> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/conference/index/sosp-21/smartnic/linefs-efficient-smartnic-offload-of-a-distributed-file-system-with-pipeline-parallelism.md).

# LineFS: Efficient SmartNIC offload of a distributed file system with pipeline parallelism

https\://sands.kaust.edu.sa/papers/linefs.sosp21.pdf

### Presentation 

* Growing DFS host resource consumption 
  * DFS is popular in cloud and HPC --> applications are consolidated 
    * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FQHFqZF0V3tNjDWSNX5FO%2Fimage.png?alt=media\&token=343e8d13-ef0d-4d67-86ee-f4ebebbb8302)
  * Resource contention between DFS and co-running applications 
    * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FFUg9uOb3yZZaFz3Iz42n%2Fimage.png?alt=media\&token=32c664bb-118c-4bbc-8a72-1ea2b6a142a1)
* Problem: performance interference 
  * Host resource contention degrades the performance of both 
    * 1\) DFS application 2) co-running applications 
* Solution: offload DFS to SmartNIC 
  * To reduce interference 
  * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FzuctqWzgsX2wuVyPTV96%2Fimage.png?alt=media\&token=b979ebc5-0bd6-4e2d-b798-db313d144921)
  * Challenges 
    * C1: high access latency from SmartNIC CPU to host PM 
    * Wimpy SmartNIC architecture 
    * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FCds1N8xlM4FIN9CBukRl%2Fimage.png?alt=media\&token=6ebd70d3-7650-4dc1-b8ff-b48f04b3c160)
* LineFS: SmartNIC offload of DFS with pipeline parallelism 
  * Goal: minimizing **host application interference** and **DFS slowdown** 
  * Design principles 
    * D1: persist-and-publish 
      * Persist data / metadata: latency critical task (fast host CPU)
      * Background publish: deferrable task (Wimpy SmartNIC CPU) 
    * D2: pipeline parallelism 
      * Increase throughput of offloaded tasks 
        * D2-1: publishing pipeline 
        * D2-2: replication pipeline 

#### D1: persist-and-publish 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FwitqyAq8lQnaPxgzKXyc%2Fimage.png?alt=media\&token=bd0c6514-932e-4b45-af1b-9a4bdf73a441)

#### D2-1: pipeline parallelism - publishing 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FXlbkIlUEzdvs0YyKL76Z%2Fimage.png?alt=media\&token=8a74293b-b52c-4ddd-ae51-671ffe809f3e)

#### D2-2: pipeline parallelism - replication 

* Replication? "persisting data to all the nodes on fsync"
* Persist data in advance: use SmartNIC resources 
* Reduce fsync latency: most of the data has already been persisted 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FE41krykArXXQN3Z1uBJR%2Fimage.png?alt=media\&token=999cbadf-519e-4832-b196-456dd3fd9add)

Other design ideas in the paper:

* Linearizability & prefix crash consistency 
* Leveraging data-path processing 
* Availability during host OS failure 
* Shared file management with lease mechanism 

Evaluation:

* Does LineFS provide adequate offloaded DFS performance?
* Does LineFS alleviate interference? 

Conclusion

* Increasing DFS host resource use causes contention 
  * Degrading applications' performance
  * Unpredictable DFS operation latencies 
* Naive DFS offload to SmartNIC has high overhead
  * Long distance to host PM
  * Wimpy processing power 
* **LineFS: Efficient DFS SmartNIC offload**
  * Design principles: persist-and-publish, pipeline parallelism 
  * 90% lower LevelDB latency, 79% higher Filebench throughput versus Assise under host resource contention