> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/fall-21-reading-list/floem-a-programming-system-for-nic-accelerated-network-applications.md). # Floem: A programming system for NIC-accelerated network applications * Challenge of accelerating server applications by off-loading applications to a programmable network card * Solution: language compiler runtime that makes it easier for developers to explore different NIC offloading designs * Recently: CPU can no longer keep up with the network * Offload the computation to programmable network card that sits between the CPU and the network * Traditionally: cannot be programmed * Now: NIC * Wimpy multi-core processor * Cavium LiquidIO * Netronome Agilio * Mellanox BlueField * Field-programmable gate array (FPGA) * Microsoft catapult * NetFPGA * Reconfigurable match table (RMT) * NIC offload * Offload computation * Fast path processing: filtering, classifying, caching, etc. * Transformation: encryption / decryption, compression etc * Steering * Congestion control * Offloading computation to a NIC requires a large amount of effort * Why? have to deal with distributed heterogenous system * CPU and NIC have o cache coherence * NIC can access CPU memory via PCIe * Hard to manage and optimize for * Heterogeneity in the system * NIC: slower cores, less power; faster cores higher power, instruction, memory hierarchy * Space of offload designs * Example: key-value store * CPU core, each handle one distinct set * NIC to steer the packet to the right CPU core: **key-based steering** * Can increase throughput * **Using NIC as cache** * 3x power efficiency * Require: enough memory on NIC * No one-size-fit-all offload. Non-trivial to predict which offload is best * System, and performance objectives * Challenge: packet marshaling (define what fields to send, copy those fields) * Tedious & error-prone, hinder exploration of different offload designs * Challenge: communication strategies * No steering, key-based steering, separate GET & SET * Exploring different offload designs requires huge amount of efforts! * Floem * **DSL** makes it easy to explore alternative offloads * **Compiler** minimizes communication and generates efficient code * **Runtime** manages data transfer over PCIe * Language overview * Data-flow programming model * Extend to support: heterogeneity, parallelism * Contributions * **Goal: explore offload designs** * **Inferred data transfer** * **Logical-to-physical queue mapping** * **Caching construct** * Goal: Integration with existing app * Interface to external programs * Compiler 7 Runtime * ![](/files/UAG8l5ioWJwaVBL6g4iZ) * Example * ![](/files/U2XgrIrURdWBA310xFzK) * Element class: allow users to embed C code to define a computation of that element * ![](/files/MHWnPVkaCpn5meBTvgsD) * One thread processes one packet at a time * Data parallelism * Annotate the segment with multiple cores * Pipeline parallelism * Multiple segments, insert queues * Offload: device * First: inferred data transfer * Solution: infer fields to send * **Per-packet** state: a packet and its metadata can be accessed anywhere in the program * **Compiler** infers which fields of packet and metadata to send * Second: logical-to-physical queue mapping * Observation: different communication strategies can be expressed by mapping logical queues to physical queues * Degrees of resource sharing * Dynamic packet steering * Packet ordering * Solution: **queue construct** with explicit logical-to-physical queue mapping * I.e. Queue(channels=2, instances=3) * Logical queues, physical queues * Example * No steering: queue construct with one physical instance * Key-based steering: multiple physical queue instances, specify which CPU cores get the data from which queue, and specify how to steer by assigning queue id to the per-packet state * Third: caching construct * Difficult to implement a complete cache protocol * Maintain consistency of data on NIC and CPU * High performance * ![](/files/AvtoRU1FOz8z8jOPiwYV) * ![](/files/Mafu0zYvqkybNRV4XGcz) * Runtime & Communication * Runtime: responsible for managing the communication over the PCIe * Hard thing: manage data synchronization between the NIC and CPU * For performance, need a lot of optimizations in order to achieve high-throughput data transfer * I/O batching * Overlapping DMA operations with useful computation * Decouple the queue logic (in queue library) from the data sync and the DMA optimizations * Less compact and more modular * Can swap up queue library but reuse the queue sync layer * Evaluation * Help the programmers explore different offload designs? * Server: with smart NIC, without smart NIC * Case study: key-value store * Code relevant to communication * Floem program: 15 lines --> C program: 240 lines * Add cache in Floem (write-through cache --> write-back cache) * Other: distributed real-time data analytics * First offload: worse than CPU-only * Second offload: 96% improvement with 23 lines of code * Take-away: high-level programming abstractions * Control implementation strategies * Avoid low-level details * Result: minimal changes to explore different designs * Questions * Storage limitation of the NIC * Cavium: pretty big memory (4GB memory), fitting nicely with it * Might not be the right strategy * Allow you to try different offloading technologies * Multiple communication? multiple times back and forth * Can we have multiple communications? * Language allows you to use how many queues as you want * Whether that is the best strategy to use? * Best to minimize the communication * Congestion control beyond single-packet * TCP stack offload * One thing: implement the entire stack in the NIC, but use a lot of resources, and only left with small number of resources to offload other applications * Right thing: offload some parts (fast-path processing), but handle the flow state carefully (share between NIC and CPU)