# Floem: A programming system for NIC-accelerated network applications * Challenge of accelerating server applications by off-loading applications to a programmable network card * Solution: language compiler runtime that makes it easier for developers to explore different NIC offloading designs * Recently: CPU can no longer keep up with the network * Offload the computation to programmable network card that sits between the CPU and the network * Traditionally: cannot be programmed * Now: NIC * Wimpy multi-core processor * Cavium LiquidIO * Netronome Agilio * Mellanox BlueField * Field-programmable gate array (FPGA) * Microsoft catapult * NetFPGA * Reconfigurable match table (RMT) * NIC offload * Offload computation * Fast path processing: filtering, classifying, caching, etc. * Transformation: encryption / decryption, compression etc * Steering * Congestion control * Offloading computation to a NIC requires a large amount of effort * Why? have to deal with distributed heterogenous system * CPU and NIC have o cache coherence * NIC can access CPU memory via PCIe * Hard to manage and optimize for * Heterogeneity in the system * NIC: slower cores, less power; faster cores higher power, instruction, memory hierarchy * Space of offload designs * Example: key-value store * CPU core, each handle one distinct set * NIC to steer the packet to the right CPU core: **key-based steering** * Can increase throughput * **Using NIC as cache** * 3x power efficiency * Require: enough memory on NIC * No one-size-fit-all offload. Non-trivial to predict which offload is best * System, and performance objectives * Challenge: packet marshaling (define what fields to send, copy those fields) * Tedious & error-prone, hinder exploration of different offload designs * Challenge: communication strategies * No steering, key-based steering, separate GET & SET * Exploring different offload designs requires huge amount of efforts! * Floem * **DSL** makes it easy to explore alternative offloads * **Compiler** minimizes communication and generates efficient code * **Runtime** manages data transfer over PCIe * Language overview * Data-flow programming model * Extend to support: heterogeneity, parallelism * Contributions * **Goal: explore offload designs** * **Inferred data transfer** * **Logical-to-physical queue mapping** * **Caching construct** * Goal: Integration with existing app * Interface to external programs * Compiler 7 Runtime * ![](/files/UAG8l5ioWJwaVBL6g4iZ) * Example * ![](/files/U2XgrIrURdWBA310xFzK) * Element class: allow users to embed C code to define a computation of that element * ![](/files/MHWnPVkaCpn5meBTvgsD) * One thread processes one packet at a time * Data parallelism * Annotate the segment with multiple cores * Pipeline parallelism * Multiple segments, insert queues * Offload: device * First: inferred data transfer * Solution: infer fields to send * **Per-packet** state: a packet and its metadata can be accessed anywhere in the program * **Compiler** infers which fields of packet and metadata to send * Second: logical-to-physical queue mapping * Observation: different communication strategies can be expressed by mapping logical queues to physical queues * Degrees of resource sharing * Dynamic packet steering * Packet ordering * Solution: **queue construct** with explicit logical-to-physical queue mapping * I.e. Queue(channels=2, instances=3) * Logical queues, physical queues * Example * No steering: queue construct with one physical instance * Key-based steering: multiple physical queue instances, specify which CPU cores get the data from which queue, and specify how to steer by assigning queue id to the per-packet state * Third: caching construct * Difficult to implement a complete cache protocol * Maintain consistency of data on NIC and CPU * High performance * ![](/files/AvtoRU1FOz8z8jOPiwYV) * ![](/files/Mafu0zYvqkybNRV4XGcz) * Runtime & Communication * Runtime: responsible for managing the communication over the PCIe * Hard thing: manage data synchronization between the NIC and CPU * For performance, need a lot of optimizations in order to achieve high-throughput data transfer * I/O batching * Overlapping DMA operations with useful computation * Decouple the queue logic (in queue library) from the data sync and the DMA optimizations * Less compact and more modular * Can swap up queue library but reuse the queue sync layer * Evaluation * Help the programmers explore different offload designs? * Server: with smart NIC, without smart NIC * Case study: key-value store * Code relevant to communication * Floem program: 15 lines --> C program: 240 lines * Add cache in Floem (write-through cache --> write-back cache) * Other: distributed real-time data analytics * First offload: worse than CPU-only * Second offload: 96% improvement with 23 lines of code * Take-away: high-level programming abstractions * Control implementation strategies * Avoid low-level details * Result: minimal changes to explore different designs * Questions * Storage limitation of the NIC * Cavium: pretty big memory (4GB memory), fitting nicely with it * Might not be the right strategy * Allow you to try different offloading technologies * Multiple communication? multiple times back and forth * Can we have multiple communications? * Language allows you to use how many queues as you want * Whether that is the best strategy to use? * Best to minimize the communication * Congestion control beyond single-packet * TCP stack offload * One thing: implement the entire stack in the NIC, but use a lot of resources, and only left with small number of resources to offload other applications * Right thing: offload some parts (fast-path processing), but handle the flow state carefully (share between NIC and CPU) --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/fall-21-reading-list/floem-a-programming-system-for-nic-accelerated-network-applications.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.