# Understanding host network stack overheads * Network & Host Hardware Trends * For internet and early generation datacenters * Bottlenecks were in the network * Challenge: sharing network resources * Switch buffers, switch bandwidth * Exponentially increasing bandwidth * Stagnant CPU capacity (slowdown of Moore's Law) * For high speed networks * Bottleneck have moved to the host * Challenge: sharing host resources * CPU cores * DRAM bandwidth * LLC capacity * Solutions: Linux stack optimization, RDMA, hardware offloads, userspace stacks * Goal: guide the design-space by a detailed understanding of today's stack * Methodology & Experimental Scenarios * Goal: understand CPU overheads of host network stack * Want to push the bottlenecks to the network stack * Measure: throughput, cpu utilization & breakdown, cache miss rate * Impact of various factors: * Optimization techniques: TSO/GRO, Jumbo frames, aRFS * HW configurations: DDIO: IOMMU * Traffic pattern: single, incast, one-to-one, outcast, all-to-all * Flow types: long flows, short, mixture * Network drops * Congestion control protocols * Main lessons from our study * **Bottlenecks have shifted from packet processing to data copy** * For 40 Gbps NICs, a single CPU core could saturate the access link bandwidth * Multiple cores needed to saturate 100 Gbps access link bandwidth * Possible solution: **zero-copy techniques** like TCP mmap/AF\_XDP * Implementation overhead * **The NIC DMA pipeline has become inefficient** * NIC overwrites data before application read them * High cache miss rate is the core reason for the inefficiency of NIC DMA pipeline * Large TCP buffers increase the delay from packet RX to data copy * More NIC Rx descriptors lead to a higher chance of cache eviction * Enabling IOMMU further degrades the performance * Additional per page operations * Possible solution: **TCP buffer size calculation** must take host resources (like L3 cache, packet processing latency) into account * Possible solution: **decouple** data copy and packet processing (so as to scale them independently) * Possible solution: an efficient cache replacement policy * **Host resource sharing leads to further performance degradation** * Multiple flows contending for host resources aggravates host bottleneck * Cache contention degrades the throughput per core * Possible solution: receiver-driven protocols for orchestrating receiver's caches * Bandwidth contention further degrades performance * Higher scheduling overheads * GRO benefits reduce (increase the # of flows per core) * Possible solution: **receiver-driven** transport protocols for orchestrating receiver's bandwidth * **Colocation of short and long flows further degrades performance** * When flows are collocated, both long/short flows suffer (degrades 48% / 42%) * Long and short flows have different bottlenecks * TCP/IP: overheads increase as we decrease the flow size * Data copy: won't improve performance of short flows as long flows * Possible solution: design different packet processing pipelines for short and long flows * Possible solution: design **application- and network-aware** CPU schedulers ![A Linux Network Stack Data Path ](/files/Ng3TwJOmkDYlBRmb8zi1) --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/fall-21-reading-list/understanding-host-network-stack-overheads.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.