# High Performance Data Center Operating Systems * Today's I/O devices are fast and getting faster * Can't we just use Linux? * Kernel mediation is too heavy weight * Arrakis (OSDI 14): separate the OS control and data plane * OS architecture that separates the control and data plane, for both networking and storage * How to skip the kernel? * ![](/files/yLFbMsGpdpfq6nhYtffG) * Design goals * Streamline network and storage I/O * Eliminate OS mediation in the common case * Application-specific customization vs. kernel one size fits all * Keep OS functionality * Process (container) isolation and protection * Resource arbitration, enforceable resource limits * Global naming, sharing semantics * POSIX compatibility at the application level * Additional performance gains from rewriting the API * Strata (SOSP 17) * File system design for low latency persistence (NVM) and multi-tier storage (NVM, SSD, HDD) * Storage diversification * NVDIMM: byte-addressable (cache-line granularity IO), direct access with load/store instructions * SSD: large erasure block (hardware GC overhead), random writes cause 5-6x slowdown by GC * Let's build a fast server * Key value store, database, file server, mail server ... * Requirements * Small updates dominate * NVM is too fast, kernel is the bottleneck * Dataset scales up to many terabytes * To save the cost, need a way to use multiple device types: NVM, SSD, HDD * Using only NVM is too expensive * For low-cast capability with high performance, must leverage multiple device types * Block-level caching manages data in blocks, but NVM is byte-addressable! * Updates must be crash consistent * Applications struggle for crash consistency * Today's file systems: limited by old design assumptions * Kernel mediates every operation * NVM is too fast, kernel is the bottleneck * Tied to single type of device * For low-cost capacity with high performance, must leverage multiple device types * Aggressive caching in DRAM, only write to device when you must (fsync) * Struggles for crash consistency * Strata: a cross media file system * Performance: especially small, random IO * Fast user-level device access * Capacity: leverage NVM, SSD & HDD for low cost * Transparent data migration across different media * Efficiently handle device IO properties * Simplicity: intuitive crash consistency model * In-order, sync IO * No fsync() required * Main design principle * LibFS: log operations to NVM at user-level * Fast user-level access * In-order, sync IO * Kernel FS: digest and migrate data in kernel * Async digest * Transparent data migration * Shared file access * TCP as a Service / FlexNIC / Floem (ASPLOS 15, OSDI 18) * OS, NIC, and app library support for fast, agile, secure protocol processing * Let's build a fast server * Small RPCs dominate * Enforceable resource sharing * Agile protocol development * Cost-efficient hardware * RDMA: read/write to (limited) region of remote server memory, no CPU involvement on the remote node (fast if app can use programming model) * Limitations: what if you need remote application computation (RPC)? lossless model is performance-fragile * Smart NICs: NIC with array of low-end CPU cores * If compute on the NIC, maybe don't need to go CPU? * Applications in high speed trading * **Step 1: Build a faster kernel TCP in software** * Q: why RPC over Linux TCP is so slow? * OS: hardware interface, highly optimized code path, buffer descriptor queues (no interrupts in common case, maximize concurrency) * Following: OS transmit packet processing * TCP layer: move from socket buffer to IP queue * Lock socket, congestion / flow control limit, fill in TCP header, calculate checksum, copy data, arm re-transmission timeout * IP layer: firewall, routing, ARP, traffic shaping * Driver: move from IP queue to NIC queue * Allocate and free packet buffers * Now: multiple sync kernel transitions * Parameter checks and copies * Cache pollution, pipeline stalls * TCP Acceleration as a Service (TaS) * TCP as a user-level OS service * SRIO-V to dedicated cores * Scale number of cores up/down to match demand * Optimized data plane for common case operations * Application uses its own dedicated cores * Avoid polluting application level cache * To the application, per-socket tx/rx queues with doorbells * Streamline common-case data path * Remove unneeded computation from data path * Congestion control, timeouts oper RTT * Minimize per-flow TCP state * Linearized code * Enforce IP level access control on control plane at connection setup * **Step 2: TaS data plane can be efficiently built in hardware** * FlexNIC design principles * RPCs are the common case: kernel bypass to application logic * Enforceable per-flow resource sharing: data plane in hardware, policy in kernel * Agile protocol development: protocol agnostic, offload both kernel and app packet handling * Cost-efficient: minimal instruction set for packet processing * FlexTCP: H/W accelerated TCP * Fast path is simple enough for FlexNIC model * Applications directly access NIC for RX/TX * Software slow-path messages NIC state * Streamlines NIC processing --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/fall-21-reading-list/high-performance-data-center-operating-systems.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.