# The Demikernel and the future of kernal-bypass systems * I/O devices are getting faster, but cpus are not * The CPU is increasingly a bottleneck in datacenters * OS kernels consume a big percentage of CPU cycles, OS kernels can no longer keep up with datacenter applications or I/O * Solution: kernel-bypass I/O devices * ![](/files/5ziJvyjgjdhKRQ784wWa) * ![](/files/kqBjUNZm7zUXbADX1gBn) * Kernel-bypass gives applications direct access to I/O devices, bypassing the OS kernel on every I/O * Pros and Cons of kernel-bypass * Pros: widely-available (GCE, AWS, and Azure all support kernel-bypass, most I/O devices support it), effective (i.e. 128x improvement using RDMA) * Cons: hard-to-use (porting an application is complex and expensive), limited (only used by specialized applications today, like scientific computing, but not at scale in data-center today) * Outline * Introduction * Background * Demikernel overview, API, liboses, evaluation ### Intro * Kernel-bypass works similarly to hardware virtualization * I/O device provides H/W support that the VM can directly issue I/O * IOMMU translation from guest to actual physical devices * Technologies: SR-IOV * Kernel-bypass * I/O device bypass the OS kernel * Application can directly issue I/O * IOMMU translate from virtual application level (user-level) addresses to machine addresses * Widely deploy: DPDK, RDMA ![](/files/kimaCxLznwmGegZD23dE) * Unlike H/W virtualization, the OS kernel does more than multiplex hardware resources ![](/files/FVSP8SfaoLaopuP9JTMC) * Take a look at networking: widely used kernel bypass technologies ![](/files/Bc3VNedrs8MrJBtJxtwL) * Above: architecture of modern system * Kernel bypass: move those features to I/O device like address translation, device multiplexing and things like that, but I/O is not capable of supporting everything * No high-level abstractions, no TCP, no socket, and lots of other things * ![](/files/ke4SqrxXgsqhYdGKqghq) * Gap? * One option: own custom messaging layer (networking stack) * If every application needs to build their own * Re-use OS networking stack and move it up to user space? * Not fast enough for kernel-bypass devices, traditional OS is built to work with ms-level NICs * mTCP * Explicitly built for kernel bypass * Implement TCP in user space, offer socket and interface same as POSIX * But only oriented with throughput, even slower than going through the Linux kernel * Different devices implement different interfaces and OS services based on hardware capabilities * ![](/files/8UsxakWm56PrPIC2v7n5) * RDMA NIC: capable, provide a lot of stuff * But depend heavily on network to do congestion control, hard to control * DPDK: virtual nics and nothing else * Spend a lot of CPU cycles to run a full networking stack in user-level * Programmable devices * Interface? * **There is no standard kernel-bypass API or architecture** ### **Demi-kernel** * Demi-kernel project * What is it? A new kernel-bypass OS \[architecture] * Design goals * Standardized kernel-bypass API * Flexible architecture for heterogenous devices (OS features, but still provide a uniform experience for application programmer) * Single microsecond i/o processing: i/o is very fast, os cannot add any more overhead ![](/files/vqBjRP8bs1NEGHAce5Qe) ![](/files/g3ajZBAMo6Stv9lRtlVP) * Demikernel supplies a different libos for each device * ![](/files/eUujApIyRBXcLTKUdis4) * Single unified interface and architecture * New hardware --> build new demi-kernel libOS to support them #### API * Key features * I/O queue API: with scatter-gather arrays to minimize latency * Queues replace UNIX pipes and scokets * In-memory queue similar to Go channels * Each push and pop should be a complete I/O, so the Demikerel libOS can immediately issue the I/O if possible * qtoken and wait: to block on I/O operations for finer-grained scheduling * Push and pop are async and return a qtoken for blocking on I/O computation * Wait blocks on one or more I/O operations and returns the result * Native zero-copy from the application heap with use-after-free protection * Critical for latency * Pushed SGA buffers are libOS-owned until qtoken returns; however, the app can free the buffers at any time * LibOS allocates buffers for incoming I/O, transferring ownership of the SGA buffers on pop. The app is responsible for freeing the buffers * Design principles * Shared execution contexts for minimizing latency * Demikernel libOSes perform OS tasks (e.g., networking processing) on shared application threads * Minimizes latency compared to threads (mTCP, NSDI '14) or processes (SNAP, SOSP '19) * Requires cooperation: application must regularly entire the libOS (e.g., by calling wait or performing I/O) * Co-routines for lightweight multiplexing of OS tasks * Light-weight scheduling abstractions * Multiplex OS tasks (e.g., packet processing, sending acks, allocating receive buffers) with application execution * Implemented with built-in C++ * Cooperatively scheduled by a Demikernel scheduler that separates runnable and blocked co-routines * Integrated memory allocator for transparent memory registration and use-after-free protection #### Challenges * Kernel-bypass scheduling * When to do OS work vs running the application * How to prioritize OS work based on the kernel-bypass device * How to scale to hundreds of co-routines * How to make fine-grained scheduling decisions in a few nanoseconds #### Summary * Demikernel is a low-latency kernel-bypass OS * Demikernel provides a portable API and flexible architecture for heterogenous kernel-bypass devices * There are still many interesting open problems in kernel-bypass OS design #### Questions * How can demikernel interacts with resources such as sockets without transitioning into the underlying OS? * RDMA, DPDK: libraries, user-level direct access to the hardware NIC. Hardware devices allow us to safely access the network device * Feasible to implement POSIX API on top of demi kernel's qAPI? Do you think one should? * Not hard to do POSIX API. Make design decisions: when to push, when to issue the I/O * Great way to slowly move applications onto kernel-bypass * Congestion control? * No, working on it. Thought about doing (MIT) --> extensible CC module in rust, porting that? * Example of the # of running for round-trip, Redis (applications). Other applications that does evaluations on? * Ported version of Redis, sub-module * Not run Redis on the most recent version * Kernel bypass wouldn't work with containers? * Need user-level driver to talk with the device * Match the versions * But same OS problem, now visible to your applications * How do you see these primitives extending to the network, in distributed context. Two demi-kernels? * LibOS: customized * Framing for TCP, UDP: receive them as a packet * TCP: don't know when start and end * What happens as the payloads increase in size? * Latency goes up a little bit? Throughput goes up * Nothing unexpected * RDMA: up to 1G, segmentation offloaded onto the NIC --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/fall-21-reading-list/the-demikernel-and-the-future-of-kernal-bypass-systems.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.