> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/networking/index/reading-list/legoos-a-disseminated-distributed-os-for-hardware-resource-disaggregation.md).

# LegoOS: A Disseminated Distributed OS for Hardware Resource Disaggregation

#### Hardware Resource Disaggregation 

![](/files/SLdzW8WAjAugdNnGMyK9)

![](/files/c3l2S5FZCPSStasVh9ya)

* Resource packing is difficult 
* More heterogeneous hardware: go into a server (not planned before hand)
* Poor elasticity: add / remove / reconfigure 
* Fault tolerance
  * CPU memory controller: whole server is down 

 

![](/files/TDBooXxpYeJJaRorRL2T)

![](/files/xQbKmjiNlhF3EqukeqBU)

![](/files/2b6PhywvoAVvnM8QAHJs)

* Why possible now?
  * Network is faster
  * More processing power at device: smartNIC, smartSSD, PIM
  * Network interface closer to device 

#### Kernel Architectures for Resource Disaggregation 

* Can existing kernels fit? 

![](/files/xK5LFQPZqDBJDfa8AOPe)

* Existing kernels don't fit
  * Remote resources 
  * Distributed resource management (network-partition)
  * Fine-grained failure handling (each component can fail independently) 
* Key idea: when hardware is disaggregated, the OS should be also! 

![](/files/8s7eMItMjLhIPyeDBUvD)

![](/files/saMbFeXq0XpMWyRKHFy3)

#### LegoOS: the first disaggregated OS 

* Outline 
  * Abstraction 
  * Design Principles
  * Implementations and Emulations 
* How should LegoOS appear to users?
  * as a set of virtual nodes (vNodes) 
    * Similar semantics to virtual machines
    * Can run on multiple processor, memory, and storage components 

![](/files/5AzgWhW5LIRlBMxJ46Vn)

![](/files/v2Jb2W1WAcJGGQOTfpDF)

![](/files/qZzarlxp0HFIPh5laHF2)

![](/files/OXw3cty5tl2RwDWWxBz5)

![](/files/CYmaziMlIFLJgtTJ5VGp)

![](/files/W7yaaq08Om937YQUEFgk)

![](/files/WcNVdntlfHHQqlrwtFs5)

![](/files/Lkb5DdRFt90TVoUxtdZ5)

* 1.3x to 1.7x slowdown when disaggregating devices with LegoOS
  * To gain better resource packing, elasticity, and fault tolerance! 

### Paper: LegoOS 

* Key idea: disaggregated, network-attached hardware components can improve resource utilization, elasticity, heterogeneity, and failure handling in data centers, however no existing OSes or software systems can properly manage it 
* New OS model: splitkernel 
  * Key: when hardware is disaggregated, the OS should be also 
    * Breaks traditional OS functionalities into monitors, each monitor manages a hardware component, virtualizes and protects its physical resources 
      * Lossely-coupled, communicate with each other
    * Run monitors at hardware components 
    * Message passing across non-coherent components
    * Global resource management and failure handling  
* LegoOS: 
  * appear to users as a set of distributed servers (vNodes)
  * separate OS functionalities: process monitor, memory monitor, storage monitor 
  * Challenges 
    * How to deliver good performance when application execution involves the access of network partitioned disaggregated HW and current network is slower than local buses?
    * How to locally manage individual HW components with limited HW resources?
    * How to manage distributed HW resources?
    * How to handle a component failure without affecting others?
    * What abstraction to expose to users? How to support existing datacenter applications? 
  * Solution: hardware + software
    * separate process, memory, and storage functionalities
      * Moves all hardware memory functionalities to mComponents (e.g., page tables, TLBs), leaves only caches at the pComponent side --> each mComponent can choose its own memory allocation technique and v to p memory addr mapping 
      * pComponent: virtual caches 
      * Separating memory for performance and for capacity: utilize locality and leave a small amount of memory (e.g., 4GB) at each pComponent 
        * ExCache  
    * monitors run at HW components and fit device constraints 
    * comparable performance to monolithic Linux servers
    * Efficient resource management and memory failure handling (space + performance)
    * Easy-to-use, backward compatible user interface 
    * Support common Linux system call interfaces 

![Splitkernel architecture ](/files/6owTiVR8Miu3BZ2WIgt9)

![](/files/s65sussOhdq6fk039JaU)

#### Process, memory, storage management 

* Process monitor: runs in the kernel space of a pComponent and manages pComponent's CPU cores and ExCache 
* Memory monitor: 
  * mComponents data: anonymous memory (i.e., heaps, stacks), memory-mapped files, and storage buffer caches 
  * Manages both virtual and physical addr spaces, their allocation, deallocation, and memory address mappings; perform actual memory read and write 
    * GMM: assign a home mComponent to each new process at its creation time 
  * Two level approaches 
    * home mComponent: coarse-grained, high-level virtual memory allocation decisions 
    * other mComponent: fine-grained virtual memory allocation 
  * Optimizations: delays physical memory allocation until write time 

![](/files/zJXAHGGeQYwpiO6iDPKF)

* Storage management 
  * Hierarchical file interface 
  * Stateless storage server design, each I/O to the storage server contains all the information needed to fulfill this request 

#### Global Resource Management 

* Two-level resource management mechanism 
  * Three global resource managers: process, memory and storage --> coarse-grained global resource allocation and load balancing 
  * At low level: each monito can employ its own policies and mechanisms to manage its local resources 

#### Reliability and Failure Handling 

* Memory reliability (focus) 
  * One primary mComponent, one secondary mComponent, and a backup file in sComponent for each vma 
  * Maintains a small append-only log at the secondary mComponent and replicate the vma tree 
  * Flushing backup to sComponent