---
description: http://today.wisc.edu/events/view/157998
---

# Enabling Hyperscale Web Services

* Web services will not grow&#x20;
* Radical shift in hyperscale computing&#x20;
  * Urgent need: redesign computer systems to enable futuristic web services&#x20;
  * Now: **target individual stack layers**&#x20;
  * Software stacks&#x20;
    * Web service application --> fragmented into small units&#x20;
    * OS & Software Stacks
  * Hardware&#x20;
    * Custom hardware (E.x. GPU)&#x20;
    * Smart NIC&#x20;
  * Disconnected Research&#x20;
* Bridge the software and hardware work to enable future web services&#x20;
  * How to redesign SW to be aware of the HW constraints? &#x20;
  * How to re-architect HW for new S paradigms post-Moore?&#x20;

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MVwpMSxpZdKEG94tzcB%2F-MVwrwQhSZhN-gK_CsM_%2Fimage.png?alt=media\&token=21926268-f1e5-479f-ba55-31f3ebbe1570)

#### Research Approach

* Characterization: understand service behavior, overheads, emerging trends&#x20;
* Novel solutions: self-navigate design space based on characterization insights&#x20;
* Real systems&#x20;

#### HW-aware SW threading&#x20;

* OSDI '18
* User request
  * Front-end microservice&#x20;
  * Hashing microservice
  * Ads microservice&#x20;
  * Caching microservice&#x20;
  * Ranking microservice&#x20;
  * End-to-end response latency&#x20;
    * Interacting in an extremely complicated way&#x20;
    * Microsecond-scale system overheads significantly affect microservices&#x20;
* Microsecond overheads: accessing OS/NW
  * Urgent need to understand SW threading interactions with OS/NW&#x20;
* Software Threading Dimensions -- taxonomy of threading models&#x20;
  * Block v.s. Poll
  * In-line v.s. dispatch
  * Synchronous v.s. asynchronous&#x20;
* Latency tradeoffs across threading models&#x20;
  * Poll better than block&#x20;
  * In-line poll faces contention; dispatch poll with one poller is best&#x20;
    * Support increase in load&#x20;
  * Dispatch block is best at high load as it does not waste CPU cycles&#x20;
  * No single threading model works best at all loads!&#x20;
* Automatic Load Adaption&#x20;
  * Exploit trade-offs among threading models at run-time&#x20;
* micro-Tune \[OSDI '18]&#x20;
  * Design challenges: synchronization, when to switch, interact with NW, thread hops, scale thread pools?
    * Abstracts threading design from service code to seamlessly manage threading&#x20;

#### Micro-tune System Design&#x20;

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MVwpMSxpZdKEG94tzcB%2F-MVwvircNuDULLFiM4Ot%2Fimage.png?alt=media\&token=7dc162ff-d4a9-473f-b50c-d6cb292ce0a5)

* Piecewise linear model
  * Input?&#x20;
    * Load, and other features&#x20;
    * Heuristics: queuing patterns and other&#x20;
  * Ecosystem changes?
    * Will need to retrain&#x20;

### Hardware&#x20;

* Diverse accelerators will break the bank&#x20;
  * Customized platforms are expensive&#x20;
  * Hardware homogeneity&#x20;
  * Avoids testing overhead &#x20;
* Facebook
  * Web
  * Feed1, Feed2
  * Ads1, Ads2
  * Cache1, Cache 2 (similar to key-value store)
* Question
  * HW overheads?
  * SW overheads are worth building HW for?&#x20;
* Observation (HW)
  * Great diversity in overheads across microservices&#x20;
  * New surprisingly dominant HW overheads&#x20;
    * Code footprints (in code cache misses, TLB misses)&#x20;
* Observation (SW)&#x20;
  * Orchestration: I/O processing, compression, encryption&#x20;
  * Orchestration overheads are significant & common across microservices&#x20;

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MVwpMSxpZdKEG94tzcB%2F-MVwxcYauqXi_19pnMZK%2Fimage.png?alt=media\&token=f0c14834-aacd-45ce-9209-e9484e1f1264)

#### How to mitigate microservice overheads?

* Great diversity in HW overheads across microservices
* Microservice orchestration logic is a significant, common SW overhead&#x20;

### Approach 1: Make better use of existing hardware

#### Soft SKUs \[ISCA' 19]

* Tune coarse HW & OS knobs on commodity HW&#x20;
* SoftSKUs achieve performance efficiency on cheap commodity HW&#x20;
  * Serves 2.7B users
  * Saved cost
  * Reduced footprint&#x20;

### Approach 2: Design and deploy new hardware architectures - accelerators&#x20;

* Accelerating the "encryption" orchestration logic&#x20;
  * HW vender only improves the accelerators, ignoring the end-to-end overhead&#x20;
    * SW interaction overhead&#x20;
    * ? Question&#x20;
      * Accelerator (PCIE)&#x20;
        * Offload, and context switches overhead&#x20;
* HW acceleration&#x20;
  * Design, Test, Deploy&#x20;
  * But performance is .. due to perf. bounds from software interactions with hardware&#x20;
* Analytical Model for HW Acceleration&#x20;
  * Simplicity&#x20;
  * **Accelerometer: Analytical Model \[ASPLOS' 20]**
    * HW accelerator&#x20;
    * SW threading design
    * Metrics
    * Offload transfer via interface&#x20;
  * E.x. synchronous offload&#x20;
    * Queuing delay, offload transfer latency...
  * E.x. asynchronous&#x20;
    * Accelerator cycles do not critically affect speedup
    * But context switch penalty&#x20;
  * Other
    * Waits for offload ack?
    * Offload size?
    * Offload preparation time?
    * Many threads? Context switches?&#x20;
* Accelerometer: estimates&#x20;
  * TPU, GPU?
    * Type of accelerometer we can build&#x20;
    * Opportunities&#x20;
      * Stage of the HW pipeline&#x20;
        * Type of the accelerator&#x20;
        * Richer model to consider paradigm &#x20;

#### Data movement

* Rethink I/O interactions in life of emerging HW and SW paradigms&#x20;
  * I/O path efficient event notification mitigating microsecond stalls&#x20;
* Rethink SW for new HW emerging technologies: non-volatile memory, accelerators
* New app paradigms & domains&#x20;
  * Serverless: locality issues
  * AR/VR, IoT: data mgmt. issues&#x20;
* ML for SW-HW design, design space exploration resource management&#x20;
  * scheduling, mgmt&#x20;
* Intersectionality, equity, fairness: first-order HW-SW design metrics&#x20;
  * discriminate based on demographics, or ethic&#x20;

Verify implementation hard?&#x20;

* slice and dice in way that still makes sense&#x20;

