# Building An Elastic Query Engine on Disaggregated Storage

* Presents SnowFlake design and implementation, along with a discussion on how recent changes in cloud infrastructure (emerging hardware, fine-grained billing, etc.) have altered the many assumptions 

### Presentation 

* Traditional shared-nothing architectures 
  * Data partitioned across servers
  * Each server handles its own partition 
* Hardware-workload mismatch
* Data re-shuffle during elasticity 
* Fundamental issue in share-nothing architecture: tight coupling of compute and storage 
* Solution: decouple compute and persistent storage 
  * Independent scaling of resources 
  * Analyze: query engine on disaggregated storage 
    * Design aspects 
    * Data-driven insights 
    * Future directions 
  * System: SnowFlake --> statistic from 70 million queries over 14 day period 
* Characteristics 
  * Diversity of queries 
    * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FzqJO8tLvL1VtQG9UcL4s%2Fimage.png?alt=media\&token=d53c9719-54ff-4395-8de2-6566c2031e21)
      * Read-only: 28% 
      * Write-only: 13%
      * Read-write: 59% 
      * Three distinct query classes (vary in orders of magnitude) 
  * Query distribution over time 
    * Read-only query load varies significantly overtime 
    * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FP6zUbiu2jsr9oDLCbkcx%2Fimage.png?alt=media\&token=3f9aefb6-d404-4fb1-8b98-4c54fc1741ea)
* High-level architecture 
  * Virtual warehouse
    * Abstractions for computational resources 
    * Under the hood --> set of VMs 
    * Distributed execution of queries 
  * Different customers --> different warehouses 
    * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FDm3gQKDPyVHPPb6LIbvk%2Fimage.png?alt=media\&token=cdce6aa1-29f4-4917-8120-1a6bcbc9563d)
  * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FWVX21uZzJ4C7k5iSblns%2Fimage.png?alt=media\&token=146e45ce-1266-49bd-b9b4-3db0ea73a982)
    * Ephemeral storage system key features 
      * Co-located with compute in VWs 
        * Use both DRAM and local SSDs 
      * Opportunistic caching of persistent data 
        * Hide S3 access latency 
      * Elasticity without data re-shuffle 
  * Intermediate data characteristics 
    * Intermediate data sizes --> variation over 5 orders of magnitude 
      * ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FgEMLBxKaExiLdv1B3gHq%2Fimage.png?alt=media\&token=b803f611-d5cb-40d1-baf8-6e3dd97cc2ae)
    * Difficult to predict intermediate data sizes upfront 
      * A bit different than ours 
    * Possibility: decouple compute & ephemeral storage 
  * Persistent data caching 
    * Intermediate data volume --> peak 
    * Opportunistic caching
      * How to ensure consistency?
        * Each file assigned to unique node 
          * Consistent hashing
        * Write-through caching 
  * Elasticity 
    * Persistent storage - easy, offloaded to S3 
    * Compute - easy, pre-warmed pool of VMs 
      * Request --> take VMs from the pool 
    * Ephemeral storage - challenging, due to co-location with compute 
      * Back to shared-nothing architecture problem (data re-shuffle) 
      * Reshuffle --> re-hashing to ensure consistency 
    * Lazy consistent hashing 
      * **Locality aware task** scheduling
      * Elasticity without data re-shuffle 
        * Waits until a task that actually reads it
  *  Do customers exploit elasticity in the wild? 
    * Resource scaling by up to 100x needed at times 
    * At what time-scales are warehouses resized? 
      * Granularity of warehouse elasticity >> changes in query load 
      * Need finer-grained elasticity in order to better match demand 
    * Resource utilization 
      * System-wide resource utilization: CPU, memory, network-tx, network-rx
      * Virtual warehouse abstraction
        * Good performance isolation
        * Trade-off: low resource utilization 
      * Significant room for improvement in resource utilization
      * Solution #1: finer-grained elasticity with current design
      * Solution #2: resource shared model 
  * Resource sharing 
    * Per-second pricing --> pre-warmed pool not cost effective!
    * Sol #2: move to resource shared model
      * Statistical multiplexing
        * better resource utilization
        * helps support elasticity 
      * Challenges 
        * Maintaining isolation guarantees 
        * Shared ephemeral storage system 
        * Sharing cache 
          * No pre-determined lifetime 
          * Co-existence with int. data 
        * Elasticity without violating isolation 
          * Cross-tenant interference 
          * Need private address spaces for tenants 
  * Findings are general applicable to other distributed system on top of disaggregated storage