# Building An Elastic Query Engine on Disaggregated Storage * Traditional shared-nothing architectures * Data partitioned across servers * Each server handles its own partition * Problems * Hardware-workload mismatch * Data re-shuffle during elasticity * Fundamental issue: tight coupling of compute & storage * Query Engines on Disaggregated Storage * Decouple compute and persistent storage * Independent scaling of resources * Work explores * Design aspects * Data-driven insights * Future directions * System focus on: snowflake * Warehousing as a service, in production for over 5 years, 1000s of customers, millions of queries per day * Statistics from 70 million queries over 14 day * Diversity of Queries * Read only --> 28% * Write only --> 13% * R/W --> 59% * Persistent data read/written varies over several orders of magnitude within each class * Query distribution over time * Read-only query load varies significantly over time * High-level architecture * Virtual warehouse: abstraction for computational resources, under the hood --> set of VMs, distributed execution of queries * Customers --> different warehouses * Decouples compute from persistent storage (S3) * Intermediate data: distributed join, or group by --> ephemeral storage * Ephemeral storage system key features * Co-located with compute in VWs * Opportunistic caching of persistent data (hide access latency) * Elasticity without data re-shuffle * Intermediate data characteristics * Data size: variation over 5 orders of magnitude * Difficult to predict intermediate data sizes upfront * Decouple compute & ephemeral storage? * Persistent data caching * Intermediate data volume: peak, average * Opportunistic caching of persistent data in ephemeral storage system, hides latency of S3 access * How to ensure consistency? * Each file assigned to unique node, consistent hashing * Write-through caching * Analysis, future directions in paper * Elasticity * Persistent storage: easy, offloaded to S3 * Compute: easy, pre-warmed pool of VMs * Ephemeral storage - challenging, due to co-location with compute * Back to shared-nothing architecture problem (data re-shuffle) * Lazy consistent hashing * Locality aware task scheduling * Elasticity without data re-shuffle * Lazy consistent hashing * Elasticity * Resource scaling by up to 100x needed at times * What time-scale? * Granularity of warehouse elasticity >> changes in query load * Take-away: need finer-grained elasticity in order to better match demand * Resource utilization * System-wide resource utilizations: significant room for improvement in resource utilizaiton * Virtual Warehouse abstraction * Good performance isolation * Trade-off: low resource utilization * Solution: finer-grained elasticity with current design, or move to resource shared model * Alternate design: resource sharing * Pricing model: per-hour usage for warehouses * Move to per-second pricing --> less cost-effective * Solution: move to resource shared model * Better resource utilization * Helps support elasticity * Challenges * Maintaining isolation guarantees * Shared ephemeral storage system * Sharing cache * No pre-determined lifetime * Co-existence with int. data * Elasticity without violating isolation * Possible cross-tenant interference * Need private address-spaces for tenants --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/fall-21-reading-list/building-an-elastic-query-engine-on-disaggregated-storage.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.