# From Laptop to Lambda: Outsourcing Everyday Jobs to Thousands of Transient Functional Containers

### Presentation 

* Occasional task that needs 10,000 cores (in the cloud)
* Many others share this dream 
  * Outsourcing computation
  * Cluster-computing framework 
  * Burst-parallel cloud functions  
* Limited speed-ups, high costs, limited applicability 
* gg: framework and a toolkit that makes it practical to outsource everyday applications using thousands of parallel threads in cloud services 
* Challenges of outsourcing applications to the cloud 
  * **Software dependencies must be managed** 
    * With data flow frameworks like Spark, Hadoop, and Dryad, the software dependencies remain unmanaged 
    * Need a warm cluster with everything preloaded
    * Not amenable to occasional one-off tasks 
    * A 10,000-core cluster on EC2 is expensive! 
    * **Thunk** abstractions 
      * **Lightweight container** 
      * Identifies an executable, along with its argument, environment, and input data 
      * Data is named by the hash of its content 
      * ![](/files/9byHgjJiN5ueWM3OyxJU)
        * Full functional footprint 
  * **Roundtrips to the cloud hurt performance** 
    * Current application-specific outsourcing tools perform better over fast networks: distcc, icecc, UCop 
    * The laptop is in the driver's seat! Want to take laptop out of the loop. Minimize the communication in-between 
    * ![](/files/0zBYsfMlGR0Joa5vvXv5)
    * Containers can reference each other's outputs: **linked containers (gg IR)** 
    * Graphs, and dynamic dependency graphs etc. 
  * **Cloud functions are promising, but hard to use well** 
    * The dream: renting a supercomputer by the second 
    * Warm clusters are expensive, cold clusters are slow to start 
    * 10,000 workers for 10 seconds on AWS Lambda costs \~$5! 
      * PyWren, Sprocket, Serverless MapReduce 
    * Using cloud functions is challenging 
    * gg on Lambda 
      * Faster speedup
      * Getting data to the cloud is faster (HTTP pipelining, multi-threading) 
    * Many applications require inter-function communication 
      * Current systems use indirect techniques such as using shared storage (e.g., S3) 
    * Using off-the-shelf **NAT-traversal techniques**, the Lambdas can talk to each other at speed up to 600 Mbps 
* Example: applications - software compilation 
  * Build systems are often large and complicated; very difficult to manually rewrite them in gg IR.
  * We need a system that works with existing build systems, like make, Cmake, nija, etc.
  * Technique: model substitution - a technique to extract gg IR from existing applications 
    * Idea: run the original build system, but replace every stage with a 'model' program that produces a thunk, instead of the actual output 
  * gg on AWS Lambda is 2-5x faster than icecc outsourcing to a 384-core cluster 
    * Performance doesn't increase as you get more cores, because the laptop becomes the bottleneck 
    * And much more cheaper 
  * Google Chrome: 18 mins... 


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/fall-21-reading-list/from-laptop-to-lambda-outsourcing-everyday-jobs-to-thousands-of-transient-functional-containers.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.