> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/conference/index/nsdi-21/distributed-system/caerus-nimble-task-scheduling-for-serverless-analytics.md).

# Caerus: NIMBLE Task Scheduling for Serverless Analytics

https\://www\.usenix.org/conference/nsdi21/presentation/zhang-hong

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBqk_EdAhlOzrK-la3%2Fimage.png?alt=media\&token=f8817abe-576c-4f2b-b82e-3d45ba3fee7d)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBqn2ez1h_XjJED6MJ%2Fimage.png?alt=media\&token=51e3132b-2125-4b52-a935-60e6456b97c6)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBqwEKMtcYWBLDLZ4J%2Fimage.png?alt=media\&token=717f54f2-083d-4c4f-8fba-f24f0bc35d05)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBr8szuG3KuFQo7gn8%2Fimage.png?alt=media\&token=95934cd9-c1e6-4f4f-8b3c-342ff72a6256)

* Lazy: Barrier between map and reduce stage 
  * Cost: aggregated duration of all the tasks 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBrNE-hY1Vpf1ONXnc%2Fimage.png?alt=media\&token=19af4c39-eeba-4550-b97a-2d1fda24490b)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBr_VYLntRy26U1v4x%2Fimage.png?alt=media\&token=79351ad9-9993-4aae-9216-522624a74942)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBriYhOgMzQHzDVRAe%2Fimage.png?alt=media\&token=32a3d0f6-f541-433d-9882-fd46314e37c8)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBrlnYoTvUQwSzu2GW%2Fimage.png?alt=media\&token=9bc42113-b2f2-4c52-b4aa-157189af4329)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBs1UtHvUsj-1N5yf-%2Fimage.png?alt=media\&token=55805b7e-b611-4cdb-975e-2bc641fbb70e)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBs89CQjxP8hRC5PP1%2Fimage.png?alt=media\&token=1bb3816f-2ee9-4d30-be5b-4cbb46b25591)

* Pipeline breaker: an operator that produces the first output only after all its input has been processed
* Map: one step
* Reduce:
  * Read all mapper output
  * Produce reduce operations 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBsOfgYmiAHcnTRrHV%2Fimage.png?alt=media\&token=0f00be64-ad70-4be8-a751-5784171c14b8)

* Mappers produce, reducers get at the same time 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBsVOu70J4PJjRnxXj%2Fimage.png?alt=media\&token=94e0abe0-82fe-4bc1-9d64-b0cb685e6f94)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBsY-47IDVVUbSjBiF%2Fimage.png?alt=media\&token=8d6da215-0962-48df-bbf1-3ffdd65220b2)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBsj0KKMxbG5hVe85F%2Fimage.png?alt=media\&token=08d3cf7f-478e-431f-a108-61ebb1ee1504)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBsmj5C9wOFpE0lYL0%2Fimage.png?alt=media\&token=29f83841-0400-497c-b6ae-dcd7dabe2f03)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBspg2YthRAPp1wWi1%2Fimage.png?alt=media\&token=eac796dc-a9b7-4122-9374-3c74034fb0e3)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBssgNmSQ3Z6E9tRj-%2Fimage.png?alt=media\&token=d3d9be2a-7c86-4546-a68e-63b7ee9d2496)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBsvjQL5MwvybyjrUT%2Fimage.png?alt=media\&token=5668a2fa-7e42-47c2-b22b-2c5a87582ce4)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBt-rJVJGvLUwhyFxH%2Fimage.png?alt=media\&token=39f778e7-ff41-42b8-a82c-a49b04e690d0)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYBqNsquRGeM3JeAUsF%2F-MYBt4UJqNgK5MHnV6oE%2Fimage.png?alt=media\&token=c01205d2-54f7-4867-b073-b230b2478bc6)