> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/shivarams-group/group-papers/lists/wavelet-efficient-dnn-training-with-tick-tock-scheduling.md).

# Wavelet: Efficient DNN Training with Tick-Tock Scheduling

https\://mlsys.org/virtual/2021/oral/1586

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJHrcvkurLhL0_80Tt%2Fimage.png?alt=media\&token=ab3d7b77-4bda-4fff-8888-37f56a2fbf7d)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJI2TizWyLnlVOVye7%2Fimage.png?alt=media\&token=fcd390dc-d837-46e6-b670-9c94c0329802)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJI5jFAkCwiVp84Q5B%2Fimage.png?alt=media\&token=4275405c-fc7f-4e72-8c23-6f10610d09c8)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJIEQ9PlY_p9VFtvJl%2Fimage.png?alt=media\&token=6b037589-8811-413c-bb68-28ab32d7e339)

1. All-reduce 
2. Parameter server 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJIkXbMCo4GXykSsCO%2Fimage.png?alt=media\&token=5093fc22-5f2d-4c75-8101-d3b890434869)

* Why?
  * Cluster-level
  * Might be more fragmentation 
  * Not something about single task utilization 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJJgXFy6cTOa9fLTEL%2Fimage.png?alt=media\&token=6d66e95a-d562-47dd-8195-cb1b142524fb)

* Not using all resources all the time 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJLDw2NZC84OyGv3NK%2Fimage.png?alt=media\&token=5109948e-857c-4524-ba65-64fca59e0423)

Gandiva:

* cluster-level
* But does not improve a single job's performance 
* Single job also takes the same time 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJLjcaAdA5Wd7vfw3t%2Fimage.png?alt=media\&token=63d2c83d-bde4-4fa5-9586-88bf61550c17)

* Increase inter-batch parallelism 

Gandiva:

* Multi-jobs and single jobs 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJMJg-RKJLQQu_rZnE%2Fimage.png?alt=media\&token=7911e42f-5f50-4848-ba54-d18d7f33e559)

Pipedream: minimizing the communication there 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJN23lrzALtlNppBHL%2Fimage.png?alt=media\&token=6405c999-ea87-4b43-9a72-dfdd4f16973d)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJNvlpnqgIlGPg1Ro3%2Fimage.png?alt=media\&token=e7d2ae4b-418d-429e-9abf-f01d70c71ce4)

Version of the model that is read? 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJOfmIK64BGEKI-OPj%2Fimage.png?alt=media\&token=de7c4a9f-e465-4889-9d05-723c5ebd272a)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJQoeZdz8q49q0Ydno%2Fimage.png?alt=media\&token=a2aeddd9-30e3-4294-98e4-8a14b187b04b)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MZJHAw0xlVfRswD973a%2F-MZJQsRX_Szk6Ojj2bpz%2Fimage.png?alt=media\&token=ac52e222-0f72-46ed-95ca-f8d07d1947f4)

### Slides