> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/shivarams-group/group-papers/lists/wavelet-efficient-dnn-training-with-tick-tock-scheduling.md).

# Wavelet: Efficient DNN Training with Tick-Tock Scheduling

![](/files/-MZJHrcvkurLhL0_80Tt)

![](/files/-MZJI2TizWyLnlVOVye7)

![](/files/-MZJI5jFAkCwiVp84Q5B)

![](/files/-MZJIEQ9PlY_p9VFtvJl)

1. All-reduce 
2. Parameter server 

![](/files/-MZJIkXbMCo4GXykSsCO)

* Why?
  * Cluster-level
  * Might be more fragmentation 
  * Not something about single task utilization 

![](/files/-MZJJgXFy6cTOa9fLTEL)

* Not using all resources all the time 

![](/files/-MZJLDw2NZC84OyGv3NK)

Gandiva:

* cluster-level
* But does not improve a single job's performance 
* Single job also takes the same time 

![](/files/-MZJLjcaAdA5Wd7vfw3t)

* Increase inter-batch parallelism 

Gandiva:

* Multi-jobs and single jobs 

![](/files/-MZJMJg-RKJLQQu_rZnE)

Pipedream: minimizing the communication there 

![](/files/-MZJN23lrzALtlNppBHL)

![](/files/-MZJNvlpnqgIlGPg1Ro3)

Version of the model that is read? 

![](/files/-MZJOfmIK64BGEKI-OPj)

![](/files/-MZJQoeZdz8q49q0Ydno)

![](/files/-MZJQsRX_Szk6Ojj2bpz)

### Slides