> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/shivarams-group/group-papers/fluid-resource-aware-hyperparameter-tuning-engine.md).

# Fluid: Resource-aware Hyperparameter Tuning Engine

![](/files/-MZsLdilqvzEu-pJHjfJ)

![](/files/-MZsLj-hsiz8S3S0ACvx)

![](/files/-MZsLlyDl7ZygPjcQ-R2)

* Successive Halving 
  * Workers underutilized, only one job to run 

![](/files/-MZsMCVAuMmWpntdBsPG)

* More efficient, but there're also some problems 

![](/files/-MZsMOgvx4UN4Fhujweo)

![](/files/-MZsMZSMbm42jQLMnt76)

* Goal: Utilized & Useful work? 
* Resource-aware hyperparameter tuning 
  * Previous work: Hypersched (resource management), but more specialized on the algorithm itself 

![](/files/-MZsNHk2H1HUnapDFSQK)

* Intra-GPU sharing (pack jobs in single GPU)
  * Current schedule: FIFO queue to manage to jobs 

### Design and Algorithms 

![](/files/-MZsNyD7d1-OoFVq4qCa)

* Intra: packing several jobs on single GPU 

![](/files/-MZsOK6nR9yBv7VlXRKE)

![](/files/-MZsOzRwLDey2u7ZBkat)

![](/files/-MZsPLLC0usKwHa2bk5m)

* Very helpful when people read the paper (very helpful in communicating the ideas) 
* Overhead of packing? 

![](/files/-MZsPiBoqoxTFdPrtLW0)

* Packing overhead of doing the placement 
  * Limit the number of packing trails 

![](/files/-MZsPxs4Wz1IQFO-c61H)

![](/files/-MZsREk6jbPfkpdew3tK)

* Model fits on the single worker 

![](/files/-MZsRjkJfCOwlKwSvObA)

* Makespan of all the trials 

![](/files/-MZsRoFE3Wco18ldEkQ6)

* Improvement more prominent when applying for asynchronous version 
* Intuition: 
  * If the runtime is very scaled 
* Parameters they are tuning
  * Learning rate, dropout rate 
  * Number of layers 
  * Batch size 
* Failures 
  * Know reasonable ranges 

![](/files/-MZsTJ5pXbzo7uUn1ah4)

* Multiple hyperparameter jobs 
  * Multiple trial groups 
  * But extra parallelism there? 
  * Different things in the same trial group 
* Variability 
  * Some 
  * More? 
* Space sharing & Parallelism 
  * Automatically parallelism, don't need anything from hyperparameter 
  * Queue problem, but in tight space