> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/reading-groups/wisr/scaling-distributed-machine-learning-within-network-aggregation.md).

# Scaling Distributed Machine Learning within-Network Aggregation

### Abstract 

* SwitchML: reduces the volume of exchanged data by aggregating the model updates from multiple workers in the network 

### Motivation 

* Network-bound workload 
  * Advances in GPU
  * The ratio of communication to computation to the workload itself has shifted 

### Challenge

* Switch: 
  * Limited computation
  * Limited storage 
  * No floating points 
  * Packet loss 

### Design

* Combined switch-host architecture 
* Pool-based streaming aggregation 
* Quantized integer operations 
* Failure-recovery protocol 
* In-switch RDMA implementation