# Network-accelerated Distributed Machine Learning for Multi-Tenant Settings

### Presentation 

* Background: 
  * ML is becoming ubiquitous in software industry 
  * ML models are trained with large volumes of data typically over shared infrastructure 
  * DML 
    * More compute cycles
    * Process lots of distributed data 
    * But require frequent synchronization (i.e. SGD: after each 'iteration') 
  * Architecture 
    * Parameter server: Google DistBelief, Microsoft Adam 
      * Async v.s Sync 
    * P2P MPI 
      * Synchronous 
* Key factors affecting performance of distributed ML training 
  * Async v.s Sync SGD
  * A-SGD:
    * Version of the model being updated v.s. used to update the gradients 
    * Communication intensive 
      * Compute update (\~100ms)
      * Fetch model (\~600ms) 
      * **R1**: Reduce network load at the server 
    * Stragglers affect convergence 
      * Halving bandwidth on 10% --> 35% in iteration through convergence 
      * **R2**: Bound delays in the presence of stragglers 
    * Fault tolerance has huge overhead 
      * Chain replication (every worker update forwards to replica) 
        * Outgoing nic: 
          * Carries both models to the workers
          * And model updates to the replica 
      * Directly forward to replica 
        * Asynchrony + stateful leads to inconsistency 
      * **R3:** Non-divergent replication without server overhead 
  * S-SGD
    * With PS architecture 
      * Server NIC overload is a bottleneck for large models
      * Stragglers increase time per-iteration
    * MPI architecture 
      * Ring reduce algorithms are typically use: but assume homogeneous network infrastructure 
      * Compute & network stragglers increase time per-iteration
    *  **R4**: bandwidth-aware aggregation 

#### Existing works 

* R1
  * Infrequent updates to the server 
    * Workers aggregate locally, then transmit to server
    * But: cannot aggregate updates across multiple workers 
* R2
  * Dropping delayed updates 
    * updates: transmitted to the servers anyway, does not reduce the server load 
* R3 
  * No prior work 
* R4
  * Hierarchical AllReduce, but assume static bandwidth setting 

#### MLFabric 

* Contributions 
  * Prove bounded delay helps convergence 
  * Network aware ordering and aggregation of updates 
    * Helps bound delay
    * Reduce network load at parameter server
  *  Model replication strategy 
    * Guarantee bounded consistency b/w server and replica 
* **Network aware ordering and aggregation of updates**
  *  Re-ordering updates 
    * Buffering updates 
      * Bounded delay (R2) at the cost of stale read 
    * Time-shared schedules 
      * Update scheduled for later can be aggregated off-server 
      * SJF + Time-sharing satisfies R1 and R2 
* Architecture 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2F7pw3CKh7QNR9uqrmgFjK%2Fimage.png?alt=media\&token=8cc5eb22-f77c-47de-a6a0-f7e453f8e9bb)

* Ordering and Aggregation algorithm 
  * Scheduler batches transfer requests to server 
    * Information: model version, size of the update, norm of gradients 
  * Orders them iteratively in SJF fashion to server 
    * Completion time is determined in network aware fashion 
    * State update in each iteration
    * Update that cannot meet deadline are dropped 
  * Queued updates are grouped, aggregated and then sent to server 
    * Minimize total completion time 
    * Aggregator is chosen randomly

### Paper 

* DML: compute and network contention, resulting in stragglers 
  * Current system takes too simplistic a view of the network (having either fixed bandwidth between all the workers or as a blackbox with unknown inter-worker bandwidth) 
* ML-fabric: contention-aware DML system
  * order transfers to improve convergence 
  * opportunistically aggregates them at idle DML workers to improve resource efficiency 
  * replicates them to support new notions of fault tolerance 
  * systematically account for compute stragglers and network contention
  * implemented as a communication layer between DML applications and AllReduce libraries 
* Key idea
  * Control update delays 
    * transfer available updates at non-overlapping times, reserving bandwidth per update, and explicitly ordering them 
    * worker's delay = the difference between the server model version a worker pulled and computed on, versus the server model version at the time the worker's compute gradient is applied 
  * Dynamically aggregating or dropping updates 
  * Replicating updates for fault tolerance 


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://sliu583.gitbook.io/blog/networking/index/reading-list/network-accelerated-distributed-machine-learning-for-multi-tenant-settings.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.