# Network-accelerated Distributed Machine Learning for Multi-Tenant Settings

### Presentation 

* Background: 
  * ML is becoming ubiquitous in software industry 
  * ML models are trained with large volumes of data typically over shared infrastructure 
  * DML 
    * More compute cycles
    * Process lots of distributed data 
    * But require frequent synchronization (i.e. SGD: after each 'iteration') 
  * Architecture 
    * Parameter server: Google DistBelief, Microsoft Adam 
      * Async v.s Sync 
    * P2P MPI 
      * Synchronous 
* Key factors affecting performance of distributed ML training 
  * Async v.s Sync SGD
  * A-SGD:
    * Version of the model being updated v.s. used to update the gradients 
    * Communication intensive 
      * Compute update (\~100ms)
      * Fetch model (\~600ms) 
      * **R1**: Reduce network load at the server 
    * Stragglers affect convergence 
      * Halving bandwidth on 10% --> 35% in iteration through convergence 
      * **R2**: Bound delays in the presence of stragglers 
    * Fault tolerance has huge overhead 
      * Chain replication (every worker update forwards to replica) 
        * Outgoing nic: 
          * Carries both models to the workers
          * And model updates to the replica 
      * Directly forward to replica 
        * Asynchrony + stateful leads to inconsistency 
      * **R3:** Non-divergent replication without server overhead 
  * S-SGD
    * With PS architecture 
      * Server NIC overload is a bottleneck for large models
      * Stragglers increase time per-iteration
    * MPI architecture 
      * Ring reduce algorithms are typically use: but assume homogeneous network infrastructure 
      * Compute & network stragglers increase time per-iteration
    *  **R4**: bandwidth-aware aggregation 

#### Existing works 

* R1
  * Infrequent updates to the server 
    * Workers aggregate locally, then transmit to server
    * But: cannot aggregate updates across multiple workers 
* R2
  * Dropping delayed updates 
    * updates: transmitted to the servers anyway, does not reduce the server load 
* R3 
  * No prior work 
* R4
  * Hierarchical AllReduce, but assume static bandwidth setting 

#### MLFabric 

* Contributions 
  * Prove bounded delay helps convergence 
  * Network aware ordering and aggregation of updates 
    * Helps bound delay
    * Reduce network load at parameter server
  *  Model replication strategy 
    * Guarantee bounded consistency b/w server and replica 
* **Network aware ordering and aggregation of updates**
  *  Re-ordering updates 
    * Buffering updates 
      * Bounded delay (R2) at the cost of stale read 
    * Time-shared schedules 
      * Update scheduled for later can be aggregated off-server 
      * SJF + Time-sharing satisfies R1 and R2 
* Architecture 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2F7pw3CKh7QNR9uqrmgFjK%2Fimage.png?alt=media\&token=8cc5eb22-f77c-47de-a6a0-f7e453f8e9bb)

* Ordering and Aggregation algorithm 
  * Scheduler batches transfer requests to server 
    * Information: model version, size of the update, norm of gradients 
  * Orders them iteratively in SJF fashion to server 
    * Completion time is determined in network aware fashion 
    * State update in each iteration
    * Update that cannot meet deadline are dropped 
  * Queued updates are grouped, aggregated and then sent to server 
    * Minimize total completion time 
    * Aggregator is chosen randomly

### Paper 

* DML: compute and network contention, resulting in stragglers 
  * Current system takes too simplistic a view of the network (having either fixed bandwidth between all the workers or as a blackbox with unknown inter-worker bandwidth) 
* ML-fabric: contention-aware DML system
  * order transfers to improve convergence 
  * opportunistically aggregates them at idle DML workers to improve resource efficiency 
  * replicates them to support new notions of fault tolerance 
  * systematically account for compute stragglers and network contention
  * implemented as a communication layer between DML applications and AllReduce libraries 
* Key idea
  * Control update delays 
    * transfer available updates at non-overlapping times, reserving bandwidth per update, and explicitly ordering them 
    * worker's delay = the difference between the server model version a worker pulled and computed on, versus the server model version at the time the worker's compute gradient is applied 
  * Dynamically aggregating or dropping updates 
  * Replicating updates for fault tolerance