# Data center TCP (DCTCP)

### Problems being solved 

#### Motivation 

* Cloud data centers: especially soft real-time applications, generate mixing workloads
  * Small predictable latency 
  * Large sustained throughput 
* In this environment, SOTA TCP protocol falls short 

#### Problems solved / improved 

* Higher throughput using less buffer space 
* High burst tolerance and low latency for short flows 
* Handles 10x the current background traffic, without impacting foreground traffic 

### Metrics of success

Partition/Aggregate workflow pattern:

* Low latency for short flows
* High burst tolerance 

Need to continuously update internal data structures of the applications: 

* High utilization / throughput for long flows 

### Key innovations 

* Measure and analyze production traffic from data centers whose network is comprised of commodity switches 
  * Impairments that hurt performance is identified, and linked to the properties of the traffic and switches 
* DCTCP that addresses these impairments to meet the need of applications 
  * Goal: switch buffer occupancies need to be persistently low, while maintaining high throughput for the long flow
  * Use Explicit Congestion Notification (ECN) 
  * Combine ECN with a novel control scheme at the sources 
    * Extract multibit feedback on congestion in the network from the single bit stream of ECN marks 

### Communications in data centers

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYbkpyf2OKZfRMo0Z4U%2F-MYbqmw571_TnBd7A8Zh%2Fimage.png?alt=media\&token=9dd36a28-e9b4-4ae2-94ee-3b8089a67c3d)

#### Partition / Aggregate (Query)

1. Motivates why latency is a critical metric
   1. Delay sensitive  
2. all-up SLA
   1. lagging instances of partition / aggregate can thus add up to threaten the all-up SLAs for queries 
   2. When a node misses its deadline, the computation continues without that response, lowering the quality of the result.
   3. Many applications find it difficult to meet these deadlines using state-of-the-art TCP, so developers often resort to complex, ad-hoc solutions
3. Missed deadline: lower quality result 

#### Short message (50KB-1MB) (Coordination, Control State)

* Delay sensitive 

#### Large flows (1MB-50MB) (Data update)

* Throughput sensitive 

### Performance impairments

* Shallow packet buffers cause three performance impairments 
  * Incast 
    * if many flows converge on the same interface of a switch over a short period of time, the packets may exhaust either the switch memory or the maximum permitted buffer for that interface, resulting in packet losses.
    * This can occur even if the flow sizes are small
    * Partition/Aggregate design pattern: as the request for data synchronizes the workers’ responses and creates incast at the queue of the switch port connected to the aggregator

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYbkpyf2OKZfRMo0Z4U%2F-MYc0Ubv7KPtgVmtzq8f%2Fimage.png?alt=media\&token=72a9f790-05ed-4151-9d22-1ef279b0be9a)

* Queue buildup 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYbkpyf2OKZfRMo0Z4U%2F-MYc1-0F9gImQV86WBPY%2Fimage.png?alt=media\&token=0cec4799-d520-41e9-91a4-b0e072f9814c)

* Buffer pressure 

### Requirements 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYbkpyf2OKZfRMo0Z4U%2F-MYc1vU3r20RB3hsNdDd%2Fimage.png?alt=media\&token=85341b0d-1e7c-48bb-80d3-e4e787f797db)

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MYbkpyf2OKZfRMo0Z4U%2F-MYc26hzmCLbJoPJKfBm%2Fimage.png?alt=media\&token=e329c944-37db-431e-b607-6b9db8be37ba)

### Algorithm

* The TCP/ECN Control Loop 
  * One batch has one ECN 

### Related assumptions (or limitations?)

* DC v.s. WAN
  * Round trip time (RTTs) 
  * Applications simultaneously need extremely high bandwidths and very low latencies 
  * Little statistical multiplexing 
  * Network: largely homogeneous and under a single administrative control 
    * Backward compatibility, incremental deployment and fairness to legacy protocols are not major concerns 
* Real rule of thumb: low variance in sending rate --> small buffers suffice