# B4: Experience with a Globally-Deployed Software Defined WAN
### Problem
* WAN links are typically provisioned at 30-40% avg utilization
* WAN links are expensive, packet loss is typically thought unacceptable
* High-end, specialized equipment that place a premium on high availability
* Treat all bits the same
* Google WAN
* Control over everything (apps, servers, LANs, edge)
* Bandwidth-intensitve app performs large-scale data copies from one site to another
* Anticipate no more than a few dozen data center deployment, making control of bandwidth feasible
* Design centers around
* Accepting failures as inevitable and common events, effects are exposed to the app
* ??
* Switch hardware that exports a simple interface to program forwarding table entries under central control
* Use cases: routing protocols and centralized traffic engineering
### Background
* Two types of WANs
* User-facing network: peers / exchange traffic with the other Internet domains
* Requirement: support a wide range of protocols, physical topology will be more dense, in content delivery must support highest level of availability
* B4: connectivity between data centers
* Workload: user data copies for availability, remote storage access for computation over inherently distributed data sources, large-scale data push synchronization state across multiple DCs
* Ordered in increasing volume, decreasing latency sensitivity, and decreasing overall priority
* Design
* Elastic bandwidth demand
* Moderate number of sites
* End application control
* Cost sensitivity
### Architecture
* Switch hardware: forward traffic
* Site controller layer: NCS hosting both OpenFlow controllers (OFC) and Network Control Applications (NCA)
* Globaly layers: logically centralized applications (e.g. SDN gateway, TE servers)
#### Switch design
* Build their own hardware
* Insight: don't need deep buffers, very large forwarding tables, hardware support for availability \[with cost and complexity]
* Motivation: careful endpoint managements, few set of DCs, switch failures typically result in software rather than hardware failure, no existing platform could support an SDN deployment
#### Network control
* Functionality runs on NCS in the site controller layer collocated with the switch hardware
* Paxos: handles leader selection for all control functionality
* At each site, perform application-layer failure detection
* When a majority of the Paxos servers detect a failure, they elect a new leader among the remaining set of available servers
#### Routing
* Routing application proxy (RAP)
* RAP translates from RIB entries forming a network-level view of global connectivity to the low-level hardware tables used by the OpenFlow data plane
* RAP translates each RIB entry into two OpenFlow tables, a Flow table which maps prefixes to entries into a ECMP Group table.
*
```
```
### Traffic Engineering
* Goal: share bandwidth among competing applications possibly using multiple paths
* Objective function: deliver **max-min fair allocation** to applications
* maximizes utilization as long as further gain in utilization is not achieved by penalizing fair share of applications
* Notion
* Flow Group (FG): TE cannot operate on granularity of individual applications; aggreage application to a Flow Group defined as {src site, dest site, QoS}
* **Bandwidth functions**
* specifies the bandwidth allocation to an application given the flow’s relative priority on an arbitrary, dimensionless scale, which we call its *fair share*
* decides from administrator-specified static weights
* q: flow detection? what about dynamic
* Bandwidth functions are configured, measured and provided to TE via *Bandwidth Enforcer*
* an FG’s bandwidth function is a piecewise linear additive composition of per-application bandwidth functions
* Each FG multiplexes multiple application demands from one site to another
* Max-min objective of TE is on this per-FG fair share dimension
* Bandwidth enforcer also aggregates bandwidth functions across multiple applications
* **Optimization algorithm:** achieve similar fairness of LP optimal and at least 99% of the bandwidth utilization with 25x faster performance relative to LP
* (1) Tunnel Group Generation
* Allocate bw to FGs using bandwidth functions to prioritize at bottleneck edges
* (2) Group Quantization
* Split ratios in each TG to match granualrity supported by the switch HW tables
* Example
* 
### TE protocol, OpenFlow, how it's implemented
* 
*
```
```
* 
* Some additional discussions on dependencies and failures
* Dependencies among ops
* Synchronizing TED between TE and OFC: compute difference, Session ID
* Ordering issues: sequence ID
* TE op failures: (Dirty/Clean) bit for each TED entry
### Deployments and Evals
* Deployment Take
* i) topology aggregation significantly reduces path churn and system load
* ii) even with topology aggregation, edge removals happen multiple times a day
* iii) WAN links are susceptible to frequent port flaps and benefit from dynamic centralized management
* TE Ops Performance: monthly distribution of ops issued, failure rate, latency distribution for two main TE operations (Tunnel addition and Tunnel Group mutation)
* Impact of Failures
* A single link failure
* An encap switch failure and separately the failure of its neighboring transit router (much longer convergence time)
* An OFC failover
* A TE server failover
* Disabling/enabling TE
* TE Algorithm Evaluation
* 14% throughput increase, main benefits come during periods of failure or high demand
* Link utilization and hashing
* Most WANs: 30-40% utilization
* But B4: \~100%
#### Lessons learned from an outage
* Planned maintenance operation --> one of the new switches was inadvertently manually configured with the same ID as an existing switch --> link flaps, switches declare interfaces down, breaking BGP adjacencies with remote cites
* Lessons
* Scalability and latency of the packet IO path between OFC and OFA is critical
* OFA should be async and multi-threaded
* Need additional performance profiling and reporting
* With TE, they "fail open" --> it is not possible to distinguish between physical failures and the associated data plane
* But the compromise as: hardware is more reliable than control software
* Require application-level signals of broken connectivity to disambiguate between WAN hardware and software failures
* TE server must be adaptive to failed / unresponsive OFCs when modifying TGs that depend on creating new Tunnels
* Most failures involve the inevitable human error that occurs in managing large, complex system
* Critical to measure system performance at its breaking point with published envelopes regarding system scale
#### Some takes
* Is the paper / problem / insight efficient?
* Efficiency argument
* Make a general observation about the traffic elasticity
* "Shift" in mindset: networking at efficiency (this is done at compute, etc..)
* Test of time award of Sigcomm
* AT\&T had large WANs but they were never aware of this --> be able to exploit this is important
* Use cases
* Enterprise internally in DC (only has 5% utilization)
* Cloud tenants
* Cloud / hot potateo routing, tiers of BW paid
* Traffic Engineering (TE)
* Improve utilization
* "Scalability" for more # of DCs is a question
* AT\&T has done the centralized controller, but twick routing
* Industry paper?
* Some student answers
* 1\) FS paper feels like other papers can built upon and extend
* 2\) Take insights from industry paper
* 3\) "Access" to things are own by the industries
* 4\) No comparisons to other solutions
* Sylvia's take:
* When reviewing the paper @ SIGCOMM, we need a good problem statement, insight, and eval insights
* In industries, it's often not realistic to compare it with some other solutions
* Industries are important participants; sigcomm reviewers said we need more industry paper
* "practical", "at-scale", "vendor-support"
