B4: Experience with a Globally-Deployed Software Defined WAN

https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf

WAN links are typically provisioned at 30-40% avg utilization
- WAN links are expensive, packet loss is typically thought unacceptable
- High-end, specialized equipment that place a premium on high availability
- Treat all bits the same
Google WAN
- Control over everything (apps, servers, LANs, edge)
- Bandwidth-intensitve app performs large-scale data copies from one site to another
- Anticipate no more than a few dozen data center deployment, making control of bandwidth feasible
Design centers around
- Accepting failures as inevitable and common events, effects are exposed to the app
  - ??
- Switch hardware that exports a simple interface to program forwarding table entries under central control
Use cases: routing protocols and centralized traffic engineering

Switch hardware: forward traffic
Site controller layer: NCS hosting both OpenFlow controllers (OFC) and Network Control Applications (NCA)
Globaly layers: logically centralized applications (e.g. SDN gateway, TE servers)

Build their own hardware
Insight: don't need deep buffers, very large forwarding tables, hardware support for availability [with cost and complexity]
Motivation: careful endpoint managements, few set of DCs, switch failures typically result in software rather than hardware failure, no existing platform could support an SDN deployment

Functionality runs on NCS in the site controller layer collocated with the switch hardware
Paxos: handles leader selection for all control functionality
- At each site, perform application-layer failure detection
- When a majority of the Paxos servers detect a failure, they elect a new leader among the remaining set of available servers

Routing application proxy (RAP)
- RAP translates from RIB entries forming a network-level view of global connectivity to the low-level hardware tables used by the OpenFlow data plane
  - RAP translates each RIB entry into two OpenFlow tables, a Flow table which maps prefixes to entries into a ECMP Group table.

Goal: share bandwidth among competing applications possibly using multiple paths
Objective function: deliver max-min fair allocation to applications
- maximizes utilization as long as further gain in utilization is not achieved by penalizing fair share of applications
Notion
- Flow Group (FG): TE cannot operate on granularity of individual applications; aggreage application to a Flow Group defined as {src site, dest site, QoS}
Bandwidth functions
- specifies the bandwidth allocation to an application given the flow’s relative priority on an arbitrary, dimensionless scale, which we call its fair share
- decides from administrator-specified static weights
Optimization algorithm: achieve similar fairness of LP optimal and at least 99% of the bandwidth utilization with 25x faster performance relative to LP
- (1) Tunnel Group Generation
  - Allocate bw to FGs using bandwidth functions to prioritize at bottleneck edges
Example

Some additional discussions on dependencies and failures
- Dependencies among ops
- Synchronizing TED between TE and OFC: compute difference, Session ID
- Ordering issues: sequence ID
- TE op failures: (Dirty/Clean) bit for each TED entry

Deployment Take
- i) topology aggregation significantly reduces path churn and system load
- ii) even with topology aggregation, edge removals happen multiple times a day
- iii) WAN links are susceptible to frequent port flaps and benefit from dynamic centralized management
TE Ops Performance: monthly distribution of ops issued, failure rate, latency distribution for two main TE operations (Tunnel addition and Tunnel Group mutation)
Impact of Failures
- A single link failure
- An encap switch failure and separately the failure of its neighboring transit router (much longer convergence time)
- An OFC failover
TE Algorithm Evaluation
- 14% throughput increase, main benefits come during periods of failure or high demand
Link utilization and hashing
- Most WANs: 30-40% utilization
- But B4: ~100%

Planned maintenance operation --> one of the new switches was inadvertently manually configured with the same ID as an existing switch --> link flaps, switches declare interfaces down, breaking BGP adjacencies with remote cites
Lessons
- Scalability and latency of the packet IO path between OFC and OFA is critical
- OFA should be async and multi-threaded
- Need additional performance profiling and reporting
- With TE, they "fail open" --> it is not possible to distinguish between physical failures and the associated data plane
  - But the compromise as: hardware is more reliable than control software
  - Require application-level signals of broken connectivity to disambiguate between WAN hardware and software failures
- TE server must be adaptive to failed / unresponsive OFCs when modifying TGs that depend on creating new Tunnels
- Most failures involve the inevitable human error that occurs in managing large, complex system
- Critical to measure system performance at its breaking point with published envelopes regarding system scale

Is the paper / problem / insight efficient?
- Efficiency argument
  - Make a general observation about the traffic elasticity
  - "Shift" in mindset: networking at efficiency (this is done at compute, etc..)
  - Test of time award of Sigcomm
  - AT&T had large WANs but they were never aware of this --> be able to exploit this is important
- Use cases
  - Enterprise internally in DC (only has 5% utilization)
  - Cloud tenants
- Traffic Engineering (TE)
  - Improve utilization
  - "Scalability" for more # of DCs is a question
Industry paper?
- Some student answers
  - 1) FS paper feels like other papers can built upon and extend

Last updated 2 years ago

Was this helpful?