B4: Experience with a Globally-Deployed Software Defined WAN
https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf
Problem
WAN links are typically provisioned at 30-40% avg utilization
WAN links are expensive, packet loss is typically thought unacceptable
High-end, specialized equipment that place a premium on high availability
Treat all bits the same
Google WAN
Control over everything (apps, servers, LANs, edge)
Bandwidth-intensitve app performs large-scale data copies from one site to another
Anticipate no more than a few dozen data center deployment, making control of bandwidth feasible
Design centers around
Accepting failures as inevitable and common events, effects are exposed to the app
??
Switch hardware that exports a simple interface to program forwarding table entries under central control
Use cases: routing protocols and centralized traffic engineering
Background
Two types of WANs
User-facing network: peers / exchange traffic with the other Internet domains
Requirement: support a wide range of protocols, physical topology will be more dense, in content delivery must support highest level of availability
B4: connectivity between data centers
Workload: user data copies for availability, remote storage access for computation over inherently distributed data sources, large-scale data push synchronization state across multiple DCs
Ordered in increasing volume, decreasing latency sensitivity, and decreasing overall priority
Design
Elastic bandwidth demand
Moderate number of sites
End application control
Cost sensitivity
Architecture
Switch hardware: forward traffic
Site controller layer: NCS hosting both OpenFlow controllers (OFC) and Network Control Applications (NCA)
Globaly layers: logically centralized applications (e.g. SDN gateway, TE servers)
Switch design
Build their own hardware
Insight: don't need deep buffers, very large forwarding tables, hardware support for availability [with cost and complexity]
Motivation: careful endpoint managements, few set of DCs, switch failures typically result in software rather than hardware failure, no existing platform could support an SDN deployment
Network control
Functionality runs on NCS in the site controller layer collocated with the switch hardware
Paxos: handles leader selection for all control functionality
At each site, perform application-layer failure detection
When a majority of the Paxos servers detect a failure, they elect a new leader among the remaining set of available servers
Routing
Routing application proxy (RAP)
RAP translates from RIB entries forming a network-level view of global connectivity to the low-level hardware tables used by the OpenFlow data plane
RAP translates each RIB entry into two OpenFlow tables, a Flow table which maps prefixes to entries into a ECMP Group table.
Traffic Engineering
Goal: share bandwidth among competing applications possibly using multiple paths
Objective function: deliver max-min fair allocation to applications
maximizes utilization as long as further gain in utilization is not achieved by penalizing fair share of applications
Notion
Flow Group (FG): TE cannot operate on granularity of individual applications; aggreage application to a Flow Group defined as {src site, dest site, QoS}
Bandwidth functions
specifies the bandwidth allocation to an application given the flow’s relative priority on an arbitrary, dimensionless scale, which we call its fair share
decides from administrator-specified static weights
q: flow detection? what about dynamic
Bandwidth functions are configured, measured and provided to TE via Bandwidth Enforcer
an FG’s bandwidth function is a piecewise linear additive composition of per-application bandwidth functions
Each FG multiplexes multiple application demands from one site to another
Max-min objective of TE is on this per-FG fair share dimension
Bandwidth enforcer also aggregates bandwidth functions across multiple applications
Optimization algorithm: achieve similar fairness of LP optimal and at least 99% of the bandwidth utilization with 25x faster performance relative to LP
(1) Tunnel Group Generation
Allocate bw to FGs using bandwidth functions to prioritize at bottleneck edges
(2) Group Quantization
Split ratios in each TG to match granualrity supported by the switch HW tables
Example
TE protocol, OpenFlow, how it's implemented
Some additional discussions on dependencies and failures
Dependencies among ops
Synchronizing TED between TE and OFC: compute difference, Session ID
Ordering issues: sequence ID
TE op failures: (Dirty/Clean) bit for each TED entry
Deployments and Evals
Deployment Take
i) topology aggregation significantly reduces path churn and system load
ii) even with topology aggregation, edge removals happen multiple times a day
iii) WAN links are susceptible to frequent port flaps and benefit from dynamic centralized management
TE Ops Performance: monthly distribution of ops issued, failure rate, latency distribution for two main TE operations (Tunnel addition and Tunnel Group mutation)
Impact of Failures
A single link failure
An encap switch failure and separately the failure of its neighboring transit router (much longer convergence time)
An OFC failover
A TE server failover
Disabling/enabling TE
TE Algorithm Evaluation
14% throughput increase, main benefits come during periods of failure or high demand
Link utilization and hashing
Most WANs: 30-40% utilization
But B4: ~100%
Lessons learned from an outage
Planned maintenance operation --> one of the new switches was inadvertently manually configured with the same ID as an existing switch --> link flaps, switches declare interfaces down, breaking BGP adjacencies with remote cites
Lessons
Scalability and latency of the packet IO path between OFC and OFA is critical
OFA should be async and multi-threaded
Need additional performance profiling and reporting
With TE, they "fail open" --> it is not possible to distinguish between physical failures and the associated data plane
But the compromise as: hardware is more reliable than control software
Require application-level signals of broken connectivity to disambiguate between WAN hardware and software failures
TE server must be adaptive to failed / unresponsive OFCs when modifying TGs that depend on creating new Tunnels
Most failures involve the inevitable human error that occurs in managing large, complex system
Critical to measure system performance at its breaking point with published envelopes regarding system scale
Some takes
Is the paper / problem / insight efficient?
Efficiency argument
Make a general observation about the traffic elasticity
"Shift" in mindset: networking at efficiency (this is done at compute, etc..)
Test of time award of Sigcomm
AT&T had large WANs but they were never aware of this --> be able to exploit this is important
Use cases
Enterprise internally in DC (only has 5% utilization)
Cloud tenants
Cloud / hot potateo routing, tiers of BW paid
Traffic Engineering (TE)
Improve utilization
"Scalability" for more # of DCs is a question
AT&T has done the centralized controller, but twick routing
Industry paper?
Some student answers
1) FS paper feels like other papers can built upon and extend
2) Take insights from industry paper
3) "Access" to things are own by the industries
4) No comparisons to other solutions
Sylvia's take:
When reviewing the paper @ SIGCOMM, we need a good problem statement, insight, and eval insights
In industries, it's often not realistic to compare it with some other solutions
Industries are important participants; sigcomm reviewers said we need more industry paper
"practical", "at-scale", "vendor-support"
Last updated
