B4: Experience with a Globally-Deployed Software Defined WAN



  • WAN links are typically provisioned at 30-40% avg utilization

    • WAN links are expensive, packet loss is typically thought unacceptable

    • High-end, specialized equipment that place a premium on high availability

    • Treat all bits the same

  • Google WAN

    • Control over everything (apps, servers, LANs, edge)

    • Bandwidth-intensitve app performs large-scale data copies from one site to another

    • Anticipate no more than a few dozen data center deployment, making control of bandwidth feasible

  • Design centers around

    • Accepting failures as inevitable and common events, effects are exposed to the app

      • ??

    • Switch hardware that exports a simple interface to program forwarding table entries under central control

  • Use cases: routing protocols and centralized traffic engineering


  • Two types of WANs

    • User-facing network: peers / exchange traffic with the other Internet domains

      • Requirement: support a wide range of protocols, physical topology will be more dense, in content delivery must support highest level of availability

    • B4: connectivity between data centers

      • Workload: user data copies for availability, remote storage access for computation over inherently distributed data sources, large-scale data push synchronization state across multiple DCs

        • Ordered in increasing volume, decreasing latency sensitivity, and decreasing overall priority

      • Design

        • Elastic bandwidth demand

        • Moderate number of sites

        • End application control

        • Cost sensitivity


  • Switch hardware: forward traffic

  • Site controller layer: NCS hosting both OpenFlow controllers (OFC) and Network Control Applications (NCA)

  • Globaly layers: logically centralized applications (e.g. SDN gateway, TE servers)

Switch design

  • Build their own hardware

  • Insight: don't need deep buffers, very large forwarding tables, hardware support for availability [with cost and complexity]

  • Motivation: careful endpoint managements, few set of DCs, switch failures typically result in software rather than hardware failure, no existing platform could support an SDN deployment

Network control

  • Functionality runs on NCS in the site controller layer collocated with the switch hardware

  • Paxos: handles leader selection for all control functionality

    • At each site, perform application-layer failure detection

    • When a majority of the Paxos servers detect a failure, they elect a new leader among the remaining set of available servers


  • Routing application proxy (RAP)

    • RAP translates from RIB entries forming a network-level view of global connectivity to the low-level hardware tables used by the OpenFlow data plane

      • RAP translates each RIB entry into two OpenFlow tables, a Flow table which maps prefixes to entries into a ECMP Group table.

Traffic Engineering

  • Goal: share bandwidth among competing applications possibly using multiple paths

  • Objective function: deliver max-min fair allocation to applications

    • maximizes utilization as long as further gain in utilization is not achieved by penalizing fair share of applications

  • Notion

    • Flow Group (FG): TE cannot operate on granularity of individual applications; aggreage application to a Flow Group defined as {src site, dest site, QoS}

  • Bandwidth functions

    • specifies the bandwidth allocation to an application given the flow’s relative priority on an arbitrary, dimensionless scale, which we call its fair share

    • decides from administrator-specified static weights

      • q: flow detection? what about dynamic

    • Bandwidth functions are configured, measured and provided to TE via Bandwidth Enforcer

      • an FG’s bandwidth function is a piecewise linear additive composition of per-application bandwidth functions

        • Each FG multiplexes multiple application demands from one site to another

      • Max-min objective of TE is on this per-FG fair share dimension

    • Bandwidth enforcer also aggregates bandwidth functions across multiple applications

  • Optimization algorithm: achieve similar fairness of LP optimal and at least 99% of the bandwidth utilization with 25x faster performance relative to LP

    • (1) Tunnel Group Generation

      • Allocate bw to FGs using bandwidth functions to prioritize at bottleneck edges

    • (2) Group Quantization

      • Split ratios in each TG to match granualrity supported by the switch HW tables

  • Example

TE protocol, OpenFlow, how it's implemented

  • Some additional discussions on dependencies and failures

    • Dependencies among ops

    • Synchronizing TED between TE and OFC: compute difference, Session ID

    • Ordering issues: sequence ID

    • TE op failures: (Dirty/Clean) bit for each TED entry

Deployments and Evals

  • Deployment Take

    • i) topology aggregation significantly reduces path churn and system load

    • ii) even with topology aggregation, edge removals happen multiple times a day

    • iii) WAN links are susceptible to frequent port flaps and benefit from dynamic centralized management

  • TE Ops Performance: monthly distribution of ops issued, failure rate, latency distribution for two main TE operations (Tunnel addition and Tunnel Group mutation)

  • Impact of Failures

    • A single link failure

    • An encap switch failure and separately the failure of its neighboring transit router (much longer convergence time)

    • An OFC failover

    • A TE server failover

    • Disabling/enabling TE

  • TE Algorithm Evaluation

    • 14% throughput increase, main benefits come during periods of failure or high demand

  • Link utilization and hashing

    • Most WANs: 30-40% utilization

    • But B4: ~100%

Lessons learned from an outage

  • Planned maintenance operation --> one of the new switches was inadvertently manually configured with the same ID as an existing switch --> link flaps, switches declare interfaces down, breaking BGP adjacencies with remote cites

  • Lessons

    • Scalability and latency of the packet IO path between OFC and OFA is critical

    • OFA should be async and multi-threaded

    • Need additional performance profiling and reporting

    • With TE, they "fail open" --> it is not possible to distinguish between physical failures and the associated data plane

      • But the compromise as: hardware is more reliable than control software

      • Require application-level signals of broken connectivity to disambiguate between WAN hardware and software failures

    • TE server must be adaptive to failed / unresponsive OFCs when modifying TGs that depend on creating new Tunnels

    • Most failures involve the inevitable human error that occurs in managing large, complex system

    • Critical to measure system performance at its breaking point with published envelopes regarding system scale

Some takes

  • Is the paper / problem / insight efficient?

    • Efficiency argument

      • Make a general observation about the traffic elasticity

      • "Shift" in mindset: networking at efficiency (this is done at compute, etc..)

      • Test of time award of Sigcomm

      • AT&T had large WANs but they were never aware of this --> be able to exploit this is important

    • Use cases

      • Enterprise internally in DC (only has 5% utilization)

      • Cloud tenants

        • Cloud / hot potateo routing, tiers of BW paid

    • Traffic Engineering (TE)

      • Improve utilization

      • "Scalability" for more # of DCs is a question

        • AT&T has done the centralized controller, but twick routing

  • Industry paper?

    • Some student answers

      • 1) FS paper feels like other papers can built upon and extend

      • 2) Take insights from industry paper

      • 3) "Access" to things are own by the industries

      • 4) No comparisons to other solutions

    • Sylvia's take:

      • When reviewing the paper @ SIGCOMM, we need a good problem statement, insight, and eval insights

      • In industries, it's often not realistic to compare it with some other solutions

      • Industries are important participants; sigcomm reviewers said we need more industry paper

      • "practical", "at-scale", "vendor-support"

Last updated