A Deep Reinforcement Learning Perspective on Internet Congestion Control

http://proceedings.mlr.press/v97/jay19a/jay19a.pdf

• Internet congestion control: one of the most fundamental and challenging problems in communication networks

  • Determines what you get out of the internet

    • At what rate data goes in

  • Always running on every connection

    • With no prior knowledge

• Water pipe

  • How fast do we send the data?

  • Too slow: harm performance

  • Too fast: overflow the pipe

• Internet CC

  • Very limited information, massive agent churn, enormous, dynamic, complicated network

  • CC revisited

    • Default algorithm: transmission control algorithm (TCP)

• Motivating deep RL

  • CC protocol

    • Blackbox

    • Locally-perceived history of feedback: loss rates, latency

    • On the other end: next sending rate

  • Hypothesis: this feedback contains information about patterns that can improve the choice of sending rates

• RL formulation

  • Goal: maximize the cumulative reward

  • Agent observe the state, new action in the environment, output a reward

  • NN: agent, and simulator for the environment

    • Observe state, based on locally-perceived feedback

• Aurora

  • Monitor interval: send ratio, latency ration, latency inflation

    • Keep design choice: scale-free observations affect robustness

  • 3-layer NN

  • Rate change factor

  • Training

    • Very simple simulated network

    • Each episode chooses different link parameters

    • Entire training platform: PCC-RL

  • Real-World testing setup

    • Real packets in linux kernel network emulation

    • Using inference only

      • Allowing faster adaptation

      • Still producing state-of-the-art results