A Deep Reinforcement Learning Perspective on Internet Congestion Control

http://proceedings.mlr.press/v97/jay19a/jay19a.pdf

Internet congestion control: one of the most fundamental and challenging problems in communication networks
- Determines what you get out of the internet
  - At what rate data goes in
- Always running on every connection
  - With no prior knowledge
Water pipe
- How fast do we send the data?
- Too slow: harm performance
- Too fast: overflow the pipe
Internet CC
- Very limited information, massive agent churn, enormous, dynamic, complicated network
- CC revisited
  - Default algorithm: transmission control algorithm (TCP)
Motivating deep RL
- CC protocol
  - Blackbox
  - Locally-perceived history of feedback: loss rates, latency
  - On the other end: next sending rate
- Hypothesis: this feedback contains information about patterns that can improve the choice of sending rates
RL formulation
- Goal: maximize the cumulative reward
- Agent observe the state, new action in the environment, output a reward
- NN: agent, and simulator for the environment
  - Observe state, based on locally-perceived feedback
Aurora
- Monitor interval: send ratio, latency ration, latency inflation
  - Keep design choice: scale-free observations affect robustness
- 3-layer NN
- Rate change factor
- Training
  - Very simple simulated network
  - Each episode chooses different link parameters
  - Entire training platform: PCC-RL
- Real-World testing setup
  - Real packets in linux kernel network emulation
  - Using inference only
    Allowing faster adaptation
    Still producing state-of-the-art results

Last updated 4 years ago