# A Deep Reinforcement Learning Perspective on Internet Congestion Control

* Internet congestion control: one of the most fundamental and challenging problems in communication networks 
  * Determines what you get out of the internet 
    * At what rate data goes in 
  * Always running on every connection 
    * With no prior knowledge 
* Water pipe
  * How fast do we send the data? 
  * Too slow: harm performance 
  * Too fast: overflow the pipe 
* Internet CC 
  * Very limited information, massive agent churn, enormous, dynamic, complicated network 
  * CC revisited 
    * Default algorithm: transmission control algorithm (TCP) 
* Motivating deep RL 
  * CC protocol
    * Blackbox 
    * Locally-perceived history of feedback: loss rates, latency 
    * On the other end: next sending rate 
  * Hypothesis: this feedback contains information about patterns that can improve the choice of sending rates 
* RL formulation 
  * Goal: maximize the cumulative reward
  * Agent observe the state, new action in the environment, output a reward 
  * NN: agent, and simulator for the environment 
    * Observe state, based on locally-perceived feedback 
* Aurora 
  * Monitor interval: send ratio, latency ration, latency inflation 
    * Keep design choice: scale-free observations affect robustness 
  * 3-layer NN 
  * Rate change factor 
  * Training 
    * Very simple simulated network
    * Each episode chooses different link parameters
    * Entire training platform: PCC-RL 
  * Real-World testing setup 
    * Real packets in linux kernel network emulation 
    * Using inference only 
      * Allowing faster adaptation 
      * Still producing state-of-the-art results