> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/wisconsin-seminar/the-lottery-ticket-hypothesis.md).

# The Lottery Ticket Hypothesis

![](/files/-MXEEwt7RpPToTN5v-vn)

### Background: Neural Network Pruning

* Neural networks are large 
* Prune: to reduce the extent of a neural network by removing unwanted parts 
* Goal: reduce the cost of inference 

![](/files/-MXEFfIB-ehsP827PKsC)

* Structure
  * Neurons, attention heads, weights, blocks? 
  * Focus on pruning weights 
* Superfluous
  * Magnitudes? Gradients? Activations? 
* Fine-tuned 
  * Retrain

![](/files/-MXEG8WFm5ikCbcZokN6)

### Contribution 

![](/files/-MXEGFxRgOhuvyNk5t7v)

### Training is expensive 

* Cost 
  * Financial cost 
  * Environmental cost (CO2) 
* Computationally 

### Research Question 

* Prune models after training, can we train smaller models?
  * Capacity for representation vs. optimization 
* Yes 
  * Early in training 

![](/files/-MXEH6pa9CNnyQvUbbCq)

### Lottery Ticket Hypothesis 

*  Hindsight 

![](/files/-MXEHG5N8t59G5iFQENz)

* Does not reach full accuracy 

![](/files/-MXEHQk-yNHv4fII98m3)

* Contribution: you need to keep the original initializations 

### Iterative Magnitude Pruning (IMP)

![](/files/-MXEI5hiy0B9HKkxRtX7)

### Result 

![](/files/-MXEISnStQqZbYtFHV2C)

![](/files/-MXEIVYam4B5Yq32jRBn)

![](/files/-MXEIy2IAi-s7Pq-ERF0)

* IMP finds subnetworks that can train from the start to full accuracy at non-trivial sparsities 
  * Random subnetwroks cannot train to full accuracy 

![](/files/-MXEJWvaZ5Qay25T69VH)