ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

https://arxiv.org/pdf/2104.07857.pdf