> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/shivarams-group/group-papers/lists/gpu-lifetimes-on-titan-supercomputer-survival-analysis-and-reliability.md).

# GPU Lifetimes on Titan Supercomputer: Survival Analysis and Reliability

### Introduction - Gray XKY Titan 

![](/files/-MYkGDEGeTxaGcxeEG1N)

* Swapping: due to the resistor 

### Data Collection & Preprocessing

![](/files/-MYkHDYSRZNX7afX5og1)

* Off the bus: cpu lose connection to GPU
* double bit error: error detection (in GPU memory) 

Reboot: software update or hardware swap 

* Aggregate data associated with that GPU, one row per GPU to show the location and period of time 

Portion of reboot or entire system? Whole system reboot 

### Data Cleansing 

![](/files/-MYkItBqEe2o0WPFL1p3)

### Intuition for GPU Lifetimes 

![](/files/-MYkJQtKvSeT9rAPOlUl)

* Install time: black dot 
* Blue square: OTB event 
* Red triangle: GDB event 
* Square: last seen event of the GPU 

2017: failing resistor 

* Had to do hardware/GPU swap 
* Installation

New batch: after 17

Old batch: more OTB event 

![](/files/-MYkJxJqDr6dylyGKtom)

Small gaps during the location, but overarchingly all operations are operational 

* 4 GPUs per blade 
  * Take the blade out and replace the GPU on the blade 
  * Put that blade back in different locations 
* 1 GPU affected 
  * Entire blade is removed 

![](/files/-MYkKFN52DDycLpTqenD)

* Distribution similar 
* Number of DBE > OTB 

![](/files/-MYkKv_5-oEbq4IwfoPK)

![](/files/-MYkLFTm33evW84UZAig)

![](/files/-MYkMAt4lhV42TFt3MhZ)

![](/files/-MYkN6fHEAgTBBv56QBI)

* Cage: where the GPU is in 

![](/files/-MYkOhGLEpS3tgNkzS8b)

Relative values compared to the other cages 

* Cage 2 higher than Cage 0 and 1 (hazard ratio) 
* Column level 
  * Columns were next to service system 

![](/files/-MYkPSD1Q4rwYqavEO_z)

* Interleaving level from column 0 to 11 

![](/files/-MYkPWyn6rd3xZoAIHcj)

* Torus: network interconnect at the super computer 
* Lower column number with lower x coordinate v.s lower column number with high x coordinate has a higher hazard ratio 
  * Job scheduling impacts the overall performance 
  * Workload is not balanced 
    * GPU with lower torus: more workloads 
    * Why?
      * Cluster is not full (70% of the cluster used) 
      * Which machine? 
        * Assign at the beginning with higher probability 
    * Awareness? 
* Good outcome: consistently around for 1 is the ideal scenario 

![](/files/-MYkQ7jqL50xDp829t9O)