# Sage: Practical & Scalable ML-Driven Performance Debugging in Microservices

### Presentation 

* Motivation 
  * Microservices become increasingly popular in cloud system
    * Fine-grained, loosely-coupled, and single-concerned 
    * Communicate with RPCs or RESTful APIs 
    * Pros 
      * Agile development 
      * Better modularity and elasticity 
      * Testing and debugging in isolation 
    * Cons 
      * Different HW & SW constraints 
      * Dependencies --> complicate cluster management 
  * SLOs govern interactive microservices 
* Challenges in microservice performance debugging 
  * Microservices are more sensitive to performance unpredictability 
  * Complex network dependences 
  * Complex tracing and monitoring 
* Critical: automatic technique (data-driven method) 
  * Existing works: poor scalability; not practical 
* Design principle 
  * No need to label data 
    * Requires a causal model 
  * Robust to sampling frequency 
    * Suitable for instrumentation in production, not using temporal patterns for inference 
  * No need for kernel-level tracing 
  * Practical adjustment to service updates 
  * Focuses on resource provisioning-related performance issues
* Sage: root cause analysis system using unsupervised learning 

  * Casual Bayesian Networks (CBN) for casual relationships among microservices
    * Edges indicate casual relationships, a tool for structural casual inference, interpretable and explainable  

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FM23Sh1SjEqGLyKodid7f%2Fimage.png?alt=media\&token=67052533-0b2a-42e2-b312-d7e39461d2c7)

* Use counterfactuals to detect root causes (services and resources) of SLO violations
  * Counterfactual queries 
    * Queries of hypothetical end-to-end latency if some metrics had been "normal"
    * Root causes: metrics that hypothetically solve the end-to-end performance issue 
  * Generating counterfactuals with generative models 
    * CVAE 
      * Prior network, encoder, decoder (MLP) 
    * GVAE: factorize CVAE according to the CBN model 
      * Connection pruning to enforce the network to follow the casual model 
      * Better interpretability 
      * Faster retraining upon microservice updates 
      * Root cause detection with GVAE 
        * Learn the latent variable (Z) from the encoder 
        * Calculate the "normal" values of metrics and latent variables
        * Two-level intervention for root cause detection 
      * Incremental & partial retraining 

![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MVORxAomcgtzVVUqmws%2Fuploads%2FIUhM8COuwhNSaxUGDFuj%2Fimage.png?alt=media\&token=18e94c8e-825d-4653-baf5-7e7e8fd7c996)