# On the Use of ML for Blackbox System Performance Prediction

* Performance prediction is increasingly important 
  * Optimization, capacity planning, SLO-aware scheduling 
  * F(parameter) --> performance 
* Challenges 
  * Accurate: precise predictions 
  * Simple / easy-to-use: in-depth understanding of the systems not required 
  * General: works across a spectrum of workloads and applications 
* Can ML provide an accurate, general, and simple performance predictor? 
* This paper: a systematic and broad study on performance prediction 
* ML for system perf. prediction? 
  * Start with the best-case scenario 
  * The best-case (BC) test 
    * Given parameters, learn 
  * ML assumptions 
    * One-feature-at-a-time: e.g., vary P2, keeping P1, P3, ..., Pk fixed 
    * Seen-config
  * System assumptions 
    * No-contention: dedicated EC2 instances, isolated experiments 
    * Identical-inputs: same input data for a given input dataset size 
* Applications and models 
  * ML models: NN, LR, RF, SVM, NN 
* Metrics and predictors
  * Accuracy metric: rMSRE
  * ML predictors --> best-of-model / BoM-err
    * rMSRE of the most accurate model
  * Oracle predictor --> O-err 
    * Allow Oracle to peek at both the error function and test data 
* Best case test results 
  * High oracle error even under our best-case setup! 
* Methodology 
  * Root cause for each of the applications 
  * Fix? 
    * With system modifications
      * For all applications, oracle error is now well within 10%!
      * Best-of-model error likewise 
  * Trade-off between predictability and other design goals!
  * E.g., disabling an optimization can lead to higher prediction accuracy but degraded performance 
  * These fixes require in-depth understanding of the app. and reasoning about the trade-offs! 
* Embrace variability: probabilistic predictions 
  * Idea: predicting a mixture distribution instead of a single value 
  * Then, use the "modes" of each distribution as the "top-k" prediction value 
  * ML: mixture density networks and probabilistic random forest 
  * Significant decrease in BoM-err with top-3 (k=3) predictions! 
* So far, best-case setup only 
  * Go beyond? 
  * Prediction errors can remain high if the underlying performance trend is difficult to learn 
* Conclusion 
  * Taken "out of the box", many apps exhibit a surprisingly high degree of irreducible error 
  * We can significantly improve the accuracy if we accept the loss of simplicity and / or generality 
    * Modify applications 
    * Modify predictions 
    * ... but they don't work in all cases 
  * Need a more nuanced methodology for applying ML