> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/conference/index/sosp-21/learning/bladerunner-stream-processing-at-scale-for-a-live-view-of-backend-data-mutations-at-the-edge.md).

# Bladerunner: Stream Processing at Scale for a Live View of Backend Data Mutations at the Edge

#### Problem statement 

* Users of Facebook services access the social graph from around the world 

![](/files/uwRjeo4aRwKith4Liclg)

* Challenges on data liveness 
  * The social graph is constantly mutating and providing every one of our end users with fresh and up-to-date data is challenging 
  * Challenges 
    * Mutability and relevance 
      * Parts of the social graph can mutate at a high rate and some contents are time sensitive 
    * Rapid changing foci 
      * Users change the focus of their interest frequently 
    * Legacy substrate  
      * Client devices have limited processing power and storage capacity 
* Polling solution 
  * Client device issuing initial query when needed and then periodically polling server for updates. 
    * Wastes network bandwidth and client battery 
    * Leads to many costly queries to server 
      * Range queries, intersection queries that are costly 
    * Polling interval introduces extra latency 
* Bladerunner architecture 

![](/files/1KBJMWKGgeWXeAoMNNnU)

![Query data and Establish subscription ](/files/VaSc1nUVdXcKi0KIsf3h)

![Issue mutation and trigger publish](/files/U5pYRvvZHYoca2kIqUHV)

![Event fanout and data delivery ](/files/aGHT8Sk1CVpr9qwinI7c)

* Deep Dive - Pylon 
  * Simple pub/sub system that delivers published messages to BRASSes across regions
  *   ![](/files/Lxg9bejDSlnpsDt63Y6C)
* Deep Dive - BRASS 
  * responsible for application-specific stream processing 
  * ![](/files/KkXHZmC44kGeOKqZVIaF)
* Deep Dive - BURST
  *  application-layer protocol developed to support long running streams between client devices and BRASSes 
  * ![](/files/Fqz1FAG2GcS3ZsYdXSAF)
* Deep Dive - Failure Handling and Delivery Guarantee 
  * BRASS and BURST are designed to provide flexibility in choosing different levels of delivery guarantees, while most applications on Bladernner today only require best effort delivery 
* Use case 
  * Comments for live videos 
* Production experience - live video comments 
  * Huge efficiency wins after switching from polling to Bladerunner, besides reduced latency in comment delivery 
* Other learnings 
  * Explicitly specify liveness requirement for product services 
  * Ranking? 
    * Polling: batchify 
    * Bladerunner: wait between latest items v.s. more engaging items in product experience