> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/mlsys/episode-18.md).

# Episode 18

### Structure is all you need

* Software 2.0 for data quality management 
* Theodoros Rekatsinas | UW-Madison 

### The Notorious data quality problem 

![](/files/-MWez17ODiXCl9a_9Vxk)

### Data quality management task 

#### Error detection tasks 

* Tuple (sample) validation
* Cell-value validation 

#### Data repairs 

* Missing data imputation
* Data repairs (value replacement) 

![](/files/-MWezRO6gYhoHaNtDZee)

* Push to ML model? 

### ML models are sensitive to low-quality data 

![](/files/-MWezd0SAOiqQAQBUETm)

### Goal: Streamline data quality management 

![](/files/-MWezkW09JtXointpP-I)

* Marius: graphs, heteregeneous-structure data 

### Example: data validation for mean estimation 

![](/files/-MWf-2mJTWHN1uV5RkxO)

![](/files/-MWf-8Ez-MvOybvGbASb)

* Discrepancies between two estimate 

![](/files/-MWf-GnHINE4H2KKfEki)

* Filling some of the values, and leaving the others 

![](/files/-MWf-OyNhibG6eVRcjic)

* If know this dependency in advanced, then we could make better estimate 

### Structure-aware data cleaning is necessary 

![](/files/-MWf-ZRJzWLMKld9kr-N)

### Heterogeneous types of structure 

![](/files/-MWf-ou1_-s1-hmtWjJ4)

### Contextual ML for automated data quality 

![](/files/-MWf0-2jHg2_iOw59QHx)

* Unsupervised manner? 
  * Can I run this inference queries fast  

![](/files/-MWf09BHKEzQjN1q4Xnc)

### HoloClean: Probabilistic Data Repairs 

![](/files/-MWf0bMy82SS-ymUass4)

![](/files/-MWf0iHeYNME52PPdUnw)

### ("Take two") 

![](/files/-MWf12DYsxWel-KP4Wvr)

### Schema-level Attention 

![](/files/-MWf1B1zzdchsvQ82QEi)

### Why Attention? 

![](/files/-MWf1M6_vyQGu9XR_JdS)

### Naturally-occurring missing data

![](/files/-MWf1eSwbEd0BtOIGlAB)

### Use case: Data Categorization 

![](/files/-MWf1uwvNwbhKjHK7L7k)

### Other use cases

* Error detection in demographic data used for policy decision
* KIP tracking
* Imputation of numerical data for industrial machinery monitoring 

### Picket: self-supervised transformers for data validation in ML pipelines 

![](/files/-MWf2PuQGHupL_1B2VkH)

![](/files/-MWf2f2DVRQW0eitqtIK)

![](/files/-MWf2ik6OQkBGQJaty1K)

### Loss-based Outlier Detection and Filtering 

1. Go back to the idea of learning a model to capture the clean data, and use this model in decisions 
   1. PicketNet: transformer 
   2. Outlier detection problem 

![](/files/-MWf33YRQ55W2EjvPPgt)

### PicketNet: two-stream transformer for tabula data 

![](/files/-MWf3GIiwG0prknqXffn)

* Benefits: 
  * Value stream: flexibility
  * Schema stream: regularization 

### Experimental Highlights: Poisoning Attacks

* Aim to destroy 

![](/files/-MWf3VQx0mHXZEa7-xrH)

### Contexual ML for automated data quality ops 

![](/files/-MWf48HAz6jHot6nNsYT)

### Scalable no-code graph learning 

![](/files/-MWf4Sv7uBxrNJNZCq7X)

![](/files/-MWf4nZthkeKoy1bi09s)

* Systematic variation of the data?
  * Types of noise: real-world data, no assumption about the noise. Types of noise can be random, or systematic error (integrating things across different sources) 
    * Random
    * Systematic: repeated instances of the noise, if we condition, then it's not random 
    * Adversarial noise: are aware of the downstream task, and go and attack that system 
* Holoclean 
  * Attention: handle this gracefully, pick up the type of strong bias
* Picket:
  * Worst-possible case (Adversarial) 
  * Not overfitting to examples 
* Distinguish between out-of-distribution or systematic change
  * Solution: two streams (scheme, value)
    * Value: robust towards the case. Kernel structure that operates on this level. 
  * Profiling mechanism
* Hetereogeneous data types 
  * Tabula data: higher level constraints 
    * Encode them as functional dependencies in DB
    * or pick them up through attention mechanism 
    * Back in the day: user specification 
  * Structure learning over the data
    * Exactly the attention matrix, in a faster and cheaper way 
  * Hetereogenous
    * Same mechanism can potentially hold for a graph 
    * Running structural learning type of profiling 
      * Identify homogenous area as a pre-processing step 
      * And preprocessing ...
      * Filtering away and keep 
    * Doing this heavily in Holoclean
* Monitor data and see if something is going on in the data pipeline 
  * Reliable data
  * How that setting is different? Or some of the goals change?
    * Reconstruction 
      * Signal and context 
      * With high confidence, then it should be outlier 
    * Information about the likelihood 

Goal:

* applying the rules at scale (ETL, ...) 
* Start-up and companies: specific problem 
  * Identify duplicates in records 
  * Infer rules to prepare and standardize 
  * AI --> platform for error detection and fixing 
* Nobody is targeting
  * Automating this 
  * Position: reasoning about noisy structured data 

Challenges and what that issue looks like:

* What is the model doing? Aspects 
  * Attention: interpretable (know the semantics of the attributes, put more weights or less)
  * Allow people not immediately accept them. Have confidence over the prediction of the model. 
    * Accept the one that makes sense 
* Also, allow users to introduce business /external features that would allow you to introduce 

Holoclean

* Accept logic rules, convert them to features 
* Support matching functions 

Which one should I trust?

* Ensemble (weighted vote)
* In real cases, people believe their rules...