> For the complete documentation index, see [llms.txt](https://sliu583.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sliu583.gitbook.io/blog/specific-work/seminar-and-talk/mlsys/episode-18.md). # Episode 18 ### Structure is all you need * Software 2.0 for data quality management * Theodoros Rekatsinas | UW-Madison ### The Notorious data quality problem ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWez17ODiXCl9a_9Vxk%2Fimage.png?alt=media\&token=987f2a75-748b-4d8a-91d5-4b57931dcb5a) ### Data quality management task #### Error detection tasks * Tuple (sample) validation * Cell-value validation #### Data repairs * Missing data imputation * Data repairs (value replacement) ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWezRO6gYhoHaNtDZee%2Fimage.png?alt=media\&token=d0228491-86a1-42dd-a02b-92e4a3605340) * Push to ML model? ### ML models are sensitive to low-quality data ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWezd0SAOiqQAQBUETm%2Fimage.png?alt=media\&token=460a9adc-470a-404e-9e15-21c212fd16ca) ### Goal: Streamline data quality management ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWezkW09JtXointpP-I%2Fimage.png?alt=media\&token=df40a257-9a72-4417-b0b0-0ba06d248fe3) * Marius: graphs, heteregeneous-structure data ### Example: data validation for mean estimation ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf-2mJTWHN1uV5RkxO%2Fimage.png?alt=media\&token=7c057428-78fd-413e-97a3-f5689a7d2879) ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf-8Ez-MvOybvGbASb%2Fimage.png?alt=media\&token=c838222d-7160-4cf5-b7bc-8e413703fff3) * Discrepancies between two estimate ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf-GnHINE4H2KKfEki%2Fimage.png?alt=media\&token=4b589cca-1002-48ca-97a5-76be8d2d40d5) * Filling some of the values, and leaving the others ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf-OyNhibG6eVRcjic%2Fimage.png?alt=media\&token=f9f82845-b6e0-4904-a45b-69dfaec1be57) * If know this dependency in advanced, then we could make better estimate ### Structure-aware data cleaning is necessary ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf-ZRJzWLMKld9kr-N%2Fimage.png?alt=media\&token=450dcd6f-3358-47ea-a93d-d28c13d022d4) ### Heterogeneous types of structure ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf-ou1_-s1-hmtWjJ4%2Fimage.png?alt=media\&token=596c9392-d5a9-446f-a25f-20661fc5cfb8) ### Contextual ML for automated data quality ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf0-2jHg2_iOw59QHx%2Fimage.png?alt=media\&token=dcbcb894-0fbe-4f82-8638-16307da4d136) * Unsupervised manner? * Can I run this inference queries fast ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf09BHKEzQjN1q4Xnc%2Fimage.png?alt=media\&token=6ce1854f-5e76-46a8-acce-443d58459488) ### HoloClean: Probabilistic Data Repairs ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf0bMy82SS-ymUass4%2Fimage.png?alt=media\&token=d6e96a92-b404-483a-801c-8a0f2de71e4d) ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf0iHeYNME52PPdUnw%2Fimage.png?alt=media\&token=38560f5c-c322-43fa-b976-bc2f18cd4a79) ### ("Take two") ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf12DYsxWel-KP4Wvr%2Fimage.png?alt=media\&token=6e1ca339-511a-4bcb-b739-6f50d9c00d6f) ### Schema-level Attention ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf1B1zzdchsvQ82QEi%2Fimage.png?alt=media\&token=9b2cf137-44d8-4825-b3b5-ae483d63cb9c) ### Why Attention? ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf1M6_vyQGu9XR_JdS%2Fimage.png?alt=media\&token=410400ec-dab9-4f19-9638-7479d2daf732) ### Naturally-occurring missing data ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf1eSwbEd0BtOIGlAB%2Fimage.png?alt=media\&token=0fb5597c-461c-491d-9b52-12008d7d2f42) ### Use case: Data Categorization ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf1uwvNwbhKjHK7L7k%2Fimage.png?alt=media\&token=36edd46f-bff8-4a4b-8314-be50456a3ec8) ### Other use cases * Error detection in demographic data used for policy decision * KIP tracking * Imputation of numerical data for industrial machinery monitoring ### Picket: self-supervised transformers for data validation in ML pipelines ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf2PuQGHupL_1B2VkH%2Fimage.png?alt=media\&token=ac81e730-720f-40c8-995f-76aab89c0a3e) ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf2f2DVRQW0eitqtIK%2Fimage.png?alt=media\&token=d07f4433-1539-4ff1-ae2b-5e77fa13e600) ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf2ik6OQkBGQJaty1K%2Fimage.png?alt=media\&token=f9006016-1bd4-4b14-91f8-ce44d7f5e079) ### Loss-based Outlier Detection and Filtering 1. Go back to the idea of learning a model to capture the clean data, and use this model in decisions 1. PicketNet: transformer 2. Outlier detection problem ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf33YRQ55W2EjvPPgt%2Fimage.png?alt=media\&token=7bf35fcd-e6ee-4777-8872-4b427c40f35f) ### PicketNet: two-stream transformer for tabula data ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf3GIiwG0prknqXffn%2Fimage.png?alt=media\&token=0ceae869-bcfb-411b-b5c7-cfbc684b4bee) * Benefits: * Value stream: flexibility * Schema stream: regularization ### Experimental Highlights: Poisoning Attacks * Aim to destroy ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf3VQx0mHXZEa7-xrH%2Fimage.png?alt=media\&token=b7eaedc6-756b-40ad-8114-8a1d8f5a7bed) ### Contexual ML for automated data quality ops ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf48HAz6jHot6nNsYT%2Fimage.png?alt=media\&token=3020de94-70f3-4ea3-a02d-06c812eb0e9b) ### Scalable no-code graph learning ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf4Sv7uBxrNJNZCq7X%2Fimage.png?alt=media\&token=15cd22d7-e208-407c-be23-76d4e826321e) ![](https://2097630930-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MVORxAomcgtzVVUqmws%2F-MWeyXSgLBGZPW1qN_M5%2F-MWf4nZthkeKoy1bi09s%2Fimage.png?alt=media\&token=4999e970-86de-4733-8f40-ede9ed37895f) * Systematic variation of the data? * Types of noise: real-world data, no assumption about the noise. Types of noise can be random, or systematic error (integrating things across different sources) * Random * Systematic: repeated instances of the noise, if we condition, then it's not random * Adversarial noise: are aware of the downstream task, and go and attack that system * Holoclean * Attention: handle this gracefully, pick up the type of strong bias * Picket: * Worst-possible case (Adversarial) * Not overfitting to examples * Distinguish between out-of-distribution or systematic change * Solution: two streams (scheme, value) * Value: robust towards the case. Kernel structure that operates on this level. * Profiling mechanism * Hetereogeneous data types * Tabula data: higher level constraints * Encode them as functional dependencies in DB * or pick them up through attention mechanism * Back in the day: user specification * Structure learning over the data * Exactly the attention matrix, in a faster and cheaper way * Hetereogenous * Same mechanism can potentially hold for a graph * Running structural learning type of profiling * Identify homogenous area as a pre-processing step * And preprocessing ... * Filtering away and keep * Doing this heavily in Holoclean * Monitor data and see if something is going on in the data pipeline * Reliable data * How that setting is different? Or some of the goals change? * Reconstruction * Signal and context * With high confidence, then it should be outlier * Information about the likelihood Goal: * applying the rules at scale (ETL, ...) * Start-up and companies: specific problem * Identify duplicates in records * Infer rules to prepare and standardize * AI --> platform for error detection and fixing * Nobody is targeting * Automating this * Position: reasoning about noisy structured data Challenges and what that issue looks like: * What is the model doing? Aspects * Attention: interpretable (know the semantics of the attributes, put more weights or less) * Allow people not immediately accept them. Have confidence over the prediction of the model. * Accept the one that makes sense * Also, allow users to introduce business /external features that would allow you to introduce Holoclean * Accept logic rules, convert them to features * Support matching functions Which one should I trust? * Ensemble (weighted vote) * In real cases, people believe their rules...