Zero Trust Data Modeling
Hi folks,
A while back I had come across the term "Zero Trust Data Modeling", the term is indicative of handling incoming data streams as if there are no guarantees on the validity or quality of that data. Many data quality issues can be traced to not catching problems in the data early enough. For some types of data, say numeric data, it isn't too difficult to validate. For others, say names, it's a bit more difficult.
Has anyone had experience implementing a sort of expectations on their data as it's brought in? There are libraries out there that are known for this; Great Expectations in Python, and Delta Live Tables in Databricks. I'm unaware of anything in R.
A while back I had come across the term "Zero Trust Data Modeling", the term is indicative of handling incoming data streams as if there are no guarantees on the validity or quality of that data. Many data quality issues can be traced to not catching problems in the data early enough. For some types of data, say numeric data, it isn't too difficult to validate. For others, say names, it's a bit more difficult.
Has anyone had experience implementing a sort of expectations on their data as it's brought in? There are libraries out there that are known for this; Great Expectations in Python, and Delta Live Tables in Databricks. I'm unaware of anything in R.
repo
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5314482/
targets
https://docs.ropensci.org/targets/
If that doesn't work and have to go manual route, my thoughts would be along developing a cleaning script that you could always run first to check for the most common data issues of any dataset, ex. are date fields in right format, any missing values, identify correct text columns, etc. and then generate a report to you of what issues each new data stream has identified from those common fields. You could have assigned values coded in as the indicator, example: if stream_file from facility_x on date_x has missing values from field_x greater than 25%, then sandbox (somewhere not in production) date_x file for further manual review and cleaning.
The DataExplorer package has a lot of examples of different things you can look for and here's an article talking about good cleaning practice checklist also. Hope this helps.
DataExplorer
https://cran.r-project.org/web/packages/DataExplorer/vignettes/dataexplorer-intro.html
checklist article
https://towardsdatascience.com/data-cleaning-in-r-made-simple-1b77303b0b17