Data Sources, Interfaces & Automation
19
Working with Data Sources
Pointblank
: The Complete Guide to Data Validation in Python
Welcome
Foundations
1
Introduction
2
Getting Started
3
Inspecting & Profiling Your Data
4
The Validation Workflow
5
Validation in the Analysis Loop
Building Validation Plans
6
Column Value Validations
7
Aggregate Validations
8
Row and Completeness Validations
9
Table Structure and Freshness
10
Missing Data and Coded Missingness
11
Segmented Validation
12
Advanced and Custom Validation
Responding to Results
13
Thresholds and Actions
14
Reports and Extracts
15
Data Quality Scoring
16
Notifications and Observability
AI-Assisted Validation
17
Semantic Validation with LLMs
18
AI-Assisted Authoring
Data Sources, Interfaces & Automation
19
Working with Data Sources
20
YAML-Based Workflows
21
The Command-Line Interface
22
The MCP Server
Data Contracts & Pipelines
23
Data Contracts
24
Validation Pipelines
Test Data Generation
25
Generating Synthetic and Test Data
Clinical & Regulated Data
26
Validating Clinical Trial Data (CDISC SDTM & ADaM)
27
CDISC Submission Conformance
Industry Playbooks
28
Playbook: Financial Services & Fintech
29
Playbook: E-commerce & Retail
30
Playbook: Data Engineering & Analytics Platforms
31
Playbook: Healthcare & Life Sciences
32
Playbook: Real-World Evidence
33
Playbook: Machine Learning & Feature Monitoring
34
Playbook: Manufacturing & IoT Sensors
35
Playbook: Insurance & Actuarial
36
Playbook: Public Sector & Official Statistics
37
Playbook: Marketing & Digital Analytics
Appendices
A
Appendix: Which Method Do I Need?
B
Appendix: Glossary
Table of contents
19.1
Two kinds of backend
19.2
In-memory DataFrames
19.3
Databases and query pushdown
19.4
Ibis table expressions
19.5
Validating files
19.6
Choosing a backend
19.7
Backend differences to watch
19.8
Summary
Data Sources, Interfaces & Automation
19
Working with Data Sources
19
Working with Data Sources
19.1
Two kinds of backend
19.2
In-memory DataFrames
19.3
Databases and query pushdown
19.4
Ibis table expressions
19.5
Validating files
19.6
Choosing a backend
19.7
Backend differences to watch
19.8
Summary
18
AI-Assisted Authoring
20
YAML-Based Workflows