small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")4 The Validation Workflow
Chapter 2 gave you the tools to run a basic validation. Effective data validation, though, requires more than knowing which methods to call. It requires understanding how those methods fit together, when each component comes into play, and how to design plans that are both thorough and maintainable.
This chapter steps back from individual methods to examine the validation workflow as a whole. Once you understand the complete lifecycle, from plan construction through interrogation to result analysis, you can make better decisions about how to structure validations, troubleshoot surprising results, and get the most out of the process.
We begin with the anatomy of a validation plan, looking at what components it contains and how they interact. We then trace what happens during interrogation, the step that turns a plan into results. Next we work with a real interrogated plan and survey the many ways to extract information from it. Finally, we discuss how to organize plans so they stay readable as they grow. Throughout, we use the built-in small_table dataset so that every example runs and every reported number is real.
4.1 Anatomy of a validation plan
A validation plan is more than a list of checks. It is a complete specification that bundles together the data to validate, metadata for reporting, thresholds that define severity, actions that respond to problems, and the validation steps themselves. Understanding these components helps you design plans that are not just correct but also maintainable and useful in an operational setting.
It helps to think of a validation plan as a contract between your data and your expectations. The more precisely you state that contract, the more useful the results become. Everything in this section builds toward one concrete plan that we carry through the rest of the chapter, so we start by loading the data it will run against.
With the data in hand, we can look at each part of a plan in turn.
4.1.1 The target data
Every validation plan operates on a specific dataset, and once a Validate object is created, that data is bound to it. The target data is what gives the plan its raw material, and it determines how many test units each step will have. For a table of thirteen rows like small_table, a column-value check produces thirteen test units, one for each row it examines.
NoteWhat exactly is a test unit?
A test unit is the smallest thing a step can pass or fail on, and what counts as one depends on the kind of check. A column-value check such as col_vals_gt() produces one test unit per row, so a thirteen-row table yields thirteen of them. An aggregate check such as col_sum_eq() reduces the whole column to a single number and therefore has exactly one test unit, and a table-level check such as col_schema_match() also has one, judging the table as a whole. This matters throughout the book, because thresholds and quality scores are computed from the fraction of test units that pass, so knowing what a step’s test units are is what lets you read its result correctly.
The data also determines which columns are available to validate and what types they hold. A plan cannot check a column that does not exist, and a check only makes sense when it matches the column’s type. Because the data is fixed at construction time, the plan and its data travel together, which is what makes results reproducible and reports self-describing.
4.1.2 Metadata
In quick exploratory work, metadata can feel like optional decoration. In a production setting where many validations run across many tables, it becomes essential for understanding what you are looking at. A plan can carry several pieces of identifying metadata that flow through to its reports.
meta_example = pb.Validate(
data=small_table,
tbl_name="small_table", # identifies the table under test
label="Daily quality check", # a human-readable description of the run
lang="en", # report language
locale="en-US", # locale for number and date formatting
)The tbl_name= parameter records which table the validation examined, which is useful when we’re using a pipeline that validates several tables. The label= parameter captures the purpose of the run (e.g., distinguishing a scheduled check from an ad-hoc investigation). The lang= and locale= settings help to make reports readable for the audience that will consume them (say, a team based in Germany) since the reporting elements that Pointblank provides will be translated to fit a specified locale. We will see some of these values appear in the report header once we display the full plan.
4.1.3 Global thresholds
Not every failure is equally serious. A small rate of invalid formatting might be acceptable noise, while the same rate of missing identifiers might be a genuine problem. Thresholds let you say, in advance, what levels of failure should raise concern. Thresholds set on the Validate object apply to every step unless a step overrides them.
threshold_example = pb.Validate(
data=small_table,
thresholds=pb.Thresholds(warning=0.1, error=0.25, critical=0.35),
)These global thresholds establish a baseline where a ten percent failure rate raises a warning, twenty-five percent raises an error, and thirty-five percent is treated as critical. Different checks often warrant different tolerances, and individual steps can set their own thresholds when they need to, a topic covered in depth in Chapter 13. For now, the important idea is that thresholds turn a raw failure count into a graded judgment about severity.
4.1.4 Global actions
Thresholds classify problems, and actions respond to them. Without actions, validation is passive, producing reports that someone has to remember to read. With actions, validation becomes active, able to send alerts, write to logs, or pause a pipeline the moment a threshold is crossed. Actions can be attached globally, and each one is a function that runs when its severity level is reached.
def warn_action():
m = pb.get_action_metadata()
print(f"Step {m['step']} ({m['type']}) reached the {m['level']} level on column '{m['column']}'")
action_example = (
pb.Validate(
data=small_table,
thresholds=pb.Thresholds(warning=0.1),
actions=pb.Actions(warning=warn_action),
)
.col_vals_lt(columns="a", value=7)
.interrogate()
)Step 1 (col_vals_lt) reached the warning level on column 'a'The single step here checks that values in column a are below 7, and because two of the thirteen values are not, the failure rate crosses the 10% warning threshold and the action (defined in the actions= parameter) runs. Note that the action itself (warn_action) takes no arguments. Instead it calls get_action_metadata() internally to learn about the step that triggered it, and that metadata carries the step number, the assertion type, the column involved, the comparison value, the severity level reached, and a ready-made failure_text message (we used only select pieces of information in the print() call). Actions are the mechanism that lets a validation plan participate in your operational infrastructure, and Chapter 13 goes quite a bit further on this topic.
4.1.5 Validation steps
The validation steps are the core of the plan. Each one states what to check and how to check it, and steps are added by chaining method calls. Here we assemble the plan that the rest of the chapter uses, combining the metadata and thresholds from above with four checks.
plan = (
pb.Validate(
data=small_table,
tbl_name="small_table",
label="Daily quality check",
thresholds=pb.Thresholds(warning=0.1, error=0.25, critical=0.35),
)
.col_vals_lt(columns="a", value=7)
.col_vals_not_null(columns="c")
.rows_distinct()
.col_vals_in_set(columns="f", set=["low", "mid", "high"])
)
planPointblank ValidationNo Interrogation Performed |
|||||||||||||
Daily quality check Polarssmall_tableWARNING0.1ERROR0.25CRITICAL0.35 |
|||||||||||||
| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C66 | 1 |
col_vals_lt()
|
— | ||||||||||
| #4CA64C66 | 2 |
col_vals_not_null()
|
— | ||||||||||
| #4CA64C66 | 3 |
rows_distinct()
|
— | ||||||||||
| #4CA64C66 | 4 |
col_vals_in_set()
|
— | ||||||||||
Displaying the plan before interrogation shows its structure without any results. The report header carries the tbl_name and label we supplied, and the body lists the four steps in the order they were added, numbered one through four. The note that no interrogation has been performed is a reminder that this is still only a specification. That numbering is stable and is used throughout reports and result access, so a report that mentions step three always refers to the third method call, which here is the rows_distinct() check. Because the numbers are meaningful, the order in which you add steps is worth some thought, a point we return to at the end of the chapter.
4.1.6 Final actions
Step actions respond to individual steps as they happen. Final actions, by contrast, run once after the entire interrogation completes, and they run regardless of whether anything failed. That makes them the right place for summary reporting, cleanup, or an aggregate go/no-go decision.
def summary_action():
s = pb.get_validation_summary()
print(f"{s['n_steps']} steps ran, {s['n_failing_steps']} had failing test units")
final_example = (
pb.Validate(
data=small_table,
final_actions=pb.FinalActions(summary_action),
)
.col_vals_lt(columns="a", value=7)
.col_vals_not_null(columns="c")
.interrogate()
)2 steps ran, 2 had failing test unitsThe final action here reports that two steps ran and both had failing test units. It draws that information from get_validation_summary(), which is available only inside a final action and returns a dictionary describing the whole run, including step counts, per-step results, the highest severity reached, and an overall health score. Because a final action always fires, it is the dependable hook for work that must happen at the end of every validation, whether or not the data was clean.
4.2 The interrogation process
Until you call interrogate(), a validation plan is only a description of what to check. Interrogation is the step that turns that description into results. You rarely need to think about its internals for a simple validation, but understanding the execution model pays off when a plan runs slowly, produces a surprising number, or needs to react to problems through actions.
The key idea is that interrogation walks the steps in order, evaluates each one against the data, records what happened, and reacts when thresholds are crossed. The subsections below trace that sequence.
4.2.1 Step-by-step execution
When you call interrogate(), Pointblank processes the steps one at a time and in the order they were added. For each step it runs the step’s logic against the target data, counts the test units that pass and fail, stores the outcome, and then checks the result against any thresholds, running an associated action if one is defined and its level is reached. Only then does it move on to the next step.
This sequential model has one consequence that’s worth stating. The failure of one step never prevents later steps from running, so you always get results for every step in the plan. A plan is a set of independent observations about the data, not a chain that breaks at the first problem.
4.2.2 Test unit evaluation
The test unit is the atom of validation. For a column-value check, each row’s value in the target column is a single test unit, evaluated independently as a pass or a fail. Consider the first step of our plan, col_vals_lt(columns="a", value=7). Each of the thirteen values in column a is one test unit, and the check asks whether that value is below seven. Two of the values, an eight and a seven, are not, so those two test units fail and the other eleven pass.
This row-by-row evaluation is why Pointblank can report not just that a check failed but exactly how much of the data failed it. That precision is what turns validation from a very simplified check into more of a detailed diagnostic, and it’s also what makes the failing rows available for inspection later (these are known as data extracts).
4.2.3 Threshold evaluation
Once a step’s test units are counted, its failure rate is compared against the thresholds. Two failing test units out of thirteen is a failure rate of about fifteen percent, which crosses our warning threshold of ten percent but stays below the error threshold of twenty-five percent. That step is therefore at the warning level.
Each severity level is its own threshold, and the failure rate is compared against each one independently. When the thresholds are set in the conventional order of increasing stringency, with warning no higher than error and error no higher than critical, they cascade upward, so a step that reaches the critical level has necessarily crossed the error and warning levels too (and reports show the highest level reached). That cascade is what lets you simplify your own logic, since asking whether a step reached the warning level is then enough to know whether it crossed any threshold at all. Pointblank does not enforce this ordering, so a non-standard arrangement such as a warning set higher than a critical is allowed and would break the cascade. So a pattern of increasing stringency is the sensible and near-universal choice. The comparison happens immediately after each step, not at the end of the run, and this is what allows actions to respond in real time.
4.2.4 Action execution
When a step crosses a threshold that has an action attached, that action runs right then, in the middle of interrogation rather than after it. We saw this earlier when the warning action printed its message as soon as the a check failed. The timing is deliberate. If a critical check fails early in a long plan, an alert can fire immediately instead of waiting for dozens of later steps to finish, which matters in time-sensitive pipelines.
An action function receives no arguments directly. Everything it needs comes from get_action_metadata(), which reports which step triggered, what kind of check it was, which column was involved, and what level was reached. That keeps action functions simple while still giving them full context about why they were called.
4.2.5 Result accumulation
As each step completes, its results accumulate inside the validation object, and after interrogation finishes they are all available for inspection. Calling interrogate() on our plan gives us the interrogated object we will use for the rest of the chapter.
validation = plan.interrogate()
validationThe interrogated report now fills in the columns that were empty before. Each step shows its total test units, how many passed and failed, and the threshold status it reached. The object stores all of this, including the assertion type, the parameters, timing information, and extracts of the failing rows, and that single accumulation is what feeds every view of the results we look at next. Crucially, the results persist after interrogation ends, so you can inspect them now, save them, or pass the object to another function without losing anything.
4.3 Working with results
A validation whose results you cannot reach is not much use. Pointblank offers several ways to get at results because different situations call for different formats. Interactive exploration wants a visual report, automation wants plain numbers, debugging wants the actual failing rows, and integration with other systems wants structured data. This section works through those options using the interrogated validation object from above, so every number shown is the real result of that plan.
4.3.1 The tabular report
The visual report is the view we just saw, and it is produced by displaying the validation object. For finer control over what the report includes, get_tabular_report() accepts arguments to toggle the header, footer, and other elements.
validation.get_tabular_report(incl_header=True, incl_footer=False)The report is designed to be easily scannable. A glance can tell you whether the plan passed, and the per-step rows let you get into exactly what was checked and how it fared. This is the right output when a person is going to read the results. However, an automation scenario needs something it can compute with, which is where the programmatic accessors come in.
4.3.2 Aggregate metrics
The methods in this group return the counts and fractions behind the report. Most of them return a dictionary keyed by step number, so a single call describes the whole plan at once. Those keys start at one and match the step numbers in the leftmost column of the tabular report, so each dictionary entry lines up with the row you would read there.
validation.n_failed(){1: 2, 2: 2, 3: 2, 4: 0}The result confirms what the report showed: steps one through three each have two failing test units, and step four has none. The companion methods n(), n_passed(), f_passed(), and f_failed() follow the same pattern, returning totals, passing counts, and passing or failing fractions per step. When you want a single step’s value as a plain number rather than a dictionary entry, pass its one-based index and use scalar=True.
validation.n_failed(i=1, scalar=True)2For an overall verdict rather than per-step detail, all_passed() returns a single boolean that is true only when every test unit in every step passed.
validation.all_passed()FalseThe result is False here because three of the four steps had failures. You might use this boolean as a means to branch in a pipeline (i.e., decision to proceed or stop entirely).
if validation.all_passed():
print("All checks passed so it is safe to proceed.")
else:
print("Some checks failed. Please review.")Some checks failed. Please review.Where the report speaks to a person, these accessors return the same results as plain numbers and booleans. This can help address common scenerios like a monitoring job recording the n_failed() counts or a test that can assert on all_passed(). You might imagine a pipeline that can branch on either condition. An interrogation can thus directly serve as inputs for automation directly rather than only documenting an outcome.
4.3.3 Threshold status
When you care about degrees of severity rather than raw pass and fail metrics, the threshold accessors can report whether each step reached a given level. Like the count methods, they return a dictionary that is keyed by step.
validation.warning(){1: True, 2: True, 3: True, 4: False}Steps 1–3 reached the warning level, and step 4 did not. This matches the roughly 15% failure rate of the first three steps against our ten percent warning threshold. Checking the error level shows that none of the steps went that far.
validation.error(){1: False, 2: False, 3: False, 4: False}For automated responses during interrogation, actions (covered earlier) are the right tool. These threshold accessors are most useful after interrogation, when you need to branch or assert on the results externally (in a CI script, a test suite, or a pipeline step that receives an already- interrogated object).
# raise an exception if any step hit the critical level
if any(validation.critical().values()):
raise RuntimeError("Critical data quality failures detected")With our data, no step reached the critical threshold, so the exception is not raised. For test suites specifically, assert_passing() and assert_below_threshold() are more idiomatic than a manual check: assert_passing() raises an AssertionError if any test unit failed anywhere, while assert_below_threshold(level="error") raises only if a step reached the error level given in error= (or an error state above that), letting bounded low-severity failures through. Both are covered in detail in Chapter 13.
The choice between responding inside the plan (via actions) and responding outside it (via post-interrogation Python) comes down to timing and coupling. Actions fire during interrogation, so a critical action that raises will stop the run immediately, before later steps execute. This matters when a failure early in a long plan should halt everything rather than waste time on subsequent checks. Post-interrogation code always sees the complete results, which makes it better suited to decisions that depend on the overall picture, to contexts where you want to inspect before acting, and to situations where the same plan runs under different response requirements (strict in CI, lenient in development). Actions are also part of the plan definition itself, so they travel with it. Post-interrogation code is external and easier to vary without touching the plan.
4.3.4 Data extracts
Knowing that two rows failed a step is useful, and seeing those two rows is better. Data extracts hand you the actual failing rows so you can understand why a check failed rather than only that it did. Asking for the extract of the first step, with frame=True to get the data frame directly, returns the offending rows.
validation.get_data_extracts(i=1, frame=True)
shape: (2, 9)
| _row_num_ | date_time | date | a | b | c | d | e | f |
|---|---|---|---|---|---|---|---|---|
| u32 | datetime[μs] | date | i64 | str | i64 | f64 | bool | str |
| 5 | 2016-01-09 12:36:00 | 2016-01-09 | 8 | "3-ldm-038" | 7 | 283.94 | true | "low" |
| 7 | 2016-01-15 18:46:00 | 2016-01-15 | 7 | "1-knw-093" | 3 | 843.34 | true | "high" |
4.3.5 Sundered data
Where extracts isolate the failing rows of a single step, sundering splits the whole table into the rows that passed and the rows that failed. This supports a common quarantine pattern, where clean rows continue downstream while problematic rows divert to a review queue.
validation.get_sundered_data(type="fail")
shape: (4, 8)
| date_time | date | a | b | c | d | e | f |
|---|---|---|---|---|---|---|---|
| datetime[μs] | date | i64 | str | i64 | f64 | bool | str |
| 2016-01-06 17:23:00 | 2016-01-06 | 2 | "5-jdo-903" | null | 3892.4 | false | "mid" |
| 2016-01-09 12:36:00 | 2016-01-09 | 8 | "3-ldm-038" | 7 | 283.94 | true | "low" |
| 2016-01-15 18:46:00 | 2016-01-15 | 7 | "1-knw-093" | 3 | 843.34 | true | "high" |
| 2016-01-30 11:23:00 | 2016-01-30 | 1 | "3-dka-303" | null | 2230.09 | true | "high" |
The failing portion has four rows. That number deserves a close look, because our plan had two failing rows in each of its first three steps, which might suggest six failing rows rather than four. The reason is that sundering considers only column-value checks, the col_vals_* family. Our first two steps are such checks, and their failing rows are the two with an out-of-range value in a and the two with a missing value in c, four distinct rows in total. The rows_distinct() step, although it did flag two duplicated rows, is a row-based check rather than a column-value one, so it does not contribute to the split. Keeping this rule in mind avoids surprise when the sundered counts differ from the per-step failure counts.
4.3.6 JSON export
For integration with monitoring dashboards, data catalogs, or custom tooling, the results are available as JSON. The get_json_report() method returns a JSON string, which parses into a list with one record per step.
import json
report = json.loads(validation.get_json_report())
len(report)44.4 Organizing validation plans
A plan with a handful of steps is easy to follow. A plan with fifty steps can become hard to read unless it is organized deliberately. As plans grow, the way you structure them decides whether they stay maintainable or turn into something no one wants to touch. Good organization also improves the report, because related checks sitting together tell a coherent story rather than presenting a jumble.
4.4.1 Brief descriptions
Step method calls describe the implementation of check but not the intent for those checks. The brief= parameter helps you attach a human-readable description that appears in the report, which closes the gap between what a check does mechanically and why it exists.
brief_demo = (
pb.Validate(data=small_table)
.col_vals_lt(
columns="a",
value=7,
brief="Values in 'a' should be all less than 7",
)
.interrogate()
)
brief_demo.n_failed(){1: 2}The check behaves exactly as before (flagging the same two failing rows) but now the report provides a sentence that a non-technical stakeholder could read to better understand the check. Put another way, a brief is documentation that resides with the validation. It can also serve to remind the author what the check is meant to accomplish.
When you do not want to write a description yourself (and this could be tedious when there are many checks in a validation plan), the Validate(brief=True) option generates briefs automatically from each step’s parameters. The same option works at the step level (brief=True on an individual method call), which overrides whatever global setting is in effect for that step.
auto_brief_demo = (
pb.Validate(data=small_table, brief=True)
.col_vals_lt(columns="a", value=7)
.interrogate()
)
auto_brief_demo.n_failed(){1: 2}You can also mix the two approaches by using "{auto}" as a token inside a custom string. The generated text replaces the token, letting you prefix it with context that the auto-generation would not know.
mixed_brief_demo = (
pb.Validate(data=small_table)
.col_vals_lt(columns="a", value=7, brief="Range check: {auto}")
.interrogate()
)
mixed_brief_demo.n_failed(){1: 2}The "{auto}" token is one of several available for composing briefs. The full set consists of:
| Token | Resolves to |
|---|---|
{auto} |
Auto-generated description of the step’s expectation |
{step} or {i} |
Step number |
{col} or {column} |
Column name (or comma-separated names, if multiple) |
{value} |
The comparison value |
{thresholds} |
Formatted threshold levels (e.g., W: 0.1 / E: 0.25 / C: 0.35) |
{segment} |
Segment column and value together |
{segment_column} |
Segment column name alone |
{segment_value} |
Segment value alone |
These tokens work in both per-step brief= strings and in the global brief= setting on the Validate constructor. A global brief like "Step {step}: {auto}" applies to every step that does not override it with its own brief= argument, and setting brief=False on an individual step cancels the global brief for that step.
In practice, brief=True is the easiest and lowest-friction option to take and really works well for exploratory plans or rapid iteration. A hand-written string is better when the automatic text is too technical for the intended audience, or you want to provide rationale for the check (among other reasons). The token-based form splits the difference as you can add context around the automatically generated portion.