small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")3 Inspecting & Profiling Your Data
You cannot validate what you do not understand. A validation rule is a statement of expectation, and an expectation has to come from somewhere. For data you know well, that knowledge lives in your head: you already know which columns should never be empty, which values are categorical, and what a plausible range looks like. For data you are meeting for the first time, you have none of that, and writing rules before you have looked is how validation plans end up either too loose to catch anything or too strict to ever pass.
This chapter is about the looking part. Pointblank ships a small set of inspection tools that answer the first questions you ask of any table: What columns are here, and what types do they hold? How many rows are there? Where are the values missing, and how are they distributed? Each tool turns a vague sense of “let me get to know this data” into a repeatable step you can run on any table, whether it lives in a local DataFrame or a remote warehouse.
We will move from the broadest view to the most detailed. First we preview a handful of rows to orient ourselves, then we profile every column at once, then we focus specifically on missing values, and finally we look at how the same tools work against data that never leaves the database. The goal throughout is not to validate anything yet, but to learn enough that the validation plan almost writes itself. Building and running that plan is the subject of Chapter 4, and letting Pointblank draft one for you from a profile is covered in Chapter 18.
3.1 The example data
Every example in this chapter uses one of the datasets bundled with Pointblank, so you can run the code exactly as written without hunting for a file. The load_dataset() function returns a ready-to-use table, and its tbl_type= argument controls whether you get a Polars DataFrame, a Pandas DataFrame, or a DuckDB-backed table. We begin with small_table, which is deliberately tiny at thirteen rows and eight columns, because it is easy to reason about while still containing the kinds of imperfections that matter.
Before looking at any values, it is worth confirming the two most basic facts about a table: how many rows and columns it has. The get_row_count() and get_column_count() functions answer exactly those questions, and they work identically across every backend Pointblank supports.
print(pb.get_row_count(small_table), "rows")
print(pb.get_column_count(small_table), "columns")13 rows
8 columnsThose two numbers are often the first sign that something is wrong. A table you expected to hold a million rows that reports a few hundred usually means a load failed or a filter ran too aggressively. With the dimensions confirmed, we are ready to look at the data itself.
3.2 A first look with preview()
The simplest inspection is to look at some rows, but a raw DataFrame print is a poor way to do it. Wide tables wrap awkwardly, long tables scroll past the screen, and neither the column types nor the overall shape are easy to pick out. The preview() function solves this by rendering a compact, formatted view that shows a few rows from the top and a few from the bottom of the table, along with each column’s type.
Function signature
preview(
data,
columns_subset=None,
n_head=5,
n_tail=5,
limit=50,
show_row_numbers=True,
max_col_width=250,
min_tbl_width=500,
incl_header=None
)Calling it on small_table shows the whole idea at a glance.
pb.preview(small_table)PolarsRows13Columns8 |
||||||||
The preview shows the first five and last five rows with a divider between them, so you see how the data looks at both ends of the table without scrolling through the middle. Each column header carries its data type, which is often where the first surprise appears: a date read as a string, or an integer column that arrived as a float. The c column here is worth noting, because a couple of its cells are empty, and that is the missing data we will return to later in the chapter.
3.2.1 Focusing the preview
On a wide table, showing every column defeats the purpose of a compact view. The columns_subset= argument narrows the preview to the columns you actually care about, and n_head= and n_tail= control how many rows appear at each end. Together they let you frame a preview around a specific question rather than dumping the entire table.
pb.preview(small_table, columns_subset=["a", "c", "f"], n_head=3, n_tail=3)PolarsRows13Columns8 |
|||
Here we have asked for just three columns and three rows at each end, which is enough to check that a looks like a small integer, c holds counts with the occasional gap, and f takes a limited set of text values. Narrowing the view this way is especially useful on tables with dozens of columns, where the signal you want is easily lost among columns you do not. A quick, focused preview is usually all it takes to decide which columns deserve closer profiling.
3.3 Profiling columns with col_summary_tbl()
A preview shows you individual rows, but it cannot tell you the character of a column across all of its values. For that you need a profile: the type of each column, how many values are missing, how many are distinct, and the range and central tendency of the numbers. The col_summary_tbl() function computes all of this in one call and presents it as a single readable table.
Function signature
col_summary_tbl(data, tbl_name=None)Running it on small_table produces a column-by-column summary of the entire table.
pb.col_summary_tbl(small_table)PolarsRows13Columns8 |
|||||||||||||
| Column | NA | UQ | Mean | SD | Min | P5 | Q1 | Med | Q3 | P95 | Max | IQR | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Datetime(time_unit='us', time_zone=None) |
0 0 |
12 0.92 |
- | - | 2016 01 04 00:32:00 |
- | - | - | - | - | 2016 01 30 11:23:00 |
- | |
Date |
0 0 |
11 0.85 |
- | - | 2016 01 04 |
- | - | - | - | - | 2016 01 30 |
- | |
Int64 |
0 0 |
7 0.54 |
3.77 | 2.09 | 1 | 1.06 | 2 | 3 | 4 | 7.4 | 8 | 2 | |
String |
0 0 |
12 0.92 |
9 | 0 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 0 | |
Int64 |
2 0.15 |
7 0.54 |
5.73 | 2.72 | 2 | 2.05 | 3 | 7 | 8 | 9 | 9 | 5 | |
Float64 |
0 0 |
12 0.92 |
2,304.7 | 2,631.36 | 108.34 | 118.88 | 837.93 | 1,035.64 | 3,291.03 | 6,335.44 | 9999.99 | 2,453.1 | |
Boolean |
0 0 |
T0.62 F0.38 |
- | - | - | - | - | - | - | - | - | - | |
String |
0 0 |
3 0.23 |
3.46 | 0.52 | 3 | 3 | 3 | 3 | 4 | 4 | 4 | 1 | |
| String columns statistics regard the string's length. | |||||||||||||
Each row of this summary describes one column of the data. You can read off the storage type, the count of missing values, and the number of unique values, and for the numeric columns you also get statistics such as the mean, the minimum, and the maximum. This is the view that turns hunches into facts. The f column reveals itself as low-cardinality text, a strong hint that it is categorical and a candidate for a set-membership check, while the numeric columns expose the ranges you would use to set sensible boundaries. Reading a profile like this is usually the moment a validation plan starts to take shape in your mind.
3.3.1 Programmatic profiles with DataScan
The summary table is ideal for a human, but sometimes you want the profile as data rather than as a display, for example to store it, compare it against a later run, or drive logic in a script. The DataScan class is the engine behind the summary, and it exposes the profile in several forms. You construct it with a table, then ask it for whichever representation you need.
Class signature
scan = DataScan(data, tbl_name=None)
scan.get_tabular_report(show_sample_data=False) # the visual summary table
scan.summary_data() # the profile as a data frame
scan.to_json() # the profile as a JSON string
scan.save_to_json(output_file) # write the JSON to a file
scan.compare(baseline) # drift vs a baseline -> DataScanDiff
DataScan.load_from_json(input_file) # read a saved profile backThe get_tabular_report() method returns the same kind of summary we just saw, which confirms that col_summary_tbl() is really a convenient shortcut over DataScan.
scan = pb.DataScan(data=small_table)
scan.get_tabular_report()PolarsRows13Columns8 |
|||||||||||||
| Column | NA | UQ | Mean | SD | Min | P5 | Q1 | Med | Q3 | P95 | Max | IQR | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Datetime(time_unit='us', time_zone=None) |
0 0 |
12 0.92 |
- | - | 2016 01 04 00:32:00 |
- | - | - | - | - | 2016 01 30 11:23:00 |
- | |
Date |
0 0 |
11 0.85 |
- | - | 2016 01 04 |
- | - | - | - | - | 2016 01 30 |
- | |
Int64 |
0 0 |
7 0.54 |
3.77 | 2.09 | 1 | 1.06 | 2 | 3 | 4 | 7.4 | 8 | 2 | |
String |
0 0 |
12 0.92 |
9 | 0 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 0 | |
Int64 |
2 0.15 |
7 0.54 |
5.73 | 2.72 | 2 | 2.05 | 3 | 7 | 8 | 9 | 9 | 5 | |
Float64 |
0 0 |
12 0.92 |
2,304.7 | 2,631.36 | 108.34 | 118.88 | 837.93 | 1,035.64 | 3,291.03 | 6,335.44 | 9999.99 | 2,453.1 | |
Boolean |
0 0 |
T0.62 F0.38 |
- | - | - | - | - | - | - | - | - | - | |
String |
0 0 |
3 0.23 |
3.46 | 0.52 | 3 | 3 | 3 | 3 | 4 | 4 | 4 | 1 | |
| String columns statistics regard the string's length. | |||||||||||||
The difference is what else the scan can give you. Calling to_json() returns the entire profile as a JSON string, and summary_data() returns it as a data frame, either of which can be persisted and diffed over time to detect when a column’s shape has changed. That comparison-over-time use is the seed of drift detection, which we return to at the end of the chapter. For now, the important point is that the same profile is available both as something to read and as something to compute with.
3.4 Finding missing values with missing_vals_tbl()
Missing values are often the earliest and clearest signal of a data quality problem, but a single count of nulls hides more than it reveals. A column that is missing five percent of its values at random is a very different situation from a column that is completely empty for the most recent week of data. The missing_vals_tbl() function shows not just how much is missing but where, by dividing the table into row sections and reporting the missing proportion for each column within each section.
Function signature
missing_vals_tbl(data, missing=None)Running it on small_table makes the pattern in the c column visible.
pb.missing_vals_tbl(small_table)| Missing Values 2 in total | ||||||||||
PolarsRows13Columns8 |
||||||||||
| Column | Row Sector | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
NO MISSING VALUES PROPORTION MISSING: 0% 100% ROW SECTORS
| ||||||||||
The table breaks the rows into sections and shades each cell according to how much of that column is missing within that section. Because small_table is short, the effect is modest, but the structure is exactly what scales up: on a large table this view immediately distinguishes a column with scattered gaps from one whose missingness is concentrated in a particular stretch of rows. Reading the layout of missingness, rather than a single summary number, is what lets you tell an expected empty field from a genuine failure.
3.4.1 Reading missing-value patterns
The shape of missingness usually points at its cause. Values missing at random across the whole table tend to come from incidental collection failures, such as an occasional dropped sensor reading, and they rarely bias the data in any systematic way. Values that go missing together across several columns for the same rows point at a related cause, such as an optional section of a form that some respondents skip. Values missing in a clear block, for instance an entire recent period or every row from one region, almost always indicate a structural problem in a pipeline or source system that deserves immediate investigation.
Recognizing which of these you are looking at changes what you do next. Random, low-rate missingness might be tolerable and handled with a permissive threshold, whereas a systematic block of missing values is the kind of thing you want a validation to fail loudly on. The point of visualizing the pattern is to make that judgment on evidence rather than assumption.
3.4.2 Coded missing values
There is a subtler kind of absence that a null-based view cannot catch on its own, where a value is present but stands in for missingness, such as a -99 that means “not asked”. The missing argument accepts a specification of those coded values so they are counted as missing too, and that whole topic has its own treatment in Chapter 10. For plain nulls, though, this function is the fastest way to see where your data thins out.
3.5 Inspecting data where it lives
Everything so far has run against an in-memory DataFrame, but the same tools work without change on data that stays in a database or warehouse. This matters because the point of inspecting data early is to catch problems where they originate, and copying a large table out of its source just to look at it defeats that purpose. When you load a dataset with tbl_type="duckdb", Pointblank returns a table backed by the database rather than by memory, and the inspection functions treat it exactly like any other table.
game_revenue_db = pb.load_dataset(dataset="game_revenue", tbl_type="duckdb")
pb.preview(game_revenue_db)DuckDBRows2,000Columns11 |
|||||||||||
The preview looks and behaves the same as it did for the Polars table, even though the rows are being read from DuckDB rather than held in memory. The same is true of col_summary_tbl(), DataScan, and missing_vals_tbl(), so the profiling workflow you have learned transfers directly to backends you cannot fit in RAM. Writing the inspection once and running it wherever the data happens to live is a recurring theme, and it is covered in full in Chapter 19.
3.5.1 Connecting to a database
When the data is in a real database rather than a bundled DuckDB file, you reach it through a connection string. The connect_to_table() function opens a connection string and returns a table you can inspect, and print_database_tables() lists what is available when you are not sure of the exact table name.
Function signatures
connect_to_table(connection_string)
print_database_tables(connection_string)A typical session against a warehouse looks like the following, where you first list the tables and then connect to the one you want.
# See what tables exist
pb.print_database_tables("duckdb:///warehouse.ddb")
# Connect to a specific table and inspect it
sales = pb.connect_to_table("duckdb:///warehouse.ddb::sales")
pb.col_summary_tbl(sales)This example does not execute here because it needs a live database, but the shape of it is the whole point: once you have a connected table, every inspection tool in this chapter applies to it unchanged. Meeting data where it lives, rather than exporting it first, keeps your inspection honest about the data your pipeline will actually see.
3.6 From inspection to a validation plan
Inspection is only worthwhile if it changes what you do next, and what it should change is the validation plan you write. The findings from this chapter map onto specific kinds of checks in a fairly direct way. A column with missing values, like c in small_table, suggests a completeness check that either forbids nulls or holds their rate below a threshold. A low-cardinality text column, like f, suggests a set-membership check against its known values. The observed range of a numeric column gives you defensible boundaries for a between check, grounded in what the data actually contains rather than a guess.
It’s worth keeping in mind that a profile shows what the data currently looks like, while a validation says what it should look like, and those aren’t always the same thing. If col_summary_tbl() shows a column ranging from one to one hundred, that’s just what’s there today, not a rule. The actual requirement might be “values must be positive”, which means the right check is a lower bound of zero, not the observed maximum. A profile helps you figure out what checks make sense and what values to use, but deciding which checks actually matter is still up to you. Pointblank can also look at a profile and suggest a starting plan for you to edit, which is covered in Chapter 18. The steps for turning any of these observations into runnable checks are in Chapter 4.
3.7 Profiling and drift detection
Profiles become most valuable when you can save them and compare across runs, because the most revealing question about a table is often not what it looks like today but how it has changed. The DataScan class supports this directly. The save_to_json() method writes a profile to a file, and the load_from_json() class method reads one back, so a baseline captured from one time period can be stored and compared against a later scan.
january_data = pl.read_csv("assets/january_sales.csv")
baseline = pb.DataScan(january_data)
baseline.save_to_json("january.json")
# on a later run, load the stored profile to compare against
baseline = pb.DataScan.load_from_json("january.json")With two scans in hand, the compare() method measures the difference between them and returns a DataScanDiff. It reports two kinds of change: schema drift, where columns are added, removed, or change type, and statistical drift, where a column present in both scans has shifted its distribution. The following example profiles two monthly slices of a sales table. The February slice has gained a channel column, lost a legacy_code column, and seen its amount distribution move upward.
import numpy as np
import polars as pl
rng = np.random.default_rng(23)
january = pl.DataFrame({
"region": rng.choice(["North", "South", "East", "West"], 800).tolist(),
"amount": rng.normal(100, 15, 800).round(2),
"legacy_code": rng.integers(0, 4, 800),
})
february = pl.DataFrame({
"region": rng.choice(["North", "South", "East", "West"], 800).tolist(),
"amount": rng.normal(118, 15, 800).round(2),
"channel": rng.choice(["web", "app"], 800).tolist(),
})
pb.DataScan(february).compare(pb.DataScan(january)).get_tabular_report()| Profile Comparison: baseline vs current | |||||
| Row count: 800 (baseline) vs 800 (current) | |||||
| Column | Status | Type (Baseline) | Type (Current) | Changed Statistics | Drift Scores |
|---|---|---|---|---|---|
| region | Stats Changed | String | String | mean: 4.497 -> 4.48 std: 0.5003 -> 0.4999 |
PSI: 0.0031 |
| amount | Stats Changed | Float64 | Float64 | n_unique: 751 -> 744 mean: 99.26 -> 118.4 median: 99.73 -> 118.7 std: 14.95 -> 15.02 min: 56.33 -> 69.51 max: 155.4 -> 169.9 p05: 59.06 -> 77.91 q_1: 89.1 -> 108.9 q_3: 109.7 -> 128.6 p95: 122.6 -> 143.1 iqr: 20.62 -> 19.69 |
PSI: 1.4620 KS: 0.4913 (p=0.0000) |
| legacy_code | Removed | Int64 | |||
| channel | Added | String | |||
The report lays out both kinds of change. It names channel as added and legacy_code as removed, and for the columns present in both slices it reports drift scores: the population stability index (PSI) and, for numeric columns, the Kolmogorov-Smirnov statistic. The amount column, whose mean shifted from roughly 100 to 118, shows a PSI near 1.5 and a KS statistic of about 0.5 with a p-value of zero—an unmistakable shift. The region column, whose categories held steady, shows a PSI near zero. By a common rule of thumb, a PSI below 0.1 signals no meaningful change, 0.1 to 0.25 a moderate shift worth watching, and above 0.25 a significant one, which places amount firmly in alarm territory.
The same results are available programmatically for a monitoring job to act on. A DataScanDiff exposes has_changes as a quick test and to_dict() for the full structured result. The result’s columns_added, columns_removed, and per-column drift_scores give a scheduled comparison the information it needs to alert when a PSI crosses a threshold or a column disappears. This profiling-and-comparison foundation is what the observability discussions of Chapter 30 and Chapter 33 build on.
3.8 Summary
Inspection is what makes every later step possible, because a validation plan is only as good as the understanding behind it. This chapter walked through the tools Pointblank provides for building that understanding, moving from the broadest view to the most detailed.
We started with get_row_count() and get_column_count() for the two most basic facts about a table, then used preview() to look at representative rows with their types (and only looking at a subset of columns via columns_subset= when a table is wide). From there we profiled entire tables with col_summary_tbl(), and saw that DataScan is the engine underneath it (exposing the same profile as a data frame or as JSON that can be saved and compared across runs). That comparison capability is what the drift detection workflow builds on: DataScan.compare() returns a DataScanDiff that identifies schema changes and quantifies distributional shift with PSI and KS scores. We used missing_vals_tbl() to locate missing values and read their patterns. And we saw that all of these tools work unchanged on database-backed tables reached through connect_to_table().
The thread running through the chapter is a progression from broad to specific: confirm the dimensions, look at a few rows, profile every column, focus on the trouble spots, and then track how those profiles change over time. Following that progression turns what was once an unfamiliar table into a set of concrete observations. And it is these types of observations that are so very useful in the next chapter, where we turn them into a validation plan.