game_revenue = pb.load_dataset(dataset="game_revenue", tbl_type="polars")
small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")
pb.preview(game_revenue)PolarsRows2,000Columns11 |
|||||||||||
6 Column Value Validations
NoteCompact validation reports
This chapter shows a validation report for almost every example, because seeing the result is the point. To keep those reports compact we hide the header and footer with pb.config(report_incl_header=False, report_incl_footer=False), which leaves just the step rows. In your own work you will usually want the header and footer visible for the context they add.
Most data quality problems show up in column values. You might find a negative number in a revenue column, for example, or an empty cell in a field that is supposed to be required. Column value validations catch problems like these by stating what you expect in a column and checking every row against it.
This chapter covers Pointblank’s column value methods in detail. It is the longest chapter in the book because these are the methods you will use most. For each family we look at more than one example, cover the edge cases that tend to trip people up, and talk about when to pick one method over another. Toward the end we look at column selectors, which let one step cover many columns, and the dimension= argument, which ties a check to the quality scoring of Chapter 15.
Two built-in datasets carry us through. The game_revenue table has a realistic mix of numeric, categorical, and identifier columns across 2,000 rows. The smaller small_table is useful when we want a result short enough to count by hand. A few tiny hand-built frames also appear, for cases where it is easier to construct an edge case than to go looking for one in real data.
The columns we will come back to most are the numeric session_duration and item_revenue, the categorical item_type and acquisition, and the identifiers player_id and session_id. Let’s start checking them.
6.1 Comparison validations
The most basic column checks compare each value against a fixed value or against another column. There are six of them, one for each of the usual comparison operators, and they all take the same arguments: columns= for what to check, value= for what to compare against, and na_pass= for how to treat nulls.
6.1.1 Greater than and less than
The col_vals_gt() method checks that values are greater than a given value. A good first use is asserting something you believe is always true, such as every session having a positive duration.
(
pb.Validate(data=game_revenue)
.col_vals_gt(columns="session_duration", value=0)
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_gt()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — |
All 2,000 test units pass (the shortest session in the table is 3.2). A check that passes may not seem to tell you much, but it will fail on the day an upstream change lets a zero or negative duration through. Often, though, you choose a comparison value to pick out a particular set of rows rather than to confirm something universal. Here we raise the bar to a revenue greater than 1.
(
pb.Validate(data=game_revenue)
.col_vals_gt(columns="item_revenue", value=1)
.interrogate()
)Now 604 rows pass and 1,396 fail. That is not a sign of bad data. Most rows in this table are ad impressions, which typically earn a small fraction of a unit, while in-app purchases usually earn more. The split is not perfectly clean, though: 172 ad rows earn more than 1, and 15 in-app purchases earn 1 or less. The counterpart, col_vals_lt(), checks an upper bound in the same way.
(
pb.Validate(data=game_revenue)
.col_vals_lt(columns="session_duration", value=40)
.interrogate()
)Here 74 of the 2,000 sessions have a duration of 40 or more and fail. Note that col_vals_gt() and col_vals_lt() are strict, so a value exactly equal to the comparison value fails. A session with a duration of exactly 40 would fail col_vals_lt(value=40), because 40 is not less than 40. When the boundary value should count as acceptable, you need the inclusive versions, which are covered next.
6.1.2 Inclusive versus strict bounds
The col_vals_ge() and col_vals_le() methods are the inclusive versions, meaning the boundary value passes. You only see a difference when a value sits exactly on the boundary, so let’s make a tiny frame that contains a zero.
import polars as pl
boundary = pl.DataFrame({"x": [0, 1, 2]})
(
pb.Validate(data=boundary)
.col_vals_gt(columns="x", value=0)
.interrogate()
)The strict col_vals_gt(value=0) fails the row with 0 and passes the rows with 1 and 2. Now the inclusive version:
(
pb.Validate(data=boundary)
.col_vals_ge(columns="x", value=0)
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_ge()
|
✓ | 3 | 3 1.00 |
0 0.00 |
— | — | — | — |
All three rows pass, because col_vals_ge() accepts 0 as meeting the bound. To choose between the two forms, ask whether the boundary value itself is valid. A stock count of zero for a sold-out product is a perfectly valid value, for instance, and the strict form would wrongly flag it. Revenue works the same way, so a non-negativity check on revenue should be inclusive.
(
pb.Validate(data=game_revenue)
.col_vals_ge(columns="item_revenue", value=0)
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_ge()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — |
Every row passes. As it happens, game_revenue has no zero-revenue rows (the smallest value is 0.004), so the strict and inclusive forms would give the same answer on this data. Using col_vals_ge() is still the right call, since it states the actual rule and will not fail if a free item shows up later.
6.1.3 Equality and inequality
The col_vals_eq() and col_vals_ne() methods check that values are equal, or not equal, to a given value. The item_type column holds only "ad" and "iap", so checking that every row equals "iap" splits the table in two.
(
pb.Validate(data=game_revenue)
.col_vals_eq(columns="item_type", value="iap")
.interrogate()
)There are 447 passing rows and 1,553 failing ones, the failures being the ad rows. In practice, equality is most useful when a column should hold a single constant, like a currency code that should always be "USD" in a domestic feed. The col_vals_ne() method forbids one specific value. Because item_type has only two values, checking that it is never "ad" gives exactly the same result as the check above: 447 passing and 1,553 failing.
Be careful using col_vals_eq() with floating-point numbers. The comparison is exact, and a value computed with arithmetic often does not exactly equal the number you would write for it.
computed = pl.DataFrame({"x": [0.1 + 0.2]})
(
pb.Validate(data=computed)
.col_vals_eq(columns="x", value=0.3)
.interrogate()
)The single row fails, because 0.1 + 0.2 comes out very slightly different from 0.3. This is not something Pointblank is doing. It is how floating-point arithmetic works, and it means col_vals_eq() is the wrong tool for computed decimals. To confirm that a computed value matches an expected one, allow a small tolerance, either by checking that the value falls within a narrow range or by writing an expression that compares with a tolerance (an approach covered in Chapter 12). Keep col_vals_eq() for exact values like integers, strings, and category codes.
NoteWhy
0.1 + 0.2 is not 0.3
Computers store most decimals in binary floating point. Just as one third has no exact finite decimal form, numbers like 0.1 and 0.2 have no exact binary form, so each is stored as the nearest value the format can represent. Those tiny rounding errors carry through arithmetic, and 0.1 + 0.2 ends up slightly away from the stored value of 0.3. Nearly every programming language uses this format for decimals, so comparing computed decimals for exact equality is a mistake everywhere, not just in Python or Pointblank.
6.1.4 Comparing against another column
The comparison value does not have to be a constant. If you wrap a column name in pb.col(), each row is compared against the value in that other column. This is how you check relationships between columns, such as a sale price being higher than its cost or an end date coming after a start date. Using small_table, we can check whether each value in column a is less than the value in column c.
(
pb.Validate(data=small_table)
.col_vals_lt(columns="a", value=pb.col("c"), na_pass=True)
.interrogate()
)Eight of the 13 rows pass and five fail, those being the rows where a is not less than c. Every comparison method accepts pb.col() in place of a literal value, so “the sale price is greater than the cost” becomes col_vals_gt(columns="sale_price", value=pb.col("cost")) and “the ship date is on or after the order date” becomes col_vals_ge(columns="ship_date", value=pb.col("order_date")). We set na_pass=True here because column c has two nulls. Without it, those two rows would also fail and the result would be 6 passing and 7 failing. How nulls are treated in comparisons is the topic of the next section.
6.1.5 Handling nulls in comparisons
Every comparison method has an na_pass= argument that decides what happens when a value is null. The default is False, which means a null counts as a failure: a missing value cannot be shown to meet the condition. Column c of small_table has two nulls, so we can see this with a check that values are at most 10.
(
pb.Validate(data=small_table)
.col_vals_le(columns="c", value=10)
.interrogate()
)Eleven rows pass and two fail. The two failures are not values above 10. They are the two nulls. If we set na_pass=True, nulls count as passing instead.
(
pb.Validate(data=small_table)
.col_vals_le(columns="c", value=10, na_pass=True)
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_le()
|
✓ | 13 | 13 1.00 |
0 0.00 |
— | — | — | — |
Now all 13 rows pass, since the 11 non-null values are all at most 10 and the two nulls are let through. Which setting is right depends on what a null means in your data. If a null means “not applicable”, such as a sale price that is null because the item is not on sale, then na_pass=True is correct because the check does not apply to that row. If a null means a value that should be there but is missing, keep the default, because then one step catches both out-of-range values and unexpected gaps. If missingness is the thing you want to measure, the methods in Chapter 10 are a better fit than toggling na_pass=.
6.2 Range validations
You could keep values within a range using two comparisons, one for the lower bound and one for the upper bound. A range validation does the same job in one step, and its report is easier to read because both bounds appear together instead of being spread across two steps.
6.2.1 Between
The col_vals_between() method checks that values fall within a range, and both ends are inclusive by default. Column a in small_table runs from 1 to 8, so a check with those bounds passes.
(
pb.Validate(data=small_table)
.col_vals_between(columns="a", left=1, right=8)
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_between()
|
✓ | 13 | 13 1.00 |
0 0.00 |
— | — | — | — |
All 13 rows pass. Because both ends are inclusive, the rows with exactly 1 and exactly 8 are accepted, which is usually what a rule like “ages between 18 and 65” means. The inclusive= argument controls this with a pair of booleans for the left and right ends. Setting both to False excludes the endpoints.
(
pb.Validate(data=small_table)
.col_vals_between(columns="a", left=1, right=8, inclusive=(False, False))
.interrogate()
)Now 11 rows pass and two fail. The failures are the row where a is 1 and the row where it is 8, since the endpoints are no longer part of the range. The two ends are set independently, so inclusive=(True, False) accepts the lower bound but not the upper, which suits a rule like “at least 0 and less than 100 percent”. As with the comparison methods, either bound can be a pb.col() reference, so you can check that a value sits between a per-row minimum and maximum. On game_revenue, a plausibility range for session duration gives a more typical mixed result.
(
pb.Validate(data=game_revenue)
.col_vals_between(columns="session_duration", left=5, right=40)
.interrogate()
)Here 1,915 rows fall between 5 and 40 and 85 fall outside. Of those 85, 14 are sessions shorter than 5 and 71 are longer than 40. Compare that with the 74 failures from col_vals_lt(value=40) earlier: three sessions last exactly 40, which the inclusive range accepts but the strict comparison rejected. These very short and very long sessions are the ones worth a closer look.
6.2.2 Outside
The col_vals_outside() method is the reverse. It passes values outside a range and fails values inside it. Let’s check that column a lies outside the range 3 to 6.
(
pb.Validate(data=small_table)
.col_vals_outside(columns="a", left=3, right=6)
.interrogate()
)Six rows pass because their values are below 3 or above 6, and seven fail because they fall between 3 and 6. Checking that values lie outside a range can seem backward, but it is a natural way to flag anomalies. If a range of numeric codes is reserved for errors, for example, any value inside that range deserves attention. Like col_vals_between(), this method takes an inclusive= pair that controls whether the endpoints count as inside the range.
6.3 Set membership validations
Ranges suit numeric data, but categorical columns need a different kind of check. When a column’s values come from a fixed list, such as a set of status codes, the right question is whether each value is on that list.
6.3.1 In set
The col_vals_in_set() method checks that each value belongs to a set of allowed values. The item_type column should only ever contain "iap" or "ad".
(
pb.Validate(data=game_revenue)
.col_vals_in_set(columns="item_type", set=["iap", "ad"])
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_in_set()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — |
Every row passes, so there are no unexpected values in the column. Keep in mind that the matching is exact, so a value like "IAP" or "iap " (with a trailing space) would fail. The acquisition column has six channels, and if we list only three of them we can see failures.
(
pb.Validate(data=game_revenue)
.col_vals_in_set(columns="acquisition", set=["organic", "facebook", "google"])
.interrogate()
)There are 1,594 passing rows and 406 failing ones, the failures being rows from the three channels we left out (other_campaign, crosspromo, and apple). It is worth knowing how this check treats a null, since a null is not one of the listed values. A small frame shows what happens.
with_null = pl.DataFrame({"s": ["a", "b", None]})
(
pb.Validate(data=with_null)
.col_vals_in_set(columns="s", set=["a", "b"])
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C66 | 1 |
col_vals_in_set()
|
✓ | 3 | 2 0.67 |
1 0.33 |
— | — | — | — |
Two rows pass and the null row fails, because the null is not in the set. If nulls are acceptable in the column, add None to the set itself (set=["a", "b", None] passes all three rows here). You can also pass a Python Enum as the set=, which lets you keep the list of allowed values in one place in your code.
6.3.2 Not in set
The col_vals_not_in_set() method works the other way, failing any value that appears in a forbidden set. This is useful when the list of valid values is too long or open-ended to write out, but you know a few specific values that indicate a problem, such as codes for discontinued products. Let’s forbid the value other_campaign in the acquisition column.
(
pb.Validate(data=game_revenue)
.col_vals_not_in_set(columns="acquisition", set=["other_campaign"])
.interrogate()
)There are 1,733 passing rows and 267 failing ones, which is every row tagged with other_campaign.
6.4 Null validations
Missing values are one of the most common data quality problems, and whether a null is acceptable depends on the column. Pointblank has a method for each case: one that requires values to be present and one that requires them to be absent.
6.4.1 Requiring values to be present
The col_vals_not_null() method checks that a required column has a value in every row. Both identifier columns in game_revenue should always be filled in.
(
pb.Validate(data=game_revenue)
.col_vals_not_null(columns="player_id")
.col_vals_not_null(columns="session_id")
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_not_null()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — | |||
| #4CA64C | 2 |
col_vals_not_null()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — |
Both steps pass for all 2,000 rows, so no identifiers are missing. Running the same check on column c of small_table finds its two nulls.
(
pb.Validate(data=small_table)
.col_vals_not_null(columns="c")
.interrogate()
)Eleven rows pass and two fail. This is probably the most common validation of all, since a missing value in a required column tends to cause trouble in everything downstream of it.
6.4.2 Requiring values to be absent
The col_vals_null() method checks the opposite, that a column is empty. You will need this less often, but it is useful when null is the correct value, such as an activation date that should stay unset for archived records, or a derived field that should not be filled in yet at a certain stage of a pipeline. Here is a small table where the column is all nulls.
archived = pl.DataFrame({"active_date": [None, None, None]}, schema={"active_date": pl.Datetime})
(
pb.Validate(data=archived)
.col_vals_null(columns="active_date")
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_null()
|
✓ | 3 | 3 1.00 |
0 0.00 |
— | — | — | — |
All three rows pass because every value is null, as expected. The two null methods are exact opposites. A column that is partly filled in and partly empty fails both of them: col_vals_not_null() fails on the empty rows and col_vals_null() fails on the filled ones.
6.5 Pattern matching
Numeric bounds and set membership do not help much with structured text. Many text columns, like product codes, follow a fixed format, and the usual way to check a format is with a regular expression.
6.5.1 Regular expressions
The col_vals_regex() method checks that text values match a regular expression, written in standard Python syntax. Column b in small_table holds values like "1-bcd-345": a digit, three lowercase letters, and three digits, separated by hyphens. We can require exactly that format.
(
pb.Validate(data=small_table)
.col_vals_regex(columns="b", pattern=r"^\d-[a-z]{3}-\d{3}$")
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_regex()
|
✓ | 13 | 13 1.00 |
0 0.00 |
— | — | — | — |
All 13 rows pass, so the column is consistently formatted. In the pattern, ^ marks the start of the string, \d is a single digit, [a-z]{3} is three lowercase letters, \d{3} is three digits, and $ marks the end, so any value that strays from this format fails. The method also has an inverse= argument that flips the check so that rows matching the pattern fail. That turns a pattern into a blocklist. Here we reject the value "high" in column f.
(
pb.Validate(data=small_table)
.col_vals_regex(columns="f", pattern="high", inverse=True)
.interrogate()
)Seven rows pass and six fail, and the six failures are exactly the rows where f is "high". Like the comparison methods, col_vals_regex() accepts na_pass= to control how nulls are handled. Regular expressions are powerful but can be hard to read, and for common formats there is an easier option, which the next section covers.
6.6 Validating against known formats
Some common formats are awkward to express as a regular expression, especially ones that include a checksum or vary by country. The col_vals_within_spec() method checks a column against a named specification and handles those details for you. Here is a short list of email addresses, some valid and some not.
contacts = pl.DataFrame({
"email": [
"grace@example.org",
"silly-sarah@example.com",
"not-an-email",
"bob@bad",
"carol@example.io",
]
})
(
pb.Validate(data=contacts)
.col_vals_within_spec(columns="email", spec="email")
.interrogate()
)Three of the five rows pass. The two failures are "not-an-email", which has no @ or domain at all, and "bob@bad", which has no top-level domain. The real advantage of a specification is that it can check things a pattern cannot. The credit card specification, for example, applies the Luhn checksum, so it can tell a number that only has the right number of digits from one whose digits actually check out.
cards = pl.DataFrame({"card": ["4111111111111111", "5500005555555559", "4111111111111112"]})
(
pb.Validate(data=cards)
.col_vals_within_spec(columns="card", spec="credit_card")
.interrogate()
)Two rows pass and the third fails. It has 16 digits like the others, but its Luhn checksum is wrong, which no digit-count pattern could detect. Some specifications depend on the country, and postal codes are one of them, so you include the country in the spec name.
zips = pl.DataFrame({"zip": ["90210", "12345", "9021", "ABCDE"]})
(
pb.Validate(data=zips)
.col_vals_within_spec(columns="zip", spec="postal_code[USA]")
.interrogate()
)Two rows pass and two fail: the four-digit "9021" and the non-numeric "ABCDE" do not fit the United States format. The country code is required, and either the two-letter or three-letter ISO code works ("postal_code[US]" or "postal_code[USA]"). The currently supported specifications are these:
"email","url","phone","ipv4","ipv6", and"mac"for contact and network values"isbn"for book numbers and"vin"for vehicle identification numbers"credit_card", which applies the Luhn checksum, and"swift"for bank identifier codes"postal_code[<country>]"and"iban[<country>]", which take an ISO country code
If the format you need is not on this list, col_vals_regex() is the fallback.
6.7 String length validations
Some columns are constrained by how long their text is rather than by what it says. A fixed-width identifier, for example, has to have exactly the right number of characters. The col_vals_str_len() method checks the number of characters in each string against a lower bound (min_val=), an upper bound (max_val=), or both, with one test unit per row.
To require an exact length, set min_val= and max_val= to the same number. Every value in column b of small_table is nine characters long, so we can check for that.
(
pb.Validate(data=small_table)
.col_vals_str_len(columns="b", min_val=9, max_val=9)
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_str_len()
|
✓ | 13 | 13 1.00 |
0 0.00 |
— | — | — | — |
All 13 rows pass. If a truncated or overlong code shows up later, this step will fail even though the bad value might otherwise look like a perfectly ordinary string.
More often a length has a range rather than a single allowed value. A username that must be between 3 and 15 characters is a typical example, and a small frame with two out-of-range values shows how the check behaves.
usernames = pl.DataFrame({"username": ["ann", "robert", "maximilian_alexander", "xy"]})
(
pb.Validate(data=usernames)
.col_vals_str_len(columns="username", min_val=3, max_val=15)
.interrogate()
)Two rows pass and two fail. "xy" has two characters, which is below the minimum, and "maximilian_alexander" has 20, which is above the maximum. You can use either bound on its own. Passing only min_val= sets a minimum with no maximum, which is handy for catching empty or truncated entries. Passing only max_val= sets a maximum with no minimum, for values that must fit a length limit. As with the other methods, na_pass= decides whether a null passes, and by default it does not.
6.8 Sequence validations
Ordered data comes with its own expectations. Timestamps in a log should move forward, for instance, and a running total should never go down. Pointblank can check these properties row by row down a column.
6.8.1 Increasing and decreasing values
The col_vals_increasing() method checks that values go up from one row to the next, and col_vals_decreasing() checks that they go down. A small table of made-up measurements shows both.
trend = pl.DataFrame({
"id": [1, 2, 3, 4, 5],
"temperature": [20, 22, 25, 28, 30],
"countdown": [100, 80, 60, 40, 20],
})
(
pb.Validate(data=trend)
.col_vals_increasing(columns="id")
.col_vals_increasing(columns="temperature")
.col_vals_decreasing(columns="countdown")
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_increasing()
|
✓ | 5 | 5 1.00 |
0 0.00 |
— | — | — | — | |||
| #4CA64C | 2 |
col_vals_increasing()
|
✓ | 5 | 5 1.00 |
0 0.00 |
— | — | — | — | |||
| #4CA64C | 3 |
col_vals_decreasing()
|
✓ | 5 | 5 1.00 |
0 0.00 |
— | — | — | — |
All three steps pass: id and temperature go up and countdown goes down. By default these checks are strict, so a value that is the same as the one before it fails. The allow_stationary= argument relaxes that. A column with the values 1, 2, 2, 3 shows the difference.
ties = pl.DataFrame({"x": [1, 2, 2, 3]})
(
pb.Validate(data=ties)
.col_vals_increasing(columns="x")
.interrogate()
)With the strict default, the repeated 2 fails, giving three passes and one failure. The same check with allow_stationary=True passes all four rows. Use the strict form when a repeat means something has gone wrong, such as a duplicated sequence number, and allow repeats when equal neighbors are normal, such as events that share a timestamp. Measured data is also often noisy, so a series that is decreasing overall might tick up slightly now and then, and a strict check would fail those rows. The tolerance arguments handle this. For example, col_vals_decreasing() takes an increasing_tol= value that allows upward steps up to a given size (and col_vals_increasing() has a matching decreasing_tol=).
readings = pl.DataFrame({"v": [100, 90, 92, 80]})
(
pb.Validate(data=readings)
.col_vals_decreasing(columns="v", increasing_tol=5)
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_decreasing()
|
✓ | 4 | 4 1.00 |
0 0.00 |
— | — | — | — |
The series goes from 90 up to 92 before continuing down, an increase of 2. Without a tolerance, that row would fail (three passes and one failure), but increasing_tol=5 allows increases of up to 5, so all four rows pass. A good starting point is a tolerance slightly larger than the normal noise in your measurements.
6.9 Selecting columns
So far each step has checked a single named column, but often you want the same check on several columns. Writing a near-identical step for each one is tedious and easy to get wrong. Column selectors let one step cover a group of columns.
6.9.1 Explicit lists and selector functions
The simplest option is a list of column names, which creates one separate step per column so each column’s result is reported on its own. To select columns by name pattern instead, Pointblank has selector functions. The contains() selector, for example, picks every column whose name includes a given substring, so we can check both identifier columns in game_revenue without naming them.
(
pb.Validate(data=game_revenue)
.col_vals_not_null(columns=pb.contains("id"))
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_not_null()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — | |||
| #4CA64C | 2 |
col_vals_not_null()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — |
The one step expands into two, one for player_id and one for session_id, and both pass. The other selectors match in different ways. starts_with("item") would pick item_type, item_name, and item_revenue. ends_with() matches a suffix. matches() takes a regular expression, so matches("^s") would pick every column starting with “s”, which here means session_id, session_start, session_duration, and start_day. You might not have expected start_day in that group, so it pays to check which columns a selector picks up.
Because selectors are resolved against the data when the plan runs, a new column that fits the pattern is picked up automatically without changing the plan. You can also combine selectors inside pb.col(). For example, pb.col(pb.starts_with("player") | pb.ends_with("_id")) picks columns that match either rule, a - between two selectors picks the first group minus the second, and everything() picks every column when a check should apply to the whole table.
6.10 Tagging checks with a quality dimension
Every method in this chapter has a dimension= argument that labels the check with the aspect of data quality it measures, such as completeness, validity, or uniqueness. If you leave it unset, Pointblank picks a dimension based on the kind of check. Setting it yourself makes your intent explicit, and it feeds into the quality scores described in Chapter 15.
(
pb.Validate(data=game_revenue)
.col_vals_in_set(columns="item_type", set=["iap", "ad"], dimension="validity")
.col_vals_not_null(columns="player_id", dimension="completeness")
.interrogate()
)| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_vals_in_set()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — | |||
| #4CA64C | 2 |
col_vals_not_null()
|
✓ | 2000 | 2000 1.00 |
0 0.00 |
— | — | — | — |
The checks give the same results as before, but each now carries a dimension label that is used when scores are calculated per dimension. In this case the labels match what Pointblank would have assigned anyway (col_vals_in_set() defaults to validity and col_vals_not_null() to completeness), so setting dimension= matters most when you want to override the default. You can validate a table perfectly well without ever setting a dimension. It becomes useful once you start tracking quality scores across the six dimensions.
6.11 Beyond single-column checks
Every method in this chapter judges one value at a time, but they all accept extra arguments that go further, and related features have their own chapters. The missing= argument lets a check recognize coded missing values, such as a -99 that means “not asked”, which Chapter 10 covers. The segments= argument splits one check into a separate result for each group, so you can see whether failures are concentrated in a particular region or cohort, which is the topic of Chapter 11. When a rule involves several columns or needs custom logic, the pre= argument, the col_vals_expr() method, and the conjointly() and specially() methods give you more flexibility, and Chapter 12 covers them.
6.12 Bringing it together
A real validation plan combines several of these checks into one report. The plan below runs a few column checks from different families against game_revenue.
(
pb.Validate(data=game_revenue, tbl_name="game_revenue", label="Column checks")
.col_vals_gt(columns="session_duration", value=0)
.col_vals_lt(columns="session_duration", value=40)
.col_vals_ge(columns="item_revenue", value=0)
.col_vals_in_set(columns="item_type", set=["iap", "ad"])
.col_vals_not_null(columns="player_id")
.interrogate()
)The report has all five checks in one place. Four of them pass for every row, confirming positive durations, non-negative revenue, valid item types, and no missing player IDs. The upper-bound check on session duration flags the same 74 long sessions we found earlier.
6.13 Summary
Column value validations are how you state what good data looks like, one column at a time. The comparison methods col_vals_gt(), col_vals_lt(), col_vals_ge(), col_vals_le(), col_vals_eq(), and col_vals_ne() check values against a fixed value or, with pb.col(), against another column. The strict and inclusive versions differ only at the boundary, and na_pass= decides how nulls are treated. Two things are worth remembering: the strict comparisons fail a value that equals the boundary, and exact equality is the wrong tool for computed floating-point numbers.
The range methods col_vals_between() and col_vals_outside() check whether values fall inside or outside an interval, with inclusive= controlling the endpoints. For categorical data, col_vals_in_set() and col_vals_not_in_set() check against allowed and forbidden sets, and a null fails an allowed-set check unless you add None to the set. The col_vals_not_null() and col_vals_null() methods require values to be present or absent. Structured text is handled by col_vals_regex(), whose inverse= option turns a pattern into a blocklist, and by col_vals_within_spec() for named formats such as email addresses, credit card numbers, and country-specific postal codes and IBANs.
String length is checked with col_vals_str_len(), using a minimum, a maximum, or an exact length. Ordered data is checked with col_vals_increasing() and col_vals_decreasing(), which are strict by default but can allow repeats with allow_stationary= and small reversals with a tolerance argument.
Column selectors let one step cover many columns, and dimension= ties each check to quality scoring.
The next chapter, Chapter 7, moves from checking individual values to checking column-level summaries such as sums and averages.