15 Data Quality Scoring

A validation report answers detailed questions: which checks failed, on how many rows, and at what severity. It does not directly answer the question a manager or a dashboard usually asks first, which is how healthy the data is overall and where it is weakest. Data quality scoring answers that. It combines the step results into a few scores, one for each of several quality dimensions and one overall, so you can describe a table’s condition with a number and track it over time.

The idea is that different checks measure different aspects of quality. A null check measures completeness, a set membership check measures validity, and a duplicate check measures uniqueness. Grouping checks by what they measure turns a long report into a short summary. This chapter covers the six dimensions Pointblank uses, how each check is assigned to one, and the methods that give you the per-dimension scores, the overall health score, a visual scorecard, and an assertion for enforcing minimum scores.

We use game_revenue and a plan whose checks cover three dimensions. A couple of the checks have failures, so the scores are not all 100.

game_revenue = pb.load_dataset(dataset="game_revenue", tbl_type="polars")

validation = (
    pb.Validate(data=game_revenue, tbl_name="game_revenue")
    .col_vals_not_null(columns="player_id", dimension="completeness")
    .col_vals_not_null(columns="session_id", dimension="completeness")
    .col_vals_in_set(columns="item_type", set=["iap", "ad"], dimension="validity")
    .col_vals_lt(columns="session_duration", value=40, dimension="validity")
    .rows_distinct(columns_subset=["session_id", "time"], dimension="uniqueness")
    .interrogate()
)

The dimension= argument on each step assigns it to a quality dimension, and the scores are grouped by these assignments.

15.1 The six quality dimensions

Pointblank uses six dimensions of data quality. Completeness is about whether required values are present, and null checks, rows_complete(), and most of the missing-value checks from Chapter 10 count toward it. Validity is about whether values have an allowed form, which covers the comparison, range, set, pattern, and format checks, along with col_vals_expr() and the schema checks. Uniqueness is about unwanted duplicates, which rows_distinct() measures. Consistency is about related values agreeing with each other, and conjointly(), col_missing_consistent(), col_vals_in_table(), and tbl_match() count toward it. Timeliness is about whether the data is recent, which data_freshness() measures. Volume is about whether the table is the right size, which row_count_match() and col_count_match() measure.

Where the six dimensions come from

These dimensions are not a Pointblank invention. Most of them (completeness, validity, uniqueness, consistency, and timeliness) appear in data quality frameworks such as the one published by DAMA, and you will see the same names in many data governance tools. Using familiar names means a scorecard makes sense to people who already work with these frameworks.

You can set a step’s dimension with the dimension= argument, as the plan above does, or leave it out and let Pointblank pick one based on the kind of check. For example, it assigns a null check to completeness and a distinctness check to uniqueness.

inferred = (
    pb.Validate(data=game_revenue)
    .col_vals_not_null(columns="player_id")
    .rows_distinct(columns_subset=["session_id", "time"])
    .interrogate()
)

inferred.get_dimension_scores()

{'completeness': 100.0, 'uniqueness': 98.9}

Without any dimension= arguments, the null check is scored under completeness and the distinctness check under uniqueness. Setting the dimension yourself is useful when you want a check counted under a different dimension than its default, or when you want the plan to state it explicitly. If you want to change the default for a kind of check across all your plans, the dimension_map= setting in pb.config() does that. For example, pb.config(dimension_map={"col_vals_lt": "consistency"}) would score every col_vals_lt() step under consistency.

A dimension only appears in the scores if at least one step measures it. A plan with only null and set checks covers completeness and validity, so the other four dimensions are missing from the result entirely. They are not reported as 100 or as 0. Adding volume checks, such as confirming that the table has 2,000 rows and 11 columns, adds volume to the scores.

scored = (
    pb.Validate(data=game_revenue)
    .col_vals_not_null(columns="player_id", dimension="completeness")
    .col_vals_in_set(columns="item_type", set=["iap", "ad"], dimension="validity")
    .rows_distinct(columns_subset=["session_id", "time"], dimension="uniqueness")
    .row_count_match(count=2000, dimension="volume")
    .col_count_match(count=11, dimension="volume")
    .interrogate()
)

scored.get_dimension_scores()

{'completeness': 100.0, 'validity': 100.0, 'uniqueness': 98.9, 'volume': 100.0}

Four dimensions now have scores. Volume is 100 because both count checks passed, and consistency and timeliness are still missing because no step measured them. This makes sense: if no check looked at a dimension, there is nothing to score, and reporting it as 100 would suggest the data had been checked when it had not. It also means your scores only cover what your plan checks, so a dimension you care about needs at least one check assigned to it.

15.2 Dimension scores

The get_dimension_scores() method returns a dictionary of scores from 0 to 100. A dimension’s score is the percentage of its test units that passed, counted across every step assigned to it. So a step that checks more data has more influence on the score than a step that checks less.

validation.get_dimension_scores()

{'completeness': 100.0, 'validity': 98.15, 'uniqueness': 98.9}

Completeness is 100, because both null checks passed on every row. Uniqueness is 98.9, because 22 of the 2,000 rows share a session and timestamp. Validity is 98.15, and this number shows how the counting works. The item type check passed on all 2,000 rows and the session duration check failed on 74, so validity is 3,926 passing test units out of 4,000. It is not the average of the two steps’ pass rates, and it is not a count of how many steps passed.

15.3 The overall health score

The get_health_score() method reduces the whole plan to a single number.

validation.get_health_score()

99.04

The score of 99.04 is the percentage of all test units that passed across every step: 9,904 out of 10,000. Steps with more test units count for more. This is the number to track from run to run, since a drop is an early sign that quality is slipping somewhere. If some dimensions matter more to you than others, the dimension_weights= setting in pb.config() gives them more influence on the overall score.

pb.config(dimension_weights={"completeness": 2.0, "validity": 1.0, "uniqueness": 1.0})
weighted_health = validation.get_health_score()

pb.config(report_incl_header=False, report_incl_footer=False)
weighted_health

99.31

With completeness counted double, the overall score goes up to 99.31. The weight applies to test units, so completeness now counts as 8,000 test units instead of 4,000, and all 8,000 passed. That gives 13,904 passing out of 14,000. The score goes up because completeness is the strongest dimension here. The second pb.config() call puts this chapter’s display settings back and clears the weights, since any setting you leave out goes back to its default. These settings are global, so it is a good idea to reset them after a one-off change like this.

15.4 The scorecard

Numbers work well for a pipeline, but a person reviewing quality usually wants something visual. The get_scorecard() method returns a Great Tables object that shows the overall health score along with the score for each dimension.

validation.get_scorecard()

DIMENSION	SCORE	UNITS
Dimension Scores — `game_revenue`
Health Score: 99%
Completeness	100%	4000 / 4000
Validity	98.15%	3926 / 4000
Uniqueness	98.9%	1978 / 2000

The scorecard shows the same figures we computed above, in a compact format that works well on a dashboard or in a report for stakeholders. Because it is a Great Tables object, you can export it the same way as the validation report, with write_raw_html() for an HTML file or save() for an image. The tabular report of Chapter 14 lists every check, while the scorecard gives the summary a non-technical reader is more likely to want.

15.5 Requiring minimum scores

Scores are also useful in automation. The assert_dimension_scores() method takes a dictionary of minimum scores and raises an AssertionError if any of the named dimensions scores below its minimum. Otherwise it does nothing. Requiring every dimension to be at least 95 passes here.

validation.assert_dimension_scores(
    thresholds={"completeness": 95, "validity": 95, "uniqueness": 95}
)
print("all dimensions met their minimums")

all dimensions met their minimums

Every dimension is above 95, so the assertion passes and the message prints. Raising the minimum for validity to 99 changes the outcome, since validity is 98.15.

try:
    validation.assert_dimension_scores(thresholds={"validity": 99})
except AssertionError as e:
    print(str(e).splitlines()[0])

Dimension health score(s) below the required minimum: validity (98.15% < 99%)

The error message names the dimension, its score, and the minimum (98.15% against 99%). In a test suite or pipeline, this is how you enforce a minimum quality level, failing the build or stopping the pipeline when a dimension drops too low, much like the assertion methods of Chapter 13. You can set default minimums once with the dimension_thresholds= setting in pb.config(), and Chapter 21 shows how to run this kind of check from the command line.

There is a catch here, and it comes from how dimensions are scored. A dimension that no step measures has no score, so a minimum set on it always passes.

validation.assert_dimension_scores(thresholds={"timeliness": 95})
print("timeliness minimum passed with no timeliness checks in the plan")

timeliness minimum passed with no timeliness checks in the plan

The assertion passes even though the plan has no timeliness check at all. A minimum score only protects the dimensions your plan actually checks, so a timeliness minimum means something only once the plan includes a freshness check. Before relying on a score assertion, make sure the plan has at least one check for every dimension you are asserting on.

15.6 Summary

Data quality scoring summarizes a validation as a few numbers by grouping checks according to what they measure. Pointblank uses six dimensions (completeness, validity, uniqueness, consistency, timeliness, and volume), and each step is assigned to one, either through its dimension= argument or automatically based on the kind of check. The get_dimension_scores() method gives the percentage of test units that passed in each dimension. Only dimensions with at least one step appear, so an assertion on a dimension with no checks always passes.

The get_health_score() method gives one overall number to track over time, and the dimension_weights= setting gives some dimensions more influence on it. The get_scorecard() method shows the overall and per-dimension scores as a table you can share, and assert_dimension_scores() raises an error when a dimension falls below a minimum. The next chapter, Chapter 16, covers sending results and scores to alerting and monitoring systems.