game_revenue = pb.load_dataset(dataset="game_revenue", tbl_type="polars")
validation = (
pb.Validate(data=game_revenue, tbl_name="game_revenue")
.col_vals_not_null(columns="player_id", dimension="completeness")
.col_vals_not_null(columns="session_id", dimension="completeness")
.col_vals_in_set(columns="item_type", set=["iap", "ad"], dimension="validity")
.col_vals_lt(columns="session_duration", value=40, dimension="validity")
.rows_distinct(columns_subset=["session_id", "time"], dimension="uniqueness")
.interrogate()
)15 Data Quality Scoring
A validation report answers detailed questions: which checks failed, on how many rows, and at what severity. It does not directly answer the question a manager or a dashboard usually asks first, which is how healthy the data is overall and where it is weakest. Data quality scoring answers that. It combines the step results into a few scores, one for each of several quality dimensions and one overall, so you can describe a table’s condition with a number and track it over time.
The idea is that different checks measure different aspects of quality. A null check measures completeness, a set membership check measures validity, and a duplicate check measures uniqueness. Grouping checks by what they measure turns a long report into a short summary. This chapter covers the six dimensions Pointblank uses, how each check is assigned to one, and the methods that give you the per-dimension scores, the overall health score, a visual scorecard, and an assertion for enforcing minimum scores.
We use game_revenue and a plan whose checks cover three dimensions. A couple of the checks have failures, so the scores are not all 100.
The dimension= argument on each step assigns it to a quality dimension, and the scores are grouped by these assignments.
15.1 The six quality dimensions
Pointblank uses six dimensions of data quality. Completeness is about whether required values are present, and null checks, rows_complete(), and most of the missing-value checks from Chapter 10 count toward it. Validity is about whether values have an allowed form, which covers the comparison, range, set, pattern, and format checks, along with col_vals_expr() and the schema checks. Uniqueness is about unwanted duplicates, which rows_distinct() measures. Consistency is about related values agreeing with each other, and conjointly(), col_missing_consistent(), col_vals_in_table(), and tbl_match() count toward it. Timeliness is about whether the data is recent, which data_freshness() measures. Volume is about whether the table is the right size, which row_count_match() and col_count_match() measure.
NoteWhere the six dimensions come from
These dimensions are not a Pointblank invention. Most of them (completeness, validity, uniqueness, consistency, and timeliness) appear in data quality frameworks such as the one published by DAMA, and you will see the same names in many data governance tools. Using familiar names means a scorecard makes sense to people who already work with these frameworks.
15.2 Dimension scores
The get_dimension_scores() method returns a dictionary of scores from 0 to 100. A dimension’s score is the percentage of its test units that passed, counted across every step assigned to it. So a step that checks more data has more influence on the score than a step that checks less.
validation.get_dimension_scores(){'completeness': 100.0, 'validity': 98.15, 'uniqueness': 98.9}Completeness is 100, because both null checks passed on every row. Uniqueness is 98.9, because 22 of the 2,000 rows share a session and timestamp. Validity is 98.15, and this number shows how the counting works. The item type check passed on all 2,000 rows and the session duration check failed on 74, so validity is 3,926 passing test units out of 4,000. It is not the average of the two steps’ pass rates, and it is not a count of how many steps passed.
15.3 The overall health score
The get_health_score() method reduces the whole plan to a single number.
validation.get_health_score()99.0415.4 The scorecard
Numbers work well for a pipeline, but a person reviewing quality usually wants something visual. The get_scorecard() method returns a Great Tables object that shows the overall health score along with the score for each dimension.
validation.get_scorecard()Dimension Scores — game_revenue |
||
Health Score: 99% |
||
| DIMENSION | SCORE | UNITS |
|---|---|---|
| Completeness | 100% |
4000 / 4000 |
| Validity | 98.15% |
3926 / 4000 |
| Uniqueness | 98.9% |
1978 / 2000 |
The scorecard shows the same figures we computed above, in a compact format that works well on a dashboard or in a report for stakeholders. Because it is a Great Tables object, you can export it the same way as the validation report, with write_raw_html() for an HTML file or save() for an image. The tabular report of Chapter 14 lists every check, while the scorecard gives the summary a non-technical reader is more likely to want.