Some expectations are hard to write as code. Whether a product review is a real, on-topic comment is a judgment about meaning, and no regular expression or set membership check can make it. The prompt() method handles cases like this. It sends each row to a language model along with a plain description of what a good row looks like, and turns the model’s answer into an ordinary pass or fail test unit that appears in the report next to your other checks.
This comes with costs that regular checks do not have. Each check costs money, takes much longer to run, and may not give exactly the same answer twice. So semantic validation is something to use on purpose, not by default. This chapter covers how the method works, how to write prompts that give reliable answers, how to include reference material as attachments, how to manage the cost, and how the results fit into the rest of a validation plan. We use a small table of product reviews for the examples.
reviews = pl.DataFrame({
"review_id": [1, 2, 3, 4],
"product": ["wireless mouse", "desk lamp", "coffee mug", "usb cable"],
"review_text": [
"Works great, very responsive and comfortable to use all day.",
"asdkjfh lorem ipsum dolor sit amet",
"The lamp is bright and the base is sturdy. Very happy with it.",
"n/a",
],
})
pb.preview(reviews, n_head=4, n_tail=0)PolarsRows4Columns3 |
|||
Reviews 1 and 3 are real comments about a product, while reviews 2 and 4 are junk. A length check would catch "n/a", but not the keyboard mash and placeholder text in review 2, which is 34 characters long. This is the kind of situation semantic validation is for.
prompt() worksThe prompt() method takes a plain-language description of what makes a row valid and the name of a model, and asks the model to judge each row against that description. Each row is a test unit that passes or fails, and the results show up in the report like any other check. The columns_subset= argument limits which columns are sent to the model, which keeps the model focused and avoids sending data it does not need.
(
pb.Validate(data=reviews)
.prompt(
prompt="The review_text should be a coherent, on-topic comment about the product.",
columns_subset=["product", "review_text"],
model="openai:gpt-4o-mini",
)
.interrogate()
)The model name has the form provider:model. The supported providers are "anthropic", "openai", "ollama", "bedrock", and "azure-openai", and the part after the colon is the specific model. You never pass an API key as an argument. Pointblank reads it from an environment variable or from a .env file in your project, so an ANTHROPIC_API_KEY or OPENAI_API_KEY variable is picked up automatically. Ollama runs models on your own machine and needs no key. Rows are sent to the model in batches (1,000 rows per batch by default, set with batch_size=), but each row still gets its own result. For this table, you would expect reviews 1 and 3 to pass and reviews 2 and 4 to fail.
Unlike every other check in this book, a semantic check sends the row values it checks to a model provider. With a hosted provider, that data leaves your environment. List only the columns the check needs in columns_subset=, and for sensitive data use the local "ollama" provider or a regular check instead. The privacy section later in this chapter goes into more detail.
Sometimes the standard depends on reference material that is hard to put into words, such as a brand style guide or a diagram. The attachments= argument accepts images (.png, .jpg, .jpeg, .gif, and .webp) and PDFs, and sends them to the model as context that applies to every row. Each row’s own values are still sent as data.
(
pb.Validate(data=reviews)
.prompt(
prompt="The review_text should match the tone and vocabulary in the attached brand guide.",
columns_subset=["product", "review_text"],
attachments=["docs/brand_guide.pdf"],
model="anthropic:claude-sonnet-5",
)
.interrogate()
)With the attachment, the model can compare each review against the guide without the guide’s rules having to be written into the prompt. Attachments do add cost, though, because an attachment is sent again with every batch. A table big enough to need several batches pays for the attachment several times. To keep this down, keep attachments small (for example, by shrinking an image or cutting a PDF down to the pages that matter) and fit more rows into each batch.
Every semantic check makes real API calls. That means it costs money for each row, runs much slower than a regular check, and may not return the same result on the same input every time, since models are not fully deterministic. The batch_size= and max_concurrent= arguments control throughput. Larger batches send more rows per call, and higher concurrency sends more calls at the same time (the default is 3). Both make the check finish sooner, and both make it more likely you hit a provider’s rate limits.
(
pb.Validate(data=reviews)
.prompt(
prompt="The review_text should be a coherent, on-topic comment about the product.",
columns_subset=["product", "review_text"],
model="openai:gpt-4o-mini",
batch_size=100,
max_concurrent=1,
)
.interrogate()
)A semantic check sends the data it checks to the model provider you choose. With a hosted provider, the row values leave your environment. For public data like product reviews that is usually fine, but for records with personal, financial, or health information, you need to think about it before using the method at all. There are two ways to reduce the risk. The first is columns_subset=, which should list only the columns the check needs. A check on review quality needs the product and the review text, not the customer ID or email address that might be in other columns. Sending less data is both cheaper and safer.
The second is your choice of provider. The "ollama" option runs a model locally, so no data leaves your machine. That makes it the natural choice for sensitive data, as long as a local model is good enough for the job. A hosted provider such as Anthropic or OpenAI is usually more capable but means sending the data out. For regulated data, a local model or a regular check is usually the safer default, and hosted models are best kept for data you are comfortable sending.
A semantic check is a regular step in a plan. Its results appear in the report like any other check’s, thresholds from Chapter 13 decide how many rows can fail before it counts as a warning or an error, a dimension= tag includes it in the scores from Chapter 15, and extracts from Chapter 14 let you pull out the rows the model flagged.
(
pb.Validate(data=reviews, thresholds=pb.Thresholds(warning=0.25))
.col_vals_not_null(columns="review_text")
.prompt(
prompt="The review_text should be a coherent, on-topic comment about the product.",
columns_subset=["product", "review_text"],
model="openai:gpt-4o-mini",
dimension="validity",
)
.interrogate()
)This plan combines a regular completeness check with a semantic validity check in one interrogation. The first step confirms that no review is missing, and the second checks whether the reviews that are present actually say something sensible. If the model fails reviews 2 and 4, that is a failure rate of 0.5, which is above the 0.25 warning threshold.
Since the model is the expensive part, you want it to judge as few rows as possible. The pre= argument from Chapter 12 lets a semantic step run on a filtered set of rows instead of the whole table. You can use it to remove rows that a cheap rule can already reject, such as empty or very short reviews, so the model only sees the rest.
def substantial_reviews(df):
return df.filter(pl.col("review_text").str.len_chars() > 15)
(
pb.Validate(data=reviews)
.prompt(
prompt="The review_text should be a coherent, on-topic comment about the product.",
columns_subset=["product", "review_text"],
model="openai:gpt-4o-mini",
pre=substantial_reviews,
)
.interrogate()
)The filter removes the "n/a" review (3 characters) before the model sees it, so the model only judges the other three rows. Review 2 is 34 characters long, so it gets through the filter and it is up to the model to reject it. This is the general pattern for keeping semantic validation affordable: let regular checks catch the obvious problems and save the model for the rows only a judgment about meaning can settle.