AI contextual targeting: how language models classify a page, and where they misread
Part of Contextual targeting
The short answer
AI contextual targeting uses a large language model to read a page's text and assign it labels, such as a topic or a sensitive theme, against definitions in a prompt. Models handle paraphrase well but misread satire, quoted speech and mixed pages, so every match should show the passage behind it and uncertain calls should go to a person.
Key takeaways
- A language model classifies a page by reading its text against label definitions and returning a label, which can come with a short rationale.
- Satire, quoted speech, mixed topic pages and text planted for the model are four misreads that can put a brand beside the wrong page.
- A rationale is only evidence if the passage it cites really appears on the page, and that can be checked automatically.
- Results can change with small edits to the prompt's format, so test across prompt variants before trusting a single score.
- Route avoidance labels, low confidence calls and hard content types to trained human reviewers.
How do large language models classify a web page?
One common way to build it has four stages. Software fetches the page and extracts the article text, dropping menus, ads and comments. The text goes to a language model with instructions: the list of allowed labels, a definition of each, and sometimes a few labeled examples. The model returns one or more labels, ideally with the sentence that justified each. A final layer checks the output and decides which pages go to a person.
Definition
- Zero shot and few shot classification
- Zero shot means the model labels a page from definitions alone, with no examples. Few shot means the prompt also includes a handful of labeled examples to show what each label looks like.
The strength of this approach is reading like a person: a model can tell that a review of a computer mouse has nothing to do with pest control, and can label a niche topic no standard category describes. The weakness is that the model is not reading for the brand's purpose unless the definitions say exactly what that purpose is, and it can be confidently wrong.
What errors do AI classifiers make on news and satire?
Satire is the classic trap because it copies the form of real reporting. Researchers at the University of Western Ontario described satirical news as content that mimics the format and style of journalistic reporting. In their 2016 study (opens in a new tab), a detector built specifically for satire reached 90 percent precision and 84 percent recall on 360 news articles. A general purpose classifier asked about topics, not satire, gets no such focused help.
| Misread | What goes wrong | Why it happens |
|---|---|---|
| Satire read as news | A parody story about a city council is labeled as real local politics news | Satire uses the vocabulary and structure of news; the joke sits in absurd details, not keywords |
| Quoted speech read as the page's stance | A news report quoting a hateful remark is labeled as hate speech | The model can label the words on the page without separating the reporter from the person quoted |
| Mixed page read by one part | A long neighborhood guide with one home section and a crime report in a sidebar gets one label | Long inputs are read unevenly; research finds information in the middle of a long input is used less reliably |
| Prompt format changes the answer | The same page gets different labels after a harmless edit to the instructions | Studies show large swings in accuracy from small formatting changes |
| Planted instructions | Hidden text on the page tells the model to label it as family friendly | The model treats page text as input it may follow, a risk known as indirect prompt injection |
The research behind the last three rows is specific. Lost in the Middle (opens in a new tab) (Liu and colleagues, 2023) found performance is often highest when relevant information occurs at the beginning or end of the input, and degrades when models must use information in the middle. Sclar and colleagues (opens in a new tab) (2023, ICLR 2024) measured differences of up to 76 accuracy points from prompt formatting alone with one open model. The OWASP Gen AI Security Project's LLM01:2025 (opens in a new tab) says indirect prompt injections occur when a model accepts input from external sources, such as websites or files.
Why should every AI match show its evidence?
Because a model's explanation can be invented. NIST's Generative AI Profile (opens in a new tab) (NIST AI 600-1, July 2024) names confabulation as a risk: the production of confidently stated but erroneous or false content. A label that comes with a fluent reason looks trustworthy whether or not the reason is true.
The fix is to require evidence that can be checked mechanically. Ask the model to return the exact sentence or passage that supports each label, then confirm that the passage appears word for word in the extracted text. A label whose quote cannot be found on the page is treated as unsupported, however confident the wording. This one check catches invented quotes without anyone reading the page.
When should a person review an AI classification?
When the cost of an error is high or the content is known to be hard. The Media Rating Council's brand safety supplement (opens in a new tab) (September 2018) says machine learning should be deployed in conjunction with focused human intervention, driven by risk based criteria that include content types that are more difficult to measure or categorize, user generated content and higher traffic pages. It also asks that reviewers be adequately trained and supervised.
Send the page to a person when
- The label is an avoidance label that would block or allow a sensitive theme.
- The model's confidence, or agreement across repeated runs, falls below the team's set threshold.
- The site publishes satire, opinion columns or heavily quoted reporting.
- The cited passage is a quotation, a headline only, or sits in a sidebar rather than the article body.
- The page is in a language or dialect the classifier was not tested on.
- The page contains text that addresses the reader as a machine or tells a system how to label it.
Human review is also the feedback loop. Pages a reviewer corrects become test cases for the next model or prompt version, so the same misread does not return unnoticed.
How do you test an AI classifier before trusting it?
A pre launch test for an AI classifier
- Build a labeled test set first. Hand label a random sample of real candidate pages against the written definitions, as described in our guide to classification accuracy.
- Add a hard case set. Include satire, reports quoting offensive speech, mixed topic pages, very long pages and pages with planted instructions.
- Run several prompt variants. Change wording, order of labels and formatting. Sclar and colleagues recommend reporting a range of performance across plausible prompt formats rather than a single format.
- Shuffle the examples. Zhao and colleagues (opens in a new tab) (ICML 2021) found few shot results can be unstable with the choice of prompt format, examples and even their order, with bias toward answers placed near the end of the prompt.
- Rerun the same pages. A page that flips labels between identical runs is a page the classifier is unsure about.
- Verify every cited passage. Count how often the quoted evidence is missing from the page.
- Record the versions. Note the model, prompt and definitions tested, so a later change triggers a new test.
| Measure | Why it matters | Warning sign |
|---|---|---|
| Precision and recall per label | Shows false matches and misses separately | Strong average hiding one weak label |
| Spread across prompt variants | Shows how fragile the result is | Large gaps between variants |
| Run to run agreement | Shows stability on identical input | Labels that flip on rerun |
| Hard case pass rate | Shows behavior on satire, quotes and planted text | Planted instructions followed |
| Unsupported passage rate | Shows how often evidence is invented | Any rate above zero on avoidance labels |
What does an AI classification review look like on a real brief?
Fictional example
Home security cameras in Pennsylvania
A fictional retailer of home security cameras and video doorbells with stores in Philadelphia, Pittsburgh and Harrisburg wants display ads beside home setup guides, renter friendly security explainers and moving checklists. It avoids reporting on violent crime.
The test set turns up four problems. A satire site's story about a doorbell camera catching raccoons planning a heist is labeled home security advice. A consumer explainer on preventing package theft, which quotes a police officer, is labeled as crime news to avoid. A long neighborhood guide is labeled home setup while a sidebar carries a shooting report the model never mentions. One small site has hidden text asking automated systems to label every page family friendly.
The team adds satire and planted text to the hard case set, routes quoted reporting and avoidance labels to a reviewer, and requires a verified passage from the article body for every match. The satire domain and the site with hidden text are left off the list, each with its reason recorded.
How does PlacementGraph keep AI judgments checkable?
Frequently asked questions
Does an AI classifier give the same answer every time?
Not always. Depending on its settings, a language model can return different labels for the same page on different runs, and small prompt changes can shift results. Rerun a sample of pages, measure how often labels flip, and treat pages that flip as uncertain rather than averaging them into a single answer.
Can a text classifier judge images and video on the page?
A classifier that receives only extracted text cannot see images, video or charts, so a page whose meaning sits in a photo or clip may be misread. Ask what input the classifier actually receives. When visual content drives the risk, review those pages by hand or use a process that examines the media itself.
Does an audit still hold after the model is updated?
No. A new model, a rewritten prompt or changed label definitions can move results in either direction. Treat each version as a new classifier, rerun the labeled test set and the hard cases, and compare the figures before switching production lists to the new version.
Should a page be blocked when the AI flags it but cites no evidence?
Send it to a person rather than blocking or allowing it automatically. An unsupported flag may be a confabulation, or the risk may sit in something the passage check cannot capture, such as an image. A short review settles the case and adds a useful example to the test set.
Sources
- Fake News or Truth? Using Satirical Cues to Detect Potentially Misleading News (opens in a new tab)Rubin, Conroy, Chen and Cornwell, ACL Anthology (2016)
- Lost in the Middle: How Language Models Use Long Contexts (opens in a new tab)Liu and colleagues, arXiv (2023)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (opens in a new tab)Sclar, Choi, Tsvetkov and Suhr, arXiv (2023)
- Calibrate Before Use: Improving Few-Shot Performance of Language Models (opens in a new tab)Zhao and colleagues, arXiv (2021)
- LLM01:2025 Prompt Injection (opens in a new tab)OWASP Gen AI Security Project
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) (opens in a new tab)NIST
- MRC Supplement to IAB Guidelines for the Conduct of Ad Verification: Enhanced Content Level Context and Brand Safety (September 2018) (opens in a new tab)Media Rating Council
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.