How to measure contextual classification accuracy with a labeled sample
Part of Contextual targeting
The short answer
Contextual classification accuracy is measured by labeling a random sample of pages by hand, then comparing the classifier's labels with the human ones. Precision says how many pages the classifier put in a category truly belong there. Recall says how many pages that belong were actually found. Report both, per label, with a confidence range.
Key takeaways
- A single accuracy percentage hides the two errors that matter, false matches and misses, so ask for precision and recall per label.
- The audit sample must be drawn at random from the pages the classifier actually scored, and labeled by people who cannot see its answer.
- Around 100 labeled pages per label gives a usable range; 30 pages gives a range too wide to support a decision.
- Whether a false match or a miss costs more depends on the job: targeting and avoidance invert the cost.
- Recheck accuracy on a calendar and after every model, taxonomy or content change, not only when a report looks wrong.
What does contextual classification accuracy actually measure?
A contextual classifier attaches labels to a page, such as an IAB Tech Lab Content Taxonomy (opens in a new tab) category, a brief topic or a theme to avoid. Accuracy asks a narrow question: when a person who knows the label definition reads the same page, do they agree with the label? It does not ask whether the page will perform.
Definition
- Labeled audit sample
- A set of pages, drawn at random from what the classifier scored, that trained people have labeled by hand without seeing the classifier's answer. Their labels are the reference the classifier is graded against.
Every comparison between the classifier and the reference falls into one of four cells. Google's machine learning course calls this the confusion matrix (opens in a new tab).
| Outcome | What happened | Example for a fishkeeping label |
|---|---|---|
| True positive | The classifier applied the label and the reference agrees | A guide to cycling a new freshwater tank, labeled fishkeeping |
| False positive (false match) | The classifier applied the label, the reference says it does not belong | A review of a public aquarium's new shark exhibit, labeled fishkeeping |
| False negative (miss) | The classifier did not apply the label, the reference says it belongs | A forum thread on treating ich in tropical fish, left unlabeled |
| True negative | The classifier left the label off and the reference agrees | A septic tank maintenance article, left unlabeled |
Plain accuracy, the share of all four cells that were right, sounds like the natural summary but misleads when a label is rare. Google's course gives the extreme case: a model that always predicts negative can score 99 percent on accuracy (opens in a new tab) on data where positives are 1 percent, despite being useless. When a contextual label is rare among the pages scored, a high overall figure mostly reflects the pile of true negatives.
What do precision and recall mean in plain words?
Precision and recall split accuracy into the two questions a buyer actually asks. Google's course defines precision as the proportion of the model's positive classifications that are actually positive, and recall as the proportion of all actual positives that were classified correctly as positives.
| Measure | The question it answers | How it is counted | A low score means |
|---|---|---|---|
| Precision | Of the pages carrying this label, how many truly belong? | True positives divided by all pages the classifier labeled | Ads land on pages that only look like a match, wasting spend or blocking good pages |
| Recall | Of the pages that truly belong, how many did the classifier find? | True positives divided by all pages the reference labeled | Good pages are left out of a target, or unsuitable pages slip past an avoidance rule |
The two measures pull against each other. A classifier that returns a score applies a label above a threshold. Google's course explains that raising the threshold usually cuts false positives but adds false negatives. Either number can be raised by moving the threshold, so a precision figure without its matching recall tells a buyer little.
How do you build a labeled audit sample that a classifier cannot game?
Six steps to a fair audit sample
- Freeze the label definitions first. One paragraph per label: what belongs, what does not, two borderline cases.
- Draw two random pools. Take one random set from pages the classifier labeled, which measures precision, and one from pages it scored but did not label, which measures recall. A sample of only labeled pages can never find a miss.
- Stratify where the plan needs it. Draw separate pools for each language, for news and for user posts when the brief covers them, so a strong result on English blog posts does not hide a weak one on French forum threads.
- Hide the classifier's answer. Labelers see the page and the definitions, never the score.
- Use two labelers per page. Splits go to a third person; frequent splits mean the definition needs work.
- Keep the sample apart from training data. Pages used to build or tune the classifier cannot grade it. The Media Rating Council's brand safety supplement (opens in a new tab) asks that training and evaluation data be distinct and mutually exclusive data sets.
The same MRC supplement, issued in September 2018, says that where reduced accuracy is expected in certain data conditions, such as smaller samples, this must be disclosed to users through estimates of error. A buyer's audit measures that error instead of assuming it is small.
How big should a labeled audit sample be?
Big enough that the range around each figure is narrower than the decision it supports. The NIST Engineering Statistics Handbook (opens in a new tab) recommends the Wilson method for confidence intervals on proportions and warns that the common normal approximation may not be accurate enough when the sample is very small. The table applies the Wilson method to a classifier that got 90 percent of labeled pages right.
| Pages labeled | Correct | Range the true figure likely sits in | Good enough for |
|---|---|---|---|
| 30 | 27 | 74.4 to 96.5 percent | Spotting a broken label, not judging a working one |
| 50 | 45 | 78.6 to 95.7 percent | A first look at a new label |
| 100 | 90 | 82.6 to 94.5 percent | A go or no go call on a label for one plan |
| 200 | 180 | 85.1 to 93.4 percent | Comparing this quarter's figure with last quarter's |
| 400 | 360 | 86.7 to 92.6 percent | A contract term or a published accuracy figure |
These are working sizes for a buyer's audit, not the MRC rule for verification companies, which requires samples built to a 99.7 percent confidence level.
Which errors cost a brand more, misses or false matches?
It depends on the job. Google's course says to favor recall when false negatives are more expensive, and precision when false positives are. In contextual planning, the same label sits on either side of that rule depending on whether it targets or avoids.
| Use of the label | Costlier error | Measure to protect | Typical response |
|---|---|---|---|
| Targeting a niche brief | False match, because spend goes to off-topic pages | Precision | Raise the threshold and accept a smaller list |
| Targeting a broad awareness brief | Miss, because reach is the goal | Recall | Lower the threshold and review a sample of what it adds |
| Avoiding floor level content | Miss, because one bad placement can become a story | Recall | Lower the threshold and accept some blocked good pages |
| Avoiding a mild sensitivity, such as news about a product recall | False match, because responsible news is blocked | Precision | Raise the threshold and send borderline pages to a person |
Platform settings show why avoidance recall is never perfect. Google's content suitability help (opens in a new tab) states that it can't guarantee that all related content will be excluded. Knowing the recall of an avoidance label tells a buyer how much slips through.
How often should classification accuracy be rechecked?
On a fixed calendar, and again whenever something changes underneath the label. The MRC supplement expects human review to work as an ongoing and continuous quality control, with its results used to update machine learning as frequently as is feasible and at least several times annually. A buyer can mirror that: a full audit each quarter for the labels that carry the plan, with spot checks between.
Triggers for an unscheduled accuracy check
- The classifier's model version or the taxonomy version has changed since the last audit.
- The plan adds a language, a country or a content type, such as forums or video pages, that the last sample did not cover.
- A large news event has changed what pages on the topic say, for example a recall, a strike or an election.
- A placement review found more than a handful of pages a reader would call off topic.
- The label's definition in the brief was rewritten, even slightly.
What does an accuracy audit look like on a real brief?
Fictional example
Freshwater fishkeeping supplies in Ontario
A fictional aquarium supply retailer with stores in Toronto, Hamilton and Ottawa wants display ads beside English articles about keeping freshwater fish at home. Its agency plans to use a classifier label mapped to Pets, Fish and Aquariums (IAB Tech Lab 3.1 ID 426) and to avoid pages about fish die offs from water contamination.
The frozen definition counts tank setup, fish health and aquarium plants, and excludes public aquariums (Zoos and Aquariums, ID 178), pet store openings and seafood recipes. Two labelers read 100 random labeled pages and 100 random unlabeled ones from the candidate sites.
Precision comes back high, with false matches clustered on public aquarium reviews. Recall is lower: forum threads on fish disease were often missed because they used species names and slang, not the word aquarium. The team keeps the label, adds forums as a separate stratum for the next audit, and records the definitions and model version the figures describe.
How does PlacementGraph make its classifications checkable?
Frequently asked questions
Is a vendor's published accuracy figure enough to rely on?
It is a starting point. Ask which labels it covers, the threshold, how many pages were labeled, whether they were kept apart from training data and when they were drawn. A figure for the whole taxonomy says little about the few labels a brief depends on, so audit those on the plan's own candidate pages.
Should the people who wrote the label definitions also label the sample?
One of them can, but not both labelers. Authors read their own intent into borderline pages and agree too easily. Pair one author with someone who knows only the written definition; their splits show where the definition needs another example.
Does a high precision score mean the campaign will perform?
No. Precision says the pages carrying a label are about what the label claims. It says nothing about whether readers act on the ad or whether the site is a quality publisher. Treat accuracy as a check on the targeting input, and test outcomes separately.
Can one audit sample cover English and French pages together?
Only if results are reported per language. A classifier can be strong in one language and weak in another, and a pooled figure averages that away. Draw a pool per language, use fluent labelers, and report precision and recall for each.
Sources
- Classification: Accuracy, recall, precision, and related metrics (opens in a new tab)Google for Developers, Machine Learning Crash Course
- Thresholds and the confusion matrix (opens in a new tab)Google for Developers, Machine Learning Crash Course
- 7.2.4.1. Confidence intervals (opens in a new tab)NIST/SEMATECH e-Handbook of Statistical Methods
- MRC Supplement to IAB Guidelines for the Conduct of Ad Verification: Enhanced Content Level Context and Brand Safety (September 2018) (opens in a new tab)Media Rating Council
- Content Taxonomy (opens in a new tab)IAB Tech Lab
- Content Taxonomy 3.1 (TSV) (opens in a new tab)IAB Tech Lab on GitHub
- About content suitability (opens in a new tab)Google Ads Help
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.