How to verify contextual relevance, one article at a time
Part of Contextual targeting
The short answer
To verify contextual relevance, a reviewer reads several recent articles from each candidate site and scores each against the topic description, separating true matches from passing mentions. Two reviewers score independently, settle disagreements with a written tie break, and save the URL, date, archived copy and matching passage for every article that qualifies the site.
Key takeaways
- Verification happens before launch and judges articles, not domains or keyword hits.
- An article is a true match when the topic is its main subject and purpose, not when the topic appears somewhere on the page.
- A four point score from no match to true match makes reviewer verdicts comparable across sites.
- Two independent reviewers plus a written tie break keep the review from depending on one person's taste.
- Evidence saved at the time of review lets anyone check a match after the page has changed.
What does it mean to verify contextual relevance?
Definition
- Contextual relevance verification
- A manual, prelaunch check in which a person reads sample articles from a candidate site and records whether each one truly matches a topic in the brief, with evidence another person can audit.
Matching tools, keyword rules and taxonomy labels all produce candidates. None of them can say, in a way a client will accept, that a specific article is about the right thing. Verification is the human step that turns a candidate into a defensible entry on the list. It sits between search and approval, and it is different from checking served placements after launch, which the contextual targeting hub covers.
How many articles should a reviewer read per site?
Enough to see whether the match is a pattern or an accident. No industry standard sets this number, so the figures below are a working rule to adapt, not a benchmark. The principle is that sites with narrow coverage need fewer reads, and sites that publish on everything need more, drawn from the section where the ads would run.
| Site type | Articles to read | Where to draw them from |
|---|---|---|
| Specialist site on the topic | Five to eight | The most recent articles across the whole site |
| Specialist site on a nearby topic | Eight to ten | Recent articles, plus any tagged with the brief's subject |
| General lifestyle or magazine site | Ten or more | The section that covers the topic, and the homepage's current stories |
| News site | Ten or more | The relevant desk or section, across at least two weeks |
Read recent articles. A site's archive can hold excellent coverage from years ago while today's output has moved on, and the ads will run beside what is being published now. Note the date of each article on the scoring sheet so a reviewer can see how current the match is.
What makes an article a true match and not a passing mention?
Google's Search Quality Rater Guidelines (opens in a new tab), in the September 11, 2025 edition, give a useful frame. They define Main Content as any part of the page that directly helps the page achieve its purpose, and separate it from Supplementary Content, such as navigation links, and from advertisements. A true match lives in the main content and serves the page's purpose. A passing mention lives in a sidebar, a related links box, or one sentence of an article about something else.
| Score | Label | What the reviewer sees |
|---|---|---|
| 3 | True match | The topic is the headline subject and the article's purpose; the narrative's counts line is met |
| 2 | Partial match | The topic is a substantial part of the main content, such as one full section, but not the purpose |
| 1 | Passing mention | The topic appears in a sentence, a caption, a list item or supplementary content only |
| 0 | No match | The topic is absent, or the article fits the narrative's does not count line |
Four questions that decide the score
- Read the headline and first paragraph. Ask whether a reader would describe the article as being about the topic.
- Find the purpose. Ask what the reader is meant to do or learn, and whether the topic is that purpose.
- Measure the share. Ask whether the topic runs through the main content or sits in one place.
- Check the narrative. Ask whether the article falls under any line that says what does not count.
Score the site, not only the articles. A site qualifies for a topic when most of its sampled articles score 2 or 3, with at least one 3. A site with one 3 and several 0s is a site that once wrote a good article, which is a reason to watch it, not to list it.
How do you spot a match driven by one repeated keyword?
Some articles match because a word appears often, not because the subject is treated. Google's spam policies (opens in a new tab) describe keyword stuffing as filling a web page with keywords or numbers in an attempt to manipulate rankings, often in a list or group, unnaturally, or out of context. Pages written that way can fool a keyword rule and sometimes a meaning model. A reviewer can catch them with a few plain tests.
Signs a match rests on one repeated keyword
- The topic word appears in the headline and many times in the body, yet the article explains nothing about it.
- Paragraphs could swap the keyword for an unrelated product without any other change.
- The article ends in a list of near identical phrases or place names.
- Several articles on the same site share the same structure with only the keyword changed.
- Removing every sentence with the keyword leaves an article about something else.
The last test carries the most weight in this protocol. Mentally strike every sentence that contains the keyword and read what remains. A true match still makes sense as an article on the topic, because the subject runs through examples, steps and detail. A keyword driven page collapses into filler, and it scores 0 regardless of how often the word appears.
How should two reviewers settle a disagreement?
Two reviewers who score separately will disagree on some articles, and that is useful information. Raw percent agreement flatters the review: Mary McHugh's guide to the kappa statistic (opens in a new tab) notes that percent agreement does not account for raters guessing and may overestimate true agreement. Kappa corrects for chance, and McHugh treats any kappa below 0.60 as inadequate agreement among raters.
A tie break routine for split scores
- Score separately first. Neither reviewer sees the other's scores until both sheets are complete.
- Discuss only the splits of two points or more. A 3 against a 2 changes little; a 3 against a 1 means the topic is being read differently.
- Argue from the topic description, not taste. Each reviewer points to the line in the description or narrative that supports their score.
- Call a third reviewer when the text is silent. The third reviewer decides, and the decision becomes a new line in the narrative.
- Rescore the affected articles. Apply the new line to every article already scored for that topic, not only the one in dispute.
What evidence should be saved for each match?
Articles change. Headlines are rewritten, pages are updated and archives are reorganized, so a match that cannot be shown later is a match the client has to take on trust. Save a small, consistent record for every article that helped qualify a site.
| Field | What to record | Why it matters |
|---|---|---|
| Address and date | The article URL, its publication date and the date it was read | Shows how current the match was at review time |
| Archived copy | A saved snapshot of the page | Proves what the page said if it later changes |
| Headline and summary | The headline and a one line summary in plain words | Lets a client check the match without opening the page |
| Matching passage | The section or paragraph that carries the topic | Shows the score rests on main content |
| Score and topic | The score from 0 to 3 and the topic ID | Makes sites comparable and reporting possible |
| Reviewer | The name of each reviewer and any tie break note | Makes the decision auditable |
For the archived copy, the Internet Archive's Save Page Now (opens in a new tab) gives a permanent URL for a single page, though its help notes that some sites prohibit crawling and a few break because of their security settings. Keep a screenshot as a fallback for those.
What does a relevance review look like on a real site list?
Fictional example
Home espresso equipment in Upstate New York
A fictional retailer of home espresso machines and grinders, with showrooms in Buffalo, Rochester and Syracuse, wants display ads beside articles that help people make espresso at home. The topic narrative counts brewing guides, equipment explainers and home barista technique, and does not count cafe reviews or coffee industry news.
On a regional food magazine, the reviewers read ten recent articles from the drinks section. Two are home brewing guides scored 3, one is a cafe opening story with a paragraph on the owner's home setup scored 1, and the rest are restaurant coverage scored 0. The site does not qualify, and the two strong articles are noted for a possible page level buy.
On a small home coffee blog, six of seven articles score 3. One reviewer scores a grinder roundup 1 because every paragraph repeats the same product phrase; the other scores it 3. The strike out test leaves almost nothing, the third reviewer scores it 0, and the narrative gains a line on roundups without explanation. The blog still qualifies on its other articles.
How does PlacementGraph show the evidence behind a match?
Frequently asked questions
Can a reviewer judge an article from its headline alone?
No. Headlines are written to attract clicks and can promise a broader or narrower subject than the article delivers. The headline is the first test in the scoring routine, not the verdict. A reviewer should read at least the opening and skim the body to confirm the topic runs through the main content.
Should sponsored or affiliate articles count as true matches?
Decide this with the client and write it into the topic narrative. A sponsored post can be squarely on topic, yet it may promote a rival product or read as advertising itself. A review can score such articles normally and flag them, so the client can see how much of a site's match rests on paid content.
Do older articles count toward a site's score?
Count them only as supporting evidence. A site's recent output predicts what ads will run beside, while its archive shows what it once covered. If the qualifying articles are all old, note it on the record and treat the site as a candidate to watch rather than a match.
How long should relevance evidence be kept?
Keep it at least for the life of the campaign and any period in which the client may ask how the list was built. Archived copies and the scoring sheet are small, and they are the only proof of what a page said when it was approved.
Sources
- Search Quality Rater Guidelines (General Guidelines, September 11, 2025) (opens in a new tab)Google
- English relevance judgements file list (opens in a new tab)Text REtrieval Conference (TREC), NIST
- Spam policies for Google web search (opens in a new tab)Google Search Central
- Interrater reliability: the kappa statistic (opens in a new tab)Mary L. McHugh, Biochemia Medica, 2012
- Save pages in the Wayback Machine (opens in a new tab)Internet Archive Help Center
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.