How many pages should you read before judging a whole website for ads?
Part of Programmatic media planning
The short answer
No standard sets a number. A workable protocol starts with a small sample drawn on purpose: recent pages, pages from every section where ads would run, and pages the site promotes, then reads more when sections differ, the site publishes fast, or reviewers disagree. Record which pages were read, how they were picked and what was missed.
Key takeaways
- A sample can show that a problem exists on a site; it cannot prove that a problem is absent, so the record should say what was not read.
- Choose pages from each section where ads would run, because one section can differ sharply from the rest of the site.
- Hand picked recent and popular pages are a convenience sample, useful for screening but not a measure of the whole site.
- Write down the page list, the selection method, the date and the gaps, so another reviewer can repeat or challenge the call.
Why sample pages before judging a whole website?
Definition
- Review sample
- The set of pages a reviewer reads to decide whether a website as a whole is fit for a campaign, chosen by a stated method and recorded with the date.
Websites are too large to read in full. The sitemaps protocol (opens in a new tab) allows each sitemap file up to 50,000 URLs, and a sitemap index can list up to 50,000 sitemaps, so even a mid sized publisher can expose more pages than any team could open. A review decision about a site is therefore a judgment from a sample, whether or not anyone calls it that. Naming it a sample makes the method visible and lets someone else check it.
The question a site review answers is broader than relevance. It asks whether the site as a whole fits the brand: what it publishes in the sections where ads would appear, how pages are laid out, whether any section touches the brand's avoid topics, and whether content looks written for readers or assembled for ad space. Checking whether pages match the brief's topics is a related but separate step, covered on the contextual relevance review page.
How many pages should be sampled before judging a whole site?
Survey statisticians tie sample size to purpose, not to a fixed rule. The US Census Bureau's sample design standard (opens in a new tab) says the size and design of a sample must reflect the level of detail needed and the precision required of key estimates, and NIST's handbook on selecting sample sizes (opens in a new tab) adds that when a population is stratified, sample size has to be worked out for each stratum. For site review, that means the count should grow with the stakes of the decision and with the number of sections the site has.
| Review purpose | Starting sample | Read more when |
|---|---|---|
| First screen of a candidate site | Six pages: the homepage, three recent articles and two from the section ads would run in | Any page touches an avoid topic or looks templated |
| Approval for a site list | Two to three pages from each section where ads would run, plus three recent pages site wide | Sections differ from each other, or the site publishes several times a day |
| A site in a deal or a large share of budget | Five pages per section where ads would run, read by two reviewers | The two reviewers disagree on any page |
| Re-review after a flag | Every page in the flagged section from the last two weeks, plus three from other sections | The issue appears outside the flagged section |
How should sample pages be chosen?
Each way of picking pages finds some problems and misses others, so combine at least two. Recency matters because ads run beside what a site publishes now. Sitemaps can help find recent pages: the protocol defines lastmod as the date of last modification of the page, and Google's sitemap guidance (opens in a new tab) says it uses lastmod only if it is consistently and verifiably accurate, and ignores the priority and changefreq values. A sitemap date is a lead to check, not proof of when a page changed.
| Method | How | Finds | Misses |
|---|---|---|---|
| Recent | The newest pages from the sitemap, feed or section front | What the site publishes now | Older pages still getting traffic |
| By section | A fixed number from each section where ads would run | Differences between sections | Problems in sections left out of the plan |
| Promoted | Pages featured on the homepage or in most read modules | What the site puts in front of readers | Long tail pages that ads also reach |
| Random | Pages drawn at random from the sitemap URL list | A fairer picture across the archive | Recent changes, if the archive is large |
Build one sample in five steps
- List the sections. Read the site's navigation and sitemap, and mark the sections where the plan's ads would run.
- Pull recent pages from each marked section. Use section fronts or sitemap dates, and open each page to confirm its date.
- Add the promoted pages. Take what the homepage features on the review day.
- Add a few random pages. Draw them from the full sitemap URL list to catch what the other methods miss.
- Write the list down before reading. Fixing the list first stops a reviewer from swapping in pages that look better.
When does a small sample mislead?
A sample picked for ease is a convenience sample. The AAPOR Task Force on Non-Probability Sampling (opens in a new tab) (June 2013) defines convenience sampling as a form of non-probability sampling in which the ease of locating participants is the primary consideration, and warns that with such samples the population exposed to selection can be a small and unrepresentative part of the target population. Homepage stories and recent articles are exactly that: easy to find, and not the whole site.
Signs the sample is too small or too narrow
- The site has sections the sample never touched, and ads could run there.
- The site publishes several times a day, so the sample covers only a few hours of output.
- Pages in the sample differ sharply in quality, tone or layout from one section to another.
- Several sampled pages could not be opened, were paywalled or redirected elsewhere.
- Two reviewers reading the same pages reached different conclusions.
- The site's owner, layout or topic mix changed recently.
Any one of these is a reason to read more before deciding, not a reason to reject the site. The extra reading should target the gap: the missing section, a longer date range, or a second reviewer.
How do you sample a site with many sections?
Treat each section as its own stratum. A general lifestyle site may run a careful home and garden section beside a celebrity gossip section that breaks a brand's avoid topics every week. A sample drawn across the whole site would lean toward whichever section publishes the most pages. Drawing a fixed number of pages from each relevant section, as stratified sampling does, keeps a quiet section from being drowned out and a noisy one from being missed.
Deciding per section first also gives the buyer an option. When one section fails and the rest pass, the site can stay on the plan with that section excluded, if the buying platform supports exclusions at that path level.
How should the sample be recorded?
Record enough for someone else to repeat the review. The Census Bureau's standards ask for documentation that can replicate and evaluate (opens in a new tab) coding operations and sample design, and its nonsampling error standard (opens in a new tab) treats coverage and missing data as quality measures in their own right. Applied to site review, the record should show what was read, how it was picked and what could not be read. Sites also change: Display & Video 360's keyword help (opens in a new tab) notes that the platform frequently crawls pages to keep up with content changes, which is a reminder that a human review is true only for the date it was done.
| Field | What to write |
|---|---|
| Site and review date | The domain and the day the pages were read |
| Sections in scope | The sections where the plan's ads would run |
| Selection method | Recent, by section, promoted, random, and how many of each |
| Page list | Every URL read, with its publication date |
| Gaps | Pages that would not open, paywalls, sections left out and why |
| Findings | Per section: fit, unfit or read more, with the page that decided it |
| Reviewers | Who read the pages, and whether a second reviewer agreed |
How does PlacementGraph sample a website before ranking it?
What does a sampled site review look like in practice?
Fictional example
A home sauna maker reviewing wellness sites for Oklahoma
An unnamed maker of home saunas and cold plunge tubs plans a winter display campaign for buyers in Tulsa and Oklahoma City, on home, wellness and regional lifestyle websites. Its brief avoids pages that make medical cure claims for heat or cold therapy.
One candidate is a regional lifestyle site with home, food, health and events sections. The planner marks home and health as in scope, writes down a list of three recent pages from each, two pages promoted on the homepage and two random pages from the sitemap, then reads them. The home pages are renovation guides. Two of the three health pages are reported features, but one presents a cold plunge routine as a cure for a chronic illness.
That one page triggers more reading. The planner pulls six more health pages from the last month, finds two more of the same kind, and a second reviewer agrees. The record shows the site approved for the home section only, the health section excluded, the twelve URLs read with their dates, and one events page that would not open.
Frequently asked questions
Does the homepage count as a sample page?
Read it, but do not count it as one of the pages that decide the call. A homepage shows what the site promotes on that day and how ads are laid out, yet a campaign's ads also run on article pages. Record the homepage separately and base the decision on pages from the sections in scope.
What if a website has no sitemap?
Use section fronts, the site's feed if it has one, and the archive or tag pages linked from the navigation. Note in the record that no sitemap was found, because random selection becomes harder and the sample leans more on recent and promoted pages.
Should paywalled pages be part of the sample?
Include the paywalled pages you can open, and record the ones you cannot as a gap. A paywall does not take a section out of scope on its own. If the pages you could open in a section are too few to judge it, say the section was not assessed.
Can an automated crawl replace a human sample?
It can cover far more pages and measure things a reader cannot, such as ad refresh timing. A person still decides what the brand's avoid topics mean on close calls. The two work together best when the crawl's page list and the human reviewer's page list are both kept in the record.
Sources
- Sitemaps XML format (opens in a new tab)sitemaps.org
- Build and submit a sitemap (opens in a new tab)Google Search Central
- Statistical Quality Standard A3: Developing and Implementing a Sample Design (opens in a new tab)US Census Bureau
- Statistical Quality Standard C3: Coding Data (opens in a new tab)US Census Bureau
- Statistical Quality Standard D3: Producing Measures and Indicators of Nonsampling Error (opens in a new tab)US Census Bureau
- Selecting sample sizes, NIST/SEMATECH e-Handbook of Statistical Methods (opens in a new tab)National Institute of Standards and Technology
- Report of the AAPOR Task Force on Non-Probability Sampling (June 2013) (opens in a new tab)American Association for Public Opinion Research
- Keyword targeting (opens in a new tab)Google, Display & Video 360 Help
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.