How to design a contextual advertising test that answers one question
Part of Contextual targeting
The short answer
A fair contextual advertising test changes one thing, the contextual plan, and holds everything else fixed: creative, budget share, bids, frequency, dates and geography. Units are assigned to test and control at random, ideally with a holdout that sees no ads. The result supports a conclusion only about that question, that market and that period.
Key takeaways
- Write the planning question and the single success metric before launch, or the test will be read to fit whatever happened.
- Random assignment is what makes a test an experiment; picking which regions or users get the plan turns it into an observation.
- Block the differences you can control, such as region size, and randomize the rest.
- A placement report proves where ads ran, not what they changed.
- A clean result supports a narrow statement about one plan, one market and one period, stated as an estimate with a range.
What question should a contextual test answer?
A contextual test can fail before launch when the question is vague. A testable question names one change and one outcome: does adding a reviewed list of home improvement publishers to the current display plan change the number of quote requests from the test regions over six weeks?
Definition
- Holdout
- A group of people or regions deliberately kept from seeing the ads being tested, so the outcome without the ads can be measured directly instead of guessed.
Separate operational questions from causal ones. Did the plan run on the approved sites, at what price and with how much reach are operational: a delivery report answers them. Did the plan cause more of the outcome is causal, and only a controlled comparison answers it. The Media Rating Council's Outcomes and Data Quality Standards (opens in a new tab) (September 2022) describe incrementality as the potential causal impact of marketing, measured by isolating results from other business factors.
How do you design a fair test of a contextual plan?
The MRC standard draws the line that matters. An experiment uses randomization to produce test and control groups, with whoever runs it controlling who gets treatment and deciding truly at random. Any approach that does not control treatment, or does not assign it at random, is in the standard's words not an experiment and would be considered an observational approach.
| Design | How units are assigned | Strengths | Limits |
|---|---|---|---|
| Geo experiment | Non-overlapping regions randomly assigned to test or control | Works without tracking individuals; outcomes such as store visits or quote requests can be counted by region | Needs enough comparable regions; regional news or weather can swamp a small effect |
| Platform split | The buying platform splits traffic or users between an original and a test campaign | Built into the tool; budget and traffic are shared on equal terms | Can rely on browser or account identifiers; the split rules are set by the platform |
| Before and after | The same market compared across two periods | Cheap and simple | Not an experiment; seasonality, prices and competitors change between periods |
Geo experiments suit contextual plans well because they need no personal data. Google researchers Jon Vaver and Jim Koehler described the method in Measuring Ad Effectiveness Using Geo Experiments (opens in a new tab) (2011): non-overlapping geographic regions are randomly assigned to a control or treatment condition, and each region receives its condition through geo targeted advertising. For a platform split, Google's custom experiments (opens in a new tab) page says the experiment shares the original campaign's traffic and budget, splits the budget evenly between control and treatment arms, and that Display campaigns always use cookie splits.
Seven steps to a test plan written before launch
- State the question in one sentence. Name the change, the outcome, the market and the period.
- Pick one primary metric. Choose an outcome the business already counts the same way in every region, such as quote requests or site visits from paid media.
- Choose the assignment unit. Regions for a geo experiment, users or browsers for a platform split.
- Pair or block the units. Group regions of similar size and history, then randomize within each pair.
- Set a holdout if the question is about lift. A control that sees the current plan answers whether the new plan is better; a control that sees nothing answers whether advertising works at all.
- Fix the run length and stopping rule. Decide the end date in advance and do not stop early because the numbers look good.
- Write down what each result would mean. One line each for higher, lower and no clear difference.
What should stay fixed between test cells?
Everything except the contextual plan. The NIST Engineering Statistics Handbook (opens in a new tab) calls the other influences nuisance factors, ones that may affect the measured result but are not of primary interest, and gives the rule: block what you can, randomize what you cannot. For a planner, blocking means matching cells on controllable factors.
Held constant across test and control cells
- Creative, ad formats and sizes, and landing pages are identical.
- Budget per unit of population, bid strategy and bid caps match.
- Frequency caps, dayparting and device settings match.
- Brand suitability settings and exclusion lists are the same in every cell, so only the contextual plan differs.
- Flight dates are the same, and no cell gets a promotion, price change or email push the others do not.
- Conversion definitions and tags are unchanged during the test.
- No one edits a cell mid flight; Google's custom experiments page notes that changes to the original campaign won't be reflected on the experiment.
Why is a placement report not proof of lift?
A placement report answers where ads ran. It cannot say what would have happened if they had not run, and that missing comparison is what lift means.
The report is also incomplete by design. Google's Where ads showed (opens in a new tab) help says placements are only shown if they meet minimum thresholds of impressions or interactions, and tells advertisers not to use the report as an exhaustive list of placements. Use it to confirm the plan ran as intended, which is a precondition for a valid test, never as the result.
| Evidence | What it shows | What it cannot show |
|---|---|---|
| Placement or domain report | Where impressions were served, above reporting thresholds | Whether those impressions changed any outcome |
| Platform attributed conversions | Conversions the platform credits to ads under its attribution rules | Conversions that would have happened anyway |
| Survey based brand lift | Differences in survey answers between exposed and control groups | Sales or actions outside the survey |
| Randomized test with holdout | An estimate of the change caused, with a range | Results in other markets, seasons or plans |
Google's Brand Lift (opens in a new tab) page shows the shape of a valid comparison: an exposed group of people who saw the ads, and a control group of people who were eligible to see them but did not because the ad was not shown. The lift is the difference in responses between the two.
How long should a contextual test run?
Long enough to cover full weekly cycles and the normal delay between seeing an ad and acting, and fixed before launch. There is no universal number. Google's custom experiments setup asks for a start date and a duration but does not recommend one, and its Brand Lift page says to meet the minimum spend within 10 days to give a study the best chance of completing. The business, not the platform, sets the real length.
| Factor | Pushes longer when | Allows shorter when |
|---|---|---|
| Purchase cycle | People research for weeks before acting | The outcome is a same day action, such as a site visit |
| Outcome volume | Few conversions per region per week | Hundreds of outcomes per cell per week |
| Weekly pattern | Weekends behave very differently from weekdays | Days look alike |
| Seasonality | The season turns during the test | The period is stable |
| Size of the change | The plans differ only slightly | The plans differ sharply in sites and spend |
What can a buyer honestly conclude from the result?
Only what the design supports. The MRC standard says that because deviations from perfect execution are almost unavoidable, results of randomized and non-experimental studies must be treated as estimates, with disclosure of sample sizes, biases and standard error. A responsible readout states the estimate, its range and the conditions it held under.
| Result | Supported statement | Overreach to avoid |
|---|---|---|
| Clear positive difference | In these regions and weeks, the plan was associated with an estimated increase in the metric, within this range | Contextual targeting increases sales |
| No clear difference | The test could not distinguish the plans at this size and length | The contextual plan does not work |
| Clear negative difference | The new plan underperformed the current one on this metric here | Context does not matter for this brand |
| Cells drifted on spend or reach | The test is inconclusive about context | Any statement about context |
What does a contextual test look like on a real brief?
Fictional example
Residential roofing quotes in Tennessee
A fictional residential roofing company serving Nashville, Knoxville, Chattanooga and Memphis wants to know whether a reviewed list of home improvement and homeowner advice publishers should replace its current broad display plan. The question: does the reviewed list change online quote requests over eight weeks of the spring season?
The planner splits the company's service area into 16 county groups, pairs them by population and past quote volume, and randomly assigns one of each pair to the reviewed list and the other to the current plan. Creative, landing page, bids, frequency caps and budget per household are identical. Two further county groups are held out of display entirely to show the baseline.
Weekly checks confirm both cells ran on their intended sites and spent evenly. At the end, the team reports the estimated difference with its range, names the regions and weeks, and states what it cannot say: nothing about other seasons, other markets or other creative. The decision recorded is only whether to keep the reviewed list for the next spring flight.
How can PlacementGraph keep the test cells clean?
Frequently asked questions
Is a before and after comparison good enough for a contextual test?
It is useful for spotting a problem, not for proving an effect. Prices, competitors and seasons change between the two periods, so any difference has several explanations. If a randomized split is impossible, run the comparison across matched regions at the same time and describe it as an observation.
Should a contextual test measure brand survey answers or sales?
Measure the outcome the question names. Survey based studies suit awareness or consideration questions and finish faster. Sales or leads suit performance questions but need more volume and time. Choosing the metric after seeing the results invalidates the test, whichever one is picked.
What should a team do if one test cell underspends?
Decide in advance how much drift is acceptable. Small gaps go in the readout; large gaps mean the test now compares budgets, not context. Fix the cause, such as too few sites for the bid, and restart rather than extending one cell.
Can one test cover websites and YouTube at the same time?
It can, but then the result describes the combined plan. If the question is about the website list alone, keep YouTube identical across cells or out of the test regions entirely. Mixing a channel change into a context test makes it impossible to say which change moved the outcome.
Sources
- MRC Outcomes and Data Quality Standards (September 2022) (opens in a new tab)Media Rating Council
- 5.3.3.2. Randomized block designs (opens in a new tab)NIST/SEMATECH e-Handbook of Statistical Methods
- Measuring Ad Effectiveness Using Geo Experiments (opens in a new tab)Google Research (Vaver and Koehler, 2011)
- Set up a custom experiment (opens in a new tab)Google Ads Help
- About Brand Lift (opens in a new tab)Google Ads Help
- About the Where ads showed report (opens in a new tab)Google Ads Help
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.