How can AI site recommendations skew a plan toward large publishers?
Part of Fair publisher planning
The short answer
AI site recommendations skew toward large publishers when popularity enters the score, when the index holds more data on big sites, and when the system learns from past delivery. NIST calls these popularity, representation and feedback loop biases. Audit a plan by testing it with known good small sites, reading the reasons behind each pick and recording overrides.
Key takeaways
- NIST groups AI bias into systemic, computational and statistical, and human-cognitive types, and says each can occur without any intent to discriminate.
- Popularity bias, which NIST defines as more popular items being more exposed, is one route by which automated lists can favor large publishers.
- A holdout set of vetted small publishers is the simplest test of whether a recommender can find them at all.
- NIST names ranking bias and automation complacency among human biases, so review only corrects bias when it is structured and its overrides are recorded.
What is AI bias in media planning?
Definition
- AI bias in media planning
- A systematic tendency of an automated recommendation system to rank or include some kinds of publishers more often than their fit to the brief justifies, for example favoring large, well known sites over small, independent or local language ones.
The NIST AI Risk Management Framework (opens in a new tab), released January 26, 2023, names three major categories of AI bias: systemic, computational and statistical, and human-cognitive. It states that "each of these can occur in the absence of prejudice, partiality, or discriminatory intent." That point matters for media planning. Nobody designs a site recommender to ignore independent publishers; the skew comes from the data it learns from and the signals it is allowed to use.
The framework lists "fair with harmful bias managed" among the characteristics of trustworthy AI, alongside valid and reliable, accountable and transparent, and explainable and interpretable. For a planning tool, those words translate into questions a buyer can ask: is the list valid for the brief, can each pick be explained, and has the skew been measured?
How do automated recommendations come to favor large publishers?
NIST's special publication 1270 (opens in a new tab) of March 2022 defines the specific biases in its glossary. Several of them map directly onto how site recommendations are built. The table quotes NIST's definitions and shows where each one appears in a planning tool.
| Bias (NIST SP 1270) | NIST definition | How it shows up in a site list | What to check |
|---|---|---|---|
| Popularity bias | Items that are more popular are more exposed and less popular items are under-represented | Traffic or reach adds to the score, so big sites rise whatever their fit | Whether popularity enters the ranking at all |
| Representation bias | Non-random sampling of subgroups, so trends for one population do not generalize to another | The index can hold more pages from big sites than from small ones | How many pages are read per site, by site size |
| Feedback loop bias | An algorithm learns from user behavior and feeds that behavior back into the model | Sites that served past campaigns get more data and rank higher next time | Whether delivery or clicks feed back into ranking |
| Ranking bias | Top-ranked results are assumed to be the most relevant and important | Planners may approve the first page and stop reading | How far down the list reviewers go |
| Automation complacency | Humans over-rely on automated systems | Reasons go unread because the tool is trusted | Whether reviewers can say why a site is in |
Data gaps add a systemic layer. Public popularity data such as the Chrome UX Report (opens in a new tab) does not count Chrome on iOS, Android WebView or other Chromium browsers, and it leaves out origins below an undisclosed popularity threshold. A model that treats that data as complete will see small sites as absent rather than unmeasured.
Where in a planning pipeline does the skew enter?
How do you test an AI recommended plan for publisher bias?
Five tests to run on any automated site recommendation
- Run a holdout test. Before the search, write down five to ten small publishers you have already vetted for this kind of brief. After the search, check where each one ranks and read the reason given for any that are missing.
- Read the reasons. Open the explanation for ten sites at the top and ten near the cutoff. Reasons that cite sampled articles point to fit; reasons that cite traffic, brand recognition or nothing at all point to popularity.
- Rephrase the brief. Run the same brief in different words. A list driven by fit changes a little; a list that always returns the same large sites is leaning on popularity.
- Check the missing data. Count how many small sites were dropped or ranked low for lack of audience data, and ask whether missing data is treated as zero.
- Log every override. Record each site a person adds or removes and why. A pattern of reviewers adding back small publishers is evidence that the scoring needs a fix.
These tests look at the recommender itself. The fair publisher planning hub describes the separate slice audit of the finished list, which measures how each group of publishers survived from candidate pool to approval.
Which methods reduce popularity skew, and what does each cost?
Research on recommender systems has studied this problem for years. A 2019 paper by Abdollahpouri, Burke and Mobasher (opens in a new tab) notes that "many recommender systems suffer from popularity bias," with niche items recommended rarely or not at all, and proposes a re-ranking step that raises the share of less popular items while keeping acceptable accuracy. No method is free, so each has real strengths and weaknesses.
| Method | Strengths | Weaknesses |
|---|---|---|
| Leave popularity out of the score and show it as information | Ranking reflects fit; reach stays visible for the buyer | Buyers must judge scale themselves; can surface sites too small for the budget |
| Re-rank the output to include more long tail sites | Works on top of an existing system as a post-processing step | Needs a chosen target share; can lift weaker sites if fit is not also checked |
| Structured human review with recorded overrides | Catches errors no model sees; builds an audit trail | Slow at scale; reviewers bring their own biases, as NIST notes |
What should a buyer ask before relying on an AI planning tool?
Questions to ask about any automated site recommendation
- Does traffic, reach or brand recognition enter the ranking score, and if so how much?
- How many pages does the system read per site, and is that the same for small and large sites?
- Is missing audience data shown as missing, or counted as zero?
- Is quality measured separately for small, independent and local language publishers?
- Do campaign delivery or clicks feed back into future rankings?
- Does every recommendation come with a reason tied to the site's own pages?
- Is the model version recorded with each plan so results can be reproduced?
- Can a person override any pick, and is the override logged?
How does PlacementGraph limit bias in its own recommendations?
What does a bias test change on a real plan?
Fictional example
A fictional continuing education program in Atlantic Canada
A fictional university continuing education program in Nova Scotia and Newfoundland and Labrador asks its agency for an English display plan beside career, trades and adult learning content. The agency's automated planning tool returns a list led by national news sites and large education portals.
Before accepting it, the planner runs a holdout test with eight regional publishers the agency already trusts, including a trades magazine and two community news sites. Only one appears, near the bottom. The reasons on the top results cite audience size, and the missing sites have no audience figure at all.
The team asks the tool's provider to remove reach from the score and show it beside each site instead, then reruns the brief in two phrasings. In the rerun the reasons on the holdout sites cite their pages, and each override the planners make is logged with the plan's model version.
Frequently asked questions
Is AI bias in media planning about audiences or about publishers?
In site planning it is mainly about publishers: which sites a system finds, scores and ranks. The checks on this page describe publishers and pages, never people. Bias in audience targeting is a separate question with its own legal and privacy rules, and contextual site planning does not need personal data to be audited.
Can human review correct a biased recommendation on its own?
Only partly. NIST's glossary defines ranking bias as the assumption that top-ranked results are the most relevant, and automation complacency as over-reliance on automated systems. Review works when reviewers read below the top results, check reasons against pages and record every change, so patterns in their overrides can fix the model.
Does generative AI in planning tools add new bias risks?
It can. NIST published a Generative Artificial Intelligence Profile, NIST AI 600-1, on July 26, 2024, to help organizations identify risks unique to generative AI. For planning, the practical test stays the same: every recommended site needs a reason tied to its own pages that a person can check.
What records should a team keep to show a plan was checked for bias?
Keep the brief as confirmed, the candidate pool, the model or tool version, the holdout sites and where they ranked, the reasons for the final picks, and every human override. Those records let a client or auditor see how the plan was made and let the team repeat the test on the next brief.
Sources
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (opens in a new tab)National Institute of Standards and Technology, January 26, 2023
- Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, NIST SP 1270 (opens in a new tab)National Institute of Standards and Technology, March 15, 2022
- AI Risk Management Framework (opens in a new tab)National Institute of Standards and Technology
- Managing Popularity Bias in Recommender Systems with Personalized Re-ranking (opens in a new tab)Himan Abdollahpouri, Robin Burke and Bamshad Mobasher, arXiv, 2019
- CrUX methodology (opens in a new tab)Google Chrome for Developers
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.