When should a planner override an AI site recommendation, and how is the call recorded?
Part of Fair publisher planning
The short answer
A planner should override an automated site recommendation when checkable evidence contradicts it, when the brief falls outside what the system was built for, or when a client rule was never in the brief. Each override needs a written reason, the evidence, the person and the date, so that a reviewer can later test whether overrides themselves skew the plan.
Key takeaways
- An override is justified by evidence a second person can open, not by taste, brand recognition or a hunch about traffic.
- Overrides run in both directions: removing a recommended site and restoring a site the system ranked low or left out.
- NIST's AI RMF Playbook suggests measuring how often people override a system and feeding the results back into improvement.
- Override records are bias data: a pattern of removals that falls on one group of publishers points to human bias, and a pattern of restorations points to system bias.
- A named reviewer outside the day to day planning team should read the override log on a fixed schedule.
When should a planner override an automated site recommendation?
Definition
- Override
- A documented human decision that reverses an automated recommendation for one site: removing a site the system put on the list, or adding back a site the system ranked low, flagged or left out.
The planner's job is to find the results where the system's evidence is wrong, incomplete or aimed at the wrong question. Five situations justify an override. Anything else is a preference, which belongs in the next brief.
| Trigger | What it looks like | Test before acting |
|---|---|---|
| Evidence contradicts the reason | The cited articles do not match the brief, or a cited fact is wrong | Open every supporting page and the public record behind each claim |
| Out of scope for the system | A language, region or publisher type the system was not built or tested for | Read the tool's stated intended use and known limits |
| Client rule missing from the brief | A competitor's owned site, a past dispute, a legal restriction on the category | Confirm the rule in writing with the client, then add it to the brief |
| Fact changed since the check | A site changed owner, layout, ad load or editorial focus after it was scored | Compare the check date with the date of the change |
| Missing data treated as a fail | A site ranked low or left out because a dataset had no record of it | Check whether the reason is an absence of data or a negative finding |
The second trigger draws on a long standing idea in machine learning documentation. The Model Cards for Model Reporting (opens in a new tab) paper by Mitchell and colleagues, presented at FAT* 2019, asks each model card to state the model's primary intended uses and out of scope uses, and to say "whether the model was developed with general or specific tasks in mind". When a brief asks for something the system was not designed to judge, such as a regional language it does not detect, the planner's reading carries more weight than the score.
What evidence justifies an override?
Evidence that another person can open, read and reach the same conclusion from. "It feels low quality" gives a reviewer nothing to check; three linked articles that are reposted press releases do. Restoring a site needs the same: the articles that fit the brief and the record that clears the concern.
Labels from automated systems are not final evidence either. A January 2026 study by Smith and colleagues (opens in a new tab) compared the ratings three brand safety services gave to 4,352 news articles from 51 domains and found that "brand safety services often produce conflicting classifications." When classifiers disagree with one another, the article itself is the tiebreaker, and an override that cites the article is stronger than one that cites a second label.
| Grade | Examples | Enough to override? |
|---|---|---|
| Strong | Linked articles read in full; a dated ads.txt or sellers.json record; written client instruction | Yes, on its own |
| Supporting | A layout check on phone and desktop; a site's own about page or masthead; a correction policy | Yes, when two or more agree |
| Weak | A second tool's label; a headline seen without the article; one unusual page | No, only a reason to look closer |
| Not evidence | Brand recognition, assumed traffic, a colleague's opinion, the site's domain ending | No |
How should overrides be recorded?
One record per site per decision, written when the decision is made, beside the list itself. A spreadsheet tab works; an email thread does not. These fields are the minimum a reviewer needs.
Fields in one override record
- The site and the list version the override applies to.
- Direction: removed from the recommendation, or restored against it.
- The system's original result and the reason it gave, copied rather than paraphrased.
- The trigger, chosen from a fixed list of reason codes, plus one sentence in plain words.
- Links to the evidence, each with the date it was opened.
- The planner who made the call and the person who approved it.
- A date to check again, for overrides based on facts that can change.
- The publisher slices the site belongs to: size band, ownership, language, base country and sensitive topic coverage.
Use fixed reason codes, because free text cannot be counted and counting is the point. The slice fields turn a log of individual calls into a bias test: they let a reviewer ask which publishers the overrides fall on.
How do overrides help find bias?
Overrides are where human judgment and the system disagree, so they show the bias of both. NIST's AI RMF Playbook (opens in a new tab) suggests organizations "Maintain statistics about downstream actions by end users and operators such as system overrides" (MEASURE 2.8) and "Measure frequency of AI systems' override decisions, evaluate and document results, and feed insights back into continual improvement processes" (MEASURE 4.2).
The human side needs the same scrutiny as the machine. The NIST AI Risk Management Framework (opens in a new tab) (AI RMF 1.0, January 2023) warns that under certain conditions "the AI part of the human-AI interaction can amplify human biases, leading to more biased decisions than the AI or human alone." A review step has to show, in its records, that it corrected errors rather than reintroducing old shortcuts.
| Pattern in the override log | What it suggests | Next step |
|---|---|---|
| Removals fall mostly on small, independent or local language sites | Human bias toward familiar brands | Re-read the removal evidence; restore sites removed without strong evidence |
| Restorations fall mostly on one slice, such as French language sites | The system underserves that slice | Report the pattern to the tool's owner with the examples |
| One planner overrides far more than colleagues on similar briefs | A personal rule that is not in any brief | Discuss the rule; write it into the brief template or drop it |
| Overrides cluster on one trigger, such as stale facts | A data freshness gap | Shorten the recheck interval for that kind of fact |
| Almost no overrides across several plans | Either a good system or a rubber stamp | Sample approved lists and check the evidence independently |
The AI bias guide tests the recommender itself and the explainable AI guide tests its reasons; the override log records what people did with them.
Who reviews override patterns?
Someone who did not make the overrides. NIST's Playbook, discussing GOVERN 1.5, describes incident response and "appeal and override" as processes that "enable real-time flagging of potential incidents, and human adjudication of system outcomes," and calls for roles and review frequency to be clearly defined. In a planning team that can mean three roles: the planner who records each override, an approver who signs off on the list, and a periodic reviewer who reads the log across plans.
Internal audit research supports the separation. In Closing the AI Accountability Gap (opens in a new tab) (ACM FAT* 2020), Raji and colleagues describe a framework in which "Each stage of the audit yields a set of documents that together form an overall audit report." An override log, reviewed on a schedule, is one of those documents for a planning team.
A quarterly override review in five steps
- Pull every override since the last review. Include both directions and every planner, not a sample chosen by the team.
- Count by slice and by reason code. Compare the share of removals and restorations for each publisher slice with that slice's share of the recommended lists.
- Read a sample of the evidence. Open the links on ten overrides at random and confirm the evidence supports the stated trigger.
- Sort findings by owner. Human patterns go to the team lead and the brief template; system patterns go to whoever maintains the tool.
- Record what changed. Note the rule, brief field or tool report that came out of the review, so the next review can check whether it worked.
How does PlacementGraph support a planner who overrides a recommendation?
What does an override review look like on a real plan?
Fictional example
A fictional auto repair and tire chain in Colorado
A fictional chain of auto repair and tire shops in Denver, Colorado Springs and Fort Collins briefs an English display plan beside car maintenance, winter driving and local commuting content. The automated recommendation returns a ranked list with reasons, and two planners review it over a week.
Their log shows eleven overrides. Three removals have strong evidence: one site reposts manufacturer press releases, and two changed owners after being checked. Two are restorations of small Colorado driving blogs that had been ranked low for missing popularity data, with linked articles showing a close fit.
The other six remove small local sites, each logged as "not a known outlet" with no evidence. Read by slice, all six unsupported removals fell on independent publishers. Four are restored after the reviewer reads their articles, and that reason code is dropped.
Frequently asked questions
Should a planner override a site the client simply dislikes?
Yes, but record it as a client instruction, not as a quality finding. A client preference is a legitimate reason to remove a site. Its own reason code keeps it apart from evidence based overrides, so a bias review does not mistake one client's taste for a flaw.
Can an override be reversed later?
It should be reversible by design. Give overrides based on facts that change, such as ownership or ad load, a date to check again, and link each override to the list version it changed. When the reason no longer holds, a new record reverses it, and the history shows both decisions rather than overwriting the first.
How many overrides on one list are too many?
There is no published threshold, so judge the evidence rather than the count. A large number of well supported overrides suggests the brief or the system needs fixing. A sudden rise for one planner, one client or one publisher slice is the signal worth investigating.
Should the client see the override log?
The client should see the outcome and, on request, the reasons. A short summary of sites removed or restored, each with a one sentence reason, lets the client challenge a call. The full log with evidence links and planner names is an internal working record for the team's own review.
Sources
- AI RMF Playbook (opens in a new tab)National Institute of Standards and Technology (NIST)
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (opens in a new tab)National Institute of Standards and Technology (NIST), January 2023
- Model Cards for Model Reporting (opens in a new tab)Mitchell et al., FAT* 2019 (arXiv, revised January 14, 2019)
- Inconsistencies in Classification of Online News Articles: A Call for Common Standards in Brand Safety Services (opens in a new tab)Smith et al., arXiv, January 3, 2026
- Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing (opens in a new tab)Raji et al., ACM FAT* 2020 (arXiv, January 3, 2020)
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.