What makes an AI site recommendation explainable?
Part of Fair publisher planning
The short answer
An AI site recommendation is explainable when it shows, in a planner's words, the evidence that truly drove the choice: the supporting pages, the signals checked with their dates, the site's weakest point and where the system was unsure. If the evidence cannot be opened and checked, it is a label, not an explanation.
Key takeaways
- NIST's four principles of explainable AI, explanation, meaningful, explanation accuracy and knowledge limits, translate directly into what a site recommendation should show.
- Every recommendation should carry its supporting pages, the signals checked with dates, its weakest point and any doubt the system had.
- An explanation is real only if it can be checked: the pages open, the public records match and the reasons change when the evidence changes.
- Explanations are what let a planner see when a site was ranked for its size instead of its fit, which makes them a fairness tool.
- Clients need a shorter view than planners, but the same evidence, so every client line can be traced back to the record.
What makes an AI site recommendation explainable?
A public standard to start from is the National Institute of Standards and Technology's Four Principles of Explainable Artificial Intelligence (opens in a new tab), NISTIR 8312, published in September 2021. It defines Explanation as a system that "delivers or contains accompanying evidence or reason(s) for outputs and/or processes", Meaningful as explanations "understandable to the intended consumer(s)", Explanation Accuracy as an explanation that "correctly reflects the reason for generating the output", and Knowledge Limits as a system that "only operates under conditions for which it was designed and when it reaches sufficient confidence in its output."
| NIST principle | What it means for a site recommendation | A recommendation that fails it |
|---|---|---|
| Explanation | Each site comes with the reasons it was chosen | A ranked list with scores and nothing else |
| Meaningful | Reasons are in a planner's words: topics, pages, sellers, audience | Reasons given as feature weights or model jargon |
| Explanation accuracy | The stated reasons are the ones that actually drove the ranking | A generic sentence written after the fact, the same for every site |
| Knowledge limits | The system says when evidence was missing or too close to call | Every site presented with equal confidence |
The four principles work together. A recommendation can be clear and still inaccurate, or accurate and still unreadable. For a buyer defending a plan, the key one is explanation accuracy, because a convincing reason that did not drive the result is worse than no reason at all: it gives false confidence to everyone who reads it.
Which evidence should each recommendation show?
Enough for a planner to agree or disagree without rerunning the search. For a website, that means evidence about content, quality, selling path and audience, each with its source and date, plus the parts the system is least sure about. The list below is the minimum record for one site.
The evidence record for one recommended site
- The supporting pages: links to the specific articles that match the brief, not only the homepage.
- The match in words: which brief topic each supporting page fits, and how closely.
- The signals checked: page quality, ad load, language, suitability categories, each with a result.
- The selling path: what the site's ads.txt (opens in a new tab) file lists, and whether the exchange's sellers.json entry agrees.
- The audience evidence: the popularity band in the brief's country and its month, or a clear statement that none exists.
- The weakest point: the one fact most likely to make a planner hesitate.
- The doubt: any check that could not run or came out too close to call, with the reason.
- The version: the date of the evidence and the model version that produced the call.
Definition
- Weakest point
- The single piece of evidence that most counts against a recommended site, stated beside its reasons. It turns an explanation from a sales pitch into a balanced record a planner can weigh.
How can a planner test whether an explanation is real?
Treat the explanation as a claim and check it. Researchers draw the same line. In their ACL 2020 paper Towards Faithfully Interpretable NLP Systems (opens in a new tab), Alon Jacovi and Yoav Goldberg argue that "the current binary definition for faithfulness sets a potentially unrealistic bar" and call for "a more graded one." For a planner, graded means asking how much of an explanation holds up, site by site, through tests anyone on the team can run.
Five tests for a site recommendation's explanation
- Open the supporting pages. Click every cited article. Each should exist, be on the recommended site and match the topic the explanation claims.
- Check the public records. Compare the stated selling path with the live ads.txt and sellers.json files, and the stated popularity band with the CrUX rank definition (opens in a new tab) and its month.
- Look for boilerplate. Read ten explanations side by side. If several say the same thing in the same words, they may have been written to sound right rather than drawn from each site's evidence.
- Change the brief slightly. Remove one topic and rerun. Sites recommended for that topic should drop or change their reasons; if nothing moves, the explanation was not tied to the topic.
- Ask for the weakest point. A real explanation can name what counts against the site. A system that finds nothing negative about any result is not reporting its evidence.
Why do explanations matter for fair planning?
Because unfair reasons hide in unexplained rankings. When a list shows only scores, a planner cannot tell whether a small local publisher sits low because its articles fit the brief less well or because it has no popularity band. An explanation that names the reason makes the difference visible, and a reason tied to size rather than fit can be challenged before the plan is approved.
| Reason shown in the explanation | Fair basis for ranking? | What the planner does |
|---|---|---|
| Fewer of the sampled articles match the brief | Yes: it measures fit | Accept, or add the site to a test group |
| Heavy or refreshing ad load on sampled pages | Yes: it measures quality | Accept and record the reason |
| No popularity band in the brief's country | No: missing data is not a fit measure | Show reach as information and rank on fit |
| Domain ending outside the target country | No, on its own: audience matters more than address | Check audience evidence before demoting |
| Sold only through resellers | Not on its own: reselling is a selling path, not a quality measure | Record the path and let the trading desk decide |
Explanations also give publishers something to answer. A publisher told that its site ranked low for a stated reason can correct a wrong fact. A publisher told nothing cannot. The AI bias guide covers how to measure skew across a whole plan; explanations are how that skew is traced back to single decisions.
What should a client see from an automated recommendation?
Less detail than the planner, never different facts. NIST notes that "one-size-fits-all explanations do not exist" and that "different users will require different types of explanations." A client needs to know why each site is there and what to watch, in a sentence or two, with the full record a click away.
| Part of the explanation | Planner view | Client view |
|---|---|---|
| Why it fits | Each supporting page linked, with the brief topic it matches | One sentence naming the topics and an example article |
| Quality and suitability | Each check and its result | Passed the agreed checks, or the one exception |
| Selling path | Seller IDs, direct or reseller, per exchange | Sold on the exchanges in the deal |
| Audience | Popularity band, country and month | Popularity band, or no public data |
| Weakest point and doubt | Full text and any check that did not run | The weakest point in plain words |
How does PlacementGraph explain each site it recommends?
What does checking explanations change on a real plan?
Fictional example
A fictional electric bicycle retailer in Quebec
A fictional electric bicycle retailer with shops in Montreal and Quebec City briefs a display plan beside cycling commute, city transport and bike maintenance content, in French and English. An automated tool returns forty sites, each with a short explanation.
The planner runs the five tests on the top twenty. Not all of them hold up. Three explanations cite supporting pages that are on a different site from the one recommended. Six read almost word for word alike. When the planner removes bike maintenance from the brief, two sites recommended for maintenance keep the same reasons.
The planner removes the three mismatched sites, asks for fresh evidence on the six alike explanations and moves the two unchanged sites to a review group. The client view of the final list carries one sentence per site, its weakest point and a link to the full record.
Frequently asked questions
Is a confidence score an explanation?
No. A score says how sure a system is, not why it chose the site. It is useful beside an explanation, particularly when it shows a call was close, but a list of scores alone leaves a planner unable to check or challenge anything. Ask for the reasons and evidence behind each score.
Can an explanation be written after the recommendation is made?
It can be, and that is the main risk. A reason composed after the ranking may describe the site well without describing why it ranked. Prefer explanations drawn from the same evidence record the ranking used, and test them by changing the brief to see whether the reasons move with the results.
What if a supporting page has changed since the recommendation?
Then the explanation is out of date, not false. Pages are edited, moved and removed. A good record dates its evidence, so a planner can see when it was gathered. If a cited page no longer matches, ask for the site to be checked again before the list is approved or exported.
Do explanations reveal anything about a site's readers?
They should not. A site recommendation is a decision about content, quality and selling path, so its explanation describes pages and publishers. Audience evidence should be aggregated, such as a country popularity band. Any explanation that infers traits of individual readers goes beyond what site planning needs.
Sources
- Four Principles of Explainable Artificial Intelligence (NISTIR 8312) (opens in a new tab)National Institute of Standards and Technology, September 2021
- Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (opens in a new tab)Alon Jacovi and Yoav Goldberg, ACL 2020
- ads.txt (Authorized Digital Sellers) (opens in a new tab)IAB Tech Lab
- CrUX metrics (opens in a new tab)Google Chrome for Developers
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.