Algorithmic fairness definitions, applied to publishers
Part of Fair publisher planning
The short answer
Algorithmic fairness is a set of measurable tests for whether a model's decisions treat groups alike. In media planning the groups are publisher slices, such as small sites or Spanish language sites. The main tests compare selection rates (parity), error rates (equal opportunity, equalized odds) and what a score means in each slice (calibration).
Key takeaways
- Fairness measures were written for decisions about people; in planning they apply to decisions about publishers: keep, exclude or send for review.
- Slice based evaluation reports a model's errors for each group of publishers, because an overall score can hide a slice that is served badly.
- Parity compares how often each slice is kept, equal opportunity compares how often good sites are kept, and calibration asks whether a score means the same thing in each slice.
- Slices should describe publishers and pages, since reader traits would need personal information about people.
What does fairness mean for an advertising algorithm?
Definition
- Algorithmic fairness (publisher sense)
- A property of a model that includes, excludes or ranks publishers, checked with a stated measure that compares how the model treats different groups of publishers judged to be equally fit for the brief.
The vocabulary of fairness grew out of decisions about people. The 2016 NeurIPS paper by Hardt, Price and Srebro (opens in a new tab) names "advertising, credit, employment, education, and criminal justice" as domains where machine learning affects decisions. A planning model decides about a business instead: keep, exclude or send to a person. The fairness question becomes whether equally good publishers are kept at equal rates, whatever their size, language or ownership.
The measures below are built from four outcomes. Take candidate sites that trained reviewers have already judged against the brief, and compare the model's call with the reviewer's call.
| Outcome | What happened | Who bears the cost |
|---|---|---|
| True positive | A site the reviewer judged fit was kept by the model | Nobody; this is the goal |
| False negative | A site the reviewer judged fit was excluded | The publisher loses revenue; the buyer loses wanted reach |
| False positive | A site the reviewer judged unfit was kept | The buyer, through spend on a site it would not have chosen |
| True negative | A site the reviewer judged unfit was excluded | Nobody, if the reviewer was right |
What is slice based evaluation?
Definition
- Slice based evaluation
- Measuring a model's results separately for defined subsets, called slices, instead of one overall score. In planning, a slice is a group of publishers sharing a checkable trait.
A model can score well overall while failing a small group, because the small group barely moves the total. The Slice Finder paper (opens in a new tab) by Chung, Kraska, Polyzotis, Tae and Whang puts it this way: "the overall model performance can fail to reflect that of the smaller subsets." Its authors look for interpretable slices, since a slice a person can name is one a person can fix.
| Slice | Example values | Evidence for the tag |
|---|---|---|
| Site size | Popularity band, or no band published | Public popularity data, with a missing band kept as its own value |
| Language of the pages | English, Spanish, Canadian French | Language detected on sampled articles |
| Ownership | Independent, part of a media group | Ownership filings, ads.txt owner fields, shared ad codes |
| Base country | United States, Canada, elsewhere | The publisher's own address and records |
| Topic profile | Covers sensitive but safe subjects, or not | A reviewer's reading of sampled articles |
What are calibration and error rate parity?
Four measures cover the core of the research vocabulary. Each compares a different cell of the outcome table, and none is a default.
Definition
- Demographic parity (selection rate parity)
- Each slice is kept at the same rate. Google's machine learning glossary describes it as satisfied when results "are not dependent on a given sensitive attribute," irrespective of whether one group is more qualified than the other.
Definition
- Equal opportunity and equalized odds
- Equal opportunity asks for the same true positive rate in each slice: equally fit sites are kept equally often. Equalized odds, defined by Hardt, Price and Srebro, also asks for equal false positive rates, so unfit sites slip through equally often too.
Definition
- Calibration within groups
- A score means the same thing in each slice. Kleinberg, Mullainathan and Raghavan describe it as asking "that the scores mean what they claim to mean, even when considered separately in each group." A 0.8 fit score should be right about as often for a small Spanish language site as for a national English one.
| Measure | What it compares across slices | Strength | Weakness |
|---|---|---|---|
| Demographic parity | Share of candidates kept | Needs no reviewer labels, so it is cheap to run on a whole pool | Ignores fit: if one slice holds more unfit sites, forcing equal rates keeps the wrong ones |
| Equal opportunity | Share of fit sites kept (true positive rate) | Targets exclusion harm: good sites left out | Says nothing about unfit sites slipping through in one slice |
| Equalized odds | True positive and false positive rates | Protects publishers and buyers at once | Harder to meet; needs labeled samples large enough in each slice |
| Calibration within groups | Whether a given score is right equally often | Lets one threshold be read the same way in each slice | Can hold while one slice still loses more good sites at the threshold |
Hardt, Price and Srebro are blunt about the first measure: demographic parity "is seriously flawed on two counts." It does not ensure fairness, and it can cripple utility when outcomes really differ between groups. Equal keep rates are a warning light, not a target to force.
Can a model be fair on every measure at once?
No, except in narrow cases. Kleinberg, Mullainathan and Raghavan (opens in a new tab) formalized calibration within groups and two balance conditions, one for the positive class and one for the negative class, and proved that "except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously." The two special cases are perfect prediction and equal base rates, meaning each group holds the same share of positives.
A year later Pleiss and coauthors (opens in a new tab) showed that "calibration is compatible only with a single error constraint," equal false negative rates across groups. Yet Lazar Reich and Vijaykumar (opens in a new tab), at FORC 2021, derived "necessary and sufficient conditions" under which calibrated scores can still give equal error rates at a group blind threshold.
Google's fairness glossary (opens in a new tab) draws the lesson: incompatibility "doesn't imply that fairness efforts are fruitless," but "fairness must be defined contextually for a given ML problem." The harm the fair publisher planning hub describes, good independent sites dropped unread, is a false negative, so equal opportunity makes a sensible lead measure, with calibration checked beside it.
Why should slices describe publishers and not people?
A publisher trait can be checked from public evidence; a reader trait cannot be known without data about people. Ownership, page language, base country and site size come from filings, the pages and public popularity records. A slice such as sites read by older women would need personal data or a guess.
Canada's privacy regulator sets out why that matters. The Office of the Privacy Commissioner's guidelines on online behavioural advertising (opens in a new tab), modified August 11, 2025, say that information involved in tracking and targeting "will generally constitute personal information," and that information "such as health and financial data, ethnic and racial origins, political opinions" is generally considered sensitive. A fairness check built on publisher slices never needs to collect any of it.
How does PlacementGraph apply these measures to its own model?
How do you run a slice based check on a site selection model?
Six steps for one model and one kind of brief
- Draw a labeled sample. Two reviewers judge candidate sites against a real brief without seeing the model's call, and settle disagreements first.
- Tag each site's slices. Use publisher traits from the slice table, and keep a no data value for sites missing a popularity band.
- Choose the lead measure in writing. Equal opportunity for exclusion harm, false positive rates for spend risk, with calibration recorded beside either.
- Compute rates per slice with counts. A rate built on a handful of sites can swing with one decision, so show the count.
- Read the failing sites. Open the excluded fit sites in the weakest slice and note which signal pushed each one out.
- Change the rule and rerun. Fix that signal, then repeat the check on a fresh sample, not the one used to find the problem.
The AI bias guide covers where popularity skew enters a recommender, and the override framework covers a planner who disagrees with one call.
What does a fairness check look like on a real brief?
Fictional example
A Utah massage therapy clinic group checks its agency's site model
A fictional massage therapy clinic group in Salt Lake City, Provo and Ogden asks its agency for a display plan beside running, desk ergonomics, sports recovery and local community news, in English and Spanish. The client asks how the agency's site model treats smaller and Spanish language publishers.
The planner draws 120 candidate sites and has two reviewers label each one fit or unfit before seeing the model's calls, tagging each by page language, size band and ownership. Equal opportunity is written down as the lead measure, since the client's concern is good local sites left out.
The report shows fit English sites kept far more often than fit Spanish language sites, on counts small enough that the result is marked provisional. The excluded Spanish sites share one cause: short community bulletins read as thin content. The team reweighs bulletin pages, reruns on a fresh sample and records both runs with the model version.
Frequently asked questions
Is a fair model the same as an accurate model?
No. A model can be accurate overall and still fail a small slice, because the slice barely moves the total. Hardt, Price and Srebro note that their equalized odds criterion gets easier to meet as the predictor gets more accurate, so the two goals can support each other.
Does removing a slice label from the model make it fair?
Not by itself. Hardt, Price and Srebro call this idea fairness through unawareness and describe it as ineffective because of redundant encodings, ways of predicting a protected attribute from other features. A model that never sees language as a field can still learn it from page text, so test the outcomes per slice.
Is calibration the same as a confidence score?
No. A confidence score is a number the model outputs. Calibration is a property you test: among sites scored near 0.8, about eight in ten should prove fit when reviewers check them, and calibration within groups runs that test per publisher slice.
Who should choose the lead fairness measure for a plan?
The team that owns the plan's risks: the agency planner, with the client's sign off. It turns on which mistake costs more for this brief, excluding good publishers or keeping poor ones. Record the choice and the reason before the check runs.
Sources
- Equality of Opportunity in Supervised Learning (NIPS 2016) (opens in a new tab)Moritz Hardt, Eric Price and Nathan Srebro, NeurIPS proceedings
- Inherent Trade-Offs in the Fair Determination of Risk Scores (opens in a new tab)Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, arXiv, revised November 17, 2016
- On Fairness and Calibration (opens in a new tab)Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg and Kilian Q. Weinberger, arXiv, revised November 3, 2017
- A Possibility in Algorithmic Fairness: Can Calibration and Equal Error Rates Be Reconciled? (FORC 2021) (opens in a new tab)Claire Lazar Reich and Suhas Vijaykumar, Schloss Dagstuhl LIPIcs
- Automated Data Slicing for Model Validation (Slice Finder) (opens in a new tab)Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae and Steven Euijong Whang, arXiv, revised January 7, 2019
- Machine Learning Glossary: Fairness (opens in a new tab)Google for Developers, last updated April 13, 2026
- Guidelines on privacy and online behavioural advertising (opens in a new tab)Office of the Privacy Commissioner of Canada, modified August 11, 2025
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.