Do brand safety tools treat some communities unfairly?
Part of Fair publisher planning
The short answer
They can. Peer reviewed research shows text classifiers rating posts in African American English as offensive up to twice as often, and models scoring mentions of disability or identity terms as more toxic. Brand safety tools built on similar methods can carry the same skew, so buyers should test a tool with paired content before trusting its blocks.
Key takeaways
- Bias in content classification is documented in peer reviewed research, not only in anecdotes from publishers.
- Classifiers learn from labeled examples, so they can link identity terms, dialect and disability with harm when the training data does.
- Pages that name an identity, such as Black, LGBTQ+ or disability coverage, can be exposed, because the research found identity terms alone raised scores.
- A buyer can test any tool with paired articles that differ only in identity terms or dialect, and compare the block rates.
- When bias is found, fix the setting or the rule that causes it, record the change and test again.
What does bias in a brand safety tool look like?
Definition
- Brand safety bias
- A pattern in which a content classification tool flags, blocks or downgrades pages about a group of people, or written in that group's language, more often than comparable pages that do not mention them, for reasons unrelated to the content's actual risk.
The pattern can be hard to see on a single page. A buyer sees one blocked article and assumes the tool had a reason. Bias shows up when blocks are counted across a large set of pages and compared between groups. An article about a Black owned bakery, a guide to disability benefits and a profile of a transgender athlete may each be blocked for a different stated reason, while the common factor is that they describe people the training data linked with conflict.
Brand safety tools can combine keyword lists with machine learning classifiers. The narrower case of keyword lists blocking news coverage is covered in news blocklisting. This page looks at the research on the classifiers themselves, and at how a buyer can check a tool before relying on it.
What research shows bias in content classification?
The evidence below comes from studies of toxicity and hate speech classifiers, the same family of models used to judge whether text is offensive. The findings are consistent across research groups, datasets and years.
| Study | What it tested | What it found |
|---|---|---|
| Dixon et al., AAAI/ACM AI, Ethics, and Society, 2018 | Toxicity classifiers, measured with demographic identity terms in the text | Imbalanced training data produced biased models; rebalancing reduced bias without hurting overall quality |
| Sap et al., ACL 2019 | Hate speech datasets and the models trained on them | Tweets in African American English were up to two times more likely to be labeled offensive |
| Davidson, Bhattacharya and Weber, Workshop on Abusive Language Online, 2019 | Classifiers trained on five annotated Twitter datasets | Systematic racial bias in all five, with African American English more often labeled abusive |
| Borkan et al., 2019 | Score distributions across identity groups in real comments | Threshold agnostic metrics revealed subtle unintended bias in public models |
| Hutchinson et al., ACL 2020 | A toxicity model and a sentiment model | Mentions of disability shifted scores, and embeddings carried similar bias |
The Sap et al. paper (opens in a new tab) also traced a cause: annotators who were unaware of dialect labeled posts as offensive more often, and when they were primed to consider dialect, they were significantly less likely to do so. Davidson and colleagues (opens in a new tab) warned that such systems could penalize the very groups that abuse detection is meant to protect. Hutchinson and colleagues (opens in a new tab) found that topics like gun violence, homelessness and drug addiction were overrepresented in texts about mental illness, which may push models to treat any mention of it as risky.
How do identity terms trigger false blocks?
A classifier learns which words tend to appear in examples people labeled harmful. In imbalanced training data, identity words such as gay, Black, Muslim or disabled appear disproportionately inside abuse, because abuse targets identity. The model then learns a shortcut: the presence of the word raises the risk score, even when the page is a celebration, a news report or a how to guide. Dixon and colleagues (opens in a new tab) built their bias measure around exactly these demographic identity terms.
Dialect works the same way. If annotators rated posts in African American English as more offensive, a model trained on their labels repeats the judgment at scale. For publishers, the consequence is structural. A site written for a Black audience, an LGBTQ+ community magazine or a disability news outlet can name its community throughout its coverage. Each mention is a small push on the score, and the site as a whole can end up rated riskier than a general site covering the same events.
| Mechanism | Example page at risk | What to check |
|---|---|---|
| Identity term as a risk signal | A profile of a gay small business owner | Does swapping the identity word change the score? |
| Dialect read as offensive | A culture column written in African American English | Do dialect and standard versions score the same? |
| Topic overlap in training data | A guide to mental health services | Is the page judged by its treatment of the subject or by its topic words? |
| Site level averaging | A community outlet whose pages name its audience | Is the site scored on sampled meaning or on word frequency? |
Disagreement between tools is a further signal that classification is not settled. A January 2026 study, Inconsistencies in Classification of Online News Articles (opens in a new tab), compared three brand safety providers on 4,352 news articles from 51 domains and found significant discrepancies between them. When tools disagree, a block is a judgment, not a fact, and judgments can be tested.
How can a buyer test a tool for bias?
The research offers a practical template. Borkan and colleagues (opens in a new tab) measured how score distributions differ across identity groups rather than relying on one overall accuracy figure, and Ribeiro and colleagues (opens in a new tab) showed that behavioral tests, borrowed from software testing, find failures that held out accuracy hides. A buyer can apply both ideas without access to the model.
A paired content test for any brand safety tool
- Collect real pages. Gather pages from the publishers the client cares about, including community focused outlets, across ordinary topics such as food, business, sports and family.
- Build matched pairs. For a set of pages, prepare a version that differs only in an identity term or dialect, and keep every other word the same.
- Run both through the tool. Record each page's category, severity and whether it would be blocked at the client's settings.
- Compare block rates by group. Count blocks for pages that mention each community and for matched pages that do not. A consistent gap is the signal to investigate.
- Read the blocked pages. Have a person judge whether each block matches the client's written policy. Count false blocks by group, not only in total.
- Repeat after every change. Rerun the same test set when the tool, the settings or the client policy changes, and keep the results with the plan.
What a fair test set includes
- Pages from community focused publishers, not only from general news sites.
- Ordinary topics such as recipes, local business and sports, where no block should occur.
- Matched versions that differ only in an identity term or dialect.
- Pages in each language the plan buys, including Canadian French where relevant.
- A written definition of what the client actually wants blocked, used to judge every result.
- Results recorded by group, with the date and the tool settings used.
How does PlacementGraph guard against bias in suitability?
What should a brand do when bias is found?
Treat a bias finding as a defect in the setup, not as a reason to drop brand safety. The goal is to keep the protection the client wants while removing blocks that have nothing to do with risk.
The fix can be a setting. A category set to its strictest level may be catching community coverage the client never meant to exclude, and loosening it by one level removes the gap. A custom keyword list may contain identity words added years ago. Where the tool itself is at fault, share the paired test results with its provider and ask for a response. In the meantime, an inclusion list of reviewed community publishers keeps them in the plan. Guidance on specific communities is in LGBTQ+ content advertising, disability content advertising and diverse owned media.
Fictional example
A fictional hair care brand tests its setup in Maryland and Washington, DC
A fictional maker of hair care products for curly and coily hair plans a display campaign beside beauty, lifestyle and local culture content in Maryland and Washington, DC, in English. The client wants to buy on Black owned and Black audience publishers as well as general beauty sites, and asks the agency to confirm that its brand safety setup will not block them.
The agency builds a test set from recent beauty, recipe and local events pages on both kinds of site, with matched versions that differ only in community references. Run at the client's settings, pages from the community publishers are blocked noticeably more often than their matched versions, almost all under one strictly set category.
A reviewer reads every blocked page and finds none that breaks the client's written policy. The agency loosens that one category by a level, shares the test with the tool's provider, adds the reviewed community publishers to an inclusion list and reruns the test. The gap closes, and the results go into the plan file.
Frequently asked questions
Is brand safety bias intentional?
The research points to training data and labeling: when people label examples, their own blind spots shape what the model learns, and identity words that appear in abusive training examples can become risk signals. The outcome can be unfair without anyone choosing it, which is why testing matters more than intent.
Can a brand safety provider prove its tool is unbiased?
No tool can be proven free of bias, but a provider can share how it measures bias across groups and what it found. Ask for results broken down by group rather than one accuracy figure, and run your own paired test on pages from the publishers your client buys.
Does testing for bias require personal data about readers?
No. The test uses pages and their wording, not information about people. It compares how a tool treats content that mentions a community with matched content that does not. Slices describe the pages and publishers, never the readers.
Do these findings apply to languages other than English?
The studies cited here tested English text. Bias in other languages has to be tested in those languages, so a plan that buys Canadian French or Spanish pages should include them in the test set rather than assuming English results carry over.
Sources
- The Risk of Racial Bias in Hate Speech Detection (opens in a new tab)Sap, Card, Gabriel, Choi and Smith, ACL 2019
- Racial Bias in Hate Speech and Abusive Language Detection Datasets (opens in a new tab)Davidson, Bhattacharya and Weber, Third Workshop on Abusive Language Online, 2019
- Measuring and Mitigating Unintended Bias in Text Classification (opens in a new tab)Dixon, Li, Sorensen, Thain and Vasserman, AAAI/ACM Conference on AI, Ethics, and Society, 2018
- Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification (opens in a new tab)Borkan, Dixon, Sorensen, Thain and Vasserman, arXiv, 2019
- Social Biases in NLP Models as Barriers for Persons with Disabilities (opens in a new tab)Hutchinson et al., ACL 2020
- Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (opens in a new tab)Ribeiro, Wu, Guestrin and Singh, ACL 2020
- Inconsistencies in Classification of Online News Articles: A Call for Common Standards in Brand Safety Services (opens in a new tab)Smith et al., arXiv, January 2026
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.