What is language variant bias, and how do buyers test for it?
Part of Fair publisher planning
The short answer
Language variant bias is when a classifier, filter or ranking model performs worse on a regional form of a language, such as Quebec French, Mexican Spanish or a regional English, than on the standard form. In advertising it can mislabel pages, lower relevance scores and quietly remove publishers. Buyers find it by testing error rates per variant.
Key takeaways
- The DIALECTBENCH benchmark found performance gaps between standard and non-standard varieties, and an average score for French or Spanish can hide the gap.
- In the 2026 QFrBLiMP benchmark, tested models showed significant degradation on Quebec French grammar, while the strongest models stayed within the margin of significance.
- Variant bias reaches a media plan through three doors: wrong language labels, false suitability flags and weaker topic matches.
- A fair test uses matched pages in each variant, native speaker review and separate error rates per task.
- Buyers can ask providers which variants they test, on which tasks, with what error rates, and what happens when a variant falls behind.
What is language variant bias in advertising tools?
Definition
- Language variant bias
- A systematic gap in a tool's accuracy between varieties of one language, so that pages written in a regional or non-standard variety are labeled, scored or ranked less accurately than pages in the standard variety.
Advertising tools run several language tasks on each page: they detect the language, classify suitability, match topics to a brief and sometimes judge page quality. Each task is trained and tested on data, and that data leans toward standard written forms. The authors of DIALECTBENCH (opens in a new tab), presented at ACL 2024, write that "Most NLP benchmarks are limited to standard language varieties." Their benchmark of "10 text-level tasks covering 281 varieties" found "performance disparities between standard and non-standard language varieties."
Language tags can carry the variant, but only when someone sets them. The BCP 47 language tag rules (RFC 5646) (opens in a new tab) say region subtags "indicate linguistic variations associated with or appropriate to a specific country, territory, or region," as in fr-CA or es-MX, and give "es-419" for Spanish appropriate to the Latin America and Caribbean region. The OpenRTB 2.6 specification (opens in a new tab) offers Content.langb, "Content language using IETF BCP 47." A bid request that sends only "fr" tells a buyer nothing about which French it is.
What does research show about variant gaps in language models?
None of these studies tested an advertising classifier, so none gives an error rate for ad tools. What they show is a pattern across tasks and languages that a buyer should test for rather than assume away.
| Study | Variety and task | Finding | Limit |
|---|---|---|---|
| Beauchemin and colleagues, QFrBLiMP, EACL 2026 Findings | Quebec French grammar, 1,761 minimal pairs judged by 12 native speakers | "a significant performance degradation for most models on Quebec-French" | "the most capable models remain within the statistical significance interval" |
| Faisal and colleagues, DIALECTBENCH, ACL 2024 | 281 varieties across 10 text tasks | "performance disparities between standard and non-standard language varieties" | Gaps differ by language cluster and task |
| Kantharuban, Vulić and Korhonen, EMNLP 2023 Findings | Regional dialects in translation and speech recognition | Training data effects "significant but not consistent across models or languages" | Speech and translation, not page classification |
| Ziems and colleagues, VALUE, ACL 2022 | African American Vernacular English in language understanding | Dialect features "can lead to a drop in model performance" | One variety of English, 11 features |
How can Canadian French content be misjudged?
Quebec French has grammatical phenomena of its own, and those are what QFrBLiMP tests. Its authors built their pairs from "an official online resource maintained by a Québec government institution" and found that "All benchmarked models consistently fail on phenomena requiring deep semantic understanding." A tool trained mostly on European French text can stumble on a Quebec page in ways an average French score never shows.
| Task | How it can go wrong | Check |
|---|---|---|
| Language detection | A page mixing French with English loanwords is labeled uncertain or English | Review pages flagged uncertain by a French reader |
| Suitability | Regional slang or idiom read literally and flagged as profanity or violence | Compare flag rates on matched Quebec and European French pages |
| Topic matching | Regional terms for everyday things miss the brief's European French or English topic words | Write brief topics with regional terms, then compare match rates |
| Page quality | Informal regional writing scored as thin or low quality | Native speaker review of a sample of low scored pages |
These are failure modes to test for, not measured rates. The Media Rating Council's 2018 supplement on content level context and brand safety (opens in a new tab) asks verification services to consider them directly: "Different languages and language-related nuances should be considered when analyzing metadata including dialects." For planning French Canadian campaigns as a whole, see the French Canadian advertising guide.
Why does variant bias reduce reach?
Each error removes or demotes pages. Research on hate speech detection shows the mechanism. Xia, Field and Tsvetkov (SocialNLP 2020 (opens in a new tab)) write that bias in training data causes "AAE text to often be mislabeled as abusive/offensive/hate speech (high false positive rate)." Applied to brand suitability, a false flag on a variant means a publisher writing in that variant is blocked for content it does not carry. The reach loss is an inference that a campaign has to test.
The losses can compound. A publisher that survives detection with an uncertain label, picks up one false flag and ranks lower on topic match can fall off a list that each step alone would have kept. The readers it serves are still in the target market; the plan just stops reaching them through the publishers they read.
How do you test quality by language variant?
Test each variant on each task separately, with people who speak it. QFrBLiMP had 12 native speakers annotate each pair, and VALUE recruited fluent speakers to validate its dialect transformations through "linguistic acceptability judgments." A buyer's test can be far smaller and still informative.
Six steps for a variant test on a site list tool
- Pick the variants the brief needs. For a Canadian brief, Quebec and Acadian French beside European French; for a US Spanish brief, Mexican and Caribbean Spanish beside a general Spanish set.
- Collect matched pages. Gather 30 or more pages per variant on the same topics and of similar length, from publishers you have read.
- Label them by hand. Have a native speaker of each variant mark language, suitability and topic for each page.
- Run the tool on all pages. Record its language label, each suitability flag and its topic match score.
- Compare error rates per variant and per task. Count false flags, missed topics and wrong labels separately; never merge them into one number.
- Repeat after changes. Run the same set when the provider updates its model or the brief adds a variant.
What should a buyer ask a tool provider about language variants?
MRC guidance gives buyers a basis for asking. The same supplement says that where "reduced performance or accuracy is expected in certain data conditions," this "must be actively disclosed to users via estimates of error," and that "Biases in training and evaluation data should be reduced to the extent possible and disclosed where material." Spliethöver, Menon and Wachsmuth (ACL 2024 Findings (opens in a new tab)) also show that modeling dialect explicitly "improves their fairness" in their experiments, so mitigation is possible and worth asking about.
Questions for a provider about language variants
- Which variants of each supported language does your test set include, by name?
- What are your error rates per variant for language detection, suitability and topic matching?
- How many pages per variant were tested, and when?
- Who labeled the test pages, and were they native speakers of each variant?
- What happens when one variant scores worse than the others: is release held, or is an exception granted?
- Can a publisher or agency report a variant problem, and who reviews it?
How does PlacementGraph check its quality by language variant?
What does a variant test change on a real plan?
Fictional example
A fictional laundry detergent brand in Southern California
An unnamed laundry detergent brand asks its agency for a Spanish and English display plan in Los Angeles, San Diego and the Inland Empire, beside family, home and parenting content. The agency's suitability tool returns a Spanish list heavy on large national publishers written in a general Spanish, and few Southern California outlets written for Mexican American readers.
The planner runs a variant test. A bilingual reviewer labels 40 Mexican Spanish pages and 40 general Spanish pages on the same family topics. The tool flags Mexican Spanish pages for profanity at a visibly higher rate, and the reviewer traces the flags to everyday regional slang read literally.
The agency sends the results to the provider with the page list, sets the profanity category to manual review for Spanish pages during the flight, and adds five regional publishers the reviewer confirmed. The plan records the test date, the sample and the open question to the provider.
- Draft: Spanish list built on default suitability settings.
- Test: 80 matched pages, labeled by a native speaker.
- Plan: regional publishers added with reasons, provider question logged.
Frequently asked questions
Is language variant bias the same as poor translation?
No. Translation errors happen when content moves between languages. Variant bias happens inside one language, when a tool reads Quebec French or Mexican Spanish less accurately than the standard form, even on original pages that were never translated.
Does a fr-CA or es-MX tag fix variant bias?
A tag tells a tool which variant a page claims to be. It does not make the tool better at reading that variant. A tag helps route pages to a variant aware model or test set, if the provider has one; ask whether it does.
Does variant bias affect English too?
Yes. VALUE found performance drops on African American Vernacular English, and regional forms of English can carry idiom and spelling a tool reads differently. A US or Canadian brief in English can still need a variant test if it targets regional publishers.
How large does a variant test need to be?
There is no industry standard. Thirty to fifty matched pages per variant is a practical start for spotting a large gap and opening a conversation with a provider. A small gap needs a larger sample before anyone acts on it, so record the sample size with the result.
Sources
- QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs (opens in a new tab)Findings of the Association for Computational Linguistics: EACL 2026, Beauchemin, Veilleux, Khoury and Roy, March 2026
- DIALECTBENCH: An NLP Benchmark for Dialects, Varieties, and Closely-Related Languages (opens in a new tab)ACL 2024, Faisal and colleagues, August 2024
- Quantifying the Dialect Gap and its Correlates Across Languages (opens in a new tab)Findings of EMNLP 2023, Kantharuban, Vulić and Korhonen, December 2023
- VALUE: Understanding Dialect Disparity in NLU (opens in a new tab)arXiv, Ziems and colleagues, ACL 2022 (revised September 13, 2022)
- Demoting Racial Bias in Hate Speech Detection (opens in a new tab)SocialNLP 2020, Xia, Field and Tsvetkov, July 2020
- Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairness (opens in a new tab)Findings of ACL 2024, Spliethöver, Menon and Wachsmuth, August 2024
- MRC Supplement to IAB Guidelines for the Conduct of Ad Verification: Enhanced Content Level Context and Brand Safety (opens in a new tab)Media Rating Council, September 2018
- RFC 5646 (BCP 47): Tags for Identifying Languages (opens in a new tab)IETF, September 2009
- OpenRTB 2.6 specification (opens in a new tab)IAB Tech Lab
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.