How is the language of a page detected?
Part of Contextual targeting
The short answer
Page language is detected by running a statistical model over the article's main text and comparing its letter and word patterns with patterns learned for each language. It describes the page, not the reader, whose browser setting is a separate signal. Detection weakens on short text, closely related languages and mixed language passages, which is when a person should check.
Key takeaways
- Language detection reads the article text and returns a best guess with some measure of confidence, not a fact.
- The language of a page and the language setting of a reader's browser are separate signals, and OpenRTB carries them in separate fields.
- Short text, closely related languages, foreign names and mixed language passages are the documented ways detection goes wrong.
- An uncertain result should be recorded as uncertain and checked by a reader of the language, not forced into one label.
- A bilingual page is best labeled with its main language plus the share and place of the second language.
How is the language of a page detected?
Detection starts with the right text. Menus, footers, cookie banners and ads are stripped away so the model sees the article itself, because a French article inside an English site template would otherwise read as partly English. The model then compares the text's patterns, such as which short letter sequences and words appear and how often, with the patterns it learned for each language, and returns the most likely language with a score.
The W3C describes the method plainly in its working draft Strings on the Web: Language and Direction Metadata (opens in a new tab) (First Public Working Draft, July 16, 2026): detectors "are usually statistically based heuristics, such as using n-gram frequency in a language, possibly coupled with other data." The same section says the approach "is NOT recommended" as a substitute for explicit language metadata in data formats, which is a reminder that detection is a fallback when no one has labeled the text.
Definition
- Content language detection
- Identifying the language of a text from the text itself, using a model trained on examples of each language. Also called language identification.
Why is page language different from a user's language setting?
The two describe different things. Page language describes what the article is written in. A browser's language setting describes what the reader asked for, and it can be wrong about the reader. The W3C's note on Accept-Language (opens in a new tab) says the header carries "information about the user's language preferences" and warns: "Many users never change the defaults for Accept-Language." It also notes that people borrow and share machines.
The ad tech standard keeps them apart as well. In the OpenRTB 2.6 specification (opens in a new tab), the Content object's language field is "Content language using ISO-639-1-alpha-2", while the Device object's language field is "Browser language using ISO-639-1-alpha-2". Both also have a langb version using IETF BCP 47 tags. A bilingual reader in San Antonio may run an English browser and read Spanish news; a contextual plan cares about the second fact.
| Signal | What it describes | Where it comes from | What it cannot tell you |
|---|---|---|---|
| Detected page language | The language of the article's main text | A model run over the text | Who is reading or what they prefer |
| Declared page language | What the publisher says the page is in | The lang attribute in the page code | Whether the text actually matches it |
| Browser language | What the reader's software requests | Accept-Language header or device settings | Which language the reader reads best, or what is on the page |
What causes language detection to fail?
The failure modes are well documented. The W3C draft lists them in its issues section. Heuristics are "more accurate the longer and more representative the text being scanned is", and "Short strings may not detect well." Detection "is limited to the languages for which one has a detector." And: "Inclusions, such as personal or brand names in another language or script, can throw off the detection."
Research adds two more. A 2025 ACL paper by Rob van der Goot, Identifying Open Challenges in Language Identification (opens in a new tab), found that "cross-domain setups are the most challenging": a model trained on one kind of text does worse on another, so a detector should be tested on the kind of pages it will classify. And a 2024 LREC-COLING paper on FastSpell (opens in a new tab) says that "commonly used language identifiers struggle to differentiate between similar or closely-related languages", its reason for adding a spell checker as a second opinion.
| Cause | Source | What it looks like on a site |
|---|---|---|
| Short text | W3C string-meta draft, 5.4.3 | Photo galleries, recipe cards and event listings with a few lines of text |
| Names from another language | W3C string-meta draft, 5.4.3 | An English review full of Spanish dish and place names |
| Language not covered | W3C string-meta draft, 5.4.3 | A language the detector was never trained on, labeled as a near neighbor |
| Closely related languages | FastSpell, LREC-COLING 2024 | Spanish and Portuguese, or similar pairs, swapped on short passages |
| Different kind of text | van der Goot, ACL 2025 | Forum posts or slang heavy columns scored by a model trained on cleaner text |
| Mixed languages | Burchell and colleagues, EACL 2024 | Articles that switch between English and Spanish within a paragraph |
When should an uncertain language result go to manual review?
No standard sets a confidence cutoff for review, so each team sets its own and writes it down. The useful rule is to send a result to a person when the cause of doubt is one the model is known to handle badly, or when a wrong label would change the plan. A Spanish language site wrongly labeled English drops out of a Spanish brief; an English site wrongly labeled Spanish fills a Spanish list with pages the client's ads do not match.
Send the page to a reader of the language when
- The main text is short, such as a gallery, listing or caption led page.
- Two languages score close to each other.
- The detected language disagrees with the publisher's declared language.
- The page mixes languages within paragraphs, not only in separate sections.
- The detected language is one the brief targets and the site would be excluded or kept on this result alone.
- Several sampled articles from the same site disagree with each other.
How should bilingual pages be labeled?
Start with how publishers are told to declare it. The W3C's guidance on declaring language in HTML (opens in a new tab) gives two rules: "Always use a language attribute on the html tag to declare the default language of the text in the page", and "When the page contains content in another language, add a language attribute to an element surrounding that content." One main language for the page, with marked passages in another, is the model a planner's label can follow.
Detection should then work at the level of passages. The EACL 2024 paper Code-Switched Language Identification is Harder Than You Think (opens in a new tab) reformulates the task "as a sentence-level multi-label tagging problem" and concludes that "no current approach is adequate" for text that switches languages. For planning, that argues for scoring paragraphs or sentences separately and keeping more than one label on the page, rather than one label for the whole article.
| Page pattern | Label to record | Planning note |
|---|---|---|
| One language throughout | Main language only | Plan it with that language's list |
| Main language with a marked second language section | Main language, plus second language and its section | Fits the main language list; note the section |
| Side by side translation | Both languages, equal | Fits both lists only if the ad can match both |
| Switches within paragraphs | Both languages, mixed | Send to a bilingual reader before deciding |
| Declared language differs from detected | Detected language, with the conflict noted | Check a few more articles from the same site |
What does language review look like on a real brief?
Fictional example
A gift basket company in San Antonio and El Paso
An unnamed gift basket and food gift company wants display ads beside English and Spanish articles about family celebrations, hosting and regional food in San Antonio and El Paso. It runs separate English and Spanish creative, so each page needs one clear language.
Detection labels the bulk of the candidate pages with confidence. Three kinds go to a bilingual reviewer: short recipe cards, where the text is too brief; English food columns full of Spanish dish names, which score close between the two languages; and a community site whose articles switch between English and Spanish within paragraphs.
The reviewer confirms the recipe cards as Spanish and the food columns as English. The community site is labeled mixed and kept off both single language lists, with a note that it would suit a bilingual creative if the client adds one.
- Kept: pages with a confirmed single language, on the matching list.
- Held: the mixed language community site, labeled mixed.
- Recorded: why each page went to a person.
How does PlacementGraph detect the language of a site?
Frequently asked questions
Can language detection tell Canadian French from European French?
Not reliably from short text. Varieties of one language are closely related, and the FastSpell authors note that identifiers struggle with closely related languages. Place names, spelling and vocabulary can hint at the variety, but a reader of the language should confirm when the plan depends on it.
Does detection work on pages with very little text?
Poorly. The W3C draft says detection is more accurate the longer and more representative the text, and that short strings may not detect well. For galleries and listings, judge the site from its longer articles, or mark the short pages as unconfirmed.
Should comments and reader posts count toward a page's language?
No. Comments can be in any language and change after publication, so they describe the readers who posted, not the article. Detect the language of the main text, and note separately if the comment section is large and in another language.
How many articles should be checked to set a site's language?
Enough to see whether the site is consistent. Check several recent articles from different sections; if they agree, label the site with that language. If they disagree, label the sections separately, since some bilingual publishers split languages by section.
Sources
- Strings on the Web: Language and Direction Metadata (First Public Working Draft, July 16, 2026) (opens in a new tab)W3C Internationalization Working Group
- Accept-Language used for locale setting (opens in a new tab)W3C Internationalization
- Declaring language in HTML (opens in a new tab)W3C Internationalization
- OpenRTB 2.6 specification (Content and Device language fields) (opens in a new tab)IAB Tech Lab
- Identifying Open Challenges in Language Identification (ACL 2025) (opens in a new tab)Association for Computational Linguistics
- Code-Switched Language Identification is Harder Than You Think (EACL 2024) (opens in a new tab)Association for Computational Linguistics
- FastSpell: The LangId Magic Spell (LREC-COLING 2024) (opens in a new tab)ELRA and ICCL
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.