Computer vision advertising: what does reading the images add?
Part of Contextual targeting
The short answer
Computer vision advertising uses image and video analysis to judge the content around an ad: objects, scenes, faces, logos and text inside pictures. It adds evidence that text alone misses, such as a photo that changes an article's meaning. It also errs, on irony, small objects and text in images, so its labels need the same checking as text labels.
Key takeaways
- Visual classifiers return labels and likelihoods, such as an object name or a racy score, not an understanding of what the page means.
- An image can reverse the meaning of a caption, which is why research datasets pair text and pictures to test models.
- Irony, text inside images and small objects are documented weak points of visual classification.
- On YouTube, Google and YouTube state that visual content, thumbnails and metadata all feed suitability and theme decisions.
- Reading the text is enough when images only illustrate it; visual review earns its cost when pictures carry the risk.
What does computer vision detect on pages and in videos?
Definition
- Computer vision in advertising
- Software that analyzes the pixels of images and video frames to produce labels, such as objects, scenes, faces, logos or embedded text, that a buying or review system can use to judge the content around an ad.
Public cloud documentation shows the building blocks. Google Cloud's Vision API lists features such as "Detect labels", "Detect logos", "Detect multiple objects", "Detect faces" and "Detect and extract text from images". Its SafeSearch detection (opens in a new tab) "uses five categories (adult, spoof, medical, violence, and racy)" and "returns the likelihood that each is present in a given image", on a scale from very unlikely to very likely.
| Task | What it returns | What a buyer can use it for | What it does not tell you |
|---|---|---|---|
| Label detection | General names for what the image shows, with confidence | Confirming a page's pictures match its topic | Whether the picture is used seriously, ironically or critically |
| Object localization | Named objects and where they sit in the frame | Spotting a product, a weapon or a bottle in a scene | Whether a small or partly hidden object was missed |
| Text detection (OCR) | Words found inside the image | Reading signs, screenshots, memes and chart labels | What the extracted words mean together with the picture |
| Logo detection | Brand marks found in the image | Noticing another brand's products on the page | Whether the brand is endorsed, reviewed or mocked |
| Safety likelihoods | Scores such as adult, violence or racy | Flagging images for human review | Where the client's own line sits between categories |
In video, the same tasks run on sampled frames, and can be joined with audio and the video's text fields. Google's help page on contextual targeting (opens in a new tab) says its system "analyzes the content of each YouTube video", considering text, metadata, visual content, audio content and channel information, and that "Based on this analysis, the central theme of each video is determined."
Why can an image change a page's meaning?
Words and pictures carry meaning together, and either one can flip the other. A neutral caption over an offensive photo becomes offensive; a harsh caption over a harmless photo can become a joke. Researchers built the Hateful Memes dataset (opens in a new tab) around this problem: its authors added "benign confounders" so that models could not rely on the text or the image alone, and constructed it "such that unimodal models struggle and only multimodal models can succeed."
The same paper reported that state-of-the-art methods "perform poorly" on that task, at 64.73 percent accuracy against 84.7 percent for humans. Those figures date from 2020 and describe one benchmark, not today's commercial systems, but the design lesson holds: a judgment that reads only the article text, or only the photos, can miss the meaning that sits between them.
What errors does visual classification make?
Visual classifiers err in patterns that research has documented. Knowing them helps a buyer read a vendor's visual label as a lead to check, not a verdict.
| Weak point | What goes wrong | What the research shows |
|---|---|---|
| Irony and sarcasm | The image and caption contradict each other on purpose, and literal labels miss the joke | A COLING 2025 paper on multimodal sarcasm says the task depends on "accurately capturing incongruities from different modalities" |
| Text inside images | A sign, label or screenshot carries the meaning, and object labels ignore it | The TextVQA authors wrote that "today's VQA models can not read!" and built 45,336 questions on 28,408 images to test reading |
| Small objects | A bottle on a table or a logo on a shirt is a few pixels wide and goes unseen | The tiny faces paper found "the cues for recognizing a 3px tall face are fundamentally different than those for recognizing a 300px tall face" |
| Order of events in video | A model labels scenes correctly but misreads what happened, or answers from text cues | A 2025 study of video benchmarks found "strong language priors, where models can answer questions without watching the video" |
| Context of depiction | A documentary or educational image is scored like gratuitous content | YouTube's own guidelines say shocking elements are judged in context: "context matters" |
The research figures above are benchmark results from specific papers, cited to show the kinds of error, not current accuracy of any product. The contextual classification accuracy guide explains how to sample and measure a classifier's errors on a client's own pages.
How does visual analysis apply to video ads?
On YouTube, the platform itself reads more than the words. Its advertiser friendly best practices (opens in a new tab) say "We take all aspects of your content into account", listing the video, Short or live stream content, title, thumbnail, description and tags. The same page admits "Our systems don't always get it right" and lets creators request human review.
Thumbnails and opening seconds carry their own weight. YouTube's advertiser friendly content guidelines (opens in a new tab) mark "Graphic game violence in the thumbnail or in the first seven seconds of the video" as earning no ad revenue, while the same violence after the first seven seconds can still earn. For a buyer, this means a channel's thumbnails are evidence in their own right, and a review that reads only titles and descriptions sees part of what YouTube judges.
A visual review of a YouTube channel in five steps
- Scan the thumbnail grid. Look at the last thirty uploads as images, before reading any title.
- Open three videos at the start. Watch the opening seconds, where YouTube's guidelines place extra weight.
- Skip to the middle of each. Check that the content matches what the title and thumbnail promised.
- Read on-screen text. Captions burned into the picture, signs and overlays can carry claims the description does not.
- Note what you could not see. Live streams and deleted videos leave gaps; record them as gaps, not as clean.
When is reading the text enough?
Text analysis reads what the publisher wrote, and on long form editorial pages that is where the subject lives. The W3C's accessibility standard asks that "All non-text content that is presented to the user has a text alternative that serves the equivalent purpose", per WCAG 2.2 success criterion 1.1.1 (opens in a new tab). Where a publisher follows it, image descriptions add text a reader can check, though alt text is written by the publisher and can be missing or thin.
| Page format | Where the meaning sits | Text review | Visual review |
|---|---|---|---|
| Long article with stock photos | In the text; photos illustrate it | Sufficient for topic and suitability | Adds little beyond a spot check |
| Product review with photos | Text, plus product shots | Sufficient for topic | Useful to confirm the product shown |
| Photo gallery or slideshow | In the images; captions are short | Weak; few words to read | Needed for suitability |
| Meme or screenshot page | In text inside images and in the pairing | Weak; text is in pixels | Needed, with text extraction and a person |
| Video page or channel | In frames, audio, thumbnails and metadata | Reads titles and descriptions only | Needed for thumbnails and opening seconds |
Signs a site list needs visual review before approval
- Sampled articles carry fewer than a few paragraphs of text around their images.
- The site publishes galleries, memes or screenshot collections as main content.
- The client's avoid topics are visual by nature, such as nudity, injuries or weapons.
- The product category is one a casual photo can place in a bad light, such as swimwear, alcohol or children's products.
- Video is the main format and thumbnails are designed to shock or tease.
What does a visual review look like on a real brief?
Fictional example
A swimwear brand on the South Carolina coast
A fictional swimwear label plans spring display ads around Charleston and Myrtle Beach, beside beach travel guides, packing lists and summer fashion articles. The brief avoids sexually suggestive content and alcohol.
The text review clears a beach travel publisher and two fashion blogs: the articles are about resorts, fabrics and sizing. A sample of image led pages tells a different story on one fashion site, whose galleries pair swimsuit photos with suggestive captions in the images themselves. That site moves to a person's review. On a travel guide, a beer bottle sits small in a beach photo; the reviewer notes it and keeps the page, since the article is about tide pools.
The planner records which pages were checked visually and why, so the client can see the evidence. No result figures are implied.
What does PlacementGraph read on a page and a channel?
Frequently asked questions
Does alt text tell a classifier what an image shows?
It tells a text reader what the publisher says the image shows. WCAG asks for text alternatives that serve the equivalent purpose, and decorative images can be marked so they are ignored. Alt text is only as good as the publisher's habits: missing, generic or stuffed alt text is a reason to look at the images directly.
Can visual analysis spot other brands on a page?
Logo detection is a documented feature of public vision services, so a system can flag brand marks in photos. It cannot say whether the brand is being reviewed, praised or mocked; that needs the text and a person. Use logo flags as a prompt to read the page, not as an exclusion rule.
Are YouTube thumbnails judged separately from the video?
YouTube's guidelines treat the thumbnail as part of the content, and some rules name it directly: graphic game violence in the thumbnail earns no ad revenue under its advertiser friendly guidelines. A buyer reviewing a channel should look at thumbnails as images, since they can carry content the video itself handles more mildly.
Does visual analysis make a list safer than text analysis alone?
It adds evidence on image led pages and video, and adds its own errors, such as missed small objects and misread irony. A list is safer when each method covers the formats it reads well and a person reviews the pages where they disagree. Neither method removes the need to read the served placement report after launch.
Sources
- Detect explicit content (SafeSearch) (opens in a new tab)Google Cloud Vision API documentation
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (Kiela et al., 2020) (opens in a new tab)arXiv
- Towards VQA Models That Can Read (Singh et al., 2019) (opens in a new tab)arXiv
- Finding Tiny Faces (Hu and Ramanan, 2016) (opens in a new tab)arXiv
- Ambiguity-aware Multi-level Incongruity Fusion Network for Multi-Modal Sarcasm Detection (Li et al., COLING 2025) (opens in a new tab)ACL Anthology
- Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding? (Feng et al., 2025) (opens in a new tab)arXiv
- About contextual targeting (opens in a new tab)Google Ads Help
- Best practices for creating advertiser friendly content (opens in a new tab)YouTube Help
- Advertiser-friendly content guidelines (opens in a new tab)YouTube Help
- Understanding Success Criterion 1.1.1: Non-text Content (WCAG 2.2) (opens in a new tab)W3C Web Accessibility Initiative
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.