MFA risk score: how should a calibrated score work?
Part of MFA avoidance
The short answer
An MFA risk score estimates how likely a website is to be made for advertising, given the evidence about it. A useful score is calibrated, so sites scored 0.8 turn out MFA about 80 percent of the time, is reported per signal area rather than as one verdict, and becomes action through thresholds set from the cost of each error.
Key takeaways
- A risk score is an estimate with uncertainty, not a label; treat 0.55 and 0.95 differently even if both are above a line.
- Report a probability for each signal area, such as ad experience, traffic sourcing and content, so reviewers see why a site scored high.
- Thresholds are a business decision: set them from what a wrongly excluded publisher and a wrongly included MFA site each cost.
- Check calibration on a labeled sample with a reliability diagram, and check it again for small publishers, languages and countries.
- Missing evidence lowers confidence; it should never raise the risk score on its own.
What is an MFA risk score?
A team that reviews sites at scale can turn its judgment into a number. An MFA risk score is that number: an estimate, between 0 and 1, of how likely it is that a site is built mainly to resell ad impressions. The trade groups' 2023 definition, published by the ANA (opens in a new tab), lists five characteristics MFA sites show in some combination and leaves each advertiser to decide its tolerance. A score is one way to apply that definition consistently across thousands of domains.
Definition
- Calibrated MFA risk score
- A probability estimate that a site is made for advertising, where scores match observed frequencies: among sites scored near 0.8, roughly eight in ten turn out to be MFA when reviewed.
The scikit-learn documentation on probability calibration (opens in a new tab) describes the same property for any classifier: among samples given a probability close to 0.8, "approximately 80% actually belong to the positive class." Without that property a score is only a ranking. It can tell you that one site looks worse than another, but not how worried to be about either, and any threshold set on it is a guess.
Why should MFA risk be a probability, not a yes or no?
A yes or no flag hides the difference between a site that barely crossed a line and one that showed every sign. It also hides the reason. When a publisher disputes a flag, or a client asks why a familiar site is missing from a plan, the reviewer needs to know how strong the evidence was and where it came from.
| Question | Binary MFA flag | Calibrated probability per area |
|---|---|---|
| Ease of use | Simple to load into a buying platform as a block | Needs a threshold before it becomes an action |
| Strength of evidence | Lost: 0.51 and 0.99 look the same | Kept: a reviewer sees how close the call was |
| Reason for the call | A single label | Visible by signal area |
| Fit to client tolerance | One line for every client | Different thresholds per client on the same score |
| Risk of false precision | Low; it claims little | Real if the score is not checked for calibration |
Which signals feed an MFA risk score?
Group signals into areas that mirror the trade group definition, and give each area its own probability. A site's profile then reads like a short report: high on ad experience, low on traffic sourcing, unknown on network ownership.
| Signal area | Evidence collected | Watch out for |
|---|---|---|
| Ad experience | Share of the screen given to ads on phone and desktop, refresh timing, sticky and autoplay units | Legitimate publishers with heavy but usable layouts |
| Traffic sourcing | Sponsored link units pointing to the site, recommendation boxes that send visitors out, weak organic popularity | Sites that buy some traffic for real audience growth |
| Content | Generic, rewritten or templated articles; no bylines or sources; bursts of publishing | Specialist sites with short, practical articles |
| Design and network | One template and shared ad or analytics IDs across several domains | Small publishers using off the shelf site themes |
| Supporting clues | Domain age, a recent change of owner, many resellers and no direct seller | Honest new or small sites can show these too |
Two rules keep the inputs honest. Evidence that could not be collected, a page that would not load or a layout that could not be measured, should widen the uncertainty for that area, never push the score up. And supporting clues should only adjust a score that other areas have already raised, so a new domain is never treated as MFA because it is new.
How do you choose review and exclusion thresholds?
The score and the decision are separate problems. The scikit-learn guide to tuning the decision threshold (opens in a new tab) splits them into "the statistical problem of learning a model to predict, ideally, class probabilities" and "the decision problem to take concrete action based on those probability predictions," and warns that a default cut-off of 0.5 is "most certainly not ideal for most use cases." For MFA, the right line depends on what each mistake costs a given client.
| Error | What happens | Who bears the cost | Pushes the threshold |
|---|---|---|---|
| False exclusion | A real publisher is left off the plan | The publisher, and the brand that loses relevant reach | Higher, so fewer good sites are removed |
| False inclusion | An MFA site stays on the plan | The brand, through spend on pages nobody chose to read | Lower, so more risky sites are caught |
Set the two lines for one client
- Agree the costs in plain words. Ask the brand whether missing a good niche publisher or running on one MFA site is worse, and by roughly how much.
- Read precision at each candidate line. On the labeled sample, note what share of sites above each score were truly MFA.
- Place the exclusion line where the sample shows agreement. Above it, the labeled sample should show the model's calls matching a person's.
- Place the review line below it. Sites between the two lines go to a person; size the band to the review time the team actually has.
- Write both lines into the policy. Record the date, the model version and the sample they were set on, as the MFA avoidance hub describes for policy records.
How do you check a score is calibrated?
Calibration is tested, not assumed. Guo, Pleiss, Sun and Weinberger's ICML 2017 paper On Calibration of Modern Neural Networks (opens in a new tab) found that "modern neural networks, unlike those from a decade ago, are poorly calibrated," and that a simple post-processing step, temperature scaling, was "surprisingly effective" at calibrating predictions on most of the datasets they tested. Any model that produces an MFA score, from a rule set to a neural network, needs the same check.
The standard tool is a reliability diagram, or calibration curve. It groups sites by predicted score and plots, for each group, the share that reviewers labeled MFA. A calibrated score sits on the diagonal. The scikit-learn guide also warns that a single summary such as the Brier score mixes calibration with how well the model separates classes, so "a lower Brier loss" does not necessarily mean better calibration. When a score is off, Platt scaling fits a sigmoid correction; isotonic regression is more flexible but "more prone to overfitting, especially on small datasets," and the guide says isotonic performs as well as or better than sigmoid when there are more than about 1,000 samples.
Calibration checks for an MFA risk score
- A labeled sample was reviewed by people who did not see the model's score.
- A reliability diagram was drawn for each signal area, not only for the combined score.
- Calibration was checked separately for small and large sites, for each language and for each target country.
- The sample includes recent sites, so new MFA formats are represented.
- Calibration is re-tested whenever the model version changes, and the result is recorded.
- Sites near each threshold were re-reviewed to confirm the lines still sit in the right place.
What does a score based review look like on a real brief?
Fictional example
A Newfoundland and Labrador heat pump brief
A fictional St. John's heat pump installer asks its agency for an English display plan in Newfoundland and Labrador, beside home energy guides, renovation advice and local weather coverage. The agency's MFA score is calibrated on a labeled sample, and for this client the review line is 0.4 and the exclusion line 0.85, set after the brand said losing local publishers would hurt more than a small amount of MFA exposure.
A home energy blog scores 0.9 on ad experience, 0.85 on traffic sourcing and 0.8 on content, with sponsored link units pointing to it and rewritten articles. All three areas agree, so it is excluded with the signals listed. A local weather site scores 0.7 on ad experience but under 0.2 elsewhere. It goes to review, where a person finds dense but usable pages and a long staff archive, and keeps it.
A new renovation site has no measurable layout because its pages timed out. Its ad experience area is marked unknown rather than high, and it is queued for a second visit instead of being scored as risky.
How does PlacementGraph score made-for-ads risk?
Frequently asked questions
What should a buyer ask a supplier of an MFA score?
Ask how the score was calibrated and on what labeled sample, whether it is broken down by signal area, how missing data is handled, and whether calibration was checked for small publishers and for each language and country. Ask too which model version produced a given score, so a later dispute can be traced.
Can several area scores be combined into one number?
They can for sorting, but keep the areas visible. A combined number is useful to order a long list for review. The decision should still show which areas drove it, because heavy ads alone and heavy ads plus bought traffic plus copied content are different findings.
Does a low MFA risk score mean a site is a quality publisher?
No. A low score says the site does not look built to resell ad impressions. It says nothing about whether the site does original work or answers for its accuracy. Use a separate positive review, such as the framework in our guide to what makes a quality publisher.
Why can the same score lead to different decisions for two clients?
Because the score describes the site and the threshold describes the client. Two brands can agree on a site's probability and still act differently: one may exclude at 0.7 to protect spend, while another reviews up to 0.9 to keep local and niche publishers in reach.
Sources
- Leading Trade Groups Define “Made For Advertising” Websites (opens in a new tab)ANA (Association of National Advertisers), September 26, 2023
- Probability calibration (opens in a new tab)scikit-learn User Guide
- Tuning the decision threshold for class prediction (opens in a new tab)scikit-learn User Guide
- On Calibration of Modern Neural Networks (opens in a new tab)Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, ICML 2017 (arXiv 1706.04599)
- AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1) (opens in a new tab)National Institute of Standards and Technology, January 26, 2023
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.