What should an AI audit trail record for each planning decision?
Part of Fair publisher planning
The short answer
An AI audit trail for a media plan records, for each automated site decision, the model version, the confirmed brief, the dated inputs it read, the evidence and reason it gave, what a person did with it, who approved the list and when. With those records a reviewer can rebuild why any site was kept or dropped, months later.
Key takeaways
- Automated site decisions depend on inputs that change monthly or weekly, so a result without dated inputs cannot be reproduced.
- Record the model's name and version with every result, as the Model Cards paper and NIST's AI RMF Playbook both recommend for version history.
- No single retention period fits every record; Canadian privacy guidance asks for minimum and maximum periods, and the FTC advises keeping data only for a legitimate business need.
- Access should follow need to know: planners write, approvers sign, reviewers read across plans, and clients see a summary.
- Slice fields on each record are what let a fairness review compare how publishers fared across plans and model versions.
Why does an automated media plan need an audit trail?
Definition
- AI audit trail (media planning)
- A dated record, kept for each automated recommendation and each human action on it, that holds enough information to reconstruct why a site or channel was included, ranked or left out of an approved plan.
Because the evidence behind a recommendation does not stand still. The ads.txt 1.1 specification (opens in a new tab) says that if no cache control directives are present, "consuming systems should default to an expiry of 7 days." The Chrome UX Report on BigQuery (opens in a new tab) "is released on the second Tuesday of the following month," so a site's popularity band can move between one plan and the next.
A plan approved in March and questioned in September will be judged on September's evidence unless March's was saved. The audit trail shows what the tool saw, which version made the call and who accepted it.
What model version should be recorded with a planning decision?
The exact version that produced the result, not the product name. The Model Cards for Model Reporting (opens in a new tab) paper (Mitchell and colleagues, FAT* 2019) lists model date and model version among a model's basic details and frames the version question as "Which version of the model is it, and how does it differ from previous versions?" NIST's AI RMF Playbook (opens in a new tab), under MANAGE 4.3, suggests organizations "Maintain a database of system changes, reason for change, and details of how the change was made, tested and deployed" and "Maintain version history information and metadata."
| Field | Example of what to store | Why it matters later |
|---|---|---|
| Model name and version | The identifier the tool prints on the result | Ties the result to the release notes for that version |
| Run date and time zone | When the search or scoring ran | Matches the result to the inputs available that day |
| Settings in force | Thresholds, filters and switches the planner chose | Separates a model change from a settings change |
| Reference vocabularies | The content taxonomy and its version, such as IAB Tech Lab Content Taxonomy 3.1 | Category labels can change between versions |
| Tool change log entry | The vendor's note for the version, if published | Explains a shift in results between two plans |
Store the version on each result, not once per plan: a list built over several days or refreshed mid flight can mix two versions.
Which inputs should be saved with each result?
Every input the result depended on, dated. NIST's Playbook, under GOVERN 4.2, suggests documenting an AI system's "data provenance, including sources, origins, transformations, augmentations, labels, dependencies, constraints, and metadata."
| Input | What to keep | How it changes |
|---|---|---|
| Confirmed brief | The structured brief as the person confirmed it, plus the original wording | Edited between rounds of review |
| Sampled articles | The URLs read and the date each was fetched | Articles are edited, moved or deleted |
| Authorized sellers | The ads.txt and sellers.json entries matched, with the fetch date | Default cache expiry is 7 days under the spec |
| Audience by country | The popularity band and the data month it came from | Released monthly |
| Quality and risk signals | The signals listed with the result and their values | Ad layouts and traffic sources change |
| Language detected | The languages found on the sampled pages | Sites add or drop language editions |
For public records such as seller files, a link and a date are enough. For articles, a short excerpt or archived copy of the passage that drove the match protects against later edits.
What belongs in the audit record for one planning decision?
The checklist below is the full record for one site on one list version. A row that cannot fill every item is a gap to note, not a reason to skip the row.
Audit record checklist for one automated site decision
- A decision ID that links the row to the plan, the client and the list version.
- The model name and version, the run date and the settings in force.
- The confirmed brief version the search used.
- Each input with its source and fetch date: sampled pages, seller files, audience data month.
- The result: included, ranked position, flagged for review, or left out, with the reason as the tool stated it.
- The human action: accepted, removed, restored or edited, with the reason code.
- The reviewer's name or role and the date of review.
- The approver and approval date for the list version that was exported.
- The export file name, its destination platform and the date it was handed over.
- The publisher slices for the site: size band, ownership, language, base country and sensitive topic coverage.
How long should planning records be kept?
No public rule sets one period for media planning records, so the answer comes from what the records contain and what they are for. An audit trail of this kind describes publishers and pages, not people. Its personal information is mainly reviewer and approver names and contact details, and that is where privacy guidance applies.
In Canada, the Office of the Privacy Commissioner's guidance on PIPEDA Principle 5 (opens in a new tab) says to "Institute maximum and minimum retention periods that take into account any legal requirements or restrictions" and to "Keep personal information only as long as it is needed to serve those purposes." Canada's privacy regulators, in their December 7, 2023 principles for generative AI (opens in a new tab), ask organizations to "Establish and abide by appropriate retention schedules for personal information." In the United States, the FTC's Start with Security (opens in a new tab) guide advises businesses to "Hold on to information only as long as you have a legitimate business need."
| Record type | Contains personal information? | What anchors the period |
|---|---|---|
| Model version, inputs, results | No, it describes sites and pages | The client contract's audit rights and the life of the campaign |
| Reviewer and approver fields | Yes, names and roles | The organization's retention schedule for staff records |
| Article excerpts or archived copies | Possibly, if articles name people | Keep only the passage needed; prefer links |
| Exported files | No | The same period as the plan they belong to |
Who should be able to see the audit trail?
People with a reason to, at the level of detail they need. The FTC's guide recommends controls "to make sure employees have access only on a 'need to know' basis," and the OPC's PIPEDA Principle 7 (opens in a new tab) page asks organizations to "Protect personal information in a way that is appropriate to how sensitive it is." For a planning team that suggests four levels of access.
| Role | Access | What they use it for |
|---|---|---|
| Planner | Write their own decisions; read the plan | Recording actions and reasons as they work |
| Approver | Read the plan's full record; sign the list version | Checking the evidence before export |
| Fairness or quality reviewer | Read across plans and clients the agency permits | Pattern reviews by slice and model version |
| Client | A summary of sites, reasons and approval | Challenging a call or confirming the plan |
Agencies holding several clients' plans in one system should keep each client's records apart.
How does an audit trail support a fairness review?
It turns a suspicion into a query. With slice fields, results and reasons stored per row, a reviewer can count how each slice fared, read why its sites were left out, and compare two model versions on similar briefs. Canada's generative AI principles ask organizations to "Regularly re-visit and re-evaluate accountability measures (including bias testing and assessments)", and that is only possible when the past decisions are still readable.
Three fairness questions the audit trail can answer
- Compare survival by slice. Count, for each publisher slice, how many candidates reached the approved list, and read the stated reasons for those that did not.
- Compare model versions. For briefs run before and after a release, check whether one slice's share of results moved while the briefs stayed similar.
- Separate people from the system. Compare the tool's results with the human actions on them, to see whether a skew entered at scoring or at review.
The AI bias guide describes how to test the recommender for skew; the audit trail is what lets that test be run on past plans rather than only on the next one.
How does PlacementGraph record what each result was based on?
PlacementGraph records a model version with every result, and its target lists keep versions.
What does an audit trail look like on a real plan?
Fictional example
A fictional utility's home energy rebate campaign in Washington
A fictional electric utility serving Seattle, Tacoma and Spokane briefs a display plan beside home improvement, heating and household budgeting content to promote its energy efficiency rebates. Because ratepayer programs are reviewed, the utility asks its agency for a record of how the site list was built.
The agency's audit table holds one row per candidate site. Each row carries the model version, the run date, the confirmed brief version, the sampled article links with fetch dates, the seller file date and the audience data month, plus the planner's action and the approver's sign off.
Five months later the utility's reviewer asks why two Spokane neighborhood news sites are missing. The rows show both lacked an authorized sellers file on the run date; one has published one since. The planner adds it after a fresh check, and the new row records the change.
Frequently asked questions
Does an audit trail need full copies of every sampled article?
No. A link with the fetch date is the base, and a short excerpt of the passage that drove the match protects against later edits. Full copies take storage, can carry personal information about people named in stories, and raise copyright questions. Keep them only where a record would be meaningless without the original text.
What happens to the audit trail when a vendor updates the model mid campaign?
Nothing is lost if each result carries its own model version: rows before and after the update can be compared directly. Ask the vendor for the change note of the new version and attach it to the plan.
Can screenshots serve as an audit record?
Only as a supplement. A screenshot cannot be sorted, counted or joined to other plans, which is what a fairness review needs. Store the fields as structured data, and keep a screenshot only where the visual layout itself is the evidence, such as an ad heavy page.
Should sites the tool left out be in the audit trail?
Yes. The sites that never reached the list are the ones a bias question is about. Store each left out candidate with its stated reason and inputs, so a reviewer can tell a site dropped for a quality finding from one dropped because a dataset had no record of it.
Sources
- AI RMF Playbook (opens in a new tab)National Institute of Standards and Technology (NIST)
- Model Cards for Model Reporting (opens in a new tab)Mitchell et al., FAT* 2019 (arXiv, revised January 14, 2019)
- PIPEDA Fair Information Principle 5: Limiting Use, Disclosure, and Retention (opens in a new tab)Office of the Privacy Commissioner of Canada, modified August 13, 2020
- PIPEDA Fair Information Principle 7: Safeguards (opens in a new tab)Office of the Privacy Commissioner of Canada, modified August 11, 2025
- Principles for responsible, trustworthy and privacy-protective generative AI technologies (opens in a new tab)Office of the Privacy Commissioner of Canada and Canadian privacy authorities, December 7, 2023
- Start with Security: A Guide for Business (opens in a new tab)US Federal Trade Commission
- ads.txt Specification Version 1.1 (opens in a new tab)IAB Tech Lab (GitHub)
- CrUX on BigQuery (opens in a new tab)Google Chrome for Developers
Researched and written by PlacementGraph Research. Read how we research, source and update these pages. Corrections: diana@placementgraph.com.