ICO AI risk toolkit: why AI teams need a data map before another model

Most AI governance problems do not begin with a dramatic failure.

They begin with a small missing answer.

• Which data does this feature use?
• Where did it come from?
• Which vendor touches it?
• Which model processes it?
• Is the output only a suggestion?
• Can a human override it?
• Can the data be removed later?
• Who owns the review?

When a team cannot answer those questions clearly, the AI feature may still work.

But it becomes harder to explain, harder to improve, harder to audit, and harder to trust.

That is the useful signal from the UK ICO’s AI and data protection risk toolkit.

The toolkit is designed to help organisations reduce risks to people’s rights and freedoms caused by their AI systems. The ICO’s broader AI audit framework also focuses on areas such as governance and accountability, transparency, and third-party contracts.

For software teams, the practical takeaway is simple:

Before adding another model, map the AI workflow that already exists.

The hidden problem: AI features spread faster than AI records

AI features rarely stay in one clean box.

A team starts with one model call.

Then it adds a vector database.

Then a vendor API.

Then a support workflow.

Then logs.

Then evaluation data.

Then analytics.

Then a fallback model.

Then a human review queue.

Then customer-facing explanations.

Then a new team uses the same data for a different purpose.

Nothing looks messy on day one.

But after a few months, the company may not have a single clear view of how the AI system actually works.

The code may know.

The product team may know part of it.

The data team may know another part.

The legal or compliance team may see only the policy.

Support may see the customer questions.

Leadership may see only the feature name.

That gap becomes the governance problem.

Why a data map matters

An AI data map is not only a compliance document.

It is a product operating tool.

It helps the team understand:

• what data enters the AI workflow,
• where that data came from,
• what the AI system does with it,
• which vendors or models process it,
• what output reaches the user,
• what decision the output may influence,
• where human review happens,
• how the system is monitored,
• and how data can be corrected, excluded, or removed.

Without this map, the team may keep adding AI capabilities on top of an unclear foundation.

That is how small risks become product-wide risks.

The practical AI data map

A useful AI data map should be simple enough for product, engineering, privacy, support, and leadership to read.

It does not need to be a 70-page document.

It needs to answer the questions the team will be asked later.

1. The data source

Start with the data.

For each AI feature, list the data sources.

Examples:

• customer messages,
• uploaded files,
• CRM records,
• product activity logs,
• public web pages,
• internal documents,
• support tickets,
• call transcripts,
• analytics events,
• workspace data,
• or third-party datasets.

Then record the basics:

• where it came from,
• why it is needed,
• whether it contains personal data,
• whether it contains sensitive data,
• whether the user or customer expects this use,
• and whether the data can be reduced before processing.

The goal is to stop vague phrases like “uses customer data.”

That phrase is not enough.

A useful map says exactly what kind of customer data, from which system, for which AI task.

2. The model or vendor

Next, list what processes the data.

This may include:

• hosted model APIs,
• internal models,
• open-source models,
• embedding models,
• rerankers,
• transcription tools,
• moderation tools,
• OCR systems,
• vector databases,
• analytics tools,
• or agent frameworks.

For each one, the team should know:

• who provides it,
• what data is sent,
• whether data is stored,
• whether it is used for training,
• where processing happens,
• what contract or privacy terms apply,
• and what fallback exists if the vendor changes.

This matters because many AI features are not one model.

They are a chain of services.

If the chain is not mapped, vendor risk becomes invisible.

3. The AI task

The map should describe what the AI system actually does.

Not in marketing language.

In workflow language.

For example:

• summarizes a support thread,
• extracts fields from a document,
• ranks candidate records,
• recommends a next action,
• drafts a reply,
• flags risky activity,
• generates a customer answer,
• routes a ticket,
• classifies user intent,
• or searches internal knowledge.

This makes the system easier to review.

A summarization tool, a decision-support tool, and an automated action tool do not need the same controls.

4. The output destination

Where does the AI output go?

This is one of the most important parts of the map.

An output may be shown to:

• a customer,
• an employee,
• a manager,
• a support agent,
• an admin,
• an internal dashboard,
• a downstream workflow,
• another AI system,
• or an external system.

The risk changes depending on where the output lands.

A draft shown to an internal reviewer is different from an answer sent directly to a customer.

A risk label shown in a dashboard is different from a label that changes access or priority.

The map should show the destination clearly.

5. The decision impact

Not every AI output is a decision.

But many outputs influence decisions.

The team should ask:

• Does this affect access?
• Does it affect money?
• Does it affect customer support priority?
• Does it affect hiring or workload?
• Does it affect account status?
• Does it affect visibility?
• Does it affect what a user is allowed to do?
• Does it affect what a person is told?

If the output influences something important, the workflow needs stronger review.

This is where a data map becomes more than documentation.

It becomes a product safety tool.

6. The human review point

The map should show where a person can review, correct, or stop the AI output.

A vague note like “human review available” is not enough.

The map should define:

• who reviews,
• when they review,
• what they can change,
• what information they see,
• what happens if they disagree,
• and whether the user can request review.

This matters because human review only helps when it is part of the actual workflow.

7. The removal or correction path

AI systems need a way to change.

A customer may delete data.

A source may become unreliable.

A document may be outdated.

A vendor may change terms.

A model may be replaced.

A user may challenge an output.

A regulator, buyer, or internal leader may ask how data is removed from active use.

The team should know:

• how to remove a source,
• how to update an index,
• how to correct wrong data,
• how to exclude certain records,
• how to retrain or reprocess where needed,
• and who approves the change.

If the team cannot change the data path, the AI feature becomes harder to govern over time.

The mistake is treating the AI feature as one thing

Most AI features are not one thing.

They are a chain.

Data source → preprocessing → model or vendor → storage → retrieval → output → human review → user action → logging → monitoring.

If the team only reviews the model, it misses the system.

If the team only reviews the policy, it misses the workflow.

If the team only reviews the output, it misses the data path.

The useful review is the full path.

A simple field note for product teams

Before adding another model, create one page for the AI feature.

The page should answer:

• What data enters the workflow?
• Where does it come from?
• Which model or vendor processes it?
• What does the AI system do?
• Where does the output go?
• What decision can it influence?
• Where can a person review it?
• How can data be corrected or removed?
• Who owns the workflow?

That one page will not solve everything.

But it will reveal what the team actually understands.

And that is often the first step toward better governance.

Founder takeaway

AI governance does not start with a long policy.

It starts with a clear map.

A team should be able to point to an AI feature and explain:

• the data it uses,
• the model or vendor involved,
• the output it creates,
• the decision it may influence,
• the review step,
• the correction path,
• and the owner.

If the team cannot map the workflow, it probably cannot govern the workflow.

Before adding the next AI feature, map the one already in production.

That may be the most useful governance work a product team can do.

Sources

• ICO: AI and data protection risk toolkit
  
  https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/ai-and-data-protection-risk-toolkit/

• ICO: Artificial intelligence audit framework
  
  https://ico.org.uk/for-organisations/advice-and-services/audits/data-protection-audit-framework/toolkits/artificial-intelligence/