Build a Privacy Filter Before Your AI Agent Remembers User Actions

AI agents are starting to remember more than chats. They can watch clicks, typed text, app switches, browser context, files, tool calls, and workflow history. That memory can make an agent feel useful fast, but it can also turn a helpful feature into a quiet privacy incident.

If you are building an AI product, do not start with “how much can we capture?” Start with “what is the smallest event stream that still helps the user?” This guide shows a practical privacy filter you can place between raw user activity and agent memory.

Why this matters now

Recent AI tooling trends point in the same direction: agents are moving from chat boxes into operating systems, browsers, IDEs, customer support tools, analytics dashboards, and workflow automation platforms. The more useful the agent becomes, the more context it wants.

That creates a new engineering problem.

Traditional app logs record requests and errors. Agent memory records intent, context, and behavior. A raw event can include:

• What the user clicked
• What they typed
• Which customer record was open
• Which browser page was active
• Which tool the agent called
• Which file or message was summarized
• Which secrets or personal details appeared nearby

This is not just observability. It is a privacy boundary.

The practical trigger is simple: computer-use agents and workflow agents now need history to resume work, personalize answers, and automate multi-step tasks. But developers, security reviewers, and buyers are asking harder questions about PII, retention, auditability, user consent, and whether agent traces can leak private business data.

The common mistake: treating memory like logs

Most teams already have logs, traces, analytics events, and support transcripts. So when they add agent memory, they often reuse the same pattern:

1. Capture the event.
2. Save it to storage.
3. Index it for search.
4. Let the agent retrieve it later.

That is easy to ship. It is also too broad.

Agent memory needs a stricter path because it may be used to generate future answers or actions. A normal log line might be seen by engineers. A memory item might be read by a model, combined with other data, and used to make a decision.

The safer pattern is:

Raw event → privacy filter → purpose check → redacted memory → retention policy → retrieval policy

The privacy filter is not a prompt. It is application code that decides what the agent is allowed to remember.

What an AI agent privacy filter should do

A good privacy filter has five jobs.

Job	Question it answers	Example
Event allowlist	Should this event be captured at all?	Save “opened invoice page,” not every mouse coordinate.
Sensitive data detection	Does the payload contain PII, secrets, or regulated data?	Detect emails, API keys, card-like numbers, tokens.
Purpose binding	Why is this memory needed?	Resume task, improve support, audit approval.
Retention control	How long can this memory live?	48 hours for raw traces, 30 days for redacted task summaries.
Retrieval control	Who or what can read it later?	Only the same user, tenant, role, and task type.

The filter should run before indexing, summarization, embedding, analytics export, or model calls.

Step 1: classify your event stream

Do not start with redaction. Start with event classes. Redaction helps when you must keep data. Classification helps you avoid collecting data in the first place.

A simple event taxonomy might look like this:

Event class	Risk	Store by default?	Notes
Navigation event	Low	Yes, redacted	Page type, not full URL if it contains IDs.
Tool call metadata	Medium	Yes	Store tool name, status, cost, policy result.
User typed text	High	No	Store only if explicitly needed and redacted.
Screen content	High	No	Prefer structured app state over screenshots.
File content	High	No	Store references and hashes, not full content.
Approval decision	Medium	Yes	Keep reviewer, action, timestamp, and reason.
Secret or credential	Critical	Never	Block and alert if detected.

Here is a small TypeScript example:

type EventClass =
  | "navigation"
  | "tool_call"
  | "typed_text"
  | "screen_content"
  | "file_content"
  | "approval"
  | "secret";

type CaptureDecision = "store" | "redact_then_store" | "summarize_only" | "drop";

const capturePolicy: Record<EventClass, CaptureDecision> = {
  navigation: "redact_then_store",
  tool_call: "store",
  typed_text: "summarize_only",
  screen_content: "drop",
  file_content: "summarize_only",
  approval: "store",
  secret: "drop",
};

This looks boring. That is the point. Privacy should not depend on a clever prompt at runtime.

Step 2: reduce the event before redacting it

A raw event often contains too much context. Reduce it into a smaller shape before running PII detection.

Bad memory candidate:

{
  "type": "typed_text",
  "value": "My card is 4242 4242 4242 4242 and my email is [e-mail]",
  "url": "https://app.example.com/customers/cus_782/orders/ord_991",
  "dom": "...full page text...",
  "timestamp": "2026-08-15T03:30:00Z"
}

Better memory candidate:

{
  "type": "task_signal",
  "summary": "User entered payment-related information during checkout setup.",
  "page_type": "checkout_settings",
  "tenant_id": "tenant_123",
  "user_id": "user_456",
  "timestamp": "2026-08-15T03:30:00Z"
}

Notice what changed:

• No full typed text
• No full DOM
• No full URL with record IDs
• No card number
• No personal email
• Enough context to resume the task

Reduction is the cheapest privacy win you can ship.

Step 3: add layered PII and secret detection

Use multiple detectors. Regex is not enough, but regex is still useful.

You want to detect:

• Email addresses
• Phone numbers
• Access tokens
• API keys
• Session cookies
• Credit-card-like patterns
• Private keys
• OAuth codes
• Personal addresses
• Customer names in risky contexts
• Health, legal, financial, or employment data when relevant

Example filter:

type Redaction = {
  redacted: string;
  findings: Array<{ type: string; count: number }>;
};

function redactSensitiveText(input: string): Redaction {
  const findings: Redaction["findings"] = [];
  let text = input;

  const patterns = [
    { type: "email", regex: /[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}/gi },
    { type: "credit_card_like", regex: /\b(?:\d[ -]*?){13,19}\b/g },
    { type: "api_key_like", regex: /\b(?:sk|pk|ghp|xoxb|AKIA)[A-Za-z0-9_\-]{16,}\b/g },
    { type: "private_key", regex: /-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----/g },
  ];

  for (const pattern of patterns) {
    const matches = text.match(pattern.regex);
    if (matches?.length) {
      findings.push({ type: pattern.type, count: matches.length });
      text = text.replace(pattern.regex, `[REDACTED_${pattern.type.toUpperCase()}]`);
    }
  }

  return { redacted: text, findings };
}

For production, combine this with a structured PII service, domain-specific dictionaries, and field-level policies. The key is to store the findings separately from the raw value.

Step 4: bind every memory to a purpose

A memory without a purpose becomes a future liability. Add a purpose field when the memory is created.

Common purposes:

• resume_task
• support_debugging
• security_audit
• billing_dispute
• quality_evaluation
• personalization

Each purpose should control retention and retrieval.

type MemoryPurpose =
  | "resume_task"
  | "support_debugging"
  | "security_audit"
  | "billing_dispute"
  | "quality_evaluation"
  | "personalization";

const retentionDays: Record<MemoryPurpose, number> = {
  resume_task: 7,
  support_debugging: 30,
  security_audit: 180,
  billing_dispute: 365,
  quality_evaluation: 14,
  personalization: 90,
};

This gives your product, legal, and engineering teams one shared control surface.

It also prevents a common failure mode: using data collected for debugging as long-term personalization memory.

Step 5: separate raw traces from agent memory

Raw traces and agent memory should not live in the same bucket.

A useful split:

1. Raw event buffer
   
   Short-lived, encrypted, tightly restricted, used for immediate debugging or user-visible replay.

2. Redacted memory store
   
   Longer-lived, purpose-bound, searchable by the agent only through policy checks.

3. Audit ledger
   
   Append-only records of decisions: what was stored, why, which policy allowed it, and when it expires.

The agent should usually retrieve from the redacted memory store, not the raw buffer.

[User activity]
      |
      v
[Raw event buffer: short TTL]
      |
      v
[Privacy filter]
      |
      +--> [Drop / block / alert]
      |
      v
[Redacted memory store]
      |
      v
[Policy-checked retrieval]
      |
      v
[Agent response or action]

This structure makes deletion easier and audits less painful.

Step 6: check consent at capture and retrieval

Consent is not a checkbox on the settings page. It is runtime state.

Check consent when:

• The event is captured
• The event is transformed into memory
• The memory is retrieved
• The memory is exported
• The user revokes access
• The tenant changes policy

Example:

type ConsentState = {
  userId: string;
  tenantId: string;
  allowAgentMemory: boolean;
  allowPersonalization: boolean;
  allowSupportReview: boolean;
  revokedAt?: string;
};

function canStoreMemory(consent: ConsentState, purpose: MemoryPurpose): boolean {
  if (!consent.allowAgentMemory || consent.revokedAt) return false;

  if (purpose === "personalization") return consent.allowPersonalization;
  if (purpose === "support_debugging") return consent.allowSupportReview;

  return true;
}

If consent is revoked, new memory should stop immediately. Existing memory should either expire, be deleted, or become inaccessible depending on your product policy and legal requirements.

Step 7: make retrieval policy stricter than storage policy

A memory can be safe to store but unsafe to retrieve in a different context.

Before retrieving memory for an agent, check:

• Same tenant
• Same user or approved team scope
• Same purpose
• Role permissions
• Data region
• Retention expiry
• Sensitivity level
• Current task relevance

This prevents awkward bugs like a support agent retrieving billing context during a product tutorial, or a workspace agent pulling private notes into a shared channel.

A retrieval policy can be simple:

function canRetrieveMemory(args: {
  requesterTenantId: string;
  requesterUserId: string;
  memoryTenantId: string;
  memoryUserId: string;
  purpose: MemoryPurpose;
  requestedPurpose: MemoryPurpose;
  expiresAt: Date;
}) {
  if (args.requesterTenantId !== args.memoryTenantId) return false;
  if (args.requesterUserId !== args.memoryUserId) return false;
  if (args.purpose !== args.requestedPurpose) return false;
  if (args.expiresAt.getTime() < Date.now()) return false;

  return true;
}

In team products, replace the user equality check with a role and resource policy. Keep the default narrow.

Step 8: log decisions, not secrets

You still need audit logs. Just do not put raw secrets in them.

A good audit record includes:

{
  "memory_id": "mem_123",
  "tenant_id": "tenant_123",
  "user_id": "user_456",
  "event_class": "typed_text",
  "decision": "summarize_only",
  "purpose": "resume_task",
  "pii_findings": [{ "type": "email", "count": 1 }],
  "policy_version": "privacy-filter-v4",
  "created_at": "2026-08-15T03:30:00Z",
  "expires_at": "2026-08-22T03:30:00Z"
}

This record helps you answer:

• Why did the agent remember this?
• Which policy allowed it?
• Was PII detected?
• When will it expire?
• Who can retrieve it?

That is much better than saving the whole raw payload and hoping nobody looks too closely.

Practical implementation checklist

Use this as a build checklist for your first privacy filter.

Capture controls

• Define event classes before instrumentation.
• Default high-risk events to drop or summarize_only.
• Avoid storing raw typed text unless the user explicitly asks the agent to remember it.
• Prefer page type, resource type, and task state over full URL or DOM text.
• Keep raw buffers short-lived and encrypted.

Redaction controls

• Run PII and secret detection before storage.
• Replace sensitive values with typed placeholders.
• Store finding counts, not raw matches.
• Add domain-specific detectors for your product.
• Block critical secrets instead of masking them.

Purpose and consent controls

• Require a purpose for every memory object.
• Map each purpose to retention and retrieval rules.
• Check consent at capture and retrieval.
• Support revocation and deletion workflows.
• Do not reuse debugging data for personalization without explicit permission.

Retrieval controls

• Enforce tenant isolation.
• Enforce user or role scope.
• Match requested purpose to stored purpose.
• Filter expired memories before retrieval.
• Keep sensitive memories out of shared contexts.

Audit controls

• Log privacy decisions with policy version.
• Keep audit logs redacted.
• Track who or what retrieved memory.
• Build a simple memory inspection page for users or admins.
• Test deletion, export, and expiry before launch.

Where most articles stop short

Most privacy guides cover PII redaction, audit logs, or broad data governance. Agent memory needs one more layer: event design. Decide what to do with clicks, typed text, page context, tool calls, desktop actions, and model-readable summaries before they enter storage.

Ask this before saving anything:

Would this memory still feel reasonable if the user inspected it, exported it, or saw it during an incident review?

If the answer feels uncomfortable, reduce it.

A simple architecture you can ship first

For a small team, do not overbuild. Start with this:

1. Create an agent_events table with short retention.
2. Create an agent_memories table with redacted summaries only.
3. Create an agent_memory_audit table for policy decisions.
4. Add a privacy filter service before any embedding or model call.
5. Add a nightly expiry job.
6. Add a user-facing “forget my agent memory” action.
7. Add tests for PII, secrets, tenant isolation, and consent revocation.

Schema sketch:

CREATE TABLE agent_memories (
  id TEXT PRIMARY KEY,
  tenant_id TEXT NOT NULL,
  user_id TEXT NOT NULL,
  purpose TEXT NOT NULL,
  sensitivity TEXT NOT NULL,
  summary TEXT NOT NULL,
  metadata JSONB NOT NULL DEFAULT '{}',
  policy_version TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at TIMESTAMPTZ NOT NULL
);

CREATE INDEX agent_memories_lookup
ON agent_memories (tenant_id, user_id, purpose, expires_at);

Then test it like a security feature, not like a logging feature.

Test cases worth adding

Add these to CI:

• A fake API key is never stored in memory.
• A card-like number is redacted before storage.
• A revoked user cannot create new personalization memory.
• Expired memory is not retrieved.
• User A cannot retrieve User B’s memory in the same tenant.
• Tenant A cannot retrieve Tenant B’s memory.
• Debug memory is not retrieved for personalization.
• Raw event buffer expires on schedule.
• Audit logs contain findings but not raw sensitive values.
• Shared-channel agents cannot access private one-to-one memory.

These tests will catch more real issues than another paragraph in your privacy policy.

Final thoughts

Agent memory is powerful because it compresses user context into future usefulness. That same compression can hide privacy mistakes if you capture too much, store it too long, or retrieve it in the wrong place.

The safest path is not “never remember anything.” That makes agents weak. The safest path is to remember less, explain why, expire it on purpose, and retrieve it only when the current task deserves it.

Build the privacy filter before the memory feature becomes popular. It is much easier to start narrow than to clean up a giant pile of raw history later.

FAQ

What is an AI agent privacy filter?

An AI agent privacy filter is application logic that decides which user activity events can become agent memory. It classifies events, redacts sensitive data, checks consent, assigns a purpose, applies retention, and controls retrieval.

Should AI agents store raw user actions?

Usually no. Raw actions such as typed text, full page content, screenshots, and file contents are high risk. Store reduced summaries, task state, tool metadata, or redacted memory instead.

How is agent memory different from normal logs?

Logs are mainly used for debugging and operations. Agent memory may be retrieved by a model and used to generate future responses or actions. That makes purpose, consent, retention, and retrieval policy more important.

What data should never be stored in agent memory?

Do not store raw secrets, API keys, session cookies, private keys, payment details, or regulated personal data unless you have a very specific, compliant reason. In most products, these should be blocked or heavily redacted.

How long should AI agent memory be retained?

Retention depends on purpose. Task-resume memory may only need days. Support debugging may need weeks. Security audit records may need longer. Avoid one global retention window for every memory type.

Do prompts provide enough privacy protection?

No. Prompts can remind a model not to reveal sensitive data, but privacy enforcement should happen in code before storage, indexing, embedding, and retrieval.

How can users trust agent memory?

Give users visibility and control. Provide settings to disable memory, inspect saved memory, delete memory, and understand what the agent remembers and why.