DEV Community

Cover image for Ghost in the Machine: Building an Open-Source Privacy Boundary for AI Agents
hassanismailtech
hassanismailtech

Posted on

Ghost in the Machine: Building an Open-Source Privacy Boundary for AI Agents

AI agents are useful precisely where the stakes are high: customer support, financial operations, clinical workflows, and internal tooling.

That is also where they can accidentally send the wrong thing to an external model.

A bank-transfer agent may receive a customer’s name, phone number, account number, BVN, or NIN. A support agent may handle email addresses and sensitive tool results. Most AI integrations still treat the model request as the boundary: collect data, serialize it, send it upstream.

I wanted to invert that assumption.

Aegis is an open-source, inspectable data boundary for AI agents handling Nigerian identifiers. It sits between an application and an AI provider. Before a request leaves the application, Aegis’s Scrubber replaces detected sensitive values with session-stable Placeholders. The external Upstream sees only those Placeholders. When the response returns, the Reinjector restores known values for the calling application.

The goal is not to claim magic privacy or perfect detection. It is to reduce unnecessary data exposure while leaving evidence of what crossed the boundary.

This is the first public open-source release of Aegis under Apache-2.0.

The ghost I wanted to make visible

The “Ghost in the Machine” theme made me think about the data that silently follows an AI agent through a workflow.

An agent can look harmless on screen:

“Please initiate a transfer for this customer.”

But that sentence may carry identity data in the message body, an assistant tool-call argument, and later a tool result. If the agent sends all of it directly to a model provider, the sensitive information becomes invisible infrastructure: present in the request, absent from the product story, and difficult to inspect afterward.

Aegis makes that boundary visible.

Agent or application
        |
        v
     Aegis Proxy
        |
        +--> Scrubber --> Session Map
        |
        v
External Upstream receives Placeholders only
        |
        v
    Reinjector
        |
        v
Agent receives a usable response
Enter fullscreen mode Exit fullscreen mode

The critical distinction is that Aegis is not an irreversible redaction tool. An agent may still need a name or account number to complete a local workflow. Instead, Aegis uses a short-lived Session Map to preserve a controlled association between an original value and its Placeholder for the current conversation.

For example, an external model can receive:

Initiate a transfer to [[PERSON_1]] using [[ACCOUNT_NO_1]].
Enter fullscreen mode Exit fullscreen mode

The caller can still receive a coherent response after the Reinjector resolves known Placeholders. The external model never needs the original values to reason about the transfer flow.

Why Nigerian identifiers need first-class support

Generic PII tooling is useful, but production systems are contextual.

Aegis starts with a detector pack for Nigerian data patterns:

  • Phone numbers, including common +234, 234, and local forms
  • BVNs
  • NINs
  • Account numbers
  • Email addresses
  • PERSON entities, using a Nigerian name gazetteer and local GLiNER detection

The detector is deliberately layered:

  1. Pattern rules identify structured values.
  2. A gazetteer catches known Nigerian naming patterns.
  3. A local GLiNER model finds PERSON entities the earlier layers miss.

Overlapping matches are merged, with the longest match winning. A repeated original value receives the same Placeholder within a Session, which matters when an agent sees a customer in one turn and uses a tool in the next.

The open-source AI component is central here, not decorative. GLiNER performs local entity detection inside the Scrubber before a request is sent to an external Upstream. Aegis uses open tooling to decide what must not leave the building in its original form.

The hard part: agents do more than chat

A text-only demo would have been easy to make look convincing. Real agents made the problem more interesting.

Aegis scrubs:

  • Message text
  • Assistant tool-call arguments
  • Tool-result content
  • OpenAI-compatible streaming responses
  • Anthropic Messages API content and tool inputs

That means the boundary follows data through the places where agents actually move it.

A normal client integration remains intentionally small:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="your-upstream-key",
)
Enter fullscreen mode Exit fullscreen mode

The application points its client at Aegis. Aegis forwards a scrubbed request to the configured Upstream and returns a reinjected response where possible.

The same principle applies to streaming. Placeholders can be split across chunks, so Aegis uses a bounded hold-back buffer before reinjecting stream content. It does not assume that one network chunk contains one complete Placeholder.

What I chose not to promise

Privacy software earns trust by being explicit about its failure modes.

Aegis does not guarantee that every sensitive value will be detected. It does not make a data transfer lawful by itself. It does not claim “zero retention,” “zero knowledge,” or perfect recall.

Instead, it follows two conservative rules:

  • If the Scrubber or Session Map fails before an Upstream request, the request is blocked.
  • If reinjection cannot recover a Placeholder because the Session Map has expired or Redis has restarted, the response keeps the unresolved Placeholder rather than guessing a value.

That second choice can be inconvenient for an agent. It is also safer than manufacturing identity data.

The Session Map is intentionally ephemeral: Redis persistence and volumes are disabled in the supplied local stack, and its default sliding TTL is 300 seconds. That does not control an Upstream provider’s own retention policy; it simply prevents Aegis from turning temporary mapping state into a permanent database.

Making the boundary inspectable

A privacy boundary should be testable, not merely asserted.

Aegis provides a Wire Tap that records what the Upstream received after scrubbing. It never exposes pre-scrub request text or Session Map contents. Safe telemetry records counts, types, timings, IDs, and request status without recording sensitive bodies.

The included dashboard can show:

  • Detected-entity totals by type
  • Proxy overhead
  • Live safe telemetry
  • A selected Wire View of the post-scrub request sent Upstream

The demo uses synthetic data and a local mock Upstream. It is designed to make the contrast visible:

AGENT SAW:
  synthetic customer details

UPSTREAM SAW:
  [[PERSON_1]], [[BVN_1]], [[PHONE_1]], ...
Enter fullscreen mode Exit fullscreen mode

That is the moment I wanted the project to create: the agent remains useful, but the “ghost” of raw identity data is no longer silently travelling across the network boundary.

Evidence, not a victory lap

I measured Aegis on a project-authored, held-out synthetic evaluation set of 220 sentences and 500 labelled entities.

Here are the results:

Measure Result
Total recall 98.8%
Total precision 80.9%
PERSON recall 96.9%
PERSON precision 61.9%
Structured entity recall in this synthetic set 100%
Leak-test result 0 of 35 planted entities reached the mock Upstream

Those numbers need context.

The evaluation set is synthetic and project-authored, not an independent audit. PERSON precision is intentionally lower because Aegis prefers over-redaction to silently passing a likely sensitive name onward. The leak test uses the project’s own prompts, not adversarial third-party penetration testing.

Performance also depends on the path taken. On the development laptop, the pattern layer’s 60-word p50 was about 0.15–0.22 ms. Cold local model detection was much more expensive roughly 350–440 ms p50 on a 60-word prompt and long text can exceed the default two-second model timeout, which blocks the request rather than sending unscreened text Upstream. Cached and skipped model calls were far faster, around 0.4 ms p50.

That trade-off is the project’s real engineering tension: protect first, measure honestly, and fail closed when the protection layer cannot make a safe decision.

Why open innovation matters here

Aegis is open source because a privacy boundary should be inspectable by the people who depend on it.

Users should be able to examine:

  • What patterns are recognized
  • Which data reaches the Upstream
  • How Placeholders are created and restored
  • How failures behave
  • How measured recall and latency were produced
  • What the project does not claim

The system can run with a local mock Upstream for demonstration and testing, and its detection layer uses local open-source tooling. That makes it possible to inspect the actual boundary rather than accept a black-box promise about it.

For organisations handling Nigerian identifiers, open innovation is not only about being able to modify source code. It is about being able to ask a precise question:

“What exactly left our environment?”

Aegis is my attempt to make that answer concrete.

Run it locally

The repository includes a Docker Compose stack with Aegis, Redis, and a mock Upstream:

git clone https://github.com/hassanismailtech/Aegis.git
cd Aegis
cp .env.example .env
docker compose up --build
Enter fullscreen mode Exit fullscreen mode

Then run the test suite:

uv run --directory backend pytest -q
Enter fullscreen mode Exit fullscreen mode

For the held-out synthetic evaluation:

uv run --directory backend python scripts/eval_heldout.py --floor 0.85
Enter fullscreen mode Exit fullscreen mode

And for the benchmark:

uv run --directory backend python scripts/bench_scrub.py
Enter fullscreen mode Exit fullscreen mode

Please use synthetic data in local demos. Do not put real customer, banking, health, or identity data into a public test environment.

What comes next

Aegis is not finished, and that is part of why I am releasing it openly.

The next useful work is not to make broader privacy claims. It is to improve the evidence:

  • Independent evaluation data and review
  • More language and regional entity packs
  • Better calibration for PERSON over-redaction
  • Clear deployment guidance for in-country or customer-controlled environments
  • More adversarial leak testing

If you build agent systems that touch sensitive data, I would value the most specific feedback possible: what data shape would your agent accidentally send, and how would you want the boundary to behave?

Source code, tests, synthetic evaluation, and documentation are available in the Aegis repository.


References and acknowledgement

The wider community is actively exploring PII minimisation and privacy boundaries for AI systems. Two useful perspectives while building Aegis were:

Aegis takes a different approach: Nigerian-aware detection, session-stable reversible Placeholders, coverage for tool calls and tool results, and a Wire Tap that makes the post-scrub Upstream boundary reviewable.

Top comments (1)

Collapse
 
shuttlewright profile image
Shuttlewright •

The session-stability choice deserves a test of its own, in both directions. Within a session, two different customers must never collapse into one placeholder - that is an attribution bug the model will reason on confidently. Across sessions, the same customer must never keep one placeholder - that is a durable identifier quietly reintroduced. Planted fixtures pin both: a two-customer conversation and a repeat-customer-across-sessions conversation, run in CI. Second, on the Nigerian identifiers specifically: recognizers tend to be strongest on the formats their maintainers test most, so BVN and NIN want their own planted recall measurements per type rather than a borrowed average from more common formats. The placeholder architecture is only as strong as the detection in front of it, and detection is where the silent misses live.