DEV Community: yim_rei

AI Data Privacy: A Three-Layer Defense for Using AI Without Leaking Secrets

yim_rei — Tue, 30 Jun 2026 14:17:59 +0000

One time we opened a daemon's log on the box, saw that some secret values were mixed into it, and decided to scrub them first with a short command that matched the pattern KEY= and TOKEN= and replaced each with the word redacted. We thought that was the end of it.

Then the text came out and two real key values were sitting right there in the transcript. The ones that leaked were named ANTHROPIC_API_KEY_FALLBACK= and CLAUDE_CODE_OAUTH_TOKEN_BAD=. Their names end in _FALLBACK= and _BAD=, not KEY= or TOKEN=, so the pattern never caught them.

The secret did not leak to some hacker. It leaked into our own transcript, a place we were not counting as "outside" at that moment, even though that is exactly what it is. A secret printed into a log has to be treated as leaked right away, and the key has to be rotated.

The lesson here is not that the command was written wrong. It is that data leaks at the seams you forgot to label, not at the spot you were already watching.

Part 1: A three-layer defense: data can flow down, but never up

The AI we all use day to day keeps its knowledge base on someone else's cloud. You ask it something and the model answers from the broad knowledge of the world. No problem there. Public knowledge flowing down to you is fine.

The trouble starts when data flows the other way. The moment you feed your private data up, the model is not just reading it to answer you. That chunk has already left your machine and now lives in someone else's system. So how do you know what is safe to send up and what is not? The approach we use is to split data into three layers.

The outer layer is the cloud, the world knowledge the model already has. Ask it general things freely. None of your own data is in there.

The middle layer is shareable work data: public documents, project context that is already open. If it flows up to the cloud, that is still acceptable.

The inner layer is secrets: financial data, health data, client data, credentials. This layer has one rule. It never flows up to the outer layer, period.

The whole point of these layers is one-way flow. Knowledge from the outer layer can flow down and help your work all it wants, but anything in the inner layer never flows back up. Once you set the direction like this, a hard question such as "can I feed this to the AI?" becomes a much easier one: which layer does this belong to?

Part 2: The leaks you don't think about

The most obvious leak is pasting secret text straight into an AI chat. Most people already guard against that. The sneakier leaks are the automatic "processing" steps you never see.

Take one close to home: the memory system our own agent actually runs, Graphiti. Each time it records a chunk of work, a small model reads the raw contents of that chunk to extract the facts. That "read to extract" step is exactly where data leaves the machine (egress). In the stack we actually run, that small model does its reading in the cloud, so your raw content has already gone up at that moment, even though all you did was tell it to "remember." You never meant to send anything up. That is why we treat "which model does the reading, and where it runs" as an inner-layer question.

The last one is logs and transcripts, like the leaked key at the start. The data does not escape to a criminal. It just pools in a place you never classified as the outer layer, even though that is what it is. Anyone who can read that log can see everything in your inner layer.

Part 3: Make it a real defense, not just an intention

Judgment that runs on instinct does not count as a defense. Three things turn it into a real one.

Separate data by layer at design time, not afterward. Building a system that holds real personal data taught us that separating data by purpose has to be structural from day one, because each kind of data has different rules. Some can be deleted the instant the owner withdraws consent. Some the law requires you to keep for 5 to 7 years. Pile them all on one table and the moment you have to delete, the whole pile breaks.

Set it to fail closed. If you cannot prove where a chunk of data will end up, assume it is going to flow up to the outer layer and block it. Do not wave it through because "no problem has shown up yet." It is the same rule as access control: if you cannot confirm who has permission, deny first. Safer than guessing and opening the door.

Filter data by what it is, not what it looks like. Go back to the leaked key. The root cause was filtering by the character pattern KEY= instead of by the meaning "this variable is a secret." Once you classify things by what they actually are, an odd name like _FALLBACK can no longer slip through the net.

The actual wiring that makes the three layers enforce themselves automatically is another part of the implementation we are holding back for now. But the three principles above you can use today, with no special tooling required.

Part 4: One question that works on any tool

Before you wire a new AI tool into your work, ask it a short question: where does the text I type end up, and which layer is that? If it cannot answer, do not feed anything from your inner layer into it yet. This question works on everything, from a plain chat to a memory system humming quietly in the background. Try it on the tools you already use every day.

Written from real work. Full version (and a Thai edition) at productize.life.

AI Coding Agents: The Expensive Part Isn't the Agents

yim_rei — Tue, 30 Jun 2026 14:15:12 +0000

I ran three AI coders in parallel last week, each on its own file. Three pull requests came back in minutes.

Then I checked the bill. It had barely moved.

We'd expected running several AI agents at once to burn a fair bit. The numbers said otherwise, and once we dug into why, we hit the thing that changed how we think about putting AI to work.

1. The coders aren't the real cost

Most people picture running a lot of AI codegen as a per-token bill that climbs and climbs, because the mental image is a metered API. But the coders we run don't connect that way. We run them through an agent framework that decides which engine drives each coder, where the engine is just the model doing the real work behind it. The engine we plug in logs in through a monthly subscription we already pay for, not an API key billed per token. So whether we launch one coder or three at once, the cost per round barely changes.

So where does the real cost go? To us. Early on we babysat the coders, ssh-ing in to check status every few seconds. That back-and-forth was what ate the resources, not the coders. The fix wasn't fewer coders, it was to stop babysitting and let a dispatcher hand out the work and close it off automatically. People set the problem and make the calls; people don't sit and stare.

Once you see that, running several coders in parallel stops being the extravagance you feared. The coders are cheap and replaceable. The expensive thing is human time and attention.

2. Cheap and parallel means you need a checker who isn't the writer

Once coders are cheap and you can launch several at once, the risk moves. It isn't about money anymore, it's about quality. Output that comes fast and in volume slips defects through just as easily if nobody checks first.

The path we chose: everything a coder writes goes through a separate reviewer first. The reviewer has one job, to read and flag, never to write. Because the moment the writer and the checker are the same agent, it glosses over the exact spots it just missed. Splitting the reviewer out gives you a second pair of eyes that isn't attached to what was just written.

What we'd like to do better: the reviewer should run a different engine from the writer, so it doesn't share the same blind spots. We tried switching the reviewer to a different engine, but it hung silently inside the system and never produced a single review, so we fell back to the engine we knew worked. A different engine is the right target, but something that actually works now beats an ideal that isn't stable yet.

3. A human keeps merge authority, and the work stays isolated

Coders running in parallel, a reviewer in between, and there's still one last line to draw: who presses merge into the main code. Our answer is a person, not any AI. The coder writes, the reviewer flags, but a human decides whether it actually goes in. We call that last gate Gate A, not because we distrust the AI, but because taking code into the main line is a hard-to-reverse decision, and someone should own that spot every time.

The other thing to set up the moment you run several at once: each coder needs its own workspace, not all writing over the same files. Let several coders edit the same space at once and the work tangles. That's the dispatcher's job, to fence off space for each one up front, not to patch collisions after the fact.

Start small: the shape you can copy

If you take one thing away, take this: running a fleet of AI coders isn't as expensive as you fear, because the coders aren't the real cost. The expensive things are human time and the defects that escape, and both are solved by the same shape.

Start small. You don't need the whole fleet on day one.

One coder, running on a subscription you already pay for, instead of a per-token API.
One reviewer that isn't the writer, reading and flagging first, every time.
A human holding merge authority, with a dispatcher handing out the rest of the work instead of you watching it.
Once that shape is stable, add coders one at a time. Because what lets you run a whole fleet without losing sleep isn't smarter coders, it's the checker and the human who still holds the decision.

Written from real work. The full version, with a Thai edition, lives at productize.life.

How to Make an Agent-Ready Website (and What to Refuse)

yim_rei — Tue, 30 Jun 2026 14:10:56 +0000

Recently I ran productize.life through a tool called isitagentready.com. It scans how ready your site is for AI agents to actually use, and the result was blunt and useful: the standard endpoints an agent checks to learn what a site offers (the /.well-known/... family) all returned 401, because they sat behind a password gate. In plain terms, an agent looking at the site saw nothing at all.

I spent hours fixing it. But the part worth telling is not "I ticked everything." Several items on the isitagentready list are ones you should not do for a content site. Once I sorted the real ones from the fake ones, I ended up doing 8, refusing 3, and skipping 1. This post is the reasoning behind each pile.

Part 1: A tool that scores how agent-ready your website is

First, a straight recommendation: isitagentready.com is a good radar. It sends the kind of requests an agent really sends, then tells you what your site is missing. Mine caught immediately that the homepage and the standard endpoints were hidden behind a password gate, so agents saw nothing. That was a real gap I did not know I had.

But every scanner shares one habit: it gives you a generic checklist that grades a large SaaS platform and a personal blog by the same yardstick. When that is the case, some items just do not fit your site. Ticking them all is not the goal. The goal is to do the items that are real for this site, and know which ones to skip.

Part 2: The 8 I did: one principle, point at something real

All 8 things I did share a single principle: everything you publish has to point at something that actually exists on the site. This blog already has that real thing, namely every article, all public:

Link headers (RFC 8288) tell an agent, right from the response header, where to go to read the list of resources

MCP server card + endpoint declares that this site has a read-only MCP server, exposing two real tools: list the articles, and fetch an article as text

Agent Skills index the full list of articles with a digest, ready for an agent to pick up

WebMCP exposes the same tools to an agent running in the browser

API catalog (RFC 9727) points at the MCP endpoint that actually exists

Web Bot Auth publishes a real public key (self-generated) so other sites can verify requests signed by us

DNS-AID publishes DNS-level records that point the way to these endpoints

Markdown for Agents if an agent asks for markdown it gets the article as markdown if a browser visits, it gets HTML as before

The last one has a useful detail. Cloudflare offers this feature, but locks it behind a paid plan. So I built it myself in code I already control. It is just content negotiation: look at the header the agent sends, and if it asks for markdown, convert the article and return it. No extra subscription. Knowing what you can build yourself for free is part of the judgment too.

Part 3: The 3 I refused: do not advertise capabilities you do not have

This is the heart of it. The scan pushed me to publish these: OAuth/OIDC discovery, OAuth Protected Resource, and auth.md for agent registration. All three ask you to expose files that announce the site has an "authentication server" where agents can register, request a token, and sign in.

But this blog has no such system. The MCP endpoint I exposed is deliberately public and read-only, with no login. If I published those files, an agent that trusted them would try to authenticate against a server that does not exist, and fail right there.

This is where a checklist fools you most easily. Ticking them all looks more agent-ready, but it actually makes the site less agent-ready, because advertising a capability that is not real wastes the time of, and breaks, the agents that trusted you. Refusing these three is what makes the site honest, not lazy.

Part 4: The 1 I skipped on purpose

The one I skipped is DNSSEC. The list recommends signing your DNS zone for security. I tried to turn it on, but hit a real wall. The domain productize.life is registered in one place, while its DNS lives in another (because the whole back end runs there). Our registrar supports DNSSEC only when you use its own name servers, so it cannot publish the signature for the current DNS provider.

The only way to earn this checkmark would be to transfer the domain registration entirely, which carries its own cost and hassle. For a blog where the DNS-AID records already work, moving the domain just to tick one box is not worth it. So I wrote down that I will revisit it only if I move the domain for some other reason. This is a "no" with a wake-up condition, not a permanent veto.

Part 5: The lesson: deployed is not the same as live

Along the way I hit a trap worth warning about. When I deployed, the system showed a nice green "READY", but the live page was still the old version. The reason was that the host the back end reads from was frozen on a build from 11 hours earlier the new deploy had landed under a different name and did not move with it.

The lesson is that "deploy succeeded" and "actually live" are two different things. What you can trust is the ground truth: send a request to the real URL and look at the bytes that come back, not a green status on a screen. The rest were small things, like returning markdown but forgetting it still carried a header saying it was compressed when it was not. I caught that because I tested locally before going live.

Use it tomorrow

Use a scanner like isitagentready as a radar it surfaces real gaps you missed

Everything you publish has to point at something that actually exists on the site

Never publish a file that claims a system (auth, registration) the site does not have a fake checkmark makes you less agent-ready

When you skip an item, skip it with a wake-up condition, not a permanent veto

Confirm with real ground truth, not a tool's "success" status

Where to start run your site through isitagentready.com, and before you do every item, ask one question: does this point at something I actually have? If not, skip it.

Written from real work. Full version (and a Thai edition) at productize.life.