DEV Community

T.M. Gunderson
T.M. Gunderson

Posted on

Your AI Agent's First Credentials Should Be Fake: The SMB Sandbox Pattern

The most expensive minute in small-business AI is the minute a newly connected agent first touches something real: the live inbox, the real CRM, the actual customer list. Everything before that minute is a free mistake. Everything after it costs trust, money, or a customer.

Most setup advice skips the step between "I signed up for this tool" and "I connected it to everything." Enterprises don't skip it. There's a cheap version of what they do, and it's worth borrowing before your next agent gets its hands on real credentials.

The pattern, from an unlikely place

Legora, a fast-growing legal-tech company, has a constraint most AI vendors never face: its CTO Jacob Lauritzen has said the company can't read a single trace of what its agent does for a lawyer, because the data belongs to the law firm's clients. So the team doesn't test on production at all. Instead, as Lauritzen described on the Latent Space podcast, a small team of three engineers and three lawyers builds test environments modeled on real client work — including a simulated data room holding 10,000 files — and the agent proves itself there first, against work that looks real but isn't.

The company's engineering write-ups describe the industrial version of this: every evaluation run boots the entire application inside its own isolated sandbox, pinned to one code version, and a typical experiment is 100 cases across 4 system variants with 5 trials each — 2,000 runs in parallel before anything counts as tested.

You will never run 2,000 parallel anything, and you don't need to. The part worth copying isn't the infrastructure. It's the principle: the agent proves itself in a copy of your business before it ever sees the real one.

Build a fake world in about 20 minutes

Here is the small-business version of the sandbox, using tools you already have:

1. Separate credentials for the test world. Before an agent touches your real systems, it gets test credentials: a free email address, a test user in your CRM, a sample company file in your accounting tool (most major platforms ship one), a payment provider's test mode. The rule is simple: the agent's first login is never your real one. If it does something absurd in the test world, nothing real breaks and no real person finds out.

2. Copy the shape, not the data. Create 15–20 fake records that look structurally identical to your real ones — fake names, fake addresses, fake dollar amounts, same fields, same messiness. Never paste a real customer's name, email, or invoice into the test world. Legora's 10,000-file data room is the enterprise scale of this; your 20 fake rows do the same job. What exposes an agent's failures is the shape of the work — messy inputs, odd requests, incomplete records — not the identity of the data.

3. Known answers only. Grade the agent against outcomes you already know. Last month's invoices, where you know the correct categorization. Your trickiest customer email, where you know what the right reply looked like. An old quote request where you know which jobs were actually profitable. If you can't verify the right answer yourself, the test tells you nothing.

4. Write the pass line before the first real connection. Decide in advance what "ready" means — for example: five clean runs in the fake world, where a clean run means the agent took the right action, invented no facts, and contacted nothing outside the test world. Write it down. The pass line is what stops "it seems fine" from becoming the standard, because "it seems fine" is how an untested agent ends up emailing a real customer a discount you never approved.

What the sandbox catches, and what it won't

Run an agent loose in a fake world and you'll catch the failures that matter most at zero cost: emails sent to the wrong place, facts the agent made up confidently, boundaries it stepped over because nobody stated them, records it edited that it should only have read. Every one of those is survivable in a fake world and expensive in a real one.

The honest boundary: a sandbox is a filter, not a proof. Your fake world only contains the failure modes you thought to include, and your real data will eventually produce edge cases your copies didn't. Isolation catches the obvious catastrophes early; it doesn't certify the agent forever. That's the point of the last piece.

Behavior changes overnight — re-run the pass

Lauritzen also described a citation-accuracy problem that had been stuck around 70% — until a single new model release, with no change to Legora's code, pushed it to 99%. The lesson cuts both ways: a vendor's model update can fix your agent overnight, and it can just as easily break something that used to work. You control none of it.

So the fake world isn't a one-time gate. Keep it. When your AI vendor ships a major model update — or quarterly, if you'd rather not track release notes — re-run the same known-answer tasks through the fake world before trusting new behavior on real work. It's the same 15 minutes, and it's the only cheap moment you'll ever get with a changed system.

Tonight

  1. Create the test credentials — email, CRM test user, sample company file.
  2. Build the 20 fake records with known right answers.
  3. Write the pass line: how many clean runs, and what "clean" means.
  4. Point the new agent at the fake world. Nothing real, yet.

Deciding which tasks are worth delegating at all is a separate exercise — this is the step that makes whatever you delegate safe to try. The agent gets real access on the day it earns it, in a copy of your business where its worst day costs you nothing.


Want the boring half pre-built? The Boring Automation Pack from SMB Scale Up includes plug-and-play automation blueprints plus the standing-instructions worksheet — the checklists and tracking files that keep delegated work inside lines you actually wrote down.

Top comments (0)