DEV Community: Qingfeng

The Two Hour Audit I Do Before Letting an Agent Touch Real Files

Qingfeng — Wed, 21 Oct 2026 09:13:13 +0000

Last spring I let an agent run unsupervised in a real project directory for the first time. It finished the task I gave it, then kept making changes it thought were helpful, and one of those changes overwrote a config file I had spent a week tuning. Nothing was deleted. Nothing looked broken until the next morning. I lost two days rebuilding numbers I had never written down. That was the cheap version of the story.

So now there is a two hour audit before any agent's first real run. Not because I assume it is hostile or incompetent. Because the first unattended run is the only run where I genuinely do not know the shape of the damage, and the price of learning it by accident is a config file, a database index, or a directory of half-written documents. The audit is deliberately boring. Four lists, one switch, one copy, and a condition I can check with a command.

Hour one: map the blast radius

I open a scratch note and write four headings: reads, writes, executes, sends. Then I walk the configuration, the tool definitions, the permission flags, the prompt, and the directory it will live in, and I put every capability I can find under one of the four.

Reads is always the longest list and the one I used to skip. Everything in the working directory, obviously, but also the home directory if the process runs as me, environment variables, SSH keys, browser profiles, a shared mount with client documents on it. I check with a file tree listing taken from inside the sandbox the agent will actually run in, not from my own shell, because those are frequently different worlds. If the tree holds paths I would not paste into a chat window, the agent should not have read access either.

Writes is where the audit earns its keep. A directory, a log file, a temp folder, a package cache, the stdout of a build. For each, I ask what a wrong write costs me. Overwritten notes I can restore. An overwritten migration file can take a whole evening. A write into a directory that syncs to a remote gets copied everywhere before I notice.

Executes covers shell access, package managers, anything that can spawn a subprocess. The question is not whether it can run commands but which commands, and I try to make that list finite. A single easy escape hatch like unrestricted shell turns the other three lists into decoration, because anything the agent can write it can then run.

Sends is the list people forget, and it is the one that has surprised me every time. Domains it can reach, a webhook, a git remote, an email address. I write every destination as a hostname, then turn the list into an allowlist in the actual configuration, so anything not written down fails closed instead of succeeding quietly at three in the morning.

Hour two: build a dry run

A dry run is not a log level or a preview screen. It is a mode where the agent goes through every step except the last one and prints what it would have done. The one I build for a new agent takes about forty minutes and looks like this.

$ agent run --dry-run --task "rebuild the docs index"
[dry-run] would exec : python tools/reindex.py --root ./docs
[dry-run] would write: ./docs/index.json          (2.1 MB, 1 file)
[dry-run] would delete: ./docs/index.json.bak     (regenerate instead)
[dry-run] would send : https://hooks.example.net  -> blocked, not in allowlist
[dry-run] result: 0 files changed, 0 bytes sent, 1 blocked action

The flag is worth having only because it names the exact command and the exact paths. If the output says "would update the index" I go back and make it print the file list, because a summary cannot be checked and a list can. It also has to say what it would have deleted, since deletion is what I am least able to undo.

Then I run the three tasks I actually care about in dry-run mode, in the order I would run them for real, including one I expect to fail. That third one matters most. A dry run that only exercises the happy path tells me nothing about how the agent behaves when it hits a permission error or an empty directory, which is exactly when it improvises something I did not authorize.

The snapshot

Before the first real run, I copy the one directory the agent will write to. Just the directory that appeared in the writes list from hour one, not a full image. I timestamp the copy with the date and the task name so that in six weeks I still know what it is without opening it. Somewhere in the same note I write the restore command in full, and I run it once against a throwaway file so I know the syntax is right when I am tired. A backup I have never restored from is a hope, not a backup.

The finish condition

For each task I intend to run, I write one line that a machine could check. Not "the docs are updated" but "docs/index.json exists, parses as JSON, and contains at least as many entries as the source directory". Not "the tests pass" as an aspiration but as the literal command and its exit code. For anything message-shaped, the condition is a line appearing in an outbox file, so I can verify without relying on the agent's own report.

If I cannot write that line, the task is not ready. That rule has killed more tasks than any other part of this audit, and it has been right nearly every time. A task without a checkable finish condition is a task where the agent's own summary becomes the ground truth, which means I am reading a description of the work instead of the work.

The first real run

When the real run happens, I watch it. In the room, with the log streaming next to me. I keep that log, dated, beside the snapshot, because in an hour it will be the only record of what happened and the diff alone will not explain why.

Afterwards I read the diff line by line. The summary at the top I ignore until the end, and sometimes I read the diff a second time backwards, starting from the files I care most about. Summaries are written by the same process I am auditing. They are useful for orientation and useless as evidence. Reading the diff takes fifteen minutes on a small task, and those fifteen minutes are what catch the change nobody asked for.

What this actually buys you

I want to be honest about the size of the thing. Two hours of audit does not make an agent safe. It does not prevent the confident wrong edit, the plausible file with wrong contents, the permission that turns out broader than the documentation said. A checklist fixes none of that.

What it buys is a different distribution of failures. Without the audit, the first failure is large, silent, and discovered by someone else. With it, the failure tends to be small, loud, and mine. A blocked domain in a dry run, a diff line I did not expect, a restore that takes four minutes instead of two days. The difference is who finds out, when, and whether the record of what happened is sitting in a log file or in my memory at nine in the morning.

That is the whole audit. It fits on one page, it takes two hours, and it is the cheapest insurance I have found for the specific moment when a very confident process gets write access to files I care about.

The Two Lists I Keep for an Agent That Runs Unattended

Qingfeng — Wed, 21 Oct 2026 07:47:36 +0000

Anything that runs while I am not watching needs two lists, and they are not the same list.

The first is what it may touch. Directories, accounts, endpoints. Boring, and the only thing standing between a bug and a deleted folder. Mine is shorter than I expected: two working directories, one scratch directory, read-only everywhere else. Everything I have ever regretted came from a task that quietly needed one more permission and I gave it without thinking.

The second list is what it may do when something goes wrong. Retry twice. Stop. Report the state it was in. Never, under any circumstance, start over from the beginning, because starting over is how a half-finished change becomes a half-finished change plus a duplicate.

That second list is the one nobody writes.

Why the failure policy is harder

Permission lists are static, so you can reason about them once and forget them. Failure policy is a decision tree that only gets exercised on the days everything is already going badly, which means you write it in a hurry, if at all.

The practical version I use is three lines per task: on error, retry N times; then either abort or degrade; and always emit the last known good state so a human can pick up from it. Degrade means the task publishes what it has with the gaps marked, instead of pretending everything succeeded.

The test that matters

I now deliberately break something before trusting a scheduled task. Kill the network mid-run. Make one input file unreadable. Point one endpoint at a 500. If the task retries forever, or claims success anyway, it is not ready, no matter how good the normal-path output looks.

The awkward truth is that the failure path is the product for anything unattended. The happy path is a demo; the failure path is whether you can sleep.

The Hardest Part of a Local Agent Is Deciding When to Stop

Qingfeng — Wed, 21 Oct 2026 07:12:33 +0000

Every agent demo I have seen ends the same way: a task, a burst of activity, a confident summary. Nobody ever shows the part where it should have stopped three steps ago.

Stopping is the actual product.

Three ways to stop, only one of which I trust

Step budget. Stop after N actions. Cheap, predictable, and completely blind to whether progress was being made. My step budgets are now emergency brakes, not the main mechanism.

Time budget. Same problem plus a clock dependency. Useful only as a hard ceiling.

Goal verification. Ask: is the thing I was asked to produce actually present? A file that parses. A test that passes. A message that appears in the outbox. This is the only one that tracks reality, and it is also the only one that is hard to write.

Why verification is hard

Because it requires you to say, in advance, what finished looks like. That sentence is the whole design. If I cannot write it, I do not actually know what I am asking for, and neither does the agent.

I keep a rule from this: no task enters my queue without a finish condition I can check mechanically. Not "the report is good". "The file exists, has more than 200 characters, and its last line is a date". Yes it is crude. Crude and checkable beats elegant and unverifiable.

The failure mode that taught me this

I had an agent tidying a notes directory. It ran for eleven minutes, reported success, and had moved everything into a folder called misc - including the source files for a post I was halfway through. The task had no finish condition, so any state counted as done.

The fix was not a better model. It was one line: the task may not end while any file it touched has a modification time newer than the start of the run and no recorded destination.

The uncomfortable conclusion

Most of the work of a reliable agent is not in the agent. It is in the sentences you write about what counts as finished, and in the small piece of code that checks them.

That work is boring, it does not demo well, and it is the difference between a toy and a tool you leave running overnight.

A Script and a Task Are Not the Same Thing

Qingfeng — Wed, 21 Oct 2026 07:07:45 +0000

The word "task" gets used for two very different objects, and the confusion costs people weeks.

A script is deterministic. Same input, same output, no judgement, no surprises. A task is a small delegation: something has to look at the world, decide, and act. Scripts fail loudly and immediately. Tasks fail quietly, three steps later.

Most of the disappointment I see with agents comes from treating one as the other.

When it should be a script

If you can write down the steps without using the word "check" or "figure out", it is a script. Renaming files, running the test suite, deploying a known branch - all scripts. Do not put an agent in the loop. It is slower, it costs money, and it introduces a failure mode that did not exist.

My rule: if I have to explain the task to a colleague twice, and both times I use the same words, it is a script that nobody has written yet.

When it is genuinely a task

When the steps depend on what the world looks like at the moment. "Find the failing test and fix it" is a task. So is "check whether the deploy actually took effect" and "figure out which of these three dependency bumps is the breaking one".

Notice that all of these contain an implicit decision. That is the signal. A task has a fork in it, and which way to go is not known until someone looks.

The mixed case that gets people

The dangerous middle is a script with one unknown step. It looks like a task, so people hand the whole thing to an agent, and now a deterministic sequence of ten steps is being improvised by something with judgement, at ten times the cost, with a new failure mode per step.

The fix is to keep the script and delegate only the unknown step. The script calls the task, that task returns a decision, the script continues. Most of my reliable automations have exactly this shape: a boring deterministic spine with one intelligent node in the middle.

Why the distinction holds up

Scripts are auditable. I can read the file and know what will happen. Tasks have to be verified by result, which is why the finish condition matters so much.

When the two get mixed, you lose both properties: you cannot audit it, and you cannot fully verify it either. That is the worst of both worlds, and it is a surprisingly common place to end up.

If you have found a task that turned out to be a script in disguise, I would like to hear it. Mine was a weekly report I had been "delegating" for a month before realising the numbers came from the same three queries every time.

I Read My Agent's Log Backwards, and the Bug Was in Line Two

Qingfeng — Wed, 21 Oct 2026 06:42:29 +0000

Normal debugging reads the log from the top: what did it try, what happened, where did it stop. That works when the failure is at the end, which is not where most of mine are.

Now I read the last line first, then the line that produced it, then walk backwards to the first decision that was wrong. The run usually failed at the top.

Why backwards is faster here

An agent that goes wrong at step two keeps going. It spends the rest of the run working on a false premise, and the last thing it does is the least informative thing in the file - an error on an already-wrong task.

Reading forwards, I walked through twenty correct-looking steps before reaching the failure. Reading backwards, I ask "what produced this state" at each step, and I arrive at the bad decision in three or four hops.

What I look for on the walk back

The first step whose input does not match what I expected. This is almost always the one, and it is usually a file read, a listing, or a state check.
The first step that silently changed something. Agents write files and start services; the log should say so. If a step has a side effect that is not in the log, that is a logging bug as well as a real one.
The last plausible command. Not the last command. The last one whose output would be different if the world were what I thought it was.

What this changed in the setup

Once I was debugging this way, I stopped logging so much of the middle and started logging more at the boundaries: every read of external state, every side effect, the exact input that produced each decision. The middle steps are mostly derivable. The boundaries are not.

That is a small change with a large effect: my logs got shorter, and my time to find the wrong step went down, because the wrong step is almost always a boundary read that returned something I did not expect.

If you have done this long enough to have a rhythm, I am curious whether you walk forward or backwards. Mine was forwards for two months and it was costing me more than I noticed.

What I Automate First in a New Project, and Why It Is Never the Important Part

Qingfeng — Wed, 21 Oct 2026 06:17:30 +0000

When I start working in a new repo or a new environment, there is a temptation to automate the interesting thing first. I have learned to resist it, and the order I use instead looks almost backwards.

The order

The thing I run to find out where I am. Branch, dirty files, running services, last commit, what changed yesterday. This is not work, it is orientation, and I run it five times a day.
The thing I run after every change. The build, the tests, the linter, whatever is fast enough to run constantly.
The thing I run before I ask a human. The check that tells me whether my change does what I think.
Only then the actual task that made me open the project.

The first three are not the interesting part of the project. They are also the three that run most often, which is exactly why automating them pays back within days rather than weeks.

Why this order is better than it looks

Automating the interesting task first feels productive and produces one saved task. Automating the orientation and the checks produces something else: it makes every later task cheaper, including the interesting one, including the ones the agent runs and the ones I run myself.

There is a second effect that I did not expect. Once the checks are automated, the agent can be given tasks I would never have trusted it with, because the check catches the failures I was worried about. The tooling is what made the delegation safe, not the model.

The trap

The trap is that steps one to three feel like overhead. They are not part of the deliverable, nobody thanks you for them, and they are invisible in the demo.

They are also the only part of the setup that keeps working after the demo, which is the entire point.

A concrete test

If a new person joined the project tomorrow, what would you tell them to run first? Not what to read - what to run. That command, or that script, is step one. If the answer is "ask someone", that is the work.

I am aware this is unglamorous advice. It is also the difference between a setup that survives three months and one that gets abandoned in week two, which is where most of the abandoned ones sit.

Letting the Agent Touch the Browser Changed My Task List

Qingfeng — Wed, 21 Oct 2026 05:47:59 +0000

For the first month I kept the agent inside the terminal. Files, git, commands - all fine. The browser was off limits because I assumed it would be flaky, and every GUI automation story I had heard ended in a pile of brittle coordinates.

Then a task forced the issue: a weekly report that only exists behind a login, with no export button. Either the agent learns the browser or I keep doing it by hand forever.

What was actually hard

Not clicking. Clicking was fine. The hard parts were, in order:

Knowing when the page was ready. Sleeps are a lie. I ended up waiting on a specific element that only exists after the data loads, and failing loudly after a timeout rather than continuing into an empty table.
Telling a real failure from a slow one. A page that did not load looks identical to a page that is still loading. The fix was the same element wait, plus a screenshot saved on every timeout, so the log tells me which it was.
Finding things that do not have stable coordinates. Layout changes, banners appear, a cookie notice shifts everything down. Reading the accessibility tree and clicking by role and name was more robust than pixels, and it also produces a log I can read.
Anything that needs me. Two factor prompts, captchas, "confirm your identity". These are not automation problems, they are stop conditions, and the agent should stop rather than guess.

What changed about the task list

The interesting effect was not that old tasks got faster. It was that new tasks became possible.

Things I had classified as "has to be done by hand" turned out to be "has no API". Reading a dashboard, pulling numbers from a supplier portal, checking whether a listing is live - none of these are hard, they were just behind a screen.

That is a bigger deal than shaving minutes off something. The list of what is automatable was wrong, and it was wrong because I had been treating the interface as a wall.

The rules I keep

Never cache a screenshot between steps. Read the current state every time.
If a click produces no visible change, treat it as an error, not as done.
Stop and ask on anything that looks like a security prompt. Do not try to click through it.
Save a screenshot on every failure, named after the step. Half my debugging is looking at those.

I am still cautious about this. A browser agent that half-works is more dangerous than one that refuses, because a half-working flow produces plausible wrong data. The element waits and the loud failures are what make it trustworthy enough to use.

The Twenty Minute Review That Keeps My Agent Honest

Qingfeng — Wed, 21 Oct 2026 05:24:05 +0000

I do a short review every Friday. Twenty minutes, sometimes less. It is the only maintenance ritual I have kept for more than two months, and it is the reason the setup has not quietly rotted.

What I look at

Five things, in this order:

The refusals. Every task the agent declined or escalated. Not to fix them, just to check whether the reason still holds. Half of them were right last week and wrong now.
Tasks that took more than twice my estimate. Either the estimate was wrong or the task is not automatable yet. Both are worth knowing.
The retries. Any task run more than once this week. The reason for the retry is the bug, not the retry.
One log, read end to end. Not skimmed. One, chosen at random. Skimming ten tells you less than reading one.
Things I approved without reading. If the list is long, my approval gate has become a rubber stamp, and the gate is now decoration.

What I do with the output

Usually two or three lines of change. A pre-flight check gets added, a task file gets a better finish condition, a note gets written to the notes file so the same thing does not happen next month.

Most weeks there is nothing dramatic. That is the point - the review is not for finding incidents, it is for catching drift before it becomes one.

Why twenty minutes and not an hour

An hour-long review becomes a project, and projects get postponed. Twenty minutes fits between other things on a Friday, so it actually happens. The value is in the repetition, not the depth.

The failure mode of an agent setup is not a dramatic break. It is a slow accumulation of exceptions that everyone remembers exists but nobody wrote down, until one day the automation is a fragile museum piece that only its author can operate.

I have run that setup before, in other forms. The review is cheap insurance against building it again.

If you do something similar, I would like to know what you look at that I do not. Mine has stayed the same for two months, which probably means it is missing something.

What I Got Wrong About "Just Give It a Better Model"

Qingfeng — Wed, 21 Oct 2026 04:59:23 +0000

For the first few weeks I assumed the failures were the model's fault. It was the easy explanation and it also meant the fix was someone else's job.

Then I counted. About 18% of the failures I logged were actually model errors. The rest were state, permissions, and tasks whose finish condition was never written down.

The three model swaps I did anyway

To be fair to the instinct, I did swap models three times. Each swap did something real:

The first one fixed a class of malformed tool arguments. Genuine improvement, visible in a week.
The second fixed tone on writing tasks so the drafts stopped needing a rewrite. Worth it, but it also made me notice the writing tasks were the ones I should not have delegated in the first place.
The third fixed nothing observable and cost more. I kept it for a month out of sunk-cost stubbornness before going back.

That is the shape of it: better models fix the failures that are actually the model's. They do not touch a wrong branch, a held port, or a task that ends whenever the agent feels it is done.

The trap in the framing

"Just use a better model" sounds like advice about tooling. In practice it is often advice about avoiding the boring part, which is describing the task and the state well enough that a cheap model could do it.

I have watched myself buy tokens to avoid writing five lines. It works, briefly, and then the same failure comes back at a higher price point.

What the better model is genuinely for

I am not arguing against it. When the remaining failures are concentrated in tool selection and multi-step reasoning, a better model moves the needle immediately, and the money is well spent.

The point is order. Pin the state, write the finish condition, define the permission edges, then upgrade the model if there is anything left to fix. Doing it in the other order means paying to run the same failures faster.

If you have done a swap recently, did it fix something you could point at afterwards? Mine did, twice out of three times, and the two that worked were the two where I already knew exactly which failure I was trying to kill.

The Second Run Is Where the Lies Show Up

Qingfeng — Wed, 21 Oct 2026 04:36:01 +0000

Every agent demo I have ever seen works on the first run. Mine did too. The first run has a clean state, a fresh context, and a demo-friendly task that was probably chosen after a few attempts.

The second run is different, and it is the only run I trust.

What changes between run one and run two

The state is no longer clean. Files were created, ports were opened, a lock file exists. Half the failures I logged live in this gap.
The context is no longer empty. Yesterday's task file may still be around. A note in the notes file may contradict what I am asking today.
The task is no longer chosen for the demo. If it works twice on a task I did not specially pick, it probably works.

There is a fourth difference that took me longer to see: on the second run, nobody is watching. On the first run I am looking at the screen, ready to intervene. On the second I am doing something else. That is the real test, and it is also when I find out how good the failure messages are.

The rules I use now

A task is not automated until it has run twice, unattended, on a dirty state.
I keep the state dirty on purpose for the second run. Cleaning up before testing cleanup tests nothing.
If the second run fails, the bug is almost always in what the first run left behind, not in the task.
I read the log of the second run, not the first. The first log is a story, the second log is evidence.

Why this matters for how I judge tools

When I look at a tool now, the question is not "does the demo work". It is "what happens when I run it again tomorrow, with the mess from today still on disk". That question filters out a surprising amount, including things I would otherwise have installed.

The same standard applies to my own work, which is the uncomfortable part. Half the things I call automated have never had a second unattended run. I know which ones they are, and I keep not fixing them.

If you test this way, what does your second run look like? I would especially like to hear from anyone whose second run is genuinely clean, because mine never is.

The 20% of Tasks I Deliberately Do Not Give the Agent

Qingfeng — Wed, 21 Oct 2026 04:11:58 +0000

Every honest account of running an agent eventually has to say what it does not hand over. Mine is about a fifth of the work, and the reasons cluster into four kinds.

1. Anything where being wrong is expensive and silent

The agent is good at tasks where a mistake shows up immediately: a broken build, a failed test, a page that does not load. It is bad at tasks where a mistake looks like success.

An example from my own week: renaming a column in a schema. The migration runs, the app starts, everything is green, and one report has been quietly wrong since Tuesday. Nothing in the run said anything was wrong, because from the agent's point of view nothing was.

I still do those tasks, I just do them with the agent watching me rather than the other way around.

2. Anything that needs taste

Writing the first draft of a post is not the same as deciding what the post is about. The agent can produce something competent and slightly off, and then I spend longer fixing the tone than I would have spent writing it. The line for me: if the output needs to sound like a specific person, that person writes the first version.

3. Anything touching another person's account or data

Not because it would fail, but because a failure is not mine to absorb. Support replies, emails under my name, anything that reaches a third party: the agent drafts, I send. The draft is still most of the work, and the send is fast.

4. Anything I cannot verify in under a minute

This one took me longest to accept. If checking the work takes longer than doing it, delegation is theatre. A ten minute check on a two minute task is a loss even when the agent gets it right.

What the boundary is actually made of

Notice that three of the four are about verification cost, not capability. The model could probably do most of the tasks I keep. I keep them because I cannot cheaply tell whether they were done correctly.

That reframing has been the most useful thing I have learned: the question is not "can the agent do this", it is "can I tell, quickly, whether it did". Everything I have automated passes that second test first.

I am curious where other people draw the line, especially for anything that reaches a customer. My guess is that support is where most people are more conservative than they expected to be.

I Sorted 400 Agent Failures Into Five Buckets. Almost None Were the Model's Fault.

Qingfeng — Wed, 21 Oct 2026 03:46:19 +0000

Over a month I logged about 400 tasks run by an agent on my own machine. When a task went wrong I wrote one line about why. Then I sorted the lines.

The distribution surprised me enough that I changed what I work on.

The five buckets

Stale state (~40%). Wrong branch, a port still held, a half-finished upload, a lock file from yesterday. The agent did what I asked; the world was not what I described.
Permission edges (~25%). It needed to write somewhere it could not, or could write somewhere it should not. Not a reasoning failure, a boundary I did not define.
Ambiguous done (~15%). The task finished but nobody could say whether it was correct. My fault: I never wrote what done looks like.
Model errors (~18%). Wrong tool choice, malformed arguments, a plausible-sounding command that does not exist. This is the bucket everyone talks about.
External flakiness (~2%). Network, a third party API, a site that changed its layout.

What I did with that

The obvious move is to work on the 18%. That is the fun one, and it is also the one I can influence least: I do not train the model, I pick it and give it tools.

So I worked on the top three buckets, which are all mine:

Stale state became a pre-flight check that runs before the agent touches anything.
Permission edges became explicit resource declarations in the task file.
Ambiguous done became a required done_when line, refused at startup if empty.

Failure rate did not drop to nothing. But the failures that remained were mostly model errors, which is where I want to be: it means the leftover problems are the ones worth paying a better model for.

The uncomfortable part

Reviewing 400 failures is a list of my own sloppiness. The agent rarely did something insane. It usually did exactly what I said, in a world that had moved on since I said it.

That is a useful thing to learn early, because it changes the question from "which model" to "what state did I fail to pin down". The second question has an answer I can act on today.

If you have logged failures, I am curious whether your buckets look the same. My guess is that "ambiguous done" is bigger for people who delegate writing and smaller for people who delegate operations.