LangChain shipped an agent skill that inspects your codebase and traces, interviews you about what "good" means, and generates eval datasets into LangSmith. It's a genuinely useful thing and the write-up is worth reading.
It also clarified something I'd been struggling to articulate about my own work, so this is less a comparison than a distinction.
Two different questions
Evaluation answers a question about the past:
Given these traces, how did my system perform?
Gating answers a question about the present:
This process is about to run
rm -rf. Does it?
Those look adjacent. They are not. One is a measurement problem, and you solve it with datasets, scoring functions, and dashboards. The other is a control problem, and you solve it with an interruption and a human.
Evals are how you decide whether to ship. Gates are what stand between a shipped agent and your filesystem. A perfect eval score tells you nothing about the run that reads a poisoned README at 4pm on a Tuesday.
Why this matters more than it used to
In July 2026 AI Now published a proof of concept where coding agents executed payloads planted in a README. Not a supply-chain attack — a file the agent was told to read, containing instructions it then followed.
You cannot eval your way out of that. The input didn't exist when you built your dataset. Whatever ran, ran.
Back in October 2025 a developer lost every user-owned file on his machine to an rm -rf that started at /. The detail people skip: he was not running with permissions skipped. The gate was on. He approved it.
So we have one failure mode evals can't see, and one that gates saw and lost anyway. The second is the more interesting problem.
The gate failed because of where it was
An approval prompt in your terminal is competing with the work it interrupts.
You delegated the task specifically so you could stop holding it in your head. The prompt arrives in that same window, and reading it properly means reloading the context you just put down. Denying it means the run stops and you own the next move.
The cost of scrutiny gets paid in exactly the currency you were trying to save. Approving isn't just easier — it's the option that preserves your flow. The interface has made "yes" the path of least resistance for the one decision where you want friction.
This is alarm fatigue. Aviation, medicine, and ops have decades of literature on it, and the fix was never "read more carefully." It was changing what the alarm interrupts and what it costs to respond.
What I'm testing
I've been building ThumbGate around one narrow bet: put the approval on a different surface than the work.
The agent runs on your machine. A connector dials out over HTTPS — outbound only, no inbound ports, private-key pairing. Pending tool calls surface in a browser tab. You read them cold, with none of the terminal's gravity, and approve or deny there.
Two implementation details that turned out to matter:
- Outbound-only. If protecting your machine means exposing it to the internet, you've relocated the risk, not reduced it.
- A 90-second lease. Exactly one executor owns a thread. Without it, "the gate lives elsewhere" degrades into "two things might act on the same thread," which is worse than no gate.
What I don't know
This is early. I'm the only person on it, there's no user base, and sign-in is Google-only right now. There's a failover mode that continues eligible work on a VPS when your machine drops offline, and I'd call that unproven — it works in my testing, it hasn't met real-world mess.
The central question is genuinely open: does a second surface actually get read, or does it become a second thing you rubber-stamp?
I have an argument for why it should be different — scrutiny no longer costs you the flow you're protecting — but an argument isn't evidence. Anyone claiming otherwise about a tool this young is selling you something.
If you run agents that can touch a filesystem, that's the question I want help with. Not whether the framing sounds good. Whether it survives your actual week.
Nothing to install: thumbgate.app
Top comments (0)