DEV Community

Nathan C.
Nathan C.

Posted on

My agent kept saying "done." Now it has to prove it.

Two days, a dozen features, one rule: check it before you hand it over.

Every agent has the same favorite word. "Done."

The code is done. It doesn't compile. The spreadsheet is done. Cell C3 says #DIV/0!. The deck is done. Slide 4 is blank.

"Done" is the cheapest word a model can say. So this week, most of what I added to Flash, my local-first agent built on Ollama, makes it prove that word first.

Code isn't done until the tests pass

Flash now has a verify tool. It finds the checks a project already has (the test runner, linter and type checker in pyproject.toml, package.json, Cargo.toml, go.mod or a Makefile) and runs them.

When something fails, it doesn't dump 400 lines of log. It names the failing tests and the files and lines to look at, then tells the model to fix the cause, never by skipping a test.

If Flash changes code and tries to finish without checking, it gets one reminder. That's the whole mechanism: a nudge, not a cage.

/review got the same treatment. It looks over a change for bugs, then argues against each finding before showing it. Whatever doesn't survive its own cross-examination never reaches you.

Office files get opened, not assumed

Code that writes a spreadsheet can't see the spreadsheet. So check_office_file opens it the way you would: LibreOffice recalculates every formula and prints every page.

Then Flash gets two things:

  • Facts: Costs!C3 shows #DIV/0!, page 2 is blank, slide 4 is empty, "TODO" left on page 1.
  • Pictures of the pages, to catch what no parser can, like text spilling out of its box or a table cut off at the margin.

It fixes the script, makes the file again, and checks again. You get the file only once it's clean.

Show it once

Record your screen doing a task. Talk over it if you like. Flash pulls a still every time the screen changes, matches your words to each one, and writes the procedure as a skill: each step, what to click, and what the screen should look like after.

Next time, there's a Do it now button. It runs the skill, checks each screen against what the last step should have left, and stops to ask before anything that sends, pays, deletes or signs in. Any screen recording works now, not just ones made in Flash.

Chess, one move at a time

Ask Flash to teach you the Italian Game and you get a board in the chat. Flash's moves slide into place. When it's your turn, you drag a piece, and the board tells you instantly whether that was the move. No model call, no waiting.

Under it is a full rules engine in plain Python: castling, en passant, promotion, mate and stalemate. It produces exactly 8,902 positions three moves deep from the start, the number every chess engine is tested against.

That engine is also the model's fact-checker. Local models misremember openings. Now a bad lesson comes back as an error:

step 1: 'e5' is not a legal move for White here. Legal moves: Na3, Nc3, Nf3, ...

The model fixes the step and sends it again. You never see the wrong version. Don't ask a model to be right. Make wrong impossible to ship.

Your house, with a guard rail

Flash now talks to Home Assistant, so it reaches nearly any light, thermostat, blind or speaker without me writing a line per brand. "Kitchen lights to 30%, warm white" just works.

One exception: locks, alarms and garage doors. Those ask you every time, even in fully autonomous mode. An agent can dim your lights on its own. It can't open your front door on its own.

Everything else

  • Read it to me. "Read me this article" pulls out the article, skips the menus, and plays it in the chat with a waveform, speed control and download. Sound files play inline too.
  • 3D you can touch. Models Flash makes now have parts that tell you what they are on hover, and open when clicked. A cabinet door swings on its hinge.
  • A journal. Every night Flash writes up the day's chats: what you worked on, what was decided, what was left open. Weeks and months roll up, and it can search them when you ask what you were doing.
  • AGENTS.md. Flash follows the project instructions in the folder it works in.
  • Small things: /compact in the browser, drafts with code blocks shown in monospace, flash://open links, and a --dev flag on the installers.

Around Flash

Flash isn't alone anymore. Riptide, my animation editor, got real 3D (lit shapes, imported models, a camera) and a Flash extension: one click sends a model or picture from a Flash chat into the project open in Riptide, as a new layer. And Ember, the watch app that feeds ideas to Flash, now lets you edit a note right on your wrist.

The pattern

Looking back at the week, it's one idea, over and over. The model is good at writing. It's bad at knowing whether what it wrote works. So give it something that knows: a test suite, a spreadsheet engine, a rules engine, a human at the door.

Then let it try again.

Flash is open source and runs on your own machine: github.com/Natuworkguy/Flash. Ask it to teach you an opening. Then drag the wrong piece, just to see.

Top comments (1)

Collapse
 
mrsaynothing profile image
Mr Say Nothing •

The done lie is the whole game. We run an agent-run site, and the fix that held was a gate: every submission passes a mechanical checker first, then a fail-closed judge — unsure lands in a pending queue, never straight to published. Done became published or pending, nothing in between. The part that surprised us: zero-violation submissions still pend sometimes, and that is the feature — the gray band is where the trust lives. How does your verify step handle the case where the artifact really is fine but no existing check covers it?