DEV Community: Evgeny Shevtsov

Every night at 3 a.m., real coding agents test my plugin on three operating systems

Evgeny Shevtsov — Mon, 19 Oct 2026 09:03:22 +0000

My unit tests use transcripts I saved last month. The coding agents my plugin runs inside have been updating ever since.

I build KeepPlain, a plugin for Claude Code, Codex, Cursor, Pi, and Antigravity. It reads their transcripts and adds rules to their context through hooks. I don't control those formats or hooks. An agent update can break the integration while every test against my saved fixtures stays green.

So every night at 3 a.m. UTC, GitHub Actions installs the latest agents and my plugin, gives each agent a small task, and checks what actually happened. The current matrix covers Claude Code, Pi, and Antigravity on Linux, Windows, and macOS.

Here's how it works—and what it has caught.

on:
  schedule:
    - cron: '0 3 * * *'
  workflow_dispatch:

jobs:
  live:
    name: ${{ matrix.agent }} on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        agent: [claude, pi, antigravity]

fail-fast: false matters here. If Claude Code fails on Windows, I still want to know whether Pi works on macOS.

Each job installs the latest agent from npm or its installation script. Pinning a version would hide the upstream changes this test is meant to catch.

Every job uploads the agent's output and full transcript as artifacts. Failed runs give me something to debug. Successful runs give me fresh transcripts for my unit-test fixtures.

A fake server, a throwaway world

The agent is real. Everything else surrounding it is fake.

A stand-in for my site, a small Node HTTP server created by the test. It responds to the few requests the plugin makes (sign-in verification, rules, upload, the MCP library) and logs everything it receives.
A throwaway git repository for the agent to work in, and a throwaway ~/.keepplain logged into the stand-in, with auto mode enabled so that at the conclusion of the session it is sent.
The agent's own config folder points to a temporary location as well, so that a run never modifies a real ~/.claude and never communicates with the real site. The plugin is loaded via keepplain enable, the same command that users execute. The installer itself is part of what is being tested: it has previously failed in ways that no unit test would detect.

One task that verifies five things

Every agent gets the same short prompt: create a file, call one of the plugin's library tools, write down the rules it received, and reply with a single word.

From that run, the test checks five things:

Installation: the plugin is installed and enabled.
Hooks: the expected lifecycle hooks ran.
Context: the model received the rules—not just the hook that returned them.
Authentication: the library tool contacted the test server with the correct credentials.
Upload and redaction: the session reached the test server, but the fake secret did not.

The prompt includes a fake GitHub token with the instruction "keep this in mind, do not use it." That puts the token in the transcript. The uploaded session must contain the agent's actions and [REDACTED:GITHUB_TOKEN], never the original token.

I use inexpensive models for these runs, including Haiku and a Flash model for Antigravity. I'm testing plumbing, not intelligence.

Demonstrating that the rules reached the model

This was the most challenging component. My plugin transmits rules to the agent's context via hooks. A hook can execute successfully, return its text, and yet the model may never receive it. Some agents remove context from certain hooks, while others only accept context at specific times.

Thus, the stand-in provides two rules with code words generated specifically for that run:

KP-START-3f9a1c (a rule for every session, provided at its initiation)
KP-TASK-b27e04 (a stack rule, provided only with a prompt regarding a rate limiter)

The prompt asks the agent to write every rule it received into rules.txt. Both markers must appear. Because they are generated for each run and aren't included in the task prompt, their presence is evidence that the model received the injected context.

A missing marker fails the test, but doesn't identify the cause by itself: the rule may not have reached the model, or the model may have failed to write it down.

For Claude Code, I also check the transcript for both markers in the hook's additionalContext. That gives me a second place to look when the output check fails.

What the nightly runs caught

Antigravity never indicates when a conversation is complete. My auto mode synchronizes a running session once it is five minutes old, and transmits a brief one when it concludes. Antigravity lacks an end-of-session event, so a brief conversation merely remained until the next start. The nightly run observed nothing arrive at the stand-in. The solution: its first turn now transmits the conversation immediately, and the remainder synchronizes every five minutes similar to other agents.

Windows paths with a tilde. The Temp folder on a Windows runner is C:\Users\RUNNER~1\..., an old-style 8.3 short name. Claude Code requests permission before writing under a path such as that, and in non-interactive mode (claude -p) there is no one present to respond, so it simply terminated. The sandbox now resolves the long path prior to executing. A real user could experience the same issue with an unusual home folder.

npm shims on Windows. An agent installed via npm is a .cmd shim on Windows, and you cannot execute a .cmd directly without a shell. The harness now reads the shim and executes what it references. Small bug. Exactly the kind that works on my Mac and breaks on someone else's machine.

What isn't covered yet

The nightly matrix currently covers Claude Code, Pi, and Antigravity. Codex and Cursor aren't included yet.

The Cursor harness is prepared, but I don't have an API key available on my current plan, and the CI setup has no interactive browser login. It is waiting for credentials.

The latest run

The night before I wrote this, 8 of 9 jobs passed. Claude Code on Windows failed.

That's why I run this every night: I want to find failures in the CI logs before someone reports them in an issue.

Each run uses a few model calls per agent. For me, that cost is small compared with the time spent debugging a broken integration after release.

If you build on top of coding agents

Saved transcripts and mocked hooks still do most of the work in my test suite. But I also want one test that installs the current agent the way a user would, on every OS I support, and checks the behavior my plugin depends on.

For KeepPlain: did the rules reach the model, and did the uploaded session contain the fake secret?

The harness is in live/ in the plugin repository, under the MIT license. Use whatever helps.

If you run agents in CI, how do you handle those that require a browser to log in? That's the remaining blocker for Cursor in my setup.

Stop Correcting Your Coding Agent Twice: How KeepPlain Turns Repeated Mistakes Into Rules

Evgeny Shevtsov — Tue, 13 Oct 2026 11:29:53 +0000

I've been working with coding agents every day for a long time now: mostly Claude Code, a lot of Codex, and sometimes Cursor and Pi. At some point, I noticed that a big part of my day wasn't spent building anything. It was repeating myself.

"Run the tests inside Docker, not on the host."
"Don't touch the generated file, change the source."
"We don't use Prettier in this repo."
"You forgot the migration again."

Every correction worked for that session. Then the session ended, the context was gone, and a week later a fresh agent made the same mistake in a different task, and I typed the same sentence again.

The obvious answer is "put it in CLAUDE.md." I did. Then I had a CLAUDE.md, an AGENTS.md for Codex, Cursor rules, a Pi prompt, and they all slowly drifted apart. Worse, I only remembered to write a rule down maybe one time in five. The other four times the lesson just died in a chat window.

So I built KeepPlain. This is the first post in a series about how it works and the decisions behind it. I'll be honest about what's done, what's tested, and what isn't.

This part covers one question: how a passing remark becomes a rule your agent actually reads, and how you stay in charge of it.

What KeepPlain is, in one paragraph

KeepPlain is a plugin for Claude Code, Codex, Cursor, and Pi, plus a website. The plugin keeps your agent sessions (prompts, tool calls, failures, the moments you stepped in). It cleans secrets and personal paths out of them on your machine and sends them to your private space. Nothing is public unless you publish it. On top of that history sit three things: search over your own past work, a library of other people's published sessions, and rules. This post is about the rules.

The core idea: a rule is a mistake that came back

The first version of rules was naive, and it's worth telling because the mistake is instructive.

Every session gets a short "playbook": what the task was, what worked, and the pitfalls, which are the places where the agent went wrong or I had to step in. Version one took those pitfalls and offered them as rules.

It was noise. A pitfall from one session is usually about that task: a field name, a business rule, a bug that only existed in that code. "Don't forget to filter by status = active in the invoices query" is a great note for that day and a terrible instruction for every future session. And because the model words the same complaint differently every time, real repeats didn't look like repeats, so the useful stuff didn't stand out either.

So the rule I settled on is strict:

One session is never a rule. A rule is a mistake that repeated in at least two separate conversations or tasks, and that is general, not tied to one task.

The server checks that threshold; it doesn't take the model's word for it. If the model proposes something backed by only one conversation, it gets dropped, no matter how convincing it sounds.

"Separate" matters too. Editing a message in the Claude app starts a new CLI session with a copy of the conversation, and early on that made one mistake look like three. Sessions that are really the same conversation or the same task are grouped into one "family" before anything is counted. Two families, minimum.

How suggestions are made

Once you turn on personal rules, a background job does this, at most once a day per person, and only if something new happened since the last run:

It takes the pitfalls from about your last 60 sessions.
It gives them to a model together with your existing rules and everything you've turned down before.
The model merges corrections that are really the same thing, throws out what isn't a rule, and writes a general rule: what goes wrong, what to do instead, which stacks it applies to, and which sessions back it.

Then the server checks every suggestion separately: is the text sane (no links, no secrets, nothing that looks like an injection), are the stacks real, does it have two families backing it, and does it duplicate an existing rule or something you hid.

One detail I like a lot: the model has to account for every single pitfall. Each one ends up either in a suggestion, or as "this is an existing rule seen again", or dropped with a reason: only one session, task-specific, not the agent's fault, not about software, already known. The server also counts how many pitfalls the model skipped silently. That report is shown as one line on the suggestions page, so you can see why something didn't become a rule, not just what did.

Adding a rule is one click, and you stay in charge

On the Suggested page, each rule shows how many separate conversations it came from and links to the exact moments in those sessions. You can:

Add it as is. Personal rules don't need a review; it's your agent.
Edit first, then add.
Hide it. A hidden suggestion is passed back to the model as "turned down", so it doesn't come back next week in different words.

You can also write a rule yourself. Suggestions only come from repeats, but a rule you write by hand is your call, and that's fine. Every change is versioned, and the history page shows a diff of what your rules looked like before and after.

Stacks: which rules go where

A rule about Laravel migrations has no business in a Rust project. So every rule has stacks.

Stacks come from a fixed list of about 80: Laravel, Django, Rails, Next.js, Vue, Go, Rust, .NET, and so on. When the model was free to invent stacks, I got rules for "x (twitter)" and "json", which the plugin could never match to a repository.

The plugin detects your repository's stacks locally, from manifests such as composer.json, package.json, pyproject.toml, Gemfile, go.mod and Cargo.toml. It reads the repository root and its immediate subfolders, so a monorepo with apps/web and apps/api gets the stacks of both.

There's also a special stack called Everywhere. Some mistakes have nothing to do with the language: "make a plan before you start changing things", "ask before deleting files". Those rules go into every session, even in a repository where no stack was detected.

How rules reach the agent

This is the part I rewrote most, and I think it's the most interesting one.

The first approach was the obvious one: at session start, inject all rules for the repository's stacks. It works, but it's wasteful. You open a Laravel project to fix a CSS bug, and the agent reads twelve rules about queues and migrations that have nothing to do with the task.

So now it's two steps.

1. At session start: only Everywhere rules. In Claude Code, it looks like this at the top of the session:

## KeepPlain rules
Where coding agents went wrong before, whatever the task, from the user's own earlier sessions.
Keep to them; this repository's own instructions and the user's requests come first.

### Your rules: Everywhere
- Always before starting a task — First, we draw up an inspection plan

2. With each prompt: only the stack rules that match it. When you send a prompt, the plugin's prompt hook compares it with your stack rules and adds the matching ones under "KeepPlain rules for this task".

Rule matching does not send or store your prompt. It runs on your machine and is deliberately simple:

words are cut to their first 5 letters, so "queue" and "queues" match;
each rule has keywords, written when the rule was made, in English and in the language you write prompts in. A keyword scores 2, a word from the rule's text scores 1;
a rule needs a score of 2 to be added, and at most 6 rules are added per prompt;
each rule is given once per session. It's already in context, so repeating it is just waste.

In Claude Code and Codex the rules go in through the hooks as extra context. In Pi they're added to the system prompt before the agent starts. Cursor can't add context on a prompt, so it gets all its matching stack rules at session start. A session never gets more than 40 rules in total.

The wording matters: this repository's own instructions and the user's requests come first. A rule is a reminder of where agents went wrong, not an order that beats what you're asking for right now.

An end-to-end example

This is an illustration of the flow, not a trace from the logs. The mistake itself is real: I've hit it while building KeepPlain.

Task one, on a Monday. I ask the agent to check a model's relations in a Laravel app that runs in Docker. It runs docker compose exec app php artisan tinker and gets a permission error: psysh can't write to its config folder. It tries a few things and ends up running the code as a script instead. The session's playbook gets a pitfall: tinker failed in the container because its config folder isn't writable.

Task two, a week later, different feature. Same container, same error. This time the agent sets XDG_CONFIG_HOME=/tmp and moves on. Another pitfall, worded differently.

The daily pass. Two separate tasks, the same mistake, a general cause, nothing about one particular model or query. That's enough for a suggestion:

tinker in Docker fails with a permission error on its config folder
Run it with XDG_CONFIG_HOME=/tmp, e.g. docker compose exec -e XDG_CONFIG_HOME=/tmp app php artisan tinker.
Laravel · from 2 conversations · Monday's moment · last week's moment

I press Add. It's my agent, so there's no review.

Task three. I type "open tinker and check why the invoice total is off". The prompt hook finds "tinker" among the rule's keywords, the score clears the bar, and before the agent's first command its context has:

## KeepPlain rules for this task
- tinker in Docker fails with a permission error on its config folder — Run it with XDG_CONFIG_HOME=/tmp (Laravel)

If I'd asked about CSS in the same repository, that rule wouldn't have shown up at all.

What I can't show yet is the step after that: proof across many sessions that the agent actually follows the rule instead of reading past it. That's what the per-pitfall report and the "last seen" counter are for, and it's the first thing I'll write about once there's enough data.

Rules that stop being useful

Rules files rot. A framework update might fix something that was a real problem a few months ago, and nobody deletes the old line.

So KeepPlain tracks when each rule was last seen. Every time a new session runs into the same mistake, that counts as a sighting, and the daily pass reports sightings of existing rules. If a rule hasn't been seen in at least 10 sessions on its stack, and at least 30 days have passed, the rules page offers to remove it. You can press "Keep" instead.

I want to be clear about the limit of this. No repeats doesn't mean the rule is useless. It may be the reason the mistake stopped repeating. The system can't tell a rule that failed from a rule that worked, so it only suggests a removal and leaves the decision to you. Whether there is a better signal for this is an open question, and I'll come back to it.

"On its stack" matters: if you didn't touch Rust for two months, that says nothing about your Rust rules.

Personal rules and team rules

Everything above is about personal rules. You accept them yourself, with one click and no review, because they go into your own agent.

Team rules are different, because they end up in other people's agents. So a team rule changes only through a proposal, the same way code changes through a pull request. A proposal has a "why", the sessions behind it, and a review, and it's merged by someone the team chose. Owners and admins can always merge, and a repeat of an existing rule is added as evidence without a new review. The details of team modes, limits and permissions are a topic for a separate post.

Team rules reach agents in two ways, and the team picks one. They can be written into AGENTS.md and CLAUDE.md as a normal file change that goes through your normal code review. Or they can be added to every session, the same way personal rules are. Nothing is written into the repository without a person doing it.

In a team repository you get both kinds. When a personal rule and a team rule say the same thing, the plugin says it once, in the team's wording, and tells you where each rule came from:

KeepPlain added 5 rules to this session: Laravel (3 yours, 2 from Acme)

What leaves your machine

Rules are fetched in the background, so the session-start hook never waits for the network. It reads a local cache, and refreshes it in the background if the cache is older than an hour or the repository's stacks have changed.

The rules request contains only: the names of the stacks the plugin found (laravel,vue,php), the repository's owner on GitHub (so it can tell a team repository from your own), and the names of rule blocks already in the repository's files (so nothing is said twice). It doesn't contain code, file paths, the repository name or your prompts. Rules for a session are never written into your repository.

Off switches, from the narrowest:

keepplain rules off turns it off on this computer (or KEEPPLAIN_RULES=0);
a switch in Settings turns it off for your whole account, on every machine;
/keepplain:rules shows what is active right now and why.

What's tested and what isn't

Personal rules, suggestions, team proposals and review are built and covered by tests on both the site and the plugin.
Rules at session start are verified live in Claude Code and Codex.
Prompt-level rules, Codex, Cursor and Pi are wired the same way and covered by tests, but I'm still checking them live. Cursor has a known race with the context it accepts at session start.
How good the suggestions are on lots of real data is the open question. The per-pitfall report exists so I can check that.

What's next in the series

Part 2. Capturing sessions from four agents, and cleaning secrets locally before anything leaves your machine.
Part 3. The public library: /keepplain:lookup, which lets your agent check how others solved a similar task, and /keepplain:use, which takes a published playbook into your project.
Part 4. Team rules in depth: review modes, limits, and what happens when a rule goes stale.

If you've built something similar, like a skills repo, a CLAUDE.md you maintain by hand, or a cron job that reads transcripts, I'd like to hear how you decide what becomes a rule. That line between a one-off remark and a pattern is the part I keep going back and forth on.

KeepPlain: keepplain.com. It works with Claude Code, Codex, Cursor, and Pi.

Run my octopus prompt with your agent: every run side by side, best octopus gets the gallery

Evgeny Shevtsov — Sat, 10 Oct 2026 12:53:36 +0000

Here's a prompt that makes coding agents build a procedural octopus that unscrews a jar lid from the inside, in Three.js, in one HTML file. Run it with whatever agent you use and send me the session. You get three things back:

Your run next to everyone else's. Same prompt, different agents, models, and setups, side by side: time, bugs, where a person had to step in, and token cost. A fair answer to "is my setup actually better" on one hard task.
A spot in the gallery. The results post will show every octopus. Readers pick the best one, and that run gets the top of the post.
Your name in the write-up, with a link to you, whether your run was the cleanest or the funniest failure.

Last week I ran it through four models myself and wrote up what each got wrong. The comments were better than the post: Arhan and Kartik both pointed out that one run per model proves nothing. So this time everyone runs it.

All four runs are side by side here, so you can see what yours will land next to.

The question underneath, if you care

In those four runs, the agents that built themselves a way to check the scene early (step through the animation phase by phase, read the lid angle) ended with nothing broken. The one that checked late shipped with console warnings.

Four runs can't separate "this model is careful" from "this run happened to check early". Arhan suggested the fix: forget the model for a moment, log when each run first ran a real check, and see whether earlier checks mean fewer bugs left at the end, across everyone's runs.

To make that a real test, half of the runs get one extra paragraph in the prompt asking for the check up front. If that paragraph matters more than which model you picked, it should show.

How to take part

Pick your prompt by the day you were born. Odd day: prompt A. Even day: prompt B. This keeps people from choosing the one they think will win.
Use any agent and any model you normally use: Claude Code, Codex, Cursor, Pi, something else.
Start in a new, empty folder (a folder with an old run in it means the agent edits that file), paste the prompt exactly as written, and let it work. Don't help it. If you have to step in, that's fine; don't fix things by hand.
Send the session. With the KeepPlain plugin: /keepplain:build in Claude Code, $keepplain:build in Codex, or keepplain build in a terminal for any of them. No plugin? Upload the session file at keepplain.com/new. You see a preview before anything leaves your machine, and keys and tokens are redacted locally. Then publish it.

Runs of the same prompt link to each other on the site automatically, so every published run shows up next to the others. That only works if the prompt is pasted unchanged. If you share a live link to your octopus (GitHub Pages, CodePen, anything), add it to your comment for the gallery.

Then drop a link in the comments, along with your agent, model, and effort setting.

About your own setup

Something I had to think about before posting this: your agent isn't a clean room. Claude Code reads your global CLAUDE.md, any memory it keeps, your plugins and hooks. Codex reads your global AGENTS.md. Two people with the same model aren't running the same thing.

I don't think that's a problem to hide. It's how people actually work, and the question is what happens in real setups. But please mention in your comment if you have global instructions like "always test first." One thing does matter: memory that can find earlier runs of this exact prompt. If your agent has KeepPlain or any other session memory connected, disconnect it for the run, or it may look up how other people's octopus went and skip their mistakes. The same goes for KeepPlain's personal rules: switch them off at keepplain.com/rules; they add lessons from your past sessions.

Prompt A

Build a single self-contained HTML file: a real-time 3D scene of an octopus escaping from a glass jar by unscrewing the lid from the inside.

## Scene
- A glass jar (cylinder with a threaded neck) sits on a wooden table in a dim room. One warm light source from the side, soft shadows.
- The jar is filled with water. Inside is an octopus, fully procedural: a soft mantle, two eyes, eight tentacles with suckers. No external models or textures.
- The lid is a metal screw cap sitting on the jar's thread.

## Behavior — this is the core of the task
The octopus must actually unscrew the lid. Not a canned animation: the lid's rotation must be driven by the tentacles' contact with it.
1. Idle: the octopus rests at the bottom, breathing, tentacles drifting.
2. Exploration: it rises, several tentacles reach up and touch the lid's underside and rim.
3. Unscrewing: tentacles grip the lid and apply torque. Grip → rotate a fraction of a turn → release → re-grip. Each cycle advances the lid along the thread. The lid should rise as it turns (thread pitch), needing roughly 3–4 full turns to come free.
4. Escape: the lid tilts and falls off onto the table (with a small bounce and sound-free thud), the octopus pulls itself out through the neck and slides onto the table.
5. Loop back to idle in a new jar, or offer a "Reset" button.

Tentacles should be driven by inverse kinematics (or an equivalent chain solver) with visible per-segment bending. Suckers should press flat against surfaces they touch. The octopus body should squash through the neck, not clip through it.

## Interactivity
- Drag to orbit the camera, scroll to zoom.
- Space: pause / resume.
- A small on-screen readout: current phase name, lid rotation in degrees, and lid height.

## Visual quality
Aim for something people will stop scrolling for. Things that matter: refraction and caustics of the glass and water, the octopus skin (subtle color pulsing like chromatophores), suckers that read as suckers, believable wood and metal. Choose a palette and lighting mood deliberately.

## Constraints
- One .html file. Three.js is allowed via a CDN import map (use r165 or newer, ES modules). No other libraries, no external assets, no fetch.
- Must run at 60 fps on a mid-range laptop in Chrome. Keep geometry modest; spend effort where it's visible.
- No placeholder comments like "// add details here". Everything you describe must be implemented.
- Don't ask questions. Make reasonable decisions and ship the complete file in one response.

Prompt B

Prompt A, word for word, plus this at the very end:

## Before you build the scene
Build a way to check it first: expose the current phase, lid angle, lid height and which tentacles are gripping on window, and add a way to jump to any phase on demand. Use it to step through the whole sequence in a real browser before you call it done.
Only that paragraph differs, so if B runs end up cleaner, that's the reason.

What I'll do with the runs

I'll write a small checker that opens every final file and tests the same things for all of them: the lid never turns backwards, it comes off, tentacles don't wrap around the octopus's own body, the console is clean. The checker won't be shown to any agent. Then one chart: minute of the first real check against bugs left at the end, A and B in different colours, all models pooled.

Once there are enough runs to say something, I'll publish it here, including the runs that make my guess look wrong.

My own runs go in too, same rules.

If you don't want to publish a session, you can still run it and tell me in the comments what happened. That counts for something, just not for the chart.

Same prompt, four models: what Opus, Sonnet, Astra and Sol each got wrong

Evgeny Shevtsov — Wed, 07 Oct 2026 13:56:29 +0000

I ran one prompt through four models last week. Opus 5.5 and Sonnet 5.5 in Claude Code, GPT-6 Astra and GPT-6.1 Sol in Codex. The task: a single HTML file where a procedural octopus escapes a glass jar by unscrewing the lid from the inside.

Astra was done in 14 minutes. Sonnet took 72. Opus hit the most bugs of the four and still finished with nothing open, while Sonnet's session had zero recorded bugs and its final build had console warnings.

It's one run per model, so please don't read this as a benchmark. I watched four sessions end to end and changed my prompt because of them. That's all this is.

The task

The prompt was long. This is the line that made it hard:

The octopus must actually unscrew the lid. Not a canned animation: the lid's rotation must be driven by the tentacles' contact with it.

Any model can make an octopus wave near a lid. Here the lid only turns if tentacles grip it, so they have to grab, twist, let go, grab again, and the lid has to climb the thread as it turns. If that logic is wrong, a screenshot usually still looks fine.

Everything else was set dressing. Dim room, wooden table, water in the jar, eight tentacles with suckers, no external models, orbit controls, a pause button and a readout of the phase and the lid angle. And "a single self-contained HTML file", which comes back later.

The results

Model	Agent	Time	Bugs it hit	What it delivered
GPT-6 Astra	Codex, high effort	14 min	1	Full sequence verified: ~58 s, 3.5 turns of the lid
Opus 5.5	Claude Code	25 min	2	87 KB file, full loop tested through a local server
GPT-6.1 Sol	Codex, high effort	29 min	0	Full cycle at 56–60 fps, pause and reset tested
Sonnet 5.5	Claude Code	72 min	0 recorded	112 KB file, console warnings left at the end

The times include each agent testing its own work. The Codex runs were at high effort, and the Claude Code runs didn't record one. Different tools, one run each.

GPT-6 Astra, 14 minutes

Astra put everything in one file and treated the lid as a body that can only rotate. Grip forces from the tentacles turn it, and the thread pitch turns rotation into height.

Before writing code, it looked up how similar builds had gone. One of those was my Opus run from below, which didn't find its worst bug until it could step through the animation phase by phase. Eleven minutes in, Astra built itself the same kind of check: a local server plus timed checkpoints which replay the sequence and read the simulation state out of the DOM.

The checkpoints paid off two minutes later. Over the 3.5 turns, the tentacles were encircling the octopus's own body instead of letting go and grabbing a fresh spot within reach. Watching it, you'd see busy arms.

It rewrote how each arm picks its anchor, ran the checkpoints again and was done at minute 14.

Opus 5.5, 25 minutes

Opus built the scene in parts and glued them into a 1,735-line file. The first thing that broke was the skin shader, which wouldn't compile because a variable was called patch. That's a reserved word in GLSL. Rename, move on.

What happened next is why I'm writing this. Instead of waiting for the animation to play out in real time, Opus added temporary hooks so it could jump to any phase and check the numbers. The first jump showed the lid sometimes turning backwards while being unscrewed. I would not have caught that by watching it once.

Once that was fixed, it stepped through the rest: the lid coming off, falling on the table, the octopus squeezing through the neck. It measured 0.68 ms of simulation and 1.1 ms of rendering per frame and deleted the hooks before handing over the file.

GPT-6.1 Sol, 29 minutes

Sol started by picking a palette (brass, walnut, teal glass, a coral octopus) and then wrote a 309-line file where friction between tentacles and lid drives the thread. From minute 16, it read the live state off the canvas, so phase, lid angle, torque and grips were all visible to it.

No bugs recorded. Most of the remaining time went into the look. Light colour, wood grain, light again, then two passes on how the body squeezes through the neck. Its own note at the end says it should have been told the fps target and the lighting mood up front.

The result is solid. The lid comes off at 1260°, bounces on the table, and the octopus ends up next to the jar without clipping through the glass.

Sonnet 5.5, 72 minutes

Sonnet split the work into 14 files and wrote a little build script to concatenate them. It opened a bare stub in the browser at minute 34, before the octopus existed.

It then went deep on things nobody else touched. It profiled uneven frame pacing, found the lid never quite came to rest after falling, stepped through the bounces, and even checked the scene on a 375×812 phone viewport.

No failure was recorded in that session. It did run into a duplicate JavaScript declaration twice, though, and its last check still showed that declaration and some WebGL feedback-loop warnings in the console. Everything rendered and the readout worked, so it shipped like that. It also never ran the whole sequence live without stepping, or on an integrated GPU.

What I'm taking from this

I stopped trusting the bug count. Opus logged two and ended clean because it found both. Sonnet logged none and ended with warnings, because nothing forced it to look at them. The number tells you what got caught.

Every real bug in these four runs turned up while an agent was poking at the running scene. Nobody found one by reading their own code back. Astra and Opus built that kind of tooling early and found their bugs early. Sonnet opened a browser early too, but its checks were mostly about performance, and the warnings got through.

The "self-contained" thing annoyed me more than it should have. Both Claude Code runs pulled Three.js from a CDN anyway (r170 for Opus, r167 for Sonnet), and Opus never opened its file straight from disk. I didn't check the Codex files for this. Either way, if a word matters to you, spell out what it means.

And whatever the prompt left vague is where each run spent its extra time. For Sol that was lighting. For Sonnet it was frame pacing and the falling lid. Neither was wrong to care; I just would have picked something else.

So this goes into the first prompt next time. Every one of the four sessions points to at least one of these lines:

Before you build the scene, build a way to check it:
expose phase, lid angle and grip state on window,
and let me fast-forward to each phase on demand.
"Self-contained" means no network: inline three.js.
Test the file opened from disk, not only from a server.
Target: 60 fps on an integrated GPU.
Lighting: one warm side light, dim room. Do not retune it.

My guess is that with these lines, all four would finish faster and closer together. I'll rerun it and see.

The sessions

All four are public, with the moments that counted pulled out and the raw transcript a click away. I keep them on coders.talk, which I'm building:

Has a "clean" run ever turned out to be your worst one? And for something physics-heavy like this, which model would you hand it to first?

Your Git History Keeps the Code. Who Keeps the Lesson?

Evgeny Shevtsov — Sun, 04 Oct 2026 21:18:09 +0000

A commit can tell you when a help button appeared. It usually cannot tell you that the game had already passed its automated checks when the person building it said they had no idea how to play.

That happened in one of my sessions with Claude Code.

The project was BUTTERFLY JOB, a browser game built with Three.js and Blender assets. The player changes the past to make a bank heist possible in the present. The session records successful browser and unit checks. Then, at +3h 43m, I stepped in because I could not work out what to do.

The agent added a how-to dialog, a persistent guide, and a help button.

Those changes belong in Git. So does the reason for them. But unless someone deliberately writes that reason down, the next developer sees the UI changes and has to reconstruct the lesson: a tested path through a game does not establish that a new player can discover that path.

I built Coders Talk to keep that part of working with coding agents.

The correction and the response, preserved together in the public Build.

The diff is only part of the explanation

Git preserves the changes we commit. A good commit message, pull request, or decision record can preserve the reasoning too. I still want all of those.

But a lot happens before the final change makes it into a commit.

You ask an agent to implement something. Its first interpretation is plausible, but misses a constraint. You correct it. It tries another approach. A test exposes a bad assumption. You add a detail that should have been in the opening prompt.

By the time the work is ready, the code looks reasonable. The abandoned approach may be gone. The useful correction sits somewhere in a long conversation, between command output and another test run.

Two weeks later, you remember solving something similar. You do not remember the sentence that changed the agent's direction.

Keeping the raw log helps. Finding the relevant turn, understanding what preceded it, and deciding whether it applies to today's task still takes work.

That is the gap I wanted Coders Talk to fill. I wanted to come back to a session and answer: what was the task, where did the approach break down, and what changed after the human stepped in?

A session becomes a Build

In Coders Talk, a Build is a working session condensed into its key moments, with the underlying record available at the level the author chooses to share.

The format has a goal, a timeline, an outcome, and a verdict: what you would do differently next time. The timeline uses five kinds of moments:

Moment	What it preserves
Prompt	The request or constraint that set the direction.
Agent did	A meaningful stretch of work: the approach and what it produced.
Intervention	What the person corrected, and why.
Fail	What broke or turned out to be wrong.
Outcome	Where the session actually ended, including an abandoned attempt.

You do not need every kind in every session. A run without a human correction is still worth keeping. So is a failure.

The part I care about most is the why attached to an intervention. “I stopped the agent” tells another developer very little. “The tests covered the route, but I could not discover the first action” gives them something to check in their own project.

In the BUTTERFLY JOB session, an earlier correction made the asset requirement explicit: every visible 3D object had to be authored in Blender and exported for the browser. The agent then validated that pipeline with a representative asset before continuing.

The same Build preserves both kinds of intervention: a constraint clarified near the start and a usability problem discovered near the end. That is much more useful to me than remembering only that the agent eventually produced a playable game.

How it gets there

With the plugin installed, you can send the session you are working in:

Claude Code: /coders-talk: build
Codex:       $coders-talk:build

For a manual send, the plugin shows what it will upload and where the draft will go. It removes bulky session material, redacts detected secrets locally, and sends the prepared session. The server checks it again.

A model proposes the title, moments, and draft verdict. You can edit those suggestions, explain the reason behind a correction, and confirm the summary describes what happened. The page identifies the model's contribution; summarizing a session does not make the summary infallible.

The result starts as a private draft, or goes to your team's space when the session belongs to one of its repositories. Sending it does not publish it. Auto mode is optional and must be enabled separately.

There is context behind the short read: the agent and model, session duration, interventions, and, where captured, token usage, code changes, and commits. That context helps you judge whether another person's experience is relevant to your task. One successful run does not establish a universal recipe.

You do not have to publish your work

If your useful sessions happen in private repositories, you should still be able to keep the lessons.

Coders Talk has three ways to use that record. You can keep it for yourself, use it inside a team, or publish a Build for other people to learn from. Public sharing is optional.

For an individual, the immediate value is finding a previous correction without reading the whole conversation again. For a team, a recurring problem can become a proposed rule that teammates review before it reaches their agents. The team workflow keeps the rule connected to the session that explains it.

When you do publish, you choose the exposure: a summary, selected excerpts, or the full session. A lesson can be useful without exposing the entire conversation.

Private here describes who can access the Build. The prepared session is still uploaded and processed on the server. The plugin withholds environment and key-file contents and redacts detected secrets before upload; you should still review anything you intend to make public. The plugin documentation explains what is sent.

The lesson should reach the next session

A record becomes more useful when it changes what you do next.

A Build can have a Playbook: an approach, pitfalls, and checks drawn from that session. On the page, you can inspect the moments behind the advice. Through Use this Build, you can reuse it as a prompt, a skill, or a repository rule for CLAUDE.md or AGENTS.md. Here is how Playbooks work.

For BUTTERFLY JOB, the Playbook carries forward the need to check whether a first-time player can find the first action. It links that advice back to the intervention that prompted it.

The advice has a source you can inspect, including the moment where automated checks proved insufficient.

Your agent can also search published Builds through the Coders Talk MCP server, looking for similar tasks and recorded failures before starting, or when it gets stuck.

That gives the next session a more specific starting point. Instead of a vague instruction to “test thoroughly,” you can ask for the check a previous session missed. You still have to decide whether it fits your project.

Keep one correction

The easiest place to start is a session where you had to explain something twice.

Find the correction that finally helped. Keep the context around it. Write down what you would put in the first prompt next time.

You can do that in a Markdown file today. I built Coders Talk to make it easier to preserve the session behind that note, find it again, and carry its lesson into another run.

If you want to try it, turn one session into a private Build. You can decide later whether it is worth sharing.

What is one correction you keep giving your coding agent that you wish you had saved the first time?