DEV Community: Vanshaj Kakkar

Crysto: how I run Claude, Codex and other AI models as one team. The models were the easy part.

Vanshaj Kakkar — Thu, 05 Nov 2026 04:24:58 +0000

Early on, two of my AI agents tried to use the same browser at the same time. The browser fell over. Neither of them had really done anything wrong. They had each followed their instructions to the letter. They just had no idea the other one was there.

That was when I realised the hard part of running AI agents isn't the AI. Getting a model to write decent code is close to solved. Getting four of them to work on the same projects for weeks, without stepping on each other, redoing each other's work, or cheerfully telling me something was done when it wasn't, is where all my time has gone.

I call the whole setup Crysto. It started as a way to keep my own projects organised, and grew into something I lean on for my professional work too, with the two sides kept completely separate. Here's how it works, and what broke along the way.

Why I built it

I'm an AI engineer, so I spend most of my day with these tools anyway. But my own projects were scattered: side projects, research, writing and the important things I needed to keep on top of, all half-done across different chat windows. Every new chat started from nothing, and I was the only one keeping track of where anything was up to.

Who's on the team

The Crysto stack. Click through it at vanshaj1999.github.io/crysto

Claude Code runs the show. It plans the work, writes briefs for the others, reviews everything they hand back, and decides what actually ships. I run it on two separate accounts, one for work and one personal, and a small shell hook switches between them depending on which folder I'm in.

Codex gets the long jobs. Give it a clear brief and it will build a small project, or grind through a pile of browser forms, while I get on with something else.

Smaller models, through a tool called OpenCode, handle the bounded fixes. I can hand the same task to five of them at once and keep whichever result is best.

Claude on my phone connects through a small MCP server on Cloudflare Workers, with KV storage behind it. That gives the phone and laptop access to the same stored data without relying on a shared chat.

The coding agents also run headless. These are the command shapes I use, with a model and brief supplied for each run:

opencode run --format json -m <model> "<brief>"

codex exec -s workspace-write "<brief>"

Each run gets its own copy of the repo. OpenCode runs one model per invocation; Codex works inside its workspace-write sandbox. Separate copies matter when I'm comparing results: otherwise a later model could inherit an earlier model's edits, and I'd have no clean way to tell who did what.

The catch is that none of them share a conversation. Everything below comes from that.

Nobody remembers anything

Codex can't see my Claude chats. Tomorrow's Claude session can't see today's. Every agent wakes up with amnesia.

What fixed it was embarrassingly low-tech: files. Every project lives in a folder with a STATUS.md that says where things are up to. Every session reads it first and updates it before it finishes. Longer-term memory is one fact per file with a one-line summary, and only a short index of those gets loaded each time. Each memory file has front matter for its name, description and type. The four types are user, feedback, project and reference. The index tells the agent what exists; it can read the relevant file when the task needs it.

The part that surprised me was how much it helps to give every rule a reason. "Only one agent uses the browser at a time" is a rule. Add "because two agents on the same debugging port will crash each other" and an agent can work out the case I never wrote down, like a second browser tool on the same machine.

It sounds almost too simple to write about. But it's the reason any agent, on any model, can pick up a project cold.

Who's doing what

Back to the browser crash. The fix was one more file, COORDINATION.md, which lives in the hub. Before an agent touches files or the browser, it writes down that it's taking the job:

# Website copy review, 4 November: CLAIMED BY CODEX
Owner: Codex. Claimed 4 Nov 10:05.
Done: 12 of 18 claims checked.
Evidence: drafts/about.md diff.
Next: check the last 6, then hand back to Claude.

A real run: creating a project, then the 17 tests in make check

The rules fit on an index card. Check the file before you start. If someone else has claimed a task, leave it and ask me. Only one agent on the browser. And nothing actually enforces any of it. There's no lock. It works because every agent's instructions tell it to respect the file, and so far they do.

It earned its keep this week. A work session already had the browser, so my personal session switched to a different browser tool instead of grabbing it.

Tests that pass and still lie

The whole setup depends on the smaller models building and Claude reviewing. That only works if the review actually catches things, so I tested it.

The bug was pydynox issue #560. Its MemoryBackend decorator wrapped an async function, but exited its context before the coroutine ran. That's a useful failure to test because returning a coroutine and executing it aren't the same event. The context has to cover the actual async execution.

I gave it to 9 models, with tests I'd written beforehand and kept hidden from them. Eight passed those hidden tests; the fastest finished in 84 seconds. Nemotron 3.5 Lightning passed, but also deleted an existing test. If I'd only read the final test result, I'd have missed that change entirely.

The refactor task was in awslabs/stickler, with a 4,091-test suite. Five models kept it green without editing a test. Flint 40B also ran ruff --fix over 8 unrelated notebooks. Its diff touched 11 files, while the others touched 3. The extra changes weren't necessary for the task, even though the suite still passed.

Photo by Kevin Ku on Unsplash

So now Claude does three things with every change. It writes tests up front that the building agent never sees, runs them afterwards, and reads the whole diff. A green test run tells you the tests passed. That's all it tells you.

The build loop. Interactive version at vanshaj1999.github.io/crysto

Everyone exaggerates a little

When an agent finishes, it writes a handover: what it changed, the proof, what went wrong, and what it didn't get to test. The rule is that "done" means a command ran and the output is in the file. Plans and good intentions don't count.

Even so, claims creep. When the smaller models built my retrieval benchmark, their handover was upfront that one feature, the Claude answer path, had never actually run. Their README still made it sound like part of the results. The review also found a threshold that had been tuned on the very questions it was being scored on. Both are fixed now, and the README says so.

That project, rag-eval-lab, uses 745 Python PEPs split into 17,933 chunks, stored in PostgreSQL with pgvector and an HNSW index. Vector search scored recall@1 of 0.70 against 0.36 for full-text search. Hybrid search with reciprocal rank fusion, or RRF, scored 0.66. Here, recall@1 asks whether the expected result came first. Adding another retrieval method didn't automatically improve that score.

Those numbers describe this benchmark. They don't prove that vector search wins on every corpus, and they say nothing about an answer path that never ran. That's the distinction I need the handover and README to preserve.

Claude isn't immune either. While drafting my last article, it wrote that my fixes had been sent to the project maintainers. They hadn't, not yet. We only caught it because the check went back to the actual pull requests.

I've stopped hoping for a model that never overstates things. What I want is a check that doesn't take its word for it.

The rules that weren't about AI

Two of my most expensive lessons had nothing to do with models.

My work and personal Claude accounts are kept apart by separate config folders. But both read the same keychain entry, so logging out of one to switch logs out every session on the machine. Now the shell hook handles the switch and prints a line every time it happens, and I never log out.

The account folders use CLAUDE_CONFIG_DIR for Claude and GH_CONFIG_DIR for the GitHub CLI. The hook checks the current directory and selects the matching config. These lines come straight from account-switch.zsh:

autoload -Uz add-zsh-hook
add-zsh-hook chpwd _crysto_claude_account
add-zsh-hook chpwd _crysto_gh_account
_crysto_claude_account
_crysto_gh_account

The chpwd hook runs the account functions when the directory changes. The final calls also select the accounts when the shell starts. Separate config folders solve the selection problem; they don't remove the shared keychain behaviour.

Then the voice assistant. My first design for "Hey Crysto" was a small Cloudflare Worker that called Claude with my subscription login. Anthropic's terms don't allow that: a subscription is for their own apps, and anything running on your own server needs a paid API key. So I split the problem. The part that matters most, Crysto's connection to my data, is built and running: a personal MCP gateway on Cloudflare Workers with OAuth and PKCE, read-only by default, where any write needs my explicit consent. Codex signs in to it and reads my mail, calendar and project data through one endpoint. For voice, I use the Claude app and Siri today. The always-on brain, a model that runs by itself and also runs the lights at home, is designed and waiting. The main thing stopping me is the API bill: I'm not ready to pay per request for a side project yet. Next time I'll read the terms before I draw the diagram.

Under the hood

Crysto is mostly Markdown files and shell commands. Claude Code coordinates the work, while headless OpenCode and Codex runs produce changes for review. Project status and task claims live beside the work, so the next session can read them with ordinary file tools.

The public repo includes a bash script to create a project, plus 17 tests, shellcheck and GitHub Actions CI. The Cloudflare Workers MCP server uses KV for stored state. PostgreSQL, pgvector and HNSW belong to the retrieval benchmark, rather than the coordination protocol. I don't need a database query to find out who owns the browser.

What it costs

Not much beyond the subscriptions I already had. Money isn't really the limit. Attention is, so the whole thing is set up to spend it on briefs and reviews rather than on typing code.

If you want to try this

Keep shared state in files, not in chats.
Have agents claim work before they start, and give the browser one driver.
Write your own tests and keep them away from the agent writing the code.
Read every diff, even when it's green.
Ask for proof, then check it.

The crysto repo on GitHub

The design, the coordination rules, the templates and a script that sets up a new project are all open source at github.com/vanshaj1999/crysto. My memory, notes and data aren't, and that's on purpose.

Want to click through it yourself? The interactive diagrams are at vanshaj1999.github.io/crysto.

Cover photo by James Harrison on Unsplash.

I gave 9 free AI coding models the same real bug. Here is what happened.

Vanshaj Kakkar — Thu, 05 Nov 2026 02:10:55 +0000

Everyone has an opinion about which AI coding model is best. Most of those opinions come from toy prompts or leaderboards. I wanted to see what happens on a real bug in a real open source repo, with tests the models cannot see.

So I gave the same bug to 9 free models at the same time and graded every result the same way.

The bug

The repo is pydynox, a Python DynamoDB library. Issue #560: the MemoryBackend test helper works as a decorator, but on an async function it exits its context before the coroutine runs. So the test runs against the real backend instead of the in-memory one. It is a small, realistic bug: easy to describe, easy to get subtly wrong.

The setup

Agent: OpenCode, run headless with opencode run, one model per run.
Models: 9 free models from the OpenCode catalogue, all started at the same time, each in its own copy of the repo.
Brief: identical for all 9. Read the issue and the repo's own AI guidelines, make the smallest correct change, add tests that fail without the fix, do not commit.
Hidden tests: before the run I wrote my own tests for the bug. They fail twice on the original code. The models never saw them. After each run I added them and ran the suite.
Grading: hidden tests pass, the model's own tests pass, and a diff review for anything outside the task.

The results

8 of 9 models produced a fix that passed the hidden tests. Big Pickle and Pulse Alpha both finished in 84 seconds. DeepSeek V4.1 Flash took 2 minutes 11 seconds. The rest took between 4 and 6.5 minutes. Total model cost: $0.

Speed was not the story, though. The failures were.

How the models went wrong

Crouton Alpha gave up after 10 seconds. No fix, no tests. That is the easy failure, because you notice it.

Nemotron 3.5 Lightning passed, but deleted an existing test. Its fix was correct and the hidden tests passed. It also removed a test that was already in the file. A green test run would never tell you that. Only reading the diff does.

That is the failure I worry about most with coding agents. Not "it did not work", but "it worked, and quietly changed something you did not ask for".

A second test: a refactor in a 4,091-test suite

One bug is a small sample, so I ran a different kind of task. awslabs/stickler #517 asks for two classes to share their state handling through one private base class. The repo has 4,091 tests, so a refactor has nowhere to hide.

I gave it to 5 free models. All 5 kept the full suite green without editing a single test. Big Pickle was fastest again, at 4 minutes 34 seconds.

One model, Flint 40B, also ran ruff --fix across 8 Jupyter notebooks that had nothing to do with the task. Its diff touched 11 files. The others touched 3. Again: the tests passed, and the change was still not one you would merge.

Free versus paid

For comparison I gave the first bug to three paid models: GLM 5.3 Flash, Kimi K3.1 Pro and Qwen 3.8 Max. All three fixed it, with tests that fail without the fix. They were not faster than the best free models.

There was one real quality difference. Only Qwen detected coroutine functions with inspect.iscoroutinefunction. The other two used asyncio.iscoroutinefunction, which is deprecated from Python 3.14. Both pass today's tests. Only one is still right next year. No test suite catches that. A reviewer who knows the language does.

What I took from it

For small, well-described bugs, free models are good enough. 8 of 9 got it right. The gap between free and paid was smaller than I expected.
Write your own tests, and keep them hidden. A model grading itself with its own tests tells you little. Tests it never saw tell you a lot.
Read every diff. The two most worrying results both passed their tests. One deleted a test, one reformatted files outside the task.
The human work moved, it did not disappear. Choosing the issue, writing the brief and the hidden tests, and reviewing the diff took longer than the fixes did. That is the job now.

One more lesson from the open source side: by the time I had a fix for one issue, a human contributor had claimed it 6 minutes earlier. Check the issue again right before you post.

I now use the same workflow for real contributions. Each pull request is reviewed and tested by me and labelled as AI-assisted, which is what the maintainers ask for.

Caveats: one run per model, on two tasks. Free models in this catalogue change often, so treat the names as a snapshot from November 2026, not a ranking.