The hardest bugs weren't crashes. They were the ones where the system looked like it was working while quietly becoming less useful.
I've been shipping coldstart in the open for the last few months. Before that, I mostly used it myself, on my own repositories, with my own workflow.
I thought I knew where the weak spots were.
I didn't.
The first people who installed it found bugs I hadn't imagined, but more interestingly, they exposed assumptions I didn't realize I'd built into the system. Looking back through the issues and release history, almost every lasting fix came from removing one of those assumptions entirely.
Automatic capture wasn't enough
One of the first feature requests wasn't for a new capability.
It was for a manual one.
Coldstart captures notes automatically as you work, but users wanted a /capture-notes command they could run whenever they felt a session had reached a natural stopping point.
Adding the command turned up a much bigger problem.
Automatic capture marked files as "already offered." The manual command treated that flag as "already captured," so if you'd edited a file earlier in the session, the explicit capture command often skipped the most important work entirely. Brand new files could disappear from the worklist, while older read-only files stayed.
That forced me to rethink what those flags actually meant.
A file being offered to the agent once isn't the same thing as a good notebook entry existing.
The capture pipeline now tracks actual work instead of whether something happened to appear in an earlier prompt, and manual capture always prioritises the files you changed during the current session.
The feature request turned into a redesign of the capture model.
Measuring recall instead of tuning it
Another complaint was harder to pin down.
Recall is the part of coldstart that automatically surfaces older notes when they seem relevant to what you're working on. It felt noisy, but "felt" isn't something you can optimise.
So I stopped guessing.
I hand-labelled 140 real recall events and measured how often the injected note was actually useful.
The baseline precision was 31%.
Three different problems showed up.
Ordinary words like "status" or "merge" matched unrelated notes because they happened to share common vocabulary.
The same note could be injected repeatedly during one session, even though it was already sitting in the model's context.
And some of the query wasn't coming from the user at all. Harness telemetry and editor boilerplate were leaking into the relevance calculation and promoting notes that had nothing to do with the task.
The fixes were surprisingly small.
Require stronger evidence before accepting a single ordinary-word match.
Don't show the same note twice in one session.
Strip infrastructure text before building the search query.
Precision increased to 47% while keeping most of the genuinely useful recalls.
The biggest lesson wasn't the number.
It was that retrieval systems are very easy to make feel better and much harder to make measurably better.
Notes existed, but couldn't be found
One of the most surprising bugs wasn't that notes were missing.
They were there.
They just weren't retrievable.
The information that makes a note searchable—aliases, symbols, verified anchors—had drifted across multiple places. The capture prompt expected one shape. The documentation described another. The MCP tool had a third.
Everything still worked.
Until it didn't.
The solution wasn't another validation rule.
I froze the note schema in one canonical definition and generated every surface from it instead. The capture prompt, documentation, repair tooling and MCP interface all derive from the same source now, and tests fail if they drift apart.
That also led to kb repair.
Instead of assuming every notebook stays healthy forever, coldstart can now inspect itself and tell you which notes have become incomplete or difficult to retrieve.
That was a change in philosophy as much as implementation.
The notebook stopped being something users had to trust blindly and became something that could explain when it needed maintenance.
Context is temporary. State shouldn't be.
The most difficult bugs all had one thing in common.
They assumed an agent would remember something later.
Notebook capture originally handed the agent a worklist and a write contract in a single prompt.
Hours later, after conversation compaction, that prompt was gone.
The agent had to reconstruct the note format from memory.
Writing multiple notes had a similar problem. A malformed note halfway through the sequence stopped every note after it from being written, and there was no single place that could answer a simple question:
"Did this session actually get captured?"
The eventual solution wasn't another prompt.
It was moving the state somewhere durable.
The worklist now lives inside the repository itself, where the agent can read it again whenever it needs to. Writing happens in one batch instead of a chain of independent commands, coverage is calculated once, malformed notes are reported together instead of silently preventing everything after them, and the worklist updates itself as notes are completed.
The implementation is more complicated.
The behaviour is much simpler.
The fixes that lasted all looked the same
The installation problems, recall changes, capture redesign, repair tools and durable worklists all looked unrelated while I was fixing them.
Looking back, they all came from the same mistake.
I kept assuming information would stay synchronised.
That prompts would stay visible.
That different copies of a schema would evolve together.
That something shown once had effectively been remembered.
That writing a note automatically meant it would still be discoverable months later.
Real users broke every one of those assumptions.
Most of the fixes that lasted weren't bigger patches.
They removed the assumption altogether.
Replacing native dependencies with WASM removed an entire class of installation failures.
Generating every interface from one note schema removed documentation drift.
Persisting the capture worklist removed the need for an agent to remember instructions from fifty turns earlier.
Adding repair commands acknowledged that notebooks, like code, slowly drift and occasionally need maintenance.
That's probably the biggest thing shipping coldstart has taught me.
Bugs rarely come from the thing you were trying to build.
They come from the assumptions you forgot you were making while you built it.
coldstart is open source under the MIT license. If you end up trying it and manage to break an assumption I haven't found yet, there's a good chance it'll become the next entry in the release notes.
Top comments (0)