DEV Community

Cover image for I indexed 3.3 GB of my coding agents' logs so I'd stop re-solving the same bugs
Vlad Shulcz
Vlad Shulcz

Posted on

I indexed 3.3 GB of my coding agents' logs so I'd stop re-solving the same bugs

A few weeks ago I spent most of an evening debugging a postgres connection leak that I was pretty sure I had already debugged. Not a similar one. The same one, in the same service, with the same pgxpool symptoms. Somewhere around midnight I found my old fix — not in git history, and not in my notes, because of course I didn't write any. It was in a Claude Code transcript from April, sitting in a JSONL file under ~/.claude/projects/, where it had been the whole time.

That file exists because every coding agent keeps full session transcripts on disk. All of them. Nobody reads these files, most people don't know they exist, and after a few months of agent-heavy work they quietly become the best engineering notebook you never wrote. Mine was 3.3 GB across three agents, roughly 50k messages.

So the obvious first move: grep it.

grep -r "connection pool" ~/.claude/projects/ | head
Enter fullscreen mode Exit fullscreen mode

This technically works and practically doesn't. Claude Code transcripts are JSONL where the text lives at .message.content, which is sometimes a string and sometimes an array of typed blocks. Tool outputs dominate the byte count, so most matches are inside a 40 KB cat result rather than anything a human said. And that's one agent out of the several I actually use. Codex has a different layout (~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl, plus a separate history.jsonl that partially duplicates it). opencode keeps everything in SQLite. You can build a jq pipeline for one of these in ten minutes; you will not maintain jq pipelines for eight.

I know eight because I ended up cataloguing them:

Agent Storage
Claude Code ~/.claude/projects/<encoded-path>/*.jsonl
Codex CLI ~/.codex/sessions/.../rollout-*.jsonl + history.jsonl
opencode SQLite
Cursor SQLite state.vscdb for IDE chats, separate transcripts for the CLI
Gemini CLI ~/.gemini/tmp/<hash>/chats/ — the same session in .json and .jsonl
aider .aider.chat.history.md, one per repo, markdown
Antigravity transcript.jsonl under ~/.gemini/antigravity/brain/
Grok Build ~/.grok/sessions/<url-encoded-cwd>/<id>/updates.jsonl

None of this is documented. Some of it is actively weird. The Claude Code project directory encodes the project path by replacing / with -, which means -Users-me-my-app is ambiguous: was that my/app or a directory literally named my-app? You can't tell from the string alone; you have to walk the filesystem and check what exists. Gemini writes the same session as both a checkpoint .json and an event-log .jsonl, and the .jsonl contains $rewindTo events, so reading it means replaying edits, not just parsing lines.

The result is deja-vu, a single Go binary that parses all of this into one local index:

curl -fsSL https://raw.githubusercontent.com/vshulcz/deja-vu/main/install.sh | sh
deja install --auto
deja "connection pool exhausted"
Enter fullscreen mode Exit fullscreen mode

The screenshot shows the property I actually wanted: hits come back from Claude Code, Codex and Antigravity sessions at once. The April fix is findable regardless of which agent I happened to be talking to in April. And because it indexes what's already on disk, there's no cold-start problem — no proxy to install, no capture hook, no "it'll be useful in a month". The history is already there; it just wasn't queryable.

Why not embeddings

Fair question, since every agent-memory project reaches for a vector store. I considered it and decided against, for a boring reason: when I search these logs, I'm searching for an error string, a function name, or a flag. "SSL_ERROR_SYSCALL", resolveEncodedPath, --force-with-lease. Exact tokens. Semantic similarity actively hurts here — I don't want things like my error, I want the session where that literal string appeared, because that's where the fix is.

So the index is embarrassingly classical: a records file, varint-encoded token postings, a small manifest. It rebuilds from scratch in about 14 seconds on my corpus and answers warm queries in single-digit milliseconds. There's no daemon; the CLI checks file sizes and mtimes on each run and incrementally ingests whatever changed. The whole thing is one static binary because it started as a tool I scp'd between machines, and I wanted that to keep working.

The parts that bit me

Incremental indexing of files that other processes are actively writing turned out to be where all the actual engineering lives.

Torn tails. An agent can be mid-write when the indexer runs, so the file ends in half a JSON line. Skipping the broken line is easy. The bug is on the next run: if you resume from the new end-of-file, the message that was half-written last time is now complete — and you've skipped it forever. The fix is to store the offset of the last complete newline at index time and resume from there, not from wherever the file ended.

History that un-happens. I assumed transcripts were append-only. Gemini and Grok both violate this: a rewind truncates the file and regrows it with different content. Size-based change detection sees "file got bigger" and happily appends, duplicating everything before the truncation point. Those files now get re-parsed in full whenever they change, and the session's records are replaced rather than extended.

The re-merge that ate seconds. An early user with 592 Cursor sessions reported that every search took seconds. Cursor's SQLite database has no per-message watermark I was tracking, so every query re-merged the entire chat history. Embarrassing to explain, easy to fix once a real corpus hit it: track the DB's last-updated timestamp and query only newer rows. My test fixtures were too small to notice. Now there's a 10k-message fixture in CI precisely because of this.

filepath.Dir at the root. A walk-up loop that terminates on dir != "/" runs forever on Windows, because filepath.Dir("C:\\") is "C:\\". Found by a hung Windows CI job, ten minutes of nothing followed by a goroutine dump pointing at filepath.Clean. The loop now breaks when Dir(dir) == dir, which is what I should have written the first time.

Wiring it into the agents

Search from a shell is half of it. The other half is the agents using the memory themselves: deja mcp is a stdio MCP server with a recall tool, and deja install --auto writes it into whatever agent configs exist on the machine — .claude.json, Codex's config.toml, Cursor's mcp.json, Grok's config.toml, and so on, each with a .bak and an uninstall that removes only its own block.

Where the agent has a hook mechanism, install goes further: Claude Code gets a SessionStart hook so relevant past sessions are injected before you type anything. Codex and opencode have equivalents. The others don't expose any way to inject context (Grok has hooks, but stdout from passive events is discarded), so for them the MCP tool is as deep as it goes. aider has no MCP client at all; deja ctx "query" > ctx.md and passing the file in is the workaround, and I won't pretend it's elegant.

One more thing, since Grok Build's cloud upload made the rounds this week: deja reads local files and writes a local index. The only network operation in the entire tool is deja sync ssh <host>, which you run yourself — I use it to keep a laptop and a mac mini seeing each other's sessions. Batches go over plain SSH, dedup is content-based, nothing else leaves the machine. Transcripts are also full of pasted secrets, so everything passes through redaction patterns (AWS keys, JWTs, PEM blocks, the usual shapes) before hitting the index; patterns aren't understanding, though, so read deja share output before sending it to anyone.

What it doesn't do

It won't find "that auth thing we discussed" — it's token search, not semantic. Cursor IDE chats need the sqlite3 CLI on PATH because I refused to take a C dependency for one harness. And eight undocumented formats means eight parsers that upstream can break at any moment; the test suite has fixtures for every format and a cross-harness suite in CI, but realistically, field reports are part of the maintenance model. If your agent isn't covered or its format changed, an issue with a redacted sample is the fastest path: the Grok Build parser went from issue to merged PR in a day, and most of it was written by the person who opened it.

github.com/vshulcz/deja-vu — MIT, no runtime dependencies.

Top comments (0)