Parsing Claude Code's JSONL: patterns for a schema that keeps moving

Every conversation you have with Claude Code is written to disk as JSONL, under ~/.claude/projects/. Your decisions, your dead ends, the bug hunt that took three sessions: it is all there. You have probably never opened one.

The catch: the format is an internal implementation detail. No documentation, no version field, no stability guarantee. The schema changes whenever the CLI updates, which is, at the current pace, almost daily.

The patterns below come from building a read-only replay and search tool on top of these files, and from keeping it alive through a dozen CLI releases. Each pattern survived contact with real data. Four were learned the hard way, from bugs worth retelling.

The ground rule: you are an archaeologist, not a validator

A parser for someone else's internal format has a different job than a parser for your own API. Rejecting malformed input is not an option: the input is the historical record, and whatever is on disk is all there will ever be.

The contract that follows from this:

1. A bad line never kills the file. Skip it, count it, move on.
2. An unknown shape is preserved, not dropped. You can re-parse later; you cannot un-drop data.
3. Normalize at the boundary, once. Downstream code (search index, UI, export) should never see the mess.

Everything below is one of these rules meeting reality.

Pattern 1: tolerant line parsing with an explicit whitelist

The naive loop (JSON.parse each line, switch on type) works on day one. The question is what happens when a CLI update introduces a type nobody has seen before. This is not hypothetical; a real batch of them appears at the end of this post.

The approach that holds up: keep an explicit whitelist of known types, and treat everything outside it as "parse failed, but preserved":

const KNOWN_MESSAGE_TYPES = new Set([
  'user', 'assistant', 'system',
  'queue-operation', 'last-prompt',
  'progress', 'attachment', 'file-history-snapshot',
  'permission-mode', 'custom-title', 'ai-title',
  'agent-name', 'pr-link',
])

function parseLine(line: string): ParsedLine | null {
  if (!line.trim()) return null

  let obj: Record<string, unknown>
  try {
    obj = JSON.parse(line)
  } catch {
    return null // malformed line: skip, never throw
  }

  const type = typeof obj.type === 'string' ? obj.type : 'unknown'
  const parseFailed = !KNOWN_MESSAGE_TYPES.has(type)
  // unknown type → keep the raw JSON string for later re-parse
  // known type → extracted fields are enough, raw can be dropped
  ...
}

The whitelist does double duty as a storage policy. For known types, the extracted columns are sufficient and the raw JSON can be discarded; that alone reclaims most of the disk space. For unknown types, the raw line goes into an archive table untouched. When a future version of the parser learns the new shape, the evidence is still there.

One more detail that pays off: cap the length of identifier fields (uuid, requestId) at something sane like 128 chars before trusting them. Parsing files you do not control calls for a little paranoia at the boundary, and it is cheap.

Pattern 2: version your derived data, not just your schema

Preserving unknowns only matters if you can act on them later. The mechanism is a SUMMARY_VERSION integer stored per session. When the parser learns new tricks, bump the version; the indexer sees stale versions and re-parses those sessions automatically on the next sync.

This turns "the schema changed again" from a migration crisis into a routine: extend the parser, bump the version, let the backfill run. No manual steps, no data loss, no "please delete your index and start over" release notes.

War story 1: the lone surrogates

One day the indexer started producing strings that crashed downstream consumers. The cause: some JSONL lines contained unpaired UTF-16 surrogates. Half an emoji, lurking in a tool-error message.

How does half an emoji end up on disk? Older Claude Code versions (up to around 2.1.132, judging by the archived sessions) truncated long tool outputs by byte length, and the cut sometimes landed mid-emoji. JSON.stringify happily writes the lone surrogate as a \udXXX escape, the file looks like clean ASCII, and JSON.parse faithfully reconstructs the broken string at read time. The corruption stays invisible until something refuses it: SQLite, an IPC bridge, a TextEncoder.

The fix is one line, if it lands in the right place:

// at the parser's exit boundary, applied to every extracted string
export function ensureWellFormed(s: string): string {
  return s.toWellFormed() // lone surrogates → U+FFFD
}

The placement is the actual lesson. Normalize once, at ingestion, and every consumer downstream (search index, renderer, Markdown export) gets to assume well-formed strings forever. Unicode normalization (NFC/NFD) deliberately stays out of this step: it would change user-visible text, which an archival tool has no business doing. Fix what is broken, touch nothing else.

(String.prototype.toWellFormed() needs Node.js 20+. Before that, the surrogate scan has to be written by hand.)

War story 2: the tokens that counted themselves twice

The tool's token dashboard once reported usage numbers roughly 2.3× higher than reality, measured across a few hundred real sessions. The cause is a JSONL quirk worth knowing even if you never touch tokens:

One API response can become several JSONL lines. When a response contains multiple content blocks (text plus tool calls, for instance), Claude Code writes one assistant entry per block, and each entry carries a copy of the same usage object. Sum them naively and every multi-block turn is counted once per block.

The entries share a requestId, which is the dedup key. But there is a trap inside the trap: it is tempting to merge the entries into one logical message. Don't: entries of different requests can interleave on disk (streaming order), and merging would scramble the conversation. The entries themselves are fine; only the usage is duplicated.

So: keep every entry, zero out the usage on all but the last entry per requestId:

function deduplicateTokensByRequestId(lines: ParsedLine[]): ParsedLine[] {
  const lastIndex = new Map<string, number>()
  lines.forEach((line, i) => {
    if (line.role === 'assistant' && line.requestId) {
      lastIndex.set(line.requestId, i)
    }
  })
  return lines.map((line, i) =>
    line.role === 'assistant' && line.requestId && lastIndex.get(line.requestId) !== i
      ? { ...line, inputTokens: null, outputTokens: null,
          cacheReadTokens: null, cacheCreationTokens: null }
      : line,
  )
}

The general lesson: one JSONL line is not one logical event. Never assume a 1:1 mapping between physical lines and semantic units in a format you do not own.

War story 3: resumed sessions replay the past

When a Claude Code session is resumed, the new JSONL file starts with copies of messages from the original session: same uuid, same content, written again. Index both files naively and every resumed conversation shows up with duplicated history.

The remedy is UUID-level dedup against what is already indexed. The trap hiding inside that fix: the dedup query must exclude the session currently being indexed. Otherwise, re-indexing an existing session matches its own previously-indexed messages, concludes that every line is a duplicate, and quietly drops the entire session. A dedup check that can self-match is a data-loss machine with good intentions.

War story 4: screenshots will eat your index

Claude Code conversations can contain images: screenshots pasted into the prompt, arriving as content blocks with base64 data inline. Store message content verbatim and a handful of screenshots will outweigh thousands of text messages in the database.

The pattern: strip the payload, keep the shape.

if (block.type === 'image' && block.source?.type === 'base64') {
  return { ...block, source: { ...block.source, data: '[base64-stripped]' } }
}

The block structure survives, so the UI can still render an "image was here" placeholder at the right position, and a has_image flag stays queryable. Only the megabytes are gone. Same archaeology principle as everywhere else: preserve the evidence of structure, not necessarily every byte of payload.

The schema will move again

In case "the schema keeps moving" sounds abstract, here is what diffing real session files before and after one CLI release (v2.1.168, June 2026) turned up: top-level attribution fields on assistant entries (which skill, plugin, or MCP server produced a reply), image content blocks, structured system subtypes carrying API error status codes, and an edited_text_file attachment type. Four schema extensions, zero announcements. A normal month.

With the patterns above, absorbing that release was: extend the whitelist, extract the new fields, bump SUMMARY_VERSION, ship. The sessions written before the parser update backfilled themselves on the next sync. Nothing was lost in the weeks where the parser did not yet understand the new shapes: the unknown parts were sitting in the archive table, waiting.

Takeaways

1. Skip bad lines, never throw. The file is the historical record; the parser's opinion of it is irrelevant.
2. Whitelist known shapes; archive unknown ones raw. Storage policy and forward compatibility in one mechanism.
3. Version your derived data. Re-parsing should be a routine background event, not a migration.
4. Normalize at the ingestion boundary, exactly once, and only what is actually broken (toWellFormed, yes; NFC, no).
5. Distrust the line/event mapping. Duplicated usage across entries, replayed messages across files: physical lines lie.

To see these patterns in production context, the tool is open source: ccRewind on GitHub, a read-only, offline replay and search tool for Claude Code history. It never writes a byte to ~/.claude/. Why that constraint exists, and what it cost, is the next post.

Disclosure: this post was drafted with Claude and edited by the human who debugged every story in it. The drafting sessions are, naturally, JSONL files under ~/.claude/projects/ now.

Comments

[Raju Dandigam]

The “archaeologist, not a validator” framing is exactly right for any tool built on top of agent session history. Once the source format is an internal implementation detail, a parser that insists on cleanliness is basically choosing data loss. I also like the distinction between versioning derived data versus trying to pretend the upstream schema is stable, because that turns format churn into a routine backfill instead of a migration fire drill. The duplicated usage objects and resumed-session replay issues are especially good examples of why agent traces need structural inspection, not line-by-line assumptions. This is the same reason I care about local-first debugging tools like agent-inspect: the interesting bugs are often in the execution record itself, not just the final response. Have you found many teams underestimate how quickly “just parse the JSONL” turns into an observability problem?

[林子超（子超）]

Yeah, and the annoying part is none of the four war stories in the post ever threw an error. The parser "succeeded" every time. Lone surrogates parse clean until something downstream chokes on them. Duplicated usage sums fine, it's just wrong. The self-match dedup bug didn't crash, it quietly deleted a whole session. Screenshots don't fail to store, they just make the database fat.

I'd frame it less as teams underestimating observability and more as a mismatch in finish lines. Parsing has one: valid JSON, correct types, done. Observability doesn't: correct semantics, held over time, through schema drift. Nothing forces the switch from one mindset to the other until a number downstream looks wrong, and by then it's decoupled from the parsing code, so nobody thinks to look there first.

Going to go read agent-inspect properly. Local-first, no dashboard sounds like the same bet this project made: the record on disk is more trustworthy than any live view of it, because you can replay it after the fact.

[Skillselion]

The whitelist-as-storage-policy point is the one I would frame more loudly, because it quietly solves a versioning problem too. One thing worth flagging for anyone building an index on top: the file is append-only, but it is not a flat one-line-per-turn log. Compaction appends a summary entry rather than editing history, and a rewind forks a new session file with a fresh sessionId, so any assumption that byte offsets map cleanly to turns breaks the first time someone compacts or branches. Keying on uuid plus parentUuid instead of position handles that, and it also lets you reconstruct the actual tree rather than a flat list. On the unknown-type archive, capping identifier fields at 128 chars is good, and I would also stash the top-level version string next to the raw line, so when you later learn a shape you can tell which release introduced it instead of bisecting by date.

[林子超（子超）]

Ugh, yeah. Went and actually checked the code instead of just agreeing.

On ordering: parentUuid gets parsed off each line, but the indexer drops it before it ever hits the database. Ordering today is just sequence position. So you’re right, once a session compacts or rewinds, position stops meaning turn order. That’s a live bug, not a hypothetical.

Compaction and rewind get zero special handling right now, worse than I expected honestly. A rewind fork is just a new file with nothing pointing back to where it branched from. A compaction summary sits in the timeline looking like a normal turn.

Same story with version. It’s sitting right there in every raw line, nothing extracts it. So “which release introduced this shape” really does mean bisecting by date today, exactly like you guessed.

All three are going on the list now: uuid/parentUuid for actual tree reconstruction, tagging compaction/rewind instead of assuming everything’s linear, and keeping version next to whatever gets archived. Can’t promise when, but it’s written down now instead of just implied by the whitelist.