A negative result, its root cause, and the feature it produced.
My last post introduced ctxfold — lossless, structure-aware compression for the bulky stuff we put in LLM prompts. Its benchmark table had one cell I wasn't proud of:
| CSV / TSV | char reduction | ~30–45%* |
*CSV readability not yet validated against a model
JSON and logs had real numbers: a model answering lookup questions off the folded form, scored against exact ground truth, matching raw field-for-field. CSV had a character count and an asterisk.
So I wrote the same harness for CSV. Generate 400 records with realistic redundancy, fold them, ask GPT-4o-mini to look up specific records in both forms, score against ground truth.
Raw CSV: 24/24. Folded CSV: 0/24.
Not "slightly worse." Zero.
Maybe the model is too small?
First hypothesis: capability threshold. Re-ran on GPT-4o.
Raw: 24/24. Folded: 6/24. A later run: 9/24. So no — a stronger model helps a little and inconsistently. The format is the problem.
The failure had structure, which is what made it diagnosable. Asked about record NW-1258, the model reported a warehouse of WH-2 (a half-applied prefix), a supplier of DAL-2 (a value from the warehouse column), and a price from some other row entirely. In another run it answered qty 1238 for sku NW-1238 — it read the record's own identifier back as data. Two distinct failures, every time: it couldn't find the right row, and it couldn't reconstruct the values in it.
The root cause is embarrassing in hindsight
ctxfold's JSON and logs encoders fold syntax. A JSON array of objects repeats every key on every record; the encoder lifts keys, braces, and quotes into a one-time header, and every value stays verbatim in its row. Logs are the same story with templates: timestamps, levels, reqId= prefixes get lifted; the payload stays put. The model reads a plain labeled table containing exactly the values it needs.
CSV has no syntax to remove. No keys, no braces — it's already a compact table. The only redundancy left is inside the values: shared prefixes like NW- on every sku, WH- on every warehouse, a constant USD column. So the CSV encoder factors those out, and each row keeps only the varying middle. Lossless, byte-exact, verified on every encode.
And unreadable. To answer a question, the model has to find a row by a partial key (the sku column only contains 258, because NW-1 was factored out — all 400 skus shared it) and then reconstruct every value through the header's prefix + middle rules. It can't do either reliably. On GPT-4o-mini it essentially can't do it at all.
I'd actually seen a milder version of this before. ctxfold's opt-in dictionary coding (low-cardinality values become small integers, mapped once in the header) pushes JSON savings from ~39% to ~46% — and in testing, models resolved the coded values slightly less reliably than plain ones. That's why it ships off by default. The CSV result is the same phenomenon at full strength:
Models read values, not reconstruction rules. Indirection through a header costs readability — and in CSV's case, the savings were the indirection.
What I didn't do
I considered "fixing" the format — don't factor unique columns, keep identifiers whole. But that only patches row-finding; the prefix-reconstruction failure remains, and the savings shrink toward nothing. CSV is already near its readable minimum. That's why there was nothing safe to fold.
So the fix was documentation, not code. As of v0.1.4, CSV folding is pipeline-mode: fold it for lossless transit or storage, call decompress() before the prompt is built, and the model never sees the folded form. For direct model reading, send CSV raw. The benchmark table now says so, with the measured numbers where the asterisk used to be. JSON and logs remain validated direct-readable — their folded output keeps every value intact, and the scores show it.
The rule I keep coming back to: ctxfold's core contract is lossless or no-op — never lossy. It turns out the same discipline applies to claims. Either a readability claim has a measurement behind it, or it gets the asterisk. And when the measurement comes in against you, the asterisk becomes documentation.
The feature the failure produced
Sitting with those numbers, the useful question turned out to be: before anyone folds anything — where do a prompt's tokens actually go, and what's safely foldable?
v0.2.0 ships that as ctxfold --profile:
$ ctxfold --profile users.json
[ctxfold profile]
format JSON array — 300 records × 7 fields
size 65,614 chars ≈ 16,404 tokens (estimated; pass a tokenizer for exact)
where the characters go
keys 27% repeated field names (with quotes)
syntax 7% braces, brackets, commas, colons
values 36% the data itself (with string quotes)
whitespace 30% indentation and spacing
foldable (lossless, verified by round-trip)
fold -66% direct-readable — validated 24/24 vs raw
+ --dictionary -73% readability tradeoff — off by default, see README
verdict: fold it — 16,404 → ~5,595 tokens (~66% fewer)
That's a real API response shape — wrapped array, pretty-printed. 64% of the file is structure and formatting, which is why the fold is fat. On a CSV file, the same command reports the fold as pipeline-only and tells you to send it raw or decompress first — the negative result is baked into the tool's own advice.
The profiler follows the rules the failure taught:
- Composition is measured in characters and attributed exactly — the categories sum to the input size. Attributing individual tokens to categories would be false precision, so token figures are totals, marked estimated unless you pass a real tokenizer.
-
The "foldable" numbers come from actually running the encoder on your input. The profiler cannot promise more than
compress()delivers — that's an invariant with a test on it, not a policy. - If nothing folds, it says why — quoted CSV, nested JSON, too few records, prose — instead of silently no-opping.
Try it with zero setup (no API key, deterministic output):
git clone https://github.com/antrixy/ctxfold && cd ctxfold
node examples/profile-demo.js
Or on your own data:
npm install -g ctxfold
ctxfold --profile your-file.json
The part I'd generalize
If you maintain a tool that makes claims — savings, accuracy, compatibility — the cheapest test you'll ever write is the one that checks your own table. Mine took an afternoon, cost a few cents of API calls, and found that a third of my format list didn't do what a reader would assume. The fix cost nothing but honesty, and the tool that fell out of it is the most useful thing in the package.
The full harness is in the repo (examples/gpt-csv-equivalence.js), along with the v0.1.4 and v0.2.0 release notes. If you push structured data into prompts and your payloads break my assumptions, I want to hear about it.
Repo & docs: https://github.com/antrixy/ctxfold · npm: npm install ctxfold · MIT licensed.
Top comments (10)
The 0/24 result is a useful failure mode, honestly. Compression schemes that look clever to the author often lose the reader at the first missing affordance. I'd probably keep the test harness and add a boring baseline next to it: original text, gzip-ish binary, and your format, then measure where humans stop trusting the reconstruction.
Agreed on the boring baseline — with one split that turned out to be the whole finding. For pipeline use (fold → transit → decompress before the model reads), gzip is the honest baseline and frankly wins: 80–90% on redundant CSV vs my 24–28%, and every transport layer does it free. That's exactly why "CSV folding is pipeline-mode" is a weak consolation prize and the README now says to just send CSV raw for direct reading.
Where gzip can't play is the direct-read case: the model consumes the folded form as the prompt, no decompress step. That's the only regime where ctxfold earns its keep, and it's the regime the harness tests — JSON/logs pass because their folded output keeps every value verbatim; CSV failed because its folding factors the data itself.
"Where humans stop trusting the reconstruction" is a lens I hadn't considered — my scorer measures where the model stops reconstructing correctly, but human debuggability of the folded form is a real adjacent property (you do end up eyeballing these payloads in logs). Noting it.
Since publishing, I've run the same harnesses across three more model families: Claude (Haiku 4.5), Llama-3.3-70B, and Qwen3.6-27B (a reasoning model). Full table in the README's cross-model section; short version:
JSON folding: 24/24 on every model, both forms. Rock solid.
Logs: no non-reasoning model counts raw logs exactly, but every one of them got closer to truth on the folded columnar form. The reasoning model was exact on both.
CSV (the 0/24 in the post): it's more interesting now. Llama fails worse than GPT (lost record alignment entirely, 0 scoreable answers). Qwen, the reasoning model, can fully reconstruct the fold — 24/24, derivation visible in its thinking — but spends 3–6k variable-length reasoning tokens to undo a 15–25% prompt saving. The fold doesn't delete the reconstruction work, it moves it into the model. And Claude Haiku 4.5 read the folded CSV 24/24 with no visible reasoning overhead — the only model to just read it.
So the honest current picture: folded-CSV readability is model-dependent, from 0/24 to 24/24 across four families. That's exactly why the recommendation in the post doesn't change — pipeline-only, decompress before the model reads it — unless you've pinned your model and measured it yourself. The harnesses in the repo take any OpenAI-compatible endpoint if you want to test yours.
This is fascinating! How do you even define 'readability' for a compression format itself, rather
Operationally: readability = the model answers exact-match lookup questions off the folded form as accurately as off the raw form. Concretely — generate N records with known ground truth, fold them, ask the same "for SKU X, report price/qty/warehouse/supplier" questions against both forms, score field-by-field against truth. Raw scored 24/24; folded CSV scored 0/24 (gpt-4o-mini) and 6–9/24 (gpt-4o). Same harness previously validated JSON and logs at parity with raw.
So it's deliberately not "can the format be decoded" (it can — losslessly, verified on every encode) but "can the model use it in place without a decode step." That gap is the entire finding. Harness is in the repo: examples/gpt-csv-equivalence.js.
(Your comment looks cut off at "itself, rather" — happy to answer the rest if there was more!)
Were the 24/24 JSON scores all lookup style questions? I'd be curious how folded JSON holds up on aggregation, like summing a column across 400 rows, where the model has to touch every record instead of finding one. If that survives too, that's a much stronger claim than lookups alone.
Yes — the JSON 24/24 was lookup-style, fair catch. Aggregation did get tested this week, but on the logs format: count every ERROR line across the file + name the top service, i.e. touch-every-record. Across GPT, Llama, and Qwen the folded form was never worse — the non-reasoning models actually landed closer to ground truth reading folded than raw, and a reasoning model was exact on both (table in the README).
JSON aggregation specifically is a real gap, and I'll add it — with one tweak to your version: exact sums over 400 rows mostly measure LLM arithmetic (models fail that on raw data too), so I'll use filtered counts and max/min style questions where ground truth is checkable and raw performance is a meaningful baseline. Will report the numbers either way, as usual.
The thing your CSV result actually exposes is that "lossless" and "readable" are orthogonal axes that everyone quietly collapses into one. Your JSON encoder isn't readable because it's lossless — it's readable because the transform it applies happens to leave every value at a stable, addressable position. Lifting keys is a positional identity op on the data cells. Prefix-factoring is not; it rewrites the cells themselves. Same losslessness guarantee, completely different readability contract, and nothing in a round-trip test can tell them apart. That's why the round-trip invariant sailed through while a reader scored 0.
Which suggests the profiler could carry one more axis you haven't named yet: not "how much folds" but "does the fold preserve value addressability." Key-lifting and template extraction keep it; dictionary coding and prefix-factoring break it. That distinction predicts your pipeline-only verdict mechanically instead of per-format — a nested-JSON case that factors shared string prefixes would fail for the same structural reason CSV does, and right now I don't think anything in the tool would flag it before a human reads garbage out of a prompt.
This is a better framing than the one I'd been using, and my cross-model runs from this week accidentally confirm it: a reasoning model (Qwen3.6-27B) can read the folded CSV — it scores 24/24 — but only by spending 3–6k thinking tokens visibly re-deriving the prefix rules. The reconstruction work doesn't disappear when the fold breaks addressability; it gets billed to the model. Cells-intact transforms (key-lifting, the logs columnarization) cost nothing extra across every model I tested. Full table here: github.com/antrixy/ctxfold#cross-m...
Taking both suggestions: an addressability flag per transform in the profiler output (it's a static property of each encoder, so it's cheap and mechanical), and treating "addressability-preserving" as a hard constraint on the v0.3 nested-JSON work — path-lifting only, no string-prefix factoring. Thanks — this comment is going straight into the roadmap.
Wonderful i'm glad that i could contribute.