Building Property-Tested Markdown Compression for LLM Context Windows

When building context layers for AI coding agents, maximizing token efficiency is critical. But if your compression pipeline corrupts a single code block delimiter, the entire agent run fails.

In TokenCap, we built src/compress/compress.js with strict preservation gates verified by property-based testing.

The Preservation Testing Contract

In test/compress-preserve.property.test.js, we generate randomized markdown documents containing complex code fences, file paths, URLs, and JSON objects:

test("preserves fenced code blocks and URLs byte-exact during compression", () => {
  const sample = fs.readFileSync("test/fixtures/complex-markdown.md", "utf8");
  const compressed = compressMarkdown(sample);

  const originalBlocks = extractCodeBlocks(sample);
  const compressedBlocks = extractCodeBlocks(compressed);

  assert.equal(originalBlocks.length, compressedBlocks.length);
  for (let i = 0; i < originalBlocks.length; i++) {
    assert.equal(compressedBlocks[i].content, originalBlocks[i].content);
  }
});

Dry-Run Compression Preview

You can test compression on your existing .tokencap/snapshot.md before applying:

tokencap compress .tokencap/snapshot.md --dry-run

CLI Output:

Original size: 18,420 tokens (74.2 KB)
Compressed size: 11,250 tokens (45.1 KB)
Savings: 38.9% reduction
Code blocks preserved: 14 / 14 exact matches

Learn more about safe context compression at tokencap.vansharora.app