DEV Community: VANSH ARORA

Designing a Local Learning Ledger with No Transcripts and No Telemetry

VANSH ARORA — Thu, 12 Nov 2026 15:29:16 +0000

TokenCap builds task-scoped context packs for AI coding agents. Retrieval is good but not perfect, so v3.1 adds a way to tell it when a pack was right or wrong. The constraint we kept throughout: the ledger records only explicit, bounded evidence, and it records it locally.

Recording a judgment

After reviewing a pack, record what happened:

tokencap analyze --learning --outcome helped \
  --task "invoice rounding" \
  --files src/billing/invoice.js,src/billing/round.js

If the pack left something out:

tokencap analyze --learning --outcome neutral \
  --task "invoice rounding" \
  --missing src/billing/currency.js

The CLI confirms what it did and what it did not do:

Recorded explicit focus-feedback (helped). Local only; no automatic behavior changed.

--outcome accepts helped, neutral or hurt. Anything else is rejected.

What a record contains

Each event in .tokencap/learning/ledger.json has an ID, a timestamp, a kind, a source (cli or companion), an outcome, a task label, and two file lists: selected and missing. That is the full schema.

The bounds are enforced in src/learning/ledger.js:

Task labels are capped at 300 characters, with newlines and tabs collapsed.
Each file list keeps at most 30 entries, deduplicated.
Only workspace-relative paths are accepted. Absolute paths, drive letters, UNC paths, ~ paths, URL schemes and anything containing .. are dropped.

Redacting the task label

A task label is free text, so it can contain things that should not be stored. Two passes run before it is written:

Paths are scrubbed. A path inside the workspace is rewritten as workspace-relative. A path outside it becomes [path]. POSIX paths are only matched under system roots such as /Users or /home, so an API route like /users/:id stays readable.
The label goes through TokenCap's central redaction engine, the same one that redacts packs.

If redaction throws, the label is dropped rather than stored unredacted.

What it never does

There is no transcript capture. The ledger does not accept chat history and does not infer outcomes from what an agent did. There is no telemetry; the file stays on disk and nothing reads it over the network. And there are no automatic changes. Recording feedback does not edit rules, the Constitution, or default ranking.

The ledger is also not shared. Team rules in .tokencap/team/ are meant to be committed. The learning ledger stays ignored with the rest of .tokencap/, so one developer's judgments do not end up in everyone's history.

Reading it back

tokencap analyze --learning

This prints a count of helpful, neutral and harmful focus events, rule feedback events, and the ten most recent records with their files. Add --json to get the full report, which states local: true and automaticChanges: false in the output itself.

The companion signal

The Chrome companion can mark a file Useful or Missed for the current task. It sends that to the local loopback bridge's /feedback endpoint, which writes the same kind of event with source: "companion". The bridge only accepts feedback for a project that has been indexed, named by absolute path.

Where the evidence is used

The ledger is input for two opt-in features: tokencap ask "<task>" --adaptive, which can re-rank candidates using matching feedback, and tokencap analyze --patterns, which reports file pairs that keep appearing together. Both are covered in the next post. Neither runs unless you ask for it.

Docs: tokencap.vansharora.app

Team Rules for AI Agents, Stored as Append-Only Files in Git

VANSH ARORA — Wed, 11 Nov 2026 14:37:19 +0000

Shared project knowledge usually needs a shared place to live. For TokenCap v3.0 we wanted that place to be the repository itself, with no server, no account and no sync service. The result is tokencap analyze team, and its storage format is the interesting part.

The layout

Everything lives under .tokencap/team/:

.tokencap/team/
  team.json
  rules/rule-<id>.json
  decisions/decision-<id>.json
  approvals/approval-<id>.json
  profiles/<hash>.json
  adrs/adr-<hash>.json

team.json holds the team name and a schema version. Every rule, decision and approval is its own file, named by a random UUID. A record holds its text, author and creation time. Nothing else is stored.

The commands

tokencap analyze team init --name "Platform"
tokencap analyze team rule --author "Ava" --text "Changes to auth require review."
tokencap analyze team approve-rule --id <rule-id> --author "Ben"
tokencap analyze team decision --author "Ava" --text "Use local Git records."
tokencap analyze team profile --author "Ava" --text "Owns authentication."
tokencap analyze team sync-adrs
tokencap analyze team status --json

Each write prints the same reminder:

Team intelligence: rule complete. Commit .tokencap/team/ to share it through Git.

TokenCap never runs that commit for you. You do it:

git add .tokencap/team .gitignore
git commit -m "chore: share TokenCap team intelligence"

Why one file per record

The obvious design is a single rules.json with an array in it. It works until two people add a rule on two branches. Both edit the same lines of the same file, and the merge stops for a human.

With one file per record, Ava's rule and Carol's rule are different files with different names. Git merges both branches cleanly because neither branch touched a file the other changed.

Approvals follow the same idea. approve-rule does not edit the rule. It writes a new approval-<id>.json that points at the rule's ID. status reads every approval and reports each rule as proposed or accepted, with the list of approvers. A rule with two approvals has two approval files, and the Git log shows when each one arrived.

Records are never rewritten after creation. The one exception is profiles: each contributor has one profile file, and profile replaces it.

Getting Git to see the directory

Many repositories ignore .tokencap/ entirely, since most of it is generated. analyze team init and analyze team status add a managed block to .gitignore that re-includes only .tokencap/team/. In v3.0 the block re-included more than that, so git add could pick up generated artifacts. v3.1 narrowed it, and running init or status rewrites an old block in place.

What it does not do

TokenCap records evidence. It does not block a commit when a rule is unapproved, and it does not resolve a disagreement between two people. If two engineers propose contradicting rules, both files exist until the team decides what to do about them. That decision is a conversation, and then a commit.

sync-adrs copies your existing architecture decision records into adrs/, one file per ADR, keyed by a hash of its content. Running it twice adds nothing new.

The whole feature is files plus Git. Review it in a pull request like any other change.

Docs and install: tokencap.vansharora.app

Writing a TokenCap Plugin, and What the Sandbox Stops It From Doing

VANSH ARORA — Tue, 10 Nov 2026 11:27:11 +0000

TokenCap 3 added a local plugin runtime. The contract is small on purpose: two hooks, named tools, and one capability, which is reading and writing files in the plugin's own folder. This post walks through a plugin and then through each limit, with how it is enforced in src/plugins/runtime.js.

Registering a plugin

Plugins are opt-in per project, from that project's own package.json:

{
  "tokencapPlugins": ["./tools/tokencap-plugin.js"]
}

Each entry must be a workspace-relative path that starts with ./ and points to a file inside the workspace. Package names and remote modules are refused.

A plugin file

// tools/tokencap-plugin.js
module.exports = {
  name: "build-log",
  version: "1.0.0",
  hooks: {
    afterBuild(ctx) {
      ctx.artifacts.write("last-build.json", JSON.stringify(ctx.build));
      return { logged: true };
    },
    beforeRedact(ctx) {
      return ctx.content.replace(/internal\.corp\.example/g, "[internal-host]");
    },
  },
  tools: [
    {
      name: "last-build",
      handler(ctx) {
        return ctx.artifacts.read("last-build.json");
      },
    },
  ],
};

The runtime validates the shape when it loads the file. The name must be 1 to 64 characters of letters, digits, - or _, starting with a letter or digit. A version is required. Any hook other than afterBuild or beforeRedact is rejected, and two plugins with the same name or the same tool name are refused.

The two hooks

afterBuild runs after tokencap make. ctx.build holds outputPath, fileCount and estimatedTokens. When any plugin hook ran, the build prints one summary line:

Plugins:                     1 hook(s), 0 error(s)

beforeRedact receives ctx.content and ctx.filePath and runs before TokenCap's own redaction. It is called synchronously, so return the new string (or { content }) directly. If a plugin throws or returns an error, its change is skipped and core redaction runs on the original content. The core engine stays authoritative.

Tools

tokencap analyze plugins list
tokencap analyze plugins run last-build --text "input"

list prints Plugins: 1 loaded, 0 error(s). run passes --text to the handler as ctx.input.text. Add --json to either command for the full result.

What the sandbox enforces

A separate realm. Each plugin gets its own V8 context with a null-prototype global, so this.constructor cannot resolve to the host's Object. Code generation from strings and WebAssembly code generation are both turned off. Host callbacks stay in a closure the plugin cannot reach. Payloads go in as JSON strings and come back as JSON, and the payload object a hook receives is frozen. The test suite tries six constructor paths back to the host realm, and all six come back blocked.

No require, no process, no network. Inside a plugin, typeof require and typeof process are both "undefined". The only I/O is ctx.artifacts.read, write and list.

One folder. Every artifact path resolves under .tokencap/plugins/<name>/. Empty paths, .. and absolute paths are rejected. Each existing path component from .tokencap down is checked with lstat, and a symlink or junction anywhere on that path is an error. The source names the case it is written for: a committed .tokencap/plugins/<name> link pointing at ~/.ssh. The check runs before and after the directory is created, and the file is opened with O_NOFOLLOW where the platform supports it.

A clock. Each hook or tool call has 1 second, and loading a plugin has 150 ms. Promise callbacks run inside the timed call, so an async hook cannot slip past the limit; a call still pending after it is reported as not completed. A plugin that overruns is disabled until its file changes or the process restarts.

That last rule is a 3.1 fix. Before it, a looping hook could hang make with the build lock held.

The limit we state up front

The runtime's own header says it plainly: node:vm is not a security boundary, and plugins are trusted code. The rules above close the known host-object escapes and stop the common accidents: a path that walks out of its folder, a committed symlink, an infinite loop. They are not designed to contain a determined attacker running code on your machine.

So the usage docs end the plugin section the same way this post does: install plugins you trust.

Plugin docs at tokencap.vansharora.app

TokenCap 3.0: One Build, Every Tool Reads It

VANSH ARORA — Thu, 05 Nov 2026 14:50:56 +0000

The 3.0 release was named "Platform". The change is less about new analysis and more about where the output goes. One local build produces .tokencap/, and every surface reads from it: coding agents, MCP hosts, the Chrome companion, an SDK, and plugins.

The build

tokencap make

Among other things, a full build writes:

.tokencap/agent/START_HERE.md, the entry point an agent reads first
.tokencap/brain/knowledge.json, the knowledge base
.tokencap/graph/graph-data.json, the dependency graph
allowed-context.json and context-manifest.json, which point an agent at task-relevant files
the snapshot

Everything stays on disk. In v3.1, tokencap update, which runs npm, is the only command that uses the network.

Reader 1: coding agents

Agents that read files start at START_HERE.md. make also creates AGENTS.md when it is missing, and as of 3.1 it leaves a hand-written AGENTS.md alone. The npm package ships an Agent Skill at skills/tokencap/ for hosts that discover skills.

Reader 2: MCP hosts

tokencap serve --mcp --init
tokencap serve --mcp --init --client vscode

--init detects the host and writes its config where that host reads it: .mcp.json for Claude Code, .cursor/mcp.json for Cursor, .vscode/mcp.json for VS Code, .codex/config.toml for Codex. Claude Desktop and Windsurf use per-user files, so their entry adds --root <repo>.

The server answers initialize at once and builds in the background. Tools such as tokencap_overview, tokencap_files, tokencap_impact and tokencap_review read the same intelligence. In 3.1, tokencap_files uses the same retrieval as tokencap ask, so MCP and task packs agree on file order.

Reader 3: the Chrome companion

tokencap serve

This binds a loopback bridge on 127.0.0.1:4545 with a pairing token and an origin allowlist. The companion gets a compressed, redacted, task-scoped pack. It never receives the raw .tokencap/ files.

Reader 4: the SDK (experimental)

@tokencap/sdk is a read-only interface to the generated artifacts:

const { loadIntelligence, getImpact, buildContext } = require("@tokencap/sdk");

const intelligence = await loadIntelligence("/workspace/my-project");
const impact = getImpact(intelligence, "src/payments/charge.js");
const context = buildContext(intelligence, {
  task: "Review the charge retry path",
  budget: 5000,
});

It needs brain/knowledge.json and graph/graph-data.json to exist, and reports TOKENCAP_INTELLIGENCE_MISSING when they do not. It does not trigger a build, start a watcher, write files, or make network calls. Graph and impact answers are static evidence, and context token totals are character-based estimates.

It is not published to npm. npm install @tokencap/sdk is not a supported setup path today. The source ships with TokenCap for evaluation, and it will be published only after external projects show the small API is stable.

Reader 5: plugins

A project can list local plugin files in its own package.json:

{
  "tokencapPlugins": ["./tools/tokencap-plugin.js"]
}

Plugins can provide afterBuild and beforeRedact hooks and named tools:

tokencap analyze plugins list
tokencap analyze plugins run <tool-name> --text "input"

They run with no require, process, or network, and can only read and write inside .tokencap/plugins/<name>/. The sandbox gets its own post.

Team intelligence

Shared rules, decisions and approvals live in Git under .tokencap/team/ as append-only records. No server, no cloud account, no commits on your behalf:

tokencap analyze team init --name "Platform"
tokencap analyze team rule --author "Ava" --text "Changes to auth require review."

Upgrading from v2

tokencap make --migrate
tokencap make

--migrate upgrades .tokencap/config.json to version 3, publishes schemas in .tokencap/schemas/, and writes .tokencap/migration.json. The next make produces the v3 artifact manifest.

The CLI is still seven commands: make, ask, analyze, serve, update, help, version. The v2 standalone aliases are gone and print their replacement instead (tokencap impact now points you to tokencap ask impact). The one exception is tokencap mcp, which still starts the MCP server so existing host configs keep working.

Docs and migration table at tokencap.vansharora.app

Measuring Context Packs Against Real Commits: Recall, Tokens, and the Limits

VANSH ARORA — Wed, 04 Nov 2026 14:46:16 +0000

Most "fewer tokens" claims compare a small number to a big number and stop there. That rewards a tool for leaving things out. The TokenCap benchmark in benchmarks/fair/ scores two things for every task: the token count, and whether the files the change actually needed are in the context.

Where the tasks come from

Each task is a real commit from one of nine repositories:

fastify-example-todo (JavaScript)
full-stack-fastapi-template (Python + TypeScript)
hoppscotch (TypeScript + Vue)
flask and httpx (Python)
gin and cobra (Go)
ripgrep (Rust)
gson (Java)

The task text is the commit subject. The ground truth is the set of non-test source files the commit changed. The repository is evaluated at the commit's parent, so the change does not exist yet.

The selection rule is recorded in tasks.json. Per repository: the most recent non-merge commits that change 1 to 5 non-test source files (at most 8 files in total), are not dependency, release, docs, lint or translation commits, and have a subject of at least three words. Commits were listed by script before any measurement, and none was dropped after seeing results.

50 tasks were used while changing retrieval and packing code. 50 were held out. Two more selected commits are skipped because none of their files exists at the parent.

The three arms

TokenCap task pack: what tokencap ask "<task>" builds from the commit subject. The companion bridge builds the same pack.
Naive, all matching files: keyword grep over the repository, every matching file read in full.
Naive, 10 best-matching files: the top ten files from that grep.

Results (all 100 tasks)

Measure	TokenCap task pack	All matching files	10 best-matching files
Needed files present	87%	96%	76%
Average tokens per task	9,997	382,258	87,761
Median saving vs. that baseline	n/a	18.9×	7.5×

On average, 73% of the exact lines a commit edited are in the pack.

Split by set: the held-out half scored 88% needed files, 74% edit-line coverage, and an 18.4× median saving. The tuning half scored 86%, 72%, and 20.4×. One caveat on the held-out side: the held-out tasks of the first three repositories were also run during an earlier round of work, before the other six repositories were added.

Running it

node benchmarks/fair/fetch-repos.js ../bench-repos
node benchmarks/fair/run.js --repos ../bench-repos --set holdout

CI runs the full set with --set all --gate. The gate fails the job when needed-file recall, edit-line coverage, or the median saving drops below the thresholds stored in tasks.json.

The limits

These are published next to the table in docs/HONEST-NUMBERS.md, not in a footnote.

It is not an answer-quality measurement. No model is called. A pack can contain the right file and still lead to a wrong answer. The harness for that check, benchmarks/fair/answer-eval.js, exists but has not been run, and there is no published model result.
Grepping everything finds more. 96% against 87%. The pack wins on tokens (38× fewer on average) and beats the 10-file baseline on both counts.
Spans can miss the edit. Large files ship as selected functions and lines. Edit-line coverage is lowest on repositories with very large files: 56-58% on cobra, ripgrep, and gson.
Small repositories gain little. The median saving on the small repository is 1.6×, and in 3 of its 9 tasks the pack was larger than the naive read.
Median, not mean. The mean saving is 34.2×, pulled up by the large repository. The median, 18.9×, is the representative figure.
Task text is terse. Commit subjects are shorter than what a developer usually types.
Scope. This measures the task pack built by tokencap ask and the companion bridge. It does not measure the tokencap make capsule.

Why it was worth building

Before the v3.1 retrieval changes, the first three repositories scored 25% needed-file recall on this benchmark. Those result sets are still in benchmarks/fair/results/; the 100-task results are in benchmarks/fair/results/v2/.

A benchmark that only counts tokens would have rated that 25% version as a success. This one did not, which is the whole point of having ground truth.

Full table, method, and limits at tokencap.vansharora.app

We Retracted Our Own 12.2 Benchmark. Here Is What Was Wrong With It.

VANSH ARORA — Tue, 03 Nov 2026 11:07:41 +0000

TokenCap v1.6 shipped with a line we liked a lot: "12.2× fewer tokens, same answer." It came from benchmarks/run.js. It was wrong in five separate ways, and it is now retracted in docs/HONEST-NUMBERS.md.

This post is the list, and what replaced it.

The five problems

1. "Same answer" was never measured. The harness printed the phrase. It called no model and checked no answer. The words were a label, not a result.

2. The naive baseline counted TokenCap's own output. The capsule was written to TOKENCAP.md inside the repository. The naive side then read the repository, capsule included. Every naive figure carried the capsule's full size, which inflated the baseline we were claiming to beat.

3. One task matched every file. Keywords were matched against the absolute path. On Windows the temp path starts with C:\Users\..., and user was one of the keywords. So one "naive" read was simply the whole repository.

4. TokenCap's side was not task-specific. One capsule per repository was reused for all five tasks. The 15 data points were really three measurements.

5. The two headline figures used different aggregations. 12.2× was the mean of per-task ratios. The 94% figure came from the ratio of the averages. Two numbers, two methods, side by side.

None of these needs bad intent. Each one is a shortcut that looks fine until someone reads the harness carefully.

What the corrected number looked like

With the baseline fixed, the small repository's capsule was 8,275 tokens. The naive read averaged 3,880 tokens. The capsule was about twice the size of what it was supposed to save.

That is exactly the kind of result a benchmark exists to surface. The old one could not surface it.

What replaced it

benchmarks/fair/ scores both sides against ground truth instead of against each other's size:

Each task is a real commit from one of nine repositories (JavaScript, TypeScript, Vue, Python, Go, Rust, Java).
The task text is the commit subject. The ground truth is the set of non-test source files the commit changed.
The repository is checked out at the commit's parent, so the answer is not in the tree yet.
Commits are selected mechanically by a rule recorded in benchmarks/fair/tasks.json. 50 were used while changing retrieval and packing code. 50 were held out.
Both sides are scored on tokens and on whether the needed files are present. A smaller pack only counts if it still holds what the task needs.

Results across all 100 tasks:

Measure	TokenCap task pack	All matching files	10 best-matching files
Needed files present	87%	96%	76%
Average tokens per task	9,997	382,258	87,761
Median saving vs. that baseline	n/a	18.9×	7.5×

Note the first row. Reading every matching file finds the needed files more often than the pack does. We publish that next to the saving, not under it.

What changed in how we publish

A CI job runs the benchmark and fails when needed-file recall, edit-line coverage, or the median saving drops below the thresholds in tasks.json.
The limits sit next to the numbers: no model is called, grepping everything finds more, small repositories gain little, and the median is quoted because the mean (34.2×) is pulled up by the large repository.
The old script and its result file stay in benchmarks/, for the record only.
Before the v3.1 retrieval changes, the first three repositories scored 25% needed-file recall on this same benchmark. That earlier result set is kept in benchmarks/fair/results/ as well.

That 25% is the number I would point to if someone asked whether the rebuild was worth it. The old benchmark could never have shown it, because it never asked whether the right files were in the context at all.

The full table and limits list are at tokencap.vansharora.app

Announcing TokenCap: The Local-First Context Layer for AI Coding Agents

VANSH ARORA — Thu, 22 Oct 2026 18:07:15 +0000

Today, we are officially launching TokenCap: a zero-network-egress repository intelligence platform designed for developers building and pairing with AI coding agents.

What is TokenCap?

TokenCap bridges the gap between raw source code and LLM context windows. Instead of blindly dumping full files or relying on fuzzy text search, TokenCap parses your codebase into an AST symbol graph, clusters modules, and budgets context deterministically.

Core Verified Benchmarks and Capabilities

12.2x Token Reduction: Compresses typical 150k-token repository payloads down to 12.5k tokens without dropping critical call signatures.
Even-Span Sampling: Proportional sampling across file heads, bodies, and exports that eliminates line-300 cutoffs.
Spatial Impact Analysis: tokencap ask impact <file>:<symbol> calculates caller blast radius and risk before an agent modifies code.
The Four Sovereignty Laws: Zero runtime network egress, zero API keys, no compilation build steps, and under 4 dependencies.
Multi-Surface Availability:
- CLI: npx tokencap make
- VS Code Extension: Auto-capture and spatial menus
- Local Bridge & Companion: Direct loopback injection into ChatGPT, Claude, and Gemini
- Agent Skill: Native discovery for OpenCode, Claude Code, Cursor, and Antigravity

Quick Start

Install globally or run directly with npx:

# Generate your repository intelligence snapshot
npx tokencap make

# Start the local loopback bridge for browser companions
npx tokencap serve --daemon

# Query the blast radius of a function
npx tokencap ask impact src/auth.js:verifyToken

Explore the documentation, read the architecture specs, and install the extensions at tokencap.vansharora.app

How We Packaged TokenCap as a Self-Discoverable Agent Skill

VANSH ARORA — Thu, 22 Oct 2026 12:13:04 +0000

When onboarding a new software engineer, you provide onboarding documentation. When onboarding an AI agent, developers often rely on ad-hoc prompts or complex custom scripts that break across different platforms.

With the TokenCap Agent Skill system in skills/tokencap/, any agent supporting open skill standards can discover, query, and respect repository intelligence without prior manual setup.

The Skill Specification

The skill is built around three standard files:

SKILL.md: Root declaration with YAML frontmatter, trigger conditions, and high-level workflows.
reference/commands.md: Exact CLI and MCP commands for snapshot generation, impact analysis, and refactoring.
reference/workflows.md: Step-by-step playbooks for PR review, onboarding to unfamiliar codebases, and blast-radius audits.


yaml

Why AI Context Packs Must Adapt Dynamically to Target Model Economics

VANSH ARORA — Thu, 22 Oct 2026 08:46:40 +0000

Different LLMs have wildly different context windows and cost economics. Providing a generic 20k-token context pack is wasteful for smaller models and insufficient for massive 200k+ models.

In TokenCap, we implemented model-aware budgeting in src/bridge/generate.js.

How Token Allocations Adjust

gpt-4o (Target: ~16k to 32k tokens): Balanced span sampling, key symbol definitions, priority diffs, and compressed helper functions.
claude-3-5-sonnet (Target: ~32k to 64k tokens): Extended call-chain depths, full component prop interfaces, and complete ADR documentation.
gemini-1.5-pro (Target: ~100k+ tokens): Deep repository breadth, full implementation bodies across related clusters, and complete test suites.
llama-3-8b / Local Models (Target: ~4k to 8k tokens): Aggressive even-span folding, signatures only, zero diffs, highest Louvain cluster focus.

Selecting Profiles via CLI or Bridge

# Generate context tailored specifically for Claude 3.5 Sonnet
tokencap make --profile claude-3-5-sonnet

CLI Output:

Loaded model profile: claude-3-5-sonnet
Token budget ceiling: 64,000 tokens
Allocated:
  - Architecture and graph overview: 2,400 tokens
  - Core targeted symbols: 18,200 tokens
  - Transitive caller bodies: 14,800 tokens
  - Active ADRs and memory: 1,600 tokens
Total Generated: 37,000 tokens (58% of ceiling, 0% overflow risk)

The bridge passes these model hints automatically when queries arrive from our Chrome companion or MCP server.

Read the model profiling guide at tokencap.vansharora.app

How We Refactored a 2,400-Line CLI Monolith into 7 Core Verbs Without Breaking Changes

VANSH ARORA — Wed, 21 Oct 2026 18:11:36 +0000

As TokenCap expanded from a simple repository snapshot generator to a full intelligence engine with graphs, memory loops, and security scanners, our CLI sprawled out of control. We had 21 distinct commands in a single 2,455-line bin/tokencap.js file.

To fix this, we completed a major CLI refactoring around 7 core verbs.

The 7 Core Verbs Hierarchy

make (src/cli/make.js): Snapshot generation, watch mode, custom budgeting profiles.
ask (src/cli/ask.js): Spatial queries (ask impact, ask dead-code, ask adr).
analyze (src/cli/analyze.js): Repo audits (analyze health, analyze debt, analyze security).
serve (src/cli/serve.js): Background daemons (serve --daemon, serve --mcp).
update (src/cli/update.js): In-place upgrades.
help & version: System introspection.

The Silent Alias Strategy

To guarantee that existing scripts and CI workflows did not break:

// bin/tokencap.js excerpt
const ALIASES = {
  graph: ["ask", "impact"],
  capsule: ["make"],
  security: ["analyze", "security"],
  health: ["analyze", "health"],
  mcp: ["serve", "--mcp"]
};

const normalizedArgs = resolveAliases(process.argv.slice(2));

The dispatcher shrank from 2,455 lines down to 234 lines of clean routing logic, with handlers decoupled into modular, testable files in src/cli/.

Inspect the CLI architecture at tokencap.vansharora.app

How TokenCap Eliminates Context Query Latency with Background Pre-Warming

VANSH ARORA — Wed, 21 Oct 2026 12:09:36 +0000

When integrating browser companions or IDE extensions with a local intelligence bridge, latency matters. If generating prompt context takes 4 or 5 seconds, developers stop using the integration.

To deliver sub-second responses, TokenCap implements background cache pre-warming in src/bridge/warmCache.js.

How Pre-Warming Operates

When you run tokencap serve, the HTTP server begins listening immediately. While the pairing handshake occurs, an asynchronous worker kicks off:

Graph Pre-Compilation: Traverses all files and loads AST symbol tables into memory.
Memory and ADR Indexing: Loads active session records and architecture decisions into RAM.
Git Diff Snapshotting: Caches uncommitted working tree diffs and affected symbols.

// src/bridge/warmCache.js logic
function preWarmBridgeContext(repoRoot) {
  setImmediate(() => {
    // Builds query-independent artifacts concurrently
    const graph = getOrBuildGraph(repoRoot);
    const memory = loadActiveMemory(repoRoot);
    const diff = computeWorkingDiff(repoRoot);
    bridgeCache.set(repoRoot, { graph, memory, diff, timestamp: Date.now() });
  });
}

Response Time Comparison

Cold Generation (No Pre-Warm): 1,840ms
With Pre-Warmed Cache: 180ms (10x faster first-click response)

When the Chrome Companion requests task-scoped context via POST /generate, TokenCap slices the pre-warmed graph and formats the prompt payload almost instantaneously.

Documentation available at tokencap.vansharora.app

How We Built an Atomic, Sharded Incremental Cache with 57ms Rebuild Times

VANSH ARORA — Tue, 20 Oct 2026 18:04:41 +0000

When building a local-first repository intelligence engine, speed is everything. A cold build on a 100k-file benchmark can take 45 to 55 seconds. But during active coding, developers expect instantaneous updates when saving a file.

In TokenCap, we implemented an atomic sharded incremental cache in src/incremental/cacheManager.js.

Cache Architecture and Integrity Guarantees

Atomic Sharding: Instead of a monolithic cache file, intelligence graphs are sharded into modular JSON blocks keyed by file path hashes.
Content-Based Checksums: Cache validity is verified via SHA-256 checksums rather than volatile filesystem mtimes.
Version Guard: Any version mismatch between the running TokenCap binary and stored cache forces an automatic, safe cold rebuild instead of mis-reading stale schemas.
Scoped Invalidation: When src/auth.js changes, TokenCap only re-parses auth.js and updates direct incoming and outgoing graph edges.

Benchmark Numbers

On our benchmark repository of 1,200 files:

Cold Build (Zero Cache): 1.14s
Warm Full Rebuild: 290ms
Scoped Incremental Build (1 File Edit): 57ms

# Watch mode with sub-60ms incremental rebuilds
tokencap make --watch

Output:

Watching for changes (rebuild after 300ms idle)...
✓ Change detected in src/auth/jwt.js
✓ Invalidation scope: 1 file, 4 affected callers
✓ Rebuilt incremental snapshot in 57ms (.tokencap/snapshot.md)

Because rebuilds take under 60ms, the VS Code extension auto-captures on save without interrupting typing cadence.

Explore the caching implementation at tokencap.vansharora.app