DEV Community: VANSH ARORA

Announcing TokenCap: The Local-First Context Layer for AI Coding Agents

VANSH ARORA — Thu, 22 Oct 2026 18:07:15 +0000

Today, we are officially launching TokenCap: a zero-network-egress repository intelligence platform designed for developers building and pairing with AI coding agents.

What is TokenCap?

TokenCap bridges the gap between raw source code and LLM context windows. Instead of blindly dumping full files or relying on fuzzy text search, TokenCap parses your codebase into an AST symbol graph, clusters modules, and budgets context deterministically.

Core Verified Benchmarks and Capabilities

12.2x Token Reduction: Compresses typical 150k-token repository payloads down to 12.5k tokens without dropping critical call signatures.
Even-Span Sampling: Proportional sampling across file heads, bodies, and exports that eliminates line-300 cutoffs.
Spatial Impact Analysis: tokencap ask impact <file>:<symbol> calculates caller blast radius and risk before an agent modifies code.
The Four Sovereignty Laws: Zero runtime network egress, zero API keys, no compilation build steps, and under 4 dependencies.
Multi-Surface Availability:
- CLI: npx tokencap make
- VS Code Extension: Auto-capture and spatial menus
- Local Bridge & Companion: Direct loopback injection into ChatGPT, Claude, and Gemini
- Agent Skill: Native discovery for OpenCode, Claude Code, Cursor, and Antigravity

Quick Start

Install globally or run directly with npx:

# Generate your repository intelligence snapshot
npx tokencap make

# Start the local loopback bridge for browser companions
npx tokencap serve --daemon

# Query the blast radius of a function
npx tokencap ask impact src/auth.js:verifyToken

Explore the documentation, read the architecture specs, and install the extensions at tokencap.vansharora.app

How We Packaged TokenCap as a Self-Discoverable Agent Skill

VANSH ARORA — Thu, 22 Oct 2026 12:13:04 +0000

When onboarding a new software engineer, you provide onboarding documentation. When onboarding an AI agent, developers often rely on ad-hoc prompts or complex custom scripts that break across different platforms.

With the TokenCap Agent Skill system in skills/tokencap/, any agent supporting open skill standards can discover, query, and respect repository intelligence without prior manual setup.

The Skill Specification

The skill is built around three standard files:

SKILL.md: Root declaration with YAML frontmatter, trigger conditions, and high-level workflows.
reference/commands.md: Exact CLI and MCP commands for snapshot generation, impact analysis, and refactoring.
reference/workflows.md: Step-by-step playbooks for PR review, onboarding to unfamiliar codebases, and blast-radius audits.


yaml

Why AI Context Packs Must Adapt Dynamically to Target Model Economics

VANSH ARORA — Thu, 22 Oct 2026 08:46:40 +0000

Different LLMs have wildly different context windows and cost economics. Providing a generic 20k-token context pack is wasteful for smaller models and insufficient for massive 200k+ models.

In TokenCap, we implemented model-aware budgeting in src/bridge/generate.js.

How Token Allocations Adjust

gpt-4o (Target: ~16k to 32k tokens): Balanced span sampling, key symbol definitions, priority diffs, and compressed helper functions.
claude-3-5-sonnet (Target: ~32k to 64k tokens): Extended call-chain depths, full component prop interfaces, and complete ADR documentation.
gemini-1.5-pro (Target: ~100k+ tokens): Deep repository breadth, full implementation bodies across related clusters, and complete test suites.
llama-3-8b / Local Models (Target: ~4k to 8k tokens): Aggressive even-span folding, signatures only, zero diffs, highest Louvain cluster focus.

Selecting Profiles via CLI or Bridge

# Generate context tailored specifically for Claude 3.5 Sonnet
tokencap make --profile claude-3-5-sonnet

CLI Output:

Loaded model profile: claude-3-5-sonnet
Token budget ceiling: 64,000 tokens
Allocated:
  - Architecture and graph overview: 2,400 tokens
  - Core targeted symbols: 18,200 tokens
  - Transitive caller bodies: 14,800 tokens
  - Active ADRs and memory: 1,600 tokens
Total Generated: 37,000 tokens (58% of ceiling, 0% overflow risk)

The bridge passes these model hints automatically when queries arrive from our Chrome companion or MCP server.

Read the model profiling guide at tokencap.vansharora.app

How We Refactored a 2,400-Line CLI Monolith into 7 Core Verbs Without Breaking Changes

VANSH ARORA — Wed, 21 Oct 2026 18:11:36 +0000

As TokenCap expanded from a simple repository snapshot generator to a full intelligence engine with graphs, memory loops, and security scanners, our CLI sprawled out of control. We had 21 distinct commands in a single 2,455-line bin/tokencap.js file.

To fix this, we completed a major CLI refactoring around 7 core verbs.

The 7 Core Verbs Hierarchy

make (src/cli/make.js): Snapshot generation, watch mode, custom budgeting profiles.
ask (src/cli/ask.js): Spatial queries (ask impact, ask dead-code, ask adr).
analyze (src/cli/analyze.js): Repo audits (analyze health, analyze debt, analyze security).
serve (src/cli/serve.js): Background daemons (serve --daemon, serve --mcp).
update (src/cli/update.js): In-place upgrades.
help & version: System introspection.

The Silent Alias Strategy

To guarantee that existing scripts and CI workflows did not break:

// bin/tokencap.js excerpt
const ALIASES = {
  graph: ["ask", "impact"],
  capsule: ["make"],
  security: ["analyze", "security"],
  health: ["analyze", "health"],
  mcp: ["serve", "--mcp"]
};

const normalizedArgs = resolveAliases(process.argv.slice(2));

The dispatcher shrank from 2,455 lines down to 234 lines of clean routing logic, with handlers decoupled into modular, testable files in src/cli/.

Inspect the CLI architecture at tokencap.vansharora.app

How TokenCap Eliminates Context Query Latency with Background Pre-Warming

VANSH ARORA — Wed, 21 Oct 2026 12:09:36 +0000

When integrating browser companions or IDE extensions with a local intelligence bridge, latency matters. If generating prompt context takes 4 or 5 seconds, developers stop using the integration.

To deliver sub-second responses, TokenCap implements background cache pre-warming in src/bridge/warmCache.js.

How Pre-Warming Operates

When you run tokencap serve, the HTTP server begins listening immediately. While the pairing handshake occurs, an asynchronous worker kicks off:

Graph Pre-Compilation: Traverses all files and loads AST symbol tables into memory.
Memory and ADR Indexing: Loads active session records and architecture decisions into RAM.
Git Diff Snapshotting: Caches uncommitted working tree diffs and affected symbols.

// src/bridge/warmCache.js logic
function preWarmBridgeContext(repoRoot) {
  setImmediate(() => {
    // Builds query-independent artifacts concurrently
    const graph = getOrBuildGraph(repoRoot);
    const memory = loadActiveMemory(repoRoot);
    const diff = computeWorkingDiff(repoRoot);
    bridgeCache.set(repoRoot, { graph, memory, diff, timestamp: Date.now() });
  });
}

Response Time Comparison

Cold Generation (No Pre-Warm): 1,840ms
With Pre-Warmed Cache: 180ms (10x faster first-click response)

When the Chrome Companion requests task-scoped context via POST /generate, TokenCap slices the pre-warmed graph and formats the prompt payload almost instantaneously.

Documentation available at tokencap.vansharora.app

How We Built an Atomic, Sharded Incremental Cache with 57ms Rebuild Times

VANSH ARORA — Tue, 20 Oct 2026 18:04:41 +0000

When building a local-first repository intelligence engine, speed is everything. A cold build on a 100k-file benchmark can take 45 to 55 seconds. But during active coding, developers expect instantaneous updates when saving a file.

In TokenCap, we implemented an atomic sharded incremental cache in src/incremental/cacheManager.js.

Cache Architecture and Integrity Guarantees

Atomic Sharding: Instead of a monolithic cache file, intelligence graphs are sharded into modular JSON blocks keyed by file path hashes.
Content-Based Checksums: Cache validity is verified via SHA-256 checksums rather than volatile filesystem mtimes.
Version Guard: Any version mismatch between the running TokenCap binary and stored cache forces an automatic, safe cold rebuild instead of mis-reading stale schemas.
Scoped Invalidation: When src/auth.js changes, TokenCap only re-parses auth.js and updates direct incoming and outgoing graph edges.

Benchmark Numbers

On our benchmark repository of 1,200 files:

Cold Build (Zero Cache): 1.14s
Warm Full Rebuild: 290ms
Scoped Incremental Build (1 File Edit): 57ms

# Watch mode with sub-60ms incremental rebuilds
tokencap make --watch

Output:

Watching for changes (rebuild after 300ms idle)...
✓ Change detected in src/auth/jwt.js
✓ Invalidation scope: 1 file, 4 affected callers
✓ Rebuilt incremental snapshot in 57ms (.tokencap/snapshot.md)

Because rebuilds take under 60ms, the VS Code extension auto-captures on save without interrupting typing cadence.

Explore the caching implementation at tokencap.vansharora.app

Moving Beyond Raw Git Diffs: Evidence-Backed Symbol Change Reports

VANSH ARORA — Tue, 20 Oct 2026 08:44:14 +0000

Git diffs are the universal standard for code review, but they operate at the line level rather than the semantic level. A one-line signature change in an exported function can have a higher blast radius than a 200-line CSS refactor, yet git treats them both as raw text changes.

In TokenCap, we built src/diff/buildDiffReport.js to produce evidence-backed semantic diff reports.

What the Semantic Diff Report Computes

Symbol Changes: Identifies exactly which functions, methods, and classes were added, modified, or deleted.
Breaking Change Detection: Flags parameter list alterations and removed exports that break existing call sites.
Test Association: Scans the call graph to list the specific test files covering the modified symbols.
Risk Rating: Computes risk (LOW, MEDIUM, HIGH, CRITICAL) based on transitive caller volume and module boundary crossings.

CLI Inspection

tokencap diff

Output:

Comparing working tree against origin/main...
Files Changed: 4 files (+42, -18 lines)
Symbol Alterations:
  - MODIFIED: src/auth/jwt.js:verifyToken (Signature changed: +options param)
Breaking Change Risk: HIGH
Downstream Impact:
  - 4 direct call sites affected
  - 18 transitive callers across 3 clusters
Recommended Tests to Run:
  - test/auth.test.js
  - test/session.test.js
Report generated in .tokencap/diff/review-packet.md

You can inject this report directly into your PR description or pass it to an AI agent to verify that all affected call sites were updated.

Learn more at tokencap.vansharora.app

How TokenCap Computes a 0-to-100 Build Health Diagnostic for Codebases

VANSH ARORA — Thu, 15 Oct 2026 17:51:14 +0000

Evaluating the health of a repository usually requires running multiple disjointed tools: linters, circular dependency checkers, dead code detectors, and test runners.

In TokenCap, we unified these signals into an objective 0-to-100 repository health diagnostic in src/health/buildHealth.js.

What Factors Drive the Health Score?

Call Resolution Ratio (30% weight): Percentage of intra-repo call expressions that successfully resolve to declared symbols.
Dead Code Ratio (20% weight): Ratio of uncalled internal functions relative to total declared symbols.
Architecture Boundaries (20% weight): Frequency of circular dependencies and unexpected cross-cluster imports.
Test Mapping Coverage (15% weight): Ratio of exported functions with direct unit test mappings.
Security Cleanliness (15% weight): Absence of unescaped static taint sinks and credential leaks.

CLI Health Diagnostic

tokencap analyze health

Output:

Evaluating repository health across 94 files...
------------------------------------------------------
OVERALL BUILD HEALTH SCORE: 88 / 100 [GRADE: A-]
------------------------------------------------------
  ✓ Call Resolution: 91% (891 / 978 call sites resolved)
  ✓ Circular Dependencies: 0 detected
  ! Dead Code: 6 uncalled functions (-4 pts)
  ! Untested Exports: 14 exported functions missing tests (-6 pts)
  ✓ Security Sinks: 0 critical taint paths
------------------------------------------------------
Report saved to .tokencap/health-report.json

By running health checks before and after agent sessions, teams can verify that automated code changes improve the overall score rather than degrading it.

Learn more at tokencap.vansharora.app

How TokenCap Turns Scattered TODO Comments into an Actionable Debt Ledger

VANSH ARORA — Thu, 15 Oct 2026 11:52:09 +0000

Almost every production codebase is littered with comments like // TODO: handle error retry or // FIXME: memory leak under load. Over time, they become invisible noise.

In TokenCap, we built src/debt/ledger.js to catalog and prioritize technical debt automatically.

How the Debt Ledger Operates

Tokenizer Scan: Scans source lines for standard debt prefixes (TODO, FIXME, HACK, DEBT, OPTIMIZE).
Git Metadata Anchoring: Runs local git blame to capture the original author, commit hash, and creation date.
Symbol and Cluster Mapping: Associates the debt marker with its enclosing function and architectural cluster.
Output Generation: Writes an auditable ledger to .tokencap/debt/ledger.json and .tokencap/debt-map.md.

CLI Command

tokencap analyze debt

CLI Output:

Indexed 38 technical debt markers across 84 files:
  - [CRITICAL] src/auth/session.js:84: FIXME: Refresh token race condition
    Authored: 2026-03-12 by vansharora21 | Cluster: Auth
  - [MEDIUM] src/pack/fold.js:120: TODO: Support nested bracket folding
    Authored: 2026-06-20 by vansharora21 | Cluster: Pack
  - [LOW] src/utils/format.js:14: HACK: Temporary regex for legacy dates
    Authored: 2026-08-01 by vansharora21 | Cluster: Utilities

Because debt items are scored and tracked, your AI coding agent can be instructed to resolve high-priority debt items alongside planned feature work.

Preventing AI Architecture Reversals with In-Repo Architecture Decision Records

VANSH ARORA — Thu, 15 Oct 2026 08:27:56 +0000

A common frustration when working with AI coding tools is architectural amnesia. You carefully decide to avoid an ORM for specific performance reasons. Two days later, a new AI session opens and immediately suggests installing Prisma.

In TokenCap, we solve this by embedding Architecture Decision Records (ADRs) directly into the agent intelligence layer in src/memory/adrs.js.

How ADR Intelligence Works

Structured Storage: ADRs are stored as markdown files in .tokencap/memory/adrs/ with title, date, status, context, decision, and consequences.
Symbol Association: Decisions can be linked to specific modules or symbol clusters.
Prompt Context Injection: When an agent edits files associated with an ADR, TokenCap injects the rationale into the top of the context pack:

### Active Architecture Decision Record: ADR-004
Title: Use Raw SQL for Ledger Entries (Do Not Use ORM)
Status: Accepted (2026-07-15)
Decision: Ledger balance operations must use raw SQL parameterized queries with strict SERIALIZABLE transaction isolation to prevent race conditions.
Consequence: Do not refactor ledger operations to ORM abstractions.

CLI and MCP Tooling

You can create and manage ADRs via the CLI or MCP tool:

tokencap ask adr --list

CLI Output:

Registered Architecture Decision Records (4):
  - ADR-001: CommonJS only for core modules (No build step)
  - ADR-002: Vendored WASM grammars for tree-sitter
  - ADR-003: Staged memory pipeline before repository commit
  - ADR-004: Raw SQL for ledger operations

With ADRs in place, AI agents respect established team choices instead of endlessly proposing architectural rewrites.

How TokenCap Traces Taint Flow from Sources to Sinks Using AST Traversal

VANSH ARORA — Wed, 14 Oct 2026 18:08:05 +0000

Generative AI is great at writing features, but notorious for re-introducing classic security vulnerabilities like SQL injection, Command Injection, and Path Traversal.

In TokenCap, we implemented intra-procedural static taint flow analysis in src/security/taint/ to flag tainted data paths before code changes are committed.

Taint Analysis Mechanics

Sources: HTTP request properties (req.body, req.query, req.params, event.body).
Propagation: Data flow tracking across variable assignments, template strings, and object destructuring within the function scope.
Sinks: Unescaped database execution calls (db.raw, query()), shell executions (exec, spawn), and filesystem operations (fs.readFile).
Sanitizers: Validating functions that cleanse taint (such as parameterized queries or validator schemas).

Inspection Example

tokencap analyze security

CLI Output:

Inspecting 62 route handlers for static taint flow...
[WARNING] Potential SQL Injection Sink:
  - Source: req.query.filter (src/api/users.js:34)
  - Taint Path: filter -> rawFilterClause -> queryStr
  - Sink: db.raw(queryStr) (src/api/users.js:48)
  - Recommendation: Use parameterized binding (:filter) or validate through zod schema.

This security finding is embedded into .tokencap/risk-map.md, alerting any coding agent working on src/api/users.js to avoid raw string concatenation.

Read our security architecture at tokencap.vansharora.app

Linking SQL DDL and GraphQL Schemas Directly to Code Entities in TokenCap

VANSH ARORA — Wed, 14 Oct 2026 12:13:02 +0000

AI coding assistants frequently write SQL or ORM queries that look plausible but fail at runtime because a column was renamed from created_at to createdAt or a field is non-nullable.

To prevent schema-to-code drift, TokenCap implements schema linking in src/schema/schemaGraph.js.

How Schema Intelligence Works

DDL Parsing: Scans .sql migrations and Prisma/Drizzle schema files for CREATE TABLE statements, extracting column names, types, primary keys, and foreign keys.
GraphQL Schema Parsing: Extracts types, queries, and mutations from .graphql files.
Code Entity Linking: Maps extracted schema entities to corresponding backend model classes, Prisma calls, or repository queries.

Generated Graph Intelligence

When TokenCap compiles .tokencap/snapshot.md:

### Schema Entity: users (Table)
- Defined in: migrations/20260401_init_users.sql
- Columns:
  - id: UUID (Primary Key, NOT NULL)
  - email: VARCHAR(255) (UNIQUE, NOT NULL)
  - tier: VARCHAR(50) (DEFAULT 'free')
  - metadata: JSONB (NULLABLE)
- Referenced By Code:
  - src/models/User.js: User schema definition
  - src/services/auth.js: findByEmail
  - src/services/billing.js: updateSubscriptionTier

When an agent needs to add a column or modify a query, it knows the exact database constraints without asking you to upload database dumps.

Learn more at tokencap.vansharora.app