Why We Replaced Native Tree-Sitter Bindings with Vendored WebAssembly Grammars

Building developer tools with tree-sitter provides incredible parsing power. However, relying on native tree-sitter bindings introduces significant installation friction:

• Windows machines fail if Microsoft C++ Build Tools are missing.
• Linux Alpine containers fail if musl-compatible gcc toolchains are not configured.
• Node.js version bumps break native ABI compatibility.

To eliminate installation failure entirely, TokenCap uses web-tree-sitter and vendors all language grammars as WASM binaries.

Supported Languages (Vendored In-Repo)

Inside dist/grammars/, TokenCap vendors compiled WASM grammars:

• JavaScript (tree-sitter-javascript.wasm)
• TypeScript & TSX (tree-sitter-typescript.wasm, tree-sitter-tsx.wasm)
• Python (tree-sitter-python.wasm)
• Go (tree-sitter-go.wasm)
• Rust (tree-sitter-rust.wasm)
• Java (tree-sitter-java.wasm)

The Sovereignty Advantage

Because every grammar file is bundled into the npm package and VSIX archive:

1. Zero Network Downloads: No fetching grammars from GitHub releases during installation.
2. Zero Compilation: Installs instantly via npx tokencap without invoking node-gyp.
3. Total Package Size: The entire package with all 7 grammars, query files, and CLI engines is under 6.5 MB packed.

Test it yourself on any machine:

npx tokencap make

Read the architecture documentation at tokencap.vansharora.app