Like Fuse.js, but it matches on meaning: a 7 MB embedding model that runs in the browser

Wen Shu Tang — Fri, 31 Jul 2026 16:24:29 +0000

Semantic search — matching on meaning instead of exact text — almost always means standing up infrastructure: an embedding API (plus a key, plus a per-call bill), a Python model server, or a vector database. For a lot of ordinary features, that's a surprising amount of backend for "make the search box understand synonyms."

So I tried to delete all of it.

ternlight is a sentence-embedding model where the model, tokenizer, and inference engine are ~7 MB total and run entirely in the browser, on the CPU. No server, no API, no GPU. You npm install it and it runs where your users already are.

The way I explain it simply: an upgraded Fuse.js. Something you drop into the page - no service to stand up, no key to manage — that happens to match on meaning instead of characters. Pagefind made full-text search on a static site a one-line install; this is the same idea, but with semantic search unlocked.

👉 Live demo - runs entirely in your browser: https://ternlight.dev/demo/

The whole integration

npm install @ternlight/base

import { embed, similar } from '@ternlight/base';

// text -> 384-dim vector, computed on the CPU, in this tab
const v = embed('how do I cancel my subscription?');

// or rank a small corpus by meaning
similar('I want my money back', docs, { topK: 3 });
// -> ranked matches · ~5 ms/embed · zero network

No fetch, no key, works offline. That's it.

What "on-device" actually costs

The interesting part isn't that it's small, it's that "on-device" makes four things bind at once that normally trade off against each other:

It has to fit: engine + tokenizer + weights arrive as one artifact the browser downloads once.
It has to stay smart: an embedding is only worth anything if it captures meaning (the one you can't fake).
It has to run anywhere: no GPU to assume, nothing to install; a runtime every browser already has.
It has to be fast on a CPU: a search box feels latency keystroke by keystroke.

Each is reasonable alone; holding all four is the hard part, and it's why the usual tools don't carry over. A transformer behind an API is great when you have a server to call. A general runtime like ONNX Runtime Web or PyTorch is built to run almost anything — which is exactly what makes it heavy to ship into a tab.

Roughly how the options compare:

	ternlight	Remote embedding API	transformers.js (in-browser)
On the wire	~7 MB, once	~0, but a network call per query	tens–hundreds of MB
Latency	~5 ms/embed, local	network round-trip each call	heavier (general runtime)
Server / API key	none	required	none
Data leaves device	no	yes	no
GPU	not needed	n/a	often wanted

How it fits in 7 MB

There are three levers that unlocked this:

1. Ternary weights. In deep neural nets, every weight has a floating point 32 representation. But here, every linear-layer weight is just -1, 0, or +1 - trained that way from the start. That's two bits per weight instead of thirty-two. And here's the part that is most cool: when weights are only -1/0/+1, the matrix multiply that dominates inference stops needing multiplication - it collapses into add, subtract, skip, exactly the shape a CPU's Single Instruction multiple data (SIMD) lanes chew through vector operations fast. So ternary does double duty: it shrinks the model and makes it quick on a plain CPU.

Left: 32-bit weights, a continuous range. Right: the same layer, ternary — every weight is one of three values.

2. Packing. Three possible values pack into ~2 bits; squeezing thousands together is what turns a normal-sized network into a few-megabyte download.

3. A from-scratch engine. Instead of shipping PyTorch/ONNX and letting the runtime dwarf the weights, the whole inference path is hand-written Rust compiled to one small WASM module - where that add/subtract/skip hot loop actually lives.

The surprise: "small, fast, portable, good - pick two" feels like a law. But training ternary from the start doesn't slide you along that tradeoff curve, it moves the curve — more quality per byte than the tradeoff should allow. That's why all four can hold at once.

Measured: ~8× smaller for ~0.05 Spearman vs the pre-quantization student — the whole point of doing it in training rather than after.

When to use it (and when not)

Honest limits: It's English-only, reads up to 128 tokens at a time, and won't top a retrieval leaderboard. It's tuned to be small and useful, not to win benchmarks.

Where it shines is the ordinary, high-volume work: semantic search over your own docs, matching a question to an FAQ or an intent, finding near-duplicates, privacy-first tasks. For those, a small model that runs locally beats a higher benchmark number sitting behind an API, no network round-trip, no API keys, nothing to deploy, and the data never leaves the device.

Try it

It's MIT and fully open sourced - the model, the training + quantization pipeline, and the Rust engine are all on GitHub. One line to try it:

npm install @ternlight/base

I wrote the longer, illustrated version - with the craft-project backstory on my site: Running embeddings in the browser: what it actually costs. Deep dives on the training, the packing, and the engine are coming soon.