Semantic search — matching on meaning instead of exact text — almost always means standing up infrastructure: an embedding API (plus a key, plus a per-call bill), a Python model server, or a vector database. For a lot of ordinary features, that's a surprising amount of backend for "make the search box understand synonyms."
So I tried to delete all of it.
ternlight is a sentence-embedding model where the model, tokenizer, and inference engine are ~7 MB total and run entirely in the browser, on the CPU. No server, no API, no GPU. You npm install it and it runs where your users already are.
The way I explain it simply: an upgraded Fuse.js. Something you drop into the page - no service to stand up, no key to manage — that happens to match on meaning instead of characters. Pagefind made full-text search on a static site a one-line install; this is the same idea, but with semantic search unlocked.
👉 Live demo - runs entirely in your browser: https://ternlight.dev/demo/
The whole integration
npm install @ternlight/base
import { embed, similar } from '@ternlight/base';
// text -> 384-dim vector, computed on the CPU, in this tab
const v = embed('how do I cancel my subscription?');
// or rank a small corpus by meaning
similar('I want my money back', docs, { topK: 3 });
// -> ranked matches · ~5 ms/embed · zero network
No fetch, no key, works offline. That's it.
What "on-device" actually costs
The interesting part isn't that it's small, it's that "on-device" makes four things bind at once that normally trade off against each other:
- It has to fit: engine + tokenizer + weights arrive as one artifact the browser downloads once.
- It has to stay smart: an embedding is only worth anything if it captures meaning (the one you can't fake).
- It has to run anywhere: no GPU to assume, nothing to install; a runtime every browser already has.
- It has to be fast on a CPU: a search box feels latency keystroke by keystroke.
Each is reasonable alone; holding all four is the hard part, and it's why the usual tools don't carry over. A transformer behind an API is great when you have a server to call. A general runtime like ONNX Runtime Web or PyTorch is built to run almost anything — which is exactly what makes it heavy to ship into a tab.
Roughly how the options compare:
| ternlight | Remote embedding API | transformers.js (in-browser) | |
|---|---|---|---|
| On the wire | ~7 MB, once | ~0, but a network call per query | tens–hundreds of MB |
| Latency | ~5 ms/embed, local | network round-trip each call | heavier (general runtime) |
| Server / API key | none | required | none |
| Data leaves device | no | yes | no |
| GPU | not needed | n/a | often wanted |
How it fits in 7 MB
There are three levers that unlocked this:
1. Ternary weights. In deep neural nets, every weight has a floating point 32 representation. But here, every linear-layer weight is just -1, 0, or +1 - trained that way from the start. That's two bits per weight instead of thirty-two. And here's the part that is most cool: when weights are only -1/0/+1, the matrix multiply that dominates inference stops needing multiplication - it collapses into add, subtract, skip, exactly the shape a CPU's Single Instruction multiple data (SIMD) lanes chew through vector operations fast. So ternary does double duty: it shrinks the model and makes it quick on a plain CPU.
Left: 32-bit weights, a continuous range. Right: the same layer, ternary — every weight is one of three values.
2. Packing. Three possible values pack into ~2 bits; squeezing thousands together is what turns a normal-sized network into a few-megabyte download.
3. A from-scratch engine. Instead of shipping PyTorch/ONNX and letting the runtime dwarf the weights, the whole inference path is hand-written Rust compiled to one small WASM module - where that add/subtract/skip hot loop actually lives.
The surprise: "small, fast, portable, good - pick two" feels like a law. But training ternary from the start doesn't slide you along that tradeoff curve, it moves the curve — more quality per byte than the tradeoff should allow. That's why all four can hold at once.
Measured: ~8× smaller for ~0.05 Spearman vs the pre-quantization student — the whole point of doing it in training rather than after.
When to use it (and when not)
Honest limits: It's English-only, reads up to 128 tokens at a time, and won't top a retrieval leaderboard. It's tuned to be small and useful, not to win benchmarks.
Where it shines is the ordinary, high-volume work: semantic search over your own docs, matching a question to an FAQ or an intent, finding near-duplicates, privacy-first tasks. For those, a small model that runs locally beats a higher benchmark number sitting behind an API, no network round-trip, no API keys, nothing to deploy, and the data never leaves the device.
Try it
It's MIT and fully open sourced - the model, the training + quantization pipeline, and the Rust engine are all on GitHub. One line to try it:
npm install @ternlight/base
I wrote the longer, illustrated version - with the craft-project backstory on my site: Running embeddings in the browser: what it actually costs. Deep dives on the training, the packing, and the engine are coming soon.
Happy to answer anything in the comments 👇
Top comments (0)