DEV Community: Ólafur Aron Jóhannsson

Semantic Search in C++ without Python, libtorch or ONNX Runtime

Ólafur Aron Jóhannsson — Sat, 05 Sep 2026 22:23:35 +0000

Ask how to run a transformer model from C++ and you get two answers: link libtorch, or convert the model and link ONNX Runtime. Both work. Both are large, both want a toolchain of their own, and both put a second inference engine inside your process.

There is a third answer, and it takes four commands.

mkdir kjarni-quickstart && cd kjarni-quickstart
curl -sL https://github.com/olafurjohannsson/kjarni/releases/latest/download/kjarni-x86_64-linux.tar.gz | tar xz
curl -sO https://raw.githubusercontent.com/olafurjohannsson/kjarni/main/crates/kjarni-ffi/examples/cpp/hello.cpp
g++ -std=c++23 hello.cpp -I. -L. -lkjarni_ffi -Wl,-rpath,'$ORIGIN' -o hello && ./hello

related:   0.5510
unrelated: -0.0630

That is a transformer model, downloaded, loaded and run, from an empty directory. No package manager, no Python, no model conversion step. The archive holds the shared library, kjarni.h (the C ABI) and kjarni.hpp (a header-only C++23 wrapper). macOS and Windows
builds are on the same [releases page(https://github.com/olafurjohannsson/kjarni/releases).

Here is what hello.cpp contains:

#include "kjarni.hpp"
#include <print>

int main() {
    // Downloaded once and cached under ~/.cache/kjarni, then loaded from disk.
    auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
    if (!embedder) {
        std::println("{}", embedder.error().message());
        return 1;
    }

    auto question = embedder->encode("How do I get my money back?");
    auto related = embedder->encode("What is your refund policy?");
    auto unrelated = embedder->encode("The weather in Reykjavik is unpredictable.");

    // No shared words with the question, but the same meaning.
    std::println("related:   {:.4f}", kjarni::cosine(*question, *related));
    std::println("unrelated: {:.4f}", kjarni::cosine(*question, *unrelated));
}

"How do I get my money back?" and "What is your refund policy?" share no words at all, and score 0.55. The sentence about the weather scores below zero. That gap is the entire idea behind semantic search.

How semantic search works

An embedding model reads text and returns a vector, an array of floats, 384 numbers for the model above. Text with similar meaning produces vectors that point in similar directions.

"refund policy"  -> [0.12, -0.34, 0.56, ...]  (384 numbers)
"get money back" -> [0.11, -0.33, 0.55, ...]  (384 numbers)  <- close
"weather today"  -> [-0.45, 0.23, -0.12, ...] (384 numbers)  <- far

You compare two vectors with cosine similarity, which measures the angle between them and ignores their length. It runs from 1 for identical direction to -1 for opposite.

What you actually link against

This is the part that decides whether the approach is worth anything, so it is worth checking rather than believing. Ask the linker:

$ ldd hello
    linux-vdso.so.1
    libkjarni_ffi.so => /home/you/kjarni-quickstart/libkjarni_ffi.so
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    /lib64/ld-linux-x86-64.so.2
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2

Kjarni, the C++ runtime, and the parts of glibc every program already uses: libc, libm, libgcc, libpthread and libdl. That is the whole list. No libtorch, no onnxruntime, no Python, no CUDA runtime. The binary is 283 KB and the library is 19.4 MB, which includes the tokenizer, the model loaders and every kernel.

The -Wl,-rpath,'$ORIGIN' in the build line is what makes that first entry resolve to the library sitting next to your binary rather than something in /usr/local/lib. Keep it, and
the directory you built in is a directory you can copy somewhere else and run.

A dependency you cannot see in ldd is a dependency that cannot break you on a machine that is not yours.

Errors are values

Every fallible call returns std::expected<T, kjarni::Error>. Nothing in the header throws except std::bad_alloc.

auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
if (!embedder) {
    std::println(stderr, "could not load model: {}", embedder.error().message());
    return 1;
}

Whether a missing model file is exceptional depends on the program. A batch job should die; a desktop application should show a message and carry on. Returning the failure lets the caller decide, and puts it in the signature where it cannot be missed.

The options are a designated-initialiser aggregate, so a call names only what it changes:

Embedder::create({.model = "mpnet-base-v2", .gpu = true}).

Searching a corpus

Encode the documents once, encode the query at search time, sort by similarity.

#include "kjarni.hpp"

#include <algorithm>
#include <print>
#include <ranges>
#include <string_view>
#include <vector>

int main() {
    auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
    if (!embedder) {
        std::println(stderr, "could not load model: {}", embedder.error().message());
        return 1;
    }

    constexpr std::array docs = {
        std::string_view{"How do I reset my password?"},
        std::string_view{"What is your refund policy?"},
        std::string_view{"Do you ship internationally?"},
        std::string_view{"How do I update my billing address?"},
        std::string_view{"Where can I track my order?"},
    };

    std::vector<std::vector<float>> corpus;
    corpus.reserve(docs.size());
    for (std::string_view d : docs) {
        auto v = embedder->encode(d);
        if (!v) {
            std::println(stderr, "encode failed: {}", v.error().message());
            return 1;
        }
        corpus.push_back(v->to_vector());
    }

    constexpr std::string_view query = "I need to change my login credentials";
    auto q = embedder->encode(query);
    if (!q) {
        std::println(stderr, "encode failed: {}", q.error().message());
        return 1;
    }

    std::vector<std::pair<float, std::string_view>> scored;
    for (auto [i, doc] : std::views::enumerate(docs))
        scored.emplace_back(kjarni::cosine(q->values(), corpus[i]), doc);

    std::ranges::sort(scored, std::ranges::greater{},
                      &std::pair<float, std::string_view>::first);

    std::println("query: \"{}\"", query);
    for (auto [score, doc] : scored)
        std::println("  {:.4f}  {}", score, doc);
}

query: "I need to change my login credentials"
  0.5981  How do I reset my password?
  0.4067  How do I update my billing address?
  0.0767  Where can I track my order?
  -0.0027  What is your refund policy?
  -0.0451  Do you ship internationally?

"Change my login credentials" matches "reset my password" at 0.60 while sharing no words with it, and "update my billing address" comes second because changing account details is a
related idea. That is what you cannot get from keyword matching.

encode returns a kjarni::Embedding, which owns the array the C API returned and frees it in its destructor. It hands out a std::span<const float> through values(), so it drops
straight into ranges, and to_vector() copies when the data has to outlive the object. There is no raw pointer to forget.

Classification and reranking

The other models follow the same shape.

auto clf = kjarni::Classifier::create({.model = "roberta-sentiment"});
if (!clf) { std::println(stderr, "{}", clf.error().message()); return 1; }

for (std::string_view t : {"I love this product!",
                           "Terrible quality, broke after one day."}) {
    auto top = clf->top(t);
    if (!top) { std::println(stderr, "{}", top.error().message()); return 1; }
    if (*top) std::println("  {:<9} {:5.1f}%  \"{}\"", (*top)->name, (*top)->score * 100, t);
}

  positive   98.5%  "I love this product!"
  negative   94.1%  "Terrible quality, broke after one day."

top() returns Result<std::optional<Label>>, which puts two separate questions in the type: did the call work, and did it produce a label. A model with nothing above threshold is
not a failure.

Reranking uses a cross-encoder, which reads the query and each document together instead of comparing two independently produced vectors. It is slower and much more precise, so it runs
as a second pass over whatever the embeddings retrieved:

auto rr = kjarni::Reranker::create();
if (!rr) { std::println(stderr, "{}", rr.error().message()); return 1; }

const std::vector<std::string> docs = {
    "Machine learning is a subset of artificial intelligence.",
    "Deep learning uses neural networks with many layers.",
    "The weather today is sunny.",
};

auto ranked = rr->rerank("What is machine learning?", docs);
if (!ranked) { std::println(stderr, "{}", ranked.error().message()); return 1; }

for (const auto& r : *ranked)
    std::println("  {:>9.4f}  {}", r.score, docs[r.index]);

    10.5139  Machine learning is a subset of artificial intelligence.
    -5.5301  Deep learning uses neural networks with many layers.
   -11.1001  The weather today is sunny.

The scores are logits, not probabilities. What matters is the ordering and the size of the gap between one document and the next. Ranked gives back an index into your input rather than a copy of the text, so whatever IDs, URLs and permissions came with your documents stay attached to them.

One signature detail: rerank takes std::span<const std::string> rather than string_view, because the C call underneath needs an array of null terminated pointers and a string_view does not promise one.

The same numbers in every language

Those figures are not specific to C++. The reranker scores 10.5139, -5.5301 and -11.1001 are
the same values the C# document search post prints, and the
similarity scores are the ones in Semantic Search in C#.

That is the reason a C ABI is the right shape here. There is one engine and one set of
kernels, and the C++ header is a wrapper over the same entry points the C#, Go and Python
packages call. There is no second implementation to drift from the first.

Choosing a model

Model	Dimensions	Input limit	Notes
`minilm-l6-v2`	384	256 tokens	Default. Fast, good quality per byte
`mpnet-base-v2`	768	384 tokens	Higher quality, slower
`nomic-embed-text`	768	8192 tokens	Long documents, though trained at 2048
`bge-m3`	1024	8192 tokens	Large, multilingual

Mind the input limit column. minilm-l6-v2 reads 256 tokens, roughly 900 characters, and silently drops the rest: no error, no warning, just a vector computed from the part it saw.
The cross-encoder has its own limit, reading query and document as one sequence capped at 512 tokens. If your documents are longer than that, chunk them. There is a measurement of
what the truncation costs in Your MiniLM Embeddings Are Probably Truncating at 256 Tokens.

Practical notes

Threading. The handles are not individually thread safe. Give each thread its own, or serialise calls. The engine already parallelises across cores inside a single call, so one embedder will use the machine.

C++23 is only needed for std::expected. Everything else in kjarni.hpp is C++20, and GCC 13 or Clang 17 and newer will build it.

There is no package manager integration yet. No Conan recipe, no vcpkg port. The four commands above are the install story on Linux, and the equivalent archives for macOS and
Windows are on the releases page.

If you are on C++11 or C++17

Plenty of codebases are, and a header that demands C++23 is not much use to them. It is worth being clear about what the requirement actually covers: kjarni.hpp needs C++23 for
std::expected, and nothing else does. kjarni.h is plain C, it is the interface every other language binding is built on, and it works from C++11 upward.

The whole of the C++23 convenience is one RAII wrapper and a copy:

#include "kjarni.h"

#include <cstdio>
#include <memory>
#include <string>
#include <vector>

namespace {

struct EmbedderDeleter {
    void operator()(KjarniEmbedder* p) const { kjarni_embedder_free(p); }
};
using EmbedderPtr = std::unique_ptr<KjarniEmbedder, EmbedderDeleter>;

std::vector<float> encode(KjarniEmbedder* emb, const char* text) {
    KjarniFloatArray arr{};
    if (kjarni_embedder_encode(emb, text, &arr) != KJARNI_ERROR_CODE_OK) {
        std::fprintf(stderr, "encode failed: %s\n", kjarni_last_error_message());
        return {};
    }
    std::vector<float> out(arr.data, arr.data + arr.len);
    kjarni_float_array_free(arr);   // copied out, so release the engine's buffer
    return out;
}

} // namespace

int main() {
    KjarniEmbedderConfig cfg = kjarni_embedder_config_default();
    cfg.model_name = "minilm-l6-v2";
    cfg.quiet = 1;

    KjarniEmbedder* raw = nullptr;
    if (kjarni_embedder_new(&cfg, &raw) != KJARNI_ERROR_CODE_OK) {
        std::fprintf(stderr, "could not load model: %s\n", kjarni_last_error_message());
        return 1;
    }
    EmbedderPtr embedder(raw);

    const std::vector<float> question  = encode(embedder.get(), "How do I get my money back?");
    const std::vector<float> related   = encode(embedder.get(), "What is your refund policy?");
    const std::vector<float> unrelated = encode(embedder.get(), "The weather in Reykjavik is unpredictable.");
    if (question.empty() || related.empty() || unrelated.empty()) return 1;

    std::printf("related:   %.4f\n",
                kjarni_cosine_similarity(question.data(), related.data(), question.size()));
    std::printf("unrelated: %.4f\n",
                kjarni_cosine_similarity(question.data(), unrelated.data(), question.size()));
}

g++ -std=c++17 hello17.cpp -I. -L. -lkjarni_ffi -Wl,-rpath,'$ORIGIN' -o hello17

related:   0.5510
unrelated: -0.0630

The same numbers as the C++23 version, because it is the same engine underneath. That compiles unchanged under -std=c++11 as well.

Two rules cover the manual memory. Any KjarniFloatArray you receive is freed with kjarni_float_array_free once you have copied what you need out of it, and any handle is freed with its matching _free function. Wrapping the handle in a unique_ptr with a
custom deleter, as above, means the second rule takes care of itself on every return path.

Error text comes from kjarni_last_error_message(), and it reports the most recent failure process wide, so read it immediately after the call that failed rather than saving it up.

Compared to the alternatives

	libtorch	ONNX Runtime	Kjarni
Install	download SDK, match ABI	package plus model conversion	one archive
Model format	TorchScript	.onnx, converted	HuggingFace safetensors and GGUF directly
Extra entries in `ldd`	many	its own stack	none beyond libc
Tokenizer	bring your own	bring your own	included
GPU	CUDA toolkit	CUDA or DirectML	WebGPU, no toolkit

The trade is scope. libtorch runs anything expressible in TorchScript. Kjarni runs the model families it implements: BERT-style encoders, cross-encoders, Llama-family decoders, T5, BART and Whisper. For an arbitrary research model, convert it and use ONNX Runtime. For embeddings, classification, reranking or chat inside a C++ program that has to ship somewhere, one archive with no toolchain is a smaller problem than either.

Getting it

kjarni.ai - documentation and the rest of these posts
Releases - the archive used above, for Linux, macOS and Windows
GitHub - source, including kjarni.hpp and the C++ examples
NuGet - the same engine from C#
npm - the same engine in the browser, via WebAssembly
Go module - the same engine from Go
crates.io - the Rust engine itself

Other Resources

Semantic Search in C# - The same engine and the same vectors, from .NET
Build a Document Search Engine in C# - Keyword and semantic retrieval combined, with reranking
Why I Built a Native ML Inference Engine in Rust - What is underneath all of this
ML from the Command Line - The same models as a UNIX tool
Your MiniLM Embeddings Are Probably Truncating at 256 Tokens - Measured, with the cost

Build a Document Search Engine in C# Without Python

Ólafur Aron Jóhannsson — Sun, 15 Feb 2026 11:05:42 +0000

Build a Document Search Engine in C

Most search implementations fall into one of two camps: send everything to Elasticsearch, or call a search API. Both work. Both add infrastructure.

Here's a third option. Index local files, search them by keyword, by meaning, or both, in about 10 lines of C#. No external services.

dotnet add package Kjarni

NuGet

using Kjarni;

using var indexer = new Indexer(model: "minilm-l6-v2", quiet: true);
indexer.Create("my_index", new[] { "docs/" });

using var searcher = new Searcher(
    model: "minilm-l6-v2",
    rerankerModel: "minilm-l6-v2-cross-encoder");

var results = searcher.Search("my_index", "how do returns work?",
    mode: SearchMode.Hybrid);

foreach (var r in results)
    Console.WriteLine($"  {r.Score:F4}: {r.Text}");

The indexer reads your files, splits them into chunks, encodes each chunk as a vector, and builds a BM25 keyword index. The searcher queries both indexes and combines the results.

Setup

Create a few text files to search over:

mkdir -p docs

docs/returns.txt:

Our return policy allows customers to return any unused item within 30 days
of purchase for a full refund. Items must be in their original packaging.
Shipping costs are non-refundable.

docs/shipping.txt:

We ship to all 50 US states and internationally to over 40 countries.
Standard shipping takes 5-7 business days. Express shipping is available
for an additional fee.

docs/account.txt:

To reset your password, click "Forgot Password" on the login page.
You will receive an email with a reset link. The link expires after 24 hours.

Three short documents. In practice these could be product manuals, support articles, internal wikis, or any text files.

Indexing

using var indexer = new Indexer(model: "minilm-l6-v2", quiet: true);
indexer.Create("my_index", new[] { "docs/" });

The indexer does three things:

Reads all files in the given directories
Chunks each file into passages (for long documents)
Encodes each chunk into a 384-dimension vector using the embedding model It also builds a BM25 keyword index over the same chunks. The result is a sel, holding a config, a manifest and its segments, that you can query repeatedlywithout re-indexing. Commit it, ship it in a container image, or drop it on a share. The default chunk size is 512 characters with an overlap of 100. Both numbers matter more than they look. minilm-l6-v2 reads 256 tokens, roughly 900 characters, and silently discards anything past that, so a larger chunk size means encoding text the model neve a sentence falling across a chunk boundary still appears whole in one of them, which is why it is a fifth of the chunk size rather than a token amount. There is a measurement of what the truncation costs in Your MiniLM Embeddings Are Probably Truncating at 256 Tokens.

Three Search Modes

Keyword Search (BM25)

Matches documents that contain the query words. The same algorithm that power

var results = searcher.Search("my_index", "return policy refund",
    mode: SearchMode.Keyword);

  3.1841: Our return policy allows customers to return any unused item
          within 30 days of purchase for a full refund...

This works because the query words — "return", "policy", "refund" — appear ind for "send items back and get money" instead, keyword search would findnothing.

One thing to know about that number: BM25 scores are relative to the corpus, not absolute. The same document and the same query will score differently once you add more documents,
because the rarity of each term changes. Compare scores within one result set

For the theory behind BM25, see BM25 vs TF-IDF: Keyword Search Explained.

Semantic Search

Matches documents by meaning, regardless of the exact words used.

var results = searcher.Search("my_index", "can I send items back and get money?",
    mode: SearchMode.Semantic);

This finds the returns document even though none of those exact words appear in it. The embedding model understands that "send items back" means "return" and "get money" means "refund."

For how embeddings and similarity work, see Semantic Search in C#.

Hybrid Search

Combines keyword and semantic results. This is usually the best default.

var results = searcher.Search("my_index", "how do returns work?",
    mode: SearchMode.Hybrid);

   1.3282: Our return policy allows customers to return any unused item
           within 30 days of purchase for a full refund. Items must be in
           their original packaging. Shipping costs are non-refundable.

 -10.5874: To reset your password, click "Forgot Password" on the login
           page. You will receive an email with a reset link. The link
           expires after 24 hours.

 -11.0939: We ship to all 50 US states and internationally to over 40
           countries. Standard shipping takes 5-7 business days. Express
           shipping is available for an additional fee.

Hybrid search catches both exact keyword matches and semantically related con reranker (more on that below), which is why the gap between relevant andirrelevant results is so large. The returns document scores 1.3, while the other two are deep in the negatives.

Reranking

The results above use a cross-encoder reranker. This is the difference between good search and great search.

The Problem with Embeddings Alone

Embedding models are fast because they encode the query and each document independently. But this means they can't model the interaction between query and document directly. They're
comparing summaries, not reading both texts together.

How Reranking Fixes This

A cross-encoder takes the query and a document as a single input and outputs oth texts at the same time, so it can attend to specific words in the documentthat answer the specific question.

Bi-encoder (embedding):     Query -> Vector    Document -> Vector    Compare
Cross-encoder (reranker):   [Query + Document] -> Relevance Score

The cross-encoder is slower because it processes each query-document pair inded as a second stage: the embedding model retrieves candidates quickly, thenthe cross-encoder reranks the top results precisely.

Using the Reranker Directly

You can also use the reranker on its own:

using var reranker = new Reranker();

var results = reranker.Rerank(
    "What is machine learning?",
    new[] {
        "Machine learning is a subset of artificial intelligence.",
        "Deep learning uses neural networks with many layers.",
        "The weather today is sunny.",
    });

foreach (var r in results)
    Console.WriteLine($"  {r.Score:F4}: {r.Document}");

  10.5139: Machine learning is a subset of artificial intelligence.
  -5.5301: Deep learning uses neural networks with many layers.
 -11.1001: The weather today is sunny.

The scores are logits, not probabilities. What matters is the relative orderi. A positive score means the cross-encoder thinks the document is relevant. Anegative score means it's not.

The cross-encoder has its own input limit, and it is easy to miss: it reads the query and the document as one sequence capped at 512 tokens, roughly 1,800 characters shared between the
two. A long document gets truncated from the end, and the query eats into thereturns whole pages, rerank the passage you would actually put in front of amodel, not the full document.

The Full Pipeline

Here's how the pieces fit together:

Query
  |
  +-- BM25 Keyword Index ----> Top N candidates by word match
  |
  +-- Vector Index ----------> Top N candidates by meaning
  |
  v
  Merge candidates (union or intersection)
  |
  v
  Cross-Encoder Reranker ----> Final ranked results
  |
  v
  Return to user

Each stage filters and refines. BM25 is cheap and catches exact matches. The matches that keywords miss. The reranker reads both query and documenttogether to produce a precise ranking.

using var indexer = new Indexer(model: "minilm-l6-v2", quiet: true);
indexer.Create("my_index", new[] { "docs/" });

using var searcher = new Searcher(
    model: "minilm-l6-v2",
    rerankerModel: "minilm-l6-v2-cross-encoder");

// Hybrid = BM25 + Semantic + Reranker
var results = searcher.Search("my_index", "how do returns work?",
    mode: SearchMode.Hybrid);

When to Use Each Mode

Mode	Best for	Misses
Keyword	Exact terms, error codes, IDs	Synonyms, rephrased queries
Semantic	Intent matching, fuzzy queries	Exact phrases, rare terms
Hybrid	General purpose (recommended)	Slightly slower

Start with Hybrid. Switch to Keyword if your users search for exact identifier users describe what they want in natural language.

Practical Patterns

Filtering Results

Apply a score threshold to filter out irrelevant results:

var results = searcher.Search("my_index", query, mode: SearchMode.Hybrid);
var relevant = results.Where(r => r.Score > 0.0);

With reranking, a score above 0 is a reasonable default threshold for "probably relevant."

Search + Classification

Find relevant documents, then classify their sentiment. This combines search and classification together:

using var searcher = new Searcher(model: "minilm-l6-v2");
using var classifier = new Classifier("roberta-sentiment");

var results = searcher.Search("reviews_index", "battery life",
    mode: SearchMode.Hybrid);

foreach (var r in results.Take(10))
{
    var sentiment = classifier.Classify(r.Text);
    Console.WriteLine($"  {sentiment}  \"{r.Text}\"");
}

See Sentiment Analysis in C# Without Python for more on classification.

Re-indexing

When documents change, re-create the index:

indexer.Create("my_index", new[] { "docs/" });

This rebuilds the full index. For large corpora where incremental updates matter, you'd manage the vector storage separately.

How It Compares

Approach	Setup	Cost	Offline
Elasticsearch	Cluster + config	Server costs	No
Azure AI Search	Portal + API key	Per-query pricing	No
Algolia	Dashboard + API key	Per-search pricing	No
Kjarni	`dotnet add package`	Free	Yes

The tradeoff: Kjarni runs in-process on a single machine. If you need distribf documents, use Elasticsearch. If you need search over thousands to millionsof documents on a single server, a local engine works well and eliminates a dependency.

How It Works Under the Hood

Kjarni builds two indexes per collection:

BM25 index — inverted index over tokenized text, with term frequency saturation and document length normalization
Vector index — encoded embeddings for each chunk, queried by cosine sim

At search time, both indexes return candidates. The results are merged and opencoder model that reads the query and each candidate together.

The engine is written in Rust. The C# package wraps a single native library. o JVM, and no external service.

The same engine runs everywhere:

NuGet
npm
Go module
GitHub

Other Resources

Semantic Search in C# - Embedding
Sentiment Analysis in C# - Positive, negative, neutral, emotions and toxicity
RAG in C# Without a Vector Database e end to end
Your MiniLM Embeddings Are Probably Truncating at 256 Tokens - Measured, with the cost
BM25 vs TF-IDF: Keyword Search Explained
What are Vector Embeddings? - How machines understand meaning through numbers

Semantic Search in C# Without Python

Ólafur Aron Jóhannsson — Sat, 14 Feb 2026 10:51:36 +0000

Semantic Search in C#

Keyword search finds documents that contain the words you typed. Semantic search finds documents that mean what you meant.

Search for "how to change my login credentials". Keyword search returns nothing because none of your documents contain those exact words. Semantic search returns "How do I reset my password?" because the meaning is the same.

dotnet add package Kjarni

NuGet

using Kjarni;

using var embedder = new Embedder("minilm-l6-v2");
Console.WriteLine(embedder.Similarity("doctor", "physician")); // 0.8598
Console.WriteLine(embedder.Similarity("doctor", "banana"));    // 0.3379

No API key. No Python. No vector database. The model runs locally on CPU.

How Semantic Search Works

The core idea: convert text into numbers that capture meaning.

A sentence embedding model reads text and outputs a vector — an array of floating-point numbers, typically 384 or 768 dimensions. Texts with similar meaning produce vectors that are close together. Texts with different meaning produce vectors that are far apart.

"doctor"    -> [0.12, -0.34, 0.56, 0.78, ...]  (384 numbers)
"physician" -> [0.11, -0.33, 0.55, 0.79, ...]  (384 numbers)  <- close
"banana"    -> [-0.45, 0.23, -0.12, 0.01, ...]  (384 numbers)  <- far

You measure the distance between vectors using cosine similarity. The score ranges from -1 (opposite) to 1 (identical).

For a deeper explanation of how embeddings work, see What are Vector Embeddings?.

Encoding Text

using var embedder = new Embedder("minilm-l6-v2");
float[] vector = embedder.Encode("Hello world");
Console.WriteLine(vector.Length);                          // 384
Console.WriteLine(string.Join(", ", vector[..5]));
// -0.034477282, 0.03102318, 0.006734989, 0.02610899, -0.03936202

The model downloads on first use (~90MB) and caches locally. Every call to Encode() runs the full transformer: tokenization, attention layers, mean pooling, normalization. The output is a unit-length vector ready for cosine similarity.

These are the same vectors you'd get from Python's sentence-transformers:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
vector = model.encode("Hello world", normalize_embeddings=True)
# [-0.03447726  0.03102319  0.00673499  0.02610895 -0.03936201]

Same model, same weights, same output.

Measuring Similarity

Cosine similarity tells you how close two vectors are:

using var embedder = new Embedder("minilm-l6-v2");

var pairs = new[] {
    ("doctor", "physician"),
    ("doctor", "hospital"),
    ("doctor", "banana"),
    ("cat", "dog"),
    ("cat", "car"),
    ("machine learning", "artificial intelligence"),
    ("machine learning", "potato soup"),
};

foreach (var (a, b) in pairs)
    Console.WriteLine($"  {embedder.Similarity(a, b):F4}  \"{a}\" / \"{b}\"");

  0.8598  "doctor" / "physician"
  0.5971  "doctor" / "hospital"
  0.3379  "doctor" / "banana"
  0.6606  "cat" / "dog"
  0.4633  "cat" / "car"
  0.7035  "machine learning" / "artificial intelligence"
  0.1848  "machine learning" / "potato soup"

The scores match intuition. "Doctor" and "physician" are near-synonyms (0.86). "Cat" and "dog" are related but different (0.66). "Machine learning" and "potato soup" have almost nothing in common (0.18).

Building a Search

Here's the pattern. Encode your documents once. Encode the query at search time. Rank by cosine similarity.

using var embedder = new Embedder("minilm-l6-v2");

var docs = new[] {
    "How do I reset my password?",
    "What is your refund policy?",
    "Do you ship internationally?",
    "How do I update my billing address?",
    "Where can I track my order?",
};

// Encode all documents (do this once, store the vectors)
var vectors = embedder.EncodeBatch(docs);

// Search
var query = embedder.Encode("I need to change my login credentials");

var results = docs
    .Select((doc, i) => (doc, score: Embedder.CosineSimilarity(query, vectors[i])))
    .OrderByDescending(x => x.score);

foreach (var (doc, score) in results)
    Console.WriteLine($"  {score:F4}: {doc}");

  0.5981: How do I reset my password?
  0.4067: How do I update my billing address?
  0.0767: Where can I track my order?
 -0.0027: What is your refund policy?
 -0.0451: Do you ship internationally?

"Change my login credentials" matches "reset my password" at 0.60 despite sharing zero keywords. That's the entire value of semantic search.

"Update my billing address" comes second. The model understands that changing account information is related even though the specific fields differ.

FAQ Matching

Route support tickets to the most relevant FAQ:

using var embedder = new Embedder("minilm-l6-v2");

var faqs = new[] {
    "How do I cancel my subscription?",
    "How do I get a refund?",
    "How do I change my email address?",
    "What payment methods do you accept?",
    "How do I contact support?",
};
var faqVectors = embedder.EncodeBatch(faqs);

string MatchFaq(string userQuestion)
{
    var queryVec = embedder.Encode(userQuestion);
    var best = faqs
        .Select((faq, i) => (faq, score: Embedder.CosineSimilarity(queryVec, faqVectors[i])))
        .OrderByDescending(x => x.score)
        .First();

    return best.score > 0.4 ? best.faq : "No matching FAQ found.";
}

Console.WriteLine(MatchFaq("I want to stop paying"));
// How do I cancel my subscription?

Console.WriteLine(MatchFaq("Can I pay with crypto?"));
// What payment methods do you accept?

Encode your FAQs once at startup, store the vectors, and only encode the user's query at request time.

Deduplication

Find near-duplicate content in a dataset:

var texts = GetAllDocuments();
var vectors = embedder.EncodeBatch(texts);

var duplicates = new List<(string, string, float)>();

for (int i = 0; i < texts.Length; i++)
    for (int j = i + 1; j < texts.Length; j++)
    {
        var sim = Embedder.CosineSimilarity(vectors[i], vectors[j]);
        if (sim > 0.85)
            duplicates.Add((texts[i], texts[j], sim));
    }

A threshold of 0.85 catches rephrased content while ignoring merely related documents.

Combining with Sentiment

Find relevant reviews about a topic, then check their sentiment. See Sentiment Analysis in C# for more on the classification side.

using var embedder = new Embedder("minilm-l6-v2");
using var classifier = new Classifier("roberta-sentiment");

var query = embedder.Encode("battery life");
var relevant = reviews
    .Select(r => (review: r, score: Embedder.CosineSimilarity(query, embedder.Encode(r))))
    .Where(x => x.score > 0.3)
    .OrderByDescending(x => x.score);

foreach (var (review, score) in relevant)
    Console.WriteLine($"{classifier.Classify(review)}  \"{review}\"");

Choosing a Model

Model	Dimensions	Speed	Quality
`minilm-l6-v2`	384	Fast	Good
`mpnet-base-v2`	768	Slower	Better

Start with minilm-l6-v2. It's the most widely used embedding model in production and handles most use cases well. Switch to mpnet-base-v2 if you need higher quality and can afford the extra latency and memory.

Both models have a 512 token input limit (~300-400 words). Longer text gets truncated. If your documents are long, split them into chunks first — which is exactly what the document search engine does for you automatically.

When to Use Semantic Search vs Keyword Search

Semantic search is not always better than keyword search.

Semantic search works best when:

Users don't know the exact terminology
You're matching intent, not words ("change login" → "reset password")
Documents are short (FAQs, product descriptions, support tickets)

Keyword search works best when:

Users search for exact terms (error codes, product IDs, proper nouns)
You need exact phrase matching
Documents are long and repetitive keywords matter

The best approach is usually both. See Build a Document Search Engine in C# for a hybrid search implementation that combines BM25 keyword search with semantic vectors and reranking.

For the theory behind keyword search, see BM25 vs TF-IDF: Keyword Search Explained.

How This Works

Kjarni loads HuggingFace sentence-transformer models directly from safetensors. The inference engine is written in Rust. The C# package wraps a single native library.

The outputs match Python's sentence-transformers

NuGet:  https://www.nuget.org/packages/Kjarni
GitHub: https://github.com/olafurjohannsson/kjarni

Other Resources

Semantic Search in C# - Embeddings and similarity from scratch
Build a Document Search Engine in C# - Full hybrid search with indexing and reranking
BM25 vs TF-IDF: Keyword Search Explained - How keyword search works under the hood
What are Vector Embeddings? - How machines understand meaning through numbers

Sentiment Analysis in C# - Without Python or External APIs

Ólafur Aron Jóhannsson — Fri, 13 Feb 2026 09:13:20 +0000

Sentiment Analysis

You have text. You want to know if it's positive, negative, or neutral.

The usual options:

Azure Cognitive Services - API call per request, pay per character
ML.NET - train your own model, bring your own dataset
Python sidecar - run Flask next to your .NET app, serialize everything as JSON

There's a simpler option. Load a pretrained transformer model and run it locally.

dotnet add package Kjarni

NuGet

using Kjarni;

using var classifier = new Classifier("roberta-sentiment");
Console.WriteLine(classifier.Classify("I love this product!"));
// positive (98.5%)

The model downloads on first use, then caches locally, and runs on CPU by default, but GPU is also available.
No API key. No Python. No container.

How Sentiment Models Work

A sentiment classifier is a neural network trained on labeled text.
The model has seen millions of examples like:

Text	Label
"This movie was fantastic"	positive
"Terrible customer service"	negative
"The package arrived on Tuesday"	neutral

At inference time, the model reads your input text, encodes it into a
high-dimensional vector, and runs that vector through a classification
head that outputs a probability for each label.

Input text -> Tokenizer -> Transformer Encoder -> Classification Head -> Probabilities
                                                                         positive: 98.5%
                                                                         neutral:   1.2%
                                                                         negative:  0.3%

You pass in text, you get back a label and a score.

Three-Class Sentiment

The roberta-sentiment model classifies text as positive, negative, or neutral.
It was trained on ~124M tweets and handles informal text, slang, and emoji well.

using var classifier = new Classifier("roberta-sentiment");

var inputs = new[] {
    "I love this product!",
    "Terrible quality, broke after one day.",
    "It's okay I guess.",
    "The packaging was nice but the product itself was mediocre.",
    "Just received my order",
};

foreach (var text in inputs)
    Console.WriteLine($"{classifier.Classify(text)}  \"{text}\"");

positive (98.5%)  "I love this product!"
negative (94.1%)  "Terrible quality, broke after one day."
positive (52.9%)  "It's okay I guess."
negative (81.4%)  "The packaging was nice but the product itself was mediocre
positive (57.5%)  "Just received my order"

Notice the nuance. "It's okay I guess" is technically positive but barely, at
The model picks up on hedging. "The packaging was nice but the product itself
is classified negative because the overall sentiment leans that way despite t
positive clause.

Getting All Scores

Classify() returns the top label. To see the full probability distribution:

var result = classifier.Classify("The packaging was nice but the product itself was mediocre.");
Console.WriteLine(result.ToJson());

{
  "label": "negative",
  "score": 0.8138,
  "predictions": [
    {"label": "negative", "score": 0.8138},
    {"label": "neutral", "score": 0.1615},
    {"label": "positive", "score": 0.0247}
  ]
}

The scores sum to 1.0. The model is 81.4% confident this is negative,
16.2% neutral, 2.5% positive. In production, you might treat anything
below 70% confidence as "uncertain" rather than taking the label at face valu

Five-Star Sentiment (Multilingual)

If you need finer granularity, bert-sentiment-multilingual maps text to a 1-5 star rating.
It works across English, German, French, Spanish, Italian, and Portuguese.

using var classifier = new Classifier("bert-sentiment-multilingual");

var inputs = new[] {
    ("en", "Absolutely amazing!"),
    ("es", "Esta es la peor compra que he hecho."),
    ("de", "Das ist ganz okay."),
    ("fr", "C'est terrible."),
    ("it", "Non male, ma potrebbe essere meglio."),
};

foreach (var (lang, text) in inputs)
    Console.WriteLine($"[{lang}] {classifier.Classify(text)}  \"{text}\"");

[en] 5 stars (96.7%)  "Absolutely amazing!"
[es] 1 star (94.1%)   "Esta es la peor compra que he hecho."
[de] 3 stars (77.7%)  "Das ist ganz okay."
[fr] 1 star (70.4%)   "C'est terrible."
[it] 3 stars (83.7%)  "Non male, ma potrebbe essere meglio."

Same API. The model handles language detection internally.

Emotion Detection

Sentiment tells you positive or negative. Emotion tells you why.

The distilroberta-emotion model classifies text into seven emotions:
anger, disgust, fear, joy, neutral, sadness, and surprise.

using var classifier = new Classifier("distilroberta-emotion");

var inputs = new[] {
    "I just got promoted!",
    "My dog passed away yesterday.",
    "I can't believe they did that to me.",
    "I'm so nervous about the interview tomorrow.",
};

foreach (var text in inputs)
    Console.WriteLine($"{classifier.Classify(text)}  \"{text}\"");

surprise (50.7%)  "I just got promoted!"
sadness (98.4%)   "My dog passed away yesterday."
surprise (89.2%)  "I can't believe they did that to me."
fear (99.4%)      "I'm so nervous about the interview tomorrow."

"I just got promoted!" is interesting. The model sees it as surprise more than joy.
If you need the full breakdown:

var result = classifier.Classify("I just got promoted!");
Console.WriteLine(result.ToJson());

{
  "label": "surprise",
  "score": 0.5066,
  "predictions": [
    {"label": "surprise", "score": 0.5066},
    {"label": "anger", "score": 0.2376},
    {"label": "joy", "score": 0.0980},
    {"label": "neutral", "score": 0.0664},
    {"label": "disgust", "score": 0.0658},
    {"label": "sadness", "score": 0.0221},
    {"label": "fear", "score": 0.0035}
  ]
}

For finer-grained emotions, the roberta-emotions model detects 28 labels
including admiration, amusement, curiosity, gratitude, and others.

Toxicity Detection

Content moderation is a specific form of classification. The toxic-bert model
scores text across six categories simultaneously:

using var classifier = new Classifier("toxic-bert");
Console.WriteLine(classifier.Classify("You are an idiot").ToDetailedString())

             toxic   98.61%  ███████████████████████████████████████
            insult   96.00%  ██████████████████████████████████████
           obscene   75.64%  ██████████████████████████████
      severe_toxic    4.56%  █
     identity_hate    1.41%
            threat    0.13%

This is a multi-label model, meaning multiple categories can be true at t
A comment can be both toxic and an insult. The scores are independent, not co

Compare with something benign:

Console.WriteLine(classifier.Classify("I respectfully disagree with your poin

             toxic    0.07%
           obscene    0.02%
            insult    0.02%
     identity_hate    0.01%
            threat    0.01%
      severe_toxic    0.01%

All scores near zero. The model is confident this is not toxic.
In production you'd set a threshold (say 80%) and only flag content above it.

Bulk Processing

For analyzing many texts, loop through them:

using var classifier = new Classifier("roberta-sentiment");

var reviews = new[] {
    "Fast shipping, great product",
    "Arrived damaged, no response from support",
    "Does what it says",
    "Best purchase I've made this year",
    "Meh",
};

foreach (var text in reviews)
    Console.WriteLine($"{classifier.Classify(text)}  \"{text}\"");

positive (97.2%)  "Fast shipping, great product"
negative (85.3%)  "Arrived damaged, no response from support"
neutral (82.0%)   "Does what it says"
positive (98.1%)  "Best purchase I've made this year"
neutral (62.3%)   "Meh"

Practical Patterns

Confidence Thresholding

Don't trust low-confidence predictions blindly:

var result = classifier.Classify(text);

if (result.Score > 0.80)
    // High confidence, act on it
    SaveSentiment(text, result.Label);
else
    // Low confidence, flag for review or use a neutral default
    FlagForReview(text, result);

Aggregating Sentiment

To get the overall sentiment of a product from many reviews:

var results = reviews.Select(r => classifier.Classify(r)).ToArray();

var summary = results
    .GroupBy(r => r.Label)
    .ToDictionary(g => g.Key, g => g.Count());

// { "positive": 847, "negative": 121, "neutral": 232 }

Combining with Search

Sentiment pairs well with [semantic search](https://kjarni.ai/blog/semanticse
Find relevant documents first, then analyze their sentiment:

using var embedder = new Embedder("minilm-l6-v2");
using var classifier = new Classifier("roberta-sentiment");

// Find reviews about shipping
var query = embedder.Encode("shipping and delivery experience");
var relevant = reviews
    .Select(r => (review: r, score: Embedder.CosineSimilarity(query, embedder
    .Where(x => x.score > 0.3)
    .OrderByDescending(x => x.score);

// Classify only the relevant ones
foreach (var (review, score) in relevant)
    Console.WriteLine($"{classifier.Classify(review)}  \"{review}\"");

One thing worth knowing before you run this over real data: minilm-l6-v2 reads
256 tokens, roughly 900 characters, and silently drops anything past that. For
short reviews it does not matter. For long ones, chunk them first. Measured i
Your MiniLM Embeddings Are Probably Truncating at 256 Tokens.

Which Model to Use

Need	Model	Labels
Quick positive/negative	`distilbert-sentiment`	2 (pos/neg)
Positive/negative/neutral	`roberta-sentiment`	3
Star rating (multilingual)	`bert-sentiment-multilingual`	5 (1-5 stars)
Basic emotion	`distilroberta-emotion`	7
Fine-grained emotion	`roberta-emotions`	28
Content moderation	`toxic-bert`	6 (multi-label)

Start with roberta-sentiment. If you need more detail, move to the multilin
or emotion models. They all work the same way, same API, different model name.

How This Works Under the Hood

Kjarni loads HuggingFace models
directly from safetensors using memory-mapped I/O. The inference engine is
written in Rust with hand-tuned SIMD kernels. The C# package wraps a single
native library.

These are the same models used by Python's transformers and `sentence-trans
libraries. The outputs match to four decimal places. The difference is you do
need a Python runtime or 2GB of pip dependencies.

The same engine runs everywhere:

NuGet
npm
Go module
GitHub

Other Resources

Semantic Search in C# - Embedding
Build a Document Search Engine in C# - Full hybrid search with indexing and reranking
ML from the Command Line - The same models as a UNIX tool that reads stdin and writes JSON
Your MiniLM Embeddings Are Probably Truncating at 256 Tokens - A bug this engine had, and what it costs you
BM25 vs TF-IDF: Keyword Search Explained - How keyword search works under the hood
What are Vector Embeddings? ning through numbers

What are Vector Embeddings?

Ólafur Aron Jóhannsson — Sun, 19 Oct 2025 07:00:25 +0000

It's just matrices

Vector embeddings serve a very important piece in technology today, as their application is so useful, as they capture
meaning of natural language.

You've used them before

Netflix/Spotify uses it for their recommendation systems (because you watched X...)
Duplicate detection
Retrieval Augmented Generation (RAG) systems, retreive relevant text from a corpora
Content moderation
Question Answering, match the intent, not just keywords (How do i reset password -> Forgot login credentials)

Turn text into numbers in high-dimensional space. More dimensions = more detail about meaning. MiniLM uses 384. Bigger models go to 1024+, but cost more compute and memory.

This is how computers know "doctor" and "physician" mean the same thing despite sharing zero letters.

Try It Yourself

→ Live Interactive Demo

Demo

How It Works

Neural networks learn to map words to vectors by training on billions of text examples. Words that appear in similar contexts end up with similar vectors.

The network learns patterns like:

"The doctor prescribed medication"
"The physician prescribed medication"

Since "doctor" and "physician" appear in similar contexts, they get similar embeddings.

Why 384 Dimensions?

Each embedding is 384 numbers (for MiniLM-L6-v2). Why so many?

Each dimension captures a different aspect of meaning:

Dimension 1 might encode "is this a profession?"
Dimension 47 might encode "medical-related?"
Dimension 203 might encode "human-related?"

The model learns these automatically from data. We can't interpret individual dimensions, but the full vector captures nuanced meaning.

Measuring Similarity

Cosine similarity measures how "aligned" two vectors are:

1.0 = identical direction (same meaning)
0.0 = perpendicular (unrelated)
-1.0 = opposite direction (antonyms)

fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
    let dot: f32 = a.iter().zip(b).map(|(x, y)| x * y).sum();
    let norm_a: f32 = a.iter().map(|x| x * x).sum::<f32>().sqrt();
    let norm_b: f32 = b.iter().map(|x| x * x).sum::<f32>().sqrt();
    dot / (norm_a * norm_b)
}

What You Can Build With This

Once you have embeddings, you can:

Semantic Search - Find documents by meaning, not keywords

Clustering - Group similar items together

Recommendations - "Users who liked X also liked Y"

Duplicate Detection - Find similar content with different wording

Classification - Categorize text by meaning

All powered by comparing vectors.

The Model Behind This Demo

This demo uses EdgeBERT, my pure Rust BERT model inference implementation that runs in browsers via WebAssembly.

Model: all-MiniLM-L6-v2 (384 dimensions)

Other Resources

EdgeBERT on GitHub - The WASM library powering this demo

Alse see BM25 vs TF-IDF - BM25 vs TF-IDF

EdgeBERT: I Built My Own Neural Network Inference Engine in Rust

Ólafur Aron Jóhannsson — Fri, 12 Sep 2025 10:52:04 +0000

Lightweight BERT Embeddings in Rust (Sentence-Transformers Alternative Without Python)

I needed semantic search in my Rust app, so users could search for "doctor" and still find documents mentioning "physician" or "medical practitioner". I wanted a lightweight BERT embeddings solution in Rust, small enough to run on edge devices, browsers, and servers without headaches.

The trick is to turn text into vectors that capture meaning. Similar words → similar numbers.

doctor → [0.2, 0.5, -0.1, 0.8, ...]
physician→ [0.2, 0.4, -0.1, 0.7, ...] ✅ similar
banana → [0.9, -0.3, 0.6, -0.2, ...] ❌ different

The Pain with Existing Solutions

To compare it with Python, the standard approach is... heavy:

Just to generate embeddings, a fresh virtual environment ballooned to 6.8 GB, mostly PyTorch, tokenizers, and model weights.

The ONNX Runtime

But i was using Rust, someone mentioned ort, i'll use ONNX Runtime from Rust. How hard could it be?

    pub fn new(model_path: &str, tokenizer_path: &str) -> Result<Self> {
        let environment = ort::environment::init()
.with_execution_providers([CUDAExecutionProvider::default().build()]).commit()?;
    }
// ... 150 lines total just to get encode to work

Binary bloat

In the end it worked, but ort pulled in 80+ crates, expanded my release build to 350 MB, and relied on system libraries like libstdc++, libpthread, libm, libc. OpenSSL version mismatches added another headache.

System library conflicts

error: OpenSSL 3.3 required
$ openssl version
OpenSSL 1.1.1k  # Can't upgrade - would break RHEL dependencies

The C++ dependencies wanted OpenSSL 3.3. My RHEL system had 1.1. Removing 1.1 would break half my system.

In the beginning i was trying to build a light-weight offline RAG solution, with this one dependency turned into a major challenge.

What i actually wanted

This was the API i was looking for

model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(texts)

The Solution

So I built it my own inference engine in Rust:

use edgebert::{Model, ModelType};
let model = Model::from_pretrained(ModelType::MiniLML6V2)?;
let embeddings = model.encode(texts, true)?;

5MB binary
200MB RAM
No dependencies hell
Same accuracy (0.9997 correlation)
BLAS optional feature flag

Performance

Initial benchmarks were promising, and after optimizing the matrix multiplication routines, here's how EdgeBERT stacks up against sentence-transformers on a CPU:

Configuration	EdgeBERT	sentence-transformers
Single-threaded	`1.76ms/sentence`	5.02ms/sentence
Multi-threaded (8)	3.04ms/sentence	3.90ms/sentence
Default threads	8.52ms/sentence	`2.88ms/sentence`

👉 EdgeBERT is up to 3× faster in single-threaded scenarios. This is because MiniLM's matrices (384×384) are small, meaning the overhead from thread coordination can outweigh the benefits of parallelization. While sentence-transformers pulls ahead when using all cores on large batches, EdgeBERT's single-threaded efficiency is a key advantage for lightweight and edge applications.

CPU performance is only half the story. Memory efficiency, especially RAM usage during encoding, is critical. We can see a significant difference here:

EdgeBERT's memory footprint is not only smaller but also more stable, avoiding the large initial allocation spikes seen with the PyTorch-based solution.

Accuracy

Comparing with Python sentence-transformers:

EdgeBERT:   [-0.0344, 0.0309, 0.0067, 0.0261, -0.0394, ...]
Python:     [-0.0345, 0.0310, 0.0067, 0.0261, -0.0394, ...]
Cosine similarity: **0.9997**

Rounding differences of floating point computations, 99.97% the same.

Why This Matters (Even If You Don’t Do ML)

This enables:

Smart search: Users find what they mean, not just what they typed
Better recommendations: "If you liked X, you'll like Y" based on meaning
Duplicate detection: Find similar issues/documents even with different wording
Content moderation: Detect harmful content regardless of phrasing
RAG/AI features: Give LLMs the right context without keyword matching

All in 5MB of Rust. No Python required.

When to Use

Use EdgeBERT when:

You need embeddings without Python
Deployment size matters (5MB vs 6.8GB)
Running on edge devices or browsers
Memory is constrained
Single-threaded performance matters

Use sentence-transformers when:

You need GPU acceleration
Using multiple model architectures
Already in Python ecosystem
Need the full Hugging Face stack

WebAssembly

Since it's pure Rust with minimal dependencies, it compiles to WASM:

import init, { WasmModel, WasmModelType } from './pkg/edgebert.js';

const model = await WasmModel.from_type(WasmModelType.MiniLML6V2);
const embeddings = model.encode(texts, true);

429KB WASM binary + 30MB model weights. Runs in browsers.

Had to implement WordPiece tokenizer from scratch - the tokenizers crate has C dependencies that don't compile to WASM.

How It Works

BERT is matrix operations in a specific order:

Tokenization - WordPiece tokens to IDs
Embedding - 384-dimensional vectors (word + position + segment)
Self-attention - Q·K^T/√d, softmax, multiply by V
Feed-forward - Linear, GELU, Linear
Pooling - Average tokens into sentence embedding

Each transformer layer repeats attention and feed-forward, refining the representations. MiniLM has 6 layers.

The core implementation is ~500 lines in src/lib.rs.

No magic, just the transformer algorithm, written in Rust.

https://github.com/olafurjohannsson/edgebert

Configuration

For best performance:

# EdgeBERT - fastest single-threaded
export OPENBLAS_NUM_THREADS=1; cargo run --release --features openblas

# Python - let it auto-tune threads
python native.py

Installation

[dependencies]
edgebert = "0.3.4"

Roadmap & Future Work

EdgeBERT is focused and minimal by design, but there are exciting directions for the future:

GPU Support: Adding wgpu support for cross-platform GPU acceleration is a top priority.
More Architectures: Expanding beyond all-MiniLM-L6-v2 to support other efficient models.
Quantization: Implementing model quantization to further reduce model size and improve performance on CPU and microcontrollers.

Pull requests are always welcome!

Code

https://github.com/olafurjohannsson/edgebert

Most of the implementation is in one file. Pull requests welcome.

I built this because I needed it.

I’m sharing it because maybe you do too. 🚀

Benchmark Details

EdgeBERT: cargo run --release --features openblas --bin native with OPENBLAS_NUM_THREADS set
Python: python native.py with OMP_NUM_THREADS set
Installed sentence_transformers using pip in a venv and inspected filesize du -sh venv/lib/python*/site-packages/
Inspected Rust dependencies with cargo tree | wc -l
Inspected venv dependencies pip list | wc -l
WASM file size ls -lh examples/pkg/edgebert_bg.wasm
Native file size ls -lh target/release/native
Used /usr/bin/time -v to measure peak Maximum resident set size (kbytes)