Local Embedding Models in 2026: Which to Run, What Hardware They Need, and What They Cost

*Published 13 October 2026 · Model cards, runtime docs and API prices checked on 13 October 2026* A local embedding model turns your text into vectors on hardware you control, so no document or query leaves your machine to get embedded. As of 13 October 2026, the one to start with on a GPU is [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) (Apache 2.0, 32K-token context, 1,024 dimensions, 2.49 GiB of peak VRAM in a published test), and the one to start with on a laptop CPU is [EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2), which Google released on 6 October under Apache 2.0 with a 270M-parameter text backbone and an 8,192-token context. If you have a 16 GB GPU and quality is the only thing you care about, run Qwen3-Embedding-8B. If you are on a Raspberry Pi or prototyping, all-MiniLM-L6-v2 still works, and it is also the model most likely to be holding your retrieval back. Picking the model is the easy decision, though, and the ranking guides from Local AI Zone and AIMultiple stop there. Local setups go wrong in four other places: a missing query prefix that silently costs retrieval quality, a context limit that silently truncates your chunks, a vector index four times bigger than it needed to be, and a model switch that forces you to re-embed the whole corpus. Every figure below comes from a model card, a runtime's own documentation, or a benchmark with its hardware named, all checked on 13 October 2026. Leaderboard ranks for open and hosted models are in our [embedding model comparison from 26 September](https://www.maximem.ai/blog/best-embedding-models-2026). ## Which local embedding model to run: Qwen3-Embedding-0.6B on a GPU, EmbeddingGemma 2 on a laptop Pick by the hardware you have, then by language and licence. The table below covers the models worth running locally today; parameter counts, context limits and licences come from each model's card or from the spec table in [Atomic Chat's 25 September benchmark](https://atomic.chat/blog/guides/best-embedding-models-for-rag), and peak VRAM comes from that same benchmark (one NVIDIA RTX A6000, mostly bfloat16). | Model | Licence | Params | Dimensions | Max tokens | Peak VRAM (A6000) | Ollama download | Best for | |---|---|---|---|---|---|---|---| | all-MiniLM-L6-v2 | Apache 2.0 | 22.7M | 384 | 256 | 0.24 GiB | 46 MB | Prototypes, edge devices | | nomic-embed-text v1.5 | Apache 2.0 | 136.7M | 768 (to 64) | 8,192 | 0.47 GiB | 274 MB | Fast CPU default, long chunks | | EmbeddingGemma 2 (text) | Apache 2.0 | 270M text backbone, 740M with all encoders | 768 (to 128) | 8,192 | not yet measured | 270m to 740m tags | Laptops and phones, multimodal | | EmbeddingGemma-300M (v1) | Gemma terms | 300M class | 768 (to 128) | 2,048 | 0.96 GiB | 622 MB | Superseded by v2 for most uses | | BGE-M3 | MIT | 567.8M | 1,024 plus sparse and multi-vector | 8,192 | 1.30 GiB | ~1.2 GB | Multilingual and hybrid search | | Qwen3-Embedding-0.6B | Apache 2.0 | 595.8M | 1,024 (to 32) | 32K | 2.49 GiB | 639 MB | Default on any GPU | | Qwen3-Embedding-4B | Apache 2.0 | 4B | 2,560 | 32K | not measured | 2.5 GB | Mid-size GPUs | | Qwen3-Embedding-8B | Apache 2.0 | 7.57B | 4,096 (to 32) | 32K | 16.60 GiB | 4.7 GB (Q4_K_M) | Highest quality you can host | Ollama download sizes come from the [Ollama library](https://ollama.com/search?c=embedding) and [Morph's Ollama guide](https://www.morphllm.com/ollama-embedding-models), and they are quantized files, which is why they sit well below the bfloat16 VRAM figures. The Qwen3-Embedding-8B download grows to 8.0 GB at q8_0 and 15 GB at fp16. On quality, Atomic Chat's run on the full [BEIR](https://github.com/beir-cellar/beir) SciFact set (5,183 documents, 300 queries) put Qwen3-Embedding-8B first at 0.7953 nDCG@10, EmbeddingGemma-300M second at 0.7861, nomic-embed-text v1.5 at 0.7156, Qwen3-Embedding-0.6B at 0.7011, BGE-M3 at 0.6510 and a BM25 keyword baseline at 0.6402. Two cautions apply to that run: it is English only and dense only, and the authors did not record batch size, library versions or model revisions, and reported no significance test. Treat close scores as ties. ### EmbeddingGemma 2 brings an 8K context and Apache 2.0 to the laptop tier [EmbeddingGemma 2](https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/) is the release that changes the laptop tier. Google published it on 6 October 2026 as a 740M-parameter model that maps text, code, images, video and audio into one embedding space, and the part that matters for local use is that you can load only the 270M text backbone; Google reports about 191 MB of active RAM for the quantized text-only weights on a Pixel 11 Pro and about 567 MB for the full multimodal model. The context window grew from 2,048 tokens in version 1 to 8,192, Google reports MTEB Code rising from 68.76 to 78.68, and it already runs in [Ollama](https://ollama.com), [llama.cpp](https://github.com/ggml-org/llama.cpp), [sentence-transformers](https://sbert.net), [MLX](https://github.com/ml-explore/mlx), [vLLM](https://github.com/vllm-project/vllm) and [transformers.js](https://huggingface.co/docs/transformers.js) in the browser. No independent retrieval benchmark of version 2 existed when we checked, so the SciFact score quoted above (0.7861) is for version 1. ### Apache 2.0 and MIT cover the strong local models, with exceptions to check Qwen3-Embedding, EmbeddingGemma 2 and nomic-embed-text v1.5 are Apache 2.0, and BGE-M3 is MIT, so all of them can go into a commercial product. EmbeddingGemma version 1 sits under Google's Gemma terms and its prohibited-use policy, which is the main practical reason to move to version 2, since version 2 changed to Apache 2.0. Read the card before you ship anything else: some strong open-weight models, such as jina-embeddings-v5-text-small, are CC BY-NC 4.0 and cannot be used commercially without a separate licence. ### Published model lists often compare MTEB scores that are not comparable Morph's Ollama guide quotes MTEB scores from different tracks side by side and then warns that they cannot be compared directly. MTEB English v1, English v2, Multilingual and the BEIR retrieval-only figures are different test sets. [Pinecone's guide](https://www.pinecone.io/learn/series/rag/embedding-models-rundown/) adds the second caution: self-reported scores can be inflated when a model has been trained on data close to the benchmark. On the MTEB leaderboard, sort by the retrieval score instead of the overall average, compare models on the same track, and let a sample of your own queries make the final call. ## Most local embedding models need under 2.5 GiB of VRAM; Qwen3-Embedding-8B needs 16.60 GiB You need much less than a chat model needs. Every measured model in the table except Qwen3-Embedding-8B peaked under 2.5 GiB of VRAM at bfloat16, and the quantized Ollama files for those smaller models are about 1.2 GB or less, so a laptop with 8 GB of RAM runs any of them. Qwen3-Embedding-8B needed 16.60 GiB of VRAM at bfloat16 in Atomic Chat's run, which means a 24 GB card at full precision or the 4.7 GB Q4_K_M file on a smaller one. CPU-only is a real option for sub-billion models. Our own retrieval study ran without a discrete GPU: all-MiniLM-L6-v2 embedded 50,000 documents in 161.6 seconds (309 documents per second) on an Apple M4 with 16 GB of RAM through [ChromaDB](https://www.trychroma.com), in a single run on 14 January 2026. Parameter count is a poor guide to speed, though: in Atomic Chat's run EmbeddingGemma-300M indexed slower than the larger BGE-M3 (315.0 against 433.0 chunks per second), so measure throughput on your own hardware. Indexing time is where model size shows up. Using Atomic Chat's measured throughput on one RTX A6000 (chunks of about 250 words), here is how long one million chunks would take: | Model | Chunks per second (A6000) | Time for 1M chunks | |---|---|---| | all-MiniLM-L6-v2 | 1,074.7 | about 16 minutes | | nomic-embed-text v1.5 | 727.3 | about 23 minutes | | BGE-M3 | 433.0 | about 38 minutes | | EmbeddingGemma-300M | 315.0 | about 53 minutes | | Qwen3-Embedding-0.6B | 169.9 | about 1.6 hours | | Qwen3-Embedding-8B | 26.1 | about 10.6 hours | Atomic Chat did not record batch size, so read these times as order of magnitude; a tuned batch can move them noticeably. Choosing between the 0.6B and 8B Qwen models is a trade: 0.0942 nDCG@10 on SciFact in exchange for an indexing job about six and a half times longer and nearly seven times the peak VRAM. ## Ollama is the fastest way to get local vectors; sentence-transformers, llama.cpp, TEI and FastEmbed suit other stacks Five runtimes cover almost every setup, and the choice depends on what the rest of your stack already uses. [Ollama](https://docs.ollama.com/api/embed) is the shortest path from nothing to vectors. Pull a model, then call the /api/embed\ endpoint, which takes a single string or an array of strings and returns an embeddings\ array along with a prompt\_eval\_count\ of tokens processed: \bash ollama pull qwen3-embedding:0.6b curl http://localhost:11434/api/embed -d '{ "model": "qwen3-embedding:0.6b", "input": \["first chunk of text", "second chunk of text"\], "truncate": false }' \\ Use /api/embed\, not the older /api/embeddings\, which takes a prompt\ field, returns one vector and does not batch. Ollama also serves an OpenAI-compatible /v1/embeddings\ endpoint that accepts a dimensions\ parameter, so most OpenAI client code points at it with a base-URL change. [Sentence-transformers](https://sbert.net) is the Python default for anything on [Hugging Face](https://huggingface.co), and it also ships the quantize\_embeddings\ helper for shrinking stored vectors: \python from sentence\_transformers import SentenceTransformer model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B") doc\_vecs = model.encode(chunks, batch\_size=64, normalize\_embeddings=True) query\_vecs = model.encode(queries, prompt\_name="query", normalize\_embeddings=True) \\ Qwen3-Embedding needs sentence-transformers 2.7.0 or later and transformers 4.51.0 or later, according to its card; older transformers versions fail with KeyError: 'qwen3'\. [llama.cpp](https://github.com/ggml-org/llama.cpp) runs GGUF files with the lowest overhead. Start llama-server -m model.gguf --embeddings --pooling mean\ (match --pooling\ to the method the model was trained with, which its card states) and it serves an OpenAI-compatible /v1/embeddings\ endpoint with normalized vectors. [Text Embeddings Inference](https://huggingface.co/docs/text-embeddings-inference) (TEI) from Hugging Face is the production server, shipped as a Docker image for GPUs and buildable for x86 CPUs or Apple Metal, and its own CPU guide uses Qwen3-Embedding-0.6B as the example model. [FastEmbed](https://qdrant.github.io/fastembed/) from [Qdrant](https://qdrant.tech) runs models on ONNX Runtime without PyTorch, which makes it the lighter choice for serverless functions and small containers. ### A missing query prefix lowers retrieval quality without any error Most modern embedding models expect a different prefix on queries than on documents (Qwen3's card calls it an instruction), and nothing fails when you leave it out; retrieval simply gets worse. [nomic-embed-text v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) states that every input must start with search\_document: \ or search\_query: \. Qwen3-Embedding wants queries as Instruct: {task description}\\nQuery:{query}\ and documents with no instruction, and its card credits that prefix with a 1% to 5% gain on most tasks. EmbeddingGemma uses task: search result | query: {query}\ for queries and title: none | text: {content}\ for documents. Morph points out that the [LangChain](https://www.langchain.com) and [LlamaIndex](https://www.llamaindex.ai) Ollama wrappers do not add these prefixes for you, and reports that a 5 to 10 point gap between published and observed retrieval quality often traces back to that. Write the prefix into your own embedding function so it cannot be forgotten. ### Ollama truncates chunks past the context limit unless you set truncate: false\ Context limits range from 256 tokens for all-MiniLM-L6-v2 and 512 for mxbai-embed-large and multilingual-e5-large to 8,192 for nomic, BGE-M3 and EmbeddingGemma 2 and 32K for Qwen3-Embedding. Anything longer is cut off without an error by default; Ollama's truncate\ parameter defaults to true\, and setting it to false\ turns overflow into an error you can catch. Ollama also loads nomic-embed-text with a shorter context than the model supports, so set num\_ctx\ to 8192 if your chunks are long. To catch silent truncation, compare the prompt\_eval\_count\ Ollama returns against your own token count for each chunk. ## A million 1,024-dimension vectors take 4.1 GB, and quantization cuts that 4x to 32x Raw storage is the number of vectors times the dimensions times four bytes for float32. For one million chunks, that is 1.5 GB at 384 dimensions, 3.1 GB at 768, 4.1 GB at 1,024 and 16.4 GB at 4,096, before the extra memory your vector database spends on its search graph. Atomic Chat's SciFact index shows the same ratio in practice: 38.1 MiB for 1,024-dimension models against 152.5 MiB for Qwen3-Embedding-8B's 4,096. Matryoshka truncation and vector quantization both cut that down, and you can use both. Matryoshka-trained models let you keep only the first part of each vector. Qwen3-Embedding-8B scored 0.7960 on SciFact at 4,096 dimensions, 0.7905 at 1,024 and 0.7672 at 256 in Atomic Chat's tests, so cutting to 1,024 dimensions gives up 0.0055 nDCG@10 for a quarter of the storage; nomic-embed-text v1.5 drops from 62.28 to 61.04 MTEB average going from 768 to 256 dimensions, by Morph's table. Re-normalize after truncating, and use the same dimension for queries and documents. Quantizing the stored vectors shrinks them further. Hugging Face's [embedding quantization study](https://huggingface.co/blog/embedding-quantization) (March 2024, mxbai-embed-large-v1 on 15 MTEB retrieval datasets) found int8 vectors are 4x smaller and keep about 97% of retrieval quality, rising to about 99% when the top candidates are rescored with full-precision query vectors, while binary vectors are 32x smaller and keep about 92.5%, or about 96% with rescoring. At the scale of their worked example, 250 million 1,024-dimension vectors take 953.67 GB as float32 and 29.80 GB as binary. Sentence-transformers ships the quantize\_embeddings\ helper for both. ## A local embedding model is rarely much cheaper than an API: embedding 1M chunks once on text-embedding-3-small costs about $6.66 Local saves less than the API price suggests, and cost is rarely the best reason to go local. [OpenAI's text-embedding-3-small](https://developers.openai.com/api/docs/models/text-embedding-3-small) costs $0.02 per million tokens and text-embedding-3-large $0.13. Suppose you have one million chunks of about 330 tokens each, which is 333 million tokens: embedding all of them once costs about $6.66 on the small model and about $43.29 on the large one. A local GPU does the same job for the price of electricity, but only if the GPU already exists, and somebody still owns the server and the on-call rota. Four situations tip the arithmetic toward local: you re-index often (every chunking change means paying again on an API), your query volume is high enough that per-call latency and cost add up, you need the exact same vectors a year from now, or the text cannot leave your network. Reproducibility and privacy matter most. A local model file gives identical embeddings for as long as you keep it, while a hosted model can change versions behind the same name. If the text cannot leave your network, an API is ruled out before cost comes up. Local is the wrong call when your volume is small and nobody on the team wants to run inference. It is also wrong when your hardware cannot hold a model good enough for your hardest queries, since multi-hop questions and rare jargon are where the largest hosted models still pull ahead, as [AI/TLDR](https://ai-tldr.dev/learn/local-open-models/open-model-ecosystem/local-embedding-models/) notes. Run both on a labelled sample of your own queries before deciding; it takes an afternoon. ## Switching embedding models means re-embedding everything Vectors from different models live in different spaces, even when the dimensions match, so you cannot query a Qwen3 index with an EmbeddingGemma query vector or mix the two in one collection. Changing models means re-embedding every document, and the indexing table above is your cost estimate; for a million chunks the move from all-MiniLM-L6-v2 to Qwen3-Embedding-8B goes from about 16 minutes to about 10.6 hours on one A6000. A migration without downtime is ordinary engineering. Write new documents to both indexes, backfill the new index in batches, compare recall on a labelled query set while reads still go to the old one, switch reads once the new index wins, and keep the old index until the rollback window passes. Store the model name, revision, prefix, dimension and normalization next to every vector, because that metadata is the only record of how the old vectors were made. ## The embedding model is often not your retrieval bottleneck, so test recall first Keyword search beat vector search in our own study once a single code dataset was taken out, which is a reason to measure before you swap models. In our [50,000-document retrieval study](https://www.maximem.ai/blog/file-rag-vs-vector-rag), vector search led keyword search across five datasets on MRR@10 by 0.6320 to 0.5325, but almost all of that lead came from one natural-language-to-code dataset; remove it and keyword search led the remaining four, 0.5931 to 0.5614, and on science exam questions keyword search scored 0.8145 against 0.6142. The setup was all-MiniLM-L6-v2 at 384 dimensions, [Tantivy](https://github.com/quickwit-oss/tantivy) 0.22.0 and ChromaDB on an Apple M4, 5,000 queries scored by exact document-ID matching with no LLM judge, in one run on 14 January 2026. A stronger embedder would narrow keyword search's lead on those four datasets. It would not remove the class of query where the exact term (a product code, a drug name, an error string, a ticket ID) is the whole point, which is why hybrid search, with BGE-M3's sparse output or a separate keyword index, belongs in most local setups. Rerankers solve a narrower problem. A cross-encoder reranker reorders the candidates the embedder already found, so it helps when the right chunk sits in the top 50 but not the top 5, and it cannot recover a chunk the first stage missed. Finding your bottleneck takes about ten lines of code: label which chunk answers each of 50 to 100 real queries, embed them with two candidate models using the correct prefixes, and measure how often the right chunk lands in the top 10. \python import numpy as np from sentence\_transformers import SentenceTransformer def recall\_at\_k(model\_id, chunks, queries, gold, k=10, query\_prompt=None): m = SentenceTransformer(model\_id) d = m.encode(chunks, normalize\_embeddings=True, batch\_size=64) q = m.encode(queries, prompt\_name=query\_prompt, normalize\_embeddings=True) top = np.argsort(-q @ d.T, axis=1)\[:, :k\] return float(np.mean(\[g in row for g, row in zip(gold, top)\])) \\ If both models land within 0.02 of each other (two percentage points of recall), the model is not your problem; look at chunking and query prefixes first. ## Running embeddings locally keeps text private, but the vectors can still leak it Running the model yourself removes a third-party processor from the embedding step, which shortens your list of sub-processors and keeps raw text inside your network. It does not make the vectors safe to treat as anonymous. In [Morris and colleagues' 2023 EMNLP paper](https://aclanthology.org/2023.emnlp-main.765), an inversion method recovered 92% of 32-token inputs exactly from their embeddings and recovered full names from a dataset of clinical notes. So the vector store needs the same access controls and residency rules as the documents it was built from, and a deletion request has to reach the vectors as well as the source rows. ## Where Maximem Synap fits: agent memory needs more than similarity A local embedding model finds passages that are similar to a query. For agent memory, similarity search leaves four questions open: it cannot tell which of two stored facts is current, that "my manager" in one conversation and "Sarah" in another are the same person, which user a fact belongs to, or when a fact should stop being returned at all. [Maximem Synap](https://www.maximem.ai) is a managed memory layer that takes care of those parts, and the embedding step runs on Maximem's own infrastructure, so text is not sent to a third party for it. Teams that need everything inside their own network can deploy Synap self-hosted or air-gapped on enterprise plans. If your retrieval feeds an agent that has to remember users across sessions, run the recall test above on your own queries first; it tells you whether the embedder or the memory around it is the part to fix. ## Frequently asked questions ### What is the best local embedding model in 2026? As of 13 October 2026, Qwen3-Embedding-0.6B is the best default if you have a GPU, Qwen3-Embedding-8B scored highest on SciFact in Atomic Chat's test and fits on one 24 GB GPU (16.60 GiB of VRAM at bfloat16), and EmbeddingGemma 2 is the strongest candidate for laptops and multimodal search, though no independent retrieval benchmark of it existed on that date. All three are Apache 2.0. Run the recall test above on two of them with 50 real queries before committing. ### Can I run an embedding model on CPU only? Yes. Sub-billion models such as all-MiniLM-L6-v2, nomic-embed-text v1.5 and EmbeddingGemma 2 run comfortably on a laptop CPU; all-MiniLM-L6-v2 embedded 50,000 documents in 161.6 seconds on an Apple M4 with no discrete GPU in our study. Keep the 4B and 8B Qwen3 models for machines with a GPU. ### What is the best Ollama embedding model? For most RAG work, use qwen3-embedding:0.6b\ (639 MB) if quality matters and nomic-embed-text\ (274 MB) if CPU speed matters; embeddinggemma-2\ is now the small multimodal option. Call /api/embed\, add the model's query and document prefixes yourself, and set num\_ctx\ for nomic if your chunks are long. ### How much storage do embeddings take? One million float32 vectors take 4.1 GB at 1,024 dimensions and 16.4 GB at 4,096, plus your vector database's index overhead. Int8 quantization makes that 4x smaller. Binary makes it 32x smaller, and Matryoshka truncation cuts it further on models trained for it. ### Do I need to re-embed my data if I change embedding models? Yes. Vectors from different models are not comparable even at the same dimension, so every document and query has to be embedded with the new model. Plan it as a backfill into a second index, and budget the time from your model's measured throughput. ### Are local embeddings private? The raw text stays on your hardware, which removes a third party from that step. The vectors themselves can be partly inverted back into text, so protect and delete them the way you would the original documents. ## Sources All retrieved 13 October 2026. - [Qwen3-Embedding-0.6B model card](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) - [EmbeddingGemma 2 announcement, Google, 6 October 2026](https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/) and [model card](https://huggingface.co/google/embeddinggemma-2) - [EmbeddingGemma-300M model card](https://huggingface.co/google/embeddinggemma-300m) - [nomic-embed-text v1.5 model card](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) - [Best Local Embedding Models for RAG: 7 Models Compared, Atomic Chat, 25 September 2026](https://atomic.chat/blog/guides/best-embedding-models-for-rag) - [Best Ollama Embedding Models 2026, Morph, 9 June 2026](https://www.morphllm.com/ollama-embedding-models) - [Ollama embedding model library](https://ollama.com/search?c=embedding) and [/api/embed reference](https://docs.ollama.com/api/embed) - [llama.cpp embeddings guide](https://mintlify.wiki/ggml-org/llama.cpp/inference/embeddings) - [Text Embeddings Inference, local CPU guide](https://huggingface.co/docs/text-embeddings-inference/local\_cpu) - [FastEmbed documentation](https://qdrant.github.io/fastembed/) - [Binary and Scalar Embedding Quantization, Hugging Face, 22 March 2024](https://huggingface.co/blog/embedding-quantization) - [OpenAI text-embedding-3-small model page](https://developers.openai.com/api/docs/models/text-embedding-3-small) - [Choosing an Embedding Model, Pinecone](https://www.pinecone.io/learn/series/rag/embedding-models-rundown/) - [Best Local Embedding Models to Run Without an API, AI/TLDR, 13 June 2026](https://ai-tldr.dev/learn/local-open-models/open-model-ecosystem/local-embedding-models/) - [Text Embeddings Reveal (Almost) As Much As Text, Morris et al., EMNLP 2023](https://aclanthology.org/2023.emnlp-main.765) - [File Search vs Vector Search for RAG, Maximem](https://www.maximem.ai/blog/file-rag-vs-vector-rag)