DEV Community

Runcrate
Runcrate

Posted on

Text embedding throughput per dollar: BGE small, base and large on RTX 5090, L40S, RTX A6000 and H100

#embeddings #rag #gpu #python

What does it cost to embed a million passages with an open model, and does a faster GPU earn its price? I rented four GPUs on Runcrate today (2026-10-29), an RTX 5090, an L40S, an RTX A6000 and an H100 PCIe, measured end-to-end encode() throughput for BGE small, base and large at 128 and 512 tokens, and deleted every machine when the numbers were saved.

Short version: the L40S, at $0.54 an hour, embedded a million passages for the least money in all six cases: $0.032 per million 128-token passages with bge-small, $0.49 per million 512-token passages with bge-large. The H100 was the fastest card only for bge-base and bge-large, and cost 3.6 to 6.8 times more per passage than the L40S. With bge-small, the H100 was slower than the RTX 5090.

Setup

  • GPUs (one per machine), on-demand price at launch time: RTX 5090 32 GB (Austin, US, $1.15 per hour, 16 vCPUs), L40S 48 GB (Des Moines, US, $0.54, 12 vCPUs), RTX A6000 48 GB (Montreal, CA, $0.65, 28 vCPUs), H100 PCIe 80 GB (US Midwest, $3.86, 20 vCPUs).
  • Models: BAAI/bge-small-en-v1.5 (33M parameters), BAAI/bge-base-en-v1.5 (109M), BAAI/bge-large-en-v1.5 (335M), fp16.
  • Software: sentence-transformers 5.7.0, transformers 4.57.1, torch 2.8.0+cu128, Python 3.12.
  • Input: 4,096 passages per run made of random words from the model's vocabulary, long enough that every passage fills the window, then truncated to exactly 128 or 512 tokens. The content does not change the compute for an encoder; the fixed length makes runs comparable.
  • Measured: passages per second over at least 20 s of repeated model.encode(texts, batch_size=128) after a warm-up, tokenization included, plus peak GPU memory from torch.cuda.max_memory_allocated. Cost per million passages = hourly price / (passages per second x 3600) x 1,000,000.

The script 

import json, sys, time, random, torch
from sentence_transformers import SentenceTransformer
model_id, seqlen = sys.argv[1], int(sys.argv[2])
m = SentenceTransformer(model_id, device="cuda", model_kwargs={"torch_dtype": torch.float16})
m.max_seq_length = seqlen
random.seed(0)
words = [w for w in m.tokenizer.get_vocab() if w.isalpha() and len(w) > 2][:20000]
texts = [" ".join(random.choice(words) for _ in range(seqlen)) for _ in range(4096)]
m.encode(texts[:512], batch_size=128)  # warm-up
torch.cuda.synchronize()
done, t0 = 0, time.time()
while time.time() - t0 < 20:
    m.encode(texts, batch_size=128); done += len(texts)
torch.cuda.synchronize()
print(json.dumps({"model": model_id, "seqlen": seqlen, "passages_per_s": done / (time.time() - t0)}))
Enter fullscreen mode Exit fullscreen mode

Results

128-token passages, fp16, batch 128

Model GPU $/hour Passages/s Tokens/s Peak memory $ per 1M passages
bge-small-en-v1.5 RTX 5090 1.15 6,415 821,094 0.2 GB $0.050
bge-small-en-v1.5 L40S 0.54 4,714 603,443 0.2 GB $0.032
bge-small-en-v1.5 RTX A6000 0.65 3,681 471,130 0.2 GB $0.049
bge-small-en-v1.5 H100 3.86 4,952 633,882 0.2 GB $0.217
bge-base-en-v1.5 RTX 5090 1.15 4,077 521,843 0.5 GB $0.078
bge-base-en-v1.5 L40S 0.54 3,054 390,925 0.5 GB $0.049
bge-base-en-v1.5 RTX A6000 0.65 2,273 290,931 0.5 GB $0.079
bge-base-en-v1.5 H100 3.86 4,123 527,706 0.6 GB $0.260
bge-large-en-v1.5 RTX 5090 1.15 1,748 223,731 1.1 GB $0.183
bge-large-en-v1.5 L40S 0.54 1,374 175,936 1.1 GB $0.109
bge-large-en-v1.5 RTX A6000 0.65 875 111,987 1.1 GB $0.206
bge-large-en-v1.5 H100 3.86 2,391 306,061 1.1 GB $0.448

512-token passages, fp16, batch 128

Model GPU $/hour Passages/s Tokens/s Peak memory $ per 1M passages
bge-small-en-v1.5 RTX 5090 1.15 1,713 877,005 0.7 GB $0.186
bge-small-en-v1.5 L40S 0.54 1,193 610,867 0.7 GB $0.126
bge-small-en-v1.5 RTX A6000 0.65 1,068 546,867 0.7 GB $0.169
bge-small-en-v1.5 H100 3.86 1,428 731,085 0.7 GB $0.751
bge-base-en-v1.5 RTX 5090 1.15 1,033 528,998 1.4 GB $0.309
bge-base-en-v1.5 L40S 0.54 763 390,605 1.4 GB $0.197
bge-base-en-v1.5 RTX A6000 0.65 603 308,838 1.4 GB $0.299
bge-base-en-v1.5 H100 3.86 1,156 591,974 1.5 GB $0.927
bge-large-en-v1.5 RTX 5090 1.15 411 210,330 2.3 GB $0.778
bge-large-en-v1.5 L40S 0.54 305 156,109 2.3 GB $0.492
bge-large-en-v1.5 RTX A6000 0.65 234 120,064 2.3 GB $0.770
bge-large-en-v1.5 H100 3.86 598 306,330 2.3 GB $1.792

What the numbers say

  1. The L40S gave the most passages per dollar for every model and length. For 10 million 512-token chunks with bge-base that is about $2.0 on the L40S, $3.0 on the RTX A6000, $3.1 on the RTX 5090 and $9.3 on the H100 PCIe.
  2. Small models don't fill a big GPU. bge-small ran at 4,952 passages per second on the H100 and 6,415 on the RTX 5090, and on every card its tokens per second changed by 16% or less between 128 and 512 tokens. The GPU is waiting on tokenization and Python between batches, so the per-batch CPU work and the CPU itself decide the speed, not the GPU.
  3. The H100 pulled ahead only when the model was large enough. bge-large at 512 tokens: 598 passages per second on the H100, 411 on the RTX 5090, 305 on the L40S, 234 on the RTX A6000. Still, the L40S did the same work for 27% of the H100's cost.
  4. Memory is not the constraint. Peak allocation was 2.3 GB or less at batch 128, even for bge-large at 512 tokens. Any of these cards could serve several embedding models at once.
  5. The RTX 5090 and the RTX A6000 cost about the same per passage (within 11% on every row), the 5090 doing it 1.6 to 2.0x faster.

What to take from it

  • For bulk embedding with BGE-class models, a mid-range data-center card at a low hourly price was the efficient choice here. Paying for an H100 bought speed only for the large model, at 3.6 to 4.1 times the L40S's cost per passage.
  • If you use a small embedding model, throughput depends on the CPU side: tokenize ahead of time, use a fast tokenizer, or run several workers per GPU before you reach for a bigger card.
  • Real corpora have mixed lengths, and sentence-transformers sorts each call's inputs by length, so short real-world text will run faster than these fixed-length 512-token numbers.

Caveats

  • One run per setup and machine, on one day. Treat differences under about 10% as noise.
  • The machines differ in CPU count (12 to 28 vCPUs) and CPU model, which matters most for the small models (point 2).
  • No ONNX, TensorRT or text-embeddings-inference server: plain sentence-transformers in fp16, the way most pipelines start. A dedicated server would raise the small-model numbers.
  • Prices are on-demand at launch time and change daily. Billed cost for the four machines, which also ran a training benchmark, was $0.14 (RTX 5090), $0.11 (L40S, which also ran a model startup test), $0.08 (RTX A6000) and $0.43 (H100).

Reproduce it

  1. Create an API key and add an SSH key (POST /api/v1/ssh-keys).
  2. Launch a machine with POST /api/v1/instances ({"name": "...", "gpu_type": "L40S", "gpu_count": 1, "ssh_key_id": "...", "max_runtime_minutes": 40}), install torch and sentence-transformers with uv.
  3. Run python embed.py BAAI/bge-base-en-v1.5 512 (the script above), then delete the instance.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Top comments (0)