DEV Community: Moksh Gupta

OmniRoute: The Free LLM Gateway With a Terms-of-Service Catch

Moksh Gupta — Tue, 18 Aug 2026 02:33:51 +0000

If you run Claude Code, Cursor and Cline against different accounts, you already know the failure mode: one provider hits a quota mid-task and the whole session stalls while you go dig up another key. OmniRoute tries to solve that by putting every provider behind one endpoint on your own machine. I wrote a longer breakdown of it on DevToolLab, but the short version covers the parts worth knowing before you install it.

What it does

OmniRoute is an MIT-licensed proxy that runs locally on localhost:20128 and speaks the OpenAI API format at /v1. Point any OpenAI-compatible client at it and it forwards requests to whichever of roughly 300 providers its router selects, then fails over automatically when one runs dry. It picked up just over 48,000 GitHub stars in about six months, which is fast growth against LiteLLM, the incumbent in this space, sitting at 56,000-plus stars built up since mid-2023.

Because it runs on your machine rather than someone else's server, API keys stay encrypted on local disk with AES-256-GCM and your prompts never pass through a third-party cloud. That is a real difference from a hosted gateway, not a marketing line.

The failover logic is the actual product

Once you get past the free-token pitch, what you are really getting is a well-built retry system. Requests move through four tiers: subscriptions you already pay for first, then your own API keys, then cheap models, then free tiers, so a coding session degrades gracefully instead of dying outright.

Under that sit three separate recovery mechanisms. A circuit breaker only trips on 408 and 5xx errors (three failures for OAuth connections, five for API keys, two for local models) before resetting on a timer and probing again. A per-connection cooldown backs off exponentially and respects Retry-After on 429s instead of guessing at a delay. A per-model lockout benches just the failing model rather than the whole provider. Anyone who has hand-written retry logic against a flaky LLM API will recognize these as sensible, specific defaults, and there are 19 routing strategies on top, including one that pins a prompt to a single account so caching actually hits.

The 1.53 billion token number needs context

OmniRoute's headline stat sums the documented free tiers of 43 provider pools across 516 models, not a pool anyone actually hands you, and shows the running total on a built-in dashboard. To its credit, the methodology is public: shared pools count once, one-time signup credits are separated from recurring ones, and the maintainers openly decline to publish an inflated ~10 billion figure that would come from counting every rate-limit window around the clock. Mistral alone accounts for roughly 1 billion tokens a month of it.

That number moves. In 2026, Chutes, Phind and Kluster all ended or paused their free tiers, and the project re-audits its figures every two weeks because of exactly that churn.

Read the terms-of-service table before enabling everything

This is the part that gets skipped in most coverage, and it lives right in the repo at docs/reference/FREE_TIERS.md: a table of 15 providers whose terms explicitly restrict proxy access. Google Antigravity bans third-party tools accessing its service via OAuth. Fireworks prohibits proxy or intermediary use outright. A handful of consumer chat products reached through session tokens ban automated access entirely. The maintainers label each entry ok, caution, or ambiguous and are upfront that this is informational, not legal advice, which is more candor than this category usually offers.

The practical risk is account suspension, and a suspended provider account can take a subscription you actually depend on with it. I go through the exact wording of the flagged terms in the full article, which is worth five minutes before you flip every connector on.

How it stacks up

	Stars	License	Created
OmniRoute	48.1k	MIT	Feb 2026
LiteLLM	56.4k	Non-standard	Jul 2023
one-api	36.4k	MIT	Apr 2023
Portkey Gateway	12.7k	MIT	Aug 2023

LiteLLM is still the safer pick for a regulated or contract-bound environment purely on track record. For a solo developer juggling agents or a cost-sensitive side project, OmniRoute's failover is worth having on its own, independent of the free-tier math.

Trying it safely

Check that port 20128 is actually free before you assume the gateway is running (our Port Checker confirms this in a browser tab), then send one request straight to /v1/chat/completions before wiring an agent to it, so you know whether a problem is the gateway or your editor config. Read the ToS table and turn off anything you are not personally entitled to use. And before leaning on the free tiers, price the same workload on a paid API with something like our LLM Token Cost Calculator, because free tiers disappear with no warning and you want to know what the fallback actually costs.

Bottom line

The routing, the fallback tiers, and the local-first key storage are genuinely good engineering, and the MIT license means you can verify all of it yourself. The 1.53 billion free tokens are real arithmetic on paper, but a chunk of it sits behind provider terms that prohibit exactly this kind of proxy access. Use OmniRoute for what it is actually good at, and treat the token count as a ceiling with some doors marked "do not open."

References

How to Safely Run AI-Generated Code: E2B, Modal and Piston in Python

Moksh Gupta — Sun, 16 Aug 2026 17:43:48 +0000

The moment you let an LLM write code and then run it for real, you have handed a decision to a model that has no idea what your .env file contains. A chatbot that only talks is safe by construction. One that executes its own Python to check its math or plot a chart is a different animal, and something has to decide where that code is allowed to run.

Most people's first instinct is exec(), and that is exactly the wrong instinct: exec() runs inside your own process, with your environment variables and your filesystem permissions along for the ride. I wrote a longer version of this walkthrough on DevToolLab, covering E2B, Modal, and Piston, the open source engine you can self-host instead of depending on either vendor. Every snippet below was checked against the SDK versions actually installed via pip, not copied from a marketing page.

The exec() problem, quickly

Drop a model-generated string straight into exec() and it can read anything your process can read:

model_output = """
import os
print(f"I can see {len(os.environ)} environment variables from this process.")
"""
exec(model_output)

On an ordinary laptop that prints something like I can see 61 environment variables from this process, and nothing about that call needed elevated permissions. It is just a normal Python process reading its own environment, which is usually where API keys and secrets live. A sandbox exists to give the model's code its own throwaway machine so that reading os.environ returns nothing interesting.

E2B: microVMs built for this exact loop

E2B runs each sandbox as a Firecracker microVM, the same virtualization AWS built for Lambda. Install the code interpreter SDK, grab an API key from the E2B dashboard, and export it as E2B_API_KEY:

pip install e2b-code-interpreter

from e2b_code_interpreter import Sandbox

with Sandbox.create(timeout=60) as sandbox:
    execution = sandbox.run_code("data = [4, 8, 15, 16, 23, 42]\nprint(sum(data))")
    print(execution.text)

Always set an explicit timeout. run_code() hands back an Execution object, not a plain string, with .text for the printed result, .results for richer outputs like PNGs from matplotlib, .logs for stdout/stderr, and .error for a structured exception (.name, .value, .traceback) instead of a thrown Python exception. Checking .error is the idiomatic move here, since a failing snippet is not a transport failure, it is a normal result the SDK reports back correctly.

E2B's Hobby tier is free with a one-time $100 usage credit, 20 concurrent sandboxes, and one-hour sessions. Pro is $150/month, stretching sessions to 24 hours and concurrency to 100.

Modal: containers instead of microVMs

Modal takes a different route: a Sandbox is a container on Modal's general compute platform, so the same image-building tools you'd use for a Modal function work here too.

pip install modal
modal setup

modal setup opens a browser and writes a token to ~/.modal.toml, no manual API key needed. A Modal sandbox starts idle with no entrypoint, and you run commands inside it with .exec():

import modal

app = modal.App.lookup("demo", create_if_missing=True)
image = modal.Image.debian_slim(python_version="3.12")
sandbox = modal.Sandbox.create(image=image, app=app, timeout=60, block_network=True)

process = sandbox.exec("python", "-c", "print(sum([4, 8, 15, 16, 23, 42]))")
print(process.stdout.read())
sandbox.terminate()

Sandbox.create() defaults its own timeout to 300 seconds if you skip it, and block_network=True is worth setting explicitly for anything that only needs to compute, not call out. Modal's Starter plan is free with $30/month in credits and no idle charges; Sandboxes bill at roughly 3x the standard per-second rate since they're non-preemptible.

Piston: the open source, self-hosted option

Piston is a genuinely open source (MIT licensed) code execution engine that's powered Discord code bots for years. Its public demo API used to be free to hit, but as of February 15, 2026 the execute endpoint went whitelist-only, confirmed directly:

curl -s -X POST "https://emkc.org/api/v2/piston/execute" \
  -H "Content-Type: application/json" \
  -d '{"language":"python","version":"3.10.0","files":[{"name":"main.py","content":"print(1)"}]}'

That returns a message pointing you to self-hosting instead. The good news is Piston ships a ready Docker image:

docker run --privileged -v $PWD:/piston -dit -p 2000:2000 --name piston_api ghcr.io/engineer-man/piston

Once it's running, point requests at your own host instead of emkc.org, using the exact same request shape:

import requests
response = requests.post(
    "http://localhost:2000/api/v2/execute",
    json={"language": "python", "version": "3.10.0", "files": [{"name": "main.py", "content": "print('hi')"}]},
)
print(response.json()["run"]["stdout"])

Piston sandboxes each execution with isolate, the same tool competitive-programming judges use, but you own patching the host and scoping container privileges yourself, work E2B and Modal absorb as part of the price.

Picking one

If you want the least infrastructure to think about, start with E2B, its run_code() model is purpose-built for this loop. If you need a custom image, mounted volumes, or you're already deploying on Modal, its container-based Sandbox fits naturally. If neither vendor works for your situation, Piston self-hosted is the real fallback, at the cost of running it yourself. Whichever you pick, the full DevToolLab guide has the complete code for all three plus the exact pricing math, and DevToolLab's .env File Generator is a quick way to scaffold the file holding your API keys without hand-typing it.

The pattern that matters more than which SDK you choose: always set an explicit timeout, always check the structured error instead of guessing, decide deliberately whether the sandbox gets network access, and always tear it down when you're done.

References

How Much VRAM Does a Local LLM Actually Need?

Moksh Gupta — Sun, 16 Aug 2026 07:18:18 +0000

"Half a gig of VRAM per billion parameters" is the rule everyone repeats when picking hardware for a local model, and it works fine right up until the model loads, runs for a few minutes, and then dies mid-conversation with an out-of-memory error. The model fit. The context didn't. I wrote up the full math on DevToolLab, and it comes down to two numbers most people never check.

First, quantized weights are bigger than the quant name implies. Second, at long context the KV cache can weigh more than the model itself. Here's what actually happens, checked against real GGUF files on Hugging Face rather than the arithmetic.

The quant name isn't the bit count

Q4 sounds like 4 bits per weight. Multiply that by parameter count and divide by 8, and an 8B model should land at 4.01 GB. Download the actual bartowski/Meta-Llama-3.1-8B-Instruct-GGUF Q4_K_M file and it's 4.92 GB, 23 percent heavier than the math promised.

That gap isn't a rounding error. K-quants store per-block scale and minimum values next to the quantized weights, and the "M"/"L" variants keep a handful of sensitive tensors (embeddings, output layer) at higher precision. The suffix names the dominant format, not the average across the whole file. The overhead shrinks as precision goes up:

Quant	Naive size	Real size	Overhead	Real bits/weight
Q4_K_M	4.01 GB	4.92 GB	+23%	4.90
Q5_K_M	5.02 GB	5.73 GB	+14%	5.71
Q6_K	6.02 GB	6.60 GB	+10%	6.58
Q8_0	8.03 GB	8.54 GB	+6%	8.51

If you need one planning number, use 0.61 GB per billion parameters at Q4_K_M. That's where the "0.6 GB per billion" folklore actually comes from.

The KV cache is what actually runs you out of memory

Weights are a one-time cost. The KV cache grows with every token in the conversation and never shrinks until you start a new one. The formula is two tensors (K and V), per layer, per token:

cache_bytes_per_token = 2 x layers x kv_heads x head_dim x bytes_per_element

Llama 3.1 8B has 32 layers, a head dimension of 128, and only 8 KV heads (not 32 - grouped-query attention shares key/value projections across query heads, which is a 4x cache saving over full multi-head attention). Plug those in at fp16 and you get 128 KB per token:

Context	KV cache (fp16)	KV cache (q8)	Total with Q4_K_M weights
4,096	0.54 GB	0.27 GB	5.46 GB
8,192	1.07 GB	0.54 GB	5.99 GB
32,768	4.29 GB	2.15 GB	9.21 GB
131,072	17.18 GB	8.59 GB	22.10 GB

Look at the last row. The model itself is under 5 GB. The cache for its advertised 128k context window is 17.18 GB on top of that - 22.10 GB total for a model everyone calls "runs on anything." That doesn't fit on a 16 GB card and leaves nothing free on a 24 GB one. The full breakdown, including the runnable script, walks through deriving this for any model from its config.json.

Concurrency multiplies the cache, not the weights

Everything above is one conversation. Put a model behind an API two people hit at once, and the cache multiplies by however many sequences are in flight, because each one keeps its own K/V state. Four concurrent 8k sessions on this model is 4.29 GB of cache against 4.92 GB of weights. Sixteen sessions is 17.2 GB, gone, on a 24 GB card, while any single conversation still looks perfectly modest. This is why serving frameworks like vLLM invest so heavily in paged attention: naive allocation reserves space for tokens nobody has generated yet.

If you're sizing for a team instead of your own laptop, multiply the cache column by peak concurrency, and quantize the cache before you do anything else.

Rough hardware guide

VRAM	Comfortable at Q4_K_M	Realistic context
8 GB	7B-8B	8k-16k
12 GB	8B-13B	16k-32k
16 GB	13B-14B	32k, or 8B at 64k
24 GB	24B-32B	32k comfortably, 8B at full 128k
48 GB	70B at Q4	32k
96 GB+	70B at Q6+	long context on large models

Apple Silicon breaks this table a little because memory is unified, not dedicated - a 32 GB M-series Mac can hold a model plus cache that would need a 32 GB discrete GPU. It's slower per token than an equivalent NVIDIA card, but the ceiling is total RAM, not what fits on a board.

Four ways to buy headroom without a new GPU

Quantize the KV cache. The single biggest lever. llama.cpp exposes --cache-type-k / --cache-type-v, Ollama has an equivalent flag, and going to q8 roughly halves cache memory for a quality hit most people won't notice in chat.
Set context deliberately. Runtimes often reserve the full advertised window whether you use it or not. Dropping from 128k to 32k on this model frees nearly 13 GB, and most chat/coding sessions never get near 32k anyway.
Drop a quant level before dropping model size. A 13B at Q4_K_M usually beats an 8B at Q8_0 for similar memory - bigger model, cheaper weights, wins until you go below Q3, where quality visibly falls off.
Offload layers instead of giving up. llama.cpp splits layers between GPU and CPU. It gets slower, sometimes a lot slower, but a slow model beats one that won't load.

Measure it, don't just estimate it

Once the model's loaded, check the real number instead of trusting the arithmetic - allocator overhead, CUDA context and fragmentation all sit on top of the theoretical figure. On NVIDIA, nvidia-smi --query-gpu=memory.used,memory.total --format=csv run during a long conversation shows the cache filling in real time. On Apple Silicon, Activity Monitor's memory tab shows the same thing, and watch for the machine starting to swap, which shows up as tokens/sec collapsing rather than a clean error.

The tell-tale sign of a cache problem: generation is snappy for the first few exchanges, then slows hard or crashes as the conversation grows. Weights load once at startup, so a failure that shows up later is almost always the cache eating into memory you didn't budget for.

To size a model you haven't downloaded yet, pull four fields from its config.json on Hugging Face: num_hidden_layers, num_key_value_heads, hidden_size, and num_attention_heads (head dimension is hidden_size / num_attention_heads). Watch num_key_value_heads especially - when it equals num_attention_heads, the model uses full multi-head attention and its cache is several times larger per token than a GQA model of the same size. Two 8B models can differ 4x on cache memory for exactly that reason.

Sizing a local model is really two separate calculations: parameters times real bits-per-weight for the weights, and 2 x layers x kv_heads x head_dim x bytes per token for the cache. At long context the second number usually matters more than the first. Check both before you buy a card - DevToolLab's Data Storage Converter handles the GB-vs-GiB conversion, and the File Size Converter is useful for checking a downloaded GGUF against the size the repo advertises - and quantize the cache before you compromise on the model.

References

Neon vs Supabase: The Real Math Behind Scale-to-Zero Postgres in 2026

Moksh Gupta — Sat, 15 Aug 2026 17:13:40 +0000

Provisioned Postgres charges you by the hour whether anyone is querying it or not, which is a bad deal the moment your traffic is spiky, or you want a fresh database per pull request. That is the whole pitch for serverless Postgres, and in 2026 the numbers actually back it up. I ran the real arithmetic on Neon and Supabase's published rates in a longer piece on DevToolLab, and the short version is below.

What "Serverless" Actually Changes

Three things happen when compute and storage split apart. Compute can suspend to zero cost when idle, which is the entire economic argument. Storage keeps living on its own, which is what makes suspend-and-resume a few seconds instead of a restore-from-backup. And you can spin up a copy-on-write branch per PR, seeded with real data, without provisioning a second instance. For most teams the branching is the bigger workflow change; the bill is just the reason it got funded.

Where the Crossover Actually Sits

The interesting question isn't whether usage billing is cheaper, it's at what utilization it stops being cheaper. That's a two-line calculation from the vendors' own rate cards.

const HOURS_PER_MONTH = 730
const NEON_CU_HOUR = 0.106       // neon.com/pricing, Launch plan, Aug 2026
const FLAT_PLAN = 25             // supabase.com/pricing, Pro, Aug 2026

const monthlyCost = (utilization) => 1 * HOURS_PER_MONTH * utilization * NEON_CU_HOUR
const breakEvenUtilization = FLAT_PLAN / (HOURS_PER_MONTH * NEON_CU_HOUR)

console.log(`always-on: $${monthlyCost(1).toFixed(2)}/mo`)
console.log(`break-even vs $${FLAT_PLAN}/mo flat: ${(breakEvenUtilization * 100).toFixed(1)}%`)

always-on: $77.38/mo
break-even vs $25/mo flat: 32.3%

A database that's genuinely busy for roughly a third of the month costs the same either way. Below that line, usage billing wins by a widening margin, an ephemeral preview branch that wakes for ten minutes per PR is basically free. Above it, an always-on single compute unit runs $77.38 a month, over three times a flat plan. I walk through the full utilization table (5% through 100%) in the original breakdown, worth a look if you're sizing a specific workload rather than eyeballing the crossover.

One catch: Supabase's $25 includes 8 GB of disk, 250 GB of egress, auth and object storage, so it's a full backend product, not a database line item. The crossover tells you about compute, not about what you'd otherwise pay to assemble the rest.

Neon: Cheap Compute, a Quiet Repo

Neon dropped its monthly minimum entirely. Launch plan is pure usage billing at $0.106 per compute-hour and $0.35 per GB-month, suspending after 5 minutes idle (configurable down to 1 minute on Scale). A paid project sitting idle all month now costs nothing for compute, which makes per-branch, per-agent databases a rounding error instead of a line item somebody has to approve.

Databricks closed its roughly $1 billion acquisition of Neon in May 2025, and the stated reason explains the pricing: about 80% of databases on Neon were being spun up by AI agents, not people, a workload that only makes sense on per-second billing with a zero floor.

The part almost nobody checks is the open source repo. neondatabase/neon is Apache 2.0 with around 22,900 stars, but it landed exactly four commits in all of 2026, one of which just swapped a README link. Most recent tagged release: July 2025.

Use Neon as the excellent managed service it is. Don't build a plan around self-hosting it, or around upstream fixes landing on your timeline.

Supabase: A Platform That Happens to Include Postgres

Supabase Pro is $25 a month flat, 8 GB disk, 250 GB egress, a Micro compute instance included. No scale-to-zero on paid plans, the flat fee is a floor, not a cap, since the database stays up continuously.

Free tier projects pause after a week of inactivity, fine for a side project, annoying for a demo you show a client monthly. What you're really paying for on Pro is everything wrapped around the database: auth, row-level security, realtime, edge functions, a generated REST layer. Its GitHub repo (supabase/supabase, ~108,000 stars) had a commit the same day this was checked, a sharp contrast to Neon's.

Quick Comparison

	Neon	Supabase
What it is	Unbundled Postgres	Backend platform with Postgres
Paid entry	No minimum, $0.106/CU-hour	$25/month flat
Scale to zero on paid	Yes	No
Repo activity (2026)	4 commits	Daily

How to Pick

Per-branch or per-preview databases: Neon, not close. Agent or batch workloads that spike and vanish: Neon, for the same reason Databricks bought it. Need auth, storage and realtime too: Supabase, and stop comparing raw compute rates, price the whole stack. Steady traffic above roughly a third utilization: run your own version of the arithmetic above with your real compute size before assuming usage billing wins.

A Few Things Worth Getting Right

Measure your actual utilization (a week of CPU-active time divided by hours in that week) before picking a billing model, that single number decides which shape wins. Use a pooled connection from serverless functions, not a direct one per invocation, or you'll exhaust Postgres connection slots before you exhaust budget. Our Connection String Parser is handy for confirming which endpoint and sslmode you actually pasted. And don't cron a keepalive ping to dodge cold starts, that quietly converts a usage bill back into an always-on bill, the one move that erases the whole point of scale-to-zero; if you do need scheduled jobs, size the interval with something like a Cron Expression Generator instead of guessing.

Neon is the better database, Supabase is the better platform, and comparing them on compute price alone will mislead you in whichever direction you were already leaning.

References

Microsoft Now Rejects Unauthenticated Email. Here Is the DNS That Fixes It.

Moksh Gupta — Thu, 13 Aug 2026 17:32:42 +0000

Since May 5, 2025, Microsoft has been refusing mail that cannot prove where it came from. Outlook.com, Hotmail and Live.com, somewhere around half a billion active mailboxes, now bounce unauthenticated messages from high-volume senders with this:

550; 5.7.515 Access denied, sending domain does not meet the required authentication level

Google and Yahoo got there first, back in February 2024. That means all three of the inbox providers that matter now check your DNS before they accept your password resets and receipts. I put the full version of this, with every record and a longer troubleshooting section, on DevToolLab. Here is the short version.

Five Records, Two of Them Non-Negotiable

Record	Job	Status
SPF	Declares which servers may send as you	Enforced by all three
DKIM	Signs each message so tampering shows	Enforced by all three
DMARC	Binds the two above to your visible From address and sets the penalty	Enforced by all three
BIMI	Puts your logo in the inbox	Nice to have
MTA-STS	Demands TLS on inbound delivery	Nice to have

Build the first three. Treat the last two as things you add once the first three are boring.

SPF: One Line, One Trap

SPF is a TXT record naming the servers permitted to send for your domain. A receiver compares the connecting IP against that list.

yourapp.com.  IN  TXT  "v=spf1 include:_spf.resend.com include:sendgrid.net -all"

End it with -all, not ~all. The tilde is a soft fail, which politely suggests a receiver might want to consider being suspicious. The hyphen is a hard fail and actually means no.

Now the trap, because it bites quietly. SPF evaluation is capped at 10 DNS lookups, and each include: spends one. Add a marketing platform, a help desk, an invoicing tool and a transactional provider and you can cross that line without noticing. Past 10 you get a PermError, and a PermError means your genuine mail starts failing authentication with nothing obviously broken. Either flatten the record into literal IP ranges or drop services you no longer send from. Our SPF Record Checker counts the lookups for you, which is faster than reasoning about nested includes by hand.

DKIM: Signing the Message, Not the Server

SPF vouches for the machine. DKIM vouches for the content. Your provider signs outbound mail with a private key; you publish the public half at a selector subdomain, usually as a CNAME they hand you:

resend._domainkey.yourapp.com.  IN  CNAME  resend._domainkey.resend.com.

Verify by mailing yourself at Gmail and opening Show Original:

Authentication-Results: dkim=pass header.i=@yourapp.com header.s=resend

A dkim=fail almost always means the key in DNS and the key doing the signing have drifted apart, which is a re-copy from your provider's dashboard rather than a debugging session. Running your own mail server instead? Use 2048-bit RSA. Every major receiver refuses 1024-bit in 2026.

DMARC: The Policy Layer

DMARC does two jobs. It decides what happens when SPF or DKIM fails, and it insists that whichever check passed belongs to the same domain the recipient can actually see in the From line. That second part is what kills spoofing, because a message can pass SPF on an attacker's own domain while displaying yours.

Resist going straight to enforcement. A p=reject rolled out blind will silently delete real mail from a service you forgot you owned.

Month one, watch and learn:

_dmarc.yourapp.com.  IN  TXT  "v=DMARC1; p=none; rua=mailto:dmarc@yourapp.com"

Nothing is blocked. You just start receiving daily XML aggregate reports, which a free parser such as Postmark's DMARC Digests will turn into something a human can read. The job this month is inventory: find every system sending as you and make sure SPF knows about it.

Month two, start applying pressure to a slice of the traffic:

"v=DMARC1; p=quarantine; pct=10; rua=mailto:dmarc@yourapp.com"

Ten percent of failures go to spam. Read the reports, close the gaps, raise pct toward 100.

Month three, commit:

"v=DMARC1; p=reject; rua=mailto:dmarc@yourapp.com"

Failures are discarded and impersonating your domain stops being practical.

BIMI: Paying for a Logo

BIMI puts your brand mark beside your messages in Gmail, Yahoo, Apple Mail and Fastmail. The entry fee is DMARC already at quarantine or reject with pct=100, plus a certificate:

A Verified Mark Certificate runs $780 to $1,668 a year and requires a registered trademark, and it is the only route to the blue checkmark in Gmail. A Common Mark Certificate costs around $650 a year, skips the trademark requirement, and Google began honoring them in 2026, though without the checkmark. Your logo has to be SVG conforming to the Tiny PS profile.

default._bimi.yourapp.com.  IN  TXT  "v=BIMI1; l=https://yourapp.com/logo.svg; a=https://yourapp.com/vmc.pem"

Consumer brand with recognition worth reinforcing? Probably worth it. B2B tool whose users read mail in a client that ignores BIMI anyway? Spend the money on getting to p=reject instead.

MTA-STS: HSTS for Mail

MTA-STS instructs sending servers that TLS is mandatory when delivering to you, closing the downgrade attack where someone strips encryption in transit and reads plaintext. It needs a hosted policy file plus a DNS pointer.

The file lives at https://mta-sts.yourapp.com/.well-known/mta-sts.txt:

version: STSv1
mode: testing
mx: mail.yourapp.com
max_age: 86400

Begin in testing. Move to enforce after reports show nothing legitimate is being refused.

_mta-sts.yourapp.com.  IN  TXT  "v=STSv1; id=20260601T000000"

Bump that id every time you edit the policy file, otherwise senders keep serving themselves a cached copy. Add TLS-RPT alongside it so failures reach you:

_smtp._tls.yourapp.com.  IN  TXT  "v=TLSRPTv1; rua=mailto:tls-reports@yourapp.com"

Confirming It Actually Works

Mail a Gmail address, open Show Original, and look for three passes on one line:

spf=pass ... dkim=pass ... dmarc=pass

Anything reading fail points at exactly which record to revisit. Our Email Header Analyzer pulls those headers apart and surfaces the pass and fail states along with routing hops and delays, which beats squinting at raw text.

Then confirm the records are actually live rather than just saved in a control panel:

dig txt yourapp.com +short                      # SPF
dig txt _dmarc.yourapp.com +short               # DMARC
dig txt resend._domainkey.yourapp.com +short    # DKIM

Five Failures You Will Probably Hit

An SPF PermError means you crossed 10 lookups. Flatten or prune.

A DKIM mismatch means your provider rotated keys and DNS did not follow.

DMARC failing while SPF passes is an alignment problem: the provider is bouncing from something like bounces.yourapp.com while your From says yourapp.com. Relax alignment with aspf=r or line the subdomains up.

No BIMI logo in Gmail means no VMC or CMC. Yahoo and Apple Mail are more forgiving; Gmail is not.

An MTA-STS 404 means the policy URL is not reachable. Check the subdomain resolves and the path returns 200.

Everything In One Block

# SPF
yourapp.com.  IN  TXT  "v=spf1 include:_spf.resend.com -all"

# DKIM
resend._domainkey.yourapp.com.  IN  CNAME  resend._domainkey.resend.com.

# DMARC (begin here, escalate later)
_dmarc.yourapp.com.  IN  TXT  "v=DMARC1; p=none; rua=mailto:dmarc@yourapp.com; adkim=r; aspf=r"

# BIMI (optional)
default._bimi.yourapp.com.  IN  TXT  "v=BIMI1; l=https://yourapp.com/logo.svg; a=https://yourapp.com/vmc.pem"

# MTA-STS
_mta-sts.yourapp.com.  IN  TXT  "v=STSv1; id=20260601T000000"

# TLS-RPT
_smtp._tls.yourapp.com.  IN  TXT  "v=TLSRPTv1; rua=mailto:tls-reports@yourapp.com"

The Order To Do This In

Publish SPF first, since it is a single record and it directly answers the Microsoft requirement. Add DKIM next by following your provider's guide, typically two or three records. Then put DMARC up at p=none and leave it alone for four weeks while the reports tell you which forgotten system is sending as you. Tighten to quarantine, walk pct up to 100, then reject, spread over roughly six to eight weeks. MTA-STS goes on once that is stable. BIMI goes last, after reject.

Total hands-on work is a few hours. The calendar time is almost entirely waiting for DMARC reports to tell you what you did not know you were running. The longer guide covers the troubleshooting cases in more depth if something refuses to pass.

References

LLM Guard Is Archived. Here Are the Guardrail Tools Still Worth Installing

Moksh Gupta — Wed, 12 Aug 2026 03:12:27 +0000

If you go looking for protectai/llm-guard right now, GitHub greets you with a read-only notice. The repo went into the archive on July 9, 2026, its last commit landed the day before, and the README says outright that neither the code nor the Hugging Face models are being maintained any more. It shipped 15 input scanners and 20 output scanners under MIT, collected 3,200 stars, and it is still sitting near the top of most "best LLM guardrails" roundups you will find today.

Worth checking before it goes into a requirements file. I dug into what replaced it and wrote the full comparison, with every license and price verified against the vendor's own page, over on DevToolLab. This is the condensed version.

Follow the Ownership and the Archive Makes Sense

Protect AI maintained LLM Guard, and Palo Alto Networks finished buying Protect AI on July 22, 2025. Look either side of that date and a pattern shows up fast. Check Point picked up Lakera on September 16, 2025, in a deal reported at roughly $300 million. Palo Alto said it intended to buy Portkey on April 30, 2026. OpenAI said it was buying promptfoo on March 9, 2026.

Inside about a year and a half, nearly every independent vendor in this space turned into a feature of somebody's larger security platform. LLM Guard's capabilities did not vanish, they moved into Palo Alto's commercial AI security module, and the free library stopped getting commits.

Three Jobs Wearing One Word

"Guardrails" gets applied to three separate problems, and picking a tool built for the wrong one is where most teams waste a sprint.

Input rails sit in front of the model and screen what arrives: injection attempts, jailbreaks, off-limits topics, personal data you do not want leaving your network. This is the adversarial surface, because attackers get to choose this text.

Output rails screen what comes back: toxic content, leaked personal data, claims that do not match your retrieved sources, malformed structure. Different failure mode entirely, since here the model is the thing misbehaving.

Red teaming is not a filter at all. It is an adversarial test suite you run in CI before shipping. Treating it as a runtime control is how you end up with a clean security report and nothing actually guarding production.

Expect to need all three. Expect them to be three separate tools.

Why This Bug Class Stays Open

The 2025 edition of the OWASP Top 10 for LLM Applications lists prompt injection at LLM01, its second consecutive run at the top spot.

There is a structural reason it does not get patched away. Instructions and data arrive through the same channel with nothing marking which is which, so content can be read as a command. SQL solved this with bound parameters. There is no equivalent primitive for a prompt.

EchoLeak is the case study worth reading. Tracked as CVE-2025-32711 at CVSS 9.3, found by Aim Security in June 2025, it is documented as the first zero-click prompt injection actually exploited against a shipping LLM product.

The delivery was an email that looked like any other, carrying instructions tucked into an HTML comment or set in white text on white. No click required. When the recipient later asked Microsoft 365 Copilot about something entirely different, retrieval pulled that message into context and the buried text got treated as direction. Getting there meant stacking bypasses: past Microsoft's cross prompt injection classifier, around link redaction using reference-style Markdown, through auto-fetched images, out via a Teams proxy the content security policy already trusted. Microsoft fixed it server-side and saw no exploitation in the wild.

Note where the payload entered. Not the chat box. A classifier watching user input would never have seen it.

The Constraint That Beats Any Classifier

Simon Willison named the useful mental model on June 16, 2025: the lethal trifecta. Private data access, untrusted content exposure, external communication. Any two of those together is survivable. All three in one session means whoever controls the untrusted content can read your private data and send it somewhere, without writing a single line of exploit code.

The reason this beats a detector is that it is deterministic. You can guarantee an architecture; you cannot guarantee a probability. Cut one leg and you have removed the capability rather than lowered the odds, at zero runtime cost. Everything below belongs behind that decision, not instead of it.

What Is Still Being Maintained

Five options here are genuinely open source and run on hardware you control.

NeMo Guardrails sits at v0.23.0 as of July 1, 2026, Apache 2.0, around 6,900 stars, commits still landing. Heads up on the URL: NVIDIA/NeMo-Guardrails now redirects to NVIDIA-NeMo/Guardrails. It is the most opinionated design of the bunch, splitting enforcement across input, dialog, retrieval, execution and output stages, with policies expressed in a DSL called Colang. Those retrieval rails are the correct place to screen RAG chunks, which is exactly the gap EchoLeak drove through. Costs: Colang is a language you have to learn, and some rail types spend an LLM call per request. If a roundup quotes you a "sub-100ms GPU-accelerated" number for it, be skeptical, since the repo ships no latency benchmarks at all.

Guardrails AI reached v0.10.2 on June 4, 2026, Apache 2.0, roughly 7,300 stars. The unit here is a validator, one per risk, which you compose into guards around a call. Good fit for field-level output validation, and easy to introduce one validator at a time. Just read each validator before you deploy it, because several reach out to a model rather than computing locally.

Presidio is at 2.2.364 as of July 22, 2026, MIT, about 10,400 stars, and it only does PII: find it, mask it, anonymize it, across text, images and structured data, entirely offline.

This one relocated too, and it went the opposite direction from everything else in this article. microsoft/presidio now 301s to data-privacy-stack/presidio, because the project is becoming community-owned under a vendor-neutral org with Microsoft's backing. The license stays MIT and the copyright line now credits Presidio Contributors. The change that will actually break your build: images live at ghcr.io/data-privacy-stack/presidio-* now, so go grep for hardcoded mcr.microsoft.com pulls. While the paid end of this market got swallowed by security giants, the best open PII tool escaped one.

Llama Prompt Guard 2 is the classifier you host yourself, labeling prompts benign or malicious across both injection and jailbreak attempts. The 86M build sits on mDeBERTa-base, the 22M build on DeBERTa-xsmall for about 75 percent less latency and compute. Get the licensing right: those base models are MIT, but Meta ships the Prompt Guard weights themselves under the Llama license behind a gated download, which is not the same thing. What I like is the honesty of the model card, which reports .998 AUC on English and 97.5 percent recall at a 1 percent false positive rate, then admits the real-world attack prevention rate is 81.2 percent at 3 percent utility loss. That gap is the most useful number in the document. It also states its limits plainly: 512-token window, no multilingual pretraining in the 22M variant, and an acknowledgment that people will build attacks specifically to defeat it.

promptfoo covers red teaming, MIT, about 24,000 stars, commits landing daily. Its red team mode manufactures adversarial prompts across injection, jailbreaks, PII leakage, SSRF, SQL injection, excessive agency and hallucination, then tells you what slipped past. Put it in CI. The OpenAI acquisition means the roadmap is no longer independent, though MIT and a public codebase keep the downside contained.

The Managed Options, With Real Numbers

Bedrock Guardrails is refreshingly easy to price because AWS publishes each policy separately, and you only pay for what you switch on. A text unit tops out at 1,000 characters.

Content filters run $0.15 per 1,000 text units for text and $0.00075 per image. Denied topics are $0.15 per 1,000 units. Sensitive information filters are $0.10 per 1,000 units, or free if you express them as regex. Word filters are free. Contextual grounding checks are $0.10 per 1,000 units. Automated Reasoning checks are $0.17 per 1,000 units per policy.

Two things jump out. The single cheapest useful control in this whole category is a deny list you write by hand, because word filters and regex PII filters bill nothing. And grounding checks, the direct fix for a RAG pipeline inventing facts, undercut the content filters.

Do the arithmetic before flipping switches. One million requests a month, an 800-character prompt and a 1,600-character response, works out to three text units per request. Filter both directions and you are billing three million units, so $450. Add denied topics on input only, $150. Add PII filtering on output, $200. You are at $800 a month, which is either budgeted or a surprise depending on whether anyone ran this multiplication.

Azure prices differently, bundling text moderation, Prompt Shields, groundedness and protected material detection into one rate.

The free allowance is 5,000 text records and 5,000 images monthly, and it halts rather than silently billing you. Standard is $0.38 per 1,000 text records and $0.75 per 1,000 images, checked in Central US with USD selected. A record covers up to 1,000 Unicode code points.

Which is cheaper genuinely depends on your policy count, since Azure's $0.38 already includes what Bedrock charges $0.15 plus $0.10 plus $0.10 for. One or two policies favor Bedrock. All of them can favor Azure. Rates vary by region, so check your own.

On the commercial side, Lakera Guard remains the best-known dedicated prompt firewall, now inside Check Point following the September 2025 deal, so ask how it is packaged before assuming old pricing survived. Protect AI's Guardian and Recon plus its red teaming layer now live in Palo Alto's Prisma AIRS, which is where LLM Guard's functionality went commercially.

Side by Side

Tool	License	Layer	Self-hosted	Status
NeMo Guardrails	Apache 2.0	Input, dialog, retrieval, output	Yes	Active, v0.23.0
Guardrails AI	Apache 2.0	Input and output validators	Yes	Active, v0.10.2
Presidio	MIT	PII detection and redaction	Yes	Active, 2.2.364
Llama Prompt Guard 2	Llama license, MIT base	Injection and jailbreak classifier	Yes	Active
promptfoo	MIT	Red teaming in CI	Yes	Active, OpenAI acquiring
Bedrock Guardrails	Commercial	Input and output	No	From $0.10 per 1k text units
Azure Prompt Shields	Commercial	Input and output	No	$0.38 per 1k text records
Lakera Guard	Commercial	Runtime prompt firewall	No	Check Point, post-acquisition
LLM Guard	MIT	Input and output scanners	Yes	Archived July 2026

Defaults Will Corrupt Your Redactions

PII stripping has the least ambiguous spec of any guardrail, so it is the one I actually measured rather than trusted. Everyone's first version is a pile of regular expressions. Here is Presidio 2.2.364 with spaCy 3.8.15 on a synthetic support ticket, straight out of the box:

from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.nlp_engine import NlpEngineProvider
from presidio_anonymizer import AnonymizerEngine

TICKET = (
    "Hi, this is Marcus Delgado from Cleveland. My order never arrived. "
    "You can reach me at marcus.delgado@northgate-supply.com or (216) 555-0142. "
    "I paid with card 4111 1111 1111 1111 and my SSN on file is 401-55-9302. "
    "The request came from 198.51.100.24 if that helps."
)

provider = NlpEngineProvider(nlp_configuration={
    "nlp_engine_name": "spacy",
    "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}],
})
analyzer = AnalyzerEngine(nlp_engine=provider.create_engine(), supported_languages=["en"])
anonymizer = AnonymizerEngine()

results = analyzer.analyze(text=TICKET, language="en")
print(anonymizer.anonymize(text=TICKET, analyzer_results=results).text)

A regex pass catches the five structured identifiers: email, phone, SSN, card, IP. Presidio catches all five plus PERSON for "Marcus Delgado" and LOCATION for "Cleveland", which is the whole reason to run a real engine, because no pattern will ever recognize a name.

It also returns things you never asked for. marcus.de came back as a URL at 0.5 confidence, since .de is a live TLD hiding inside the email address. The digits 1111 were classified DATE_TIME at 0.85. The literal token "SSN" was tagged ORGANIZATION at 0.85. That last one is not cosmetic, because it wrecks the output:

Hi, this is <PERSON> from <LOCATION>. My order never arrived. You can reach me
at <EMAIL_ADDRESS> or <PHONE_NUMBER>. I paid with card <CREDIT_CARD> and my
<ORGANIZATION> on file is <US_SSN>. The request came from <IP_ADDRESS> if that helps.

"My <ORGANIZATION> on file" is a bug your customers would read. Two arguments fix it, by naming the entities you actually want and setting a confidence floor:

ENTITIES = [
    "PERSON", "LOCATION", "EMAIL_ADDRESS",
    "PHONE_NUMBER", "CREDIT_CARD", "US_SSN", "IP_ADDRESS",
]

results = analyzer.analyze(
    text=TICKET, language="en", entities=ENTITIES, score_threshold=0.4,
)

Seven findings, all correct, clean text. Two caveats: I ran en_core_web_sm to keep the download small when Presidio really wants en_core_web_lg, and PHONE_NUMBER scored exactly 0.4, right on the line, which is your cue to tune that floor on your own data instead of copying mine. The full write-up has both runs side by side with every score.

The transferable lesson: any guardrail left on defaults with no threshold will produce false positives, and in a redaction path a false positive is a shipped bug, not a security win.

Picking One

Conversational app with topic rules, go NeMo Guardrails and use the retrieval rails. Field-level output validation, Guardrails AI, one validator at a time. Hard requirement that personal data never leaves your network, self-hosted Presidio with an explicit entity list. Want your own injection classifier, Llama Prompt Guard 2 at whichever size fits your latency budget, with the license reviewed by whoever reviews licenses. Already on Bedrock, turn on the free word and regex filters today and add paid policies only after measuring. No adversarial tests at all, start with promptfoo in CI, because knowing which attacks land beats a filter you cannot evaluate. And if LLM Guard is already in your requirements file, Guardrails AI plus Presidio covers most of what you were using it for.

A Rollout Order That Works

Start by auditing for the lethal trifecta, since removing one leg outperforms every tool listed here and costs nothing at runtime. Then write down every route text takes into your context window, not just the prompt field, remembering that EchoLeak came in through retrieved mail.

Normalize before inspecting, because payloads hide inside zero-width characters and get base64-wrapped to slip past literal matching. The Invisible Character Remover strips the hidden ones and the Base64 Encoder Decoder shows you what the model will really see. Test every deny-list pattern against adversarial input in a Regex Tester before it touches a request path, because a bad pattern either blocks real users or waves everything through.

After that, add PII redaction with an explicit entity list rather than defaults, get promptfoo running in CI so a system-prompt regression breaks a build, and log every block alongside the input that caused it. A guardrail you cannot audit is a guardrail you cannot tune.

Where This Leaves You

Eighteen months of consolidation: Lakera to Check Point, Protect AI and Portkey to Palo Alto, promptfoo to OpenAI, LLM Guard to the archive. Check a repo's own status page before you trust any comparison article, this one included.

The open layer is still in good shape. Two Apache 2.0 frameworks under active development, an MIT PII engine that just gained community ownership, and a self-hosted classifier whose own documentation publishes the distance between its benchmark score and its real-world hit rate.

None of it closes prompt injection, because instructions and data still share a channel. Presidio broke its own output on default settings in the test above, and Meta's numbers put roughly a fifth of genuine attacks through a well-trained classifier. Spend your first hour on architecture and on which leg of the trifecta you can amputate. Treat the rest as depth behind that call.

References

Faker, SDV, MOSTLY AI or NeMo: Picking a Synthetic Data Tool in 2026

Moksh Gupta — Mon, 10 Aug 2026 15:42:34 +0000

Search for gretel.ai today and you get redirected to nvidia.com. NVIDIA acquired Gretel in March 2025 for reportedly north of its $320 million valuation and relaunched it as NeMo Data Designer under Apache 2.0. That single redirect captures the last eighteen months in this space pretty well: the venture-backed vendors got bought or repriced, and the open-source tooling got noticeably better.

The change that actually affects a decision you're making this week is quieter. SDV, the most widely cited open-source synthesizer, left the MIT license behind for the Business Source License. I wrote up the full comparison, with every license and price checked against the vendor's own page in August 2026, on DevToolLab - this is the condensed version, plus the local test I'd run before trusting any of these tools.

Three different jobs share one name

"Synthetic data" gets used for three distinct problems, and most of the bad tool picks in this space come from solving for one while actually needing another.

Mock data fabricates rows with no connection to anything real - names, emails, cities. Good for seed scripts and test fixtures. Statistical synthesis trains on your actual table and generates new rows that preserve the distributions and correlations between columns, which matters the moment something downstream learns from the output. De-identification takes real production data and scrubs it so individuals can't be re-identified while keeping row-level structure - a compliance problem, not a data-generation one.

The gap between the first two is the expensive one, and it's what the test near the end of this post is built to expose.

The open-source landscape, quickly

Faker is MIT, ubiquitous, and exactly as good as its job description: independent, plausible-looking fields. Seed it and you get byte-identical output across runs, which is genuinely useful for CI fixtures:

from faker import Faker

fake = Faker("en_US")
Faker.seed(42)

for _ in range(3):
    print(f"{fake.name()},{fake.email()},{fake.city()}")

One gotcha worth knowing: the seed locks the entire random stream, not each field separately. Add one fake.* call anywhere earlier in the script and every value after it shifts. Pin the script alongside the seed.

SDV (Synthetic Data Vault) is still the deepest open synthesizer - relational tables with foreign keys, sequential data, built-in quality evaluation - but it's Business Source License now, not MIT, the same license family Terraform and Vault moved to. My infrastructure as code piece ran into the identical question. BUSL allows plenty, but check the terms yourself rather than trusting a roundup that still calls it "MIT licensed" - plenty do.

MOSTLY AI's SDK is Apache 2.0 and local-first by design: training and generation run on your own machine by default, no cloud account required. If SDV's license is a blocker, this is the first thing to try.

NVIDIA NeMo Data Designer - what Gretel became - targets generating training data for models rather than cloning a production table. Dependency-aware fields, Python/SQL validators, LLM-as-judge scoring on the output, and a preview mode before you generate at scale. Apache 2.0.

The commercial tier, in one sentence each

Tonic.ai splits into three products worth telling apart: Fabricate (mock generation, the only one with public self-serve pricing - free tier, then $29/month), Structural (de-identification against real databases, quote-based), and Textual (unstructured redaction, billed per word count). The pattern holds across the category - K2view, Syntho, YData are all quote-based enterprise deals. If you need a signed DPA and a live database connector, you're in procurement, not a credit card checkout.

The test that actually separates these tools

Averages lie here. Generate a real dataset with a deliberate relationship - salary rising with years of experience - then compare it against columns generated independently versus columns run through an actual synthesizer (SDV's GaussianCopulaSynthesizer, in my test):

real data              corr=+0.988   mean_salary=$119,977   mean_years=20.3
independent columns    corr=-0.061   mean_salary=$123,101   mean_years=20.4
SDV synthetic          corr=+0.880   mean_salary=$118,319   mean_years=21.5

Every mean is within a few percent across all three. Any "does this look reasonable" check passes all of them. But the correlation between experience and pay goes from +0.988 to essentially zero once the columns are generated independently - the relationship isn't weakened, it's erased. A model trained on that data learns that experience has no bearing on pay. SDV holds the correlation at +0.880, and crucially, zero of its generated rows matched a real row in the source table.

Run this on your own data, with your own columns, before picking a tool based on a vendor's demo.

Picking one

Dev database or test fixtures: Faker, seeded, nothing else needed. Training or evaluating on data you can't touch directly: MOSTLY AI if you need a permissive license, SDV if you want the deeper feature set and BUSL works for you. Generating eval or instruction datasets for LLMs: NeMo Data Designer with preview mode and judge scoring turned on. Real production database plus compliance sign-off: a commercial platform on a quote, budgeted as procurement.

For quick one-off rows without installing anything, DevToolLab's Fake Data Generator runs in the browser, and the SQL Insert Generator turns the output into statements you can paste into a seed script.

References

OpenAI, Voyage, Cohere or Open Weights: Picking an Embedding Model in 2026

Moksh Gupta — Sun, 09 Aug 2026 12:23:25 +0000

An embedding model does one thing: it turns text into a list of numbers so that similar meaning ends up close together in that number space. Every product in this post is a variation on that idea, and 2026 rearranged the lineup more than any year since embeddings went mainstream. Voyage AI shipped a 4-series after MongoDB's acquisition closed, Google shipped both a multimodal Gemini Embedding 2 and a 308-million-parameter on-device model, and open weights stopped being the budget option and started beating the paid APIs on public benchmarks.

I wrote a longer version of this comparison on DevToolLab, Best Embedding Models and APIs in 2026, with the full pricing table and every source linked. Here is the short version, including the one experiment worth running yourself before you trust any of it.

Two Shapes, Not Ten Products

Every option here is one of two things: a hosted API you call over HTTP and pay per token, or open weights you download and run yourself for free. Three properties separate the options inside each shape: dimension count (which drives your storage and search cost), context window (how much text one call can see), and whether Matryoshka Representation Learning is baked in.

A Matryoshka-trained model packs the most useful signal into the first dimensions of its output, so truncating a 3,072-dimension vector down to 256 keeps most of the retrieval quality while cutting storage 12x. OpenAI's text-embedding-3 models, Voyage's 4-series, and Qwen3-Embedding all do this natively through a dimensions parameter. A model that was not trained this way degrades faster when you truncate it, which the experiment near the end of this post shows directly.

The Hosted APIs

OpenAI's text-embedding-3-small and text-embedding-3-large have not moved in price since January 2024: $0.02 and $0.13 per million tokens, batch pricing cutting both in half. It remains the safe default: cheapest at the small tier, an 8,191-token context that covers most chunking strategies, and the widest library ecosystem.

Voyage AI became part of MongoDB in a $220 million deal in February 2025 and kept shipping independently. voyage-4-large is $0.12 per million tokens with a 32,000-token context, four times OpenAI's window, and every model in the 4-series shares the same Matryoshka range (1,024 default, up to 2,048), so the cheaper voyage-4-lite is a price cut rather than a smaller vector.

Cohere's Embed v4 is the long-context, multimodal specialist: 128,000 tokens of context against OpenAI's 8,191, at $0.12 per million text tokens and $0.47 per million image tokens, handling text, images, and mixed documents like PDFs in one model. Gemini Embedding 2 goes further still, natively embedding text, image, audio, video, and PDF into the same vector space, priced per modality from $0.20 per million text tokens up to $12.00 per million video tokens. Text-only it is the most expensive API here; its case is that nothing else lets you compare a video frame against a text query at all.

Open Weights Stopped Being the Compromise

Qwen3-Embedding is genuinely Apache 2.0, not the more restrictive license some aggregator sites report, and its model card states it ranked first on the MTEB multilingual leaderboard as of June 2025. It ships in 0.6B, 4B, and 8B sizes with a 32,000-token context and Matryoshka output from 32 to 4,096 dimensions; the 0.6B variant runs on CPU and is still competitive, which makes it the model to try first if you want to self-host.

BGE-M3, MIT licensed, does something none of the hosted APIs do: dense, sparse, and multi-vector ColBERT-style output from one pass, so hybrid semantic-plus-keyword search does not need two separate models. Most self-hosted production RAG stacks default to it for exactly that reason.

The model most people miss is Google's EmbeddingGemma, released in September 2025, built on Gemma 3, and aimed at a use case none of the above cover: running entirely offline on a phone or laptop with no API call and no GPU. Google's own numbers: under 200MB of RAM with quantization, embeddings generated in under 22 milliseconds on an EdgeTPU, a 2,048-token context, and Matryoshka output from 768 down to 128. The one thing to get right is the license: it ships under Google's Gemma Terms of Use, not Apache or MIT, so it permits commercial use but is not the drop-in dependency the other three are. Nomic Embed v2 sits next to it as the more permissively licensed alternative at a similar size, trading context length (512 tokens) for Apache 2.0 with no usage restrictions at all.

What Cosine Similarity Actually Measures, Run Locally

Every model above outputs vectors compared the same way. Here is the whole thing, run with sentence-transformers on a laptop CPU, no API key required:

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
sentences = [
    "The cat sat on the mat.",
    "A feline rested on the rug.",
    "The stock market fell sharply today.",
]
emb = model.encode(sentences, normalize_embeddings=True)

def cos(a, b):
    return float(np.dot(a, b))

print(f"sim(cat/feline)  = {cos(emb[0], emb[1]):.4f}")
print(f"sim(cat/stock)   = {cos(emb[0], emb[2]):.4f}")

sim(cat/feline)  = 0.5560
sim(cat/stock)   = 0.0747

Two sentences about a cat sitting somewhere score seven times higher on similarity than one about a cat and one about the stock market, despite sharing no words beyond "The". That gap over a plain keyword match is the entire value of everything in this post; a tool like DevToolLab's Word Frequency Counter would find these two cat sentences almost entirely dissimilar by word overlap, while the embedding correctly reads them as close in meaning.

Truncating that same 384-dimension output down to 128 and 64 dimensions and re-normalizing shows what an untrained-for-Matryoshka model does under pressure: cat/feline stays clearly higher than cat/stock at every size, but the gap narrows fast, from 0.556 versus 0.075 at full size to 0.601 versus 0.229 at 64 dimensions. A model actually trained for Matryoshka degrades far more gracefully, which is the entire point of training for it rather than truncating an arbitrary model and hoping. The full writeup has the complete truncation table and the reasoning behind every pricing figure in this post.

How to Pick, Briefly

No strong constraints: text-embedding-3-small. Need more than 8,191 tokens of context: Cohere Embed v4 at 128,000, or Voyage at 32,000 for less money. Corpus includes real images or video: Cohere for text-plus-image, Gemini Embedding 2 if audio or video need to share the same vector space. Self-hosting with the fewest restrictions: Qwen3-Embedding. Hybrid dense-plus-keyword from one model: BGE-M3. Running on-device with no server at all: EmbeddingGemma, with Nomic Embed v2 as the fallback if the Gemma license is a problem.

References

Best Embedding Models and APIs in 2026 - the original, with the full comparison table and every price verified
OpenAI embeddings guide, Voyage AI pricing, Cohere Embed v4 docs
Gemini API pricing, EmbeddingGemma overview
Qwen3-Embedding-8B model card, BGE-M3 model card
Best Vector Databases in 2026 - where these embeddings actually get stored and searched

Terraform Is an IBM Product With a Meter Now. Here Is What the IaC Landscape Looks Like

Moksh Gupta — Sun, 09 Aug 2026 07:03:17 +0000

Try opening a state file written by OpenTofu with encryption turned on using Terraform 1.5.7. You get Unsupported state file format. That single error is a better description of where infrastructure as code sits in 2026 than any licensing debate, because the fork is no longer about which license you prefer. It is about a file one of your tools cannot read.

Two things happened that most teams only noticed on an invoice. HashiCorp archived the Cloud Development Kit for Terraform on 10 December 2025, read-only, with the recommendation to go back to HCL. Then on 31 March 2026 the legacy HCP Terraform free plan reached end of life and every organization still on it landed on resource metering, in a migration you cannot reverse.

The longer version of this comparison lives on DevToolLab, with every price sourced from the vendor's own page and the full ten-tool table. This is the short version.

Stop Treating It as One Choice

Three decisions hide inside "which IaC tool", and they are independent. The engine parses your config and calls cloud APIs. The runner executes it with credentials and leaves an audit trail. The state layer holds the mapping from config to real resource IDs, and decides whether that mapping is encrypted.

Most cost surprises come from conflating the first and third. You can run OpenTofu as the engine, GitHub Actions as the runner and an encrypted bucket as state, and pay nobody anything.

Terraform: Still the Default, Still Shipping

IBM closed its $6.4 billion HashiCorp acquisition on 27 February 2025, and Terraform has been under the Business Source License 1.1 since August 2023, with 1.5.7 the last MPL release. Feature work has not stalled. Version 1.14 introduced list resources in .tfquery.hcl files with a terraform query command that discovers unmanaged resources and writes import blocks for them. Version 1.15 arrived on 29 April 2026 with dynamic module sources and a deprecated marker for variables and outputs, and Stacks went generally available as a terraform stacks subcommand. Current stable is 1.15.8.

The part worth your attention if you use coding agents is the official Terraform MCP server, generally available since 13 June 2026 and open source. It exposes the registry and your workspaces over Model Context Protocol, which means an agent can read the real inputs of your private modules instead of guessing at resource arguments.

OpenTofu Went Somewhere Different

OpenTofu forked from Terraform 1.5, kept MPL 2.0, lives under the Linux Foundation and became a CNCF sandbox project on 23 April 2025. Harness, Gruntwork, Spacelift, env0 and Scalr committed nineteen full-time engineers for five years, which is why the pace held up rather than trailing off after the initial news cycle.

Early variable and locals evaluation shipped in 1.8, roughly twenty months ahead of Terraform's dynamic module sources. Version 1.11 brought ephemeral values that never touch state or plan files and an enabled meta-argument that retires the count = var.x ? 1 : 0 trick. Version 1.12.0, on 14 May 2026, added dynamic prevent_destroy, import by resource identity and destroy = false for removing a resource from state while leaving the real object alone.

State encryption is the feature with no counterpart. Configure it through an environment variable, so the passphrase stays out of your repository:

export TF_ENCRYPTION='key_provider "pbkdf2" "demo" {
  passphrase = "correct-horse-battery-staple-42"
}
method "aes_gcm" "secure" {
  keys = key_provider.pbkdf2.demo
}
state { method = method.aes_gcm.secure }'
tofu init && tofu apply -auto-approve

Applied against a config containing a random_password resource on OpenTofu 1.12.5, the resulting state file gives up nothing:

$ grep -c 'random_password' terraform.tfstate
0

The readable part of the file is a PBKDF2 header at 600,000 iterations. Everything else is ciphertext. Change the passphrase and the next command reports decryption failed for all provided methods.

Pulumi Bet on Agents

Pulumi is the choice for teams who want TypeScript, Python, Go or C# instead of HCL, and its 2026 pitch is agentic. Co-founder Joe Duffy said in May that language models now carry out more than 20 percent of deployments on the platform, up from effectively zero twelve months earlier, and expects that past 50 percent within the year. The same release added a Neo CLI, GitHub and Slack apps, scheduled agent tasks that arrive as pull requests, and a pulumi do command.

Billing is per resource per month: free for one user, $40 a month on Team covering up to 500 resources and then $0.1825 each, $400 a month on Enterprise. Neo tokens are $3 per million. The engine and CLI are Apache 2.0, so self-hosted state without Pulumi Cloud is a legitimate path.

Crossplane for Platform Teams

Crossplane reframes the problem: cloud resources become Kubernetes objects a controller reconciles continuously, so correcting drift is the default rather than a cron job. It graduated in CNCF on 28 October 2025, and v2 made composite and managed resources namespaced, dropped the claim indirection, and added Operations that run function pipelines for jobs like certificate rotation.

It earns its keep when other teams consume your platform through Kubernetes APIs. For one VPC and a handful of services, you have just adopted a control plane to operate.

Who Actually Holds Your State

This is the decision teams make by accident rather than on purpose. There are really only three options: a bucket you own with encryption turned on, a vendor's hosted state bundled into a platform plan, or a self-hosted backend like OpenTaco's that speaks the same protocol as the commercial ones so nothing downstream has to change if you move.

Two things are worth knowing before you pick. State holds credentials in plain text unless something encrypts it, which is the entire reason OpenTofu's TF_ENCRYPTION above is worth setting up on day one rather than after an incident. And encryption only goes one way between engines: once OpenTofu encrypts a state file, Terraform cannot read it, so decrypt before any trip back. If you want the fuller picture on where those credentials belong in the first place, DevToolLab's secrets management comparison covers Vault, OpenBao and the SaaS options this post does not.

The Free Runner Layer Is Good Now

Terragrunt reached 1.0 on 30 March 2026 with a backwards compatibility guarantee across CLI flags, HCL config and serialized output, which matters a lot for something wired into everyone's CI. MIT licensed, around 9,800 stars.

Atlantis is still the reference self-hosted pattern, posting plans to the pull request and applying on a comment so credentials never leave your infrastructure. Digger became OpenTaco in November 2025 and now ships a self-hosted state backend with access control, history and rollback behind an HCP Terraform compatible interface. An open source state backend speaking the commercial protocol is the sharpest answer yet to per-resource billing.

The Meters Are Not Comparable

HCP Terraform charges per managed resource per month: $0.10 on Essentials, $0.47 on Standard, $0.99 on Premium, free up to 500 resources with unlimited users and a surprisingly complete feature set including SSO and policy as code. Spacelift covers the most engines and keeps a two-user free tier, but its entry paid plan is now $20,000 a year on annual commitment. Scalr charges per run instead: 50 free monthly, then $0.99 each, with no per-user or per-resource fee. env0, trading as env zero, gives 250 runs and 30 environments free.

At 1,000 managed resources that works out to $100 a month on Essentials, $470 on Standard, roughly $131 on Pulumi Team, and $99 on Scalr at 100 runs regardless of resource count. Model your own estate before signing anything, because per-resource billing penalizes stable, well-factored infrastructure specifically. The full pricing table on DevToolLab lays all ten options side by side.

Migrating Takes About Fifteen Minutes

Back up state, install OpenTofu, then run tofu init in the unchanged directory. No edits are needed, because hashicorp/* provider addresses resolve through OpenTofu's registry. Run tofu plan and insist on an empty diff. On a project whose state was written by Terraform 1.5.7, OpenTofu reported found no differences, so no changes are needed. Enable state encryption before the next apply, swap the binary in CI, and audit any wrapper script that greps output, since the word "Terraform" becomes "OpenTofu". Plan the return trip too: encrypted state has to be decrypted first.

If you are picking today with nothing to migrate, OpenTofu plus GitHub Actions plus encrypted remote state is the sensible default. It is a superset of the Terraform you would otherwise learn, and there is no license review at the end of it. Two small things that help before you commit anything: the Subnet Calculator for working out CIDR ranges before they get hard-coded into a VPC module, and the Cron Expression Parser for sanity-checking a drift detection schedule that will run unattended.

References

Best Infrastructure as Code Tools in 2026 - the original, with the full comparison table and verified sources
Best Secrets Management Tools in 2026 - Vault, OpenBao and the SaaS options for the credentials your state file holds
Terraform documentation and the Terraform MCP server
OpenTofu, Pulumi pricing, Crossplane
Terragrunt, Atlantis, OpenTaco
Scalr pricing, Spacelift pricing, env zero pricing
GitHub Actions Best Practices - hardening the runner that applies all of this

A2A Hit 150 Organizations in a Year. Most Projects Reaching For It Do Not Need It Yet

Moksh Gupta — Fri, 07 Aug 2026 16:48:34 +0000

Agent2Agent turned one year old under the Linux Foundation in April 2026 with numbers that are genuinely impressive: more than 150 supporting organizations, up from roughly 50 twelve months earlier, over 22,000 stars on the core repository, and SDKs in Python, JavaScript, Java, Go and .NET. Azure AI Foundry, Amazon Bedrock AgentCore and Google Cloud's Agent Development Kit all ship it.

None of that tells you whether you should use it. "150 companies signed a support letter" and "engineers pick this over an HTTP call" are separate claims, and the gap between them is where most of the confusion about A2A lives. I wrote a fuller version of this on DevToolLab, A2A Protocol Explained, with the complete working example. Short version below.

Four Concepts, Then You Have It

MCP wires an agent to tools. A2A wires an agent to other agents, possibly on another vendor's stack, in another language, owned by a team you never talk to. The design goal is that agent A never learns how agent B is built, only what B can do and how to hand it work.

An Agent Card is a JSON document at /.well-known/agent-card.json listing skills, input and output modes, and auth requirements. A Task is a unit of work with a real lifecycle: submitted, working, sometimes input-required when it needs more from the caller, then completed, failed or canceled. A Message is one turn in the exchange. An Artifact is the output, built from typed parts that can be text, a file or structured data.

Transport is JSON-RPC 2.0 over HTTP with Server-Sent Events for streaming and push notifications for tasks that outlive one request. Version 0.3 in August 2025 added gRPC as an alternate transport and JWS-signed Agent Cards, which matters more than it first sounds: without signing, an Agent Card is an unauthenticated JSON file at a predictable URL.

It Is Not Competing With MCP

The framing that survives contact with real systems: MCP is the tool-integration layer, A2A is the agent-collaboration layer. MCP answers "give my agent a function to call." A2A answers "hand this task to an agent I do not control and get a result back." Non-trivial multi-agent systems need both.

The differences that matter in practice are the unit of work and who owns the other side. MCP moves a tool call; A2A moves a stateful task that can span turns. MCP servers are usually yours; the agent behind an Agent Card usually is not.

When You Genuinely Need It

The criticism doing the rounds this year is fair. Plenty of A2A demos show three agents accomplishing what three function calls would. Standing up an Agent Card, a task store and JSON-RPC plumbing for one in-process call is pure overhead.

It earns its keep when the other agent is actually outside your control, when the interaction is a real task rather than a function call (multiple turns, mid-flight requests for more input, long enough that you want streaming instead of a blocking response), or when you need to discover capabilities at runtime instead of hardcoding them. Skip it when a direct API call, a shared MCP server or in-process orchestration already works. And read "150 organizations" as an interoperability signal, not as proof of production load. The number that would prove that is how many teams keep it after their first authentication failure and compliance review, which is far smaller.

A Working Agent, Start to Finish

The reference SDK is a2a-sdk, compatible with protocol versions 1.0 and 0.3:

pip install "a2a-sdk[http-server]"

Describe the capability, then publish it in a card:

from a2a.types import AgentCard, AgentCapabilities, AgentSkill

skill = AgentSkill(
    id="summarize_ticket",
    name="Summarize Support Ticket",
    description="Reads a support ticket thread and returns a one-paragraph summary.",
    input_modes=["text/plain"],
    output_modes=["text/plain"],
    tags=["support", "summarization"],
    examples=["Summarize ticket #4021"],
)

agent_card = AgentCard(
    name="Ticket Summarizer Agent",
    description="Summarizes support tickets on request.",
    url="http://localhost:9999",
    version="1.0.0",
    default_input_modes=["text/plain"],
    default_output_modes=["text/plain"],
    capabilities=AgentCapabilities(streaming=True),
    skills=[skill],
)

The executor is where the work happens. Note the status update before the slow part, which is what gives a caller something to stream:

from a2a.server.agent_execution import AgentExecutor, RequestContext
from a2a.server.events import EventQueue
from a2a.server.tasks import TaskUpdater
from a2a.types import TaskState
from a2a.utils import new_task_from_user_message, new_text_message, new_text_part


class TicketSummarizerExecutor(AgentExecutor):
    async def execute(self, context: RequestContext, event_queue: EventQueue) -> None:
        task = context.current_task or new_task_from_user_message(context.message)
        updater = TaskUpdater(event_queue, task_id=task.id, context_id=task.context_id)

        await updater.update_status(
            state=TaskState.working,
            message=new_text_message("Reading ticket thread..."),
        )

        summary = summarize(context.message)  # your own logic or LLM call

        await updater.add_artifact(parts=[new_text_part(text=summary)])
        await updater.update_status(state=TaskState.completed)

    async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None:
        raise NotImplementedError("This agent doesn't support cancellation yet.")

Then serve it:

from a2a.server.apps import A2AStarletteApplication
from a2a.server.request_handlers import DefaultRequestHandler
from a2a.server.tasks import InMemoryTaskStore
import uvicorn

handler = DefaultRequestHandler(
    agent_executor=TicketSummarizerExecutor(),
    task_store=InMemoryTaskStore(),
)
app = A2AStarletteApplication(agent_card=agent_card, http_handler=handler).build()

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=9999)

Before pointing a client at it, confirm the card is actually being served:

curl -s http://localhost:9999/.well-known/agent-card.json

A malformed Agent Card is a common reason a client silently discovers no skills, so it is worth running that response through a JSON Schema validator while you are iterating. The cURL command generator is also handy for the task-submission call, since hand-typing a JSON-RPC envelope every time you change one parameter gets old fast. The original post walks through each piece in more detail.

The Security Gap Nobody Demos

Every serious writeup lands on the same hole: an Agent Card says what an agent can do, not who owns it or what it should be trusted with. Four things worth doing on day one rather than retrofitting.

Verify signed cards, because JWS signing in v0.3 exists precisely so you can confirm a card was not swapped in transit, and HTTPS alone does not give you that. Scope what a remote agent may request instead of trusting every skill its card advertises, the same way you would scope an OAuth grant. Decode the tokens moving between agents rather than assuming an aud or sub claim means what you expect. And never forward a caller's token to a third-party agent unmodified: if agent A hands agent B its own credential, B now acts with A's authority on systems it was never meant to reach. That is the confused-deputy problem, and it does not stop being one because both parties are agents.

The Short Answer

A2A is not hype in the sense of not working. The spec is real, the SDKs work, the adoption is genuine. It is hype in the narrower sense that most projects reaching for it in 2026 do not need it yet. Use MCP to give your agent tools. Reach for A2A once you have a second agent you do not control that must accept a task, work across several turns and hand back a result. If your case is "agent A calls agent B once and gets a string," that is a function call wearing a protocol.

References

A2A Protocol Explained: 150+ Companies Backed It in a Year - the original, with the full example and every source
A2A protocol surpasses 150 organizations - Linux Foundation, first-year numbers
Agent2Agent protocol is getting an upgrade - the v0.3 gRPC and signed-card release
Best MCP Servers and Build an MCP Server in Python - the tool layer underneath
MCP Server Authentication with OAuth 2.1 - the confused-deputy problem in detail

Qdrant Raised $50M, Pinecone Explored a Sale: Picking a Vector Database in 2026

Moksh Gupta — Thu, 06 Aug 2026 19:44:32 +0000

The two most useful facts about vector databases in 2026 are both business facts. Qdrant closed a $50M Series B in March 2026, taking total funding to about $87.8M. Pinecone, the company that turned "vector database" into a product category, reportedly hired bankers to explore a sale and replaced its founder-CEO with a hire from Google.

Meanwhile AWS, Google Cloud, Databricks, Snowflake and MongoDB all added vector search to databases they already sell, and Postgres does it with a free extension. The pure-play vendors are squeezed from both directions, so they are competing on latency, memory cost and hybrid search rather than on storing vectors at all. If you are building retrieval-augmented generation or agent memory, that competition is working in your favor, and the reflexive 2023 answer of "just use Pinecone" deserves rechecking.

I published a fuller version of this comparison on DevToolLab, Best Vector Databases in 2026, with the full pricing table and every source linked. Here is the compressed version.

The Search Problem in One Paragraph

An embedding model turns text, images or audio into a list of floats. Similar content lands close together in that space, so answering a question means embedding the question and finding the nearest stored vectors. Doing that exactly means scoring every vector you have, which is accurate and slow, so these systems use approximate nearest neighbor indexes, usually HNSW, and give up a sliver of recall for a large speedup. Everything that distinguishes one product from another sits around that: which indexes exist, how metadata filtering interacts with the index, how aggressively vectors can be quantized to cut RAM, and who runs the servers.

Three Shapes, Not Five Products

The market splits three ways, and that narrows the decision faster than any feature table.

Pinecone is managed and proprietary: no infrastructure to touch, no self-hosting escape hatch, usage-based bills. Weaviate, Qdrant and Milvus are open source with a managed cloud, which is where most of the current engineering effort is going. pgvector is a feature of a database you already operate, with vectors sitting beside your relational rows.

Pinecone

Fastest path from nothing to a working retrieval endpoint. Serverless separates storage from compute so idle indexes are cheap, namespaces handle multi-tenancy, and sparse-dense hybrid search is supported. Nothing to tune, which is the point.

The costs are strategic rather than technical. You cannot self-host, the index internals are undisclosed so you cannot swap algorithms or chase recall yourself, and the vendor is publicly in motion. The Starter tier is free up to 2GB with 1M read units and 2M write units a month. Builder is a flat $20/month; Standard carries a $50/month minimum and Enterprise $500/month, both pay-as-you-go above that. On Standard, usage lands around $16 to $18 per million read units, $4 to $4.50 per million write units and $0.33/GB/month of storage. Forecast your read volume before you commit, because read-heavy RAG is exactly where a usage meter surprises people.

Weaviate

Go, BSD-3-Clause, managed cloud and bring-your-own-cloud. Its real differentiator is hybrid search: dense similarity and BM25 keyword scoring fused natively, which beats pure vector search whenever exact tokens matter, and they matter more than people expect for SKUs, surnames and error strings.

Release 1.37.0 in April 2026 added a built-in MCP server in preview, so coding agents can query Weaviate over the Model Context Protocol, plus MMR diversity search, query profiling and incremental backups. Built-in vectorizer and reranker modules mean it can call your embedding model instead of making you embed client-side. The tradeoffs: the module system is configuration surface you have to learn, and single-node latency generally trails Qdrant. Self-hosting is free, the cloud free tier covers 100,000 objects, Flex starts at $45/month billed per million vector dimensions from $0.00465, and Premium starts at $400/month prepaid. That resource model replaced a $25/month serverless tier retired in October 2025, which annoyed a lot of small projects.

Qdrant

Rust and Apache 2.0, and the one I reach for when latency and a forecastable bill both matter. No garbage collector pauses, SIMD distance math, over 250 million package downloads and 29,000-plus GitHub stars, with Canva, HubSpot and Bosch in production.

Version 1.18 in May 2026 shipped TurboQuant, a quantization method derived from Google Research that claims roughly scalar-quantization recall at about half the memory, plus per-collection memory monitoring and named vectors you can add or drop without recreating the collection. That sits on top of scalar, product and binary quantization and filterable HNSW, so tight metadata filters do not destroy recall.

Cloud pricing is per provisioned resource (vCPU, RAM, disk) rather than per query, which makes cost flat no matter how much traffic you send. Fewer AI modules than Weaviate, so bring your own embeddings, and the enterprise feature set is younger than the incumbents. Qdrant also publishes its own benchmark suite showing itself ahead on QPS and latency; the methodology is open source, but read any single-vendor benchmark as directional. The free managed cluster is 1GB with no card and no expiry, which is genuinely enough to prototype on.

Milvus

Apache 2.0, built by Zilliz for billion-scale work, compute separated from storage. It supports the widest index range here by a distance: HNSW, IVF variants, DiskANN and GPU indexes including NVIDIA CAGRA.

Milvus 2.6 went open source in June 2025 and reached general availability on Zilliz Cloud on January 20, 2026, themed entirely on cost at scale. Zilliz claims RaBitQ 1-bit quantization cuts memory up to 72%, that tiered storage cuts storage cost substantially, and that its BM25 full-text search beats Elasticsearch by several times. Those are vendor numbers.

The honest cost is operational. Self-hosting means etcd, an object store, and several coordinator and worker components, which is why most teams that pick Milvus end up on Zilliz Cloud anyway. For a few million vectors it is straightforwardly overkill. Self-hosting is free, Zilliz Cloud has a 5GB free tier, serverless starts at $0 and bills around $4 per million vCUs, and dedicated clusters run from roughly $126/GB/month. Storage dropped to $0.04/GB/month in January 2026, down 87% from $0.30.

pgvector

Not a database. An extension under the PostgreSQL License that adds a vector column type and similarity search to Postgres, and for a large share of applications it is the correct answer that gets skipped for being boring. Embeddings live next to your relational data, inside the same transactions and joins, with no new system to operate. Every managed Postgres supports it.

0.8.0 added iterative index scans, fixing the old over-filtering problem where a strict WHERE plus vector search returned too few rows. Note that 0.8.2, released February 26, 2026, was a security fix for a buffer overflow in parallel HNSW index builds, tracked as CVE-2026-3172, so self-managed Postgres should be upgraded. It will not reach billions of vectors as gracefully as Milvus or Qdrant, and high recall at high QPS lags the purpose-built engines. Hybrid search is assembly work with Postgres full-text search rather than a feature. pgvectorscale from Timescale adds a StreamingDiskANN index if you need to close part of that gap.

The rule I would apply: stay on pgvector until you can prove you have outgrown one large Postgres node. Plenty of teams never do.

Five Questions, In Order

Already on Postgres with fewer than a few million vectors? pgvector. Do not add infrastructure yet.
Want zero ops and accept usage billing plus lock-in? Pinecone.
Is keyword-plus-vector hybrid search core, or are you multi-tenant SaaS? Weaviate.
Need the lowest latency and a flat bill? Qdrant.
Hundreds of millions of vectors, or GPU search? Milvus, most likely on Zilliz Cloud.

Prototyping on pgvector or a free managed tier and migrating later is a legitimate plan, because embeddings are portable. Migration is mostly re-upserting vectors.

What the Database Is Actually Doing

Before an index makes it fast, retrieval is one similarity calculation. This runs with NumPy alone:

import numpy as np

def cosine_similarity(a, b):
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

query = [0.11, 0.92, 0.34]          # real embeddings are 768-3072 dims
docs = {"doc-a": [0.10, 0.90, 0.30], "doc-b": [0.88, 0.05, 0.42]}

for doc_id, vec in sorted(docs.items(), key=lambda kv: cosine_similarity(query, kv[1]), reverse=True):
    print(f"{doc_id}: {cosine_similarity(query, vec):.4f}")

doc-a: 0.9995
doc-b: 0.2964

doc-a wins because it points nearly the same direction as the query. On Postgres the same idea is plain SQL, where <=> is cosine distance and smaller means closer:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
  id        bigserial PRIMARY KEY,
  content   text,
  embedding vector(1536)
);

CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

SELECT id, content
FROM documents
ORDER BY embedding <=> '[0.11, 0.92, 0.34, ...]'
LIMIT 5;

Quantization, hybrid fusion, distributed indexing and multi-tenancy all exist to keep that query working at a scale where one ORDER BY collapses. Two things worth having open while you build: the Cosine Similarity Calculator for checking your distance math by hand, and the Docker Compose Generator for standing up self-hosted Qdrant, Weaviate or Milvus locally before you pay anyone.

There is no winner here, only clean best-fit answers. Pick for the scale you have today, prove retrieval quality on your real corpus, and move when you hit a concrete limit rather than a hypothetical one.

References

Best Vector Databases in 2026: Tested and Ranked - the original, with the full comparison table and every source
Qdrant Series B announcement and Qdrant 1.18
Pinecone pricing, Weaviate pricing, Zilliz pricing, Qdrant pricing
Weaviate release notes and Milvus 2.6
pgvector 0.8.2 security release
Best MCP Servers - what Weaviate's new MCP server sits alongside

OpenAI Is Closing Fine-Tuning in January 2027. Here Is Where to Train Instead

Moksh Gupta — Wed, 05 Aug 2026 16:58:26 +0000

Go and read OpenAI's deprecations page. Not a blog post about it, the page itself, dated 7 May 2026. If your organisation has never run a fine-tuning job, you cannot start one. Since 2 July 2026, new jobs need inference on a fine-tuned model within the previous 60 days. On 6 January 2027 that door shuts for everybody, and fine-tuned gpt-3.5-turbo, gpt-4, gpt-4.1-nano, babbage-002 and davinci-002 go dark on 23 October 2026.

I put together the full comparison on DevToolLab with every price sourced from the provider's own page. This is the compressed version, because the headline deserves to travel: the most visible fine-tuning API in the industry is being retired while every alternative got cheaper and better in the same twelve months.

Behaviour, Not Facts

Before shopping for a platform, be sure you need one. Fine-tuning changes how a model behaves. It does not install knowledge. "The model has never heard of our internal billing API" is a retrieval problem. "The model understands the task but keeps returning prose when I asked for JSON, or picks the wrong label out of six" is a fine-tuning problem, and prompt engineering will keep almost-fixing it forever.

One filter decides it: could you sit down and write 200 to 1,000 examples of the output you want? If the answer is no, stop here. Nothing on this list rescues a dataset you cannot describe.

Four methods, briefly. SFT learns from input and ideal-output pairs, and covers most real needs. DPO learns from pairs where one answer beats another, which suits taste-driven work like tone or summary quality. RFT, in practice GRPO, learns against a grader that scores attempts, so it fits problems you can measure but cannot hand-write, such as code that must pass a test suite. Distillation copies a big model's outputs into a small one purely to cut serving cost.

Why Everyone Reaches for LoRA

LoRA freezes the base weights and trains a small adapter beside them. The ratio is the whole story, and it is easy to see for yourself. This ran on my laptop against peft 0.20.0:

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M")
model = get_peft_model(base, LoraConfig(
    r=16, lora_alpha=32, task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
))
model.print_trainable_parameters()

trainable params: 1,843,200 || all params: 136,358,208 || trainable%: 1.3517

Under one and a half percent of the weights carry the training. At fp16 that adapter is 3.5 MiB against a 260 MiB model, which is why a single GPU can serve dozens of task-specific variants off one base.

The Big Clouds Are Not All Retreating

OpenAI's own model optimization guide now warns that its fine-tuning material is moving to legacy documentation, which tells you the same thing the deprecation table does. What still runs for existing customers: SFT and DPO on gpt-4.1, mini and nano, vision tuning on gpt-4o-2024-08-06, RFT only on o4-mini-2025-04-16. Training is $25 per million tokens on gpt-4.1, $5 on mini, $1.50 on nano, and RFT bills $100 per hour of core training. No GPT-5 model was ever fine-tunable, and the migration notes send you to untuned gpt-5.4-mini and gpt-5.5.

Microsoft went the other way with the same model family. Foundry runs RFT on o4-mini with Global Training across 13 regions at lower per-token rates than standard, the trade being data residency. GPT-4.1, mini and nano serve as graders, and Microsoft's own advice is to grade with nano first. If you need managed OpenAI-family tuning that is not on a shutdown schedule, this is the shortest move.

Google bills Vertex tuning as dataset tokens multiplied by epochs, then charges above base rates to serve the tuned endpoint, which is the part that surprises people at invoice time. Check which snapshot is actually tunable before you plan around it, since Gemini 3.x tuning has been arriving through preview and allowlist rather than open self-serve.

AWS made the most interesting bet by treating open weights as the main event. Bedrock added RFT for Qwen3-32B and gpt-oss-20b on 17 February 2026, and the tuned model answers immediately through OpenAI-compatible Responses and Chat Completions endpoints. SageMaker AI followed on 25 March 2026 with serverless SFT, DPO and RFT across twelve more open models including gpt-oss-120b and Qwen3 14B, in N. Virginia, Oregon, Tokyo and Ireland. Anthropic is the gap: no fine-tuning in the public Claude API at all, only Bedrock customisation on whichever Claude models it exposes.

The Specialists Compete on Price

Together AI is the number to benchmark everyone else against: $0.48 per million training tokens for LoRA up to 16B, $1.50 from 17B to 69B, $2.90 from 70B to 100B, with full fine-tuning and DPO also available. Fireworks AI opens at a near-identical $0.50 up to 16B but climbs to $3.00 by 80B and $6.00 by 300B, so the model size you actually intend to train decides which is cheaper, not the logo.

Tinker from Thinking Machines Lab is the one I would reach for if I wanted control without babysitting GPUs. It meters per million tokens instead of per GPU-hour, Qwen3-8B moved from $0.40 to $0.44 on 17 July 2026, so a 50 million token run lands near $22 plus $0.10 per GB-month of checkpoints. It only does LoRA at rank 32, and it exports a plain PEFT adapter, so leaving costs you nothing.

Predibase deserves a caveat more than a recommendation. GRPO runs $10 per million tokens to 16B and $20 from there to 32B, with generous free serverless inference, but it has belonged to Rubrik since June 2025 and predibase.com no longer serves its own site as of August 2026. It redirects to Rubrik Agent Cloud. Confirm the product is still sold on its own before you build a roadmap on it. The original post has the full pricing table for all of these side by side.

Or Skip the Bill Entirely

With one GPU, or a rented hour of one, the free tooling is the fastest path in the category. Unsloth 2026.8.3 is the throughput specialist: GRPO roughly 1.3x faster, mixture-of-experts training 3 to 5x faster, and up to 80 percent less VRAM on reinforcement learning, which is precisely what puts GRPO within reach of a consumer card. Axolotl 0.18.0 keeps every run in a YAML file, so a training change arrives as a reviewable diff instead of a notebook nobody can reproduce. LLaMA-Factory 0.9.5 claims the widest model surface at 100-plus and ships a UI. Underneath all of them sit TRL 1.9.2 and PEFT 0.20.0, which now expose SFT, DPO and GRPO trainers directly if you would rather own the loop.

What you inherit is the GPU, the CUDA versions and the evaluation harness. A sane split: rent a hosted platform to find out whether the idea works at all, then move in-house once the same job runs every week.

Check the Dataset Before the Meter Starts

Training is billed per token, so a broken JSONL file costs money before it costs you an experiment. This is short enough to paste into any repo, and the output below is a real run against a deliberately damaged six-row file:

import json, sys, tiktoken

enc = tiktoken.get_encoding("o200k_base")
rows = bad = tokens = 0

for n, line in enumerate(open(sys.argv[1], encoding="utf-8"), 1):
    if not line.strip():
        continue
    rows += 1
    try:
        msgs = json.loads(line)["messages"]
        assert any(m["role"] == "assistant" for m in msgs), "no assistant turn"
    except Exception as e:
        print(f"line {n}: unusable - {e}")
        bad += 1
        continue
    tokens += sum(len(enc.encode(m.get("content", ""))) for m in msgs)

print(f"{rows} rows, {bad} unusable, {tokens:,} dataset tokens")
print(f"3 epochs = {tokens * 3:,} training tokens = ${tokens * 3 / 1e6 * 0.48:.2f} at $0.48/1M")

line 5: unusable - no assistant turn
line 6: unusable - 'messages'
6 rows, 2 unusable, 53 dataset tokens
3 epochs = 159 training tokens = $0.00 at $0.48/1M

Change the rate and the token count becomes a quote. Take something realistic: 5,000 examples averaging 400 tokens, three epochs, so 6 million training tokens. That is $2.88 on Together's LoRA tier, $2.64 on Tinker, $30 on gpt-4.1-mini, and $150 on full gpt-4.1. Small open models are not where the money goes. Serving them carelessly is.

Two housekeeping steps worth doing before upload, because training data ends up inside checkpoints you may keep for months: scrub it with the PII Redactor, and drop repeats with Remove Duplicate Lines, since near-identical rows both inflate the bill and teach the model to over-weight whatever you pasted twice.

The Step Teams Skip

Reserve 10 to 20 percent of your examples as an evaluation set the training run never sees. Then run one job on default hyperparameters and compare the result against the untuned base model with a good prompt. That single comparison is the only evidence that matters, and it is the one people leave out because it sometimes says the prompt was fine.

If the tuned model wins, ship it and watch cost per request. If it does not, the answer is nearly always more or better examples, not a different vendor.

What I Would Actually Do

Still on OpenAI with tuning in the roadmap: treat 6 January 2027 as immovable and pick a destination now. Foundry disrupts least, Together or Tinker on an open model costs least and outlives the decision.

Chasing the lowest price on something small: Together at $0.48, with Fireworks as your second quote. Below 16B the gap is pennies, so choose on which base models are offered.

Able to score outputs but not write them: that is the RFT case, so Bedrock or SageMaker on AWS, or GRPO in TRL and Unsloth if you want it in-house.

Not certain fine-tuning is even the answer: it probably is not, yet. Better prompt, then retrieval, then distil the working prompt into a smaller model.

References

Best AI Fine-Tuning Platforms in 2026: Hosted and Open Source Compared - the original, with the full pricing table and every source
OpenAI deprecations and the model optimization guide
Microsoft Foundry fine-tuning
Amazon Bedrock and SageMaker AI
Together AI, Fireworks AI, Tinker
Unsloth, Axolotl, LLaMA-Factory, TRL, PEFT
LLM Evals Guide - how to run the comparison that decides whether tuning helped