DEV Community: Alex Georgiev

MongoDB 8.2's encrypted substring search costs 20 times more per insert than equality

Alex Georgiev — Wed, 16 Sep 2026 08:00:00 +0000

I inserted 200 documents into a MongoDB collection with one encrypted, substring-searchable field and it took 4.6 seconds. The same 200 documents into a collection with only encrypted equality search took 232 milliseconds. Plaintext took 7 milliseconds.

MongoDB 8.2 added public preview support for prefix, suffix and substring queries against Queryable Encryption fields, through three new aggregation operators: $encStrStartsWith, $encStrEndsWith and $encStrContains. Until now, Queryable Encryption could only do equality and range lookups on an encrypted field — nothing that resembles a LIKE '%foo%'. I built a small encrypted collection, indexed an email field for substring search, and measured what that actually costs against the same field indexed for plain equality, and against no encryption at all.

Setup

I ran MongoDB 8.2.12 (docker run mongo:8.2, Community Edition, confirmed via db.runCommand({buildInfo:1}).modules returning an empty array — no Enterprise module) as a single-node replica set, which Queryable Encryption requires. Automatic encryption needs a query-analysis component; I used the crypt_shared shared library, downloaded directly from MongoDB's public download server with no login or Enterprise license involved. The client was PyMongo 4.18.1 with pymongo[encryption] and a local (non-production) KMS key. This is a public preview feature — MongoDB's own docs say not to use it in production, and preview functionality will be incompatible with the eventual GA version.

One thing worth correcting up front: I'd read that automatic Queryable Encryption needs MongoDB Enterprise. It doesn't. The crypt_shared library is a free download, and the whole setup above ran against stock Community Edition without complaint.

What it costs

I built three versions of the same collection, differing only in how the email field is indexed, inserted the same 200 generated documents into each (three trials, fastest reported), and compared to an unencrypted collection:

Configuration	Insert cost (ms/doc, fastest of 3)	Avg document size
Plaintext, no encryption	0.03	133 bytes
Queryable Encryption, equality only	1.16	489 bytes
Queryable Encryption, substringPreview	22.9	24,129 bytes

The equality number already reflects the tax you already know about with Queryable Encryption: roughly 35x the insert cost and 3.7x the document size of plaintext, because every write also has to touch the hidden ESC and ECOC metadata collections that back the encrypted index. Substring search pays that same tax and then a much larger one on top: 20x the insert cost of equality-only encryption, and storage that dwarfs both.

For a 500-document run I measured the full picture, including the metadata collections:

Configuration	Main collection	ESC	ECOC	Total for 500 docs	Per doc
Plaintext	66.5 KB	—	—	66.5 KB	133 B
Equality QE	244.5 KB	23.5 KB	51.5 KB	319.5 KB	639 B
substringPreview QE	12.06 MB	4.07 MB	9.17 MB	25.3 MB	50.6 KB

That's 380 times the storage of plaintext, and 79 times the storage of the same field encrypted for equality only. The reason is visible in the ECOC collection: it held 86,500 entries for 500 documents, or about 173 encrypted tokens per email address. strMaxLength was set to 40 and the query length range to 3–10 characters, and MongoDB has to generate an indexable token for every substring in that length range so any of them can later be matched without decrypting the field. A 25-character email address has roughly that many valid substrings between 3 and 10 characters long, so the token count checks out.

What it refuses

The field-length and query-length bounds are enforced, not advisory. Inserting a 47-character email into a field configured with strMaxLength: 40:

EncryptionError: StrEncode: String passed in was longer than the maximum
length for substring indexing -- String len: 47, max len: 40

Querying with a 2-character substring against a field configured with strMinQueryLength: 3:

EncryptionError: StrQuery: string value was shorter than the minimum query
length for this field after folding -- folded codepoint len: 2, min query len: 3

Both fail on the client, before anything reaches the server. So does a plain $regex against an encrypted field, run through the same encrypted client — it doesn't silently return nothing, it's rejected outright: Invalid match expression operator on encrypted field 'email'.

The sharper limitation is in how many query types one field can carry. I first tried configuring email with all three preview types — substring, prefix and suffix — on the theory that one indexed field should support all three lookup styles. MongoDB refused:

OperationFailure: The number of query types for an encrypted field cannot
exceed two

Dropping to two types, substring plus prefix, still failed:

OperationFailure: Multiple query types may only include the suffixPreview
and prefixPreview query types

So substringPreview cannot be combined with anything, not even equality, on the same field. The only legal pair is prefix and suffix together. I set that up — email configured with both prefixPreview and suffixPreview — and it worked: $encStrStartsWith and $encStrEndsWith both ran correctly against it, returning 61 and 113 matches respectively, sane numbers for the test data. That combined field cost 4.3 ms/doc to insert — cheaper than substring alone, but still 130x the 0.03 ms/doc plaintext baseline — and averaged 2,526 bytes per document, about a tenth of what substring alone needed. If your schema needs both prefix/suffix search and substring search on the same logical value, today that means two separate encrypted fields carrying the same plaintext, because one field can't carry both.

Query latency

Against 500 documents, $encStrContains for the string "acme" (112 matches) took a fastest time of 25.3 ms across five runs. The equivalent plaintext $regex query against the same data took 1.0 ms. Both returned exactly 112 documents, so correctness wasn't in question, only cost — roughly 25x slower on a dataset barely large enough to notice.

What I got wrong on the way

My first concurrency test used a Python thread pool: ten threads, each running the same $encStrContains query concurrently through a shared PyMongo client. Single-threaded, the query took 28.7 ms. At ten concurrent threads, the average jumped to 485.7 ms — a 17x slowdown that looked like severe server-side lock contention on the encrypted index.

It wasn't. Query analysis for these operators happens client-side, inside crypt_shared, and that work is CPU-bound. Python threads share one interpreter lock, so ten threads doing CPU-bound cryptographic work serialize against each other regardless of what the server is doing. I reran the same test using separate OS processes instead of threads, each with its own client and no GIL to contend over. The baseline (including per-process client setup) was 34–43 ms, and at ten concurrent processes the average was 87.2 ms — about 2x, not 17x.

The lesson isn't about MongoDB, it's about benchmarking anything that does client-side crypto from Python: a thread-pool concurrency test measures your interpreter's lock contention as much as it measures the database.

Watching it in production

db.serverStatus().fle reports live counts of which query types are in use across encrypted fields on the deployment:

indexTypeStats: {
  unindexed: 2,
  equality: 1,
  substringPreview: 1,
  suffixPreview: 0,
  prefixPreview: 0,
  ...
}

That's the one place I found to check, at a glance, whether anyone has put a substringPreview field into a cluster — useful given the preview warning against doing that in production. The same section also reports compactStats and cleanupStats for the ESC and ECOC collections, which is where you'd watch the housekeeping that keeps those metadata collections from growing forever as documents are updated or deleted.

Run it yourself

This needs Docker and Python 3.11+.

docker run -d --name mongoqe --network host mongo:8.2 --replSet rs0 --port 27117 --bind_ip_all
sleep 4
docker exec mongoqe mongosh --port 27117 --quiet \
  --eval 'rs.initiate({_id:"rs0", members:[{_id:0, host:"localhost:27117"}]})'

curl -sL -o crypt_shared.tgz \
  "https://downloads.mongodb.com/linux/mongo_crypt_shared_v1-linux-x86_64-enterprise-ubuntu2204-8.2.12.tgz"
mkdir crypt_shared && tar xzf crypt_shared.tgz -C crypt_shared

python3 -m venv venv && source venv/bin/activate
pip install "pymongo[encryption]"

Then, in Python:

import os
from pymongo import MongoClient
from pymongo.encryption import ClientEncryption, AutoEncryptionOpts

uri = "mongodb://localhost:27117/?replicaSet=rs0&directConnection=true"
key = os.urandom(96)
kms_providers = {"local": {"key": key}}

client = MongoClient(uri)
ce = ClientEncryption(kms_providers, "encryption.__keyVault", client, client.codec_options)

fields = {"fields": [{
    "path": "email", "bsonType": "string",
    "queries": [{"queryType": "substringPreview", "contention": 0,
                 "strMaxLength": 40, "strMinQueryLength": 3, "strMaxQueryLength": 10,
                 "caseSensitive": False, "diacriticSensitive": False}],
}]}
coll, _ = ce.create_encrypted_collection(client["qedemo"], "users", fields, "local")

auto_opts = AutoEncryptionOpts(kms_providers, "encryption.__keyVault",
                                crypt_shared_lib_path="crypt_shared/lib/mongo_crypt_v1.so")
ecoll = MongoClient(uri, auto_encryption_opts=auto_opts)["qedemo"]["users"]
ecoll.insert_one({"email": "alice@acme.com"})

print(list(ecoll.aggregate([
    {"$match": {"$expr": {"$encStrContains": {"input": "$email", "substring": "acme"}}}},
    {"$project": {"__safeContent__": 0}},
])))

That last $project matters: every document in an encrypted collection carries a __safeContent__ array of encrypted search tags, and without excluding it you'll get a wall of binary blobs back with your result. I verified every command above against a fresh container before publishing.

If you're evaluating Queryable Encryption for a field that needs pattern matching, measure the insert and storage cost on your own document size and access pattern before committing to it — the multiplier scales with how many substrings your strMaxLength/query-length settings force it to index, so a longer field or a wider query-length range will cost more than what I measured here, not less. And if you need more than one style of pattern lookup on the same value, budget for a second encrypted field, because MongoDB won't let one field carry substring search alongside anything else.

Tigris marks every read with the region and cache layer that actually served it

Alex Georgiev — Tue, 15 Sep 2026 13:00:00 +0000

Disclosure: I have no affiliation with Tigris. Nobody asked me to write this and nobody paid for it. I signed up and generated an access key like anyone else would.

I created a bucket with no region argument at all, put one object in it, then read it back and looked at the response headers instead of just the body. Three of them named something most S3-compatible services never tell you: X-Tigris-Regions: fra, X-Tigris-Served-From: fra, X-Tigris-Read-Source: cache. The read wasn't served from an abstraction, it was served from Frankfurt, from a cache layer, and the API told me so without being asked.

What Tigris is

Tigris is S3-compatible object storage built around one specific pitch: write from anywhere, read from anywhere, and the data follows access patterns globally instead of living in one region you have to pick upfront. It's a $25M Series A company (Spark Capital led, a16z participating, October 2025), founded 2021, and despite claiming 4,000+ customers, its dev.to and Hacker News presence is almost entirely other people's tutorials using it as an S3/R2 alternative inside broader posts, not dedicated coverage of the company itself.

Global by default, pinnable if you want it

Creating a bucket with the plain S3 API and no location constraint:

aws s3api create-bucket --bucket my-bucket --endpoint-url https://t3.storage.dev

aws s3api get-bucket-location --bucket my-bucket --endpoint-url https://t3.storage.dev

{
    "LocationConstraint": "global"
}

global is the default, not a special mode you opt into. If you want the opposite, standard single-region behavior, that's one parameter away:

aws s3api create-bucket --bucket my-pinned-bucket \
  --create-bucket-configuration LocationConstraint=iad \
  --endpoint-url https://t3.storage.dev

aws s3api get-bucket-location --bucket my-pinned-bucket --endpoint-url https://t3.storage.dev

{
    "LocationConstraint": "iad"
}

Both requests succeeded on the first try, no special account flag or enterprise-tier gate to reach either mode.

What the read path actually reveals

Back on the global bucket, head-object and get-object both return the same three headers on every response, not just the first one:

aws s3api get-object --bucket my-bucket --key test/object.txt /tmp/out.txt --debug 2>&1 | grep -oE "X-Tigris-[A-Za-z-]+': '[a-z]+'"

X-Tigris-Read-Source': 'cache'
X-Tigris-Regions': 'fra'
X-Tigris-Served-From': 'fra'

I ran that same request three times in a row and got the identical answer each time: fra, cache, fra. That's the honest limit of what a single test from a single network location can show, this confirms the metadata is real and consistently exposed, not that I've proven true global read-latency optimization, which would need requests actually originating from multiple regions to test properly. What it does prove is that the abstraction isn't a black box: if a read from your application looks slow, the response headers tell you which physical region and cache layer actually handled it, instead of leaving you to guess.

Presigned URLs work the way you'd expect and don't require the requester to have any credentials at all:

aws s3 presign s3://my-bucket/test/object.txt --expires-in 300

Fetching that URL with a plain curl, no AWS credentials anywhere on that machine, returned the exact same bytes I'd uploaded.

What it gets wrong

The S3 API surface isn't complete, and where it's incomplete, the failure mode is more confusing than it needs to be. Trying to enable versioning on an existing bucket:

aws s3api put-bucket-versioning --bucket my-bucket --versioning-configuration Status=Enabled --endpoint-url https://t3.storage.dev

An error occurred (BucketAlreadyExists) when calling the PutBucketVersioning operation:
The requested bucket name is not available. The bucket namespace is shared by
all users of the system. Specify a different name and try again.

That error is for bucket creation, not versioning, on a bucket that already exists and that I already own. It reads like the versioning call got routed into the same handler as CreateBucket rather than into real versioning logic, or rejected into whatever error path was closest at hand. Either way, get-bucket-versioning before and after the failed call returned an empty response both times, so versioning is not silently on, but a user hitting this would spend real time debugging a bucket-naming problem that doesn't exist before concluding the feature just isn't there.

The zero-egress claim

Tigris's own pricing page states $0.02/GB/month for standard storage and zero egress fees, with the same rate everywhere rather than per-region pricing. I didn't generate enough traffic in a short test to see a real bill, so I can't independently confirm the egress number the way I confirmed the region headers or the versioning error, this is what their pricing page says, not something I measured. Worth stating plainly rather than repeating it as if I'd verified it myself.

Run it yourself

export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
aws s3api create-bucket --bucket your-test-bucket --endpoint-url https://t3.storage.dev
aws s3api get-bucket-location --bucket your-test-bucket --endpoint-url https://t3.storage.dev
echo "test" > test.txt
aws s3api put-object --bucket your-test-bucket --key test.txt --body test.txt --endpoint-url https://t3.storage.dev
aws s3api get-object --bucket your-test-bucket --key test.txt out.txt --debug --endpoint-url https://t3.storage.dev 2>&1 | grep -i tigris

Clean up when you're done:

aws s3 rm s3://your-test-bucket --recursive --endpoint-url https://t3.storage.dev
aws s3api delete-bucket --bucket your-test-bucket --endpoint-url https://t3.storage.dev

If you're weighing other object storage

DigitalOcean Spaces and AWS S3 are the two most likely comparison points if you're evaluating this space, and they price the global-vs-regional tradeoff very differently from Tigris.

DigitalOcean Spaces charges $5/month for a base subscription covering 250GiB of storage and 1,024GiB of outbound transfer, with $0.02/GiB for additional storage and $0.01/GiB for additional egress beyond that. Egress is only free in specific same-region Spaces-to-Droplet pairs (for example Spaces in FRA1 to Droplets in FRA1), not globally, so a Spaces bucket is a regional resource you pay to move data out of once you leave that pairing.

AWS S3 Standard charges $0.023/GB for the first 50TB of storage per month in us-east-1, and egress to the internet is free only for the first 100GB per month (aggregated across all AWS services), then $0.09/GB for the next roughly 10TB, stepping down at higher volume tiers. Both storage and egress are priced per region, and moving data between AWS regions costs more again.

Tigris's pitch is specifically that neither of those tradeoffs applies: one global price, zero egress, and the region-pinning option still there if you genuinely need data to stay put. Whether that's worth it depends on how much of your bill today is egress versus storage, that's a question only your own traffic pattern can answer, not something a single test bucket can tell you.

Redis 8.10's compact hashes need HIMPORT or a restart to kick in

Alex Georgiev — Tue, 15 Sep 2026 08:00:00 +0000

Redis 8.10 shipped in July with a feature called compact hashes: if many hashes share the same field names, Redis can store those field names once instead of once per key. The release blog claims up to 50% lower memory and 2x higher loading throughput. I wanted to know what "up to" actually means for a normal schema, and what you have to do to get it, because the answer to the second question turned out to be less obvious than I expected.

I ran Redis 8.10.1 in Docker and tested it against two schemas, a throughput benchmark, a concurrency check, and the two ways to actually trigger the new encoding: the new HIMPORT command, and a server restart.

The straightforward result

Redis 8.10 adds hash templates: an internal encoding where hashes with the same field names keep one shared copy of those names. The only documented way to create one directly is HIMPORT, a new command pair that declares a field list once per connection and then loads values against it:

HIMPORT PREPARE u name email country last_login
HIMPORT SET user:1 u Alice alice@example.com UK 2026-07-14
HIMPORT SET user:2 u Bob bob@example.com US 2026-07-14

I loaded 100,000 hashes with a four-field user-profile schema (name, email, country, last_login) two ways: plain HSET per key, and HIMPORT against a prepared fieldset. I measured used_memory from INFO memory before and after each load, flushing the database between runs.

Method	Bytes/hash	Reduction
Plain HSET	159.9	—
HIMPORT	127.6	20.2%

Nowhere near the advertised 50%. So I built a second schema designed to favour the feature: eight fields with longer names (account_status_code, two_factor_enabled_flag, and so on) but short single-character or short-word values, the shape you'd get from flags and codes rather than free text.

Method	Bytes/hash	Reduction
Plain HSET	271.3	—
HIMPORT	87.3	67.8%

So the 50% figure is workload-dependent in both directions: my ordinary schema landed well under it, my field-name-heavy schema went well over it. The saving comes specifically from field names, so the more your schema's memory is field names rather than values, the more you get back.

The number that didn't hold up

The blog also claims "up to 2x higher hash loading throughput" for HIMPORT versus per-key HSET. I loaded 500,000 hashes both ways, three times each, timed from inside the container with bash's time builtin to avoid docker exec overhead skewing the numbers:

Method	Fastest of 3 runs	Hashes/sec
Plain HSET	2.698s	185,323
HIMPORT	1.992s	251,004

That's a 35% improvement, not 2x. I don't know what workload produces the blog's number, maybe a wider schema, maybe a network-bound client where skipping repeated field names in the wire protocol matters more than it does over a Unix socket to a local container. What I can say is that on a bulk pipe load with redis-cli --pipe, the throughput gain was real but far more modest than advertised.

Plain HSET gets nothing, ever, until you restart

This was the part I hadn't expected going in. I loaded the same 100,000-hash, shared-schema dataset with ordinary HSET calls, then checked INFO stats:

hash_templates:0
hash_template_keys:0

Zero. Redis does not look at a live hash, notice it shares a field set with a thousand others, and fold it into a template. That only happens two ways: through HIMPORT, or by reloading an RDB file with three new config parameters set, all of which default to off:

hash-rdb-load-min-template-entries    0
hash-rdb-load-max-template-entries    0
hash-rdb-load-template-disassembly-threshold  0

I set these to 3, 20 and 2 and restarted the container expecting the conversion to happen. It didn't, and hash_templates stayed at 0. The reason turned out to be my mistake, covered below. Once I fixed it, a BGSAVE followed by a container restart converted all 100,000 plain-HSET hashes into a single shared template, and used_memory dropped from 17,459,464 to 14,182,656 bytes, an 18.8% reduction, consistent with the direct HIMPORT comparison on the same schema.

The practical upshot: if your application already writes hashes with plain HSET and a consistent schema, compact hashes give you nothing until you either rewrite your write path to use HIMPORT, or turn on the RDB-load config and restart (or fail over) every node that holds the data. A long-lived primary that never restarts and never uses HIMPORT sees zero benefit from this feature, indefinitely.

Small numbers of shared keys make things worse, not better

A template has its own fixed overhead, and something has to pay for it. I varied the number of keys sharing one fieldset and read MEMORY USAGE on the first key each time:

Keys sharing the template	MEMORY USAGE (bytes)
2	156
3	129
4	116
5	108
20	84
100	77
1,000	76
5,000	76

An equivalent plain hash with the same fields reports 112 bytes. So with two or three keys sharing a template, you're worse off than not using the feature at all. The crossover is around four to five keys, and the saving plateaus by about a hundred. If you're grouping small, short-lived batches of hashes under one fieldset, this feature can quietly cost you memory rather than save it.

What happens if you mutate a templated hash

I ran HSET to add a field that wasn't in the schema on one of my 1,000-key templated hashes, and HDEL to remove a field on another:

> HSET user:1 extra_field zzz
(integer) 1
> MEMORY USAGE user:1
(integer) 265

I expected the key to fall back to being a normal hash. It didn't. INFO stats showed hash_templates go from 1 to 2, then to 3 after the HDEL, while hash_template_keys stayed at 1,000 throughout. Redis had forked off a brand new, single-key template for each mutated hash rather than detaching it back to plain encoding. user:1 went from 76 bytes to 265, and the HDEL'd key went to 208, both worse than the 112-byte plain hash baseline. The other 998 untouched keys stayed at 76 bytes, unaffected. So a single stray field on a templated hash doesn't just lose the sharing benefit for that key, it costs more than never having used the feature at all.

Refusals and connection scoping

HIMPORT's fieldsets live on the connection, not the server, and Redis enforces this strictly. Sending the wrong number of values:

> HIMPORT PREPARE u2 name email country last_login
OK
> HIMPORT SET user:3 u2 Carol carol@example.com
(error) ERR value count does not match fieldset field count

Referencing a name nobody prepared:

> HIMPORT SET user:4 doesnotexist X Y Z W
(error) ERR no such fieldset

And, importantly, opening a fresh connection and trying to use a fieldset prepared on another one fails with the same error. I confirmed this by preparing a fieldset in one redis-cli session and calling HIMPORT SET against it from a second, separate redis-cli invocation. Any connection-pooled client, which most production Redis clients are, will hit this the moment a request lands on a different pooled connection than the one that ran PREPARE. You'd need to re-run PREPARE on every connection in the pool, or pin the whole import to a single connection. RESET also drops the fieldset immediately, which I checked directly.

Concurrency behaved better than I expected

I ran ten separate redis-cli --pipe processes in parallel, each opening its own connection, each declaring the same four-field schema under its own PREPARE, each writing 2,000 disjoint keys. All ten finished with zero errors in under a third of a second combined. The interesting part was in INFO stats afterwards:

hash_templates:1
hash_template_keys:20000

One template, not ten. Even though each connection prepared its fieldset independently, Redis deduplicated them server-side by the actual field names, not by the client-chosen fieldset label. Ten unrelated connections converged on one shared template with no coordination required and no errors under concurrent writes.

What I got wrong on the way

My first attempt to test the RDB auto-conversion config failed silently. I set the three hash-rdb-load-* parameters with CONFIG SET, ran BGSAVE, then did docker restart on the container. After the restart, hash_templates was still 0, and the config values had reverted to 0 as well. I assumed the container had picked up the same settings, since docker restart re-runs the same command. It does, but I'd started the original container with only --save "" --appendonly no, no config file and none of the template flags on the command line, so CONFIG SET values I'd applied at runtime were never persisted anywhere Redis would read them again. docker restart doesn't remember CONFIG SET history. I had to stop the container, remove it, and start a new one with the three parameters passed directly as redis-server arguments before the RDB reload conversion worked at all.

Run it yourself

This loads 100,000 hashes both ways and prints the memory delta for each.

docker run -d --name redis810 -p 16379:6379 redis:8.10.1 \
  redis-server --save "" --appendonly no

python3 -c "
for i in range(100000):
    print(f'HSET plain:{i} name User{i} email user{i}@example.com country UK last_login 2026-07-14')
" > plain.txt

python3 -c "
print('HIMPORT PREPARE u name email country last_login')
for i in range(100000):
    print(f'HIMPORT SET himp:{i} u User{i} user{i}@example.com UK 2026-07-14')
" > himp.txt

docker exec redis810 redis-cli FLUSHALL
docker exec redis810 redis-cli INFO memory | grep ^used_memory:
cat plain.txt | docker exec -i redis810 redis-cli --pipe
docker exec redis810 redis-cli INFO memory | grep ^used_memory:

docker exec redis810 redis-cli FLUSHALL
docker exec redis810 redis-cli INFO memory | grep ^used_memory:
cat himp.txt | docker exec -i redis810 redis-cli --pipe
docker exec redis810 redis-cli INFO memory | grep ^used_memory:
docker exec redis810 redis-cli INFO stats | grep hash_template

I ran this exact script against Redis 8.10.1 while writing this post; the numbers match the first table above within a few percent between runs.

What to do with this

If you're already running a schema where thousands of hashes share the same fields, HIMPORT is worth adopting for bulk loads and ETL paths specifically, not as a drop-in replacement for HSET in your normal write path. Check your actual field-name-to-value ratio before trusting the 50% figure either way. If you can't change your write path, the RDB-reload config is a lower-effort option, but remember it does nothing until the next restart or failover, and it needs the three hash-rdb-load-* parameters set explicitly since none of them default to on. Either way, watch INFO stats for hash_templates and hash_template_keys before and after, because at low key-sharing counts, or after a single stray HSET on one of the keys, the feature can leave you worse off than not using it.

Django 6.1's FETCH_PEERS collapses a 2,001-query loop into 2

Alex Georgiev — Mon, 14 Sep 2026 08:30:00 +0000

A loop over 2,000 Django model instances that touched a foreign key on each one fired 2,001 queries against Postgres and took, at its fastest run, 1.14 seconds on localhost. That is the N+1 problem everyone who has used Django has hit at least once. Django 6.1, released on 5 August 2026, added a way to fix it without adding select_related() or prefetch_related() calls at every call site: a queryset-level setting called fetch_mode.

I built a small Postgres-backed project, ran the loop both ways, then spent the rest of the afternoon trying to break the new mode. It mostly held up. One combination silently didn't work at all, and I only found it because I went looking for the cases the documentation doesn't mention.

The setup

Two models, a classic one-to-many:

class Author(models.Model):
    name = models.CharField(max_length=100)
    bio = models.TextField(default="")

class Book(models.Model):
    title = models.CharField(max_length=200)
    author = models.ForeignKey(Author, on_delete=models.CASCADE, related_name="books")
    description = models.TextField(default="")
    price_cents = models.IntegerField(default=999)

50 authors, 2,000 books, seeded with bulk_create into Postgres 17 running in Docker, no cloud, no managed service. Django 6.1.1 on Python 3.12, psycopg2-binary as the driver. I counted queries with django.test.utils.reset_queries() and len(connection.queries), and timed each run with time.perf_counter(), five repetitions per case.

The headline number

def default_n_plus_1():
    total = 0
    for book in Book.objects.all():
        total += len(book.author.name)
    return total

def fetch_peers():
    total = 0
    for book in Book.objects.fetch_mode(models.FETCH_PEERS):
        total += len(book.author.name)
    return total

approach	queries	fastest of 5 runs
default (`FETCH_ONE`)	2,001	1,144.6 ms
`fetch_mode(FETCH_PEERS)`	2	13.1 ms
`select_related("author")`	1	13.7 ms

FETCH_PEERS took the 2,001-query loop down to 2 queries and cut the wall time by roughly 87x. It landed within a millisecond of select_related, despite firing one extra query, because that second query is a single WHERE id IN (...) batch fetch rather than 2,000 round trips. I checked the actual SQL Django ran:

SELECT "blog_book"."id", "blog_book"."title", "blog_book"."author_id", ... FROM "blog_book"
SELECT "blog_author"."id", "blog_author"."name", "blog_author"."bio" FROM "blog_author" WHERE ("blog_author"."id") IN ((1), (2), (3), ...

That is exactly what prefetch_related() produces, except you didn't have to write it. The documentation's claim that this "works like an on-demand prefetch_related()" and "reduces most cases of the N+1 problem to two queries" held up precisely, down to the query count.

It held for deferred fields too, not just foreign keys. Loading books with .only("id", "title") and then touching the deferred description field, under FETCH_PEERS, also went from 2,001 queries and roughly 1.3 seconds to 2 queries and 14ms.

What it refuses: the many side

FETCH_PEERS documentation lists what it applies to: forward foreign keys, one-to-one fields and their reverse accessors, deferred fields, generic relations. It does not list reverse foreign key managers, and testing confirmed that gap is real, not an oversight in my reading.

def reverse_fk_fetch_peers():
    total = 0
    for author in Author.objects.fetch_mode(models.FETCH_PEERS):
        for book in author.books.all():
            total += len(book.title)
    return total

approach	queries
default, `author.books.all()` per author	51
`fetch_mode(FETCH_PEERS)`, same loop	51

Identical. Setting fetch_mode on the Author queryset does nothing for author.books.all(), because that call returns a fresh RelatedManager queryset rather than fetching a field value. If your N+1 problem is on the "many" side of a relation, fetch_mode will not touch it; you still need prefetch_related("books"). That is a real limitation for anyone reading the feature announcement and assuming it replaces prefetch_related() everywhere.

What it rejects outright

FETCH_RAISE is the third mode, meant to catch accidental lazy loading in code that should already have everything it needs:

book = Book.objects.only("id", "title").fetch_mode(models.FETCH_RAISE).first()
book.description

django.core.exceptions.FieldFetchBlocked: Fetching of Book.description blocked.

Same exception, same message shape, for a blocked forward foreign key: Fetching of Book.author blocked. Both are precise enough to paste into a bug report and know exactly which field and which model tripped it.

The gotcha nobody warns you about

FETCH_PEERS works by looking at every other instance that came out of the same queryset evaluation and batching the missing field across all of them. That requires the queryset to have materialised its full result list in memory, because peer tracking is done through weak references stored on each instance once the list exists.

QuerySet.iterator() exists specifically to avoid that materialisation, for cases where a queryset is too large to hold in memory at once. So I tried combining them:

for book in Book.objects.fetch_mode(models.FETCH_PEERS).iterator(chunk_size=200):
    total += len(book.author.name)

Result: 2,001 queries. Not an error, not a warning, just the full N+1 pattern the feature exists to prevent. I checked the Django source (django/db/models/fetch_modes.py) to confirm this wasn't a bug in my own code:

class FetchPeers(FetchMode):
    def fetch(self, fetcher, instance):
        instances = [p for w in instance._state.peers if (p := w()) is not None]
        if len(instances) > 1:
            fetcher.fetch_many(instances)
        else:
            fetcher.fetch_one(instance)

With .iterator(), each instance's peers list never grows past itself, so len(instances) > 1 is never true, and every access quietly falls through to a single-row fetch. I could not find this interaction called out anywhere in the 6.1 release notes or the fetch modes documentation page I read. Anyone who reaches for .iterator() on a large table for memory reasons, which is exactly when you'd want the query-count win most, gets none of it and no signal that anything went wrong.

What else it costs

FETCH_PEERS batches eagerly, not lazily per access. I built a 2,000-row queryset and touched the author field on only every fourth book, 500 accesses out of 2,000 possible:

books=2000 accessed=500 queries=2

Still 2 queries. The first access to any instance's author field fetches authors for every peer in the queryset, whether or not you go on to touch the rest. That's the right trade-off if you're going to touch most of the set, and a waste if you only needed a slice of it, in which case a plain filtered queryset with select_related would cost less.

Peer tracking also turned out to be correctly scoped even when instances get mixed together after the fact. I built two separate querysets, both under FETCH_PEERS, concatenated their result lists into one Python list, then accessed .author across the combined list:

queries after touching .author on ALL 199 combined elements: 2

Two queries, one per original queryset, not one per Python list. Objects remember which evaluation they came from regardless of how you recombine them afterwards, which is the behaviour I'd want but hadn't seen stated anywhere.

Memory cost was measurable but small at this scale: materialising 2,000 Book instances added about 1.77 MiB over baseline, and the FETCH_PEERS batch fetch of 50 authors added a further 400 KiB. For a table with a genuinely large row count, that first number is the one to watch, since it scales with how many rows you load before touching anything, independent of whether you use FETCH_PEERS at all.

Under 10 concurrent threads, each running its own independent FETCH_PEERS loop against the same Postgres instance, every thread still saw exactly 2 queries, and results were correct in all 10. Per-thread wall time rose from 13ms solo to between 87ms and 179ms under contention, which is Postgres connection load, not a fetch_mode problem: the query-count discipline held regardless of how many callers were doing it at once.

What I got wrong on the way

My first version of the query-counting harness used django.test.utils.CaptureQueriesContext, which slices into connection.queries_log, a deque capped at 9,000 entries. Running the 2,001-query N+1 case five times in a row, inside the same process, blew past that cap partway through, and every benchmark after the cap was hit silently reported 0 queries instead of an error. I only noticed because a select_related case that should have shown 1 query showed 0, which was too suspicious to accept. Switching to reset_queries() before each run, so the deque never had to hold more than one run's worth of history, fixed it. The lesson generalises: any tool that logs to a fixed-size buffer will lie to you quietly once you exceed it, and a suspiciously clean zero is worth more suspicion than a suspiciously large number.

Run it yourself

This needs Python 3.12+ (Django 6.1 requires it) and either a local Postgres or Docker:

docker run -d --name pgtest -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=testdb -p 5432:5432 postgres:17
python3.12 -m venv venv && ./venv/bin/pip install django psycopg2-binary
./venv/bin/django-admin startproject testproj .
# add 'blog' to INSTALLED_APPS, point DATABASES at testdb/postgres/postgres/127.0.0.1
./venv/bin/python manage.py makemigrations blog && ./venv/bin/python manage.py migrate

Then the benchmark itself, saved and run as bench.py from inside the project directory (so it can import testproj.settings):

import os, django, time
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "testproj.settings")
django.setup()
from django.db import models, connection
from django.test.utils import reset_queries
from blog.models import Book

def timed(fn, reps=5):
    for _ in range(reps):
        reset_queries()
        t0 = time.perf_counter()
        fn()
        print(len(connection.queries), (time.perf_counter() - t0) * 1000, "ms")

def default_n_plus_1():
    for book in Book.objects.all():
        _ = book.author.name

def fetch_peers():
    for book in Book.objects.fetch_mode(models.FETCH_PEERS):
        _ = book.author.name

timed(default_n_plus_1)
timed(fetch_peers)

What to do with this

If you're on Django 6.1 and have a view or serializer with a loop that touches a related object per instance, fetch_mode(models.FETCH_PEERS) on the queryset is a genuine drop-in fix, cheaper to write than auditing every call site for missing select_related. Don't reach for it on the reverse side of a relation, it won't help there. And if your reason for using .iterator() in the first place was a large table, check your query count after adding fetch_mode, don't assume it's working just because the code runs without error.

Docker Engine 29's default image store lets --storage-opt size fail silently

Alex Georgiev — Sun, 13 Sep 2026 08:00:00 +0000

Docker Engine 29 switches new installs from the classic overlay2 graphdriver to the containerd snapshotter as the default image store. I wanted to know what actually changes when that flag flips, so I ran the same host, the same images, and the same flags against both backends and compared what came out.

The interesting result wasn't a speed number, although I got one of those too. It was a container disk quota that stopped being enforced and didn't tell me.

The setup

I have root on a fresh Docker 29.3.1 install (overlay2 root filesystem is plain ext4, no project quota mount option). I can flip the backend with a daemon flag and restart:

{ "features": { "containerd-snapshotter": false } }

docker info confirms which one is active:

Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1

versus

Storage Driver: overlay2

Finding 1: the quota flag stops erroring, and stops working

--storage-opt size=<limit> is supposed to cap a container's writable layer. It has a well-known precondition: it only works on overlay2 over XFS with the pquota mount option. My ext4 root doesn't have that. Under the classic graphdriver, Docker knows this and refuses:

$ docker run --rm --storage-opt size=100M alpine:3.20 echo ok
docker: Error response from daemon: --storage-opt is supported only for overlay over xfs with 'pquota' mount option

Exit code 125, every time, three runs in a row. The container never starts. That's a correct, loud failure on a host that can't honour the option.

Under the containerd snapshotter — the Docker 29 default — the exact same command and the exact same filesystem behave differently:

$ docker run --rm --storage-opt size=100M alpine:3.20 sh -c \
    "dd if=/dev/zero of=/bigfile bs=1M count=300; ls -lh /bigfile"
300+0 records in
300+0 records out
314572800 bytes (300.0MB) copied, 1.81 s, 165.5MB/s
-rw-r--r--  1 root root  300.0M Sep 13 05:06 /bigfile

Exit code 0. I asked for a 100MB limit and wrote 300MB with no resistance, no warning, no log line. I repeated it three times to be sure it wasn't a one-off race:

Backend	`--storage-opt size=100M`, write 300MB	Exit code
overlay2 (graphdriver)	refuses to start	125, ×3
containerd snapshotter	writes all 300MB	0, ×3

It gets worse. docker inspect still reports the option as if it applied:

$ docker inspect quotatest --format '{{.HostConfig.StorageOpt}}'
map[size:100M]

If you have anything that checks container config to confirm a quota is set — a compliance script, an audit tool, a human reading docker inspect output — it will tell you the limit is in place. Nothing enforces it. This is the kind of thing that looks fine in every dashboard until a runaway container fills the disk that three other containers are also writing to.

I don't think this is a deliberate design choice so much as an unimplemented check: the containerd image store path doesn't currently validate that the backing filesystem supports the quota before accepting the option, where the graphdriver path does. Whatever the cause, the behaviour changed under a default flip, and nothing in the CLI tells you.

Finding 2: pulls are genuinely faster, independent of why

I pulled node:22 (8 layers, 1.64GB unpacked) from a clean state, alternating backends, always removing the local image first:

Backend	Run 1	Run 2	Run 3	Average
containerd snapshotter	19.9s	17.7s	16.4s	18.0s
overlay2 (graphdriver)	28.9s	28.5s	28.1s	28.5s

That's roughly 37% faster on the new default, consistently, and the graphdriver runs happened later in my test sequence, so it isn't warm-cache bias in the snapshotter's favour — if anything a registry or proxy cache would have made the later runs faster, and they were the slow ones.

Finding 3: the concurrency knob barely matters, which wasn't what I expected

Docker 29's own release notes mention a bug, fixed in 29.7.0, where concurrent pull limits weren't honoured on the snapshotter path. I'm running 29.3.1, so I expected to see it: set max-concurrent-downloads to 1 and watch pulls slow down a lot more under the snapshotter than under the graphdriver.

That's not what happened. Setting the limit to 1 versus 8 moved the snapshotter average from 18.0s to 15.0s — about 17%. Under the graphdriver it moved from 28.5s to 27.5s — about 3.5%, close to noise:

Backend	max-concurrent-downloads=1	max-concurrent-downloads=8
containerd snapshotter	18.0s	15.0s
overlay2 (graphdriver)	28.5s	27.5s

Neither backend shows the dramatic serialization I was looking for. My conclusion, from these numbers, is that unpacking the layers into the filesystem dominates the wall-clock time for a pull far more than the number of concurrent HTTP downloads does, at least on this network path. The backend-versus-backend gap in finding 2 is real and reproducible; the concurrency-setting gap inside either backend is not where the time goes.

Finding 4: the new mount type does what it says, mostly

Docker 29 also graduates --mount type=image out of experimental status — in 29.7.0, according to the release notes. On 29.3.1 it's still flagged:

$ docker run --rm --mount type=image,source=alpine:3.20,target=/mnt \
    busybox:latest ls /mnt
WARNING: Image mount is an experimental feature
bin  dev  etc  home  lib  ...

It correctly refuses writes:

$ docker run --rm --mount type=image,source=alpine:3.20,target=/mnt \
    busybox:latest touch /mnt/testfile
touch: /mnt/testfile: Read-only file system

Exit 1, as it should be — this is the one place where the new feature enforced a restriction I expected and the quota flag didn't. It also fails cleanly on a bad source image (No such image, exit 125) and on a missing target (field Target must not be empty). What I couldn't get working on this build was a subpath option to mount only part of an image; every spelling I tried (subpath=, src.subpath=) came back as an unrecognised option. That may simply not exist yet in 29.3.1's experimental implementation — I'm reporting what this specific version accepts, not what later docs promise.

Finding 5: dedup accounting still works

One thing I checked because I assumed it might have regressed, and it hadn't: docker system df -v still correctly attributes shared base layers under the snapshotter. python:3.12-slim and python:3.13-slim share a Debian base:

REPOSITORY   TAG         SIZE     SHARED SIZE   UNIQUE SIZE
python       3.12-slim   190MB    87.45MB       102.9MB
python       3.13-slim   189MB    87.45MB       101.7MB

87.45MB shared, matching on both images, which is what you'd want to see if you're trying to estimate real disk cost from a fleet of related images. This is the one finding in this post that's a non-finding — I looked for a problem and didn't find one.

What I got wrong on the way

I set out to reproduce the "concurrent download limits not honoured" bug mentioned in the 29.7.0 changelog, expecting my max-concurrent-downloads=1 runs to be dramatically slower than =8 on the snapshotter. When the difference came out small on both backends, my first instinct was that my harness was broken — maybe the proxy in front of my registry was capping throughput regardless of client-side concurrency, hiding the effect. I re-ran with a completely fresh image (python:3.13-slim, untouched by earlier tests) to rule out any registry-side caching of my specific pull pattern, and got the same shape of result. The actual finding wasn't the one I went looking for: the backend swap (finding 2) explains far more of the wall-clock time than the concurrency setting does (finding 3). I'd rather report that than force a bug I couldn't actually reproduce on this version.

Run it yourself

# check which image store is active
docker info | grep -i "storage driver"

# flip to the classic graphdriver (needs root, restarts the daemon)
echo '{"features": {"containerd-snapshotter": false}}' | sudo tee /etc/docker/daemon.json
sudo pkill dockerd && sudo dockerd &

# the quota test — compare exit codes between backends
docker run --rm --storage-opt size=100M alpine:3.20 \
  sh -c "dd if=/dev/zero of=/bigfile bs=1M count=300; echo EXIT=\$?"

# flip back to the containerd snapshotter (Docker 29 default)
echo '{"features": {"containerd-snapshotter": true}}' | sudo tee /etc/docker/daemon.json
sudo pkill dockerd && sudo dockerd &

# pull timing, repeat 3x per backend, always from a clean state
docker rmi -f node:22
start=$(date +%s.%N); docker pull node:22 >/dev/null; end=$(date +%s.%N)
echo "$end - $start" | bc

# the read-only mount check
docker run --rm --mount type=image,source=alpine:3.20,target=/mnt \
  busybox:latest touch /mnt/testfile

If you're running Docker 29 on ext4 or any filesystem without project quota support, and you rely on --storage-opt size anywhere — a shared build host, a CI runner, a multi-tenant container platform — go check which image store you're actually running, and don't trust docker inspect to tell you whether the limit is real. Run the write test above against your own host. It takes ten seconds and it either fills up or it doesn't.

Nile isolates Postgres tenants with a tenant_id column and one session variable

Alex Georgiev — Sat, 12 Sep 2026 08:32:08 +0000

Disclosure: I have no affiliation with Nile. Nobody asked me to write this and nobody paid for it. I signed up for their free tier like anyone else would. I'm on my way to test products and share my findings with the community just in case someone wondered how will this perform and etc.

I created two tenants, put a row in each, then ran the same query without telling the connection which tenant it was. It returned both rows. Turns out that's intentional, documented behaviour, not a bug I'd stumbled onto — but it raised a question worth asking if you're building on this: what actually guarantees your app sets that context every time?

What Nile is

Nile is Postgres reengineered for multi-tenant B2B apps: instead of managing one database per customer, you mark a table as tenant-aware and Nile handles isolating each tenant's rows underneath a single connection string. It's an $11.6M seed company (Benchmark, January 2024, board seat taken by Eric Vishria), founded by Sriram Subramanian and Gwen Shapira, and it's had zero dev.to coverage despite that.

A tenant-aware table needs one thing: a tenant_id uuid column.

CREATE TABLE todos (
  id uuid DEFAULT gen_random_uuid(),
  tenant_id uuid,
  title varchar(256),
  complete boolean,
  PRIMARY KEY (tenant_id, id)
);

Isolation is a session variable:

SET nile.tenant_id = '11111111-1111-1111-1111-111111111111';

Where the isolation actually lives

I created two tenants, set the session to tenant A, inserted two rows, switched to tenant B, inserted one row. Querying as each tenant worked exactly as advertised:

=== query AS tenant A ===
              tenant_id               |    title
--------------------------------------+-------------
 11111111-1111-1111-1111-111111111111 | Acme task 1
 11111111-1111-1111-1111-111111111111 | Acme task 2

=== query AS tenant B ===
              tenant_id               |     title
--------------------------------------+---------------
 22222222-2222-2222-2222-222222222222 | Widget task 1

Then I ran the identical SELECT tenant_id, title FROM todos; on a fresh connection, with nile.tenant_id never set at all:

              tenant_id               |     title
--------------------------------------+---------------
 11111111-1111-1111-1111-111111111111 | Acme task 1
 11111111-1111-1111-1111-111111111111 | Acme task 2
 22222222-2222-2222-2222-222222222222 | Widget task 1

All three rows, both tenants. Nile's own docs confirm this is deliberate: without a tenant context set, a connection can read across all tenants, by design, presumably so admin and migration connections aren't locked out. So the isolation lives in the session rather than the table itself. The open question, for anyone building a pooled-connection app on top of this, is which layer is responsible for making sure SET nile.tenant_id gets called on every request. Worth checking early rather than assuming.

What happens if a write's tenant doesn't match the session

Naturally I tried inserting a row for tenant B while the session was set to tenant A:

SET nile.tenant_id = '11111111-1111-1111-1111-111111111111';
INSERT INTO todos (tenant_id, title, complete)
VALUES ('22222222-2222-2222-2222-222222222222', 'Smuggled into Widget', false);

ERROR:  Multiple tenant IDs specified in write query
DETAIL:  Writes to tenant-aware tables must specify exactly one tenant ID

Rejected outright, with a clear error naming the problem. Good sign — the write path actively checks the session's tenant against the row's tenant.

What it refuses

The tenants table itself is tenant-aware on its own id column, and that column has a rule I didn't expect: it has to be a literal or a bind parameter, not an expression.

INSERT INTO tenants (id, name) VALUES (gen_random_uuid(), 'Acme Corp');

ERROR:  cannot determine tenant ID. Tenant ID must be a constant or a parameter reference (i.e: $1)

A literal UUID works fine; you just can't compute it inline. And deleting from tenants only accepts one shape of WHERE clause:

DELETE FROM tenants WHERE name = 'Acme Corp';

ERROR:  DELETE operations on the tenants table have to provide a 'id'=tenant_id condition. Further conditions are not supported.

WHERE id = '...' is the only accepted filter. I hit both by trial and error, not from anything in the docs I'd read going in.

EXPLAIN doesn't pass through

Every connection to Nile goes through a proxy layer, and I found one command it doesn't forward:

EXPLAIN ANALYZE SELECT * FROM todos;

ERROR:  command tag EXPLAIN unhandled

That rules out the usual first move for debugging a slow query. It also left me curious about something an earlier error had already hinted at: a constraint-violation message named a physical relation todos_200149e, which doesn't show up in pg_class or pg_tables from this connection. So todos isn't stored as one plain table underneath — I just couldn't see the plan or the structure to say more than that.

What you get for it

The one thing that's unambiguously generous: \dx on a fresh free-tier database lists 35 extensions already installed beyond the standard plpgsql every Postgres ships with, including vector and vectorscale (DiskANN) for embeddings, postgis for geospatial, pg_trgm and pg_bigm for text search, and less common ones like h3 and financial. On a stock RDS Postgres instance you'd be enabling most of these one at a time and, for some, not have the option at all.

What I got wrong on the way

My first attempt at inserting a row assumed SET nile.tenant_id would populate the column for me:

SET nile.tenant_id = '11111111-1111-1111-1111-111111111111';
INSERT INTO todos (title, complete) VALUES ('Acme task 1', false);

ERROR:  null value in column "tenant_id" of relation "todos_200149e" violates not-null constraint

It doesn't. The session variable scopes what you can read and enforces what you're allowed to write, but you still have to supply tenant_id explicitly on every insert. That error message is also where the hidden relation name first showed up.

Run it yourself

Free tier, no credit card: sign up at console.thenile.dev, create a database, and grab the connection string from Settings → Connection. A tenants table already exists in every new Nile database, so the INSERTs below work immediately.

psql "$NILE_CONNECTION_STRING" <<'SQL'
CREATE TABLE todos (
  id uuid DEFAULT gen_random_uuid(),
  tenant_id uuid,
  title varchar(256),
  complete boolean,
  PRIMARY KEY (tenant_id, id)
);

INSERT INTO tenants (id, name) VALUES ('11111111-1111-1111-1111-111111111111', 'Acme Corp');
INSERT INTO tenants (id, name) VALUES ('22222222-2222-2222-2222-222222222222', 'Widget LLC');

SET nile.tenant_id = '11111111-1111-1111-1111-111111111111';
INSERT INTO todos (tenant_id, title, complete) VALUES ('11111111-1111-1111-1111-111111111111', 'Acme task 1', false);

SET nile.tenant_id = '22222222-2222-2222-2222-222222222222';
INSERT INTO todos (tenant_id, title, complete) VALUES ('22222222-2222-2222-2222-222222222222', 'Widget task 1', false);
SQL

# now compare a scoped read against an unscoped one
psql "$NILE_CONNECTION_STRING" -c "SET nile.tenant_id = '11111111-1111-1111-1111-111111111111'; SELECT tenant_id, title FROM todos;"
psql "$NILE_CONNECTION_STRING" -c "SELECT tenant_id, title FROM todos;"

If you're evaluating Nile for a real app, the question worth answering early is which layer of your stack is responsible for calling SET nile.tenant_id on every pooled connection before it touches tenant data. That's a question about your own architecture as much as theirs.

Bun 1.4's Rust rewrite cuts script startup time from 10.5ms to 4.2ms

Alex Georgiev — Sat, 12 Sep 2026 08:00:00 +0000

Bun 1.4 shipped on 20 August as the first stable release built on Bun's new Rust-based core, replacing the Zig runtime layer that shipped everything since 1.0. The release notes claim faster startup, less idle CPU, and lower memory. I installed the last pre-rewrite build and the new one side by side and measured all three, plus what happens when you point a node symlink at the new binary, which is how a lot of people actually run Bun in production.

I put Bun 1.3.14 in ~/.bun-old and 1.4.2 in ~/.bun-new using the official install script with a pinned version tag, so both binaries sit on the same machine and I can call either one directly.

Startup time

My first pass measured bun --version twenty times per binary. The difference was almost nothing: 2.24ms median for 1.3.14, 2.00ms for 1.4.2. That command exits before it initialises the JS engine, so it wasn't measuring what the release notes were talking about. Once I switched to actually running a script, the gap showed up.

console.log("hi")

Thirty runs of bun run hello.js each, timed with time.perf_counter() around a subprocess.run call:

	median	min	max
1.3.14	10.54ms	9.32ms	18.00ms
1.4.2	4.22ms	3.90ms	5.25ms

That's a 60% drop in median startup time, better than the "50% faster on Linux" the release notes claim. The old binary also had a much longer tail (up to 18ms), while the new one stayed inside a 1.4ms band across all 30 runs.

Idle memory

I ran a two-line Bun.serve echo server on each version, hit it once to warm it up, then read VmRSS from /proc/<pid>/status after a further 60 seconds of doing nothing. Three runs each:

	RSS after warmup	RSS after 60s idle
1.3.14 (run 1)	34,876 KB	34,960 KB
1.3.14 (run 2)	34,924 KB	34,928 KB
1.3.14 (run 3)	34,872 KB	34,936 KB
1.4.2 (run 1)	18,864 KB	18,920 KB
1.4.2 (run 2)	19,068 KB	19,136 KB
1.4.2 (run 3)	19,056 KB	19,144 KB

Averaging the idle figures: 34.9MB down to 19.0MB, a 46% reduction. The release notes say "up to 35%", so on this workload it beat its own number, though a bare echo server is close to the best case for a claim like this — there's no application memory to dwarf the runtime's own footprint.

Idle CPU: this is where the claim didn't hold up

The release notes claim 5x less idle CPU. This is the one I'd flag as not backed by what I measured.

I sampled utime + stime from /proc/<pid>/stat before and after a 60-second idle window, three times per version (100 clock ticks per second, so each tick is 10ms of CPU time):

run	1.3.14 ticks/60s	1.4.2 ticks/60s
1	11	6
2	11	4
3	10	3
mean	10.67	4.33

That's roughly 0.178% average CPU for 1.3.14 against 0.072% for 1.4.2 — a real improvement, and in the right direction, but a 2.5x reduction, not 5x. Both numbers are small enough that a single bad sample would swing the ratio a lot, which is exactly why I went from a 5-second window (0 to 2 ticks total, useless) to 60 seconds before trusting any of it. It's possible Bun's 5x figure comes from a different idle workload — something with an active file watcher or timer loop rather than a bare Bun.serve with no traffic — but on the simplest possible idle server, I got 2.5x.

Throughput under load

The release notes don't claim a throughput number, so I measured one anyway, since idle numbers only tell you about servers doing nothing. I ran autocannon -c 50 -d 8 against the same echo server, three runs per version:

run	1.3.14 avg req/s	1.4.2 avg req/s
1	29,456	62,310
2	29,578	61,998
3	28,042	62,922
mean	29,025	62,410

That's 2.15x more requests per second at 50 concurrent connections, and p50 latency dropped from 1ms to sub-millisecond in autocannon's output. This one held up better under concurrency than at idle, which is the opposite of what I expected going in — I assumed a rewrite aimed at idle efficiency would show its biggest gains with nothing happening, not under load.

The regression: invoking Bun as `node` silently drops your `.env`

Bun positions itself as a drop-in replacement for Node, and one common way people use that is symlinking a node binary to bun so existing tooling picks it up without changes. I tested exactly that setup.

$ mkdir -p /tmp/nodebin && ln -s /root/.bun-old/bin/bun /tmp/nodebin/node
$ echo 'MY_SECRET=hello123' > .env
$ /tmp/nodebin/node check.js
MY_SECRET=hello123

That's 1.3.14, invoked through a symlink literally named node. It loads .env as expected. Now the same setup against 1.4.2:

$ mkdir -p /tmp/nodebin && ln -s /root/.bun-new/bin/bun /tmp/nodebin/node
$ /tmp/nodebin/node check.js
MY_SECRET=undefined

No error, no warning, exit code 0. The process just runs with MY_SECRET undefined. I checked whether this was specific to the symlink trick or a wider regression: plain bun run check.js and bun --bun run check.js both still load .env correctly on 1.4.2. It's specifically the "binary is named node" detection path that stopped auto-loading environment files.

This matters because "point a node symlink at Bun" is a documented migration path, not an edge case I invented. Anyone running Bun that way in a container image or CI pipeline, expecting Node-compatible behaviour, will get a process that starts cleanly and silently runs with missing configuration. The workaround exists and works:

$ /tmp/nodebin/node --env-file=.env check.js
MY_SECRET=hello123

But you have to know to add it. Nothing in the normal run output tells you your .env was skipped.

What I got wrong on the way

I nearly wrote off the startup claim entirely after my first benchmark. bun --version is a fast path that never spins up the JS engine, so it wasn't exercising the thing the release notes described, and it showed almost no difference between versions. I only found the real 60% gap after switching to a script that actually gets parsed and run. If a "no difference" result comes out of the first thing you try, it's worth asking whether you measured the feature or just the binary's exit path.

The same thing happened with idle CPU: my first sampling window was 5 seconds, which produced 0-2 clock ticks total per run — not enough resolution to say anything. Stretching the window to 60 seconds is what made the numbers usable, and it's also what exposed that the claimed 5x doesn't hold at this scale.

Run it yourself

# install both versions side by side
BUN_INSTALL=~/.bun-old bash -c 'curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.14"'
BUN_INSTALL=~/.bun-new bash -c 'curl -fsSL https://bun.sh/install | bash'

# startup timing
echo 'console.log("hi")' > hello.js
python3 -c "
import subprocess, time, statistics
for label, binpath in [('old', '$HOME/.bun-old/bin/bun'), ('new', '$HOME/.bun-new/bin/bun')]:
    times = []
    for _ in range(30):
        t0 = time.perf_counter()
        subprocess.run([binpath, 'run', 'hello.js'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
        times.append((time.perf_counter() - t0) * 1000)
    times.sort()
    print(label, 'median', round(times[15], 2), 'ms')
"

# the node-symlink .env regression
mkdir -p /tmp/nodebin && ln -sf ~/.bun-new/bin/bun /tmp/nodebin/node
echo 'MY_SECRET=hello123' > .env
echo 'console.log("MY_SECRET=" + process.env.MY_SECRET)' > check.js
/tmp/nodebin/node check.js

If you run Bun 1.4 through anything that presents it as node — a container base image, an nvm-style shim, a CI runner that symlinks a runtime — go and check one thing before you rely on it: put a throwaway variable in your .env, run your actual entrypoint through that symlink, and print the variable back out. If it comes back undefined, you've found the same gap I found, and --env-file on your start command is the fix. The startup and memory gains are worth having either way; I'd rather find out about the missing config here than from a support ticket.

Built anything cool with AI lately?

Alex Georgiev — Fri, 11 Sep 2026 08:42:08 +0000

I train a lot, and with AI's help I built a small app that pulls my Garmin and Strava data and helps me structure my training around it.

Made me curious what else people are building. What's something you've put together recently with AI's help, big or small?

Valkey 9.1's hash field TTL triples memory use for the pattern its own docs show

Alex Georgiev — Fri, 11 Sep 2026 08:00:00 +0000

Valkey 9.0 added the ability to put a TTL on a single field inside a hash, rather than on the whole key. Valkey 9.1 followed up with HSETEX, which sets a field and its expiry in one round trip. The example in Valkey's own blog post is a per-user auth token:

HSETEX user:123 EX 900 FIELDS 1 auth_token "eyJhbGciOiJ..."

That reads like a drop-in replacement for SET user:123:auth_token "..." EX 900. I ran both patterns against 20,000 keys and measured memory with INFO memory. The hash version used three times as much RAM as the plain string.

Setup

I ran Valkey 9.1.2 in Docker on a single host, nothing exotic:

$ docker run -d --name valkey9 -p 6399:6379 valkey/valkey:9.1
$ docker exec valkey9 valkey-server --version
Valkey server v=9.1.2 sha=00000000:0 malloc=jemalloc-5.3.0 bits=64

All measurements below are the fastest of three runs unless noted, using redis-py 8.1.0 talking to the container over the exposed port. For the valkey-cli snippets I aliased valkey-cli to docker exec valkey9 valkey-cli so the commands read the same as they would against a local install.

The headline number

I built three versions of "one item, one expiring value" for 20,000 items: a hash per item with one field carrying a TTL (the pattern in the docs), a hash per item with no TTL, and a plain string key with EXPIRE.

Pattern	Bytes used	Bytes / key	Encoding
Hash per item, field TTL set (`HSETEX`)	5,226,560	261.33	hashtable
Hash per item, no TTL (`HSET`)	1,754,720	87.74	listpack
Plain string + `EXPIRE`	1,733,680	86.68	n/a

Correction (2026-09-12): "Almost exactly the same" overstates it. A reader added the missing control, a hash forced into hashtable with CONFIG SET hash-max-listpack-entries 0 but no TTL, which isolates the encoding switch from the TTL metadata. That put the encoding conversion at about 82 percent of the regression, with the remaining ~18 percent being the TTL bookkeeping itself. I reproduced their numbers independently on a fresh Valkey 9.1.2 instance and got the same split.

Giving a single field a TTL costs almost exactly the same extra memory as switching the whole hash's encoding from listpack to hashtable. It is not a small tax on top of the hash: it roughly triples it, and it lands you back at the same memory cost as the string-based pattern the feature was meant to improve on, except worse, because now you are also paying the hash's own bookkeeping.

I checked encoding directly to confirm this was the cause:

$ valkey-cli DEL h3
$ valkey-cli HSET h3 x 1
$ valkey-cli OBJECT ENCODING h3
listpack
$ valkey-cli HEXPIRE h3 100 FIELDS 1 x
(integer) 1
$ valkey-cli OBJECT ENCODING h3
hashtable

hash-max-listpack-entries was still the default 512 the whole time. It made no difference: the very first field TTL set on a hash forces it out of listpack and into hashtable, regardless of how few fields it has or how small they are. This is a known, tracked gap: Valkey issue #2618 describes exactly this and proposes encoding the expiry inside the listpack itself for small hashes, but as of 9.1.2 that has not landed.

The pattern that actually works

The docs' one-hash-per-item example is the wrong shape for this feature. It is meant for hashes that already hold several fields, where only some carry a TTL, not for a hash that exists solely to wrap one expiring value.

I tested that shape instead: one hash, 20,000 fields, each field set through HSETEX with its own TTL, against the same hash with the same 20,000 fields set without any TTL.

Pattern	Bytes used	Bytes / field	Encoding
One hash, 20,000 fields, each with a TTL	1,563,728	78.19	hashtable
One hash, 20,000 fields, no TTL	1,095,056	54.75	hashtable

Both end up hashtable-encoded regardless, because 20,000 fields is well past the 512-entry listpack ceiling anyway. The difference here, 23.44 bytes per field, is the actual cost of tracking an expiry once you are already paying for hashtable encoding. That lines up with the 16-to-29-byte range the Valkey maintainers cite in the same GitHub issue. Used this way, the feature does what it says: cheap per-field TTLs on a shared structure.

The lesson is really about encoding, not about the TTL feature being expensive. If your hash already has enough fields to sit in hashtable encoding, field TTLs are close to free. If your hash is small enough to want listpack, adding a single TTL field takes that away from you completely.

Reads are not slower

Valkey's blog claims the tracking structure behind field TTLs does not degrade normal hash operations. I benchmarked plain HGET against a 5,000-field hash where no field had a TTL, then against an identical hash where every field did:

Hash	HGET throughput	Latency
No TTL on any field	4,885 ops/s	204.7 us
TTL on every field	4,889 ops/s	204.6 us

Three runs each, same spread. No measurable difference. This is the one claim in the marketing that held up exactly as stated. The per-op latency here is dominated by the network round trip through Docker's port mapping, not by anything server-side, but the comparison is apples to apples and the two numbers are indistinguishable.

What it refuses

A few error paths worth knowing before you hit them in production:

$ valkey-cli SET plainstring hello
OK
$ valkey-cli HEXPIRE plainstring 100 FIELDS 1 f1
(error) WRONGTYPE Operation against a key holding the wrong kind of value

$ valkey-cli HEXPIRE h5 -5 FIELDS 1 a
(error) ERR invalid expire time in 'hexpire' command

$ valkey-cli HEXPIRE h5 100 NX XX FIELDS 1 a
(error) ERR NX and XX, GT or LT options at the same time are not compatible

$ valkey-cli HEXPIRE h5 100 FIELDS 0
(error) ERR wrong number of arguments for 'hexpire' command

HEXPIRE on a field that does not exist, or on a hash that does not exist, returns -2 rather than an error, matching plain TTL's convention. HEXPIRE ... GT against a field that has no TTL (infinite, by convention) is a documented no-op: it returns 0 and leaves the field persistent, because "greater than infinite" can never be true.

Concurrency does not change the picture

I ran 10 threads, 5,000 HSETEX calls each, pipelined in batches of 500, against three layouts: one hash key per thread, one shared hash key for all ten threads, and 5,000 independent string keys per thread with SET ... EX.

Layout	Throughput
One hash per thread	64,800-68,900 ops/s
All threads on one shared hash	61,700-68,000 ops/s
Independent string keys	61,900-62,600 ops/s

Across three runs each, the numbers overlap within noise. I expected the single shared hash to show contention against the per-thread hashes. It did not, because Valkey executes commands on one thread regardless of how many client connections are pushing at it; there is no per-key lock to contend over in the first place. If you were worried that concentrating expiring fields into one big hash creates a hot-key bottleneck under load, this test did not find one at this scale.

Field TTLs survive a restart, and expire on schedule

I set a field TTL, forced a background save, and restarted the container outright rather than trusting DEBUG RELOAD (which this image disables by default under enable-debug-command no):

$ valkey-cli HTTL persisthash FIELDS 1 tok
(integer) 292
$ docker restart valkey9
$ valkey-cli HTTL persisthash FIELDS 1 tok
(integer) 288

The TTL survived the RDB round trip and kept counting down rather than resetting, which is the behaviour you want and not something I'd have bet on without checking.

For timing, I set 2,000 fields in one hash to a 2-second TTL and polled HLEN every 200ms:

t=2.01s hlen=2000
t=2.21s hlen=0

All 2,000 fields disappeared in the same 200ms polling window, not gradually. INFO stats backed this up:

expired_fields:2000
expire_cycle_cpu_milliseconds:3

That is the number to watch in production, not a guess I made. There is no expired_subkeys stat, despite what you might expect from the naming used elsewhere in Redis-family docs; the counter is called expired_fields.

What I got wrong on the way

My first pass at measuring expiry precision checked INFO stats for a field called expired_subkeys, because that's the name I'd seen used informally when this feature was being discussed. It does not exist. r.info('stats').get('expired_subkeys') silently returned None both before and after 2,000 fields expired, and for a few minutes I nearly wrote down "no visible counter for field expiry" as a finding. The actual field is expired_fields, sitting right there in the same INFO stats block. Grepping the raw output instead of asking for one named key by guesswork caught it. It's a good reminder that a None from a stats API is not evidence of absence, it's evidence you guessed the wrong key.

Run it yourself

Start Valkey 9.1 and confirm the encoding transition directly:

docker run -d --name valkey9 -p 6399:6379 valkey/valkey:9.1
docker exec valkey9 valkey-cli HSET h a 1
docker exec valkey9 valkey-cli OBJECT ENCODING h        # listpack
docker exec valkey9 valkey-cli HEXPIRE h 100 FIELDS 1 a
docker exec valkey9 valkey-cli OBJECT ENCODING h        # hashtable

The memory comparison, using redis-py (pip install redis):

import redis, time

r = redis.Redis(host='localhost', port=6399, decode_responses=True)
N = 20000

def used_mem():
    return int(r.info('memory')['used_memory'])

r.flushall(); time.sleep(0.3)
base = used_mem()
pipe = r.pipeline(transaction=False)
for i in range(N):
    pipe.execute_command("HSETEX", f"h:ttl:{i}", "EX", "600", "FIELDS", "1", "tok", "x" * 20)
pipe.execute(); time.sleep(0.3)
print("hash-per-item w/ field TTL:", (used_mem() - base) / N, "bytes/key")

r.flushall(); time.sleep(0.3)
base = used_mem()
pipe = r.pipeline(transaction=False)
for i in range(N):
    pipe.set(f"s:ttl:{i}", "x" * 20, ex=600)
pipe.execute(); time.sleep(0.3)
print("string + EXPIRE:", (used_mem() - base) / N, "bytes/key")

On my run this printed 262.4 bytes/key for the hash version against 85.6 for the string version, matching the table above within normal run-to-run variance.

What to do with this

If you are storing one expiring value per logical item, the string-plus-EXPIRE pattern Valkey has always had is still cheaper than the new hash field TTL commands, because a lone TTL field forces the whole hash into hashtable encoding. Reach for HEXPIRE/HSETEX when you already have a multi-field hash and only some of its fields need to expire; that is the case where the per-field overhead is a genuinely small 16-to-29 bytes and the feature earns its keep. Before you migrate a per-user token cache to the pattern shown in the release notes, run the OBJECT ENCODING check above against your own field sizes and counts, because the answer depends entirely on how many fields already live in that hash.

SQLite 3.53's self-healing index only repairs the rows you write to

Alex Georgiev — Thu, 10 Sep 2026 10:00:00 +0000

I asked a database for the row where a computed bucket equalled 500. It gave me row 309. The correct row, using the same formula, was 308. No error, no warning, just a wrong answer served from an index that no longer matched the function that built it.

This is what SQLite calls a stale expression index, and it's been a known but rarely discussed failure mode for years. SQLite 3.53.0, released in April 2026, added a "self-healing" mechanism aimed at exactly this problem. I built one on purpose, broke it on an old SQLite build, then fixed it on the new one, to see what the fix actually does and does not cover.

How an expression index goes stale

An expression index stores the computed value of an expression, not the raw column. CREATE INDEX idx ON docs(lower(email)) stores lowercased emails in the index tree. If the function behind that expression ever returns a different answer for the same input — a bug fix in a custom function, a library upgrade, or (SQLite's own documented case) a one-ULP shift in floating point conversion between versions — the stored index entries no longer match what the expression would compute today. The row is still there. The index just points at the wrong place, or the wrong bucket doesn't point anywhere.

I couldn't easily reproduce SQLite's own internal float-conversion trigger without hunting for the exact input that shifts by one ULP between two specific point releases, so I reproduced the other documented cause instead: a custom SQL function whose output changes. I registered a scalar function classify(x) that returns floor(x*100), built a table and an expression index on classify(x), then reopened the same database file with a "fixed" version of the function that returns floor(x*100) + 1 — standing in for an app-level bug fix that nobody thought to pair with a REINDEX.

static void classify_func(sqlite3_context *ctx, int argc, sqlite3_value **argv) {
    double x = sqlite3_value_double(argv[0]);
    long bucket = (long)(x * 100.0);
    if (g_variant == 1) bucket += 1;   /* the "fixed" version */
    sqlite3_result_int64(ctx, bucket);
}

Both versions are registered SQLITE_DETERMINISTIC, which is exactly what SQLite's docs warn against doing carelessly: it lets the engine trust the index without re-checking.

I built and ran two SQLite builds compiled from the official amalgamation, so I could isolate the version as the only variable: 3.51.0 (before self-healing existed) and 3.53.4, the latest release as of this test.

The wrong answer, on the old build

With 1,000 rows and the index built under the old function, I reopened the database with the "fixed" function and asked for classify(x) = 500:

indexed (INDEXED BY idx_classify) ids for classify(x)=500: (id=309 x=5.004000)
full scan (NOT INDEXED) ids for classify(x)=500: (id=308 x=4.991000)

Two different rows, same query, same data, same function. The index path used the b-tree key that was written when the old function built it; the scan path recomputed the expression fresh. Nothing in the indexed query's result told me it was wrong.

Then I tried writing to the affected rows on SQLite 3.51.0:

update_rc=11 err=database disk image is malformed (range 1..50)

11 is SQLITE_CORRUPT. That is the honest, current behaviour of every SQLite release before 3.53: a stale expression index can turn an ordinary UPDATE into a corruption error, on a database that isn't actually corrupt in any other sense.

What 3.53.4 does instead

Same database, same broken function pairing, but opened with the 3.53.4 build. The pre-write query showed the identical wrong answer (309 instead of 308) — self-healing does not run on reads, only on writes. Then I ran an UPDATE touching id 300 through 320, a range that happens to include both 308 and 309:

update_rc=0 err=none (range 300..320) time=0.0009

No error. Querying classify(x) = 500 again afterwards, both the indexed path and the full scan now agreed on row 308. That part of SQLite's claim held up: "no errors are raised, the application never knows that something was ever amiss" is literally what I observed.

What I didn't expect is how narrow the fix is. Before that UPDATE, an integrity check with the row limit raised to 5,000 counted every stale entry in the table:

State	Stale index entries (of 1,000 rows)
Before any write	1,000
After `UPDATE` on rows 300–320 (21 rows)	979
After `REINDEX EXPRESSIONS`	0

Self-healing repairs exactly the rows a write statement touches, and nothing else. The other 979 rows stayed just as wrong as they were on 3.51.0 — they simply hadn't been written to. A read-heavy table with a stale expression index could sit there returning wrong answers indefinitely on 3.53, with no error and no write ever forcing a fix.

REINDEX EXPRESSIONS is the actual fix, and it's fast

REINDEX EXPRESSIONS is new in 3.53 too: it rebuilds only expression indexes, skipping ordinary column indexes on the same table. On 3.51.0 the statement doesn't parse as a special form at all:

reindex_expr_rc=1 err=unable to identify the object to be reindexed

SQLite 3.51 reads EXPRESSIONS as the name of an index or table to reindex, finds nothing by that name, and fails with that message. On 3.53.4 it's a real command, and it healed all 979 remaining rows in my 1,000-row table instantly.

To see whether "only touches expression indexes" is actually worth anything, I built a 500,000-row table with four indexes — one expression index on classify(x), three ordinary ones on other columns — and timed REINDEX EXPRESSIONS against a full REINDEX of the table, fastest of three runs each:

Operation	Time (fastest of 3)
`REINDEX EXPRESSIONS` (1 of 4 indexes)	0.198s
`REINDEX t` (all 4 indexes)	0.990s

About five times faster, roughly proportional to touching one index instead of four. If a table has several plain indexes alongside one suspect expression index, this is the difference between a maintenance job that runs in the background and one that locks the whole table for a full rebuild.

What self-healing costs you on every write

Since 3.53 quietly rewrites stale entries as it goes, I wanted to know if that costs anything on writes that don't need it. I built two 200,000-row tables — one where the index was already stale relative to the current function, one where it matched from the start — then ran the identical single-statement UPDATE touching every row, fastest of three runs:

Table state	Update time (fastest of 3)
Index already consistent	0.164s
Index stale, every row healed	0.239s

Healing added roughly 46% to that statement's time, or about 0.4 microseconds per row. That's a real cost, not a rounding error, though it's a one-time tax paid the first time each row is touched after the function changes — after that the row is clean and stays clean.

Under concurrent writers, nothing new happens

I opened five separate connections against the same stale database, each running an UPDATE on an overlapping range of ids, with no busy_timeout set:

update_rc=5 err=database is locked (range 801..2000)
update_rc=5 err=database is locked (range 1601..3000)
update_rc=5 err=database is locked (range 2401..4000)
update_rc=0 err=none (range 1..1000)
update_rc=0 err=none (range 3201..5000)

Three of five got SQLITE_BUSY immediately, two succeeded. That's just SQLite's ordinary single-writer lock, the same thing that happens with any concurrent write and no timeout configured. Self-healing didn't introduce a new failure mode here, and the two successful writes healed exactly the rows in their ranges — the remaining stale count matched what I'd expect from simple arithmetic on which ranges got through. I didn't find anything specific to self-healing under contention worth a longer write-up.

What I got wrong on the way

My first pass at the "wrong answer" query used bucket 500 arbitrarily without checking my data's range, and separately I first tried bucket 50 — which was below the minimum value my generated x column could ever produce, so both the indexed and scan queries correctly returned zero rows. That looked like "no staleness detected" and briefly convinced me my repro wasn't working, before I checked the actual range of x*100 in my dataset and picked a bucket inside it.

The second mistake was quieter. I wrote a separate small program to run PRAGMA integrity_check(5000) for a precise count, and it reported zero problems on a database I already knew was full of them. I'd forgotten to register the classify function on that connection before running the check. SQLite apparently treats an index expression it can't evaluate as something to skip rather than something to flag, so the check silently passed over the one index I cared about. Once I registered the function on that connection too, the count came back as 1,000, matching everywhere else.

Correction (2026-09-11): The paragraph above is wrong, a reader's reproduction on 3.53.4 caught it. With classify unregistered, PRAGMA integrity_check does not skip the index, sqlite3_prepare_v2 fails outright with unknown function: classify(). My separate checker program never checked that return code, so the check never actually ran, and the zero problems I reported came from nothing being counted, not from SQLite skipping anything. A safeguard has to confirm the check actually executed, not just trust a clean result.

Run it yourself

This needs a C compiler and the official amalgamation source, no Docker required.

mkdir sqlite-test && cd sqlite-test
curl -sSL -o new.zip https://sqlite.org/2026/sqlite-amalgamation-3530400.zip
curl -sSL -o old.zip https://sqlite.org/2025/sqlite-amalgamation-3510000.zip
unzip -q new.zip && unzip -q old.zip

Save this as harness.c (trimmed to the create/query/update paths used above):

#include "sqlite3.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static int g_variant = 0;

static void classify_func(sqlite3_context *ctx, int argc, sqlite3_value **argv) {
    double x = sqlite3_value_double(argv[0]);
    long bucket = (long)(x * 100.0);
    if (g_variant == 1) bucket += 1;
    sqlite3_result_int64(ctx, bucket);
}

int main(int argc, char **argv) {
    const char *dbpath = argv[1], *mode = argv[2];
    g_variant = atoi(argv[3]);
    int n = argc > 4 ? atoi(argv[4]) : 1000;
    sqlite3 *db; sqlite3_open(dbpath, &db);
    sqlite3_create_function(db, "classify", 1, SQLITE_UTF8 | SQLITE_DETERMINISTIC,
                             NULL, classify_func, NULL, NULL);
    char *err = NULL;
    if (!strcmp(mode, "create")) {
        sqlite3_exec(db, "CREATE TABLE t(id INTEGER PRIMARY KEY, x REAL)", 0, 0, &err);
        sqlite3_exec(db, "CREATE INDEX idx_classify ON t(classify(x))", 0, 0, &err);
        sqlite3_exec(db, "BEGIN", 0, 0, &err);
        sqlite3_stmt *stmt;
        sqlite3_prepare_v2(db, "INSERT INTO t(x) VALUES (?)", -1, &stmt, 0);
        for (int i = 0; i < n; i++) {
            sqlite3_bind_double(stmt, 1, 1.0 + i * 0.013);
            sqlite3_step(stmt); sqlite3_reset(stmt);
        }
        sqlite3_finalize(stmt);
        sqlite3_exec(db, "COMMIT", 0, 0, &err);
    } else if (!strcmp(mode, "query")) {
        int bucket = n;
        char sql[256]; sqlite3_stmt *stmt;
        snprintf(sql, sizeof sql, "SELECT id,x FROM t INDEXED BY idx_classify WHERE classify(x)=%d", bucket);
        sqlite3_prepare_v2(db, sql, -1, &stmt, 0);
        while (sqlite3_step(stmt) == SQLITE_ROW)
            printf("indexed: id=%d x=%f\n", sqlite3_column_int(stmt,0), sqlite3_column_double(stmt,1));
        sqlite3_finalize(stmt);
        snprintf(sql, sizeof sql, "SELECT id,x FROM t NOT INDEXED WHERE classify(x)=%d", bucket);
        sqlite3_prepare_v2(db, sql, -1, &stmt, 0);
        while (sqlite3_step(stmt) == SQLITE_ROW)
            printf("scan:    id=%d x=%f\n", sqlite3_column_int(stmt,0), sqlite3_column_double(stmt,1));
        sqlite3_finalize(stmt);
    } else if (!strcmp(mode, "update")) {
        int lo = n, hi = argc > 5 ? atoi(argv[5]) : n;
        char sql[256];
        snprintf(sql, sizeof sql, "UPDATE t SET x=x WHERE id BETWEEN %d AND %d", lo, hi);
        int rc = sqlite3_exec(db, sql, 0, 0, &err);
        printf("update_rc=%d err=%s\n", rc, err ? err : "none");
    }
    sqlite3_close(db);
}

Build both versions and reproduce the wrong answer, then the fix:

gcc -O2 -I sqlite-amalgamation-3510000 harness.c sqlite-amalgamation-3510000/sqlite3.c \
    -lpthread -ldl -lm -o harness_old
gcc -O2 -I sqlite-amalgamation-3530400 harness.c sqlite-amalgamation-3530400/sqlite3.c \
    -lpthread -ldl -lm -o harness_new

rm -f test.db
./harness_old test.db create 0 1000      # build the index with the "buggy" function
./harness_old test.db query 1 500        # variant 1 = the "fixed" function: wrong answer
./harness_old test.db update 1 1 50      # SQLITE_CORRUPT on the old build

rm -f test.db
./harness_new test.db create 0 1000
./harness_new test.db update 1 300 320   # succeeds silently on 3.53.4
./harness_new test.db query 1 500        # now correct, because 308/309 got touched

What to do about it

If you have expression indexes on top of application-defined functions or extensions, treat a fix to that function the same way you'd treat a schema migration: run REINDEX EXPRESSIONS right after deploying it. Don't rely on self-healing to catch up on its own — it only touches rows your application happens to write to, and a read-mostly table can carry silently wrong answers for as long as you let it.

Run PRAGMA integrity_check on any database where you suspect this, but register every custom function the schema depends on before you do — mine reported zero problems the one time I forgot, on a database with a thousand stale rows. And if you're still on a version before 3.53, know that the current failure mode for a stale expression index isn't a wrong answer, it's SQLITE_CORRUPT on write.

nginx silently rejects the new HTTP QUERY method

Alex Georgiev — Wed, 09 Sep 2026 21:11:00 +0000

RFC 10008 went to Proposed Standard in June. It adds QUERY, a new HTTP method. Safe and idempotent like GET, but it carries a body like POST. On paper, that's it. A new verb.

I nearly didn't bother writing this up because of that. Then I read further into the RFC and found a line saying older proxies, frameworks and load balancer configs might not recognise the method yet. It doesn't say which ones. It doesn't say what "not recognise" even means in practice. Does it 404? 405? Does it just eat the body and treat it as GET? Nobody tells you, so I rented a box and found out myself.

Most codebases I've touched have a POST /search somewhere, and it's always for the same reason: GET can't carry a real filter object, and nobody fancies fighting a URL length limit over it. QUERY fixes that, in theory. Whether it works in practice depends on every layer between the client and your view agreeing to let the method through. That's not something the RFC can tell you. Only running it can.

So that's what I did. One backend, three reverse proxies in front of it, one separate Django app on the side, and a droplet I could throw away the second I had numbers.

What I built

A FastAPI backend on port 8001. nginx, Caddy and Traefik each fronting it on their own port. A separate Django project too, with two class-based views, because Django isn't built on Starlette and dispatches methods completely differently. All of it on one DigitalOcean Droplet in Frankfurt, fra1, s-2vcpu-4gb, Ubuntu 24.04. I killed the droplet the moment testing was done.

Versions, in case you're checking this later: curl 8.5.0. FastAPI 0.141.1 on Starlette 1.6.0. Django 6.1.1. nginx 1.24.0. Caddy 2.11.4. Traefik 3.7.10.

curl already does this properly

First question, before anything else: can the tooling even send a QUERY request with a body? I pointed curl at a bare netcat listener to see the raw bytes.

curl -s -m 2 -X QUERY -H "Content-Type: application/json" \
  -d '{"q":"test"}' http://127.0.0.1:8000/search

QUERY /search HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/8.5.0
Accept: */*
Content-Type: application/json
Content-Length: 12

{"q":"test"}

No special flag. No workaround. curl just sends whatever method you give it through -X, body and all. It didn't need anything. That's one layer down.

FastAPI

I gave it a route with methods=["QUERY"]:

@app.api_route("/search", methods=["QUERY"])
async def search(request: Request):
    body = await request.body()
    return JSONResponse({"received_method": request.method, "body": body.decode()})

That worked. 200, body echoed back. Not surprising, since I'd told it to expect QUERY. What I actually wanted to know was what happens on a route I hadn't touched. So I sent the same request to /docs, which only has GET on it:

HTTP/1.1 405 Method Not Allowed
allow: GET, HEAD
content-type: application/json

{"detail":"Method Not Allowed"}

Correct Allow header, listing GET and HEAD. Nothing broke, and I'll say that plainly since most of this post is about things that did break. But it also means QUERY doesn't just piggyback on your GET handler. You want ten routes to answer QUERY, that's ten edits. Not a flag, not a setting.

Django: I wrote a handler, and it still said no

Django's class-based views dispatch by looking up a lowercased method name. get for GET. post for POST. So query should work for QUERY the same way. I wrote one:

class SearchView(View):
    def query(self, request, *args, **kwargs):
        return JsonResponse({"received_method": request.method,
                              "body": request.body.decode()})

HTTP/1.1 405 Method Not Allowed
Allow: OPTIONS
Content-Length: 0

Refused. Empty body. An Allow header that doesn't even mention the method I just wrote a handler for, and only lists OPTIONS, because OPTIONS is the one method this class gets automatically and I hadn't implemented get or post on it either. Here's why it happened: View.http_method_names is a hardcoded list, and Django checks the request against it before it ever looks at your class.

>>> from django.views import View
>>> View.http_method_names
['get', 'post', 'put', 'patch', 'delete', 'head', 'options', 'trace']

No query in there. Doesn't matter that I wrote the method. Django never gets that far. One line fixes it, but you have to know to write it:

class SearchViewFixed(View):
    http_method_names = View.http_method_names + ["query"]
    def query(self, request, *args, **kwargs):
        return JsonResponse({"received_method": request.method,
                              "body": request.body.decode()})

HTTP/1.1 200 OK
Content-Type: application/json

{"received_method": "QUERY", "body": "{\"q\":\"hello\"}"}

Same handler. Same request. The only change is telling the class it's allowed to answer at all. This is the sharp edge of that RFC warning I mentioned. It doesn't crash. It just 405s, looking exactly like a typo in your URL, and the traceback won't point you anywhere near the real fix.

nginx, and the config pattern that breaks it

Plain proxy_pass, nothing restricting methods:

location / {
    proxy_pass http://127.0.0.1:8001;
}

HTTP/1.1 200 OK
{"received_method":"QUERY","body":"{\"q\":\"hello\"}"}

Works fine. Then I added limit_except, the block that turns up in a huge share of nginx hardening guides, restricting a location to only the methods it's supposed to need:

location / {
    limit_except GET POST HEAD {
        deny all;
    }
    proxy_pass http://127.0.0.1:8001;
}

HTTP/1.1 403 Forbidden

Flat. No explanation. Never even reaches the backend. And to be fair to nginx: the default was fine two paragraphs ago. This isn't nginx's fault, it's a hardening snippet copied into configs for years, written back when GET, POST and HEAD covered every method a location would ever need. Whoever wrote it had no reason to think about a verb that didn't exist yet. If your config has limit_except anywhere in it, go look at what's on that list right now.

Caddy and Traefik: nothing to report

I expected at least one of these to have an opinion about a method it didn't recognise. Neither did.

# Caddy, default reverse_proxy, no config changes
HTTP/1.1 200 OK
Via: 1.1 Caddy
{"received_method":"QUERY","body":"{\"q\":\"hello\"}"}

# Traefik, default file-provider router, no config changes
HTTP/1.1 200 OK
{"received_method":"QUERY","body":"{\"q\":\"hello\"}"}

They just pass whatever the client sends. No allowlist to trip over. Nothing to configure. Caddy and Traefik are also the two newer tools of the three here, which fits the RFC's "older tooling" line better than I expected going in.

Under load, and in the logs

Ten concurrent QUERY requests through nginx's plain proxy, all 200. No concurrency surprise there. The access log picked them up cleanly too, no config change needed:

127.0.0.1 - - [09/Sep/2026:19:23:13 +0000] "QUERY /search HTTP/1.1" 200 51 "-" "curl/8.5.0"

Request line, status, size, all where you'd expect them. If you're watching this in production already, your log pipeline already sees it fine. The gap isn't observability. It's earlier, at the config and framework layer, before the request even gets that far.

What I got wrong

My first Traefik config wouldn't load. The error wasn't helpful: yaml: line 4: found unknown escape character. I'd written the router rule as one long string passed straight through an SSH command, and the backtick in PathPrefix(`/`) got mangled by an extra layer of shell escaping I hadn't planned for. Writing the same YAML through a quoted heredoc over ssh ... bash -s, instead of one inline string, fixed it straight away. That one was on me, not Traefik.

Run it yourself

Five checks, that's all of this. Every command below is exactly what I ran, against a backend already listening on port 8001.

# 1. curl sends QUERY with a body natively
curl -s -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8001/search

# 2. Explicit FastAPI route works; unregistered route doesn't
curl -s -i -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8001/search
curl -s -i -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8001/docs

# 3. Django: default View rejects it, extended http_method_names accepts it
curl -s -i -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8002/search/
curl -s -i -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8002/search-fixed/

# 4. nginx: plain proxy_pass works, limit_except blocks it
curl -s -i -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8010/search   # plain
curl -s -i -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8011/search   # limit_except

# 5. Caddy and Traefik, both unmodified
curl -s -i -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8020/search
curl -s -i -X QUERY -d '{"q":"hello"}' http://127.0.0.1:8030/search

The Django fix is that one http_method_names line, plus the query method itself. The nginx fix is either drop limit_except, or add QUERY to its list by hand.

What to do about it

Don't test this on a clean install of anything. I did, here, and it made the whole thing look easier than it will be on a real system. Grep your actual nginx config for limit_except first. That one line is what decided whether QUERY got anywhere near my backend at all. On Django, check http_method_names on the actual views you'd be changing, not some throwaway subclass.

Going in, I expected the framework side to be the messy one. I came out thinking the proxy layer is worse, just because it's older, more copied from tutorial to tutorial, and less likely to get a second look before this breaks on someone. curl got QUERY right before I'd written a single line of my own code. A five year old nginx snippet didn't.

That's the whole shape of it. The method itself is sound, and that POST /search I mentioned at the start really can become a QUERY without changing what it does. It just can't skip the audit of everything sitting in front of it first.

What is your daily routine?

Alex Georgiev — Mon, 10 May 2021 13:32:47 +0000

I would like to hear what is everyone doing during WFH (Working from Home). Have you established some goals for each day, for example, to exercise or to read a book, take small breaks each hour?

Do you have a to-do list that you follow each day of the week or you just try to stay productive but do not follow a routine?

DEV Community: Alex Georgiev

MongoDB 8.2's encrypted substring search costs 20 times more per insert than equality

Setup

What it costs

What it refuses

Query latency

What I got wrong on the way

Watching it in production

Run it yourself

Tigris marks every read with the region and cache layer that actually served it

What Tigris is

Global by default, pinnable if you want it

What the read path actually reveals

What it gets wrong

The zero-egress claim

Run it yourself

If you're weighing other object storage

Redis 8.10's compact hashes need HIMPORT or a restart to kick in

The straightforward result

The number that didn't hold up

Plain HSET gets nothing, ever, until you restart

Small numbers of shared keys make things worse, not better

What happens if you mutate a templated hash

Refusals and connection scoping

Concurrency behaved better than I expected

What I got wrong on the way

Run it yourself

What to do with this

Django 6.1's FETCH_PEERS collapses a 2,001-query loop into 2

The setup

The headline number

What it refuses: the many side

What it rejects outright

The gotcha nobody warns you about

What else it costs

What I got wrong on the way

Run it yourself

What to do with this

Docker Engine 29's default image store lets --storage-opt size fail silently

The setup

Finding 1: the quota flag stops erroring, and stops working

Finding 2: pulls are genuinely faster, independent of why

Finding 3: the concurrency knob barely matters, which wasn't what I expected

Finding 4: the new mount type does what it says, mostly

Finding 5: dedup accounting still works

What I got wrong on the way

Run it yourself

Nile isolates Postgres tenants with a tenant_id column and one session variable

What Nile is

Where the isolation actually lives

What happens if a write's tenant doesn't match the session

What it refuses

EXPLAIN doesn't pass through

What you get for it

What I got wrong on the way

Run it yourself

Bun 1.4's Rust rewrite cuts script startup time from 10.5ms to 4.2ms

Startup time

Idle memory

Idle CPU: this is where the claim didn't hold up

Throughput under load

The regression: invoking Bun as node silently drops your .env

What I got wrong on the way

Run it yourself

Built anything cool with AI lately?

Valkey 9.1's hash field TTL triples memory use for the pattern its own docs show

Setup

The headline number

The pattern that actually works

Reads are not slower

What it refuses

Concurrency does not change the picture

Field TTLs survive a restart, and expire on schedule

What I got wrong on the way

Run it yourself

What to do with this

SQLite 3.53's self-healing index only repairs the rows you write to

How an expression index goes stale

The wrong answer, on the old build

What 3.53.4 does instead

The regression: invoking Bun as `node` silently drops your `.env`