MongoDB Search and Vector Search for Community Edition went GA on 30 June 2026. Self-managed deployments get the same mongot search process that used to be Atlas-only, running alongside mongod instead of bolted onto a separate engine. I followed MongoDB's own Docker installation guide word for word, and the search index never left PENDING.
Setting it up
The docs describe two containers on one network: mongod (MongoDB Community Server 9.0.2) and mongot-community (mongot 1.70.5), joined by a gRPC connection and a SCRAM user with the searchCoordinator role. I ran it exactly as documented: a single-member replica set, a mongod.conf pointing at the search host, a mongot.conf pointing back at mongod, then createSearchIndex on a 50,000-document test collection.
The index came back immediately, as expected:
[ { name: 'default', status: 'PENDING', queryable: false } ]
PENDING is normal for the first few seconds. It was not normal five minutes later, still retrying every 30 seconds with the same message in the mongot log:
"msg":"Transient error while trying to sync ... Retrying in 30000 milliseconds"
"stack_trace":"... PauseInitialSyncException: Initial syncs are paused ..."
Finding the actual cause
mongot exposes Prometheus metrics on port 9946, and one of them settled it:
mongot_system_disk_space_data_path_free_bytes 2.8686217216E10
mongot_system_disk_space_data_path_total_bytes 2.70553174016E11
28.7GB free out of 270.6GB, which is 89.4% used. MongoDB's own self-managed troubleshooting page states the rule directly: "Replication stops when disk usage exceeds roughly 90% and resumes after usage drops below roughly 85%. For a new index or rebuild, expect the definition to be accepted but the build to stay stuck if disk pressure is already above the protective threshold." My host had drifted into that band and the index was never going to move.
Nothing about this showed up on the client side. createSearchIndex returned normally, $listSearchIndexes just said PENDING, and df -h on the same path reported 28% used, not 89%, because df was measuring against a quota-limited allowance while mongot was dividing by the underlying device's full reported capacity. Two tools looking at the same disk gave answers 60 points apart.
I moved mongot's data directory onto tmpfs to get a mount with headroom, and the same index went from PENDING to serving queries in under five seconds:
"msg":"Finished a collection scan phase.","attr":{"numDocumentsIndexed":50000}
"msg":"Completed initial sync. Beginning first commit.","attr":{"duration":"4.850 s", ...}
"msg":"Transitioning from INITIAL_SYNC to STEADY_STATE."
The block was real and the fix was disk headroom, nothing else.
A status field that keeps lying
After I recreated the mongot container, $listSearchIndexes kept reporting default as PENDING even though queries against it worked. The full statusDetail array explained why: it still listed the old, now-dead mongot host as PENDING alongside the new host, which was READY. The aggregated top-level status is the worst of all hosts it has ever seen, including ones that no longer exist. A vector index I created fresh, with no dead host in its history, reported READY cleanly. If you restart mongot in place, don't trust the summary status; read statusDetail per host.
What it refuses, and what it doesn't
Querying an index that exists but isn't ready yet is a hard, specific error:
OperationFailure: cannot query search index ... while in state NOT_STARTED (code 8)
Querying an index name that doesn't exist at all is not an error. It returns zero results silently, with only a WARN "No index in catalog" line in mongot's own log, invisible to the client. A typo in an index name and a genuinely empty result set look identical from the application side.
The speed comparison I expected to favour $search
Once the index was live, I ran 15 repetitions each of five single-word and two-word queries against $search, a classic MongoDB $text index, and a plain case-insensitive regex, all on the same 50,000-document collection.
| Method | min | median | p95 | max |
|---|---|---|---|---|
$search (mongot) |
6.25ms | 9.36ms | 15.43ms | 56.05ms |
classic $text index |
0.48ms | 0.67ms | 0.89ms | 1.57ms |
| regex scan | 0.48ms | 0.79ms | 2.86ms | 3.95ms |
For a plain term lookup, the new search stack was 10 to 14 times slower than the mechanism it's positioned to replace. That's the gRPC round trip to a separate process rather than an in-process index scan, and it's not a reason to avoid mongot — it's a reason not to switch to it for cases the old $text index already handled well.
Where $search earns the extra hop is fuzzy matching. I queried all four methods with backpak, a one-letter typo for backpack:
| Method | results for "backpak" |
|---|---|
$search with fuzzy: {maxEdits: 1}
|
5 |
$search, no fuzzy option |
0 |
classic $text index |
0 |
| regex | 0 |
Only the fuzzy option found anything. That's the actual capability being sold here, and it worked as documented.
Vector search against doing it by hand
I ran $vectorSearch against a 32-dimension cosine-similarity index over the same 50,000 documents, and compared it to scoring every vector in Python after fetching them once:
| Method | min | median | max |
|---|---|---|---|
$vectorSearch |
10.38ms | 15.52ms | 93.93ms |
| brute-force Python cosine scan | 123.23ms | 134.89ms | 243.87ms |
$vectorSearch was roughly 8.7 times faster at the median, and that's generous to the brute-force side: I excluded the time to fetch all 50,000 embeddings over the wire, which a real application doing this in-process would also have to pay unless it already held every vector in memory.
Concurrency and cost
Ten concurrent clients issuing $search queries finished 20 total queries in 98.6ms of wall time, against 180.7ms for one client doing the same 20 sequentially — a real throughput gain. But per-query latency degraded: the median went from 8.41ms to 28.77ms, and the worst case rose from 13.07ms to 83.40ms. mongot is a shared, single-threaded-ish gRPC service from the client's point of view, and concurrency buys you overall throughput at the cost of tail latency.
I also checked docker stats while both containers sat idle, with nothing but two small indexes over 50,000 short documents behind them. mongod held 229.5MiB resident. mongot held 1.033GiB, over four times as much, and it hadn't answered a single query yet. Anyone budgeting a self-managed box for mongod alone needs to add a second, JVM-sized line item for mongot, and that line item doesn't shrink if the collection is small.
MongoDB's launch post says $search runs on "the same query model you already know." I wanted to see if that survived contact with a real pipeline rather than a one-line example, so I chained $match, $project, $sort, {$meta: "searchScore"} and $limit after a $search stage. It worked without any special-casing, filtering on price after the text match and sorting on the filtered set, which is a normal thing to want and not something the docs show directly.
What I got wrong on the way
The sync config in the docs sets readPreference: secondaryPreferred, and when the index sat in PENDING for the first few minutes I decided that was the problem: a single-node replica set with no secondary to prefer. So I built one. I added a second mongod container, ran rs.add(), watched rs.status() report a healthy SECONDARY, and waited for the index to move. It didn't. Another five minutes of the same 30-second retry loop passed before I gave up on that theory and went looking at mongot's metrics instead, where the disk numbers had been sitting the entire time. df -h on the same host never once suggested there was a problem.
Run it yourself
This needs Docker with a working daemon, not just the CLI.
docker network create search-community
# mongod.conf: net.bindIpAll true, replication.replSetName rs0,
# plus the setParameter block pointing at mongot-community:27028
docker run -d --name mongod --network search-community \
--network-alias mongod.search-community \
-v "$PWD/mongod.conf:/etc/mongod.conf:ro" \
-v "$PWD/data/db:/data/db" -p 27017:27017 \
mongodb/mongodb-community-server:latest --config /etc/mongod.conf
docker exec mongod mongosh --port 27017 --quiet --eval '
rs.initiate({_id:"rs0", members:[{_id:0, host:"mongod.search-community:27017"}]})'
docker exec mongod mongosh --port 27017 --quiet --eval '
db.getSiblingDB("admin").createUser({user:"mongotUser", pwd:"testpass123", roles:["searchCoordinator"]})'
# mongot.conf: syncSource pointing at mongod, storage.dataPath /data/mongot
docker run -d --name mongot-community --network search-community \
--network-alias mongot-community.search-community \
-v "$PWD/data/mongot:/data/mongot" \
-v "$PWD/mongot.conf:/mongot-community/config.default.yml:ro" \
-v "$PWD/passwordFile:/passwordFile:ro" \
-p 8080:8080 -p 9946:9946 \
mongodb/mongodb-community-search:latest
curl localhost:8080/health # expect {"status":"SERVING"}
curl localhost:9946/metrics | grep disk_space_data_path # check your own headroom first
Check your disk metric before you file a bug: (total - free) / total from mongot_system_disk_space_data_path_total_bytes and _free_bytes, not df -h. If you're above roughly 85%, free space before you touch the index definition.
What to do with this
Before you create a first index on a self-managed deployment, pull mongot_system_disk_space_data_path_free_bytes and do the arithmetic yourself rather than trusting df on the same box; the two disagreed by 60 percentage points on my host, and only one of them is the number mongot actually acts on. Don't rip out a working $text index or a regex query just because $search is newer — on plain term lookups mine was an order of magnitude slower. Reach for it when you need fuzzy matching or relevance scoring you can't already get, and reach for $vectorSearch once scoring vectors in application code starts to hurt, which on my 50,000-row test happened well before the collection felt large. If you restart mongot, open statusDetail before you believe whatever the top-level queryable field tells you.
Top comments (18)
The two-tools-one-disk gap (df 28% vs mongot 89%) is the part that will bite everyone: mongot enforces its own math, so the alert should too - a recording rule on mongot_system_disk_space_data_path_free_bytes / total_bytes catches the drift before any index build wedges, while df on a quota-limited overlay never will. Same asymmetry in the API: NOT_STARTED is a hard error but a typo'd index name is a silent zero, so the cheap guard is asserting the index exists via $listSearchIndexes at startup - turns 'identical from the application side' into a boot-time failure. And the worst-of-all-hosts-ever aggregation is really append-only truth without decay: a statusDetail-vs-actual-members diff is a one-line drift check on it.
The recording rule on mongot_system_disk_space_data_path_free_bytes/total_bytes, yeah that's just... correct, and kind of embarrassing that I didn't say it outright. df was never gonna save anyone here, it's not even doing the same math mongot is doing, so alerting on it is basically alerting on vibes. Two tools, two different denominators, good luck.
The $listSearchIndexes boot check for the typo thing is the one I keep thinking about honestly. "Looks identical from the app side" was the whole annoying part of that finding, like, a typo and an empty collection give you the exact same nothing back, and the only hint is a WARN line buried in mongot's own logs that basically nobody's tailing at 2am. Turning that into a startup crash instead of a silent shrug is just... obviously better? Not sure why I didn't land there myself, might've been too deep in "huh that's weird" mode to go write the fix.
And the statusDetail-vs-members diff, yep, would've saved me some genuine head scratching with the dead-host thing. The fact that the aggregate status is "worst thing this index has EVER seen, including hosts that are long gone" is such a sneaky little trap, a fresh index with no baggage reports clean no problem, so you'd never suspect the pattern exists until the one day you restart mongot and the summary just straight up lies to you from then on.
Anyway good comment, you basically did the "what would you actually build" homework I skipped. Appreciate it.
The two-denominator trap is worth naming explicitly for the post: df and mongot disagree because they measure different things — df sees the filesystem, mongot sees its data path, and on XFS with speculative preallocation the gap is gigabytes. The recording rule on mongot's own metric is the only alert that means what it says. On the $listSearchIndexes boot check: it's three lines and it catches the typo class that looks identical from the app side — cheap enough that skipping it is the actual bug.
Small correction on the denominator bit, cause I don't want it to stick around in the thread wrong. There's no XFS or speculative preallocation in the post, and that's not where the gap came from. Both tools were looking at the same path. df was dividing by the quota-limited allowance, mongot was dividing by the full capacity the underlying device reports. That's how you get 28% vs 89% on one disk, a 60 point gap, way bigger than preallocation would ever explain.
Have you actually seen preallocation throw mongot's numbers off on XFS? If so I'd be curious, that'd be a separate trap on top of this one.
Agree on the boot check though. It's a few lines, and skipping it is how you end up debugging an empty result that was really a typo.
You're right and thanks for correcting it before it sticks — I reached for XFS speculative preallocation because it's a classic 'two tools disagree' story, but it wasn't in your post and the quota-vs-capacity split is the actual mechanism, and it's cleaner: both tools read the same path, just against different denominators. df tracks the quota-limited allowance, so it's the one that screams early, while mongot keeps reporting comfortable numbers against full capacity — which also explains why mongot's own metric never fires an alert there. To your question: no, I haven't seen preallocation move mongot's number; speculative preallocation inflates apparent usage at the filesystem level, so it would bias df upward, not mongot. The general lesson holds though: when two measurements of the same disk disagree by tens of points, check the denominators first — quota, cgroup limits, reserved blocks — before suspecting either tool.
“Alerting on vibes” is the reframe I’ll reuse with teams — two tools with two denominators means every rule on the wrong one is trust-debt you pay at 2am. For the statusDetail trap: once you confirm the worst-ever semantics, check whether the per-host membership view resets cleanly on restart — if the aggregate lies after a restart, the fix is alerting on the delta across restarts. One rule then catches both the stuck-pending and the ghost-host.
Ha, glad the vibes line landed.
On the restart thing, that part's actually already answered in the post: it doesn't reset. After I recreated the mongot container, statusDetail still listed the dead host as PENDING next to the new READY one, and the top-level status stayed PENDING because of it. So the aggregate does lie after a restart, that's the whole trap.
I'm not sure one rule covers both though. The stuck-pending case in the post happened on the very first build with no restart at all, it was purely disk headroom, so a delta-across-restarts rule would never fire on it. I'd still split it in two: alert on mongot's own free_bytes / total_bytes for the disk case, and for the ghost host, only look at statusDetail entries for hosts that are actually alive (your members diff from earlier). Two boring rules, each watching one cause.
That split is the right decomposition — one rule per cause, each boring. Two additions though. For the disk rule, mongot's free_bytes can dip briefly during compaction and recover within minutes, so a naive threshold fires false positives: trigger only when free_bytes stays under the line across N consecutive samples with a near-zero delta between them. For the ghost-host rule, take the members diff from replSetGetStatus, not from statusDetail — statusDetail is mongot's own cache and it's precisely the thing that lies, which is how the dead host survived the container recreate in the first place. Worth also surfacing which index(es) are bound to the ghost entry: the only real recovery is reindexing those, and knowing that upfront turns the alert from a mystery into a to-do.
For the disk rule, a plain
for: 10mon the alert already does the N-consecutive-samples thing, Prometheus only fires once the condition has held the whole window. A short dip during a merge never makes it that far. The stuck case in the post wasn't a dip anyway, it sat above the line the whole time, so a long-ishforcosts nothing there.On replSetGetStatus though, I don't think that works, and I kinda walked into it myself when I said "members diff" last time. replSetGetStatus lists the mongod replica set members. mongot isn't a member, so none of the hosts in statusDetail would ever show up there and every entry would look like a ghost. The list of live mongot hosts has to come from wherever you run them. If you're already scraping mongot on 9946,
up{job="mongot"}is basically that list for free.And on recovery, in my case it wasn't a reindex. Queries against the index worked fine the whole time, the new host was READY. The only broken thing was the summary status, cause it still counted the dead host. So the alert's to-do is more "ignore the ghost entry" than "rebuild the index". I didn't test whether dropping and recreating the index clears the ghost out, that'd be worth knowing though.
Agreed on the for semantics — a plain for: 10m at a 1m scrape interval is already ten consecutive samples, and Prometheus resets the timer the moment the expression goes false even once, so flapping metrics never fire. The cost of a long for isn't zero though: if the condition is genuinely marginal and toggles around the threshold, the alert stays silent forever, which is the opposite failure from a noisy page. That's why I'd pair the for with a hysteresis band instead of lengthening the window further. On the ghost entry, the approach that has held up for me is to alert on the count delta rather than on presence: compare the number of hosts in the summary status against count(up{job="mongot"} == 1), and page when the delta is non-zero for more than fifteen minutes. That signal self-clears when the dead entry ages out, so you never need to know whether re-creating the index clears it — though my read is that it won't, because the ghost lives in the summary document, not in the index. If the summary is rebuilt from a heartbeat table, the durable fix is an expiry on the heartbeat row.
Fair point on the marginal case, a long for on something hovering right at the line can stay quiet forever. Something like max_over_time over the window instead of the raw value would handle that without making the window even longer.
On the count delta, two things I'd push back on. First, statusDetail isn't a Prometheus metric, it only comes back from $listSearchIndexes, so you'd need a small exporter or a cron script that runs the aggregation and pushes the host count somewhere before you can compare it to up{job="mongot"}. Doable, just not free.
Second, "self-clears when the dead entry ages out" is the part I didn't see happen. In my run the dead host just stayed in statusDetail, and I never saw anything suggesting it expires. If it doesn't, that alert never clears, it just pages until someone silences it.
The heartbeat table with an expiry is a guess about mongot's internals, right? Nothing I tested shows where statusDetail is actually stored, so I wouldn't build on that until someone checks.
Agreed on both — and the exporter route is the honest one: a small cron that runs $listSearchIndexes, turns the host count into a gauge, and lets Prometheus own the alerting beats scraping it back out of logs. One addition on shape: for an 89%-disk threshold, level alerts are the wrong tool anyway — predict_linear over a 4-6h window tells you "crosses the line in N hours", which fires before the index goes stuck. The stalled index is the symptom; the runway is the signal. Keeps your max_over_time question clean too: slope for alerts, level for dashboards.
predict_linear on mongot_system_disk_space_data_path_free_bytes is a good addition for runway, but I wouldn't call level alerts the wrong tool. My case is the one it misses: the disk was already at 89% before I ever created the index, so the line was flat and there was nothing to predict. The index just went straight to PENDING. A slope alert tells you you're going to cross, a level alert tells you you already did, and you kind of need both.
Also worth remembering the threshold has hysteresis, it pauses around 90% and only resumes below roughly 85%, so "runway" really means runway to 90 and then cleanup down to 85, not just back under the line.
Anyway I think between the thread and the post there's enough here for anyone to build the alerting, so I'll leave it there. Thanks for digging in.
Fair — "you kind of need both" is the right frame, and the hysteresis detail is what sharpens it: if alerting re-arms only below ~85, then 85 is the number to compute runway against, not 90. 90 is when you stop hearing about it, not when you're safe — so "clean back under the line" really has to mean "under with margin", or the next slow climb happens while the warnings are still paused. The pairing I'll keep from this thread: a level rule for "already above and flat" — your case, where nothing needed predicting because it had already happened — and a slope rule for the "below and climbing" runway. Two messages, same disk, no state in between where nothing fires. Thanks for the corrections along the way — the denominator one tightened the whole thread.
Thanks, appreciate you sticking with it.
Glad it was useful, Alex. If that index ever does come unstuck, the pending-to-ready timing would make a great follow-up — it's the transition number almost nobody publishes. Cheers.
cheers 👍
Any time, Alex. If the index ever does shift, even a rough pair of timestamps — last seen pending, first seen ready — makes the transition number publishable; the shape of the wait is usually more interesting than its length. If it ever turns into that follow-up, I'll be reading. Good luck with it.