A reproducible comparison of CognoDB Cloud, Neo4j AuraDB, Memgraph, FalkorDB and ArangoDB, and the two things that nearly made it lie.
I was asked to benchmark CognoDB Cloud against four other managed graph databases. Same data, same workloads, honest numbers. I figured the hard part would be writing fair queries across five different engines. It wasn't. The hard part was realising that my first clean looking results table was mostly a map of undersea internet cables, and that one of the databases was quietly throwing away my data while telling me everything was fine.
Here's what actually happened.
The number that looked great and meant nothing
My first latency table said FalkorDB was about 2,700 times faster than CognoDB at a single indexed lookup. 0.09 ms versus 239.61 ms. That number is real. I can reproduce it any time I want. It's also completely meaningless.
CognoDB's free instance provisioned into Google Cloud's us-east4 region, which is Northern Virginia. I'm in Telangana, India. The query itself takes about a millisecond to run. The network round trip to Virginia and back takes roughly 243. So for that row, my benchmark was measuring the speed of light through glass across the Atlantic. FalkorDB wasn't faster at anything. It was just running in a container under my desk instead of on another continent.
And I couldn't fix it the obvious way. The other managed database, Neo4j AuraDB, landed in Singapore, about 88 ms from me, which had me worried right away, because now my two cloud databases weren't even on the same continent as each other. But the free tier of neither one lets you pick a region. There's no dropdown. You get what you're given. So I was stuck comparing a database in Virginia against a database in Singapore against three databases on localhost, and my honest first reaction was just: how am I supposed to compare these at all? If I report raw response times, I'm publishing a geography quiz, not a benchmark.
I did at least confirm Aura's location instead of guessing at it. A DNS lookup resolved its hostname to an IP inside Google's published Singapore range. The roughly 15,000 km gap between Singapore and Virginia matches the 156 ms difference I measured almost exactly.
The way out was to stop trusting wall clock time and start asking each database how long it actually spent on its own. Bolt drivers give you exactly that. The server reports its own execution time with the network excluded. So I report every latency three ways. Raw wall clock, which is what a user in India actually feels. RTT adjusted, which is wall clock minus that platform's ping. And server reported, which is the engine's own number. The third one is the only honest way to compare a database in Virginia against a container on my desk, and it changes the whole story. In pure engine time the platforms are far closer than the raw table makes them look.
The 150,000 relationships that never existed
Then came the part that actually rattled me.
My loader finished importing CognoDB and happily reported 23,211 relationships per second. Then my verification step read the database back and found 0 relationships.
My first thought was the boring one. The writes didn't go in properly, or I'd written the Cypher wrong, or there was a bug somewhere in my own loader. That's the honest first instinct when something breaks: assume it's your fault. So I started taking the thing apart.
I wrote a script to send edges in different batch sizes. 10, 100, 1,000, all the way up to 10,000. Every single one worked. Created exactly what I asked for. I tried writing edges right after building the indexes, then right after dropping them, in two different Cypher phrasings. All fine. At this point I genuinely didn't understand what I was looking at. Every test in isolation passed, but the full load created nothing. That's the worst kind of bug, the one that vanishes the moment you look straight at it.
The clue was the version. CognoDB runs v0.9.11, and it had already been refusing things that normal Neo4j supports. In particular it didn't recognise db.awaitIndexes(), which is the standard call for "wait until the indexes are ready." That refusal is what finally turned my attention toward the indexes instead of my queries. What if the writes were fine, and the real question was when they ran compared to the index being ready?
That was it. CognoDB builds indexes in the background, and on my data it took 28.8 seconds. On top of that there's no clean way to ask whether it's finished. The await call isn't supported, and SHOW INDEXES returns nothing useful for the status. And here's the actual bug. While an index is still building, a lookup that uses that index returns nothing at all, instead of either throwing an error or falling back to a full scan. My edge loading query matches two nodes by their indexed ID and then creates a relationship between them. During those 28.8 seconds the match silently found nobody, so the CREATE ran against nothing, succeeded, and created zero edges. No error. No warning. Just relationships_created = 0, batch after batch.
My original loader would have published 23,211 rel/s for 150,000 relationships that never existed, and I'd never have known, because I was counting the rows I sent, not the rows the database said it wrote. The only reason I caught it is that the harness checks the server's own counters and stops when they don't match.
The fix was to poll. After building the indexes, keep doing a lookup for a node I know exists until it actually comes back, and only then start loading edges. I record that waiting time separately so it never leaks into the throughput number.
The lesson goes well past CognoDB, and it's the thing I'm taking with me to every system I touch after this. Never trust the count of rows you submitted. Only trust the count the database says it wrote.
The scaling that ran backwards
One more result I didn't see coming.
I ran a concurrency test. 1 client, then 10, then 40, all hammering each database with a read heavy mix. The two cloud databases scaled beautifully. CognoDB went from 3.4 to 112 queries per second, Aura from 11 to 400. Roughly linear with the number of clients.
The three local databases got slower. Memgraph did 909 queries per second with one client and 468 with forty.
Once it clicked it made complete sense. The cloud databases are latency bound. Every query spends 88 to 244 ms just travelling, so a single client leaves the server mostly sitting idle. Add more clients and you're just filling that dead time, so throughput climbs. The local databases have no network wait at all, so at 0.5 vCPU they're already maxed out with a single client. Add more and all you're adding is queue, so per query latency balloons (Memgraph's p50 went from 0.27 ms to 14.6 ms) and total throughput drops.
Which means the single client latency table and the concurrency table are answering two different questions, and neither one on its own tells you what to actually deploy.
A smaller thing I liked. At 40 clients FalkorDB started flat out refusing about 1.5% of queries with a "max pending queries exceeded" message, and in return it kept its latency low and steady. ArangoDB accepted every query and let its latency climb 5 times over. Neither is wrong. They're just two different promises about what happens when you overload them, and the only way to find out which promise you bought is to overload them.
What I'd do differently
The geography problem is the big asterisk over all of this. With free tiers I couldn't put the client in the same region as the servers, so the server reported timing is doing a lot of heavy lifting. With more time I'd run the whole suite from a cloud VM sitting right next to each database in turn, and I'd repeat every run several times to measure how much the numbers wobble, instead of trusting a single pass.
But the thing I'm actually glad about is the boring little safeguard that caught the CognoDB bug. I almost didn't add the check that compares the rows I sent against the count the server reports. If I'd skipped it, this would have been a clean, professional, completely wrong benchmark, and I'd have had no idea anything was off.
The full harness, every number, and all the raw data are on GitHub: github.com/FANZ3R/graph-database-cloud-benchmark
Top comments (0)