I was asked to benchmark CognoDB Cloud, a managed graph database, against four other graph platforms on the same hardware, the same dataset, and the same queries. The brief was clear about one thing: this isn't about proving a winner, it's about proving the comparison itself is fair. That's the part I actually cared about getting right, coming from a DevOps background — a benchmark is only as good as the infrastructure decisions behind it.
The biggest finding wasn't about CognoDB at all. Under a memory cap tight enough to be realistic for a free tier, Neo4j — one of the most widely used graph databases out there — failed 740 out of 744 operations the moment I pushed it to 40 concurrent clients. Not slower. Failed.
Here's what I set up, what I found, and a couple of real bugs I hit along the way.
The setup
CognoDB Cloud (the platform being evaluated) against Neo4j Community, Memgraph Community, ArangoDB Community, and FalkorDB. I picked those four deliberately, not at random: Neo4j because CognoDB documents itself as drop-in compatible with the official Neo4j driver, so it's the closest apples-to-apples comparison available. Memgraph because it speaks the same protocol but is an in-memory C++ engine, not a JVM. ArangoDB because it's deliberately not Cypher — it forces the "same logical query" methodology to hold up across a completely different query language. FalkorDB because it's built specifically for low-memory environments, which is exactly what a free tier is.
Dataset: MovieLens 100K, 2,665 nodes, 103,836 relationships — a real public dataset, nothing synthetic. Every platform ran the same six measurements: load throughput, 1/2/3-hop traversal latency, indexed lookups, an aggregation query, and a mixed read/write sweep across 1, 10, and 40 concurrent clients. And every platform got the same resource cap: 0.5 vCPU, 1 GB disk, 512 MB RAM.
That RAM number is where the first problem showed up.
The spec sheet was wrong
The assignment doc said CognoDB's free tier gives you 256 MB of RAM. When I actually spun up a live instance and checked the console, it was provisioned with 512 MB. Double what was documented.
If I'd capped the four comparator databases at 256 MB because the doc said so, while CognoDB itself was running on double that, the whole benchmark would've been rigged against the comparators without me even meaning to. I re-capped everything to match what CognoDB was actually running, not what the doc claimed. Small thing, but it's the difference between a benchmark that's fair and one that just looks fair on paper.
Neo4j didn't slow down — it fell over
Neo4j Community, running under the same 512 MB cap as everyone else, handled 1 and 10 concurrent clients fine, decent latencies across the board. Then I pushed it to 40 concurrent clients and it didn't degrade gracefully — it collapsed. 740 failed operations out of 744, in a 15-second window.
To be clear about what I didn't do: I didn't loosen the memory cap to make that number go away, and I didn't drop the 40-client run from the results because it looked bad. Both would've been easy, both would've defeated the whole point. My best guess without deeper server-side profiling is that Neo4j's JVM, under real memory pressure with 40 threads each opening their own bolt session per operation, just runs out of headroom. That's a guess, not a confirmed root cause, and I'm calling it that in the writeup rather than presenting it as fact.
CognoDB, over an actual network connection to a cloud instance, handled all three concurrency levels — including 40 clients — with zero errors.
Same protocol, same driver, opposite behavior
Memgraph and Neo4j both speak Cypher over the same bolt protocol with the same official driver. I expected similar behavior. Instead Memgraph's throughput curve runs the opposite direction from CognoDB's — fastest with one client, and it gets slower, not faster, as concurrency climbs, all the way to 40 clients, without ever throwing an error. Two platforms on the same protocol and driver, two totally different personalities under load. Not a bug in either one — a real difference in how each engine handles contention, and exactly the kind of thing you'd never catch running a benchmark at a single concurrency level.
Where CognoDB actually looked weak
Not going to bury this one: CognoDB's slowest result anywhere in the entire benchmark, across all eight platforms tested, was its own aggregation query — a group-by-average that took over three seconds at the median. Neo4j and Memgraph ran the identical query locally in about a quarter of a second. Some of that gap is the network hop to a cloud instance, but that alone doesn't explain a number twelve times higher than CognoDB's own simplest operations. My best guess is a burstable, shared vCPU throttling harder under a scan-heavy aggregation than a single-hop lookup — but the free tier doesn't expose any metrics to actually confirm that, so I'm flagging it as an open question, not a conclusion.
A bug that wasn't mine to begin with
Worth mentioning because it's a good reminder that "my benchmark is broken" and "the database is broken" look identical at first. FalkorDB's aggregation query kept silently timing out — no error message, just nothing coming back. Turned out FalkorDB's Docker image ships with a one-second default query timeout, something other people had already flagged as a surprise on FalkorDB's own GitHub issues. No other platform in this benchmark enforces any query deadline at all, so I disabled it. Not to flatter FalkorDB's numbers — to keep the comparison fair, since an unrelated software default silently discarding results isn't part of anyone's resource-cap methodology.
What I'd actually tell someone evaluating CognoDB
Not "CognoDB wins" — that was never the question, and answering it that way would throw out everything actually useful in the data. What the data shows is that graph databases that look interchangeable on paper — same protocol, same query language, same free-tier numbers — behave completely differently once you put real, identical pressure on them. If you're evaluating CognoDB specifically: it's consistent, it didn't error once across three concurrency levels where a well-known competitor fell over, and its weak spot is aggregation-heavy workloads on the free tier. Worth testing against your own actual access patterns before you commit either way — not something you can read off a spec sheet.
Full methodology, every raw result, and every caveat — including the ones that don't flatter CognoDB — are public: github.com/SachinMerugune/cognodb-graph-benchmark.
Top comments (1)
The best benchmark posts are the ones where the surprising result changes the measurement method. If the interesting finding had nothing to do with the product, that usually means the test surfaced a real workload assumption.