Graph database benchmarks are easy to get wrong.
Most public benchmarks compare expensive production instances, use different datasets, ignore network latency, hide free-tier throttling, or quietly optimize one database more than the others.
For a recent engineering assignment from Wexa AI, I decided to do the opposite.
I built a small benchmark harness that compared CognoDB Cloud against four other graph database platforms using:
- the same dataset
- the same logical queries
- the same client machine
- free or entry-level resource tiers
- warm-up before measurement
- percentile-based latency reporting
- automated one-command runs
- honest caveats
The platforms tested were:
- CognoDB Cloud
- Neo4j Aura Free
- Neo4j Sandbox
- Docker-capped Neo4j Community
- Memgraph Cloud
The dataset was the SNAP cit-HepTh citation network:
- 27,769 nodes
- 352,768 relationships
- papers citing other papers
Small enough to fit inside free tiers, but large enough to make the benchmark meaningful.
The full benchmark code, results, and documentation are available here:
GitHub repository: [Add your GitHub repository link]
The Goal Was Not to Find a “Winner”
The assignment was clear: the goal was not to crown a database.
The goal was to evaluate engineering rigor:
- fair methodology
- reproducible automation
- honest reporting
- clear analysis
- documented caveats
That matters because real-world database selection is messy.
There is no universal “best graph database.”
There is only the best database for:
- your data size
- your query patterns
- your latency requirements
- your operational constraints
- your budget
- your deployment model
So this benchmark was designed to measure behavior under constrained free-tier conditions, not to simulate an unlimited production environment.
The Fairness Rules
Before writing code, I set a few rules.
1. Same dataset everywhere
Every platform received the exact same graph.
2. Same logical queries everywhere
All platforms ran the same workloads, even where minor Cypher dialect adjustments were required.
3. Free or entry tiers only
No paid production instance was allowed to sneak into the comparison.
4. Small resource footprint
CognoDB’s free tier is intentionally small:
- burstable 0.5 vCPU
- 256 MB RAM
- 1 GB disk
So the other platforms were run on their free, trial, or capped self-hosted tiers.
5. Warm-up before measurement
Every read workload was warmed up first. Cold-start behavior was excluded from the main latency numbers and documented as a caveat.
6. Percentiles, not just averages
Averages can hide bad requests.
So the benchmark reported:
- p50 latency: typical behavior
- p95 latency: tail behavior
7. No hidden caveats
If something failed, throttled, crashed, or behaved strangely, it got documented.
That rule ended up being one of the most valuable parts of the project.
The Dataset
I used the SNAP cit-HepTh citation network.
It represents arXiv High Energy Physics Theory papers and their citations.
If paper A cites paper B, the graph contains a directed edge from A to B.
The final cleaned graph contained:
| Metric | Value |
|---|---|
| Nodes | 27,769 |
| Relationships | 352,768 |
| Node label | Paper |
| Relationship type | CITES |
Each paper node had:
idyearfield
The year property came from the original SNAP metadata.
The field property was generated deterministically for aggregation testing, so every platform received identical values.
This dataset was a good fit because it had enough relationships to make traversals and aggregations interesting, but it was still small enough to fit into free tiers.
The Benchmark Harness
I built the benchmark harness in C# using .NET 8 and the official Neo4j .NET driver.
For each platform, the harness did the following:
- downloaded and prepared the dataset if needed
- connected using secrets from environment variables
- wiped old benchmark data
- created indexes and constraints
- loaded nodes and relationships in batches
- ran warm-up queries
- measured read workloads over 100 iterations
- reported p50 and p95 latency
- ran mixed read/write workloads at 1, 10, and 40 concurrent clients
- saved results as JSON
Then a report generator converted the JSON files into Markdown tables.
The workflow was simple:
dotnet run -- cognodb
dotnet run -- aura
dotnet run -- sandbox
dotnet run -- memgraph
dotnet run -- docker
And then:
dotnet run -- report
No passwords were stored in the repository. All credentials were read from environment variables.
The Workloads
The benchmark measured six categories.
1. Ingest throughput
How fast can the platform load the graph?
Measured as:
- nodes per second
- relationships per second
- total wall-clock load time
2. Traversals
One-hop, two-hop, and three-hop queries from randomly selected start nodes.
Example:
MATCH (p:Paper {id: $id})-[:CITES]->()-[:CITES]->(q)
RETURN count(q) AS c;
3. Lookups
Point lookup by indexed id:
MATCH (p:Paper {id: $id})
RETURN p.year AS year;
Filtered lookup using indexed year:
MATCH (p:Paper)
WHERE p.year >= $y1 AND p.year <= $y2
RETURN count(p) AS c;
4. Aggregation
Group-by query over a node property:
MATCH (p:Paper)
RETURN p.field AS field, count(*) AS c
ORDER BY c DESC;
5. Mixed read/write workload
The mixed workload used:
- 80% reads
- 20% writes
- 1, 10, and 40 concurrent clients
- 30 seconds per concurrency level
6. Footprint
Where observable, the benchmark recorded:
- node count
- relationship count
- resource limits
- available platform metadata
Where a platform did not expose internals, the result was marked as not observable.
Results: Ingest Throughput
Memgraph dominated ingest.
| Platform | Relationships/sec | Total Load Time |
|---|---|---|
| Memgraph | 14,966 | 25.7s |
| Docker Neo4j | 8,144 | 55.2s |
| Neo4j Aura | 7,865 | 47.6s |
| Neo4j Sandbox | 2,339 | 167.9s |
| CognoDB | 1,209 | 317.9s |
Visual summary:
memgraph 14966 rels/sec ████████████████████████████████
docker 8144 rels/sec █████████████████
aura 7865 rels/sec ████████████████
sandbox 2339 rels/sec █████
cognodb 1209 rels/sec ██
Memgraph’s in-memory architecture gave it a major advantage during batched loading.
Aura also performed strongly.
Docker Neo4j benefited from local execution and no cloud network overhead.
CognoDB and Sandbox were slower under the tested free-tier conditions.
Results: Traversal Latency
For traversal latency, Docker Neo4j had the lowest p50 numbers because it ran locally.
| Platform | 1-hop p50 | 2-hop p50 | 3-hop p50 |
|---|---|---|---|
| Docker Neo4j | 3.1 ms | 3.2 ms | 3.3 ms |
| Memgraph | 95.3 ms | 95.8 ms | 96.2 ms |
| Neo4j Aura | 146.2 ms | 146.5 ms | 146.8 ms |
| CognoDB | 644.2 ms | 652.3 ms | 654.2 ms |
| Neo4j Sandbox | 697.8 ms | 697.7 ms | 698.0 ms |
One pattern stood out.
For most platforms, latency barely increased from one hop to three hops.
For example:
- Aura: 146.2 ms to 146.8 ms
- Memgraph: 95.3 ms to 96.2 ms
- Sandbox: 697.8 ms to 698.0 ms
- CognoDB: 644.2 ms to 654.2 ms
That tells us something important.
At this dataset size, the actual traversal work was small compared with fixed overhead.
That overhead may include:
- network round-trip time
- request parsing
- query planning
- serialization
- connection handling
- free-tier CPU scheduling
In other words, the query engine may not have been the dominant cost. The request path was.
Results: Lookups and Aggregations
Lookup latency followed a similar pattern.
Point lookup p50
| Platform | Point Lookup p50 |
|---|---|
| Docker Neo4j | 3.3 ms |
| Memgraph | 95.1 ms |
| Neo4j Aura | 146.1 ms |
| CognoDB | 643.9 ms |
| Neo4j Sandbox | 698.8 ms |
Aggregation p50
| Platform | Aggregation p50 |
|---|---|
| Docker Neo4j | 13.9 ms |
| Memgraph | 106.0 ms |
| Neo4j Aura | 156.8 ms |
| CognoDB | 683.7 ms |
| Neo4j Sandbox | 704.6 ms |
Docker’s low numbers again reflected its localhost advantage.
Among cloud platforms, Memgraph and Aura were the strongest in this test.
CognoDB remained stable, but its latency was dominated by what appeared to be fixed request overhead under the tested free-tier environment.
Results: Mixed Read/Write Workload
The mixed workload used 80% reads and 20% writes.
At 40 concurrent clients:
| Platform | QPS | Errors |
|---|---|---|
| Docker Neo4j | 378.4 | 0 |
| Memgraph | 374.4 | 5 |
| Neo4j Aura | 287.7 | 0 |
| CognoDB | 61.2 | 0 |
| Neo4j Sandbox | 51.3 | 0 |
Visual summary:
docker 378.4 QPS ████████████████████████████████
memgraph 374.4 QPS ███████████████████████████████
aura 287.7 QPS ████████████████████████
cognodb 61.2 QPS █████
sandbox 51.3 QPS ████
Docker and Memgraph achieved the highest throughput.
Aura scaled well and completed the workload with no recorded errors.
CognoDB and Sandbox showed lower absolute throughput, but both completed with zero recorded errors.
Memgraph produced five errors during the 40-client workload. I kept those errors in the results because hiding them would defeat the purpose of an honest benchmark. The likely explanation is resource pressure on the entry-tier instance under concurrent writes.
Surprise #1: Java Does Not Like Strict 256 MB Containers
One of the most interesting findings came from Docker Neo4j.
CognoDB’s free tier gives you:
- 0.5 vCPU
- 256 MB RAM
- 1 GB disk
I wanted the local Docker Neo4j instance to be as close to that as possible.
So I first capped the Docker container at 256 MB.
Neo4j crashed during startup.
It did not fail during the benchmark. It failed before it could serve queries.
The fix was to distinguish between two different memory concepts:
- container memory ceiling
- database JVM heap
The final Docker configuration used:
- 512 MB container ceiling
- 256 MB Neo4j JVM heap max
- 50 MB Neo4j page cache
That distinction mattered.
The database heap was still limited to 256 MB, but the JVM needed additional headroom just to run.
This became one of my favorite architectural observations from the project:
A database memory limit is not always the same thing as a process memory limit.
It also highlighted a difference between lightweight engines and JVM-based engines. CognoDB’s free tier operates within a very small total footprint, while Java-based Neo4j requires more baseline headroom.
Surprise #2: Free-Tier Latency Is Often About Fixed Overhead
The traversal results were interesting because latency barely changed as query complexity increased.
If the graph traversal itself were the dominant cost, we would expect three-hop queries to be noticeably slower than one-hop queries.
But for several platforms, they were almost identical.
That suggests that the measured latency was dominated by fixed overhead:
- network round-trip time
- request handling
- query planning
- serialization
- scheduling
- free-tier throttling
This is an important lesson for anyone reading database benchmarks.
A slow result does not always mean the engine is slow.
It may mean:
- the instance is tiny
- the network path is distant
- the CPU is burstable
- the environment is shared
- the platform is throttling free-tier traffic
Honest benchmarks need to say that clearly.
Surprise #3: Docker Is Useful, but It Is Not a Fair Cloud Comparison
Docker Neo4j had the lowest latency in almost every category.
But that result must be interpreted carefully.
Docker ran on the same machine as the benchmark client.
That means:
- no internet round-trip time
- no cloud routing
- no managed-service request overhead
Docker results are useful as a local engine baseline, but they are not directly comparable to cloud platforms over a network.
This is one of the reasons I included Docker in the first place.
It helped separate two questions:
- How does the engine behave locally?
- How does the managed platform behave over a real network path?
Those are related, but they are not the same.
The Importance of Percentiles
One of the requirements I’m glad the assignment enforced was percentile reporting.
Docker Neo4j had very low p50 latency:
- around 3 ms for lookups
- around 14 ms for aggregation
But its p95 latency jumped to around 80 ms in several workloads.
That gap matters.
Averages could make Docker look almost perfect. But p95 revealed occasional pauses, likely related to JVM behavior, caching, or local scheduling.
In production, users often notice the tail more than the average.
A system that usually responds in 5 ms but occasionally responds in 100 ms can feel worse than a system that consistently responds in 30 ms.
Percentiles expose that.
What the Results Suggest
Under the conditions of this benchmark:
Memgraph
Memgraph showed strong ingest throughput and strong cloud latency. Its in-memory architecture appears to benefit both batch loading and read workloads.
Neo4j Aura
Aura showed strong managed-cloud behavior, especially considering it was running on a free tier. It scaled well under concurrency and completed mixed workloads with zero recorded errors.
Docker Neo4j
Docker Neo4j produced the lowest local latency, but its results include a localhost advantage and should not be treated as a direct cloud comparison.
Neo4j Sandbox
Sandbox behaved like a managed trial environment with relatively high fixed request latency. It is useful for evaluation, but its internal resource guarantees are not publicly detailed.
CognoDB
CognoDB completed all benchmark workloads with zero mixed-workload errors. Its free tier is intentionally small, and under the tested network path it showed higher request latency and lower throughput than some other platforms.
The important caveat is that this reflects the free-tier conditions and client network path used in this test. It does not necessarily represent CognoDB’s behavior under larger instances, closer regions, or production configurations.
Honest Caveats
This benchmark has limitations, and they matter.
Free tiers are noisy
Free instances may be shared, throttled, or burstable. Results can vary over time.
Network path matters
Cloud platforms were tested from a residential internet connection. Different regions or networks could produce different results.
Not all specs are public
Some platforms do not expose exact vCPU, RAM, or storage details for free tiers.
Docker has a localhost advantage
Docker Neo4j avoids cloud network latency, so it should be interpreted as a local baseline.
Docker needed more container memory than CognoDB
Neo4j’s JVM required extra headroom beyond the 256 MB database heap.
Warm numbers only
The reported latency numbers are warm-run numbers. Cold-start behavior was not separately benchmarked.
One full measurement pass
Each read workload used 100 iterations, but the full suite was not repeated across multiple days due to the 48-hour assignment window.
Engine diversity
Aura, Sandbox, and Docker all use Neo4j-based engines. They were included to compare different deployment models. Memgraph provided a different in-memory architecture. Future work could include additional distinct engines such as FalkorDB, ArangoDB, NebulaGraph, or TigerGraph.
What I Would Improve Next
If I had more time, I would extend the benchmark in several ways.
Add more distinct engines
FalkorDB, ArangoDB, NebulaGraph, TigerGraph, and Kùzu would make the comparison more diverse.
Run repeated full-suite passes
Multiple runs across different times would help measure variance.
Add cold-start benchmarking
Cold-start latency is important for serverless and scale-to-zero environments.
Collect platform metrics
Where available, I would collect:
- CPU usage
- memory usage
- disk usage
- request errors
- saturation metrics
Generate charts automatically
The current report uses Markdown tables and simple text charts. A chart generator would make the results easier to read.
Test larger datasets
The current dataset fits free tiers. A larger dataset would stress indexing, memory, and storage more aggressively.
The Bigger Lesson
The biggest lesson from this project was not “Database X is faster.”
The bigger lesson was that benchmarking is a systems problem.
You are not only measuring the database engine.
You are measuring:
- the network
- the driver
- the query language
- the instance size
- the storage engine
- the runtime
- the cloud tier
- the client machine
- the workload design
- the honesty of the methodology
If you ignore those things, your benchmark may be technically reproducible but practically misleading.
If you document them, your benchmark becomes useful.
That was the spirit of this assignment.
Not to hide the messy parts.
Not to smooth over the caveats.
Not to declare a winner.
But to measure carefully, explain clearly, and leave the next engineer enough information to reproduce the work.
Repository
The benchmark harness, results, and full documentation are available here:
GitHub repository: https://github.com/kalbashi09/Benchmark
The README includes:
- full results tables
- platform specs
- dataset details
- query workloads
- methodology
- caveats
- reproducibility instructions
If you are evaluating graph databases, I hope this helps you ask better questions before trusting any benchmark.
Including mine.
Top comments (0)