Benchmarking 5 Graph Database Platforms on the Same 352k-Edge Graph: What Free Tiers Hide

John David Paul — Thu, 27 Aug 2026 03:38:43 +0000

Graph database benchmarks are easy to get wrong.

Most public benchmarks compare expensive production instances, use different datasets, ignore network latency, hide free-tier throttling, or quietly optimize one database more than the others.

For a recent engineering assignment from Wexa AI, I decided to do the opposite.

I built a small benchmark harness that compared CognoDB Cloud against four other graph database platforms using:

the same dataset
the same logical queries
the same client machine
free or entry-level resource tiers
warm-up before measurement
percentile-based latency reporting
automated one-command runs
honest caveats

The platforms tested were:

CognoDB Cloud
Neo4j Aura Free
Neo4j Sandbox
Docker-capped Neo4j Community
Memgraph Cloud

The dataset was the SNAP cit-HepTh citation network:

27,769 nodes
352,768 relationships
papers citing other papers

Small enough to fit inside free tiers, but large enough to make the benchmark meaningful.

The full benchmark code, results, and documentation are available here:

GitHub repository: [Add your GitHub repository link]

The Goal Was Not to Find a “Winner”

The assignment was clear: the goal was not to crown a database.

The goal was to evaluate engineering rigor:

fair methodology
reproducible automation
honest reporting
clear analysis
documented caveats

That matters because real-world database selection is messy.

There is no universal “best graph database.”

There is only the best database for:

your data size
your query patterns
your latency requirements
your operational constraints
your budget
your deployment model

So this benchmark was designed to measure behavior under constrained free-tier conditions, not to simulate an unlimited production environment.

The Fairness Rules

Before writing code, I set a few rules.

1. Same dataset everywhere

Every platform received the exact same graph.

2. Same logical queries everywhere

All platforms ran the same workloads, even where minor Cypher dialect adjustments were required.

3. Free or entry tiers only

No paid production instance was allowed to sneak into the comparison.

4. Small resource footprint

CognoDB’s free tier is intentionally small:

burstable 0.5 vCPU
256 MB RAM
1 GB disk

So the other platforms were run on their free, trial, or capped self-hosted tiers.

5. Warm-up before measurement

Every read workload was warmed up first. Cold-start behavior was excluded from the main latency numbers and documented as a caveat.

6. Percentiles, not just averages

Averages can hide bad requests.

So the benchmark reported:

p50 latency: typical behavior
p95 latency: tail behavior

7. No hidden caveats

If something failed, throttled, crashed, or behaved strangely, it got documented.

That rule ended up being one of the most valuable parts of the project.

The Dataset

I used the SNAP cit-HepTh citation network.

It represents arXiv High Energy Physics Theory papers and their citations.

If paper A cites paper B, the graph contains a directed edge from A to B.

The final cleaned graph contained:

Metric	Value
Nodes	27,769
Relationships	352,768
Node label	`Paper`
Relationship type	`CITES`

Each paper node had:

id
year
field

The year property came from the original SNAP metadata.

The field property was generated deterministically for aggregation testing, so every platform received identical values.

This dataset was a good fit because it had enough relationships to make traversals and aggregations interesting, but it was still small enough to fit into free tiers.

The Benchmark Harness

I built the benchmark harness in C# using .NET 8 and the official Neo4j .NET driver.

For each platform, the harness did the following:

downloaded and prepared the dataset if needed
connected using secrets from environment variables
wiped old benchmark data
created indexes and constraints
loaded nodes and relationships in batches
ran warm-up queries
measured read workloads over 100 iterations
reported p50 and p95 latency
ran mixed read/write workloads at 1, 10, and 40 concurrent clients
saved results as JSON

Then a report generator converted the JSON files into Markdown tables.

The workflow was simple:

dotnet run -- cognodb
dotnet run -- aura
dotnet run -- sandbox
dotnet run -- memgraph
dotnet run -- docker

And then:

dotnet run -- report

No passwords were stored in the repository. All credentials were read from environment variables.

The Workloads

The benchmark measured six categories.

1. Ingest throughput

How fast can the platform load the graph?

Measured as:

nodes per second
relationships per second
total wall-clock load time

2. Traversals

One-hop, two-hop, and three-hop queries from randomly selected start nodes.

Example:

MATCH (p:Paper {id: $id})-[:CITES]->()-[:CITES]->(q)
RETURN count(q) AS c;

3. Lookups

Point lookup by indexed id:

MATCH (p:Paper {id: $id})
RETURN p.year AS year;

Filtered lookup using indexed year:

MATCH (p:Paper)
WHERE p.year >= $y1 AND p.year <= $y2
RETURN count(p) AS c;

4. Aggregation

Group-by query over a node property:

MATCH (p:Paper)
RETURN p.field AS field, count(*) AS c
ORDER BY c DESC;

5. Mixed read/write workload

The mixed workload used:

80% reads
20% writes
1, 10, and 40 concurrent clients
30 seconds per concurrency level

6. Footprint

Where observable, the benchmark recorded:

node count
relationship count
resource limits
available platform metadata

Where a platform did not expose internals, the result was marked as not observable.

Results: Ingest Throughput

Memgraph dominated ingest.

Platform	Relationships/sec	Total Load Time
Memgraph	14,966	25.7s
Docker Neo4j	8,144	55.2s
Neo4j Aura	7,865	47.6s
Neo4j Sandbox	2,339	167.9s
CognoDB	1,209	317.9s

Visual summary:

memgraph   14966 rels/sec  ████████████████████████████████
docker      8144 rels/sec  █████████████████
aura        7865 rels/sec  ████████████████
sandbox     2339 rels/sec  █████
cognodb     1209 rels/sec  ██

Memgraph’s in-memory architecture gave it a major advantage during batched loading.

Aura also performed strongly.

Docker Neo4j benefited from local execution and no cloud network overhead.

CognoDB and Sandbox were slower under the tested free-tier conditions.

Results: Traversal Latency

For traversal latency, Docker Neo4j had the lowest p50 numbers because it ran locally.

Platform	1-hop p50	2-hop p50	3-hop p50
Docker Neo4j	3.1 ms	3.2 ms	3.3 ms
Memgraph	95.3 ms	95.8 ms	96.2 ms
Neo4j Aura	146.2 ms	146.5 ms	146.8 ms
CognoDB	644.2 ms	652.3 ms	654.2 ms
Neo4j Sandbox	697.8 ms	697.7 ms	698.0 ms

One pattern stood out.

For most platforms, latency barely increased from one hop to three hops.

For example:

Aura: 146.2 ms to 146.8 ms
Memgraph: 95.3 ms to 96.2 ms
Sandbox: 697.8 ms to 698.0 ms
CognoDB: 644.2 ms to 654.2 ms

That tells us something important.

At this dataset size, the actual traversal work was small compared with fixed overhead.

That overhead may include:

network round-trip time
request parsing
query planning
serialization
connection handling
free-tier CPU scheduling

In other words, the query engine may not have been the dominant cost. The request path was.

Results: Lookups and Aggregations

Lookup latency followed a similar pattern.

Point lookup p50

Platform	Point Lookup p50
Docker Neo4j	3.3 ms
Memgraph	95.1 ms
Neo4j Aura	146.1 ms
CognoDB	643.9 ms
Neo4j Sandbox	698.8 ms

Aggregation p50

Platform	Aggregation p50
Docker Neo4j	13.9 ms
Memgraph	106.0 ms
Neo4j Aura	156.8 ms
CognoDB	683.7 ms
Neo4j Sandbox	704.6 ms

Docker’s low numbers again reflected its localhost advantage.

Among cloud platforms, Memgraph and Aura were the strongest in this test.

CognoDB remained stable, but its latency was dominated by what appeared to be fixed request overhead under the tested free-tier environment.

Results: Mixed Read/Write Workload

The mixed workload used 80% reads and 20% writes.

At 40 concurrent clients:

Platform	QPS	Errors
Docker Neo4j	378.4	0
Memgraph	374.4	5
Neo4j Aura	287.7	0
CognoDB	61.2	0
Neo4j Sandbox	51.3	0

Visual summary:

docker     378.4 QPS  ████████████████████████████████
memgraph   374.4 QPS  ███████████████████████████████
aura       287.7 QPS  ████████████████████████
cognodb     61.2 QPS  █████
sandbox     51.3 QPS  ████

Docker and Memgraph achieved the highest throughput.

Aura scaled well and completed the workload with no recorded errors.

CognoDB and Sandbox showed lower absolute throughput, but both completed with zero recorded errors.

Memgraph produced five errors during the 40-client workload. I kept those errors in the results because hiding them would defeat the purpose of an honest benchmark. The likely explanation is resource pressure on the entry-tier instance under concurrent writes.

Surprise #1: Java Does Not Like Strict 256 MB Containers

One of the most interesting findings came from Docker Neo4j.

CognoDB’s free tier gives you:

0.5 vCPU
256 MB RAM
1 GB disk

I wanted the local Docker Neo4j instance to be as close to that as possible.

So I first capped the Docker container at 256 MB.

Neo4j crashed during startup.

It did not fail during the benchmark. It failed before it could serve queries.

The fix was to distinguish between two different memory concepts:

container memory ceiling
database JVM heap

The final Docker configuration used:

512 MB container ceiling
256 MB Neo4j JVM heap max
50 MB Neo4j page cache

That distinction mattered.

The database heap was still limited to 256 MB, but the JVM needed additional headroom just to run.

This became one of my favorite architectural observations from the project:

A database memory limit is not always the same thing as a process memory limit.

It also highlighted a difference between lightweight engines and JVM-based engines. CognoDB’s free tier operates within a very small total footprint, while Java-based Neo4j requires more baseline headroom.

Surprise #2: Free-Tier Latency Is Often About Fixed Overhead

The traversal results were interesting because latency barely changed as query complexity increased.

If the graph traversal itself were the dominant cost, we would expect three-hop queries to be noticeably slower than one-hop queries.

But for several platforms, they were almost identical.

That suggests that the measured latency was dominated by fixed overhead:

network round-trip time
request handling
query planning
serialization
scheduling
free-tier throttling

This is an important lesson for anyone reading database benchmarks.

A slow result does not always mean the engine is slow.

It may mean:

the instance is tiny
the network path is distant
the CPU is burstable
the environment is shared
the platform is throttling free-tier traffic

Honest benchmarks need to say that clearly.

Surprise #3: Docker Is Useful, but It Is Not a Fair Cloud Comparison

Docker Neo4j had the lowest latency in almost every category.

But that result must be interpreted carefully.

Docker ran on the same machine as the benchmark client.

That means:

no internet round-trip time
no cloud routing
no managed-service request overhead

Docker results are useful as a local engine baseline, but they are not directly comparable to cloud platforms over a network.

This is one of the reasons I included Docker in the first place.

It helped separate two questions:

How does the engine behave locally?
How does the managed platform behave over a real network path?

Those are related, but they are not the same.

The Importance of Percentiles

One of the requirements I’m glad the assignment enforced was percentile reporting.

Docker Neo4j had very low p50 latency:

around 3 ms for lookups
around 14 ms for aggregation

But its p95 latency jumped to around 80 ms in several workloads.

That gap matters.

Averages could make Docker look almost perfect. But p95 revealed occasional pauses, likely related to JVM behavior, caching, or local scheduling.

In production, users often notice the tail more than the average.

A system that usually responds in 5 ms but occasionally responds in 100 ms can feel worse than a system that consistently responds in 30 ms.

Percentiles expose that.

What the Results Suggest

Under the conditions of this benchmark:

Memgraph

Memgraph showed strong ingest throughput and strong cloud latency. Its in-memory architecture appears to benefit both batch loading and read workloads.

Neo4j Aura

Aura showed strong managed-cloud behavior, especially considering it was running on a free tier. It scaled well under concurrency and completed mixed workloads with zero recorded errors.

Docker Neo4j

Docker Neo4j produced the lowest local latency, but its results include a localhost advantage and should not be treated as a direct cloud comparison.

Neo4j Sandbox

Sandbox behaved like a managed trial environment with relatively high fixed request latency. It is useful for evaluation, but its internal resource guarantees are not publicly detailed.

CognoDB

CognoDB completed all benchmark workloads with zero mixed-workload errors. Its free tier is intentionally small, and under the tested network path it showed higher request latency and lower throughput than some other platforms.

The important caveat is that this reflects the free-tier conditions and client network path used in this test. It does not necessarily represent CognoDB’s behavior under larger instances, closer regions, or production configurations.

Honest Caveats

This benchmark has limitations, and they matter.

Free tiers are noisy

Free instances may be shared, throttled, or burstable. Results can vary over time.

Network path matters

Cloud platforms were tested from a residential internet connection. Different regions or networks could produce different results.

Not all specs are public

Some platforms do not expose exact vCPU, RAM, or storage details for free tiers.

Docker has a localhost advantage

Docker Neo4j avoids cloud network latency, so it should be interpreted as a local baseline.

Docker needed more container memory than CognoDB

Neo4j’s JVM required extra headroom beyond the 256 MB database heap.

Warm numbers only

The reported latency numbers are warm-run numbers. Cold-start behavior was not separately benchmarked.

One full measurement pass

Each read workload used 100 iterations, but the full suite was not repeated across multiple days due to the 48-hour assignment window.

Engine diversity

Aura, Sandbox, and Docker all use Neo4j-based engines. They were included to compare different deployment models. Memgraph provided a different in-memory architecture. Future work could include additional distinct engines such as FalkorDB, ArangoDB, NebulaGraph, or TigerGraph.

What I Would Improve Next

If I had more time, I would extend the benchmark in several ways.

Add more distinct engines

FalkorDB, ArangoDB, NebulaGraph, TigerGraph, and Kùzu would make the comparison more diverse.

Run repeated full-suite passes

Multiple runs across different times would help measure variance.

Add cold-start benchmarking

Cold-start latency is important for serverless and scale-to-zero environments.

Collect platform metrics

Where available, I would collect:

CPU usage
memory usage
disk usage
request errors
saturation metrics

Generate charts automatically

The current report uses Markdown tables and simple text charts. A chart generator would make the results easier to read.

Test larger datasets

The current dataset fits free tiers. A larger dataset would stress indexing, memory, and storage more aggressively.

The Bigger Lesson

The biggest lesson from this project was not “Database X is faster.”

The bigger lesson was that benchmarking is a systems problem.

You are not only measuring the database engine.

You are measuring:

the network
the driver
the query language
the instance size
the storage engine
the runtime
the cloud tier
the client machine
the workload design
the honesty of the methodology

If you ignore those things, your benchmark may be technically reproducible but practically misleading.

If you document them, your benchmark becomes useful.

That was the spirit of this assignment.

Not to hide the messy parts.

Not to smooth over the caveats.

Not to declare a winner.

But to measure carefully, explain clearly, and leave the next engineer enough information to reproduce the work.

Repository

The benchmark harness, results, and full documentation are available here:

GitHub repository: https://github.com/kalbashi09/Benchmark

The README includes:

full results tables
platform specs
dataset details
query workloads
methodology
caveats
reproducibility instructions

If you are evaluating graph databases, I hope this helps you ask better questions before trusting any benchmark.

Including mine.

DEV Community: John David Paul