Originally published on the Semurg blog: https://one.semurg.io/blog/graph-on-one-server. Every figure below is a row on https://one.semurg.io/benchmark#graph with the release it was measured on.
Graph databases are judged on traversals: start from one node, follow relationships out several hops, and return what you reach. We
ran that question on a public social graph against six graph databases, on one bare-metal server, with every engine's answer checked
against two independent reference implementations before anything was timed. This post has the results, the one place where it is
close, a mistake in our own benchmark kit, and the one command that reruns the whole thing on your own machine.
What we ran
- The data: SNAP's com-Orkut social graph: 3.07 million people and 117.2 million relationships, loaded as published.
- The machine: one Vultr bare-metal server, AMD EPYC 4345P (8 cores, 16 threads), 64 GB of memory, two NVMe disks, no GPU.
- The engines: Semurg, Neo4j, ArangoDB, FalkorDB, FalkorDBLite, ArcadeDB and SurrealDB, each in Docker on that server, one after another, each with every thread, both data disks and no memory cap, its own recommended settings, and its fastest query form.
- Two shapes of use: inside your application (Semurg linked into the client; the other engine's embedded mode where it has one, otherwise its fastest connection) and as a server (each engine through its own fastest connection).
- Three cycles: cold, warm, warm again; the figure is the median of the warm cycles.
One query, 4 hops
Starting from one person, every distinct person within 4 hops (455,604 of them). As a server, Semurg answered in 3.83 ms; ArcadeDB
took 73.14 ms, FalkorDBLite 164.14 ms, FalkorDB 205.89 ms, Neo4j 281.40 ms and SurrealDB 1,484.92 ms. Across those
five server rows that is between 19 and 388 times faster. Inside the application the picture is the same: 3.84 ms for
Semurg against 93.08 ms for ArcadeDB's embedded mode and 1,360.65 ms for SurrealDB's.
Many queries at once
The same 4-hop question from 16 clients at once, each side driven by the same number of clients. As a server, Semurg answered 240.2
queries per second, ArcadeDB 28.0, FalkorDB 8.5 and Neo4j 2.6.
Walks to 131,072 hops
The ladder walks from the same person to every depth from 2 to 131,072 hops, checking each step against the reference. On this graph
everything reachable is reached within a few dozen hops, so past that point every engine is timing one full walk of the reachable
graph. As a server, Semurg's full walk took 95.22 ms; ArcadeDB's took 113.71 ms, FalkorDB's 412.67 ms and Neo4j's
14,033.83 ms. Inside the application Semurg's took 96.88 ms and ArcadeDB's embedded mode 459.74 ms.
The server comparison with ArcadeDB is the close one, and we care about it most. ArcadeDB answers its traversals from what it calls a
Graph Analytical View: an index of the whole graph it builds in memory when the database opens (the build is counted in its load
time). Semurg needs no such build when it opens. The next release carries two speed-ups to the graph walk; in our own test build they took the full walk from 96.4 ms to 81.1 ms on 8 threads (a test build, not a published release), and the published number follows the release.
What did not run, and why
A result we could not judge is shown as held, never dropped. In this run:
- ArangoDB answered the 4-hop question correctly, but it was stopped by the out-of-memory killer during the many-at-once test, and the walk that followed found it already stopped. Our benchmark kit read that as a failure of the walk itself and set aside all of ArangoDB's results, its correct ones included. That was our kit's mistake, not ArangoDB's: the kit now checks that an engine is running before a walk starts, and ArangoDB is re-run on the next release. Until then its cells on the benchmark page read results pending.
- FalkorDBLite's many-at-once run had failed requests and SurrealDB's had a zero median, so neither has a throughput figure; load times are held for every engine until each load has its run-to-run band.
Bigger than memory
Every domain we publish runs four classes: in the application and as a server, each on a dataset that fits in memory and one that
does not. This graph fits. The bigger-than-memory run uses SNAP's com-Friendster, a far larger social graph, with each engine held to a
fixed memory budget; its reference answers are still being computed, and the results go on the benchmark page when they finish.
Run it yourself
The kit that produced every number above is published. On a Linux machine with Docker, one command runs the graph benchmark for every
engine: curl -fsSL https://one.semurg.io/bench | sudo bash -s -- --graph. On a machine like ours it needs about 22 GB of free disk,
about 48 GB of memory for the largest engine, and about eight and a half hours (test tree, not a published release). Every figure,
every held row and the reason for each is on the benchmark page.
Top comments (0)