Amitesh0512

Posted on Oct 24 Originally published at amiteshsurwar.com

Neo4j Graph-Based Recommendation Engine: Scaling to 25M DAUs

#neo4j #performancetuning #scalability #graphdatabaserecommendation

Quick Answer

Explore how to design, score, and scale a Neo4j Graph-Based Recommendation Engine that powers high‑throughput matching in modern dating apps.

Real‑Time Graph Scoring

In a high‑velocity matchmaking service, the core pain point isn’t the schema or the persistence layer; it’s the ability to compute relationship‑based scores against a constantly evolving, densely connected graph while keeping per‑request latency under 10 ms. Traditional relational designs treat every interaction as a row and force a cascade of joins, which explode in both CPU and I/O as edges grow into the billions. The architectural decision boils down to: Do you want to pay for expensive graph traversals in a relational engine or ship a graph‑native store that can walk 3‑hop neighborhoods in constant time?

Real‑world Example

Consider a dating platform that serves 25 M daily active users (DAUs) and logs ~3 billion interactions per day. The matching service must generate a fresh top‑10 list for every user every 30 seconds. In a relational stack, each request triggers a 3‑join, a sub‑query for mutual likes, and a window function for ordering, all against a table that cannot be indexed efficiently for the 2‑hop pattern. The result: query times balloon past 200 ms during peak traffic, and the system has to throttle writes to keep the database alive.

Switching to Neo4j, the same logic becomes a single Cypher traversal that hops from the requester to their liked users, then to those users’ liked users, filtering out blocked edges. The graph engine keeps adjacency in memory and can push the traversal down to the kernel, eliminating the join overhead.

Trade‑offs

Latency vs. Consistency: Neo4j’s causal cluster guarantees ACID per transaction but write throughput is bounded by Raft quorum. For write‑heavy workloads, you’ll see a 5‑10 ms write amplification. If your business can tolerate eventual consistency for likes, you can shard writes across read‑replicas and batch them into the core.
Operational Complexity: Managing a 3‑core cluster with read replicas, page cache tuning, and transaction event handlers requires a dedicated Ops team. In contrast, a relational stack can run on a single node with simpler backups.
Developer Productivity: Cypher’s declarative syntax is concise, but complex scoring logic often ends up in Java procedures. This can create a cognitive load split between Cypher and Java, and unit testing those procedures demands a full Neo4j instance.
Cost vs. Scale: A single large‑memory Neo4j instance can hold ~200 M edges for 30 GB of page cache, but scaling beyond that forces a cluster. The cluster’s read replicas add cost, but they also unlock linear read scaling and geographic latency reduction.

Edge Cardinality, Write Patterns, Depth

Assess Edge Cardinality: If you expect < 100 M edges and can tolerate a single node, start with a single‑core Neo4j. If you need > 200 M edges or high read concurrency, move to a causal cluster.
Determine Write Patterns: For bursty interaction events (e.g., a marketing push), buffer writes in Kafka and apply them in bulk via apoc.periodic.iterate to keep the core from throttling.
Choose Traversal Depth: 2‑hop mutual likes are cheap; 3‑hop or beyond quickly become expensive. Profile your Cypher queries with PROFILE and enforce a hard depth limit in the application layer.
Cache Strategy: Hot subgraphs (top 1 M active users) should stay in the page cache. Pre‑compute top‑N matches per user using a scheduled job and store them in Redis with a 30‑second TTL. Invalidate on USER_DELETED or BLOCKED_BY events.
Observability: Enable Neo4j’s kernel metrics, integrate with OpenTelemetry, and surface query latency, page cache hit ratio, and transaction start times in your monitoring stack.
Compliance & Privacy: For GDPR, implement a single DETACH DELETE transaction per user and back up the graph state to a cold store before deletion.

When This Fails in Production

Unbounded Fan‑Out: A celebrity profile that accumulates > 5 M likes creates a hot node that saturates the core. The cluster’s write path becomes a bottleneck, and GC pauses spike.
Stale Cache: A 60 s TTL on Redis allows users to see matches that were deleted or blocked within that window, leading to a spike in error responses.
Back‑Pressure Ignored: Writing directly to Neo4j during a viral event without batching causes the core to throttle, increasing latency beyond SLA and forcing the matchmaking service to timeout.
Incorrect Decay Calibration: Using a decay constant tuned on a small dataset leads to old interactions dominating, reducing the freshness of matches and lowering engagement.

Common Mistakes Engineers Make

Assuming Neo4j’s default page cache is sufficient for hot subgraphs; in reality you need to tune dbms.memory.pagecache.size to 70‑80% of RAM.
Embedding complex scoring logic directly in Cypher; this bloats the query plan and makes debugging impossible.
Neglecting relationship‑type indexes; a missing index on INTERACTS_WITH forces a full scan for each hop.
Over‑sharding by user ID only; this ignores natural locality of interactions, causing cross‑shard joins that the application layer has to stitch.
Under‑estimating the cost of maintaining a causal cluster; read replicas need their own page cache and memory, doubling the hardware footprint.

Better Approach Based on Experience

In a recent 10‑month rollout for a global dating service, we adopted the folLowing pattern:

Hybrid Graph + Cache Engine: The core Neo4j cluster handled write‑heavy events and maintained a 30 GB page cache. We pre‑computed top‑10 matches every 15 seconds and pushed them to Redis. The matchmaking API served from Redis first; on cache miss, it fell back to a lightweight Cypher that only traversed up to 2 hops.
Write Buffering: All interaction events were streamed to Kafka. A microservice consumed the topic in batches of 10 k and applied them via apoc.periodic.iterate with commitSize=2000, keeping the core from throttling during spikes.
Adaptive Decay: We introduced a per‑action decay factor that was auto‑tuned via an online learning loop: every day the system recomputed λ based on the change in match acceptance rates.
Hot‑Node Isolation: For users with > 1 M outgoing edges, we created a lightweight “fan‑page” node that held the fan‑out as a single relationship, and routed all fan‑out queries to a dedicated read‑replica. This reduced GC pressure on the core.
Observability & Alerting: We instrumented the query latency histogram and set an alert on the 99th percentile > 20 ms. This early warning prevented a silent latency drift that had previously caused a 15 % drop in daily active matches.

Result: 10× reduction in per‑user latency, 25 % lower infrastructure cost (thanks to the read‑replica strategy), and a 12 % increase in match acceptance after the adaptive decay tuning.

Dimension	Approach 1	Approach 2	Trade‑offs
Scoring Method	Node2Vec embeddings + cosine similarity	Graph algorithm: Personalized PageRank	Embeddings capture latent preferences but need retraining; PageRank is easier to maintain but less nuanced.
Data Model Design	Flat node types (User, Interest, Location) with relationship types (LIKES, MATCHES)	Hierarchical sub‑nodes (InterestCategory, SubInterest) to model nested preferences	Flat model is simpler and faster for queries; hierarchical offers richer semantics but adds traversal overhead.
Deployment Scale	Neo4j Aura Cloud (managed, auto‑scaling)	Self‑hosted Neo4j Enterprise cluster (manual scaling)	Aura reduces ops but limits custom tuning; cluster gives control but requires ops expertise.
Matching Strategy	Batch pre‑computed recommendations nightly	Real‑time on‑demand queries per user session	Batch is efficient for high volume but stale; real‑time is fresh but heavier load.

Performance Considerations

Page Cache Size: Allocate 70‑80 % of RAM to the page cache; the rest goes to JVM metaspace and OS buffers.
Relationship‑Type Indexes: Enable an index on INTERACTS_WITH and HAS_TAG. For high‑cardinality relationships, consider a composite index with the type property.
Batch Size: For bulk writes, a commitSize of 2000 balances transaction overhead and memory usage.
GC Tuning: Use G1GC with a 100 ms pause target and a maximum heap of 4 GB for the core node to keep GC pauses below 10 ms.
Network Latency: In a causal cluster, the write latency is roughly 2 * networkRoundTripTime + writeCommitTime. Keep the cluster in the same region or use a low‑latency inter‑region link.

Scaling Notes

Vertical scaling hits a ceiling around 200 M edges for a single node; beyond that, a 3‑core causal cluster is the only path to linear read scaling.
Read replicas can be deployed to edge locations (e.g., AWS us‑east‑1, ap‑south‑1) to serve match requests with <10 ms latency. Remember that write traffic still funnels to the core.
When sharding by geography, each shard becomes its own causal cluster. Cross‑shard queries must be performed by the application, so keep the shard boundaries tight (e.g., country or region).
For global services, consider a hybrid approach: a global causal cluster for core interactions and regional read‑only clusters for cached top‑N lists.

Checklist for a Production‑Ready Graph Recommendation Engine

Deploy a 3‑core causal cluster with a 30 GB page cache.
Create relationship‑type indexes for INTERACTS_WITH and HAS_TAG.
Implement a Java procedure for weighted similarity and expose it via Cypher.
Set up a Kafka topic for interaction events and a batch writer using apoc.periodic.iterate.
Schedule a nightly job to pre‑compute top‑10 matches per user and push to Redis.
Integrate Neo4j kernel metrics with OpenTelemetry; alert on dbms.page.cache.hitRatio < 0.95 and query latency > 20 ms.
Implement a USER_DELETED event listener to purge Redis keys immediately.
Run A/B tests on decay constants and adjust λ daily based on match acceptance metrics.
Document hot‑node handling: split fan‑out into a dedicated read‑replica and monitor GC pause times.
Perform regular GDPR compliance drills: delete a user and verify that all incident edges are removed within a single transaction.

Conclusion

Building a Neo4j‑based recommendation engine isn’t a silver bullet; it’s a disciplined trade‑off between graph traversal speed and cluster complexity. By treating the graph as a first‑class citizen, you unlock constant‑time adjacency lookups that relational joins cannot match. The key to production success lies in:

careful page cache tuning
batching writes to avoid core throttling
pre‑computing hot subgraphs and caching them aggressively
monitoring Neo4j‑specific metrics to catch hot‑node regressions early

When you combine these practices, you can serve millions of users with sub‑10 ms latency while keeping infrastructure costs in check. The next step is to prototype the pre‑compute pipeline, instrument the metrics, and iterate on the decay model until your match acceptance rates hit the desired KPI.

DEV Community