DEV Community: Satyam Shree

A Practical Guide to Temporal Versioning in Neo4j: Nodes, Relationships, and Historical Graph Reconstruction

Satyam Shree — Fri, 05 Dec 2025 08:38:36 +0000

Modern graph databases often represent dynamic systems: applications evolving over time, relationships appearing and disappearing, and entities acquiring new attributes as data changes.
When the underlying graph is user-facing, maintaining a complete history of nodes and relationships becomes a critical capability.

This article presents a production-grade, bitemporal versioning model for Neo4j, supporting:

Accurate historical reconstruction
Time-travel queries
Temporal relationship tracking
Efficient ingestion
Minimal impact on existing “current” queries

The approach is designed for high-read systems where graph state changes incrementally and users must view data at any point in time.

1. Design Goals

A temporal graph versioning system must satisfy the following constraints:

1.1 Minimal disruption to existing queries

Everyday queries (fetching the “current” graph) must remain simple:

MATCH (n) WHERE NOT n:Deleted
MATCH ()-[r]->() WHERE r.Status = "Active"

No complex temporal logic in the majority of queries.

1.2 Complete bitemporal representation

Every node or relationship must encode:

StartDate — when it became valid
EndDate   — when it stopped being valid (NULL = current)

This enables time-travel queries and historical reconstruction.

1.3 Deterministic version merging

Each node and relationship must have a stable primary key so the ingestion pipeline can decide:

Should this entity be created?
Should it be updated?
Should old versions be closed?

1.4 Efficient deletion detection

We cannot “blindly” delete nodes. Instead, the pipeline must:

Mark entities touched in this ingestion cycle (via lastUpdated)
Infer deletions by comparing against the process date

1.5 Neo4j MERGE limitations must be respected

Neo4j does not support:

MERGE (a)-[r:LINK {EndDate: NULL}]->(b)

This is why relationships use a Status property rather than attempting NULL-based merges.

2. Data Model

2.1 Versioned Nodes

Each logical entity is represented as multiple immutable node versions:

(:Entity {
    Id: "E123",
    StartDate: datetime("2024-01-10T00:00:00Z"),
    EndDate: null,
    lastUpdated: datetime("2024-12-01T10:00:00Z")
})

When a node becomes invalid:

EndDate is set
:Deleted label is added

ASCII Diagram

+------------------+        +------------------+
| Entity (v1)      | ----> | Entity (v2)      |
| Id: E123         |       | Id: E123         |
| Start: T1        |       | Start: T2        |
| End: T2          |       | End: null        |
| Label: Deleted   |       | Label: <none>    |
+------------------+        +------------------+

2.2 Versioned Relationships

Like nodes, relationships also maintain temporal state:

(a)-[:LINK {
    Id: "R987",
    StartDate: datetime("2024-01-10T00:00:00Z"),
    EndDate: null,
    Status: "Active",
    lastUpdated: datetime("2024-12-01T10:00:00Z")
}]->(b)

Why we need `Status`

Neo4j cannot MERGE on EndDate = NULL, so we use:

Status = "Active"
Status = "Deleted"

This provides a safe, deterministic merge target.

3. Ingestion Architecture (Multi-Phase)

Your ingestion pipeline comprises three phases, ensuring consistent versioning.

+-----------------------------------------------------+
|                Ingestion Pipeline                   |
+-----------------------------------------------------+
|                                                     |
| Phase 1: Nodes      → Create or update nodes        |
| Phase 2: Links      → Create or update relationships|
| Phase 3: Clean-up   → Close missing versions        |
|                                                     |
+-----------------------------------------------------+

3.1 Phase 1 — Node Ingestion

For each incoming node:

MERGE by Id
If node exists and attributes differ → close old version, create new
Update lastUpdated = processTime

Cypher (simplified)

MERGE (n:Entity {Id: $id})
ON MATCH SET
    n.lastUpdated = $processDate
ON CREATE SET
    n.StartDate = $processDate,
    n.lastUpdated = $processDate

When detecting changes, the ingestion process may:

Set EndDate on the previous version
Add :Deleted
Create a fresh version

3.2 Phase 2 — Relationship Ingestion

For each incoming relationship:

MATCH (a:Entity {Id: $src})
MATCH (b:Entity {Id: $dst})

MERGE (a)-[r:LINK {Id: $id}]->(b)
ON MATCH SET
    r.lastUpdated = $processDate
ON CREATE SET
    r.StartDate = $processDate,
    r.Status = "Active",
    r.lastUpdated = $processDate

If a relationship changed (attribute changes), the pipeline must:

Mark old relationship as:
- r.EndDate = $processDate
- r.Status = "Deleted"
Create a new version:
- StartDate = $processDate
- Status = "Active"

3.3 Phase 3 — Version Closure (Deletion Detection)

After phases 1 & 2, you detect deletions:

Any node whose lastUpdated != processDate is no longer valid:

MATCH (n:Entity)
WHERE n.lastUpdated <> $processDate AND NOT n:Deleted
SET n.EndDate = $processDate, n:Deleted

Same for relationships:

MATCH ()-[r:LINK]->()
WHERE r.lastUpdated <> $processDate AND r.Status = "Active"
SET r.EndDate = $processDate, r.Status = "Deleted"

This allows ingestion to determine “missing = deleted” without manual intervention.

4. Querying the Current Graph

Your versioning design enables extremely simple “current state” queries:

Nodes

MATCH (n:Entity)
WHERE NOT n:Deleted
RETURN n

Relationships

MATCH (a)-[r:LINK]->(b)
WHERE r.Status = "Active"
RETURN a, r, b

Minimal logic.
High performance.
Clean integration with UI/API.

5. Querying Historical Snapshots

To reconstruct graph state for a given timestamp T:

Nodes

MATCH (n:Entity)
WHERE n.StartDate <= $T AND (n.EndDate IS NULL OR n.EndDate > $T)
RETURN n

Relationships

MATCH (a)-[r:LINK]->(b)
WHERE r.StartDate <= $T AND (r.EndDate IS NULL OR r.EndDate > $T)
RETURN a, r, b

This produces an accurate, complete view of the graph at time T.

6. Go + Neo4j Driver Pseudo-code

Below is idiomatic Go pseudocode demonstrating versioned ingestion logic.

6.1 Creating/Updating a Node

func ingestNode(id string, props map[string]interface{}, processDate time.Time) {
    session := driver.NewSession(neo4j.SessionConfig{AccessMode: neo4j.AccessModeWrite})
    defer session.Close()

    _, err := session.WriteTransaction(func(tx neo4j.Transaction) (interface{}, error) {
        params := map[string]interface{}{
            "id":          id,
            "processDate": processDate,
            "props":       props,
        }

        query := `
            MERGE (n:Entity {Id: $id})
            ON MATCH SET 
                n.lastUpdated = $processDate
            ON CREATE SET 
                n.StartDate = $processDate,
                n.lastUpdated = $processDate,
                n += $props
        `
        return tx.Run(query, params)
    })
    if err != nil {
        log.Fatal(err)
    }
}

6.2 Closing Stale Nodes

func closeStaleNodes(processDate time.Time) {
    session := driver.NewSession(neo4j.SessionConfig{AccessMode: neo4j.AccessModeWrite})
    defer session.Close()

    _, err := session.Run(`
        MATCH (n:Entity)
        WHERE n.lastUpdated <> $processDate AND NOT n:Deleted
        SET n.EndDate = $processDate, n:Deleted
    `, map[string]interface{}{
        "processDate": processDate,
    })
    if err != nil {
        log.Fatal(err)
    }
}

7. Common Pitfalls & How This Model Solves Them

7.1 MERGE cannot match on NULL

Many developers attempt:

MERGE (a)-[r:LINK {EndDate: NULL}]->(b)

This does not work in Neo4j.

Solution:
Use Status for deterministic relationship merging.

7.2 Avoid overwriting nodes

You never update older versions.
Instead:

Close old version (EndDate, :Deleted)
Create new version

This preserves full history.

7.3 Efficient current-state filtering

Instead of comparing timestamps, we rely on:

NOT n:Deleted
r.Status = "Active"

These are extremely fast and index-friendly.

8. Performance Considerations

Indexes

You should index:

Node: Entity(Id)
Node: Entity(Deleted)
Rel: LINK(Id)
Rel: LINK(Status)

Batching

Batching ingestion improves performance substantially.

Avoiding deep history scans

Historical reconstruction always uses date filtering, not traversal of version chains.

9. Summary of the Model

Nodes:
- Id
- StartDate
- EndDate
- lastUpdated
- :Deleted label

Relationships:
- Id
- StartDate
- EndDate
- Status ("Active"/"Deleted")
- lastUpdated

Ingestion phases:

1. Node ingest
2. Relationship ingest
3. Close stale versions

Key benefits:

Clean, fast “current” queries
Complete historical accuracy
Deterministic version merging
No risk of MERGE-on-NULL issues
Proven scalability

10. Conclusion

Temporal versioning in Neo4j is not just a schema change—it is an architectural decision that affects ingestion pipelines, storage models, and query semantics.
The strategy described above enables:

Efficient ingestion without overwriting data
Simple current-state queries
Accurate time-travel analysis
Clean separation of active vs. historical data
A scalable, deterministic versioning model

This design supports both high-performance applications and advanced tooling such as diffing, history exploration, and lineage tracking.

If you are building any graph system where state changes matter, this approach provides a strong, production-grade foundation for temporal graph modeling.

[Boost]

Satyam Shree — Tue, 11 Nov 2025 04:15:57 +0000

Building a Mini Kafka in Go — My Journey Creating go-pub-sub

Satyam Shree ・ Nov 10

#go #grpc #pubsub #kafka

Building a Mini Kafka in Go — My Journey Creating go-pub-sub

Satyam Shree — Mon, 10 Nov 2025 13:15:03 +0000

🌀 Building a Mini Kafka in Go — My Journey Creating `go-pub-sub`

A hands-on dive into distributed messaging, gRPC streams, and pluggable storage — all in Go.

👋 Intro

Over the past few weeks, I’ve been exploring how systems like Kafka and Pulsar handle messaging, partitioning, and consumer groups. Instead of just reading papers, I wanted to build one myself — from scratch — to truly understand how Pub/Sub systems work under the hood.

That’s how go-pub-sub was born — a lightweight, gRPC-based Publish–Subscribe broker written entirely in Go.

It started as a learning project but grew into a deployable, extensible system that can run locally or in containers.

⚙️ What is `go-pub-sub`?

go-pub-sub is a mini distributed broker that lets you:

Publish messages over gRPC
Stream them to consumers (with acknowledgments)
Organize them by topics, partitions, and consumer groups
Plug in different storage layers — in-memory, Redis, or Postgres
Expose Admin APIs to manage topics dynamically

It’s meant to be small, clear, and hackable — a Kafka-like learning platform.

🧩 Architecture Overview

Publisher(s) → gRPC (Publish)
Consumer(s)  ← gRPC Stream (Subscribe)
↳ Ack → gRPC (Ack)
↳ Admin APIs → CreateTopic, ListTopics, TopicInfo

The broker exposes two main services:

PubSub — handles Publish, Subscribe, Ack
Admin — topic creation, inspection, listing

Internally, the broker consists of:

Broker layer → core logic for pub/sub, offsets, consumer groups
Storage layer → in-memory / Redis / Postgres adapters
Middleware → logging, recovery, optional auth
Config layer → YAML-based configuration
Clients → simple Go-based producer and consumer binaries

💡 Why I Built It

Like most Go devs, I’d read about channels, goroutines, and concurrency patterns — but hadn’t seen them used in distributed systems.

Building a Pub/Sub broker was the perfect playground:

gRPC streams → long-lived connections
Consumer groups → concurrency and coordination
Storage adapters → interfaces and dependency inversion
Message acknowledgment → reliable delivery patterns
Docker-based testing → reproducibility

And honestly, nothing teaches you distributed thinking like debugging your own broker 😅.

🧱 Core Design Concepts

Topics and Partitions

Each topic is split into one or more partitions for parallelism.

Messages are appended to a partition and assigned an incremental offset.

Consumer Groups

Multiple consumers in the same group share partitions.

Each group tracks committed offsets per partition, ensuring at-least-once delivery.

Storage Backends

Memory: great for tests and local development
Redis: fast ephemeral storage (supports consumer state tracking)
Postgres: durable persistence with offset commits and retention

🧰 Running It

1️⃣ Run the broker (in-memory)

go run ./cmd/server --config=config/config.yaml

2️⃣ Start a consumer

go run ./client/consumer --addr=localhost:50051 --topic=events --group=g1

3️⃣ Publish messages

go run ./client/producer --addr=localhost:50051 --topic=events --msg="hello world"

Output:

INFO  2025/11/10 22:31:12 📤 published topic=events partition=0 offset=42
INFO  2025/11/10 22:31:12 ➡ received topic=events partition=0 offset=42 value=hello world
INFO  2025/11/10 22:31:12 ✅ acked offset=42

🧪 Smoke Testing (Docker Style)

I wrote a full bash smoke test that spins up a broker container, waits for readiness using logs, and runs a real consumer + producer pair locally.

./scripts/smoke_with_docker.sh -n 100 -i go-pub-sub:local -p 50051

It verifies that all messages are received and acknowledged, even across partitioned topics.

When it prints ✅ SUCCESS, you know your broker is behaving like a pro.

🗄️ Production-Ready Direction

Right now, go-pub-sub supports three storage types:
| Backend | Type | Use Case |
|----------|------|----------|
| memory | In-memory | Fast testing |
| redis | Ephemeral | Temporary persistence |
| postgres | Durable | Production use |

I’m currently building a Postgres-backed store that supports:

atomic offset assignment via SQL sequences
durable message and offset persistence
configurable retention cleanup
per-topic partition tables for scalability

This will make the broker durable and crash-safe while keeping it minimal.

🔍 Lessons Learned

gRPC streams are powerful but tricky — they demand careful connection management.
Consumer group coordination can get complex even with small datasets.
Interfaces are your friend — swapping backends was easy once interfaces were clean.
Dockerized smoke testing saved hours of debugging inconsistencies.
Go’s concurrency model shines for pub/sub patterns.

🚀 What’s Next

🗄️ Complete Postgres backend (durable persistence)
🔎 Add /health and /metrics endpoints
🧹 Implement retention cleanup goroutine
🌐 Add gRPC-Web / WebSocket gateway for browser-based apps
🧰 Build a simple chat app demo using go-pub-sub as backend

📚 Tech Stack

Language: Go 1.25
Transport: gRPC
Storage: In-memory / Redis / Postgres
Orchestration: Docker Compose
Testing: grpcurl, smoke scripts, Go integration tests

💬 Wrap-up

go-pub-sub started as a weekend curiosity and ended up teaching me more about distributed coordination than any book could.

It’s small, educational, and — with Postgres — production-capable for internal systems or side projects.

If you’re learning gRPC, Go interfaces, or concurrency patterns, this is one of those projects that will click everything into place.

🧠 Repo: github.com/satya-sudo/go-pub-sub

📦 Docker image (coming soon): docker pull satya-sudo/go-pub-sub

Built with ❤️ in Go — because understanding systems is better than just using them.

💡 Tags

#go #grpc #opensource #pubsub

If you’ve ever wanted to understand Kafka’s internals by building one, check out the repo and try running the smoke test!

Drop your thoughts, feature ideas, or PRs in the comments 👇

Sometimes small ideas teach you the biggest lessons.

Satyam Shree — Sat, 25 Oct 2025 04:45:21 +0000

🧠 Go-URL: Building a Lightweight, Scalable URL Shortener in Go

Sometimes small projects end up teaching you the biggest lessons.

Go-URL started as a simple weekend idea — “let’s build a basic URL shortener in Go.”

But as I started designing it, I realized it was a perfect opportunity to explore system design, caching, persistence, and scalability — all within a compact, real-world problem space.

🚀 The Idea

The goal was simple:

Build a fast, reliable URL shortener with:

Clean architecture
Stateless services
High scalability
Low latency

And do it all using technologies I love working with: Go, Redis, PostgreSQL, and Kubernetes.

⚙️ Tech Stack Overview

Component	Purpose
Go (Golang)	Core backend logic, API server, concurrency handling
Redis	Fast caching for redirect lookups
PostgreSQL	Persistent storage for original and shortened URLs
Kubernetes	Deployment, load balancing, auto-scaling
Docker	Containerization for portability

🧩 System Design

1. Stateless API Layer

The API is built using Go’s net/http package and structured around clean architecture principles — keeping business logic, repository layers, and handlers decoupled.

This design allows horizontal scaling — multiple instances can run behind a load balancer with no shared state.

2. Redis for Speed

URL redirection is read-heavy.

Instead of querying PostgreSQL for every hit, Redis caches the mappings:

short_url -> original_url

Whenever a new short URL is created, it’s stored in both Redis and PostgreSQL.

On lookups, if Redis misses, the record is fetched from the DB and re-cached — a classic read-through caching pattern.

3. PostgreSQL for Durability

Redis is blazing fast but volatile.

To ensure persistence, PostgreSQL stores all URL mappings permanently, complete with timestamps and access logs.

A background worker syncs write operations asynchronously to maintain API responsiveness.

4. Kubernetes for Scalability

I containerized the service with Docker and deployed it on Kubernetes.

K8s handles:

Load balancing across pods
Auto-scaling based on CPU/memory metrics
Rolling updates without downtime

This setup ensures Go-URL can handle sudden traffic spikes smoothly.

🧠 Key Challenges and Learnings

Balancing cache and consistency

Designing the Redis sync flow without introducing stale data was tricky. A hybrid TTL-based strategy worked best.
Async syncs without blocking requests

I built a simple worker queue in Go to offload DB writes, improving response times significantly.
Observability

Added basic logging and metrics using Go’s expvar and structured logs — it helped a lot during debugging and load testing.
Scaling on Kubernetes

Watching pods auto-scale under simulated load was incredibly satisfying — a real-world validation of clean design.

💡 Performance Results

After load testing with hey and wrk, the results were impressive:

Median response time: <10ms (with Redis warm)
Sustained 10K+ RPS across multiple instances
Zero downtime during redeployments via Kubernetes rolling updates

✨ What I Took Away

Go-URL reminded me that even the simplest systems can be architecturally deep if you think about them right.

It taught me:

How clean architecture helps with maintainability.
Why caching layers are crucial for scalability.
How Kubernetes orchestration simplifies deployment complexity.
And most importantly — how small design decisions impact performance at scale.

🔗 Check it Out

You can explore the project here:

👉 GitHub: Go-URL

If you’re into backend systems, Go, or distributed design — I’d love to hear how you would scale or extend this system further.

DEV Community: Satyam Shree

A Practical Guide to Temporal Versioning in Neo4j: Nodes, Relationships, and Historical Graph Reconstruction

1. Design Goals

1.1 Minimal disruption to existing queries

1.2 Complete bitemporal representation

1.3 Deterministic version merging

1.4 Efficient deletion detection

1.5 Neo4j MERGE limitations must be respected

2. Data Model

2.1 Versioned Nodes

ASCII Diagram

2.2 Versioned Relationships

Why we need Status

3. Ingestion Architecture (Multi-Phase)

3.1 Phase 1 — Node Ingestion

Cypher (simplified)

3.2 Phase 2 — Relationship Ingestion

3.3 Phase 3 — Version Closure (Deletion Detection)

4. Querying the Current Graph

Nodes

Relationships

5. Querying Historical Snapshots

Nodes

Relationships

6. Go + Neo4j Driver Pseudo-code

6.1 Creating/Updating a Node

6.2 Closing Stale Nodes

7. Common Pitfalls & How This Model Solves Them

7.1 MERGE cannot match on NULL

7.2 Avoid overwriting nodes

7.3 Efficient current-state filtering

8. Performance Considerations

Indexes

Batching

Avoiding deep history scans

9. Summary of the Model

10. Conclusion

[Boost]

Building a Mini Kafka in Go — My Journey Creating go-pub-sub

Satyam Shree ・ Nov 10

Building a Mini Kafka in Go — My Journey Creating go-pub-sub

🌀 Building a Mini Kafka in Go — My Journey Creating go-pub-sub

👋 Intro

⚙️ What is go-pub-sub?

🧩 Architecture Overview

💡 Why I Built It

🧱 Core Design Concepts

Topics and Partitions

Consumer Groups

Storage Backends

🧰 Running It

1️⃣ Run the broker (in-memory)

2️⃣ Start a consumer

3️⃣ Publish messages

🧪 Smoke Testing (Docker Style)

🗄️ Production-Ready Direction

🔍 Lessons Learned

🚀 What’s Next

📚 Tech Stack

💬 Wrap-up

💡 Tags

Sometimes small ideas teach you the biggest lessons.

🧠 Go-URL: Building a Lightweight, Scalable URL Shortener in Go

🚀 The Idea

⚙️ Tech Stack Overview

🧩 System Design

1. Stateless API Layer

2. Redis for Speed

3. PostgreSQL for Durability

4. Kubernetes for Scalability

🧠 Key Challenges and Learnings

💡 Performance Results

✨ What I Took Away

🔗 Check it Out

#golang #backend #systemdesign #redis #kubernetes #scalability #learningbybuilding

Why we need `Status`

🌀 Building a Mini Kafka in Go — My Journey Creating `go-pub-sub`

⚙️ What is `go-pub-sub`?