DEV Community: Archit Agarwal

Your Delete Button Is Lying to You in Production - Here's the Data Structure That Fixes It

Archit Agarwal — Sun, 31 May 2026 18:30:00 +0000

When you hit delete and the data comes right back, the problem isn't a bug. It's a design gap that only shows up at scale.

You've seen this before.

A user gets banned. You delete their entry from the database. The system confirms the deletion. Fifteen minutes later, another server in your cluster still shows them as active. You delete again. It comes back again. You feel like you're in a horror movie.

Or consider a simpler scenario: your team is running a feature-flag system across multiple data centres. Someone deprecates a flag - it needs to die permanently. But your distributed nodes keep merging state, and the deleted flag keeps resurrecting. It's not malicious. It's just physics.

This is one of the most infuriating classes of bugs in distributed systems: deletes that don't stick.

The good news? This has a known cause, a clean mental model, and a data structure purpose-built to solve it - the 2-Phase Set (2P-Set).

Decoding the Problem: Why Deletes Are Hard at Scale

On a single server, deletes are easy. You remove a row. Done. The authoritative truth lives in one place.

The moment you scale horizontally - multiple nodes, multiple replicas, multiple data centres - that single source of truth disappears. Now you have competing sources of truth, and they need to sync up periodically. This sync is called reconciliation or merge.

Here's where things get interesting.

When two nodes merge their state, what happens to a delete? Node A deleted a record. Node B never got that delete message (maybe a network partition, maybe just a delayed sync). Node B still has the record. When A and B merge - Node B's "present" wins. The delete is gone.

This is called the delete wins vs add wins problem, and it's not a theoretical edge case. It's what happens at 3am when your on-call engineer is trying to figure out why a banned user is still sending messages.

The naive fix - timestamps - doesn't help much either. "Most recent write wins" sounds clean until you realise clocks across distributed nodes are never perfectly in sync. You end up with a different class of inconsistency.

The scale-triggered part matters here: on a single node, you never see this. On two nodes with reliable sync, you might not notice. With ten nodes, partial connectivity, and concurrent writes - the problem is unavoidable. This isn't a bug that appears in development. It's a bug that appears in production, under load, after your third customer complaint.

How to Fix It: The Two-Toybox Mental Model

Before we talk algorithms, let me give you a mental model that will make everything else obvious.

Imagine you're a kid with two toyboxes.

The first one is the Add-box. Any toy you want in your collection goes here. You can keep adding things indefinitely.

The second one is the Remove-box. When you're done with a toy - permanently done - you move it here. Not back to the shelf. Not to a friend. Into the Remove-box, and it stays there forever.

The toys you actually "have" at any given moment are: everything in the Add-box that is NOT in the Remove-box.

That's it. That's the 2P-Set.

The magic of this model is what happens when two kids merge their toyboxes. Alice has an Add-box and a Remove-box. Bob has an Add-box and a Remove-box. When they combine:

Combined Add-box = Alice's Add-box ∪ Bob's Add-box
Combined Remove-box = Alice's Remove-box ∪ Bob's Remove-box
Final collection = Combined Add-box − Combined Remove-box

Notice something critical: once a toy lands in any Remove-box across any node, that removal propagates everywhere during the next merge - and it can never be undone. The remove is permanent and irrevocable.

This is why 2P-Set works for the banned-user problem. You don't just delete the user. You move them into the Remove-set. That Remove-set is a grow-only set (just like the G-Set from Episode 4). It only ever gets bigger, never smaller. So when nodes merge, removals always survive.

Real-world rule of thumb: Use a 2P-Set when you have a distributed collection where deletions must be permanent and must survive node merges.

Where 2P-Set Fits Perfectly

Permanent bans and blacklists. Add users to the Allow-set on signup. Move them to the Ban-set on moderation action. Any node that gets the update will propagate the ban everywhere. No re-emergence after sync.

Feature flags with permanent deprecation. Your config system adds feature keys freely. When a flag is killed, it goes into the removed-set. Even if a stale node briefly re-surfaces it, the next merge corrects course.

Service registry tombstones. A microservice is decommissioned. You don't just unregister it - you add it to the removed-set. Any service-discovery node that tries to re-register the old name gets blocked, because the tombstone is present.

Access revocation. A user loses a permission. That revocation needs to hold across all nodes, even if one node was partitioned when the revocation happened. The Remove-set is how you make that hold.

The Trade-off You Need to Know About

2P-Set has one hard constraint that you shouldn't try to work around: once removed, the element is gone forever. You cannot re-add a removed element.

This sounds limiting, but it's actually the source of the guarantee. If you could re-add something after removal, you'd re-introduce the "whose update wins" problem you were trying to escape.

The practical implication: if you need to "re-add" something, model it differently. Don't re-add the same logical identifier - create a new one. A banned user who creates a new account gets a new user ID, not a resurrection of the old one.

This is a clean design constraint that forces your domain model to be explicit about intent.

The Algorithm

The formal structure is straightforward, and once you see it, it clicks immediately.

State: Each node maintains two grow-only sets:

A - the Add-set (all elements ever added)
R - the Remove-set (all elements ever removed)

Operations:

Add(e):

Precondition: e must not be in R (can't re-add a removed element)
Effect: Insert e into A

Remove(e):

Precondition: e must be in A (can only remove what was added)
Effect: Insert e into R

Lookup(e):

Returns true if e ∈ A and e ∉ R
Equivalently: e ∈ (A − R)

Value():

Returns A − R (all elements in Add-set not in Remove-set)

Merge(local, remote):

merged.A = local.A ∪ remote.A
merged.R = local.R ∪ remote.R

Both A and R are G-Sets under the hood - they only grow. Merging G-Sets via union is always safe, always commutative, always idempotent. Apply the merge in any order, any number of times, you get the same result. That's the CRDT guarantee.

Why this is conflict-free: The only operation that's "destructive" is a remove. But removes go into R, and R only grows. So across any merge, a removal that happened on any node will eventually appear in every node's R-set. The deletion propagates monotonically. There is no conflict to resolve.

Go Implementation

package crdt

import "sync"

// TwoPhaseSet implements a 2P-Set CRDT.
// Once an element is removed, it can never be re-added.
type TwoPhaseSet[T comparable] struct {
    mu        sync.RWMutex
    addSet    map[T]struct{}
    removeSet map[T]struct{}
}

func NewTwoPhaseSet[T comparable]() *TwoPhaseSet[T] {
    return &TwoPhaseSet[T]{
        addSet:    make(map[T]struct{}),
        removeSet: make(map[T]struct{}),
    }
}

// Add inserts an element into the set.
// Returns false if the element was previously removed - once removed, it stays removed.
func (s *TwoPhaseSet[T]) Add(item T) bool {
    s.mu.Lock()
    defer s.mu.Unlock()

    if _, removed := s.removeSet[item]; removed {
        return false // 2P-Set invariant: removed elements cannot be re-added
    }
    s.addSet[item] = struct{}{}
    return true
}

// Remove moves an element from the add-set to the remove-set.
// The element must have been added first. Returns false otherwise.
func (s *TwoPhaseSet[T]) Remove(item T) bool {
    s.mu.Lock()
    defer s.mu.Unlock()

    if _, exists := s.addSet[item]; !exists {
        return false // can only remove what has been added
    }
    s.removeSet[item] = struct{}{}
    return true
}

// Contains returns true if the element is in the add-set but not the remove-set.
func (s *TwoPhaseSet[T]) Contains(item T) bool {
    s.mu.RLock()
    defer s.mu.RUnlock()

    _, inAdd := s.addSet[item]
    _, inRemove := s.removeSet[item]
    return inAdd && !inRemove
}

// Items returns a snapshot of all live elements (add-set minus remove-set).
func (s *TwoPhaseSet[T]) Items() []T {
    s.mu.RLock()
    defer s.mu.RUnlock()

    items := make([]T, 0, len(s.addSet))
    for item := range s.addSet {
        if _, removed := s.removeSet[item]; !removed {
            items = append(items, item)
        }
    }
    return items
}

// Merge combines another 2P-Set into this one.
// Both add-sets and remove-sets are merged via union - safe, idempotent, commutative.
func (s *TwoPhaseSet[T]) Merge(other *TwoPhaseSet[T]) {
    other.mu.RLock()
    defer other.mu.RUnlock()

    s.mu.Lock()
    defer s.mu.Unlock()

    for item := range other.addSet {
        s.addSet[item] = struct{}{}
    }
    for item := range other.removeSet {
        s.removeSet[item] = struct{}{}
    }
}

Walkthrough with the banned-user scenario:

func main() {
    nodeA := crdt.NewTwoPhaseSet[string]()
    nodeB := crdt.NewTwoPhaseSet[string]()

    // Both nodes add the user during normal operation
    nodeA.Add("user:42")
    nodeB.Add("user:42")

    // Node A bans the user - this is the remove
    nodeA.Remove("user:42")

    // Network partition: Node B hasn't received the ban yet
    fmt.Println(nodeB.Contains("user:42")) // true - B doesn't know yet

    // Sync happens - B merges A's state
    nodeB.Merge(nodeA)

    // Now the ban has propagated
    fmt.Println(nodeB.Contains("user:42")) // false - ban survived the merge

    // Trying to re-add the banned user on Node B fails
    fmt.Println(nodeB.Add("user:42")) // false - once removed, gone forever
}

The merge doesn't care about the order of operations or how many times you call it. The ban is permanent.

Java Implementation


import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;

public class TwoPhaseSet<T> {

    private final Set<T> addSet = ConcurrentHashMap.newKeySet();
    private final Set<T> removeSet = ConcurrentHashMap.newKeySet();

    /**
     * Adds an element to the set.
     * Returns false if the element was previously removed - 2P-Set invariant.
     */
    public boolean add(T item) {
        if (removeSet.contains(item)) {
            return false; // once removed, cannot be re-added
        }
        return addSet.add(item);
    }

    /**
     * Removes an element permanently.
     * Element must have been added first; returns false otherwise.
     */
    public boolean remove(T item) {
        if (!addSet.contains(item)) {
            return false; // can only remove what was added
        }
        return removeSet.add(item);
    }

    /**
     * Returns true if the element is in addSet but not in removeSet.
     */
    public boolean contains(T item) {
        return addSet.contains(item) && !removeSet.contains(item);
    }

    /**
     * Returns a snapshot of all live elements (addSet minus removeSet).
     */
    public Set<T> items() {
        Set<T> result = new HashSet<>(addSet);
        result.removeAll(removeSet);
        return Collections.unmodifiableSet(result);
    }

    /**
     * Merges another 2P-Set into this one via union of both add and remove sets.
     * Safe to call multiple times - idempotent and commutative.
     */
    public void merge(TwoPhaseSet<T> other) {
        addSet.addAll(other.addSet);
        removeSet.addAll(other.removeSet);
    }
}

Walkthrough:


public class Main {
    public static void main(String[] args) {
        TwoPhaseSet<String> nodeA = new TwoPhaseSet<>();
        TwoPhaseSet<String> nodeB = new TwoPhaseSet<>();

        // Both nodes add the feature flag during rollout
        nodeA.add("flag:new-checkout");
        nodeB.add("flag:new-checkout");

        // Flag is deprecated - permanently removed on Node A
        nodeA.remove("flag:new-checkout");

        // Node B is partitioned - it still sees the flag
        System.out.println(nodeB.contains("flag:new-checkout")); // true

        // Sync: B merges A's state
        nodeB.merge(nodeA);

        // Deprecation has propagated
        System.out.println(nodeB.contains("flag:new-checkout")); // false

        // Someone tries to re-add the deprecated flag on Node B
        System.out.println(nodeB.add("flag:new-checkout")); // false - tombstone wins
    }
}

The Java version uses ConcurrentHashMap.newKeySet() for thread-safe operations without explicit locking - appropriate for read-heavy workloads where you're merging state from background sync threads.

Comparing to What Came Before

You've seen the G-Set in Episode 4 - it only grows, never shrinks. Simple and elegant, but useless if you need to remove anything.

The 2P-Set is a direct evolution: you layer a second G-Set (the Remove-set) on top. The Remove-set is your delete log, and it only ever grows. Deletions become a monotonic write rather than a destructive one. That's the insight.

The One Catch Worth Repeating

The Remove-set never shrinks. In a long-lived system with lots of deletions, this means your Remove-set grows indefinitely. You can't garbage-collect it without coordination - because the whole point is that it outlives any single node.

For most use cases (bans, tombstones, feature deprecations) the Remove-set is small and this is fine. For high-churn systems where elements are added and removed at high frequency, 2P-Set isn't the right tool. That's where the OR-Set comes in - and that's exactly what Episode 6 covers.

What's Next

The next piece steps back from the counters entirely.

What if instead of tracking numbers, you need to track membership - items in a set, users in a group, tags on a document?
The same concurrent update problems apply. And a surprisingly simple grow-only constraint solves the basic case cleanly.

Episode 6: OR sets, solves the re-add problem without breaking conflict-freedom. It does this using a concept called unique tags, and it's the most practically useful set CRDT for general-purpose distributed state.

The full series:

Episode 1: G-Counter - why YouTube's view count is approximate
Episode 2: PN-Counter - Reddit karma and the two-notebook trick
Episode 3: Bounded Counter - inventory floors and distributed permission
Episode 4: G-Set - grow-only sets
Episode 5: 2P-2 Phase sets (you're here)
Episode 6: OR-Set - the fix for "deleted but still there"
Episode 7: The full picture - when to use CRDTs and when to walk away

Subscribe to the newsletter to get each episode as it drops.
If you're building distributed systems and want to think through invariant enforcement, CRDT design, or distributed architecture with someone who has been deep in this space, I do 1:1 sessions on Topmate.

You're Writing Conflict Resolution Code for Data That Can Never Conflict

Archit Agarwal — Tue, 26 May 2026 04:00:00 +0000

You've built complex sync logic, last-write-wins strategies, and vector clocks to handle merges between distributed nodes. But for append-only data — events, logs, audit trails — none of that conflict resolution was ever necessary. At scale, that accidental complexity quietly drops data, creates race conditions, and pages you at 3 AM. There's a data structure that makes the entire problem disappear.

Here's a scenario that plays out more often than people admit.

You're running a multi-region analytics service. Writes come in from Singapore and Amsterdam simultaneously. Both nodes accept events independently — which is correct, because you don't want a transatlantic network hiccup to stop recording user activity. Then the nodes sync.

Now what? Someone has to decide what the "merged" state looks like. Last-write-wins? You'll silently drop events that lost the timestamp race. Vector clocks? Now you're maintaining causal history for every key. Operational transforms? You've just signed up to maintain a distributed consensus protocol. Each of these approaches is a moving part that can fail, drift, or produce surprising behavior under weird race conditions.

Here's what makes this painful: the data was never actually conflicting. An event recorded in Singapore and an event recorded in Amsterdam are two different facts about the world. There's no conflict to resolve. You invented the complexity by reaching for a data structure that was never the right fit.

For append-only data, there's a data structure that makes the entire problem structurally impossible: the Grow-only Set, or G-Set. It's one of the simplest CRDTs (Conflict-free Replicated Data Types), and once you understand it, you'll recognize the pattern everywhere — and stop reaching for solutions to problems you don't actually have.

Let's pull this apart properly.

The Problem: Two Replicas, One Truth, Zero Agreement

To understand why G-Sets matter, you need to feel the pain they solve first.

Imagine you're running a user analytics service. You have two data center nodes — call them Node A (Singapore) and Node B (Amsterdam). Both are recording user events. Both accept writes independently so that a network partition between Singapore and Amsterdam doesn't bring your whole system down. That's the right call. That's the whole point of distributed systems.

Here's where it gets interesting.

A user in Singapore clicks "Purchase". Node A records the event. A user in Amsterdam completes a form. Node B records that event. So far, so good.

Now Node A and Node B sync up. In a naive system backed by a regular set or a plain key-value store, you'd ask: who wins? If both nodes added different things with the same key, you'd need a conflict resolution strategy. Last-write-wins? Vector clocks? Operational transforms? Each of these is a moving part that can fail, drift, or produce surprising behavior under weird race conditions.

And here's the subtle, nasty part: the problem doesn't show up when you have two nodes. It shows up when you have twenty. It shows up when network partitions happen mid-sync. It shows up when one node is lagging by 40 seconds and another is lagging by 4 minutes. It shows up when you're merging states that went through three intermediate nodes before reaching the final destination.

At small scale, your ad-hoc conflict resolution seems fine. At scale — multiple data centers, hundreds of nodes, millions of events per minute — the edge cases multiply faster than your ability to reason about them. Eventually, you're not building a feature. You're building a distributed consensus protocol from scratch, badly, while also shipping product.

The Fix, in Plain English

What if you just… didn't allow the conflicts to exist in the first place?

That's the core insight behind G-Sets. Instead of building conflict resolution for a general-purpose mutable set, you restrict the operation set to only what can never conflict: addition only.
Think about it like a physical toy box with a one-way slot — like a ballot box. You can drop things in. You cannot take things out. The slot only goes one direction.

Now imagine two kids (two nodes) each have their own toy box. They both drop toys in independently, all day long, without talking to each other. At the end of the day, you want one combined box with everything both kids put in.

The merge is trivial: you just take everything from both boxes. There is no conflict because there was never a competing operation. Kid A adding a rubber duck doesn't conflict with Kid B adding a LEGO set. Addition + Addition = Union. Always. No exceptions.

That's a G-Set.

The formal guarantee is this: at any point in time, on any replica, the set of elements only grows. It never shrinks. When two replicas merge, they compute the union of their states. The union of two sets is commutative (order doesn't matter), associative (grouping doesn't matter), and idempotent (merging the same thing twice gives the same result). Those three properties together mean you can sync replicas in any order, any number of times, across any topology, and you always converge to the same state.

No locks. No coordination. No conflict resolution. No 3 AM pages.

When You'd Actually Use This

G-Sets shine everywhere you naturally think "append-only". Here are five real-world patterns where this isn't just theoretical — it's the right tool:

User event logs in analytics. Every click, login, page view, or purchase is an event that happened. Events don't un-happen. You record them, and they stay forever. Each node adds events to its local G-Set replica. When replicas sync, the union is the full history. No event is lost, no event is double-counted from merging.
Training data pipelines in ML. Different ingestion nodes might be collecting labeled training samples from different regions. Once a sample is added to the corpus, it stays. The "global training set" is just the union of everything every node has seen. Merging two replicas is just "take all samples from both" — which is exactly what you want.
Config version history. A service that tracks every version of a configuration key — so you can replay history, debug what changed, or roll back — needs to keep all version metadata permanently. You only ever insert version entries; you never delete them. A G-Set gives you that audit trail with automatic merge semantics across replicas.
Social graph: follow edges. In a "follow" relationship model (think Twitter/X, GitHub stars), you add edges to a graph: "user A followed user B". If your system doesn't support unfollow (or handles unfollow via a different mechanism), the follow-graph itself is a G-Set. New follows can be added from any replica without coordination, and they all merge cleanly.
Permission grant audit trail. Every time a user is granted a permission — "can read resource X", "can deploy to environment Y" — that grant is recorded permanently. You don't delete grants; you expire them or supersede them via a newer grant. The history of all grants ever issued is a G-Set across your auth service replicas.

Notice the pattern: G-Sets are perfect wherever the business semantics naturally say "this thing happened and we never want to forget it."

The Algorithm

The G-Set is deliberately minimal. Three operations, all of them simple:

State: A set S of elements.

Add(element): Insert the element into the local set. This is always safe. It never conflicts with anything. Complexity: O(1) average.

Lookup(element): Check if an element is in the local set. Returns true if present. Complexity: O(1) average.

Merge(remote_state): Given another replica's set, compute the union of both sets and store it locally. Formally: S_local = S_local ∪ S_remote. Complexity: O(|S_remote|).

The convergence proof is embarrassingly simple:

Any two replicas that have seen the same set of Add operations will have the same state.
Merge is monotonically non-decreasing — the set can only grow, never shrink.
Because merge is a union operation, it's commutative (A ∪ B = B ∪ A), associative ((A ∪ B) ∪ C = A ∪ (B ∪ C)), and idempotent (A ∪ A = A).
Therefore, regardless of the order in which replicas sync, they will all eventually converge to the union of all elements ever added.

This property — where any number of merges in any order reaches the same final state — is called Strong Eventual Consistency (SEC). It's a stricter and more useful guarantee than plain eventual consistency, because it means you don't just eventually agree; you agree immediately once you've seen the same information.

One important constraint to keep in mind: G-Sets have no Remove operation. That's not a bug — it's the feature. The moment you add removal, you open the door to conflicts (what if one replica removes an element that another replica just added?). If you need removal semantics, you'd graduate to a 2P-Set (Two-Phase Set) or an OR-Set (Observed-Remove Set), which are more complex CRDTs. But if your use case is naturally append-only, don't reach for the complex tool.

Implementation in Go

package crdt

import "sync"

// GSet is a Grow-only Set — elements can be added but never removed.
// Safe for concurrent use via a read-write mutex.
type GSet[T comparable] struct {
    mu   sync.RWMutex
    data map[T]struct{}
}

func NewGSet[T comparable]() *GSet[T] {
    return &GSet[T]{
        data: make(map[T]struct{}),
    }
}

// Add inserts an element into the set.
// Always safe: no conflict is possible with a pure add operation.
func (s *GSet[T]) Add(item T) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.data[item] = struct{}{}
}

// Contains returns true if the element is present in the set.
func (s *GSet[T]) Contains(item T) bool {
    s.mu.RLock()
    defer s.mu.RUnlock()
    _, ok := s.data[item]
    return ok
}

// Items returns a snapshot of all elements currently in the set.
func (s *GSet[T]) Items() []T {
    s.mu.RLock()
    defer s.mu.RUnlock()
    items := make([]T, 0, len(s.data))
    for item := range s.data {
        items = append(items, item)
    }
    return items
}

// Merge computes the union of this set and another replica's set.
// Commutative, associative, idempotent — safe to call in any order, any number of times.
func (s *GSet[T]) Merge(other *GSet[T]) {
    other.mu.RLock()
    snapshot := make([]T, 0, len(other.data))
    for item := range other.data {
        snapshot = append(snapshot, item)
    }
    other.mu.RUnlock()

    s.mu.Lock()
    defer s.mu.Unlock()
    for _, item := range snapshot {
        s.data[item] = struct{}{}
    }
}

// Example: two analytics nodes recording events independently, then syncing.
func ExampleTwoReplicas() {
    // Node A: Singapore data center
    nodeA := NewGSet[string]()
    nodeA.Add("event:user_123:purchase")
    nodeA.Add("event:user_456:login")

    // Node B: Amsterdam data center
    nodeB := NewGSet[string]()
    nodeB.Add("event:user_789:pageview")
    nodeB.Add("event:user_456:login") // same event recorded on both nodes

    // Sync: merge B into A
    nodeA.Merge(nodeB)

    // nodeA now contains all three unique events.
    // The duplicate "event:user_456:login" is handled automatically — idempotency in action.
    // Result: {"event:user_123:purchase", "event:user_456:login", "event:user_789:pageview"}
    fmt.Println(nodeA.Items())
}

A few things worth pointing out in this implementation:

The mutex strategy matters here. We take a snapshot of other.data under other's read lock first, then release it before acquiring s's write lock. This avoids a classic deadlock scenario where two goroutines simultaneously try to merge into each other, each waiting for the other's lock. Snapshot-then-write is the safe pattern.

The generic constraint [T comparable] is load-bearing — Go's map keys must be comparable, so string, int, UUID types all work. Custom structs work too as long as all fields are comparable.

Implementation in Java

import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;

/**
 * GSet — a Grow-only Set CRDT.
 *
 * Elements can only be added, never removed.
 * Merge is a union operation: commutative, associative, and idempotent.
 * Thread-safe via a ReentrantReadWriteLock.
 *
 * @param <T> the type of elements; must implement equals() and hashCode() correctly.
 */
public class GSet<T> {

    private final Set<T> data = new HashSet<>();
    private final ReadWriteLock lock = new ReentrantReadWriteLock();

    /**
     * Add an element to the set.
     * Always safe — a pure add can never conflict with any other operation.
     */
    public void add(T item) {
        lock.writeLock().lock();
        try {
            data.add(item);
        } finally {
            lock.writeLock().unlock();
        }
    }

    /**
     * Check if an element is present in the set.
     */
    public boolean contains(T item) {
        lock.readLock().lock();
        try {
            return data.contains(item);
        } finally {
            lock.readLock().unlock();
        }
    }

    /**
     * Returns an immutable snapshot of all elements currently in the set.
     */
    public Set<T> items() {
        lock.readLock().lock();
        try {
            return Collections.unmodifiableSet(new HashSet<>(data));
        } finally {
            lock.readLock().unlock();
        }
    }

    /**
     * Merge another replica's state into this set (union operation).
     *
     * Safe to call in any order, any number of times.
     * A snapshot of the other set is taken under the other set's read lock
     * before acquiring this set's write lock — prevents deadlock on mutual merge.
     */
    public void merge(GSet<T> other) {
        // Snapshot first, under other's read lock
        Set<T> snapshot = other.items();

        lock.writeLock().lock();
        try {
            data.addAll(snapshot);
        } finally {
            lock.writeLock().unlock();
        }
    }
}

// Example: simulating two service nodes syncing their event logs
public class GSetsExample {
    public static void main(String[] args) {
        // Node A: US-East
        GSet<String> nodeA = new GSet<>();
        nodeA.add("grant:user_101:read:resource_A");
        nodeA.add("grant:user_202:write:resource_B");

        // Node B: EU-West
        GSet<String> nodeB = new GSet<>();
        nodeB.add("grant:user_303:read:resource_C");
        nodeB.add("grant:user_202:write:resource_B"); // same grant, recorded independently

        // Sync: merge B into A
        nodeA.merge(nodeB);

        // nodeA now contains all three unique grants.
        // Duplicate handled automatically via HashSet semantics.
        nodeA.items().forEach(System.out::println);
        // Output (order may vary):
        // grant:user_101:read:resource_A
        // grant:user_202:write:resource_B
        // grant:user_303:read:resource_C
    }
}

The Java version's items() returning an unmodifiable snapshot does double duty: it protects the internal state from external mutation and serves as the snapshot mechanism that merge() relies on for the deadlock-safe two-phase lock acquisition.

One Java-specific note: make sure your element type T has a correct equals() and hashCode() implementation. HashSet.add() uses these to determine uniqueness. If you're storing custom objects (like a UserEvent record), and you forget to override these, you'll end up with duplicate "equal" events in your set — which defeats the idempotency guarantee. Java records (record UserEvent(String userId, String type)) handle this correctly by default.

The Same Data Structure, Everywhere You Look

Once you internalize G-Sets, you start recognizing the pattern in places you never expected. The question to ask is simple: "Does this data only ever grow? Is removing something not a real operation, just a display concern?" If yes, you have a G-Set waiting to happen.

IoT sensor readings. A fleet of temperature sensors across a warehouse each record readings independently. Readings never get un-taken. The "global history of all readings" is the union of every sensor's local set. G-Set handles the merge across all nodes with zero coordination.

Distributed tag systems. Content tagging in a CMS or knowledge base: articles accumulate tags across different editor nodes. Tags are added freely; "removing" a tag is typically a business-level soft delete handled separately. The growth of tags across replicas is a pure G-Set problem.

Service discovery registries. When microservices register themselves with a discovery service, each new registration is an append. The set of "services that have ever been registered" is a G-Set. Deregistration is a separate concern (and a different CRDT problem).

Feature flag rollout tracking. Which user IDs have had a feature enabled for them — for audit or rollback purposes — is naturally append-only. You enable features for more users over time, and you want every region's view of "who saw this feature" to converge to the same answer.

Distributed deduplication registries. Idempotency keys for payment systems or message queues. Once a key is "seen", it's in the set forever. Multiple nodes can independently record the same key without conflict — the G-Set's idempotency handles duplicates automatically.

Blockchain-style ledgers. Transaction IDs in a distributed ledger. New transactions are appended; nothing is ever removed. Each participating node maintains a local G-Set of confirmed transaction IDs, and merging across nodes gives you the complete global ledger with zero conflict resolution needed.

The unifying mental model: any time your data represents facts that happened, rather than state that changes, you're looking at a G-Set. Events happened. Grants were issued. Sensors fired. Services registered. None of those facts un-happen. Stop treating them like mutable state — and stop writing conflict resolution code for them.

What G-Sets Can’t Do (and That’s Fine)

It would be dishonest to end without stating the obvious limitation: you cannot remove elements. A G-Set grows forever.

This means:

If you need “soft deletes” or element removal, G-Set is the wrong tool. Look at 2P-Sets or OR-Sets.
If your set grows unboundedly and you care about memory, you’ll need a tombstone strategy or a compaction layer on top.
If the “has this element ever been added” semantic doesn’t match your domain (i.e., you need “is this element currently active”), you’re modeling the wrong thing as a G-Set.

But for the patterns where it does fit — event logs, audit trails, append-only graphs, training data — the G-Set gives you something rare in distributed systems: a correctness guarantee that doesn’t require you to think harder. It just works.

And in systems at scale, “it just works” is worth more than almost anything else.

What’s Next

The next piece steps back from the counters entirely.

What if instead of tracking numbers, you need to track membership — items in a set, users in a group, tags on a document? The same concurrent update problems apply. And a surprisingly simple grow-only constraint solves the basic case cleanly.

Episode 5: 2P — 2-Phase sets, where we add the ability to remove elements — and discover why even that simple extension requires careful design.

The full series:

Episode 1: G-Counter — why YouTube’s view count is approximate
Episode 2: PN-Counter — Reddit karma and the two-notebook trick
Episode 3: Bounded Counter — inventory floors and distributed permission
Episode 4: G-Set — grow-only sets (you’re here)
Episode 5: 2P-2 Phase sets
Episode 6: OR-Set — the fix for “deleted but still there”
Episode 7: The full picture — when to use CRDTs and when to walk away

Subscribe to the newsletter to get each episode as it drops.

If you’re building distributed systems and want to think through invariant enforcement, CRDT design, or distributed architecture with someone who has been deep in this space, I do 1:1 sessions on Topmate.

Your Inventory Counter Just Went Negative. Here’s Why — and How to Fix It.

Archit Agarwal — Mon, 18 May 2026 06:14:44 +0000

Black Friday. 9:02am. Your e-commerce platform has been live for two minutes.

A limited-edition sneaker drops - 500 pairs in stock. Within seconds, the inventory counter is spinning. Add to cart. Checkout. Payment confirmed. Your servers in the US, EU, and Asia are all taking orders simultaneously to keep latency low for every customer on the planet.

By 9:04am, you've sold 620 pairs.

You have 500.

Your ops channel lights up. Support tickets flood in. You're now emailing 120 paying customers to tell them their confirmed order doesn't exist. The refunds go out. The trust doesn't come back.

What went wrong wasn't a bug in your payment code. It wasn't a race condition in the traditional sense. It was something more fundamental: your distributed counter had no floor. And when multiple servers decremented it concurrently, each one saw a local view that looked safe - and the global invariant silently broke.

This is the problem Bounded Counters were designed to solve. And the solution is one of the most elegant ideas in the CRDT literature.

What an Inventory Counter Actually Is

Before the distributed systems, let's ground this in something simple.

An inventory counter is just a number. It represents how many units of something you have available. When a customer buys one, the number goes down. When a shipment arrives, the number goes up. The rule is straightforward:

The counter must never go below zero.

In SQL, this is trivial:

CREATE TABLE inventory (
    product_id  BIGINT PRIMARY KEY,
    stock       INT DEFAULT 0 CHECK (stock >= 0)
);

The CHECK constraint enforces the floor. If a transaction tries to bring stock below zero, the database rejects it. One server. One source of truth. The constraint holds perfectly.

The same logic applies to a wallet: balance must never go below zero. To an ad budget: spend must never exceed the limit. To a rate limiter: requests must never exceed the ceiling.

These are bounded counters - counters with a numeric invariant that must never be violated.

Simple when you have one server. Deeply interesting when you have many.

The Moment Simplicity Breaks

Your platform grows. One server can't handle Black Friday traffic. You distribute.

You deploy database replicas in the US, EU, and Asia. Each region accepts writes locally to keep latency low. The system is eventually consistent - replicas sync in the background, but any given server might be working from a slightly stale view of the world.

Here's what happens to your inventory counter.

500 pairs in stock. Three servers, each seeing the same initial state: stock = 500. In the first two minutes, customers are hammering all three regions simultaneously.

The US server processes 200 orders. Locally: 500 - 200 = 300. Looks fine.

The EU server, working from the same initial state it loaded before the US orders propagated, processes 180 orders. Locally: 500 - 180 = 320. Also looks fine.

The Asia server processes 170 orders. Locally: 500 - 170 = 330. Still fine locally.

When the three servers finally sync: 500 - 200 - 180 - 170 = -50.
You've oversold by 50 pairs. Every check passed locally. The global invariant shattered silently.

Why the Solutions You're Thinking Of Don't Work Here

At this point, two approaches come to mind. Both fail at scale in ways worth understanding.

Distributed locks: Before decrementing, acquire a global lock on the inventory key across all replicas. Only one server writes at a time.

This works - and kills your latency. A lock that spans data centres separated by hundreds of milliseconds of network round-trip adds that latency to every single purchase. On Black Friday, when the whole point of distributing was to make checkout fast, a global lock turns your purchase flow into a queue. Users abandon cart. Revenue drops.

Strong consistency everywhere: Route all inventory writes through a single primary node or quorum. Every decrement gets serialised.

Also works. Also slow. Same problem as distributed locks, different mechanism. The CAP theorem is not negotiable: if you want strong consistency across geographically distributed nodes, you pay in latency and availability.

In Episode 2, PN-Counter gave us increment and decrement without coordination. But I ended that piece with a warning: PN-Counter has no floor. Nothing in its design prevents the score from going to -50,000. For Reddit karma, nobody cares. For inventory, that's a P0 incident.

We need something new.

The Insight: Stop Distributing the Counter. Distribute the Permission.

Here's the shift in thinking that makes Bounded Counters work.

With PN-Counter, we stopped trying to coordinate on every write. We accepted that replicas might temporarily disagree on the exact value. That was fine because eventual consistency was acceptable.

Bounded Counters need something stronger: a hard floor that can never be violated, even under concurrent writes, even during network partitions, even when replicas have stale views.

The key insight: you don't need every replica to know the exact current value to enforce the floor. You just need every replica to know it has permission to decrement.

Think of it like tickets.

You have 100 items in stock. The floor is 0. That means you have 100 units of "decrement permission" - 100 tickets. You distribute those tickets across your three servers before the sale starts:

US server: 50 tickets
EU server: 30 tickets
Asia server: 20 tickets

Now the rule is simple: a server can only decrement if it has at least one ticket to spend. When it decrements by N, it spends N tickets.

The US server takes 50 orders: spends 50 tickets, now has 0. Stops accepting orders locally. The EU server takes 30 orders: spends 30 tickets, now has 0. Stops accepting orders locally. The Asia server takes 20 orders: spends 20 tickets, now has 0. Stops.

Total orders: 100. Exactly the stock. Global invariant: never violated.

Nobody coordinated on a single order. Every check was purely local. The constraint held globally because the math is airtight: if total tickets = total available decrements, and each replica can only spend what it owns, the sum of decrements can never exceed the total tickets.

This is the escrow idea, applied to distributed systems. You're not distributing the counter. You're distributing the permission to decrement.

The Wallet Walk-Through

Let's trace this concretely with a wallet that must never go below zero.

Initial state: Wallet value = 100. Floor = 0. Available decrement rights = 100 − 0 = 100.

Rights distributed:

DC_A: 50 rights
DC_B: 30 rights
DC_C: 20 rights

Payment at DC_A: User pays 15 units.

DC_A checks local rights: 50 ≥ 15 ✓. Executes locally. No coordination needed.

DC_A: 35 rights remaining
DC_B: 30 rights remaining
DC_C: 20 rights remaining
Global value: 85

Concurrent payment at DC_B: User pays 25 units.

DC_B checks local rights: 30 ≥ 25 ✓. Executes locally. DC_A and DC_B never talked during either payment.

DC_A: 35 rights remaining
DC_B: 5 rights remaining
DC_C: 20 rights remaining
Global value: 60

Both payments went through. Both were validated purely locally. The global floor - wallet ≥ 0 - was never at risk, because the sum of all rights (35 + 5 + 20 = 60) exactly equals the remaining balance.

Failed payment at DC_C: User tries to pay 25 units.

DC_C checks local rights: 20 < 25 ✗. Cannot guarantee the floor locally.

Two options: fail immediately, or coordinate to borrow rights from another replica.

Rights transfer: DC_C asks DC_A for 10 rights. DC_A has 35 and can spare them.

DC_A: 25 rights  (gave 10 to DC_C)
DC_B: 5 rights
DC_C: 30 rights  (received 10 from DC_A)

DC_C re-attempts: 30 ≥ 25 ✓. Payment goes through.

DC_A: 25 rights
DC_B: 5 rights
DC_C: 5 rights
Global value: 35

At no point did the wallet go below zero. At no point did all three servers need to agree on the exact current value before processing a payment. The invariant held through local checks alone - with one brief coordination event only when a replica ran out of its allocated rights.

The Data Structure

The Bounded Counter state at each replica tracks three things:

type BoundedCounter struct {
    Min int64     // lower bound (e.g. 0)
    N   int       // number of replicas
    ID  int       // this replica's index

    R [][]int64   // R[i][j]: rights replica i transferred to j
                  // R[i][i]: rights replica i created for itself (via increments)
    U []int64     // U[i]: decrements executed at replica i
}

The rights matrix R is the core. Each replica owns the diagonal - R[i][i] is the rights it created. Off-diagonal entries R[i][j] track rights transferred from replica i to replica j. Every entry only ever increases. This is the grow-only constraint, applied to a matrix instead of a single value.

Reading the current value:

func (c *BoundedCounter) Value() int64 {
    var inc, dec int64
    for i := 0; i < c.N; i++ {
        inc += c.R[i][i]
        dec += c.U[i]
    }
    return c.Min + inc - dec
}

Sum the diagonal (total increments), subtract total decrements, add the floor.

Checking local rights:

func (c *BoundedCounter) LocalRights() int64 {
    id := c.ID
    var created, received, sent, spent int64
    created = c.R[id][id]
    for i := 0; i < c.N; i++ {
        if i == id { continue }
        received += c.R[i][id]
        sent += c.R[id][i]
    }
    spent = c.U[id]
    return created + received - sent - spent
}

Rights I created, plus rights I received, minus rights I gave away, minus rights I already spent.

Decrementing:

func (c *BoundedCounter) Dec(n int64) error {
    if c.LocalRights() < n {
        return fmt.Errorf("not enough rights")
    }
    c.U[c.ID] += n
    return nil
}

Only decrement if you have the tickets. Otherwise, reject or trigger a rights transfer.

Transferring rights:

func (c *BoundedCounter) Transfer(to int, n int64) error {
    if c.LocalRights() < n {
        return fmt.Errorf("not enough rights to transfer")
    }
    c.R[c.ID][to] += n
    return nil
}

The merge - identical pattern to every CRDT we've built:

func (c *BoundedCounter) Merge(other *BoundedCounter) error {
    for i := 0; i < c.N; i++ {
        for j := 0; j < c.N; j++ {
            if other.R[i][j] > c.R[i][j] {
                c.R[i][j] = other.R[i][j]
            }
        }
        if other.U[i] > c.U[i] {
            c.U[i] = other.U[i]
        }
    }
    return nil
}

Element-wise maximum. Every entry in R and U only ever grows. Merge is commutative, associative, idempotent. Replicas always converge.

Why the Invariant Can Never Break

Here's the mathematical guarantee, stated plainly.

Total decrement rights across all replicas = current value − floor.

Each replica can only decrement by spending rights it owns. A replica cannot own negative rights. Therefore, the total decrements across all replicas can never exceed the total rights in the system. Therefore, the global value can never drop below the floor.

This holds even when replicas have stale views of each other's state. Each replica's local check - "do I have enough rights?" - is conservative by design. If you have 20 rights and the operation needs 25, you say no. You might technically have been able to borrow rights from another replica - but you don't know that yet, and the safe answer is always no.

Coordination is infrequent and amortised. For operations that fit within local rights, zero coordination. For operations that exceed local rights, one round-trip to request a transfer. Compared to coordinating on every write, this is a dramatic improvement in practice.

The Trade-Off You Should Know

Bounded Counter is the most complex CRDT in this series so far - and it comes with trade-offs worth naming.

Rights starvation: if a replica runs out of rights and the replica it needs to borrow from is unreachable (network partition, server failure), the operation fails. The invariant is safe - you'd rather fail an operation than violate the floor - but the user experience is a rejection, not a delay. For inventory, that means "out of stock" even when stock technically exists elsewhere. Acceptable for correctness. Worth designing around.

State size scales with replica count: the R matrix is N×N. With 3 replicas, 9 cells. With 100 replicas, 10,000. For most geo-distributed systems (2-7 data centres), this is negligible. For edge computing with thousands of nodes, you'd want a different approach.

Rights rebalancing is a background concern: a replica that consistently gets more traffic than its rights allocation will hit starvation more often. Operational tooling to monitor and rebalance rights distribution is not optional - it's part of running this in production.

The Bigger Pattern

Three episodes in, the shape of CRDT design is clear.

Every CRDT we've built solves a hard distributed problem by identifying a constraint that eliminates ambiguity.

G-Counter said: make it grow-only. The maximum is always correct.

PN-Counter said: never decrement - use two grow-only counters and subtract at read time.

Bounded Counter says: never coordinate on every operation - distribute the permission to operate instead.

The constraint is always the solution. The art is finding the right one.

What's Next

The next piece steps back from counters entirely.

What if instead of tracking numbers, you need to track membership - items in a set, users in a group, tags on a document? The same concurrent update problems apply. And a surprisingly simple grow-only constraint solves the basic case cleanly.

Episode 4: G-Set - grow-only sets, where the items go in but never come out, and why that constraint enables a clean distributed implementation.

The full series:

Episode 1: G-Counter - why YouTube's view count is approximate
Episode 2: PN-Counter - Reddit karma and the two-notebook trick
Episode 3: Bounded Counter - inventory floors and distributed permission (you're here)
Episode 4: G-Set - grow-only sets
Episode 5: LWW Registry - last-write-wins
Episode 6: OR-Set - the fix for "deleted but still there"
Episode 7: The full picture - when to use CRDTs and when to walk away

Reddit's Karma Score Is a Lie. Here's the Math Behind It.

Archit Agarwal — Mon, 11 May 2026 03:00:00 +0000

Two comments. Same subreddit. Same topic. Posted minutes apart.

One has +847 karma. It sits at the top, gets gilded, spawns a thread of 200 replies. The person who wrote it gets DMs. Their next post gets upvoted reflexively because people remember the name.

The other has +1. It exists somewhere below the fold, unseen by 99% of readers. Same words, different fate.

That number — Reddit karma — is one of the most consequential invisible forces on the internet. It shapes what millions of people read, what opinions gain traction, whose voices carry weight in a community. And most Reddit users have a vague sense of how it works: upvotes good, downvotes bad, net score determines visibility.

What almost nobody knows is what's actually happening inside Reddit's infrastructure every time someone clicks that arrow.

The score you see? It's a lie. Not a malicious one. A mathematically necessary, deliberately constructed approximation — and the algorithm behind it is one of the most elegant ideas in distributed systems.

This is that story.

What Reddit Karma Actually Is

Let's start simple, because the simple version matters.

Every post and comment on Reddit has a score. That score is calculated as:

Score = Upvotes − Downvotes

A post with 1,200 upvotes and 200 downvotes shows a score of +1,000. A comment with 50 upvotes and 80 downvotes shows -30. Reddit uses this score to rank content — higher scores float up, lower scores sink. In highly active subreddits, the first few minutes of score accumulation can determine whether a post reaches the front page or disappears forever.

The score also feeds into account-level karma, which affects posting privileges, community credibility, and how other users perceive you before they've read a single word you've written.

Simple math. Massive consequences.

Now let's build the system that computes it.

The Simple Version (That Works Fine Until It Doesn't)

Imagine Reddit in its early days. A few thousand users. One web server. One database.

The schema for tracking votes is straightforward:

CREATE TABLE post_scores (
    post_id   BIGINT PRIMARY KEY,
    score     INT DEFAULT 0
);

When someone upvotes:

UPDATE post_scores SET score = score + 1 WHERE post_id = ?;

When someone downvotes:

UPDATE post_scores SET score = score - 1 WHERE post_id = ?;

User loads the page, you return score. Done.

This works beautifully for a small community. Your database handles concurrent updates with row-level locking. Reads are fast. Writes are fast. The score is always accurate.

And then the site grows.

The Moment Everything Gets Complicated

Reddit today serves over 50 million daily active users. Popular posts on r/worldnews or r/AskReddit can accumulate thousands of votes in minutes. During major news events, that number spikes even higher.

One database server cannot handle that write throughput. So you do what every scaling team eventually does — you distribute.

You spin up multiple database replicas. You deploy servers in the US, Europe, and Asia so votes from each region get handled locally. Latency drops. Throughput goes up. Everything seems fine.

Until two servers try to update the same post's score at the same time.

Here's the problem in its simplest form. The US server reads the score: 142. The EU server reads the score: 142. A user in the US clicks upvote — the US server wants to write 143. Simultaneously, a user in Europe clicks downvote — the EU server wants to write 141.

Both writes land. One of them overwrites the other. A real vote disappears.

At small scale this is a rounding error. At Reddit scale, with thousands of concurrent votes per second on a single viral post, you're losing a meaningful number of real votes from real users. Users who clicked an arrow and expected their opinion to count.

Let's be honest — this is not a hypothetical. This is the exact class of problem that has caused production incidents at companies you use every day.

The Approaches That Don't Work

The instinct is to reach for coordination. Make the servers talk to each other before writing.

Distributed locks: before updating the score, acquire a lock on that post's row across all replicas. Only one server writes at a time. Others wait.

This works. It's also slow. Lock acquisition across data centres separated by hundreds of milliseconds of network latency adds that latency to every single vote. During peak traffic, lock contention turns your vote system into a queue. Users notice.

Two-phase commit (2PC): a coordinator server proposes the write, all participants acknowledge, then the coordinator commits. Guarantees consistency across all nodes.

Also works. Also slow. 2PC requires multiple round trips between servers. It assumes all participants are available — if any node is down or unreachable during the commit phase, the transaction blocks. In a system operating across multiple continents, "all participants available" is never guaranteed.

Both approaches trade availability for consistency. That's a valid trade-off in some systems. For a vote counter on a social platform — where a few hundred milliseconds of delay on every click would be immediately noticeable and wildly unacceptable — it's not.

There had to be a better way.

A Different Question

The distributed systems community eventually asked a different question.

Instead of "how do we make all servers agree before writing?" — what if we asked: "what constraints can we place on the data so that disagreements become impossible?"

This line of thinking led to CRDTs — Conflict-free Replicated Data Types. Data structures specifically designed so that concurrent updates on multiple replicas can always be merged correctly, without coordination, without locks, without consensus rounds.

The key insight: if you design your data structure such that the merge operation is always unambiguous — no matter what order updates arrive, no matter how long replicas were disconnected — you get eventual consistency for free.

I covered the simplest CRDT — the G-Counter — in Episode 1 of this series. The short version: each replica keeps its own local count, only ever increments it, and merge means "take the maximum per replica." YouTube's view counter works this way. There's no conflict possible because a count that can only grow is always unambiguous — the higher number is always more up to date.

But G-Counters have an obvious limitation.

They only go up.

Why Upvotes Alone Aren't Enough

Reddit's score isn't a count of upvotes. It's upvotes minus downvotes. You need both directions.

A G-Counter can track upvotes fine. But downvotes require decrement. And decrement is where things get complicated — because "subtract one" from a shared counter across distributed replicas brings back exactly the ambiguity we were trying to avoid.

Two servers both see the score as 50. Both receive a downvote simultaneously. Both want to write 49. Same lost vote problem as before, just through a different door.

So: we need increment and decrement, without coordination, with correct merging across replicas.

Here's where things get interesting.

The Insight: You Never Actually Decrement

This is the moment the whole thing clicks.

What if you never decremented anything? What if every downvote was secretly an increment — just to a different counter?

Go back to the lemonade stand mental model. You and your friends are tracking the cash register. Money comes in from customers. Money goes out for supplies. You need to track both.

You already know the G-Counter trick: give everyone a notebook where they can only add tally marks. Works for money coming in.

For money going out, the solution is almost embarrassingly simple: give everyone two notebooks.

A green notebook for every dollar that came in
A red notebook for every dollar that went out

Neither notebook ever loses a mark. Both only ever grow. At the end of the day, you sync the green notebooks (take max per person), sync the red notebooks (take max per person), and do one subtraction:

Green total − Red total = what's in the register

That's a PN-Counter. Positive-Negative Counter.

Two G-Counters. One subtraction at read time. No coordination. No locks.

The "decrement" you thought you needed was never a real operation. It was always just an increment to a second counter that you subtract when you read the value. The counter doesn't go down — your accounting does.

The Code

gotype PNCounter struct {
    inc GCounter
    dec GCounter
}

func NewPNCounter() PNCounter {
    return PNCounter{
        inc: Zero(),
        dec: Zero(),
    }
}

Two G-Counters. That's the entire data structure.
Reading the score:

func (p PNCounter) Value() int64 {
    return p.inc.Value() - p.dec.Value()
}

The subtraction only happens here — at read time, when someone loads the page. Never during updates. Never during merges.

Upvoting:

func (p PNCounter) Inc(r ReplicaId) PNCounter {
    return PNCounter{
        inc: p.inc.Inc(r),
        dec: p.dec,
    }
}

Downvoting:

func (p PNCounter) Dec(r ReplicaId) PNCounter {
    return PNCounter{
        inc: p.inc,
        dec: p.dec.Inc(r),
    }
}

Look at Dec. It doesn't subtract anything. It calls .Inc() on the decrement counter.

A downvote is an upvote to the wrong notebook.

If you showed this to someone without context they'd think it was a bug. It's the entire algorithm.

Merging two replicas:

func MergePN(a, b PNCounter) PNCounter {
    return PNCounter{
        inc: Merge(a.inc, b.inc),
        dec: Merge(a.dec, b.dec),
    }
}

Run the G-Counter merge twice — once on the increment counters, once on the decrement counters. Same "take the maximum per replica" logic. Nothing new to prove. Everything PN-Counter needs, it inherits from G-Counter.

What This Looks Like on Reddit's Servers

Let's trace a viral comment across three servers: US, EU, Asia.
T=0: Comment just posted. All counters at zero.

US:   inc={US:0}, dec={US:0}  →  score: 0
EU:   inc={EU:0}, dec={EU:0}  →  score: 0
Asia: inc={As:0}, dec={As:0}  →  score: 0

T=1: Activity floods in. US gets 8 upvotes. EU gets 3 upvotes and 5 downvotes. Asia gets 2 downvotes. All happening simultaneously, all independent.

US:   inc={US:8}, dec={US:0}   →  local score: 8
EU:   inc={EU:3}, dec={EU:5}   →  local score: -2
Asia: inc={As:0}, dec={As:2}   →  local score: -2

T=2: US and EU sync.

inc = Merge({US:8}, {EU:3}) = {US:8, EU:3}
dec = Merge({US:0}, {EU:5}) = {US:0, EU:5}
Score = (8+3) - (0+5) = 6

US and EU now agree: score is +6.

T=3: Asia reconnects after a brief network partition and syncs.

inc = {US:8, EU:3, As:0}  →  total: 11
dec = {US:0, EU:5, As:2}  →  total: 7
Score = 11 - 7 = 4

Final score: +4. Every upvote counted. Every downvote counted. No vote lost during the partition. No coordination during any of the voting itself.

This is the lie in the title: the +4 you see wasn't computed from one authoritative source. It was assembled from fragments of state scattered across three continents and merged after the fact.

It's an approximation. A consistent, mathematically guaranteed approximation — but an approximation nonetheless.

The Trade-Off Worth Naming

PN-Counter inherits G-Counter's main cost: every sync ships full state. Two maps, N entries each, across every gossip round. For tens of replicas this is fine. For thousands of edge nodes, the bandwidth adds up.

The more important trade-off is eventual consistency. When you're actively voting on a hot post, the score you see is the best approximation your nearest server has right now. Somewhere in Europe, a server might briefly disagree. They'll converge — they always do — but there's a window where different users see slightly different scores.

For Reddit, this is entirely acceptable. Nobody's life depends on whether the karma score reads 1,042 or 1,047 for 200 milliseconds. The system is highly available, survives network partitions cleanly, and handles write throughput that would collapse a coordinated system.

The trade-off is: exact consistency in exchange for scale and availability. Reddit made that trade. Most social platforms do.

The Problem PN-Counter Doesn't Solve

Here's where it gets interesting again — and where the next problem is quietly waiting.

PN-Counter has no floor. No ceiling. Nothing prevents the score from going to -50,000 if enough downvotes arrive. And nothing stops it from going to +10,000,000 either.

For Reddit karma, that's fine. But consider:

A distributed rate limiter that needs to cap requests at 1,000 per minute. The counter must not go above 1,000, enforced across all replicas.
An inventory system where stock can't go below zero. You can't sell items you don't have.
A per-user karma floor — some subreddits prevent karma from dropping below a certain value to protect new users from pile-ons.

These are bounded operations. And PN-Counter, with its two unbounded G-Counters, cannot enforce bounds without coordination.

You'd think: just check the current value before decrementing, and reject the operation if you'd go below the floor.

Try that across distributed replicas and you immediately reintroduce the problem we just spent this entire article solving. Two replicas both see the counter at 1 (the floor). Both receive a decrement. Both check locally — "1 is above 0, I can proceed." Both decrement. Counter is now -1 across the board. The floor was violated.

This is the problem the Bounded Counter CRDT was designed for. It's the next piece in this series — and it's the most complex CRDT we'll cover, because enforcing limits without coordination requires a genuinely different approach.

The short version: you can't do it with just max and subtraction. You need to distribute the permission to decrement itself.

But that's next time.

The Thread Running Through All of This

Two episodes in, the pattern is clear.

Every CRDT solves a hard distributed problem by finding a constraint that eliminates ambiguity. G-Counter said: make it grow-only, and max is always correct. PN-Counter said: never decrement — maintain two grow-only counters and subtract at read time.

The constraint is never a workaround. It's the design.

When you next open Reddit and see a karma score, you're looking at the output of two grow-only counters being summed and subtracted on a server near you — a server that may not have talked to its peers in the last few hundred milliseconds and doesn't need to.

The number is an approximation. But it's a principled one, built on math that guarantees eventual correctness without sacrificing the availability and performance that make Reddit usable at scale.

That's not a lie. That's engineering.

What's Next

This is Episode 2 of a series on CRDTs — the data structures powering Google Docs, Redis, Riak, and distributed infrastructure at scale.

Episode 1: G-Counter — why YouTube's view count is always approximate
Episode 2: PN-Counter — Reddit karma and the two-notebook trick (you're here)
Episode 3: G-Set — grow-only sets, and where they appear in the wild
Episode 4: LWW Registry — when "latest timestamp wins" is good enough
Episode 5: OR-Set — the elegant fix for the "deleted but still there" problem
Episode 6: Map CRDT — composing CRDTs into richer structures
Episode 7: The full picture — when to reach for CRDTs and when to walk away

Subscribe to the newsletter to get each episode as it drops.

If you're building distributed systems and want to think through architecture decisions with someone who has spent years deep in Go and distributed infrastructure — I do 1:1 sessions on Topmate.

YouTube Doesn't Actually Know How Many Views Your Video Has

Archit Agarwal — Tue, 05 May 2026 11:26:39 +0000

And that's not a bug. It's one of the most elegant algorithms in distributed systems.

Every time you refresh a viral YouTube video and watch the counter tick up, you're looking at a lie.

Not a malicious one. A mathematically precise, deliberately constructed one — built by some of the smartest distributed systems engineers on the planet, for a very good reason.

The actual view count? Nobody knows it exactly. Not YouTube's servers. Not their databases. Not the team that built the counter.
Here's what's actually going on, and why the algorithm behind it — called a G-Counter — is one of the most satisfying ideas in computer science.

The Problem With Counting at Scale

Let's be honest about what YouTube is dealing with.

A viral video gets 50 million plays in its first 24 hours. That's roughly 578 plays every second. These plays are happening from Los Angeles, London, Mumbai, São Paulo — simultaneously, all the time.
YouTube doesn't run one server. It runs thousands, distributed across data centres on multiple continents. Every server is independently accepting "play" events.

The naive solution: have one authoritative counter. Every time someone hits play, that event travels to the central counter, which increments and returns the new value.

The problem: that central counter immediately becomes the bottleneck. At 578 writes per second, you'd need a heavily optimised database just for the counter. And when it goes down — because everything goes down eventually — your entire view counting system goes with it.

The slightly less naive solution: use a distributed lock. Servers coordinate before writing. They take turns.

The problem with locks at scale: latency multiplies, throughput collapses, and network partitions turn the whole thing into a mess. In distributed systems, the CAP theorem tells you this is a fundamental tradeoff — you can't have consistency, availability, and partition tolerance all at once. When you choose strong consistency (everyone agrees on the exact count), you sacrifice availability and performance.

So YouTube made a different choice: stop trying to coordinate at all.

The Insight That Changes Everything

Here's the mental shift that makes G-Counters work.

What does it mean for a view count to be "correct"?

If the US server has recorded 1.2 million views, and the EU server has recorded 800 thousand views, and the Asia server has recorded 600 thousand views — and none of these servers have talked to each other recently — the total isn't ambiguous. It's 2.6 million.

You just add them up.

The reason this is safe is the key constraint: view counts can only go up. Nobody un-watches a video. Nobody subtracts views. The counter is monotonically increasing.

This constraint completely eliminates the problem of conflicting state. If the US server thinks it has 1.2M views and later discovers the EU server also thought the US had 1.1M views (because the EU server's information was slightly stale), there's no conflict to resolve — 1.2M is clearly more up-to-date, because the counter can only increase. You just take the higher number.

That insight — that the maximum value is always the correct value for a grow-only counter — is the foundation of the G-Counter algorithm.

A Simpler Way to See It

Before we look at the code, here's the mental model I find most useful.

Imagine three friends — call them US, EU, and Asia — who are each counting birds in different sections of a massive park. The park is too big to shout across, so they can't coordinate in real time.

Each person has their own notebook. The rule is simple: you can add tally marks, but you can never erase them (because you can't un-see a bird).

At the end of the day, they meet up. US has 3 marks. EU has 5 marks. Asia has 2 marks.

Total birds seen: 10.

Here's what makes this work without any drama: even if they ran into each other in the middle of the day and shared their notebooks, there's no conflict possible. If EU had shared their notebook earlier and it showed 4 marks at the time, and now it shows 5 — 5 is just the more current number. Take the maximum. Move on.

When you replace "friends" with "servers" and "notebook" with "replica state", you have a G-Counter.

The Data Structure

A G-Counter is, at its core, just a map:

type ReplicaId string
type GCounter map[ReplicaId]int64

func Zero() GCounter {
    return make(GCounter)
}

Each replica (each server) owns exactly one key in the map. A replica only ever writes to its own key. This invariant is everything — it's what makes the merge operation safe.
Reading the total is simple addition:

func (c GCounter) Value() int64 {
    var total int64
    for _, v := range c {
        total += v
    }
    return total
}

Incrementing means bumping your own key by 1:

func (c GCounter) Inc(r ReplicaId) GCounter {
    newC := make(GCounter, len(c))
    for k, v := range c {
        newC[k] = v
    }
    newC[r] = newC[r] + 1
    return newC
}

We copy the map to keep things immutable — each increment returns a new GCounter rather than mutating in place. This is a conscious design choice that makes reasoning about state considerably easier.

The Merge Function — Where the Magic Lives

This is the core of the whole algorithm:

func Merge(a, b GCounter) GCounter {
    result := make(GCounter)

    for k, v := range b {
        result[k] = v
    }

    for k, v := range a {
        if val, ok := result[k]; ok {
            if v > val {
                result[k] = v
            }
        } else {
            result[k] = v
        }
    }
    return result
}

For each replica ID, we take the maximum value seen across both maps.
That's it. No locks. No consensus rounds. No leader election. 15 lines of Go.

Why "Take the Maximum" Is Mathematically Safe

Here's where things get interesting — because this works for a deeper reason than it might seem.

For a merge function to be safe in a system where replicas can sync asynchronously, in any order, any number of times — it needs to satisfy three mathematical properties:

Commutativity: Merge(A, B) must equal Merge(B, A). It shouldn't matter which server initiates the sync.
Associativity: Merge(Merge(A, B), C) must equal Merge(A, Merge(B, C)). It shouldn't matter how you group the merges.
Idempotency: Merge(A, A) must equal A. If a server accidentally receives the same state twice — because of a network retry, for example — nothing should break.

The maximum function satisfies all three. Maximum(3, 5) equals Maximum(5, 3). Maximum(Maximum(3, 5), 7) equals Maximum(3, Maximum(5, 7)). Maximum(5, 5) equals 5.

A data structure whose merge function has these three properties is called a Convergent Replicated Data Type (CvRDT), or a state-based CRDT. The "convergent" part is the guarantee: no matter what order messages arrive in, no matter how long servers are disconnected, they will always end up agreeing on the same state when they finally sync.

This guarantee — eventual consistency without coordination — is the entire value proposition of CRDTs.

A Concrete Walk-Through

Let's trace through exactly what happens with three servers.
T=0: All servers start at zero.

US:   {US: 0}
EU:   {EU: 0}
Asia: {Asia: 0}

T=1: US receives 3 plays, EU receives 5, Asia receives 2. All independently.

US:   {US: 3}
EU:   {EU: 5}
Asia: {Asia: 2}

T=2: US and EU sync. Merge({US:3}, {EU:5}) → {US:3, EU:5}.

US:   {US: 3, EU: 5}  →  Value() = 8
EU:   {US: 3, EU: 5}  →  Value() = 8
Asia: {Asia: 2}        →  Value() = 2  (still hasn't synced)

T=3: Asia comes back online after a brief network partition. It merges with US.
Merge({US:3, EU:5}, {Asia:2}) → {US:3, EU:5, Asia:2}

All servers: {US:3, EU:5, Asia:2}  →  Value() = 10

Every play counted. No coordination required during the counting itself. The merge handles everything.

Where This Shows Up in the Real World

G-Counters (or something very close to them) are embedded in more infrastructure than most engineers realise:

YouTube and Twitch view counts — the exact use case we opened with. Regional servers count locally, merge periodically. The displayed count is always approximate, always eventually consistent.

Redis in distributed mode — Redis Enterprise uses CRDT-based counters when data is replicated across data centres. The INCR command on a distributed counter works precisely because of this algorithm.

Analytics pipelines — tools like Segment and Amplitude track event counts at edge nodes before rolling up. The edge nodes are essentially running G-Counters before flushing to a central store.

Riak — the distributed database that helped popularise CRDTs in production, built native G-Counter support into its data model specifically for use cases like these.

Gaming leaderboards — any system tracking "total kills" or "total distance" across game sessions that can happen offline is solving the same problem.

The Trade-Off You Should Know

G-Counters are simple and powerful, but they come with a cost worth understanding.

Because this is a state-based CRDT (also called CvRDT), every time two replicas sync, they exchange their entire state. With 3 replicas, that's a tiny map — 3 key-value pairs. With 1,000 replicas, every gossip round ships 1,000 entries over the wire, even if only one entry changed.

This is the fundamental tension in state-based CRDTs: simplicity of reasoning versus efficiency of replication.

The solution to this is Delta-state CRDTs, where replicas only send the delta — what changed since the last sync — rather than the whole state. Same mathematical guarantees, much lower bandwidth. That's a topic for a future article.

The Constraint Is the Solution

Here's the thing I find most elegant about G-Counters.

The constraint that initially seems limiting — you can only ever increment, never decrement — is exactly what makes the algorithm so clean. It eliminates an entire class of conflict. It makes the merge function trivially provable. It turns a hard distributed systems problem into a single line: take the max.

This pattern shows up everywhere in good system design. When you can't have everything, the question becomes: what do I give up, and what do I get in return? YouTube gave up exact real-time accuracy on view counts. In return, they got a counter that scales to any number of servers, works through network partitions, needs no coordination, and never loses a view.

That's a trade-off most engineering teams would take without hesitation.

What's Next

This is Episode 1 of a series I'm writing on CRDTs — the data structures that power Google Docs, distributed databases, offline-first applications, and collaborative tools.

Episode 2 covers PN-Counters — what happens when you need to both increment and decrement, without conflicts. Spoiler: you just use two G-Counters and subtract. The simplicity is almost frustrating.
The full series:

Episode 1: G-Counter (you're here)
Episode 2: PN-Counter — increment and decrement
Episode 3: G-Set — grow-only sets
Episode 4: LWW Registry — last-write-wins
Episode 5: OR-Set — observed-remove sets
Episode 6: Map CRDT — composing CRDTs into richer structures
Episode 7: The full picture — when to use CRDTs and when not to

Follow along on [your newsletter/Substack URL] to get each episode as it drops.

If you're building distributed systems and want to think through architecture decisions with someone who has been in the weeds on this — I do 1:1 sessions on Topmate.

How to Build Heartbeats in Go: Let Your Goroutines Say 'Still Breathing!'

Archit Agarwal — Wed, 21 May 2025 10:45:00 +0000

Ever wonder how to make your Go services show signs of life… even when they’re bored out of their mind?

You’re not alone.

Imagine this: You’ve got a bunch of goroutines quietly waiting for work to show up. All seems peaceful. Until… boom — you discover one of them died hours ago, and no one told you. No logs, no panics, no error traces — just pure ghosting.

Wanna avoid that silent death? Let’s teach our goroutines to breathe — or more specifically, send heartbeats.

🫀 Why Bother with Heartbeats?

Ever had a background task silently die while your main process happily spins along? Or had to debug why a job queue worker went unresponsive in the middle of the night?

Heartbeats help answer:

“Hey, is that goroutine still alive… or did it take an early retirement?”

If you’re building anything slightly concurrent, heartbeat signals become your tiny, periodic signs of life from those goroutines. You can monitor them, restart them, or just breathe a little easier knowing things are ticking.

🛠️ Let’s Build It!

Let’s say you’ve got a worker goroutine that waits for signals to do some work. While it’s waiting, you also want it to occasionally let the outside world know:

“Still alive, boss. Just waiting for the next gig.”

Here’s how we make that happen:

func dowork(done <-chan interface{}, pulseInterval time.Duration) (<-chan interface{}, <-chan struct{}) {
 heartbeater := make(chan interface{})
 result := make(chan struct{})
 go func() {
  defer close(result)
  defer close(heartbeater)

  pulse := time.NewTicker(pulseInterval)
  workGen := time.NewTicker(3 * pulseInterval)

  defer pulse.Stop()
  defer workGen.Stop()

  sendPulse := func() {
   select {
   case heartbeater <- struct{}{}:
   default:
    // drop if nobody's listening
   }
  }

  sendResult := func(res struct{}) {
   for {
    select {
    case <-done:
     return
    case <-pulse.C:
     sendPulse()
    case result <- res:
     return
    }
   }
  }

  for {
   select {
   case <-done:
    return
   case <-pulse.C:
    sendPulse()
   case <-workGen.C:
    sendResult(struct{}{})
   }
  }
 }()
 return heartbeater, result
}

This function returns two channels:

heartbeater: a non-blocking signal that says, "I'm alive!"
result: a channel that emits actual work when done.

In the main function, we simulate a 10-second lifetime for our job using time.AfterFunc. We read from both the heartbeat and result channels.

📡 The Main Function: Listening for Life

Here's what that looks like:

func main() {
 done := make(chan interface{})
 time.AfterFunc(10*time.Second, func() { close(done) })

 pulseInterval := 1 * time.Second
 heartbreater, result := dowork(done, pulseInterval)

 go func() {
  for {
   select {
   case _, ok := <-heartbreater:
    if !ok {
     fmt.Println("worker heartbeat stopped")
     return
    }
    fmt.Println("worker heartbeat")
   case _, ok := <-result:
    if !ok {
     return
    }
    fmt.Println("worker completed work")
   }
  }
 }()

 time.Sleep(20 * time.Second)
}

You'll see heartbeats printed every second, and actual work output every 3 seconds. After 10 seconds, the goroutine closes done, and everything wraps up gracefully.

🧠 A Few Takeaways

Heartbeats let your monitoring tools or orchestration layer know things are alive, even when no work is happening.
Non-blocking sends (select { case ch <- val: default: }) are perfect for "if you're listening, here's a signal-otherwise, no big deal."
Keep your goroutines polite: if they die, let them clean up after themselves.

👀 Final Thoughts
Implementing heartbeats in Go isn't just a cool concurrency trick - it's a must-have tool when you're building resilient systems. You don't want to wait until 3 AM to find out that your queue worker ghosted you. Be proactive. Make your goroutines send a little "I'm alive" wave now and then.
Because of silent failures?
They're not just frustrating - they're expensive.

🔗 Full source code available on GitHub:

👉 architagr/The-Weekly-Golang-Journal - heart beat

Stay Connected!

💡 Follow me on LinkedIn: Archit Agarwal
🎥 Subscribe to my YouTube: The Exception Handler
📬 Sign up for my newsletter: The Weekly Golang Journal
✍️ Follow me on Medium: @architagr
👨‍💻 Join my subreddit: r/GolangJournal
👨‍💻 Follow me on twitter: @architagr

The Silent Killers in Your Go App: Unhandled Errors

Archit Agarwal — Wed, 14 May 2025 10:45:00 +0000

Do you know what's scarier than a panic in a Golang application? 🤔
An error that doesn't panic, doesn't log, doesn't show up - but silently fails in production.
These are like landmines scattered across your codebase, patiently waiting to blow up at the worst possible time - like when you're sipping on a cold beer 🍺, celebrating a successful release.
And then…

📞 "Hey, the feature isn't working. Logs look clean. What's going on?"

Oh no. That's your PM. On a weekend. With no error trace.
I've been there. And yes, I left my beer mid-sip. 😞

When I first started with Go, I didn't fully grasp that errors in Go are first-class citizens - passed around just like any other value. If you don't handle them, they don't scream. They whisper. And that whisper can turn into a production outage.

So this article is not just a rant. It's a survival guide - to help you prevent these ghost errors from ruining your weekend or your system's reputation. 🧯

Let's walk through:

How to structure your error objects for better debugging
How to distinguish between expected and unexpected errors
And how to turn unhandled errors into actionable alerts for your team

Why Error Handling in Go Needs Your Full Attention

Not all errors are created equal. Some are edge cases you anticipate. Some are wild cards. But all of them need structure and context if you want to trace them later.

✅ Expected (or edge-case) errors:

User requests a resource that doesn't exist
Invalid query parameter
Authorization failure
Timeout or network delay

These are normal. They happen. You handle them, maybe return a 400 or 403, and move on.

❌ Unexpected errors (the real troublemakers):

DB connection fails
Memory exhaustion
Filesystem full
Internal services misbehaving
Panic due to nil pointer dereference

These should never happen - but if they do, you want to know immediately.

What Should Your Error Object Include?

Think of your error like a black box recorder. When things go wrong, it should tell you exactly what happened and how to reproduce it. Here's what it must include:

🧩 What happened?

A clear description: e.g. "invalid DB credentials", "failed to marshal JSON", or "timeout while calling payment gateway".

📍 Where did it happen?

Add stack traces or function names - Go makes this easy with tools like runtime.Caller() or libraries like pkg/errorsor xerrors.

⏱️ When did it happen?

Include request context: user ID, request ID, trace ID, timestamp - all of these help recreate the scenario and debug faster.

💬 What should the user see?

Don't leak internals! Instead, return friendly, actionable messages:

"Something went wrong. Please get in touch with support with Reference ID: xyz123."

🔍 How can the devs debug it further?

Include traceable metadata in logs or bug tracking systems so your team has breadcrumbs to follow. Reference IDS go a long way.

In the next section, we'll build a real REST API example and demonstrate how to put this into practice, gracefully handling, categorising, and propagating errors up to monitoring and alerting systems.

But before that, just remember this:
Handling errors isn't just about avoiding panics - it's about staying in control of your system even when it misbehaves.

Building a REST API in Go (Without Losing Your Mind Over Errors)

So, imagine we're building a REST API server - nothing fancy, just your good old layered Go backend:

Router Layer: Parses the HTTP request and routes it appropriately.
Service Layer: Decides whether to hit the DB, cache, or file system.
Persistence Layer: Deals directly with the database.

Let's zoom into the persistence layer, where two common issues can ruin your day:

The database isn't reachable (classic).
The data just… doesn't exist (also classic).

For the second case, we create a custom error structure to wrap all the "not found" drama:

type ObjectNotFoundError struct {
 ObjectType        string         `json:"objectType"`
 ObjectIdentifiers map[string]any `json:"objectIdentifier"`
 StackTrace        string         `json:"stackTrace"`
}

func (e *ObjectNotFoundError) Error() string {
 return fmt.Sprintf("%s not found, filter properties: [%v]", e.ObjectType, e.ObjectIdentifiers)
}
func (e *ObjectNotFoundError) LogMessage() string {
 b, _ := json.Marshal(e)
 return string(b)
}

🧠 What's Cool About This?

Error() gives a user-friendly summary.
LogMessage() gives devs all the gritty details.
ObjectType tells us what was missing.
ObjectIdentifiers explains how we looked for it.
StackTrace points us right to where the problem happened.

Now, let's see this in action in the persistence layer:

func (usrPer *UserPersistence) Get(ctx context.Context, id int) (*entities.User, error) {
 sqlQuery := fmt.Sprintf("select * from users where id = %d", id)
 data, err := usrPer.baseDb.Query(ctx, sqlQuery)
 if err != nil {
  return nil, err   // --- 1️⃣
 }
 if id <= 0 {
  return nil, &ObjectNotFoundError{.  // --- 2️⃣
   ObjectType:        "User",
   ObjectIdentifiers: map[string]any{"id": id},
   StackTrace:        string(debug.Stack()),
  }
 }
 fmt.Println(data)
 // else we will extract data from the data, for this we will hard code this return data
 return &entities.User{
  Id:          id,
  Name:        fmt.Sprintf("user-%d", id),
  CreatedDate: time.Now().Add(-10 * time.Hour),
 }, nil
}

Breakdown

1️⃣ We forward the DB error if it happens.
2️⃣ We simulate an invalid user ID scenario (I know validation doesn't belong here, but bear with me - this is just for the demo).

A small disclaimer: I know this is not the right place for this validation, but I have only done this to explain how to gracefully propagate the error from the innermost layer.

Now Let's Head Over to the Service Layer 🚪

Here's where we catch and convert those deep, scary errors into something the client can actually digest.

type ServiceError struct {
 InnerError     error
 HttpStatusCode int
 HttpResponse   *dto.ErrorResponse
}

func (e *ServiceError) Error() string {
 return e.InnerError.Error()
}

func (e *ServiceError) GetHttpStatusCode() int {
 return e.HttpStatusCode
}

func (e *ServiceError) GetErrorResponse() *dto.ErrorResponse {
 return e.HttpResponse
}

We also define a handy interface:

type ILogMessageError interface {
 error
 LogMessage() string
}

Now, here's the service logic in action:

func (svc UserService) Get(ctx context.Context, id int) (*dto.User, error) {
 fmt.Println("get in the svc")
 userInfo, err := svc.userPersistenceObj.Get(ctx, id)
 if err != nil {
  if e, ok := err.(ILogMessageError); ok {
   return nil, svc.WrapError(e)
  }
  logBug(err)
  return nil, fmt.Errorf("can not retrive user: %d", id)
 }
 return (&dto.User{}).Init(userInfo), nil
}

🧪 What's Happening Here?

Interface Check: We see if the error implements ILogMessageError - if yes, it's a known error.
WrapError: Transforms the error into a format with HTTP status and message.
logBug(): If it's an unexpected error, alert the engineering team. PagerDuty bells ring. People panic. You know the drill.

// handle all types of errors that we expect the persistence to return
// or we can have a builder to build the ServiceError, and also follow
// The open and closed principle, and single responsibility principal
func (svc UserService) WrapError(err ILogMessageError) *ServiceError {
 var objNotFound *persistence.ObjectNotFoundError
 var result *ServiceError
 if errors.As(err, &objNotFound) {
  result = &ServiceError{
   InnerError:     err,
   HttpStatusCode: http.StatusNotFound,
   HttpResponse: &dto.ErrorResponse{
    ErrorCode:        "usr-404",
    ErrorDescription: err.Error(),
   },
  }
 }
 return result
}
func logBug(err error) {
 // log a bug and page enginnering team about the error
 log.Println("logged a bug for error", err, string(debug.Stack()))
}

Finally, The Router - The Gatekeeper to the Client 🌐

Here, we check whether the service-level error can be converted to an HTTP response.

type IHttpResponseError interface {
 GetHttpStatusCode() int
 GetErrorResponse() *dto.ErrorResponse
}

func (r UserRouter) Get(c *gin.Context) {
 var userReq dto.User
 if err := c.ShouldBindUri(&userReq); err != nil {
  c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
  return
 }
 userInfo, err := r.userServiceObj.Get(c.Request.Context(), userReq.Id)
 if err != nil {
  if e, ok := err.(IHttpResponseError); ok {
   c.JSON(e.GetHttpStatusCode(), e.GetErrorResponse())
   return
  }
  c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
  return
 }
 c.JSON(http.StatusOK, userInfo)
}

🧩 Why This Works

Handled errors bubble up in a predictable format with HTTP codes.
Unhandled errors raise the flag to devs and get returned as generic 500s.
Clients stay informed. Devs stay sane. Production stays alive. 😅

🎯 Final Thoughts

And there you have it - an end-to-end error handling strategy that:

Keeps your layers clean and decoupled
Makes debugging a breeze with structured error logs
Ensures your users never see a dreaded "500 Internal Server Error" without context

🔗 Full source code available on GitHub:

👉 architagr/The-Weekly-Golang-Journal - error-propagation

Stay Connected!

Stop Spawning Infinite Goroutines: Use Tee-Channels in Go for Scalable Event Handling

Archit Agarwal — Wed, 23 Apr 2025 10:44:00 +0000

Imagine you're working on an app that allows users to sign up for free. But you also want to make sure that someone from the sales team follows up to nudge the user to subscribe. So you have two tasks after the user is successfully signed up:

Send a welcome email to the user.
Notify the sales team on Slack.

Naturally, your first version looks something like this:

User signs up
You send a welcome email
Once that's successful, you notify the sales team

Sounds clean.
But here's the thing - you've now tied the success of the sales notification to the welcome email, which is not just a coupling nightmare, it's also a recipe for "Oops, we never told the sales team."

⚠️ The Goroutine Trap: Scaling Gone Wild

So you think, "Let's parallelise it! Fire two goroutines!"
Great. Until…
You host a workshop, and suddenly, a hundred thousand users are signing up.
Now your server is firing two goroutines per user and context switching like it's in a dance battle. CPU usage spikes, latency increases, and your devops team starts slacking you 😅.

💡 Observing the Pattern: The Tasks are Independent!

Take a closer look. The welcome email and the Slack notification are independent. They don't need to wait on each other. They don't even need to happen in real-time.
Which begs the question - why not broadcast the event and let each downstream process consume it at its own pace?
Enter: Tee-Channels in Go.

🧵 What is a Tee-Channel in Go?

A tee-channel is like a broadcast system for Go channels. One input channel, multiple output channels. Each downstream consumer listens to their dedicated output and does their thing.
You don't spawn two goroutines per user anymore. You pipe the user once, and split it out - cleanly.

👨‍💻 Let's Build It: A Tee-Channel Implementation in Go

Here's a simple, scalable version that processes signups, and then concurrently (but safely) fans out to:

Send welcome emails
Notify the marketing team

Step 1: Define a simple DTO

type userDTO struct {
 id    int
 name  string
 email string
}

Step 2: Create a helper to respect done context

We'll use orDone so that everything shuts down gracefully if the context is cancelled.

func orDone(done <-chan struct{}, c <-chan userDTO) <-chan userDTO {
 valStream := make(chan userDTO)
 go func() {
  defer close(valStream)
  for {
   select {
   case <-done:
    return
   case v, ok := <-c:
    if !ok {
     return
    }
    select {
    case valStream <- v:
    case <-done:
    }
   }
  }
 }()
 return valStream
}

Step 3: Implement the Tee-Channel logic

func tee(
 done <-chan struct{},
 in <-chan userDTO,
) (<-chan userDTO, <-chan userDTO) {
 out1 := make(chan userDTO, 1_000)
 out2 := make(chan userDTO, 1_000)

 go func() {
  defer close(out1)
  defer close(out2)
  for val := range orDone(done, in) {
   var o1, o2 = out1, out2
   for i := 0; i < 2; i++ {
    select {
    case <-done:
    case o1 <- val:
     o1 = nil
    case o2 <- val:
     o2 = nil
    }
   }
  }
 }()
 return out1, out2
}

Here's what's happening:

We listen to the input stream (in)
Each input value is sent to both output channels exactly once.
It's concurrency-safe and honours context cancellation.

Step 4: Handlers for Email and Slack

func sendWelcomeEmail(done <-chan struct{}, ch <-chan userDTO) {
 for u := range orDone(done, ch) {
  fmt.Println("📧 Sending welcome email to", u.email)
 }
 fmt.Println("✅ Email handler done")
}

func sendNotification(done <-chan struct{}, ch <-chan userDTO, slackChannel string) {
 for u := range orDone(done, ch) {
  fmt.Println("📣 Notifying", slackChannel, "team for user", u.name)
 }
 fmt.Println("✅ Slack handler done")
}

Step 5: Putting It All Together

func singnup(u userDTO, ch chan<- userDTO) {
 // Simulate saving to DB
 ch <- u
}

func main() {
 ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
 defer cancel()

 notificationChannel := make(chan userDTO, 1_000)

 // Split the input to two output handlers
 ch1, ch2 := tee(ctx.Done(), notificationChannel)

 // Start handlers
 go sendNotification(ctx.Done(), ch1, "marketing")
 go sendWelcomeEmail(ctx.Done(), ch2)

 // Simulate signups
 singnup(userDTO{id: 1, name: "Archit", email: "a@example.com"}, notificationChannel)
 singnup(userDTO{id: 2, name: "Lilly", email: "lilly@example.com"}, notificationChannel)

 time.Sleep(2 * time.Second) // Let handlers finish
}

You can find this complete code in my GitHub : https://github.com/architagr/The-Weekly-Golang-Journal/tree/main/tee-channel

🧠 A Few Pro Tips

The orDone pattern ensures your goroutines stop listening when the context is cancelled.
tee() is the magic sauce - it splits the input stream cleanly.
This is future-proof. You can later plug more consumers like analytics, onboarding workflows, etc., without modifying the producer.

🎯 When Should You Use Tee-Channels?

✅ When multiple consumers need the same data
✅ When you want to decouple event producers from consumers
✅ When you want to avoid unbounded goroutine creation
Not a good fit if:

You need real-time delivery with strict ordering guarantees
You expect high fan-out (many consumers) - in that case, use a pub-sub system

🧵 Final Thoughts: Small Pattern, Big Impact

Honestly, this pattern looks simple. But in high-scale systems, it becomes a lifesaver. It gives you concurrency without chaos and scalability without the CPU meltdown.
So next time someone signs up on your app - just tee it up! 😄

Stay Connected!

Master the Chain of Responsibility Pattern in Go with This Real-World Example

Archit Agarwal — Wed, 09 Apr 2025 10:45:00 +0000

🧶 Intro: From Spaghetti to Structured

Ever written a function that starts by fetching user data, dumps it into a file, zips the file, then emails it out - all in one go?
It feels like this:

func exportEverything() {
  fetchUsers()
  parseUsers()
  zipFile()
  emailZip()
}

And sure, it works. Until the day something changes.

Suddenly, you need to skip zipping for certain users, or plug in an S3 upload. You end up sprinkling conditionals everywhere. Before you know it, you've summoned a micro-monolith. And worse? It's not even testable.

So… what do we do?

🧩 Enter: Chain of Responsibility Pattern

The Chain of Responsibility is a behavioral design pattern that lets you pass requests down a chain of handlers. Each handler does its job and optionally passes the request forward.
Think of it like a tech support escalation:

Level 1 takes your call
If they can't help, they pass it to Level 2
If that fails, well… the manager steps in

In Go, we can use this pattern to cleanly break up our export logic into small, reusable, testable chunks.

🛠️ The Real-World Example: Exporting User Data

Let's say we have a process that:

Fetches users from the DB
Parses them into a file
Zips that file
Emails the zip to the requester

Using the Chain of Responsibility, each step becomes a handler:

type exportStepHandler interface {
 execute(req *exportRequest) error
 setNext(next exportStepHandler)
}

Each handler implements this interface and decides what to do with the request. Here's a quick peek:

🏃 Fetch Users

type exportFetchUsers struct {
 next exportStepHandler
}

func (h *exportFetchUsers) execute(req *exportRequest) error {
 fmt.Printf("RequestId(%d): Users fetched from DB\n", req.requestid)
 return h.callNext(req)
}

🧻 Parse Into File

type exportParseIntoFile struct {
 next exportStepHandler
}

func (h *exportParseIntoFile) execute(req *exportRequest) error {
 fmt.Printf("RequestId(%d): Users parsed into file\n", req.requestid)
 return h.callNext(req)
}

📦 Zip It Up

type exportZipFiles struct {
 next exportStepHandler
}

func (h *exportZipFiles) execute(req *exportRequest) error {
 fmt.Printf("RequestId(%d): file zipped\n", req.requestid)
 return h.callNext(req)
}

✉️ Email the Zip

type exportEmailZip struct {
 next exportStepHandler
}

func (h *exportEmailZip) execute(req *exportRequest) error {
 fmt.Printf("RequestId(%d): Zip emailed\n", req.requestid)
 return h.callNext(req)
}

🧩 Connecting the Chain

We then wire everything like a good old-fashioned factory:

func InitExportData() exportStepHandler {
 emailStep := &exportEmailZip{}

 zipStep := &exportZipFiles{}
 zipStep.setNext(emailStep)

 parseStep := &exportParseIntoFile{}
 parseStep.setNext(zipStep)

 fetchStep := &exportFetchUsers{}
 fetchStep.setNext(parseStep)

 return fetchStep
}

And then… 🧙 magic:

func main() {
 exportChain := InitExportData()
 exportChain.execute(&exportRequest{
  requestid: 1,
 })
}

🧠 Why This Pattern Rocks

✅ Readable: Each handler does one thing - clean and simple.

✅ Maintainable: Change one step without touching others.

✅ Flexible: Easily rewire or skip steps depending on context.

✅ Testable: Unit test each handler in isolation.

✅ Extensible: Want to log something or upload to S3? Just add a new handler!

🧵 Final Thoughts

If you've ever struggled with long, procedural workflows in Go, this pattern is a game-changer.
The Chain of Responsibility gives you a structure that scales. It turns a rigid flow into a flexible, testable, and modular pipeline.
So next time you think:

"Hmm, maybe I'll just add another if…"

Take a deep breath. Then chain it up. 🪢

Stay Connected!

The Strategy Pattern in Go: A Simple Guide to Writing Extensible Code

Archit Agarwal — Wed, 02 Apr 2025 10:45:00 +0000

Imagine you're running a coffee shop ☕, and customers ask for their order receipts in different formats - some want their receipts printed, some in PDF sent over Email, and one guy asks for a receipt over SMS. Now, imagine that you have written this billing software to support a printed receipt.🫣
Every time a new type of receipt is requested, you rewrite your entire billing system. Sounds painful, right? Well, that's precisely what happened in our Go code when UserService was initially responsible for exporting users as JSON and CSV, and now we onboard a client who wants this in XML. Let's look at the initial UserService.

Initial UserService: Without using Strategy Design Pattern

Imagine you have a UserService that fetches user data and allows exporting it in different formats. Right now, you support JSON and CSV, but what if tomorrow you need XML, PDF, or Parquet?

var (
 Err_NotSupportedExportType = errors.New("Not Supported export Type")
)

type IUserPersistence interface {
 ListUsers(paginationObj models.Pagination) []models.UserModel
}
type UserService struct {
 userPersistence IUserPersistence
}

func InitUserService(userPersistence IUserPersistence) *UserService {
 return &UserService{
  userPersistence: userPersistence,
 }
}

func (userSvc *UserService) GetUsers(paginationObj models.Pagination) []models.UserModel {
 return userSvc.userPersistence.ListUsers(paginationObj)
}

func (userSvc *UserService) ExportUsers(paginationObj models.Pagination, 
exportConfig models.ExportConfig) (string, error) {

 users := userSvc.GetUsers(paginationObj)

switch exportConfig.Type {
 case enums.ExportType_Json:
  jsonData, err := json.Marshal(users)
  return string(jsonData), err
 case enums.ExportType_CSV:
  sb := strings.Builder{}
  sb.WriteString("id,name,email\n")
  for _, u := range users {
   sb.WriteString(fmt.Sprintf("%d,%s,%s\n", u.Id, u.Name, u.Email))
  }
  return sb.String(), nil
 default:
  return "", Err_NotSupportedExportType
 }
}

The Problem with This Approach

With the current implementation, every time you add a new export format, you modify the ExportUsers method. This violates two key SOLID principles:

Single Responsibility Principle (SRP) - UserService should focus only on fetching users, not handling export logic.
Open/Closed Principle (OCP) - The class should be open for extension but closed for modification. Right now, adding a new export type means editing existing code, increasing the risk of breaking something.

Issues with this approach:

💔 UserService becomes a bottleneck.
💀 Every new format requires modifying existing code.
💥 One small change could break everything.
😭 Teams adding new export formats need to touch your code.
😫 Testing becomes harder as the service grows.

How Can We Fix This?

Instead of making UserService aware of every export format, we need a pluggable solution. This is where the Strategy Pattern shines! ✨

💡 Introducing the Strategy Pattern

Think of it like choosing a shipping method on Amazon: You pick Standard Shipping, Express, or Same-Day - but Amazon doesn't care how it's delivered. The delivery company handles that.
Similarly, our UserService should just fetch users and let the export strategy decide how to format them.

Are you ready to start the code repair 👩🏻‍🔧?

Step 1: Define a Common Export Interface

// Strategy interface
type IUserExport interface {
 Export(data []models.UserModel) (string, error)
}

Implement Concrete Strategies

// CSV Export: Concrete Strategy
type CsvExportStrategy struct {
}

func (e *CsvExportStrategy) Export(data []models.UserModel) (string, error) {
 if len(data) == 0 {
  return "", errors.New("No data")
 }
 sb := strings.Builder{}
 sb.WriteString(data[0].GetHeader())
 for _, u := range data {
  sb.WriteString(u.GetRow())
 }
 return sb.String(), nil
}

// JSON Export: Concrete Strategy
type JSONExportStrategy struct {
}

func (e *JSONExportStrategy) Export(data []models.UserModel) (string, error) {
 if len(data) == 0 {
  return "", errors.New("No data")
 }
 jsonData, err := json.Marshal(data)
 return string(jsonData), err
}

Step 3: Create a Strategy Builder

var (
 Err_NotSupportedExportType = errors.New("Not Supported export Type")
)

func UserExportBuilder(config *models.ExportConfig) (IUserExport, error) {
 if config.Type == enums.ExportType_CSV {
  return &CsvExportStrategy{}, nil
 } else if config.Type == enums.ExportType_Json {
  return &JSONExportStrategy{}, nil
 }
 return nil, fmt.Errorf("%w, %d", Err_NotSupportedExportType, config.Type)
}

Step 4: Update UserService 🥁

func (userSvc *UserService) ExportUsers(paginationObj models.Pagination, 
exportConfig exportstrategy.IUserExport) (string, error) {

 users := userSvc.GetUsers(paginationObj)
 return exportConfig.Export(users)
}

Now, ExportUsers does not care how data is exported. It only retrieves users and sends the data to the correct export strategy.

Step 5: Adding a New Format (XML)

// XML Export: Concrete Strategy
type XMLExportStrategy struct {
}

func (e *XMLExportStrategy) Export(data []models.UserModel) (string, error) {
 if len(data) == 0 {
  return "", errors.New("No data")
 }
 xmlData, err := xml.Marshal(data)
 return string(xmlData), err
}

Step 6: Update the Strategy Builder

func UserExportBuilder(config *models.ExportConfig) (IUserExport, error) {
 switch config.Type {
 case enums.ExportType_CSV:
  return &CsvExportStrategy{}, nil
 case enums.ExportType_Json:
  return &JSONExportStrategy{}, nil
 case enums.ExportType_Xml:
  return &XMLExportStrategy{}, nil
 default:
  return nil, fmt.Errorf("%w, %d", Err_NotSupportedExportType, config.Type)
 }
}

You can find this code on my Github: https://github.com/architagr/design_patterns/tree/main/golang/behavioral/strategy

🚀 Final Thoughts & Takeaways

Before: Every new format meant editing UserService, risking bugs.
After: Adding new formats is just creating a new strategy, with no changes to the core logic.
Note:
If your if-else statements are growing faster than your stress levels, it's time for the Strategy Pattern.

Stay Connected!

Never Miss an Update: Master the Observer Pattern in Go with Real-World Examples

Archit Agarwal — Wed, 26 Mar 2025 10:45:00 +0000

Why Should You Care About the Observer Pattern?

Have you ever felt the FOMO when you miss out on an important update? Imagine a world where you must constantly refresh your email to check for new messages. Sounds painful, right? 😩

Well, that’s exactly how inefficient polling works in software. Instead of constantly checking, wouldn’t it be better if you just got notified automatically when something changed?

That’s where the Observer Pattern comes in! This pattern lets different parts of your system subscribe to updates and automatically receive notifications when an event occurs. It’s the backbone of event-driven programming and is used in real-world systems like:

✅ Live sports score updates
✅ Stock market price alerts
✅ Job portal notifications
✅ E-commerce product price drop alerts
✅ Hashtag-based content subscription (which we’ll build today!)

In this article, we’ll demystify the Observer Pattern and implement a Hashtag Subscription System in Golang, where users can subscribe to specific hashtags and receive notifications when new articles are published.

What is the Observer Pattern?

The Observer Pattern is a behavioural design pattern where:

A Subject (Publisher) maintains a list of Observers (Subscribers).
Whenever the subject’s state changes, it notifies all registered observers automatically.
Observers can subscribe or unsubscribe dynamically without affecting the subject’s core logic. This pattern decouples objects, making systems more modular and scalable.

Implementing the Observer Pattern in Go

Let’s build a Hashtag Subscription System, where:

📌 Users subscribe to hashtags they’re interested in.
📌 When an article is published under a hashtag, subscribers get notified.
📌 Users can unsubscribe from hashtags at any time.
📌 Users can block specific authors if they don’t want to see their content.

Step 1: Define the Observer Interface (Users)

package publisher

import "fmt"

// Observer Interface (Subscribers)
type IHashtagObserver interface {
 Notify(articleID, author string)
}

Each user who wants to receive hashtag updates must implement the Notify method.

Step 2: Define the Subject (Hashtag Manager)

// Hashtag acts as a subject (Publisher)
type Hashtag struct {
 observers map[string]IHashtagObserver
 name      string
 articles  map[string]bool
}

// Register an observer (User)
func (tag *Hashtag) Register(name string, observer IHashtagObserver) {
 tag.observers[name] = observer
}

// Deregister an observer (User)
func (tag *Hashtag) Deregister(name string) {
 delete(tag.observers, name)
}

// Publish a new article
func (tag *Hashtag) NewArticlePublished(articleID, author string) {
 fmt.Printf("\n📢 Notifying subscribers of #%s about new article (ID: %s) by %s\n", tag.name, articleID, author)
 tag.notify(articleID, author)
}

// Notify all observers
func (tag *Hashtag) notify(articleID, author string) {
 for _, observer := range tag.observers {
  observer.Notify(articleID, author)
 }
}

The Hashtag Manager maintains a list of subscribers and notifies them when a new article is published.

Step 3: Implement the Concrete Observer (User)

// User struct acts as an Observer
type User struct {
 id            string
 name          string
 email         string
 slackUserID   string
 hashtags      map[string]bool
 blockedAuthor map[string]bool
}

// Add an author to the blocked list
func (u *User) AddBlockedAuthor(author string) {
 u.blockedAuthor[author] = true
}

// Remove an author from the blocked list
func (u *User) RemoveBlockedAuthor(author string) {
 delete(u.blockedAuthor, author)
}

// Subscribe to a hashtag
func (u *User) AddHashtag(tagName string) {
 u.hashtags[tagName] = true
 publisher.GetHashtag(tagName).Register(u.name, u)
}

// Unsubscribe from a hashtag
func (u *User) RemoveHashtag(tagName string) {
 delete(u.hashtags, tagName)
 publisher.GetHashtag(tagName).Deregister(u.name)
}

// Receive notifications for new articles
func (u *User) Notify(articleID, author string) {
 if u.blockedAuthor[author] {
  fmt.Printf("🚫 %s skipped notification for article (ID: %s) - blocked author: %s\n", u.name, articleID, author)
  return
 }
 fmt.Printf("\n📨 Sending notification to %s\n", u.name)
 if u.email != "" {
  fmt.Printf("📧 Email sent to %s for new article (ID: %s) by %s\n", u.email, articleID, author)
 }
 if u.slackUserID != "" {
  fmt.Printf("💬 Slack notification sent to %s for new article (ID: %s) by %s\n", u.slackUserID, articleID, author)
 }
}

Each user can subscribe/unsubscribe from hashtags dynamically!

Step 4: Bringing It All Together

func main() {
 var (
  tagCleanCode           = "Clean Code"
  tagSoftwareDevelopment = "Software Development"
 )
 publisher.AddNewHashtag(tagSoftwareDevelopment)
 publisher.AddNewHashtag(tagCleanCode)

 // Create users and subscribe them to hashtags
 user.InitUser("1", "Alice", "alice@example.com", "alice_slack", []string{tagSoftwareDevelopment}, nil)
 user.InitUser("2", "Bob", "bob@example.com", "bob_slack", []string{tagSoftwareDevelopment, tagCleanCode}, nil)
 user3 := user.InitUser("3", "Charlie", "charlie@example.com", "charlie_slack", []string{tagSoftwareDevelopment, tagCleanCode}, []string{"author1"})

 // Publish new articles under hashtags
 publisher.GetHashtag(tagSoftwareDevelopment).NewArticlePublished("article1", "author2")
 publisher.GetHashtag(tagSoftwareDevelopment).NewArticlePublished("article2", "author1") // Charlie blocked author1
 publisher.GetHashtag(tagCleanCode).NewArticlePublished("article3", "author1")

 // Charlie unsubscribes from "Clean Code" hashtag
 user3.RemoveHashtag(tagCleanCode)
 publisher.GetHashtag(tagCleanCode).NewArticlePublished("article4", "author2") // Charlie won't be notified
}

Expected Output

📢 Notifying subscribers of #Software Development about new article (ID: article1) by author2

📨 Sending notification to Alice
📧 Email sent to alice@example.com for new article (ID: article1) by author2
💬 Slack notification sent to alice_slack for new article (ID: article1) by author2

📨 Sending notification to Bob
📧 Email sent to bob@example.com for new article (ID: article1) by author2
💬 Slack notification sent to bob_slack for new article (ID: article1) by author2

📨 Sending notification to Charlie
📧 Email sent to charlie@example.com for new article (ID: article1) by author2
💬 Slack notification sent to charlie_slack for new article (ID: article1) by author2

📢 Notifying subscribers of #Software Development about new article (ID: article2) by author1

📨 Sending notification to Alice
📧 Email sent to alice@example.com for new article (ID: article2) by author1
💬 Slack notification sent to alice_slack for new article (ID: article2) by author1

📨 Sending notification to Bob
📧 Email sent to bob@example.com for new article (ID: article2) by author1
💬 Slack notification sent to bob_slack for new article (ID: article2) by author1
🚫 Charlie skipped notification for article (ID: article2) - blocked author: author1

📢 Notifying subscribers of #Clean Code about new article (ID: article3) by author1

📨 Sending notification to Bob
📧 Email sent to bob@example.com for new article (ID: article3) by author1
💬 Slack notification sent to bob_slack for new article (ID: article3) by author1
🚫 Charlie skipped notification for article (ID: article3) - blocked author: author1

📢 Notifying subscribers of #Clean Code about new article (ID: article4) by author2

📨 Sending notification to Bob
📧 Email sent to bob@example.com for new article (ID: article4) by author2
💬 Slack notification sent to bob_slack for new article (ID: article4) by author2

Boom! 🎉 Alice and Bob got their notifications instantly and Charlie also was able to block a particular author! No polling, no wasted CPU cycles. Just clean, efficient event-driven programming.

You can find this complete working code on my Github.

Key Takeaways

✅ The Observer Pattern helps create event-driven, decoupled systems.
✅ It's used in stock updates, job alerts, social media notifications, and more.
✅ Our Hashtag Subscription System shows how users can subscribe, unsubscribe, and receive notifications in real-time.

Final Thoughts

This is just the tip of the iceberg. You can extend this pattern for WebSockets, message brokers, or even real-time dashboards! 🚀

Stay Connected!

💡 Follow me here on LinkedIn for more insights on software development and architecture.
🎥 Subscribe to my YouTube channel for in-depth tutorials.
📬 Sign up for my newsletter, The Weekly Golang Journal, for exclusive content.
✍️ Follow me on Medium for detailed articles.
👨‍💻 Join the discussion on my subreddit, r/GolangJournal, and be part of the community!

Stop Writing Rigid Code! Master the Decorator Pattern in Go for Ultimate Flexibility

Archit Agarwal — Wed, 12 Mar 2025 10:45:00 +0000

Imagine you're a software developer building an ice cream vending machine for Disneyland. The menu was initially fixed - customers could only order predefined flavors with no customizations (boring, right? 😒). You wrote simple code to calculate costs based on these fixed options.

But this is Disneyland! Nobody wants a plain old scoop. People want triple scoops, extra chocolate sauce, stacked wafer cones, sprinkles, and more. You realize you can't hardcode every possible combination, or your codebase will become a nightmare! 🫠

So, how do you handle dynamic customizations without rewriting everything? Enter the Decorator Pattern, a lifesaver for scenarios where you need to dynamically extend an object's functionality without modifying its structure.

Let's first explore the problem with a naive approach and then refactor it using the Decorator Pattern for a flexible and scalable solution. 🚀

The Traditional Approach: Hardcoding Ice Cream Variants

Imagine our first attempt at coding the vending machine. A simple function creates a Butterscotch Ice Cream:

func CreateButterscotchIcecream(scoops int) IceCream {
    ingredients := []IceCreamIngredient{
        BASE_PLAIN_CONE,
    }
    for i := 0; i < scoops; i++ {
        ingredients = append(ingredients, FLAVOUR_BUTTERSCOTCH)
    }
    ingredients = append(ingredients, TOPPING_SPRINKLES)
    return Create(CreateIceCreamRequest{Ingrediants: ingredients})
}

Now, we add Vanilla Ice Cream:

func CreateVanillaIcecream(scoops int) IceCream {
    ingredients := []IceCreamIngredient{
        BASE_PLAIN_CONE,
    }
    for i := 0; i < scoops; i++ {
        ingredients = append(ingredients, FLAVOUR_VANILLA)
    }
    return Create(CreateIceCreamRequest{Ingrediants: ingredients})
}

This works fine for fixed flavours, but what if customers want to mix and match? What if someone wants vanilla with chocolate chips or a chocolate cone with butterscotch scoops? We can't keep adding new functions for every combination. 😵‍💫

The Problem? 🚨

Hardcoding every variation is not scalable.
The code becomes repetitive and difficult to maintain.
There is no flexibility to create custom orders dynamically.

Time to refactor using the Decorator Pattern!

Introducing the Decorator Pattern 🎩✨

The Decorator Pattern allows us to dynamically extend an object's functionality without modifying its structure. Instead of hardcoding different types of ice cream, we use composition over inheritance to wrap functionalities around each other.

Think of your manager adding their name to your presentation after changing the font size - that's the Decorator Pattern at work. 😂

Step 1: Define a Common Interface

We start by creating an interface that all ice cream components (base, flavours, and toppings) will implement:

type IIceCreamIngredient interface {
    GetPreparationSteps() []string
    GetCost() int
}

Step 2: Implement the Base Ingredient (Chocolate Cone)

Now, let's implement the Chocolate Cone decorator, which wraps around another ingredient:

type BaseChocolateCone struct {
 ExistingIngrediant IIceCreamIngrediant
}

func (ingrediant *BaseChocolateCone) GetPreperationSteps() []string {
 step := "Chocolate Cone"
 if ingrediant.ExistingIngrediant != nil {
  return append(ingrediant.ExistingIngrediant.GetPreperationSteps(), step)
 }
 return []string{step}
}
func (ingrediant *BaseChocolateCone) GetCost() int {
 oldCost := 0
 if ingrediant.ExistingIngrediant != nil {
  oldCost = ingrediant.ExistingIngrediant.GetCost()
 }
 return 12 + oldCost
}

✅ Breakdown:

If this ingredient is added on top of something else, it appends the steps to the existing preparation list.
If it's the first ingredient, it starts the list.
Cost calculation is handled dynamically by summing up all wrapped ingredients.

Step 3: Implement More Decorators

Let's implement a Butterscotch Flavor decorator:

type FlavourButterscotch struct {
 ExistingIngrediant IIceCreamIngrediant
}

func (ingrediant *FlavourButterscotch) GetPreperationSteps() []string {
 step := "1 scoop Butterscotch"
 if ingrediant.ExistingIngrediant != nil {
  return append(ingrediant.ExistingIngrediant.GetPreperationSteps(), step)
 }
 return []string{step}
}
func (ingrediant *FlavourButterscotch) GetCost() int {
 oldCost := 0
 if ingrediant.ExistingIngrediant != nil {
  oldCost = ingrediant.ExistingIngrediant.GetCost()
 }
 return 5 + oldCost
}

With this setup, we can now chain ingredients dynamically! 🎉

Step 4: Dynamically Creating Ice Cream Orders 🍨

Now, let's take a user's order and create their dream ice cream dynamically:

func main() {
    iceCream := &BaseChocolateCone{
        ExistingIngredient: &FlavourButterscotch{
            ExistingIngredient: &FlavourVanilla{},
        },
    }

    fmt.Println("Steps to prepare your ice cream:")
    for index, step := range iceCream.GetPreparationSteps() {
        fmt.Printf("Step %d: %s\n", index+1, step)
    }
    fmt.Printf("Total Cost: $%d\n", iceCream.GetCost())
}

Output:

Steps to prepare your ice cream:
Step 1: Chocolate Cone
Step 2: 1 scoop Butterscotch
Step 3: 1 scoop Vanilla
Total Cost: $20

✅ Now, any combination is possible dynamically!

No hardcoding different combinations
Easily scalable for new flavours & toppings
Follows the Open-Closed Principle (extend behavior without modifying existing code)

The complete Code can be found on my GitHub https://github.com/architagr/design_patterns/tree/main/golang/structural/decorator

Real-World Use Cases of the Decorator Pattern

Logging Middleware (Wrap API requests to log data)
Authentication Middleware (Add security checks dynamically)
Rate Limiting (Restrict API calls per user)
Caching Mechanism (Cache responses dynamically)
Monitoring & Metrics Collection (Track execution time)
Encryption & Compression (Wrap data processing functions)
Feature Flags (Enable/disable features at runtime)
Payment Processing (Add discounts/taxes dynamically)
Data Validation (Validate user input before processing)
Web Scraping Pipelines (Apply transformation layers dynamically)

Wrapping Up 🎁

The Decorator Pattern is a powerful technique in Go that allows you to dynamically compose behaviours while keeping your codebase clean and scalable. Next time you face a scenario where functionalities need to be added dynamically without modifying the base code think Decorators!

Have you used the Decorator Pattern in your projects? Share your experience in the comments! 👇

Stay Connected!

💡 Follow me here on LinkedIn for more insights on software development and architecture.

🎥 Subscribe to my YouTube channel for in-depth tutorials.

📬 Sign up for my newsletter, The Weekly Golang Journal, for exclusive content.

✍️ Follow me on Medium for detailed articles

👨‍💻 Join the discussion on my subreddit, r/GolangJournal, and be part of the community!