DEV Community: Odilon HUGONNOT

Testing a production Bash script: 38 tests without a framework

Odilon HUGONNOT — Wed, 29 Apr 2026 09:00:02 +0000

The context: a Bash script to display Claude Code rate limits in a terminal status bar. At first it seems trivial — a few functions, reading a JSON file, some formatting. But the script quickly ends up handling: JSON parsing with jq, time formatting with timezone, percentage calculations, Unicode progress bar generation, countdown logic. And above all: several edge cases (missing data, corrupted JSON, quotas at 0, unknown timezone).

Without tests, every change becomes a lottery. You change the countdown formatting, you don't know if you broke the percentage calculation. You add active model handling, you don't know if the corrupted JSON fixtures still pass. With 38 unit tests on a Bash script, you modify with confidence — and bash tests/test_statusline.sh in 2 seconds confirms nothing is broken.

Why no Bash testing framework

Bash testing frameworks exist: bats-core, shunit2, shellspec. They're good. For this project, the choice was not to add one — zero external dependencies in the repo, one-liner install. A test framework is a dependency. The constraint was simple: tests run with native bash, nothing to install.

Result: ~100 lines of homemade test helpers. No magic, no DSL, no plugin to update — just functions and conventions. For a 300-line script, that's the right level of engineering. A testing framework to test a Bash script is often more complexity than the script itself.

The basic structure: assertions and counters

The test file starts with two global counters and two assertion functions:

#!/usr/bin/env bash
# tests/test_statusline.sh

PASS=0
FAIL=0

assert_eq() {
    local description="$1"
    local expected="$2"
    local actual="$3"

    if [[ "$expected" == "$actual" ]]; then
        echo "  ✓ $description"
        ((PASS++))
    else
        echo "  ✗ $description"
        echo "    expected: $(echo "$expected" | cat -A)"
        echo "    actual:   $(echo "$actual" | cat -A)"
        ((FAIL++))
    fi
}

assert_contains() {
    local description="$1"
    local needle="$2"
    local haystack="$3"

    if [[ "$haystack" == *"$needle"* ]]; then
        echo "  ✓ $description"
        ((PASS++))
    else
        echo "  ✗ $description"
        echo "    expected to contain: $needle"
        echo "    actual: $haystack"
        ((FAIL++))
    fi
}

# At the end of the test file:
echo ""
echo "Results: $PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]] || exit 1

The cat -A in the error message displays invisible characters (spaces, tabs, \n, $ at end of line) — essential for debugging tests that fail on an extra space in a formatted string. Without it, you spend 20 minutes comparing two strings that look identical visually but aren't equal.

Testing Bash functions: the isolation problem

The main script (statusline.sh) isn't designed to be sourced in tests — it executes top-level code as soon as it's launched. The pattern to isolate functions without modifying production behavior:

# In statusline.sh, wrap the executable code:
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    # Executed only when the script is run directly
    main "$@"
fi

# Functions defined in the file remain available when sourced

In the test file, source the functions without triggering main:

# Source functions without executing main
source "$(dirname "$0")/../statusline.sh"

# Now we can test functions directly
result=$(format_percentage 67)
assert_eq "67% gives yellow" "🟡" "$result"

BASH_SOURCE[0] holds the path of the currently executing script. $0 holds the path of the script launched by the user. When you source a file, these two values differ — the if doesn't execute.

Mocking external dependencies

The script calls jq, reads JSON files, uses date with timezone. For unit tests on Bash scripts, you mock by redefining commands in the test scope — Bash resolves functions before binaries in PATH:

# Mock jq to return controlled data
jq() {
    echo "42"
}
export -f jq

result=$(get_context_percentage)
assert_eq "context at 42%" "42" "$result"

# Clean up the mock after the test
unset -f jq

For more complex JSON data, create fixtures in tests/fixtures/ and override the path variable via the environment:

USAGE_FILE="tests/fixtures/usage_normal.json" \
    result=$(render_statusline)
assert_contains "displays the model" "Sonnet 4.6" "$result"

USAGE_FILE="tests/fixtures/usage_corrupted.json" \
    result=$(render_statusline)
assert_contains "corrupted JSON → fallback" "N/A" "$result"

The advantage: fixtures are real readable JSON files, versionable, easy to update when the format changes. No need to mock jq for each case — you provide the input data directly.

A concrete example: testing the Unicode progress bar

The progress bar (▓▓▓░░░░░) is one of the most tested functions — it must correctly handle edge cases of Bash integer division:

test_progress_bar() {
    echo "--- Progress bar ---"
    assert_eq "0%  → 8 empty blocks"     "░░░░░░░░" "$(make_bar 0)"
    assert_eq "50% → 4 full 4 empty"     "▓▓▓▓░░░░" "$(make_bar 50)"
    assert_eq "100%→ 8 full blocks"      "▓▓▓▓▓▓▓▓" "$(make_bar 100)"
    assert_eq "34% → rounded down"       "▓▓░░░░░░" "$(make_bar 34)"
    assert_eq "99% → 7 full"             "▓▓▓▓▓▓▓░" "$(make_bar 99)"
}

5 tests, 5 edge cases. The 34% case is the most important: 34 * 8 / 100 = 2.72 → rounded to 2 by Bash integer division (no bc, no awk). Without an explicit test, this behavior is discovered in production when the bar displays one too many blocks. The 99% case verifies that the last block stays empty — 7 full, not 8.

The result: 38 tests, 0 external framework

Final organization of tests by category:

Progress bar (5 tests) — boundary values, integer division rounding
Color coding (6 tests) — green/yellow/red thresholds by percentage
Percentage formatting (4 tests) — display with %, zero, 100
Time formatting (8 tests) — countdown, timezone, reset times, midnight
JSON parsing (7 tests) — missing data, null values, corrupted JSON
Full statusline render (8 tests) — complete output with real data fixtures

Run: bash tests/test_statusline.sh. Output:

--- Progress bar ---
  ✓ 0% → 8 empty blocks
  ✓ 50% → 4 full 4 empty
  ✓ 100% → 8 full blocks
  ✓ 34% → rounded down
  ✓ 99% → 7 full

--- Color coding ---
  ✓ 0% → green
  ✓ 74% → green
  ✓ 75% → yellow
  ✓ 89% → yellow
  ✓ 90% → red
  ✓ 100% → red

[...]

Results: 38 passed, 0 failed

What it changes in practice

The statusline evolved 12+ times in a single day of development: adding the active model, overhauling the countdown, adding git dirty status, handling exhausted quotas. With each change, bash tests/test_statusline.sh in 2 seconds. Regressions are caught immediately — not after restarting Claude Code to see that the bar displays N/A instead of the percentage.

Bash has no typing, no strict linter, no IDE that highlights logic errors. Tests compensate. This isn't a substitute for a real language — it's the minimal safety net that makes the script modifiable without anxiety.

Conclusion

100 lines of helpers. 38 Bash unit tests. Zero external dependencies. For a production Bash script that needs to be modifiable safely, that's the right investment. The pattern is portable to any Bash project: assert_eq, assert_contains, mock by function redefinition, fixtures as JSON files. No need for a testing framework to test a 300-line script.

The full project is on GitHub: github.com/ohugonnot/claude-code-statusline

Migrating Postman to Bruno in a monorepo: the practical guide

Odilon HUGONNOT — Tue, 28 Apr 2026 09:00:02 +0000

The ticket lands on a Monday morning: "Migrate Postman collections to Bruno, integrate them into the monorepo." You've used Postman for years without thinking too hard about it. Bruno you've heard of but never opened. The official docs list features. Medium articles explain "the 10 reasons to switch." Neither tells you concretely how to start clean on a monorepo with 6 services — patient API, lab, doctor portal, backoffice, billing, drug database — 3 environments, and secrets you can't commit.

This is that guide. No preamble, no progressive intro — straight to what you need to know so the result is clean from the first commit.

Why Bruno

The real reason isn't "Bruno is better than Postman." It's that Postman has drifted toward a cloud-first model that creates concrete problems in a team:

Postman collections live in Postman's cloud, not in your repo. They're not reviewable in a PR, they don't follow branches, and if someone modifies a request without telling anyone, it shows up nowhere in your version history.
Workspace sharing requires an account and a paid Postman organization plan once you go beyond 4 or 5 active developers.
The Postman client weighs 300+ MB (Electron). Anecdotal on its own, but representative of the model: grow the tool to justify the cloud value.
Postman environment variables are stored in Postman's cloud, which creates friction for secrets: either you put them there (and lose control) or you manage them outside (and you're maintaining two separate configurations).

Bruno solves all of this with one architectural decision: collections are text files (the .bru format, a simple DSL) stored in a directory. No account, no cloud, no magic sync. A git clone and everything is there. Request changes show up in diffs, PRs, and git blame. That's the only real reason to migrate.

The concepts in 5 minutes

Bruno has three concepts to understand before opening the interface. Two are similar to Postman, one is meaningfully different.

Collection — a directory of .bru files. When you open Bruno and create a collection, you pick a folder on your disk. That's it. Each request is a .bru file in that folder or a subfolder.

Environment — a named set of variables, defined in the Bruno UI and stored in the collection directory under an environments/ subfolder, one .bru file per environment. These files are version-controlled. They hold non-sensitive variables: base URLs, ports, domain names, feature flags. You can put base_url = http://localhost:8080 in there; you should not put a JWT token or a client certificate.

.env — a standard .env file at the root of the collection, gitignored. It holds secrets: tokens, passwords, paths to certificates. Bruno loads it automatically. In your .bru files, you access environment variables with {{TOKEN}} (which looks first in the active Environment, then in the .env) or with process.env.TOKEN in scripts.

Concept

Version-controlled?

Use for

environments/*.bru

Yes

base_url, ports, domain names

.env

No (gitignored)

Tokens, certificates, passwords

.bru (requests)

Yes

API call definitions

Structure in the monorepo

First decision: where to put the collections in the monorepo. Options include postman/ (legacy name, avoid it), api-collection/ (verbose), api/ (ambiguous if you already have Go packages called api), bruno/ (the tool name — explicit, short, unambiguous).

The bruno/ choice has an added benefit: it's visible in the tree that this is a Bruno collection, not a Go code directory. When someone clones the repo for the first time, they don't have to guess what's in that folder.

Recommended structure for 6 services:

bruno/
├── environments/
│   ├── local.bru
│   ├── dev.bru
│   └── prod.bru
├── .env              # gitignored — tokens, certs
├── .gitignore
├── patient-api/      # Patient API — REST
│   ├── health.bru
│   ├── patients/
│   │   ├── create-patient.bru
│   │   └── get-patient.bru
│   └── ...
├── lab-service/      # Lab Service — gRPC
│   ├── lab-results.bru
│   └── ...
├── doctor-portal/    # REST
│   └── ...
├── backoffice/       # REST
│   └── ...
├── billing-service/  # REST
│   └── ...
└── drug-db-api/      # REST — external API (e.g. OpenFDA)
    └── ...

The .gitignore inside bruno/ contains at minimum:

.env

One service = one subfolder. Inside, you can create as many subfolders as needed to organize requests by resource or feature. Bruno renders the directory tree as-is in its explorer.

Managing environments

Environment files are .bru files with a slightly different syntax from request files. Here's what a realistic local.bru looks like:

vars {
  patient_url: http://localhost:8080
  lab_url: localhost:9090
  doctor_url: http://localhost:8081
  backoffice_url: http://localhost:8082
  billing_url: http://localhost:8083
  drug_db_url: https://api.openfda.gov
  grpc_tls: false
}

vars:secret [
  AUTH_TOKEN
  DRUG_DB_API_KEY
]

The vars:secret block lists the names of secret variables — it declares their existence without their value. The actual values come from the .env. This is what allows Bruno to handle display (masked in the UI) without storing secrets in the versioned file.

The dev.bru file looks like this:

vars {
  patient_url: https://patient-dev.acme-internal.com
  lab_url: lab-dev.acme-internal.com:443
  doctor_url: https://doctor-dev.acme-internal.com
  backoffice_url: https://backoffice-dev.acme-internal.com
  billing_url: https://billing-dev.acme-internal.com
  drug_db_url: https://api.openfda.gov
  grpc_tls: true
}

vars:secret [
  AUTH_TOKEN
  DRUG_DB_API_KEY
]

To switch environments in Bruno: dropdown menu in the top right of the interface, select local, dev, or prod. All requests immediately use the new environment's variables.

Secrets in .env

The .env at the root of bruno/ holds the actual values of secret variables declared in the environments:

AUTH_TOKEN=
DRUG_DB_API_KEY=
CLIENT_CERT_PATH=/home/user/.certs/client.crt
CLIENT_KEY_PATH=/home/user/.certs/client.key

In a .bru request file, you reference these variables with double-brace syntax:

headers {
  Authorization: Bearer {{AUTH_TOKEN}}
  X-DrugDB-Key: {{DRUG_DB_API_KEY}}
}

Bruno resolves {{AUTH_TOKEN}} by looking in order: active Environment variables, then .env. If you've defined AUTH_TOKEN in both, the Environment wins. That's the expected behavior: you can override a secret per environment without touching the .env file.

For client certificates (mutual TLS), Bruno supports configuration in the collection settings. You can reference paths from the .env: {{CLIENT_CERT_PATH}} in the certificate path field.

gRPC with server reflection

Acme's lab service exposes a gRPC service — analysis results stream in as each machine finishes a measurement, without waiting for the full batch to complete. The good news: Bruno supports gRPC natively, and it supports server reflection mode — you don't need the .proto file to discover and call methods. The server responds to a reflection request and Bruno builds the list of available methods.

For reflection to work, your Go service needs to have registered the reflection service. If it hasn't yet, two lines to add in main.go:

import "google.golang.org/grpc/reflection"

// In the function that configures the gRPC server:
reflection.Register(grpcServer)

Here's what a .bru file for a unary gRPC request looks like:

meta {
  name: Get Lab Results
  type: grpc
  seq: 1
}

url: {{lab_url}}

body {
  {
    "patient_id": "{{patient_id}}",
    "test_type": "blood_panel",
    "date_from": "2026-01-01"
  }
}

grpc {
  method: LabResultService/GetResults
  reflect: true
  tls: {{grpc_tls}}
}

The reflect: true field tells Bruno to use server reflection to fetch the schema. tls: {{grpc_tls}} uses the environment variable — false locally, true on dev/prod. The URL is just host:port, no scheme. The patient_id in the body can be injected from a pre-request script or copied from a previous REST response (patient record creation).

For gRPC streaming (server-stream, client-stream, bidirectional), Bruno supports that too but the configuration differs slightly — the official docs cover that case better than any summary can.

What a real .bru file looks like

This is where people coming from Postman are most surprised: requests are plain text files with a simple syntax, not 200-line nested JSON. Here's a complete REST request with bearer auth, JSON body, and custom headers:

meta {
  name: Create Patient Record
  type: http
  seq: 1
}

post {
  url: {{patient_url}}/v1/patients
  body: json
  auth: bearer
}

auth:bearer {
  token: {{AUTH_TOKEN}}
}

headers {
  X-Request-ID: {{$randomUUID}}
  X-Client-Version: 2.1.0
}

body:json {
  {
    "last_name": "Dupont",
    "first_name": "Marie",
    "dob": "1985-03-12",
    "nss": "{{$randomUUID}}"
  }
}

tests {
  test("status is 201", function() {
    expect(res.status).to.equal(201);
  });

  test("has patient id", function() {
    expect(res.body.patient_id).to.be.a("string");
  });
}

A few things worth noting about this format:

seq controls the display order in Bruno's explorer.
{{$randomUUID}} is a built-in dynamic variable — Bruno provides several ($timestamp, $randomInt, etc.).
The tests block uses Chai for assertions — same syntax as Postman if you were using it before v10.
There's no pre-request script in some weird JSON/DSL format — it's plain JavaScript in a script:pre-request { } block.

This file is readable in a diff. If someone changes the body or adds a header, you see it in the PR. That's the fundamental difference from Postman.

Migrating from Postman

Bruno supports import from Postman Collection v2.1.

What migrates cleanly: simple REST requests, headers, JSON/form-data bodies, non-secret environment variables. Bruno generates one .bru file per request, folder structure is preserved.

What doesn't migrate correctly:

Complex pre-request scripts — Postman scripts using pm.sendRequest() or advanced environment manipulation need to be rewritten manually. Bruno's syntax is similar but not identical.
Postman Flows — completely Postman-specific, no equivalent in Bruno.
Monitors and mock servers — Postman cloud features, out of scope for Bruno.
Secret variables — you normally don't export these, so they won't appear in the exported JSON anyway.

Pragmatic recommendation: import one collection, check the result, fix what's broken. Don't bulk-import all 6 services without verifying. Critical requests (auth, resource creation) are worth recreating by hand in 5 minutes rather than importing and leaving with unpredictable behavior. The .bru format is simple enough for that.

Step by step

Download and install Bruno — available at usebruno.com/downloads for macOS, Linux, and Windows. No account required, no cloud, standard installation.
Export from Postman — in Postman, right-click on the collection → Export → Collection v2.1 → save the JSON file locally.
Create the Bruno collection in the repo — in Bruno: File → Open Collection → select the bruno/ folder in the monorepo (or create a new folder if it doesn't exist yet). Bruno opens that folder as the collection root.
Import the Postman collection — in Bruno: File → Import Collection → Postman Collection v2.1 → select the JSON exported in step 2. Bruno creates one .bru file per request in the current folder, preserving the folder structure.
Create the environments — in Bruno: Environments menu (gear icon, top right) → Create Environment → name it local. Add your variables: base_url, ports, etc. Repeat for dev and prod. Bruno creates the corresponding files in bruno/environments/.
Create the .env and .gitignore — create bruno/.env with your secrets (tokens, API keys, certificate paths). Create bruno/.gitignore with at minimum .env in it. Commit the .gitignore, never commit the .env.
Verify and fix — test each imported request one by one. Fix variables if needed ({{variable}} syntax from Postman is compatible with Bruno, but complex pre-request scripts need to be rewritten manually in a script:pre-request { } block using plain JavaScript).
Commit — git add bruno/ → git commit -m "feat: migrate API collections from Postman to Bruno". The .bru files, environments/*.bru, and .gitignore are version-controlled. The .env never is.

Conclusion

The Postman → Bruno migration takes half a day for a monorepo of this size. Half of that time is initial setup (structure, environments, .env). The other half is verifying that imported requests actually work.

What you gain isn't "a better API tool." It's that collections become code: reviewable in PRs, diffable commit by commit, consistent across all developers without manual sync. When someone adds an endpoint or changes a parameter, it's in the same commit as the Go code.

That's enough to justify the migration.

Testing a Go exchange API client: mocks, httptest and testcontainers

Odilon HUGONNOT — Mon, 27 Apr 2026 09:00:03 +0000

The Go service runs locally, tests pass. Then you remember the tests are hitting the real Binance API. The next day, CI fails: rate limit exceeded, unstable network, or the exchange quietly renamed a field in its response. Tests that depend on external APIs are flaky at best, unusable in CI at worst.

The problem is structural: exchanges don't have usable sandbox environments (Binance testnet exists, but with constraints that make automated integration testing difficult), rate limits are aggressive, and some operations like PlaceOrder have real side effects. A different approach is needed.

The problem with external APIs in tests

Three concrete reasons you can't test against real exchange APIs in CI:

Rate limits: Binance caps at 1200 requests/minute on the order book endpoint. Ten developers running tests in parallel, and the first ones get banned.
No reliable sandbox: OKEx and Coinbase Pro have test environments, but their availability isn't guaranteed, the data isn't representative, and authentication is yet another CI problem to solve.
Side effects: a test that places a real order during a demo is the kind of incident you don't forget.

Three approaches to work around this, in order of complexity:

Approach

Use case

Speed

HTTP mock (httptest.NewServer)

Test API response parsing

Very fast

Interface mock (mockgen)

Test business logic in isolation

Very fast

Testcontainers

Integration tests with a real DB

Slow (10-30s)

Defining an Exchange interface

The key insight: define a Go interface that all exchange clients implement. This is the foundation of testability. Without it, the service depends on a concrete type and becomes impossible to mock properly.

// exchange/client.go

type OrderBook struct {
    Bids []Level `json:"bids"`
    Asks []Level `json:"asks"`
}

type Level struct {
    Price decimal.Decimal
    Qty   decimal.Decimal
}

type Order struct {
    Pair     string
    Side     string // "buy" | "sell"
    Quantity decimal.Decimal
    Price    decimal.Decimal
}

type OrderResult struct {
    OrderID   string
    Status    string
    FilledQty decimal.Decimal
}

// ExchangeClient is the interface all exchange clients implement.
// The service depends on this interface, never on concrete types.
type ExchangeClient interface {
    GetOrderBook(ctx context.Context, pair string) (*OrderBook, error)
    PlaceOrder(ctx context.Context, order Order) (*OrderResult, error)
    GetBalance(ctx context.Context) (map[string]decimal.Decimal, error)
}

Each exchange implements the interface. The service receives ExchangeClient as a parameter — not *BinanceClient or *CoinbaseClient:

// exchange/binance.go
type BinanceClient struct {
    baseURL    string
    apiKey     string
    secretKey  string
    httpClient *http.Client
}

func NewBinanceClient(apiKey, secretKey string) *BinanceClient {
    return &BinanceClient{
        baseURL:    "https://api.binance.com",
        apiKey:     apiKey,
        secretKey:  secretKey,
        httpClient: &http.Client{Timeout: 10 * time.Second},
    }
}

// BinanceClient implements ExchangeClient.
func (c *BinanceClient) GetOrderBook(ctx context.Context, pair string) (*OrderBook, error) {
    // real HTTP call to Binance
}

// exchange/coinbase.go
type CoinbaseClient struct { /* ... */ }

func (c *CoinbaseClient) GetOrderBook(ctx context.Context, pair string) (*OrderBook, error) {
    // real HTTP call to Coinbase
}

// aggregator/service.go
type AggregatorService struct {
    exchanges []ExchangeClient
}

func NewAggregatorService(exchanges ...ExchangeClient) *AggregatorService {
    return &AggregatorService{exchanges: exchanges}
}

// In tests, pass mocks. In production, pass real clients.
func main() {
    binance := exchange.NewBinanceClient(os.Getenv("BINANCE_KEY"), os.Getenv("BINANCE_SECRET"))
    coinbase := exchange.NewCoinbaseClient(os.Getenv("COINBASE_KEY"), os.Getenv("COINBASE_SECRET"))

    svc := aggregator.NewAggregatorService(binance, coinbase)
}

Mocks with mockgen

Once the interface is defined, mockgen generates mocks automatically. Don't write them by hand — maintenance becomes a nightmare as soon as the interface evolves.

// exchange/client.go
// Add this directive to generate mocks

//go:generate mockgen -destination=../mocks/mock_exchange.go -package=mocks . ExchangeClient

go generate ./exchange/...

The generated file at mocks/mock_exchange.go provides a ready-to-use MockExchangeClient. Table-driven test for a service that aggregates order books from multiple exchanges:

// aggregator/service_test.go

func TestAggregateOrderBooks(t *testing.T) {
    tests := []struct {
        name        string
        binanceOB   *exchange.OrderBook
        coinbaseOB  *exchange.OrderBook
        binanceErr  error
        coinbaseErr error
        wantBestBid decimal.Decimal
        wantErr     bool
    }{
        {
            name: "normal case — best bid on Coinbase",
            binanceOB: &exchange.OrderBook{
                Bids: []exchange.Level{
                    {Price: decimal.NewFromFloat(65000), Qty: decimal.NewFromFloat(0.5)},
                },
            },
            coinbaseOB: &exchange.OrderBook{
                Bids: []exchange.Level{
                    {Price: decimal.NewFromFloat(65100), Qty: decimal.NewFromFloat(0.3)},
                },
            },
            wantBestBid: decimal.NewFromFloat(65100),
        },
        {
            name:       "exchange down — return an error",
            binanceErr: errors.New("connection refused"),
            wantErr:    true,
        },
        {
            name: "empty order book — skip that exchange",
            binanceOB: &exchange.OrderBook{
                Bids: []exchange.Level{
                    {Price: decimal.NewFromFloat(64900), Qty: decimal.NewFromFloat(1.0)},
                },
            },
            coinbaseOB: &exchange.OrderBook{Bids: []exchange.Level{}},
            wantBestBid: decimal.NewFromFloat(64900),
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            ctrl := gomock.NewController(t)
            defer ctrl.Finish()

            binanceMock := mocks.NewMockExchangeClient(ctrl)
            coinbaseMock := mocks.NewMockExchangeClient(ctrl)

            binanceMock.EXPECT().
                GetOrderBook(gomock.Any(), "BTCUSDT").
                Return(tt.binanceOB, tt.binanceErr)

            if tt.binanceErr == nil {
                coinbaseMock.EXPECT().
                    GetOrderBook(gomock.Any(), "BTCUSDT").
                    Return(tt.coinbaseOB, tt.coinbaseErr)
            }

            svc := aggregator.NewAggregatorService(binanceMock, coinbaseMock)
            result, err := svc.GetBestBid(context.Background(), "BTCUSDT")

            if tt.wantErr {
                require.Error(t, err)
                return
            }
            require.NoError(t, err)
            assert.True(t, tt.wantBestBid.Equal(result),
                "expected %s, got %s", tt.wantBestBid, result)
        })
    }
}

The test verifies the service's behavior, not its implementation. If OKEx is added tomorrow, you add a mock in the tests without touching any existing logic.

HTTP mock server for testing parsing

Interface mocks test business logic, but not the HTTP layer. If Binance changes its response format, the mocks won't catch it — they return Go structs, not JSON.

To test that BinanceClient.GetOrderBook() correctly parses the real API format, use httptest.NewServer:

// exchange/binance_test.go

func TestBinanceOrderBookParsing(t *testing.T) {
    // Real format from Binance API GET /api/v3/depth
    server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        assert.Equal(t, "/api/v3/depth", r.URL.Path)
        assert.Equal(t, "BTCUSDT", r.URL.Query().Get("symbol"))

        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]interface{}{
            "lastUpdateId": 1027024,
            "bids": [][]string{
                {"65000.00000000", "0.50000000"},
                {"64950.00000000", "1.20000000"},
            },
            "asks": [][]string{
                {"65001.00000000", "0.30000000"},
                {"65010.00000000", "0.80000000"},
            },
        })
    }))
    defer server.Close()

    // Point the client at the test server
    client := exchange.NewBinanceClientWithBaseURL(server.URL, "test-key", "test-secret")

    ob, err := client.GetOrderBook(context.Background(), "BTCUSDT")

    require.NoError(t, err)
    require.Len(t, ob.Bids, 2)
    require.Len(t, ob.Asks, 2)

    assert.True(t, decimal.NewFromFloat(65000).Equal(ob.Bids[0].Price))
    assert.True(t, decimal.NewFromFloat(0.5).Equal(ob.Bids[0].Qty))
    assert.True(t, decimal.NewFromFloat(65001).Equal(ob.Asks[0].Price))
}

This test catches real bugs: renamed field, unexpected decimal format, unexpected element order in the array. It also documents exactly what the client expects from the API — useful when Binance ships a breaking change.

For this to work, the constructor needs a configurable baseURL. Good practice anyway — it lets you point to Binance testnet or a proxy in production:

func NewBinanceClientWithBaseURL(baseURL, apiKey, secretKey string) *BinanceClient {
    return &BinanceClient{
        baseURL:    baseURL,
        apiKey:     apiKey,
        secretKey:  secretKey,
        httpClient: &http.Client{Timeout: 10 * time.Second},
    }
}

func NewBinanceClient(apiKey, secretKey string) *BinanceClient {
    return NewBinanceClientWithBaseURL("https://api.binance.com", apiKey, secretKey)
}

Testcontainers for integration tests

Mocks cover business logic and HTTP parsing. One case remains: verifying that trades are correctly persisted to the database. A real PostgreSQL is needed here — DB mocks don't test real SQL queries, constraints, or transactions.

testcontainers-go starts a Docker container from within the test, then tears it down when done. No manual setup, no shared database where developers step on each other:

// repository/trade_test.go
// +build integration

func TestSaveTradeIntegration(t *testing.T) {
    ctx := context.Background()

    pg, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
        ContainerRequest: testcontainers.ContainerRequest{
            Image:        "postgres:16",
            ExposedPorts: []string{"5432/tcp"},
            Env: map[string]string{
                "POSTGRES_PASSWORD": "test",
                "POSTGRES_DB":       "trades_test",
                "POSTGRES_USER":     "test",
            },
            WaitingFor: wait.ForListeningPort("5432/tcp"),
        },
        Started: true,
    })
    require.NoError(t, err)
    defer pg.Terminate(ctx)

    host, _ := pg.Host(ctx)
    port, _ := pg.MappedPort(ctx, "5432/tcp")

    dsn := fmt.Sprintf("postgres://test:test@%s:%s/trades_test?sslmode=disable", host, port.Port())
    db, err := sqlx.ConnectContext(ctx, "pgx", dsn)
    require.NoError(t, err)
    defer db.Close()

    // Apply migrations
    require.NoError(t, runMigrations(db))

    repo := repository.NewTradeRepository(db)

    trade := &model.Trade{
        Exchange:   "binance",
        Pair:       "BTCUSDT",
        Side:       "buy",
        Price:      decimal.NewFromFloat(65000),
        Quantity:   decimal.NewFromFloat(0.1),
        ExecutedAt: time.Now().UTC(),
    }

    err = repo.Save(ctx, trade)
    require.NoError(t, err)
    require.NotEmpty(t, trade.ID)

    // Verify persistence
    saved, err := repo.GetByID(ctx, trade.ID)
    require.NoError(t, err)
    assert.Equal(t, "binance", saved.Exchange)
    assert.True(t, trade.Price.Equal(saved.Price))
}

The // +build integration build tag isolates these tests from unit tests. Run them separately:

# Unit tests only (fast, run on every commit)
go test ./...

# Integration tests (slow, run on main branch or pre-deploy)
go test -tags=integration ./...

Testcontainers takes between 10 and 30 seconds to start the container (depending on whether the image is cached). Acceptable for an integration suite, not for unit tests run 50 times a day.

What not to test

With mocks and httptest, it's tempting to test everything. Three things to avoid:

Testing that the Binance API works. That's not your code. If the Binance API is down, your service should detect the error and propagate it correctly — that's what you test, not Binance's uptime.

Over-mocking. If every call returns exactly what you told it to return, you end up testing the mock, not the service. A test that passes because the mocks are perfectly configured to pass is a false positive. The httptest.NewServer with real JSON is more honest than a mock that returns a Go struct directly.

Testing implementation rather than behavior. A test that verifies GetOrderBook was called exactly twice in a specific order is brittle. What matters: the service returns the best bid. How it gets there is its own business.

Conclusion

The layered structure — Exchange interface → business service → repository — isn't an abstraction for its own sake. It's what makes testing possible without hitting real APIs. The business service doesn't know whether it's talking to Binance or a mock. The test mock doesn't know whether the service actually calls the exchange.

In practice, all three test levels coexist in the project: unit tests with mockgen for fast logic, httptest.NewServer to validate API response parsing, and testcontainers for persistence integration tests. Each has its role. None replaces the others.

The most valuable test is often the HTTP parsing one. Business logic changes slowly. Exchange API formats change without warning.

DDD in Go applied to crypto exchange APIs

Odilon HUGONNOT — Sun, 26 Apr 2026 09:00:04 +0000

You write CQRS. You talk about aggregates. You emit domain events. But where do these concepts come from? Domain-Driven Design. Without understanding DDD, CQRS is just a pattern you copy-paste and hope it holds. With DDD, it becomes a tool you use consciously, for the right reasons.

This article lays the conceptual foundations. The concrete terrain: a Go service consuming APIs from multiple crypto exchanges (Binance, OKEx, Coinbase). Because an Order in trading is something any trader understands — that's exactly the kind of domain where DDD pays off.

Domain first, technology second

DDD's core principle is uncomfortable when you come from a technical background: the business model takes precedence over technology. You don't start with "what database", or "what API does Binance expose". You start with "what is an Order in this system? What is a Position? What happens when an order is partially filled?"

For a crypto trading service, the domain is: orders you place, positions that evolve, market data arriving continuously, a portfolio whose value you monitor. Not "a Binance wrapper". Not "a websocket consumer". Those are implementation details.

The practical difference: if your model is domain-centric, you can replace Binance with OKEx without touching your business logic. If your model is API-centric, you rewrite everything every time an exchange changes an endpoint.

DDD doesn't say "ignore technology". It says "don't let technology contaminate your business model".

Bounded contexts: splitting without cutting to the bone

This is the most practical DDD concept, and the most frequently misunderstood. A bounded context is an explicit boundary inside which a model has a precise, coherent meaning. Outside that boundary, the same word can mean something different — and that's acceptable.

For a crypto trading service, three natural bounded contexts emerge:

Market Data

Prices, order books, tickers, recent trades. This context is read-heavy, near real-time. Binance can push 10 updates per second on BTC/USDT. The model here is simple: immutable snapshots. A Ticker in Market Data has no mutable state — it's just a value at a point in time.

Trading

Orders, executions, positions. This context is command-heavy. An Order here has a full lifecycle: created, submitted, partially filled, cancelled. Business logic is concentrated here: validation rules, exchange rejection handling, fee calculation. This is where your aggregates live.

Portfolio

Balances, P&L, trade history. This context is a read model oriented toward reporting. It consumes events emitted by Trading to maintain a coherent view of the portfolio. Updates can be slightly delayed — a few seconds of lag is acceptable for reporting.

Why separate them? Because they have radically different constraints. Market Data at 10 updates/sec doesn't need the same model as Portfolio reporting aggregating data over 3 months. Putting everything in the same model means making compromises that make everything more complex without any benefit.

Bounded Context

Load

Consistency

Business complexity

Market Data

Very high (websockets)

Eventual OK

Low

Trading

Moderate (commands)

Strong (transactional)

High

Portfolio

Low (read model)

Eventual OK

Low

Aggregates and Value Objects in Go

Two core DDD building blocks that Go implements naturally, without any framework.

A Value Object is immutable. Its identity is defined by its value, not by a reference. Price{Amount: 42000, Currency: "USD"} equals another Price with the same fields. In Go, this translates to a struct without a pointer, passed by value.


// Value Object — immutable, equality by value
type Price struct {
    Amount   decimal.Decimal
    Currency string
}

func (p Price) IsZero() bool {
    return p.Amount.IsZero()
}

func (p Price) Add(other Price) (Price, error) {
    if p.Currency != other.Currency {
        return Price{}, fmt.Errorf("currency mismatch: %s vs %s", p.Currency, other.Currency)
    }
    return Price{Amount: p.Amount.Add(other.Amount), Currency: p.Currency}, nil
}

type TradingPair struct {
    Base  string // "BTC"
    Quote string // "USDT"
}

func (tp TradingPair) String() string {
    return tp.Base + "/" + tp.Quote
}

An Aggregate is a unit of consistency. All modifications go through its methods — never through direct field access. It guarantees that its business invariants are always upheld. And in event sourcing, it emits events to describe what happened.


type OrderStatus int

const (
    StatusPending   OrderStatus = iota
    StatusPartial
    StatusFilled
    StatusCancelled
)

// Aggregate — consistency guaranteed, modification through methods only
type Order struct {
    id           OrderID
    pair         TradingPair
    side         Side // Buy / Sell
    quantity     decimal.Decimal
    filledQty    decimal.Decimal
    status       OrderStatus
    events       []DomainEvent
}

func NewOrder(pair TradingPair, side Side, qty decimal.Decimal) (*Order, error) {
    if qty.IsZero() || qty.IsNegative() {
        return nil, fmt.Errorf("invalid quantity: %s", qty)
    }
    o := &Order{
        id:       NewOrderID(),
        pair:     pair,
        side:     side,
        quantity: qty,
        status:   StatusPending,
    }
    o.events = append(o.events, OrderCreated{
        OrderID: o.id,
        Pair:    pair,
        Side:    side,
        Qty:     qty,
    })
    return o, nil
}

func (o *Order) Fill(qty decimal.Decimal, price Price) error {
    if o.status != StatusPending && o.status != StatusPartial {
        return ErrOrderNotFillable
    }
    if qty.GreaterThan(o.quantity.Sub(o.filledQty)) {
        return fmt.Errorf("fill qty %s exceeds remaining %s", qty, o.quantity.Sub(o.filledQty))
    }

    o.filledQty = o.filledQty.Add(qty)
    if o.filledQty.Equal(o.quantity) {
        o.status = StatusFilled
    } else {
        o.status = StatusPartial
    }

    o.events = append(o.events, OrderFilled{
        OrderID: o.id,
        Qty:     qty,
        Price:   price,
    })
    return nil
}

func (o *Order) Cancel() error {
    if o.status == StatusFilled || o.status == StatusCancelled {
        return ErrOrderAlreadyTerminal
    }
    o.status = StatusCancelled
    o.events = append(o.events, OrderCancelled{OrderID: o.id})
    return nil
}

// Events are retrieved then purged after persistence
func (o *Order) PopEvents() []DomainEvent {
    events := o.events
    o.events = nil
    return events
}

Notice what's absent: no database access, no HTTP calls, no external dependencies. The aggregate is pure Go. It expresses business rules and generates events — that's it. Persistence is someone else's problem.

Anti-Corruption Layer: isolating external APIs

Binance returns prices as strings: "4.00000000". OKEx uses a different format. Coinbase yet another. If you let these external formats leak into your domain, you have a problem: your model becomes dependent on the whims of three different exchanges.

The Anti-Corruption Layer (ACL) is the translation boundary. Outside: the exchange's model. Inside: your domain. The ACL translates, and your domain never hears about Binance.


// External model — what Binance actually sends
type binanceOrderBook struct {
    LastUpdateID int64      `json:"lastUpdateId"`
    Bids         [][]string `json:"bids"` // [["price", "qty"], ...]
    Asks         [][]string `json:"asks"`
}

// ACL: translates Binance model to domain
func (c *BinanceClient) GetOrderBook(ctx context.Context, pair TradingPair) (*domain.OrderBook, error) {
    raw, err := c.fetchRaw(ctx, pair.String())
    if err != nil {
        return nil, fmt.Errorf("binance order book %s: %w", pair, err)
    }
    return translateBinanceOrderBook(raw)
}

func translateBinanceOrderBook(raw *binanceOrderBook) (*domain.OrderBook, error) {
    bids := make([]domain.PriceLevel, 0, len(raw.Bids))
    for _, b := range raw.Bids {
        if len(b) != 2 {
            return nil, fmt.Errorf("unexpected bid format: %v", b)
        }
        price, err := decimal.NewFromString(b[0])
        if err != nil {
            return nil, fmt.Errorf("invalid bid price %q: %w", b[0], err)
        }
        qty, err := decimal.NewFromString(b[1])
        if err != nil {
            return nil, fmt.Errorf("invalid bid qty %q: %w", b[1], err)
        }
        bids = append(bids, domain.PriceLevel{
            Price:    domain.Price{Amount: price, Currency: "USDT"},
            Quantity: qty,
        })
    }
    // same for asks...
    return &domain.OrderBook{Bids: bids}, nil
}

And for OKEx, the same interface, a different translation:


// External OKEx model — completely different format
type okexOrderBook struct {
    Data []struct {
        Asks      [][]string `json:"asks"` // [["price", "qty", "liquidated", "count"], ...]
        Bids      [][]string `json:"bids"`
        Timestamp string     `json:"ts"`
    } `json:"data"`
}

func (c *OKExClient) GetOrderBook(ctx context.Context, pair TradingPair) (*domain.OrderBook, error) {
    raw, err := c.fetchRaw(ctx, pair)
    if err != nil {
        return nil, fmt.Errorf("okex order book %s: %w", pair, err)
    }
    return translateOKExOrderBook(raw)
}

Your trading code has no idea who's providing the order book. It calls an OrderBookProvider interface and receives a domain.OrderBook. If Binance changes its API from v3 to v4 with a new response format, only translateBinanceOrderBook changes. Zero impact on business logic.


// Interface defined by the domain — not by the exchanges
type OrderBookProvider interface {
    GetOrderBook(ctx context.Context, pair TradingPair) (*OrderBook, error)
}

// The trading service has no dependency on Binance, OKEx or Coinbase
type TradingService struct {
    orderBook  OrderBookProvider
    orderRepo  OrderRepository
}

DDD and CQRS: how they fit together

If you've read the articles on CQRS aggregates and command handlers, you've already seen DDD in action without it being explicitly named.

DDD gives you the model: aggregates, Value Objects, domain events, bounded contexts. CQRS is the architectural pattern that exploits that model. The relationship is direct:

A Command expresses a business intent (PlaceOrder, CancelOrder)
The command handler loads the aggregate, calls a business method
The aggregate validates the invariant and emits a DomainEvent
The event is persisted, then propagated to read models (Portfolio, history)


// Command — business intent
type PlaceOrderCommand struct {
    Pair     TradingPair
    Side     Side
    Quantity decimal.Decimal
}

// Handler — orchestrates without business logic
func (h *PlaceOrderHandler) Handle(ctx context.Context, cmd PlaceOrderCommand) error {
    order, err := domain.NewOrder(cmd.Pair, cmd.Side, cmd.Quantity)
    if err != nil {
        return fmt.Errorf("invalid order: %w", err)
    }

    // Submit to exchange via ACL
    exchangeID, err := h.exchange.SubmitOrder(ctx, order)
    if err != nil {
        return fmt.Errorf("exchange submission: %w", err)
    }
    order.SetExchangeID(exchangeID)

    // Persist the aggregate and its events
    return h.orderRepo.Save(ctx, order)
}

Without DDD, the command handler accumulates business logic. With DDD, it orchestrates: load, call, persist. The logic belongs to the aggregate.

What DDD doesn't solve

DDD adds complexity. Abstraction layers, interfaces everywhere, translations between models. On a simple CRUD service — user list, a few REST endpoints, no complex business rules — it's clear over-engineering.

For a crypto trading engine with business rules spanning multiple exchanges, invariants to maintain, divergent external formats, and logic that evolves with the market — it pays off. The cost of abstraction is offset by the ability to change one piece without bringing down the others.

The most honest test: if a domain expert — a trader — can read your code and recognize their business concepts, DDD is working. If your code talks about BinanceWebsocketMessageParser and RestAPIResponseDTO, you've let technology invade the domain.

Conclusion

DDD is not an architecture, it's a modeling philosophy. It says: code should speak the language of the people who understand the problem, not the language of the people who understand the technology. In a crypto trading service, that means Order, Position, TradingPair — not BinanceResponseWrapper.

Bounded contexts provide the structure. Aggregates provide consistency. Anti-Corruption Layers provide isolation. And when you add CQRS on top, you have a system where each part has a clear responsibility and explicit boundaries.

Next time you write a command handler, ask yourself: does this logic belong in the handler or in the aggregate? If the answer is "in the aggregate", you're doing DDD.

Advanced PostgreSQL: JSONB, partial indexes and partitioning

Odilon HUGONNOT — Sat, 25 Apr 2026 09:00:03 +0000

A trades table with 50 million rows. The query "which active trades for this user this month" takes 4 seconds. After applying the four techniques in this article: 12 ms.

This isn't magic — it's a solid understanding of what PostgreSQL actually does under the hood. I already covered slow query diagnosis with EXPLAIN ANALYZE basics. Here we go further: the four levers that make a real difference on high-volume production tables.

JSONB: storing flexible schemas without sacrificing performance

Every crypto exchange exposes a different API. Binance returns a clientOrderId, Kraken a userref, Coinbase a client_order_id nested inside a sub-object. If you create a dedicated column for each field from each exchange, your table ends up with 40 columns where 35 are NULL for every single row.

JSONB solves this problem — provided you understand when to use it.

`json` vs `jsonb` — always use `jsonb`

json stores the raw text as-is. jsonb decomposes it into a binary representation at insert time. The consequences:

jsonb is indexable (GIN, expression indexes)
jsonb supports containment operators (@>, ?)
jsonb is slightly slower to write, but significantly faster to read

Simple rule: always use jsonb.

Essential operators

-- Access a key (returns jsonb)
SELECT exchange_meta -> 'exchange' FROM trades;

-- Access a key (returns text)
SELECT exchange_meta ->> 'exchange' FROM trades;

-- Containment: does the JSON contain this sub-object?
SELECT * FROM trades WHERE exchange_meta @> '{"exchange": "binance"}';

-- Key existence
SELECT * FROM trades WHERE exchange_meta ? 'client_order_id';

Adding the column and GIN index

-- Store exchange-specific metadata without extra columns
ALTER TABLE trades ADD COLUMN exchange_meta JSONB;

-- GIN index for containment queries (@> operator)
CREATE INDEX idx_trades_exchange_meta ON trades USING GIN (exchange_meta);

-- Query: all trades with a specific exchange order ID
SELECT * FROM trades WHERE exchange_meta @> '{"order_id": "12345"}';

-- Query: all Binance trades with status filled
SELECT * FROM trades WHERE exchange_meta @> '{"exchange": "binance", "status": "filled"}';

The GIN index on exchange_meta automatically covers all keys in your documents. No need to anticipate upfront which keys you'll query against.

When not to use JSONB

JSONB is not an excuse to skip data modeling. If you regularly filter on a field — user_id, status, created_at — that field must be a real column with a proper B-tree index. Putting those fields inside JSON forfeits all planner optimizations.

Rule: JSONB for variable data that is rarely filtered directly. Typed columns for everything that is frequently filtered, sorted, or joined.

Partial indexes: only index what matters

This is the most underused PostgreSQL feature. A partial index only covers rows that satisfy a WHERE clause. Its size can be 100x smaller than a full index — and it fits entirely in RAM.

The problem with full indexes

-- Full index on status: indexes ALL 50 MILLION rows,
-- including the 49.9 million closed trades nobody queries
CREATE INDEX idx_trades_status ON trades(status);

-- Approximate size
SELECT pg_size_pretty(pg_relation_size('idx_trades_status'));
-- → 1 GB

The solution: partial index on active rows only

-- Partial index: only the ~100,000 active trades
CREATE INDEX idx_trades_active ON trades(user_id, created_at)
WHERE status = 'active';

-- Actual size
SELECT pg_size_pretty(pg_relation_size('idx_trades_active'));
-- → 3 MB  ← fits entirely in shared memory

-- This query uses the partial index directly
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
ORDER BY created_at DESC;

PostgreSQL knows that a query with WHERE status = 'active' can use this index — the index predicate is a subset of the query condition. The planner selects the partial index automatically.

Other common use cases

-- Unverified emails (minority of users)
CREATE INDEX idx_users_unverified ON users(created_at)
WHERE verified = false;

-- Pending jobs in a queue
CREATE INDEX idx_jobs_pending ON jobs(priority, created_at)
WHERE status = 'pending';

-- Open orders
CREATE INDEX idx_orders_open ON orders(user_id, amount)
WHERE closed_at IS NULL;

The principle is always the same: identify the subset of rows your hot queries target, and index only that subset.

EXPLAIN ANALYZE: reading between the lines

"Seq Scan bad, Index Scan good" — that's the naive reading. The reality is more nuanced. Here's how to read an execution plan correctly.

The full command

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
ORDER BY created_at DESC
LIMIT 20;

ANALYZE actually executes the query and measures real timings. BUFFERS reports cache and disk activity. Never run this on a DELETE or UPDATE without wrapping it in a transaction.

Decoding the plan

Limit  (cost=0.56..18.42 rows=20 width=112) (actual time=0.124..0.198 rows=20 loops=1)
  ->  Index Only Scan Backward using idx_trades_active on trades
        (cost=0.56..2891.33 rows=3201 width=112)
        (actual time=0.121..0.183 rows=20 loops=1)
      Index Cond: (user_id = 123)
      Heap Fetches: 0
      Buffers: shared hit=5 read=0
Planning Time: 0.312 ms
Execution Time: 0.221 ms

The key nodes:

Index Only Scan — the best case. PostgreSQL reads the index without touching the heap (the physical table). Heap Fetches: 0 confirms zero table access. This is possible because all needed columns are covered by the index.
rows=3201 estimated vs rows=20 actual — a moderate discrepancy, acceptable. A 100x divergence means stale statistics; run ANALYZE trades;.
Buffers: shared hit=5 read=0 — everything served from memory cache, zero disk I/O. If read is high, your working set doesn't fit in RAM.

Pattern to watch for: the expensive Sort node

Sort  (cost=15234.12..15489.23 rows=102044 width=112)
      (actual time=1823.44..2104.12 rows=102044 loops=1)
  Sort Key: created_at DESC
  Sort Method: external merge  Disk: 8432kB

external merge Disk means the sort spilled to disk — work_mem is insufficient, or the index should include the sort column. Fix: include created_at in the index so rows come out pre-sorted.

-- Before: index without sort order
CREATE INDEX idx_trades_user ON trades(user_id) WHERE status = 'active';

-- After: composite index with native sort direction
CREATE INDEX idx_trades_user_date ON trades(user_id, created_at DESC)
WHERE status = 'active';
-- → The Sort node disappears from the plan

Date partitioning: the solution for massive tables

Beyond 10 million rows on an append-only table (trades, logs, events), date-range partitioning becomes necessary. The idea: physically split the table into sub-tables by time period. A query for "this month" only scans one partition instead of the whole thing.

Creating the partitioned table

-- Parent table — defines the structure and partition key
CREATE TABLE trades (
    id          BIGSERIAL,
    user_id     BIGINT       NOT NULL,
    pair        TEXT         NOT NULL,
    amount      NUMERIC(20, 8) NOT NULL,
    status      TEXT         NOT NULL DEFAULT 'active',
    exchange_meta JSONB,
    created_at  TIMESTAMPTZ  NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (created_at);

-- Monthly partitions
CREATE TABLE trades_2026_01 PARTITION OF trades
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE trades_2026_02 PARTITION OF trades
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

CREATE TABLE trades_2026_03 PARTITION OF trades
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

Partition pruning in action

EXPLAIN SELECT * FROM trades
WHERE created_at >= '2026-03-01'
  AND created_at < '2026-04-01'
  AND user_id = 123;

Append  (cost=0.29..45.23 rows=18 width=156)
  Partitions: trades_2026_03
  ->  Index Scan using trades_2026_03_user_id_idx on trades_2026_03
        (cost=0.29..45.23 rows=18 width=156)
      Index Cond: (user_id = 123)

Partitions: trades_2026_03 — PostgreSQL eliminated all other partitions at planning time. It physically accesses only March 2026.

Operational benefits

Instant archiving: DROP TABLE trades_2024_01 removes a full month of data without a massive DELETE or table lock.
Targeted VACUUM: autovacuum runs partition by partition. Frozen old partitions are never reprocessed.
Local indexes: indexes created on the parent table are automatically created on each partition — each one smaller, faster to build, and more likely to fit in memory.

-- Creating an index applies to all existing and future partitions
CREATE INDEX ON trades(user_id, created_at DESC) WHERE status = 'active';

-- Archive January 2024 without locking the table
DROP TABLE trades_2024_01;

-- Or detach first, then archive
ALTER TABLE trades DETACH PARTITION trades_2024_01;

Combining all four

Here is the final state of the table after applying all four techniques:

-- 1. Table partitioned by month
CREATE TABLE trades (
    id            BIGSERIAL,
    user_id       BIGINT         NOT NULL,
    pair          TEXT           NOT NULL,
    amount        NUMERIC(20, 8) NOT NULL,
    status        TEXT           NOT NULL DEFAULT 'active',
    exchange_meta JSONB,                          -- 2. JSONB for variable metadata
    created_at    TIMESTAMPTZ    NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (created_at);

-- Monthly partitions (automate with pg_partman in production)
CREATE TABLE trades_2026_03 PARTITION OF trades
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

-- 3. Partial index: only active trades, composite with sort direction
CREATE INDEX idx_trades_active ON trades(user_id, created_at DESC)
WHERE status = 'active';

-- 4. GIN index for containment queries on exchange_meta
CREATE INDEX idx_trades_exchange_meta ON trades USING GIN (exchange_meta);

The query that used to take 4 seconds:

-- Before: Seq Scan over 50M rows, 4 seconds
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
  AND created_at >= date_trunc('month', NOW())
ORDER BY created_at DESC
LIMIT 20;

-- After: partition pruning + composite partial index
-- Execution Time: 12 ms

Index Only Scan Backward using idx_trades_active on trades_2026_03
  Index Cond: (user_id = 123)
  Filter: (created_at >= date_trunc('month', now()))
  Heap Fetches: 0
  Buffers: shared hit=4 read=0

The gain comes from cumulative effect: partitioning narrows the scope to a single monthly partition, the partial index only covers active trades within that partition, and the index column order eliminates the sort. PostgreSQL touches only the 4 memory pages it strictly needs.

Conclusion

The logical optimization order for a high-volume table:

Partial indexes first: immediate gain, zero schema change, spectacular impact on hot queries targeting a data subset.
JSONB for variable data: when external API responses have unpredictable fields, JSONB with a GIN index avoids a proliferation of NULL columns. It does not replace typed columns for frequently filtered fields.
Partitioning when the table is truly large: from 10M rows on append-only data. The upfront investment is higher (migration, partition automation, pg_partman), but the long-term operational benefits are substantial.

And at every step, EXPLAIN (ANALYZE, BUFFERS) to verify the planner is doing what you expect. Row estimates and the hit/read ratio don't lie.

PostgreSQL: debugging a slow query and optimizing it

Odilon HUGONNOT — Fri, 24 Apr 2026 09:00:03 +0000

It's Friday at 6:30 PM. Your query takes 4 seconds to respond in prod. The client sent you a screenshot of their loading screen. Your phone is ringing. There's a cold beer waiting for you. That's the context in which you're going to debug a slow PostgreSQL query.

Good news: 80% of PostgreSQL performance problems have an identifiable cause in under 10 minutes with the right tools. This guide follows the logical order of a real investigation — not an exhaustive PG feature catalog, but a method that works.

EXPLAIN vs EXPLAIN ANALYZE: one predicts, the other executes

The classic first mistake: using EXPLAIN alone and wondering why the numbers don't match what you observe in prod.

EXPLAIN displays the execution plan that PostgreSQL thinks it will use, based on the statistics it has in memory. It doesn't execute the query. Costs are estimates, rows are predictions.

EXPLAIN ANALYZE actually executes the query and enriches the plan with measured values: real time, actual rows processed, number of loops. That's what you want. Warning: it actually runs the query — don't run it on a DELETE or UPDATE without a BEGIN/ROLLBACK.

-- Safe for write queries
BEGIN;
EXPLAIN ANALYZE
    UPDATE orders SET status = 'processed' WHERE created_at < NOW() - INTERVAL '30 days';
ROLLBACK;

-- For a SELECT, no special precautions needed
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
    SELECT u.email, COUNT(o.id) AS order_count
    FROM users u
    JOIN orders o ON o.user_id = u.id
    WHERE u.created_at >= '2025-01-01'
    GROUP BY u.email
    ORDER BY order_count DESC
    LIMIT 20;

The BUFFERS option is valuable: it shows how many blocks were read from cache (shared hit) vs from disk (read). A query with lots of disk reads on a table that should be cached is a strong signal.

What to look for in the output

Three things to spot immediately:

Seq Scan on a large table: PostgreSQL scans every row one by one. On a 50,000-row table, this is often acceptable. On 10 million, it's not.
Large gap between rows estimated and rows actual: if PostgreSQL expected to filter 50 rows and processed 80,000, its statistics are stale or the query is poorly written.
Node with disproportionate time: the plan is a tree. The slowest node is the culprit. The displayed time is cumulative — subtract the time of child nodes to isolate a node's own cost.

-- Example of problematic EXPLAIN ANALYZE output
Seq Scan on orders  (cost=0.00..48520.00 rows=2000 width=120)
                    (actual time=0.042..3841.223 rows=1847291 loops=1)
  Filter: (status = 'pending' AND created_at > '2024-01-01')
  Rows Removed by Filter: 152709
Planning Time: 1.2 ms
Execution Time: 4102.8 ms

-- What we see:
-- 1. Seq Scan on "orders" → no index used
-- 2. rows estimated = 2,000, rows actual = 1,847,291 → catastrophic statistics
-- 3. 4 seconds → matches exactly what the client is seeing

The usual suspects

1. Missing index on a filtered column

The most common case. An orders table with 2 million rows, a query filtering on user_id, no index. PostgreSQL does a full Seq Scan. The fix takes 30 seconds.

-- The slow query
SELECT id, total, status
FROM orders
WHERE user_id = 42
ORDER BY created_at DESC;

-- EXPLAIN ANALYZE shows:
-- Seq Scan on orders  (actual time=0.031..2341.5 rows=47 loops=1)
-- Filter: (user_id = 42)
-- Rows Removed by Filter: 1999953

-- The fix
CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders(user_id);
-- CONCURRENTLY: creates the index without locking the table for writes

-- After creation, the same query:
-- Index Scan using idx_orders_user_id on orders
-- (actual time=0.041..0.189 rows=47 loops=1)
-- 2.3 seconds → 0.2 ms

2. Index exists but isn't used

A trickier situation: the index exists, but PostgreSQL doesn't use it. Three main reasons.

Function in the WHERE clause: the moment you apply a function to an indexed column, the index is unusable — PostgreSQL can't scan the index on the transformed value.

-- Index on email, but unused
CREATE INDEX idx_users_email ON users(email);

-- The index does nothing here
SELECT * FROM users WHERE LOWER(email) = 'alice@example.com';

-- Expression index — solves the problem
CREATE INDEX idx_users_email_lower ON users(LOWER(email));

-- Or rewrite the query if data is already lowercase
SELECT * FROM users WHERE email = 'alice@example.com';

LIKE with leading wildcard: LIKE '%dupont' can't use a B-tree index, because it requires scanning all values to find those that end with "dupont". LIKE 'dupont%', on the other hand, can.

-- Guaranteed Seq Scan, even with an index on last_name
SELECT * FROM customers WHERE last_name LIKE '%martin';

-- Uses the B-tree index
SELECT * FROM customers WHERE last_name LIKE 'martin%';

-- For "contains" searches, use pg_trgm + GIN
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX idx_customers_last_name_trgm
    ON customers USING GIN(last_name gin_trgm_ops);

-- Now this LIKE uses the index
SELECT * FROM customers WHERE last_name LIKE '%martin%';

Poor cardinality: if a column has very few distinct values (e.g. status with 3 possible values on 2 million rows), PostgreSQL may estimate that a Seq Scan is faster than an Index Scan followed by 600,000 random lookups. It's often right. The solution: a partial index.

-- 2 million orders, 95% have status = 'completed', 5% status = 'pending'
-- A simple index on status is not very useful for 'completed'
-- But very useful for 'pending' (100,000 rows out of 2 million)

-- Partial index: only indexes pending rows
CREATE INDEX idx_orders_pending
    ON orders(created_at)
    WHERE status = 'pending';

-- This query now uses the partial index
SELECT id, user_id, total
FROM orders
WHERE status = 'pending'
  AND created_at > NOW() - INTERVAL '7 days'
ORDER BY created_at DESC;

3. The N+1 problem via ORM

Not specific to PostgreSQL, but PostgreSQL suffers from it as much as any other database. The ORM loads 100 articles, then for each article makes a separate query to load the author. Result: 101 queries instead of one.

-- What the ORM generates (N+1)
SELECT * FROM articles LIMIT 100;
SELECT * FROM users WHERE id = 1;
SELECT * FROM users WHERE id = 2;
-- ... 98 more times

-- What it should do
SELECT a.id, a.title, a.body, a.published_at,
       u.id AS author_id, u.name AS author_name, u.avatar_url
FROM articles a
JOIN users u ON u.id = a.author_id
ORDER BY a.published_at DESC
LIMIT 100;

Detection happens in the logs or via pg_stat_statements (see below): dozens of identical queries with just one parameter changing, executed in rapid succession. In Go: plain SQL with a single JOIN. In Laravel/Django: eager loading.

4. Stale statistics

PostgreSQL maintains statistics on data distribution in each table (via autovacuum). If these statistics are outdated — after a large import, a bulk DELETE — the planner makes bad decisions. Symptom: large gap between rows estimated and rows actual in EXPLAIN ANALYZE.

-- Update statistics without blocking writes
ANALYZE orders;

-- After a bulk load or data purge
VACUUM ANALYZE orders;

-- Check statistics freshness
SELECT schemaname, tablename, last_analyze, last_autoanalyze, n_live_tup, n_dead_tup
FROM pg_stat_user_tables
WHERE tablename = 'orders';

5. Too many rows returned

Sometimes the query is correctly indexed but returns 50,000 rows when you're displaying 20. Add LIMIT and paginate properly. For sorted lists on large volumes, offset-based pagination itself becomes a problem beyond a few thousand pages: OFFSET 100000 LIMIT 20 forces PG to scan 100,020 rows to return 20.

-- Offset pagination — slow on large pages
SELECT id, title, published_at
FROM articles
ORDER BY published_at DESC
LIMIT 20 OFFSET 10000;

-- Cursor pagination (keyset pagination) — fast regardless of page number
SELECT id, title, published_at
FROM articles
WHERE published_at < '2024-06-15 10:23:00'  -- last value from the previous page
ORDER BY published_at DESC
LIMIT 20;

Choosing the right index type

PostgreSQL offers several index types. In practice, you use three of them.

B-tree (default)

Covers 95% of cases: equality, inequality, sorting, range queries. It's the default when you write CREATE INDEX without specifying a type. Effective on high-cardinality columns (identifiers, emails, dates, amounts).

CREATE INDEX idx_orders_created_at ON orders(created_at);
CREATE INDEX idx_users_email ON users(email);

-- Composite index: column order matters.
-- Useful for queries that filter on user_id AND sort on created_at.
CREATE INDEX idx_orders_user_created ON orders(user_id, created_at DESC);

GIN — Full-text search and JSONB

For searches in text (tsvector), arrays, or JSONB columns. A B-tree index on a JSONB field doesn't help for queries like WHERE metadata @> '{"role": "admin"}'.

-- GIN index for JSONB queries
CREATE INDEX idx_users_metadata ON users USING GIN(metadata);

-- Now efficient
SELECT * FROM users WHERE metadata @> '{"role": "admin", "active": true}';

-- GIN index for full-text search
ALTER TABLE articles ADD COLUMN search_vector tsvector;
UPDATE articles SET search_vector =
    to_tsvector('english', COALESCE(title, '') || ' ' || COALESCE(body, ''));

CREATE INDEX idx_articles_search ON articles USING GIN(search_vector);

SELECT title FROM articles
WHERE search_vector @@ plainto_tsquery('english', 'postgresql optimization index');

BRIN — Sequential timestamps

BRIN (Block Range INdex) is tiny in size and highly efficient for columns whose values grow naturally with insertion: timestamps, sequential IDs. The principle: instead of indexing each value, it stores min/max values per data block. On a log table of several hundred GB, it can drastically reduce query time with an index of just a few MB.

-- Log table with 500 million rows.
-- A B-tree index on created_at would weigh ~15 GB.
-- A BRIN index weighs a few MB.
CREATE INDEX idx_logs_created_at_brin ON access_logs USING BRIN(created_at);

-- Efficient for time range queries
SELECT COUNT(*), path
FROM access_logs
WHERE created_at >= '2026-02-01' AND created_at < '2026-03-01'
GROUP BY path
ORDER BY count DESC;

pg_stat_statements: finding expensive queries in production

Up to now we knew which query to analyze. In production, you often don't. pg_stat_statements accumulates statistics on all executed queries: total time, call count, mean time. This is where you find queries that run 10,000 times per hour and take 200ms each — less visible than a 4-second query, but far more impactful in aggregate.

-- Enable the extension (requires a PostgreSQL restart)
-- In postgresql.conf: shared_preload_libraries = 'pg_stat_statements'
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- The 10 most expensive queries by total time
SELECT
    LEFT(query, 100) AS query_short,
    calls,
    ROUND(total_exec_time::numeric, 2) AS total_ms,
    ROUND(mean_exec_time::numeric, 2) AS mean_ms,
    ROUND(stddev_exec_time::numeric, 2) AS stddev_ms,
    rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

-- Queries with the highest standard deviation (time instability)
-- Possible symptom: different plan depending on parameters, or random cache miss
SELECT
    LEFT(query, 100) AS query_short,
    calls,
    ROUND(mean_exec_time::numeric, 2) AS mean_ms,
    ROUND(stddev_exec_time::numeric, 2) AS stddev_ms
FROM pg_stat_statements
WHERE calls > 100
ORDER BY stddev_exec_time DESC
LIMIT 10;

-- Reset stats (useful after a deployment or optimization)
SELECT pg_stat_statements_reset();

pg_stat_statements doesn't interrupt service and has no measurable performance impact. There's no reason not to enable it in production on all your PostgreSQL clusters.

Quick tips

Force PostgreSQL to show the plan with an index

Sometimes PostgreSQL chooses a Seq Scan when an index exists, and you want to see what the plan would look like with the index. SET enable_seqscan = off forces the planner to prefer index scans for the current session. Useful for diagnosis, not as a permanent fix.

-- In your session only — no global change
SET enable_seqscan = off;

EXPLAIN ANALYZE
    SELECT * FROM orders WHERE user_id = 42;

-- Restore to normal
SET enable_seqscan = on;

-- Interpretation:
-- If the plan with index is faster → create the index or run ANALYZE (stale stats).
-- If the plan with index is slower → PostgreSQL was right to do the Seq Scan.
--   Perhaps the query returns too many rows for an index to help.

work_mem for sorts and hash joins

When you see Sort Method: external merge Disk in EXPLAIN ANALYZE, PostgreSQL ran out of memory and had to write to disk for sorting. Increasing work_mem for the session can fix the problem immediately.

-- Default work_mem = 4MB — insufficient for large aggregations
SET work_mem = '64MB';

EXPLAIN ANALYZE
    SELECT user_id, SUM(total) AS revenue
    FROM orders
    WHERE created_at >= '2025-01-01'
    GROUP BY user_id
    ORDER BY revenue DESC;

-- With sufficient work_mem:
-- Sort Method: quicksort  Memory: 12kB  ← all in RAM, fast

-- Without sufficient work_mem:
-- Sort Method: external merge  Disk: 48MB  ← writes to disk, slow

-- Warning: don't increase work_mem globally without calculation.
-- It applies PER sort operation AND per concurrent connection.
-- 100 connections × 5 sorts × 64MB = potentially 32 GB consumed.
-- Reserve it for sessions that need it, or adjust max_connections.

VACUUM ANALYZE after a bulk load

-- After an import or bulk DELETE
VACUUM ANALYZE orders;

-- VACUUM reclaims space from dead tuples (after UPDATE/DELETE)
-- ANALYZE updates planner statistics
-- Both together: essential after any large data modification

-- Check table bloat
SELECT n_dead_tup, n_live_tup,
       ROUND(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 1) AS dead_pct
FROM pg_stat_user_tables
WHERE tablename = 'orders';
-- If dead_pct > 10-20%, a VACUUM is needed

The method in summary

When a PostgreSQL query is slow, here's the investigation order that resolves the vast majority of cases:

EXPLAIN (ANALYZE, BUFFERS) on the offending query. Identify the slowest node. Look for Seq Scan on a large table, gap between rows estimated / actual.
Missing index? Create it with CONCURRENTLY in production. Verify the query is written to benefit from it — no function on the indexed column, no LIKE '%...'.
Stale statistics? VACUUM ANALYZE on the affected table.
In production with no specific query to target? pg_stat_statements to identify offenders by total time or call count.

80% of PostgreSQL performance problems are solved with EXPLAIN ANALYZE and a well-placed index. The remaining 20% involve configuration tuning (shared_buffers, work_mem, max_connections), rewriting complex queries, or table partitioning — but that's a topic for another article.

The beer can wait another 10 minutes. The client can't.

📄 Associated CLAUDE.md

View • Download • Catalog

CQRS in Go — Part 4: PostgreSQL as an event store

Odilon HUGONNOT — Thu, 23 Apr 2026 09:00:02 +0000

CQRS in Go series:

Part 1: the aggregate, Transition() and Clone()
Part 2: command handlers without side effects
Part 3: sagas and event choreography
Part 4: PostgreSQL as an event store

The first three parts laid the groundwork: immutable aggregates, pure command handlers, sagas through choreography. One central question remains — where do we persist the events? This part answers that question using PostgreSQL as an append-only event store, with no additional dependencies.

Why PostgreSQL and not EventStoreDB or Kafka

EventStoreDB is an excellent tool, designed specifically for event sourcing. But it's an additional component to deploy, monitor, back up, and maintain. For a team already running PostgreSQL in production, adding EventStoreDB means doubling the operational footprint for a database.

Kafka is often mentioned in this context. It's a message bus, not an event store. Reading by aggregate ID is not its strong suit — Kafka is optimized for sequential partition consumption, not for "give me all events for order CMD-4521 in order". Default retention is time-limited. Reconstituting an aggregate from Kafka requires non-trivial work.

PostgreSQL, you already have it. Native ACID, JSONB with indexing, backups integrated into your existing infrastructure, monitoring your team knows, pg_dump, replication, point-in-time recovery. For 90% of projects, an append-only table in PostgreSQL is more than enough. Moving to a dedicated tool makes sense when you're processing millions of events per second — a threshold most projects never reach.

The es_events table — the core

Everything rests on a single table:

CREATE TABLE es_events (
    id            BIGSERIAL PRIMARY KEY,
    aggregate_id  UUID NOT NULL,
    aggregate_type VARCHAR(64) NOT NULL,
    event_type    VARCHAR(128) NOT NULL,
    event_data    JSONB NOT NULL,
    metadata      JSONB DEFAULT '{}',
    version       INT NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    UNIQUE (aggregate_id, version)
);

CREATE INDEX idx_es_events_aggregate ON es_events (aggregate_id, version);

Each column has a specific role:

id (BIGSERIAL): global position in the stream. This is the cursor used by projectors to track where they are. Monotonically increasing, never reused.
aggregate_id: the identifier of the entity involved — the UUID of the order, payment, or shipment. This is the primary read key for reconstituting an aggregate.
aggregate_type: the type of the aggregate — "order", "payment", "shipping". Allows filtering events by domain and avoids UUID collisions between different types.
event_type: the name of the event — "OrderPlaced", "PaymentConfirmed", "ShipmentDispatched". Used to deserialize event_data into the correct Go type.
event_data (JSONB): the serialized event payload. JSONB enables indexing and field-level queries when needed, without sacrificing flexibility.
metadata (JSONB): traceability information — who triggered the action (user ID), correlation ID, causation ID, client timestamp. Not domain data, but indispensable in production.
version: sequence number per aggregate. The first event of an aggregate is at version 1, the second at version 2, and so on.

The UNIQUE (aggregate_id, version) constraint is the centerpiece of concurrent safety. It makes it impossible to insert two events with the same version on the same aggregate. This is the optimistic locking mechanism — no read lock, a constraint violation on write in case of conflict.

Append — writing an event

type EventStore struct {
    db *sqlx.DB
}

func (s *EventStore) Append(ctx context.Context, aggregateID uuid.UUID, aggregateType string, expectedVersion int, events []StorableEvent) error {
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback()

    for i, event := range events {
        version := expectedVersion + i + 1

        data, err := json.Marshal(event.Data)
        if err != nil {
            return fmt.Errorf("marshal event: %w", err)
        }

        _, err = tx.ExecContext(ctx, `
            INSERT INTO es_events (aggregate_id, aggregate_type, event_type, event_data, metadata, version)
            VALUES ($1, $2, $3, $4, $5, $6)
        `, aggregateID, aggregateType, event.Type, data, event.Metadata, version)

        if err != nil {
            // UNIQUE violation = version conflict = optimistic locking
            if isUniqueViolation(err) {
                return ErrConcurrencyConflict
            }
            return fmt.Errorf("insert event: %w", err)
        }
    }

    return tx.Commit()
}

The expectedVersion parameter is the number of the last known event at the time the aggregate was loaded. If the handler produces two events, they will be inserted at versions expectedVersion + 1 and expectedVersion + 2.

If another goroutine wrote an event between loading and writing, the UNIQUE constraint kicks in — PostgreSQL rejects the insert with a constraint violation, translated into ErrConcurrencyConflict. The client can retry: it will reload the aggregate with the new event, recalculate the state, and re-execute the handler. This is optimistic locking — no read lock, just a check at write time.

Load — reloading an aggregate

func (s *EventStore) Load(ctx context.Context, aggregateID uuid.UUID) ([]StoredEvent, error) {
    var events []StoredEvent

    err := s.db.SelectContext(ctx, &events, `
        SELECT id, aggregate_id, aggregate_type, event_type, event_data, metadata, version, created_at
        FROM es_events
        WHERE aggregate_id = $1
        ORDER BY version ASC
    `, aggregateID)

    if err != nil {
        return nil, fmt.Errorf("loading events: %w", err)
    }

    return events, nil
}

The aggregate's state is never stored directly. It's always recalculated by replaying events in order, calling Transition() on each one (see Part 1). This is the replay. The result is deterministic — the same events always produce the same state. This is what makes the system testable and auditable.

Subscriptions — feeding projections

Projectors — the components that maintain read views — must be notified of new events. The simplest solution: a positions table.

CREATE TABLE es_subscriptions (
    subscriber_id  VARCHAR(128) PRIMARY KEY,
    last_event_id  BIGINT NOT NULL DEFAULT 0,
    updated_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Each projector records the identifier of the last event it processed. On startup or after a crash, it resumes from this position. The polling pattern:

type Subscriber struct {
    db           *sqlx.DB
    subscriberID string
    handler      EventHandler
}

func (s *Subscriber) Poll(ctx context.Context) error {
    // 1. Read the last processed position
    var lastID int64
    s.db.GetContext(ctx, &lastID,
        "SELECT last_event_id FROM es_subscriptions WHERE subscriber_id = $1",
        s.subscriberID,
    )

    // 2. Load new events
    var events []StoredEvent
    s.db.SelectContext(ctx, &events,
        "SELECT * FROM es_events WHERE id > $1 ORDER BY id ASC LIMIT 100",
        lastID,
    )

    // 3. Process within a transaction
    for _, event := range events {
        tx, _ := s.db.BeginTxx(ctx, nil)

        if err := s.handler.Handle(ctx, tx, event); err != nil {
            tx.Rollback()
            return fmt.Errorf("handling event %d: %w", event.ID, err)
        }

        // 4. Update the position in the same transaction
        tx.ExecContext(ctx,
            "UPDATE es_subscriptions SET last_event_id = $1, updated_at = NOW() WHERE subscriber_id = $2",
            event.ID, s.subscriberID,
        )

        tx.Commit()
    }

    return nil
}

The critical detail is in point 4: the position update and event processing are in the same transaction. If the process crashes between processing and updating the position, the event will be reprocessed on the next poll. This is at-least-once delivery — handlers must be idempotent to absorb duplicates without side effects.

Idempotency in practice: a handler that inserts a row into a read view can use INSERT ... ON CONFLICT DO NOTHING, or check whether the row already exists. The deduplication key is generally the event_id or a combination of unique business fields.

Real-time notifications with LISTEN/NOTIFY

Polling introduces latency — at best a few hundred milliseconds if the poll runs frequently, at worst several seconds. For read views that need to be reactive, PostgreSQL offers a native mechanism: LISTEN/NOTIFY.

// Event store side: notify after each INSERT
func (s *EventStore) Append(ctx context.Context, aggregateID uuid.UUID, aggregateType string, expectedVersion int, events []StorableEvent) error {
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback()

    // ... insert events ...

    // Notify subscribers
    if _, err := tx.ExecContext(ctx, "NOTIFY es_events_channel"); err != nil {
        return fmt.Errorf("notify: %w", err)
    }

    return tx.Commit()
}

// Subscriber side: listen instead of polling
func (s *Subscriber) Listen(ctx context.Context) error {
    conn, err := s.db.Conn(ctx)
    if err != nil {
        return fmt.Errorf("acquire conn: %w", err)
    }
    defer conn.Close()

    if _, err := conn.ExecContext(ctx, "LISTEN es_events_channel"); err != nil {
        return fmt.Errorf("listen: %w", err)
    }

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
        }

        // Wait for notification (blocking with timeout)
        _, err := conn.WaitForNotification(ctx)
        if err != nil {
            return fmt.Errorf("wait notification: %w", err)
        }

        // Poll immediately after the notification
        if err := s.Poll(ctx); err != nil {
            return err
        }
    }
}

The notification is sent inside the Append transaction — it's only delivered to listeners once the transaction commits. No false alarms on rollback.

In production, combine both approaches: LISTEN for real-time reactivity under normal conditions, periodic polling (every 5 to 30 seconds) as a safety net in case of reconnection or missed notification. The dedicated LISTEN connection must not be taken from the standard connection pool — it occupies a connection permanently.

The command gateway — putting it all together

With the event store in place, the complete flow for a command becomes explicit:

type CommandGateway struct {
    store *EventStore
}

func (gw *CommandGateway) Execute(ctx context.Context, aggregateID uuid.UUID, cmd Command, handler CommandHandler) error {
    // 1. Load events
    storedEvents, err := gw.store.Load(ctx, aggregateID)
    if err != nil {
        return fmt.Errorf("load aggregate: %w", err)
    }

    // 2. Reconstruct state
    state := Replay(storedEvents)
    currentVersion := len(storedEvents)

    // 3. Execute the handler (pure, no side effects)
    newEvents, err := handler.Handle(ctx, state, cmd)
    if err != nil {
        return err
    }

    // 4. Persist the new events
    return gw.store.Append(ctx, aggregateID, cmd.AggregateType(), currentVersion, newEvents)
}

If two commands arrive simultaneously on the same aggregate, the second fails with ErrConcurrencyConflict. The client can retry — it will reload the aggregate including the events from the first command, recalculate the state, and re-execute the handler on the updated state. In practice, real conflicts are rare on well-bounded aggregates; most concurrent operations touch different aggregates.

This is also the pattern that makes command handlers testable without infrastructure (Part 2): the gateway injects the state, the handler produces events, the gateway persists. Each responsibility is isolated.

Performance — when events accumulate

An aggregate with a few dozen events reloads in microseconds. An aggregate with 10,000 events — a long-lived order, a very active user account — starts to weigh on the replay. The solution is snapshots.

CREATE TABLE es_snapshots (
    aggregate_id   UUID PRIMARY KEY,
    aggregate_type VARCHAR(64) NOT NULL,
    version        INT NOT NULL,
    state_data     JSONB NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

A snapshot is a captured, serialized state at a given point in time, associated with its version. Loading with a snapshot proceeds in two steps:

Load the most recent snapshot for the aggregate (if it exists)
Load only the events after the snapshot's version
Replay the partial events on the snapshot's state

The simplest practical rule: create a snapshot every 100 events. Most aggregates never reach this limit — a typical e-commerce aggregate generates 5 to 20 events over its lifetime. Snapshots are only necessary for high-volume aggregates: accounts, wallets, inventories updated hundreds of times per day.

Important: snapshots are an optimization, not a change to the data model. Events remain the source of truth. A corrupted or deleted snapshot is not data loss — the aggregate can be reconstituted from the beginning of its events.

Series summary

Across four parts, we built a complete CQRS implementation in Go:

Part 1 — The aggregate: immutable state, Transition() for applying events, Clone() for dry-runs. The foundation that makes everything else testable.
Part 2 — Command handlers: pure functions that take a state and a command, return events or an error. No database, no HTTP, testable in complete isolation.
Part 3 — Sagas and choreography: coordination between aggregates through published events. Each service reacts to what interests it, without direct coupling. Sagas handle compensation in case of distributed failure.
Part 4 — The event store: PostgreSQL as the persistence infrastructure. Append-only table, optimistic locking through UNIQUE constraint, subscriptions for projectors, LISTEN/NOTIFY for reactivity, snapshots for large-scale performance.

These four patterns combine: the command gateway loads via the event store, calls the pure handler, persists the produced events. Subscribers read the global stream and maintain read views. Sagas listen for certain event types and trigger commands on other aggregates.

The result is a system that is auditable by construction — the complete history is in the events —, testable at every layer, and operable with the PostgreSQL infrastructure you already manage.

📄 Associated CLAUDE.md

View • Download • Catalogue

CQRS in Go — Part 3: sagas and event choreography

Odilon HUGONNOT — Wed, 22 Apr 2026 09:00:03 +0000

CQRS in Go series:

Part 1: the aggregate, Transition() and Clone()
Part 2: command handlers without side effects
Part 3: sagas and event choreography
Part 4: PostgreSQL as an event store

The problem — two aggregates need to cooperate

Concrete scenario: a payment is validated. The Payment aggregate emits a PaymentConfirmed event. In response, a shipment needs to be created in the Shipping aggregate.

The problem: the two aggregates don't know each other, and that's intentional. An aggregate that directly calls another aggregate is a fundamental violation of the pattern. The aggregate is an isolated unit of consistency. If Payment knows about Shipping, you're recreating tight coupling — exactly what you were trying to avoid by leaving the classic transactional model.

The question is: how do you make Shipping react to something that happened in Payment, without one knowing the other? There are two families of answers: orchestration and choreography.

Orchestration vs choreography

Orchestration sets up a central conductor — a SagaManager — that knows all the steps, all transitions, and all possible rollbacks. When PaymentConfirmed arrives, the SagaManager explicitly calls the Shipping service, then the Email service, then the Stock service. The flow is readable in one place. In case of error, the manager knows exactly what to compensate.

The downside: the SagaManager knows everyone. Adding a new step to the flow forces you to modify this central component. It becomes a tight coupling point between domains that should ignore each other.

Choreography works differently. Each aggregate emits events. Independent handlers listen to these events and trigger commands on other aggregates. Nobody centralizes knowledge of the flow — it emerges from the sum of reactions.

Choreography wins in most cases because it scales better, adding new behavior on an existing event doesn't touch existing code, and each handler can evolve, be redeployed, or even fail without blocking the others. The price to pay: the flow is not readable in one place — we'll come back to that in section 6.

The event handler — the link between aggregates

The central tool of choreography is the event handler. It listens to events from one aggregate and triggers commands on another. Its responsibility is narrow: translate an event into a command. Nothing more.

// PaymentEventHandler reacts to Payment events to trigger Shipping.
type PaymentEventHandler struct {
    shippingCmd ShippingCommandService
}

func (h PaymentEventHandler) Handle(ctx context.Context, event DomainEvent) error {
    switch e := event.(type) {
    case PaymentConfirmed:
        return h.shippingCmd.CreateShipment(ctx, CreateShipmentCmd{
            OrderID:    e.OrderID,
            CustomerID: e.CustomerID,
            Items:      e.Items,
            Address:    e.ShippingAddress,
        })
    }
    return nil
}

The event handler contains no business logic. It doesn't decide whether the shipment should be created — that decision was already made by the fact that PaymentConfirmed was emitted. It doesn't validate data — that's the Shipping command handler's job. It translates, nothing more.

If the logic in an event handler starts to grow — conditions, branches, multiple calls — it's a signal that an aggregate is missing somewhere, or that a responsibility hasn't been properly carved out.

Saga idempotency

An event can be redelivered. This happens whenever a handler successfully processes an event but crashes before acknowledging the message. The broker (Kafka, RabbitMQ, or even a simple queue table in PostgreSQL) doesn't know the processing succeeded — it redelivers. If the saga isn't idempotent, you create two shipments for a single order.

The simplest solution: check before acting. If a shipment already exists for this OrderID, treat it as a duplicate and ignore it.

func (h PaymentEventHandler) Handle(ctx context.Context, event DomainEvent) error {
    switch e := event.(type) {
    case PaymentConfirmed:
        err := h.shippingCmd.CreateShipment(ctx, CreateShipmentCmd{
            OrderID:    e.OrderID,
            CustomerID: e.CustomerID,
            Items:      e.Items,
            Address:    e.ShippingAddress,
        })
        // If the shipment already exists, it's a duplicate — ignore it.
        if errors.Is(err, ErrShipmentAlreadyExists) {
            return nil
        }
        return err
    }
    return nil
}

This ErrShipmentAlreadyExists error must be a sentinel error defined in the Shipping package, and the CreateShipment command handler must check uniqueness on OrderID before inserting — ideally with a unique constraint in the database to make the check atomic.

The other approach is an idempotency key: pass the event's ID as an idempotency key to the command handler, which stores it and silently rejects any duplicate with the same key. This is more generic but requires a dedicated tracking table. Both approaches are valid; the choice often depends on whether the command handler is already instrumented for idempotency keys (see Part 2 of the series).

Partial failures

An event can trigger multiple independent reactions: create the shipment, send the confirmation email, decrement stock. If all these reactions are in a single handler and the email fails, the shipment is created but the stock is never updated. The handler failed midway and the final state is inconsistent.

// Bad: one big handler doing everything sequentially.
func (h BigHandler) Handle(ctx context.Context, event DomainEvent) error {
    switch e := event.(type) {
    case PaymentConfirmed:
        if err := h.shipping.Create(ctx, toShipmentCmd(e)); err != nil {
            return err // email and stock will never be processed
        }
        if err := h.email.Send(ctx, toConfirmationEmail(e)); err != nil {
            return err // stock will never be processed
        }
        if err := h.stock.Update(ctx, toStockCmd(e)); err != nil {
            return err
        }
    }
    return nil
}

The solution is to never put multiple responsibilities in the same handler. One handler, one action.

// One handler per responsibility. Each can fail independently.

type ShippingOnPayment struct {
    shippingCmd ShippingCommandService
}

func (h ShippingOnPayment) Handle(ctx context.Context, event DomainEvent) error {
    if e, ok := event.(PaymentConfirmed); ok {
        return h.shippingCmd.CreateShipment(ctx, CreateShipmentCmd{
            OrderID: e.OrderID,
            Items:   e.Items,
            Address: e.ShippingAddress,
        })
    }
    return nil
}

type EmailOnPayment struct {
    mailer EmailService
}

func (h EmailOnPayment) Handle(ctx context.Context, event DomainEvent) error {
    if e, ok := event.(PaymentConfirmed); ok {
        return h.mailer.SendConfirmation(ctx, e.CustomerEmail, e.OrderID)
    }
    return nil
}

type StockOnPayment struct {
    stockCmd StockCommandService
}

func (h StockOnPayment) Handle(ctx context.Context, event DomainEvent) error {
    if e, ok := event.(PaymentConfirmed); ok {
        return h.stockCmd.DecrementStock(ctx, DecrementStockCmd{
            Items: e.Items,
        })
    }
    return nil
}

These three handlers are registered separately on the PaymentConfirmed event. If the email fails, it will be retried independently — without blocking the shipment or the stock update. A failure in one branch doesn't poison the others.

Retry with exponential backoff is handled by the infrastructure (broker, worker), not the handler. The handler has one responsibility: try to perform its action and return an error if it fails. The system decides how many times to retry.

Eventual consistency is an accepted consequence. After a crash, the stock will be updated on the next retry — in a few seconds or minutes depending on the retry policy. For most e-commerce use cases, this is acceptable. If it's not acceptable, you need a distributed transaction, and you're leaving the scope of choreography.

Tracing the flow — the event/handler table

In choreography, the flow is not readable in any single file. It's distributed across dozens of handlers. Without documentation, it's impossible to know what gets triggered when PaymentConfirmed is emitted. That's the cost of choreography.

The practical answer is a mapping table, maintained manually or generated from handler registrations. For each event, list the handlers and the action they perform. This is the system's map.

Event

Handler

Action

PaymentConfirmed

ShippingOnPayment

Creates the shipment

PaymentConfirmed

EmailOnPayment

Order confirmation email

PaymentConfirmed

StockOnPayment

Decrements stock

ShipmentCreated

EmailOnShipment

"Your package has shipped" email

ShipmentCreated

TrackingOnShipment

Creates the delivery tracking

This table should live in the project documentation, not in a comment buried in some file. It answers the question "what happens when this event is emitted?" — a question you ask frequently in production when something goes wrong.

In Go, you can also centralize handler registrations in a wire.go or handlers.go file per domain, which makes the mapping readable in code:

func registerPaymentHandlers(bus EventBus, deps Dependencies) {
    bus.Register(PaymentConfirmed{}, ShippingOnPayment{shippingCmd: deps.ShippingCmd})
    bus.Register(PaymentConfirmed{}, EmailOnPayment{mailer: deps.Mailer})
    bus.Register(PaymentConfirmed{}, StockOnPayment{stockCmd: deps.StockCmd})
}

All handlers tied to PaymentConfirmed are visible in one function. No need to grep the entire codebase to find what reacts to this event.

When choreography isn't enough

Choreography has one limit: flows with compensation. Compensation = undoing an already-completed step because a subsequent step failed. The classic example: booking a hotel, a flight, and a rental car for a trip. If the flight is unavailable, you need to cancel the hotel reservation that already succeeded.

In pure choreography, this scenario forces you to emit a FlightBookingFailed event and have a handler that cancels the hotel. This works, but the handler needs to know whether the hotel was booked for this trip — which implies shared state. You're rebuilding an orchestrator, but implicitly and scattered across the codebase.

In this case, a process manager is justified. A process manager is an event handler with its own state. It listens to events from different steps, maintains progress state, and decides on compensations if needed.

type BookingProcessManager struct {
    bookingID uuid.UUID
    hotelOK   bool
    flightOK  bool
    carOK     bool
    hotelCmd  HotelCommandService
}

func (pm *BookingProcessManager) Handle(ctx context.Context, event DomainEvent) error {
    switch e := event.(type) {
    case HotelReserved:
        pm.hotelOK = true
        // Trigger the flight booking
        return pm.flightCmd.BookFlight(ctx, BookFlightCmd{
            BookingID: pm.bookingID,
            FlightRef: e.FlightRef,
        })

    case FlightBooked:
        pm.flightOK = true
        // Trigger the car reservation
        return pm.carCmd.ReserveCar(ctx, ReserveCarCmd{
            BookingID: pm.bookingID,
        })

    case FlightBookingFailed:
        // Compensation: cancel the hotel if already booked
        if pm.hotelOK {
            return pm.hotelCmd.CancelReservation(ctx, CancelHotelCmd{
                BookingID: pm.bookingID,
            })
        }
    }
    return nil
}

The difference from a global SagaManager: this process manager only knows the travel booking flow. It's not a centralization point for all system flows. One process manager per saga, not one manager for everything.

Its state must be persisted — in a database or the event store — to survive restarts. This is the only place in a choreographed architecture where you explicitly accept having a component with its own state and centralized flow logic. The rest of the system continues to operate through choreography.

Summary

An aggregate cannot call another aggregate directly — choreography through events is the standard solution.
The event handler is a translator: it transforms an event into a command on another aggregate. No business logic inside.
Sagas must be idempotent: an event can be redelivered, the handler must detect and ignore duplicates.
One handler per responsibility: never put multiple independent actions in a single handler. A partial failure must not block other branches.
Eventual consistency is an accepted consequence: a branch that fails will be retried independently, without blocking the others.
The event/handler table is the essential documentation for a choreographed system. Without it, the flow is unreadable.
For flows with compensation, a process manager with its own state is justified — but one per saga, not a global manager.

Part 4 covers the PostgreSQL event store: how to persist events, manage aggregate versioning, and implement replay from the database.

📄 Associated CLAUDE.md

View • Download • Catalogue

CQRS in Go — Part 2: side-effect-free command handlers

Odilon HUGONNOT — Tue, 21 Apr 2026 09:00:02 +0000

CQRS in Go series:

Part 1: the aggregate, Transition() and Clone()
Part 2: command handlers without side effects
Part 3: sagas and event choreography
Part 4: PostgreSQL as an event store

If you followed Part 1, you know what an aggregate is and how Transition() reconstructs state from events. Now, a more concrete question: who decides to emit those events? Who validates that a command is legitimate? That's the role of the command handler.

And that's where many CQRS projects drift. The handler starts clean, then accumulates dependencies. A repo here, an email service there. Three months later, you have a test that spins up a database to verify that a required field is actually required.

There's a better way.

The problem with classic handlers

Here's what a "standard" handler looks like — the one you write when you apply CQRS without pushing the reasoning all the way through:

// ❌ Classic handler — depends on everything
func (h *OrderHandler) PlaceOrder(ctx context.Context, cmd PlaceOrderCmd) error {
    customer, err := h.customerRepo.Get(ctx, cmd.CustomerID)
    if err != nil {
        return err
    }

    order := Order{
        ID:         uuid.New(),
        CustomerID: customer.ID,
        Items:      cmd.Items,
    }

    if err := h.orderRepo.Save(ctx, order); err != nil {
        return err
    }
    if err := h.emailService.Send(customer.Email, "Order placed"); err != nil {
        return err
    }

    return nil
}

To test this function, you need: a mock customerRepo, a mock orderRepo, a mock emailService. Three mocks for a single test. And each test becomes a wiring exercise — you spend more time configuring the environment than testing the business logic.

The worst part: the actual business logic fits in five lines. The rest is I/O. And that's precisely the problem: I/O and business logic are mixed together.

The magic signature

The solution fits in one signature:

Handle(ctx context.Context, state Order, cmd PlaceOrderCmd) (Events, error)

Three important things in this signature:

The handler receives the current state of the aggregate — not a repository, not a DB connection. The state is already reconstructed when the handler is called.
It returns events — not a modified state, not a boolean. Facts: "here's what happened".
Zero I/O, zero side effects. The handler doesn't even know PostgreSQL exists.

Here's the complete implementation:

type PlaceOrderCmd struct {
    OrderID    uuid.UUID
    CustomerID uuid.UUID
    Items      []OrderItem
}

func (cmd PlaceOrderCmd) Validate() error {
    if cmd.OrderID == uuid.Nil {
        return fmt.Errorf("order ID required")
    }
    if len(cmd.Items) == 0 {
        return fmt.Errorf("at least one item required")
    }
    return nil
}

type PlaceOrderHandler struct{}

func (h PlaceOrderHandler) Handle(ctx context.Context, state Order, cmd PlaceOrderCmd) (Events, error) {
    if err := cmd.Validate(); err != nil {
        return nil, err
    }

    // Business invariant: can't place an order that's already placed
    if state.Status == OrderStatusPlaced {
        return nil, ErrOrderAlreadyPlaced
    }

    total := 0
    for _, item := range cmd.Items {
        total += item.UnitPrice * item.Quantity
    }

    var events Events
    events.Append(OrderPlaced{
        OrderID:    cmd.OrderID,
        CustomerID: cmd.CustomerID,
        Items:      cmd.Items,
        Total:      total,
        PlacedAt:   time.Now(),
    })

    return events, nil
}

This handler is a pure function disguised as a method. Same inputs, same output, every time. No internal state. No hidden dependencies. Testable in complete isolation.

Validation in two layers

Look at the structure: validation is split into two distinct levels with different responsibilities.

Layer 1: cmd.Validate() — structural validation. Required fields, correct formats, simple constraints. This validation can be called before even loading the aggregate from the store. No business context needed. It belongs to the command itself.

Layer 2: business invariants in the handler — these validations require the current state. "Can we cancel this order?" depends on the order's current status. Without state, you can't answer.

type CancelOrderCmd struct {
    OrderID uuid.UUID
    Reason  string
}

func (cmd CancelOrderCmd) Validate() error {
    if cmd.Reason == "" {
        return fmt.Errorf("cancellation reason required")
    }
    return nil
}

func (h CancelOrderHandler) Handle(ctx context.Context, state Order, cmd CancelOrderCmd) (Events, error) {
    if err := cmd.Validate(); err != nil {
        return nil, err
    }

    // Business invariants — depend on state
    if state.Status == OrderStatusShipped {
        return nil, fmt.Errorf("cannot cancel shipped order")
    }
    if state.Status == OrderStatusCancelled {
        return nil, fmt.Errorf("order already cancelled")
    }

    var events Events
    events.Append(OrderCancelled{
        OrderID:     cmd.OrderID,
        Reason:      cmd.Reason,
        CancelledAt: time.Now(),
    })

    return events, nil
}

The separation is clean. Structural validation errors surface early, before any loading from the store. Business invariant errors surface later, when context is available.

Tests — 10 lines, no mocks

This is where the design pays off. Here's a complete test for the happy path:

func TestPlaceOrder_Success(t *testing.T) {
    handler := PlaceOrderHandler{}
    state := Order{} // empty state = new order

    cmd := PlaceOrderCmd{
        OrderID:    uuid.New(),
        CustomerID: uuid.New(),
        Items:      []OrderItem{{ProductID: "SKU-1", Quantity: 2, UnitPrice: 1500}},
    }

    events, err := handler.Handle(context.Background(), state, cmd)

    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }
    if len(events) != 1 {
        t.Fatalf("expected 1 event, got %d", len(events))
    }

    placed, ok := events[0].(OrderPlaced)
    if !ok {
        t.Fatalf("expected OrderPlaced, got %T", events[0])
    }
    if placed.Total != 3000 {
        t.Errorf("expected total 3000, got %d", placed.Total)
    }
}

No mocks. No DB. No Docker. No setup/teardown. This test runs in microseconds on any machine.

Business invariants test just as simply — just pass a state with the right status:

func TestPlaceOrder_AlreadyPlaced(t *testing.T) {
    handler := PlaceOrderHandler{}
    state := Order{Status: OrderStatusPlaced} // order already placed

    _, err := handler.Handle(context.Background(), state, PlaceOrderCmd{
        OrderID: uuid.New(),
        Items:   []OrderItem{{ProductID: "SKU-1", Quantity: 1, UnitPrice: 100}},
    })

    if err != ErrOrderAlreadyPlaced {
        t.Fatalf("expected ErrOrderAlreadyPlaced, got %v", err)
    }
}

And table-driven tests are particularly readable because the only variable between cases is the initial state:

func TestCancelOrder(t *testing.T) {
    tests := []struct {
        name    string
        state   Order
        wantErr bool
    }{
        {"draft order", Order{Status: OrderStatusDraft}, false},
        {"placed order", Order{Status: OrderStatusPlaced}, false},
        {"shipped order", Order{Status: OrderStatusShipped}, true},
        {"already cancelled", Order{Status: OrderStatusCancelled}, true},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            _, err := CancelOrderHandler{}.Handle(context.Background(), tt.state, CancelOrderCmd{
                OrderID: uuid.New(),
                Reason:  "changed mind",
            })
            if (err != nil) != tt.wantErr {
                t.Errorf("got err=%v, wantErr=%v", err, tt.wantErr)
            }
        })
    }
}

Four status cases, four lines in the table. The test logic is entirely in the data — which is exactly what table-driven tests aim for.

The Events type and Append()

The Events type is intentionally simple:

type Events []DomainEvent

func (e *Events) Append(event DomainEvent) {
    *e = append(*e, event)
}

It's a slice. No event bus, no dispatcher, no middleware. Just a slice with a helper method so the calling code is a bit more readable than *e = append(*e, event).

Why so simple? Because the handler doesn't need to know what happens next. It emits events. That's it. What happens after — persistence, projections, sagas — is the command gateway's problem.

The full flow — from HTTP to storage

For the complete picture, here's what the command gateway does when it receives a command. The handler sees none of this:

// What the CommandGateway does when it receives a command:
// 1. Load the aggregate's existing events from the store
// 2. Replay events to reconstruct state (via Transition())
// 3. Call handler.Handle(ctx, state, cmd)
// 4. Store the new events in the store
// 5. Notify event handlers (projections, sagas)

func (g *CommandGateway) Dispatch(ctx context.Context, aggregateID uuid.UUID, cmd Command) error {
    // Load and reconstruct the aggregate
    storedEvents, err := g.store.Load(ctx, aggregateID)
    if err != nil {
        return fmt.Errorf("loading aggregate %s: %w", aggregateID, err)
    }

    state := g.aggregate.Rebuild(storedEvents)

    // The handler knows nothing of what came before
    newEvents, err := g.handler.Handle(ctx, state, cmd)
    if err != nil {
        return err
    }

    // Store the new events
    if err := g.store.Append(ctx, aggregateID, newEvents); err != nil {
        return fmt.Errorf("appending events: %w", err)
    }

    // Notify projections and sagas (async)
    g.bus.Publish(newEvents)

    return nil
}

The handler is called at step 3. It receives an already-reconstructed state, returns events, and doesn't know what comes before or after. This deliberate ignorance is what makes it testable.

The details of the PostgreSQL store and event-based reconstruction are covered in Part 4. What matters here: the handler is pure.

Summary

Pure handler: no dependency on repos, DBs, or external services. Receives a state, returns events.
Signature Handle(ctx, state, cmd) (Events, error): that's the contract. The entire architecture follows from it.
Two-layer validation: structural on the command (cmd.Validate()), business in the handler. Each where it has the necessary context.
Tests without mocks: build a state, call the handler, verify the events. No setup, no infrastructure, no Docker.
Events as a slice: type Events []DomainEvent. No bus built into the handler. Routing complexity belongs elsewhere.
The gateway orchestrates: loading from the store, state reconstruction, handler call, event persistence. The handler sees none of this — and that's intentional.

Part 3 covers sagas: how to react to an event to trigger other commands, and how to orchestrate business processes that span multiple aggregates.

📄 Associated CLAUDE.md

View • Download • Catalogue

CQRS in Go — Part 1: the aggregate, Transition() and the Clone() you forget

Odilon HUGONNOT — Mon, 20 Apr 2026 09:00:01 +0000

CQRS in Go series:

Part 1: the aggregate, Transition() and Clone()
Part 2: command handlers without side effects
Part 3: sagas and event choreography
Part 4: PostgreSQL as an event store

What an aggregate is

The aggregate is the guardian of business consistency. It's the one that says "no" when an operation is invalid: you can't cancel an order that's already been shipped, you can't add an item to a closed cart. This logic lives in the aggregate and nowhere else.

In Event Sourcing, the aggregate has no directly persisted state. Its state is the result of replaying all its events from the beginning. Each event represents something that happened — immutable, factual, in the past. The aggregate replays them in order and reconstructs its current state.

What the aggregate does not do: I/O. It doesn't read from the database, doesn't make HTTP requests, doesn't send emails. It receives a command, validates whether it can be applied to its current state, and emits events. That's it. This constraint is what makes it testable at zero cost.

The structure in Go

An Order aggregate in Go looks like this:

type OrderStatus string

const (
    OrderStatusDraft     OrderStatus = "draft"
    OrderStatusPlaced    OrderStatus = "placed"
    OrderStatusShipped   OrderStatus = "shipped"
    OrderStatusCancelled OrderStatus = "cancelled"
)

type OrderItem struct {
    ProductID string
    Quantity  int
    UnitPrice int // in cents
}

type Order struct {
    OrderID    uuid.UUID
    CustomerID uuid.UUID
    Status     OrderStatus
    Items      []OrderItem
    Total      int
    CreatedAt  time.Time
}

Two deliberate choices here. First, value struct, not a pointer. When you pass an Order, Go makes a copy of it. That's exactly what we want: two versions of a state don't share memory by default (except for slices and maps — we'll get to that). An aggregate passed by pointer between multiple goroutines is a source of silent bugs.

Second, fields are exported here to simplify the example, but in production you keep them private to the package and only expose what's needed through methods. The aggregate is not a DTO.

Events as the source of truth

Events are facts that have occurred. They don't describe an intention — that's the role of commands — they describe what happened.

type OrderPlaced struct {
    OrderID    uuid.UUID
    CustomerID uuid.UUID
    Items      []OrderItem
    Total      int
    PlacedAt   time.Time
}

type OrderItemAdded struct {
    OrderID  uuid.UUID
    Item     OrderItem
    NewTotal int
}

type OrderCancelled struct {
    OrderID     uuid.UUID
    Reason      string
    CancelledAt time.Time
}

Once emitted, an event never changes. You don't correct a past event — you emit a new event that compensates. That's the fundamental difference from an UPDATE in a database.

The aggregate declares the event types it handles via EventKinds(). This is useful for the event store which needs to know how to deserialize stored events:

func (o Order) EventKinds() []DomainEvent {
    return []DomainEvent{
        OrderPlaced{},
        OrderItemAdded{},
        OrderCancelled{},
    }
}

Transition() — the heart of the pattern

Transition() is the method that applies an event to the current state and returns a new state. It's the only place where the aggregate's state evolves.

func (o Order) Transition(event DomainEvent) Order {
    switch e := event.(type) {
    case OrderPlaced:
        o.OrderID = e.OrderID
        o.CustomerID = e.CustomerID
        o.Status = OrderStatusPlaced
        o.Items = e.Items
        o.Total = e.Total
        o.CreatedAt = e.PlacedAt

    case OrderItemAdded:
        o.Items = append(o.Items, e.Item)
        o.Total = e.NewTotal

    case OrderCancelled:
        o.Status = OrderStatusCancelled
    }
    return o
}

The receiver is a value, not a pointer. When Go calls this method, it copies the current Order into o. We modify this copy and return it. If you ignore the return value, nothing has changed — the original Order is intact.

It's a pure reducer, exactly like Array.prototype.reduce in JavaScript or fold in Haskell. The current state is a deterministic function of all past events:

state = events.reduce(initialState, transition)

The practical consequence: you can reconstruct the aggregate's state at any historical point in time by replaying events up to that point. To debug a bug that occurred on February 14th at 2:37 PM, replay the events up to that timestamp and inspect the state. No logs to dig through, no snapshots to restore.

The Clone() trap — the silent bug

This is where most CQRS implementations in Go fall apart. And it's silent: no panic, no error, just corrupted data.

The problem comes from Go's memory model for slices. A slice is a three-field structure: a pointer to the backing array, a length, and a capacity. When Go copies a struct containing a slice, it copies these three fields — but not the backing array. Both copies point to the same array in memory.

append is particularly treacherous. If the slice has enough capacity to accommodate the new element, append writes directly into the existing backing array without allocating a new one. Result: the old state is modified without the code knowing it.

package main

import "fmt"

type OrderItem struct {
    ProductID string
    Quantity  int
}

type Order struct {
    Items []OrderItem
}

func main() {
    // Slice with capacity 4, length 1 — enough room for append without reallocation
    original := Order{
        Items: make([]OrderItem, 1, 4),
    }
    original.Items[0] = OrderItem{ProductID: "A", Quantity: 1}

    // Simulate a naive Transition: copy the struct, append to the copy
    modified := original
    modified.Items = append(modified.Items, OrderItem{ProductID: "B", Quantity: 2})

    // The trap: append wrote into the shared backing array.
    // original.Items still points to len=1, but index [1] in memory
    // now contains product "B". Reslice and it reappears.
    fmt.Println("original length:", len(original.Items))  // 1 — reassuring
    fmt.Println("modified length:", len(modified.Items))  // 2

    // But:
    ghost := original.Items[:2] // Reslice within existing capacity
    fmt.Println("ghost item:", ghost[1].ProductID) // "B" — it's there
}

In practice in a CQRS system, this scenario occurs as soon as you store a snapshot or pass a state between goroutines. The old state you thought was immutable has been silently modified.

The solution is Clone(). Before any Transition(), explicitly clone the state to break the link with the backing array:

func (o Order) Clone() Order {
    c := o
    if o.Items != nil {
        c.Items = make([]OrderItem, len(o.Items))
        copy(c.Items, o.Items)
    }
    return c
}

copy allocates a new backing array and copies the elements into it. After that, c.Items and o.Items are completely independent. Any subsequent append on one doesn't touch the other.

The same constraint applies to maps. Go copies the reference, not the content. A Cart with a map requires an explicit deep clone:

type Cart struct {
    Items map[uuid.UUID]CartItem
}

func (c Cart) Clone() Cart {
    clone := c
    clone.Items = make(map[uuid.UUID]CartItem, len(c.Items))
    for k, v := range c.Items {
        clone.Items[k] = v
    }
    return clone
}

If CartItem itself contains slices or maps, you need to go one level deeper. That's the only case where the depth of the clone depends on the actual data structure — there's no safe generic shortcut in Go.

The production rule: always call Clone() before Transition(). You can also integrate it directly into Transition() as the first instruction, which makes the pattern foolproof at the cost of a systematic copy even when it's not needed. For most aggregates, this overhead is negligible.

Replay — reconstructing state from events

With Clone() and Transition() in place, the replay fits in five lines:

func Replay(events []DomainEvent) Order {
    var state Order
    for _, event := range events {
        state = state.Clone().Transition(event)
    }
    return state
}

Start from an empty Order, apply each event in order, and get the final state. Proof that the aggregate is pure: this function is deterministic. Same list of events, same result, every time. No network calls, no cache, no system clock. You can call it from a test without any mocks.

Replay is also what allows you to fix bugs after the fact. If the logic in Transition() was incorrect and you fix it, you can replay all historical events and get corrected states — provided the events themselves are correct, which they are by definition since they represent what actually happened.

Summary

The aggregate is the guardian of business consistency. It does no I/O.
Its state is a pure function of the events it has received since its creation.
Transition(event) applies an event and returns a new state — method on a value, not a pointer.
Go copies structs by value, but slices and maps share their backing store. Without Clone(), mutations via append can silently corrupt old states.
Clone() must deep copy all collections in the struct.
Replay reconstructs any historical state by replaying events — no I/O, therefore trivially testable.

Part 2 covers command handlers: how to validate a command against the aggregate's current state, emit the appropriate events, and keep everything free of side effects and implicit dependencies.

📄 Associated CLAUDE.md

View • Download • Catalogue

Idempotency in CQRS and Event Sourcing — Part 2: commands, projections and outbox

Odilon HUGONNOT — Sun, 19 Apr 2026 09:00:02 +0000

In HTTP, idempotency is straightforward: store the key, return the cache. In a CQRS/Event Sourcing system, it's more subtle. The command may be idempotent, but what about the event it generates? The projection consuming it? Idempotency must cross the entire stack.

If you haven't read part 1, it covers the basics and the HTTP implementation in Go — with a complete PostgreSQL store that replaces Redis. Here we go one level deeper: command store, optimistic locking, idempotent projections, outbox pattern. These are the patterns found in financial systems, e-commerce platforms, anywhere a processing duplicate costs money or trust. Everything runs on PostgreSQL — no Redis, no external broker for idempotency.

The problem specific to Event Sourcing

In Event Sourcing, the application state is the event stream. An account balance is not a column in the database — it's the sum of AccountCredited and AccountDebited events since opening. If the same event is recorded twice, the reconstructed state is wrong. And often irreparable without manual intervention.

Concrete scenario on a payment system:

The command DebitAccount(amount=100, idempotencyKey=abc123) arrives
The event AccountDebited(amount=100) is recorded, balance goes from 200 to 100
Network drops. Client retries. The same command arrives a second time
Without protection: a second AccountDebited(amount=100) is recorded, balance drops to 0
The user has been charged twice for a single purchase

In a classic relational database, you could fix this with an UPDATE. In Event Sourcing, you never modify the past — you can only append a corrective event, which complicates audit and support. Better to prevent the duplicate from getting in at all.

Command idempotency with the idempotency key

The principle is the same as in HTTP: associate each command with a unique key provided by the client, and check before processing whether that key has already been seen. The difference is that you store the result in a dedicated command store, and the save must be atomic with the events.

CREATE TABLE processed_commands (
    idempotency_key UUID PRIMARY KEY,
    aggregate_id    UUID NOT NULL,
    command_type    VARCHAR(100) NOT NULL,
    result_event_id UUID,
    processed_at    TIMESTAMPTZ DEFAULT NOW()
);

The handler follows a three-step schema:

func (h *PaymentHandler) Handle(ctx context.Context, cmd DébiterCompte) error {
    // 1. Check if already processed
    existing, err := h.commandStore.Get(ctx, cmd.IdempotencyKey)
    if err == nil {
        // Already processed — return silent success
        _ = existing
        return nil
    }
    if !errors.Is(err, ErrNotFound) {
        return fmt.Errorf("checking idempotency: %w", err)
    }

    // 2. Process the command
    events, err := h.aggregate.Handle(ctx, cmd)
    if err != nil {
        return err
    }

    // 3. Save events + mark command as processed — ATOMICALLY
    return h.store.SaveEventsAndMarkCommand(ctx, events, cmd.IdempotencyKey)
}

The atomicity of step 3 is non-negotiable. If you save the events in one transaction and mark the command in a second separate transaction, a crash between the two gives you: events in the database, command not marked. On the next retry, the handler doesn't see the key, processes again, records a duplicate event. Both operations must live in the same PostgreSQL transaction.

Optimistic locking — detecting concurrent conflicts

The idempotency key protects against duplicates of the same command. It doesn't protect against two different commands modifying the same aggregate at the same time. That's the role of optimistic locking.

Each aggregate has a version number. When a client reads the state of an aggregate, it receives its current version. When it sends a command, it includes that version. If in the meantime someone else has written to the same aggregate, the versions no longer match and the command is rejected — it's up to the client to re-read the state and decide whether its command still holds.

CREATE TABLE account_events (
    id             UUID PRIMARY KEY,
    aggregate_id   UUID NOT NULL,
    version        INT NOT NULL,
    event_type     VARCHAR(100) NOT NULL,
    payload        JSONB NOT NULL,
    created_at     TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(aggregate_id, version) -- key constraint
);

The UNIQUE(aggregate_id, version) constraint does all the work. If two concurrent commands try to write version 5 of the same aggregate, PostgreSQL lets one through and rejects the other with a uniqueness violation.

type DébiterCompte struct {
    AggregateID     uuid.UUID
    ExpectedVersion int // version the client read
    Montant         float64
    IdempotencyKey  uuid.UUID
}

func (s *EventStore) AppendEvents(
    ctx context.Context,
    aggregateID uuid.UUID,
    expectedVersion int,
    events []Event,
) error {
    for i, event := range events {
        _, err := s.db.ExecContext(ctx,
            `INSERT INTO account_events (id, aggregate_id, version, event_type, payload)
             VALUES ($1, $2, $3, $4, $5)`,
            event.ID,
            aggregateID,
            expectedVersion+i+1,
            event.Type,
            event.Payload,
        )
        if isUniqueViolation(err) {
            return ErrVersionConflict // caller can retry with the new version
        }
        if err != nil {
            return fmt.Errorf("appending event: %w", err)
        }
    }
    return nil
}

By returning ErrVersionConflict, we leave the decision to retry or surface the error to the user to the caller. In a financial system, you often surface an explicit error ("your operation was cancelled because the account was modified in the meantime") rather than retrying silently.

Idempotent projections

Projections consume the event stream to build denormalized views — the current balance of an account, the order list for a customer, etc. With Kafka or any at-least-once system, the same event can arrive multiple times. The projection must produce the same result whether it sees it once or ten times.

Approach 1: tracking position in the stream

Each projection remembers the ID of the last event it processed. Events older than or equal to the checkpoint are ignored.

CREATE TABLE projection_checkpoints (
    projection_name VARCHAR(100) PRIMARY KEY,
    last_event_id   UUID NOT NULL,
    updated_at      TIMESTAMPTZ DEFAULT NOW()
);

This approach works well when events are ordered and IDs can be compared (UUIDs v7 or sequences). It assumes the event store guarantees a stable order, which is generally true per aggregate but may be less so across different aggregates.

Approach 2: idempotent upsert

Rather than filtering duplicates upstream, write in a way that rewriting the same data is harmless. PostgreSQL's INSERT ON CONFLICT DO UPDATE instruction is built for this:

INSERT INTO account_balances (account_id, balance, last_event_id)
VALUES ($1, $2, $3)
ON CONFLICT (account_id) DO UPDATE
    SET balance       = EXCLUDED.balance,
        last_event_id = EXCLUDED.last_event_id
WHERE account_balances.last_event_id < EXCLUDED.last_event_id;
-- The WHERE clause prevents overwriting a more recent state with an older event

The WHERE clause matters. Without it, if an old event arrives after a recent event (which can happen with multiple Kafka partitions), you overwrite a correct state with a stale one. With it, you only update if the incoming event is newer than what you already have.

In practice, both approaches are complementary: the checkpoint avoids replaying thousands of events unnecessarily, and the idempotent upsert acts as a safety net for duplicates that slip through anyway.

The Outbox pattern — publishing events without data loss

Suppose your system saves events in PostgreSQL and publishes them to Kafka for other services. If the process crashes after the PostgreSQL commit but before publishing to Kafka, the event is lost from the perspective of consumers. If you reverse the order, you might publish an event that the transaction will then rollback. It's impossible to do an atomic commit across two independent systems without a distributed coordinator.

The Outbox pattern sidesteps the problem without two-phase commit:

In the same PostgreSQL transaction: save the event and write it to an outbox table
A dedicated worker reads outbox and publishes to Kafka
Once successfully published, mark the row as sent

CREATE TABLE outbox (
    id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_type   VARCHAR(100) NOT NULL,
    payload      JSONB NOT NULL,
    published    BOOLEAN DEFAULT FALSE,
    created_at   TIMESTAMPTZ DEFAULT NOW(),
    published_at TIMESTAMPTZ
);

func (s *Store) SaveEventAndOutbox(ctx context.Context, tx *sqlx.Tx, event Event) error {
    // 1. Save the event in the event store
    if err := s.saveEvent(ctx, tx, event); err != nil {
        return err
    }
    // 2. Write to the outbox — same transaction, same commit or same rollback
    _, err := tx.ExecContext(ctx,
        `INSERT INTO outbox (event_type, payload) VALUES ($1, $2)`,
        event.Type,
        event.Payload,
    )
    return err
}

// Separate worker — runs in the background
func (w *OutboxWorker) Run(ctx context.Context) {
    ticker := time.NewTicker(100 * time.Millisecond)
    defer ticker.Stop()
    for {
        select {
        case <-ticker.C:
            if err := w.publishPending(ctx); err != nil {
                slog.Error("outbox publish failed", "error", err)
            }
        case <-ctx.Done():
            return
        }
    }
}

func (w *OutboxWorker) publishPending(ctx context.Context) error {
    rows, err := w.db.QueryContext(ctx,
        `SELECT id, event_type, payload FROM outbox
         WHERE published = FALSE
         ORDER BY created_at
         LIMIT 100`,
    )
    if err != nil {
        return fmt.Errorf("querying outbox: %w", err)
    }
    defer rows.Close()

    for rows.Next() {
        var id uuid.UUID
        var eventType string
        var payload []byte
        if err := rows.Scan(&id, &eventType, &payload); err != nil {
            return fmt.Errorf("scanning outbox row: %w", err)
        }
        if err := w.kafka.Publish(ctx, eventType, payload); err != nil {
            return fmt.Errorf("publishing event %s: %w", id, err)
        }
        _, err = w.db.ExecContext(ctx,
            `UPDATE outbox SET published = TRUE, published_at = NOW() WHERE id = $1`,
            id,
        )
        if err != nil {
            return fmt.Errorf("marking event published %s: %w", id, err)
        }
    }
    return rows.Err()
}

The worker may publish the same event twice if it crashes after the Kafka publish but before the UPDATE outbox. That's at-least-once delivery, not exactly-once. Kafka consumers must therefore be idempotent — which the upsert approach described in the previous section guarantees.

For high-volume systems, polling every 100ms can be replaced by PostgreSQL LISTEN/NOTIFY to trigger the worker immediately after each outbox insertion, without waiting for the next tick.

Summary — what protects what

Problem

Solution

Command received twice (client retry)

Idempotency key in processed_commands

Two concurrent commands on the same aggregate

Optimistic locking (UNIQUE constraint on version)

Event published twice by the broker

Idempotent projection (checkpoint or upsert)

Crash between save and publish

Outbox pattern (atomicity via transaction)

Each layer protects against a different type of duplicate. The idempotency key doesn't replace optimistic locking, and the outbox doesn't make the projection idempotent — these are orthogonal guarantees that each cover a distinct failure point.

Conclusion

Idempotency in CQRS/Event Sourcing is not a detail you add after the fact — it's what separates a system that works in a demo from one that holds up in production. Retries happen. Processes crash. Kafka re-delivers messages. That's not paranoia, it's the normal behavior of any distributed system under load.

The good news: these four patterns are well-known, well-supported by PostgreSQL, and assemble cleanly in Go. Once in place, they work silently — and that's exactly what you expect from them. The "duplicate payment" ticket that never comes in, that's the real success.

📄 Associated CLAUDE.md

View • Download • Catalogue

Idempotency explained — Part 1: basics, idempotency key and Go implementation

Odilon HUGONNOT — Sat, 18 Apr 2026 09:00:03 +0000

The user clicks "Pay". Nothing happens. The network is slow, the spinner keeps spinning. They click again. This time it goes through. Your API received two identical requests 800ms apart.

Two possible scenarios. Either your customer just got charged twice — and you have a legal problem, a chargeback to handle, and a difficult conversation ahead. Or you thought about idempotency, and the second request is treated as a duplicate: same response, zero additional side effects.

What is idempotency?

The mathematical definition: an operation f is idempotent if f(f(x)) = f(x). Applying it multiple times gives the same result as once.

In practice: pressing the elevator call button five times — the elevator arrives once. That's idempotent. Ordering the same dish five times at a restaurant — you get five plates. That's not idempotent, and neither is your bill.

In HTTP, verbs have precise guarantees:

GET — idempotent. Calling GET /orders/42 a hundred times changes nothing server-side.
PUT — idempotent. Updating a resource with the same data multiple times: same result.
DELETE — idempotent. The first deletion removes the resource; subsequent ones return 404. The final result is the same: the resource no longer exists.
POST — not idempotent by default. Each call to POST /orders creates a new order.

A commonly misunderstood point: idempotent doesn't mean "no side effects". DELETE /users/42 does delete the user — that's a very concrete side effect. But that effect is stable: after the first call, subsequent ones no longer change the system's state. That's idempotency.

Why it matters in practice

Duplicates arrive in three ways in a distributed system. None of them are rare or theoretical.

1. Automatic retries

The client sends a request. The network times out after 30 seconds. The client automatically retries. Except your server had already received the first request — it was just busy processing it. You now have two executions for the same intent.

This is the classic case of a payment SDK that retries three times on network failure. If your POST /payments endpoint is not idempotent, the customer is potentially charged three times.

2. Double-click

The user clicks "Confirm order". The interface gives no immediate feedback. They click again. Two identical requests go out a few hundred milliseconds apart. Two orders created in the database. Customer support is going to love it.

3. At-least-once delivery

Kafka, RabbitMQ, Stripe or GitHub webhooks guarantee message delivery at least once — not exactly once. If the consumer crashes after processing the message but before acknowledging receipt, the broker resends the message. That's a normal protocol guarantee, not a bug. Your consumer must be able to receive the same message twice without creating a duplicate.

The concrete consequences:

Double bank charge — chargeback, critical incident.
Confirmation email sent three times — the customer thinks it's a bug or an attack.
Order created twice — incorrect stock, duplicated delivery, wrong accounting.

The idempotency key — the universal pattern

The canonical solution for making a POST endpoint idempotent. Stripe has been using it for years and makes it a prerequisite for payments in production.

The principle: the client generates a UUID v4 for each intent of an operation and sends it in the Idempotency-Key header. The server stores the result of the first processing. For any subsequent request with the same key, it returns the stored result without reprocessing.

POST /payments
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
Content-Type: application/json

{"amount": 100, "currency": "EUR", "customer_id": "cus_8Rn2xM"}

The complete flow:

First request with the key: normal processing, result stored.
Same key, same payload: stored result returned immediately. Zero reprocessing.
Same key, different payload: 422 error. Client-side inconsistency.
Different key: new intent, normal processing.

The key is generated client-side, not server-side — the client is the one who knows that "this request is a repeat of the one 30 seconds ago". The server can't guess that.

Go implementation — HTTP middleware

package middleware

import (
    "bytes"
    "net/http"
    "sync"
    "time"
)

type CachedResult struct {
    StatusCode int
    Body       []byte
    CreatedAt  time.Time
}

// IdempotencyStore — in-memory map, demo only.
// For production, see the next section: PostgreSQL store.
type IdempotencyStore struct {
    mu      sync.RWMutex
    results map[string]CachedResult
}

func NewIdempotencyStore() *IdempotencyStore {
    return &IdempotencyStore{results: make(map[string]CachedResult)}
}

func (s *IdempotencyStore) Get(key string) (CachedResult, bool) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    result, ok := s.results[key]
    return result, ok
}

func (s *IdempotencyStore) Set(key string, result CachedResult) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.results[key] = result
}

// responseRecorder captures the response for storage.
type responseRecorder struct {
    http.ResponseWriter
    code int
    body bytes.Buffer
}

func (r *responseRecorder) WriteHeader(code int) {
    r.code = code
    r.ResponseWriter.WriteHeader(code)
}

func (r *responseRecorder) Write(b []byte) (int, error) {
    r.body.Write(b)
    return r.ResponseWriter.Write(b)
}

func IdempotencyMiddleware(store *IdempotencyStore) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            if r.Method != http.MethodPost {
                next.ServeHTTP(w, r)
                return
            }

            key := r.Header.Get("Idempotency-Key")
            if key == "" {
                http.Error(w, `{"error": "Idempotency-Key header required"}`, http.StatusBadRequest)
                return
            }

            // Already cached? Return immediately.
            if cached, ok := store.Get(key); ok {
                w.Header().Set("X-Idempotent-Replayed", "true")
                w.WriteHeader(cached.StatusCode)
                w.Write(cached.Body)
                return
            }

            // First pass: capture and store the response.
            rec := &responseRecorder{ResponseWriter: w, code: http.StatusOK}
            next.ServeHTTP(rec, r)

            store.Set(key, CachedResult{
                StatusCode: rec.code,
                Body:       rec.body.Bytes(),
                CreatedAt:  time.Now(),
            })
        })
    }
}

This version has two unacceptable limitations in production:

Multi-instance: each pod has its own map. A request on pod A isn't visible to pod B.
No TTL: keys accumulate in memory until restart.

The usual solution is Redis. But if your stack doesn't include Redis — which is often the case on projects that want to stay simple — PostgreSQL does exactly the same job.

The PostgreSQL store — zero Redis, multi-instance, native TTL

The idea: a dedicated idempotency_keys table that acts as the shared store. All pods read and write to the same database. PostgreSQL handles atomicity itself.

CREATE TABLE idempotency_keys (
    key         UUID PRIMARY KEY,
    status      VARCHAR(16) NOT NULL DEFAULT 'pending', -- pending | completed
    status_code INT,
    body        TEXT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Automatic cleanup of old keys (Stripe keeps 24h)
-- Run periodically via a job or a background goroutine
DELETE FROM idempotency_keys
WHERE created_at < NOW() - INTERVAL '24 hours';

The middleware uses INSERT ... ON CONFLICT DO NOTHING to acquire the lock atomically — exactly like SETNX under Redis, but guaranteed by PostgreSQL.

package middleware

import (
    "bytes"
    "database/sql"
    "fmt"
    "net/http"
    "time"

    "github.com/jmoiron/sqlx"
)

type PgIdempotencyStore struct {
    db *sqlx.DB
}

func NewPgIdempotencyStore(db *sqlx.DB) *PgIdempotencyStore {
    return &PgIdempotencyStore{db: db}
}

// TryAcquire attempts to set the PENDING lock on the key.
// Returns true if this is the first request, false if another is already in progress or completed.
func (s *PgIdempotencyStore) TryAcquire(ctx context.Context, key string) (bool, error) {
    res, err := s.db.ExecContext(ctx,
        `INSERT INTO idempotency_keys (key) VALUES ($1) ON CONFLICT (key) DO NOTHING`,
        key,
    )
    if err != nil {
        return false, fmt.Errorf("acquiring idempotency key: %w", err)
    }
    n, _ := res.RowsAffected()
    return n == 1, nil // true = first pass
}

// GetCompleted retrieves the result of an already-processed key.
func (s *PgIdempotencyStore) GetCompleted(ctx context.Context, key string) (statusCode int, body []byte, found bool, err error) {
    var row struct {
        Status     string `db:"status"`
        StatusCode sql.NullInt32 `db:"status_code"`
        Body       sql.NullString `db:"body"`
    }
    err = s.db.GetContext(ctx, &row,
        `SELECT status, status_code, body FROM idempotency_keys WHERE key = $1`,
        key,
    )
    if err == sql.ErrNoRows {
        return 0, nil, false, nil
    }
    if err != nil {
        return 0, nil, false, fmt.Errorf("getting idempotency key: %w", err)
    }
    if row.Status != "completed" {
        return 0, nil, false, nil // PENDING — still in progress
    }
    return int(row.StatusCode.Int32), []byte(row.Body.String), true, nil
}

// Complete marks the key as finished and stores the response.
func (s *PgIdempotencyStore) Complete(ctx context.Context, key string, statusCode int, body []byte) error {
    _, err := s.db.ExecContext(ctx,
        `UPDATE idempotency_keys SET status = 'completed', status_code = $2, body = $3 WHERE key = $1`,
        key, statusCode, string(body),
    )
    return err
}

// StartCleanup launches a background goroutine that deletes expired keys.
func (s *PgIdempotencyStore) StartCleanup(ctx context.Context, ttl time.Duration) {
    go func() {
        ticker := time.NewTicker(1 * time.Hour)
        defer ticker.Stop()
        for {
            select {
            case <-ticker.C:
                s.db.ExecContext(ctx,
                    `DELETE FROM idempotency_keys WHERE created_at < NOW() - $1::interval`,
                    ttl.String(),
                )
            case <-ctx.Done():
                return
            }
        }
    }()
}

// responseRecorder captures the response for storage.
type responseRecorder struct {
    http.ResponseWriter
    code int
    body bytes.Buffer
}

func (r *responseRecorder) WriteHeader(code int) {
    r.code = code
    r.ResponseWriter.WriteHeader(code)
}

func (r *responseRecorder) Write(b []byte) (int, error) {
    r.body.Write(b)
    return r.ResponseWriter.Write(b)
}

// IdempotencyMiddleware guarantees idempotency of POST requests via PostgreSQL.
func IdempotencyMiddleware(store *PgIdempotencyStore) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            if r.Method != http.MethodPost {
                next.ServeHTTP(w, r)
                return
            }

            key := r.Header.Get("Idempotency-Key")
            if key == "" {
                http.Error(w, `{"error": "Idempotency-Key header required"}`, http.StatusBadRequest)
                return
            }

            // Already processed? Return the cached result.
            if statusCode, body, found, err := store.GetCompleted(r.Context(), key); found && err == nil {
                w.Header().Set("X-Idempotent-Replayed", "true")
                w.WriteHeader(statusCode)
                w.Write(body)
                return
            }

            // Attempt to acquire the PENDING lock.
            acquired, err := store.TryAcquire(r.Context(), key)
            if err != nil {
                http.Error(w, `{"error": "internal error"}`, http.StatusInternalServerError)
                return
            }
            if !acquired {
                // Another request is already processing this key.
                // Simple option: 409 — client should retry in a few seconds.
                http.Error(w, `{"error": "request in progress, retry shortly"}`, http.StatusConflict)
                return
            }

            // First pass: process and store the result.
            rec := &responseRecorder{ResponseWriter: w, code: http.StatusOK}
            next.ServeHTTP(rec, r)

            store.Complete(r.Context(), key, rec.code, rec.body.Bytes())
        })
    }
}

This middleware works in multi-instance deployments without additional configuration. The INSERT ... ON CONFLICT DO NOTHING is atomic: if two pods receive the same key simultaneously, only one gets the lock. The other receives a 409 and the client can retry a few seconds later — at that point the key will be COMPLETED and it will get the cached response.

To start the automatic cleanup at application init:

store := middleware.NewPgIdempotencyStore(db)
store.StartCleanup(ctx, 24*time.Hour) // deletes keys > 24h in background

handler := middleware.IdempotencyMiddleware(store)(myHandler)

The UNIQUE constraint directly on your data

For simple cases where the idempotency key can live directly on the business table (payments, orders), no dedicated table needed. A UNIQUE constraint is enough — it's the most solid layer because it's atomic by construction.

CREATE TABLE payments (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key  UUID UNIQUE NOT NULL,
    customer_id      VARCHAR(64) NOT NULL,
    amount           DECIMAL(10,2) NOT NULL,
    currency         CHAR(3) NOT NULL,
    status           VARCHAR(20) NOT NULL DEFAULT 'pending',
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

INSERT INTO payments (idempotency_key, customer_id, amount, currency)
VALUES ($1, $2, $3, $4)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id, status, created_at;

func CreatePayment(ctx context.Context, db *sqlx.DB, req PaymentRequest) (*Payment, error) {
    var payment Payment

    err := db.GetContext(ctx, &payment, `
        INSERT INTO payments (idempotency_key, customer_id, amount, currency)
        VALUES ($1, $2, $3, $4)
        ON CONFLICT (idempotency_key) DO NOTHING
        RETURNING id, idempotency_key, customer_id, amount, currency, status, created_at
    `, req.IdempotencyKey, req.CustomerID, req.Amount, req.Currency)

    if err == sql.ErrNoRows {
        // Key already known — return the existing payment.
        err = db.GetContext(ctx, &payment,
            "SELECT * FROM payments WHERE idempotency_key = $1",
            req.IdempotencyKey,
        )
        if err != nil {
            return nil, fmt.Errorf("fetching existing payment: %w", err)
        }
        return &payment, nil
    }
    if err != nil {
        return nil, fmt.Errorf("inserting payment: %w", err)
    }

    return &payment, nil
}

Two concurrent requests with the same key: PostgreSQL places a row-level lock during insertion. Only one INSERT succeeds. No race condition, no Redis, no intermediate state to manage.

Summary

Duplicates come from three places: automatic retries, double-click, at-least-once delivery from brokers.
The idempotency key is the universal pattern for POST: the client generates a UUID per intent, the server deduplicates.
The PostgreSQL store replaces Redis: idempotency_keys table, INSERT ON CONFLICT for atomic lock, TTL cleanup goroutine. Multi-instance, zero external dependency.
The UNIQUE constraint in the database is the most solid layer: atomic by construction, no possible race condition.

Part 2 goes further: in a CQRS and Event Sourcing architecture, idempotency touches commands, events and aggregates. Optimistic locking, outbox pattern — the four layers that make a distributed system never create a duplicate, even under load.

📄 Associated CLAUDE.md

View • Download • Catalogue

DEV Community: Odilon HUGONNOT

Testing a production Bash script: 38 tests without a framework

Why no Bash testing framework

The basic structure: assertions and counters

Testing Bash functions: the isolation problem

Mocking external dependencies

A concrete example: testing the Unicode progress bar

The result: 38 tests, 0 external framework

What it changes in practice

Conclusion

Migrating Postman to Bruno in a monorepo: the practical guide

Why Bruno

The concepts in 5 minutes

Structure in the monorepo

Managing environments

Secrets in .env

gRPC with server reflection

What a real .bru file looks like

Migrating from Postman

Step by step

Conclusion

Testing a Go exchange API client: mocks, httptest and testcontainers

The problem with external APIs in tests

Defining an Exchange interface

Mocks with mockgen

HTTP mock server for testing parsing

Testcontainers for integration tests

What not to test

Conclusion

DDD in Go applied to crypto exchange APIs

Domain first, technology second

Bounded contexts: splitting without cutting to the bone

Market Data

Trading

Portfolio

Aggregates and Value Objects in Go

Anti-Corruption Layer: isolating external APIs

DDD and CQRS: how they fit together

What DDD doesn't solve

Conclusion

Advanced PostgreSQL: JSONB, partial indexes and partitioning

JSONB: storing flexible schemas without sacrificing performance

json vs jsonb — always use jsonb

Essential operators

Adding the column and GIN index

When not to use JSONB

Partial indexes: only index what matters

The problem with full indexes

The solution: partial index on active rows only

Other common use cases

EXPLAIN ANALYZE: reading between the lines

The full command

Decoding the plan

Pattern to watch for: the expensive Sort node

Date partitioning: the solution for massive tables

Creating the partitioned table

Partition pruning in action

Operational benefits

Combining all four

Conclusion

PostgreSQL: debugging a slow query and optimizing it

EXPLAIN vs EXPLAIN ANALYZE: one predicts, the other executes

What to look for in the output

The usual suspects

1. Missing index on a filtered column

2. Index exists but isn't used

3. The N+1 problem via ORM

4. Stale statistics

5. Too many rows returned

Choosing the right index type

B-tree (default)

GIN — Full-text search and JSONB

BRIN — Sequential timestamps

pg_stat_statements: finding expensive queries in production

Quick tips

Force PostgreSQL to show the plan with an index

work_mem for sorts and hash joins

VACUUM ANALYZE after a bulk load

The method in summary

CQRS in Go — Part 4: PostgreSQL as an event store

`json` vs `jsonb` — always use `jsonb`