DEV Community: lukman lukman

Database Isolation: Why Two Committed Transfers Can Create Money

lukman lukman — Fri, 02 Oct 2026 07:16:37 +0000

Two wallet transfers can both commit successfully while creating money. In Lab 08, the problem is a specific query pattern: read a balance, validate it in Go, calculate a replacement value, and write that value later. A transaction boundary does not make the original read remain current.

The useful question is whether overlapping transactions preserve the business invariant. This lab approaches that question through a three-account wallet, PostgreSQL isolation experiments, and concurrent tests.

Start with what must remain true

Alice, Bob, and Charlie each begin with 1,000,000. A transfer should change the distribution of those balances while keeping their sum at 3,000,000. Every balance must also remain non-negative.

The schema implements the row-level invariant:

balance BIGINT NOT NULL CHECK (balance >= 0)

That constraint does not enforce conservation of money across accounts. A broken transfer can leave every individual balance positive while increasing their sum. Correctness therefore needs both checks.

TransferNaive uses sql.LevelReadCommitted. It begins a transaction, reads the sender, checks sufficient funds, reads the receiver, updates both accounts, inserts an audit record, and commits. These operations share an atomic boundary. Their decisions can still depend on stale application values.

Follow the overlap instead of the happy path

Transfer A sends 800,000 from Alice to Bob. Transfer B sends 800,000 from Alice to Charlie. Both transactions read Alice at 1,000,000 before either writes. Each concludes that the transfer is allowed.

Each then calculates Alice's replacement balance as 200,000. The naive sender update uses that calculated value rather than deriving the new balance from the current row:

// 4. Overwrite sender balance with calculated value (Lost Update vulnerability)
    _, err = tx.ExecContext(ctx, "UPDATE isolation_accounts SET balance = $1 WHERE id = $2", fromBalance-amount, fromID)
    if err != nil {
        return fmt.Errorf("update from balance: %w", err)
    }

Excerpt from transfer.go; surrounding validation, receiver update, audit, and commit are omitted.

The second write can wait for the first writer and still overwrite Alice with the same 200,000. Bob and Charlie receive separate credits of 800,000. The expected final state in TestNaiveTransfer_LostUpdate is Alice 200,000, Bob 1,800,000, Charlie 1,800,000: total 3,800,000.

The test orchestrates this overlap with channels. Both transactions signal that their reads have completed; the coordinator then releases their writes. It reproduces the read-calculate-write pattern explicitly, rather than calling TransferNaive directly. That distinction matters when describing what the test covers.

These numbers are assertions in the repository, not measurements from a test run for this article. No lab tests were executed during preparation.

Why the constraint does not reject it

Alice still has 200,000. There is no negative balance to reject. The missing debit is a lost update, and the violated invariant is the cross-account sum. A successful commit and a satisfied CHECK constraint are insufficient evidence that the transfer was correct.

This does not mean every READ COMMITTED update loses data. The vulnerable implementation reads, calculates in application memory, and later writes an absolute value. Its query pattern is part of the explanation.

Separate the SQL standard from PostgreSQL behavior

The README describes the standard anomaly ladder: READ UNCOMMITTED permits dirty reads; READ COMMITTED prevents dirty reads but permits non-repeatable and phantom reads; standard REPEATABLE READ prevents dirty and non-repeatable reads but may permit phantoms; SERIALIZABLE prevents those anomalies.

PostgreSQL provides stronger behavior in two important places. READ UNCOMMITTED behaves as READ COMMITTED and does not permit dirty reads. PostgreSQL REPEATABLE READ uses a stable snapshot and prevents the classic phantom read demonstrated by the lab.

PostgreSQL level	Visibility in this lab	Remaining concern
READ UNCOMMITTED	Same behavior as READ COMMITTED	Does not grant a stable transaction snapshot
READ COMMITTED	New snapshot per statement	Repeated reads may change; naive writes may lose updates
REPEATABLE READ	Snapshot acquired at the first read	Concurrent same-row update can fail with 40001
SERIALIZABLE	SSI with serial execution equivalence	Conflicts may abort a transaction; retry must be handled

The table describes the engine used by the repository. It should not be treated as a portable implementation table for other databases.

Read Committed: two statements, two views

TestReadCommitted_NonRepeatableRead coordinates a reader and a writer. The reader first sees Alice at 1,000,000. The writer adds 500,000 and commits. The reader's second SELECT, still inside the same transaction, is expected to see 1,500,000.

The change is committed, so this is not a dirty read. It is a non-repeatable read caused by statement snapshots.

The phantom experiment uses a predicate instead of a single account. Two invoices initially have status PAID. A writer inserts a third PAID invoice and commits between the reader's queries:

SELECT COUNT(*) FROM isolation_invoices WHERE status = 'PAID'

Under READ COMMITTED, the test expects counts of two and three. Under PostgreSQL REPEATABLE READ, it expects two and two. The predicate is unchanged; the visible set of matching rows differs only in the first strategy.

A stable snapshot is not exclusive access

PostgreSQL MVCC lets ordinary readers observe versions without taking a FOR UPDATE row lock. TestRepeatableRead_SnapshotIsolation expects both reads to return the original 1,000,000 even after another transaction commits an increase.

The concurrent-update experiment is different. Two REPEATABLE READ transactions read the same account, then try to deduct 800,000. Its assertions require exactly one success and one serialization failure, SQLSTATE 40001.

The application must therefore handle an aborted transaction. Choosing REPEATABLE READ does not make the second transfer silently succeed correctly. It changes the failure behavior.

The README also discusses write skew as a possible serialization anomaly under snapshot isolation. The supplied experiments demonstrate same-row update conflicts; they do not reproduce a separate write-skew scenario. Keep the conceptual warning separate from the experiment's actual evidence.

Row locking moves validation behind coordination

TransferWithLock stays at READ COMMITTED but obtains both account locks before validating the sender. Here is the locking excerpt:

firstID, secondID := DeterministicLockOrder(fromID, toID)

    // Lock both accounts deterministically
    var b1, b2 int64
    err = tx.QueryRowContext(ctx, "SELECT balance FROM isolation_accounts WHERE id = $1 FOR UPDATE", firstID).Scan(&b1)
    if err != nil {
        return fmt.Errorf("lock account %d: %w", firstID, err)
    }
    err = tx.QueryRowContext(ctx, "SELECT balance FROM isolation_accounts WHERE id = $1 FOR UPDATE", secondID).Scan(&b2)
    if err != nil {
        return fmt.Errorf("lock account %d: %w", secondID, err)
    }

Excerpt from transfer.go; the rest of the transfer is omitted.

A competing FOR UPDATE or UPDATE on those rows waits until the holder ends its transaction. An ordinary SELECT continues to use MVCC. The lock is coordination for the write path, not a promise that every reader is blocked.

After locking, the implementation identifies which locked balance belongs to the sender and checks sufficient funds. Debit and credit use arithmetic SQL updates, and the audit record is written before commit. The second competing transfer validates after obtaining access to the row, rather than trusting an earlier unlocked balance.

Acquire locks in a shared order

If A-to-B locks A first while B-to-A locks B first, the two transactions can create circular wait. The lab uses a helper independent of transfer direction:

func DeterministicLockOrder(id1, id2 int) (firstID, secondID int) {
    if id1 < id2 {
        return id1, id2
    }
    return id2, id1
}

Both directions acquire the smaller account ID, then the larger. The bidirectional test runs two workers with 50 transfers each. It checks errors and total balance. This order addresses the two-account lock pattern used here; it does not guarantee that arbitrary additional locking elsewhere can never deadlock.

Lock waits are a real cost. The README's phrase about deterministic latency should not be read as a guarantee from this implementation. Contention and transaction duration still affect waiting, and the repository provides no latency benchmark.

Serializable means serial equivalence, not a global queue

PostgreSQL SERIALIZABLE uses Serializable Snapshot Isolation. Concurrent transactions can run together; committed results must be equivalent to an allowed serial execution. Conflicts can require an abort.

TestSerializable_ConcurrentUpdate_SerializationFailure coordinates two same-row updates and expects one success plus one 40001. It is a conflict experiment, not a throughput comparison or proof of every possible invariant.

The recovery path needs a new transaction with fresh reads and validation. Retrying only the failed statement would not repeat the decision that caused it.

Retry the whole operation, with a limit

The wrapper passes a complete transfer as the operation:

func (r *PostgresWalletRepo) TransferSerializableWithRetryPolicy(ctx context.Context, fromID, toID int, amount int64, maxAttempts int, policy RetryPolicy) error {
    repo := r
    return RetryTransaction(ctx, maxAttempts, policy, func(ctx context.Context) error {
        return repo.TransferSerializable(ctx, fromID, toID, amount)
    })
}

Complete wrapper from transfer.go.

Each attempt calls TransferSerializable again, including BeginTx, read, validation, debit, credit, audit, and commit. Failed attempts defer rollback. The retry helper accepts only serialization failure 40001 or deadlock 40P01, through the PostgreSQL error or corresponding sentinel.

func IsRetryableTxError(err error) bool {
    if err == nil {
        return false
    }
    if errors.Is(err, ErrSerializationFailure) || errors.Is(err, ErrDeadlockDetected) {
        return true
    }
    var pqErr *pq.Error
    if errors.As(err, &pqErr) {
        return pqErr.Code == "40001" || pqErr.Code == "40P01"
    }
    return false
}

Complete filter from transfer.go.

WrapTxError preserves both sentinel matching with errors.Is and access to *pq.Error with errors.As. Constraint violations such as 23505 and 23503, generic errors, and insufficient funds are not automatically retried.

maxAttempts includes the initial attempt. Non-positive values are rejected. The default base delay is 10 milliseconds and the maximum is one second. Before another attempt, the helper computes an exponential upper bound, caps it, and multiplies it by a random value for full jitter.

Context cancellation is checked before attempts and can interrupt the default sleep. When retryable errors exhaust the limit, ErrMaxRetryExceeded wraps the last error. retry_test.go covers filtering, error preservation, success after a conflict, attempt exhaustion, cancellation, invalid limits, and delay behavior.

Test the invariant under contention

The stress fixture differs from the introductory wallet: three accounts begin at 50,000 each, giving total 150,000. One hundred goroutines attempt transfers of 1,000 using TransferWithLock after a shared start gate is released.

The test checks transfer errors, non-negative final balances, and the conserved sum of 150,000. A start gate creates competing work; it does not force every goroutine to reach every query at an identical moment. The naive test uses a more specific barrier after reads to reproduce its anomaly.

To run the supplied experiments, start the repository's PostgreSQL infrastructure and execute:

cd labs/08-database-isolation-level
go test -v ./...

The database helper skips integration tests when PostgreSQL is unavailable unless REQUIRE_POSTGRES=1 makes absence a failure. A test command that reports success alongside skipped experiments is not evidence that database behavior was verified.

Choose from the invariant outward

For this wallet's known rows, READ COMMITTED with FOR UPDATE and a shared lock order coordinates the critical read-check-write path. REPEATABLE READ serves a stable multi-query view but requires conflict handling for updates. SERIALIZABLE addresses serial equivalence at the cost of aborts, retry work, and dependency tracking.

The README mentions an atomic conditional update as an alternative, but the lab does not implement that strategy. It also offers no measured throughput or latency ranking. The practical conclusion is to specify the invariant, identify the relevant reads and writes, choose coordination or conflict detection, and verify the resulting state under overlap.

Repository and source map

Observability: Making an Invoice Request Explain Itself

lukman lukman — Wed, 30 Sep 2026 23:55:00 +0000

A successful response says the invoice workflow finished. It does not explain where the request spent its time. Lab 07 explores that gap through a Go service that loads an invoice, reserves inventory, calculates commission, generates a PDF, and sends a notification.

The useful question is not just “Is the endpoint working?” It is “What happened inside this request, and which dependency explains its latency or failure?”

One workflow, two levels of visibility

The unsafe service executes the same five dependency operations but exposes only a coarse completion duration or a generic failure log. Its HTTP response can still contain a request ID. What is missing is the service-level breakdown needed to investigate the request.

The safe service adds structured logs, Prometheus metrics, and OpenTelemetry spans around the workflow. This changes the evidence available during investigation; it does not make the business operations themselves faster.

The demo dependencies simulate work using configurable delays and errors. A delay waits on a timer while also listening for context cancellation. These are controlled diagnostic scenarios, not measurements of a real database, inventory service, or PDF engine.

Logs, metrics, and traces answer different questions

Logs describe discrete events. The safe service emits a structured event when a dependency completes, with its component, operation, duration, and outcome. On failure, the event also carries an error.

Metrics summarize many requests. Request counters describe traffic, duration histograms describe latency, error counters describe failures, and an in-flight gauge describes active requests. The gauge is a concurrency signal; it is not a complete measurement of CPU capacity or resource saturation.

Traces explain a particular execution. A server span contains an invoice-processing span, which contains one span for each dependency operation. That hierarchy connects the HTTP request to the work performed inside it.

A latency histogram can reveal a shift across requests. A trace can identify the slow operation in one request. A structured log can provide the event details at that operation. None replaces the other two.

Make the slow step visible

The demo's normal configuration assigns 20 ms to invoice loading, 15 ms to inventory, 10 ms to commission, 30 ms to PDF generation, and 20 ms to notification. The configured delays sum to approximately 95 ms before runtime and instrumentation overhead.

In the slow-PDF scenario, PDF generation receives a 4,800 ms delay. Keeping the other fake dependencies unchanged gives a configured total of approximately 4,865 ms. This is scenario arithmetic, not a benchmark result. An HTTP notification client also changes the execution path when enabled.

The point of the scenario is the contrast between “the request took several seconds” and “pdf.generate consumed most of the request.” A total-duration log provides the first statement. Dependency spans provide the second.

POST /invoices/{id}/process
  invoice.process
    database.load_invoice
    inventory.reserve
    commission.calculate
    pdf.generate
    notification.send

This is the hierarchy for the five-dependency workflow. A real HTTP notification call adds transport and downstream spans when those boundaries are instrumented.

Carry context through every dependency

The safe service starts a span for each dependency and passes that span's context into the dependency call. The context carries the causal relationship and cancellation signal together.

On a dependency failure, the implementation records the error on the dependency span and the invoice-processing span, marks them as failed, observes an error outcome, and returns a wrapped error identifying the component and operation. Processing stops at that failed step.

The HTTP handler then maps a processing error to a response: a deadline becomes 504, cancellation becomes 499, and other processing failures become 502. The response uses the generic message “invoice processing failed” and includes the request ID. Detailed failure information stays in telemetry.

These are mappings implemented by this lab, rather than a universal status-code policy.

Correlate events without turning IDs into metric labels

ContextLogger attaches request_id and, when an active span is valid, trace_id and span_id. The request ID helps connect the HTTP response to logs. The trace ID groups spans in one execution. The span ID identifies the active operation within that trace.

The correlation fields are useful in logs, but they are deliberately absent from metric labels. Request metrics use method, a route template, and status class. Dependency metrics use component, operation, and outcome.

The route label is /invoices/{id}/process, rather than a separate label value for each invoice URL. The registry tests also check that request IDs and invoice IDs do not leak into label values.

That distinction preserves aggregate metrics while leaving request-level investigation to logs and traces. The fixture tests check selected sensitive keys in emitted logs; they do not implement a general-purpose redaction system.

Read the metric names as an investigation map

The collector exposes six metric families:

Metric	What it records
lab07_http_requests_total	Requests by method, route, and status class
lab07_http_request_duration_seconds	Request duration observations
lab07_http_request_errors_total	Processing errors at the HTTP boundary
lab07_http_in_flight_requests	Currently active requests for the route
lab07_dependency_duration_seconds	Dependency duration by component, operation, and outcome
lab07_dependency_errors_total	Dependency failures

Histograms store duration observations in seconds. Structured dependency logs report duration_ms. Keep those units explicit when comparing the two.

The missing-ID branch records a 4xx request but does not increment the processing-error counter. The name “request errors” therefore needs to be read alongside the implementation's accounting rules.

Cross an HTTP boundary without losing the trace

The handler extracts incoming W3C trace context before starting its server span. The demo configures TraceContext and Baggage propagation globally.

For outgoing notifications, HTTPNotificationClient creates a request with the current context and uses an OpenTelemetry HTTP transport. The transport propagates trace context, while the client explicitly copies X-Request-ID from the context into the downstream request.

The default notification client has a five-second timeout. A supplied client is copied before its transport is wrapped, and an already instrumented transport is not wrapped again. Request cancellation still reaches the outgoing HTTP call through its context.

The local /notifications endpoint lets the demo exercise this HTTP boundary. It runs in the same demo process; this is not evidence of a deployed multi-service production system.

Exercise specific failure modes

The demo supports normal, slow-pdf, slow-database, slow-external, and pdf-error scenarios. They isolate different dependency delays or a PDF failure so the resulting telemetry can be inspected.

For example, with the demo and its configured telemetry services running:

curl -X POST   'http://localhost:8087/invoices/INV-007/process?scenario=slow-pdf'   -H 'X-Request-ID: lab07-slow-pdf'

The corresponding unsafe route is /unsafe/invoices/INV-007/process. Comparing the routes makes the difference in diagnostic detail visible.

Use a stable request ID to find the relevant logs, inspect the request's dependency spans, and compare request-level duration observations with dependency duration observations. The useful outcome is an explanation of the configured scenario, not merely confirmation that an endpoint returned a response.

Treat telemetry behavior as a contract

The tests inspect span hierarchy, shared trace IDs, incoming trace-context extraction, outgoing propagation, request-ID preservation, structured log fields, and metric accounting.

The fake-dependency hierarchy test expects seven ended spans: one HTTP server span, one invoice-processing span, and five dependency spans. The HTTP notification path has additional instrumentation and should not be forced into that seven-span expectation.

The slow-PDF test uses a shorter, 10 ms delay and checks the recorded PDF span. Concurrent-request tests check mixed success and error responses, matching counters, and an in-flight gauge that returns to zero.

There is also a test for response-encoding failures. Such failures are logged with correlation fields, but the implementation has already recorded success accounting before encoding the successful response. That limitation matters when interpreting metrics.

These observations come from reading the source and its assertions. The Go tests were not executed in this content-generation environment.

The mental model

Observability is useful when the evidence explains a request: what happened, where its time went, and which operation failed. This lab makes that idea concrete by connecting dependency events, aggregate measurements, and a causal trace through one invoice workflow.

A system should explain what happened, where time was spent, and why it was slow or failed.

Repository: https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/07-observability

API Versioning with Separate Invoice Contracts

lukman lukman — Wed, 30 Sep 2026 07:23:01 +0000

An invoice endpoint returns customer information as a string. A new representation replaces that string with an object containing the customer's ID, name, and phone number. The response remains valid JSON and the server still returns HTTP 200. The legacy consumer expects a string and cannot decode the new object.

Lab 06 of Software Engineering Lab demonstrates this change using Go HTTP handlers and an explicit consumer model. It then keeps the old representation on V1 and introduces the nested representation on V2. A third example adds a field without changing the fields that the legacy model already understands.

The tests described here are source-defined scenarios and assertions. They were inspected, not executed for this article. The legacy consumer is simulated by a Go struct; this is not an Android device test.

The original invoice contract

The documented invoice response contains four fields:

{
  "id": 1001,
  "customer": "Budi",
  "total": 500000,
  "status": "PAID"
}

domain.go makes the consumer's expectation concrete:

type LegacyInvoice struct {
    ID       int    `json:"id"`
    Customer string `json:"customer"`
    Total    int64  `json:"total"`
    Status   string `json:"status"`
}

The helper ParseLegacyInvoice calls json.Unmarshal into that struct. This turns compatibility into an operation that the tests can exercise. The question is whether the existing model can still consume the body and obtain the expected values.

The fixture uses invoice ID 1001, customer name Budi, total 500000, and status PAID. These are demonstration values, not a production invoice dataset.

The unsafe handler changes an existing field's type

UnsafeHandler returns the following customer representation while retaining the invoice's other fields:

{
  "id": 1001,
  "customer": {
    "id": 15,
    "name": "Budi",
    "phone": "08123"
  },
  "total": 500000,
  "status": "PAID"
}

The new information is useful, but it occupies a field whose published representation was a string. LegacyInvoice.Customer cannot receive this object through the supplied decoder.

TestBreakingChange_LegacyClientFails first requires HTTP 200, then calls ParseLegacyInvoice on the response body. It expects decoding to fail. A passing test in this example would confirm the deliberate incompatibility; it would not certify the handler as backward compatible.

The unsafe endpoint is synthetic. It creates its payload directly after validating the path, rather than retrieving the invoice through the repository used by V1 and V2. It demonstrates the shape change, not a complete persistence-backed invoice service.

One domain model, two response models

The internal Invoice model already contains a nested Customer. The public V1 response does not have to copy that shape.

In safe_server.go, the response types establish separate contracts:

type InvoiceV1Response struct {
    ID       int    `json:"id"`
    Customer string `json:"customer"`
    Total    int64  `json:"total"`
    Status   string `json:"status"`
}

type CustomerV2Response struct {
    ID    int    `json:"id"`
    Name  string `json:"name"`
    Phone string `json:"phone"`
}

type InvoiceV2Response struct {
    ID       int                `json:"id"`
    Customer CustomerV2Response `json:"customer"`
    Total    int64              `json:"total"`
    Status   string             `json:"status"`
}

The mapper for V1 selects only the customer name:

func mapToV1(domain Invoice) InvoiceV1Response {
    return InvoiceV1Response{
        ID:       domain.ID,
        Customer: domain.Customer.Name,
        Total:    domain.Total,
        Status:   domain.Status,
    }
}

mapToV2 maps the same domain invoice into the nested customer DTO, preserving ID, name, and phone. Both handlers use the same repository interface. The repository fixture supplies the same invoice to each mapper.

This is the boundary the lab implements: the domain model holds the invoice information, while a version-specific mapper decides what the consumer receives. There is no database schema or database migration in this lab.

Route each contract explicitly

The supported endpoints are:

GET /api/v1/invoices/1001
GET /api/v2/invoices/1001

V1Handler maps its result through mapToV1; V2Handler maps through mapToV2. Registering V2 does not replace V1's handler or DTO.

TestVersionedRoutes_CanRunSideBySide registers both handlers on the same mux and requests both paths. It decodes each response into its version's type and checks the customer values.

Another regression test registers V2 and then checks V1's wire contract again. That test targets a specific risk: a new route exists, but the old response must still retain the fields the legacy consumer understands.

The lab uses URL versioning. Header-based selection appears in the README as a design alternative; there is no header-version negotiation implementation here.

Test the consumer and inspect the response

The safe tests use two complementary views of the body. One decodes V1 into LegacyInvoice and checks its invoice fields. Another reads the response into a map of json.RawMessage and checks field presence and values.

For the central breaking change, the important assertion is explicit: V1's customer is decoded into a string and compared with Budi. V2's customer is decoded into an object, then its id, name, and phone are checked separately.

Typed response tests also check the invoice ID, total, and status. This gives the tests a concrete fixture-based contract rather than merely requiring that the body be JSON.

These assertions cover the fields and requests they exercise. They are not an exhaustive schema validator for all payloads, nullability combinations, or consumer implementations.

Adding a field is a different change

AdditiveHandler keeps customer as a string and adds currency:

{
  "id": 1001,
  "customer": "Budi",
  "total": 500000,
  "status": "PAID",
  "currency": "IDR"
}

TestAdditiveField_LegacyClientStillWorks verifies that the body contains currency with value IDR. It then decodes the response into the old LegacyInvoice, which has no currency field, and checks the customer and total.

The supplied decoder tolerates the extra field. The README qualifies the broader design rule: additive response fields are usually compatible when consumers tolerate unknown fields. Consumers using strict validation may behave differently.

The demonstrated outcome belongs to this Go decoding path. The lab does not include a strict-schema consumer test, so it does not prove compatibility with every possible reader.

Compatibility includes request requirements

The source also protects an expectation outside the JSON body. TestV1Contract_DoesNotIntroduceRequiredTenantHeader sends a V1 request without X-Tenant-ID and expects HTTP 200.

That test records that this V1 handler does not introduce the header as a requirement. It does not establish a broader authentication policy; the lab contains no tenant authentication middleware.

The README lists other contract changes, such as renamed fields, required inputs, date formats, and changed error representations. The implemented string-to-object case is the main demonstration. The listed possibilities should not be mistaken for separately implemented experiments.

Routing behavior is tested separately

The helper parseInvoiceID validates the prefix, requires an ID, rejects extra path segments, parses a numeric value, and requires it to be positive. The versioned handlers then look up that ID.

For the versioned fixture, the expected cases include:

Request	Expected response
`GET /api/v1/invoices/1001`	200, invoice JSON
`GET /api/v1/invoices/`	400, missing ID
`GET /api/v1/invoices/abc`	400, invalid numeric ID
`GET /api/v1/invoices/0`	400, nonpositive ID
`GET /api/v1/invoices/1001/extra`	400, extra segment
`GET /api/v1/invoices/9999`	404, invoice not found
`POST /api/v1/invoices/1001`	405, `Allow: GET`
`GET /api/v3/invoices/1001`	404 from the router

The fixture repository contains only invoice 1001. Thus 9999 is a valid ID format but has no corresponding invoice in the fixture.

newVersionedMux registers both the exact collection path and the subtree path for each version. The request without a trailing slash therefore reaches the handler and produces the lab's 400 response, rather than relying on an implicit redirect.

Handler-generated responses use application/json, including method errors. An unmatched V3 path is handled by the mux's default 404 behavior; the README documents that response as plain text. The common JSON writer does not wrap every router response.

Migration remains a documented policy

The README describes releasing V2, retaining V1 for older consumers, monitoring adoption, communicating deprecation, and removing V1 after sunset criteria are met.

It includes example timeframes and traffic thresholds. They are illustrative policy values, not measured adoption data or an implemented shutdown schedule. This directory contains no traffic-monitoring pipeline or automated sunset mechanism.

The code demonstrates the prerequisite: both representations can be served through separate routes, with tests that continue checking V1. Deciding when V1 can be removed requires the consumer information described in the README.

Run the lab from its module

The directory contains its own go.mod, declaring Go 1.25.0. Its README lists these test commands:

cd labs/06-api-versioning
go test -v ./...
go test -race -v ./...

The tests use httptest and a mock repository rather than an external database. No test execution results are claimed here.

The lab's mental model is precise: HTTP 200 describes a server response, while compatibility must be evaluated against the consumer's contract. Separate DTOs and regression tests keep that contract visible when a new representation is introduced.

Source

Based solely on the README, Go handlers, models, and tests in Software Engineering Lab, Lab 06. Test outcomes are described from source assertions, not a new run.

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/06-api-versioning

Race Condition on Concurrent Stock Updates

lukman lukman — Tue, 29 Sep 2026 05:58:05 +0000

A stock update can be implemented with a sequence that looks reasonable when only one request is running:

read stock
check availability
decrease stock
save the new value

Lab 05 focuses on what happens when multiple operations execute that sequence against the same state.

The important part of the experiment is not whether each request contains valid business logic in isolation. The test is whether the invariant still holds when those requests overlap.

The lab covers this from several angles:

an unsafe read/check/write flow;
deterministic lost-update tests;
an atomic conditional SQL update;
PostgreSQL row locking;
concurrent inventory attempts;
concurrent booking attempts protected by a database uniqueness constraint;
a separate Go memory data-race example.

The main experiment is the application/database race.

The Smallest Inventory Case

Start with one unit of stock:

stock = 1

Two operations attempt to consume it.

The expected business result is straightforward:

successful operations = 1
remaining stock       = 0

The unsafe implementation separates the decision into multiple steps.

Conceptually, the execution is:

READ
  ↓
CHECK
  ↓
CALCULATE
  ↓
WRITE

With one caller, the sequence behaves as expected.

With two callers, those steps can interleave.

Two Operations Read the Same State

Consider this execution:

Request A                    Request B
---------                    ---------
READ stock = 1

                             READ stock = 1

CHECK stock > 0              CHECK stock > 0

newStock = 0                 newStock = 0

WRITE stock = 0

                             WRITE stock = 0

Both operations observed the same original value.

Both checks passed.

Both calculated the same replacement value.

The final database value is:

stock = 0

Looking only at the final stock value does not expose the full problem.

Both operations may have been treated as successful even though only one unit existed.

The broken invariant is therefore not simply:

stock must never be negative

The test also needs to account for the number of successful decrements.

For one available item:

successful decrements <= 1

The Lost Update

The unsafe flow calculates a new value outside the final write:

read stock
calculate stock - 1
write calculated value

That creates a window where another operation can read the same state.

The important interleaving is:

A reads 1
B reads 1

A calculates 0
B calculates 0

A writes 0
B writes 0

The second write is not based on the state produced by the first write.

It is based on the earlier value both callers observed.

That is the lost-update behavior exercised by the lab.

The repository uses deterministic concurrency coordination for these tests rather than depending only on timing or arbitrary sleeps. Barriers/start gates are used to force the relevant operations to overlap at the point required by the experiment.

That matters because a concurrency test that passes or fails only depending on scheduler luck is a weak proof.

Moving the Condition Into the Update

One implementation in the lab removes the separate read/check/write sequence.

The SQL statement is:

UPDATE inventory_products
SET stock = stock - 1
WHERE id = $1
  AND stock > 0
RETURNING stock;

There are two important details in this statement.

First:

stock = stock - 1

The application does not read a value, calculate a replacement, then send that replacement back.

Second:

WHERE stock > 0

The availability condition is part of the statement that mutates the row.

The operation becomes:

attempt decrement
only when stock > 0

instead of:

read
decide in application
write later

This removes the application-level race window demonstrated by the unsafe implementation.

The Invariant Is Checked at the Write Boundary

The atomic version changes where the decision happens.

Unsafe model:

Application:
    read stock
    check stock
    calculate new stock

Database:
    write supplied value

Atomic model:

Database:
    decrement stock
    only if stock > 0

The difference is visible under contention.

The repository includes a concurrent inventory scenario with:

initial stock = 100
concurrent attempts = 500

The useful assertion is not merely that all concurrent calls return.

The final state must still be consistent with the initial inventory.

The number of successful decrements cannot exceed the amount that was available.

PostgreSQL Row Locking

The lab also contains a PostgreSQL row-lock implementation and tests.

The relevant read uses FOR UPDATE:

SELECT stock
FROM inventory_products
WHERE id = $1
FOR UPDATE;

The row lock changes the execution shape.

Instead of two transactions independently reading the same stock and later trying to update it, the critical read/check/update flow is coordinated around the same row.

A simplified execution is:

Transaction A
-------------
BEGIN

SELECT ... FOR UPDATE

check stock

update stock

COMMIT

A concurrent transaction targeting the same locked row does not proceed through the same critical section independently while the first transaction still owns the lock.

The repository demonstrates this as another way to protect the same business invariant.

The lab does not use that result to claim that every concurrency problem should be solved with row locking. It is one implementation demonstrated against the stock-update problem.

Atomic Update and Row Lock Solve the Problem Differently

Both implementations target the same unsafe boundary, but the shape is different.

With the atomic update:

UPDATE ...
SET stock = stock - 1
WHERE stock > 0
RETURNING stock;

the condition and mutation fit into one statement.

With row locking:

lock row
↓
read state
↓
make decision
↓
update
↓
commit

the operation retains a multi-step decision while coordinating access to the row.

The lab therefore gives two concrete implementations rather than reducing the topic to "put everything in a transaction."

A transaction establishes a transaction boundary.

The concurrency behavior still depends on how the shared row is accessed inside that boundary.

Booking Uses a Different Invariant

The repository also applies the concurrency problem to booking.

The invariant is no longer a numeric stock value.

A booking is identified by a combination including:

branch_id
service_date
slot_time

The schema protects that combination with a unique constraint:

UNIQUE(branch_id, service_date, slot_time)

The important concurrency case is:

many requests
↓
same branch
same date
same slot

An application can check whether the slot exists before inserting.

That check is useful for normal control flow, but it is not enough to enforce the invariant under concurrency.

Two requests can both observe:

slot is available

before either insert is visible to the other operation.

The database constraint is the final storage boundary.

Concurrent Booking Test

The booking test exercises a single slot with:

500 concurrent attempts

Only the valid database state should remain.

PostgreSQL uniqueness conflicts are mapped through SQLSTATE:

The conflict is treated as an expected concurrency outcome rather than as proof that the database is broken.

The database is enforcing the invariant that the application needs.

This is the same design principle as the inventory test, applied to a different state shape.

Inventory invariant:

successful decrements <= available stock

Booking invariant:

one row for one unique branch/date/slot combination

The mechanism is chosen around the invariant being protected.

Application Pre-Checks Do Not Replace Constraints

Consider the booking sequence:

SELECT booking
WHERE branch/date/slot = requested slot

if not found:
    INSERT booking

Under concurrent execution:

Request A                       Request B
---------                       ---------
SELECT → not found

                                SELECT → not found

INSERT

                                INSERT

Without storage enforcement, both requests reached the same application decision.

The unique constraint prevents the database from accepting both final rows for the same slot.

This is another example of why code that is correct sequentially can still be incomplete as concurrent business logic.

Memory Data Race vs Business Race

Lab 05 also keeps a Go memory data-race example.

That experiment is useful, but it is not the same failure as the inventory and booking cases.

A memory data race is concerned with unsynchronized memory access between goroutines.

The inventory and booking experiments focus on business invariants over shared application/database state.

Those two categories can overlap in some systems, but they should not be treated as synonyms.

The repository keeps the intentional memory race separated so the application/database concurrency tests can be reasoned about independently.

What the Tests Need to Prove

Concurrency tests need stronger assertions than:

no panic

or:

all goroutines completed

For inventory, useful assertions concern the invariant:

successful operations
remaining stock
initial stock

For booking, the relevant assertion concerns how many rows survive for the unique slot.

The final repository revision also made the PostgreSQL lost-update reproduction deterministic rather than relying on an arbitrary timing window.

That is an important testing property.

The test should deliberately create the interleaving being investigated.

Commands

Lab 05 is a nested Go module.

The repository exposes dedicated Make targets including:

make lab-05-test
make lab-05-test-race
make lab-05-vet
make lab-05-fmt
make lab-05-integration

The nested module boundary matters when running tests from the repository root: a plain root-level go test ./... does not implicitly replace the dedicated Lab 05 targets.

The README was updated to reflect that layout.

What This Lab Demonstrates

The unsafe inventory example starts from code that is locally understandable:

read
check
modify
write

Concurrency changes the correctness requirement.

Another operation can run between those steps.

The fixes in the repository move the invariant to a boundary that can protect it:

atomic database update

or:

row lock around the critical database flow

For booking, the invariant is enforced through:

database uniqueness

The broader result from the experiment is specific:

sequential correctness
does not prove
concurrent correctness

The invariant has to survive the interleavings that the system actually permits.

Source

This article is based on an implementation from my Software Engineering Lab:

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/05-race-condition

Caching: Why Faster Reads Create Consistency Problems

lukman lukman — Tue, 22 Sep 2026 07:48:22 +0000

A dashboard request can look harmless.

The problem appears when hundreds of users ask for the same expensive data at the same time.

In Software Engineering Lab 04, the scenario is a workshop dashboard. Without caching, every request runs six database queries with joins and aggregations.

500 concurrent users
        ↓
Dashboard request
        ↓
6 queries + join/aggregation per request
        ↓
3000 total DB queries

The first instinct is easy:

put Redis in front of the database

That reduces repeated work.

But it also creates a new set of problems.

How stale may the cached value be?
When should it be invalidated?
What happens when the key expires under load?
What happens if Redis is unavailable?
Can the cache return data from the wrong tenant?

That is the part I wanted to explore in Lab 04.

The main mental model is not:

slow query → add cache

It is closer to:

reduce repeated work
        ↓
accept a consistency boundary
        ↓
design expiration, invalidation, failure, and concurrency behavior

PostgreSQL is still the source of truth

The lab uses two storage layers:

Layer	Technology	Role
Primary	PostgreSQL	Durable, persistent, authoritative storage that can rebuild the cache
Cache	Redis	Derived data, TTL-bound, rebuilt on demand

The important decision is ownership of correctness.

In this lab, PostgreSQL remains authoritative for business data. Redis stores derived data that can disappear, expire, or be rebuilt.

That means cache correctness has to work even when the cache is empty.

The cache is an optimization layer, not the only copy of the business state.

The read path: Cache Aside

Lab 04 uses Cache Aside for reads.

GET cache
   ↓
hit? ───── yes ───→ return cached value
   ↓ no
query PostgreSQL
   ↓
populate Redis
   ↓
return value

A cache miss is not an application failure.

It means the application has to rebuild the value from the authoritative source.

The application explicitly knows about both storage layers:

Redis
PostgreSQL

That control is useful when Redis is unavailable because the application can still attempt to read from PostgreSQL, as long as the database and fallback capacity can handle the traffic.

But Cache Aside also means the application now owns cache freshness.

That is where the interesting failures start.

TTL is really a staleness decision

The useful question is not simply:

Does this data change?

The lab asks:

How long can stale data be accepted?

Examples recorded in the lab:

Data	Max staleness	Reasonable?
Dashboard statistics	30s–2min	Yes, operational metrics
Stock display	1–5s	Yes, UI/UX only
Wallet balance	0s	No, audit risk

The difference matters.

A dashboard metric can tolerate a freshness window that would be unacceptable for a transactional balance.

So TTL is not just an expiry configuration. In this implementation, it represents an accepted freshness window and also acts as a recovery path when stale cache survives longer than expected.

Invalidating before commit can reintroduce stale data

Consider this write flow:

DELETE cache
↓
update database
↓
COMMIT

It looks reasonable. Remove the old cache first, then write the new database value.

The race appears when a reader enters between those operations.

Lab 04 describes this timeline:

T1 Writer: DELETE cache
T2 Reader: cache MISS → reads old DB value
T3 Reader: SET old value into cache
T4 Writer: DB COMMIT new value

Final state:

Database = new value
Cache    = old value

This is stale cache, not data loss. The authoritative business data in PostgreSQL is still correct.

A safer order is:

DB COMMIT
↓
DELETE cache

Now a reader that misses after the delete can rebuild from the committed database value.

But even this is not strong consistency.

Another interleaving still exists:

T1 Reader: cache MISS
T2 Reader: reads old DB value
T3 Writer: DB COMMIT new value
T4 Writer: DELETE cache
T5 Reader: SET old DB result into cache

Final state again:

Database = new value
Cache    = old value

So the conclusion is narrower:

COMMIT → DELETE

is safer than:

DELETE → COMMIT

but Cache Aside is still an eventually consistent optimization in this lab.

TTL remains useful as a safety net for the residual stale window.

Updating Redis after a database write is not atomic either

The lab also uses an application-managed update-on-write flow:

DB update
↓
DB COMMIT succeeds
↓
best-effort Redis SET
↓
return success

The database returns the authoritative value, then the application tries to update the cache.

The problem is the boundary between PostgreSQL and Redis.

These are separate systems, so this sequence is not atomic:

DB COMMIT
↓
Redis SET

A process can fail between them:

DB COMMIT succeeds
↓
process crashes
↓
Redis SET never happens
↓
old cache remains

Concurrent writers create another race:

Writer A commits value A
Writer B commits value B
Writer B SET cache = B
Writer A performs a late SET cache = A

Final state:

Database = B
Cache    = A

This was an important distinction for me:

Cache Aside
→ read strategy

Invalidate-on-write / update-on-write
→ write strategy

They solve different parts of the cache lifecycle.

Neither turns PostgreSQL and Redis into one atomic system.

One miss is normal. One thousand simultaneous misses are not

Now consider a popular key reaching expiration.

cache expires
      ↓
1000 concurrent requests arrive
      ↓
1000 cache misses
      ↓
1000 parallel DB queries
      ↓
database overload / crash

This is the cache stampede scenario used in Lab 04.

The cache successfully removes repeated database work while the entry exists, but expiration can suddenly send that work back to the database at the same time.

For duplicate rebuilds inside one process, the lab uses golang.org/x/sync/singleflight.

The flow includes a second cache check:

initial cache GET
      ↓
miss
      ↓
singleflight.Do
      ↓
check cache again
      ↓
query DB
      ↓
populate cache
      ↓
share result

The second check matters because another caller may already have populated the cache between the first miss and the point where this caller becomes the rebuild leader.

Singleflight stops at the process boundary

Singleflight coordinates callers inside one process.

A multi-instance deployment needs a different coordination boundary.

Lab 04 implements a distributed-lock primitive:

WithLock() = try-once lock primitive

The conceptual cache regeneration flow is:

cache GET
   ↓ miss
acquire distributed lock
   ↓
check cache again
   ↓
query DB
   ↓
populate cache
   ↓
safe release

The lock requirements in the lab are explicit:

unique token or owner;
TTL to avoid a permanent deadlock;
atomic compare-and-delete when releasing;
one holder must not remove another holder's lock.

But even this has a boundary.

If regeneration takes longer than the lock TTL:

Instance A acquires lock
↓
lock expires
↓
Instance B acquires a new lock
↓
Instance A is still rebuilding
↓
duplicate rebuild can run

The lock in Lab 04 reduces duplicate cache regeneration. It is not used as a correctness primitive for business transactions.

Expiration can also be spread out

The lab adds TTL jitter so many keys do not expire at nearly the same moment.

Example:

60s + random 0–15s

The implementation produces values in:

[base, base + maxJitter)

The TTL never goes below base, and the upper bound is exclusive.

The lab also discusses background refresh: refresh the cached value before expiry while clients continue receiving the existing cache value.

Both techniques target expiration behavior rather than changing the source of truth.

Cache keys are part of the data boundary

The canonical key format in the lab is:

{app}:{tenant}:{branch}:{resource}:{dimension}

Example:

cmms:tenant:42:branch:7:dashboard:2026-09-01

The rule is simple:

Every input that changes the result belongs in the cache key.

For the dashboard example, that includes tenant, branch, and business date.

This is also a security boundary in a multi-tenant system.

A sensitive cached value must include tenant scope. Missing isolation can expose data under the wrong tenant context.

Key design also affects reuse.

If 10,000 users read the same branch dashboard, this shape:

cache:tenant:42:user:{user_id}:dashboard

creates high-cardinality entries for data that is actually shared.

The lab contrasts it with:

cache:tenant:42:branch:{branch_id}:dashboard

One branch-scoped value can be reused by those readers.

The trade-off is not "specific keys are bad." The point is that key dimensions should match the result boundary.

Redis failure changes where the traffic goes

Cache Aside allows database fallback when Redis fails, but only while the authoritative dependency and fallback capacity remain available.

The traffic pattern becomes:

traffic previously absorbed by Redis
      ↓
directly reaches PostgreSQL
      ↓
load spike / cache-failure amplification

So graceful degradation does not mean the outage has no effect.

It means the main function may continue with degraded performance while PostgreSQL can still handle the fallback traffic.

The lab also separates:

cache_miss

from:

cache_error

because an expected miss and an unavailable cache backend are different operational events.

Hit ratio alone does not tell me whether the cache is worth it

Lab 04 evaluates cache value together with:

avoided query cost
cache latency
memory cost
invalidation complexity
failure amplification

The README gives this comparison:

30% hit ratio for a 100ms operation
may still be valuable

99% hit ratio for a 0.1ms operation
may not justify the complexity

The useful signal is not the percentage alone.

The cost of the work being avoided matters.

Cache comes after understanding the database work

The diagnostic order in the lab is:

measure endpoint
↓
check N+1 queries
↓
inspect execution plan
↓
add / optimize indexes
↓
reduce selected columns
↓
optimize joins / subqueries
↓
evaluate caching if the workload needs it

This prevents cache from becoming a way to hide an unexplained query problem.

For a very cheap query and some workloads, the extra cache hop and operational complexity may not provide meaningful end-to-end benefit.

The mental model I take from this lab

The naive model is:

query is expensive
→ add Redis
→ problem solved

Lab 04 forces a longer chain of questions:

What is the source of truth?
↓
How stale may this value be?
↓
What dimensions belong in the key?
↓
How is the value invalidated after writes?
↓
What happens during concurrent misses?
↓
What happens when Redis fails?
↓
Is the avoided work worth the added complexity?

That is the main lesson from the experiment.

Caching can reduce repeated work in a read-heavy workload.

But the moment a second storage layer is introduced, freshness, invalidation, concurrency, failure behavior, and isolation become part of the design.

Running the lab

From the repository root:

docker compose up -d redis

make lab-04-test
make lab-04-test-race
make lab-04-vet
make lab-04-demo
make lab-04-integration

The demo scenarios can also be run directly:

cd labs/04-caching

go run ./cmd/demo -scenario=without-cache
go run ./cmd/demo -scenario=cache-aside
go run ./cmd/demo -scenario=stampede-unprotected
go run ./cmd/demo -scenario=stampede-protected
go run ./cmd/demo -scenario=write-through

Source Code

Software Engineering Lab 04 — Caching

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/04-caching

Repository:

https://github.com/lukman-ss/software-engineering-lab

Author: Lukman (lukman-ss)

Database Transactions Are a Boundary, Not a Safety Blanket

lukman lukman — Mon, 21 Sep 2026 02:21:39 +0000

The uncomfortable part of a failed payment flow is not always the error.

Sometimes the function returns an error and the database still keeps half of the work.

That is the failure Lab 03 demonstrates. A payment is inserted. The order is marked as paid. Then the flow fails before the wallet transaction is inserted.

The result is not a clean failure:

payment persisted = 1
order.status = paid
wallet_transactions = 0

That is the useful lesson: database transactions are not just a way to call ROLLBACK. They are a boundary. Anything inside the boundary can commit or roll back together. Anything outside it cannot be magically undone by the database.

The Scenario

The lab starts with a local payment operation.

Initial state:

orders.id = 101
orders.status = pending

invoices.order_id = 101
invoices.status = unpaid

The service needs to do three local writes:

INSERT INTO payments (order_id, amount, status)
VALUES ($1, $2, 'completed')

UPDATE orders
SET status = 'paid'
WHERE id = $1

INSERT INTO wallet_transactions (order_id, amount, type)
VALUES ($1, $2, 'credit')

The injected failure happens after the first two statements, but before the wallet transaction insert.

The Problem

PaymentServiceUnsafe executes statements directly with db.ExecContext.

The flow is:

INSERT payment
  ↓
UPDATE order to paid
  ↓
injected failure
  ↓
wallet transaction is not inserted

There is no shared transaction boundary around the related writes. So when the failure happens, the database does not roll the earlier statements back.

The test result is the whole point:

payment persisted = 1
order.status = paid
wallet_transactions = 0

The application failed. The database did not return to the original state.

What Actually Went Wrong

The bug is not that SQL failed.

The bug is that the service treated a multi-step business operation as separate database statements.

From the business side, the payment row, the paid order status, and the wallet transaction belong to the same local invariant. From the unsafe database flow, they are just separate statements executed in order.

Without a transaction, the database has no reason to undo statement one and statement two just because the application never reaches statement three.

Using a Local Database Transaction

PaymentServiceSafe changes the boundary.

It starts a transaction:

tx, err := s.db.BeginTx(ctx, nil)

It keeps rollback as the default if commit does not happen:

committed := false
defer func() {
    if !committed {
        _ = tx.Rollback()
    }
}()

Then the related writes are executed through tx.ExecContext instead of db.ExecContext.

If all local writes succeed, the transaction commits:

if err := tx.Commit(); err != nil {
    return err
}

Successful Flow

The successful local flow is:

BEGIN TRANSACTION
  ↓
INSERT payment
  ↓
UPDATE order status
  ↓
INSERT wallet transaction
  ↓
COMMIT

At commit time, the local database state becomes complete together.

The lab uses the same idea again in the outbox example. InvoiceServiceOutbox updates the invoice and inserts an outbox event in one local transaction:

BEGIN
  ↓
UPDATE invoices SET status = 'paid'
  ↓
INSERT INTO outbox_events (... status = 'pending' ...)
  ↓
COMMIT

After commit, both the business state and the event intent exist locally.

Failure Flow

The failure path is where the difference becomes visible:

BEGIN TRANSACTION
  ↓
INSERT payment
  ↓
UPDATE order status
  ↓
injected failure
  ↓
ROLLBACK

The lab verifies the final state:

payments = 0
order.status = pending
wallet_transactions = 0

This is a clean local failure. The service still returns an error, but the database does not keep half of the payment flow.

Before vs After

Unsafe failure:

payments = 1
order.status = paid
wallet_transactions = 0

Safe local transaction failure:

payments = 0
order.status = pending
wallet_transactions = 0

This is what local atomicity buys you.

But the lab does not stop there, because this is only the easy boundary.

Rollback Does Not Undo WhatsApp

The next example uses DistributedOrderService.

The service opens a database transaction, inserts a payment, updates an invoice, sends a WhatsApp notification, then hits a simulated ERP integration error.

BEGIN TRANSACTION
  ↓
INSERT payment
  ↓
UPDATE invoice
  ↓
Send WhatsApp notification
  ↓
simulated ERP integration error
  ↓
ROLLBACK database transaction

The final state:

WhatsApp sent count = 1
payments = 0
paid invoices = 0

The database rollback worked. The WhatsApp message was still sent.

That is not a contradiction. WhatsApp is outside the database transaction boundary. The database can roll back its own rows. It cannot recall a message that has already been sent through another system.

The lab uses the same boundary idea for email, SMS, ERP APIs, payment gateway APIs, and message broker publishes.

HTTP Inside a Transaction

The lab also shows a blocking external call while a database transaction is open:

BEGIN TRANSACTION
  ↓
UPDATE invoice SET status = 'paid'
  ↓
HTTP call blocks
  ↓
transaction stays open

The test verifies that the transaction remains open during the blocking external call, then closes after commit.

The issue here is not only failure. The lifetime of the database transaction is now tied to an external resource.

The Dual-Write Gap

Another failure appears when database commit and event publish are separate operations.

UPDATE invoice
  ↓
COMMIT succeeds
  ↓
process crashes
  ↓
Publish event never happens

The test shows:

invoice.status = paid
published events = 0

The invoice was paid, but no event was published.

Reversing the order does not make the operation atomic. Publishing before commit can produce an event for a database state that never commits.

Transactional Outbox

The lab uses transactional outbox to keep the business state and event intent in one local transaction.

Local transaction
  ↓
Record business state + event intent
  ↓
COMMIT
  ↓
Dispatcher publishes pending events

InvoiceServiceOutbox inserts an outbox_events row with status pending. The dispatcher later reads pending events, publishes them to the broker, and marks them as published.

The important part is the local atomic step: invoice state and event intent are saved together.

Idempotent Consumer

The lab then shows the consumer side.

The commission worker stores a processed-event marker and the business state in the same transaction:

INSERT INTO processed_events (consumer_name, event_id, processed_at)
VALUES ($1, $2, $3)
ON CONFLICT (consumer_name, event_id) DO NOTHING

If the same consumer receives the same event again, the insert affects zero rows and the event is skipped.

The test result:

same consumer: processed once
same consumer duplicate: skipped
same event, different consumer: processed independently

This matters because the outbox side and the consumer side are connected. Recording an event intent is not enough. The receiver also has to handle repeated delivery safely.

What This Lab Demonstrates

A local transaction is the right tool for local state that must commit or roll back together.

It does not roll back external side effects.

It does not make database commit and broker publish atomic when those are executed as separate operations.

Outbox solves the local database/event-intent part by storing both in one transaction.

Idempotent consumer logic handles repeated processing on the consumer side by storing a dedup marker with the business update.

Key Takeaways

First define the transaction boundary.
Put related local writes inside the same transaction.
Do not treat external calls as rollbackable database work.
Do not keep a transaction open longer than necessary while waiting on external systems.
Use outbox when database state and event publishing must be coordinated.
Make consumers idempotent when the same event can be processed more than once.

Source Code

Repository:

https://github.com/lukman-ss/software-engineering-lab

Lab:

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/03-database-transaction

Author:

Lukman (lukman-ss)

Database Index: Why Queries Become Slow as Data Grows

lukman lukman — Thu, 17 Sep 2026 07:36:43 +0000

This query looks ordinary:

SELECT *
FROM service
WHERE branch_id = 2
  AND status = 'FINISHED'
  AND service_date BETWEEN '2026-01-01' AND '2026-01-31'
ORDER BY service_date DESC;

With a small dataset, a query like this may appear completely fine.

Lab 02 does not rely on assumptions. The query is tested using EXPLAIN (ANALYZE, BUFFERS) against the same dataset.

Dataset

Table: service

Total rows:

500,000

Status distribution:

Status	Fraction	Rows
FINISHED	70.00%	350,000
CANCELLED	20.00%	100,000
IN_PROGRESS	5.00%	25,000
WAITING	4.90%	24,500
PENDING_REFUND	0.10%	500

Branch 2 is the busiest branch, containing 25.00% of the data.

Baseline

The baseline only has constraint-backed indexes:

service_pkey
service_invoice_no_key

There is no secondary index on:

branch_id
status
service_date

Baseline query:

EXPLAIN (ANALYZE, BUFFERS)
SELECT *
FROM service
WHERE branch_id = 2
  AND status = 'FINISHED'
  AND service_date BETWEEN '2026-01-01' AND '2026-01-31'
ORDER BY service_date DESC;

What should be inspected:

plan node;
actual rows;
rows removed by filter;
explicit Sort;
shared read / shared hit;
planning time;
execution time.

Add the Candidate Index

CREATE INDEX idx_service_branch_status_date
ON service(branch_id, status, service_date DESC);

This index follows the query shape:

branch_id equality
→ status equality
→ service_date range/order

It is a candidate for this specific query, not a universal index.

Run the Same Query Again

EXPLAIN (ANALYZE, BUFFERS)
SELECT *
FROM service
WHERE branch_id = 2
  AND status = 'FINISHED'
  AND service_date BETWEEN '2026-01-01' AND '2026-01-31'
ORDER BY service_date DESC;

Compare:

Condition	What to inspect
Before index	Seq Scan, Sort, buffers, rows, time
After index	Index usage, Index Cond, Sort presence, buffers, rows, time

The lab does not store fixed execution-time numbers. Those values should come from the local run.

Cardinality Is Not Match Fraction

status has 5 distinct values. That is its cardinality.

Match fraction is the proportion of rows that match a predicate.

Predicate	Match fraction	Expected / likely plan
`status = 'FINISHED'`	70.0%	Seq Scan likely cheaper
`status = 'PENDING_REFUND'`	0.1%	index-based plan likely cheaper

The lesson is that low cardinality does not automatically make an index useless.

Column Order Matters

The lab compares:

CREATE INDEX idx_service_a_branch_status_date
    ON service(branch_id, status, service_date);

CREATE INDEX idx_service_b_date_branch_status
    ON service(service_date, branch_id, status);

CREATE INDEX idx_service_c_status_date_branch
    ON service(status, service_date, branch_id);

For the main query, the first index becomes a strong candidate because the leading equality predicates can narrow the scan range more effectively.

ORDER BY + LIMIT

Dashboard query:

SELECT *
FROM service
WHERE branch_id = 2
  AND status = 'FINISHED'
ORDER BY service_date DESC
LIMIT 20;

Index:

CREATE INDEX idx_service_branch_status_date_desc
    ON service(branch_id, status, service_date DESC);

What to inspect:

whether Sort is absent;
whether the plan uses Index Scan or Index Only Scan;
how many rows are examined before LIMIT is satisfied.

Index Is Not Free

The lab also tests:

inserting 1,000 rows without secondary indexes;
inserting 1,000 rows with 1 composite index;
inserting 1,000 rows with 4 secondary indexes;
updating indexed vs non-indexed columns;
storage cost using pg_relation_size, pg_indexes_size, and pg_total_relation_size.

An index can improve specific reads, but it also adds write and storage cost.

Key Takeaways

Do not optimize based on guesses.
Use EXPLAIN (ANALYZE, BUFFERS).
A composite index should follow the query shape.
Column order matters.
Cardinality is different from match fraction.
A Seq Scan can be the correct plan.
Indexes have write and storage costs.

Hands-on Lab

Software Engineering Lab #02 — Database Index

Author: Lukman

GitHub: lukman-ss

Repository:
https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/02-database-index

Making POST Requests Safe to Retry with Idempotency Keys

lukman lukman — Tue, 15 Sep 2026 08:05:50 +0000

A practical guide to duplicate execution, request fingerprints, concurrency, and safe retries.

A request timeout does not mean the operation failed.

Sometimes the server has already completed the operation, but the response never reaches the client.

Consider:

POST /orders/order-123/pay

with:

{
  "amount": 500000
}

The timeline may look like this:

Client -> POST /pay
Server -> payment succeeds
Server -> response lost
Client -> timeout
Client -> retry
Server -> payment succeeds again

The problem is not the duplicate HTTP request itself. The problem is duplicate execution.

The target invariant

For a mutation API, the goal should be:

1 logical operation
=
1 effective side effect

even if the request is delivered multiple times.

Introduce an Idempotency Key

The client generates an identifier for one logical operation:

Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000

The key must remain the same across retries.

same logical operation -> same key
new logical operation  -> new key

The basic flow becomes:

Request
  ↓
Check Idempotency-Key
  ↓
New key?
  ↓
Process operation
  ↓
Store result
  ↓
Return response

On retry:

Same key
  ↓
Existing completed result
  ↓
Replay response
  ↓
Do not execute side effect again

Why check-then-act fails

This is not enough:

if key does not exist:
    process payment

Two concurrent requests may both observe the key as missing:

A -> key not found
B -> key not found

A -> process
B -> process

This is a race condition. The uniqueness decision needs to be atomic.

For a relational database, that normally means enforcing the invariant in storage:

CREATE UNIQUE INDEX idx_idempotency_key
ON payments(idempotency_key);

Application checks are useful. The unique constraint is the final guard.

Same key, different payload

An idempotency key must represent exactly one logical operation.

This should be accepted:

key: abc
amount: 500000

retry

key: abc
amount: 500000

This should not:

key: abc
amount: 500000

retry

key: abc
amount: 800000

The lab uses a request fingerprint based on SHA-256:

func hashRequest(data []byte) string {
    sum := sha256.Sum256(data)
    return hex.EncodeToString(sum[:])
}

Then:

same key + same fingerprint
-> replay safely

same key + different fingerprint
-> 409 Conflict

PROCESSING vs COMPLETED

The safe flow uses two important states:

PROCESSING
COMPLETED

When a second request arrives:

PROCESSING
-> return 409
-> do not execute payment again

When the operation is already finished:

COMPLETED
-> load stored response
-> replay it

Idempotency is not a transaction

A database transaction provides atomicity inside one execution.

Idempotency protects a logical operation across multiple execution attempts.

These can both succeed:

Request A
BEGIN
create payment
COMMIT

Request B
BEGIN
create payment
COMMIT

The database is consistent. The business result is not. The customer was charged twice.

External side effects have a different boundary

Consider:

Call payment provider
↓
Provider SUCCESS
↓
Local commit fails
↓
ROLLBACK

The local rollback does not undo the external charge.

local rollback
!=
external rollback

If the provider supports provider-side idempotency, the backend should reuse a stable key for that external request too.

Lab implementation

The lab intentionally keeps the infrastructure small.

Unsafe:

func ProcessPayment(req PaymentRequest) (PaymentResult, error) {
    return gateway.Charge(req)
}

Safe:

validate key
↓
calculate fingerprint
↓
reserve operation
↓
PROCESSING
↓
execute payment
↓
store result
↓
COMPLETED

The storage implementation uses:

map + sync.RWMutex

This simulates atomic uniqueness without introducing a real database into the lab.

Run both versions:

go test ./labs/01-idempotency/unsafe/... -v -count=1

go test ./labs/01-idempotency/safe/... -v -count=1

Final mental model

Retry is normal.
Duplicate execution is the problem.
Idempotency makes retry safe.

Source code:

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/01-idempotency