DEV Community: lukman lukman

Load Testing: Read the Queue Behind Successful Bookings

lukman lukman — Thu, 15 Oct 2026 11:36:00 +0000

A booking endpoint can keep returning success while concurrent users wait much longer. Lab 15 makes that behavior visible with a local HTTP server, a constrained work pool, and a small Go load generator.

Load testing reveals how latency changes when demand competes for limited resources.

Fast bookings can queue under load

One successful booking does not explain concurrent behavior. Lab 15 sends POST /booking to a local HTTP server, comparing low and high concurrency to expose queueing and latency distributions. It models bounded work slots, not a production database.

Two scenarios share one server

The demo uses five simulated database slots and a 20 ms base query. Smoke runs two VUs; stress runs 50. Both run for two seconds, sequentially, with the same endpoint and booking body. These are demonstration settings, not production capacity targets.

Each VU waits before sending again

Each VU has one goroutine and sends its next request after the previous attempt finishes. There is no think time, ramp-up, or arrival-rate scheduler. This is a closed loop: slower responses reduce each VU's request rate.

A buffered channel bounds simulated work

A buffered channel acts as a semaphore. Handlers acquire a token before simulated query work and release it with defer. Full capacity makes additional handlers wait; there is no explicit bounded waiting queue or overload rejection policy.

select {
case s.semaphore <- struct{}{}:
case <-r.Context().Done():
    return
}
defer func() { <-s.semaphore }()

The database pool is simulated

There is no SQL driver, real query, booking table, or database transaction. A timer simulates work. ActiveConnections returns len(semaphore): acquired slots, not waiting handlers, TCP connections, or real database telemetry.

Random slow work amplifies queueing

activeReq includes waiting and working POST handlers. After admission, when activeReq exceeds capacity, a 10% random branch multiplies query duration by 25. The demo's 20 ms becomes 500 ms. Stress results therefore combine queueing with deliberately injected slow work.

if atomic.LoadInt64(&s.activeReq) >
    int64(s.cfg.MaxDBConnections) {
    if rand.Float32() < 0.10 {
        dur = s.cfg.DBQueryDuration * 25
    }
}

Context bounds the test lifetime

Run uses context.WithTimeout for the configured duration and passes that context to each HTTP request. A five-second client timeout also applies. The server checks cancellation while acquiring a slot and waiting for simulated work; it does not return a dedicated overload response.

Canceled attempts are not all counted

Request-construction and client.Do errors are counted only while ctx.Err() is nil. End-of-test canceled attempts can be omitted. TotalRequests is recorded success samples plus counted errors, not every attempt started.

if err != nil {
    if ctx.Err() == nil {
        errs++
    }
    continue
}

Success follows an HTTP status rule

The runner drains and closes response bodies. Status >=400 increments errors; status <400 records success latency, which is broader than a 2xx-only rule. JSON business fields are not validated, and body-read errors are ignored.

Latency includes reading the body

Timing starts before client.Do and ends after the successful response body is drained and closed. It is not query-only time. DNS, dialing, TLS, first-byte wait, pool wait, and body transfer are not separately measured.

Percentiles use successful samples only

Failed responses and transport errors have no latency samples. Percentiles, average, min, and max use successful samples only. With no successes, latency fields remain zero; that does not mean the service responded instantly.

RPS includes counted errors

Total is len(latencies)+errors. RPS divides that total by Run's actual elapsed duration after goroutines finish, so it is not success-only throughput. With no total requests, an empty Result is returned, including zero Duration.

total := len(latencies) + errors
res.RPS = float64(total) /
    totalDuration.Seconds()

Percentiles select a floored index

The calculator copies and sorts samples, then selects int((n-1)*p/100) without interpolation. For the test fixture 1–100 ms, P50=50 ms, P95=95 ms, and P99=99 ms. Other tools may use different percentile conventions.

idx := int(float64(len(sorted)-1) *
    (pct / 100.0))
return sorted[idx]

A fixture makes the calculation explicit

The 1–100 ms fixture has min 1 ms, max 100 ms, and average 50.5 ms. time.Duration retains nanosecond units, so division does not force whole-millisecond rounding. These are calculated fixture values, not HTTP performance measurements.

Small samples make the tail coarse

With one 42 ms sample, every percentile is 42 ms. Small datasets can map several tail percentiles to the same sample and may exclude the maximum under this formula. Sample count and method matter when interpreting precision.

Average and tail answer different questions

Average summarizes all successful samples; P95 selects a high-ranked sample. Unit tests verify ordered percentile invariants. The integration test separately checks stress P95 exceeds stress average; that relationship is not guaranteed for every possible distribution.

Demo numbers are recorded examples

The repository records a sample smoke P95 of about 21.37 ms and stress P95 of about 1.357 s, with zero counted errors. These are historical example results, not a fresh measurement. Random spikes and scheduling vary; omitted canceled attempts also qualify the zero-error interpretation.

Transport settings are not a full generator audit

MaxIdleConns and MaxIdleConnsPerHost are both 1000; these are idle-connection retention settings, not proof that every client bottleneck is eliminated. All successful latencies are retained per VU, merged, copied, and sorted, so memory grows with collected samples.

A booking response is not a real transaction

Non-POST methods receive 405. Completed POST requests return 201, confirmed, and the fixed ID BK-1001. Request JSON is not decoded or validated, and there is no persistence, authentication, idempotency, or unique booking creation.

Tests establish specific behaviors

Five metric tests cover fixtures, empty input, one sample, RPS, and ordering. Seven integration tests cover load comparison, HTTP 500, method rejection, dial errors, cancellation, slot limits, and total-count invariants. They do not prove queue fairness or maximum capacity.

The lab bottleneck is known by design

The slot bottleneck is deliberately constructed. In production, rising P95 alone does not isolate application queues, database work, network, or external dependencies. The runtime has no exporter, tracing, or automated SLA threshold; the demo prints summaries.

Rerun and interpret in context

The module declares Go 1.22 and uses the standard library. Go is unavailable in this authoring environment, so source and assertions were inspected without rerunning tests. Recorded repository results are prior-run evidence. Load testing reveals how latency changes when demand competes for limited resources.

cd labs/15-load-testing
go test -v ./...
go test -race ./...
go run ./cmd/demo

Source

Software Engineering Lab 15, snapshot commit 82af988b89439fa92b1200406c5afbd59ce72a56. Source: README, runner, metric calculator and tests, server, integration tests, demo, execution record, and go.mod. Code excerpts belong in their original package context.

Circuit Breakers Protect Callers from Failing Dependencies

lukman lukman — Wed, 14 Oct 2026 11:29:00 +0000

A payment service slows down. Checkout continues sending requests and waiting for timeouts. Lab 14 places a circuit breaker before the payment client so repeated failures can stop subsequent attempts.

A circuit breaker protects the caller; it does not repair the dependency.

The lab uses real local HTTP through an httptest fake server. It does not perform real payments, balance mutations, or production gateway integration.

The implemented path

Checkout calls Breaker.Execute, whose callback calls payment.Client.ProcessPayment. The client sends POST /pay. The fake server supports HEALTHY (200), DOWN (500), and SLOW (sleep, then 200).

The README motivates the pattern with cascading resource exhaustion. The tests demonstrate narrower behavior: state transitions, callback rejection, HTTP request counts, timeout errors, and recovery probes. They do not measure thread exhaustion, memory pressure, or a platform-wide outage.

CLOSED counts consecutive failures

The breaker starts CLOSED and admits callbacks. A successful completion resets its failure count. A nonnil error increments it; reaching the threshold opens the circuit:

// circuit_breaker.go: onFailureLocked excerpt
b.failures++
if b.failures >= b.cfg.FailureThreshold {
    b.state = Open
    b.openedAt = now
    b.generation++
}

This is a consecutive-error counter, not a sliding-window failure rate. With concurrent calls, valid completion order determines the counter updates.

Defaults are three failures, a 300 ms cooldown, and one HALF_OPEN call. New substitutes defaults for nonpositive values. These are lab settings, not universal production recommendations.

Callbacks execute outside the mutex. CLOSED has no general in-flight limit, so a concurrent burst can dispatch more requests than the threshold before the first failures complete. A circuit breaker is not automatically a bulkhead.

OPEN rejects before invoking the callback

// circuit_breaker.go: Execute state branch
case Open:
    b.mu.Unlock()
    return ErrCircuitOpen

During OPEN before cooldown expires, no callback runs. When the callback wraps the payment client, that rejected call produces no new network request. TestOpenDoesNotCallDownstream checks one initial failed callback followed by five rejections, with the callback count remaining one.

Already admitted calls are not canceled. “No downstream calls while rejecting” is different from claiming no older request is still running.

Cooldown advancement is lazy

There is no background timer that sleeps and sends a probe. Both State() and Execute() call advanceLocked. Once enough time has elapsed since openedAt, the state becomes HALF_OPEN, the probe counter resets, and the generation increases.

Calling State() can therefore mutate state. An expired cooldown does not itself dispatch a recovery request; a later admitted Execute does that.

HALF_OPEN limits admission, not the number of successes required

// circuit_breaker.go: HALF_OPEN branch
case HalfOpen:
    if b.halfOpenIn >= b.cfg.HalfOpenMaxCalls {
        b.mu.Unlock()
        return ErrCircuitOpen
    }
    b.halfOpenIn++

Extra calls receive the same ErrCircuitOpen used for OPEN. The error alone does not distinguish an open circuit from a full probe allowance.

One successful probe closes the breaker. One failed probe reopens it and restarts cooldown. When the configured allowance exceeds one, the implementation does not wait for every admitted probe to succeed: the first state-changing result advances the generation, making remaining old results stale.

That is a specific recovery policy, not a requirement for N consecutive successful probes.

Generation guards separate old work from new state

Execute captures a generation before releasing the mutex to run the callback. Result bookkeeping begins with this guard:

// circuit_breaker.go: result bookkeeping guard
if b.generation != gen {
    return
}

A trailing request from an earlier generation cannot reset or reopen a newer state. The trailing-in-flight test starts a slow CLOSED request, opens the circuit using another request, advances to HALF_OPEN, and checks that the old failure does not corrupt HALF_OPEN.

The old callback still runs and its error still returns to its caller. Generation filtering changes its authority to update breaker state; it is not cancellation.

Panic cleanup does not swallow panic

Deferred bookkeeping treats a callback panic as a failure, including reopening HALF_OPEN. The panic continues to the caller rather than becoming an ordinary returned error. The test recovers outside the breaker and checks that a later recovery probe can run.

This prevents a probe panic from leaving the breaker permanently stuck with its allowance consumed, while preserving the panic's normal propagation.

Timeouts remain necessary

The HTTP client has a configured timeout. Checkout(ctx) passes the caller's context into the payment request; the two demo helpers use context.Background().

Breaker.Execute itself has no context argument or execution deadline. An admitted callback can still wait indefinitely without its own timeout, including a HALF_OPEN probe occupying a slot.

Retry and exponential backoff are discussed in the README but not implemented in this client or checkout service. The breaker does not schedule retries or automatically control an external retry loop. It only admits or rejects each callback evaluation.

The fake SLOW handler sleeps without checking request cancellation. A client timeout is therefore not proof that the server has stopped its work.

Failure classification is intentionally broad

The client accepts only HTTP 200. Every other status becomes ErrPaymentFailed; transport errors, timeouts, and cancellation also return errors. The breaker counts all nonnil errors without classification, so a 4xx or canceled request can affect the threshold if it reaches this path.

A 200 response is not decoded to validate payment business semantics. A successful probe means this client returned nil, not that a real transaction was completed correctly.

Read the demo as four scenarios, not one benchmark

The first scenario uses SLOW with a 200 ms server delay and a 100 ms client timeout, making three sequential calls without a breaker. The second uses DOWN responses, a threshold of three, and six sequential calls with a breaker.

Those are different failure modes. They should not be presented as a controlled latency benchmark of an identical workload. The N × timeout approximation describes sequential waiting, not general concurrent request latency.

The third scenario makes the server HEALTHY before probing and demonstrates HALF_OPEN → CLOSED. The fourth keeps it DOWN and demonstrates HALF_OPEN → OPEN.

Nanosecond or microsecond durations shown in the README are example output, not a latency guarantee or a newly measured result here.

Fallback must preserve business meaning

The implemented checkout returns an error on payment failure or rejection. It has no cached success, queue, or automatic alternative payment flow.

The README's CMMS and PPOB examples discuss critical versus noncritical dependencies. Invoice creation, WhatsApp delivery, provider fulfillment, queue retention, and debit protection are architectural discussion, not implemented workflows in this folder.

A fast payment rejection still means checkout has failed. The breaker reduces futile attempts; it does not manufacture a successful payment during an outage.

What the tests and instrumentation establish

Unit tests cover thresholds, success resets, rejected callbacks, cooldown, both probe outcomes, probe limits, panic cleanup, stale generations, and concurrent access. Integration tests use the HTTP fake server to check 500 responses, recovery, request counts, and slow-call timeouts.

The general concurrency test checks a valid final state. The interleaved test waits for goroutines without additional semantic assertions. A race-detector run can provide evidence about executed paths, but does not independently prove every interleaving or policy.

Production gauges, open/rejected counters, and dependency latency percentiles are recommendations in the README. This implementation exposes state and the fake server's request counter, not a production metrics exporter.

cd labs/14-circuit-breaker
go test -v ./...
go test -race ./...
go run ./cmd/demo

The module declares Go 1.22. Go is unavailable in this authoring environment, so I inspected code and assertions rather than rerunning the tests. No new passing-test result or benchmark is claimed.

The reusable lesson is to bound permitted calls and interpret recovery precisely: timeout limits waiting, the breaker controls subsequent admission, and probes decide whether that path should reopen.

Source

Software Engineering Lab 14, snapshot commit 82af988b89439fa92b1200406c5afbd59ce72a56: README, circuit breaker implementation/tests, payment client/fake server, checkout service, integration tests, demo, and go.mod. Excerpts belong in their existing package context.

Backward Compatibility Must Survive the Transition

lukman lukman — Tue, 13 Oct 2026 10:14:00 +0000

The old system stores one phone number in users.phone. The new requirement is a list of numbers per user. Replacing a JSON string with an array is easy to code, but an old consumer still expects that string.

Lab 13 uses Expand → Migrate → Contract to let old and new representations coexist. The important question is not only whether the final design works. It is whether readers and writers can survive every intermediate state.

Compatibility must survive the transition, not just the final state.

The implementation uses an in-memory Go store. It teaches a migration sequence; it does not demonstrate a live database migration or establish zero downtime.

Expand the contract before retiring the old representation

The enriched response preserves phone as a string and adds a separate phones array:

// model.go: complete response type
type UserResponse struct {
    ID     int          `json:"id"`
    Name   string       `json:"name"`
    Phone  string       `json:"phone,omitempty"`
    Phones []PhoneEntry `json:"phones"`
}

PhoneEntry contains an ID, user ID, number, and primary marker. LegacyConsumerDTO expects the scalar field; ModernConsumerDTO expects the array.

TestSerializationBackwardCompatibility creates a dual-written user, marshals one enriched payload, and unmarshals it into both DTOs. Its assertions check the legacy number, two modern entries, and the primary number. That is useful evidence for these two Go consumers and this fixture, not a universal claim about all strict decoders or generated clients.

Empty representations also matter. The scalar has omitempty, so an empty string may disappear from the enriched payload. CreateUser in legacy-only mode leaves Phones nil, which can serialize as null; the store's empty read slice can serialize as []. Those are different payload shapes.

API version and storage mode are separate choices

The V1 handler returns LegacyConsumerDTO; the V2 handler returns ModernConsumerDTO. They do not both return the same enriched payload used by the serialization test.

The service also has independent write and read modes:

Mode	Behavior
`WriteLegacyOnly`	Store the scalar; ignore extra phone arguments.
`WriteDual`	Store scalar and modern primary, then add extras to modern storage.
`WriteNewOnly`	Store modern entries with a nil legacy phone.
`ReadLegacyOnly`	Derive the response's scalar from legacy storage.
`ReadFallback`	Prefer modern primary/first entry, otherwise use legacy.
`ReadNewOnly`	Derive the scalar from modern entries.

There is a subtle boundary here: GetUser always fetches modern entries before selecting the scalar's source. ReadLegacyOnly does not hide the enriched response's Phones array. GetModernUser returns that array.

Dual-write protects what the old reader still needs

MemoryStore.CreateDual creates the user and modern primary phone under one mutex. The old scalar stays populated, allowing a legacy read of records created during the dual-write phase.

Additional numbers exist only in the modern representation. Backward compatibility preserves the old view; it does not give an old client all new capabilities.

The atomicity claim needs a precise scope. Service.CreateUser saves extraPhones through separate SavePhoneEntry calls after CreateDual, ignoring errors from those extra calls. The complete multi-phone operation is not one database transaction. The store is a collection of maps, while schema.sql is a schema illustration rather than an executed storage adapter.

Backfill historical data in bounded batches

Dual-write covers creates passing through that mode. It does not migrate historical users by itself.

BackfillWorker scans IDs after LastProcessedID. It copies a nonempty legacy phone only when modern entries are absent. Existing modern entries are skipped, so backfill is not a repair mechanism for every mismatch.

The storage helper prevents duplicate numbers for the same user:

// store.go: SavePhoneEntry excerpt
for _, p := range s.userPhones[userID] {
    if p.Number == number {
        return p, nil
    }
}

Returning an existing entry does not update its primary marker. The behavior is deduplication by user and number, not a general merge policy.

The worker keeps LastProcessedID, TotalMigrated, and IsComplete in memory. Its test processes ten users as batches of three, three, then four remaining records, and expects zero from rerunning the completed worker.

“Resumable” therefore means continuing the same worker object. There is no durable checkpoint or restart-recovery demonstration. Once IsComplete is set, that worker stops scanning; newly created legacy-only records are not automatically revisited. A completed scan does not establish that all writers have switched.

A fallback read can repair storage without repairing this response

ReadFallback prefers modern data. If the modern list is empty and the legacy pointer exists, it sets the scalar from legacy and performs a lazy SavePhoneEntry.

However, the local phones slice was loaded before that write and is not reloaded afterward. The first fallback response can contain the correct scalar and an empty array, even though storage contains the new entry by the time the method returns. GetModernUser can therefore receive an empty array on that first read.

TestFallbackRead checks the scalar and inspects storage after the call. It does not assert the first response's array. That difference is exactly why reading assertions matters more than inferring guarantees from a test name.

Reconciliation detects drift; it does not fix it

ReconcileData compares a populated legacy phone with the modern primary number. Missing entries and mismatched primary numbers count as drift. It skips users whose legacy pointer is nil and returns immediately after contract has been applied.

The function returns the number of findings in that scan. DriftDetected, by contrast, is cumulative: rescanning the same mismatch increments it again. A clean current scan and a nonzero historical counter can both be correct.

There is no automatic repair or periodic scheduler. The demo and tests invoke reconciliation explicitly. Zero drift within this comparison is not proof that every aspect of modern data is correct.

Rollback safety depends on write history

TestRollbackScenarios contrasts two states:

Record creation mode	Result after returning to legacy reads
Dual-write	The primary phone remains visible.
New-only	The scalar is empty, without a read error.

The modern number may still exist. What fails is visibility through the old representation. Restoring an old application mode does not populate data that the old reader needs.

This rollback boundary appears before the column is dropped: merely retaining a legacy column is insufficient if new records have stopped writing it. The demo changes flags in one process; it does not deploy and roll back separate binaries against a shared database.

Contract needs evidence that the current guard does not gather

The initial guard in ApplyContract is:

// service.go: ApplyContract excerpt
if !force && s.obs.LegacyReadHits.Load() > 0 {
    return fmt.Errorf("%w: recorded %d legacy reads",
        ErrContractViolation, s.obs.LegacyReadHits.Load())
}

LegacyReadHits is a cumulative counter, not a traffic rate or observation window. The guard has no managed reset or time window. GetLegacyUser increments it before checking errors, so failed attempts count too.

The demo calls ApplyContract(true), bypassing this guard. That does not prove a zero-traffic period. Even without force, a zero counter can allow contract before historical data has been backfilled: the method does not check worker completion, drift, or old writers.

It then changes write mode, changes read mode, marks contract applied, and clears legacy pointers in storage. Those steps do not form one atomic service-wide transition.

After contract, legacy service reads return an error and the V1 handler responds with 410 Gone. But the enriched UserResponse type still has Phone, and new-only reads can populate it from modern entries. This is retirement of the legacy route and storage value, not removal of the enriched field from source code.

Do not promote illustrative SQL or headers into production guarantees

The example SQL keeps users.phone TEXT NOT NULL during expansion. The in-memory CreateModern creates users with a nil legacy phone. The SQL file does not implement the complete sequencing needed to reconcile new-only inserts with the still-present database constraint.

No test applies the DDL to a database or measures database locks. The example is not evidence of an online, zero-downtime schema migration.

The V1 handler emits hardcoded Deprecation and Sunset values. Its test checks Deprecation == "true" and a nonempty Sunset, not standards compliance or time-triggered retirement. 410 follows ApplyContract, not a sunset scheduler. This article therefore makes no RFC-compliance claim for those values.

Read concurrency coverage narrowly

The concurrency test overlaps creates, legacy/modern reads, backfill, and reconciliation. Many returned errors are ignored, and its final explicit assertion checks only DualWriteErrors.

That does not establish final consistency of every record or safe concurrent contract transitions. Running a race detector is valuable evidence for executed paths, but is not a substitute for semantic assertions about the migration.

To reproduce the lab from its own module directory:

cd labs/13-backward-compatibility
go test -v ./...
go test -race ./...
go run ./cmd/demo

The module declares Go 1.26.7. Go is unavailable in this authoring environment, so I inspected the source and tests rather than executing them. Execution reports committed in the repository are historical author reports, not a fresh run performed for this article.

Preserve the transition, not just the destination

Expand keeps both representations available. Migrate moves writes, historical data, and consumers with explicit checks. Contract retires the old path after its dependencies are actually gone.

The pattern supplies an order of work, not an automatic safety guarantee. Compatibility still depends on the real reader behavior, write history, checkpoint durability, and retirement gates implemented along that path.

Source

Software Engineering Lab 13, commit 82af988b89439fa92b1200406c5afbd59ce72a56: internal/compat/*.go, tests/*.go, cmd/demo/main.go, README.md, schema.sql, and go.mod. Code excerpts belong in their existing package context.

Feature Flags: Deploy Code, Control the Release

lukman lukman — Mon, 12 Oct 2026 10:27:00 +0000

The new online-booking flow is ready to deploy. Does that mean every workshop customer must use it immediately?

Lab 12 separates those decisions. Deployment puts code in place; release exposes its behavior to users. The online_booking flag controls that exposure through internal targeting, percentage rollout, and a global OFF switch.

Deploy does not have to mean release.

The lab is a small Go simulation, not a production flag platform. Reading its boundaries is part of understanding what a feature flag actually guarantees.

Keep the decision at a clear boundary

FeatureService owns flag configuration and evaluation. BookingService.CreateBooking makes one routing decision rather than scattering targeting rules throughout business logic:

// booking_service.go: complete method
func (s *BookingService) CreateBooking(req BookingRequest) BookingResponse {
    if s.featureService.IsEnabled("online_booking", req.UserID) {
        return s.newBookingFlow(req)
    }
    return s.legacyBookingFlow(req)
}

Both flows return simulated responses. The new flow succeeds unless simulateFail is true; the legacy flow always succeeds. There is no database write, mechanic reservation, or real vendor call. Vendor Error (Simulated) is deliberately injected behavior.

This distinction matters: the lab demonstrates choosing a path, not implementing the booking transaction itself.

Evaluation order is part of the contract

IsEnabled applies rules in this order:

Order	Rule
1	Unknown flag returns false.
2	`Enabled == false` returns false for everyone.
3	A matching ID in `InternalUsers` returns true.
4	Percentage <= 0 returns false for ordinary users; >= 100 returns true.
5	For percentages 1–99, compare the user's bucket with the threshold.

Global OFF wins over internal targeting. Conversely, Enabled: true with a zero percentage can release to internal users while leaving ordinary users on legacy.

The User struct has an IsInternal field, but evaluation does not use it. Internal access comes from matching userID against the configured list. Do not infer targeting behavior from a field that is never read.

The percentage comment says 0–100, but there is no rejecting validation. Negative values behave like zero; values above 100 behave like 100 for ordinary users.

Deterministic cohorts, not random requests

Randomizing every request could send the same user through different flows on consecutive calls. The implemented calculation is deterministic:

// feature_service.go: bucketing excerpt
hashInput := fmt.Sprintf("%s:%s", userID, featureKey)
hash := sha256.Sum256([]byte(hashInput))
bucketVal := binary.BigEndian.Uint64(hash[:8])
bucket := int(bucketVal % 100)
return bucket < flag.RolloutPercentage

The exact input includes a colon separator. SHA-256 produces the digest; the first eight bytes are interpreted as a big-endian uint64; modulo 100 gives a bucket from 0 to 99.

At 10%, buckets 0–9 qualify. At 50%, buckets 0–49 qualify. With the same IDs, feature key, algorithm, and targeting rules, the original 10% cohort remains included as the threshold increases.

For online_booking, recalculating the source algorithm puts user-3 in bucket 25 and user-4 in bucket 83. The former enters at 50%; the latter remains excluded until a higher threshold or the 100% shortcut. These are derived examples, not recorded Go execution output.

Consistency is conditional. Changing configuration is supposed to change eligibility. Changing the user ID or feature key can also change the bucket.

A 10% threshold does not promise exactly ten requests

The CLI generates 100 users: three internal IDs, then user-3 through user-99. It issues one request per user in each phase. Internal users remain allowlisted through the percentage phases.

Re-evaluating this exact fixture with the source hash algorithm gives:

Phase	New flow	Legacy
OFF	0	100
Internal only	3	97
10% plus internal allowlist	15	85
50% plus internal allowlist	48	52
100%	100	0

I calculated these counts in Python from the repository's fixture and algorithm; I did not run the Go demo. At 10%, 12 of the 97 ordinary users qualify, plus all three internal users. At 50%, 45 ordinary users qualify, plus the three internal users.

Two effects explain the difference from the threshold label: a finite sample need not contain exactly 10% of users in buckets 0–9, and internal targeting overrides the percentage gate.

User exposure is also not necessarily traffic exposure. Users can send unequal request volumes. The demo makes those quantities align by sending exactly one request per user per phase. A 10% cohort is therefore not a hard ceiling on every possible production impact.

Observe before increasing exposure

The README recommends OFF, internal, 10%, 50%, and 100%, with observation between increases. The CLI demonstrates that sequence, but its transitions are hardcoded. It does not wait for an observation window or inspect a threshold before advancing.

That makes it an illustration of rollout stages, not an automatic rollout controller.

The runner's phase function records metrics after calling CreateBooking:

Total requests.
New-flow and legacy-flow request counts.
Aggregate booking successes and failures.
Response-time samples and their mean.

Although BookingService accepts a metrics pointer, CreateBooking does not update those counters. Calling the service directly does not provide the CLI's instrumentation automatically.

The error-rate formula in Metrics.Snapshot is:

// metrics.go: struct-literal field excerpt
ErrorRate: float64(m.BookingFailed) /
    float64(max(1, m.TotalRequests)) * 100,

The denominator is all requests, not just requests using the new flow. During a partial rollout, aggregate error rate can dilute a severe failure rate inside the new cohort. The current metrics split request volume by flow but combine success/failure outcomes. Per-flow error rates would be an extension, not an existing feature.

Metrics reset at each phase. AvgResponseTime is a mean, not a percentile. It measures a local simulated method with no database or network work, so its latency should not be presented as a production booking benchmark.

A kill switch changes subsequent decisions

After reaching 100%, the CLI sets simulateFail to true. Every request in phase 6a selects the new flow and fails. Then it applies this configuration:

// cmd/simulate/main.go: kill-switch configuration
service.SetFlag(featureflags.Flag{
    Key:     "online_booking",
    Enabled: false,
})

Requests in phase 6b select legacy and succeed, even though the failure simulation remains enabled. Turning off the feature avoids the failing path on subsequent evaluations that read the updated configuration.

It does not retry an already failed request through legacy. It does not cancel an evaluation that already read the old configuration, undo completed side effects, or remove deployed code. The distinction between rerouting future work and repairing past work is essential to the kill-switch claim.

Runtime configuration has an operational boundary

FeatureService stores flags in an in-memory map protected by an RWMutex. SetFlag replaces the entire Flag value. It is not a partial-field update: leaving out RolloutPercentage and InternalUsers resets them to their zero values.

The CLI sends complete targeting configuration during rollout. A caller that wants to preserve targeting must do the same rather than assume omitted fields remain unchanged.

There is no persistent configuration store, cross-replica distribution, operator dashboard, or administration endpoint. “Without redeploy” means updating the object in this running process. The map's mutex also does not establish concurrency safety for every other field: simulateFail is a plain bool, and toggling it concurrently with booking reads is not synchronized by this implementation. The demo itself runs sequentially.

A flag is not authorization

Authorization asks whether the user is allowed to create a booking. The feature flag asks which booking behavior is selected for that user.

The README explicitly distinguishes these questions. This implementation answers the second one; it does not implement an authorization check. An internal allowlist for release targeting should not be described as a permission system.

Test the paths, then remove the temporary branch

The test file covers global OFF, zero percentage, internal inclusion, examples on both sides of a 10% gate, repeated evaluation, increased rollout, kill switch, and unknown flags. Booking tests check ON/OFF responses and a simulated failure followed by an OFF update and a successful legacy response.

The percentage test searches for one ON user and one OFF user. It does not prove an exact 10% distribution. There are no metric assertions or concurrent execution tests in this file. Test names should not be read as broader guarantees than their assertions provide.

Use the repository's Makefile targets:

# From the repository root
make lab-12-demo
make lab-12-test

# Or enter the nested lab module
cd labs/12-feature-flags
go run ./cmd/simulate
go test -v -count=1 ./...

The lab's own go.mod declares Go 1.26.7. Entering that module matches the Makefile; the README's root-relative go run alternative needs to account for the nested-module boundary. Go is unavailable in this authoring environment, so I inspected the tests and demo rather than executing them. No passing test result is claimed.

Finally, the README's lifecycle continues beyond 100%: observe stability, remove the flag, then remove unused legacy code. That cleanup is a recommended future change, not a refactor already performed in this lab.

A release flag is useful because it separates code delivery from user exposure. Its value depends on consistent targeting, meaningful evidence, tested fallback behavior, and a plan to retire the extra branch.

Source

Software Engineering Lab 12, inspected at commit 82af988b89439fa92b1200406c5afbd59ce72a56: the README, feature service, booking service, metrics, tests, CLI simulation, lab go.mod, and Lab 12 targets in the root Makefile. Code excerpts belong in their existing package context.

Production Debugging: Test Hypotheses, Not Guesses

lukman lukman — Sun, 11 Oct 2026 10:14:00 +0000

A purchase-order request for a one-year range is slow. The README frames the incident as a 504 Gateway Timeout. Is the handler serializing too much JSON? Is the service doing expensive work? Is the repository slow?

Lab 11 turns those guesses into an investigation. Its most useful lesson is not simply “measure every layer.” It is understand what each measurement includes before using it as evidence.

Debugging is testing hypotheses, not guessing causes.

The scenario and the implementation are different things

The lab contains README.md, main.go, and main_test.go. The implementation is an in-process Go simulation: a handler calls a service, which calls a repository. Tests invoke HandleGetOrders directly.

There is no HTTP server, load balancer, database query, or JSON serialization in these files. The 504 is the motivating scenario, not an HTTP response generated by the tests. Missing indexes and inefficient queries are analogies in the README, not demonstrated database findings.

Here is the complete slow repository method from main.go, formatted for readability:

func (r *SlowRepository) FetchOrders(
    ctx context.Context, rangeType string,
) ([]PurchaseOrder, error) {
    if rangeType == "1y" {
        time.Sleep(4 * time.Second)
    } else {
        time.Sleep(10 * time.Millisecond)
    }
    return []PurchaseOrder{{ID: "po-1", Amount: 100}}, nil
}

The one-year input selects a four-second sleep. Every other input selects a ten-millisecond sleep. Both paths return one order, not different-sized datasets. There is no date filtering or range validation. This matters: the experiment models latency, not the cost of retrieving a larger result set.

Reproduce one specific condition

TestReproduceBug wires SlowRepository into the service and calls the handler with "1y". It fails when TotalDuration < 4*time.Second. Precisely, the assertion accepts a duration of at least four seconds; the failure message's > 4s is slightly looser wording.

From the repository root, the README's workflow is:

cd labs/11-debugging-production
go test -v -run TestReproduceBug
go test -v -run TestCollectEvidence
go test -v -run TestVerifyFix

The root go.mod specifies Go 1.25.0. Go is unavailable in the environment used to prepare this article, so I inspected these tests rather than running them. The timing arithmetic below is derived from source, not benchmark output or a claim that tests passed.

Give each hypothesis a falsifiable question

README hypothesis	What the evidence must distinguish
H1: handler / JSON work	Handler-local work versus time spent downstream; JSON is not implemented here.
H2: service business logic	Service-local work versus waiting inside the repository call.
H3: repository	Whether changing the repository while holding the other parts fixed removes the modeled delay.

A slow service measurement does not automatically support H2. A caller's timer can include its dependency. Similarly, a small preparation timer cannot establish that JSON serialization is fast when serialization never occurs.

Service time is inclusive

This is the complete Service.GetOrders method, with its signature wrapped:

func (s *Service) GetOrders(
    ctx context.Context, rangeType string,
) ([]PurchaseOrder, time.Duration, error) {
    start := time.Now()
    time.Sleep(10 * time.Millisecond)
    orders, err := s.repo.FetchOrders(ctx, rangeType)
    dur := time.Since(start)
    return orders, dur, err
}

The timer starts before the service's simulated work and stops after FetchOrders returns. ServiceDuration therefore includes repository time. With the slow repository, the configured sleep contribution is 10 ms + 4000 ms, not 10 ms alone.

The handler then derives repository time using this excerpt:

totalDur := time.Since(totalStart)
repoDur := svcDur - 10*time.Millisecond

That is a subtraction, not an independently measured repository span. Any excess time in service scheduling or other overhead can remain in the derived value. Calling the field RepositoryDuration does not change its measurement boundary.

Field	Actual meaning in the code
`HandlerDuration`	Handler preparation only, including a 5 ms sleep.
`ServiceDuration`	Inclusive service work plus the repository call.
`RepositoryDuration`	Service duration minus a fixed 10 ms.
`TotalDuration`	Elapsed time across the successful handler path.

Do not add handler, service, and repository durations together. Service already includes repository time, so that would double-count the repository. The general habit is to ask where every timer starts and stops before comparing its value with another timer.

The example log is not output from this implementation

The README shows service_duration=12ms alongside repository_duration=4200ms. That pairing is inconsistent with the current calculation: repository duration equals service duration minus 10 ms. It cannot be treated as actual output from this code.

The configured sleep budgets are:

Path	Sum of configured sleeps
Slow repository, `1y`	5 + 10 + 4000 = 4015 ms
Slow repository, other input	5 + 10 + 10 = 25 ms
Fast repository, `1y`	5 + 10 + 10 = 25 ms

These are source-derived sums, not measured runtimes. Scheduling and overhead affect real elapsed time. The README's claim that 1m takes less than 20 ms also conflicts with the 25 ms of configured sleeps in the current code.

TestCollectEvidence prints the duration fields but does not assert a bottleneck relationship. Its request_id=abc123 line is hardcoded; it is not a unique-ID generator or a tracing implementation. Printing evidence and automatically verifying an interpretation are different operations.

Change one variable

TestVerifyFix replaces SlowRepository with FastRepository while retaining the same handler, service, and "1y" input. The fast repository always sleeps for 10 ms and returns the same one-order payload.

This controlled replacement isolates the artificial repository delay. It does not demonstrate a SQL optimization or a production deployment. Both implementations visibly return the same data, but the tests do not assert payload correctness.

The verification assertion from main_test.go is:

if res.TotalDuration > 100*time.Millisecond {
    t.Fatalf("expected request to be fast (< 100ms), but took %v", res.TotalDuration)
}

The condition accepts a total duration of at most 100 ms, including exactly 100 ms. That is a lab test threshold, not a production SLO or latency guarantee. Because these are wall-clock tests, execution conditions can influence the result.

Keep the conclusion inside the evidence

All three tests use "1y" and context.Background(). They do not test the 1m path, returned order contents, HTTP status codes, concurrent load, or cancellation.

Although context is passed through the call chain, neither repository checks it, and the sleeps are not cancellation-aware. A context parameter alone does not establish timeout behavior.

The error path also deserves careful reading. The service returns the repository error, but the handler returns an empty RequestResult on error. Timing is therefore not returned on that path. Both provided repositories always return a nil error, and no test injects a failing repository. These are possible follow-up checks, not existing capabilities.

The reusable debugging loop

Define the exact condition you can reproduce.
Separate implementation facts, documentation examples, and assumptions.
Write hypotheses with evidence that could refute them.
Check the boundaries and overlap of your measurements.
Change one variable and verify against an explicit criterion.
State what the experiment did not establish.

For this lab, the source identifies a four-second artificial repository delay. That is a useful, bounded conclusion. Claiming a missing production index would go beyond the evidence.

The point is not to sound certain sooner. It is to make each conclusion traceable to the code, measurement, or controlled change that supports it.

Source

Software Engineering Lab 11: README.md, main.go, and main_test.go, inspected at commit 82af988b89439fa92b1200406c5afbd59ce72a56. The repository root supplies go.mod. Code excerpts belong in the existing lab package; they are not standalone programs.

Project Estimation: Eight Screens Are Not Eight Days of Work

lukman lukman — Fri, 09 Oct 2026 10:36:00 +0000

An application has eight features. Assigning one day to each produces an eight-day estimate, but says nothing about webhook behavior, sandbox access, booking validation, or the engineer's actual availability.

Lab 10 replaces that shortcut with a small Go estimation model. Its case study is a vehicle-service booking application: login, online booking, branch selection, mechanic selection, payment, WhatsApp notifications, an admin dashboard, and Excel reports. The repository implements estimation arithmetic and validation—not those application features or vendor integrations.

The important result is not a more convincing single number. It is an explanation of scope, effort, uncertainty, dependencies, assumptions, and the capacity needed to turn effort into elapsed working time.

Page count hides the work

The naive estimator deliberately returns one day per page:

// estimation.go; complete function
func EstimateByPageCount(pageCount int) int {
    if pageCount <= 0 {
        return 0
    }
    return pageCount
}

TestNaiveEstimator expects ten days for ten pages. That assertion confirms the function's behavior; it does not establish that a ten-page project can be delivered in ten days.

The README's payment breakdown explains what page count misses: documentation discovery, authentication, transaction creation, callback verification, error paths, retry/idempotency, and sandbox testing. A screen is an interface boundary, not a unit of implementation effort.

Similarly, branch selection involves master data and a selector, mechanic selection needs relationships and availability checks, and Excel reporting includes filtered data and file generation. These are planning examples in the README, not implemented booking endpoints.

Make the estimation unit explicit

Each executable planning task carries a name, a three-point effort range, a risk label, spike effort, and assumptions:

// estimation.go; complete type definitions
type EstimateRange struct {
    Min        float64
    MostLikely float64
    Max        float64
}

type Task struct {
    Name        string
    Estimate    EstimateRange
    Risk        RiskLevel
    SpikeDays   float64
    Assumptions []string
}

Project.Estimate combines the tasks directly attached to the project with tasks inside each feature. Feature names organize the input; the calculation flattens their tasks into one collection. TestFeatureBreakdownAffectsTotal checks that three feature tasks contribute six minimum implementation days and a task count of three.

There is no task identity or deduplication. Listing the same work in both collections counts it twice. Nor does the model store task-to-task dependencies or create a schedule from feature order.

Unknown work needs a bounded investigation

The README distinguishes familiar work from work whose authentication, webhook behavior, or vendor environment is not understood. Unknown does not mean zero effort. It also should not become an unsupported implementation guess.

A spike is timeboxed discovery with a specific output: enough evidence to identify and estimate the implementation tasks. The payment example investigates authentication, transaction creation, signature verification, sandbox behavior, and failure/retry handling. It is not a production integration delivered inside the discovery timebox.

The actual validation rule is strict:

// estimation.go; excerpt from Task.Validate
if t.Risk == RiskUnknown && t.SpikeDays == 0 {
    return fmt.Errorf("task %q: %w", t.Name, ErrUnknownRiskNoSpike)
}

Every RiskUnknown task must have positive SpikeDays, even when its implementation range is nonzero. TestUnknownRiskWithoutSpikeRejected supplies a 1/2/3 estimate with no spike and expects rejection. TestUnknownRiskWithSpikeAndExplicitEstimateAccepted supplies a positive spike and expects acceptance.

This contradicts the README's failure-scenario row claiming that an unknown task with zero spike and a nonzero estimate is allowed after discovery. The executable model has no completed-spike flag. Once discovery is complete, keeping RiskUnknown while setting SpikeDays to zero still fails validation; a re-estimated task needs an appropriate supported risk classification.

Also, RequiredSpikes includes any task with positive spike days, not just unknown-risk tasks. Spike time is counted as fixed effort in all three range positions.

A weighted estimate is not a deadline probability

// estimation.go; complete function
func (r EstimateRange) Expected() float64 {
    return (r.Min + 4.0*r.MostLikely + r.Max) / 6.0
}

The middle input gets four times the weight of each endpoint. The model validates nonnegative values and Min <= MostLikely <= Max, then sums minimum, weighted expected, and maximum values across tasks.

The word Expected is an output name for this weighted calculation. There is no sampled distribution, Monte Carlo simulation, percentile, or historical calibration. A maximum estimate is not a promised latest delivery date, and the output range is not a statistical confidence interval.

For the arithmetic below, the source snapshot is commit 82af988b89439fa92b1200406c5afbd59ce72a56. Go is unavailable in this environment, so the tests were inspected but not executed. The numbers are independently checked arithmetic derived from the fixture and formulas, not captured Go test output.

Work through the booking fixture

TestBookingServiceCaseStudy has six direct tasks and two tasks inside an admin feature:

Task	Min	Most likely	Max	Risk	Spike days
Login	1	2	3	Low	0
Booking Online	3	5	8	Medium	0
Pilih Cabang	1	2	4	Low	0
Pilih Mekanik	2	3	5	Medium	0
Payment Gateway	4	6	12	Unknown	2
WhatsApp Notification	1	2	4	Unknown	0.5
Dashboard Admin	3	5	7	Medium	0
Laporan Excel	2	4	6	High	0

Task estimates use engineer-days. These are illustrative fixture inputs, not measured productivity or a quotation for a real booking project.

The minimum values sum to 17, the most-likely inputs sum to 29, and the maximum values sum to 49. The sum of weighted expectations is (17 + 4 × 29 + 49) / 6 = 30.333…, rounded by the implementation to 30.33.

Adding the two fixed spikes gives 2.5 engineer-days and a base range of 19.50 / 32.83 / 51.50. This is one complete fixture calculation; it should not be combined with the separate reference example in the README, which uses different task estimates and spike lengths.

The fixture does not contain separate testing/UAT or deployment tasks. Some README breakdowns include testing inside a task, but the fixture does not specify how much of those activities its ranges cover. The model cannot detect omitted work.

Risk controls a heuristic, not external reality

Project risk is derived from task counts:

Any unknown task makes overall risk High.
Otherwise, at least 30% High tasks makes overall risk High.
Otherwise, any High or Medium task makes overall risk Medium.
Otherwise, it is Low.

This is not effort-weighted: a short unknown investigation can elevate the whole project. The booking fixture is High because it contains unknown tasks.

Automatic contingency applies only when AutoContingency is true and the supplied rate is zero. It selects 25% for High, 15% for Medium, and 10% for Low. A positive explicit rate takes precedence, even if automatic selection is enabled. A zero rate with automatic selection disabled means no contingency.

The defaults are teaching choices, not universal planning rules. The README's broader percentage ranges should not be presented as additional executable settings.

Contingency is applied to every endpoint

The model computes ContingencyEffort as base expected effort times the rate. It scales minimum and maximum by the same percentage, rather than adding that single expected contingency amount to both endpoints:

// estimation.go; excerpt from Project.Estimate
contingencyDays := baseRange.ExpectedDays * contingencyRate

For this fixture, 25% produces expected contingency of 8.2075 engineer-days and a rounded final range of 24.38 / 41.04 / 64.38 engineer-days. ContingencyEffort itself is not rounded in the stored result; formatted output may display fewer decimals.

This is an explicit assumption about risk buffering. It does not discover vendor delays, validate assumptions, or guarantee that the chosen buffer is sufficient.

Effort becomes duration through capacity

// estimation.go; excerpt from Project.Estimate
effectiveDailyCapacity := float64(p.EngineerCount) * p.Availability

Each final effort endpoint is divided by this capacity and rounded to one decimal. One engineer at 70% availability gives 0.7 engineer-days of modeled daily capacity. For the fixture, that produces 34.8 / 58.6 / 92.0 working days.

DurationRange.Weeks divides each duration by five. The endpoint range is therefore approximately 7.0–18.4 five-working-day weeks, not literal calendar days and not a date-aware schedule. Holidays, vendor waiting, dependency ordering, and the critical path are not represented.

TestMultipleEngineersReducesCalendar checks that the formula gives a smaller duration with two engineers. It does not prove that every real task can be split evenly or that a project finishes twice as fast. The linear capacity formula assumes that the modeled effort can use that capacity.

The README's separate reference example says 15–23 engineer-days with one engineer at 70%, yet mentions three to four weeks and converts only the lower endpoint. Dividing both endpoints yields approximately 21.4–32.9 working days, or 4.3–6.6 five-working-day weeks. The executable fixture above uses different inputs; neither example should be quoted as a verified delivery commitment.

Confidence is a communication label

calculateConfidence returns Low for any unknown task or High overall risk. Medium risk returns Medium. Low risk with at least one assumption returns High; without assumptions it returns Low.

The function does not inspect whether assumption text is true, specific, or meaningful. contingencyRate is passed into it but is unused. Adding buffer therefore does not increase confidence under this implementation. The booking fixture remains Low confidence despite its assumptions and contingency.

TestAssumptionsIncreaseConfidence and TestMissingAssumptionsReduceConfidence exercise the low-risk label behavior. They do not establish a statistical probability of meeting a deadline.

Validation is useful, but bounded

Tests cover invalid range order, negative effort/spike/contingency, unsupported risk labels, empty projects, empty task names, invalid availability, and nonpositive engineer counts. Other tests check feature aggregation, range ordering, risk labels, spike accounting, and the distinction between effort and duration.

Several validations only assert that an error exists. TestBookingServiceCaseStudy asserts High risk and at least two spikes, then logs calculated values; it does not pin every numeric endpoint to an expected constant.

The implementation checks an empty task name with Name == "", so whitespace-only names pass. Floating-point validation does not explicitly reject NaN or infinity. It has no dependency graph, waiting-time field, task deduplication, missing-scope detection, or automatic re-estimation after a spike. These are limits of the teaching model, not features silently provided by its result type.

Communicate the conditions with the number

For the fixture, an honest summary is: eight planning tasks, 2.5 engineer-days of spikes, 24.38–64.38 final engineer-days under the model's 25% contingency, and 34.8–92.0 modeled working days with one engineer at 70% availability. Overall risk is High and confidence is Low. Assumptions include final UI design, available vendor sandbox/documentation, accessible provider API, and no major scope changes.

Vendor credentials are dependencies. Unexpected webhook behavior and changing booking allocation rules are risks. A capacity formula does not automatically add time spent waiting for either.

An estimate is a range with assumptions, not a promise without conditions. The model is valuable when its arithmetic and limitations make those conditions visible.

Repository sources

Lab 10
README.md: scope, breakdown, risks, and communication examples.
estimation.go: types, validation, arithmetic, and heuristics.
estimation_test.go: fixtures and assertions.
go.mod: module and Go 1.25 requirement.

Code Review: When Checkout Success Still Breaks Inventory

lukman lukman — Wed, 07 Oct 2026 23:31:00 +0000

A checkout can return success after reducing stock without creating an order. It can also accept two purchases when only one unit exists. Lab 09 makes those risks concrete by comparing CheckoutNaive with CheckoutImproved and inspecting the assertions around both implementations.

The useful review unit is a change in behavior: which requirement it serves, what can fail, who is affected, and what evidence protects the fix. A transaction or an idempotency interface is meaningful only when its actual boundary addresses that failure.

Start with requirements and observable behavior

For this checkout, the important requirements are that stock stays nonnegative, stock reservation and order creation succeed together, a retry does not create another order, and a principal can only check out their own cart.

Review in this order: requirement → changed behavior → failure mode → impact → simplest sufficient fix → test evidence. Naming and extraction suggestions can wait when inventory integrity or authorization is at risk.

All technical observations below come from Lab 09, inspected at commit 82af988b89439fa92b1200406c5afbd59ce72a56. Go is unavailable in this content-generation environment; the tests were read, not executed. Described outcomes are assertions, not recorded run results. Go excerpts are copied from the source; surrounding code is omitted where indicated. The README's PHP example is illustrative pseudocode, not the executable implementation.

Finding 1: success hides a partial checkout

Code

The naive service updates stock inside its product loop and then ignores the result of order creation:

// checkout_naive.go; excerpt, surrounding code omitted
_ = c.repo.CreateOrder(ctx, order)

_ = c.notify.SendOrderConfirmation(ctx, userID, orderID)

return &CheckoutResponse{Success: true, OrderID: orderID, Total: total}, nil

Failure mode and impact

If order creation fails, previously applied stock deductions remain. The caller receives success and a notification is attempted for an order that may not exist. This is a correctness issue, not merely an error-message problem.

The actual naive code does propagate GetCart failure. It also skips an item when GetProduct fails; it does not dereference a nil product on that branch. Yet toOrderItems still includes all cart items, while the total only includes items that passed product lookup. That creates another inconsistency the abbreviated README example does not explain.

Fix and test

The improved version reserves every item and creates the order inside one callback:

// checkout_improved.go; excerpt
 txErr := c.txManager.WithinTransaction(ctx, func(tx CheckoutTx) error {
    for _, item := range cartItems {
        if reserveErr := tx.ReserveStock(ctx, item.ProductID, item.Quantity); reserveErr != nil {
            return reserveErr
        }
    }
    if createErr := tx.CreateOrder(ctx, order); createErr != nil {
        return createErr
    }
    return nil
})

TestImprovedCheckout_IdempotencyFailureCanRetry asserts that an injected order-creation failure leaves stock unchanged, creates no order, sends no notification, and permits a later retry. TestImprovedCheckout_MultiProductFailureRollsBackEverything checks that both product stocks retain their original values when order creation fails.

Those assertions describe the mock transaction contract. They do not validate a database adapter, because the lab contains no production database transaction implementation.

Finding 2: synchronized memory can still oversell

Code

// checkout_naive.go; excerpt
product.Stock -= item.Quantity
_, _ = c.products.UpdateStock(ctx, item.ProductID, -item.Quantity)

In MockProductRepository, GetProduct returns a new Product. Changing its Stock field does not persist that value. UpdateStock applies a delta under a mutex, but it has no condition preventing a negative result. Its error is ignored by the naive service.

Failure mode and impact

Two requests can each deduct one unit from an initial stock of one, leaving negative inventory. The business failure is overselling. This does not require an unsynchronized memory access: locks can protect each map access while still allowing an invalid business operation.

The barrier in TestNaiveCheckout_DeterministicallyDemonstratesOverselling makes both product reads occur before the requests proceed. Its assertions expect two logical successes and negative final stock. The naive path would also accept a sequential deduction larger than available stock; concurrency is not its only problem.

Fix and test

The improved service calls CheckoutTx.ReserveStock. The actual mock transaction stages an accumulated deduction per product, rejects insufficient stock, and applies deductions on commit. Repeated entries for the same product share that accumulated reservation.

The README supplies this production-oriented SQL illustration:

UPDATE products
SET stock = stock - $1
WHERE id = $2
  AND stock >= $1;

Its explanation requires checking affected rows. This SQL is documentation, not a database adapter implemented or exercised by the lab. BEGIN and COMMIT alone do not express the stock condition.

TestImprovedCheckout_ConcurrentStockReservationPreservesInvariant asserts one success, one failure, one order, and final stock zero for two buyers competing for one unit. TestImprovedCheckout_DuplicateProductInCartAccumulatesReservation checks both sufficient and insufficient accumulated reservations.

What the mock actually proves

MockTransactionManager holds one mutex per manager for the complete callback and commit. For requests sharing that manager, transactions are serialized. This is the global serialization boundary of the demonstration, not production database concurrency control.

go test -race ./... is relevant to unsafe concurrent memory access. It cannot establish the stock invariant by itself. Also, despite its name and comment, TestImprovedCheckout_DeterministicNoRace does not use the naive read hook on the improved path: improved calls GetProducts, which does not invoke that hook. Its outcome assertions remain useful, but it does not force the same read interleaving.

Finding 3: retries need a claim, not just a key

Code

// checkout_improved.go; excerpt
scopedKey := fmt.Sprintf("checkout:%s:%s", principal.UserID, cmd.IdempotencyKey)

hashRequest marshals CheckoutCommand and hashes it with SHA-256. That command contains only CartOwnerID and IdempotencyKey; it does not contain quantities, product IDs, prices, or a cart version.

Failure mode and impact

Without retry protection, repeating a checkout can deduct stock and create an order again. Hashing mutable cart contents would also make a completed retry depend on the cart's later state.

The improved service first checks for a completed record before loading the cart. If it finds a stored response, it returns that response immediately. Otherwise it loads and validates the cart, calculates the command hash, and calls Claim. Contrary to the README's broad wording, the atomic claim and hash comparison happen after cart loading; only the completed-record lookup happens before it.

Fix and test

The mock repository protects Claim with a mutex. A new key becomes PROCESSING. The same hash while processing returns ErrDuplicateRequest; a different hash returns ErrIdempotencyConflict; a completed matching claim returns the stored response. TestIdempotencyRepository_StateMachineDirect asserts those repository behaviors with explicit test hashes.

At the service level, TestImprovedCheckout_DuplicateRequestReturnsSameResult expects the same order ID and one order after two requests. TestImprovedCheckout_RetryAfterCartMutationReturnsOriginalResponse empties the cart between attempts and asserts replay without another stock deduction. The same-key concurrency test asserts one order, one stock deduction, and one notification; it does not require both callers to receive success.

The conflict boundary is narrower than it sounds

The early Get replay does not compare RequestHash. With the current command shape and ownership check, the same scoped key normally implies the same authorized command fields. Still, the service does not implement hash validation on every completed replay, and the hash cannot distinguish changed cart contents. If the command grows, the fast path needs review. The direct repository conflict test is not evidence that an expanded service payload would be checked correctly.

Failures before business commit generally release the claim so another attempt can run. Release can fail too. The transaction-error path joins that error with errors.Join; the missing-product-in-batch branch logs a release failure but returns only ErrProductNotFound. Error preservation is therefore not uniform across all branches.

Finding 4: authentication context must constrain cart access

Code

// checkout_improved.go; excerpt
if principal.UserID != cmd.CartOwnerID {
    c.logger.Error(ctx, "authorization failed", "userID", principal.UserID, "cartOwnerID", cmd.CartOwnerID)
    return nil, ErrForbidden
}

Failure mode, impact, fix, and test

Accepting a caller-selected cart owner without comparing it with the principal could permit checkout of another user's cart. Improved checks ownership before cart loading or replay. TestImprovedCheckout_CannotCheckoutAnotherUsersCart asserts forbidden access, unchanged stock, and no order.

Per-user key scoping solves a separate problem: two users submitting the same key string should not share an operation. TestImprovedCheckout_IdempotencyKeyIsScopedPerUser asserts two orders and the combined stock deduction.

The naive service only accepts a userID; it has no independent principal/owner comparison. There is no HTTP authentication middleware in this lab, so it does not establish how that identity is authenticated upstream. The equality check also does not validate that a principal ID is nonempty.

Finding 5: repeated lookup is a measurable call pattern

Code and failure mode

Naive calls GetProduct for each cart item. Improved collects unique product IDs and calls GetProducts once. Repeated product lookup is the N+1-style issue here; multiple stock updates are a separate write pattern.

Impact, fix, and test

Batch loading reduces repeated repository lookup work. TestNaiveCheckout_NPlusOneProductLookups expects three single-product calls for three items. TestImprovedCheckout_BatchLoadsProducts expects one batch call and zero single calls. TestImprovedCheckout_BatchLoadUsesUniqueProductIDs checks that repeated product entries produce two unique requested IDs in its fixture.

These counters measure repository calls, not database round trips or query latency. There is no benchmark result to report.

The improved total uses product.UnitPrice, whereas naive uses the cart's price. This changes the price source, but quantity positivity is the explicit validation; negative prices, multiplication overflow, and accumulated-total overflow are not guarded. The mock fixes product prices at 1000, and the batch-loading test does not assert pricing behavior.

Finding 6: commit is not the end of every operation

Code

// checkout_improved.go; excerpt
if markErr := c.idempotency.MarkCompleted(ctx, scopedKey, resp); markErr != nil {
    c.logger.Error(ctx, "failed to finalize idempotency record; business transaction was committed", "userID", principal.UserID, "idempotencyKey", scopedKey, "error", markErr.Error())
    return nil, fmt.Errorf("%w: %v", ErrIdempotencyFinalize, markErr)
}

Failure mode and impact

Business commit occurs before MarkCompleted. Finalization failure can therefore return an error after stock has been deducted and the order exists. The claim remains processing in the mock, and notification is not reached. This is an atomicity gap between two state transitions.

ErrIdempotencyFinalize is wrapped for errors.Is; the underlying markErr is formatted with %v, not wrapped as a second cause. This error must not be read as proof that checkout rolled back.

Fix boundary and test

TestImprovedCheckout_IdempotencyFinalizeFailureAfterCommit injects a finalization failure and asserts the sentinel error, stock 8 from an initial 10, one order, no notification, and no completed record. The README discusses a shared transactional boundary, reconciliation, and a unique business constraint as production options. Those options are not implemented here, nor is an Outbox.

After successful finalization, notification runs synchronously outside the transaction. Its error is logged without changing checkout success. A crash before delivery can lose the message, and completed replay does not resend it. The supplied notification mock always succeeds, so the test suite does not exercise a notification-send failure.

Contextual logger calls include user, key, order, and error information where applicable. MockLogger only stores messages, however, so these tests do not verify preservation of the structured fields. TestImprovedCheckout_PreservesRepositoryErrors checks that a batch infrastructure error is not mislabeled as ErrProductNotFound; it does not assert errors.Is(err, infraErr).

Rank the risk before polishing the code

Inventory corruption and cross-user access deserve attention before naming. Partial state, swallowed errors, duplicate execution, and post-commit ambiguity change what a client can safely assume. Batch loading matters too, but its urgency depends on workload and impact; a call count does not establish severity on its own.

The README's general findings include hardcoded environment configuration and duplicate validation code. Those are discussion topics, not verified concrete defects in the executable checkout. Several README test names are illustrative rather than the names present in checkout_test.go; the references above use the actual test functions.

The improved service is a teaching implementation with in-memory repositories. Its atomic.Int64 counter belongs to each service instance, so IDs are only unique within that instance's running counter; they can collide across instances or replicas and after restart. There is no implemented TTL/recovery mechanism for stale processing claims, database uniqueness constraint, durable notification delivery, or general exactly-once guarantee.

Code review reduces risk by protecting behavior and business invariants. A review is stronger when it connects a concrete branch to a failure, a bounded fix, and an assertion that would expose regression.

Source files

README.md: review framework and documented SQL illustration.
checkout_naive.go: intentionally broken checkout.
checkout_improved.go: service orchestration and remaining gaps.
domain.go: command, interfaces, and errors.
mock.go: in-memory transaction and idempotency semantics.
checkout_test.go: behavior assertions.
go.mod: module and Go 1.25 requirement.

Database Isolation: Why Two Committed Transfers Can Create Money

lukman lukman — Fri, 02 Oct 2026 07:16:37 +0000

Two wallet transfers can both commit successfully while creating money. In Lab 08, the problem is a specific query pattern: read a balance, validate it in Go, calculate a replacement value, and write that value later. A transaction boundary does not make the original read remain current.

The useful question is whether overlapping transactions preserve the business invariant. This lab approaches that question through a three-account wallet, PostgreSQL isolation experiments, and concurrent tests.

Start with what must remain true

Alice, Bob, and Charlie each begin with 1,000,000. A transfer should change the distribution of those balances while keeping their sum at 3,000,000. Every balance must also remain non-negative.

The schema implements the row-level invariant:

balance BIGINT NOT NULL CHECK (balance >= 0)

That constraint does not enforce conservation of money across accounts. A broken transfer can leave every individual balance positive while increasing their sum. Correctness therefore needs both checks.

TransferNaive uses sql.LevelReadCommitted. It begins a transaction, reads the sender, checks sufficient funds, reads the receiver, updates both accounts, inserts an audit record, and commits. These operations share an atomic boundary. Their decisions can still depend on stale application values.

Follow the overlap instead of the happy path

Transfer A sends 800,000 from Alice to Bob. Transfer B sends 800,000 from Alice to Charlie. Both transactions read Alice at 1,000,000 before either writes. Each concludes that the transfer is allowed.

Each then calculates Alice's replacement balance as 200,000. The naive sender update uses that calculated value rather than deriving the new balance from the current row:

// 4. Overwrite sender balance with calculated value (Lost Update vulnerability)
    _, err = tx.ExecContext(ctx, "UPDATE isolation_accounts SET balance = $1 WHERE id = $2", fromBalance-amount, fromID)
    if err != nil {
        return fmt.Errorf("update from balance: %w", err)
    }

Excerpt from transfer.go; surrounding validation, receiver update, audit, and commit are omitted.

The second write can wait for the first writer and still overwrite Alice with the same 200,000. Bob and Charlie receive separate credits of 800,000. The expected final state in TestNaiveTransfer_LostUpdate is Alice 200,000, Bob 1,800,000, Charlie 1,800,000: total 3,800,000.

The test orchestrates this overlap with channels. Both transactions signal that their reads have completed; the coordinator then releases their writes. It reproduces the read-calculate-write pattern explicitly, rather than calling TransferNaive directly. That distinction matters when describing what the test covers.

These numbers are assertions in the repository, not measurements from a test run for this article. No lab tests were executed during preparation.

Why the constraint does not reject it

Alice still has 200,000. There is no negative balance to reject. The missing debit is a lost update, and the violated invariant is the cross-account sum. A successful commit and a satisfied CHECK constraint are insufficient evidence that the transfer was correct.

This does not mean every READ COMMITTED update loses data. The vulnerable implementation reads, calculates in application memory, and later writes an absolute value. Its query pattern is part of the explanation.

Separate the SQL standard from PostgreSQL behavior

The README describes the standard anomaly ladder: READ UNCOMMITTED permits dirty reads; READ COMMITTED prevents dirty reads but permits non-repeatable and phantom reads; standard REPEATABLE READ prevents dirty and non-repeatable reads but may permit phantoms; SERIALIZABLE prevents those anomalies.

PostgreSQL provides stronger behavior in two important places. READ UNCOMMITTED behaves as READ COMMITTED and does not permit dirty reads. PostgreSQL REPEATABLE READ uses a stable snapshot and prevents the classic phantom read demonstrated by the lab.

PostgreSQL level	Visibility in this lab	Remaining concern
READ UNCOMMITTED	Same behavior as READ COMMITTED	Does not grant a stable transaction snapshot
READ COMMITTED	New snapshot per statement	Repeated reads may change; naive writes may lose updates
REPEATABLE READ	Snapshot acquired at the first read	Concurrent same-row update can fail with 40001
SERIALIZABLE	SSI with serial execution equivalence	Conflicts may abort a transaction; retry must be handled

The table describes the engine used by the repository. It should not be treated as a portable implementation table for other databases.

Read Committed: two statements, two views

TestReadCommitted_NonRepeatableRead coordinates a reader and a writer. The reader first sees Alice at 1,000,000. The writer adds 500,000 and commits. The reader's second SELECT, still inside the same transaction, is expected to see 1,500,000.

The change is committed, so this is not a dirty read. It is a non-repeatable read caused by statement snapshots.

The phantom experiment uses a predicate instead of a single account. Two invoices initially have status PAID. A writer inserts a third PAID invoice and commits between the reader's queries:

SELECT COUNT(*) FROM isolation_invoices WHERE status = 'PAID'

Under READ COMMITTED, the test expects counts of two and three. Under PostgreSQL REPEATABLE READ, it expects two and two. The predicate is unchanged; the visible set of matching rows differs only in the first strategy.

A stable snapshot is not exclusive access

PostgreSQL MVCC lets ordinary readers observe versions without taking a FOR UPDATE row lock. TestRepeatableRead_SnapshotIsolation expects both reads to return the original 1,000,000 even after another transaction commits an increase.

The concurrent-update experiment is different. Two REPEATABLE READ transactions read the same account, then try to deduct 800,000. Its assertions require exactly one success and one serialization failure, SQLSTATE 40001.

The application must therefore handle an aborted transaction. Choosing REPEATABLE READ does not make the second transfer silently succeed correctly. It changes the failure behavior.

The README also discusses write skew as a possible serialization anomaly under snapshot isolation. The supplied experiments demonstrate same-row update conflicts; they do not reproduce a separate write-skew scenario. Keep the conceptual warning separate from the experiment's actual evidence.

Row locking moves validation behind coordination

TransferWithLock stays at READ COMMITTED but obtains both account locks before validating the sender. Here is the locking excerpt:

firstID, secondID := DeterministicLockOrder(fromID, toID)

    // Lock both accounts deterministically
    var b1, b2 int64
    err = tx.QueryRowContext(ctx, "SELECT balance FROM isolation_accounts WHERE id = $1 FOR UPDATE", firstID).Scan(&b1)
    if err != nil {
        return fmt.Errorf("lock account %d: %w", firstID, err)
    }
    err = tx.QueryRowContext(ctx, "SELECT balance FROM isolation_accounts WHERE id = $1 FOR UPDATE", secondID).Scan(&b2)
    if err != nil {
        return fmt.Errorf("lock account %d: %w", secondID, err)
    }

Excerpt from transfer.go; the rest of the transfer is omitted.

A competing FOR UPDATE or UPDATE on those rows waits until the holder ends its transaction. An ordinary SELECT continues to use MVCC. The lock is coordination for the write path, not a promise that every reader is blocked.

After locking, the implementation identifies which locked balance belongs to the sender and checks sufficient funds. Debit and credit use arithmetic SQL updates, and the audit record is written before commit. The second competing transfer validates after obtaining access to the row, rather than trusting an earlier unlocked balance.

Acquire locks in a shared order

If A-to-B locks A first while B-to-A locks B first, the two transactions can create circular wait. The lab uses a helper independent of transfer direction:

func DeterministicLockOrder(id1, id2 int) (firstID, secondID int) {
    if id1 < id2 {
        return id1, id2
    }
    return id2, id1
}

Both directions acquire the smaller account ID, then the larger. The bidirectional test runs two workers with 50 transfers each. It checks errors and total balance. This order addresses the two-account lock pattern used here; it does not guarantee that arbitrary additional locking elsewhere can never deadlock.

Lock waits are a real cost. The README's phrase about deterministic latency should not be read as a guarantee from this implementation. Contention and transaction duration still affect waiting, and the repository provides no latency benchmark.

Serializable means serial equivalence, not a global queue

PostgreSQL SERIALIZABLE uses Serializable Snapshot Isolation. Concurrent transactions can run together; committed results must be equivalent to an allowed serial execution. Conflicts can require an abort.

TestSerializable_ConcurrentUpdate_SerializationFailure coordinates two same-row updates and expects one success plus one 40001. It is a conflict experiment, not a throughput comparison or proof of every possible invariant.

The recovery path needs a new transaction with fresh reads and validation. Retrying only the failed statement would not repeat the decision that caused it.

Retry the whole operation, with a limit

The wrapper passes a complete transfer as the operation:

func (r *PostgresWalletRepo) TransferSerializableWithRetryPolicy(ctx context.Context, fromID, toID int, amount int64, maxAttempts int, policy RetryPolicy) error {
    repo := r
    return RetryTransaction(ctx, maxAttempts, policy, func(ctx context.Context) error {
        return repo.TransferSerializable(ctx, fromID, toID, amount)
    })
}

Complete wrapper from transfer.go.

Each attempt calls TransferSerializable again, including BeginTx, read, validation, debit, credit, audit, and commit. Failed attempts defer rollback. The retry helper accepts only serialization failure 40001 or deadlock 40P01, through the PostgreSQL error or corresponding sentinel.

func IsRetryableTxError(err error) bool {
    if err == nil {
        return false
    }
    if errors.Is(err, ErrSerializationFailure) || errors.Is(err, ErrDeadlockDetected) {
        return true
    }
    var pqErr *pq.Error
    if errors.As(err, &pqErr) {
        return pqErr.Code == "40001" || pqErr.Code == "40P01"
    }
    return false
}

Complete filter from transfer.go.

WrapTxError preserves both sentinel matching with errors.Is and access to *pq.Error with errors.As. Constraint violations such as 23505 and 23503, generic errors, and insufficient funds are not automatically retried.

maxAttempts includes the initial attempt. Non-positive values are rejected. The default base delay is 10 milliseconds and the maximum is one second. Before another attempt, the helper computes an exponential upper bound, caps it, and multiplies it by a random value for full jitter.

Context cancellation is checked before attempts and can interrupt the default sleep. When retryable errors exhaust the limit, ErrMaxRetryExceeded wraps the last error. retry_test.go covers filtering, error preservation, success after a conflict, attempt exhaustion, cancellation, invalid limits, and delay behavior.

Test the invariant under contention

The stress fixture differs from the introductory wallet: three accounts begin at 50,000 each, giving total 150,000. One hundred goroutines attempt transfers of 1,000 using TransferWithLock after a shared start gate is released.

The test checks transfer errors, non-negative final balances, and the conserved sum of 150,000. A start gate creates competing work; it does not force every goroutine to reach every query at an identical moment. The naive test uses a more specific barrier after reads to reproduce its anomaly.

To run the supplied experiments, start the repository's PostgreSQL infrastructure and execute:

cd labs/08-database-isolation-level
go test -v ./...

The database helper skips integration tests when PostgreSQL is unavailable unless REQUIRE_POSTGRES=1 makes absence a failure. A test command that reports success alongside skipped experiments is not evidence that database behavior was verified.

Choose from the invariant outward

For this wallet's known rows, READ COMMITTED with FOR UPDATE and a shared lock order coordinates the critical read-check-write path. REPEATABLE READ serves a stable multi-query view but requires conflict handling for updates. SERIALIZABLE addresses serial equivalence at the cost of aborts, retry work, and dependency tracking.

The README mentions an atomic conditional update as an alternative, but the lab does not implement that strategy. It also offers no measured throughput or latency ranking. The practical conclusion is to specify the invariant, identify the relevant reads and writes, choose coordination or conflict detection, and verify the resulting state under overlap.

Repository and source map

Observability: Making an Invoice Request Explain Itself

lukman lukman — Wed, 30 Sep 2026 23:55:00 +0000

A successful response says the invoice workflow finished. It does not explain where the request spent its time. Lab 07 explores that gap through a Go service that loads an invoice, reserves inventory, calculates commission, generates a PDF, and sends a notification.

The useful question is not just “Is the endpoint working?” It is “What happened inside this request, and which dependency explains its latency or failure?”

One workflow, two levels of visibility

The unsafe service executes the same five dependency operations but exposes only a coarse completion duration or a generic failure log. Its HTTP response can still contain a request ID. What is missing is the service-level breakdown needed to investigate the request.

The safe service adds structured logs, Prometheus metrics, and OpenTelemetry spans around the workflow. This changes the evidence available during investigation; it does not make the business operations themselves faster.

The demo dependencies simulate work using configurable delays and errors. A delay waits on a timer while also listening for context cancellation. These are controlled diagnostic scenarios, not measurements of a real database, inventory service, or PDF engine.

Logs, metrics, and traces answer different questions

Logs describe discrete events. The safe service emits a structured event when a dependency completes, with its component, operation, duration, and outcome. On failure, the event also carries an error.

Metrics summarize many requests. Request counters describe traffic, duration histograms describe latency, error counters describe failures, and an in-flight gauge describes active requests. The gauge is a concurrency signal; it is not a complete measurement of CPU capacity or resource saturation.

Traces explain a particular execution. A server span contains an invoice-processing span, which contains one span for each dependency operation. That hierarchy connects the HTTP request to the work performed inside it.

A latency histogram can reveal a shift across requests. A trace can identify the slow operation in one request. A structured log can provide the event details at that operation. None replaces the other two.

Make the slow step visible

The demo's normal configuration assigns 20 ms to invoice loading, 15 ms to inventory, 10 ms to commission, 30 ms to PDF generation, and 20 ms to notification. The configured delays sum to approximately 95 ms before runtime and instrumentation overhead.

In the slow-PDF scenario, PDF generation receives a 4,800 ms delay. Keeping the other fake dependencies unchanged gives a configured total of approximately 4,865 ms. This is scenario arithmetic, not a benchmark result. An HTTP notification client also changes the execution path when enabled.

The point of the scenario is the contrast between “the request took several seconds” and “pdf.generate consumed most of the request.” A total-duration log provides the first statement. Dependency spans provide the second.

POST /invoices/{id}/process
  invoice.process
    database.load_invoice
    inventory.reserve
    commission.calculate
    pdf.generate
    notification.send

This is the hierarchy for the five-dependency workflow. A real HTTP notification call adds transport and downstream spans when those boundaries are instrumented.

Carry context through every dependency

The safe service starts a span for each dependency and passes that span's context into the dependency call. The context carries the causal relationship and cancellation signal together.

On a dependency failure, the implementation records the error on the dependency span and the invoice-processing span, marks them as failed, observes an error outcome, and returns a wrapped error identifying the component and operation. Processing stops at that failed step.

The HTTP handler then maps a processing error to a response: a deadline becomes 504, cancellation becomes 499, and other processing failures become 502. The response uses the generic message “invoice processing failed” and includes the request ID. Detailed failure information stays in telemetry.

These are mappings implemented by this lab, rather than a universal status-code policy.

Correlate events without turning IDs into metric labels

ContextLogger attaches request_id and, when an active span is valid, trace_id and span_id. The request ID helps connect the HTTP response to logs. The trace ID groups spans in one execution. The span ID identifies the active operation within that trace.

The correlation fields are useful in logs, but they are deliberately absent from metric labels. Request metrics use method, a route template, and status class. Dependency metrics use component, operation, and outcome.

The route label is /invoices/{id}/process, rather than a separate label value for each invoice URL. The registry tests also check that request IDs and invoice IDs do not leak into label values.

That distinction preserves aggregate metrics while leaving request-level investigation to logs and traces. The fixture tests check selected sensitive keys in emitted logs; they do not implement a general-purpose redaction system.

Read the metric names as an investigation map

The collector exposes six metric families:

Metric	What it records
lab07_http_requests_total	Requests by method, route, and status class
lab07_http_request_duration_seconds	Request duration observations
lab07_http_request_errors_total	Processing errors at the HTTP boundary
lab07_http_in_flight_requests	Currently active requests for the route
lab07_dependency_duration_seconds	Dependency duration by component, operation, and outcome
lab07_dependency_errors_total	Dependency failures

Histograms store duration observations in seconds. Structured dependency logs report duration_ms. Keep those units explicit when comparing the two.

The missing-ID branch records a 4xx request but does not increment the processing-error counter. The name “request errors” therefore needs to be read alongside the implementation's accounting rules.

Cross an HTTP boundary without losing the trace

The handler extracts incoming W3C trace context before starting its server span. The demo configures TraceContext and Baggage propagation globally.

For outgoing notifications, HTTPNotificationClient creates a request with the current context and uses an OpenTelemetry HTTP transport. The transport propagates trace context, while the client explicitly copies X-Request-ID from the context into the downstream request.

The default notification client has a five-second timeout. A supplied client is copied before its transport is wrapped, and an already instrumented transport is not wrapped again. Request cancellation still reaches the outgoing HTTP call through its context.

The local /notifications endpoint lets the demo exercise this HTTP boundary. It runs in the same demo process; this is not evidence of a deployed multi-service production system.

Exercise specific failure modes

The demo supports normal, slow-pdf, slow-database, slow-external, and pdf-error scenarios. They isolate different dependency delays or a PDF failure so the resulting telemetry can be inspected.

For example, with the demo and its configured telemetry services running:

curl -X POST   'http://localhost:8087/invoices/INV-007/process?scenario=slow-pdf'   -H 'X-Request-ID: lab07-slow-pdf'

The corresponding unsafe route is /unsafe/invoices/INV-007/process. Comparing the routes makes the difference in diagnostic detail visible.

Use a stable request ID to find the relevant logs, inspect the request's dependency spans, and compare request-level duration observations with dependency duration observations. The useful outcome is an explanation of the configured scenario, not merely confirmation that an endpoint returned a response.

Treat telemetry behavior as a contract

The tests inspect span hierarchy, shared trace IDs, incoming trace-context extraction, outgoing propagation, request-ID preservation, structured log fields, and metric accounting.

The fake-dependency hierarchy test expects seven ended spans: one HTTP server span, one invoice-processing span, and five dependency spans. The HTTP notification path has additional instrumentation and should not be forced into that seven-span expectation.

The slow-PDF test uses a shorter, 10 ms delay and checks the recorded PDF span. Concurrent-request tests check mixed success and error responses, matching counters, and an in-flight gauge that returns to zero.

There is also a test for response-encoding failures. Such failures are logged with correlation fields, but the implementation has already recorded success accounting before encoding the successful response. That limitation matters when interpreting metrics.

These observations come from reading the source and its assertions. The Go tests were not executed in this content-generation environment.

The mental model

Observability is useful when the evidence explains a request: what happened, where its time went, and which operation failed. This lab makes that idea concrete by connecting dependency events, aggregate measurements, and a causal trace through one invoice workflow.

A system should explain what happened, where time was spent, and why it was slow or failed.

Repository: https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/07-observability

API Versioning with Separate Invoice Contracts

lukman lukman — Wed, 30 Sep 2026 07:23:01 +0000

An invoice endpoint returns customer information as a string. A new representation replaces that string with an object containing the customer's ID, name, and phone number. The response remains valid JSON and the server still returns HTTP 200. The legacy consumer expects a string and cannot decode the new object.

Lab 06 of Software Engineering Lab demonstrates this change using Go HTTP handlers and an explicit consumer model. It then keeps the old representation on V1 and introduces the nested representation on V2. A third example adds a field without changing the fields that the legacy model already understands.

The tests described here are source-defined scenarios and assertions. They were inspected, not executed for this article. The legacy consumer is simulated by a Go struct; this is not an Android device test.

The original invoice contract

The documented invoice response contains four fields:

{
  "id": 1001,
  "customer": "Budi",
  "total": 500000,
  "status": "PAID"
}

domain.go makes the consumer's expectation concrete:

type LegacyInvoice struct {
    ID       int    `json:"id"`
    Customer string `json:"customer"`
    Total    int64  `json:"total"`
    Status   string `json:"status"`
}

The helper ParseLegacyInvoice calls json.Unmarshal into that struct. This turns compatibility into an operation that the tests can exercise. The question is whether the existing model can still consume the body and obtain the expected values.

The fixture uses invoice ID 1001, customer name Budi, total 500000, and status PAID. These are demonstration values, not a production invoice dataset.

The unsafe handler changes an existing field's type

UnsafeHandler returns the following customer representation while retaining the invoice's other fields:

{
  "id": 1001,
  "customer": {
    "id": 15,
    "name": "Budi",
    "phone": "08123"
  },
  "total": 500000,
  "status": "PAID"
}

The new information is useful, but it occupies a field whose published representation was a string. LegacyInvoice.Customer cannot receive this object through the supplied decoder.

TestBreakingChange_LegacyClientFails first requires HTTP 200, then calls ParseLegacyInvoice on the response body. It expects decoding to fail. A passing test in this example would confirm the deliberate incompatibility; it would not certify the handler as backward compatible.

The unsafe endpoint is synthetic. It creates its payload directly after validating the path, rather than retrieving the invoice through the repository used by V1 and V2. It demonstrates the shape change, not a complete persistence-backed invoice service.

One domain model, two response models

The internal Invoice model already contains a nested Customer. The public V1 response does not have to copy that shape.

In safe_server.go, the response types establish separate contracts:

type InvoiceV1Response struct {
    ID       int    `json:"id"`
    Customer string `json:"customer"`
    Total    int64  `json:"total"`
    Status   string `json:"status"`
}

type CustomerV2Response struct {
    ID    int    `json:"id"`
    Name  string `json:"name"`
    Phone string `json:"phone"`
}

type InvoiceV2Response struct {
    ID       int                `json:"id"`
    Customer CustomerV2Response `json:"customer"`
    Total    int64              `json:"total"`
    Status   string             `json:"status"`
}

The mapper for V1 selects only the customer name:

func mapToV1(domain Invoice) InvoiceV1Response {
    return InvoiceV1Response{
        ID:       domain.ID,
        Customer: domain.Customer.Name,
        Total:    domain.Total,
        Status:   domain.Status,
    }
}

mapToV2 maps the same domain invoice into the nested customer DTO, preserving ID, name, and phone. Both handlers use the same repository interface. The repository fixture supplies the same invoice to each mapper.

This is the boundary the lab implements: the domain model holds the invoice information, while a version-specific mapper decides what the consumer receives. There is no database schema or database migration in this lab.

Route each contract explicitly

The supported endpoints are:

GET /api/v1/invoices/1001
GET /api/v2/invoices/1001

V1Handler maps its result through mapToV1; V2Handler maps through mapToV2. Registering V2 does not replace V1's handler or DTO.

TestVersionedRoutes_CanRunSideBySide registers both handlers on the same mux and requests both paths. It decodes each response into its version's type and checks the customer values.

Another regression test registers V2 and then checks V1's wire contract again. That test targets a specific risk: a new route exists, but the old response must still retain the fields the legacy consumer understands.

The lab uses URL versioning. Header-based selection appears in the README as a design alternative; there is no header-version negotiation implementation here.

Test the consumer and inspect the response

The safe tests use two complementary views of the body. One decodes V1 into LegacyInvoice and checks its invoice fields. Another reads the response into a map of json.RawMessage and checks field presence and values.

For the central breaking change, the important assertion is explicit: V1's customer is decoded into a string and compared with Budi. V2's customer is decoded into an object, then its id, name, and phone are checked separately.

Typed response tests also check the invoice ID, total, and status. This gives the tests a concrete fixture-based contract rather than merely requiring that the body be JSON.

These assertions cover the fields and requests they exercise. They are not an exhaustive schema validator for all payloads, nullability combinations, or consumer implementations.

Adding a field is a different change

AdditiveHandler keeps customer as a string and adds currency:

{
  "id": 1001,
  "customer": "Budi",
  "total": 500000,
  "status": "PAID",
  "currency": "IDR"
}

TestAdditiveField_LegacyClientStillWorks verifies that the body contains currency with value IDR. It then decodes the response into the old LegacyInvoice, which has no currency field, and checks the customer and total.

The supplied decoder tolerates the extra field. The README qualifies the broader design rule: additive response fields are usually compatible when consumers tolerate unknown fields. Consumers using strict validation may behave differently.

The demonstrated outcome belongs to this Go decoding path. The lab does not include a strict-schema consumer test, so it does not prove compatibility with every possible reader.

Compatibility includes request requirements

The source also protects an expectation outside the JSON body. TestV1Contract_DoesNotIntroduceRequiredTenantHeader sends a V1 request without X-Tenant-ID and expects HTTP 200.

That test records that this V1 handler does not introduce the header as a requirement. It does not establish a broader authentication policy; the lab contains no tenant authentication middleware.

The README lists other contract changes, such as renamed fields, required inputs, date formats, and changed error representations. The implemented string-to-object case is the main demonstration. The listed possibilities should not be mistaken for separately implemented experiments.

Routing behavior is tested separately

The helper parseInvoiceID validates the prefix, requires an ID, rejects extra path segments, parses a numeric value, and requires it to be positive. The versioned handlers then look up that ID.

For the versioned fixture, the expected cases include:

Request	Expected response
`GET /api/v1/invoices/1001`	200, invoice JSON
`GET /api/v1/invoices/`	400, missing ID
`GET /api/v1/invoices/abc`	400, invalid numeric ID
`GET /api/v1/invoices/0`	400, nonpositive ID
`GET /api/v1/invoices/1001/extra`	400, extra segment
`GET /api/v1/invoices/9999`	404, invoice not found
`POST /api/v1/invoices/1001`	405, `Allow: GET`
`GET /api/v3/invoices/1001`	404 from the router

The fixture repository contains only invoice 1001. Thus 9999 is a valid ID format but has no corresponding invoice in the fixture.

newVersionedMux registers both the exact collection path and the subtree path for each version. The request without a trailing slash therefore reaches the handler and produces the lab's 400 response, rather than relying on an implicit redirect.

Handler-generated responses use application/json, including method errors. An unmatched V3 path is handled by the mux's default 404 behavior; the README documents that response as plain text. The common JSON writer does not wrap every router response.

Migration remains a documented policy

The README describes releasing V2, retaining V1 for older consumers, monitoring adoption, communicating deprecation, and removing V1 after sunset criteria are met.

It includes example timeframes and traffic thresholds. They are illustrative policy values, not measured adoption data or an implemented shutdown schedule. This directory contains no traffic-monitoring pipeline or automated sunset mechanism.

The code demonstrates the prerequisite: both representations can be served through separate routes, with tests that continue checking V1. Deciding when V1 can be removed requires the consumer information described in the README.

Run the lab from its module

The directory contains its own go.mod, declaring Go 1.25.0. Its README lists these test commands:

cd labs/06-api-versioning
go test -v ./...
go test -race -v ./...

The tests use httptest and a mock repository rather than an external database. No test execution results are claimed here.

The lab's mental model is precise: HTTP 200 describes a server response, while compatibility must be evaluated against the consumer's contract. Separate DTOs and regression tests keep that contract visible when a new representation is introduced.

Source

Based solely on the README, Go handlers, models, and tests in Software Engineering Lab, Lab 06. Test outcomes are described from source assertions, not a new run.

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/06-api-versioning

Race Condition on Concurrent Stock Updates

lukman lukman — Tue, 29 Sep 2026 05:58:05 +0000

A stock update can be implemented with a sequence that looks reasonable when only one request is running:

read stock
check availability
decrease stock
save the new value

Lab 05 focuses on what happens when multiple operations execute that sequence against the same state.

The important part of the experiment is not whether each request contains valid business logic in isolation. The test is whether the invariant still holds when those requests overlap.

The lab covers this from several angles:

an unsafe read/check/write flow;
deterministic lost-update tests;
an atomic conditional SQL update;
PostgreSQL row locking;
concurrent inventory attempts;
concurrent booking attempts protected by a database uniqueness constraint;
a separate Go memory data-race example.

The main experiment is the application/database race.

The Smallest Inventory Case

Start with one unit of stock:

stock = 1

Two operations attempt to consume it.

The expected business result is straightforward:

successful operations = 1
remaining stock       = 0

The unsafe implementation separates the decision into multiple steps.

Conceptually, the execution is:

READ
  ↓
CHECK
  ↓
CALCULATE
  ↓
WRITE

With one caller, the sequence behaves as expected.

With two callers, those steps can interleave.

Two Operations Read the Same State

Consider this execution:

Request A                    Request B
---------                    ---------
READ stock = 1

                             READ stock = 1

CHECK stock > 0              CHECK stock > 0

newStock = 0                 newStock = 0

WRITE stock = 0

                             WRITE stock = 0

Both operations observed the same original value.

Both checks passed.

Both calculated the same replacement value.

The final database value is:

stock = 0

Looking only at the final stock value does not expose the full problem.

Both operations may have been treated as successful even though only one unit existed.

The broken invariant is therefore not simply:

stock must never be negative

The test also needs to account for the number of successful decrements.

For one available item:

successful decrements <= 1

The Lost Update

The unsafe flow calculates a new value outside the final write:

read stock
calculate stock - 1
write calculated value

That creates a window where another operation can read the same state.

The important interleaving is:

A reads 1
B reads 1

A calculates 0
B calculates 0

A writes 0
B writes 0

The second write is not based on the state produced by the first write.

It is based on the earlier value both callers observed.

That is the lost-update behavior exercised by the lab.

The repository uses deterministic concurrency coordination for these tests rather than depending only on timing or arbitrary sleeps. Barriers/start gates are used to force the relevant operations to overlap at the point required by the experiment.

That matters because a concurrency test that passes or fails only depending on scheduler luck is a weak proof.

Moving the Condition Into the Update

One implementation in the lab removes the separate read/check/write sequence.

The SQL statement is:

UPDATE inventory_products
SET stock = stock - 1
WHERE id = $1
  AND stock > 0
RETURNING stock;

There are two important details in this statement.

First:

stock = stock - 1

The application does not read a value, calculate a replacement, then send that replacement back.

Second:

WHERE stock > 0

The availability condition is part of the statement that mutates the row.

The operation becomes:

attempt decrement
only when stock > 0

instead of:

read
decide in application
write later

This removes the application-level race window demonstrated by the unsafe implementation.

The Invariant Is Checked at the Write Boundary

The atomic version changes where the decision happens.

Unsafe model:

Application:
    read stock
    check stock
    calculate new stock

Database:
    write supplied value

Atomic model:

Database:
    decrement stock
    only if stock > 0

The difference is visible under contention.

The repository includes a concurrent inventory scenario with:

initial stock = 100
concurrent attempts = 500

The useful assertion is not merely that all concurrent calls return.

The final state must still be consistent with the initial inventory.

The number of successful decrements cannot exceed the amount that was available.

PostgreSQL Row Locking

The lab also contains a PostgreSQL row-lock implementation and tests.

The relevant read uses FOR UPDATE:

SELECT stock
FROM inventory_products
WHERE id = $1
FOR UPDATE;

The row lock changes the execution shape.

Instead of two transactions independently reading the same stock and later trying to update it, the critical read/check/update flow is coordinated around the same row.

A simplified execution is:

Transaction A
-------------
BEGIN

SELECT ... FOR UPDATE

check stock

update stock

COMMIT

A concurrent transaction targeting the same locked row does not proceed through the same critical section independently while the first transaction still owns the lock.

The repository demonstrates this as another way to protect the same business invariant.

The lab does not use that result to claim that every concurrency problem should be solved with row locking. It is one implementation demonstrated against the stock-update problem.

Atomic Update and Row Lock Solve the Problem Differently

Both implementations target the same unsafe boundary, but the shape is different.

With the atomic update:

UPDATE ...
SET stock = stock - 1
WHERE stock > 0
RETURNING stock;

the condition and mutation fit into one statement.

With row locking:

lock row
↓
read state
↓
make decision
↓
update
↓
commit

the operation retains a multi-step decision while coordinating access to the row.

The lab therefore gives two concrete implementations rather than reducing the topic to "put everything in a transaction."

A transaction establishes a transaction boundary.

The concurrency behavior still depends on how the shared row is accessed inside that boundary.

Booking Uses a Different Invariant

The repository also applies the concurrency problem to booking.

The invariant is no longer a numeric stock value.

A booking is identified by a combination including:

branch_id
service_date
slot_time

The schema protects that combination with a unique constraint:

UNIQUE(branch_id, service_date, slot_time)

The important concurrency case is:

many requests
↓
same branch
same date
same slot

An application can check whether the slot exists before inserting.

That check is useful for normal control flow, but it is not enough to enforce the invariant under concurrency.

Two requests can both observe:

slot is available

before either insert is visible to the other operation.

The database constraint is the final storage boundary.

Concurrent Booking Test

The booking test exercises a single slot with:

500 concurrent attempts

Only the valid database state should remain.

PostgreSQL uniqueness conflicts are mapped through SQLSTATE:

The conflict is treated as an expected concurrency outcome rather than as proof that the database is broken.

The database is enforcing the invariant that the application needs.

This is the same design principle as the inventory test, applied to a different state shape.

Inventory invariant:

successful decrements <= available stock

Booking invariant:

one row for one unique branch/date/slot combination

The mechanism is chosen around the invariant being protected.

Application Pre-Checks Do Not Replace Constraints

Consider the booking sequence:

SELECT booking
WHERE branch/date/slot = requested slot

if not found:
    INSERT booking

Under concurrent execution:

Request A                       Request B
---------                       ---------
SELECT → not found

                                SELECT → not found

INSERT

                                INSERT

Without storage enforcement, both requests reached the same application decision.

The unique constraint prevents the database from accepting both final rows for the same slot.

This is another example of why code that is correct sequentially can still be incomplete as concurrent business logic.

Memory Data Race vs Business Race

Lab 05 also keeps a Go memory data-race example.

That experiment is useful, but it is not the same failure as the inventory and booking cases.

A memory data race is concerned with unsynchronized memory access between goroutines.

The inventory and booking experiments focus on business invariants over shared application/database state.

Those two categories can overlap in some systems, but they should not be treated as synonyms.

The repository keeps the intentional memory race separated so the application/database concurrency tests can be reasoned about independently.

What the Tests Need to Prove

Concurrency tests need stronger assertions than:

no panic

or:

all goroutines completed

For inventory, useful assertions concern the invariant:

successful operations
remaining stock
initial stock

For booking, the relevant assertion concerns how many rows survive for the unique slot.

The final repository revision also made the PostgreSQL lost-update reproduction deterministic rather than relying on an arbitrary timing window.

That is an important testing property.

The test should deliberately create the interleaving being investigated.

Commands

Lab 05 is a nested Go module.

The repository exposes dedicated Make targets including:

make lab-05-test
make lab-05-test-race
make lab-05-vet
make lab-05-fmt
make lab-05-integration

The nested module boundary matters when running tests from the repository root: a plain root-level go test ./... does not implicitly replace the dedicated Lab 05 targets.

The README was updated to reflect that layout.

What This Lab Demonstrates

The unsafe inventory example starts from code that is locally understandable:

read
check
modify
write

Concurrency changes the correctness requirement.

Another operation can run between those steps.

The fixes in the repository move the invariant to a boundary that can protect it:

atomic database update

or:

row lock around the critical database flow

For booking, the invariant is enforced through:

database uniqueness

The broader result from the experiment is specific:

sequential correctness
does not prove
concurrent correctness

The invariant has to survive the interleavings that the system actually permits.

Source

This article is based on an implementation from my Software Engineering Lab:

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/05-race-condition

Caching: Why Faster Reads Create Consistency Problems

lukman lukman — Tue, 22 Sep 2026 07:48:22 +0000

A dashboard request can look harmless.

The problem appears when hundreds of users ask for the same expensive data at the same time.

In Software Engineering Lab 04, the scenario is a workshop dashboard. Without caching, every request runs six database queries with joins and aggregations.

500 concurrent users
        ↓
Dashboard request
        ↓
6 queries + join/aggregation per request
        ↓
3000 total DB queries

The first instinct is easy:

put Redis in front of the database

That reduces repeated work.

But it also creates a new set of problems.

How stale may the cached value be?
When should it be invalidated?
What happens when the key expires under load?
What happens if Redis is unavailable?
Can the cache return data from the wrong tenant?

That is the part I wanted to explore in Lab 04.

The main mental model is not:

slow query → add cache

It is closer to:

reduce repeated work
        ↓
accept a consistency boundary
        ↓
design expiration, invalidation, failure, and concurrency behavior

PostgreSQL is still the source of truth

The lab uses two storage layers:

Layer	Technology	Role
Primary	PostgreSQL	Durable, persistent, authoritative storage that can rebuild the cache
Cache	Redis	Derived data, TTL-bound, rebuilt on demand

The important decision is ownership of correctness.

In this lab, PostgreSQL remains authoritative for business data. Redis stores derived data that can disappear, expire, or be rebuilt.

That means cache correctness has to work even when the cache is empty.

The cache is an optimization layer, not the only copy of the business state.

The read path: Cache Aside

Lab 04 uses Cache Aside for reads.

GET cache
   ↓
hit? ───── yes ───→ return cached value
   ↓ no
query PostgreSQL
   ↓
populate Redis
   ↓
return value

A cache miss is not an application failure.

It means the application has to rebuild the value from the authoritative source.

The application explicitly knows about both storage layers:

Redis
PostgreSQL

That control is useful when Redis is unavailable because the application can still attempt to read from PostgreSQL, as long as the database and fallback capacity can handle the traffic.

But Cache Aside also means the application now owns cache freshness.

That is where the interesting failures start.

TTL is really a staleness decision

The useful question is not simply:

Does this data change?

The lab asks:

How long can stale data be accepted?

Examples recorded in the lab:

Data	Max staleness	Reasonable?
Dashboard statistics	30s–2min	Yes, operational metrics
Stock display	1–5s	Yes, UI/UX only
Wallet balance	0s	No, audit risk

The difference matters.

A dashboard metric can tolerate a freshness window that would be unacceptable for a transactional balance.

So TTL is not just an expiry configuration. In this implementation, it represents an accepted freshness window and also acts as a recovery path when stale cache survives longer than expected.

Invalidating before commit can reintroduce stale data

Consider this write flow:

DELETE cache
↓
update database
↓
COMMIT

It looks reasonable. Remove the old cache first, then write the new database value.

The race appears when a reader enters between those operations.

Lab 04 describes this timeline:

T1 Writer: DELETE cache
T2 Reader: cache MISS → reads old DB value
T3 Reader: SET old value into cache
T4 Writer: DB COMMIT new value

Final state:

Database = new value
Cache    = old value

This is stale cache, not data loss. The authoritative business data in PostgreSQL is still correct.

A safer order is:

DB COMMIT
↓
DELETE cache

Now a reader that misses after the delete can rebuild from the committed database value.

But even this is not strong consistency.

Another interleaving still exists:

T1 Reader: cache MISS
T2 Reader: reads old DB value
T3 Writer: DB COMMIT new value
T4 Writer: DELETE cache
T5 Reader: SET old DB result into cache

Final state again:

Database = new value
Cache    = old value

So the conclusion is narrower:

COMMIT → DELETE

is safer than:

DELETE → COMMIT

but Cache Aside is still an eventually consistent optimization in this lab.

TTL remains useful as a safety net for the residual stale window.

Updating Redis after a database write is not atomic either

The lab also uses an application-managed update-on-write flow:

DB update
↓
DB COMMIT succeeds
↓
best-effort Redis SET
↓
return success

The database returns the authoritative value, then the application tries to update the cache.

The problem is the boundary between PostgreSQL and Redis.

These are separate systems, so this sequence is not atomic:

DB COMMIT
↓
Redis SET

A process can fail between them:

DB COMMIT succeeds
↓
process crashes
↓
Redis SET never happens
↓
old cache remains

Concurrent writers create another race:

Writer A commits value A
Writer B commits value B
Writer B SET cache = B
Writer A performs a late SET cache = A

Final state:

Database = B
Cache    = A

This was an important distinction for me:

Cache Aside
→ read strategy

Invalidate-on-write / update-on-write
→ write strategy

They solve different parts of the cache lifecycle.

Neither turns PostgreSQL and Redis into one atomic system.

One miss is normal. One thousand simultaneous misses are not

Now consider a popular key reaching expiration.

cache expires
      ↓
1000 concurrent requests arrive
      ↓
1000 cache misses
      ↓
1000 parallel DB queries
      ↓
database overload / crash

This is the cache stampede scenario used in Lab 04.

The cache successfully removes repeated database work while the entry exists, but expiration can suddenly send that work back to the database at the same time.

For duplicate rebuilds inside one process, the lab uses golang.org/x/sync/singleflight.

The flow includes a second cache check:

initial cache GET
      ↓
miss
      ↓
singleflight.Do
      ↓
check cache again
      ↓
query DB
      ↓
populate cache
      ↓
share result

The second check matters because another caller may already have populated the cache between the first miss and the point where this caller becomes the rebuild leader.

Singleflight stops at the process boundary

Singleflight coordinates callers inside one process.

A multi-instance deployment needs a different coordination boundary.

Lab 04 implements a distributed-lock primitive:

WithLock() = try-once lock primitive

The conceptual cache regeneration flow is:

cache GET
   ↓ miss
acquire distributed lock
   ↓
check cache again
   ↓
query DB
   ↓
populate cache
   ↓
safe release

The lock requirements in the lab are explicit:

unique token or owner;
TTL to avoid a permanent deadlock;
atomic compare-and-delete when releasing;
one holder must not remove another holder's lock.

But even this has a boundary.

If regeneration takes longer than the lock TTL:

Instance A acquires lock
↓
lock expires
↓
Instance B acquires a new lock
↓
Instance A is still rebuilding
↓
duplicate rebuild can run

The lock in Lab 04 reduces duplicate cache regeneration. It is not used as a correctness primitive for business transactions.

Expiration can also be spread out

The lab adds TTL jitter so many keys do not expire at nearly the same moment.

Example:

60s + random 0–15s

The implementation produces values in:

[base, base + maxJitter)

The TTL never goes below base, and the upper bound is exclusive.

The lab also discusses background refresh: refresh the cached value before expiry while clients continue receiving the existing cache value.

Both techniques target expiration behavior rather than changing the source of truth.

Cache keys are part of the data boundary

The canonical key format in the lab is:

{app}:{tenant}:{branch}:{resource}:{dimension}

Example:

cmms:tenant:42:branch:7:dashboard:2026-09-01

The rule is simple:

Every input that changes the result belongs in the cache key.

For the dashboard example, that includes tenant, branch, and business date.

This is also a security boundary in a multi-tenant system.

A sensitive cached value must include tenant scope. Missing isolation can expose data under the wrong tenant context.

Key design also affects reuse.

If 10,000 users read the same branch dashboard, this shape:

cache:tenant:42:user:{user_id}:dashboard

creates high-cardinality entries for data that is actually shared.

The lab contrasts it with:

cache:tenant:42:branch:{branch_id}:dashboard

One branch-scoped value can be reused by those readers.

The trade-off is not "specific keys are bad." The point is that key dimensions should match the result boundary.

Redis failure changes where the traffic goes

Cache Aside allows database fallback when Redis fails, but only while the authoritative dependency and fallback capacity remain available.

The traffic pattern becomes:

traffic previously absorbed by Redis
      ↓
directly reaches PostgreSQL
      ↓
load spike / cache-failure amplification

So graceful degradation does not mean the outage has no effect.

It means the main function may continue with degraded performance while PostgreSQL can still handle the fallback traffic.

The lab also separates:

cache_miss

from:

cache_error

because an expected miss and an unavailable cache backend are different operational events.

Hit ratio alone does not tell me whether the cache is worth it

Lab 04 evaluates cache value together with:

avoided query cost
cache latency
memory cost
invalidation complexity
failure amplification

The README gives this comparison:

30% hit ratio for a 100ms operation
may still be valuable

99% hit ratio for a 0.1ms operation
may not justify the complexity

The useful signal is not the percentage alone.

The cost of the work being avoided matters.

Cache comes after understanding the database work

The diagnostic order in the lab is:

measure endpoint
↓
check N+1 queries
↓
inspect execution plan
↓
add / optimize indexes
↓
reduce selected columns
↓
optimize joins / subqueries
↓
evaluate caching if the workload needs it

This prevents cache from becoming a way to hide an unexplained query problem.

For a very cheap query and some workloads, the extra cache hop and operational complexity may not provide meaningful end-to-end benefit.

The mental model I take from this lab

The naive model is:

query is expensive
→ add Redis
→ problem solved

Lab 04 forces a longer chain of questions:

What is the source of truth?
↓
How stale may this value be?
↓
What dimensions belong in the key?
↓
How is the value invalidated after writes?
↓
What happens during concurrent misses?
↓
What happens when Redis fails?
↓
Is the avoided work worth the added complexity?

That is the main lesson from the experiment.

Caching can reduce repeated work in a read-heavy workload.

But the moment a second storage layer is introduced, freshness, invalidation, concurrency, failure behavior, and isolation become part of the design.

Running the lab

From the repository root:

docker compose up -d redis

make lab-04-test
make lab-04-test-race
make lab-04-vet
make lab-04-demo
make lab-04-integration

The demo scenarios can also be run directly:

cd labs/04-caching

go run ./cmd/demo -scenario=without-cache
go run ./cmd/demo -scenario=cache-aside
go run ./cmd/demo -scenario=stampede-unprotected
go run ./cmd/demo -scenario=stampede-protected
go run ./cmd/demo -scenario=write-through

Source Code

Software Engineering Lab 04 — Caching

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/04-caching

Repository:

https://github.com/lukman-ss/software-engineering-lab

Author: Lukman (lukman-ss)