DEV Community: Sagar Maheshwary

How I Debug Logs Across Services (and Why I Built a CLI Tool)

Sagar Maheshwary — Mon, 30 Mar 2026 16:21:46 +0000

A while back at work, I was debugging an issue across a few services.
I had the request_id, but the logs were scattered across different files.

Not every environment I work in has full tracing or centralized logging set up, especially local or staging setups, so I often rely on raw logs.

The usual workflow looked something like this:

open multiple terminals
run grep on each service
copy the same ID again and again
try to mentally piece together what happened

It worked… but it was slow and honestly pretty frustrating.
I kept thinking — there has to be a simpler way to do this.

Logs are usually:

spread across multiple services or files
in different formats (plain text, JSON, sometimes pretty logs)
hard to follow in order

Even when you do find the right logs, understanding the full request flow takes time. During live debugging, it gets worse — you’re trying to figure things out quickly while jumping between files and commands.

What I Wanted

A simpler workflow that lets me:

search by a key (like request_id, trace_id, etc)
scan multiple log files at once
see everything in a clean, chronological flow

Something lighter than full observability tools, but more structured than chaining grep commands.

Introducing `reqlog`

So I built a small CLI tool called reqlog.

Basic usage:

reqlog --dir ./logs --key request_id abc123

It searches across log files and prints a timeline like this:

2026-03-20T14:10:00Z [api-gateway]   | request_id=abc123 start request
2026-03-20T14:10:01Z [order-service] | request_id=abc123 fetching order
2026-03-20T14:10:02Z [inventory]     | request_id=abc123 checking stock

Instead of jumping between files, you can just follow the request from start to finish.

Here’s what it looks like in action:

Why Not Just Use `grep`?

grep is great — I still use it all the time.
But when you’re debugging across multiple services, it starts to fall apart:

recursive search can touch all files, but you still have to mentally stitch together logs from different services
no automatic sense of timeline across services
structured logs (like JSON) aren’t convenient
lots of copy/paste to track the same request

reqlog sits in between:

more structured than grep, but much simpler than full observability tooling.

Features (v1)

Nothing fancy — just what I needed:

works with plain text and JSON logs (single-line)
key-based search (--key request_id abc123)
scans logs across a directory by default (recursive)
optionally filter specific services (--service)
time filtering (--since)
live tailing (--follow)

Performance

I tested it on multi-million line logs (~9M+ lines across services).

Not as fast as grep (which is highly optimized and only does string matching), but still quick enough for real debugging workflows.
Most searches finish in a couple of seconds.

The extra work (JSON parsing, sorting into a timeline, etc.) adds some overhead — but that’s also what makes the output useful.

Tradeoffs

For v1, I kept things simple:

only supports single-line logs
no handling for multi-line / pretty logs
processes logs sequentially

There’s plenty of room to improve in future versions.

What’s Next

Some things I’m considering:

Docker / Kubernetes log support
performance improvements (parallel scanning)
support for more structured log formats

The main goal is to make debugging across services easier, even without tools like Jaeger or centralized logging systems.

Feedback

If you debug logs across multiple services, I’d love to hear:

would something like this be useful for you?
what would you want it to do (multi-line logs, k8s/docker support, etc.)?

If this looks interesting, check it out here: reqlog on GitHub

Transactional Outbox with RabbitMQ (Part 2): Handling Retries, Dead-Letter Queues, and Observability

Sagar Maheshwary — Fri, 23 Jan 2026 16:43:24 +0000

In Part 1, we built a correct and production-oriented foundation for the transactional outbox pattern using Go, PostgreSQL, and RabbitMQ. We ensured that domain state changes and event creation were atomic, published events asynchronously via a poll-based outbox worker, and enforced exactly-once processing on the consumer side using idempotency.

That baseline system already solves the hardest problem: correctness under partial failure.

But correctness alone isn’t enough in real-world systems.

Once this pattern hits production, new questions immediately surface:

What happens when publishing to RabbitMQ fails repeatedly?
How do we retry safely without flooding the broker or duplicating work?
How do consumers handle poison messages?
When should events be retried, and when should they be sent to a dead-letter queue?
How do we observe failures early, before downstream systems silently degrade?
At higher scale, is polling still the right choice or should CDC-based approaches be considered?

This article answers those questions.

In Part 2, we extend the system from Part 1 with failure-aware behavior and operational guardrails, without changing the core model. The transactional outbox remains the source of truth; we simply make it resilient, observable, and easier to operate under load.

Specifically, we’ll cover:

Retry strategies for the outbox publisher with exponential backoff
Dead Letter Queues (DLQs) for events that cannot be published
Consumer-side retries and dead-letter exchanges
Operational metrics and dashboards that surface failure modes early
A practical discussion of Polling vs CDC-based outbox implementations (Debezium + Kafka), including trade-offs and when each approach makes sense

This is not a rewrite of the system from Part 1.
Instead, it’s an evolution, the kind that happens after the first real incident.

By the end of this article, you’ll understand how to take a correct outbox implementation and turn it into one that’s boring to operate, even when things go wrong.

Table of Content

Failure as a First-Class Design Constraint
Retrying Outbox Publishing (Producer-Side Retries)
Moving permanently failed events to a DLQ
Consumer-Side Retries Using Dead-Letter Exchanges
What To Do With Messages in the DLQ
Retry & Failure Metrics
Tracing & Outcome Visibility
CDC-Based Outbox as an Alternative
Conclusion

Failure as a First-Class Design Constraint

In distributed systems, failure isn’t an edge case, it’s the normal operating condition.

Message brokers go down, consumers crash and restart, networks partition, bad data makes it to production.

Retries exist because things will fail, not because we expect them to succeed eventually.

But retries are also dangerous.

Poorly designed retries don’t fix problems, they amplify them:

Retrying inline in request handlers ties user latency to broker availability
Tight retry loops can overwhelm databases and brokers
Retrying without backoff creates retry storms
Retrying forever hides broken messages instead of surfacing them

Not all failures are transient. Some messages will never succeed due to invalid payloads or schema mismatches. A healthy system must know when to stop retrying and move on.

Well-designed retries share a few properties:

asynchronous for background workflows and message processing
bounded (no infinite loops)
backoff-aware (to avoid overload)
observable (failures are visible, not silent)
built on idempotency

The transactional outbox gives us a natural place to implement this safely: durable retry state, explicit event lifecycle, and isolation from request paths.

In the next section, we’ll extend the outbox worker with retry logic and exponential backoff, turning failures into delayed work instead of cascading outages.

Retrying Outbox Publishing (Producer-Side Retries)

In Part 1, a failed publish immediately moved the event to a failed state. That was intentional, it gave us a correct baseline without hiding failure.

In real systems, however, most publish failures are transient: brief broker outages, network hiccups, or short-lived resource pressure. Retrying these events is both safe and expected, as long as retries are designed carefully.

In this section, we extend the outbox worker to support bounded, delayed retries with exponential backoff, while keeping all retry state durable and observable.

Schema changes for retries

To support retries, the outbox needs to track how many times an event has been attempted and when it becomes eligible again.

The Part 2 schema extends outbox_events with two new columns:

CREATE TABLE outbox_events (
  id TEXT PRIMARY KEY,
  event_key TEXT NOT NULL,
  payload JSONB NOT NULL,
  status OutboxEventStatus NOT NULL,
  retry_count INT NOT NULL DEFAULT 0,
  next_retry_at TIMESTAMP NULL,
  locked_at TIMESTAMP DEFAULT NULL,
  locked_by VARCHAR(128) NULL,
  failure_reason VARCHAR(128) DEFAULT NULL,
  failed_at TIMESTAMP DEFAULT NULL,
  traceparent TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
);

Conceptually:

retry_count tracks how many publish attempts have failed
next_retry_at enforces backoff by delaying eligibility
Events remain in the outbox until they are either published or explicitly dead-lettered

Retries change how workers select events, so indexing needs to reflect that.

CREATE INDEX idx_outbox_events_pending_ready
ON outbox_events (created_at)
WHERE status = 'pending';

CREATE INDEX idx_outbox_events_retryable
ON outbox_events (locked_at, next_retry_at, created_at)
WHERE status = 'in_progress';

idx_outbox_events_pending_ready Optimizes the common case: picking the oldest pending events ready to publish.
idx_outbox_events_retryable Speeds up recovery scenarios where:

a worker crashed (locked_at expired), or
an event is waiting for its retry delay (next_retry_at <= now())

Using partial indexes keeps them small and focused, avoiding unnecessary index scans on completed or failed events.

Making the worker retry-aware

Retries are modeled as delayed eligibility, not as a new status.

An event is eligible for publishing if:

it is pending, or
it was abandoned by a crashed worker (in_progress lease expired)
and its retry delay has elapsed

The worker’s polling query is updated accordingly:

UPDATE outbox_events
SET
  status = ?,
  locked_at = NOW(),
  locked_by = ?
WHERE id IN (
  SELECT id
  FROM outbox_events
  WHERE
    (
      status = ?
      OR (
        status = ?
        AND locked_at < NOW() - INTERVAL '30 seconds'
      )
    )
    AND (
      next_retry_at IS NULL
      OR next_retry_at <= NOW()
    )
  ORDER BY created_at
  LIMIT ?
  FOR UPDATE SKIP LOCKED
)
RETURNING *

This ensures that:

events under backoff are skipped
retries are naturally spread out over time
multiple workers can run safely without coordination

Retrying on transient publish failures

When publishing fails, the worker no longer immediately marks the event as failed.

Instead, it schedules a retry by resetting the event to pending and pushing next_retry_at into the future.

func (o *Outbox) scheduleRetry(ctx context.Context, event *model.OutboxEvent) {
    backoff := backoff(event.RetryCount, o.config.RetryDelaySeconds)

    err := o.outboxEventService.UpdateState(ctx, event.ID, map[string]interface{}{
        "status":        model.OutboxEventStatusPending,
        "retry_count":   event.RetryCount + 1,
        "next_retry_at": time.Now().Add(backoff),
        "locked_at":     nil,
        "locked_by":     nil,
    })
    if err != nil {
        o.log.Error("Failed to schedule retry for event",
            logger.Field{Key: "error", Value: err.Error()},
        )
    }
}

Key points:

Retry state is fully durable
No in-memory timers or loops
Worker crashes do not lose retry intent
Backoff is enforced by the database, not the process

Exponential backoff with jitter

Retries use exponential backoff with jitter to avoid retry storms when multiple workers fail at once:

func backoff(retryCount int, baseDelay time.Duration) time.Duration {
    jitter := time.Duration(randInt(0, 1000)) * time.Millisecond
    return time.Duration(1<<uint(retryCount))*baseDelay + jitter
}

func randInt(min int, max int) int {
    return min + rand.Intn(max-min)
}

This gives us:

fast recovery for transient blips
increasing delays under sustained failure
natural spreading of retry attempts across workers

Retries are capped (e.g. max 3 attempts) to prevent infinite loops.

Moving permanently failed events to a DLQ

A Dead Letter Queue (DLQ) is a dedicated queue for messages that could not be processed successfully and should no longer be retried.
Its role is not automatic recovery, but isolation, visibility, and controlled handling of bad events.

Not all failures are transient.

If an outbox event exceeds the maximum retry count, it is treated as permanently failed and published to a DLQ instead of the normal exchange.

func (o *Outbox) publishToDLQ(
    ctx context.Context,
    ch *amqp091.Channel,
    event *model.OutboxEvent,
    procErr error,
) {
    dlqEvent := map[string]interface{}{
        "event_id":       event.ID,
        "event_key":      event.EventKey,
        "payload":        event.Payload,
        "failed_at":      time.Now(),
        "failure_reason": procErr.Error(),
    }

    err := o.rabbitmq.Publish(
        ctx,
        &rabbitmq.PublishOpts{
            Ch:         ch,
            Exchange:   o.amqpConfig.DLX,
            RoutingKey: o.amqpConfig.DLQ,
            Body:       dlqEvent,
            Headers:    amqp091.Table{"message_id": event.ID},
        },
    )
    if err != nil {
        o.log.Error("Failed to publish event to DLQ", logger.Field{Key: "error", Value: err.Error()})
        return
    }
}

We use a dedicated exchange:

outbox.dlx

Instead of reusing the original event routing key (e.g. order.created), we publish using the DLQ name itself as the routing key:

order-service.dlq

The Order Service declares and binds:

Queue: order-service.dlq
Binding key: order-service.dlq

Using the DLQ name as the routing key makes DLQ ownership explicit and keeps failed events isolated per microservice, without overloading domain event routing keys with failure semantics.

Once published to the DLQ, the outbox event is marked as failed and is no longer eligible for retries.

If DLQ publishing fails, the event remains failed and is surfaced via logs and metrics. DLQs are an operational signal, not a delivery guarantee.

At this point, retries are no longer a best-effort loop, they’re a durable, observable state machine, with the outbox as the single source of truth.

Consumer-Side Retries Using Dead-Letter Exchanges

Once a message is successfully published, failures shift from the broker to the consumer. These failures fall into two broad categories:

Permanent failures — malformed payloads, missing headers, invalid data
Transient failures — database outages, timeouts, temporary dependency issues

Retrying both the same way is a mistake. Retrying permanent failures wastes resources, while retrying transient failures incorrectly can amplify load and slow down the entire system.

For consumer-side retries, we rely on RabbitMQ dead-letter exchanges (DLX) instead of retrying in application code.

Retry Topology

Retries are modeled as explicit retry levels, each with its own delay:

RetryConfig{
    Levels: []RetryLevel{
        {"retry.30s", 30 * time.Second},
        {"retry.1m",  1 * time.Minute},
        {"retry.5m",  5 * time.Minute},
    },
}

Each retry level consists of:

A retry exchange (e.g. outbox.events.retry.30s)
A retry queue with:
- x-message-ttl
- x-dead-letter-exchange pointing back to the main exchange

When a message expires in a retry queue, RabbitMQ automatically republishes it to the original exchange using the same routing key.

This design allows a single retry topology to handle multiple event types (order.created, order.updated, etc.) without creating per-routing-key queues.

Classifying Failures

The consumer explicitly signals whether a failure is retryable:

var (
    ErrPermanent = errors.New("permanent error")
    ErrTransient = errors.New("transient error")
)

Examples:

JSON unmarshalling failure → permanent
Database query failure → transient

if err := json.Unmarshal(message.Body, &body); err != nil {
    return fmt.Errorf("%w: failed to unmarshal message body", ErrPermanent)
}

This keeps retry decisions deterministic and easy to reason about.

Retrying a Message

Retries are handled by republishing the message to the next retry exchange.

Retry state is tracked via headers:

x-retry-count increments on each retry
The message body and routing key remain unchanged

headers["x-retry-count"] = retryCount + 1

Publish(
    exchange: "outbox.events.retry.30s",
    routingKey: message.RoutingKey,
)

The original message is always ACKed, regardless of success or failure.

We never rely on broker re-delivery. Retries are explicit and fully controlled by retry queues.

Why There Is No Native Jitter on the Consumer Side

TTL-based retries in RabbitMQ are deterministic: messages expire at fixed intervals and are re-queued at the same time they were originally published. Unlike producer-side retry logic, consumers do not have a natural place to introduce per-message jitter without delaying acknowledgements or holding messages in memory.

As a result, when many messages fail simultaneously, retries tend to align and create short-lived spikes in traffic. In high-throughput systems, one way to mitigate this is to shard retry queues per retry level and route messages deterministically using a hash of message identity. This spreads retries over time without introducing artificial sleeps or per-message jitter, though it’s generally better to observe retry behavior and throughput in production before optimizing the retry strategy further.

When Retries Are Exhausted

If no retry level remains, the message is treated as permanently failed and published to a dead-letter queue (DLQ):

Routing key is the DLQ name (service-owned)
Publishing is best-effort
No further retries are attempted

Example:

Exchange: outbox.dlx
Queue: notification-service.dlq
Routing key: notification-service.dlq

At this point, the failure is considered operational or data-related and requires manual inspection.

What To Do With Messages in the DLQ

A DLQ is not a trash can. It’s a decision point.

Once a message lands in the DLQ, automated retries have stopped and the system is intentionally asking for human judgment.

In practice, DLQ handling usually falls into three categories:

1. Inspect and discard

Some messages are fundamentally invalid:

schema mismatches
corrupted payloads
missing required fields

After inspection, these events are discarded, and the root cause is fixed upstream.
The DLQ acts as a safety net, not a backlog.

2. Fix data and replay

Some failures are recoverable with intervention:

missing reference data
temporary downstream misconfiguration
bad assumptions fixed in code

These messages can be:

corrected manually, or
replayed after the consumer or producer is fixed

Because DLQ messages preserve payload and metadata, replay is deliberate, not accidental.

3. Escalate operationally

A growing DLQ is often a system-level signal:

a consumer deployment is broken
an external dependency changed behavior
retry limits are too low

In this case, the DLQ is less about the messages themselves and more about triggering alerts, dashboards, and on-call investigation.

DLQs shift failure from automation to operations.
They make failures visible, bounded, and actionable. which is exactly what a reliable system needs.

Retry & Failure Metrics

In the previous article, we focused on whether events are flowing at all.
Once retries and DLQs enter the picture, the question shifts slightly:

Are failures behaving the way we expect them to?

Retries are not inherently bad but unbounded or misaligned retries are.
The following metrics make retry behavior visible, bounded, and explainable, without turning this into an observability deep dive.

Retry attempts

outbox_retries_total

This counter tracks how many times an outbox event is scheduled for retry.

It answers a simple but important question:

How often are we failing, but recovering later?

A low, steady retry rate is usually acceptable.
A sudden spike often indicates:

a downstream dependency outage
transient broker or network failures
a bad deploy causing temporary publish errors

This metric is incremented only when an event fails to publish and is rescheduled, not when it is first written.

metrics.OutboxRetriesTotal.Inc()

In practice, an alert would trigger only if retry spikes correlate with other symptoms, such as:

a growing outbox backlog
increasing publish latency

Retries without backlog growth usually mean the system is degrading gracefully.

Retry exhaustion

outbox_retry_exhaustions_total

This counter tracks how many events have exceeded the maximum retry limit.

At this point, the system has made a conscious decision:

This event will not be published through the normal path.

A non-zero value here is expected over long periods.
However, a sudden increase usually signals a persistent failure, while sustained growth means retries are no longer helping and are only delaying failure.

This metric is incremented exactly once per event, when retry limits are exceeded.

metrics.OutboxRetryExhaustionsTotal.Inc()

In practice, an alert would trigger if:

retry exhaustions suddenly spike above baseline
or retry exhaustions grow steadily relative to total event volume

Dead-letter publishing

Once retries are exhausted, events are sent to a Dead Letter Queue (DLQ).

We track this explicitly using two counters:

outbox_dlq_published_total
outbox_dlq_publish_failed_total

These metrics distinguish between:

events intentionally routed to the DLQ
failures while publishing to the DLQ itself (a much more serious issue)

metrics.OutboxDLQPublishedTotal.Inc()

metrics.OutboxDLQPublishFailedTotal.Inc()

If outbox_dlq_publish_failed_total increases, it usually indicates:

misconfigured DLX / DLQ bindings
broker-level issues
permission or exchange configuration problems

In practice, an alert would trigger if:

DLQ publishes spike unexpectedly
or any DLQ publish failures occur, since this implies events are no longer safely retained

Consumer-side retry & DLQ visibility

On the consumer side, we also track retry and DLQ behavior, alongside core processing metrics like:

consumer_messages_total
consumer_processing_failed_total

Together, these metrics feed into a dedicated Consumer Processing Dashboard, giving a clear view of throughput, failure rates, and terminal outcomes without overloading the system with cardinality-heavy signals.

Separating concerns with dedicated dashboards

To avoid mixing responsibilities, we intentionally use two focused dashboards:

Outbox Reliability Dashboard

This dashboard answers one core question:

Is the event publishing pipeline healthy?

It focuses on:

outbox backlog growth
publish latency
retry and retry-exhaustion rates
DLQ routing from the outbox

Consumer Processing Dashboard

This dashboard answers a different question:

Are downstream business operations succeeding?

It focuses on:

message consumption rate
processing failure rate
retry queue activity
DLQ volume and trends

Keeping these dashboards separate prevents alert fatigue and makes ownership clearer:
outbox issues are usually infrastructure or configuration problems, while consumer issues tend to be business logic or dependency failures.

Together, they provide a complete picture of the event lifecycle, from transactional write to final business outcome without overloading any single view.

Tracing & Outcome Visibility

In this article, we deliberately keep tracing lightweight.

There are no new spans for retries, DLQ hops, or internal helpers. Instead, we focus on making the existing spans more expressive by attaching the right attributes.

This keeps traces readable while still answering the questions you actually ask during incidents.

Outcomes as first-class signals

Internally, each processing attempt is classified into one of a small, well-defined set of outcomes:

published — successfully sent to the broker
retry — failed, but scheduled for another attempt
dlq — retries exhausted, sent to DLQ
failed — internal failure (e.g. DB update or retry scheduling failure)

While counters remain intentionally simple, these outcomes are attached as trace attributes, allowing us to reason about individual event lifecycles.

This avoids exploding metric cardinality while still preserving detail in traces.

Enriching spans with outcomes

Both the outbox publisher and the consumer record a single span per message, and attach an explicit outcome to it.

For the outbox:

attribute.String("outbox.outcome", "published|retry|dlq|failed")

For the consumer:

attribute.String("consumer.outcome", "success|retry|dlq|failed")

This makes it trivial to:

filter traces by failure mode
distinguish retries from terminal failures
answer “what happened to this message?” without reading logs

Instead of inferring intent from error messages or span status, the outcome is made explicit.

Retry visibility without span explosion

Retries are represented via attributes rather than additional spans:

attribute.Int("event.retry_count", event.RetryCount) //Outbox
attribute.Int("rabbitmq.retry_count", getRetryCount(message.Headers)) //Consumer

This avoids a common pitfall where retry-heavy systems generate deeply nested or noisy traces.

You still get full visibility into retry behavior, but without sacrificing trace clarity.

Trace–log correlation

While the dashboards in this article focus on metrics, the codebase also supports trace–log correlation.

All log statements accept a context via:

logger.WithContext(ctx)

When enabled, this automatically attaches trace and span IDs to logs, allowing you to:

jump from a trace in Jaeger to the exact logs that belong to it
correlate retries, DLQ publishes, and failures without manual searching

For brevity, we don’t include log-based dashboards here, but the hooks are intentionally present so readers can experiment and extend the system if needed.

CDC-Based Outbox as an Alternative

Another way to implement the outbox pattern is by combining it with Change Data Capture (CDC).

Instead of running a background worker that periodically polls the outbox_events table, CDC-based approaches stream database changes directly from the database’s write-ahead log (WAL) into a message broker. Tools like Debezium can capture inserts into the outbox table and publish them to Kafka topics with very low latency.

At a high level, the flow looks like this:

From an application perspective, the outbox table remains exactly the same. Domain events are still created explicitly, with well-defined payloads and clear business intent. The difference lies entirely in how those events leave the database.

What CDC Changes

With CDC, new outbox rows are emitted almost immediately after the transaction commits. There is no polling interval, no periodic SELECT queries, and no need to tune batch sizes. Event delivery becomes effectively push-based, driven by database writes rather than background scans.

This can be especially attractive when:

low end-to-end latency is important
the outbox table grows quickly
polling overhead on the primary database needs to be minimized

In practice, CDC turns the outbox into a streaming source, while preserving the core benefits of the transactional outbox pattern: atomic writes, durable state, and explicit event lifecycles.

Trade-offs and Operational Cost

While CDC reduces polling-related concerns, it introduces a different kind of complexity.

A CDC-based outbox typically requires:

logical replication enabled at the database level
replication slots and WAL retention management
a CDC connector (e.g., Debezium) running reliably
Kafka (or a similar log-based broker) as infrastructure

These components add operational surface area and failure modes that don’t exist in a simple polling-based design. Connector configuration, schema evolution, topic management, and offset handling become part of the system’s responsibility.

For many teams, this is a reasonable trade-off. For others, especially smaller teams or services with moderate throughput, the added infrastructure may outweigh the latency benefits.

Choosing Between Polling and CDC

Both polling-based and CDC-based outbox implementations solve the same core problem: reliably publishing domain events without breaking transactional guarantees.

Polling favors simplicity, explicit control, and minimal infrastructure.
CDC favors lower latency and push-based delivery, at the cost of operational complexity.

There is no universally “correct” choice. The right approach depends on scale, team maturity, and operational constraints.

As with most distributed systems patterns, real-world implementations may vary in complexity. The comparison here is intentionally simplified to highlight trade-offs, not prescribe a one-size-fits-all solution.

Conclusion

In this article, we extended the transactional outbox pattern with producer-side retries, consumer-side retries, and dead-letter handling, while adding observability through metrics, dashboards, and distributed tracing. With these building blocks, the outbox becomes robust, debuggable, and ready to handle transient and permanent failures gracefully.

The provided Prometheus metrics and Jaeger traces give you visibility into backlog, retry attempts, publish outcomes, and DLQ traffic, while logger.WithContext() allows logs to correlate with spans for deeper insight. Together, these tools make it easy to monitor and troubleshoot the system, and even experiment with load, stress, or chaos testing on your own.

Thanks for reading! If you have any questions or insights, please leave a comment.
You can also check out the full working example on GitHub.

Transactional Outbox with RabbitMQ (Part 1): Building Reliable Event Publishing in Microservices

Sagar Maheshwary — Mon, 19 Jan 2026 08:18:37 +0000

Publishing events directly to a message broker from an API handler is a common source of inconsistency in distributed systems. If the database transaction commits but the broker publish fails (or vice versa), the system ends up in a partially updated state: an order exists in the database, but downstream services never receive the event.

This problem cannot be solved reliably with retries alone. The database and the message broker are two independent systems with no shared transaction boundary, which makes atomic updates across both impossible.

The Transactional Outbox Pattern addresses this by storing domain events in a database table within the same transaction as the business operation. A separate background worker then publishes those events to the message broker asynchronously, ensuring consistency between state changes and event delivery.

In this article, we’ll build a minimal but production-oriented implementation of this pattern using Go, PostgreSQL, RabbitMQ, and Docker Compose. We’ll develop two microservices:

Order Service – exposes a /orders endpoint, persists orders, and writes outbox events atomically
Notification Service – consumes events and demonstrates exactly-once processing using consumer-side idempotency

To keep the focus on correctness and clarity, the outbox publisher uses a poll-based worker that queries the database every 1–2 seconds. This approach is simple, reliable, and sufficient for moderate-scale systems.

This is a two-part series:

Part 1 (this article) focuses on core functionality: service setup, transactional outbox writes, poll-based publishing, consumer idempotency via a processed_messages table, essential metrics for SLOs, and trace context propagation across services.
Part 2 extends the system with retries and exponential backoff for both publisher and consumer, dead-letter queues, observability dashboards, and a discussion of CDC-based outbox implementation using Debezium and Kafka.

System Overview
Database Schema
Order Service (transactional write)
Outbox Worker (publisher)
Using RabbitMQ Topic Exchange
Notification Service (consumer)
Consumer side Idempotency
Deduplication with a Unique Constrait
End-to-end test with Docker Compose
What to Do with Published Outbox Events
Operational Metrics
Trace Propagation Through the Outbox
What's Next
Conclusion

System Overview

At a high level, the Order Service persists business state and records domain events atomically using the outbox pattern.
A background outbox worker polls the database, publishes pending events to RabbitMQ, and marks them as published.
Downstream services, such as the Notification Service, consume these events and ensure exactly-once processing using idempotency at the consumer side.

In production, outbox worker often runs as a separate process, for this article we keep it in the same binary to reduce operational complexity.

Database Schema

This section defines the database tables that enable reliable event publishing using the outbox pattern and exactly-once message processing on the consumer side.

`outbox_events` table

This table is the core of the outbox pattern. Every domain event is persisted here in the same transaction as the business operation, ensuring atomicity between state changes and event creation.

CREATE TYPE OutboxEventStatus AS ENUM (
  'pending',
  'in_progress',
  'published',
  'failed'
);

CREATE TABLE outbox_events (
  id UUID PRIMARY KEY,
  event_key VARCHAR(64) NOT NULL,
  payload JSONB NOT NULL,
  status OutboxEventStatus NOT NULL,
  locked_at TIMESTAMP DEFAULT NULL,
  locked_by VARCHAR(128) DEFAULT NULL,
  failure_reason TEXT DEFAULT NULL,
  failed_at TIMESTAMP DEFAULT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

Column breakdown

id A UUID generated at event creation time. This value is reused as the message_id on the consumer side to implement idempotent processing.
event_key The routing key used by RabbitMQ to determine which consumers should receive the event.
payload The event body stored as JSON. In a real system this would typically contain identifiers and minimal state required by downstream services.
status Tracks the lifecycle of the event:
- pending → created but not yet picked up
- in_progress → locked by a worker
- published → successfully sent to RabbitMQ
- failed → publishing failed
locked_at / locked_by
- Used to implement lease-based locking.
- When a worker picks up events, it acquires a lease by setting these fields. If a worker crashes mid-processing, the lease can expire (e.g. after 30 seconds), allowing another worker to safely reclaim and retry those events.
failure_reason / failed_at Capture error context when publishing fails, making retries and debugging easier.

Indexes

The outbox worker frequently queries events by status and creation time, and needs to efficiently detect expired leases. The following indexes support these access patterns:

CREATE INDEX idx_outbox_events_status_created_at
ON outbox_events (status, created_at);

CREATE INDEX idx_outbox_events_locked_at
ON outbox_events (locked_at)
WHERE locked_at IS NOT NULL;

We’ll look at the exact SQL used by the outbox worker in a later section.

`orders` table

This table represents a minimal business entity and exists solely to demonstrate an atomic business operation combined with outbox event creation.

CREATE TABLE orders (
  id BIGSERIAL PRIMARY KEY,
  status VARCHAR(10) NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
);

`processed_messages` table

This table is used by the Notification Service (the RabbitMQ consumer) to enforce idempotency and achieve effectively-once processing.

CREATE TABLE processed_messages (
  message_id UUID PRIMARY KEY,
  processed_at TIMESTAMP NOT NULL DEFAULT NOW()
);

Each consumed message attempts to insert a row using its message_id.
If the insert fails due to a primary key conflict, the message has already been processed and can be safely ignored.

This pattern protects the consumer from:

message redeliveries
consumer restarts
broker retries

Order Service (transactional write)

Let’s start with the Order Service. Below is the directory structure to give you a sense of how responsibilities are separated:

.
├── cmd
│   └── server
│       └── main.go
├── internal
│   ├── config
│   ├── database
│   │   └── model
│   │       ├── order.go
│   │       └── outbox_event.go
│   ├── http
│   │   └── handler
│   ├── outbox
│   ├── rabbitmq
│   └── service
│       ├── order.go
│       └── outbox_event.go
└── schema.sql

Boilerplate related to configuration, logging, database setup, and HTTP server wiring is intentionally omitted from the article to keep the focus on the outbox pattern. The complete, runnable code is available in the repository.

Application lifecycle and graceful shutdown

The Order Service runs two long-lived components:

An HTTP server that handles /orders requests
An outbox worker that continuously polls and publishes events

Both are started from main.go and share a common context.Context.

On shutdown (for example, SIGTERM in Kubernetes or Docker), the context is cancelled, which allows:

The HTTP server to stop accepting new requests
The outbox worker to stop polling
In-flight event processing to complete before exit

This ensures that:

No new outbox events are claimed during shutdown
Already claimed events are either published or released safely
The service can be terminated without corrupting outbox state

The full main.go and shutdown wiring are intentionally omitted here to keep the focus on the outbox pattern. You can find the complete implementation in the repository.

Atomic order creation + outbox write

The most important guarantee we want from the outbox pattern is this:

Either the order and its corresponding event are both persisted, or neither is.

To achieve this, we write the order row and the outbox event within the same database transaction. At this stage, we do not interact with RabbitMQ at all, publishing is handled later by a background worker.

Here's the Order creation logic:

func (o *orderService) Create(ctx context.Context, req *CreateOrder) (*model.Order, error) {
    var order *model.Order

    err := withTransaction(ctx, o.db, func(tx *gorm.DB) error {
        order = &model.Order{
            Status: "pending",
        }

        if err := tx.Create(order).Error; err != nil {
            return err
        }

        outboxEvent := &model.OutboxEvent{
            ID:       uuid.NewString(),
            EventKey: "order.created",
            Payload: model.JSONB{
                "id":         order.ID,
                "product_id": req.ProductID,
                "quantity":   req.Quantity,
            },
            Status:      model.OutboxEventStatusPending,
        }

        if err := o.outboxEventService.Create(ctx, tx, outboxEvent); err != nil {
            return err
        }

        return nil
    })

    if err != nil {
        return nil, err
    }

    return order, nil
}

If the transaction commits, we are guaranteed that the event exists in the outbox table.
If it fails, neither the order nor the event is persisted.

Outbox Worker (Publisher)

Once events are written to the outbox_events table, a background worker is responsible for publishing them to RabbitMQ.

Each event follows a simple lifecycle:

pending → in_progress → published | failed

The key requirement is that multiple worker instances must be able to run concurrently without publishing the same event twice.

Claiming events safely

The outbox worker periodically queries the database and claims a batch of events for processing:

func (o *outboxEventService) ClaimEvents(
    ctx context.Context,
    workerID string,
    limit int,
) ([]*model.OutboxEvent, error) {
    var events []*model.OutboxEvent

    err := o.db.WithContext(ctx).
        Raw(`
        UPDATE outbox_events
        SET
                    status = ?,
                    locked_at = NOW(),
                    locked_by = ?
        WHERE id IN (
                    SELECT id
                    FROM outbox_events
                    WHERE
                            status = ?
                            OR (
                                status = ?
                                AND locked_at < NOW() - INTERVAL '30 seconds'
                            )
                    ORDER BY created_at
                    LIMIT ?
                    FOR UPDATE SKIP LOCKED
        )
        RETURNING *`,
            model.OutboxEventStatusInProgress,
            workerID,
            model.OutboxEventStatusPending,
            model.OutboxEventStatusInProgress,
            limit,
        ).
        Scan(&events).Error

    return events, err
}

This query does several important things atomically:

Selects only events that are:
- pending, or
- in_progress but abandoned by a crashed worker (lease expired)
Orders events by creation time
Limits the batch size
Locks rows using FOR UPDATE SKIP LOCKED
Updates their state to in_progress
Returns the claimed rows in one round-trip

Because this is a single SQL statement, we avoid race conditions that would occur with a SELECT → UPDATE approach.

SKIP LOCKED ensures that if another worker has already claimed certain rows, they are skipped instead of blocking, allowing horizontal scaling without coordination.

Concurrent publishing with bounded workers

Once events are claimed, the worker publishes them concurrently using a bounded pool of goroutines:

func (o *Outbox) Start(ctx context.Context, workerID string) {
    o.log.Info("Outbox worker started")

    if err := o.initChannels(o.config.MaxConcurrency); err != nil {
        o.log.Error("Failed to initialize channels", logger.Field{Key: "error", Value: err.Error()})
        return
    }
    defer o.closeChannels()

    eventsCh := make(chan *model.OutboxEvent, o.config.MaxConcurrency)
    wg := sync.WaitGroup{}

    for i := 0; i < o.config.MaxConcurrency; i++ {
        wg.Add(1)
        go func(workerID int) {
            defer wg.Done()
            o.processEvents(ctx, workerID, o.channels[i], eventsCh)
        }(i)
    }

    ticker := time.NewTicker(o.config.MaxConcurrency)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            close(eventsCh)
            wg.Wait()
            return

        case <-ticker.C:
            o.dispatchPendingEvents(ctx, workerID, eventsCh)
        }
    }
}

Each goroutine uses its own RabbitMQ channel.

RabbitMQ channels are lightweight but not thread-safe, so the recommended pattern is one channel per goroutine while sharing a single TCP connection.

This design allows:

Back-pressure via bounded concurrency
Safe parallel publishing
Independent scaling of outbox workers

Publishing and failure handling

Each worker publishes the event and updates its state based on the outcome:

func (o *Outbox) processEvents(
    ctx context.Context,
    workerID int,
    ch *amqp091.Channel,
    events <-chan *model.OutboxEvent,
) {
    for event := range events {
        err := o.PublishEvent(ctx, ch, event)
        if err == nil {
            o.markPublished(ctx, event)
            continue
        }

        o.markFailed(ctx, event, err)
    }
}

In this first article, we intentionally keep failure handling simple:

On success → mark event as published
On error → mark event as failed

Retries, exponential backoff, and DLQs will be introduced in Part 2.

Propagating message identity for idempotency

Each outbox event has a globally unique identifier (outbox_events.id).
We reuse this identifier as the message ID when publishing to RabbitMQ.

Instead of embedding it in the payload, we attach it as a message header:

amqp091.Publishing{
    ContentType: "application/json",
    Body:        messageData,
    Headers: amqp091.Table{
        "message_id": messageID,
    },
}

This gives us a stable, explicit identity for every message that flows through the system.

Note
In the article-1 tag, the message ID is propagated using a custom message_id header. In later revisions, this was updated to use the native amqp091.Publishing.MessageId field instead. This is the default and recommended way to propagate message identity in AMQP, while preserving the same idempotency guarantees described here.

On the consumer side, this message_id is used as the primary key in the processed_messages table.

This approach gives us exactly-once processing semantics at the consumer, even though:

Message delivery from RabbitMQ is at-least-once
Publishers or consumers may crash and restart
Messages may be redelivered

By propagating the outbox event ID all the way to the consumer, we create a single source of truth for message identity across services.

Using RabbitMQ Topic Exchanges

Events are published to a topic exchange, not directly to queues.

Each outbox event stores a routing key in outbox_events.event_key, for example:

order.created

Queues declare their interest by binding to specific routing keys.
In the Notification Service, the queue binds to order.created, so it automatically receives all order creation events.

This decouples publishers from consumers and allows multiple services to react to the same domain event independently.

Notification Service (Consumer)

The Notification Service is intentionally boring, and that’s a feature.

Its structure mirrors the Order Service, minus the outbox worker. Keeping both services shaped the same reduces cognitive overhead and lets us focus on behavior, not scaffolding.

.
├── cmd
│   └── server
│       └── main.go
├── internal
│   ├── config
│   ├── database
│   │   └── model
│   │       └── processed_message.go
│   ├── http
│   │   └── handler
│   ├── rabbitmq
│   └── service
│       └── processed_message.go
└── schema.sql

This service exists to demonstrate consumer-side idempotency.

Consumer-side Idempotency

Message handling and deduplication are performed inside a single database transaction.

The flow is simple:

Receive a message from RabbitMQ
Extract the message_id header
Attempt to insert it into processed_messages
Perform side effects only if the insert succeeds

If the message has already been processed, the insert is a no-op and the handler exits safely.

func (r *RabbitMQ) handleMessage(ctx context.Context, message amqp091.Delivery) (err error) {
    var body map[string]any
    if err = json.Unmarshal(message.Body, &body); err != nil {
        return err
    }

    messageID, ok := message.Headers["message_id"].(string)
    if !ok {
        return errors.New("invalid message id")
    }

    return service.WithTransaction(ctx, r.DB, func(tx *gorm.DB) error {
        inserted, err := r.ProcessedMessageService.TryInsert(
            ctx,
            tx,
            &model.ProcessedMessage{
                MessageID:   messageID,
                ProcessedAt: time.Now(),
            },
        )
        if err != nil {
            return err
        }

        // Idempotency: message already processed → commit & ack
        if !inserted {
            return nil
        }

        r.Log.Info(
            "Order email sent to customer",
            logger.Field{Key: "payload", Value: body},
        )

        return nil
    })
}

Deduplication with a Unique Constraint

The processed_messages table uses the message ID as a primary key.
Insertion relies on ON CONFLICT DO NOTHING, allowing us to detect duplicates without locking or coordination.

func (p *processedMessageService) TryInsert(
    ctx context.Context,
    tx *gorm.DB,
    row *model.ProcessedMessage,
) (bool, error) {

    query := `
        INSERT INTO processed_messages (message_id, processed_at)
        VALUES (?, ?)
        ON CONFLICT (message_id) DO NOTHING
    `

    res := tx.WithContext(ctx).Exec(
        query,
        row.MessageID,
        row.ProcessedAt,
    )

    if res.Error != nil {
        return false, res.Error
    }

    return res.RowsAffected > 0, nil
}

RowsAffected tells us whether this message was processed for the first time:

> 0 → new message, proceed
= 0 → duplicate delivery, skip

This pattern is cheap, deterministic, and works across restarts.

End-to-end test with Docker Compose

To validate the full flow, we run everything using Docker Compose.

The setup includes:

Two services: Order and Notification
Two isolated Postgres databases (one per service)
A shared RabbitMQ broker

Each service:

Owns its own schema
Initializes its database at startup
Waits for Postgres and RabbitMQ health checks before starting

There is no shared database between services.

Start the system:

docker compose up

Send a request to create an order:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"product_id":"sku123","quantity":1}' \
  http://localhost:4000/orders | jq

A successful run shows the request flowing through every layer:

# HTTP request arrives
order-service         | {"level":"info","message":"Create Order Request Arrived"}

# Outbox worker picks up the event
order-service         | {"level":"info","count":1,"message":"Fetched outbox events"}
order-service         | {"level":"info","worker_id":2,"event_id":"a6f6d2df-f2f9-4a0f-98b7-73c99e96f75b","event_key":"order.created","message":"Worker processing event"}
order-service         | {"level":"info","routing_key":"order.created","message":"RabbitMQ message published"}

# Notification service consumes the message
notification-service  | {"level":"info","message":"Broker Message Arrived"}
notification-service  | {"level":"info","payload":{"id":5,"product_id":"sku123","quantity":1},"message":"Order email sent to customer"}

At this point, the basic outbox + consumer idempotency flow is complete.

Messages are:

Written atomically to the database
Published asynchronously
Delivered at least once
Processed exactly once

In Part 2, we’ll build on this foundation by adding retries, backoff, dead-letter queues, and more realistic failure handling, without changing the core model.

What to Do with Published Outbox Events

The outbox table is a coordination mechanism, not a long-term event store.
Once an event has been successfully published to the broker, it has reached a terminal state from the outbox’s perspective.

However, the pattern itself does not mandate what happens to rows after publication. This is a deliberate design choice left to the system owner.

Two common approaches are worth calling out.

Hard delete after publish

One option is to delete outbox rows immediately after a successful publish.

This keeps the system simple:

the outbox table remains small
polling queries stay fast and predictable
there is no ambiguity about which events still require work

In this model, the message broker becomes the source of truth after publication, and observability (metrics, logs, traces) is used to understand what happened rather than inspecting historical rows.

This approach works well when:

events are transient
replay from the database is not required
operational visibility is handled elsewhere

Retention window (delayed deletion)

Another option is to retain published events for a limited period before deleting them.

Typically:

events are marked as published
workers ignore published rows
a background job periodically deletes rows older than a retention threshold

This provides:

a short-lived audit trail
easier post-incident inspection
the ability to manually replay very recent events if necessary

The trade-off is added complexity:

table growth over time
cleanup jobs that must be maintained
additional indexing considerations

Correctness boundary

Regardless of the approach chosen:

only events in a terminal state should ever be deleted
in-progress or retryable events must remain intact
failed events should be handled explicitly (for example, via retries or a DLQ) before removal

Operational Metrics

Even a minimal outbox implementation needs some visibility. Without metrics, failures usually surface only when downstream consumers silently stop doing useful work.

For this article, we intentionally expose a small, high-signal set of Prometheus metrics, just enough to reason about correctness and system health, without turning this into an observability deep dive.

Outbox backlog

outbox_backlog

This gauge represents the number of outbox events waiting to be processed.

It answers the most important question in an outbox system:

Are events piling up faster than we can publish them?

A steadily growing backlog indicates one of the following:

the outbox worker is down or stuck
publishing to the broker is slow or failing
worker concurrency is insufficient for the current load

In our implementation, the backlog is updated periodically (e.g. every 10 seconds) by counting pending events in the database.

func (o *outboxEventService) CountBacklog(ctx context.Context) (int64, error) {
    var count int64

    err := o.db.WithContext(ctx).
        Model(&model.OutboxEvent{}).
        Where(`
            status = ?
            OR (
                status = ?
                AND locked_at < NOW() - INTERVAL '30 seconds'
            )
        `,
            model.OutboxEventStatusPending,
            model.OutboxEventStatusInProgress,
        ).
        Count(&count).Error

    return count, err
}

This is typically the first metric you alert on.

Event outcomes

outbox_events_total{status="published|failed"}

This counter tracks how many outbox events were successfully published versus how many failed.

It allows you to:

spot spikes in publishing failures
compute success/failure ratios over time
distinguish “slow but working” from “actively broken”

In practice, an alert would trigger if:

failures suddenly increase
or failures exceed a small percentage of total events

Publish latency

outbox_publish_latency_seconds

This histogram measures the end-to-end delay from when an outbox event is written to the database to when it is successfully published to the broker.

It captures more than just RabbitMQ latency:

database contention
worker scheduling delays
backpressure from limited concurrency

We record this latency only on successful publishes, using the event’s created_at timestamp as the starting point.

Sustained increases here usually correlate with a growing backlog and are an early signal that the system is under stress.

Recording metrics in the worker

Metrics like outbox_event_total and outbox_publish_latency_seconds are updated directly in the outbox worker during event publication:

func (o *Outbox) PublishEvent(
    ctx context.Context,
    ch *amqp091.Channel,
    event *model.OutboxEvent,
) error {
    err := o.rabbitmq.Publish(
        ctx,
        ch,
        event.EventKey,
        event.Payload,
        event.ID,
    )
    if err != nil {
        metrics.OutboxEventsTotal.WithLabelValues("failed").Inc()
        return err
    }

    metrics.OutboxPublishLatency.Observe(
        time.Since(event.CreatedAt).Seconds(),
    )
    metrics.OutboxEventsTotal.WithLabelValues("published").Inc()

    return nil
}

and outbox_backlog is updated via a separate goroutine:

func (o *Outbox) startBacklogReporter(ctx context.Context) {
    ticker := time.NewTicker(o.config.BacklogReportInterval)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            count, err := o.outboxEventService.CountBacklog(ctx)
            if err != nil {
                o.log.Error("Failed to count outbox backlog",
                    logger.Field{Key: "error", Value: err.Error()},
                )
                continue
            }
            metrics.OutboxBacklog.Set(float64(count))
        }
    }
}

Visualization

To make these metrics actionable, we’ve created a minimal Grafana dashboard for quick insight into system health:

Trace Propagation Through the Outbox

Outbox introduces an asynchronous boundary, which means trace context no longer flows automatically.
If we don’t handle this explicitly, traces will stop at the database commit.

For this article, we use a simple and common pattern:

Capture the incoming trace context
Persist it with the outbox event
Rehydrate it when publishing and consuming messages

Capturing trace context

At the HTTP boundary, OpenTelemetry middleware attaches trace context to context.Context.
When creating an outbox event, we extract the W3C traceparent and store it alongside the event.

traceparent := tracing.ExtractTraceParent(ctx)

outboxEvent := &model.OutboxEvent{
    ID:          uuid.NewString(),
    EventKey:    "order.created",
    Payload:     payload,
    Traceparent: traceparent,
    Status:      model.OutboxEventStatusPending,
}

This keeps the outbox table self-contained and avoids relying on in-memory state.

Outbox worker spans

When the outbox worker publishes events, it reconstructs the trace context and starts a new span:

ctx := tracing.ContextFromTraceParent(event.Traceparent)
ctx, span := tracing.Tracer.Start(ctx, "outbox.publish")
defer span.End()

This links the publish operation back to the original request, even though it may happen seconds later and in a different process.

Consumer spans

On the consumer side, we extract trace headers from RabbitMQ message headers and start a span per message:

ctx = contextWithOtelHeaders(ctx, message.Headers)
ctx, span := tracing.Tracer.Start(ctx, "rabbitmq.consume")
defer span.End()

At this point, the trace is fully reconnected:
HTTP → Order Service → Outbox → RabbitMQ → Consumer.

Why this approach

Works across process and time boundaries
Requires no vendor-specific features
Scales naturally with retries and worker restarts
Easy to reason about during incidents

Other patterns exist (baggage, external context stores), but persisting trace context with the outbox event is simple and explicit.

What’s Next

This article focused on correctness, idempotency, and observability fundamentals. In the next part, we’ll extend this baseline toward real-world failure handling and making the system more resilient and production-ready:

Producer-side retries: Bounded, delayed retries with exponential backoff for outbox events
Dead Letter Queues (DLQs): Handling permanently failed events, microservice ownership, and routing
Consumer-side retries: Using dead-letter exchanges and retry exchanges for safe message processing
Observability: Richer metrics, dashboards, and tracing (Jaeger) with retry and DLQ awareness
CDC-based outbox as an alternative: A brief overview of how change-data-capture can replace polling in some cases

Conclusion

In this article, we built a minimal and practical implementation of the transactional outbox pattern.

We started with the core problem, reliable event publishing across service boundaries, and worked through a concrete solution: persisting events alongside business data, publishing them asynchronously, and handling duplicates safely on the consumer side. Along the way, we added just enough observability, metrics and tracing, to make the system debuggable without overwhelming the design.

The goal wasn’t to build a perfect or fully hardened system, but to establish a correct and observable baseline that behaves predictably under failure. Once those fundamentals are in place, retries, DLQs, and scaling strategies become incremental improvements rather than emergency patches.

Thanks for reading! If you have any questions or insights, please leave a comment.

You can also explore the full working example on GitHub (article-1)

Understanding gRPC Client-Side Load Balancing with DNS

Sagar Maheshwary — Fri, 19 Dec 2025 08:32:44 +0000

gRPC is a high-performance RPC framework widely used in microservices architectures. Because gRPC is built on HTTP/2, multiple RPCs are multiplexed over a single long-lived TCP connection. This makes traditional load balancing approaches such as placing gRPC services behind a simple L4 or L7 load balancer ineffective, often resulting in all traffic being sent to a single backend instance.

gRPC addresses this problem by supporting client-side load balancing natively. Instead of relying on an external proxy, the client itself discovers backend instances and decides where each RPC should be sent.

In this article, we’ll explore gRPC client-side load balancing in practice. We’ll build two simple Go services: an API Gateway that exposes a REST endpoint, and an Order Service that implements a GetOrder gRPC RPC. The Order Service will return both the order ID and the instance (container/pod) that handled the request, making load-balancing behavior easy to observe.

We’ll start by running the system using Docker Compose, observe how requests are distributed across multiple Order Service containers, and see what happens when instances go down or come back up. We’ll then configure retries to handle transient failures. Finally, we’ll deploy the same setup to a local Kubernetes cluster using KIND to understand how gRPC load balancing behaves in a Kubernetes environment.

Creating the Services
gRPC Client-Side Load Balancing Basics
Running with Docker Compose
Client-Side Retries
Running on Kubernetes with KIND
Alternatives to Client-Side Load Balancing
Conclusion

Creating the Services

Directory Structure

We’ll keep the code for both services minimal and focused purely on gRPC. Below is the directory structure we’ll be building:

.
├── api-gateway/
│   ├── cmd/
│   ├── proto/
│   └── internal/
│     ├── grpc/
│     └── http/
├── order-service/
│   ├── cmd/
│   ├── proto/
│   └── internal/
│     ├── grpc/
│     └── http/
├── k8s/
└── docker-compose.yaml

We use a REST endpoint in the API Gateway purely as a convenience layer. It allows us to trigger gRPC requests using simple curl commands and observe load-balancing behavior without requiring a dedicated gRPC client tool. This setup also mirrors real-world microservice architectures, where API gateways often expose HTTP/REST APIs while communicating with backend services over gRPC.

API Gateway

Let's start with the API Gateway and bootstrap a Go project:

mkdir api-gateway
cd api-gateway
go mod init github.com/sagarmaheshwary/go-grpc-load-balancing/api-gateway

Create the proto definition in proto/order/order.proto:

syntax = "proto3";

package order;

option go_package = "github.com/SagarMaheshwary/go-grpc-load-balancing/api-gateway/proto/order";

service OrderService {
  rpc GetOrder(GetOrderRequest) returns (GetOrderResponse);
}

message GetOrderRequest {
  string id = 1;
}

message GetOrderResponse {
  string id = 1;
  string served_by = 2;
}

To generate Go code from proto files, you’ll need to install:

protoc compiler
Go plugins for protobuf and gRPC:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Now we can generate the code with below command:

protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative ./proto/order/order.proto

Next, let’s create a minimal gRPC client that establishes a non-TLS connection to the server in internal/grpc/client.go:

package grpc

import (
    "context"
    "log"

    proto "github.com/sagarmaheshwary/go-grpc-load-balancing/api-gateway/proto/order"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func NewClient(ctx context.Context, url string) (*OrderClient, *grpc.ClientConn, error) {
    conn, err := grpc.NewClient(url,
        grpc.WithTransportCredentials(insecure.NewCredentials()),
    )

    if err != nil {
        log.Println("Order gRPC client failed to connect", err)
        return nil, nil, err
    }

    log.Println("Order gRPC client connected on " + url)
    client := NewOrderClient(proto.NewOrderServiceClient(conn))

    return client, conn, nil
}

and the handler for the GetOrder RPC in internal/grpc/handler.go:

package grpc

import (
    "context"

    proto "github.com/sagarmaheshwary/go-grpc-load-balancing/api-gateway/proto/order"
)

type OrderClient struct {
    client proto.OrderServiceClient
}

func NewOrderClient(c proto.OrderServiceClient) *OrderClient {
    return &OrderClient{
        client: c,
    }
}

func (o *OrderClient) GetOrder(ctx context.Context, in *proto.GetOrderRequest) (*proto.GetOrderResponse, error) {
    res, err := o.client.GetOrder(ctx, in)
    if err != nil {
        return nil, err
    }

    return res, nil
}

Now let’s create a simple HTTP server using gin for the REST API in internal/http/server.go:

package http

import (
    "net/http"

    "github.com/gin-gonic/gin"
    "github.com/sagarmaheshwary/go-grpc-load-balancing/api-gateway/internal/grpc"
)

type HTTPServer struct {
    URL    string
    Server *http.Server
}

func NewServer(url string, grpcClient *grpc.OrderClient) *HTTPServer {
    gin.SetMode(gin.ReleaseMode)
    r := gin.New()

    handler := &HTTPHandler{
        GRPCClient: grpcClient,
    }

    r.GET("/livez", handler.Livez)
    r.GET("/readyz", handler.Readyz)
    r.GET("/orders/:id", handler.GetOrder)

    return &HTTPServer{
        URL: url,
        Server: &http.Server{
            Addr:    url,
            Handler: r,
        },
    }
}

func (h *HTTPServer) Serve() error {
    return h.Server.ListenAndServe()
}

Create the HTTP handlers in internal/http/handler.go:

package http

import (
    "net/http"

    "github.com/gin-gonic/gin"
    "github.com/sagarmaheshwary/go-grpc-load-balancing/api-gateway/internal/grpc"
    proto "github.com/sagarmaheshwary/go-grpc-load-balancing/api-gateway/proto/order"
)

type HTTPHandler struct {
    GRPCClient *grpc.OrderClient
}

func (h *HTTPHandler) Livez(c *gin.Context) {
    c.JSON(http.StatusOK, map[string]string{"status": "ok"})
}

func (h *HTTPHandler) Readyz(c *gin.Context) {
    c.JSON(http.StatusOK, map[string]string{"status": "ready"})
}

func (h *HTTPHandler) GetOrder(c *gin.Context) {
    orderID := c.Param("id")
    if orderID == "" {
        c.JSON(http.StatusBadRequest, map[string]string{"error": "missing order id"})
        return
    }

    req := &proto.GetOrderRequest{
        Id: orderID,
    }
    res, err := h.GRPCClient.GetOrder(c.Request.Context(), req)
    if err != nil {
        // For demo simplicity, map all gRPC errors to 500.
        // In real systems, you'd inspect status.Code(err).
        c.JSON(http.StatusInternalServerError, map[string]string{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, res)
}

Apart from the /orders endpoint, we also expose /livez and /readyz endpoints for Kubernetes health probes. We’ll discuss their role in more detail later, when running the services on KIND.

Now let’s bootstrap the gRPC client and HTTP server in cmd/server/main.go, which serves as the API Gateway’s entrypoint:

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "time"

    "github.com/sagarmaheshwary/go-grpc-load-balancing/api-gateway/internal/grpc"
    httpserver "github.com/sagarmaheshwary/go-grpc-load-balancing/api-gateway/internal/http"
)

var (
    httpServerURL       = "0.0.0.0:4000"
    grpcOrderServiceURL = "order-service:5000"
)

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    grpcClient, conn, err := grpc.NewClient(ctx, grpcOrderServiceURL)
    if err != nil {
        //An API Gateway should not crash if the downstream service is down.
        //However, for simplicity, we are logging the error and exiting.
        log.Fatal(err)
    }
    defer conn.Close()

    httpServer := httpserver.NewServer(httpServerURL, grpcClient)
    go func() {
        log.Println("Starting HTTP server on", httpServerURL)
        err := httpServer.Serve()
        if err != nil && !errors.Is(err, http.ErrServerClosed) {
            stop()
        }
    }()

    log.Println("API Gateway is running...")

    <-ctx.Done()
    log.Println("shutting down API Gateway...")

    shutdownCtx, cancelShutdown := context.WithTimeout(context.Background(), 3*time.Second)
    if err := httpServer.Server.Shutdown(shutdownCtx); err != nil {
        log.Fatalf("failed to shutdown HTTP server: %v", err)
    }
    cancelShutdown()

    log.Println("API Gateway stopped")
}

The main function also handles graceful shutdown by listening for the os.Interrupt (SIGINT) signal and properly shutting down the HTTP server and gRPC connection before exiting.

Order Service

The Order Service is similar to the API Gateway, except it exposes a gRPC server along with a lightweight HTTP server for health checks.

Let’s create the Order Service:

cd ..
mkdir order-service
cd order-service
go mod init github.com/sagarmaheshwary/go-grpc-load-balancing/order-service

For order.proto, we reuse the proto file from the API Gateway, update the go_package path, and regenerate the code.

Create the GetOrder handler in internal/grpc/handler.go. This handler simply returns the same order ID along with a ServedBy value. The ServedBy field helps us visually confirm which instance handled the request during load-balancing experiments. It uses the INSTANCE_ID environment variable if present, or falls back to the hostname:

package grpc

import (
    "context"
    "log"
    "os"

    proto "github.com/sagarmaheshwary/go-grpc-load-balancing/order-service/proto/order"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)

type OrderServer struct {
    proto.OrderServiceServer
}

func (o *OrderServer) GetOrder(ctx context.Context, in *proto.GetOrderRequest) (*proto.GetOrderResponse, error) {
    instanceID := os.Getenv("INSTANCE_ID")
    if instanceID == "" {
        instanceID, _ = os.Hostname()
    }

    log.Println("Order request received for ID:", in.Id, "served by instance:", instanceID)

    return &proto.GetOrderResponse{
        Id:       in.Id,
        ServedBy: instanceID,
    }, nil
}

Next, create the gRPC server in internal/grpc/server.go and register the OrderServer:

package grpc

import (
    "log"
    "net"

    proto "github.com/sagarmaheshwary/go-grpc-load-balancing/order-service/proto/order"
    "google.golang.org/grpc"
)

type GRPCServer struct {
    Server *grpc.Server
    URL    string
}

func NewServer(url string) *GRPCServer {
    srv := grpc.NewServer()

    proto.RegisterOrderServiceServer(srv, &OrderServer{})

    return &GRPCServer{
        Server: srv,
        URL:    url,
    }
}

func (s *GRPCServer) Serve() error {
    lis, err := net.Listen("tcp", s.URL)
    if err != nil {
        log.Printf("failed to listen on %s: %v", s.URL, err)
        return err
    }
    return s.Server.Serve(lis)
}

We also need an HTTP server to expose health-check endpoints.

internal/http/handler:

package http

import (
    "net/http"

    "github.com/gin-gonic/gin"
)

func Livez(c *gin.Context) {
    c.JSON(http.StatusOK, map[string]string{"status": "ok"})
}

func Readyz(c *gin.Context) {
    c.JSON(http.StatusOK, map[string]string{"status": "ready"})
}

internal/http/server.go:

package http

import (
    "net/http"

    "github.com/gin-gonic/gin"
)

type HTTPServer struct {
    URL    string
    Server *http.Server
}

func NewServer(url string) *HTTPServer {
    gin.SetMode(gin.ReleaseMode)
    r := gin.New()

    r.GET("/livez", Livez)
    r.GET("/readyz", Readyz)

    return &HTTPServer{
        URL: url,
        Server: &http.Server{
            Addr:    url,
            Handler: r,
        },
    }
}

func (h *HTTPServer) Serve() error {
    return h.Server.ListenAndServe()
}

Finally, create the main function in cmd/server/main.go:

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "time"

    grpcserver "github.com/sagarmaheshwary/go-grpc-load-balancing/order-service/internal/grpc"
    httpserver "github.com/sagarmaheshwary/go-grpc-load-balancing/order-service/internal/http"
    "google.golang.org/grpc"
)

var (
    httpServerURL = "0.0.0.0:4000"
    grpcServerURL = "0.0.0.0:5000"
)

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    httpServer := httpserver.NewServer(httpServerURL)
    go func() {
        log.Println("Starting HTTP server on", httpServerURL)
        err := httpServer.Serve()
        if err != nil && !errors.Is(err, http.ErrServerClosed) {
            stop()
        }
    }()

    grpcServer := grpcserver.NewServer(grpcServerURL)
    go func() {
        log.Println("Starting gRPC server on", grpcServerURL)
        if err := grpcServer.Serve(); err != nil && errors.Is(err, grpc.ErrServerStopped) {
            stop()
        }
    }()

    log.Println("Order Service is running...")

    <-ctx.Done()
    log.Println("shutting down Order Service...")

    grpcServer.Server.GracefulStop()

    shutdownCtx, cancelShutdown := context.WithTimeout(context.Background(), 3*time.Second)
    if err := httpServer.Server.Shutdown(shutdownCtx); err != nil {
        log.Fatalf("failed to shutdown HTTP server: %v", err)
    }
    cancelShutdown()

    log.Println("Order Service stopped")
}

gRPC Client-Side Load Balancing Basics

In client-side load balancing, the gRPC client resolves multiple backend addresses, maintains a sub-connection to each backend, and uses a load balancing policy to decide which backend handles each RPC. The server is completely unaware of the load balancing; all logic lives inside the gRPC client.

This model is particularly well-suited for Kubernetes and modern microservices because it is more efficient, avoiding an extra proxy hop and works well with long-lived HTTP/2 connections, which gRPC relies on. For many gRPC workloads, this approach is preferred when you want direct client-to-backend communication and simple, transparent load balancing.

Service Discovery via DNS

The simplest and most common form of client-side load balancing uses DNS-based service discovery. When we configure the gRPC client with a DNS target like dns:///order-service:5000, the DNS resolver returns multiple A records, with each record representing a backend instance (pod or container). The gRPC client then creates a SubConn for each of these addresses, and the configured load balancing policy distributes RPCs across them.

In Docker Compose, this works because multiple containers can share the same hostname, and Docker DNS will return all container IPs. In Kubernetes, we achieve the same behavior using Headless Services (clusterIP: None), where each pod gets its own DNS A record that the gRPC client can discover and use for load balancing.

Load Balancing Policies in gRPC

gRPC supports multiple load balancing policies. The most important ones are:

Pick First (Default)

Client picks one backend and sticks to it
No real load balancing
Bad fit for scalable systems

Round Robin

RPCs are distributed across all healthy backends
Most commonly used policy
Ideal for stateless services

Weighted load balancing allows sending different proportions of traffic to different backends (for example, 90% to v1 and 10% to v2).

This is not supported with plain DNS-based client-side load balancing. In gRPC, weighted policies are typically implemented using xDS and Envoy and are commonly used for canary deployments and gradual rollouts.

We will use Round Robin policy for this article.

To enable it, update the internal/grpc/client.go in API Gateway:

conn, err := grpc.NewClient(url,
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithDefaultServiceConfig(`{
        "loadBalancingPolicy":"round_robin"
    }`),
)

See this proto to find out all the supported values for service config.

Running with Docker Compose

Dockerfile and Docker Compose

We start by creating a Dockerfile for each service:

FROM golang:1.25 AS builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
    go build -o /app/main ./cmd/server/main.go

FROM alpine:3.22 AS production

WORKDIR /app
COPY --from=builder /app/main .

EXPOSE 4000 5000

CMD ["./main"]

Next, we create a docker-compose.yml in the root directory:

services:
  api-gateway:
    build:
      context: ./api-gateway
      target: production
    ports:
      - 4000:4000

  order-service-1:
    build:
      context: ./order-service
      target: production
    hostname: order-service
    environment:
      - INSTANCE_ID=order-service-1

  order-service-2:
    build:
      context: ./order-service
      target: production
    hostname: order-service
    environment:
      - INSTANCE_ID=order-service-2

  order-service-3:
    build:
      context: ./order-service
      target: production
    hostname: order-service
    environment:
      - INSTANCE_ID=order-service-3

Here, we are creating three order-service containers and assigning them the same hostname order-service. Docker DNS will return all container IPs when queried, enabling client-side load balancing.

Testing Load Balancing

We can now start the services:

docker compose up

Making a request to the API Gateway:

curl localhost:4000/orders/1

Should return a response like:

{ "id": "1", "served_by": "order-service-1" }

Repeated requests will be distributed across all containers:

{"id":"1","served_by":"order-service-2"}
{"id":"1","served_by":"order-service-3"}
{"id":"1","served_by":"order-service-1"}

Each container also logs the request it serves:

order-service-1-1  | Order request received for ID: 1 served by instance: order-service-1
order-service-2-1  | Order request received for ID: 1 served by instance: order-service-2
order-service-3-1  | Order request received for ID: 1 served by instance: order-service-3

Inspecting gRPC Internal Logs

For a deeper look at client-side load balancing, we can enable verbose gRPC logs:

environment:
  - GRPC_GO_LOG_SEVERITY_LEVEL=info
  - GRPC_GO_LOG_VERBOSITY_LEVEL=99

On the first request, the gRPC client resolves DNS and establishes one SubConn per backend:

Subchannel picks a new address "172.19.0.5:5000"
Subchannel picks a new address "172.19.0.2:5000"
Subchannel picks a new address "172.19.0.4:5000"

This confirms that the client discovered all three containers and created independent connections to each.

Stopping and Restarting Backends

Let’s stop one backend container:

docker compose stop order-service-1

The gRPC client detects the closed TCP connection and logs:

Closing: connection error: desc = "error reading from server: EOF"

The affected SubChannel transitions through:

READY → IDLE → CONNECTING → TRANSIENT_FAILURE → SHUTDOWN

Eventually, it is removed from the load balancer, and traffic is routed only to healthy instances.

Restarting the container:

docker compose start order-service-1

Does not immediately change client behavior. gRPC maintains long-lived HTTP/2 connections and will not refresh DNS while all existing connections are healthy.

Improving Responsiveness with Keepalive

To make the client more responsive to backend changes, we can configure keepalives:

conn, err := grpc.NewClient(
    url,
    grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithKeepaliveParams(keepalive.ClientParameters{
        Time:                30 * time.Second,
        Timeout:             5 * time.Second,
        PermitWithoutStream: false,
    }),
)

Keepalive does not refresh DNS by itself. It periodically pings backends to detect unresponsive connections. When a backend fails, the resulting state change triggers internal reconnection logic, which may include DNS re-resolution—allowing new backends to be discovered earlier.

DNS-based client-side load balancing is reactive by design. It prioritizes connection stability over aggressive backend discovery. For most Kubernetes use cases like pod crashes, restarts, and rolling updates naturally trigger DNS refreshes.

Client-Side Retries

Why Client-Side Retries Matter

In a distributed system, failures are expected rather than exceptional. Even when client-side load balancing is in place, individual backends can fail due to pod crashes, container restarts during rolling deployments, transient network issues, or momentary server overload. Without retries, a single transient failure immediately propagates as an error to the caller, despite the presence of other healthy backends.

Client-side retries allow the gRPC client to handle these failures gracefully. When a request fails, the client can mark the backend as unhealthy, select another backend from the load balancer, and retry the RPC transparently. This improves overall availability without pushing retry logic into application code.

When to Retry

Retries at the load-balancing layer are intentionally conservative. Only the UNAVAILABLE gRPC status code is retried automatically. UNAVAILABLE indicates that the server could not be reached, the connection was dropped, or the backend was temporarily unable to serve requests. This usually means the request never reached application logic, making it safe to retry on another backend.

Other status codes are not retried at the LB level because retries could introduce correctness issues:

DEADLINE_EXCEEDED: server may still be processing the request
ABORTED: often signals concurrency conflicts
INVALID_ARGUMENT or FAILED_PRECONDITION: indicate client or business-logic errors where retries are ineffective

For application-level errors, a custom interceptor is recommended, applying retries only to explicitly idempotent RPCs.

Configuring Client-Side Retries in Go

In gRPC Go, retries are configured using the service config. Here, we extend our client configuration to enable retries for the OrderService:

conn, err := grpc.NewClient(url,
    grpc.WithDefaultServiceConfig(`{
        "loadBalancingPolicy":"round_robin",
        "methodConfig":[
            {
                "name":[{"service":"order.OrderService"}],
                "retryPolicy":{
                    "maxAttempts":3,
                    "initialBackoff":"0.1s",
                    "maxBackoff":"1s",
                    "backoffMultiplier":2,
                    "retryableStatusCodes":["UNAVAILABLE"]
                }
            }
        ]
    }`),
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithKeepaliveParams(keepalive.ClientParameters{
        Time:                30 * time.Second,
        Timeout:             5 * time.Second,
        PermitWithoutStream: false,
    }),
)

This configuration allows up to three attempts per RPC (one original request + two retries) using exponential backoff: 100ms → 200ms → capped at 1s. Retries are restricted to UNAVAILABLE, ensuring safe and meaningful retry behavior.

Demonstrating Retries in Action

If one Order Service container becomes unhealthy, the gRPC client retries the request on another container selected by the round-robin policy without exposing failure to the caller.

We can simulate failure in the GetOrder RPC using the GRPC_FORCE_UNAVAILABLE environment variable:

func (o *OrderServer) GetOrder(ctx context.Context, in *proto.GetOrderRequest) (*proto.GetOrderResponse, error) {
    if os.Getenv("GRPC_FORCE_UNAVAILABLE") == "true" {
        log.Println("gRPC simulated failure, another instance will handle the request!")
        return nil, status.Error(codes.Unavailable, "simulated failure")
    }

    instanceID := os.Getenv("INSTANCE_ID")
    if instanceID == "" {
        instanceID, _ = os.Hostname()
    }

    log.Println("Order request received for ID:", in.Id, "served by instance:", instanceID)

    return &proto.GetOrderResponse{
        Id:       in.Id,
        ServedBy: instanceID,
    }, nil
}

Enable this mode for one container in docker-compose.yaml:

order-service-1:
    build:
        context: ./order-service
        target: production
    hostname: order-service
    environment:
        - INSTANCE_ID=order-service-1
        - GRPC_FORCE_UNAVAILABLE=true

Observing Retry Behavior

With all containers running, multiple requests demonstrate automatic retries. Only healthy backends respond:

{"id":"1","served_by":"order-service-2"}
{"id":"1","served_by":"order-service-3"}
{"id":"1","served_by":"order-service-2"}

Logs from the containers confirm the process:

order-service-1-1  | 2025/12/16 17:46:39 gRPC simulated failure, another instance will handle the request!
order-service-2-1  | 2025/12/16 17:46:39 Order request received for ID: 1 served by instance: order-service-2
order-service-3-1  | 2025/12/16 17:46:42 Order request received for ID: 1 served by instance: order-service-3
order-service-1-1  | 2025/12/16 17:46:45 gRPC simulated failure, another instance will handle the request!
order-service-2-1  | 2025/12/16 17:46:45 Order request received for ID: 1 served by instance: order-service-2

This illustrates how client-side retries and round-robin load balancing work together: failures are isolated, healthy backends continue serving traffic, and the system remains resilient without additional application-level retry logic.

Running on Kubernetes with KIND

Now let’s run the same setup on Kubernetes and see how client-side gRPC load balancing works using a Headless Service to return all Pod IPs to the client.

We’ll use KIND (Kubernetes IN Docker) to run a local Kubernetes cluster. Make sure you have KIND and kubectl installed.

We’ll keep the manifests minimal and focus only on what’s required for gRPC-based client-side load balancing.

API Gateway manifests

For the API Gateway, we’ll create k8s/api-gateway.yaml with a Deployment and a standard ClusterIP Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
  labels:
    app: api-gateway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:
        - name: api-gateway
          image: api-gateway:latest
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /livez
              port: 4000
          readinessProbe:
            httpGet:
              path: /readyz
              port: 4000
          env: # Optional env variables for gRPC logging
            - name: GRPC_GO_LOG_SEVERITY_LEVEL
              value: "info"
            - name: GRPC_GO_LOG_VERBOSITY_LEVEL
              value: "99"

---
apiVersion: v1
kind: Service
metadata:
  name: api-gateway
  labels:
    app: api-gateway
spec:
  type: ClusterIP
  selector:
    app: api-gateway
  ports:
    - name: http
      protocol: TCP
      port: 4000
      targetPort: 4000

This Service is only used to expose the API Gateway internally and for port-forwarding during local testing.

Order Service manifests (Headless Service)

Next, we’ll create k8s/order-service.yaml for the Order Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
        - name: order-service
          image: order-service:latest
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /livez
              port: 4000
          readinessProbe:
            httpGet:
              path: /readyz
              port: 4000

---
apiVersion: v1
kind: Service
metadata:
  name: order-service
  labels:
    app: order-service
spec:
  clusterIP: None
  selector:
    app: order-service
  ports:
    - name: grpc
      protocol: TCP
      port: 5000
      targetPort: 5000

We’re running three replicas of the Order Service.
The key detail here is clusterIP: None, which makes this a Headless Service.

A Headless Service does not allocate a virtual ClusterIP. Instead, Kubernetes publishes one DNS record per Pod IP. When the gRPC client resolves order-service, it receives a list of all backing Pod IPs, which is exactly what we want for client-side load balancing.

Why probes matter for DNS-based LB

Although liveness and readiness probes are important for all services, they are especially important here:

When a Pod is created and becomes Ready, it is added to DNS
When a Pod becomes NotReady or is terminated, it is removed from DNS
The gRPC client will only see Ready Pods when it re-resolves DNS

In this demo, /livez and /readyz simply return 200 OK.
In a real system, /readyz would check service dependencies like Redis, Kafka, RabbitMQ, or databases and return 503 if the service is not ready to handle traffic.

Creating the KIND cluster

Let’s create a local Kubernetes cluster using KIND:

kind create cluster
kubectl config set-context kind-kind

One advantage of KIND is that it can load Docker images directly, so we don’t need a container registry. This works because our Deployments use imagePullPolicy: IfNotPresent.

Build and load the images:

docker build -t api-gateway:latest ./api-gateway
docker build -t order-service:latest ./order-service

kind load docker-image api-gateway:latest
kind load docker-image order-service:latest

Apply all manifests:

kubectl apply -f k8s/

Watch the Pods come up:

kubectl get pods -w

Once everything is ready, port-forward the API Gateway so we can access it from our host machine:

kubectl port-forward svc/api-gateway 4000:4000

Verifying load balancing

Now let’s send a few requests to the API Gateway:

curl localhost:4000/orders/1

We should see responses coming from different Pods:

{"id":"1","served_by":"order-service-644849c549-prxms"}
{"id":"1","served_by":"order-service-644849c549-lbbp7"}
{"id":"1","served_by":"order-service-644849c549-cp8cp"}

In Kubernetes, os.Hostname() resolves to the Pod name, which makes it easy to see which Pod handled each request.

We can also verify this via logs:

kubectl logs -l app=order-service -f

2025/12/17 08:43:07 Order request received for ID: 1 served by instance: order-service-644849c549-prxms
2025/12/17 08:43:08 Order request received for ID: 1 served by instance: order-service-644849c549-lbbp7
2025/12/17 08:43:08 Order request received for ID: 1 served by instance: order-service-644849c549-cp8cp

Testing retries with a faulty Pod

To test retries on the UNAVAILABLE status, we’ll deploy a deliberately faulty Order Service instance.

Create k8s/order-service-faulty.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-faulty
  labels:
    app: order-service-faulty
spec:
  replicas: 1
  selector:
    matchLabels:
      app: order-service
      type: faulty
  template:
    metadata:
      labels:
        app: order-service
        type: faulty
    spec:
      containers:
        - name: order-service
          image: order-service:latest
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /livez
              port: 4000
          readinessProbe:
            httpGet:
              path: /readyz
              port: 4000
          env:
            - name: INSTANCE_ID
              value: order-service-faulty
            - name: GRPC_FORCE_UNAVAILABLE
              value: "true"

Both Deployments share the label app: order-service, so the Headless Service will now return four Pod IPs: three healthy Pods and one faulty one.

Apply the manifest:

kubectl apply -f k8s/order-service-faulty.yaml

Once the Pod is ready, scale down the original Order Service to trigger a DNS update on the client:

kubectl scale --replicas 2 deployment order-service

Make a few requests and check the logs again:

2025/12/17 08:46:51 gRPC simulated failure, another instance will handle the request!
2025/12/17 08:46:51 Order request received for ID: 1 served by instance: order-service-644849c549-prxms
2025/12/17 08:46:52 Order request received for ID: 1 served by instance: order-service-644849c549-lbbp7
2025/12/17 08:47:02 gRPC simulated failure, another instance will handle the request!
2025/12/17 08:47:02 Order request received for ID: 1 served by instance: order-service-644849c549-prxms

The faulty Pod consistently returns UNAVAILABLE, and the gRPC client transparently retries the request on a healthy Pod using the round-robin policy.

With this, we’ve validated gRPC client-side load balancing using DNS on both Docker Compose (for quick local experimentation) and Kubernetes with KIND, showing that the same principles apply in production-like environments.

Alternatives to Client-Side Load Balancing

Client-side load balancing using DNS works well because it is simple, transparent, and has very few moving parts. For many internal microservice setups, especially stateless services running on Kubernetes, this approach is often sufficient and production-ready.

However, DNS-based client-side load balancing has clear limits. The client reacts to failures and topology changes, but it does not actively manage traffic. We don’t get instant rebalancing, fine-grained routing rules, or full visibility into per-request behavior beyond what the client exposes.

A common alternative is proxy-based load balancing, for example with Envoy, typically configured through xDS APIs. The mental model here is different: instead of each client making load-balancing decisions, traffic flows through a dedicated proxy with a global view of the backend topology. This enables more advanced capabilities such as dynamic configuration updates, traffic shifting, consistent retries and timeouts, and richer observability.

For many teams, DNS-based client-side load balancing is a pragmatic first step. More advanced solutions can be introduced later, once the system’s complexity and traffic patterns actually justify them.

Conclusion

Thanks for reading! In this article, we explored how gRPC client-side load balancing works using DNS-based discovery, first with Docker Compose and then on Kubernetes using a Headless Service and KIND. We looked at how gRPC manages long-lived connections, how failures trigger re-resolution, and how retries on UNAVAILABLE help route traffic away from unhealthy backends.

While DNS-based client-side load balancing doesn’t aggressively discover new backends, it provides a practical, production-proven approach for many internal microservice workloads and is a great way to understand the fundamentals before moving to more advanced approaches like xDS or service meshes.

If you found this useful, I’d love to hear your thoughts in the comments, and feel free to check out the full working example on GitHub: https://github.com/SagarMaheshwary/go-grpc-load-balancing

Kubernetes Liveness vs Readiness Probes: What They Actually Mean

Sagar Maheshwary — Fri, 05 Dec 2025 15:43:46 +0000

Health checks are essential in distributed systems. They help Kubernetes and load balancers understand the state of your service. But not all health checks serve the same purpose.

This post explains the difference between liveness and readiness probes, why they exist, and how to implement them correctly in real microservices.

Liveness Probe

A liveness probe answers one question:
Is the application still running, or should Kubernetes restart it?

Key points:

Designed to detect deadlocks, stuck event loops, memory corruption, or unrecoverable internal states.
A failing liveness probe results in a container restart.
Should be lightweight and must not depend on external systems (database, cache, message broker, etc.).
If your liveness probe depends on a database and the database has an outage, your pod will get stuck in a restart loop — which is exactly what you do not want.

A simple 200 OK is usually enough.

Readiness Probe

A readiness probe answers a different question:
Is this service ready to receive traffic right now?

Key points:

If a readiness probe fails, Kubernetes keeps the pod running but removes it from the load balancer.
Safe to include dependency checks: database, Redis, message broker, external API, cache warmup, etc.
Useful during startup, warm-up, graceful shutdown, or when dependencies temporarily fail.

This ensures your service only receives traffic when it can actually handle it.

HTTP-Based Implementation

The common pattern is to expose two endpoints:

/livez → basic check to confirm the application is alive
/readyz → deeper check to confirm the service is ready

Below are examples in Go and Node.js.

Go Example

Liveness:

func Livez(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("OK"))
}

Readiness with database and Redis checks:

type Dependencies struct {
    DB    *sql.DB
    Redis *redis.Client
}

func (d *Dependencies) Readyz(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()

    if err := d.DB.PingContext(ctx); err != nil {
        http.Error(w, "db not ready", http.StatusServiceUnavailable)
        return
    }

    if err := d.Redis.Ping(ctx).Err(); err != nil {
        http.Error(w, "redis not ready", http.StatusServiceUnavailable)
        return
    }

    w.WriteHeader(http.StatusOK)
    w.Write([]byte("READY"))
}

Kubernetes configuration:

livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

Node.js Example (Express)

Liveness:

app.get("/livez", (req, res) => {
  res.status(200).send("OK");
});

Readiness with DB + Redis:

app.get("/readyz", async (req, res) => {
  try {
    await db.ping();
    await redis.ping();
    res.status(200).send("READY");
  } catch (err) {
    res.status(503).send("NOT READY");
  }
});

gRPC Health Checks

Kubernetes supports gRPC probes via grpc.health.v1.Health/Check.
However, this API only exposes a single health state, making it difficult to cleanly separate liveness from readiness.

Because of this limitation, many real-world teams expose HTTP-based /livez and /readyz endpoints, even when the service itself is gRPC-only.
This approach is simple, flexible, and aligns well with Kubernetes.

Real-World Examples

If you want to see real world implementations, check out:

go-microservice-boilerplate
Includes ready-to-use /livez and /readyz endpoints.
kind-microservices-demo
A demo microservices setup, with KIND and Docker Compose environments.

You can use these as reference implementations or a starting point for your own projects.

Conclusion

Liveness and readiness probes are simple concepts, but misusing them can lead to cascading failures, stuck restarts, or pods receiving traffic during startup.

A good rule of thumb:

Liveness is about process health.
Readiness is about dependency and traffic readiness.

Separate them cleanly, keep liveness lightweight, and let readiness reflect the true runtime health of your system.

Hands-On Kubernetes: Running Event-Driven Microservices Locally with KIND

Sagar Maheshwary — Mon, 01 Dec 2025 19:00:00 +0000

Kubernetes can feel heavy to experiment with, but KIND (Kubernetes IN Docker) makes it easy to run a full, multi-node Kubernetes cluster locally using Docker containers. It’s fast, lightweight, and behaves very similarly to a real cluster, supporting ingress controllers, custom configurations, and multi node setups. This makes KIND an excellent option for testing microservices architectures, experimenting with Kubernetes manifests, or learning cluster-level components without provisioning any cloud infrastructure.

In this article, we’ll walk through running a small event-driven microservices inside a Kind cluster. You’ll learn how to:

Configure and create a KIND cluster
Set up a local Docker registry and integrate it with the cluster
Install NGINX Ingress Controller and expose services
Deploy microservices using ConfigMaps, Secrets, Deployments, and Services
Run RabbitMQ using a StatefulSet
Use readiness/liveness probes for clean startup behavior

The demo contains three simple services:

API Gateway (Go) — exposes a REST /users endpoint
User Service (Go) — publishes a user.created event to RabbitMQ
Notification Service (NestJS) — consumes the event and logs a welcome message

Here’s the architecture we’ll be deploying:

This article focuses on the Kubernetes side, not the application code, so we won’t be walking through any code snippets—but feel free to explore the services in the Github Repo if you're interested.

Repository Structure
KIND Cluster and Registry Setup
The Basics of Kubernetes
Setup NGINX Ingress Controller
Understanding the Microservices
Healthchecks
Graceful Shutdowns
Kubernetes Manifests
Running Everything
Calling the API
Cleaning Up
Automating Everything (Makefile)
Conclusion

Repository Structure

This project is intentionally structured as a single repository to keep the tutorial simple and easy to clone.

├── api-gateway/               # Go service exposing REST APIs
├── user-service/              # Go service publishing RabbitMQ events
├── notification-service/      # NestJS service consuming RabbitMQ events
│
├── k8s/
│   ├── api-gateway/           # Deployment, Service, ConfigMap, Ingress
│   ├── user-service/
│   ├── notification-service/
│   ├── rabbitmq/              # StatefulSet, ConfigMap, Secrets
│   ├── namespace.yaml         # Namespace definition
│   └── kind-config.yaml       # Multi-node KIND cluster + local registry
│
├── Makefile                   # Automates cluster creation and deployments
├── docker-compose.yaml        # Optional for local dev
└── devbox.json                # Reproducible development environment

If you intend to follow along and build everything step-by-step, you can clone the repo and switch to the no-k8s branch, which contains only the microservices:

git clone https://github.com/SagarMaheshwary/kind-microservices-demo.git
cd kind-microservices-demo
git checkout no-k8s

KIND Cluster and Registry Setup

First, you need to install Docker, KIND, and kubectl (the Kubernetes CLI). I’m using Ubuntu for this demo, but these tools are also available for macOS and Windows.

If you prefer, you can install Devbox and simply run devbox shell in the project root. Devbox will read devbox.json and automatically install KIND, kubectl, and cloud-provider-kind for you. (you still need to install Docker separately)

We’ll start by setting up a local Docker registry running as a container. This lets us push and pull images locally without relying on Docker Hub or any external registry.

Run the registry:

docker run -d \
 -p 5000:5000 \
 --net kind \
 -v registry-data:/var/lib/registry \
 --name kind-registry \
 registry:2

--net kind → uses the same Docker network as KIND so the cluster can access kind-registry:5000
registry-data volume → keeps images persistent
registry:2 → official lightweight Docker registry image

Next, we’ll create the KIND cluster. We define k8s/kind-config.yaml that connects KIND to the local registry and sets up a multi-node cluster:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4

nodes:
  - role: control-plane
  - role: worker
  - role: worker

containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."kind-registry:5000"]
      endpoint = ["http://kind-registry:5000"]

Node roles:

control-plane → runs Kubernetes components like the API server, scheduler, and controller manager
worker → runs your Deployments, StatefulSets, and other workloads

Create the cluster:

kind create cluster --config=./k8s/kind-config.yaml

# Set kubectl context to the KIND cluster and default namespace to microservices.
kubectl config set-context kind-kind --namespace=microservices

Verify that it’s running:

kubectl get nodes

You should see one control-plane node and two workers in the Ready state.

Next, we’ll create two namespaces for this demo in k8s/namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: microservices
---
apiVersion: v1
kind: Namespace
metadata:
  name: datastores

microservices → where all application services will run
datastores → where databases and brokers will run (RabbitMQ in our case)

Apply the manifest:

kubectl apply -f ./k8s/namespace.yaml

We’ll also install metrics-server so Kubernetes can report CPU/memory usage for Pods and Nodes (useful for debugging, HPA, and general observability):

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Patch the metrics-server deployment to allow insecure TLS connections to the kubelet (required for KIND).
kubectl patch deployment metrics-server \
  --namespace kube-system \
  --type='json' \
  -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]'

Check for metrics-server readiness:

kubectl get pods -n kube-system -l k8s-app=metrics-server

Now you can use the top command to see CPU and memory usage:

kubectl top nodes

The Basics of Kubernetes

Before deploying our microservices, let’s quickly cover the core Kubernetes concepts used in this demo. This isn’t a deep-dive, just enough practical knowledge to understand the manifests we’ll apply later.

Manifests

Kubernetes resources are defined using manifests, usually in YAML format. These files describe the desired state of your cluster.

apiVersion – defines which Kubernetes API version the resource belongs to
kind – specifies the type of resource (Pod, Deployment, Service, etc.)
metadata – names and labels for identifying the resource
spec – configuration details of the resource

Think of manifests as instructions for Kubernetes, telling it “create/update/delete these objects” to match your declared state.

Core Resources (Quick Overview)

Pod
The smallest deployable unit. A pod can run one or more containers (including sidecars, which are helper containers for logging, metrics, proxies, etc.). Pods are ephemeral, Kubernetes may recreate them at any time.

Deployment
Manages pods for you. Handles rolling updates, restarts, scaling, and ensures the correct number of pods are running. In real-world setups, you almost never create pods directly—Deployments take care of their lifecycle.

Service
Provides a stable network endpoint for accessing pods.

Types of Services:

ClusterIP — internal-only (default)
NodePort — exposes a port on every node
LoadBalancer — creates a cloud load balancer (not supported directly in KIND)
Headless — no cluster IP; used for StatefulSets or service discovery patterns

Ingress
Routes external HTTP(S) traffic into the cluster. Requires an Ingress Controller (we use NGINX Ingress).

ConfigMap
Stores non-sensitive configuration such as environment variables or app configs. Lets you avoid baking configuration into images.

Secret
Stores sensitive values (passwords, tokens). The data is Base64-encoded (Base64 is not encryption, it only ensures safe handling of special characters). In real clusters, Secrets are typically integrated with external secret managers such as AWS Secrets Manager or Vault.

StatefulSet
Used for stateful applications. Unlike Deployments, StatefulSets guarantee:

stable pod names
ordered startup/shutdown
persistent storage attachment

PersistentVolume (PV) / PersistentVolumeClaim (PVC)
Provide persistent disk storage. A PVC is a "request" for storage; a PV fulfills it. KIND auto-provisions storage for PVCs, which is perfect for local testing (e.g., RabbitMQ).

Namespace
Logical grouping of resources. Helps separate environments and teams. In this demo, we’ll use:

microservices
datastores

These fundamentals are enough to follow the manifests we’ll apply later.

Kubectl Basics

We’ll interact with the cluster using kubectl, the Kubernetes command-line tool. This section introduces a few essential commands you’ll commonly use to apply manifests, check resource status, and debug your services. It’s not a complete reference, just enough to follow along with this demo.

Apply a manifest to create or update resources:

kubectl apply -f k8s/user-service/deployment.yaml

Check the status of resources:

kubectl get pods
kubectl get pods -n microservices

This same pattern works for other resources such as Deployments, Services, or Ingress resources:

kubectl get deployment
kubectl get service
kubectl get ingress

Restart or trigger a rollout for a deployment:

kubectl rollout restart deployment/user-service

Kubernetes will gradually replace old pods with new ones while respecting readiness and liveness probes, enabling zero-downtime deployments if your app handles graceful shutdowns.

Create a resource directly without a manifest:

kubectl create configmap example-config --from-literal=LOG_LEVEL=debug

Port forward a service or pod to your host machine for debugging:

kubectl port-forward service/api-gateway 8080:80

You can use labels to filter pods when checking status or viewing logs:

kubectl get pods -l app=user-service
kubectl logs -f pod/<pod-name>

Setup NGINX Ingress Controller

An Ingress Controller is a Kubernetes component that manages external access to services inside the cluster, primarily HTTP/HTTPS traffic. It allows you to define routes, hostnames, and TLS for your services without exposing each one individually as a NodePort or LoadBalancer.

In cloud environments, a Service of type LoadBalancer automatically provisions a real network load balancer (AWS ELB, GCP LB, Azure LB, etc.).
However, this does not work on local Kubernetes distributions like KIND, because there is no cloud provider available to create that external load balancer. This is where cloud-provider-kind comes in.

cloud-provider-kind simulates a cloud environment for KIND by providing a fake external IP for Kubernetes LoadBalancer Services. It watches for Services of type LoadBalancer, allocates a local IP address for them, and routes traffic from that address to the underlying NodePort. This gives KIND clusters production-like behavior for ingress controllers—without requiring a cloud provider.

How traffic flows:

Install the Ingress Controller:

kubectl apply -f https://kind.sigs.k8s.io/examples/ingress/deploy-ingress-nginx.yaml

Download the cloud-provider-kind binary and run it:

curl -L https://github.com/kubernetes-sigs/cloud-provider-kind/releases/download/v0.6.0/cloud-provider-kind_0.6.0_linux_amd64.tar.gz | tar -xzf - cloud-provider-kind
chmod +x cloud-provider-kind
./cloud-provider-kind

Once the Ingress Controller and cloud-provider-kind are running, check the LoadBalancer IP:

kubectl get service -n ingress-nginx

You’ll see the external IP under the EXTERNAL-IP column.

Add it to your OS hosts file (Linux/macOS: /etc/hosts, Windows: C:\Windows\System32\drivers\etc\hosts):

[External-IP] microservices.local

Now, any request to http://microservices.local will be routed through the NGINX Ingress Controller to the correct service and pod inside the cluster.

Understanding the Microservices

Before deploying anything, it’s helpful to understand the core components of our demo system and how they interact.

The API Gateway (Go) exposes a simple REST endpoint /users for clients and forwards requests to the User Service. It acts as the main entry point for all client requests.

The User Service (Go) also exposes a REST endpoint /users (we avoid gRPC here to keep things simple). It stores user information in-memory and publishes a user.created event to RabbitMQ whenever a new user is created.

The Notification Service (NestJS) consumes the user.created events from RabbitMQ and logs a message like “Welcome email sent to [user name]”.

Healthchecks

Each service exposes /livez and /readyz endpoints to enable Kubernetes probes.

The /livez endpoint simply returns HTTP 200 if the service is alive. If Kubernetes does not receive a 200 response, it restarts the pod.
The /readyz endpoint verifies that all dependencies (such as RabbitMQ connections for the User Service and Notification Service) are available before returning HTTP 200. If dependencies aren’t ready, it returns HTTP 503, preventing Kubernetes from sending traffic to the pod until it is fully ready.

Checkout this article for implementing Healthchecks in Golang/NodeJS.

Graceful Shutdowns

All services implement graceful shutdowns. When Kubernetes sends a SIGTERM signal, the services allow in-progress requests to complete, close connections to dependencies like RabbitMQ, and then shut down the HTTP server.

Without graceful shutdowns, requests could be dropped mid-progress, causing potential errors or lost messages. Implementing this ensures smooth rolling updates and proper pod termination during scaling or redeployments.

Kubernetes Manifests

Microservices

Now that we understand the microservices, let’s see how they’re deployed in Kubernetes using manifests. We’ll start with user-service:

Let's create a configmap for our user-service in k8s/user-service/configmap.yaml and define our env variables:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-service
  namespace: microservices
data:
  HTTP_SERVER_URL: "0.0.0.0:4000"
  HTTP_SHUTDOWN_TIMEOUT: "5s"
  GIN_MODE: "release"

  AMQP_HOST: "rabbitmq.datastores.service.cluster.local"
  AMQP_PORT: "5672"
  AMQP_CONNECTION_RETRY_INTERVAL_SECONDS: "5s"
  AMQP_CONNECTION_RETRY_ATTEMPTS: "10"
  AMQP_PUBLISH_TIMEOUT_SECONDS: "2s"

rabbitmq.datastores.service.cluster.local points to the RabbitMQ service in the datastores namespace. Other services can use just the service name if they are in the same namespace, like user-service for the API Gateway to forward requests.

for rabbitmq username/password, we define a secret k8s/user-service/secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: user-service
  namespace: microservices
data:
  AMQP_USERNAME: ZGVmYXVsdA== #base64 for "default"
  AMQP_PASSWORD: ZGVmYXVsdA== #base64 for "default"

Use the following commands to encode/decode Base64 values:

echo -n "hello world" | base64 # -n prevents echo from adding a newline, which would change the encoded value.
echo "aGVsbG8gd29ybGQ=" | base64 --decode

Create a Deployment manifest in k8s/user-service/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
  namespace: microservices
  labels:
    app: user-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
    spec:
      containers:
        - name: user-service
          image: kind-registry:5000/user-service:1.0
          envFrom:
            - configMapRef:
                name: user-service
            - secretRef:
                name: user-service
          ports:
            - containerPort: 4000
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /livez
              port: 4000
            initialDelaySeconds: 20 # Wait 20s for the service to start
            periodSeconds: 10 # Check every 10s
            timeoutSeconds: 3 # Timeout after 3s
            failureThreshold: 4 # Restart pod after 4 consecutive failures (~40s)
          readinessProbe:
            httpGet:
              path: /readyz
              port: 4000
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2 # Mark pod not ready after 2 consecutive failures (~10s)
          resources:
            requests:
              memory: "100Mi"
              cpu: "150m"
            limits:
              memory: "150Mi"
              cpu: "300m"

Key fields in our deployment manifest:

replicas: 1 → Number of identical Pods to create.
selector.matchLabels → Tells the Deployment which Pods it owns. Must match the labels in template.metadata.labels.
template → The actual Pod definition.
image → Container image to deploy.
envFrom (configMapRef and secretRef) → Pull environment variables from ConfigMap/Secret automatically.
ports.containerPort → Exposes port 4000 inside the Pod.
livenessProbe → Detects stuck containers and restarts them.
readinessProbe → Sends traffic only when the app is ready.
initialDelaySeconds → Useful for slow-start apps to avoid false failures.
resources.requests → Minimum guaranteed resources for the Pod (100Mi = 100 MB RAM, 150m = 0.15 CPU cores, checkout resource unit docs).
resources.limits → Maximum resources the container can consume.

Define the Service in k8s/user-service/service.yaml. We use ClusterIP, as this service is internal:

apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: microservices
  labels:
    app: user-service
spec:
  type: ClusterIP
  selector:
    app: user-service
  ports:
    - protocol: TCP
      port: 4000 # exposed by the Service
      targetPort: 4000 # Pod container port
      name: http # A human-readable name for the port; useful for referencing this port in other resources (like NetworkPolicy or Ingress).

Create an Ingress for the API Gateway in k8s/api-gateway/ingress.yaml to expose it externally via the NGINX Ingress Controller:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-gateway
  namespace: microservices
  annotations: # Extra settings for the ingress controller.
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: "nginx" # Uses the NGINX ingress controller.
  rules: # Define host and URL paths, mapping traffic to the appropriate backend service and port.
    - host: microservices.local
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: api-gateway
                port:
                  number: 4000

The Deployment, Service, and Secret manifests for API Gateway and Notification Service follow the same patterns. Only names, labels, and container images differ. ConfigMaps define service-specific settings such as the API Gateway pointing to the User Service, and Notification Service pointing to RabbitMQ.

# API Gateway doesn't need secret.yaml, so make sure to remove "envFrom.secretRef" from copied deployment.yaml
cp ./k8s/user-service/deployment.yaml ./k8s/api-gateway/deployment.yaml
cp ./k8s/user-service/service.yaml ./k8s/api-gateway/service.yaml

mkdir ./k8s/notification-service
cp ./k8s/user-service/deployment.yaml ./k8s/notification-service/deployment.yaml
cp ./k8s/user-service/service.yaml ./k8s/notification-service/service.yaml
cp ./k8s/user-service/secret.yaml ./k8s/notification-service/secret.yaml

Create a ConfigMap for the API Gateway (k8s/api-gateway/configmap.yaml):

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-gateway
  namespace: microservices
data:
  HTTP_SERVER_URL: "0.0.0.0:4000"
  HTTP_SERVER_SHUTDOWN_TIMEOUT: "5s"
  GIN_MODE: "release"

  USER_SERVICE_CLIENT_URL: "http://user-service:4000"
  USER_SERVICE_CLIENT_TIMEOUT: "2s"

And for the Notification Service (k8s/notification-service/configmap.yaml):

apiVersion: v1
kind: ConfigMap
metadata:
  name: notification-service
  namespace: microservices
data:
  HTTP_SERVER_URL: "0.0.0.0:4000"

  AMQP_HOST: "rabbitmq.datastores.service.cluster.local"
  AMQP_PORT: "5672"
  AMQP_QUEUE: "notification-service"

Copy-pasting raw Kubernetes manifests for every service works for demos, but it doesn’t scale well. In production setups, you often package your manifests as a Helm chart and give each microservice its own values.yaml. That file defines only what varies between services, things like labels, image name/tag, environment variables, configs, and resource settings, while all the shared templates live in one place. This keeps your configuration clean, DRY, and much easier to manage as your number of microservices grows.

Setup RabbitMQ

We’ll run RabbitMQ inside the cluster using a StatefulSet.

First, create a ConfigMap for in k8s/user-service/configmap.yaml and define environment variables:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq
  namespace: datastores
data:
  enabled_plugins: |
    [rabbitmq_management].
  rabbitmq.conf: |
    listeners.tcp.default = 5672
    management.tcp.port = 15672
    loopback_users.guest = false

This ConfigMap mounts two config files into the RabbitMQ container:

enabled_plugins → Enables the Management UI
rabbitmq.conf → Port settings and default runtime config

Next, define a Secret for RabbitMQ credentials in k8s/rabbitmq/secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq
  namespace: datastores
type: Opaque
data:
  RABBITMQ_DEFAULT_USER: ZGVmYXVsdA== # Base64 for "default"
  RABBITMQ_DEFAULT_PASS: ZGVmYXVsdA== # Base64 for "default"

Now let's create the StatefulSet in k8s/rabbitmq/statefulset.yaml. A StatefulSet is similar to a Deployment but adds a few important capabilities for stateful systems like message brokers:

volumeMounts → Mounts RabbitMQ configuration and persistent storage
volumeClaimTemplates → Creates a dedicated PVC per Pod for durable message data
stable hostnames → Ensures Pods get predictable DNS entries

The postStart script is required because the official RabbitMQ Docker image does not automatically apply user tags, permissions, or certain admin settings at startup. Without this script:

the default user exists but has no administrator tag
required permissions are not granted
the Management UI will show limited capabilities
your microservices may fail permission checks when publishing/consuming messages

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: datastores
spec:
  serviceName: rabbitmq
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
        - name: rabbitmq
          image: rabbitmq:3.12-management
          ports:
            - containerPort: 5672
            - containerPort: 15672
          envFrom:
            - secretRef:
                name: rabbitmq
          volumeMounts:
            - name: rabbitmq
              mountPath: /etc/rabbitmq
            - name: rabbitmq-data
              mountPath: /var/lib/rabbitmq
          lifecycle:
            postStart:
              exec:
                command: [
                    "/bin/sh",
                    "-c",
                    "echo 'Configuring RabbitMQ...'; \
                    for i in $(seq 1 60); do \
                    if rabbitmqctl status; then \
                    rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbit@$(hostname).pid && \
                    echo 'RabbitMQ is ready!' && break; \
                    else \
                    echo 'Waiting for RabbitMQ to be ready...'; \
                    fi; \
                    sleep 5; \
                    done; \
                    rabbitmqctl set_user_tags ${RABBITMQ_DEFAULT_USER} administrator && \
                    rabbitmqctl set_permissions -p / ${RABBITMQ_DEFAULT_USER} '.*' '.*' '.*'; \
                    echo 'RabbitMQ configured successfully.'",
                  ]
      volumes:
        - name: rabbitmq
          configMap:
            name: rabbitmq
  volumeClaimTemplates:
    - metadata:
        name: rabbitmq-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

The last piece is a Headless Service for the StatefulSet, defined in k8s/rabbitmq/headless-service.yaml. It provides each RabbitMQ Pod a stable DNS identity:

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: datastores
spec:
  ports:
    - name: tcp
      port: 5672
    - name: management
      port: 15672
  clusterIP: None
  selector:
    app: rabbitmq

Tip: For local development and experiments, it’s perfectly fine to run datastores like Postgres or RabbitMQ outside the KIND cluster using Docker Compose. This dramatically reduces complexity since you avoid dealing with StatefulSets, PVCs, and storage provisioning. I’ve experimented with both approaches, and running stateful components externally often leads to a smoother and faster feedback loop.
Even in production, databases and message brokers are typically not run inside Kubernetes due to operational complexity, durability concerns, and failover constraints.

Running Everything

With all manifests in place, the final step is to build the service images, push them to the local registry, deploy all components, and verify that everything is running correctly.

First, build and push the images for each microservice (api-gateway, user-service, notification-service) to the local KIND registry:

docker build --target production -t localhost:5000/api-gateway:1.0 ./api-gateway
docker build --target production -t localhost:5000/user-service:1.0 ./user-service
docker build --target production -t localhost:5000/notification-service:1.0 ./notification-service

docker push localhost:5000/api-gateway:1.0
docker push localhost:5000/user-service:1.0
docker push localhost:5000/notification-service:1.0

Next, deploy RabbitMQ and the microservices by applying the manifests:

kubectl apply -f k8s/rabbitmq
kubectl apply -f k8s/user-service
kubectl apply -f k8s/notification-service
kubectl apply -f k8s/api-gateway

User service and notification service may restart a few times initially if RabbitMQ isn’t ready. Kubernetes assumes each service is self-healing, and the readiness probe ensures that pods only receive traffic once their dependencies are available.

To validate that everything is running properly, watch the pods come up:

kubectl get pods -A -w

If a pod isn’t behaving as expected, you can inspect it further. To see detailed information:

kubectl describe pod <pod-name> -n <namespace>

To follow logs in real-time, for example for the Notification Service:

kubectl logs -f deployment/notification-service -n microservices

Some common issues you might encounter:

ImagePullBackOff — The image isn’t in the registry or KIND isn’t configured to pull from it. Double-check your registry setup and image tags.
CrashLoopBackOff — Usually occurs when a service starts before RabbitMQ is ready. Make sure readiness probes are correctly defined and RabbitMQ is healthy.
Ingress 404 — Either the host entry isn’t in /etc/hosts, or the Ingress isn’t mapped correctly. Check it with kubectl get ingress -n microservices.

Calling the API

Once all Pods are running and the Ingress is active, you can create a user through the API Gateway:

curl -X POST http://microservices.local/users \
  -H "Content-Type: application/json" \
  -d '{"name": "daniel", "email": "daniel@example.com", "password": "123"}'

If everything is wired correctly, the User Service will publish a user.created event, and the Notification Service will consume it.

Check the logs:

kubectl logs -f deployment/notification-service -n microservices

You should see something like:

{
  "level": "log",
  "pid": 25,
  "timestamp": 1764442767463,
  "message": "Sending welcome email to daniel",
  "context": "NotificationController"
}

This confirms the full event flow works across the cluster.

Cleaning Up

When you’re done experimenting, delete the KIND cluster:

kind delete cluster

Stop the local registry:

docker stop kind-registry
docker rm kind-registry

Automating Everything (Makefile)

The repository includes a Makefile that automates every step end-to-end, from cluster creation to deployment and teardown.

Quick summary of available commands:

make help                       # List all available commands
make kind-create-cluster        # Create KIND cluster, local registry, metrics service, namespace
make kind-deploy-nginx-ingress  # Install and run NGINX Ingress + cloud-provider-kind
make kind-build-images          # Build all microservice Docker images
make kind-push-images           # Push images to local registry
make kind-deploy-services       # Deploy RabbitMQ + all microservices
make kind-delete-cluster        # Delete cluster and stop registry

Conclusion

Thanks for reading! This walkthrough showed how to run a local Kubernetes environment using KIND with a multi-node cluster, local registry, NGINX Ingress, RabbitMQ, readiness/liveness probes, and clean Kubernetes manifests. While real-world Kubernetes deployments are often larger and more complex, this guide provides a solid, hands-on starting point for understanding how Kubernetes deploys, manages, and operates microservices, even on a local machine.

If you found this useful, leave a like or comment, and feel free to explore the Github repo for the full setup!

Operational Excellence in Go Microservices: Caching, Healthchecks, Metrics, and Tracing

Sagar Maheshwary — Sat, 01 Nov 2025 13:29:51 +0000

In Part Two, we integrated Postgres using GORM, along with migrations and seeders, discussed Service layer pattern, created UserService example which we integrated with SayHello RPC, and ended the article with an integration test using testcontainers.

In this part, we will look at Redis integration for caching, Healthchecks for service monitoring, Observability with Prometheus metrics and OpenTelemetry tracing to make our microservice production-ready. We’ll also walk through a sample Grafana dashboard for metrics visualization and spin up a second service to demonstrate end-to-end distributed tracing in action.

Redis Integration
Healthchecks
Observability (Metrics and Tracing)
Wrapping Up the Series

Redis Integration

Caching is an important part of scaling any service. It reduces database load and speeds up response times by storing frequently accessed data in a fast in-memory store. We’ll use Redis, the most popular in-memory key–value store, for this purpose.

Redis supports far more than simple caching — features like Pub/Sub messaging, geospatial queries, and time series are built in — but for this project, we’ll keep things focused on basic caching functions that can easily be extended later.

Setup

We start by updating our config.go with a new Redis struct:

type Config struct {
    GRPCServer *GRPCServer `validate:"required"`
    Database   *Database   `validate:"required"`
    Redis      *Redis      `validate:"required"`
}

type Redis struct {
    Addr         string        `validate:"required"`
    Password     string        `validate:"required"`
    DB           int           `validate:"gte=0"`
    DialTimeout  time.Duration `validate:"gte=0"`
    ReadTimeout  time.Duration `validate:"gte=0"`
    WriteTimeout time.Duration `validate:"gte=0"`
    PoolSize     int           `validate:"gte=0"`
    MinIdleConns int           `validate:"gte=0"`
}

Then we load it inside NewConfigWithOptions():

    cfg := &Config{
        Redis: &Redis{
            Addr:         getEnv("REDIS_ADDR", "localhost:6379"),
            Password:     getEnv("REDIS_PASSWORD", "default"),
            DB:           getEnvInt("REDIS_DB", 0),
            DialTimeout:  getEnvDuration("REDIS_DIAL_TIMEOUT", time.Second*5),
            ReadTimeout:  getEnvDuration("REDIS_READ_TIMEOUT", time.Second*3),
            WriteTimeout: getEnvDuration("REDIS_WRITE_TIMEOUT", time.Second*3),
            PoolSize:     getEnvInt("REDIS_POOL_SIZE", 20),
            MinIdleConns: getEnvInt("REDIS_MIN_IDLE_CONNECTIONS", 5),
        },
        //...
    }

Next, we’ll create internal/cache/cache.go where our Redis implementation will live:

package cache

import (
  "fmt"
    "time"
    "context"
    "github.com/redis/go-redis/v9"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

type CacheService interface {
    Set(ctx context.Context, key string, value interface{}, expiration time.Duration) error
    Get(ctx context.Context, key string) (string, error)
    Delete(ctx context.Context, key string) error
    Ping(ctx context.Context) error
    Close() error
}

type RedisCache struct {
    client *redis.Client
}

type Opts struct {
    Config *config.Redis
    Logger logger.Logger
}

func NewRedisCache(ctx context.Context, opts *Opts) (CacheService, error) {
    cfg := opts.Config
    rdb := redis.NewClient(&redis.Options{
        Addr:         cfg.Addr,
        Password:     cfg.Password,
        DB:           cfg.DB,
        DialTimeout:  cfg.DialTimeout,
        ReadTimeout:  cfg.ReadTimeout,
        WriteTimeout: cfg.WriteTimeout,
        PoolSize:     cfg.PoolSize,
        MinIdleConns: cfg.MinIdleConns,
    })

    r := &RedisCache{client: rdb}

    if err := r.Ping(ctx); err != nil {
        return nil, fmt.Errorf("failed to connect to redis: %v", err)
    }

    opts.Logger.Info("Redis connected")

    return r, nil
}

func (r *RedisCache) Set(ctx context.Context, key string, value interface{}, expiration time.Duration) error {
    return r.client.Set(ctx, key, value, expiration).Err()
}

func (r *RedisCache) Get(ctx context.Context, key string) (string, error) {
    return r.client.Get(ctx, key).Result()
}

func (r *RedisCache) Delete(ctx context.Context, key string) error {
    return r.client.Del(ctx, key).Err()
}

func (r *RedisCache) Ping(ctx context.Context) error {
    return r.client.Ping(ctx).Err()
}

func (r *RedisCache) Close() error {
    if r == nil || r.client == nil {
        return fmt.Errorf("cannot close: redis is not initialized")
    }
    return r.client.Close()
}

Here, we’ve followed the same modular pattern used across other components. We define a CacheService interface that can be easily mocked in unit tests, and a concrete RedisCache struct implementing basic caching functions along with Ping for health checks and Close for graceful shutdown. Finally, we provide a NewRedisCache() constructor to instantiate the cache service cleanly.

Implementation

Now we initialize our cache service in main.go:

package main

import (
  "os"
    "context"
    "os/signal"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/cache"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    log := logger.NewZerologLogger("info", os.Stderr)

    cfg, err := config.NewConfig(log)
    if err != nil {
        log.Fatal(err.Error())
    }

    redisCache, err := cache.NewRedisCache(ctx, &cache.Opts{
        Config: cfg.Redis,
        Logger: log,
    })
    if err != nil {
        log.Fatal(err.Error())
    }

    //...Database, gRPC server etc

  <-ctx.Done()

  // Gracefully close redis client
    if err := redisCache.Close(); err != nil {
        log.Error("failed to close cache client", logger.Field{Key: "error", Value: err.Error()})
    }
}

This setup ensures that Redis starts alongside our other dependencies and closes gracefully when the service shuts down.

To integrate caching into our application logic, we’ll update the UserService.FindByID method. It will now first look for the user data in Redis before querying the database. If the record isn’t found in cache, it’s fetched from the database and then written back to Redis for subsequent requests.

Since we’ve added another dependency, we’ll extend the UserService constructor to accept the cache service:

package service

import (
    "gorm.io/gorm"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/cache"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database"
)

type userService struct {
    database *gorm.DB
    cache    cache.CacheService
}

type UserServiceOpts struct {
    Database database.DatabaseService
    Cache    cache.CacheService
}

func NewUserService(opts *UserServiceOpts) UserService {
    return &userService{
        database: opts.Database.DB(),
        cache:    opts.Cache,
    }
}

Next, we update our gRPC server to pass the cache instance from main.go:

type Opts struct {
    Config   *config.GRPCServer
    Logger   logger.Logger
    Database database.DatabaseService
    Cache    cache.CacheService
}

func NewServer(opts *Opts) *GRPCServer {
    srv := grpc.NewServer(grpc.UnaryInterceptor(interceptor.LoggerInterceptor(opts.Logger)))

    helloworld.RegisterGreeterServer(srv, handler.NewGreeterServer(
        service.NewUserService(&service.UserServiceOpts{
            Database: opts.Database,
            Cache:    opts.Cache,
        }),
    ))

    return &GRPCServer{
        Server: srv,
        Config: opts.Config,
        Logger: opts.Logger,
    }
}

Finally, here’s how caching looks inside FindByID:

func (s *userService) FindByID(ctx context.Context, id uint) (*model.User, error) {
    cacheKey := fmt.Sprintf("user:%d", id)

    // Try cache
    if cached, err := s.cache.Get(ctx, cacheKey); err == nil && cached != "" {
        var u model.User

        if err := json.Unmarshal([]byte(cached), &u); err != nil {
            return nil, err
        }

        return &u, nil
    }

    u := &model.User{}
    if err := s.database.First(u, id).Error; err != nil {
        return nil, err
    }
    data, _ := json.Marshal(u)

    if err := s.cache.Set(ctx, cacheKey, data, time.Minute); err != nil { // cache for 1 minute
        return nil, err
    }

    return u, nil
}

Now, when you call the SayHello RPC, the user data will come from Redis on the second request (after being cached during the first). The cache expires after one minute, after which a new request will repopulate it. You can also log cache hits/misses to observe behavior.

Integration Test

Since we added caching to UserService, we’ll update our integration tests accordingly. To ensure realistic conditions, we’ll spin up a real Redis instance using Testcontainers. Let’s start by creating a helper function SetupRedis in testutils:

package testutils

import (
  "io"
  "fmt"
    "log"
    "time"
    "testing"
    "context"
    "github.com/stretchr/testify/require"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/modules/redis"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/cache"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

func SetupRedis(t *testing.T) cache.CacheService {
    ctx := context.Background()

    redisContainer, err := redis.Run(ctx,
        "redis:7.2",
        redis.WithSnapshotting(10, 1),
        redis.WithLogLevel(redis.LogLevelVerbose),
    )
    require.NoError(t, err)

    t.Cleanup(func() {
        if err := testcontainers.TerminateContainer(redisContainer); err != nil {
            log.Printf("failed to terminate container: %s", err)
        }
    })

    host, _ := redisContainer.Host(ctx)
    port, _ := redisContainer.MappedPort(ctx, "6379")

    redisCache, err := cache.NewRedisCache(ctx, &cache.Opts{
        Config: &config.Redis{
            Addr:         fmt.Sprintf("%s:%s", host, port.Port()),
            Password:     "",
            DB:           0,
            DialTimeout:  time.Second * 5,
            ReadTimeout:  time.Second * 3,
            WriteTimeout: time.Second * 3,
            PoolSize:     20,
            MinIdleConns: 5,
        },
        Logger: logger.NewZerologLogger("info", io.Discard),
    })
    require.NoError(t, err)

    return redisCache
}

And then update the FindByID test to include Redis:

package service_test

import (
    "context"
    "testing"
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/service"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database/model"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/tests/testutils"
)

func TestUserService_FindByID(t *testing.T) {
    db := testutils.SetupPostgres(t)
    redis := testutils.SetupRedis(t)

    // Seed test data
    u := &model.User{Name: "Alice", Email: "alice@example.com"}
    require.NoError(t, db.DB().Create(u).Error)

    userService := service.NewUserService(&service.UserServiceOpts{
        Database: db,
        Cache:    redis,
    })

    got, err := userService.FindByID(context.Background(), u.ID)
    require.NoError(t, err)

    assert.Equal(t, "Alice", got.Name)
    assert.Equal(t, "alice@example.com", got.Email)
}

For this test, we only need to confirm that FindByID behaves correctly — whether the data came from the database or cache isn’t important here. The goal is to ensure the service works seamlessly with both persistence and caching layers under test conditions.

Healthchecks

Service monitoring is critical in production environments. Although gRPC includes a built-in health check protocol, it only exposes a single RPC endpoint. This makes it less flexible if you want to differentiate between basic service liveness and full readiness to serve traffic.

In this project, we’ll implement two simple REST endpoints — /livez and /readyz — for clarity and flexibility.

We’ll start by defining a HealthService in internal/service/health.go. This service will be used by our /readyz endpoint to verify that all dependencies (like the database and cache) are healthy before declaring the service ready.

package service

import (
    "context"

    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/cache"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database"
    "gorm.io/gorm"
)

type HealthService interface {
    Check(ctx context.Context) HealthStatus
    SetReady(ready bool)
}

type HealthStatus struct {
    Status  string            `json:"status"`
    Details map[string]string `json:"details,omitempty"`
}

type healthService struct {
    database *gorm.DB
    cache    cache.CacheService
    ready    atomic.Bool
}

type HealthServiceOpts struct {
    Database database.DatabaseService
    Cache    cache.CacheService
}

func NewHealthService(opts *HealthServiceOpts) HealthService {
    h := &healthService{
        database: opts.Database.DB(),
        cache:    opts.Cache,
    }
    h.ready.Store(true)
    return h
}

func (h *healthService) SetReady(ready bool) {
    h.ready.Store(ready)
}

func (h *healthService) Check(ctx context.Context) HealthStatus {
    if !h.ready.Load() {
        return HealthStatus{
            Status:  "unready",
            Details: map[string]string{"service": "shutting down"},
        }
    }

    status := HealthStatus{Status: "ready", Details: map[string]string{}}

    if err := h.database.Exec("SELECT 1").Error; err != nil {
        status.Status = "unready"
        status.Details["database"] = err.Error()
    } else {
        status.Details["database"] = "ok"
    }

    if err := h.cache.Ping(ctx); err != nil {
        status.Status = "unready"
        status.Details["cache"] = err.Error()
    } else {
        status.Details["cache"] = "ok"
    }

    return status
}

The Check method reports dependency health via /readyz. The SetReady method lets us manually toggle readiness — useful during startup and graceful shutdown.

Next, we’ll implement the REST handlers for /livez and /readyz inside internal/transports/http/server/handler/health.go:

package handler

import (
    "encoding/json"
    "net/http"

    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/service"
)

type Opts struct {
    HealthService service.HealthService
    Logger        logger.Logger
}

type HealthHandler struct {
    healthService service.HealthService
    logger        logger.Logger
}

func NewHealthHandler(opts *Opts) *HealthHandler {
    return &HealthHandler{healthService: opts.HealthService, logger: opts.Logger}
}

func (h *HealthHandler) Livez(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)

    writeJSON(w, map[string]string{"status": "ok"}, h.logger)
}

func (h *HealthHandler) Readyz(w http.ResponseWriter, r *http.Request) {
    status := h.healthService.Check(r.Context())
    w.Header().Set("Content-Type", "application/json")

    if status.Status == "ready" {
        w.WriteHeader(http.StatusOK)
    } else {
        w.WriteHeader(http.StatusServiceUnavailable)
    }

    writeJSON(w, status, h.logger)
}

//common JSON helper (can be separated in a file if you add more apis)
func writeJSON(w http.ResponseWriter, data any, log logger.Logger) {
    w.Header().Set("Content-Type", "application/json")
    if err := json.NewEncoder(w).Encode(data); err != nil {
        log.Error("failed to write JSON: " + err.Error())
    }
}

The /livez handler simply returns "ok" with an HTTP 200 — it’s only responsible for confirming that the service process is running.

The /readyz handler, on the other hand, uses the HealthService to confirm that all required dependencies (like the database and Redis) are operational.

Now, let’s wire up our HTTP server in internal/transports/http/server/server.go:

package server

import (
    "net"
    "net/http"

    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/cache"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/service"
)

type Opts struct {
    Config   *config.HTTPServer
    Logger   logger.Logger
    Database database.DatabaseService
    Cache    cache.CacheService
    Health   service.HealthService
}

type HTTPServer struct {
    Config *config.HTTPServer
    Server *http.Server
    Logger logger.Logger
}

func NewServer(opts *Opts) *HTTPServer {
    mux := http.NewServeMux()

    healthHandler := handler.NewHealthHandler(&handler.Opts{
        HealthService: opts.Health,
        Logger:        opts.Logger,
    })

    mux.HandleFunc("/livez", healthHandler.Livez)
    mux.HandleFunc("/readyz", healthHandler.Readyz)

    return &HTTPServer{
        Config: opts.Config,
        Server: &http.Server{
            Addr:    opts.Config.URL,
            Handler: mux,
        },
        Logger: opts.Logger,
    }
}

func (h *HTTPServer) ServeListener(listener net.Listener) error {
    h.Logger.Info("HTTP server started", logger.Field{Key: "address", Value: listener.Addr().String()})
    if err := h.Server.Serve(listener); err != nil && err != http.ErrServerClosed {
        h.Logger.Error("HTTP server failed", logger.Field{Key: "error", Value: err.Error()})
        return err
    }
    return nil
}

func (h *HTTPServer) Serve() error {
    listener, err := net.Listen("tcp", h.Config.URL)
    if err != nil {
        h.Logger.Error("Failed to create HTTP listener",
            logger.Field{Key: "address", Value: h.Config.URL},
            logger.Field{Key: "error", Value: err.Error()},
        )
        return err
    }

    return h.ServeListener(listener)
}

This follows the same structure as our GRPCServer. We define both ServeListener (for custom listeners) and Serve (for normal TCP-based startup from configuration).

Next, update your configuration to fetch the HTTP server’s host and port from environment variables:

type Config struct {
    GRPCServer *GRPCServer `validate:"required"`
    HTTPServer *HTTPServer `validate:"required"`
    Database   *Database   `validate:"required"`
    Redis      *Redis      `validate:"required"`
}

//Define struct
type HTTPServer struct {
    URL string `validate:"required,hostname_port"`
}

//Load from env
func NewConfigWithOptions(opts LoaderOptions) (*Config, error) {
  //...

  cfg := &Config{
    HTTPServer: &HTTPServer{
            URL: getEnv("HTTP_SERVER_URL", ":4000"),
        },
    //...
  }

  //...
}

Finally, let’s initialize and start the HTTP server in main.go:

package main

func main() {
  ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

  log := logger.NewZerologLogger("info", os.Stderr)

    cfg, err := config.NewConfig(log)
    if err != nil {
        log.Fatal(err.Error())
    }

  //...

    healthService := service.NewHealthService(&service.HealthServiceOpts{
        Database: db,
        Cache:    redisCache,
    })

  httpServer := httpserver.NewServer(&httpserver.Opts{
        Config:   cfg.HTTPServer,
        Logger:   log,
        Database: db,
        Cache:    redisCache,
        Metrics:  metricsService,
        Health:   healthService,
    })
    go func() {
        err = httpServer.Serve()
        if err != nil && !errors.Is(err, http.ErrServerClosed) {
            stop()
        }
    }()

  <-ctx.Done()

  // Marking the service as not ready (/readyz) during shutdown.
    healthService.SetReady(false)

  //shutdown grpc, db, redis etc

    // Shut down the health server last so it can continue responding to liveness checks (/livez).
  if err := httpServer.Server.Shutdown(ctx); err != nil {
        log.Error("failed to close http server", logger.Field{Key: "error", Value: err.Error()})
    }
}

Once everything is wired up, we can test the endpoints locally with curl:

Livez:

curl localhost:4000/livez

Expected response:

{ "status": "ok" }

Readyz:

curl localhost:4000/readyz

Expected response:

{ "status": "ready", "details": { "cache": "ok", "database": "ok" } }

With these two endpoints, you now have both basic and dependency-aware healthchecks that can be used by Kubernetes probes, load balancers, or monitoring tools.

Observability (Metrics and Tracing)

In Part 1, we added structured logging to capture what’s happening inside our services. Logs are great for understanding what happened in a specific instance — but in a distributed system, we also need to understand how things behave and how requests flow across multiple services.

That’s where metrics and tracing come in.

Metrics capture quantitative data about our services — like request counts, error rates, and latency. They’re lightweight, easy to visualize, and ideal for alerting and performance dashboards.
Tracing gives us a request-level view of our system — showing how a request moves between microservices and how much time is spent in each hop.

Together, logging, metrics, and tracing give us full observability into our system’s behavior. In this part, we’ll implement metrics and tracing, then spin up Prometheus, Grafana, and Jaeger to see them in action.

Prometheus Metrics

Prometheus is a popular open-source monitoring system that scrapes metrics from your services, stores them as time-series data, and makes them queryable through PromQL. We'll use the official prometheus/client_golang library to instrument our services and exposes metrics for scraping.

We’ll begin by creating a MetricsService inside internal/observability/metrics/metrics.go. This service will handle Prometheus setup, expose default metrics, and provide a simple way to register custom collectors.

package metrics

import (
    "net/http"

    "github.com/prometheus/client_golang/prometheus"
    promcollectors "github.com/prometheus/client_golang/prometheus/collectors"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
)

type MetricsService interface {
    Register(collectors ...MetricCollector)
    RegisterDefault()
    Handler() http.Handler
}

type metricsService struct {
    Config   *config.Metrics
    Registry *prometheus.Registry
}

func NewMetricsService(cfg *config.Metrics, collectors ...MetricCollector) MetricsService {
    registry := prometheus.NewRegistry()

    m := &metricsService{
        Registry: registry,
        Config:   cfg,
    }

    if cfg.EnableDefaultMetrics {
        m.RegisterDefault()
    }
    m.Register(collectors...)

    return m
}

func (m *metricsService) Register(collectors ...MetricCollector) {
    for _, c := range collectors {
        c.Register(m.Registry)
    }
}

func (m *metricsService) RegisterDefault() {
    m.Registry.MustRegister(
        promcollectors.NewGoCollector(),
        promcollectors.NewProcessCollector(promcollectors.ProcessCollectorOpts{}),
    )
}

func (m *metricsService) Handler() http.Handler {
    return promhttp.HandlerFor(
        m.Registry,
        promhttp.HandlerOpts{},
    )
}

To keep metrics modular, we define a small MetricCollector interface. Each component — like gRPC, HTTP, or Redis — can implement this interface to expose its own metrics while keeping code organized.

type MetricCollector interface {
    Register(r *prometheus.Registry)
}

Next, we add gRPC-specific metrics inside a new metrics/grpc.go file. This includes counters and histograms for request counts, and latency — useful for tracking performance trends across gRPC calls.

package metrics

import "github.com/prometheus/client_golang/prometheus"

var (
    GRPCRequestCounter = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "grpc_requests_total",
            Help: "Total number of gRPC requests received, labeled by method and status.",
        },
        []string{"method", "status"},
    )

    GRPCRequestLatency = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "grpc_request_duration_seconds",
            Help:    "Histogram of gRPC request latencies (seconds).",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method"},
    )
)

type GRPCMetrics struct{}

func (GRPCMetrics) Register(r *prometheus.Registry) {
    r.MustRegister(GRPCRequestCounter, GRPCRequestLatency)
}

We then create a gRPC interceptor that automatically records these metrics for every incoming request. This way, we don’t need to manually instrument every handler — metrics are collected transparently as part of the request lifecycle.

package interceptor

import (
    "context"
    "time"

    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/observability/metrics"
    "google.golang.org/grpc"
    "google.golang.org/grpc/status"
)

func MetricsInterceptor() grpc.UnaryServerInterceptor {
    return func(
        ctx context.Context,
        req interface{},
        info *grpc.UnaryServerInfo,
        handler grpc.UnaryHandler,
    ) (interface{}, error) {
        start := time.Now()
        res, err := handler(ctx, req)

        method := info.FullMethod
        statusCode := status.Code(err).String()
        metrics.GRPCRequestCounter.WithLabelValues(method, statusCode).Inc()
        metrics.GRPCRequestLatency.WithLabelValues(method).Observe(time.Since(start).Seconds())

        return res, err
    }
}

We then attach the interceptor to gRPC server via grpc.ChainUnaryInterceptor:

    srv := grpc.NewServer(
        grpc.ChainUnaryInterceptor(
            interceptor.LoggerInterceptor(opts.Logger),
            interceptor.MetricsInterceptor(),
        ),
  )

Once metrics are collected, we need a way for Prometheus to access them. We expose a /metrics endpoint in our HTTP server, which Prometheus will scrape periodically.

package server

type Opts struct {
    Config   *config.HTTPServer
    Logger   logger.Logger
    Database database.DatabaseService
    Cache    cache.CacheService
    Metrics  metrics.MetricsService
}

func NewServer(opts *Opts) *HTTPServer {
    mux := http.NewServeMux()

    healthService := service.NewHealthService(&service.HealthServiceOpts{
        Database: opts.Database,
        Cache:    opts.Cache,
    })
    healthHandler := handler.NewHealthHandler(&handler.Opts{
        HealthService: healthService,
        Logger:        opts.Logger,
    })

    mux.HandleFunc("/livez", healthHandler.Livez)
    mux.HandleFunc("/readyz", healthHandler.Readyz)
    mux.Handle("/metrics", opts.Metrics.Handler())

    return &HTTPServer{
        Config: opts.Config,
        Server: &http.Server{
            Addr:    opts.Config.URL,
            Handler: mux,
        },
        Logger: opts.Logger,
    }
}

Now Let's register metrics to HTTPServer in main.go:

package main

import (...)

func main() {
  //...

    metricsService := metrics.NewMetricsService(cfg.Metrics, metrics.GRPCMetrics{})

    httpServer := httpserver.NewServer(&httpserver.Opts{
        Config:   cfg.HTTPServer,
        Logger:   log,
        Database: db,
        Cache:    redisCache,
        Metrics:  metricsService,
    })
    go func() {
        err = httpServer.Serve()
        if err != nil && !errors.Is(err, http.ErrServerClosed) {
            stop()
        }
    }()
}

Finally, we update our configuration to include a METRICS_ENABLE_DEFAULT_METRICS option. This flag lets us control whether we also expose Go’s built-in runtime metrics like garbage collection and goroutine counts — useful in production for monitoring resource usage.

type Metrics struct {
    EnableDefaultMetrics bool
}

func NewConfigWithOptions(opts LoaderOptions) (*Config, error) {
  cfg := &Config{
    Metrics: &Metrics{
            EnableDefaultMetrics: getEnvBool("METRICS_ENABLE_DEFAULT_METRICS", false),
        },
  }
}

//New helper for parsing boolean values
func getEnvBool(key string, defaultVal bool) bool {
    if val, err := strconv.ParseBool(os.Getenv(key)); err == nil {
        return val
    }

    return defaultVal
}

With this in place, our services now export rich, structured metrics that can be scraped by Prometheus and visualized in Grafana. These metrics will serve as the backbone for our dashboards and alerts once we deploy to production.

OpenTelemetry Tracing

Tracing is the third pillar of observability, alongside metrics and logs. While metrics tell you how your system behaves overall, and logs describe what happened at a specific moment, tracing shows you how a request flows across services. It’s especially valuable in microservice architectures where a single client request might touch multiple services, databases, or queues before completing.

We’ll be using OpenTelemetry — a popular open-source standard for collecting distributed traces — and connect it with Jaeger for visualization. You can also integrate it later with other backends like Zipkin, Tempo, or AWS X-Ray if needed.

Let’s start by defining our TracerService in internal/observability/tracing/tracing.go:

package tracing

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

type Opts struct {
    Config *config.Tracing
    Logger logger.Logger
}

type TracerService struct {
    Config *config.Tracing
    Logger logger.Logger
    tp     *sdktrace.TracerProvider
}

func NewTracerService(ctx context.Context, opts *Opts) (*TracerService, error) {
    cfg := opts.Config

    exporter, err := otlptrace.New(
        ctx,
        otlptracehttp.NewClient(
            otlptracehttp.WithEndpoint(cfg.CollectorURL),
            otlptracehttp.WithInsecure(),
        ),
    )
    if err != nil {
        return nil, err
    }

    res, err := resource.New(ctx,
        resource.WithAttributes(semconv.ServiceName(cfg.ServiceName)),
    )
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})

    opts.Logger.Info("Tracing initialized (exporter: otlptracehttp)",
        logger.Field{Key: "serviceName", Value: cfg.ServiceName},
    )

    return &TracerService{
        Config: cfg,
        Logger: opts.Logger,
        tp:     tp,
    }, nil
}

func (t *TracerService) Shutdown(ctx context.Context) error {
    return t.tp.Shutdown(ctx)
}

Next, we’ll update our configuration to include tracing settings such as service name and exporter in config.go:

type Tracing struct {
    ServiceName  string `validate:"required"`
    CollectorURL string `validate:"required,hostname_port"`
}

func NewConfigWithOptions(opts LoaderOptions) (*Config, error) {
    cfg := &Config{
        Tracing: &Tracing{
            ServiceName:  getEnv("TRACING_SERVICE_NAME", "go-microservice-boilerplate"),
            CollectorURL: getEnv("TRACING_COLLECTOR_URL", "localhost:4318"),
        },
        //...
    }
}

Now that the config is ready, we can initialize the tracer in main.go so that every part of our application has access to it:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    //...

    tracerService, err := tracing.NewTracerService(ctx, &tracing.Opts{
        Config: cfg.Tracing,
        Logger: log,
    })
    if err != nil {
        log.Fatal(err.Error())
    }

    //...

    <-ctx.Done()

    //gracefully shutdown the client
    if err := tracerService.Shutdown(ctx); err != nil {
        log.Error("failed to close tracing client", logger.Field{Key: "error", Value: err.Error()})
    }
}

To propagate trace context across service boundaries, we add a gRPC StatsHandler. This ensures that when a request comes into our service, it either starts a new trace or continues an existing one passed from another service:

srv := grpc.NewServer(
    grpc.ChainUnaryInterceptor(
        interceptor.LoggerInterceptor(opts.Logger),
        interceptor.MetricsInterceptor(),
    ),
    grpc.StatsHandler(otelgrpc.NewServerHandler(
        otelgrpc.WithTracerProvider(otel.GetTracerProvider()),
        otelgrpc.WithPropagators(otel.GetTextMapPropagator()),
    )),
)

From now on, every gRPC call will automatically create a trace that’s sent to Jaeger.

Jaeger UI: http://localhost:16686/search

Each trace is composed of multiple spans — where a trace represents the entire journey of a request (for example, a user calling your API), and spans represent individual operations within that request (like “fetch user from DB” or “send email”).

To demonstrate this, let’s add spans to our SayHello handler and the UserService. Each span will capture the work done by that specific function, and together they’ll form a full picture of a single request’s flow through the system.

func (g *GreeterServer) SayHello(ctx context.Context, in *helloworld.SayHelloRequest) (*helloworld.SayHelloResponse, error) {
    tr := otel.Tracer("hello_world.Greeter")
    ctx, span := tr.Start(ctx, "helloworld.SayHello")
    defer span.End()

    //...
}

func (s *userService) FindByID(ctx context.Context, id uint) (*model.User, error) {
    tr := otel.Tracer("UserService")
    ctx, span := tr.Start(ctx, "FindByID")
    span.SetAttributes(attribute.String("UserId", strconv.Itoa(int(id))))
    defer span.End()

    //...
}

When you look at the trace in Jaeger, you’ll see a clear chain of execution — the main request trace leading into the SayHello span, which then calls into the UserService span.

The SetAttributes method lets you attach useful context (like user IDs, order IDs, or error states) that can make debugging much easier when viewing a trace.

Context propagation here is critical — notice that we always pass ctx forward. This context carries the traceID created by our gRPC StatsHandler, ensuring that every downstream function and service is linked to the same request chain.

End to End Observability in Action

The repo comes with a ready-to-use observability setup using Docker Compose, available in the examples branch. If you want to follow along, clone the repo and switch to that branch.

We’ll first bring up our core application stack — which includes the service itself, PostgreSQL, and Redis:

docker compose up

Once the application is running, we can start the observability stack. This will spin up Grafana, Prometheus, and Jaeger — all preconfigured to work with our service.

docker compose -f docker-compose.observability.yml up

Jaeger runs via the jaeger/all-in-one image using its default in-memory storage to keep things lightweight and simple. Prometheus uses a basic scrape configuration that pulls metrics from our service’s /metrics endpoint every eight seconds:

global:
  scrape_interval: 8s

scrape_configs:
  - job_name: go-microservice-boilerplate
    static_configs:
      - targets:
          - "go-microservice-boilerplate:4000"

Grafana comes with a sample dashboard already wired up to Prometheus, showcasing the gRPC metrics we defined earlier. Open a browser and navigate to http://localhost:3000 (default credentials are admin / admin).

The dashboard has four panels visualizing data from the grpc_requests_total and grpc_request_duration_seconds metrics:

Average gRPC Request Duration — time series
gRPC Request Latency (95th Percentile) — time series
gRPC Requests per Method — time series
Total gRPC Requests — counter/gauge

This sample dashboard offers a quick glimpse into request throughput and latency trends — perfect for demonstration and local testing. In a real production environment, you’d typically design a more comprehensive dashboard tailored to your service’s specific KPIs, error rates, and resource utilization patterns.

Now that we’ve seen metrics in action, let’s turn to tracing. We already explored what a trace looks like for a single service, but the real power of tracing emerges when requests flow through multiple microservices. Distributed traces help you visualize the entire request path, pinpoint slow hops, and quickly identify where failures occur.

To demonstrate distributed tracing, let’s spin up a second instance of our service that will act as another microservice in the request chain. Clone the same repository inside your project directory, switch to the examples branch, and create an environment file:

git clone https://github.com/SagarMaheshwary/go-microservice-boilerplate.git go-microservice-boilerplate2
cd go-microservice-boilerplate2
git checkout examples
cp .env.example .env

Now let’s add this new service to our main docker-compose.yml file:

services:
  app2:
    build:
      context: ./go-microservice-boilerplate2
      target: development
    container_name: go-microservice-boilerplate2
    ports:
      - 4001:4000
      - 5001:5000
    volumes:
      - ./go-microservice-boilerplate2:/app
    networks:
      - microservices_net
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

Next, open the SayHello() handler of our main service and replace it with the following code.
This is just for demonstration — in a real-world setup, you would create a dedicated client inside transports/grpc/client and reuse it across the service instead of creating a new connection for every request (since gRPC clients are long-lived):

func (g *GreeterServer) SayHello(ctx context.Context, in *helloworld.SayHelloRequest) (*helloworld.SayHelloResponse, error) {
    tr := otel.Tracer("hello_world.Greeter")
    ctx, span := tr.Start(ctx, "helloworld.SayHello")
    defer span.End()

    conn, err := grpc.NewClient(
        "go-microservice-boilerplate2:5000",
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
    )
    if err != nil {
        return nil, status.Error(codes.Internal, err.Error())
    }
    defer conn.Close()

    greeter := helloworld.NewGreeterClient(conn)
    res, err := greeter.SayHello(ctx, &helloworld.SayHelloRequest{UserId: in.UserId})
    if err != nil {
        return nil, status.Error(codes.Internal, err.Error())
    }

    return res, nil
}

Note: In the internal/tracing/tracing.go file, make sure to include otel.SetTextMapPropagator(propagation.TraceContext{}). This line is missing from the part-3 source, and without it, traces between multiple services won’t connect properly. You can use either the master or examples branch, as both already include this fix.

Now, let’s trigger a request from the first service, which will internally call the second service. We’ll use grpcurl for this:

grpcurl -d '{"user_id": 1}' -proto ./proto/hello_world/hello_world.proto -plaintext localhost:5000 hello_world.Greeter/SayHello

Open Jaeger in your browser, and you should now see a single trace spanning across two services — one initiating the request, and the other handling it downstream:

Clicking into the trace reveals the chain of spans showing each step of the process:

This connected trace gives a clear view of how the request travels through the system — from the first service’s SayHello() call, to the gRPC client invocation, and finally to the second service’s SayHello() and its database lookup via UserService.

And that wraps up our Observability section.
We’ve now covered all three pillars — logging, metrics, and tracing — giving you complete visibility into your services. With this setup, you can detect issues early, measure system health, and trace requests across microservices, building a solid foundation for a production-grade monitoring stack.

Wrapping Up the Series

In this final part, we strengthened our service with Redis caching, health checks, and full observability using Prometheus metrics and OpenTelemetry tracing. We visualized performance and operational insights in Grafana, and used Jaeger to understand request behavior across services using distributed tracing — allowing us to see the system the way production teams do.

With this, the series comes to a close. Across all three parts, we built a Go microservice with a strong architectural foundation: clear structure, configuration management, structured logging, graceful shutdowns, PostgreSQL with a clean data layer, integration testing, Redis integration, metrics, health endpoints, and tracing. Together, these form a practical blueprint for designing maintainable, reliable, and observable Go services.

The repository is meant to be a reusable boilerplate for spinning up new Go microservices — giving you a clean, production-ready starting point for any project.

The master branch contains a clean, production-ready setup, while the examples branch includes all the working examples and demos featured throughout the series.

You can check out the complete code for Part Three here.

Thanks for reading! I hope you found this series helpful. If you have any questions, feedback, or suggestions, feel free to drop a comment. I’d love to hear your thoughts.

Designing a Reliable Data Layer in Go: PostgreSQL, GORM, and Integration Testing

Sagar Maheshwary — Sat, 11 Oct 2025 09:30:21 +0000

In Part One, we built the foundation, setting up configuration management, structured logging, a gRPC server, graceful shutdowns, containerized workflows, and a Makefile. That gave us a solid starting point, but most real-world microservices also need a database layer.

In this part, we’ll integrate PostgreSQL using GORM, set up migrations and seeders, and introduce a service layer with a UserService. We’ll also write integration tests with Testcontainers to ensure our code works against a real Postgres instance.

By the end, you’ll have a complete request → service → database → response flow, with persistent data and tests that give you confidence it all works together.

Database Setup
Database Migrations
Database Seeders
Service Layer Pattern (UserService Example)
Integration Testing with Testcontainers
Using the UserService in SayHello RPC
Conclusion

Database Setup

Almost every microservice needs persistence. In this project, we’ll use PostgreSQL, a reliable, production-grade relational database.

For quick local development or testing, we’ll also support SQLite. It doesn’t require a running database server, which makes it convenient for unit tests or rapid prototyping. However, PostgreSQL remains the primary focus throughout this series.

Our database code lives in internal/database/:

internal/
├── database/
│   ├── database.go   # opens DB connection (Postgres/SQLite)
│   ├── migrations/   # migration files
│   ├── seeder/      # seed data for dev/tests

Configuration

We’ll start by adding database connection settings to our .env file:

# PostgreSQL DSN
DATABASE_DSN=postgres://postgres:password@localhost:5432/boilerplate?sslmode=disable

# Which driver to use (postgres | sqlite)
DATABASE_DRIVER=postgres

# Connection pooling
DATABASE_POOL_MAX_IDLE=10
DATABASE_POOL_MAX_OPEN=100
DATABASE_POOL_MAX_LIFETIME=1h

Next, update config.go with a new Database struct:

type Config struct {
    GRPCServer *GRPCServer `validate:"required"`
    Database   *Database   `validate:"required"`
}

type Database struct {
    DSN                 string        `validate:"required"`
    Driver              string        `validate:"required,oneof=postgres sqlite"`
    PoolMaxIdleConns    int           `validate:"gte=0"`
    PoolMaxOpenConns    int           `validate:"gte=0"`
    PoolConnMaxLifetime time.Duration `validate:"gte=0"` // must be non-negative
}

Finally, load the database config inside NewConfigWithOptions():

    cfg := &Config{
        Database: &Database{
            DSN:                 getEnv("DATABASE_DSN", ""),
            Driver:              getEnv("DATABASE_DRIVER", "postgres"),
            PoolMaxIdleConns:    getEnvInt("DATABASE_POOL_MAX_IDLE", 10),
            PoolMaxOpenConns:    getEnvInt("DATABASE_POOL_MAX_OPEN", 100),
            PoolConnMaxLifetime: getEnvDuration("DATABASE_POOL_MAX_LIFETIME", time.Hour),
        },
        //...
    }

The Database Service

To keep the code modular, we define a simple DatabaseService interface in database.go that abstracts database operations:

type DatabaseService interface {
    DB() *gorm.DB
    Close() error
}

This makes it easy to swap drivers, mock the database in tests, or extend functionality later without rewriting everything.

We then implement NewDatabase to initialize a new database connection:

package database

import (
    "fmt"
    "gorm.io/gorm"
    "gorm.io/driver/sqlite"
    "gorm.io/driver/postgres"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

func NewDatabase(opts *Opts) (DatabaseService, error) {
    var (
        db  *gorm.DB
        err error
    )

    switch opts.Config.Driver {
    case "postgres":
        db, err = gorm.Open(postgres.Open(opts.Config.DSN), &gorm.Config{})
    case "sqlite":
        if opts.Config.DSN == "" {
            return nil, fmt.Errorf("invalid DSN: sqlite requires a non-empty DSN")
        }
        db, err = gorm.Open(sqlite.Open(opts.Config.DSN), &gorm.Config{})
    default:
        return nil, fmt.Errorf("unsupported database driver %s", opts.Config.Driver)
    }

    if err != nil {
        return nil, fmt.Errorf("failed to connect to %s: %v", opts.Config.Driver, err)
    }

    sqlDB, err := db.DB()
    if err != nil {
        return nil, fmt.Errorf("failed to get db instance: %v", err)
    }

  //Connection pooling
    sqlDB.SetMaxIdleConns(opts.Config.PoolMaxIdleConns)
    sqlDB.SetMaxOpenConns(opts.Config.PoolMaxOpenConns)
    sqlDB.SetConnMaxLifetime(opts.Config.PoolConnMaxLifetime)

    opts.Logger.Info("Database connected", logger.Field{Key: "driver", Value: opts.Config.Driver})

    return &Database{db: db, Logger: opts.Logger}, nil
}

NewDatabase reads the driver from config, opens the connection using the correct GORM driver, and applies pooling settings for efficient connection reuse under load.

We can now initialize the database in main.go:

package main

import (
    "context"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

func main() {
    log := logger.NewZerologLogger("info", os.Stderr)

    cfg, err := config.NewConfig(log)
    if err != nil {
        log.Fatal(err.Error())
    }

    db, err := database.NewDatabase(&database.Opts{
        Config: cfg.Database,
        Logger: log,
    })
    if err != nil {
        log.Fatal(err.Error())
    }

    //...grpc server etc
}

Closing Connections Gracefully

When the service shuts down, open connections should be released cleanly. The Close() method ensures that:

func (d *Database) Close() error {
    if d == nil || d.db == nil {
        return fmt.Errorf("cannot close: database is not initialized")
    }
    sqlDB, err := d.db.DB()
    if err != nil {
        return err
    }
    return sqlDB.Close()
}

We can then use it in the graceful shutdown pattern from Part 1:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    //logger, config, database, grpcServer setup...

    <-ctx.Done()

    grpcServer.Server.GracefulStop()

    if err := db.Close(); err != nil {
        log.Error("failed to close database client", logger.Field{Key: "error", Value: err.Error()})
    }
}

Next, we’ll set up migrations (to version-control schema changes) and seeders (to populate test or development data), followed by a clean service layer built around GORM.

Database Migrations

Schema drift is one of the most common issues in backend projects. When database changes are applied manually, it’s only a matter of time before development, staging, and production databases start to diverge — often leading to subtle bugs that surface at runtime. The best way to prevent this is through migrations: version-controlled SQL scripts that describe schema evolution in a predictable, reversible way.

In this project, we use golang-migrate/migrate, a popular migration tool that integrates well with both automation scripts and CI/CD pipelines. It helps ensure that every environment runs the same schema version — no matter where or when migrations are applied.

To create a new migration, we use the Makefile command:

make migrate-new name=create_users_table

This automatically generates both the “up” and “down” migration files using the proper naming convention:

internal/database/migrations/
├── 000001_create_users_table.up.sql
├── 000001_create_users_table.down.sql

Let’s define a simple users table in our first migration:

-- 000001_create_users_table.up.sql
CREATE TABLE
  IF NOT EXISTS users (
    id BIGSERIAL PRIMARY KEY,
    name VARCHAR(25) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT NOW (),
    updated_at TIMESTAMP
  );

-- 000001_create_users_table.down.sql
DROP TABLE IF EXISTS users;

Applying or rolling back migrations is just as straightforward. The Makefile includes helper commands that wrap golang-migrate, making it easy to run migrations without typing long CLI commands:

make migrate-up dsn="postgres://postgres:password@localhost:5432/boilerplate?sslmode=disable"
make migrate-down dsn="postgres://postgres:password@localhost:5432/boilerplate?sslmode=disable"

This setup keeps your schema versioned, portable, and reproducible — ensuring that every team member and environment stays in sync as the project evolves.

Database Seeders

Seeders play an essential role in keeping development and testing environments fast, consistent, and predictable. During development, they instantly populate your database with realistic sample data so you can start testing APIs and RPCs without manually inserting rows. In testing, seeders ensure that every run starts from a clean, known state — making your results reproducible across environments and CI pipelines.

All seeders live inside the internal/database/seeder directory:

internal/database/seeder/
├── runner.go
└── user.go

Let’s start simple with a user seeder that inserts a few sample records. This gives our service something to work with right away.

package seeder

import (
    "gorm.io/gorm"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/model"
)

func SeedUsers(db *gorm.DB) error {
    users := []model.User{
        {Name: "Alice", Email: "alice@example.com"},
        {Name: "Bob", Email: "bob@example.com"},
    }

    return db.Create(&users).Error
}

To keep things organized, we register all individual seeders in a central runner (runner.go). This way, running one command can execute all seeders in sequence.

package seeder

import (
    "gorm.io/gorm"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

type SeederFunc struct {
    Name string
    Func func(db *gorm.DB) error
}

var defaultSeeders = []SeederFunc{
    {Name: "SeedUsers", Func: SeedUsers},
    // Add more seeders here
}

type Opts struct {
    DB      *gorm.DB
    Log     logger.Logger
    Seeders []SeederFunc
}

func RunAll(opts *Opts) error {
    log := opts.Log
    seeders := opts.Seeders
    if len(seeders) == 0 {
        seeders = defaultSeeders
    }

    for _, s := range seeders {
        log.Info("[Seeder] Running " + s.Name)

        if err := s.Func(opts.DB); err != nil {
            log.Info("[Seeder] Failed "+s.Name, logger.Field{Key: "error", Value: err.Error()})
            return err
        }

        log.Info("[Seeder] Completed " + s.Name)
    }

    log.Info("[Seeder] All seeders completed successfully")
    return nil
}

Finally, we expose a CLI entrypoint cmd/cli/main.go to run all seeders directly from the terminal. This CLI can be extended with more commands in future.

package main

import (
    "os"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database/seeder"
)

func main() {
    log := logger.NewZerologLogger("info", os.Stderr)

    if len(os.Args) < 2 {
        log.Info("Usage: go run cmd/cli/main.go seed")
        os.Exit(1)
    }

    cmd := os.Args[1]

    cfg, err := config.NewConfig(log)
    if err != nil {
        log.Fatal(err.Error())
    }

    switch cmd {
    case "seed":
        db, err := database.NewDatabase(&database.Opts{
            Config: cfg.Database,
            Logger: log,
        })
        if err != nil {
            log.Fatal(err.Error())
        }
        defer db.Close()

        err = seeder.RunAll(&seeder.Opts{
            DB:  db.DB(),
            Log: log,
        })
        if err != nil {
            log.Fatal(err.Error())
        }
    default:
        log.Error("Unknown command " + cmd)
    }
}

We can run all seeders with a single Make command:

make seed

This quickly populates your database with initial data — like test users — so you can immediately start calling APIs and RPCs without manually adding records. It’s a small step, but it makes your local and CI workflows far more efficient and consistent.

Service Layer Pattern (UserService Example)

Now that our database setup is ready, the next step is to structure how we interact with it. In any well-designed microservice, business logic shouldn’t live inside gRPC or HTTP handlers — that’s where the service layer comes in.

The service layer acts as a clean boundary between your transport layer (gRPC/HTTP) and your data layer (Postgres via GORM). It encapsulates all business logic, leaving handlers focused only on request/response handling. This approach keeps your codebase modular, testable, and easier to evolve as the system grows.

Let’s start with a simple UserService, which lives under:

internal/service/user.go

By isolating logic in a dedicated service, we can easily mock it in tests or extend it later without touching transport logic. It also makes refactoring safer — for instance, swapping GORM for another ORM or even a different persistence layer won’t affect higher layers of the application.

Implementation

package service

import (
    "context"
    "gorm.io/gorm"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database/model"
)

type UserService interface {
    FindByID(ctx context.Context, id uint) (*model.User, error)
}

type userService struct {
    db *gorm.DB
}

func NewUserService(db database.DatabaseService) UserService {
    return &userService{db: db.DB()}
}

func (s *userService) FindByID(ctx context.Context, id uint) (*model.User, error) {
    var user model.User
    if err := s.db.WithContext(ctx).First(&user, id).Error; err != nil {
        return nil, err
    }
    return &user, nil
}

The UserService interface defines the contract for user-related operations. This abstraction makes it easy to plug in mocks for unit testing or replace the implementation later. Its concrete type, userService, handles actual database operations using GORM — keeping queries neatly contained within the service layer.

Integration Testing with Testcontainers

Now that we’ve defined a clean service layer, let’s make sure our business logic actually works when connected to a real database — not just in mocks.

Unit tests are great for verifying logic in isolation, but they can’t always catch real-world issues like invalid SQL, schema drift, or subtle differences between database drivers. To bridge that gap, we’ll use Testcontainers for Go — a Go library that spins up lightweight, disposable Docker containers during tests.

This allows us to run integration tests against a real Postgres instance, ensuring our code works end-to-end just like it would in production.

All integration tests live under:

internal/tests/

and reusable helpers (like database setup) go under:

internal/tests/testutils/

This keeps our tests organized and makes setup logic easy to reuse across services.

Let’s look at an example that tests UserService.FindByID.
Instead of relying on mocks, this test runs against a temporary Postgres container spun up just for this test:

package service_test

import (
    "context"
    "testing"
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database/model"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/service"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/tests/testutils"
)

func TestUserService_FindByID(t *testing.T) {
    db := testutils.SetupPostgres(t)

    // Seed test data
    u := &model.User{Name: "Alice", Email: "alice@example.com"}
    require.NoError(t, db.DB().Create(u).Error)

    userService := service.NewUserService(db)

    got, err := userService.FindByID(context.Background(), u.ID)
    require.NoError(t, err)

    assert.Equal(t, "Alice", got.Name)
    assert.Equal(t, "alice@example.com", got.Email)
}

Setting Up Postgres with Testcontainers

The helper below, SetupPostgres, takes care of everything:

Starts a temporary Postgres container using Testcontainers
Builds a DSN dynamically
Connects through our existing database abstraction
Runs embedded migrations before returning the connection

Here’s the complete implementation:

package testutils

import (
    "io"
    "fmt"
    "time"
    "context"
    "testing"
    "github.com/stretchr/testify/require"
    "github.com/golang-migrate/migrate/v4"
    "github.com/golang-migrate/migrate/v4/source/iofs"
    "github.com/golang-migrate/migrate/v4/database/postgres"
    pgcontainer "github.com/testcontainers/testcontainers-go/modules/postgres"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/database"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

func SetupPostgres(t *testing.T) database.DatabaseService {
    ctx := context.Background()
    log := logger.NewZerologLogger("info", io.Discard)

    dbName := "testdb"
    username := "test"
    password := "test"

    pgContainer, err := pgcontainer.Run(ctx,
        "postgres:15-alpine",
        pgcontainer.WithDatabase(dbName),
        pgcontainer.WithUsername(username),
        pgcontainer.WithPassword(password),
        pgcontainer.BasicWaitStrategies(),
    )
    require.NoError(t, err)

    t.Cleanup(func() {
        if err := pgContainer.Terminate(ctx); err != nil {
            t.Fatalf("failed to terminate container: %s", err.Error())
        }
    })

    host, _ := pgContainer.Host(ctx)
    port, _ := pgContainer.MappedPort(ctx, "5432")
    dsn := fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable",
        username,
        password,
        host,
        port.Port(),
        dbName,
    )

    db, err := database.NewDatabase(&database.Opts{
        Config: &config.Database{
            DSN:                 dsn,
            Driver:              "postgres",
            PoolMaxIdleConns:    10,
            PoolMaxOpenConns:    100,
            PoolConnMaxLifetime: time.Hour,
        },
        Logger: log,
    })
    require.NoError(t, err)

    sqlDB, err := db.DB().DB()
    require.NoError(t, err)

    // Migration driver
    driver, err := postgres.WithInstance(sqlDB, &postgres.Config{})
    require.NoError(t, err)

    // Load embedded migrations
    d, err := iofs.New(database.MigrationsFS, "migrations")
    require.NoError(t, err)

    m, err := migrate.NewWithInstance("iofs", d, dbName, driver)
    require.NoError(t, err)

    // Apply migrations before returning
    err = m.Up()
    if err != nil && err != migrate.ErrNoChange {
        t.Fatalf("failed to run migrations: %v", err)
    }

    return db
}

To make our migrations portable, we embed them directly into the Go binary:

// internal/database/migrations_embed.go

package database

import "embed"

//go:embed migrations/*.sql
var MigrationsFS embed.FS

Embedding ensures the tests (and even compiled binaries) always ship with the correct schema — no filesystem dependencies, no surprises in CI/CD.

The first run might take a bit longer because Docker needs to pull the Postgres image.

Finally, run your integration tests via Make:

make test-integration

Keeping integration tests separate from your fast unit tests ensures a smooth developer workflow while still guaranteeing your application works end-to-end in realistic conditions.

Using the UserService in SayHello RPC

In Part 1, our SayHello RPC only returned a static greeting. Now that we’ve built the user service and connected it to the database, let’s update the RPC to demonstrate how application code can integrate with services and fetch data.

We’ll extend the request to include a user_id, and the response will return both a greeting and the corresponding user object from the database.

message HelloRequest {
  uint64 user_id = 1;
}

message HelloResponse {
  string message = 1;
  User user = 2;
}

message User {
  int64 id = 1;
  string name = 2;
  string email = 3;
}

To use UserService in the gRPC handler, we first pass the database from main.go to NewServer and include it in Opts and GRPCServer:

type Opts struct {
    Config   *config.GRPCServer
    Logger   logger.Logger
    Database database.DatabaseService
}

type GRPCServer struct {
    Server   *grpc.Server
    Config   *config.GRPCServer
    Logger   logger.Logger
    Database database.Database
}

func NewServer(opts *Opts) *GRPCServer {
    srv := grpc.NewServer(grpc.UnaryInterceptor(interceptor.LoggerInterceptor(opts.Logger)))
    helloworld.RegisterGreeterServer(srv, handler.NewGreeterServer(
        service.NewUserService(&service.Opts{
            Database: opts.Database,
        }),
    ))

    return &GRPCServer{
        Server: srv,
        Config: opts.Config,
        Logger: opts.Logger,
    }
}

Add UserService to GreeterServer:

type GreeterServer struct {
    helloworld.GreeterServer
    userService service.UserService
}

func NewGreeterServer(userService service.UserService) *GreeterServer {
    return &GreeterServer{userService: userService}
}

Then update SayHello() to query user by user_id:

func (s *Server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloResponse, error) {
    user, err := s.userService.FindByID(ctx, in.UserId)
    if err != nil {
      return nil, status.Errorf(codes.NotFound, "user not found")
    }

    return &pb.HelloResponse{
            Message: fmt.Sprintf("Hello, %s!", user.Name),
            User: &helloworld.User{
                Id:    int64(user.ID),
                Name:  user.Name,
                Email: user.Email,
            },
    }, nil
}

With a user already seeded in the database, you can test the RPC directly:

grpcurl -d '{"user_id": 1}' -proto ./proto/hello_world/hello_world.proto -plaintext localhost:5000 hello_world.Greeter/SayHello

And get back a response:

{
  "message": "Hello, Alice!",
  "user": {
    "id": 1,
    "name": "Alice",
    "email": "alice@example.com"
  }
}

This small change completes the first true end-to-end flow in our service — from gRPC → service → database → back to gRPC.
It also demonstrates how cleanly each layer interacts, thanks to the structure we’ve built so far.

Conclusion

In Part Two, we:

Integrated PostgreSQL with GORM.
Added migrations and seeders with a CLI entrypoint.
Introduced a service layer pattern with a UserService.
Wrote integration tests using Testcontainers.
Wired the service into our gRPC handler.

In Part Three, we’ll build on this foundation with caching using Redis, observability with Prometheus + OpenTelemetry, and health checks to make the service production-ready.

The repository is meant to be a reusable boilerplate for spinning up new Go microservices — giving you a clean, production-ready starting point for any project.

The master branch contains a clean, production-ready setup, while the examples branch includes all the working examples and demos featured throughout the series.

You can check out the complete code for Part Two here.

Thanks for reading! I hope you found this part helpful. If you have any questions, feedback, or suggestions, feel free to drop a comment. I’d love to hear your thoughts.

Architecting a Maintainable Go Microservice: Structure, Logging, and Reliability

Sagar Maheshwary — Sun, 05 Oct 2025 08:31:57 +0000

Welcome to this series on architecting maintainable and reliable Go microservices.

Instead of jumping straight into code or “yet another boilerplate,” this series focuses on building a service the way real production systems are built, with clear structure, strong foundations, and attention to operational concerns from day one.

We’ll start with the essentials: a predictable project layout, configuration management, structured logging, and a gRPC server with graceful shutdowns. From there, we’ll progressively evolve the service by adding a PostgreSQL-backed data layer, Redis caching, health checks, metrics, tracing, and full observability.

This series is divided into three parts:

Part 1: Project structure, configuration, structured logging, gRPC server, graceful shutdowns, Docker, and Makefile.
Part 2: Database layer with PostgreSQL and GORM, migrations, seeders, service layer, and integration testing with Testcontainers.
Part 3: Redis caching, Prometheus metrics, OpenTelemetry tracing, health checks, and a complete observability demo.

By the end, you’ll have a practical, well-engineered Go microservice foundation you can confidently use in real projects, whether you’re starting something new, improving an existing system, or simply strengthening your backend engineering skills.

Project Structure
Logger Setup
Configuration Setup
gRPC Server
gRPC Interceptors
Graceful Shutdowns
Docker Setup
Makefile
Conclusion

Project Structure

Let’s start by outlining the directory structure.

At the root, we’ll keep standard files (Dockerfile, Makefile, .env, etc.). The main application code will live inside the internal/ directory, following Go’s convention for private packages.

Here’s what we’ll start with in Part 1:

.
├── proto/          # Protobuf definitions and generated code
├── cmd/            # Application entrypoints (cmd/server/main.go)
├── internal/       # Application modules (logger, config, transports, etc.)
├── Dockerfile      # Multi-stage build for dev/prod
├── Makefile        # Workflow automation
├── .env.example    # Example environment variables
└── README.md       # Documentation

Key design choice:

cmd/server/main.go: where the service starts.
internal/: holds actual application logic. Inside, we’ll group modules like logger/, config/, transports/grpc/. Later, we’ll also add database/, cache/, and more.
transports/: keeps protocol-specific code. For example, grpc/server/ for server code, and later grpc/client/ if we want to add gRPC clients. This makes it easy to add more protocols (HTTP, WebSockets, etc.) without cluttering the codebase.

Logger Setup

Logging is one of the first things every service needs. Without good logs, debugging production issues is a nightmare.

We’ll define a Logger interface so we can easily swap implementations later (e.g. zerolog, zap, or a custom logger):

type Logger interface {
    Info(msg string, fields ...Field)
    Warn(msg string, fields ...Field)
    Debug(msg string, fields ...Field)
    Error(msg string, fields ...Field)
    Fatal(msg string, fields ...Field)
    Panic(msg string, fields ...Field)
}

type Field struct {
    Key   string
    Value interface{}
}

We’ll use Zerolog for logging. It’s a lightweight, zero-allocation JSON logger that’s fast enough for production and structured out of the box. That means every log entry is machine-readable (great for aggregators like Loki or ELK) while still being human-friendly in development.

Our NewZerologLogger method initializes the logger with:

Log level parsing – falls back to info if the input is invalid.
Output writer – defaults to stderr but can be swapped (e.g., for testing or file logging).
Timestamps – automatically included with each log line.

Here’s the implementation:

package logger

import (
    "io"
    "os"
    "encoding/json"
    "github.com/rs/zerolog"
)

func NewZerologLogger(level string, out io.Writer) *ZerologLogger {
    lvl, err := zerolog.ParseLevel(level)
    if err != nil {
        lvl = zerolog.InfoLevel
    }
    zerolog.SetGlobalLevel(lvl)

    if out == nil {
        out = os.Stderr
    }

    l := zerolog.New(out).With().Timestamp().Logger()
    return &ZerologLogger{log: l}
}

Now we can initialize it in cmd/server/main.go:

package main

import "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"

func main() {
    log := logger.NewZerologLogger("info", os.Stderr)
    log.Info("Hello, World!", logger.Field{Key: "foo", Value: "bar"})
}

When running the service, we'll see structured JSON logs like:

{
  "level": "info",
  "foo": "bar",
  "time": "2025-09-26T15:23:50Z",
  "message": "Hello, World!"
}

By default, the service keeps log output in structured JSON but during development, you might prefer human-friendly pretty printing. Zerolog supports this via the ConsoleWriter:

package main

import (
    "github.com/rs/zerolog"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

func main() {
    log := logger.NewZerologLogger("info", zerolog.ConsoleWriter{Out: os.Stderr})
    log.Info("Hello, World!", logger.Field{Key: "foo", Value: "bar"})
}

Which produces colorized, easy-to-read output:

The project comes with unit tests for each module. Even though logging might seem trivial, testing it ensures two key guarantees:

Log levels are applied correctly, so you don’t silently miss critical information.
Structured fields are formatted as expected, keeping logs consistent and machine-readable.

Here’s an example test that validates the default log level and structured message output:

package logger_test

import (
    "testing"
    "bytes"
    "github.com/stretchr/testify/assert"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
)

func TestNewZerologLogger_DefaultLevel(t *testing.T) {
    var buf bytes.Buffer
    l := logger.NewZerologLogger("invalid-level", &buf)

    l.Info("hello world", logger.Field{Key: "foo", Value: "bar"})

    entry := parseLog(t, &buf)
    assert.Equal(t, "info", entry["level"])
    assert.Equal(t, "hello world", entry["message"])
    assert.Equal(t, "bar", entry["foo"])
}

//simple helper, transforms JSON to map
func parseLog(t *testing.T, buf *bytes.Buffer) map[string]any {
    t.Helper()

    var logEntry map[string]any
    err := json.Unmarshal(buf.Bytes(), &logEntry)
    require.NoError(t, err)

    return logEntry
}

Let's run the test:

go test ./internal/logger -v

You should see something like this:

=== RUN   TestNewZerologLogger_DefaultLevel
--- PASS: TestNewZerologLogger_DefaultLevel (0.00s)
PASS

Configuration Setup

Next, let’s add configuration management. Services shouldn’t rely on hardcoded values — ports, URLs, database credentials should all come from configs.

We’ll use gofor-little/env to load values from a .env file (useful locally), but also support system environment variables (useful in environments like Kubernetes where env variables are injected from ConfigMaps/Secrets).

Here’s the setup:

package config

import (
    "os"
    "github.com/go-playground/validator/v10"
    "github.com/gofor-little/env"
)

type Config struct {
    GRPCServer *GRPCServer `validate:"required"`
}

type GRPCServer struct {
    URL string `validate:"required,hostname_port"`
}

type LoaderOptions struct {
    EnvPath   string
    EnvLoader func(string) error
    Logger    logger.Logger
}

//helper for NewConfigWithOptions with default values
func NewConfig() (*Config, error) {
    return NewConfigWithOptions(LoaderOptions{
        EnvPath: path.Join(rootDir(), "..", ".env"),
    })
}

func NewConfigWithOptions(opts LoaderOptions) (*Config, error) {
    log := opts.Logger
    if log == nil {
        log = logger.NewZerologLogger("info", os.Stderr)
    }

    envLoader := opts.EnvLoader
    if envLoader == nil {
        envLoader = func(path string) error {
            _, err := os.Stat(path)
            if err != nil {
                return err
            }

            return env.Load(path)
        }
    }

    if err := envLoader(opts.EnvPath); err == nil {
        log.Info("Loaded environment variables from" + opts.EnvPath)
    } else {
        log.Info("failed to load .env file, using system environment variables")
    }

    cfg := &Config{
        GRPCServer: &GRPCServer{
            URL: getEnv("GRPC_SERVER_URL", ":5000"),
        },
    }

    validate := validator.New()
    if err := validate.Struct(cfg); err != nil {
        return nil, fmt.Errorf("invalid config: %w", err)
    }

    return cfg, nil
}

func getEnv(key string, defaultVal string) string {
    if val := os.Getenv(key); val != "" {
        return val
    }

    return defaultVal
}

This setup gives the config loader flexibility. The LoaderOptions struct allows you to customize how environment variables are loaded — for example, by specifying a different .env path or providing your own loading function (useful for tests or different environments). The EnvPath defines where the .env file is located, while the EnvLoader controls how it’s loaded.

Inside NewConfigWithOptions, the loader first tries to read variables from the .env file. If it doesn’t exist, it falls back to system environment variables instead. Finally, all fields are validated using go-playground/validator to ensure the configuration is complete and valid. Centralizing this logic in a single config package keeps configuration handling consistent across all services.

Usage in main.go:

package main

import (
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
)

func main() {
    log := logger.NewZerologLogger("info", nil)

    cfg, err := config.NewConfig()
    if err != nil {
        log.Fatal(err.Error())
    }

    log.Info("gRPC server started!", logger.Field{Key: "Addr", Value: cfg.GRPCServer.URL})
}

Let’s write a test to verify config works as expected:

package config_test

import (
    "os"
    "io"
    "testing"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"
)

func TestNewConfigWithEnvFile(t *testing.T) {
    tmpFile, _ := os.CreateTemp("", "test.env")
    defer os.Remove(tmpFile.Name())
    tmpFile.Write([]byte(`GRPC_SERVER_URL=127.0.0.1:6000`))
    tmpFile.Close()

    cfg, err := config.NewConfigWithOptions(config.LoaderOptions{
        EnvPath: tmpFile.Name(),
        Logger:    logger.NewZerologLogger("info", io.Discard), //doesn't output any logs
    })
    require.NoError(t, err)

    assert.Equal(t, "127.0.0.1:6000", cfg.GRPCServer.URL)
}

This test creates a temporary .env file with a gRPC server config, loads it through NewConfigWithOptions, and verifies the value is correctly mapped to the struct.

gRPC Server

Now that we have logging and configuration in place, let’s bring our service to life with a gRPC server.

Why gRPC?

gRPC (Google Remote Procedure Call) is a high-performance communication framework built on top of HTTP/2. It:

Uses Protocol Buffers (Protobufs) instead of JSON. Protobufs are compact, binary-encoded, and strongly typed.
Supports multiplexing — multiple requests over a single TCP connection.
Is generally faster and more efficient than REST (often cited as up to 10x faster).
Is widely used in microservice architectures, where services often sit behind an API Gateway that translates REST ↔ gRPC (since browsers don’t natively support gRPC).

In internal/transports/grpc/server/server.go, let’s scaffold our server:

package server

import (
    "net"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/config"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    helloworld "path-to-root/proto/hello_world"
    "google.golang.org/grpc"
)

type Opts struct {
    Config *config.GRPCServer
    Logger logger.Logger
}

type GRPCServer struct {
    Server *grpc.Server
    Config *config.GRPCServer
    Logger logger.Logger
}

func NewServer(opts *Opts) *GRPCServer {
    srv := grpc.NewServer()
    return &GRPCServer{
        Server: srv,
        Config: opts.Config,
        Logger: opts.Logger,
    }
}

func (s *GRPCServer) ServeListener(listener net.Listener) error {
    return s.Server.Serve(listener)
}

func (s *GRPCServer) Serve() error {
    listener, err := net.Listen("tcp", s.Config.URL)
    if err != nil {
        return err
    }
    s.Logger.Info("gRPC server started", logger.Field{Key: "Addr", Value: s.Config.URL})
    return s.ServeListener(listener)
}

The Opts struct acts as a dependency container for the gRPC server — it holds configuration and logger references, keeping the server constructor clean and extendable. The NewServer function creates a new gRPC server instance using the provided configuration and logger. The ServeListener method starts the gRPC server on an existing network listener, which can be handy for testing or more customized setups. Finally, the Serve method creates a TCP listener using the configured address and begins serving incoming gRPC requests. Together, these functions form a clean and reusable foundation for running gRPC services in your microservice.

Let's start the server in main.go (in a goroutine since Serve() blocks):

package main

import (
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/transports/grpc/server"
)

func main() {
    //create logger...
    //create config...

    grpcServer := server.NewServer(&server.Opts{
        Config: cfg.GRPCServer,
        Logger: log,
    })

    go func() {
        if err := grpcServer.Serve(); err != nil {
            log.Fatal(err.Error())
        }
    }()
}

To make sure our gRPC server starts correctly, we can write a simple test that boots the server on a random free port (:0), waits briefly for startup, and checks the logs for the expected message:

package server_test
import (
    "bytes"
    "time"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    "github.com/stretchr/testify/assert"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/transports/grpc/server"
)

func TestServe(t *testing.T) {
    var buf bytes.Buffer
    log := logger.NewZerologLogger("info", &buf)

    srv := server.NewServer(&server.Opts{
        Config: &config.GRPCServer{URL: ":0"}, // :0 picks a random free port
        Logger: log,
    })

    go func() { _ = srv.Serve() }()
    defer srv.Server.Stop()

    time.Sleep(100 * time.Millisecond) // give server some time to start

    assert.Contains(t, buf.String(), "gRPC server started")
}

Right now, the server runs but doesn’t expose any RPCs. To define RPCs in gRPC, we use protocol buffer (.proto) files, which describe the API contract. A proto file typically contains:

syntax – the proto version (we’ll use proto3).
package – a namespace for your definitions, preventing name collisions.
option go_package – tells the Go compiler where to place the generated code. Without this, Go packages can clash or be generated in undesired paths.
service – defines a gRPC service and its available RPC methods.
rpc – describes an individual remote procedure call, with input and output message types.
message – defines structured request/response payloads.

Let’s create a simple SayHello RPC that responds with "Hello World":

proto/hello_world/hello_world.proto

syntax = "proto3";

package hello_world;

option go_package = "github.com/sagarmaheshwary/go-microservice-boilerplate/proto/hello_world";

service Greeter {
  rpc SayHello(SayHelloRequest) returns (SayHelloResponse);
}

message SayHelloRequest {}
message SayHelloResponse {
  string message = 1;
}

To generate Go code from proto files, you’ll need to install:

protoc compiler
Go plugins for protobuf and gRPC:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Now we can generate the code with below command:

protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative ./proto/hello_world/hello_world.proto

Next, let’s implement the RPC in Go. The generated code from our .proto file provides an interface (GreeterServer) that we need to implement. We’ll define our handler in grpc/server/handler/greeter.go:

package handler

import (
    "context"
    helloworld "path-to-root/proto/hello_world"
)

type GreeterServer struct {
    helloworld.GreeterServer
}

func NewGreeterServer() *GreeterServer {
    return &GreeterServer{}
}

func (e *GreeterServer) SayHello(ctx context.Context, in *helloworld.SayHelloRequest) (*helloworld.SayHelloResponse, error) {
    return &helloworld.SayHelloResponse{Message: "Hello, World!"}, nil
}

To make this service available, register it inside NewServer():

srv := grpc.NewServer()
helloworld.RegisterGreeterServer(srv, handler.NewGreeterServer())

Now we can start the service and test it using grpcurl, a handy CLI tool for interacting with gRPC servers:

grpcurl -proto ./proto/hello_world/hello_world.proto localhost:5000 hello_world.Greeter/SayHello

Expected response:

{
  "message": "Hello, World!"
}

gRPC Interceptors

In gRPC, an interceptor works similarly to middleware in HTTP frameworks: it lets you inject logic before and after each RPC call. Common use cases include logging, authentication, metrics, and tracing.

An interceptor gives you access to useful fields such as:

info.FullMethod – the full RPC method name (e.g. /hello_world.Greeter/SayHello).
Request/response payloads – the actual objects passed to/from the handler.
Context metadata – request-scoped values, including headers/trailers.
Execution time – you can measure how long an RPC takes.
Error – any error returned from the handler.

Unlike HTTP middleware (which can be attached to specific routes), gRPC interceptors are attached at the server level and therefore run for every RPC. If you need to apply logic only to certain RPCs, you can use a switch or if block inside the interceptor to match against info.FullMethod.

Since logging is a core part of the service, let’s add a simple unary interceptor that logs the RPC method, execution duration, and any error.

Code: grpc/server/interceptor/logger.go

package interceptor

import (
    "context"
    "fmt"
    "time"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/logger"
    "google.golang.org/grpc"
)

func LoggingInterceptor(log logger.Logger) grpc.UnaryServerInterceptor {
    return func(
        ctx context.Context,
        req interface{},
        info *grpc.UnaryServerInfo,
        handler grpc.UnaryHandler,
    ) (resp interface{}, err error) {
        start := time.Now()

        // Call the actual RPC handler
        res, err = handler(ctx, req)

        elapsedMs := fmt.Sprintf("%.2fms", time.Since(start).Seconds()*1000)

        if err == nil {
            log.Info("gRPC request completed",
                logger.Field{Key: "method", Value: info.FullMethod},
                logger.Field{Key: "duration", Value: elapsedMs},
            )
        } else {
            log.Error("gRPC request failed",
                logger.Field{Key: "method", Value: info.FullMethod},
                logger.Field{Key: "duration", Value: elapsedMs},
                logger.Field{Key: "error", Value: err.Error()},
            )
        }

        return res, err
    }
}

let's attach the interceptor when creating the server:

import (
    "google.golang.org/grpc"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/transports/grpc/server/interceptor"
)

server := grpc.NewServer(
    grpc.UnaryInterceptor(interceptor.LoggingInterceptor(logger)),
)

Example log output:

{
  "level": "info",
  "method": "/hello_world.Greeter/SayHello",
  "duration": "0.00ms",
  "time": "2025-09-26T15:34:13Z",
  "message": "gRPC request completed"
}

With this setup, every RPC call is logged consistently without adding boilerplate code in each handler.

Graceful Shutdowns

When shutting down, we want the service to finish active requests and close connections cleanly. Without this, you risk dropping in-flight requests or leaving resources (like DB connections) in a bad state.

In Go, the standard way to handle this is with signal.NotifyContext. It creates a context that gets canceled when the process receives an OS signal (e.g., CTRL+C in dev, or a SIGTERM from Kubernetes during pod termination).

Here’s how we use it in main.go:

package main

import (
    "context"
    "errors"
    "os"
    "os/signal"
    "github.com/sagarmaheshwary/go-microservice-boilerplate/internal/transports/grpc/server"
)

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    //create logger...
    //create config...

    grpcServer := server.NewServer(&server.Opts{
        Config: cfg.GRPCServer,
        Logger: log,
    })

    go func() {
        if err := grpcServer.Serve(); err != nil && !errors.Is(err, grpc.ErrServerStopped) {
            stop() // cancel context if server crashes
        }
    }()

    // Blocks until we receive an interrupt signal
    <-ctx.Done()

    log.Info("Signal received, shutting down gracefully...")
    grpcServer.Server.GracefulStop()
}

This ensures the gRPC server stops accepting new requests but completes any in-flight ones.

Docker Setup

One of the goals of this project is to make running and shipping your service as simple as possible. Docker helps us achieve that by packaging the Go service together with all its dependencies into a portable container.

To support different workflows, the provided Dockerfile is multi-stage:

Builder stage → Compiles the Go binary.
Production stage → Runs only the compiled binary in a minimal Alpine image.
Development stage → Runs with Air for hot reloading during local development.

Here’s the Dockerfile:

# Stage 1: Build Go binary
FROM golang:1.25 AS builder
WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /app/main ./cmd/server/main.go

# Stage 2: Production image (only binary + Alpine)
FROM alpine:3.22 AS production
WORKDIR /app

COPY --from=builder /app/main .

EXPOSE 5000
CMD ["./main"]

# Stage 3: Development image with Air
FROM golang:1.25 AS development
WORKDIR /app

# Copy go.mod and go.sum first for dependency caching
COPY go.mod go.sum ./
RUN go mod download

RUN go install github.com/air-verse/air@v1.52.3

COPY . .

EXPOSE 5000
CMD ["air", "-c", ".air.toml"]

Development mode (hot reload)

Useful when iterating locally. The container mounts your source code, so changes are picked up instantly without rebuilding the image.

docker build --target development -t go-microservice-boilerplate:dev .
docker run -it --rm -p 5000:5000 -v $(pwd):/app go-microservice-boilerplate:dev

Production mode (lightweight & optimized)

Optimized for deployment. The final image contains only the statically compiled binary, making it extremely small and fast to start.

docker build --target production -t go-microservice-boilerplate:latest .
docker run -it --rm -p 5000:5000 go-microservice-boilerplate:latest

Makefile

When working on a project, you often end up typing the same long commands again and again — generating code, building Docker images, or starting the app. A Makefile makes this easier by turning those commands into short, memorable shortcuts.

For example, instead of remembering:

protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative ./proto/**/*.proto

You can just run:

make proto

Note: Using ./proto/**/*.proto tells protoc to look through all subfolders inside proto/ and generate code for every .proto file it finds (e.g. proto/hello_world/hello_world.proto). This is different from the earlier example in the gRPC server section where we only generated code for a single file.

Here’s a simplified Makefile from the project repo:

proto:
    protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative ./proto/**/*.proto

run:
    go run cmd/main.go

docker-build:
    docker build -t go-microservice-boilerplate:latest .

docker-run:
    docker run -it --rm -p 5000:5000 -v .env:/app/.env go-microservice-boilerplate:latest

Now you can simply run:

make proto        # Generate proto code
make run          # Start the app locally
make docker-build # Build the Docker image
make docker-run   # Run the service as docker container

Think of the Makefile as your project’s command center — one place where you and your teammates can find all the common commands without digging through docs or wikis.

You can use make help to find out which commands are available in the repo.

Conclusion

In Part One, we:

Set up project structure.
Added logging and configuration.
Built a gRPC server with an example SayHello RPC.
Implemented graceful shutdown.
Wrote a multi-stage Dockerfile for development and production.
Automated workflows with Makefile.

In Part Two, we’ll integrate PostgreSQL with GORM, set up migrations and seeders, introduce a service layer pattern, and write integration tests using Testcontainers.

The repository is meant to be a reusable boilerplate for spinning up new Go microservices, giving you a clean, production-ready starting point for any project.

The master branch contains a clean, production-ready setup, while the examples branch includes all the working examples and demos featured throughout the series.

You can check out the complete code for Part One here.

Thanks for reading! I hope you found this part helpful. If you have any questions, feedback, or suggestions, feel free to drop a comment. I’d love to hear your thoughts.

DEV Community: Sagar Maheshwary

How I Debug Logs Across Services (and Why I Built a CLI Tool)

What I Wanted

Introducing reqlog

Why Not Just Use grep?

Features (v1)

Performance

Tradeoffs

What’s Next

Feedback

Transactional Outbox with RabbitMQ (Part 2): Handling Retries, Dead-Letter Queues, and Observability

Table of Content

Failure as a First-Class Design Constraint

Retrying Outbox Publishing (Producer-Side Retries)

Schema changes for retries

Making the worker retry-aware

Retrying on transient publish failures

Exponential backoff with jitter

Moving permanently failed events to a DLQ

Consumer-Side Retries Using Dead-Letter Exchanges

Retry Topology

Classifying Failures

Retrying a Message

Why There Is No Native Jitter on the Consumer Side

When Retries Are Exhausted

What To Do With Messages in the DLQ

1. Inspect and discard

2. Fix data and replay

3. Escalate operationally

Retry & Failure Metrics

Retry attempts

Retry exhaustion

Dead-letter publishing

Consumer-side retry & DLQ visibility

Separating concerns with dedicated dashboards

Outbox Reliability Dashboard

Consumer Processing Dashboard

Tracing & Outcome Visibility

Outcomes as first-class signals

Enriching spans with outcomes

Retry visibility without span explosion

Trace–log correlation

CDC-Based Outbox as an Alternative

What CDC Changes

Trade-offs and Operational Cost

Choosing Between Polling and CDC

Conclusion

Transactional Outbox with RabbitMQ (Part 1): Building Reliable Event Publishing in Microservices

Table of Contents

System Overview

Database Schema

outbox_events table

Column breakdown

Indexes

orders table

processed_messages table

Order Service (transactional write)

Application lifecycle and graceful shutdown

Atomic order creation + outbox write

Outbox Worker (Publisher)

Claiming events safely

Concurrent publishing with bounded workers

Publishing and failure handling

Propagating message identity for idempotency

Using RabbitMQ Topic Exchanges

Notification Service (Consumer)

Consumer-side Idempotency

Deduplication with a Unique Constraint

End-to-end test with Docker Compose

What to Do with Published Outbox Events

Hard delete after publish

Retention window (delayed deletion)

Correctness boundary

Operational Metrics

Outbox backlog

Event outcomes

Publish latency

Recording metrics in the worker

Visualization

Trace Propagation Through the Outbox

Introducing `reqlog`

Why Not Just Use `grep`?

`outbox_events` table

`orders` table

`processed_messages` table