DEV Community: Freddie Thompson

Context Propagation Across gRPC and REST: One Broken Hop Splits Every Trace

Freddie Thompson — Mon, 05 Oct 2026 07:50:06 +0000

Your trace shows api-gateway calling orders, then stops. orders definitely called inventory, but inventory appears as a separate, parentless trace.

The context did not propagate. It happens at four specific boundaries, and each needs a different fix.

Boundary 1: protocol translation

W3C Trace Context travels as a traceparent HTTP header. gRPC has metadata, not headers, and the mapping is not automatic unless an instrumented interceptor does it.

from __future__ import annotations

import grpc
from opentelemetry import context as otel_context
from opentelemetry import propagate, trace
from opentelemetry.trace import SpanKind


class _MetadataSetter:
    def set(self, carrier: list, key: str, value: str) -> None:
        # gRPC metadata keys MUST be lowercase. A capitalised key is
        # silently dropped by some implementations and raises in others,
        # which is a fun way to lose context on one language's client only.
        carrier.append((key.lower(), value))


class _MetadataGetter:
    def get(self, carrier, key: str):
        if carrier is None:
            return None
        vals = [v for k, v in carrier if k.lower() == key.lower()]
        return vals or None

    def keys(self, carrier):
        return [k for k, _ in (carrier or [])]


_SETTER, _GETTER = _MetadataSetter(), _MetadataGetter()
_tracer = trace.get_tracer(__name__)


class TracingClientInterceptor(grpc.UnaryUnaryClientInterceptor):
    def intercept_unary_unary(self, continuation, client_call_details, request):
        with _tracer.start_as_current_span(
            client_call_details.method, kind=SpanKind.CLIENT
        ) as span:
            metadata = list(client_call_details.metadata or [])
            propagate.inject(metadata, setter=_SETTER)

            new_details = client_call_details._replace(metadata=metadata)
            response = continuation(new_details, request)

            code = getattr(response, "code", lambda: None)()
            if code not in (None, grpc.StatusCode.OK):
                span.set_attribute("rpc.grpc.status_code", str(code))

            return response


class TracingServerInterceptor(grpc.ServerInterceptor):
    def intercept_service(self, continuation, handler_call_details):
        ctx = propagate.extract(
            list(handler_call_details.invocation_metadata or []),
            getter=_GETTER
        )

        token = otel_context.attach(ctx)

        try:
            return continuation(handler_call_details)
        finally:
            # Detaching is not optional. A leaked context attaches itself to
            # whatever request the worker thread handles next, and you get
            # spans from request B parented under request A.
            otel_context.detach(token)

The lowercase-metadata rule is worth internalising. It fails asymmetrically — a Go client talking to a Python server may work while the reverse does not — which makes it look like a language bug rather than a casing bug.

Boundary 2: message queues

A trace that crosses Kafka or SQS has a producer and a consumer that may run hours apart. Parent-child is the wrong relationship; use a link.

from opentelemetry.trace import Link, NonRecordingSpan, SpanContext


def produce(topic: str, payload: bytes, producer) -> None:
    with _tracer.start_as_current_span(
        f"publish {topic}",
        kind=SpanKind.PRODUCER
    ):
        headers: list = []
        propagate.inject(headers, setter=_SETTER)
        producer.send(topic, value=payload, headers=headers)


def consume(msg) -> None:
    parent_ctx = propagate.extract(
        list(msg.headers or []),
        getter=_GETTER
    )

    parent_span = trace.get_current_span(parent_ctx)

    # A link, not a parent. The consumer may run long after the producer's
    # trace ended; making it a child produces traces with multi-hour gaps
    # that break every latency percentile you compute from span duration.
    links = []

    sc = parent_span.get_span_context()

    if sc.is_valid:
        links.append(Link(sc))

    with _tracer.start_as_current_span(
        f"process {msg.topic}",
        kind=SpanKind.CONSUMER,
        links=links
    ) as span:
        span.set_attribute("messaging.kafka.partition", msg.partition)
        span.set_attribute("messaging.kafka.offset", msg.offset)
        handle(msg)

Boundary 3: thread pools and async

Context is stored in a context-local. Hand work to a thread pool and the new thread has a fresh, empty one.

import contextvars
from concurrent.futures import ThreadPoolExecutor
from functools import partial


class ContextPreservingExecutor(ThreadPoolExecutor):
    """
    Copies the caller's context into the worker thread.

    Without this, every span produced inside a pool task is a new root and
    your trace loses everything that happens in a background worker - which
    is usually the slow part you were trying to see.
    """

    def submit(self, fn, /, *args, **kwargs):
        ctx = contextvars.copy_context()
        return super().submit(
            partial(ctx.run, fn),
            *args,
            **kwargs
        )

asyncio.create_task copies context automatically. loop.run_in_executor and raw Thread do not. Neither does most third-party library code that manages its own pool, which is why a trace can break inside a database driver you did not write.

Boundary 4: sampling decisions

If service A samples a trace out and service B makes its own independent decision, you get a child span with no parent. Always use ParentBased sampling so the root’s decision is authoritative downstream.

Finding the broken hop

Do not hunt for these manually. Query your tracing backend for spans with a parent ID that does not resolve:

def find_orphans(spans: list[dict]) -> dict[str, int]:
    """
    Group orphaned spans by service. The service that shows up is the
    one RECEIVING broken context - so the bug is in its caller, or in its
    own extract path. Run this weekly; it catches a regression the day a
    new service ships without the interceptor.
    """
    known = {s["span_id"] for s in spans}
    orphans: dict[str, int] = {}

    for s in spans:
        parent = s.get("parent_span_id")

        if parent and parent not in known:
            svc = s.get("service_name", "unknown")
            orphans[svc] = orphans.get(svc, 0) + 1

    return dict(
        sorted(
            orphans.items(),
            key=lambda kv: -kv[1]
        )
    )

Caveat: a parent can be legitimately missing because it was sampled out or is still in flight. Run this over a completed window and treat a rate as the signal, not individual cases. A service that jumps from 2% orphans to 60% after a deploy has a propagation bug; one that sits steadily at 3% probably has sampling.

Propagate more than trace context

traceparent gets you the trace. Baggage carries application context — tenant ID, feature flag cohort, request priority — to every downstream service without threading it through every function signature.

Two warnings. Baggage travels in headers to every hop, including third parties, so it must never contain anything sensitive. And it costs bytes on every call; a few small keys are fine, a serialised user object is not.

We build distributed backend systems at SoluLab — more on our custom software development work.

Multi-Tenant Database Isolation: RLS, Schema-per-Tenant, and the Migration Wall

Freddie Thompson — Mon, 05 Oct 2026 07:47:25 +0000

Three ways to keep tenants apart in one Postgres cluster:

Discriminator column — tenant_id on every table, filtered in application code.
Row-level security — tenant_id plus RLS policies the database enforces.
Schema-per-tenant — each tenant gets its own set of tables.

Option 1 is not isolation. It is a convention, and it fails the first time someone writes a query without the filter. That is not hypothetical — it is one of the most common sources of cross-tenant data exposure, and it happens in a reporting endpoint or an admin tool rather than the main code path, which is why review misses it.

The real choice is between 2 and 3, and it comes down to a trade most teams discover too late: RLS scales with tenant count, schema-per-tenant scales with tenant size.

Row-level security, done correctly

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;

ALTER TABLE invoices FORCE ROW LEVEL SECURITY;
-- applies to the table owner too

CREATE POLICY tenant_isolation ON invoices
    USING (tenant_id = current_setting('app.tenant_id', true)::uuid)
    WITH CHECK (tenant_id = current_setting('app.tenant_id', true)::uuid);

Three things in there that are load-bearing:

FORCE ROW LEVEL SECURITY. Without it, the table owner bypasses the policy entirely. If your application connects as the owner — which it does by default in most setups — RLS is enabled and doing absolutely nothing. This is the single most common way an RLS deployment provides no protection while appearing to.

WITH CHECK as well as USING. USING filters reads. Without WITH CHECK, a tenant can insert a row with someone else’s tenant_id. You get read isolation and write contamination.

The true second argument to current_setting. Makes it return NULL instead of erroring when unset. That sounds convenient and is the trap — tenant_id = NULL is NULL, which is not true, so the policy denies everything. Failing closed is right; just know that an unset variable produces “no rows” rather than an error, which is a confusing thing to debug.

And the part that actually enforces it:

from contextlib import contextmanager


@contextmanager
def tenant_scope(conn, tenant_id: str):
    """
    SET LOCAL is scoped to the transaction and cannot leak into the next
    checkout from the pool. Plain SET persists on the connection - with
    PgBouncer in transaction mode that means the next tenant inherits it,
    which is a cross-tenant read with no bug in your query.
    """
    with conn.transaction():
        conn.execute("SET LOCAL app.tenant_id = %s", (tenant_id,))
        yield conn


# The role matters as much as the policy.
# CREATE ROLE app_user NOLOGIN;
# GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_user;
# -- app_user is NOT the table owner, so FORCE is not the only thing
# -- standing between you and a full-table read.

SET LOCAL versus SET is the whole connection-pooling story. Get it wrong and the failure is intermittent, load-dependent, and cross-tenant.

Where each model breaks

RLS breaks on the query planner. Policies are predicates, and they interact with indexes in ways that are not always obvious. A composite index on (tenant_id, created_at) is essential; without it the planner may filter after scanning. Worse, RLS can prevent certain plan shapes entirely, and a query that is fast for a small tenant becomes a sequential scan for a large one. Check EXPLAIN output as a tenant, not as a superuser, or you are reading a plan the application will never get.

Schema-per-tenant breaks on migrations. At 4,000 schemas, ALTER TABLE runs 4,000 times. A migration that takes 8 seconds per schema is nine hours, and it is not atomic — you will be half-migrated when something fails, with application code that has to tolerate both shapes. Add connection-level catalog bloat: Postgres’s system catalogs grow with schema count, and planning time degrades measurably in the thousands.

Both break on cross-tenant queries. Admin dashboards, usage billing, aggregate analytics. Under RLS you need a separate role that bypasses policies, which is a privileged path you must guard as carefully as the policies themselves. Under schema-per-tenant you need a UNION across thousands of schemas, which is not viable, so you end up with a separate analytics store and a pipeline to keep it current.

The decision, concretely

Use RLS when you have many small tenants (SaaS, self-serve), you need one migration to cover everyone, and no tenant demands physical separation. This is most products.

Use schema-per-tenant when tenants are few and large, they have genuinely different schemas (custom fields as real columns), or a contract requires demonstrable physical separation. Enterprise and regulated deployments.

Use a separate database or cluster when a tenant’s contract requires it, or when one tenant’s load would otherwise degrade everyone. This is the honest answer for a handful of large accounts and it is fine to run a hybrid: RLS for the long tail, dedicated clusters for the top ten. The routing layer that picks the connection is a small amount of code and it saves you from designing for the worst case everywhere.

Testing isolation, which is the part that gets skipped

Isolation is a security property, so test it like one — adversarially, in CI, on every build:

import pytest


@pytest.mark.parametrize("table", ALL_TENANT_TABLES)
def test_no_cross_tenant_read(db, table):
    seed(db, tenant="A", table=table, rows=3)
    seed(db, tenant="B", table=table, rows=5)

    with tenant_scope(db, tenant_id=TENANT_A) as conn:
        rows = conn.execute(
            f"SELECT tenant_id FROM {table}"
        ).fetchall()

    assert rows, f"{table}: tenant A should see its own rows"
    assert {r[0] for r in rows} == {TENANT_A}, (
        f"{table}: cross-tenant leak"
    )


@pytest.mark.parametrize("table", ALL_TENANT_TABLES)
def test_cannot_write_other_tenant(db, table):
    with tenant_scope(db, tenant_id=TENANT_A) as conn:
        with pytest.raises(Exception):
            conn.execute(
                f"INSERT INTO {table} (tenant_id, ...) VALUES (%s, ...)",
                (TENANT_B,),
            )


def test_every_tenant_table_has_rls(db):
    """
    Catches the new table someone added without a policy. This test is
    worth more than the other two combined, because the failure it prevents
    is the one that actually happens.
    """
    missing = db.execute("""
        SELECT c.relname FROM pg_class c
        JOIN pg_namespace n ON n.oid = c.relnamespace
        WHERE n.nspname = 'public' AND c.relkind = 'r'
          AND c.relname = ANY(%s)
          AND (NOT c.relrowsecurity OR NOT c.relforcerowsecurity)
    """, (ALL_TENANT_TABLES,)).fetchall()

    assert not missing, f"tables without forced RLS: {missing}"

That third test is the important one. RLS is opt-in per table, so the realistic failure is not a broken policy — it is a table shipped six months from now by someone who did not know the convention. A catalog assertion in CI catches it; code review does not.

We build multi-tenant platforms where this decision gets made early and lived with for years — the MedicGraph patient management case study is one of them.

Zero-Downtime Database Migration: The Dual-Write Window Is Where It Goes Wrong

Freddie Thompson — Mon, 05 Oct 2026 07:44:24 +0000

The standard playbook for moving a table to a new store is four phases: dual-write, backfill, verify, and cut over. Every phase has a failure mode that produces silent, permanent divergence, and the order the phases run in is load-bearing.

The one that catches most teams: if you backfill before dual-writing, you lose every row written during the backfill. The backfill reads a snapshot; concurrent writes land in the old store only; you finish, see matching row counts, and ship. The missing rows are the ones written during the window, which is exactly your busiest data.

Dual-write must be live before the backfill starts, not after.

Phase 1: Dual-Write, Old Store Authoritative

from __future__ import annotations

import logging
import time
from dataclasses import dataclass
from enum import Enum
from typing import Any, Protocol

log = logging.getLogger(__name__)


class Phase(str, Enum):
    OLD_ONLY = "old_only"
    DUAL_OLD_AUTH = "dual_old_auth"  # write both, read old, compare async
    DUAL_NEW_AUTH = "dual_new_auth"  # write both, read new, old is fallback
    NEW_ONLY = "new_only"


class Store(Protocol):
    def write(self, key: str, value: dict) -> None: ...
    def read(self, key: str) -> dict | None: ...


@dataclass
class MigrationRouter:
    old: Store
    new: Store
    phase: Phase
    metrics: Any

    def write(self, key: str, value: dict) -> None:
        if self.phase is Phase.NEW_ONLY:
            self.new.write(key, value)
            return

        # Authoritative store first, and a failure there propagates.
        auth, shadow = (
            (self.old, self.new)
            if self.phase in (Phase.OLD_ONLY, Phase.DUAL_OLD_AUTH)
            else (self.new, self.old)
        )

        auth.write(key, value)

        if self.phase is Phase.OLD_ONLY:
            return

        # Shadow write failures must NOT fail the request. If they do, the
        # migration has coupled your availability to the store you are still
        # testing - and you will roll back a migration that was fine.
        try:
            shadow.write(key, value)
        except Exception:
            self.metrics.incr("migration.shadow_write_failed")
            log.exception("shadow write failed key=%s", key)

    def read(self, key: str) -> dict | None:
        if self.phase in (Phase.OLD_ONLY, Phase.DUAL_OLD_AUTH):
            return self.old.read(key)

        result = self.new.read(key)

        if result is None and self.phase is Phase.DUAL_NEW_AUTH:
            # Fallback during cutover. Every hit here is a real gap - alert on it.
            self.metrics.incr("migration.new_store_miss")
            return self.old.read(key)

        return result

Two decisions in there matter more than they look.

1. Swallow and Count Shadow-Write Failures

Shadow-write failures are swallowed and counted. The instinct is to fail loudly. Do not — you have just made your production availability depend on the store you are still validating.

Count them, alert on the rate, and let the reconciler repair them.

2. Alert on `new_store_miss`

new_store_miss gets an alert, not a log line. Once you flip to new-authoritative, every fallback read is a row that should be in the new store and is not.

If that counter is non-zero, you are not ready to cut over. This is the single metric that tells you so.

Phase 2: Backfill With a Stable Cursor

def backfill(
    router: MigrationRouter,
    source_scan,
    batch_size: int = 500
) -> int:
    """
    Copy historical rows.

    MUST run with dual-write already live.

    Uses a keyset cursor, not OFFSET:
    OFFSET pagination over a table taking concurrent writes skips
    and duplicates rows as pages shift underneath it.
    """
    copied = 0
    cursor = ""

    while True:
        rows = source_scan(after=cursor, limit=batch_size)

        if not rows:
            break

        for key, value in rows:
            existing = router.new.read(key)

            # Never overwrite. A live dual-write may have already put a NEWER
            # value here; the backfill is reading an older snapshot. Clobbering
            # it silently reverts a customer's write.
            if existing is None:
                router.new.write(key, value)
                copied += 1

        cursor = key

        # Crude throttle; prefer a token bucket keyed on replica lag
        # so the backfill self-limits.
        time.sleep(0.05)

    return copied

The if existing is None guard is the second classic corruption.

A backfill that blindly writes will overwrite fresher dual-written values with stale snapshot data, and the damage is invisible until a customer reports that an edit reverted.

For rows that genuinely need updating, you need a version or updated_at comparison rather than a presence check.

The source of truth for that comparison has to be a column the application actually maintains, not a database-generated timestamp that the backfill itself bumps.

Phase 3: Verify Continuously, Not Once

A one-shot comparison at the end tells you nothing about steady state. Sample continuously through the whole dual-write period:

def sample_and_compare(
    router: MigrationRouter,
    keys: list[str]
) -> dict:
    mismatches, missing, checked = [], [], 0

    for key in keys:
        old_val = router.old.read(key)
        new_val = router.new.read(key)

        checked += 1

        if new_val is None and old_val is not None:
            missing.append(key)
        elif old_val != new_val:
            mismatches.append((key, old_val, new_val))

    return {
        "checked": checked,
        "missing": len(missing),
        "mismatched": len(mismatches),
        "samples": mismatches[:5],
    }

Expect a non-zero mismatch rate from timing alone — you read the two stores microseconds apart, and a write can land between those reads.

Re-check a mismatch after a short delay before counting it:

A mismatch that persists is real.
A mismatch that resolves is replication lag.

Failing to make that distinction produces an alert channel everyone learns to ignore, which is worse than no alerting.

Phase 4: Cut Over and Keep the Way Back

Flip:

DUAL_OLD_AUTH → DUAL_NEW_AUTH

Do this behind a runtime flag, not a deploy.

You want the rollback to be a configuration change measured in seconds because the thing you are protecting against is a failure that shows up under production load and could not be reproduced in staging.

Keep dual-write running for at least one full business cycle after the flip — a week, a month-end, or whatever your longest periodic job is.

The rollback path only exists while the old store is still current.

The moment you stop writing to it, NEW_ONLY is irreversible. That step should therefore be a separate, deliberate, scheduled change rather than the tail end of the cutover.

What Makes This Genuinely Hard

Transactions That Span Both Stores

If a single logical operation writes to two tables and only one has migrated, you have lost atomicity for the duration of the migration.

There is no clean fix.

The options are to:

Migrate the whole transactional unit together.
Accept a reconciliation job for the migration window.

Pick deliberately rather than discovering the problem mid-migration.

Schema Changes During the Migration

Someone will want to add a column while you are three weeks into a dual-write.

The migration router has to tolerate the change on both sides. That means:

The new store's schema has to remain a superset.
The write path has to tolerate unknown fields.

Reads That Are Not Key-Value

Everything above assumes point reads.

Range queries, aggregates, and joins do not dual-read cleanly.

A COUNT(*) disagreeing between stores during a backfill is expected, not necessarily a bug. Any verification built on aggregates will therefore produce noise for the entire migration.

Final Takeaway

A zero-downtime database migration is not simply a matter of copying data and switching traffic.

The dual-write window is the critical safety mechanism. Start dual-writing before the backfill, protect newer writes from stale backfill data, verify continuously, monitor fallback reads, and keep the old store writable long enough to preserve a real rollback path.

We do data platform and legacy modernization work at SoluLab — the InfuseNet case study covers one of these migrations end to end.