DEV Community: Freddie Thompson

Building AI-Assisted Tax Document Workflows: Engineering Considerations for Tokenization and Evidence

Freddie Thompson — Tue, 27 Oct 2026 12:45:58 +0000

AI and tokenization are often discussed as separate technology trends. Tax-related workflows offer an opportunity to examine how they might fit into a broader information-management system.

The important engineering question is not simply how to extract data from a form or create a digital token. It is how to maintain traceability, protect sensitive information, and ensure that downstream decisions can be reviewed.

SoluLab's AI-powered tax form tokenization and compliance evidence platform case study is a useful project-specific reference for this topic.

Model the Workflow Before Selecting the Stack

Start by documenting the stages that a tax document passes through.

Depending on the use case, the requirements may include ingestion, classification, extraction, validation, review, evidence storage, and retrieval.

Each stage should have a defined input, output, failure state, and access policy.

For example, an extraction failure should not silently produce an accepted compliance record. It should create an exception that can be investigated.

Make AI Outputs Verifiable

AI-assisted extraction can reduce repetitive work, but extracted values must remain connected to their sources.

A robust design should consider:

Field-level validation.

Confidence thresholds and human-review queues.

Handling of missing or conflicting values.

Versioning of documents and corrections.

Logging of important processing and approval events.

The exact controls will depend on the data, risk level, and applicable compliance requirements.

Define the Tokenization Boundary

Before implementing tokenization, answer a basic question: What exactly does the token represent?

It might represent a defined asset or another explicitly modeled entity. The architecture must specify the relationship between that representation and its authoritative data.

Avoid assuming that sensitive tax documents should be stored directly on-chain. Data placement, access requirements, privacy obligations, and the need for corrections must all be assessed first.

Likewise, do not select a blockchain or token standard until the functional requirements justify that choice.

Design Evidence Retrieval as a Core Feature

Evidence management should not be an afterthought.

Developers need to consider provenance, permissions, integrity checks, retention, and how an authorized reviewer can retrieve the supporting source for a particular record.

A system may demonstrate that a record has not changed without proving that the information was originally correct. That distinction matters in compliance-related applications.

Test Failure Paths, Not Just the Happy Path

Useful test scenarios include:

Documents with missing fields.

Conflicting values across records.

Low-confidence extraction results.

Unauthorized evidence requests.

Corrected or superseded documents.

Failed processing and retry behavior.

Testing should evaluate the complete workflow rather than only the performance of the AI component.

Takeaway

AI processing, tokenization, and evidence management solve different problems. A dependable implementation must define the boundaries between them and make those boundaries testable.

For developers, the strongest starting point is a clear data model, explicit validation rules, and a documented evidence lifecycle. Technology choices should follow from those requirements—not the other way around.

Blockchain in Banking: What Developers Should Evaluate Before Building

Freddie Thompson — Mon, 26 Oct 2026 11:45:15 +0000

Blockchain-based banking applications involve more than writing smart contracts or storing transactions on a ledger. Developers must understand the business workflow, system boundaries, security requirements, and integration constraints before choosing an implementation.

Define the Use Case

Start by documenting the existing workflow.

Which participants need access to the records?

What information must be shared?

Who is authorized to submit or verify transactions?

What are the expected transaction volumes?

Which existing systems must remain operational?

These questions help determine whether a blockchain architecture is appropriate.

Design for Security

Financial applications require careful handling of identities, permissions, sensitive data, and transaction authorization.

Developers should consider which information belongs on-chain and which should remain in protected off-chain systems. The answer depends on the application and its privacy requirements.

Plan for Integration

Blockchain applications frequently need to exchange information with existing software. API design, data consistency, error handling, monitoring, and recovery procedures should be part of the architecture discussion.

Validate Performance

Before deployment, test transaction throughput, latency, failure scenarios, and expected infrastructure requirements. Performance assumptions should be validated under realistic workloads.

Review a Banking Case Study

SoluLab's Chichin Blockchain in Banking project is a relevant starting point for exploring a blockchain application in the banking domain.

Consult the original project details before drawing conclusions about the selected framework, architecture, consensus mechanism, or implementation results.

Final Thoughts

The engineering challenge in banking is not simply to introduce blockchain. It is to design a secure, maintainable solution that addresses a clearly identified business requirement.

Start with the workflow, document the constraints, evaluate the architecture, and test the assumptions before scaling the solution.

Building a Multi-Chain DEX: What Blockchain Developers Should Consider

Freddie Thompson — Fri, 23 Oct 2026 11:36:41 +0000

Multi-chain development introduces a practical engineering challenge: how do you support different blockchain environments without making the application unnecessarily complex?

For decentralized exchanges (DEXs), this question affects blockchain integrations, transaction handling, smart contract security, and the user interface.

Start with Network Requirements

Before writing integration code, define which blockchain networks the product needs to support and why.

For each network, document:

Transaction and confirmation behavior

Smart contract compatibility

Available tooling and infrastructure

Asset identification requirements

Network-specific failure scenarios

Avoid adding chains without a clear product or technical reason. Every additional integration increases the testing and maintenance surface.

Separate Network Logic from Application Logic

A maintainable multi-chain application benefits from clear boundaries between the user interface, business logic, and network-specific integrations.

A conceptual architecture might look like this:

User Interface
|
Application Services
|
Network Integration Layer
|
Supported Blockchain Networks

This is a general architectural illustration, not a claim about the implementation used in any specific project.

The purpose is to make responsibilities easier to understand and network-specific changes easier to manage.

Design for Transaction Failures

A transaction may be pending, rejected, reverted, or delayed. The interface should communicate these states clearly and avoid treating transaction submission as proof of successful execution.

Developers should test:

Wallet rejection

RPC or provider failures

Insufficient funds

Reverted transactions

Delayed confirmations

Network switching errors

Include Security in the Development Lifecycle

Smart contracts, external dependencies, and transaction flows need appropriate security reviews. If the architecture uses cross-chain messaging or bridging, those components require additional threat analysis.

Do not assume that connecting multiple networks automatically creates cross-chain asset interoperability. That depends on the specific implementation.

Plan for Ongoing Maintenance

Network integrations change over time. Monitoring, regression tests, dependency updates, and documented recovery procedures help keep the application maintainable.

For a project reference, see SoluLab's Multi-Chain DEX case study.

The project can be explored alongside these general engineering considerations, while implementation-specific details should be taken from the published case study.

Takeaway

The difficult part of multi-chain DEX development is managing differences between networks without compromising security or usability.

Start with clear network requirements, define integration boundaries, test failure scenarios, and treat every supported chain as an ongoing engineering responsibility.

What has been your biggest challenge with multi-chain application development: RPC reliability, transaction handling, or smart contract security?

blockchain #web3 #defi #programming

Building Trust into Digital Document Workflows with Blockchain

Freddie Thompson — Thu, 22 Oct 2026 11:24:04 +0000

The problem developers should start with

Document management sounds straightforward: upload a file, store it, and retrieve it when needed.

Real-world requirements can be more complicated. Depending on the application, developers may also need to support document verification, activity traceability, permissions, and reliable records of important actions.

Blockchain can be considered when these requirements involve a need for tamper-resistant records.

What should the architecture consider?

Before writing smart contracts or selecting a blockchain network, define the system's responsibilities.

Document storage: Determine where files will live and how they will be retrieved.

Verification: Establish what users need to verify and what evidence the system should maintain.

Identity: Define how users and organizations are authenticated.

Authorization: Separate authentication from permissions. Knowing who a user is does not automatically mean they should be able to access every document.

Privacy: Avoid assuming that putting information on a blockchain makes it private.

These decisions should be based on the actual application requirements.

A useful distinction: the file versus its record

A blockchain-based design may maintain selected records related to a document without placing the entire file on-chain.

For example, a system could be designed to verify a document against a previously recorded cryptographic reference. Whether that approach is suitable depends on the implementation and the required security properties.

This is a general architectural pattern, not a claim about the specific implementation used in DocTrace.

Exploring DocTrace

SoluLab's DocTrace project provides a case study focused on blockchain and document management.

You can review the DocTrace project case study to explore the documented project context.

Practical lessons for blockchain developers

Start with a clear problem statement.

Decide what belongs on-chain and what belongs off-chain.

Design permissions and key management early.

Consider privacy and data-retention requirements.

Test failure scenarios, not just successful transactions.

Make verification understandable to the people using the application.

Blockchain is one component of a larger system. The engineering challenge is to combine it with storage, identity, application logic, and a usable interface to meet a defined business need.

What would you prioritize first in a blockchain document management project: integrity verification, access control, or integration with existing systems?

GenAI + Tokenized RWAs: What Developers Should Consider Before Building

Freddie Thompson — Wed, 21 Oct 2026 10:58:22 +0000

Real-world asset (RWA) tokenization and generative AI solve different problems, but they can potentially complement each other in an application.

Tokenization provides a digital framework for representing assets. Generative AI can help users interact with information through natural-language interfaces, summarization, and other information-processing tasks.

The interesting engineering challenge is deciding where AI belongs in the architecture.

Start with the Data Model

Before introducing an AI component, identify the information the application manages.

Depending on the project, this may include asset metadata, supporting documents, business rules, and transaction-related records. Each data source should have a clear owner, access policy, and level of authority.

Do not assume that an AI-generated summary is equivalent to a verified asset record.

Define the AI Component's Responsibility

A useful design exercise is to separate informational tasks from authoritative operations.

For example, an application might use AI to help users understand documentation. Any actual asset update or transaction would still need to follow the platform's established validation and authorization rules.

This separation reduces the risk of treating generated content as an instruction that can execute without review.

Think About Security Early

When connecting AI to asset-related data, consider:

Which users can access each category of information?
Can the model retrieve information the current user is not authorized to see?
How will incorrect or unsupported answers be handled?
Which actions require explicit approval?
How can important operations be audited?

The right controls depend on the application's risk profile and architecture.

Evaluate the Actual Use Case

Before choosing a model or adding more infrastructure, define what success means.

Possible evaluation criteria include answer accuracy, retrieval relevance, latency, task completion, and the rate of outputs requiring human correction. Select metrics based on the intended functionality rather than using them as generic performance claims.

A Case Study to Explore

SoluLab's GenAI in Tokenized RWA case study is relevant to developers investigating the intersection of generative AI and tokenized real-world assets.

When reviewing the project, look for the documented use case, implemented features, technology choices, and evidence of outcomes. Avoid inferring a specific model, blockchain, token standard, or deployment architecture unless the source confirms it.

Final Thought

The goal is not to put an LLM in every part of a tokenization stack. It is to identify a genuine information or workflow problem and determine whether AI can solve it without weakening the system's controls.

For developers, clear component boundaries are just as important as choosing the right technology.

Crypto Launchpad Development: Planning the Architecture Before Writing Code

Freddie Thompson — Mon, 19 Oct 2026 11:11:53 +0000

Crypto launchpad projects present an interesting engineering challenge: the application must support a defined product workflow while accounting for blockchain-specific behavior.

Before implementing contracts or integrating a blockchain SDK, developers should understand the platform's functional requirements.

Step 1: Define the Requirements

Start by documenting:

User roles and permissions
Project discovery and information requirements
The intended token launch workflow
On-chain versus off-chain responsibilities
Transaction states and error scenarios
Security and testing expectations

This requirement list becomes the basis for architectural decisions.

Step 2: Separate Application Concerns

A typical design discussion may cover the following responsibilities:

Frontend: Displays project information and guides users through the application.
Backend: Manages application-specific logic and operational workflows.
Blockchain integration: Handles communication with the selected network.
Smart contracts: Enforce required on-chain rules, if applicable.
Testing and monitoring: Help validate expected behavior and identify failures.

This is a general planning model, not a claim about the exact architecture used in the Token World project. The final design should follow the actual requirements.

Step 3: Design for Failure

Blockchain interactions introduce scenarios that a conventional interface cannot simply ignore.

Developers should plan for:

Rejected transactions
Network interruptions
Delayed transaction confirmation
Invalid user inputs
Unexpected contract responses
Inconsistent application state

The interface should communicate what happened and what the user can safely do next.

Step 4: Make Security Part of Development

Security review should cover the application and any on-chain components. Depending on the implementation, this may include permission checks, input validation, contract testing, and careful handling of transaction-related information.

Avoid assuming that a successful deployment automatically means an application is secure.

Learning From a Real Project

SoluLab's Token World crypto launchpad case study offers a project reference for developers exploring crypto launchpad solutions.

When evaluating a case study, look for documented requirements, technology choices, implementation details, and evidence of the resulting capabilities. These details are more useful for architecture planning than assumptions based on the project category alone.

Final Takeaway

For blockchain developers, a successful launchpad project begins with clear requirements, a deliberate architecture, well-defined failure handling, and security-conscious engineering.

Choose the technology stack after understanding the workflow it must support.

Canonical URL: Configure the original SoluLab URL as the canonical URL if DEV Community's current publishing interface supports it. Verify the setting before publishing.

Gold Tokenization: What Blockchain Developers Should Consider Before Building

Freddie Thompson — Fri, 16 Oct 2026 10:35:00 +0000

Gold tokenization sits at the intersection of smart contracts, asset management, and real-world operational processes.

For developers, the interesting challenge is not just creating a token. It is designing a system in which the digital representation has a clearly defined relationship with the underlying asset.

1. Define the Asset Model First

Before choosing a token standard or writing a smart contract, establish what the token represents.

Possible structures may involve direct ownership, a beneficial interest, or a contractual claim. Each structure has different legal and operational implications.

Developers should not treat these models as interchangeable.

2. Establish the Connection Between Physical and Digital Records

A blockchain can record token issuance and transfers, but it cannot independently verify the existence or condition of physical gold.

A complete design needs a defined process for verifying the underlying assets and reconciling the relevant records.

Questions to resolve include:

Who verifies the gold?
How are asset records maintained?
What authorizes token issuance?
How are discrepancies handled?
What happens when an asset's status changes?

3. Design Token Controls Around Real Requirements

Smart-contract controls should reflect the platform's documented operating model.

Depending on the requirements, engineers may need to consider role-based permissions, controlled issuance, transfer restrictions, event logging, and secure administrative operations.

These are potential design considerations, not claims about the specific implementation of any individual platform.

4. Treat Security and Operations as Part of the System

Smart-contract security is only one layer of the overall solution. Operational procedures, access controls, supporting records, and incident-response plans also matter.

A contract can execute exactly as written while the wider asset-management process still has weaknesses. This is why technical testing and operational validation should be considered together.

5. Review the Project Context

SoluLab's Tangible Gold case study is a relevant project reference for developers researching gold and digital asset tokenization.

When reviewing the project, focus on the documented requirements, implemented functionality, technical decisions, and any explicitly reported outcomes.

Final Takeaway

The key engineering lesson is to start with the asset and ownership model, then design the blockchain layer around it.

A well-designed tokenization solution needs a consistent relationship between its digital records and the real-world rights those records are intended to represent.

Building a Tokenized Clean Energy Investment Platform: What Developers Should Consider

Freddie Thompson — Thu, 15 Oct 2026 10:37:11 +0000

Tokenization projects require more than smart contracts. When the underlying use case involves clean energy investment, developers must think about asset representation, transaction records, application security, and the relationship between digital tokens and real-world rights.

1. Define the Asset Model First

Before choosing a blockchain or token standard, establish what the platform represents.

Questions to answer include:

What is the underlying asset?
What rights does a token represent?
Who verifies the asset information?
How are changes to the underlying asset recorded?
What happens when an asset or investment arrangement changes?

These decisions influence the data model, application logic, and contract design.

2. Design the Tokenization Workflow

A conceptual workflow may include asset onboarding, verification, token issuance, investor participation, and transaction record management.

Each step needs defined permissions, validation rules, error handling, and auditability. The actual workflow depends on the project's legal and technical requirements.

3. Treat Blockchain as One Layer

Blockchain can provide transaction records and programmable rules, where appropriate. The broader application may also require conventional backend services, interfaces, data storage, monitoring, and identity or access-management controls.

The architecture should reflect actual business requirements instead of introducing blockchain components without a clear purpose.

4. Account for Security and Compliance

Developers should consider contract testing, authorization, secure key management, application vulnerabilities, and transaction monitoring.

The team must also establish how digital tokens relate to enforceable investment rights. Technical implementation alone cannot establish legal ownership or guarantee compliance.

A Real-World Project to Explore

SoluLab's Tokenized Clean Energy Investment Platform case study offers a relevant project reference for developers interested in the intersection of clean energy investment and tokenization.

Review the original project details before drawing conclusions about its specific blockchain, architecture, or implemented features.

Final Takeaway

The most important development decision is not simply which token standard to use. It is defining the asset model, investment rights, workflow, and security requirements first.

A well-designed tokenization platform connects these elements into a coherent system rather than treating token issuance as the entire solution.

Context Propagation Across gRPC and REST: One Broken Hop Splits Every Trace

Freddie Thompson — Mon, 05 Oct 2026 07:50:06 +0000

Your trace shows api-gateway calling orders, then stops. orders definitely called inventory, but inventory appears as a separate, parentless trace.

The context did not propagate. It happens at four specific boundaries, and each needs a different fix.

Boundary 1: protocol translation

W3C Trace Context travels as a traceparent HTTP header. gRPC has metadata, not headers, and the mapping is not automatic unless an instrumented interceptor does it.

from __future__ import annotations

import grpc
from opentelemetry import context as otel_context
from opentelemetry import propagate, trace
from opentelemetry.trace import SpanKind


class _MetadataSetter:
    def set(self, carrier: list, key: str, value: str) -> None:
        # gRPC metadata keys MUST be lowercase. A capitalised key is
        # silently dropped by some implementations and raises in others,
        # which is a fun way to lose context on one language's client only.
        carrier.append((key.lower(), value))


class _MetadataGetter:
    def get(self, carrier, key: str):
        if carrier is None:
            return None
        vals = [v for k, v in carrier if k.lower() == key.lower()]
        return vals or None

    def keys(self, carrier):
        return [k for k, _ in (carrier or [])]


_SETTER, _GETTER = _MetadataSetter(), _MetadataGetter()
_tracer = trace.get_tracer(__name__)


class TracingClientInterceptor(grpc.UnaryUnaryClientInterceptor):
    def intercept_unary_unary(self, continuation, client_call_details, request):
        with _tracer.start_as_current_span(
            client_call_details.method, kind=SpanKind.CLIENT
        ) as span:
            metadata = list(client_call_details.metadata or [])
            propagate.inject(metadata, setter=_SETTER)

            new_details = client_call_details._replace(metadata=metadata)
            response = continuation(new_details, request)

            code = getattr(response, "code", lambda: None)()
            if code not in (None, grpc.StatusCode.OK):
                span.set_attribute("rpc.grpc.status_code", str(code))

            return response


class TracingServerInterceptor(grpc.ServerInterceptor):
    def intercept_service(self, continuation, handler_call_details):
        ctx = propagate.extract(
            list(handler_call_details.invocation_metadata or []),
            getter=_GETTER
        )

        token = otel_context.attach(ctx)

        try:
            return continuation(handler_call_details)
        finally:
            # Detaching is not optional. A leaked context attaches itself to
            # whatever request the worker thread handles next, and you get
            # spans from request B parented under request A.
            otel_context.detach(token)

The lowercase-metadata rule is worth internalising. It fails asymmetrically — a Go client talking to a Python server may work while the reverse does not — which makes it look like a language bug rather than a casing bug.

Boundary 2: message queues

A trace that crosses Kafka or SQS has a producer and a consumer that may run hours apart. Parent-child is the wrong relationship; use a link.

from opentelemetry.trace import Link, NonRecordingSpan, SpanContext


def produce(topic: str, payload: bytes, producer) -> None:
    with _tracer.start_as_current_span(
        f"publish {topic}",
        kind=SpanKind.PRODUCER
    ):
        headers: list = []
        propagate.inject(headers, setter=_SETTER)
        producer.send(topic, value=payload, headers=headers)


def consume(msg) -> None:
    parent_ctx = propagate.extract(
        list(msg.headers or []),
        getter=_GETTER
    )

    parent_span = trace.get_current_span(parent_ctx)

    # A link, not a parent. The consumer may run long after the producer's
    # trace ended; making it a child produces traces with multi-hour gaps
    # that break every latency percentile you compute from span duration.
    links = []

    sc = parent_span.get_span_context()

    if sc.is_valid:
        links.append(Link(sc))

    with _tracer.start_as_current_span(
        f"process {msg.topic}",
        kind=SpanKind.CONSUMER,
        links=links
    ) as span:
        span.set_attribute("messaging.kafka.partition", msg.partition)
        span.set_attribute("messaging.kafka.offset", msg.offset)
        handle(msg)

Boundary 3: thread pools and async

Context is stored in a context-local. Hand work to a thread pool and the new thread has a fresh, empty one.

import contextvars
from concurrent.futures import ThreadPoolExecutor
from functools import partial


class ContextPreservingExecutor(ThreadPoolExecutor):
    """
    Copies the caller's context into the worker thread.

    Without this, every span produced inside a pool task is a new root and
    your trace loses everything that happens in a background worker - which
    is usually the slow part you were trying to see.
    """

    def submit(self, fn, /, *args, **kwargs):
        ctx = contextvars.copy_context()
        return super().submit(
            partial(ctx.run, fn),
            *args,
            **kwargs
        )

asyncio.create_task copies context automatically. loop.run_in_executor and raw Thread do not. Neither does most third-party library code that manages its own pool, which is why a trace can break inside a database driver you did not write.

Boundary 4: sampling decisions

If service A samples a trace out and service B makes its own independent decision, you get a child span with no parent. Always use ParentBased sampling so the root’s decision is authoritative downstream.

Finding the broken hop

Do not hunt for these manually. Query your tracing backend for spans with a parent ID that does not resolve:

def find_orphans(spans: list[dict]) -> dict[str, int]:
    """
    Group orphaned spans by service. The service that shows up is the
    one RECEIVING broken context - so the bug is in its caller, or in its
    own extract path. Run this weekly; it catches a regression the day a
    new service ships without the interceptor.
    """
    known = {s["span_id"] for s in spans}
    orphans: dict[str, int] = {}

    for s in spans:
        parent = s.get("parent_span_id")

        if parent and parent not in known:
            svc = s.get("service_name", "unknown")
            orphans[svc] = orphans.get(svc, 0) + 1

    return dict(
        sorted(
            orphans.items(),
            key=lambda kv: -kv[1]
        )
    )

Caveat: a parent can be legitimately missing because it was sampled out or is still in flight. Run this over a completed window and treat a rate as the signal, not individual cases. A service that jumps from 2% orphans to 60% after a deploy has a propagation bug; one that sits steadily at 3% probably has sampling.

Propagate more than trace context

traceparent gets you the trace. Baggage carries application context — tenant ID, feature flag cohort, request priority — to every downstream service without threading it through every function signature.

Two warnings. Baggage travels in headers to every hop, including third parties, so it must never contain anything sensitive. And it costs bytes on every call; a few small keys are fine, a serialised user object is not.

We build distributed backend systems at SoluLab — more on our custom software development work.

Multi-Tenant Database Isolation: RLS, Schema-per-Tenant, and the Migration Wall

Freddie Thompson — Mon, 05 Oct 2026 07:47:25 +0000

Three ways to keep tenants apart in one Postgres cluster:

Discriminator column — tenant_id on every table, filtered in application code.
Row-level security — tenant_id plus RLS policies the database enforces.
Schema-per-tenant — each tenant gets its own set of tables.

Option 1 is not isolation. It is a convention, and it fails the first time someone writes a query without the filter. That is not hypothetical — it is one of the most common sources of cross-tenant data exposure, and it happens in a reporting endpoint or an admin tool rather than the main code path, which is why review misses it.

The real choice is between 2 and 3, and it comes down to a trade most teams discover too late: RLS scales with tenant count, schema-per-tenant scales with tenant size.

Row-level security, done correctly

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;

ALTER TABLE invoices FORCE ROW LEVEL SECURITY;
-- applies to the table owner too

CREATE POLICY tenant_isolation ON invoices
    USING (tenant_id = current_setting('app.tenant_id', true)::uuid)
    WITH CHECK (tenant_id = current_setting('app.tenant_id', true)::uuid);

Three things in there that are load-bearing:

FORCE ROW LEVEL SECURITY. Without it, the table owner bypasses the policy entirely. If your application connects as the owner — which it does by default in most setups — RLS is enabled and doing absolutely nothing. This is the single most common way an RLS deployment provides no protection while appearing to.

WITH CHECK as well as USING. USING filters reads. Without WITH CHECK, a tenant can insert a row with someone else’s tenant_id. You get read isolation and write contamination.

The true second argument to current_setting. Makes it return NULL instead of erroring when unset. That sounds convenient and is the trap — tenant_id = NULL is NULL, which is not true, so the policy denies everything. Failing closed is right; just know that an unset variable produces “no rows” rather than an error, which is a confusing thing to debug.

And the part that actually enforces it:

from contextlib import contextmanager


@contextmanager
def tenant_scope(conn, tenant_id: str):
    """
    SET LOCAL is scoped to the transaction and cannot leak into the next
    checkout from the pool. Plain SET persists on the connection - with
    PgBouncer in transaction mode that means the next tenant inherits it,
    which is a cross-tenant read with no bug in your query.
    """
    with conn.transaction():
        conn.execute("SET LOCAL app.tenant_id = %s", (tenant_id,))
        yield conn


# The role matters as much as the policy.
# CREATE ROLE app_user NOLOGIN;
# GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_user;
# -- app_user is NOT the table owner, so FORCE is not the only thing
# -- standing between you and a full-table read.

SET LOCAL versus SET is the whole connection-pooling story. Get it wrong and the failure is intermittent, load-dependent, and cross-tenant.

Where each model breaks

RLS breaks on the query planner. Policies are predicates, and they interact with indexes in ways that are not always obvious. A composite index on (tenant_id, created_at) is essential; without it the planner may filter after scanning. Worse, RLS can prevent certain plan shapes entirely, and a query that is fast for a small tenant becomes a sequential scan for a large one. Check EXPLAIN output as a tenant, not as a superuser, or you are reading a plan the application will never get.

Schema-per-tenant breaks on migrations. At 4,000 schemas, ALTER TABLE runs 4,000 times. A migration that takes 8 seconds per schema is nine hours, and it is not atomic — you will be half-migrated when something fails, with application code that has to tolerate both shapes. Add connection-level catalog bloat: Postgres’s system catalogs grow with schema count, and planning time degrades measurably in the thousands.

Both break on cross-tenant queries. Admin dashboards, usage billing, aggregate analytics. Under RLS you need a separate role that bypasses policies, which is a privileged path you must guard as carefully as the policies themselves. Under schema-per-tenant you need a UNION across thousands of schemas, which is not viable, so you end up with a separate analytics store and a pipeline to keep it current.

The decision, concretely

Use RLS when you have many small tenants (SaaS, self-serve), you need one migration to cover everyone, and no tenant demands physical separation. This is most products.

Use schema-per-tenant when tenants are few and large, they have genuinely different schemas (custom fields as real columns), or a contract requires demonstrable physical separation. Enterprise and regulated deployments.

Use a separate database or cluster when a tenant’s contract requires it, or when one tenant’s load would otherwise degrade everyone. This is the honest answer for a handful of large accounts and it is fine to run a hybrid: RLS for the long tail, dedicated clusters for the top ten. The routing layer that picks the connection is a small amount of code and it saves you from designing for the worst case everywhere.

Testing isolation, which is the part that gets skipped

Isolation is a security property, so test it like one — adversarially, in CI, on every build:

import pytest


@pytest.mark.parametrize("table", ALL_TENANT_TABLES)
def test_no_cross_tenant_read(db, table):
    seed(db, tenant="A", table=table, rows=3)
    seed(db, tenant="B", table=table, rows=5)

    with tenant_scope(db, tenant_id=TENANT_A) as conn:
        rows = conn.execute(
            f"SELECT tenant_id FROM {table}"
        ).fetchall()

    assert rows, f"{table}: tenant A should see its own rows"
    assert {r[0] for r in rows} == {TENANT_A}, (
        f"{table}: cross-tenant leak"
    )


@pytest.mark.parametrize("table", ALL_TENANT_TABLES)
def test_cannot_write_other_tenant(db, table):
    with tenant_scope(db, tenant_id=TENANT_A) as conn:
        with pytest.raises(Exception):
            conn.execute(
                f"INSERT INTO {table} (tenant_id, ...) VALUES (%s, ...)",
                (TENANT_B,),
            )


def test_every_tenant_table_has_rls(db):
    """
    Catches the new table someone added without a policy. This test is
    worth more than the other two combined, because the failure it prevents
    is the one that actually happens.
    """
    missing = db.execute("""
        SELECT c.relname FROM pg_class c
        JOIN pg_namespace n ON n.oid = c.relnamespace
        WHERE n.nspname = 'public' AND c.relkind = 'r'
          AND c.relname = ANY(%s)
          AND (NOT c.relrowsecurity OR NOT c.relforcerowsecurity)
    """, (ALL_TENANT_TABLES,)).fetchall()

    assert not missing, f"tables without forced RLS: {missing}"

That third test is the important one. RLS is opt-in per table, so the realistic failure is not a broken policy — it is a table shipped six months from now by someone who did not know the convention. A catalog assertion in CI catches it; code review does not.

We build multi-tenant platforms where this decision gets made early and lived with for years — the MedicGraph patient management case study is one of them.

Zero-Downtime Database Migration: The Dual-Write Window Is Where It Goes Wrong

Freddie Thompson — Mon, 05 Oct 2026 07:44:24 +0000

The standard playbook for moving a table to a new store is four phases: dual-write, backfill, verify, and cut over. Every phase has a failure mode that produces silent, permanent divergence, and the order the phases run in is load-bearing.

The one that catches most teams: if you backfill before dual-writing, you lose every row written during the backfill. The backfill reads a snapshot; concurrent writes land in the old store only; you finish, see matching row counts, and ship. The missing rows are the ones written during the window, which is exactly your busiest data.

Dual-write must be live before the backfill starts, not after.

Phase 1: Dual-Write, Old Store Authoritative

from __future__ import annotations

import logging
import time
from dataclasses import dataclass
from enum import Enum
from typing import Any, Protocol

log = logging.getLogger(__name__)


class Phase(str, Enum):
    OLD_ONLY = "old_only"
    DUAL_OLD_AUTH = "dual_old_auth"  # write both, read old, compare async
    DUAL_NEW_AUTH = "dual_new_auth"  # write both, read new, old is fallback
    NEW_ONLY = "new_only"


class Store(Protocol):
    def write(self, key: str, value: dict) -> None: ...
    def read(self, key: str) -> dict | None: ...


@dataclass
class MigrationRouter:
    old: Store
    new: Store
    phase: Phase
    metrics: Any

    def write(self, key: str, value: dict) -> None:
        if self.phase is Phase.NEW_ONLY:
            self.new.write(key, value)
            return

        # Authoritative store first, and a failure there propagates.
        auth, shadow = (
            (self.old, self.new)
            if self.phase in (Phase.OLD_ONLY, Phase.DUAL_OLD_AUTH)
            else (self.new, self.old)
        )

        auth.write(key, value)

        if self.phase is Phase.OLD_ONLY:
            return

        # Shadow write failures must NOT fail the request. If they do, the
        # migration has coupled your availability to the store you are still
        # testing - and you will roll back a migration that was fine.
        try:
            shadow.write(key, value)
        except Exception:
            self.metrics.incr("migration.shadow_write_failed")
            log.exception("shadow write failed key=%s", key)

    def read(self, key: str) -> dict | None:
        if self.phase in (Phase.OLD_ONLY, Phase.DUAL_OLD_AUTH):
            return self.old.read(key)

        result = self.new.read(key)

        if result is None and self.phase is Phase.DUAL_NEW_AUTH:
            # Fallback during cutover. Every hit here is a real gap - alert on it.
            self.metrics.incr("migration.new_store_miss")
            return self.old.read(key)

        return result

Two decisions in there matter more than they look.

1. Swallow and Count Shadow-Write Failures

Shadow-write failures are swallowed and counted. The instinct is to fail loudly. Do not — you have just made your production availability depend on the store you are still validating.

Count them, alert on the rate, and let the reconciler repair them.

2. Alert on `new_store_miss`

new_store_miss gets an alert, not a log line. Once you flip to new-authoritative, every fallback read is a row that should be in the new store and is not.

If that counter is non-zero, you are not ready to cut over. This is the single metric that tells you so.

Phase 2: Backfill With a Stable Cursor

def backfill(
    router: MigrationRouter,
    source_scan,
    batch_size: int = 500
) -> int:
    """
    Copy historical rows.

    MUST run with dual-write already live.

    Uses a keyset cursor, not OFFSET:
    OFFSET pagination over a table taking concurrent writes skips
    and duplicates rows as pages shift underneath it.
    """
    copied = 0
    cursor = ""

    while True:
        rows = source_scan(after=cursor, limit=batch_size)

        if not rows:
            break

        for key, value in rows:
            existing = router.new.read(key)

            # Never overwrite. A live dual-write may have already put a NEWER
            # value here; the backfill is reading an older snapshot. Clobbering
            # it silently reverts a customer's write.
            if existing is None:
                router.new.write(key, value)
                copied += 1

        cursor = key

        # Crude throttle; prefer a token bucket keyed on replica lag
        # so the backfill self-limits.
        time.sleep(0.05)

    return copied

The if existing is None guard is the second classic corruption.

A backfill that blindly writes will overwrite fresher dual-written values with stale snapshot data, and the damage is invisible until a customer reports that an edit reverted.

For rows that genuinely need updating, you need a version or updated_at comparison rather than a presence check.

The source of truth for that comparison has to be a column the application actually maintains, not a database-generated timestamp that the backfill itself bumps.

Phase 3: Verify Continuously, Not Once

A one-shot comparison at the end tells you nothing about steady state. Sample continuously through the whole dual-write period:

def sample_and_compare(
    router: MigrationRouter,
    keys: list[str]
) -> dict:
    mismatches, missing, checked = [], [], 0

    for key in keys:
        old_val = router.old.read(key)
        new_val = router.new.read(key)

        checked += 1

        if new_val is None and old_val is not None:
            missing.append(key)
        elif old_val != new_val:
            mismatches.append((key, old_val, new_val))

    return {
        "checked": checked,
        "missing": len(missing),
        "mismatched": len(mismatches),
        "samples": mismatches[:5],
    }

Expect a non-zero mismatch rate from timing alone — you read the two stores microseconds apart, and a write can land between those reads.

Re-check a mismatch after a short delay before counting it:

A mismatch that persists is real.
A mismatch that resolves is replication lag.

Failing to make that distinction produces an alert channel everyone learns to ignore, which is worse than no alerting.

Phase 4: Cut Over and Keep the Way Back

Flip:

DUAL_OLD_AUTH → DUAL_NEW_AUTH

Do this behind a runtime flag, not a deploy.

You want the rollback to be a configuration change measured in seconds because the thing you are protecting against is a failure that shows up under production load and could not be reproduced in staging.

Keep dual-write running for at least one full business cycle after the flip — a week, a month-end, or whatever your longest periodic job is.

The rollback path only exists while the old store is still current.

The moment you stop writing to it, NEW_ONLY is irreversible. That step should therefore be a separate, deliberate, scheduled change rather than the tail end of the cutover.

What Makes This Genuinely Hard

Transactions That Span Both Stores

If a single logical operation writes to two tables and only one has migrated, you have lost atomicity for the duration of the migration.

There is no clean fix.

The options are to:

Migrate the whole transactional unit together.
Accept a reconciliation job for the migration window.

Pick deliberately rather than discovering the problem mid-migration.

Schema Changes During the Migration

Someone will want to add a column while you are three weeks into a dual-write.

The migration router has to tolerate the change on both sides. That means:

The new store's schema has to remain a superset.
The write path has to tolerate unknown fields.

Reads That Are Not Key-Value

Everything above assumes point reads.

Range queries, aggregates, and joins do not dual-read cleanly.

A COUNT(*) disagreeing between stores during a backfill is expected, not necessarily a bug. Any verification built on aggregates will therefore produce noise for the entire migration.

Final Takeaway

A zero-downtime database migration is not simply a matter of copying data and switching traffic.

The dual-write window is the critical safety mechanism. Start dual-writing before the backfill, protect newer writes from stale backfill data, verify continuously, monitor fallback reads, and keep the old store writable long enough to preserve a real rollback path.

We do data platform and legacy modernization work at SoluLab — the InfuseNet case study covers one of these migrations end to end.

DEV Community: Freddie Thompson

Building AI-Assisted Tax Document Workflows: Engineering Considerations for Tokenization and Evidence

Blockchain in Banking: What Developers Should Evaluate Before Building

Building a Multi-Chain DEX: What Blockchain Developers Should Consider

blockchain #web3 #defi #programming

Building Trust into Digital Document Workflows with Blockchain

GenAI + Tokenized RWAs: What Developers Should Consider Before Building

Start with the Data Model

Define the AI Component's Responsibility

Think About Security Early

Evaluate the Actual Use Case

A Case Study to Explore

Final Thought

Crypto Launchpad Development: Planning the Architecture Before Writing Code

Step 1: Define the Requirements

Step 2: Separate Application Concerns

Step 3: Design for Failure

Step 4: Make Security Part of Development

Learning From a Real Project

Final Takeaway

Gold Tokenization: What Blockchain Developers Should Consider Before Building

1. Define the Asset Model First

2. Establish the Connection Between Physical and Digital Records

3. Design Token Controls Around Real Requirements

4. Treat Security and Operations as Part of the System

5. Review the Project Context

Final Takeaway

Building a Tokenized Clean Energy Investment Platform: What Developers Should Consider

1. Define the Asset Model First

2. Design the Tokenization Workflow

3. Treat Blockchain as One Layer

4. Account for Security and Compliance

A Real-World Project to Explore

Final Takeaway

Context Propagation Across gRPC and REST: One Broken Hop Splits Every Trace

Boundary 1: protocol translation

Boundary 2: message queues

Boundary 3: thread pools and async

Boundary 4: sampling decisions

Finding the broken hop

Propagate more than trace context

Multi-Tenant Database Isolation: RLS, Schema-per-Tenant, and the Migration Wall

Row-level security, done correctly

Where each model breaks

The decision, concretely

Testing isolation, which is the part that gets skipped

Zero-Downtime Database Migration: The Dual-Write Window Is Where It Goes Wrong

Phase 1: Dual-Write, Old Store Authoritative

1. Swallow and Count Shadow-Write Failures

2. Alert on new_store_miss

Phase 2: Backfill With a Stable Cursor

Phase 3: Verify Continuously, Not Once

Phase 4: Cut Over and Keep the Way Back

What Makes This Genuinely Hard

Transactions That Span Both Stores

Schema Changes During the Migration

Reads That Are Not Key-Value

Final Takeaway

2. Alert on `new_store_miss`