DEV Community: Andrii Mahalias

Graceful Multi-Tenant Lifecycle: Managing Isolated Containers, Drains, and Deferred Purges in Python

Andrii Mahalias — Mon, 17 Aug 2026 20:20:34 +0000

When building a multi-tenant SaaS control plane that provisions dedicated infrastructure per client - such as isolated Docker containers running alongside dynamic PostgreSQL databases - managing the container lifecycle is rarely as simple as issuing docker stop or docker rm.

If an orchestrator tears down an infrastructure instance while a tenant is mid-transaction, executing background workers, or handling state initialization, you face:

Orphaned Database State: Dropping tenant databases while active connection pools remain open.
Corrupted In-Flight Payloads: Force-killing containers (SIGKILL) during active job execution.
Blocked Event Loops: Freezing asynchronous FastAPI orchestrators with synchronous Docker SDK calls.
Resource Leaks: Leaving orphaned directories, allocated ports, or half-provisioned containers in limbo after a failure.

Let's examine how to structure a robust Provisioning, Offloading, and Deprovisioning Architecture in Python using asyncio, Docker, PostgreSQL, and distributed Redis locks.

1. The Core Lifecycle Pitfall: force=True vs. Graceful Drain
In Python’s docker-py SDK, it is tempting to wipe non-responsive or deprovisioned containers using:

# ❌ DANGEROUS: Immediately sends SIGKILL (Signal 9)
container.remove(v=True, force=True)

Passing force=True bypasses SIGTERM (Signal 15) and immediately kills the container process. If your tenant container is processing a queue or executing database transactions, operations cut off mid-flight.

The Correct Graceful Stop Pattern
Always issue an explicit stop timeout first to allow the application inside the container to flush buffers and close connection pools before executing container removal:

def stop_container(self, container_name: str, remove: bool = True, timeout: int = 30):
    if not self.client:
        return
    try:
        container = self.client.containers.get(container_name)

        # 1. Send SIGTERM and grant a 30s grace window
        container.stop(timeout=timeout)
        logger.info(f"Container {container_name} stopped gracefully.")

        # 2. Clean up container and associated volumes safely
        if remove:
            container.remove(v=True)
            logger.info(f"Container {container_name} and volumes purged.")

    except docker.errors.NotFound:
        logger.warning(f"Container {container_name} not found, skipping.")
    except Exception as e:
        logger.error(f"Error stopping/removing {container_name}: {e}")

2. Dynamic Orchestration & Distributed Locks
When dynamically allocating infrastructure assets (e.g., picking an unused host port, generating distinct DB credentials, and writing docker-compose.yml configs), concurrent HTTP requests create severe race conditions.

To prevent two worker routines from claiming the same network port or overlapping tenant resources, wrap key allocation steps inside Distributed Redis Locks:

async def provision(self, tenant: Tenant, db: AsyncSession):
    # Tenant-level lock prevents duplicate concurrent provisioning
    async with OrchestrationLock(settings.REDIS_URL) as locker:
        lock_name = locker.tenant_provision_lock(tenant.id)
        if not await locker.lock.acquire(lock_name):
            raise RuntimeError(f"Provisioning already in progress for tenant {tenant.id}")

        try:
            return await self._execute_provisioning_flow(tenant, db)
        except Exception as e:
            logger.error(f"Provisioning failed for {tenant.subdomain}: {e}")
            await self._rollback()
            raise
        finally:
            await locker.lock.release(lock_name)

3. Atomic Provisioning Flow via Reversible Rollback Stacks
Provisioning multi-tenant infrastructure involves multiple distinct stages across different subsystems:

[1. Allocate DB] ➔ [2. Render Configs] ➔ [3. Spin Docker] ➔ [4. Healthcheck] ➔ [5. Seed State]

If Step 4 fails (e.g., the container fails health checks after 60 seconds), steps 1, 2, and 3 must unwind completely in reverse order so no orphaned database or container remains on the host.

An elegant design pattern to solve this is a LIFO (Last-In, First-Out) Rollback Stack:

class ProvisioningOrchestrator:
    def __init__(self):
        self._rollback_stack: List[Callable[[], Coroutine[Any, Any, None]]] = []

    async def _execute_provisioning_flow(self, tenant: Tenant, db: AsyncSession):
        self._rollback_stack = []

        # Step 1: Create isolated PostgreSQL tenant DB
        db_success = await self.postgres.create_tenant_db(db_name, db_user, db_pass)
        if not db_success:
            raise RuntimeError("Database creation failed")

        # Register rollback handler for Step 1
        self._rollback_stack.append(
            lambda: self.postgres.drop_tenant_db(db_name, db_user)
        )

        # Step 2: Spin up container using CLI subprocess
        await self.docker.start_instance(tenant_dir)

        # Register rollback handler for Step 2
        self._rollback_stack.append(
            lambda: self._async_wrapper(self.docker.stop_container, container_name)
        )

        # Step 3: Wait for Healthcheck
        if not await self.wait_for_health(internal_url):
            raise RuntimeError("Instance failed healthcheck")

    async def _rollback(self):
        logger.warning(f"Initiating rollback for {len(self._rollback_stack)} steps...")
        # Execute registered rollbacks in reverse order
        for rollback_task in reversed(self._rollback_stack):
            try:
                await rollback_task()
            except Exception as e:
                logger.error(f"Rollback step failed: {e}")

4. Non-Blocking Async Orchestration with Synchronous SDKs
Python's standard Docker SDK (docker-py) is blocking and synchronous. Calling container.stop() or container.remove() directly inside an async def function blocks FastAPI’s main Event Loop, ruining response times for all other active tenants.

To safely bridge blocking synchronous SDK calls inside async flows, offload them to background threads via asyncio.to_thread:

async def _async_wrapper(self, sync_fn, *args, **kwargs):
    # Offloads blocking Docker SDK calls to a background worker thread
    return await asyncio.to_thread(sync_fn, *args, **kwargs)

5. Two-Tier Lifecycle: Compute Freeze vs. Hard Storage Purge
Keeping non-paying or canceled tenant containers running 24/7 wastes precious CPU and RAM. However, permanently deleting a client's database the second their subscription expires is disastrous for customer retention.

Implement a Two-Tier Retention Model driven by two key variables:

SUBSCRIPTION_FREEZE_DAYS = 7   # Soft Stop: Release RAM/CPU compute
SUBSCRIPTION_PURGE_DAYS = 365  # Hard Purge: Delete Database & Disk Files

[Active Tenant] ──(Subscription Lapses)──> [Suspended] ──(Day 7 Expiry)──> [Frozen (0 MB RAM)] ──(Day 365 Expiry)──> [Hard Purged]
                                            Container stopped/removed.       PostgreSQL DB dropped.
                                            PostgreSQL DB preserved.         Filesystem wiped.

Tier 1: Freezing Compute Resources (SUBSCRIPTION_FREEZE_DAYS = 7)
If a tenant cancels or fails payment, grant a 7-day grace period. Once hit, stop and remove the Docker container to free up host RAM. The PostgreSQL database and filesystem remain completely intact.

async def handle_frozen_tenants(db: AsyncSession):
    """Cron job: Offloads compute containers for tenants inactive > 7 days."""
    freeze_threshold = datetime.utcnow() - timedelta(days=settings.SUBSCRIPTION_FREEZE_DAYS)
    inactive_tenants = await tenant_crud.get_tenants_eligible_for_freeze(db, freeze_threshold)

    for tenant in inactive_tenants:
        instance = await listmonk_crud.get_instance_by_tenant(db, tenant.id)
        if instance and instance.container_name:
            # Stop and remove container, but preserve DB and files
            await asyncio.to_thread(orchestrator.docker.stop_container, instance.container_name, remove=True)
            await listmonk_crud.update_instance_status(db, instance.id, "frozen")
            logger.info(f"Tenant {tenant.subdomain} container frozen. Compute freed.")

If the client resubscribes on day 30, the control plane simply executes start_instance() over the existing database, restoring their workspace instantly.

Tier 2: Safe Deprovisioning Sequence (SUBSCRIPTION_PURGE_DAYS = 365)
If the tenant remains inactive for a full year, execute a permanent deprovision(). Execution order is critical: dismantle the architecture from the edge inward to prevent deadlocks or corrupted logs.

1. Remove Docker Container (Stop incoming traffic & queues)
       │
       ▼
2. Drop PostgreSQL DB (Clean storage once process handles detach)
       │
       ▼
3. Purge File Directory (Wipe local compose configs and assets)
       │
       ▼
4. Update Control Plane DB (Commit deactivated status)

async def deprovision(self, tenant: Tenant, db: AsyncSession) -> Dict[str, Any]:
    instance = await listmonk_crud.get_instance_by_tenant(db, tenant.id)
    if not instance:
        tenant.status = "deactivated"
        await db.commit()
        return {"status": "not_provisioned"}

    cleanup_errors = []

    # 1. Stop and remove container first
    if instance.container_name:
        try:
            await asyncio.to_thread(self.docker.stop_container, instance.container_name, remove=True)
        except Exception as e:
            cleanup_errors.append(f"docker_container: {e}")

    # 2. Drop the isolated PostgreSQL database
    if instance.database_name and instance.database_user:
        try:
            await self.postgres.drop_tenant_db(instance.database_name, instance.database_user)
        except Exception as e:
            cleanup_errors.append(f"postgres_db: {e}")

    # 3. Clean files off the disk
    tenant_dir = os.path.abspath(self.config_svc.get_tenant_dir(tenant.subdomain))
    try:
        if os.path.exists(tenant_dir):
            shutil.rmtree(tenant_dir)
    except Exception as e:
        cleanup_errors.append(f"disk_directory: {e}")

    # 4. Clean up relational metadata & commit
    await db.execute(delete(SMTPConfig).where(SMTPConfig.tenant_id == tenant.id))
    await listmonk_crud.update_instance_status(db, instance.id, "deleted")

    tenant.status = "deactivated"
    tenant.listmonk_api_url = None
    await db.commit()

    if cleanup_errors:
        logger.warning(f"Tenant {tenant.subdomain} deprovisioned with partial failures: {cleanup_errors}")
    else:
        logger.info(f"Tenant {tenant.subdomain} fully purged after {settings.SUBSCRIPTION_PURGE_DAYS} days.")

    return {"status": "deactivated"}

Summary Checklist for Multi-Tenant Resilience

Avoid force=True blindly: Grant container processes an explicit stop timeout before executing removal.
Rollback Stack Pattern: Use a LIFO stack to tear down partially created DBs/containers if health checks fail during setup.
Offload Sync SDKs: Wrap synchronous docker-py SDK calls in asyncio.to_thread inside async frameworks.
Distributed Locking: Secure dynamic port assignment and tenant provisioning steps with Redis locks.
Two-Tier Retention Strategy: Separate compute offloading (FREEZE_DAYS=7) from permanent storage wipes (PURGE_DAYS=365) to optimize server costs without killing customer win-back capability.

Why I Give Every User Their Own Database (And Why It’s Slowly Becoming My Biggest Headache)

Andrii Mahalias — Sun, 16 Aug 2026 19:04:12 +0000

When building QeakFlow — a privacy-first, GDPR-compliant email marketing SaaS — one architectural decision became surprisingly difficult: how much tenant isolation does a SaaS actually need?

The conventional answer is straightforward: one application, one database, and a tenant_id on every relevant table.

I chose a different approach. Each QeakFlow workspace runs in its own container with its own PostgreSQL database.

This gives us a much stronger isolation boundary, but it also introduces infrastructure costs and operational complexity that a conventional multi-tenant architecture avoids.

This is why I chose it, how the architecture works, and where the approach starts to hurt.

Why hard tenant isolation?
QeakFlow is an email marketing platform, which means every tenant can potentially store subscriber PII and its own SMTP credentials.

That made data isolation more important than it would be for many other SaaS products.

The architecture looks roughly like this:

There is no shared application database containing all subscriber data with a tenant_id column. Each tenant has its own isolated application instance and database.

A smaller blast radius for data-access bugs In a traditional multi-tenant database, every query needs to respect tenant boundaries:

SELECT *
FROM subscribers
WHERE tenant_id = :tenant_id;

That seems simple until you have complex joins, background jobs, exports, reporting queries, migrations, and administrative tooling. One missing tenant filter can potentially expose another customer's data.

Separate databases remove this particular class of application-level isolation bug. They don't eliminate every possible security problem, but they make accidental cross-tenant queries much harder.

Simpler tenant deletion
With separate databases, deleting a tenant's application data has a very clear boundary. Instead of finding every table containing that tenant's records, the tenant's database can eventually be removed entirely. That makes the data lifecycle easier to reason about.
Resource isolation
Email marketing can create surprisingly uneven workloads. One customer might import 100,000 contacts while another sends a small newsletter once a week.

With isolated containers, CPU and memory limits can be applied independently. For example:

Tenant A -> CPU: 0.5 | RAM: 256 MB
Tenant B -> CPU: 0.5 | RAM: 256 MB
Tenant C -> CPU: 0.5 | RAM: 256 MB

A large import from Tenant A doesn't automatically consume all resources available to Tenant B.

SMTP isolation QeakFlow uses a BYO-SMTP model. Each customer connects their own email delivery provider.

That means background workers can maintain SMTP connections, process bounces, send campaigns, and handle provider-specific events. If something goes wrong with one tenant's SMTP connection or worker, the failure is contained within that tenant's environment rather than taking down a shared worker responsible for everyone.

The infrastructure
The current architecture is deliberately simple.

The control plane manages tenants, billing, provisioning, and configuration. Redis is used as the orchestration backbone for things such as distributed locks, container state, and background coordination.

The actual tenant workloads run separately. The infrastructure currently uses:

Docker & cgroups (for resource boundaries)
PostgreSQL (isolated databases)
Redis (orchestration state)
Prometheus & cAdvisor (visibility into container health)
Ubuntu 24.04 LTS (bare-metal host environment)

The goal isn't to build a Kubernetes cluster for a handful of customers. The goal is to keep the isolation boundary while keeping the infrastructure understandable enough to operate as a small team.

The downside: memory
This is where the architecture becomes expensive.

Every isolated application instance has a baseline memory footprint. In our testing, approximately:

10 idle tenants → ~1.5 GB RAM
100 idle tenants → ~15 GB RAM

Those numbers are only a rough measurement from our current setup, not a universal Docker or Listmonk requirement. But the underlying problem is real: the cost of isolation scales with the number of tenants.

With a shared application instance, adding the 100th tenant doesn't require another application process. With dedicated containers, it does. At small scale this is acceptable. At larger scale, it becomes one of the main architectural constraints.

Database migrations become more complicated
A shared database makes schema migrations relatively straightforward:

1 Migration  --->  1 Shared Database

With isolated databases:

1 Migration  --->  Tenant A DB
             --->  Tenant B DB
             --->  Tenant C DB
             --->  Tenant D DB

Every tenant database needs to be upgraded. That means the provisioning and migration system needs to understand database versions and handle failures safely.

A migration that succeeds for 99 tenants but fails for tenant 100 isn't just a technical inconvenience. It becomes an operational state that needs to be tracked and recovered. This is one of the biggest trade-offs of the architecture.

Orchestration becomes part of your application
Once every tenant has its own container, you need to answer questions that don't exist in a simple shared application:

When should a container start?
What happens when it crashes?
How do you detect an unhealthy instance?
How do you update all instances?
Can an idle tenant be stopped?
How do you provision a new tenant safely?
What happens if provisioning succeeds halfway?
How do you recover from a failed migration?

At that point, you're no longer just building a SaaS application — you're also building a small orchestration layer.

For QeakFlow, Redis became particularly useful here. It provides coordination between provisioning workers and helps prevent two workers from modifying the same tenant at the same time.

Does separate infrastructure make the system more secure?
Not automatically. This distinction is important.

Dedicated databases eliminate some risks associated with shared multi-tenant databases, but they introduce other attack surfaces. You still have to secure:

The control plane API
Container networking
Database credentials
Provisioning logic
Secrets management
Reverse proxy configuration
Host-level access

A separate database doesn't magically make a SaaS GDPR-compliant. It simply gives you a stronger technical isolation boundary to build on.

Why not just use a shared database?
For many SaaS products, I think shared multi-tenancy is the right answer. It's cheaper, easier to operate, easier to migrate, and easier to monitor. Mature multi-tenant applications can provide strong isolation with the right testing, authorization layers, and database constraints.

If I were building a generic project-management SaaS, I probably wouldn't choose a dedicated database and container for every tenant.

Email marketing is a somewhat different case. The data includes subscriber information, campaigns, and potentially SMTP credentials. The workloads can also be unpredictable. That makes stronger isolation more valuable.

What I would do differently at larger scale
I don't think this architecture is the final answer for thousands of tenants. At some point, the operational overhead would probably outweigh the benefits.

A more scalable architecture could use different isolation levels depending on the tenant tier:

Small / Free tenants → Shared application infrastructure
Larger / Sensitive tenants → Dedicated database
Enterprise tenants → Dedicated application + database

In other words, tenant isolation doesn't have to be binary. You can choose the isolation boundary based on workload, sensitivity, and business requirements.

So, was it worth it?
At the current stage of QeakFlow, I think so. We're trading infrastructure efficiency for a simpler security boundary and stronger workload isolation.

The cost is real: more memory, more containers, more databases, more migrations, and more orchestration.

But the architecture also makes an important property of the system explicit: one tenant's data and workload should not casually become another tenant's problem.

For a few tenants, dedicated infrastructure is probably overkill. For highly sensitive workloads, it can be a reasonable trade. For thousands of small tenants, I'd expect a hybrid approach to win.

How are you handling tenant isolation in your SaaS? Did you stick with shared multi-tenancy, separate databases, dedicated containers, or some hybrid approach?

Andrii is building QeakFlow — a privacy-first email marketing platform with per-tenant isolation and BYO-SMTP. Early access at
qeakflow.com