DEV Community: Acel

How I Moved Real-Time Out of Django (and Made Everything Simpler)

Acel — Tue, 30 Jun 2026 08:44:58 +0000

Your users want live updates: an order flips to paid, a notification pops, a dashboard number ticks up without a refresh. The obvious answer in Django is Channels. I reached for it first too. But as our scale grew, the costs of keeping connections in-process pushed me to rethink the architecture. I let Django stop handling WebSockets entirely, and almost everything got simpler. Here's the pattern that replaced it.

The Problem

Django is a request/response framework. It processes a request, returns a response, and moves on. Real-time notifications — a dashboard ticking up, an order status flipping live — need a server that holds a connection open for every connected client. That is a fundamentally different job.

The standard answer in the Django ecosystem is Channels. It moves you to ASGI, gives you consumers, and works well, especially when your real-time needs are deep and bidirectional.

Ours weren't. We needed server→client push: notify a user when an order completed, or when a payment landed. The connections were mostly idle. Channels gave us that, but it came with costs that started hurting at scale.

The concrete problem was this: every open WebSocket connection consumed memory and a small slice of CPU time on our app servers. A thousand connected dashboards was fine. Ten thousand, across dozens of tenants, each holding a persistent socket. Suddenly, our workers were doing two very different jobs at once. Serving fast API responses and babysitting long-lived, mostly-quiet connections. Both suffered.

What made it brittle was the reconnect storm. Deploy a new build, restart a server, or hit a brief network partition, and every connected client disconnects and immediately reconnects. All at once. That thundering herd of TLS handshakes, authentication, and subscription requests would spike CPU to 100% on every app server as it tried to spin up thousands of ASGI consumer instances simultaneously, occasionally knocking them over entirely before the retry backoff kicked in.

There's also the smaller but telling friction of auth. Authenticating the WebSocket handshake means reaching your existing auth (DRF/JWT) from inside an ASGI scope. I ended up writing middleware that reconstructed a DRF Request out of the raw scope just to reuse the auth I already had. It worked, but it was a sign: I was bending the framework to do something it wasn't built for.

The turning point was asking: why is my web framework in the business of holding connections at all?

For a fire-and-forget server→client push, it shouldn't be. There are servers built specifically to hold millions of open connections and do nothing else. The only two things your backend needs to do are say who's allowed to listen, and publish events.

The Solution

Put a dedicated pub/sub gateway in front of your app. I used Centrifugo, a standalone real-time server. Clients connect to it, not to Django. Django goes back to being a plain, boring WSGI/sync app and plays two small roles:

Issue a short-lived connection token that embeds which channels this user may subscribe to.
Publish events to the gateway over its HTTP API.

The gateway holds the connections. Django holds nothing.

Step 1: Issue a token that carries its own permissions

When a client wants to connect, it asks your backend (over a normal authenticated request) for a connection token. The token is a JWT whose claims list the exact channels the user may subscribe to. The gateway verifies the signature and authorizes subscriptions without a single database lookup:

# realtime/auth.py
import time
import jwt
from django.conf import settings


class RealtimeTokenService:
    """Issues short-lived JWTs for connecting to the realtime gateway."""

    def generate_connection_token(self, user) -> str:
        tenant_id = user.tenant_id

        # Channel grants are baked into the token, so the gateway authorizes
        # subscriptions with zero database lookups on connect.
        channels = [
            f"user_{user.id}",
            f"critical:user_{user.id}",
            f"tenant_{tenant_id}",
            f"critical:tenant_{tenant_id}",
        ]

        claims = {
            "sub": str(user.id),
            "exp": int(time.time()) + settings.REALTIME_TOKEN_TTL_SECONDS,
            "info": {"name": user.get_full_name(), "email": user.email},
            "channels": channels,
        }
        return jwt.encode(claims, settings.REALTIME_HMAC_SECRET, algorithm="HS256")

This is the key idea: token-as-permission. The channel names encode your authorization model. A user gets their own user_{id} channel and their tenant's tenant_{id} channel, and nothing else. Multi-tenant isolation falls out of the channel naming: a user simply has no grant for another tenant's channel, so the gateway will refuse the subscription.

Step 2: Hand the token out behind your normal auth

The endpoint that mints the token is an ordinary authenticated view. Your existing auth stack protects it; no ASGI gymnastics:

# realtime/views.py
class RealtimeTokenView(APIView):
    permission_classes = [IsAuthenticated]

    def post(self, request):
        token = RealtimeTokenService().generate_connection_token(request.user)
        return Response(
            {"token": token, "socket_url": settings.REALTIME_SOCKET_URL}
        )

The client takes {token, socket_url}, opens a WebSocket to the gateway, and subscribes to its channels. Django is now out of the connection entirely.

Step 3: Publish events over HTTP

To push an update, your backend POSTs to the gateway's publish API. No connection state, no sockets: just an HTTP call:

# realtime/publisher.py
import requests
from django.conf import settings


def push_event(channel, event_type, payload, *, recoverable=False, timeout=10.0):
    # The "critical:" namespace is configured on the gateway with message
    # history + recovery, so clients can catch up after a reconnect.
    final_channel = f"critical:{channel}" if recoverable else channel

    resp = requests.post(
        f"{settings.REALTIME_API_URL}/publish",
        json={
            "channel": final_channel,
            "data": {"type": event_type, "payload": payload},
        },
        headers={"X-API-Key": settings.REALTIME_API_KEY},
        timeout=timeout,
    )
    resp.raise_for_status()

Notice the critical: channel prefix. On the gateway, that namespace is configured to keep a short history so a client that briefly drops can recover missed messages on reconnect:

{
  "namespaces": [
    { "name": "critical", "history_size": 50, "history_ttl": "120s", "force_recovery": true }
  ]
}

Ephemeral events (a "user is typing" blip) publish to the bare channel; events you can't afford to lose (a completed order) publish to critical: and survive a reconnect.

Step 4: Make publishing reliable, and don't block the request

Publishing is a network call to a separate service. You do not want it on the request's critical path, and you want it to survive a transient blip. Wrap it in a Celery task with retries and backoff:

# realtime/tasks.py
import logging
from requests.exceptions import RequestException, HTTPError
from celery import shared_task
from .publisher import push_event

logger = logging.getLogger(__name__)


@shared_task(bind=True, max_retries=3, default_retry_delay=5, retry_backoff=True)
def publish_realtime_event(self, channel, event_type, payload, recoverable=False):
    try:
        push_event(channel, event_type, payload, recoverable=recoverable)
    except HTTPError as exc:
        # A 4xx error (bad API key, malformed payload) is a deterministic
        # failure that retries can't fix. Log it and drop it.
        # A 5xx error might be transient, so retry it.
        if 400 <= exc.response.status_code < 500:
            logger.error("Deterministic 4xx error on %s, dropping.", channel)
            return
        raise self.retry(exc=exc)
    except RequestException as exc:
        # Pure network timeouts or connection errors — retry with backoff.
        raise self.retry(exc=exc)
    except Exception:
        # A serialization bug won't fix itself on a retry. Log and drop it,
        # rather than burning three attempts on a guaranteed failure.
        # By not re-raising, Celery marks the task as successful and removes
        # it from the queue — no poison pill looping forever.
        logger.exception("Dropping unrecoverable realtime event on %s", channel)

That except split matters: retry the transient, drop the deterministic. A flaky network deserves another try; a payload that can't be JSON-encoded will fail identically three times in a row, so retrying it is just noise.

Firing an event from anywhere in your app is now a one-liner:

publish_realtime_event.delay(
    channel=f"tenant_{order.tenant_id}",
    event_type="order.completed",
    payload={"order_id": order.id, "total": str(order.total)},
    recoverable=True,
)

Why This Works

Your web tier goes back to being stateless. Django stays on WSGI. No ASGI migration, no async server, no sockets held in your workers. You can scale, restart, and deploy the web tier like any other stateless service. The open connections live on the gateway and aren't disturbed.

Connections scale independently. Holding a lot of idle connections is a specialized job. A purpose-built gateway does it on its own box, tuned for that, while your app servers stay sized for request throughput.

Auth has zero connect-time cost. Because the token carries its channel grants, the gateway never calls back into your database to authorize a subscription. Ten thousand clients reconnecting after a blip is ten thousand signature checks, not ten thousand DB queries.

Multi-tenancy is just naming. Tenant isolation isn't a separate access-control layer. It's the channel names in the token. There's no code path by which a user can subscribe to a channel they weren't granted.

Delivery is fire-and-forget but durable. Publishing happens off the request path via Celery, retries on transient failure, and the critical: namespace lets clients recover what they missed across a reconnect.

Design Decision: The Token Is a Cache, So It Can Go Stale

Baking permissions into the token is what buys you DB-free connects. But a token is a snapshot. If a user's access changes (you remove them from a tenant, say) and their token still has 24 hours to live, the gateway will keep honoring the old grants until it expires. You've traded revocation freshness for connect-time performance.

That's usually the right trade for a notification stream, but be deliberate about it:

Keep the TTL short and have the client transparently re-fetch a token. Shorter tokens = smaller stale window.
For true revocation, the gateway can be told to force-disconnect a user or refuse channels server-side. Reach for that only when a stream is sensitive enough to need it.

One more operational gotcha: a single gateway is tempting to share across environments, but tenant_42 in staging and tenant_42 in production are the same channel name: cross-talk waiting to happen. If you must share one broker, prefix channels with the environment (prod_tenant_42, staging_tenant_42) so the namespaces can't collide.

And to be honest about the path here: I shipped the Channels version first. It worked. But the friction of running an async server and bridging my existing auth into the ASGI scope is exactly what made the gateway approach worth it. If your real-time needs are deep and bidirectional (collaborative editing, presence), in-process Channels may genuinely fit better. For "push updates to subscribed clients," letting something else hold the sockets is the simpler system.

The Result

Adding a live update anywhere in the app is now one .delay() call. The web tier never learned what a WebSocket is. Connections, fan-out, history, and recovery are the gateway's problem; authorization is a signed list of channel names; delivery is a retrying background task. Django went back to doing the thing it's good at. And the real-time problem moved to a server built for it.

How are you doing real-time in your Django apps — in-process with Channels, or have you pushed connections out to a gateway? I'd love to hear what's worked for you in the comments.

¡Hasta luego!

How I Built a "Set It and Forget It" Sync System with Django Signals

Acel — Tue, 16 Jun 2026 06:00:00 +0000

Change a product's price anywhere in your app, and it instantly syncs to a third-party marketplace. No manual triggers, no polling, no fragile save() overrides. Here's the signal pattern that powers it.

The Problem

In our app, a product's name or price can change from eight different places. The edit page. Bulk import. The variant editor. A pricing rule engine. An API endpoint that processes webhooks from suppliers. Every few weeks, someone adds a new feature and creates yet another code path that mutates a product.

We needed every one of those changes, no matter where they came from, to sync to an external marketplace. The naive approach would be to add a sync call to each code path. That's eight places to maintain (and counting), eight chances to forget, and eight places that break if the sync API changes.

Django's post_save signals solve the discovery problem: hook into a model's save event and you catch every change, from every code path, in one place. Signals get a bad rap — fairly, they make control flow hard to trace. But when you need to react to changes from everywhere without touching anywhere, this specific discovery problem is exactly what they were designed for.

There's a catch, though. If ten prices change in a single request, say, a bulk import, a naive signal handler fires ten times. That's ten API calls in rapid succession. At best, you are wasting resources. At worst, the external API rate-limits you.

We needed signals to detect changes, but we needed to batch them.

The Solution

Here's the three-part pattern:

A signal handler that collects change references instead of acting on them immediately.
A per-thread set that deduplicates for free — adding the same product twice does nothing.
A flush callback deferred to transaction.on_commit that processes everything once the database transaction lands.

Step 1: Register the signal

In your app's apps.py, connect post_save to the model that holds pricing:

# shopping/apps.py
from django.apps import AppConfig

class ShoppingAppConfig(AppConfig):
    name = "shopping"

    def ready(self):
        from django.db.models.signals import post_save
        from shopping.models import PriceRecord
        from shopping.signals import on_price_change

        post_save.connect(
            on_price_change,
            sender=PriceRecord,
            dispatch_uid="shopping_price_change",
        )

dispatch_uid prevents duplicate connections if ready() runs twice, a common gotcha during development with auto-reload.

Step 2: Collect, don't act

The signal handler doesn't call an API. It just adds a reference to a set and registers a flush callback:

# shopping/signals.py
import threading
from django.db import transaction

_local = threading.local()


def _get_pending():
    if not hasattr(_local, "pending"):
        _local.pending = set()
    return _local.pending


def on_price_change(sender, instance, **kwargs):
    # Record just enough to look up the product later
    _get_pending().add(instance.object_id)
    transaction.on_commit(flush_changes)

That's it. Four lines of logic. The handler doesn't care how many prices changed or where the change came from. It just records what changed and defers action to the flush.

Notice that we use threading.local() instead of a standard module-level variable. This ensures that each thread gets its own isolated storage.

Why transaction.on_commit? If the transaction rolls back, the price change never happened, so the flush callback is discarded. You never sync data that wasn't committed.

Step 3: Flush once per transaction

When the transaction lands, the flush handler fires. It snapshots the set, clears it immediately, and processes everything in one batch:

def flush_changes():
    pending = _get_pending()
    refs = pending.copy()
    pending.clear()

    if not refs:
        return

    from shopping.services import SyncService
    SyncService.handle_price_changes(refs)

The key detail: we copy the set before clearing it. If clearing happened after processing, and processing raised an exception, stale refs would linger into future transactions. Snapshot-first is defensive.

This clear() is also what keeps requests perfectly isolated. Once a transaction commits and the flush runs, the set is empty and ready for the next request.

Step 4: Resolve and dispatch

The service layer resolves the raw references into business entities, groups them by destination, and dispatches one task per group. The grouping is the important part. One API call per store, regardless of how many products changed:

# shopping/services.py
from collections import defaultdict

class SyncService:

    @classmethod
    def _resolve_to_products(cls, refs: set[int]):
        """
        Resolve product ID references to actual Product instances
        with their Store relationship.  Fetches everything in ONE
        query using filter(id__in=...), silently ignoring stale
        IDs that no longer exist.
        """
        products = Product.objects.filter(
            id__in=refs
        ).select_related("store")

        resolved = []
        for product in products:
            resolved.append((product, product.store_id))
        return resolved

    @classmethod
    def handle_price_changes(cls, refs: set[int]):
        resolved = cls._resolve_to_products(refs)

        # Group changes by the store they belong to
        by_store = defaultdict(list)
        for product, store_id in resolved:
            by_store[store_id].append(product)

        # One API call per store, regardless of how many products changed
        for store_id, products in by_store.items():
            sync_to_marketplace.delay(store_id=store_id, products=products)

The sync_to_marketplace task is a Celery task that calls the external API. It's configured with retries and backoff for transient failures:

@shared_task(bind=True, max_retries=3, default_retry_delay=60)
def sync_to_marketplace(self, store_id, products):
    try:
        provider.bulk_update(store_id, products)
    except Exception as exc:
        raise self.retry(exc=exc)

Why This Works

Deduplication is free. A set naturally deduplicates, adding the same object_id twice is a no-op. If a price changes twice in the same request (say, a pricing rule recalculates it), the flush handler sees it once. And when it fires, it reads the latest price from the database. The most recent value always wins.

Transaction safety is built-in. transaction.on_commit guarantees the flush only runs after the database confirms the change. If the transaction rolls back (a validation error, a constraint violation, a raise somewhere), the callback is discarded. You never sync phantom data.

Batch resolution avoids N+1. The service resolves all references in bulk queries, not one at a time. For 50 changed products, it takes exactly 1 query regardless of count. No per-product lazy loading.

Design Decision: Avoiding the Global State Pitfall

You might be wondering why we used threading.local() in Step 2 instead of a simple module-level variable like _pending_changes = set().

A basic module-level set shares state across every request handled by the same worker process. If User A's request rolls back, stale refs from their failed transaction sit in the global set. When User B's request commits ten minutes later on the same worker, the flush accidentally picks up both User A's and User B's products.

By using threading.local(), we protect against this cross-request leakage. Even if a transaction rolls back, any leftover references in that thread's set will simply be cleared on its next successful commit. And in the rare event a stale reference makes it to the resolver, it evaluates to nothing and is skipped silently.

One thing to watch for: use .filter() when looking up products in your resolver, not .get(). .filter() returns an empty queryset gracefully. .get() throws DoesNotExist and tanks your flush.

The Result

The user experience is deceptively simple. A user toggles "Sync product prices" on a store configuration page. From that moment on, any price change from any code path in the application syncs to the external marketplace within seconds.

No one has to remember to add a sync call to new features. No one has to track down every place a price can change. No one monitors a queue for failures (Celery retries handle that). The system just works.

How are you handling external API syncs in your Django apps? Are you using signals, or do you prefer a different pattern? Drop your approach in the comments.

¡Hasta luego!

How to Get Free SSL Certificates with Docker & LetsEncrypt

Acel — Tue, 23 Jul 2024 20:44:05 +0000

"An SSL certificate is a digital certificate that authenticates a website's identity and enables an encrypted connection. SSL stands for Secure Sockets Layer, a security protocol that creates an encrypted link between a web server and a web browser." - kaspersky

Majority of websites built today use SSL certificates, and if you are building a website in 2024, you will need to learn how to get one too!

There are several approaches to getting an SSL certificate for your domain. But in this article, we will take a look at generating SSL certificates with Let's Encrypt - a nonprofit certificate authority (CA) that provides free SSL/TLS certificates for enabling HTTPS (secure HTTP) on websites. In addition to this, automating the renewal process with Certbot and using a Docker image that was built on top of the official Nginx Docker images.

Prerequisites

To successfully complete this guide, you should be familiar with the following:

Docker and Docker Compose
Virtual machines from cloud providers, e.g. Azure VMs, AWS EC2 etc.
Linux

It is a good idea to containerize your app with Docker, but if you don't want to, you can still follow along just fine!

Configuring Certbot and Nginx

In the root directory of your project, please create a docker-compose file.

touch docker-compose.yml

Then paste the following code in the docker-compose.yml file.

version: '3'

services:
  nginx:
    image: jonasal/nginx-certbot:latest
    restart: unless-stopped
    environment:
      - CERTBOT_EMAIL
    env_file:
      - ./nginx-certbot.env
    ports:
      - 80:80
      - 443:443
    volumes:
      - nginx_secrets:/etc/letsencrypt
      - ./user_conf.d:/etc/nginx/user_conf.d

volumes:
  nginx_secrets:

The code we just pasted above is a YAML configuration for our Nginx docker image. It makes use of the jonasal/nginx-certbot:latest image that has certbot built on top of Nginx.

Now, let us create a nginx-certbot.env file to store the variables needed for our Nginx image.

touch nginx-certbot.env

Then paste this into your nginx-certbot.env file

# Required
CERTBOT_EMAIL=your@email.com

# Optional (Defaults)
DHPARAM_SIZE=2048
RSA_KEY_SIZE=2048
ELLIPTIC_CURVE=secp256r1
RENEWAL_INTERVAL=8d
USE_ECDSA=1
STAGING=1

Next, we create a server configuration file to configure our Nginx server.

touch server.conf

server {
    # Listen to port 443 on both IPv4 and IPv6.
    listen 443 ssl default_server reuseport;
    listen [::]:443 ssl default_server reuseport;

    # Domain names this server should respond to.
    server_name yourdomain.com www.yourdomain.com;

    # Load the certificate files.
    ssl_certificate         /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key     /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/yourdomain.com/chain.pem;

    # Load the Diffie-Hellman parameter.
    ssl_dhparam /etc/letsencrypt/dhparams/dhparam.pem;

    # Redirect non-https traffic to https
    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    }

    return 200 'Let\'s Encrypt certificate successfully installed!';
    add_header Content-Type text/plain;
}

Now, all we need to do is build the docker image and run it on our server.

docker compose -f docker-compose.yml build

Next, move the images to the server whose IP address is mapped to a domain name you own and run this command:

docker compose -f docker-compose.yml up

The command starts the server and Certbot automatically creates new SSL certificates for us. Try navigating to the domain name mapped to the server, you should see a "not trusted" page.

This is because we used "test" certificates to make sure that our configuration works fine. Issuing of live certificates is rate limited, so using test/staging certificates first is a better approach to ensure that it runs fine.

Now, we will update our nginx-certbot.env file to enable us generate live SSL certificates. Change the staging value from 1 to 0.

...
STAGING=0

Stop the server with ctrl+c and run the command below to regenerate SSL certificates.

docker exec -it <container_name> /scripts/run_certbot.sh force

Start up the server again

docker compose -f docker-compose.yml up

Now, navigate to your domain name. Your site should open up just fine!

Certbot and Nginx configurations for Dockerized apps

Assuming you dockerized your app, you will need to make a few more changes to what we have done so far.
First, we update the docker-compose.yml file so that the Nginx service depends on your app.

    ...
    volumes:
      - nginx_secrets:/etc/letsencrypt
      - ./user_conf.d:/etc/nginx/user_conf.d
    depends_on:
      - web

  web:
    container_name: core_app
    build: .
    restart: always
    ...

volumes:
  nginx_secrets:

Where web is the name of your app service.

Then, we update the server.conf file to point to our web server.

upstream webapp {
    server core_app:8000;
    # core_app is the container name of the web server
    # 8000 is the port for the web container
}
server {
    # Listen to port 443 on both IPv4 and IPv6.
    listen 443 ssl default_server reuseport;
    listen [::]:443 ssl default_server reuseport;

    # Domain names this server should respond to.
    server_name yourdomain.com www.yourdomain.com;

    server_tokens off;
    client_max_body_size 20M;

    # Load the certificate files.
    ssl_certificate         /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key     /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/yourdomain.com/chain.pem;

    # Load the Diffie-Hellman parameter.
    ssl_dhparam /etc/letsencrypt/dhparams/dhparam.pem;

    # Redirect non-https traffic to https
    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    }

    location / {
        proxy_pass http://webapp;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }
}

Now, Nginx should be routing traffic directly to your web server (docker container).

Conclusion

We have successfully created free SSL certificates for our domain name that renew automatically, so we never have to worry about it unless our server goes down.

In addition to this article, you can learn more about this approach here: https://github.com/JonasAlfredsson/docker-nginx-certbot/tree/master

Thanks for reading.

¡Hasta luego!