DEV Community: Rahim Ranxx

Django + Celery + Redis Sentinel: A Real Failover Test (With Metrics)

Rahim Ranxx — Sat, 04 Apr 2026 17:36:44 +0000

Redis Sentinel + Celery Failover: What Actually Happens in Production

Most tutorials on Redis Sentinel stop at “it elects a new master”.
Very few show what happens to a real system under failover pressure.

I ran a failover drill on a Django + Celery stack backed by Redis Sentinel and Prometheus monitoring.

Here’s what actually happened.

Architecture Overview
Sentinel Integration (Django + Celery)
Observability with Prometheus
Failover Drill Walkthrough
Celery Behavior During Failover
Performance Impact
Production Readiness Assessment
How to Reduce Failover Latency

Architecture Overview

flowchart LR
    Client --> Django
    Django -->|Cache| Sentinel
    Django -->|Tasks| Celery
    Celery -->|Broker| Sentinel
    Celery -->|Result Backend| Sentinel

    Sentinel --> RedisMaster
    Sentinel --> RedisReplica1
    Sentinel --> RedisReplica2

    Prometheus --> RedisExporter
    RedisExporter --> Sentinel

Stack Components

Django → Redis cache via Sentinel
Celery → Broker + result backend via Sentinel
Redis Sentinel → High availability + failover
Prometheus + redis_exporter → Monitoring

Sentinel Integration (Django + Celery)

All services were switched to Sentinel using environment configuration:

REDIS_ADDR=redis://host.docker.internal:26379

Validation steps:

Django cache → successful round-trip
Celery broker → connected via Sentinel
Celery result backend → SentinelBackend initialized
Test suite passed:

  pytest tests/test_settings_redis_sentinel.py

At this stage, the system is fully Sentinel-aware

Observability with Prometheus

After pointing redis_exporter to Sentinel:

Key metrics exposed:

redis_sentinel_master_status
redis_sentinel_master_ok_sentinels
redis_sentinel_master_ok_slaves
redis_sentinel_masters

Verification:

redis_instance_info{redis_mode="sentinel", tcp_port="26379"}

This confirms monitoring is tracking cluster state, not a single node.

Failover Drill Walkthrough

Initial State

flowchart LR
    Sentinel -->|Master| Redis1["172.20.0.3:6379"]
    Sentinel --> Redis2["Replica"]
    Sentinel --> Redis3["Replica"]

Prometheus reported:

master_address="172.20.0.3:6379"

Induced Failure

Current master was stopped manually

Sentinel Election

flowchart LR
    Sentinel -->|New Master| Redis2["172.20.0.2:6379"]
    Sentinel --> Redis3["Replica"]
    Sentinel --> Redis1["Down"]

New master elected on first poll
Prometheus updated on next scrape

Failover was immediate and correct

Celery Behavior During Failover

Timeline

sequenceDiagram
    participant App as Django App
    participant Celery
    participant Sentinel
    participant Redis

    App->>Celery: Submit Task
    Celery->>Redis: Send to Master
    Redis-->>Celery: Connection Lost

    Sentinel->>Sentinel: Elect New Master

    Celery->>Sentinel: Retry Connection
    Note over Celery: ~54.7s delay

    Celery->>Redis: Reconnect to New Master
    Redis-->>Celery: OK

    Celery-->>App: Task SUCCESS

Observed Task

Task ID: 9b57ba3b-a707-4c13-9255-d74de411b64b
Status during failover: PENDING
Delay: ~54.7 seconds
Final state: SUCCESS

Performance Impact

Phase	Behavior
Normal operation	Immediate execution
During failover	~55s delay
Post-recovery	Normal

Production Readiness Assessment

What Works

Redis Sentinel failover is reliable
Prometheus reflects cluster changes correctly
Django cache survives failover
No task loss in Celery

What Needs Attention

Celery introduces significant delay during failover
Reconnection is not instantaneous

When This Architecture Is Production-Ready

Use this setup if:

Tasks are asynchronous/background
Eventual completion is acceptable
Temporary latency spikes are tolerable

When This Is Not Enough

Avoid this setup (as-is) if you need:

Real-time task execution
Sub-10s failover recovery
User-facing async operations

How to Reduce Failover Latency

To push recovery closer to 10–15 seconds:

Tune Celery broker retry settings
Reduce reconnect backoff intervals
Optimize worker heartbeat and visibility timeout
Re-run failover drills with timing instrumentation

Key Takeaway

Redis Sentinel ensures infrastructure recovery.
Celery determines how fast your system actually resumes work.

In this test:

Sentinel recovery: instant
Application recovery: ~55 seconds

That gap is the real engineering challenge.

Final Thoughts

If you're using Redis Sentinel with Celery:

Don’t stop at:

“Failover works.”

Measure:

“How long until my system behaves normally again?”

Because that’s what production users experience.

Escaping Cache Fragmentation: How Misconfigured PHP Workers Flooded My Token System

Rahim Ranxx — Sun, 22 Mar 2026 18:02:23 +0000

🚨 The Symptom

I started noticing something strange in my observability stack:

Integration tokens were being minted repeatedly
My token endpoint showed activity even when no user interaction was happening
Metrics suggested constant “traffic” to an otherwise idle system

At first glance, it looked like:

A security issue
A rogue client
Or a broken API consumer

It was none of those.

🔍 The Root Cause

The issue came down to a subtle but critical architectural mistake:

I was using a non-shared cache in a multi-worker environment.

Stack involved:

PHP-FPM (2 workers)
APCu (in-memory cache)
Token-based integration between services

⚙️ What Went Wrong

APCu is process-local, not shared.

That means:

Worker A cache ≠ Worker B cache

Each PHP-FPM worker had its own isolated memory.

💥 The Cascade Effect

My token logic was straightforward:

if token not in cache:
    mint_new_token()

But in reality, the system behaved like this:

Request hits Worker A → token exists → OK
Next request hits Worker B → cache miss → mint new token
Repeat across workers → continuous token regeneration

📈 Why Observability Looked “Wrong”

From the outside, it looked like traffic was hitting the token endpoint.

But in reality:

The system was generating its own traffic due to cache inconsistency.

This is a key lesson:

Not all traffic is external
Some is emergent behavior from system design

✅ The Fix

I switched from APCu to:

Redis (shared cache)

Now:

All workers → same cache → consistent token state

Result:

Tokens minted once
Reused across all workers
Metrics stabilized instantly

🔒 Production Hardening (What I Added Next)

Fixing the cache wasn’t enough — I hardened the system further.

1. Distributed Locking

To prevent race conditions:

if token exists:
    return token

acquire lock
    re-check cache
    mint token if still missing
release lock

2. TTL Buffering

Avoid edge expiration issues:

cache_ttl = token_expiry - safety_margin

3. Observability Metrics

I added:

token_cache_hits
token_cache_misses
token_mint_count

Now anomalies show up immediately.

🧠 Key Takeaway

This wasn’t just a bug.

It was a distributed systems failure mode:

Cache locality + multi-worker architecture → inconsistent state → emergent traffic

⚡ Final Insight

If your system:

Runs multiple workers
Uses in-memory caching
Relies on shared state

Then this rule applies:

If your cache isn’t shared, your state isn’t real.

🔗 Closing

This issue reinforced something critical in my engineering journey:

You don’t debug systems by staring at code —
you debug them by understanding how state flows across boundaries.

If you're building distributed APIs, token systems, or high-concurrency services —
this is one edge case worth designing for early.

From 80-Second APIs to Sub-Second: Rebuilding a Geospatial Backend with Async Pipelines

Rahim Ranxx — Sat, 21 Mar 2026 16:37:10 +0000

From 80-Second APIs to Sub-Second: Fixing Latency with Async Pipelines (Django + Celery)

Introduction

At some point, every backend engineer hits this wall:

The API works perfectly… until it doesn’t.

I hit that wall with a farm analytics endpoint computing NDVI (Normalized Difference Vegetation Index) from satellite imagery. The system was correct, the logic was sound, and the results were accurate.

But the numbers told a different story:

P95 latency: 1.25 minutes

That’s not an API. That’s a blocking compute job pretending to be one.

This is the story of how I redesigned the system—from a synchronous request-driven model to an asynchronous data pipeline—and brought latency down to sub-second performance (P95 ≈ 725ms).

The Original Architecture (The Hidden Problem)

At first glance, the system looked clean:

[Client]
   ↓
[Django API]
   ↓
[STAC API → Satellite Data]
   ↓
[Raster Processing (NDVI)]
   ↓
[Response]

What happened on each request?

Query satellite imagery via STAC
Fetch raster bands (Red & NIR) from remote storage
Process NDVI using rasterio
Aggregate coverage
Return result

Why this seemed fine

It worked locally
It returned correct data
It followed a “pure API” mindset

But under the hood:

Remote I/O (S3-backed satellite data)
Heavy raster decoding (JPEG2000)
Sequential band reads
Full computation per request

The Breaking Point

Logs told the truth.

Each request looked like:

STAC request → ~5s
Raster read (B04) → ~5–10s
Raster read (B08) → ~5–10s
Processing → ~5s+
Total → ~80+ seconds

And the key realization:

I wasn’t building an API—I was executing a geospatial compute pipeline on every request.

The Core Insight

This is the shift that changes everything:

APIs should serve data, not compute it on demand.

The problem wasn’t Python.
The problem wasn’t Django.
The problem was architecture.

The New Architecture (Async Pipeline)

I redesigned the system around asynchronous computation + caching:

             (Scheduled / Triggered)
                    ↓
             [Celery Worker]
                    ↓
         [NDVI Computation Pipeline]
                    ↓
             [Redis / Database]
                    ↓
[Client] → [Django API] → [Cache Lookup]

Key changes

NDVI computation moved out of the request path
Results cached in Redis
Background jobs compute and refresh data
API returns instantly (no heavy compute)

Diagram 1 — Before vs After

Before (Request-driven)

Request
   ↓
STAC API
   ↓
Raster I/O
   ↓
NDVI Compute
   ↓
Response (80s)

After (Pipeline-driven)

Request → Cache → Response (~725ms P95)
              ↓ (miss)
         Async Task
              ↓
       Compute + Store

Implementation

1. Fast API Path (Non-blocking)

from django.core.cache import cache
from ndvi.tasks import compute_farm_state_coverage

def get_farm_state(farm_id: int) -> dict:
    cache_key = f"farm_state:{farm_id}"

    data = cache.get(cache_key)
    if data:
        return data

    compute_farm_state_coverage.delay(farm_id=farm_id)

    return {
        "coverage_pct": None,
        "status": "processing"
    }

2. Celery Task (Async Compute)

from celery import shared_task
from django.core.cache import cache

@shared_task(bind=True, autoretry_for=(Exception,), retry_backoff=True)
def compute_farm_state_coverage(self, farm_id: int) -> None:
    coverage = compute_ndvi_coverage(farm_id)

    cache.set(
        f"farm_state:{farm_id}",
        {
            "coverage_pct": coverage,
            "status": "ready"
        },
        timeout=60 * 60 * 6,
    )

3. Daily Backfill (Critical)

from celery import shared_task

@shared_task
def enqueue_daily_farm_state_coverage():
    farm_ids = get_active_farm_ids()

    for farm_id in farm_ids:
        compute_farm_state_coverage.delay(farm_id=farm_id)

Observability (The Real Upgrade)

Metrics added:

Task duration
Task success/failure
Queue depth

Metrics (Grafana Observations)

📊 Grafana Screenshots

1. Latency Graph

Before

P95 latency: ~1.25 minutes

After

API latency: ~725ms (P95)
Background tasks: 60–90s

Before vs After Summary

Metric	Before	After
API latency	1.25 min	~725 ms (P95)
System type	Request-driven	Pipeline-driven
Scalability	Poor	Strong
Observability	Minimal	Improved

Final Thought

I stopped treating my API like a calculator and started treating my system like a data pipeline.

That’s when everything changed.

Designing a One-Way Farm Sync Architecture (Nextcloud Django DRF)

Rahim Ranxx — Sun, 15 Mar 2026 07:11:34 +0000

From Nextcloud to Django: Designing a Farm Sync Architecture with DRF

Modern applications rarely live in a single system. As projects grow, different components begin to specialize: one system handles identity and user workflows, while another focuses on computation and domain logic.

This week I explored a small but interesting distributed architecture problem: how to synchronize farm data between a Nextcloud application and a Django REST API backend.

The goal was simple in theory:

Nextcloud provides the user interface.
Django performs geospatial computation (NDVI, raster processing).
Farm data must stay consistent between the two systems.

But as any engineer knows, “simple in theory” is where architecture decisions start to matter.

The Initial Problem

In the system I’m building, users manage farms from a Nextcloud application while a Django service handles geospatial workloads.

The challenge was deciding where farm data should live and how it should propagate between systems.

Three architectural questions emerged:

Which system is the source of truth?
How do we synchronize data between services?
How do we avoid identity conflicts between systems?

These are classic distributed systems questions, even in relatively small projects.

Option 1: Bidirectional Sync (The Dangerous Path)

One tempting solution is letting both systems create farms and then syncing them.

Nextcloud <--> Django

At first glance this feels flexible. In practice it creates difficult problems:

conflicting updates
race conditions
reconciliation logic
versioning requirements

Large distributed databases solve this with vector clocks and conflict resolution strategies. For most applications, that complexity is unnecessary.

So I rejected bidirectional replication early.

Option 2: API-First Architecture

Instead of replicating farms between databases, Nextcloud simply delegates creation to the Django API.

When a user creates a farm in Nextcloud:

User
  ↓
Nextcloud UI
  ↓
Nextcloud Controller
  ↓
POST /api/v1/farms/ (Django REST Framework)
  ↓
Django database

Django becomes the source of truth for farm data, while Nextcloud acts as the interface layer.

This pattern has several advantages.

Immediate Consistency

Since farms are created directly in Django, the backend always has the latest data.

There is no delayed replication.

Clear Ownership

Each system has a defined responsibility:

Nextcloud – user interface, identity, workflow
Django – domain logic, geospatial processing, data storage

Clear boundaries reduce architectural complexity.

Extensibility

Once Django exposes a clean API, other systems can integrate easily:

mobile apps
data pipelines
satellite processing services
analytics dashboards

Everything interacts with the same API.

Solving the Cross-System Identity Problem

Once multiple systems talk about the same object, a subtle problem appears:

identity consistency.

If each system generated its own farm IDs, synchronization would become fragile:

Nextcloud Farm ID = 12
Django Farm ID = 47

Now every integration requires mapping tables.

Instead, the architecture introduces a stable external identifier.

Nextcloud generates a UUID called:

external_farm_id

That UUID is sent to Django whenever a farm is synchronized.

Conceptually the model looks like this:

Farm
 ├─ id (internal database id)
 └─ external_farm_id (shared UUID)

Now both systems reference the farm using the same identifier.

When Nextcloud syncs a farm:

POST /api/v1/farms/sync

Payload example:

{
  "external_farm_id": "uuid",
  "external_user_id": "nextcloud_uid",
  "name": "Demo Farm",
  "bbox": "...",
  "centroid": "..."
}

This approach provides several benefits.

No ID Collisions

UUIDs are globally unique, preventing conflicts between systems.

Clean Synchronization

Updates and raster requests can reference farms using the same external identifier.

/api/v1/farms/{external_farm_id}

Future Integration

If other services appear later (mobile apps, analytics pipelines, satellite processors), they can all reference farms using the same UUID.

This pattern is common in distributed systems and prevents a large class of synchronization bugs.

Identity Translation Between Systems

Another challenge is mapping users between Nextcloud and Django.

Nextcloud users authenticate normally and then communicate with Django using an integration token.

Conceptually:

Nextcloud User
      ↓
Integration Token
      ↓
Django API
      ↓
Farm Sync

The Django service stores farms under a service user while still preserving the original external_user_id from Nextcloud.

This keeps authentication simple while preserving user ownership information.

Measuring the Integration Layer

During development I analyzed the Nextcloud app using cloc to understand the size of the integration layer.

The results showed roughly 11,000 lines of code, split mainly between PHP backend logic and JavaScript UI.

Around this size, architecture decisions begin to matter more than raw implementation.

Systems become complex enough that clear service boundaries become essential.

Lessons Learned

Several practical lessons emerged from this integration work.

1. Avoid bidirectional replication

Two systems writing to the same domain model creates unnecessary complexity.

2. Establish a clear source of truth

In this architecture, Django owns farm data while Nextcloud orchestrates the workflow.

3. Use stable external identifiers

UUIDs dramatically simplify synchronization across systems.

4. Prefer API-first architectures

APIs make it easier to expand systems and integrate future services.

5. Keep compute close to the data

Since Django handles geospatial processing, storing farms there keeps the compute layer efficient.

Final Thoughts

What started as a simple integration between Nextcloud and Django turned into a useful exercise in distributed system design.

Even relatively small systems benefit from clear service boundaries and stable identity strategies.

By combining:

an API-first architecture
external UUID identifiers
and clear ownership of farm data

the system stays simple today while remaining extensible for future services like satellite analytics or mobile farming applications.

Sometimes good architecture isn’t about complexity at all — it’s about clarity of responsibility between systems.

Escaping the Sync Trap: How I Slashed Latency by 10x in a Django-Rust API Gateway

Rahim Ranxx — Sun, 01 Mar 2026 13:24:13 +0000

How I diagnosed and eliminated synchronous bottlenecks in a Django-Rust API gateway, migrating to ASGI and pre-warming caches for millisecond responses.

When building a high-performance backend, the standard playbook is well-known: offload heavy computational tasks to faster microservices (like Rust) and implement an aggressive caching strategy.

Recently, I did exactly that. My architecture is built around a Django REST Framework gateway sitting behind Caddy, heavily monitored with Prometheus and Grafana.

But despite the raw speed of Rust and my caching layers, my dashboards were flashing red. Latency was spiking to brutal 10-second flatlines for my most critical endpoints. Worse, my observability itself started failing, creating silent blind spots exactly when I needed data the most.

Here is the detective story of how I used telemetry to hunt down synchronous traps, migrate to a non-blocking async architecture, and implement proactive pre-warming to bring response times down to the millisecond range — all while reclaiming 30% of my idle CPU.

The Architecture: A Gateway and its Heavy Lifters

Before diving into the problem, here is a quick look at my setup.

          ┌──────────────────────────────┐
          │          Nextcloud           │
          │ (Authenticated Client Calls) │
          └──────────────┬───────────────┘
                         │  JWT / API Key
                         ▼
               ┌────────────────────┐
               │  Django Gateway    │
               │ (ASGI, DRF, Caddy) │
               └──────┬─────────────┘
                      │
     ┌────────────────┴────────────────┐
     │                                 │
     ▼                                 ▼
┌───────────────┐              ┌────────────────┐
│ NDVI Service  │              │ Weather Service│
│ (Rust, 8081)  │              │ (Rust, 8090)   │
│  → Postgres    │              │  → MySQL       │
└───────────────┘              └────────────────┘

       ▲
       │ Prometheus & Grafana
       │ (Observability Stack)
       ▼
   System Telemetry + Metrics

Django stays as the public API gateway, accepting requests authenticated via JWT or API keys.
It enforces a shared JSON response envelope containing status, message, data, and errors to keep all client interactions standardized.

Specific traffic routes — namely /api/v1/ndvi and /api/v1/weather/* — are forwarded directly to my Rust backends:

🛰 NDVI microservice ingests satellite data into a dedicated Postgres database.
🌦 Weather microservice relies on a MySQL database and communicates with external providers like Open-Meteo and NASA POWER.

My Nextcloud instance acts like any other client, presenting either an Authorization: Bearer token or an X-API-Key. Django manages this traffic using a specific nextcloud_hmac throttle configuration before passing the authorized call down to Rust with the original headers intact.

The False Cure: The Worker Starvation Anomaly

To protect the system, I implemented aggressive TTL caching (e.g., 1 hour for schema data, 5 minutes for API tokens). However, once I added traffic, my Grafana dashboard revealed a chaotic reality.

I saw brutal, perfectly flat 8–10 second latency spikes on key endpoints. Crucially, perfectly timed with these latency spikes, my internal /metrics request rate dropped to zero.

The Diagnosis: The Gateway Caching Itself to Death

The telemetry told a story of hidden synchronous bottlenecks:

The Trigger: When a high-traffic endpoint like farm-weather-current/GET experienced a cache miss, the Django gateway had to fetch fresh data.
The Trap: My Django deployment was running using standard synchronous workers. It called the Rust service, which then called the external weather API (taking 2.2+ seconds).
The Impact (Worker Starvation): Because the Django worker was synchronous, it blocked entirely for those 2.2 seconds. All incoming traffic got stuck in a queue.

The Trap: Synchronous Gateway Routing

import httpx
from django.http import JsonResponse
from django.views import View

async def async_weather_proxy_view(request):
    async with httpx.AsyncClient(timeout=5.0) as client:
        resp = await client.get("http://weather-service:8090/api/v1/weather-current")
        resp.raise_for_status()
        data = resp.json()
    return JsonResponse({"status": "success", "data": data, "errors": None})

I had successfully offloaded work to Rust — but my synchronous Django workers completely nullified the speed gains.

The First Fix: Embracing the Non-Blocking Gateway

I needed to decouple the speed of the gateway from the speed of the external API calls it was routing.
I migrated the Django deployment from synchronous workers to an ASGI (Asynchronous Server Gateway Interface) setup, allowing my gateway to handle requests asynchronously.

I rewrote my proxy views to use asynchronous HTTP clients like httpx:

The Fix: Asynchronous Non-Blocking Routing

import httpx
from django.http import JsonResponse

async def async_weather_proxy_view(request):
    # The event loop is freed! Django can serve other requests while waiting
    async with httpx.AsyncClient() as client:
        rust_response = await client.get("http://weather-service:8090/api/v1/weather-current")

    return JsonResponse({
        "status": "success",
        "data": rust_response.json(),
        "errors": None
    })

The visual evidence on my dashboards was a massive, instant victory:

Observability Restored: The metrics scrape line remained unbroken. Django could finally pause a slow weather request, instantly answer the Prometheus scrape, and resume without blocking.
⚡ Instant Internal Routing: In my initial setup, a simple internal metrics scrape took ~84ms. After the ASGI migration, that duration dropped to 11ms.

The Second Fix: Proactive Caching

While the infrastructure was now bulletproof, the end-user experience was still occasionally sluggish.

With an “on-demand” caching strategy, the very first user to request the weather after a 1-hour cache expiration had to pay the Cache Miss Penalty (waiting ~2.2 seconds for the external API).

To eliminate this, I decoupled the data-fetching time from the user-request cycle entirely.
I implemented a Proactive Background Pre-warming pattern using a background task (like Celery) that runs every 55 minutes, independently fetching slow data and silently overwriting the cache before it expires.

The Cache Pre-Warmer (Celery Example)

from celery import shared_task
from django.core.cache import cache
import requests

@shared_task
def pre_warm_weather_cache():
    # Runs in the background every 55 minutes, shielding the user from the 2.2s wait
    response = requests.get("http://weather-service:8090/api/v1/weather-current")
    if response.status_code == 200:
        cache.set("weather_current_data", response.json(), timeout=3600)

The result? The average latency for critical weather endpoints plummeted from seconds to milliseconds as the hot cache permanently took over.

The Grand Slam: Optimizing the Rust Build Pipeline

The final victory of this new architecture came from pure server efficiency.
During deployments, compiling Rust crates (sqlx, syn) from scratch was pegging my 4-core server at 100% CPU, artificially causing timeouts.

To fix this, I implemented cargo-chef in a multi-stage Dockerfile to strictly cache Rust dependencies.

Multi-stage Dockerfile for the Rust Microservice

FROM rust:1.88-slim AS chef
USER root
RUN cargo install cargo-chef
WORKDIR /app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
# Docker caches this heavy dependency build!
RUN cargo chef cook --release --recipe-path recipe.json
COPY . .
RUN cargo build --release --bin weather-service

FROM debian:bookworm-slim AS runtime
WORKDIR /app
RUN apt-get update && apt-get install -y libssl-dev ca-certificates
COPY --from=builder /app/target/release/weather-service /usr/local/bin/
EXPOSE 8090
ENTRYPOINT ["/usr/local/bin/weather-service"]

Between ASGI, background caching, and Docker layer caching, my total Node CPU now rests comfortably between 11% and 13%.
I fundamentally reclaimed 30% of my total server compute capacity.

Conclusion: Finding the Next Bottleneck

Building high-performance API gateways is an ongoing journey of shifting bottlenecks.

By relying strictly on my telemetry, I proved that synchronous workers nullify microservice speed, validated the immense power of ASGI, and eliminated cache miss penalties.

With the gateway running unburdened, my dashboards have revealed one final bottleneck — a 6-to-8 second delay on my token generation endpoint.
Because my CPU is mostly idle, I know exactly what this is: a database connection pool limitation in the Rust service.

And thanks to my new observability baseline, I know exactly where to strike next.

Boost Performance by Migrating Django Endpoints to Rust: NDVI & Weather Services (Phase 2 Complete)

Rahim Ranxx — Sat, 21 Feb 2026 16:55:43 +0000

Migrating Django Endpoints to Rust: My NDVI & Weather Services Journey

When I started rethinking my NDVI and weather endpoints, the goal was simple: improve performance, enforce strong auth, and gain full observability. Over the last few weeks, I migrated critical services from Django to Rust, and the process turned out to be an engineering adventure worth sharing.

Phase 0 – Contract Freeze: Locking the APIs

Before touching Rust, I froze all NDVI and weather API contracts in Django. This ensured that the front-end and other consumers could continue working without disruptions. Think of it as putting a protective glass over your APIs: nothing moves until Rust is ready to take over.

Output: Frozen NDVI + weather contracts from Django.

Phase 1 – Multi-Service Architecture & Shared Auth/Throttle

Next, I set up a Rust workspace with multiple services:

NDVI service: Handles vegetation index calculations.
Weather service: Will eventually serve weather data.
Shared auth & throttling module: Ensures consistent authentication and rate limiting across all services.

This phase established the skeleton for independent Rust microservices while maintaining the same contract as Django.

Output: Rust workspace, shared auth/throttle, NDVI envelope.

Phase 2 – Weather Migration

With the workspace ready, I migrated weather endpoints from Django to Rust. Key steps included:

Implementing shared authentication and throttling.
Integrating MySQL connections safely with Rust’s type system.
Ensuring the endpoints conformed to the frozen contract from Phase 0.

After this phase, all weather requests were fully handled by Rust services, improving throughput and reliability.

Output: Weather endpoints implemented in Rust.

Phase 3 – Gateway Cutover (Planned)

The final phase will transition Django routes to forward requests to Rust microservices. This will include:

Canary deployments to avoid downtime.
Metrics and alerting for observability.
CI enforcement for Rust formatting, clippy lints, and tests across the workspace.

End state after Phase 3:

Django acts as a gateway, routing NDVI + weather requests to Rust services.
NDVI is fully served by Rust/Postgres.
Weather is fully served by Rust/MySQL.
Shared auth and throttling are enforced in Rust.
Observability and canary rollouts ensure safe production deployment.
CI checks formatting, linting, and tests across the workspace.

Lessons Learned

Contract first: Freezing contracts before migration prevents chaos.
Shared modules are gold: Auth and throttling reused across services reduces duplication.
Rust’s type system and ownership model force careful database and network design.
Incremental migration avoids “big bang” outages.

Why Rust?

Migrating to Rust allowed me to:

Serve high-throughput endpoints with lower latency.
Reduce runtime errors with compile-time guarantees.
Scale services independently while sharing critical modules like auth and throttling.

Example: Rust Weather Service (axum + sqlx)

// src/main.rs
#![deny(clippy::all)]
#![forbid(unsafe_code)]

use axum::{routing::{get, post}, Router, Json, extract::Extension, response::IntoResponse, http::StatusCode};
use serde::{Deserialize, Serialize};
use sqlx::mysql::MySqlPoolOptions;
use std::sync::Arc;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

mod auth;
mod rate_limit;

#[derive(Clone)]
struct AppState {
    db: sqlx::MySqlPool,
}

#[derive(Deserialize)]
struct WeatherQuery {
    lat: f64,
    lon: f64,
    ts: Option<i64>,
}

#[derive(Serialize)]
struct WeatherResponse {
    temp_c: f32,
    precip_mm: f32,
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    tracing_subscriber::registry()
        .with(tracing_subscriber::EnvFilter::from_default_env())
        .with(tracing_subscriber::fmt::layer())
        .init();

    let db = MySqlPoolOptions::new()
        .max_connections(20)
        .connect(&std::env::var("WEATHER_DATABASE_URL")?)
        .await?;

    let state = Arc::new(AppState { db });

    let app = Router::new()
        .route("/api/v1/weather/point", get(get_weather_point))
        .route("/api/v1/weather/bulk", post(post_weather_bulk))
        .layer(auth::AuthLayer::new())
        .layer(rate_limit::RateLimitLayer::new(100));

    let addr = std::env::var("LISTEN_ADDR").unwrap_or_else(|_| "0.0.0.0:8080".into());
    tracing::info!("listening on {}", addr);
    axum::Server::bind(&addr.parse()?)
        .serve(app.into_make_service())
        .await?;

    Ok(())
}

Django Gateway Proxy Example

# views/proxy.py
import httpx
from django.http import HttpResponse
from django.conf import settings

PROXY_TARGET = settings.PROXY_TARGET  # e.g., "http://rust-weather:8080"

async def proxy_request(request):
    upstream = f"{PROXY_TARGET}{request.path}"
    headers = {"x-forwarded-for": request.META.get("REMOTE_ADDR", "")}
    if "Authorization" in request.headers:
        headers["Authorization"] = request.headers["Authorization"]

    async with httpx.AsyncClient(timeout=30.0) as client:
        upstream_resp = await client.request(
            request.method,
            upstream,
            headers=headers,
            params=request.GET,
            content=await request.body()
        )

    return HttpResponse(
        content=upstream_resp.content,
        status=upstream_resp.status_code,
        headers={k: v for k, v in upstream_resp.headers.items() if k.lower() != "transfer-encoding"},
    )

CI Example (GitHub Actions)

name: CI
on: [push, pull_request]
jobs:
  rust-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain-action@v1
      - run: cargo fmt --all -- --check
      - run: cargo clippy --workspace --all-targets -- -D warnings
      - run: cargo test --workspace --all-features

  python-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install ruff mypy bandit
      - run: ruff check .
      - run: mypy .
      - run: bandit -r .

This setup ensures a production-ready, highly observable Rust microservices environment while keeping Django as a stable gateway. Phase 3 will finalize the gateway cutover with canary deployment and metrics monitoring.

From Django to Rust Microservices: What Prometheus Taught Me About Backend Performance

Rahim Ranxx — Sun, 15 Feb 2026 06:48:42 +0000

Django Performance and Prometheus Observability
I operate a stack combining Django REST Framework, Nextcloud integrations, Prometheus for metrics, and Grafana dashboards — all served behind Caddy with strict CI/CD and Dockerized isolation.

Everything looked stable until my Prometheus metrics told a different story.

In Grafana, the /prometheus-django-metrics endpoint consistently showed 250 ms latency spikes, while other endpoints like /farm-weather-hourly and /home averaged under 50 ms. Scrape durations varied between 80 ms and 430 ms, even when request rates stayed flat at 0.08 req/s.

That meant the latency wasn’t due to load — it was intrinsic to Python’s runtime and how Django handled metrics serialization.

Why Prometheus Exposed Django’s Bottleneck
Each Prometheus scrape forces Django to:

Lock the Global Interpreter Lock (GIL)
Gather live counters and histograms
Serialize JSON or text payloads
Reallocate memory on every request

Even low-volume systems suffer because this happens repeatedly at fixed intervals. Observability itself became a performance cost.

The graphs made it clear: the bottleneck was the runtime, not the code.

Why Migrate Django Microservices to Rust
Rust’s asynchronous ecosystem (Tokio / Actix Web) solves these exact issues.

No GIL: True multi-core concurrency.
Predictable latency: Consistent under heavy I/O.
Memory safety: Compile-time guarantees without a garbage collector.
Low overhead I/O: Async networking with minimal allocations.

In my benchmarks, Rust microservices consistently stay under 40 ms latency, use 30–40 % less CPU, and make Prometheus scrape times nearly constant.

Rust Microservices Architecture with Django and Prometheus
The new architecture keeps Django as the orchestrator — managing authentication, APIs, and admin routes — while Rust handles performance-intensive modules:

NDVI raster computation
Weather data transformation
Metrics aggregation

They communicate via REST or gRPC. Prometheus exports data from both runtimes into unified Grafana dashboards.
Caddy provides HTTPS termination and reverse-proxy routing, maintaining secure observability across the stack.

This hybrid model keeps Django’s flexibility while giving me Rust’s efficiency where it matters.

Lessons from Observability

Metrics are architectural signals, not just health checks.
Python’s runtime trade-offs appear first under introspection, not user load.
Rust isn’t a replacement for Django — it’s a reinforcement for its weak spots.
Observability drives evolution when used as feedback, not just monitoring.

The Road Ahead
My next experiment involves measuring CPU cycles per request across Django and Rust services under sustained Prometheus scrapes. The goal: prove observability-driven performance scaling in production.

If your /metrics endpoint is your slowest route, don’t ignore it — that graph might be pointing directly toward your next architectural upgrade.

Wired Django, Nextcloud, Grafana, Loki & Prometheus into a secure observability mesh over Tailnet (metrics & logs, dashboards).

Rahim Ranxx — Sun, 08 Feb 2026 05:10:03 +0000

Building an Observability Mesh with Grafana, Loki, and Prometheus
When multiple backend services start running in isolation, debugging becomes guesswork. My recent sprint was about turning that guesswork into clarity — by wiring up full observability across Django, Nextcloud, Grafana, Loki, and Prometheus.

Goal
Unify logs and metrics across services in a distributed setup — all communicating over Caddy TLS and my Tailnet domain.
I wanted one dashboard that could tell me everything about my system’s health without SSH-ing into individual servers.

Architecture
Here’s the high-level design:

Stack Overview

Prometheus → scrapes metrics from Django and Nextcloud API endpoints
Loki → ingests logs from both services
Grafana → visualizes metrics and logs together
Caddy → reverse proxy with trusted TLS for all endpoints
Tailnet (Tailscale) → private network with identity-based access

Everything talks securely — no exposed ports, no unencrypted traffic.

Challenges

1. Grafana showed logs but no metrics
Root cause: Prometheus targets weren’t reachable after moving from localhost to tailnet hostnames.

2. TLS verification issues in Prometheus
Solved by updating Caddy’s certificates and confirming Prometheus scrape configs pointed to HTTPS endpoints.

3. Cross-service routing
Caddy needed to handle routes like /metrics, /api/schema, and /api/* correctly between Django and Nextcloud.

Config Highlights

Here’s a simplified Prometheus scrape config example:

scrape_configs:

job_name: "django" metrics_path: /metrics static_configs:
targets: ["X.tail.ts.net:8000"]
job_name: "nextcloud" metrics_path: /metrics static_configs:
targets: ["X.tail.ts.net:8080"]

Both routes sit behind Caddy, which handles TLS termination using trusted Tailnet certificates.

Results
Once Prometheus started scraping successfully, Grafana dashboards came alive.

Now I can:

Correlate logs and metrics per request
Track uptime and performance trends
Visualize distributed system behavior across all nodes

It feels like operating my own mini control plane — distributed, secure, and explainable.

Next Steps

Add distributed tracing (OpenTelemetry)
Define Prometheus alert rules for critical endpoints
Automate observability config rollout via CI/CD

Key Takeaway
Observability isn’t an add-on — it’s the nervous system of your infrastructure.
When your servers start talking, you start listening differently.

CGNAT Escape Plan: Private Access + Browser-Trusted TLS for My Self-Hosted Stack

Rahim Ranxx — Sun, 01 Feb 2026 08:19:29 +0000

Most self-hosting stories end at “it works on my laptop.” Mine only felt real when my phone loaded the stack remotely with a clean TLS lock in the browser—no warnings—and my services stayed private by default.

This is how I made my self-hosted setup—* *Nextcloud + a Django REST framework API—reachable from anywhere without opening public ports, and how I verified that both the network boundary and TLS trust were actually correct.

The constraint

CGNAT is the quiet villain of home labs. You can build a great system locally, but inbound access becomes fragile or impossible. Even when you can port-forward, “expose 443 to the internet” is a great way to collect bots, scans, and stress you didn’t ask for.

So I set a different goal:

Reach my services from my phone and laptop
Keep them off the public internet
Use browser-trusted TLS (real lock icon, no prompts)
Validate it like a production system: boundaries + proof

Threat model (what I was defending against)

I’m not building a bank, but I am defending against the common stuff:

Random internet scanning and opportunistic attacks
Misconfiguration that accidentally exposes admin panels
“It’s HTTPS” illusions where the browser still doesn’t trust the cert
Curl working while browsers fail because of redirects/headers/mixed content

The strategy: private reachability + controlled entry point + trusted TLS + verification.

The design (high-level)

I used a private network overlay—Tailscale—so my devices can reach the services securely without public inbound ports.

Flow:

Phone/Laptop
→ private network
→ reverse proxy (TLS termination + routing)
→ Nextcloud + Django API

Reverse proxy-wise, you can do this with Caddy or Nginx—the important part is the role: one controlled front door, consistent routing, and TLS handled properly.

What “network was clean” means (not vibes, boundaries)

For me, “clean network” meant:

The services are reachable only over the private network (tailnet)
No public IP exposure for app ports
Reverse proxy is the only entry point I maintain
Everything uses explicit hostnames + stable routing

That’s the difference between “it works” and “it’s defensible.”

The TLS win (browser-trusted)

I didn’t want “HTTPS” that still nags the browser. I wanted the lock icon and a certificate chain the browser trusts.

My acceptance criteria:

The hostname matches the certificate (SAN is correct)
The full trust chain is valid (no warnings)
Works on mobile browsers (the harshest judge)
No weird “mixed content” failures when the UI loads assets

Once that was done, the system stopped feeling like a lab toy and started feeling like a real service.

Proof: how I validated end-to-end

I like checks that produce receipts.

1) Browser trust proof

From my phone:

The site loads with a lock icon
Certificate details match the expected hostname and validity

2) HTTP behavior proof (sanity headers + redirect behavior)

From a trusted client device:

curl -I https:///... confirms:
expected status codes (200/302)
redirects aren’t downgrading to HTTP
headers look intentional, not accidental

3) Boundary proof (private-only reachability)

On the server:

Confirm services bind where you expect (private interfaces / localhost behind proxy)
Confirm the reverse proxy is the only exposed listener you intended

A simple sanity check is reviewing active listeners (ss -tulpn) and verifying only the expected ports are bound to the expected interfaces.

4) Integration proof (UI ↔ API)

The best proof wasn’t curl:

The Nextcloud app successfully calls the Django API
Data renders end-to-end over the private path

That’s the “real system” test.

The debugging lesson: curl worked, browser didn’t

This was the most educational part.

Curl can succeed while browsers fail because browsers enforce:

stricter TLS validation
redirect rules and canonical hostnames
mixed content blocking
security policies (CSP, framing, referrer policy)

So when I saw “works in Termux, blank in browser,” I treated it as a routing/trust signal—not mystery magic. Tightening hostnames, TLS, and consistent proxy routing fixed the mismatch.

Hardening checklist (what I locked down before calling it done)

This is the part that turns “reachable” into “safe enough to run.”

Network & exposure

Services reachable only via private network path
Reverse proxy is the only entry point
Avoid binding internal services to public interfaces

TLS & trust

Browser-trusted certs, correct hostname coverage
No HTTP downgrade paths
Renewals handled automatically (so security doesn’t rot)

App security hygiene

Secrets live in environment config (not in repo)
Strong auth on the API endpoints (and no unauth admin surfaces)
Consistent logging so failures aren’t invisible

Practical resilience

Basic monitoring and dashboards are next (I’m aiming for Grafana + Prometheus)

-Backups and recovery plan (because “it works” is not the same as “it survives”)

What I learned

Private access is not a compromise—it’s a security strategy.
Browser-trusted TLS is a real quality bar. If the browser trusts it, you eliminate a whole class of hidden problems.
Verification beats vibes. I now validate: reachability, trust, boundaries, and integration.

Closing

This wasn’t just “getting Nextcloud running.” It was turning a home lab into something closer to production:

private-by-default access
browser-trusted TLS
one controlled entry point
verifiable behavior end-to-end

That’s the kind of engineering I want to keep doing: systems that work—and deserve trust.

DEV Community: Rahim Ranxx

Django + Celery + Redis Sentinel: A Real Failover Test (With Metrics)

Here’s what actually happened.

Table of Contents

Architecture Overview

Stack Components

Sentinel Integration (Django + Celery)

Observability with Prometheus

Failover Drill Walkthrough

Initial State

Induced Failure

Sentinel Election

Celery Behavior During Failover

Timeline

Observed Task

Performance Impact

Production Readiness Assessment

What Works

What Needs Attention

When This Architecture Is Production-Ready

When This Is Not Enough

How to Reduce Failover Latency

Key Takeaway

Final Thoughts

Escaping Cache Fragmentation: How Misconfigured PHP Workers Flooded My Token System

🚨 The Symptom

🔍 The Root Cause

Stack involved:

⚙️ What Went Wrong

💥 The Cascade Effect

📈 Why Observability Looked “Wrong”

✅ The Fix

Result:

🔒 Production Hardening (What I Added Next)

1. Distributed Locking

2. TTL Buffering

3. Observability Metrics

🧠 Key Takeaway

⚡ Final Insight

🔗 Closing

From 80-Second APIs to Sub-Second: Rebuilding a Geospatial Backend with Async Pipelines

From 80-Second APIs to Sub-Second: Fixing Latency with Async Pipelines (Django + Celery)

Introduction

The Original Architecture (The Hidden Problem)

What happened on each request?

Why this seemed fine

The Breaking Point

The Core Insight

The New Architecture (Async Pipeline)

Key changes

Diagram 1 — Before vs After

Before (Request-driven)

After (Pipeline-driven)

Implementation

1. Fast API Path (Non-blocking)

2. Celery Task (Async Compute)

3. Daily Backfill (Critical)

Observability (The Real Upgrade)

Metrics (Grafana Observations)

📊 Grafana Screenshots

Before

After

Before vs After Summary

Final Thought

Designing a One-Way Farm Sync Architecture (Nextcloud Django DRF)

From Nextcloud to Django: Designing a Farm Sync Architecture with DRF

The Initial Problem

Option 1: Bidirectional Sync (The Dangerous Path)

Option 2: API-First Architecture

Immediate Consistency

Clear Ownership

Extensibility

Solving the Cross-System Identity Problem

No ID Collisions

Clean Synchronization

Future Integration

Identity Translation Between Systems

Measuring the Integration Layer

Lessons Learned

Final Thoughts