DEV Community: Rahim Ranxx

Decoupled Media Streams: A Django and Nextcloud Radio Architecture

Rahim Ranxx — Mon, 25 May 2026 12:41:31 +0000

I recently added a radio integration to a platform built around Django REST Framework (DRF), and Nextcloud.

The existing architecture was already doing a lot of heavy lifting, powering authentication, farm management, NDVI processing pipelines, weather data ingestion, API key orchestration, and Nextcloud application integrations.

The new requirement was to introduce internet radio support seamlessly inside the Nextcloud ecosystem. However, there was a strict architectural constraint: we needed to do this without turning Django into a media relay.

That single distinction shaped the entire implementation strategy.

The Core Challenge: Avoiding the Proxy Trap
Instead of proxying heavy audio streams through the backend, the architecture relies on direct playback. Django is strictly responsible for exposing radio metadata and playback endpoints (routed under /api/v1/radio/). Meanwhile, the Nextcloud clients stream the audio directly from the source providers, such as BBC, SomaFM, and TuneIn.

The result is a much cleaner separation of responsibilities:

Nextcloud UI / Web Client: The presentation layer.

Django + DRF API: Radio metadata and stream information logic.

Radio Providers: Direct playback of media transport.

Architectural Separation in Action
The following diagram illustrates exactly how we achieved this decoupling. The critical path is that thick dark orange arrow (3), showing the media stream bypassing the Django API server entirely.

(Diagram: Metadata requests [blue] are routed through Django, while heavy media streams [orange] flow directly from providers like BBC/SomaFM to the Nextcloud user.)

This separation keeps the backend highly performant and lightweight, while allowing the Nextcloud frontend to integrate radio discovery naturally alongside the rest of the platform's services.

How Nextcloud Fits Into the Architecture
The radio integration was explicitly designed to plug into a broader, Nextcloud-driven ecosystem rather than operating as an isolated, standalone media application. By defining strict boundaries, each system handles what it does best.

Nextcloud provides:

The frontend user experience

Authenticated user workflows

App integration surfaces and dashboard presentation

Native media interaction capabilities

Django provides:

API orchestration and provider abstraction

Station metadata and stream discovery

Data normalization logic

Backend consistency

This clear separation creates a strong boundary between backend platform orchestration and frontend client experience. Instead of embedding complex streaming logic directly into Nextcloud—or forcing Django to waste resources proxying media—the architecture keeps each layer focused entirely on its primary responsibility.

Built for Future Expansion
Because the backend already behaves like a pure metadata platform rather than a streaming server, the architecture leaves massive room for future expansion.

Without needing to redesign the streaming layer itself, this setup easily supports adding:

Personalized stations and user favorites

Listening history tracking

Podcast aggregation

Recommendation systems

Analytics pipelines

Multi-provider federation

By treating media transport and metadata orchestration as two distinct problems, the integration remains scalable, fast, and ready for whatever features the platform requires next.

Debugging a Cross-Language HMAC Signature Failure Between Nextcloud and Django

Rahim Ranxx — Sat, 16 May 2026 14:47:34 +0000

Introduction

A few days ago, I hit a frustrating issue while integrating a custom Nextcloud application with a Django REST Framework backend.

Everything looked correct:

shared HMAC secret ✔️
canonical request string ✔️
HMAC-SHA256 ✔️
timestamps synchronized ✔️

Yet every authenticated request failed with:

invalid nextcloud signature

The interesting part?

Both implementations were technically correct.

The failure came from something much smaller — and much more dangerous in distributed systems:

Different string encodings of the exact same HMAC digest.

This article walks through the full debugging process, the root cause, and the engineering lessons learned from debugging cryptographic interoperability between PHP and Python services.

System Architecture

The integration architecture looked like this:

┌──────────────────────┐
│  Nextcloud App (PHP) │
│  Generates HMAC      │
└──────────┬───────────┘
           │
           │ Signed HTTP Request
           ▼
┌──────────────────────┐
│ Django DRF Backend   │
│ Verifies Signature   │
└──────────────────────┘

The request flow:

Nextcloud generates a canonical request string
PHP computes an HMAC-SHA256 signature
Signature is attached to request headers
Django reconstructs the canonical string
Django recomputes the HMAC
Signatures are compared

Simple in theory.

Except it kept failing.

Initial Symptoms

The backend logs showed repeated authorization failures:

nextcloud_hmac.denied
code=invalid_signature

Even more confusing:

the integration had worked before
secrets matched
clocks matched
payloads matched

At first glance, it looked like a replay issue, timestamp skew problem, or cache corruption.

It turned out to be none of those.

The Root Cause

The issue came from a mismatch in how the HMAC digest was encoded.

Nextcloud (PHP)

The PHP client generated the signature like this:

base64_encode(
    hash_hmac('sha256', $canonical, $secret, true)
);

Notice the important detail:

true

That parameter returns the raw digest bytes.

Those bytes were then encoded as Base64.

Django (Python)

Meanwhile, Django verified signatures like this:

hmac.new(
    secret,
    canonical.encode(),
    hashlib.sha256,
).hexdigest()

hexdigest() returns a hexadecimal string representation.

So both systems produced:

the same HMAC bytes
using the same algorithm
using the same secret

But converted those bytes into different string formats.

The Hidden Interoperability Bug

This was the breakthrough moment.

The exact same digest bytes produced:

Hex:
44c39c4ecc7268547ca51db72c6f27125251e6ea8ce3c659d918a9542522b612

Base64:
RMOcTsxyaFR8pR23LG8nElJR5uqM48ZZ2RipVCUithI=

Both values represent the same underlying bytes.

But string comparison obviously fails.

The Second Bug

While investigating, I found another subtle issue.

The Django verifier lowercased the incoming signature before comparison:

signature = signature.lower()

That may appear harmless for hexadecimal values.

But Base64 is case-sensitive.

Meaning:

ABC != abc

So even after fixing the encoding mismatch, lowercasing would still break verification.

This was a protocol normalization bug hiding inside the verification pipeline.

The Fix

I updated Django to verify signatures using Base64 instead of hexadecimal.

New Verification Function

import base64
import hashlib
import hmac


def compute_hmac_signature_b64(
    *,
    secret: bytes,
    canonical_string: str,
) -> str:
    """Compute Base64 encoded HMAC-SHA256 signature."""

    digest = hmac.new(
        secret,
        canonical_string.encode("utf-8"),
        hashlib.sha256,
    ).digest()

    return base64.b64encode(digest).decode()

Then all verification calls were updated to use:

compute_hmac_signature_b64()

instead of:

.hexdigest()

Finally, I removed:

.lower()

from the verification flow.

Verification Results

After deploying the fix:

Ping Endpoint

GET /api/v1/integrations/nextcloud/ping/

200 OK

Token Issuance

POST /api/v1/integrations/token/

200 OK

Authentication immediately started working again.

Secondary Investigation Findings

While debugging, I validated several other production concerns.

1. Time Drift

I suspected clock skew initially.

Both services were checked:

Nextcloud epoch: 1778841776
Django epoch:    1778841776
Drift:            0 seconds

Time synchronization was perfect.

2. Shared Secrets

Client IDs and secrets matched correctly across both systems.

This eliminated:

environment mismatch
stale secrets
config drift

3. Redis and Cache State

I flushed:

Redis
Django cache
integration token caches

This helped eliminate stale token artifacts and replay-state inconsistencies.

4. Infrastructure Validation

I also verified:

loopback networking
gunicorn binding
uvicorn workers
allowlists
HTTP dev mode configuration

At this point the investigation became less about cryptography and more about systematic elimination of variables.

Why It “Worked Before”

This was the most interesting systems question.

I had not changed the signing logic recently.

So why did the failure suddenly appear?

The likely answer is:

Infrastructure state had been masking a latent protocol incompatibility.

Possible contributors:

cached tokens
stale replay windows
inactive code paths
existing sessions bypassing verification
Redis persistence behavior

This is an important engineering lesson:

A system can contain dormant interoperability bugs for weeks before infrastructure conditions expose them.

Engineering Lessons Learned

1. Cryptographic Bytes ≠ String Representation

HMAC output is binary data.

Hexadecimal and Base64 are merely different textual encodings of the same bytes.

They are not interchangeable.

2. Cross-Language Integrations Need Explicit Contracts

Never assume:

encoding format
canonicalization rules
normalization behavior

Define them explicitly.

Especially across:

PHP
Python
Go
Node.js
Java

3. Normalization Can Break Security

Lowercasing signatures looked harmless.

It was not.

Cryptographic values should only be normalized if the protocol explicitly defines normalization behavior.

4. Infrastructure State Can Hide Bugs

Cache layers and token persistence can temporarily conceal protocol inconsistencies.

Sometimes:

restarts
cache flushes
clock resets

suddenly expose issues that already existed.

5. Production Debugging Requires Elimination Discipline

The investigation involved validating:

clocks
secrets
caches
workers
networking
encoding
replay protection
request canonicalization

Good debugging is often less about guessing and more about systematically removing uncertainty.

Final Thoughts

The most dangerous bugs are not always algorithm failures.

Sometimes:

the crypto is correct
the infrastructure is healthy
the logic is valid

…but the protocol contract between systems is inconsistent.

In this case:

The cryptography was correct on both sides. The protocol contract was not.

And that single mismatch was enough to break the entire authentication flow.

Why I Added Redis Streams Between My Django API and Celery Workers.

Rahim Ranxx — Sun, 03 May 2026 07:01:10 +0000

A practical engineering breakdown of how I introduced Redis Streams into a live Django + Celery NDVI pipeline without rewriting the worker layer.

Introduction

I run a Django API backed by Celery workers for NDVI processing workloads.

The execution layer worked fine.

The queue semantics didn’t.

I needed:

durable ingestion
replay visibility
dead-letter handling
stale consumer recovery
rollback safety
observability during incidents

…but I did not want to rewrite the worker system or destabilize production.

So instead of replacing Celery, I inserted Redis Streams between the API and the workers.

This article explains why I made that decision, how the architecture works, and what I learned while implementing reliable stream-backed NDVI ingestion in Django.

The Original Problem

The problem was not task execution.

The problem was everything before execution.

Originally, NDVI ingestion looked like this:

Django API → Celery Broker → Celery Workers

At first, this worked well.

But as the system evolved, operational gaps became more obvious:

Direct .delay() calls tightly coupled request ingestion to broker behavior.
Queue visibility was limited during incidents.
Failed ingestion paths were harder to replay safely.
In-flight recovery semantics were weak.
There was no dead-letter workflow for poisoned messages.
Worker interruptions could leave messages in uncertain states.

The architecture was fast.

It was not durable enough.

Why I Did Not Replace Celery

One of the biggest architectural decisions was choosing not to replace Celery.

That decision reduced risk dramatically.

Celery already handled:

worker orchestration
task retries
execution concurrency
scheduling
routing
operational familiarity

Replacing the worker layer would have increased migration complexity and expanded the failure domain.

Instead, I treated Redis Streams as an ingestion and reliability layer.

The resulting architecture looked like this:

Django API
    ↓
Dispatch Boundary
    ↓
Redis Streams (XADD)
    ↓
Consumer Group (XREADGROUP)
    ↓
Celery Queue
    ↓
NDVI Workers

Failures route into a dead-letter stream.

Stale consumers are recovered through reclaim logic.

Most importantly, rollback remains simple.

Centralizing Dispatch Before Adding Redis Streams

Before introducing Redis Streams, I centralized every NDVI enqueue path.

This was the most important migration step.

Instead of scattering direct .delay() calls across the codebase, everything flowed through dispatch helpers.

from ndvi.dispatch import dispatch_ndvi_job

job = enqueue_job(...)
dispatch_ndvi_job(job)

That allowed one configuration flag to control the ingestion backend.

NDVI_QUEUE_BACKEND = env("NDVI_QUEUE_BACKEND", default="celery")

Supported modes:

celery
stream

This created a clean migration boundary.

The system could switch between direct Celery dispatch and Redis Streams without changing every call site.

Operationally, this mattered more than the stream code itself.

Publishing NDVI Jobs into Redis Streams

The producer layer publishes deterministic NDVI payloads into a Redis stream.

Example:

payload = {
    "job_id": job.id,
    "request_hash": job.request_hash,
    "farm_id": job.farm_id,
    "engine": job.engine,
    "job_type": job.job_type,
    "enqueue_timestamp": time.time(),
}

redis_client.xadd(
    settings.NDVI_STREAM_NAME,
    payload,
    maxlen=settings.NDVI_STREAM_MAXLEN,
    approximate=True,
)

Key design decisions:

request_hash acts as the idempotency key.
XTRIM keeps memory bounded.
Stream payloads remain deterministic.
Producers do not execute business logic.

The stream became the ingestion ledger.

Redis Streams Consumer Design

The consumer reads from Redis Streams and forwards work into Celery.

Example:

messages = redis_client.xreadgroup(
    groupname=settings.NDVI_STREAM_GROUP,
    consumername=consumer_name,
    streams={settings.NDVI_STREAM_NAME: ">"},
    count=settings.NDVI_STREAM_BATCH_SIZE,
    block=settings.NDVI_STREAM_BLOCK_MS,
)

For every message:

Deserialize payload
Validate structure
Apply idempotency safeguards
Enqueue Celery task
Acknowledge stream entry

process_ndvi_job.delay(job_id)

redis_client.xack(
    settings.NDVI_STREAM_NAME,
    settings.NDVI_STREAM_GROUP,
    message_id,
)

The stream consumer remains intentionally thin.

Its job is reliable transport and recovery.

Celery still handles execution.

Why Consumer Groups Matter

Redis Streams consumer groups solved several operational problems immediately.

They provided:

cooperative work distribution
independent consumer identities
pending-entry tracking
reclaim support
replay visibility

Unlike simple queue semantics, Redis Streams expose message lifecycle state.

That visibility becomes extremely valuable during failures.

Message lifecycle:

XADD → pending → reclaimed → acknowledged
                          ↓
                         DLQ

This made queue recovery observable instead of implicit.

Recovering Stale Messages with XAUTOCLAIM

The most important recovery primitive ended up being XAUTOCLAIM.

If a consumer dies after reading a message but before acknowledging it, the entry remains pending indefinitely unless another consumer reclaims it.

Without reclaim logic, stream durability is incomplete.

Example reclaim loop:

messages = redis_client.xautoclaim(
    name=settings.NDVI_STREAM_NAME,
    groupname=settings.NDVI_STREAM_GROUP,
    consumername=consumer_name,
    min_idle_time=settings.NDVI_STREAM_CLAIM_IDLE_MS,
    start_id="0-0",
    count=settings.NDVI_STREAM_BATCH_SIZE,
)

This allows healthy consumers to recover abandoned work automatically.

That changed the reliability profile of the ingestion pipeline significantly.

Dead-Letter Queue Handling

I also introduced a dedicated dead-letter stream.

Messages are routed into the DLQ when:

validation fails
delivery ceilings are exceeded
payloads become structurally invalid
repeated execution attempts fail

Example:

redis_client.xadd(
    settings.NDVI_STREAM_DLQ_NAME,
    dlq_payload,
)

Every DLQ entry includes:

original message ID
delivery count
failure reason
serialized payload
timestamps

This made operational debugging dramatically easier.

The Hardest Problem: Idempotency

Redis Streams provide at-least-once delivery.

That means duplicate delivery is expected.

Exactly-once delivery is not guaranteed.

To prevent duplicate NDVI execution, I added multiple protection layers.

Layer 1: Deterministic Request Hash

Every NDVI job already had a deterministic request_hash.

That became the execution identity.

Layer 2: Distributed Redis Lock

The consumer acquires a Redis lock before execution.

lock_key = f"ndvi:lock:{request_hash}"

Acquisition uses SETNX semantics with expiration.

Layer 3: Token-Based Lock Release

Locks are released through an atomic Lua script.

This prevents blind deletion.

if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
end
return 0

Layer 4: Database Status Recheck

Before execution begins, the worker re-checks terminal job state.

This acts as a second safety boundary.

The result is effectively-once execution semantics.

At-least-once delivery + idempotent execution = effectively-once processing

Observability Added During the Rollout

One major lesson from this migration:

Do not enable stream mode before queue visibility exists.

I added dedicated metrics before enabling the rollout broadly.

Examples:

redis_stream_pending_entries
redis_stream_pending_age_max
ndvi_stream_consumer_heartbeat
ndvi_stream_consumer_failures_total

I also expanded upstream visibility:

ndvi_upstream_requests_total
ndvi_upstream_failures_total
ndvi_upstream_duration_seconds

Grafana dashboards now expose:

pending stream backlog
reclaim frequency
DLQ volume
consumer liveness
upstream API failures
queue drain rate

This transformed rollout decisions from guesswork into measurable operations.

Rollback Strategy

Rollback was designed before rollout.

That mattered.

The stream backend is fully feature-flagged:

NDVI_QUEUE_BACKEND = "celery"

NDVI_QUEUE_BACKEND = "stream"

Rollback requires:

environment variable change
process restart

No redeploy.

No task rewrite.

No schema rollback.

This significantly reduced operational fear during rollout.

What Shipped This Week

This week’s rollout included:

a 528-line Redis Streams consumer
reclaim + DLQ lifecycle handling
distributed execution locking
token-safe lock release
approximately 400 lines of stream-focused tests
Prometheus metrics for queue health
Grafana visibility for consumer state and lag
feature-flag rollback support

Most of the work was not adding Redis.

Most of the work was making failure recovery predictable.

Was It Worth It?

Redis Streams did not simplify the system.

They made failure states explicit.

That introduced additional complexity:

reclaim logic
idempotency handling
consumer lifecycle management
DLQ operations
stream observability

But the reliability gains were substantial:

durable ingestion
replay visibility
safer recovery semantics
backlog introspection
controlled rollback
observable queue state

For this NDVI pipeline, the tradeoff was worth it.

Final Thoughts

One of the biggest lessons from this migration is that queue evolution is not just about throughput.

It is about operational recovery.

Redis Streams gave the ingestion layer explicit lifecycle semantics:

pending
acknowledged
reclaimed
dead-lettered

That visibility fundamentally changed how the system behaves during failures.

And importantly, I achieved that without rewriting the worker layer.

Sometimes the best migration strategy is not replacing your stack.

It is inserting a safer boundary in front of it.

Building a Resilient NDVI Pipeline with Redis Streams (Event-Driven Architecture)

Rahim Ranxx — Sun, 26 Apr 2026 09:02:14 +0000

A practical breakdown of moving an NDVI processing pipeline from a synchronous design to an event-driven architecture using Redis Streams — including concurrency challenges, distributed locking pitfalls, and production-safe patterns.

Introduction

Most pipelines work — until concurrency and failure expose their limits.

At first, processing NDVI (Normalized Difference Vegetation Index) data seems straightforward:

receive a request

process imagery

return results

But once you introduce:

concurrent jobs

long-running processing

distributed components

you’re no longer building a simple pipeline.

You’re designing a distributed system.

This article walks through how I transformed an NDVI processing pipeline from a synchronous model into an event-driven architecture using Redis Streams, and the real-world engineering challenges that came with it.

System Overview

The system is built using:

Django REST Framework (backend API)

Nextcloud (client-facing integration layer)

Celery (asynchronous task processing)

Redis Streams (event ingestion and coordination)

The Initial Architecture (Synchronous Design)

Client → API → Celery Task → NDVI Processing → Result

This design works well at small scale, but it introduces hidden risks when the system grows.

The Core Problems

Tight Coupling

The request lifecycle is directly tied to processing.

If processing fails:

the request fails

the user experiences errors

retries become difficult

Concurrency Issues

When multiple requests target the same job:

Request A ─┐
├──> Same Job → Duplicate Processing
Request B ─┘

This leads to:

duplicated work

inconsistent outputs

race conditions

Fragile Execution Model

Without coordination:

jobs execute immediately

no buffering exists

failure handling is reactive, not controlled

The Shift to Event-Driven Architecture

To solve these issues, I introduced Redis Streams and redesigned the system into an event-driven model.

New Architecture (Event-Driven Pipeline)

Client → API → Redis Stream → Consumer → Celery → Processing

Why Redis Streams?

Redis Streams provide:

Event buffering (decouples ingestion from execution)

At-least-once delivery (ensures reliability)

Ordered processing

Scalability for distributed systems

What Changed

Instead of executing tasks immediately:

The API publishes events to a Redis Stream

A stream consumer controls task execution

Celery workers process jobs asynchronously

This separates:

ingestion

scheduling

execution

Distributed Locking: The Critical Bug

To prevent duplicate processing, a locking mechanism was introduced.

The naive approach:

cache.delete(lock_key)

This looks harmless — but in distributed systems, it’s dangerous.

Why This Fails

Consider this sequence:

Process A acquires a lock
The lock expires
Process B acquires the same lock
Process A deletes the lock

Now:

Process B is running without protection

This creates a race condition — one of the hardest problems in distributed systems.

The Fix: Token-Based Distributed Locking

To solve this, each lock is assigned a unique token.

SET lock_key = token_A (TTL)

Release only if:
stored_token == token_A

Key Principles

Only the owner of the lock can release it

If ownership does not match → do nothing

TTL ensures eventual cleanup

This ensures:

safe concurrency

no accidental unlocks

predictable system behavior

Stream Consumer Design

Redis Streams operate with:

At-least-once delivery semantics

This means:

messages can be delivered more than once

consumers must be idempotent

Consumer Processing Flow

Read → Validate → Enqueue → Acknowledge

Critical Rule

Never acknowledge a message before it is safely enqueued.

Idempotency and Reliability

To handle duplicate events:

processing must be idempotent

tasks must tolerate retries

state transitions must be safe

This is essential in any event-driven system.

Final Architecture (Layered System Design)

The system now operates in clear layers:

Ingestion Layer

receives requests

publishes events

Stream Layer

buffers and orders events

decouples system components

Consumer Layer

controls execution

validates and dispatches tasks

Execution Layer

Celery workers process NDVI jobs

Coordination Layer

distributed locking

idempotency

concurrency control

Key Lessons from Building an Event-Driven System

Event-Driven Architecture Does Not Reduce Complexity

It shifts complexity into:

coordination

state management

failure handling

Concurrency Is the Real Challenge

Not performance.
Not frameworks.

Concurrency.

Safety Must Be Designed Explicitly

Small shortcuts (like naive lock deletion) can lead to major production issues.

Idempotency Is Non-Negotiable

In systems with retries and event delivery:

duplicate execution is expected

safe handling is required

Observability Becomes Critical

In asynchronous systems, you must answer:

“What happened to this job?”

This requires:

structured logging

tracing across components

visibility into system flow

Conclusion

This shift changed the system from:

"Run this task now"

to:

"This event will be processed safely"

That difference is fundamental.

Because in distributed systems:

You don’t design for success.
You design for failure.

What’s Next

The next phase is observability-driven engineering:

tracing event lifecycles

monitoring stream lag

correlating logs across services

Because once a system becomes event-driven:

Visibility is what makes it understandable.

Hardening Distributed Systems: Retries, Circuit Breakers & Observability.

Rahim Ranxx — Sun, 12 Apr 2026 05:12:28 +0000

Building Resilient Distributed Systems: A Solo Engineer's Journey

How I turned flaky upstream APIs into a predictable, observable, and operator-friendly reliability layer — with code you can steal.

Introduction

If you've ever built a service that depends on external APIs (STAC catalogs, SentinelHub, weather data providers, etc.), you know the pain:

429s when you hit rate limits
502s when upstreams hiccup
Silent timeouts that leave jobs hanging
Retry storms that make bad days worse

Last month, I undertook a focused effort to harden the retry and resilience logic for an NDVI (Normalized Difference Vegetation Index) processing pipeline. What started as "let's clean up some duplicate retry code" evolved into a production-grade reliability subsystem that now governs every upstream interaction.

In this article, I'll walk through:

Phase 1: Consolidating retry policy into a single source of truth
Phase 2: Adding circuit breakers with observability and admin controls
Phase 3 (preview): Decoupling dispatch with Redis Streams for back-pressure resilience
Key principles I learned that you can apply to your own distributed systems

All code is Python/Django/Celery, but the patterns are language-agnostic. And yes — I did this alone. No team, no dedicated SRE, no platform squad. Just me, a codebase, and a lot of careful thinking.

The Problem Space

The NDVI pipeline I was working on orchestrates vegetation index calculations by:

Querying STAC catalogs for satellite imagery metadata
Fetching raster data from SentinelHub
Computing NDVI values per farm/plot
Returning results to farmers/agronomists

The challenge: Each upstream service has different failure modes:

STAC: occasional 502s, auth errors (401/403)
SentinelHub: strict rate limits (429), validation errors (422), transient 5xx
Network: timeouts, DNS failures, TLS handshake issues

Before my refactor, retry logic was scattered across 4+ modules, with inconsistent error classification and no centralized observability. Result? Hard-to-debug failures, wasted Celery retries, and on-call pages at 3 AM.

As a solo engineer, I couldn't afford to keep firefighting. I needed a system that would just work — or fail gracefully, with clear signals.

Phase 1: One Source of Truth for Retries

The Core Insight

Not all errors are retryable. Not all retries are equal.

I started by defining a canonical truth table mapping HTTP status codes to retry behavior:

# ndvi/retry_policy.py
def classify_status_code(status_code: int | None) -> RetryClassification:
    """
    Canonical truth table: HTTP status → retry decision.

    | Status      | Retryable | Category           |
    |-------------|-----------|--------------------|
    | 401, 403    | False     | AUTH               |
    | 400, 422    | False     | VALIDATION         |
    | 429         | True      | RATE_LIMIT         |
    | >= 500      | True      | TRANSIENT_UPSTREAM |
    | Other/None  | False     | UNKNOWN            |
    """
    if status_code in (401, 403):
        return RetryClassification(retryable=False, category="AUTH")
    if status_code in (400, 422):
        return RetryClassification(retryable=False, category="VALIDATION")
    if status_code == 429:
        return RetryClassification(retryable=True, category="RATE_LIMIT")
    if status_code is not None and status_code >= 500:
        return RetryClassification(retryable=True, category="TRANSIENT_UPSTREAM")
    return RetryClassification(retryable=False, category="UNKNOWN")

Unified Exception Hierarchy

I made all upstream errors inherit from a common base, ensuring consistent attributes:

class UpstreamFailureError(NdviFailureError):
    """Base for all retryable upstream failures."""
    def __init__(self, message: str, status_code: int | None = None, response: Response | None = None):
        super().__init__(message)
        self.status_code = status_code
        self.response = response
        # Delegate to canonical classifier
        classification = classify_status_code(status_code)
        self.retryable = classification.retryable
        self.category = classification.category
        self.delay = self._compute_delay(classification)

class StacUpstreamError(UpstreamFailureError, StacError): ...
class SentinelHubUpstreamError(UpstreamFailureError): ...
class SentinelHubRasterError(UpstreamFailureError): ...

Centralized Retry Decision

@dataclass
class RetryDecision:
    retry: bool
    delay: float
    reason: str

def should_retry(exc: Exception, response_headers: dict | None = None) -> RetryDecision:
    if not isinstance(exc, UpstreamFailureError):
        return RetryDecision(retry=False, delay=0.0, reason="non-retryable-exception")

    # Respect Retry-After header for 429s
    if exc.status_code == 429 and response_headers:
        server_delay = parse_retry_after(response_headers.get("Retry-After"))
        if server_delay is not None:
            return RetryDecision(retry=True, delay=server_delay, reason="retry-after-header")

    return RetryDecision(
        retry=exc.retryable,
        delay=exc.delay,
        reason=f"{exc.category}-classification"
    )

Impact

28 parametrized tests covering all 13 truth-table branches
Removed 3 duplicate retry implementations
Celery tasks now use shared should_retry() logic
Network errors properly wrapped → no more silent failures

Lesson #1: Centralize failure classification. When retry logic lives in one place, you can test it thoroughly, document it clearly, and evolve it safely — even when you're the only one maintaining it.

Phase 2: Circuit Breakers with Teeth

Retries alone aren't enough. When an upstream is truly down, you want to fail fast and avoid thundering herds.

The Circuit Breaker State Machine

I implemented a simple but effective three-state breaker:

CLOSED → (failures ≥ threshold) → OPEN → (timeout elapsed) → HALF_OPEN → (success) → CLOSED
                              ↘ (failure) ↗

class _CircuitBreaker:
    def __init__(self, threshold: int = 3, timeout_secs: float = 300):
        self.state = "CLOSED"
        self.failure_count = 0
        self.last_failure_time: float | None = None
        self.threshold = threshold
        self.timeout_secs = timeout_secs

    def record_success(self):
        self.failure_count = 0
        self._transition_to("CLOSED")

    def record_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.threshold:
            self._transition_to("OPEN")

    def allow_request(self) -> bool:
        if self.state == "CLOSED":
            return True
        if self.state == "OPEN":
            if time.time() - self.last_failure_time >= self.timeout_secs:
                self._transition_to("HALF_OPEN")
                return True
            return False
        # HALF_OPEN: allow one probe request
        return True

    def _transition_to(self, new_state: str):
        old_state = self.state
        self.state = new_state
        logger.info(f"Circuit breaker: {old_state} → {new_state}")
        # Export Prometheus metric
        circuit_breaker_state.labels(engine=self.engine).set(STATE_VALUES[new_state])
        circuit_breaker_transitions.labels(
            engine=self.engine, from_state=old_state, to_state=new_state
        ).inc()

Observability First

I didn't just build the breaker — I made it visible:

# Prometheus metrics
circuit_breaker_state = Gauge(
    'ndvi_circuit_breaker_state',
    'Current circuit breaker state (0=CLOSED, 1=OPEN, 2=HALF_OPEN)',
    labelnames=['engine']
)

circuit_breaker_transitions = Counter(
    'ndvi_circuit_breaker_transitions_total',
    'Count of circuit breaker state transitions',
    labelnames=['engine', 'from_state', 'to_state']
)

And added a Grafana dashboard with:

Stat panels showing current state per engine (color-coded: 🟢 CLOSED, 🔴 OPEN, 🟡 HALF_OPEN)
Time series of transition rates
Correlation with upstream failure rates

Operator Controls

Because things will go wrong — and when you're solo, you are the operator — I added an admin endpoint to manually reset breakers:

POST /api/v1/ndvi/circuit-breaker/reset/
Content-Type: application/json
Authorization: Bearer <admin-token>

{ "engine": "stac" }

→ { "data": { "previous_state": "OPEN", "new_state": "CLOSED" } }

Lesson #2: Resilience patterns need observability and escape hatches. If you can't see it or control it, you don't own it — and when you're the only one on call, "owning it" means sleeping at night.

Phase 3 Preview: Decoupling with Redis Streams

As I scaled the system, I hit a new challenge: Celery broker unavailability during Redis Sentinel failover (~55 seconds of downtime). For background jobs, this was acceptable. But for real-time dispatch, I needed better.

The Architecture Decision

Instead of relying on Celery's built-in Redis transport, I chose a separate consumer pattern:

API → [Redis Stream] → Consumer → [Celery Queue] → Worker

Why?

Avoids Celery/Kombu stream support uncertainty
Easier to observe and debug (explicit XREADGROUP/XACK)
Natural back-pressure via XPENDING monitoring
Cleaner rollback path (just flip a feature flag)

Key Design Decisions I Made Early

1. Idempotency by Design

stream_payload = {
    "job_id": job.id,
    "request_hash": job.request_hash,  # Primary idempotency key
    "schema_version": 1,                # Future-proofing
    "colormap_normalization": "histogram",  # Evolved schema
    # ... other fields
}
# Consumer checks request_hash before enqueueing to Celery

2. Error Classification at Consumer Boundary

Not all failures should retry:

ERROR_STRATEGY = {
    "no_items": "DLQ",           # Permanent: no data exists
    "missing_assets": "DLQ",     # Permanent: schema mismatch
    "network_timeout": "RETRY",  # Transient: try again
    "celery_unavailable": "RETRY_WITH_BACKOFF",  # Infrastructure blip
}

3. Back-Pressure Strategy

PENDING_WARNING = 1_000
PENDING_CRITICAL = 5_000

pending_count = redis.xpending(stream_name, group_name)["pending"]
if pending_count > PENDING_CRITICAL:
    # Return 429 on API to slow producers
    return HttpResponseTooManyRequests("Upstream backlog critical")
elif pending_count > PENDING_WARNING:
    logger.warning(f"Stream backlog growing: {pending_count} pending")

4. Graceful Shutdown

# In consume_ndvi_stream.py
signal.signal(signal.SIGTERM, handle_shutdown)

def handle_shutdown(signum, frame):
    shutdown_flag.set()  # Stop accepting new entries
    # Finish current entry, XACK if successful
    # Exit cleanly → orchestrator restarts

Lesson #3: Decoupling isn't just about scalability — it's about failure isolation. When one component fails, the rest can keep moving. And when you're solo, isolation means you can debug one piece without bringing down the whole system.

Principles I Learned (That You Can Steal)

1. Make Failure Explicit

Don't hide errors behind generic exceptions. Classify them, tag them, and route them intentionally. Your future self — especially at 3 AM — will thank you.

2. Observability Is a Feature, Not an Afterthought

If you can't measure it, you can't improve it. Export metrics at the point of decision (retry? circuit open? stream lag?) — not just at the edges. When you're the only one debugging, every metric is a lifeline.

3. Design for the "Boring" Failure Modes

Everyone plans for the 500 error. Few plan for:

Broker failover latency
Consumer restart mid-processing
Schema evolution mid-deploy
Clock skew in distributed timestamps

Document these. Test them. Build escape hatches. When you don't have a team to lean on, preparation is your best defense.

4. Centralize, Then Specialize

Start with a single source of truth (like classify_status_code()). Then layer on engine-specific behavior on top of that foundation. This prevents drift and duplication — critical when you're the only one maintaining the code.

5. Operator Experience Matters

Admin endpoints, health checks, clear logs, and meaningful metrics aren't "nice to have" — they're what let you sleep at night. Build them in from day one. When you're solo, you are the operator.

A Note on Solo Engineering

Working alone doesn't mean working in isolation. I leaned heavily on:

Public documentation: Google SRE book, AWS Well-Architected, Martin Fowler's patterns
Open source: Studying how Celery, Kombu, and Redis clients handle resilience
Community: Reading post-mortems, blog posts, and conference talks from engineers who've been there

And I documented everything. Not for a team — for my future self. Every architecture decision, every tradeoff, every "why" is written down. Because six months from now, I won't remember why I chose 300s for the circuit breaker timeout. But my docs will.

If you're also building alone: you're not behind. You're just optimizing for a different constraint. Depth over breadth. Clarity over velocity. Resilience over features.

Conclusion

Building resilient distributed systems isn't about fancy algorithms or cutting-edge tools. It's about discipline: clear contracts, explicit failure handling, observable behavior, and operator empathy.

The NDVI pipeline I built isn't perfect. My circuit breakers are still process-local (not cluster-wide). My stream consumer doesn't yet support distributed tracing. But it's predictable, testable, and recoverable — and that's what matters.

If you take one thing from this article, let it be this:

Resilience isn't a feature you add at the end. It's a mindset you build in from the start — whether you're on a team of 50 or flying solo.

All code examples are simplified for clarity; production versions include additional error handling and logging. This work reflects my personal approach — your mileage may vary, and that's okay.

💡 Pro Tip: Want to try the circuit breaker pattern? Start small:

Add a failure_count and last_failure_time to your HTTP client

Skip requests when failure_count >= 3 and time_since_failure < 300

Log state transitions

Add one Prometheus gauge

You'll be 80% of the way there — and you'll learn what actually matters for your workload.

Django + Celery + Redis Sentinel: A Real Failover Test (With Metrics)

Rahim Ranxx — Sat, 04 Apr 2026 17:36:44 +0000

Redis Sentinel + Celery Failover: What Actually Happens in Production

Most tutorials on Redis Sentinel stop at “it elects a new master”.
Very few show what happens to a real system under failover pressure.

I ran a failover drill on a Django + Celery stack backed by Redis Sentinel and Prometheus monitoring.

Here’s what actually happened.

Architecture Overview
Sentinel Integration (Django + Celery)
Observability with Prometheus
Failover Drill Walkthrough
Celery Behavior During Failover
Performance Impact
Production Readiness Assessment
How to Reduce Failover Latency

Architecture Overview

flowchart LR
    Client --> Django
    Django -->|Cache| Sentinel
    Django -->|Tasks| Celery
    Celery -->|Broker| Sentinel
    Celery -->|Result Backend| Sentinel

    Sentinel --> RedisMaster
    Sentinel --> RedisReplica1
    Sentinel --> RedisReplica2

    Prometheus --> RedisExporter
    RedisExporter --> Sentinel

Stack Components

Django → Redis cache via Sentinel
Celery → Broker + result backend via Sentinel
Redis Sentinel → High availability + failover
Prometheus + redis_exporter → Monitoring

Sentinel Integration (Django + Celery)

All services were switched to Sentinel using environment configuration:

REDIS_ADDR=redis://host.docker.internal:26379

Validation steps:

Django cache → successful round-trip
Celery broker → connected via Sentinel
Celery result backend → SentinelBackend initialized
Test suite passed:

  pytest tests/test_settings_redis_sentinel.py

At this stage, the system is fully Sentinel-aware

Observability with Prometheus

After pointing redis_exporter to Sentinel:

Key metrics exposed:

redis_sentinel_master_status
redis_sentinel_master_ok_sentinels
redis_sentinel_master_ok_slaves
redis_sentinel_masters

Verification:

redis_instance_info{redis_mode="sentinel", tcp_port="26379"}

This confirms monitoring is tracking cluster state, not a single node.

Failover Drill Walkthrough

Initial State

flowchart LR
    Sentinel -->|Master| Redis1["172.20.0.3:6379"]
    Sentinel --> Redis2["Replica"]
    Sentinel --> Redis3["Replica"]

Prometheus reported:

master_address="172.20.0.3:6379"

Induced Failure

Current master was stopped manually

Sentinel Election

flowchart LR
    Sentinel -->|New Master| Redis2["172.20.0.2:6379"]
    Sentinel --> Redis3["Replica"]
    Sentinel --> Redis1["Down"]

New master elected on first poll
Prometheus updated on next scrape

Failover was immediate and correct

Celery Behavior During Failover

Timeline

sequenceDiagram
    participant App as Django App
    participant Celery
    participant Sentinel
    participant Redis

    App->>Celery: Submit Task
    Celery->>Redis: Send to Master
    Redis-->>Celery: Connection Lost

    Sentinel->>Sentinel: Elect New Master

    Celery->>Sentinel: Retry Connection
    Note over Celery: ~54.7s delay

    Celery->>Redis: Reconnect to New Master
    Redis-->>Celery: OK

    Celery-->>App: Task SUCCESS

Observed Task

Task ID: 9b57ba3b-a707-4c13-9255-d74de411b64b
Status during failover: PENDING
Delay: ~54.7 seconds
Final state: SUCCESS

Performance Impact

Phase	Behavior
Normal operation	Immediate execution
During failover	~55s delay
Post-recovery	Normal

Production Readiness Assessment

What Works

Redis Sentinel failover is reliable
Prometheus reflects cluster changes correctly
Django cache survives failover
No task loss in Celery

What Needs Attention

Celery introduces significant delay during failover
Reconnection is not instantaneous

When This Architecture Is Production-Ready

Use this setup if:

Tasks are asynchronous/background
Eventual completion is acceptable
Temporary latency spikes are tolerable

When This Is Not Enough

Avoid this setup (as-is) if you need:

Real-time task execution
Sub-10s failover recovery
User-facing async operations

How to Reduce Failover Latency

To push recovery closer to 10–15 seconds:

Tune Celery broker retry settings
Reduce reconnect backoff intervals
Optimize worker heartbeat and visibility timeout
Re-run failover drills with timing instrumentation

Key Takeaway

Redis Sentinel ensures infrastructure recovery.
Celery determines how fast your system actually resumes work.

In this test:

Sentinel recovery: instant
Application recovery: ~55 seconds

That gap is the real engineering challenge.

Final Thoughts

If you're using Redis Sentinel with Celery:

Don’t stop at:

“Failover works.”

Measure:

“How long until my system behaves normally again?”

Because that’s what production users experience.

Escaping Cache Fragmentation: How Misconfigured PHP Workers Flooded My Token System

Rahim Ranxx — Sun, 22 Mar 2026 18:02:23 +0000

🚨 The Symptom

I started noticing something strange in my observability stack:

Integration tokens were being minted repeatedly
My token endpoint showed activity even when no user interaction was happening
Metrics suggested constant “traffic” to an otherwise idle system

At first glance, it looked like:

A security issue
A rogue client
Or a broken API consumer

It was none of those.

🔍 The Root Cause

The issue came down to a subtle but critical architectural mistake:

I was using a non-shared cache in a multi-worker environment.

Stack involved:

PHP-FPM (2 workers)
APCu (in-memory cache)
Token-based integration between services

⚙️ What Went Wrong

APCu is process-local, not shared.

That means:

Worker A cache ≠ Worker B cache

Each PHP-FPM worker had its own isolated memory.

💥 The Cascade Effect

My token logic was straightforward:

if token not in cache:
    mint_new_token()

But in reality, the system behaved like this:

Request hits Worker A → token exists → OK
Next request hits Worker B → cache miss → mint new token
Repeat across workers → continuous token regeneration

📈 Why Observability Looked “Wrong”

From the outside, it looked like traffic was hitting the token endpoint.

But in reality:

The system was generating its own traffic due to cache inconsistency.

This is a key lesson:

Not all traffic is external
Some is emergent behavior from system design

✅ The Fix

I switched from APCu to:

Redis (shared cache)

Now:

All workers → same cache → consistent token state

Result:

Tokens minted once
Reused across all workers
Metrics stabilized instantly

🔒 Production Hardening (What I Added Next)

Fixing the cache wasn’t enough — I hardened the system further.

1. Distributed Locking

To prevent race conditions:

if token exists:
    return token

acquire lock
    re-check cache
    mint token if still missing
release lock

2. TTL Buffering

Avoid edge expiration issues:

cache_ttl = token_expiry - safety_margin

3. Observability Metrics

I added:

token_cache_hits
token_cache_misses
token_mint_count

Now anomalies show up immediately.

🧠 Key Takeaway

This wasn’t just a bug.

It was a distributed systems failure mode:

Cache locality + multi-worker architecture → inconsistent state → emergent traffic

⚡ Final Insight

If your system:

Runs multiple workers
Uses in-memory caching
Relies on shared state

Then this rule applies:

If your cache isn’t shared, your state isn’t real.

🔗 Closing

This issue reinforced something critical in my engineering journey:

You don’t debug systems by staring at code —
you debug them by understanding how state flows across boundaries.

If you're building distributed APIs, token systems, or high-concurrency services —
this is one edge case worth designing for early.

From 80-Second APIs to Sub-Second: Rebuilding a Geospatial Backend with Async Pipelines

Rahim Ranxx — Sat, 21 Mar 2026 16:37:10 +0000

From 80-Second APIs to Sub-Second: Fixing Latency with Async Pipelines (Django + Celery)

Introduction

At some point, every backend engineer hits this wall:

The API works perfectly… until it doesn’t.

I hit that wall with a farm analytics endpoint computing NDVI (Normalized Difference Vegetation Index) from satellite imagery. The system was correct, the logic was sound, and the results were accurate.

But the numbers told a different story:

P95 latency: 1.25 minutes

That’s not an API. That’s a blocking compute job pretending to be one.

This is the story of how I redesigned the system—from a synchronous request-driven model to an asynchronous data pipeline—and brought latency down to sub-second performance (P95 ≈ 725ms).

The Original Architecture (The Hidden Problem)

At first glance, the system looked clean:

[Client]
   ↓
[Django API]
   ↓
[STAC API → Satellite Data]
   ↓
[Raster Processing (NDVI)]
   ↓
[Response]

What happened on each request?

Query satellite imagery via STAC
Fetch raster bands (Red & NIR) from remote storage
Process NDVI using rasterio
Aggregate coverage
Return result

Why this seemed fine

It worked locally
It returned correct data
It followed a “pure API” mindset

But under the hood:

Remote I/O (S3-backed satellite data)
Heavy raster decoding (JPEG2000)
Sequential band reads
Full computation per request

The Breaking Point

Logs told the truth.

Each request looked like:

STAC request → ~5s
Raster read (B04) → ~5–10s
Raster read (B08) → ~5–10s
Processing → ~5s+
Total → ~80+ seconds

And the key realization:

I wasn’t building an API—I was executing a geospatial compute pipeline on every request.

The Core Insight

This is the shift that changes everything:

APIs should serve data, not compute it on demand.

The problem wasn’t Python.
The problem wasn’t Django.
The problem was architecture.

The New Architecture (Async Pipeline)

I redesigned the system around asynchronous computation + caching:

             (Scheduled / Triggered)
                    ↓
             [Celery Worker]
                    ↓
         [NDVI Computation Pipeline]
                    ↓
             [Redis / Database]
                    ↓
[Client] → [Django API] → [Cache Lookup]

Key changes

NDVI computation moved out of the request path
Results cached in Redis
Background jobs compute and refresh data
API returns instantly (no heavy compute)

Diagram 1 — Before vs After

Before (Request-driven)

Request
   ↓
STAC API
   ↓
Raster I/O
   ↓
NDVI Compute
   ↓
Response (80s)

After (Pipeline-driven)

Request → Cache → Response (~725ms P95)
              ↓ (miss)
         Async Task
              ↓
       Compute + Store

Implementation

1. Fast API Path (Non-blocking)

from django.core.cache import cache
from ndvi.tasks import compute_farm_state_coverage

def get_farm_state(farm_id: int) -> dict:
    cache_key = f"farm_state:{farm_id}"

    data = cache.get(cache_key)
    if data:
        return data

    compute_farm_state_coverage.delay(farm_id=farm_id)

    return {
        "coverage_pct": None,
        "status": "processing"
    }

2. Celery Task (Async Compute)

from celery import shared_task
from django.core.cache import cache

@shared_task(bind=True, autoretry_for=(Exception,), retry_backoff=True)
def compute_farm_state_coverage(self, farm_id: int) -> None:
    coverage = compute_ndvi_coverage(farm_id)

    cache.set(
        f"farm_state:{farm_id}",
        {
            "coverage_pct": coverage,
            "status": "ready"
        },
        timeout=60 * 60 * 6,
    )

3. Daily Backfill (Critical)

from celery import shared_task

@shared_task
def enqueue_daily_farm_state_coverage():
    farm_ids = get_active_farm_ids()

    for farm_id in farm_ids:
        compute_farm_state_coverage.delay(farm_id=farm_id)

Observability (The Real Upgrade)

Metrics added:

Task duration
Task success/failure
Queue depth

Metrics (Grafana Observations)

📊 Grafana Screenshots

1. Latency Graph

Before

P95 latency: ~1.25 minutes

After

API latency: ~725ms (P95)
Background tasks: 60–90s

Before vs After Summary

Metric	Before	After
API latency	1.25 min	~725 ms (P95)
System type	Request-driven	Pipeline-driven
Scalability	Poor	Strong
Observability	Minimal	Improved

Final Thought

I stopped treating my API like a calculator and started treating my system like a data pipeline.

That’s when everything changed.

Designing a One-Way Farm Sync Architecture (Nextcloud Django DRF)

Rahim Ranxx — Sun, 15 Mar 2026 07:11:34 +0000

From Nextcloud to Django: Designing a Farm Sync Architecture with DRF

Modern applications rarely live in a single system. As projects grow, different components begin to specialize: one system handles identity and user workflows, while another focuses on computation and domain logic.

This week I explored a small but interesting distributed architecture problem: how to synchronize farm data between a Nextcloud application and a Django REST API backend.

The goal was simple in theory:

Nextcloud provides the user interface.
Django performs geospatial computation (NDVI, raster processing).
Farm data must stay consistent between the two systems.

But as any engineer knows, “simple in theory” is where architecture decisions start to matter.

The Initial Problem

In the system I’m building, users manage farms from a Nextcloud application while a Django service handles geospatial workloads.

The challenge was deciding where farm data should live and how it should propagate between systems.

Three architectural questions emerged:

Which system is the source of truth?
How do we synchronize data between services?
How do we avoid identity conflicts between systems?

These are classic distributed systems questions, even in relatively small projects.

Option 1: Bidirectional Sync (The Dangerous Path)

One tempting solution is letting both systems create farms and then syncing them.

Nextcloud <--> Django

At first glance this feels flexible. In practice it creates difficult problems:

conflicting updates
race conditions
reconciliation logic
versioning requirements

Large distributed databases solve this with vector clocks and conflict resolution strategies. For most applications, that complexity is unnecessary.

So I rejected bidirectional replication early.

Option 2: API-First Architecture

Instead of replicating farms between databases, Nextcloud simply delegates creation to the Django API.

When a user creates a farm in Nextcloud:

User
  ↓
Nextcloud UI
  ↓
Nextcloud Controller
  ↓
POST /api/v1/farms/ (Django REST Framework)
  ↓
Django database

Django becomes the source of truth for farm data, while Nextcloud acts as the interface layer.

This pattern has several advantages.

Immediate Consistency

Since farms are created directly in Django, the backend always has the latest data.

There is no delayed replication.

Clear Ownership

Each system has a defined responsibility:

Nextcloud – user interface, identity, workflow
Django – domain logic, geospatial processing, data storage

Clear boundaries reduce architectural complexity.

Extensibility

Once Django exposes a clean API, other systems can integrate easily:

mobile apps
data pipelines
satellite processing services
analytics dashboards

Everything interacts with the same API.

Solving the Cross-System Identity Problem

Once multiple systems talk about the same object, a subtle problem appears:

identity consistency.

If each system generated its own farm IDs, synchronization would become fragile:

Nextcloud Farm ID = 12
Django Farm ID = 47

Now every integration requires mapping tables.

Instead, the architecture introduces a stable external identifier.

Nextcloud generates a UUID called:

external_farm_id

That UUID is sent to Django whenever a farm is synchronized.

Conceptually the model looks like this:

Farm
 ├─ id (internal database id)
 └─ external_farm_id (shared UUID)

Now both systems reference the farm using the same identifier.

When Nextcloud syncs a farm:

POST /api/v1/farms/sync

Payload example:

{
  "external_farm_id": "uuid",
  "external_user_id": "nextcloud_uid",
  "name": "Demo Farm",
  "bbox": "...",
  "centroid": "..."
}

This approach provides several benefits.

No ID Collisions

UUIDs are globally unique, preventing conflicts between systems.

Clean Synchronization

Updates and raster requests can reference farms using the same external identifier.

/api/v1/farms/{external_farm_id}

Future Integration

If other services appear later (mobile apps, analytics pipelines, satellite processors), they can all reference farms using the same UUID.

This pattern is common in distributed systems and prevents a large class of synchronization bugs.

Identity Translation Between Systems

Another challenge is mapping users between Nextcloud and Django.

Nextcloud users authenticate normally and then communicate with Django using an integration token.

Conceptually:

Nextcloud User
      ↓
Integration Token
      ↓
Django API
      ↓
Farm Sync

The Django service stores farms under a service user while still preserving the original external_user_id from Nextcloud.

This keeps authentication simple while preserving user ownership information.

Measuring the Integration Layer

During development I analyzed the Nextcloud app using cloc to understand the size of the integration layer.

The results showed roughly 11,000 lines of code, split mainly between PHP backend logic and JavaScript UI.

Around this size, architecture decisions begin to matter more than raw implementation.

Systems become complex enough that clear service boundaries become essential.

Lessons Learned

Several practical lessons emerged from this integration work.

1. Avoid bidirectional replication

Two systems writing to the same domain model creates unnecessary complexity.

2. Establish a clear source of truth

In this architecture, Django owns farm data while Nextcloud orchestrates the workflow.

3. Use stable external identifiers

UUIDs dramatically simplify synchronization across systems.

4. Prefer API-first architectures

APIs make it easier to expand systems and integrate future services.

5. Keep compute close to the data

Since Django handles geospatial processing, storing farms there keeps the compute layer efficient.

Final Thoughts

What started as a simple integration between Nextcloud and Django turned into a useful exercise in distributed system design.

Even relatively small systems benefit from clear service boundaries and stable identity strategies.

By combining:

an API-first architecture
external UUID identifiers
and clear ownership of farm data

the system stays simple today while remaining extensible for future services like satellite analytics or mobile farming applications.

Sometimes good architecture isn’t about complexity at all — it’s about clarity of responsibility between systems.

Escaping the Sync Trap: How I Slashed Latency by 10x in a Django-Rust API Gateway

Rahim Ranxx — Sun, 01 Mar 2026 13:24:13 +0000

How I diagnosed and eliminated synchronous bottlenecks in a Django-Rust API gateway, migrating to ASGI and pre-warming caches for millisecond responses.

When building a high-performance backend, the standard playbook is well-known: offload heavy computational tasks to faster microservices (like Rust) and implement an aggressive caching strategy.

Recently, I did exactly that. My architecture is built around a Django REST Framework gateway sitting behind Caddy, heavily monitored with Prometheus and Grafana.

But despite the raw speed of Rust and my caching layers, my dashboards were flashing red. Latency was spiking to brutal 10-second flatlines for my most critical endpoints. Worse, my observability itself started failing, creating silent blind spots exactly when I needed data the most.

Here is the detective story of how I used telemetry to hunt down synchronous traps, migrate to a non-blocking async architecture, and implement proactive pre-warming to bring response times down to the millisecond range — all while reclaiming 30% of my idle CPU.

The Architecture: A Gateway and its Heavy Lifters

Before diving into the problem, here is a quick look at my setup.

          ┌──────────────────────────────┐
          │          Nextcloud           │
          │ (Authenticated Client Calls) │
          └──────────────┬───────────────┘
                         │  JWT / API Key
                         ▼
               ┌────────────────────┐
               │  Django Gateway    │
               │ (ASGI, DRF, Caddy) │
               └──────┬─────────────┘
                      │
     ┌────────────────┴────────────────┐
     │                                 │
     ▼                                 ▼
┌───────────────┐              ┌────────────────┐
│ NDVI Service  │              │ Weather Service│
│ (Rust, 8081)  │              │ (Rust, 8090)   │
│  → Postgres    │              │  → MySQL       │
└───────────────┘              └────────────────┘

       ▲
       │ Prometheus & Grafana
       │ (Observability Stack)
       ▼
   System Telemetry + Metrics

Django stays as the public API gateway, accepting requests authenticated via JWT or API keys.
It enforces a shared JSON response envelope containing status, message, data, and errors to keep all client interactions standardized.

Specific traffic routes — namely /api/v1/ndvi and /api/v1/weather/* — are forwarded directly to my Rust backends:

🛰 NDVI microservice ingests satellite data into a dedicated Postgres database.
🌦 Weather microservice relies on a MySQL database and communicates with external providers like Open-Meteo and NASA POWER.

My Nextcloud instance acts like any other client, presenting either an Authorization: Bearer token or an X-API-Key. Django manages this traffic using a specific nextcloud_hmac throttle configuration before passing the authorized call down to Rust with the original headers intact.

The False Cure: The Worker Starvation Anomaly

To protect the system, I implemented aggressive TTL caching (e.g., 1 hour for schema data, 5 minutes for API tokens). However, once I added traffic, my Grafana dashboard revealed a chaotic reality.

I saw brutal, perfectly flat 8–10 second latency spikes on key endpoints. Crucially, perfectly timed with these latency spikes, my internal /metrics request rate dropped to zero.

The Diagnosis: The Gateway Caching Itself to Death

The telemetry told a story of hidden synchronous bottlenecks:

The Trigger: When a high-traffic endpoint like farm-weather-current/GET experienced a cache miss, the Django gateway had to fetch fresh data.
The Trap: My Django deployment was running using standard synchronous workers. It called the Rust service, which then called the external weather API (taking 2.2+ seconds).
The Impact (Worker Starvation): Because the Django worker was synchronous, it blocked entirely for those 2.2 seconds. All incoming traffic got stuck in a queue.

The Trap: Synchronous Gateway Routing

import httpx
from django.http import JsonResponse
from django.views import View

async def async_weather_proxy_view(request):
    async with httpx.AsyncClient(timeout=5.0) as client:
        resp = await client.get("http://weather-service:8090/api/v1/weather-current")
        resp.raise_for_status()
        data = resp.json()
    return JsonResponse({"status": "success", "data": data, "errors": None})

I had successfully offloaded work to Rust — but my synchronous Django workers completely nullified the speed gains.

The First Fix: Embracing the Non-Blocking Gateway

I needed to decouple the speed of the gateway from the speed of the external API calls it was routing.
I migrated the Django deployment from synchronous workers to an ASGI (Asynchronous Server Gateway Interface) setup, allowing my gateway to handle requests asynchronously.

I rewrote my proxy views to use asynchronous HTTP clients like httpx:

The Fix: Asynchronous Non-Blocking Routing

import httpx
from django.http import JsonResponse

async def async_weather_proxy_view(request):
    # The event loop is freed! Django can serve other requests while waiting
    async with httpx.AsyncClient() as client:
        rust_response = await client.get("http://weather-service:8090/api/v1/weather-current")

    return JsonResponse({
        "status": "success",
        "data": rust_response.json(),
        "errors": None
    })

The visual evidence on my dashboards was a massive, instant victory:

Observability Restored: The metrics scrape line remained unbroken. Django could finally pause a slow weather request, instantly answer the Prometheus scrape, and resume without blocking.
⚡ Instant Internal Routing: In my initial setup, a simple internal metrics scrape took ~84ms. After the ASGI migration, that duration dropped to 11ms.

The Second Fix: Proactive Caching

While the infrastructure was now bulletproof, the end-user experience was still occasionally sluggish.

With an “on-demand” caching strategy, the very first user to request the weather after a 1-hour cache expiration had to pay the Cache Miss Penalty (waiting ~2.2 seconds for the external API).

To eliminate this, I decoupled the data-fetching time from the user-request cycle entirely.
I implemented a Proactive Background Pre-warming pattern using a background task (like Celery) that runs every 55 minutes, independently fetching slow data and silently overwriting the cache before it expires.

The Cache Pre-Warmer (Celery Example)

from celery import shared_task
from django.core.cache import cache
import requests

@shared_task
def pre_warm_weather_cache():
    # Runs in the background every 55 minutes, shielding the user from the 2.2s wait
    response = requests.get("http://weather-service:8090/api/v1/weather-current")
    if response.status_code == 200:
        cache.set("weather_current_data", response.json(), timeout=3600)

The result? The average latency for critical weather endpoints plummeted from seconds to milliseconds as the hot cache permanently took over.

The Grand Slam: Optimizing the Rust Build Pipeline

The final victory of this new architecture came from pure server efficiency.
During deployments, compiling Rust crates (sqlx, syn) from scratch was pegging my 4-core server at 100% CPU, artificially causing timeouts.

To fix this, I implemented cargo-chef in a multi-stage Dockerfile to strictly cache Rust dependencies.

Multi-stage Dockerfile for the Rust Microservice

FROM rust:1.88-slim AS chef
USER root
RUN cargo install cargo-chef
WORKDIR /app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
# Docker caches this heavy dependency build!
RUN cargo chef cook --release --recipe-path recipe.json
COPY . .
RUN cargo build --release --bin weather-service

FROM debian:bookworm-slim AS runtime
WORKDIR /app
RUN apt-get update && apt-get install -y libssl-dev ca-certificates
COPY --from=builder /app/target/release/weather-service /usr/local/bin/
EXPOSE 8090
ENTRYPOINT ["/usr/local/bin/weather-service"]

Between ASGI, background caching, and Docker layer caching, my total Node CPU now rests comfortably between 11% and 13%.
I fundamentally reclaimed 30% of my total server compute capacity.

Conclusion: Finding the Next Bottleneck

Building high-performance API gateways is an ongoing journey of shifting bottlenecks.

By relying strictly on my telemetry, I proved that synchronous workers nullify microservice speed, validated the immense power of ASGI, and eliminated cache miss penalties.

With the gateway running unburdened, my dashboards have revealed one final bottleneck — a 6-to-8 second delay on my token generation endpoint.
Because my CPU is mostly idle, I know exactly what this is: a database connection pool limitation in the Rust service.

And thanks to my new observability baseline, I know exactly where to strike next.

Boost Performance by Migrating Django Endpoints to Rust: NDVI & Weather Services (Phase 2 Complete)

Rahim Ranxx — Sat, 21 Feb 2026 16:55:43 +0000

Migrating Django Endpoints to Rust: My NDVI & Weather Services Journey

When I started rethinking my NDVI and weather endpoints, the goal was simple: improve performance, enforce strong auth, and gain full observability. Over the last few weeks, I migrated critical services from Django to Rust, and the process turned out to be an engineering adventure worth sharing.

Phase 0 – Contract Freeze: Locking the APIs

Before touching Rust, I froze all NDVI and weather API contracts in Django. This ensured that the front-end and other consumers could continue working without disruptions. Think of it as putting a protective glass over your APIs: nothing moves until Rust is ready to take over.

Output: Frozen NDVI + weather contracts from Django.

Phase 1 – Multi-Service Architecture & Shared Auth/Throttle

Next, I set up a Rust workspace with multiple services:

NDVI service: Handles vegetation index calculations.
Weather service: Will eventually serve weather data.
Shared auth & throttling module: Ensures consistent authentication and rate limiting across all services.

This phase established the skeleton for independent Rust microservices while maintaining the same contract as Django.

Output: Rust workspace, shared auth/throttle, NDVI envelope.

Phase 2 – Weather Migration

With the workspace ready, I migrated weather endpoints from Django to Rust. Key steps included:

Implementing shared authentication and throttling.
Integrating MySQL connections safely with Rust’s type system.
Ensuring the endpoints conformed to the frozen contract from Phase 0.

After this phase, all weather requests were fully handled by Rust services, improving throughput and reliability.

Output: Weather endpoints implemented in Rust.

Phase 3 – Gateway Cutover (Planned)

The final phase will transition Django routes to forward requests to Rust microservices. This will include:

Canary deployments to avoid downtime.
Metrics and alerting for observability.
CI enforcement for Rust formatting, clippy lints, and tests across the workspace.

End state after Phase 3:

Django acts as a gateway, routing NDVI + weather requests to Rust services.
NDVI is fully served by Rust/Postgres.
Weather is fully served by Rust/MySQL.
Shared auth and throttling are enforced in Rust.
Observability and canary rollouts ensure safe production deployment.
CI checks formatting, linting, and tests across the workspace.

Lessons Learned

Contract first: Freezing contracts before migration prevents chaos.
Shared modules are gold: Auth and throttling reused across services reduces duplication.
Rust’s type system and ownership model force careful database and network design.
Incremental migration avoids “big bang” outages.

Why Rust?

Migrating to Rust allowed me to:

Serve high-throughput endpoints with lower latency.
Reduce runtime errors with compile-time guarantees.
Scale services independently while sharing critical modules like auth and throttling.

Example: Rust Weather Service (axum + sqlx)

// src/main.rs
#![deny(clippy::all)]
#![forbid(unsafe_code)]

use axum::{routing::{get, post}, Router, Json, extract::Extension, response::IntoResponse, http::StatusCode};
use serde::{Deserialize, Serialize};
use sqlx::mysql::MySqlPoolOptions;
use std::sync::Arc;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

mod auth;
mod rate_limit;

#[derive(Clone)]
struct AppState {
    db: sqlx::MySqlPool,
}

#[derive(Deserialize)]
struct WeatherQuery {
    lat: f64,
    lon: f64,
    ts: Option<i64>,
}

#[derive(Serialize)]
struct WeatherResponse {
    temp_c: f32,
    precip_mm: f32,
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    tracing_subscriber::registry()
        .with(tracing_subscriber::EnvFilter::from_default_env())
        .with(tracing_subscriber::fmt::layer())
        .init();

    let db = MySqlPoolOptions::new()
        .max_connections(20)
        .connect(&std::env::var("WEATHER_DATABASE_URL")?)
        .await?;

    let state = Arc::new(AppState { db });

    let app = Router::new()
        .route("/api/v1/weather/point", get(get_weather_point))
        .route("/api/v1/weather/bulk", post(post_weather_bulk))
        .layer(auth::AuthLayer::new())
        .layer(rate_limit::RateLimitLayer::new(100));

    let addr = std::env::var("LISTEN_ADDR").unwrap_or_else(|_| "0.0.0.0:8080".into());
    tracing::info!("listening on {}", addr);
    axum::Server::bind(&addr.parse()?)
        .serve(app.into_make_service())
        .await?;

    Ok(())
}

Django Gateway Proxy Example

# views/proxy.py
import httpx
from django.http import HttpResponse
from django.conf import settings

PROXY_TARGET = settings.PROXY_TARGET  # e.g., "http://rust-weather:8080"

async def proxy_request(request):
    upstream = f"{PROXY_TARGET}{request.path}"
    headers = {"x-forwarded-for": request.META.get("REMOTE_ADDR", "")}
    if "Authorization" in request.headers:
        headers["Authorization"] = request.headers["Authorization"]

    async with httpx.AsyncClient(timeout=30.0) as client:
        upstream_resp = await client.request(
            request.method,
            upstream,
            headers=headers,
            params=request.GET,
            content=await request.body()
        )

    return HttpResponse(
        content=upstream_resp.content,
        status=upstream_resp.status_code,
        headers={k: v for k, v in upstream_resp.headers.items() if k.lower() != "transfer-encoding"},
    )

CI Example (GitHub Actions)

name: CI
on: [push, pull_request]
jobs:
  rust-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain-action@v1
      - run: cargo fmt --all -- --check
      - run: cargo clippy --workspace --all-targets -- -D warnings
      - run: cargo test --workspace --all-features

  python-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install ruff mypy bandit
      - run: ruff check .
      - run: mypy .
      - run: bandit -r .

This setup ensures a production-ready, highly observable Rust microservices environment while keeping Django as a stable gateway. Phase 3 will finalize the gateway cutover with canary deployment and metrics monitoring.

From Django to Rust Microservices: What Prometheus Taught Me About Backend Performance

Rahim Ranxx — Sun, 15 Feb 2026 06:48:42 +0000

Django Performance and Prometheus Observability
I operate a stack combining Django REST Framework, Nextcloud integrations, Prometheus for metrics, and Grafana dashboards — all served behind Caddy with strict CI/CD and Dockerized isolation.

Everything looked stable until my Prometheus metrics told a different story.

In Grafana, the /prometheus-django-metrics endpoint consistently showed 250 ms latency spikes, while other endpoints like /farm-weather-hourly and /home averaged under 50 ms. Scrape durations varied between 80 ms and 430 ms, even when request rates stayed flat at 0.08 req/s.

That meant the latency wasn’t due to load — it was intrinsic to Python’s runtime and how Django handled metrics serialization.

Why Prometheus Exposed Django’s Bottleneck
Each Prometheus scrape forces Django to:

Lock the Global Interpreter Lock (GIL)
Gather live counters and histograms
Serialize JSON or text payloads
Reallocate memory on every request

Even low-volume systems suffer because this happens repeatedly at fixed intervals. Observability itself became a performance cost.

The graphs made it clear: the bottleneck was the runtime, not the code.

Why Migrate Django Microservices to Rust
Rust’s asynchronous ecosystem (Tokio / Actix Web) solves these exact issues.

No GIL: True multi-core concurrency.
Predictable latency: Consistent under heavy I/O.
Memory safety: Compile-time guarantees without a garbage collector.
Low overhead I/O: Async networking with minimal allocations.

In my benchmarks, Rust microservices consistently stay under 40 ms latency, use 30–40 % less CPU, and make Prometheus scrape times nearly constant.

Rust Microservices Architecture with Django and Prometheus
The new architecture keeps Django as the orchestrator — managing authentication, APIs, and admin routes — while Rust handles performance-intensive modules:

NDVI raster computation
Weather data transformation
Metrics aggregation

They communicate via REST or gRPC. Prometheus exports data from both runtimes into unified Grafana dashboards.
Caddy provides HTTPS termination and reverse-proxy routing, maintaining secure observability across the stack.

This hybrid model keeps Django’s flexibility while giving me Rust’s efficiency where it matters.

Lessons from Observability

Metrics are architectural signals, not just health checks.
Python’s runtime trade-offs appear first under introspection, not user load.
Rust isn’t a replacement for Django — it’s a reinforcement for its weak spots.
Observability drives evolution when used as feedback, not just monitoring.

The Road Ahead
My next experiment involves measuring CPU cycles per request across Django and Rust services under sustained Prometheus scrapes. The goal: prove observability-driven performance scaling in production.

If your /metrics endpoint is your slowest route, don’t ignore it — that graph might be pointing directly toward your next architectural upgrade.