DEV Community: Debjit Dey

Tracking Chaos: Building a Real-Time Flight Anomaly Engine with Django, Celery, and Machine Learning

Debjit Dey — Sun, 24 May 2026 11:25:09 +0000

Imagine walking outside on a quiet afternoon. You hear a sharp roar overhead, pull out your phone, and open a flight-tracking app. You find a tiny airplane icon ✈️ iding smoothly along a solid line. It looks clean, structured, and completely predictable.

But what if that airplane icon suddenly starts making frantic, tight loops over a residential area? What if it begins a terrifyingly rapid descent, or behaves in a way that defies normal flight paths?

If a human air traffic controller isn’t watching that specific screen, how can software automatically flag that something is wrong?

When I set out to build SkyWatch Live, an open-source airspace and satellite tracking dashboard, my focus quickly shifted from simply drawing dots on a map to a much more interesting engineering problem: How do you build a real-time machine learning pipeline that can detect unusual airborne behavior across thousands of concurrent flights using chaotic public data?

Repo: https://github.com/debjit450/skywatch-live

Whether you are an AI engineer or someone who has never looked at aviation data in your life, the architecture behind processing volatile, high-frequency telemetry into clean, explainable alerts contains lessons that apply to any real-time streaming system.

The Operational Blueprint

Before diving into the math, it helps to see the pipeline that keeps everything alive. The application runs a split architecture:

The UI Layer: A React 19 and TanStack Start dashboard using MapLibre GL and deck.gl to paint real-time positions and historical playback tracks smoothly at 60 frames per second.
The Ingestion & Inference Layer: A Django ASGI core backed by Celery background workers, a fast Redis state cache, and a durable PostgreSQL time-series database.

Every 15 seconds, a scheduled Celery Beat task pulls raw telemetry vectors from multiple fragmented public radio networks. It de-duplicates them, caches the latest snapshot in Redis, and broadcasts them down a live WebSocket channel directly to the map.

But as soon as that data hits the database, a post-commit hook hands the raw flight data over to our dedicated machine learning pipeline.

The 3-Gate Anomaly Pipeline

If you feed raw, noisy public data straight into a complex neural network, your system will instantly drown in false positives. External sensors glitch, transponders experience signal dropouts, and rate limits throttle incoming coordinates.

To prevent false alarms, SkyWatch Live runs data through three computational gates.

Gate 1: Deterministic Physical Rules

The first line of defense doesn't use AI at all. It uses fast, binary checks written in raw Python to catch immediate high-signal events:

Emergency Squawk Codes: Radio transponders broadcasting specific numbers like 7700 (general emergency) or 7600 (radio communication failure).
Kinematic Violations: Physical impossibilities, such as a non-military cargo plane suddenly executing a 90-degree turn mid-air or dropping altitude faster than its structural limits allow.

Gate 2: Feature Engineering & Spatial Grids

If a flight path passes basic physics checks, the pipeline extracts hidden behavioral features out of raw coordinate sequences (latitude, longitude, altitude, heading, velocity). This happens inside backend/ml/features.py:

Spatial Indexing: The engine dynamically hashes coordinate points into a geometric grid to calculate local proximity metrics (spotting close-proximity events or dense airspace deviations).
Angular Velocity Tracking: By computing statistical variance across rolling windows of an aircraft's heading history, the system converts raw directional degrees into a "loitering score" that exposes circling or tracking patterns.
The Behavioral Baseline: The telemetry is cross-referenced with a historical AircraftProfile table, checking whether the current aircraft type is operating outside its typical operational envelope.

Gate 3: The Statistical Machine Learning Ensemble

Once the features are compiled into a normalized vector, they strike a three-model ensemble powered by scikit-learn. Using a blend of different algorithmic approaches balances out the blind spots of any single model:

Isolation Forest: A tree-based model that isolates anomalies by randomly partitioning features. Because anomalies require fewer splits to isolate than normal data, they appear near the shallow roots of the trees. It is great for spotting overall global outliers.
Local Outlier Factor (LOF): A density-based algorithm that measures how locally isolated a data point is relative to its surrounding neighborhood. This catches contextual anomalies—like a plane flying at a speed that is normal globally, but highly irregular for that specific crowded corridor.
MLP Autoencoder: A neural network that attempts to compress the multi-dimensional feature vector down into a tiny bottleneck layer and reconstruct it perfectly on the other side.

The autoencoder flags structural anomalies by calculating the Mean Squared Error (MSE) reconstruction error between the original feature vector x and the reconstructed output x̂ across n dimensions:

If the structural configuration of a flight path is weird, the autoencoder fails to reconstruct it accurately, causing the error E to spike beyond a dynamic, self-calibrating threshold.

Going Deeper: Time-Series Sequence Analysis

While spatial snapshots catch immediate deviations, flight data is fundamentally a time-dependent sequence. To track subtle, slow-building irregularities over time, SkyWatch Live features an optional deep learning path using a Long Short-Term Memory (LSTM) network inside backend/ml/lstm.py.

[State at t-3] ──► [State at t-2] ──► [State at t-1] ──► [Current State t]
                                                               │
                                                               ▼
                                                    Deep Sequence Inference
                                                               │
                                                               ▼
                                                    Trajectory Anomaly Score

Engineering for Accessibility: TensorFlow and Keras are intentionally excluded from the project's default dependency file. This ensures open-source contributors can download, run, and modify the UI or standard ingestion loops without requiring massive deep-learning runtimes or expensive GPUs. The LSTM modules load conditionally, initializing only when a user explicitly activates them via python manage.py train_lstm_anomaly.

Explainability: Busting the "Black Box"

An anomaly alert is completely useless if a user doesn't know why it went off. If the map flashes red, an operator needs to know what triggered the alarm instantly.

To fix this, our explainability module decomposes the ensemble's mathematical scoring matrix into an explicit plain-text payload. When the UI hits /api/v1/anomalies/<id>/explanation/, it receives a clean breakdown:

{
  "anomaly_id": "8f3b2a-7c",
  "detector_type": "Ensemble_LOF",
  "severity": "CRITICAL",
  "confidence_score": 0.91,
  "explanation": "Triggered due to an 87% deviation in rolling heading variance (circling behavior) paired with atypical low-velocity thresholds for this airframe profile."
}

The frontend reads this payload to display clear warnings alongside the live flight profile. Users can even submit structured feedback to mark a detection as a false alarm, creating a clean, labeled dataset that our automated Celery jobs use to retrain the models every week.

Building Systems That Expect Uncertainty

SkyWatch Live shows that you can build highly performant, intelligent monitoring tools out of raw public data if you architect your pipeline to expect imperfections. By separating your fast live-state caches from your analytical data stores and guarding your machine learning models with strict validation layers, you build a system that tells the truth about chaotic real-world inputs.

If you have a laptop, a curious mindset, and want to dig into the background tasks, training engines, or the mapping layer, the repository is open source and ready for setup. Hop into the codebase, explore how the real-time data flows, and let me know your thoughts!

👉 GitHub Repository: debjit450/skywatch-live

Adaptive Rate Limiting with Redis and Lua

Debjit Dey — Tue, 28 Apr 2026 17:36:33 +0000

Making Rate Limiting Correct Under Concurrency

Most rate limiting tutorials stop at the single-instance case.
That’s fine for learning, but it breaks quickly in production.

Once you have multiple instances and real traffic patterns, the problem changes.
It’s no longer just about picking an algorithm — it’s about correctness under concurrency.

This article walks through what actually goes wrong and how to fix it.

The In-Memory Trap

The first implementation most people write looks like this:

keep a counter in memory
increment on each request
reject when the limit is reached

This works perfectly in a single instance.

Now deploy two instances.

Each instance has its own counter. A client can exceed your intended limit just by hitting different instances.

At that point, you don’t have a rate limiter anymore.
You have a suggestion.

Redis Fixes Distribution, Not Concurrency

The next step is moving state to Redis.

Now all instances share the same counters. Good.

A typical implementation looks like this:

Read current count from Redis
Check against limit
Increment and write back

This seems correct, but it isn’t.

These are separate operations. Under concurrent load:

two requests read the same value
both pass the check
both increment

Now your limit is no longer strict. It’s approximate.

The Real Problem: Atomicity

The issue isn’t Redis.

It’s that the decision is split across multiple steps.

What you need is:

a single, atomic operation that reads state, applies logic, and updates state

The Fix: Lua Scripts in Redis

Redis supports Lua scripts that execute atomically.

No other command runs between the start and end of the script.

Instead of multiple round trips:

read state
apply limiter logic
update state
return decision

You do everything inside one script.

Example (simplified):

local current = redis.call("GET", KEYS[1]) or 0
if tonumber(current) >= tonumber(ARGV[1]) then
  return {0, current}
end

current = redis.call("INCR", KEYS[1])
redis.call("EXPIRE", KEYS[1], ARGV[2])

return {1, current}

This ensures:

no race conditions
consistent decisions across instances
predictable behavior under load

Where Algorithms Fit In

At this point, you can plug in different strategies:

Token Bucket → allows bursts, smooths over time
Sliding Window → more accurate but heavier
Leaky Bucket → enforces steady flow

But here’s the key point:

The algorithm matters less than where the decision happens.

If your logic isn’t atomic, the algorithm won’t save you.

Static Limits Miss Real Traffic Behavior

Even with correct enforcement, static limits are too rigid.

Real traffic looks like:

legitimate bursts
scrapers probing endpoints
repeated identical requests
denial loops

A fixed limit treats all of these the same.

Adding a Behavior Layer

A simple improvement is to track short-term behavior:

request volume over a short window (burst detection)
repeated request fingerprints
number of unique routes hit (scan detection)
repeated denials

This produces a basic risk score.

That score maps to tiers:

normal
elevated
suspicious
blocked

The important part is separation:

Limiter → enforces limits
Policy → decides how strict to be

This keeps the system easier to reason about and tune.

Tradeoffs

This approach is not free.

Lua scripts add complexity
debugging moves closer to Redis
Redis becomes a critical dependency

But for systems that need consistency under concurrency, the tradeoff is worth it.

Key Takeaway

The biggest lesson is not about token buckets or sliding windows.

It’s this:

Correctness in rate limiting comes from atomic decision-making.

Once you ensure:

a single source of truth
atomic execution
consistent state across instances

the rest becomes much easier.

Closing

I built this approach into a small system to explore the problem end-to-end.

If you’re interested in seeing a full implementation (TypeScript + Redis + Lua), you can check it out here:

👉 https://github.com/debjit450/arce

If you’ve dealt with this problem in production, I’d be interested to hear how you approached it.

StableJSON: A Practical Workspace for Serious JSON Work

Debjit Dey — Wed, 31 Dec 2025 06:14:37 +0000

JSON is everywhere — APIs, configs, logs, exports, payloads, diffs.
If you build software, you already spend a non-trivial part of your life working with it.

Most JSON tools fall into two camps:

Too basic — format and validate, then you’re on your own
Overengineered — cluttered UIs, half-useful features, slow workflows

StableJSON sits in between.
It’s a focused workspace for actually working with JSON, not just prettifying it.

Live app: https://stablejson.vercel.app
Repo: https://github.com/debjit450/stablejson

Formatting Isn’t the Hard Part

Pretty-printing JSON is solved. Every editor can do that.

The real problems show up when you need to:

Compare large API responses
Debug payload mismatches
Clean data before storage or hashing
Generate types from unknown or evolving structures
Ensure identical data always produces identical output

That’s where things usually break down:

Key order changes
Nulls and empty values add noise
Diffs become unreadable
Hashes change unexpectedly
Generated types drift from reality

StableJSON is built specifically around these failure points.

What StableJSON Actually Gives You

StableJSON is a browser-based JSON workspace with tools that cover the entire JSON workflow:

Validate & format JSON with proper indentation
Clean null, undefined, and empty values in one step
Side-by-side JSON diff with clear highlighted changes
Path inspection & extraction for deeply nested data
Automatic TypeScript interface and Zod schema generation
Canonical JSON output for consistent hashing and comparison
JSONPath queries to slice complex objects
Structure analysis (depth, size, data types)
Transform, flatten, and reshape JSON safely
Custom validation rules when default checks aren’t enough

It’s not a collection of random utilities — it’s one coherent workflow.

Canonical JSON Is the Quiet Power Feature

One of the most important features in StableJSON is canonical output.

When JSON is deterministic:

Hashes stop changing randomly
Cache keys become reliable
Diffs stay readable
CI checks stop failing for no apparent reason

If you’ve ever had two JSON objects that look identical but serialize differently, you already know why this matters.

Canonical output isn’t flashy — but it prevents entire classes of bugs.

Who This Is Built For

StableJSON is for developers who:

Debug APIs regularly
Care about correctness, not just visuals
Work with large or dynamic JSON structures
Need reliable diffs, hashes, and generated types
Want one tool instead of five scattered ones

If JSON is part of your daily workflow, this tool earns its keep very quickly.

Closing

Most JSON tools stop at “looks nice”.

StableJSON goes further — it helps you understand, compare, clean, and stabilize your data.

When you need JSON to be right, not just readable, this is the tool you reach for.

Try it: https://stablejson.vercel.app
Explore the code: https://github.com/debjit450/stablejson