Building a Real-Time Shipment Tracking Platform that Scales to Millions

Building a Real-Time Shipment Tracking Platform that Scales to Millions

Quick Answer

Explore a production‑grade design for a real-time shipment tracking platform, covering scalable state management, event‑driven pipelines, and Azure‑native microservices.

Building a Production‑Ready Real‑Time Shipment Tracking Platform

For a senior architect, the headline is simple: you must keep the UI < 500 ms while ingesting millions of telemetry events per day and staying under a predictable Azure bill. This article shows how to turn that headline into a concrete, production‑grade design. It skips the fluff, dives straight into the decision points you face, and ends with a checklist that you can copy‑paste into your next sprint.

State Exposure & Operational Constraints

The core problem is not the GPS driver; it is the *state* you expose to the front‑end. Every container, pallet, or vehicle is a moving object whose location, status, and environmental data must be queryable in real‑time. The constraints are:

• Event ingestion rate: 10–12 M events per second during peak.
• Latency target: < 500 ms from event arrival to UI refresh.
• Consistency: eventual consistency is acceptable for UI, but the underlying state must never be corrupted.
• Cost: < 10 USD per million events processed, < 200 USD per GB of hot memory.
• Operational complexity: no single point of failure; auto‑recovery must be in‑built.

Real‑World Example

Consider FastTrack Logistics, a mid‑size provider that grew from 10 k shipments/day to 2 M shipments/day over 18 months. They have Three data sources per shipment:

• GPS pings every 10 s (≈ 2 B events/month).
• Carrier status callbacks (≈ 5 M events/month).
• Temperature sensors for refrigerated goods (≈ 1 M events/month).

The platform needed to surface the current location, ETA, and alerts to a web dashboard and a mobile app, with a 95 % SLA on UI latency. They also had to keep the Azure bill < 50 k/month.

Trade‑offs

Every decision in a real‑time tracking stack involves at least two competing axes: latency vs durability, cost vs complexity, and consistency vs scalability. Below are the key trade‑offs we faced and the rationale that guided us.

Axis	Option A	Option B	Why we chose B
State Storage	Single PostgreSQL row per shipment (write‑through)	Hot Redis + PostgreSQL snapshot	PostgreSQL write‑through would choke on 12 M events/sec; Redis gives sub‑ms reads.
Event Ordering	Kafka partition per shipment	Hash‑based partitioning across 500 partitions	Per‑shipment partitioning leads to hot partitions; hash spreads load.
Consistency Model	Strict ACID (transactional write‑through)	Event‑sourced with idempotent projector	Transactional writes add >200 ms latency; event sourcing keeps writes fast.
Cost of Hot Memory	Dedicated Redis cluster (10 GB)	Tiered cache: Redis for hot 24 h, Blob for older snapshots	Reduces memory footprint by 70 % while keeping SLA.
Observability	Manual logs per component	OpenTelemetry + Azure Monitor	Provides end‑to‑end tracing for 500 ms SLA.

Choosing Ingestion & State Layers

1. Choose Ingestion If you need >5 M events/sec, pick Event Hubs (Kafka protocol). It auto‑scales partitions and has built‑in geo‑replication. For smaller workloads, Azure Service Bus is cheaper but limited to <1 M events/sec.
2. Define Partitioning Calculate the expected events per second per shipment. If < 10 events/s, hash‑based partitioning across 500 partitions keeps each partition < 25 k events/s, which a single processor can handle.
3. Select State Layer Use Redis for sub‑ms reads. Persist snapshots to PostgreSQL every 5 min. If your SLA relaxes to 1 s, you can drop Redis and keep everything in PostgreSQL with TimescaleDB.
4. Implement Idempotency Store the last processed offset per partition in Redis. On restart, replay from the last offset and skip duplicates.
5. Backpressure & Throttling Set max.poll.records to 200 for processors and use EventProcessorClient with EventProcessorOptions to pause consumer on lag > 30 s.
6. Observability Add OpenTelemetry instrumentation to every producer and consumer. Export traces to Azure Monitor Logs; use ServiceMap to surface latency by carrier.
7. Security Use Managed Identities for all Azure resources. Never embed SAS keys in code.

When This Fails in Production

• Consumer Lag Spikes – A carrier’s bulk upload can flood a single partition. The processor CPU saturates, lag > 2 min, UI stutters. Fix: back‑pressure + throttling microservice.
• Redis Eviction – During a surge, LRU policy evicts the most‑queried shipment keys. UI falls back to PostgreSQL, latency jumps to 1.5 s. Fix: dedicate a hot‑key Redis tier with noeviction policy.
• Snapshot Corruption – A pod crash during snapshot write leaves a partially written blob. On recovery, the read model is corrupted. Fix: atomic rename + checksum validation.
• Memory Fragmentation – Heavy updates to large Redis hash maps cause >20 % overhead. Fix: use hash-max-ziplist-entries tuning and fixed field sets.
• Event Replay Failure – Event Hubs Capture disabled leads to data loss after a network partition. Fix: enable Capture to Blob and replay from the earliest offset.

Common Mistakes Engineers Make

• Assuming Kafka gives you exactly‑once semantics – you’ll see duplicate locations unless you idempotent the write.
• Hard‑coding 10 partitions for a PoC – it collapses under 12 M events/sec. Use dynamic partitioning.
• Storing every GPS ping as a row in PostgreSQL – the table grows beyond 1 TB in a month. Use TimescaleDB or batch inserts.
• Exposing Event Hub endpoints without Managed Identity – credentials get leaked in CI/CD pipelines.
• Ignoring backpressure – a burst of events can overwhelm a single processor, causing lag and eventual data loss.
• Over‑caching – putting the entire shipment object in Redis without TTL leads to stale data and memory bloat.

Better Approach Based on Experience

After 3 years of running FastTrack Logistics’ platform, the following practices consistently reduce incidents and cost:

1. Event Sourcing + CQRS Hybrid All telemetry is persisted as immutable events in Event Hubs. A background projector builds a read model in Redis. This keeps writes fast (< 5 ms) and reads instant (< 1 ms). If you need audit, the raw events are available in Blob.
2. Consumer Group with Offset Store Each processor runs as a consumer group member. Offsets are stored in Redis (key: offset:{partition}) so a crash can resume exactly where it left off. No duplicate processing.
3. Back‑off Strategy for Hot Partitions When a partition’s lag exceeds 30 s, the processor pauses, and a throttle-service splits the incoming batch into 1‑minute chunks. This keeps CPU < 70 % and latency < 400 ms.
4. Tiered Cache Hot 24 h state in Redis; older snapshots in Blob with Cool tier. Snapshot writes are incremental (AOF) and scheduled at 3 am UTC to avoid peak traffic.
5. OpenTelemetry Tracing All spans are annotated with shipmentId and carrierCode. Alerts on span duration > 500 ms trigger an auto‑scale event.
6. Security First All services use Managed Identities. Event Hub and Blob use Azure RBAC. No SAS keys in code.
7. Automated Recovery Kubernetes HPA scales processors based on consumer group lag. A sidecar monitors Redis maxmemory usage and triggers a graceful shutdown if memory > 90 %.
8. Cost Monitoring Azure Cost Management dashboards track per‑resource spend. If Redis memory > 70 % for > 2 h, an alert triggers a review of the retention policy.

Performance Considerations

• Partition Count – 500 partitions gives ~24 k events/sec per partition at 12 M events/sec. Scale processors to match partitions; each .NET instance consumes < 70 % CPU.
• Consumer Group Lag – Keep lag < 30 s; beyond that, enable back‑pressure. Monitor EventProcessorClient.Lag via Application Insights.
• Redis Latency – Use Cluster mode; keep key size < 1 KB. For 2 M concurrent shipments, 10 GB memory is enough if you store only {lat,lng,ts,status} per shipment.
• Snapshot Frequency – Every 5 min snapshot keeps the read model fresh. The snapshot file is ~200 MB; writing to Blob takes < 2 s with AOF.
• Back‑pressure Settings – max.poll.records=200 keeps the event loop from blocking; max.poll.interval=30s allows graceful handling of slow processors.

Scaling Notes

• Horizontal Scaling – Deploy processors in a Kubernetes Deployment with replica count = partition count. Use PodDisruptionBudget to avoid simultaneous restarts.
• Multi‑Region Replication – For global customers, deploy a read replica of Redis in each region. Use Azure Front Door to route UI traffic based on latency.
• Autoscaling – HPA based on Redis consumer lag and CPU. Set min replicas to 10, max to 500.
• Cost‑Optimized Tiering – Use Azure Reserved Instances for Redis if you can commit 1‑year; otherwise pay‑as‑you‑go with Standard tier.
• Observability at Scale – Export logs to Azure Log Analytics; use Kusto queries to detect patterns like “carrier X has > 5 % duplicate events”.

Actionable Checklist for Your Next Sprint

1. Define event schemas in Avro; enforce at the edge gateway.
2. Spin up Event Hub with 500 partitions; enable Capture to Blob.
3. Deploy .NET processors as consumer group members; store offsets in Redis.
4. Set up Redis cluster with noeviction for hot keys; schedule AOF snapshots.
5. Implement OpenTelemetry instrumentation across producers, processors, Redis, and gRPC API.
6. Configure HPA for processors based on lag; set sidecar to monitor memory.
7. Write a throttle-service to split bulk carrier uploads into manageable chunks.
8. Set up Azure Cost Management alerts for Redis memory > 70 % and Event Hub throughput units > 50.
9. Run a load test with 12 M events/sec; verify lag < 30 s and UI latency < 500 ms.
10. Document the recovery procedure for snapshot corruption (atomic rename + checksum).

With these patterns, you can go from a prototype that processes 10 k events/sec to a production platform that reliably handles 12 M events/sec, keeps the UI snappy, and stays within a predictable cost envelope.

Related Articles

• Agentic AI Customer Support Platform Architecture: A Production‑Ready Design Walkthrough
• Hardening WebMCP Security Considerations for ASP.NET Core Applications – A Production Guide
• Benchmarking .NET vs Node.js for Building Scalable AI Agents
• NVIDIA NOOA for .NET: Reducing Latency in Microservices
• Three Layers of AI Infrastructure, Thirty-Three Minutes Apart: Debugging a Multi-Provider AI Orchestration Stack