DEV Community: Tuấn Anh

Escaping the Monolith: How We Replaced Magento with 21 Go Microservices

Tuấn Anh — Fri, 26 Jun 2026 15:00:00 +0000

![Cover Image: A monolithic block breaking into Go microservices]

Answer-first: How we escaped Magento's licensing and scaling walls by migrating to a Composable Commerce Platform built on 21 Go microservices, Dapr PubSub, and a 3-phase Strangler Fig strategy without dropping a single order.

At exactly midnight during a major campaign, a monolithic Magento server can quickly become your single point of failure. Every engineering team that builds seriously on Magento eventually hits the same walls: the licensing wall ($100k-$200k/year for Enterprise), the scaling wall (scaling the entire massive codebase just to handle a traffic spike on the checkout page), and the developer velocity wall.

When we decided to migrate away from Magento, the standard industry advice was to split the monolith into "4 to 6 microservices". We ignored that advice. For serious e-commerce at scale, that approach inevitably leads to a distributed monolith — where services are deployed separately but remain tightly coupled through HTTP chains or shared database tables.

Instead, we built a Composable Commerce Platform using 21 Go microservices, Google's Kratos v2 framework, and Dapr PubSub. Here is the blueprint of how we structured it, and how we migrated with zero downtime.

For the complete architecture deep-dive across all domains, see the Composable Commerce Migration Series.

The 21-Service Blueprint: Domain-Driven Design in Practice

Before writing a single line of Go code, we established boundaries using Domain-Driven Design (DDD). Every domain owns its own PostgreSQL database. There are absolutely no cross-domain queries.

Here is the breakdown of our 6 core domains:

Commerce Flow (Checkout, Order, Payment): The highly critical money path, orchestrated using Saga patterns.
Product & Content (Catalog, Pricing, Promotion, Search): Read-heavy, heavily cached, requiring sub-50ms latency.
Logistics (Warehouse, Fulfillment, Shipping): Integration with the physical world and 3PLs.
Post-Purchase (Returns, Loyalty): Customer retention and post-sale state machines.
Identity & Access (Auth, User, Customer): Strict separation between internal staff (RBAC) and external customer PII.
Platform Operations (Gateway, Analytics, Notification): Shared infrastructure utilities.

Here is how traffic flows through the ecosystem:

![Commerce Ecosystem Architecture]
(For the full traffic anatomy and Elasticsearch CQRS implementation, see the Architecture Blueprint).

The 3 Rules of Decoupling: Avoiding the Distributed Monolith

The most common failure mode when migrating from a monolith is building a system where a single failure in a non-critical service cascades and takes down the checkout flow. We avoided this with three hard rules:

1. No cross-domain database queries.
If the Order service needs product data, it cannot query the Catalog database. It must either hold a denormalized copy (via CQRS) or call the Catalog service's gRPC API. This ensures schema changes in one domain never break another.

2. No synchronous calls in the async event path.
Once an event (e.g., order.paid) enters the Dapr PubSub mesh, it is processed independently. Downstream services like Fulfillment or Loyalty react to the event, but they never synchronously call back into the producer. If the Loyalty service is down, the order still succeeds.

3. No shared deployment pipelines.
Every service lives in a Rush monorepo but has its own ArgoCD Application and container registry path. A bug or a blocked deployment in the Notification service cannot block a hotfix for the Checkout service.

The Zero-Downtime Migration Strategy: The 3-Phase Strangler Fig

Magento's EAV (Entity-Attribute-Value) schema is a trap. You cannot just run an ETL job over the weekend and cut over traffic. We used a strict 3-Phase Strangler Fig approach to ensure zero dropped orders:

Phase 1 (Read-Only via CDC): We deployed the Go microservices in read-only mode behind our API Gateway. We used Debezium to stream Magento MySQL changes to our Go services via Dapr. Writes still went to Magento.
Phase 2 (Dual-Write & Conflict Resolution): Microservices started accepting writes, persisting to their own Postgres DBs, and publishing domain events. A sync-adapter service listened to these events and wrote back to Magento. Conflicts were resolved by explicit policies (e.g., timestamp-wins).
Phase 3 (Full Cutover via GitOps): Using ArgoCD, we gradually shifted traffic per service: 25% → 50% → 75% → 100%. Magento was kept on "hot standby" for a 30-day rollback window.

Key Takeaways for Engineers Migrating Monoliths

1. Establish boundaries before writing code. Do not extract services based on UI pages. Extract them based on business domains (DDD) and data ownership.
2. Enforce the DB-per-service rule strictly. The moment you allow two services to read the same database table, you have coupled their deployment lifecycles.
3. CDC is your best friend during migration. Using Debezium and Kafka/Dapr to stream data out of the legacy monolith is much safer than building massive REST APIs on the legacy system just for synchronization.
4. Do not do a "big bang" release. The Strangler Fig pattern takes longer to build (due to dual-write adapters), but it is the only way to migrate an active e-commerce platform without risking catastrophic revenue loss.

Frequently Asked Questions

Why Go (Golang) instead of Node.js or Java for microservices?

Go provides the perfect balance for e-commerce microservices: it compiles to a single static binary (fast startup for scaling), uses very little memory compared to JVM languages, and has excellent concurrency primitives (goroutines) for handling high-throughput API gateways and event consumers.

How do you handle distributed transactions across 21 services?

We don't do distributed locks or 2PC (Two-Phase Commit). We use the Saga Pattern orchestrated by the Checkout service, leveraging Dapr PubSub. If a step fails, the orchestrator fires compensating events to roll back previous steps.

Does the API Gateway become a new monolith?

No. The gateway only handles Auth, Rate Limiting, and basic routing. We strictly forbid placing business logic in the API Gateway. It acts purely as a traffic shield and BFF (Backend-For-Frontend).

For the full engineering blueprint — including the exact EAV extraction SQL and our Dapr Saga implementation — read the complete Composable Commerce Migration Series.

[System Design] Chapter 2: Flash Sale Engine - Solving Overselling and Hot Keys

Tuấn Anh — Tue, 23 Jun 2026 23:38:50 +0000

← Series hub
← Prev • Next →

Chapter 2: Flash Sale Engine - The Mystery Behind Redis and Hot Keys

Flash Sale events are the ultimate stress test for system architecture. When an iPhone is sold for $1, millions of users will smash the "Buy Now" button in the exact same millisecond. If this massive spike hits a MySQL database directly, the system will instantly crash due to Row Locks and Deadlocks.

1. The Hot Key Problem and Two-Tier Caching

A highly discounted product is known as a Hot Key.
Many developers mistakenly believe that "just putting inventory in Redis" solves everything. However, a single Redis node has Network Bandwidth and CPU limits (typically maxing out at ~100k Ops/sec). One million clicks on a single key will saturate the network interface card (NIC) of that Redis node.

Shopee's Solution: Multi-Level Caching

Tier 1 (Local Cache): Built directly into the RAM of the Golang Application Servers (using tools like sync.Map or BigCache). This local cache only stores a boolean flag: "Is the item still in stock?". It has a TTL of just 1-2 seconds but successfully blocks 90% of useless traffic from hitting the network once the item is sold out.
Tier 2 (Distributed Cache - Redis): Only when the Local Cache reports that the item is available does the request proceed to the Redis cluster.

2. Preventing Overselling with Atomic Lua Scripts

When a user buys an item, the system must deduct the inventory. But if you use standard commands: Read stock (GET) -> Check if > 0 -> Write new stock (SET), you will face a critical Race Condition. Two parallel threads might both read a stock value of 1, both decrement it, and result in selling two items when only one existed (Overselling).

The Solution: Shopee wraps the inventory deduction logic inside Lua Scripts running natively within Redis. Because Redis is fundamentally single-threaded, executing a Lua script acts as an Atomic Transaction—no other requests can interrupt it mid-execution.

-- Example Lua Script for Inventory Deduction
local stock_key = KEYS[1]
local stock = tonumber(redis.call('GET', stock_key))

if stock and stock > 0 then
    redis.call('DECR', stock_key)
    return 1 -- Purchase Successful
else
    return 0 -- Out of Stock
end

Fail Fast: Thanks to this mechanism, if the Lua script returns 0, the request is immediately rejected and the user sees an "Out of Stock" message. This RAM-level operation takes mere microseconds.

3. Inventory Sharding

For mega-campaigns, a single Hot Key on a single Redis Node is still too risky. Shopee employs Inventory Sharding.
If there are 1,000 iPhones, they do not store the number 1,000 in a single key iphone_stock. Instead, they slice it into 10 shards: iphone_stock_1 to iphone_stock_10. Each key holds 100 items and is distributed across 10 different physical Redis Nodes.

A load balancer or router randomly routes incoming user traffic to one of those 10 keys, instantly dividing the massive system pressure by 10.

sequenceDiagram
    participant User
    participant App as Golang Server<br/>(Local Cache)
    participant Redis as Redis Cluster<br/>(Sharded)
    participant Worker as Kafka Worker

    User->>App: Click "Buy Now"
    Note over App: Check Local Cache.<br/>Block if Out of Stock
    App->>Redis: Route to shard (e.g. stock_3)
    Note over Redis: Execute Atomic Lua Script
    Redis-->>App: If 0: Return Error
    Redis-->>Worker: If 1: Push Order Event to Queue
    Worker-->>User: Process Order Asynchronously

Developer Takeaway: RAM and caching are your strongest weapons against heavy traffic. However, do not blindly rely on a Distributed Cache. Combine it with Local Caches on the App Server to save network bandwidth, and always use Lua Scripts to guarantee data consistency when handling sensitive numbers like inventory or wallet balances.

References & Further Reading

This post was originally published on my blog at Chapter 2: Flash Sale Engine - Solving Overselling and Hot Keys.

Hi, I'm Lê Tuấn Anh (vesviet) 👋
I am a Senior Go Backend Architect & Distributed Systems Engineer with 17+ years of experience building high-traffic platforms (25M+ requests/month).
If you enjoyed this deep-dive, let's connect on LinkedIn or explore my consulting services at tanhdev.com/hire.

[System Design] Banking Microservices Architecture: Event Sourcing, CQRS & Saga Patterns for Core Banking

Tuấn Anh — Sun, 21 Jun 2026 00:28:28 +0000

Series context (Part 4 of 8): This article assumes familiarity with ACID transactions and database concurrency. Understanding why consistency guarantees are hard at the database layer is essential context before introducing distributed patterns here.

Why Microservices in Banking?

Microservices in banking is the architectural pattern where a core banking system is broken into independently deployable, domain-owned services (CIF, Payments, Lending, Notifications) connected by an event bus instead of direct database calls. This replaces monolithic systems like T24 or Flexcube — where a single change to the Payments module requires redeploying the entire application and risks taking down unrelated services.

High-risk deployments: Modifying a small module requires redeploying the entire system. A patch to the Payments module can take down CIF.
Inefficient scaling: You cannot scale just the Payments module during peak loads without scaling everything else — including parts that don't need more capacity.
Technology lock-in: Bound to a single programming language and database. Adding a modern ML risk engine becomes an 18-month integration project.

The current trend is transitioning to Headless Core Banking — decoupling the domain logic from the delivery channels (Mobile App, Internet Banking, ATM) using a banking microservices architecture.

Overall Architecture

                    ┌─────────────────────────────────────┐
  CHANNELS          │  Mobile App  │  Internet Banking  │  ATM/POS  │
                    └──────────────────────┬──────────────────────────┘
                                           │ REST/gRPC
                    ┌──────────────────────▼──────────────────────────┐
  API GATEWAY       │         API Gateway (Auth, Rate Limit, Routing)  │
                    └──────────────────────┬──────────────────────────┘
                                           │
         ┌─────────────────────────────────┼──────────────────────────┐
         │                                 │                          │
  ┌──────▼──────┐               ┌──────────▼─────────┐    ┌──────────▼──────────┐
  │ CIF Service │               │  Account Service   │    │ Payment Service     │
  │ (Customer)  │               │  (CASA, GL)        │    │ (Transfers, Fees)   │
  └─────────────┘               └────────────────────┘    └─────────────────────┘
         │                                 │                          │
         └─────────────────────────────────┼──────────────────────────┘
                                           │ Events (Kafka/Dapr)
                    ┌──────────────────────▼──────────────────────────┐
  EVENT BUS         │              Message Broker (Kafka / Redis)      │
                    └──────────────────────┬──────────────────────────┘
                                           │
         ┌─────────────────────────────────┼──────────────────────────┐
         │                                 │                          │
  ┌──────▼──────┐               ┌──────────▼─────────┐    ┌──────────▼──────────┐
  │ Loan Service│               │ Notification Svc   │    │ Reporting Service   │
  │ (Lending)   │               │ (SMS, Push, Email) │    │ (CQRS Read Side)    │
  └─────────────┘               └────────────────────┘    └─────────────────────┘

Pattern 1: Event Sourcing for the Ledger

In traditional architectures, we store the current state. In Event Sourcing, we store a sequence of immutable events that produce that state.

Why does Event Sourcing fit Core Banking?

The ledger is already essentially Event Sourcing — every entry is an immutable event. The current balance is simply the result of replaying all entries from the beginning.

// Events in the Account domain
type AccountOpened struct {
    AccountID    string
    CIFNumber    string
    Currency     string
    OpenedAt     time.Time
}

type MoneyDeposited struct {
    AccountID     string
    Amount        int64
    TransactionID string
    OccurredAt    time.Time
}

type MoneyWithdrawn struct {
    AccountID     string
    Amount        int64
    TransactionID string
    OccurredAt    time.Time
}

// Calculate balance by replaying events
func calculateBalance(events []Event) int64 {
    var balance int64
    for _, event := range events {
        switch e := event.(type) {
        case MoneyDeposited:
            balance += e.Amount
        case MoneyWithdrawn:
            balance -= e.Amount
        }
    }
    return balance
}

Pattern 2: CQRS — Command Query Responsibility Segregation

Core Banking has a unique characteristic: writes must be exceptionally robust (ACID) but reads need to be lightning fast (dashboards, reports). CQRS completely separates these two flows:

WRITE SIDE (Command)                READ SIDE (Query)
────────────────────────            ──────────────────────────
POST /transfers            →        Materialized Views
POST /accounts             →        Elasticsearch Index
PUT /loans/repay           →        Redis Cache

↓ Event Published ↓                ↑ Subscribe & Update ↑
         └──────────────────────────┘
              (Event Bus / Kafka)

Real-world Example:

Write Side: Processes transfers using PostgreSQL with full ACID compliance, guaranteeing money isn't lost.
Read Side: Dashboards display transaction history from Elasticsearch — ultra-fast queries, full-text search, and multi-condition filtering.

Pattern 3: Saga — Distributed Transactions Across Services

When a cross-bank transfer requires coordinating 3 services: Account Service (deduct money), Payment Service (send to clearing house), and Notification Service (send SMS), how do you ensure integrity?

Choreography Saga (Event-Driven)

Account Service                Payment Service           Notification Service
      │                               │                          │
      │── TransferInitiated ──────────▶│                          │
      │                               │── PaymentSubmitted ──────▶│
      │                               │                          │── SMS Sent
      │◀── PaymentCompleted ──────────│                          │
      │                               │                          │
   (release hold)                                            (done)

If Payment fails:
      │◀── PaymentFailed ─────────────│
      │                               │
   (cancel hold, refund)

Outbox Pattern — Guaranteeing Events are Never Lost

Problem: What if a service successfully commits to the database but fails to publish the event to Kafka?

Solution: Write the event to the database within the same transaction, then have a separate worker read and publish it to Kafka.

-- Outbox table: written in the same transaction as business data
CREATE TABLE outbox_events (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    topic       VARCHAR(100) NOT NULL,  -- 'account.transfer.completed'
    payload     JSONB        NOT NULL,
    status      VARCHAR(20)  NOT NULL DEFAULT 'PENDING',
    created_at  TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    published_at TIMESTAMPTZ
);

-- Inside the same Database Transaction:
-- 1. Update account balance
-- 2. Write ledger entries  
-- 3. INSERT into outbox_events

-- Separate worker running periodically:
-- SELECT * FROM outbox_events WHERE status = 'PENDING'
-- → Publish to Kafka
-- → UPDATE status = 'PUBLISHED'

API Design for Financial Transactions

Design Principles

Stateless APIs: Every request must contain all necessary information.
Mandatory Idempotency headers for all state-changing APIs.
Strict separation between Request (commands) and Status (polling).

POST /v1/transfers                    → Initiate transfer command
  Header: Idempotency-Key: <uuid>
  Body: { from, to, amount, currency }
  Response: { transfer_id, status: "PROCESSING" }

GET  /v1/transfers/{transfer_id}      → Check result
  Response: { status: "COMPLETED" | "FAILED", ... }

Never design a transfer API as a synchronous block because processing through a central clearing network (like SWIFT) can take anywhere from seconds to minutes.

Technical Stack Selection

Layer	Popular Choices	Reason
Service Framework	Go (Kratos, Fiber), Java (Spring Boot)	High performance, type-safe
Primary Database	PostgreSQL	Strong ACID, flexible JSONB
Cache	Redis	Balances, sessions, rate limiting
Event Bus	Apache Kafka, Dapr PubSub	Durable, ordered, replayable
Service Mesh	Istio, Dapr	mTLS, circuit breaking
Orchestration	Kubernetes	Auto-scaling, self-healing

References & Further Reading

🔗 Previous Step: Explore the foundational database layer in Part 3 — Database Design for Financial Transactions (ACID & Concurrency).

🔗 Next Step: Now that you understand banking microservices architecture and its event-driven patterns, see how these services communicate with the outside world through international financial standards. Continue reading Part 5 — International Integration Standards: ISO 8583 & ISO 20022.

🔗 Deep Dive: For a complete engineering guide to the full composable banking stack — ledger concurrency patterns, Strangler Fig migrations, RFC 8705 mTLS, and the next-gen vendor landscape — see Composable Banking Architecture: From Monolith to Modular Core.

This post was originally published on my blog at Banking Microservices Architecture: Event Sourcing, CQRS & Saga Patterns for Core Banking.

[System Design] GraphHopper Distance Matrix: Self-Host OSRM vs Haversine for Route Optimization

Tuấn Anh — Fri, 19 Jun 2026 01:02:51 +0000

Series context: This is Part 7 of the E-commerce Order Allocation series. The distance matrix built here feeds directly into the OR-Tools VRP solver in Part 6.

The Invisible yet Most Expensive Bottleneck in E-commerce Routing

Answer-first: For 1 warehouse + 100 delivery stops, you need 10,201 pairwise distances. Self-hosting GraphHopper or OSRM on a $20/month VPS delivers this in under 50ms per batch — completely free. Using Google Maps Distance Matrix API for the same workload costs $510/day. This guide shows you exactly when to use which tool, with Docker setup and production Python code.

For any VRP (Vehicle Routing Problem) solver to find the optimal delivery route, it needs to know the exact cost between every pair of stops — this is the distance matrix. For 1 warehouse + 100 orders (101 points), that is 101 × 101 = 10,201 pairs. Choosing the wrong tool for this step can cost $510/day in API fees or cause multi-second latency spikes under load.

1. As the Crow Flies: The Haversine Formula

This calculates the straight-line distance between two points on a sphere (the Earth) using their Latitude and Longitude.

Pros

Extremely fast: Calculating 10,000 pairs takes milliseconds on a CPU.
No external data needed: Pure mathematics; zero reliance on APIs or map data.

Cons

Inaccurate: It ignores roads, rivers, tunnels, and one-way streets. In urban reality, actual driving distance is usually 1.2x to 1.5x the Haversine distance.

Python Example

import math

def haversine(lat1, lon1, lat2, lon2):
    R = 6371.0 # Earth radius in km

    dlat = math.radians(lat2 - lat1)
    dlon = math.radians(lon2 - lon1)

    a = (math.sin(dlat / 2)**2 + 
         math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) * 
         math.sin(dlon / 2)**2)

    c = 2 * math.atan2(math.sqrt(a), math.sqrt(1 - a))
    return R * c # Returns km

# Building the Distance Matrix:
def build_haversine_matrix(locations):
    n = len(locations)
    matrix = [[0] * n for _ in range(n)]
    for i in range(n):
        for j in range(n):
            if i != j:
                matrix[i][j] = haversine(
                    locations[i].lat, locations[i].lng,
                    locations[j].lat, locations[j].lng
                )
    return matrix

Practical usage: Amazon and Grab often use Haversine as a "Candidate Filter" to eliminate points that are obviously too far away before calling more expensive algorithms.

2. The Ideal Solution for the Base Problem: Routing Engine (OSRM / GraphHopper)

If your problem only requires delivering from a fixed warehouse to customers, without caring about real-time traffic or rush hour histories, and solely needs distance based on the actual existing road network — then this is the perfect, most cost-effective solution.

Prerequisite: This section assumes familiarity with Order Allocation Algorithms and how a VRP solver consumes a pre-built distance matrix.

You need a Routing Engine to load road network graph data. This data is usually downloaded for free from OpenStreetMap (OSM) — which the community constantly updates whenever a new road is built, an alley is removed, or weight restrictions change. When the roads change, you simply re-download the map file (.osm.pbf) and restart the engine.

Why not standard Dijkstra or A*?

If you run a standard Dijkstra or A* algorithm to find paths between 10,000 pairs on a city-wide map (which has millions of nodes), your server will freeze because the graph is too large to traverse per query.

Therefore, modern Routing Engines use "pre-processing" techniques:

Contraction Hierarchies (CH): Spends a few hours pre-analyzing the map to create "shortcuts" (virtual highways) between areas. At query time, the algorithm jumps across these shortcuts rather than traversing every alley → Query time drops to microseconds. The downside is that if a road closes, you must rebuild the entire map.
Multi-Level Dijkstra (MLD) / Customizable Route Planning (CRP): Divides the map into small cells. Updating the cost of a road only requires updating that specific cell (taking seconds) rather than the whole map. Great for real-time traffic injection.

The Open-Source Giants: OSRM and GraphHopper Distance Matrix

In custom logistics systems, the two most famous open-source engines are OSRM (C++) and GraphHopper (Java).

Feature	OSRM (Open Source Routing Machine)	GraphHopper
Language	C++	Java
Main Algorithms	MLD and CH	CH, A*, Landmark
Routing Profiles	Written in Lua (driving, cycling, walking)	Written in Java / YAML
Matrix API	Unbelievably fast (C++ optimized)	Good, but OSRM edges out on massive matrices
Flexibility	Quite rigid, hard to change rules at runtime	Highly flexible (Custom Models) allows runtime rule changes
Community	Backed by Mapbox, widely used for static routing	Easy to embed in Java apps, very flexible

Routing Profiles:
The route for a motorcycle navigating tight alleys is completely different from a 10-ton truck (which is banned from small roads). Both OSRM and GraphHopper allow you to define Profiles. In OSRM, you write Lua files: car.lua, truck.lua, bike.lua to tell the engine which roads are legal.

OSRM Table API: Lightning Fast Matrices

Assuming you self-host an OSRM server (very easy via Docker), it provides a table API that generates distance matrices optimally:

# Request to local OSRM to calculate a 3x3 matrix
curl "http://localhost:5000/table/v1/driving/106.70,10.77;106.71,10.78;106.72,10.79?annotations=distance,duration"

# Returns both durations (seconds) and distances (meters)
{
  "durations": [
    [0, 150, 320],
    [155, 0, 180],
    [330, 175, 0]
  ],
  "distances": [
    [0, 1200, 2400],
    [1250, 0, 1100],
    [2500, 1150, 0]
  ]
}

Note: OSRM Table API can return a 100x100 matrix (10,000 elements) in just a few dozen milliseconds!

Pros of self-hosting: Free, insanely fast, zero rate limits.
Cons: RAM heavy (especially if loading an entire country's map).

How to Calculate a Distance Matrix with GraphHopper

GraphHopper is the most developer-friendly open-source routing engine for building a distance matrix in Java or via a self-hosted HTTP API. Unlike OSRM, GraphHopper supports runtime routing rule changes via Custom Models — making it the preferred choice when your delivery fleet has vehicle-specific constraints (weight limits, road class restrictions).

Option A: GraphHopper Matrix API (Self-hosted)

Self-host the GraphHopper server with Docker and call the /matrix endpoint:

# Start GraphHopper server with a local OSM map file
docker run -d -p 8989:8989 \
  -v $(pwd)/data:/data \
  israelhikingmap/graphhopper \
  --url https://download.geofabrik.de/asia/vietnam-latest.osm.pbf \
  --host 0.0.0.0

import requests
import json

def build_graphhopper_distance_matrix(locations: list[dict]) -> dict:
    """
    Calculate a full GraphHopper distance matrix for a list of lat/lng points.
    Returns duration (seconds) and distance (meters) for every pair.
    Args:
        locations: List of dicts with 'lat' and 'lng' keys.
    Returns:
        dict with 'durations' and 'distances' 2D arrays.
    """
    url = "http://localhost:8989/matrix"

    # GraphHopper Matrix API requires [lng, lat] order (GeoJSON convention)
    points = [[loc["lng"], loc["lat"]] for loc in locations]

    payload = {
        "points": points,
        "profile": "car",                     # Routing profile: car, bike, foot
        "out_arrays": ["times", "distances"],  # Request both duration and distance
        "fail_fast": False                     # Return partial results if a pair is unreachable
    }

    response = requests.post(url, json=payload, timeout=30)
    response.raise_for_status()
    data = response.json()

    return {
        "durations": data["times"],       # N×N matrix in seconds
        "distances": data["distances"]    # N×N matrix in meters
    }

# Example: 3 warehouses / delivery points in Ho Chi Minh City
locations = [
    {"lat": 10.7712, "lng": 106.7011},  # Warehouse
    {"lat": 10.7780, "lng": 106.7100},  # Customer A
    {"lat": 10.7650, "lng": 106.6980},  # Customer B
]

matrix = build_graphhopper_distance_matrix(locations)
print(f"Duration matrix (seconds): {matrix['durations']}")
print(f"Distance matrix (meters):  {matrix['distances']}")
# Output:
# Duration matrix (seconds): [[0, 320, 185], [315, 0, 410], [180, 405, 0]]
# Distance matrix (meters):  [[0, 2100, 1350], [2050, 0, 2900], [1300, 2880, 0]]

Option B: GraphHopper Java SDK (Embedded)

For Java-based logistics backends, embed GraphHopper directly without a network hop:

// Embedded GraphHopper distance matrix calculation
// Purpose: Build an NxN distance matrix without requiring a running HTTP server
import com.graphhopper.GraphHopper;
import com.graphhopper.config.CHProfile;
import com.graphhopper.config.Profile;
import com.graphhopper.routing.util.EncodingManager;

GraphHopper hopper = new GraphHopper();
hopper.setOSMFile("/data/vietnam-latest.osm.pbf");
hopper.setGraphHopperLocation("/data/graph-cache");
hopper.setProfiles(new Profile("car").setVehicle("car").setWeighting("fastest"));
hopper.getCHPreparationHandler().setCHProfiles(new CHProfile("car"));
hopper.importOrLoad();

// Use GHMatrixAPI to compute full NxN matrix
// See: https://github.com/graphhopper/graphhopper/tree/master/web-api

GraphHopper vs. OSRM: Which to Choose for Distance Matrix?

Criterion	GraphHopper Distance Matrix	OSRM Table API
Speed (large matrices)	Fast (CH/MLD)	Slightly faster (C++ optimized)
Custom vehicle rules	✅ Runtime Custom Models	❌ Requires recompile + Lua
Java ecosystem	✅ Native SDK	❌ HTTP only
Docker ease	✅ Official image	✅ Official image
OSM data format	`.osm.pbf`	`.osm.pbf`
Best for	Flexible profiles, Java backends	Max throughput, static profiles

Rule of thumb: Use OSRM if you have a fixed vehicle type and need maximum raw speed. Use GraphHopper if you need to change routing rules at runtime (truck weight limits, toll avoidance, time-dependent costs).

3. Expensive Overkill: Commercial APIs (Google Maps / Mapbox)

If you are building a Ride-Hailing app that requires minute-perfect Estimated Time of Arrival (ETA) so customers don't cancel, you need real-time traffic data and historical rush hour patterns. In that case, you must use commercial APIs (like Google Maps).

However, for static delivery routing from a fixed warehouse, this is entirely overkill and extremely wasteful.

# Calling Google Maps Distance Matrix API (Very Expensive)
import requests

url = "https://maps.googleapis.com/maps/api/distancematrix/json"
params = {
    "origins": "10.77,106.70|10.78,106.71",
    "destinations": "10.77,106.70|10.78,106.71",
    "departure_time": "now",  # Current traffic
    "key": "YOUR_API_KEY"
}

The Exorbitant Cost

Google Maps charges about $0.005 per Element (Pair).
Returning to our 1 warehouse + 100 orders = 10,201 pairs.
Every time you run the allocation algorithm, you pay: 10,201 × $0.005 = $51.
If your warehouse dispatches 10 batches a day, you lose $510/day just to calculate distances!

This is exactly why domestic delivery companies self-host OSRM or GraphHopper instead of throwing money at Google Maps for warehouse routing. The OpenStreetMap network is more than adequate for this need.

4. System Design Strategies to Save Compute

To maintain high accuracy without overloading servers, large systems use these tricks:

Technique 1: Clustering

If 5 orders are going to the same apartment building, cluster them into a single coordinate. A 101x101 matrix drops to 97x97.

Technique 2: Hybrid Matrix (Haversine + OSRM)

Only calculate precise road distances for points that could realistically follow one another.

If Point A and Point B are > 15km apart (via Haversine), assign an infinite cost. The VRP solver will naturally never route a vehicle from A to B.
Only call OSRM for nearby pairs (< 5km).

Technique 3: H3 Hexagon Caching (How Uber/Grab does it)

Recalculating the distance between two nearby alleys day after day is a waste of resources. Logistics giants use Uber's H3 (Hexagonal Hierarchical Spatial Index) to cache distance results.

Why Hexagons and not Geohash (Squares)?

In a square grid, the distance from the center to the 4 straight edges differs from the distance to the 4 corners. In a hexagon grid, the distance from the center to all neighboring cells is exactly equal. This is critical in routing because distance error margins are uniformly controlled in all directions.

How it works:

1. Pick a Resolution:
H3 Resolution 9 is typically used (each hexagon is about 174 meters wide). This perfectly encapsulates a small residential block or a street segment.

2. Convert Coordinates (Lat/Lng) to H3 Index:

import h3

# Convert Point A and Point B
h3_A = h3.geo_to_h3(10.7712, 106.7011, 9) # Result: '8965a6a0033ffff'
h3_B = h3.geo_to_h3(10.7780, 106.7100, 9) # Result: '8965a6a0027ffff'

3. Check Cache (Redis) before invoking the Engine:

redis_key = f"route_cost:{h3_A}:{h3_B}"

cache_result = redis.get(redis_key)
if cache_result:
    return json.loads(cache_result) # Cache Hit! Zero computation.
else:
    # Cache Miss! Call OSRM
    result = call_osrm_api(lat1, lon1, lat2, lon2)

    # Save to Redis for next time (TTL = 30 days)
    redis.set(redis_key, json.dumps({
        "distance_m": result.distance,
        "duration_s": result.duration
    }), ex=2592000)

    return result

Pre-warming the Cache

Instead of waiting for a customer to order, the system can run a batch job at night:

Fetch all H3 cells that contain residential areas in the city.
Use OSRM (since it's free and local) to pre-calculate the distance between all H3 pairs (within 10km of each other).
Pump this data into Redis.

When the OR-Tools system runs the next day, it reads purely from Redis RAM at the speed of light without needing to traverse any graphs. Because road networks rarely change (only when there's long-term construction), the Cache Hit ratio can exceed 95%, giving you the speed of Haversine but the accuracy of a Routing Engine!

Summary of the Series: Order Fulfillment is not just a CRUD application tracking statuses. It is a symphony of event-driven microservices, real-time inventory algorithms, OR-Tools optimization, and ultra-fast Routing Engines processing spatial data.

References & Further Reading

🔗 Next Step: Now that you understand how to build an accurate distance matrix with GraphHopper, see how this feeds into Part 6 — Building a Mini Allocation Engine and the complete Series Executive Summary.

This post was originally published on my blog at GraphHopper Distance Matrix: Self-Host OSRM vs Haversine for Route Optimization.

[System Design] Part 4 — Amazon CONDOR & Anticipatory Shipping

Tuấn Anh — Thu, 18 Jun 2026 00:02:11 +0000

Amazon Fulfillment: The Three Tiers of Optimization

Amazon processes billions of orders annually through a network of over 175 fulfillment centers globally. To maintain their 1-2 day (or same-day) delivery guarantees, they built a 3-tier optimization architecture:

┌─────────────────────────────────────────────────────────────┐
│  TIER 1: ANTICIPATORY SHIPPING (Long-term — weeks/months)  │
│  → ML predicts demand → Moves inventory close to customers │
│     BEFORE they place an order                             │
├─────────────────────────────────────────────────────────────┤
│  TIER 2: REGIONALIZATION (Medium-term — days/weeks)        │
│  → Partitions the fulfillment network into autonomous zones│
│  → Ensures 70-80% of orders are fulfilled intra-region     │
├─────────────────────────────────────────────────────────────┤
│  TIER 3: CONDOR (Short-term — hours)                       │
│  → Continuously re-optimizes the fulfillment plan within   │
│     a 5-6 hour window before pick-and-pack begins.         │
└─────────────────────────────────────────────────────────────┘

Anticipatory Shipping — Shipping Before You Buy

A Crazy but Effective Idea

Amazon holds a patent (US Patent 8,615,473) describing a system that begins shipping items BEFORE a customer places an order. It sounds like science fiction, but it's a reality.

Traditional Model:
  Customer orders → Warehouse processes → Ships → Delivered (2-5 days)

Anticipatory Shipping:
  ML predicts: "Customers in Region X will buy 200 iPhone 16s in the next 3 days"
  → Amazon ships 200 iPhones from a central hub to local delivery hubs in Region X
  → Customer places order → The item is already locally staged → Delivered same-day!

ML Model Input Features

Input Feature	Significance
Purchase history	What do they buy, and how often?
Browsing behavior	What are they looking at? Cart abandonment?
Wishlists	Explicitly desired items
Seasonal patterns	Winter coats in November, sunscreen in June
Regional demographics	High-income areas? Young families? College towns?
Trending products	Items going viral on social media
Weather forecast	Rain forecasted → move umbrellas to local hubs
Events calendar	Black Friday, Prime Day, major sports events

Late-Select Addressing — The Key Technique

The Anticipatory Flow:

1. ML Model: "Zip code 10001 (NY) has an 87% probability of ordering 200 cases of water in the next 3 days."

2. System: Ships 200 cases from the Midwest Central Hub → NY Local Hub.
   These packages DO NOT HAVE A SPECIFIC CUSTOMER ADDRESS YET → They are just labeled "Destination: NY Hub".

3. Customer A in NY orders 2 cases of water:
   → The system assigns an address to 2 of the pre-staged cases at the NY Hub.
   → Delivered in 2 hours!

4. If predictions are slightly off (e.g., 50 cases remain unsold):
   → Amazon might run a targeted flash sale for that zip code.
   → Or re-route them back to the central hub.

CONDOR — Customer Order and Network Density OptimizeR

The Problem CONDOR Solves

When you place an order, Amazon doesn't process it immediately. There is a 5-6 hour window between the order being placed and the warehouse actually starting the pick-and-pack process. CONDOR exploits this window to optimize delivery routes.

17:00 — Order A is placed in Zone 1
  → CONDOR Plan v1: Ship from WH Alpha, individual truck.

17:15 — Order B is placed in Zone 1 (near Order A)
  → CONDOR Plan v2: Consolidate A+B onto the same route → saves 1 truck trip.

17:30 — Order C is placed in Zone 2 (along the same route)
  → CONDOR Plan v3: Consolidate A+B+C → highly dense, efficient route.

17:45 — Order D is placed in Zone 9 (opposite direction)
  → CONDOR Plan v4: Route 1 (A+B+C) + Route 2 (D only).

→ Every 15 minutes, CONDOR re-evaluates the entire network to find better plans.
→ Deadline: When the window closes (e.g., 23:00), the warehouse executes the final optimized plan.

The CONDOR Algorithm

CONDOR solves a variation of the Prize Collecting Vehicle Routing Problem (PCVRP), which is vastly more complex than standard VRP:

PCVRP:
  Maximize: Total "prize" (value of orders delivered on time)
  Minimize: Total transportation cost
  Subject to:
    - Capacity constraints (vehicle limits)
    - Time windows (delivery SLAs)
    - Fleet size limits
    - Network density bonuses: bundling orders in the same neighborhood significantly reduces cost-per-package.

Solving Techniques:
  1. Mathematical optimization (LP/MIP relaxation)
  2. Local search heuristics (2-opt, 3-opt swaps between routes)
  3. Iterative re-optimization (running the solver continuously as new data arrives)

The Major Breakthrough

Amazon has stated that CONDOR reduces the number of feasible routing decisions for a given area from millions to under 10 viable options, transforming an NP-hard problem into something solvable in near real-time.

Regionalization — Partitioning the Network

Prior to 2022, if a customer in New York ordered an item, it might have shipped from a warehouse in California (3,000 miles away) if the local warehouse was out of stock. This was incredibly inefficient.

Amazon restructured its US network into 8 autonomous regions:

Pre-Regionalization:
  Customer in NY → Order fulfilled from CA (3,000 miles) → 3-5 day delivery.

Post-Regionalization:
  Customer in NY → Order fulfilled from NJ or PA (100 miles) → Same/Next day delivery.

Results:
  - Average travel distance per package dropped by ~60%
  - Significant reduction in shipping costs
  - Delivery times dropped by 1-2 days
  - Massive reduction in carbon footprint

Regional Inventory Strategy

SKU "IPHONE-16-256GB":
  National Demand: 100,000/month

  Northeast Region (NY, NJ, PA):  25,000/mo → Stock 30,000
  West Region (CA, WA, OR):       20,000/mo → Stock 25,000
  South Region (TX, FL, GA):      18,000/mo → Stock 22,000
  ...

  Buffer: 22,000 kept at a central cross-dock facility for overflow/rebalancing.

Comparing Amazon vs. eBay vs. Regional Marketplaces

Aspect	Amazon	eBay	Regional Marketplaces
Model	1P + FBA (Owns warehouses)	Marketplace (Sellers ship)	Marketplace + Fulfillment (e.g., Shopee)
Facilities	175+ Global FCs	Seller warehouses + 3PLs	Regional fulfillment hubs
Allocation	CONDOR (Global/Continuous optimization)	Rule-based (Seller-defined)	Regional matching engines
Anticipatory	Yes (Late-Select Addressing)	No	No
Structure	8 Autonomous Regions (US)	Decentralized	Geographic partitioning

Lessons Learned for System Design

You don't need to build CONDOR to apply its principles:

Batch processing over real-time: Don't dispatch an order the second it arrives. Hold it in a 15-30 minute batch window and optimize the entire batch together. It is always mathematically superior.
Re-optimization: The best route at 17:00 may be terrible by 17:30 as new orders arrive. Run iterative optimizations.
Predictive placement: If data shows consistent regional demand, stage the inventory there beforehand.
Partitioning: Break massive NP-hard routing problems into smaller regional chunks to make them solvable.

Next, we explore split shipments, consolidation, and the last-mile delivery problem—which accounts for 53% of all logistics costs. Read Part 5 — Split Shipment, Consolidation & Last-Mile Delivery.

References & Further Reading

This post was originally published on my blog at Part 4 — Amazon CONDOR & Anticipatory Shipping.

[System Design] Uber RAMEN: How Ride-Hailing Apps Push Real-Time Notifications to Millions of Devices

Tuấn Anh — Tue, 16 Jun 2026 23:22:25 +0000

The Problem: Pushing Instant Notifications to Millions of Devices

Answer-first: Uber's RAMEN system maintains persistent gRPC bidirectional streams to every active driver app. When a match is made, the ride offer travels from the matching engine to the driver's phone in under 100ms — without polling. This is how millions of connections are held open simultaneously without crashing the backend.

When DISCO decides to match you with Driver John Doe, the system must:

Send the ride offer to exactly John Doe's phone (out of millions of connected phones).
Deliver it in milliseconds (not seconds).
Ensure the driver receives it even if their 4G connection is weak.
Simultaneously push the driver's location back to your app so you can watch the car move on the map.

There are two main approaches: Polling (asking continuously) and Push (proactively sending).

Polling vs. Push

Polling (The Old Way — Inefficient)

The Driver App asks the server every 3 seconds: "Do you have any rides for me?"

GET /api/v1/offers?driver_id=abc123
→ Response: { "offers": [] }  ← Nothing here

GET /api/v1/offers?driver_id=abc123  (3 seconds later)
→ Response: { "offers": [] }  ← Still nothing

GET /api/v1/offers?driver_id=abc123  (3 seconds later)
→ Response: { "offers": [...] }  ← Got a ride! But it's already 0-3 seconds delayed

The Issues:
  - 5 million drivers × 1 request/3s = 1.67 million requests/second (just asking)
  - 99% of requests return empty → Massive waste of server resources
  - Latency of 0-3 seconds → Another driver might accept the ride first
  - Battery drain: the radio chip must be constantly active

Push (RAMEN — Efficient)

The server maintains an OPEN connection with every driver app.
When a ride offer is ready, the server PROACTIVELY pushes it down instantly.

Driver App ◄═══ gRPC Stream (live connection) ═══► RAMEN Server

Advantages:
  - Latency: < 100ms (near-instantaneous)
  - No wasted empty requests
  - The radio chip is only active when there's actual data to receive
  - Significant battery savings

RAMEN — Real-time Asynchronous Messaging Network

RAMEN is the push messaging infrastructure built in-house by Uber. It maintains millions of concurrent persistent connections to push real-time data to rider and driver apps.

Three-Tier Architecture

┌────────────────────────────────────────────────────────────┐
│                     RAMEN Architecture                      │
│                                                              │
│  ┌──────────────────┐                                       │
│  │  Fireball Service │  "When to push?"                     │
│  │  (Decision Engine)│  • Consumes Kafka events              │
│  │                    │  • Evaluates business rules           │
│  │                    │  • Handles priority, localization     │
│  └────────┬─────────┘                                       │
│           │                                                  │
│           ▼                                                  │
│  ┌──────────────────┐                                       │
│  │  API Gateway      │  "What to push?"                     │
│  │  (Payload Builder)│  • Aggregates data from services      │
│  │                    │  • Builds message payloads            │
│  │                    │  • Serializes (Protobuf)              │
│  └────────┬─────────┘                                       │
│           │                                                  │
│           ▼                                                  │
│  ┌──────────────────┐                                       │
│  │  RAMEN Server     │  "How to push?"                      │
│  │  (Delivery Layer) │  • Manages millions of connections    │
│  │                    │  • Routes to the correct device       │
│  │                    │  • Guarantees at-least-once delivery  │
│  └──────────────────┘                                       │
│           │                                                  │
│           ▼                                                  │
│     Mobile Devices (Millions)                                │
└────────────────────────────────────────────────────────────┘

The Evolution of the Transport Protocol

Generation 1: Server-Sent Events (SSE) over HTTP/1.1

Client ──── HTTP/1.1 Connection ────► RAMEN Server
       ◄═══ SSE (Server → Client only) ═══

Characteristics:
  ✓ Simple, browser-friendly
  ✗ Unidirectional: only Server → Client
  ✗ ACK (acknowledgments) must be sent via separate HTTP POST requests (wasting connections)
  ✗ Head-of-line blocking: large messages block smaller heartbeats
  ✗ Text-based JSON: heavy payloads

Generation 2: gRPC over QUIC/HTTP3 (Current)

Client ◄══ gRPC Bidirectional Stream ══► RAMEN Server
              (HTTP/3 + QUIC)

Characteristics:
  ✓ Full-duplex: both sides send data simultaneously on the same connection
  ✓ Binary framing (Protobuf): small payloads, low CPU usage
  ✓ Multiplexing: multiple streams on 1 connection, no head-of-line blocking
  ✓ QUIC: UDP-based, much more resilient on weak mobile networks
  ✓ ACKs sent right on the current stream (no separate connection needed)
  ✓ Connection migration: switching networks (WiFi → 4G) doesn't break the connection

Scalability: Managing Millions of Connections

Apache Helix + ZooKeeper

RAMEN cannot run on a single server — it requires a cluster of hundreds of servers, each holding tens of thousands of live connections.

RAMEN Cluster Management:

ZooKeeper: Stores topology metadata (which servers are alive?)
Apache Helix: Manages sharding and automatic rebalancing

User UUID "abc123" → hash() → Shard 42 → RAMEN Server #7
User UUID "def456" → hash() → Shard 18 → RAMEN Server #3

If server #7 dies:
  1. Helix detects it (heartbeat timeout)
  2. Helix moves Shard 42 to server #9
  3. Client "abc123" reconnects → Load Balancer → server #9
  4. Continues receiving messages seamlessly

Stateful Servers — The Unique Challenge

Unlike standard REST API servers (which are stateless), RAMEN servers are stateful — each server holds specific TCP/gRPC sockets for specific users. Routing must be exact: a message for user "abc123" must reach the exact server holding that user's socket.

Routing Flow:

Fireball: "Push ride offer to driver abc123"
  │
  ▼
Routing Layer: hash("abc123") → Server #7
  │
  ▼
Server #7: Finds the socket for abc123 in memory → Pushes the message

Ensuring Reliability

At-Least-Once Delivery

Mobile networks are notoriously unreliable — 4G signals can drop for a few seconds and return. RAMEN guarantees that a message is delivered at least once by using:

Persistence Layer:

  Cassandra (Durable Storage)     Redis (In-Memory Cache)
  ┌────────────────────┐         ┌────────────────────┐
  │ Source of truth     │         │ Absorb traffic     │
  │ Stores messages    │◄───────│ bursts             │
  │ permanently for     │         │ Thundering herd    │
  │ retries             │         │ protection         │
  └────────────────────┘         └────────────────────┘

Flow:
  1. Message arrives → Written to Cassandra (backup)
  2. Cached in Redis (fast access)
  3. Pushed via gRPC stream to the device
  4. Device sends an ACK
  5. If no ACK is received within 10s → Retry from Cassandra
  6. Receives ACK → Marks as delivered

Sequence Numbers — Handling Reconnects

When a client loses its connection and reconnects, how does the server know which messages it has already received?

Every message has an incrementing sequence number:

Server → Client:
  [seq=1] Ride offer
  [seq=2] ETA update
  [seq=3] Driver location  ← Connection lost here

Client reconnects:
  "Last received: seq=2"

Server:
  → Resends everything from seq=3 onwards (doesn't resend seq=1, seq=2)
  [seq=3] Driver location (retry)
  [seq=4] Driver location (new)
  ...

Connection Draining — Preventing Thundering Herds

When deploying new code or scaling down a cluster, RAMEN cannot simply drop millions of connections at once — this would cause a thundering herd (millions of clients reconnecting simultaneously, crashing the entire system).

Graceful Shutdown Flow:

1. RAMEN Server #7 needs to shut down
2. Server #7 stops accepting NEW connections
3. Sends a "Graceful Disconnect" to all currently connected clients
   Message: { type: "DISCONNECT", backoff_hint_ms: random(1000, 30000) }
4. Each client receives a different backoff_hint:
   - Client A: waits 2.3 seconds, then reconnects
   - Client B: waits 15.7 seconds, then reconnects
   - Client C: waits 8.1 seconds, then reconnects
5. Reconnections are spread evenly over 30 seconds → no thundering herd

Fallback: Silent Push Notifications

When a driver app is pushed to the background by the operating system (due to Android/iOS power management), the gRPC stream will be closed. In this scenario, RAMEN uses Silent Push Notifications via APNs (Apple) / FCM (Google) to "wake up" the app:

Primary Flow (App in foreground):
  RAMEN Server ══► gRPC Stream ══► Driver App  ✓ (< 100ms)

Fallback Flow (App in background):
  RAMEN Server ──► FCM/APNs ──► OS wakes app ──► App reconnects gRPC
                                                    (Takes 1-5 seconds)

Grab's Approach: WebSocket + Istio

Grab didn't build a massive custom system like RAMEN; they use a simpler approach:

WebSockets for real-time bidirectional communication.
Istio Service Mesh manages routing, load balancing, and mTLS.
FCM/APNs act as fallbacks when the app is backgrounded.

Pros: Simpler to maintain, uses open standards. Cons: WebSockets over HTTP/1.1 suffer from head-of-line blocking and aren't as robust as gRPC/QUIC on weak networks.

Summary of the End-to-End Real-Time Flow

 THE COMPLETE REAL-TIME PIPELINE:

 ① Driver moves
    → GPS Sensor → Kalman Filter → Batches 3 points
    → gRPC Stream → Load Balancer → Location Service

 ② Location Service
    → Converts GPS → H3 Index
    → Produces to Kafka "driver.location.updates"

 ③ Kafka → Consumers:
    ├── Redis GEO (updates driver's location)
    ├── Flink (calculates supply-demand → Surge Pricing)
    └── Analytics Pipeline (Data Lake)

 ④ Rider requests a car
    → Demand Service → Kafka "ride.requests"

 ⑤ DISCO Matching Engine
    → Queries Redis (finds nearby drivers)
    → Routing Service (calculates actual ETA)
    → Hungarian Algorithm (batched matching)
    → Selects the optimal driver

 ⑥ RAMEN Push
    → Fireball (decision) → API Gateway (payload)
    → RAMEN Server → gRPC Stream → Driver Phone
    → "You have a new ride request!"

 ⑦ Driver accepts
    → Trip Service → Kafka "ride.status.changes"
    → RAMEN Push → Rider App: "Your driver is on the way!"
    → Location stream starts pushing driver's position to the Rider App
    → Car moves smoothly on the map 🚗

 Total elapsed time: < 2 seconds

Official Reference Sources

Source	Content
Uber Eng: H3 Hexagonal Index	Hexagonal gridding algorithm
Uber Eng: RAMEN	Push messaging architecture
Uber Eng: DISCO	Matching Engine
Uber Eng: DeepETA	ML model for ETA prediction
Grab Eng: Fulfilment Platform	Dispatch platform architecture
Grab Eng: DispatchGym	RL framework for dispatch
Lyft Eng: Real-time Map Matching	Kalman Filters & Map Matching
H3 Documentation	API reference for H3
Google S2 Geometry	API reference for S2

Congratulations on completing the series! You now deeply understand every architectural layer behind that smoothly moving car on your map. From the GPS sensor → Kalman Filter → Kafka → H3 → DISCO → RAMEN → Your App. **Every layer represents a fascinating distributed engineering challenge.**

This post was originally published on my blog at Uber RAMEN: How Ride-Hailing Apps Push Real-Time Notifications to Millions of Devices.

[System Design] Surge Pricing Algorithm: How Ride-Hailing Engines Calculate Surge Rate in Real Time

Tuấn Anh — Tue, 16 Jun 2026 12:02:51 +0000

Series context: This is Part 5 of the Real-Time Ride-Hailing Architecture series. For location ingestion and geospatial indexing, start at Part 1.

What Is Surge Rate? (And How Is It Calculated?)

Surge rate is the real-time price multiplier (e.g., 2.0×) applied by ride-hailing platforms when ride demand in a geographic zone exceeds available driver supply. It is recalculated every 30–60 seconds per H3 hexagon cell using a demand/supply ratio fed into a lookup table or ML model.

{{< faq q="What is surge rate?" >}}
Surge rate (also called surge pricing or surge multiplier) is the real-time price multiplier that ride-hailing platforms like Uber and Grab apply when demand for rides exceeds the available supply of drivers in a geographic zone. A surge rate of 2.0x means the rider pays twice the base fare.
{{< /faq >}}

{{< faq q="How is surge rate calculated?" >}}
The surge rate is calculated by a pricing engine that evaluates the ratio of incoming ride requests (demand) versus available drivers (supply) in a specific H3 hexagon cell over a rolling time window (typically 5 minutes). The ratio is fed into a lookup table or ML model that outputs the surge multiplier.
{{< /faq >}}

Why is Surge Pricing Necessary?

On New Year's Eve, during heavy rain, or at rush hour — the demand for rides skyrockets, but the number of available drivers remains unchanged. If prices were kept fixed:

Riders wouldn't be able to book a ride because there are no available drivers.
Drivers in other areas would have no incentive to move to the hot zones.
The system would be overwhelmed, leading to massive wait times.

Surge Pricing (or Dynamic Pricing) is not merely a tool to increase revenue — it is a marketplace equilibrium mechanism:

Price increases → Two simultaneous effects:

1. SUPPLY INCREASES: Drivers see red zones (high prices) on their heatmap
                     → They move toward those areas to earn more
                     → The number of available drivers in the area increases

2. DEMAND DECREASES: Riders see high prices → Some choose to wait, take a bus,
                     or walk → The number of ride requests drops

→ Supply and demand gradually return to EQUILIBRIUM
→ Wait times are reduced for riders who truly need a car

Surge Pricing Engine Architecture

┌────────────────────────────────────────────────────────────────┐
│                      DATA PIPELINE                              │
│                                                                  │
│  Kafka Topic              Flink Stream Processing                │
│  "driver.location"  ───►  ┌────────────────────┐                │
│  "ride.requests"    ───►  │  Supply-Demand      │                │
│                           │  Aggregator         │                │
│                           │  (per H3 cell,      │                │
│                           │   5-min window)     │                │
│                           └─────────┬──────────┘                │
│                                     │                            │
│                                     ▼                            │
│                           ┌────────────────────┐                │
│                           │  Pricing Engine     │                │
│                           │  (Surge Calculator) │                │
│                           └─────────┬──────────┘                │
│                                     │                            │
│                           ┌─────────▼──────────┐                │
│                           │  Redis Cache        │                │
│                           │  (Surge Multipliers) │                │
│                           └─────────┬──────────┘                │
│                                     │                            │
│                    ┌────────────────┼────────────────┐           │
│                    ▼                ▼                ▼           │
│              Rider App        Driver App       Matching Engine   │
│              (Shows price)    (Heatmap)        (Weighs cost)    │
└────────────────────────────────────────────────────────────────┘

Step 1: Geofencing with H3

Surge pricing is not calculated for an entire city — it is calculated for individual H3 hexagons. Uber uses Resolution 7 (each cell ~5 km²), which is large enough to be statistically significant but small enough to reflect hyper-local conditions.

Ho Chi Minh City is divided into ~200 H3 cells (Resolution 7)

Cell A (District 1 - Center): Supply=5,  Demand=30  → Surge 3.2x
Cell B (District 7 - Suburb): Supply=20, Demand=15  → Surge 1.0x (normal)
Cell C (Airport):             Supply=8,  Demand=40  → Surge 4.0x
Cell D (District 9 - Outskirts): Supply=12, Demand=3   → Surge 1.0x

Step 2: Calculating the Surge Multiplier

The Basic Model: Supply-Demand Ratio and Allocation Algorithms

Just like in e-commerce allocation algorithms that decide which warehouse fulfills an order, the surge engine evaluates resources dynamically.

surge_multiplier = f(demand / supply)

Where:
  supply  = Number of AVAILABLE drivers in the H3 cell (last 5 mins)
  demand  = Number of ride requests in the H3 cell (last 5 mins)

Example simple formula (illustrative):
  ratio = demand / supply

  if ratio <= 1.0:  surge = 1.0 (normal price)
  if ratio == 2.0:  surge = 1.5
  if ratio == 3.0:  surge = 2.0
  if ratio >= 5.0:  surge = 3.5 (maximum cap)

Advanced Model: Machine Learning

In reality, Uber doesn't use a simple linear formula. They use ML models to calculate optimal prices based on a multitude of factors:

Input Feature	Meaning
Supply count	Number of idle drivers in the cell
Demand count	Number of requests in a sliding window
Historical patterns	Supply-demand patterns by hour/day of the week
Weather data	Raining → demand rises, supply drops
Events	Large events (concerts, football games)?
Conversion rate	What % of riders still book at the current surge price?
Neighboring cells	Surge levels in adjacent cells (spillover effect)

The Feedback Loop — Preventing Over-Pricing

Continuous feedback loop:

1. Surge = 3.0x → Many riders cancel (conversion rate drops from 70% → 30%)
2. Engine realizes: price is too high, riders are abandoning
3. Lowers surge to 2.0x → Conversion rate recovers to 60%
4. Simultaneously, drivers arrive (supply increases) → ratio drops
5. Surge continues dropping to 1.5x → 1.0x

This entire process happens automatically over a few minutes.

Step 3: The Driver Heatmap

Surge pricing doesn't just affect the cost for the rider — it generates a Heatmap displayed on the driver's app, guiding them to areas with high demand.

Heatmap Visualization:

  ┌────────────────────────────────────┐
  │                                    │
  │      🟢          🟡               │
  │            🟢         🟡          │
  │      🟢   District 7     🔴       │
  │            🟢    🟡  🔴 District 1│
  │      🟢        🟡  🔴 🔴         │
  │                   🔴              │
  │               🟡                  │
  │                                    │
  └────────────────────────────────────┘

  🟢 = 1.0x (normal, surplus of drivers)
  🟡 = 1.5-2.0x (moderate demand)
  🔴 = 2.5x+ (very high demand, great earning potential)

Real-time Heatmap Updates

The heatmap is pushed to the driver app via WebSockets (or gRPC streams):

Server → WebSocket Push → Driver App

Payload every 30 seconds:
{
  "heatmap": [
    {"h3": "872a100d6ffffff", "surge": 3.2, "color": "#FF0000"},
    {"h3": "872a100d7ffffff", "surge": 1.0, "color": "#00FF00"},
    {"h3": "872a100d8ffffff", "surge": 1.8, "color": "#FFAA00"}
  ],
  "updated_at": "2026-05-06T20:30:00Z"
}

Predictive Surge — Anticipating Demand

Uber and Grab don't just react to current surges — they predict surges before they happen:

Predictive Model:

Inputs:
  - Current time: 17:00 (rush hour approaching)
  - Day: Friday (weekend → demand rises)
  - Weather: Rain forecasted at 17:30
  - Events: Concert at the Stadium at 20:00
  - History: The last 4 Fridays also surged to 2.5x at 17:30

Output:
  - Prediction: Surge will hit 2.8x in the Stadium area at 17:30
  - Action: Send notifications to nearby drivers 15 minutes BEFORE
    "High demand expected near the Stadium soon, drive there to earn more!"

Upfront Pricing vs. Surge Multiplier

The Old Model: Surge Multiplier (Uber before 2017)

Price shown to rider: "Surge 2.5x"
Final Price = Base Fare × 2.5
Problem: Riders didn't know the total cost before getting in → Surprises, complaints

The New Model: Upfront Pricing (Current Uber, Grab)

Shown to rider: "Price: 125,000 VND" (fixed before booking)

Price is calculated from:
  base_fare + (distance × per_km_rate) + (time × per_min_rate)
  + surge_premium
  + route_specific_adjustments (e.g., predicted traffic jams)

The rider knows the exact price upfront → Much more transparent

Storing the Surge State

-- Redis: Stores the surge multiplier for each H3 cell
-- Key pattern: surge:{resolution}:{h3_cell_id}
-- TTL: 60 seconds (auto-expires if not updated → falls back to 1.0x)

SET surge:7:872a100d6ffffff "3.2" EX 60
SET surge:7:872a100d7ffffff "1.0" EX 60
SET surge:7:872a100d8ffffff "1.8" EX 60

-- When Rider App requests a price:
GET surge:7:872a100d6ffffff → "3.2"
-- The API Gateway uses this value to calculate the Upfront Price

Anti-Abuse Mechanisms

Risk	Solution
Drivers deliberately turning off apps to create artificial scarcity	Detect patterns: many drivers going offline simultaneously → flag
Drivers only accepting high surge rides, rejecting normal rides	Low acceptance rate → lower priority in matching algorithm
Extremely high surges causing massive backlash	Maximum cap (e.g., 5.0x), soft caps based on conversion rates
"Flickering" surge (rapidly fluctuating prices)	Smoothing: surge can only increase/decrease by a max of 0.5x every 30 seconds

Surge Rate FAQ: Common Questions Answered

What is surge rate in ride-hailing?
Surge rate is the multiplier applied to a ride's base price during peak demand periods. A surge rate of 1.5x on a $10 base fare means the rider pays $15. The surge rate is calculated per geographic zone (H3 cell) and updated every 30–60 seconds.

Why does surge rate exist?
Surge rate is a market-clearing mechanism, not just a revenue tool. When demand outpaces supply, a higher surge rate simultaneously attracts more drivers to the area (supply increase) and filters out lower-urgency ride requests (demand decrease), restoring equilibrium and reducing wait times for riders who genuinely need a car.

What is a normal surge rate vs. a high surge rate?
A surge rate of 1.0x is the baseline (no surge). Rates between 1.2x–1.8x are considered moderate surge. Rates above 2.0x indicate heavy demand imbalance. Most platforms enforce a hard cap (e.g., 5.0x) to prevent extreme price spikes that damage user trust.

How long does a surge rate last?
Surge rates are recalculated every 30–60 seconds using a sliding window over the last 5 minutes of supply-demand data. In most cases, a surge event lasts 15–45 minutes before drivers repositioning to the zone restore equilibrium.

How does the surge pricing engine work technically?
The engine ingests driver location events and ride request events from a message broker (Kafka). A stream processor (Apache Flink) aggregates supply and demand counts per H3 cell on a 5-minute tumbling window. The output is a demand/supply ratio that maps to a surge multiplier via a lookup table or an ML model. The resulting multiplier is cached in Redis with a 60-second TTL and read by the API gateway at price-calculation time.

In the final part, we will explore RAMEN — Uber's real-time communication infrastructure, which solves the problem of pushing instant notifications to millions of devices simultaneously. Continue reading Part 6 — RAMEN & Real-time Communication.

This post was originally published on my blog at Surge Pricing Algorithm: How Ride-Hailing Engines Calculate Surge Rate in Real Time.

[System Design] Ride-Hailing Dispatch Algorithm: How Uber DISCO & Grab DispatchGym Match Drivers

Tuấn Anh — Mon, 15 Jun 2026 00:01:48 +0000

Every time you tap "Book Ride," a system makes dozens of decisions in under two seconds: Which driver? What route? What's the real ETA? This article breaks down exactly how the dispatch algorithm works — from the greedy approach that fails at scale, to the bipartite graphs, batched matching, and surge pricing mechanics that power Uber, Lyft, Grab, and Gojek today.

Why a Greedy Dispatch Algorithm Fails (Closest Driver Problem)

The first instinct when designing a matching system is to pair every customer with their nearest driver. However, this Greedy approach causes massive losses at a system-wide scale:

Example: 3 riders (R1, R2, R3) and 3 drivers (D1, D2, D3)

Greedy Matching (closest driver):
  R1 ← D1 (ETA 2 mins)  ✓
  R2 ← D3 (ETA 8 mins)  ← D2 was "taken" by R1, even though D2 is closer to R2
  R3 ← D2 (ETA 10 mins) ← Terrible outcome

  Total ETA: 2 + 8 + 10 = 20 minutes

Optimal Matching (global optimal):
  R1 ← D2 (ETA 3 mins)
  R2 ← D1 (ETA 3 mins)
  R3 ← D3 (ETA 4 mins)

  Total ETA: 3 + 3 + 4 = 10 minutes  ← 50% better!

Uber refers to this problem as Global Optimization — finding an assignment strategy that minimizes the total ETA of the entire system, rather than optimizing just for individual pairs.

Bipartite Graph Matching: The Mathematical Foundation (Lyft)

Before diving into the systems, it helps to understand the mathematical model that all ride-hailing matching engines share at their core.

Lyft formalizes dispatch as a bipartite graph matching problem:

Bipartite Graph:
  Set A (Riders):  { R1, R2, R3, R4 }
  Set B (Drivers): { D1, D2, D3, D4, D5 }

  Edges: every possible Rider ↔ Driver pair
  Edge Weight: cost of that match (e.g., ETA, driver rating, distance)

  Goal: Find a set of edges (a "matching") where:
    - No rider is matched to more than one driver
    - No driver is matched to more than one rider
    - The total cost of all selected edges is minimized

This is known as the Minimum Weight Bipartite Matching problem. The classical algorithm for solving it is the Hungarian Algorithm (also called the Kuhn-Munkres algorithm), which runs in O(n³) time.

Why Batching Matters for Bipartite Matching

The key insight is that you can only find a globally optimal bipartite matching if you have multiple riders and drivers available simultaneously. If you match greedily (one-by-one as requests arrive), you lose the ability to find the global optimum.

This is why all major ride-hailing platforms introduce a batching window:

Batching Strategy:
  1. Collect all ride requests in a 2-5 second window
  2. Build a complete Rider × Driver cost matrix
  3. Run the Hungarian Algorithm on the full batch
  4. Dispatch all assignments simultaneously

Result: System-wide optimal — not just locally optimal for each individual request

Approach	Latency	Global Optimality	Scalability
Pure Greedy	Near-zero	Poor	High
Batched Matching	2-5 seconds	Excellent	Medium
RL-Adaptive Batching	Dynamic	Excellent	High

Uber DISCO: The Core Dispatch Algorithm Architecture

DISCO (Dispatch Optimization) is Uber's matching system, responsible for pairing millions of ride requests with drivers every day.

Overall Architecture

┌─────────────────┐     ┌─────────────────┐
│  Rider App      │     │  Driver App      │
│  "Book Ride"    │     │  GPS, Status     │
└────────┬────────┘     └────────┬────────┘
         │                       │
         ▼                       ▼
┌─────────────────┐     ┌─────────────────┐
│  Demand Service │     │  Supply Service  │
│  (Ride Requests)│     │  (Driver Pool)   │
└────────┬────────┘     └────────┬────────┘
         │                       │
         └──────────┬────────────┘
                    ▼
         ┌─────────────────────┐
         │    DISCO Engine      │
         │                     │
         │  1. Candidate Filter │ ← S2/H3 Geospatial Query
         │  2. ETA Calculator   │ ← Routing Service + DeepETA
         │  3. Batch Optimizer  │ ← Hungarian Algorithm
         │  4. Dispatch         │ ← RAMEN Push
         └─────────────────────┘

Step 1: Candidate Filtering

When a ride request arrives, DISCO doesn't check every driver. It uses the rider's S2 Cell ID (or H3) to rapidly narrow down the list:

Input:  Rider location → S2 Cell Level 12
Action: Find all neighboring S2 cells (covering circle radius ~3km)
Query:  Redis/Memory → Retrieve list of drivers in those cells

Filters:
  ✓ Status = AVAILABLE (not currently carrying a passenger)
  ✓ Vehicle type matches (Car, Bike, SUV, etc.)
  ✓ Capacity is sufficient (if a group ride)
  ✓ Not blocked by the rider

Output: ~10-30 candidate drivers

S2 vs H3: Choosing the Right Spatial Index

Uber originally used S2 geometry (quad-tree squares) for spatial sharding but later developed H3 (hexagons). The difference matters for how the engine finds candidate drivers:

	S2 Geometry (Google)	H3 (Uber)
Cell shape	Square	Hexagon
Neighbor equidistance	No (diagonal cells are further)	Yes (all 6 neighbors equidistant)
Best for	Precise geofencing, hierarchical sharding	Proximity search, surge heatmaps
Used by	Lyft, early Uber DISCO	Modern Uber, Grab

Step 2: ETA Calculation with DeepETA

Straight-line (crow-fly) distance is meaningless in the real world — 500m straight-line might be a 3km drive (due to bridges, intersections, or one-way streets).

Candidate drivers: [D1, D2, D3, D4, D5]
Rider position: R1

Routing Service calculates actual ETA based on:
  - Digitized road network graph
  - Real-time traffic data
  - One-way streets, overpasses, tunnels
  - Rush hours / historical traffic patterns

Results:
  D1: 800m crow-fly → 2.3km road → ETA 4 mins (heavy traffic)
  D2: 1.2km crow-fly → 1.5km road → ETA 3 mins (clear roads)  ← Better!
  D3: 600m crow-fly → 3.8km road → ETA 7 mins (must U-turn)

Uber developed DeepETA — a hybrid deep learning approach that sits on top of the traditional routing engine:

DeepETA Architecture:
  1. Traditional Router → "Naive ETA" (e.g., 4 mins based on road graph)
  2. Deep Neural Network → "Residual Prediction" (e.g., +1.5 min due to
     traffic signal, weather, specific intersection complexity)
  3. Final ETA = Naive ETA + Residual

Input features to the DNN:
  - GPS coordinates (cleaned by Kalman Filter)
  - Time of day, day of week
  - Historical traffic at this location
  - Driver/vehicle attributes
  - Trip type (airport, downtown, etc.)

Kalman Filter role: Raw GPS signals are noisy in urban environments (tall buildings, tunnels). A Kalman Filter smooths the GPS stream before it feeds into DeepETA, ensuring the model learns from accurate positional data rather than jittery raw coordinates.

Step 3: Batched Matching

This is the most crucial and complex step. Instead of processing each request individually, DISCO batches requests within a short time window (a few seconds) and solves the optimal assignment problem for the entire batch.

Batching Window: 2 seconds

Within 2 seconds, the system receives:
  Ride Requests: [R1, R2, R3, R4, R5]
  Available Drivers: [D1, D2, D3, D4, D5, D6, D7]

Build Cost Matrix (ETA between every rider-driver pair):

         D1   D2   D3   D4   D5   D6   D7
  R1  [  3    5    8    2    7    9    4  ]
  R2  [  6    2    4    7    3    5    8  ]
  R3  [  4    7    3    5    6    2    9  ]
  R4  [  8    4    6    3    5    7    2  ]
  R5  [  5    6    2    8    4    3    7  ]

The Problem: Find a 1-to-1 assignment such that the total cost is minimized.
→ Hungarian Algorithm (or Auction Algorithm)

The Hungarian Algorithm

This is a classic algorithm used to solve the Assignment Problem with a time complexity of O(n³):

Input: N×M Cost Matrix (N riders, M drivers, M ≥ N)
Output: Optimal 1-to-1 assignment

Optimal Results:
  R1 ← D4 (ETA 2 mins)
  R2 ← D2 (ETA 2 mins)
  R3 ← D6 (ETA 2 mins)
  R4 ← D7 (ETA 2 mins)
  R5 ← D3 (ETA 2 mins)

  Total ETA: 10 minutes (compared to a Greedy approach which might yield 20+ minutes)

Ringpop — Distributed Coordination

DISCO runs on multiple servers. How do we ensure two different DISCO servers don't attempt to assign the exact same driver to two different riders?

Uber developed Ringpop — a consistent hashing library based on the SWIM (gossip) protocol:

Ringpop Hash Ring:

Server A ────── Server B ────── Server C ────── Server A
   │                │                │
   ▼                ▼                ▼
  Riders &       Riders &         Riders &
  Drivers        Drivers          Drivers
  in Zone 1      in Zone 2        in Zone 3

Each S2 Cell is hashed to a specific DISCO server.
→ All requests and drivers within the same area are processed by the SAME DISCO server.
→ No conflict during assignment.

SWIM Protocol: Every DISCO node periodically "pings" other nodes to check if they are alive. If a node goes down, the ring auto-rebalances — the S2 cells of the dead node are transferred to neighboring nodes in the ring.

Fault Tolerance: State Digest

The problem: If an Uber data center goes down abruptly, will the states of all in-flight trips be lost?

Uber's clever solution: Encrypted State Digest — DISCO periodically pushes the trip state (encrypted) down to be stored directly on the driver's phone.

Normal Flow:
  DISCO Server ◄──── state ────► Driver Phone
  (Source of truth)               (Backup copy)

When a data center crashes:
  1. A new DISCO Server boots up
  2. Driver phones reconnect
  3. DISCO requests driver phones to send back the state digest
  4. DISCO decrypts and restores the state of all in-flight trips
  5. The system continues operating without dropping a single ride

From Heuristics to Machine Learning: Gojek Jaeger

Gojek's evolution from a simple dispatch heuristic to a production ML system is a masterclass in how marketplace complexity forces you to rethink single-objective optimization.

The Problem with a Single ML Model

Gojek's initial approach used a single machine learning model to rank and select drivers. It worked — until it didn't. The model optimized aggressively for its primary metric (say, acceptance rate) and created feedback loops:

Feedback Loop Problem:
  1. Model learns: "Drivers in Zone A accept 90% of the time"
  2. Model always sends orders to Zone A drivers
  3. Zone A drivers get overwhelmed, acceptance rate drops
  4. Zone B drivers are idle, marketplace liquidity drops
  5. Riders in Zone B wait longer → lower satisfaction

A single-objective model cannot balance competing goals without explicit guardrails.

Jaeger: Multi-Objective Allocation

Gojek built Jaeger — a multi-objective allocation framework that simultaneously optimizes for:

Jaeger Optimization Objectives:
  ↓ Minimize pickup ETA          (rider experience)
  ↑ Maximize driver utilization  (driver earnings)
  ↑ Maximize acceptance rate     (marketplace flow)
  ↑ Ensure fairness              (prevent driver starvation)

Architecture:
  [Real-time Features]  ←── GPS, traffic, demand heatmaps
        │
        ▼
  [ML Models]           ←── Acceptance probability, ETA model
        │
        ▼
  [Manual Configs]      ←── Business rules, fairness floors
        │
        ▼
  [Jaeger Aggregator]   ←── Weights & combines all signals
        │
        ▼
  [Driver Score]        ←── Final ranking for dispatch

The Manual Configs layer is intentional: it gives business teams control to override ML decisions in edge cases (market launches, weather events, regulatory requirements) without retraining the entire model.

The Future: Reinforcement Learning & MDP in Dispatching (Grab DispatchGym)

The Hungarian algorithm solves the immediate assignment optimally. But it has a fundamental limitation: it only optimizes for the current batch. It cannot answer questions like:

Should I assign a driver now, or wait 30 seconds for a better match?
If I send this driver 5km across town, will that leave a coverage gap for the next surge?

This is where Reinforcement Learning (RL) and the Markov Decision Process (MDP) formulation come in.

Dispatch as a Markov Decision Process

An MDP models dispatch as a sequence of decisions where each action affects future states:

MDP Formulation for Dispatch:

State (S):
  - Current positions of all idle drivers
  - Pending ride requests and their locations
  - Time of day, predicted demand heatmap
  - Traffic conditions across all zones

Action (A):
  - Assign Driver D to Rider R
  - Hold Driver D idle (wait for a better request)
  - Reposition Driver D to Zone X (proactive repositioning)

Transition (T):
  - Probability that action A in state S leads to state S'
  - e.g., P(driver ends up in Zone B | assigned to trip starting in Zone A)

Reward (R):
  - Positive: completed trip, short pickup ETA, high acceptance
  - Negative: long idle time, deadhead mileage, rider cancellation

The key difference from greedy matching: the RL agent learns to make decisions that maximize cumulative long-term reward — not just the immediate reward for a single trip.

Grab DispatchGym

Grab built DispatchGym to make RL research accessible for dispatching problems. Its design solves a core challenge: training RL agents in production is dangerous (bad policies lose money and drivers). DispatchGym provides a safe, simulated environment:

DispatchGym Architecture:

┌───────────────────────────────────┐
│         Simulation Layer           │
│  - Replays historical trip data    │
│  - Injects synthetic demand spikes │
│  - Models driver behavior          │
└────────────────┬──────────────────┘
                 │  State observation
                 ▼
┌───────────────────────────────────┐
│         RL Agent (Policy)          │
│  - Gymnasium API compatible        │
│  - Trainable with any RL algo      │
│    (PPO, SAC, DQN...)              │
└────────────────┬──────────────────┘
                 │  Action (dispatch decision)
                 ▼
┌───────────────────────────────────┐
│        Reward Computation          │
│  - Total completed trips           │
│  - Average pickup ETA              │
│  - Driver earnings equity          │
│  - Acceptance rate                 │
└───────────────────────────────────┘

Deployment:
  Reward stabilizes → A/B test in production → Gradual rollout

Multi-Agent RL: When One Agent Isn't Enough

A single RL agent controlling an entire city's fleet hits the curse of dimensionality — the state-action space is too large. The solution is Multi-Agent Reinforcement Learning (MARL):

MARL for Dispatch:
  - Each geographic zone (or each driver) is an independent agent
  - Agents observe their local state (nearby riders, driver density)
  - Agents take local actions (match, hold, reposition)
  - Coordination mechanism ensures global objectives are met

Paradigm: Centralized Training, Decentralized Execution (CTDE)
  - During training: agents share global information to learn cooperation
  - During execution: each agent acts on local observations only
  - Result: Scales to city-wide fleets without centralized bottleneck

Academic research on MARL-based dispatch suggests wait time reductions of 25–40% compared to greedy baselines, with significant improvements in driver idle mileage — though real-world results vary by city density and fleet size.

Grab's Fulfilment Platform Architecture

Grab doesn't just run ride-hailing — it runs food, groceries, express delivery, and financial services on the same driver network. Early on, each vertical had its own dispatch engine, causing massive inefficiency.

Grab solved this with the Fulfilment Platform — a unified three-layer architecture:

┌────────────────────────────────────────┐
│         Business Verticals             │
│  GrabCar | GrabFood | GrabMart | ...   │
└───────────────┬────────────────────────┘
                │ Demand signals
                ▼
┌────────────────────────────────────────┐
│         Fulfilment Platform            │
│  - Unified dispatch engine             │
│  - Supply shaping & driver incentives  │
│  - Global optimization across verticals│
└───────────────┬────────────────────────┘
                │ Infrastructure
                ▼
┌────────────────────────────────────────┐
│         Technology Infrastructure      │
│  - DynamoDB (OLTP: live orders)        │
│  - MySQL partitioned (OLAP: analytics) │
│  - 1,000+ microservices on AWS/GCP     │
└────────────────────────────────────────┘

The critical innovation: a driver finishing a GrabFood delivery can be immediately available for a GrabCar ride — the same Fulfilment Platform sees the full picture and can optimize across verticals. This eliminated "dead time" between trips and significantly improved driver earnings.

Key Metrics for the Matching Engine

Metric	Meaning	Target
P50/P99 Matching Latency	Time from request received to dispatch	< 2 seconds (P99)
Acceptance Rate	% of drivers accepting when offered	> 85%
ETA Accuracy	Error margin between predicted and actual ETA	< 20%
Match Rate	% of requests successfully matched	> 95%
Total Wait Time	Total time a rider waits (rider + driver travel time)	Minimize
Driver Idle Mileage	Distance driven without a passenger	Minimize
Marketplace Liquidity	Balance of supply/demand across zones	Maintain

FAQ: Dispatch Algorithms

What is a dispatch algorithm?
A dispatch algorithm is a system used by ride-hailing platforms like Uber and Grab to optimally match riders with available drivers. It goes beyond finding the closest driver by using batched matching and global optimization to minimize the total wait time (ETA) for all users.

How does Uber match riders and drivers?
Uber uses a system called DISCO that batches ride requests every 2-5 seconds and applies the Hungarian algorithm for global optimization across the entire batch. This ensures system-wide efficiency, not just locally optimal matches for each individual request.

Why is reinforcement learning used in ride-hailing?
RL models treat dispatching as a Markov Decision Process (MDP), allowing the system to optimize for long-term marketplace liquidity and driver earnings, rather than just immediate wait times. Platforms like Grab use RL via DispatchGym to learn when to hold a driver for a better future match, rather than always assigning immediately.

What is the difference between S2 and H3 in dispatch systems?
S2 (Google) uses square cells and is excellent for precise geofencing and hierarchical sharding. H3 (Uber) uses hexagonal cells, which offer uniform adjacency — all 6 neighbors are equidistant — making proximity searches and surge pricing heatmaps more accurate for real-time matching.

Next, we will look into Surge Pricing — the dynamic pricing system based on real-time supply and demand ratios. Continue reading Part 5 — Surge Pricing: Dynamic Pricing Based on Real-time Supply and Demand.

This post was originally published on my blog at Ride-Hailing Dispatch Algorithm: How Uber DISCO & Grab DispatchGym Match Drivers.

[System Design] H3 Geospatial Indexing: How Uber Finds Nearby Drivers with Hexagonal Spatial Index

Tuấn Anh — Sun, 14 Jun 2026 02:06:33 +0000

The Problem: Finding a Needle in a Haystack

Answer-first: Uber and Grab find the nearest available driver in under 100ms by dividing the Earth's surface into hexagonal cells (H3 index at Resolution 8, each ~0.74 km²). Instead of calculating distance to every driver, they look up only the 7 cells nearest to the rider — reducing millions of comparisons to dozens.

When you tap "Book" on Grab, the system must find the most suitable driver within a radius of a few kilometers. But the system is tracking millions of drivers simultaneously. The naive approach — calculating the distance from you to every driver — is impossible:

The Naive Approach (Brute Force):
  SELECT * FROM drivers
  WHERE ST_Distance(driver_location, rider_location) < 2000 -- 2km
  ORDER BY ST_Distance(driver_location, rider_location)

With 5 million drivers → 5 million distance calculations
Latency: takes seconds → Unacceptable

The solution: Divide the map into a grid (Spatial Indexing) to narrow down the search space from millions to just a few dozen.

Method 1: Geohash

Geohash encodes coordinates (latitude, longitude) into a character string. The longer the string, the smaller and more precise the grid square:

Coordinates: 10.7769, 106.7009 (District 1, HCMC)
Geohash: w3gvk1   (cell ~1.2km × 0.6km)
         w3gvk1e  (cell ~153m × 153m)
         w3gvk1e7 (cell ~38m × 19m)

Important characteristic: Prefix sharing
  w3gvk1e  ← Cells sharing a prefix are located near each other
  w3gvk1f
  w3gvk1g

Advantages

Simple and easy to understand.
Database-friendly: you can use a LIKE query (WHERE geohash LIKE 'w3gvk1%').
Redis supports it natively (GEOADD, GEOSEARCH).

A Serious Disadvantage: The Edge Problem

Geohash divides the map into squares. Two points right next to each other, but situated on the edges of two completely different squares, will have entirely different prefixes — the system won't find the driver even if they are only 50 meters away.

┌──────────┬──────────┐
│          │          │
│  w3gvk1  │  w3gvk3  │  ← Two cells with completely different prefixes
│          │          │
│    ●──────────○     │  ← Very close but in different cells
│          │          │
└──────────┴──────────┘

The Workaround: Always search the current cell + the 8 neighboring cells (a 3x3 grid). But this creates many false positives in the corners.

Method 2: H3 — Uber's Hexagonal Grid

Uber developed H3 (Hexagonal Hierarchical Spatial Index) to resolve all the limitations of Geohash. It is an open-source spatial indexing system that partitions the entire surface of the Earth into hexagons.

Why are hexagons better than squares?

Squares (Geohash):             Hexagons (H3):

┌────┬────┬────┐                 ╱╲    ╱╲
│    │    │    │                ╱    ╲╱    ╲
│  d │  ? │    │               │  d  ││  d  │
│    │    │    │                ╲    ╱╲    ╱
├────┼────┼────┤                 ╲╱    ╲╱
│  d │  ● │    │                 ╱╲  ● ╱╲
│    │    │    │                │  d  ││  d │
├────┼────┼────┤                ╲    ╱╲    ╱
│    │    │    │                 ╲╱    ╲╱
└────┴────┴────┘                 ╱╲    ╱╲
                                │  d  ││  d │
d = distance to ●              ╲    ╱╲    ╱
                                  ╲╱    ╲╱

Squares: 4 near neighbors (edges)   Hexagons: 6 neighbors
         4 far neighbors (corners)            ALL at the SAME distance!
         → Uneven distances                   → Uniform distance

Equidistant: Every neighboring cell of a hexagon is an equal distance from its center. With squares, corner cells are √2 times (~1.41 times) further away than edge cells. This unevenness creates biases in search algorithms.

16 Levels of Resolution

H3 supports 16 levels of resolution, from level 0 (covering continents) to level 15 (a few square meters):

Resolution	Average Area	Used For
0	~4,357,449 km²	Continental level
4	~1,770 km²	City/Province level
7	~5.16 km²	Surge Pricing zones
8	~0.74 km²	Finding drivers (most common)
9	~0.105 km²	Precise matching
12	~0.003 km²	Street-level

Uber uses Resolutions 7-9 for the vast majority of its operations: finding drivers, calculating surge pricing, and analyzing supply and demand.

K-Ring — Neighborhood Search

A K-Ring is a collection of all cells within K steps from a center cell:

K=0 (center cell only):     K=1 (center cell + 6 neighbors):

       ╱╲                           ╱╲    ╱╲
      ╱  ╲                        ╱    ╲╱    ╲
     │ ●  │                       │    ││    │
      ╲  ╱                        ╲    ╱╲    ╱
       ╲╱                          ╲╱    ╲╱
                                    ╱╲  ● ╱╲
     1 cell                       │    ││    │
                                   ╲    ╱╲    ╱
                                    ╲╱    ╲╱
                                    7 cells

K=2:  19 cells     K=3: 37 cells

Algorithm to find drivers:

Convert the rider's coordinates into an H3 index (resolution 8).
Calculate K-Ring(1) → 7 hexagonal cells.
Find all drivers currently in these 7 cells from Redis.
If not enough drivers are found → expand to K-Ring(2) → 19 cells.
Sort by actual ETA (not just straight-line distance).

Code Example (Go)

import "github.com/uber/h3-go/v4"

// Convert coordinates to H3 index
func findNearbyDrivers(riderLat, riderLng float64, radius int) []string {
    // Resolution 8: each cell is ~0.74 km²
    riderCell := h3.LatLngToCell(h3.LatLng{Lat: riderLat, Lng: riderLng}, 8)

    // K-Ring: get all cells within a radius of K steps
    searchCells := h3.GridDisk(riderCell, radius)

    var driverIDs []string
    for _, cell := range searchCells {
        // Look up in Redis: key = H3 cell, value = list of driver IDs
        cellKey := fmt.Sprintf("drivers:h3:%s", cell.String())
        ids := redisClient.SMembers(ctx, cellKey).Val()
        driverIDs = append(driverIDs, ids...)
    }
    return driverIDs
}

Method 3: Google S2 Geometry

S2 (developed by Google) is also utilized by Uber in their DISCO system (Matching Engine). S2 divides the Earth's surface into square cells based on a projection onto a cube (following a Hilbert Curve).

Characteristics:

Each cell is represented by a 64-bit integer → much faster for comparing, hashing, and sharding.
30 levels of resolution.
Google Maps, Foursquare, and MongoDB Geospatial all use S2.

import "github.com/golang/geo/s2"

// S2: Find all cells covering a 2km radius from a point
func getCoveringCells(lat, lng float64, radiusM float64) []s2.CellID {
    center := s2.PointFromLatLng(s2.LatLngFromDegrees(lat, lng))
    cap := s2.CapFromCenterAngle(center, s2.Angle(radiusM/6371000.0))

    coverer := &s2.RegionCoverer{
        MinLevel: 14,
        MaxLevel: 16,
        MaxCells: 20,
    }
    return coverer.Covering(cap)
}

Storing Locations: Redis GEO

The locations of all online drivers are stored entirely in RAM (Redis) because query speeds are < 1ms. Traditional databases are not used because they are far too slow for 1.25 million writes/second.

Approach 1: Redis GEO (Built-in)

-- Add/update a driver's location
GEOADD drivers:active 106.7009 10.7769 "driver:abc123"

-- Find drivers within a 2km radius
GEOSEARCH drivers:active FROMLONLAT 106.7009 10.7769 BYRADIUS 2 km ASC COUNT 20
-- Result: ["driver:abc123", "driver:def456", ...]

Approach 2: Redis SET + H3 Index (Uber's approach)

-- Each H3 cell is a Redis SET containing driver IDs
SADD drivers:h3:882a100d6dfffff "driver:abc123"
SADD drivers:h3:882a100d6dfffff "driver:def456"

-- When a driver moves to a new cell:
SREM drivers:h3:882a100d6dfffff "driver:abc123"  -- Remove from old cell
SADD drivers:h3:882a100d71fffff "driver:abc123"  -- Add to new cell

-- TTL to automatically clean up offline drivers:
-- Combine SET + EXPIRE or use a Sorted Set with timestamps

Comparing the Two Approaches

Feature	Redis GEO	Redis SET + H3
Radius Search	✅ Built-in `GEOSEARCH`	❌ Must calculate K-Rings manually
Sharding	❌ Hard to shard (1 key holds everything)	✅ Each H3 cell is a separate key → easy to shard
Scale Limit	~a few million	~tens of millions
Surge/Analytics Integration	❌	✅ Count drivers/cell → calculate supply/demand instantly

Uber uses the second approach (Redis + H3) because its sharding capabilities and integration with the Pricing Engine are far superior.

Consistent Hashing — Distributing GEO Data

When a massive city like Ho Chi Minh City has 200,000 online drivers, a single Redis node might not have enough RAM. The solution is Consistent Hashing to distribute the H3 cells across multiple Redis nodes:

H3 Cell "882a100d6dfffff"  →  hash() → Node A
H3 Cell "882a100d71fffff"  →  hash() → Node B
H3 Cell "882a100d73fffff"  →  hash() → Node A

Neighboring cells (a K-Ring) might reside on different nodes
→ A K-Ring query = a scatter-gather operation across multiple nodes

References & Further Reading

Next, we will delve into the backbone of the entire system — Apache Kafka — where every GPS event, ride request, and acceptance flows. Continue reading Part 3 — Event Streaming: The Apache Kafka & Flink Backbone.

This post was originally published on my blog at H3 Geospatial Indexing: How Uber Finds Nearby Drivers with Hexagonal Spatial Index.

[System Design] GPS Location Ingestion at Scale: gRPC Streaming, MQTT & Kalman Filter in Ride-Hailing

Tuấn Anh — Sat, 13 Jun 2026 10:38:16 +0000

The Challenge: Millions of Drivers, Every 4 Seconds

Answer-first: Uber and Grab handle 1.25 million GPS write operations per second from ~5 million active drivers. HTTP REST fails at this scale due to per-request TCP+TLS handshake overhead. The solution is persistent connections (gRPC streams or MQTT) with Protobuf serialization, Kalman Filter noise reduction, and batched coordinate uploads — cutting network calls by 67% while maintaining sub-200ms end-to-end latency.

Grab has approximately 5 million drivers operating in Southeast Asia. Uber has over 5 million drivers globally. If every driver sends a GPS coordinate every 4 seconds, the system must receive:

5,000,000 drivers ÷ 4 seconds = 1,250,000 GPS packets / second

That is 1.25 million write operations per second — just for location data alone, not counting other requests. Traditional HTTP REST cannot handle this load efficiently.

Why is HTTP REST Unsuitable?

Connection Overhead

Every HTTP request requires:

TCP Handshake (3-way handshake)
TLS Handshake (an additional 1-2 round trips if using HTTPS)
HTTP Headers (~200-800 bytes of overhead per request)
Closing the connection (or keep-alive timeouts)

With a GPS payload of only about 50-100 bytes (latitude, longitude, timestamp, speed, heading), the HTTP overhead is 5-10 times larger than the actual data.

Battery Drain

Every time a new HTTP connection is established, the phone's radio chip (4G/5G) must transition from an idle state to an active state, consuming significant energy. If a driver works 8-12 hours a day, the battery will drain rapidly.

The Solution: Ultra-lightweight Protocols

1. MQTT (Message Queuing Telemetry Transport)

MQTT was originally designed for IoT devices with limited bandwidth and small batteries — perfect for continuous GPS tracking.

Why MQTT is efficient:

Persistent Connection: Handshakes only once, then continuously sends coordinates without re-establishing the connection.
Tiny Headers: Only 2 bytes of overhead compared to hundreds of bytes for HTTP.
Flexible Quality of Service (QoS):
- QoS 0: Fire-and-forget — suitable for GPS because a new coordinate will arrive a few seconds later anyway.
- QoS 1: Guarantees delivery at least once.
Last Will and Testament (LWT): If a driver loses connection abruptly, the MQTT broker automatically notifies the server that the driver has gone offline.

MQTT Data Flow:

Driver App → MQTT Publish → MQTT Broker Cluster → Kafka
  Topic: "driver/{driver_id}/location"
  Payload: {
    "lat": 10.7769,
    "lng": 106.7009,
    "ts": 1717689600,
    "speed": 35.2,
    "heading": 127,
    "accuracy": 8
  }

2. gRPC Bidirectional Streaming

Uber later transitioned to using gRPC Streams (and QUIC/HTTP3) instead of MQTT for many real-time data streams.

Advantages over MQTT:

Protobuf serialization: Binary payloads are 3-10 times smaller than JSON.
Bidirectional: The server can push data back (ride offers, navigation) over the same connection.
Flow control: gRPC has built-in backpressure, crucial when servers are overloaded.
Multiplexing: HTTP/2 allows multiple streams over a single TCP connection, avoiding head-of-line blocking.

// gRPC service definition for Location Streaming
service LocationService {
  // Driver sends a continuous stream of coordinates
  rpc StreamLocation(stream LocationUpdate) returns (stream ServerCommand);
}

message LocationUpdate {
  double latitude = 1;
  double longitude = 2;
  int64 timestamp_ms = 3;
  float speed_mps = 4;       // meters/second
  float heading_degrees = 5;  // direction (0-360)
  float accuracy_meters = 6;  // GPS accuracy
  string driver_id = 7;
}

message ServerCommand {
  oneof command {
    NewRideOffer ride_offer = 1;
    NavigationUpdate nav = 2;
    PingRequest ping = 3;
  }
}

Optimization Techniques on the Mobile App

Batching — Grouping Coordinates

Instead of sending each coordinate individually, the app batches 3-5 GPS points and sends them together:

Without batching:      With batching:
GPS 1 → Send          GPS 1 ─┐
GPS 2 → Send          GPS 2 ─┤ → Send 1 batch
GPS 3 → Send          GPS 3 ─┘
GPS 4 → Send          GPS 4 ─┐
GPS 5 → Send          GPS 5 ─┤ → Send 1 batch
GPS 6 → Send          GPS 6 ─┘

6 network calls        2 network calls (67% savings)

Adaptive Frequency

The app doesn't need to send coordinates at a strict 4-second interval. It adjusts the frequency based on context:

Driver State	GPS Frequency	Reason
Waiting for a ride (idle)	Every 15-30 seconds	Saves battery, driver isn't moving much
Moving to find passengers	Every 4-5 seconds	Needs updated location for matching
On-trip (carrying a passenger)	Every 2-3 seconds	Needs high accuracy for ETA and mapping
High speed (>60 km/h)	Every 1-2 seconds	Moving fast, location changes rapidly
Stationary (at a red light)	Paused	GPS hasn't changed, 100% savings

Dead Reckoning — Position Interpolation

Between receiving GPS points, the rider app uses Dead Reckoning (trajectory prediction) to make the car appear to move smoothly on the map:

Actual received GPS positions: ● (every 4 seconds)
Interpolated positions: ○ (every 100ms)

●───○───○───○───○───○───○───●───○───○───○───○───●
t=0                         t=4s                t=8s

Simple interpolation formula:
  predicted_lat = last_lat + (speed × cos(heading) × Δt)
  predicted_lng = last_lng + (speed × sin(heading) × Δt)

Filtering GPS Noise: The Kalman Filter

GPS signals from phones are often noisy — especially in cities with tall buildings where signals reflect (the urban canyon effect). The result: cars on the map jump erratically or appear to drive through buildings.

How the Kalman Filter Works

The Kalman Filter is a continuous predict → update algorithm:

Loop for every new GPS point:

1. PREDICT:
   Based on the previous position, speed, and heading,
   predict the current position using a physics model.

   predicted_position = previous_position + velocity × Δt

2. UPDATE (Correct):
   Compare the prediction with the actual received GPS point.
   Assign weights: trust the prediction more, or the GPS more?

   If GPS accuracy = 5m (accurate): trust the GPS more
   If GPS accuracy = 50m (noisy):   trust the prediction more

   final_position = weighted_average(predicted, gps_measured)

Lyft Engineering described in their technical blog that they use a Marginalized Particle Filter — a more advanced version of the Kalman Filter — which can track multiple possible trajectories simultaneously when it's uncertain which road the car is on (e.g., a main road vs. a parallel frontage road).

Map Matching — Snapping to the Road

After filtering the noise, the system performs Map Matching — snapping the filtered coordinates onto a digitized road network:

Raw GPS (after Kalman):   Map Matched:

    ○  ○                     ●──●
   ○    ○                   ●    ●
  ○      ○   ──────►       ●      ●
   ○    ○                   ●    ●
    ○  ○                     ●──●

(Scattered coordinates)   (Snapped closely to the road)

The Complete Location Pipeline Architecture

Driver Phone GPS Sensor
        │
        ▼
  ┌───────────────┐
  │  Kalman Filter │ (On the phone)
  │  + Batching    │
  └───────┬───────┘
          │ gRPC Stream / MQTT
          ▼
  ┌───────────────┐
  │  Load Balancer │ (Nginx / Envoy)
  └───────┬───────┘
          │
          ▼
  ┌───────────────┐
  │  Location     │ → Validate, enrich, convert to H3 index
  │  Service      │
  └───────┬───────┘
          │ Produce to Kafka
          ▼
  ┌───────────────┐
  │  Apache Kafka  │  Topic: "driver.location.updates"
  └───┬───┬───┬───┘
      │   │   │
      ▼   ▼   ▼
   Redis  Flink  Data Lake
   GEO    (Stream  (S3/HDFS
   Cache  Processing) Analytics)

Real-world Numbers

Metric	Estimated Value
Concurrent online drivers (Grab SEA)	~1.5 million
Average GPS frequency	Every 4 seconds
Location Service Throughput	~375,000 msg/s
GPS payload size (Protobuf)	~40-60 bytes
Raw GPS bandwidth	~15-22 MB/s
End-to-end latency (phone → Redis)	< 200ms

Next, we will explore how Uber uses the H3 algorithm to divide the map into millions of hexagons and find the closest driver in the blink of an eye. Continue reading Part 2 — Geospatial Indexing: H3, S2 Geometry & Redis GEO.

This post was originally published on my blog at GPS Location Ingestion at Scale: gRPC Streaming, MQTT & Kalman Filter in Ride-Hailing.

The Landscape of Core Banking Developers: Building Financial Systems from Scratch

Tuấn Anh — Sat, 06 Jun 2026 15:17:38 +0000

This post is the Executive Summary of my 8-part long-form series: *Learning Path to Become a Core Banking Developer** originally published on my blog.*

Who is a Core Banking Developer?

A Core Banking Developer is a software engineer responsible for building, operating, and extending the system that processes all of a bank's core financial operations — from managing accounts, processing money transfers, and calculating interest rates, to ensuring every single penny is recorded with absolute accuracy.

Unlike a typical developer, a mistake for a Core Banking Developer doesn't just mean a 404 error page — it means customers' money being lost, duplicated, or the general ledger becoming unbalanced. This intense pressure defines their entire approach to writing code and designing systems.

Why is this field special?

1. Absolute Accuracy

In regular software development, "eventual consistency" is often acceptable. In Core Banking, a transaction either completely succeeds or does not happen at all. There is no in-between state. This is why ACID database transactions are an indispensable foundation.

2. Extremely High Concurrency

Millions of users can perform transactions simultaneously within the same second. The system must handle concurrency without allowing race conditions that could lead to incorrect deductions or double credits.

3. Compliance and Legal Requirements

Every action in a Core Banking system must have an audit trail. The State Bank, tax authorities, and international organizations reserve the right to review the entire transaction history at any given time.

The Knowledge Map of a Core Banking Developer

If you want to transition from a regular backend engineer to a financial systems architect, you need to bridge both domain knowledge and heavy technical architecture.

┌─────────────────────────────────────────────────────────────────┐
│                   CORE BANKING DEVELOPER                        │
│                                                                 │
│  DOMAIN KNOWLEDGE          TECHNICAL SKILLS                     │
│  ─────────────────          ────────────────                    │
│  • Double-Entry (GL)       • Database (ACID, Locking)           │
│  • CASA (Deposits)         • Distributed Transactions           │
│  • Lending (Credit)        • Event-Driven Architecture          │
│  • Payments & Clearing     • API Design (REST/gRPC)             │
│  • Trade Finance           • Security & Encryption              │
│                                                                 │
│  STANDARDS & PROTOCOLS     ARCHITECTURE PATTERNS                │
│  ─────────────────────     ─────────────────────                │
│  • ISO 8583 (Card/ATM)     • Saga Pattern                       │
│  • ISO 20022 (SWIFT)       • Outbox Pattern                     │
│  • BIAN Framework          • CQRS & Event Sourcing              │
│  • PCI-DSS                 • Idempotency Keys                   │
└─────────────────────────────────────────────────────────────────┘

The Market Landscape

Popular Core Banking Systems

To understand the ecosystem, here are the dominant legacy and modern systems you will encounter in the banking industry (with examples of adoptions in regional markets like Vietnam):

System	Core Technology	Banks Using It
Temenos T24	Java, jBASE/BASIC	Techcombank, VPBank, MB Bank, Sacombank
Oracle Flexcube	Java EE, Oracle DB	VietinBank, BIDV
Infosys Finacle	Java	Agribank (Under deployment)
In-house (Custom)	Go, Java, Kotlin	MoMo, ZaloPay, VCB Digibank

Next-Generation Trends

Digital banks and fintechs are no longer purchasing off-the-shelf monolithic Core Banking systems — they are building their own Core Banking systems using Microservices. This represents a massive opportunity for full-stack developers with a mindset for distributed systems.

Ready to dive deeper?

This was just the executive summary. If you want to actually build these systems, I have written a complete step-by-step roadmap for developers:

Part 1 — The Double-Entry Ledger Foundation (The mental foundation you cannot skip)
Part 2 — Core Banking Domain: CASA & Lending
Part 3 — Database Design for Financial Transactions
Part 4 — Modern Core Banking Architecture
Part 5 — International Integration Standards (ISO 8583)
Part 6 — Security, Compliance & Audit
Part 7 — Build a Mini Core Banking System from Scratch

👉 Read the full 8-part series on my blog

I'm Lê Tuấn Anh, a Go Backend Architect specializing in microservices migrations and financial platforms. If you found this useful, let's connect on LinkedIn.