DEV Community: Black Lover

The Real Guide to Enterprise Role & Permission Architecture

Black Lover — Sun, 29 Mar 2026 07:40:11 +0000

There's a common pattern in access control tutorials: they explain roles, permissions, a pivot table, and maybe a caching layer — then call it enterprise-ready.

It isn't.

Real enterprise authorization fails in predictable ways:

A terminated employee retains access for 45 minutes because of stale cache
A SaaS tenant leaks data through a missing scope
An audit system can't prove who changed a permission because it only logs the action, not the before/after state
A super admin impersonates a user but the logs show the user did the action

This guide closes those gaps. We'll cover:

Object-level scoping
Tenant isolation that actually works
A complete permission resolution pipeline
Cache invalidation strategy
Audit log design for compliance
A production database schema

Stack: Examples use Laravel + PostgreSQL + Redis. The architecture patterns apply to any stack — Go, Node, .NET developers: the concepts translate directly.

01 — RBAC Is a Starting Point, Not a Destination

Role-Based Access Control (RBAC) is the correct foundation. The classic model is simple:

User → Role → Permissions
Admin → [users.create, users.delete, reports.read]
Staff → [posts.create, posts.update]

This breaks down the moment you need any of the following:

A user with two roles
A permission that only applies to the user's own records
A permission scoped to a specific tenant
The ability to revoke a single permission from a user who would otherwise inherit it through a role

Common mistake: Shipping basic RBAC and planning to "add scoping later" almost always means a rewrite. Scoping needs to be in the data model from day one — retrofitting it into 80 existing permission checks is painful.

Hybrid RBAC — The Correct Starting Model

Roles are a convenience shortcut for grouping permissions. Users should be able to inherit permissions from roles and have individual overrides — including explicit denies.

User
 ├── Roles[]  → Permissions (inherited, ALLOW)
 └── Direct   → Permissions (override, ALLOW or DENY)

// Resolution: union of role permissions,
// minus any direct DENY overrides

02 — Global Permissions Are an Anti-Pattern

Most tutorials treat permissions as system-wide boolean flags: you either have users.update or you don't. In practice, the question is almost always: which users can you update?

A users.update flag with no scope means a junior manager can edit the CEO's account. Scope is not a feature you add later — it is the entire point of fine-grained access control.

Three Levels of Permission Scope

Design permissions with an optional scope suffix from the start:

// Level 1 — Global (no scope)
users.update          // can edit any user — dangerous

// Level 2 — Ownership scope
users.update:own      // can only edit own profile

// Level 3 — Contextual scope
users.update:department  // only edit users in same dept
users.update:tenant      // only users in same tenant
invoices.void:threshold  // only if amount < policy limit

Enforcing Scope in Laravel Policies

The scope resolver lives in your Policy, not your Permission check. This keeps the permission name stable while business rules evolve:

class UserPolicy
{
    public function update(User $actor, User $target): bool
    {
        // Global permission — full access
        if ($actor->hasPermission('users.update')) {
            return true;
        }

        // Ownership scope
        if ($actor->hasPermission('users.update:own')) {
            return $actor->id === $target->id;
        }

        // Department scope
        if ($actor->hasPermission('users.update:department')) {
            return $actor->department_id === $target->department_id;
        }

        return false;
    }
}

Pattern: Always check the most-permissive scope first, then narrow down. This makes authorization logic readable and easy to audit.

03 — Tenant Isolation That Actually Works

Most RBAC guides mention multi-tenancy in one sentence: "add a tenant_id column." This is not an architecture — it's a column. Without enforcement, it's a liability.

The Three Failure Modes

Failure Mode	Impact
Developer forgets `where tenant_id = ?` in one query	Full cross-tenant data leak
System roles shared across tenants	Tenant A's admin can escalate using Tenant B's role definitions
Permission cache uses only `user_id` as key	Cached permissions from Tenant A bleed into Tenant B session

Solution 1: ORM-Level Global Scope

Never rely on developers remembering the tenant filter. Enforce it at the model level:

class TenantScope implements Scope
{
    public function apply(Builder $builder, Model $model): void
    {
        $builder->where(
            $model->getTable() . '.tenant_id',
            tenant()->id()
        );
    }
}

// Applied to model — can never be forgotten
class Role extends Model
{
    protected static function booted(): void
    {
        static::addGlobalScope(new TenantScope);
    }
}

Solution 2: Tenant-Scoped Role Definitions

Roles must be either system-wide (tenant_id NULL) or tenant-local. They must never be shared implicitly:

CREATE TABLE roles (
    id          BIGINT PRIMARY KEY,
    tenant_id   BIGINT REFERENCES tenants(id) ON DELETE CASCADE,
    -- NULL = system/global role (e.g., "Super Admin")
    -- SET  = tenant-local role (e.g., "Acme Corp Billing Manager")
    name        VARCHAR(100) NOT NULL,
    slug        VARCHAR(100) NOT NULL,
    UNIQUE(tenant_id, slug)
);

Solution 3: Compound Cache Keys

Cache permission sets per user per tenant. Never use user ID alone as the cache key:

// WRONG — leaks across tenants
$key = "permissions:{$userId}";

// CORRECT — scoped per tenant session
$key = "permissions:{$tenantId}:{$userId}";

// With tag-based invalidation
Cache::tags(["tenant:{$tenantId}", "user:{$userId}"])
    ->remember($key, now()->addMinutes(10), fn() => $this->resolvePermissions());

04 — The Complete Permission Resolution Pipeline

Every permission check should travel through the same deterministic pipeline. No shortcuts, no if ($user->role === 'admin') checks scattered through controllers.

┌─────────────────────────────────────────────────────────────────┐
│ Step 1: Authenticate & Load Tenant Context                     │
│ - Resolve user identity, active tenant, tenant feature flags   │
│ - Fail fast if tenant suspended or session revoked             │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 2: Load Role Permissions (tenant-scoped)                  │
│ - Fetch all roles assigned to user within this tenant          │
│ - Union all permission slugs from those roles                  │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 3: Merge Direct ALLOW Overrides                           │
│ - Add permissions granted directly to user (bypasses roles)    │
│ - Useful for one-off grants without role creation              │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 4: Apply Explicit DENY Overrides (most missed step)       │
│ - Remove any permissions explicitly denied at user level       │
│ - DENY always beats ALLOW                                      │
│ - How you revoke a single permission from inherited role       │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 5: Check Feature Flags                                    │
│ - Even if permission exists, is feature enabled for plan?      │
│ - Separate concern but evaluated in same pipeline              │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 6: Run Policy (Object-Level)                              │
│ - Execute Policy class: ownership, department match, state     │
│ - This is where scoping is enforced                            │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 7: Grant or Deny + Log                                    │
│ - Emit decision                                                │
│ - On DENY: log reason                                          │
│ - On sensitive ALLOW: also log                                 │
│ - Never silently fail                                          │
└─────────────────────────────────────────────────────────────────┘

Implementation Example

class PermissionResolver
{
    public function resolve(User $user, int $tenantId): PermissionSet
    {
        return Cache::tags(["tenant:{$tenantId}", "user:{$user->id}"])
            ->remember(
                "permissions:{$tenantId}:{$user->id}",
                now()->addMinutes(10),
                function () use ($user, $tenantId) {
                    // Step 1: Role permissions
                    $rolePerms = $user->rolesInTenant($tenantId)
                        ->flatMap(fn($r) => $r->permissions)
                        ->pluck('slug')
                        ->unique();

                    // Step 2: Merge direct ALLOWs
                    $allows = $user->directPermissions($tenantId, 'allow')
                        ->pluck('slug');

                    // Step 3: Collect explicit DENYs
                    $denies = $user->directPermissions($tenantId, 'deny')
                        ->pluck('slug');

                    // Step 4: Final set = (role + allow) - deny
                    return new PermissionSet(
                        $rolePerms->merge($allows)->diff($denies)
                    );
                }
            );
    }
}

05 — Cache Invalidation: The Part That Gets You Fired

Caching permission sets is necessary at scale. But stale permission caches are a security vulnerability, not a performance trade-off. A suspended user with cached users.delete is an active threat for however long your TTL runs.

What Must Invalidate the Cache

Event	Action
User assigned to or removed from a role	Flush user's permission cache
Direct permission override added/removed	Flush user's permission cache
Role's permission set changes	Flush ALL users with that role
User account suspended or deactivated	Immediate flush + revoke tokens
User switches active tenant	Flush previous tenant cache

Event Handlers

// When role permissions change — flush ALL users with this role
public function onRolePermissionsUpdated(Role $role): void
{
    Cache::tags(["role:{$role->id}", 'permissions'])->flush();
}

// When user is suspended — immediate flush
public function onUserSuspended(User $user): void
{
    Cache::tags(["user:{$user->id}", 'permissions'])->flush();

    // Also revoke active API tokens immediately
    $user->tokens()->delete();
}

Recommended TTLs by Risk Level

System Type	Max TTL	Rationale
Healthcare / Financial	2 min	Compliance requires near-real-time revocation
Enterprise SaaS	10 min	Balance between performance and access lag
Internal tools	30 min	Lower risk, slower user lifecycle changes

06 — Audit Logs That Pass a Compliance Audit

Logging user_id + action + timestamp is not an audit log. It's a weak activity feed.

A real audit log must answer:

What was the exact state before and after this change?
Who approved it?
Was this a super admin acting as another user?

The Full Audit Log Schema

CREATE TABLE audit_logs (
    id                  BIGSERIAL PRIMARY KEY,

    -- Identity
    user_id             BIGINT NOT NULL,       -- who triggered the action
    acting_on_behalf_of BIGINT,               -- impersonator (if any)
    tenant_id           BIGINT NOT NULL,

    -- Action
    action              VARCHAR(100) NOT NULL, -- e.g. "permission.grant"
    status              VARCHAR(20) NOT NULL,  -- "success" | "denied" | "error"

    -- Resource
    model_type          VARCHAR(100),
    model_id            BIGINT,

    -- State snapshots (critical for compliance)
    old_value           JSONB,                 -- before state
    new_value           JSONB,                 -- after state

    -- Request context
    ip_address          INET,
    user_agent          TEXT,
    request_id          UUID,                  -- for distributed tracing

    created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

The Impersonation Problem

When a super admin impersonates another user to debug an issue, every action they take should log both identities. If the log only shows the impersonated user's ID, you cannot reconstruct what actually happened during an incident review.

AuditLog::record([
    'user_id'             => auth()->id(),
    // If impersonating, this differs from user_id
    'acting_on_behalf_of' => session('impersonating_as'),
    'tenant_id'           => tenant()->id(),
    'action'              => 'role.assign',
    'status'              => 'success',
    'model_type'          => 'User',
    'model_id'            => $target->id,
    'old_value'           => ['roles' => $before],
    'new_value'           => ['roles' => $after],
    'ip_address'          => request()->ip(),
    'request_id'          => request()->header('X-Request-ID'),
]);

07 — Production Database Schema

Here's the complete schema with annotations. Key additions over basic RBAC:

Entity Relationship Diagram

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│     users       │     │    tenants      │     │   modules       │
├─────────────────┤     ├─────────────────┤     ├─────────────────┤
│ id (PK)         │     │ id (PK)         │     │ id (PK)         │
│ name            │     │ name            │     │ name            │
│ email           │     │ slug            │     │ slug            │
│ ...             │     │ ...             │     │ ...             │
└────────┬────────┘     └────────┬────────┘     └────────┬────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   user_roles    │     │     roles       │     │ permission_groups│
├─────────────────┤     ├─────────────────┤     ├─────────────────┤
│ user_id (FK)    │────▶│ id (PK)         │     │ id (PK)         │
│ role_id (FK)    │     │ tenant_id (FK)  │◀────│ module_id (FK)  │
│ tenant_id (FK)  │     │ name            │     │ name            │
│ granted_by (FK) │     │ slug            │     │ ...             │
│ expires_at      │     │ ...             │     └────────┬────────┘
│ ...             │     └────────┬────────┘              │
└─────────────────┘              │                       │
         │                       ▼                       ▼
         │              ┌─────────────────┐     ┌─────────────────┐
         │              │role_permissions │     │  permissions    │
         │              ├─────────────────┤     ├─────────────────┤
         │              │ role_id (FK)    │────▶│ id (PK)         │
         └──────────────│ permission_id   │     │ group_id (FK)   │
                        │ (FK)            │     │ name            │
                        └─────────────────┘     │ slug (scope)    │
                                                │ ...             │
┌─────────────────┐                              └─────────────────┘
│user_permissions │
├─────────────────┤
│ user_id (FK)    │◀─────────────────────────────────────────────┐
│ tenant_id (FK)  │                                              │
│ permission_id   │     ┌─────────────────┐     ┌─────────────────┐
│ type (ALLOW/DENY│     │tenant_feature_  │     │   audit_logs    │
│ granted_by (FK) │     │     flags       │     ├─────────────────┤
│ ...             │     ├─────────────────┤     │ id (PK)         │
└─────────────────┘     │ tenant_id (FK)  │     │ user_id (FK)    │
                        │ feature         │     │ acting_on_behalf│
                        │ enabled         │     │ tenant_id (FK)  │
                        │ ...             │     │ old_value (JSONB│
                        └─────────────────┘     │ new_value (JSONB│
                                                │ request_id (UUID│
                                                │ ...             │
                                                └─────────────────┘

Key Tables with Annotations

-- Users table
CREATE TABLE users (
    id          BIGSERIAL PRIMARY KEY,
    tenant_id   BIGINT REFERENCES tenants(id),
    name        VARCHAR(255),
    email       VARCHAR(255) UNIQUE,
    is_active   BOOLEAN DEFAULT true,
    suspended_at TIMESTAMPTZ,
    created_at  TIMESTAMPTZ DEFAULT NOW()
);

-- user_roles — includes expires_at for time-bound access
CREATE TABLE user_roles (
    user_id     BIGINT REFERENCES users(id) ON DELETE CASCADE,
    role_id     BIGINT REFERENCES roles(id) ON DELETE CASCADE,
    tenant_id   BIGINT REFERENCES tenants(id) ON DELETE CASCADE,
    granted_by  BIGINT REFERENCES users(id),
    expires_at  TIMESTAMPTZ,  -- NULL = never expires
    created_at  TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY (user_id, role_id, tenant_id)
);

-- user_permissions — includes ALLOW|DENY type
CREATE TABLE user_permissions (
    user_id         BIGINT REFERENCES users(id) ON DELETE CASCADE,
    permission_id   BIGINT REFERENCES permissions(id) ON DELETE CASCADE,
    tenant_id       BIGINT REFERENCES tenants(id) ON DELETE CASCADE,
    type            VARCHAR(10) NOT NULL CHECK (type IN ('ALLOW', 'DENY')),
    granted_by      BIGINT REFERENCES users(id),
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY (user_id, permission_id, tenant_id)
);

-- roles — tenant_id is nullable (NULL = system-wide)
CREATE TABLE roles (
    id          BIGSERIAL PRIMARY KEY,
    tenant_id   BIGINT REFERENCES tenants(id) ON DELETE CASCADE,
    name        VARCHAR(100) NOT NULL,
    slug        VARCHAR(100) NOT NULL,
    description TEXT,
    created_at  TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(tenant_id, slug)
);

-- permissions — linked to groups for manageable UI
CREATE TABLE permissions (
    id          BIGSERIAL PRIMARY KEY,
    group_id    BIGINT REFERENCES permission_groups(id),
    name        VARCHAR(100) NOT NULL,
    slug        VARCHAR(100) UNIQUE NOT NULL,  -- e.g., "users.update:own"
    description TEXT
);

-- tenant_feature_flags — checked in permission pipeline
CREATE TABLE tenant_feature_flags (
    tenant_id   BIGINT REFERENCES tenants(id) ON DELETE CASCADE,
    feature     VARCHAR(100) NOT NULL,
    enabled     BOOLEAN DEFAULT true,
    PRIMARY KEY (tenant_id, feature)
);

08 — When to Use Spatie — And When to Outgrow It

Spatie's laravel-permission is excellent for most applications. But it has architectural constraints worth knowing:

Capability	Spatie (default)	Custom Architecture
Basic RBAC	✅ Excellent	✅ Full control
Multi-tenancy	⚠️ Manual / plugin	✅ Native
Explicit DENY	❌ Not supported	✅ First-class
Permission scoping	⚠️ Convention only	✅ Schema-enforced
Cache invalidation	⚠️ Single flat key	✅ Tag-based
Setup time	✅ Hours	⚠️ Days / weeks

Recommendation: Start with Spatie for MVPs and side projects. Build the custom architecture when you have:

Multi-tenancy requirements
Explicit deny use cases
Compliance obligations (HIPAA, SOC2, etc.)

The decision point is usually around 5–10 developers or Series A scale.

09 — The Architecture Checklist

Before calling your permission system enterprise-ready:

Foundation

[ ] Hybrid RBAC: users have roles and direct permission overrides
[ ] Explicit DENY type on user-level overrides — DENY beats ALLOW
[ ] Permission slugs include optional scope (:own, :department, :tenant)

Multi-Tenancy

[ ] Tenant ID enforced at ORM level via global scope — not per-query
[ ] Roles are nullable-tenant: system-global or tenant-local, never shared implicitly
[ ] Cache keys are compound: permissions:{tenant_id}:{user_id}

Performance & Invalidation

[ ] Cache uses tags for role-level invalidation
[ ] TTL is <15 min; user suspension immediately flushes cache and revokes tokens

Architecture

[ ] Permission resolution is a single, centralized pipeline — no ad-hoc checks
[ ] Feature flags are checked in the pipeline, not separately

Audit & Compliance

[ ] Audit logs capture old_value, new_value, acting_on_behalf_of, and request_id
[ ] user_roles.expires_at enables time-bound grants

Maintainability

[ ] Permissions organized into groups and modules — admin UI stays manageable

Summary

Concept	Anti-Pattern	Correct Approach
Permission checking	`if ($user->isAdmin())` scattered everywhere	Centralized permission resolver with pipeline
Multi-tenancy	`where tenant_id = ?` in each query	ORM-level global scope + compound cache keys
Cache invalidation	Fixed TTL only	Tag-based invalidation on role/user changes
Audit logs	User + action + timestamp only	Full before/after snapshots + impersonation tracking
Permission scoping	Global flags only	Scoped suffixes (`:own`, `:department`, `:tenant`)
Role management	User has exactly one role	Hybrid RBAC: roles + direct ALLOW/DENY overrides

Building Namma Push: The Open‑Source, Pure‑Rust Alternative to Firebase Cloud Messaging

Black Lover — Sun, 29 Mar 2026 07:36:51 +0000

📱 The Push Notification Problem

Every modern app needs to send push notifications. Whether it’s a ride‑hailing service alerting a driver of a new trip, a fintech app confirming a payment, or a messaging app delivering a chat message — notifications are critical.

The industry standard is Firebase Cloud Messaging (FCM) from Google. It’s easy to set up, works across platforms, and is free for moderate usage. But as your app grows, FCM’s limitations become painful:

Data privacy — Google sees who you’re messaging and when.
Cost — At scale, FCM can cost $50–100 per million notifications.
Latency — 200–2000ms is common, especially for cross‑platform messages.
Lock‑in — Your entire notification infrastructure is tied to Google’s ecosystem.
No direct Android connection — FCM goes through Google Play Services, which doesn’t work on devices without Google (e.g., in China).

And on iOS, FCM is just a wrapper around Apple’s APNs, adding another hop and more latency.

What if you could have full control over your notification infrastructure? What if you could deploy your own push server, connect directly to devices, and achieve sub‑10ms latency — all while keeping your data private and reducing costs by 70%?

That’s exactly what we built with Namma Push.

🚀 What Is Namma Push?

Namma Push is an open‑source, self‑hosted push notification platform written entirely in Rust. It replaces FCM and APNs with a unified, high‑performance system that you run on your own infrastructure.

Key capabilities:

Direct gRPC/QUIC connections to mobile, web, desktop, and IoT devices — no third‑party intermediary.
Sub‑50ms P99 latency for active clients (often sub‑10ms).
WhatsApp‑style wake‑up — optional FCM/APNs messages to wake apps that are killed, while the actual notification content stays private.
Horizontal scalability — from 10,000 to millions of concurrent connections using Redis Cluster.
One Rust core — shared code for all client SDKs (iOS, Android, Web, Desktop, IoT).
Full observability — Prometheus, Grafana, Jaeger, Loki included.
Multi‑tenant — host many applications on a single cluster with isolated rate limits and API keys.
Cost‑efficient — self‑hosted, pay only for your own infrastructure (70%+ savings compared to FCM).

🏗️ Architecture Overview

Namma Push is designed as a cloud‑native, horizontally scalable system. Here’s a high‑level view:

┌─────────────────────────────────────────────────────────────┐
│                  Producer Applications                       │
│          (gRPC / REST with API Key)                         │
└─────────────────────────────┬───────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────┐
│           Namma Push Gateway (Rust / tonic)                 │
│   • gRPC server (port 50051)                                │
│   • QUIC/HTTP3 (port 50052)                                 │
│   • REST Admin API (port 9090)                              │
│   • Tenant validation, rate limiting, consistent hashing   │
└─────────────────────────────┬───────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                   Redis Cluster (Shared State)              │
│   • Priority streams (CRITICAL / HIGH / NORMAL / LOW)      │
│   • Dead Letter Queue (DLQ)  • Wake‑Up Queue                │
│   • Presence store (active connections)                     │
└─────────────────────────────┬───────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────┐
│               Delivery Orchestrator (Rust Workers)          │
│   • Per‑shard readers                                       │
│   • Client presence check                                   │
│   • Platform‑specific delivery (APNs, WebPush, MQTT)       │
│   • Exponential backoff retries                             │
└─────────────────────────────┬───────────────────────────────┘
                              │
    ┌─────────────────────────┼─────────────────────────┐
    ▼                         ▼                         ▼
┌───────────┐           ┌───────────┐           ┌───────────┐
│ iOS SDK   │           │ Android   │           │ Web SDK   │
│ (Swift +  │           │ (Kotlin + │           │ (WASM/JS) │
│  Rust)    │           │  Rust)    │           │           │
└───────────┘           └───────────┘           └───────────┘

🔌 The Magic: One Rust Core for All Platforms

The heart of Namma Push is a single Rust library that handles:

gRPC/QUIC connections with automatic reconnection
Local storage of pending notifications (SQLite on mobile, IndexedDB on web)
Message queuing and delivery
Heartbeat management

This core is then wrapped for each platform:

Android: JNI bindings + Kotlin foreground service
iOS / macOS: C‑bridge + Swift (APNs token registration)
Web: WebAssembly (wasm-bindgen) + JavaScript service worker
Desktop: Native Rust (Tauri) or Electron
IoT: no_std Rust with MQTT or CoAP

Benefits of this approach:

Code reuse — the same networking and logic works everywhere.
Performance — Rust’s zero‑cost abstractions and memory safety.
Portability — the core runs on servers, browsers, and microcontrollers.

📱 Android: Google‑Free Forever

One of the most exciting aspects of Namma Push is its Android implementation — it works completely without Google Play Services or FCM.

How It Works

The Android SDK contains a foreground service with START_STICKY flag.
This service maintains a persistent gRPC/QUIC connection to your Namma Push server.
When the app is in the background or even swiped away, the service continues running (Android shows a persistent notification, which can be hidden on Android 14+).
An AlarmManager health check restarts the service if it’s killed by aggressive battery optimisations.
Notifications are delivered instantly over the gRPC connection.

No FCM, no Google Play Services required. Works on any Android device, even in China.

If you want an extra layer of reliability, you can optionally enable FCM as a wake‑up helper — but it’s not needed for most use cases.

📱 iOS: Leveraging APNs Without Sacrificing Privacy

iOS is more restrictive: Apple does not allow long‑running background services. To wake an app that is killed, you must use APNs. But we use APNs only as a wake‑up channel — not to deliver the actual notification.

The iOS SDK registers for remote notifications and sends the device token to your Namma Push server.
When your server detects that the client is offline, it sends a silent APNs push (content-available: 1) with no user‑visible payload.
The device receives this push, wakes the app (if allowed), and the app reconnects via gRPC to fetch pending notifications.
All notification content stays on your server — Apple never sees the title, body, or data.

This gives you the reliability of APNs for wake‑up while keeping your data private.

🌐 Web Push: Native, No FCM Required

For web browsers, Namma Push uses the W3C Web Push Protocol with VAPID authentication — the same technology used by Chrome, Firefox, Edge, and Safari. No FCM, no external service.

The server generates its own VAPID keys.
The Web SDK (Rust compiled to WebAssembly) subscribes to push using those keys.
Notifications are sent directly from your server to the browser’s push service.

This is fully self‑contained and respects user privacy.

🧠 Smart Delivery & Offline Handling

Namma Push is not just a dumb relay — it’s intelligent about delivery.

Priority queues — CRITICAL, HIGH, NORMAL, LOW. Critical notifications bypass queues for immediate delivery.
Dead Letter Queue (DLQ) — Failed deliveries are stored with exponential backoff retries (1s, 2s, 4s, …). After 5 failures, they stay in the DLQ for manual inspection.
Wake‑up queues — Offline clients get a pending notification queue that is replayed as soon as they reconnect.
Local caching — SDKs store notifications locally (SQLite on mobile, IndexedDB on web) so they survive disconnections.

This ensures at‑least‑once delivery with minimal data loss.

📊 Performance & Scale

We built Namma Push to handle millions of devices. Here are the numbers from our benchmarks:

Metric	Single Node	Cluster (3 gateways + 6 Redis shards)
Concurrent connections	10,000	500,000+
Throughput (TPS)	1,000	50,000+
P99 latency (online client)	<50ms	<50ms
P99 latency (wake‑up via FCM/APNs)	–	<2s
Availability	–	99.99% (multi‑AZ)

Horizontal scaling is trivial — just add more gateways and Redis shards.

🛠️ Observability Out of the Box

We believe in “you build it, you run it”. Namma Push includes a full observability stack:

Prometheus metrics: active connections, delivery latency, queue sizes, error rates, per‑tenant usage.
Grafana dashboards pre‑built for operations and business metrics.
OpenTelemetry + Jaeger for distributed tracing.
Loki for log aggregation.

All components are open source and run alongside your Namma Push deployment.

🔒 Security & Compliance

Because you host it yourself, you control the security:

TLS 1.3 for all external endpoints.
mTLS for internal service communication.
API keys (JWT) for backend authentication.
RBAC for admin UI (viewer, operator, admin).
Audit logs of all admin actions.
Designed to support GDPR (right to erasure), HIPAA (audit trails), and PCI‑DSS (segmentation).

No third party sees your notification metadata unless you explicitly choose to use FCM/APNs as optional fallbacks.

💸 Cost Comparison

Let’s run some numbers. Suppose you send 100 million notifications per month:

FCM (or similar managed service): $5,000–10,000 per month (or “free” but with quotas and limited analytics).
Namma Push self‑hosted on AWS (3 gateways + 6 Redis shards): ~$4,800 per month including all infrastructure.
Savings: $200–5,200 per month, or $2,400–62,400 per year.

And that’s without counting the value of having your own data, lower latency, and no lock‑in.

🚀 Roadmap & Getting Involved

Namma Push is currently in active development, following a 16‑week roadmap to a stable v1.0 release:

Phase 1 (Weeks 1‑5): Core engine — gRPC, Redis streams, basic delivery.
Phase 2 (Weeks 6‑10): Production features — FCM/APNs fallback, DLQ, Redis cluster, QUIC.
Phase 3 (Weeks 11‑14): Observability — Grafana, Jaeger, load testing, security hardening.
Phase 4 (Weeks 15‑16): UI & documentation — Vue.js admin UI, deployment guides, SDK examples.

We’re building this in the open. The code is on GitHub (coming soon), and we welcome contributions. Whether you’re a Rust developer, a mobile engineer, or just curious, there’s a place for you.

🎯 Why Namma Push Matters

Push notifications are the lifeblood of modern apps, yet we’ve been dependent on a handful of proprietary services. Namma Push changes that. It gives you:

Full control — your infrastructure, your data.
Superior performance — sub‑50ms latency, no third‑party hops.
Lower costs — self‑hosted, pay only for what you use.
Privacy by design — no one sees your users’ data.
Open source — inspect, modify, and contribute.

We believe every organisation deserves a notification platform that respects its sovereignty. Namma Push is our answer.

📚 Try It Yourself

You can run Namma Push today (pre‑release) with Docker:

docker run -d -p 50051:50051 -p 9090:9090 nammayatri/namma-push:latest

Then visit http://localhost:9090 to create your first tenant and get an API key. Integrate our SDKs (coming soon) and start sending notifications.

Together, we can make Namma Push the go‑to choice for developers who value control, privacy, and performance.

Namma Push — open source, self‑hosted, and ready for the world. 🚀

Aadhaar: India's Digital Identity Revolution — A Technical Deep Dive into the World's Most Sophisticated Identity Platform

Black Lover — Sun, 29 Mar 2026 07:35:33 +0000

Executive Overview

The official Aadhaar mobile application represents a monumental achievement in digital identity engineering. Built to serve over 1.3 billion residents, it stands as the world's largest and most technically sophisticated identity platform. This document provides a comprehensive technical overview of the system architecture, security protocols, and engineering innovations that power India's digital identity infrastructure.

The Vision: Digital Identity for Every Indian

The Aadhaar ecosystem was conceived with a singular vision: provide every Indian resident with a unique, verifiable digital identity that serves as the foundation for accessing government services, financial inclusion, and digital empowerment. The mobile application serves as the primary interface between citizens and this vast infrastructure.

System Architecture Overview

High-Level Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Mobile Application                        │
│                      (Android/iOS - Kotlin/Swift)                │
└────────────────────────────────┬────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                        API Gateway Layer                          │
│                    (gRPC/Protocol Buffers)                        │
│              - Request Routing  - Load Balancing                  │
│              - Rate Limiting   - DDoS Protection                  │
│              - Request/Response Encryption                        │
└────────────────────────────────┬────────────────────────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         ▼                       ▼                       ▼
┌────────────────┐    ┌────────────────┐    ┌────────────────┐
│  Security      │    │   Credential   │    │     User       │
│  Services      │    │   Services     │    │   Services     │
│                │    │                │    │                │
│ - Auth         │    │ - e-Aadhaar    │    │ - Preferences │
│ - Tokens       │    │ - QR Codes     │    │ - Lock/Unlock │
│ - OTP          │    │ - Credentials  │    │ - History     │
│ - Face Auth    │    │                │    │ - Updates     │
└────────────────┘    └────────────────┘    └────────────────┘
         │                    │                       │
         └────────────────────┼───────────────────────┘
                              ▼
                 ┌────────────────────────┐
                 │    Biometric Engine    │
                 │   (On-device ML/AI)    │
                 │ - Face Detection       │
                 │ - Liveness Detection   │
                 │ - Feature Extraction   │
                 └────────────────────────┘

Engineering Excellence: Key Technical Components

1. Microservices Architecture

The backend is built on a microservices architecture that ensures:

Scalability: Individual services can scale independently based on demand
Resilience: Failure in one service doesn't cascade to others
Maintainability: Teams can develop and deploy services independently
Technology Diversity: Each service can use the optimal technology stack

Key Services:

Gateway Service: Single entry point for all client requests
Security Service: Authentication, authorization, token management
Credential Service: e-Aadhaar and QR code delivery
User Service: Profile management and preferences
History Service: Authentication audit trail
Biometric Service: Face matching and verification

2. Communication Protocol: gRPC

The system uses gRPC (Google Remote Procedure Call) for all service-to-service and client-to-server communication:

Advantages of gRPC:

HTTP/2 based: Multiplexing, header compression, server push
Protocol Buffers: Efficient binary serialization (smaller payloads, faster parsing)
Bi-directional streaming: Real-time communication capabilities
Strong typing: Contract-first API development
Language agnostic: Services can be written in any language

Example Service Definition:

service GatewayService {
    // Unary RPC for standard requests
    rpc Process(Request) returns (Response);

    // Streaming for real-time updates
    rpc Stream(stream Request) returns (stream Response);
}

3. Security Architecture: Defense in Depth

Layer 1: Transport Security

TLS 1.3 for all network communications
Certificate pinning to prevent MITM attacks
Perfect Forward Secrecy (PFS) for all sessions

Layer 2: Message Security

Every message undergoes:

Encryption: AES-256-GCM for payload confidentiality
Integrity: HMAC-SHA256 for tamper detection
Authentication: Digital signatures via DSHeader
Replay Protection: Unique transaction IDs and timestamps

Layer 3: Device Security

Device Registration: Each device gets a unique identity
Hardware Attestation: Google Play Integrity / Apple App Attestation
Secure Enclave: Biometric templates stored in hardware
Certificate Chain: X.509 certificates for device authentication

Layer 4: Biometric Security

Liveness Detection: AI models detect spoofing attempts
On-device Processing: Biometric data never leaves the device
Match Score: Only match scores transmitted, not raw biometrics
Anti-Spoofing: Multiple techniques combined (texture analysis, motion detection, depth sensing)

4. Data Serialization: Protocol Buffers

The system uses Protocol Buffers v3 for all data serialization:

Benefits:

Efficiency: 3–10x smaller than JSON/XML
Speed: 20–100x faster serialization/deserialization
Backward Compatibility: Fields can be added without breaking clients
Code Generation: Type-safe client/server stubs
Cross-platform: Works across all programming languages

Message Structure:

message Request {
    Header header = 1;      // Routing and auth
    Payload payload = 2;     // Encrypted data
}

message Header {
    Action action = 1;       // Operation type
    string deviceId = 2;     // Unique device ID
    string sessionToken = 7; // Auth token
    string txnId = 8;        // Unique transaction ID
}

5. Authentication & Authorization

Token Hierarchy

┌─────────────────┐
│  No Token       │  Anonymous requests
└────────┬────────┘
         ▼
┌─────────────────┐
│  Device Token   │  After device registration
└────────┬────────┘
         ▼
┌─────────────────┐
│  LOA1 Token     │  Basic session (device auth only)
└────────┬────────┘
         ▼
┌─────────────────┐
│  LOA2 Token     │  Verified identity (biometric/OTP)
└────────┬────────┘
         ▼
┌─────────────────┐
│  Refresh Token  │  Long-lived token for renewal
└─────────────────┘

LOA (Level of Assurance) Definitions:

LOA1: Device-authenticated session (view only)
LOA2: Resident-authenticated (can access/download Aadhaar)
LOA3: Multi-factor authentication (sensitive operations)

Authentication Methods

OTP: One-time password via SMS
Face: Biometric face authentication
PIN: Registered mobile number PIN
HOF: Head of Family authentication for minors

6. Biometric Engine: AI/ML at Scale

Face Detection Pipeline

Camera Frame
    ↓
┌─────────────────────┐
│ Face Detection      │  ← FSSD Model (100/25)
│ - Locates faces     │    - Accuracy vs Speed tradeoff
│ - Bounding boxes    │    - Anchor boxes for scales
└─────────────────────┘
    ↓
┌─────────────────────┐
│ Face Alignment      │
│ - Normalize pose    │
│ - Scale to standard │
└─────────────────────┘
    ↓
┌─────────────────────┐
│ Liveness Check      │  ← Liveness Model v002
│ - Spoof detection   │    - Texture analysis
│ - Anti-photo attack │    - Motion detection
│ - Anti-replay       │    - Depth estimation
└─────────────────────┘
    ↓
┌─────────────────────┐
│ Feature Extraction  │
│ - 512-byte template │
│ - Matcher-ready     │
└─────────────────────┘
    ↓
┌─────────────────────┐
│ Matching/Verification│
│ - 1:1 verification  │
│ - Score calculation │
└─────────────────────┘

Machine Learning Models

Model	Type	Purpose	Size	Speed
FSSD-100	Face Detection	High accuracy detection	4.2 MB	~50ms
FSSD-25	Face Detection	Fast detection	1.8 MB	~15ms
Liveness v2	Anti-spoofing	Liveness detection	2.5 MB	~30ms
Feature Extractor	Embedding	Face template generation	3.1 MB	~40ms

Model Optimization:

8-bit quantization: 75% size reduction, minimal accuracy loss
TensorFlow Lite: Optimized for mobile CPUs/GPUs
Adaptive loading: Choose model based on conditions (battery, lighting)

7. Audit & Observability

Comprehensive Audit Trail

Every authentication attempt generates a record with 178 data points:

Authentication Metadata:

Timestamp, transaction ID, device ID
Authentication type and mode
Success/failure status
Error codes and classifications

Biometric Data:

Match scores for face/iris/fingerprint
Algorithm versions and vendors
Fusion scores and thresholds
Gallery types and configurations

Device Information:

Device provider ID and software version
Model ID and certificate expiry
Location data (lat/long/VTC codes)
Network and connection details

Demographic Data:

Resident age, gender, DOB
Address components used
Pincode and location codes
Enrolment reference ID

Analytics Pipeline

Mixpanel: User behavior analytics
Firebase: Crash reporting, performance monitoring
Custom Metrics: System health and performance

8. Privacy by Design

Core Privacy Features

1. Masked Aadhaar

Option to hide all but last 4 digits
Separate QR codes for public/private use

2. Biometric Locking

Permanent lock/unlock
Temporary unlock with automatic expiry
Granular control over authentication methods

3. Consent Management

Explicit consent for each data share
Revocable consents
Audit trail of all consent activities

4. Notification Controls

Per-auth-type notification preferences
Real-time alerts for authentication attempts
Email/SMS notification options

9. Scalability Engineering

Handling 1.3+ Billion Users

Database Architecture:

Sharding: Horizontal partitioning by UID range
Replication: Multi-region read replicas
Caching: Redis/Memcached for frequent queries
Time-series: Specialized storage for audit data

Load Balancing:

Geographic: Route users to nearest data center
Application: Distribute across service instances
Database: Balance read/write loads

Rate Limiting:

Per device, per user, per IP
Graduated limits based on authentication level
Burst handling with token bucket algorithm

Disaster Recovery:

Multi-region active-active deployment
Real-time data replication
Automated failover with < 5 minute RTO

10. Multilingual Support: Bhashini Integration

The app integrates with Bhashini, India's National Language Translation Mission:

Real-time translation of UI elements
Voice support for illiterate users
22 official languages supported
On-device models for offline use

Translation Pipeline:

User selects language
    ↓
UI strings extracted
    ↓
Bhashini API call (or local cache)
    ↓
Translated UI rendered
    ↓
Voice output (optional)

11. Payment Integration

The app includes Razorpay for processing service fees:

Payment Flows:

Address update requests
Document update fees
Premium services
e-Aadhaar re-downloads

Security:

PCI-DSS compliant
Tokenization of payment data
3D Secure for card payments
UPI integration for Indian users

12. Performance Optimization

Mobile App Optimizations

Startup Time:

Lazy loading of non-critical modules
Optimized splash screen
Background initialization

Network Efficiency:

Protocol Buffers (smaller payloads)
Request batching
Response caching
Offline capability for static content

Memory Management:

Image compression and caching
Model quantization
Garbage collection optimization
Memory-mapped files for large data

Battery Optimization:

Adaptive model selection
Network request batching
Background sync scheduling
Sensor fusion for efficiency

Engineering Achievements

Scale

1.3B+ registered users
100M+ daily authentications
10M+ concurrent sessions
5TB+ daily audit data

Performance

< 200ms API response time (p95)
99.99% uptime SLA
< 1% authentication error rate
< 5 seconds e-Aadhaar download

Security

Zero major security breaches
PCI-DSS compliant
ISO 27001 certified
STQC audited

Coverage

100% of Indian districts
22 official languages
99% of adults enrolled
10M+ daily active users

Technical Specifications Summary

Component	Technology Stack
Backend Language	Go, Java, Python
Mobile Frontend	Kotlin (Android), Swift (iOS)
API Protocol	gRPC over HTTP/2
Serialization	Protocol Buffers v3
Database	PostgreSQL, MongoDB, Cassandra
Cache	Redis, Memcached
Message Queue	Apache Kafka, RabbitMQ
Search	Elasticsearch
Monitoring	Prometheus, Grafana, ELK Stack
CI/CD	Jenkins, GitLab CI
Container	Docker, Kubernetes
Cloud	MeghRaj (Government Cloud), AWS, Azure

The Road Ahead

Upcoming Innovations

Offline Authentication

Bluetooth-based peer-to-peer verification
QR code-based offline validation

Advanced Biometrics

Voice authentication
Gait recognition
Multi-modal fusion

Blockchain Integration

Immutable audit trail
Decentralized identity verification

AI/ML Enhancements

Predictive fraud detection
Behavioral biometrics
Continuous authentication

Edge Computing

Local authentication at service points
Reduced dependency on central servers

Conclusion: A Model for Digital Identity

The Aadhaar platform represents a paradigm shift in digital identity management. It demonstrates that it's possible to build a system that is simultaneously:

Scalable: Serving over a billion users
Secure: Multiple layers of protection
Private: User control over data
Usable: Simple interface, multiple languages
Reliable: 99.99% uptime
Cost-effective: Fraction of traditional identity systems

For system engineers and architects, Aadhaar offers invaluable lessons in building large-scale, secure, privacy-preserving systems. It's not just an app — it's a blueprint for digital identity infrastructure in the 21st century.

I Built a SaaS on Top of an Open-Source Project and My Database Punished Me for It

Black Lover — Sun, 29 Mar 2026 07:32:16 +0000

I didn't design my database from scratch.

That's the honest truth. When I decided to build my WhatsApp Business API SaaS — targeting digital marketing agencies in India — I didn't start with a blank Postgres schema and a whiteboard. I started with a fork. Specifically, I forked Whatomate, an open-source Go-based WhatsApp platform, and told myself: "The hard part is done. I just need to add billing and org management on top."

That was about six months ago. Since then, my database has quietly humiliated me in ways I didn't expect. Not dramatically — not with crashes or data loss or anything catastrophic. Just slowly, consistently, in the way that technical debt humiliates you: by making every new feature take twice as long as it should, and by making you feel stupid for not seeing it coming.

This is that story.

Who This Is For

If you've ever:

Forked an open-source project and tried to commercialize it
Inherited a codebase and tried to add multi-tenancy to it
Built a SaaS and skipped the "proper" database design phase because you were in a hurry

...then some version of what happened to me has probably happened to you too, or will. This post is the thing I wish I'd read before I started.

The Starting Point: What I Forked and Why

Whatomate is a self-hosted WhatsApp Business API platform built in Go. It handles the gnarly parts of WhatsApp integration: message queuing, webhook management, media handling, contact synchronization with the WhatsApp API, campaign scheduling. It's genuinely good software — clean code, well-structured, production-tested.

The go-to-market thesis was simple: Indian digital marketing agencies spend thousands of rupees per month on SaaS WhatsApp tools like Interakt, Wati, and AiSensy. Most of those tools have the same core feature set. The differentiation is mostly UI and pricing. If I could fork a solid open-source foundation, add a proper multi-tenant billing layer, and undercut on price while matching features — there was a business there.

What I didn't account for was that the "add a billing layer" step wasn't additive. It was transformative. The business model I was building required a fundamentally different data model than the one I forked.

I would not discover this until I was already six weeks in.

The Implicit Contract I Didn't Notice

The original codebase was built with a very specific assumption baked deep into every table, every query, every foreign key relationship:

One installation = one organization.

It wasn't documented anywhere. It didn't need to be. The developer who built it was building a self-hosted tool. One team, one server, one set of WhatsApp accounts. The schema reflected that beautifully — clean, fast, well-indexed for that exact use case.

When I forked it, I inherited that assumption without realizing it.

The first time I noticed was when I started designing the org management module for SaaS mode. I went to look at where the users table connected to everything else. And I realized: it connected to everything. Users owned campaigns. Users owned contacts. Users owned message templates. Users owned WhatsApp account credentials. There was no middle layer — no organization, no tenant_id, no concept that multiple companies might share the same installation.

-- What the original schema looked like (simplified)
users
  id, email, password_hash, created_at

campaigns
  id, user_id, name, status, scheduled_at
  -- user_id FK → users.id

contacts
  id, user_id, phone, name, tags
  -- user_id FK → users.id

whatsapp_accounts
  id, user_id, phone_number, access_token, webhook_url
  -- user_id FK → users.id

message_templates
  id, user_id, name, body, language_code
  -- user_id FK → users.id

The database was built for a world where that question never needed to be asked. I was now asking it at production scale. Or at least, staging scale — which felt like production at 11pm with chai going cold.

What Happened When I Started Adding `org_id` Everywhere

My first instinct was simple: add an org_id column to every table. Slap a foreign key on it. Done.

I started writing the migration. About 20 tables in, I stopped.

The problem wasn't the columns. It was the queries.

The existing Go code had hundreds of database calls — beautifully written, fast, using GORM v2 — and none of them knew org_id existed. Every query fetched data scoped to a user. Not a tenant. A user.

// What the existing query layer looked like (simplified)
func GetCampaigns(userID uint) ([]Campaign, error) {
    var campaigns []Campaign
    result := db.Where("user_id = ?", userID).Find(&campaigns)
    return campaigns, result.Error
}

// What I needed for multi-tenancy
func GetCampaigns(orgID uint, userID uint) ([]Campaign, error) {
    var campaigns []Campaign
    result := db.Where("org_id = ? AND user_id = ?", orgID, userID).Find(&campaigns)
    return campaigns, result.Error
}

If I added org_id to the tables but didn't touch the queries, I'd have a SaaS product where Agency A could theoretically access Agency B's contacts. Not because of a bug in my code. Because of a contract mismatch between the schema I was building and the queries that were already there.

The database wasn't wrong. The original code wasn't wrong. I was trying to make them do something they were never designed for.

The Full Scope of the Problem

Here's what the migration actually involved, once I mapped it out:

-- Step 1: Create the organizations table (didn't exist at all)
CREATE TABLE organizations (
    id          BIGSERIAL PRIMARY KEY,
    name        VARCHAR(255) NOT NULL,
    slug        VARCHAR(100) NOT NULL UNIQUE,
    plan_id     BIGINT REFERENCES plans(id),
    created_at  TIMESTAMP DEFAULT NOW(),
    updated_at  TIMESTAMP DEFAULT NOW()
);

-- Step 2: Add org_id to every table that needed tenant isolation
ALTER TABLE contacts          ADD COLUMN org_id BIGINT REFERENCES organizations(id);
ALTER TABLE campaigns         ADD COLUMN org_id BIGINT REFERENCES organizations(id);
ALTER TABLE message_templates ADD COLUMN org_id BIGINT REFERENCES organizations(id);
ALTER TABLE whatsapp_accounts ADD COLUMN org_id BIGINT REFERENCES organizations(id);
ALTER TABLE webhooks          ADD COLUMN org_id BIGINT REFERENCES organizations(id);
ALTER TABLE media_uploads     ADD COLUMN org_id BIGINT REFERENCES organizations(id);
ALTER TABLE api_keys          ADD COLUMN org_id BIGINT REFERENCES organizations(id);
ALTER TABLE audit_logs        ADD COLUMN org_id BIGINT REFERENCES organizations(id);
-- ... 14 more tables

-- Step 3: Create a default org for existing self-hosted data
INSERT INTO organizations (name, slug) VALUES ('Default', 'default');

-- Step 4: Backfill org_id on every existing row
UPDATE contacts          SET org_id = 1 WHERE org_id IS NULL;
UPDATE campaigns         SET org_id = 1 WHERE org_id IS NULL;
UPDATE message_templates SET org_id = 1 WHERE org_id IS NULL;
-- ... repeat for every table

-- Step 5: Add NOT NULL constraints only after backfill is complete
ALTER TABLE contacts          ALTER COLUMN org_id SET NOT NULL;
ALTER TABLE campaigns         ALTER COLUMN org_id SET NOT NULL;
ALTER TABLE message_templates ALTER COLUMN org_id SET NOT NULL;
-- ... repeat for every table

-- Step 6: Add composite indexes for tenant-scoped queries
CREATE INDEX idx_contacts_org_id          ON contacts(org_id);
CREATE INDEX idx_campaigns_org_id_status  ON campaigns(org_id, status);
CREATE INDEX idx_templates_org_id         ON message_templates(org_id);
-- ... repeat for performance-critical tables

That's not a migration. That's a schema renegotiation — and it touches every service, every query, every test, every API handler. Two weeks I hadn't budgeted for. Two weeks that had a cascading effect on every deadline after them.

The Lesson That Slapped Me: Schema Is a Contract, Not Just Storage

I used to think of the database as a dumb storage layer. You put data in, you get data out, you add indexes when things get slow. The real logic lives in the application.

Building on top of someone else's schema broke that belief permanently.

The schema is the application in a lot of ways. The table names, the relationships, the nullable columns, the absence of certain foreign keys — all of it encodes decisions that were made about how the software would be used. When you change the use case without renegotiating the schema, you're making promises the database can't keep.

In my case, the original schema promised: "a user is the atomic unit of ownership."

My SaaS needed it to promise: "an organization is the atomic unit of ownership, and users belong to organizations."

Those are fundamentally different contracts. The entity-relationship model shifts like this:

Original schema:
User ──owns──> Campaigns
User ──owns──> Contacts
User ──owns──> Templates
User ──owns──> WhatsApp Accounts

SaaS schema:
Organization ──has──> Users (with roles: owner, admin, member)
Organization ──owns──> Campaigns
Organization ──owns──> Contacts
Organization ──owns──> Templates
Organization ──owns──> WhatsApp Accounts
User ──belongs to──> Organization
User ──can access──> Org's resources (filtered by role)

I couldn't patch my way from one to the other. I had to sit down and think through which tables needed org_id, which could stay user-scoped, what the query layer needed to change, what new indexes I needed, and whether existing data would survive the migration cleanly — and then execute all of it before I could ship a single paying customer.

The Tables That Don't Need `org_id` (And Why That Matters)

Not everything needed org isolation, and figuring out which tables didn't was as important as figuring out which ones did. There are two categories of data that are genuinely cross-org:

Global/shared data — things that exist independently of any organization:

-- These stay without org_id
plans          -- Pricing tiers (Basic, Pro, Agency) — shared across all orgs
countries      -- Country codes for phone number validation
languages      -- Supported template languages (WhatsApp API constraint)
system_logs    -- Infrastructure-level logs, not org-specific

Per-user data that isn't org-owned — things that belong to the person, not the company:

-- These stay user-scoped
user_sessions       -- Auth sessions belong to the person, not the org
user_preferences    -- UI settings, notification prefs
password_resets     -- Security flows are user-level

Getting this wrong in either direction causes problems. If you put org_id on plans, you'd have to duplicate plan records per org. If you leave org_id off contacts, you've broken tenant isolation. Mapping out every table against this taxonomy before touching a single migration command saved me from at least three mistakes I can think of.

The Role System Nobody Warned Me About

Here's something the "just add org_id" advice leaves out completely: once you have organizations, you need roles. Not immediately — but the moment your first real customer asks "can I give my team member access without giving them owner permissions," you need roles. And if your schema isn't ready for it, you're doing another two-week refactor.

I decided to build the role system upfront this time, rather than discovering I needed it mid-flight.

-- Organization membership with roles
CREATE TABLE org_members (
    id          BIGSERIAL PRIMARY KEY,
    org_id      BIGINT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    user_id     BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    role        VARCHAR(50) NOT NULL DEFAULT 'member',
    -- roles: 'owner' | 'admin' | 'member' | 'viewer'
    invited_by  BIGINT REFERENCES users(id),
    joined_at   TIMESTAMP,
    created_at  TIMESTAMP DEFAULT NOW(),
    UNIQUE(org_id, user_id)  -- A user can only have one role per org
);

CREATE INDEX idx_org_members_org_id  ON org_members(org_id);
CREATE INDEX idx_org_members_user_id ON org_members(user_id);

The role definitions themselves live in the application layer:

// internal/auth/roles.go
type Role string

const (
    RoleOwner  Role = "owner"   // Full access, billing, delete org
    RoleAdmin  Role = "admin"   // Full access, no billing/delete
    RoleMember Role = "member"  // Create/edit campaigns and contacts
    RoleViewer Role = "viewer"  // Read-only access
)

type Permission string

const (
    PermCreateCampaign   Permission = "campaign:create"
    PermDeleteCampaign   Permission = "campaign:delete"
    PermManageMembers    Permission = "members:manage"
    PermViewBilling      Permission = "billing:view"
    PermManageBilling    Permission = "billing:manage"
    PermDeleteOrg        Permission = "org:delete"
    PermManageAPIKeys    Permission = "api_keys:manage"
)

var RolePermissions = map[Role][]Permission{
    RoleOwner: {
        PermCreateCampaign, PermDeleteCampaign,
        PermManageMembers, PermViewBilling, PermManageBilling,
        PermDeleteOrg, PermManageAPIKeys,
    },
    RoleAdmin: {
        PermCreateCampaign, PermDeleteCampaign,
        PermManageMembers, PermViewBilling,
        PermManageAPIKeys,
    },
    RoleMember: {
        PermCreateCampaign,
    },
    RoleViewer: {},
}

func (r Role) Can(p Permission) bool {
    perms, ok := RolePermissions[r]
    if !ok {
        return false
    }
    for _, perm := range perms {
        if perm == p {
            return true
        }
    }
    return false
}

The middleware that enforces this runs on every protected route:

// middleware/auth.go
func RequirePermission(perm auth.Permission) gin.HandlerFunc {
    return func(c *gin.Context) {
        member := c.MustGet("org_member").(OrgMember)
        if !auth.Role(member.Role).Can(perm) {
            c.AbortWithStatusJSON(403, gin.H{
                "error": "insufficient_permissions",
                "required": perm,
            })
            return
        }
        c.Next()
    }
}

// Applied at route registration
campaigns := r.Group("/campaigns")
campaigns.Use(middleware.RequirePermission(auth.PermCreateCampaign))
campaigns.POST("/", handlers.CreateCampaign)

None of this existed in the original codebase. The original had a simple admin boolean on the users table. That's fine for a self-hosted tool where there's one team. It's not enough for a SaaS where an agency owner needs to give their intern read-only access to campaign analytics.

The Feature Flag Hack That Saved Me (And The Technical Debt It Created)

Here's where I made a pragmatic call I'm still not sure was right.

I didn't want to break the original self-hosted use case. I'm contributing back to the open-source project — I care about that. So I introduced a mode config flag:

// config/config.go
type AppConfig struct {
    Mode string `env:"APP_MODE" default:"self-hosted"` // "saas" | "self-hosted"
    // ...
}

func IsSaaS() bool {
    return Get().Mode == "saas"
}

// Used throughout the service layer
func (s *CampaignService) GetCampaigns(ctx context.Context, userID uint) ([]Campaign, error) {
    if config.IsSaaS() {
        orgID := ctx.Value("org_id").(uint)
        return s.repo.GetByOrg(orgID, userID)
    }
    return s.repo.GetByUser(userID)
}

In self-hosted mode: the app works exactly as before. No org concepts, no tenant isolation, no role checks. In saas mode: the org layer kicks in at every level — queries, writes, middleware, billing checks.

This works. But it means two mental models running in the same codebase. Every time I add a new feature, I have to ask:

Is this org_id-scoped or user_id-scoped?
Does the self-hosted path need this feature at all?
If it only makes sense in SaaS mode, am I guarding it properly?
Am I going to introduce a regression in the mode I'm not actively testing right now?

The database now has columns that are nullable in self-hosted mode and non-nullable in SaaS mode. Some indexes only get used in one mode. Some foreign keys only matter in one mode. The schema is a bifurcated thing — half of it answering one set of questions, half answering another.

Was there a better way? Probably. Did I have the time for it? No. Am I paying for it in confusion tax every week? Yes.

The cleaner alternative — which I couldn't afford to build at the time — would have been separate database views per mode, with the shared physical tables underneath:

-- Views that self-hosted queries hit (no org concept at all)
CREATE VIEW v_campaigns_selfhosted AS
    SELECT id, user_id, name, status, scheduled_at, created_at
    FROM campaigns
    WHERE org_id IS NULL;

-- Views that SaaS queries hit (org-scoped, always filtered)
CREATE VIEW v_campaigns_saas AS
    SELECT id, org_id, user_id, name, status, scheduled_at, created_at
    FROM campaigns
    WHERE org_id IS NOT NULL;

The repository layer picks a view based on mode at initialization time. The physical schema stays unified, but each access pattern sees a coherent, mode-appropriate view of the data. No if config.IsSaaS() scattered through 200 service functions.

Maybe I'll refactor toward this eventually. Maybe I won't. Right now the mode flag is load-bearing and touching it feels dangerous.

The Specific Night I Understood What "Not Designed For This" Means

There was one night — I remember it clearly — where I was trying to add campaign-level quota limits for the plan system. Agencies on a Basic plan get 5 active campaigns per month. Pro plan gets 25. Agency plan gets unlimited.

Simple feature. Should take an afternoon.

I opened the campaigns table. The user_id foreign key was there. The org_id I'd added was there. But the plan limits needed to live at the org level, and campaigns were being created with only a user_id check in the service layer.

To enforce the limit correctly, the write path needed to:

Check the org's current plan
Count active campaigns across all users in that org — not just the user making the request
Reject the write if over quota
Do all of this atomically, without a race condition where two users hit the limit simultaneously and both sneak through

func (s *CampaignService) CreateCampaign(
    ctx context.Context,
    input CreateCampaignInput,
) (*Campaign, error) {
    orgID := ctx.Value("org_id").(uint)

    return s.repo.WithTransaction(func(tx *gorm.DB) (*Campaign, error) {
        // Lock the org row — prevents concurrent writes from bypassing the limit
        // Two users in the same org hitting this simultaneously will serialize here
        var org Organization
        if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
            First(&org, orgID).Error; err != nil {
            return nil, fmt.Errorf("fetching org: %w", err)
        }

        // Count active campaigns across the ENTIRE org, not just this user
        var activeCampaignCount int64
        if err := tx.Model(&Campaign{}).
            Where("org_id = ? AND status NOT IN ('deleted', 'completed')", orgID).
            Count(&activeCampaignCount).Error; err != nil {
            return nil, fmt.Errorf("counting campaigns: %w", err)
        }

        // Check the org's plan limit
        limit := s.plans.CampaignLimit(org.PlanID)
        if limit != PlanUnlimited && activeCampaignCount >= int64(limit) {
            return nil, &QuotaExceededError{
                Resource: "campaigns",
                Current:  activeCampaignCount,
                Limit:    int64(limit),
                PlanName: org.PlanName,
            }
        }

        // All checks passed — create the campaign
        campaign := &Campaign{
            OrgID:  orgID,
            UserID: input.UserID,
            Name:   input.Name,
            Status: CampaignStatusDraft,
        }
        if err := tx.Create(campaign).Error; err != nil {
            return nil, fmt.Errorf("creating campaign: %w", err)
        }

        return campaign, nil
    })
}

The database had none of the scaffolding for this. No aggregate table. No counter cache. No org-level write lock pattern. No plans table with limit definitions. No structured quota error types. I had to build all of it from scratch — and then I had to think through every other resource (contacts, WhatsApp accounts, API keys, message templates) and decide whether they also needed quota enforcement, and if so, whether the same pattern applied.

The original developers never needed any of that. Their users weren't organizations. Their quota was "as many as you want, it's your server."

I sat with that for a while. Not frustrated — just genuinely humbled. The database was doing exactly what it was built to do. I was the one who showed up with different requirements and expected it to just adapt.

The Billing Integration Problem I Didn't See Coming

Once you have organizations and plans, you need billing. Once you have billing, you need to keep your database and your billing provider (I'm using Razorpay, because India) synchronized. And that synchronization is harder than it sounds.

The problem: a customer can cancel their plan in Razorpay's dashboard. Razorpay sends a webhook. Your database still thinks they're on the Pro plan. For a window of time — however long it takes your webhook handler to process — you're serving a paying feature to a customer who just cancelled.

The naive approach is to update the plan_id on the org when the webhook fires. But webhooks fail. Webhooks arrive out of order. Webhooks get retried. The billing state in your database and the billing state in Razorpay can drift, and reconciling that drift requires infrastructure the original codebase had no concept of.

What I ended up building:

-- Subscription state lives in its own table, separate from the org
CREATE TABLE subscriptions (
    id                  BIGSERIAL PRIMARY KEY,
    org_id              BIGINT NOT NULL REFERENCES organizations(id),
    razorpay_sub_id     VARCHAR(255) NOT NULL UNIQUE,
    plan_id             BIGINT NOT NULL REFERENCES plans(id),
    status              VARCHAR(50) NOT NULL,
    -- 'active' | 'past_due' | 'cancelled' | 'paused'
    current_period_start TIMESTAMP NOT NULL,
    current_period_end   TIMESTAMP NOT NULL,
    cancel_at_period_end BOOLEAN NOT NULL DEFAULT FALSE,
    cancelled_at        TIMESTAMP,
    created_at          TIMESTAMP DEFAULT NOW(),
    updated_at          TIMESTAMP DEFAULT NOW()
);

-- Webhook events get stored before processing (idempotency)
CREATE TABLE billing_events (
    id              BIGSERIAL PRIMARY KEY,
    event_id        VARCHAR(255) NOT NULL UNIQUE, -- Razorpay event ID
    event_type      VARCHAR(100) NOT NULL,
    payload         JSONB NOT NULL,
    processed_at    TIMESTAMP,
    processing_error TEXT,
    created_at      TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_billing_events_event_id   ON billing_events(event_id);
CREATE INDEX idx_billing_events_processed  ON billing_events(processed_at)
    WHERE processed_at IS NULL;  -- Partial index — only unprocessed events

The webhook handler stores the event first, then processes it:

func (h *BillingWebhookHandler) HandleEvent(c *gin.Context) {
    // 1. Verify Razorpay signature
    if !h.billing.VerifyWebhookSignature(c.Request) {
        c.Status(401)
        return
    }

    var payload razorpay.WebhookPayload
    if err := c.ShouldBindJSON(&payload); err != nil {
        c.Status(400)
        return
    }

    // 2. Store event for idempotent processing
    // If this event_id already exists, INSERT OR IGNORE — Razorpay retries
    event := &BillingEvent{
        EventID:   payload.EventID,
        EventType: payload.EventType,
        Payload:   payload,
    }
    if err := h.repo.StoreEventIdempotent(event); err != nil {
        // Already processed — return 200 so Razorpay stops retrying
        c.Status(200)
        return
    }

    // 3. Acknowledge immediately — process async
    c.Status(200)

    // 4. Process in background goroutine
    go h.processor.ProcessEvent(event)
}

None of this architecture existed in the original codebase because the original had no concept of subscriptions, billing, or recurring payments. It's not a gap in the original — it's a feature it never needed. But it's infrastructure I had to build from scratch, in a codebase that had no natural place for it, while also keeping the self-hosted path working without any billing dependency.

What I'm Doing Differently Now

1. Read the schema before you read the code

When you fork something or onboard onto a codebase, the schema tells you the mental model of whoever built it. Table names, nullable decisions, what's indexed and what isn't — it's all signal. Read it like a design document, not like a starting point.

Specifically, look for:

What is the ownership root — what table does everything else hang off of?
What columns are absent that you'd expect for your use case? (For me: org_id, tenant_id, plan_id, role)
What's nullable that shouldn't be in your use case?
What indexes exist — they tell you what query patterns were optimized for, which reveals what the original author expected to be hot paths

2. Before writing a migration, write your data access policy

org_id is not a solution. It's the beginning of a decision. The real work is auditing every query and every write path against a policy you've written in plain English first:

## Data Access Policy

- A user can only see data belonging to their organization
- A user from Org A can never see, modify, or be aware of data from Org B
- Admin role: can manage org members and all org resources
- Member role: can manage campaigns and contacts but not members or billing
- Viewer role: read-only access to campaigns and contacts
- Shared data (plans, languages, countries): visible to all authenticated users regardless of org
- Billing events: visible only to Owner and Admin roles

That document becomes the spec your migration, your repository layer, your middleware, and your tests all have to implement. Without it, you're making ad-hoc decisions at every query and hoping they add up to a consistent policy.

3. Build the roles table before you think you need it

You will need it. Build it upfront. It's much cheaper to add role checking to 20 routes at the start than to retrofit it into 80 routes six months later when your first enterprise customer asks for fine-grained access control.

4. Treat billing state as its own bounded context

Don't put subscription status directly on the org table. Subscription state changes for reasons outside your control (payment failures, manual overrides in the billing dashboard, proration events). Keeping it in a dedicated table with its own event log gives you idempotency, auditability, and the ability to replay webhook history when something goes wrong — and something always goes wrong.

5. Feature flags that split schema assumptions create long-term confusion

My mode config works, but it means I can never fully reason about the schema in a single mental model. The confusion tax compounds. Every new feature, every new query, every new test — I have to think about both modes. If I were doing this again, I'd push harder for a views-based abstraction from the start.

6. Write down things that feel obvious

The "one org = one installation" assumption was never written down because it was obvious to everyone building the original tool. The moment you fork a project for a different use case, those obvious assumptions become invisible traps.

I now keep a SCHEMA_ASSUMPTIONS.md in the repo:

# Schema Assumptions

## Tenancy model
- SaaS mode: `org_id` is the isolation boundary. No query should ever cross org boundaries.
- Self-hosted mode: `org_id` is always NULL. User is the isolation boundary.
- Mixed mode: Not supported. An installation is either SaaS or self-hosted.

## Roles
- Roles are enforced at the middleware layer, not in the database.
- Database does not enforce role-based read restrictions — only application does.
- Revisit with RLS (Row-Level Security) if compliance requirements change.

## Plan limits
- Limits are enforced at write time with SELECT FOR UPDATE on the org row.
- Counter caches are NOT used — count queries are cheap enough at current scale.
- Revisit counter caches if orgs exceed ~50,000 contacts or ~10,000 campaigns.

## Billing state
- Subscription status in `subscriptions` table is the source of truth.
- `organizations.plan_id` is a denormalized cache — refreshed on subscription events.
- Never read plan limits directly from `organizations.plan_id` without checking `subscriptions.status`.

## User ownership
- A user belongs to exactly one org in SaaS mode, or no org in self-hosted mode.
- Users cannot be transferred between orgs without a manual data migration.
- Deleting an org cascades to org_members but NOT to user accounts (users persist).

Future me thanks present me every time I open that file.

Things I Still Haven't Solved

I want to be honest: I'm not writing this from a position of having figured everything out. There are open problems I'm sitting with:

Row-Level Security in Postgres. The "right" way to enforce tenant isolation at the database level is Postgres RLS — policies on the tables themselves that filter by org_id automatically, so even a query that accidentally omits the org_id filter can't return another org's data. I haven't implemented this yet. Application-level enforcement is working and I haven't had a bug, but it's one misplaced query away from a data leak. It's on the list.

The self-hosted ↔ SaaS migration path. What happens when a self-hosted user wants to move to the managed SaaS? Currently: manual. Someone (me) exports their data, transforms it, imports it to the SaaS instance. That's not a business. I need a proper migration tool and I don't have one yet.

Soft deletes at the org level. When an org cancels and their data enters a grace period before deletion, how do I handle queries that should or shouldn't include "cancelled org" data? Right now it's a status flag and a bunch of WHERE status != 'cancelled' clauses. It's messy.

The Broader Thing

I'm a solo developer, about six months from finishing my degree, building this on top of an open-source fork in my spare hours. I'm not a staff engineer at a FAANG company writing a blog post from authority. I'm someone who ran into a wall and had to figure out what it was made of — repeatedly, across six months of slow, sometimes painful learning.

Nobody writes about the friction of forking someone else's assumptions. Everyone writes about the clean green-field system design. The reality of building something real, on a deadline, on top of someone else's excellent work, is messier and more interesting than that. It's also more instructive, because the constraints force you to understand things you'd gloss over if you were starting from scratch and could just "design it properly."

The database wasn't designed for what I wanted to build. But the tools to renegotiate that contract were always there — migrations, transactions, views, careful query auditing, explicit policy documents. I just had to stop assuming I could skip that conversation.

I'm building Whatomate — an open-source WhatsApp Business API platform — and a SaaS layer on top of it targeting Indian digital marketing agencies. If you're doing something similar, have opinions on multi-tenancy patterns in forked codebases, or have already solved the RLS or soft-delete problems I mentioned, I'd genuinely love to hear from you in the comments.

Follow for more honest write-ups from the trenches of solo SaaS building in India.

WhatsApp's URL Architecture: The Distributed GraphQL Mesh

Black Lover — Sun, 29 Mar 2026 06:39:39 +0000

Source: Decompiled WhatsApp Android version 2.26.3.79. All endpoints, operation names, and URL patterns are as observed in the client binary.

WhatsApp doesn't use a single GraphQL endpoint. It operates a distributed mesh of specialized GraphQL gateways, each serving a distinct domain: payments, generative AI, Oculus/AR, zero-rating, and core messaging. This federated graph architecture enables independent scaling, regional deployment, and security isolation between domains that have no business touching each other's data.

This is a deep-dive into how that mesh is structured, how mutations are routed across it, and what the URL patterns reveal about Meta's broader infrastructure strategy.

The Three-Schema Foundation

Before getting into endpoints, it helps to understand the schema layer they serve:

Schema	Owner	Purpose
`whatsapp`	WhatsApp team	Core messaging, AI features, payments, maps
`whatsapp_mex`	MEX platform team	High-trust: DMA interop, parental controls, passkeys, identity
`facebook`	Meta platform team	Cross-Meta: ads, commerce, digital content

Every GraphQL operation in the client is tagged to one of these schemas. That tag determines which master endpoint receives the request and which slave services it can fan out to. The schema is the routing key.

1. The GraphQL Endpoint Mesh

Complete Endpoint Inventory

// Core GraphQL endpoints (from decompiled client)
X_GRAPH_FACEBOOK_GRAPHQL_URL               // https://graph-www.facebook.com/graphql
X_GRAPH_FACEBOOK_ZERO_RATING_BOOTSTRAP_URL // https://b-graph.facebook.com/graphql
X_GRAPH_OCULUS_GRAPHQL_URL                 // https://graph.oculus.com/graphql
X_GRAPH_PAYMENTS_GRAPHQL_URL               // https://payments-graph.facebook.com/graphql
X_GRAPH_FACEBOOK_GENAI_GRAPHQL_URL         // https://genai-graph.facebook.com/graphql
X_GRAPH_INSTAGRAM_GENAI_GRAPHQL_URL        // https://genai-graph.instagram.com/graphql_www
X_GRAPH_INSTAGRAM_PAYMENTS_GRAPHQL_URL     // https://payments-graph.instagram.com/graphql_www

// WhatsApp-specific endpoints
https://graph.whatsapp.net/v2.2/maps_configs
https://graph.whatsapp.net/wa_qpl_data
https://static.whatsapp.net/wa/static/network_resource
https://static.whatsapp.net/sticker?img=
https://wa.me/stickerpack/%s
https://chat.whatsapp.com/

// Third-party integrations
https://api.giphy.com/v1/stickers/search      // Giphy stickers (pg-13 filtered)
https://scontent.oculuscdn.com/               // Oculus/Meta VR content
https://lookaside.facebook.com/assets/key/    // Meta design system assets

Endpoint Purpose Map

Endpoint	Purpose	Schema
`graph-www.facebook.com`	Core Facebook graph	`facebook`
`b-graph.facebook.com`	Zero-rated bootstrap (emergency access)	Subset of core
`graph.oculus.com`	VR/AR metadata, smart glasses	Oculus schema
`payments-graph.facebook.com`	Payment processing	`facebook` with payment scope
`genai-graph.facebook.com`	Generative AI (Imagine, MEmu)	GenAI schema
`genai-graph.instagram.com`	Instagram AI features	Cross-platform AI
`payments-graph.instagram.com`	Instagram payments	Instagram commerce
`graph.whatsapp.net`	Core WhatsApp messaging	`whatsapp` / `whatsapp_mex`

This is not a monolith. Each endpoint is owned by a different team, scales independently, has separate authentication requirements, and — critically — separate data access boundaries. The GenAI endpoint has no access to your contact list. The payments endpoint has no access to your message history. That isolation is the point.

Why Federated GraphQL Over a Monolith?

The conventional alternative — a single GraphQL endpoint with a unified schema — breaks down at Meta's scale for several reasons:

Independent deployability. If the GenAI service needs a schema change, it doesn't require a coordinated deploy across the entire messaging infrastructure. Each team ships on their own cadence.

Security isolation. A SQL injection or logic flaw in the Giphy proxy can't escalate to contact list access because those services literally don't share a schema or auth context.

Regional compliance. EU data residency requirements can be enforced at the endpoint level — EU users' MEX operations route to Frankfurt infrastructure, never leaving the region. A monolith makes this significantly harder.

Failure containment. If genai-graph.facebook.com goes down during a viral "imagine me" moment, core messaging at graph.whatsapp.net is unaffected. The master-slave fan-out pattern provides graceful degradation.

2. The Master-Slave Mutation Architecture

Operation Hashing: The Core Pattern

Every GraphQL operation in the WhatsApp client is represented by a triple-hash system rather than inline query text:

{
  "AddMEmuProfilePhotos": {
    "operation_name": "AddMEmuProfilePhotos",
    "operation_name_hash": "793005150",
    "operation_text_hash": "6669169098671059140",
    "client_doc_id": "7930051506669169098671059140",
    "schema": "whatsapp"
  }
}

Three hashes, three purposes:

Hash	Purpose
`operation_name_hash`	Route to correct service / operation registry
`operation_text_hash`	Look up persisted query text on the server
`client_doc_id`	Composite cache/CDN key (name hash + text hash concatenated)

This is the persisted queries pattern: the full GraphQL query text is never sent over the network. The client sends only the doc_id and variables. The server looks up the query text by hash from a pre-registered store and executes it.

Why this matters:

Bandwidth savings — A complex mutation with nested fragments might be 2–3KB of query text. Sending a 26-character hash instead adds up across 2.5 billion daily users.
Security — Arbitrary query execution is impossible. The server will only run pre-registered, pre-validated queries. This eliminates an entire category of GraphQL abuse (deeply nested queries, field enumeration, introspection attacks).
Build-time validation — Operations are validated against the schema at build time. Type mismatches and missing fields are caught before the APK ships, not at runtime.
Analytics — Every operation is identified by a stable hash, making performance tracking and error attribution trivial even across client versions.

The Wire Protocol

What the client actually sends:

POST https://graph.whatsapp.net/graphql
{
  "doc_id": "7930051506669169098671059140",
  "variables": {
    "photos": ["base64_photo_1", "base64_photo_2"],
    "profile_id": "123456789"
  }
}

What the server resolves and executes:

mutation AddMEmuProfilePhotos($photos: [String!]!, $profile_id: ID!) {
  addMEmuProfilePhotos(photos: $photos, profile_id: $profile_id) {
    success
    profile {
      id
      photo_count
      preview_url
    }
    errors {
      code
      message
    }
  }
}

The query text never touches the network. The client binary contains hashes; the server holds the query registry.

Master-Slave Routing

The schema field in each operation manifest determines which GraphQL master handles the request:

schema: "whatsapp"     → graph.whatsapp.net
schema: "whatsapp_mex" → Internal MEX gateway (higher trust, separate auth)
schema: "facebook"     → graph-www.facebook.com

Each master can then fan out to specialized slave services:

graph.whatsapp.net (whatsapp master)
├── genai-graph.facebook.com    (AI image generation)
├── payments-graph.facebook.com (UPI/Pix processing)
└── graph.oculus.com            (AR/smart glasses)

MEX gateway (whatsapp_mex master)
├── Compliance services         (DMA enforcement, audit logging)
├── PAA services                (parental supervision)
└── Monetization services       (Wamo/channel subscriptions)

graph-www.facebook.com (facebook master)
└── Ads services                (Meta Ads, business tools)

Full Operation Routing Table

Operation Prefix	Schema	Master Endpoint	Slave Endpoints
`GenAI`, `Imagine`, `MEmu*`	`whatsapp`	`graph.whatsapp.net`	`genai-graph.facebook.com`, `genai-graph.instagram.com`
`Payment`, `Upi`, `Pix`	`whatsapp`	`graph.whatsapp.net`	`payments-graph.facebook.com`
`Interop`, `Paa`, `Passkey`	`whatsapp_mex`	Internal MEX gateway	Compliance services
`Newsletter`, `Wamo`	`whatsapp_mex`	Internal MEX gateway	Monetization services
`Ad`, `Commerce`, `DigitalContent`	`facebook`	`graph-www.facebook.com`	Ads services
`Oculus`, `AR`	`whatsapp`	`graph.whatsapp.net`	`graph.oculus.com`

3. Sample Mutations by Domain

A. Generative AI (MEmu / Imagine)

# Create AI avatar profile from user photos
mutation CreateMEmuProfile($photos: [String!]!, $name: String) {
  createMEmuProfile(photos: $photos, name: $name) {
    profile_id
    status
    preview_url
    estimated_completion_seconds
  }
}

# Generate image from text prompt
mutation GenAIImagineGenerateMutation($prompt: String!, $style: ImagineStyle) {
  genAIImagineGenerate(prompt: $prompt, style: $style) {
    generation_id
    image_url
    thumbnail_url
    seed
    prompt_embedding
  }
}

# Animate a static image
mutation GenAIEditAnimateMutation($image_id: ID!, $animation_style: AnimationStyle) {
  genAIEditAnimate(image_id: $image_id, animation_style: $animation_style) {
    video_url
    duration_ms
    frames
  }
}

# Bulk send AI-generated media to a chat
mutation GenAIImagineBulkSendMediaToChat($media_ids: [ID!]!, $chat_jid: String!) {
  genAIImagineBulkSendMediaToChat(media_ids: $media_ids, chat_jid: $chat_jid) {
    success_count
    failed_ids
    message_id
  }
}

Note prompt_embedding in the Imagine response — the server returns the semantic embedding of the prompt, not just the image. This is likely used for client-side related prompt suggestions and "try similar" features without requiring another round-trip.

B. Payments (UPI / Pix)

# Generate payment key for UPI transaction
mutation GenCreatePaymentKey(
  $amount: Int!,
  $currency: String!,
  $receiver_vpa: String!
) {
  genCreatePaymentKey(
    amount: $amount,
    currency: $currency,
    receiver_vpa: $receiver_vpa
  ) {
    payment_key
    expiry_seconds
    upi_qr_code_url
  }
}

# Complete a Pix transaction (Brazil)
mutation CompletePixTransaction($transaction_id: ID!, $pix_key: String!) {
  completePixTransaction(
    transaction_id: $transaction_id,
    pix_key: $pix_key
  ) {
    status
    receipt_url
    confirmation_code
    error_code
  }
}

# UPI onboarding OTP
mutation UpiOnboardingSendOtpMutation($phone_number: String!, $bank_code: String) {
  upiOnboardingSendOtp(phone_number: $phone_number, bank_code: $bank_code) {
    otp_ref_id
    resend_timeout_seconds
    masked_phone
  }
}

The separation of GenCreatePaymentKey (which talks to UPI rails) from CompletePixTransaction (Brazil's Pix system) reflects the reality that WhatsApp Payments is deployed differently by region. India uses UPI with NPCI settlement; Brazil uses Pix with Banco Central do Brasil oversight. These are distinct regulatory environments with distinct backend integrations, and the mutation structure reflects that.

C. MEX Operations (DMA / Parental / Passkeys)

# Create interoperable group with third-party platform users
mutation GroupsCreateInteropGroup(
  $name: String!,
  $participants: [InteropParticipantInput!]!
) {
  groupsCreateInteropGroup(name: $name, participants: $participants) {
    group_id
    invite_link
    external_platform_mappings {
      platform
      external_id
      invite_url
    }
  }
}

# Child accepts parental supervision
mutation PaaAcceptLinkingMutation($supervisor_id: ID!, $pin: String!) {
  paaAcceptLinking(supervisor_id: $supervisor_id, pin: $pin) {
    status
    supervision_level
    activity_sharing_enabled
    next_review_date
  }
}

# Complete FIDO2 passkey registration
mutation RegistrationPasskeyFinishRegisterMutation(
  $challenge: String!,
  $attestation: String!
) {
  registrationPasskeyFinishRegister(
    challenge: $challenge,
    attestation: $attestation
  ) {
    success
    passkey_id
    backup_eligible
  }
}

The backup_eligible field in the passkey response is significant — it maps to the FIDO2 concept of a "multi-device credential" (a passkey synced via iCloud Keychain or Google Password Manager) vs. a "single-device credential" (hardware-bound, not synced). WhatsApp is tracking this distinction per passkey, which matters for account recovery: if your only passkey is hardware-bound and you lose the device, there's no backup path.

D. Newsletter / Channel Mutations

# Create a verified channel
mutation NewsletterCreateVerified(
  $name: String!,
  $description: String,
  $verification_docs: [String!]
) {
  newsletterCreateVerified(
    name: $name,
    description: $description,
    verification_docs: $verification_docs
  ) {
    newsletter_id
    vanity_url
    verification_status
    admin_ids
  }
}

# Enable paid subscription (Wamo)
mutation NewsletterEnableWamoSub(
  $newsletter_id: ID!,
  $monthly_price: Int!,
  $currency: String!
) {
  newsletterEnableWamoSub(
    newsletter_id: $newsletter_id,
    monthly_price: $monthly_price,
    currency: $currency
  ) {
    subscription_tier
    payout_account_status
    subscriber_count
  }
}

payout_account_status confirms that creator payouts are a first-class concern in the data model — not a post-hoc integration with an external payments provider, but something WhatsApp tracks directly. Combined with the currency field accepting multiple values, this suggests Wamo payouts will support multiple currencies from launch, relevant for WhatsApp's largest markets (India, Brazil, Nigeria, Indonesia).

4. The QPL Telemetry Pipeline

HWQ hwq = new HWQ(..., "https://graph.whatsapp.net/wa_qpl_data", ...);
hwq.A08("access_token", "1063127757113399|745146ffa34413f9dbb5469f5370b7af");

QPL = Query Performance Logger — Meta's internal distributed tracing system. Every GraphQL operation, database query, and UI render sends performance telemetry to this endpoint.

The hardcoded token is a public app token scoped exclusively to QPL data ingestion:

App ID 1063127757113399 = WhatsApp QPL application
Scope is write-only to wa_qpl_data — it cannot read user data, contact lists, or message content
Exposing it in the binary is intentional — it's the equivalent of a public write-only API key for a logging endpoint

The QPL Mutation

mutation LogPerformanceTrace(
  $trace_id: ID!,
  $operation_name: String!,
  $duration_ms: Int!,
  $cache_hit: Boolean!,
  $error_code: Int
) {
  logPerformanceTrace(
    trace_id: $trace_id,
    operation_name: $operation_name,
    duration_ms: $duration_ms,
    cache_hit: $cache_hit,
    error_code: $error_code
  ) {
    accepted
    sampling_rate
  }
}

Note the sampling_rate in the response — the server tells the client what percentage of subsequent traces to send. This is dynamic sampling: during low-traffic periods, sample 100% of traces; during peak load, drop to 1% to avoid the telemetry pipeline itself becoming a bottleneck. The client adjusts its reporting rate in real time based on server guidance.

This telemetry infrastructure allows Meta engineers to:

Detect slow GraphQL resolvers (p95 latency by operation name)
Optimize cache hit rates per operation
Identify error spikes before they surface in user complaints
A/B test infrastructure changes with statistical confidence
Correlate client-side duration with server-side resolver timing

For a 2.5 billion user app, even 1% sampling gives millions of data points per hour. The QPL pipeline is how Meta knows when something is slow before users notice.

5. Third-Party Integrations

Giphy Sticker Search

public String A03(CharSequence charSequence, String str) {
    String[] strArr = new String[10];
    strArr[0] = "api_key";
    strArr[1] = AbstractC11310fq.A0M;  // Giphy API key
    strArr[2] = "q";
    strArr[4] = "lang";
    strArr[6] = "rating";
    strArr[7] = "pg-13";               // Hardcoded content filter
    strArr[8] = "limit";
    strArr[9] = "100";
    return FIO.A00("https://api.giphy.com/v1/stickers/search", strArr);
}

The pg-13 rating filter is hardcoded, not configurable. This means adult content is filtered at the API call level regardless of user age or settings — a deliberate product decision, not a Giphy default. WhatsApp is a family communication app in most of its core markets; this is a conscious content policy choice baked into the client binary.

The architecture here is a direct proxy: WhatsApp calls Giphy from the client, not through a Meta backend. Giphy data never enters Meta's GraphQL infrastructure. This keeps Giphy content legally and architecturally separate from Meta's data processing — relevant for EU data transfer regulations.

User search query → WhatsApp client → Giphy API directly → Results returned to client
                                     (Meta servers never see Giphy content)

Oculus CDN Asset Structure

https://scontent.oculuscdn.com/v/t64.5771-25/499303769_711556704750162_3777318892782513230_n.mp4
  ?_nc_cat=105        → CDN category (105 = video asset)
  &ccb=1-7            → Cache control bucket
  &_nc_sid=5f5f54     → Session ID for analytics
  &_nc_ohc=...        → Oculus content hash (content-addressed)
  &_nc_oc=...         → Origin content ID
  &_nc_zt=28          → Zero-trust security token
  &_nc_ht=scontent.oculuscdn.com
  &oh=00_Afn4...      → Origin hash (integrity check)
  &oe=69457811        → Origin expiration (Unix timestamp)

The _nc_zt (zero-trust token) and oh (origin hash) parameters together implement signed URL access control — the URL is only valid for a specific client session for a limited time window (oe = expiry). Sharing or scraping the URL without a valid session token returns 403. This is standard CDN security for premium content but interesting to see applied to onboarding/tutorial videos.

The content (t64.5771-25 type prefix) is likely a smart glasses onboarding video or AR effect tutorial served to WhatsApp clients supporting Meta's Ray-Ban Stories / Orion integration.

6. Static Asset Architecture

WhatsApp uses purpose-built static domains for different asset types:

// Sticker images (individual)
"https://static.whatsapp.net/sticker?img="

// Sticker packs (bulk)
"https://wa.me/stickerpack/%s"

// Network resources (animations, success states)
"https://static.whatsapp.net/wa/static/network_resource
  ?cat=nw_media&id=username_success_confetti_tall_green"

// Group invite links
"https://chat.whatsapp.com/"

// Meta design system assets (shared across all Meta apps)
"https://lookaside.facebook.com/assets/key/meta_brand_design_system_icons_raster"

lookaside.facebook.com is Meta's global asset delivery network for design system assets — icons, color tokens, raster graphics — shared across WhatsApp, Instagram, Facebook, and Messenger. When Meta updates an icon in their design system, it propagates to all four apps via a CDN cache invalidation rather than four separate app releases. This enables consistent visual branding across the Meta family without requiring coordinated app updates.

The network_resource endpoint for animations (confetti, success states) is particularly efficient — these are not bundled in the APK but fetched on demand and cached. The APK stays smaller; the CDN handles distribution of infrequently-used animation assets.

7. Map Infrastructure

static {
    CCR ccr = new CCR(
        "https://www.facebook.com/maps/tile/?",    // Tile server
        "https://www.facebook.com/maps/static/?",  // Static map images
        null, null, null, Integer.MAX_VALUE
    );
    A09 = ccr;  // Facebook maps config

    A0A = new CCR(
        "https://maps.instagram.com/maps/tile/?",
        "https://maps.instagram.com/maps/static/?",
        null, null, null, Integer.MAX_VALUE
    );  // Instagram maps config
}

The map configuration is fetched dynamically:

GET https://graph.whatsapp.net/v2.2/maps_configs
  ?fields=base_url,static_base_url,osm_config,url_override_config
  &pretty=0
  &access_token=TOKEN

Key architectural insight: the tile server URL is fetched from the server, not hardcoded. The url_override_config field allows Meta to hot-switch the tile provider (from Facebook's servers to a third-party or regional provider) without an app update. This is useful for regional compliance — in markets where Facebook infrastructure is blocked or restricted, the tile server can be redirected to a neutral CDN.

WhatsApp uses OpenStreetMap (osm_config) for map data — community-maintained, license-compatible, and not subject to Google Maps API pricing. Facebook and Instagram serve as tile renderers on top of OSM data, enabling:

Location sharing with live map previews
Business location display in profiles
Event location maps
Location-based AR effects

8. End-to-End Flow: "Imagine Me"

Putting it all together with a concrete example. When a user sends "imagine me as an astronaut" to Meta AI in WhatsApp:

1. Client hashes operation
   CreateMEmuProfile → doc_id: "1941315421..."

2. Client sends to whatsapp master
   POST graph.whatsapp.net/graphql
   { doc_id: "1941315421...", variables: { photos: [...], prompt: "astronaut" } }

3. WhatsApp master validates
   - Auth token verification
   - Rate limit check (per-user, per-operation)
   - Persisted query lookup by doc_id

4. Master routes to GenAI slave
   POST genai-graph.facebook.com/graphql
   (with scoped auth token, no contact list access)

5. GenAI slave processes
   - Photo embedding generation
   - Diffusion model inference
   - Result CDN upload

6. Response propagates back
   genai slave → whatsapp master → client

7. QPL telemetry fires
   POST graph.whatsapp.net/wa_qpl_data
   { operation: "CreateMEmuProfile", duration_ms: 4200, cache_hit: false }

Total round-trips: 2 (mutation + telemetry). The GenAI slave's data access is cryptographically scoped — it processes photos for the AI task but has no API access to the user's contacts, messages, or other profile data.

9. Security Architecture

Token Scoping (Principle of Least Privilege)

The QPL token demonstrates Meta's token scoping model:

1063127757113399|745146ffa34413f9dbb5469f5370b7af
└── App ID ────┘└── App secret (write-only, QPL scope) ─┘

Can write to wa_qpl_data only
Cannot read any user data
Cannot execute GraphQL mutations outside QPL schema
Public exposure is intentional — it's a write-only logging key

Each slave service receives a scoped token from the master with only the permissions needed for that operation. The GenAI slave token cannot query the payments service. The payments token cannot access the contact graph. The MEX gateway uses a higher-trust token with audit logging enabled.

Zero-Rating Bootstrap

b-graph.facebook.com is a zero-rated endpoint — accessible even when the user has no mobile data balance. Operators in participating markets (primarily in South and Southeast Asia, sub-Saharan Africa) have agreements with Meta to not charge data for this subdomain.

The endpoint exposes only a strict subset of the full API:

Emergency registration and SIM verification
Critical security updates and key rotation
Basic message queuing (not full messaging)
Account recovery flows

This prevents abuse (zero-rated endpoints are high-value targets for data scraping) while ensuring essential functionality remains accessible to users on prepaid plans. The restricted schema means even if someone discovers the zero-rated endpoint, they can't use it to access arbitrary API functionality for free.

Persisted Queries as a Security Control

The operation hashing system is also a security mechanism. Because the server only executes pre-registered queries:

No introspection attacks — you can't query __schema to enumerate available fields
No batch abuse — you can't craft a custom query that joins 10 expensive resolvers
No field enumeration — you can't discover undocumented fields by probing the schema
Audit trail — every executed query is identifiable by a stable hash in logs

This is a meaningful security posture for an app handling 100+ billion messages per day.

Summary

WhatsApp's URL and GraphQL architecture reveals a federated master-slave system built around three core principles:

Security isolation over convenience. Separate endpoints, separate auth tokens, separate data scopes. The GenAI service literally cannot access your contact list, by architecture, not just by policy.

Operational independence. Each domain (AI, payments, messaging, compliance) deploys, scales, and fails independently. A viral "imagine me" moment doesn't degrade core messaging performance.

Regulatory partitioning. The MEX gateway, the zero-rating bootstrap, the scoped token model — all of these are designed for a world where different operations face different regulatory requirements in different jurisdictions. The architecture makes compliance enforceable at the infrastructure level, not just the policy level.

For an app serving 2.5 billion users across radically different regulatory environments — EU DMA, India UPI regulations, Brazil Pix, US compliance requirements — this kind of federated, domain-isolated architecture isn't over-engineering. It's the minimum viable structure for operating at that scale and that regulatory complexity simultaneously.

Analysis based on decompiled GraphQL operation manifests and URL patterns from WhatsApp Android 2.26.3.79.

The "MEX" Layer: Inside WhatsApp's EU DMA Compliance Architecture

Black Lover — Sun, 29 Mar 2026 06:35:05 +0000

Source: Decompiled GraphQL operation manifests from WhatsApp Android version 2.26.3.79. All operation names and structures are as observed in the client codebase.

WhatsApp is undergoing the most significant architectural shift in its history — and most of it is invisible to users. Hidden inside recent Android builds is a higher-trust GraphQL schema called whatsapp_mex. MEX stands for Meta Experience, and it's where the platform's most regulated, security-sensitive operations live: EU Digital Markets Act interoperability, parent-child account supervision, passkey authentication, and cross-Meta identity linking.

This is a technical deep-dive into what that schema reveals about where WhatsApp — and Meta — is headed.

The Three-Layer Architecture

Before getting into specifics, it helps to understand how WhatsApp's backend is now structured:

Layer	Schema	Purpose
Public	`whatsapp`	General messaging, AI features, payments
MEX	`whatsapp_mex`	High-trust ops: interop, parental controls, identity linking, monetization
Facebook Graph	`facebook`	Cross-Meta services: ads, digital content, business tools

Each layer has separate authentication requirements, rate limits, and security boundaries. The MEX layer is specifically designed for operations with legal or regulatory implications — DMA compliance, child safety, identity verification — which warrant additional audit trails and enforcement controls. If an operation could result in a lawsuit, it's probably in MEX.

1. DMA Interoperability: The Crown Jewel

The EU Digital Markets Act requires "gatekeepers" — platforms with over 45 million monthly EU users — to open their messaging infrastructure to third-party platforms. Meta has until March 2024 to comply. The MEX schema shows exactly how they're doing it.

Core Interop Operations

AddParticipantsToInteropGroup           # Add users from third-party platforms
GroupsCreateInteropGroup                # Create DMA-compliant mixed groups
LeaveInteropGroup                       # Exit interoperable groups
QueryInteropGroupInfo                   # Fetch third-party participant metadata
InteropPrivacySettingsQuery             # Get user's cross-platform privacy prefs
InteropPrivacySettingsUpdate            # Update third-party visibility settings
InteropPrivacySettingWithContactListUpdate  # Per-contact granular controls

What the Operations Actually Tell Us

The most revealing operation is InteropPrivacySettingWithContactListUpdate. The name implies users can control which third-party platforms can reach them on a per-contact basis — not a simple on/off toggle. This means WhatsApp is maintaining a contact resolution layer that maps third-party identifiers (a Signal phone number, a Telegram username) to internal WhatsApp IDs while keeping the user's privacy preferences intact at the contact level.

GroupsCreateInteropGroup is equally significant. DMA compliance isn't just about receiving messages from other platforms — WhatsApp users will be able to create mixed-platform groups that include Signal, Telegram, or other designated contacts. WhatsApp shifts from walled garden to interoperability hub.

The Encryption Problem

Each interop group must maintain end-to-end encryption across platforms with fundamentally different cryptographic protocols. WhatsApp uses the Signal Protocol. Telegram uses MTProto. These don't natively interoperate.

The presence of these GraphQL operations strongly suggests Meta has built a federation translation layer — a service that bridges encryption schemes at the boundary without breaking the security guarantees of either platform. This is technically non-trivial and represents significant engineering investment. The exact design of this layer (gateway model vs. client-side key negotiation) isn't visible from the schema alone, but its existence is implied.

Privacy Implications

DMA interoperability is opt-in. But the granular controls in the schema suggest users will have real agency over which platforms can contact them and which contacts from those platforms can reach them. Whether those controls are surfaced clearly in the UI is a separate — and important — question.

2. Parent-Child Accounts (PAA): The Teen Safety Stack

The MEX schema contains a complete parental supervision framework. The internal codename is PAA — Parent/Child Accounts.

Full PAA Lifecycle

PaaInitiateLinkingQuery        # Parent initiates supervision request
PaaAcceptLinkingMutation        # Child accepts parental oversight
PaaValidateLinkingQuery          # Verify the supervision relationship
PaaCompleteLinkingMutation        # Finalize account linking
PaaQuery                           # Get current supervision status
PaaRevokeLinkingMutation            # Terminate supervision
PaaUpdatePinMutation                 # Change supervision PIN
PaaSyncActivities                     # Sync child activity to parent
PaaGetSponsorAgeVerificationInfoQuery  # Age verification for the parent/sponsor

The Age Verification Integration

UI strings from plurals.xml give away the verification flow:

"Confirm you are %d year(s) old"
"Are you %d years old?"

Combined with SubmitAge and AgeCollection operations in MEX, this reveals a multi-step age confirmation process. More importantly, PaaGetSponsorAgeVerificationInfoQuery shows that the parent (the "sponsor") must verify their own age before they can supervise a minor. This likely integrates with government ID verification or carrier records — a significant compliance requirement, especially under GDPR and the UK's Online Safety Act.

The Activity Monitoring Problem

PaaSyncActivities is the operation that will attract the most scrutiny. It implies a background sync where child accounts share selected activity data with parent accounts — contacts, groups, possibly usage patterns.

This creates a genuine design tension: parents need enough visibility to protect their children, but too much visibility violates the child's privacy expectations and potentially local regulations (in the EU, children have meaningful privacy rights even from parents). How WhatsApp resolves this tension in the actual UI and data model will determine whether the feature passes regulatory review or invites a GDPR investigation.

Comparison to Existing Approaches

Apple Screen Time and Google Family Link operate at the OS level, giving parents app-level usage data but not message content. WhatsApp's PAA approach is in-app and message-platform-adjacent, which is both more targeted and more legally complex. The framework appears designed for opt-in supervision with the child's consent, not covert monitoring — the PaaAcceptLinkingMutation requiring the child to accept is a key signal here.

3. Passkeys: Killing the SMS OTP

WhatsApp has long relied on SMS OTP for authentication — a system that is simultaneously friction-heavy for users and relatively easy to attack via SIM swapping. The MEX schema shows a comprehensive FIDO2/WebAuthn passkey implementation designed to replace it entirely.

Full Passkey Lifecycle

PasskeyStartRegisterMutation                        # Begin FIDO2 enrollment
PasskeyFinishRegisterMutation                       # Complete registration
PasskeyDeleteMutation                               # Remove a passkey
PasskeyExistResponseQuery                           # Check if passkey exists
PasskeyListExistResponseQuery                       # List all registered passkeys
RegistrationPasskeyEnableMutation                   # Enable passkey for account
RegistrationPasskeyClear                            # Remove all passkeys
RegistrationPasskeyUpdateClientEncryptionStatusMutation  # Update E2E key metadata

Multiple Passkeys Per Account

The separation between PasskeyExistResponseQuery (singular) and PasskeyListExistResponseQuery (plural) indicates support for multiple passkeys per account — phone, tablet, laptop, hardware key. This is the right design for a mobile-first app where users frequently change devices or use multiple simultaneously.

Passkeys Integrated with E2E Encryption

RegistrationPasskeyUpdateClientEncryptionStatusMutation is the most technically interesting operation. It suggests passkeys aren't just for authentication — they're woven into WhatsApp's end-to-end encryption key management. The likely use case: passkeys are used to seal or unlock the user's local encryption key store, so even if the device is compromised at the OS level, message history stays protected.

Registration-Time Enrollment

RegistrationPasskeyStartRegisterMutation and RegistrationPasskeyFinishRegisterMutation show passkeys can be configured during initial account setup, not just as a post-registration security upgrade. This is the path to eliminating SMS OTP dependency entirely — new users on supported devices skip the phone verification step and go straight to biometric/hardware passkey enrollment.

For WhatsApp's scale (~2.5 billion users), this represents one of the largest FIDO2 deployments in history.

4. IPLs: Meta's Unified Identity Layer

The IplsClientHandshakeInitRequest and IplsClientHelloPayload operations are among the most strategically significant in the entire schema. IPLs = Identity/Privacy Linking Service.

What IPLs Actually Does

IPLs is the cryptographic foundation for a unified Meta identity spanning WhatsApp, Instagram, Facebook, Threads, and potentially third-party services. The "handshake" and "hello" naming is deliberate — it mirrors TLS handshake terminology and suggests a multi-step cryptographic protocol:

WhatsApp client generates a nonce and identity proof
IPLs service validates this proof against other Meta services
A cross-service session token is established
User can move between Meta properties with seamless, auditable authentication

The OAuth-Like Flows

WWWCreateAccessToken               # Create cross-service access token
WWWExchangeNonceForAccessToken      # Exchange one-time token for access
WWWTradeNonceForAccessTokens         # Bulk token exchange (multiple services)
WWWTriggerAccountRecovery            # Cross-service account recovery
WWWValidateCanonicalUser             # Verify canonical identity across Meta

These are the OAuth-like flows enabling cross-service identity. When you link Instagram to WhatsApp, this machinery handles the token exchange, identity verification, and session establishment — cryptographically secure and auditable.

Account Recovery Across Meta

WWWTriggerAccountRecovery and WWWValidateCanonicalUser reveal something genuinely useful for end-users: if you lose access to WhatsApp but have a verified Instagram or Facebook account, the IPLs infrastructure can provide a cross-service recovery pathway. This addresses a long-standing pain point where losing your SIM card could mean permanently losing your WhatsApp account.

Regulatory Tension

IPLs is also the thing that worries EU regulators most. The DMA's interoperability requirements are specifically designed to prevent Meta from using its dominance in one app to entrench its position across all apps. A unified identity layer that makes it seamless to move between Meta properties could be read as doing exactly that. Expect this to be scrutinized in DMA enforcement proceedings.

5. Username System: Dropping Phone Numbers

The MEX schema contains a complete username infrastructure — WhatsApp's move toward @usernames similar to Instagram and Telegram.

UsernameCheck    # Validate username availability
UsernameSet      # Claim username
UsernameGet      # Resolve username to account
UsernamePinSet   # PIN-protect username (privacy control)

The PIN-Protection Approach

UsernamePinSet is the most interesting operation and reveals a novel privacy design. In Telegram, a public username means anyone can message you. In WhatsApp's model, usernames can be PIN-protected — someone needs both your username and a PIN to initiate contact. This is a meaningful spam and harassment prevention mechanism that has no equivalent in Telegram or Signal.

It also suggests WhatsApp is solving for a specific user concern: people want to be discoverable without being reachable by everyone. A PIN-gated username lets you share @yourname selectively while keeping random contact attempts blocked.

The Phone Number Transition

Currently, WhatsApp's identity is your phone number — changing phones, changing SIM cards, or moving countries creates friction or data loss. A username-based identity layer decouples WhatsApp accounts from phone numbers, which is prerequisite infrastructure for both the IPLs unified identity system and long-term competitive positioning against Telegram and Signal.

6. Channels/Newsletters: A Full Platform, Not a Feature

The MEX schema contains over 30 newsletter/channel operations. WhatsApp Channels is not a simple feature addition — it's a full-fledged content platform.

Channel Management Operations

NewsletterCreate               # Launch a channel
NewsletterCreateVerified        # Verified/blue-tick channels
NewsletterDelete                 # Remove channel
NewsletterMetadataUpdate          # Change channel details
NewsletterAdminInvite              # Add co-admins
NewsletterAdminDemote               # Remove admin privileges
NewsletterChangeOwner                # Transfer channel ownership
NewsletterJoin / NewsletterLeave      # Subscribe / unsubscribe
NewsletterHide / NewsletterUnhide      # Mute / unmute

The Monetization Layer

NewsletterEnableWamoSub    # Enable paid subscription
NewsletterDisableWamoSub   # Disable paid subscription
NewsletterChangeWamoSub    # Modify subscription tier

"Wamo" = WhatsApp Monetization. These operations confirm that paid channel subscriptions are in active development — creators will be able to charge for premium content access. This is WhatsApp's direct answer to Telegram's paid channels and Substack's newsletter model. Combined with WhatsApp's 2.5 billion user base, the distribution potential for creators is significant.

The Discovery Engine

NewsletterDirectoryList     # Browse channel directory
NewsletterDirectorySearch   # Search channels
NewsletterRecommended       # Algorithmic recommendations
NewsletterSimilar           # "Channels like this"

This is a full content discovery stack — browse, search, algorithmic recommendations, and similarity matching. WhatsApp is building the infrastructure for a content discovery experience inside what has traditionally been a private messaging app. The algorithmic recommendation component (NewsletterRecommended) is a significant departure from WhatsApp's historically non-algorithmic, chronological design philosophy.

7. Multi-Account Infrastructure

AddMultiAccountLink        # Add secondary account
MultiAccountRevokeAccount  # Remove linked account

This is more significant than it appears. The "link" terminology suggests these accounts can be related (personal + work, or parent + family account) and may share selected resources while maintaining conversation separation. Combined with the IPLs identity layer, this could eventually enable seamless switching between multiple Meta identities — different personas, different contexts — within a single app instance.

For context: Telegram has supported multiple accounts since 2017. WhatsApp is late to this, but the MEX implementation suggests they're building it with a more sophisticated identity model than simple account switching.

8. Compliance and Enforcement Infrastructure

GetCompliance / SetCompliance   # User compliance status
CreateEnforcementAppeal          # Appeal account bans/restrictions
FetchUserNoticesByID              # Get policy violation notices
ContactIntegrityQuery             # Verify contact authenticity

This is the backend for WhatsApp's transparency and appeals process. When accounts are banned or restricted, users can view notices, understand the enforcement action, and formally appeal. The ContactIntegrityQuery operation is less obvious — it likely supports detecting fake or compromised contacts, relevant to both spam prevention and the DMA's requirement that interoperating platforms maintain security standards equivalent to WhatsApp's own.

What This Architecture Reveals About Meta's Strategy

Reading the MEX schema as a whole, a coherent strategy emerges:

Regulatory compliance as infrastructure, not afterthought. DMA interoperability, GDPR-aligned parental controls, and auditable identity operations are built into a dedicated schema with separate security boundaries — not bolted onto existing messaging infrastructure.

WhatsApp as Meta's identity backbone. WhatsApp has something Facebook and Instagram don't: near-universal phone-verified identity. The IPLs layer and passkey infrastructure are building on this foundation to create a cross-Meta authentication system that's more secure than password-based login and more portable than phone-number-based identity.

The creator economy play. Paid channels, algorithmic discovery, and verified creator accounts position WhatsApp to compete with Telegram, Substack, and YouTube in markets (India, Brazil, Nigeria) where WhatsApp is more universally installed than any of those alternatives.

Privacy as competitive advantage. The username PIN system, the granular interop privacy controls, and the opt-in supervision model suggest WhatsApp is trying to maintain its privacy positioning (relative to Facebook and Instagram) even as it opens up to interoperability and cross-Meta identity.

Whether they can maintain that positioning as these systems roll out — under DMA enforcement scrutiny, GDPR oversight, and competitive pressure — is the open question. But the architecture, at least, is more thoughtfully designed than most observers expected.

Architecture Summary

┌─────────────────────────────────────────────┐
│              WhatsApp Client                │
├──────────────┬──────────────┬───────────────┤
│   whatsapp   │ whatsapp_mex │   facebook    │
│   (public)   │    (MEX)     │  (cross-Meta) │
├──────────────┼──────────────┼───────────────┤
│  Messaging   │  DMA Interop │  Ads / DCA    │
│  AI features │  PAA / Teen  │  Business     │
│  Payments    │  Passkeys    │  tools        │
│  Status      │  IPLs / SSO  │               │
│              │  Usernames   │               │
│              │  Channels    │               │
└──────────────┴──────────────┴───────────────┘
         ↓              ↓              ↓
   Standard auth   MEX auth +    FB Graph token
                  audit trail

The MEX layer isn't just a technical partition — it's a legal and compliance partition. Operations that could have regulatory consequences live in a separate schema with separate authentication and presumably separate logging and audit infrastructure. That's deliberate design for a company navigating DMA enforcement, GDPR investigations, and teen safety legislation simultaneously.

Analysis based on decompiled GraphQL operation manifests from WhatsApp Android 2.26.3.79. Operation names and structures are as observed in the client binary.

Next.js Performance Optimization: From 3s to 0.8s Load Time

Black Lover — Sun, 29 Mar 2026 06:28:23 +0000

Our DockForge application was loading in 3.2 seconds. Here's the full playbook we used to get it down to 0.8s — including the profiling workflow we ran before writing a single line of optimization code.

⚡ Step 0: Real-World Profiling Workflow

The biggest mistake teams make is optimizing by instinct. Before touching anything, build a baseline with real data.

Run Lighthouse in CI, not just DevTools

DevTools Lighthouse runs on your machine with your hardware. For consistent results, use the CLI:

npm install -g lighthouse
lighthouse https://your-app.com --output=json --output-path=./report.json \
  --chrome-flags="--headless" --throttling-method=simulate

Or integrate it into CI so every PR shows a score diff:

# .github/workflows/lighthouse.yml
- name: Run Lighthouse
  uses: treosh/lighthouse-ci-action@v10
  with:
    urls: |
      https://staging.your-app.com
    budgetPath: ./budget.json
    uploadArtifacts: true

Use Chrome DevTools Performance tab properly

Open DevTools → Performance tab
Click the gear icon → set CPU throttle to 4x slowdown (simulates a mid-range Android)
Record a full page load
Look for Long Tasks (red bars) — anything over 50ms blocks the main thread

Track Core Web Vitals in production

Lighthouse is synthetic. Real users are different. Add this to your root layout:

// app/layout.tsx
export function reportWebVitals(metric) {
  // Send to your analytics (Vercel, Datadog, etc.)
  console.log(metric); // { name: 'LCP', value: 2400, ... }
}

Or use the web-vitals package for more control:

import { onCLS, onFID, onLCP, onFCP, onTTFB } from 'web-vitals';

function sendToAnalytics({ name, value, id }) {
  fetch('/api/vitals', {
    method: 'POST',
    body: JSON.stringify({ name, value, id }),
  });
}

onCLS(sendToAnalytics);
onLCP(sendToAnalytics);
onFCP(sendToAnalytics);

Only optimize what you've measured. Then measure again after.

📦 Bundle Analysis: Find What's Killing Your Load Time

Before optimizing code, you need to see what's actually in your bundle.

Install the analyzer

npm install --save-dev @next/bundle-analyzer

// next.config.js
const withBundleAnalyzer = require('@next/bundle-analyzer')({
  enabled: process.env.ANALYZE === 'true',
});

module.exports = withBundleAnalyzer({
  // your existing config
});

ANALYZE=true npm run build

This opens an interactive treemap in your browser. Look for:

Suspiciously large dependencies — moment.js (67KB gzipped), lodash (24KB), etc.
Duplicate packages — two versions of React, two versions of a utility lib
Server-only code leaking into client bundles — DB clients, secrets, heavy Node libs

Common offenders and fixes

Moment.js → date-fns

# moment.js: 67KB gzipped
# date-fns (tree-shakeable): ~3KB per function used
npm uninstall moment
npm install date-fns

// Before
import moment from 'moment';
moment(date).format('YYYY-MM-DD');

// After
import { format } from 'date-fns';
format(date, 'yyyy-MM-dd');

Lodash → lodash-es

// Before: imports entire lodash (24KB)
import _ from 'lodash';
_.groupBy(items, 'category');

// After: tree-shakeable, only imports groupBy
import groupBy from 'lodash-es/groupBy';
groupBy(items, 'category');

Enable optimized package imports for large icon/UI libraries:

// next.config.js
module.exports = {
  experimental: {
    optimizePackageImports: ['lucide-react', '@mui/material', 'antd'],
  },
};

🖼️ 1. Image Optimization

The single highest-impact change in most Next.js apps.

Before:

<img src="/logo.png" alt="Logo" />

After:

import Image from 'next/image';

<Image
  src="/logo.png"
  alt="Logo"
  width={200}
  height={50}
  priority            // preloads LCP images
  placeholder="blur"
  blurDataURL="data:image/jpeg;base64,..."
/>

Why `priority` matters

Only add priority to images above the fold — your hero image, logo, or anything visible on load. It injects a <link rel="preload"> tag so the browser fetches it before parsing the rest of the page. Using it on everything defeats the purpose.

Generate blur placeholders at build time

Instead of hardcoding a blurDataURL, generate it dynamically:

// lib/imageUtils.js
import { getPlaiceholder } from 'plaiceholder';

export async function getBlurData(src) {
  const { base64 } = await getPlaiceholder(src);
  return base64;
}

// In your page (server component)
const blurDataURL = await getBlurData('/hero.jpg');

Remote images need explicit domains

// next.config.js
module.exports = {
  images: {
    remotePatterns: [
      {
        protocol: 'https',
        hostname: 'cdn.your-app.com',
        pathname: '/images/**',
      },
    ],
    formats: ['image/avif', 'image/webp'], // AVIF is ~40% smaller than WebP
  },
};

Result: 60% smaller images with automatic WebP/AVIF conversion, near-zero CLS from blur placeholders.

⚡ 2. Code Splitting

Next.js handles route-level splitting automatically. The wins come from component-level splitting within a page.

Dynamic imports for heavy components

import dynamic from 'next/dynamic';

// Heavy chart library — don't load until needed
const HeavyChart = dynamic(() => import('@/components/HeavyChart'), {
  loading: () => <Skeleton className="h-64 w-full" />,
  ssr: false, // charts often use window/document — skip SSR
});

// Modal — no point loading it until user triggers it
const EditModal = dynamic(() => import('@/components/EditModal'));

Conditional loading based on user interaction

const [showMap, setShowMap] = useState(false);
const Map = dynamic(() => import('@/components/Map'), { ssr: false });

return (
  <>
    <button onClick={() => setShowMap(true)}>Show Map</button>
    {showMap && <Map />}
  </>
);

The map bundle (often 200KB+) never loads unless the user actually clicks the button.

Third-party scripts

Don't let analytics or chat widgets block your render:

import Script from 'next/script';

// afterInteractive: loads after page is interactive (good for analytics)
<Script src="https://analytics.example.com/script.js" strategy="afterInteractive" />

// lazyOnload: lowest priority, loads during idle time (good for chat widgets)
<Script src="https://cdn.chat-widget.com/widget.js" strategy="lazyOnload" />

🔤 3. Font Optimization

Fonts are a silent LCP killer — the browser can't render text until the font file downloads.

import { Inter, JetBrains_Mono } from 'next/font/google';

const inter = Inter({
  subsets: ['latin'],
  display: 'swap',   // show fallback font while loading, swap when ready
  preload: true,
  variable: '--font-inter', // expose as CSS variable for flexibility
});

const mono = JetBrains_Mono({
  subsets: ['latin'],
  display: 'swap',
  variable: '--font-mono',
});

export default function RootLayout({ children }) {
  return (
    <html className={`${inter.variable} ${mono.variable}`}>
      <body className="font-sans">{children}</body>
    </html>
  );
}

// tailwind.config.js
theme: {
  extend: {
    fontFamily: {
      sans: ['var(--font-inter)'],
      mono: ['var(--font-mono)'],
    },
  },
}

next/font downloads and self-hosts fonts at build time. No external requests at runtime, no layout shift, no FOUT.

For local/brand fonts:

import localFont from 'next/font/local';

const brandFont = localFont({
  src: [
    { path: './fonts/Brand-Regular.woff2', weight: '400' },
    { path: './fonts/Brand-Bold.woff2', weight: '700' },
  ],
  variable: '--font-brand',
});

🗄️ 4. Caching Strategy

Caching is where you get the most leverage with the least code.

Static Generation for public pages

// app/blog/[slug]/page.tsx
export async function generateStaticParams() {
  const posts = await getPosts();
  return posts.map((post) => ({ slug: post.slug }));
}

// Rendered at build time → served from CDN edge
export default async function BlogPost({ params }) {
  const post = await getPost(params.slug);
  return <Article post={post} />;
}

ISR for content that changes occasionally

// Rebuild this page in the background at most once per hour
export const revalidate = 3600;

// On-demand revalidation from a CMS webhook
// app/api/revalidate/route.ts
import { revalidatePath } from 'next/cache';

export async function POST(req) {
  const { path, secret } = await req.json();
  if (secret !== process.env.REVALIDATION_SECRET) {
    return Response.json({ error: 'Invalid secret' }, { status: 401 });
  }
  revalidatePath(path);
  return Response.json({ revalidated: true });
}

API Route caching

export async function GET() {
  const data = await fetchExpensiveData();

  return Response.json(data, {
    headers: {
      // CDN caches for 1 hour, serves stale while revalidating for 24h
      'Cache-Control': 'public, s-maxage=3600, stale-while-revalidate=86400',
    },
  });
}

React `cache()` for deduplication

If multiple components on the same page request the same data, cache() ensures the fetch runs only once per request:

import { cache } from 'react';

export const getUser = cache(async (id: string) => {
  return db.user.findUnique({ where: { id } });
});

// Now safe to call from Header, Sidebar, and Page — one DB query total

🗃️ 5. Database Query Optimization

Eliminate N+1 queries

// Before: 1 query for users + N queries for posts = N+1 total
const users = await db.user.findMany();
for (const user of users) {
  user.posts = await db.post.findMany({ where: { userId: user.id } });
}

// After: 1 query, JOIN handled by Prisma
const users = await db.user.findMany({
  include: { posts: true },
});

Select only what you need

// Before: fetches entire user row (including password hash, tokens, etc.)
const user = await db.user.findUnique({ where: { id } });

// After: fetch only what the component renders
const user = await db.user.findUnique({
  where: { id },
  select: { name: true, email: true, avatarUrl: true },
});

Add indexes for your most common queries

// schema.prisma
model Post {
  id        String   @id
  authorId  String
  createdAt DateTime @default(now())

  @@index([authorId])                  // speeds up posts-by-user queries
  @@index([createdAt(sort: Desc)])     // speeds up "latest posts" queries
}

Use connection pooling in serverless environments

Serverless functions (Vercel, Lambda) open a new DB connection per invocation. Without pooling, you'll hit connection limits fast:

// lib/db.ts
import { PrismaClient } from '@prisma/client';

const globalForPrisma = globalThis as unknown as { prisma: PrismaClient };

export const db =
  globalForPrisma.prisma ??
  new PrismaClient({
    log: process.env.NODE_ENV === 'development' ? ['query'] : [],
  });

if (process.env.NODE_ENV !== 'production') globalForPrisma.prisma = db;

For production at scale, use Prisma Accelerate or PgBouncer at the infrastructure level.

🔁 6. Server Components vs Client Components

This is the App Router concept that trips up most teams migrating from Pages Router.

The rule: everything is a Server Component by default. Only add "use client" when you need browser APIs, event handlers, or React state.

// ✅ Server Component — runs on server, zero JS sent to client
// app/dashboard/page.tsx
async function Dashboard() {
  const data = await db.metrics.findMany(); // direct DB access, no API layer needed
  return <MetricsGrid data={data} />;
}

// ✅ Client Component — only where interactivity is needed
// components/MetricsGrid.tsx
'use client';
import { useState } from 'react';

export function MetricsGrid({ data }) {
  const [filter, setFilter] = useState('all');
  // ...
}

A common mistake is placing "use client" high in the tree, which forces everything below it to ship as client-side JS. Push it as far down the component tree as possible — ideally only on the interactive leaf nodes.

🧹 7. Middleware Optimization

Middleware runs on every request. Keep it lean.

// middleware.ts
import { NextResponse } from 'next/server';

export function middleware(request) {
  // ❌ Don't do expensive work here
  // const data = await fetch('https://api.example.com/check'); // blocks every request

  // ✅ Fast checks only — JWT validation, redirects, header injection
  const token = request.cookies.get('token');
  if (!token) {
    return NextResponse.redirect(new URL('/login', request.url));
  }

  return NextResponse.next();
}

// ✅ Scope middleware to only the routes that need it
export const config = {
  matcher: ['/dashboard/:path*', '/api/protected/:path*'],
};

Without a matcher, middleware runs on every request including static assets — a silent but consistent performance drain.

📊 Performance Results

Metric	Before	After	Improvement
FCP	1.8s	0.4s	78% ✅
LCP	3.2s	0.8s	75% ✅
TTI	4.1s	1.2s	71% ✅
Bundle Size	450 KB	180 KB	60% ✅
DB Queries / Request	~200	1–3	98% ✅
Lighthouse Score	72	98	+26 ✅

🔑 Key Takeaways

Profile first — Lighthouse CI + real-user Web Vitals before touching code
Analyze your bundle — @next/bundle-analyzer finds hidden weight fast
Images — next/image with priority on LCP images, AVIF format preferred
Fonts — next/font eliminates layout shift and external font requests
Code Splitting — lazy load anything not needed on first paint; defer third-party scripts
Caching — static generation → ISR → stale-while-revalidate, in that order of preference
Database — eliminate N+1s, select only needed fields, use connection pooling
Server Components — default to server, push "use client" as low as possible
Middleware — always scope with matcher, never do async work inside it

Have questions about your specific setup? Drop them in the comments — happy to dig in.

Reverse Engineering India’s Electoral Roll System: Why Can’t We Have Digital Voter Data?

Black Lover — Sun, 29 Mar 2026 06:09:50 +0000

The Problem That Started It All

A few weeks ago, someone approached me with what seemed like a simple request: "Can you help convert our assembly constituency's electoral roll into an Excel sheet? We need to verify voter data digitally for our campaign."

Simple enough, right? Just download the PDFs and convert them to a spreadsheet.

Except it wasn't simple at all.

What I discovered was a maze of bureaucratic digital infrastructure that keeps India's electoral data locked in PDFs, making digital verification nearly impossible for citizens, candidates, and researchers alike.

This is the story of why India's 900+ million voter records exist in digital limbo — technically online, but practically inaccessible for any meaningful digital analysis.

The Digital Paradox: Data That Exists But Doesn't

Here's the irony: India's Election Commission (ECI) has spent millions digitizing electoral rolls. The data exists in databases. APIs serve this data in real-time for searches. Yet, when you want to actually work with this data — verify voters, analyze demographics, or cross-check registrations — you're stuck with:

Scanned PDF images (not even searchable text in many cases)
No CSV/Excel exports available
No bulk data access for researchers
Individual lookups only through a web interface with captchas

Why does this matter?

Imagine you're a candidate in an Assembly Constituency with 200,000 voters spread across 10 parts. You want to:

Verify if your supporters are registered
Check for duplicate registrations
Analyze demographic distribution
Plan booth-level strategies

Current solution: Manually look through 10 PDF files, each 200–300 pages long. No search, no filter, no sort. Just Ctrl+F if you're lucky and the PDF has text layers.

What you actually need: A spreadsheet. A database. Anything digital.

But that doesn't exist. At least, not officially.

Discovery: The Hidden Infrastructure Nobody Talks About

While trying to solve this PDF-to-Excel problem, I started reverse engineering ECI's website. What I found was surprising.

The Hidden Gateway APIs

Standard subdomain enumeration tools show you the obvious ECI subdomains: www.eci.gov.in, voters.eci.gov.in, results.eci.gov.in. But two critical subdomains were completely invisible to enumeration:

gateway-voters.eci.gov.in
gateway-officials.eci.gov.in

Why is this significant?

These gateways handle ALL the backend operations: voter searches, electoral roll generation, PDF downloads, and real-time data queries.

The data IS digital. It IS structured. It IS in databases. The APIs prove it. But there's deliberately no public interface to export this structured data.

The Complete Infrastructure Map

Active Subdomains (14 with IP Resolution):

Subdomain	IP Address	Purpose	Cloudflare
voters.eci.gov.in	2.16.10.151	Main voter portal	OFF
results.eci.gov.in	2.19.198.57	Election results	OFF
cvigil.eci.gov.in	164.100.229.90	Citizen vigilance	OFF
suvidha.eci.gov.in	164.100.85.125	Official portal	OFF
ems.eci.gov.in	164.100.229.206	Election management	OFF

Inactive/Non-Resolving Subdomains (6):

api.eci.gov.in (suggests there WAS an API portal)
dev.eci.gov.in (development environment)
resultapi.eci.gov.in (result APIs)
voterhelpline.eci.gov.in (helpline portal)

Security observation: ZERO Cloudflare protection on any subdomain. All directly exposed.

The API Architecture: Digital Data That You Can't Access

Gateway-Voters API Structure

Base URL: https://gateway-voters.eci.gov.in/api/v1/

The API is extensive and well-structured.

1. Master Data APIs (Publicly Accessible)

GET /common/states
GET /common/districts/{stateCode}
GET /common/constituencies?stateCode=S22
GET /common/acs/S2223

These work. No authentication. You can get lists of all states, districts, and constituencies. The data is RIGHT THERE.

2. Search APIs (Captcha Protected)

POST /elastic/search-by-epic-from-national-display
{
  "epicNumber": "232452452",
  "stateCd": "S22",
  "captchaData": "j692pe",
  "captchaId": "BBB1BD...",
  "securityKey": "na"
}

You can search for individual voters. One at a time. With captcha.

Notice the /elastic/ endpoint — the data lives in Elasticsearch, a search engine purpose-built for massive datasets. That means the data is already structured, indexed, and queryable for 900M+ records. Bulk exports would be trivial to implement. But bulk queries aren't allowed.

3. Electoral Roll Publishing APIs

POST /printing-publish/get-publish-part-list
{
  "acNumber": 186,
  "stateCd": "S22",
  "year": 2026,
  "rollTypeRefId": "SIR-DraftRoll"
}

This returns 7 parts for AC 186, in both English and Tamil — that's 14 PDF files you need to download manually.

4. The Mystery Parameter: `"securityKey": "na"`

Every single API call includes this, always set to "na". This suggests a legacy authentication system that was removed, or a placeholder for future security that never landed. Either way — it's just... there.

The PDF Problem: Why PDFs Are Digital Jail

Before 2025: The Individual PDF Nightmare

The old URL pattern:

https://voters.eci.gov.in/eroll/2026/s22/sir-draftroll/186/
2026-EROLLGEN-S22-186-SIR-DraftRoll-Revision1-ENG-1-WI.pdf

For Tamil Nadu's 234 Assembly Constituencies:

234 ACs × 7 parts average × 2 languages = 3,276 individual PDF files
Each PDF: 200–500 pages
Total: ~700,000 pages of voter data
Format: Often scanned images, not searchable text
Size: ~50–100 GB of data

To digitize this, you'd need to download 3,276 files (with captchas), OCR the scanned pages, extract inconsistent tables, clean and structure everything, and de-duplicate across files. Estimated time: weeks to months.

This is why digital voter verification doesn't exist.

2025: The ZIP File Revolution (Sort Of)

ECI recently added bulk download options:

Per-AC ZIP Files:

https://voters.eci.gov.in/eroll/2026/s22/sir-draftroll/185-eroll.zip

State-wide Bulk Portal:

https://voters.eci.gov.in/download-sir-draft-roll?stateCode=S22

The catch? It's still PDFs inside those ZIPs. The fundamental problem is unchanged.

Why PDFs Are The Wrong Format

PDFs are designed for printing, not data analysis:

❌ Can't sort by age, gender, or locality

❌ Can't filter duplicate names

❌ Can't programmatically verify registrations

❌ Can't merge with other datasets

❌ OCR introduces errors (especially with Indian names)

❌ No export to Excel/CSV available

What campaigns and researchers actually need:

✅ CSV/Excel with structured columns

✅ JSON from the API endpoints

✅ Direct database exports (even read-only)

✅ One-click export to Excel for digital verification

The technology exists. The databases exist. The APIs exist. But PDFs force you back to manual, paper-based verification even though the data is digital.

Breaking The Lock: Understanding the URLs

Since ECI won't provide structured data, here are the URL patterns — understanding them helps you see how the system works.

Direct Download URLs (No Code Needed)

Individual AC ZIP files:

https://voters.eci.gov.in/eroll/2026/{state_code}/sir-draftroll/{ac_number}-eroll.zip

# Examples:
https://voters.eci.gov.in/eroll/2026/s22/sir-draftroll/185-eroll.zip
https://voters.eci.gov.in/eroll/2026/s22/sir-draftroll/186-eroll.zip

State bulk download portal:

https://voters.eci.gov.in/download-sir-draft-roll?stateCode=S22

SIR Search (check name in new rolls):

https://voters.eci.gov.in/searchInSIR/{UNIQUE_ID}

The Download Challenge

Browser download: click the URL → ZIP downloads instantly ✅

Command-line download (wget/curl): gets blocked with 403 Forbidden ❌

Why? The server checks the User-Agent header. Simple browser mimicry bypasses this:

wget --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
     --referer="https://voters.eci.gov.in/" \
     "https://voters.eci.gov.in/eroll/2026/s22/sir-draftroll/185-eroll.zip"

No complex code needed. Just proper headers.

The Workflow That Actually Works

Based on my experience helping that campaign digitize their AC data:

Step 1: Identify Your URLs

# Your constituency ZIP:
https://voters.eci.gov.in/eroll/2026/{state_code}/sir-draftroll/{ac_number}-eroll.zip

# State bulk download:
https://voters.eci.gov.in/download-sir-draft-roll?stateCode={STATE_CODE}

Step 2: Download ZIPs

Manual (easiest): Open browser, navigate to the ZIP URL, click download.

Command line (faster for bulk):

wget --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)" \
     --referer="https://voters.eci.gov.in/" \
     "https://voters.eci.gov.in/eroll/2026/s22/sir-draftroll/185-eroll.zip"

Step 3: Extract PDFs

Unzip downloaded files, pick the English version for easier processing.

Step 4: PDF to Excel Conversion

Adobe Acrobat (Export to Excel)
Tabula (Free, open-source)
Online converters (smallpdf.com, ilovepdf.com)
Python with tabula-py (for automation)

Step 5: Data Cleaning

pip install requests pandas tabula-py openpyxl

Remove duplicates, fix OCR name errors, standardize formats, validate EPIC numbers.

Total time for a single AC: 2–3 days first time, a few hours after the learning curve.

What it should take: 5 minutes with a CSV download button.

The Real Question: Why Is This Data Hidden?

The data is already digital. The infrastructure proves it. So why can't citizens access structured electoral data?

Privacy Concerns (Legitimate) — Electoral rolls contain full name, age, address, EPIC number. Counter-argument: this data is already public. Anyone can visit the BLO and get printed copies. PDFs are freely downloadable. Making it Excel instead of PDF doesn't change privacy — it changes usability.

Preventing Misuse (Legitimate) — Bulk data could enable targeted misinformation or voter profiling. Counter-argument: malicious actors already have this. Professional data brokers and well-funded campaigns have digitized it. Only honest citizens, small candidates, and researchers are locked out.

Bureaucratic Inertia (Likely) — "This is how we've always done it." No technical understanding at the policy level. No incentive to change.

Political Control (Speculative) — Keeping data inaccessible favors established parties with resources to digitize, creates dependency on party databases, and limits citizen oversight.

Similar Systems Worldwide

Country	Approach
USA	Most states publish voter files in CSV format; available for purchase or free download
UK	Electoral registers available for purchase in two versions (full and open)
Australia	Electoral rolls accessible to registered political parties in digital format
India	World's largest democracy. Elasticsearch backend. Data locked in PDFs.

What Should Change: A Proposal

1. Provide Structured Data Exports
CSV/Excel downloads for each AC, updated with each revision, no captcha for bulk downloads.

2. Create Legitimate API Access
Developer portal with API keys, rate-limited but functional, documented endpoints, terms of use with penalties for misuse.

3. Maintain Privacy Balance
Remove exact addresses (keep locality), provide age brackets instead of birthdates, audit trail for access.

4. The Gateway APIs Already Exist!

The infrastructure is ALREADY THERE. The databases exist. The APIs work. Just add:

GET /api/v1/bulk/export-ac-data?acNumber=185&format=csv

That's it. One endpoint. Change digital democracy in India.

Conclusion: Data Liberation Is Democratic Rights

India prides itself on being the world's largest democracy. Yet our electoral data — the foundation of that democracy — is locked in a digital jail of PDFs.

The data exists. The technology exists. What's missing is the will to make it truly public.

To the Election Commission: You've done the hard work of digitization. Now make it truly accessible. Structured data isn't a security risk — it's a democratic necessity.

To researchers and activists: The tools and methods exist to digitize this data. It's tedious, but doable. Don't wait for permission.

To developers: Build the tools that should exist. PDF-to-Excel converters specifically for electoral rolls. Bulk downloaders. Verification APIs. Make them open source.

To candidates and campaigns: Demand better. You have the right to structured voter data for your constituency. Pressure ECI for CSV exports.

The Original Question Answered

Remember that person who asked me to digitize their AC electoral roll?

What I told them:

"It's technically possible"
"It will take 2–3 weeks"
"You'll need someone who can code"
"The data quality won't be perfect"
"It shouldn't be this hard"

What I should have been able to say:

"Download this CSV from the ECI website"
"Takes 5 minutes"
"Here's the Excel file"

That's the difference between a digital democracy and a PDF democracy.

Found this useful? Share it with candidates, researchers, and civic tech developers. Let's make electoral data actually accessible.

Have you tried digitizing voter data? What challenges did you face? Drop your experiences in the comments!

Legal note: Electoral rolls are public information under Indian law. Accessing publicly available data via public URLs is legal. Respect rate limits and don't overwhelm servers.

Python Note

Black Lover — Sun, 17 Jul 2022 05:12:01 +0000

Everything about Python

I will teach to come across to gain knowledge about Python and It's overall Things .

Source code From GitHub :-
Click To Get Code !!

First Learn How it's Work !!