DEV Community: Kishan Agarwal

How WhatsApp Works Without Internet: Offline Messaging and Sync Explained

Kishan Agarwal — Tue, 16 Jun 2026 07:01:00 +0000

Before we get overwhelmed by phrases like "eventual consistency" and "offline-first architecture," let's just talk about something you've probably done a hundred times. You're on a train through a tunnel, the signal drops to zero, and you send a message anyway. WhatsApp shows a single grey tick almost immediately. The message didn't go anywhere — there's no internet. So what just happened?

That single tick is doing a lot of quiet engineering work. And understanding it unlocks how a huge category of modern apps think about reliability, state, and user trust.

This is not as mysterious as it sounds. Let's walk through it.

The Moment You Hit Send With No Internet

Picture this: you're on a flight, airplane mode is on, and you type "landed safely, will call soon" and tap send. WhatsApp shows the message instantly in your chat — a single grey tick. The app didn't freeze. It didn't say "no network, try again." It just... worked.

The obvious question here is: where did that message actually go?

It went nowhere yet. It went into your phone's local storage — a small database sitting right on your device. WhatsApp writes your message to this local store first, marks it as "pending," and shows it to you immediately. The network hasn't been involved at all.

This is the core idea behind offline-first design: the app treats your local device as the source of truth, not the server. You write locally, you sync later.

Sounds simple. The engineering to make it reliable is not.

Your Phone Is a Mini Post Office

Think of your phone as a local post office in a small town cut off by floods. You walk in, drop a letter, and get a receipt (the grey tick). The letter sits in the post office's outbox. The moment the roads reopen, the truck leaves and delivers everything.

The post office doesn't tell you, "Sorry, roads are closed, come back later." It accepts your letter, stores it, and handles delivery when it can. That's exactly what the message queue on your device does.

Every message you send while offline enters this queue — an ordered list of pending actions your app will retry as soon as connectivity returns. The queue persists even if you close the app, because it's written to local storage (usually SQLite on mobile), not just held in memory.

What "Local Storage" Actually Means Here

WhatsApp (and apps like it) keep a local copy of your entire recent conversation history on your device. This isn't just a cache for speed — it's a deliberate design decision, so the app can work without a server.

When you open a chat, you're reading from this local database. When you type and send, you're writing to it. The server sync happens in the background, asynchronously.

The local database tracks a few critical things for each message:

The message content
A unique ID generated on your device
A timestamp from the moment you wrote it
A delivery status: pending, sent, delivered, read

That device-generated ID is more important than it looks. It's what lets the server recognize your message as unique even if it receives it multiple times (say, after a network retry). No duplicates, no confusion.

The Queue Comes Alive When You Reconnect

The moment your phone gets a signal back — WiFi, 4G, doesn't matter — the app's sync process kicks in. It reads through the pending queue, oldest first, and starts pushing messages to the server.

Now wait a minute — what if two messages were queued up and they arrive at the server out of order?

Good catch. This is why the device-generated timestamp matters. The server uses it to reconstruct the correct message order, even if network delivery got scrambled. The queue on your device sends in order; the timestamp is a fallback guarantee.

Once the server receives a message and stores it, it sends back an acknowledgement. Your app updates that message's status to sent — and the grey tick becomes a double grey tick.

The Three Ticks: What Each State Actually Means

This is where offline messaging connects to something you see every single day.

Single grey tick — Sent. Your device has written the message to the server. That's it. The recipient hasn't been involved yet. If they're offline, it just means the server is holding it for them.

Double grey tick — Delivered. The server pushed the message to the recipient's device, and their WhatsApp app acknowledged receipt. The message now lives in their local database too. The recipient may not have opened it — delivered just means their phone got it.

Double blue tick — Read: The recipient opened the conversation. Their app sent a read receipt back to the server, which relayed it to you. Blue ticks require the recipient to actively open the chat.

Each state is a separate network event. Each can be delayed independently. That's why you sometimes see a message go from single tick to double blue all at once — the recipient was offline when it was delivered, then opened it later, and both events reached you simultaneously.

Media Is a Separate Problem

A text message is a few hundred bytes. A video can be 50MB. Offline handling for media is fundamentally different.

When you send a photo while offline, the app queues an upload intent — not the photo itself. The local database notes: "When we're back online, upload this file from this local path, then send the message that references it."

When connectivity returns, the media uploads first to WhatsApp's servers. Once the upload completes and the server gives back a URL, the text message referencing that URL is sent. Only then does the recipient see the media.

Sounds like a massive headache, right? But this separation is intentional. If both the upload and the send were one atomic operation, a failed upload midway would mean starting over completely. By splitting them, the app can resume an interrupted upload from where it left off — not from scratch.

The Real Engineering Part

Conflict Resolution and Message Ordering

Here's where it gets genuinely tricky. Imagine you and a friend both go offline at the same time and both send messages to a group chat. When you both reconnect, whose messages get sent first? What order do they appear in?

The short answer is: the server decides, and it uses timestamps to do it. But timestamps from different devices are never perfectly synchronized. Two phones with a 2-second clock drift can disagree on which message came first.

Most messaging systems solve this with a technique called logical ordering — the server assigns a monotonically increasing sequence number to each message as it arrives. This becomes the "official" order, regardless of what the device clocks said. Your local UI then reorders messages to match the server's sequence.

This creates an interesting moment you may have noticed: you send a message, it shows at the bottom of your chat, then jumps slightly upward after syncing. That's message reordering happening in real time.

Eventual Consistency at the User Level

Eventual consistency is a scary-sounding term from distributed systems. In plain English, it means: not everyone will see the same state at the same moment, but eventually, everyone will agree.

WhatsApp is eventually consistent. If you send a message to a group, some members might get it in 2 seconds, others in 30 seconds. For a moment, the group chat looks different on different phones. But within some window of time — usually seconds — everyone catches up.

The tradeoff being made here is availability over perfect real-time accuracy. The app prioritizes letting you send and receive messages even under bad conditions, accepting that momentary inconsistency is less annoying than failure.

The Catch: When Offline-First Gets Complicated

The edge cases are where this gets interesting.

Message expiry is one. If your message sits in the pending queue for too long — say, you're offline for 30 days — the server may no longer accept it. The session might have expired, encryption keys may have rotated (WhatsApp uses ephemeral keys via the Signal Protocol), and the queued message simply fails silently or returns an error your app has to handle gracefully.

Another edge case: editing and deletion. When you delete a message "for everyone," that's a new queued action — a delete event with the target message ID. If you're offline when you tap delete, the delete event queues up. But if the recipient comes online before you do and reads the message before your delete event reaches the server, the delete may still succeed, but they've already seen it. Eventually consistent can mean "eventually deleted, but too late."

Clock manipulation is another real issue. If a user manually sets their phone clock backwards, device-generated timestamps can look like messages from the past. Servers defend against this with server-side timestamps as authoritative when there's a large discrepancy.

DIY: Build It Yourself

Here's a minimal offline-first message queue in TypeScript — the core pattern without the complexity of a full chat app.

import AsyncStorage from '@react-native-async-storage/async-storage';

interface QueuedMessage {
  id: string;
  content: string;
  recipientId: string;
  timestamp: number;
  status: 'pending' | 'sent' | 'failed';
}

async function queueMessage(content: string, recipientId: string): Promise<void> {
  const message: QueuedMessage = {
    id: `${Date.now()}-${Math.random().toString(36).slice(2)}`,
    content,
    recipientId,
    timestamp: Date.now(),
    status: 'pending',
  };

  const raw = await AsyncStorage.getItem('pendingMessages');
  const queue: QueuedMessage[] = raw ? JSON.parse(raw) : [];
  queue.push(message);
  await AsyncStorage.setItem('pendingMessages', JSON.stringify(queue));
}

async function flushQueue(sendToServer: (msg: QueuedMessage) => Promise<boolean>): Promise<void> {
  const raw = await AsyncStorage.getItem('pendingMessages');
  if (!raw) return;

  const queue: QueuedMessage[] = JSON.parse(raw);
  const remaining: QueuedMessage[] = [];

  for (const msg of queue) {
    const success = await sendToServer(msg);
    if (!success) {
      remaining.push({ ...msg, status: 'failed' });
    }
  }

  await AsyncStorage.setItem('pendingMessages', JSON.stringify(remaining));
}

queueMessage writes to local storage immediately and returns — no network call. flushQueue is called when connectivity is detected, iterates through pending messages oldest-first, and removes only the ones that succeed. Failed messages stay in the queue for the next retry.

Notice the ID format: a timestamp plus a random suffix. That's a poor person's globally unique ID, enough for small-scale use. Production systems use proper UUIDs or server-assigned IDs with client-side temporary IDs that get swapped on acknowledgement.

Try implementing this with a real network listener — the NetInfo API on React Native fires an event every time connectivity changes. Wire flushQueue to that event and watch your "pending" messages go live the moment you turn off airplane mode. It is one thing to read about offline queuing, but it is a whole different feeling when you watch your queued messages flush in order the instant the signal comes back.

What's Next

If you are thinking of how media is handled on these gigantic platforms, we have already discussed it in the previous blog. You can check it out here: How Instagram Stores Your Reels, Photos, and Drafts Without Losing a Frame

Happy Exploration!

React Navigation vs Expo Router: Which One Should You Actually Use?

Kishan Agarwal — Mon, 15 Jun 2026 06:44:00 +0000

Liquid syntax error: Variable '{{ title: "'Feed'," tabBarIcon: ({ color }' was not properly terminated with regexp: /\}\}/

Architecting Production React Native Apps: How Instagram, WhatsApp, and Uber Think About Scale

Kishan Agarwal — Sun, 14 Jun 2026 07:00:00 +0000

Ok, before we go into the depths of these concepts, I want to tell you that we will take it easy. I don't want you to get overwhelmed by the company names in the title. We are not cloning Instagram. We are not rebuilding WhatsApp. We are going to use them as lenses — each one teaches you something specific about how production mobile engineering actually thinks.

By the end of this, you will have a mental model for structuring React Native applications that can grow without collapsing under their own weight. Whether you are building a solo project or joining a team that already has 40 engineers, this is the thinking you need.

Let's get into it.

The Dhaba Problem

There is a dhaba near my college that made the best chole bhature I have ever eaten. One cook, one kitchen, no menu, no system. The cook just knew where everything was. It worked perfectly.

Then they opened a second location.

Everything fell apart. The second cook didn't know the recipe. There was no system to teach it. The original cook couldn't be in two places. A dhaba can survive on one person's tribal knowledge. A chain cannot.

Most React Native codebases start as dhabas. Everything in one components/ folder. Navigation in App.js. API calls are scattered across screens. State managed with useState wherever it's needed. For the first few months, it works perfectly. The original developer knows where everything is.

Then the team grows. Or the app grows. Or both. Suddenly, nobody knows where anything is, touching one screen breaks three others, and onboarding a new developer takes a week just for them to understand the folder structure.

This is the problem architecture solves. Not performance — a flat structure performs just fine. Architecture is about how many people can work on this simultaneously without stepping on each other.

Feature-Based Architecture: Thinking in Districts

The shift you need to make is from type-based to feature-based organization.

Type-based is the instinct everyone starts with:

src/
├── components/
├── screens/
├── hooks/
├── utils/
└── services/

Looks clean. Falls apart at 30 screens. Your components/ folder has 80 files. hooks/ has 40. Finding the hook for the feed screen means digging through everything unrelated to it.

Feature-based reorganizes around product domains:

src/
├── features/
│   ├── feed/
│   │   ├── components/
│   │   ├── hooks/
│   │   ├── api/
│   │   └── store/
│   ├── messaging/
│   │   ├── components/
│   │   ├── hooks/
│   │   ├── api/
│   │   └── store/
│   ├── auth/
│   └── profile/
├── shared/
│   ├── components/    ← truly reusable UI only
│   ├── hooks/
│   └── utils/
└── core/
    ├── api/           ← base API client
    ├── navigation/
    └── storage/

Everything that belongs to the feed lives inside features/feed/. A developer working on messaging never needs to open the feed folder. A developer onboarding to the team can open one feature folder and understand that entire slice of the product.

The rule is simple: if a file is only used by one feature, it lives inside that feature. If it's used by two or more features, it moves to shared/.

Expo Router at Production Scale

Expo Router's file-based routing maps cleanly onto feature-based architecture. The app/ folder is your navigation layer. The src/features/ folder is your business logic layer. They stay separate.

├── app/                          ← Navigation (Expo Router)
│   ├── _layout.tsx               ← Root: fonts, providers, theme
│   ├── +not-found.tsx
│   │
│   ├── (auth)/
│   │   ├── _layout.tsx
│   │   ├── login.tsx
│   │   ├── register.tsx
│   │   └── forgot-password.tsx
│   │
│   └── (app)/
│       ├── _layout.tsx           ← Auth guard
│       ├── (tabs)/
│       │   ├── _layout.tsx       ← Tab navigator
│       │   ├── feed.tsx
│       │   ├── explore.tsx
│       │   ├── inbox.tsx
│       │   └── profile.tsx
│       │
│       ├── post/
│       │   ├── [id].tsx
│       │   └── [id]/comments.tsx
│       │
│       ├── chat/
│       │   ├── index.tsx
│       │   └── [conversationId].tsx
│       │
│       └── settings/
│           ├── index.tsx
│           ├── account.tsx
│           └── notifications.tsx
│
└── src/                          ← Business logic (feature-based)
    ├── features/
    ├── shared/
    └── core/

Your screen files in app/ are thin shells. They import from src/features/ and render. All the logic — data fetching, state, business rules — lives in the feature folder, not in the screen file.

// app/(app)/(tabs)/feed.tsx — thin shell
import { FeedContainer } from '@/features/feed/components/FeedContainer';

export default function FeedScreen() {
  return <FeedContainer />;
}

The screen file is five lines. The complexity lives where it belongs.

Authentication Architecture

Auth is the first place most apps get complicated. And the first place most developers cut corners, and they regret it later.

The architecture decision that matters is: where does auth state live, and who is allowed to read it?

// core/auth/AuthProvider.tsx
import { createContext, useContext, useEffect, useState } from 'react';
import * as SecureStore from 'expo-secure-store';
import { authApi } from './authApi';

type AuthState = {
  user: User | null;
  token: string | null;
  isLoading: boolean;
};

const AuthContext = createContext<AuthState & {
  login: (credentials: Credentials) => Promise<void>;
  logout: () => Promise<void>;
}>(null!);

export function AuthProvider({ children }: { children: React.ReactNode }) {
  const [state, setState] = useState<AuthState>({
    user: null,
    token: null,
    isLoading: true,
  });

  useEffect(() => {
    // On mount, check if a token exists in secure storage
    async function loadToken() {
      const token = await SecureStore.getItemAsync('auth_token');
      if (token) {
        const user = await authApi.getMe(token);
        setState({ user, token, isLoading: false });
      } else {
        setState(s => ({ ...s, isLoading: false }));
      }
    }
    loadToken();
  }, []);

  async function login(credentials: Credentials) {
    const { user, token } = await authApi.login(credentials);
    await SecureStore.setItemAsync('auth_token', token);
    setState({ user, token, isLoading: false });
  }

  async function logout() {
    await SecureStore.deleteItemAsync('auth_token');
    setState({ user: null, token: null, isLoading: false });
  }

  return (
    <AuthContext.Provider value={{ ...state, login, logout }}>
      {children}
    </AuthContext.Provider>
  );
}

export const useAuth = () => useContext(AuthContext);

Notice: the token lives in SecureStore, not AsyncStorage. Sensitive credentials belong in the device keychain, not in a plain key-value store.

The layout-level guard in Expo Router reads from this context:

// app/(app)/_layout.tsx
import { Redirect, Stack } from 'expo-router';
import { useAuth } from '@/core/auth/AuthProvider';

export default function AppLayout() {
  const { user, isLoading } = useAuth();

  if (isLoading) return <SplashScreen />;
  if (!user) return <Redirect href="/login" />;

  return <Stack />;
}

One guard, covering every screen under (app)/. Add a new protected screen — create the file, and it's automatically guarded.

State Management at Scale

This is the question every growing React Native team debates. Redux? Zustand? Jotai? Context? Just React Query?

The honest answer: it depends on the type of state you are managing. There are three distinct categories, and they need different tools.

Server state — data that lives on a server and needs to be fetched, cached, synchronized, and invalidated. This is most of your app's data: user profiles, feed posts, and messages. Use TanStack Query (React Query). It handles caching, background refetching, optimistic updates, and loading/error states. This alone replaces 60% of what most apps use Redux for.
Client state — UI state that doesn't come from the server and doesn't need to persist: which tab is active, whether a modal is open, and the current theme. Use Zustand for the shared client state, useState for local. Zustand is 1KB, has no boilerplate, and works without a Provider wrapper.
Persistent state — data that needs to survive app restarts: user preferences, auth tokens, offline cache, draft messages. Use MMKV for fast synchronous key-value storage, SQLite (via expo-sqlite) for structured data like messages and posts.
```
// A Zustand store for UI state — no boilerplate, no actions, no reducers
import { create } from 'zustand';
```



type UIStore = {

  activeTab: string;

  isComposerOpen: boolean;

  setActiveTab: (tab: string) => void;

  toggleComposer: () => void;

};

export const useUIStore = create<UIStore>((set) => ({ activeTab: 'feed', isComposerOpen: false, setActiveTab: (tab) => set({ activeTab: tab }), toggleComposer: () => set((s) => ({ isComposerOpen: !s.isComposerOpen })), }));
No actions. No reducers. No dispatch. Just a store and setters.

The rule for choosing:

If TanStack Query can handle it, use TanStack Query.
If it's UI state, use Zustand.
If it needs to persist, use MMKV or SQLite.
Redux is the right choice when you need complex state machines, time-travel debugging, or your team already has deep Redux expertise.

Otherwise, the simpler stack wins.

The API Layer: One Client, Not Many

A mistake almost every app makes early on: API calls directly inside components. fetch('https://api.yourapp.com/users') scattered across 30 different files.

The problem hits when your base URL changes, you add auth headers, you switch from REST to GraphQL, or you need to add request logging. You make the change in 30 places. You miss one.

The API layer is a single module that owns all network communication.

// core/api/client.ts
import axios from 'axios';
import { getToken } from '@/core/auth/tokenStorage';

export const apiClient = axios.create({
  baseURL: process.env.EXPO_PUBLIC_API_URL,
  timeout: 10000,
});

// Attach auth token to every request automatically
apiClient.interceptors.request.use(async (config) => {
  const token = await getToken();
  if (token) {
    config.headers.Authorization = `Bearer ${token}`;
  }
  return config;
});

// Handle 401s globally — redirect to login
apiClient.interceptors.response.use(
  (response) => response,
  async (error) => {
    if (error.response?.status === 401) {
      await clearToken();
      router.replace('/login');
    }
    return Promise.reject(error);
  }
);

Every feature's API module uses this client, not fetch directly.

// features/feed/api/feedApi.ts
import { apiClient } from '@/core/api/client';
import type { Post } from '../types';

export const feedApi = {
  getPosts: async (page: number): Promise<Post[]> => {
    const { data } = await apiClient.get('/posts', { params: { page } });
    return data;
  },

  likePost: async (postId: string): Promise<void> => {
    await apiClient.post(`/posts/${postId}/like`);
  },
};

Change the base URL once. Add a header once. Log every request once. The features don't care about any of it.

Realtime Systems: Chat, Live Updates, and Ride Tracking

This is where architecture gets genuinely interesting. Real-time systems are a different class of problem from standard request-response API calls.

Chat (WhatsApp mental model)

WhatsApp's core problem is not sending messages — that's a POST request. The hard problem is receiving messages when the app is in the background, displaying them in order, handling failed sends, and syncing across devices.

The real-time transport for chat is WebSockets. A persistent connection between the client and server that lets the server push messages to the client without the client asking.

// features/messaging/hooks/useMessageSocket.ts
import { useEffect, useRef } from 'react';
import { useQueryClient } from '@tanstack/react-query';

export function useMessageSocket(conversationId: string) {
  const ws = useRef<WebSocket | null>(null);
  const queryClient = useQueryClient();

  useEffect(() => {
    ws.current = new WebSocket(
      `wss://api.yourapp.com/ws/conversations/${conversationId}`
    );

    ws.current.onmessage = (event) => {
      const message = JSON.parse(event.data);

      // Push the incoming message into TanStack Query's cache
      // The UI updates automatically — no manual setState
      queryClient.setQueryData(
        ['messages', conversationId],
        (old: Message[] = []) => [...old, message]
      );
    };

    ws.current.onclose = () => {
      // Reconnection logic here
    };

    return () => ws.current?.close();
  }, [conversationId]);
}

The incoming message lands directly in TanStack Query's cache. Every component subscribed to ['messages', conversationId] re-renders automatically. No event emitters, no global state, no manual dispatch.

Live Location (Uber mental model)

Uber's driver location problem is different from chat. Messages can tolerate a two-second delay. A driver's location on a map cannot, the user expects it to move smoothly in near real-time.

The transport for this is WebSockets or Server-Sent Events, with the client receiving location updates at a high frequency and the map component interpolating between positions for smooth animation.

// features/ride/hooks/useDriverLocation.ts
import { useEffect, useState } from 'react';

type LatLng = { latitude: number; longitude: number };

export function useDriverLocation(rideId: string) {
  const [location, setLocation] = useState<LatLng | null>(null);

  useEffect(() => {
    const ws = new WebSocket(`wss://api.yourapp.com/ws/rides/${rideId}/location`);

    ws.onmessage = (event) => {
      const { latitude, longitude } = JSON.parse(event.data);
      setLocation({ latitude, longitude });
    };

    return () => ws.close();
  }, [rideId]);

  return location;
}

The map component takes this location and animates the driver marker. The WebSocket delivers new coordinates every 1–2 seconds. The map library handles the smooth animation between points.

Live Content Updates (Instagram/Netflix mental model)

Instagram's feed updates when someone you follow posts. Netflix showing a download's progress. These don't need WebSockets; the urgency is lower, and maintaining a persistent connection for every user is expensive.

Server-Sent Events (SSE) work here. A one-directional push channel from server to client, lighter than WebSockets, reconnects automatically.

For lower-frequency updates, polling with TanStack Query is often the right answer. Refetch the feed every 30 seconds. Simple, predictable, no persistent connection cost.

// Polling with TanStack Query — Instagram "new posts" pattern
const { data: newPosts } = useQuery({
  queryKey: ['feed', 'new'],
  queryFn: feedApi.getNewPosts,
  refetchInterval: 30_000,       // Every 30 seconds
  refetchIntervalInBackground: false, // Pause when app is backgrounded
});

Sounds hard, right? TanStack Query handles all of it in two lines.

Offline-First Support

You might be thinking, "My app uses an API. If there's no internet, there's no data. What is there to cache?"

A lot. And this is the difference between an app that feels native and one that feels like a web page in a frame.

Offline-first means the app renders content from local cache immediately on launch, syncs with the server in the background, and queues writes when there's no connection to replay when connectivity returns.

Reading offline: TanStack Query caches responses in memory. For persistence across app restarts, combine it with MMKV as the persistence layer.

// core/api/queryClient.ts
import { QueryClient } from '@tanstack/react-query';
import { createSyncStoragePersister } from '@tanstack/query-sync-storage-persister';
import { MMKV } from 'react-native-mmkv';

const storage = new MMKV();

const mmkvStorageAdapter = {
  getItem: (key: string) => storage.getString(key) ?? null,
  setItem: (key: string, value: string) => storage.set(key, value),
  removeItem: (key: string) => storage.delete(key),
};

export const persister = createSyncStoragePersister({
  storage: mmkvStorageAdapter,
});

export const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      gcTime: 1000 * 60 * 60 * 24, // Cache for 24 hours
    },
  },
});

Writing offline: Queue mutations locally when offline, replay them when the connection returns.

// Optimistic update with offline queue
const { mutate: likePost } = useMutation({
  mutationFn: feedApi.likePost,
  onMutate: async (postId) => {
    // Update the cache immediately — the UI feels instant
    await queryClient.cancelQueries({ queryKey: ['posts'] });
    queryClient.setQueryData(['posts'], (old: Post[]) =>
      old.map(p => p.id === postId ? { ...p, liked: true, likes: p.likes + 1 } : p)
    );
  },
  onError: (err, postId, context) => {
    // If it fails, roll back
    queryClient.setQueryData(['posts'], context.previousPosts);
  },
});

The user taps like. The UI updates instantly. The network request happens in the background. If it fails, the UI rolls back (Optimistic Updates). If the user is offline, TanStack Query's offline mutation queuing holds the request until connectivity returns.

App Startup Optimization

The two numbers that matter most for a startup: Time to Interactive (TTI) and perceived launch time, like LightHouse WebAnalytics for web

TTI is how long before the user can actually use the app. Perceived launch time is how quickly it feels like something is happening, which you can improve even without reducing actual TTI.

The strategies:

1. Defer heavy initialization. Don't initialize analytics, crash reporting, push notification handlers, and feature flag SDKs synchronously on startup. Defer them.

// app/_layout.tsx
export default function RootLayout() {
  useEffect(() => {
    // These run after the first frame is painted, not before
    initAnalytics();
    initCrashReporting();
    registerPushHandlers();
  }, []);

  return <Stack />;
}

2. Prefetch critical data. Before the user navigates anywhere, start fetching the data they will almost certainly need.

// Prefetch feed data while auth is being verified
useEffect(() => {
  queryClient.prefetchQuery({
    queryKey: ['feed'],
    queryFn: feedApi.getPosts,
  });
}, []);

3. Use a native splash screen until fonts and critical assets are loaded, not until all data is fetched. Users accept a splash screen for 500ms. They don't accept a blank white screen for 2 seconds.

import * as SplashScreen from 'expo-splash-screen';
import { useFonts } from 'expo-font';

SplashScreen.preventAutoHideAsync();

export default function RootLayout() {
  const [fontsLoaded] = useFonts({ 'Inter': require('./assets/Inter.ttf') });

  useEffect(() => {
    if (fontsLoaded) SplashScreen.hideAsync();
  }, [fontsLoaded]);

  if (!fontsLoaded) return null;

  return <Stack />;
}

4. Lazy load heavy screens. React Native's Metro bundler doesn't code-split by default, but you can manually lazy-load screens that users rarely visit.

The Real Engineering Part

Let me share a very interesting problem my friend faced, where all of this stopped being theory and became urgent. He was working on a consumer app that had grown from 5 screens to 38 over 14 months. The original architecture was type-based — one big components/ folder, API calls in screens, state everywhere. Everything in App.js navigated to everything else via React Navigation string references.

At 38 screens and 4 developers, the codebase became genuinely dangerous. Changing the profile screen broke the settings screen because they shared a component that had grown side-effect dependencies on profile-specific state. Nobody knew. There were no boundaries.

The refactor took three weeks and touched nearly every file. The lesson is not "refactor early", that's too vague. The lesson is: the cost of architecture is paid once, the cost of no architecture is paid forever.

The specific decisions that matter at scale, and when they matter:

Feature isolation — matters from the first day a second developer joins. Two people in the same folder guarantee merge conflicts.
Typed API client — matters from the first time you add auth headers, change an endpoint, or need to mock the API in tests.
Server state separation — matters from the first screen that shows data from two different API endpoints. Without TanStack Query, you write loading/error/cache logic by hand, every time.
SQLite for structured offline data — matters when your app has more than a few thousand cacheable records. MMKV is fast, but it stores everything as strings. SQLite gives you queries, indexes, and relational structure.
WebSockets for real-time — matters the moment your app needs any data to update without user action. Polling works up to a point. WebSockets scale better and feel more alive.

The Catch: When Architecture Becomes the Product

You might be thinking, "This is a lot of setup before writing a single feature."

It is. And there is a real trap here.

Over-architected apps fail just as often as under-architected ones. The difference is who fails: under-architected apps fail when the team tries to scale. Over-architected apps fail when the team is too slow to ship anything before the runway runs out.

A solo developer building an MVP does not need feature-based architecture, a typed API client, offline sync, and a WebSocket layer. They need to ship. A flat structure, useState everywhere, and direct fetch calls is the right architecture for week one.

The question to ask is: What is the next forcing function?

Second developer joins → add feature boundaries
Auth gets complex → extract auth layer
API changes break multiple screens → add API client abstraction
Users complain about loading times → add TanStack Query
Need chat → add WebSockets

Architecture should lag slightly behind complexity, not anticipate it by six months. Building for Instagram's scale when you have 200 users is not engineering, it's procrastination with better vocabulary.

DIY: A Production-Grade Starting Point

Here is a starter structure you can clone for your next serious React Native project. Not a toy. Not a tutorial app. Something you can build a real product on.

my-app/
│
├── app/                          ← Expo Router (navigation only)
│   ├── _layout.tsx
│   ├── +not-found.tsx
│   ├── (auth)/
│   │   ├── _layout.tsx
│   │   ├── login.tsx
│   │   └── register.tsx
│   └── (app)/
│       ├── _layout.tsx           ← Auth guard here
│       ├── (tabs)/
│       │   ├── _layout.tsx
│       │   ├── index.tsx         ← feed
│       │   ├── explore.tsx
│       │   └── inbox.tsx
│       └── [feature]/
│           └── [id].tsx
│
├── src/
│   ├── features/                 ← Business logic, one folder per domain
│   │   ├── feed/
│   │   │   ├── components/
│   │   │   ├── hooks/
│   │   │   │   ├── useFeed.ts
│   │   │   │   └── useLikePost.ts
│   │   │   ├── api/
│   │   │   │   └── feedApi.ts
│   │   │   └── types.ts
│   │   ├── messaging/
│   │   │   ├── components/
│   │   │   ├── hooks/
│   │   │   │   ├── useConversations.ts
│   │   │   │   └── useMessageSocket.ts
│   │   │   ├── api/
│   │   │   └── types.ts
│   │   └── auth/
│   │       ├── AuthProvider.tsx
│   │       ├── authApi.ts
│   │       └── tokenStorage.ts
│   │
│   ├── shared/                   ← Used by 2+ features
│   │   ├── components/
│   │   │   ├── Button.tsx
│   │   │   ├── Avatar.tsx
│   │   │   └── EmptyState.tsx
│   │   ├── hooks/
│   │   │   ├── useNetworkStatus.ts
│   │   │   └── useDebounce.ts
│   │   └── utils/
│   │       ├── formatDate.ts
│   │       └── formatCurrency.ts
│   │
│   └── core/                     ← Infrastructure, not business logic
│       ├── api/
│       │   ├── client.ts         ← Axios instance + interceptors
│       │   └── queryClient.ts    ← TanStack Query client + MMKV persister
│       ├── storage/
│       │   └── mmkv.ts
│       └── config/
│           └── env.ts            ← process.env wrappers with types
│
├── assets/
├── constants/
│   └── theme.ts
└── app.json

The rule that keeps this clean: app/ imports from src/features/. src/features/ imports from src/shared/ and src/core/. Nothing imports from app/. One direction. No circles.

Try setting this up for your next project. It feels like overhead on day one. By week four, when you add a new feature without touching anything outside its folder, you will understand why it exists.

Happy Exploration!

React's Virtual DOM: How React Decides What (and What Not) to Redraw

Kishan Agarwal — Sat, 13 Jun 2026 07:00:00 +0000

Ok, before we go into the depths of these concepts, I want to tell you that we will take it easy. I don't want you to get overwhelmed by the jargon. Virtual DOM, reconciliation, diffing, these words sound like they belong in a research paper, not a blog post.

Here is the thing. Every React developer has heard "React is fast because of the Virtual DOM." And every React developer has nodded along. But if you ask them why the Virtual DOM makes it fast, most of them go quiet.

Let's fix that.

The Problem: Why Touching the DOM Directly Hurts

Think about the last time you had to repaint an entire wall just to fix one scratch near the skirting board.

That's what browsers do when you manipulate the DOM without strategy. The browser doesn't just change the one element you touched; it recalculates styles, reflows the layout, and repaints everything that might have been affected. One small change, full repaint.

Now, imagine a user clicks "like" on one post in a feed of 500. Without any optimization, the browser re-renders the whole feed. Sounds like a massive headache, right?

This is the exact problem the Virtual DOM was built to solve.

The Real DOM: Why It's So Expensive

The DOM (Document Object Model) is a tree-like representation of your HTML that browsers maintain internally. Every element is a live node with styles, event listeners, and layout information all attached to it.

Let me explain it in simple words.

Every direct DOM operation triggers a chain reaction inside the browser. document.getElementById('title').innerText = 'Hello' sounds innocent. But the browser recalculates styles, re-runs layout for anything affected, and queues a repaint. It is not broken, it is just expensive by design.

For a static webpage, this is fine. For an app updating 30 times a second, it becomes a serious bottleneck.

The Virtual DOM: The Architect's Blueprint

You might be thinking, "Why not just be more careful about which DOM updates you trigger? Why add an entire extra layer?"

Fair question. For a small app, maybe you could manage this manually. But the moment you have dozens of components all reacting to user input, server responses, timers, and route changes, tracking what changed and what didn't becomes a full-time job on its own.

The Virtual DOM hands that job to React.

Think of an architect before a renovation. They don't tear down walls to figure out what to change. They work on a blueprint first, a lightweight paper representation of the building. Once all the changes are clear on paper, they hand over a precise list to the construction crew. The crew does only what's on that list. Not the whole building.

The Virtual DOM is React's blueprint. It is a plain JavaScript object that mirrors the structure of your UI, but with none of the expensive browser machinery attached.

// A simplified Virtual DOM node looks something like this
{
  type: 'div',
  props: { className: 'post-card', id: 'post-42' },
  children: [
    {
      type: 'h2',
      props: {},
      children: ['React Virtual DOM Explained']
    },
    {
      type: 'span',
      props: { className: 'likes' },
      children: ['128 likes']
    }
  ]
}

This is a plain JavaScript object. Creating and comparing these costs almost nothing, no layout engine, no browser API calls, no repaint.

The Initial Render

When your React app loads for the first time, React runs your component functions, converts the returned JSX into a Virtual DOM tree, and uses that tree to build and paint the actual Real DOM nodes.

The first render is always a full build. There's no way around it, you're starting from nothing. The magic kicks in when something changes.

State or Props Change: Drawing a New Blueprint

When a user clicks a button, submits a form, or new API data arrives, React state or props update.

React does not immediately reach for the Real DOM. Instead, it re-runs the affected component functions and builds a brand new Virtual DOM tree, a fresh blueprint of what the UI should now look like.

You now have two blueprints side by side: the old one (what's on screen) and the new one (what it should look like after the change).

Sounds familiar, right? This is where diffing comes in.

Diffing and Reconciliation: The Spot-the-Difference Game

Reconciliation is the process by which React compares the old Virtual DOM tree with the new one to determine the minimal set of changes required to update the Real DOM.

Let me explain it in simple words.

You've played "spot the difference" puzzles, two nearly identical images, five things changed. React plays this game but with JavaScript tree structures.

It walks through both trees simultaneously, node by node. Where nothing changed, it does nothing. Where something changed, it records the operation: update this text, swap this class, remove this node entirely. At the end, React holds a precise, minimal patch.

React does this using a few rules:

Elements of different types (a <div> becoming a <section>) get torn down and rebuilt from scratch.
Elements of the same type get their props compared; only the changed props get updated.
Lists use key props to track which items were added, removed, or moved. This is exactly why React warns you when you forget to add keys.

Insane, right? You write setLikeCount(c => c + 1) and React plays spot-the-difference on your entire component tree to figure out the smallest possible set of Real DOM operations to run.

Applying the Diff

Once React has the patch, it enters the commit phase, the only moment it actually touches the Real DOM.

It applies the diff in order, synchronously. Insert this node. Update that attribute. Remove this one. Instead of re-rendering your 500-post feed, React updates the one <span> whose like count changed. The browser recalculates layout for that tiny corner of the screen, not the whole page.

That's why React apps feel snappy even as they grow complex.

The Real Engineering Part

Here is the full render-to-commit lifecycle, without the hand-waving.

Trigger → Render Phase → Commit Phase

Trigger

A state or props change schedules a re-render. React doesn't process it immediately; it batches multiple updates from the same event handler into a single pass.

// Both updates collapse into ONE re-render, ONE diff, ONE commit
const handleLike = () => {
  setLikeCount(c => c + 1);
  setHasLiked(true);
};

Two state updates. One re-render. React collects them, rebuilds the Virtual DOM once, diffs once, commits once.

Render Phase

React calls the component functions. They return JSX, which becomes new Virtual DOM nodes. React builds the new tree, then diffs it against the previous one.

This phase is pure, no Real DOM contact. If React needs to interrupt this phase (in Concurrent Mode), it can because no real side effects have happened yet.

Commit Phase

React applies the diff to the Real DOM synchronously, in order. After the DOM is updated, it fires your useEffect hooks.

useEffect(() => {
  // Runs AFTER React has committed all changes to the Real DOM
  document.title = `${likeCount} people liked this`;
}, [likeCount]);

useEffect fires after commit, not during. This guarantees the DOM is fully updated before your effect runs.

The three phases: Render → Diff → Commit repeat on every state or props change. Understanding this loop is what separates React developers who write fast apps from those who keep wondering why their app lags.

The Catch: When the Virtual DOM Is Not Enough

You might be thinking, "If the Virtual DOM handles all this automatically, why do React apps still go slow sometimes?"

Because the Virtual DOM optimizes Real DOM writes. It doesn't optimize component execution.

If a parent component re-renders, all its children re-render by default even if their props didn't change. React diffs those children, confirms nothing changed, skips the DOM update. Correct behavior. But you still paid the cost of running every child function and building every child's Virtual DOM nodes.

In a small app, this is invisible. In a large one, it destroys your frame rate.

Let me share a very interesting problem I faced where this bit me hard. I was building a theme editor, the kind where users pick brand colors and see a live preview of their UI update in real time. Dragging the hue slider on the color picker was visibly janky. The preview felt like it was running at 10fps.

I dropped React Scan into the project. If you haven't used it, React Scan overlays a colored flash on every component that re-renders, you literally watch your UI light up in real time. The moment I moved the hue slider, the entire page lit up. The color picker was sitting inside a parent that held the theme state. Every onColorChange fired a setState on that parent. That parent re-rendered. Every child of that parent, the font selector, the spacing controls, the preview card, the export button, all of them re-rendered on every single mousemove event.

The Virtual DOM was correctly diffing all of them and making zero DOM changes for most of them. But the render phase was still executing every single one of those component functions, 60 times a second, as the user dragged.

The fix was two things: React.memo on every sibling component that didn't consume the color state, and moving the color state down into a smaller subtree closer to where it was actually used. React Scan went quiet. The slider felt instant.

The Virtual DOM wasn't the problem. The unchecked render phase was. React.memo, useMemo, and useCallback are the tools you reach for when the render phase is your bottleneck, and React Scan is how you find out you have the problem in the first place.

DIY: Build It Yourself (Minimal Reconciler)

You don't need to build a full React reconciler for this to click. Three small functions are enough.

1. A Minimal Virtual DOM Node

// vdom.js
function createElement(type, props = {}, ...children) {
  return { type, props, children: children.flat() };
}

const vNode = createElement(
  'div',
  { id: 'post-card' },
  createElement('h2', {}, 'React Virtual DOM'),
  createElement('span', { className: 'likes' }, '128 likes')
);

console.log(JSON.stringify(vNode, null, 2));

Plain JavaScript object. No browser API. No paint. Creating a tree like this is essentially free.

2. A Minimal Differ

// differ.js
function diff(oldNode, newNode) {
  if (!newNode) return { type: 'REMOVE' };
  if (!oldNode) return { type: 'ADD', newNode };
  if (typeof oldNode !== typeof newNode) return { type: 'REPLACE', newNode };
  if (typeof oldNode === 'string') {
    return oldNode !== newNode ? { type: 'TEXT_UPDATE', newNode } : { type: 'NONE' };
  }
  if (oldNode.type !== newNode.type) return { type: 'REPLACE', newNode };

  const propChanges = {};
  const allProps = new Set([
    ...Object.keys(oldNode.props || {}),
    ...Object.keys(newNode.props || {})
  ]);

  for (const key of allProps) {
    if ((oldNode.props || {})[key] !== (newNode.props || {})[key]) {
      propChanges[key] = (newNode.props || {})[key];
    }
  }

  return {
    type: 'UPDATE',
    propChanges,
    childDiffs: diffChildren(oldNode.children || [], newNode.children || [])
  };
}

function diffChildren(oldChildren, newChildren) {
  const length = Math.max(oldChildren.length, newChildren.length);
  const diffs = [];
  for (let i = 0; i < length; i++) {
    diffs.push(diff(oldChildren[i], newChildren[i]));
  }
  return diffs;
}

Call diff(oldTree, newTree) and inspect the output. You'll see NONE where nothing changed, TEXT_UPDATE where text changed, REPLACE where a node type flipped. That object is the patch only what needs work.

3. Apply the Diff to the Real DOM

// commit.js
function applyDiff(domNode, patch) {
  if (patch.type === 'NONE') return;

  if (patch.type === 'REMOVE') {
    domNode.parentNode.removeChild(domNode);
    return;
  }

  if (patch.type === 'ADD') {
    domNode.parentNode.appendChild(createDomNode(patch.newNode));
    return;
  }

  if (patch.type === 'REPLACE') {
    domNode.parentNode.replaceChild(createDomNode(patch.newNode), domNode);
    return;
  }

  if (patch.type === 'TEXT_UPDATE') {
    domNode.textContent = patch.newNode;
    return;
  }

  // UPDATE — apply only the changed props
  for (const [key, value] of Object.entries(patch.propChanges)) {
    if (value === undefined) {
      domNode.removeAttribute(key);
    } else {
      domNode.setAttribute(key, value);
    }
  }

  patch.childDiffs.forEach((childPatch, i) => {
    applyDiff(domNode.childNodes[i], childPatch);
  });
}

The commit function does exactly what the diff says, nothing more. A node with no changes? Skipped. One changed attribute? One setAttribute call.

Try implementing this! It is one thing to read about reconciliation, but it is a whole different feeling when you build two trees by hand, run diff(), and watch it produce a minimal, precise patch for exactly the nodes that changed.

Happy Exploration!

Authentication Demystified: Every Login Method, Actually Explained

Kishan Agarwal — Fri, 12 Jun 2026 06:31:00 +0000

Before we get overwhelmed by the jargon, let's talk about the actual problem we are solving. At some point, every app needs to answer one question: is this person actually who they say they are? That is it. Authentication is just the collection of strategies engineers have invented to answer that reliably.

Think of it the way you would in the real world. You lock your house because you do not want intruders accessing your stuff. Authentication is exactly that, but for your digital resources.

Every method needs two ingredients: a unique identifier that picks you out from everyone else (a username, an email address, a phone number, a device) and a secret that proves that identifier genuinely belongs to you. Every method in this post is just a different answer to what those two ingredients should be.

Let's go through all of them.

1. Basic Authentication: Username and Password

The classic. You are given a unique username, and you set a password for it. The server saves that password, and when you come back to log in, the server verifies what you entered against what it has stored for your username. Alternatively, and more commonly these days, your email serves as the unique identifier instead of a username. Same idea, same flow.

Advantages: Easy to implement and set up. You can roll out your own auth for this without too much trouble, and you own the whole flow.

Disadvantages: It is too basic and can be bypassed relatively easily. The entire burden of proof is on the user. They are responsible for picking a strong password, not reusing it elsewhere, and not handing it over to a phishing page. Most people do at least one of those things wrong.

2. Basic Authentication with Two-Step Verification

This is a step up from the above. Here we verify the user twice before confirming what they are claiming to be.

Email and password with a verification link: The user enters their email and password. Before we grant access, we send them a verification link. The user has to click that link before we let them through to their resources.
Email and password with an OTP on email: The same flow as above, just instead of sending a verification link, we send a time-limited OTP. The user has to enter it before it expires.
Phone number with password and OTP via SMS: Same flow as all the above, but this time the unique identifier is the phone number, and the OTP comes to the user's SMS inbox, verifying that they actually have access to that phone number.

Advantages: Still relatively easy to implement on your own. The user now needs to have physical access to the identity resource (their inbox or their phone) to verify themselves.

Disadvantages: Auth can be bypassed using burner emails or prepaid phone numbers, and this can be misused to access your resources without real accountability.

3. OIDC, or most famously known as OAuth: Let Someone Credible Vouch for You

OpenID Connect (OIDC) is an authentication protocol built on top of the OAuth 2.0 framework. It allows applications to securely verify a user's identity and retrieve basic profile information (like name and email) without forcing the user to manage and share passwords

Let me explain it in simple words. Here we use social providers to vouch for your identity.

Think of it like applying for a bank loan. The bank does not know you personally, so they ask for a guarantor: someone credible who can vouch that you will pay back what you owe. The guarantor's word stands in for the bank's direct knowledge of you. Critically, the guarantor has to be someone the bank already trusts.

If you have ever clicked "Sign in with Google," "Continue with Apple," or "Allow this app to access your GitHub," you have used OAuth.

Here is how the whole thing actually works:

First, you need credible guarantors. In our case, these are apps that have agreed to act as guarantors for the people coming to your app. Some famous examples are Google, Facebook, GitHub, Apple, and Microsoft. You go to these providers and ask them to be a guarantor for your users. For this, you register your app's URLs with the provider, and in return the provider gives you a client ID and a client secret. This is essentially authentication for your app itself, so that the provider can verify that the service redirecting users to its login page is actually yours and not someone impersonating you.

Now, when a user comes to your app and clicks "Sign in with Google," a process begins. Your app reaches out to the provider and says: "Someone wants to log in. Are you willing to vouch for them?" The provider then checks if it knows this person. If yes, it shows the user a screen that says: "I will act as a guarantor for you with this app. Would you like to continue?" The user clicks continue.

Under the hood, two things happen at this point. The provider sends a challenge, which the user completes at a redirect URI you defined during registration. If that is successful, the provider calls the /callback route you set up in your app and sends back a token. You use that token to fetch the user's details (name, email, profile picture), and then you save whatever you need and issue your own access tokens or sessions based on your auth strategy from there.

Advantages: No password to remember for every site. The guarantor handles identity verification. Also, it is easy for the user to try, since they already have an account with these providers.

Disadvantages: If the user loses access to their guarantor (locked out of Google, for instance), they lose access to everything they signed into through that provider. Your app knows users only through the intermediary, like a friend-of-a-friend situation, and cannot reach them directly if the connection breaks.

4. Passwordless Authentication: Magic Links

This one is for those who want to really test a user's patience. Personally, I get irritated every time I encounter this flow.

The idea is to remove passwords entirely. Every time a user wants to log in, you send them a code or a link to their registered email or phone. A link with the code embedded is more common, since it means the user does not have to manually type anything. They just click. Whoever opens that link is authenticated as that user.

The most important thing to understand here: if your magic link leaks, your account is compromised. The entire security model rests on that link staying private.

Advantages: The user has to have access to their identifier at all times. The moment they lose access to their inbox or phone is the moment they cannot log in, which means you know the person logging in actually controls that resource. Burner emails are much harder to exploit here because the attacker would also need access to the inbox, not just the address.

Disadvantages: User friction. They have to open their email, find the link, and click it every single time they want to log in. If you log in frequently, this gets old fast.

5. Passkeys: Your Device Is the Proof

Passkeys are the absolute best, if you ask me. The idea is that you yourself, or your device, serves as the identity. Think about how, on the internet, your identity is your IP address or your domain name. Passkeys do something similar: your device or your biometrics act as a unique identifier for verifying who you are.

How this works is genuinely fascinating if you understand even a little about encryption. So let us deviate from the main topic for a moment, because it is worth it.

Encryption, in simple terms, is hiding information in such a way that it cannot be read or decoded easily. A famous and very simple example is ROT13, where you rotate each character in the alphabet by 13 positions, a well-known problem in programming circles as well. That is symmetric encryption at its simplest: the same transformation both encodes and decodes.

Now, encryption is broadly of two types: symmetric and asymmetric.

Symmetric encryption is easy to picture. Think of a lock and a key. You have one key, and it can both lock and unlock the lock. Same key, both directions.

Asymmetric encryption changes this. You have a separate key to lock and a separate key to unlock. These are called the private key and the public key. As the names suggest, your private key should stay completely private. Never share it with anyone. Your public key can be shared freely with the world. This separation ensures that even if someone intercepts the public key in transit, they cannot do anything with it. They can only lock things with it. They cannot open what was locked.

Now let's come back to authentication.

In passkeys, your device acts as your private key. When you register a passkey with a service, your device generates a public/private key pair. The public key is sent to the server and stored there. The private key stays locked inside your device, stored in the Keychain on iOS and macOS, the Keystore on Android, or the TPM chip on Windows.

When you come back to log in, the server sends you a challenge: a piece of random data. You sign that challenge using your private key (unlocked by your biometric, either fingerprint or Face ID). The signed response goes back to the server, and the server verifies it using the public key it stored during registration. If the signature matches, your identity is confirmed.

No password. Nothing that can be phished or stolen in transit. Secure as long as you do not lose your device or compromise your biometrics.

Advantages: Highly secure. Extremely convenient for the user. One touch and they are in.

Disadvantages: Implementation is on the harder side compared to the other methods above.

6. Authenticator Apps (TOTP)

This is the next level of security. If you have ever tried making your Google account "extra secure," you would have noticed the option to use an authenticator app. What this does is generate a TOTP (Time-Based One-Time Password).

TOTP is an algorithm that generates a unique password for each login attempt, using time as one of its parameters. Here is the key detail most explanations miss: it does not use time alone. During setup, the server generates a shared secret (usually shown as a QR code that your authenticator app scans and stores). From that point forward, both your server and your authenticator app independently run the same mathematical algorithm, using the shared secret combined with the current timestamp divided into 30-second windows, and they arrive at the same six-digit number.

You enter that number, the server checks that it matches what it computed on its end, and you are in.

Because both sides compute the code locally using the shared secret, there is nothing travelling over the network to intercept. This is what makes TOTP meaningfully more secure than SMS OTPs. SIM-swapping attacks that can intercept SMS codes simply have no target here.

Works entirely offline, too. Your authenticator app does not need a cell signal or internet connection to generate the code.

Advantages: Highly secure. Resistant to interception and SIM-swapping. Works offline. Requires the attacker to have physical access to your device with the app installed.

Disadvantages: Requires the user to download a separate third-party app. If the user loses their phone and has not saved backup codes, they are effectively locked out.

7. SSO: The Enterprise Special

Imagine you run a company. For every new hire, that person has to create 50 separate accounts across every internal tool, then request that their account be added to the right team or group for each one so they can actually start working. Isn't that a massive headache? It would take several hours just to get set up on day one.

This is exactly where SSO, which stands for Single Sign-On, comes in. As the name suggests, you sign in only once. You log into the company's identity provider, and that single authentication gives you access to all the different software and teams you are supposed to have access to. No manual account creation, no requesting access to 50 different tools.

When someone leaves the company, an admin revokes their access in one place, and they are immediately locked out of everything simultaneously. Onboarding and offboarding that used to take hours now take minutes.

Advantages: Massive productivity boost for employees. Centralized control for IT admins. Clean, auditable access management.

Disadvantages: Complex and expensive to set up correctly. If the SSO provider goes down, nobody can log into anything. It is a deliberate single point of failure, which is why IdP providers invest heavily in availability.

The Real Engineering Part

Every method above is trading off three things: security, user experience, and implementation complexity. No single method wins on all three, and the right choice depends entirely on your actual threat model.

Basic auth is the fastest to ship but leaves the most attack surface. OAuth hands identity to a trusted third party, good for both security and UX, but you now have a dependency you do not control. Passkeys are the most secure and the most frictionless once set up, but carry the largest implementation surface. TOTP occupies a useful middle ground for high-security flows without external dependencies.

The question most engineers skip: what are you actually defending against? A personal notes app does not need passkeys. A payment flow that skips two-factor authentication is negligent.

In most production systems, you do not pick one method. You layer them. OAuth for initial sign-in, TOTP as a second factor for sensitive operations, short-lived sessions throughout, SSO if you are building internal tooling. The combinations matter more than any individual choice.

The Catch

A few things that do not fit neatly into the comparison above but matter when you actually build this.

OAuth token storage is where most implementations go wrong first. Access tokens in localStorage are readable by any JavaScript on the page, including third-party scripts. Prefer httpOnly cookies or hold tokens in memory only. Short-lived access tokens with refresh token rotation are not optional in production; they are the baseline.

Magic link integrity has two non-negotiable rules: the token must be single-use (invalidated immediately after the first click), and it must expire even if it is never clicked. Skip either of these, and you are not implementing magic links. You are implementing persistent session tokens that happen to arrive by email.

TOTP clock drift is a real support ticket waiting to happen. The standard allows a one-window tolerance: the server checks the previous and next 30-second window alongside the current one, but a device with a significantly drifted clock will fail consistently. Always return a useful, specific error message rather than a generic "invalid code."

In the next post, we will get into the actual implementation: code for handling sessions, issuing JWTs, setting up OAuth with a provider, and wiring up TOTP from scratch. If the theory clicks here, the code will make it permanent.

Happy Exploration!

The Journey of a Request: What Happens Before Your Code Even Runs?

Kishan Agarwal — Wed, 10 Jun 2026 10:30:00 +0000

Okay, before we go into the depths of these concepts, I want to tell you that we will take it easy. I don’t want you to get overwhelmed by the jargon.

We spend hours arguing about which programming language is the fastest, or how to write the most optimized database query. We debate whether Go is better than Rust, or if Node.js can handle the load.

But have you ever thought about the gauntlet a user's request has to run through before it even touches your beautiful code?

Let's break down the layers of the modern web infrastructure. We have CDNs, WAFs, Load Balancers, API Gateways, Internal Reverse Proxies, Rate Limiting, and Input Sanitization.

Sounds like a massive headache, right? Let's take them one by one.

Content Delivery Network (CDN): The Franchise Model

The formal definition: A CDN is a network of interconnected servers that speeds up webpage loading for data-heavy applications.

Sounds boring. Let me explain it in simple words.

Think of McDonald’s. If McDonald's only cooked burgers in the US, would you wait a month for your food to arrive in India? Of course not. To expand, they open multiple franchises so you can order from the outlet nearest to your house.

A CDN does exactly this for your application data. When you have users from all over the globe, you don't want to serve them from a single server in Virginia. You want to get as close to them as possible. Services like Cloudflare or AWS CloudFront act as your franchises.

When a user requests an image, a video, or even a static HTML file, the CDN fetches it from your main server once, caches it, and then directly serves it to anyone else in that region who asks for it.

WAF (Web Application Firewall): The Bouncer at the Door

A WAF protects web applications by filtering and monitoring HTTP traffic. It stops attacks like Cross-Site Scripting (XSS) and SQL Injection.

Think of it this way: Do you sleep with your front door wide open? Absolutely not. So why should the ports of your application remain open to anyone on the internet?

A WAF is the bouncer securing those doors. We write specific rules that dictate who the gate will open for and who gets turned away. If a request comes in carrying a malicious payload that looks like a database command, the WAF literally slams the door in its face before your server even knows someone knocked.

Load Balancer: The Traffic Cop

A load balancer acts as a "traffic cop" sitting in front of your servers, distributing incoming client requests across a group of backend servers.

This is exactly what it sounds like. Imagine a busy intersection. If everyone tries to go down the same lane, there will be a massive traffic jam. The load balancer looks at your cluster of 10 servers and says, "Okay, Server 1 is busy sweating over a heavy calculation, let's send this new request to Server 2."

It maximizes speed and ensures high availability. If one server crashes and burns, the traffic cop just routes traffic to the surviving ones. The user never notices a thing.

API Gateway vs. Internal Reverse Proxy: The Watchman and the Guide

If you read my last blog on proxies, you know what a Reverse Proxy is. But wait, if we have an API Gateway, why do we need a reverse proxy? Let's clear the confusion.

The API Gateway (The Watchman): This is the security guard at the main gate of your residential society. They check your ID, verify if you are authorized to be there (Authentication/Authorization), check if you paid your subscription, and make sure you aren't trying to sneak in.

The Internal Reverse Proxy (The Guide): Once the watchman lets you in, you are inside a massive cluster of buildings (microservices). The Load Balancer points to the Reverse Proxy, and the Reverse Proxy looks at your request and says, "Ah, you want user data? Go to building A. You want payment history? Go to building B."

Rate Limiting: Preventing the Stampede

Rate limiting is a system design technique that restricts the number of requests a user can make to a service within a specific timeframe.

Why do we do this? Because someone out there might try to overwhelm our services by sending 10,000 requests per second. If we don't stop them, our servers will spend all their CPU power serving this one malicious user, while legitimate users wait in an endless queue. This is known as a Denial of Service (DoS) attack.

Rate limiting is our safety valve. "You've had your 100 requests for this minute, buddy. Come back later."

The Real Engineering Part: How do we actually do this? Usually, we just keep a rapid-fire counter in a fast, in-memory database like Redis. Every time a user makes a request, we increment their counter. If the counter hits 101, we return an HTTP 429 (Too Many Requests) error. Simple, but life-saving.

Input Sanitization: The Most Overlooked Lifesaver

Believe me, this is the most overlooked layer, but it is the absolute most critical one.

Users lie. Users will send you malicious scripts pretending to be standard text inputs. If you don't clean (sanitize) that input, your system can be entirely compromised.
hackerbotclaw
What did the bot do? It found repositories with automated workflow files and made Pull Requests where the "branch name" was actually a malicious script. Because the system didn't sanitize the branch name before echoing it out into the terminal, the script executed right there on GitHub's servers. Just like that, the bot gained control.

Never trust user input. Always sanitize.

What's Next?

This was just a bird's-eye view of the gauntlet. In the upcoming articles, we are going to tear each of these layers apart. We will look at the code, write our own rate limiters, and configure our own load balancers.

Happy Exploration!

The Invisible Middleman: Understanding Forward & Reverse Proxies

Kishan Agarwal — Mon, 08 Jun 2026 20:53:43 +0000

Okay, so today we are going to talk about proxies.

Sounds familiar, right? I know, I know. We have all been there during our college days, asking a friend to give a "proxy" to help us meet that strict 75% attendance criteria. Yeah, I was part of that game too.

But putting college hacks aside, we are going to talk about proxies in web development. We use this word all the time, but what does it actually mean to a software developer? Let's break it down so you understand what it truly is.

The Concept

A proxy, in the simplest terms, is something that helps us identify ourselves as another individual or server. It is a middleman.

Think about a VPN. What does a VPN do? It protects our IP address. It takes our web request, sends it to a VPN server, and then that server forwards the request to the final website. The VPN acts as a middleman. That is exactly what a proxy does.

Now, you might be thinking, "Isn't a middleman intercepting my data a bit fishy?"

It can be! A malicious middleman is literally called a "Man-in-the-Middle" attack. But in web development, we use trusted proxies for internal purposes to help our services communicate securely and efficiently.

Real-World Engineering: The Header Injection

Let me share a very interesting problem I faced where a proxy saved my life.
hello.anydomain.com
You might be thinking, "Yeah, very easy, just fetch it from the request."

The problem was that the hosting service I was using to deploy the app was overwriting the subdomain data before it reached my code. I couldn't get it.

Enter the proxy.

I set up a proxy so that before the request hit my main server, the proxy inspected it. It pulled out the subdomain, added a custom HTTP header with that information, and then forwarded the request to my server. It added a tiny bit of latency, but it worked perfectly.

The Port 80 Problem (Multiple Projects, One Server)

Where else do we use proxies? Recently, I used a single server to deploy six different Go projects.

Six projects on one server? Yeah, very easy, we can just run them on different local ports (like 3000, 8080, 9000).
amazon.com
Why? Because web standards dictate that HTTPS requests automatically go to Port 443, and HTTP requests go to Port 80. You only have one Port 443 on your server. So if all traffic comes through one port, how do we route it to six different services?

We can't do it directly. We need a proxy.

We put a proxy at the front door listening to Port 443.

When a request comes in for Service A, the proxy internally routes it to Port 3000.
When a request comes for Service B, the proxy routes it to Port 8080.

Think of it as a watchman at the entry gate of a residential society, telling visitors where to go and guiding them to the correct buildings. Now, this can sometimes generate confusion, because a real watchman also checks if you are allowed to enter the society (you can't just let a thief in!). When a proxy starts doing those security checks and authentications, it becomes an API Gateway—but we will save that for another discussion!

You have probably heard of Nginx, Apache, or Caddy. These are basically reverse proxy servers. Can you build your own in Go or Node.js? Yes, it's actually quite easy. But we use these established tools because they are battle-tested and provide incredible customization. Caddy, for instance, even gives you free, automated SSL certificates!

Forward vs. Reverse Proxies

To wrap the theory up, proxies mainly fall into two distinct categories:

1. The Forward Proxy

A forward proxy sits in front of the client. It prevents the backend server from seeing who the actual user is. (Your VPN is a forward proxy).

Goal: Hide the client from the server.

2. The Reverse Proxy

A reverse proxy sits in front of the backend. It hides your internal architecture from the client. Nginx, Apache, and Caddy are examples of reverse proxies.

Goal: Hide the server from the client.

The user just sees that all requests are magically handled by one IP address, but behind that reverse proxy, there could be a mammoth cluster of microservices running. The user never knows what is happening behind the curtain.

DIY 1: The Supabase Rescue (Build Your Own Proxy)

Want to try building a reverse proxy yourself? Now is the absolute best time to learn, and I'll tell you why.
*.supabase.co
The rescue mission circulating on X (Twitter)? Reverse Proxies.

Instead of changing client-side code or asking thousands of users to install VPNs, developers used Cloudflare Workers to act as a reverse proxy. Here is how you can do it:

Get your domain on Cloudflare and turn on the "Orange Cloud" (Proxy status) on a specific route (like db.yourapp.com).
Inside Cloudflare Workers, write a simple Node.js or Bun script.
Add a custom rule: Whenever a request hits db.yourapp.com, your Worker intercepts it, changes the host header behind the scenes, and forwards the traffic to your actual your-project.supabase.co URL.

To the ISPs, the traffic just looks like it's going to your custom domain via Cloudflare. The block is completely bypassed. It’s a perfect, real-world example of how understanding web infrastructure and proxies can literally save your startup from going dark.

DIY 2: Look Under the Hood (Real Code Examples)

I know some of you are curious. You don't just want to read about the theory; you want to see the guts of how this looks in production.

Here are two stripped-down, real-world examples from my own production environments demonstrating how these proxies are actually built.

1. The Header Injection & CDN Router (Cloudflare Workers)

Remember my story about the hosting service eating my subdomain data? Here is the exact architecture of how I solved it using a Cloudflare Worker as an Edge Proxy.

X-Subdomain

export default {

  async fetch(request) {

    const url = new URL(request.url);

    const subdomain = url.hostname.split(".")[0];

// Smart Routing based on Hostname
if (url.hostname === "cdn.letshost.dpdns.org") {
  return await handleCDNRequest(request, url);
} else if (url.hostname === "www.letshost.dpdns.org") {
  return Response.redirect("https://letshost.dpdns.org", 302);
} else {
  // The Header Injection for the main backend
  const proxyUrl = `${process.env.BACKEND_URL}${url.pathname}`;
  const modifiedRequest = new Request(proxyUrl, {
    method: request.method,
    headers: request.headers,
    body: request.body,
    redirect: "manual",
  });

  modifiedRequest.headers.set("X-Subdomain", subdomain);
  return fetch(modifiedRequest);
}


},

};

// Edge Router Logic for CDN

async function handleCDNRequest(request, url) {

  let targetUrl;

  const pathWithoutQuery = url.pathname;

  const fileExtension = pathWithoutQuery.split(".").pop().toLowerCase();

// Route Media to Transformers or Direct Storage based on extension

  if (["mp4", "webm", "avi", "mov", "mkv", "flv", "m4v", "jpg", "jpeg", "png", "gif", "webp", "svg"].includes(fileExtension)) {

    if (url.search) {

      targetUrl = ${process.env.TRANSFORMER_URL}${url.pathname}${url.search};

    } else {

      targetUrl = ${process.env.IMAGE_STORAGE_URL}${url.pathname};

    }

  } else {

    targetUrl = ${process.env.FILE_STORAGE_URL}${url.pathname}${url.search || ""};

  }

try {

    const proxyRequest = new Request(targetUrl, {

      method: request.method,

      headers: request.headers,

      body: request.body,

      redirect: "manual",

    });

    proxyRequest.headers.delete("Host"); // Prevent host header conflicts

const response = await fetch(proxyRequest);
return new Response(response.body, {
  status: response.status,
  statusText: response.statusText,
  headers: response.headers,
});


} catch (error) {


    return new Response(Proxy Error: ${error.message}, { status: 500 });


  }


}

2. The Port Multiplexing Router (Caddy)

Caddyfile7999

:7999 { # 1. Clean up trailing slashes redir /avatars /avatars/ redir /calendars /calendars/ redir /charts /charts/ redir /github /github/ redir /institutions /institutions/ redir /locations /locations/ redir /npm /npm/ # 2. Serve Static Frontend handle / { header Content-Type text/html file_server { root ./public index index.html } } # 3. Multiplexing microservices to different local ports handle_path /avatars/* { reverse_proxy localhost:8000 } handle_path /calendars/* { reverse_proxy localhost:8001 } handle_path /charts/* { reverse_proxy localhost:8002 } handle_path /github/* { reverse_proxy localhost:8003 } handle_path /institutions/* { reverse_proxy localhost:8004 } handle_path /locations/* { reverse_proxy localhost:8005 } handle_path /npm/* { reverse_proxy localhost:8006 }

}

3. The CORS Killer (Frontend Development Proxy)

If you write frontend code (React, Vue, Svelte), you know the absolute nightmare that is the CORS error.
localhost:5173
You could configure your backend to allow all CORS requests, but that can be a security risk if it accidentally ships to production. The smarter, cleaner fix? A Frontend Dev Proxy.

vite.config.js
JavaScript

import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';

export default defineConfig({
  plugins: [react()],
  server: {
    proxy: {
      // Any request starting with /api will be intercepted
      '/api': {
        target: 'http://localhost:8000', // Your backend URL
        changeOrigin: true,
        // Rewrite the URL: remove '/api' before sending to the backend
        rewrite: (path) => path.replace(/^\/api/, ''),
      },
    },
  },
});

How it works:

In your frontend code, you just make a request to /api/users.
The browser allows it without any CORS errors because it thinks it is talking to the frontend server (localhost:5173/api/users).
But under the hood, the Vite development server intercepts that request, strips away the /api part, and silently forwards it to your actual backend at http://localhost:8000/users.
This is useful and is definitely used, but please be mindful that it's always better to handle CORS on the backend rather than on the frontend.

Try implementing these! It is one thing to read about proxies passing data around, but it is a whole different feeling when you write that routing logic and see your custom headers hit your backend perfectly.

Happy Exploration!

How Instagram Stores Your Reels, Photos, and Drafts Without Losing a Frame

Kishan Agarwal — Sun, 31 May 2026 11:47:50 +0000

Ok, before we go into the depths of these concepts, I want to tell you that we will take it easy. I don't want you to get overwhelmed by terms like pre-signed URLs, object storage, chunked upload pipelines, or CDN edge nodes.

We are going to explore all of this through the lens of how a web developer would understand it. If you are coming from a web background and stepping into mobile for the first time, this transition introduces a completely new domain — and covering it from that angle will make it click much faster.

You've probably saved a Reel draft at 2 AM, opened Instagram four hours later, and it was right there — untouched. Or you've watched a Reel, swiped away, came back, and it loaded instantly without buffering. None of that happens by accident.

The system behind it is thoughtful, layered, and surprisingly easy to reason about once you see the full picture.

It Starts With a Familiar Problem: Pre-Signed URLs

If you have ever built a frontend with your own backend, you've almost certainly encountered this pattern.

When a user uploads a file, say, a profile picture or a video, you don't want to route that file through your own backend server. Instead, you generate a pre-signed URL: a link signed by your private key, valid strictly for the purpose you assigned to it, and handed directly to the client. The client uploads straight to a storage provider like AWS S3. Your backend never touches the file.

This keeps your servers fast and your upload costs low. The storage provider handles the heavy lifting.

But here's the catch. The moment you hand that URL to the client, you lose visibility. You no longer control the process. Did the upload succeed? Did it fail at 60%? You have no idea.

That's where webhooks come in.

Think of a webhook as a system that says, "Let me know when the job is done."

Let me explain it in simple words. You expose a secure endpoint on your backend. Once the upload completes, AWS S3 calls that endpoint automatically, saying "job done", ping. You didn't watch the upload happen. You just got notified when it was over.

Web developers solve this on the browser side easily enough. The browser temporarily holds the file in memory. If a page refresh happens, sure, the data is gone, but for the duration of a normal upload session, the browser manages it just fine.

Now take that same problem to mobile. And suddenly everything gets harder.

The Mobile Upload Problem Nobody Talks About

On mobile, starting a direct upload means the media sits in memory. That's it, just memory.

Networks drop. Apps close. The OS kills background processes when it needs resources. Imagine a user uploading a 3-minute video. It reaches 98%, and then the connection drops or the app refreshes. Back to 0%. They wait for 100% all over again.

That is a terrible user experience. And it is entirely avoidable.

You might be thinking, "Can't we just retry the upload automatically in the background?"

You can. But if the media only lives in memory, there is nothing left to retry after the app closes. The file is gone. Retrying from zero is the only option. The fix has to happen earlier, the moment the upload begins.

Writing to the File System First

The moment a user starts an upload, we immediately write the file to a local directory.

On mobile, tools like Expo File System give apps access to a sandboxed environment on the user's device — a real file system with real directories. The app can read and write files there freely, independent of what the network is doing.

There are generally two types of directories in this sandbox. A temporary directory for short-lived data and a persistent data directory for things that must survive app restarts, OS cleanup, or low-storage pressure.

Since we only need the media until the upload finishes, the temporary directory is the right call here. The moment the user initiates the upload, we write the file into temp storage. The user sees the upload progress as normal. Behind the scenes, we've created our own local backup, and now we have something to resume from if anything goes wrong.

Chunking and Streaming: Why Not Just Upload the Whole File?

Once the file is written locally, we take the storage URL, divide the file into chunks, and stream them one by one.

Chunking does two important things.

First, it handles memory efficiently. The entire media file never loads into RAM at once. A 500MB video doesn't need 500MB of device memory to upload; each chunk gets loaded, sent, and released.

Second, it enables background processing. Because we're streaming chunks rather than sending one massive request, the app can hand the task off to a background process. The UI stays responsive. The app doesn't feel laggy. And if a chunk fails, only that chunk retries — not the entire file.

This is why you've seen a progress bar on Instagram pause and then quietly resume. The upload wasn't frozen. It was retrying one chunk.

The Storage Comparison: Web vs Mobile

If this pattern feels familiar, it should. Web developers have been solving the same problem in the browser for years.

On the web, Local Storage handles quick key-value data, but it caps at around 5MB. For anything larger, there's IndexedDB, which offers a much bigger capacity. More recently, browsers added OPFS (Origin Private File System), which essentially runs a sandboxed file system through WebAssembly, structured, persistent, and purpose-built for larger data.

Mobile mirrors this hierarchy exactly. Async Storage on React Native is the direct equivalent of Local Storage, with the same ~5MB ceiling and the same "never put media here" rule. For real files, Expo File System is your IndexedDB — a sandboxed directory that can handle actual binary data.

You might be thinking, "Why not just throw everything into a SQLite database on the device?"

Because you never store blobs, i.e. media files, in an SQL database. Technically possible. Never recommended. SQL databases are for structured relational data. Blob storage belongs in systems built for it: file systems, object storage, CDNs. The moment you start storing video frames in SQLite rows, you have a performance problem and a maintenance nightmare.

What Happens When You Save a Draft

Whenever a user saves a draft, like the "Save as Draft" option on Instagram Reels or LinkedIn, the app grabs all the serialized data: video frames, audio, filter state, caption text, edit history. It writes everything to a structured location in local storage and leaves it there.

Think of a photographer you've hired to take your pictures at a wedding. They don't back up every single photo to Google Drive between shots. They capture everything on their memory card first, stay focused on the job, and deal with the backup later. Losing Wi-Fi mid-shoot doesn't cost them a single frame.

Instagram works the same way. Capture locally first. Upload when the time is right.

Draft data survives app restarts because the OS writes it to disk, not just to memory. This is also why, when you check "App Info" on your phone, you see the initial install size versus the much larger current size. That difference is data; it is split between things like cache storage and app storage. Everything the app is quietly managing on your behalf.

Sounds simple, right? But there's a subtle problem: if the user clears the app's cache, or if the OS decides to reclaim storage under pressure, drafts can disappear. This is exactly why heavy apps often warn you before clearing the cache or quietly sync draft metadata to their servers even before you tap publish.

Cloud Storage — Where Your Media Lives After Upload

Once you hit Share, Instagram's servers take over.

Instagram doesn't store your media in a regular file system like your laptop's hard drive. They use object storage — a system where every photo or video becomes a single blob of data with a unique identifier.

Object storage is a data storage architecture where data is managed as discrete units called "objects," each with its own ID, metadata, and content.

This gives Instagram the flexibility to store billions of photos across countless servers and retrieve any one of them in milliseconds.

The Real Engineering Part-Backend handling of uploads

Here is the portion that almost stays the same for web or mobile apps, and you could follow if you are ever dealing with systems involving media

When your Reel lands on Instagram's servers, it goes through a media processing pipeline before anyone else can watch it.

Here's what that pipeline does:

Transcoding: Your phone records in whatever format it supports — H.264, H.265, HEVC. Instagram converts it into multiple output formats and multiple resolutions. One source video becomes dozens of versions: 1080p, 720p, 480p, and lower for HLS streaming ( i.e. the player picks the right one based on the viewer's connection speed at the time of playback ). In this stage, the most commonly used library is ffmpeg, which is the standard for these conversions and the most notorious of all to handle on the server

Thumbnail generation: The system extracts the first frame (or a smart-selected moment) and generates a low-resolution preview image. As you scroll your feed, thumbnails just keep appearing instantly, so you don't have to wait. That's because object storage extracts the first frame at processing time, generates a low-res photo, and stores it directly in cache. The feed serves it from there. No network call per thumbnail. Instead, batch calls happen all at once, the data gets saved, and it sits ready for whenever you scroll to it.

Audio processing: Background music rights checks, audio normalization, and sync validation all happen here.

Metadata extraction: Duration, aspect ratio, codec — all indexed so the content delivery layer knows exactly what it is about to serve.

The whole pipeline runs asynchronously. You tap Share, get a "processing" confirmation, and Instagram's servers work through it without blocking your session.

Content Delivery Using CDNs

Once the pipeline finishes, Instagram pushes the final video files to a CDN — a Content Delivery Network.

A CDN is a distributed network of servers placed physically close to users around the world. When someone in Bengaluru watches your Reel, they are not fetching that video from a data center in Virginia. They fetch it from the nearest CDN edge server — which might sit in Mumbai.

I have already covered CDN in a previous blog, you can check it out here: CDNs

Caching — The Secret Behind Instant Loads

CDN caching handles content at the network level. But caching also happens inside the app itself.

When you scroll your feed, Instagram prefetches the next few videos before you reach them. The app estimates your scroll speed, predicts which Reels you'll hit next, and starts downloading them in the background. By the time you arrive at that Reel, the app already has it sitting in memory.

This prefetched data lives in an in-memory cache, which is fast to access, and is gone when you close the app. A second layer, the disk cache, keeps recently viewed content around a bit longer so reopening the app feels instant.

You might be thinking, "Won't caching everything eat up my phone storage?"

It does, which is why the cache has a size limit. Instagram's app typically caps its disk cache at a few hundred MBs. When that limit hits, the oldest cached files get evicted first. The content always remains available from the CDN — the local cache is just a shortcut that saves a network round-trip.

DIY: Build It Yourself

Here is a simplified yet production-ready version of the local-write-then-chunk-upload pattern in a React Native app, using the Expo File System and a pre-signed URL from S3.

import * as FileSystem from "expo-file-system";

const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB per chunk

async function writeToTempAndUpload(
  localUri: string,
  uploadId: string,
  getPresignedUrl: (partNumber: number) => Promise<string>
) {
  // Step 1: Copy to temp directory for resilience
  const tempPath = `${FileSystem.cacheDirectory}upload_${uploadId}.mp4`;
  await FileSystem.copyAsync({ from: localUri, to: tempPath });

  const fileInfo = await FileSystem.getInfoAsync(tempPath, { size: true });
  const totalSize = (fileInfo as FileSystem.FileInfo & { size: number }).size;
  const totalChunks = Math.ceil(totalSize / CHUNK_SIZE);

  const uploadedETags: string[] = [];

  for (let i = 0; i < totalChunks; i++) {
    const partNumber = i + 1;
    const start = i * CHUNK_SIZE;
    const end = Math.min(start + CHUNK_SIZE, totalSize);

    // Step 2: Get a fresh pre-signed URL per part
    const presignedUrl = await getPresignedUrl(partNumber);

    // Step 3: Upload this chunk
    const result = await FileSystem.uploadAsync(presignedUrl, tempPath, {
      httpMethod: "PUT",
      headers: {
        "Content-Type": "video/mp4",
        "Content-Range": `bytes ${start}-${end - 1}/${totalSize}`,
      },
      uploadType: FileSystem.FileSystemUploadType.BINARY_CONTENT,
    });

    if (result.status !== 200 && result.status !== 206) {
      throw new Error(`Part ${partNumber} failed: ${result.status}`);
    }

    const eTag = result.headers["ETag"];
    uploadedETags.push(eTag);
    console.log(`Part ${partNumber}/${totalChunks} done — ETag: ${eTag}`);
  }

  // Step 4: Signal completion to your backend (it finalizes the S3 multipart upload)
  await fetch("/api/complete-upload", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ uploadId, parts: uploadedETags }),
  });

  // Step 5: Clean up temp file after success
  await FileSystem.deleteAsync(tempPath, { idempotent: true });
}

The cacheDirectory here is your temp storage — it survives app restarts but can be reclaimed by the OS. The copy step in Step 1 is what buys you resilience. If the upload fails at chunk 3, the full file is still sitting at tempPath. Only that chunk needs to retry.

Notice that each part returns an ETag. S3's multipart upload system uses these to verify and reassemble all parts in the correct order when your backend calls CompleteMultipartUpload. You aren't uploading blind — every chunk is tracked and acknowledged server-side.

Try implementing this yourself! It is one thing to read about chunked uploads, but it is a whole different feeling when you watch a progress bar actually pause, retry a single chunk, and keep climbing.

With all this discussion, I hope you had a better understanding of how the uploads on devices behave differently from the web. The backend handling is almost the same for web or mobile.

Happy Exploration!

Cuckoo Filters vs. Bloom Filters: Let’s Take it Easy

Kishan Agarwal — Sat, 30 May 2026 07:55:26 +0000

Ok, before we go into the depths of these concepts, I want to tell you that we will take it easy. I don’t want you to get overwhelmed about the topics and the depth of the discussion.

Bloom Filters

A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set.

Sounds hard, right? In just one sentence: "probabilistic," then "data structure," and all this is what the definition says. Let's break it down so you understand what it truly is.

The Concept

Ok, I give you a problem. Think of it for a second. Let's say you have an array of characters (English alphabet characters) of size n, and you have to find whether a char exists or not.

Approach 1 (Loop): Quite simple, we can just loop the array once. This gives us a time complexity of O(n)
Approach 2 (Sort + Binary Search): We sort the array (O(nlog n)) and then perform a binary search (O(log n)). Steps increased, but at least we reduced the time.
Approach 3 (HashMap): We store it in a HashMap. This way we take double space, but at least we can optimise the time to O(1)

Can we optimise it further? For most of us, this may be ok because you have to prioritise over time and space. Now, for those who want both, the discussion is for you, my friend.

Think of this: Have you read about ASCII values before?

Let's say a user gives me input 'a'. I don't store 'a'. Instead, I calculate its relative position from 'a' and store it in a bit array of length 26.

We get a number, consider it as an index.
Store 1 in the position, else 0.

Now, to check for a char, we just repeat the process. We can access this in, and the space it will take is just 26 bits. Insane, right?

The Real Engineering Part (The Hashing Reality)

But then, that's for a single char. For a string, we can't do that. And Bloom filters are mostly used on strings. So now get to the real engineering part.

In Bloom filters, we use a hashing function known as MurmurHash. What is hashing? Hashing is a technique of converting a string to an equivalent number.

But here is the catch: Hashing is deterministic (Input A always gives Output A), but the universe of strings is infinite, while our array is finite.

For A, you get a hash.
Then B gets a hash.
Then C can also generate the same hash as B.

Can you see what is happening? These are collisions. The main problem is this: hashing can sometimes give the same index for two strings. So in order to prevent this, we need to be clever.

Bloom Filters in Production (Making it Robust)

Now, you might be thinking, "If collisions are inevitable, isn't this useless?"

In a real production setup, we don't rely on just one hash function. That would be too risky. In the real world, we use Multiple Hash Functions (let's say functions).

When a string comes in, we run it through Hash Function 1, Hash Function 2, and Hash Function 3.
We get 3 different indexes.
We set all 3 positions to 1.

Now, when we query:

We check all 3 positions.
If all of them are 1, then we say "Yes, it's probably here."
If even one of them is 0, we can say with 100% guarantee: "It is not here."

This drastically reduces the false positive rate. It’s like asking three different friends if a restaurant is open. If all three say yes, it's probably open. If one says, "No, I'm standing in front of it, and it's closed," then you know for sure.

The Major Problem

Now, can you see a major problem in this Bloom filter? I also recently came to know about it.

The problem is deletion.

Once a Bloom filter gets filled, let's say User A deletes their account. Technically, this username is free. But your Bloom filter will still say that the username exists.

We can query in O(1)
We can insert in O(1)
But we cannot delete data.

Why? If I go to those 3 positions and set them back to 0, maybe User B was also using one of those spots! If I delete User A, I might accidentally "corrupt" User B. This is why deletion is extremely challenging in Bloom filters.

So, is there a fix? (The Counting Bloom Filter)

Now you might be thinking, "Can’t we just simply count the items instead of just flipping bits?"

Actually, yes. There is something called a Counting Bloom Filter.

The logic is super simple:

In a normal Bloom filter, we have bits (0 or 1).
In a Counting filter, we keep a counter (a number).

So when you add an item, you just increment the counter at that index (+1). When you want to delete, you just decrement it (-1).

Sounds perfect, right? Problem solved! But here is the major catch.

To store a counter (even a small number), you need way more space than a single tiny bit. A standard Bloom filter uses 1 bit per spot. A Counting filter might need 4 bits or more just to hold that number.

So, we effectively quadrupled the space usage. We solved the deletion problem, but we killed the whole purpose of using a Bloom filter: Space Efficiency. If it takes too much RAM, we might as well just use a HashMap, right?

We need something that allows deletion without blowing up our memory.

Cuckoo Filters: The Solution

So you might be thinking, "yeah, then there might be a solution." Yes, there is a fantastic solution known as the Cuckoo Filter.

Does the name give you some hint? Cuckoo.

The bird that doesn't build its own nest; it takes over the nests of other birds. But why this?

Cuckoo Hashing: The "Kicking" Strategy

Cuckoo filters work on the principle of partial-key cuckoo hashing. Bloom filters store bits (0 or 1). Cuckoo filters store a Fingerprint.

What is a fingerprint? Instead of just marking a spot as "occupied," we take a small signature of the data (like an 8-bit or 12-bit hash) and store that. This is the game changer.

Here is how the "Kicking" works (The Cuckoo Logic):

Hash 1: String A comes. I hash it and try to put its fingerprint in Nest 1.
Collision: If Nest 1 is full, we don't just give up. The new entry kicks out the old entry!
Re-homing: The old entry (the victim) is now homeless. It has to find a new spot.
The Magic Math: In production, we use a math trick (XOR operation) to calculate the "alternative nest" using just the current position and the fingerprint. The victim flies to Nest 2.
Loop: If Nest 2 is also full? It kicks that one out.

It’s a chain reaction. It can very easily turn into an infinite loop. This is the only slight problem.

The Fix: We assign Max Iterations (let's say 500). It will try to kick items around 500 times. If no solution is reached, then we know the filter is too full, and we need to expand.

Production Setup: Buckets and Efficiency

In a real production setup (like in large-scale databases), we don't just have single slots. We use Buckets.

Imagine the array is divided into buckets, and each bucket has 4 slots.
This means when a hash maps to a bucket, we can fit 4 different items there before we have to start kicking anyone out.
This makes the Cuckoo filter insanely dense. We can fill it up to 95% capacity, and it still works fast. Bloom filters usually start failing around 50% or 70% full.

Why does this solve Deletion?

Because we store Fingerprints, not just bits.

If I want to delete User A:

I calculate the hash and look in the bucket.
I check the fingerprints stored there.
I find the fingerprint that matches User A.
I remove it.

Since I am matching a specific fingerprint, I am highly unlikely to delete User B by accident. This gives us ** Deletion** without the huge memory cost of Counting Bloom Filters.

Want to try it? (RedisBloom)

Now, you might be thinking, "This is cool, but do I have to write all this complex math and XOR logic myself?"

No, my friend. You don't.

If you are using Redis (and you probably are), there is a module called RedisBloom. It has all this implemented for you.

For Bloom Filters: You just run BF.ADD key item and `BF.EXISTS key item. It handles the multiple hashes for you.
For Cuckoo Filters: You run CF.ADD key item the magic command. CF.DEL key item.

It’s production-ready, highly optimised, and you don't have to worry about the "kicking" logic—it happens behind the scenes. So if you need to optimise space, just load this module, and you are good to go.

DIY: Build It Yourself (Pseudocode)

I know some of you are curious. You don't just want to use a library; you want to see the guts of the algorithm.

If you want to implement this in Python, Go, or Java to learn, here is the blueprint. I’ve written this in pseudocode so you can translate it into your favourite language.

1. The Bloom Filter Logic

This is straightforward. We need a bit array and multiple hash functions.

// Configuration
ArraySize = 1000
BitArray = [0] * ArraySize

// Function to Add an Item
Function Add(item):
    idx1 = Hash1(item) % ArraySize
    idx2 = Hash2(item) % ArraySize
    idx3 = Hash3(item) % ArraySize

    BitArray[idx1] = 1
    BitArray[idx2] = 1
    BitArray[idx3] = 1

Function Exists(item):
    idx1 = Hash1(item) % ArraySize
    idx2 = Hash2(item) % ArraySize
    idx3 = Hash3(item) % ArraySize

    IF (BitArray[idx1] == 1 AND BitArray[idx2] == 1 AND BitArray[idx3] == 1):
        RETURN True
    ELSE:
        RETURN False

2. The Cuckoo Filter Logic

This is where the magic happens. The key here is the "Kicking" loop and the XOR math to find the alternate index.

// Configuration
MaxKicks = 500
Buckets = [Size 1000] // Each bucket can hold multiple fingerprints

// Helper: Calculate Fingerprint (a small hash)
Function GetFingerprint(item):
    RETURN Hash(item) -> truncated to 8 bits

// Helper: Calculate Two Possible Locations
Function GetLocations(item, fingerprint):
    idx1 = Hash(item)
    
    // The Magic: We can find idx2 using only idx1 and the fingerprint!
    idx2 = idx1 XOR Hash(fingerprint)
    
    RETURN idx1, idx2

// Function to Add Item
Function Insert(item):
    fp = GetFingerprint(item)
    idx1, idx2 = GetLocations(item, fp)

    // 1. Try to fit in the first nest
    IF Buckets[idx1] has space:
        Buckets[idx1].add(fp)
        RETURN Success

    // 2. Try to fit in the second nest
    IF Buckets[idx2] has space:
        Buckets[idx2].add(fp)
        RETURN Success

    // 3. Both full? Time to KICK!
    current_idx = RandomlyPick(idx1, idx2)
    
    LOOP for count from 0 to MaxKicks:
        // Swap the new fingerprint with the one currently in the bucket
        victim_fp = Buckets[current_idx].swap_with(fp)
        
        // Make the victim the new item to insert
        fp = victim_fp
        
        // Calculate where the victim should go (The Magic XOR again)
        current_idx = current_idx XOR Hash(fp)

        IF Buckets[current_idx] has space:
            Buckets[current_idx].add(fp)
            RETURN Success

    RETURN Failure // Filter is too full, we need to expand!

Try implementing this! It’s one thing to read about "kicking" items, but it’s a whole different feeling when you write that loop and see your code shuffling data around to make space.

Happy Exploration!