DEV Community: Jatin Gupta

Forward Proxy vs Reverse Proxy: The Complete Guide Every Developer Should Know

Jatin Gupta — Thu, 25 Jun 2026 03:26:11 +0000

If you've ever worked with Nginx, Cloudflare, Load Balancers, CDNs, Kubernetes, or Microservices, you've probably heard the terms Forward Proxy and Reverse Proxy.

Many developers memorize the definitions but struggle during interviews when asked:
"What's the difference between a Forward Proxy and a Reverse Proxy?"

What is a Proxy Server?
A Proxy Server is a middleman between two systems.
Instead of communicating directly, requests pass through a proxy first.

Client → Proxy → Server

The proxy receives the request, processes it, and forwards it to the destination.

Think of it as a receptionist in an office:
You don't directly meet the CEO.
You first talk to the receptionist.
The receptionist forwards your request. The same idea applies in networking.

Why Do We Need Proxies?
Proxies are used for:

✅ Security

✅ Privacy

✅ Caching

✅ Load Balancing

✅ Monitoring

✅ Access Control

✅ Hiding Network Details

Types of Proxy Servers
There are two major types:

Forward Proxy
Reverse Proxy

The main difference is:

| Proxy Type    | Hides  |
| ------------- | ------ |
| Forward Proxy | Client |
| Reverse Proxy | Server |

If you remember only one thing from this article, remember this table.

Understanding Forward Proxy
A Forward Proxy sits between the client and the internet.

Client → Forward Proxy → Internet

The website sees the proxy's IP instead of the client's IP.
Forward Proxy Architecture

+--------+
| Client |
+--------+
     |
     v
+---------------+
| Forward Proxy |
+---------------+
     |
     v
+------------+
| Website    |
| Server     |
+------------+

How Forward Proxy Works
Step 1
The client sends a request to the proxy.

Client → Proxy

Step 2
The proxy forwards the request to the website.

Proxy → Website

Step 3
The website sends a response back to the proxy.

Website → Proxy

Step 4
The proxy sends the response to the client.

Proxy → Client

What Does Forward Proxy Hide?
It hides the client.
The server doesn't know who the real client is.

It only sees the proxy.

Actual Client ❌ Hidden

Proxy IP ✅ Visible

Real-Life Example

Imagine your company blocks social media websites.
You use a proxy server.

You → Proxy → Facebook

Facebook sees:

Request from Proxy

instead of:

Request from You

Your identity is hidden.

Common Uses of Forward Proxy

Privacy Hide user's IP address.

Example:

Anonymous browsing
Privacy protection

Access Control Organizations control internet access.

Example:

Block YouTube
Block Gaming Websites
Block Social Media

Content Filtering Schools and colleges often use proxies.

Example:

Student → Proxy → Internet

Proxy blocks unwanted websites.

Caching Frequently visited websites are cached.

Benefits:

Faster loading
Reduced bandwidth

Popular Forward Proxy Tools

Squid Proxy
HAProxy
Privoxy
Blue Coat Proxy

Understanding Reverse Proxy
A Reverse Proxy sits between users and servers.

Client → Reverse Proxy → Server

Clients think they are talking directly to the server.
In reality, the reverse proxy handles everything.

Reverse Proxy Architecture

                 +-----------+
                 | Server 1  |
                 +-----------+

Client
   |
   v
+----------------+
| Reverse Proxy  |
+----------------+
   |
   +------------> +-----------+
                  | Server 2  |
                  +-----------+

   |
   +------------> +-----------+
                  | Server 3  |
                  +-----------+

How Reverse Proxy Works
Step 1
Client sends request.

Client → Reverse Proxy

Step 2
Proxy chooses a backend server.

Reverse Proxy → Server

Step 3
Server processes request.

Server → Reverse Proxy

Step 4
Proxy returns response.

Reverse Proxy → Client

What Does Reverse Proxy Hide?
It hides the servers.

Clients don't know:

Number of servers
Server IP addresses
Internal architecture

Server Details ❌ Hidden
Proxy Address ✅ Visible

Real-Life Example
Suppose Amazon receives millions of requests.

Instead of exposing backend servers:

Users
   |
   v
Cloudflare / Nginx
   |
   +--> Server 1
   +--> Server 2
   +--> Server 3

Users only see:

amazon.com

They never see actual backend servers.

Why Reverse Proxy Is So Important
Almost every large-scale application uses it.

Examples:

Netflix
Amazon
YouTube
Facebook
Instagram
Uber

Features of Reverse Proxy

Load Balancing Distributes traffic across multiple servers.

Without Load Balancer:

1000 Requests
      |
      v
   Server 1

Server crashes.

With Reverse Proxy:

1000 Requests
      |
      v
 Reverse Proxy
   /   |   \
  /    |    \
 S1   S2    S3

Traffic gets distributed.

Benefits:

Better performance
High availability

Security Backend servers remain hidden. Attackers cannot directly access them.

Benefits:

DDoS Protection
IP Masking
Firewall Integration

SSL Termination Reverse proxy handles HTTPS.

Client (HTTPS)
      |
      v
Reverse Proxy
      |
      v
Server (HTTP)

Backend servers don't need SSL certificates.
This reduces server workload.

Caching Frequently requested responses are cached.

Benefits:

Faster responses
Lower server load

Compression Can compress data before sending it.

Result:

Faster website loading
Less bandwidth usage

Most Popular Reverse Proxy Solutions

Nginx
Most widely used reverse proxy.

server {
    listen 80;

    location / {
        proxy_pass http://backend;
    }
}

HAProxy
Excellent for load balancing.

Cloudflare
Acts as a global reverse proxy.

Provides:

DDoS Protection
CDN
Caching
Security

Traefik

Popular in Docker and Kubernetes.

Forward Proxy vs Reverse Proxy

| Feature           | Forward Proxy            | Reverse Proxy              |
| ----------------- | ------------------------ | -------------------------- |
| Sits In Front Of  | Clients                  | Servers                    |
| Hides             | Client                   | Server                     |
| Used By           | Users                    | Website Owners             |
| Purpose           | Privacy & Access Control | Security & Scalability     |
| Traffic Direction | Outgoing                 | Incoming                   |
| Main Goal         | Hide Client Identity     | Hide Server Infrastructure |
| Example           | VPN, Squid Proxy         | Nginx, Cloudflare          |

Easy Interview Trick
If interviewer asks:

"How do I remember the difference?"

Use this rule:
Forward Proxy

FORWARD = FOR USER

Hides the client.

Reverse Proxy

REVERSE = FOR SERVER

Hides the server.

Interview Questions

Q1: What is a Proxy Server?

A proxy server acts as an intermediary between a client and a server.

Q2: What is Forward Proxy?

A server that sits between the client and the internet and hides the client identity.

Q3: What is Reverse Proxy?

A server that sits between users and backend servers and hides server details.

Q4: Which Proxy Provides Load Balancing?

Reverse Proxy. Examples:

Nginx
HAProxy
Traefik

Q5: Which Proxy Is Used for Privacy?

Forward Proxy.

Q6: Does Cloudflare Work as a Reverse Proxy?

Yes. Cloudflare sits between users and origin servers.

Q7: Does Nginx Act as a Reverse Proxy?

Yes. Nginx is one of the most commonly used reverse proxy servers.

Q8: Why Is Reverse Proxy Used in Microservices?
Because it provides:

Routing
Load Balancing
SSL Termination
Security
Service Discovery

Why Companies Like Google Prefer gRPC for Microservices

Jatin Gupta — Sun, 21 Jun 2026 03:39:42 +0000

REST vs gRPC
REST changed how applications communicate over the internet. But when companies started building hundreds of microservices that needed to talk to each other millions of times per second, REST began to show its limitations. That's where gRPC came in.

Introduction
If you've built APIs before, you've probably used REST.
A client sends an HTTP request, the server responds with JSON, and everything works perfectly.

But what happens when:

Thousands of services communicate internally?
Low latency becomes critical?
Network traffic becomes massive?
Multiple programming languages are involved? This is exactly the challenge Google faced while running its enormous distributed systems.

To solve this problem, Google developed gRPC (Google Remote Procedure Call) and open-sourced it in 2015.

Today, companies like Google, Netflix, Uber, Square, and many cloud-native applications use gRPC for internal service communication.

What is REST?

REST (Representational State Transfer) is an architectural style for designing APIs.
A REST API exposes resources through URLs.

Example:

GET /users/101

Response:

{
  "id": 101,
  "name": "Jatin Gupta",
  "email": "jatin@example.com"
}

REST commonly uses:

HTTP/1.1
JSON
URLs
GET, POST, PUT, DELETE methods

REST Workflow

Client
   |
HTTP Request
   |
Server
   |
JSON Response
   |
Client

What is RPC?

RPC stands for Remote Procedure Call.
Instead of accessing resources, the client directly calls a function that exists on another machine.

Example:

user = getUser(101)

Even though the function runs on a remote server, it feels like calling a local method.

What is gRPC?
gRPC is a modern implementation of RPC created by Google.
It allows services to communicate efficiently using:

HTTP/2
Protocol Buffers (Protobuf)
Code Generation
Bi-directional Streaming Instead of exchanging JSON, gRPC exchanges compact binary data.

gRPC Architecture

+-------------------+
|  Client Service   |
+-------------------+
          |
          |
          v
+-------------------+
|   gRPC Channel    |
+-------------------+
          |
          |
          v
+-------------------+
|   Server Service  |
+-------------------+

The client calls a method directly:

userService.getUser(101)

The network communication happens automatically behind the scenes.

How gRPC Works
Step 1: Define Service
Create a .proto file.

syntax = "proto3";

service UserService {
  rpc GetUser(UserRequest)
      returns (UserResponse);
}

message UserRequest {
  int32 id = 1;
}

message UserResponse {
  int32 id = 1;
  string name = 2;
}

Step 2: Generate Code
gRPC generates:

Java classes
Go classes
Python classes
Node.js classes
C# classes

Automatically.

Step 3: Implement Server

public UserResponse getUser(UserRequest request) {
    return UserResponse.newBuilder()
        .setId(1)
        .setName("Jatin")
        .build();
}

Step 4: Call From Client

UserResponse user =
    client.getUser(request);

No manual JSON parsing.
No URL creation.
No request serialization.

REST vs gRPC

| Feature         | REST      | gRPC             |
| --------------- | --------- | ---------------- |
| Protocol        | HTTP/1.1  | HTTP/2           |
| Data Format     | JSON      | Protocol Buffers |
| Speed           | Slower    | Faster           |
| Payload Size    | Large     | Small            |
| Streaming       | Limited   | Native           |
| Type Safety     | No        | Yes              |
| Code Generation | No        | Yes              |
| Browser Support | Excellent | Limited          |
| Microservices   | Good      | Excellent        |

Why gRPC is Faster

Binary Serialization REST uses JSON.

Example:

{
  "name":"Jatin"
}

gRPC converts the same data into compact binary format.

Benefits:

Smaller payloads
Less bandwidth
Faster parsing

2. HTTP/2 Multiplexing
REST mostly uses HTTP/1.1.

Problem:
Multiple requests often require multiple connections.
gRPC uses HTTP/2.

Benefits:

Multiple requests on one connection
Reduced latency
Better throughput

Single Connection
 ├── Request 1
 ├── Request 2
 ├── Request 3
 └── Request 4

3. Strongly Typed Contracts
REST documentation can become outdated.

gRPC uses .proto files as the source of truth.

Benefits:

Clear contracts
Automatic validation
Better developer experience

Streaming: A Superpower of gRPC

REST follows:

Request -> Response

gRPC supports:

1. Unary RPC

Request -> Response

2. Server Streaming

Request -> Response 1
          Response 2
          Response 3

3. Client Streaming

Request 1
Request 2
Request 3
        -> Response

4. Bidirectional Streaming

Client <----> Server

Both can send messages simultaneously.
This is extremely useful for:

Live chats
Gaming
Video streaming
IoT devices
Real-time analytics

Why Google Uses gRPC
Google runs one of the world's largest distributed systems.
A single user action may trigger dozens of services.

Example:

Search Service
      |
      +--> Ranking Service
      |
      +--> Ad Service
      |
      +--> User Profile Service
      |
      +--> Analytics Service

If every communication used JSON:

More bandwidth
More CPU usage
More latency

gRPC solves these issues by providing:

✅ Smaller messages

✅ Faster communication

✅ Automatic code generation

✅ Better scalability

✅ Streaming support

Does gRPC Replace REST Completely?

No. This is one of the biggest misconceptions.

REST is still the best choice for:

Public APIs
Browser applications
Third-party integrations
Mobile app backends
Simpler systems Example:

Frontend App
      |
     REST
      |
API Gateway
      |
     gRPC
      |
Microservices

This hybrid architecture is extremely common today.

When Should You Use REST?
Choose REST when:

Building public APIs
Working with browsers
Simplicity matters
Human-readable responses are needed
Third-party developers consume your API

When Should You Use gRPC?

Choose gRPC when:

Building microservices
Performance matters
Low latency is critical
High traffic systems exist
Streaming is required
Multiple programming languages are involved

Real-World Example
Imagine an e-commerce platform.

REST Architecture

Frontend
   |
 REST
   |
Backend

Works perfectly.

Microservice Architecture

Frontend
    |
 REST API Gateway
    |
 --------------------------------
 |        |        |            |
User    Order   Payment   Inventory
         gRPC Services

Now hundreds of service-to-service calls happen every second.
gRPC becomes the better option.

Conclusion

REST revolutionized web APIs and remains the standard for public-facing services. However, as applications evolved into large-scale distributed systems, the need for faster and more efficient communication became critical.

That's why Google created gRPC.
The real question isn't "REST or gRPC?"

It's:

"Where should I use REST, and where should I use gRPC?"

The most successful modern systems combine both:

REST for external communication.
gRPC for internal microservice communication. And that's why gRPC has become the backbone of many high-performance cloud-native applications today.

"Kubernetes Sidecar Pattern Explained: The Secret Helper Container Running Beside Your App"

Jatin Gupta — Fri, 19 Jun 2026 17:34:18 +0000

When developers start learning Kubernetes, they usually focus on Pods, Deployments, Services, and Ingress. But once applications grow in production, another powerful design pattern becomes very important: the Sidecar Pattern.

The Sidecar Pattern is one of the most widely used Kubernetes patterns because it helps developers add extra functionality to an application without changing the application's code.

What is the Sidecar Pattern?
Imagine you have a car and attach a small sidecar to it.
The motorcycle does its main job of transportation, while the sidecar provides additional functionality without changing the motorcycle itself.
The same idea exists in Kubernetes.

A Sidecar Container is an additional container that runs alongside the main application container inside the same Pod.

Both containers share:

Network
Storage Volumes
Lifecycle This allows them to communicate easily and work together as a team.

Simple Representation

+--------------------------------+
|            POD                 |
|                                |
|  +---------+   +------------+  |
|  | Main    |<->| Sidecar    |  |
|  | App     |   | Container  |  |
|  +---------+   +------------+  |
|                                |
+--------------------------------+

The main application focuses only on business logic, while the sidecar handles supporting tasks.

Why Do We Need a Sidecar?
Many applications require additional features such as:

Log collection
Monitoring
Security
Data synchronization
Traffic management

Without a Sidecar, developers often have to build these features directly into the application.

This creates several problems:

❌ More code to maintain

❌ Increased complexity

❌ Duplicate logic across services

❌ Difficult upgrades

The Sidecar Pattern solves these issues by separating responsibilities.

How Sidecar Works

Inside a Kubernetes Pod:

The main container runs the application.
The sidecar container runs helper processes.
Both containers share the same network namespace.
They can access shared storage volumes.
Kubernetes manages them together.

Example

Suppose your application generates logs.

Instead of modifying the application to send logs to Elasticsearch or Splunk:

Application writes logs to a file.
Sidecar reads the file.
Sidecar sends logs to a centralized logging system.

The application remains clean and focused.

Common Use Cases of Sidecar Pattern

1. Logging
This is one of the most popular uses of Sidecars.

Application
     |
Writes Logs
     |
     v
Log Collector Sidecar
     |
     v
Central Logging System

Examples:

Fluent Bit
Fluentd
Vector

Benefits:

- No logging code in application
- Centralized log management
- Easy replacement of logging tools

2. Monitoring
Applications expose metrics, while Sidecars collect and forward them.

Examples:

Prometheus exporters
Monitoring agents

Benefits:

- Better observability
- Performance tracking
- Resource monitoring

3. Service Mesh
Modern service meshes rely heavily on Sidecars.

Examples:

Istio Envoy Proxy
Linkerd Proxy

The Sidecar intercepts network traffic and provides:

- Load balancing
- Traffic routing
- Encryption
- Retry mechanisms

Without changing application code.

4. Security

Security-related functionality can be delegated to a Sidecar.

Examples:

TLS certificate management
Authentication proxies
Secret rotation

Benefits:

Improved security
Centralized policy enforcement

5. Data Synchronization & Caching
Some applications need data from external systems.

A Sidecar can:

Fetch data periodically
Sync configuration
Maintain local caches

The application simply consumes the data.

Sidecar vs Without Sidecar
Without Sidecar

Application
 ├─ Logging Logic
 ├─ Monitoring Logic
 ├─ Security Logic
 └─ Business Logic

Problems:

Tightly coupled
Difficult maintenance
Larger codebase
Harder scaling

With Sidecar

Application
 └─ Business Logic

Sidecar
 ├─ Logging
 ├─ Monitoring
 └─ Security

Benefits:

✅ Cleaner architecture

✅ Better separation of concerns

✅ Easier maintenance

✅ Reusable functionality

✅ Independent updates

Real-World Example: Log Collection
Let's say an e-commerce application generates logs.

Main Container

E-commerce App
      |
Writes Logs
      |
      v
Shared Volume

Sidecar Container

Fluent Bit Sidecar
      |
Reads Logs
      |
      v
Elasticsearch

The application never needs to know where logs are stored.
If tomorrow the company switches from Elasticsearch to another logging platform, only the Sidecar configuration changes.
The application code remains untouched.

Advantages of Sidecar Pattern
1. Separation of Concerns

Each container focuses on a specific responsibility.

2. Reusability

The same Sidecar can be reused across multiple services.

3. Independent Updates

Update logging or monitoring without redeploying application code.

4. Improved Security

Sensitive functionality can be isolated.

5. Better Maintainability

Applications stay lightweight and easier to understand.

Challenges of Sidecar Pattern
Although powerful, Sidecars are not free.

Additional Resource Usage
Each Sidecar consumes:

CPU
Memory
Storage

Increased Complexity
More containers mean:

More monitoring
More debugging
More configuration

Pod Dependency
If the Sidecar fails, application behavior may be affected.

Best Practices
Keep Sidecars Focused

A Sidecar should have a single responsibility.

Good:

Logging Sidecar
Monitoring Sidecar

Bad:

One Sidecar doing five different tasks.

Monitor Resource Usage

Always define:

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

Use Shared Volumes Carefully
Only share the data that is necessary.

Avoid Business Logic

Business logic should stay in the main application.
The Sidecar should only provide supporting capabilities.

Conclusion

The Sidecar Pattern is one of the most important Kubernetes design patterns used in modern cloud-native applications.

Instead of stuffing logging, monitoring, security, and networking features into your application, Kubernetes allows you to run helper containers alongside the main container. This keeps applications clean, modular, and easier to maintain.

Whether you're using Fluent Bit for logging, Prometheus exporters for monitoring, or Envoy proxies in a service mesh, chances are you're already benefiting from the Sidecar Pattern.

The biggest takeaway is simple:

Let your application focus on business logic, and let Sidecars handle everything else.

That's the beauty of the Kubernetes Sidecar Pattern.

The Unsung Hero Behind Reliable Distributed Systems - Apache ZooKeeper

Jatin Gupta — Thu, 18 Jun 2026 03:45:20 +0000

If you've ever worked on distributed systems, you've probably heard people mention Apache ZooKeeper. For a long time, I knew it was important, but I didn't fully understand why so many large-scale systems depended on it.
Then I started digging deeper into concepts like leader election, distributed coordination, service discovery, and fault tolerance. That's when ZooKeeper finally clicked.
In this article, I'll explain ZooKeeper in simple terms, why it exists, the problems it solves, and how it works behind the scenes.

Why Do We Need ZooKeeper?

Imagine you're running a distributed application with multiple servers.

Server A
Server B
Server C

All three servers are capable of performing the same task.
Now suppose a cron job is scheduled to run every day at 9 AM.

Without coordination:

Server A → Executes Job
Server B → Executes Job
Server C → Executes Job

Instead of one report being generated, you now have:

3 reports
3 emails sent
Duplicate processing
Data inconsistencies Clearly, this is a problem. What we really want is:

Server A → Executes Job
Server B → Waits
Server C → Waits

Only one server should be responsible for executing the task.
But who decides which server gets the responsibility?
This is where ZooKeeper enters the picture.

What Is Apache ZooKeeper?
Apache ZooKeeper is a distributed coordination service that helps multiple servers agree on shared state.
Think of it as a referee for distributed systems.

Instead of every server making independent decisions, ZooKeeper acts as a central coordination layer that answers questions like:

Who is the leader?
Which servers are alive?
Who owns a lock?
Which node should process a task?
Where is a service located? In short, ZooKeeper helps distributed systems stay organized.

The Core Problem: Coordination
Distributed systems are hard because servers can:

Crash unexpectedly
Lose network connectivity
Restart at any time
Become temporarily unavailable Without coordination, each server may make conflicting decisions.

For example:

Server A thinks it is leader.
Server B thinks it is leader.
Server C thinks both are dead.

Now your system is in chaos.
ZooKeeper prevents these situations by maintaining a consistent view of the cluster.

ZooKeeper Architecture
A ZooKeeper cluster typically consists of multiple ZooKeeper servers.

Example:

ZooKeeper Server 1
ZooKeeper Server 2
ZooKeeper Server 3
ZooKeeper Server 4
ZooKeeper Server 5

Among them:

One becomes the Leader
The rest become Followers

Leader
├── Follower
├── Follower
├── Follower
└── Follower

The leader handles write operations while followers replicate data and serve read requests.
This ensures consistency across the cluster.

Why Multiple ZooKeeper Servers?
Imagine ZooKeeper itself runs on only one machine.

ZooKeeper Server

If that machine crashes:

Coordination = Gone

Your entire distributed system becomes vulnerable.
To avoid this, ZooKeeper runs as a cluster.
As long as a majority of ZooKeeper nodes are alive, the system continues functioning.
This concept is called a quorum.

Understanding Quorum
ZooKeeper requires a majority of nodes to be available.

Formula:

Majority = (N / 2) + 1

Examples:

| Total Nodes | Required Majority |
| ----------- | ----------------- |
| 3           | 2                 |
| 5           | 3                 |
| 7           | 4                 |

For a 5-node ZooKeeper cluster:

Node1
Node2
Node3
Node4
Node5

The cluster continues operating even if:

Node4 fails
Node5 fails

Because:

Node1 + Node2 + Node3 = Majority

Leader Election Explained
One of ZooKeeper's most famous use cases is leader election.
Suppose three application servers are running:

App Server A
App Server B
App Server C

Each server registers itself with ZooKeeper.
ZooKeeper elects one server as the leader.

Leader → Server B
Followers → Server A, Server C

Now only Server B executes critical tasks.
If Server B crashes:

Server B ❌

ZooKeeper automatically elects a new leader.

Leader → Server A

The system keeps running without manual intervention.
This is one of the reasons distributed systems remain highly available.

Distributed Locking
Another common problem:
Two servers try to update the same resource simultaneously.

Example:

Server A updates Inventory
Server B updates Inventory

This may create inconsistent data.
ZooKeeper solves this using distributed locks.

Before modifying the resource:

Server A requests lock

ZooKeeper grants it.

Lock Owner → Server A

Now:

Server B waits

When Server A finishes:

Lock Released

Then ZooKeeper allows Server B to proceed.
This prevents race conditions.

Service Discovery
In microservice architectures, services constantly come and go.

Imagine:

Payment Service
Order Service
Notification Service

How does the Order Service know where the Payment Service is running?
ZooKeeper can maintain a registry of available services.

When a service starts:

Register Service

When it stops:

Deregister Service

Other services can query ZooKeeper to discover active instances.
This process is called Service Discovery.

Cluster Membership
ZooKeeper continuously tracks active servers.

For example:

Server A ✅
Server B ✅
Server C ✅

If Server B crashes:

Server B ❌

ZooKeeper immediately updates cluster membership information.
Other servers become aware of the change and adjust accordingly.
This is critical for maintaining system stability.

How ZooKeeper Stores Data

ZooKeeper stores information in a hierarchical structure called a ZNode Tree.
It looks similar to a file system.

/
├── services
│   ├── payment
│   └── order
│
├── leaders
│   └── appLeader
│
└── locks
    └── inventoryLock

Each node is called a ZNode.

A ZNode can store:

Metadata
Configuration
Leader information
Lock ownership details
Service registration data

Ephemeral Nodes: The Secret Sauce
ZooKeeper provides a special type of node called an Ephemeral Node.
These nodes exist only while the client connection remains active.

Example:

Server A registers

ZooKeeper creates:

/servers/serverA

If Server A crashes:

Connection Lost

ZooKeeper automatically removes:

/servers/serverA

This makes failure detection extremely reliable.

Watches: Real-Time Notifications
Instead of continuously polling ZooKeeper:

Are there changes?
Are there changes?
Are there changes?

Applications can register watches.

Example:

Watch Leader Node

If leadership changes:

Leader Changed

ZooKeeper immediately notifies interested clients.
This makes coordination efficient and responsive.

Real-World Systems That Use ZooKeeper
Many large-scale distributed systems have relied on ZooKeeper, including:

Historically, Kafka used ZooKeeper heavily for broker coordination and leader election before moving toward KRaft mode in newer versions.

Advantages of ZooKeeper

Reliable Coordination
Provides a consistent source of truth for distributed systems.
Automatic Leader Election
Handles leader selection and failover automatically.
Distributed Locking

Prevents conflicting operations across multiple servers.

Service Discovery

Helps services find each other dynamically.

Fault Tolerance

Continues functioning even when some nodes fail.

Limitations of ZooKeeper
ZooKeeper is powerful, but it's not a database.

Avoid using it for:

Large data storage
Heavy write workloads
Analytics
Transactional business data ZooKeeper is designed for coordination metadata, not application data.

A Simple Analogy
Imagine a group of friends planning a road trip.

Without coordination:

Everyone books hotels.
Everyone chooses routes.
Everyone buys tickets.

Chaos.
Now imagine one coordinator managing everything.

Who drives?
Who books hotels?
Who handles payments?

Everyone follows the same plan.
ZooKeeper plays that coordinator role inside distributed systems.

Final Thoughts

When people first learn distributed systems, concepts like load balancing, replication, fault tolerance, and microservices seem exciting.

But behind all of these lies a less glamorous challenge: coordination.

Distributed systems fail not because servers are weak, but because multiple servers struggle to agree on what should happen next.

Apache ZooKeeper solves exactly that problem.

It provides a reliable mechanism for leader election, distributed locking, service discovery, cluster membership tracking, and synchronization across nodes. Instead of every server making its own decisions, ZooKeeper ensures the entire system operates with a shared understanding of reality.

The next time you hear someone mention leader election or distributed coordination in a system design interview, you'll know why ZooKeeper has been one of the most important building blocks in distributed computing for over a decade.

How File Upload Works at Scale?

Jatin Gupta — Sat, 13 Jun 2026 08:56:12 +0000

Ever wondered what actually happens after you click the "Upload" button?
You select a file, and within seconds it appears in Google Drive or Amazon S3. But behind that simple button is a highly optimized distributed system designed to handle millions of uploads every day.

The System Design Behind Google Drive & Amazon S3 Uploads

The Naive Approach ❌
A beginner might think the process is simply:

Client
    |
    |
Upload File
    |
    v
Server
    |
    |
Store File
    |
    v
Storage

Seems easy...
But imagine:

1 GB video
10 million users
Slow internet
Network failures
Server crashes This architecture would fail quickly.

Problems:

Server bandwidth becomes bottleneck
High CPU usage
Upload restarts if connection drops
Difficult to scale So companies use a much smarter architecture.

Step 1: User Selects a File
When you choose a file,
the client immediately gathers metadata:

{
  filename: "vacation.mp4",
  size: 2.1 GB,
  type: "video/mp4"
}

Notice:
The actual file is not uploaded yet.
Only metadata is prepared.

Step 2: Client Sends Metadata to API

Client
      |
      | filename
      | size
      | contentType
      |
      v
API Gateway

The API validates:

User authentication
Storage quota
File type
Permissions If everything is valid, it proceeds.

Step 3: Backend Generates a Pre-Signed URL
Instead of sending the file through the application server,
the backend requests a secure upload URL.

                Backend

                     |
                     |
          Generate Upload URL
                     |
                     |
                     v

                  Amazon S3

Example:

https://bucket.s3.amazonaws.com/file123
?signature=abcxyz
&expires=600

This is called a Pre-Signed URL.
It is:

Temporary
Secure
Limited permission
Expires automatically

Why Use Pre-Signed URLs?
Without it:

Client

     |

Application Server

     |

Storage

Every byte passes through your server.
Bad idea.

With pre-signed URLs:

Client

     |---------------------->

              Storage

The application server only handles authorization.
The heavy file upload goes directly to cloud storage.

Benefits:

Less server load
Lower cost
Better scalability
Faster uploads

Step 4: Client Uploads Directly to S3
Now the client uploads directly:

Client
      |
      |
      | 2 GB file
      |
      |
      v
Amazon S3

The backend is no longer in the data path.
This is exactly why systems can support millions of users simultaneously.

Step 5: Storage Returns Success
After upload:

S3

  |

200 OK

  |

Client

The client now knows the upload succeeded.

Step 6: Metadata is Saved
The application server stores information like:

Files Table

-----------------------------------

id

userId

filename

storageKey

size

mimeType

createdAt

updatedAt

-----------------------------------

Notice:
The database stores metadata, not the actual file.
The actual file remains inside object storage.

Final Architecture

                Metadata

Client -------------------->

                     API

                      |

                      |

          Generate Pre-Signed URL

                      |

                      |

                      v

                    S3

                      ^

                      |

                      |

Client -----------------------> Upload File


                      |

                      |

               Save Metadata

                      |

                      v

                  Database

What Happens if Internet Disconnects?
Suppose:

Uploading...

███████████░░░░░░░░
         60%

Internet goes off.
Without special handling:

Start Again ❌

Uploading a 5 GB file again is frustrating.
Modern systems avoid this using Resumable Uploads.

Resumable Upload

Instead of one huge file,
the client divides it into chunks.
Example:

File

|

|

-----------------------------

Chunk 1

Chunk 2

Chunk 3

Chunk 4

Chunk 5

-----------------------------

Maybe:

20 MB each

Upload Process

Chunk 1 ✅

Chunk 2 ✅

Chunk 3 ✅

Chunk 4 ❌

Chunk 5 ❌

Connection lost.

Later:

Reconnect

|

|

Resume

|

|

Chunk 4 ✅

Chunk 5 ✅

Only missing chunks are uploaded.
Huge bandwidth savings.

Multipart Upload in Amazon S3
Amazon S3 supports Multipart Upload:

Initialize Upload

        |

        |

Upload Part 1

Upload Part 2

Upload Part 3

Upload Part 4

        |

        |

Complete Upload

Internally, S3 assembles all parts into a single object.

Advantages:

Retry individual parts
Parallel uploads
Better reliability
Faster performance

Parallel Upload
Instead of:

Chunk1

↓

Chunk2

↓

Chunk3

↓

Chunk4

Systems do:

Chunk1 ----->

Chunk2 ----->

Chunk3 ----->

Chunk4 ----->

All at once.
This significantly reduces upload time.

What About Very Large Files?

For files like:

10 GB
20 GB
100 GB

Systems use:
Chunking
Multipart upload
Retry logic
Checksum verification
Background processing

This ensures reliability even over unstable networks.

How Sync Works Across Multiple Devices

Suppose you upload from your laptop.

Laptop

      |

      |

Cloud Storage

     /   \

    /     \

Phone    Tablet

When the upload completes:

Metadata is updated
Sync service detects changes
Other devices receive notifications
Only changed files are downloaded That's why your phone quickly shows the new file without manually refreshing.

Why Don't Companies Store Files in Databases?
Imagine storing a 2 GB video directly inside MySQL or PostgreSQL.

Problems:

Massive database growth
Slow backups
Expensive replication
Poor performance

Instead:

Database

↓

Stores:

- filename
- owner
- path
- size
- permissions

Object Storage

↓

Stores:

Actual binary file

This separation makes systems scalable and easier to maintain.

Real Production Flow

                    User

                      |

                      |

                Select File

                      |

                      |

             Send Metadata

                      |

                      v

               API Gateway

                      |

          Authentication

                      |

      Generate Pre-Signed URL

                      |

                      v

                Object Storage

          <--------------------

             Direct Upload

                      |

                      |

             Upload Success

                      |

                      |

             Save Metadata

                      |

                      v

                  Database

                      |

                      |

             Notify Sync Service

                      |

          -----------------------

          |                     |

       Laptop               Mobile

          |                     |

       Synced ✅             Synced ✅

Interview Questions

Q1. Why shouldn't files pass through the application server?

Because it creates a bandwidth bottleneck, increases server cost, and limits scalability. Direct uploads to object storage are more efficient.

Q2. What is a Pre-Signed URL?

A temporary, secure URL generated by the backend that allows a client to upload directly to object storage without exposing permanent credentials.

Q3. Why store metadata in a database instead of the file itself?

Databases are optimized for structured data and queries, while object storage is optimized for storing large binary files reliably and cost-effectively.

Q4. What is Multipart Upload?

It splits a large file into multiple parts that can be uploaded independently and then combined by the storage service into one object.

Q5. What is Resumable Upload?

A mechanism where interrupted uploads continue from the last successfully uploaded chunk instead of restarting from zero.

Have you ever wondered how Zoom works?

Jatin Gupta — Fri, 12 Jun 2026 03:22:08 +0000

If only two people are talking, does Zoom send the video?
And if 100 people join a meeting, how can everyone see each other without your laptop sending 99 separate videos?
Let's understand the complete architecture (Peer-to-Peer (P2P), WebRTC, and SFU) in a simple way.

The Basic Idea

Whenever you join a Zoom meeting, there are actually multiple types of data traveling over the internet:

🎤 Audio
🎥 Video
💬 Chat messages
📄 Live transcript
🖥️ Screen sharing

These are called Real-Time Media Streams because they need to reach everyone almost instantly.

You
   │
   ├── Audio
   ├── Video
   ├── Chat
   ├── Screen Share
   └── Transcript
        │
        ▼
      Zoom
        │
        ▼
Other Participants

What happens in a 1-on-1 call?

Suppose:

You
Your friend

Only two participants exist.
In this case, Zoom can establish a Peer-to-Peer (P2P) connection.

You  ------------------> Friend
        Direct Connection

Your laptop sends one video stream directly to your friend's laptop.
Similarly,

Friend -------------> You

One stream each.
No complicated routing is required.

How do two devices connect?
They usually use WebRTC (Web Real-Time Communication).

WebRTC allows browsers and applications to exchange:

Video
Audio
Data in real time. However, before sending media, both devices need to discover each other.

This process is called Signaling.

Example:

You
   │
Signaling
   │
STUN Server
   │
Signaling
   │
Friend

After exchanging connection information, media can flow directly.

You  ==========> Friend
        Video

Friend ========> You
        Video

What is a STUN Server?
A STUN Server helps devices discover their public IP address and determine how they can communicate through NAT (Network Address Translation).
Think of it like this:

You ask,

"What address does the outside world see for me?"

The STUN server replies with that information so the two peers can attempt a direct connection.

Why P2P is great.

Advantages:

Very low latency
Less server cost
Fast communication
Direct transmission

You ---> Friend

Simple and efficient.

But what happens when more people join?
Imagine:

You
Person B
Person C

Now you must send your video to both people.

You now upload 2 streams.

Suppose there are 10 participants.

        You
      / / | \ \
     A B C D E F G H I

Now your laptop must upload 9 different video streams.
That is a huge problem.

Why is this impossible?

Because your:

Internet upload bandwidth is limited
CPU is limited
Memory is limited
Battery is limited If each stream is 2 Mbps,

For 10 people:

2 Mbps × 9

= 18 Mbps upload

Most users cannot continuously upload this much.
If 100 participants exist:

2 Mbps × 99

≈198 Mbps upload

Practically impossible.

So how does Zoom solve this?
Instead of sending separate streams to everyone,
You upload only one stream.

You
   │
   │
   ▼
 Zoom Server

The Zoom server then distributes that stream to everyone else.

             You
              │
              ▼
        Zoom Server
        /    |     \
       /     |      \
      B      C       D

Your laptop uploads only once.
The server does the rest.

This server is called SFU
SFU stands for
Selective Forwarding Unit

It is a media server.
Its job is simple:

Receive your stream
Forward it to other participants It does not create new videos. It simply forwards them intelligently.

How SFU works
Step 1

You upload one stream.

You
   │
   ▼
 SFU

Step 2

SFU receives it.

You
   │
   ▼
 SFU

Step 3

SFU forwards it.

        SFU
      /  |  \
     /   |   \
    B    C    D

Now each participant does the same.

Everyone uploads only one stream.
The SFU distributes everything.

Why is it called "Selective" Forwarding?
Because it forwards only what is needed.
Example:

If your screen shows only 9 participants,
The SFU may send only those visible videos in high quality.
If someone is hidden or inactive,
The server can reduce quality or stop forwarding temporarily.

This saves:

Bandwidth
CPU
Battery

Does SFU mix videos together?
No.
It simply forwards.

You ---> SFU ---> B

You ---> SFU ---> C

You ---> SFU ---> D

No video mixing happens.

Then what is MCU?
Another architecture is MCU (Multipoint Control Unit).
MCU receives all streams,

A
 \
  \
B ---> MCU
  /
 /
C

It decodes everything,
mixes videos,
creates one combined video,
then sends that back.

        MCU
          │
          ▼
     Combined Video
          │
          ▼
     All Participants

SFU vs MCU

| SFU                              | MCU                              |
| -------------------------------- | -------------------------------- |
| Simply forwards streams          | Mixes streams                    |
| Faster                           | More processing                  |
| Lower latency                    | Higher latency                   |
| Less CPU on server               | Heavy server computation         |
| Scales well                      | More expensive                   |
| Used by many modern meeting apps | Used in some specialized systems |

Why Zoom uses SFU
Because it allows:

Millions of users
Lower latency
Better scalability
Lower upload requirements
Efficient bandwidth usage Instead of every laptop sending many streams,

❌ You → B
❌ You → C
❌ You → D
❌ You → E

it becomes

✅ You
      │
      ▼
     SFU
   / / | \ \
  B C D E F

Only one upload from your device.

Complete Flow

User

   │

Capture Camera

   │

Encode Video

   │

Upload One Stream

   │

──────────────

       SFU

──────────────

Receives Stream

Forwards Stream

──────────────

Participants

B

C

D

E

F

Final Takeaway

1-on-1 calls can often use Peer-to-Peer (P2P) with WebRTC, where devices communicate directly after signaling and STUN-based connection setup.
Group calls cannot rely on P2P because each participant would need to upload many separate streams, quickly exhausting bandwidth and device resources.
To solve this, Zoom uses an SFU (Selective Forwarding Unit). Each participant uploads one media stream to the SFU, and the SFU intelligently forwards that stream to the other participants.
This architecture keeps latency low, reduces upload bandwidth requirements, and allows Zoom to scale to meetings with many participants efficiently.

In one line:
"P2P works well for 1-to-1 calls, but for group meetings Zoom scales by using an SFU, where you upload one stream and the server forwards it to everyone else."

How URL Shorteners Generate Unique Links Instantly

Jatin Gupta — Thu, 11 Jun 2026 08:41:19 +0000

Every day, billions of links are shared through services like Bitly, TinyURL, and many custom URL shorteners.
But have you ever wondered:
How can they generate unique short URLs instantly, even when millions of users are creating links at the same time?
Let's understand the complete system step by step-

What is a URL Shortener?
A URL shortener converts a long URL into a shorter and easier-to-share URL.
Example
Long URL:

https://www.example.com/blog/how-url-shorteners-work-in-distributed-systems

Short URL:

https://short.ly/g8

When someone opens short.ly/g8, the service redirects them to the original long URL.

The Main Challenge
Imagine:

Millions of users creating links
Thousands of requests every second
No duplicate short URLs allowed

The system must:

✅ Generate links quickly

✅ Never create duplicates

✅ Scale to millions of users

✅ Keep URLs short

Approach 1: Using URL Prefix
One simple method is:
Take the first few characters of the original URL.

Example:

https://linkedin.com/jobs
→ link

Problem:
Many URLs start with the same letters.

https://linkedin.com/jobs
https://linkedin.com/feed
https://linkedin.com/profile

All become:

link

This creates collisions.
Therefore, real-world systems don't use this approach.

Approach 2: Unique Counter + Base62 Encoding
This is one of the most common approaches used in large systems.
The process looks like:

User submits URL
        ↓
Generate unique ID
        ↓
Convert ID to Base62
        ↓
Store mapping
        ↓
Return short URL

Step 1: User Sends Long URL
Example:

https://www.linkedin.com/posts/example-post

The request reaches the backend server.

Step 2: Generate a Unique Number
Instead of generating random strings, the system first generates a unique number.
Example:

Every new URL gets the next number.

Why Not Use Database Auto Increment?
Databases can generate IDs, but under very high traffic:

Database becomes a bottleneck
Slower performance
Increased load Therefore, many systems use Redis.

What is Redis?
Redis is an extremely fast in-memory database.
It stores data in RAM instead of disk.
Because RAM is much faster than disk:

Database → milliseconds
Redis → microseconds

This makes Redis perfect for counters.

Redis INCR Command
Redis provides a command called:

INCR

It increases a number by 1.

Example:

Counter = 1000

Request 1:
INCR
→ 1001

Request 2:
INCR
→ 1002

Request 3:
INCR
→ 1003

Why Redis INCR Is Special
Redis operations are atomic.
Atomic means:

Only one operation happens at a time.

Even if:

1 million requests arrive simultaneously

Redis guarantees:

No duplicates.
No race conditions.
No conflicts.
This is why Redis is commonly used in URL shortening systems.

What Is a Race Condition?
Imagine two users create a URL at exactly the same moment.
Without atomic operations:

User A gets ID 1001
User B gets ID 1001

Now both URLs have the same short code.
System breaks.
Redis prevents this problem.

Step 3: Convert Number to Base62
Now we have:

Using this directly would make URLs long.
So we convert it into Base62.

What Is Base62?
Base62 uses:

a-z  → 26 characters
A-Z  → 26 characters
0-9  → 10 characters

Total:

26 + 26 + 10 = 62 characters

Character set:

abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789

Why Base62?
Because it is:

✅ URL friendly

✅ Compact

✅ Easy to encode

✅ No special characters

Example Base62 Conversion
Suppose:

1000 → g8
1001 → g9
1002 → ga

Now:

https://short.ly/g8

Is much shorter than:

https://short.ly/1000

How Base62 Conversion Works
Think of it like converting:

Decimal → Binary

But instead:

Decimal → Base62

Example:

1000 ÷ 62

Repeated division generates characters from the Base62 character set.
The resulting string becomes the short code.

How Many URLs Can Base62 Create?

Formula:

62ⁿ

Where:

62 = available characters
n = length of short code

1 Character

62 combinations

2 Characters

62² = 3,844

3 Characters

62³ = 238,328

6 Characters

626ⁿ

Result:

56,800,235,584

Over 56 billion URLs.

Step 4: Store Mapping
The system saves:
Short Code

| Short Code | Original URL                                                           |
| ---------- | ---------------------------------------------------------------------- |
| g8         | [https://example.com/very-long-url](https://example.com/very-long-url) |
| g9         | [https://linkedin.com/post](https://linkedin.com/post)                 |
| ga         | [https://youtube.com/watch?v=abc](https://youtube.com/watch?v=abc)     |

Database record:

{
  "id": 1000,
  "shortCode": "g8",
  "originalUrl": "https://example.com/very-long-url",
  "createdAt": "2026-06-08",
  "expiryDate": null
}

Step 5: Return Short URL
Backend returns:

https://short.ly/g8

The user can now share this URL anywhere.

Step 6: Redirection Process
When someone visits:

https://short.ly/g8

The backend:

Reads g8
Finds corresponding URL
Returns HTTP redirect

Example:

g8
↓
Database lookup
↓
Original URL found
↓
302 Redirect
↓
User reaches destination

Why Not Use Random Strings?

Example:

a8KzP1

Random generation causes problems:

Collision checks required
More database queries
Slower performance

Example:

Generated:
abc123

Already exists?

Need extra lookup.
Counter + Base62 avoids this completely.

System Design Used by Large Companies

A simplified architecture:

Client
  ↓
Load Balancer
  ↓
Application Servers
  ↓
Redis (INCR Counter)
  ↓
Database

Redis
Generates unique IDs.

Application Server

Converts IDs to Base62.

Database

Stores URL mappings.

Load Balancer

Distributes traffic among servers.

Interview Question: Why Base62 Instead of Base64?
Many people confuse Base62 and Base64.

Base64 Characters

A-Z
a-z
0-9
+
/

Problem:

+
/
=

Special characters are not ideal inside URLs.

Example:

ab+c/

Needs encoding.

Base62 Characters

a-z
A-Z
0-9

No special characters.
Completely URL-friendly.
Therefore URL shorteners usually use Base62, not Base64.

Real-World Optimizations
Large URL shorteners also add:

Caching
Frequently accessed URLs stay in Redis.

Analytics

Track:
Click count
Device type
Browser
Country

Expiration
Links can expire automatically.

Custom Aliases

short.ly/jatin

instead of:

short.ly/g8

Final Summary

A modern URL shortener works like this:

User submits URL
        ↓
Redis INCR generates unique ID
        ↓
ID converted to Base62
        ↓
Mapping stored in database
        ↓
Short URL returned
        ↓
User opens short URL
        ↓
Database lookup
        ↓
Redirect to original URL

Key Takeaways

Redis INCR generates unique IDs atomically.
Atomic operations prevent duplicate URLs.
Base62 makes IDs short and URL-friendly.
6 Base62 characters can generate over 56 billion unique combinations.
This approach is fast, scalable, and widely used in production URL shortening systems.

This is why services like Bitly can create unique short URLs almost instantly, even when handling millions of requests every second. 🚀