DEV Community: Gouranga Das Samrat

Database Scaling

Gouranga Das Samrat — Sun, 07 Jun 2026 02:00:00 +0000

One-liner: Databases are the hardest part to scale — the strategy is: vertical first → read replicas → caching → sharding.

📌 Scaling Ladder

Stage 1: Single DB (< 10K users)
Stage 2: Separate DB server from app server
Stage 3: Add caching (Redis) — reduce DB reads by 80%
Stage 4: Read Replicas — distribute read traffic
Stage 5: Database Sharding — distribute write traffic + data
Stage 6: CQRS + Event Sourcing (advanced)

Never jump to Stage 5 without exhausting Stages 3 and 4.

📊 The Read/Write Split

Most web apps: ~90% reads, ~10% writes.

Optimize reads first:
  → Add Redis cache (most impactful)
  → Add read replicas
  → Add CDN for static data

Only then optimize writes:
  → Sharding
  → Async writes (queue + batch)

🗃️ Connection Pooling

Never create a new DB connection per request. Use a pool.

// pg (Node.js PostgreSQL)
const pool = new Pool({
  max: 20, // max 20 connections
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});

const result = await pool.query("SELECT * FROM users WHERE id = $1", [id]);

Without pooling: 1000 req/sec = 1000 new connections = DB crashes.

Consistent Hashing

Gouranga Das Samrat — Sat, 06 Jun 2026 02:00:00 +0000

One-liner: A hashing technique that minimizes the number of keys that need to be remapped when servers are added or removed.

📌 The Problem It Solves

Imagine you have 3 cache servers and you distribute keys using:

server = hash(key) % N  (N = number of servers)

Works fine until N changes:

N=3: hash("user123") % 3 = 1 → Server 1
N=4: hash("user123") % 4 = 3 → Server 3  ← DIFFERENT!

When you add or remove a server, almost ALL keys remap → massive cache miss storm → database gets hammered.

💡 The Solution: Consistent Hashing

How It Works

Map both servers and keys onto the same circular hash ring (0 to 2³²)
Each key is assigned to the first server clockwise from its position
Adding/removing a server only affects adjacent keys

         0
      /     \
  S3         S1
    \       /
         S2
        360°

Key K1 → clockwise → lands on S1
Key K2 → clockwise → lands on S2

Adding a Server

Before:  K1→S1, K2→S1, K3→S2, K4→S3
Add S4 between S1 and S2:
After:   K1→S1, K2→S4, K3→S2, K4→S3  ← Only K2 moved!

Removing a Server

Before: K1→S1, K2→S2, K3→S2, K4→S3
Remove S2:
After:  K1→S1, K2→S3, K3→S3, K4→S3  ← K2 and K3 moved to next server

⚖️ Problem: Uneven Distribution

With few servers, the ring can be unbalanced:

S1 handles 60% of keyspace
S2 handles 30% of keyspace
S3 handles 10% of keyspace

Solution: Virtual Nodes (VNodes)

Instead of 1 point per server, use many virtual points:

S1 → S1_1, S1_2, S1_3 ... S1_100  (spread around ring)
S2 → S2_1, S2_2, S2_3 ... S2_100
S3 → S3_1, S3_2, S3_3 ... S3_100

Result: Much more even distribution. Each physical server maps to ~N/total keys.

🏗️ Implementation Sketch

class ConsistentHash {
  constructor(replicas = 100) {
    this.replicas = replicas; // virtual nodes per server
    this.ring = new Map(); // hash → serverName
    this.sortedKeys = [];
  }

  addServer(server) {
    for (let i = 0; i < this.replicas; i++) {
      const hash = md5(`${server}:${i}`);
      this.ring.set(hash, server);
      this.sortedKeys.push(hash);
    }
    this.sortedKeys.sort();
  }

  getServer(key) {
    const hash = md5(key);
    // Find first server clockwise
    for (const k of this.sortedKeys) {
      if (k >= hash) return this.ring.get(k);
    }
    return this.ring.get(this.sortedKeys[0]); // wrap around
  }
}

📊 Consistent Hashing vs Modulo Hashing

Dimension	Modulo Hashing	Consistent Hashing
Keys remapped on resize	~100%	~1/N
Uneven distribution	Even	Possible (fix with VNodes)
Complexity	O(1)	O(log N) per lookup
Use case	Fixed set of servers	Dynamic server pool

🌍 Real-World Use Cases

System	How it's used
Cassandra	Partitions data across nodes
DynamoDB	Key-based partitioning
Memcached/Redis Cluster	Distributing cache keys
Nginx	Consistent upstream selection
CDN	Routing requests to edge nodes
Kafka	Partition assignment

🎨 Excalidraw Diagram

The diagram shows:

The hash ring as a circle
Server nodes placed on the ring
Keys mapped to nearest server clockwise
Virtual nodes distributed around the ring
Before/after adding a server (minimal key movement)

🔑 Key Takeaways

Modulo hashing causes thundering herd on any server change
Consistent hashing limits disruption to ~1/N keys per change
Virtual nodes (VNodes) solve the uneven distribution problem
This is used everywhere in distributed databases and caches

Back of Envelope Calculations

Gouranga Das Samrat — Fri, 05 Jun 2026 02:00:00 +0000

One-liner: Rough estimates done quickly to understand the scale of a system before designing it. They tell you what tier of infrastructure you need.

📌 Why This Matters

Before designing ANY system, you need to know:

How many users will use it?
How many requests per second?
How much storage do we need?
How much bandwidth is required?

These estimates shape every architectural decision.

🔢 Numbers Every Engineer Should Know

Time

Unit	Value
1 millisecond	10⁻³ seconds
1 microsecond	10⁻⁶ seconds
1 nanosecond	10⁻⁹ seconds
Seconds in a day	~86,400 ≈ 10⁵
Seconds in a month	~2.5M ≈ 2.5 × 10⁶
Seconds in a year	~31.5M ≈ 3 × 10⁷

Latency Numbers (Approximate)

Operation	Latency
L1 cache reference	1 ns
L2 cache reference	4 ns
RAM reference	100 ns
SSD random read	150 µs
HDD seek	10 ms
Network: same datacenter	500 µs
Network: cross-region (US→EU)	150 ms
Network: cross-continent	200–300 ms

Storage Units

Unit	Size
1 KB	10³ bytes
1 MB	10⁶ bytes
1 GB	10⁹ bytes
1 TB	10¹² bytes
1 PB	10¹⁵ bytes

📐 The Estimation Framework

Step 1: DAU → QPS

Daily Active Users (DAU)    = 10 million
Avg requests per user/day   = 10
Total daily requests        = 100 million

QPS = 100M / 86,400 ≈ 1,200 req/sec
Peak QPS (2-3x average)     ≈ 3,000 req/sec

Step 2: Storage

New posts per day    = 1M
Avg post size        = 1 KB (text) + 500 KB (image)
Daily storage        = 1M × 501 KB ≈ 500 GB/day
Yearly storage       = 500 GB × 365 ≈ 180 TB/year
5-year storage       ≈ 1 PB

Step 3: Bandwidth

Reads per second     = 100K req/sec
Avg response size    = 10 KB
Outbound bandwidth   = 100K × 10 KB = 1 GB/sec

🔬 Real Example: Design Twitter

Assumptions

300M DAU
Each user reads 10 tweets/day
Each user writes 1 tweet/week ≈ 0.14 tweets/day
Avg tweet: 280 chars = 280 bytes ≈ 300 bytes
10% of tweets have an image (~200 KB)

Read QPS

300M users × 10 reads/day = 3B reads/day
QPS = 3B / 86,400 ≈ 35,000 reads/sec
Peak QPS ≈ 100,000 reads/sec

Write QPS

300M × 0.14 = 42M tweets/day
QPS = 42M / 86,400 ≈ 500 writes/sec

Storage per day

Text: 42M × 300 bytes = 12.6 GB
Images: 42M × 10% × 200 KB = 840 GB
Total: ~850 GB/day ≈ 310 TB/year

Bandwidth

Read traffic: 35K req/sec × 1 KB/response ≈ 35 MB/sec = ~300 Gbps

🔬 Real Example: Design WhatsApp

Assumptions

2B DAU
Each user sends 20 messages/day
Avg message: 100 bytes
30% are media messages (500 KB avg)

QPS

2B × 20 = 40B messages/day
QPS = 40B / 86,400 ≈ 460,000 msg/sec
Peak QPS ≈ 1M msg/sec

Storage per day

Text: 40B × 100B = 4 TB
Media: 40B × 30% × 500 KB = 6 PB/day (too high → add retention/compression policy)

💡 Tips for Interviews

Always ask before estimating — "Should I assume 10M or 100M users?"
Round aggressively — 86,400 → 10⁵, 3.14 → 3
Peak = 2-3× average (or 5-10× for viral/event-based systems)
Show your math — interviewers care about the process, not the exact number
Derive storage needs from estimates — don't pull numbers from thin air
1 char = 1 byte (ASCII), 1 char = 2-4 bytes (Unicode)

📋 Estimation Cheat Sheet

1K users  → Single server fine
10K users → Need to think about DB separation
100K users → Load balancer, read replicas
1M users  → Caching layer, CDN, sharding
10M users → Distributed systems, microservices
100M+     → You work at a FAANG (or interview there)

🔑 Key Takeaways

Back-of-envelope is about order of magnitude, not precision
Know the key numbers cold: bytes, seconds in a day, latency tiers
Always separate read QPS from write QPS — they're usually very different
Storage estimations reveal whether you need sharding/archival early

TypeScript 7.0 Beta is Here — and It's Rewritten in Go. Here's What Actually Changed.

Gouranga Das Samrat — Thu, 04 Jun 2026 04:00:00 +0000

I've been watching the TypeScript Go port news since it first leaked, and I'll be honest — when they said "10x faster," I was ready to call marketing spin. Then I ran it on a real codebase.

It's fast. Like, uncomfortably fast. The kind of fast where you assume something must be broken.

TypeScript 7.0 Beta dropped on April 21st, and it's not your usual TypeScript release. This isn't "we added a few new types and fixed some edge cases." The entire compiler was ported from TypeScript (the language) to Go — and then they're calling the result TypeScript 7.0.

Let me break down what this actually means for you.

Wait, Why Go?

First question everyone asks. Why not Rust? Why not WASM?

The honest answer: speed of porting, not peak performance. Go's concurrency model and memory layout happened to align well with how the existing TypeScript compiler was structured. A Rust rewrite would've taken years and needed a ground-up redesign. The Go port took roughly a year, and it's architecturally identical to TypeScript 6.0's type-checking logic — same semantics, same results.

Pragmatic call. I think it was the right one.

How to Try It Right Now

npm install -D @typescript/native-preview@beta

Then run tsgo instead of tsc:

npx tsgo --version
# Version 7.0.0-beta

Note: the package is @typescript/native-preview for now. The stable release will eventually ship as the regular typescript package with the normal tsc command.

For VS Code, grab the TypeScript Native Preview extension. It's not a rough prototype — teams at Bloomberg, Figma, Slack, Google, and others have been running pre-release builds for over a year. It's been solid for months.

The Performance Is the Main Event

The Go rewrite isn't just a compiler curiosity. It unlocks parallelization that the old JS runtime couldn't do:

--checkers controls how many type-checking workers run in parallel (default: 4). Bigger codebases on beefy machines can push this higher. CI runners with limited cores should drop it down.

--builders controls parallel project reference builds — a big deal for monorepos. Combined with --checkers, it multiplies. --checkers 4 --builders 4 = up to 16 type-checkers running simultaneously.

--singleThreaded if you need to debug or benchmark against TypeScript 6 without parallelism noise.

The team reports ~10x speedups on large codebases. Based on what early adopters are saying publicly, that tracks.

Running 7.0 Alongside 6.0 (Without Chaos)

This part tripped me up at first. Here's the clean way to handle it.

There's a new compatibility package: @typescript/typescript6. It exposes a tsc6 entry point, so you can run both without naming collisions.

If you use tools like typescript-eslint that peer-depend on typescript directly, do this in your package.json:

{
  "devDependencies": {
    "typescript": "npm:@typescript/typescript6@^6.0.0"
  }
}

This aliases the typescript package to 6.0, so your existing tooling stays happy while you experiment with tsgo on the side.

Breaking Changes — The Ones That Will Actually Bite You

TypeScript 7.0 adopts 6.0's defaults as hard requirements. A lot of these were opt-in before. Now they're just on, with no escape hatch.

Defaults that changed:

strict is true by default
module defaults to esnext
noUncheckedSideEffectImports is true
types defaults to [] — your @types/node won't be picked up automatically anymore

The rootDir change is subtle but will catch you:

If your tsconfig.json sits outside your src/ folder (pretty common), you need to add this:

{
  "compilerOptions": {
    "rootDir": "./src"
  },
  "include": ["./src"]
}

The types change will also catch you:

{
  "compilerOptions": {
    "types": ["node", "jest"]
  }
}

List every @types package your project needs explicitly. No more implicit globals from everything in your node_modules/@types.

Things that are just gone:

target: es5 — not supported
moduleResolution: node / node10 — use nodenext or bundler
module: amd, umd, systemjs, none — gone
baseUrl — use paths relative to the project root instead
downlevelIteration — gone

If you're still on any of these, migrating to TypeScript 6.0 first will make the 7.0 transition cleaner.

JavaScript Support Got a Full Rethink

This one flew under my radar until I read the details. TypeScript 7.0 rewrites how .js files are handled — moving away from Closure-style JSDoc support toward patterns consistent with .ts files.

Key changes:

@enum is no longer special — you need @typedef instead
@class on a function doesn't make it a constructor anymore — use a class declaration
Closure-style function syntax like function(string): void is dropped — use (s: string) => void
values can't be used where types are expected — use typeof someValue

If you have a pure-JS codebase that relied on TypeScript's JSDoc inference, check the CHANGES.md before upgrading.

What's Still Missing

The beta label is real, but it's not hiding a disaster — it's flagging a few known gaps:

No stable programmatic API yet (that's TypeScript 7.1 at earliest)
--watch mode is less efficient than it'll eventually be
Some VS Code features are still coming: semantics-enhanced highlighting, more granular import commands
Declaration file emit from .js files is still being finished

If you depend on the TypeScript compiler API for custom tooling, wait for 7.1. For everything else — type-checking in CI, editor experience, build times — 7.0 Beta is ready.

My Take

The Go port is a genuinely impressive engineering decision. Not the most glamorous language choice, not the theoretical peak, but it shipped in about a year and already works on multi-million line codebases.

The breaking changes are real, but most of them were coming anyway. TypeScript 6.0 gave everyone a year to see them coming. If you've been keeping up with releases, 7.0 should feel like a minor migration with a huge performance payoff.

If you haven't touched your tsconfig.json in two years and still have moduleResolution: node in there... you have some cleanup to do. But that cleanup was overdue regardless of whether TypeScript 7 existed.

Try it today:

npm install -D @typescript/native-preview@beta
npx tsgo --version

Run it on something real and let the speed be someone else's excuse to finally upgrade the old config.

Found issues? The team wants feedback at the microsoft/typescript-go issue tracker — not the main TypeScript repo. That distinction matters during the beta.

Single Server Setup

Gouranga Das Samrat — Wed, 03 Jun 2026 02:00:00 +0000

One-liner: The simplest possible architecture — one machine running everything. Great for starting out, but a single point of failure.

📌 What Is It?

[Browser] ─── HTTP ──► [Single Server]
                          ├── Web Server (Nginx/Apache)
                          ├── App Code (Node/Python/Java)
                          └── Database (MySQL/PostgreSQL)

One machine handles everything: web serving, business logic, database.

✅ When It's Fine

Prototype / MVP stage
<1,000 daily users
Internal tool / side project
Learning / development

❌ Problems

Problem	Impact
SPOF	Server dies → everything down
No scaling	Can't add more machines
Resource contention	App and DB fight for CPU/RAM
Hard to update	Any deploy = full downtime

🚀 First Step: Separate App & DB

[Browser] → [App Server] → [DB Server]

Now you can scale each independently.

DNS — Domain Name System

Gouranga Das Samrat — Mon, 01 Jun 2026 02:00:00 +0000

One-liner: DNS is the internet's phone book — it translates human-readable domain names (google.com) into machine-readable IP addresses (142.250.195.78).

📌 Why DNS Exists

Computers communicate using IP addresses (numbers), but humans remember names. DNS bridges this gap.

Without DNS, you'd have to type 142.250.195.78 instead of google.com.

🔍 DNS Resolution — Step by Step

You type: www.google.com

Browser Cache — Did I look this up recently? (TTL-based)
OS Cache / hosts file — /etc/hosts on Linux/Mac
Recursive Resolver (your ISP or 8.8.8.8) — The middleman
Root Name Server — Knows who handles .com
TLD Name Server — Knows who handles google.com
Authoritative Name Server — Returns the final IP address
Response cached at each level with a TTL

🗂️ DNS Record Types

Record	Purpose	Example
A	Domain → IPv4 address	`google.com → 142.250.195.78`
AAAA	Domain → IPv6 address	`google.com → 2607:f8b0::...`
CNAME	Alias → another domain	`www.google.com → google.com`
MX	Mail server for domain	`google.com → smtp.google.com`
NS	Name server for domain	`google.com → ns1.google.com`
TXT	Arbitrary text (SPF, verification)	`v=spf1 include:...`
PTR	Reverse lookup (IP → domain)	`142.250.195.78 → google.com`
SOA	Start of Authority — zone metadata	Serial, TTL defaults

⏱️ TTL (Time to Live)

TTL controls how long a DNS record is cached before being re-fetched.

TTL	Use Case
`60s`	Frequent changes (A/B deployments)
`300s`	Default for most records
`86400s` (1 day)	Stable records (MX, NS)

⚠️ Before migrating servers: lower TTL to 60s first, wait for propagation, then change IP.

🌍 DNS Hierarchy

               Root (.)
              /    |    \
          .com   .org   .io
          /
      google.com
      /         \
  www           mail

🔄 DNS Caching Layers

Browser Cache (seconds-minutes)
    ↓ miss
OS / hosts file
    ↓ miss
Recursive Resolver (ISP / 8.8.8.8)  ← caches for TTL
    ↓ miss
Root → TLD → Authoritative

🛡️ DNS Security

Threat	Description	Defense
DNS Spoofing	Attacker poisons cache with fake records	DNSSEC
DNS Hijacking	ISP/government redirects queries	Use encrypted DNS (DoH, DoT)
DDoS on DNS	Flood resolver to take down domains	Anycast DNS (Cloudflare, AWS Route 53)

DNSSEC — Cryptographically signs DNS records
DoH (DNS over HTTPS) — Encrypts DNS queries
DoT (DNS over TLS) — Alternative encryption standard

⚡ DNS in System Design

Load Balancing via DNS

api.myapp.com  →  [DNS Round Robin]
                  → 10.0.0.1
                  → 10.0.0.2
                  → 10.0.0.3

Simple but no health checks — if a server dies, DNS still routes to it until TTL expires.

GeoDNS

Different IPs returned based on user's location:

api.myapp.com from India → 13.235.x.x (Mumbai)
api.myapp.com from US   → 54.144.x.x (Virginia)

Failover via DNS

Change A record to backup server when primary fails. But limited by TTL propagation delay.

🏢 Popular DNS Providers

Provider	Feature
Cloudflare (1.1.1.1)	Fast, privacy-focused, free
Google (8.8.8.8)	Reliable, global
AWS Route 53	Deep AWS integration, health checks
Azure DNS	Azure ecosystem

🔑 Key Takeaways

DNS resolution is hierarchical and cached
Lower TTL before changing server IPs to avoid downtime
DNS alone is a bad load balancer (no health checks)
Modern DNS services (Cloudflare, Route 53) offer much more than just name resolution

PgBouncer: Effectively Managing Your PostgreSQL Connection Pool

Gouranga Das Samrat — Sun, 31 May 2026 13:00:00 +0000

Why HikariCP might not be enough and where PgBouncer steps in.

1. Why are Connections Expensive in PostgreSQL?

PostgreSQL forks a separate process at the operating system level for each new connection. This differs from the thread-based model of other databases like MySQL. Each idle connection consumes approximately 5–10 MB of RAM (with default work_mem; this value can be much higher for active queries).

Consequently, the math is simple:

PostgreSQL’s default max_connections value is 100. While increasing this might work in the short term, it directly increases RAM consumption and complicates PostgreSQL’s shared memory management. The correct way to scale the number of connections is to use fewer connections but use them efficiently — that’s exactly where a connection pooler comes in.

2. What is PgBouncer and How Does it Work?

PgBouncer is a lightweight connection pooling proxy that sits between your application and PostgreSQL. Because it’s written in C, its memory footprint is minimal. It fully supports the PostgreSQL wire protocol, making it indistinguishable from PostgreSQL to applications.

Architecturally, it works like this:

[App instance 1] \
[App instance 2] ---> [PgBouncer :6432] ---> [PostgreSQL :5432]
[App instance 3] /

The application connects to PgBouncer as if it were connecting to PostgreSQL (port 6432). PgBouncer allocates an existing PostgreSQL connection from its pool. When the operation (transaction or session) finishes, the connection is returned to the pool.
Effect: 500 application connections → PgBouncer → only 20 actual PostgreSQL connections. From PostgreSQL’s perspective, there are only 20 clients.

3. PgBouncer Pooling Modes

PgBouncer has three main modes that determine when connections are returned to the pool. Choosing the right mode for your needs is critical:

Session Pooling

Connection Assignment: A database connection is allocated when the client (application) connects.
Return Process: That database connection remains busy until the client completely closes the connection.
Ideal Use Case: Stateful sessions requiring things like LISTEN/NOTIFY or the use of temporary tables.

Transaction Pooling (Recommended)

Connection Assignment: A connection is allocated when a database transaction begins.
Return Process: The connection returns to the pool immediately after the COMMIT or ROLLBACK command runs.
Ideal Use Case: This is the most efficient mode for most web applications and microservices architectures.

Statement Pooling

Connection Assignment: A separate connection is allocated for each individual SQL query (statement).
Return Process: The connection becomes free as soon as the query runs and returns results.
Ideal Use Case: Systems running only with “Autocommit.”
Warning: In this mode, BEGIN…COMMIT blocks (transactions containing multiple queries) are not supported.

⚠️ Transaction Mode Gotcha: Prepared Statements

There’s an important limitation in Transaction mode: server-side prepared statements don’t work. The reason is simple — each transaction can go to a different backend connection, so a statement created with PREPARE on one connection cannot be accessed with EXECUTE on another connection.

Furthermore, SET commands and session variables are not preserved outside the active transaction.

The solution — disable prepared statements at the driver level:

pgx (Go): Add default_query_exec_mode=simple_protocol to the connection string.
JDBC (Java): Set prepareThreshold=0.
Alternatively, you can use Session mode, but multiplexing efficiency will decrease.
Note: The server_reset_query = DISCARD ALL parameter is ineffective in transaction mode — PgBouncer runs this command only in session mode.

4. What’s the Difference with HikariCP?

HikariCP is a widely used in-application connection pool in the Java/Spring Boot ecosystem. It aims for the same goal as PgBouncer — managing connections — but their fundamental approaches differ.

Critical difference: HikariCP keeps a separate pool for each application instance. If you’re running 10 pods and each pod opens 20 connections, 200 connections in total go to PostgreSQL. Because PgBouncer is centralized, these 200 connections are reduced to 20 actual connections in the pool.

5. When is PgBouncer Needed?

If you’re running a single application instance, in-application pooling is mostly sufficient. PgBouncer becomes critical in the following scenarios:

High Number of Application Instances (Kubernetes): When each pod opens its own pool, the max_connections limit is quickly exceeded.
Polyglot Architecture: If Go, Python, and Node.js services use the same DB, a centralized pool is a must.
Serverless (Lambda, Cloud Run): Each invocation tends to open a new connection; PgBouncer absorbs this load.
Error Management: If you see FATAL: too many connections errors, this is an alarm state.

6. Pros and Cons

Pros:

Dramatically reduces the number of connections, lowering RAM load.
Pause mode: Holds connections during maintenance so the application doesn’t get errors.
Language and framework independent.

Cons:

Additional infrastructure component (Maintenance overhead).
Limitations on prepared statements and session variables in Transaction mode.
Requires a redundant setup for HA (High Availability).

7. Conclusion: Should I Use It?

Single application instance?
└── Yes → HikariCP / pgx pool is sufficient.
└── No → How many instances?
├── 3–5 pods → Monitor max_connections.
└── 10+ pods or serverless → PgBouncer is a must.

PgBouncer is a small binary that solves a big problem. When every new service you add to your infrastructure loads directly onto PostgreSQL, your database finds it hard to breathe. Anticipating this problem in advance is always better than managing a crisis in production.

IP & Networking Basics

Gouranga Das Samrat — Sat, 30 May 2026 02:00:00 +0000

One-liner: IP (Internet Protocol) is the addressing system of the internet — every device gets a unique address, and packets are routed between them.

📌 IP Addresses

IPv4

32-bit address → 2³² = ~4.3 billion addresses (running out!)
Format: 192.168.1.1 (four 8-bit octets, 0–255)
Private ranges (not routable on internet):
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16

IPv6

128-bit address → 2¹²⁸ = ~340 undecillion addresses
Format: 2001:0db8:85a3:0000:0000:8a2e:0370:7334
Solves IPv4 exhaustion problem

🔢 Ports

A port is a logical channel on an IP address. One server can run many services on different ports.

Port	Service
22	SSH
25	SMTP (email)
53	DNS
80	HTTP
443	HTTPS
3306	MySQL
5432	PostgreSQL
6379	Redis
27017	MongoDB

Ports 0–1023 = well-known (require root). Ports 1024–65535 = available.

📦 TCP vs UDP

TCP (Transmission Control Protocol)

Connection-oriented — 3-way handshake (SYN, SYN-ACK, ACK)
Reliable — guarantees delivery, ordering, error checking
Slower — overhead of acknowledgments
Use: HTTP, HTTPS, SSH, databases

UDP (User Datagram Protocol)

Connectionless — just fire and forget
Unreliable — no delivery guarantee, no ordering
Fast — no handshake overhead
Use: DNS, video streaming, gaming, VoIP

TCP: SYN ──► SYN-ACK ◄── ACK (connection established, then data)
UDP: DATA ──► (no response needed, no connection)

🏠 NAT (Network Address Translation)

Your router at home has ONE public IP. All your devices share it via NAT.

Device (192.168.1.5:1234) ──► Router ──► Internet (Public IP:5000)
                               NAT table: 192.168.1.5:1234 ↔ PublicIP:5000

This is why IPv4 hasn't completely run out — NAT multiplies addresses.

🔗 OSI Model (7 Layers)

Layer	Name	Example
7	Application	HTTP, DNS, SSH
6	Presentation	TLS, encryption
5	Session	Session management
4	Transport	TCP, UDP
3	Network	IP routing
2	Data Link	Ethernet, MAC
1	Physical	Cables, WiFi signals

For system design: mostly care about L3 (IP), L4 (TCP/UDP), L7 (HTTP).

🔑 Key Takeaways

Every server has an IP + port — that's how clients find it
TCP = reliable but slower; UDP = fast but unreliable
Load balancers can operate at L4 (TCP) or L7 (HTTP)

HTTP & HTTPS

Gouranga Das Samrat — Thu, 28 May 2026 02:00:00 +0000

One-liner: HTTP is the protocol for transferring data on the web; HTTPS adds TLS encryption so that data can't be snooped or tampered with in transit.

📌 HTTP — HyperText Transfer Protocol

HTTP is a stateless, request-response application-layer protocol. Every interaction is independent — the server has no memory of previous requests by default.

Request Structure

GET /api/users/42 HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGci...
Accept: application/json

Response Structure

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: max-age=3600

{ "id": 42, "name": "Rahul" }

🔢 HTTP Methods

Method	Purpose	Body?	Idempotent?
`GET`	Retrieve resource	❌	✅
`POST`	Create resource	✅	❌
`PUT`	Replace resource entirely	✅	✅
`PATCH`	Partially update resource	✅	❌
`DELETE`	Remove resource	❌	✅
`HEAD`	Like GET but no body	❌	✅
`OPTIONS`	Ask what methods are allowed	❌	✅

Idempotent = calling it N times has same effect as calling it once.

📊 HTTP Status Codes

Range	Category	Examples
1xx	Informational	`100 Continue`
2xx	Success	`200 OK`, `201 Created`, `204 No Content`
3xx	Redirection	`301 Moved Permanently`, `304 Not Modified`
4xx	Client Error	`400 Bad Request`, `401 Unauthorized`, `403 Forbidden`, `404 Not Found`, `429 Too Many Requests`
5xx	Server Error	`500 Internal Server Error`, `502 Bad Gateway`, `503 Service Unavailable`

🔄 HTTP Versions

Version	Key Feature	Year
HTTP/1.0	New TCP connection per request	1996
HTTP/1.1	Keep-alive, persistent connections, pipelining	1997
HTTP/2	Multiplexing, header compression (HPACK), binary protocol	2015
HTTP/3	QUIC protocol (UDP-based), 0-RTT, faster handshake	2022

HTTP/1.1 vs HTTP/2 — Multiplexing

HTTP/1.1:  REQ1 ──► | wait | ◄── RES1
           REQ2 ──► | wait | ◄── RES2   (sequential)

HTTP/2:    REQ1 ─┐         ┌─ RES1
           REQ2 ─┤─ wire ──┤─ RES2      (parallel, same connection)
           REQ3 ─┘         └─ RES3

🔐 HTTPS — HTTP over TLS

HTTPS = HTTP + TLS (Transport Layer Security)

What TLS Provides

Property	Meaning
Encryption	Data is unreadable to third parties
Authentication	Server proves it's who it says it is (via certificate)
Integrity	Data can't be tampered with without detection

TLS Handshake (Simplified)

Client                          Server
  │──── ClientHello (TLS ver) ──►│
  │◄─── ServerHello + Cert ──────│
  │── Verify cert with CA ───────│ (checks certificate authority)
  │──── Key Exchange ────────────│
  │◄─── Key Exchange ────────────│
  │   [Symmetric key derived]    │
  │◄════ Encrypted data ═════════│

SSL vs TLS

SSL is deprecated (SSL 2.0, 3.0 — broken)
TLS 1.2 is widely used
TLS 1.3 is modern (faster handshake, better ciphers)
When people say "SSL certificate" they mean TLS certificate

🍪 Cookies & Sessions (Handling Statelessness)

Since HTTP is stateless, we use:

Mechanism	How it works
Cookies	Server sends `Set-Cookie` header; browser sends it back on every request
Sessions	Server stores session data, client stores session ID in cookie
JWT	Stateless token — server signs a token, client stores it (localStorage or cookie)

📡 Long-Polling, SSE, WebSockets

Pattern	Direction	Use Case
HTTP Polling	Client → Server (repeated)	Simple updates, high latency
Long Polling	Client holds connection open	Chat, notifications
SSE (Server-Sent Events)	Server → Client (one way)	Live scores, feeds
WebSocket	Bidirectional	Chat apps, multiplayer games, trading

🔑 Key Takeaways

HTTP is stateless — use cookies/JWT to add state
Always use HTTPS in production (free certs via Let's Encrypt)
HTTP/2 gives huge performance gains via multiplexing
Understand status codes cold — interviewers ask about 401 vs 403, 502 vs 503, etc.

Client-Server Model

Gouranga Das Samrat — Tue, 26 May 2026 02:00:00 +0000

One-liner: A client requests resources/services, and a server responds with them. Every web interaction follows this model.

📌 What Is It?

The client-server model is the foundational architecture of the internet. It defines a relationship between two parties:

Client — the requester (browser, mobile app, CLI tool)
Server — the provider (web server, API server, database server)

This is a request-response cycle. The client always initiates; the server always reacts.

🔁 How a Request Works (Step by Step)

You type google.com in the browser
Browser checks its local DNS cache
If not found → asks OS DNS cache
If not found → asks ISP's DNS resolver
DNS resolver returns the IP address (e.g., 142.250.195.78)
Browser opens a TCP connection to that IP on port 80/443
Browser sends an HTTP request
Server processes the request and sends back an HTTP response
Browser renders the HTML/CSS/JS

🧩 Key Components

Component	Role
Client	Initiates requests (browser, app)
Server	Handles requests, returns responses
Protocol	Rules for communication (HTTP, WebSocket, gRPC)
IP Address	Unique address of the server on the network
Port	Channel on the IP (80=HTTP, 443=HTTPS, 3306=MySQL)

🌐 Types of Clients

Thin Client — Only rendering/display logic (browser)
Thick Client — Has business logic locally (desktop apps)
Mobile Client — App that hits APIs

🔄 Communication Models

Request-Response (HTTP)

Client ──── GET /users ────► Server
Client ◄─── 200 OK + data ── Server

Long Polling

Client ──── GET /updates ──► Server (holds connection)
Server ◄─── responds ONLY when data is ready

WebSockets (Bi-directional)

Client ◄──────────────────► Server
      (real-time, both ways)

🏗️ Single Server Architecture

[User's Browser]
      |
      | HTTP Request
      ▼
[Web + App Server] ──── reads/writes ────► [Database]
      |
      | HTTP Response
      ▼
[User's Browser]

Problems with a single server:

Single Point of Failure (SPOF)
Can't scale horizontally
Web traffic + DB on same machine = resource contention

📊 Real-World Analogy

Think of it like a restaurant:

Client = Customer placing an order
Server = Waiter taking the order
Database = Kitchen preparing the food
Protocol = Language/menu used to communicate

⚠️ Limitations of Simple Client-Server

Problem	Impact
Single server → SPOF	Entire system goes down
No caching	Every request hits DB
Stateful server	Hard to scale horizontally
Tight coupling	Hard to change one part

🔑 Key Takeaways

The client always initiates the connection
HTTP is stateless — server doesn't remember previous requests
Modern systems have many layers between client and actual data
Every "server" is just another computer listening on a port

Turning PostgreSQL Into an Integration Engine

Gouranga Das Samrat — Sun, 24 May 2026 13:00:00 +0000

I wanted to see how far PostgreSQL extensibility could go.

Most People Think PostgreSQL Is Just a Database, but PostgreSQL is actually a programmable system. With extensions and procedural languages, it can do far more than simply store and retrieve data.

Recently I started thinking about a simple idea:

What if PostgreSQL could call REST APIs directly from SQL queries?

Instead of always relying on a backend service to act as an integration layer, some interactions with external systems might happen directly from the database.

This idea led me to run a small technical experiment.

Project Idea

In most architectures today, integrations usually look like this:

Database → Backend Service → External API

The backend service is responsible for calling APIs, processing responses, and storing results in the database.

But PostgreSQL has something many people forget: extensibility.

It supports:

procedural languages (PL/Python, PL/pgSQL, etc.)
native extensions written in C
custom SQL-callable functions

This raises an interesting possibility:

Could PostgreSQL itself perform HTTP requests?

If that is possible, SQL queries could potentially interact with external services during execution.

Technical Possibility

PostgreSQL allows developers to extend the database with custom functions.

These functions can be implemented using:

procedural languages (Python, Perl, etc.)
native extensions written in C

That means we can implement a function capable of performing HTTP requests and expose it directly to SQL.

Conceptually, the goal is to enable something like this:

SELECT http_request(
    'httpbin.org',
    443,
    '/headers',
    'GET',
    NULL,
    '{"Authorization":"Bearer demo-token"}'::jsonb,
    true,
    30
);

This query sends an HTTPS request to an external endpoint and returns the response to the SQL session.

Because headers are passed as JSON, they can also be dynamically constructed inside SQL queries.

3. Experiment & Result

To explore the idea, I implemented two different versions of the HTTP function.

Experiment 1 — PL/Python Implementation

The first version uses PL/Python, one of PostgreSQL’s supported procedural languages.

Using Python inside PostgreSQL makes it relatively easy to perform HTTP requests using common Python libraries.

Example usage:

select http.post('https://httpbin.org/post','{"data":{"a": 1,"b": 2}}');

The query triggers an HTTPS request and the response is returned to the SQL client.

To simplify experimentation, this version was packaged as a Docker image with PostgreSQL and PL/Python preconfigured.

Experiment 2 — Native PostgreSQL extension

The second experiment goes deeper by implementing the HTTP functionality as a native PostgreSQL extension written in C.

Extensions written in C integrate directly with the PostgreSQL engine and offer more control over performance and behavior.

This version explores how HTTP capabilities could potentially be embedded closer to the database layer.

Example usage:

SELECT http_request(
    'httpbin.org',
    443,
    '/headers',
    'GET',
    NULL,
    '{"Authorization":"Bearer demo-token"}'::jsonb,
    true,
    30
);

Development Process

Another interesting aspect of this experiment was the development workflow.

I intentionally used AI-assisted development (or what I like to call vibe coding) while building the project, especially when working with PostgreSQL extension code in C.

AI helped accelerate:

extension scaffolding
exploration of low-level C patterns
rapid prototyping
documentation

It was interesting to see how AI could help speed up systems-level experimentation, not just application development.

Design Considerations

Allowing a database to call external APIs raises several architectural considerations.

Latency

Database queries are expected to be fast and predictable.
External API calls introduce network latency, which can slow down query execution.

Transaction Behavior

If an HTTP request is executed inside a transaction, the transaction might be blocked while waiting for the external service.

Reliability

External APIs can fail, timeout, or become unavailable.
Proper timeout handling and error management are essential.

Security

Allowing outbound HTTP requests from the database may introduce security concerns, especially if unrestricted endpoints are allowed.

Because of these considerations, this approach should not replace backend services in most production architectures.

However, it can still be useful for:

experimentation
rapid integration prototypes
certain data enrichment scenarios

Final Thoughts

This experiment started with a simple question:

Can the database participate directly in integration workflows?

PostgreSQL’s extensibility makes it a surprisingly powerful playground for exploring ideas like this.

And with modern AI-assisted workflows, experimenting with systems-level concepts has become significantly faster.

I’m still exploring how far the idea of database-driven integrations can go.

System Design — Beginner to Advance: Complete Curriculum

Gouranga Das Samrat — Tue, 19 May 2026 02:00:00 +0000

Every topic, every date. Bookmark this and follow along.

If you've ever wondered how Netflix streams to millions, how Twitter handles trends, or how Google serves search results in milliseconds — this series is for you.

This is a structured, week-by-week journey through System Design — from the absolute basics to production-grade distributed systems.

🟢 Beginner — Foundations

#	Topic	Publish Date
1	Client-Server Model	May 26
2	IP & Networking Basics	May 30
3	HTTP & HTTPS	May 28
4	DNS — Domain Name System	Jun 1
5	Single Server Setup	Jun 3
6	Back of Envelope Calculations	Jun 5

🔵 Intermediate — Core Building Blocks

#	Topic	Publish Date
7	Database Scaling	Jun 7
8	Consistent Hashing	Jun 6
9	Database Sharding	Jun 13
10	Read Replicas	Jun 14
11	SQL vs NoSQL	Jun 20
12	API Gateway	Jun 21
13	CDN — Content Delivery Network	Jun 27
14	Load Balancers	Jun 28
15	Reverse Proxy	Jul 4
16	Caching Strategies	Jul 5
17	Redis	Jul 11
18	Message Queues (SQS)	Jul 12
19	Pub-Sub (SNS) & Fan-out Architecture	Jul 18
20	Fan-out Architecture	Jul 19
21	Microservices	Jul 25
22	Containers & Docker	Jul 26

🔴 Advanced — Distributed Systems

#	Topic	Publish Date
23	Data Consistency in Microservices — Saga Pattern	Aug 1
24	API Versioning	Aug 2
25	Service-to-Service Communication — gRPC vs REST	Aug 8
26	Rate Limiting	Aug 9
27	CAP Theorem	Aug 15
28	Bloom Filters	Aug 16
29	HLD: URL Shortener (like bit.ly)	Aug 22
30	HLD: Notification System	Aug 23
31	HLD: Instagram Feed	Aug 29
32	HLD: Twitter / X	Aug 30
33	Authentication & Authorization — JWT & OAuth 2.0	Sep 5
34	DDoS Protection & Rate Limiting Abuse Cases	Sep 6
35	Circuit Breaker Pattern	Sep 12
36	Distributed Locks	Sep 13
37	Idempotency	Sep 19
38	Observability — Logs, Metrics, Tracing	Sep 20
39	Service Discovery	Sep 26
40	Multi-Region & Geo-Distribution	Sep 27
41	Cost Optimization	Oct 3
42	Edge Computing	Oct 4
43	Failure Handling & Resilience	Oct 10
44	API Design — Advanced	Oct 11
45	Cache Invalidation Patterns	Oct 17
46	Distributed Cron Jobs	Oct 18
47	Leader Election	Oct 24
48	Rate Limiter Designs	Oct 25

📅 How to Follow This Series

New posts drop every Saturday & Sunday
Start from the beginning if you're new to System Design
Each post is self-contained but builds on previous ones

Follow to get notified when each post drops.