DEV Community: CodeWithDhanian

SYSTEM DESIGN ROADMAP

CodeWithDhanian — Wed, 09 Sep 2026 21:00:57 +0000

THE JOURNEY FROM WRITING CODE TO DESIGNING SYSTEMS

Software engineering begins with a deceptively simple question:

"How do I make this application work?"

System design begins with a much harder question:

"How do I make this system continue to work when everything around it becomes more complicated?"

A small application can often run on a single server.

The application receives a request.

→ The server processes it.

→ The server talks to a database.

→ The database returns the data.

→ The server sends a response.

At small scale, this can be enough.

But imagine the application suddenly has millions of users.

Now the single server becomes a bottleneck.

You add more servers.

→ Now requests need to be distributed.

You add a load balancer.

→ Now your application servers need to share data.

You introduce a database.

→ The database becomes a bottleneck.

You add replication.

→ Now different copies of data can temporarily disagree.

You add caching.

→ Now you have to think about cache invalidation.

You introduce asynchronous processing.

→ Now you need queues.

You deploy services independently.

→ Now you have distributed systems problems.

You operate across multiple regions.

→ Now network partitions, replication lag, disaster recovery, and global consistency become important.

The architecture evolves because the constraints evolve.

This is the essence of system design.

A system is not designed once.

It evolves.

And an engineer who understands system design can anticipate that evolution rather than simply reacting to it.

WHAT IS SYSTEM DESIGN?

System design is the process of defining how the components of a software system interact to satisfy functional and non-functional requirements.

At a high level, you are answering questions such as:

→ What components does the system need?

→ How do those components communicate?

→ Where does data live?

→ How is data retrieved?

→ How does the system scale?

→ What happens when a component fails?

→ How do we maintain availability?

→ How do we maintain consistency?

→ How do we control latency?

→ How do we protect the system?

→ How do we monitor it?

→ How much will it cost?

→ How do we evolve the architecture as requirements change?

A good architecture is therefore not just a diagram.

It is a collection of decisions.

Every box represents a responsibility.

Every arrow represents communication.

Every database represents a storage decision.

Every cache represents a performance and consistency decision.

Every queue represents an asynchronous processing decision.

Every replica represents a reliability or scalability decision.

Every boundary represents an architectural trade-off.

Once you start thinking this way, architecture diagrams stop being pictures.

They become maps of engineering decisions.

WHY SYSTEM DESIGN MATTERS

System design becomes increasingly important as software grows in scale and complexity.

The architecture that works for 100 users may fail at 100,000 users.

The architecture that works for 100,000 users may require significant changes at 10 million users.

The architecture that works in one geographic region may not be appropriate for a global application.

The architecture that works for a prototype may be completely inappropriate for a financial system where correctness is more important than raw throughput.

This is why system design is not a collection of universal recipes.

Instead, it is a framework for reasoning.

Consider two applications.

Application A is a photo-sharing platform.

Application B is a banking transaction system.

Both may have:

→ APIs

→ databases

→ authentication

→ caching

→ background jobs

But their architectural priorities are different.

The photo-sharing platform may prioritize:

→ massive read scalability

→ media delivery

→ CDN usage

→ asynchronous processing

→ feed generation

→ high availability

The banking system may prioritize:

→ correctness

→ transactional integrity

→ authorization

→ auditability

→ strong consistency

→ fraud detection

The components can look similar.

The priorities are different.

That is why memorizing architecture diagrams is insufficient.

You need to understand why a particular component exists.

THE CORE SYSTEM DESIGN MINDSET

Before learning individual technologies, develop the mindset that connects them.

A strong system designer repeatedly asks five questions.

1. WHAT ARE WE BUILDING?

Understand the functional requirements.

→ What can users do?

→ What data does the system manage?

→ What operations must happen synchronously?

→ What operations can happen asynchronously?

→ What are the most important user flows?

2. HOW MUCH LOAD WILL IT HANDLE?

Estimate scale.

→ Number of users

→ Requests per second

→ Read/write ratio

→ Storage requirements

→ Bandwidth

→ Peak traffic

→ Growth rate

3. WHAT MUST THE SYSTEM GUARANTEE?

Understand non-functional requirements.

→ Availability

→ Latency

→ Durability

→ Consistency

→ Security

→ Reliability

→ Scalability

4. WHAT CAN FAIL?

Assume failure.

→ Server failure

→ Database failure

→ Network failure

→ Cache failure

→ Queue failure

→ Dependency failure

→ Region failure

→ Deployment failure

5. WHAT TRADE-OFFS ARE ACCEPTABLE?

There is almost always a trade-off.

→ Strong consistency vs availability

→ Latency vs durability

→ Simplicity vs flexibility

→ Cost vs performance

→ Centralization vs decentralization

→ Synchronous processing vs asynchronous processing

→ Operational complexity vs scalability

This mindset should become automatic.

STAGE 1: MASTER THE FOUNDATIONS

Do not begin system design by memorizing Kafka, Kubernetes, Redis, or microservices.

Begin with fundamentals.

Advanced architecture is built on simple concepts.

If you do not understand networking, databases, operating systems, APIs, and basic distributed-system behavior, advanced system design will become a collection of vocabulary rather than understanding.

The first stage is therefore about building your mental model.

1. NETWORKING FUNDAMENTALS

Start with networking.

Understand how a request travels from a client to a server.

Study:

→ IP addresses

→ Ports

→ TCP

→ UDP

→ DNS

→ HTTP

→ HTTPS

→ TLS

→ Proxies

→ Reverse proxies

→ Load balancers

→ Network latency

→ Bandwidth

→ Connection management

You should eventually be able to explain what happens when someone enters:

https://example.com

into a browser.

A simplified flow is:

→ Browser checks available information.

→ DNS resolution identifies the destination.

→ A connection is established.

→ TLS negotiates secure communication for HTTPS.

→ The client sends an HTTP request.

→ Infrastructure routes the request.

→ An application server processes it.

→ The application may query caches or databases.

→ The response travels back to the client.

This seemingly simple operation contains many layers.

Understanding those layers makes later architecture decisions easier.

For example:

Why use a CDN?

Because geographic distance and network latency matter.

Why use a reverse proxy?

Because you may want a centralized entry point for routing, TLS termination, security policies, and traffic management.

Why use HTTP/2 or HTTP/3?

Because communication protocols influence connection behavior, multiplexing, latency, and performance.

The system designer must understand the foundation beneath the abstraction.

2. CLIENT-SERVER ARCHITECTURE

Next, understand the client-server model.

The basic pattern is:

CLIENT → NETWORK → SERVER → DATA STORE → SERVER → CLIENT

But production systems become more sophisticated.

You may eventually have:

CLIENT → DNS → CDN → WAF → LOAD BALANCER → API GATEWAY → SERVICES → CACHE → DATABASE

And some requests may additionally go through:

SERVICE → MESSAGE QUEUE → WORKER → DATABASE

Your job is to understand why each layer exists.

Do not add components simply because large companies use them.

Architecture should follow requirements.

3. HTTP AND API FUNDAMENTALS

Learn HTTP deeply.

Understand:

→ Methods

→ Status codes

→ Headers

→ Cookies

→ Sessions

→ Authentication

→ Authorization

→ Idempotency

→ Pagination

→ Rate limiting

→ Caching headers

→ Webhooks

→ Long polling

→ Server-Sent Events

→ WebSockets

Then learn API design.

Study:

→ REST

→ GraphQL

→ gRPC

→ API versioning

→ Resource modeling

→ Error handling

→ Pagination

→ Filtering

→ Sorting

→ Idempotency keys

→ Backward compatibility

The API is the contract between systems.

Poor API design creates long-term architectural problems.

Good API design makes systems easier to evolve.

4. DATABASE FUNDAMENTALS

Once networking and APIs are comfortable, move into databases.

Begin with relational databases.

Understand:

→ Tables

→ Rows

→ Columns

→ Primary keys

→ Foreign keys

→ Constraints

→ Joins

→ Indexes

→ Transactions

→ Isolation levels

→ ACID

→ Query optimization

You should understand not only how SQL works, but how database design affects system performance.

For example:

Suppose your application frequently performs:

GET USER BY EMAIL

Without an appropriate index, the database may need to examine many rows.

With an index:

→ Lookup becomes more efficient.

But indexes are not free.

They consume storage.

They increase write overhead.

They must be maintained.

This is a recurring system-design principle:

Every optimization introduces a cost.

5. DATABASE INDEXING

Indexing deserves special attention.

Learn:

→ B-tree indexes

→ Hash indexes

→ Composite indexes

→ Covering indexes

→ Index selectivity

→ Query execution plans

→ Read/write trade-offs

→ Index maintenance

Do not merely learn:

"Add an index to make queries faster."

Learn to ask:

→ Which queries are slow?

→ What is the access pattern?

→ How selective is the indexed field?

→ What happens to writes?

→ How large will the index become?

→ Does the query actually use the index?

This is the beginning of architectural thinking.

6. DATA MODELING

Learn how to model data based on application behavior.

Understand:

→ Normalization

→ Denormalization

→ Relationships

→ Cardinality

→ Access patterns

→ Entity boundaries

→ Aggregation

A data model should not exist independently of application requirements.

If your system frequently needs a particular view of data, you may eventually choose to denormalize or maintain derived data.

That decision should come from workload characteristics rather than fashion.

STAGE 2: UNDERSTAND SCALABILITY

Once fundamentals are strong, learn scalability.

Scalability is the ability of a system to handle increased workload while maintaining acceptable performance and operational characteristics.

Two fundamental approaches are:

→ Vertical scaling

→ Horizontal scaling

Vertical scaling means giving a machine more resources.

→ More CPU

→ More RAM

→ Faster storage

Horizontal scaling means adding more machines.

→ Server 1

→ Server 2

→ Server 3

→ Server 4

Horizontal scaling becomes increasingly important for large distributed applications because workloads can be distributed across multiple instances.

But horizontal scaling introduces another requirement:

The application should generally avoid depending on local instance state.

This leads to the concept of stateless services.

7. STATELESS VS STATEFUL SYSTEMS

A stateless application server does not rely on local memory or local disk to maintain important user state between requests.

Instead:

→ Session data → shared store

→ Files → object storage

→ Persistent data → database

→ Cache → distributed cache

This allows:

CLIENT → LOAD BALANCER → SERVER A

and later:

CLIENT → LOAD BALANCER → SERVER C

without breaking the user's session.

That is one of the foundations of horizontal scaling.

8. LOAD BALANCING

Learn load balancers deeply.

A load balancer distributes traffic across multiple servers.

Instead of:

CLIENT → SERVER

you get:

CLIENT → LOAD BALANCER → SERVER A

** → SERVER B**

** → SERVER C**

Study:

→ Layer 4 load balancing

→ Layer 7 load balancing

→ Round robin

→ Weighted routing

→ Least connections

→ Health checks

→ Session affinity

→ Failover

→ Traffic distribution

→ Service discovery

A load balancer is not simply a traffic distributor.

It becomes an important reliability boundary.

If one application instance fails:

→ Health check detects the failure.

→ Traffic stops going to the unhealthy instance.

→ Other instances continue serving requests.

This is the beginning of fault-tolerant architecture.

STAGE 3: MASTER CACHING

Caching is one of the most important concepts in system design.

A cache stores frequently accessed data closer to where it is needed.

The basic flow is:

CLIENT → APPLICATION → CACHE

If the data exists:

CACHE → APPLICATION → CLIENT

If it does not:

CACHE MISS → DATABASE → CACHE → APPLICATION → CLIENT

The goal is usually to reduce:

→ Latency

→ Database load

→ Expensive computation

Caching appears everywhere.

You can have:

→ Browser cache

→ CDN cache

→ Reverse-proxy cache

→ Application cache

→ Distributed cache

→ Database buffer cache

Study:

→ Cache-aside

→ Read-through

→ Write-through

→ Write-back

→ TTL

→ Eviction

→ LRU

→ Cache invalidation

→ Cache stampede

→ Hot keys

→ Cache penetration

Caching improves performance, but it introduces consistency complexity.

The moment you store the same information in two places, you have to ask:

"What happens when those copies disagree?"

That question will appear repeatedly throughout system design.

STAGE 4: DATABASE SCALING

Eventually, even a well-designed database may become a bottleneck.

Learn how to scale databases.

Start with:

→ Read replicas

→ Replication

→ Partitioning

→ Sharding

→ Connection pooling

→ Query optimization

→ Materialized views

→ Data archiving

A common architecture is:

APPLICATION → PRIMARY DATABASE

and:

APPLICATION → READ REPLICA 1

** → READ REPLICA 2**

Writes may go to the primary.

Reads may be distributed across replicas.

But replication can introduce lag.

Now the system designer has a trade-off:

→ Scale reads

but potentially accept:

→ Delayed visibility of writes.

Again:

Architecture is trade-offs.

9. SHARDING

Sharding means distributing data across multiple database partitions or machines.

Instead of:

ALL DATA → ONE DATABASE

you can have:

DATA → SHARD 1

DATA → SHARD 2

DATA → SHARD 3

DATA → SHARD 4

A shard key determines where data belongs.

Possible keys might include:

→ User ID

→ Tenant ID

→ Geographic region

→ Hash of an identifier

Choosing a shard key is critical.

A poor key can create:

→ Hot partitions

→ Uneven storage

→ Uneven traffic

→ Difficult migrations

A good shard key distributes workload effectively while supporting important queries.

Learn consistent hashing because partitioning strategies become increasingly important in distributed architectures.

STAGE 5: MASTER MESSAGE QUEUES

Not every operation needs to happen during the user's request.

Imagine a user uploads a video.

The user request might only need to:

→ Validate upload

→ Store the video

→ Create a processing job

Then the heavy work can happen asynchronously.

The architecture becomes:

CLIENT → API → QUEUE

Then:

QUEUE → WORKER → PROCESSING SYSTEM

This separates request handling from background processing.

Study:

→ Message queues

→ Pub/Sub

→ Event-driven architecture

→ Producers

→ Consumers

→ Consumer groups

→ Delivery semantics

→ Ordering

→ Retry mechanisms

→ Dead-letter queues

→ Idempotency

→ Backpressure

→ Event replay

→ Event retention

Asynchronous architecture allows systems to absorb bursts.

Suppose traffic suddenly increases.

Instead of forcing every request to perform expensive work immediately:

REQUEST → QUEUE → WORKER

The queue becomes a buffer.

The system can process work at a sustainable rate.

This is called queue-based load leveling.

But queues introduce their own challenges.

You must understand:

→ Duplicate messages

→ Lost messages

→ Delayed processing

→ Ordering

→ Poison messages

→ Consumer failure

→ Backpressure

Again, every abstraction introduces new failure modes.

STAGE 6: EVENT-DRIVEN ARCHITECTURE

Move beyond simple request-response systems.

In a traditional synchronous architecture:

SERVICE A → SERVICE B → SERVICE C

Service A may wait for Service B.

Service B may wait for Service C.

The entire request path can become fragile.

An event-driven design might look like:

SERVICE A → EVENT BUS

Then:

EVENT BUS → SERVICE B

EVENT BUS → SERVICE C

EVENT BUS → SERVICE D

Now consumers can process events independently.

This enables:

→ Loose coupling

→ Asynchronous processing

→ Independent scaling

→ Event replay

→ Independent evolution of consumers

But event-driven architecture requires strong discipline around:

→ Event schemas

→ Idempotency

→ Ordering

→ Delivery semantics

→ Observability

→ Data ownership

→ Failure recovery

Do not adopt event-driven architecture because it sounds sophisticated.

Use it when asynchronous communication and loose coupling provide meaningful value.

STAGE 7: DISTRIBUTED SYSTEMS

This is where system design becomes significantly deeper.

A distributed system is a system whose components operate across multiple machines and communicate over a network.

The moment you distribute computation, you inherit new problems.

A local function call may look like:

FUNCTION → RESULT

A network call looks more like:

SERVICE A → NETWORK → SERVICE B

But the network can:

→ Delay packets

→ Drop packets

→ Duplicate requests

→ Reorder messages

→ Partition systems

→ Disconnect machines

→ Experience congestion

→ Fail partially

This leads to one of the most important principles in distributed systems:

A network call is never equivalent to a local function call.

You must design for failure.

10. CAP THEOREM

Understand CAP theorem carefully.

In the presence of a network partition, a distributed system must trade off between strong consistency and availability.

The important lesson is not to memorize:

"CAP means choose two."

The deeper lesson is:

Distributed systems force you to make explicit decisions about consistency and availability under failure.

Ask:

→ What happens when nodes cannot communicate?

→ Can the system continue accepting writes?

→ Can users read potentially stale data?

→ Should the system reject requests?

→ What consistency guarantees are required?

Different applications make different choices.

A social feed may tolerate some stale information.

A financial transaction may have far stricter correctness requirements.

Understanding the business requirement is therefore essential.

11. CONSISTENCY MODELS

Study:

→ Strong consistency

→ Eventual consistency

→ Causal consistency

→ Read-your-writes consistency

→ Monotonic reads

→ Session consistency

The key is understanding what users actually observe.

Suppose a user updates their profile.

They immediately refresh.

Should they always see the new profile?

If yes, the architecture needs to provide an appropriate guarantee.

Now imagine a social-media like counter.

Does the number need to be perfectly accurate every millisecond?

Probably not.

This distinction can dramatically change architecture.

12. REPLICATION

Replication means maintaining multiple copies of data.

Common patterns include:

→ Primary-replica

→ Multi-primary

→ Leader-follower

→ Synchronous replication

→ Asynchronous replication

Replication can improve:

→ Availability

→ Read scalability

→ Disaster recovery

But it introduces:

→ Replication lag

→ Conflict resolution

→ Failover complexity

→ Operational overhead

The question is never:

"Should I replicate?"

The better question is:

"Why do I need replication, what guarantee must it provide, and what complexity am I willing to accept?"

STAGE 8: LEARN RELIABILITY ENGINEERING

A system that works under normal conditions is not necessarily a reliable system.

Reliable systems are designed around failure.

Learn:

→ Health checks

→ Timeouts

→ Retries

→ Exponential backoff

→ Jitter

→ Circuit breakers

→ Bulkheads

→ Graceful degradation

→ Failover

→ Redundancy

→ Disaster recovery

→ Backups

→ Recovery procedures

One of the most dangerous mistakes is retrying everything blindly.

Imagine:

SERVICE A → SERVICE B

Service B becomes overloaded.

Service A times out and retries.

Now:

SERVICE A → SERVICE B

Thousands of clients do the same thing.

The retry storm makes the original failure worse.

This is why retries should generally be controlled.

Use:

→ Timeouts

→ Exponential backoff

→ Jitter

→ Maximum retry limits

→ Idempotent operations where appropriate

Reliability engineering is about controlling failure rather than pretending failure will not happen.

13. CIRCUIT BREAKERS

A circuit breaker protects a system from repeatedly calling an unhealthy dependency.

Conceptually:

HEALTHY → REQUESTS ALLOWED

Then repeated failures occur:

FAILURE THRESHOLD REACHED → CIRCUIT OPEN

Further requests are rejected or handled through a fallback.

After a recovery period:

HALF OPEN → TEST REQUEST

If successful:

CLOSED

This prevents cascading failures.

14. RATE LIMITING

Rate limiting protects systems from excessive traffic.

It can control:

→ Requests per user

→ Requests per IP

→ Requests per API key

→ Requests per tenant

→ Requests per service

Common algorithms include:

→ Token bucket

→ Leaky bucket

→ Fixed window

→ Sliding window

Rate limiting is useful for:

→ Abuse prevention

→ Resource protection

→ Fairness

→ API quotas

→ Cost control

A system that accepts unlimited requests is vulnerable to both accidental overload and deliberate abuse.

STAGE 9: OBSERVABILITY

You cannot reliably operate a system you cannot see.

Learn the three fundamental observability signals:

→ Metrics

→ Logs

→ Traces

Metrics answer:

"What is happening?"

Examples:

→ CPU utilization

→ Request rate

→ Error rate

→ Latency

→ Queue depth

Logs answer:

"What happened?"

Traces answer:

"Where did the request spend its time?"

Imagine:

CLIENT → API → SERVICE A → SERVICE B → DATABASE

The request takes three seconds.

Where did the time go?

Without distributed tracing, you may not know.

With tracing, you can identify:

→ 100 ms in API gateway

→ 200 ms in Service A

→ 2.5 seconds in Service B

→ 200 ms in database

Now you know where to investigate.

Observability transforms debugging from guessing into investigation.

STAGE 10: PERFORMANCE ENGINEERING

Do not confuse scalability with performance.

Performance asks:

"How efficiently does the system handle its workload?"

Scalability asks:

"How does the system behave as workload increases?"

Study:

→ Latency

→ Throughput

→ Concurrency

→ CPU utilization

→ Memory utilization

→ Network bandwidth

→ Disk I/O

→ Database performance

→ Cache hit ratio

→ Queue latency

→ Tail latency

Pay particular attention to tail latency.

Average latency can hide serious problems.

Suppose:

→ Average latency = 100 ms

But:

→ p95 = 500 ms

→ p99 = 2 seconds

Your average looks healthy.

Your slowest users are having a terrible experience.

System designers therefore think about distributions, not just averages.

15. BACK-OF-THE-ENVELOPE ESTIMATION

One of the most important skills in system design is estimation.

You should be able to make rough calculations before choosing architecture.

Suppose you have:

→ 10 million daily active users

→ Each user makes 20 requests per day

Then:

10,000,000 × 20 = 200,000,000 requests/day

Convert that into requests per second.

200,000,000 / 86,400 ≈ 2,315 requests/second

Then account for peak traffic.

If peak traffic is several times the average:

→ Average QPS ≈ 2,315

→ Peak QPS may be many thousands

Now architecture decisions become easier.

You can reason about:

→ Number of application servers

→ Database capacity

→ Cache capacity

→ Network bandwidth

→ Queue throughput

→ Storage requirements

→ CDN requirements

You do not need perfect numbers.

You need reasonable orders of magnitude.

STAGE 11: STORAGE ARCHITECTURE

Different data requires different storage systems.

Learn the strengths and weaknesses of:

→ Relational databases

→ Key-value stores

→ Document databases

→ Wide-column databases

→ Graph databases

→ Object storage

→ Search engines

→ Time-series databases

→ Vector databases

Do not ask:

"Which database is the best?"

Ask:

"Which storage model matches this workload?"

For example:

Structured transactional data may fit naturally into a relational database.

Large media files may belong in object storage.

High-volume key-based lookups may fit a key-value system.

Search workloads may benefit from a dedicated search engine.

AI retrieval workloads may require vector representations alongside traditional metadata.

The right storage technology depends on the problem.

STAGE 12: CDN AND EDGE ARCHITECTURE

A CDN distributes content closer to users geographically.

Instead of:

USER IN AFRICA → SERVER IN NORTH AMERICA

you can have:

USER → NEARBY EDGE LOCATION → CACHED CONTENT

This can significantly reduce latency for cacheable content.

Study:

→ CDN caching

→ Cache invalidation

→ Edge locations

→ Origin servers

→ Cache-control headers

→ Geographic routing

→ Edge computing

CDNs are especially valuable for:

→ Images

→ Videos

→ JavaScript

→ CSS

→ Static assets

→ Downloadable files

Large systems frequently move computation and content closer to users where it makes economic and technical sense.

STAGE 13: API GATEWAYS AND SERVICE COMMUNICATION

As systems become larger, you need to understand communication boundaries.

Study:

→ API gateways

→ Service-to-service communication

→ REST

→ gRPC

→ Asynchronous messaging

→ Service discovery

→ Timeouts

→ Retries

→ Authentication between services

→ Request tracing

You should understand when synchronous communication is appropriate and when asynchronous communication is better.

A synchronous request might be:

ORDER SERVICE → PAYMENT SERVICE

The order may require payment confirmation.

An asynchronous operation might be:

ORDER CREATED → EVENT → EMAIL SERVICE

The customer does not necessarily need the email system to complete before the order is accepted.

This distinction is fundamental.

STAGE 14: MICROSERVICES AND MONOLITHS

Do not learn microservices as a universal upgrade from monoliths.

Learn the trade-offs.

A monolith can provide:

→ Simplicity

→ Easy local development

→ Straightforward transactions

→ Simple deployment

→ Low operational overhead

Microservices can provide:

→ Independent deployment

→ Independent scaling

→ Team autonomy

→ Stronger service boundaries

→ Technology isolation

But microservices introduce:

→ Network calls

→ Distributed failures

→ Operational complexity

→ Deployment complexity

→ Observability requirements

→ Data ownership challenges

→ Distributed transactions

→ Service coordination

A useful progression is often:

SIMPLE SYSTEM → MODULAR MONOLITH → SELECTIVE SERVICE EXTRACTION → DISTRIBUTED ARCHITECTURE

Do not distribute a system simply because you can.

Distribute it when the organizational or technical benefits justify the complexity.

STAGE 15: DISTRIBUTED TRANSACTIONS

Once you have multiple services, traditional database transactions become harder.

Suppose:

ORDER SERVICE

must coordinate with:

PAYMENT SERVICE

and:

INVENTORY SERVICE

What happens if payment succeeds but inventory reservation fails?

You need a strategy.

Study:

→ Two-phase commit

→ Saga pattern

→ Compensating transactions

→ Outbox pattern

→ Idempotency

→ Event-driven workflows

The important lesson is that distributed transactions are difficult because there is no single local transaction boundary covering every system.

Instead, you need carefully designed workflows.

STAGE 16: CONSENSUS AND LEADER ELECTION

For advanced distributed systems, study:

→ Consensus

→ Leader election

→ Quorum

→ Raft

→ Paxos

→ Distributed coordination

You do not necessarily need to implement every algorithm from scratch.

But you should understand the problem they solve.

When multiple machines must agree on an important state, coordination becomes necessary.

For example:

→ Who is the leader?

→ Who is allowed to write?

→ What is the committed state?

→ What happens after a leader fails?

This leads to some of the deepest ideas in distributed systems.

STAGE 17: SECURITY ARCHITECTURE

System design is incomplete without security.

Learn:

→ Authentication

→ Authorization

→ OAuth

→ OpenID Connect

→ JWT

→ Sessions

→ TLS

→ Encryption at rest

→ Encryption in transit

→ Secret management

→ Role-based access control

→ API security

→ WAF

→ DDoS protection

→ Network segmentation

→ Audit logging

Security should not be a final layer added after the architecture is complete.

It should influence architecture from the beginning.

For every component ask:

→ Who can access it?

→ What credentials are required?

→ What data does it expose?

→ What happens if credentials are compromised?

→ How is access audited?

→ How are secrets rotated?

Security is part of system design, not a separate topic.

STAGE 18: CLOUD ARCHITECTURE

Once you understand system design fundamentals, map them to cloud infrastructure.

Learn the conceptual categories:

→ Compute

→ Networking

→ Storage

→ Databases

→ Messaging

→ Caching

→ Identity

→ Monitoring

→ Security

→ Containers

→ Orchestration

Cloud providers give you managed building blocks.

But the underlying system-design concepts remain the same.

For example:

A managed load balancer is still a load balancer.

A managed database is still a database.

A managed queue is still a queue.

A managed cache is still a cache.

Do not learn cloud services as isolated products.

Map them to the architectural problem they solve.

STAGE 19: CONTAINERS AND ORCHESTRATION

Learn:

→ Containers

→ Docker

→ Container images

→ Networking

→ Volumes

→ Registries

Then learn orchestration concepts.

→ Scheduling

→ Service discovery

→ Health checks

→ Rolling deployments

→ Autoscaling

→ Resource limits

→ Secrets

→ Configuration

→ Failure recovery

You do not need Kubernetes to understand system design.

But understanding orchestration becomes valuable when managing large distributed workloads.

The deeper concept is:

How do we reliably run many instances of software across a fleet of machines?

STAGE 20: DEPLOYMENT AND CI/CD

Production architecture includes the deployment pipeline.

Study:

→ Continuous integration

→ Continuous delivery

→ Automated testing

→ Infrastructure as code

→ Blue-green deployments

→ Canary releases

→ Rolling deployments

→ Feature flags

→ Rollbacks

Imagine deploying a new version to millions of users.

You do not want:

VERSION 1 → VERSION 2 FOR EVERYONE

without safeguards.

Instead:

VERSION 1

→ Deploy small percentage to Version 2

→ Observe metrics

→ Verify errors

→ Increase traffic

→ Continue monitoring

→ Roll out gradually

This is where system design meets operational engineering.

STAGE 21: DISASTER RECOVERY

A production system must have a plan for catastrophic failure.

Study:

→ Backups

→ Replication

→ Failover

→ Recovery Point Objective

→ Recovery Time Objective

→ Multi-zone architecture

→ Multi-region architecture

→ Disaster recovery testing

A backup that has never been restored is an assumption, not a guarantee.

You need to understand:

How much data can we afford to lose?

and:

How long can we afford to be unavailable?

Those answers directly influence architecture and cost.

STAGE 22: MULTI-REGION ARCHITECTURE

Global systems introduce another layer of complexity.

A simplified architecture might be:

US REGION

→ Application

→ Database

→ Cache

EU REGION

→ Application

→ Database

→ Cache

ASIA REGION

→ Application

→ Database

→ Cache

Now you must solve:

→ Global routing

→ Data replication

→ Failover

→ Regional isolation

→ Consistency

→ Conflict resolution

→ Disaster recovery

→ Data residency

→ Cost

Multi-region architecture should not be treated as a badge of sophistication.

It should exist because the requirements justify it.

STAGE 23: DESIGNING REAL-WORLD SYSTEMS

At this stage, stop studying concepts in isolation.

Start designing complete systems.

Begin with simpler problems.

→ URL shortener

→ Pastebin

→ Rate limiter

→ File storage service

→ Notification service

Then move to intermediate systems.

→ Chat application

→ Social media feed

→ Search engine

→ News feed

→ Video streaming platform

→ Ride-hailing platform

Then advanced systems.

→ Global payment system

→ Distributed analytics platform

→ Global messaging system

→ Multi-region data platform

→ AI-powered search system

→ Recommendation platform

The purpose of case studies is not memorizing their architecture.

The purpose is learning how to reason.

24. DESIGN A URL SHORTENER

A URL shortener looks simple.

User sends:

POST /shorten

The service generates a short identifier.

Then:

GET /abc123

redirects to the original URL.

But now ask:

→ How many URLs?

→ How many redirects?

→ Read/write ratio?

→ How do we generate unique IDs?

→ Do IDs need to be sequential?

→ How do we store mappings?

→ How do we cache popular URLs?

→ How do we handle expiration?

→ How do we prevent abuse?

→ How do we collect analytics?

A simple problem becomes an architecture exercise.

That is the point.

25. DESIGN A SOCIAL MEDIA FEED

A feed introduces another level of complexity.

You have:

→ Users

→ Posts

→ Followers

→ Likes

→ Comments

→ Media

→ Notifications

The difficult question becomes:

How do we construct the feed efficiently?

One approach is:

USER OPENS FEED → QUERY FOLLOWED USERS → FETCH POSTS

This may become expensive for users following thousands of accounts.

Another approach is:

POST CREATED → DISTRIBUTE POST TO FOLLOWER FEEDS

This improves read performance but increases write amplification.

Now you face a trade-off.

WRITE HEAVY

versus

READ HEAVY

Neither is universally correct.

The workload determines the design.

26. DESIGN A CHAT SYSTEM

A chat application introduces:

→ Real-time communication

→ WebSockets

→ Message persistence

→ Presence

→ Delivery status

→ Ordering

→ Offline users

→ Push notifications

→ Message synchronization

A simplified architecture might be:

CLIENT → GATEWAY → CHAT SERVICE → MESSAGE STORE

with:

CHAT SERVICE → MESSAGE QUEUE → NOTIFICATION SERVICE

Then:

NOTIFICATION SERVICE → PUSH PROVIDER → MOBILE DEVICE

Now ask:

→ What happens if the recipient is offline?

→ How do messages get stored?

→ How do we preserve ordering?

→ What happens if a message is delivered twice?

→ How do clients reconnect?

→ How do we synchronize missed messages?

The architecture becomes an exercise in distributed systems.

27. DESIGN A VIDEO STREAMING PLATFORM

Video systems introduce:

→ Large objects

→ Upload pipelines

→ Transcoding

→ Multiple resolutions

→ Object storage

→ CDN

→ Metadata

→ Recommendations

A typical high-level flow could become:

CLIENT → API

CLIENT → OBJECT STORAGE

Then:

UPLOAD → EVENT → TRANSCODING WORKERS

Then:

TRANSCODED VIDEO → OBJECT STORAGE → CDN → USER

This illustrates why asynchronous processing, object storage, and CDNs become critical at scale.

28. DESIGN A RIDE-HAILING SYSTEM

A ride-hailing system introduces:

→ Real-time location

→ Matching

→ Geospatial queries

→ Driver availability

→ Trip state

→ Pricing

→ Notifications

→ Payments

→ Event processing

The architecture may involve:

DRIVER → LOCATION SERVICE

RIDER → REQUEST SERVICE

REQUEST SERVICE → MATCHING ENGINE

MATCHING ENGINE → DRIVER

TRIP SERVICE → EVENT STREAM

EVENT STREAM → NOTIFICATION / PAYMENT / ANALYTICS

Now you have to reason about real-time systems and distributed state.

STAGE 24: AI SYSTEM DESIGN

System design is increasingly expanding into AI-enabled applications.

The fundamentals remain the same.

You still need:

→ APIs

→ Databases

→ Caches

→ Queues

→ Authentication

→ Observability

→ Reliability

But AI introduces additional components.

Study:

→ Model serving

→ Inference APIs

→ Embeddings

→ Vector databases

→ Retrieval-augmented generation

→ RAG pipelines

→ Prompt management

→ AI gateways

→ Model routing

→ Rate limits

→ Token budgets

→ Evaluation

→ Guardrails

→ AI observability

→ Agent architectures

A simplified RAG system may look like:

USER → API → RETRIEVAL → VECTOR DATABASE

Then:

RETRIEVED CONTEXT → LLM → RESPONSE

But production systems require much more.

You must consider:

→ Retrieval latency

→ Token consumption

→ Model availability

→ Prompt injection

→ Data privacy

→ Context limits

→ Hallucination

→ Evaluation

→ Cost

→ Caching

The architecture becomes a combination of traditional distributed systems and AI infrastructure.

The lesson remains the same:

Understand the workload before selecting the architecture.

STAGE 25: SYSTEM DESIGN INTERVIEW FRAMEWORK

Learning system design is valuable for real engineering work, but it is also commonly evaluated through system-design interviews.

Do not approach interviews as architecture trivia.

Use a structured process.

Start with:

REQUIREMENTS → ESTIMATION → API → DATA MODEL → HIGH-LEVEL ARCHITECTURE → DEEP DIVE → BOTTLENECKS → TRADE-OFFS

STEP 1: CLARIFY REQUIREMENTS

Never immediately draw a diagram.

Ask:

→ Who are the users?

→ What are the core operations?

→ What is in scope?

→ What is out of scope?

→ What are the expected scale requirements?

→ What latency is acceptable?

→ What availability is required?

→ What consistency guarantees matter?

This prevents solving the wrong problem.

STEP 2: DEFINE SCALE

Estimate:

→ Users

→ Requests per second

→ Storage

→ Bandwidth

→ Read/write ratio

→ Peak traffic

Do rough calculations.

You are not trying to predict the future perfectly.

You are establishing architectural boundaries.

STEP 3: DESIGN THE API

Define the major operations.

For example:

POST /users

GET /users/{id}

POST /posts

GET /feed

The API forces you to clarify what the system actually needs to do.

STEP 4: DESIGN THE DATA MODEL

Identify:

→ Main entities

→ Relationships

→ Access patterns

→ Indexes

→ Storage choices

Do not blindly normalize everything.

Do not blindly denormalize everything.

Design around actual queries and requirements.

STEP 5: DRAW THE HIGH-LEVEL ARCHITECTURE

Start simple.

For example:

CLIENT

↓

LOAD BALANCER

↓

APPLICATION SERVERS

↓

DATABASE

Then introduce components only when you can explain why they are necessary.

Perhaps:

CLIENT

↓

CDN / WAF

↓

LOAD BALANCER

↓

API SERVERS

↓

CACHE

↓

DATABASE

And:

API SERVERS

↓

MESSAGE QUEUE

↓

WORKERS

The diagram should evolve as the requirements demand it.

STEP 6: IDENTIFY BOTTLENECKS

Ask:

→ What breaks first?

→ Database?

→ CPU?

→ Memory?

→ Network?

→ Cache?

→ Queue?

→ External dependency?

Then solve the bottleneck.

This is far more valuable than adding technologies randomly.

STEP 7: DISCUSS FAILURE MODES

Ask:

→ What if the database fails?

→ What if the cache fails?

→ What if a service times out?

→ What if messages are duplicated?

→ What if the region goes down?

→ What if traffic suddenly increases tenfold?

A strong design explains not only how the system works.

It explains how the system fails.

STEP 8: EXPLAIN TRADE-OFFS

This is where architectural maturity becomes visible.

Say:

"We could use X, but because requirement Y is more important, I would choose Z."

For example:

→ We could use synchronous processing, but asynchronous processing reduces user-facing latency.

→ We could use strong consistency everywhere, but eventual consistency is sufficient for this feature and improves availability.

→ We could shard the database immediately, but that introduces complexity; read replicas may be sufficient at the current scale.

→ We could introduce microservices, but a modular monolith may be operationally simpler until independent scaling becomes necessary.

The ability to explain trade-offs is one of the strongest indicators of system-design maturity.

THE SYSTEM DESIGN LEARNING ORDER

A practical learning sequence looks like this:

FOUNDATIONS

↓

NETWORKING

↓

HTTP & APIs

↓

DATABASES

↓

SCALABILITY

↓

LOAD BALANCING

↓

CACHING

↓

DATABASE SCALING

↓

MESSAGE QUEUES

↓

EVENT-DRIVEN ARCHITECTURE

↓

DISTRIBUTED SYSTEMS

↓

CONSISTENCY

↓

REPLICATION

↓

SHARDING

↓

RELIABILITY

↓

RATE LIMITING

↓

OBSERVABILITY

↓

SECURITY

↓

CLOUD ARCHITECTURE

↓

MICROSERVICES

↓

DISTRIBUTED TRANSACTIONS

↓

MULTI-REGION SYSTEMS

↓

REAL-WORLD CASE STUDIES

↓

AI SYSTEM DESIGN

↓

ARCHITECTURAL MASTERY

Do not rush this sequence.

Each stage gives you vocabulary and mental models required by the next.

THE MOST IMPORTANT SYSTEM DESIGN CONCEPTS TO MASTER

If you want to build a strong foundation, make sure you can explain these concepts without memorizing definitions.

FUNDAMENTALS

→ Scalability

→ Performance

→ Availability

→ Reliability

→ Durability

→ Latency

→ Throughput

→ Concurrency

→ Statelessness

→ Stateful systems

NETWORKING

→ DNS

→ TCP

→ UDP

→ HTTP

→ HTTPS

→ TLS

→ Reverse proxy

→ Load balancing

→ CDN

→ WebSockets

→ SSE

DATABASES

→ SQL

→ NoSQL

→ ACID

→ Transactions

→ Isolation levels

→ Indexes

→ Replication

→ Read replicas

→ Sharding

→ Partitioning

→ Consistency

DISTRIBUTED SYSTEMS

→ CAP theorem

→ Consensus

→ Quorum

→ Leader election

→ Eventual consistency

→ Strong consistency

→ Distributed transactions

→ Saga

→ Idempotency

→ Event sourcing

→ CQRS

PERFORMANCE

→ Caching

→ Cache invalidation

→ Cache eviction

→ Connection pooling

→ Batching

→ Compression

→ CDN

→ Asynchronous processing

RELIABILITY

→ Timeouts

→ Retries

→ Exponential backoff

→ Circuit breakers

→ Bulkheads

→ Failover

→ Replication

→ Disaster recovery

→ Graceful degradation

OPERATIONS

→ Metrics

→ Logs

→ Tracing

→ Monitoring

→ Alerting

→ SLOs

→ SLIs

→ Error budgets

SECURITY

→ Authentication

→ Authorization

→ OAuth

→ Encryption

→ TLS

→ Secrets

→ WAF

→ DDoS protection

→ Access control

ARCHITECTURE

→ Monolith

→ Modular monolith

→ Microservices

→ Event-driven architecture

→ Service-oriented architecture

→ Serverless

→ Multi-region systems

→ Edge architecture

DO NOT MEMORIZE ARCHITECTURE DIAGRAMS

This deserves emphasis.

You can memorize the architecture of a social network.

You can memorize the architecture of a video platform.

You can memorize the architecture of a ride-sharing application.

But memorization will eventually fail.

Someone will change one requirement.

Now your memorized diagram may no longer work.

Instead, learn the reasoning process.

For example:

HIGH READ TRAFFIC

↓

Consider caching

↓

DATABASE READ BOTTLENECK

↓

Consider replicas

↓

DATABASE WRITE BOTTLENECK

↓

Consider partitioning or sharding

↓

EXPENSIVE BACKGROUND WORK

↓

Consider queues and workers

↓

GLOBAL USERS

↓

Consider CDN and geographic distribution

↓

REGIONAL FAILURE

↓

Consider redundancy and failover

↓

STRICT CONSISTENCY

↓

Evaluate appropriate consistency mechanisms

This is system design thinking.

BUILD SYSTEMS, DO NOT ONLY READ ABOUT THEM

Reading gives you concepts.

Building gives you intuition.

Choose a simple application.

Start with:

ONE SERVER → ONE DATABASE

Then evolve it.

Add:

→ Load balancing

→ Multiple application instances

→ Redis

→ Database replicas

→ Message queue

→ Background workers

→ Monitoring

→ Rate limiting

→ Authentication

→ CDN

Then intentionally break it.

Kill an application instance.

Stop the database.

Introduce latency.

Fill the queue.

Remove the cache.

Send a traffic spike.

Now observe what happens.

This is where architecture becomes real.

You begin to understand why reliability patterns exist.

You stop seeing a circuit breaker as a definition.

You see it as a response to a real failure.

You stop seeing caching as a performance buzzword.

You see what happens when the database becomes overloaded.

You stop seeing queues as another technology.

You understand how they decouple workloads.

PRACTICE ARCHITECTURAL TRADE-OFFS

For every major technology, ask:

WHAT PROBLEM DOES THIS SOLVE?

Then ask:

WHAT NEW PROBLEM DOES THIS CREATE?

For caching:

→ Solves database load and latency.

→ Creates invalidation and consistency challenges.

For replication:

→ Improves availability and read scalability.

→ Creates replication lag and failover complexity.

For microservices:

→ Improves service isolation and independent scaling.

→ Creates distributed-system complexity.

For asynchronous processing:

→ Reduces synchronous request work.

→ Creates eventual completion and message-processing complexity.

For sharding:

→ Allows databases to scale horizontally.

→ Creates partitioning and query-routing complexity.

For multi-region deployment:

→ Improves geographic resilience and latency.

→ Creates replication and consistency complexity.

This single habit will dramatically improve your system-design ability.

SYSTEM DESIGN IS A BUSINESS PROBLEM TOO

Architecture does not exist in isolation.

A technically beautiful system can still be a bad business decision.

Suppose a startup has:

→ 10,000 users

→ Moderate traffic

→ A small engineering team

→ Limited budget

Building an extremely complex multi-region microservice architecture may create more problems than it solves.

The system may require:

→ More infrastructure

→ More monitoring

→ More deployment pipelines

→ More operational expertise

→ More failure modes

→ More development time

A simpler architecture may be better.

Now imagine the same company has:

→ Hundreds of millions of users

→ Global traffic

→ Strict availability requirements

→ Multiple engineering organizations

The architecture will likely need substantially more sophisticated infrastructure.

This is why system design is fundamentally about constraints.

FROM JUNIOR ENGINEER TO SYSTEM DESIGNER

The progression often looks like this:

JUNIOR ENGINEER

Thinks primarily about:

→ Writing correct code.

↓

MID-LEVEL ENGINEER

Starts thinking about:

→ Performance

→ Maintainability

→ APIs

→ Database design

↓

SENIOR ENGINEER

Thinks about:

→ Scalability

→ Reliability

→ Architecture

→ Operational impact

→ Team boundaries

↓

STAFF / PRINCIPAL ENGINEER

Thinks about:

→ Organization-wide architecture

→ Long-term evolution

→ Platform capabilities

→ Business constraints

→ Technical strategy

→ Risk

→ Cost

The goal of learning system design is not simply to pass an interview.

It is to move your thinking upward.

From:

"How do I implement this?"

to:

"How should this system evolve?"

A 12-WEEK SYSTEM DESIGN STUDY PLAN

If you want structure, you can organize your learning into twelve weeks.

WEEK 1: FOUNDATIONS

Study:

→ System design principles

→ Scalability

→ Performance

→ Availability

→ Reliability

→ Latency

→ Throughput

→ Capacity estimation

Goal:

Understand the vocabulary of system design.

WEEK 2: NETWORKING

Study:

→ DNS

→ TCP/IP

→ UDP

→ HTTP

→ HTTPS

→ TLS

→ Proxies

→ Load balancing

Goal:

Understand how requests move through a system.

WEEK 3: API DESIGN

Study:

→ REST

→ GraphQL

→ gRPC

→ API versioning

→ Authentication

→ Authorization

→ Pagination

→ Rate limiting

Goal:

Learn how systems expose and consume interfaces.

WEEK 4: DATABASES

Study:

→ SQL

→ NoSQL

→ Data modeling

→ Indexes

→ Transactions

→ ACID

→ Isolation

Goal:

Understand how data should be modeled and stored.

WEEK 5: DATABASE SCALING

Study:

→ Replication

→ Read replicas

→ Partitioning

→ Sharding

→ Consistency

Goal:

Learn how storage systems evolve under increasing load.

WEEK 6: CACHING AND PERFORMANCE

Study:

→ Redis

→ Cache-aside

→ TTL

→ Eviction

→ Cache invalidation

→ Cache stampede

→ CDN

Goal:

Learn how to reduce latency and infrastructure pressure.

WEEK 7: MESSAGE QUEUES

Study:

→ Queues

→ Pub/Sub

→ Producers

→ Consumers

→ Consumer groups

→ Retry mechanisms

→ Dead-letter queues

→ Idempotency

Goal:

Understand asynchronous architectures.

WEEK 8: DISTRIBUTED SYSTEMS

Study:

→ CAP theorem

→ Consistency

→ Replication

→ Quorum

→ Consensus

→ Leader election

Goal:

Understand what changes when computation is distributed.

WEEK 9: RELIABILITY

Study:

→ Timeouts

→ Retries

→ Backoff

→ Circuit breakers

→ Bulkheads

→ Failover

→ Disaster recovery

Goal:

Learn how to design systems that survive failure.

WEEK 10: MICROSERVICES AND CLOUD

Study:

→ Monoliths

→ Modular monoliths

→ Microservices

→ Service discovery

→ API gateways

→ Containers

→ Kubernetes concepts

→ Cloud architecture

Goal:

Understand large application architectures.

WEEK 11: OBSERVABILITY AND SECURITY

Study:

→ Metrics

→ Logging

→ Tracing

→ Monitoring

→ Authentication

→ Authorization

→ Encryption

→ WAF

→ Secrets

Goal:

Understand how production systems are operated and protected.

WEEK 12: REAL-WORLD SYSTEM DESIGN

Design:

→ URL shortener

→ Chat application

→ Social feed

→ Notification system

→ Video streaming platform

→ Search engine

→ Ride-hailing platform

→ Payment system

Goal:

Bring everything together.

HOW TO KNOW WHEN YOU ARE GETTING BETTER

Do not measure progress by the number of articles you have read.

Measure your ability to reason.

At the beginning, you may see:

CLIENT → SERVER → DATABASE

and think that is enough.

Later, you begin asking:

→ What happens when traffic increases?

→ What happens when the database becomes unavailable?

→ What happens when the network becomes slow?

→ How do we handle retries?

→ Can we cache this?

→ Can this work asynchronously?

→ Do we need replication?

→ What consistency is required?

→ What is the expected peak load?

→ What happens during a regional outage?

That progression is real growth.

Eventually, you will be able to look at a system and instinctively identify:

→ Bottlenecks

→ Failure points

→ Scaling limits

→ Data boundaries

→ Communication boundaries

→ Consistency requirements

→ Operational risks

That is when system design starts becoming intuition.

THE FINAL SYSTEM DESIGN MENTAL MODEL

When facing any system-design problem, think in layers.

1. REQUIREMENTS

→ What does the system need to do?

2. USERS AND SCALE

→ Who uses it?

→ How much traffic?

→ How much data?

3. INTERFACES

→ What APIs and communication mechanisms are required?

4. DATA

→ What needs to be stored?

→ Where?

→ How is it accessed?

5. COMPUTE

→ Where does business logic execute?

6. PERFORMANCE

→ Where are the expensive operations?

→ What can be cached?

7. SCALABILITY

→ What happens when traffic increases?

8. RELIABILITY

→ What happens when components fail?

9. CONSISTENCY

→ What guarantees does the system need?

10. SECURITY

→ Who can access what?

11. OBSERVABILITY

→ How will we know the system is healthy?

12. OPERATIONS

→ How do we deploy, monitor, recover, and evolve it?

13. COST

→ Is the architecture economically sustainable?

14. TRADE-OFFS

→ Why did we choose this design over alternatives?

This framework can be applied to almost any system.

THE DEEPEST LESSON OF SYSTEM DESIGN

The most important lesson is simple:

There is no perfect architecture.

There is only an architecture that is appropriate for a particular set of requirements and constraints.

A system optimized for latency may sacrifice cost.

A system optimized for consistency may sacrifice availability during certain failures.

A system optimized for simplicity may sacrifice some scalability.

A system optimized for massive scale may introduce operational complexity.

A system optimized for rapid development may require architectural evolution later.

The engineer's responsibility is not to eliminate trade-offs.

It is to make them consciously.

That is what separates architecture from technology selection.

BUILD YOUR ARCHITECTURAL JUDGMENT

Do not aim to become the person who knows the most technologies.

Aim to become the engineer who can look at a problem and understand what actually matters.

When traffic grows:

→ Know where the bottleneck will appear.

When data grows:

→ Know how storage must evolve.

When latency grows:

→ Know where to measure.

When a dependency fails:

→ Know how the system should respond.

When consistency becomes difficult:

→ Know what guarantees the business actually needs.

When the architecture becomes complicated:

→ Know whether the complexity is justified.

When the system becomes global:

→ Understand the implications of geography.

When AI enters the application:

→ Understand how model behavior, inference cost, retrieval, latency, security, and reliability affect the architecture.

And when someone asks:

"How would you design this system?"

Do not rush to draw boxes.

Start with the problem.

Then follow the reasoning.

REQUIREMENTS → SCALE → DATA → APIs → COMPONENTS → COMMUNICATION → BOTTLENECKS → FAILURE MODES → TRADE-OFFS → EVOLUTION

That sequence is the foundation of strong system design.

THE ROAD AHEAD

System design can feel overwhelming because it connects almost every major area of software engineering.

You cannot truly understand distributed systems without understanding networking.

You cannot understand database scaling without understanding data access patterns.

You cannot understand caching without understanding consistency.

You cannot understand microservices without understanding service boundaries and distributed communication.

You cannot understand reliability without understanding failure.

You cannot understand cloud architecture without understanding the infrastructure underneath it.

And you cannot become a strong system designer by simply consuming more information.

You need to connect the information.

That connection is what creates architectural thinking.

Start with one server.

Understand it.

Then add a second.

Ask what breaks.

Add a load balancer.

Ask what breaks next.

Add a cache.

Ask what becomes inconsistent.

Add a replica.

Ask what happens when replication lags.

Add a queue.

Ask what happens when messages are duplicated.

Add another region.

Ask what happens when regions cannot communicate.

Add millions of users.

Ask where the bottleneck moves.

Keep asking.

That is the practice.

Eventually, system design stops looking like a collection of complicated diagrams.

It becomes a way of thinking.

YOUR SYSTEM DESIGN ROADMAP

The complete progression can be remembered as:

FOUNDATIONS

→ Understand computers, networks, APIs, databases, and basic architecture.

↓

SCALABILITY

→ Learn vertical scaling, horizontal scaling, statelessness, and load balancing.

↓

STORAGE

→ Master SQL, NoSQL, indexing, replication, partitioning, and sharding.

↓

PERFORMANCE

→ Learn caching, CDNs, batching, compression, and latency optimization.

↓

ASYNC SYSTEMS

→ Master queues, Pub/Sub, workers, event-driven architecture, retries, and idempotency.

↓

DISTRIBUTED SYSTEMS

→ Learn consistency, CAP, replication, consensus, quorum, and coordination.

↓

RELIABILITY

→ Learn timeouts, circuit breakers, retries, failover, redundancy, and disaster recovery.

↓

SECURITY

→ Learn authentication, authorization, encryption, secrets, WAF, and network protection.

↓

OBSERVABILITY

→ Learn metrics, logs, traces, monitoring, alerting, and operational debugging.

↓

CLOUD

→ Map architecture concepts to compute, storage, networking, managed databases, messaging, and orchestration.

↓

MICROSERVICES

→ Learn service boundaries, communication, discovery, distributed transactions, and organizational architecture.

↓

GLOBAL SYSTEMS

→ Learn multi-region deployment, geographic routing, replication, disaster recovery, and global consistency.

↓

REAL-WORLD DESIGN

→ Design feeds, chat systems, search engines, video platforms, payment systems, notification systems, and other production architectures.

↓

AI SYSTEM DESIGN

→ Learn inference, RAG, vector search, AI gateways, model routing, evaluation, observability, and AI-specific reliability and security.

↓

ARCHITECTURAL MASTERY

→ Develop the judgment to choose the simplest architecture that satisfies the requirements today while providing a sensible path for tomorrow.

QUICK THOUGHT

The difference between an engineer who can write software and an engineer who can design systems is not simply the number of technologies they know.

It is the depth of their reasoning.

Anyone can learn the name of a database.

A system designer understands the workload that database must handle.

Anyone can draw a load balancer.

A system designer understands why traffic needs to be distributed and what happens when an instance disappears.

Anyone can add a cache.

A system designer understands the relationship between latency, load, freshness, invalidation, memory, and consistency.

Anyone can say "microservices."

A system designer understands when separating services creates value and when it creates unnecessary complexity.

Anyone can draw a distributed architecture.

A system designer understands that every network boundary introduces another potential failure.

That is why system design is worth learning.

It teaches you to think beyond the code.

You begin to see software as a living system:

REQUESTS → COMPUTATION → DATA → NETWORKS → PEOPLE → FAILURE → SCALE → RECOVERY → EVOLUTION

And once you understand that system, you are no longer simply implementing features.

You are designing the future of the software.

Start small.

Understand deeply.

Build.

Measure.

Break things.

Study the failures.

Redesign.

Scale.

Repeat.

That is how system designers are made.

GRAB THE SYSTEM DESIGN EBOOK

If you want to go deeper and turn this roadmap into a structured system-design learning journey, grab the System Design ebook here:

GET THE SYSTEM DESIGN EBOOK

Use the roadmap as your map.

Use the ebook as your deeper study resource.

Then take what you learn and build real systems.

LEARN → DESIGN → BUILD → BREAK → MEASURE → IMPROVE → REPEAT

That is the path from understanding system design to actually becoming a system designer.

1000 Linux Shortcuts to Master to Become a Pro

CodeWithDhanian — Wed, 09 Sep 2026 09:39:16 +0000

Linux is one of the most powerful operating systems for developers, DevOps engineers, system administrators, cybersecurity professionals, cloud engineers, and power users. Its real strength becomes visible when you stop relying on graphical interfaces and start working confidently from the command line.

The terminal allows you to navigate files, manage processes, inspect hardware, configure networks, control services, automate repetitive tasks, work with containers, troubleshoot systems, and manage remote servers.

This guide contains 1,000 Linux shortcuts, commands, shell techniques, and productivity tricks designed to take you from basic terminal usage to professional Linux administration.

1. Master the Linux Terminal First

Before learning hundreds of commands, become comfortable controlling the terminal itself. Keyboard shortcuts can save enormous amounts of time because you don't need to repeatedly move your cursor or retype commands.

For example, Ctrl + C stops a running command, Ctrl + L clears the screen, and Ctrl + R searches your command history.

Essential terminal shortcuts

#	Shortcut	What it does
1	`Ctrl + C`	Stop the current command
2	`Ctrl + D`	Exit the shell or send EOF
3	`Ctrl + Z`	Suspend the current process
4	`Ctrl + L`	Clear the terminal
5	`Ctrl + A`	Move to the beginning of the line
6	`Ctrl + E`	Move to the end of the line
7	`Ctrl + B`	Move backward one character
8	`Ctrl + F`	Move forward one character
9	`Alt + B`	Move backward one word
10	`Alt + F`	Move forward one word
11	`Ctrl + U`	Delete from cursor to line beginning
12	`Ctrl + K`	Delete from cursor to line end
13	`Ctrl + W`	Delete the previous word
14	`Alt + D`	Delete the next word
15	`Ctrl + Y`	Paste deleted text
16	`Ctrl + T`	Swap the previous two characters
17	`Alt + T`	Swap the previous two words
18	`Ctrl + R`	Search command history
19	`Ctrl + G`	Exit history search
20	`Ctrl + P`	Show previous command
21	`Ctrl + N`	Show next command
22	`Up Arrow`	Previous command
23	`Down Arrow`	Next command
24	`Home`	Beginning of command line
25	`End`	End of command line
26	`Alt + Backspace`	Delete previous word
27	`Ctrl + Backspace`	Delete previous word in supported terminals
28	`Ctrl + Shift + V`	Paste terminal text
29	`Ctrl + Shift + C`	Copy terminal selection
30	`Ctrl + Shift + T`	Open a new terminal tab
31	`Ctrl + Shift + W`	Close terminal tab
32	`Ctrl + Shift + N`	Open a new terminal window
33	`Ctrl + S`	Pause terminal output
34	`Ctrl + Q`	Resume terminal output
35	`Ctrl + \`	Quit a process with SIGQUIT
36	`Ctrl + X Ctrl + E`	Edit command in an external editor
37	`Ctrl + X Ctrl + U`	Undo a readline change
38	`Ctrl + X Ctrl + X`	Toggle cursor/mark position
39	`Alt + U`	Uppercase the next word
40	`Alt + L`	Lowercase the next word
41	`Alt + C`	Capitalize the next word
42	`Alt + .`	Insert the previous command's last argument
43	`Alt + <`	Move to first history entry
44	`Alt + >`	Move to last history entry
45	`Ctrl + X Ctrl + R`	Reload readline configuration
46	`Ctrl + X Ctrl + V`	Show readline version
47	`Ctrl + X Ctrl + J`	Accept the current line
48	`Ctrl + X Ctrl + X`	Toggle cursor position
49	`Ctrl + D`	Log out when the command line is empty
50	`reset`	Reset a broken terminal

These shortcuts become second nature with practice. A professional Linux user often navigates and edits commands without touching the mouse.

2. Master Filesystem Navigation

Linux organizes everything into a hierarchical filesystem. Learning how to move through that hierarchy quickly is one of the first skills you should master.

The most important commands are pwd, ls, and cd.

Navigation commands

#	Command	What it does
51	`pwd`	Print current directory
52	`ls`	List directory contents
53	`ls -l`	Detailed listing
54	`ls -a`	Show hidden files
55	`ls -la`	Detailed hidden-file listing
56	`ls -lh`	Human-readable sizes
57	`ls -lt`	Sort by modification time
58	`ls -ltr`	Reverse time sorting
59	`ls -lS`	Sort by file size
60	`ls -R`	Recursively list directories
61	`cd /`	Go to filesystem root
62	`cd ~`	Go to home directory
63	`cd ..`	Go to parent directory
64	`cd -`	Return to previous directory
65	`cd .`	Reference current directory
66	`cd /tmp`	Enter `/tmp`
67	`cd /var/log`	Enter system logs
68	`cd /etc`	Enter configuration directory
69	`cd /usr/bin`	Enter common executable directory
70	`cd /opt`	Enter optional software directory
71	`pushd /path`	Push directory onto stack
72	`popd`	Return to stacked directory
73	`dirs`	Display directory stack
74	`tree`	Display directory tree
75	`tree -L 2`	Display two directory levels
76	`tree -a`	Include hidden files
77	`find . -maxdepth 1`	Search current level
78	`find . -type f`	Find files
79	`find . -type d`	Find directories
80	`find . -name "*.log"`	Find log files
81	`find . -iname "*.jpg"`	Case-insensitive filename search
82	`find . -empty`	Find empty files/directories
83	`find . -type f -size +100M`	Find large files
84	`find . -mtime -1`	Find files modified recently
85	`find . -mmin -60`	Find files modified within an hour
86	`find . -user username`	Find files owned by a user
87	`find . -perm 777`	Find files with permission 777
88	`find . -type f -delete`	Delete matching files
89	`realpath file`	Show absolute path
90	`readlink -f file`	Resolve absolute path
91	`basename /path/file.txt`	Extract filename
92	`dirname /path/file.txt`	Extract directory
93	`namei /path/file`	Inspect path components
94	`ls /bin`	List system binaries
95	`ls /sbin`	List administrative binaries
96	`ls /home`	List user directories
97	`ls /dev`	List devices
98	`ls /proc`	List process information
99	`ls /sys`	List kernel/device information
100	`ls /run`	List runtime information

The goal isn't simply knowing what /etc or /var contains. You should become comfortable jumping between directories without thinking about the path.

3. Master File Management

Linux gives you precise control over files and directories. Commands such as cp, mv, rm, ln, and stat are fundamental.

File-management commands

#	Command	What it does
101	`touch file`	Create an empty file
102	`touch -a file`	Update access time
103	`touch -m file`	Update modification time
104	`mkdir dir`	Create directory
105	`mkdir -p a/b/c`	Create nested directories
106	`mkdir -m 755 dir`	Create directory with permissions
107	`cp file dest`	Copy file
108	`cp -r dir dest`	Copy directory recursively
109	`cp -a src dest`	Preserve attributes
110	`cp -i file dest`	Ask before overwriting
111	`cp -u file dest`	Copy only newer files
112	`cp -v file dest`	Show copy operations
113	`mv file dest`	Move or rename
114	`mv -i file dest`	Ask before overwriting
115	`mv -v file dest`	Show move operations
116	`rm file`	Delete file
117	`rm -i file`	Confirm deletion
118	`rm -r dir`	Remove directory recursively
119	`rm -f file`	Force deletion
120	`rmdir dir`	Remove empty directory
121	`ln file link`	Create hard link
122	`ln -s target link`	Create symbolic link
123	`readlink link`	Show symlink target
124	`stat file`	Show file metadata
125	`file file`	Identify file type
126	`du file`	Show disk usage
127	`du -h file`	Human-readable usage
128	`du -sh dir`	Show total directory size
129	`du -sh *`	Show size of each item
130	`du -ah`	Show recursive sizes
131	`df`	Show filesystem usage
132	`df -h`	Human-readable filesystem usage
133	`df -T`	Show filesystem types
134	`df -i`	Show inode usage
135	`sync`	Flush filesystem buffers
136	`truncate -s 0 file`	Empty a file
137	`truncate -s 1G file`	Set file size
138	`shred file`	Overwrite file contents
139	`install file dir`	Copy/install file with attributes
140	`mktemp`	Create temporary file
141	`mktemp -d`	Create temporary directory
142	`basename file`	Extract filename
143	`dirname file`	Extract directory
144	`pathchk file`	Check pathname validity
145	`lsattr file`	Show filesystem attributes
146	`chattr +i file`	Make file immutable
147	`chattr -i file`	Remove immutable attribute
148	`getfacl file`	Show ACL permissions
149	`setfacl -m u:user:rwx file`	Set ACL
150	`setfacl -x u:user file`	Remove ACL entry

Important: Commands such as rm -rf and shred can have serious consequences. Always verify paths before executing destructive commands.

4. Master File Inspection

A Linux professional spends a significant amount of time reading logs, configuration files, application output, and command results.

Viewing and searching content

#	Command	What it does
151	`cat file`	Display file
152	`cat -n file`	Show line numbers
153	`tac file`	Display lines in reverse
154	`less file`	Browse large files
155	`more file`	Paginate content
156	`head file`	Show first lines
157	`head -n 20 file`	Show first 20 lines
158	`tail file`	Show last lines
159	`tail -n 20 file`	Show last 20 lines
160	`tail -f log`	Follow a growing log
161	`tail -F log`	Follow logs across rotation
162	`wc file`	Count lines, words, bytes
163	`wc -l file`	Count lines
164	`wc -w file`	Count words
165	`wc -c file`	Count bytes
166	`wc -m file`	Count characters
167	`nl file`	Number lines
168	`od file`	Display binary/octal data
169	`xxd file`	Display hexadecimal data
170	`hexdump file`	Hexadecimal dump
171	`strings file`	Extract printable strings
172	`cut -d: -f1 file`	Extract first field
173	`paste a b`	Merge files line-by-line
174	`join a b`	Join files on fields
175	`sort file`	Sort lines
176	`sort -r file`	Reverse sort
177	`sort -n file`	Numeric sort
178	`sort -h file`	Human-readable sort
179	`sort -u file`	Sort and remove duplicates
180	`uniq file`	Remove adjacent duplicates
181	`uniq -c file`	Count duplicates
182	`uniq -d file`	Display duplicates
183	`comm a b`	Compare sorted files
184	`diff a b`	Compare files
185	`diff -u a b`	Unified diff
186	`cmp a b`	Compare byte-by-byte
187	`sed -n '1,10p' file`	Print selected lines
188	`awk '{print $1}' file`	Print first column
189	`grep pattern file`	Search text
190	`grep -i pattern file`	Case-insensitive search
191	`grep -n pattern file`	Show matching line numbers
192	`grep -r pattern dir`	Recursive search
193	`grep -v pattern file`	Invert matching
194	`grep -w word file`	Match whole words
195	`grep -E 'regex' file`	Extended regex
196	`grep -F 'text' file`	Literal search
197	`grep -C 2 pattern file`	Show surrounding context
198	`grep -l pattern files`	Show matching filenames
199	`grep -L pattern files`	Show nonmatching files
200	`grep -o pattern file`	Show only matching text

These commands become especially powerful when combined with pipes.

5. Master Text Processing

Linux becomes extremely powerful when you learn to manipulate text directly from the command line.

grep, sed, awk, cut, tr, sort, uniq, and xargs form a powerful toolkit for processing structured and unstructured text.

Text-processing mastery

#	Command	What it does
201	`sed 's/a/b/' file`	Replace first match
202	`sed 's/a/b/g' file`	Replace all matches
203	`sed -i 's/a/b/g' file`	Edit file in place
204	`sed -n '/error/p' file`	Print matching lines
205	`sed -n '5p' file`	Print line 5
206	`sed '5d' file`	Delete line 5 from output
207	`sed '1d' file`	Skip first line
208	`sed '$d' file`	Remove final line from output
209	`awk '{print $NF}' file`	Print last field
210	`awk '{print NR,$0}' file`	Add line numbers
211	`awk -F: '{print $1}' /etc/passwd`	Extract usernames
212	`awk '$3 > 100 {print}' file`	Filter numeric values
213	`awk 'BEGIN{print "Start"}'`	Run initialization
214	`awk 'END{print NR}' file`	Count records
215	`cut -c1-10 file`	Extract characters
216	`cut -d, -f2 file`	Extract CSV field
217	`tr a-z A-Z`	Convert case
218	`tr -d '\r'`	Remove carriage returns
219	`tr -s ' '`	Collapse repeated spaces
220	`rev file`	Reverse lines
221	`fold -w 80 file`	Wrap long lines
222	`fmt file`	Format paragraphs
223	`column -t file`	Align columns
224	`expand file`	Convert tabs to spaces
225	`unexpand file`	Convert spaces to tabs
226	`printf '%s\n' text`	Formatted output
227	`echo "$var"`	Print variable
228	`echo -n text`	Print without newline
229	`printf '%q\n' "$var"`	Shell-escaped output
230	`seq 1 10`	Generate sequence
231	`seq 10 -1 1`	Generate reverse sequence
232	`yes`	Repeatedly output text
233	`xargs command`	Build command arguments
234	`xargs -n1 command`	One argument per execution
235	`xargs -P4 command`	Run four jobs in parallel
236	`tee file`	Display and save output
237	`tee -a file`	Append output
238	`shuf file`	Randomize lines
239	`shuf -n 5 file`	Select random lines
240	`split file`	Split files
241	`split -l 1000 file`	Split by line count
242	`csplit file pattern`	Split around pattern
243	`paste -sd, file`	Join lines
244	`grep \	sort \
245	{% raw %}`sort \	uniq -c`
246	`sort \	uniq -c \
247	{% raw %}`awk '{sum += $1} END {print sum}'`	Sum a column
248	`awk '{sum += $1} END {print sum/NR}'`	Calculate average
249	`grep -oE '[0-9]+' file`	Extract numbers
250	`grep -oE 'https?://[^ ]+' file`	Extract URLs

6. Master Linux Permissions

Linux permissions are fundamental to system administration and security.

Every file has an owner, a group, and permission settings for the owner, group, and others.

Permissions and ownership

#	Command	What it does
251	`ls -l`	Inspect permissions
252	`chmod 755 file`	Set common executable permissions
253	`chmod 644 file`	Set common file permissions
254	`chmod 700 file`	Owner-only access
255	`chmod 600 file`	Owner read/write only
256	`chmod +x file`	Add execute permission
257	`chmod -x file`	Remove execute permission
258	`chmod u+x file`	Add owner execute permission
259	`chmod g+w file`	Add group write permission
260	`chmod o-r file`	Remove others' read permission
261	`chmod a+r file`	Add read permission
262	`chmod -R 755 dir`	Recursively modify permissions
263	`chown user file`	Change owner
264	`chown user:group file`	Change owner and group
265	`chown -R user:group dir`	Recursively change ownership
266	`chgrp group file`	Change group
267	`umask`	Show default permission mask
268	`umask 022`	Set default permission mask
269	`id`	Show user identity
270	`id user`	Show another user's identity
271	`whoami`	Show current username
272	`groups`	Show group memberships
273	`groups user`	Show user's groups
274	`who`	Show logged-in users
275	`w`	Show user activity
276	`last`	Show login history
277	`lastlog`	Show last-login information
278	`passwd`	Change password
279	`sudo command`	Run command as administrator
280	`sudo -i`	Open root login shell
281	`sudo -s`	Open privileged shell
282	`sudo -u user command`	Run as another user
283	`sudo -l`	Show sudo permissions
284	`su user`	Switch user
285	`su -`	Switch to root/login shell
286	`visudo`	Safely edit sudoers configuration
287	`getent passwd user`	Query user database
288	`getent group group`	Query group database
289	`useradd user`	Create user
290	`userdel user`	Delete user
291	`usermod -aG group user`	Add user to group
292	`groupadd group`	Create group
293	`groupdel group`	Delete group
294	`gpasswd -a user group`	Add user to group
295	`gpasswd -d user group`	Remove user from group
296	`chage -l user`	Show password aging
297	`chage -M 90 user`	Set password expiration
298	`loginctl list-users`	List users
299	`loginctl user-status`	Show session status
300	`getfacl file`	Inspect ACLs

Understanding permissions is more important than memorizing numerical modes. Learn what r, w, and x mean for files and directories, then master symbolic and numeric notation.

7. Master Processes and Jobs

A Linux system can run hundreds or thousands of processes simultaneously. Professional Linux users need to know how to inspect, control, prioritize, and troubleshoot them.

Process management

#	Command	What it does
301	`ps`	Show processes
302	`ps aux`	Show all processes
303	`ps -ef`	Full process listing
304	`ps aux --sort=-%cpu`	Sort by CPU
305	`ps aux --sort=-%mem`	Sort by memory
306	`top`	Interactive process monitor
307	`htop`	Enhanced process monitor
308	`pgrep name`	Find process IDs
309	`pidof program`	Find program PIDs
310	`pstree`	Display process tree
311	`kill PID`	Send termination signal
312	`kill -9 PID`	Force-kill process
313	`kill -STOP PID`	Stop process
314	`kill -CONT PID`	Continue process
315	`killall name`	Kill processes by name
316	`pkill pattern`	Kill matching processes
317	`nice command`	Start with adjusted priority
318	`nice -n 10 command`	Start with lower priority
319	`renice 10 -p PID`	Change process priority
320	`jobs`	Show shell jobs
321	`fg`	Bring job to foreground
322	`fg %1`	Bring job 1 foreground
323	`bg`	Resume background job
324	`bg %1`	Resume job 1
325	`command &`	Run command in background
326	`nohup command &`	Keep process running after logout
327	`disown`	Detach shell job
328	`disown -h %1`	Keep job alive after logout
329	`wait`	Wait for background jobs
330	`wait PID`	Wait for process
331	`time command`	Measure execution time
332	`/usr/bin/time command`	Detailed resource timing
333	`watch command`	Repeat command
334	`watch -n 1 command`	Repeat every second
335	`timeout 10 command`	Limit execution
336	`timeout 10s command`	Ten-second timeout
337	`strace command`	Trace system calls
338	`strace -p PID`	Trace running process
339	`lsof -p PID`	Show files opened by process
340	`lsof -i`	Show network connections
341	`lsof /path/file`	Find process using file
342	`fuser file`	Find processes using file
343	`fuser -k port/tcp`	Kill process using port
344	`vmstat`	System statistics
345	`vmstat 1`	Monitor statistics
346	`iostat`	CPU/I/O statistics
347	`mpstat`	CPU statistics
348	`pidstat`	Per-process statistics
349	`sar`	System activity reporting
350	`uptime`	Show load averages

A key professional habit is to inspect before killing. Identify the process, understand what it is doing, and then choose the least disruptive action.

8. Master Environment Variables and Bash

Environment variables control much of the behavior of Linux applications and shells.

The PATH variable, for example, determines where the shell searches for executable commands.

Environment and shell techniques

#	Command	What it does
351	`env`	Show environment variables
352	`printenv`	Print environment
353	`printenv PATH`	Show PATH
354	`echo $HOME`	Show home directory
355	`echo $USER`	Show username
356	`echo $SHELL`	Show shell
357	`echo $PWD`	Show current directory
358	`echo $OLDPWD`	Show previous directory
359	`echo $?`	Show last exit status
360	`echo $$`	Show shell PID
361	`echo $!`	Show last background PID
362	`echo $#`	Show argument count
363	`echo $0`	Show script name
364	`export VAR=value`	Export variable
365	`unset VAR`	Remove variable
366	`VAR=value command`	Set variable for one command
367	`readonly VAR=value`	Make variable read-only
368	`set`	Show shell variables
369	`set -x`	Enable tracing
370	`set +x`	Disable tracing
371	`set -e`	Exit on command failure
372	`set -u`	Error on unset variables
373	`set -o pipefail`	Detect pipeline failures
374	`set -euxo pipefail`	Strict shell mode
375	`source file`	Load shell configuration
376	`. file`	Source a file
377	`alias`	Show aliases
378	`alias ll='ls -lah'`	Create alias
379	`unalias ll`	Remove alias
380	`type command`	Identify command
381	`type -a command`	Show all command locations
382	`command -v command`	Locate executable
383	`which command`	Locate executable
384	`whereis command`	Locate binary and documentation
385	`hash`	Show shell command cache
386	`hash -r`	Clear command cache
387	`history`	Show history
388	`history 20`	Show recent history
389	`!!`	Repeat previous command
390	`!$`	Reuse last argument
391	`!*`	Reuse all arguments
392	`!^`	Reuse first argument
393	`!n`	Execute history entry
394	`!-2`	Execute previous-previous command
395	`!string`	Execute latest matching command
396	`^old^new^`	Replace text in previous command
397	`HISTCONTROL=ignoredups`	Ignore duplicate history
398	`HISTSIZE=10000`	Increase history size
399	`HISTFILE=~/.bash_history`	Define history file
400	`history -c`	Clear current history

9. Master Pipes and Redirection

The Unix philosophy is built around small programs that do one thing well and can be combined.

The pipe operator | is one of the most important concepts to understand.

For example:

ps aux | grep nginx

The output of ps becomes the input of grep.

Pipes and redirection

#	Command	What it does
401	`command > file`	Redirect stdout
402	`command >> file`	Append stdout
403	`command < file`	Redirect stdin
404	`command 2> file`	Redirect stderr
405	`command 2>> file`	Append stderr
406	`command &> file`	Redirect stdout and stderr
407	`command 2>&1`	Redirect stderr to stdout
408	`command 1>&2`	Redirect stdout to stderr
409	`command > /dev/null`	Discard output
410	`command 2> /dev/null`	Discard errors
411	`command &> /dev/null`	Discard all output
412	`command \	less`
413	`command \	grep text`
414	`command \	sort`
415	`command \	uniq`
416	`command \	wc -l`
417	`command \	tee file`
418	`command1 \	command2`
419	`command1 && command2`	Run second after success
420	`command1 \	\
421	{% raw %}`command1 ; command2`	Run commands sequentially
422	`(command1; command2)`	Run in subshell
423	`{ command1; command2; }`	Run in current shell
424	`$(command)`	Command substitution
425	`command`	Legacy command substitution
426	`VAR=$(command)`	Store command output
427	`<(command)`	Process substitution
428	`>(command)`	Output process substitution
429	`xargs`	Convert input into arguments
430	`xargs -0`	Process null-delimited input
431	`find ... -print0 \	xargs -0`
432	`cmd1 \	cmd2 \
433	{% raw %}`yes \	command`
434	`printf ... \	command`
435	`cat file \	command`
436	`command <<EOF`	Start heredoc
437	`command <<'EOF'`	Literal heredoc
438	`command <<< "$var"`	Here-string
439	`exec > file`	Redirect shell stdout
440	`exec 2> errors.log`	Redirect shell errors
441	`exec 3>file`	Open file descriptor 3
442	`>&3`	Redirect to descriptor 3
443	`0<&3`	Read from descriptor 3
444	`command &`	Background command
445	`command &>/dev/null &`	Silent background command
446	`nohup command >/tmp/out 2>&1 &`	Persistent background command
447	`command \	& other`
448	`coproc name command`	Start coprocess
449	`wait $!`	Wait for background process
450	`exit $?`	Exit with previous status

Once you understand pipes and redirection, Linux commands stop being isolated tools and become building blocks.

10. Master Archives and Compression

Linux systems frequently deal with compressed logs, backups, source-code packages, application releases, and deployment artifacts.

Compression commands

#	Command	What it does
451	`tar -cf archive.tar files`	Create tar archive
452	`tar -xf archive.tar`	Extract tar archive
453	`tar -tf archive.tar`	List archive contents
454	`tar -czf archive.tar.gz dir`	Create gzip archive
455	`tar -xzf archive.tar.gz`	Extract gzip archive
456	`tar -cjf archive.tar.bz2 dir`	Create bzip2 archive
457	`tar -xjf archive.tar.bz2`	Extract bzip2 archive
458	`tar -cJf archive.tar.xz dir`	Create xz archive
459	`tar -xJf archive.tar.xz`	Extract xz archive
460	`tar -tvf archive.tar`	Verbose listing
461	`tar -rf archive.tar file`	Append to tar
462	`tar -uf archive.tar dir`	Update archive
463	`tar --delete -f archive.tar file`	Delete archive member
464	`gzip file`	Compress with gzip
465	`gzip -d file.gz`	Decompress gzip
466	`gunzip file.gz`	Decompress gzip
467	`gzip -k file`	Keep original
468	`bzip2 file`	Compress with bzip2
469	`bunzip2 file.bz2`	Decompress bzip2
470	`xz file`	Compress with xz
471	`unxz file.xz`	Decompress xz
472	`zip archive.zip files`	Create ZIP
473	`zip -r archive.zip dir`	ZIP directory
474	`unzip archive.zip`	Extract ZIP
475	`unzip -l archive.zip`	List ZIP contents
476	`zipinfo archive.zip`	Inspect ZIP
477	`7z a archive.7z files`	Create 7z archive
478	`7z x archive.7z`	Extract 7z
479	`zcat file.gz`	Read gzip directly
480	`bzcat file.bz2`	Read bzip2 directly
481	`xzcat file.xz`	Read xz directly
482	`zgrep pattern file.gz`	Search gzip
483	`zless file.gz`	Browse gzip
484	`zmore file.gz`	Paginate gzip
485	`tar -C dir -xf archive.tar`	Extract to directory
486	`tar --exclude='*.log'`	Exclude files
487	`tar --exclude=dir`	Exclude directory
488	`tar -czvf archive.tar.gz dir`	Verbose archive creation
489	`tar -xvzf archive.tar.gz`	Verbose extraction
490	`gzip -9 file`	Maximum gzip compression
491	`gzip -1 file`	Faster gzip compression
492	`xz -9 file`	High xz compression
493	`zip -9 archive.zip file`	Maximum ZIP compression
494	`tar -cf - dir \	gzip > archive.tar.gz`
495	`tar -xf - < archive.tar`	Extract from stdin
496	`tar -czf - dir \	ssh host 'cat > archive.tar.gz'`
497	`tar -czf - dir \	ssh host 'tar -xzf -'`
498	`gzip -l file.gz`	Show compression statistics
499	`file archive.tar.gz`	Identify archive
500	`du -sh archive`	Check archive size

11. Master Disk and Storage Management

Storage problems are common in production environments. Knowing how to inspect disks, partitions, filesystems, mounts, and inodes is essential.

Storage commands

#	Command	What it does
501	`lsblk`	List block devices
502	`lsblk -f`	Show filesystem information
503	`blkid`	Show block-device UUIDs
504	`df -h`	Show filesystem capacity
505	`du -sh .`	Show current directory size
506	`du -sh *`	Show item sizes
507	`du -ah . \	sort -h`
508	`ncdu`	Interactive disk analyzer
509	`mount`	Show mounted filesystems
510	`mount /dev/sdb1 /mnt`	Mount filesystem
511	`umount /mnt`	Unmount filesystem
512	`findmnt`	Show mounts
513	`findmnt /path`	Identify mount
514	`cat /etc/fstab`	View mount configuration
515	`swapon --show`	Show swap
516	`free -h`	Show memory and swap
517	`sync`	Flush buffers
518	`fdisk -l`	List partitions
519	`parted -l`	List partition information
520	`file -s /dev/sdb`	Identify device data
521	`dd if=/dev/zero of=file bs=1M count=10`	Create test file
522	`dd if=file of=copy`	Raw copy
523	`lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT`	Custom block listing
524	`du --max-depth=1 -h`	Directory-level usage
525	`df -Th`	Filesystem type and usage
526	`df -ih`	Inode usage
527	`tune2fs -l /dev/device`	Inspect ext filesystem
528	`e2fsck`	Check ext filesystem
529	`fsck`	Check filesystem
530	`mkswap /dev/device`	Create swap area
531	`swapon /dev/device`	Enable swap
532	`swapoff /dev/device`	Disable swap
533	`fstrim -av`	Trim supported filesystems
534	`ls /dev/disk/by-uuid`	List disks by UUID
535	`ls /dev/disk/by-label`	List disks by label
536	`blkid /dev/sda1`	Show partition UUID
537	`mount -o ro device mountpoint`	Mount read-only
538	`mount -a`	Mount fstab filesystems
539	`umount -l mountpoint`	Lazy unmount
540	`umount -f mountpoint`	Force unmount where supported
541	`du -x`	Stay on one filesystem
542	`du -d 1`	Limit depth
543	`df -P`	POSIX-style output
544	`lsblk -S`	List SCSI devices
545	`lsblk -t`	Show device topology
546	`udevadm info`	Inspect device information
547	`udevadm monitor`	Monitor device events
548	`dmesg \	grep -i usb`
549	`dmesg \	tail`
550	`journalctl -k`	Show kernel journal

12. Master Linux Networking

Networking is one of the most important areas for Linux professionals.

Whether you're troubleshooting a web server, configuring a cloud instance, debugging DNS, or investigating an unreachable service, networking commands help you understand what is happening.

Networking commands

#	Command	What it does
551	`ip addr`	Show IP addresses
552	`ip link`	Show interfaces
553	`ip route`	Show routing table
554	`ip neigh`	Show neighbor table
555	`ip -br addr`	Compact IP display
556	`ip -br link`	Compact interface display
557	`ip link set eth0 up`	Enable interface
558	`ip link set eth0 down`	Disable interface
559	`ip addr add IP/24 dev eth0`	Add IP address
560	`ip addr del IP/24 dev eth0`	Remove IP address
561	`ip route get 8.8.8.8`	Show route to destination
562	`ping host`	Test connectivity
563	`ping -c 4 host`	Send four pings
564	`ping6 host`	Test IPv6
565	`traceroute host`	Trace route
566	`tracepath host`	Trace route and MTU
567	`mtr host`	Combine ping and traceroute
568	`curl URL`	Make HTTP request
569	`curl -I URL`	Show HTTP headers
570	`curl -L URL`	Follow redirects
571	`curl -O URL`	Download resource
572	`curl -o file URL`	Save response to file
573	`curl -s URL`	Silent request
574	`curl -v URL`	Verbose request
575	`curl -X POST URL`	Send POST request
576	`curl -d 'data' URL`	Send request data
577	`curl -H 'Header: value' URL`	Add HTTP header
578	`wget URL`	Download resource
579	`wget -c URL`	Resume download
580	`wget -q URL`	Quiet download
581	`wget -r URL`	Recursive download
582	`dig domain`	DNS query
583	`dig +short domain`	Concise DNS result
584	`dig MX domain`	Query MX records
585	`dig NS domain`	Query NS records
586	`host domain`	DNS lookup
587	`nslookup domain`	DNS lookup
588	`resolvectl status`	Show DNS configuration
589	`resolvectl query domain`	Query DNS
590	`ss -tuln`	Show listening sockets
591	`ss -tunap`	Show connections and processes
592	`ss -ltn`	Show listening TCP ports
593	`ss -lun`	Show listening UDP ports
594	`nc -vz host port`	Test TCP port
595	`nc -l port`	Listen on port
596	`hostname`	Show hostname
597	`hostname -I`	Show local IP addresses
598	`uname -n`	Show hostname
599	`arp -a`	Show ARP cache where available
600	`ethtool eth0`	Inspect Ethernet interface

13. Master SSH and Remote Linux

Linux professionals rarely work only on their local machine. SSH is the foundation of remote Linux administration.

SSH commands

#	Command	What it does
601	`ssh user@host`	Connect to remote host
602	`ssh -p 2222 user@host`	Connect using custom port
603	`ssh -i key user@host`	Use private key
604	`ssh -v user@host`	Debug SSH
605	`ssh -vvv user@host`	Detailed SSH debugging
606	`ssh-keygen`	Generate SSH key
607	`ssh-keygen -t ed25519`	Generate Ed25519 key
608	`ssh-keygen -t rsa`	Generate RSA key
609	`ssh-keygen -lf key.pub`	Show fingerprint
610	`ssh-copy-id user@host`	Install public key
611	`ssh-add key`	Add key to agent
612	`ssh-add -l`	List agent keys
613	`ssh-add -D`	Remove agent keys
614	`ssh-agent bash`	Start SSH agent shell
615	`scp file user@host:/path`	Copy file remotely
616	`scp user@host:/path/file .`	Download remote file
617	`scp -r dir user@host:/path`	Copy directory
618	`rsync file host:/path`	Synchronize file
619	`rsync -av dir host:/path`	Synchronize directory
620	`rsync -avz dir host:/path`	Compress transfer
621	`rsync --delete src dest`	Mirror directories
622	`rsync --dry-run src dest`	Preview synchronization
623	`ssh host command`	Execute remote command
624	`ssh host 'sudo command'`	Execute privileged command
625	`ssh -L 8080:localhost:80 host`	Local port forwarding
626	`ssh -R 8080:localhost:80 host`	Remote port forwarding
627	`ssh -D 1080 host`	SOCKS proxy
628	`ssh -N host`	Don't execute remote command
629	`ssh -fN host`	Background SSH forwarding
630	`ssh -o ServerAliveInterval=60 host`	Keep connection alive
631	`ssh -G host`	Show effective configuration
632	`ssh-keyscan host`	Retrieve host keys
633	`~/.ssh/config`	User SSH configuration
634	`~/.ssh/authorized_keys`	Authorized keys
635	`~/.ssh/known_hosts`	Known hosts
636	`chmod 700 ~/.ssh`	Protect SSH directory
637	`chmod 600 ~/.ssh/id_ed25519`	Protect private key
638	`chmod 644 ~/.ssh/id_ed25519.pub`	Protect public key
639	`ssh-keygen -R host`	Remove known host
640	`scp -P port file host:/path`	SCP using custom port
641	`rsync -e ssh src host:/path`	Rsync over SSH
642	`rsync -e 'ssh -p 2222' src host:/path`	Rsync over custom SSH port
643	`ssh user@host 'uptime'`	Check remote uptime
644	`ssh user@host 'df -h'`	Check remote storage
645	`ssh user@host 'free -h'`	Check remote memory
646	`ssh user@host 'ps aux'`	Inspect remote processes
647	`ssh user@host 'journalctl -n 50'`	Inspect recent logs
648	`ssh user@host 'systemctl status service'`	Check remote service
649	`ssh user@host 'ip addr'`	Inspect remote network
650	`ssh user@host 'uname -a'`	Inspect remote kernel

14. Master Package Management

Linux distributions use different package managers. Learn the package manager used by the distribution you administer.

Debian/Ubuntu

#	Command	What it does
651	`apt update`	Refresh package indexes
652	`apt upgrade`	Upgrade packages
653	`apt install package`	Install package
654	`apt remove package`	Remove package
655	`apt purge package`	Remove package and configuration
656	`apt autoremove`	Remove unused dependencies
657	`apt search package`	Search packages
658	`apt show package`	Show package information
659	`apt list --installed`	List installed packages
660	`apt policy package`	Show package versions
661	`apt-cache search package`	Search package cache
662	`apt-cache policy package`	Show package policy
663	`dpkg -i package.deb`	Install DEB
664	`dpkg -r package`	Remove package
665	`dpkg -l`	List packages
666	`dpkg -L package`	List package files
667	`dpkg -S /path/file`	Find package owning file

Fedora/RHEL

#	Command	What it does
668	`dnf install package`	Install package
669	`dnf update`	Update packages
670	`dnf remove package`	Remove package
671	`dnf search package`	Search packages
672	`dnf info package`	Show package information
673	`dnf list installed`	List installed packages
674	`dnf history`	Show package transactions
675	`yum install package`	Install using legacy yum
676	`rpm -i package.rpm`	Install RPM
677	`rpm -e package`	Remove RPM
678	`rpm -qa`	List RPM packages
679	`rpm -qi package`	Show RPM information
680	`rpm -ql package`	List RPM files

Arch Linux

#	Command	What it does
681	`pacman -S package`	Install package
682	`pacman -Syu`	Update system
683	`pacman -R package`	Remove package
684	`pacman -Ss package`	Search packages
685	`pacman -Qi package`	Show package information

Other package systems

#	Command	What it does
686	`zypper install package`	Install openSUSE package
687	`zypper update`	Update packages
688	`zypper remove package`	Remove package
689	`snap list`	List Snap packages
690	`snap install package`	Install Snap
691	`snap remove package`	Remove Snap
692	`flatpak list`	List Flatpak applications
693	`flatpak install package`	Install Flatpak
694	`flatpak update`	Update Flatpaks
695	`flatpak uninstall package`	Remove Flatpak
696	`apt-mark hold package`	Hold package
697	`apt-mark unhold package`	Remove package hold
698	`apt-mark manual package`	Mark manually installed
699	`apt-mark auto package`	Mark automatically installed
700	`apt clean`	Clear package cache

15. Master System Information

When troubleshooting Linux, you need to know how to inspect the machine itself.

CPU, memory, kernel, devices, operating-system information, logs, and system state are all available from the command line.

System information commands

#	Command	What it does
701	`uname -a`	Complete kernel information
702	`uname -r`	Kernel version
703	`uname -m`	CPU architecture
704	`hostnamectl`	System information
705	`lsb_release -a`	Distribution information
706	`cat /etc/os-release`	OS information
707	`uptime`	System uptime
708	`free -h`	Memory usage
709	`lscpu`	CPU information
710	`lsmem`	Memory ranges
711	`lsusb`	USB devices
712	`lspci`	PCI devices
713	`lspci -k`	PCI devices and drivers
714	`lshw`	Hardware information
715	`lsmod`	Loaded kernel modules
716	`modinfo module`	Module information
717	`modprobe module`	Load module
718	`modprobe -r module`	Remove module
719	`dmesg`	Kernel messages
720	`dmesg -T`	Human-readable kernel timestamps
721	`journalctl`	System journal
722	`journalctl -b`	Current boot logs
723	`journalctl -b -1`	Previous boot logs
724	`journalctl -p err`	Error-level logs
725	`journalctl -f`	Follow logs
726	`journalctl -u service`	Service logs
727	`journalctl --since today`	Today's logs
728	`journalctl --since "1 hour ago"`	Recent logs
729	`timedatectl`	Time configuration
730	`timedatectl status`	Clock status
731	`localectl`	Locale settings
732	`locale`	Locale variables
733	`date`	Current date/time
734	`date -u`	UTC time
735	`cal`	Calendar
736	`whoami`	Current user
737	`id`	User identity
738	`who`	Logged-in users
739	`w`	User activity
740	`last`	Login history
741	`uptime -p`	Pretty uptime
742	`cat /proc/cpuinfo`	Detailed CPU data
743	`cat /proc/meminfo`	Detailed memory data
744	`cat /proc/version`	Kernel build information
745	`cat /proc/loadavg`	Load averages
746	`cat /proc/uptime`	Kernel uptime
747	`free -m`	Memory in MB
748	`free -g`	Memory in GB
749	`vmstat -s`	System statistics
750	`dmesg --level=err,warn`	Kernel warnings/errors

16. Master systemd and Services

Modern Linux distributions commonly use systemd for service management.

Understanding systemctl and journalctl is essential for Linux administration.

systemd commands

#	Command	What it does
751	`systemctl status service`	Show service status
752	`systemctl start service`	Start service
753	`systemctl stop service`	Stop service
754	`systemctl restart service`	Restart service
755	`systemctl reload service`	Reload configuration
756	`systemctl enable service`	Enable at boot
757	`systemctl disable service`	Disable at boot
758	`systemctl enable --now service`	Enable and start
759	`systemctl disable --now service`	Disable and stop
760	`systemctl is-active service`	Check active state
761	`systemctl is-enabled service`	Check boot enablement
762	`systemctl is-failed service`	Check failure
763	`systemctl list-units`	List active units
764	`systemctl list-units --type=service`	List services
765	`systemctl list-unit-files`	List installed units
766	`systemctl list-dependencies service`	Show dependencies
767	`systemctl cat service`	Show service configuration
768	`systemctl edit service`	Create override
769	`systemctl daemon-reload`	Reload unit configuration
770	`systemctl mask service`	Prevent service startup
771	`systemctl unmask service`	Remove service mask
772	`systemctl reboot`	Reboot
773	`systemctl poweroff`	Power off
774	`systemctl suspend`	Suspend
775	`systemctl hibernate`	Hibernate
776	`systemctl rescue`	Enter rescue mode
777	`systemctl emergency`	Enter emergency mode
778	`systemctl get-default`	Show default target
779	`systemctl set-default graphical.target`	Set graphical target
780	`systemctl set-default multi-user.target`	Set multi-user target
781	`systemctl list-timers`	List timers
782	`systemctl status timer`	Show timer status
783	`systemctl start timer`	Start timer
784	`systemctl enable timer`	Enable timer
785	`journalctl -u service`	Service logs
786	`journalctl -fu service`	Follow service logs
787	`systemctl reset-failed service`	Clear failure state
788	`systemctl show service`	Show service properties
789	`systemctl list-sockets`	List socket units
790	`systemctl list-mounts`	List mounts
791	`systemctl list-paths`	List path units
792	`systemctl list-scopes`	List scopes
793	`systemctl reboot --firmware-setup`	Reboot to firmware setup where supported
794	`loginctl list-sessions`	List sessions
795	`loginctl session-status`	Show session status
796	`loginctl terminate-session ID`	Terminate session
797	`loginctl lock-session`	Lock session
798	`loginctl unlock-session`	Unlock session
799	`loginctl terminate-user user`	Terminate user sessions
800	`systemd-analyze`	Analyze system boot

17. Master Scheduling and Automation

Linux becomes significantly more powerful when you stop manually repeating tasks and start automating them.

Cron, systemd timers, shell loops, find, xargs, and at are useful tools for automation.

Scheduling and automation

#	Command	What it does
801	`crontab -e`	Edit user cron jobs
802	`crontab -l`	List cron jobs
803	`crontab -r`	Remove cron jobs
804	`at 10:00`	Schedule a command
805	`atq`	List scheduled jobs
806	`atrm ID`	Remove scheduled job
807	`batch`	Run jobs when load permits
808	`sleep 10`	Pause execution
809	`sleep 1h`	Pause for an hour
810	`watch -n 5 command`	Repeat command
811	`timeout 60 command`	Limit execution time
812	`date`	Generate timestamp
813	`date +%F`	Output ISO date
814	`date +%T`	Output current time
815	`date +%s`	Unix timestamp
816	`date -d tomorrow`	Calculate tomorrow
817	`date -d "next monday"`	Calculate next Monday
818	`date -d "@TIMESTAMP"`	Convert Unix timestamp
819	`sleep 5 && command`	Delay command
820	`command \	at now`
821	`while true; do command; sleep 5; done`	Repeat command
822	`for f in *; do command "$f"; done`	Process files in loop
823	`find . -exec command {} \;`	Execute per match
824	`find . -exec command {} +`	Batch execute matches
825	`find . -print0 \	xargs -0 command`
826	`watch 'df -h'`	Monitor storage
827	`watch 'free -h'`	Monitor memory
828	`watch 'ss -tuln'`	Monitor ports
829	`watch 'ps aux --sort=-%cpu \	head'`
830	`watch 'du -sh *'`	Monitor directory sizes
831	`crontab -u user -l`	View another user's cron
832	`systemctl list-timers`	List systemd timers
833	`systemd-run --on-active=1m command`	Run delayed command
834	`systemd-run --on-calendar=daily command`	Schedule systemd task
835	`nohup command &`	Persist background process
836	`disown -h`	Detach shell job
837	`trap 'command' EXIT`	Execute on script exit
838	`trap 'command' INT`	Handle interrupt
839	`trap -p`	Show configured traps
840	`while read line; do command; done`	Process input lines
841	`until command; do sleep 1; done`	Retry until success
842	`for i in {1..10}; do echo "$i"; done`	Iterate numbers
843	`seq 1 10 \	xargs -n1 command`
844	`parallel command ::: a b c`	Parallel execution
845	`flock file command`	Prevent concurrent execution
846	`timeout 5s curl URL`	Time-limit HTTP request
847	`watch -d command`	Highlight output changes
848	`watch -g command`	Exit when output changes
849	`cron`	Background scheduling service
850	`systemd-run`	Create transient systemd unit

18. Master Git from Linux

Linux and Git are deeply connected in modern software development.

You should be comfortable managing repositories directly from the terminal.

Git commands

#	Command	What it does
851	`git init`	Initialize repository
852	`git clone URL`	Clone repository
853	`git status`	Show repository status
854	`git add file`	Stage file
855	`git add .`	Stage changes
856	`git commit -m "message"`	Create commit
857	`git log`	Show commit history
858	`git log --oneline`	Compact history
859	`git log --graph --oneline`	Graph history
860	`git diff`	Show unstaged changes
861	`git diff --cached`	Show staged changes
862	`git diff HEAD`	Show all changes
863	`git branch`	List branches
864	`git branch name`	Create branch
865	`git switch name`	Switch branch
866	`git switch -c name`	Create and switch branch
867	`git checkout name`	Switch branch
868	`git merge branch`	Merge branch
869	`git rebase branch`	Rebase
870	`git fetch`	Download remote changes
871	`git pull`	Fetch and integrate
872	`git push`	Upload commits
873	`git push -u origin branch`	Push and track branch
874	`git remote -v`	Show remotes
875	`git remote add origin URL`	Add remote
876	`git stash`	Temporarily save changes
877	`git stash pop`	Restore stash
878	`git stash list`	List stashes
879	`git stash apply`	Apply stash
880	`git stash drop`	Delete stash
881	`git reset file`	Unstage file
882	`git reset --soft HEAD~1`	Undo commit, preserve staged changes
883	`git reset --mixed HEAD~1`	Undo commit, preserve working changes
884	`git reset --hard HEAD~1`	Reset commit and working tree
885	`git restore file`	Restore working-tree file
886	`git restore --staged file`	Unstage file
887	`git revert COMMIT`	Create inverse commit
888	`git tag`	List tags
889	`git tag v1.0.0`	Create tag
890	`git show COMMIT`	Show commit details
891	`git blame file`	Show line authorship
892	`git cherry-pick COMMIT`	Apply specific commit
893	`git clean -n`	Preview untracked deletion
894	`git clean -fd`	Delete untracked files/directories
895	`git reflog`	Show reference history
896	`git bisect start`	Start binary search
897	`git bisect good`	Mark good commit
898	`git bisect bad`	Mark bad commit
899	`git grep pattern`	Search repository
900	`git shortlog -sn`	Summarize contributors

19. Master Docker from the Terminal

Containers are an important part of modern Linux development and DevOps.

If you understand Docker from the command line, you can build, run, inspect, debug, network, and manage containers without depending on graphical tools.

Docker commands

#	Command	What it does
901	`docker ps`	List running containers
902	`docker ps -a`	List all containers
903	`docker images`	List images
904	`docker pull image`	Download image
905	`docker run image`	Run container
906	`docker run -it image bash`	Open interactive shell
907	`docker run -d image`	Run detached
908	`docker stop container`	Stop container
909	`docker start container`	Start container
910	`docker restart container`	Restart container
911	`docker rm container`	Remove container
912	`docker rm -f container`	Force-remove container
913	`docker rmi image`	Remove image
914	`docker exec -it container bash`	Enter container
915	`docker logs container`	View logs
916	`docker logs -f container`	Follow logs
917	`docker inspect container`	Inspect container
918	`docker stats`	Show resource usage
919	`docker top container`	Show container processes
920	`docker cp file container:/path`	Copy into container
921	`docker cp container:/path file`	Copy from container
922	`docker build -t app .`	Build image
923	`docker tag image repository:tag`	Tag image
924	`docker push repository:tag`	Push image
925	`docker login`	Authenticate registry
926	`docker logout`	Log out
927	`docker network ls`	List networks
928	`docker network create network`	Create network
929	`docker network connect network container`	Connect container
930	`docker network disconnect network container`	Disconnect container
931	`docker volume ls`	List volumes
932	`docker volume create volume`	Create volume
933	`docker volume inspect volume`	Inspect volume
934	`docker volume rm volume`	Remove volume
935	`docker system df`	Show Docker disk usage
936	`docker system prune`	Remove unused resources
937	`docker container prune`	Remove stopped containers
938	`docker image prune`	Remove unused images
939	`docker volume prune`	Remove unused volumes
940	`docker compose up`	Start Compose services
941	`docker compose up -d`	Start detached
942	`docker compose down`	Stop and remove services
943	`docker compose ps`	Show Compose containers
944	`docker compose logs`	Show Compose logs
945	`docker compose logs -f`	Follow Compose logs
946	`docker compose build`	Build Compose images
947	`docker compose pull`	Pull Compose images
948	`docker compose restart`	Restart services
949	`docker compose exec service bash`	Enter service container
950	`docker compose config`	Validate Compose configuration

20. Become a Linux Power User

The final 50 techniques bring everything together.

The difference between a beginner and a professional isn't the number of commands memorized. It's the ability to combine commands into efficient workflows.

Advanced Linux productivity

#	Command	What it does
951	`fzf`	Interactive fuzzy finder
952	`history \	fzf`
953	`find . \	fzf`
954	`Ctrl + R`	Search command history
955	`!!`	Repeat previous command
956	`sudo !!`	Repeat previous command with sudo
957	`cd $(dirname "$(fzf)")`	Navigate using fuzzy selection
958	`alias ll='ls -lah'`	Create listing shortcut
959	`alias gs='git status'`	Create Git shortcut
960	`alias ..='cd ..'`	Parent-directory shortcut
961	`alias ...='cd ../..'`	Go up two directories
962	`alias c='clear'`	Clear shortcut
963	`alias ports='ss -tuln'`	Quickly inspect ports
964	`alias myip='curl -s ifconfig.me'`	Check public IP
965	`alias update='sudo apt update && sudo apt upgrade'`	Update Debian/Ubuntu
966	`clear`	Clear terminal
967	`reset`	Reset terminal
968	`tput cols`	Show terminal width
969	`tput lines`	Show terminal height
970	`tput clear`	Clear using terminal capabilities
971	`printf '\033c'`	Reset terminal display
972	`env \	sort`
973	`alias \	less`
974	`compgen -c \	sort -u`
975	`compgen -b`	List Bash builtins
976	`help command`	Show Bash builtin help
977	`man command`	Open manual
978	`man -k keyword`	Search manual pages
979	`apropos keyword`	Search command descriptions
980	`info command`	Open GNU documentation
981	`command --help`	Show command help
982	`command -h`	Short help where supported
983	`whatis command`	Show brief description
984	`tldr command`	Show concise examples if installed
985	`watch -n 1 'date'`	Live clock
986	`watch -n 1 'free -h'`	Live memory monitor
987	`watch -n 1 'df -h'`	Live disk monitor
988	`watch -n 1 'uptime'`	Live load monitor
989	`watch -n 1 'ss -s'`	Live socket monitor
990	`find . -type f -print \	wc -l`
991	`du -ah . \	sort -rh \
992	{% raw %}`ps aux --sort=-%cpu \	head`
993	`ps aux --sort=-%mem \	head`
994	`journalctl -p err -b`	Find boot errors
995	`dmesg -T \	tail -50`
996	`history \	tail -50`
997	`command1 && command2`	Chain successful commands
998	`command1 \	\
999	{% raw %}`command \	tee output.log`
1000	`man man`	Learn how to use manual pages

The Real Secret to Linux Mastery

Learning 1,000 Linux commands doesn't make you a Linux professional.

Knowing how to combine them does.

A beginner might run:

ls

A more experienced user might run:

ls -lah

A power user might use:

find /var/log -type f -name "*.log" -size +100M -print

A system administrator might investigate further:

find /var/log -type f -name "*.log" -size +100M -exec ls -lh {} \;

And an experienced engineer understands why those commands are useful, what the output means, what could go wrong, and what command should come next.

That is the mindset you want to develop.

Your Linux Mastery Roadmap

LINUX BEGINNER
      ↓
Terminal Navigation
      ↓
Files & Directories
      ↓
Text Processing
      ↓
Permissions
      ↓
Processes
      ↓
Pipes & Redirection
      ↓
Networking
      ↓
SSH
      ↓
Storage
      ↓
Package Management
      ↓
systemd & Services
      ↓
Bash Scripting
      ↓
Automation
      ↓
Git
      ↓
Docker
      ↓
Troubleshooting
      ↓
System Administration
      ↓
LINUX POWER USER
      ↓
LINUX PROFESSIONAL

The 10 Commands You Should Know Extremely Well

If you want to prioritize, start with these:

cd
ls
find
grep
sed
awk
chmod
chown
ps
systemctl

Then learn how to combine them with:

|
>
>>
2>
&&
||
$
xargs
find -exec

For example:

ps aux | grep nginx

find /var/log -type f | grep nginx

journalctl -u nginx | grep -i error

du -ah /var | sort -rh | head -20

ps aux --sort=-%mem | head

These combinations demonstrate the real power of Linux: inspect → filter → transform → act → verify.

Final Challenge

Don't just read these 1,000 Linux shortcuts.

Practice them.

Create a virtual machine or safe test environment and work through progressively harder tasks:

Day 1–5
Terminal + Navigation

Day 6–10
Files + Permissions

Day 11–15
Text Processing

Day 16–20
Processes + Jobs

Day 21–25
Pipes + Redirection

Day 26–30
Networking

Day 31–35
SSH + Remote Administration

Day 36–40
Storage + Packages

Day 41–45
systemd + Logs

Day 46–50
Bash + Automation

Day 51–55
Git + Docker

Day 56–60
Troubleshooting + Advanced Workflows

By the end, your goal shouldn't be to say:

"I know 1,000 Linux commands."

Your goal should be:

"Give me a Linux problem, and I'll know how to investigate it from the terminal."

That is when you start becoming a Linux pro.

Middleware & Request-Response Pipeline

CodeWithDhanian — Sat, 15 Aug 2026 06:11:11 +0000

Middleware is one of the most important concepts in backend engineering because it controls what happens between receiving an HTTP request and sending the HTTP response.

In frameworks such as Express.js, middleware functions sit inside a request-processing pipeline. Each middleware can inspect the request, modify the request or response, perform business-independent work, terminate the request, or pass control forward with next(). ([Express.js][1])

A useful mental model is:

Client
  |
  | HTTP Request
  v
┌─────────────────────┐
│ Request Parsing     │
└──────────┬──────────┘
           v
┌─────────────────────┐
│ Logging Middleware  │
└──────────┬──────────┘
           v
┌─────────────────────┐
│ Authentication      │
└──────────┬──────────┘
           v
┌─────────────────────┐
│ Authorization        │
└──────────┬──────────┘
           v
┌─────────────────────┐
│ Validation          │
└──────────┬──────────┘
           v
┌─────────────────────┐
│ Route Handler       │
└──────────┬──────────┘
           v
┌─────────────────────┐
│ Database / Service  │
└──────────┬──────────┘
           v
┌─────────────────────┐
│ Error Middleware    │
└──────────┬──────────┘
           v
        Response

4.1 What Middleware Actually Is

An Express middleware function generally receives three important arguments:

(req, res, next)

req — the incoming HTTP request
res — the outgoing HTTP response
next — function used to transfer control to the next middleware

A minimal middleware looks like this:

function logger(req, res, next) {
  console.log(`${req.method} ${req.originalUrl}`);

  next();
}

app.use(logger);

The important operation is:

next();

If middleware does not send a response, end the request, or call next(), the request can remain unresolved. ([Express.js][1])

4.2 The Request-Response Lifecycle

Consider:

POST /api/users
Content-Type: application/json
Authorization: Bearer TOKEN

The backend can process it through multiple stages:

HTTP Request
     |
     v
Node.js HTTP Server
     |
     v
Express Application
     |
     v
Global Middleware
     |
     +---- Request ID
     |
     +---- Logging
     |
     +---- JSON Parsing
     |
     +---- CORS
     |
     +---- Security
     |
     v
Router
     |
     v
Authentication
     |
     v
Authorization
     |
     v
Validation
     |
     v
Controller
     |
     v
Service
     |
     v
Database
     |
     v
Controller
     |
     v
Response Middleware
     |
     v
HTTP Response

Middleware order matters. Express executes middleware in the order in which it is registered. ([Express.js][2])

For example:

app.use(authMiddleware);
app.use(validationMiddleware);

app.post("/users", createUser);

The request must pass through:

authMiddleware
      ↓
validationMiddleware
      ↓
createUser

Changing the order changes the behavior.

4.3 Middleware Can Do Four Fundamental Things

1. Execute logic

function logger(req, res, next) {
  console.log(req.method);
  next();
}

2. Modify the request

function attachRequestId(req, res, next) {
  req.requestId = crypto.randomUUID();
  next();
}

Later code can access:

console.log(req.requestId);

3. Modify the response

function securityHeaders(req, res, next) {
  res.setHeader("X-API-Version", "1");
  next();
}

4. End the request

function maintenanceMode(req, res, next) {
  return res.status(503).json({
    error: "Service temporarily unavailable"
  });
}

There is no next() because the middleware has already completed the request-response cycle. Express documents middleware as being able to execute code, modify req/res, end the cycle, or call next(). ([Express.js][1])

4.4 `next()` Is the Control Mechanism

Think of next() as:

"Middleware finished its job.
Continue processing this request."

Example:

function authenticate(req, res, next) {
  const token = req.headers.authorization;

  if (!token) {
    return res.status(401).json({
      error: "Authentication required"
    });
  }

  next();
}

The important distinction is:

next();

means:

Continue

while:

return res.status(401).json(...);

means:

Stop

A common mistake is accidentally doing both:

if (!token) {
  res.status(401).json({ error: "Unauthorized" });
}

next();

This can cause the pipeline to continue after a response has already been sent.

Prefer:

if (!token) {
  return res.status(401).json({
    error: "Unauthorized"
  });
}

next();

4.5 Application-Level Middleware

Application-level middleware applies to the entire application or a broad section of it.

app.use(express.json());

app.use((req, res, next) => {
  console.log(`${req.method} ${req.url}`);
  next();
});

For example:

Every Request
      |
      v
express.json()
      |
      v
Request Logger
      |
      v
Router

Express provides built-in middleware such as express.json(), express.urlencoded(), and express.static(). ([Express.js][1])

4.6 Router-Level Middleware

Large applications should not put every middleware function directly into app.js.

Use routers:

src/
├── app.js
├── middleware/
│   ├── auth.js
│   ├── validate.js
│   └── error.js
├── routes/
│   ├── users.js
│   └── orders.js
├── controllers/
│   ├── users.js
│   └── orders.js
└── services/
    ├── users.js
    └── orders.js

Example:

import express from "express";

const router = express.Router();

function authenticate(req, res, next) {
  const token = req.headers.authorization;

  if (!token) {
    return res.status(401).json({
      error: "Unauthorized"
    });
  }

  next();
}

router.use(authenticate);

router.get("/", (req, res) => {
  res.json({
    message: "Authenticated users"
  });
});

export default router;

Mount it:

app.use("/api/users", userRouter);

Now:

GET /api/users

passes through the router's authentication middleware before reaching the handler. Express supports both application-level and router-level middleware. ([Express.js][1])

4.7 Authentication Middleware

Authentication determines:

Who is making this request?

A simplified JWT middleware:

import jwt from "jsonwebtoken";

export function authenticate(req, res, next) {
  const header = req.headers.authorization;

  if (!header?.startsWith("Bearer ")) {
    return res.status(401).json({
      error: "Missing authentication token"
    });
  }

  const token = header.substring(7);

  try {
    const payload = jwt.verify(
      token,
      process.env.JWT_SECRET
    );

    req.user = payload;

    next();
  } catch {
    return res.status(401).json({
      error: "Invalid or expired token"
    });
  }
}

The pipeline becomes:

Request
   |
   v
Authorization Header
   |
   v
Extract Token
   |
   v
Verify Signature
   |
   +---- Invalid → 401
   |
   v
req.user = payload
   |
   v
next()
   |
   v
Controller

The middleware creates a controlled boundary between unauthenticated HTTP traffic and authenticated application logic.

4.8 Authorization Middleware

Authentication answers who you are.

Authorization answers what you are allowed to do.

export function requireRole(...allowedRoles) {
  return (req, res, next) => {
    if (!req.user) {
      return res.status(401).json({
        error: "Authentication required"
      });
    }

    if (!allowedRoles.includes(req.user.role)) {
      return res.status(403).json({
        error: "Forbidden"
      });
    }

    next();
  };
}

Use it:

router.delete(
  "/users/:id",
  authenticate,
  requireRole("admin"),
  deleteUser
);

The pipeline becomes:

Request
   ↓
authenticate
   ↓
requireRole("admin")
   ↓
deleteUser

4.9 Validation Middleware

Validation should happen before business logic.

function validateCreateUser(req, res, next) {
  const { name, email, password } = req.body;

  if (
    typeof name !== "string" ||
    typeof email !== "string" ||
    typeof password !== "string"
  ) {
    return res.status(400).json({
      error: "Invalid request body"
    });
  }

  if (password.length < 8) {
    return res.status(400).json({
      error: "Password must contain at least 8 characters"
    });
  }

  next();
}

Then:

router.post(
  "/",
  authenticate,
  validateCreateUser,
  createUser
);

This prevents the controller from becoming responsible for every HTTP-level validation rule.

4.10 Error-Handling Middleware

Errors should have a centralized path instead of every controller independently formatting errors.

Express identifies error-handling middleware through its four-argument signature:

(err, req, res, next)

([Express.js][1])

Example:

export function errorHandler(err, req, res, next) {
  console.error(err);

  const statusCode = err.statusCode || 500;

  res.status(statusCode).json({
    error: err.message || "Internal server error"
  });
}

app.use("/api/users", userRouter);
app.use("/api/orders", orderRouter);

app.use(errorHandler);

A typical structure is:

Request
   ↓
Middleware
   ↓
Router
   ↓
Controller
   ↓
Service
   ↓
Error
   ↓
next(error)
   ↓
Error Handler
   ↓
HTTP Response

4.11 Async Middleware

Modern backend applications perform many asynchronous operations:

Database queries
HTTP requests
File operations
Cache operations
Message queues
External APIs

Example:

async function loadUser(req, res, next) {
  try {
    const user = await userService.findById(req.params.id);

    if (!user) {
      return res.status(404).json({
        error: "User not found"
      });
    }

    req.userRecord = user;

    next();
  } catch (error) {
    next(error);
  }
}

The important pattern is:

await operation
      |
      +---- success → next()
      |
      +---- failure → next(error)

In current Express behavior, rejected promises from middleware are propagated to next() automatically in Express 5, while explicitly calling next(error) remains a clear pattern for asynchronous control flow. ([Express.js][2])

4.12 Complete Backend Pipeline

A clean production-oriented structure can look like:

import express from "express";
import crypto from "node:crypto";

const app = express();

app.use(express.json());

app.use((req, res, next) => {
  req.requestId = crypto.randomUUID();

  console.log({
    requestId: req.requestId,
    method: req.method,
    path: req.originalUrl
  });

  next();
});

function authenticate(req, res, next) {
  const token = req.headers.authorization;

  if (!token) {
    return res.status(401).json({
      error: "Authentication required"
    });
  }

  req.user = {
    id: "user-123",
    role: "admin"
  };

  next();
}

function requireRole(role) {
  return (req, res, next) => {
    if (req.user?.role !== role) {
      return res.status(403).json({
        error: "Forbidden"
      });
    }

    next();
  };
}

function validateUser(req, res, next) {
  const { name, email } = req.body;

  if (!name || !email) {
    return res.status(400).json({
      error: "name and email are required"
    });
  }

  next();
}

app.post(
  "/api/users",
  authenticate,
  requireRole("admin"),
  validateUser,
  async (req, res, next) => {
    try {
      const user = {
        id: crypto.randomUUID(),
        name: req.body.name,
        email: req.body.email
      };

      res.status(201).json(user);
    } catch (error) {
      next(error);
    }
  }
);

app.use((err, req, res, next) => {
  console.error({
    requestId: req.requestId,
    error: err.message
  });

  res.status(500).json({
    error: "Internal server error",
    requestId: req.requestId
  });
});

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

The complete execution path is:

POST /api/users
       |
       v
express.json()
       |
       v
Request ID Middleware
       |
       v
Authentication
       |
       v
Authorization
       |
       v
Validation
       |
       v
Route Handler
       |
       v
Business Logic
       |
       v
Response

If something fails:

Any Middleware / Controller
          |
          v
     next(error)
          |
          v
   Error Middleware
          |
          v
    HTTP Error Response

4.13 Middleware Composition

Middleware becomes powerful when small functions are composed.

Instead of one enormous function:

function everything(req, res, next) {
  // authentication
  // authorization
  // validation
  // logging
  // caching
  // rate limiting
  // business logic
  // error handling
}

Prefer:

router.post(
  "/orders",
  requestLogger,
  rateLimiter,
  authenticate,
  requireRole("customer"),
  validateOrder,
  createOrder
);

This creates a readable pipeline of responsibilities.

Each middleware should ideally have one clear responsibility.

4.14 Route-Specific vs Global Middleware

Global middleware:

app.use(requestLogger);

is appropriate when almost every request needs the behavior.

Route-specific middleware:

app.post(
  "/payments",
  authenticate,
  requireRole("admin"),
  processPayment
);

is appropriate when the behavior only applies to particular endpoints.

A practical structure is:

Global
├── JSON parsing
├── Request ID
├── CORS
├── Security headers
└── General logging

Router
├── Authentication
├── Authorization
├── Validation
└── Resource-specific rules

Controller
└── Business operation

4.15 The Most Important Middleware Rules

Middleware order is execution order.

next() transfers control forward.

next(error) transfers control to error-handling middleware.

Sending a response should normally terminate the current path.

Authentication should occur before protected business logic.

Authorization should occur after authentication.

Validation should happen before expensive business operations.

Error middleware should be centralized.

Middleware should remain focused and composable.

Business logic should not be buried inside generic middleware.

A well-designed request-response pipeline makes the backend easier to secure, test, debug, observe, and maintain. Express's middleware model is explicitly built around this ordered chain of functions that can modify requests and responses, terminate processing, or delegate to subsequent middleware. ([Express.js][1])

Backend Engineering Ebook

Grab the Backend Engineering ebook:
https://codewithdhanian.gumroad.com/l/ungqng

" width="800" height="1200">

Authentication & Authorization (JWT, OAuth2, Sessions)

CodeWithDhanian — Sat, 15 Aug 2026 05:38:54 +0000

Authentication and authorization are the security foundation of a backend application.

Authentication (AuthN) answers:

Who are you?

Authorization (AuthZ) answers:

What are you allowed to do?

A production backend usually combines password authentication, sessions or tokens, OAuth 2.0 / OpenID Connect, role-based authorization, secure cookies, MFA, and audit logging.

A secure design should also treat authentication, session management, and authorization as separate responsibilities.

1. Authentication vs Authorization

Consider an API endpoint:

DELETE /api/users/42

The backend must answer two different questions:

Authentication
    ↓
Is this request associated with a valid authenticated user?
    ↓
Authorization
    ↓
Does that authenticated user have permission to delete user 42?

A valid login does not automatically grant permission to perform every operation.

For example:

User: alice@example.com
Authenticated: YES
Role: USER

GET /api/profile
→ ALLOWED

DELETE /api/users/42
→ DENIED

An administrator might have:

User: admin@example.com
Authenticated: YES
Role: ADMIN

GET /api/profile
→ ALLOWED

DELETE /api/users/42
→ ALLOWED

This separation prevents a common security mistake: treating identity as permission.

2. The Complete Authentication Architecture

A typical backend authentication system looks like this:

                    ┌──────────────────────┐
                    │      Browser / App   │
                    └──────────┬───────────┘
                               │
                         HTTPS Request
                               │
                               ▼
                    ┌──────────────────────┐
                    │    API Gateway /     │
                    │    Load Balancer     │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ Authentication Layer │
                    │                      │
                    │ Session / JWT / OAuth│
                    └──────────┬───────────┘
                               │
                    Authenticated Identity
                               │
                               ▼
                    ┌──────────────────────┐
                    │ Authorization Layer  │
                    │                      │
                    │ RBAC / ABAC / Scopes │
                    └──────────┬───────────┘
                               │
                         Authorized Request
                               │
                               ▼
                    ┌──────────────────────┐
                    │ Application Services │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ PostgreSQL / Redis   │
                    └──────────────────────┘

The client should never be trusted to determine whether a request is authorized. The backend must enforce authorization on protected operations.

3. Password Authentication

A traditional authentication flow is:

Registration
    ↓
User submits email + password
    ↓
Validate input
    ↓
Hash password
    ↓
Store password hash
    ↓
Login
    ↓
Find user
    ↓
Verify password against hash
    ↓
Create session/token
    ↓
Return authenticated response

Never store:

password = "MyPassword123"

Store a strong password hash instead.

Modern password hashing should use a dedicated password-hashing algorithm such as Argon2id, bcrypt, or another appropriately configured password hashing mechanism. OWASP recommends secure password storage rather than reversible encryption or plaintext storage.

Node.js Example

npm install express argon2 jsonwebtoken cookie-parser

import express from "express";
import argon2 from "argon2";
import jwt from "jsonwebtoken";
import cookieParser from "cookie-parser";

const app = express();

app.use(express.json());
app.use(cookieParser());

const users = new Map();

app.post("/register", async (req, res) => {
  const { email, password } = req.body;

  if (!email || !password) {
    return res.status(400).json({
      error: "Email and password are required"
    });
  }

  if (users.has(email)) {
    return res.status(409).json({
      error: "User already exists"
    });
  }

  const passwordHash = await argon2.hash(password);

  users.set(email, {
    email,
    passwordHash,
    role: "USER"
  });

  return res.status(201).json({
    message: "User created"
  });
});

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

The important distinction is:

Plain password
     ↓
Argon2id
     ↓
Password hash
     ↓
Database

During login, the server does not decrypt a password. It verifies whether the submitted password matches the stored hash.

4. JWT Authentication

JSON Web Token (JWT) is a compact token format commonly used to represent claims between systems.

A JWT normally contains:

HEADER.PAYLOAD.SIGNATURE

Example structure:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
.
eyJzdWIiOiIxMjMiLCJyb2xlIjoiVVNFUiJ9
.
signature

The payload can contain claims such as:

{
  "sub": "123",
  "role": "USER",
  "iss": "api.example.com",
  "aud": "web-app",
  "exp": 1786780000
}

Important JWT claims include:

sub — subject or user identifier
iss — token issuer
aud — intended audience
exp — expiration time
iat — issued-at time
scope — permitted OAuth scopes

A JWT is signed, not automatically encrypted. Therefore, sensitive secrets should not be placed inside its payload.

JWT Authentication Flow

Client
  │
  │ POST /login
  │ email + password
  ▼
Backend
  │
  ├── Find user
  ├── Verify password
  └── Issue access token
  │
  ▼
Client
  │
  │ Authorization: Bearer <token>
  ▼
Authentication Middleware
  │
  ├── Verify signature
  ├── Validate expiration
  ├── Validate issuer
  └── Validate audience
  │
  ▼
Authorization Middleware
  │
  ├── Check role
  └── Check permissions
  │
  ▼
Protected Controller

Complete JWT Middleware

import jwt from "jsonwebtoken";

const JWT_SECRET = process.env.JWT_SECRET;

export function authenticate(req, res, next) {
  const authorization = req.headers.authorization;

  if (!authorization) {
    return res.status(401).json({
      error: "Authentication required"
    });
  }

  const [scheme, token] = authorization.split(" ");

  if (scheme !== "Bearer" || !token) {
    return res.status(401).json({
      error: "Invalid authorization header"
    });
  }

  try {
    const payload = jwt.verify(token, JWT_SECRET, {
      issuer: "api.example.com",
      audience: "web-app"
    });

    req.user = {
      id: payload.sub,
      role: payload.role
    };

    next();
  } catch {
    return res.status(401).json({
      error: "Invalid or expired token"
    });
  }
}

The middleware converts:

HTTP Authorization Header
            ↓
Token Verification
            ↓
Validated Identity
            ↓
req.user

A Bearer token must be protected carefully because possession of the token is generally sufficient to use it.

5. JWT Access Tokens and Refresh Tokens

A common architecture separates short-lived access tokens from long-lived refresh tokens.

Login
  │
  ├──────────────► Access Token
  │                 Short lifetime
  │                 Used for APIs
  │
  └──────────────► Refresh Token
                    Longer lifetime
                    Used to obtain new access tokens

Example:

Access Token
    ↓
15 minutes

Refresh Token
    ↓
Days / weeks depending on security requirements

When the access token expires:

Client
  │
  │ refresh token
  ▼
Backend
  │
  ├── Validate refresh token
  ├── Check revocation/session state
  └── Rotate refresh token
  │
  ▼
New access token

For browser applications, authentication credentials should generally not be stored in localStorage or sessionStorage, because JavaScript running in the origin can access them. OWASP recommends appropriately protected cookies or architectures such as a Backend-for-Frontend (BFF).

6. Secure Cookie Configuration

A session or refresh token stored in a cookie should use appropriate security attributes:

res.cookie("refresh_token", refreshToken, {
  httpOnly: true,
  secure: true,
  sameSite: "lax",
  path: "/auth",
  maxAge: 7 * 24 * 60 * 60 * 1000
});

Important attributes:

HttpOnly
    ↓
JavaScript cannot directly read the cookie

Secure
    ↓
Cookie is sent only over HTTPS

SameSite
    ↓
Controls cross-site cookie transmission

Path
    ↓
Limits where the cookie is sent

Max-Age
    ↓
Controls cookie lifetime

Cookie-based authentication must also consider CSRF protection, particularly when cookies are automatically attached to cross-site requests.

7. Server-Side Sessions

A session-based authentication system stores authentication state on the server.

The browser receives only an opaque session identifier:

Browser
   │
   │ session_id=abc123
   ▼
Backend
   │
   ▼
Session Store
   │
   └── abc123
        ├── userId: 42
        ├── role: USER
        └── expiresAt: ...

The session identifier should be random, unpredictable, and contain no sensitive user information. OWASP recommends strong server-generated session identifiers and careful lifecycle management.

Redis Session Example

import crypto from "crypto";

function createSession(userId) {
  const sessionId = crypto.randomBytes(32).toString("hex");

  const session = {
    userId,
    createdAt: Date.now(),
    expiresAt: Date.now() + 1000 * 60 * 60 * 24
  };

  return {
    sessionId,
    session
  };
}

The architecture becomes:

Client
  │
  │ Cookie: session_id
  ▼
API
  │
  ▼
Redis
  │
  └── session_id → user identity

This makes logout, session revocation, and server-side session invalidation straightforward.

8. OAuth 2.0

OAuth 2.0 is primarily an authorization framework, not a password-login protocol.

It allows a client application to obtain limited access to protected resources without receiving the resource owner's password.

The important roles are:

Resource Owner
      │
      ▼
Authorization Server
      │
      ▼
Client Application
      │
      ▼
Resource Server

For example:

User
 │
 │ "Allow application to access profile"
 ▼
Google Authorization Server
 │
 │ authorization code
 ▼
Application
 │
 │ exchange code
 ▼
Access Token
 │
 ▼
Protected API

For modern applications, the Authorization Code flow with PKCE is the standard pattern to understand, especially for public clients where a client secret cannot safely be kept confidential.

9. OAuth 2.0 Authorization Code + PKCE

The architecture:

┌────────────┐
│   Client   │
└─────┬──────┘
      │
      │ 1. Generate code_verifier
      │
      │ 2. Create code_challenge
      ▼
┌─────────────────────┐
│ Authorization       │
│ Server              │
└──────────┬──────────┘
           │
           │ 3. User authenticates
           │
           │ 4. User grants access
           ▼
      Authorization Code
           │
           ▼
┌──────────┴──────────┐
│ Client              │
│                     │
│ code + verifier     │
└──────────┬──────────┘
           │
           │ 5. Token request
           ▼
┌─────────────────────┐
│ Authorization       │
│ Server              │
└──────────┬──────────┘
           │
           │ 6. Access Token
           ▼
┌─────────────────────┐
│ Resource Server     │
└─────────────────────┘

PKCE binds the authorization-code exchange to the client instance that initiated the flow.

Conceptually:

code_verifier
      │
      ▼
SHA-256
      │
      ▼
code_challenge

The client sends the code_challenge during authorization and later sends the original code_verifier during token exchange.

The authorization server verifies that they match.

10. OAuth 2.0 Scopes

OAuth permissions should be expressed using narrow scopes.

Instead of:

access = everything

use:

profile:read
orders:read
orders:write

Example:

Authorization: Bearer ACCESS_TOKEN

The backend validates:

Token valid?
      ↓
Scope present?
      ↓
orders:write
      ↓
Allow operation

A token with:

scope = "orders:read"

should not be accepted for:

DELETE /orders/123

because deletion requires a stronger permission such as:

orders:delete

11. OpenID Connect

OAuth 2.0 provides authorization.

OpenID Connect (OIDC) adds an identity layer on top of OAuth 2.0.

The conceptual difference is:

OAuth 2.0
    ↓
"What can this application access?"

OpenID Connect
    ↓
"Who authenticated this user?"

OIDC commonly introduces an ID Token, which contains identity-related claims intended for the client.

A backend integrating with an identity provider should validate tokens according to the provider's documented issuer, audience, signature, expiration, and other required claims.

12. Authorization with RBAC

Role-Based Access Control (RBAC) assigns permissions through roles.

Example:

USER
 ├── profile:read
 └── order:create

EDITOR
 ├── profile:read
 ├── article:create
 └── article:update

ADMIN
 ├── user:read
 ├── user:update
 ├── user:delete
 └── system:manage

Authorization Middleware

function authorize(...allowedRoles) {
  return (req, res, next) => {
    if (!req.user) {
      return res.status(401).json({
        error: "Authentication required"
      });
    }

    if (!allowedRoles.includes(req.user.role)) {
      return res.status(403).json({
        error: "Forbidden"
      });
    }

    next();
  };
}

Usage:

app.delete(
  "/api/users/:id",
  authenticate,
  authorize("ADMIN"),
  deleteUser
);

The pipeline becomes:

Request
   ↓
authenticate()
   ↓
Identity established
   ↓
authorize("ADMIN")
   ↓
Permission verified
   ↓
deleteUser()

401 Unauthorized generally means the request lacks valid authentication.

403 Forbidden means the server understands the identity but refuses the requested operation.

13. Resource-Level Authorization

Checking only the user's role is not always sufficient.

Suppose:

GET /api/orders/500

The user may have:

role = USER

but the backend must additionally verify:

Does order 500 belong to this user?

Correct:

const order = await db.order.findUnique({
  where: {
    id: req.params.id
  }
});

if (!order) {
  return res.status(404).json({
    error: "Order not found"
  });
}

if (order.userId !== req.user.id && req.user.role !== "ADMIN") {
  return res.status(403).json({
    error: "Forbidden"
  });
}

This protects against broken object-level authorization, where an authenticated user attempts to access another user's resource by changing an identifier.

14. RBAC vs ABAC

RBAC evaluates roles:

role = ADMIN

Attribute-Based Access Control (ABAC) evaluates multiple attributes:

user.department
resource.owner
resource.classification
request.time
request.location
operation

Example policy:

ALLOW

IF
    user.role = MANAGER
AND
    user.department = resource.department
AND
    operation = "READ"

This becomes valuable in large systems where simple roles are insufficient.

15. Authentication Middleware Structure

A production backend can separate security responsibilities into layers:

HTTP Request
     │
     ▼
Rate Limiting
     │
     ▼
Authentication
     │
     ▼
Identity Context
     │
     ▼
Authorization
     │
     ▼
Validation
     │
     ▼
Business Logic
     │
     ▼
Database

A clean project structure might look like:

src/
├── auth/
│   ├── auth.controller.js
│   ├── auth.service.js
│   ├── auth.middleware.js
│   ├── token.service.js
│   ├── password.service.js
│   └── authorization.js
│
├── users/
│   ├── user.controller.js
│   ├── user.service.js
│   └── user.repository.js
│
├── middleware/
│   ├── rateLimit.js
│   ├── validation.js
│   └── errorHandler.js
│
├── routes/
│   ├── auth.routes.js
│   └── user.routes.js
│
├── database/
│   └── client.js
│
└── server.js

This prevents authentication logic from becoming mixed into every controller.

16. Complete Protected API Example

import express from "express";
import jwt from "jsonwebtoken";

const app = express();

app.use(express.json());

const JWT_SECRET = process.env.JWT_SECRET;

function authenticate(req, res, next) {
  const header = req.headers.authorization;

  if (!header) {
    return res.status(401).json({
      error: "Missing Authorization header"
    });
  }

  const [scheme, token] = header.split(" ");

  if (scheme !== "Bearer" || !token) {
    return res.status(401).json({
      error: "Invalid Authorization header"
    });
  }

  try {
    const payload = jwt.verify(token, JWT_SECRET, {
      issuer: "api.example.com",
      audience: "web-app"
    });

    req.user = {
      id: payload.sub,
      role: payload.role
    };

    next();
  } catch {
    return res.status(401).json({
      error: "Invalid or expired access token"
    });
  }
}

function authorize(...roles) {
  return (req, res, next) => {
    if (!roles.includes(req.user.role)) {
      return res.status(403).json({
        error: "Insufficient permissions"
      });
    }

    next();
  };
}

app.get("/api/profile", authenticate, (req, res) => {
  res.json({
    userId: req.user.id,
    role: req.user.role
  });
});

app.delete(
  "/api/users/:id",
  authenticate,
  authorize("ADMIN"),
  async (req, res) => {
    // Delete user from database here.

    res.json({
      message: "User deleted"
    });
  }
);

app.listen(3000, () => {
  console.log("API running on port 3000");
});

The important security boundary is:

authenticate()
      ↓
WHO IS THE USER?
      ↓
authorize()
      ↓
WHAT MAY THE USER DO?
      ↓
controller
      ↓
business operation

17. Multi-Factor Authentication

Multi-Factor Authentication (MFA) combines independent authentication factors.

Common factors include:

Something you know
    → Password

Something you have
    → Security key / authenticator device

Something you are
    → Biometric characteristic

A stronger login might therefore become:

Email + Password
       ↓
Password Verification
       ↓
MFA Challenge
       ↓
MFA Verification
       ↓
Authenticated Session

MFA is especially important for administrative accounts, financial operations, account recovery, and other high-impact actions. OWASP recommends stronger authentication and re-authentication for sensitive operations and risk events.

18. Session Lifecycle

Authentication is not complete when a user logs in.

A secure session has a lifecycle:

CREATE
  ↓
AUTHENTICATED
  ↓
ACTIVE
  ↓
RENEWED
  ↓
EXPIRED / REVOKED
  ↓
DESTROYED

Important events include:

login
logout
session_created
session_renewed
session_expired
session_revoked
password_changed
MFA_enabled
MFA_failed
authorization_denied

Session identifiers should be regenerated when appropriate, particularly across authentication state transitions, to reduce session fixation risks.

19. Logout and Token Revocation

With server-side sessions:

POST /logout
     ↓
Destroy session
     ↓
Clear cookie

With refresh-token systems:

POST /logout
     ↓
Revoke refresh token
     ↓
Clear authentication cookie

A simple cookie cleanup:

res.clearCookie("refresh_token", {
  httpOnly: true,
  secure: true,
  sameSite: "lax",
  path: "/auth"
});

res.status(204).send();

For high-security systems, maintain server-side refresh-token/session state so stolen credentials can be revoked before their natural expiration.

20. Authentication Security Controls

A production authentication system should include:

HTTPS / TLS
    +
Secure password hashing
    +
MFA
    +
Rate limiting
    +
Credential stuffing protection
    +
Account recovery protection
    +
Secure session management
    +
Short-lived access tokens
    +
Refresh-token rotation
    +
Token/session revocation
    +
Authorization checks
    +
Audit logging
    +
Re-authentication for sensitive operations

Authentication endpoints should also avoid revealing whether an account exists.

Instead of:

{
  "error": "Email does not exist"
}

prefer a generic response such as:

{
  "error": "Invalid email or password"
}

This reduces user enumeration.

Authentication systems should also log security-relevant events while avoiding passwords, access tokens, refresh tokens, and other secrets in logs. OWASP specifically recommends monitoring the lifecycle of sessions and authentication-related events.

21. Authentication Decision Structure

                         REQUEST
                            │
                            ▼
                    Is HTTPS enabled?
                       │          │
                      NO         YES
                       │          │
                    REJECT        ▼
                              Authenticate
                                   │
                          ┌────────┴────────┐
                          │                 │
                       INVALID            VALID
                          │                 │
                       401                  ▼
                                      Authorize
                                          │
                                 ┌────────┴────────┐
                                 │                 │
                              DENIED             ALLOWED
                                 │                 │
                                403                 ▼
                                                  API
                                                  │
                                                  ▼
                                               Database

The core security model is therefore:

IDENTITY
   ↓
AUTHENTICATION
   ↓
SESSION / TOKEN
   ↓
AUTHORIZATION
   ↓
PERMISSION
   ↓
RESOURCE

Grab the Backend Engineering ebook

CRUD Operations with a Relational Database (SQL)

CodeWithDhanian — Thu, 13 Aug 2026 13:21:04 +0000

CRUD stands for Create, Read, Update, and Delete—the four fundamental operations performed on data in almost every backend application. Whether building an e-commerce platform, banking system, hospital management system, social media application, or inventory management system, every backend revolves around storing, retrieving, modifying, and removing structured data efficiently.

A Relational Database Management System (RDBMS) stores information inside tables, where data is organized into rows and columns. Relationships between tables are established using Primary Keys and Foreign Keys, allowing complex applications to maintain data consistency, integrity, and normalization.

Core CRUD Operations

1. CREATE — Inserting New Records

The Create operation inserts new information into a database.

Example Users table:

id	name	email
1	Alice	alice@example.com

Adding another user:

INSERT INTO users (name, email)
VALUES ('John Doe', 'john@example.com');

Explanation

INSERT INTO specifies the target table.
(name, email) lists the columns receiving values.
VALUES contains the actual data.
The id is usually generated automatically using AUTO_INCREMENT, SERIAL, or IDENTITY depending on the database.

2. READ — Retrieving Data

The Read operation fetches stored information.

Retrieve every user:

SELECT *
FROM users;

Retrieve only specific columns:

SELECT id, name, email
FROM users;

Retrieve a single user:

SELECT *
FROM users
WHERE id = 1;

Retrieve users sorted alphabetically:

SELECT *
FROM users
ORDER BY name ASC;

Limit returned rows:

SELECT *
FROM users
LIMIT 10;

Important SQL keywords

SELECT chooses columns.
FROM specifies the table.
WHERE filters records.
ORDER BY sorts results.
LIMIT restricts the number of returned rows.

3. UPDATE — Modifying Existing Data

The Update operation changes existing records.

UPDATE users
SET email = 'newemail@example.com'
WHERE id = 1;

Explanation

UPDATE specifies the table.
SET assigns new values.
WHERE identifies which row to modify.

Without a WHERE clause:

UPDATE users
SET email = 'changed@example.com';

Every row in the table would be updated, making WHERE one of the most important clauses in SQL.

4. DELETE — Removing Records

Delete a specific record:

DELETE FROM users
WHERE id = 1;

Delete every record:

DELETE FROM users;

Completely remove the table:

DROP TABLE users;

These commands are very different:

DELETE removes rows.
DROP TABLE removes the entire table structure and its data.

Complete Database Structure

Database
│
├── users
│   ├── id (Primary Key)
│   ├── name
│   ├── email
│   └── created_at
│
├── products
│   ├── id (Primary Key)
│   ├── name
│   ├── price
│   └── stock
│
└── orders
    ├── id (Primary Key)
    ├── user_id (Foreign Key)
    ├── product_id (Foreign Key)
    ├── quantity
    └── order_date

Here:

users.id uniquely identifies every user.
products.id uniquely identifies every product.
orders.user_id references users.id.
orders.product_id references products.id.

This relationship prevents duplicate information and keeps data consistent across multiple tables.

Creating Tables

CREATE TABLE users (
    id INT PRIMARY KEY AUTO_INCREMENT,
    name VARCHAR(100) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Column breakdown

INT stores whole numbers.
PRIMARY KEY uniquely identifies each row.
AUTO_INCREMENT automatically generates IDs.
VARCHAR(100) stores variable-length text.
NOT NULL requires a value.
UNIQUE prevents duplicate emails.
TIMESTAMP records creation time automatically.

Typical Backend CRUD Flow

Client
   │
   ▼
HTTP Request
   │
   ▼
Backend API
   │
   ▼
Validation
   │
   ▼
Business Logic
   │
   ▼
SQL Query
   │
   ▼
Relational Database
   │
   ▼
Query Result
   │
   ▼
Backend Response
   │
   ▼
Client

Each request follows this sequence:

Client sends an HTTP request.
Backend API receives it.
Input is validated.
Business logic determines what should happen.
SQL performs the required CRUD operation.
Database executes the query.
Results are returned.
Backend formats the response as JSON.
Client displays the updated information.

Example CRUD API (Node.js + Express)

import express from "express";

const app = express();

app.use(express.json());

const users = [];

// CREATE
app.post("/users", (req, res) => {
  users.push(req.body);
  res.status(201).json(req.body);
});

// READ
app.get("/users", (req, res) => {
  res.json(users);
});

// UPDATE
app.put("/users/:id", (req, res) => {
  users[req.params.id] = req.body;
  res.json(users[req.params.id]);
});

// DELETE
app.delete("/users/:id", (req, res) => {
  users.splice(req.params.id, 1);
  res.status(204).send();
});

app.listen(3000);

This example demonstrates the four CRUD endpoints:

POST /users creates a new resource.
GET /users retrieves resources.
PUT /users/:id updates an existing resource.
DELETE /users/:id removes a resource.

In production, the users array would be replaced by SQL queries against a relational database.

Best Practices

Always validate incoming data before executing SQL.
Use Primary Keys to uniquely identify records.
Enforce Foreign Keys to maintain relationships.
Use transactions when multiple related operations must either all succeed or all fail.
Avoid SELECT * when only specific columns are required.
Always include WHERE when updating or deleting individual records.
Parameterize queries to prevent SQL Injection attacks.
Create indexes on frequently searched columns to improve query performance.
Normalize tables to eliminate redundant data while preserving consistency.

Backend Engineering eBook

Build a stronger foundation in backend development with the complete Backend Engineering eBook:

https://codewithdhanian.gumroad.com/l/ungqng

Learn HTTP & RESTful API Design

CodeWithDhanian — Thu, 13 Aug 2026 13:05:28 +0000

HTTP (Hypertext Transfer Protocol) is the application-layer communication protocol that allows clients and servers to exchange information across a network. Every modern web application, mobile application, backend service, microservice, and cloud platform relies on HTTP to send requests and receive responses.

A RESTful API (Representational State Transfer Application Programming Interface) is a collection of HTTP endpoints that expose resources using predictable URLs, standard HTTP methods, meaningful status codes, and structured JSON responses.

Unlike direct database access, clients never communicate with databases. Every interaction passes through a backend API where validation, authentication, authorization, business logic, and security are enforced.

Client–Server Architecture

Every HTTP request follows a predictable path.

+-------------------+
| Client            |
| Browser / Mobile  |
+---------+---------+
          |
          | HTTP Request
          v
+---------+---------+
| Web Server        |
| Nginx / Apache    |
+---------+---------+
          |
          |
          v
+---------+---------+
| Backend API       |
| Express / Django  |
| Spring Boot       |
+---------+---------+
          |
          |
          v
+---------+---------+
| Business Logic    |
+---------+---------+
          |
          |
          v
+---------+---------+
| Database          |
| PostgreSQL/MySQL  |
+-------------------+

The client only knows how to call the API. The backend decides how data is processed, where it is stored, and what response should be returned.

HTTP Request Structure

Every HTTP request contains several important parts.

GET /users/15 HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhb...
Content-Type: application/json
Accept: application/json

Request Line

Defines:

HTTP Method
Resource Path
HTTP Version

Example

GET /users/15 HTTP/1.1

Headers

Headers provide additional information.

Authorization: Bearer TOKEN
Content-Type: application/json
Accept: application/json
User-Agent: Chrome

Examples:

Authorization → identifies the authenticated user.
Content-Type → tells the server what format the body uses.
Accept → tells the server which response formats are acceptable.

Body

Only requests like POST, PUT, and PATCH usually include a body.

{
    "name": "Alice",
    "email": "alice@example.com"
}

The backend parses this JSON before validating it.

HTTP Response Structure

Example response:

HTTP/1.1 200 OK

Content-Type: application/json

{
    "id": 15,
    "name": "Alice",
    "email": "alice@example.com"
}

A response contains:

Status Line
Headers
Body

HTTP Methods

REST maps different operations to different HTTP methods.

Method	Purpose
GET	Retrieve data
POST	Create a resource
PUT	Replace an entire resource
PATCH	Update specific fields
DELETE	Remove a resource

Example resource:

/users

Operations become

GET    /users
GET    /users/15

POST   /users

PUT    /users/15

PATCH  /users/15

DELETE /users/15

Each endpoint performs exactly one responsibility.

REST Resource Design

REST is resource-oriented, not action-oriented.

Good

/users
/products
/orders
/invoices

Poor

/createUser
/getUsers/deleteUser
/updateUser

Resources should be represented as nouns, while the HTTP method describes the action.

URL Hierarchy

Well-designed URLs represent relationships.

/users

/users/15

/users/15/orders

/users/15/orders/8

The URL itself explains the resource hierarchy.

Building a REST API with Express

const express = require("express");

const app = express();

app.use(express.json());

let users = [
    {
        id: 1,
        name: "Alice",
        email: "alice@example.com"
    }
];

The server:

enables JSON parsing
stores sample data
exposes REST endpoints

GET

app.get("/users", (req, res) => {
    res.json(users);
});

Returns every user.

GET by ID

app.get("/users/:id", (req, res) => {

    const user = users.find(
        u => u.id === Number(req.params.id)
    );

    if (!user) {
        return res.status(404).json({
            message: "User not found"
        });
    }

    res.json(user);
});

The route parameter

:id

captures the requested resource identifier.

POST

app.post("/users", (req, res) => {

    const user = {
        id: users.length + 1,
        name: req.body.name,
        email: req.body.email
    };

    users.push(user);

    res.status(201).json(user);

});

This endpoint:

receives JSON
creates a new resource
returns 201 Created

HTTP Status Codes

Every response should communicate what happened.

Code	Meaning
200 OK	Request succeeded
201 Created	Resource created
204 No Content	Success without response body
400 Bad Request	Invalid client input
401 Unauthorized	Authentication required
403 Forbidden	Permission denied
404 Not Found	Resource does not exist
409 Conflict	Duplicate or conflicting data
422 Unprocessable Entity	Validation failed
500 Internal Server Error	Unexpected server failure

Choosing the correct status code makes APIs easier to consume and debug.

JSON Response Standards

Successful responses should remain consistent.

{
    "success": true,
    "data": {
        "id": 15,
        "name": "Alice"
    }
}

Error responses should also follow a predictable structure.

{
    "success": false,
    "message": "Email already exists",
    "errors": [
        {
            "field": "email",
            "message": "Duplicate email"
        }
    ]
}

Consistent response formats simplify frontend development and automated API integrations.

REST Design Principles

A well-designed REST API follows several fundamental principles:

Stateless Communication — Every request contains all information required to process it. The server does not depend on previous requests.
Resource-Based URLs — Endpoints represent resources instead of actions.
Uniform Interface — Similar resources follow the same naming and behavior patterns.
Standard HTTP Methods — Use GET, POST, PUT, PATCH, and DELETE according to their intended purpose.
Meaningful Status Codes — Responses clearly indicate whether an operation succeeded or failed.
JSON Payloads — Structured, language-independent data format for requests and responses.
Idempotency — Repeating GET, PUT, or DELETE should produce the same result without unintended side effects.

Backend Engineering eBook

Master backend engineering from fundamentals to advanced concepts with practical projects and production-ready examples:

Backend Engineering eBook: https://codewithdhanian.gumroad.com/l/ungqng

How RAG actually works (clearly explained in under 2 mins):

CodeWithDhanian — Fri, 07 Aug 2026 08:55:49 +0000

RAG (Retrieval-Augmented Generation) is a system that retrieves relevant data and feeds it into an LLM before generating a response.

It lets models answer questions using external knowledge, not just what they were trained on.

If you’re building with these patterns, here's a great guide on scaling multi-agent RAG systems: https://codewithdhanian.gumroad.com/l/miwqgy

Here’s a simple mental model to understand it:

𝟭) 𝗗𝗮𝘁𝗮 𝗶𝘀 𝗶𝗻𝗴𝗲𝘀𝘁𝗲𝗱
↳ Documents (PDFs, docs, APIs) are collected and split into chunks
↳ Each chunk is cleaned and formatted ready for embedding

𝟮) 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 𝗮𝗿𝗲 𝗰𝗿𝗲𝗮𝘁𝗲𝗱
↳ Each chunk is converted into a vector representation
↳ Similar meaning → closer vectors

𝟯) 𝗗𝗮𝘁𝗮 𝗶𝘀 𝘀𝘁𝗼𝗿𝗲𝗱
↳ Vectors are stored in a vector database
↳ Enables fast similarity search across large datasets

𝟰) 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝘁 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗶𝘀 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗲𝗱
↳ The user's query is converted into an embedding (vector representation)
↳ The system compares it against stored vectors and retrieves the most relevant chunks

𝟱) 𝗧𝗵𝗲 𝗟𝗟𝗠 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝘀 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿
↳ The query + retrieved context are combined into a prompt
↳ The model generates a grounded response

That's the foundation of RAG. There are several types of RAG, each designed for different use cases and levels of complexity.

If you’re curious what this actually looks like in practice (beyond diagrams), this repo is a great place to start: https://codewithdhanian.gumroad.com/l/miwqgy

It has:
↳ E2E implementations of RAG, AI applications, agents, and systems
↳ Resources covering AI agent architecture, reasoning strategies, and memory systems
↳ Hands-on workshops and guided learning

Star it to keep it bookmarked. This repo will keep growing, and you'll want it on hand as you build.

What else would you add?

——

♻️ Repost to help others learn AI engineering.
🙏 Remember to bookmark.
➕ Follow me ( @e_opore ) to improve at AI engineering.

Day 5: Inserting Data – INSERT INTO, Multiple Rows, and RETURNING

CodeWithDhanian — Sun, 24 May 2026 12:25:46 +0000

Introduction: Why Data Insertion Matters in Real-World Development

In any production database system, inserting data is the primary way information enters your application. Whether you are building a user registration flow, logging e-commerce orders, importing CSV analytics data, or syncing IoT sensor readings, the INSERT statement is the gateway between your application code and persistent storage.

Mastering insertion is not just about syntax — it directly impacts:

Application performance (single-row vs. bulk operations)
Data integrity (handling defaults, constraints, and generated values)
User experience (returning fresh IDs or computed fields instantly)
Scalability (avoiding costly round-trips between app and database)

By the end of this tutorial you will confidently insert single records, bulk-load hundreds or thousands of rows efficiently, retrieve auto-generated values in one atomic operation using the RETURNING clause (PostgreSQL), and apply production-grade best practices that senior engineers use daily.

We will use PostgreSQL for the main examples because the RETURNING clause is native and extremely powerful here. MySQL equivalents and differences are clearly noted so you can apply the same concepts regardless of your stack.

1. The INSERT INTO Statement – Core Concepts

The INSERT statement tells the database: “Here is new data — please store it according to the table’s structure.”

Basic Syntax

INSERT INTO table_name (column1, column2, ..., columnN)
VALUES (value1, value2, ..., valueN);

You explicitly list the columns you want to populate.
The database fills any omitted columns with their DEFAULT value or NULL (if allowed).
Order of values must match the order of columns in the parentheses.

Why list columns explicitly?

It makes your code resilient to future schema changes (new columns added later) and improves readability/maintainability.

2. Inserting a Single Row – Step-by-Step

Assume we have this table (created in Day 4):

CREATE TABLE products (
    product_id SERIAL PRIMARY KEY,        -- Auto-incrementing ID (PostgreSQL)
    name TEXT NOT NULL,
    price NUMERIC(10,2) DEFAULT 0.00,
    stock INTEGER DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

Single-row insertion example

-- Example 1: Full column list (most explicit and safest)
INSERT INTO products (name, price, stock)
VALUES ('Wireless Bluetooth Headphones', 89.99, 150);

-- Example 2: Omit columns that have defaults or are auto-generated
INSERT INTO products (name, price)
VALUES ('USB-C Cable', 12.50);

What happens behind the scenes

Database validates constraints (NOT NULL, data types, CHECK constraints).
Generates default values where needed.
Writes the row to disk (within a transaction).
Returns success or raises an error.

MySQL note: Use the same syntax. The SERIAL type becomes AUTO_INCREMENT and the primary key.

3. Inserting Multiple Rows Efficiently

Inserting rows one by one in a loop is a performance anti-pattern. Databases are optimized for set-based operations.

Multi-row INSERT syntax

INSERT INTO products (name, price, stock)
VALUES 
    ('Smart Watch', 249.99, 75),
    ('Laptop Stand', 34.50, 200),
    ('Wireless Mouse', 29.99, 120);

Benefits

One network round-trip instead of dozens.
Single transaction → atomicity.
Database can optimize batch writing and indexing.

Production tip: For thousands of rows, use PostgreSQL’s COPY command or prepared statements with batching in your application language (e.g., pg driver in Node.js with multi-row mode).

4. Handling NULL, DEFAULT, and Constraints

-- Explicitly set NULL (only if column allows it)
INSERT INTO products (name, price, stock)
VALUES ('Test Product', NULL, NULL);

-- Let DEFAULT kick in
INSERT INTO products (name)
VALUES ('Free Sample Item');

Warning: Never rely on implicit column order for critical applications. Always name columns.

5. The RETURNING Clause – The Magic of Getting Data Back Instantly (PostgreSQL)

One of the most powerful features in PostgreSQL. After inserting, you can immediately retrieve any columns — especially auto-generated IDs, timestamps, or computed values — without a second SELECT.

Syntax

INSERT INTO products (name, price, stock)
VALUES ('4K Webcam', 129.99, 40)
RETURNING *;                    -- Return every column

Practical examples

-- Return only the new ID and creation timestamp
INSERT INTO products (name, price, stock)
VALUES ('Mechanical Keyboard', 79.99, 60)
RETURNING product_id, created_at;

-- Return multiple rows when inserting many
INSERT INTO products (name, price, stock)
VALUES 
    ('HDMI Cable', 8.99, 300),
    ('Monitor Arm', 45.00, 25)
RETURNING product_id, name, created_at;

Real-world use case: In a REST API, when a user creates a new order, you can return the full order object (including the generated order_id) in the same HTTP response. No extra query needed.

MySQL equivalent (versions 8.0.21+): MySQL now supports RETURNING in INSERT, but older versions require LAST_INSERT_ID() after the insert:

-- MySQL alternative
INSERT INTO products (name, price, stock) VALUES ('Example', 19.99, 10);
SELECT LAST_INSERT_ID() AS product_id;

6. Best Practices & Production Techniques

Always name columns — never use INSERT INTO table VALUES (...) in production code.
Use parameterized queries / prepared statements to prevent SQL injection.
Batch inserts for bulk data (100–10,000 rows).
Wrap in transactions when inserting related data across multiple tables.
Validate data in the application layer before hitting the database.
Monitor insert performance with EXPLAIN ANALYZE on large operations.
Consider ON CONFLICT (PostgreSQL) or INSERT ... ON DUPLICATE KEY UPDATE (MySQL) for upsert patterns.

Example of safe, parameterized insert (conceptual — shown in SQL for clarity)

-- In application code you would use placeholders
PREPARE insert_product AS
INSERT INTO products (name, price, stock)
VALUES ($1, $2, $3)
RETURNING product_id;

7. Common Pitfalls & Troubleshooting

Pitfall	Symptom	Fix
Forgetting column names	Data goes into wrong columns	Always list columns
Violating NOT NULL	Error: null value in column	Provide value or DEFAULT
Type mismatch	Error: invalid input syntax	Cast or format correctly
Forgetting RETURNING	Extra SELECT query needed	Add RETURNING when possible
Inserting in loop	Slow performance, high latency	Use multi-row or COPY

8. Real-World Project Example: E-Commerce Product Import

Imagine you receive a CSV file from your supplier with 5,000 new products.

Instead of looping 5,000 times:

-- PostgreSQL efficient bulk import (Day 5 level)
COPY products (name, price, stock)
FROM '/path/to/products.csv'
DELIMITER ','
CSV HEADER;

Then query the newly imported rows using RETURNING or a timestamp filter.

This pattern is used by Shopify, Amazon, and every serious e-commerce platform.

Conclusion

You now understand not just how to insert data, but why certain patterns matter in production systems. The combination of explicit column lists, multi-row inserts, and the RETURNING clause gives you both safety and speed — the hallmarks of professional database engineering.

Key Takeaways

Always specify column names in INSERT statements.
Prefer multi-row VALUES for performance.
Use RETURNING (PostgreSQL) to eliminate extra queries.
Validate early, insert atomically, and monitor with EXPLAIN.
Master these fundamentals before moving to advanced topics like triggers or stored procedures.

You have completed Day 5. Your database now has data — the foundation for every query, join, and report you will build in the coming days.

Day 4: Creating Tables – Data Types, NULL, and DEFAULT Constraints

CodeWithDhanian — Sun, 24 May 2026 02:46:06 +0000

The Foundation of Every Relational Database: Why Table Creation Matters

When you build any production application—whether it is a customer management system, an inventory tracker, or a full-scale e-commerce platform—the table becomes the core structural unit that holds your data. A table in a relational database is not just a spreadsheet; it is a precisely defined container that enforces rules about what data can live inside it, how that data is stored, and how the database engine will interact with it for years to come.

The CREATE TABLE statement is the very first place where you, as a developer or architect, make decisions that affect performance, data integrity, storage costs, query speed, and even the long-term maintainability of your entire system. Today we will explore this statement in exhaustive detail: from the underlying theory of data types to the subtle but critical behavior of NULL values and the practical power of DEFAULT constraints. We will examine real engineering trade-offs, internal database workflows, and the exact syntax you will use in both PostgreSQL and MySQL—the two databases you set up on Day 2.

The CREATE TABLE Statement: Syntax, Structure, and Execution Flow

At its core, the CREATE TABLE command instructs the database management system to allocate storage space, define column metadata, and register the table within the database’s system catalog. The engine parses your statement, validates data-type compatibility, applies any immediate constraints, and then creates the physical files (or pages) that will eventually hold rows.

The basic structure looks like this:

CREATE TABLE table_name (
    column_name1 data_type [constraints],
    column_name2 data_type [constraints],
    ...
);

Every column definition consists of three mandatory parts and one optional but extremely important part:

The column name (must be unique within the table)
The data type (determines storage format, range, and behavior)
Optional constraints such as NOT NULL or DEFAULT

When the database engine processes this statement, it performs several internal steps in sequence:

Acquires a schema lock to prevent concurrent modifications.
Validates that the chosen data types exist and are compatible with the storage engine.
Allocates space in the heap (the main table storage area).
Records the table definition in the system catalog tables (pg_class and pg_attribute in PostgreSQL, or information_schema.columns in both engines).
If the table is created successfully, it becomes immediately queryable.

Choosing the wrong data type or forgetting a constraint at this stage can force you to run expensive ALTER TABLE operations later—which lock the table and can cause downtime in high-traffic systems.

Data Types: The Language Your Database Speaks

Every value you store must be declared with an explicit data type. The database engine uses this declaration to decide how much disk space to reserve, how to encode the value in memory, how to compare values during queries, and how to index the column efficiently. Selecting the correct data type is both an art and a science; it directly impacts storage costs, query performance, and application correctness.

Numeric Data Types

Integer types are the workhorses of most applications. In PostgreSQL you have SMALLINT (2 bytes, ±32,767), INTEGER (4 bytes, ±2.1 billion), and BIGINT (8 bytes, ±9 quintillion). MySQL offers the nearly identical TINYINT, SMALLINT, MEDIUMINT, INT, and BIGINT.

For fractional numbers, use NUMERIC(precision, scale) (also called DECIMAL in MySQL). This type stores exact decimal values—critical for financial data—unlike REAL or DOUBLE PRECISION, which use floating-point approximation and can introduce rounding errors that accumulate over millions of transactions.

Real-world example: an order total column should never use DOUBLE PRECISION; always use NUMERIC(12,2) to guarantee penny-level accuracy across years of compounding calculations.

Character Data Types

Text storage comes in two families. VARCHAR(n) stores variable-length strings up to n characters and is the default choice in modern applications. TEXT (or LONGTEXT in MySQL) removes the length limit entirely and is ideal for product descriptions or user comments.

CHAR(n) is fixed-length and right-pads with spaces; it can be slightly faster for very short, consistently sized fields (e.g., country codes), but it wastes space otherwise. In high-scale systems, the performance difference between VARCHAR and TEXT is negligible because both are stored out-of-line when they grow large, but the choice still matters for index size and memory usage during sorting.

Date and Time Data Types

DATE, TIME, TIMESTAMP, and TIMESTAMPTZ (timestamp with time zone in PostgreSQL) are essential. PostgreSQL’s TIMESTAMPTZ automatically normalizes everything to UTC internally while preserving the original time zone for display—an architectural decision that prevents daylight-saving bugs that plague many legacy MySQL applications using plain DATETIME.

Always store timestamps with time zone awareness unless your entire system operates in a single fixed timezone. The internal workflow is simple: the engine converts the input to UTC on write and converts back on read based on the client’s session timezone.

Boolean and Specialized Types

The BOOLEAN type (or BOOL) stores true, false, or NULL. It occupies only one byte yet saves you from the classic “0/1 magic number” anti-pattern that leads to bugs when someone later interprets 2 as “maybe.”

PostgreSQL also offers JSONB for semi-structured data and UUID for globally unique identifiers—both of which we will use in production examples later in the series.

NULL: The Concept of “Value Unknown”

NULL is not a value; it is the explicit absence of a value. The SQL standard treats NULL through three-valued logic (true, false, unknown). This means that any comparison involving NULL—even NULL = NULL—evaluates to unknown, not true.

From an architectural standpoint, the database engine stores a special null bitmap for each row. When a column is NULL, the engine skips storing any data for that column, saving space. However, this bitmap adds a tiny overhead to every row, which becomes measurable only at billions of rows.

In real systems, NULL represents legitimate business states: “user has not yet set a middle name,” “order has not yet shipped,” or “product has no discount expiration date.” Misusing NULL for “zero” or “empty string” is one of the most common sources of incorrect analytics.

Enforcing Presence with NOT NULL Constraints

The NOT NULL constraint tells the database engine to reject any INSERT or UPDATE that would leave the column without a value. This is enforced at write time, before the row reaches disk.

Placing NOT NULL on columns that your application logic depends on (email, created_at, status) prevents entire classes of runtime errors. It also allows the query planner to generate more efficient execution plans because the engine knows the column will never be unknown.

Supplying Sensible Defaults with the DEFAULT Constraint

The DEFAULT constraint provides an automatic value when none is supplied during an INSERT. The engine evaluates the default expression at write time and substitutes it seamlessly.

Common realistic defaults include:

CURRENT_TIMESTAMP or NOW() for audit columns
A literal string such as 'active' for status columns
A numeric literal such as 0.00 for price fields
A complex expression such as gen_random_uuid() in PostgreSQL

MySQL and PostgreSQL both support DEFAULT on almost every data type, but PostgreSQL allows far richer expressions (including subqueries and functions) because its parser is more expressive.

Building a Production-Grade Table: A Complete Realistic Example

Let us create a users table that reflects real engineering decisions made every day in SaaS companies.

-- PostgreSQL version
CREATE TABLE users (
    id              BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    email           VARCHAR(255) NOT NULL,
    full_name       VARCHAR(255),
    username        VARCHAR(100) UNIQUE,
    status          VARCHAR(20)  DEFAULT 'pending' NOT NULL,
    created_at      TIMESTAMPTZ  DEFAULT NOW() NOT NULL,
    last_login_at   TIMESTAMPTZ,
    preferences     JSONB        DEFAULT '{}'::JSONB,
    is_verified     BOOLEAN      DEFAULT FALSE NOT NULL
);

-- Equivalent MySQL version (note AUTO_INCREMENT and slight syntax differences)
CREATE TABLE users (
    id              BIGINT AUTO_INCREMENT PRIMARY KEY,
    email           VARCHAR(255) NOT NULL,
    full_name       VARCHAR(255),
    username        VARCHAR(100) UNIQUE,
    status          VARCHAR(20)  DEFAULT 'pending' NOT NULL,
    created_at      DATETIME(3)  DEFAULT CURRENT_TIMESTAMP(3) NOT NULL,
    last_login_at   DATETIME(3),
    preferences     JSON         DEFAULT (JSON_OBJECT()) NOT NULL,
    is_verified     TINYINT(1)   DEFAULT 0 NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Walk through the logic:

id uses an auto-incrementing or identity column so the engine guarantees uniqueness without application-level coordination.
email is NOT NULL because every user must have one; the database engine will reject any attempt to create a row without it.
status has a DEFAULT so new users start in a safe pending state even if the application forgets to specify it.
created_at defaults to the exact moment of insertion, providing a reliable audit trail without requiring application code to remember to set it.
preferences as JSONB (PostgreSQL) or JSON (MySQL) allows flexible storage of user settings without schema changes.

When you execute this statement, the database engine creates the table, builds the internal row header with the null bitmap, reserves space for the default values, and registers the column metadata so future queries can leverage type information for optimization.

Engineering Trade-offs, Performance Implications, and Best Practices

Choosing data types and constraints is never free.

Wider numeric types (BIGINT vs INTEGER) double storage and index size but prevent overflow bugs that crash production systems.
Variable-length types (VARCHAR, TEXT) are more space-efficient but require an extra length prefix and can fragment pages over time.
NOT NULL columns allow the query planner to skip null-check branches, improving index usage and reducing CPU cycles.
Over-use of DEFAULT expressions that call functions (e.g., NOW()) adds microseconds per row; in a table receiving 10,000 inserts per second, this becomes measurable.

Best practice: declare the narrowest, most restrictive data type and the strictest constraints that still match real business rules. Run EXPLAIN ANALYZE on representative queries after creation to validate that the engine is using the types and constraints you intended. In high-scale environments, these early decisions determine whether your database stays on a single node or needs sharding later.

For teams working in both PostgreSQL and MySQL, maintain separate migration scripts or use an abstraction layer that understands the subtle syntax differences—especially around default timestamps and identity columns.

If you want to take these concepts from theory to production-grade mastery with dozens of additional exercises, real schema evolution patterns, and battle-tested migration strategies, grab the comprehensive SQL Playbook at https://codewithdhanian.gumroad.com/l/hjmix —it was built exactly for engineers who want to move beyond tutorials and ship reliable systems.

Day 3: Creating and Dropping Databases – CREATE DATABASE, DROP DATABASE

CodeWithDhanian — Sat, 23 May 2026 12:38:47 +0000

Creating a database is one of the most fundamental operations you will perform as a SQL developer or database engineer. It marks the moment you move from simply connecting to a database management system to actually owning and organizing your own structured data environment. In this tutorial, we will explore exactly how databases are created and removed, why these operations behave the way they do under the hood, and how to handle them responsibly in both development and production environments.

We will work with the two most popular open-source relational database management systems introduced earlier: PostgreSQL and MySQL. Although both support the core SQL commands CREATE DATABASE and DROP DATABASE, their implementations differ in important architectural ways that affect performance, security, and scalability.

What a Database Actually Represents

Before writing any code, it is essential to understand what a database truly is from an architectural perspective. In a relational database management system, a database is a named container that holds multiple schemas, tables, indexes, views, functions, and other database objects. It acts as a logical boundary for data isolation, access control, and resource allocation.

Physically, a database is more than just a name. In PostgreSQL, each database corresponds to a separate set of files on disk, managed by the postmaster process. These files store the actual data pages, transaction logs (WAL), and catalog metadata. In MySQL (using the default InnoDB storage engine), each database typically maps to a subdirectory under the MySQL data directory, with table definitions and data stored in .frm, .ibd, or similar files depending on the engine.

This separation is deliberate. It allows a single database server instance to host dozens or even hundreds of independent databases, each with its own security context and performance characteristics. This architecture is what enables multi-tenant applications, separate development/staging/production environments, and efficient resource isolation.

The CREATE DATABASE Statement

The primary command to create a new database is CREATE DATABASE. The basic syntax is intentionally simple, yet the optional clauses reveal deep control over how the database will behave.

-- PostgreSQL version
CREATE DATABASE my_app_production
    WITH OWNER = postgres
         ENCODING = 'UTF8'
         LC_COLLATE = 'en_US.UTF-8'
         LC_CTYPE = 'en_US.UTF-8'
         TEMPLATE = template0;

-- MySQL version
CREATE DATABASE my_app_production
    CHARACTER SET utf8mb4
    COLLATE utf8mb4_unicode_ci;

Let us break down every important part of these statements.

WITH (PostgreSQL only): Introduces a list of optional parameters that configure the new database at creation time.
OWNER: Specifies which role (user) will own the database. The owner has full rights to create objects, grant privileges, and drop the database later. In production systems, it is a best practice to create a dedicated application role with the least privileges necessary rather than using the default superuser.
ENCODING / CHARACTER SET: Defines how text data will be stored. UTF8 (PostgreSQL) or utf8mb4 (MySQL) is almost always the correct choice today because it properly supports international characters, emojis, and modern languages without data corruption.
LC_COLLATE and LC_CTYPE: Control how string sorting and character classification work. These settings affect query performance on text columns and the results of ORDER BY operations. Choosing the wrong collation can lead to unexpected sorting behavior or degraded index performance.
TEMPLATE (PostgreSQL): Every new database is cloned from a template. The default template1 includes standard objects and extensions. Using template0 gives you a completely clean slate, which is preferable when you want strict control over what exists in the new database.

In MySQL, the equivalent parameters are simpler because MySQL handles character sets at both the server and database level. The COLLATE clause determines the default sorting and comparison rules for all tables created inside that database unless overridden at the table or column level.

Why These Options Matter in Real Engineering

Choosing the right encoding and collation at creation time is not a cosmetic decision. Once a database is created, changing these properties is extremely difficult and often requires a full data export/import cycle. In large-scale systems, incorrect collation choices have caused production incidents where searches and sorts returned inconsistent results across environments.

Performance implications are also significant. A database with the wrong collation can prevent the query planner from using indexes efficiently during text comparisons, leading to full table scans even when indexes exist.

Creating Databases in Practice: Development Workflows

Professional developers rarely create databases manually in production. Instead, they follow repeatable workflows:

Use infrastructure-as-code tools or migration scripts.
Create separate databases for development, testing, and production.
Automate creation as part of CI/CD pipelines so every developer works with identical starting conditions.

Here is a realistic example of a script you might keep in your project repository:

-- create_dev_db.sql
DO $$
BEGIN
    IF NOT EXISTS (SELECT FROM pg_database WHERE datname = 'my_app_dev') THEN
        PERFORM dblink_exec('host=localhost user=postgres',
            'CREATE DATABASE my_app_dev WITH OWNER = app_user ENCODING = ''UTF8''');
    END IF;
END
$$;

This pattern uses a conditional check to make the script idempotent, meaning it can be run multiple times without error.

The DROP DATABASE Statement

Removing a database is the counterpart to creation and must be treated with respect. The syntax is:

-- PostgreSQL
DROP DATABASE my_app_production;

-- Safer version
DROP DATABASE IF EXISTS my_app_production;

-- MySQL
DROP DATABASE my_app_production;
DROP DATABASE IF EXISTS my_app_production;

The IF EXISTS clause prevents an error if the database does not exist. This is especially valuable in automated scripts.

Important behavior differences:

In PostgreSQL, you cannot drop a database while active connections exist. The server will reject the command unless you use the WITH FORCE option (available in newer versions) or first terminate connections manually.
In MySQL, dropping a database is more permissive but still requires appropriate privileges.

DROP DATABASE is a permanent, non-recoverable operation in most cases. The files on disk are removed, and all data disappears. This is why production systems almost never allow direct DROP DATABASE commands except during controlled maintenance windows.

Internal Workflows and System Behavior

When you execute CREATE DATABASE:

The database server validates your permissions.
It allocates a new database OID (object identifier).
It copies the template files (PostgreSQL) or creates a new subdirectory (MySQL).
It updates the system catalogs (pg_database in PostgreSQL, mysql system database in MySQL).
It initializes the default schema (public in PostgreSQL, the database itself in MySQL).

When you execute DROP DATABASE:

The server checks for active connections and may block or terminate them.
It removes all associated files from the filesystem.
It cleans up metadata entries across system catalogs.
Any open transactions involving that database are rolled back or invalidated.

These operations affect the entire server instance. On very large systems, creating or dropping databases can momentarily increase I/O load because of file system operations.

Best Practices and Scalability Considerations

Never create databases with generic names like "test" or "db1" in production. Use descriptive, versioned names that reflect purpose and environment.
Prefer schemas over multiple databases when you need logical separation within the same data set. This is more efficient because it avoids the overhead of managing separate file sets.
In multi-tenant SaaS applications, some teams create one database per customer for strong isolation, while others use a single database with tenant_id columns. Each approach has different scalability and backup implications.
Always back up critical databases before any DROP operation.
Grant the minimal privileges necessary. Application users should rarely have CREATE DATABASE or DROP DATABASE rights.

Real-World Engineering Example

Imagine you are building an e-commerce platform. You might maintain:

ecommerce_production – live customer data
ecommerce_staging – pre-production testing
ecommerce_dev – local development

You would create these using scripts that also initialize required extensions, default roles, and security policies. When a new feature requires schema changes that cannot be rolled back easily, you create a fresh database from a recent backup of production, test the migration thoroughly, then promote the change.

These patterns ensure consistency, reduce human error, and make your infrastructure reproducible.

For even more in-depth examples and exercises, consider purchasing this SQL Playbook: https://codewithdhanian.gumroad.com/l/hjmix

Day 2: Installing PostgreSQL/MySQL and Setting Up Your First Database

CodeWithDhanian — Sat, 23 May 2026 12:16:57 +0000

In database development, the quality of your local environment directly determines how effectively you will learn and how reliably you will build production systems later. Today we focus entirely on installing PostgreSQL and MySQL, understanding the client-server architecture that powers them, and creating your first functional database. We will examine every step from a professional engineering perspective so you can make informed decisions whether you are a complete beginner or an experienced developer setting up a new machine.

The Client-Server Architecture Behind Every Relational Database

A relational database management system (RDBMS) is never just a file on your computer. It follows a strict client-server model. The database server is a background process (daemon) that runs continuously, manages data files on disk, enforces referential integrity, handles concurrent connections, and executes SQL statements. The client is any tool or application that connects to this server over the network or locally via a socket.

PostgreSQL and MySQL both implement this architecture, but they differ in design philosophy. PostgreSQL is a fully object-relational system with strong emphasis on standards compliance, extensibility, and advanced features such as JSONB support and custom data types. MySQL is optimized for read-heavy web workloads and remains the default choice for many content-driven applications due to its speed and widespread ecosystem support.

When the server starts, it listens on a specific TCP port: 5432 for PostgreSQL and 3306 for MySQL. Clients connect using these ports, authenticate with a username and password, and then send SQL commands. The server parses the query, checks permissions, optimizes the execution plan using its internal query planner, and returns results. Understanding this flow helps you debug connection issues later and appreciate why proper installation and configuration matter for both development speed and long-term scalability.

Choosing Between PostgreSQL and MySQL

Both systems are excellent open-source RDBMS options, but they serve slightly different needs. PostgreSQL excels in complex queries, window functions, full ACID compliance out of the box, and handling large analytical workloads. MySQL is often lighter on resources and has tighter integration with popular web stacks such as PHP and certain cloud platforms. Many professional developers install both so they can work on different projects without friction. We will cover the complete installation process for each.

Installing PostgreSQL

Installation on Major Operating Systems

The installation process places the PostgreSQL server binaries, client tools, and default data directory on your system.

On macOS, the recommended developer approach uses Homebrew. Open your terminal and run:

brew install postgresql
brew services start postgresql

This command installs the latest stable version, creates the necessary directories, and registers PostgreSQL as a background service that starts automatically.

On Ubuntu/Debian Linux, use the package manager:

sudo apt update
sudo apt install postgresql postgresql-contrib
sudo systemctl start postgresql

On Windows, download the official EnterpriseDB installer from the PostgreSQL website and follow the wizard. Choose the default port 5432 and remember the password you set for the postgres superuser.

Post-Installation Verification and Initialization

After installation, PostgreSQL runs an initialization process called initdb (automatically handled by the installer or package manager). This creates the data directory (often /var/lib/postgresql/data on Linux or ~/Library/Application Support/Postgres on macOS) containing the system catalogs, template databases, and configuration files.

Verify everything is running by checking the service status or using the client tool:

psql --version

This confirms the psql command-line client is available and in your PATH.

Switch to the default postgres user (on Linux/macOS) or open pgAdmin on Windows to continue setup.

Setting Up Your First PostgreSQL Database

Create a dedicated database and user instead of using the default postgres superuser for daily work. This follows security best practices and prevents accidental destructive commands.

First, create a new user account:

CREATE USER myappuser WITH PASSWORD 'strong_password_here';

Then create the actual database:

CREATE DATABASE myfirstdb OWNER myappuser;

Grant necessary privileges:

GRANT ALL PRIVILEGES ON DATABASE myfirstdb TO myappuser;

Connect to your new database using the psql client:

psql -U myappuser -d myfirstdb -h localhost

The -U flag specifies the user, -d the database, and -h localhost forces a TCP connection so you can see network behavior. Once inside psql, you can run \l to list databases and confirm your new database exists.

For a graphical interface, install pgAdmin. It provides a visual browser of databases, schemas, tables, and query tools while still connecting to the same underlying server. This dual workflow (command line for speed, GUI for exploration) is standard in professional development environments.

Installing MySQL

Installation Approaches

MySQL installation follows a similar client-server pattern but uses different binaries and configuration files.

On macOS with Homebrew:

brew install mysql
brew services start mysql

On Ubuntu/Debian:

sudo apt update
sudo apt install mysql-server
sudo systemctl start mysql

During installation on Linux, the system creates the root user with a temporary password that you must secure immediately using the mysql_secure_installation script.

On Windows, use the official MySQL Installer and select the Developer Default setup type.

MySQL Post-Installation Steps

MySQL stores its data in a directory controlled by the mysqld daemon. The main configuration file is my.cnf (or my.ini on Windows), where you can later tune buffer sizes and connection limits.

Verify installation:

mysql --version

Secure the root user immediately and create a dedicated application user:

ALTER USER 'root'@'localhost' IDENTIFIED BY 'strong_root_password';
CREATE USER 'myappuser'@'localhost' IDENTIFIED BY 'strong_password_here';
CREATE DATABASE myfirstdb;
GRANT ALL PRIVILEGES ON myfirstdb.* TO 'myappuser'@'localhost';
FLUSH PRIVILEGES;

Connect to your new database:

mysql -u myappuser -p myfirstdb

The -p flag prompts for the password, keeping it out of command history.

MySQL Workbench serves as the official graphical client, offering schema design, query execution, and server status monitoring.

Best Practices for Production-Ready Local Setup

Never leave default passwords or the root user exposed in development. Always create application-specific users with the minimum required privileges. On both systems, configure the server to listen only on localhost during development to reduce the attack surface.

Monitor the server logs early. PostgreSQL logs are in the data directory under log/, while MySQL uses the error log defined in the configuration file. These logs reveal connection attempts, slow queries, and authentication failures.

Consider using Docker for repeatable environments in larger projects. A simple Docker Compose setup can spin up isolated PostgreSQL or MySQL containers with version pinning, making your development environment identical across team members and machines.

Performance implications appear even at this stage: proper data directory placement on SSDs rather than HDDs dramatically affects query speed. Memory allocation in the configuration files determines how much caching the server can perform before hitting disk.

These foundational choices affect scalability later. A correctly installed and secured RDBMS will support everything from simple prototypes to high-traffic applications without requiring major rework.

If you want a complete, project-driven SQL learning experience that builds directly on these setup foundations with guided exercises and real application patterns, consider purchasing the SQL Playbook at https://codewithdhanian.gumroad.com/l/hjmix. It provides the structured depth many developers wish they had when starting out.

Day 1: What is SQL – Databases, Tables, and Relational Concepts

CodeWithDhanian — Sat, 23 May 2026 11:59:28 +0000

The Foundation of Modern Data Management

SQL stands as the universal language for interacting with relational data. Before writing a single query, it is essential to understand exactly what SQL is, why it exists, and how it sits at the heart of virtually every significant application you use today. SQL, which expands to Structured Query Language, is a declarative programming language designed specifically for managing, organizing, and retrieving data stored in relational databases.

At its core, SQL allows you to tell a database system what you want rather than how to retrieve it. This declarative nature distinguishes SQL from imperative languages where you must write step-by-step instructions. The database engine handles the optimization, execution plan, and data access internally, freeing developers to focus on business logic instead of low-level file operations.

What Is a Database?

A database is an organized collection of structured data that is stored and accessed electronically. Think of it as a highly efficient digital filing cabinet designed for speed, reliability, and concurrent access by thousands or even millions of users simultaneously.

Unlike a simple spreadsheet or text file, a production database must satisfy strict requirements: it must maintain data integrity, support atomic transactions, allow concurrent access without corruption, provide backup and recovery mechanisms, and scale as data volumes grow. These capabilities are delivered by specialized software known as a Relational Database Management System (RDBMS).

The RDBMS acts as the intermediary between your application and the raw data files on disk. It manages memory buffers, disk I/O, query optimization, concurrency control through locking and multiversion concurrency control (MVCC), and crash recovery. Popular RDBMS implementations include systems that follow the same fundamental principles you will master in this series.

The Relational Model: The Theoretical Backbone

The entire concept of SQL rests on the relational model, first formally defined by Edgar F. Codd. In this model, all data is represented in two-dimensional structures called tables (also known as relations). Each table represents a specific entity or concept in your domain.

The power of the relational model comes from its mathematical foundation in set theory and predicate logic. This foundation guarantees that data can be combined, filtered, and transformed in predictable, mathematically sound ways. Relationships between tables are expressed through shared values rather than physical pointers, providing flexibility and reducing data duplication.

Key Principles of Relational Design

Several core rules govern how data should be structured in a relational database:

Atomicity: Each cell in a table holds a single, indivisible value.
Uniqueness: Every row in a table must be uniquely identifiable.
Order independence: The sequence of rows or columns does not matter; the data is defined by its content and relationships.
Consistency: All values in a column must belong to the same data type and follow the same rules.

These principles lead to the practice of normalization, which organizes data to minimize redundancy while preserving relationships. Although we will explore normalization in greater depth later, understanding that it exists helps explain why SQL databases feel so structured and reliable.

Tables: The Fundamental Building Blocks

A table is the primary structure where data lives. Each table has a name that should clearly describe the entity it represents, such as users, orders, or products.

Inside a table you find:

Columns (also called fields or attributes): These define the structure and type of data the table can hold. Each column has a name and a specific data type that enforces what kind of information can be stored.
Rows (also called records or tuples): These are the actual data entries. Each row represents one complete instance of the entity described by the table.

For example, consider an e-commerce application. You might have a products table with columns like product_id, name, price, category, and stock_quantity. Each row would represent one specific product available for sale.

Visualizing Table Structure

A table can be imagined as a grid. The column headers define the blueprint, while each subsequent row fills in actual values according to that blueprint. The RDBMS enforces rules at the column level, such as requiring certain columns to be filled (NOT NULL) or ensuring values fall within acceptable ranges.

Rows, Columns, and Data Relationships

Every row in a table must be uniquely identifiable. This is typically achieved using a primary key — a column or combination of columns whose value is unique for every row. The primary key serves as the definitive address for that record.

Relationships between tables are established using foreign keys. A foreign key in one table references the primary key in another table, creating a logical link. This linkage is what makes the database relational — data in separate tables can be meaningfully connected without duplicating entire records.

For instance:

An orders table might contain a customer_id column that references the customer_id primary key in a customers table.
This design allows one customer to place many orders while storing customer details only once.

This approach dramatically reduces data redundancy, improves consistency, and makes complex queries possible.

Why SQL and Relational Databases Dominate

SQL combined with the relational model offers several powerful advantages:

Declarative power: You describe the desired result, and the query optimizer determines the most efficient way to retrieve it.
Data integrity: Built-in constraints protect against invalid data.
Standardization: SQL is an ANSI/ISO standard, meaning core concepts transfer across different RDBMS implementations.
Scalability: Well-designed relational databases can handle terabytes of data while maintaining performance through proper indexing and query design.
Transaction safety: Changes can be grouped into transactions that either fully succeed or fully roll back.

Of course, no technology is perfect. Relational databases can face challenges with extremely high-velocity write workloads or when dealing with highly unstructured data. These limitations have led to the rise of other database types, but for structured business data — finance, e-commerce, healthcare, inventory, user management — the relational model remains the gold standard.

Practical Developer Perspective

As a software engineer, you will interact with SQL daily. Whether building a new feature, debugging performance issues, or designing a new data model, a deep understanding of these foundational concepts prevents costly mistakes later.

When designing tables, always ask:

What entity does this table represent?
What are the natural relationships to other entities?
Which columns must be unique?
What constraints protect data quality?

These questions guide you toward clean, maintainable database schemas that scale gracefully as your application grows.

If you are serious about mastering SQL and want a complete hands-on playbook that follows this exact structured learning path with exercises, challenges, and advanced patterns, consider purchasing the SQL Playbook at https://codewithdhanian.gumroad.com/l/hjmix. It is designed to complement this series perfectly.