API Design 101: The Ultimate Guide to Building APIs That Don't Suck
You've used APIs. You've probably built a few. And if you're being honest, at least one of them was held together with duct tape and prayers.
Bad APIs are everywhere. Inconsistent naming, mystery status codes, error messages that say "something went wrong" with zero context, pagination that breaks when you look at it funny. We've all been there.
This guide is the antidote. We're going to cover everything about designing APIs that developers actually enjoy using — from URL structure to authentication, rate limiting to webhooks, error handling to documentation. With real code, real patterns, and real opinions.
Let's build APIs that don't suck.
Table of Contents
- What Makes a Great API?
- URL and Resource Design
- HTTP Methods Done Right
- Status Codes — The Full Picture
- Request/Response Design
- Error Handling Patterns
- Authentication in APIs
- Versioning Strategies
- Rate Limiting
- Input Validation
- Filtering, Sorting, and Pagination
- Bulk Operations
- Webhooks
- HATEOAS and Hypermedia
- API Documentation
- API Security Checklist
- Bad vs Good API Patterns
- Putting It All Together — Express.js Example
- Decision Framework
What Makes a Great API?
Before we get into the specifics, let's establish what separates a great API from a mediocre one. It boils down to three pillars:
+---------------------------------------------------+
| GREAT API DESIGN |
+---------------------------------------------------+
| |
| +-------------+ +----------------+ +-----------+ |
| | Consistency | | Predictability | | Developer | |
| | | | | | Experience| |
| +-------------+ +----------------+ +-----------+ |
| |
| "If I know one "I can guess "I can get |
| endpoint, I what this does started in |
| know them all" without docs" 5 minutes" |
| |
+---------------------------------------------------+
Consistency means if GET /users returns a list, then GET /orders returns a list in the exact same shape. Same envelope, same pagination format, same error structure. Always.
Predictability means a developer can guess the endpoint for "get a single order" after seeing GET /users/:id. It should obviously be GET /orders/:id. No surprises.
Developer Experience (DX) means clear errors, good docs, sensible defaults, and the feeling that whoever designed this API actually thought about the person consuming it.
Here's a quick litmus test:
| Question | Great API | Bad API |
|---|---|---|
| Can I guess the endpoint? | Yes, follows conventions | No, every resource is different |
| Are errors helpful? | Specific field-level messages | "error": "Bad Request" |
| Is pagination consistent? | Same pattern everywhere | Different for each resource |
| Can I onboard in 5 min? | Good docs, clear examples | Need to read source code |
| Are breaking changes versioned? | Yes, with deprecation notices | Stuff just breaks randomly |
URL and Resource Design
Your URLs are the front door of your API. They should read like a well-organized filing cabinet, not a junk drawer.
Rule 1: Use Nouns, Not Verbs
Resources are things. Use nouns to name them. The HTTP method already tells us the action.
BAD:
GET /getUsers
POST /createUser
PUT /updateUser/123
DELETE /deleteUser/123
GOOD:
GET /users
POST /users
PUT /users/123
DELETE /users/123
The verb is in the HTTP method. Don't repeat it in the URL.
Rule 2: Use Plurals
Pick plural nouns and be consistent. Even for a single resource, the collection is plural.
BAD: GOOD:
GET /user/123 GET /users/123
GET /user GET /users
GET /product-catalog GET /products
Rule 3: Nest for Relationships (But Not Too Deep)
Nesting shows resource ownership. But stop at two levels — deeper nesting gets messy.
GOOD (one level of nesting):
GET /users/123/orders # Orders for user 123
GET /orders/456/items # Items in order 456
TOO DEEP (avoid this):
GET /users/123/orders/456/items/789/reviews
BETTER (flatten it):
GET /orders/456/items/789
GET /items/789/reviews
Rule 4: Use Query Parameters for Filtering, Sorting, and Pagination
Don't pollute your URL path with non-resource concerns.
GOOD:
GET /users?role=admin&sort=-created_at&page=2&limit=20
GET /products?category=electronics&min_price=100&max_price=500
GET /orders?status=shipped&created_after=2026-01-01
BAD:
GET /users/admins/sorted-by-date/page/2
GET /products/electronics/price-range/100-500
Rule 5: Use kebab-case for Multi-Word Resources
GOOD: BAD:
/order-items /orderItems
/user-profiles /user_profiles
/payment-methods /PaymentMethods
Complete URL Design Cheatsheet
Resource Collection: GET /resources
Single Resource: GET /resources/:id
Create Resource: POST /resources
Update Resource: PUT /resources/:id
Partial Update: PATCH /resources/:id
Delete Resource: DELETE /resources/:id
Sub-resources: GET /resources/:id/sub-resources
Actions (rare): POST /resources/:id/actions/archive
Search: GET /resources/search?q=term
HTTP Methods Done Right
Each HTTP method has specific semantics. Using them correctly makes your API predictable.
The Big Five
| Method | Purpose | Idempotent? | Request Body? | Success Code |
|---|---|---|---|---|
GET |
Read a resource | Yes | No | 200 |
POST |
Create a resource | No | Yes | 201 |
PUT |
Replace a resource entirely | Yes | Yes | 200 |
PATCH |
Partially update a resource | Yes* | Yes | 200 |
DELETE |
Remove a resource | Yes | Rarely | 204 |
Idempotency Explained
An idempotent operation produces the same result no matter how many times you call it.
Idempotent (safe to retry):
PUT /users/123 { "name": "Alex" }
-> Call 1: Updates name to "Alex" -> 200
-> Call 2: Name is still "Alex" -> 200 (same result)
DELETE /users/123
-> Call 1: Deletes the user -> 204
-> Call 2: User already gone -> 404 (but state is same)
NOT idempotent (calling twice creates duplicates):
POST /orders { "product": "laptop" }
-> Call 1: Creates order #1001 -> 201
-> Call 2: Creates order #1002 -> 201 (duplicate!)
This matters because network requests fail. If a client doesn't get a response and retries, idempotent operations are safe to retry. Non-idempotent ones (POST) need idempotency keys.
PUT vs PATCH
This trips people up. Here's the deal:
// Current user:
{ "id": 123, "name": "Alex", "email": "alex@example.com", "role": "admin" }
// PUT /users/123 — full replacement
// You MUST send the complete resource
{ "name": "Alex Updated", "email": "alex@example.com", "role": "admin" }
// Missing fields get wiped to null/default!
// PATCH /users/123 — partial update
// Send only what changed
{ "name": "Alex Updated" }
// Other fields remain untouched
Use PUT when the client has the full resource and wants to replace it.
Use PATCH when updating one or two fields (which is 90% of real-world updates).
Making POST Idempotent with Idempotency Keys
// Client generates a unique key
POST /orders
Headers:
Idempotency-Key: a1b2c3d4-e5f6-7890
// Server logic:
app.post('/orders', async (req, res) => {
const idempotencyKey = req.headers['idempotency-key'];
// Check if we've already processed this key
const existing = await cache.get(`idempotency:${idempotencyKey}`);
if (existing) {
return res.status(200).json(existing); // Return cached response
}
// Process the order
const order = await createOrder(req.body);
// Cache the response with the key (TTL: 24 hours)
await cache.set(`idempotency:${idempotencyKey}`, order, 86400);
return res.status(201).json(order);
});
Status Codes — The Full Picture
Status codes are your API's way of communicating what happened. Don't just throw 200s and 500s around.
The Must-Know Status Codes
+-------+---------------------------------------------+
| Code | When to Use |
+-------+---------------------------------------------+
| | 2xx — SUCCESS |
+-------+---------------------------------------------+
| 200 | OK — general success, returning data |
| 201 | Created — resource created (POST) |
| 204 | No Content — success, nothing to return |
| | (DELETE, PUT with no response body) |
+-------+---------------------------------------------+
| | 4xx — CLIENT ERROR |
+-------+---------------------------------------------+
| 400 | Bad Request — malformed syntax, invalid |
| | JSON, missing required fields |
| 401 | Unauthorized — no auth or invalid auth |
| | (should really be "Unauthenticated") |
| 403 | Forbidden — authenticated but no permission |
| 404 | Not Found — resource doesn't exist |
| 409 | Conflict — resource state conflict |
| | (duplicate email, version mismatch) |
| 422 | Unprocessable Entity — valid syntax but |
| | semantically wrong (age = -5) |
| 429 | Too Many Requests — rate limit exceeded |
+-------+---------------------------------------------+
| | 5xx — SERVER ERROR |
+-------+---------------------------------------------+
| 500 | Internal Server Error — unhandled bug |
| 503 | Service Unavailable — overloaded or in |
| | maintenance, try again later |
+-------+---------------------------------------------+
When to Use What — Real Examples
// 200 — Successful read or update
GET /users/123 -> 200 { "id": 123, "name": "Alex" }
PATCH /users/123 -> 200 { "id": 123, "name": "Updated" }
// 201 — Resource created (include Location header!)
POST /users -> 201 { "id": 124, "name": "New User" }
Location: /users/124
// 204 — Success but nothing to send back
DELETE /users/123 -> 204 (empty body)
// 400 — Malformed request
POST /users { name: } -> 400 "Invalid JSON syntax"
// 401 — Missing or invalid credentials
GET /admin/users -> 401 "Authentication required"
(no token provided)
// 403 — Valid auth but insufficient permissions
GET /admin/users -> 403 "Admin role required"
(token valid, but user is not admin)
// 404 — Resource not found
GET /users/99999 -> 404 "User not found"
// 409 — Conflict with current state
POST /users { email: "taken@email.com" }
-> 409 "Email already exists"
// 422 — Valid JSON but fails business logic
POST /users { age: -5 }
-> 422 "Age must be a positive number"
// 429 — Too many requests
GET /api/anything -> 429 "Rate limit exceeded"
Retry-After: 30
// 500 — Unexpected server error
GET /users -> 500 "Internal server error"
// 503 — Server overloaded or in maintenance
GET /api/anything -> 503 "Service temporarily unavailable"
Retry-After: 300
The 401 vs 403 Decision
This one confuses people. Here's the simple rule:
+------------------+----------------------------------+
| 401 | 403 |
+------------------+----------------------------------+
| WHO are you? | I know who you are, but NO. |
| No token sent | Valid token, wrong permissions |
| Expired token | User role can't access this |
| Invalid token | IP blocked, account suspended |
+------------------+----------------------------------+
Decision flow:
Is there a valid auth token?
NO -> 401 Unauthorized
YES -> Does this user have permission?
NO -> 403 Forbidden
YES -> Process request
Request/Response Design
Consistency in your request and response shapes is what makes an API feel polished.
The Envelope Pattern
Wrapping responses in a consistent envelope gives you room for metadata without changing the core data shape.
// Consistent envelope for single resources
{
"status": "success",
"data": {
"id": 123,
"name": "Alex",
"email": "alex@example.com"
}
}
// Consistent envelope for collections
{
"status": "success",
"data": [
{ "id": 123, "name": "Alex" },
{ "id": 124, "name": "Jordan" }
],
"meta": {
"total": 245,
"page": 1,
"per_page": 20,
"total_pages": 13
}
}
// Consistent envelope for errors
{
"status": "error",
"error": {
"code": "VALIDATION_ERROR",
"message": "Request validation failed",
"details": [
{ "field": "email", "message": "Must be a valid email address" },
{ "field": "age", "message": "Must be between 1 and 150" }
]
}
}
Flat vs Envelope — When to Use What
| Approach | Pros | Cons |
|---|---|---|
Envelope { data, meta }
|
Consistent shape, room for metadata | Slightly more verbose |
| Flat (just the resource) | Simpler, less nesting | Hard to add metadata later |
My recommendation: use envelopes for collections (you need pagination metadata) and flat for single resources unless you need metadata. Many teams just go full envelope for consistency, and that's fine too.
RFC 7807 Problem Details — The Error Standard
Instead of inventing your own error format, use the RFC 7807 standard. It's becoming the industry norm.
// RFC 7807 Problem Details format
{
"type": "https://api.example.com/errors/validation",
"title": "Validation Error",
"status": 422,
"detail": "The request body contains invalid fields",
"instance": "/users",
"errors": [
{
"field": "email",
"message": "Must be a valid email address",
"value": "not-an-email"
}
]
}
// Another example — rate limiting
{
"type": "https://api.example.com/errors/rate-limit",
"title": "Rate Limit Exceeded",
"status": 429,
"detail": "You have exceeded 100 requests per minute",
"instance": "/users",
"retryAfter": 30
}
The type field is a URI that identifies the error type (and can point to docs). The title is human-readable. The detail gives specifics for this occurrence.
Error Handling Patterns
Good error handling is the difference between "I fixed the bug in 2 minutes" and "I spent 3 hours guessing what went wrong."
Custom Error Classes
// errors.js — Define a hierarchy of errors
class AppError extends Error {
constructor(message, statusCode, code) {
super(message);
this.statusCode = statusCode;
this.code = code;
this.isOperational = true;
Error.captureStackTrace(this, this.constructor);
}
}
class NotFoundError extends AppError {
constructor(resource, id) {
super(`${resource} with id '${id}' not found`, 404, 'NOT_FOUND');
}
}
class ValidationError extends AppError {
constructor(errors) {
super('Validation failed', 422, 'VALIDATION_ERROR');
this.errors = errors; // Array of field-level errors
}
}
class ConflictError extends AppError {
constructor(message) {
super(message, 409, 'CONFLICT');
}
}
class UnauthorizedError extends AppError {
constructor(message = 'Authentication required') {
super(message, 401, 'UNAUTHORIZED');
}
}
class ForbiddenError extends AppError {
constructor(message = 'Insufficient permissions') {
super(message, 403, 'FORBIDDEN');
}
}
class RateLimitError extends AppError {
constructor(retryAfter = 60) {
super('Rate limit exceeded', 429, 'RATE_LIMIT_EXCEEDED');
this.retryAfter = retryAfter;
}
}
Centralized Error Handler
// middleware/errorHandler.js
const errorHandler = (err, req, res, next) => {
// Log the error (but not in tests)
if (process.env.NODE_ENV !== 'test') {
console.error(`[${new Date().toISOString()}] ${err.stack}`);
}
// Operational errors — we know what happened
if (err.isOperational) {
const response = {
type: `https://api.example.com/errors/${err.code.toLowerCase().replace(/_/g, '-')}`,
title: err.code.replace(/_/g, ' ').toLowerCase(),
status: err.statusCode,
detail: err.message,
instance: req.originalUrl,
};
// Add field-level errors for validation
if (err.errors) {
response.errors = err.errors;
}
// Add retry-after for rate limiting
if (err.retryAfter) {
res.set('Retry-After', err.retryAfter);
response.retryAfter = err.retryAfter;
}
return res.status(err.statusCode).json(response);
}
// Programming errors — don't leak details
return res.status(500).json({
type: 'https://api.example.com/errors/internal',
title: 'Internal Server Error',
status: 500,
detail: process.env.NODE_ENV === 'production'
? 'An unexpected error occurred'
: err.message,
instance: req.originalUrl,
});
};
Using Errors in Route Handlers
// Clean route handlers that throw meaningful errors
app.get('/users/:id', async (req, res) => {
const user = await db.users.findById(req.params.id);
if (!user) {
throw new NotFoundError('User', req.params.id);
}
res.json(user);
});
app.post('/users', async (req, res) => {
const existing = await db.users.findByEmail(req.body.email);
if (existing) {
throw new ConflictError('A user with this email already exists');
}
const user = await db.users.create(req.body);
res.status(201)
.location(`/users/${user.id}`)
.json(user);
});
Authentication in APIs
There are several ways to authenticate API requests. Each has a sweet spot.
Comparison Table
| Method | Best For | Complexity | Security |
|---|---|---|---|
| API Keys | Server-to-server, internal services | Low | Medium |
| Bearer Tokens (JWT) | User-facing apps, SPAs | Medium | High |
| OAuth 2.0 | Third-party integrations | High | Very High |
| Session Cookies | Traditional web apps | Low | High (with CSRF protection) |
API Keys
Simple and effective for server-to-server communication.
// Client sends key in header
GET /api/weather?city=london
Headers:
X-API-Key: sk_live_abc123def456
// Server validates
const authenticateApiKey = async (req, res, next) => {
const apiKey = req.headers['x-api-key'];
if (!apiKey) {
throw new UnauthorizedError('API key is required');
}
// Hash the key and look it up (never store keys in plain text!)
const hashedKey = crypto.createHash('sha256').update(apiKey).digest('hex');
const keyRecord = await db.apiKeys.findByHash(hashedKey);
if (!keyRecord || keyRecord.revoked) {
throw new UnauthorizedError('Invalid API key');
}
req.apiClient = keyRecord.client;
next();
};
Bearer Tokens (JWT)
The standard for user-facing APIs. Token-based, stateless authentication.
// Client sends JWT in Authorization header
GET /api/profile
Headers:
Authorization: Bearer eyJhbGciOiJIUzI1NiIs...
// Server middleware
const authenticateJWT = (req, res, next) => {
const authHeader = req.headers.authorization;
if (!authHeader?.startsWith('Bearer ')) {
throw new UnauthorizedError('Bearer token required');
}
const token = authHeader.split(' ')[1];
try {
const payload = jwt.verify(token, process.env.JWT_SECRET);
req.user = payload;
next();
} catch (err) {
if (err.name === 'TokenExpiredError') {
throw new UnauthorizedError('Token expired');
}
throw new UnauthorizedError('Invalid token');
}
};
OAuth 2.0
For when third-party apps need access to your user's data.
+--------+ +---------------+
| |---(1) Authorization Request-->| Resource |
| | | Owner |
| |<--(2) Authorization Grant----| (User) |
| | +---------------+
| Client |
| App |---(3) Auth Grant------------>+---------------+
| | | Authorization |
| |<--(4) Access Token-----------| Server |
| | +---------------+
| |
| |---(5) Access Token---------->+---------------+
| | | Resource |
| |<--(6) Protected Resource-----| Server |
+--------+ +---------------+
When to Use What
Decision tree:
Is the consumer a server/service?
YES -> API Keys (simple, rotatable)
NO -> Is it your own frontend app?
YES -> JWT Bearer Tokens
NO -> Is it a third-party app?
YES -> OAuth 2.0
NO -> JWT Bearer Tokens
Versioning Strategies
APIs evolve. Breaking changes happen. Versioning gives you a way to evolve without breaking existing clients.
The Three Approaches
1. URL Path Versioning (Most Common)
GET /v1/users/123
GET /v2/users/123
// Express.js implementation
const v1Router = express.Router();
const v2Router = express.Router();
v1Router.get('/users/:id', (req, res) => {
// V1: returns flat user object
res.json({ id: 1, name: 'Alex', email: 'alex@example.com' });
});
v2Router.get('/users/:id', (req, res) => {
// V2: returns nested user with profile
res.json({
id: 1,
name: 'Alex',
profile: { email: 'alex@example.com', avatar: '...' }
});
});
app.use('/v1', v1Router);
app.use('/v2', v2Router);
2. Header Versioning
GET /users/123
Headers:
Accept: application/vnd.myapi.v2+json
3. Query Parameter Versioning
GET /users/123?version=2
Comparison
| Strategy | Pros | Cons |
|---|---|---|
URL path /v1/
|
Obvious, easy to route, cacheable | "Pollutes" the URL, feels like a different API |
Header Accept: vnd.v2
|
Clean URLs, content negotiation | Hidden, harder to test in browser |
Query param ?version=2
|
Simple, visible | Easy to forget, can't cache well |
My recommendation: URL path versioning. It's the most explicit, the easiest to understand, and the most widely used. GitHub, Stripe, and Twilio all use it.
Deprecation Strategy
Don't just kill old versions. Give clients time.
// Deprecation middleware
const deprecationWarning = (version, sunsetDate) => (req, res, next) => {
res.set('Deprecation', 'true');
res.set('Sunset', sunsetDate); // RFC 8594
res.set('Link', '</v3/docs>; rel="successor-version"');
console.warn(`[DEPRECATION] ${version} hit: ${req.method} ${req.originalUrl}`);
next();
};
app.use('/v1', deprecationWarning('v1', 'Sat, 01 Jun 2026 00:00:00 GMT'), v1Router);
Rate Limiting
Without rate limiting, one misbehaving client can take down your entire API. Rate limiting protects you and ensures fair usage.
Common Algorithms
TOKEN BUCKET:
+-------------------+
| Bucket: 10 tokens | Tokens refill at steady rate
| Refill: 1/second | Each request costs 1 token
+-------------------+ Burst allowed up to bucket size
Time 0s: [##########] 10 tokens (full)
Burst 5 requests:
Time 0s: [##### ] 5 tokens
Time 3s: [######## ] 8 tokens (3 refilled)
Time 3s: [# ] 1 token (7 requests)
SLIDING WINDOW:
Window: 60 seconds, Limit: 100 requests
|<-------- 60 seconds -------->|
| req req req ... (98 total) | -> OK
| req | -> 99, OK
| req | -> 100, OK
| req | -> 101, REJECTED (429)
| window slides -> |
Implementation with Express
// Simple rate limiter using express-rate-limit
import rateLimit from 'express-rate-limit';
// General API rate limit
const apiLimiter = rateLimit({
windowMs: 60 * 1000, // 1 minute window
max: 100, // 100 requests per window
standardHeaders: true, // Return rate limit info in RateLimit-* headers
legacyHeaders: false, // Disable X-RateLimit-* headers
message: {
type: 'https://api.example.com/errors/rate-limit',
title: 'Rate Limit Exceeded',
status: 429,
detail: 'You have exceeded 100 requests per minute',
},
keyGenerator: (req) => {
// Rate limit by API key or IP
return req.headers['x-api-key'] || req.ip;
},
});
// Stricter limit for auth endpoints (prevent brute force)
const authLimiter = rateLimit({
windowMs: 15 * 60 * 1000, // 15 minutes
max: 10, // 10 attempts
skipSuccessfulRequests: true, // Only count failures
});
app.use('/api/', apiLimiter);
app.use('/api/auth/login', authLimiter);
Rate Limit Headers
Always tell clients where they stand:
HTTP/1.1 200 OK
RateLimit-Limit: 100
RateLimit-Remaining: 67
RateLimit-Reset: 1679529600
HTTP/1.1 429 Too Many Requests
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1679529600
Retry-After: 30
Tiered Rate Limits
Different clients get different limits. This is how Stripe, GitHub, and every serious API works.
const tierLimits = {
free: { windowMs: 60000, max: 20 },
starter: { windowMs: 60000, max: 100 },
pro: { windowMs: 60000, max: 1000 },
enterprise: { windowMs: 60000, max: 10000 },
};
const dynamicRateLimit = async (req, res, next) => {
const client = req.apiClient; // Set by auth middleware
const tier = client?.tier || 'free';
const limits = tierLimits[tier];
const key = `ratelimit:${client?.id || req.ip}`;
const current = await redis.incr(key);
if (current === 1) {
await redis.pexpire(key, limits.windowMs);
}
res.set('RateLimit-Limit', limits.max);
res.set('RateLimit-Remaining', Math.max(0, limits.max - current));
if (current > limits.max) {
throw new RateLimitError(Math.ceil(limits.windowMs / 1000));
}
next();
};
Input Validation
Never trust client input. Validate everything, fail fast, and return clear errors.
Zod — The Modern Choice
Zod gives you TypeScript-first validation with excellent error messages.
import { z } from 'zod';
// Define a schema
const createUserSchema = z.object({
name: z
.string()
.min(2, 'Name must be at least 2 characters')
.max(100, 'Name must be under 100 characters'),
email: z
.string()
.email('Must be a valid email address'),
age: z
.number()
.int('Age must be a whole number')
.min(13, 'Must be at least 13 years old')
.max(150, 'Invalid age'),
role: z
.enum(['user', 'admin', 'moderator'])
.default('user'),
preferences: z.object({
newsletter: z.boolean().default(false),
theme: z.enum(['light', 'dark']).default('light'),
}).optional(),
});
// Validation middleware factory
const validate = (schema) => (req, res, next) => {
const result = schema.safeParse(req.body);
if (!result.success) {
const errors = result.error.issues.map(issue => ({
field: issue.path.join('.'),
message: issue.message,
code: issue.code,
}));
throw new ValidationError(errors);
}
req.validatedBody = result.data; // Use parsed data (with defaults applied)
next();
};
// Usage in routes
app.post('/users', validate(createUserSchema), async (req, res) => {
// req.validatedBody is guaranteed to be valid
const user = await db.users.create(req.validatedBody);
res.status(201).json(user);
});
Validation for Query Parameters
Don't forget to validate query params too:
const listUsersQuery = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
sort: z.enum(['name', 'created_at', 'email']).default('created_at'),
order: z.enum(['asc', 'desc']).default('desc'),
role: z.enum(['user', 'admin', 'moderator']).optional(),
search: z.string().max(200).optional(),
});
const validateQuery = (schema) => (req, res, next) => {
const result = schema.safeParse(req.query);
if (!result.success) {
const errors = result.error.issues.map(issue => ({
field: issue.path.join('.'),
message: issue.message,
}));
throw new ValidationError(errors);
}
req.validatedQuery = result.data;
next();
};
app.get('/users', validateQuery(listUsersQuery), async (req, res) => {
const { page, limit, sort, order, role, search } = req.validatedQuery;
// All values are validated and typed
});
Filtering, Sorting, and Pagination
Any API that returns lists needs these three. Get them right from day one — retrofitting pagination into an API is painful.
Pagination: Cursor vs Offset
OFFSET PAGINATION:
GET /users?page=3&limit=20
-> Skip 40, take 20
Pros: Simple, jump to any page
Cons: Slow on large datasets, inconsistent if data changes
Page 1: [1, 2, 3, ..., 20 ]
Page 2: [21, 22, 23, ..., 40 ] <- If item 5 is deleted,
Page 3: [41, 42, 43, ..., 60 ] page 2 shifts and you miss item 21
CURSOR PAGINATION:
GET /users?limit=20
GET /users?limit=20&cursor=eyJpZCI6MjB9
Pros: Consistent, fast on large datasets
Cons: Can't jump to page N, slightly more complex
First: [1, 2, 3, ..., 20 ] cursor -> "20"
Next: [21, 22, 23, ..., 40 ] cursor -> "40"
Next: [41, 42, 43, ..., 60 ] cursor -> "60"
(Stable regardless of inserts/deletes)
Offset Pagination Implementation
app.get('/users', async (req, res) => {
const page = Math.max(1, parseInt(req.query.page) || 1);
const limit = Math.min(100, Math.max(1, parseInt(req.query.limit) || 20));
const offset = (page - 1) * limit;
const [users, total] = await Promise.all([
db.users.findMany({ skip: offset, take: limit }),
db.users.count(),
]);
res.json({
data: users,
meta: {
total,
page,
per_page: limit,
total_pages: Math.ceil(total / limit),
has_next: page * limit < total,
has_prev: page > 1,
},
});
});
// Response:
{
"data": [...],
"meta": {
"total": 245,
"page": 3,
"per_page": 20,
"total_pages": 13,
"has_next": true,
"has_prev": true
}
}
Cursor Pagination Implementation
app.get('/users', async (req, res) => {
const limit = Math.min(100, parseInt(req.query.limit) || 20);
const cursor = req.query.cursor
? JSON.parse(Buffer.from(req.query.cursor, 'base64url').toString())
: null;
const where = cursor ? { id: { gt: cursor.id } } : {};
const users = await db.users.findMany({
where,
take: limit + 1, // Fetch one extra to check if there's a next page
orderBy: { id: 'asc' },
});
const hasNext = users.length > limit;
if (hasNext) users.pop(); // Remove the extra item
const nextCursor = hasNext
? Buffer.from(JSON.stringify({ id: users[users.length - 1].id }))
.toString('base64url')
: null;
res.json({
data: users,
meta: {
has_next: hasNext,
next_cursor: nextCursor,
},
});
});
// Response:
{
"data": [...],
"meta": {
"has_next": true,
"next_cursor": "eyJpZCI6NDB9"
}
}
Filtering and Sorting
app.get('/products', async (req, res) => {
const {
category, // ?category=electronics
min_price, // ?min_price=100
max_price, // ?max_price=500
in_stock, // ?in_stock=true
sort = 'created_at', // ?sort=price or ?sort=-price (descending)
} = req.query;
// Build filter object dynamically
const where = {};
if (category) where.category = category;
if (in_stock) where.inStock = in_stock === 'true';
if (min_price) where.price = { ...where.price, gte: parseFloat(min_price) };
if (max_price) where.price = { ...where.price, lte: parseFloat(max_price) };
// Parse sort: "-price" means descending
const sortField = sort.startsWith('-') ? sort.slice(1) : sort;
const sortOrder = sort.startsWith('-') ? 'desc' : 'asc';
const products = await db.products.findMany({
where,
orderBy: { [sortField]: sortOrder },
});
res.json({ data: products });
});
// Usage:
// GET /products?category=electronics&min_price=100&sort=-price&in_stock=true
When to Use Which Pagination
| Scenario | Use |
|---|---|
| Admin dashboard with "go to page 5" | Offset |
| Infinite scroll feed | Cursor |
| Real-time data (social feeds, logs) | Cursor |
| Small dataset (< 10k rows) | Either works |
| Large dataset (millions of rows) | Cursor (offset gets slow) |
Bulk Operations
Sometimes clients need to create, update, or delete many resources at once. Don't make them call your API 500 times.
Batch Endpoints
// Batch create
POST /users/batch
{
"items": [
{ "name": "Alice", "email": "alice@example.com" },
{ "name": "Bob", "email": "bob@example.com" },
{ "name": "Charlie", "email": "invalid-email" }
]
}
// Response — report per-item results
{
"results": [
{ "index": 0, "status": "created", "data": { "id": 501, "name": "Alice" } },
{ "index": 1, "status": "created", "data": { "id": 502, "name": "Bob" } },
{ "index": 2, "status": "failed", "error": { "field": "email", "message": "Invalid email" } }
],
"summary": {
"total": 3,
"succeeded": 2,
"failed": 1
}
}
Async Processing for Large Jobs
For operations that take too long for a synchronous response, use the async job pattern.
// Client kicks off a large import
POST /imports
{
"type": "users",
"file_url": "https://storage.example.com/users.csv"
}
// Server returns immediately with a job ID
// 202 Accepted
{
"job_id": "job_abc123",
"status": "queued",
"status_url": "/jobs/job_abc123",
"estimated_duration": "2-5 minutes"
}
// Client polls for status
GET /jobs/job_abc123
{
"job_id": "job_abc123",
"status": "processing", // queued | processing | completed | failed
"progress": {
"total": 5000,
"processed": 3247,
"failed": 12,
"percent": 64.9
},
"created_at": "2026-03-15T10:00:00Z",
"updated_at": "2026-03-15T10:02:15Z"
}
// When complete:
GET /jobs/job_abc123
{
"job_id": "job_abc123",
"status": "completed",
"result": {
"total": 5000,
"imported": 4988,
"failed": 12,
"errors_url": "/jobs/job_abc123/errors"
}
}
Client API Server Job Queue
| | |
|--POST /imports---------->| |
| |--Enqueue job------------>|
|<--202 { job_id }--------| |
| | |
|--GET /jobs/abc123------->| |
|<--{ status: processing }-| |
| | Worker picks up |
| ... wait ... | and processes |
| | |
|--GET /jobs/abc123------->| |
|<--{ status: completed }--| |
Webhooks
Webhooks are the flip side of APIs — instead of clients calling you, you call them when something interesting happens.
Sending Webhooks
// Webhook dispatcher
class WebhookDispatcher {
async send(event, payload, webhookUrl, secret) {
const body = JSON.stringify({
id: crypto.randomUUID(),
event,
created_at: new Date().toISOString(),
data: payload,
});
// Sign the payload so receivers can verify it's from you
const signature = crypto
.createHmac('sha256', secret)
.update(body)
.digest('hex');
const response = await fetch(webhookUrl, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Webhook-Signature': `sha256=${signature}`,
'X-Webhook-Event': event,
'X-Webhook-Delivery': crypto.randomUUID(),
},
body,
signal: AbortSignal.timeout(10000), // 10s timeout
});
return { status: response.status, success: response.ok };
}
}
// Usage
const dispatcher = new WebhookDispatcher();
await dispatcher.send(
'order.completed',
{ order_id: 456, total: 99.99, customer_id: 123 },
'https://client-app.com/webhooks',
'whsec_abc123'
);
Retry Logic with Exponential Backoff
async sendWithRetry(event, payload, webhookUrl, secret, maxRetries = 5) {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
const result = await this.send(event, payload, webhookUrl, secret);
if (result.success) return result;
// 4xx errors (except 429) — don't retry, client's problem
if (result.status >= 400 && result.status < 500 && result.status !== 429) {
return result;
}
} catch (err) {
// Network error, timeout, etc.
}
if (attempt < maxRetries) {
// Exponential backoff: 1s, 2s, 4s, 8s, 16s
const delay = Math.pow(2, attempt) * 1000;
await new Promise(resolve => setTimeout(resolve, delay));
}
}
// All retries exhausted — mark webhook as failing
await this.markFailing(webhookUrl);
}
Receiving and Verifying Webhooks
// On the receiving end — verify the signature
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), (req, res) => {
const signature = req.headers['x-webhook-signature'];
const expectedSignature = 'sha256=' + crypto
.createHmac('sha256', process.env.WEBHOOK_SECRET)
.update(req.body)
.digest('hex');
if (!crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
)) {
return res.status(401).json({ error: 'Invalid signature' });
}
const event = JSON.parse(req.body);
// Process the event...
// Always respond 200 quickly — do heavy processing async
res.status(200).json({ received: true });
});
HATEOAS and Hypermedia
HATEOAS (Hypermedia as the Engine of Application State) is the idea that API responses include links to related actions and resources. It makes APIs self-discoverable.
What It Looks Like
// Without HATEOAS — client must hardcode URLs
{
"id": 123,
"name": "Alex",
"status": "active"
}
// With HATEOAS — response tells client what it can do next
{
"id": 123,
"name": "Alex",
"status": "active",
"_links": {
"self": { "href": "/users/123", "method": "GET" },
"update": { "href": "/users/123", "method": "PATCH" },
"delete": { "href": "/users/123", "method": "DELETE" },
"orders": { "href": "/users/123/orders", "method": "GET" },
"suspend": { "href": "/users/123/actions/suspend", "method": "POST" }
}
}
// For a suspended user — different links available
{
"id": 123,
"name": "Alex",
"status": "suspended",
"_links": {
"self": { "href": "/users/123", "method": "GET" },
"reactivate": { "href": "/users/123/actions/reactivate", "method": "POST" }
// No update, delete, or suspend links — not available in this state
}
}
When Does HATEOAS Matter?
Honestly? For most APIs, it doesn't. HATEOAS matters when:
- You're building a truly public API used by many third-party clients
- Your API has complex state machines where available actions change based on state
- You want clients to be resilient to URL changes
For internal APIs or simple CRUD, don't over-engineer it. Just document your endpoints well.
API Documentation
The best-designed API is useless if nobody can figure out how to use it.
OpenAPI/Swagger
OpenAPI is the industry standard for describing REST APIs. Write it, and you get interactive docs, client generation, and validation for free.
# openapi.yaml
openapi: 3.0.3
info:
title: Users API
version: 1.0.0
description: User management endpoints
paths:
/users:
get:
summary: List all users
parameters:
- name: page
in: query
schema:
type: integer
default: 1
- name: limit
in: query
schema:
type: integer
default: 20
maximum: 100
responses:
'200':
description: Paginated list of users
content:
application/json:
schema:
type: object
properties:
data:
type: array
items:
$ref: '#/components/schemas/User'
meta:
$ref: '#/components/schemas/PaginationMeta'
post:
summary: Create a user
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/CreateUser'
responses:
'201':
description: User created
components:
schemas:
User:
type: object
properties:
id:
type: integer
name:
type: string
email:
type: string
format: email
Auto-Generation from Code
With tools like swagger-jsdoc, you can keep docs next to your code:
/**
* @openapi
* /users/{id}:
* get:
* summary: Get a user by ID
* parameters:
* - in: path
* name: id
* required: true
* schema:
* type: integer
* responses:
* 200:
* description: User found
* 404:
* description: User not found
*/
app.get('/users/:id', getUser);
Documentation Checklist
- [ ] Every endpoint documented with description
- [ ] Request/response examples for every endpoint
- [ ] Error response examples
- [ ] Authentication requirements explained
- [ ] Rate limit information included
- [ ] Pagination explained with examples
- [ ] Runnable in Postman or similar tools
- [ ] Changelog for API versions
API Security Checklist
Security is not optional. Here's your checklist, aligned with the OWASP API Security Top 10.
The Essentials
import helmet from 'helmet';
import cors from 'cors';
import hpp from 'hpp';
import mongoSanitize from 'express-mongo-sanitize';
const app = express();
// 1. Security headers
app.use(helmet());
// 2. CORS — be explicit, never use wildcard in production
app.use(cors({
origin: ['https://myapp.com', 'https://admin.myapp.com'],
methods: ['GET', 'POST', 'PUT', 'PATCH', 'DELETE'],
allowedHeaders: ['Content-Type', 'Authorization'],
credentials: true,
maxAge: 86400,
}));
// 3. Body size limits — prevent payload attacks
app.use(express.json({ limit: '10kb' }));
// 4. Prevent HTTP parameter pollution
app.use(hpp());
// 5. Sanitize input against NoSQL injection
app.use(mongoSanitize());
// 6. Rate limiting (covered above)
app.use('/api/', apiLimiter);
OWASP API Security Top 10 Quick Reference
| # | Risk | Mitigation |
|---|---|---|
| 1 | Broken Object-Level Auth | Always verify resource ownership |
| 2 | Broken Authentication | Strong password policy, MFA, short-lived tokens |
| 3 | Broken Object Property-Level Auth | Whitelist fields, never return everything from DB |
| 4 | Unrestricted Resource Consumption | Rate limits, pagination limits, payload size limits |
| 5 | Broken Function-Level Auth | RBAC middleware, check permissions per endpoint |
| 6 | Unrestricted Access to Sensitive Flows | CAPTCHA, rate limit on sensitive operations |
| 7 | Server-Side Request Forgery (SSRF) | Validate/sanitize URLs, allowlist external calls |
| 8 | Security Misconfiguration | Disable debug in prod, review CORS, use helmet |
| 9 | Improper Inventory Management | Document all endpoints, deprecate unused ones |
| 10 | Unsafe Consumption of APIs | Validate data from third-party APIs too |
Authorization Middleware Pattern
// Role-based access control
const requireRole = (...roles) => (req, res, next) => {
if (!req.user) {
throw new UnauthorizedError();
}
if (!roles.includes(req.user.role)) {
throw new ForbiddenError(`Requires one of: ${roles.join(', ')}`);
}
next();
};
// Object-level authorization — CRITICAL
const requireOwnership = (resourceFetcher) => async (req, res, next) => {
const resource = await resourceFetcher(req);
if (!resource) {
throw new NotFoundError('Resource', req.params.id);
}
if (resource.userId !== req.user.id && req.user.role !== 'admin') {
throw new ForbiddenError('You do not own this resource');
}
req.resource = resource;
next();
};
// Usage
app.delete('/orders/:id',
authenticateJWT,
requireOwnership((req) => db.orders.findById(req.params.id)),
async (req, res) => {
await db.orders.delete(req.resource.id);
res.status(204).end();
}
);
app.get('/admin/users',
authenticateJWT,
requireRole('admin'),
listAllUsers
);
Bad vs Good API Patterns
Let's put it all together with side-by-side comparisons.
URL Design
BAD GOOD
---- ----
GET /getAllUsers GET /users
POST /createNewUser POST /users
GET /getUserById?id=123 GET /users/123
POST /deleteUser DELETE /users/123
GET /getOrdersForUser/123 GET /users/123/orders
POST /user/update PATCH /users/123
Error Responses
// BAD — Unhelpful error
{
"error": true,
"message": "Bad Request"
}
// GOOD — Actionable error with context
{
"type": "https://api.example.com/errors/validation",
"title": "Validation Error",
"status": 422,
"detail": "Request validation failed for 2 fields",
"errors": [
{ "field": "email", "message": "Must be a valid email", "value": "notanemail" },
{ "field": "age", "message": "Must be at least 13", "value": 5 }
]
}
Pagination
// BAD — No pagination info, no way to know total
{
"users": [...]
}
// GOOD — Complete pagination metadata
{
"data": [...],
"meta": {
"total": 1532,
"page": 2,
"per_page": 20,
"total_pages": 77,
"has_next": true,
"has_prev": true
}
}
Authentication Errors
// BAD — Same vague error for everything
{
"error": "Access denied"
}
// GOOD — Distinguish between auth issues
// No token:
{
"status": 401,
"title": "Unauthorized",
"detail": "Authentication required. Provide a Bearer token."
}
// Valid token, wrong permissions:
{
"status": 403,
"title": "Forbidden",
"detail": "Your role 'editor' cannot access admin endpoints."
}
Response Shapes
// BAD — Inconsistent responses across endpoints
// GET /users returns:
[{ "user_name": "Alex" }]
// GET /products returns:
{ "result": [{ "productName": "Laptop" }] }
// GOOD — Same envelope everywhere
// GET /users returns:
{ "data": [{ "name": "Alex" }], "meta": { "total": 100 } }
// GET /products returns:
{ "data": [{ "name": "Laptop" }], "meta": { "total": 50 } }
Putting It All Together — Express.js Example
Here's a production-ready Express.js API skeleton that incorporates every pattern we've discussed.
// app.js
import express from 'express';
import helmet from 'helmet';
import cors from 'cors';
import rateLimit from 'express-rate-limit';
import { z } from 'zod';
const app = express();
// ==================== MIDDLEWARE ====================
app.use(helmet());
app.use(cors({ origin: process.env.ALLOWED_ORIGINS?.split(',') }));
app.use(express.json({ limit: '10kb' }));
// Rate limiting
app.use('/api/', rateLimit({
windowMs: 60 * 1000,
max: 100,
standardHeaders: true,
legacyHeaders: false,
}));
// ==================== ERROR CLASSES ====================
class AppError extends Error {
constructor(message, statusCode, code) {
super(message);
this.statusCode = statusCode;
this.code = code;
this.isOperational = true;
}
}
class NotFoundError extends AppError {
constructor(resource, id) {
super(`${resource} with id '${id}' not found`, 404, 'NOT_FOUND');
}
}
class ValidationError extends AppError {
constructor(errors) {
super('Validation failed', 422, 'VALIDATION_ERROR');
this.errors = errors;
}
}
// ==================== VALIDATION ====================
const createUserSchema = z.object({
name: z.string().min(2).max(100),
email: z.string().email(),
role: z.enum(['user', 'admin']).default('user'),
});
const paginationSchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
sort: z.string().default('-created_at'),
});
const validate = (schema, source = 'body') => (req, res, next) => {
const result = schema.safeParse(req[source]);
if (!result.success) {
const errors = result.error.issues.map(i => ({
field: i.path.join('.'),
message: i.message,
}));
throw new ValidationError(errors);
}
req.validated = result.data;
next();
};
// ==================== ROUTES ====================
const router = express.Router();
// List users with pagination, filtering, sorting
router.get('/users', validate(paginationSchema, 'query'), async (req, res) => {
const { page, limit, sort } = req.validated;
const offset = (page - 1) * limit;
// Parse sort: "-created_at" -> { field: "created_at", order: "desc" }
const sortField = sort.startsWith('-') ? sort.slice(1) : sort;
const sortOrder = sort.startsWith('-') ? 'desc' : 'asc';
const [users, total] = await Promise.all([
db.users.findMany({
skip: offset,
take: limit,
orderBy: { [sortField]: sortOrder },
}),
db.users.count(),
]);
res.json({
data: users,
meta: {
total,
page,
per_page: limit,
total_pages: Math.ceil(total / limit),
has_next: page * limit < total,
has_prev: page > 1,
},
});
});
// Get single user
router.get('/users/:id', async (req, res) => {
const user = await db.users.findById(req.params.id);
if (!user) throw new NotFoundError('User', req.params.id);
res.json({ data: user });
});
// Create user
router.post('/users', validate(createUserSchema), async (req, res) => {
const user = await db.users.create(req.validated);
res
.status(201)
.location(`/api/v1/users/${user.id}`)
.json({ data: user });
});
// Update user
router.patch('/users/:id', async (req, res) => {
const user = await db.users.findById(req.params.id);
if (!user) throw new NotFoundError('User', req.params.id);
const updated = await db.users.update(req.params.id, req.body);
res.json({ data: updated });
});
// Delete user
router.delete('/users/:id', async (req, res) => {
const user = await db.users.findById(req.params.id);
if (!user) throw new NotFoundError('User', req.params.id);
await db.users.delete(req.params.id);
res.status(204).end();
});
// Mount versioned routes
app.use('/v1', router);
// ==================== ERROR HANDLER ====================
app.use((err, req, res, next) => {
if (err.isOperational) {
const response = {
type: `https://api.example.com/errors/${err.code.toLowerCase()}`,
title: err.code.replace(/_/g, ' '),
status: err.statusCode,
detail: err.message,
instance: req.originalUrl,
};
if (err.errors) response.errors = err.errors;
return res.status(err.statusCode).json(response);
}
console.error(err);
res.status(500).json({
type: 'https://api.example.com/errors/internal',
title: 'Internal Server Error',
status: 500,
detail: 'An unexpected error occurred',
});
});
app.listen(3000, () => console.log('API running on port 3000'));
Decision Framework
Not sure which approach to pick? Use this cheatsheet.
| Decision | Default Choice | Consider Alternative When |
|---|---|---|
| Pagination | Cursor-based | Small dataset, need "jump to page N" -> Offset |
| Auth | JWT Bearer Tokens | Server-to-server -> API Keys; Third party -> OAuth |
| Versioning | URL path /v1/
|
Internal-only API -> Headers |
| Response format | JSON with envelope | Need streaming -> NDJSON; Need efficiency -> Protobuf |
| IDs | UUIDs | Need ordering -> ULID; Human-readable -> Nanoid |
| Validation | Zod | Already in Joi/Hapi ecosystem -> Joi |
| Docs | OpenAPI + SwaggerUI | Simple internal -> Markdown is fine |
| Rate limiting | Sliding window | Need burst tolerance -> Token bucket |
| Bulk operations | Batch endpoint | Very large (>1000) -> Async job queue |
| Error format | RFC 7807 | Already have a standard in org -> use that |
The API Design Checklist
Before you ship, walk through this:
- [ ] URLs — Nouns, plural, kebab-case, max 2 nesting levels
- [ ] Methods — Correct HTTP method for each operation
- [ ] Status codes — Right code for every scenario (not just 200/500)
- [ ] Validation — Every input validated, clear field-level errors
- [ ] Auth — Every endpoint has appropriate authentication
- [ ] Authorization — Object-level access control (not just role checks)
- [ ] Pagination — Every list endpoint is paginated
- [ ] Rate limiting — Enforced with proper headers
- [ ] Errors — Consistent format, actionable messages
- [ ] Versioning — Strategy in place before v1 ships
- [ ] Docs — OpenAPI spec, examples for every endpoint
- [ ] Security — Helmet, CORS, input limits, OWASP top 10 covered
- [ ] Idempotency — POST endpoints support idempotency keys
- [ ] Logging — Request IDs, structured logs for debugging
Build APIs like you're the one who has to consume them at 2 AM during an outage. Be kind to your future self.
Let's Connect!
If you found this guide helpful, I'd love to connect with you! I regularly share deep dives on system design, backend engineering, and software architecture.
Connect with me on LinkedIn — let's grow together.
Drop a comment, share this with someone who needs this, and follow along for more guides like this!
Top comments (0)