# When a 500-Request Burst Hits at 2 AM, Your Architecture Either Drives or Dies
Let us be honest. Nobody wakes up proud of their checkout service. Not until it burns down at 02:17 on a Tuesday during a flash sale and you are reading your own error logs like a crime scene report.
The alert that night was brutal: **checkout-service: 500, DB connection pool exhausted**. A junior dev had glued everything into one file with a global pool. Another team shipped the same weekend with a plug-in mesh. Only one survived.
I have been building production systems since before microservices was a LinkedIn buzzword. The code below is from real builds I have shipped. [Full-stack reference codebase here](https://www.shipmvp.tech) if you want to see the patterns in context.
---
## 1. What Went Wrong: The Junior's Single-File Trap
js
// checkout.js - 300 lines of "just get it done"
const { Pool } = require('pg')
const pool = new Pool({ connectionString: process.env.DATABASE_URL })
const express = require('express')
const app = express()
app.use(express.json())
app.post('/checkout', async (req, res) => {
const client = await pool.connect()
try {
await client.query('BEGIN')
const { items, userId } = req.body
if (!items?.length) throw new Error('Empty cart')
// Sequential loop: each SKU blocks the event loop
for (const i of items) {
await client.query(
'UPDATE inventory SET qty = qty - $1 WHERE sku = $2 AND qty >= $1',
[i.qty, i.sku]
)
}
const orderRes = await client.query(
'INSERT INTO orders(user_id, status) VALUES($1, $2) RETURNING id',
[userId, 'pending']
)
const orderId = orderRes.rows[0].id
for (const i of items) {
await client.query(
'INSERT INTO order_items(order_id, sku, qty) VALUES($1, $2, $3)',
[orderId, i.sku, i.qty]
)
}
await client.query('COMMIT')
res.json({ orderId })
} catch (e) {
await client.query('ROLLBACK')
console.error(e)
res.status(500).send('checkout failed')
} finally {
client.release()
}
})
app.listen(3000)
This is not just bad code. It is a bomb with three fuses:
1. **Global pool defaults to size 10.** Five hundred parallel requests queue behind ten connections. Pool exhausts. Gateway times out. PagerDuty screams.
2. **Sequential `for...await` loops.** Each inventory update blocks the event loop. One request walks its SKU list while the entire service sits idle.
3. **Zero separation of concerns.** Validation, inventory, ordering, and transaction handling all live together. One typo and the whole thing detonates.
---
## 2. The Senior Approach: Plug-in Mesh Architecture
The fix treats each responsibility as a swappable module. The orchestrator stays thin. The plugins do the work.
### Directory layout
plaintext
src/
├─ adapters/db/pgPool.js
├─ domain/checkout/
│ ├─ validator.js
│ ├─ inventoryService.js
│ ├─ orderService.js
│ └─ useCase.js
└─ index.js
### Pool Sized for 8 GB RAM, Not Guesswork
js
// src/adapters/db/pgPool.js
const { Pool } = require('pg')
// 1 PG connection ≈ 50 MB heap. On 8 GB, leave headroom for Node runtime.
const availBytes = process.memoryUsage().heapTotal
const poolSize = Math.max(2, Math.floor(availBytes / 1024 / 1024 / 50))
const pool = new Pool({
connectionString: process.env.DATABASE_URL,
max: Math.min(poolSize, 20),
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 2000
})
module.exports = pool
Stop hardcoding pool sizes. Your heap tells you what it can handle.
### Parallel Inventory Reservations
js
// src/domain/checkout/inventoryService.js
module.exports = class InventoryService {
constructor(pool) { this.pool = pool }
async reserve(items, client) {
// Promise.all fires all SKU updates concurrently within the same transaction
const queries = items.map(i =>
client.query(
'UPDATE inventory SET qty = qty - $1 WHERE sku = $2 AND qty >= $1',
[i.qty, i.sku]
)
)
// Fails fast: one out-of-stock SKU rolls back everything
await Promise.all(queries)
}
}
### The Orchestrator
js
// src/domain/checkout/useCase.js
module.exports = class CheckoutUseCase {
constructor({ validator, inventoryService, orderService, pool }) {
this.validator = validator
this.inventoryService = inventoryService
this.orderService = orderService
this.pool = pool
}
async execute(payload) {
this.validator(payload)
const client = await this.pool.connect()
try {
await client.query('BEGIN')
await this.inventoryService.reserve(payload.items, client)
const orderId = await this.orderService.create(
payload.userId, payload.items, client
)
await client.query('COMMIT')
return orderId
} catch (err) {
await client.query('ROLLBACK')
throw err
} finally {
client.release()
}
}
}
Clean. Testable. Swap any plugin without touching the others.
---
## 3. Failure Modes Under Burst Load
| Scenario | Junior Code | Senior Code |
|----------|-------------|-------------|
| 500 parallel checkouts | Pool exhausts, 500 errors | Parallel `Promise.all` cuts latency; pool auto-sizes to 16 |
| One SKU out of stock | Transaction hangs in sequential loop | Fails fast; ROLLBACK fires immediately |
| GC pressure on 8 GB box | No heap awareness; pool grows unbounded | Pool capped at 20; idle connections released after 30s |
| Partial failure mid-transaction | Rollback may silently fail | Explicit `try/catch/finally` guarantees ROLLBACK |
The race condition wins here. `Promise.all` within a single transaction eliminates the TOCTOU gap. The database enforces atomicity: all reservations succeed or all roll back. No gray areas.
---
## 4. Observability: Because Console Logs Are for Hobbyists
js
// After successful checkout:
console.log(JSON.stringify({
event: 'checkout_success',
orderId,
latencyMs: Date.now() - start,
items: payload.items.length,
traceId: req.headers['x-trace-id']
}))
Prometheus counters. Latency histograms. A circuit breaker on the inventory service. When the pool hits 90 percent utilization, return `503 Service Unavailable` instead of queuing until the server buckles.
Graceful degradation beats heroic collapse every time.
---
## 5. The Question That Keeps You Up at Night
**If you size your connection pool purely by heap memory, how do you account for load balancers, sidecars, and logging agents eating the same 8 GB?** The answer separates services that survive Black Friday from services that become cautionary tales on your ops team's Slack channel.
My Lamborghini Urus is not the fastest supercar. It is not the most refined. But it handles situations no other car in its class touches: gravel, rain, steep mountain passes, and city traffic. Same philosophy here. Your architecture does not need to be the prettiest. It needs to handle whatever the road throws at it.
The code above ships. [See the full reference](https://www.shipmvp.tech) if you want production-hardened patterns, not just theory.
---
**Open Loop:** When your connection pool is sized dynamically from heap memory, which other hidden consumers of that same memory footprint should factor into your capacity calculation: the Node.js V8 engine itself, OS-level buffer caches, or the overhead of your process manager's forked workers? And how do you measure those without guessing?
Top comments (0)