DEV Community: Olawale Afuye

Your Ticket Was Closed. The User Still Couldn't Pay.

Olawale Afuye — Wed, 17 Jun 2026 15:49:13 +0000

Your backend returned 200.

The mobile app showed an error.

The user tapped "Pay" three times.

Three pending charges hit their account. One order was placed. Their balance was short. And your incident log showed zero failures.

Every engineer on the team did their job. Nobody solved the problem.

This is the most common way engineering teams fail, not through incompetence, but through excellent execution of the wrong unit of work. And until you recognise the difference between completing a task and solving a business problem, you will keep shipping systems that work perfectly and experiences that don't.

The Ticket-Thinker vs. The System-Owner

Most engineers early in their careers think in tickets.

Ticket assigned → code written → tests pass → PR merged → ticket closed. Done.

This is fine when you're learning. It's a liability when you're trying to grow.

The engineer who closes tickets is useful. The engineer who asks "what problem does this ticket actually solve, and am I solving it in the right place?" that engineer is dangerous in the best way.

Here's the distinction in practice.

The backend engineer builds a payment endpoint. It processes charges correctly, returns the right status codes, has proper error handling. 100% test coverage. Ticket closed.

The mobile engineer builds the payment screen. It calls the endpoint, handles the response, shows confirmation or error. Smooth UI. Ticket closed.

The problem nobody owned: what happens when the network drops after the backend processes the charge but before the mobile app receives the confirmation?

The backend: charge processed. No error.
The mobile: timeout. Shows "Payment failed." User retries.
The user: charged twice.

Both engineers solved their assigned problem correctly. The business problem — charge the user once and confirm it reliably — went unsolved. Because that problem lived in the space between their tickets, and nobody was watching that space.

Real Scenario 1: The Payment That Worked and Failed at the Same Time

This happens in production more than any team admits.

In a payment flow, the sequence is: mobile initiates → backend charges → payment processor confirms → backend responds → mobile confirms to user.

Network latency exists at every arrow in that chain.

If the connection between the backend and mobile drops after the payment processor confirms but before the backend responds to the mobile, both the backend log and the payment processor log show success. The mobile app shows "Payment failed. Please try again."

A user who trusts the mobile app retries. Now they're charged twice.

The fix isn't purely a backend fix. It isn't purely a mobile fix. It requires:

Idempotency keys — the mobile generates a unique key per payment attempt and sends it with every request. The backend uses it to guarantee that retrying the same request never creates a duplicate charge, regardless of how many times the network drops and retries.

// Mobile: generate and persist the idempotency key per payment intent
const idempotencyKey = `pay_${userId}_${orderId}_${Date.now()}`;

// Store it locally before the request
localStorage.setItem('pending_payment_key', idempotencyKey);

// Send with every retry of this specific payment
const response = await fetch('/api/payments', {
  method: 'POST',
  headers: {
    'Idempotency-Key': idempotencyKey,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({ amount, currency, orderId })
});

// Backend: check for existing successful charge with this key
async function processPayment(req) {
  const idempotencyKey = req.headers['idempotency-key'];

  const existing = await db.payments.findOne({ idempotencyKey });
  if (existing?.status === 'success') {
    return existing; // Return the same result. Don't charge again.
  }

  const charge = await paymentProcessor.charge(req.body);
  await db.payments.create({ idempotencyKey, ...charge });
  return charge;
}

This solution only exists if a backend engineer and mobile engineer sat down together and asked: what does the user experience look like when the network misbehaves? Not: does my component work?

That's the difference.

Real Scenario 2: The Smart Device That "Works"

A team builds a smart home device. Hardware, mobile app, cloud backend, three separate engineering workstreams.

The hardware engineer ships firmware that correctly sends state changes to the cloud API. Tests pass. Ticket closed.

The mobile engineer ships an app that correctly receives state changes from the cloud and updates the UI. Tests pass. Ticket closed.

The backend engineer ships an API that receives from hardware and sends to mobile. Load tested. Ticket closed.

Users buy the device. They press the button to turn on their light.

The light turns on 11 seconds later.

Nobody's system is broken. The latency was distributed across three components, each one individually fine, each one adding 3–4 seconds of its own processing and polling delay. Nobody measured the end-to-end journey. Nobody owned the number that the user actually experiences: the time between button press and light turning on.

The product reviews say "laggy" and "unresponsive." The engineering team looks at their metrics and sees nothing wrong.

This is what happens when reliability is treated as a component property instead of a system property.

Real reliability — the kind users actually experience only exists at the intersection of every layer. The backend can be 99.9% available. If the mobile SDK polls every 5 seconds, the effective user-facing response time is up to 5 seconds before the backend is even consulted. Hardware transmission latency on top of that. Cloud-to-mobile push latency on top of that.

The only way to catch this is to instrument the entire journey, not individual components:

// Instrument the user-facing journey end to end
// Not just "did the API respond?" but "did the user get feedback?"

const journeyStart = performance.now();

await hardwareCommandAPI.send(deviceId, 'toggle_light');

// Poll for state change confirmation from device
await waitForDeviceStateChange(deviceId, 'on', { timeoutMs: 2000 });

const journeyEnd = performance.now();
const userFacingLatency = journeyEnd - journeyStart;

metrics.record('light_toggle_user_latency_ms', userFacingLatency);

When this number starts living in your dashboards, cross-functional conversations change. "The API is fast" stops being the end of the discussion.

Why Engineers Stay Stuck in the Ticket Mindset

It's not laziness. It's incentive structure.

Most engineering teams measure and reward what's visible: tickets closed, PRs merged, features shipped, uptime of individual services.

Nobody measures "how many times did an engineer spot a problem outside their lane and raise it?" Nobody gives performance review credit for the mobile engineer who asked the backend team: "what happens to our payment UI if your charge endpoint takes 8 seconds instead of 200ms?" And then followed up with: "here's what the user sees, here's the drop-off in our funnel."

The ticket system creates invisible walls between components. Each engineer optimizes for their component. The user lives in the space between the walls and has no advocate unless someone consciously takes on that role.

One of the clearest signs of engineering maturity is the ability to think beyond the ticket and own the user outcome.

Not deeper technical expertise in one domain. The willingness to hold the end-to-end user journey in your head while working in one specific layer of it.

What Cross-Functional Reliability Actually Looks Like

Collaboration here doesn't mean more meetings. It means shared ownership of outcomes rather than outputs.

Practically, this looks like:

Defining end-to-end SLOs, not just component SLOs.
Your backend's 99.9% availability means nothing to a user whose mobile app never got the response. Define what the user-facing journey reliability looks like and measure it across every layer together.

Writing integration tests that simulate the user, not the component.

// Component test (insufficient):
test('payment endpoint returns 200', async () => {
  const res = await request(app).post('/payments').send(payload);
  expect(res.status).toBe(200);
});

// Integration test (what actually matters):
test('user can complete payment even on slow network', async () => {
  // Simulate 3G latency
  networkCondition.set({ latency: 500, packetLoss: 0.05 });

  const result = await simulateUserPaymentJourney({
    userId: 'test-user',
    amount: 5000,
    retryOnTimeout: true
  });

  expect(result.charged).toBe(true);
  expect(result.chargeCount).toBe(1); // Exactly once. Not zero. Not two.
  expect(result.userConfirmed).toBe(true);
});

Doing joint failure mode analysis before shipping, not after an incident.
Get backend, mobile, and hardware/infrastructure engineers in the same room with one question: what happens to the user if this part fails? Run through every component. Write down what the user experiences at each failure point. Fix the ones that are unacceptable.

Instrumenting the user journey, not just the service.
Every system already has dashboards showing API response times, error rates, DB query performance. How many have a dashboard showing: user tapped Pay → charge confirmed → confirmation visible to user, with a latency distribution for the whole sequence? Build that one. It will tell you things your component dashboards never will.

The Mental Model Shift

Here's the reframe that changes how you approach your work:

Your job title describes your skill set. It doesn't describe the boundary of your responsibility.

You are a backend engineer who is responsible for users being able to pay reliably. You are a mobile engineer who is responsible for users having confidence in the product. You are a hardware engineer who is responsible for users trusting that the physical interaction works.

The moment you accept that your responsibility extends to the user outcome not just the technical component, you start asking different questions. You start talking to engineers in other layers. You start caring about what your API response time does to the mobile engineer's loading UX. You start caring about what the mobile engineer's retry logic does to your backend's duplicate-detection. You start caring about what the hardware's transmission delay does to the entire chain.

This isn't extra work. This is the actual work.

Closing tickets is a floor, not a ceiling. The engineers who grow fast are the ones who figure that out early. 🔧

What's the most painful cross-stack failure you've shipped or inherited? The ones where every component technically worked and the user still got hurt are the best learning stories. Drop them in the comments.

Your Codebase Is a Mess Because Your Team Can't Agree on What a "Customer" Is

Olawale Afuye — Sun, 07 Jun 2026 03:45:49 +0000

Nobody wants to hear this.

But the reason your software is hard to change, hard to test, and hard to explain to a new engineer isn't your tech stack.

It's that your code doesn't reflect how your business actually works.

Your engineers are using one word — "customer," "order," "student," "subscriber" — and meaning six different things depending on which part of the system they're touching. Your domain expert says "order" and means something completely different from what your database schema says "order" is.

That gap? That's where complexity lives. That's where bugs are born. That's where senior engineers spend their Fridays.

Domain-Driven Design is the discipline of closing that gap. Here's what it actually means, practically, without the academic noise.

The Core Problem: One Model Trying to Mean Everything

Imagine a map that tried to show subway routes, underwater hazards, hiking trails, and flight paths — all at once.

It would be useless.

A subway map works because it only shows what matters for navigating trains. A nautical chart works because it only shows what matters for sailing. Each map is an abstraction built for a specific purpose, valid within a specific context.

Your software models need to work the same way.

The moment you build a single "Customer" class that has to satisfy your billing team, your marketing team, your support team, and your logistics team simultaneously — that class becomes a bloated, ambiguous disaster. Everyone adds their fields. Nobody removes anything. The model stops meaning anything specific to anyone.

This is the monolithic model trap. And most large codebases are sitting right inside it.

Strategic Design: Understand the Problem Before You Touch Code

DDD separates design into two layers. Strategic design comes first — it's the work you do before writing a single line of code.

Step 1: Find Your Subdomains

A subdomain is a slice of the business problem. Ordering. Shipping. Notifications. Payments. Inventory. These aren't your microservices — they're your business problems, identified during analysis.

This identification should never happen in an engineering room alone. If your engineers figure out the subdomains without the business, they will model what they think the business does. That's not the same thing. Get the domain experts in the room. The goal of DDD is software that reflects real-world processes — that requires everyone on the same page.

Once you have your subdomains, classify them:

Core subdomains — your competitive advantage. Custom-build these. This is where your best engineers spend their time.
Supporting subdomains — necessary but not differentiating. Can be built simply or outsourced.
Generic subdomains — solved problems. Use off-the-shelf software. Don't reinvent email delivery.

Step 2: Establish a Ubiquitous Language

This is the one DDD concept that pays dividends even if you implement nothing else.

Ubiquitous language means the business and engineering teams use exactly the same words to describe the same things. Not synonyms. Not approximations. The same words.

If the business calls it a "subscription" and your code calls it a "plan" and your database calls it a "contract," you have a translation layer in every conversation. Every ticket. Every bug report. Every onboarding session for a new hire.

Eliminate the translations. Name things in code what the business calls them. The domain expert should be able to read your entity names and recognize their own vocabulary.

Step 3: Event Storming

Before designing your architecture, run an event storming session. Put domain experts and engineers in a room with sticky notes. Map out every event that happens in the system — things that occurred in the business domain, written in past tense: "Order Placed," "Payment Confirmed," "Drone Dispatched."

Then add the commands that trigger those events. Then the actors who issue those commands.

The wall of sticky notes becomes your shared understanding. Cluster the events, and you'll see your subdomains emerge naturally from the patterns.

Bounded Contexts: Where the Model Gets Its Boundaries

A bounded context is the architectural answer to the ubiquitous language question.

Here's the uncomfortable truth: the same word can legitimately mean different things in different parts of your business, and that's fine.

A tomato is a fruit in botany. A vegetable in culinary arts. Both are correct within their respective contexts. The botanical definition doesn't need to win. The chef's definition doesn't need to lose. Each is accurate within its domain.

"Subscriber" in your billing context means a paying entity with a plan, a billing cycle, and a payment method. "Subscriber" in your notification context means an endpoint that receives messages. Forcing these into one model creates friction in both directions.

Bounded contexts draw explicit lines. Within that line, the ubiquitous language is valid and consistent. Across the line, you manage translation deliberately — not accidentally.

Three kinds of boundaries a context creates:

Physical boundaries — the context operates as an independent service or deployable unit
Logical boundaries — the context is a module or package within a larger codebase, with strict interfaces
Ownership boundaries — one team owns the context entirely, end-to-end

That last one matters more than most teams realize. Shared ownership of a bounded context means two teams are making decisions that affect each other constantly. That friction shows up in your code as coupling. Give each context a clear owner. The organizational boundary reinforces the architectural one.

Context Mapping: Managing the Lines Between Contexts

Bounded contexts don't exist in isolation. An order context needs to talk to a shipping context. A payment context needs to signal a notification context.

A context map documents which domains interact, how they communicate, and which direction the relationship flows.

When two contexts need to exchange information, the critical tool is the Anti-Corruption Layer (ACL). This is a translation interface that sits between contexts. When the shipping context receives data from the order context, the ACL translates it into the shipping context's own model and language — preventing the order domain's concepts from bleeding into shipping's internal logic.

Without an ACL, you end up with leaky abstractions. The order team changes a field name, and suddenly the shipping team's tests break. That dependency is a design failure, not bad luck.

Tactical Design: Building the Model in Code

Strategic design tells you what to build and where the boundaries are. Tactical design tells you how to represent the domain in code.

Entities

An entity is an object with a unique identity that persists over time.

A Drone is an entity. Drone #A47 is not the same as Drone #B12, even if they have the same battery level and model. Identity is what matters, not attribute values. Entities are mutable — their state changes, but their identity doesn't.

Value Objects

A value object is defined entirely by its attributes. It has no identity of its own.

A battery level of 80% is just a value. One 80% is identical to any other 80%. There's no BatteryLevel #4291 — there's just the value.

Value objects are immutable. If the battery level changes, you don't mutate the existing object — you create a new one. This eliminates a whole class of bugs where shared references to mutable state lead to unexpected side effects.

Use value objects aggressively. An email address isn't a string — it's a value object with validation baked in. A money amount isn't a float — it's a value object with currency, precision, and arithmetic rules. Let the type system do work your service layer shouldn't be doing.

Aggregates

An aggregate is a cluster of entities and value objects that are treated as a single unit for data changes.

Every aggregate has a root entity — the gateway through which all external access happens. You don't reach directly into a child entity from outside the aggregate. You go through the root. The root enforces the business rules that maintain consistency across the whole cluster.

The critical property of an aggregate: it is a transactional boundary. When you update an aggregate, the entire unit updates atomically. Either all of it changes, or none of it does. This is how you prevent partial state — the silent killer of data integrity.

One important constraint: keep aggregates small. An aggregate that encapsulates too many business rules becomes expensive to load and update. If every operation on your Order aggregate requires loading its full history of 500 line items, you have a design problem. When aggregates become performance bottlenecks, split them.

Domain Events

When something meaningful happens inside a bounded context — an order is placed, a payment fails, a drone completes its delivery — that's a domain event.

Domain events are how bounded contexts communicate without coupling. The payment context doesn't call the notification context directly. It publishes a PaymentConfirmed event. The notification context subscribes to it and does whatever it needs to do.

This keeps your contexts genuinely independent. The payment context doesn't care what happens after confirmation. It just records that it happened.

Repositories

A repository is the abstraction between your domain logic and your persistence layer.

Your OrderService should not contain SQL. It should call orderRepository.findById(id) and get back a domain object — an aggregate root, fully loaded and ready to work with. The repository handles the mechanics of talking to whatever database you're using.

This matters for the same reason bounded contexts matter: it's about boundaries. The domain logic should not know or care that you're using PostgreSQL today and might migrate to something else in two years. The repository is where that translation lives.

Anemic Models vs. Rich Models

This is where a lot of teams go wrong after learning the vocabulary.

An anemic model is a class full of getters and setters with no logic. It's a data container dressed up as a domain object. All the business logic lives in services that manipulate these containers from the outside.

A rich model is an entity that owns its logic and validation. The Drone entity doesn't just store battery level — it exposes a method that enforces the business rule about minimum charge before dispatch. The validation isn't scattered across five different services hoping they all remember to check. It's in the entity. It's always enforced.

Anemic models are seductive because they look simple. They feel like they're keeping things clean. They're not. They're moving complexity from where it belongs (the domain object) into services where it becomes redundant, inconsistent, and invisible to the type system.

// Anemic — business logic leaked into service
class DroneService {
  dispatch(drone) {
    if (drone.batteryLevel < 20) throw new Error("Battery too low");
    if (!drone.maintenanceCleared) throw new Error("Not cleared");
    drone.status = "DISPATCHED";
  }
}

// Rich — domain object owns its rules
class Drone {
  dispatch() {
    if (this.batteryLevel < 20) throw new DomainError("Battery too low");
    if (!this.maintenanceCleared) throw new DomainError("Not cleared");
    this.status = DroneStatus.DISPATCHED;
    this.addEvent(new DroneDispatched(this.id));
  }
}

The difference isn't just aesthetic. In the anemic version, nothing stops another service from setting drone.status = "DISPATCHED" without running any validation. In the rich version, that's structurally impossible. The invariants are enforced by design.

When DDD Is Worth It (And When It Isn't)

DDD has a real cost. Event storming takes time. Bounded contexts add integration work. Rich models require more upfront design thinking.

For a simple CRUD application — a form that saves to a database with minimal business logic — DDD is overkill. The ceremony will cost more than the complexity it manages.

But for a system where the business logic is the hard part? Where the same word means five things to five different stakeholders? Where changes in one area keep breaking things in unexpected places?

That's exactly the problem DDD was built to solve.

The irony is that most teams who would benefit most from DDD adopt it latest — because they're too deep in the mess to step back and redesign. The teams who need it most are the ones who've been "moving fast" for three years and are now buried under a codebase that nobody fully understands.

The best time to adopt DDD thinking was at the start. The second best time is now.

The One Thing to Take Away

If you implement nothing else from this article, implement ubiquitous language.

Before your next sprint, get your engineers and your domain experts in the same conversation and align on terminology. Pick the words. Write them down. Name your classes those words. Name your tables those words. Name your API endpoints those words.

The translation tax you're currently paying — in miscommunications, in onboarding friction, in bugs born from misunderstood requirements — is silent and constant.

Stop paying it.

DDD has a deep literature if you want to go further. Eric Evans' "Domain-Driven Design" is the original text. Vaughn Vernon's "Implementing Domain-Driven Design" is the practitioner's companion. Start with the vocabulary. Then the bounded contexts. The rest follows.

Your Microservices Are Not Resilient. Your Architecture Is the Real Problem

Olawale Afuye — Fri, 05 Jun 2026 14:39:50 +0000

Most teams building microservices are one bad deployment away from a full system meltdown.

Not because their engineers are bad.

Not because they picked the wrong cloud provider.

Because they built a distributed monolith — and dressed it up like a real microservices architecture.

I've watched brilliant teams do this. Long chains of synchronous REST calls. No timeouts. No circuit breakers. No queue monitoring. Everything holding hands in production, pretending that's fine.

It isn't fine.

Here's the full breakdown of what resilience actually requires — and why most teams skip the parts that matter most.

1. Stop Building Distributed Monoliths

Here's the thing nobody wants to say out loud: most "microservices" architectures are just monoliths with extra network hops.

You have Service A calling Service B, which calls Service C, which calls Service D. All synchronously. All blocking. All waiting on each other like a Lagos traffic queue in the rain.

The moment Service C hiccups, Service B hangs. Service A times out. Your user sees a spinner. Your Slack blows up.

That's not microservices. That's a monolith wearing a Halloween costume.

The actual problem: Cascading failures. One slow service accumulates failures upstream, consuming threads and connections until the whole system chokes.

The fix: Stop treating synchronous REST chains as a default. Question every service-to-service call. Ask whether it has to be synchronous or whether it's just convenient.

2. The Bulkhead Pattern: Give Failures Nowhere to Go

On a ship, bulkheads are watertight compartments. One hull breach doesn't sink the whole ship — it sinks one section. The rest stays afloat.

Your services need the same thing.

The bulkhead pattern isolates components so that failure in one part of your system cannot cascade into everything else. Concretely, this means:

Separate thread pools for separate service calls
Dedicated connection pools per downstream dependency
Hard resource limits per consumer group

If your payment service and your recommendation engine share the same thread pool, a recommendation spike can starve your payments. That's not an edge case. That's a design flaw.

Implementation rule: Define your boundaries. Enforce them. A failure in your notification service should never be able to reach your checkout flow.

3. Timeouts Are Not Optional

This one is embarrassingly basic. And yet.

Every blocking call in a distributed system must have a timeout. Every single one.

Without timeouts, a slow downstream service doesn't just slow your service down — it holds your threads open indefinitely. Enough of those and you've got resource exhaustion. Enough resource exhaustion and you've got an outage.

Default timeouts in most HTTP clients are either disabled or set to something absurd like 30 seconds. In a distributed system, 30 seconds of waiting is an eternity.

The rule: Set timeouts that are aggressive enough to protect you, but generous enough not to fail healthy traffic. Start with your 99th percentile response time, add a margin, and set that as your ceiling.

One more thing: if your operation is not idempotent, think carefully before adding retries. Retrying a payment without idempotency checks is how you charge a customer twice and earn a very unhappy support ticket.

4. Circuit Breakers: Fail Fast, Recover Clean

Here's the intuition: if a service is already down, why are you still sending it traffic?

A circuit breaker monitors the failure rate and timeout frequency of calls to a downstream dependency. Once failures cross a defined threshold, it opens — and stops sending requests entirely. No more piling onto a service that's already struggling.

After a cool-down window, it moves to a half-open state. A few test requests go through. If they succeed, the circuit closes and normal traffic resumes. If they fail, it opens again.

Three states. Simple logic. Massive resilience benefit.

CLOSED → normal traffic flows
    ↓ (failures exceed threshold)
OPEN → requests fail fast, no traffic sent
    ↓ (after cool-down)
HALF-OPEN → test requests sent
    ↓ (success)
CLOSED again

The implementation exists in every major language. Resilience4j for Java. Polly for .NET. opossum for Node. There's no reason to roll your own.

5. Throttling: Protect Your Critical Flows

Not all traffic is equal. A user refreshing their dashboard feed is not as important as a user completing a payment.

Throttling means imposing artificial load limits to protect the flows that actually matter. If a background analytics job is hammering your database and slowing down your checkout API, something has gone badly wrong in your prioritization logic.

Practical approach:

Define your critical business flows
Assign them dedicated capacity
Rate-limit everything else before it touches that capacity

Bounded queues are your friend here. A queue with no upper bound will accept traffic until your system collapses. A bounded queue with a sane limit will reject or backpressure early, giving you a chance to recover before everything explodes.

6. Go Asynchronous. Seriously.

The real fix for long synchronous call chains is to stop making them.

Messaging infrastructure — Kafka, RabbitMQ, SQS — decouples your services temporally. Service A publishes an event and moves on. It doesn't care when Service B processes it or whether Service C is currently up.

This eliminates a whole class of resilience problems:

No cascading timeouts from downstream slowness
No resource exhaustion from blocked threads
Natural load levelling during traffic spikes

The mental model shift is real. You lose the clean request-response stack trace you're used to. Debugging across asynchronous flows requires distributed tracing — and that brings us to correlation IDs.

Always attach a correlation ID to every event. When a transaction touches five services across three queues, that ID is the only thing that lets you reconstruct what happened. Without it, you facethe pain of debugging and you're reading logs in the dark.

And watch your queues. Seriously. A growing queue waiting time is one of the earliest signals that something downstream is struggling. Most teams don't monitor this until it's too late.

7. Embrace Eventual Consistency (Or Suffer the Alternative)

The monolith gave you something seductive: strict consistency. One transaction, one database, one source of truth.

Microservices take that away. You now have multiple services with their own data stores. Forcing strict consistency across them creates tight coupling, distributed transactions, and the kind of complexity that ages engineers prematurely.

Eventual consistency is the trade. You accept that different parts of your system may be temporarily out of sync — and you design for it. Your inventory service might briefly show a product as available while it's being purchased. Your notification service might send an email seconds after the transaction completes, not simultaneously.

For most business domains, this is fine. Genuinely fine. The obsession with real-time consistency is often a reflex from monolith thinking, not an actual business requirement.

Identify your truly consistency-critical flows. Design strict guarantees only where they're mandatory. Everywhere else, let eventual consistency do its job.

8. Event Sourcing: When History Becomes Infrastructure

Standard CRUD stores state. Event sourcing stores what happened.

Every change is an immutable event appended to an event log. The current state is derived by replaying those events. This gives you:

Full audit trail — you know exactly what happened and when
Point-in-time reconstruction — replay to any moment in history
Scalability — pair with CQRS to separate reads and writes, deploy multiple consumers

The complexity cost is real. Event versioning, out-of-order message handling, schema evolution — none of this is free. Don't reach for event sourcing for a simple CRUD service. Do reach for it when audit history, temporal queries, or complex state transitions are core requirements.

9. The Robustness Principle: Be Strict in What You Send, Tolerant in What You Accept

Postel's Law. Often cited. Rarely implemented.

When you're producing data for other services, be strict. Follow your contract. Don't add unexpected fields, don't change types, don't break your schema.

When you're consuming data from other services, be tolerant. If you only need two fields from a 20-field response, don't fail the request because one of the other 18 is missing. You didn't need it. Don't act like you did.

Over-validation of incoming data is a quiet source of fragility. A downstream service makes a minor additive change — adds a new optional field — and suddenly your service is throwing 500s because your schema validator rejects it.

Validate what you actually depend on. Ignore what you don't.

10. Observability Is Not Optional. It's How You Know Anything.

Here's the uncomfortable truth: most teams don't know their system is degraded until a user complains.

That's too late.

Real observability means:

Health checks that actually tell you something. A liveness check that just returns 200 OK is nearly useless. A readiness check that reports whether your payment gateway is reachable, your database is responsive, and your critical dependencies are up — that's useful.

Distributed tracing. Zipkin, Jaeger, Honeycomb. When a request touches six services, you need to see the entire timeline, with durations, to find the bottleneck. Without tracing, you're guessing.

Metrics with alerts. Response time degradation, error rate spikes, queue depth growth — these need thresholds and automated alerts, not manual dashboard checking.

DevOps ownership. If the team that writes the code doesn't own the production health of that code, nobody does. The siloed model where developers throw services over the wall and operations catches whatever breaks — that model is where resilience patterns go to die.

The Real Problem Is Culture, Not Code

Every pattern in this list has a library. Most have battle-tested implementations in your language of choice.

The reason teams skip them isn't technical ignorance. It's deadline pressure, underestimating distributed system complexity, and the slow-creep assumption that "it'll probably be fine."

It will not be fine.

Resilience is a feature. It deserves to be designed, implemented, tested with tools like Toxiproxy (which simulates network failures so you can validate your assumptions before production does it for you), and monitored in perpetuity.

Your users don't care that you had a network partition. They care that it worked anyway.

Build for that.

Quick-Reference: Resilience Pattern Checklist

[ ] No long synchronous REST call chains in critical paths
[ ] Bulkheads isolate failure domains — separate thread/connection pools
[ ] Every blocking call has an explicit timeout configured
[ ] Retries only on idempotent operations
[ ] Circuit breakers on all external dependencies
[ ] Throttling protects critical business flows from lower-priority traffic
[ ] Bounded queues prevent unbounded load accumulation
[ ] Asynchronous messaging used where synchronous coupling isn't required
[ ] Correlation IDs attached to all events and async flows
[ ] Queue waiting time and depth monitored with alerts
[ ] Eventual consistency embraced where strict consistency isn't a real requirement
[ ] Readiness health checks report dependency status, not just liveness
[ ] Distributed tracing deployed and covering service boundaries
[ ] Team owns production health of their services end-to-end

Building in distributed systems? The patterns here apply whether you're running three services or three hundred. Start with timeouts and circuit breakers — they're the fastest wins. Then work backwards from your most critical user flows and ask: what happens when each dependency here fails?

The answer will tell you exactly where to go next.

Your Security Scanner Found 7 Missing Headers. Don't Fix Them Blindly.

Olawale Afuye — Fri, 05 Jun 2026 09:04:02 +0000

Your security scanner just came back with 6 flagged items.

All missing HTTP headers.

You did what any reasonable developer does: Googled each one, copy-pasted the recommended config, and shipped a fix in 20 minutes. Job done. Security score green. PR merged.

You also probably shipped at least two of them wrong.

Here is the thing nobody tells you about HTTP security headers: knowing what to add is the easy part. Understanding why it matters, when it actually doesn't, and how a misconfigured one breaks your app in production — that's where most developers fall short.

This isn't another "add these 7 headers to secure your app" post.

This is the one that explains what's actually happening.

First, The Contrarian Take

Missing a security header is not automatically a vulnerability.

If you do bug bounties, this will save you a rejection. If you're a dev, it'll save you from cargo-culting configs that don't apply to your app.

Context is king.

X-Frame-Options: DENY is a valid security header. YouTube doesn't use it. Because the entire point of YouTube is for people to embed its videos in iframes. Applying that header would break a core product feature. That's not a security oversight — it's a deliberate design decision.

A missing Content-Security-Policy header is not a vulnerability in itself. It only becomes relevant if you already have an XSS problem to mitigate. CSP is defense-in-depth. Not a fix for a broken input sanitisation layer.

This matters because a lot of developers (and worse, automated scanners) treat these headers like a binary checklist. Present = secure. Missing = vulnerable.

Reality is messier than that.

Now — with that said — let's talk about what each one actually does.

#1. HTTP Strict Transport Security (HSTS)

Most developers think HSTS is just "force HTTPS." It's more precise than that.

When your app redirects http:// to https://, that first request is still unencrypted. For a fraction of a second, on a public network, that window exists. An attacker on that network can intercept the request, redirect the user, inject content — before HTTPS ever kicks in.

HSTS closes that window.

Once a browser receives the HSTS header, it never sends an unencrypted request to that domain again. No matter what. Even if the user types http://.

Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

Breaking down the directives:

max-age=31536000 — How long the browser enforces this. One year. Don't go lower.
includeSubDomains — Extend the policy to all subdomains. Only add this if you're certain every subdomain serves HTTPS.
preload — Submit your domain to be hardcoded into browsers. First-time visitors are covered before they even make a request. Submit here.

The catch: Once you set this, you're committed. If your cert expires or your HTTPS breaks, users can't access your site. Test in staging. Be sure.

#2. Content Security Policy (CSP)

The most powerful header on this list.

Also the most commonly misconfigured one.

CSP is a whitelist. You tell the browser: "Only execute scripts from these sources. Only load styles from here. Block everything else." This makes XSS attacks dramatically harder — even if an attacker injects a script tag, if the source isn't on your whitelist, the browser refuses to run it.

Content-Security-Policy: default-src 'self'; script-src 'self' https://cdn.trusted.com; style-src 'self'

The directives that matter most:

default-src — The fallback for any content type not explicitly specified. Start here.
script-src — Controls JavaScript. The most important directive for XSS mitigation.
style-src — Controls stylesheets.
img-src — Controls image sources.
connect-src — Controls where your JS can make network requests (fetch, XHR, WebSocket).
frame-ancestors — Controls who can embed your page in an iframe. This replaces X-Frame-Options.

The values that will hurt you:

unsafe-inline — Allows inline scripts. Defeats a large portion of XSS protection. Avoid it.

unsafe-eval — Allows eval(). Avoid this too.

default-src * — Allows everything. Worse than no CSP. Gives a false sense of security.

How to implement without breaking your app:

Don't go straight to enforcement. Use report-only mode first.

Content-Security-Policy-Report-Only: default-src 'self'; report-uri /csp-violations

This logs all violations to the browser console (and to your endpoint) without blocking anything. Watch it for a week. Fix the violations. Then flip to enforcement.

This is the only sane way to add CSP to an existing app.

#3. X-Content-Type-Options

One directive. One line. No excuses for not having this.

X-Content-Type-Options: nosniff

The problem it solves: browsers try to be smart. They "sniff" content — sometimes overriding the declared MIME type of a file if the content looks like something else. A .txt file that contains JavaScript? Some browsers might decide to execute it.

nosniff tells the browser: trust the declared Content-Type. Don't guess. Don't override.

This one is a no-brainer. Add it everywhere.

#4. X-Frame-Options

Controls whether your page can be loaded in an <iframe>.

The attack it prevents is clickjacking — where an attacker loads your app invisibly inside an iframe on their site and tricks users into clicking things they didn't intend to.

X-Frame-Options: SAMEORIGIN

Three options:

DENY — Nobody can embed you. Not even yourself.
SAMEORIGIN — Only your own domain can embed you.
ALLOW-FROM uri — Specific URI only. Note: not supported in some modern browsers.

Important context: This header has technically been superseded by the CSP frame-ancestors directive. If you're adding CSP, use frame-ancestors instead. But if you're not yet using CSP, X-Frame-Options still works fine.

And as mentioned earlier — if your app intentionally allows embedding (a widget, a video player, a public component), think twice before slapping DENY on it.

#5. Cache-Control

This one lives at the intersection of performance and security.

Most developers think about Cache-Control as a performance header. It is. But it's also a security concern.

The scenario: A user logs into your banking app on a shared computer. They log out. The next person opens the browser, hits the back button — and sees the previous user's account page, served from cache.

That's a real attack surface.

Cache-Control: no-store, no-cache, must-revalidate

The directives:

no-store — Don't cache this response anywhere. Not the browser. Not intermediate proxies.
no-cache — Cache it, but revalidate with the server before using it.
must-revalidate — Expired cache must be revalidated before serving.
private — Only the user's browser can cache this. Not shared proxies.
public — Shared caches (CDNs, proxies) can cache this.

For sensitive pages: no-store, no-cache, must-revalidate. No exceptions.

For static assets (JS, CSS, images): public, max-age=31536000, immutable with versioned filenames. Cache aggressively, but tie it to a content hash so updates invalidate the cache automatically.

These are two completely different strategies. Don't apply one to everything.

#6. CORS (Access-Control-Allow-Origin)

The most misunderstood header in web development.

Let's be precise about what CORS actually is.

By default, browsers block scripts on one domain from making requests to a different domain. This is the Same-Origin Policy. It's a browser security feature.

CORS is how servers say: "It's okay, I trust this origin. Let them through."

Access-Control-Allow-Origin: https://app.yourdomain.com

The wildcard problem:

Access-Control-Allow-Origin: *

This allows any website to make requests to your server. For truly public, read-only, non-authenticated resources — that's fine. A public weather API. A CDN. An open dataset.

For anything that involves session cookies, authentication tokens, or user-specific data — this is dangerous. A malicious site can make requests to your API as the logged-in user.

The rule: Use * only when you're absolutely certain no sensitive data is involved. For everything else, whitelist specific origins explicitly.

And CORS misconfiguration only becomes a real vulnerability when it allows authenticated requests — meaning an attacker can leverage the user's session to make requests on their behalf. That's the bar. Without that, it's a low-severity issue at best.

#7. Permissions-Policy

The one almost nobody adds.

Modern browsers give websites access to powerful hardware features: camera, microphone, geolocation, payment APIs, gyroscope, and more. By default, a page (or any third-party script running on it) can request access to these.

Permissions-Policy lets you define exactly which features your app actually needs — and lock out everything else.

Permissions-Policy: camera=(), microphone=(), geolocation=(self), payment=()

() means nobody gets access. Not your code, not third-party scripts.

(self) means only your own origin can request it.

(self "https://trusted-partner.com") extends it to a specific third party.

This is particularly important if you load third-party widgets, analytics scripts, or chat SDKs. You don't always know what those scripts are doing. This header puts a hard boundary on what they can access.

The Referrer-Policy Bonus

Quick one.

Every time a user clicks a link to another site, the browser sends a Referer header with the URL they came from. If your URL contains a session token, a user ID, or any sensitive query parameter — that data just leaked to a third party.

Referrer-Policy: strict-origin-when-cross-origin

This sends the origin (domain) for cross-origin requests but drops the path and query string. Enough for analytics. Not enough to leak tokens.

no-referrer is the most restrictive — no referrer data sent at all. Use this if your URLs can contain sensitive information.

How to Actually Ship These

For Nginx:

server {
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header Permissions-Policy "camera=(), microphone=(), geolocation=(self)" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self'; img-src 'self' data:; connect-src 'self'" always;
}

For Apache:

<IfModule mod_headers.c>
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    Header always set X-Content-Type-Options "nosniff"
    Header always set X-Frame-Options "SAMEORIGIN"
    Header always set Referrer-Policy "strict-origin-when-cross-origin"
    Header always set Permissions-Policy "camera=(), microphone=(), geolocation=(self)"
    Header always set Content-Security-Policy "default-src 'self';"
</IfModule>

For Node.js/Express — use Helmet:

npm install helmet

import helmet from 'helmet';

app.use(helmet()); // Sane defaults for most headers
app.use(helmet.contentSecurityPolicy({
  directives: {
    defaultSrc: ["'self'"],
    scriptSrc: ["'self'", "https://cdn.trusted.com"],
    styleSrc: ["'self'"],
  }
}));

Helmet handles HSTS, X-Content-Type-Options, X-Frame-Options, and several others out of the box. You still need to configure CSP manually for your specific app.

The Summary That Actually Matters

Here's the real order of priority:

Add immediately, no excuses:

X-Content-Type-Options: nosniff — one line, zero risk.

Strict-Transport-Security — if you're on HTTPS (you should be).

Referrer-Policy — lightweight, meaningful privacy win.

Add with context:

X-Frame-Options or CSP frame-ancestors — depends on whether your app is embeddable.

Cache-Control — depends on what kind of resource you're serving.

Permissions-Policy — depends on what third-party scripts you're loading.

Add carefully, test first:

Content-Security-Policy — use report-only mode. Understand your app's resource tree first.

CORS — understand your authentication model before you configure this.

The rule that overrides everything:

A security header you don't understand is more dangerous than a missing one.

default-src * is worse than no CSP.

A misconfigured Cache-Control that serves sensitive pages publicly is worse than no caching directive.

Wildcard CORS on an authenticated API is worse than missing CORS headers entirely.

Understand what you ship.

Security isn't a scanner score.

It's a set of deliberate decisions made by someone who understands what their system actually does.

What header did you add wrong before you knew better? Drop it in the comments.

The Gateway Layer Explained: Reverse Proxies, Load Balancers, and API Gateways — Once and For All

Olawale Afuye — Thu, 04 Jun 2026 22:56:03 +0000

Most backend engineers have used all three. Few can explain the difference without stuttering.

And honestly? That's not entirely their fault. The tools themselves blur the lines — Nginx can be a proxy, a balancer, and a gateway simultaneously. Kong does all three. So does Envoy. Cloudflare will happily do all of that plus serve your coffee.

But the confusion isn't just about tooling. It's about not having a mental model for why these three things exist as separate concepts. Once you have that model, the tools stop being confusing and start being obvious.

This post gives you that model — clearly, technically, and without the hand-waving.

The Gateway Layer: What It Actually Is

Before your requests touch a single line of application code, they pass through a layer of infrastructure whose entire job is to control, inspect, protect, and route traffic. That's the Gateway Layer.

It sits in front of your backend. It is not your backend. And it has three distinct personas, each solving a different class of problem:

Component	Core Problem It Solves
Reverse Proxy	Single-server protection and optimization
Load Balancer	Distributing load across multiple servers
API Gateway	Managing API complexity in microservices

Think of it as a spectrum, not three separate boxes. As your system grows, you add layers. Let's walk through each one.

1. Reverse Proxy — The First Line of Defense

What it is

A reverse proxy sits in front of your origin server and intercepts all incoming requests before they ever reach your application. From the client's perspective, they're talking to one server. In reality, they're talking to a proxy.

Client → Reverse Proxy → Origin Server

What it actually does

SSL/TLS Termination

This is probably the most important thing a reverse proxy does, and the most underappreciated.

When the proxy terminates TLS, it handles the entire encrypted session — certificate validation, key exchange, cipher negotiation — and then forwards requests to your backend over plain HTTP on a secure private network. Your backend never has to manage certificates or touch cryptography.

The primary benefit today is operational, not computational. TLS 1.3 and hardware-accelerated AES-NI have significantly reduced the CPU cost of cryptographic work compared to earlier TLS versions. But the centralization benefit remains compelling at any scale: one certificate to manage, one place to renew, one place to enforce cipher policies and TLS version requirements. Rotate a cert in one place and every backend is covered. Enforce TLS 1.3-only in one config and every service inherits it.

CPU offloading is still relevant under very high connection volumes, but if someone asks you why you're doing SSL termination, "centralized certificate management and policy enforcement" is the honest first answer in 2025.

Caching

Your backend doesn't need to re-compute the same response 10,000 times. The proxy can cache common responses and serve them directly, dramatically reducing the load on your origin server. This is especially valuable for read-heavy APIs serving mostly static or semi-static data.

Compression

The proxy can gzip or brotli-compress response payloads before sending them to clients, reducing bandwidth consumption without touching your application code.

IP Obfuscation and Traffic Filtering

The proxy hides your server's real IP address from the public internet. Clients only ever see the proxy's IP. Beyond that, it can filter malicious traffic — blocking known bad actors, rejecting suspicious patterns, and acting as a basic security checkpoint before anything touches your application.

When is a reverse proxy sufficient?

If you're running a single server, or a small application that doesn't need horizontal scaling yet, a reverse proxy is all you need. It handles SSL, adds basic security, reduces load through caching, and insulates your origin server from the internet.

Once you need multiple servers, you graduate to the next layer.

2. Load Balancer — Scaling Horizontal

What it is

Before diving in: a load balancer and a reverse proxy are not the same abstraction, even though they're often conflated.

A reverse proxy is a traffic pattern — it mediates and hides access to backend servers. A load balancer is a function — it distributes traffic across multiple instances. These overlap in HTTP-based systems (most L7 load balancers are implemented as reverse proxies), but they're not equivalent.

AWS NLB, for example, is a load balancer that operates at L4 — it doesn't behave like a reverse proxy at all. HAProxy and Nginx do both. The distinction matters once you're choosing infrastructure rather than just reading about it.

With that said: the core problem a load balancer solves is how do you distribute traffic intelligently across multiple backend instances?

Client → Load Balancer → [Server 1]
                       → [Server 2]
                       → [Server 3]

Traffic Distribution Algorithms

Round Robin
Requests are sent to each server in sequence — 1, 2, 3, 1, 2, 3. Dead simple, works well when all your servers have similar specs and roughly equal request complexity.

Least Connections
Instead of cycling mechanically, the balancer sends each new request to whichever server currently has the fewest active connections. Smarter than round robin when your requests have variable processing time — a slow query shouldn't pile up on an already-struggling server.

Weighted Distribution
Not all servers are equal. If you have a 16-core machine and an 8-core machine in the same pool, you don't want equal traffic. Weights let you say "this server can handle twice the load — give it twice the requests." Useful when running mixed-hardware clusters or during gradual capacity upgrades.

IP Hashing
Uses the client's IP address to deterministically route them to the same backend server every time. This is useful for session affinity — cases where a user's session is stored locally on a server (WebSocket connections, in-memory caches, etc.).

Worth noting: modern systems generally prefer stateless architectures, where session state lives in an external store like Redis. That way, any server can handle any request. IP hashing is a workaround for systems that haven't made that transition yet.

Health Checks and Failover

This is where load balancers earn their keep in production.

A load balancer continuously sends health check probes to each server in the pool. If a server fails to respond — whether from a crash, an OOM kill, a deployment mishap, or a runaway process — the balancer automatically removes it from rotation. Requests stop being sent to the dead instance. Users don't see the failure.

When the server recovers, health checks pass again, and it's automatically re-added to the pool.

This is the foundational mechanism behind high availability. No manual intervention required, no on-call engineer frantically removing servers from a config file at 2am.

Layer 4 vs Layer 7: Choosing the Right Level

Load balancers can operate at two different network layers, and the choice has real performance implications.

Layer 4 (Transport Layer)

L4 balancers work at the TCP/UDP level. They route traffic based purely on IP addresses and ports — they never look inside the packet. This makes them extremely fast. Low overhead, minimal CPU usage, capable of handling millions of connections per second.

Use L4 when:

You need raw throughput at the edge
You're routing non-HTTP traffic (databases, game servers, VoIP, IoT protocols)
You want SSL pass-through (the backend owns the cert end-to-end, required for some compliance scenarios)
You're fronting a fleet of L7 balancers in a tiered architecture

Layer 7 (Application Layer)

L7 balancers understand HTTP. They can inspect headers, URL paths, query params, and even body content. This unlocks intelligent, content-aware routing:

Route /api/v2/* to a new cluster, /api/v1/* to legacy services
Send traffic from mobile clients to a specific backend tier
Route based on custom headers, cookies, or authenticated user identity

The trade-off is overhead — inspecting packet content is more expensive than routing by IP. But for most web applications, L7 is the right default.

Common production pattern: Put an L4 balancer at the very edge to absorb raw traffic volume, then distribute it across a fleet of L7 balancers that handle content-aware routing. You get the throughput of L4 with the intelligence of L7.

3. API Gateway — Taming Microservices

The Problem It Solves

Here's what happens when you split a monolith into microservices:

You go from one codebase to twelve. Each team owns their service. Fast, independent, scalable — the dream.

Then reality arrives.

Every service needs authentication. Every service needs rate limiting. Every service needs logging. Every service needs request validation. Every team implements these things slightly differently, with different libraries, different error formats, different token validation logic.

Six months later, your "User Service" has JWT validation on version 3.1 of a library, your "Order Service" is on 2.8, and your "Notification Service" has a subtly wrong implementation that passed code review because the reviewer was rushing a deadline.

This is logic drift. And it's one of the most expensive problems in distributed systems — not because it breaks things immediately, but because it breaks things inconsistently, and inconsistent failures are the hardest to debug.

An API Gateway is the solution. It's a single entry point that handles all infrastructure concerns before any request touches a microservice.

Client → API Gateway → User Service
                     → Order Service
                     → Payment Service
                     → Notification Service

What an API Gateway Does

Centralized Authentication and Authorization

The gateway validates OAuth2 tokens, JWTs, or API keys in one place. Invalid requests are rejected at the edge. Your microservices never see unauthenticated traffic. More importantly, they don't need to implement authentication — that's no longer their problem.

Rate Limiting

Define throttling rules once, enforce them everywhere. No per-service implementation, no inconsistent limits, no team accidentally shipping a service without rate limiting because it "wasn't in scope this sprint."

Request and Response Transformation

The gateway can translate between formats. Your legacy internal services speak XML? Your mobile clients send JSON? The gateway handles the translation. Services don't need to know how their clients represent data — they work in their native format and the gateway handles the conversion.

API Versioning and Traffic Routing

Route /v1/users to your stable legacy service, /v2/users to the new one being rolled out. Migrate traffic incrementally. Kill the old version when you're confident. Do all of this in one config, not scattered across twelve service codebases.

Unified Observability

Every request flows through the gateway. That means one centralized source of metrics, logs, and traces. When an incident happens, you don't have to correlate logs from twelve different services to understand what's failing. The gateway tells you.

Request Aggregation and Backend for Frontend (BFF)

This one gets skipped in most explainers, but it's a core gateway capability.

Consider a mobile dashboard that needs to display user info, recent orders, recommendations, and notifications — all in one screen load. Without a gateway, the mobile client makes four separate API calls, waits for four responses, and assembles the data itself. Over a mobile network, that round-trip cost compounds.

With an API Gateway acting as a BFF (Backend for Frontend), the client makes a single call to /dashboard. The gateway fans out to the relevant services in parallel, aggregates the responses, and returns one unified payload:

Client → GET /dashboard → API Gateway → User Service
                                       → Order Service
                                       → Recommendation Service
                                       → Notification Service
                              ↓
                     Single combined response → Client

This keeps clients thin, reduces network chattiness, and lets backend services remain focused on their own domains instead of knowing what every client type needs.

What an API Gateway Costs You

Most articles about API Gateways read like product marketing. Let's be honest about the tradeoffs.

Single point of failure. If the gateway goes down, every service goes down with it. A misconfigured routing rule, a bad deployment, a gateway-level memory leak — any of these can take your entire platform offline. This makes gateway reliability a first-class engineering concern, not an afterthought. High availability, blue-green deployments, and exhaustive config validation are not optional.

Added latency. Every request gets an extra network hop. Under normal conditions, this is negligible — a well-configured gateway adds low single-digit milliseconds. Under high load or with a misconfigured gateway doing expensive work (complex transformations, slow auth lookups), that hop starts to matter.

Team bottlenecks. In practice, the team that owns the gateway config becomes a bottleneck. New service? You need a gateway route. New auth policy? Gateway team. Rate limit change? Gateway team. This can be mitigated with good self-service tooling and declarative config, but it's a real organizational friction point.

Configuration complexity at scale. A Kong or Apigee installation managing hundreds of routes, policies, plugins, and environments can become a product unto itself. It needs versioning, testing, staging, and on-call ownership. Factor this into your operational cost estimates.

None of these are reasons to avoid API Gateways — they're reasons to operate them deliberately.

Why the Lines Blur (And Why That's Fine)

Here's the honest answer to "why does everyone get confused about this?"

Because Nginx does all three. Kong does all three. Envoy does all three. AWS API Gateway, Traefik, Caddy, HAProxy — they all blur the lines.

You can configure Nginx as a dumb reverse proxy in ten lines. You can configure it as a full L7 load balancer. You can add Lua plugins and make it behave like an API gateway. Same binary, radically different behavior.

The important mental shift: stop thinking about these as three separate products. Think of them as a spectrum of capabilities.

The question is never "which one do I use?" The question is: "which capability do I need for the problem I'm solving right now?"

Need SSL offloading and basic security? Enable proxy capabilities.
Need to distribute load across replicas? Enable balancing capabilities.
Need centralized auth and rate limiting for microservices? Enable gateway capabilities.

Sometimes you need all three from the same tool. Sometimes you need dedicated tools at each layer. Let your architecture's requirements drive the decision, not vendor marketing.

Here's a quick reference for how common tools map to capabilities:

Tool	Reverse Proxy	Load Balancer	API Gateway
Nginx	✅	✅	Limited
HAProxy	✅	✅	Limited
Envoy	✅	✅	✅
Kong	✅	✅	✅
Traefik	✅	✅	✅
Cloudflare	✅	✅	Partial
AWS ALB	Partial	✅	Partial
AWS NLB	❌	✅	❌
AWS API Gateway	❌	Partial	✅

AWS NLB is worth noting specifically — it's a load balancer that operates at L4 and does not behave like a reverse proxy, which is why the function/pattern distinction matters in practice.

What a Modern Production Architecture Actually Looks Like

When systems reach scale, these components don't replace each other — they layer.

Here's one common pattern:

Internet
    ↓
[CDN] — Static assets, edge caching, initial SSL termination, DDoS absorption
    ↓
[API Gateway] — Authentication, rate limiting, routing, observability
    ↓
[Load Balancer] — Traffic distribution across service clusters
    ↓
[Internal Proxies] — Service-to-service communication
    ↓
[Microservices]

That said — this is one common architecture, not the canonical one. In practice, teams arrive at different layering depending on their infrastructure choices. Other valid production patterns include:

# Kubernetes-native
Internet → CDN/WAF → Cloud Load Balancer → Ingress Controller → Services

# API Gateway at the edge
Internet → API Gateway (Envoy/Kong) → Kubernetes Ingress → Pods

# Cloudflare-heavy
Cloudflare → Envoy Gateway → Service Mesh (Istio) → Pods

The principles are consistent across all of them — edge, entry point, distribution, internal communication. The specific tools filling those roles vary.

If You Work in Kubernetes

These concepts don't disappear in Kubernetes — they get mapped to Kubernetes-specific primitives:

Gateway Layer Concept	Kubernetes Equivalent
Reverse Proxy	Ingress Controller (Nginx Ingress, Traefik)
Load Balancer	Service (type: LoadBalancer) / Cloud LB
API Gateway	Gateway API / Kong Ingress / Ambassador
Internal Proxy	Envoy Sidecar (in service mesh setups)
Service Mesh	Istio / Linkerd / Cilium

When you configure an Nginx Ingress rule, you're configuring reverse proxy behavior. When you define a Kubernetes Service, you're setting up internal load balancing. The vocabulary changes; the underlying concepts don't.

Each layer has a specific job:

CDN: Handles static content and absorbs high-volume traffic before it hits your origin. Think Cloudflare, Fastly, CloudFront.
API Gateway: The main entry point for dynamic requests. Handles auth and routes to the right service.
Load Balancer: Distributes traffic within a service cluster, manages health checks and failover.
Internal Proxies / Service Mesh: Manages service-to-service communication at scale with mTLS, circuit breaking, and retries.

You don't need all of this on day one. A startup running a single service needs a reverse proxy, not a service mesh. But knowing this map means you understand where you're headed as you scale, and you can design with that future in mind instead of painting yourself into corners.

Security Considerations Across Every Layer

The Gateway Layer is also your primary security perimeter. Here's where each component contributes:

Reverse Proxy

Hides internal server IPs and architecture from the public internet
Enforces HTTPS via SSL termination; manages certificate rotation centrally
Can integrate a Web Application Firewall (WAF) to block SQLi, XSS, and known attack patterns

Load Balancer

Health checks remove compromised or unresponsive instances automatically
Can enforce IP allowlisting or geo-blocking at the traffic layer
L4 pass-through mode preserves end-to-end TLS for compliance requirements

API Gateway

Validates JWT/OAuth tokens, rejects unauthenticated traffic at the edge
Enforces rate limits, preventing brute force and credential stuffing attacks
Validates request payloads against OpenAPI schemas before forwarding
Provides a complete audit trail: who called what, when, with what parameters

Internal Layer (Service Mesh)

Mutual TLS (mTLS) between services: even on the internal network, every service must prove its identity
Circuit breaking prevents a single failing service from cascading failures across the system
Automatic retries with backoff, handled by the infrastructure — not your application code

The key principle: defense in depth. Each layer enforces its own set of controls. A request that somehow bypasses the gateway still hits load balancer rules. A request that somehow bypasses those still hits service-level auth. No single point of failure in your security posture.

Decision Framework: What Do You Actually Need?

Are you running a single server?
    └─ YES → Reverse Proxy is sufficient.
              (SSL termination, caching, basic security)

Are you scaling horizontally?
    └─ YES → Add a Load Balancer.
              (Round robin, health checks, failover)

Are you running multiple independent services?
    └─ YES → Add an API Gateway.
              (Auth, rate limiting, routing, observability)

Do you have 20+ services with complex inter-service communication?
    └─ YES → Consider a Service Mesh.
              (mTLS, circuit breaking, distributed tracing)

The cardinal rule: don't over-architect for problems you don't have yet. A two-person startup shipping an MVP does not need Kong, Istio, and a 4-layer CDN strategy. A fintech processing millions of daily transactions running 40 microservices does.

Design for where you are. Build with an eye on where you're going. Layer when you actually need the layer, not when a conference talk made it sound cool.

Key Takeaways

A Reverse Proxy protects and optimizes a single backend server: SSL termination, caching, compression, IP hiding.
A Load Balancer is a function that distributes traffic across multiple instances. In HTTP systems it's often implemented as a reverse proxy, but not always — AWS NLB is a load balancer that isn't. The two concepts solve different problems: the proxy hides and mediates access, the balancer distributes load.
An API Gateway centralizes infrastructure concerns for microservices — auth, rate limiting, versioning, transformation, request aggregation (BFF) — and prevents logic drift across teams. It also introduces real costs: single point of failure, latency overhead, and team bottlenecks. Operate it deliberately.
These are not mutually exclusive. Modern tools like Nginx, Kong, and Envoy implement all three capabilities. The distinction is conceptual, not product-based.
In high-scale systems, these layer on top of each other — but there's no single universal arrangement. CDN → Gateway → Load Balancer → Proxies → Services is one common pattern; Kubernetes-native setups, Cloudflare-heavy stacks, and gateway-first architectures all look different.
Security is not one layer's job. Every component in the gateway layer enforces its own controls.

The moment you stop asking "which tool is the right one?" and start asking "which capability solves my current engineering problem?" — the entire space becomes a lot less confusing.

If this was useful, consider sharing it with a backend engineer who's been staring at an Nginx config wondering why it has 400 lines. You might save them an afternoon.

Tags: #backend #architecture #webdev #devops #systemdesign

OpenTelemetry for Node.js Developers: A Practical Guide to Observability in Distributed Systems

Olawale Afuye — Thu, 04 Jun 2026 09:15:29 +0000

TL;DR: Your app is running. Users are complaining. You have no idea why. This is what happens when you skip observability. OpenTelemetry (OTel) fixes that and this guide shows you exactly how to implement it in Node.js.

The Problem No Dashboard Will Tell You About

You've deployed your microservices. Everything looks green on the surface. Then at 2am, Slack goes off: "Checkout is broken."

You open your logs. You see... something. You check your metrics. You see... a spike. But which service caused it? Was it the payment service timing out? Was the inventory service returning bad data? Was the API gateway dropping requests?

Without proper observability, you're debugging in the dark.

This is the exact problem OpenTelemetry (OTel) was designed to solve.

What Is Observability (And Why "Monitoring" Isn't Enough)

Monitoring tells you that something is wrong. Observability tells you why.

Formally: observability is the measure of how well you can understand the internal state of a system based on the data it produces as output.

In a distributed, microservice-based architecture, you can't step inside a service and watch it run. Observability is how you get that visibility from the outside in.

It relies on four data types, collectively known as M.E.L.T.:

Pillar	What It Is	What It Tells You
Metrics	Numeric measurements at regular intervals	System performance trends over time (e.g. error rate, p99 latency)
Events	Discrete actions at a point in time	Business-level triggers (e.g. user purchase, payment initiated)
Logs	Granular, timestamped application output	Millisecond-by-millisecond reconstruction of events
Traces	A record of a request's full journey	Causal chains, bottlenecks, and cross-service latency

Most teams start with logs and call it a day. That's like having a CCTV system with no timestamps and no audio. Traces and metrics are what turn raw logs into a story you can actually debug.

OTel's actual signal taxonomy: OpenTelemetry formalizes three primary signals — Traces, Metrics, and Logs. In OTel, Events are expressed as Span Events (attached to a trace span) or structured log records, not a standalone fourth signal. M.E.L.T. is a broader observability framework concept popularized by vendors like New Relic. It's a useful mental model, but don't conflate it with OTel's own data model.

A Brief History: Why OpenTelemetry Exists

Before 2019, the observability ecosystem was fragmented. Two major CNCF projects competed for adoption:

OpenTracing — a vendor-neutral API for distributed tracing
OpenCensus — Google's solution for metrics and tracing

Both were good. Both were different. Neither was a standard.

In 2019, the two projects merged to form OpenTelemetry — a single, vendor-neutral, CNCF-backed framework for generating, collecting, and exporting telemetry data. Today it is the second most active CNCF project after Kubernetes.

The mandate is clear: write your instrumentation once, send it anywhere.

The Four Concepts You Must Understand Before Writing Code

1. Spans — The Atomic Unit of a Trace

A span represents a single unit of work: an HTTP call, a database query, a function execution. Every span has:

A name
A start and end timestamp (giving you latency)
Status (including whether it errored shown as red in Zipkin)
Optional attributes (metadata)

Spans are linked in parent-child relationships. When your Dashboard service calls your Movies service, the Dashboard's span becomes the parent, and the Movies service creates a child span. Together, they form a trace the complete map of a single request.

[Dashboard Service] ──────────────────────────── 320ms
    └── [Movies Service API Call] ────────── 290ms  ← bottleneck
          └── [DB Query: find_all_movies] ── 270ms  ← root cause

This is exactly the kind of visualization Zipkin gives you — and why distributed tracing is invaluable.

2. Span Context and Correlation Context

For spans across services to be linked into one trace, metadata must travel with each request. This is handled by two mechanisms:

Span Context: carries the traceId, spanId, and traceFlags — the IDs that tell the backend these spans belong together. Mandatory.
Correlation Context: carries user-defined properties like customerId, dataRegion, or providerHostname. Optional, but powerful for business-level debugging.

The OTel HTTP auto-instrumentation plugin propagates this context automatically via HTTP headers.

3. Metrics vs. Traces, When to Use Which

Concern	Use Traces	Use Metrics
"Why is this request slow?"	✅	❌
"What's our p99 latency over 30 days?"	❌	✅
"Which service is the bottleneck?"	✅	❌
"How many requests per second are we handling?"	❌	✅

Traces answer why at the individual request level. Metrics answer what at aggregate scale over time.

4. The OpenTelemetry Collector — Your Telemetry Router

Without a Collector, your service sends data directly to a specific backend (Zipkin, Jaeger, etc.). Change the backend and you re-instrument every service.

The OTel Collector sits in between:

[Service A] ──┐
[Service B] ──┼──► (OTLP) ──► [OTel Collector] ──► [Zipkin]     (local/dev)
[Service C] ──┘                      │              [New Relic]  (production)
                                     └────────────► [Prometheus] (metrics)

Your services always speak OTLP (OpenTelemetry's native protocol) to the Collector. The Collector then speaks whatever protocol each backend requires. This is the key architectural point: your application code is completely decoupled from the backend's data format.

It has three stages:

Receiver — accepts data in multiple formats (Zipkin, Jaeger, Prometheus, FluentBit)
Processor — filters, batches, or transforms data before export
Exporter — forwards to one or more backends

Swap backends without touching a single line of application code. This is the right architecture for production.

Setting Up Distributed Tracing in Node.js

The recommended architecture in 2025+ is: Node.js → OTLP → OTel Collector → backend. Your application always exports via OTLP (OTel's native protocol), and the Collector handles routing to whatever backend you're using.

We'll use Zipkin as the local visualization backend here — it's an excellent learning tool for seeing traces. In production you'd swap the Collector's exporter to New Relic, Jaeger, or wherever you're sending data, without touching the application.

Step 1: Install Dependencies

npm install @opentelemetry/sdk-node \
            @opentelemetry/exporter-trace-otlp-http \
            @opentelemetry/auto-instrumentations-node

@opentelemetry/sdk-node is the modern entry point. It wires together the trace provider, resource detection, and auto-instrumentations in one place. The older pattern of manually constructing NodeTracerProvider and calling provider.register() still works but is more verbose and prone to misconfiguration.

Step 2: Create Your `tracing.js` File

Keep instrumentation code in its own file, completely separate from your application logic.

// tracing.js
'use strict';

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  serviceName: 'movies-service',
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces', // OTel Collector OTLP/HTTP endpoint
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// Graceful shutdown — flush spans before the process exits
process.on('SIGTERM', () => {
  sdk.shutdown().finally(() => process.exit(0));
});

console.log('Tracing initialized');

NodeSDK handles resource attribute detection automatically (service name, runtime version, host info) and manages the lifecycle of the SDK cleanly. The SIGTERM handler ensures buffered spans are flushed rather than dropped when the process shuts down.

Step 3: Load It Before Your App Starts

Use the node -r flag to require the tracing file before your application code runs. This ensures that all subsequent modules are instrumented from the start.

node -r ./tracing.js index.js

Or in your package.json:

{
  "scripts": {
    "start": "node -r ./tracing.js index.js"
  }
}

Why -r? OTel works by monkey-patching Node.js core modules (like http). If your app loads before the instrumentation, those patches won't apply to already-loaded modules.

Step 4: Run the OTel Collector + Zipkin via Docker

# docker-compose.yml
services:
  zipkin:
    image: openzipkin/zipkin
    ports:
      - "9411:9411"

  otel-collector:
    image: otel/opentelemetry-collector-contrib
    command: ["--config=/etc/otel-config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
    depends_on:
      - zipkin

# otel-config.yaml (learning setup)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch: {}

exporters:
  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [zipkin]

Start the stack with docker-compose up, run your service, make some requests, and open http://localhost:9411. You'll see traces appear with parent-child span relationships, latency measurements per hop, and red error flags where things fail.

Setting Up Metrics Collection with Prometheus

Metrics are cheaper to store and better for trend analysis than traces. The standard framework for service-level metrics is RED: Rate (requests/second), Errors (error rate), Duration (latency). OTel supports all three through a Meter.

A counter covers Rate and Errors. For Duration, you need a histogram — not another counter. Histograms are what actually give you p95 and p99 latency, which are far more useful for SLOs than raw request counts.

Step 1: Install Dependencies

npm install @opentelemetry/sdk-metrics \
            @opentelemetry/exporter-prometheus

Step 2: Initialize Your Meter, Counter, and Histogram

// metrics.js
const { MeterProvider } = require('@opentelemetry/sdk-metrics');
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');

const exporter = new PrometheusExporter({ port: 9464 }, () => {
  console.log('Prometheus scrape endpoint: http://localhost:9464/metrics');
});

const meterProvider = new MeterProvider();
meterProvider.addMetricReader(exporter);

const meter = meterProvider.getMeter('movies-service');

const requestCounter = meter.createCounter('http_requests_total', {
  description: 'Total number of HTTP requests received',
});

// Histogram for latency — the correct instrument for p95/p99 analysis
const requestDuration = meter.createHistogram('http_request_duration_ms', {
  description: 'HTTP request latency in milliseconds',
  unit: 'ms',
});

module.exports = { requestCounter, requestDuration };

Step 3: Instrument Your Express Middleware

Note the res.on('finish', ...) pattern below. Reading res.statusCode directly in the middleware body gives you the default value (usually 200) before the response is actually sent — the finish event gives you the real status code and accurate duration.

// index.js
const { requestCounter, requestDuration } = require('./metrics');

app.use((req, res, next) => {
  const startTime = Date.now();

  res.on('finish', () => {
    const attributes = {
      method: req.method,
      route: req.route?.path ?? req.path,
      status_code: res.statusCode,
    };
    requestCounter.add(1, attributes);
    requestDuration.record(Date.now() - startTime, attributes);
  });

  next();
});

Step 4: Configure Prometheus to Scrape Your App

# prometheus.yml
global:
  scrape_interval: 15s  # Balance between granularity and performance

scrape_configs:
  - job_name: 'movies-service'
    static_configs:
      - targets: ['host.docker.internal:9464']

Environment note: host.docker.internal resolves on Docker Desktop (macOS and Windows). On Linux, Docker does not add this hostname by default use 172.17.0.1 (the default Docker bridge gateway) or the host's actual IP. In Kubernetes, replace this entirely with a proper service discovery config or a PodMonitor.

Why 15 seconds? Granular enough for accurate per-minute rate calculations and fast anomaly detection, without generating an excessive volume of data points.

Routing Everything Through the OTel Collector to New Relic

Once you're ready for production, use the same OTLP-first architecture your services already speak OTLP to the Collector. Only the Collector's exporter changes.

`otel-config.yaml` (production)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch: {}

exporters:
  otlp:
    endpoint: "https://otlp.nr-data.net:4317"
    headers:
      api-key: "${NEW_RELIC_LICENSE_KEY}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

`docker-compose.yml` (partial, production)

otel-collector:
  image: otel/opentelemetry-collector-contrib
  command: ["--config=/etc/otel-config.yaml"]
  volumes:
    - ./otel-config.yaml:/etc/otel-config.yaml
  ports:
    - "4317:4317"   # OTLP gRPC
    - "4318:4318"   # OTLP HTTP
  environment:
    - NEW_RELIC_LICENSE_KEY=${NEW_RELIC_LICENSE_KEY}

Nothing in your application code changes between the local Zipkin setup and this production config. That's the whole point of OTLP-first: your app doesn't know or care what's downstream of the Collector. In New Relic's Explorer view, you can visualize all services, their dependencies, and drill into latency across your full distributed system.

Sampling: The Production Topic Nobody Puts in Tutorials

In development, tracing every request is fine. In production, with thousands of requests per second, tracing everything is a fast way to generate a very expensive observability bill and a lot of noise.

Sampling is how you control what percentage of traces you actually record.

Head Sampling (Probabilistic)

The decision is made at the start of a trace, before any data is collected:

const { TraceIdRatioBased } = require('@opentelemetry/sdk-trace-base');

const sdk = new NodeSDK({
  serviceName: 'movies-service',
  sampler: new TraceIdRatioBased(0.1), // Record 10% of all traces
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
  instrumentations: [getNodeAutoInstrumentations()],
});

Simple, low-overhead. The tradeoff: you may drop the exact traces you needed errors and slow requests have the same chance of being dropped as fast successful ones.

Tail Sampling (Collector-side)

The decision is made at the Collector after seeing the full trace. This lets you express rules like: "always keep error traces, always keep traces over 1 second, sample everything else at 5%." This is the correct approach for production.

# In otel-config.yaml — requires otel/opentelemetry-collector-contrib
processors:
  tail_sampling:
    decision_wait: 10s   # Wait up to 10s for all spans in a trace to arrive
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow-traces
        type: latency
        latency: { threshold_ms: 1000 }
      - name: sample-everything-else
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

decision_wait needs to be long enough that all spans from across your services have arrived before the Collector makes its keep/drop call. Tune this based on your slowest service-to-service call.

The rule: never run OTel in high-traffic production without a sampling strategy. Head sampling is a quick win; tail sampling is the right architecture.

The Security Angle: What OWASP Says About Telemetry

This is the part most tutorials skip. Don't.

The OWASP Top 10 includes A09:2021 – Security Logging and Monitoring Failures precisely because bad observability is a security risk, not just an operational one.

Risks specific to telemetry:

1. Sensitive Data Leaking into Spans and Logs

Over-instrumentation is real. If you're logging request bodies, database queries, or user-facing errors without sanitization, you may be storing passwords, credit card numbers, session tokens, or PII directly inside your observability platform which is almost never protected as strictly as your production database.

Mitigation: Use OTel Collector processors to scrub sensitive fields before export:

processors:
  attributes:
    actions:
      - key: http.request.body
        action: delete
      - key: db.statement
        action: hash

2. Log Injection

If user-controlled input flows directly into span attributes or log messages without sanitization, attackers can inject crafted entries to manipulate your log analysis tools or hide their activity.

Mitigation: Never log raw user input. Sanitize and validate before adding to span attributes.

3. Insufficient Monitoring Leading to Undetected Breaches

If you're collecting telemetry but not alerting on it, you're warehousing evidence of your own compromise. Data alone isn't observability active monitoring is.

Mitigation:

Configure alerts for repeated authentication failures, unusual spike patterns, and anomalous service-to-service traffic
Route security-relevant telemetry to a SIEM, not just a tracing backend
Periodically audit what your services are actually sending

4. Credential Exposure in Collector Configuration

Your OTel Collector config references API keys and ingest tokens. These must come from environment variables or secrets managersnever hardcoded in otel-config.yaml.

# ❌ Never do this
api-key: "NRAK-XXXXXXXXXXXXXXXXXXXX"

# ✅ Do this
api-key: "${NEW_RELIC_LICENSE_KEY}"

Frontend Telemetry: It Works There Too

OTel isn't just for the backend. The @opentelemetry/sdk-trace-web package brings the same tracing model to the browser capturing document load times, XHR/fetch requests, and user interactions.

The critical win: trace propagation. When a user clicks a button in your browser app, the trace context is forwarded with the API call, linking the browser span to the backend span. You get a single trace that shows the full journey from UI click to database response.

Useful for Node.js BFFs, Next.js backends, and any system where front-to-back latency matters.

What to Implement First (A Practical Priority Order)

If you're starting from zero on an existing Node.js project, here's the sequence that gives you the fastest signal:

Set up NodeSDK with OTLP + the Collector locally — route to Zipkin for visualization. Even in local dev, using the Collector means your application code never needs to change when you switch backends.
Add a request duration histogram — a histogram on your most trafficked endpoint gives you p95/p99 latency immediately. A counter alone won't tell you what's slow.
Add sampling before production — head sampling is a 5-minute change. Tail sampling via the Collector is the right long-term answer. Skip this and high traffic will surprise you.
Audit your span attributes for PII — do this before you go to production. Retrofitting data redaction is painful.
Configure alerts — at minimum, alert on error rate and p95 latency. If you only get paged when users tweet at you, you're already too late.

Summary

Concept	Tool	What You Get
Instrumentation	NodeSDK + OTLP	Modern, vendor-neutral trace export
Local Tracing	OTel Collector → Zipkin	Request flow, latency, bottlenecks, errors
Metrics	Prometheus + Grafana	Rate/Error/Duration (RED), p95/p99 histograms
Sampling	Head or Tail (Collector)	Cost control without losing critical traces
Production Export	OTel Collector → New Relic	Full-stack service map and alerting
Security	OTel Processor + SIEM	PII redaction, breach detection

OpenTelemetry is not a monitoring tool, it's a contract. A contract between your application and any tool that wants to understand it. Write that contract once, and you're free to change backends, scale services, or bring in new tooling without starting over.

Resources to Go Deeper

Have you implemented OTel in a production Node.js system? What was the first thing traces revealed that you didn't expect? Drop it in the comments, genuinely curious.

Your Servers Have Passports. Are They Expiring Without You Knowing?

Olawale Afuye — Tue, 02 Jun 2026 12:14:40 +0000

Picture this: it's 2 AM. Your on-call phone explodes. Your payments API is down. Users are screaming. The infra team is deep in logs trying to figure out what broke — firewall rules, a bad deploy, infrastructure drift?

Turns out your TLS certificate expired six hours ago and nobody noticed.

That's not a hypothetical. It's a recurring nightmare for engineering teams all over the world. And with the industry aggressively shrinking certificate lifespans — down to 47 days by 2029 — it's about to get a lot worse for teams that aren't paying attention.

This post is your primer. We'll cover what digital certificates actually are, why they matter more than most developers realise, what "machine identity sprawl" is, and how to stop treating cert management as an afterthought.

First: What Even Is a Digital Certificate?

Here's the simplest mental model.

Certificates are passports for machines, not people.

When your browser connects to https://api.yourbank.com, it needs to answer a critical question before sending any data: "Is this actually the server I think it is, or could someone be intercepting this connection?" A digital certificate is the server's answer. It says:

"Here is my name, here is my public key, and here is the signature of a trusted authority that vouches for both."

Technically, a certificate bundles:

The server's hostname (what it claims to be)
The server's public key (used to establish encrypted communication)
A digital signature from a Certificate Authority (CA) — a trusted third party that vouches for the binding of that name to that key

Think of the CA as the government that issued the passport. You don't personally know the bearer, but you trust the issuing authority enough to accept the document.

The Three Pillars This Enables

Once a valid certificate is established, it unlocks three critical security guarantees:

Pillar	What It Means
Authentication	You're talking to the real server, not an impersonator
Confidentiality	Your data is encrypted in transit and only the server with the matching private key can read it
Integrity	The data hasn't been modified between sender and receiver

Remove any one of these, and your "secure" connection is theatre.

The Man-in-the-Middle Threat (And Why Certs Stop It)

Here's the attack that certificates are specifically designed to prevent.

An attacker positions themselves between your user and your server. They intercept the request, pretend to be your server to the user, and pretend to be the user to your server. All traffic flows through them. They can read everything, modify anything, and neither side is any wiser.

Without cert validation:
User ──── ATTACKER ──── Your Server
            ↑
        intercepts and relays everything

With cert validation:
User ──[checks cert]──✓──── Your Server
     Attacker can't forge the CA's signature

Without a valid certificate — one signed by a CA the browser trusts — the attacker cannot present a credential that passes verification. The browser (or client) catches it. The connection is rejected.

But here's the thing: if your certificate expires, the browser treats it exactly the same as a forged one. Because from the browser's perspective, it is just as untrustworthy. Which brings us to the real problem.

The Problem: Machine Identity Sprawl

Ten years ago, you might have had a handful of certificates to manage. One for your main domain, maybe one for your API subdomain.

That era is gone.

Modern enterprises run:

Web servers and subdomains
REST and gRPC APIs
Microservices talking to each other over mTLS
Load balancers and reverse proxies
IoT devices and edge nodes
Internal tooling: CI/CD pipelines, Kubernetes clusters, internal dashboards
Third-party integrations, SaaS connectors, partner APIs

Each of these can have one or more certificates. A mid-sized engineering organisation can easily have hundreds or thousands of active certificates across its infrastructure.

This is machine identity sprawl: the explosion of machine-level credentials distributed across systems, teams, clouds, and environments — most of which were issued, forgotten, and are now quietly ticking toward expiry on nobody's radar.

The dangerous part isn't complexity. It's invisibility. Nobody sends you a calendar invite for cert expiry. There's no build failure. No test suite catches it. You find out when the production API starts returning connection errors at scale, usually at the worst possible time.

SSL vs TLS: A Quick Clarification

You'll hear "SSL certificate" constantly — in documentation, in vendor dashboards, in job descriptions. It's worth being precise here.

SSL (Secure Sockets Layer) is the original protocol. It's been deprecated. SSL 2.0 and 3.0 both have known, exploitable vulnerabilities and should not be used.

TLS (Transport Layer Security) is the current standard. TLS 1.2 and TLS 1.3 are what you want. TLS 1.3 (released 2018) cut unnecessary handshake round-trips, removed weak cipher suites, and is meaningfully faster and more secure.

The certificates themselves haven't fundamentally changed in shape — they still use the same X.509 format. But when someone says "SSL certificate" today, they mean a certificate used for TLS. The name is a legacy holdover that stuck.

If you're configuring a new server and you see options for SSL 2.0, SSL 3.0, or TLS 1.0/1.1 — disable them. All of them.

Why Shorter Lifespans Are Actually a Good Thing (Even If They're Painful)

Here's the uncomfortable trade-off the industry is making.

Certificate lifespans have been shrinking aggressively:

2015: Up to 5 years
2018: 2 years max
2020: 1 year max (13 months)
2029 target: 47 days

This feels like a headache being manufactured by the CA/Browser Forum. But the reasoning is sound.

If an attacker compromises your server's private key, they can impersonate your server until that certificate expires or is manually revoked. A certificate valid for 2 years gives an attacker a 2-year window to exploit a compromised credential — assuming you even detect the compromise.

Short lifespans shrink that window dramatically. A 47-day certificate means even a successful key compromise has a limited blast radius before the certificate naturally rotates out of existence.

It also forces cryptographic hygiene. Every renewal is an opportunity to use stronger key sizes, updated cipher suites, and current security standards. Organisations with 2-year certs can sit on weak configurations for years without touching them.

The catch, of course, is that a 47-day lifespan makes manual renewal not just inconvenient — it makes it mathematically impossible at enterprise scale. You cannot have a human manually renewing hundreds of certificates every six weeks. The industry is forcing automation, and it's the right call.

The Certificate Lifecycle: What You Need to Manage

Treating cert management as "buy, install, forget" is how you end up in the 2 AM outage. A proper lifecycle has four stages:

1. Discovery

You cannot manage what you cannot see.

The first step is finding every certificate across your entire infrastructure — including the ones that were issued years ago by a developer who has since left, deployed on a server that isn't in your main dashboard, and which nobody has touched since.

Automated discovery tools scan your network, check endpoints, and build a full inventory. This is often the most surprising step. Teams consistently find dozens of "unknown" certificates when they first run a discovery scan.

2. Issue & Deploy

Automate the issuance and deployment pipeline entirely. Tools like Let's Encrypt (with Certbot), HashiCorp Vault, or enterprise platforms like Venafi and AppViewX can handle this end to end.

A good setup issues the certificate, deploys it to the right server or load balancer, triggers a reload (without downtime), and logs the event — all without human intervention.

# Example: Certbot automatic renewal via cron
0 0,12 * * * root certbot renew --quiet --post-hook "systemctl reload nginx"

For internal services or mTLS between microservices, a private CA (like Vault's PKI secrets engine) handles issuance internally without going through public CAs.

3. Monitor

Every certificate in your fleet should have active monitoring on:

Expiry date — alerts at 30 days, 14 days, 7 days out
Validity — is the cert still being served correctly?
Chain integrity — is the full trust chain intact?
Coverage — are all subdomains and SANs still accurate?

This is your early warning system. If your automation pipeline breaks, monitoring catches it before users do.

4. Rotate & Revoke

Certificates need to be replaced on schedule (rotation) and immediately if a compromise is suspected (revocation).

Revocation is important and under-implemented. If a private key is exposed — through a breach, a misconfigured server, a leaked secrets file in a public repo — the certificate must be revoked immediately through the CA. A revoked certificate tells clients: "Do not trust this, regardless of the expiry date."

The failure mode when certificates are not retired is subtle but serious: old certificates associated with deprecated services, decommissioned servers, or former employees' infrastructure can become silent attack surfaces. If the private key still exists somewhere and the certificate hasn't been revoked, it's a live credential that nobody is watching.

Why "Cryptographic Hygiene" Is Bigger Than Just Certs

Certificates are the most visible part of your cryptographic surface, but they're not the whole picture.

A genuine cryptographic hygiene audit also looks at:

Key sizes: RSA 2048-bit is a current minimum. RSA 4096 or ECDSA P-256/P-384 are preferred.
Cipher suites: Weak or deprecated ciphers (RC4, DES, 3DES) should be disabled even if your server technically supports them.
Library versions: OpenSSL, BoringSSL, and similar libraries have their own vulnerability histories. Are you running patched versions?
Protocol versions: TLS 1.0 and 1.1 are deprecated. Are they still enabled on any of your services?
Post-quantum readiness: NIST standardised its first quantum-resistant algorithms in 2024. Forward-thinking teams are beginning to inventory what a migration path looks like, even if it's years away.

Certificates are the fire you can see. These are the smoldering ones.

What Does Automation Actually Change?

Here's the honest answer: automation doesn't eliminate security risk. It eliminates the specific, unnecessary, entirely preventable risk that comes from human forgetfulness at scale.

Automated certificate management means:

Renewals happen on schedule, not when someone checks a spreadsheet
Deployment is consistent, not dependent on which engineer is available that weekend
Expiry monitoring doesn't rely on someone reading an email from six months ago
Rotation is a routine event, not an emergency

What automation doesn't do is protect you from a compromised CA, a misconfigured deployment script, or a zero-day in your TLS implementation. Those require different controls. But the "two-year-cert-in-a-forgotten-spreadsheet" class of incident? That's fully solvable, right now, with the tooling that exists today.

Quick Reference: The Toolkit

Need	Open Source Option	Enterprise Option
Public cert issuance	Let's Encrypt + Certbot	DigiCert, Sectigo
Internal / private CA	HashiCorp Vault PKI	Venafi, AppViewX
Discovery & inventory	ssl-cert-check, Shodan	Keyfactor, Sectigo SCM
Monitoring	Prometheus + custom exporter	Datadog, New Relic
mTLS between services	cert-manager (K8s)	Istio, Linkerd

The Bottom Line

Certificates are infrastructure. Not a one-time setup task. Not a DevOps checklist item. Infrastructure — like your database, your load balancer, your secrets manager. It requires the same treatment: automation, monitoring, documented runbooks, and ownership.

By 2029, the industry will not give you a choice. 47-day certificates make manual management impossible by design. The teams that start treating certificate lifecycle as a first-class engineering concern today will have the tooling and culture in place before the deadline. The teams that don't will be having a lot of 2 AM conversations.

Your servers have passports. Make sure they're not expiring in a drawer somewhere.

Found this useful? Drop a comment below with how your team currently handles cert management — spreadsheet, automation, or "we'll figure it out when it breaks." No judgment. Mostly judgment.

Your Servers Have Passports. Are They Expiring Without You Knowing?

Olawale Afuye — Tue, 02 Jun 2026 12:14:22 +0000

Turns out your TLS certificate expired six hours ago and nobody noticed.

First: What Even Is a Digital Certificate?

Here's the simplest mental model.

Certificates are passports for machines, not people.

"Here is my name, here is my public key, and here is the signature of a trusted authority that vouches for both."

Technically, a certificate bundles:

The server's hostname (what it claims to be)
The server's public key (used to establish encrypted communication)
A digital signature from a Certificate Authority (CA) — a trusted third party that vouches for the binding of that name to that key

Think of the CA as the government that issued the passport. You don't personally know the bearer, but you trust the issuing authority enough to accept the document.

The Three Pillars This Enables

Once a valid certificate is established, it unlocks three critical security guarantees:

Pillar	What It Means
Authentication	You're talking to the real server, not an impersonator
Confidentiality	Your data is encrypted in transit and only the server with the matching private key can read it
Integrity	The data hasn't been modified between sender and receiver

Remove any one of these, and your "secure" connection is theatre.

The Man-in-the-Middle Threat (And Why Certs Stop It)

Here's the attack that certificates are specifically designed to prevent.

Without cert validation:
User ──── ATTACKER ──── Your Server
            ↑
        intercepts and relays everything

With cert validation:
User ──[checks cert]──✓──── Your Server
     Attacker can't forge the CA's signature

The Problem: Machine Identity Sprawl

Ten years ago, you might have had a handful of certificates to manage. One for your main domain, maybe one for your API subdomain.

That era is gone.

Modern enterprises run:

Web servers and subdomains
REST and gRPC APIs
Microservices talking to each other over mTLS
Load balancers and reverse proxies
IoT devices and edge nodes
Internal tooling: CI/CD pipelines, Kubernetes clusters, internal dashboards
Third-party integrations, SaaS connectors, partner APIs

Each of these can have one or more certificates. A mid-sized engineering organisation can easily have hundreds or thousands of active certificates across its infrastructure.

SSL vs TLS: A Quick Clarification

You'll hear "SSL certificate" constantly — in documentation, in vendor dashboards, in job descriptions. It's worth being precise here.

SSL (Secure Sockets Layer) is the original protocol. It's been deprecated. SSL 2.0 and 3.0 both have known, exploitable vulnerabilities and should not be used.

If you're configuring a new server and you see options for SSL 2.0, SSL 3.0, or TLS 1.0/1.1 — disable them. All of them.

Why Shorter Lifespans Are Actually a Good Thing (Even If They're Painful)

Here's the uncomfortable trade-off the industry is making.

Certificate lifespans have been shrinking aggressively:

2015: Up to 5 years
2018: 2 years max
2020: 1 year max (13 months)
2029 target: 47 days

This feels like a headache being manufactured by the CA/Browser Forum. But the reasoning is sound.

Short lifespans shrink that window dramatically. A 47-day certificate means even a successful key compromise has a limited blast radius before the certificate naturally rotates out of existence.

The Certificate Lifecycle: What You Need to Manage

Treating cert management as "buy, install, forget" is how you end up in the 2 AM outage. A proper lifecycle has four stages:

1. Discovery

You cannot manage what you cannot see.

2. Issue & Deploy

A good setup issues the certificate, deploys it to the right server or load balancer, triggers a reload (without downtime), and logs the event — all without human intervention.

# Example: Certbot automatic renewal via cron
0 0,12 * * * root certbot renew --quiet --post-hook "systemctl reload nginx"

For internal services or mTLS between microservices, a private CA (like Vault's PKI secrets engine) handles issuance internally without going through public CAs.

3. Monitor

Every certificate in your fleet should have active monitoring on:

Expiry date — alerts at 30 days, 14 days, 7 days out
Validity — is the cert still being served correctly?
Chain integrity — is the full trust chain intact?
Coverage — are all subdomains and SANs still accurate?

This is your early warning system. If your automation pipeline breaks, monitoring catches it before users do.

4. Rotate & Revoke

Certificates need to be replaced on schedule (rotation) and immediately if a compromise is suspected (revocation).

Why "Cryptographic Hygiene" Is Bigger Than Just Certs

Certificates are the most visible part of your cryptographic surface, but they're not the whole picture.

A genuine cryptographic hygiene audit also looks at:

Key sizes: RSA 2048-bit is a current minimum. RSA 4096 or ECDSA P-256/P-384 are preferred.
Cipher suites: Weak or deprecated ciphers (RC4, DES, 3DES) should be disabled even if your server technically supports them.
Library versions: OpenSSL, BoringSSL, and similar libraries have their own vulnerability histories. Are you running patched versions?
Protocol versions: TLS 1.0 and 1.1 are deprecated. Are they still enabled on any of your services?
Post-quantum readiness: NIST standardised its first quantum-resistant algorithms in 2024. Forward-thinking teams are beginning to inventory what a migration path looks like, even if it's years away.

Certificates are the fire you can see. These are the smoldering ones.

What Does Automation Actually Change?

Here's the honest answer: automation doesn't eliminate security risk. It eliminates the specific, unnecessary, entirely preventable risk that comes from human forgetfulness at scale.

Automated certificate management means:

Renewals happen on schedule, not when someone checks a spreadsheet
Deployment is consistent, not dependent on which engineer is available that weekend
Expiry monitoring doesn't rely on someone reading an email from six months ago
Rotation is a routine event, not an emergency

Quick Reference: The Toolkit

Need	Open Source Option	Enterprise Option
Public cert issuance	Let's Encrypt + Certbot	DigiCert, Sectigo
Internal / private CA	HashiCorp Vault PKI	Venafi, AppViewX
Discovery & inventory	ssl-cert-check, Shodan	Keyfactor, Sectigo SCM
Monitoring	Prometheus + custom exporter	Datadog, New Relic
mTLS between services	cert-manager (K8s)	Istio, Linkerd

The Bottom Line

Your servers have passports. Make sure they're not expiring in a drawer somewhere.

Found this useful? Drop a comment below with how your team currently handles cert management — spreadsheet, automation, or "we'll figure it out when it breaks." No judgment. Mostly judgment.

Web Security Is Everyone's Job: A Developer's Field Guide

Olawale Afuye — Tue, 02 Jun 2026 06:39:01 +0000

Most web security guides cover the classics. XSS. SQL injection. CSRF. The OWASP Top 10. Those matter, and if you haven't read them, start there.

But modern attacks don't stop at the classics.

Today's breaches happen through forgotten API endpoints, leaked secrets in .env files committed to public repos, authorization checks that were never written, and npm packages that were compromised six months before anyone noticed.

This guide covers the full picture — including the five areas most security articles skip entirely.

Part 1: Authentication and Session Security

Authentication answers one question: who are you?

Getting it wrong is catastrophic. But "getting it right" means more than hashing passwords and setting a cookie. Sessions have their own threat surface, and most guides don't go deep enough on it.

Password Hashing

Never store passwords in plaintext or with reversible encryption. Use a slow, purpose-built hashing algorithm.

// ✅ Correct — bcrypt adds salt automatically and is deliberately slow
const bcrypt = require('bcrypt');
const hash = await bcrypt.hash(password, 12); // cost factor of 12

// ✅ Also acceptable — Argon2 is the modern standard
const argon2 = require('argon2');
const hash = await argon2.hash(password);

// ❌ Never do this
const md5 = require('md5');
const hash = md5(password); // fast = bad for passwords

Use bcrypt, Argon2, or scrypt. Never MD5, SHA-1, or SHA-256 alone — these are fast by design, which makes brute-force practical.

CSRF Protection

Cross-Site Request Forgery tricks an authenticated user's browser into making a request on a malicious site's behalf. If your state-changing endpoints don't verify request origin, any site can trigger them.

// Express + csurf middleware
const csrf = require('csurf');
app.use(csrf({ cookie: true }));

// In your route, pass the token to the client
app.get('/form', (req, res) => {
  res.render('form', { csrfToken: req.csrfToken() });
});

// The form includes the token
// <input type="hidden" name="_csrf" value="<%= csrfToken %>">

Modern SameSite cookie attributes reduce CSRF risk significantly — but they're not a complete replacement for CSRF tokens on sensitive endpoints.

Secure Cookie Flags

Cookies are the primary session transport. Three flags make them dramatically harder to steal or misuse:

Set-Cookie: session=abc123;
  HttpOnly;       // JS cannot read this cookie — prevents XSS theft
  Secure;         // Only sent over HTTPS
  SameSite=Strict // Not sent on cross-site requests — kills CSRF

Flag	What It Prevents
`HttpOnly`	XSS scripts reading your session cookie
`Secure`	Cookie transmission over plain HTTP
`SameSite=Strict`	CSRF attacks via cross-origin requests
`SameSite=Lax`	Most CSRF, while allowing top-level navigations

Set all three. There is almost no legitimate reason not to.

Session Fixation and Session Rotation

Session fixation is a lesser-known but serious attack. The attacker tricks a user into using a session ID they already know — then waits for the user to authenticate. Once the user logs in, the attacker has a valid authenticated session.

The fix is simple: rotate the session ID on login.

// Express-session example
app.post('/login', async (req, res) => {
  const user = await authenticate(req.body);
  if (!user) return res.status(401).send('Invalid credentials');

  // ✅ Regenerate session after login — destroys old session ID
  req.session.regenerate((err) => {
    if (err) return next(err);
    req.session.userId = user.id;
    res.redirect('/dashboard');
  });
});

Never reuse a pre-login session ID after authentication. New user state = new session ID. Always.

JWT Security

JWTs are everywhere. They're also misunderstood in ways that create serious vulnerabilities.

The none algorithm attack:

// ❌ Vulnerable — accepts unsigned tokens if alg is "none"
jwt.verify(token, secret, { algorithms: ['HS256', 'none'] });

// ✅ Explicitly specify only the algorithm you expect
jwt.verify(token, secret, { algorithms: ['HS256'] });

What to store in a JWT — and what not to:

// ✅ Minimal, safe payload
const token = jwt.sign(
  { sub: user.id, role: user.role },
  process.env.JWT_SECRET,
  { expiresIn: '15m' } // Short-lived access tokens
);

// ❌ Never put sensitive data in the payload
// JWT payloads are base64-encoded, not encrypted — anyone can decode them
const bad = jwt.sign({ password: user.password, ssn: user.ssn }, secret);

Key JWT rules:

Keep access tokens short-lived (15 minutes is a reasonable default)
Never store sensitive data in the payload — it's encoded, not encrypted
Use strong secrets (32+ random bytes minimum)
Always validate exp, iss, and aud claims

Refresh Token Rotation

Short-lived access tokens are good. But users can't re-authenticate every 15 minutes. The solution is a long-lived refresh token that issues new access tokens — but only once.

// On login — issue both tokens
const accessToken = jwt.sign({ sub: user.id }, ACCESS_SECRET, { expiresIn: '15m' });
const refreshToken = generateSecureToken(); // e.g., crypto.randomBytes(64).toString('hex')

// Store refresh token hash in DB, associated with user
await db.refreshTokens.create({
  userId: user.id,
  tokenHash: hash(refreshToken),
  expiresAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000)
});

// On refresh — rotate: invalidate old, issue new
app.post('/auth/refresh', async (req, res) => {
  const { refreshToken } = req.cookies;

  const record = await db.refreshTokens.findByHash(hash(refreshToken));
  if (!record || record.expiresAt < new Date()) {
    return res.status(401).send('Invalid or expired refresh token');
  }

  // ✅ Invalidate the used token immediately (rotation)
  await db.refreshTokens.delete(record.id);

  // Issue new pair
  const newAccess = jwt.sign({ sub: record.userId }, ACCESS_SECRET, { expiresIn: '15m' });
  const newRefresh = generateSecureToken();
  await db.refreshTokens.create({ userId: record.userId, tokenHash: hash(newRefresh) });

  res.json({ accessToken: newAccess });
  // Set newRefresh as HttpOnly cookie
});

Rotation means a stolen refresh token can only be used once. If you detect a replay (the old token used after rotation), it signals a compromise — invalidate all sessions for that user immediately.

Token Revocation

JWTs are stateless, which is their strength and their weakness. Once issued, a token is valid until it expires — even if the user logs out, resets their password, or gets banned.

Strategies to solve this:

Option 1 — Blocklist invalidated tokens:

// On logout or password change
await redis.setex(`revoked:${jti}`, tokenTTL, 'true');

// On every request
const isRevoked = await redis.get(`revoked:${jti}`);
if (isRevoked) return res.status(401).send('Token revoked');

Option 2 — Short expiry + refresh token as the revocation point:
Keep access tokens short (15m). On logout, delete the refresh token from the database. The access token expires naturally and can't be renewed.

For high-security operations (password changes, role changes), always invalidate all active sessions immediately regardless of approach.

Part 2: Authorization

Authentication answers who are you?

Authorization answers: what are you allowed to do?

This distinction matters enormously. Many modern breaches aren't authentication failures — the attacker authenticated just fine. The breach happened because authorization was never properly implemented or enforced.

IDOR — Insecure Direct Object Reference

This is one of the most common API vulnerabilities in production systems today. It looks like this:

GET /api/users/123/profile      ← You
GET /api/users/124/profile      ← Not you — but does the server stop you?

If the server returns user 124's data without checking that the requesting user owns or has permission to view that record, that's an IDOR vulnerability.

// ❌ Broken — fetches based on URL param, no ownership check
app.get('/api/users/:id/profile', auth, async (req, res) => {
  const user = await db.users.findById(req.params.id);
  res.json(user);
});

// ✅ Fixed — verify the requesting user owns this resource
app.get('/api/users/:id/profile', auth, async (req, res) => {
  if (req.user.id !== req.params.id && req.user.role !== 'admin') {
    return res.status(403).send('Forbidden');
  }
  const user = await db.users.findById(req.params.id);
  res.json(user);
});

The pattern to follow: never trust the client to tell you what they're allowed to access. Always verify server-side.

Broken Object Level Authorization (BOLA)

BOLA is the API-era name for IDOR, and it's OWASP API Security's #1 risk. Every object accessed via an API must be authorization-checked, not just authenticated.

// Scenario: User requests their own order
// ❌ No authorization check
app.get('/api/orders/:orderId', auth, async (req, res) => {
  const order = await db.orders.findById(req.params.orderId);
  res.json(order);
});

// ✅ Enforce ownership at the query level
app.get('/api/orders/:orderId', auth, async (req, res) => {
  const order = await db.orders.findOne({
    id: req.params.orderId,
    userId: req.user.id  // Ownership enforced in the query itself
  });

  if (!order) return res.status(404).send('Not found');
  res.json(order);
});

A practical rule: if you're doing findById followed by a permission check, refactor so the permission is part of the query. Fewer round trips, fewer gaps.

Broken Function Level Authorization

This is about endpoints, not objects. The question isn't "can you see this record" but "can you perform this action at all?"

// ❌ Route exists but has no role check
app.delete('/api/admin/users/:id', auth, async (req, res) => {
  await db.users.delete(req.params.id);
  res.send('Deleted');
});

// ✅ Role enforced via middleware
const requireRole = (role) => (req, res, next) => {
  if (req.user.role !== role) return res.status(403).send('Forbidden');
  next();
};

app.delete('/api/admin/users/:id', auth, requireRole('admin'), async (req, res) => {
  await db.users.delete(req.params.id);
  res.send('Deleted');
});

Admin routes that don't enforce admin roles are surprisingly common. Treat every route as untrusted and define authorization explicitly — don't rely on the UI not exposing a link.

A Practical Authorization Checklist

Before any endpoint goes to production:

[ ] Does this endpoint require authentication?
[ ] Does this endpoint require a specific role or permission?
[ ] If it returns or modifies a specific resource, does it verify the requester owns or has access to that resource?
[ ] Are these checks server-side, not client-side?
[ ] Are these checks covered by automated tests?

Part 3: API Security

Most modern systems aren't browser → server. They look more like this:

Mobile App / SPA
      ↓
  API Gateway
      ↓
  Microservices
      ↓
   Databases

Every arrow in that diagram is an attack surface. Here's how to secure it.

Rate Limiting

Unprotected APIs are trivially brute-forced, scraped, or abused. Rate limiting is non-negotiable.

const rateLimit = require('express-rate-limit');

// General API rate limit
const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100,
  message: { error: 'Too many requests, please try again later.' }
});

// Stricter limit on auth endpoints
const authLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 10, // 10 login attempts per 15 minutes per IP
  skipSuccessfulRequests: true
});

app.use('/api/', apiLimiter);
app.use('/api/auth/', authLimiter);

For distributed systems, use a Redis-backed store so limits work across multiple instances:

const RedisStore = require('rate-limit-redis');
const rateLimit = rateLimit({
  store: new RedisStore({ client: redisClient }),
  windowMs: 15 * 60 * 1000,
  max: 100
});

API Key Management

API keys are credentials. Treat them accordingly.

// ❌ Raw key stored in DB
await db.apiKeys.create({ key: rawKey, userId });

// ✅ Hash the key — store only the hash
const crypto = require('crypto');
const keyHash = crypto.createHash('sha256').update(rawKey).digest('hex');
await db.apiKeys.create({ keyHash, userId, lastUsed: null });

// On verification
const incoming = crypto.createHash('sha256').update(req.header('X-API-Key')).digest('hex');
const record = await db.apiKeys.findOne({ keyHash: incoming });

Additional API key practices:

Scope keys to specific permissions (read-only, write, admin)
Implement expiration and rotation
Log usage — who used this key, when, for what
Provide a revocation mechanism
Never log the raw key value anywhere

OAuth 2.0 — Getting It Right

OAuth is the standard for delegated authorization. It's also frequently misimplemented.

User → Your App → Authorization Server (e.g. Google)
                         ↓
               Authorization Code
                         ↓
            Your Backend exchanges for tokens
                         ↓
                   Access Token

Critical implementation notes:

// ✅ Always validate the state parameter — prevents CSRF on OAuth flow
app.get('/auth/callback', async (req, res) => {
  const { code, state } = req.query;

  // Verify state matches what you stored before redirect
  if (state !== req.session.oauthState) {
    return res.status(403).send('State mismatch — possible CSRF');
  }

  // Exchange code for tokens on the backend, not the frontend
  const tokens = await exchangeCodeForTokens(code);
  // ...
});

Never use the Implicit Flow (tokens returned directly in URL fragments). Always use Authorization Code + PKCE for public clients (SPAs, mobile apps).

Service-to-Service Authentication

Internal services calling each other need authentication too. Assume a compromised microservice could make requests on behalf of another.

Options:

mTLS (Mutual TLS): Both sides present certificates — common in service meshes like Istio
Short-lived service tokens: Each service gets a JWT for internal calls with a very short TTL
API keys with IP allowlisting: Simpler, but less flexible

// Service-to-service token validation middleware
const validateServiceToken = async (req, res, next) => {
  const token = req.header('X-Service-Token');
  try {
    const payload = jwt.verify(token, process.env.INTERNAL_SERVICE_SECRET);
    if (payload.type !== 'service') return res.status(403).send('Forbidden');
    req.callerService = payload.service;
    next();
  } catch {
    res.status(401).send('Invalid service token');
  }
};

GraphQL Security

GraphQL introduces security challenges that REST doesn't have.

Introspection in production:

// ❌ Default — exposes your entire schema to anyone
const server = new ApolloServer({ schema });

// ✅ Disable introspection in production
const server = new ApolloServer({
  schema,
  introspection: process.env.NODE_ENV !== 'production'
});

Query depth and complexity limits:

const depthLimit = require('graphql-depth-limit');
const { createComplexityLimitRule } = require('graphql-validation-complexity');

const server = new ApolloServer({
  schema,
  validationRules: [
    depthLimit(5),  // Prevents deeply nested query attacks
    createComplexityLimitRule(1000)  // Prevents complexity-based DoS
  ]
});

N+1 and batching: Use DataLoader to prevent N+1 queries that could be exploited for resource exhaustion.

Request Signing

For high-security API communication, request signing ensures the request wasn't tampered with in transit and proves it came from a specific client.

// Signing (client side)
const crypto = require('crypto');

function signRequest(method, path, body, secret) {
  const timestamp = Date.now().toString();
  const payload = `${method}\n${path}\n${timestamp}\n${JSON.stringify(body)}`;
  const signature = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  return { signature, timestamp };
}

// Verification (server side)
function verifySignature(req, secret) {
  const { signature, timestamp } = req.headers;

  // Reject requests older than 5 minutes — prevents replay attacks
  if (Date.now() - parseInt(timestamp) > 5 * 60 * 1000) return false;

  const payload = `${req.method}\n${req.path}\n${timestamp}\n${JSON.stringify(req.body)}`;
  const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}

AWS uses this pattern (SigV4). Stripe uses it for webhooks. It's the right approach whenever you need cryptographic proof of request integrity.

Part 4: Supply Chain Security

You write your own code carefully. But your application is also made of hundreds of thousands of lines of code you didn't write, by people you've never met, distributed as packages you installed in 30 seconds.

That's your supply chain. And it's increasingly the target.

The Real Threat Surface

Typosquatting — malicious packages with names one character off from popular ones:

npm install cross-env     # Legitimate
npm install crossenv      # Malicious package — real incident, 2018
npm install lodash        # Legitimate
npm install 1odash        # Hypothetical typosquat — lowercase L vs 1

Dependency confusion — publishing a public package with the same name as an internal private one to intercept installations in misconfigured environments. This is how a researcher compromised Apple, Microsoft, and Shopify in 2021.

Compromised maintainers — legitimate packages taken over through account compromise or maintainer handoff. The event-stream incident in 2018 involved exactly this.

Poisoned CI/CD — SolarWinds demonstrated that attackers who compromise your build pipeline can inject malicious code that never appears in your source repo.

Practical Defenses

Lock your dependency tree:

# Always commit package-lock.json or yarn.lock
# Use --frozen-lockfile in CI to prevent silent upgrades
npm ci --frozen-lockfile

# Pin exact versions for critical dependencies
npm install lodash@4.17.21  # Exact pin, not ^4.17.21

Audit regularly:

npm audit
npm audit --audit-level=high  # Fail CI on high-severity issues

# For more comprehensive scanning
npx snyk test

Subresource Integrity (SRI) for CDN assets:

<!-- ✅ Browser verifies the file hash before executing -->
<script
  src="https://cdn.example.com/lib.min.js"
  integrity="sha384-abc123..."
  crossorigin="anonymous">
</script>

Vet before you install:

Check the package's weekly downloads and GitHub star trajectory
Look at when it was last published and whether it's actively maintained
Review the install script (npm install can execute arbitrary code via preinstall hooks)
Use socket.dev or Snyk for automated supply chain analysis

Scope your npm tokens in CI:

# Use read-only tokens in CI where you only need to install
# Use publish-scoped tokens only in release pipelines
# Rotate tokens regularly
# Never use personal access tokens in shared environments

Securing Your CI/CD Pipeline

# GitHub Actions — use commit SHA pinning, not tag pinning
# ❌ Tags can be moved to point to different code
- uses: actions/checkout@v3

# ✅ Pinning to a specific commit SHA is immutable
- uses: actions/checkout@8e5e7e5a8e3...  # Full SHA

# Restrict permissions to minimum required
permissions:
  contents: read        # Don't grant write unless needed
  id-token: write       # Only for OIDC-based cloud auth

# Use environment secrets, not repo secrets for production
# Run security scans as a required CI step

Part 5: Secrets Management

Developers leak secrets constantly. Not maliciously — carelessly. A .env file committed to a public repo. A hardcoded API key in a utility script. A database password in a log line.

This section is the most practical in this entire guide.

What Counts as a Secret

Everything in this category must be treated as a secret:

Database credentials (connection strings, passwords)
API keys and tokens (third-party services)
JWT signing secrets
Encryption keys
Cloud credentials (AWS access keys, GCP service account keys)
Private certificates and keys
Webhook signing secrets

The Anti-Patterns That Get People Fired

// ❌ Hardcoded secrets
const db = new Client({ password: 'myS3cretP@ss' });

// ❌ Secrets in environment variable names that get logged
console.log('Config:', process.env); // This logs everything

// ❌ Secrets in client-side code
const apiKey = 'sk-live-abc123'; // Visible to anyone viewing source

// ❌ .env files committed to version control
// (even if you delete them, they remain in git history)

The Right Approach: Secrets Managers

For production systems, don't manage secrets yourself. Use a dedicated secrets manager.

// AWS Secrets Manager
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');

const client = new SecretsManagerClient({ region: 'us-east-1' });

async function getSecret(secretName) {
  const response = await client.send(
    new GetSecretValueCommand({ SecretId: secretName })
  );
  return JSON.parse(response.SecretString);
}

// Usage — fetch at startup, not hardcoded
const { DB_PASSWORD, JWT_SECRET } = await getSecret('prod/myapp/secrets');

// HashiCorp Vault
const vault = require('node-vault')({ endpoint: 'https://vault.company.com' });
await vault.approleLogin({ role_id: ROLE_ID, secret_id: SECRET_ID });

const { data } = await vault.read('secret/data/myapp');
const dbPassword = data.data.DB_PASSWORD;

Comparison:

Tool	Best For
HashiCorp Vault	Self-hosted, complex access policies, dynamic secrets
AWS Secrets Manager	AWS-native workloads, automatic rotation
Azure Key Vault	Azure workloads
GCP Secret Manager	GCP workloads
Doppler / Infisical	Developer-friendly, cloud-agnostic

Preventing Accidental Secret Leaks

Pre-commit hooks to catch secrets before they land:

# Install git-secrets or gitleaks
brew install gitleaks

# Run in CI
gitleaks detect --source . --verbose

# Or add as a pre-commit hook
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
gitleaks protect --staged -v
if [ $? -ne 0 ]; then
  echo "⛔ Secrets detected. Commit blocked."
  exit 1
fi
EOF
chmod +x .git/hooks/pre-commit

If you've already committed a secret:

Rotate the secret immediately — assume it's compromised
Remove it from history with git filter-repo (not git filter-branch)
Force-push all affected branches
Audit access logs for the exposed secret

# Remove a secret from git history
pip install git-filter-repo
git filter-repo --path-glob '*.env' --invert-paths

The most important rule: rotating the secret is always the first step. Git history cleanup comes second. Don't spend 30 minutes cleaning history while the leaked key is still active.

Secret Rotation

Secrets should be rotated regularly and always immediately on suspected compromise.

// Implement rotation-aware config loading
class SecretManager {
  constructor() {
    this.cache = new Map();
    this.ttl = 15 * 60 * 1000; // Refresh every 15 minutes
  }

  async get(secretName) {
    const cached = this.cache.get(secretName);
    if (cached && Date.now() - cached.fetchedAt < this.ttl) {
      return cached.value;
    }

    const value = await fetchFromSecretsManager(secretName);
    this.cache.set(secretName, { value, fetchedAt: Date.now() });
    return value;
  }
}

Automated rotation for database credentials is supported natively in AWS Secrets Manager and Vault. For JWT secrets, coordinate rotation: support both old and new secrets for a brief transition window.

Putting It All Together: A Security Checklist

Authentication & Sessions

[ ] Passwords hashed with bcrypt, Argon2, or scrypt (never MD5/SHA alone)
[ ] Session ID rotated on login (prevents fixation)
[ ] All cookies set with HttpOnly, Secure, SameSite
[ ] JWTs: algorithm pinned, payload contains no sensitive data, expiry is short
[ ] Refresh tokens rotated on every use
[ ] Token revocation implemented for logout and credential changes

Authorization

[ ] Every endpoint explicitly defines who can access it
[ ] Resource-level ownership verified server-side (not just authenticated)
[ ] Admin/privileged routes protected with role middleware
[ ] Authorization covered by integration tests

API Security

[ ] Rate limiting on all public endpoints (stricter on auth routes)
[ ] API keys hashed in storage, scoped, and expirable
[ ] OAuth using Authorization Code + PKCE (not Implicit Flow)
[ ] GraphQL introspection disabled in production
[ ] Service-to-service calls authenticated
[ ] Request signing on high-security internal APIs

Supply Chain

[ ] package-lock.json / yarn.lock committed and used in CI (npm ci)
[ ] npm audit run in CI pipeline
[ ] New dependencies reviewed before installation
[ ] SRI hashes on CDN-loaded scripts
[ ] GitHub Actions (and equivalent) pinned to commit SHAs
[ ] gitleaks or equivalent scanning in pre-commit hook and CI

Secrets Management

[ ] No secrets in source code, config files, or environment variable logs
[ ] .env in .gitignore, no .env files in git history
[ ] Production secrets stored in a secrets manager (Vault, AWS SM, etc.)
[ ] Secret rotation policy defined and automated where possible
[ ] Pre-commit hooks scanning for accidentally added secrets

Conclusion

Security isn't a feature you add at the end. It's a discipline you build into every layer of what you ship.

The checklist above is not a one-time exercise — it's a review you run before every significant release, and a habit you build into every PR.

The teams that get this right aren't necessarily the ones with dedicated security engineers. They're the ones where every developer treats security as part of their job description.

That's the only way it works at scale.

Something missing from this guide? A pattern you've seen exploited in production? Drop it in the comments — this document should keep evolving.

Frontend System Design Interviews: What They Are, Why They Matter, and How to Actually Prepare

Olawale Afuye — Mon, 01 Jun 2026 23:16:07 +0000

For mid-to-senior developers who want to stop winging it.

If you've been a developer for 3+ years and you're targeting senior roles, there's a round in your upcoming interviews that will either make or break your offer — and it's not the LeetCode round.

It's the Frontend System Design interview.

Most developers underprepare for it. Some don't even know it exists until they're sitting in one. And if you're in the "I'll figure it out in the room" camp — this post is especially for you.

Why Frontend System Design Has Never Mattered More

The market has shifted. AI is eating junior-level work. Entry-level roles are shrinking. Companies are hiring fewer engineers but expecting each one to do more, own more, and think more.

In that environment, the ability to design systems — not just implement them — is the difference between being hireable at a senior level and being stuck in the mid-level ceiling.

But here's what most developers miss: this isn't just interview prep.

These skills are what you use every day as a senior engineer. Architectural discussions with your team. Scaling decisions. Choosing between a short-term fix and a long-term design. Knowing when to push back on a product requirement because the technical cost is non-trivial. These all require system design thinking.

The interview is just the most visible moment where that thinking gets tested. Your actual career depends on having it long after the interview ends.

What These Interviews Are Actually Testing

Here's a misconception worth killing early: frontend system design interviews are not asking you to design a backend.

You're not drawing distributed databases and microservice meshes. You're being evaluated on your ability to architect a complex frontend application at scale — the kind where real decisions need to be made about state management, rendering strategies, component architecture, API contracts, performance budgets, and more.

The interviewer is watching for two things simultaneously:

High-level thinking — Can you zoom out, see the full picture, and make structured decisions before touching any implementation detail?
Domain expertise — Do you actually know why certain choices are better, not just that they exist?

The balance between these two is what separates strong candidates from exceptional ones. Go too broad and you sound shallow. Go too deep too fast and you miss the architecture entirely.

The Five Steps That Structure Every Strong Answer

Regardless of what you're asked to design — a Figma clone, Google Calendar, a real-time collaborative editor — a strong response follows this structure:

1. Collect Requirements

Before drawing a single box or naming a single component, ask questions.

What are the functional requirements? What does this system need to do? Clarify features, user flows, and scope boundaries.

What are the non-functional requirements? Does it need to work offline? How many concurrent users? Is performance the priority or is it accessibility? Are there latency constraints?

This step signals to the interviewer that you don't just build — you think before you build. In real life, this is the difference between an engineer who delivers and one who delivers the wrong thing efficiently.

2. High-Level Architecture

Now sketch the structural blueprint. What are the major components of the system? How do they relate to each other? Don't code anything. Don't go deep on implementation. You're drawing the map, not building the roads.

This is where most developers stumble — they jump straight to how (React with Redux, I'll use a custom hook here...) before establishing what the system looks like from above.

3. Data Modeling

Define the key entities in your system and their relationships. What does a User object look like? What about an Event or a Document? How does data flow between components? What lives on the server vs. what lives on the client?

Data modeling in frontend isn't just a backend concern. How you model data shapes your component tree, your state management decisions, and your API contracts.

4. API Design

Plan how the different pieces communicate. What does the frontend request from the backend, and in what shape? REST vs. GraphQL vs. WebSockets — and why for this specific case?

This is also where you surface your thinking on caching, optimistic updates, and error states. These aren't afterthoughts in production systems; they're first-class design decisions.

5. Additional Considerations

This is where strong candidates separate themselves. After covering the core design, address the cross-cutting concerns:

Performance — Code splitting, lazy loading, rendering strategy (SSR, SSG, CSR), image optimization
Accessibility — ARIA roles, keyboard navigation, focus management
Offline strategy — Service workers, local caching, sync on reconnect
Security — XSS, CSRF, content policies
Observability — Error tracking, performance monitoring, logging

You won't cover all of these in every interview. But naming them, and discussing trade-offs intelligently, shows that you think in systems — not just features.

The Signals Interviewers Are Actually Scoring

Green flags (what to do)

Ask clarifying questions before proposing solutions
Explicitly call out trade-offs when making a choice ("I'm choosing client-side rendering here because the data is user-specific, which reduces caching benefit — the trade-off is initial load time, which we can address with a skeleton UI")
Adapt your design when the interviewer introduces new constraints — this is intentional, and how you respond tells them more than your original design did
Show range: breadth at the architecture level, depth when you drill into your area of expertise

Red flags (what to avoid)

Jumping into design without asking a single question
Claiming expertise in areas where you'd clearly be guessing
Ignoring trade-offs — every decision has a cost, and pretending otherwise raises doubts
Getting stuck in implementation details before the architecture is clear

The interview is less about getting the "right" answer and more about demonstrating how you think. Two candidates can design the same system and get very different scores based on how they communicate their reasoning.

How to Prepare Without Wasting Your Time

The single most effective preparation method is designing real applications from scratch.

Pick something you know well as a user — Figma, Notion, Google Calendar, Twitter, Spotify. Open a blank document or whiteboard tool. Set a 45-minute timer. Design it.

Go through all five steps. Force yourself to make decisions. Write down your trade-offs. When the timer ends, review: where did you get stuck? What did you skip? What would you do differently?

Do this weekly with different systems and you'll build pattern recognition faster than reading any article (including this one).

The T-Shaped Knowledge Target

Aim to be T-shaped: broad enough to have an opinion on most frontend concerns (performance, state management, rendering, API design, accessibility), and deep enough in at least one area to go several levels down when pressed.

If your deep area is performance optimization — know it well enough to talk about browser rendering pipeline, CLS, LCP, paint budgets, and real-world measurement. If it's accessibility — know WCAG 2.1 levels, not just "use alt text."

Your depth is your credibility anchor. Your breadth is what keeps the conversation flowing.

The Stakes Are Higher Than You Think

This round has an outsized impact on how your offer comes out.

Not just whether you get an offer — but what level and what salary the offer reflects. For senior and staff-level roles especially, strong system design performance can move you up a level. Weak performance can move you down one, or out entirely.

That means the ROI on preparing for this round is higher than almost anything else you can do in your job search. One well-prepared system design interview can be worth tens of thousands of dollars in total compensation.

Treat it accordingly.

A Last Word

The developers who rise to senior, staff, and principal levels aren't always the ones who can implement the fastest. They're the ones who can design well, communicate trade-offs clearly, and bring structure to ambiguity.

Frontend system design interviews exist precisely to test that. And the good news is: unlike algorithmic puzzle rounds, this one rewards real engineering experience when you've taken the time to develop the vocabulary for it.

Start building that vocabulary now. Design something this weekend. See where you get stuck.

That's where the real preparation begins.

Have you gone through a frontend system design interview recently? What was the most unexpected thing you were asked to design? Drop it in the comments.

Your Deleted Google API Key Is Still Working — Here's Why That's a Security Crisis

Olawale Afuye — Mon, 01 Jun 2026 22:28:21 +0000

You just discovered your Google API key was leaked. Maybe it showed up in a GitHub search. Maybe a secret scanner flagged it. You panic, open the Google Cloud Console, and delete it.

Done. Crisis averted.

Except it isn't.

That key is still working. For the next 23 minutes, an attacker can keep using it — making requests, racking up your cloud bill, or accessing data you thought was already cut off.

This is not a theoretical risk. It's a documented vulnerability that Google initially dismissed as "expected behavior" — before later upgrading it to a P0/S0 critical bug.

Let's break down exactly what happened, why it matters, and what you actually need to do to protect yourself.

The Old Mental Model: "Google API Keys Are Low-Risk"

For years, Google told developers something that made sense at the time: it's fine to include certain API keys in your frontend code. These were project identifiers — used for services like Google Maps — not secrets. They were designed to be public.

So developers did exactly what they were told. They hardcoded these keys into JavaScript, committed them to GitHub, and shipped them in client bundles. It became standard practice.

Secret scanners now crawl the internet and find thousands of these keys in public repositories every day.

Here's where things changed.

Gemini Changed Everything

When Google launched Gemini, they plugged it into the same API key infrastructure that Maps had always used. Same key type. Same issuance flow. Same developer habits.

But the blast radius of a leaked Gemini key is completely different from a leaked Maps key.

A leaked Maps key might let someone make geocoding requests on your bill. Annoying, costly, but bounded.

A leaked Gemini key could:

Let attackers send unlimited LLM requests billed to your account
Potentially expose cached or uploaded data associated with your project
Serve as a foothold for broader GCP exploration depending on project permissions

The keys hadn't changed. The services they unlocked had.

The 23-Minute Window: What "Eventual Consistency" Actually Means in Practice

Security researcher Joe Leon discovered something that breaks the most basic assumption in incident response: deleting a key does not immediately stop it from working.

Here's the technical reason.

Google Cloud Platform runs on a globally distributed architecture. When you delete an API key, that deletion is written to a primary data store. But the authentication layer — the servers that actually validate incoming requests — relies on cached copies of credential data distributed across many nodes.

These caches are updated asynchronously. The system prioritizes availability and performance. The trade-off is that credential state changes don't propagate instantly.

The result: deleted keys remain valid for up to 23 minutes while those caches catch up.

This is called eventual consistency — a well-understood distributed systems pattern. The problem is that security teams don't think about it, because their mental model is:

Delete key → key is dead → attacker is locked out.

That model is wrong.

Google's Initial Response (And Why Disclosure Still Mattered)

When Joe Leon initially reported this, Google's response was to classify it as expected behavior — not a bug. The reasoning: eventual consistency is a known property of distributed systems, and this is documented.

This is technically true. It is also deeply insufficient.

The issue isn't whether eventual consistency is expected internally at Google. The issue is that it breaks the mental model of every developer and incident responder who presses the delete button and assumes the key is dead.

After Leon went public with the research, something shifted. The right people inside Google saw it. It got reclassified as a critical P0/S0 bug and escalated for remediation.

This is why responsible public disclosure matters — not to embarrass vendors, but because internal triage teams don't always have the context to understand the real-world security implications of architectural choices. Sometimes the right path through the bureaucracy is around it.

Does This Apply to AWS and Other Services?

Yes — though the specifics vary.

AWS (Amazon Web Services)
IAM access key deletions generally propagate quickly, but AWS also has edge cases:

There is a small propagation window (~4 seconds) for new requests
More critically: deleting an IAM key does not invalidate active STS sessions generated from that key. Those temporary credentials remain valid until they naturally expire — which could be hours

Postmark
Postmark is a positive outlier here. Their API tokens are designed for immediate invalidation — delete it and it's dead, no propagation delay.

The General Rule
Treat revocation as an asynchronous event, not a synchronous kill switch. The safer assumption during any security incident is: the key may still work for some window of time after deletion.

What This Means for Incident Response

Stop treating "I deleted the key" as the end of your response process.

Here's a more realistic incident response mindset:

1. Delete the key — but don't stop there.
Deletion starts the clock. It doesn't end the threat.

2. Rotate associated credentials immediately.
Any service accounts, tokens, or roles that interacted with the compromised key should be rotated. Don't assume one deletion closes the blast radius.

3. Invalidate active sessions explicitly.
On AWS, this means finding and revoking STS tokens derived from the key. On GCP, check for active OAuth sessions or service account tokens linked to the same project.

4. Monitor logs for the next 30 minutes.
After deletion, watch your Cloud Logging or CloudTrail for continued usage of the revoked key. If you're still seeing hits, your window is still open.

5. Apply emergency access restrictions.
If the situation is critical, use Service Control Policies (AWS) or VPC Service Controls (GCP) to add a hard constraint while the deletion propagates.

How to Properly Manage API Keys in a Frontend App

Since frontend code is inherently visible to users, the goal isn't to perfectly hide keys — it's to minimize what a leaked key can do.

1. Apply Strict Restrictions in the Cloud Console

Never deploy an unrestricted key. In GCP under APIs & Services > Credentials, configure:

Application Restrictions: Lock the key to specific HTTP referrers (e.g., https://yourdomain.com/*). A key stolen from your bundle becomes nearly useless when called from any other origin.
API Restrictions: Limit the key to only the APIs your app actually uses. A Maps key should not be able to touch Gemini. Full stop.

2. Never Hardcode Keys in Source Code

Use environment variables and keep them out of version control:

# .env.local
NEXT_PUBLIC_MAPS_API_KEY=your_key_here

# .gitignore
.env
.env.local
.env*.local

This doesn't prevent exposure — bundlers often inline these values — but it keeps secrets out of your git history and prevents accidental repository exposure.

3. Use a Backend Proxy for Sensitive APIs

If you need Gemini, billing-sensitive APIs, or anything with significant data access: don't call it from the frontend.

Route requests through your own backend:

Browser → Your API Server (holds the key securely) → Google Cloud API

Your backend can authenticate the user, apply rate limits, and call Google — without ever exposing the key to the client. This is the only approach that genuinely protects high-sensitivity credentials.

4. Set Up Billing Alerts

A leaked key often shows up as an unexpected spike in usage before anything else. Configure budget alerts in Google Cloud Billing so you get a notification the moment something looks abnormal. This is your earliest warning system.

5. Rotate Keys on a Schedule

Don't wait for a breach to rotate. Build key rotation into your regular ops cycle — monthly for sensitive services, quarterly at minimum for everything else. The shorter the valid lifetime of any given key, the smaller the window of any eventual compromise.

The Broader Lesson

The real vulnerability here isn't a code bug. It's a broken assumption baked into how developers and security teams think about credentials.

Google told developers keys were safe to expose. Developers believed them. Google's distributed architecture made deletion asynchronous. Nobody documented that clearly for incident responders. A researcher had to find it, get rejected, go public, and escalate through media coverage before the right people inside a trillion-dollar company treated it as critical.

That's a systems failure — technical, organizational, and communicative.

For you as a developer, the takeaway is straightforward:

Restrictions > secrecy for frontend keys
Backend proxies > frontend exposure for sensitive APIs
Revocation is asynchronous — respond accordingly
Monitoring doesn't stop when you press delete

The delete button is not a kill switch. Build your security posture around that fact.

Cryptographic Failures: The Silent Killer in Your Codebase (OWASP #2)

Olawale Afuye — Sat, 30 May 2026 14:55:32 +0000

You ship a feature. Tests pass. Deployment goes smooth. Everyone's happy.

Meanwhile, somewhere in your codebase, you're storing passwords with MD5.

And someone, right now, is cracking them in under a second.

That's the thing about Cryptographic Failures — they don't throw errors. They don't break your CI pipeline. They sit quietly in production until the day they don't.

What Are Cryptographic Failures?

OWASP ranks them #2 on the Top 10 list of most critical web application vulnerabilities. Not #7. Not #5. Number two.

And the definition is deceptively simple: sensitive data is not protected by cryptography — or it's protected badly.

That second part is where most developers get caught. It's not that they skipped encryption entirely. It's that they used the wrong algorithm, mismanaged their keys, or trusted a default that hasn't been safe since 2004.

The result is the same either way: unauthorized access to data that was supposed to be locked.

Why Does This Keep Happening?

Three root causes show up again and again.

1. Weak or Deprecated Algorithms

Not all encryption is created equal. Some algorithms that were considered secure a decade ago are now trivially breakable with modern hardware.

The most common offender? MD5 for password hashing.

MD5 was never designed for passwords — it was designed for fast checksums. "Fast" is the exact opposite of what you want when hashing credentials. Fast means attackers can run billions of attempts per second against a leaked hash database.

Here's a concrete example of what not to do:

import hashlib

# ❌ Never do this
password = "mypassword123"
hashed = hashlib.md5(password.encode()).hexdigest()

This is what led to the Freecycle data breach — a real-world incident where attackers accessed user credentials because the platform was using MD5. Once they had the hash database, cracking the passwords wasn't a challenge. It was a formality.

What you should use instead:

import bcrypt

# ✅ Use a proper password hashing algorithm
password = b"mypassword123"
hashed = bcrypt.hashpw(password, bcrypt.gensalt())

bcrypt, argon2, and scrypt are slow by design. That's the point.

The same principle applies to transport protocols. SSL 2.0, SSL 3.0, and TLS 1.0/1.1 are all deprecated. If your server still supports them, you're offering attackers a downgrade path. TLS 1.3 is the minimum you should be running.

2. Poor Key Management

You encrypted the data.

You stored the encryption key in the same repository.

Congratulations — you haven't protected anything.

This is embarrassingly common. Developers hardcode secrets directly into source code, commit them to GitHub (sometimes public repos), and ship them to production. The encryption is technically present. The protection is not.

// ❌ This is not secret management
const SECRET_KEY = "my_super_secret_key_123";

The fix isn't complicated, but it requires intentionality:

// ✅ Pull from environment or a secrets manager
const SECRET_KEY = process.env.SECRET_KEY;

Better yet, don't use environment variables for truly sensitive credentials. Use a dedicated secrets manager — AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager. These give you rotation, access control, audit logs, and a single source of truth for your credentials.

3. No Encryption at Rest or in Transit

Some applications simply don't encrypt sensitive data at all.

No HTTPS enforcement. Database fields stored in plaintext. PII sitting in logs. Backups with no encryption.

If your threat model includes "what happens when our database is dumped?" — and it should — plaintext storage is catastrophic. Encryption at rest means that even if an attacker exfiltrates your data, they get ciphertext, not customer information.

How to Prevent Cryptographic Failures

Prevention isn't one thing. It's a stack of practices.

Use Modern, Appropriate Algorithms

Use Case	Recommended	Avoid
Password hashing	`bcrypt`, `argon2`, `scrypt`	`MD5`, `SHA-1`, plain `SHA-256`
Data encryption	`AES-256-GCM`	`DES`, `3DES`, `RC4`
Transport security	`TLS 1.3`	`SSL`, `TLS 1.0/1.1`
Key exchange	`ECDH`, `RSA-2048+`	Anything < 2048-bit RSA

Manage Secrets Properly

Never hardcode credentials in source code
Never commit .env files to version control (add to .gitignore immediately)
Use a secrets manager for production
Rotate keys regularly and have a rotation plan
Enforce least-privilege access to secrets

Bake Security Into Your Dev Workflow

The best time to catch cryptographic failures is before they reach production. These four tool categories should be in your pipeline:

SAST — Static Application Security Testing

Scans your source code without running it. Catches hardcoded secrets, insecure function calls, and deprecated API usage.

→ Try: Bandit (Python), Semgrep

DAST — Dynamic Application Security Testing

Tests your running application. Finds vulnerabilities that only appear at runtime — weak cipher suites, missing security headers, misconfigured TLS.

→ Try: OWASP ZAP

Secrets Detection

Scans your git history and codebase for leaked credentials, API keys, and tokens. Because git push is forever.

→ Try: GitLeaks, TruffleHog

SCA — Software Composition Analysis

Audits your dependencies for known vulnerabilities. That npm package you haven't updated in two years? It might be pulling in an insecure crypto library.

→ Try: Trivy, npm audit, Snyk

A Practical Checklist

Before you ship, run through these:

[ ] Are passwords hashed with bcrypt, argon2, or scrypt?
[ ] Is TLS 1.3 enforced? SSL and early TLS versions disabled?
[ ] Are encryption keys stored outside the codebase?
[ ] Is sensitive data encrypted at rest in the database?
[ ] Are .env files in .gitignore?
[ ] Is SAST running in your CI pipeline?
[ ] Have you scanned your git history for leaked secrets?
[ ] Are dependencies up to date and audited?

The Uncomfortable Truth

Most cryptographic failures aren't the result of developers not caring.

They're the result of developers moving fast, inheriting legacy code, or trusting defaults that were never safe to trust. MD5 was everywhere in tutorials for years. Hardcoded secrets are convenient. "We'll fix it later" is a sentence said in every company on earth.

The problem is that "later" sometimes arrives looking like a breach notification email.

Cryptography isn't optional infrastructure. It's the floor. Everything else you build sits on top of it — and if the floor is weak, it doesn't matter how good the rest of the building is.

DEV Community: Olawale Afuye

Your Ticket Was Closed. The User Still Couldn't Pay.

The Ticket-Thinker vs. The System-Owner

Real Scenario 1: The Payment That Worked and Failed at the Same Time

Real Scenario 2: The Smart Device That "Works"

Why Engineers Stay Stuck in the Ticket Mindset

What Cross-Functional Reliability Actually Looks Like

The Mental Model Shift

Your Codebase Is a Mess Because Your Team Can't Agree on What a "Customer" Is

The Core Problem: One Model Trying to Mean Everything

Strategic Design: Understand the Problem Before You Touch Code

Bounded Contexts: Where the Model Gets Its Boundaries

Context Mapping: Managing the Lines Between Contexts

Tactical Design: Building the Model in Code

When DDD Is Worth It (And When It Isn't)

The One Thing to Take Away

Your Microservices Are Not Resilient. Your Architecture Is the Real Problem

1. Stop Building Distributed Monoliths

2. The Bulkhead Pattern: Give Failures Nowhere to Go

3. Timeouts Are Not Optional

4. Circuit Breakers: Fail Fast, Recover Clean

5. Throttling: Protect Your Critical Flows

6. Go Asynchronous. Seriously.

7. Embrace Eventual Consistency (Or Suffer the Alternative)

8. Event Sourcing: When History Becomes Infrastructure

9. The Robustness Principle: Be Strict in What You Send, Tolerant in What You Accept

10. Observability Is Not Optional. It's How You Know Anything.

The Real Problem Is Culture, Not Code

Quick-Reference: Resilience Pattern Checklist

Your Security Scanner Found 7 Missing Headers. Don't Fix Them Blindly.

First, The Contrarian Take

#1. HTTP Strict Transport Security (HSTS)

#2. Content Security Policy (CSP)

#3. X-Content-Type-Options

#4. X-Frame-Options

#5. Cache-Control

#6. CORS (Access-Control-Allow-Origin)

#7. Permissions-Policy

The Referrer-Policy Bonus

How to Actually Ship These

The Summary That Actually Matters

The Gateway Layer Explained: Reverse Proxies, Load Balancers, and API Gateways — Once and For All

The Gateway Layer: What It Actually Is

1. Reverse Proxy — The First Line of Defense

What it is

What it actually does

When is a reverse proxy sufficient?

2. Load Balancer — Scaling Horizontal

What it is

Traffic Distribution Algorithms

Health Checks and Failover

Layer 4 vs Layer 7: Choosing the Right Level

3. API Gateway — Taming Microservices

The Problem It Solves

What an API Gateway Does

What an API Gateway Costs You

Why the Lines Blur (And Why That's Fine)

What a Modern Production Architecture Actually Looks Like

If You Work in Kubernetes

Security Considerations Across Every Layer

Decision Framework: What Do You Actually Need?

Key Takeaways

OpenTelemetry for Node.js Developers: A Practical Guide to Observability in Distributed Systems

The Problem No Dashboard Will Tell You About

What Is Observability (And Why "Monitoring" Isn't Enough)

A Brief History: Why OpenTelemetry Exists

The Four Concepts You Must Understand Before Writing Code

1. Spans — The Atomic Unit of a Trace

2. Span Context and Correlation Context

3. Metrics vs. Traces, When to Use Which

4. The OpenTelemetry Collector — Your Telemetry Router

Setting Up Distributed Tracing in Node.js

Step 1: Install Dependencies

Step 2: Create Your tracing.js File

Step 3: Load It Before Your App Starts

Step 4: Run the OTel Collector + Zipkin via Docker

Setting Up Metrics Collection with Prometheus

Step 1: Install Dependencies

Step 2: Initialize Your Meter, Counter, and Histogram

Step 3: Instrument Your Express Middleware

Step 2: Create Your `tracing.js` File

`otel-config.yaml` (production)

`docker-compose.yml` (partial, production)