DEV Community: Abdul Wahab

gRPC: how a local call becomes a network request

Abdul Wahab — Tue, 03 Nov 2026 20:42:31 +0000

Suppose you run a ride-hailing app with five services: trip, dispatch, location, pricing, and payments.

The trip service owns each ride. Dispatch matches riders to nearby drivers. Location tracks driver positions.

Pricing quotes the fare. Payments charges the card.

When a rider taps Find a ride, the trip service calls dispatch to find a driver. The first version of that call is hand-written JSON. Build a URL, encode JSON, set a content type, parse the response, and repeat that code in every service that calls dispatch.

Then the dispatch service renames driver_id to driver_ref. The trip service still compiles. At runtime the field is missing, dispatch checks the field, rejects the request, and the ride fails.

gRPC starts from the contract instead of the HTTP call. You define the service and its messages in one file. From that file, a compiler generates the client and server code, and the generated client makes the call look like a local method.

The gRPC introduction describes this contract-first workflow.

A call that crosses the network

gRPC is a remote procedure call system. The client invokes a method on a stub, a generated local object that stands in for the remote service. The stub serializes the request, sends it to the server, waits for the reply, and returns the result to the caller.

The method name and the message types come from a service definition. The server implements that definition. The client never hand-writes the URL or the JSON body.

Suppose the trip service calls FindDriver with pickup_zone = "airport". The call site reads like a function call.

The network is still between the two processes. The stub hides how the message is encoded and sent. It does not hide the latency, the failure, or the version of the contract on the other side.

The .proto file is the contract

A gRPC service lives in a .proto file. The file defines messages, which are typed records, and a service, which lists the methods that can be called.

service Dispatch {
  rpc FindDriver (RideRequest) returns (DriverReply);
  rpc WatchDrivers (AreaRequest) returns (stream DriverLocation);
  rpc ReportRoute (stream RoutePoint) returns (RouteSummary);
  rpc TrackTrip (stream TripEvent) returns (stream TripUpdate);
}

message RideRequest {
  string pickup_zone = 1;
  string rider_id = 2;
}

message DriverReply {
  string driver_id = 1;
  int32 eta_seconds = 2;
}

Each message field has a name and a number. The number is the field's identity on the wire, and the binary encoding uses the number, not the name.

Changing the name can break generated source code or a JSON client, but it does not change the field number. Reusing a number for a different field is not safe, which is why deleted numbers should be reserved (protobuf overview).

The service block is the contract the compiler consumes. It says FindDriver takes a RideRequest and returns a DriverReply, and the other three methods stream. It does not say which programming language either side uses.

The compiler writes the plumbing

protoc reads the .proto file. A gRPC plugin adds the client stub and server skeleton to the normal message classes.

The client gets a stub with a FindDriver method. The server gets an interface it must implement. The generated code serializes the request, dispatches the method, and makes the transport call.

A .proto file generates a client stub and a server skeleton.

This is a build step, not a runtime library you can skip. Change the .proto file and you regenerate both sides. That keeps the contract honest, and it adds a toolchain every contributor must run.

What the stub sends

Suppose the client calls FindDriver with pickup_zone = "airport". The stub produces a request message and encodes it with protobuf, a schema-based binary format. It opens or reuses an HTTP/2 stream, a multiplexed connection that carries several calls at once.

The request carries the method path, the encoded message, and optional metadata, key-value headers attached to the call, such as an authorization token. HTTP/2 multiplexes several calls over one connection. The client does not need a new TCP connection for every request (RFC 9113).

The server decodes the message, dispatches to FindDriver, and sends back an encoded reply with a gRPC status. A non-OK status is not an HTTP 500. It is a gRPC status such as NOT_FOUND or DEADLINE_EXCEEDED. The status travels in the response trailers, the headers sent after the response body (gRPC status codes).

The client can set a deadline. When the deadline passes, the call ends with DEADLINE_EXCEEDED, even if the server is still working. The server can query the remaining time. It can stop work that no client is waiting for (gRPC core concepts).

Four ways to stream

A unary call sends one request and gets one reply. That is the shape most people mean by RPC.

gRPC also defines three streaming shapes:

Server streaming: one request, then a stream of replies. WatchDrivers uses this shape to send live driver locations.
Client streaming: a stream of requests, then one reply. ReportRoute uses this shape to send GPS points and receive one route summary.
Bidirectional streaming: both sides send independent streams. TrackTrip uses this shape to exchange trip events and status updates.

The shapes matter because the method signature carries the direction. returns (stream ItemReply) is a server stream. (stream ItemRequest) is a client stream. Both sides are visible in the .proto file before any code is generated.

The four call shapes differ in which side sends one message and which side sends a stream.

Streaming fits a live driver feed, a GPS upload, or a trip-status channel. It does not make a slow query faster. The same database work still happens behind the stream.

What the contract buys

First, the client is typed. The compiler catches a wrong argument type before the request leaves the process. A hand-written JSON client catches that at runtime, if it catches it at all. The same .proto file can generate a Go client and a Java server (gRPC introduction).

The second gain is one contract across languages. A Go client and a Java server both generate from the same file. Neither team hand-writes a serializer.

The third gain is streaming without a second protocol. The four call shapes use the same service definition and the same generated code path.

Fourth, a deadline travels with the call. The client sets it, the server sees it, and a cancelled call can stop work instead of finishing for a client that has gone.

None of this makes gRPC automatically faster than JSON over HTTP. The network can dominate a small call. Measure the call path before assuming the binary format is the bottleneck.

What it costs

Every contributor must run a build step for the contract. A new contributor needs protoc, the language plugin, and a generated-code step in CI. A stale generated file can compile while disagreeing with the service, so CI should regenerate it.

When a message is binary, curl shows bytes, not a readable JSON object. You need a decoder, a proxy, or a logging path that understands protobuf.

Browsers cannot call a normal gRPC service directly. You need gRPC-Web and a proxy, or a JSON gateway (gRPC-Web). That is extra infrastructure for a public web client.

HTTP/2 changes connection behavior. A long-lived connection can pin a client to one backend. The balancer then has to balance requests or streams, not only connections. A layer 4 balancer, which balances whole connections, can leave one server busy while another idles (gRPC load balancing).

When you change the schema, take care. Adding a field is usually safe. Renaming a field, changing its type, or reusing a field number can break old clients (protobuf update guide). The compiler checks types, not whether old clients can read the new field.

When JSON is the better fit

Use a JSON API when the client is a browser you do not control. A browser needs gRPC-Web, a browser-compatible variant, plus a proxy. It can also use a gateway that translates between gRPC and JSON (gRPC-Web). Use JSON when the API is public, or when the payload is small and the team values readable requests over generated code.

Use gRPC when both sides are services you build. Use it when the contract is shared across languages, when you need streaming, or when a deadline and typed errors remove repeated plumbing.

You can keep a public JSON edge and call internal gRPC services. The edge translates once. The internal services get the typed contract.

Takeaway

gRPC replaces hand-written request plumbing with a typed contract and generated stubs; the contract and streaming are the gain, while code generation, binary payloads, and HTTP/2 are the cost.

Common questions

Is gRPC faster than REST?

Not by definition. It often uses a compact binary format and one multiplexed connection, but latency, payload size, connection reuse, and server work decide the result.

Does gRPC replace HTTP?

No. It uses HTTP/2 as its transport (RFC 9113). The method call is mapped onto an HTTP/2 request and response, with gRPC status in the trailers.

Can a browser call gRPC directly?

Not a normal gRPC service. Browsers need gRPC-Web plus a proxy, or a JSON gateway.

What is protobuf?

A schema language and binary encoding for typed messages. gRPC uses it by default for the service definition and the payload.

Do I need HTTP/2?

gRPC is designed around HTTP/2 features such as multiplexed streams. A proxy or gateway may expose a different protocol at the edge.

How do errors work?

The server returns a gRPC status code and message in the trailers. A deadline or cancellation ends the call even if the server would have succeeded later. If either side ignores the deadline, work continues for a client that has already given up.

Sources

Originally published at awahab.dev.

Load balancing: what a load balancer does and costs

Abdul Wahab — Mon, 02 Nov 2026 20:07:45 +0000

Suppose one of three order servers runs out of database connections on a Friday afternoon. It still accepts connections.

A load balancer sits in front of the three and hands each request to one of them. It keeps choosing the broken server. Every order that lands there fails.

The load balancer gave clients one address. It did not remove the failing server. An address alone is not enough if the balancer keeps using a broken server.

One address in front of many servers

A load balancer sits in front of a group of servers. That group is the pool. A client sends every request to the balancer's address, and the balancer forwards the request to one server in the pool.

This role has a name in the HTTP specification: a gateway. A gateway is also called a reverse proxy, which is a server that forwards requests to other servers. RFC 9110 section 3.7 says gateways "enable partitioning or load balancing of HTTP services across multiple machines." The same section calls a gateway a reverse proxy (RFC 9110, section 3.7).

Suppose the service has servers at 10.0.0.1:8080, 10.0.0.2:8080, and 10.0.0.3:8080. Clients use one name, api.example.com. The balancer owns that name and sends each request to one of the three addresses.

Clients send every request to one address. The load balancer chooses a server from the pool.

The client does not need to know how many servers exist, what their addresses are, or which one is busy. It needs the balancer's address.

Without one, every client keeps the list

Suppose one API has three servers and three client apps. Without a load balancer, every app needs the list of three addresses. Each app must pick one, notice when one dies, and try another. That logic then lives in every client you ship.

You must change the address list in every client when you add a server, or leave the new server idle. You must change the list again when you remove a server to maintain it.

A load balancer chooses in one place. Clients get one address. You can add, remove, or replace servers behind it.

One address, more room, fewer visible failures

Suppose traffic doubles. You add a server and tell the balancer. Clients keep the same address, so no client changes.

Suppose a server dies. The balancer stops sending it traffic. The next request goes to another server, and the client may not notice.

Suppose you need to upgrade a server. Take it out of the pool, upgrade it, and put it back. The other servers keep serving.

The cost is that the balancer now sits in the request path. The rest of the post explains what that means.

How the request moves

A request for /images/logo.png arrives at the balancer at 203.0.113.10:443. The balancer picks 10.0.0.2:8080 and forwards the request. The response comes back through the balancer.

The balancer can choose once for a connection, or once for every request. Layer 4 and layer 7 differ in what they can see. Layer 4 is the transport layer, where connections live. Layer 7 is the application layer, where HTTP requests live.

What layer 4 sees, and what layer 7 sees

A layer 4 load balancer works with connections. It sees IP addresses and ports on TCP (Transmission Control Protocol) connections (RFC 9293). It does not see HTTP paths or headers. It chooses a server when a connection opens and sends every request on that connection to the same server.

Layer 4 does not parse the request, so it can forward protocols it does not understand. It is also blind to the request. If one request is a slow search and the next is a quick order, the balancer cannot move the quick one to another server. Both sit on the same connection.

A layer 7 load balancer reads the request. It can see the method, path, and headers. That means it can choose a server for each request, and it can send /images to one pool and /checkout to another.

Layer 7 also depends on the protocol. It is useful for HTTP and less useful for a protocol it cannot parse.

At layer 4, one connection stays on one server. At layer 7, the balancer can choose a server for each request.

Unequal requests break round robin

The simplest method is round robin. Server A gets the first request, B gets the second, C gets the third, then A again. Round robin keeps one counter for the next turn. It does not use request cost.

Round robin gives each server one turn. The rule has no input for request cost. One request may finish quickly, while another holds its server for much longer. Equal turns then become unequal work.

Suppose two slow searches and one quick order arrive together. Round robin sends the first search to A, the order to B, and the second search to A. A stays busy while B finishes and sits idle.

Round robin sends the second search to the busy instance and leaves it working until 1.8 seconds. Least connections sends it to the idle instance.

Round robin has one weakness: it ignores active work. This method sends the next request to the server with the fewest active connections. A busy server gets skipped until it catches up.

Least connections looks at the number of active connections. It does not use request cost. A slow request holds one connection for a long time. A fast request holds one for a short time.

A shallow check can miss a failed order

A load balancer needs to know which servers can take traffic. It can send a small request to each server on a schedule. That is an active check.

Another option is to watch real requests fail. That is a passive check.

The balancer sends a health check as a request:

GET /health HTTP/1.1
Host: 10.0.0.1

The balancer reads the status code. Status 200 means the path answered. It does not prove that the server can serve an order.

After enough checks fail, the server leaves the pool. After enough checks succeed, the balancer puts it back.

A check that tests only the process can pass while orders fail. A check that queries the database can fail when the database fails. If every server checks the same dependency, the dependency can fail every server at once.

This is why the check should match the problem you care about. Check that the process can answer, then watch the error rate for the work that matters.

The extra network step has a price

A load balancer adds a hop, which is one network step. The request goes to the balancer, then to a server. The balancer is another machine to run and monitor.

The balancer is now part of the request path. If it fails, clients cannot reach the pool. If you run a second balancer, it can take over when the first fails.

The balancer can keep a client on one server. That is a sticky session.

The server must then keep the client's state. If it dies, the state goes with it. If the state lives in a store both servers can reach, any server can serve the next request.

Sometimes the hop buys nothing

The first case is a service with one server and steady traffic. One server handles the whole load, so a second server would sit idle.

The second case is a static site with no dynamic requests. A cache close to the client already serves the files.

A service whose database is the bottleneck does not get faster by adding servers. Fix the database or the query first. The database still does the same work.

Takeaway

A load balancer does not make one server faster. It lets clients share a pool of servers without knowing which server answers.

Common questions

Is a load balancer a reverse proxy?

RFC 9110 calls a gateway a reverse proxy. See RFC 9110, section 3.7. A reverse proxy forwards requests to servers. A load balancer is a reverse proxy that also chooses among servers.

Does it make my app faster?

The balancer receives the request, then forwards it to a server. The balancer helps with capacity and failures, not with the speed of one request.

What happens if the balancer fails?

If there is only one, the service is down. Run more than one and give clients a way to reach a working balancer.

Do I need sticky sessions?

Only when a server holds state the next request needs, such as an in-memory cart. A store both servers can reach lets any server serve the next request.

Sources

Originally published at awahab.dev.

Text embeddings and cosine similarity

Abdul Wahab — Mon, 26 Oct 2026 15:20:34 +0000

Text embeddings represent sentences as vectors; cosine similarity compares their directions to find related candidates. A high score still needs checking against the task. For example, "I forgot my password" and "How can I reset my login?" use different words to ask about a similar problem.

Give each sentence a list of numbers

An embedding is a vector: an ordered list of numbers that represents an item. For text, a trained model produces that list from a word, sentence, or passage. The geometry of these vectors can capture useful relationships for a task. Google's introduction to embedding spaces.

Picture a map of help articles. Password-reset instructions and login troubleshooting might sit near one another. A recipe for banana bread should be somewhere else. The map is an analogy for comparing those representations.

The model learns its representation during training. You do not usually assign one coordinate to "password" and another to "bread." Real embeddings have many dimensions. Individual coordinates need not have an obvious human label.

Start with a small, deliberately invented map

The sketch above uses three hand-written, two-dimensional vectors to illustrate the comparison. The coordinates are invented for teaching, and the scores come from those coordinates. Your browser does not run an embedding model.

Our query is [1, 0.1]. A password-reset answer points in almost the same direction, while a recipe points elsewhere. Select a different answer to see its direction and similarity score.

This toy map makes the arithmetic visible. With a real model, the numbers come from encoding text. A visualization in two dimensions simplifies the full space.

The sketch calculates its score from invented vectors. The score does not measure how a particular model ranks these sentences.

Compare directions with cosine similarity

Cosine similarity divides the dot product of two vectors by the product of their lengths. Its value ranges from −1 to 1 for nonzero vectors. A value near 1 means the vectors point in almost the same direction. The choice of similarity measure depends on the model and task. Sentence Transformers documents its similarity measures.

These are example vectors. Real searches use numbers from an embedding model.

save as similarity.mjs, then run node similarity.mjs

function cosine(a, b) {
  const dot = a.reduce((sum, value, i) => sum + value * b[i], 0);
  const length = Math.hypot(...a) * Math.hypot(...b);
  return length === 0 ? 0 : dot / length;
}

const query = [1, 0.1];
const answers = [
  { text: 'Reset your password', vector: [0.92, 0.2] },
  { text: 'Troubleshoot signing in', vector: [0.72, 0.43] },
  { text: 'Bake banana bread', vector: [-0.1, 1] },
];

const ranked = answers
  .map(answer => ({
    text: answer.text,
    score: cosine(query, answer.vector),
  }))
  .sort((a, b) => b.score - a.score);

console.table(ranked);

This example assumes the vectors have the same dimensions. Its zero-length check returns zero as a fallback. Mathematically, cosine similarity is undefined for a zero vector. The reset answer should rank first and the recipe last.

Turn the comparison into a search

For a real experiment, encode a small set of support articles with a sentence-embedding model. Encode a question with the same model. Calculate the similarities. Return the closest candidates. Sentence Transformers shows text encoding and comparison.

Try ten short questions before building a larger system. Include alternate wording, spelling mistakes, and an unrelated question. Record which answer appears first. Read that answer to decide whether it helps.

Keep the question list as a repeatable check. If you change the model or split articles into smaller passages, run the same questions again. This lets you compare results across changes instead of relying on one impressive result.

A high score still needs interpretation

"I can sign in" and "I cannot sign in" discuss almost the same subject while describing different states. Include pairs like these in your experiments. They help distinguish topic similarity from the exact relationship your application needs.

Similarity scores depend on the representation and comparison method. Even familiar cosine scores can behave unexpectedly across models and training choices. Research on the limits of cosine similarity explains why the score needs context.

Retrieval gives you candidates to inspect. In a support search, a good candidate should also match the product, version, and situation of the user. Ranking alone does not make that decision.

Takeaway

Embeddings turn text into vectors you can compare. Similarity helps find candidates, but you still need to check whether they suit the task.

Sources

Originally published at awahab.dev.

Idempotency and duplicate requests

Abdul Wahab — Mon, 26 Oct 2026 15:20:33 +0000

An idempotency key identifies retries of the same operation. A database uniqueness constraint can prevent duplicate records, while other side effects need their own coordination. This matters when an order succeeds but its reply is lost: the client must be able to retry without creating a second order.

A repeated request is an ordinary situation

Imagine the server receives an order and saves it successfully. The connection drops before the reply reaches your browser, so the browser cannot tell whether the order succeeded. A retry might be sensible, but creating another order can duplicate the work.

Repeating an idempotent operation gives the same intended effect. APIs can use an idempotency key to identify retries and return a previously recorded result. Stripe documents one practical implementation.

The sketch above models two requests for one action. Disable protection. Submit twice. Then reset the example. Enable protection. Submit twice again.

The request count stays at two while the number of orders changes.

Give the action an identity before sending it

two attempts, one intended order

POST /orders HTTP/1.1
Idempotency-Key: checkout-7f3a
Content-Type: application/json

{"item": "notebook", "quantity": 1}

Create a key when the user starts an operation. Preserve it across retries. A fresh key for every request makes those requests look like different operations. A new order needs a new key.

A server should associate the key with the caller and operation. It should check that repeated uses have the same request data. If an order changes, the server should reject reuse of its key. Otherwise, it could return an unrelated result. Stripe's idempotency behavior shows how it checks repeated parameters.

Disabling the Submit button improves the interface. The server also needs to handle repeats from retries, multiple tabs, and clients that do not use that interface.

Make the database enforce uniqueness

Consider two requests arriving together. Both ask whether a key exists. Each sees that it does not exist and tries to create an order. An application-level check followed by a separate insert leaves room for that race.

A database uniqueness constraint can enforce that a key appears only once within a chosen scope. PostgreSQL documents uniqueness across groups of columns.

In this database example, retries with the same customer and key return the existing order.

a PostgreSQL schema for this example

CREATE TABLE orders (
  id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  customer_id BIGINT NOT NULL,
  idempotency_key TEXT NOT NULL,
  request_body JSONB NOT NULL,
  UNIQUE (customer_id, idempotency_key)
);

For an operation that only creates this row, the winning insert records the order. A conflicting attempt can read the existing row, compare the request, and reuse the result. More complicated workflows must also coordinate their side effects and recorded results.

A small experiment you can run

This Python example uses an in-memory SQLite database, so it needs no database server. It models one customer ordering one item. The uniqueness constraint rejects the repeated key, and the function retrieves the saved result. Python's sqlite3 documentation covers parameter binding and transaction handling.

save as orders.py, then run python3 orders.py

import sqlite3

db = sqlite3.connect(':memory:')
db.execute('''
  CREATE TABLE orders (
    id INTEGER PRIMARY KEY,
    customer TEXT NOT NULL,
    key TEXT NOT NULL,
    item TEXT NOT NULL,
    UNIQUE(customer, key)
  )
''')

def submit(customer, key, item):
    try:
        with db:
            db.execute(
                'INSERT INTO orders(customer, key, item) VALUES (?, ?, ?)',
                (customer, key, item),
            )
    except sqlite3.IntegrityError:
        pass

    row = db.execute(
        'SELECT id, item FROM orders WHERE customer = ? AND key = ?',
        (customer, key),
    ).fetchone()
    if row is None or row[1] != item:
        raise ValueError('Key reused with different request data')
    return row[0]

print(submit('alice', 'checkout-1', 'notebook'))
print(submit('alice', 'checkout-1', 'notebook'))
print(db.execute('SELECT COUNT(*) FROM orders').fetchone()[0])
# Expected output: 1, 1, 1 (on separate lines)

Change the item in the second call while keeping its key. You should get an error. Change the key instead. You should get a second order. These variations show which requests count as the same operation.

Keep the guarantee within its boundaries

The example only records database rows. It does not send emails, reserve inventory, or contact another service. A unique order row alone cannot prevent those separate effects from happening twice. A real design must coordinate their identities and recovery paths.

Decide how long keys remain valid. Define what a second request sees while the first still runs. Keeping keys forever and expiring them quickly have different consequences. Document that contract so clients know when a retry remains safe.

Draw three moments: before the insert, after the insert, and after the reply. Imagine a failure at each point. Record what a retry should return. Include any work that might already be complete.

Takeaway

Use the same identity for retries of an operation. Enforce that identity where you record the operation.

Sources

Originally published at awahab.dev.

Request timeouts: how long to wait, and when to retry

Abdul Wahab — Mon, 26 Oct 2026 15:16:38 +0000

A client sends a request at 0 seconds. The service answers at 30 seconds. Nothing in the client says stop. For those 30 seconds it holds a connection, a worker, and memory for an answer that has not arrived.

A request timeout sets that limit. It decides how long your app waits before it gives up.

How long do you wait?

A timeout is the maximum time a client waits for a request to complete. Amazon Web Services (AWS) sets one on every remote call, and it sets two: a connection timeout and a request timeout. The Requests library ships without a default, and its docs say why: "Failure to do so can cause your program to hang indefinitely."

Example: a client with a 2-second timeout sends a request at 0 seconds. The service answers at 30 seconds. The client leaves at 2 seconds, and the answer arrives to nobody.

Those resources stay reserved until the answer arrives or the wait ends. AWS puts the risk this way. "If a large number of requests hold on to resources for a long time, the server can run out of those resources."

The limit is narrower than the name suggests. In Requests it does not cover the whole download. It fires when no bytes arrive for that many seconds.

The pool fills up

Example: 50 requests arrive each second, and each call hangs for 30 seconds. That is 50 × 30 = 1,500 calls in flight. Workers and connections are fixed, so new requests queue behind the stuck ones.

In the same setup, a pool of 100 workers faces 1,500 calls. That is 15 times more work than it can run. The queue grows, callers give up, and the app appears down while the code is fine.

Choosing the number

Measure how long the downstream service takes. AWS chooses an acceptable false-timeout rate, 0.1% for example.

AWS takes the matching percentile from the dependency. In this example it is p99.9. 99.9% of calls finish faster than that value. A limit near that value fails about one call in a thousand on a healthy day.

If the limit is too high, resources stay pinned. If it is too low, healthy calls fail. Those calls get retried, and the retries add traffic.

The limit splits in two: a connection timeout and a request timeout.

The AWS Builders' Library describes one team that traced rare timeouts to connection setup. A new secure connection took longer than 20 milliseconds, so a few requests failed right after a deploy. Opening connections at startup removed the problem.

Retries make it worse

When a service is busy, "frequent client retries can prevent the service from recovering and worsen the problem." Azure's guide calls that a retry storm. When the retries arrive together, it is a thundering herd.

The AWS Builders' Library says the same. Retries "can amplify the load on a dependent system," and against an overloaded service they can "make the overload worse instead of better." AWS's client libraries retry only when the dependency looks healthy. They stop when retries stop helping.

Retries also multiply work. When N clients retry together, the total work grows with N squared, according to AWS's jitter analysis.

Backoff, waiting longer after each failure, helps a little. Jitter, a random delay before each retry, spreads the spikes. In AWS's 100-client simulation, jitter cut call counts by more than half.

Cap the attempts and the total retry time. Azure's guide puts the cap at seconds or minutes, not forever.

Retry only calls you can repeat. A write that timed out may still have happened, so retries need idempotent operations, which can run twice without harm.

Deadlines for nested calls

Three calls with 2-second limits can burn 6 seconds before the first error reaches the user.

Give the whole request one deadline and pass what remains to each step. A 2-second budget spends 300 milliseconds on the first call. The rest of the chain gets 1.7 seconds and fails before the user does.

a 2-second limit in a browser

// The fetch aborts with a TimeoutError DOMException after 2 seconds.
const response = await fetch(url, { signal: AbortSignal.timeout(2000) });

The signal aborts with a TimeoutError when the time passes, as MDN documents. A caller can catch the error and tell it apart from a network failure.

Abort closes your side

Aborting a call closes your side of the connection. The remote service may keep working unless that service watches for the client to cancel. AWS is blunt about what that leaves open: "A timeout or failure doesn't necessarily mean that side effects haven't happened."

The timeout buys back your resources. It does not buy back theirs.

Without that number, a service has no limit on how long it can take. Its slowest dependency decides.

Takeaway

Every request needs a limit. Without one, the slowest dependency sets the pace of your app.

Common questions

Is a timeout the same as a retry?

No. A timeout ends one attempt. A retry starts another, and it needs a safe call, a capped delay, and a limit on attempts.

Does a timeout stop the server work?

No. It ends your wait. The remote service keeps running unless that service watches for the client to cancel, and a 30-second call can finish long after your 2-second limit.

Sources

Originally published at awahab.dev.

Circuit breakers: when to stop calling a service

Abdul Wahab — Mon, 26 Oct 2026 15:16:37 +0000

Imagine a payments service that starts timing out at 14:02. Your app keeps sending requests. Each one waits two seconds and fails. By 14:04 the worker pool is full, and checkout is down because one dependency is sick.

A circuit breaker stops the sending. After a set number of failures it returns an error without making the call. Your app keeps the capacity it would have spent on calls that cannot succeed.

The calls that cannot succeed

A breaker wraps a remote call and watches the result. Past a threshold it stops calling. Azure's architecture guide states the purpose: the pattern "helps prevent an application from repeatedly trying to run an operation that's likely to fail."

Martin Fowler credits Michael Nygard's book Release It! with popularizing the pattern. In Fowler's account, the breaker trips once failures reach a threshold. Then "all further calls to the circuit breaker return with an error, without the protected call being made at all."

You save the wait, not the error. A failing call still holds a connection and a worker until its timeout fires. Azure warns that blocked requests "might hold critical system resources, such as memory, threads, and database connections." Enough of them and unrelated parts of the app fail too.

Example: a checkout service calls payments with a 2-second timeout. Payments hangs. At 50 checkout requests a second, 100 workers sit waiting on the same broken dependency, and the pool is gone in under a minute.

The breaker sits between the caller and the dependency; when it opens, the call stops there and the error comes back at once.

Closed, open, half-open

The breaker is a small state machine.

Closed. Calls pass. The breaker counts failures inside a rolling window, a moving slice of time such as the last 10 seconds.

Open. Calls fail at once. Nothing reaches the dependency, and a timer runs.

Half-open. When the timer ends, a few trial calls pass. A success closes the breaker and resets the count. A failure reopens it and restarts the timer.

Azure explains why the third state earns its place. A recovering service "might be able to support a limited volume of requests until the recovery is complete." A flood of work can knock it over again.

Failures open the breaker, the cooldown leaves it half-open, and the trial call decides the next state.

What trips it

Start with a count of failures in a window. Azure describes a counter that resets at intervals, so the occasional error does not open the circuit. The threshold trips only when enough failures land inside the period.

Fowler suggests a rate instead: open at a 50% failure rate. Different errors can have different limits, such as "a threshold of 10 for timeouts but 3 for connection failures."

Count only errors that say the dependency is unhealthy. A 404 for a missing record, or a rejected input, is a normal answer. Fowler is direct: "Not all errors should trip the circuit."

Scope the breaker to one dependency, often to one operation. Wide scope causes a problem: a single breaker over every call to a service will open for a fault that affects one endpoint.

a breaker with three states

// A breaker for one dependency. The numbers are examples.
function breaker(fn, { threshold = 3, cooldown = 30000 /* ms */ } = {}) {
  let failures = 0;
  let openedAt = 0;
  return async (...args) => {
    const open = openedAt > 0 && Date.now() - openedAt < cooldown;
    if (open) throw new Error("circuit open");
    try {
      const result = await fn(...args);
      failures = 0;
      openedAt = 0;
      return result;
    } catch (err) {
      if (++failures >= threshold) openedAt = Date.now();
      throw err;
    }
  };
}

The next call after the cooldown is the probe. A success resets the breaker. A failure starts the cooldown again.

What the caller sees

An open breaker returns an error at once. The caller needs a plan for that error. Fowler lists the choices: fail the operation, use a workaround, show stale data, or put the work on a queue for later.

Example: a product page calls recommendations and inventory. With the recommendations breaker open, the page renders without the recommendations strip and keeps the buy button. The inventory call still runs. One broken dependency costs a strip, not the sale.

Hystrix, Netflix's breaker library, calls this fallback and graceful degradation. It also rejects a call when its pool of threads is full, instead of queueing work that will wait.

Where the breaker lives

You can put the breaker in the client. Hystrix, Netflix's library, and resilience4j, a Java library, wrap each call in process. The breaker knows which operation it protects, so per-endpoint thresholds are possible.

You can put it in the network. Envoy's outlier detection is passive health checking: it removes hosts that fail or slow down from the load-balancing set. Envoy caps how many hosts it ejects, and each repeat lengthens the time a host stays out.

The two see different things. A proxy sees host health, not what the operation means. A client library sees the operation, but each instance keeps its own count unless you share state.

It does not fix the service

A breaker protects your capacity. It does not repair the dependency, and it does not make a failed call succeed. The half-open probe exists to notice recovery, nothing more.

Traffic returns when the breaker closes. If every instance probes at the same moment, the recovering service gets a wave. Stagger the probes, or let one instance probe first.

Order matters with retries. Retries must respect the breaker. Azure says retry logic must watch what the breaker returns. Retries should stop "if the circuit breaker indicates that a fault isn't transient."

Log every state change with the failure count that caused it. An open breaker is a signal about the dependency. Fowler argues that operations staff should be able to trip or reset a breaker by hand.

Without a breaker, the slowest dependency decides how much of your capacity it gets. With one, you decide how many calls to waste before you stop.

Takeaway

A timeout bounds one call. A breaker bounds the number of calls, and protects your capacity, not the dependency.

Common questions

Is a circuit breaker the same as a timeout?

No. A timeout bounds one call. A breaker stops new calls to a dependency that keeps failing. Use both: the timeout ends the call, and the breaker ends the attempts.

Which errors should open it?

Only errors that mean the dependency is unhealthy: timeouts, connection errors, and 5xx responses. A 404 or a rejected input is a normal result and should not count.

What happens to calls already in flight?

They finish or hit their timeout. The breaker only blocks new calls.

Sources

Originally published at awahab.dev.

HTTP caching and stale responses

Abdul Wahab — Sun, 18 Oct 2026 19:40:26 +0000

Clearing your browser cache removes saved responses, so the next request can fetch a newer version. It does not change the server or clear a CDN's cache. If your browser shows an old page while a friend sees the update, the difference may be in which cached response each browser can reuse.

A saved copy with permission to reuse it

Imagine copying a café menu into your notebook. Checking the notebook is quicker than walking back to the counter. The trade-off is that the café might change its menu while your copy stays the same.

An HTTP (Hypertext Transfer Protocol) cache keeps a response so a later request can reuse it. A browser cache lives on your device. A shared cache, such as a content delivery network (CDN), can serve many visitors. Reuse can avoid a request to the origin server. MDN explains the different cache layers.

The cache needs a rule for how long it can reuse a saved copy. Without one, visitors can receive confusing answers.

A fresh response can differ from the server

an example response

HTTP/1.1 200 OK
Cache-Control: max-age=60
ETag: "menu-v1"

Here, max-age=60 gives the response a freshness lifetime of 60 seconds. During that time, an ordinary request can reuse the stored response. If the café changes its menu after ten seconds, the saved copy can still count as fresh under this rule.

Your browser can reuse the old page until its cache timer expires.

Once the response is stale, the cache needs another way to decide whether it can reuse it. In this example, it asks the server to validate its copy. The Cache-Control reference explains the headers.

In the sketch above, publish a new version. Then request the page. Advance time past 60 seconds. Request it again. The server changes immediately, but the cached copy follows its freshness rule.

Revalidating a saved response

The server supplies a version identifier called an ETag. The client can send its saved identifier in an If-None-Match request header. If the identifier still matches, the server can answer 304 Not Modified without sending the response body again. If it differs, the server returns the new representation. ETag reference.

the version check

GET /menu HTTP/1.1
If-None-Match: "menu-v1"

# If the server still has menu-v1:
HTTP/1.1 304 Not Modified

The cache checks for an update and can still reuse the bytes it already downloaded. To see this in the sketch, request an expired copy without publishing an update.

Clearing the browser cache

After you remove the saved response from your browser, its next request cannot use that local copy. This can reveal a newer version. It does not automatically empty the cache in a CDN or an application. Those are separate layers.

Which cache supplied the response? Inspect the request in the Network panel of your browser. Check the response headers. Compare a normal load with one that bypasses the browser cache. MDN covers reloads and stored responses.

Suppose three colleagues see different versions. Record who saw each version and when. Then inspect the cache headers. That timeline can make the freshness rule easier to understand.

Three rules worth recognizing

Directive	Meaning
max-age=60	Allow reuse while the response is fresh for 60 seconds.
no-cache	Allow storage. Validate before reuse.
no-store	Do not store the response in an HTTP cache.

Despite its name, no-cache permits storage. It requires validation before reuse. no-store prevents storage. These directives control HTTP caching. An application can also manage separate caches. Cache-Control reference.

For practice, open the Network panel on any public page. Select one document and one image. Compare their caching headers. Would you accept the same freshness lifetime for both resources? Explain your choice.

Takeaway

A cache can treat a response as fresh even after the server changes. Choose its freshness rules deliberately.

Sources

Originally published at awahab.dev.