A Lambda circuit breaker protects your functions from a third-party system that has stopped answering properly. In practice, however, it is often the difference between “one function briefly shows a placeholder” and “the whole API hangs because a partner currently needs 25 seconds for every response”.
The topic matters to me because Serverless sets a trap here: Lambda happily scales out while the slow system behind it keeps buckling. Also, you pay for every millisecond of waiting, and your users wait along with you. In this fourth part of the series I build a circuit breaker for the fictional bookshop that shares its state in DynamoDB, with TypeScript, tests and a CDK stack.
In short
A circuit breaker counts failures of a third-party system and stops calling it once a threshold is reached. Instead, it returns a fallback right away and, after a wait, lets exactly one trial call through. In Lambda the state has to live outside the function, for example in DynamoDB; otherwise every instance has its own breaker.
What you will learn in this article
- How the three states of a circuit breaker work together
- Why the state must not live in memory in Lambda
- How to share it safely with conditional writes in DynamoDB
- A tested TypeScript project with a CDK stack and an alarm
- Which errors should count and which never should
What is a circuit breaker?
The name comes from the breaker in your fuse box. When too much current flows, it trips before the wire overheats. Michael Nygard carried the idea over to software in his book “Release It!”, and Martin Fowler made it widely known with a short article in 2014.
In software, the circuit breaker sits between your code and a remote system. It knows three states:
- Closed: All normal, calls go through. On the side, the breaker counts failures.
- Open: The threshold has been reached. Calls no longer go out at all; they get a substitute answer right away, the fallback.
- Half-open: The wait is over. A single trial call may go through. If it succeeds, the breaker closes. If it fails, it opens again.
The benefit is not that errors go away. They still happen at the third-party system. Instead, the benefit is that your system fails fast and predictably instead of slowly and unpredictably. On top of that, the struggling system gets room to recover instead of being flooded with even more retries.
🎬 States of a circuit breaker in Lambda – this animated diagram only runs on my site: watch it on sascha-ms.de
Why a Lambda circuit breaker works differently
The classic libraries for this pattern keep the state in the memory of the process. On a server with a long-running process, that works. In Lambda, however, exactly this assumption breaks down.
Every instance would have its own breaker
For concurrent requests, Lambda starts several execution environments. Each one also has its own memory. If 40 instances are running and each of them has to count five failures before it opens, 200 calls go to a system you already knew was struggling after five. In addition, the AWS documentation states clearly that execution environments are replaced regularly and that you should not rely on them persisting. As a result, an open breaker held in memory will simply disappear at some point.
That is why I keep the state in a small DynamoDB table. All instances therefore read and write the same item. The read costs a few milliseconds per call; in return, the breaker is open for everyone at the same time.
Waiting costs twice in Lambda
First of all, Lambda bills by duration. A function that waits 10 seconds for a response costs ten times as much as one that gives up after one second. At the same time, every waiting instance blocks a share of your concurrency limit, meaning the number of instances allowed to run in your account at once. A slow partner can therefore slow down other functions in the same account that have nothing to do with it.
Then there is API Gateway: an HTTP API waits at most 30 seconds for the integration, and that cannot be raised. Without a timeout of your own, your customer gets an error after 30 seconds, and the Lambda function keeps running in the background anyway. You can see the effect in the Lambda cost calculator by changing the duration from 300 ms to 10 seconds.
The architecture for the bookshop
The bookshop from the earlier parts shows the delivery time on every book page. It comes from a wholesaler’s API, which, however, is slow on some days. The goal: when the wholesaler is stuck, the customer sees “delivery time follows by email” after one second at most, instead of a spinner.
The architecture: Lambda checks the state in DynamoDB before it calls the wholesaler.
Only a few building blocks are needed. First, an HTTP API forwards GET /books/{isbn}/delivery-time to a Lambda function. Then the function reads the breaker state from the Circuits table, calls the wholesaler only while the breaker is closed and aborts after 800 milliseconds. Finally, it writes every opening as a structured log line, and a CloudWatch metric filter turns that into an alarm.
Building a Lambda circuit breaker step by step
Prerequisites
You need Node.js 22, an AWS account with credentials set up, and an account where the CDK has already been prepared once with cdk bootstrap. The article Deploy a Lambda function with CDK and TypeScript shows how. Basic TypeScript knowledge is enough; I explain everything else along the way.
Step 1: Create the project
First, create a folder and install the packages. I pin the versions I tested the example with.
mkdir circuit-breaker-lambda && cd circuit-breaker-lambda
npm init -y
npm install @aws-sdk/client-dynamodb@3.1157.0 @aws-sdk/lib-dynamodb@3.1157.0
npm install -D typescript@7.0.2 vitest@5.0.5 tsx@4.23.15 esbuild@0.28.2 aws-cdk-lib@2.275.0 aws-cdk@2.1146.1 constructs@10.8.1 @types/aws-lambda@8.10.164 @types/node@22.20.5
mkdir -p bin lib src test
Also set "type": "module" in your package.json so Node loads the files as ES modules. Two small configuration files complete the setup:
// tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"skipLibCheck": true,
"types": ["node"],
"noEmit": true
},
"include": ["bin", "lib", "src", "test"]
}
The cdk.json file tells the CDK how to start the app:
{
"app": "npx tsx bin/app.ts"
}
Step 2: Define the states and rules
The core is a small class without any AWS dependency. It also receives a store (CircuitStore) so you can test it without the cloud. However, the half-open state is not stored as a separate value in the table. It follows from “open, but the wait has expired”, and a short lease called probeUntil decides who may make the trial call.
// src/circuit-breaker.ts
export type CircuitState = 'CLOSED' | 'OPEN';
export interface CircuitRecord {
state: CircuitState;
failures: number;
windowStart: number; // start of the counting window (epoch milliseconds)
openUntil: number; // until when the breaker stays open
probeUntil: number; // lease for the single trial call (half-open)
}
// State storage – DynamoDB in Lambda, a Map in tests
export interface CircuitStore {
get(name: string): Promise<CircuitRecord | undefined>;
recordFailure(name: string, now: number, windowMs: number): Promise<number>;
open(name: string, now: number, openUntil: number, fromProbe: boolean): Promise<boolean>;
tryAcquireProbe(name: string, now: number, leaseMs: number): Promise<boolean>;
close(name: string, now: number): Promise<void>;
}
export interface BreakerOptions {
failureThreshold: number; // this many failures in the window open the breaker
windowMs: number; // length of the counting window
openMs: number; // how long the breaker stays open
probeLeaseMs: number; // how long one instance owns the trial call
isFailure?: (error: unknown) => boolean; // which errors count
now?: () => number;
log?: (entry: Record<string, unknown>) => void;
}
export type CallResult<T> =
| { source: 'live'; value: T }
| { source: 'fallback'; value: T; reason: 'open' | 'probe-busy' | 'error' };
export class CircuitBreaker {
private readonly now: () => number;
private readonly log: (entry: Record<string, unknown>) => void;
private readonly isFailure: (error: unknown) => boolean;
constructor(
private readonly name: string,
private readonly store: CircuitStore,
private readonly options: BreakerOptions,
) {
this.now = options.now ?? Date.now;
this.log = options.log ?? ((entry) => console.log(JSON.stringify(entry)));
this.isFailure = options.isFailure ?? (() => true);
}
async call<T>(action: () => Promise<T>, fallback: () => T): Promise<CallResult<T>> {
const now = this.now();
const record = await this.store.get(this.name);
let isProbe = false;
if (record?.state === 'OPEN') {
if (now < record.openUntil) {
return { source: 'fallback', value: fallback(), reason: 'open' };
}
// Wait time is over: exactly one instance may test (half-open)
isProbe = await this.store.tryAcquireProbe(this.name, now, this.options.probeLeaseMs);
if (!isProbe) {
return { source: 'fallback', value: fallback(), reason: 'probe-busy' };
}
this.log({ event: 'circuit_half_open', circuit: this.name });
}
try {
const value = await action();
// Only write when something changes – saves write requests
if (isProbe || (record?.failures ?? 0) > 0) {
await this.store.close(this.name, this.now());
if (isProbe) this.log({ event: 'circuit_closed', circuit: this.name });
}
return { source: 'live', value };
} catch (error) {
if (!this.isFailure(error)) throw error;
await this.onFailure(isProbe, error);
return { source: 'fallback', value: fallback(), reason: 'error' };
}
}
private async onFailure(isProbe: boolean, error: unknown): Promise<void> {
const now = this.now();
const message = error instanceof Error ? `${error.name}: ${error.message}` : String(error);
if (isProbe) {
await this.store.open(this.name, now, now + this.options.openMs, true);
this.log({ event: 'circuit_opened', circuit: this.name, cause: 'probe-failed', error: message });
return;
}
const failures = await this.store.recordFailure(this.name, now, this.options.windowMs);
if (failures >= this.options.failureThreshold) {
const opened = await this.store.open(this.name, now, now + this.options.openMs, false);
// Only the instance that actually flipped the breaker logs it
if (opened) {
this.log({ event: 'circuit_opened', circuit: this.name, failures, error: message });
}
}
}
}
Two details matter to me. First, on success the class only writes when something actually changes, so in normal operation each call costs a single read. Second, isFailure decides which errors count. More on that in a moment.
Step 3: Share the state in DynamoDB
Now comes the tricky part, because many instances write at the same time. The answer is conditional writes (condition expressions): DynamoDB only performs an update if a condition holds at the moment of writing; otherwise you get a ConditionalCheckFailedException. Thus I handle three things without locks or transactions.
// src/dynamo-circuit-store.ts
import { ConditionalCheckFailedException, DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, GetCommand, PutCommand, UpdateCommand } from '@aws-sdk/lib-dynamodb';
import type { CircuitRecord, CircuitStore } from './circuit-breaker.js';
export class DynamoCircuitStore implements CircuitStore {
constructor(
private readonly tableName: string,
private readonly db = DynamoDBDocumentClient.from(new DynamoDBClient({})),
) {}
async get(name: string): Promise<CircuitRecord | undefined> {
// Eventually consistent is enough: a few hundred milliseconds of lag don't matter
const { Item } = await this.db.send(new GetCommand({ TableName: this.tableName, Key: { circuit: name } }));
if (!Item) return undefined;
return {
state: Item.state === 'OPEN' ? 'OPEN' : 'CLOSED',
failures: Number(Item.failures ?? 0),
windowStart: Number(Item.windowStart ?? 0),
openUntil: Number(Item.openUntil ?? 0),
probeUntil: Number(Item.probeUntil ?? 0),
};
}
async recordFailure(name: string, now: number, windowMs: number): Promise<number> {
try {
// Atomically count failures in the current window
const result = await this.db.send(
new UpdateCommand({
TableName: this.tableName,
Key: { circuit: name },
UpdateExpression:
'SET failures = if_not_exists(failures, :zero) + :one, windowStart = if_not_exists(windowStart, :now), #state = if_not_exists(#state, :closed)',
ConditionExpression: 'attribute_not_exists(windowStart) OR windowStart >= :floor',
ExpressionAttributeNames: { '#state': 'state' },
ExpressionAttributeValues: { ':zero': 0, ':one': 1, ':now': now, ':closed': 'CLOSED', ':floor': now - windowMs },
ReturnValues: 'UPDATED_NEW',
}),
);
return Number(result.Attributes?.failures ?? 1);
} catch (error) {
if (!(error instanceof ConditionalCheckFailedException)) throw error;
// Window has expired: start again at 1
await this.db.send(
new UpdateCommand({
TableName: this.tableName,
Key: { circuit: name },
UpdateExpression: 'SET failures = :one, windowStart = :now',
ExpressionAttributeValues: { ':one': 1, ':now': now },
}),
);
return 1;
}
}
async open(name: string, now: number, openUntil: number, fromProbe: boolean): Promise<boolean> {
try {
await this.db.send(
new UpdateCommand({
TableName: this.tableName,
Key: { circuit: name },
UpdateExpression: 'SET #state = :open, openUntil = :until, failures = :zero, windowStart = :now REMOVE probeUntil',
// Only the first trigger sets the wait time, later ones don't extend it
ConditionExpression: fromProbe ? '#state = :open' : 'attribute_not_exists(#state) OR #state = :closed',
ExpressionAttributeNames: { '#state': 'state' },
ExpressionAttributeValues: { ':open': 'OPEN', ':until': openUntil, ':zero': 0, ':now': now },
}),
);
return true;
} catch (error) {
if (error instanceof ConditionalCheckFailedException) return false;
throw error;
}
}
async tryAcquireProbe(name: string, now: number, leaseMs: number): Promise<boolean> {
try {
await this.db.send(
new UpdateCommand({
TableName: this.tableName,
Key: { circuit: name },
UpdateExpression: 'SET probeUntil = :lease',
ConditionExpression:
'#state = :open AND openUntil <= :now AND (attribute_not_exists(probeUntil) OR probeUntil < :now)',
ExpressionAttributeNames: { '#state': 'state' },
ExpressionAttributeValues: { ':open': 'OPEN', ':now': now, ':lease': now + leaseMs },
}),
);
return true;
} catch (error) {
if (error instanceof ConditionalCheckFailedException) return false;
throw error;
}
}
async close(name: string, now: number): Promise<void> {
await this.db.send(
new PutCommand({
TableName: this.tableName,
Item: { circuit: name, state: 'CLOSED', failures: 0, windowStart: now, openUntil: 0 },
}),
);
}
}
When counting, the condition makes sure that old failures outside the 60-second window no longer count. Similarly, when opening, only the first instance sets the wait time. Later instances that see the threshold at the same moment do not extend it, which is also what the AWS guidance on the pattern recommends. Finally, for the trial call, exactly one instance wins the probeUntil lease, and all others keep returning the fallback.
You might wonder why I do not use TTL to let the open state expire. According to the documentation, DynamoDB deletes expired items only “within a few days”. For a 30-second wait, however, that is useless, so I compare against openUntil myself.
Step 4: The handler with timeout and fallback
The handler calls the wholesaler with fetch and aborts after 800 milliseconds via AbortSignal.timeout(800). The breaker is created outside the handler so that warm instances reuse the DynamoDB client.
// src/delivery-time.ts
import type { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from 'aws-lambda';
import { CircuitBreaker } from './circuit-breaker.js';
import { DynamoCircuitStore } from './dynamo-circuit-store.js';
export interface DeliveryInfo {
isbn: string;
days: number | null;
source: 'wholesaler' | 'fallback';
note?: string;
}
export class WholesalerError extends Error {
constructor(readonly status: number) {
super(`Wholesaler responded with HTTP ${status}`);
this.name = 'WholesalerError';
}
}
// Counts as an outage: timeout, network error, 5xx and 429 – but not a 4xx business error
export function isOutage(error: unknown): boolean {
if (error instanceof WholesalerError) return error.status >= 500 || error.status === 429;
if (!(error instanceof Error)) return false;
if (error.name === 'TimeoutError' || error.name === 'AbortError') return true;
// fetch reports network errors as TypeError “fetch failed” – other TypeErrors are bugs on our side
return error.name === 'TypeError' && error.message === 'fetch failed';
}
export async function fetchDeliveryDays(baseUrl: string, isbn: string, timeoutMs: number): Promise<number | null> {
const response = await fetch(`${baseUrl}/stock/${encodeURIComponent(isbn)}`, {
signal: AbortSignal.timeout(timeoutMs),
});
if (response.status === 404) return null; // Unknown book: business case, not an outage
if (!response.ok) throw new WholesalerError(response.status);
const body = (await response.json()) as { deliveryDays: number };
return body.deliveryDays;
}
// Outside the handler: reused in warm instances
const breaker = new CircuitBreaker('wholesaler-stock', new DynamoCircuitStore(process.env.CIRCUIT_TABLE ?? ''), {
failureThreshold: Number(process.env.FAILURE_THRESHOLD ?? 5),
windowMs: 60_000,
openMs: Number(process.env.OPEN_SECONDS ?? 30) * 1000,
probeLeaseMs: 5_000,
isFailure: isOutage,
});
export async function handler(event: APIGatewayProxyEventV2): Promise<APIGatewayProxyResultV2> {
const isbn = event.pathParameters?.isbn ?? '';
if (!/^\d{10}(\d{3})?$/.test(isbn)) {
return { statusCode: 400, body: JSON.stringify({ message: 'Expected an ISBN with 10 or 13 digits' }) };
}
const result = await breaker.call<DeliveryInfo | null>(
async () => {
const days = await fetchDeliveryDays(process.env.WHOLESALER_URL ?? '', isbn, 800);
return days === null ? null : { isbn, days, source: 'wholesaler' };
},
() => ({ isbn, days: null, source: 'fallback', note: 'Delivery time unavailable right now – we will email you.' }),
);
if (result.value === null) {
return { statusCode: 404, body: JSON.stringify({ message: 'Book not found at the wholesaler' }) };
}
return {
statusCode: 200,
headers: { 'content-type': 'application/json', 'x-circuit': result.source === 'live' ? 'live' : result.reason },
body: JSON.stringify(result.value),
};
}
Here the fallback is an honest answer with status 200 and a note that the delivery time will follow. In addition, the x-circuit header tells you while testing whether the response was live or why the fallback kicked in.
Step 5: Test without AWS
For the tests, I replace DynamoDB with an in-memory store that follows the same rules. The breaker also gets its clock from outside, so I can fast-forward 30 seconds in a single line.
// test/memory-store.ts
import type { CircuitRecord, CircuitStore } from '../src/circuit-breaker.js';
// Same rules as the DynamoDB store, just in memory
export class MemoryCircuitStore implements CircuitStore {
readonly records = new Map<string, CircuitRecord>();
async get(name: string) {
const record = this.records.get(name);
return record ? { ...record } : undefined;
}
async recordFailure(name: string, now: number, windowMs: number) {
const record = this.records.get(name) ?? { state: 'CLOSED', failures: 0, windowStart: now, openUntil: 0, probeUntil: 0 };
if (record.windowStart < now - windowMs) {
record.failures = 0;
record.windowStart = now;
}
record.failures += 1;
this.records.set(name, record);
return record.failures;
}
async open(name: string, now: number, openUntil: number, fromProbe: boolean) {
const record = this.records.get(name);
const allowed = fromProbe ? record?.state === 'OPEN' : record?.state !== 'OPEN';
if (!allowed) return false;
this.records.set(name, { state: 'OPEN', failures: 0, windowStart: now, openUntil, probeUntil: 0 });
return true;
}
async tryAcquireProbe(name: string, now: number, leaseMs: number) {
const record = this.records.get(name);
if (!record || record.state !== 'OPEN' || record.openUntil > now || record.probeUntil >= now) return false;
record.probeUntil = now + leaseMs;
return true;
}
async close(name: string, now: number) {
this.records.set(name, { state: 'CLOSED', failures: 0, windowStart: now, openUntil: 0, probeUntil: 0 });
}
}
// test/circuit-breaker.test.ts
import { createServer } from 'node:http';
import type { AddressInfo } from 'node:net';
import { describe, expect, it } from 'vitest';
import { CircuitBreaker } from '../src/circuit-breaker.js';
import { WholesalerError, isOutage } from '../src/delivery-time.js';
import { MemoryCircuitStore } from './memory-store.js';
function setup() {
let clock = 1_000_000;
const logs: Record<string, unknown>[] = [];
const store = new MemoryCircuitStore();
const breaker = new CircuitBreaker('wholesaler', store, {
failureThreshold: 3,
windowMs: 60_000,
openMs: 30_000,
probeLeaseMs: 5_000,
isFailure: isOutage,
now: () => clock,
log: (entry) => logs.push(entry),
});
const advance = (ms: number) => (clock += ms);
return { breaker, store, logs, advance };
}
const fails = () => Promise.reject(new WholesalerError(503));
const works = () => Promise.resolve(7);
const fallback = () => -1;
describe('CircuitBreaker', () => {
it('opens after three failures and stops calling the third-party system', async () => {
const { breaker, logs } = setup();
for (let i = 0; i < 3; i++) await breaker.call(fails, fallback);
let called = false;
const result = await breaker.call(async () => ((called = true), 7), fallback);
expect(result).toEqual({ source: 'fallback', value: -1, reason: 'open' });
expect(called).toBe(false);
expect(logs.filter((l) => l.event === 'circuit_opened')).toHaveLength(1);
});
it('lets exactly one trial call through after the wait and closes on success', async () => {
const { breaker, store, advance } = setup();
for (let i = 0; i < 3; i++) await breaker.call(fails, fallback);
advance(30_000);
// A second instance arrives during the trial call: it gets the fallback
const second = store.tryAcquireProbe('wholesaler', 1_030_000, 5_000);
const result = await breaker.call(works, fallback);
expect(await second).toBe(true);
expect(result).toEqual({ source: 'fallback', value: -1, reason: 'probe-busy' });
advance(5_001); // the other instance's lease has expired
expect(await breaker.call(works, fallback)).toEqual({ source: 'live', value: 7 });
expect((await store.get('wholesaler'))?.state).toBe('CLOSED');
});
it('opens again right away when the trial call fails', async () => {
const { breaker, store, advance } = setup();
for (let i = 0; i < 3; i++) await breaker.call(fails, fallback);
advance(30_000);
await breaker.call(fails, fallback);
const record = await store.get('wholesaler');
expect(record?.state).toBe('OPEN');
expect(record?.openUntil).toBe(1_030_000 + 30_000);
});
it('ignores failures outside the window', async () => {
const { breaker, store, advance } = setup();
await breaker.call(fails, fallback);
await breaker.call(fails, fallback);
advance(61_000);
await breaker.call(fails, fallback);
expect((await store.get('wholesaler'))?.state).toBe('CLOSED');
});
it('does not treat 404 or our own bugs as an outage', async () => {
const { breaker } = setup();
const notFound = () => Promise.reject(new WholesalerError(404));
await expect(breaker.call(notFound, fallback)).rejects.toThrow('HTTP 404');
expect(isOutage(new TypeError("Cannot read properties of undefined (reading 'days')"))).toBe(false);
});
it('treats timeouts from AbortSignal.timeout as an outage', async () => {
// A server that never answers – like a hanging wholesaler
const server = createServer(() => {}).listen(0);
const { port } = server.address() as AddressInfo;
const error = await fetch(`http://127.0.0.1:${port}/`, { signal: AbortSignal.timeout(50) }).catch((e: unknown) => e);
server.closeAllConnections();
server.close();
expect((error as Error).name).toBe('TimeoutError');
expect(isOutage(error)).toBe(true);
});
});
npx tsc --noEmit
npx vitest run
Expected result:
✓ test/circuit-breaker.test.ts (6 tests)
Test Files 1 passed (1)
Tests 6 passed (6)
The last test is especially important to me: it starts a local server that never answers and checks that a timeout from AbortSignal.timeout arrives as a TimeoutError and counts as an outage. That is exactly the case people tend to forget.
Step 6: Deploy the stack with CDK
The stack creates the table, the function, the HTTP API and the alarm. At 3 seconds, the function timeout is well above the 800 milliseconds for the wholesaler, which leaves time for DynamoDB and the response.
// lib/delivery-time-stack.ts
import { CfnOutput, Duration, RemovalPolicy, Stack, type StackProps } from 'aws-cdk-lib';
import { HttpApi, HttpMethod } from 'aws-cdk-lib/aws-apigatewayv2';
import { HttpLambdaIntegration } from 'aws-cdk-lib/aws-apigatewayv2-integrations';
import { Alarm, ComparisonOperator, TreatMissingData } from 'aws-cdk-lib/aws-cloudwatch';
import { AttributeType, Billing, TableV2 } from 'aws-cdk-lib/aws-dynamodb';
import { Architecture, Runtime } from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction, OutputFormat } from 'aws-cdk-lib/aws-lambda-nodejs';
import { FilterPattern, LogGroup, MetricFilter, RetentionDays } from 'aws-cdk-lib/aws-logs';
import type { Construct } from 'constructs';
export class DeliveryTimeStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
// One item per breaker, the partition key is the name of the third-party system
const circuits = new TableV2(this, 'Circuits', {
partitionKey: { name: 'circuit', type: AttributeType.STRING },
billing: Billing.onDemand(),
removalPolicy: RemovalPolicy.DESTROY, // demo only
});
const logGroup = new LogGroup(this, 'DeliveryTimeLogs', {
retention: RetentionDays.TWO_WEEKS,
removalPolicy: RemovalPolicy.DESTROY,
});
const deliveryTime = new NodejsFunction(this, 'DeliveryTime', {
entry: 'src/delivery-time.ts',
handler: 'handler',
runtime: Runtime.NODEJS_22_X,
architecture: Architecture.ARM_64,
memorySize: 256,
timeout: Duration.seconds(3), // well above the fetch timeout of 800 ms
logGroup,
environment: {
CIRCUIT_TABLE: circuits.tableName,
WHOLESALER_URL: 'https://wholesaler.example.com/api',
FAILURE_THRESHOLD: '5',
OPEN_SECONDS: '30',
},
bundling: { format: OutputFormat.ESM, minify: true, sourceMap: true },
});
circuits.grantReadWriteData(deliveryTime);
const api = new HttpApi(this, 'BookshopApi');
api.addRoutes({
path: '/books/{isbn}/delivery-time',
methods: [HttpMethod.GET],
integration: new HttpLambdaIntegration('DeliveryTimeIntegration', deliveryTime),
});
// Every opening becomes a metric – and from the first one on, an alarm
const opened = new MetricFilter(this, 'CircuitOpened', {
logGroup,
filterPattern: FilterPattern.stringValue('$.event', '=', 'circuit_opened'),
metricNamespace: 'Bookshop',
metricName: 'CircuitOpened',
metricValue: '1',
});
new Alarm(this, 'CircuitOpenedAlarm', {
metric: opened.metric({ statistic: 'Sum', period: Duration.minutes(5) }),
threshold: 1,
evaluationPeriods: 1,
comparisonOperator: ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treatMissingData: TreatMissingData.NOT_BREACHING,
alarmDescription: 'The circuit breaker for the wholesaler has opened',
});
new CfnOutput(this, 'ApiUrl', { value: api.apiEndpoint });
}
}
// bin/app.ts
import { App } from 'aws-cdk-lib';
import { DeliveryTimeStack } from '../lib/delivery-time-stack.js';
const app = new App();
new DeliveryTimeStack(app, 'DeliveryTimeStack');
npx cdk synth
npx cdk deploy
After the deployment, the CDK prints the API URL (example output):
Outputs:
DeliveryTimeStack.ApiUrl = https://abc123xyz.execute-api.eu-central-1.amazonaws.com
Put a real address into WHOLESALER_URL first. For experiments, a mock endpoint of your own that you can deliberately slow down works well.
Step 7: Try it out
API=https://abc123xyz.execute-api.eu-central-1.amazonaws.com
for i in $(seq 1 8); do
curl -s -o /dev/null -w "%{http_code} %{time_total}s %header{x-circuit}\n" "$API/books/9783161484100/delivery-time"
done
Each line shows the status code, the duration and the x-circuit header (%header{…} needs curl 7.84 or later). When the wholesaler is failing, you first see error five times with a duration of just under 800 milliseconds, and then open with a few milliseconds. Then, after 30 seconds, exactly one request goes through as the trial call. In CloudWatch Logs you will also find the lines circuit_opened, circuit_half_open and circuit_closed, and the alarm fires on the first opening.
Which errors should trip the breaker
This is where I see the most debate. My rule of thumb: only count what points to a problem at the third-party system, not what points to a problem with the request.
| Error | Counts? | Why |
|---|---|---|
Timeout (TimeoutError) |
yes | The core case: the system is too slow. |
Network error (TypeError: fetch failed) |
yes | DNS, dropped connections, TLS problems. |
| HTTP 500–599 | yes | The system itself reports a failure. |
| HTTP 429 | yes | Too many requests: firing more makes it worse. |
| HTTP 404 or 400 | no | A business case or a mistake in your request, not an outage. |
| Bug in your own code | no | A bug on your side should not “switch off” the partner. |
Especially that last point goes wrong easily. If your function throws an exception of its own on a broken response and that exception counts, the breaker opens because of your bug. In my example, isOutage therefore only returns true for the clearly defined outages; everything else is rethrown as usual. Watch out for TypeError: fetch reports network errors as a TypeError with the message “fetch failed”, but a typo in your code also produces a TypeError. That is why I check the message as well.
Common mistakes with a Lambda circuit breaker
The timeout is missing or too long. Without a timeout of your own, fetch waits until Lambda kills the function. Your failure counter then never goes up, because the code never gets that far. The timeout for the call has to be clearly shorter than the function timeout.
Retries in front of the breaker. If you retry three times with backoff first and only then count a failure, you triple the load on the struggling system. Instead, retries belong behind the breaker, or they are skipped while it is open. How to build the retries behind the breaker with jitter and a time budget is covered in my article on exponential backoff with jitter.
The wait keeps getting extended. If every instance sets openUntil again when opening, the wait slides further and further back under high load. The condition on opening therefore prevents this.
Many trial calls at once. Without a lease, all instances pounce on the system at the same moment once the wait expires. That is the opposite of “testing carefully”.
No meaningful fallback. A breaker that returns a 500 instead of a timeout is faster, but not better. Decide per call what the customer should see instead: a placeholder, a cached value or an honest notice.
Asynchronous work still done synchronously. If the result does not have to be there right away, a queue is often the better answer. The job pattern from part 1 of the series buffers jobs in SQS, and a failing partner then only delays the processing, not the response.
Costs and security
The table runs in on-demand mode. In normal operation, the breaker costs one eventually consistent read per call. Writes, however, only happen on failures and on reset. With very high traffic, you can additionally cache the state in the instance’s memory for a second or two, which reduces the reads considerably. The price, though, is that an instance notices a change a little later.
As for permissions, grantReadWriteData gives the function access to this one table only. Make sure the fallback and the logs never output response content from the partner that could contain personal data; my log line only contains the error name and message. And plan for a way to open or close the breaker by hand: an item in the table with state: OPEN and an openUntil in the future is all it takes.
My take
I consider the circuit breaker one of the underrated patterns in Serverless architectures. Many teams set retries and timeouts but do not think about the moment when a partner goes down for a quarter of an hour. That is exactly when Lambda cheerfully scales out and turns a small problem at the partner into a big one on your side.
Still, I do not build it in everywhere. For calls between my own Lambda functions via queues I do not need it; dead-letter queues and partial batch responses do the job there. However, I find it worthwhile for synchronous calls to systems I do not control and where a usable fallback exists. If there is no fallback, the more important question is often whether the call needs to be synchronous at all.
In its guidance, AWS also describes a variant with Step Functions. I find it elegant when a workflow exists anyway. For a single API call, though, one extra Express workflow execution per request is too much machinery for me. Besides, the small class with DynamoDB is quicker to explain and quicker to debug. One more honest point: the breaker only measures what it sees. You have to base the thresholds on real numbers, otherwise it trips too early or too late. I described how to get those numbers in Monitoring Lambda: Alarms, Dashboard and Budget with CDK.
FAQ
What is the difference between a retry and a circuit breaker?
A retry repeats a single call because the error might have been temporary. A circuit breaker looks across many calls and stops entirely when the system is clearly failing. The two complement each other, as long as the retries do not happen in front of the breaker.
Do I need Step Functions for a Lambda circuit breaker?
No. AWS describes a variant with Step Functions, but for individual API calls a small class with a DynamoDB table is enough. Step Functions pays off when the call is part of a workflow anyway.
Which thresholds make sense?
There are no universally correct numbers. For example, a good start is five failures in 60 seconds and a 30-second wait. After that, tune the values to the partner’s normal error rate and response time.
Can I use ElastiCache instead of DynamoDB?
Yes, the AWS guidance mentions Redis as a faster alternative. You then need a VPC for the function and a running cache, though. For most Serverless applications, DynamoDB is simpler.
Your next step
Find a synchronous call to an external partner in your system and check whether it has a timeout of its own. If it does not, that is the first and cheapest step. After that, think about the fallback, and only then about the breaker.
Which third-party system has dragged everything down with it on your side, and how did you react? Let me know in the comments.
Continue the series: Overview “Serverless patterns in practice” · Previous part: Fan-out with SNS and SQS: One Event, Many Consumers · Next part: Strangler fig pattern: migrate to Lambda step by step
Sources
- Martin Fowler: CircuitBreaker
- AWS Prescriptive Guidance: Circuit breaker pattern
- AWS Lambda: Understanding the execution environment lifecycle
- Amazon DynamoDB: Condition expressions
- Amazon DynamoDB: Time to Live (TTL)
- Node.js: AbortSignal.timeout()
- Amazon API Gateway: Quotas for HTTP APIs
- Tested with Node.js 22.23, TypeScript 7.0.2, Vitest 5.0.5, aws-cdk-lib 2.275.0, aws-cdk 2.1146.1 and @aws-sdk/client-dynamodb 3.1157.0:
tsc, the 6 tests from the article andcdk synth. Deployment and curl output are marked as examples.
This post was created with AI assistance.
👉 Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator – it runs right in your browser.
Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en
Top comments (0)