- Book: Event-Driven Architecture Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Sooner or later a pull request like this lands in your queue. It deletes the S3 upload step, base64-encodes the invoice PDF into the message body, and the description reads "SQS takes 1 MiB now, we don't need the bucket."
The description is right about the limit. On August 4, 2025, AWS raised the SQS maximum message size from 256 KiB to 1 MiB for standard and FIFO queues, and updated the Lambda event source mapping to match. EventBridge followed on January 29, 2026, going from 256 KB to 1 MB in most commercial Regions. On September 18, 2026, SNS added 1 MiB payloads behind a new MaximumMessageSize topic attribute, and a topic set above 256 KiB can only deliver to SQS, Lambda and Data Firehose, with at most 100 subscriptions.
The bigger limit helps events that were squeezed at 256 KiB, and it tempts teams to use the queue as a file transport. The alternative is the claim-check pattern, and in TypeScript it fits in a few small files: a content hash, an idempotent consumer, both failure orders handled, and cleanup that nobody has to remember.
Why 1 MiB still doesn't make the file a good passenger
Four reasons, in the order they usually bite.
Binary has to be encoded. An SQS body can only contain XML, JSON and unformatted text, so a PDF goes in as base64. Base64 turns every 3 bytes into 4. At 256 KiB the cutoff was 192 KiB of binary; at 1 MiB it is 768 KiB, before any envelope fields.
You pay per 64 KB. The SQS pricing page bills "each 64 KB chunk of a payload" as one request, and its own example is a 1 MiB payload billed as 16 requests. A pointer is a few hundred bytes: one request. Fan the event out through SNS to several queues and every copy carries the full weight.
Retries move the whole thing. SQS standard queues deliver at least once. Every redelivery, every trip to the dead-letter queue and every redrive drags the file along with it.
The file ends up in places you didn't plan. Messages can sit in a queue for up to 14 days, then in a DLQ, and then in whatever log line prints the failed message body. A customer's invoice now has three retention policies, and you only chose one of them.
The claim check keeps the file in storage you control and sends a ticket. AWS ships this as the SQS Extended Client Library, but only for Java and Python. In TypeScript you write it yourself, and it is small enough that you should.
The envelope: a pointer with a hash
The message carries an event ID, a type, and either a small inline JSON payload or a claim check. A discriminated union makes the consumer handle both cases explicitly.
// envelope.ts
export type ClaimCheck = {
bucket: string;
key: string;
sha256: string; // hex digest of the payload bytes
size: number;
contentType: string;
};
type Base = { v: 1; eventId: string; type: string };
export type Envelope =
| (Base & { inline: string })
| (Base & { claim: ClaimCheck });
export function httpStatus(err: unknown): number | undefined {
return (err as { $metadata?: { httpStatusCode?: number } })
?.$metadata?.httpStatusCode;
}
The clients take an optional endpoint so the same code runs against AWS or a local emulator:
// clients.ts
import { S3Client } from "@aws-sdk/client-s3";
import { SQSClient } from "@aws-sdk/client-sqs";
const endpoint = process.env.LOCAL_AWS_ENDPOINT;
export const s3 = new S3Client(
endpoint ? { endpoint, forcePathStyle: true } : {},
);
export const sqs = new SQSClient(endpoint ? { endpoint } : {});
Two notes if you run this locally in late 2026. LocalStack moved to a single image, and since March 23, 2026 its latest releases need an auth token, with a free plan for hobbyists and individual developers. The MinIO community repository was archived on April 25, 2026 and ships source only, so check what your docker-compose.yml is actually pulling before you trust it.
The producer: payload first, pointer second
The ordering rule is the whole design: write the payload, wait for S3 to confirm it, and only then send the message. Start with the upload:
// producer.ts
import { createHash } from "node:crypto";
import { PutObjectCommand } from "@aws-sdk/client-s3";
import { SendMessageCommand } from "@aws-sdk/client-sqs";
import { s3, sqs } from "./clients";
import { httpStatus, type Envelope } from "./envelope";
const BUCKET = process.env.PAYLOAD_BUCKET!;
const QUEUE_URL = process.env.QUEUE_URL!;
async function putOnce(
key: string, body: Uint8Array,
sha256b64: string, contentType: string,
): Promise<void> {
for (let attempt = 1; ; attempt++) {
try {
await s3.send(new PutObjectCommand({
Bucket: BUCKET, Key: key, Body: body,
ContentType: contentType,
ChecksumSHA256: sha256b64, // S3 rejects a bad upload
IfNoneMatch: "*", // never overwrite
}));
return;
} catch (err) {
const status = httpStatus(err);
if (status === 412) return; // an earlier try wrote it
if (status === 409 && attempt < 3) continue;
throw err;
}
}
}
ChecksumSHA256 is the base64-encoded SHA-256 of the body. S3 recomputes it on arrival and refuses the write if they differ, so a truncated upload never becomes an object.
IfNoneMatch: "*" is a conditional write: the put succeeds only if the key doesn't exist yet. A 412 means it already does, and because publish (below) builds the key from both the event ID and the hash, "already exists" means "an earlier attempt of this exact publish got there first." Treat it as success. The PutObject reference says to retry on a 409 ConditionalRequestConflict, so the loop retries, up to three attempts in total. If you want every writer held to the rule, a bucket policy can require the conditional header on writes.
publish inlines small JSON and sends everything else through the bucket:
// producer.ts (cont.)
const INLINE_LIMIT = 64 * 1024; // about one SQS billing chunk
export async function publish(
eventId: string, type: string,
payload: Uint8Array, contentType: string,
): Promise<void> {
let env: Envelope;
if (contentType === "application/json"
&& payload.byteLength <= INLINE_LIMIT) {
const inline = Buffer.from(payload).toString("utf8");
env = { v: 1, eventId, type, inline };
} else {
const digest = createHash("sha256").update(payload).digest();
const hex = digest.toString("hex");
const key = `payloads/${eventId}/${hex}`;
await putOnce(key, payload, digest.toString("base64"),
contentType);
env = { v: 1, eventId, type, claim: {
bucket: BUCKET, key, sha256: hex,
size: payload.byteLength, contentType,
} };
}
await sqs.send(new SendMessageCommand({
QueueUrl: QUEUE_URL,
MessageBody: JSON.stringify(env),
}));
}
Why not a pure content-addressed key like payloads/<hash>? It looks tidier and deduplicates identical files. But with IfNoneMatch a second event carrying the same file never rewrites the object, so the object's age is the age of the first event. Your lifecycle rule (below) counts from that first write and can delete the file while the second event is still in the queue. One object per event keeps the clock honest.
The consumer: verify, dedupe, then delete
The consumer fetches the payload, checks the bytes against the hash in the pointer, applies the event inside a PostgreSQL transaction that also records the event ID, and only then deletes the message. The dedupe table needs a primary key on the event ID. That key is what makes ON CONFLICT fire; without it every insert succeeds and duplicates get processed silently.
CREATE TABLE processed_events (
event_id text PRIMARY KEY,
processed_at timestamptz NOT NULL DEFAULT now()
);
First, the fetch and the check:
// consumer.ts
import { createHash } from "node:crypto";
import { GetObjectCommand } from "@aws-sdk/client-s3";
import { Pool, type PoolClient } from "pg";
import { s3 } from "./clients";
import { httpStatus, type ClaimCheck, type Envelope }
from "./envelope";
export class PayloadUnreachable extends Error {}
export class PayloadCorrupt extends Error {}
async function fetchVerified(c: ClaimCheck): Promise<Uint8Array> {
let bytes: Uint8Array;
try {
const res = await s3.send(new GetObjectCommand({
Bucket: c.bucket, Key: c.key,
}));
bytes = await res.Body!.transformToByteArray();
} catch (err) {
const status = httpStatus(err);
if (status === 404 || status === 403) {
throw new PayloadUnreachable(`${c.key}: HTTP ${status}`);
}
throw err;
}
const hex = createHash("sha256").update(bytes).digest("hex");
if (hex !== c.sha256 || bytes.byteLength !== c.size) {
throw new PayloadCorrupt(`${c.key}: hash or size mismatch`);
}
return bytes;
}
Then the transaction. applyEvent stands in for your business logic; it gets the transaction client so its writes commit with the dedupe row.
// consumer.ts (cont.)
declare function applyEvent(
db: PoolClient, type: string, bytes: Uint8Array,
): Promise<void>;
const pool = new Pool();
export async function handle(env: Envelope): Promise<void> {
const bytes = "claim" in env
? await fetchVerified(env.claim)
: Buffer.from(env.inline, "utf8");
const db = await pool.connect();
try {
await db.query("BEGIN");
const seen = await db.query(
`INSERT INTO processed_events (event_id)
VALUES ($1) ON CONFLICT DO NOTHING`,
[env.eventId],
);
if (seen.rowCount === 0) { // duplicate delivery
await db.query("ROLLBACK");
return;
}
await applyEvent(db, env.type, bytes);
await db.query("COMMIT");
} catch (err) {
await db.query("ROLLBACK");
throw err;
} finally {
db.release();
}
}
The processed_events insert and the side effect commit together or not at all. If the process dies after COMMIT but before the message is deleted, the redelivered copy hits the conflict, does nothing, and gets deleted. The download happens before BEGIN, so no transaction stays open while a file streams in.
The poll loop deletes only on success and logs the message ID, never the body:
// poll.ts
import {
ReceiveMessageCommand, DeleteMessageCommand,
} from "@aws-sdk/client-sqs";
import { sqs } from "./clients";
import { handle } from "./consumer";
import type { Envelope } from "./envelope";
export async function poll(queueUrl: string): Promise<void> {
for (;;) {
const { Messages = [] } = await sqs.send(
new ReceiveMessageCommand({
QueueUrl: queueUrl,
MaxNumberOfMessages: 10,
WaitTimeSeconds: 20,
}),
);
for (const m of Messages) {
try {
await handle(JSON.parse(m.Body!) as Envelope);
await sqs.send(new DeleteMessageCommand({
QueueUrl: queueUrl, ReceiptHandle: m.ReceiptHandle!,
}));
} catch (err) {
// No delete: the message comes back after the
// visibility timeout; maxReceiveCount sends it to the DLQ.
console.error("event failed", m.MessageId, err);
}
}
}
}
The two failure orders
There are two ways the halves can come apart, and they deserve very different reactions.
Payload written, message lost. The put succeeds, then SendMessage throws or the process crashes. You have an object nobody points at. That is harmless. If the producer retries with the same event ID, putOnce gets a 412, treats it as done, and sends the message. If it never retries, the lifecycle rule deletes the orphan. This is the cheap failure, and the ordering rule exists to make sure it's the only one the producer can cause.
Message delivered, payload missing. The consumer gets a pointer and S3 answers 404. Don't wait it out. S3 delivers strong read-after-write consistency for all applications, so if the producer awaited the put before sending, a 404 is not a replication race. Something deleted the object, a lifecycle rule expired it too early, or a producer somewhere sends before it uploads. Also check your IAM policy: without s3:ListBucket, GetObject on a missing key returns 403 instead of 404, which is why fetchVerified treats both as unreachable.
The code throws, the message isn't deleted, and after maxReceiveCount receives it lands in the dead-letter queue with its pointer intact. Put an alarm on DLQ depth. A missing payload is data loss. You want the DLQ alarm to fire before the customer emails about the missing invoice. PayloadCorrupt follows the same path: retrying won't change the bytes, but the DLQ keeps the evidence.
Cleanup belongs to a lifecycle rule, not a consumer
It's tempting to delete the object right after COMMIT. Don't. With SNS fan-out, several queues hold pointers to the same object and no single consumer owns it. A duplicate delivery or a DLQ redrive would also find nothing to fetch. Let the bucket expire payloads on a schedule instead:
{
"Rules": [{
"ID": "expire-claim-check-payloads",
"Filter": { "Prefix": "payloads/" },
"Status": "Enabled",
"Expiration": { "Days": 30 },
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 1
}
}]
}
aws s3api put-bucket-lifecycle-configuration \
--bucket my-payload-bucket \
--lifecycle-configuration file://lifecycle.json
Size the expiration from the longest a pointer can live, not the average. SQS retention maxes out at 14 days. For standard queues, a message moved to a DLQ keeps its original enqueue timestamp, but for FIFO queues the timestamp resets, so a FIFO message can live 14 days in the source and another 14 in the DLQ. Thirty days covers that with room to spare. If your team redrives DLQs by hand weeks later, size for that too, or make the redrive script check that the object still exists.
Expiration is asynchronous: S3 queues expired objects for removal and there can be a delay, but you aren't charged for storage after the expiration date. The AbortIncompleteMultipartUpload line clears out half-finished multipart uploads, which AWS recommends so abandoned parts don't keep billing you.
The claim check isn't free. Each large event now costs an S3 PUT and a GET on top of a one-chunk SQS message, and you own a bucket policy. For a 2 KB order event, inline it and skip all of this, which is what INLINE_LIMIT does. For PDFs, images, exports and anything that might grow, send the pointer.
If this was useful
The claim check is one of a handful of patterns that decide whether an event-driven system survives its first bad week: the transactional outbox for the "DB committed, message never sent" case, idempotent consumers, sagas, and DLQs that someone actually reads. The Event-Driven Architecture Pocket Guide covers those, plus the traps that only show up in production. If you just wired a pointer through a queue, it's the next thing I'd read.
Top comments (0)