DEV Community

Cover image for S3 multipart upload with presigned URLs and AWS CDK
Sascha Mirow-Stanitzok
Sascha Mirow-Stanitzok

Posted on Originally published at sascha-ms.de

S3 multipart upload with presigned URLs and AWS CDK

An S3 multipart upload is the answer as soon as files get too big for a single request. In my article on the S3 presigned URL I showed how the browser pushes files straight into a bucket without them passing through the Lambda function. That works great for profile pictures and PDFs. With a 3 GB audiobook, a video recording or a data export, things look different: a single PUT can be at most 5 GB, and if the connection drops at 90 percent, everything starts over.

This topic keeps me busy because most guides stop exactly where it gets tricky. They show how to sign one URL per part and leave open who actually controls the file size, what happens to half-finished uploads, and why the browser suddenly can’t see an ETag. So here we build the complete pattern with AWS CDK and TypeScript: four small endpoints, a DynamoDB table, a lifecycle rule, and a browser client that uploads parts in parallel and retries failures on its own.

In short

With an S3 multipart upload, the client splits a large file into parts, uploads each part through its own presigned URL straight to S3, and your backend assembles the parts at the end. The key insight: if the server signs the size of every part and counts the parts in S3 itself before completing, the client can neither cheat nor forget anything. The biggest pitfall is incomplete uploads that sit invisibly in the bucket and cost money until a lifecycle rule removes them.

What you will learn in this article

  • When you need an S3 multipart upload and which limits S3 sets
  • How to calculate a sensible part size and part count
  • How Lambda signs fixed-size part URLs and completes the upload safely
  • How a browser client uploads parts in parallel and retries errors
  • Which errors are common and how a lifecycle rule prevents costs

When do you need an S3 multipart upload?

AWS recommends multipart uploads from around 100 MB. That’s a rule of thumb, not a hard limit. From this size on, the benefits become obvious. A failed part is retried on its own, several parts run in parallel and make better use of the bandwidth, and you can resume an upload later because S3 remembers every finished part. From 5 GB you no longer have a choice anyway, since a single PUT can’t be any larger.

The limits you have to plan for are manageable. These values come from the S3 documentation (as of November 2026):

Limit Value
Largest possible object 48.8 TiB
Parts per upload up to 10,000, numbered 1 to 10,000
Size of a part 5 MiB to 5 GiB, only the last part may be smaller
Parts per ListParts response at most 1,000
Single PUT without multipart at most 5 GB

For small files, the simple approach with a single presigned URL remains the better choice. It needs less code and fewer requests. In practice I would draw the line at 100 MB and stick with the simple PUT below that.

How an S3 multipart upload works

The flow consists of three S3 calls. With CreateMultipartUpload you register the upload and get an upload ID back. After that, you upload any number of parts with UploadPart, each with a part number. For every part, S3 returns an ETag, a kind of fingerprint of that part. Finally, CompleteMultipartUpload sends the list of all part numbers with their ETags, and only then does S3 build an object from them. Until that point, there is no file in the bucket that anyone could download.

S3 multipart upload diagram: browser sends parts straight to S3, Lambda signs URLs and checks with ListParts

The architecture: Lambda only manages plan and status, the bytes go straight from the browser to S3.

Here’s the trick with presigned URLs: the parts don’t have to go through your backend. The Lambda function signs a separate URL for each part, and the browser sends the bytes directly to S3. A presigned URL is a link that allows one specific S3 action for a short time, without the client needing AWS credentials of its own. Your function only handles a few small JSON requests while gigabytes flow past it.

One quirk you should know before you start: parts of an unfinished upload stay in the bucket until someone completes or aborts the upload. You pay for that storage, but you don’t see the parts as objects in the S3 console. If you forget about this, invisible gigabytes pile up over the months.

How large should the parts be?

Part size is a trade-off. Small parts mean little loss when one fails, but many requests. Large parts save requests, yet cost more time when something breaks. I start with 16 MiB. A 3 GiB file then has exactly 192 parts, and even over a mediocre connection a retried part finishes in a few seconds.

The upper limit of 10,000 parts also matters. At 16 MiB per part you get to about 156 GiB. For larger files the part size has to grow, and the planParts function further down handles that automatically: it divides the file size by 10,000, rounds up to whole MiB and takes the larger of the two values.

The architecture with four endpoints

In many examples, the client sends its collected ETags to the server at the end, and the server passes them on to S3 unchecked. That works, but it leaves two questions open: how big is the file really, and did the client upload every part? That’s why I would build it a little differently. When the upload starts, the server stores in a DynamoDB table how large the file should be and how many parts it splits into. Everything else follows from that:

  • To start, POST /uploads checks file type and size, starts the multipart upload and creates a record.
  • Next, POST /uploads/{id}/parts signs URLs for specific part numbers, each with the exact length of that part.
  • At the end, POST /uploads/{id}/complete asks S3 via ListParts which parts have arrived, compares count and sizes, and only then completes.
  • If something goes wrong, DELETE /uploads/{id} aborts the upload and frees the storage.

Why the server signs the part size

The decisive trick sits in the second endpoint. If you pass a ContentLength to the SDK when signing, the content-length header ends up in the signature. If the browser then sends a part with a different size, the signature no longer matches and S3 rejects the request. So the client can’t simply stuff 5 GB into every part. Together with the size check on completion, the file size is controlled by the server, even though the bytes never pass through it.

🎬 Flow of an S3 multipart upload with presigned URLs – this animated diagram only runs on my site: watch it on sascha-ms.de

The example is a fictional audiobook archive of a city library. Volunteers upload recordings there as MP3, M4B or ZIP, which quickly grow to several gigabytes. For this article I limit files to 20 GiB; that’s a rule of the app, not of S3.

Step by step: S3 multipart upload with AWS CDK

Prerequisites

  • Node.js 22 or newer and npm
  • An AWS account with permissions for CloudFormation, Lambda, S3, DynamoDB and API Gateway (only for deployment, the tests run without it)
  • A one-time cdk bootstrap in the target region
  • Basic TypeScript knowledge; I explain CDK along the way

Step 1: Set up the project

We start with an empty folder and install the packages. AWS CDK describes infrastructure as TypeScript code and generates a CloudFormation template from it. I tested with aws-cdk-lib 2.276, AWS SDK 3.1167, TypeScript 7.0 and Vitest 5.1.

mkdir audiobook-upload && cd hoerbuch-upload
npm init -y
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb
npm install -D aws-cdk aws-cdk-lib constructs esbuild tsx typescript vitest @types/node @types/aws-lambda
mkdir bin lib src test web
Enter fullscreen mode Exit fullscreen mode
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "lib": ["ES2022", "DOM"],
    "types": ["node"],
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "noEmit": true
  },
  "include": ["bin", "lib", "src", "test", "web", "vitest.config.ts"]
}
Enter fullscreen mode Exit fullscreen mode
{
  "app": "npx tsx bin/app.ts"
}
Enter fullscreen mode Exit fullscreen mode

In package.json, also add "type": "module" and the scripts "test": "vitest run" and "build": "tsc --noEmit". The full file is in the example repository.

Step 2: Plan parts and validate input

The logic without AWS calls goes into its own file. That way you can cover it with simple unit tests. partLength calculates the length of a single part: all parts have the same size, only the last one takes the rest.

// src/plan.ts
// Pure logic without AWS calls: validate input and plan parts.
// This makes it testable without mocks.

export const MiB = 1024 * 1024;
export const MIN_PART_SIZE = 5 * MiB; // S3 minimum for every part except the last
export const MAX_PARTS = 10_000; // S3 maximum per multipart upload
export const DEFAULT_PART_SIZE = 16 * MiB;

// Limit of our example app (audiobook archive), not of S3
export const MAX_FILE_SIZE = 20 * 1024 * MiB; // 20 GiB

const ALLOWED_TYPES = new Set([
  "audio/mpeg",
  "audio/mp4",
  "audio/x-m4b",
  "application/zip",
]);

export interface UploadRequest {
  fileName: string;
  contentType: string;
  size: number;
}

export interface UploadPlan {
  partSize: number;
  partCount: number;
}

export class ValidationError extends Error {}

export function validateRequest(input: unknown): UploadRequest {
  const body = (input ?? {}) as Partial<UploadRequest>;
  if (typeof body.fileName !== "string" || body.fileName.trim() === "") {
    throw new ValidationError("fileName is missing");
  }
  if (typeof body.contentType !== "string" || !ALLOWED_TYPES.has(body.contentType)) {
    throw new ValidationError("contentType not allowed");
  }
  if (!Number.isInteger(body.size) || (body.size as number) <= 0) {
    throw new ValidationError("size must be a positive integer");
  }
  if ((body.size as number) > MAX_FILE_SIZE) {
    throw new ValidationError("File is too large");
  }
  return { fileName: body.fileName, contentType: body.contentType, size: body.size as number };
}

// Picks the part size so that there are at most MAX_PARTS parts.
// Rounds up to whole MiB to keep the math in the client simple.
export function planParts(size: number, preferred = DEFAULT_PART_SIZE): UploadPlan {
  const needed = Math.ceil(size / MAX_PARTS);
  const partSize = Math.max(preferred, MIN_PART_SIZE, Math.ceil(needed / MiB) * MiB);
  return { partSize, partCount: Math.max(1, Math.ceil(size / partSize)) };
}

// "My Audiobook (Part 1).m4b" becomes "my-audiobook-part-1-.m4b" – only safe characters in the key
export function safeFileName(name: string): string {
  const cleaned = name
    .toLowerCase()
    .replace(/[^a-z0-9._-]+/g, "-")
    .replace(/^-+/, "")
    .slice(0, 100);
  return cleaned === "" ? "file" : cleaned;
}

export function assertPartNumbers(input: unknown, partCountLimit = MAX_PARTS): number[] {
  if (!Array.isArray(input) || input.length === 0 || input.length > 100) {
    throw new ValidationError("partNumbers: 1 to 100 entries");
  }
  for (const n of input) {
    if (!Number.isInteger(n) || n < 1 || n > partCountLimit) {
      throw new ValidationError(`invalid part number: ${String(n)}`);
    }
  }
  return input as number[];
}

// Length of a part: all parts have partSize, only the last one is shorter
export function partLength(plan: UploadPlan & { size: number }, partNumber: number): number {
  if (partNumber < plan.partCount) return plan.partSize;
  return plan.size - plan.partSize * (plan.partCount - 1);
}
Enter fullscreen mode Exit fullscreen mode

Step 3: Store the upload state in DynamoDB

Each upload gets a record with its own ID. The client never sees the S3 upload ID or the object key; it only knows our UUID. Through expiresAt, DynamoDB TTL cleans up old items after one day. TTL stands for time to live: DynamoDB deletes items automatically once the timestamp in that field is reached. The close method uses a condition so that an upload can’t be completed twice or aborted after completion.

// src/store.ts
// Small abstraction over DynamoDB so the handlers stay testable without AWS.
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, GetCommand, PutCommand, UpdateCommand } from "@aws-sdk/lib-dynamodb";

export type UploadStatus = "OPEN" | "COMPLETED" | "ABORTED";

export interface UploadRecord {
  id: string;
  s3UploadId: string;
  key: string;
  contentType: string;
  size: number;
  partSize: number;
  partCount: number;
  status: UploadStatus;
  expiresAt: number; // Unix seconds, DynamoDB TTL removes old items
}

export interface UploadStore {
  put(record: UploadRecord): Promise<void>;
  get(id: string): Promise<UploadRecord | undefined>;
  // Sets the status only while the upload is still open; returns false otherwise
  close(id: string, status: Exclude<UploadStatus, "OPEN">): Promise<boolean>;
}

export function dynamoStore(tableName: string, client = new DynamoDBClient({})): UploadStore {
  const doc = DynamoDBDocumentClient.from(client);
  return {
    async put(record) {
      await doc.send(
        new PutCommand({
          TableName: tableName,
          Item: record,
          ConditionExpression: "attribute_not_exists(id)",
        }),
      );
    },
    async get(id) {
      const res = await doc.send(new GetCommand({ TableName: tableName, Key: { id } }));
      return res.Item as UploadRecord | undefined;
    },
    async close(id, status) {
      try {
        await doc.send(
          new UpdateCommand({
            TableName: tableName,
            Key: { id },
            UpdateExpression: "SET #s = :new",
            ConditionExpression: "#s = :open",
            ExpressionAttributeNames: { "#s": "status" },
            ExpressionAttributeValues: { ":new": status, ":open": "OPEN" },
          }),
        );
        return true;
      } catch (err) {
        if ((err as Error).name === "ConditionalCheckFailedException") return false;
        throw err;
      }
    },
  };
}
Enter fullscreen mode Exit fullscreen mode

Step 4: Write the Lambda handlers

The handlers are created in a factory function that receives the S3 client and the store as parameters. That lets you feed them fakes in tests, without a mocking framework. Pay attention to two places while reading: ContentLength when signing, and the loop with ListParts on completion. That loop is needed because S3 returns at most 1,000 parts per response.

// src/handlers.ts
// Four endpoints for the multipart upload: start, sign parts, complete, abort.
import { randomUUID } from "node:crypto";
import {
  AbortMultipartUploadCommand,
  CompleteMultipartUploadCommand,
  CreateMultipartUploadCommand,
  ListPartsCommand,
  type Part,
  S3Client,
  UploadPartCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import type { APIGatewayProxyEventV2, APIGatewayProxyStructuredResultV2 } from "aws-lambda";
import {
  assertPartNumbers,
  partLength,
  planParts,
  safeFileName,
  ValidationError,
  validateRequest,
} from "./plan.js";
import type { UploadRecord, UploadStore } from "./store.js";

type Result = APIGatewayProxyStructuredResultV2;
type Handler = (event: APIGatewayProxyEventV2) => Promise<Result>;

const URL_TTL_SECONDS = 15 * 60; // each part URL is valid for 15 minutes
const RECORD_TTL_SECONDS = 24 * 60 * 60; // matches the lifecycle rule on the bucket

const json = (statusCode: number, body: unknown): Result => ({
  statusCode,
  headers: { "content-type": "application/json" },
  body: JSON.stringify(body),
});

function parseBody(event: APIGatewayProxyEventV2): unknown {
  if (!event.body) return {};
  const raw = event.isBase64Encoded ? Buffer.from(event.body, "base64").toString("utf8") : event.body;
  try {
    return JSON.parse(raw);
  } catch {
    throw new ValidationError("Body is not valid JSON");
  }
}

// Catches validation errors and turns them into a 400
const safe =
  (fn: Handler): Handler =>
  async (event) => {
    try {
      return await fn(event);
    } catch (err) {
      if (err instanceof ValidationError) return json(400, { message: err.message });
      console.error(err);
      return json(500, { message: "Internal error" });
    }
  };

export interface Deps {
  s3: S3Client;
  store: UploadStore;
  bucket: string;
  now?: () => number;
}

export function createHandlers({ s3, store, bucket, now = Date.now }: Deps) {
  // Returns either the open upload or a ready-made error response
  async function loadOpen(
    event: APIGatewayProxyEventV2,
  ): Promise<{ record: UploadRecord } | { error: Result }> {
    const id = event.pathParameters?.id ?? "";
    const record = await store.get(id);
    if (!record) return { error: json(404, { message: "Upload not found" }) };
    if (record.status !== "OPEN") {
      return { error: json(409, { message: `Upload is ${record.status}` }) };
    }
    return { record };
  }

  const start = safe(async (event) => {
    const req = validateRequest(parseBody(event));
    const plan = planParts(req.size);
    const id = randomUUID();
    const key = `uploads/${id}/${safeFileName(req.fileName)}`;

    const created = await s3.send(
      new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: req.contentType }),
    );

    await store.put({
      id,
      s3UploadId: created.UploadId!,
      key,
      contentType: req.contentType,
      size: req.size,
      ...plan,
      status: "OPEN",
      expiresAt: Math.floor(now() / 1000) + RECORD_TTL_SECONDS,
    });

    return json(201, { id, key, ...plan });
  });

  const signParts = safe(async (event) => {
    const found = await loadOpen(event);
    if ("error" in found) return found.error;
    const { record } = found;
    const body = parseBody(event) as { partNumbers?: unknown };
    const numbers = assertPartNumbers(body.partNumbers, record.partCount);

    const urls = await Promise.all(
      numbers.map(async (partNumber) => {
        const contentLength = partLength(record, partNumber);
        const url = await getSignedUrl(
          s3,
          new UploadPartCommand({
            Bucket: bucket,
            Key: record.key,
            UploadId: record.s3UploadId,
            PartNumber: partNumber,
            // becomes part of the signature: S3 rejects parts with a different size
            ContentLength: contentLength,
          }),
          { expiresIn: URL_TTL_SECONDS },
        );
        return { partNumber, contentLength, url };
      }),
    );
    return json(200, { parts: urls });
  });

  const complete = safe(async (event) => {
    const found = await loadOpen(event);
    if ("error" in found) return found.error;
    const { record } = found;

    // We get the ETags straight from S3 instead of trusting the client
    const uploaded: Part[] = [];
    let marker: string | undefined;
    do {
      const page = await s3.send(
        new ListPartsCommand({
          Bucket: bucket,
          Key: record.key,
          UploadId: record.s3UploadId,
          PartNumberMarker: marker,
        }),
      );
      uploaded.push(...(page.Parts ?? []));
      marker = page.IsTruncated ? page.NextPartNumberMarker : undefined;
    } while (marker);

    const byNumber = new Map(uploaded.map((p) => [p.PartNumber!, p]));
    const missing: number[] = [];
    for (let n = 1; n <= record.partCount; n++) {
      const part = byNumber.get(n);
      if (!part || part.Size !== partLength(record, n)) missing.push(n);
    }
    if (missing.length > 0) {
      return json(409, { message: "Parts are missing or have the wrong size", missing });
    }

    const result = await s3.send(
      new CompleteMultipartUploadCommand({
        Bucket: bucket,
        Key: record.key,
        UploadId: record.s3UploadId,
        MultipartUpload: {
          Parts: [...byNumber.values()]
            .filter((p) => p.PartNumber! <= record.partCount)
            .sort((a, b) => a.PartNumber! - b.PartNumber!)
            .map((p) => ({ PartNumber: p.PartNumber, ETag: p.ETag })),
        },
      }),
    );
    await store.close(record.id, "COMPLETED");
    return json(200, { key: record.key, etag: result.ETag });
  });

  const abort = safe(async (event) => {
    const found = await loadOpen(event);
    if ("error" in found) return found.error;
    const { record } = found;
    await s3.send(
      new AbortMultipartUploadCommand({ Bucket: bucket, Key: record.key, UploadId: record.s3UploadId }),
    );
    await store.close(record.id, "ABORTED");
    return { statusCode: 204 };
  });

  return { start, signParts, complete, abort };
}
Enter fullscreen mode Exit fullscreen mode

The entry point wires up the real clients. The setting requestChecksumCalculation: "WHEN_REQUIRED" is not a detail: without it, the SDK has been adding a CRC32 checksum of the empty body to every signed URL since version 3.729.0, and S3 rejects the real part. This applies to UploadPart just like it does to the simple PUT.

// src/index.ts
// Lambda entry point: wire up real clients, export handlers.
import { S3Client } from "@aws-sdk/client-s3";
import { createHandlers } from "./handlers.js";
import { dynamoStore } from "./store.js";

// Without WHEN_REQUIRED the SDK signs a CRC32 checksum of the empty body into the URL
const s3 = new S3Client({ requestChecksumCalculation: "WHEN_REQUIRED" });

const handlers = createHandlers({
  s3,
  store: dynamoStore(process.env.TABLE_NAME!),
  bucket: process.env.BUCKET_NAME!,
});

export const start = handlers.start;
export const signParts = handlers.signParts;
export const complete = handlers.complete;
export const abort = handlers.abort;
Enter fullscreen mode Exit fullscreen mode

Step 5: Bucket, table and API in the CDK stack

This stack holds the three settings that decide between success and trouble. First, the CORS rule allows PUT from your web app and exposes the ETag. Second, a lifecycle rule aborts incomplete uploads after one day. On top of that, each function only gets the IAM permissions it needs. CreateMultipartUpload and UploadPart run under the s3:PutObject permission, and for ListParts you need s3:ListMultipartUploadParts.

// lib/upload-stack.ts
import * as path from "node:path";
import { fileURLToPath } from "node:url";
import { CfnOutput, Duration, RemovalPolicy, Stack, type StackProps } from "aws-cdk-lib";
import { CorsHttpMethod, HttpApi, HttpMethod } from "aws-cdk-lib/aws-apigatewayv2";
import { HttpLambdaIntegration } from "aws-cdk-lib/aws-apigatewayv2-integrations";
import { AttributeType, Billing, TableV2 } from "aws-cdk-lib/aws-dynamodb";
import { PolicyStatement } from "aws-cdk-lib/aws-iam";
import { Architecture, Runtime } from "aws-cdk-lib/aws-lambda";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import { BlockPublicAccess, Bucket, BucketEncryption, HttpMethods } from "aws-cdk-lib/aws-s3";
import type { Construct } from "constructs";

const here = path.dirname(fileURLToPath(import.meta.url));

export interface UploadStackProps extends StackProps {
  // Origin of the web app that may upload, e.g. https://archive.example.org
  allowedOrigin: string;
}

export class AudiobookUploadStack extends Stack {
  constructor(scope: Construct, id: string, props: UploadStackProps) {
    super(scope, id, props);

    const bucket = new Bucket(this, "AudiobookBucket", {
      blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
      encryption: BucketEncryption.S3_MANAGED,
      enforceSSL: true,
      cors: [
        {
          allowedOrigins: [props.allowedOrigin],
          allowedMethods: [HttpMethods.PUT],
          allowedHeaders: ["*"],
          // Without ETag in the CORS header the browser cannot read it from the UploadPart response
          exposedHeaders: ["ETag"],
          maxAge: 3000,
        },
      ],
      lifecycleRules: [
        // Remove abandoned uploads after one day, otherwise the parts keep costing money
        { abortIncompleteMultipartUploadAfter: Duration.days(1) },
      ],
      removalPolicy: RemovalPolicy.RETAIN,
    });

    const table = new TableV2(this, "UploadTable", {
      partitionKey: { name: "id", type: AttributeType.STRING },
      billing: Billing.onDemand(),
      timeToLiveAttribute: "expiresAt",
      removalPolicy: RemovalPolicy.DESTROY,
    });

    const fn = (name: string, handler: string) =>
      new NodejsFunction(this, name, {
        entry: path.join(here, "../src/index.ts"),
        handler,
        runtime: Runtime.NODEJS_24_X,
        architecture: Architecture.ARM_64,
        memorySize: 256,
        timeout: Duration.seconds(10),
        environment: { BUCKET_NAME: bucket.bucketName, TABLE_NAME: table.tableName },
        // Bundle the SDK so the tested version runs, not the one from the runtime
        bundling: { externalModules: [], minify: true },
      });

    const startFn = fn("StartUpload", "start");
    const partsFn = fn("SignParts", "signParts");
    const completeFn = fn("CompleteUpload", "complete");
    const abortFn = fn("AbortUpload", "abort");

    // Least privilege: each function only gets what it really needs
    const objects = bucket.arnForObjects("uploads/*");
    startFn.addToRolePolicy(new PolicyStatement({ actions: ["s3:PutObject"], resources: [objects] }));
    partsFn.addToRolePolicy(new PolicyStatement({ actions: ["s3:PutObject"], resources: [objects] }));
    completeFn.addToRolePolicy(
      new PolicyStatement({
        actions: ["s3:PutObject", "s3:ListMultipartUploadParts"],
        resources: [objects],
      }),
    );
    abortFn.addToRolePolicy(
      new PolicyStatement({ actions: ["s3:AbortMultipartUpload"], resources: [objects] }),
    );
    table.grantReadWriteData(startFn);
    table.grantReadData(partsFn);
    table.grantReadWriteData(completeFn);
    table.grantReadWriteData(abortFn);

    const api = new HttpApi(this, "UploadApi", {
      corsPreflight: {
        allowOrigins: [props.allowedOrigin],
        allowMethods: [CorsHttpMethod.POST, CorsHttpMethod.DELETE],
        allowHeaders: ["content-type", "authorization"],
      },
    });
    const route = (p: string, method: HttpMethod, target: NodejsFunction) =>
      api.addRoutes({
        path: p,
        methods: [method],
        integration: new HttpLambdaIntegration(`${target.node.id}Integration`, target),
      });

    route("/uploads", HttpMethod.POST, startFn);
    route("/uploads/{id}/parts", HttpMethod.POST, partsFn);
    route("/uploads/{id}/complete", HttpMethod.POST, completeFn);
    route("/uploads/{id}", HttpMethod.DELETE, abortFn);

    new CfnOutput(this, "ApiUrl", { value: api.apiEndpoint });
    new CfnOutput(this, "BucketName", { value: bucket.bucketName });
  }
}
Enter fullscreen mode Exit fullscreen mode
// bin/app.ts
import { App } from "aws-cdk-lib";
import { AudiobookUploadStack } from "../lib/upload-stack.js";

const app = new App();
new AudiobookUploadStack(app, "AudiobookUploadStack", {
  allowedOrigin: app.node.tryGetContext("allowedOrigin") ?? "http://localhost:5173",
});
Enter fullscreen mode Exit fullscreen mode

With externalModules: [], esbuild bundles the AWS SDK into the function. By default, the Lambda runtime would use its own SDK version, and then you don’t know exactly how signing behaves. The package grows to about 730 KB, which is no problem for cold starts.

Step 6: Test the S3 multipart upload without an AWS account

Signing works completely offline as long as the S3 client has some credentials. So in the test you can generate real URLs and check whether content-length really is part of the signature. Only send() is replaced with a small fake function.

// test/handlers.test.ts
// Test the handlers with a fake S3 and an in-memory store – no AWS account needed.
import { S3Client } from "@aws-sdk/client-s3";
import type { APIGatewayProxyEventV2 } from "aws-lambda";
import { describe, expect, it } from "vitest";
import { createHandlers } from "../src/handlers.js";
import { MiB } from "../src/plan.js";
import type { UploadRecord, UploadStore } from "../src/store.js";

function memoryStore(): UploadStore & { items: Map<string, UploadRecord> } {
  const items = new Map<string, UploadRecord>();
  return {
    items,
    async put(r) {
      items.set(r.id, { ...r });
    },
    async get(id) {
      return items.get(id);
    },
    async close(id, status) {
      const r = items.get(id);
      if (!r || r.status !== "OPEN") return false;
      r.status = status;
      return true;
    },
  };
}

// Real S3Client with dummy credentials: signing works offline,
// we replace send() with a fake implementation.
function fakeS3(parts: { PartNumber: number; Size: number; ETag: string }[]) {
  const s3 = new S3Client({
    region: "eu-central-1",
    credentials: { accessKeyId: "AKIDEXAMPLE", secretAccessKey: "example" },
    requestChecksumCalculation: "WHEN_REQUIRED",
  });
  const calls: string[] = [];
  s3.send = (async (cmd: { constructor: { name: string } }) => {
    const name = cmd.constructor.name;
    calls.push(name);
    if (name === "CreateMultipartUploadCommand") return { UploadId: "s3-upload-1" };
    if (name === "ListPartsCommand") return { Parts: parts, IsTruncated: false };
    if (name === "CompleteMultipartUploadCommand") return { ETag: '"abc-3"' };
    return {};
  }) as unknown as S3Client["send"];
  return { s3, calls };
}

const event = (body: unknown, id?: string) =>
  ({ body: JSON.stringify(body), isBase64Encoded: false, pathParameters: id ? { id } : undefined }) as unknown as APIGatewayProxyEventV2;

const size = 40 * MiB + 123;

describe("Multipart upload handlers", () => {
  it("starts an upload and plans three parts", async () => {
    const store = memoryStore();
    const { s3 } = fakeS3([]);
    const h = createHandlers({ s3, store, bucket: "audiobooks" });

    const res = await h.start(event({ fileName: "Mystery.m4b", contentType: "audio/x-m4b", size }));
    const body = JSON.parse(res.body as string);

    expect(res.statusCode).toBe(201);
    expect(body.partCount).toBe(3);
    expect(store.items.get(body.id)?.s3UploadId).toBe("s3-upload-1");
  });

  it("signs the size of each part, without a CRC32 checksum", async () => {
    const store = memoryStore();
    const { s3 } = fakeS3([]);
    const h = createHandlers({ s3, store, bucket: "audiobooks" });
    const { id } = JSON.parse((await h.start(event({ fileName: "a.m4b", contentType: "audio/x-m4b", size }))).body as string);

    const res = await h.signParts(event({ partNumbers: [3] }, id));
    const [part] = JSON.parse(res.body as string).parts;
    const url = new URL(part.url);

    expect(part.contentLength).toBe(8 * MiB + 123);
    expect(url.searchParams.get("X-Amz-SignedHeaders")).toContain("content-length");
    expect(url.searchParams.has("x-amz-checksum-crc32")).toBe(false);
  });

  it("refuses to complete when a part is missing", async () => {
    const store = memoryStore();
    const { s3, calls } = fakeS3([
      { PartNumber: 1, Size: 16 * MiB, ETag: '"e1"' },
      { PartNumber: 3, Size: 8 * MiB + 123, ETag: '"e3"' },
    ]);
    const h = createHandlers({ s3, store, bucket: "audiobooks" });
    const { id } = JSON.parse((await h.start(event({ fileName: "a.m4b", contentType: "audio/x-m4b", size }))).body as string);

    const res = await h.complete(event({}, id));

    expect(res.statusCode).toBe(409);
    expect(JSON.parse(res.body as string).missing).toEqual([2]);
    expect(calls).not.toContain("CompleteMultipartUploadCommand");
  });

  it("completes the upload when all parts match", async () => {
    const store = memoryStore();
    const { s3 } = fakeS3([
      { PartNumber: 1, Size: 16 * MiB, ETag: '"e1"' },
      { PartNumber: 2, Size: 16 * MiB, ETag: '"e2"' },
      { PartNumber: 3, Size: 8 * MiB + 123, ETag: '"e3"' },
    ]);
    const h = createHandlers({ s3, store, bucket: "audiobooks" });
    const { id } = JSON.parse((await h.start(event({ fileName: "a.m4b", contentType: "audio/x-m4b", size }))).body as string);

    const res = await h.complete(event({}, id));

    expect(res.statusCode).toBe(200);
    expect(store.items.get(id)?.status).toBe("COMPLETED");
    // A second completion is rejected
    expect((await h.complete(event({}, id))).statusCode).toBe(409);
  });
});
Enter fullscreen mode Exit fullscreen mode

The repository also contains tests for planParts, for the browser client and for the stack. The stack test uses the CDK assertions to check that the lifecycle rule and the CORS setting end up in the template. You’ll find the details in the article testing CDK stacks with Vitest.

npx tsc --noEmit
npm test
Enter fullscreen mode Exit fullscreen mode

Expected result:

 ✓ test/plan.test.ts (6 tests)
 ✓ test/handlers.test.ts (4 tests)
 ✓ test/upload-client.test.ts (1 test)
 ✓ test/stack.test.ts (3 tests)

 Test Files  4 passed (4)
      Tests  14 passed (14)
Enter fullscreen mode Exit fullscreen mode

Step 7: Deploy and try it with curl

When deploying, pass the origin of your web app as a context value. Without it, the stack only allows http://localhost:5173, which is handy for local development.

npx cdk deploy -c allowedOrigin=https://archive.example.org
# Note the output: AudiobookUploadStack.ApiUrl = https://…execute-api…amazonaws.com
API=https://YOUR-API-ID.execute-api.eu-central-1.amazonaws.com

# Create a 20 MiB test file
head -c 20971520 /dev/urandom > mystery.m4b

curl -s -X POST "$API/uploads" \
  -H "content-type: application/json" \
  -d '{"fileName":"Mystery.m4b","contentType":"audio/x-m4b","size":20971520}'
Enter fullscreen mode Exit fullscreen mode

The response contains the ID and the plan, for example {"id":"3f0c…","key":"uploads/3f0c…/mystery.m4b","partSize":16777216,"partCount":2}. Next, you get a URL signed, upload a part and complete:

ID=THE-ID-FROM-THE-RESPONSE

# Get the URL for part 1 (16 MiB)
URL=$(curl -s -X POST "$API/uploads/$ID/parts" \
  -H "content-type: application/json" \
  -d '{"partNumbers":[1]}' | node -pe 'JSON.parse(require("fs").readFileSync(0)).parts[0].url')

# Upload the first 16 MiB
head -c 16777216 mystery.m4b | curl -s -X PUT --data-binary @- "$URL" -o /dev/null -w "%{http_code}\n"

# Do the same for part 2 (the remaining 4 MiB), then complete
curl -s -X POST "$API/uploads/$ID/complete" -H "content-type: application/json" -d '{}'
Enter fullscreen mode Exit fullscreen mode

With a small 20 MiB test file you get two parts: 16 MiB and 4 MiB. If you leave out the second part, complete answers with status 409 and "missing":[2]. That’s exactly how it should be.

Step 8: The browser client with parallel parts

The client cuts the file into pieces with Blob.slice. This doesn’t copy any data into memory; the browser only reads the bytes when sending. Four workers take part numbers from a queue one after the other. If a part fails, the client fetches a fresh URL and tries again after a short pause. If nothing works anymore, it aborts the upload so that no leftovers remain.

// web/upload-client.ts
// Browser client: slice the file into parts, upload in parallel, retry on errors.

interface StartResponse {
  id: string;
  key: string;
  partSize: number;
  partCount: number;
}

interface SignedPart {
  partNumber: number;
  contentLength: number;
  url: string;
}

export interface UploadOptions {
  apiBase: string; // e.g. https://abc123.execute-api.eu-central-1.amazonaws.com
  concurrency?: number; // how many parts run at the same time
  maxRetries?: number;
  onProgress?: (uploadedBytes: number, totalBytes: number) => void;
  fetchFn?: typeof fetch; // replaceable for tests
}

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

export async function uploadLargeFile(file: Blob & { name: string }, opts: UploadOptions) {
  const { apiBase, concurrency = 4, maxRetries = 3, onProgress, fetchFn = fetch } = opts;

  const post = async <T>(path: string, body: unknown): Promise<T> => {
    const res = await fetchFn(`${apiBase}${path}`, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify(body),
    });
    if (!res.ok) throw new Error(`${path}: HTTP ${res.status}`);
    return (await res.json()) as T;
  };

  // 1. Register the upload
  const upload = await post<StartResponse>("/uploads", {
    fileName: file.name,
    contentType: file.type,
    size: file.size,
  });

  const sign = async (partNumbers: number[]) =>
    (await post<{ parts: SignedPart[] }>(`/uploads/${upload.id}/parts`, { partNumbers })).parts;

  let uploadedBytes = 0;
  const queue = Array.from({ length: upload.partCount }, (_, i) => i + 1);

  // 2. Upload one part – with a fresh URL on every attempt
  const uploadPart = async (partNumber: number) => {
    const start = (partNumber - 1) * upload.partSize;
    const chunk = file.slice(start, start + upload.partSize);
    for (let attempt = 0; ; attempt++) {
      try {
        const [signed] = await sign([partNumber]);
        const res = await fetchFn(signed.url, { method: "PUT", body: chunk });
        if (!res.ok) throw new Error(`Part ${partNumber}: HTTP ${res.status}`);
        uploadedBytes += chunk.size;
        onProgress?.(uploadedBytes, file.size);
        return;
      } catch (err) {
        if (attempt >= maxRetries) throw err;
        await sleep(2 ** attempt * 500); // 0.5 s, 1 s, 2 s …
      }
    }
  };

  // 3. Several "workers" pull part numbers from the queue
  try {
    const worker = async () => {
      for (let n = queue.shift(); n !== undefined; n = queue.shift()) await uploadPart(n);
    };
    await Promise.all(Array.from({ length: Math.min(concurrency, upload.partCount) }, worker));

    // 4. Complete – the server checks count and size of all parts
    return await post<{ key: string; etag: string }>(`/uploads/${upload.id}/complete`, {});
  } catch (err) {
    // Clean up so no orphaned parts cost storage
    await fetchFn(`${apiBase}/uploads/${upload.id}`, { method: "DELETE" }).catch(() => undefined);
    throw err;
  }
}
Enter fullscreen mode Exit fullscreen mode

Wiring it up takes only a few lines, for example on a file input:

// web/main.ts
import { uploadLargeFile } from "./upload-client.js";

const input = document.querySelector<HTMLInputElement>("#audiobook")!;
const status = document.querySelector<HTMLOutputElement>("#status")!;

input.addEventListener("change", async () => {
  const file = input.files?.[0];
  if (!file) return;
  try {
    const result = await uploadLargeFile(file, {
      apiBase: "https://YOUR-API-ID.execute-api.eu-central-1.amazonaws.com",
      onProgress: (done, total) => {
        status.textContent = `${Math.round((done / total) * 100)} % uploaded`;
      },
    });
    status.textContent = `Done: ${result.key}`;
  } catch (err) {
    status.textContent = `Upload failed: ${(err as Error).message}`;
  }
});
Enter fullscreen mode Exit fullscreen mode

Here the client requests a separate URL for each part. With 16 MiB parts that overhead is negligible, and it has an advantage: every URL is fresh, even if the upload takes an hour. If you want to save requests, ask for up to 100 URLs at once; the endpoint already supports that.

Common S3 multipart upload errors

403 SignatureDoesNotMatch when uploading a part

There are two common causes. Either requestChecksumCalculation: "WHEN_REQUIRED" is missing on the S3 client, so a wrong CRC32 checksum is baked into the URL. Or the part has a different length than what was signed. In our pattern this happens when the client slices with a different part size than the server planned. In that case, check whether the client really uses partSize from the start response.

The browser can’t see the ETag

If your client collects the ETags itself, the bucket’s CORS rule must list ETag under ExposedHeaders. Otherwise response.headers.get("ETag") simply returns null, even though the upload worked. Our pattern fetches the ETags via ListParts on the server and therefore doesn’t strictly need this setting. I keep it anyway, because it helps when debugging in the network tab and costs nothing.

EntityTooSmall on completion

S3 requires at least 5 MiB for every part except the last. If you let the client choose the part size freely, or split files into equal pieces without minding this limit, you only get this error in the very last step. The planParts function prevents that because it never goes below 5 MiB.

NoSuchUpload after a longer pause

The upload ID becomes invalid once the upload has been completed or aborted. Often your own lifecycle rule is behind it: if an upload runs longer than the configured period, S3 cleans it up. So choose the period more generously than your longest realistic uploads, and keep the DynamoDB TTL and the lifecycle rule in step.

Hidden costs from orphaned parts

Without a lifecycle rule, parts of abandoned uploads stay forever. You can find them with aws s3api list-multipart-uploads --bucket <bucket>. For me, the abortIncompleteMultipartUploadAfter rule belongs in every bucket that clients write to directly. Keep in mind that S3 evaluates lifecycle rules once a day, so cleanup doesn’t happen to the minute.

Keeping an eye on security and cost

The example deliberately has no sign-in yet, so the core stays visible. For real use, put a JWT authorizer in front of the HTTP API and store the user’s ID in the DynamoDB record. Every endpoint then checks whether the upload belongs to the calling user. Otherwise anyone who guesses or intercepts an upload ID could abort someone else’s upload. Guessing a UUID is practically hopeless, but that’s no substitute for an authorization check.

Also, S3 only checks the size, not the content. A file registered as audio/mpeg doesn’t necessarily contain MP3 data. Check uploaded files after completion, for example via an S3 event and a Lambda function that reads the first bytes, or with a malware scan. Until the check has passed, the file should live in an area other users can’t access.

On the cost side, requests matter most: each part is a PUT request in S3 and, in our pattern, one Lambda invocation for signing. With 16 MiB per part, this stays manageable even for many gigabytes. The bigger cost driver is forgotten storage, and the lifecycle rule takes care of that. The Lambda functions only run for milliseconds because they never touch a file.

My take

I like this pattern because it shows what Serverless does well: a managed service does the heavy lifting, and my code only takes care of rules and state. Four small functions and one table are enough to get gigabytes delivered reliably.

I would push back on the common shortcut of handing the client all URLs at once and accepting its ETag list at the end. It’s quick to build, but then the browser decides how big the file gets and whether it’s complete. Moving that to the server takes little effort: one table, one signed header and one ListParts call. To be fair, the DynamoDB table is the part you can argue about most. If you don’t need size control, you can do without it. But as soon as people upload whom you don’t blindly trust, I would always include it.

One alternative you should know: with Amplify or the Upload helper from @aws-sdk/lib-storage, a library does the splitting for you. Both require the client to have AWS credentials, for example through Cognito identity pools. For applications without an AWS identity in the browser, presigned URLs are the more direct route.

FAQ

From what file size is an S3 multipart upload worth it?

AWS recommends it from around 100 MB. From 5 GB it is mandatory, because a single PUT can’t be larger.

How many parts can a multipart upload have?

No more than 10,000. Apart from the final part, each one needs a size of 5 MiB to 5 GiB, and the finished object may grow to 48.8 TiB.

Can I resume an interrupted upload?

Yes, as long as the upload ID is still valid. ListParts shows which parts have already arrived, and you only upload the missing ones. In the example, the complete endpoint returns exactly that list as missing.

Do incomplete multipart uploads cost money?

Yes. Uploaded parts are billed like regular storage until the upload is completed or aborted. A lifecycle rule with AbortIncompleteMultipartUpload removes them automatically.

How long is a presigned URL for a part valid?

As long as you specify when signing, at most seven days. If a Lambda function signs with temporary credentials, the URL expires no later than those credentials. Short lifetimes like 15 minutes plus fresh URLs on retries are the safer choice.

Your next step

Clone the example, run npm test and deploy the stack to a test account. Then upload a file of a few hundred megabytes and briefly pull the network cable or switch the browser offline halfway through. You’ll see individual parts being retried instead of everything starting over. After that, take a look at your existing buckets: does every bucket that clients write to have a lifecycle rule for incomplete uploads?

Now over to you: how large are the files you push to S3, and do you trust the client or control the size on the server? Let me know in the comments.

All code from this article, ready to run: smirowstanitzok/sascha-ms-examples

Sources

This post was created with AI assistance.


👉 Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator – it runs right in your browser.

📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Top comments (0)