DEV Community

Cover image for Copy a remote video into R2 from a Cloudflare Worker without buffering it
Movoa
Movoa

Posted on

Copy a remote video into R2 from a Cloudflare Worker without buffering it

#webdev #serverless #cloudflare #javascript

If your Worker copies files it didn't create (AI generation results, imports from a URL, exports from another service), the obvious line, await res.arrayBuffer(), loads the whole file into an isolate whose 128 MB is shared by every request it is serving. For video that's the wrong default. You can stream the file straight from the source into R2 instead, and it takes three pieces: FixedLengthStream, pipeTo, and an S3 PUT signed as UNSIGNED-PAYLOAD.

The code below is trimmed from the copy step at Movoa, which moves each finished video from a provider's temporary URL into our own bucket.

TL;DR

  • Workers memory is 128 MB per isolate, not per request, so buffered copies add up across concurrent requests.
  • Stream when the source sends Content-Length: response.body.pipeTo(new FixedLengthStream(length).writable) and PUT the readable side.
  • Sign the stream with aws4fetch; it uses UNSIGNED-PAYLOAD, which R2's S3 API accepts over HTTPS.
  • Reject tiny bodies before writing, and fall back to buffering only when there's no length (or you're in Node).

Why buffering is the trap

The tempting version is three lines:

const res = await fetch(resultUrl);
const body = new Uint8Array(await res.arrayBuffer());
await bucket.put(key, body);
Enter fullscreen mode Exit fullscreen mode

Cloudflare's limits page (read 2026-10-16) is specific about why this bites: each isolate "can consume up to 128 MB of memory", and "this limit is per-isolate, not per-invocation. A single isolate can handle many concurrent requests." When an isolate goes over, in-flight requests are allowed to finish and new requests go to a fresh isolate; under heavy load some incoming requests can be cancelled.

So the risk isn't one big file. It's two or three ordinary video files being copied by the same isolate at the same moment, each fully materialised in memory.

Buffered copy holds the whole file in isolate memory; a streamed copy only holds the chunks in flight

The PUT needs a length, so stream only when you have one

A single S3 PUT to R2 wants a Content-Length. A raw ReadableStream doesn't carry one, which is where FixedLengthStream comes in. It's a Workers runtime TransformStream that you construct with the expected byte count: it tells the runtime the body's length up front, and it errors if the bytes that flow through don't match. That second property is a truncation check: if the source sends fewer than length bytes, the stream errors and the upload fails instead of leaving half a file in your bucket.

Here is the streaming branch, lightly trimmed from our code:

const response = await fetch(url, {
  headers,                 // forward auth if the result URL needs it
  redirect: 'follow',
  signal: AbortSignal.timeout(timeoutMs),
});
if (!response.ok || !response.body) return fail(`HTTP ${response.status}`);

const contentType = response.headers.get('content-type') || 'application/octet-stream';
const length = Number(response.headers.get('content-length') || NaN);

const FixedLength = (globalThis as any).FixedLengthStream; // only exists on workerd
if (FixedLength && Number.isFinite(length) && length > 0) {
  if (length < minBytes) {
    void response.body.cancel().catch(() => {});
    return fail(`too small (${length} bytes)`);
  }
  const { readable, writable } = new FixedLength(length);
  const piping = response.body.pipeTo(writable);   // source → FixedLengthStream
  const result = await putStream(env, { body: readable, length, key, contentType });
  await piping.catch(() => {});                    // surface nothing twice
  return result;
}
Enter fullscreen mode Exit fullscreen mode

Two details matter. The pipeTo promise is started but not awaited before the PUT, because the PUT is what drains the readable side; awaiting the pipe first would deadlock. And the feature check on globalThis.FixedLengthStream keeps the same module working under Node in local dev, where that class doesn't exist (more on that below).

Signing a body you haven't read

R2 speaks the S3 API, so the PUT is a signed request. SigV4 normally includes a hash of the payload, which is impossible for bytes you haven't received yet. aws4fetch handles this by signing a stream body as UNSIGNED-PAYLOAD, and R2 accepts that over HTTPS:

import { AwsClient } from 'aws4fetch';

// env: R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY (an R2 API token),
// R2_ENDPOINT = https://<account-id>.r2.cloudflarestorage.com, R2_BUCKET
async function putStream(env, { body, length, key, contentType }) {
  const client = new AwsClient({
    accessKeyId: env.R2_ACCESS_KEY_ID,
    secretAccessKey: env.R2_SECRET_ACCESS_KEY,
    region: 'auto',
    service: 's3',
  });
  const res = await client.fetch(`${env.R2_ENDPOINT}/${env.R2_BUCKET}/${key}`, {
    method: 'PUT',
    headers: {
      'Content-Type': contentType,
      'Content-Length': String(length),
    },
    body,                              // the FixedLengthStream readable
  });
  if (!res.ok) {
    const detail = (await res.text().catch(() => '')).slice(0, 200);
    return fail(`Upload failed: ${res.status} ${detail}`);
  }
  return { success: true, key, url: publicUrlFor(key) };
}
Enter fullscreen mode Exit fullscreen mode

Checks before you write anything

  • A 200 that is really an error. Some providers answer the result URL with a short JSON error and a 200 status when a file isn't ready or has expired. A minBytes threshold checked against Content-Length catches that before anything is written; cancel the body so the connection is released.
  • Redirects and auth. Signed result URLs often redirect to object storage. Set redirect: 'follow' and pass through whatever headers the provider requires.
  • Timeouts. AbortSignal.timeout(ms) on the fetch keeps a stalled source from holding the request open indefinitely. We give the whole download-and-upload 100 seconds.

Decision flow: response not OK, then minBytes, then stream with FixedLengthStream when Content-Length is known on workerd, otherwise buffer

What about the R2 binding?

If your Worker has the bucket bound (env.BUCKET), put() accepts a ReadableStream too. The catch is the same: the stream needs a known length, which a fetch() response body with a Content-Length has, and the readable side of a FixedLengthStream has. So await env.BUCKET.put(key, response.body) streams when the source sends a length; for anything else, wrap it in FixedLengthStream first.

We use the S3 API instead because the same storage module also runs under Node (local dev and scripts), where there is no binding, and one signed-request path serves both.

When you can't stream

Two cases still take the buffered path:

  1. The source sends no Content-Length (chunked transfer). Without a length there's nothing to give FixedLengthStream, and a single PUT can't be sent. Buffering is the simple answer; for large files, a multipart upload with fixed-size parts is the scalable one.
  2. Local development under Node (next dev, next start), where FixedLengthStream isn't defined. The feature check routes these to arrayBuffer(), which is fine on a laptop. 
// No length, or not on workerd: buffer, and know this is the memory path.
const body = new Uint8Array(await response.arrayBuffer());
if (body.length < minBytes) return fail(`too small (${body.length} bytes)`);
return uploadFile({ body, key, contentType });
Enter fullscreen mode Exit fullscreen mode

Checklist

  • Check response.ok and Content-Length before touching storage.
  • On workerd with a known length: pipeTo into new FixedLengthStream(length), PUT the readable side, then settle the pipe promise.
  • Sign with aws4fetch (UNSIGNED-PAYLOAD); send Content-Length explicitly.
  • Reject bodies under a sane minimum; follow redirects; set a timeout.
  • Buffer only as the fallback, and treat it as the path that costs memory.

On the streaming branch the file never sits in your isolate's memory, and a short read fails loudly instead of saving half a video. The buffered branch is the exception you keep for chunked sources and local dev.

The videos this copies come from the Movoa video generator, which turns one sentence or a photo into a clip with sound; the first one is free and needs no account.

Top comments (1)

Collapse
 
markhale_tsld profile image
Mark Hale •

Really clear write-up. The "128 MB is per isolate, not per request" point is the one that bites people, and FixedLengthStream + UNSIGNED-PAYLOAD is the right combo for a single PUT.

One extra footgun I have hit: if the upstream Content-Length is wrong (or a CDN lies after a transform), FixedLengthStream will error mid-upload and you can still leave a partial object depending on how the S3 client surfaces the failure. Worth either HEADing the source first when the provider allows it, or writing to a temporary key and only promoting to the final key after the PUT returns 200 with the expected size.

Also agreed that env.BUCKET.put(key, stream) is nicer when you are Workers-only; the aws4fetch path is worth the complexity once you need the same module in Node. Thanks for publishing the deadlock note on awaiting pipeTo before the PUT. That one is easy to get backwards.