Smart Placement on Cloudflare Workers for Backend-Heavy AI Calls

#ai #machinelearning #tutorial #llm

By default a Worker runs in the data centre nearest the user, which is the right answer when the Worker is the thing being talked to and the wrong answer when it is mostly a client of something far away. Smart Placement is one config key, and whether it helps you is arithmetic you can do beforehand.

What the default placement costs you

Running at the edge minimises the distance between the user and the Worker. It does nothing about the distance between the Worker and everything the Worker needs. If a user in Sydney reaches a Worker in Sydney which then calls a model API served from Virginia, the request crosses the Pacific and back, and it does so once for every call the Worker makes.

For a single upstream call, the edge placement is close to neutral: the long hop happens once either way, and running near the user at least makes TLS setup with the browser fast. The picture changes when the Worker makes several sequential calls, because each one pays the long hop again.

The arithmetic that decides it

Let n be the number of sequential round trips your Worker makes to a distant backend, and rtt the round-trip time between the edge location and that backend. The Worker’s own contribution to latency is roughly n × rtt plus the time the backends actually spend working. Move the Worker next to the backend and that term collapses towards rtt — one crossing, for the user’s request and response — plus n local round trips that cost almost nothing.

So the saving is approximately (n − 1) × rtt.

One model call and nothing else. n = 1. The saving is zero. Do not expect Smart Placement to help a thin proxy.
A retrieval pipeline. Fetch the user record, embed the query, query the vector store, call the model, write a log row. That is four or five sequential hops, and if they share a region the saving is three or four crossings of whatever distance separates your users from that region.
An agent loop. Several sequential model calls with tool calls between them. n can be a dozen, and this is where placement is worth the most.

The condition hiding in all of that is that the backends must be near each other. Moving the Worker next to your Postgres does not help if the model API is on a different continent from the Postgres — you have swapped which hop is long. Smart Placement optimises for one centre of gravity, so it works when your dependencies have one.

Enabling it

One key in your Wrangler configuration, and a deploy.

// wrangler.jsonc
{
  "name": "rag-worker",
  "main": "src/index.ts",
  "compatibility_date": "2026-08-11",
  "placement": { "mode": "smart" }
}

# wrangler.toml
[placement]
mode = "smart"

There is no region to choose and no hint to supply. Cloudflare documents Smart Placement as requiring consistent traffic to the Worker from multiple locations to make a placement decision, and as taking up to 15 minutes to analyse. A newly deployed Worker with no traffic will not be placed, and a Worker that only ever receives requests from one city has nothing to optimise.

Add the placement key and deploy.
Let real traffic run. Synthetic traffic from a single CI region is not enough to trigger a decision.
Check the placement status after at least fifteen minutes, using the Workers API for the script.
Compare latency before and after using your own per-request timings — see below for why the total number alone will mislead you.

Verifying that it took effect

Cloudflare documents a placement state on the Worker’s API response, with values SUCCESS, INSUFFICIENT_INVOCATIONS and UNSUPPORTED_APPLICATION, and no field at all before the Worker has been analysed. Those three answers are exactly the three questions worth asking: it worked, you do not have enough traffic yet, or your Worker is not a shape that can be placed.

The status field and the 15-minute analysis window are documented behaviour at the time of writing. Cloudflare, Smart Placement

For the latency side, instrument the segments rather than the total. The total includes the model’s own generation time, which varies by prompt and swamps a change of a hundred milliseconds in network time.

const t0 = Date.now();
const user = await getUser(env, userId);
const t1 = Date.now();
const vector = await embed(env, query);
const t2 = Date.now();
const hits = await search(env, vector);
const t3 = Date.now();

console.log(JSON.stringify({
  colo: request.cf?.colo,
  user_ms: t1 - t0,
  embed_ms: t2 - t1,
  search_ms: t3 - t2,
}));

Logging request.cf?.colo alongside the segment timings is what makes the comparison legible: after placement takes effect, the segments that talk to your backend should compress, and the colo field tells you where the Worker actually ran. Write these rows to a D1 table if you want to compare distributions rather than eyeball a log stream.

Where it does nothing

Cloudflare documents several exclusions, and they are not edge cases:

Only fetch handlers. Placement affects the execution of fetch event handlers. A Worker without one is ignored, and RPC methods and named entrypoints are not affected.
Static assets are unaffected. They are always served from the location nearest the incoming request, which is the right behaviour and means a mixed Worker gets placement for its API routes and nearest-edge delivery for its files.
Anything user-latency-bound. If your Worker returns a cached or computed answer without calling anything distant, moving it away from the user makes it slower. Placement is a trade, and the side you are trading away is proximity to the browser.

One further consideration specific to model calls: if you stream, the user’s time to first token now includes the crossing to wherever the Worker was placed. For a pipeline with several hops before the first token, that is still a large net win. For a thin streaming proxy, it is a small net loss. The arithmetic at the top of this page is the same arithmetic; streaming just makes the first hop more visible.