Cloud inference is a variable cost that scales with volume. Edge inference is a fixed cost that scales with fleet size. The choice is therefore not about which is cheaper but about where the two lines cross, and the crossover is easy to compute once the inputs are named.
Two different cost structures
Cloud spend is a rate: some amount per inference, or per GPU-hour divided by achievable throughput. It is zero when nothing runs, grows linearly, and can be reduced later by optimising the model or renegotiating a price. It also compounds — a fleet that doubles doubles the bill.
Edge spend is committed. The extra silicon capable of running the model is paid for per device at manufacture, amortised over the device’s service life, and it does not care whether the model runs once a day or continuously. Doubling the inference rate is free; doubling the fleet is not, and neither is discovering after deployment that the model no longer fits, because the fix involves a truck.
That asymmetry is the real content of the decision. Cloud keeps optionality and charges for volume; edge buys volume cheaply and gives up the ability to change your mind.
It also means the two are not usually compared at the same point in time. The cloud number is known today and revisable monthly. The edge number is a commitment made at hardware design, six to eighteen months before the first device ships and three to seven years before the last one is retired, against a workload that has to be forecast rather than observed. Comparing a measured cloud bill with a forecast edge cost and declaring one cheaper is comparing two different kinds of number, and the honest form of the comparison carries the forecast uncertainty explicitly — usually as a range of fleet sizes and inference rates rather than a single crossover point.
There is also a third structure that gets left out of the comparison and frequently wins: self-hosted inference on a server you already own, on-premises or in a rack at the site. It behaves like edge cost in that the hardware is a fixed amortised commitment, and like cloud in that one machine serves the whole fleet and can be upgraded without touching a device. For a site with a few hundred sensors and a local network, a single machine at the site often removes both the per-inference charge and the metered link at once.
The crossover, derived
Every figure below is a stated assumption. Prices for hosted inference and for silicon change constantly and vary by region, contract and volume, so substitute your own quotes; the structure is the part that holds.
ASSUMPTIONS
fleet 500 devices
device service life 36 months
extra BOM for on-device inference $28 per device
hosted inference price $0.40 per 1,000 inferences (assumed)
fleet inference rate varies; this is what we solve for
EDGE FIXED COST
total silicon 500 * $28 = $14,000
per month $14,000 / 36 = $388.89 / month
CLOUD VARIABLE COST
per inference $0.40 / 1,000 = $0.0004
CROSSOVER
N such that N * $0.0004 = $388.89
N = 388.89 / 0.0004 = 972,222 inferences / month (fleet)
per device = 972,222 / 500 = 1,944 / month
per device per day = 1,944 / 30 = 65 / day
READ IT AS
fewer than ~65 inferences per device per day -> cloud is cheaper
more than ~65 -> edge is cheaper
at 1 inference/minute (1,440/day) edge is ~22x cheaper on compute
Sixty-five inferences per device per day is a strikingly low threshold, and it is the honest consequence of these assumptions: continuous monitoring workloads clear it before breakfast. Anything triggered by a human, or by a rare event, may never reach it. The shape of the workload, not its importance, decides.
The sensitivity worth noting is that the crossover moves linearly in both directions. Halve the hosted price and the threshold doubles to 130 per device per day. Halve the silicon premium — which happens as capable parts commoditise — and it halves to 33. Neither input is stable, which is why this belongs in a spreadsheet you re-run rather than in a decision made once.
The $0.40 per thousand and the $28 BOM premium are illustrative assumptions, not quoted prices. Hosted inference pricing in particular has moved by more than an order of magnitude on some workloads within a year, so re-derive with current quotes before committing to hardware.
The term that usually decides it
Compute is frequently not the dominant term. Getting the data to the cloud has its own cost, and on cellular-connected devices it commonly exceeds the inference cost by a wide margin.
ASSUMPTIONS
raw payload per inference 2,400 bytes (128-sample, 6-channel window)
result payload 8 bytes
cellular IoT data price $0.30 per MB (assumed small-plan rate)
rate 1,440 inferences per device per day
CLOUD: SEND RAW
per device per day 1,440 * 2,400 B = 3.456 MB
per device per month 3.456 * 30 = 103.7 MB
cost 103.7 * $0.30 = $31.10 / device / month
fleet of 500 = $15,552 / month
EDGE: SEND RESULTS
per device per month 1,440 * 8 B * 30 = 345.6 kB = 0.346 MB
cost 0.346 * $0.30 = $0.10 / device / month
fleet of 500 = $52 / month
AGAINST the $388.89/month amortised silicon: the bandwidth saving alone
repays the hardware roughly 40x over.
On a cellular fleet the compute crossover is almost irrelevant, because the data transfer term dwarfs it. On devices behind a wired connection or site Wi-Fi with no metered link, that term collapses to nearly zero and the compute crossover is the whole decision. This is why the same question gets opposite answers at two companies with similar workloads, and why the connectivity assumption should be the first thing stated in any such comparison. The related question of how much preprocessing to do before the link is the preprocessing split.
The costs that do not appear in either column
- Fleet model deployment. Updating a cloud model is a deploy. Updating five hundred devices needs a signed over-the-air update system, staged rollout, rollback, and a way to know which device is on which version. That system has to exist before the first model update, and building it is not free.
- Version skew becomes a data problem. With devices on three model versions, the outputs are not comparable, and any downstream analysis has to condition on version. Cloud inference has exactly one version at a time.
- The edge model is frozen at the hardware. A better model that needs twice the memory cannot ship to devices already in the field. Sizing the silicon for today’s model with no headroom is the mistake that ends the programme, and headroom is cheap at design time and impossible later.
- Observability is harder and matters more. You cannot log every input from a device with a metered link, soedge inference systems need sampling strategies for the telemetry about the inference itself, which is a design task cloud deployments get for free.
- Cloud has a floor you keep paying. Endpoints with provisioned capacity bill whether or not traffic arrives, and idle provisioned capacity is the most common source of surprise on the cloud side. Serverless avoids it at the price of cold starts.
Using this on your own numbers
The whole calculation reduces to one comparison, and it takes four numbers you probably already have.
monthly edge cost = fleet * bom_premium / life_months
monthly cloud cost = fleet * rate_per_device_month * price_per_inference
+ fleet * bytes_per_month / 1e6 * price_per_MB
crossover rate per device per month
= (bom_premium / life_months)
/ (price_per_inference + bytes_per_inference/1e6 * price_per_MB)
Two practical cautions on applying it. Get a real quote for the silicon premium rather than a list price, because at fleet volumes the difference is large. And run the arithmetic at the fleet size you expect in three years rather than today’s, since the hardware decision is made once and lives for the whole device generation, while the cloud bill is a decision you get to revisit every month.
Top comments (0)