WES: Why Tokens Per Watt Isn't Enough for Edge Inference

Jeff Geiser — Wed, 11 Mar 2026 16:33:17 +0000

Edge inference is still nascent.

I work at Zenlayer helping companies deploy compute in hard to reach places. Spin up a VM with Ollama, pull a model, running inference in minutes. The infrastructure is there. The tooling is maturing. But the metrics for understanding what's actually happening on those nodes is still catching up.

I've also been building Wicklee in my weekend time — a sovereign GPU fleet monitor written in Rust with an embedded React dashboard. Running a mixed fleet of Apple Silicon and AMD CPU nodes, I kept running into the same problem: the standard metrics weren't telling me quite enough..

Everyone in AI talks about efficiency at the data center level. Jensen talks tokens per watt. Google reports Gemini in watt-hours. Microsoft targets 8-20x energy reductions per query. Great work — but these are hyperscaler metrics, built for environments with precision cooling, facilities teams, and controlled everything.

That's not edge inference.

Here's a scenario that'll be familiar if you're running a distributed fleet:

tok/s drops slightly
board power creeps up slightly
thermal state moves from Normal to Fair

You do get a drop in tokens/watt. But now what? Is it a blip? Is it meaningful? What do you chase?

Honest answer: hard to tell. Tokens/watt can't distinguish a legitimate workload increase from thermal throttling. They look identical in the number. One means the node is doing its job. The other means it's quietly degrading and inserting some wait sates to prevent reaching a critical state. They require completely different responses.

And in practice, a 15% drift on one node in a six-node fleet at 2am looks like noise. You don't chase it. The node keeps running. Keeps burning power. Keeps delivering worse inference. Until something obvious breaks.

The data was there. Nothing put it in front of you.

In other cases, the token/watt metric can stand still even if token/s is dropping if power output is also dropping. So, efficiency looks stable but throughput is actually dropping.

So I wanted to add thermal state to the equation.

WES — the Wicklee Efficiency Score:
WES = tok/s ÷ (Watts_adjusted × ThermalPenalty)

The ThermalPenalty comes directly from what the device reports — on Apple Silicon that's IOPMCopyCPUPowerStatus via IOKit, on NVIDIA it's the nvmlDeviceGetCurrentClocksThrottleReasons() bitmask. Not temperature guesses. Not externally imposed thresholds. The hardware's own classification of its thermal condition, amplified into the score.

Thermal StatePenaltyNormal1.0Fair1.25Serious1.75Critical2.0+

When thermals are clean, penalty is 1.0 and WES equals tokens/watt. When throttling starts, the penalty amplifies the drop — turning a subtle drift you'd dismiss as noise into something that screams at you.

Higher WES = better. Miles per gallon for inference.

Why the leaderboard is the real insight

WES on a single node is useful. The Wicklee fleet leaderboard is where it gets interesting.

Stack rank every node by WES. A thermally degraded node doesn't just show a number that drifted. It falls in the ranking. Drops below nodes it was beating yesterday. That positional change is impossible to miss — you don't need to be actively monitoring anything, you just notice your #1 node is now #3.

That's the moment tok/watt never creates.

WES surfaces the signal. The thermal panel explains the cause. Route requests to the top of the leaderboard and you're automatically routing away from degraded nodes without lifting a finger.

Real numbers from my fleet
Running llama3.2:3b via Ollama across hardware:

tok/s makes this look like a 6x gap. WES shows a 1,293x efficiency difference. The Ryzen is fast. It is not efficient.

Now throw the M2 into thermal throttling — WES drops from 181.5 to 83.6. Still #1 on the leaderboard. But the drop is visible. The thermal panel tells you why. WES made you notice. Thermal data gave you the diagnosis. They work together.

*Raw WES vs Penalized WES
*
Wicklee reports WES two ways:
Raw WES — ThermalPenalty forced to 1.0. Hardware ceiling under clean conditions. Essentially tok/watt.

Penalized WES — live thermal penalty applied. Operational reality.
The gap is your Thermal Cost — efficiency being lost to throttling right now:

Thermal Cost % = (Raw WES − Penalized WES) / Raw WES × 100

A node with Raw WES 181.5 and Penalized WES 83.6 is losing 54% of its potential efficiency to thermals. That's the number that drives action — not a raw temperature reading, not a wattage blip.

Here's the implementation if you want to compute it yourself:

javascriptconst THERMAL_PENALTIES = { Normal: 1.0, Fair: 1.25, Serious: 1.75, Critical: 2.0 };

// Clean node:
computeWESPair(108.9, 0.6, "Normal"); // → { raw: 181.5, penalized: 181.5, thermalCostPct: 0 }

// Throttled node:
computeWESPair(94.1, 0.9, "Fair"); // → { raw: 104.6, penalized: 83.6, thermalCostPct: 20 }
And in Rust for the monitoring agent side:
rustpub struct WESResult {
pub raw: Option,
pub penalized: Option,
pub thermal_cost_pct: Option,
}

pub fn compute_wes_pair(
tps: Option, watts: f64, thermal_penalty: f64, pue: f64
) -> WESResult {
let compute = |p: f64| tps.and_then(|t| {
let w = watts * pue;
if w <= 0.0 { return None; }
Some((t / (w * p) * 10.0).round() / 10.0)
});
let raw = compute(1.0);
let penalized = compute(thermal_penalty);
let thermal_cost_pct = raw.zip(penalized)
.map(|(r, p)| ((1.0 - p / r) * 100.0).round());
WESResult { raw, penalized, thermal_cost_pct }
}

WES is derived — compute it at render time from fields you're already collecting. No telemetry layer changes required.

How WES relates to existing work
Stanford and Together AI published "Intelligence per Watt" (IPW) last year — accuracy divided by power, measured offline against benchmarks. Solid research. It answers "what is this hardware capable of per watt?"

WES answers "what is it delivering right now?"

Raw WES and IPW are the same question from different vantage points — IPW from a benchmark lab, WES from live fleet telemetry. IPW tells you the ceiling. WES tells you how close you're running to it, under real thermal conditions, continuously.

**What's coming in Wicklee
**The Fleet WES Leaderboard is shipping soon — every node ranked by Penalized WES, Raw WES as a secondary column, Thermal Cost % visible at a glance.

After that, a series of benchmark posts:

Cross-platform WES benchmarks — Apple Silicon vs AMD CPU vs NVIDIA GPU, same model, same prompt. Raw WES per platform.
Thermal stress testing — deliberately inducing throttling and watching Raw vs Penalized WES diverge in real time.
Sustained load degradation — how long before each platform throttles, how fast does WES collapse when it does.
Edge enclosure testing — WES in a fanless case vs open air. Spoiler: not pretty.

Goal: a reproducible WES dataset across hardware. Not just a formula — empirical data behind it.
If you're running a local inference fleet, try computing your WES from the formula above and drop your numbers in the comments. Curious where different hardware lands.
Miles per gallon for inference. When you need a race car, gas be damned — go for it. But at the edge, efficiency wins.

Distributed Inference Observability gaps

Jeff Geiser — Fri, 16 Jan 2026 19:10:12 +0000

It seems that distributed inference observability has some gaps.

In terms of framing this, I am referring to inference deployments at the edge (or so called near edge).. pops close to end users. Let's say you are using ollama for some early testing and/or scaling but are using vllm in production.

Traditional monitoring platforms will report on GPU/CPU load, memory usage, network status, etc, etc.

However, other stuff is also happening:

GPU throttled - 100% utilization but clock speed dropped 33%
KV cache saturated causing some queue backlog
Time to first token spiked 200% from CPU contention
Another tenant's PCIe traffic impacted inference

maybe some contextual drift - some hardware stresses that degrade inference performance but it is happening in ways that is generally invisible to system metrics.

Most of the monitoring in the market is built for servers and takes a peek at intervals that may not make sense for inference

token generation:20-100 per second
cache saturation: spikes in seconds
thermal throttling happens instantly

but traditional monitoring might see this as smooth if only glancing at the server every 30 seconds. But, you also can't grab data every 2 seconds or you might contribute to some cpu scheduling pressure.

So, if you are going to run both ollama (for dev/test or smaller loads) and vLLM for production they have completely different failure modes but traditional monitoring would treat them the same.

We also have a blind spot with regard to time to first token (ttft) and time per output token (tpot). We might show request latency spiking, but we need to know whether ttft spiked or tpot spiked..

so, I am thinking about an open source project that would be a lightweight observability agent.. large companies will likely solve this by building a giant observability layer on top of their distributed inference solution -- but I think having a more bottoms up approach that can be deployed might make sense..

the observability agent would strive to:

have limited cpu impact/overhead
2 second sampling with some intelligent backoff
built in ttft/tpot splitting
contextual drift detection
works with vLLM/Prometheus and Ollama API stats
embedded DB storage (duckDB?) - no external dependencies
runs at edge.. maybe federates..

Curious to get feedback on where people are hitting observability gaps.. this is a new area for me to spend time on so curious about all feedback.

What are you doing to monitor vLLM and/or other inference engines?

What metrics do you wish you had?

Drop the war stories here.. thanks..

(apologies for lack of formatting.. maybe I will get better over time..)

DEV Community: Jeff Geiser

WES: Why Tokens Per Watt Isn't Enough for Edge Inference

Distributed Inference Observability gaps