If your definition of "green software" is still dark mode and compressed images, your mental model is a decade out of date. For any app shipping AI features, the front end is a rounding error. The dominant energy line item is inference: a prompt routed through a large model on power-hungry accelerators can consume orders of magnitude more energy than the database query sitting next to it. And because GPU-seconds map almost linearly onto both watts and dollars, every optimization below is a carbon cut and a bill cut at the same time. This is the engineer's version of the argument — concrete levers, in the order I would pull them.
1. Model routing: stop paying frontier prices for commodity tasks
The single biggest win is refusing to send every request to your largest model. Most production traffic is classification, extraction, short rewrites, or routing decisions that a 7B–13B model handles at parity with a frontier model — at a fraction of the FLOPs.
Put a router in front of inference. The cheap version is a rules table keyed on task type and input length; the better version is a small classifier that scores difficulty and dispatches accordingly, with a confidence threshold that escalates only genuinely hard prompts to the expensive tier.
def route(task, tokens, difficulty):
if task in ("classify", "extract", "route") and tokens < 2000:
return "small-8b"
if difficulty < 0.6:
return "mid-32b"
return "frontier" # reserved for genuine reasoning
Track the escalation rate as a first-class metric. If more than a quarter of traffic reaches the frontier tier, your thresholds are too timid and you are leaving both carbon and money on the table.
2. Semantic caching: the calls you never make
Identical and near-identical prompts should not trigger fresh inference. An exact-match cache on a normalized prompt hash is table stakes. The real gains come from semantic caching: embed the incoming prompt, run an approximate-nearest-neighbor lookup against a vector store of previous prompt/response pairs, and serve the cached response when cosine similarity clears a threshold you tune against a quality set.
q = embed(prompt)
hit = vector_store.query(q, top_k=1)
if hit and hit.score > 0.92:
return hit.response # zero GPU time
resp = model.generate(prompt)
vector_store.upsert(q, resp)
On assistants and support surfaces with a long tail of repeated intent, semantic caching commonly removes a double-digit percentage of calls outright. That is the greenest possible request — the one that never reaches the GPU.
3. Continuous batching and efficient serving
A GPU serving one request at a time is burning its full power envelope at a fraction of its useful throughput. Modern serving stacks (vLLM, TGI, TensorRT-LLM) use continuous batching and paged KV-cache to interleave many in-flight sequences, pushing utilization up and energy-per-token down. Same silicon, same wattage, several times the work.
Two knobs matter most. Cap max_num_seqs to protect tail latency under load, and set batch/wait windows so you are trading a few milliseconds of queueing for a large utilization gain. Watch tokens-per-second-per-GPU alongside p99 latency — the goal is a saturated accelerator, not an idle one holding a reservation.
4. Distillation and task-specific fine-tuning
For a stable, high-volume feature, a small model distilled or fine-tuned on your own task frequently matches a giant generalist on that narrow job at a fraction of the runtime cost. Unlike routing and caching, which reduce calls, distillation lowers the energy floor of every remaining call permanently. The trade is a one-time training cost against a recurring inference saving — and for anything above modest traffic, that math pays back quickly.
Pair it with prompt and context discipline: shorter system prompts, tighter retrieval, and fewer few-shot examples all reduce tokens processed per call, and tokens map directly to compute. Trimming a bloated 4K-token context to 1.5K is a real energy reduction on every single request.
5. Autoscaling and right-sizing the fleet
Idle accelerators are the purest waste there is — full standby power, zero output, metered spend. Right-size aggressively: scale GPU replicas on queue depth or concurrency rather than CPU, scale to zero for spiky internal workloads, and separate latency-critical online serving from batch inference so you are not holding premium capacity for jobs that could run later. The gap between reserved and used capacity is carbon and cost with no return.
6. Carbon-aware scheduling for the batch tier
Everything above cuts total energy. Carbon-aware scheduling cuts the emissions per unit of the energy you still spend, by moving flexible work to when and where the grid is cleaner. Non-urgent jobs — retraining, embedding backfills, nightly evals, ETL — can be shifted in time or region using a grid-intensity signal, cutting the carbon of identical compute without changing the output.
if grid_intensity(region) > THRESHOLD and job.deferrable:
reschedule(job, to=cleanest_window(region, within="6h"))
This applies only to deferrable background work, so it never touches user-facing latency. It is the one lever here that is purely a carbon play rather than a joint carbon-and-cost play — though shifting to off-peak windows often lands on cheaper capacity too.
7. Measure it, or you are guessing
You cannot optimize what you do not track, and intuition about where energy goes is usually wrong — teams blame the front end and find a nightly batch job or an over-eager AI call. Instrument tokens and GPU-seconds per feature, expose them in the same dashboards as latency and error rate, and layer provider carbon dashboards plus open estimators on top. You do not need a perfect number; you need a directional signal reliable enough to rank the backlog. Because spend tracks carbon so closely, your billing console is a usable first proxy on day one.
This is the discipline behind serious LLM integration and production AI development: efficiency treated as an engineering property, not a marketing badge. For the leadership-level framing of why carbon and cost are the same variable, read the full guide on TechCirkle.
Frequently Asked Questions
Won't a routing layer add latency to every request?
A well-built router adds single-digit milliseconds — a rules table is effectively free, and a small classifier runs in a few ms on CPU. That is negligible against the hundreds of milliseconds an LLM generation takes, and it is more than repaid when the router sends most traffic to a smaller, faster model. Net latency usually improves.
How do I stop semantic caching from serving a wrong answer?
Tune the similarity threshold against a labeled quality set rather than eyeballing it, keep it conservative (0.9+ cosine is a common starting point), and scope caches to contexts where near-duplicate prompts genuinely expect the same answer. Add a TTL for anything time-sensitive, and exclude personalized or user-specific responses from the shared cache entirely.
Is continuous batching worth it for low-traffic services?
At low, bursty traffic the batch rarely fills, so the utilization gain is small and scale-to-zero autoscaling delivers more. Continuous batching earns its keep once you have sustained concurrency. Match the technique to the traffic shape rather than applying it everywhere by reflex.
When does distillation actually pay back?
When a feature is stable and high-volume enough that the recurring inference saving outweighs the one-time training and evaluation cost. For low-traffic or rapidly-changing features, routing and caching give you most of the benefit with none of the training overhead. Reach for distillation on your hottest, most settled paths.
How much does carbon-aware scheduling really save?
It does not reduce energy at all — it reduces the emissions intensity of the energy you spend, and only for deferrable work. Savings depend entirely on how variable your grid is; a region with a wide clean/dirty swing offers meaningful gains, a consistently clean or consistently dirty grid offers little. Treat it as a bonus on top of the compute reductions, not the main event.
What is the smallest measurement setup that is actually useful?
Emit two counters per AI feature — tokens processed and GPU-seconds consumed — and chart them next to latency and errors. That alone tells you which feature dominates your inference footprint, which is enough to rank the backlog. Add provider carbon dashboards and per-call estimators later to refine; do not let their absence delay the obvious wins your billing data already reveals.
Top comments (0)