DEV Community: Rosana

The Watt-Per-Token Era: Why Energy-Efficient Inference Is the Next Core Developer Skill

Rosana — Mon, 09 Nov 2026 20:01:31 +0000

For the last few years, the AI conversation has been about capability: bigger models, longer context windows, better benchmarks. But if you’ve shipped anything with an LLM in production, you’ve already met the other constraint: every token costs something, and a lot of that cost is electricity.

In my last two posts, I looked at how energy markets ripple through cloud bills, and how AI is moving out of the data center and onto physical hardware. This post is where those threads meet. When power is expensive and compute is everywhere, efficiency stops being an optimization and becomes a design requirement.

Inference Is the Bill That Never Stops

Training gets the headlines, but inference is the workload that scales with your users. A model is trained once, then queried millions of times. Every feature you ship, from autocomplete to chat support to agentic workflows, adds to a recurring energy and compute bill.

Two things make this harder than traditional web workloads:

Requests aren’t uniform. A trivial lookup and a multi-step reasoning chain can hit the same endpoint but differ by orders of magnitude in compute.
Defaults favor waste. It’s easy to route everything to the biggest model because it “just works,” the same way we once over-provisioned servers because nobody was watching the bill.

The Efficiency Toolkit

The good news is that developers already have a growing set of levers.

Right-size the model. Not every task needs a frontier model. Classification, extraction, and routing are often handled well by smaller models, which are cheaper, faster, and far less power-hungry.

Route intelligently. A lightweight router that sends easy requests to small models and escalates only hard ones can cut spend dramatically without users noticing.

Quantize and distill. Reducing numerical precision and distilling larger models into smaller ones trades a modest accuracy hit for large gains in memory use, latency, and power draw.

Cache aggressively. Prompt caching and semantic caching mean you stop paying to recompute answers you’ve already produced.

Batch and stream wisely. Batching improves hardware utilization for non-interactive work. Streaming improves perceived latency for interactive work. Know which one you’re building.

Trim your tokens. Bloated system prompts and oversized context windows are the “unindexed database query” of the LLM era. Tokens you don’t send are tokens you don’t pay for.

Why This Connects to Physical AI

This is where the edge story matters. A robot or a phone can’t call out to an unlimited power supply. It has a battery, a thermal envelope, and a fixed NPU. Developers who learn to build within those limits now will be well placed as more AI moves on-device.

Edge deployment forces good habits that cloud abundance lets us skip:

Budget thinking: a fixed watt and memory budget instead of “scale it up.”
Latency discipline: local inference means no network round trip, but also no elastic capacity.
Graceful degradation: a smaller fallback model when power or heat is constrained.

Measure It or You’re Guessing

You can’t optimize what you don’t measure. A few practical starting points:

Track cost per successful task, not just cost per request.

Log tokens in and out per feature, and review the outliers.

Profile latency and throughput alongside quality metrics, since a cheaper model that fails twice as often isn’t cheaper.

Where tooling allows, estimate energy per request and watch the trend, even if the numbers are rough.

Efficiency without a quality check is just cutting corners. The goal is the best result per unit of compute.

Let’s Discuss

I’d like to hear from people running this in production:

Routing vs. one big model: Has model routing paid off for you, or did the added complexity eat the savings?

Measurement: What are you using to track cost or energy per request? Is there a tool you’d recommend?

Quality tradeoffs: Where have you found the line where a smaller or quantized model stops being good enough?

Culture: Does your team treat inference cost as an engineering metric, or is it still just “finance’s problem”?

Share your wins, failures, and hot takes in the comments. 👇

Sources

International Energy Agency, Energy and AI (executive summary): data center electricity demand and AI's role in its growth.
Anthropic, Prompt caching documentation: how prefix caching works and what it saves.
Zhang et al., EfficientLLM: Efficiency in Large Language Models: benchmarks of quantization and other compression methods.
Basta, Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference (University of Waterloo, 2026): energy vs. latency vs. accuracy for quantized models.
Sustainable LLM Inference for Edge AI: quantized LLM energy and accuracy on edge hardware.
Dekoninck et al., A Unified Approach to Routing and Cascading for LLMs: routing queries across models of different sizes and costs.
SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading: routing and cascading strategies between small and large models.

Beyond the Pump: What Record Energy Profits Mean for Tech, Infrastructure, and Everyday Developers

Rosana — Fri, 30 Oct 2026 18:34:28 +0000

When headline news reports massive profit spikes for energy conglomerates like Exxon and Chevron alongside quotes like "we're making a freaking fortune", it is easy to view the story strictly through a political or macroeconomic lens.

For everyday consumers, the reality hits directly at the gas pump and in monthly utility bills. But for software engineers, developers, and tech workers, this growing divide between legacy energy profits and consumer financial strain raises deeper questions about our industry:

How does energy volatility impact our compute infrastructure and cloud costs?

What role do our technical architecture choices play in global power consumption?

How can developers navigate career stability when cost of living rises faster than tech compensation?

1. The Cloud Is Built on the Grid

As developers, we are accustomed to abstracting physical infrastructure into simple software declarations and managed services. But beneath every virtual machine, serverless function, and cloud database lies a physical data center consuming real electricity.

When oil and natural gas prices surge globally, cloud providers—AWS, GCP, Azure, and regional data centers—face direct operational cost increases. Data centers require massive amounts of power not just to run processing units, but to cool them continuously.

While hyperscalers continue investing in renewables, much of the world's power grid remains tied to fossil fuel markets. When legacy energy companies post record windfalls during geopolitical or supply crunches, those underlying energy costs trickle down to every tier of the software stack:

Cloud bill inflation: API pricing, bandwidth, and managed database rates scale upward as hosting costs rise.

Cost-conscious engineering: Efficient code and resource optimization shift from optional best practices to critical financial imperatives for startups and enterprises alike.

2. Green Tech Stack vs. Legacy Extraction

The contrast between modern tech industry initiatives and legacy energy couldn't be starker:

Primary Driver:

Legacy Energy Dynamics: Commodity scarcity & price surges

Modern Tech Imperative: Efficiency, optimization & compute scale

Environmental Impact:

Legacy Energy Dynamics: Carbon extraction

Modern Tech Imperative: Minimizing carbon per API call or transaction

Economic Model:

Legacy Energy Dynamics: Record profits during consumer squeeze

Modern Tech Imperative: Open-source leverage & shared utility

As developers, our daily architectural choices directly influence real-world compute demands:

Algorithmic Efficiency: Prioritizing efficient data structures and algorithms isn't just about passing technical interviews—it reduces CPU cycles, power draw, and cloud costs.

Resource Management: Spinning down idle instances and leveraging auto-scaling ensures we aren't powering idle servers around the clock.

Green Hosting: Choosing data center regions powered primarily by nuclear, wind, or solar energy reduces the carbon footprint of our applications.

3. The Cost of Living Pressure on Developers

It is easy to assume tech workers are insulated from broader economic shocks, but cost-of-living increases affect every tier of the workforce:

Remote Work Realities: High electricity and heating bills mean home office overhead costs have climbed significantly for remote engineers.

Tech Hiring Shifts: While legacy commodities rake in windfalls, tech companies continue to optimize headcount, demand higher productivity, and scrutinize software budgets.

Side Projects & Startups: Bootstrapping a software product becomes harder when basic living expenses eat up disposable income.

4. What Can We Build About It?

As software engineers, we have a unique superpower: we write the software that optimizes real-world systems.

If you are looking for meaningful side projects or open-source contributions in this climate, consider:

Energy Efficiency Tooling: Contributing to CLI tools and libraries that measure local CPU energy consumption and process efficiency.

Open Source Energy Tracking: Supporting open-data platforms that monitor grid carbon intensity in real time.

Smart Home & Automation Systems: Building home automation integrations that optimize household energy usage during peak pricing hours.

Final Thoughts

The gap between record corporate windfalls and individual economic strain is a sobering reminder of how brittle traditional systems can be.

As developers, we may not control global energy markets, but we do control how efficiently we build, the infrastructure choices we make, and the problems we choose to solve. By prioritizing computational efficiency, pushing for clean-energy cloud providers, and building open tools that solve real-world problems, we can help build a tech ecosystem that works for everyone—not just the few making a "freaking fortune."

Sources

Fortune: Exxon and Chevron post massive profits as Trump brags about energy companies making a "freaking fortune" ahead of midterms (October 30, 2026).

Goldman Sachs & Industry Reports: Economic analyses on energy markets, inflation trends, and cost-of-living impacts on everyday consumers.

From Chatbots to Actuators: Why "Physical AI" is the Next Big Engineering Battleground

Rosana — Fri, 23 Oct 2026 17:56:19 +0000

If you’ve spent the last few years drowning in LLM fine-tuning, prompt engineering, and optimizing inference costs for text tokens, I have some news: the sandbox is expanding into the physical world.

According to recent industry tech trends (like Deloitte’s 2026 outlook on Physical AI and robotics), the bleeding edge of AI isn't just sitting in a cloud data center anymore—it’s rolling, walking, and manipulating objects in 3D space.

As software developers and systems architects, we need to talk about what happens when generative models are strapped to wheels, legs, and robotic arms. Because building software that writes code is one thing; building software that survives gravity, friction, and edge-case physics in a messy warehouse is a completely different beast.

What Actually Is "Physical AI"?

Forget rigid, industrial automation robots programmed with hard-coded paths to do one repetitive task for a decade. Physical AI is the convergence of advanced machine learning, computer vision, and mechanical actuation that allows machines to perceive, reason about, and adapt to unpredictable physical environments in real time.

What makes this possible right now is a heavy convergence of tech stack layers:

Vision-Language-Action (VLA) Models: Just like LLMs bridge text and context, VLA models take multimodal input (vision, spatial telemetry, and natural language prompts) and output motor control. You can literally tell a robot, "Hey, move that blue crate to the top shelf, but watch out for the wet floor," and the model translates that semantic command into physical trajectories.

Synthetic Data & Physics-Based Simulation: Training robots strictly in the physical world is too slow and dangerous. Instead, engineers are using platforms like Isaac Sim and neural graphics to let reinforcement learning agents make millions of mistakes in virtual worlds—learning gravity, momentum, and friction—before deploying the policy weights to edge hardware.

Edge Compute & NPUs: We’re moving away from cloud-dependent robotics (which fail the moment Wi-Fi drops) to powerful Neural Processing Units (NPUs) onboard the chassis. Low-latency edge inference is mandatory when a robot has milliseconds to react to a forklift swerving around a blind corner.

The New Developer Stack: Welcome to Embodied Systems

For web and cloud developers looking at this space, the paradigm shift is fascinating. Your stack isn't just React, Node, and Postgres anymore. It looks a bit more like this:

The Middleware Layer: ROS2 (Robot Operating System) and modern C++/Python bindings acting as the nervous system.

The Perception Layer: Handling point clouds, LiDAR streams, and spatial computing data streams with minimal latency.

The Safety & Control Loop: Writing deterministic safety fallback code because unlike an infinite loop in a web API, a software bug in an actuator can literally put a hole through drywall.

Let's Discuss: Where is the Bottleneck?

Reading through these trends raises a lot of questions for the engineering community. I want to hear from folks currently working in robotics, IoT, edge AI, or even systems architecture:

Hardware vs. Software: Are we gated more by mechanical engineering (battery density, actuator durability, thermal management) or by algorithmic limitations in generalization (the "Sim-to-Real" gap)?

The Developer Experience: The tooling around robotics has historically been notoriously fragmented and painful compared to modern web/cloud developer ecosystems. How close are we to a true "Docker for Robots" moment?

Safety & Ethics: As humanoid and autonomous mobile robots enter shared human spaces, what testing methodologies will we need to prove determinism and safety in neural-network-driven physical agents?

Drop your thoughts, hot takes, or war stories from working with edge AI and robotics in the comments below! 👇