Founder of SAGEWORKS AI — building the Web4 layer where AI, blockchain & time flow as one. Creator of Mind’s Eye and BinFlow. Engineering the future of temporal, network-native intelligence.
The infrastructure strain from agentic AI feels like an early warning of a problem we haven't fully named yet. It's not just "we need more GPUs." It's that the shape of the compute demand changes when the AI isn't just responding but planning.
A single autocomplete request is stateless. Fire and forget. An agent that reasons through a multi-step task, potentially backtracking or trying alternatives, is stateful. It's holding context, maintaining a working memory of its own partial solutions, and consuming tokens not just for output but for its own internal deliberation. That's a fundamentally different load profile on whatever's serving it.
What I'm chewing on is whether this pushes us toward more local inference by default. If agentic workflows are inherently expensive to run centrally at scale, maybe the economic equilibrium lands differently than it did for simpler models. A code completion model makes sense as a cloud service because the inference is cheap relative to the value. But an agent that burns through a few hundred thousand tokens to refactor a module? At some scale, running that locally on a decent GPU starts looking less like a preference and more like the only math that works.
The GitHub pause is probably just growing pains. But it's the kind of growing pain that hints at the ceiling of the current model. Curious if you've found yourself reaching for local models more often as the capabilities get more agentic, or if the convenience of the cloud still wins out despite the wait?
Full-stack developer building AI-powered tools that are free, fast, and actually useful. Creator of Hocks AI & PromptCraft AI. I ship products, write about AI/web dev, and open-source everything.
Stateful vs stateless , that's exactly the right framing and I don't think enough people have named it yet. The sneaky part: the token cost of an agent's internal deliberation is often larger than the cost of its final output, and it's invisible unless you go looking at thoughts . I just wrapped a benchmark on the Gemini 3 family and some Pro calls were burning 1,500+ thinking tokens to produce a one-sentence structured answer from the outside the "request" looks identical to a cheap autocomplete. Multiply that by an agent that also backtracks and you get a 10–30× compute multiplier hiding behind a single billable call.
On the local question, I think the answer is hybrid, not either-or. For short stateless calls (suggestions, extraction, routing), cloud Flash-Lite-class models are still the right math. For a stateful session that's going to burn 100K–500K tokens of planning on a single task, local starts to win less, because per-token inference is cheaper and more because you stop paying roundtrip latency on every step and your working context isn't re-serialized over the wire 20 times. Latency compounds brutally when an agent takes 20 sequential turns.
The pattern I see emerging is explicit routing: a cheap fast model (Flash-Lite, a local 7B) handles the 80% of trivial steps, and a stronger model (Pro, or a local 30B) only gets invoked for the planner. The Copilot pause might actually accelerate that because it forces people to stop pricing agent traffic like autocomplete when the load profiles clearly aren't the same.
Curious if you're seeing orgs bake that routing in explicitly yet, or are most still funneling every turn through the top-tier model?
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The infrastructure strain from agentic AI feels like an early warning of a problem we haven't fully named yet. It's not just "we need more GPUs." It's that the shape of the compute demand changes when the AI isn't just responding but planning.
A single autocomplete request is stateless. Fire and forget. An agent that reasons through a multi-step task, potentially backtracking or trying alternatives, is stateful. It's holding context, maintaining a working memory of its own partial solutions, and consuming tokens not just for output but for its own internal deliberation. That's a fundamentally different load profile on whatever's serving it.
What I'm chewing on is whether this pushes us toward more local inference by default. If agentic workflows are inherently expensive to run centrally at scale, maybe the economic equilibrium lands differently than it did for simpler models. A code completion model makes sense as a cloud service because the inference is cheap relative to the value. But an agent that burns through a few hundred thousand tokens to refactor a module? At some scale, running that locally on a decent GPU starts looking less like a preference and more like the only math that works.
The GitHub pause is probably just growing pains. But it's the kind of growing pain that hints at the ceiling of the current model. Curious if you've found yourself reaching for local models more often as the capabilities get more agentic, or if the convenience of the cloud still wins out despite the wait?
Stateful vs stateless , that's exactly the right framing and I don't think enough people have named it yet. The sneaky part: the token cost of an agent's internal deliberation is often larger than the cost of its final output, and it's invisible unless you go looking at thoughts . I just wrapped a benchmark on the Gemini 3 family and some Pro calls were burning 1,500+ thinking tokens to produce a one-sentence structured answer from the outside the "request" looks identical to a cheap autocomplete. Multiply that by an agent that also backtracks and you get a 10–30× compute multiplier hiding behind a single billable call.
On the local question, I think the answer is hybrid, not either-or. For short stateless calls (suggestions, extraction, routing), cloud Flash-Lite-class models are still the right math. For a stateful session that's going to burn 100K–500K tokens of planning on a single task, local starts to win less, because per-token inference is cheaper and more because you stop paying roundtrip latency on every step and your working context isn't re-serialized over the wire 20 times. Latency compounds brutally when an agent takes 20 sequential turns.
The pattern I see emerging is explicit routing: a cheap fast model (Flash-Lite, a local 7B) handles the 80% of trivial steps, and a stronger model (Pro, or a local 30B) only gets invoked for the planner. The Copilot pause might actually accelerate that because it forces people to stop pricing agent traffic like autocomplete when the load profiles clearly aren't the same.
Curious if you're seeing orgs bake that routing in explicitly yet, or are most still funneling every turn through the top-tier model?