Plan inference capacity before an agent feature ships

Shipping an agent feature on a capacity guess is how you find the ceiling in production. Jeetu Patel, president and chief product officer at Cisco, names that ceiling as infrastructure, ahead of another round of model comparison.

There is just not enough power, compute, network bandwidth, and data center capacity to go satiate the needs of AI.

Jeetu Patel, President and Chief Product Officer at Cisco, on Chain of Thought episode 44

He pairs that list with a simple split: infrastructure is the input, intelligence is the output. For one feature you are about to launch, the input is a worksheet. I would fill it for a single successful task. Any number the episodes do not give you stays a blank until you measure it. Token counts, concurrency, kilowatts, and link speed are blanks on purpose.

What counts as load for one agent feature?

Inference is the phase that recurs. Training builds the model. Inference is each prompt in and each answer out. For teams building on existing models, that recurring cost is the one that scales with use.

An agent feature spreads that cost across a compound AI system: several model calls, retrieval, tool use, validation, and the control logic that ties them together. In that shape, performance depends on how the parts fit, and swapping in a better model often helps less than fixing the system around it.

Write the success path first.

• Model steps in one completed task.
• Retrieval and tool calls that trigger more inference.
• Validation passes.
• Retries you will actually allow.

Anush Elangovan, corporate VP of AI software at AMD, argues that this path will not look like a person sending one request. He contrasts older interactive use with agentic traces that keep issuing work, then says:

So the entire profile is now shifting towards like, okay, if that is the base level of operation, we need to plan for 100x of what our engineers or customers may have been using.

Anush Elangovan, Corporate VP of AI Software at AMD, on Chain of Thought episode 72

That 100x is his planning claim about sustained agent traffic versus prior human use. Put your own multiplier in the cell. Use his figure as a stress case when you have no trace yet: sustained calls, file this, search this, grab this, over and over. If you cannot say what your feature does under that pattern, cap the launch.

Jeetu expects a different model mix on each side of the market. On the consumer side, he expects about a dozen models to carry most inference volume. On the enterprise side, he expects many models, and a routing layer so some queries take a cheaper compute path while others stay on a costlier one. For this feature, write which step may move and which step must stay, and only after you have tested both.

He also reports that an open source 8 billion parameter model, trained on the right data, can now perform better than a 70 billion parameter model, and that the smaller one can be quantized and run on a CPU in a laptop. Scope that to his report. If a step in your feature can run that way, the GPU cell can stay empty, and you still owe a latency number.

How do I fill the worksheet before launch?

This sketch is an illustration for the sheet. Guests did not publish it, and it calls no vendor API. Every None is a number missing from the source.

# Sketch only. Replace each None with your own measurement.

def calls_per_week(steps, tasks_per_user, users, extra_calls, retries):
    return steps * tasks_per_user * users * (1 + extra_calls) * retries

feature = {
    "steps_per_task": None,
    "extra_calls_per_step": None,
    "retries": None,
    "tasks_per_user_week": None,
    "users_at_launch": None,
    "tokens_in": None,
    "tokens_out": None,
    "peak_concurrent_tasks": None,
    "latency_budget_ms": None,
    "power_budget_kw": None,
    "link_gbps": None,
    "hosts_or_gpus": None,
    "measured_kernel_fraction": None,
}

After the blanks are real, I would estimate token volume as weekly calls times tokens in plus tokens out. As a conservative bound, I would treat peak concurrent tasks times steps as the burst that has to meet latency_budget_ms, until a trace shows how many steps actually overlap. If measured_kernel_fraction is still None, treat GPU capacity as unknown. AI hardware work keeps separating the chip you bought from the throughput you get after kernels, runtime, and the serving stack.

The decision rule I would use: launch only at a concurrency where power, hosts, link speed, and kernel fraction are filled. Unknown means a smaller user cap, or this feature waits.

Where do power, network, and placement go on the sheet?

Jeetu's hardware picture is specific. Power is the constraint, the GPU is the core asset, and the network is the force multiplier. Without the network, the GPUs do not operate. He traces training from one GPU, to a server of four to eight GPUs, to a rack, to a row, then to two sites hundreds of kilometers apart acting as one cluster: scale up, scale out, and scale across. A training run at 300,000, 500,000, or a million GPUs may not fit a data center that holds 50,000, 70,000, or 100,000. He says a dropped packet there forces a restart costing millions of dollars.

One agent feature is inference, so the training restart cost does not belong in your token math. Placement and the path between machines do. He says data centers are being built where power availability is highest, and he does not expect that to reverse. Write the site where you can get power and hosts.

Latency is the second placement test. He argues for inference on the edge when the round trip will not do, including robotics, and describes Cisco Unified Edge as networking, security, and compute in one box for a branch, hospital, or factory floor without local IT staff. If your latency budget fails from the site that has power, add an edge host with its own power and bandwidth cells.

Kernel headroom stays on the compute row. Anush describes profiling that finds kernel and fusion hotspots, then a search that keeps numerical accuracy, including a pass over 14,000 models when enough compute is available. That is his account of AMD's stack. Until you measure a fraction of peak for this model, this batch, and this context length, hosts_or_gpus is a nameplate.

Jeetu names two other brakes beside infrastructure: a trust deficit, because nondeterministic models sit under applications people want to be predictable, and a data gap as public human text runs thin while machine data grows. He offers 10 or 100 agents per human as the scale of logs, metrics, events, and traces. I would keep trust and that data gap off this sheet, and add one trace-volume line, because those traces consume bandwidth and storage while the agent runs. Leave it blank until you have one trace from this feature.

Checklist

• Pick one feature and one success path, and count model steps, retrieval, tools, validation, and retries.
• Fill token, concurrency, latency, power, link, and host cells from measurement, and treat any blank as a launch blocker.
• Run a sustained-traffic case with your own multiplier, using the 100x claim only as a stress prompt.
• Choose the serving site because power and hosts exist there, and add an edge row if latency fails from that site.
• Measure the kernel fraction for this workload, or leave GPU capacity marked unknown.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.