Profile the model, find the kernel or fusion hotspots, and treat later kernel changes as candidates in a search. Run the trials in parallel on idle GPUs, including overnight, and keep going while there is compute until the workload is performant.
Anush Elangovan, Corporate VP of AI Software and Solutions at AMD, laid this out on Chain of Thought in September 2026. ROCm 10 is a stack an agent can install and then use to serve a model. Hyperloom profiles for kernel or fusion hotspots; GEAK searches kernel changes while maintaining numerical accuracy. With enough compute, he said, Hyperloom optimized about 14,000 models in one pass, and that led to a big lift in performance.
How do you keep a faster kernel numerically honest?
Write the tolerance down before the first trial, and drop any config that breaks it. Speed ranks only the configs that still match the reference.
Elangovan split the work into two tools. A profiling and analysis step takes the model and finds kernel hotspots or fusion hotspots. A toolkit he called GEAK then takes the slow kernels and searches for better utilization while maintaining the same numerical accuracy.
Freeze a small set of real inputs and the outputs of the kernel you run today. After each candidate, compare against that reference with a max absolute difference, or a relative difference when values span a wide range. Store the diff beside the timing.
Search itself is old. At a company he founded before AMD acquired it, an autotuner defined a search space and walked it with a particle filter or a genetic algorithm. What he called new is using AI and LLMs for more educated picks and for route finding inside that space. A smarter proposer does not replace the check. The model can suggest a fusion. The tolerance still decides whether you keep it.
What should an overnight GPU tuning loop run?
Profile once, generate candidates, reject accuracy misses, time the rest, and repeat while the GPUs are free.
Hyperloom can optimize on the fly while you sleep, and the same work can run in parallel when you are not using the system. Smaller modules can run on their own, starting from profiling. The tuner structures the job into well-defined problems and defined goals. Pointing Claude or Codex at a loose speed request may not give you enough. The spec those goals stand in for is short: inputs, reference outputs, tolerance, the metric you care about, and a stop rule.
This is an illustrative sketch, not a production library or anyone's actual implementation.
import math
def tune_until_budget(candidates, reference, tolerance, gpu_hours):
if not math.isfinite(tolerance) or tolerance < 0:
raise ValueError("tolerance must be finite and non-negative")
if not math.isfinite(gpu_hours) or gpu_hours < 0:
raise ValueError("gpu_hours must be finite and non-negative")
best_config = None
best_time = None
spent = 0.0
for config in candidates:
remaining = gpu_hours - spent
if remaining <= 0:
break
# Helpers cancel and return None when the launch would pass the cap.
launched = run_kernel(config, remaining)
if launched is None:
break
outputs, elapsed, cost = launched
if not math.isfinite(cost) or cost < 0 or cost > remaining:
break
spent += cost
diff = max_abs_diff(outputs, reference)
if not math.isfinite(diff) or diff > tolerance:
continue
remaining = gpu_hours - spent
if remaining <= 0:
break
timed = benchmark(config, remaining)
if timed is None:
break
timing, bench_cost = timed
if (
not math.isfinite(bench_cost)
or bench_cost < 0
or bench_cost > remaining
or not math.isfinite(timing)
):
break
spent += bench_cost
if best_time is None or timing < best_time:
best_config = config
best_time = timing
return best_config, best_time
run_kernel and benchmark are stand-ins for whatever launches your kernel and times a steady-state batch. Accuracy is a hard filter. Time is the sort key. gpu_hours is the budget. Independent configs do not wait on each other, so queue them on idle devices.
Specifically about Hyperloom, you know, I think we had a case where we optimized like 14,000 models in one pass, right? So we took all the models and just as long as you have enough compute, you can try different versions of our search space explorations of the same kind of model.
Anush Elangovan, Corporate VP of AI Software and Solutions at AMD, on Chain of Thought ep 72
Tuning GPU Performance with AI… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts
Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · September 9 · 53m
podcasts.apple.com
And then it also can do performance optimizations when you're sleeping. So you can run a tool called Hyperloom that we have that allows you to just optimize on the fly. And so you just continue to do that. until it gets to a point where it's really performant. And this can work in parallel, especially when you're not using the system, we can continue to optimize.
Anush Elangovan, Corporate VP of AI Software and Solutions at AMD, on Chain of Thought ep 72
How do you budget kernel experiments across hardware?
Cap GPU-hours and candidate count, and count a win only on the devices you ship.
Set a GPU-hour cap, a max number of candidates, and a patience rule: stop after a fixed number of accepted trials with no improvement. Parallel runs spend that cap sooner. They do not raise it.
His team stood up an order of magnitude more CI over the last year, across multiple hardware generations, on the frameworks they track. PyTorch, vLLM, and SGLang are at 100% enablement, so upstream pull requests can be blocked when they would regress AMD hardware after being tested only on NVIDIA. A tuning loop has the same hole if every candidate is timed on one box. Put one device from each generation you support on the overnight queue, even if that means fewer candidates per device.
Agents can bring the stack up before the search. A developer with Claude Code or Codex can tell it to install ROCm, because the skills are already integrated. Asked to serve a model with vLLM, the agent reads the docs, pulls in the skills, resolves dependencies, and brings the service up. That baseline comes first. The accuracy-gated loop runs after it exists.
Agent traces change capacity planning, not the kernel-experiment budget. A human session might be someone logging in and asking for a poem. Agent traces are sustained queries, file reads, and searches. If that pattern is the base load, plan for 100x what engineers or customers had been using.
So the entire profile is now shifting towards like, okay, if that is the base level of operation, we need to plan for 100x of what our engineers or customers may have been using.
Anush Elangovan, Corporate VP of AI Software and Solutions at AMD, on Chain of Thought ep 72
FAQ
Where does the accuracy check sit in the loop?
Before you rank speed. Elangovan described GEAK taking slow kernels and searching while maintaining the same numerical accuracy. In the sketch above, a config past tolerance is skipped, and benchmark only updates the winner after that check.
Can the search keep going while you are away?
Yes. Hyperloom can keep testing performance optimizations while you sleep, in parallel, when you are not using the system. Leave it a GPU-hour cap so a free machine does not run without a stop.
Does CI on one vendor cover a kernel change?
Code tested only on NVIDIA can still regress AMD. An order of magnitude more CI, across generations, on PyTorch, vLLM, and SGLang is how his team blocks those pull requests. Time your candidates on each generation you ship.
Takeaway
- Freeze reference outputs and a numeric tolerance before any kernel edit.
- Profile first, and spend trials on kernel or fusion hotspots.
- Reject any candidate that misses the tolerance, then rank the rest by speed.
- Run independent configs in parallel on idle GPUs, and stop on a GPU-hour budget.
- Re-test on every hardware generation you ship, and benchmark with a sustained agent trace.
The full conversation, with the transcript, is on Chain of Thought.
You can also watch it on YouTube.
I write about what AI builders are learning every week in the Chain of Thought newsletter.
Drafted with AI assistance from the episode transcripts, then edited by me.
Top comments (1)
The predeclared tolerance is the right admission gate. I would also retain a small adversarial input suite alongside representative inputs—edge shapes, denormals if relevant, and values near branch thresholds—because a candidate can match a happy-path reference while changing a rare numerical regime. Recording the input-set hash with timing results makes a later speed claim comparable.