Much of the cost of switching accelerators lives in the software around the model. Memory capacity is the part a roadmap can close; a kernel tuned for one device is the part you may have to redo. I would start with a saved run from the old device and a list of which code may move unchanged.
Which parts of the model should already run?
Anush Elangovan at AMD calls the older experience a port to platform. A model launches, then someone else makes it work. He now points to day-zero support for recent frontier releases, including DeepSeek, Llama, and Qwen, running natively, as well as they do on the competing platform. That confidence is what a buyer is paying NVIDIA for: a model announced in the morning is supposed to run that afternoon.
In a later conversation he treats PyTorch and vLLM as fully enabled, and says AMD can block upstream pull requests that were tested only on NVIDIA so those changes do not regress AMD. I would confirm both claims for the model you actually serve. Day-one support names the models. Upstream testing is how you catch a framework change that was only proven on NVIDIA.
But most of the frameworks like PyTorch, VLLM, SG-Line, all of those are now currently like 100%, you know, like we have 100% enablement. And then there's obviously, you know, the benefits of having that kind of CRA like we can block PRs that go in and are always tested on AMD, which gives us the ability to ensure that the upstream projects do not get into a stage where they just regress AMD and the code was just tested on Nvidia.
Anush Elangovan, Corporate VP of AI Software at AMD, on Chain of Thought episode 72
Export the modules from the trusted run. Mark each one as a stock op in an enabled framework, or as code your repo builds. My working assumption: the stock ops are where enablement work has already happened, and the code you compile yourself is where your port effort goes.
The 350 series ships 288 GB per GPU, enough on AMD's framing for a 500-billion-parameter model on one card. Blackwell Ultra matches that 288 GB. The MI455X, launched in July 2026, reaches 432 GB. I would not pick the target from that number. Hardware generations are compared across the AI hardware episodes, and those comparisons still leave your kernel list untouched.
Where do custom kernels and attention bite?
Elangovan says AMD's internal and external repositories are identical and that outside contributions are taken seriously and merged. Contributors used that path to land Triton on Windows and PyTorch on Windows for the Strix Halo laptop. A kernel that lives only in your private repo does not benefit from that community work, so plan to port it yourself.
Standard attention compares every token with every other token, so time and memory grow with the square of the sequence length. FlashAttention, from Tri Dao and Stanford colleagues in 2022, targets a different cost: traffic between high-bandwidth memory and on-chip SRAM. It computes attention in tiles that fit in SRAM, so the full attention matrix is never written out to high-bandwidth memory. The output is the same attention result. The paper reports a 15% end-to-end training speedup on BERT-large, sequence length 512, against the MLPerf 1.1 record.
Check that match on your own tokens before you keep a speed number. Because the tiles are sized to fit on-chip SRAM, I would re-benchmark any tuned attention kernel on the new hardware rather than assume the old speedup carries over.
The inference-first cut comes from Sharon Zhou, AMD's VP of AI when we spoke in June 2025.
So right now, there's a lot of focus on inference and being able to actually make that workload really effective and reliable and efficient. And so how do we actually engage developers on that specifically today? It's more ready than training right now.
Sharon Zhou, AMD, on Chain of Thought episode 28
She also splits agent work into GPU model calls and CPU tools, including tools reached through MCP. I would put those CPU tools on their own line. She sees room to balance the two because AMD ships both kinds of compute. The port list should still show both.
How do I check numerics and speed against the old run?
Autotuning is the older pattern Elangovan describes: define a search space, sample configurations, and measure the results. An AI-guided tuner can choose the next sample with more information. The part I would not skip is the measurement. A faster candidate still has to produce the output you already trust.
Hyperloom is the inference loop AMD describes. It profiles a workload, searches serving, framework, and GPU-kernel changes, then measures the candidates again. Elangovan splits that into a profiling pass for hotspots, including kernel and fusion hotspots, and a later pass over slow kernels that keeps the same numerical accuracy. Save the reference before that search starts.
Use the same checkpoint, the same input ids, the same batch, and the same sequence length. Keep the logits, any per-module tensors you can dump, and the step time. This function is a sketch I wrote for the diff. It is not an AMD library, and it is not a method a guest demonstrated. Set atol and rtol for the dtype you stored.
# Sketch only. Each item is (layer_name, tensor, latency_ms).
def check_port(reference, candidate, atol=1e-3, rtol=1e-3):
rows = []
for (name, ref, ref_ms), (_, new, new_ms) in zip(reference, candidate):
max_abs = (ref - new).abs().max().item()
scale = ref.abs().max().clamp(min=1e-8).item()
rows.append({
"layer": name,
"max_abs": max_abs,
"ok": max_abs <= atol or (max_abs / scale) <= rtol,
"speedup": (ref_ms / new_ms) if new_ms else None,
})
return rows
Reject any row that fails, including a row with a large speedup. When fusion removes the boundaries you saved, compare final logits on the same token ids first. Then turn fusion off and bisect until one module explains the gap.
Paige Bailey describes a stack this checklist does not stand in for. TPUs sit under XLA, JAX sits on XLA, and she says people love JAX partly because it uses TPUs so well, including across data centers for training Gemini. Those choices come from owning the path from the compiler up. Teams renting someone else's stack do not get that vertical integration, which is why the port checklist matters.
Checklist
- Save the old run: input ids, per-layer tensors where you can get them, final logits, and latencies, on one fixed batch and sequence length.
- Split modules into stock ops, custom kernels such as Triton, and any tuned attention implementation.
- Confirm PyTorch and vLLM are enabled on the new hardware, and ask whether upstream pull requests are tested there before they merge.
- Diff numerics against the reference before you keep a speedup. Drop a candidate that is faster and wrong.
- For an agent, list GPU model calls and CPU-side tools as two ports.
The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)