Running a Local LLM on an eGPU Over Thunderbolt

#ai #llm #machinelearning #programming

Everything written about external GPUs is written about gaming, where the link carries a frame’s worth of traffic sixty times a second and the penalty is real. Local inference has the opposite traffic profile, and the conclusion inverts.

What the link actually is

A Thunderbolt enclosure is a PCIe tunnel. The card inside sits on a PCIe bus that is bridged over the Thunderbolt cable to the host, so from CUDA’s point of view it is an ordinary GPU on a narrow, high-latency link.

Thunderbolt 3 and 4 present a 40 Gbit/s link that is shared between display traffic, USB and tunnelled PCIe. Thunderbolt 4’s requirements set a floor on the PCIe share — commonly stated as 32 Gbit/s, which is 4 GB/s — and Thunderbolt 5 roughly doubles it. Treat those as inputs to confirm rather than as given: the share your machine grants depends on the host controller and on what else is plugged in, and the only figure that matters is the one your card negotiates.

The 32 Gbit/s figure is quoted here as the widely-stated Thunderbolt 4 requirement, but it should be checked against the specification sheet for the specific host. The section below gives the command that reads the link the card actually got, which is the number to reason from.

For comparison against the internal slots derived on the PCIe lanes page: 4 GB/s is a little more than PCIe 3.0 x4 and about half of PCIe 4.0 x4. It is a narrow slot, not a catastrophic one.

Load time, derived

The weights cross the link once, on load. That is where the whole cost sits, and it is straightforward arithmetic against published file sizes:

weights over a 4 GB/s tunnel (theoretical), and 2.5 GB/s (with overhead)

  Llama 3.2 3B Q4_K_M   2.02 GB  ->  0.51 s  /  0.81 s
  Llama 3.1 8B Q4_K_M   4.92 GB  ->  1.23 s  /  1.97 s
  Llama 3.1 8B Q8_0     8.54 GB  ->  2.14 s  /  3.42 s
  a 24 GB card filled  ~23   GB  ->  5.75 s  /  9.20 s

same models over PCIe 4.0 x16 (31.5 GB/s)
  Llama 3.1 8B Q4_K_M   4.92 GB  ->  0.16 s

File sizes are those published on the bartowski GGUF model cards (retrieved 11 August 2026). So an 8B loads in about two seconds instead of a fraction of one, and filling a 24 GB card takes something under ten seconds rather than under two. That is the entire penalty, it is paid once, and for a server that loads a model at startup and keeps it resident it is invisible.

It is not invisible if you switch models constantly, or if a runtime unloads after an idle timeout and reloads on the next request. On a Thunderbolt link that idle timeout is worth setting generously, because the reload it causes is four to eight times more expensive than it would be internally.

Why generation does not care

Once the weights are in the enclosure’s VRAM, every read of them happens inside the card, at the card’s own memory bandwidth. The Thunderbolt link carries only the prompt in and the tokens out — a few bytes per token in each direction, plus small control traffic.

24 GB card at 1,008 GB/s internal bandwidth
Thunderbolt PCIe tunnel at ~4 GB/s

ratio: 252x

per generated token, across the link: tens of bytes
per generated token, inside the card: the whole weight set

The link is two and a half orders of magnitude slower than the memory the token loop actually reads, and the token loop does not touch it. This is precisely the opposite of the gaming case, where the host is sending draw commands and vertex data across the link continuously and then pulling a rendered frame back to the laptop’s own display. Inference sends a prompt and receives text.

The consequence worth stating plainly: an external 24 GB card on a Thunderbolt link runs a model that fits in it at close to the rate the same card would achieve internally, and the machine it is plugged into barely matters. That is a genuinely better deal than the eGPU reputation suggests, and it is why an enclosure is a reasonable answer to the capacity ceiling on the laptop GPU page.

Where the argument breaks

Partial offload. If the model does not fit in the external card and layers spill to system RAM, those layers’ activations cross the Thunderbolt link on every token, in both directions, and the link goes from irrelevant to dominant. The rule is stricter here than internally: on an eGPU, make it fit.
Prompt processing at very long contexts. The prompt itself is small, but a runtime that recomputes or re-uploads cache state across the link at each request pays for it. Prefer a long-lived server holding its cache over a per-request process.
Sharing the cable with a display. Driving a monitor through the same Thunderbolt port takes bandwidth from the PCIe tunnel. On a load-heavy workflow, give the enclosure its own port.
Hot-plug and sleep. Suspending the host with an enclosure attached commonly leaves the GPU in a state CUDA will not re-enumerate, producing no CUDA-capable device is detected from an application that worked before the lid closed. See the headless page for how to tell that apart from a driver fault.
Platform support. External GPU support is a property of the host firmware and operating system as much as the enclosure, and it is not universal. Confirm the specific combination before buying either half.

Verifying the link you got

Confirm the card enumerates at all: nvidia-smi --query-gpu=index,name,memory.total --format=csv.
Read the negotiated PCIe link while a model is loading, not at idle: nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv --loop-ms=500. Idle cards drop their link, so an idle reading understates it.
Time a cold load and compare it with the derivation above. A load markedly slower than the arithmetic predicts usually means the source file is being read from slow storage rather than the link being at fault — check by loading the same file twice, since the second load comes from the page cache.
Benchmark generation with llama-bench -m model.gguf -ngl 99 -p 512 -n 128 -r 5 and compare it against the card’s bandwidth ceiling rather than against a review. If generation is near the ceiling, the link is doing nothing to you.
If generation is far below the ceiling, check the offload count in the load log first. A silent partial offload is the most likely explanation and the most damaging one on this link.

The eGPU pattern is usually half of a split: the enclosure runs the local model when you are at the desk, and the same application needs to reach a hosted model when the laptop is unplugged from it. Those two paths differ in authentication, streaming event shape and error semantics, which is a lot of branching for what is conceptually one call. A gateway keeps it one call site with a routing rule, so “is the enclosure attached” stays a deployment question rather than a code path.