Buying a Used GPU for Local LLM Inference

Most advice about buying a used graphics card is written for gaming, where the risks are cosmetic and the workload is bursty. Inference is neither. It is a sustained, memory-bound, thermally flat load, and it fails on different things.

Why inference stresses a card differently

A game pushes the shader cores hard and the memory moderately, in bursts, with the fans ramping up and down. Single-stream generation does close to the opposite: the arithmetic units are largely idle waiting on memory, because every token requires reading every weight once and the compute per byte read is tiny. The card sits at high memory utilisation and moderate core utilisation for hours at a time.

Three practical consequences. First, memory temperature matters more than core temperature, and GDDR6X in particular runs hot; a card whose thermal pads have degraded will throttle on memory junction temperature long before the core is uncomfortable. Second, the load is flat, so a marginal power delivery stage that survives a gaming session by getting idle gaps will not get them. Third, a fault in a memory module is far more likely to show up as a wrong token or a silent corruption than as a visual artefact you would notice.

The often-repeated worry — that a card used for mining is ruined — is the wrong shape. Mining is also a flat, memory-heavy load, often undervolted, and a card run that way at a controlled temperature can be in better condition than one that spent three years in a hot case with no case fans. What matters is the thermal history and whether the pads and paste were ever serviced, and neither of those is visible from the listing.

VRAM is the only spec with no substitute

Everything else on a slower card degrades gracefully. Less bandwidth means fewer tokens per second, which is annoying. Fewer shader cores means slower prompt processing, which is annoying. Insufficient VRAM means the model does not run, or runs across the partial-offload cliff described on the 12GB page, which is not a degradation but a different order of magnitude.

So the ranking for a used purchase is: capacity first, bandwidth second, everything else a long way behind. A 24 GB card two generations old beats a 12 GB card one generation old for this workload, and it is not close, because the older card runs models the newer one cannot load at all. Work out the capacity you need with the arithmetic on the 24GB page before you look at a single listing, and treat the resulting number as a hard filter.

One genuine advantage of the older end of the market: NVIDIA’s RTX 3090 product page lists 24 GB of GDDR6X on a 384-bit interface, 350 W board power, and an NVLink connector — the last GeForce generation to have one. Whether that connector is worth anything for inference is covered on the two-GPU page, and the answer is less than people assume.

The driver support window is a purchase date

A GPU stops being useful when its driver branch stops receiving updates, because the CUDA runtime version that new builds of inference software require moves forward. This is the check people skip and it is the one that decides how many years the card is worth.

The mechanism to understand: each GPU has a compute capability, the driver has a supported architecture list, and each CUDA toolkit release drops support for the oldest architectures. When a runtime is built against a toolkit that no longer emits code for your card, you get no kernel image is available for execution on the device — the card is fine, the binary simply has nothing in it for that architecture. Prebuilt wheels and prebuilt llama.cpp releases hit this before source builds do, because you can still compile for an older architecture yourself for a while after the shipped binaries stop including it.

Which architectures a current driver branch and toolkit still support changes with every major CUDA release. Check the compute capability of the specific card against the current toolkit’s supported list before buying, rather than against a list from a year ago.

The same argument applies on the AMD side with ROCm, which supports a notably narrower set of consumer cards than the marketing implies, and on Intel’s side with oneAPI. In all three cases the question is not “does this card work today” but “is it on the current support list, or is it one release from falling off”.

What to check physically

• Power connectors and their history. Count them and match them to what your supply has, then size the supply properly — see PSU sizing. On any card using the 12VHPWR connector, inspect the connector housing for discolouration or melted plastic; that failure mode is well-documented and it is caused by incomplete seating, which a previous owner may have lived with.
• Physical length, slot width and case clearance. The 4090 is listed by NVIDIA at 304 mm long and three slots wide. A card that does not fit is a return, and a card that fits with no clearance for airflow will run hot for the whole of a sustained load.
• Fan and bearing condition. Spin each fan by hand. Grinding or resistance means a replacement, which for many cards is cheap and easy, but it should come off the price.
• Signs of a repad or a reflow. Neither is disqualifying — a repad is maintenance — but stripped screws, damaged warranty stickers and residue tell you the card has been open, which you want to know before rather than after.

A verification pass before the money moves

If you can run anything on the card before buying, run something that touches all of the memory, because that is the failure you care about and the one a five-minute gaming test will not find.

1. Confirm the card is what the listing says: nvidia-smi --query-gpu=name,memory.total,pcie.link.width.max --format=csv. A relabelled or modified card shows up here.
2. Check the link it negotiates, not just the one it supports: nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv under load. A card that trains down to a narrower link has a problem — see what lane width costs.
3. Fill the memory and keep it filled. Loading a model sized to within a gigabyte of the card’s capacity and generating for ten minutes exercises the whole memory array under a realistic thermal load.
4. Watch for throttling while it runs: nvidia-smi -q -d PERFORMANCE reports clocks throttle reasons. HW Slowdown or a memory-temperature throttle appearing within minutes is a card with a cooling problem.
5. Compare against the ceiling, not against a review. Divide the card’s published bandwidth by the model’s file size to get the batch-size-one ceiling, and treat anything wildly below it as a reason to investigate the configuration or the card.

Related

• Local LLMs on a 12GB RTX 3060, by the Numbers
• Picking a GPU by VRAM per Dollar for Local Inference
• Sizing a Power Supply for a Local LLM GPU Rig