“Cannot Import Name ‘AutoGPTQForCausalLM’”

#ai #debugging #llm #machinelearning

ImportError: cannot import name 'AutoGPTQForCausalLM' from 'auto_gptq' — or from transformers, or from a partially initialised module. The words after “from” are the diagnosis, and they point at four unrelated problems.

Read the module name in the message

Python raises this exact wording when a module imported successfully but does not export the requested name. Which module it names tells you where to look:

from 'auto_gptq' — the package is installed but its package initialiser did not bind the class. Almost always an inner failure that got swallowed, or a version where the symbol moved.
from partially initialized module 'auto_gptq' with a note about a circular import — a file or directory in your working directory is named auto_gptq and is shadowing the installed package. Python searches the current directory first.
from 'transformers' — the class was never in transformers. Transformers exposes GPTQ through GPTQConfig and AutoModelForCausalLM, not through this name.
A different name entirely, such as cannot import name 'shard_checkpoint', means you are past this error and looking at the real one. Follow that message instead.

Check for shadowing first because it is instant and it makes every other theory irrelevant:

python -c "import auto_gptq, sys; print(auto_gptq.__file__, auto_gptq.__version__)"
ls -d auto_gptq auto_gptq.py 2>/dev/null   # anything here is shadowing the package

Cause 1: the import chain failed underneath

auto-gptq compiles CUDA extensions, and its package initialiser imports modules that load them. When an extension cannot load, the failure propagates out of the package initialiser, and depending on how the calling code catches it you can end up with a name that was never bound.

The reported instances of this are specific. A widely-discussed case on hosted notebooks was ImportError: libcudart.so.12: cannot open shared object file raised while importing the exllama kernels: the wheel had been built against CUDA 12 while the environment provided only the CUDA 11 runtime, so the extension could not load even though nvidia-smi reported a CUDA 12 driver. The driver version and the installed runtime libraries are different things, and only the second matters for loading a compiled extension.

The way to see it is to import the layer below and let the real error surface:

python - <<'PY'
import traceback
try:
    import auto_gptq
    print("ok", auto_gptq.__version__)
except Exception:
    traceback.print_exc()      # the real error is at the bottom of this chain
PY

python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

If the bottom of that traceback names a shared library, it is a build-versus-environment mismatch and no amount of reinstalling auto-gptq alone will fix it; the wheel must match the CUDA runtime and the torch build. If it names an exllama kernel specifically, the exllama kernel page covers that path.

Cause 2: transformers removed what auto-gptq imports

auto-gptq imports internals from transformers, and transformers removes internals. The prominent example is shard_checkpoint, which was deprecated and then removed from transformers.modeling_utils; libraries importing it broke on transformers 4.49 and later, producing cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' during the auto-gptq import. The same pattern recurs whenever a private helper is removed.

This is the version-mismatch case the search results are usually about, and it has three possible resolutions, in descending order of how long they last: move to the maintained successor package; pin transformers to a version the library was written against; or patch the import. The middle one is a holding action — pinning transformers backwards drags in a tokenizers version and blocks every newer model architecture — and the third is not something to carry in production.

Cause 3: the class is not in that package

AutoGPTQ development stopped and the project was archived in April 2025; its maintainers direct users to GPTQModel, and Hugging Face transformers moved its GPTQ backend to GPTQModel accordingly. That matters here because the two packages do not share the class name. Installing GPTQModel and keeping from auto_gptq import AutoGPTQForCausalLM produces exactly this error, and no amount of reinstalling will produce the symbol.

Package status moves. AutoGPTQ’s archival and GPTQModel’s role as the maintained backend are as of the writing of this page; check the transformers GPTQ documentation for the current recommended package before pinning anything.

The modern path does not import a GPTQ class at all. Transformers loads a GPTQ-quantised checkpoint through its normal entry point, detecting the quantisation from the checkpoint’s own config:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "TheBloke/Mistral-7B-Instruct-v0.2-GPTQ"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

ids = tok("Explain GPTQ in one sentence.", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**ids, max_new_tokens=64)[0]))

Before adopting that path, confirm the checkpoint really is GPTQ. The repository’s config.json carries a quantization_config block naming the method, the bit width, the group size and whether activation ordering was used. If that block says awq or bitsandbytes, no GPTQ package will load it and the import error was a detour from the start. If it says gptq, the same block tells the backend which kernel to select, which is why loading through transformers needs no arguments from you.

The path forward

Rule out shadowing: no file or directory named auto_gptq in your working directory, and auto_gptq.__file__ pointing into site-packages.
Import the package alone and read the whole traceback. If the bottom line names a shared object or another module, that is the error to work on.
If it names a removed transformers symbol, decide between the maintained successor and a pin. Prefer the successor.
If you are on a maintained fork, delete the import line and load the checkpoint through transformers directly. There is no GPTQ-specific class in that path.
If you only need inference on a machine without a working CUDA toolchain, a GGUF build of the same model run through one of the local inference runtimes avoids compiled Python extensions entirely.

One thing worth doing once rather than repeatedly: build the environment from a pinned set rather than installing packages in the order you discover you need them. GPTQ inference in Python depends on a compiled extension matched to a torch build matched to a CUDA runtime, and pip will happily upgrade torch underneath an extension that was compiled against the previous one, at which point a working setup starts producing import errors nobody changed anything to cause. A requirements file that pins torch, transformers and the quantisation package together, installed into a fresh virtual environment, converts this from a recurring mystery into a one-time decision.

The wider lesson is that any import error in that stack is usually the outermost symptom of a mismatch two layers down, and the message Python prints names the outermost layer only. How GPTQ differs from the other quantisation schemes covers why those kernels exist in the first place.