DEV Community

niuniu
niuniu

Posted on

Quick Tip: Run Any Hugging Face Model on Colab's Free T4 GPU (No Credit Card, No Local Setup)

Quick Tip

Your laptop has no GPU and you want to test a model from Hugging Face? Don't set up anything. Colab's free T4 runs most 7B-8B models fine in fp16:

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "Qwen/Qwen2.5-Coder-7B-Instruct"  # any causal LM on the Hub
pipe = pipeline(
    "text-generation",
    model=AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="float16", device_map="auto"),
    tokenizer=AutoTokenizer.from_pretrained(model_id),
)
out = pipe("Write a Python function to retry failed HTTP requests:", max_new_tokens=200)
print(out[0]["generated_text"])
Enter fullscreen mode Exit fullscreen mode

Why this beats what you're doing now

Option Cost Setup time GPU
Buy cloud GPU (A10G) ~$0.75/hr 10 min + billing Yes
Run on CPU locally $0 30 min of pain No — 8B model = 3 tok/sec, unusable
Colab free T4 $0 2 min Yes — 25-30 tok/sec on 7B fp16

The tricks that save you from the usual Colab pain:

  1. torch_dtype="float16" — halves VRAM. 7B model = ~15GB in fp16, fits the T4's 16GB with room to spare. Skip this and you OOM immediately.
  2. device_map="auto" — requires pip install accelerate. Spills to CPU automatically if you pick a model that's slightly too big.
  3. Use huggingface_hub[hf_transfer] for downloads — a 15GB model pulls in ~2 minutes instead of 20: 
!pip install -q accelerate "huggingface_hub[hf_transfer]"
import os; os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
Enter fullscreen mode Exit fullscreen mode
  1. Session limits: free tier gives you ~12 hours max per session and disconnects on idle. Save outputs to Drive, not local disk: 
from google.colab import drive
drive.mount('/content/drive')
Enter fullscreen mode Exit fullscreen mode

Genuinely useful models that fit on the free T4 right now: Qwen2.5-Coder-7B (code), Llama-3.1-8B-Instruct (chat, needs license acceptance), Phi-4-mini (fast, reasoning), Mistral-7B-Instruct-v0.3.

No credit card, no API key for public models. I keep a running list of which models actually fit in 16GB here: https://ly.cyberserval.tech/iIETXiF

What's the biggest model you've managed to squeeze onto Colab's free tier? Anyone gotten a 13B to work with 4-bit quantization without it being painfully slow?

Top comments (0)