DEV Community

niuniu
niuniu

Posted on

Quick Tip: Run Any Hugging Face Model on Google Colab's Free T4 GPU — No Credit Card, No Setup

#ai #python #tips #opensource

You don't need a $400 GPU or an OpenAI API key to run serious LLMs. Google Colab gives you a free NVIDIA T4 (16GB VRAM) — enough to run 7B-8B models at usable speed. Here's the 5-cell setup I use daily.

Cell 1: Check your GPU (Runtime → Change runtime type → T4) 

!nvidia-smi
# Tesla T4, 15360MiB — that's $0.35/hr on AWS, free here
Enter fullscreen mode Exit fullscreen mode

Cell 2: Install 

!pip install -q transformers accelerate bitsandbytes
Enter fullscreen mode Exit fullscreen mode

Cell 3: Load a model in 4-bit (fits any 8B model in 16GB) 

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL = "Qwen/Qwen2.5-Coder-7B-Instruct"

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    load_in_4bit=True,          # bitsandbytes magic
    device_map="auto",
    torch_dtype=torch.float16,
)
Enter fullscreen mode Exit fullscreen mode

Cell 4: Generate 

def ask(prompt, max_new_tokens=300):
    inputs = tok(prompt, return_tensors="pt").to("cuda")
    out = model.generate(**inputs, max_new_tokens=max_new_tokens)
    return tok.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)

print(ask("Write a Python function that retries an async HTTP call with exponential backoff:"))
Enter fullscreen mode Exit fullscreen mode

Runs at ~25 tokens/sec on the free T4. Not GPT-4 speed, but $0.

Cell 5 (pro tip): Persist your models

Colab wipes your disk every session. Mount Google Drive so the 5GB model downloads once:

from google.colab import drive
drive.mount('/content/drive')

import os
os.environ['HF_HOME'] = '/content/drive/MyDrive/hf_cache'
# reload cells 3-4 — model loads from Drive in 10s instead of re-downloading
Enter fullscreen mode Exit fullscreen mode

The catch (there's always one)

  • Sessions cap at ~12 hours and idle-timeout after ~90 min
  • Free tier is "priority access" — on busy days you wait in a queue
  • T4 is 2018 hardware; anything over 8B parameters crawls

For anything longer-running, I run the same models locally with Ollama, and write the glue code with a free AI assistant (MonkeyCode — no subscription): https://ly.cyberserval.tech/iIETXiF

What do you use Colab's free tier for — fine-tuning, inference, or something weirder?

Top comments (0)