DEV Community

niuniu
niuniu

Posted on

Quick Tip: Run Hugging Face Models on Google Colab's Free GPU (No Local Setup)

You don't need a $1,000 GPU to experiment with 7B-parameter models. Google Colab's free tier gives you a T4 GPU for up to 12 hours, and you can run any Hugging Face model in about 30 lines of Python.

The One-Cell Setup 

# Cell 1: Install and load (run once per session)
!pip install -q transformers accelerate bitsandbytes

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "microsoft/Phi-3-mini-4k-instruct"  # 3.8B params, fits on T4

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    load_in_4bit=True  # 4-bit quantization: 3.8B → ~2GB VRAM
)
Enter fullscreen mode Exit fullscreen mode

Generate Text 

# Cell 2: Chat with the model
def chat(prompt: str, max_new_tokens: int = 256):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

print(chat("Explain Python decorators like I'm a JavaScript developer:"))
Enter fullscreen mode Exit fullscreen mode

Free Tier Reality Check

Resource Colab Free Colab Pro ($10/mo) Local (RTX 4090)
GPU T4 (16GB) A100 (40GB) 24GB
Session limit 12 hours 24 hours Unlimited
Idle timeout 90 min 90 min Never
7B model inference ✅ 4-bit ✅ bf16 ✅ bf16
Cost $0 $10 $1,600

Pro Tips

Save your session: Colab disconnects after 90 min idle. Keep the tab active or use:

# Auto-reconnect helper
from google.colab import output
output.eval_js('new Audio("https://upload.wikimedia.org/wikipedia/commons/0/05/Beep-09.ogg").play()')
Enter fullscreen mode Exit fullscreen mode

Model size guide for T4 (16GB):

  • ✅ 3B models (Phi-3, Llama-3.2-3B): bf16 or 4-bit
  • ✅ 7B models (Mistral-7B, Llama-3.1-8B): 4-bit only
  • ❌ 13B+ models: Won't fit, use Colab Pro or API

Persist models across sessions:

from google.colab import drive
drive.mount('/content/drive')
# Save tokenizer and model to Drive, reload later
Enter fullscreen mode Exit fullscreen mode

Integration with Local Workflows

I prototype on Colab, then move to local Ollama + MonkeyCode for the actual coding work. The free AI coding assistant runs 100% on my machine, no cloud dependency: https://ly.cyberserval.tech/iIETXiF

# MonkeyCode config for Colab-generated code
{
  "provider": "ollama",
  "model": "codellama:7b",
  "base_url": "http://localhost:11434"
}
Enter fullscreen mode Exit fullscreen mode

What's your Colab workflow? I'm looking for tricks to stretch the 12-hour limit without paying for Pro.

Top comments (0)