DEV Community

niuniu
niuniu

Posted on

Quick Tip: Run Llama 3.1 8B on Colab's Free T4 GPU in 5 Lines of Python

I needed to test a fine-tuning script but my laptop has integrated graphics. Cloud GPU rentals start at $0.50/hour. Google Colab's free tier gave me a T4 for $0.

The 5 lines 

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto",
    load_in_4bit=True  # fits in 16GB T4
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

inputs = tokenizer("Explain quantum computing in one sentence", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Enter fullscreen mode Exit fullscreen mode

What you get for $0

Resource Colab Free Colab Pro ($10/mo) Lambda Labs
GPU T4 (16GB) A100 (40GB) A100 (40GB)
VRAM 16GB 40GB 40GB
Cost/hour $0 ~$0.50 $1.10
Session limit 12 hours 24 hours Unlimited

The catch

  • 12-hour session limit — fine for experiments, not for training runs
  • Idle timeout — closes if you don't interact for ~90 minutes
  • Queue priority — free tier waits behind Pro users

For a one-off test or demo, it's unbeatable. For production training, you'll need paid.

I sketched the Colab notebook template with MonkeyCode — free, open-source, no cloud dependency: https://ly.cyberserval.tech/iIETXiF

What's the largest model you've successfully run on Colab's free tier?

Top comments (0)