In this video I continue the fine-tuning series on my channel. This time I go through the whole pipeline, not just the training part: renting GPUs on Runpod, fine-tuning a model with Unsloth Studio, deploying it as an inference endpoint using Runpod's Serverless, and then actually using that endpoint inside a Pydantic AI Agent.
This is meant to cover the full path: rent the GPU, train the model, deploy it, and get it hooked up to something that can actually call it.
Setting Up the Pod
I go in-depth on creating a Pod (a dedicated GPU instance container), picking a GPU, setting up storage, and getting SSH access working. The whole setup also works through the accompanying Jupyter Notebook, but I show the terminal option too.
One thing I didn't call out clearly enough in the video: run apt update && apt upgrade -y right after you SSH in, before installing anything else. A fresh Pod's package index is often out of date, so skipping this can mean installs failing or pulling older versions of tools than you'd expect.
Storage Options You Should Know About
Before you get into training, it's worth understanding how storage works on a Pod, because it's easy to get caught out. You've got 3 options:
- container disk — files disposed when pod stops
- volume disk — files persist until pod terminated
- network volume — files persist beyond Pod's termination
Volume disk is usually what's mounted on your /workspace directory.
A Gotcha With Unsloth Studio's Install Path
If you install Unsloth Studio without setting UNSLOTH_STUDIO_HOME first, like this:
export UNSLOTH_STUDIO_HOME=/workspace
Unsloth Studio will quietly ignore /workspace and fall back to /root/.unsloth/studio instead, which is outside the volume disk and sitting on the container disk. You won't notice anything's wrong until you stop the pod to save some money, come back the next day, and your training setup is gone. Ask me how I know!
Keeping Runpod Costs Down
I didn't go in-depth on optimizing Runpod usage in the video itself, but a couple of things are worth knowing:
- Runpod charges you for renting the pod and GPUs the whole time it's running, so it's worth preparing your dataset before you spin the pod up, rather than doing that work while the GPU meter's running.
- For Runpod's serverless endpoint, you can reduce cold start by keeping at least one GPU worker active, but that comes at a cost, so just be aware of the trade-off.
From Model to Agent
Once the model was trained and deployed, the last piece was actually using it. I set up a Pydantic AI agent that calls the fine-tuned model through the endpoint. I went with Pydantic AI's OpenAI provider for this, since vLLM (the inference framework running behind the endpoint) supports the OpenAI response format.
A Quick Note on Fine-Tuning vs. RAG
Fine-tuning is best when you're trying to modify the behavior of a model, like response style, tone etc. You can also teach it new knowledge, but even after fine-tuning, models can still hallucinate or fall back on old data. For anything that needs to stay accurate and up to date, pairing your fine-tuned model with Retrieval Augmented Generation (RAG) is still the safer route, since RAG pulls in fresh, sourced info at request time instead of relying on whatever got baked in during training.
If you enjoyed this, feel free to subscribe to the channel.
Thanks and happy coding!
Top comments (0)