DEV Community: Blessed Josiah

Fine-Tune, Deploy and Use LLM As AI Agent

Blessed Josiah — Tue, 22 Sep 2026 08:10:00 +0000

In this video I continue the fine-tuning series on my channel. This time I go through the whole pipeline, not just the training part: renting GPUs on Runpod, fine-tuning a model with Unsloth Studio, deploying it as an inference endpoint using Runpod's Serverless, and then actually using that endpoint inside a Pydantic AI Agent.

This is meant to cover the full path: rent the GPU, train the model, deploy it, and get it hooked up to something that can actually call it.

Setting Up the Pod

I go in-depth on creating a Pod (a dedicated GPU instance container), picking a GPU, setting up storage, and getting SSH access working. The whole setup also works through the accompanying Jupyter Notebook, but I show the terminal option too.

One thing I didn't call out clearly enough in the video: run apt update && apt upgrade -y right after you SSH in, before installing anything else. A fresh Pod's package index is often out of date, so skipping this can mean installs failing or pulling older versions of tools than you'd expect.

Storage Options You Should Know About

Before you get into training, it's worth understanding how storage works on a Pod, because it's easy to get caught out. You've got 3 options:

container disk — files disposed when pod stops
volume disk — files persist until pod terminated
network volume — files persist beyond Pod's termination

Volume disk is usually what's mounted on your /workspace directory.

A Gotcha With Unsloth Studio's Install Path

If you install Unsloth Studio from Jupyter Notebook in the /workspace directory without setting UNSLOTH_STUDIO_HOME first, like this:

export UNSLOTH_STUDIO_HOME=/workspace

Unsloth Studio will quietly ignore /workspace and fall back to /root/.unsloth/studio instead, which is outside the volume disk and sitting on the container disk. You won't notice anything's wrong until you stop the pod to save some money, come back the next day, and your training setup is gone. Ask me how I know!

Keeping Runpod Costs Down

I didn't go in-depth on optimizing Runpod usage in the video itself, but a couple of things are worth knowing:

Runpod charges you for renting the pod and GPUs the whole time it's running, so it's worth preparing your dataset before you spin the pod up, rather than doing that work while the GPU meter's running.
For Runpod's serverless endpoint, you can reduce cold start by keeping at least one GPU worker active, but that comes at a cost, so just be aware of the trade-off.

From Model to Agent

Once the model was trained and deployed, the last piece was actually using it. I set up a Pydantic AI agent that calls the fine-tuned model through the endpoint. I went with Pydantic AI's OpenAI provider for this, since vLLM (the inference framework running behind the endpoint) supports the OpenAI response format.

A Quick Note on Fine-Tuning vs. RAG

Fine-tuning is best when you're trying to modify the behavior of a model, like response style, tone etc. You can also teach it new knowledge, but even after fine-tuning, models can still hallucinate or fall back on old data. For anything that needs to stay accurate and up to date, pairing your fine-tuned model with Retrieval Augmented Generation (RAG) is still the safer route, since RAG pulls in fresh, sourced info at request time instead of relying on whatever got baked in during training.

If you enjoyed this, feel free to subscribe to the channel.

Thanks and happy coding!

How to Fine-Tune an LLM Locally with Unsloth Studio

Blessed Josiah — Mon, 14 Sep 2026 08:00:00 +0000

Fine-tuning is the process of further training a language model on new data so it learns new information, behaviors, or style — updating the model's own weights, rather than just showing it information at the moment you ask a question.

There are two common ways to get an LLM to work with new information: RAG and fine-tuning. RAG (Retrieval-Augmented Generation) retrieves relevant information and feeds it to the model as context at the time of the query. Fine-tuning instead trains the model on new examples so that knowledge becomes part of the model itself.

In this video, I start a new series on fine-tuning, where I go over everything you need to know to fine-tune a model using Unsloth Studio.

Prepare Dataset

First, we need to source our data. For this video, I manually created a PDF with facts about the just-concluded 2026 FIFA World Cup. You could also use the Wikipedia page, but be mindful of noise in that data — you'll need to manually clean it so you don't end up training your model on noise instead of facts.

After cleaning your PDF, you can head to Claude or ChatGPT and ask for a multi-turn conversational dataset in the ChatML format:

[
  {
    "messages": [
      {"role": "system", "content": "..."},
      {"role": "user", "content": "..."},
      {"role": "assistant", "content": "..."}
    ]
  },
  {
    "messages": [
      {"role": "system", "content": "..."},
      {"role": "user", "content": "..."},
      {"role": "assistant", "content": "..."}
    ]
  }
]

Or take advantage of Unsloth Studio's Data Recipes and generate a ChatML dataset right from the application. I explain how to do this in depth in the video — the prompt and response schema I used are below.

Response Schema (JSON Array)

{
  "type": "array",
  "minItems": 5,
  "items": {
    "type": "object",
    "properties": {
      "role": {
        "type": "string",
        "enum": ["system", "user", "assistant"]
      },
      "content": {
        "type": "string"
      }
    },
    "required": ["role", "content"]
  }
}

Prompt

Generate a conversation from the chunk below: a system message, then exactly two user/assistant exchanges.

Start with a system message setting the assistant's role as a helpful assistant knowledgeable about the 2026 FIFA World Cup. Then the user asks a factual question about something in the chunk, and the assistant answers accurately. The user then asks a follow-up question that builds on the assistant's answer — asking for more detail, a related fact, or clarification — and the assistant answers that too.

Rules:
- Never invent facts not in the chunk. If the chunk is too narrow for a genuine follow-up, find a second distinct fact from the same chunk to ask about instead — do not fabricate details to fill the follow-up.
- Answers must be self-contained (no "the text says...").
- Use full names, not pronouns.
- After the system message, roles must alternate: user, assistant, user, assistant.

Output only a JSON array matching this schema — no wrapper object, no extra text:
[{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]

Chunk:
'{{chunk_text}}'

I used the unsloth/Qwen2.5-Coder-3B-Instruct-GGUF model for this step, since it reliably returned a valid JSON array. I tried the unsloth/gemma-4-E2B-it-GGUF model first, but it couldn't handle the JSON array output correctly.

Base Model

I used the unsloth/Llama-3.1-8B-Instruct model for two reasons:

It needs about 8.6GB of GPU memory, which fits comfortably within my machine's 32GB of unified memory (which serves as VRAM on Apple Silicon).
It's built for conversational chat, since it ships with a chat template out of the box.

I initially tried the base version, unsloth/Llama-3.1-8B, and ran into this error:

This page from Unsloth's documentation explains why: base models don't come with a chat template at all, while Instruct models do.

You can still fine-tune a base model for conversational chat, but you'll need to manually create a chat template for it first. Unsloth Studio doesn't yet support this out of the box — there's an open GitHub issue tracking it as a feature request, so hopefully it lands soon.

Hyperparameters

For hyperparameter configuration, check Unsloth's LoRA Hyperparameters Guide for guidance on getting optimal values for your training run.

One of the most important settings is eval_steps. Without it, evaluation never runs alongside training. I set mine to 0.1 in the video, meaning evaluation runs every 10% of the way through training. This gives you an eval loss graph, which shows how your model is actually performing — not just how well it fits its own training data.

Here's how to read it:

If the graph stays flat and never comes down, that's underfitting — the model isn't learning the patterns in your dataset.
If it comes down and then goes back up, that's overfitting — the model has stopped generalizing and started memorizing the training data instead.
What you want is the eval loss graph trending smoothly downward, alongside the training loss graph.

If you found this useful, drop the video a like and subscribe to the channel for more content.

Thanks, happy coding!

How To Build A Production-Ready Design System With Claude Code, Mobbin, And Penpot

Blessed Josiah — Mon, 31 Aug 2026 08:00:00 +0000

I started a new series on my channel on Claude Code to help you understand and use it confidently, and in this video, I show you how to build a production-ready design system.

The workflow uses Mobbin, a platform with a huge library of real application screenshots, to research how real, shipped products solve the same design problems. Claude Code takes those references and connects to Penpot — a free, open-source design platform you can self-host — to turn that research into an actual, working design system.

Using a collection of skills I built for my workflow, the /design skill returns two files when invoked: DESIGN.md and tokens.json. Claude Code derives these tokens from its research, then uses them to build consistent, traceable designs directly in Penpot.

The Mobbin MCP server does require a paid plan. If you'd rather skip that, you can point Claude Code at your own collection of reference screenshots instead, and still have it connect to Penpot to build the designs from there.

If you need a quick brush-up on Claude Code itself, check out my previous video in the Mastering Claude Code Series.

While you're there, consider subscribing to the channel

How To Evaluate An AI Agent

Blessed Josiah — Tue, 04 Aug 2026 08:00:00 +0000

In classical test-driven development, we deal with deterministic outcomes. We write assertions against values and types we already know in advance — assert result == expected, and we move on.

AI agents don't play by those rules. Because the underlying LLM is non-deterministic, giving it the same input twice can produce two differently-worded outputs. That alone breaks the classical assertion model.

But there's a second problem, beyond just consistency. Say we have a customer support agent, and we want it to be helpful, empathetic, professional, and friendly when it talks to customers. Even if the output were consistent, how do we test for qualities like that? There's no fixed value or type to assert against — "empathetic" isn't a type or an exact string, it's a subjective judgment call.

So we're dealing with two distinct problems: unpredictable output, and qualities that are inherently subjective. Testing an AI agent means solving for both.

Enter Pydantic Evals

Pydantic Evals is Pydantic AI's recommended way to test the output of your AI agents. It's an evaluation library that gives you a set of evaluators purpose-built for exactly this kind of testing — some for the deterministic parts of your agent's behavior, and some for the parts that require judgment.

Testing Deterministic Behavior: IsInstance

Not everything about an AI agent is unpredictable. If your agent is set up with a structured output type, you can — and should — still test that the shape of the response is correct, even if the content varies.

For example, say we have an agent that returns a structured output type:

from pydantic_ai import Agent
from dataclasses import dataclass

# Response Model
@dataclass
class Response:
    message: str
    sentiment: str

# Agent Setup
support_agent = Agent(
    model="openai/gpt-5.5",
    name="customer_support_agent",
    instructions="""
        You are a customer support agent for an online store.
    """,
    output_type=Response
)

We can define a test case that checks the output is an instance of that Response type, using the IsInstance evaluator:

from pydantic_evals import Case
from pydantic_evals.evaluators import IsInstance

test_case = Case(
    name="check_agent_response",
    inputs="where is my order?",
    evaluators=[
        IsInstance(type_name='Response')
    ]
)

This is a classic assertion in spirit — we know exactly what type we expect back, so we test for it directly. No judgment call required.

Testing Non-Deterministic Behavior: LLMJudge

Structural checks like IsInstance don't help us with the harder problem: was the response actually good? Was it empathetic? Professional? Did it address the customer's question?

This is where LLMJudge comes in. It uses an LLM as a judge to evaluate your agent's output against a set of criteria you define — scoring qualitative aspects of the response that a type check could never catch.

from pydantic_evals.evaluators import LLMJudge

quality_case = Case(
    name="check_response_quality",
    inputs="where is my order?",
    evaluators=[
        LLMJudge(
            rubric="Response should be empathetic, professional, and directly address the customer's question about their order status."
        )
    ]
)

Under the hood, LLMJudge sends your agent's output and your rubric to a judge model, which returns a score (and often a reason) reflecting how well the response meets your criteria. Rather than asserting on an exact value, you're asserting on a standard.

Together, IsInstance and LLMJudge cover both problems from earlier: structural correctness for the deterministic parts, and rubric-based scoring for everything else.

Watch the Full Walkthrough

I go deeper into LLMJudge — including setting up rubrics, reading scores, and combining multiple evaluators in a single test suite — in this video, part of the Master Pydantic AI series on my YouTube channel.

If you find it useful, I'd really appreciate a like and a subscribe on the channel.

Build Durable Agents With Pydantic AI And Temporal

Blessed Josiah — Sat, 18 Jul 2026 11:00:00 +0000

Agentic applications can fail mid-run: a network call drops, a server process crashes, an LLM request times out. Without durability built in, these failures mean lost progress and repeated (wasted) LLM calls when the process has to start over.

Pydantic AI provides a native integration with Temporal, a durable execution engine. Temporal records each step of a workflow's execution as an event history, so if a process crashes, it can resume from where it left off instead of starting over.

Pydantic AI exposes this through the TemporalAgent class, which wraps an existing agent and converts its tool calls, MCP calls, and model requests into Temporal Activities.

from pydantic_ai import Agent
from pydantic_ai.durable_exec.temporal import TemporalAgent

planning_agent = Agent(
    model='anthropic:claude-fable-5',
)
temporal_agent = TemporalAgent(
    wrapped=planning_agent
)

You call .run(...) on temporal_agent the same way you would on the underlying agent. The difference is under the hood: each tool call, MCP call, or model request now runs as a Temporal Activity.

Three Core Concepts

Activity — a function that performs a unit of work: an LLM call, an API request, a file write. If it fails, Temporal can retry it based on a configured retry policy.

from temporalio import activity

@activity.defn
async def create_sandbox(user_id: str) -> str:
    sandbox = await AsyncSandbox.create(template="node-expo-builder")
    return sandbox.sandbox_id

Workflow — orchestrates Activities, defining what runs and in what order. Workflow code must be deterministic (no direct network calls, no random values, no reading the system clock), since Temporal replays it to reconstruct state.

from temporalio import workflow
from datetime import timedelta

@workflow.defn
class ExampleWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        sandbox_id = await workflow.execute_activity(
            create_sandbox,
            args=[self.user_id],
            start_to_close_timeout=timedelta(minutes=2),
        )
        return sandbox_id

Worker — polls a task queue for scheduled work and executes it. A Workflow or Activity must be registered with a Worker before it can run.

import asyncio
from temporalio.client import Client
from temporalio.worker import Worker

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(
        client,
        task_queue="coding-agent",
        workflows=[ExampleWorkflow],
        activities=[create_sandbox],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())

The task queue name is the only link between the code that starts a Workflow and the Worker that executes it — they never call each other directly.

Workers run Activities and Workflows differently: Activities run their function body when scheduled, and may be retried on failure per their retry policy. Workflows are replayed against their event history each time they need to advance, which is what lets a Workflow resume correctly on a new Worker after a crash.

Signal and Query

Once a Workflow is running, Signal and Query let you interact with it from outside.

Signal sends data into a running Workflow, asynchronously, with no return value:

@workflow.signal
async def approve_plan(self) -> None:
    self._plan_approved = True

@workflow.signal
async def reject_plan(self, feedback: str = "") -> None:
    self._plan_approved = False
    self._plan_feedback = feedback

Signals only work while a Workflow is still running — once it's completed, failed, or been terminated, it can no longer receive one.

Query reads a Workflow's current state, synchronously and without side effects:

@workflow.query
def state(self) -> WorkflowState:
    return WorkflowState(
        status=self._status,
        project_name=self._project_name,
        preview_url=self._preview_url,
    )

Queries can also run against a completed Workflow, within your namespace's retention period. This doesn't extend to terminated Workflows, which aren't safely queryable.

Human In The Loop

Because Temporal keeps a full event history, a Workflow can pause and resume without losing state — a pattern suited to human approval steps.

from temporalio import workflow

@workflow.defn
class ExampleWorkflow:
    @workflow.run
    async def run(self):
        while True:
            self._plan_approved = None
            await workflow.wait_condition(lambda: self._plan_approved is not None)

            if self._plan_approved:
                break

workflow.wait_condition pauses execution until the given condition is true. Here, the Workflow waits until a Signal (like approve_plan above) sets self._plan_approved. If the process running this Workflow crashes while it's paused, a new Worker can pick it up and it resumes waiting at the same point.

This example has no timeout, so it waits indefinitely. In production, you'd typically pass a timeout to wait_condition and handle the case where no decision ever comes.

Resources

Watch Part 4 of this series: https://youtu.be/J0_GeI8Srzc
Full code for this example: https://github.com/bjoxiah/pydantic-ai-series/tree/agent-workflow

This is part of an ongoing series on my channel, Mastering Pydantic AI.

Building Modular AI Agent Features with Pydantic AI Capabilities

Blessed Josiah — Sun, 14 Jun 2026 08:30:00 +0000

If you're building AI Agents with Pydantic AI, understanding Capabilities is invaluable - it's the recommended way to add modular, reusable features to your agents.

This tutorial is part of my ongoing Pydantic AI series on YouTube, where I build a full no-code AI agent platform from scratch.

What is a Capability?

A capability in Pydantic AI is a modular unit of behavior that can be passed to an AI agent.

A capability can give your agent:

Custom toolsets
Instructions
Model settings
Lifecycle hooks

Think of it as a plug-and-play feature module - build it once, attach it to any agent.

How to Create a Capability

Capabilities are created using the Capability or AbstractCapability class:

from pydantic_ai.capabilities import AbstractCapability

Using AbstractCapability gives you full control over instructions, tools, and behavior. It's Pydantic AI's recommended pattern if you're building a library or platform on top of the framework.

Example: A Research Capability

In this tutorial, I build a Research Capability powered by the Tavily Search API, and an Email Capability powered by Resend.

Here's the research capability:

from pydantic_ai import FunctionToolset
from pydantic_ai.capabilities import AbstractCapability
from dataclasses import dataclass
from pydantic_ai.common_tools.tavily import tavily_search_tool
from settings import settings


@dataclass
class ResearchCapability(AbstractCapability):
    def get_instructions(self):
        return "You can use the Tavily search tool for research"

    def get_toolset(self):
        toolset = FunctionToolset()
        toolset.add_tool(tavily_search_tool(api_key=settings.tavily_api_key))
        return toolset

That's it - get_instructions() tells the agent what it can do, and get_toolset() gives it the tools to do it. Attach this to any agent, and it instantly gains web research abilities.

Going Further: RAG + GraphRAG

I also built a Company Knowledge Capability that combines:

pgvector for semantic search over your own data
Neo4j + Graphiti for knowledge graph retrieval (GraphRAG)

This lets an agent answer questions from your company's documents, website content, or any knowledge base with both vector search and relationship-aware graph queries.

Watch the Full Tutorial

This post covers the concept — the full video walks through building both capabilities live, plus the RAG/GraphRAG ingestion pipeline, observability with Logfire, and wiring it all into a working agent.

🎥 Watch on YouTube →
💻 Full source code →

While you're there, subscribe for more software and AI related content!