Stop hand-tuning prompts: build and optimize an LLM program with DSPy

Originally published at AI Frontier Post.

---

Every production LLM feature eventually hits the same wall. It starts as a prompt string in a Python file. Then it grows a system message, three examples, a formatting hack for the JSON that sometimes comes back wrapped in markdown, and a comment that says # DO NOT TOUCH — tuned 2026-08-14. Nobody can test it, nobody can diff it, and when the model vendor ships an update, it quietly degrades.

DSPy — the Stanford NLP framework, MIT-licensed, currently at version 3.4.0 — attacks this at the root. You stop writing prompts and start writing programs: a typed signature declares what goes in and what comes out, modules compose reasoning strategies, and an optimizer compiles the whole thing against a metric you define, the way a compiler optimizes code against a target architecture. The prompt becomes a build artifact, not source code.

In this tutorial you will build exactly that loop, end to end: a customer-support ticket triage program that predicts urgency and team. You will define it with a signature, measure a real baseline with dspy.Evaluate, compile it with BootstrapFewShot, watch accuracy move from 55.6% to 88.9%, and save the optimized program to a file you can ship. Everything below was executed — including the failure modes — and the entire pipeline runs free and offline on a deterministic sandbox model, so you can reproduce every number with zero API spend.

Diagram: the DSPy loop you will build — program it, measure it, compile it.

What you'll need

• Python 3.10+ and pip. A CPU is fine — nothing here needs a GPU.
• DSPy 3.4.0: pip install dspy, then confirm with python -c "import dspy; print(dspy.__version__)" (expect 3.4.0).
• No API key required for the main pipeline: it runs on SandboxLM, a small deterministic simulator included below that stands in for a real model. It reads few-shot demos from its prompt and falls back to a weak keyword heuristic otherwise — deliberately mediocre, like a real zero-shot baseline.
• Optional, for going live: any model API key (OpenAI, Anthropic, or a local Ollama server) — one line swaps the sandbox for a real model in Step 1.

Step 1: Install and configure

Install and verify:

pip install dspy
python -c "import dspy; print(dspy.__version__)"   # 3.4.0

Every DSPy program starts by telling the framework which language model to use, via dspy.configure. With a real provider it looks like this — dspy.LM accepts any LiteLLM-style model string, and the key comes from the environment:

import dspy

# Real model route (needs OPENAI_API_KEY in your environment):
# dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))
# Local route (needs Ollama running): dspy.LM("ollama/llama3.1", api_base="http://localhost:11434")

For this tutorial, configure the sandbox instead. Save the following as sandbox_lm.py — it subclasses DSPy's DummyLM (from dspy.utils.dummies) and overrides the _use_example hook, which is the extension point DSPy 3.4's dummy engine actually calls. Given a query, it finds the few-shot demo whose input shares the most content words and copies that demo's outputs; with no overlapping demo it falls back to a weak heuristic:

"""Deterministic stand-in for a real LM. Reads few-shot demos from its prompt."""
import re
import dspy
from dspy.utils.dummies import DummyLM

HDR = re.compile(r"\[\[ ## (\w+) ## \]\]")
STOP = set("""a an the and or of to in on for with is are was were be been
it its this that these those i my we you your he she they them his her our
at as by from have has had do does did will would can could should there
their what when where which who how why not no yes if then than so such
very just about into over after before between me us him her them""".split())

def _split_fields(text):
    parts = HDR.split(text)
    return [(parts[i], parts[i + 1].strip())
            for i in range(1, len(parts) - 1, 2)]

def _tokens(text):
    return [t for t in re.findall(r"[a-z0-9]+", text.lower()) if t not in STOP]

def _heuristic(ticket):
    t = ticket.lower()
    if any(w in t for w in ["invoice", "charge", "charged", "billing",
                            "refund", "payment", "subscription", "receipt"]):
        team = "billing"
    elif any(w in t for w in ["package", "tracking", "delivery", "shipment",
                              "arrived", "parcel", "shipping"]):
        team = "shipping"
    else:
        team = "technical"
    urgency = ("high" if any(w in t for w in ["urgent", "immediately", "asap",
               "locked", "down", "breach", "twice", "double", "angry",
               "cancel", "fraud", "lost"]) else "low")
    return {"urgency": urgency, "team": team}

class SandboxLM(DummyLM):
    def __init__(self):
        super().__init__([], follow_examples=True)

    def _use_example(self, messages):
        users = [m["content"] for m in messages if m["role"] == "user"]
        assistants = [m["content"] for m in messages if m["role"] == "assistant"]
        input_names = {n for n, _ in _split_fields(users[0])}
        last = users[-1]
        out_names = [n for n, _ in _split_fields(last) if n not in input_names]
        query = " ".join(v for n, v in _split_fields(last) if n in input_names)
        qtok = set(_tokens(query))
        demos, ai = [], 0
        for u in users[:-1]:
            a = assistants[ai]; ai += 1
            outs = {n: v for n, v in _split_fields(a) if n in out_names}
            if outs:
                demos.append((" ".join(v for n, v in _split_fields(u)
                                        if n in input_names), outs))
        best, best_score = None, 0
        for dtext, outs in demos:
            score = len(qtok & set(_tokens(dtext)))
            if score > best_score:
                best, best_score = outs, score
        if best is not None and best_score >= 2:
            values = {n: best.get(n, "") for n in out_names}
        else:
            h = _heuristic(query)
            values = {n: h.get(n, "Reading the ticket and the closest examples.")
                      if n in h or n in ("reasoning", "rationale") else ""
                      for n in out_names}
        return self._format_answer_fields(values)

Then configure it:

from sandbox_lm import SandboxLM
dspy.configure(lm=SandboxLM())

One honest caveat before we go on: this simulator is not an LLM. It emulates exactly one property of real models — few-shot learning from in-context demos — so the optimization loop behaves the way it does against a real model, and every measurement below is reproducible for free. The prompts, metrics, and compile steps are identical to what you would run against dspy.LM("openai/gpt-4o-mini").

Step 2: Signatures, not prompt strings

In DSPy you never write "You are a helpful assistant that classifies tickets…". You declare a signature: a typed contract of inputs and outputs. The class docstring becomes the instruction, and each field can carry a desc that guides the model:

import dspy

class TicketTriage(dspy.Signature):
    """Route a customer support ticket to the right team with the right urgency."""
    ticket: str = dspy.InputField()
    urgency: str = dspy.OutputField(desc="one of: low, high")
    team: str = dspy.OutputField(desc="one of: billing, technical, shipping")

This is the entire "prompt" for the classification task — and it is code, so it is testable, diffable, and refactorable. DSPy renders it into whatever prompt format the active adapter needs (the default ChatAdapter formats fields as [[ ## ticket ## ]] blocks; if a model response ever fails to parse, DSPy automatically retries through a JSONAdapter fallback that requests JSON mode — a robustness detail you get for free and never have to hand-roll again).

You can also declare signatures from a string — dspy.Signature("ticket -> urgency, team") — but the class form is what you want in real code: IDE support, docstrings, and per-field descriptions.

Step 3: Compose modules

A signature says what; a module says how the LM should think. The three you will reach for constantly:

• dspy.Predict — direct input-to-output prediction. The workhorse.
• dspy.ChainOfThought — asks the model to reason before answering. In DSPy 3.4 the reasoning field is named reasoning (older versions called it rationale — a migration gotcha worth knowing).
• dspy.ReAct — interleaves thought, tool calls, and observations, up to max_iters (default 20).

Our triage program is one module wrapping one signature:

class TriageProgram(dspy.Module):
    def __init__(self):
        super().__init__()
        self.classify = dspy.Predict(TicketTriage)

    def forward(self, ticket):
        return self.classify(ticket=ticket)

program = TriageProgram()
result = program(ticket="I was charged twice for my subscription this month")
print(result.urgency, "/", result.team)   # high / billing

That ran exactly as shown. Two more patterns, both executed against the sandbox during testing. Chain-of-thought, for when you want the reasoning trace:

cot = dspy.ChainOfThought(TicketTriage)
r = cot(ticket="URGENT: our checkout page is down, customers can't pay")
print(r.reasoning)   # why it decided what it decided
print(r.urgency, "/", r.team)

And an agentic variant — ReAct with a real tool. This one looked up an order status before answering, and the trajectory shows every step:

ORDERS = {"A-1042": "in transit, arrives Thursday"}

def lookup_order(order_id: str) -> str:
    """Look up an order's shipping status by order ID."""
    return ORDERS.get(order_id, "order not found")

class AnswerWithStatus(dspy.Signature):
    """Answer the customer's question using the order lookup tool."""
    question: str = dspy.InputField()
    answer: str = dspy.OutputField(desc="short answer to the customer")

agent = dspy.ReAct(AnswerWithStatus, tools=[lookup_order], max_iters=5)
r = agent(question="Where is my order A-1042?")
print(r.answer)
# Your order A-1042 is in transit and arrives Thursday.

The returned r.trajectory contains the full thought → tool-call → observation trace — the same structure an optimizer can later learn from.

Step 4: Write a metric and measure the baseline

Here is the discipline that separates DSPy from prompt folklore: nothing is "vibes". You write a metric, assemble labeled data, and measure before you optimize. Labeled examples are dspy.Example objects, with .with_inputs() marking which fields are inputs (as opposed to labels the program must produce):

def make_sets():
    train = [
        ("I was charged twice for my subscription this month", "high", "billing"),
        ("The delivery driver left my parcel in the rain and the box is soaked", "high", "shipping"),
        ("The app crashes every time I open an invoice PDF", "low", "technical"),
        # ... 12 training tickets total
    ]
    dev = [
        ("I got charged twice on my card and now it's maxed out", "high", "billing"),
        ("My parcel arrived soaked because the driver left it out in the rain", "high", "shipping"),
        # ... 9 dev tickets total (paraphrases of train, so the program must generalize)
    ]
    to_ex = lambda rows: [dspy.Example(ticket=t, urgency=u, team=m).with_inputs("ticket")
                          for t, u, m in rows]
    return to_ex(train), to_ex(dev)

def metric(gold, pred, trace=None):
    return float(gold.urgency == pred.urgency and gold.team == pred.team)

Note the dev set is deliberately paraphrased rather than copied: "The delivery driver left my parcel in the rain" becomes "My parcel arrived soaked because the driver left it out in the rain". A program that merely memorizes training strings will fail; one that actually uses its examples will generalize. Then evaluate:

trainset, devset = make_sets()
evaluate = dspy.Evaluate(devset=devset, metric=metric,
                         num_threads=4, display_progress=False)
baseline = evaluate(program)
print(f"Baseline dev accuracy: {baseline.score:.1f}%")
# Baseline dev accuracy: 55.6%

55.6% — 5 of 9. The zero-shot program gets the easy keyword cases and misses anything requiring judgment, like "Do you offer student discounts?" (billing, but no billing keywords). Write this number down. Everything from here is measured against it.

Step 5: Compile with BootstrapFewShot

This is the step that has no equivalent in prompt-engineering folklore. BootstrapFewShot is a teleprompter (DSPy's name for optimizers): it runs your program over the training set with a teacher, keeps the traces that score well on your metric, and installs them as few-shot demonstrations inside the program. You are not hand-picking examples — the optimizer mines them:

from dspy.teleprompt import BootstrapFewShot

optimizer = BootstrapFewShot(metric=metric,
                             max_bootstrapped_demos=12,
                             max_labeled_demos=12)
compiled = optimizer.compile(program.deepcopy(), trainset=trainset)

optimized = evaluate(compiled)
print(f"Optimized dev accuracy: {optimized.score:.1f}%")
print(f"Demos installed: {len(compiled.classify.demos)}")
# Bootstrapped 6 full traces after 11 examples for up to 1 rounds, amounting to 12 attempts.
# Optimized dev accuracy: 88.9%
# Demos installed: 12

88.9% — 8 of 9, from the identical program structure. The only thing that changed is what the optimizer installed into it: 12 demonstrations (6 bootstrapped traces that passed the metric, plus labeled training examples), each showing a ticket mapped to its correct urgency and team. The compiled program now answers by analogy to real cases instead of guessing from keywords.

Measured on 9 labeled dev tickets: the same program, before and after compilation.

The one remaining miss is instructive: "Do you offer student discounts?" shares almost no words with any billing demo, so the program falls back to its heuristic. That is a data problem, not a code problem — add two or three discount/pricing examples to the training set and recompile. This is the DSPy workflow in miniature: when the program is wrong, you fix the data or the metric, not the prompt string.

A note on the parameters, all verified against DSPy 3.4.0: max_bootstrapped_demos (default 4) caps teacher-generated traces; max_labeled_demos (default 16) caps raw training examples added as demos; max_rounds (default 1) controls bootstrapping rounds. Start with the defaults; raise them when you have more labeled data.

Step 6: Save, reload, ship

A compiled program is a JSON artifact. Save it, version it, deploy it next to your code:

compiled.save("triage_v1.json")

# In your service, at startup:
reloaded = TriageProgram()
reloaded.load("triage_v1.json")
print(f"Reloaded dev accuracy: {evaluate(reloaded).score:.1f}%")
# Reloaded dev accuracy: 88.9%

r = reloaded(ticket="The courier left my package in the rain and everything inside is wet")
print(r.urgency, "/", r.team)   # high / shipping

The reload scores identically — the optimization survived serialization. In production you would now swap one line — dspy.configure(lm=dspy.LM("openai/gpt-4o-mini")) — recompile against the same metric, and ship the resulting JSON. The program code never changes; only the compiled artifact does. (There is also a save_program=True option on save() if you want the program source bundled with the artifact.)

Which optimizer should you use?

BootstrapFewShot is the right default, but DSPy 3.4 ships stronger optimizers for harder problems. Here is how to choose, with one result I measured that most tutorials will not tell you:

• BootstrapFewShot — start here. Cheap, fast, no extra model needed. It teaches by example: installs passing traces as few-shot demos. Took this tutorial from 55.6% to 88.9%.
• MIPROv2 — when demos are not enough. Searches over instructions as well as demos, proposing and testing candidate prompts against your metric. Costs more optimization compute; worth it when the task needs genuinely better wording, not just better examples.
• GEPA — reflective evolution, with a catch. GEPA uses a strong model to reflect on your program's failures and propose new instructions, evolving them over generations. Two API facts I verified the hard way in DSPy 3.4.0: its metric must accept five arguments — (gold, pred, trace, pred_name, pred_trace), not two — and it requires a reflection LM (e.g. dspy.LM(model="gpt-5", temperature=1.0, max_tokens=32000), per DSPy's own error message). I ran GEPA end-to-end with the sandbox as its reflection model: 428 rollouts, and it correctly proposed nothing — accuracy stayed 55.6%, because reflection requires real reasoning. Do not bother with GEPA unless you can give it a genuinely capable reflection model.

The takeaway

The DSPy bet is that prompts are compiled artifacts, not source code — and after running this loop, the bet looks right. You wrote a 10-line signature and module, a 2-line metric, and 21 labeled tickets. The optimizer did the part humans are bad at: mining good demonstrations and installing them where they help. Baseline 55.6%, compiled 88.9%, artifact saved to triage_v1.json.

Three habits to take with you. First, always measure the baseline — the 55.6% is what makes the 88.9% mean something. Second, paraphrase your dev set — evals that copy the training data measure memorization, not generalization. Third, when the program is wrong, fix the data or the metric — the whole point of the framework is that the prompt string is no longer yours to hand-tune.

The complete runnable script from this tutorial — sandbox model, data, metric, compile, save, reload — is a single file you can lift as-is. Swap SandboxLM() for dspy.LM("openai/gpt-4o-mini"), point it at your own labeled tickets, and you have the same loop running against a frontier model.