Your compiled DSPy program re-sends up to 20 few-shot demos on every single call

DSPy's pitch is that you stop hand-writing prompts and let an optimizer compile them for you. You write a program out of modules, hand it a metric and a trainset, run a teleprompter, and it finds good few-shot examples for each step. It genuinely works. The part the tutorials don't put a number on is what "found good few-shot examples" costs you — not once, but on every call your compiled program makes in production.

Compiling attaches demos. Every call re-sends them.

When you compile with the default optimizer, DSPy bootstraps few-shot demonstrations and pins them onto each predictor:

# dspy/teleprompt/bootstrap.py
def __init__(self, ..., max_bootstrapped_demos=4, max_labeled_demos=16, ...):

That's up to 20 demos per predictor by default — 4 bootstrapped (full input→output traces, including the chain-of-thought rationale) plus up to 16 labeled examples. They live on the compiled program, not in any prompt string you wrote.

Then, on every inference, the module hands all of them to the adapter:

# dspy/predict/predict.py
demos = kwargs.pop("demos", self.demos)

self.demos is the full set the optimizer attached. There's no "use them for the first call only" — each forward() defaults to sending the whole list.

What "sending the demos" actually means in tokens

The adapter turns every demo into a pair of chat messages — a user turn and an assistant turn — and appends all of them ahead of your real input:

# dspy/adapters/base.py — format()
messages.append({"role": "system", "content": system_message})
messages.extend(self.format_demos(signature, demos))
...
# format_demos(), per complete demo:
messages.append({"role": "user", "content": self.format_user_message_content(signature, demo)})
messages.append({"role": "assistant", "content": ...})

So a predictor compiled with 12 demos prepends 24 messages to every call. Because bootstrapped demos carry the full reasoning trace, those messages aren't small. This is fixed overhead you pay on request #1 and request #1,000,000 alike — and it's invisible in your code, because you never wrote those messages. The optimizer did.

And a real program has more than one predictor

DSPy's whole point is composition: a pipeline is several modules — a couple of ChainOfThought steps, a retriever-reader, a router. Each is a predictor, each gets its own demo set, each re-sends it on every call. The per-call prompt overhead is roughly:

demos_per_predictor  ×  predictors  ×  (rationale is long, so each demo is not cheap)

Compile a 4-module pipeline with the defaults and you can be prepending 60–80 demo messages across the pipeline for a single end-user request — none of which appear anywhere in your source.

The knob is real — set it on purpose

This isn't a bug and it isn't a strawman: DSPy hands you the lever at compile time. Choose the demo budget instead of inheriting 4 + 16:

from dspy.teleprompt import BootstrapFewShot

optimizer = BootstrapFewShot(metric=my_metric,
                             max_bootstrapped_demos=2,
                             max_labeled_demos=2)
compiled = optimizer.compile(program, trainset=trainset)

And check what actually got attached before you ship it:

for p in compiled.predictors():
    print(len(p.demos))   # how many few-shot pairs ride along on every call

The point isn't "DSPy is expensive" — it's that the number of demos re-sent per call is a decision the optimizer makes for you, and the default is generous.

Measure it before you argue about it

Before you tune anything, put a dollar figure on one real run of the compiled program — priced, not guessed. That's what @wartzar-bee/tokenscope does (npm i @wartzar-bee/tokenscope): it takes real usage and prices each bucket — input, output, cache-write (~1.25×), cache-read (~0.1×) — into an actual per-run cost, so "the compiled pipeline costs N× the zero-shot one" stops being a hunch.

If it runs in CI, gate it: wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action (built on tokenscope) that fails the check when a run crosses an absolute max-usd ceiling — so a recompile that bumps the demo count doesn't ship as a silent 3× before anyone notices.

- uses: wartzar-bee/ci-guardrail@v1
  with:
    max-usd: "0.50"

If you run compiled DSPy programs: how many demos are on each predictor, and how long is each one? Worth pricing one real run before the next invoice does it for you.