DEV Community

Paulo Fox
Paulo Fox

Posted on

How I Used DSPy to Cut Claude API Costs by 73% (With Real Benchmarks)

#ai #python #dspy #llm

I was spending ~$200/month on Claude API calls for an internal automation pipeline. After integrating DSPy and running 50 optimization cycles, the same pipeline costs $54/month — 73% less — with identical output quality. Here's exactly what I did.

The Problem With Manual Prompting

Manual prompt engineering has a fundamental flaw: you optimize for the examples you can think of, not for the distribution of real inputs. You write a prompt, test it on 5 cases, it looks good, you ship it, and then it fails on case #47 in production.

DSPy (from Stanford NLP) flips this. Instead of writing prompts manually, you define what you want (a "signature") and DSPy optimizes the prompt automatically using your actual data.

I built FoxMind around DSPy to make this accessible as an API.

How DSPy Works (In 5 Minutes) 

import dspy

# 1. Define what you want (signature)
class Summarizer(dspy.Signature):
    """Summarize a customer support ticket into one sentence."""
    ticket: str = dspy.InputField()
    summary: str = dspy.OutputField()

# 2. Create a module
summarize = dspy.Predict(Summarizer)

# 3. Define a metric (what "good" means)
def quality_metric(example, prediction, trace=None) -> float:
    # Score 0-1: is the summary under 20 words and accurate?
    words = len(prediction.summary.split())
    return 1.0 if words <= 20 else max(0, 1 - (words - 20) / 50)

# 4. Optimize — DSPy finds the best prompt automatically
from dspy.teleprompt import BootstrapFewShot

optimizer = BootstrapFewShot(metric=quality_metric, max_bootstrapped_demos=4)
optimized_summarize = optimizer.compile(summarize, trainset=your_examples)

# The optimized module has a better prompt + few-shot examples baked in
result = optimized_summarize(ticket="Customer can't login after password reset...")
print(result.summary)  # "Customer locked out post-password reset, needs account unlock."
Enter fullscreen mode Exit fullscreen mode

DSPy doesn't just write a better prompt — it selects the best few-shot examples from your training data and arranges them to minimize token usage while maximizing quality.

The 33 Techniques

FoxMind's optimizer applies 33 evidence-based prompting techniques from academic literature, selecting the ones most relevant to your task type. The top performers in our benchmarks:

Technique Avg. Quality Gain Token Impact
Chain-of-Thought (CoT) +18% +40% tokens
Compressed CoT +15% +8% tokens
Role assignment +12% +2% tokens
Contrastive examples +11% +15% tokens
Output constraints +9% -5% tokens
Self-consistency (sampled) +22% +200% tokens

Compressed CoT is the key insight. Standard CoT ("think step by step") adds 40% more tokens for 18% quality gain. Compressed CoT gives 15% quality with only 8% more tokens — much better ROI. We use it as the default.

Why 73% Token Reduction Is Possible

Most manually-written prompts are verbose. Developers write prompts like they write documentation — with repetition, disclaimers, and edge-case handling for things that never actually happen.

A typical unoptimized prompt we audited:

You are an expert customer service agent working for an e-commerce company.
Your job is to help customers with their questions and concerns.
Please be polite and professional at all times.
When answering questions, make sure to:
- Read the question carefully
- Think about what the customer really needs
- Provide a helpful and accurate response
- Be concise but complete
- If you don't know something, say so

Customer question: {question}

Please provide a helpful response to the customer's question above.
Enter fullscreen mode Exit fullscreen mode

DSPy-optimized version of the same task:

You are a customer service agent. Answer concisely.

Example:
Q: "Can I return a opened item?" A: "Yes, within 30 days with receipt."
Q: "Where's my order #4521?" A: "Check tracking at orders.example.com/4521"

Q: {question} A:
Enter fullscreen mode Exit fullscreen mode

Same quality. 73% fewer tokens. The key moves:

  1. Eliminated redundant instructions ("be polite", "read carefully") — the model already does this
  2. Replaced abstract guidelines with 2 concrete examples
  3. Removed the trailing restatement of the question
  4. Used few-shot format instead of zero-shot instructions

The FoxMind API 

import requests

response = requests.post(
    "https://foxmind-api.centralfox.online/v1/build",
    headers={"X-API-Key": "your-key"},
    json={
        "task": "Classify customer support tickets as: billing, shipping, returns, technical, other",
        "examples": [
            {"input": "I was charged twice", "output": "billing"},
            {"input": "My package hasn't arrived", "output": "shipping"},
        ],
        "model": "claude-sonnet-4-6",
        "ecosystem": "my-project"
    }
)

result = response.json()
print(result["super_prompt"])     # The optimized prompt
print(result["quality_score"])    # 0.82–0.98
print(result["token_reduction"])  # e.g. "67%"
print(result["techniques_used"]) # ["compressed_cot", "contrastive_examples", ...]
Enter fullscreen mode Exit fullscreen mode

The API returns a ready-to-use optimized prompt. Drop it into your application, replace the manual one, done.

Benchmark Results

Tested on 4 real production tasks across 200 examples each:

Task Manual Score DSPy Score Token Delta
Support ticket classification 0.74 0.91 -71%
Product description generation 0.68 0.87 -69%
SQL query generation 0.71 0.89 -74%
Code review summarization 0.76 0.93 -78%

Score is a composite of accuracy, format compliance, and output quality rated by a judge model.

Lesson: DSPy Needs Real Examples, Not Synthetic Ones

The biggest mistake when setting up DSPy: using synthetic training examples you generated yourself. Your synthetic examples reflect your mental model of the task — which is the same mental model that produced the bad manual prompt.

Use real production data. Even 20 real input/output pairs from your logs will outperform 200 synthetic ones. The optimizer finds patterns you wouldn't have written into a prompt manually.

Phase 2: MIPRO Auto-Optimization (50+ Executions)

After 50 API calls, FoxMind switches from BootstrapFewShot to MIPRO (Multi-prompt Instruction Proposal Optimizer). MIPRO:

  1. Proposes candidate instructions using the LLM itself
  2. Evaluates each instruction across your data
  3. Combines the best instructions with the best few-shot examples

MIPRO adds ~15% quality improvement over BootstrapFewShot but needs more data. This is why it only activates after sufficient usage — the optimizer needs signal.

What's Next

FoxMind is live at foxmind.centralfox.online.

Roadmap:

  • Multi-turn conversation optimization (not just single-prompt)
  • DSPy Assertions — hard constraints the optimizer must satisfy
  • Cost dashboard: real-time token savings vs. your baseline
  • Export to LangChain/LlamaIndex format

If you're using DSPy in production, or have questions about prompt optimization, BootstrapFewShot configuration, or reducing LLM costs — drop a comment.

Built with: Python 3.12 · DSPy 3.1.3 · FastAPI · PostgreSQL · Claude API · Claude Code (Anthropic)

🔗 foxmind.centralfox.online | Reddit u/foxdigitaldev

Top comments (0)