Prompt Engineering is Dead. Long Live DSPy: How to Program LLMs Instead of Prompting Them

For the past few years, building AI-powered applications has felt less like software engineering and more like digital alchemy. We’ve all been there: sitting in front of a playground or a code editor, meticulously tweaking a system prompt, adding "please think step-by-step," or begging the model to "take a deep breath" and format its output as valid JSON.

We called this "prompt engineering." But let’s be honest with ourselves: it isn't engineering. It’s an artisan craft. It’s the equivalent of a master clockmaker hand-filing gears. Each interaction is polished by human intuition, and the final behavior of the AI agent is a delicate sculpture formed by hours of trial and error.

This approach is fundamentally broken. It is fragile, opaque, and completely non-transferable.

If you want to build AI systems that can scale, adapt, and self-improve—systems like the self-evolving Hermes Agent—you must abandon manual prompt engineering. It is time to move from artisan craft to systematic engineering. This is where DSPy (Declarative Self-improving Language Programs, from Stanford NLP) enters the stage.

DSPy replaces fragile natural-language prompts with programmatic, optimizable modules that can be automatically tuned through closed-loop learning. In this post, we’ll explore why thinking of AI tasks as programs with typed signatures is a paradigm shift—one that mirrors the transition from hand-written assembly to high-level compilers in the history of computer science.

The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the ebook Hermes Agent, The Self-Evolving AI Workforce: details link, check also the 9 volumes TypeScript AI & Agentic Engineer Masterclass that includes also a volume on Jev.

---

The Three Walls of Manual Prompting

To understand why DSPy is necessary, we must first diagnose the disease it cures. Manual prompt engineering suffers from three fundamental limitations that act as brick walls for production-grade AI agents:

1. Fragility: A single-word change in a 500-word prompt can cause an entire agent pipeline to collapse. You update your system prompt to fix a minor formatting issue, and suddenly the model starts hallucinating or refusing to perform a completely unrelated task.
2. Opacity: The reasoning behind why a prompt succeeds or fails is buried deep within the LLM’s black box. When an agent fails, developers are left guessing at root causes, leading to a cycle of "voodoo debugging" where prompts are modified based on superstition rather than data.
3. Non-Transferability: A prompt meticulously optimized for GPT-4 often performs poorly on Claude 3.5 Sonnet, and completely falls apart on an open-source model like LLaMA 3. If you switch models, you have to throw away your prompts and start the trial-and-error process all over again.

These limitations prevent AI agents from truly learning and evolving over time. To build an agent that grows with you, we need a system where prompts are treated as variables that can be compiled, optimized, and validated automatically.

---

From Assembly to High-Level Compilers: A History Lesson

The transition we are currently experiencing in AI history is not new. It is the exact same transition software engineering underwent decades ago: the shift from assembly language to high-level compilers.

In the early days of computing, programmers wrote assembly code. Every instruction was hand-coded for a specific CPU architecture. The programmer had absolute control over registers and memory addresses, but the code was incredibly fragile. A single typo in a memory address would crash the entire machine. Porting a program from one processor to another meant rewriting it from scratch.

Then came high-level languages like Fortran and C, along with compilers.

[ Assembly Era ]  --> Hand-coded instructions for specific hardware (Fragile, Non-portable)
[ Compiler Era ]  --> High-level code + Compiler maps to hardware instructions (Robust, Portable)

Instead of managing registers, programmers defined abstract logic using variables and data types. The compiler took care of the dirty work, automatically mapping the abstract code to efficient machine instructions optimized for the target hardware.

In the world of AI, prompts are the new assembly language. You are writing low-level, model-specific instructions.

DSPy acts as the high-level compiler. Instead of writing concrete prompt strings, you write clean, abstract Python code defining the flow of data. You define your inputs and outputs, and let the DSPy compiler translate that abstract program into the optimal prompt or fine-tuning instructions for whatever LLM you happen to be using today.

---

The Core Pillars of DSPy Theory

To understand how DSPy enables self-evolving systems, we must dissect its three foundational concepts: typed signatures, optimizable modules, and the compiler.

1. Typed Signatures: The Data Type System of AI Programs

In traditional software engineering, a data type is a classification that specifies what kind of value a variable holds, determining what operations can be performed on it. In DSPy, typed signatures serve as the data type system for AI modules.

A typed signature is a declarative string or Python class of the form input_fields -> output_fields. It enforces a strict contract between your program and the LLM.

For example, a signature might look like this:
"document: str, max_words: int -> summary: str"

This is not syntactic sugar. This signature serves multiple critical roles:

• Contract Enforcement: The signature declares exactly what the module expects and produces. The DSPy runtime can automatically build validation functions to check these types at runtime.
• Automatic Data Generation: Given a signature, DSPy can generate synthetic training data by sampling from the input distribution and using a teacher model to produce target outputs. This is crucial for agents that need to learn new skills but lack real-world training data.
• Composability: Signatures allow modules to be chained together like lego blocks. A FileSearch module (query: str -> file_path: str) can be seamlessly piped into a ReadFile module (file_path: str -> content: str) to build a robust pipeline.

2. Optimizable Modules: Prompts as Variables

A DSPy module is a Python class that inherits from dspy.Module. It encapsulates one or more predictors (such as dspy.Predict, dspy.ChainOfThought, or dspy.ReAct).

The key theoretical insight here is that each predictor has internal parameters that can be optimized. These parameters include:

• The instruction text (the prompt given to the LLM)
• The few-shot examples (the in-context exemplars)
• Inference hyper-parameters (temperature, top-p, stop tokens)

In traditional prompting, these parameters are hardcoded. In DSPy, they are variables—named storage locations whose values can be changed. The optimizer (the DSPy compiler) treats these variables as a search space, mutating them to find the configuration that yields the highest performance.

3. The DSPy Compiler: The Meta-Learning Engine

The compiler is the heart of DSPy. It does not translate high-level code to binary; instead, it is a meta-learning algorithm that learns how to prompt an LLM for a given task.

The compilation process runs in an iterative loop:

[ Current Module ] 
       │
       ▼
[ Evaluate on Metric ] ──> Low Score? ──> [ Generate Candidate Mutations ]
       │                                                │
       ▼                                                ▼
[ Keep Best Variant ] <─── High Score? <─── [ Score Candidates ]

1. Evaluate the current module on a validation dataset using a specific metric.
2. Generate candidates by perturbing parameters (using LLM-based prompt proposals, selecting different few-shot examples, or adjusting hyper-parameters).
3. Score each candidate against the metric.
4. Select the best-performing candidate to become the new baseline.
5. Repeat until the optimization budget is exhausted or performance converges.

This process allows the system to learn how to solve tasks without updating the underlying model's weights. It treats the LLM as a black box and optimizes the interface, making the optimization process incredibly cost-effective—often costing only a few dollars in API calls.

---

Code Walkthrough: From Fragile Prompt to DSPy Module

Let’s look at a concrete example. Imagine we are building a code review agent.

The Traditional, Fragile Approach

In a traditional pipeline, you might write a prompt like this:

# Traditional, fragile prompt-based approach
def review_code(code: str) -> str:
    system_prompt = (
        "You are an expert software engineer. Analyze the following code "
        "and provide constructive feedback. Focus on security, performance, "
        "and readability. Format your output as a bulleted list. "
        "Do not include any introductory or concluding remarks."
    )

    # Call the LLM API directly
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": f"Code to review:\n{code}"}
        ]
    )
    return response.choices[0].message.content

This looks fine, but what happens if you switch to an open-source model like LLaMA-3-8B? It might completely ignore the instruction to "not include introductory remarks," returning a conversational greeting that breaks your downstream parser.

The DSPy Programmatic Approach

Now, let’s rewrite this using DSPy. We start by defining our typed signature and encapsulating it within an optimizable module:

import dspy

# Step 1: Define the signature (the contract)
class CodeReviewSignature(dspy.Signature):
    """Analyze the given code and provide feedback on security, performance, and readability."""
    code: str = dspy.InputField(desc="The source code to be reviewed")
    feedback: str = dspy.OutputField(desc="Constructive, bulleted feedback focusing on security, performance, and readability")

# Step 2: Define the module
class CodeReviewer(dspy.Module):
    def __init__(self):
        super().__init__()
        # We use ChainOfThought to force the model to reason before outputting feedback
        self.reviewer = dspy.ChainOfThought(CodeReviewSignature)

    def forward(self, code: str) -> dspy.Prediction:
        # The forward pass executes the predictor
        return self.reviewer(code=code)

Notice what is missing here: there are no prompt strings. We haven't told the model how to behave; we have simply declared the structure of the input and output, and selected a reasoning pattern (ChainOfThought).

Compiling the Module

To make this module truly robust, we can compile it. We provide a few examples of code and desired feedback, define a validation metric, and run the compiler:

from dspy.teleprompt import BootstrapFewShot

# Small dataset of examples (inputs and expected outputs)
trainset = [
    dspy.Example(
        code="def add(a, b): return a + b", 
        feedback="- Code is clean and simple.\n- Consider adding type hints for clarity: `def add(a: int, b: int) -> int`."
    ).with_inputs('code'),
    dspy.Example(
        code="import os\ndef run_cmd(cmd):\n    os.system(cmd)", 
        feedback="- CRITICAL SECURITY RISK: `os.system` is vulnerable to shell injection.\n- Use the `subprocess` module with `shell=False` instead."
    ).with_inputs('code')
]

# Define a simple metric to validate output format
def formatting_metric(example, pred, trace=None):
    # Ensure the feedback starts with a bullet point
    return pred.feedback.strip().startswith("-")

# Set up the optimizer (compiler)
optimizer = BootstrapFewShot(metric=formatting_metric)

# Compile the module
compiled_reviewer = optimizer.compile(CodeReviewer(), trainset=trainset)

# Run our compiled reviewer
result = compiled_reviewer(code="def process(data):\n    print(data)")
print(result.feedback)

During the compile step, DSPy does something magical: it runs the training examples through the LLM, evaluates the outputs against the formatting_metric, identifies which reasoning paths led to success, and automatically formats those successful runs into few-shot exemplars that are injected into the prompt.

If you swap out the underlying LLM from GPT-4 to Claude or LLaMA, you simply re-run the compiler. The code remains completely unchanged, but the generated prompts adapt to the strengths and weaknesses of the new model.

---

Request Hooks and Persistent Memory: The Infrastructure of Self-Evolution

In advanced architectures like the Hermes Agent, DSPy is not used in isolation. It is integrated with infrastructure components like request hooks and persistent memory to create a closed-loop system that evolves in production.

Request Hooks as Middleware

In web frameworks like Flask, request hooks (such as @app.before_request) allow you to run code automatically at specific points in the request-response lifecycle.

DSPy uses a similar pattern. The compiler can inject hooks before and after each module's execution:

• Pre-Execution Hooks: Log inputs, validate schema constraints, and inject contextual memory.
• Post-Execution Hooks: Compute performance metrics, log execution traces, and flag failures.

This instrumentation means the optimization engine doesn't just guess what went wrong; it analyzes the exact execution trace of the failure.

[ User Request ] ──> [ Pre-Execution Hook ] ──> [ DSPy Module ] ──> [ Post-Execution Hook ] ──> [ Trace Database ]

Persistent Memory as a Learning Substrate

An agent cannot evolve without memory. In a self-improving system, persistent memory is not just a cache of past chats; it is a learning substrate.

The DSPy compiler leverages this substrate by using real-world session history as an optimization source:

1. Failure Capturing: When an agent fails a task in production, the failure (and the associated execution trace) is logged to persistent memory.
2. Dataset Synthesis: The optimization engine routinely scans the memory database, grouping failures into patterns.
3. Targeted Evolution: The engine triggers a DSPy compilation run, using the captured failures as new training examples. The compiler rewrites the module's instructions and selects new exemplars to prevent that specific class of failure from ever occurring again.

This is the core of the GEPA (Genetic-Pareto Prompt Evolution) engine used by Hermes. It reads execution traces to understand why things failed, proposes targeted improvements, runs them through the DSPy compiler, and deploys the optimized skills back to the agent via automated Pull Requests.

---

Guardrails and Constraints: Solving the Constrained Optimization Problem

When you allow an AI system to optimize its own prompts, you run the risk of semantic drift—the system optimizing for a narrow metric while breaking other, unmeasured behaviors. For example, a code reviewer optimized solely for brevity might stop reporting critical security bugs because security explanations require too many words.

To prevent this, the optimization loop must be treated as a constrained optimization problem. In Hermes, every evolved variant must pass through a strict set of guardrails before deployment:

• Size Limits: Evolved skills must remain compact (e.g., ≤15KB) to prevent token bloat.
• Semantic Preservation: The mutated module is tested against a held-out validation set to ensure it hasn't drifted from its original core purpose.
• Caching Compatibility: Prompts are structured to maximize prefix-caching, keeping latency and API costs low.
• The Pareto Front: Using multi-objective Pareto optimization, the system balances competing metrics—such as accuracy, speed, and cost—ensuring that an improvement in one area doesn't cause a catastrophic regression in another.

---

Conclusion: The Future of AI is Compiled

The era of hand-crafting prompts is drawing to a close. As AI systems grow more complex, relying on human intuition to write natural-language instructions is no longer viable.

By treating AI tasks as programs with typed signatures, DSPy allows us to apply the rigorous principles of software engineering to the wild world of LLMs. We can compile, optimize, test, and version-control our prompts just like we do with traditional code.

If you are still writing raw system prompts in your codebase, it is time to put down the chisel. Stop prompting, and start programming.

---

Let's Discuss

1. How do you see the role of the "Prompt Engineer" changing over the next 18 months? Will the job shift entirely toward designing metrics and validation datasets rather than writing text?
2. What are the biggest risks you foresee in letting an AI agent compile and deploy its own system prompts and skills in a production environment? How would you design the ultimate safety guardrail?

Leave your thoughts in the comments below!

The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the ebook Hermes Agent, The Self-Evolving AI Workforce: details link, you can find also my programming ebooks with AI here: Programming & AI eBooks.

Take a look at my eBooks

• Python
• JavaScript & TypeScript
• C# / .NET
• Swift & Apple Platform
• Kotlin & Android
• Rust

---

Python

The Foundations of Python

Data Structures and the Standard Library

Web Development with Python
Building backend services and dynamic websites with a framework like Flask

Advanced Python & AI Integration
Deep dive into OOP, decorators, asyncio, and orchestrating LLMs with LangChain.

Gemini 3 Python Programming - The Complete Guide
Agents, Veo 3.1, Lyria, Nano Banana/Pro, Function Calling, Grounding, Computer Use and Robotics

AI Autonomous Agents with Python Programming
Master LangGraph, CrewAI, and RAG to Build Self-Correcting Swarms and Autonomous Digital Workers

Finance & AI Trading with Python Programming
Master Algorithmic Trading, Financial NLP, and Vectorized Backtesting to Build Autonomous 'News + Math' Strategies

Cloud-Native Python, DevOps & LLMOps. Containerization, Kubernetes, and Serving AI Models at Scale
From Docker and Kubernetes to Serving LLMs with Pulumi

Defensive Cybersecurity with Python Programming
A Practical Guide to System Monitoring, Network Defense, and Automated Security Hardening

Data Science & Analytics with Python Programming

Neural Networks & Deep Learning with Python Programming

Architecting Neuro-Symbolic Agents with Python Programming
Integrating LLMs, Wolfram Alpha, IBM Watson and Open Source Stacks for Near-Zero Hallucination Systems

Bioinformatics & AI with Python Programming
Master Genomic Data Science, Protein Folding with AlphaFold, and AI-Driven Drug Discovery

Geospatial AI (GeoAI) with Python Programming
Building Autonomous GIS Agents, Deep Learning Models, and Interactive Dashboards

Astrophysics & AI with Python Programming
Building Research Agents for Astronomy, Cosmology, and SETI

Open-Source LLMs & Local Fine-Tuning
Mastering LoRA, vLLM, Ollama, and Custom SLMs

Unsloth: Efficient Fine-Tuning for Large Language Models
Methods and Workflows for Fine-Tuning and Deploying Large Language Models on Limited Hardware

Hermes Agent: The Self-Evolving AI Workforce
Architecting Autonomous Systems that Learn, Remember, and Grow.

Frontier AI Safety, Mechanistic Interpretability & Alignment Engineering
Inspecting Neural Circuits, Steering Vectors, Autonomous Capability Evals, and Scalable Oversight for Superintelligent Systems.

---

JavaScript & TypeScript

Foundations
OpenAI API, Zod, and LangChain.js

The Modern Stack
Building Generative UI with Next.js, Vercel AI SDK, and React Server Components.

Master Your Data
Production RAG, Vector Databases, and Enterprise Search.

Autonomous Agents
Building Multi-Agent Systems and Workflows with LangGraph.js

The Edge of AI
Local LLMs (Ollama), Transformers.js, WebGPU, and Performance Optimization

The AI-Ready SaaS Boilerplate. Auth, Database with Vector Support, and Payment Stack
Auth, Database with Vector Support, and Payment Stack.

Backend for Frontend & Intelligent APIs. tRPC, Edge Functions, and LLM Data Transformation
tRPC, Edge Functions, and LLM Data Transformation.

The Monetization Engine. Stripe, Smart Dunning, and AI Customer Support Agents
Stripe, Smart Dunning, and AI Customer Support Agents.

AI-Driven Growth Engineering. Programmatic SEO with GPT-4, Content Automation, and Analytics.
Programmatic SEO with GPT-4, Content Automation, and Analytics.

No More Localhost. Mastering Docker, Linux, and Containerization for JS & AI Apps
Mastering Docker, Linux, and Containerization for JS & AI Apps.

The Perfect Pipeline. Advanced CI/CD with GitHub Actions, Automated Testing, and AI Code Reviews
Advanced CI/CD with GitHub Actions, Automated Testing, and AI Code Reviews.

Kubernetes & Orchestration. Deploying Scalable Node.js & AI Clusters without Tears
Deploying Scalable Node.js & AI Clusters without Tears.

React Native for Web Developers
From Next.js to Expo, NativeWind, and Universal App

Offline AI & Local LLMs. Running Llama 3 and Vector Search directly on the Smartphone
Running Llama 3 and Vector Search directly on the Smartphone.

App Store Engineering. CI/CD for Mobile (EAS), OTA Updates, and AI-Driven App Store Optimization
CI/CD for Mobile (EAS), OTA Updates, and AI-Driven App Store Optimization.

The TypeScript-First Architect. Building Robust Applications with Effect, Zod, and Drizzle
Building Robust Applications with Effect, Zod, and Drizzle.

The Native Era. Modern Node.js, Bun & Deno without Bundlers or Transpilers
Modern Node.js, Bun & Deno without Bundlers or Transpilers.

Local-First Systems in TypeScript. Collaborative & Offline-Ready Web Apps with CRDTs and WASM DBs
Collaborative & Offline-Ready Web Apps with CRDTs and WASM DBs.

TypeScript Metaprogramming. Advanced Type Gymnastics, Modern Decorators, and Compiler Internals
Advanced Type Gymnastics, Modern Decorators, and Compiler Internals.

Model Context Protocol (MCP) & Computer Use. Standardizing Tool Integration, Vision-Driven Browser Automation, and Agent Governance in TypeScript

Standardizing Tool Integration, Vision-Driven Browser Automation, and Agent Governance in TypeScript.

Generative Media & Visual Workflow Engines. Node-Based AI Canvases, Real-Time Media Streaming Pipelines, and WebGPU Processing in TypeScript
Node-Based AI Canvases, Real-Time Media Streaming Pipelines, and WebGPU Processing in TypeScript.

Neuro-Symbolic AI & Knowledge Graphs. Deterministic Solvers, GraphDBs, Ontologies, and Zero-Hallucination Architectures
Deterministic Solvers, GraphDBs, Ontologies, and Zero-Hallucination Architectures in TypeScript.

Event-Driven Architecture & DDD in TypeScript. Event Sourcing, CQRS, and Microservices at Scale
Event Sourcing, CQRS, and Microservices at Scale.

Building Desktop Apps & Developer Tools with Tauri 2.0, Rust, and TypeScript
Cross-platform desktop tools with Tauri, Rust, and TypeScript.

FinTech Architecture in TypeScript. Precision Math, Double-Entry Ledgers, and High-Reliability Payment Pipelines
Precision Math, Double-Entry Ledgers, and High-Reliability Payment Pipelines.

Hardened TypeScript. Passkeys, Supply Chain Defense, and Zero-Trust Architectures
Passkeys, Supply Chain Defense, and Zero-Trust Architectures.

Spatial Web Development. Building Interactive 3D and WebXR Experiences with React Three Fiber & TypeScript
Building Interactive 3D and WebXR Experiences with React Three Fiber & TypeScript.

Multiple-choice test book for: Foundations (Volume 1)

Spatial Web Development. Building Interactive 3D and WebXR Experiences with React Three Fiber & TypeScript
Building Interactive 3D and WebXR Experiences with React Three Fiber & TypeScript.

Jev: The Definitive Guide to System One AI
Building Sub-100ms Decision Engines, Calibrated Guardrails, and Two-Speed Architectures with Jev and Generative LLMs

---

C# / .NET

Get all the Ten C# & AI volumes at a discounted price, or choose an ebook:

The Foundations
Syntax, Type System, and Logic for Modern Developers.

Advanced OOP & AI Data Structures
Modeling Complex Systems and Tensors.

Data Manipulation, LINQ & Vectors
From Collections to AI Embeddings

Asynchronous AI Pipelines
Async/Await, Parallelism, and Streaming LLM Responses.

Building AI Web APIs with ASP
NET Core. Serving Models and Chat Endpoints

Intelligent Data Access with EF Core
Vector Databases, RAG, and Memory Storage.

Cloud-Native AI & Microservices
Containerizing Agents and Scaling Inference.

The Core of AI Engineering: Microsoft Semantic Kernel & Agentic Patterns

Edge AI & Local Inference
Running LLMs (Llama/Phi) locally with C# and ONNX.

High-Performance C# for AI
Span, SIMD, and Optimizing Token Processing

Full Stack AI with Blazor. Building Interactive Copilots and WASM AI
Building Interactive Copilots and WASM AI.

Enterprise AI Integration & Process Automation. Connecting LLMs to legacy systems, internal APIs, and real-world business processes
Connecting LLMs to legacy systems, internal APIs, and real-world business processes.

AI for Game Development & Interactive Simulation. Using LLMs and generative AI to create dynamic worlds and intelligent characters in Unity
Using LLMs and generative AI to create dynamic worlds and intelligent characters in Unity.

---

Swift & Apple Platform

Get the Swift & AI 10 Volumes Bundle at a discoutend price, or choose an ebook:

Core ML & Vision Framework
On-device image classification, object detection, and custom model integration with Core ML and Vision.

Apple Intelligence & Foundation Models
Building apps with Apple's on-device LLM APIs, Writing Tools, and the Apple Intelligence framework

Natural Language & Speech
NLP, sentiment analysis, text classification, and Speech-to-Text with Apple's Natural Language and Speech frameworks.

SwiftUI for AI Apps
Building reactive, intelligent interfaces that respond to model outputs, stream tokens, and visualize AI predictions in real time

Create ML Studio
Training custom models without Python: tabular, image, sound, and motion classifiers using Create ML in Swift.

MLX Swift & Local LLMs. Deep dive into Apple's MLX framework for high-performance machine learning.
Building custom inference engines, fine-tuning local models (LoRA), and leveraging Unified Memory directly from Swift.

visionOS & Spatial AI with Swift

Swift + OpenAI & LangChain
Integrating external LLM APIs, RAG pipelines, and agentic workflows in iOS and macOS apps

CoreData, CloudKit & Vector Search

Shipping AI Apps to the App Store

---

Kotlin & Android

On-Device GenAI with Android Kotlin
Mastering Gemini Nano, AICore, and local LLM deployment using MediaPipe and Custom TFLite models

Edge AI Performance with Android Kotlin
Optimizing hardware acceleration via NPU, GPU, and DSP. Advanced quantization and model pruning

Android AI Agents
Building autonomous apps that use Tool Calling, Function Injection, and Screen Awareness to perform tasks for the user

---

Rust

Rust Advanced Memory Patterns for AI
Mastering Lifetimes, Smart Pointers, and custom allocators for managing large models and datasets

Extending Python with Rust. Creating high-performance Python modules with PyO3.
Creating high-performance Python modules with PyO3 for data processing, tokenization, and inference, replacing slow Python code.

---