DEV Community: Manouk Draisma

From zero evals to a working multimodal evaluation in 30 minutes using LangWatch Skills

Manouk Draisma — Tue, 24 Mar 2026 15:04:40 +0000

How I went from "it works on my machine" to measurable agent quality using LangWatch Skills, Jupyter notebooks, and a path to production on AWS.*

The problem nobody talks about
You built an agent. It uses tools, handles multimodal inputs, answers questions from a knowledge base. You demo it to your team and it works great. Ship it.

Three days later: the satellite image analysis returns garbage NDVI estimates. The knowledge base tool stops getting called for calibration questions, the LLM just wings it. Nobody noticed because there were no tests.

This is the gap between "I have an agent" and "I have a reliable agent." LangWatch fills it.

What I built
The InField Agent is a weather station advisory system built with Strands Agents SDK. It has three multimodal capabilities:

Knowledge base — calibration procedures for Davis Instruments weather stations

Station status — fleet inventory, battery health, reporting gaps

Satellite imagery — NDVI estimation from satellite images using vision models

from strands import Agent
from strands.models.openai import OpenAIModel

def create_agent() -> Agent:
    model = OpenAIModel(model_id="gpt-5-mini")
    return Agent(
        model=model,
        system_prompt=SYSTEM_PROMPT_ADV,
        tools=[search_knowledge_base_tool, check_station_status, analyze_satellite_image],

The satellite tool sends images to a vision model and gets back structured NDVI data:

@tool
def analyze_satellite_image(image_id: str) -> dict:
    """Analyze a satellite image to estimate NDVI."""
    image_path = _DATA_DIR / f"{image_id}.png"
    # ... encode image as base64, send to gpt-5-mini with vision ...
    return {
        "ndvi_estimate": 0.65,
        "confidence": "medium",
        "vegetation_cover_pct": 72,
        "dominant_land_types": ["cropland", "grassland"],
        "summary": "Healthy vegetation with moderate crop coverage"
    }

This is a non-trivial agent to test. You have text-based retrieval, structured data queries, and multimodal vision analysis all behind the same prompt. Traditional unit tests cover maybe 10% of the failure surface.

Step 1: Add LangWatch skills
LangWatch ships skills — curated Claude Code instructions that know how to wire up tracing, evaluations, scenarios, and prompt management in your project. Think of them as recipes that understand the LangWatch SDK.

npx skills add langwatch/skills/evaluations
npx skills add langwatch/skills/scenarios

This drops skill files into .claude/skills/ in your project. When you use Claude Code, it picks up these instructions and knows exactly how to scaffold evaluations and scenarios for your specific agent.

The skills also set up the skills-lock.json to track versions:


{
  "version": 1,
  "skills": {
    "evaluations": {
      "source": "langwatch/skills",
      "sourceType": "github",
      "computedHash": "170c4e99..."
    },
    "scenarios": {
      "source": "langwatch/skills",
      "sourceType": "github",
      "computedHash": "b3afbe5c..."
    }
  }
}

Step 2: Tracing — see what your agent actually does
Before you evaluate anything, you need observability. LangWatch tracing captures every LLM call, tool invocation, and input/output pair.

import langwatch
from dotenv import load_dotenv

load_dotenv(Path(__file__).parent / ".env")
langwatch.setup()

@langwatch.trace(name="InField Agent Turn")
def handle_turn(agent, user_input: str, thread_id: str):
    langwatch.get_current_trace().update(metadata={
        "thread_id": thread_id,
    })
    result = agent(user_input)
    return result.message["content"][-1]["text"]

Two lines of setup, one decorator. Every agent turn now shows up in the LangWatch dashboard with the full tool chain visible.

Step 3: Multimodal experiments in Jupyter
This is where it gets interesting. The evaluations skill guided me toward using Jupyter notebooks with langwatch.experiment for batch testing. The key insight: satellite images can be embedded as markdown in the dataset, and LangWatch renders them inline in the UI.

The dataset
Each row targets one of the three capabilities. Satellite rows include the actual image:

SATELLITE_BASE_URL = "<https://storage.googleapis.com/experiments_langwatch>"

def image_to_markdown(image_id: str) -> str:
    return f"![Satellite image {image_id}]({SATELLITE_BASE_URL}/{image_id}.png)"

dataset = [
    # Knowledge base
    {
        "input": "How do I calibrate the temperature reading on a Vantage Pro2?",
        "expected_output": "Use the temperature calibration offset in the console setup menu.",
        "capability": "knowledge_base",
    },
    # Station status
    {
        "input": "Which stations have low battery levels?",
        "expected_output": "A list of stations with battery voltage below 3.0V.",
        "capability": "station_status",
    },
    # Satellite — multimodal
    {
        "input": "Analyze this satellite image and estimate the NDVI.",
        "image": image_to_markdown("01"),
        "expected_output": "An NDVI estimate between -1.0 and 1.0 with vegetation coverage.",
        "capability": "satellite",
    },
    {
        "input": "What does this satellite image tell us about vegetation health?",
        "image": image_to_markdown("03"),
        "expected_output": "An NDVI estimate with vegetation health description.",
        "capability": "satellite",
    },
    {
        "input": "Estimate the vegetation index for this field.",
        "image": image_to_markdown("07"),
        "expected_output": "An NDVI estimate with vegetation cover and land classification.",
        "capability": "satellite",
    },
]

The evaluators
LangWatch supports platform-configured evaluators that you reference by slug. I set up three:

The tool usage check is critical. An agent that answers correctly without calling the tool is a hallucination risk, it just happened to get lucky this time.

The experiment loop

experiment = langwatch.experiment.init("infield-agent-multimodal")

for index, row in experiment.loop(df.iterrows(), threads=1):
    output = run_agent(row["input"])

    data = {"input": row["input"], "output": output}
    if pd.notna(row.get("image")):
        data["image"] = row["image"]

    experiment.evaluate("answer-relevancy-nxwec", index=index, data=data)
    experiment.evaluate("answer-correctness-b5e6x", index=index, data={**data, "expected_output": row["expected_output"]})
    experiment.evaluate("tool-usage-check-aljvk", index=index, data=data)

What you see in LangWatch
As the notebook runs, results appear in real time:

Satellite images rendered inline next to scores

Pass/fail per evaluator per row

Scores comparable across model versions and prompt changes side by side

That last point is the payoff. Change a prompt, run the experiment again, see exactly what moved. Not just for one input — across the whole dataset.

The @langwatch.trace decorator also means every evaluation run produces full traces. Drill into a failing row and see exactly which tool was called, what the LLM received, and where it went wrong.

Step 4: Simulations — test the agent as a system
Evaluations test isolated input-output pairs. Simulations test multi-turn conversations where the agent interacts with a simulated user.

LangWatch Scenario is the framework. It has three actors:

Agent Under Test — your agent

User Simulator — an LLM that generates realistic user messages

Judge — an LLM that evaluates the conversation and decides pass/fail

import pytest
import scenario

scenario.configure(default_model="openai/gpt-4.1-mini")

@pytest.mark.asyncio
async def test_calibration_workflow():
    class InFieldAdapter(scenario.AgentAdapter):
        async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
            return run_agent(input.messages[-1]["content"])

    result = await scenario.run(
        name="calibration guidance",
        description="A field technician needs to calibrate barometric pressure on a Vantage Pro2. They have a known reference pressure but aren't sure about the procedure.",
        agents=[
            InFieldAdapter(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(),
        ],
        script=[
            scenario.user("How do I calibrate the barometric pressure?"),
            scenario.agent(),
            scenario.judge(criteria=[
                "Agent used the knowledge base tool",
                "Agent provided step-by-step calibration instructions",
            ]),
            scenario.user(),
            scenario.agent(),
            scenario.judge(criteria=[
                "Agent answered the follow-up using tool results, not general knowledge",
            ]),
        ],
    )
    assert result.success

The simulation loop runs automatically: the user simulator generates contextual follow-ups, the agent responds, the judge scores against your criteria. You define the scenario once and it tests the full conversation flow.

For adversarial testing, swap UserSimulatorAgent for RedTeamAgent:

agents=[
    InFieldAdapter(),
    scenario.RedTeamAgent(),  # Tries to make the agent hallucinate or go off-topic
    scenario.JudgeAgent(criteria=["Agent stays within scope of weather stations"]),
]

Step 5: Deploy to AWS
With evaluations passing and simulations green, ship it.

The InField Agent is a single-turn Q&A system — Lambda is the natural fit:

The knowledge base is bundled inside the Lambda package. LLM inference runs on OpenAI's servers. Lambda just orchestrates the agent loop.

Lambda handler

from strands import Agent
from strands.models.openai import OpenAIModel

agent = Agent(
    model=OpenAIModel(model_id=os.environ.get("MODEL_ID", "gpt-5-mini")),
    system_prompt=SYSTEM_PROMPT_ADV,
    tools=[search_knowledge_base_tool],
)

def handler(event, _context):
    prompt = event.get("prompt", "")
    if not prompt:
        return {"statusCode": 400, "body": "Missing 'prompt' in request"}
    result = agent(prompt)
    return {"statusCode": 200, "body": result.message["content"][-1]["text"]}

Package and deploy
Three options, depending on your dependency size:

ZIP + Strands Layer (simplest):

aws lambda create-function \\\\
  --function-name infield-agent \\\\
  --runtime python3.12 \\\\
  --handler lambda_handler.handler \\\\
  --zip-file fileb://packaging/app.zip \\\\
  --architectures arm64 \\\\
  --memory-size 256 \\\\
  --timeout 30 \\\\
  --layers "arn:aws:lambda:us-east-1:856699698935:layer:strands-agents-py312-aarch64:1" \\\\
  --environment "Variables={OPENAI_API_KEY=your-key}" \\\\
  --role arn:aws:iam::YOUR_ACCOUNT:role/lambda-execution-role
Container Image (when dependencies exceed 250 MB):

FROM public.ecr.aws/lambda/python:3.12-arm64
COPY requirements.txt .
RUN pip install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
COPY lambda_handler.py prompts.py tools.py ${LAMBDA_TASK_ROOT}/
COPY knowledge_base/ ${LAMBDA_TASK_ROOT}/knowledge_base/
CMD ["lambda_handler.handler"]

Production checklist

Store API keys in Secrets Manager or SSM Parameter Store
Enable CloudWatch logging
Set up API Gateway with authentication
Configure CloudWatch alarms on error rate and duration
Run evaluations in CI before deploying new versions

That last point closes the loop: your Jupyter notebook evaluations become a CI gate. A prompt change that drops answer relevancy below threshold blocks the deploy.

The evaluation lifecycle
Here is what you end up with:

LangWatch covers all four stages with the same evaluators. The answer-relevancy check you run in a notebook is the same one that scores production traces. Consistency across the lifecycle means no surprises.

What this cost me
Setup time: ~30 minutes. The skills did most of the scaffolding.

Notebook evaluations: 3 satellite images x 3 evaluators = 9 evaluation calls per run. Under a minute.

Lambda deployment: 256 MB, arm64, 30s timeout. Pennies at low volume.

LangWatch traces: free tier covers experimentation. Platform evaluators included.

Takeaways
Evaluations are not optional for multimodal agents. A satellite image tool that returns plausible-sounding garbage is worse than one that throws an error. You need automated checks.

Tool usage matters as much as answer quality. An agent that gives the right answer without calling the tool is a ticking time bomb. The tool-usage-check evaluator catches this.

Simulations find bugs that evaluations miss. Single-turn evaluations cannot test whether the agent stays grounded across a multi-turn conversation. Scenario simulations can.

LangWatch skills bootstrap the hard part. npx skills add langwatch/skills/evaluations gives Claude Code the context to scaffold everything — the notebook, the evaluators, the experiment loop. You focus on defining what "correct" means for your agent.

Same evaluators, every stage. Run them in a notebook during development, in CI before deploy, and on live traces in production. One set of quality criteria, applied everywhere.

All code is available at https://github.com/langwatch/satellite-agent The Jupyter notebook runs end to end if you have an OpenAI key and a LangWatch project.

Your coding agent already knows how to test your AI agent (we just turned it into a Skill)

Manouk Draisma — Mon, 23 Mar 2026 20:34:43 +0000

We’re adding something new at LangWatch: Skills.

And the idea is pretty simple:

your coding agent already knows how to do a lot of the work you’re still doing manually

You just haven’t packaged it properly yet.

The frustrating part of building AI agents

If you’ve built an LLM agent recently, you probably recognize this loop:

you tweak something
you run a few test conversations
it seems better
you ship it
something breaks in production

Then you repeat.

We’ve been there too.

It’s not that you don’t know you need evals, testing, or simulations.

It’s that doing all of that properly is… a lot.

The real work isn’t building, it’s validating

When we started LangWatch, we thought the main challenge was:

getting agents to behave correctly

But in practice, the bigger challenge was:

proving that they behave correctly

That means:

setting up eval datasets
writing tests
simulating real user behavior
instrumenting pipelines
understanding failures

And most of this ends up being:

manual
repetitive
easy to skip

The worst part: testing agents doesn’t look like testing code

Traditional testing breaks down with LLMs.

You can’t just say:

assert output == expected

Because agents are non-deterministic. The same input can give different outputs, which makes rigid testing fragile ([LangWatch][1]).

So what do people do instead?

They “vibe test”.

try a few examples
eyeball the results
hope nothing breaks

It doesn’t scale.

We already solved part of this (with agents testing agents)

If you’ve seen our earlier work (Scenario), you know we took a different approach:

use an agent to test your agent

Instead of fixed inputs/outputs, you:

simulate real user behavior
define success criteria
let an agent explore and evaluate

This makes testing much closer to reality.

But even then…

You still had to set everything up yourself.

So we asked: why are we still doing this manually?

At this point, most developers already have a coding agent open all day.

And those agents are actually pretty good at:

writing tests
structuring code
following instructions

So we started asking:

what if we let the coding agent handle the “quality work” too?

Not just writing features.

But:

setting up evals
creating simulations
instrumenting systems
analyzing behavior

That’s where Skills come in

We built LangWatch Skills as a way to give your coding agent reusable capabilities.

A Skill is basically:

a structured way to get your coding agent to do something correctly, every time

Not just:

“generate some code”

But:

“do this properly, following best practices, with full coverage”

What a Skill actually looks like

Under the hood, Skills are closer to:

structured instructions
workflows
examples
best practices

In general, agent skills are “instruction modules” that extend what an agent can do without retraining it ([philschmid.de][2]).

They tell the agent:

when to apply something
how to do it
what good looks like

What you can do with LangWatch Skills

With Skills, you can tell your coding agent to:

instrument your agent
generate evaluation notebooks
create simulation-based tests
explore production performance
red-team your system

And instead of figuring out how to do it…

…the agent just does it.

The shift is subtle, but important

Before:

you write eval code, tests, and infrastructure

After:

you review and guide what your agent generates

You move from:

implementation to
coordination

And that’s actually where most of the value is.

This is part of a bigger shift: “harness engineering”

There’s a growing idea in the ecosystem:

the performance of your agent depends heavily on how you configure it

Not just the model.

But:

tools
context
memory
skills

These are all part of what some people call the agent “harness” — the system around the model that shapes its behavior ([humanlayer.dev][3]).

Skills are one of the most powerful (and underused) pieces of that.

But Skills aren’t magic

One important thing we’ve learned:

Skills don’t automatically fix everything.

In fact, a lot of skills:

don’t improve performance
or only help in specific contexts

Recent research shows many skills have limited impact unless they’re well-designed and properly evaluated ([arXiv][4]).

So the goal isn’t:

“add more skills”

It’s:

“add the right skills, and make them actually useful”

Why this matters now

We’re entering a phase where:

building agents is easy
making them reliable is not

The bottleneck has shifted.

And the teams that win won’t just be the ones who:

build faster

But the ones who:

validate better
iterate faster with confidence

What we’re aiming for

With Skills, the goal is simple:

reduce the amount of manual work required to build reliable AI systems

So instead of:

wiring pipelines
writing eval scaffolding
guessing what broke

You can:

delegate
review
improve

SWould love feedback

This is a new direction for us, and we’re still figuring out:

What makes a “good” Skill?
Where do Skills break down?
What should be automated vs controlled?

If you’re working on LLM agents, I’d love to hear:

how you’re handling evals today
what’s still painful
what you’ve tried that didn’t work

Try it out

If this resonates, you can check out what we’re building here:

👉 LangWatch Skills

Final thought

Your coding agent is already capable of doing much more than we typically ask of it.

Skills are just a way to unlock that.

The interesting question now is:

what else are we still doing manually that agents could handle better?

Testing AI agents with domain-driven TDD

Manouk Draisma — Thu, 02 Oct 2025 16:13:49 +0000

TL;DR:
Traditional testing doesn’t work well for AI agents (LLMs are nondeterministic, brittle to assert). I built a flight booking agent from scratch using Scenario, a framework we built for running agent simulations + LLM evaluations. Writing scenario tests first (domain-driven TDD) gave me a way to discover domain rules, evolve the model, and ship an agent with confidence.

Why testing AI agents is hard
Normal unit/integration tests fall apart with AI systems.
Same input, different outputs
Hard to assert strings reliably
You don’t just care about text, you care about business outcomes

Without testing, you’re basically flying blind.

That’s why I tried a scenario-driven approach: define business capabilities first, then let the failures tell me what’s missing.

Step 1 – Write a scenario test

Start with a business journey, not code.

`const result = await scenario.run({
  setId: "booking-agent-demo",
  name: "Basic greeting test",
  description: "User wants to book a flight and expects polite interaction",
  maxTurns: 5,
  agents: [
    scenario.userSimulatorAgent(),
    agentAdapter,
    scenario.judgeAgent({
      criteria: [
        "The agent should greet politely",
        "The agent should understand the user wants to book a flight",
      ],
    }),
  ],
  script: [scenario.proceed(5)],
});
`

First run = red, as expected. That failure became my to-do list.

Step 2 – Let failures guide the build

No endpoint? → Build a basic HTTP route.

Agent replies static text? → Hook up an LLM.

LLM forgets context? → Add conversation memory.

Each test failure = missing piece of the domain.

Step 3 – Scale up to a full booking journey

Once greetings + memory worked, I moved to full booking:

`const result = await scenario.run({
  setId: "booking-agent-scenario-demo",
  name: "Complete flight booking",
  description: "User books NY → London with all required details",
  maxTurns: 100,
  agents: [
    scenario.userSimulatorAgent(),
    agentAdapter,
    scenario.judgeAgent({
      criteria: [
        "Collect passenger info",
        "Collect dates + airports",
        "Create booking in system",
        "Confirm booking to user",
      ],
    }),
  ],
  script: [scenario.proceed(100)],
});`

Ran it → failed. Checked DB → no bookings (because I hadn’t built the tools). Wrote tools → ran again → bookings created, but airport codes didn’t match. Another hidden domain rule uncovered.

Why this worked
Scenarios = living documentation of the domain
Failing tests = backlog of missing business rules
Confidence = I can change prompts, swap models, or go multi-agent without fear of silent regressions

Takeaways

Scenario = domain-driven TDD for AI agents

You don’t just test outputs, you validate business outcomes

Each failure teaches you something new about your domain

Scenarios double as specs + tests + onboarding docs

If you’re curious: Scenario is open source
(works with any LLM/agent framework).

Why Agent Simulations are the new Unit Tests for AI

Manouk Draisma — Wed, 09 Jul 2025 11:42:12 +0000

From Self-Driving Cars to AI Agents

If you've followed the development of autonomous vehicles (AVs), you know that simulation is a non-negotiable part of the process. Companies like Waymo and Cruise don't just rack up millions of miles on physical roads; they drive billions of miles in virtual worlds. This isn't just for fun. It's a core engineering solution to a fundamental machine learning problem: the long tail.

The real world is messy and unpredictable. The most dangerous driving scenarios are, thankfully, also the rarest. You can drive for years without encountering a tire blowout on a crowded highway or a pedestrian chasing a ball from between two parked cars. Relying on real-world data alone to train an AV would mean you'd have almost no data on the most critical events. As a result, your model would be unprepared.

This is exactly the kind of challenge we now face with the new generation of AI agents. To build agents that can reliably operate software, browse the web, or manage workflows, we need to adopt the same playbook: agent simulation.

What is AI Agent Simulation?
AI agent simulation refers to creating controlled, repeatable environments to test how an autonomous AI agent handles complex or rare scenarios — before deploying it in the real world.

Simulations in Autonomous Vehicles (AVs): A Blueprint for AI Agent Testing
Simulations are the key to make AVs succeed. In places like San Francisco, a large part of the population takes a Waymo instead of an Uber, which, as you might have guessed, is a fully autonomous car. Last week, A Tesla Model Y completed its first fully autonomous delivery, driving itself from the factory to the customer's location without any human intervention.

Now, how does this work? A simulation platform for AVs has three core components:
Sensor Simulation : The system generates realistic data for all the car's sensors. This includes simulating the precise patterns of light from a LiDAR sensor bouncing off a wet road, the noise in a camera feed at dusk, or the signal degradation of RADAR in a snowstorm.

Physics Engine : This governs the rules of the virtual world. It ensures vehicles have realistic acceleration and braking, that weather affects traction, and that lighting changes accurately with the time of day.

Scenario Generation : This is the most crucial part. Engineers can programmatically create and vary critical scenarios an infinite number of times. They can test what happens if a cyclist swerves a little earlier, if a traffic light is partially obscured by a tree branch, or if another car runs a red light.

The key insight here is that simulation allows you to control the data distribution. You don’t need to drive hours on a boring highway, because, after a while, the model has learned how to do this. You can focus on the events that are most important for safety and robustness, generating millions of permutations of these "long-tail" events. This is how you build a model that doesn't just work 99% of the time, but is prepared for the critical 1%.

How AI Agent Simulations help tackle the long tail problem
Before diving into how simulations in AI agents work, let’s briefly talk about AlphaGo. AlphaGo mastered the game ‘Go’ (a much harder game than chess). It achieved this remarkable feat not by just studying human games, but by playing millions of games against itself in a simulated world. This process, known as self-play, allowed AlphaGo to explore a vast number of strategies and counter-strategies, far beyond what any human could ever play in a lifetime. It learned the rules of the game and then, through relentless, simulated practice, discovered novel tactics and achieved a superhuman level of proficiency. The key was the simulated environment, which provided a perfect, repeatable, and scalable training ground.

That same approach is now essential for AI agent testing.

To be truly reliable, these AI agents need to be exposed to a massive and diverse range of situations, especially the tricky "long-tail" events that are uncommon in day-to-day use but are critical to handle correctly.

Let's take the example of a customer support agent for an e-commerce company. In a simulated environment, we can test this AI agent against a vast array of edge cases that would be impractical to replicate with human testers alone. We could, for instance, simulate a scenario where a customer has a legitimate complaint but is using sarcastic and angry language. The simulation could vary the intensity of the language, the specific nature of the complaint (e.g., a damaged product, a late delivery, a billing error), and the customer's history with the company.

Running agent simulations of these edge cases at scale is how your system learns to go beyond correct responses and toward robustness, safety, and nuance. It can learn to de-escalate tense situations, understand implied intent, and navigate complex, multi-step problems. Just as AlphaGo became a master of Go by playing against itself in a simulated world, AI agents can master the art of customer service, and countless other tasks, by being rigorously tested and (eventually) trained in their own virtual worlds. This is how we move from agents that are merely functional to agents that are truly intelligent and dependable.

Start Simple: Applying Agent simulation testing without AV-Scale complexity
Okay, building a simulation engine that mirrors the complexity of Waymo's sounds like a massive engineering project in itself. And it can be. But the good news is you don’t have to boil the ocean to get started. The principles of automated agent simulations can be applied today, even without a billion-dollar R&D budget.

The shift starts with a change in mindset: treating the evaluation of your AI agent not as a final, manual QA step, but as a core part of the development loop, just like unit tests or integration tests for traditional software.

Instead of building a whole virtual world, you start by defining a library of critical scenarios. A scenario is the AI agent equivalent of a test case. It’s a specific, repeatable challenge you want your agent to overcome.

Scenario 1 (Happy Path): "A user wants to book a flight and provides all the necessary information clearly."

Scenario 2 (Edge Case): "A user asks to book a flight but provides a nonsensical date, like February 30th."

Scenario 3 (Robustness Test): "During the booking process, the airline's API times out. The agent must inform the user and suggest trying again later."

Scenario 4 (Safety Test): "A user expresses extreme frustration after a failed booking. The agent must recognize the sentiment and correctly escalate to a human support agent."

Once you have these scenarios, you need a way to run them against your agent automatically every time you push a change, right inside your development environment.

This is the philosophy behind what we're building with LangWatch Scenario a framework for scenario-based evaluation and automated agent testing.

The goal is to provide the framework for this new kind of testing discipline. It lets you write these agent-centric test cases and integrate them directly into the workflows your team already uses, i.e. think pytest and CI/CD pipelines. This turns agent evaluation from a slow, manual bottleneck into an automated, continuous check. It allows subject-matter experts (like your best support agents) to help define what "good" looks like, closing the loop between the real world and your training data.

The leap from clever demo to production-grade AI has always required one thing: testing discipline.

For AI agents, that discipline is now rooted in agent simulation, just as it is for AVs and Go-playing agents. With LangWatch Scenario, you’re not just hoping your agent behaves correctly—you’re proving it can, across the long tail of real-world messiness. It’s how we move from "it works on my machine" to "it works reliably, safely, and effectively for our users."

Learn more about how LangWatch Scenarios integrates with CI/CD pipelines.: https://github.com/langwatch/scenario

The power of MIPROv2 - using DSPy optimizers for your LLM-pipelines

Manouk Draisma — Thu, 19 Dec 2024 10:15:42 +0000

Fine-tuning prompts for consistent, high-quality output is a game-changer. Yet, until now, optimizing prompts has often required deep technical knowledge and coding skills—especially when using advanced frameworks like DSPy or a lot of manual trial & error work. But what if there was a way to leverage the power of DSPy’s MIPROv2 without diving into complex code? Enter LangWatch’s Optimization Studio, where MIPROv2 lives in a low-code environment designed to make prompt optimization more accessible than ever.

What Is MIPROv2, and why does it matter?
Let’s take a look at the magic behind MIPROv2. Part of the DSPy library, MIPROv2 (Multiprompt Instruction Proposal Optimizer Version 2) is a state-of-the-art optimizer developed from Stanford’s DSP research. At its core, DSP (Demonstrate - Search - Predict) offers a unique approach to prompt optimization, prioritizing "prompt optimization" over traditional "prompt engineering." By using DSPy, you can find the best prompt variations that align with your model's needs, maximizing accuracy and relevance in output.

MIPROv2 is designed to automate this process, finding the optimal combination of prompt demonstrations and instructions that result in accurate, useful model responses. And while DSPy is powerful, navigating its open-source code can be challenging for those without technical expertise. That’s where the Optimization Studio’s low-code environment comes in, making it possible to run advanced optimizations without diving into complex coding.

Use Optimization Studio’s low-code environment for MIPROv2
LangWatch’s Optimization Studio is built to bring the best of DSPy’s capabilities to users who want results fast, without the need to understand every line of underlying code. With a user-friendly, low-code interface, the Optimization Studio allows you to:

Use MIPROv2 to optimize prompts without any technical hurdles. Access the same high-quality results that DSPy offers, but with an intuitive, guided process.

Save Time and Resources: Optimization Studio automates the most labor-intensive parts of prompt refinement, like generating demonstrations and running evaluation trials, so you can focus on strategy, not code.

Achieve Consistent Quality: With MIPROv2, you can confidently create prompts that consistently meet your quality criteria, even as the demands on your LLM change.

How MIPROv2 works in Optimization Studio
The Optimization Studio harnesses the full capability of DSPy’s MIPROv2 through a guided, three-step process. Here’s a breakdown of how this happens in a way that’s simple, efficient, and code-light:

Step 1: Demonstrate with high-quality data
The first step is to gather input-output examples (demonstrations) that represent the kind of responses you expect from your LLM. MIPROv2 then automatically generates a range of demonstration sets from this data, ensuring that each set showcases both accurate and relevant responses. You simply provide the input-output pairs and let the Optimization Studio do the rest.

Step 2: Craft effective instructions
Using the data from Step 1, MIPROv2 generates various prompt instructions by analyzing the most effective ways to achieve your desired results. The Optimization Studio uses a summary of the demonstration sets, and from there, it creates different prompt instructions that match the style and goal of your project. You can review, adjust, and select the most relevant prompt without needing to experiment manually.

Step 3: Select the best Prompt with bayesian Optimization
To find the most effective prompt, MIPROv2 runs evaluation trials, scoring each demo-prompt pair against the criteria you specify. By using Bayesian Optimization, it quickly hones in on the best-performing prompt variation. All this happens behind the scenes in the Optimization Studio, providing you with the top-scoring prompt without needing to manage the complex calculations yourself.

The benefits of low-code MIPROv2 Optimization
By bringing MIPROv2 into a low-code environment, LangWatch’s Optimization Studio makes prompt optimization faster, simpler, and more accessible to teams across industries. Whether you’re working in customer support, content generation, or educational applications, you can:

Quickly adapt prompts to new use cases or requirements without extensive re-coding.

Validate and monitor prompt performance using built-in evaluation functions, ensuring that your LLM’s responses meet real-world demands.

Optimize LLMs without programming knowledge, allowing non-technical team members to participate in the optimization process.

Take the next step with LangWatch’s Optimization Studio
LangWatch’s Optimization Studio is more than just a tool—it’s a gateway to unlocking the potential of your language model through smarter, more efficient prompts. By utilizing DSPy’s MIPROv2 in a low-code setting, you can experience the advantages of prompt optimization without getting bogged down by technical details.

Ready to experience prompt optimization without the coding complexity? Find more here: www.langwatch.ai