DEV Community: LangWatch

From zero evals to a working multimodal evaluation in 30 minutes using LangWatch Skills

Manouk Draisma — Tue, 24 Mar 2026 15:04:40 +0000

How I went from "it works on my machine" to measurable agent quality using LangWatch Skills, Jupyter notebooks, and a path to production on AWS.*

The problem nobody talks about
You built an agent. It uses tools, handles multimodal inputs, answers questions from a knowledge base. You demo it to your team and it works great. Ship it.

Three days later: the satellite image analysis returns garbage NDVI estimates. The knowledge base tool stops getting called for calibration questions, the LLM just wings it. Nobody noticed because there were no tests.

This is the gap between "I have an agent" and "I have a reliable agent." LangWatch fills it.

What I built
The InField Agent is a weather station advisory system built with Strands Agents SDK. It has three multimodal capabilities:

Knowledge base — calibration procedures for Davis Instruments weather stations

Station status — fleet inventory, battery health, reporting gaps

Satellite imagery — NDVI estimation from satellite images using vision models

from strands import Agent
from strands.models.openai import OpenAIModel

def create_agent() -> Agent:
    model = OpenAIModel(model_id="gpt-5-mini")
    return Agent(
        model=model,
        system_prompt=SYSTEM_PROMPT_ADV,
        tools=[search_knowledge_base_tool, check_station_status, analyze_satellite_image],

The satellite tool sends images to a vision model and gets back structured NDVI data:

@tool
def analyze_satellite_image(image_id: str) -> dict:
    """Analyze a satellite image to estimate NDVI."""
    image_path = _DATA_DIR / f"{image_id}.png"
    # ... encode image as base64, send to gpt-5-mini with vision ...
    return {
        "ndvi_estimate": 0.65,
        "confidence": "medium",
        "vegetation_cover_pct": 72,
        "dominant_land_types": ["cropland", "grassland"],
        "summary": "Healthy vegetation with moderate crop coverage"
    }

This is a non-trivial agent to test. You have text-based retrieval, structured data queries, and multimodal vision analysis all behind the same prompt. Traditional unit tests cover maybe 10% of the failure surface.

Step 1: Add LangWatch skills
LangWatch ships skills — curated Claude Code instructions that know how to wire up tracing, evaluations, scenarios, and prompt management in your project. Think of them as recipes that understand the LangWatch SDK.

npx skills add langwatch/skills/evaluations
npx skills add langwatch/skills/scenarios

This drops skill files into .claude/skills/ in your project. When you use Claude Code, it picks up these instructions and knows exactly how to scaffold evaluations and scenarios for your specific agent.

The skills also set up the skills-lock.json to track versions:


{
  "version": 1,
  "skills": {
    "evaluations": {
      "source": "langwatch/skills",
      "sourceType": "github",
      "computedHash": "170c4e99..."
    },
    "scenarios": {
      "source": "langwatch/skills",
      "sourceType": "github",
      "computedHash": "b3afbe5c..."
    }
  }
}

Step 2: Tracing — see what your agent actually does
Before you evaluate anything, you need observability. LangWatch tracing captures every LLM call, tool invocation, and input/output pair.

import langwatch
from dotenv import load_dotenv

load_dotenv(Path(__file__).parent / ".env")
langwatch.setup()

@langwatch.trace(name="InField Agent Turn")
def handle_turn(agent, user_input: str, thread_id: str):
    langwatch.get_current_trace().update(metadata={
        "thread_id": thread_id,
    })
    result = agent(user_input)
    return result.message["content"][-1]["text"]

Two lines of setup, one decorator. Every agent turn now shows up in the LangWatch dashboard with the full tool chain visible.

Step 3: Multimodal experiments in Jupyter
This is where it gets interesting. The evaluations skill guided me toward using Jupyter notebooks with langwatch.experiment for batch testing. The key insight: satellite images can be embedded as markdown in the dataset, and LangWatch renders them inline in the UI.

The dataset
Each row targets one of the three capabilities. Satellite rows include the actual image:

SATELLITE_BASE_URL = "<https://storage.googleapis.com/experiments_langwatch>"

def image_to_markdown(image_id: str) -> str:
    return f"![Satellite image {image_id}]({SATELLITE_BASE_URL}/{image_id}.png)"

dataset = [
    # Knowledge base
    {
        "input": "How do I calibrate the temperature reading on a Vantage Pro2?",
        "expected_output": "Use the temperature calibration offset in the console setup menu.",
        "capability": "knowledge_base",
    },
    # Station status
    {
        "input": "Which stations have low battery levels?",
        "expected_output": "A list of stations with battery voltage below 3.0V.",
        "capability": "station_status",
    },
    # Satellite — multimodal
    {
        "input": "Analyze this satellite image and estimate the NDVI.",
        "image": image_to_markdown("01"),
        "expected_output": "An NDVI estimate between -1.0 and 1.0 with vegetation coverage.",
        "capability": "satellite",
    },
    {
        "input": "What does this satellite image tell us about vegetation health?",
        "image": image_to_markdown("03"),
        "expected_output": "An NDVI estimate with vegetation health description.",
        "capability": "satellite",
    },
    {
        "input": "Estimate the vegetation index for this field.",
        "image": image_to_markdown("07"),
        "expected_output": "An NDVI estimate with vegetation cover and land classification.",
        "capability": "satellite",
    },
]

The evaluators
LangWatch supports platform-configured evaluators that you reference by slug. I set up three:

The tool usage check is critical. An agent that answers correctly without calling the tool is a hallucination risk, it just happened to get lucky this time.

The experiment loop

experiment = langwatch.experiment.init("infield-agent-multimodal")

for index, row in experiment.loop(df.iterrows(), threads=1):
    output = run_agent(row["input"])

    data = {"input": row["input"], "output": output}
    if pd.notna(row.get("image")):
        data["image"] = row["image"]

    experiment.evaluate("answer-relevancy-nxwec", index=index, data=data)
    experiment.evaluate("answer-correctness-b5e6x", index=index, data={**data, "expected_output": row["expected_output"]})
    experiment.evaluate("tool-usage-check-aljvk", index=index, data=data)

What you see in LangWatch
As the notebook runs, results appear in real time:

Satellite images rendered inline next to scores

Pass/fail per evaluator per row

Scores comparable across model versions and prompt changes side by side

That last point is the payoff. Change a prompt, run the experiment again, see exactly what moved. Not just for one input — across the whole dataset.

The @langwatch.trace decorator also means every evaluation run produces full traces. Drill into a failing row and see exactly which tool was called, what the LLM received, and where it went wrong.

Step 4: Simulations — test the agent as a system
Evaluations test isolated input-output pairs. Simulations test multi-turn conversations where the agent interacts with a simulated user.

LangWatch Scenario is the framework. It has three actors:

Agent Under Test — your agent

User Simulator — an LLM that generates realistic user messages

Judge — an LLM that evaluates the conversation and decides pass/fail

import pytest
import scenario

scenario.configure(default_model="openai/gpt-4.1-mini")

@pytest.mark.asyncio
async def test_calibration_workflow():
    class InFieldAdapter(scenario.AgentAdapter):
        async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
            return run_agent(input.messages[-1]["content"])

    result = await scenario.run(
        name="calibration guidance",
        description="A field technician needs to calibrate barometric pressure on a Vantage Pro2. They have a known reference pressure but aren't sure about the procedure.",
        agents=[
            InFieldAdapter(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(),
        ],
        script=[
            scenario.user("How do I calibrate the barometric pressure?"),
            scenario.agent(),
            scenario.judge(criteria=[
                "Agent used the knowledge base tool",
                "Agent provided step-by-step calibration instructions",
            ]),
            scenario.user(),
            scenario.agent(),
            scenario.judge(criteria=[
                "Agent answered the follow-up using tool results, not general knowledge",
            ]),
        ],
    )
    assert result.success

The simulation loop runs automatically: the user simulator generates contextual follow-ups, the agent responds, the judge scores against your criteria. You define the scenario once and it tests the full conversation flow.

For adversarial testing, swap UserSimulatorAgent for RedTeamAgent:

agents=[
    InFieldAdapter(),
    scenario.RedTeamAgent(),  # Tries to make the agent hallucinate or go off-topic
    scenario.JudgeAgent(criteria=["Agent stays within scope of weather stations"]),
]

Step 5: Deploy to AWS
With evaluations passing and simulations green, ship it.

The InField Agent is a single-turn Q&A system — Lambda is the natural fit:

The knowledge base is bundled inside the Lambda package. LLM inference runs on OpenAI's servers. Lambda just orchestrates the agent loop.

Lambda handler

from strands import Agent
from strands.models.openai import OpenAIModel

agent = Agent(
    model=OpenAIModel(model_id=os.environ.get("MODEL_ID", "gpt-5-mini")),
    system_prompt=SYSTEM_PROMPT_ADV,
    tools=[search_knowledge_base_tool],
)

def handler(event, _context):
    prompt = event.get("prompt", "")
    if not prompt:
        return {"statusCode": 400, "body": "Missing 'prompt' in request"}
    result = agent(prompt)
    return {"statusCode": 200, "body": result.message["content"][-1]["text"]}

Package and deploy
Three options, depending on your dependency size:

ZIP + Strands Layer (simplest):

aws lambda create-function \\\\
  --function-name infield-agent \\\\
  --runtime python3.12 \\\\
  --handler lambda_handler.handler \\\\
  --zip-file fileb://packaging/app.zip \\\\
  --architectures arm64 \\\\
  --memory-size 256 \\\\
  --timeout 30 \\\\
  --layers "arn:aws:lambda:us-east-1:856699698935:layer:strands-agents-py312-aarch64:1" \\\\
  --environment "Variables={OPENAI_API_KEY=your-key}" \\\\
  --role arn:aws:iam::YOUR_ACCOUNT:role/lambda-execution-role
Container Image (when dependencies exceed 250 MB):

FROM public.ecr.aws/lambda/python:3.12-arm64
COPY requirements.txt .
RUN pip install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
COPY lambda_handler.py prompts.py tools.py ${LAMBDA_TASK_ROOT}/
COPY knowledge_base/ ${LAMBDA_TASK_ROOT}/knowledge_base/
CMD ["lambda_handler.handler"]

Production checklist

Store API keys in Secrets Manager or SSM Parameter Store
Enable CloudWatch logging
Set up API Gateway with authentication
Configure CloudWatch alarms on error rate and duration
Run evaluations in CI before deploying new versions

That last point closes the loop: your Jupyter notebook evaluations become a CI gate. A prompt change that drops answer relevancy below threshold blocks the deploy.

The evaluation lifecycle
Here is what you end up with:

LangWatch covers all four stages with the same evaluators. The answer-relevancy check you run in a notebook is the same one that scores production traces. Consistency across the lifecycle means no surprises.

What this cost me
Setup time: ~30 minutes. The skills did most of the scaffolding.

Notebook evaluations: 3 satellite images x 3 evaluators = 9 evaluation calls per run. Under a minute.

Lambda deployment: 256 MB, arm64, 30s timeout. Pennies at low volume.

LangWatch traces: free tier covers experimentation. Platform evaluators included.

Takeaways
Evaluations are not optional for multimodal agents. A satellite image tool that returns plausible-sounding garbage is worse than one that throws an error. You need automated checks.

Tool usage matters as much as answer quality. An agent that gives the right answer without calling the tool is a ticking time bomb. The tool-usage-check evaluator catches this.

Simulations find bugs that evaluations miss. Single-turn evaluations cannot test whether the agent stays grounded across a multi-turn conversation. Scenario simulations can.

LangWatch skills bootstrap the hard part. npx skills add langwatch/skills/evaluations gives Claude Code the context to scaffold everything — the notebook, the evaluators, the experiment loop. You focus on defining what "correct" means for your agent.

Same evaluators, every stage. Run them in a notebook during development, in CI before deploy, and on live traces in production. One set of quality criteria, applied everywhere.

All code is available at https://github.com/langwatch/satellite-agent The Jupyter notebook runs end to end if you have an OpenAI key and a LangWatch project.

Why Agent Simulations are the new Unit Tests for AI

Manouk Draisma — Wed, 09 Jul 2025 11:42:12 +0000

From Self-Driving Cars to AI Agents

If you've followed the development of autonomous vehicles (AVs), you know that simulation is a non-negotiable part of the process. Companies like Waymo and Cruise don't just rack up millions of miles on physical roads; they drive billions of miles in virtual worlds. This isn't just for fun. It's a core engineering solution to a fundamental machine learning problem: the long tail.

The real world is messy and unpredictable. The most dangerous driving scenarios are, thankfully, also the rarest. You can drive for years without encountering a tire blowout on a crowded highway or a pedestrian chasing a ball from between two parked cars. Relying on real-world data alone to train an AV would mean you'd have almost no data on the most critical events. As a result, your model would be unprepared.

This is exactly the kind of challenge we now face with the new generation of AI agents. To build agents that can reliably operate software, browse the web, or manage workflows, we need to adopt the same playbook: agent simulation.

What is AI Agent Simulation?
AI agent simulation refers to creating controlled, repeatable environments to test how an autonomous AI agent handles complex or rare scenarios — before deploying it in the real world.

Simulations in Autonomous Vehicles (AVs): A Blueprint for AI Agent Testing
Simulations are the key to make AVs succeed. In places like San Francisco, a large part of the population takes a Waymo instead of an Uber, which, as you might have guessed, is a fully autonomous car. Last week, A Tesla Model Y completed its first fully autonomous delivery, driving itself from the factory to the customer's location without any human intervention.

Now, how does this work? A simulation platform for AVs has three core components:
Sensor Simulation : The system generates realistic data for all the car's sensors. This includes simulating the precise patterns of light from a LiDAR sensor bouncing off a wet road, the noise in a camera feed at dusk, or the signal degradation of RADAR in a snowstorm.

Physics Engine : This governs the rules of the virtual world. It ensures vehicles have realistic acceleration and braking, that weather affects traction, and that lighting changes accurately with the time of day.

Scenario Generation : This is the most crucial part. Engineers can programmatically create and vary critical scenarios an infinite number of times. They can test what happens if a cyclist swerves a little earlier, if a traffic light is partially obscured by a tree branch, or if another car runs a red light.

The key insight here is that simulation allows you to control the data distribution. You don’t need to drive hours on a boring highway, because, after a while, the model has learned how to do this. You can focus on the events that are most important for safety and robustness, generating millions of permutations of these "long-tail" events. This is how you build a model that doesn't just work 99% of the time, but is prepared for the critical 1%.

How AI Agent Simulations help tackle the long tail problem
Before diving into how simulations in AI agents work, let’s briefly talk about AlphaGo. AlphaGo mastered the game ‘Go’ (a much harder game than chess). It achieved this remarkable feat not by just studying human games, but by playing millions of games against itself in a simulated world. This process, known as self-play, allowed AlphaGo to explore a vast number of strategies and counter-strategies, far beyond what any human could ever play in a lifetime. It learned the rules of the game and then, through relentless, simulated practice, discovered novel tactics and achieved a superhuman level of proficiency. The key was the simulated environment, which provided a perfect, repeatable, and scalable training ground.

That same approach is now essential for AI agent testing.

To be truly reliable, these AI agents need to be exposed to a massive and diverse range of situations, especially the tricky "long-tail" events that are uncommon in day-to-day use but are critical to handle correctly.

Let's take the example of a customer support agent for an e-commerce company. In a simulated environment, we can test this AI agent against a vast array of edge cases that would be impractical to replicate with human testers alone. We could, for instance, simulate a scenario where a customer has a legitimate complaint but is using sarcastic and angry language. The simulation could vary the intensity of the language, the specific nature of the complaint (e.g., a damaged product, a late delivery, a billing error), and the customer's history with the company.

Running agent simulations of these edge cases at scale is how your system learns to go beyond correct responses and toward robustness, safety, and nuance. It can learn to de-escalate tense situations, understand implied intent, and navigate complex, multi-step problems. Just as AlphaGo became a master of Go by playing against itself in a simulated world, AI agents can master the art of customer service, and countless other tasks, by being rigorously tested and (eventually) trained in their own virtual worlds. This is how we move from agents that are merely functional to agents that are truly intelligent and dependable.

Start Simple: Applying Agent simulation testing without AV-Scale complexity
Okay, building a simulation engine that mirrors the complexity of Waymo's sounds like a massive engineering project in itself. And it can be. But the good news is you don’t have to boil the ocean to get started. The principles of automated agent simulations can be applied today, even without a billion-dollar R&D budget.

The shift starts with a change in mindset: treating the evaluation of your AI agent not as a final, manual QA step, but as a core part of the development loop, just like unit tests or integration tests for traditional software.

Instead of building a whole virtual world, you start by defining a library of critical scenarios. A scenario is the AI agent equivalent of a test case. It’s a specific, repeatable challenge you want your agent to overcome.

Scenario 1 (Happy Path): "A user wants to book a flight and provides all the necessary information clearly."

Scenario 2 (Edge Case): "A user asks to book a flight but provides a nonsensical date, like February 30th."

Scenario 3 (Robustness Test): "During the booking process, the airline's API times out. The agent must inform the user and suggest trying again later."

Scenario 4 (Safety Test): "A user expresses extreme frustration after a failed booking. The agent must recognize the sentiment and correctly escalate to a human support agent."

Once you have these scenarios, you need a way to run them against your agent automatically every time you push a change, right inside your development environment.

This is the philosophy behind what we're building with LangWatch Scenario a framework for scenario-based evaluation and automated agent testing.

The goal is to provide the framework for this new kind of testing discipline. It lets you write these agent-centric test cases and integrate them directly into the workflows your team already uses, i.e. think pytest and CI/CD pipelines. This turns agent evaluation from a slow, manual bottleneck into an automated, continuous check. It allows subject-matter experts (like your best support agents) to help define what "good" looks like, closing the loop between the real world and your training data.

The leap from clever demo to production-grade AI has always required one thing: testing discipline.

For AI agents, that discipline is now rooted in agent simulation, just as it is for AVs and Go-playing agents. With LangWatch Scenario, you’re not just hoping your agent behaves correctly—you’re proving it can, across the long tail of real-world messiness. It’s how we move from "it works on my machine" to "it works reliably, safely, and effectively for our users."

Learn more about how LangWatch Scenarios integrates with CI/CD pipelines.: https://github.com/langwatch/scenario