DEV Community: Susheem Koul

I built an RPG that teaches Claude Code by making you actually use it

Susheem Koul — Wed, 26 Aug 2026 15:14:07 +0000

Most tool documentation teaches by just telling you things. You read a page about
/model, or hooks, or subagents, nod along, and forget it by the time you'd
actually need it. I wanted something closer to how people actually learn a CLI,
by using it, with something checking whether you did the thing right or not.

So I built claude-quest, a text RPG that runs entirely inside a real
claude session and teaches the Claude Code CLI zero to hero.

the idea isn't new, I borrowed it

This is basically GameShell's philosophy
applied to Claude Code. GameShell teaches Unix shell commands by dropping you
into a real shell wrapped in a themed fake filesystem, and grading your
progress by checking real shell/filesystem state instead of asking you to self
report or answer a quiz question.

claude-quest does the same thing, except the "filesystem" is a real Claude
Code environment. missions live in real sandbox directories, and progress is
checked by inspecting what you actually did, config files you wrote, hooks
that fired, tool calls that happened.

what it actually looks like

There's no separate app, no fancy terminal UI, none of that. You say "let's
play claude quest" inside a normal claude session, and Claude itself becomes
the game master. It reads real output from the engine and narrates on top of
it, something like this:

> lets play claude quest

**The Gatehall**

You've just been let through the outer gate. The tower keeps no secrets
from those who bother to read the walls...

**Mission: First Contact** (Tier 1, Artifact)

Create a CLAUDE.md file in this sandbox recording how to run the
project's tests. Let me know when you're done and I'll check it.

> done

CLAUDE.md records how to run the tests. Claude Code will know next time.
MISSION_STATUS: complete

**What you actually learned:** Claude Code reads a file named CLAUDE.md
in your project root automatically, at the start of every session...

that's it, that's the whole interface. it's just a chat, with real commands
running underneath it and real files getting checked. nothing gets rendered,
nothing gets animated, it's plain text the way claude sessions already look.

missions are graded on real state, not vibes

Every mission is one of four tiers, and the grading gets less mechanical the
further you go:

Artifact, write or edit a file (a CLAUDE.md, a config) and the engine checks the file itself.
Invocation, actually run a claude session and do the thing (switch models, compact, check cost). the engine checks what landed afterward (like .claude/settings.json), never what you said in the session.
Mastery, fix something that's conceptually broken (a skill whose description doesn't say what its for, a subagent that won't fire), then the engine runs a held-out test battery against your own claude CLI.
Judgment, write something graded on a rubric (a commit message, a skill description) and an isolated judge call scores it per criteria.

None of these ask you to type an answer into a prompt, they ask you to leave
a trace and then the engine checks the trace.

try it

Inside any claude session:

/plugin marketplace add susheem-k/claude-quest
/plugin install claude-quest@claude-quest

Then just say "let's play claude quest." Progress saves to ~/.claude-quest/,
so it doesn't really matter which project your in when you pick it back up.

where it's at, not all the levels exist yet

This is a playable alpha, not a finished game. The full loop already works
(characters, saves, hints, grading for all four tiers) but the campaign
doesn't cover everything Claude Code can do yet. Right now there's only a
handful of example missions, one or two per tier, across five arcs. More
missions are getting added.

And since this is open source, that part doesn't have to be just me. If
there's a Claude Code feature you wish more people knew how to use, a
command, a hook, a workflow you had to learn the hard way, I'd genuinely like to turn it into a mission. The mission format is just a directory with a goal, a hint, and a check.js (the design doc has the exact file contract for it), so adding one doesn't require touching
the engine at all.

Open an issue with a
mission idea, or just send a PR with one built already, both are welcome.

if you try it and it teaches you something, a star on the repo genuinely
helps, it's how other people find it.

Repo: https://github.com/susheem-k/claude-quest

From Validating Code to Evaluating Behavior: Rethinking Testing for AI Agents

Susheem Koul — Mon, 17 Aug 2026 17:25:57 +0000

Traditional software testing assumes that system behavior is governed by deterministic code paths and configurations. Once an LLM enters the execution loop, that assumption weakens. The challenge shifts from verifying that code followed the expected path to evaluating whether a system behaved appropriately and achieved the desired outcome.

That distinction fundamentally changes what we test, how we write assertions, and even what "correctness" means. We are moving from verifying deterministic code to evaluating probabilistic system behavior.

Traditional Code Testing and Where It Starts Breaking Down

In traditional software systems, behavior is encapsulated into methods, APIs, configuration, and external dependencies. Even when a system becomes extremely complex, the behavior is still ultimately determined by code.

The output of a request flowing through a service can be largely thought of as a combination of:

Output ≈ Input × Configuration × Infrastructure × Dependencies × Code Version

The resulting state space may be enormous, but it remains largely open to systematic testing against an enumerated set of cases. This is the foundation upon which software testing is built. Cover enough of the state space and you gain confidence that the system will behave correctly.

A Concrete Example

Consider a workflow that reads configuration from a config store, calls a customer profile API, calls a pricing API, applies deterministic business rules, and produces a recommendation. Testing this system is relatively straightforward. We can:

Mock the APIs
Control configuration state
Enumerate edge cases
Verify outputs

Distributed systems introduce their own challenges—network latency, node failures, and partial outages—but established testing techniques such as chaos engineering can provide confidence that the system behaves correctly under those conditions.

Where Things Change with LLMs

Now replace the deterministic business rules with an LLM-driven planner. The APIs, tools, and data remain unchanged, but the workflow must now decide:

Which tools to call
In what order
Whether more information is needed
How to handle ambiguity
How to formulate the final response

The upside is obvious: the system becomes dramatically more flexible and can handle scenarios that were never explicitly programmed for.

The downside is that the testing surface area grows significantly. The behavior driver is no longer only code; it is now a probabilistic reasoning system operating inside the application itself. Even temperature-0 inference does not always guarantee perfectly reproducible outputs in many production LLM deployments due to model updates, infrastructure differences, and implementation details.

The result is a system whose behavior is harder to predict, harder to bound, and therefore harder to test.

The Testing Challenge Shifts

From:

"Did the code execute exactly as expected?"

To:

"Did the system behave appropriately?"

Once we accept that agent behavior cannot always be validated through deterministic assertions, the next question becomes: what exactly are we evaluating?

What Are We Actually Testing?

Agent testing significantly expands what correctness means. Instead of asking whether a function returned the correct value, we ask whether the overall behavior was acceptable. A response can be factually correct while being incomplete, violate instructions, or arrive at the right answer through an inefficient or risky sequence of actions.

As a result, evaluating an agent often means evaluating it across multiple dimensions:

Outcome Metrics

These metrics focus on whether the agent successfully completed the task.

Response Completeness — Did the agent fully complete the task, or only address part of the request?
Response Correctness — Is the answer factually accurate, grounded in the available context, and compliant with instructions?

Operational Metrics

These metrics evaluate how efficiently the agent performed.

Runtime and Latency — How quickly does the agent respond and complete its work?
Cost Efficiency — How many tokens, model calls, and external resources were required to complete the task?

Reliability Metrics

These metrics measure how the system behaves when things do not go according to plan.

Failure Handling — Does the agent recover gracefully from tool failures, timeouts, malformed inputs, and missing data?

Quality Metrics

These metrics focus on the usefulness and presentation of the final output.

Response Quality — Is the output clear, concise, well-structured, and appropriate for the audience?

Governance Metrics

These metrics ensure the system operates within acceptable boundaries.

Security and Safety — Is the agent resilient to prompt injection, jailbreaks, data exfiltration attempts, and other adversarial inputs?

Behavioral Metrics

These metrics evaluate how the agent arrived at its answer, not just the answer itself.

Trajectory Quality — Did the agent follow a sensible path to reach its answer, or waste effort on unnecessary reasoning and tool calls?

The Paradigm Shift

Taken together, these dimensions reveal an important shift:

We are no longer evaluating whether a piece of code returned the expected value. We are evaluating whether an intelligent system behaved effectively across a range of often competing objectives.

How Do We Measure These Metrics?

Once we know what we want to evaluate, the next challenge is figuring out how to measure it.

In traditional software testing, assertions are usually deterministic. We know the expected output ahead of time and can compare actual behavior against it.

assert calculate_tax(100) == 18

Many agent evaluation dimensions do not naturally translate to this style of "hard" testing.

How do we write deterministic assertions for questions such as:

Was the answer complete?
Was the response faithful to the provided context?
Did the agent use retrieved information appropriately?
Was the tone professional?

These are semantic questions rather than computational ones.

As a result, modern AI evaluation frameworks have largely converged on three broad approaches.

1. Mathematical Metrics

Some dimensions can be measured using statistical techniques.

For example, here's an excerpt from the RAGAS metrics documentation on Answer Relevancy:

The Answer Relevancy metric measures how relevant a response is to the user input. It ranges from 0 to 1, with higher scores indicating better alignment with the user input.

An answer is considered relevant if it directly and appropriately addresses the original question. This metric focuses on how well the answer matches the intent of the question, without evaluating factual accuracy. It penalizes answers that are incomplete or include unnecessary details.

This metric is calculated using the user_input and the response as follows:

Generate a set of artificial questions (default is 3) based on the response. These questions are designed to reflect the content of the response.

Compute the cosine similarity between the embedding of the user input (E_u) and the embedding of each generated question (E_q).

Take the average of these cosine similarity scores to get the Answer Relevancy.

Answer Relevancy = (1/N) Σ cosine_similarity(E_q, E_u)

You can find the entire catalog of metrics provided by RAGAS here

2. Specialised Models

Some evaluation dimensions are measured using dedicated machine learning models trained for specific tasks.

Examples include:

Sentiment analysis
PII detection
Bias detection

In these cases, the evaluator itself is a model that has been optimised for a particular classification or scoring task.

3. LLM-as-a-Judge with Pre-defined Rubrics

One of the increasingly common approaches is using another LLM to evaluate the output of the system under test.

Rather than comparing outputs against exact expected values, an evaluator model is asked to assess whether a response satisfies a particular rubric.

Frameworks such as DeepEval and RAGAS package these techniques into reusable metrics that engineers can apply directly within their test suites.

For example, rather than implementing faithfulness scoring from scratch, a framework may expose a pre-built metric:

from deepeval.metrics import FaithfulnessMetric

metric = FaithfulnessMetric()

The result is that engineers can focus on defining what good behavior looks like for their application rather than building evaluation logic from first principles.

Building Metrics on Top of These Approaches

This is an important shift. In traditional software, we primarily wrote assertions against outputs. In agent systems, we increasingly write assertions against behaviours, and those behaviours are often quantified through metrics rather than exact expected values.

Traditional Testing Layers Still Matter

Most traditional testing techniques remain just as important. The difference is that they now validate different parts of the system. An agent is still built on top of code, APIs, databases, configuration, authentication systems, and external dependencies. These deterministic components should continue to be tested using the same techniques that have worked for decades.

Unit tests validate building blocks like API clients, parsers, and retry logic.
Integration tests verify tool contracts and connections to databases or vector stores.
Functional/E2E tests validate outcomes, ensuring the agent achieves goal Z given input X and environment Y.
Chaos and Performance tests ensure the system remains performant and recovers gracefully from tool outages or timeouts.

Traditional testing therefore does not disappear in agent systems. It continues to provide confidence in the deterministic layers of the stack.

Testing Methodologies for AI Agents

Understanding what to measure is only half the problem. The next challenge is designing tests that produce meaningful and repeatable signals.

Define the Test Unit

The first step is deciding what a test actually represents. For agents, a test is usually centered around a scenario or behavior.

A typical test contains:

A scenario definition describing the behavior being evaluated
An input prompt that triggers the scenario
A controlled environment consisting of tool responses, configuration, and datasets
A set of assertions used to evaluate the outcome

Control the Environment

One of the easiest ways to create flaky agent tests is to allow the environment to change between runs.

Tool responses, configuration state, datasets, and external dependencies should be controlled wherever possible. This is no different from traditional testing, but becomes even more important because the LLM itself already introduces variability.

Without environmental control, it becomes difficult to determine whether a failure was caused by the agent or by a changing dependency.

Hard vs Soft Assertions

Some behaviours are non-negotiable and should fail immediately when violated. Examples include:

Invalid schemas
Incorrect computed values
Required tool usage
Missing mandatory fields

These are hard assertions.

Other behaviours are inherently subjective and are often better represented as scores or thresholds. Examples include:

Completeness
Clarity
Tone
Faithfulness
Trajectory quality

These are soft assertions.

In practice, most agent tests combine both.

Account for Variability

Because agent behavior is probabilistic, a single test execution is rarely sufficient. Instead of asking:

"Did the test pass?"

We must ask:

"How consistently does the agent pass?"

Reliability is measured through success rates across multiple runs rather than binary outcomes.

Common Capabilities Across Agent Testing Frameworks

After exploring several agent evaluation frameworks, I found that most of them converge on a similar set of capabilities.

Whether you're looking at DeepEval, RAGAS, LangSmith, Arize or other emerging tools, the implementation details vary but the core ideas remain largely the same.

Most frameworks focus on three major areas:

Capturing and understanding agent behavior
Evaluating that behavior using reusable metrics
Persisting and re-running evaluations over time

Tracing and Observability

Most modern frameworks provide some form of tracing or instrumentation. This enables understanding:

The sequence of tool calls
Retrieved documents
Intermediate decisions
Model interactions

These captured data points provide the raw signals from which many evaluation metrics are derived.

For example, DeepEval allows developers to instrument portions of an application and collect execution traces that can later be inspected or evaluated:

from deepeval.tracing import observe

@observe
def retrieve_documents(query):
    ...

Metrics and Assertions

Once behavior can be observed, the next challenge is evaluating it.

Most frameworks provide a collection of reusable metrics that abstract away the complexity of evaluation. Instead of building custom scoring logic for every use case, teams can compose existing metrics into test assertions.

A typical evaluation for DeepEval, for example, may look like:

from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

metric = FaithfulnessMetric()
test_case = LLMTestCase(
    input="What is Kubernetes?",
    actual_output=response,
    retrieval_context=context
)
metric.measure(test_case)
assert metric.score > 0.8

The important idea here is not the specific metric itself, but the abstraction. Engineers are no longer writing assertions directly against outputs. They are increasingly writing assertions against behavioural qualities.

Test Datasets and Regression Testing

The final pillar is persistence.

Once tests have been defined, they need to be stored, versioned, and re-executed as systems evolve.

Prompt changes, model upgrades, workflow modifications, tool additions, and retrieval improvements can all introduce unexpected regressions. Without a persistent evaluation suite, it becomes difficult to determine whether changes are improving the system or silently degrading it.

A test case becomes a reusable artifact containing:

Inputs
Context
Expected behavior
Evaluation criteria

Once persisted, these datasets become regression suites that can be executed repeatedly throughout development and deployment.

Observability and Self-Learning is the Next Frontier

As I worked through these ideas, one thing became increasingly clear:

Even with comprehensive evaluation datasets, no test suite can realistically capture every scenario an autonomous agent will encounter in the real world.

This is why modern evaluation frameworks place so much emphasis on tracing and metrics. The same signals used to evaluate agents during testing can also be used to understand how they behave in production.

What I find particularly interesting is that these capabilities are not just useful for testing. They are foundational building blocks for systems that can learn from their own behavior.

That's a topic I'll explore in future articles, but it starts with a simple idea:

Before a system can learn from its behavior, we first need to be able to measure it.

Key Takeaways

Traditional software testing breaks down with LLM-driven agents due to probabilistic behavior
Agent evaluation requires measuring across 6+ dimensions (outcome, operational, reliability, quality, governance, behavioral)
Three primary measurement approaches: mathematical metrics, specialized models, and LLM-as-a-Judge
Traditional testing remains critical for deterministic system layers
Modern frameworks converge on tracing, metrics, and persistent test datasets
The future lies in observability-driven development where testing metrics inform production behavior

Originally published on Substack

The promise of AI agents is flexibility. They can solve problems we never anticipated. The challenge is control. Traditional testing can't capture every scenario an agent will encounter. That's why modern evaluation frameworks are moving toward observability: continuous measurement of agent behavior across outcome, reliability, cost, and safety dimensions.

We're building Agentplane to provide the infrastructure layer for this shift. TokenOps is our FinOps control plane that enforces token budgets and governance policies across agent delegation trees, giving you real-time cost attribution and risk enforcement. Chronicle captures execution traces and behavioral signals from every agent run, turning observability data into actionable insights for improvement and compliance.

Together, they close the gap between testing and production: measure your agents during development, enforce policies at runtime, and learn from real-world behavior.

How coding agents like Cursor quietly cut input costs by reusing KV states across turns — and what actually breaks the cache

Susheem Koul — Thu, 30 Jul 2026 19:11:26 +0000

Why my Cursor bill looked weird

I was poking around my usage dashboard in Cursor and noticed a metric I'd never paid attention to before: Cache Read.

Date            Type       Model    Tokens
May 2, 06:50 PM  Included   auto     898K   → Cache Read
May 2, 06:49 PM  Included   auto     1.4M   → Cache Write
May 2, 06:47 PM  Included   auto     346.8K → Input / Output

Almost 99% of my input tokens for that session were served from cache. That sent me down a rabbit hole on what prompt caching actually is, and why it matters so much for agentic coding tools specifically. Here's what I found.

What prompt caching actually is

Agentic coding tools ship a lot of context on every single call — system instructions, workspace rules, file contents, conversation history, the works. LLMs are stateless by default, so naively, all of that gets reprocessed from scratch on every request.

Prompt caching short-circuits this. It saves the intermediate computation from the transformer's prefill stage — specifically the Key/Value states — so that when a later request shares the same leading sequence of tokens, the model reuses those states instead of recomputing them.

One important caveat: this only applies to input tokens. Output generation is still sequential, token-by-token, and gets no speedup from caching.

How the payload is actually built

Coding agents don't just fire your question at the model. They assemble a large composite prompt, and — critically — they structure it with the most static content first and the most volatile content last:

<system_instructions> You are an expert coding assistant... </system_instructions>
<project_rules> Use functional React components... </project_rules>
<file_context_1> ... </file_context_1>
<file_context_2> ... </file_context_2>
<conversation_history> ... </conversation_history>
<user_input> Refactor this function to handle edge cases. </user_input>

Once that sequence hits the model, the transformer tokenizes it and starts computing attention. For every token, it derives a Key (K) and a Value (V) — essentially a map of how that token relates to every other token before it. This is the expensive part: attention computation scales quadratically with sequence length, so a big context window (lots of files, long history) gets disproportionately costly to prefill.

The model doesn't refer back to your raw text when generating a response — it refers to these computed KV matrices. That's the artifact caching actually preserves.

Providers typically charge less for input tokens than output tokens, but agentic coding tools push such enormous input payloads that shaving cost off the input side is still very much worth doing.

Turn 1 vs. every turn after

Here's the flow, roughly:

Turn 1 (cold start):

Build the prompt: system instructions + project rules + file context + your first message.
No cache exists yet → full prefill. The model computes KV states for every token (the expensive O(n²) pass).
Those KV states get written to cache storage.
Generation proceeds normally.

Turn 2...N (warm):

Build the new prompt: same system instructions + same files + prior conversation + your new message.
Check for a prefix cache hit against system + context.
Hit: read the cached KV states for the unchanged prefix, and only compute new KV states for the incremental conversation turn + your latest input.
Concatenate cached + new KV states, then generate.

So instead of re-running the full O(n²) prefill on every message, only the marginal new tokens get freshly processed. That's the whole trick, and it's why long coding sessions don't get proportionally slower and more expensive turn after turn.

Most major providers now do this optimization implicitly under the hood. Google's Vertex AI docs on context caching are a good primary-source reference if you want to see how one provider exposes this explicitly.

What breaks the cache

Caching depends entirely on prefix stability. The moment a token diverges from what's cached, everything downstream of that point has to be recomputed. In practice, here's what kills your hit rate:

Dynamic injections — a timestamp or other changing value stuffed near the top of the prompt invalidates the prefix on every single request.
Context reordering — shuffling the order in which files or context blocks get loaded prevents prefix matching entirely.
Upstream file edits — editing a file that sits early in the prompt structure invalidates the cache for that file and everything that comes after it in the sequence.

Most systems rely on exact token-prefix matches, though some providers (Google's docs linked above, for instance) expose more explicit cache handles or partial-reuse mechanisms.

The part that confused me: why appending doesn't break the cache

This is the part that seemed contradictory at first. If attention connects every token to every other token, shouldn't adding a new message at the end of the prompt ripple backward and change how the earlier tokens (system instructions, file context) get attended to?

It would — if the model attended in both directions. But modern generative LLMs are decoder-only and use causal (masked) self-attention: a token can only attend to tokens that came before it, never after.

That one architectural detail is what makes caching possible at all. We're not caching a fragile, bidirectional global attention matrix — we're caching the KV tensors of the prefix. Those K and V states for the early context are computed once and never change, no matter what you append afterward. New tokens just generate their own Queries and attend against the already-cached Keys of everything before them.

Where this goes next

Prompt caching solves the "don't recompute what hasn't changed" problem. It doesn't solve the separate problem of what to put in the context window in the first place. Instead of stuffing every file into the prompt and hoping for a cache hit, coding agents increasingly rely on retrieval — RAG-style embeddings over the codebase — to selectively pull in only the relevant slices of code.

That's next on my list to dig into: how coding tools build queryable views of a codebase with embeddings, and what that does to the cost equation.

If you want the deeper transformer-architecture background behind KV caching and attention, 3Blue1Brown's "Large Language Models explained briefly" is a genuinely excellent, low-jargon primer.

Originally posted on Susheem's Jottings — cross-posting here for the dev.to crowd.