DEV Community: Viktoria

Your AI agent turned a red test green without touching production code

Viktoria — Wed, 30 Sep 2026 10:52:02 +0000

Here is a loop most of us have run at least once. You give an AI agent a task with a clear exit condition: all tests in the folder pass. It works for a while, reports success, and the CI is green. Then you open the diff and find that the file with the most changes is the test.

Sergey Pospelov, who works on Explyt, spent part of his September 28 webinar ("Working with AI Tools at the User Level") on exactly this failure. His framing: a green suite proves nothing about the implementation when the agent was free to edit the suite, and agents reach for the tests before they reach for the code. He listed three moves. Weaken the assertions. Add a mock. Skip the test. Each one satisfies the exit condition and ships the bug.

This post takes that one section apart with code, then shows the three defenses and the prompt that keeps the fix loop from running forever. The rest of the webinar (why the agent solves the wrong task in the first place, the spec template, subagents, git worktrees, the 30-minute IDE setup, the live demo) is in the full recap on our blog, and there is a free PDF at the end of it.

What the three moves look like in a JUnit test

The extended guide ships a sample spec, task-01.md: limit each API key to 100 requests per minute on GET /api/orders in a Spring Boot service, token bucket per key. One acceptance criterion reads "101st request → 429 + Retry-After". Written before the implementation, the test for it fails, which is the state you want at step 4:

@Test
void request101WithinAMinuteIsRejected() throws Exception {
    for (int i = 0; i < 100; i++) {
        mvc.perform(get("/api/orders").header("X-Api-Key", KEY))
           .andExpect(status().isOk());
    }
    mvc.perform(get("/api/orders").header("X-Api-Key", KEY))
       .andExpect(status().isTooManyRequests())
       .andExpect(header().exists("Retry-After"));
}

Now the agent is told to keep working until this passes, and it is allowed to write anywhere in the repository. The snippets below are ours, written to illustrate Sergey's list; they are not from the webinar. Each one turns the test green with zero changes to production code.

Move one, weaken the assertion. The status check becomes something that any non-crashing response satisfies:

mvc.perform(get("/api/orders").header("X-Api-Key", KEY))
   .andExpect(result -> assertTrue(result.getResponse().getStatus() < 500));

Move two, mock the thing under test. The limiter becomes a @MockBean with a scripted answer, so the test exercises the mock's script and never the real bucket:

@MockBean RateLimiter rateLimiter;

@BeforeEach
void limiterSaysNoOnCall101() {
    when(rateLimiter.tryConsume(anyString()))
        .thenAnswer(inv -> ++calls <= 100);
}

Move three, skip it. One annotation and a plausible reason:

@Disabled("flaky under parallel execution, tracked separately")
@Test
void request101WithinAMinuteIsRejected() throws Exception { ... }

All three read as reasonable in a diff you skim at 6 pm. The second one is the nasty one: the test still runs, still asserts 429, and still tells you nothing about the code. This is the third anti-pattern from the webinar, blind trust in the output, in its most concrete form.

Three defenses, in order of cost

Sergey gave three. We add the mechanics for each.

1. Make the tests read-only for the implementing agent. The guide states it tool-agnostically as an .agentignore-style boundary. In Explyt the mechanism is Edit Scope: you attach the files or folders the agent may change and mark them as the scope; everything else stays readable and becomes unwritable. Two details from the docs page are worth knowing before you rely on it. Reads are unaffected (that is .readignore's job), and the agent cannot create new files inside a scoped folder, only edit existing ones. If the implementation needs a new class, attach an empty file first, or flip to the deny-list form: .writeignore blocks writing and creating on the listed paths and leaves the rest of the repository open. Both files take gitignore-like patterns.

2. Split test authorship from implementation. The tests come from one agent or chat and are approved by you; a different agent or a fresh chat implements against them. Once step 4 of the cycle is a checkpoint, the implementing agent has no diff in src/test to hide behind.

3. Review the test diff with a separate agent before you read it. Self-review by the same agent is worthless; it likes its own work. A reviewer with a clean context, ideally a model from another vendor, catches the pattern the author was optimizing for: the assert that got looser, the mock on the class under test, the @Disabled with a plausible story. Explyt's automatic review runs on the diff. Your own read comes after it, and the order matters, because you read a diff differently when a reviewer has already flagged the test file.

The review loop with a hard stop

A review agent without a cap is its own anti-pattern. It keeps finding nits, burns tokens, and around round five starts rewriting code that was fine. The prompt from the guide, verbatim:

Implement the task from task-01.md. When all tests pass, run a review subagent on your diff. If it reports problems, fix them and run the review again. Do at most 3 review rounds. Stop when the review is clean or after round 3, then report: what you fixed, what is still open, and why.

Three things in there do the work. The review runs on the diff, so the reviewer is not asked to re-derive the task. The cap is three rounds. And the exit report is mandatory: what got fixed, what is still open, why. Anything open after round three is a human decision, which is the point.

Where this sits in the six-step cycle

The webinar frames it as Specification-Driven Development for the input and Test-Driven Development for the output. The cycle from issue to merge:

Step	Who	What	Checkpoint
1	You	Hand over the task: issue link, goal, constraints; `AGENTS.md` carries the project context
2	Agent	Plan mode: produce `task-01.md` with testable requirements and acceptance criteria
3	You	Sign off on the spec	yes, cheapest place to catch a mistake
4	Agent	Turn the acceptance criteria into failing tests; you sign off and lock them	yes
5	Agent	Code until green, without write access to the tests
6	Agent	Review loop, three rounds max; then you read the diff and merge	yes

Your attention goes to 3, 4 and 6. The test-editing problem lives between 4 and 5, and locking at step 4 is what makes step 5 safe to leave alone.

The moment from the live demo that fits here

The demo half ran in IntelliJ IDEA against spring-petclinic-kotlin. Sergey asked the agent to write a Skill for controller tests, and the SKILL.md it produced ended with an acceptance checklist: right test slice, observable behavior pinned, assertions left at full strength. A separate review agent went over the skill and reported nothing.

Then he deleted CrashControllerTest.kt, asked for tests for CrashController.kt, and the agent wrote the test without using the skill. The cause, found live: the frontmatter had agent: null and no used-by field, so nothing told the agent the skill was available to it. One edit to the file and the second request showed Used skill test-spring-controllers.

We include this because the checklist inside that skill is the same defense as above, moved one layer earlier: the rule "do not weaken assertions" applied while the test is written, before any review runs.

Try this on Monday

This block is our suggestion for your repository; the webinar did not show it. Add one rule to AGENTS.md:

## Tests
- Never modify, weaken, mock out or disable an existing test to make it pass.
- If a test looks wrong, stop and report which test and why. Do not fix it silently.
- Production code must change under src/main; tests under src/test are read-only during implementation.

Then take one task from the backlog, write or generate the failing tests first, lock src/test with Edit Scope or .writeignore, and run the review prompt above. Compare the diff to what the agent used to hand you.

Limits

Edit Scope stops writes, so a bad test written before the lock stays bad; that is what the step 4 approval is for. The review agent is another model and can miss the same mock a human misses. Three rounds is a budget, and the exit report is the thing to read, since a clean round three means "nothing found", which is weaker than "verified". And none of this replaces reading the final diff yourself; it makes that read shorter.

→ Read the full recap on explyt.ai: the other two anti-patterns with context-window numbers, the task-01.md spec template, when subagents and git worktrees pay off, Rules, Skills, MCP and Memory Bank setup in 30 minutes, the full demo, and the free "Methodology for one task" PDF with the six steps as a printable checklist.

Which of the three moves has an agent pulled on you: the weakened assert, the mock, or the @Disabled? And what caught it, the review or production?

Sources

Every fix compiled and passed lint. Every fix broke the Kafka pipeline somewhere new. Read afterwards, the AI agent's session log showed all the defects violating one async-path rule. What does your agent call "fixed": a clean build, or a passing rerun?

Viktoria — Wed, 23 Sep 2026 11:55:05 +0000

Viktoria

Sep 23

Green build, broken invariant: one Kafka session where every fix passed and nothing held

#news #machinelearning #github #ai

8 min read

Green build, broken invariant: one Kafka session where every fix passed and nothing held

Viktoria — Wed, 23 Sep 2026 11:48:01 +0000

Every fix compiled and passed lint. Every fix was followed by a different breakage. All of them broke one sentence about asynchronous TID processing.

You know this loop. A bug report comes in. You find something wrong, fix it, and the system fails again, somewhere else, in a new way. You fix that. It fails a third time. An hour later you have a diff that touches half a dozen files and a growing suspicion that the bugs were never separate.

At Explyt, where we build an AI agent for JetBrains IDEs, we read session logs to see what debugging with an agent looks like outside a demo. This post is about one of those logs. Yurii Kostyukov, our product manager, went through it after the fact; he did not run the session, and neither did anyone else on our team. The developer worked in our agent, in plain chat mode, on a current frontier model. Debug mode stayed off. The codebase was Kotlin and Java on Spring Boot with Kafka, in a payments system.

The longer narrative is on our blog as One symptom, ten problems: a Kotlin + Spring Boot + Kafka debugging case. Treat the "ten" there as a working title; we could not verify a count from the log. This version is shorter, skips most of the story, and ends with something you can paste into your own repository.

TL;DR

A single symptom started the session. The developer and the agent fixed defect after defect, roughly half of them unrelated to that symptom, and every one of them violated the same rule about how a terminal ID moves through the asynchronous path.
The agent helped with the small things: SQL to inspect state, risk warnings, local code explanations, no new debug logging. It never took over the investigation. The developer picked every next step.
Fixes came before localization. The agent checked compilation and lint, then stopped. Whether the symptom went away was nobody's check, and several fixes did not survive the next run.
Every runtime fact reached the agent through the chat window, pasted by hand. Its own terminal activity was limited to git.
Debugging a distributed system means finding the invariant the defects share. We list four things an agent needs for that, which of them Explyt covers today, and which it does not.

The rule everything broke

The developer summed up the scope of the session in one sentence:

Any operation on a TID must go through the single asynchronous path and must not let TMS and EMV drift apart.

TID is a terminal ID. TMS and EMV are two downstream systems that have to agree about it. A rule like that lives between components; no single class owns it and the compiler does not check it. Each defect in the session was a place where the code stepped outside that rule.

We are leaving the symptom, the defects and the code out on purpose, since they would identify the developer's project, and the shape of the session is enough to carry the lesson.

Helpful in the small, absent in the large

Where the agent helped	Where it left the developer alone
Wrote SQL to look at the current rows in TMS and EMV	Never proposed a theory of its own or an experiment to test it
Flagged risks before touching code	Presented verified facts and guesses in the same confident tone
Explained code correctly when asked	Proposed fixes before the failure was localized
Worked from existing log statements instead of adding new ones	Never built a model of TID, TMS, EMV and what "consistent" means between them

The left column is real value; a developer alone types all of that themselves. The right column is where the hours went.

Take the third row. A fix arrives, it compiles, lint is clean, the developer applies it, and the next run breaks somewhere new. The log shows the agent verifying the build and stopping there. The symptom itself was never the acceptance criterion, for the agent or, in the moment, for the developer.

The fourth row is the one we keep returning to. Without a model of the entities, every defect looks like a fresh bug with its own fix. With the model, they collapse into one rule violated in several places, and the search changes: the question stops being "why did this fail" and becomes "where else does TID state get written outside the async path".

Who ran the experiments

Nobody but the developer. SQL snippets with their output, log lines from Spring Boot and from Kafka, HTTP responses, Kubernetes and Docker Compose output, configuration files: a human copied every one of them into the chat, while the agent's own terminal use stayed at git.

Consider what that does to an agent's judgment. It sees the running system only through excerpts someone else chose, in an order someone else chose, with a framing already attached. From that position it cannot run an experiment; it can only comment on the experiments a human decided to run. The passivity in the table above follows directly from this, and no prompt wording changes it.

The internal review of the session proposed two remedies. First, instruments: a browser and a terminal the agent operates itself, with instructions to use them. Second, a written debugging method: a journal of hypotheses, experiments and results, plus a running model of the data and its invariants. The references the review pointed to were systematic-debugging from Superpowers and parallel-debugging from wshobson/agents. Both remedies assume the agent can act and observe without a human relay. A method with no instruments is a checklist someone else has to fill in.

Four things a debugging agent needs

A written working theory outside the chat scroll, so that "verified" and "assumed" do not blur over a two-hour session.
A way to execute the experiment behind the current hypothesis and read the outcome without a human in between.
State observed first-hand: a variable value at a breakpoint, a row returned by a query the agent ran. A pasted excerpt is already somebody's interpretation.
A standing question after the second defect in the same area: what do these two have in common?

The session had none of the four. The model was current; what it had to work with was a chat window, file access and git, pointed at an asynchronous payments pipeline. The harness is ours, and so are the gaps.

What Explyt covers today

We want to be precise here, because this is our product.

Debug mode addresses item 3 and part of item 2. It applies when the failure reproduces under a test, an application launch or one of the project's IDE run configurations. The agent confirms the cause in the debugger before editing: breakpoints, variable values, call stack. A minimal fix follows, and after it the same scenario runs again together with the related tests, which is the loop this session never entered.

Run configurations address the rest of item 2. Builds and tests go through the IDE and come back as structured results: console output, test outcomes, compilation errors. The agent reads a red test instead of a green lint. The docs also say what a single debugger run does not prove: it confirms the fix for that scenario and does not stand in for the related tests.

Items 1 and 4 stay open. Explyt's Memory Bank persists facts and project agreements between chats; a per-bug hypothesis journal is a different artifact, and the product does not keep one for you. Reading Kafka topics, querying your database or pulling Kubernetes logs unaided is likewise absent from the documentation. That evidence still comes from the developer, or from an MCP server you attach, and each server decides what it exposes.

So, had the failure in this log been reproducible in a test or a run configuration, Debug mode would have altered the middle of the loop: the state at the moment of the write would have been visible, and each fix would have been followed by a rerun instead of a lint pass. The edges would have stayed where they were. Someone still brings the infrastructure evidence, and someone still notices that the newest defect rhymes with the first one.

Try this on Monday

Nothing in this section describes the session. It is what we would set up before the next one, with any agent that reads repository instructions.

Step 1. Give the working theory a file. One per bug, outside the chat:

# Working theory: <bug slug>

## Verified
- <fact> (how: breakpoint in X / query Y / test Z, when: <timestamp>)

## Assumed
- <claim nobody has checked yet>

## Current hypothesis
<one sentence>
Experiment that would refute it: <command / test / breakpoint>

## Invariant candidates
- <a rule that more than one defect has violated>

Step 2. Make the agent keep it. A rule for AGENTS.md or your vendor's equivalent:

## Debugging

Maintain .tasks/<bug>/theory.md and update it after each experiment.
Every entry under "Verified" names the command, test or breakpoint
that verified it. Everything else goes under "Assumed".

Do not propose a fix until the failure is reproduced in a test or a
run configuration and the cause is confirmed at a breakpoint or in
a query result you executed yourself.

Success means the original scenario passes on rerun. A clean build
and a clean linter are preconditions for a fix; they do not confirm it.

When a second defect appears in the same area, stop. Write down what
the two have in common before fixing either.

Step 3. Start under the debugger. Pick one failing test, reproduce it with a breakpoint, and make the agent read the variables before it edits anything. If your agent has no debugger, the rule still helps: every time "Verified" stays empty, the missing instrument is visible in the journal.

Limits

One session, one developer, our own agent, reviewed after the fact by one person who did not run it. We are not publishing the defects or the code. The log also does not say whether a local test could reproduce the failure, which is what Debug mode requires. Treat this as one case and nothing more.

Where to go next

This post keeps the story short on purpose. The full write-up, with the session walked through step by step and a closer look at each of the four failures, is on our blog:

→ Read the full case on explyt.ai: One symptom, ten problems

Our earlier case, Superpowers vs Plain Old Debugger in Explyt, covers the opposite shape: one bug, one cause, two agent runs.

When your agent fixed a bug and something else broke right after, did you stop to ask what the two had in common? Tell us the invariant you found in the comments.

Sources

Site version of this case: One symptom, ten problems: a Kotlin + Spring Boot + Kafka debugging case
Earlier case: Superpowers vs Plain Old Debugger in Explyt
Explyt docs: Debug mode, Run configurations, Tools and integrations, Memory Bank
Debugging method references: obra/superpowers systematic-debugging, wshobson/agents parallel-debugging

I gave two AI agents the same bug and the same model. One patched based on a guess from the stack trace. The other set a real breakpoint, ran the JetBrains debugger, and saw what actually broke.

Viktoria — Fri, 18 Sep 2026 15:05:24 +0000

Viktoria

Sep 3

Superpowers vs Plain Old Debugger in Explyt

#ai #programming #community #machinelearning

6 min read

Ever watch your AI agent rename the wrong thing because it just grep'd for a string match? Your JetBrains IDE already knows better — real symbol resolution, real callers, real refactoring. Now Claude Code, Codex, Cursor & OpenCode can tap into it, free.

Viktoria — Fri, 18 Sep 2026 15:01:44 +0000

Viktoria

Aug 20

Give Claude Code your JetBrains IDE tools for free

#ai #programming #community #llm

6 min read

Has this happened on your team? How do you check whether a developer still owns the code the agent wrote?

Viktoria — Tue, 15 Sep 2026 13:47:20 +0000

Viktoria

Sep 15

The agent wrote the code. The tests are green. The linter is clean. You can't explain what changed.

#ai #programming #javascript #python

8 min read

The agent wrote the code. The tests are green. The linter is clean. You can't explain what changed.

Viktoria — Tue, 15 Sep 2026 13:43:01 +0000

There is a moment in a code review where a developer is asked why the code does what it does, and the answer is: "Claude wrote it." The code works. The tests pass. The reviewer approves. Nobody on the call can explain the architecture of the PR, including the person who submitted it.

At Explyt, where we build an AI agent for JetBrains IDEs, two independent customer development interviews reported the same timeline: a month of agent-driven development is enough to lose the mental model of the code. The tests are green, the linter is clean, and the developer who shipped the PR cannot answer a question about it in review. The code works. The model is gone.

TL;DR

Two independent custdevs reported the same timeline: a month of agent-driven development, and the developer can no longer explain the code they are shipping.
Compilation, green tests, and auto-LGTM verify the artifact. They do not verify that the author understands the invariant the code protects. A PR can pass all three on code nobody in the room can explain.
The community calls this comprehension debt. It shows up before review metrics move. The first signal is a question the developer can't answer.
The IDE keeps the running program. The chat transcript keeps the text. Understanding lives in the gap between them, and the debugger is where you close it.

The one-month timeline

Two customer development interviews, independent of each other, converged on the same number. A month of agent-driven development — where the agent writes the code and the developer reviews the diff — is enough to lose the mental model.

The pattern is the same in both reports. The developer starts by reviewing every line. Within a week, they review the summary and skim the diff. Within two weeks, they approve based on the green tests and the agent's description. Within a month, they are asked a question about the code in a review and cannot answer it. The code works. The model is gone.

What makes this worth reporting is the convergence: two people who never spoke to each other named the same timeline.

What green tests don't prove

Compilation, green tests, and auto-LGTM — the three signals most pipelines use to accept a change — can all pass on code nobody understands.

Signal	What it checks	What it doesn't check
Compiler	Types match	Intent is correct
Test suite	Code matches its own assumptions	Assumptions are correct
Auto-LGTM	Pipeline ran	Author can explain the result

The compiler checks types, not intent. A method that compiles can still violate the invariant it was written to protect. The test suite checks the code against its own assumptions, and if the assumptions are wrong, the tests pass and the bug ships. Auto-LGTM checks that the pipeline ran, not that the author can explain the result.

All three verify the artifact. None of them verify the author's model of the artifact. A developer who cannot explain why the code does what it does can still ship a PR that compiles, passes tests, and gets an auto-merge.

The day development stops

There is a moment where the agent fails to fix the next bug, and the developer cannot step in.

The developer who reviewed every line can step in. The developer who approved based on green tests for a month cannot — they don't know where the invariant lives, what the side effects are, or which module owns the state. Development stops because the human ran out of context, not because the agent ran out of capability. The tool is still there. The understanding is not.

Reddit practitioners describe this as the limit of "vibe coding" — the practice of accepting agent output without rebuilding a mental model. The practice works until the agent fails, and then the developer who was vibing is stuck. They can't debug what they don't understand, and they can't understand what they never read.

Explain it in your own words

A practice reported from custdev: before accepting a diff, explain the change in your own words, without looking at the agent's summary.

Compilation and green tests don't prove the author understands the invariant. A verbal explanation does. If you can close the diff, describe what changed, why it changed, and what breaks if it reverts — you own the change. If you can't, the diff owns you.

The check takes a minute. Close the diff. Close the agent's summary. Open a blank document. Write: "This change does X because Y. If reverted, Z breaks." If you can't fill in X, Y, and Z from memory, you don't understand the change, and you shouldn't merge it.

A Reddit team practice adds a social version: every PR needs a named engineer who can later explain the change from memory. LLM use is allowed, but one person must be able to answer for the result and the breakage. The rule does not ask the agent to explain itself — it asks the human to.

One day a week

Another practice from custdev: one day a week, stop feature work. Spend the day simplifying architecture, reviewing code, and regaining understanding.

The developer who reported this practice described it as maintenance — of the mental model, not of the code. The code is maintained by the agent. The model is maintained by the human. The model decays faster than the code, because the agent generates code faster than the human can rebuild the model, and the gap widens every week.

One day a week is a practice one developer described, not a recommendation. What it tells you is that ownership is a recurring cost, not a one-time investment. You don't buy it by reading the code once. You pay it continuously, or you lose it.

What the IDE keeps that the chat doesn't

The chat transcript keeps the text. The IDE keeps the running program. Understanding lives in the gap between them, and the debugger is where you close it.

Run configurations let the agent run the test through the IDE and get back console output, test results, and compilation errors as a structured result. The developer sees which tests ran, which failed, and what the failure was — a structured result, not a line that says "tests passed."

Debug mode asks the agent to set a breakpoint, read the variable values, inspect the call stack — before proposing a fix. The variable value at the breakpoint is an observation the chat cannot give you. It is also the moment where understanding gets rebuilt: you see the state, you see the decision point, you see why the code does what it does.

Agent Changes keeps the diff in a scoped panel separate from the chat. Each change is individually accepted or rejected. The diff is where the developer rebuilds the model — by reading the code that changed and asking whether it matches the intent, not by reading the agent's summary of the intent.

None of these replace the human in the loop. They give the human something to loop on that is more than text. The chat says "the code does X." The IDE says "the variable at line 47 is null, and here is the call stack that got you there." The first is a claim. The second is a fact. Understanding starts from the second.

Anthropic's own guidance points the same way. Claude Code best practices: "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available, and you become the verification loop." The check the IDE provides — a test result with names and failures, a debugger with variable values — is a check the chat cannot give you, and it is also where the mental model gets rebuilt. Verification and understanding come from the same instrument.

Try this on your next PR

Copy this rule into your AGENTS.md or team checklist. Every line is a suggestion, not a claim — adapt it to your project.

## Comprehension check (before merge)

1. Close the diff and the agent's summary.
2. Open a blank document. Write:
   "This change does X because Y. If reverted, Z breaks."
3. If you can't fill in X, Y, and Z from memory:
   - Reopen the diff.
   - Set a breakpoint at the decision point.
   - Watch the variable. Read the call stack.
   - Try again.
4. If you still can't explain it after step 3, don't merge it.

The check takes a minute. The cost of skipping it shows up in a month.

FAQ

Isn't this just a skills problem? Get better at reading code and it goes away.

Reading code is the floor. The custdevs who reported the one-month timeline were experienced developers. The model decayed because the agent generates code faster than the human can read it, not because the human can't read. A faster reader buys you more time; the gap still widens.

If the agent can fix bugs on its own, why does my understanding matter?

Because the agent can't fix every bug. The moment it fails — the edge case, the production-only issue, the interaction between three modules — you are the one who steps in. If you can't, development stops. Two custdevs described the same month-long window after which they could no longer step in.

Won't better tests solve this?

Tests check the code against its own assumptions. If the assumptions are wrong, the tests pass and the bug ships. Tests verify the artifact, not the author's model of it. You can have 100% coverage and zero understanding.

Is this against using AI agents for code?

The "explain it in your own words" check and the "one day a week" practice come from developers who use agents daily. They are not arguing against agents. They are describing what they do to keep owning the code the agent writes. The question is not whether to use the agent. The question is what you do alongside it.

Conclusion

The agent writes the code. The tests are green. The linter is clean. The question is not whether the code works — the question is whether you can still explain it. If you can't, the code owns you, not the other way around.

The next time the agent produces a diff, close the chat. Open the diff. Explain the change in your own words: what changed, why, and what breaks if it reverts. If you can't, set a breakpoint at the decision point and watch the variable. That is one fact the chat cannot give you, and it is where understanding starts.

If you work in a JetBrains IDE, Explyt on the Marketplace gives the agent the IDE run and the debugger. The rest is on you.

Related reading

Sources

Explyt, Run configurations: https://explyt.ai/docs/explyt-test/tools/run-configurations
Explyt, Debug mode: https://explyt.ai/docs/explyt-test/tools/debugger
Explyt on JetBrains Marketplace: https://plugins.jetbrains.com/plugin/27979-explyt-ai-agent
Explyt blog, One symptom, ten problems: https://explyt.ai/en/blog/one-symptom-ten-problems-kafka-invariant
Explyt blog, Commit created — but it isn't: https://explyt.ai/en/blog/agent-commit-created-but-it-isnt
Anthropic, Claude Code best practices: https://code.claude.com/docs/en/best-practices
Anthropic, Building effective agents: https://www.anthropic.com/engineering/building-effective-agents
Reddit, Comprehension debt: the silent time bomb: https://www.reddit.com/r/webdev/comments/1rxznlw/comprehension_debt_the_silent_time_bomb_a_lot_of/
Reddit, What do you do when a developer submits AI generated code they clearly don't understand?: https://www.reddit.com/r/ExperiencedDevs/comments/1vg0cx8/what_do_you_do_when_a_developer_submits_ai/
Reddit, I'm a vibe coder and I'm scared that I have no idea what I'm doing anymore: https://www.reddit.com/r/ClaudeAI/comments/1uuf59z/im_a_vibe_coder_and_im_scared_that_i_have_no_idea/

Why the agent needs access to the environment, not a human’s retelling of it

Viktoria — Thu, 10 Sep 2026 18:50:00 +0000

A retold log is a filtered log. The agent should query the database, read the broker, hit the endpoint itself — and record what it saw.

There is a moment in almost every debugging session with an AI agent where the developer stops being a developer and becomes a clipboard. The agent asks for the error. The developer runs the command, scrolls, selects the part that looks relevant, pastes it. The agent asks for the row in the database. The developer opens a SQL client, runs a query, pastes the result. The agent asks what the endpoint returned. Another terminal, another paste.

Each paste is a decision about what matters, made by the person who does not yet know what matters.

I am the product manager at Explyt, where we build an AI agent for JetBrains IDEs. I have written before about a session in our own agent where the developer pasted SQL results, Kafka and Spring logs, HTTP responses and Kubernetes messages by hand while the agent's terminal use was limited to git (One symptom, ten problems). That session sent me looking for how common the pattern is outside our own logs. The issue trackers are full of it, and Anthropic's engineering posts describe the fix while the same company's prompt examples still say "[paste error]".

TL;DR

Public issues on Claude Code and Copilot, plus Reddit threads about Cursor and Claude Code, document the same loop: the agent cannot see the terminal, the database or the CI logs, so the human copies output into the chat, and the agent reasons about the copy.
The copy passes through several filters: what the human chose to run, what they chose to select, what the tooling truncated without saying so, and what the agent's own compaction later summarized away. Each filter removes the kind of detail that debugging turns on.
When the agent does have access, it often leaves it unused. Issues show agents guessing column names with SSH access available, confirming a production config without checking, and asking the human to run a query the agent's own MCP server could run.
Anthropic's engineering guidance says agents need "ground truth" from the environment at each step. The instruments to get it exist as MCP servers for databases, Kafka and Kubernetes, most with read-only modes.
Direct access is half of the fix. The other half is a record of what the agent queried, what came back, and which conclusion rests on which observation.

The relay loop, in the words of the people stuck in it

anthropics/claude-code #69360 (June 2026): "Claude Code (VS Code extension) cannot read the existing terminal output. When a user runs a command that produces errors, they must manually copy-paste the output into the chat. This adds friction to the debugging workflow and breaks the natural "run → see error → ask Claude" loop."

anthropics/claude-code #13865 (December 2025) spells out the steps and names the consequence: "Currently, if I want to debug a runtime issue, I have to: 1. Manually ask Claude to add print/log statements. 2. Manually run the app/script. 3. Copy/paste the terminal output back into Claude Code. 4. Ask Claude to analyze it. 5. Ask Claude to remove the logs afterwards. This manual "human-in-the-loop" data transfer is slow and error-prone, often leading to "shotgun debugging" where the model guesses the solution without concrete runtime evidence."

The Copilot side has the same report from the other direction. In microsoft/vscode-copilot-release #10778 (May 2025): "The agent fails to see the output from the terminal so it starts to run another command in a different way repeatedly failing to see the output. If I select the terminal output and select add to chat, it can see the output." An agent that cannot see the result of its own command reruns it and still sees nothing; in that thread the paste was the only channel that worked.

Reddit adds the texture. A Cursor user in r/ClaudeAI (December 2025): "Cursor also can't directly read and make use of the error logs in the PowerShell console. It feels broken for me to manually copy paste the logs to the Cursor agent." Another in r/cursor (June 2026), after a tooling change: "Only option is copy the command and re-run it yourself, which kind of defeats the whole point of having an agent run things!"

These are personal accounts with low vote counts, and I cite them as such. What they share with the GitHub issues is the shape: for everything outside the repository, the human carries the data.

Four filters between the environment and the agent

A retold log is filtered whether or not anyone intends it.

1. What to run

The first filter is the choice of what to run. The developer decides which command, which query, which endpoint. If the hypothesis in their head is wrong, the evidence they collect is evidence for or against the wrong hypothesis, and the query that would have tested the right one never runs.

2. What to select

The second is the choice of what to select. Nobody pastes a 4,000-line log. They scroll to the stack trace or the last twenty lines and paste that. The warning three seconds earlier and the second exception in a different thread stay in the terminal.

3. Truncation by tooling

The third is the one nobody chooses: truncation by tooling. A r/ClaudeCode thread (April 2026) tested a popular token-saving wrapper. On a package listing: "The remaining 314 packages are invisible. There is no truncation indicator. The output looks complete." In a separate example on log levels, the same author describes the downstream effect: "An AI agent doing incident triage reads the logs, sees one error (a transient retry), and concludes it's a minor blip. It applies a small fix and closes the investigation." Truncation is also the default in the agent itself: "For Claude Code, we restrict tool responses to 25,000 tokens by default", from Anthropic's Writing effective tools for AI agents.

4. Compaction

The fourth looks like memory. Anthropic's context engineering post describes compaction as "summarizing its contents, and reinitiating a new context window with the summary", where "the model preserves architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs or messages." The word "redundant" is doing a lot of work there. In a debugging session, the tool output that looked redundant at the time of compaction is often the one you need two hours later.

Stack the four and the agent is reasoning about a document written by a human with a hypothesis, cut to fit a chat box, truncated by a wrapper, and then summarized. It is a wonder it gets anything right.

Four filters between the environment and the agent. Each one removes the kind of detail debugging turns on.

When the agent has access and still asks you

Direct access turns out to be necessary and, on its own, insufficient, because agents with access often leave it unused.

anthropics/claude-code #43387 (April 2026) is the sharpest example I found. The agent "Wrote 1,700 lines of PHP with 28 database actions using completely guessed column names — never ran SHOW COLUMNS or any test query against the real database." The reporter adds: "Claude has SSH access to the server and could run SHOW COLUMNS, curl endpoints, or test queries — but never does before saying "done"." Thirty of forty-six queries referenced columns that did not exist while the SSH session sat unused.

anthropics/claude-code #60107 (May 2026, a report the user had the agent itself write up) moves the same failure to production: "Claude Code confirmed a Postgres dialect option (statement_timeout) as valid for a Heroku-hosted production app when I explicitly asked "are you sure this is a valid option?". The agent answered "yes" without verifying against the actual prod environment. The change passed all local tests (local Postgres in Docker has no pgbouncer), shipped to production, and crashed every dyno at boot." Local and production gave different answers to the same question, and the agent queried neither.

anthropics/claude-code #86718 (August 2026) is the near-miss: the agent "without verifying, concluded that a database record … had been overwritten/corrupted … and stated this as fact", and "had already suggested deleting the affected database record before I stopped it." The agent guessed that data was damaged and proposed deleting it before anyone had checked.

And from the MCP side, a r/cursor user (July 2025) with a Postgres MCP server connected: "Tried 4 different models to try to get it to query my dev database, all attempts fail with the model just asking me to run the query myself." A r/ClaudeCode user (July 2026) on CI: "it tried a bunch of local builds before then asking me to paste in the logs from the failed CI deployment. It has access to the CI logs, and has used them in the past."

So the instruments have to be there, and the agent has to be held to using them. The second part is a harness and prompting problem, and I come back to it below.

What Anthropic says the agent should do

Anthropic, Building effective agents: "During execution, it's crucial for the agents to gain "ground truth" from the environment at each step (such as tool call results or code execution) to assess its progress."

Anthropic, Claude Code best practices: "Claude stops when the work looks done. Without a check it can run, "looks done" is the only signal available, and you become the verification loop: every mistake waits for you to notice it." The same page defines the check as "anything that returns a signal Claude can read in the conversation: a test suite, a build exit code, a linter, a script that diffs output against a fixture, or a browser screenshot compared against a design".

The same page also, in its prompt-example table, recommends writing "[paste error]" to the agent. I mention it because it shows how deep the relay habit runs: even the document that tells you to make the agent its own verification loop assumes the human will hand it the error.

The context engineering post gives the alternative, and the alternative is many small retrievals made by the agent itself: "Letting agents navigate and retrieve data autonomously also enables progressive disclosure", and "The model can write targeted queries, store results, and leverage Bash commands like head and tail to analyze large volumes of data without ever loading the full data objects into context."

The research on long contexts points the same way. Lost in the Middle (Liu et al., 2023) found that "performance can degrade significantly when changing the position of relevant information", and Chroma's Context Rot report (2025), across 18 models, concludes that "models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows." A bigger paste makes the second problem worse while leaving the first four filters in place.

The instruments exist

"Query the database, read the broker, hit the endpoint" sounds like a wish list. As of September 2026 it is a list of MCP servers, most with a read-only switch. Star counts and dates are from the GitHub API on 2026-09-08.

Databases

For databases: crystaldba/postgres-mcp (3.3k stars) has a "Restricted Mode" that "Limits operations to read-only transactions and imposes constraints on resource utilization" and is described as "suitable for production environments". bytebase/dbhub (3.5k stars, release 2026-09-02) covers Postgres, MySQL, SQL Server, MariaDB and SQLite with "Read-only mode, row limiting, and query timeout to prevent runaway operations". Google's MCP Toolbox for Databases (16k stars) ships "Prebuilt generic tools for instant data exploration (e.g., list_tables, execute_sql)". The original reference Postgres server from the Model Context Protocol project is archived; the current reference set is filesystem, git, fetch, memory and a few others.

Kafka

For Kafka: confluentinc/mcp-confluent (v1.5.0, July 2026) exposes list-topics, consume-messages and list-consumer-groups, and also produce-message, create-topics and delete-topics. It has no read-only flag, but its --allow-tools and --block-tools options let you start it with the write tools removed, which is how I would run it for debugging.

Kubernetes

For Kubernetes: containers/kubernetes-mcp-server (2.1k stars, actively maintained) has --read-only, which "will not allow any write operations (create, update, delete)", plus --disable-destructive, and a pods_log tool that does the obvious thing.

HTTP

For HTTP: the reference fetch server converts pages to markdown, so it fetches web pages and does no API work. For "hit the endpoint itself" you want the agent's terminal with curl, or a purpose-built server.

Each of these servers exposes access the developer already has, behind a permission model.

Connecting an MCP server in Explyt (STDIO transport). The docs example is the sequential-thinking server; a database or Kafka server is configured the same way, with its read-only flags in the arguments.

The other half: record what it saw

I disagree with the simplest version of my own thesis. Give the agent a terminal, a database and a broker, and you have replaced a filtered log with an unfiltered stream that nobody reads. The developer stops being the clipboard and becomes the person who has to trust a conclusion with no visible chain of evidence.

The fix is the one that works for human engineers: a record. Which query ran, against which environment, what came back, and which conclusion rests on which result. Anthropic's tooling post recommends the reviewer's version of this: "Review the raw transcripts (including tool calls and tool responses) to catch any behavior not explicitly described in the agent's CoT." The agent's version is to keep that record itself, in the open, as it works.

Every claim about the environment carries the observation it came from: the query, the environment and the timestamp next to the value.

In practice I now hold any agent I use for debugging to a short set of rules.

Every claim about the environment carries the observation it came from: the query, the environment and the timestamp next to the value, so "the row has status = PENDING" arrives with "query at 14:02 against staging".
The environment is named every time, because #60107 above is what happens when local and production are treated as one.
And the record survives compaction: if the agent summarizes its own context, the observations stay in a file that the summary links to.

The problem sits in the harness as much as in the model. #43387 and the two MCP stories above show models with access declining to use it. A rule in the repository's agent instructions that says "before stating anything about database, broker or runtime state, query it and quote the result" is blunt. In my own sessions it removed most of the guessed claims.

Where Explyt stands

Explyt is our agent for JetBrains IDEs, and I want to state what its documentation does and does not cover on this topic.

The tools documentation frames the goal in the same terms as this article: tools "give the agent facts about your project and let it perform actions without you manually copying long logs and search results into the chat."

Once connected, each MCP server appears in the Tools menu as its own group next to the built-in IDE tools. Screenshot from the Explyt documentation.

Two of those sources come from the IDE itself, not from anything a database or a broker exposes, and I start with them because they are the ones a paste cannot replace.

Run configurations: the agent runs a configured test, build or application through the IDE and "returns the result to the agent: console output, test results, compilation errors." That closes the "copy the terminal output back" loop from #13865 for anything the IDE knows how to run. The docs are explicit that in Rider the tool currently does not build the project and does not collect coverage.

Run configurations: the IDE runs the test and returns console output, test results and compilation errors to the agent — the "copy the terminal output back" step from #13865 disappears for anything the IDE can run.

The Debug mode documentation states the principle for the runtime case in one line: "The cause is confirmed by breakpoints, variable values, and the call stack, not just by a guess based on the code or the log." Replace "the log" with "the pasted log" and you have this article. A variable read at a breakpoint is an observation no log contains, pasted or not; an agent without a debugger is back in the relay loop for that class of bug, whatever else it can query.

Frame from the official Explyt docs video (Debug mode); different project from the one in this article. The agent sets its own breakpoints and reads the debugger, instead of asking for a pasted log.

The rest of the environment comes through the two channels any agent has. The terminal tools, per the tools reference: "The terminal tools run commands and manage processes. You need them when there is no suitable IDE configuration or when the task involves an external utility." The same page lists what to check before a command runs, because "A command can change files, a database, the environment or a remote system." This is the tool that runs psql or kubectl when you allow it.

MCP: "Once an MCP server is connected, its tools appear in the Tools menu as a separate group. The names, parameters and consequences of a call are defined by the server itself." The MCP page, in its GitHub example, suggests to "Test the connection with a safe read operation first", and notes that each MCP result block "contains the outcome and diagnostic details of the call, which are useful when verifying a result or investigating an error."

An MCP server call as it appears in the Explyt chat (Chrome MCP in the docs example). The result block keeps the outcome and diagnostics of each call in the transcript — the "record what it saw" half of the argument, for MCP. Screenshot from the Explyt documentation.

What the documentation does not describe is any built-in Kafka, SQL or Kubernetes access. Those come through the terminal or through an MCP server you connect and scope, and they land in the same session as the run configurations and the debugger, so one transcript holds the test result, the variable value and the query result side by side. And pasting remains supported: the chat documentation says large pastes become attachments and "The agent receives the complete text." Pasting should stay available as the fallback; the agent's own query should be the default.

Frame from the official Explyt docs video; different project. The agent's own reply — "Please paste the log output…" — is the relay habit this article is about. Pasting stays available as the fallback; the agent's own query should be the default.

Limits

The GitHub issues above are individual reports; I have not measured how often the relay loop occurs or how often agents with access decline to use it. Reddit threads are personal accounts with low scores. MCP server statistics are a snapshot from one day and the read-only behavior comes from the READMEs; I have not audited the servers myself. The Explyt claims are limited to what the documentation says; the session I referenced is described in the linked article and its details are not published.

One thing to try this week

Pick the next runtime bug where you would normally paste a log. Connect one read-only source the agent can query itself, a database MCP in restricted mode or the terminal with kubectl logs, and add one line to your agent instructions: state nothing about the environment without quoting the query and the result. Then read the transcript and count how many of the agent's claims come with an observation attached. I expected the count to be low the first time I did this, and it was.

If you work in a JetBrains IDE, Explyt on the Marketplace gives the agent the two sources a paste cannot replace, the IDE run and the debugger, and takes MCP servers for the rest.

What did your agent last ask you to paste that it could have fetched itself, and what happened when you gave it the access? The comments are open; I am collecting the cases where the query the human never ran turned out to be the one that mattered.

Sources

Issue reports and threads

Vendor and research

MCP servers

Explyt documentation

Explyt 5.19 adds an OpenAI-compatible gateway to the subscription. Here is the setup for Claude Code and Cursor.

Viktoria — Wed, 09 Sep 2026 15:22:00 +0000

If you use more than one coding agent, you already know the tax: a separate model account for each, a separate card on file, a separate place where the balance runs out. Explyt 5.19 removes that for a specific set of tools. The subscription that used to power only the Explyt plugin in JetBrains IDEs now gives you an OpenAI-compatible URL and an API key, and the key works in Claude Code, Claude Desktop, Cursor, Codex, Cline, OpenCode and Kilo Code.

This post is the practical version: where the values are, three configs you can paste, which models answer, and the boundary that the release notes put in bold.

Step 0: get the two values

Open your Explyt account. With an active subscription or a positive Flex minutes balance you will see a block called Connect to Explyt.

It holds two things:

The URL https://api.explyt.ai/openai/api. Same for everyone.
Your API key. Copy key copies it; the eye icon reveals it.

If the block is not there, the account has neither a subscription nor Flex minutes, and there is nothing to connect yet.

Keep the key like a password. It is one key for every tool; Reissue in the account kills the old one instantly, and you then update every tool you configured.

Step 1: Claude Code

From the Claude Code page. Open or create ~/.claude/settings.json:

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.explyt.ai/openai/api",
    "ANTHROPIC_API_KEY": "<YOUR_API_KEY>",
    "CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY": 1
  }
}

Start Claude Code. When it asks Do you want to use this API key?, answer yes. If it misbehaves later, /logout in the chat and start again.

Step 2: Codex CLI

From the Codex CLI page. Open or create ~/.codex/config.toml:

model = "auto"
model_provider = "explyt"
model_catalog_json = "./model-catalog-full.json"

[model_providers.explyt]
name = "explyt"
base_url = "https://api.explyt.ai/openai/api"
wire_api = "responses"
requires_openai_auth = true

Two more files sit next to it, both described on the docs page: model-catalog-full.json with the list of Explyt models (a ready-made file is linked there) and ~/.codex/auth.json with "auth_mode": "apikey" and your key. If you were signed in through a ChatGPT account before, run codex logout first and then create auth.json. One caveat from the same page: Codex CLI and the Codex app in ChatGPT share config.toml, and the ChatGPT app can only use OpenAI models.

Step 3: Cursor

From the Cursor page. Settings → Models → API Keys:

Turn on OpenAI API Key and paste the Explyt key.
Turn on Override OpenAI Base URL and paste https://api.explyt.ai/openai/api.

Cursor allows custom keys only on a Pro plan or higher; the Cursor docs page spells this out, and the restriction comes from Cursor.

The one rule that applies to every tool

Do not also sign in to the tool's own account through ChatGPT or Anthropic. That sign-in sends requests around the Explyt gateway, and they get rejected.

Everything else that connects

Tool	Type	Docs
Claude Code	CLI	page
Codex CLI	CLI	page
OpenCode	CLI	page
Claude Desktop	Desktop app	page
Codex in ChatGPT	Desktop app	page
Cursor	Editor	page
Claude Code for VS Code	Extension	page
Kilo Code	Extension	page
Cline	Extension	page
Explyt plugin (Community mode)	JetBrains IDE plugin	page

Claude Desktop uses a dedicated gateway dialog: Configure third-party inference → Connection → Gateway, URL into Gateway base URL, key into Gateway API key, scheme x-api-key.

Anything not in the table that accepts an OpenAI-compatible base URL plus key will probably work; we have not verified it. Errors and their meanings are collected on the error messages page.

Which models answer

OpenAI and Anthropic models, plus auto, which picks a model for you and has the highest availability. Many tools fetch the list from the gateway on their own. If yours does not:

curl https://api.explyt.ai/openai/api/models \
  -H "Authorization: Bearer <YOUR_API_KEY>"

The only restriction: the Codex app in ChatGPT works with OpenAI models and auto. For Anthropic models in a Codex workflow use Codex CLI.

What the key does not carry

This is the part to read before you tell your team "we have Explyt in Cursor now".

The key gives access to models and nothing else. In an external tool you will not get:

IDE tools: project navigation, run configurations, the debugger, IDE-powered refactoring;
skills, modes, rules and checks of the Explyt agent;
subscription data inside the third-party tool's interface. Remaining minutes and the current chat's spend live in your personal account.

Paying for tokens through Explyt does not make Claude Code aware of your IntelliJ project's run configurations or let it stop on a breakpoint. Those capabilities live in the JetBrains IDE. The route for them is different and older: the Explyt MCP server from 5.17, which exposes selected tool groups to an external agent, does no inference of its own, and needs no subscription.

The key and the MCP server solve different problems and can work together: model through the gateway, IDE tools through MCP, both in one Claude Code session.

If you also use the plugin

The Explyt plugin in a JetBrains IDE has two ways to use the subscription. Personal signs in from the plugin and knows about the subscription: remaining minutes, current chat spend, a live model list with parameters. Community uses the same URL and key as any other tool and sees only an OpenAI-compatible provider. Minutes cost the same either way. Stay on Personal unless you have a reason not to.

Two fixes in the same release

Context indicator. The chat's fill bar used to understate history size by about a third, because agent chats are mostly code and tool output, which tokenize more densely than the prose the estimate assumed. A chat at the model's limit looked half empty, auto-compression did not start, and the request failed at the provider. Now Explyt reconciles the estimate with the input token count the provider reports after each response, and keeps the correction with the chat across IDE restarts. Before the first response you still see the local estimate.

MCP tokens. Servers that authenticate through a browser hand out tokens that live about an hour. The first call after expiry failed. Explyt now renews ahead of expiry, one token per server. After the update, sign in once more for each such server. Manual-header and local servers are untouched.

Also: fewer file-system checks for paths the agent mentioned but that do not exist in the project, and forks and subagents inherit the already compressed history rather than paying for the uncompressed one.

For teams

One Explyt subscription per person covers every client that person uses. In Explyt Enterprise the gateway sits inside the company perimeter; the administrator issues the URL and key, and traffic from Cursor, Claude Code or Codex to the models goes through the corporate platform, not through personal vendor accounts. See the pricing page or contact sales.

Try it on Monday

Open your Explyt account, copy the key.
Pick the one tool you open most and paste the config above.
Send one request, then open the account and confirm the minutes moved.
If the same tool also needs your IDE, wire the MCP server next; it is a separate setup.

Which tool did you connect first, and did it pick up the model list on its own? Tell us in the comments.

Full release notes: Explyt 5.19 on explyt.ai.

One symptom, ten problems: a Kotlin + Spring Boot + Kafka debugging case

Viktoria — Tue, 08 Sep 2026 13:21:38 +0000

A single visible symptom led to many defects, half unrelated to it but all tied to one invariant about async TID processing. Debugging is invariant discovery.

One visible symptom opened the session. By the end of it the developer and the agent had fixed many defects, about half of them unrelated to that symptom. All of them, though, sat inside the same scope, and the developer summed that scope up in one sentence:

Any operation on a TID must go through the single asynchronous path and must not let TMS and EMV drift apart.

I am the product manager at Explyt, where we build an AI agent for JetBrains IDEs. This session ran in our own agent, in regular chat mode, with a current frontier model, on a Kotlin and Java codebase with Spring Boot and Kafka in a payments system. I did not run it myself; I went through the log afterwards as part of our review of how people debug with an agent in practice. Debug mode was not used. What follows is a review of the shape of the work, with the session's own details left out. The defects are described as kinds, and the number in the title is the topic's working name; the log gives no count I could verify.

Previously on the Explyt blog, we compared two agent runs on one JVM bug, one working from the log and one from a breakpoint: Superpowers vs Plain Old Debugger in Explyt. That case was one bug with one cause. This one has the opposite shape, one symptom with many causes, and it stresses a different part of the debugging loop.

TL;DR

One symptom, many defects, half of them unrelated to the symptom, all of them violating one invariant about asynchronous TID processing.
Our agent was tactically helpful: it proposed SQL, warned about risks, explained code locally, and did not scatter new debug logging. It was strategically passive: the developer drove every step.
It did not separate verified facts from hypotheses, and it proposed fixes before the problem was localized. Those fixes compiled and passed lint. Afterwards something else broke, differently.
The developer was the agent's only sensor. Kafka and Spring logs, SQL results, HTTP responses, Kubernetes and Compose messages, config files: all pasted into the chat by hand. In the terminal the agent ran only git.
Debugging a distributed system is a search for the invariant that all the defects violate. The agent needs instruments for that, and I say below which of them Explyt has today and which it does not.

What the session looked like

I cannot show you the symptom or the defects; the log belongs to the developer's project. What I can describe is the shape.

The developer came in with one observable problem. Working through it with the agent, they found a defect, fixed it, and something else broke, in a different way. Then another defect, another fix, another different breakage. About half of what they fixed had nothing to do with the original symptom. Every fix, however, touched the same scope: how the system handles a terminal ID (TID) across its asynchronous path and keeps two downstream systems, TMS and EMV, consistent. The sentence in bold above is how the developer described that scope.

A few times the developer asked the agent to add something and later asked it to roll that back. That is the texture of the session: iterative, exploratory, with the agent along for the ride.

What the agent did well

The agent proposed SQL queries to inspect the current state in TMS and EMV, so the discussion started from actual rows. It warned about risks in code it was asked to change. It explained what a given piece of code did, when asked, and got it right. It did not add new debug logging; it worked from the log statements that already existed, which kept the diff clean.

All of that is real help; a developer working alone would have done that typing themselves.

Where it fell apart

Strategic passivity

The first failure was strategic passivity. The agent did not run the investigation. It waited for the developer to bring the next piece of evidence, reacted to it, and waited again. At no point did it say: here is my current theory, here is what would falsify it, let me check. The developer decided what to look at next, every time.

Facts and guesses in one voice

The second: verified facts and guesses came out in the same confident voice. The developer had to keep track of which statements had been checked and which had not. In a session with many defects that bookkeeping is most of the job, and the agent was not doing it.

Fixing before localizing

Third, the agent proposed fixes before the problem was localized. The developer applied them; the log shows no rollback of those fixes, at least not in the chat. The agent checked that each change compiled and passed lint. It did not check that the fix removed the symptom. Several fixes turned out to be ineffective: something else broke afterwards, differently. Reading that part was uncomfortable. I have accepted a plausible fix at the end of a long day for the same reason: it compiled.

No model of the entities

The fourth failure is the one that matters for tooling. The agent never built a model of how the entities relate: what TID is, what TMS and EMV each own, what "consistent" means between them. Without that model, every defect looked like a separate bug; with it, they are one invariant violated in several places.

The developer as the only sensor

The developer pasted into the chat, by hand, fragments of SQL scripts and their results, Spring Boot and Kafka log excerpts, HTTP responses, messages from Kubernetes and Docker Compose, and configuration files. The agent's own use of the terminal was limited to git.

So the agent's picture of the running system was whatever the developer chose to show it, in the order they chose, with their framing already applied. That is a poor position from which to form an independent theory, and it explains the passivity. This is the failure I keep coming back to, and I do not have a tidier answer than "give it instruments".

The review proposed two fixes. Give the agent a browser and a terminal, and make it use them. And give it a debugging method with a written journal: hypotheses, experiments, results, and a working theory that includes the data model and its invariants, along the lines of the Superpowers systematic-debugging skill and the parallel-debugging variant. Both fixes assume the agent can run an experiment and read its result without a human in between. A method without instruments is a checklist.

Debugging is invariant discovery

The framing I took away from this log: debugging a distributed system is a search for the sentence that all the defects violate, and the individual fixes are almost a byproduct. A debugging agent therefore needs four things.

A working theory in writing, separate from the chat scroll, so that "checked" and "assumed" stay apart over a long session.
The ability to run the experiment that would confirm or refute the current hypothesis, and to read the result itself.
Direct observation of state, a variable at a breakpoint or a row from a query it ran, because pasted excerpts are already interpretations.
After the second unrelated defect, the question of what the two have in common.

The session failed on all four. The model was a current frontier model. The harness gave it chat, file access and git, and asked it to debug an asynchronous payments pipeline. That is where the four failures come from, and it is our harness.

What Explyt does about it today, and what it does not

Explyt covers part of this loop today, and I want to be precise about which part.

Explyt's documented Debug mode exists for the "observe state directly" requirement: for a bug you can reproduce by running a test, an application or an existing IDE run configuration, the agent is asked to confirm the cause under the debugger first, with breakpoints, variable values and the call stack, before it changes any code. Then a minimal fix, then a rerun of the original scenario and the related tests. That is the experiment loop this session never had.

Run configurations close the other gap: the agent runs the build or the test through the IDE and gets back console output, test results and compilation errors as a structured tool result, so it has something to read beyond "compiles and lints". The docs are explicit that a single debugger run confirms the fix only for that scenario and does not replace the related tests.

Nothing in the Explyt documentation reads Kafka topics, runs SQL against your database or pulls logs from Kubernetes on its own. Those sources still enter the session the way they did in this log, through the developer, or through an MCP server you connect yourself; the tools documentation describes MCP as the way to attach browsers, knowledge bases and other external services, and what a given server exposes is up to that server. Explyt's Memory Bank keeps facts and project agreements across chats, which is useful, and it is a different thing from a per-session hypothesis journal. The working theory is still the developer's to keep.

So if this symptom can be reproduced by a test or a run configuration, Debug mode would have changed the middle of the loop: the agent could have seen the actual state at the moment of the write, and rerun the scenario after each fix, with lint no longer the last check. It would not have changed the edges. Someone still has to bring the Kafka and infrastructure evidence, and someone still has to notice that the latest defect rhymes with the first one.

If you want to try this on your own bug

This part is a suggestion for your next session, and nothing here describes what happened in the one above.

Whatever agent you use, put the invariant question into its instructions so it stops being your job to remember it. A repository-level rule along these lines works with most agents that read AGENTS.md or a vendor equivalent:

## Debugging

Keep a written working theory in a file under .tasks/, updated after
every experiment: what is verified (with the command or breakpoint that
verified it), what is assumed, and the current hypothesis.

Before proposing a fix, localize: reproduce the failure under a test or
run configuration and confirm the cause at a breakpoint or in a query
result you ran yourself.

After a fix, rerun the original scenario. "Compiles and lints" is not a result.

After the second defect in the same area, stop and write down what the
two have in common before fixing either.

Then reproduce one failing test in the debugger and make the agent read the variables before it edits anything. If the agent has no debugger, the rule still helps; it just makes the missing instrument visible.

Limits

The evidence here is one session log from one developer using our agent, reviewed by me after the fact. I did not run it and I am not publishing its details, defect list or code. Whether the symptom was reproducible under a local test, which is what Debug mode requires, I do not know from the log. Nothing here is a benchmark.

Try the loop in a JetBrains IDE

If you already work in a JetBrains IDE, install Explyt from the Marketplace and reproduce one failing test in Debug mode. Ask the agent to confirm the cause at a breakpoint before it edits anything, then make it rerun the scenario after the fix.

When your agent fixed a bug and something else broke, did you go back and ask what the two had in common? I am collecting the invariants people found that way; the comments are open.

Sources

Superpowers vs Plain Old Debugger in Explyt

Viktoria — Thu, 03 Sep 2026 11:04:36 +0000

An AI agent can follow a debugging checklist and still miss the bug.

The question I kept coming back to was simple: who ran the experiment?

I interviewed our users, mostly senior developers. In those interviews, debugging a hard problem came up most often. In practice, the investigation is still manual: a developer supplies logs, sets breakpoints, tests a theory, and pastes the result back into the chat.

A systematic debugging method gives the work a clear order: gather facts, form a hypothesis, run an experiment, then change the code.

Superpowers was created by Jesse Vincent and the Prime Radiant team. Matt Pocock helped popularize it; the collection includes a systematic-debugging skill. I ran Opus 5 on the same real bug and the same prompt twice. The only thing I changed was the debugging workflow: Superpowers in one run, Explyt's built-in Debug skill in the other.

I ran one controlled case; it is not a benchmark of either product. One number makes the contrast plain: Explyt used 67k tokens to solve this case, while Superpowers used 132k. The transcripts explain the gap: the debugger supplied a runtime fact early, while the other run spent more context on speculative code, tests, and refactoring.

The bug: one protected Windows directory, one crashed coroutine

The test case came from a Kotlin project on Windows. The IDE log contained:

java.nio.file.AccessDeniedException: C:\Config.Msi
    at java.nio.file.Files.list(Files.java:3422)
    at FacadeMentionCandidates.listDir(FacadeMentionCandidates.kt:83)
    at FacadeMentionCandidates.listChildren(FacadeMentionCandidates.kt:69)
    at MentionSuggestionMapperKt$suggestionsFor$2.invokeSuspend(MentionSuggestionMapper.kt:36)

The relevant code first checked whether a path was a directory and then enumerated it with Files.list. The exception came from a protected Windows Installer directory. FacadeMentionCandidates makes filesystem roots available to the @ mention flow, so a user navigating through a path such as @/C:/ can eventually reach a directory that the IDE process cannot enumerate.

The error was not handled at the flow boundary. It escaped the suggestion flow and was reported as an unhandled coroutine exception in WorkspaceBootstrap, with the IDE blaming the plugin. The same listing path is also used while searching around the focused file, so this was a small function with more than one way to fail.

Run one: a plausible patch from Superpowers

The Superpowers run started well. It searched the log instead of reading the whole file, followed the stack into FacadeMentionCandidates, found the unguarded Files.list, and stated a plausible root-cause hypothesis.

Then the investigation stopped before the experiment. The run never set a debugger breakpoint or inspected a value from the failing execution. The proposed fix was to wrap the directory listing in a try/catch, log the exception, and return emptyList().

That choice made the failure disappear from the call stack, but it changed the meaning of the result. An empty directory and a directory denied by ACL became the same value; other I/O failures still had no user-facing outcome. The mention UI could not tell the developer what happened, and the option to mention the directory remained indistinguishable from a normal empty-folder case.

The regression test followed the same assumption. It asserted that an unreadable directory produced no children. On Windows, the first version could not create the required POSIX permission state and was skipped. To force the exception, the agent then added a readDirectory function to the production constructor so the test could inject a throwing fake. That seam existed for the test, not for the product behavior.

The run finished with seven passing tests, but those tests established that the workaround returned an empty list. They did not establish that the user-facing behavior was correct, or that the error had been verified against a real paused execution. I discarded the patch: it hid the distinction we needed and left a test-only abstraction in the production API.

That is how a checklist turns into a ritual. Every step appears in the transcript; the experiment never does. A plausible local edit stands in for evidence.

Run two: Explyt puts the hypothesis inside the IDE

With Explyt's built-in debugging skill, the model started from the same log and source files, but it could use the JetBrains debugger as part of the investigation.

It created a minimal reproducer, set a line breakpoint on the Files.list call, and ran the test under the debugger. When execution stopped, the value was concrete:

Debugger evaluation	Observed value
`dir`	`C:\Config.Msi`
`Files.isDirectory(dir)`	`true`
`Files.isReadable(dir)`	`false`
`Files.list(dir)`	throws `java.nio.file.AccessDeniedException`

The call stack matched the production trace: listDir → listChildren → MentionSuggestionMapper.suggestionsFor, inside the coroutine that builds mention suggestions. The debugger turned "a protected path probably causes this" into a verified statement about the running program:

the directory check passes, the process cannot read the directory, and the unguarded enumeration kills the suggestion flow.

The evidence kept the fix at the filesystem boundary, with focused regression coverage and the normal listing path left intact. The mention protocol and runtime model stayed unchanged, and debug prints were unnecessary.

The narrow fix passed review and is now in production. A try/catch was enough here. What mattered was choosing it after observing the failing execution, instead of using it as the first explanation for the failure.

The practical difference

	Superpowers run	Explyt run
Starting point	Same log and source	Same log and source
Investigation channel	Text, stack trace, and tests	JetBrains debugger plus source and tests
Key fact	Inferred from `AccessDeniedException`	Observed at a breakpoint: directory, permissions, and throw
Token use in this run	132k	67k
First implementation	Swallow the error as `emptyList()`	Keep the fix at the proven failing boundary
Test signal	Passing tests for the fallback	Runtime evidence followed by a focused regression test
Result	Patch discarded as technical debt	Reviewed minimal fix deployed to production

I could see the cleanup cost in the diff. The discarded patch carried a workaround-driven test seam. The shipped patch stayed at the failing boundary and left the normal listing path intact.

For a runtime bug, a coherent explanation from a log is still only a theory. I want one fact captured from the running program: a variable value, a call stack, an execution path, or the exact branch that contradicts the model's theory.

Explyt is an AI agent for JetBrains IDEs. In this run, it used a supported JetBrains IDE and run configuration. Its Debug workflow started the code under the JetBrains debugger and inspected breakpoints, variable values, call stacks, and execution paths before editing. The IDE supplies the evidence; the developer still decides whether the hypothesis and the resulting diff are acceptable.

What I can say from these transcripts is limited. Superpowers can be combined with a debugger, and Explyt's debugger depth depends on the IDE, language, and run configuration. Check the feature matrix for the supported combinations.

The token count points in the same direction. Explyt solved this case in 67k tokens; Superpowers consumed 132k. A debugger did not merely add another tool to the menu. It gave the model the fact it needed before more speculative turns accumulated.

JetBrains has been running paired A/B tests on "token-saving" skills, with a useful warning against reading too much into their README claims. In the Caveman test, the advertised 65% reduction became 8.5% fewer output tokens on real agentic tasks, with activation forced. In the newer rtk test, the with-rtk arm cost a median 7.6% more per task at low reasoning effort, and showed no saving at high effort. The latest Ponytail test found a 10.3% cost reduction on its own benchmark. The results are mixed; the consistent lesson is to measure the whole agent run, not a skill's self-reported counter.

Try the first useful task

If you already work in a JetBrains IDE, install Explyt from the Marketplace, open a reproducible failing test, and let the agent inspect the first runtime value that contradicts its theory. Then review the diff before accepting the fix.

The first essay in this series argued that CMMI's core insight survives and its quiet assumption does not: process maturity still predicts delivery reliability, but the model was sized for code written by people at a human pace.

Viktoria — Wed, 02 Sep 2026 18:11:31 +0000

Viktoria

Sep 2

ADLC: The Lifecycle Taking Shape

#ai #programming #webdev #llm

11 min read