TL;DR
I introduced a SKILL layer while improving fare extraction in a MCP-based transit agent. After that, the agent stopped reliably using the MCP tool and started favoring web search, breaking real-time accuracy. I rolled back to a stable baseline, reintroduced SKILL more carefully, and restored MCP tool usage.
After the last post, the system felt stable.
The MCP-based transit agent was working. Fare retrieval was consistent, routes were predictable, and tool calls behaved the way I expected. It felt like I had finally reached a baseline I could build on.
Then I started iterating again.
The goal was still narrow. I was trying to improve fare extraction for more complex routes. Before introducing any new abstraction, I actually modified the MCP tool itself. I adjusted parts of the parsing logic to better handle complex route structures.
That change alone did not break anything immediately, but it shifted the foundation slightly.
After that, I introduced a SKILL layer to structure how the agent should use MCP tools. At the time, it felt like a logical next step. The idea was to make tool usage more consistent and less dependent on implicit behavior from the agent.
Nothing broke immediately. The system still ran, responses still came back, and at a glance everything looked fine.
But I had effectively changed two layers that depend on each other without fully understanding how they would interact.
The Failure Signal (What Broke)
The first sign was not an error.
It was a shift in where answers were coming from.
The agent started relying on backup web search instead of the MCP transit tool. This was not limited to fare extraction. Even for basic transit queries, the MCP tool was no longer being used, and the responses were coming from general web-based results instead of real-time transit data.
I tried to correct it at the SKILL layer.
I adjusted the instructions, made the tool usage more explicit, and expected the behavior to shift back. But the pattern stayed the same. The agent continued to avoid the MCP tool, even when the intent clearly required real-time transit information.
That is when it became obvious that the output was no longer grounded in the system I built. It was producing responses, but not based on the real-time transit pipeline anymore.
The Point of Uncertainty (And My Frustration)
At this point, I did not know which layer was responsible.
The MCP tool still worked when called directly. The SKILL layer still looked correct in isolation. Even the parsing changes I had made earlier did not clearly explain the behavior I was seeing.
But the system behavior told a different story.
The agent consistently avoided the MCP tool, and even adjustments in the SKILL instructions did not change that. It kept producing answers that were not grounded in real-time transit data.
It felt less like a single bug and more like a breakdown in coordination between layers that were supposed to work together.
The Rollback and What Changed
At this point, I stopped trying to patch it.
Nothing I changed was bringing the MCP tool back in consistent use anymore. The agent was still bypassing it, and the output was still not grounded in real-time transit data. The SKILL layer had not improved control, and it made the behavior harder to reason about.
So, I rolled everything back to v0.1.1
That meant returning to the simpler parsing logic in the MCP tool and removing the expanded SKILL-driven behavior. I wanted a baseline where I already knew the system behaved correctly.
Then, I reintroduced SKILL, but only in a minimal form, closer to the original intent rather than an orchestration layer that tried to do too much.
After testing again, the behavior stabilized.
The agent started using the MCP tool reliably again. Fare retrieval worked. Even some complex routing cases began working better than before.
It is not fully perfect. There are still edge cases that break. But compared to everything before this point, it is in a much more stable state.
My working hypothesis is that the regression was not in the MCP server itself, but somewhere in the interaction between the SKILL layer, tool selection, and my parsing changes.
This is still something I am iterating on.
What I learned from this round is that SKILL only behaves well when it stays close to orchestration. Once it starts influencing tool selection too strongly, it becomes harder to predict how the agent will actually behave.
This is the version I ended up tagging.
Closing Insight
I think the issue was not the MCP tool itself. It was how additional layers influenced when it got used. SKILL is not a system prompt replacement, but an orchestration layer that can unintentionally affect tool selection. In my case, that interaction changed the agent's routing behavior in subtle ways.
Rolling back restored a simpler, more predictable decision path, which is why the system stabilized again.
For More Info About This Project
🔗GitHub Finish-Up-A-Thon Challenge post
Transparency Note: I used AI as an editor, not as the author. For this article, it helped refine the structure and improve the English grammar. The technical content, experiments, opinions, and conclusions are my own and were reviewed by me before publishing.
Top comments (7)
The silent regression part deserves to be framed on a wall. "It still produces an answer" is the most dangerous state a system can be in — everything looks alive while the core quietly died. I'm a non-developer who runs internal tools with AI agents and MCP daily (hospital setting, so wrong-but-confident answers have real cost), and the failures that hurt were never the loud ones. They were the agent falling back to a plausible-looking source without telling anyone.
Two habits your post just reinforced for me: (1) never change two layers in the same breath — you had no way to know if SKILL or the parser broke it, and that ambiguity is the real cost, more than the bug itself; (2) my smoke tests now check where the answer came from, not just that an answer came. "Did you use the tool?" turns a silent regression into a loud one.
Also: rolling back and writing it up honestly is the least glamorous, most useful genre on this site. Thanks for not dressing it up as a win.
Thank you for sharing your perspective. Your point about not changing two layers at the same time is 100% right. Looking back, I should have isolated those changes better. The hardest part of these failures isn't always fixing the bug; it's figuring out which layer introduced the regression.
I also like your idea of testing "where the answer came from" instead of only checking that an answer exists. That is a great mental model for building reliable AI agent systems.
"Figuring out which layer introduced the regression" — you just named the real cost better than I did. The fix is usually minutes; the archaeology is what eats the week.
And full disclosure: the "where did the answer come from" habit was learned the embarrassing way. My system gave confident, plausible answers for a while before I noticed it wasn't actually reading what I thought it was reading. Your post saving someone from that archaeology is exactly what these writeups are for. Good luck with the rebuild!
This is a useful failure mode to write down because it is not a “tool broke” problem. It is a routing problem.
The MCP tool still worked directly, but the agent stopped choosing it when another layer changed the decision path. That is exactly the kind of regression that is hard to catch if tests only validate tool output in isolation.
For systems like this I’d want two separate test layers:
The second one is easy to skip, but it is where the real product behavior lives.
Rollback was the right instinct here. Once multiple layers change at once, restoring a known-good decision path is often faster than trying to prompt your way out of uncertainty.
That's a great point. I agree that the routing layer is where the real product behavior lives. The rollback gave me a clean baseline, and I'm moving into a more structured evaluation phase with dedicated routing tests to catch regressions like this much earlier. I'll also be documenting the evals more thoroughly along the way. Thank you for the insight.
The tension between adding structure through a skill layer and preserving reliable tool selection is a really difficult tradeoff, and your detailed writeup of the regression is valuable. What stood out to me is how the skill layer did not actually break anything in isolation, it subtly shifted the agent's routing preference without an obvious error state. That kind of silent degradation is one of the hardest problems to catch in agent systems because there is no crash, no stack trace, just a gradual drift in output quality. Your approach of rolling back to a known good baseline and reintroducing the skill layer in a more minimal form mirrors what I have seen work well in practice. One question I wanted to ask: did you try logging the exact reasoning trace or tool selection probability from the model during the regression period? Being able to see which signals in the prompt were steering the agent toward web search versus the MCP tool might reveal whether the skill layer was adding noise to the tool selection prompt or actually overriding it at a deeper level in the model's attention patterns.
Thanks you. I really appreciate your insights. I didn't capture the reasoning trace or tool selection probabilities during the regression. I'm now moving into a more structured evaluation phase, and that's definitely something I plan to investigate. I'm also considering trying Google ADK's tracing capabilities to better observe the agent's tool routing and execution flow. I think that could provide much clearer insight into why the routing behavior changed.