TL;DR
I introduced a SKILL layer while improving fare extraction in a MCP-based transit agent. After that, the agent stopped reliably using the MCP tool and started favoring web search, breaking real-time accuracy. I rolled back to a stable baseline, reintroduced SKILL more carefully, and restored MCP tool usage.
After the last post, the system felt stable.
The MCP-based transit agent was working. Fare retrieval was consistent, routes were predictable, and tool calls behaved the way I expected. It felt like I had finally reached a baseline I could build on.
Then I started iterating again.
The goal was still narrow. I was trying to improve fare extraction for more complex routes. Before introducing any new abstraction, I actually modified the MCP tool itself. I adjusted parts of the parsing logic to better handle complex route structures.
That change alone did not break anything immediately, but it shifted the foundation slightly.
After that, I introduced a SKILL layer to structure how the agent should use MCP tools. At the time, it felt like a logical next step. The idea was to make tool usage more consistent and less dependent on implicit behavior from the agent.
Nothing broke immediately. The system still ran, responses still came back, and at a glance everything looked fine.
But I had effectively changed two layers that depend on each other without fully understanding how they would interact.
The Failure Signal (What Broke)
The first sign was not an error.
It was a shift in where answers were coming from.
The agent started relying on backup web search instead of the MCP transit tool. This was not limited to fare extraction. Even for basic transit queries, the MCP tool was no longer being used, and the responses were coming from general web-based results instead of real-time transit data.
I tried to correct it at the SKILL layer.
I adjusted the instructions, made the tool usage more explicit, and expected the behavior to shift back. But the pattern stayed the same. The agent continued to avoid the MCP tool, even when the intent clearly required real-time transit information.
That is when it became obvious that the output was no longer grounded in the system I built. It was producing responses, but not based on the real-time transit pipeline anymore.
The Point of Uncertainty (And My Frustration)
At this point, I did not know which layer was responsible.
The MCP tool still worked when called directly. The SKILL layer still looked correct in isolation. Even the parsing changes I had made earlier did not clearly explain the behavior I was seeing.
But the system behavior told a different story.
The agent consistently avoided the MCP tool, and even adjustments in the SKILL instructions did not change that. It kept producing answers that were not grounded in real-time transit data.
It felt less like a single bug and more like a breakdown in coordination between layers that were supposed to work together.
The Rollback and What Changed
At this point, I stopped trying to patch it.
Nothing I changed was bringing the MCP tool back in consistent use anymore. The agent was still bypassing it, and the output was still not grounded in real-time transit data. The SKILL layer had not improved control, and it made the behavior harder to reason about.
So, I rolled everything back to v0.1.1
That meant returning to the simpler parsing logic in the MCP tool and removing the expanded SKILL-driven behavior. I wanted a baseline where I already knew the system behaved correctly.
Then, I reintroduced SKILL, but only in a minimal form, closer to the original intent rather than an orchestration layer that tried to do too much.
After testing again, the behavior stabilized.
The agent started using the MCP tool reliably again. Fare retrieval worked. Even some complex routing cases began working better than before.
It is not fully perfect. There are still edge cases that break. But compared to everything before this point, it is in a much more stable state.
My working hypothesis is that the regression was not in the MCP server itself, but somewhere in the interaction between the SKILL layer, tool selection, and my parsing changes.
This is still something I am iterating on.
What I learned from this round is that SKILL only behaves well when it stays close to orchestration. Once it starts influencing tool selection too strongly, it becomes harder to predict how the agent will actually behave.
This is the version I ended up tagging.
Closing Insight
I think the issue was not the MCP tool itself. It was how additional layers influenced when it got used. SKILL is not a system prompt replacement, but an orchestration layer that can unintentionally affect tool selection. In my case, that interaction changed the agent's routing behavior in subtle ways.
Rolling back restored a simpler, more predictable decision path, which is why the system stabilized again.
For More Info About This Project
🔗GitHub Finish-Up-A-Thon Challenge post
🔗GitHub Repo
Transparency Note: I used AI as an editor, not as the author. For this article, it helped refine the structure and improve the English grammar. The technical content, experiments, opinions, and conclusions are my own and were reviewed by me before publishing.
Top comments (0)