I spent two days trying to make LangChain's AgentExecutor do what I wanted.
It worked. Sort of. The agent called tools. It produced answers. But I couldn't explain why it made the choices it made. When I tried to swap the LLM provider, the tracing broke. When I tried to add a second tool, the dispatch logic got tangled in layers of abstraction I didn't write and couldn't easily inspect.
So I deleted it. I wrote a ~270-line ReAct loop instead.
This post is about that decision, the architecture that came out of it, and the one thing most agent tutorials skip entirely: observability.
What I Was Building
The project is called RepoPilot. The end goal is a CI-fixing agent: given a failing GitHub Actions run, it should eventually diagnose the failure and propose a fix that a human can approve.
But before any of that, I needed a foundation. An agent that could:
•Inspect a local Git repository through controlled tools
•Talk to an LLM through a provider-agnostic interface
•Record every decision it made, so I could debug it later
That's Phase 1. This post is about how I built it — and why the framework wasn't the right tool.
The Case for LangChain (And Why It Didn't Hold)
LangChain, CrewAI, and similar frameworks offer real value:
•Pre-built tool integrations
•Memory abstractions
•Callback systems for observability
•Multi-agent orchestration out of the box
For a certain kind of project — one that needs many integrations quickly, or one where the framework's opinions match yours — they're a legitimate accelerator.
But for my project, three things went wrong.
1. Opacity
When AgentExecutor made a decision, I couldn't easily see why. The chain of prompts, the tool selection logic, the intermediate parsing — it was all happening inside layers I hadn't written. Debugging meant reading framework source code instead of my own.
2. Provider Lock-In
LangChain abstracts LLM providers, but the abstraction has its own shape. When I tried to swap Gemini for a local Ollama model, tracing broke. The framework's callback system was tied to the LLM abstraction in ways I didn't fully control.
3. Framework Churn
LangChain's API changes frequently. Code from six months ago often needs rewriting. For a project I plan to maintain and extend, that's a liability.
None of these are dealbreakers on their own. Together, they pointed to a clear answer: for a project whose entire value is auditability and control, a black box is the wrong foundation.
The ReAct Loop, Hand-Rolled
ReAct (Reason + Act) is a simple pattern:
1.The LLM reasons about what to do next.
2.It picks a tool and provides arguments.
3.The tool runs and returns a result.
4.The result is added to the conversation.
5.Repeat until the LLM produces a final text response.
That's it. Here's the core of my implementation:
python
def run(self, goal: str) -> str:
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": goal},
]
for step in range(self.max_steps):
response = self.llm.chat_with_tools(messages, self.tools.schemas())
if not response.tool_calls:
return response.content
messages.append({"role": "assistant", "content": response.content})
for call in response.tool_calls:
result = self.tools.execute(call.name, call.arguments)
messages.append({
"role": "tool",
"name": call.name,
"content": json.dumps(result),
})
return self._request_final_answer(messages)
Roughly 20 lines of logic. It does everything I need:
•Bounded iterations via max_steps
•Tool dispatch through a registry
•Structured messages that I can inspect at any point
•A final-answer fallback if the step limit is reached
I wrote it. I understand every line. When something goes wrong, I know exactly where to look.
The Provider Abstraction
One design decision I got right early: core/ never imports a provider SDK.
The agent talks to an interface, not to Gemini or OpenAI or Ollama directly.
python
class LLMProvider(ABC):
@abstractmethod
def chat(self, messages: list[dict]) -> str: ...
@abstractmethod
def chat_with_tools(self, messages: list[dict], tools: list[dict]) -> NormalizedResponse: ...
The Gemini adapter implements this. Swapping providers means writing a new adapter — not rewriting the agent.
I can develop against a free cloud API and later switch to a local model by changing one environment variable. The agent doesn't know the difference.
Why this matters: The moment I wrote from openai import OpenAI inside core/, I would have locked myself in. The adapter pattern cost me about 50 extra lines. It bought me flexibility I'll need in Phase 2.
Observability: The Part Most Tutorials Skip
Here's the thing I built before adding a second tool: tracing.
Every agent run produces a structured trace. Every LLM call, every tool invocation, every result, every error. Not log lines — a data model.
python
@dataclass
class AgentStep:
step_number: int
timestamp: float
type: str # "llm_call" | "tool_call" | "tool_result" | "error"
model: str | None
tool_name: str | None
arguments: dict | None
result: Any | None
error: str | None
duration_ms: float
At the end of every run, RepoPilot prints a summary:
text
─────────────────────────────────────────────
RepoPilot Run Summary
─────────────────────────────────────────────
Goal: Check the repository status
Total steps: 4
LLM calls: 2
Tool calls: 1
Errors: 0
Total duration: 78.41s
─────────────────────────────────────────────
And it writes the full trace to .runs/{run_id}.json.
This is what makes the agent debuggable. When a run fails, I don't guess. I open the trace and see exactly what happened: which prompt went in, which tool was called, what came back, and where the loop broke.
Why I built this before adding more tools: If I had 5 tools and a broken loop, I wouldn't know which was the problem. With tracing, I can see the loop working perfectly while a tool misbehaves — or vice versa.
What I Got Wrong
Two honest mistakes worth sharing.
1. I put tracing in the provider adapter first.
My first version recorded LLM calls inside the Gemini adapter. It seemed logical — that's where the API calls happen. But when I tried to swap providers, tracing would be lost. I moved it to the core loop instead. Now tracing works identically regardless of which provider is running.
Lesson: Put cross-cutting concerns (tracing, safety, approval) in the core layer, not in the provider layer.
2. I started with 5 tools.
In my first draft, I wrote git_status, git_diff, read_file, list_files, and search_code all at once. Then I spent a day debugging tool schemas and error handling — before I even knew if my ReAct loop was correct.
I cut back to one tool: git_status. Just enough to prove the loop worked.
Lesson: One tool is enough to prove the architecture. Features come after the foundation is solid.
The Result
Phase 1 is complete. The agent:
•Talks to Gemini through a provider-agnostic interface
•Calls a read-only git_status tool
•Records a structured JSON trace of every run
•Passes 68 automated tests
•Runs end-to-end on a real local repository
It's not a CI-fixing agent yet. It's the foundation for one. But the foundation is clean, tested, and — most importantly — understandable.
What's Next
Phase 2 is where RepoPilot earns its name:
•Parse GitHub Actions logs deterministically (no LLM for known failure patterns)
•Escalate to the LLM only when the pattern is ambiguous
•Propose a fix as a diff, but never apply it without human approval
•Trace every decision
The deterministic-first design is the differentiator. Most CI-fixing agents throw logs at an LLM and hope. RepoPilot will classify failures with regex first, and only use the LLM when it has to.
The Takeaway
Frameworks are not wrong. They're just not always right.
If you're building a quick prototype, use LangChain. If you're building a system whose value is in its auditability, control, and safety guarantees — write the loop yourself. It's ~100 lines. You'll understand every one of them.
And build observability first. Not because it's fun, but because it's the difference between an agent you can debug and an agent you can only hope works.
The code is at https://github.com/Ujma21/RepoPilot.git . Phase 2 is in progress. If you're building agents and hit the same framework wall I did, I'd love to hear how you solved it.
Top comments (1)
interesting post very intellectual.