Deepansh Bhargava

Posted on Jul 27

Building an AI Coding Agent: 80% Engineering, 20% LLM

#ai #llm #webdev #programming

Few months into building Piper, I watched the agent confidently edit the wrong file.

The user asked: "Add pagination to the Jobs page."

The agent searched the codebase, found a file with "Jobs" in the name, and modified it.

Except it was a test file, not the component.

The resulting PR broke the entire test suite.

That's when I realized something fundamental: building an AI coding agent isn't about choosing the right LLM.

It's about everything that happens before and after the LLM thinks.

What Makes a Coding Agent Useful?

A coding agent is more than autocomplete on steroids.

Instead of answering questions, it completes tasks.

Real tasks:

Understand an existing codebase
Find the right files to modify
Make changes without breaking things
Run tests to verify the fix
Commit and open a pull request

I spent the first month thinking the hard part was prompting the LLM. It wasn't.

The hard part was everything else.

How Context Selection Broke Everything

The Problem

I started simple. I'd send the entire repository to the LLM.

It worked for small codebases.

Then I tried Piper on a 50,000-line monorepo.

Two things happened:

Token limits kicked in — I could only send 10% of the code
The agent got confused — too much irrelevant context made it hallucinate

So I tried searching. If the user asked about "pagination," I'd find files with "pagination" in the name.

Better, but still broken.

The Jobs example shows why: the test file was named jobs.test.ts. It matched the search perfectly. The agent couldn't distinguish between the real component and the test.

The Iteration

I realized the indexer needed to understand repository structure, not just filenames.

The breakthrough came from thinking about how I search a codebase:

I look at the folder structure
I identify patterns (components in /src/components, tests in __tests__)
I read imports to trace dependencies
I understand which files are related

So I built an indexer that does exactly that.

It analyzes:

Directory patterns
Import relationships
File naming conventions
The AGENTS.md file (if it exists)

Now when asked about "Jobs," the indexer understands that src/pages/jobs.tsx is probably more relevant than __tests__/jobs.test.ts.

It's still not perfect, but it's radically better.

The lesson: Good retrieval beats bigger models. I've seen a smaller model with the right 5 files outperform a bigger model with 50 irrelevant ones.

When the Model Kept Editing the Wrong Files

The Failure

Even with better retrieval, the agent still made mistakes.

It would understand the problem, read the right files, then somehow edit the wrong one.

I realized the agent needed to think before acting.

So I added a planning phase.

Plan Mode

Before executing code changes, the agent now:

Analyzes the problem
Searches the repository
Reads relevant files
Proposes a plan
Waits for user approval
Then executes

Here's what that looks like:

The user can review this before the agent touches any files.

It sounds simple, but it eliminates the majority of mistakes.

The lesson: Making the model "think out loud" before acting reduces errors dramatically. Users also trust the agent more when they can see its reasoning.

Why One Model Wasn't Enough

The Discovery

I was using the biggest, most powerful model for everything.

Searching for files? Overkill.

Refactoring complex code? Necessary.

But I was paying for powerful reasoning on every simple task.

My costs were terrible. My latency was worse.

The Solution

I built automatic model routing.

For simple tasks (searching, reading files, answering questions), the agent uses Gemini 2.5 Flash Lite.

For complex tasks (refactoring, debugging, architecture decisions), it uses Gemini 2.5 Flash.

How does it decide? By analyzing the user's message:

Keywords like "implement," "debug," "refactor" → use Flash
Just reading or searching → use Flash Lite
Code blocks in the message → use Flash
Message longer than 80 words → use Flash

If the primary model fails, it automatically falls back to the other.

Cost dropped 60%. Latency improved. Quality stayed the same.

The lesson: Not every task needs maximum capability. Intelligent routing is worth the complexity.

Git Integration Made It Feel Like a Teammate

Why It Matters

Making code changes is only half the workflow.

A developer still needs to:

Review changes
Commit with a message
Push the branch
Open a PR

Adding this in the cycle changes everything.

An Example

User: "Fix the sorting bug in the Jobs page."

↓ Agent reads files

readFile("src/pages/jobs.tsx")
readFile("src/hooks/useSortJobs.ts")

↓ Agent identifies the bug

terminal("npm test") — tests fail
terminal("npm test -- --grep 'sorting'") — isolated failure

↓ Agent makes changes

writeFile("src/hooks/useSortJobs.ts") — fix the bug

↓ Agent verifies

terminal("npm test") — all pass

You can ask the agent to commit the changes & raise the PR, it'll ask for your approval and you can approve or reject it.

The lesson: Finishing the workflow matters as much as the individual steps.

Things That Didn't Work

Looking back, here's what I tried and abandoned:

Sending the Entire Repository

What I tried: Load every file, let the LLM figure out what matters.

Why it failed: Token limits, context confusion, hallucinations.

What replaced it: Intelligent indexing with repository structure awareness.

One Giant System Prompt

What I tried: Put all instructions in the system message.

Why it failed: Too long, contradictory, hard to maintain.

What replaced it: Modular instructions, runtime context, AGENTS.md support.

Letting the Model Edit Files Directly

What I tried: writeFile with no verification.

Why it failed: Syntax errors, broken imports, wrong files edited.

What replaced it: Plan mode + user approval before write operations.

Using the Biggest Model for Everything

What I tried: Always use the most powerful LLM.

Why it failed: Unnecessary cost, slower latency, no benefit for simple tasks.

What replaced it: Automatic model routing based on task complexity.

Skipping the Planning Phase

What I tried: Jump straight to implementation.

Why it failed: Logical errors, editing wrong files, breaking code.

What replaced it: Mandatory planning before any write operations.

No Checkpointing or State Management

What I tried: Each tool call was independent.

Why it failed: Lost conversation history, no audit trail, couldn't resume interrupted tasks.

What replaced it: SQLite checkpointing with LangGraph.

No Human Approval Mechanism

What I tried: Let the agent do everything automatically.

Why it failed: Users didn't trust it. Risky operations happened without consent.

What replaced it: ask_user tool for approval gates.

What I'd Do Differently

Start With Trust, Not Features

I built tools first and trust later.

I should have done it backwards.

Start by asking: "What operations do users need to approve? What's irreversible?"

Then build the approval mechanism first.

Measure Against Real Workflows

I tested with toy examples.

Real workflows are messier: interrupted tasks, conflicting edits, unclear requirements.

Testing against real repositories would have surfaced the planning problem months earlier.

Plan Mode From Day One

Every agent eventually needs planning.

Instead of bolting it on later, build it in from the start.

What's Actually True About AI Agents

After building Piper, here's what I believe:

The LLM is 20% of the work.

The other 80% is:

Context retrieval (finding the right files)
State management (knowing what happened)
Planning (thinking before acting)
Verification (did it work?)
User control (approval gates)
Error recovery (what to do when things break)

Bigger models help less than you think.

The difference between Flash and Flash Lite disappears when you give the LLM good context.

A smaller model with the right 5 files beats a bigger model with 50 irrelevant ones.

Human approval is non-negotiable.

Users trust agents that ask permission.

Agents that "just do things" lose trust fast.

Shipping teaches you more than planning.

I learned more from the broken agent than from architecture diagrams.

The planning phase, the timeouts, the state management — all came from failures, not design documents.

What This Means If You're Building Something Similar

Don't spend months perfecting your prompt.

Don't optimize model selection before you have state management.

Don't skip the planning phase.

Instead:

Build retrieval first (find the right context)
Add checkpointing immediately (preserve state)
Implement planning (think before acting)
Add approval gates (build trust)
Then optimize everything else

The fundamentals matter more than any particular LLM or framework.

Four months ago, I thought the hard part was choosing the right model.

Today I spend far more time thinking about context retrieval, state management, planning, and user trust than I do about prompts or which LLM to use.

That's the biggest lesson Piper taught me.

If you're building an AI agent, spend less time on prompts and more time on everything around it.

That's where the magic actually happens.

Try Piper

Want to see these lessons in action?

Piper is live at https://piper.codecoves.in

Clone a GitHub repository, ask it to fix a bug or add a feature, and watch it:

Search the right files
Propose a plan
Ask for approval
Make changes
Commit and open a PR

It's built exactly as described in this article — retrieval first, planning before acting, human approval for risky operations.

What's Next

Piper is intentionally a V1. My goal wasn't to compete with Cursor or Claude Code overnight. It was to build a complete end-to-end coding agent that could take a request from understanding a repository all the way to creating a pull request. There are dozens of features I still want to build, but I wanted to ship a solid foundation first and learn from real usage.

DEV Community