Building a Coding Agent from Scratch with Llama 70B: Lessons Learned

Aidan Do — Sat, 04 Jan 2025 03:51:34 +0000

I recently built a coding agent using Llama 3.3 70B (see demo on x). While my agent's performance on SWE Bench Lite (5%) is not that impressive compared to the leaders (48.33%), I'd like to share some of my learnings in case there's something here you didn't already know.

Why do this

If you take a look on the SWE bench lite leaderboard, 80-90% of the leaderboard are closed models. I want to see how far we can push open source models and am excited to see the point where people are using open source models for real coding assistant use-cases.

Key Learnings

1. Keep It Simple

Sticking to the simplest (sometimes brute force) solutions gave me a lot more control and visibility into the agent's behavior. For example, I could quickly gather operational metrics using simple string searches over logs:

Syntax error: 46
File not found: 114
Total errors (post tool call): 122
Total tool calls: 670
list_files tool calls: 245
edit_file tool calls: 78

I also left out bash tool and python execution at the start, because I saw on OpenHands, Llama 405b gets fairly lost when trying to execute arbitrary bash commands and python scripts. So I started with just 3 tool calls edit_file, list_file and view_file to start off with and I was able to get to 5% with just those three tools.

Anthropic says it pretty well here:

When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed.

source

2. The Power of Raw Prompts

Raw prompts seem intimidating at first, but they're pretty straight forward and give you way more power. For instance all chat completion APIs I could find with Llama, don't allow text content alongside tool calls. Looking under the hood, the standard Meta tool call prompt explicitly states not to have any text alongside tool calls:

If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]
You SHOULD NOT include any other text in the response.

source

With raw prompts, not only could I achieve thinking alongside tool calls, I got a lot less tool call parsing errors since I had control over the parsing logic and could parse tool calls that some implementations would have otherwise rejected.

I suspect this is a big advantage open source models have over closed source models - you have access to the prompt formats. Ashwin, one of the core maintainers of Llama Stack has also mentioned this point. Meta post their prompt formats, and you can view llama-stack for a reference implementation.

3. Less Context Can Be More

More context isn't always better. Initially, I included the entire repository file tree in the context, thinking it would help navigation. This not only cost me $250 in evaluation runs but actually hurt performance. Excluding the file tree completely led to:

2-3x performance boost in SWE bench pass rate
10x reduction in costs (from ~$30 to ~$3 for 50 instances)

4. Tool Names Matter

Something as simple as renaming a tool from replace_in_file to edit_file decreased empty patch files by 22% (from 77% to 60.3%). The agent also complained less about not having the right tools to solve problems. I suspect there are a lot of opportunities for improvement here - finding tools that LLM find easy to use.

Current Limitations

To be honest - at 5% on SWE bench lite, this agent isn't ready for production use. It's still far behind agents using Claude (30-50% range). And is still behind the top open source performers on SWE Bench Lite:

IBM AI Agent SWE-1.0: 23.67%
Moatless tree search (Llama 3.1 70B): 17.7%
OpenHands CodeAct (Llama 3.1 405B): 14%
OpenHands CodeAct (Llama 3.1 70B): 9%

But I'm optimistic, I see a lot of room for improvement in areas like:

High rate of empty patch files (59%)
Tool call errors
Unproductive looping

So I'm keen to see how far I can push this agent.

Conclusion

So those were the key learnings I got from doing this project. I hope you took something away from it. The code is open source at github.com/aidando73/l2-llama.

Feel free to comment below or message me on x.com if you have any feedback or questions.

DEV Community: Aidan Do