Familiar Beginnings
A few months ago I decided to finally start making my dream game in Unreal Engine 5, a 3D action RPG game. With some purchased Udemy lessons I was off to the races, and started the learning process. I am a software developer by trade, so at least the coding shouldn't be a problem. Everything went fine for a week or two, until I hit an issue where my game did not work like the one in the tutorial video despite following all of the steps exactly.
Solving even the smallest of problems at this stage of learning is immensely difficult, you are completely new to a massive set of complex interconnected systems and have no idea how they work. My gut tells me it's not a fundamental issue, I followed the tutorial step by step so it's likely something small. Some small error I didn't see, something that needs tweaking to work in a different version perhaps. So I think to myself, I will have my good friend Claude help me with this issue as I am sure it's something simple. And this is where the journey really begins.
The Problem Emerges
You see the problem is, inside Unreal Engine everything is serialized binary data, stored as a .uasset file. So when I fire up Claude Code to have him review my blueprints, he can't. I search all over the internet, scouring in shock and dismay that somehow this is not a well solved problem. How is it even possible there isn't a full solution for this already, that an LLM can understand what's inside a .uasset? Admirable folks had begun putting up the foundations, but the gap was still very real. I started with the UnrealClaude plugin https://github.com/Natfii/UnrealClaude as the base (shoutout to Natfii, he created a lot of the base capabilities I built on top of, could not have done it without him). It has capability to create blueprints, but limited capability to parse existing ones as an exposed operation. So I leverage existing code in the plugin, and extend it so it can read blueprints in Unreal. And after that — aha! One of the boxes in one of the nodes was 1 when it should have been 10, Claude caught it first try once he could see it. Roadblock removed, I am back on track!
But as the tutorial progresses in complexity, the "I did it exactly as the tutorial yet it doesn't work" just keeps popping up. No fault on the tutorial maker, Unreal Engine is very complex and version updates sometimes break things. But again, solving these with limited knowledge of Unreal is really tough. Documentation, Google, forums, and tutorials are all talking about different situations with mixed advice, largely about systems you don't know. So I turn back to my friend Claude. Not so he can fix the problem, so he can help me navigate these complex systems to help me solve the problem myself. But there is another issue in our way first, Claude can't read animation montages or animation graphs so he can't help me break this issue down. So it's back to the plugin code, to teach Claude how to parse and dump these things into a format he can work with. Get that working, get all the info now in one place, and Claude can now correlate info from code, blueprints, montages, and helps me identify root cause. Problem solved. Using this I am able to finally finish the tutorial, feels like progress!
The Real Work Begins
I make my new project for the actual game and get to work. But immediately the workflow itself feels so slow and clunky. I know what I am doing now well enough, I know what I would need to click and enter to set up a new feature but that would take forever. It would be faster if I just taught Claude how to write things now instead of just read. Not to put the creation details in his hands, to give him the ability to execute my plans efficiently. So it's back to the plugin code, give him the ability to write blueprints, animation montages, anim graphs.
Now my pace of implementation is starting to speed up. I can playtest or tune while Claude is planning or implementing the next feature (with my very specific prompts on implementation details). I start to see this plugin as a time investment, every gap I encounter I immediately cover. Can't see blendspaces? Back to the plugin code. Can't modify Control Rigs? Back to the plugin code. Piece by piece its capabilities grow. I add more complex handling for debugging. It should be able to correlate image captures from a run with logs for debug? I write a low latency frame grabber, no "take a screenshot" is going to work here the overhead is too much. Now we can get per-frame image captures for granular analysis. I solve many bugs, my combat improves, I am heading in the right direction.
Optimizing the Solution
Claude can now see all, but seeing all is starting to cost a lot of tokens. It's still within my plan limits, but it's obviously not scaling well. It also feels entirely unnecessary. I watch Claude work to form curl commands over and over sending commands to the plugin, clearly spinning and guessing parameters, although he does usually get there eventually. Each response is a full JSON parse, each blueprint graph problem it reads the entire graph just to inspect one node. Now our tooling needs tooling, but that's ok. I write a little interface layer, just a script in between so he doesn't construct raw curl requests or parse whole JSONs, that gets pushed into the script. Token usage cut more than half. Put in some more functionality for node querying capabilities, now he doesn't parse a whole blueprint every time, token usage cut by nearly half again. I create a domain system to compartmentalize all of the very useful Unreal Engine specific knowledge we acquire.
Open Source Release
At this point I think I have really built something that others might also find valuable, so I am releasing this open source as the new Unreal Engine LLM Toolkit (https://github.com/ColtonWilley/ue-llm-toolkit). I will be continuing to build it out as I need to for my game, so it will be receiving updates as I need new capabilities.
Top comments (1)
I have been trying to develop games with AI assistance or even relying primarily on AI, but currently, I can only create very simple games. What open-source resources are available on GitHub regarding game engines and utilizing AI programming tools to assist in game development? I am also looking to gain a deeper understanding and learn more about this field.