NotNull

Posted on Jun 23

Building a Lightweight Unity Editor Bridge for AI Coding Agents

#unity3d #gamedev #ai #opensource

AI coding agents are getting better at writing code.

But when I started using them seriously with Unity, I kept running into the same problem:

They could edit C# files, but they could not reliably know what happened inside the Unity Editor.

In Unity development, the file system is only part of the truth.

The real questions are often:

Did Unity compile the project?
Are there Console errors?
Which scene is active?
Does this GameObject actually exist?
Did the component value change in the Inspector?
Can Play Mode start?
Do the EditMode or PlayMode tests pass?
If the agent changed UI, does it actually look right?

Without access to the running Editor, an AI agent has to guess.

That is why I started building hera-agent-unity.

GitHub:
https://github.com/NotNull92/hera-agent-unity

What is hera-agent-unity?

hera-agent-unity is an open-source Go CLI plus Unity Editor package.

It lets AI coding agents talk to a running Unity Editor over localhost HTTP.

The agent can call commands such as:

hera-agent-unity status
hera-agent-unity console --type error --lines 20
hera-agent-unity scene info
hera-agent-unity editor play --wait
hera-agent-unity test --mode PlayMode
hera-agent-unity find_gameobjects --ids

It can also execute C# inside the Editor:
hera-agent-unity exec 'return UnityEditor.EditorSceneManager.GetActiveScene().name;'
The goal is not to make AI “magically understand Unity.”
The goal is more practical:
Let the agent edit the project, ask the real Unity Editor what happened, fix problems, and verify again before saying done.

Why not just use MCP?

MCP is useful. I am not against it.
But for this project, I intentionally chose a CLI-first design.
The reason is that Unity Editor automation has a very specific workflow.
Most of the time, I do not need a general tool protocol, server negotiation, or large structured responses.
I need a coding agent to ask small, repeatable questions:

Is the Editor ready?
Did the project compile?
Are there recent Console errors?
What objects exist in the scene?
Can Play Mode start?
Did my change actually apply?

For that loop, response size matters.
AI agents are limited by context, latency, and cost. If every Editor operation returns a huge object graph or verbose schema, the tool becomes less practical to use repeatedly.
So Hera is designed around compact commands.

For example:

list --compact is around 93 tokens
find_gameobjects --ids is around 49-55 tokens
side-effecting commands usually return just OK

That may sound small, but it changes the workflow.
If a command is cheap enough, the agent can call it often.
That means the agent can verify instead of guess.

The workflow I wanted

The workflow I wanted looks like this:

AI agent edits Unity code
        ↓
asks the real Editor what happened
        ↓
checks compile / Console / scene state
        ↓
fixes errors
        ↓
verifies again
        ↓
only then says done

This is especially important for Unity because many errors do not appear as ordinary compiler feedback inside the text editor.
A script may compile, but the scene may still be wrong.
A prefab may exist, but the Inspector value may not be what the agent intended.
A UI may be technically created, but visually broken.
A Play Mode issue may only appear when the Editor actually runs.

Agent-specific rules

Another part of the project is agent rule generation.

Hera can generate project rules for tools such as:

Codex
Claude
Cursor
GitHub Copilot
Google AntiGravity

The idea is simple.
When an AI coding agent enters the project, it should immediately know:

which Hera commands to use
how to inspect Unity state
how to avoid returning huge Unity objects
how to read Console errors
how to verify before claiming completion

For example, instead of returning a full GameObject or Transform, the rules push the agent to return compact identifiers:

var go = GameObject.Find("Player");
return new { name = go.name, instanceID = go.GetInstanceID() };

Small details like this matter a lot when the tool output becomes part of the agent context.

Ultra Hera

The newest feature I added is called Ultra Hera.
It is a verification workflow for AI-assisted Unity work.
It does not do the AI work by itself. Instead, it tells the agent how carefully to check its Unity work after using Hera.
There are two main modes.

Light Mode

Light Mode is the default.

It nudges the agent to:

confirm the goal
read only the needed Unity state
make the change
verify compile or state
check recent Console errors
re-read only the changed target
retry once or twice if needed
report short evidence

This is meant for normal Unity coding, Editor, and Inspector tasks.

Ultra Mode

Ultra Mode is for stricter requests.

For example:
“play it and confirm”
“check the Inspector too”
“match this UI”
“run the tests”
“verify this exactly”

In those cases, the workflow can escalate to:

Play Mode checks
EditMode or PlayMode tests
screenshot capture
UI capture
deeper Inspector or asset state checks

The point is not to make every task heavy.
The point is to match the verification depth to the risk of the task.

UI Juicy Mode

One of my favorite parts is UI Juicy Mode.
AI-generated UI often works, but feels static.
It can place buttons, panels, and labels, but it may forget the game-feel layer that makes UI feel alive.
When UI Juicy Mode is enabled, Hera can send game-feel recipes to the agent through agent_hint.

Examples include:

button hover feedback
press squash
release bounce
popup overshoot
damage text motion
progress bar feedback
lightweight animation suggestions

So the agent does not stop at placing UI elements.
It gets nudged toward making UI that feels more like game UI.

Unity version support

I also spent time checking Unity versions separately.
So far I have verified:

Unity 2022.3 LTS
Unity 2023.2
Unity 6.3
Unity 6.5

Dogfooding

I am dogfooding Hera while building my personal Unity game project, NoMoreRolls.
The README includes a Play Mode preview, because I wanted to show the tool being used in an actual Unity workflow rather than only as a command list.

What I learned

The biggest lesson from building this is that AI-assisted development is not just about code generation.
For real projects, the more important question is:
Can the AI safely operate inside the actual product environment?

For Unity, that means the Editor.
For web apps, it might mean the browser.
For backend systems, it might mean logs, tests, local services, databases, or deployment previews.
The pattern is the same:
The AI should not only write changes.
It should be able to observe the system, verify the result, and correct itself.
That is the direction I am exploring with hera-agent-unity.

Repository