DEV Community

Takashi Mori
Takashi Mori

Posted on

I Asked an AI Agent to Build Robot Simulators — Here's What Happened

#ai #opensource #robotics #ros

AI is making simulator development much easier.

Today, we can ask AI to work with high-fidelity physics simulators such as MuJoCo, or use tool integrations to help build simulation environments that previously required a fair amount of specialized knowledge.

But when you actually try to build a simulator for your own project, the requirements quickly become more specific:

"I want to use an existing ROS URDF."

"I want to reuse ROS 2 message definitions."

"I want to visualize it in a web browser."

"And I want it to run on my Mac."

Each requirement by itself is not particularly difficult.

The hard part is combining all of them into the simulation environment you actually want.

So I tried an experiment:

Can an AI coding agent build working robot simulators from high-level requirements by combining existing open-source components?

For this experiment, I used Hakoniwa together with an AI coding agent (Codex).

Here is the latest demo:

Interestingly, this article itself follows the same approach: the AI coding agent that worked on the simulations helped draft the article, and I edited and refined it as a human. Later in the article, I also asked the AI to reflect on what made this development process work.

What I built

I tried the same AI-driven development approach with three very different robots:

  • AgileX Tracer — converted a model derived from its ROS URDF for MuJoCo and controlled it using a PDU based on ROS 2 Twist
  • FR5 robot arm — turned the arm into a Hakoniwa asset and controlled it using a PDU based on JointTrajectory
  • Unitree Go1 — used the MuJoCo Menagerie model as a Hakoniwa asset and executed open-loop motion for its 12 joints

I also used the existing Hakoniwa Drone with a Three.js web viewer.

Before going into the experiment, let me briefly explain what Hakoniwa is.

What is Hakoniwa?

Hakoniwa is an open-source platform for composing robot simulators, control applications, visualization tools, and other software components into a simulation environment.

For example, you can combine:

  • a physics simulator such as MuJoCo
  • ROS / ROS 2 applications
  • visualization applications running in a web browser
  • custom control programs

Hakoniwa treats these as independent components called assets.

Instead of implementing everything inside one monolithic simulator, the basic idea is:

Combine the components you need to build the simulation environment you want.

The components exchange data using structures called PDUs (Protocol Data Units).

This separation turned out to be particularly useful when working with AI.

Development environment

I built these simulations on the Mac I normally use for development.

The main environment was:

  • macOS
  • MuJoCo
  • Hakoniwa
  • Codex as the AI coding agent

I did not use a dedicated simulation workstation or a special GPU environment.

Also, I wasn't simply asking AI to operate an already completed simulator.

The goal was more ambitious:

Start from a requirement, investigate the necessary software and robot models, design the architecture, implement it, and get the resulting simulation actually running.

To help the AI understand the Hakoniwa ecosystem, I used a repository called Hakoniwa Business Pack.

It contains catalogs, recipes, runtime knowledge, and other information intended to help both humans and AI agents understand how Hakoniwa components can be combined.

First experiment: AgileX Tracer

The first robot I tried was the AgileX Tracer.

A ROS repository for Tracer is publicly available and includes a URDF model.

My requirements were simple:

  • reuse the existing ROS URDF
  • use MuJoCo instead of Gazebo
  • run it on macOS
  • use a control interface based on the common ROS 2 Twist message
  • avoid making the asset unnecessarily specific to Tracer
  • first make it work in MuJoCo, then turn it into a Hakoniwa asset, then control it from Python

In other words:

"I want to take this existing ROS robot model, run it in MuJoCo on my Mac, and control it using a Twist-like velocity command."

I gave those requirements to the AI.

I did not tell it:

Edit this file, create this C++ class, and call this API.

Instead, I first asked it to understand the Hakoniwa Business Pack.

Understand hakoniwa-business-pack.
Enter fullscreen mode Exit fullscreen mode

The AI read its README, catalog, recipes, and documentation to understand the available components, runtime conventions, and existing examples.

Then I asked:

Using the AgileX Tracer URDF,
create a recipe for building a Hakoniwa simulation environment.
Enter fullscreen mode Exit fullscreen mode

The architecture that emerged looked roughly like this:

AgileX Tracer URDF / robot model
        ↓
hakoniwa-mbody-registry
        ↓
MuJoCo-compatible model
        ↓
hakoniwa-mujoco-robots
        ↓
Hakoniwa asset
        ↑
Twist-derived PDU
        ↑
Python sender
Enter fullscreen mode Exit fullscreen mode

The important point is that the Python program does not directly manipulate MuJoCo.

MuJoCo runs as a Hakoniwa asset.

The controller sends commands through a PDU based on ROS 2's geometry_msgs/msg/Twist.

This allows us to reuse concepts and data definitions from ROS while composing the simulation itself using MuJoCo and Hakoniwa.

What the AI actually did

Based on the requirements and existing Hakoniwa knowledge, the AI worked through tasks such as:

  • inspecting the existing Tracer URDF and converted models
  • checking the minimum world required to load the model in MuJoCo
  • adding actuators for the left and right wheels
  • preparing a PDU corresponding to geometry_msgs/Twist
  • implementing a Hakoniwa asset that converts Twist commands into wheel velocities
  • implementing a Python sender for linear.x and angular.z
  • documenting the execution procedure
  • updating the recipe with what had actually been validated

For example, the resulting command interface could be used like this:

python3.12 examples/actuators/agilex_tracer/send_rover_twist.py \
  --linear-x 0.2 \
  --duration-sec 4
Enter fullscreen mode Exit fullscreen mode

The execution sequence also matters in Hakoniwa.

First, start the MuJoCo asset:

./src/cmake-build/examples/actuators/agilex_tracer/rover-twist-hakoniwa-asset
Enter fullscreen mode Exit fullscreen mode

The asset registers itself and waits:

hako_asset_register :RoverTwistAsset
asset(RoverTwistAsset) is registered.
WAIT START
Enter fullscreen mode Exit fullscreen mode

Then start the Python sender:

python3.12 examples/actuators/agilex_tracer/send_rover_twist.py \
  --linear-x 0.2 \
  --duration-sec 4
Enter fullscreen mode Exit fullscreen mode

It also waits:

Rover Twist sender is registered.
WAIT START
Enter fullscreen mode Exit fullscreen mode

Once the assets are ready, start the simulation:

/usr/local/hakoniwa/bin/hako-cmd start
Enter fullscreen mode Exit fullscreen mode

The Python sender then publishes the Twist PDU, and the Tracer moves forward in MuJoCo.

The asset logs showed the command and the changing base position:

Rover Twist Hakoniwa asset started.
time=0.502 base=(0.017, 0.000, 0.142) cmd=(0.200, 0.000)
time=1.002 base=(0.055, 0.000, 0.142) cmd=(0.200, 0.000)
time=2.002 base=(0.167, 0.000, 0.142) cmd=(0.200, 0.000)
Enter fullscreen mode Exit fullscreen mode

So the complete path was:

ROS URDF
  → MuJoCo model
  → Hakoniwa asset
  → Twist PDU
  → Python sender
  → movement in the MuJoCo viewer
Enter fullscreen mode Exit fullscreen mode

And all of this ran on my Mac.

The failures were useful too

The AI did not get everything right on the first try.

For example, after converting the robot model, a large wheel-like primitive appeared near the center of the robot and affected its posture.

This led to a useful lesson: when primitives or collision geometry are added during URDF/MJCF conversion, they can affect both visualization and contact behavior.

Instead of treating this as a one-off fix, we fed the knowledge back into the Hakoniwa Business Pack.

This became an important pattern during the experiment:

The development loop

  1. The AI reads existing knowledge.
  2. The AI builds and runs something.
  3. A failure or missing assumption appears.
  4. Human and AI investigate it together.
  5. The result is turned into reusable knowledge.
  6. The next AI session starts from a better baseline.

That feedback loop became at least as interesting to me as the simulator itself.

Trying the same approach with a robot arm

Next, I wanted to see whether the same pattern would work for a completely different type of robot.

I tried the FR5 robot arm.

Instead of velocity commands such as Twist, an arm needs coordinated joint motion.

So we used a PDU based on ROS's trajectory_msgs/JointTrajectory.

Again, I gave the AI requirements at the architectural level:

I want to run the FR5 arm as a Hakoniwa asset.
I want the interface to be based on JointTrajectory.
Let's begin by replaying a predefined trajectory.
Enter fullscreen mode Exit fullscreen mode

The resulting structure was:

FR5 URDF / MuJoCo model
        ↓
Hakoniwa arm asset
        ↑
JointTrajectory-derived PDU
        ↑
Python trajectory sender
Enter fullscreen mode Exit fullscreen mode

In the demo, the Python sender transmitted a nine-point joint trajectory.

The asset reported:

Accepted JointTrajectory: joints=6 points=9
Enter fullscreen mode Exit fullscreen mode

and the arm replayed the trajectory in the MuJoCo viewer before returning to its home pose.

The robot was completely different from Tracer, but the Hakoniwa runtime pattern remained largely the same:

register asset → wait for start → exchange PDUs → run simulation

Then a quadruped: Unitree Go1

The third experiment was the Unitree Go1.

This robot has four legs and 12 controlled joints, making it significantly different from both a rover and a robot arm.

For this experiment, I used the Go1 model from MuJoCo Menagerie.

The control interface was based on a 12-element std_msgs/Float64MultiArray:

Go1 MuJoCo Menagerie model
        ↓
Hakoniwa quadruped joint asset
        ↑
Float64MultiArray[12]-derived PDU
        ↑
Python open-loop motion sender
Enter fullscreen mode Exit fullscreen mode

One important lesson here was that the ordering of the 12 joint values matters. If the ordering is wrong, individual legs move in unexpected ways.

Another important distinction is terminology.

What we implemented was open-loop motion playback.

It was not a walking controller that dynamically balances the robot against terrain or disturbances.

We explicitly recorded that distinction in the Business Pack knowledge so that future AI sessions would not incorrectly describe the demo as a walking controller.

Three robots, one development pattern

The robots and their control interfaces were quite different:

Robot Type PDU Control
AgileX Tracer Mobile robot Twist Linear/angular velocity
FR5 Robot arm JointTrajectory Joint trajectory playback
Unitree Go1 Quadruped Float64MultiArray[12] Open-loop joint motion

But the process followed by the AI was surprisingly similar.

It repeatedly had to answer questions like:

  • What existing assets can be reused?
  • Which Hakoniwa components are needed?
  • What should the control PDU look like?
  • What is the smallest useful simulation we can validate first?
  • How do we know that the result actually works?

This is where the Hakoniwa Business Pack became important.

A second demo: the three robot experiments together

The first video at the beginning of this article shows the latest result.

I also recorded an earlier demo that focuses specifically on the three robot experiments described above — AgileX Tracer, FR5, and Unitree Go1.

Seeing these three very different robots side by side helped confirm something important: the robot-specific control interface changes, but the overall development pattern stays surprisingly consistent.

Why the Hakoniwa Business Pack matters

Hakoniwa itself provides many components, but from an AI agent's perspective, a large ecosystem can be difficult to navigate.

Which repository should it inspect?

Should it use the PDU Registry or the MBody Registry?

Who starts the Conductor?

When should hako-cmd start be executed?

Is a command in a README just a standalone example, or part of a validated workflow?

The Hakoniwa Business Pack acts as a map.

Conceptually, it organizes information into things such as:

Catalog
  → What components exist?

Recipes
  → How can components be combined?

Runtime Primer
  → What are the runtime conventions?

Knowledge
  → What did previous experiments teach us?
Enter fullscreen mode Exit fullscreen mode

This changes how an AI approaches the ecosystem.

Instead of opening random repositories and trying to infer everything from source code, it can start from the user's goal and work backward.

I asked the AI what it thought

After completing the experiments, I asked the AI coding agent a question:

What did Hakoniwa make possible, and what was useful about the Hakoniwa Business Pack from an AI agent's perspective?

Its answer was interesting.

The AI explained that without Hakoniwa, it would naturally tend toward one of two approaches:

  1. implement everything inside MuJoCo, or
  2. build a dedicated bridge directly between ROS 2 and MuJoCo.

Either approach tends to require designing the robot model, communication, timing, process startup, data formats, and visualization as one custom system.

Hakoniwa allowed the AI to decompose the problem:

Physics simulation → MuJoCo asset
Control application → Python / ROS / external app
Communication → PDU
Execution management → Hakoniwa runtime
Visualization → MuJoCo viewer / Three.js viewer
Enter fullscreen mode Exit fullscreen mode

Instead of building everything as one system, the AI could reason about which components should be connected.

For Tracer:

Twist → rover asset
Enter fullscreen mode Exit fullscreen mode

For FR5:

JointTrajectory → arm asset
Enter fullscreen mode Exit fullscreen mode

For Go1:

Float64MultiArray[12] → joint target asset
Enter fullscreen mode Exit fullscreen mode

This common structure also made debugging easier.

If something failed, the AI could investigate separately:

  • Is the robot model wrong?
  • Is the actuator configuration wrong?
  • Is the PDU arriving?
  • Was hako-cmd start executed at the wrong time?
  • Is the problem only in the viewer?

That decomposition is particularly useful for an AI coding agent.

Turning implicit knowledge into reusable knowledge

The most interesting part, however, was not simply that the AI could use the Business Pack.

It could also improve it.

During development, I had to teach the AI several pieces of Hakoniwa-specific knowledge.

For example:

  • don't call hako-cmd start before the assets are registered
  • wait until the asset reaches WAIT START
  • use the Python 3.12 environment where hakopy is installed
  • using Twist makes the Tracer implementation easier to generalize into a rover asset
  • the Go1 command contains 12 joint values and their ordering matters
  • Go1 open-loop motion should not be described as a walking controller
  • the Drone mission should be started after the viewer connects

If these lessons remained only in a chat session, the next AI session could make the same mistakes again.

Instead, we moved useful lessons back into recipes, documentation, and knowledge in the Business Pack.

So the development process started to look like this:

From one task to accumulated knowledge

1. Human requirement

Describe what should be achieved, not how every file should be implemented.

2. AI reads Catalog / Recipes / Knowledge

The agent starts from the ecosystem map instead of exploring repositories blindly.

3. AI proposes a simulation architecture

It chooses reusable assets, PDUs, models, and runtime components.

4. AI implements and runs it

The result is not considered complete just because the code was generated.

5. Human and AI inspect the failures

Runtime ordering, model conversion, actuator settings, PDU semantics, and other implicit assumptions become visible.

6. Reusable knowledge is updated

Useful discoveries are promoted into recipes, documentation, validation scripts, or runtime knowledge.

7. The next task starts from a stronger baseline

The AI was not just generating code. It was helping build the knowledge required for the next AI agent to do a better job.

What I learned

AI certainly makes writing simulator code easier.

But after these experiments, I don't think code generation is the most interesting part.

The harder problem has always been integration:

I want to use this URDF.

I want to reuse ROS assets.

I want MuJoCo for physics.

I want it to run on macOS.

I want web visualization.

I want to connect my own control application.

The challenge is not any single technology.

The challenge is combining multiple technologies according to the user's actual requirements.

Hakoniwa turns simulators, control applications, communication, and visualization into independent components.

The Hakoniwa Business Pack then describes those components and their composition patterns in a form that an AI agent can reason about.

That means a human does not necessarily need to begin with:

"Use this API, create this class, edit this configuration file..."

Instead, we can increasingly begin with:

"This is the robot I have, this is the environment I want, and this is how I want to control it."

The AI can then investigate existing assets, choose components, build the architecture, implement it, run it, and learn from the result.

That is what I mean by AI-driven simulator development.

Not just AI generating simulator code.

But AI helping move from:

requirements → architecture → integration → execution → validation → reusable knowledge

There is still a lot to improve, but after seeing the same pattern work with a rover, a robot arm, a quadruped, and a drone, I think this direction is worth exploring further.

The Hakoniwa Business Pack is open source, so if you're interested in AI-driven robotics simulation, feel free to take a look:

https://github.com/hakoniwalab/hakoniwa-business-pack

And here is the demo again:

I'd be very interested to hear how other robotics developers are using AI agents for simulation development — especially where integration, rather than code generation itself, is the difficult part.

Top comments (0)