TLDR Overview:
- I was able to use Gemini and Claude to make a chart generator for StepMania / ITGmania that doesn't seem to be bad, if anything good
- It is rough around the edges, but ready for other folks to try out.
- The model is only
~33MB(100MBfull precision) , runs on CPU, and generates complete maps in< 20s - I want to build a new version of the generator and model built purely on opt-in charts
- I am looking for feedback and help from the general community to keep building this
The Story Begins...
At 3:35 a.m. on a Saturday I was sitting at my computer clacking away, sight-reading a Hard tier chart written by a model that did not exist eight hours earlier.
I scored 81%. Not the best, but the chart was... actually good? It felt intentional. It followed the song.
That model is now StepForge: a free desktop app that turns any song into Beginner through Challenge charts for StepMania and ITGmania. It runs on your CPU. No account, no API key, no upload, no cost per song.

A StepForge chart, exactly as the app wrote it, on autoplay in ITGmania. No hand edits.
- Download and site: step.fofo.dev
- Code, model card and training credits: GitHub
This post is the story of how I got here. It took four months of doing it the hard way, one weekend of doing it a different way, and a few moments where the numbers made me feel a bit silly.
The Game That Couldn't Chart Its Own Songs
This started in June with a neon rhythm game I was building for fun. I make music, so I had songs. What I did not have was charts.
The first charts were scripted by hand, and they were bad. Hardcoded BPM, repeating lane patterns, one track charted at 92 BPM when the song was actually 136. Haha!
So I did what a lot of us would do in 2026. I said: A.I. can listen to audio now. Let it write the charts.
Round One: Let the LLM Listen
My first brief to my coding agent was straight to the point:
"Use Gemini 3.5 Flash specifically for the LLM to analyze audio, that is non-negotiable and the whole point of this."
The design that came out of it was actually OK, and I still stand by the core rule:
Signal processing owns time. The LLM owns taste. Plain code owns legality.
The LLM never wrote a timestamp and never picked an arrow. Signal processing found every onset in the song and gave each one an ID. The model listened to the audio and could only say "use onset p017, skip m042." Anything it invented got rejected by the parser. Then a foot-flow solver turned those picks into steps a human with two feet can actually hit.
It worked! I charted packs of my own songs and played them. Over four months I tried six different charting models, including a Gemma fine-tune running on my own GPU.
The Bill
Here's where it got uncomfortable. About $80 to chart 30 songs. And I had no idea where the money was going.
So we instrumented one song. 18 model calls, $4.28. Everyone's first guess (mine included) was that attaching the audio was the expensive part.
| Where the money went | Share of the bill |
|---|---|
| The model "thinking" | 45.8% |
| Attaching the audio | 2.1% |
The audio was basically free. The thinking and the retries were the bill.
Then we tried the obvious fix, a smaller prompt. But then...:
"The prompt got 61% smaller and the bill went UP 17%."
Removing detail from the prompt made the model think harder and fail more checks, which meant more retries. Smaller input, bigger bill.
One more from this era, because it still makes me laugh. One audio model I tested would confidently describe audio it could not hear. It called a 3 kHz tone "a low-pitched drone" and heard wind in digital silence. So now I test every audio model with two tones and a forced choice: HIGH or LOW. If it can't get that right, it isn't listening.
By October I had a pipeline that was slow, pricey and fragile. It made decent charts. It was also being held together by its referees.
"Rhythm Charting Is Not a Language Problem"
On the evening of Friday, November 13 (yes, really), I asked a different question. What if we stop asking a language model to do this at all?
I pitched the idea to Gemini, and what came back was a spec for a small, purpose-built neural network. Around 22 million parameters. No LLM anywhere in it. The write-up afterwards summed up the whole idea in one line:
"Rhythm charting is not a natural language generation problem."
I handed that spec to Claude Code with one rule: no paid model APIs. Only my home lab and my Google Colab credits.
About 33 hours later there was a trained, benchmarked model. That includes a 92 minute pause because my login expired at 1:46 a.m. and nobody was awake to notice.
To be clear about who did what: I did not hand-write this network. A.I. designed most of it and A.I. built most of it. My job was the question, the constraints, the playtesting, and refusing to believe numbers until I understood them. That last part turned out to matter a lot, actually.
The pivot only took a weekend because most of the old project was still useful. The signal processing, the foot model, the quality checks, the file writer, the benchmark tools and the desktop app all carried over. Four months of "the hard way" was the scaffolding.
How StepForge Works
Three ideas do most of the work.
The music is laid out on a musical grid. 48 ticks per bar. The network never has to guess what a 16th note is, because the grid already knows.
The model thinks in feet, not arrows. Its vocabulary is 20 tokens like "other foot steps to its home panel" or "same foot again." A chart and its mirror image become the exact same sequence. Older approaches predict arrow columns and end up with a coin flip between left and right.
Playability lives outside the weights. The last stage is not a neural network. It is an exact search that will not output a spin, a step with a foot that is holding a freeze, or a double-step nobody asked for. That is why every chart is physically playable no matter what the networks say.

Preview a chart in the app, re-roll any section you don't like, then install to ITGmania with one click.
Three Times the Numbers Made Me Feel Silly
1. My clock was wrong for four months. While building the new model we found that my own signal processing had been writing every chart's offset about 70 ms early. Since July. An early benchmark "win" I was proud of was partly graded on that broken clock. There is now a correction note at the top of that old benchmark.
2. The first decoding method produced zero crossovers. The spec said to fill in the most confident steps first. That collapsed every chart into left, right, left, right. Filling in random order instead gave 4.2% crossovers. Human charts have 3.4%.
3. The zero. The first version scored poorly on Beginner charts. The plan said: too many notes. We fixed that. Nothing improved. Then a per-song table printed this:
mean Beginner F1 where the grid is right: 0.621
mean Beginner F1 where the grid is off: 0.0
On 27% of songs, the beat grid was sitting exactly half a beat off. Harder charts hide that because they use both halves of the beat. A Beginner chart only uses quarter notes, so every single step landed between the beats and scored exactly zero. The average of "pretty good" and "zero" looked like "mediocre."
Nothing simple in the audio could tell the two cases apart. The model could. Ask it "where would you put a Beginner step?" on the grid as given and on the grid shifted half a beat, and compare. That fixed 44 of 45 bad grids and broke none.
Look at your logs before you turn the knobs.
The Training Data Part
This is the part rhythm game folks will care about most:
The first research models were trained on a large archive of community simfiles. About 12,000 songs. That is how I found out the approach worked at all. Then I ran an audit of what was actually in there, and 41.5% of it was Konami-related, including official DDR charts.
I was not comfortable releasing weights trained on that. Those models have not been released and will not be.
But the audit set up the most useful experiment of the project. How much data does this ACTUALLY need to be good still?
| Trained on | Songs | Step placement vs human charts (F1) |
|---|---|---|
| Research model, full archive (not released) | 12,055 | 0.713 |
| Research model, everything Konami-related removed (not released) | 7,058 | 0.708 |
| Research model, small set (not released) | 1,000 | 0.686 |
| Research model, small set (not released) | 100 | 0.637 |
| StepForge Flow v1.4, the one that ships | 227 | 0.656 |
| Best open-source auto-charter I tested | 0.674 | |
| Two human charters, same song | ≈ 0.75 |
Removing every Konami-related song cost half a point. And the playability numbers did not move at all, even at 100 songs, because those come from the foot search and not from the data.
So the model in the app, v1.4, was trained from scratch on a small open set: 254 songs from the StepMix 1 to 5 community contest packs, OutFox Serenity, and the StepMania 5 default songs (227 for training, 27 held out to check it).
I want to be careful here and not oversell it:
- Licenses vary by song. Some are Creative Commons. Most are contest entries that were cleared for redistribution. Most of those licenses say nothing about machine learning either way.
- Every song is credited. Artist, step artists, pack and license, in CREDITS.csv.
- If your song or chart is in there and you want it out of future models, tell me and it is gone. Open an issue or reach out.
- Every chart the app writes is credited "StepForge (AI-generated)." Please do not upload them to human-curated simfile archives as if a person made them.
- The weights are non-commercial (CC BY-NC-SA 4.0). The code is MIT. The app is free.
Where I want this to go is a model trained only on songs and charts people have chosen to contribute. If you are a step artist or musician and want in, I would love to hear from you.
So... How Good Is It?
I charted 180 songs that already have human charts, without showing the model those charts, and compared step by step.
| 180 held-out songs | StepForge v1.4 | Best open-source tool tested | Two humans |
|---|---|---|---|
| Steps where the human put one (F1, ±20 ms) | 0.656 | 0.674 | ≈ 0.75 |
| Beginner | 0.519 | 0.587 | |
| Hard / Challenge | 0.714 / 0.726 | 0.703 / 0.712 | |
| Chart statistics inside the human range | 98% | 90% | 94% |
| Unforced double-steps | 0.01% | 1.04% | 0.35% |
The honest read:
- On placement it is level with the best open-source tool overall. The small lead at Hard and Challenge is not statistically significant.
- Beginner is its weakest difficulty, and it is clearly behind there. Easy is next. If you play on Beginner, you will notice.
- Where it clearly wins is playability. The charts are clean to dance and feel intuitive (at least for me).
- It is not as good as a skilled human charter. Two humans agree with each other more than it agrees with either of them.
I also checked whether it copies its training charts. It does not. Its longest exact copy of a training chart is 9 steps, and it matches charts it trained on no better than charts it never saw. The details are in the model card.
What I'd Tell Another Developer
Make the model select, not generate. Whether it was Gemini or a tiny custom network, the best results came when the model chose from options that were already valid. Time and legality were never its job.
Your token bill is thinking and retries, not always inputs. Measure before you optimize. My "obvious" cost fix made things worse.
Don't trust metrics that grade your system against itself. Two of my early headline numbers did exactly that. Benchmark against something you did not make. For me that was human charts.
Build a ceiling and a floor. "0.656" means nothing alone. Next to "two humans agree at 0.75" it means something.
Keep track of where your data came from, starting on day one. Because I had that, the audit, the credits file and "remove this song and retrain" were each one command. If I had not, the responsible version of this release would have taken weeks instead of a day.
The not-an-LLM option is real. A.I. helped me design and build this. The thing it built is 25 million parameters and runs on a laptop CPU. Sometimes the best use of a big model is building the small one.
Beyond StepMania / ITGmania
The hope is that a similar approach will work well for other music games like Beat Saber, etc. If we can prove out a process and technical approach for this, we can do other games.
Try It
StepForge is in public beta for macOS, Windows and Linux. Expect rough edges! I typically test software more than this one got a run at, but it has been working well on Windows so far. macOS is where it was built, so that should be sturdy too. Linux is the unknown.
- Download: step.fofo.dev
- Code and model card: github.com/Sweet-Papa-Technologies/StepForge
- Feedback, bugs, "this chart did something weird": Discord or GitHub Issues
Drop in a song you love that nobody has ever charted, and tell me how it plays. Especially if it plays badly. That is how the next one gets better.
And if you chart or make music and want your work in an opt-in model, or want it kept out, reach out. I am listening.
Sweet-Papa-Technologies
/
StepForge
Turn any song into playable StepMania / ITGmania dance charts, on your own computer. Powered by StepForge Flow, a small neural model that runs locally.
StepForge
Turn any song into playable StepMania / ITGmania dance charts, on your own computer.
Drop in a song, and StepForge writes Beginner through Challenge charts for it in a few seconds, then installs them straight into ITGmania. The charting is done by StepForge Flow, a small neural model that ships inside the app and runs on your CPU: no account, no API key, no upload, no per-song cost.
Beta. StepForge is new. Expect rough edges, and tell us about them on Discord or in Issues.

A StepForge chart (Hard) of Lucky Lucky by Sweet Papa & the Tones, played on autoplay in ITGmania
More footage: docs/media.
Download
Get the latest build from step.fofo.dev/download or the Releases page.
| Platform | File | Notes |
|---|---|---|
| macOS 12.3+ (Apple silicon: M1 and later) | StepForge-…-macos-arm64.dmg |
Signed and notarized by Apple |
| macOS 12.3+ (Intel) | StepForge-…-macos-x64.dmg |
Signed and notarized by Apple |
| Windows 10 / |
Tech stack for the curious: Tauri v2 + Vue 3 (desktop app), Python + ONNX Runtime (charting engine, no PyTorch at runtime), a Conformer (RhythmNet) and a masked discrete diffusion transformer (ChoreoNet), trained on Google Colab A100s. Built with Claude Code - Fable 5.2, designed with help from Gemini 4 - Carbon on High thinking.
StepForge is made by Sweet Papa Technologies. It is not affiliated with StepMania, ITGmania, OutFox or any game publisher.




Top comments (0)