DEV Community: Joe Buckle

Translating a Japanese Sega Saturn Game From 1998: The User Interface

Joe Buckle — Tue, 04 Aug 2026 17:40:48 +0000

This is part 2 of a 3 part series, probably...

Part	Title
1	Translating a 1998 Japanese Visual Novel on Sega Saturn I Bought by Accident
2	Translating a Japanese Sega Saturn Game From 1998: The User Interface
3	...incoming

It is worth noting up front that without AI I don't think this project would be possible for me. It has been instrumental in discovering the source code and figuring out where all this stuff goes.

Part 1 ended with a Sega Saturn in the corner of the room playing a Japanese visual novel in English.

The dialogue worked. That was the important milestone. The characters were speaking English, the script was extracted, translated, rebuilt, and running on the original hardware.

But of course when I tried to play it the dialogue was English but everything around it was still Japanese.

Menus. Locations. Profile cards. The save screen - the parts of the game that tell you what to do.

The UI was about to be a very different problem.

The text that isn't in the script

The dialogue was relatively straightforward because it lived in a place designed for dialogue.

This wasn't the case for the user interface.

The menus and labels are stored inside the game's executable as raw strings. They are not in the script files, and they are not stored in a nice, editable format.

Each string sits in a fixed space, with the next string immediately after it.

The UI was built around fixed Japanese labels, with no extra space reserved for longer text in another language, so English strings simply overflow into whatever is next to them.

アルバイト

fits perfectly.

Part-time Job

does not.

The extra bytes have to come from somewhere.

This is what it looks like when it goes wrong. That label should say one thing. It is drawing several things at once, on top of each other.

The first UI mystery

Luckily, some English text already looked better than expected.

If you read the last article, you'll know the dialogue renderer needed a patch because every character was treated as the same width.

The UI was different.

Some labels were compressed. Others were spaced out.

Before I had it looking like the above, P L A N S was letter-spaced across a wide gap and overflowing its container.

The game actually contains two versions of its font. A wide one at 24 pixels for Japanese, and a narrow one at 16 pixels that covers everything that isn't a kanji, including the full Latin alphabet.

The renderer automatically chooses between them depending on the character being drawn.

This was something the original developers had already solved in 1998!

They built an interface that could mix Japanese and English characters, even though English was never going to be the primary language. Lucky us!

When English exposes old assumptions

The problems started appearing when English got longer.

One menu centres its text using a simple calculation:

x = (96 - advance * length) / 2 + 4;

96 pixels of usable width, divide the leftover space, nudge it right by 4. Perfectly sensible.

Then you give it a seven character English word:

// "PROFILE" -> 7 characters at 24 pixels each
x = (96 - 168) / 2 + 4;
// x = -32

There is no safety check, I'm guessing because there never needed to be. No Japanese label was ever long enough to go negative. Mine were, immediately, and the label starts drawing outside its own area.

That title plate is meant to say "Travel Map". The text has escaped the box and collided with whatever was already on screen.

So rather than making this a complicated rom hack, I decided to get Claude to change the words so they still made sense but fit the rules the original game expected.

STATUS survives. PROFILE does not.

The plan was simple

Most UI strings have pointers - so the obvious solution would be to move the longer English string elsewhere and update the pointer.

This seems to be a common technique in reverse engineering.

And mostly, it worked. Locations, menu entries, options, 64 profile fields. All moved without complaint.

Then the schedule screen locked the game solid.

I got this wrong three times. I blamed the line length, then the memory region I was writing into, then the renderer. What I should have done first was build a version with no UI changes at all, to see whether the screen still worked.

It worked perfectly. So the fault was my text, not the game.

From there it was just bisection. Hold back the whole screen, works. Put the labels back, works. Put the nine job messages back but leave them exactly where they were, works.

Put them back and let them move, locks.

The pointers were never the problem. They all existed, and they were all real. The problem is that the code also reads forward from one message to the next:

char *p = pool[MSG_BASE];
for (int i = 0; i < 9; i++) {
    draw(p);
    p += strlen(p) + 1;   // walk to the next one
}

So message 4 has to physically sit directly after message 3. Move any of them somewhere roomier and the walk carries on into whatever bytes happen to be sitting there instead.

RELOCATABLE                      BLOCK-WALKED
  ptr --> "PLANS"                  base --> "msg 1"
  ptr --> "TRAVEL"                          "msg 2"   must follow
  ptr --> "PHONE"                           "msg 3"   must follow

  move it, rewrite the pointer     move one, break every one after it

Having a pointer wasn't enough. Some of these strings are addressed twice, in two different ways, and only one of them is visible to a tool that scans for pointers.

The rule I ended up with: message blocks stay where they are. Fit the English to the space rather than moving the space.

The padding that wasn't padding

The executable appeared to contain spare space. There were gaps between strings filled with zeros which looked like enough space to fit the extra English.

But it turns out the Saturn's processor expects certain data to be aligned correctly in memory.

1,319 of the 1,321 strings begin on 4 byte boundaries. Those gaps are there to maintain alignment.

Removing them causes the game to crash to a black screen.

Three times, before I worked out what I was looking at.

Finding space that isn't there

Looking back at the executable it appeared to have large sections of empty-looking data. Surely some of that must be unused?

Not necessarily.

Some areas are tables that are only ever read, so nothing writes to them and they look dead from the outside. Others are buffers belonging to one screen. One region I was confident about turned out to be the save screen's working memory, which I had simply never opened while testing.

A region can look empty until the exact moment the game needs it.

Proving something is unused is a much harder problem than it sounds.

There was no reliable free space :(

Creating space instead

The executable loads into memory just below the stack, and the stack grows downwards. Between them is a small gap.

After measuring normal gameplay across a couple of sessions, the stack never came closer than about 1.8 KB. I extended the executable into that area, keeping 512 bytes clear as a margin, which gave 1,344 bytes of extra room.

It is not a perfect solution, but the failure mode is better.

If I accidentally overwrite game data, the result could be anything.

If the stack ever reaches the new strings, the worst case is broken text.

Saving bytes with words

The obvious space saving exercise would be modifying the translation itself. Some punctuation was removed, long labels were shortened, and words were changed to fit the available space.

Save to this file?

became:

Save here?

長崎駅 is Nagasaki Station. It became:

Nagasaki

None of these changes mattered individually, but together they made the interface fit.

The screens that aren't menus

A menu is the easy case. Most of the interface isn't one.

This little window caused me more trouble than any menu. The script has location labels in it, I had translated all of them, and the window was still Japanese. There are two completely separate sets of place names in this game: the ones in the script, and a second list of 160 inside the executable. The window uses the second one. So "the labels are 100% translated" was perfectly true and completely useless.

The calendar and travel panels were the opposite, and a genuine relief. They aren't sentences at all, they're assembled from pieces at runtime: place name plus 着 for an arrival, day plus 日, hour plus 時. Translate the small joining pieces and the English just appends itself. A rare case of a 1998 design decision doing me a favour.

Now - I know the settings page isn't perfect, but it does the job.

The limits you cannot negotiate

Some parts of the UI cannot be expanded.

Character names are stored in fixed size records, 26 bytes each, and the name is only the first 7 of them:

82 D9 82 CC 82 A9 00   ほのか + terminator   <- the name slot
02 BC 0A 04 2F         stats
00 00 00 00 00         padding
54 41 45 00            "TAE"                  <- voice stream tag
88 C0 92 42 00         安達                    <- surname

The space immediately after the name is live game data. Making the name field larger would overwrite her stats.

And these particular strings have no pointer anywhere. The game works out their address by arithmetic, so there is nothing to update and nowhere to move them to.

So that leaves 6 usable bytes. Japanese text is 2 bytes per character, which the renderer requires even for English.

Six divided by two.

Hon.

Tae.

Emi.

Three letters wasn't a style decision. It was long division.

The finished interface

After all of that, the game became playable without knowing Japanese.

The menus work.
The locations are readable.
The profiles make sense.
The save screen no longer requires guessing.

This was a different kind of translation.

The dialogue was about language, the UI was about archaeology.

Finding out what assumptions the original developers made, and then trying to work within them.

If you want to try this yourself

The latest version of the GitHub repo is here:

https://github.com/joebuckle-dev/sentimental-graffiti-translation-claude

The part I had not counted on

Now then, the script, executable, fonts and graphics are only part of the disc.

The rest is audio.

I wasn't going to bother with the he voices. But I am quite a ways down this rabbit hole now and I'm wondering if I'd see light by at least attempting to get AI to do these voices for me, with my English text.

Literally hundreds of megabytes of voices. 426 of them, to be exact, which is very nearly three quarters of the disc.

Twelve characters and over twelve hours of spoken Japanese.

That's next...

Translating a 1998 Japanese Visual Novel on Sega Saturn I Bought by Accident

Joe Buckle — Sun, 02 Aug 2026 18:51:25 +0000

This is part 1 of a 3 part series, probably...

Part	Title
1	Translating a 1998 Japanese Visual Novel on Sega Saturn I Bought by Accident
2	Translating a Japanese Sega Saturn Game From 1998: The User Interface
3	...incoming

This is all new to me.

I haven’t done this kind of reverse engineering before, I don’t read Japanese, and I didn’t set out to translate a Sega Saturn game.

I just bought something I didn’t understand and kept going long after it would have been reasonable to stop.

There’s a trader at my local market who’d just got back from a trip to Japan.

He doesn’t do it regularly. This was a one-off as far as I can tell. He had come back with boxes of games: SNES, N64, PS1, all of it in Japanese.

Most of it had already been picked through by the time I got there.

What was left were two Sega Saturn games.

I assumed someone else had already taken anything worth having.

I bought them anyway.

One of them was Sentimental Graffiti (1998), what I later learned is a dating sim.

The game never left Japan, it isn’t especially well regarded (I don't think?), and I don’t have any particular interest in the genre.

Naturally, I tried to play it.

I rummaged through boxes until I found my old Sega Saturn, dug out the cables, and dusted off the Trinitron CRT that had been sitting around waiting for exactly this sort of unnecessary exercise.

There was something oddly satisfying about putting a 1998 Japanese game into the original hardware it was made for.

Obviously I couldn’t read a single thing.

The first thing the game asks you to do is enter your name.

Every label on the screen is in Japanese, including the instructions for using the grid.

It goes without saying, this is as far as I got 😂.

At that point, the sensible thing would have been to stop.

Instead, I realised how much dialogue the game would likely contain and wondered whether it would be possible to extract it and translate it instead.

It's not something I would have considered in the past, but with AI now available to assist us, I thought this would be an interesting experiment.

Step 1: Let Claude loose on the disc

This genuinely surprised me.

There isn’t a toolchain for this game. No documentation, no format notes, nothing describing how the data is organised.

Normally, this is where you would start digging through forums, hunting for old reverse engineering notes, writing scripts, and slowly building up an understanding of the file formats.

Instead, I copied the contents of the disc onto my Linux machine and asked Claude to start looking through it.

I honestly didn’t expect much.

But Claude started finding structure.

It identified blocks of data that looked like text, repeated patterns that appeared to be framing information, and encodings that were likely to be Shift-JIS.

If you know what Shift-JIS is, congratulations. You are already much further down this particular rabbit hole than I was.

I had never heard of it.

It turns out Shift-JIS is a character encoding that was widely used for Japanese text. In simple terms, it is a set of rules that tells a computer how to turn stored bytes into readable characters.

The game doesn’t store text as something obvious like:

Hello, my name is...

It stores numbers that need to be interpreted correctly before they become the Japanese characters you see on screen.

Without understanding that mapping, the dialogue is just a pile of meaningless bytes.

From there Claude began separating large portions of the script from the rest of the disc contents.

It reduced the problem from:

"Understand everything on this disc"

to:

"Find the parts that look like dialogue"

The fact that I could copy a Japanese Sega Saturn disc onto a Linux machine, point an AI at it, and have it start making sense of the contents was, frankly, a little ridiculous.

And that was the moment this particular idea started feeling possible, and the rabbit warren I was about to disappear into began to reveal itself.

Step 2: Find the letters

The game carries its own font - every character it is capable of drawing, stored as raw pixels with no header and nothing to say which picture belongs to which character.

Before you can change any text, you have to work out that mapping.

Claude worked it out, and we tested it in the most direct way available: take the five most common Japanese characters in the entire script, redraw them as big obvious Latin letters, rebuild the disc, and see what the console does.

The screen we tested it on was, appropriately enough, the one that had stopped me on the first day: the name-entry grid.

I turned the disc back on, reached the same screen that had been completely unreadable a few days earlier, and it came up on the CRT looking like this:

A B, a D and a C sitting exactly where three Japanese characters should be.

It is not much to look at - but it meant the font mapping was correct, the disc could be rebuilt, and the Saturn was drawing something I had changed!

A 30(odd)-year-old console, running a game that had never been designed to display English, was now showing my characters on screen.

Step 3: Pull the script out into JSON

This was the point where the disc stopped looking like a mysterious black box.

I knew, obviously, that games were made from code and data. I have been writing software for a long time.

What I had never done was take a finished game from 1998 and start pulling apart the layers inside it.

The script turned out to be stored in plain Shift-JIS, wrapped inside a small repeating structure in the game's bytecode.

If that sentence makes you think "what the heck does that mean?", you are not alone.

The important bit was that we could now extract it.

The dialogue could be pulled out into ordinary JSON files. One file per script.

21,321 records in total.

Here's what the opening conversation looks like after extraction:

{
  "file": "COMMON.SCR",
  "records": [
    {
      "i": 3,
      "type": "dialogue",
      "jp": "$C：ご、ごめん、沢渡…$n　　　　　その、最近あまり会いにこれなくて"
    },
    {
      "i": 4,
      "type": "dialogue",
      "jp": "ほのか　：別に…　あなたに会えなくたって$n　　　　　私、なんとも思わないもん",
      "voice": "VO_HON0"
    },
    {
      "i": 5,
      "type": "dialogue",
      "jp": "ほのか　：嘘…$n　　　　　ホントはちょっと寂しかった",
      "voice": "VO_HON0"
    }
  ]
}

A few things that I learned along the way:

$C is the player's name. The game substitutes it at runtime, so it appears wherever the protagonist speaks.
$n is a line break. The spaces after it are the original developers manually aligning the dialogue underneath the speaker's name.
voice is the sound clip that plays with the line.

The voice information turned out to be particularly interesting.

The filenames on the disc are not always telling the truth.

Every dialogue file actually contains the person after the one named in the filename.

That was discovered because Claude's translator looked at one of the extracted lines and said:

"That isn't Honoka. That's Taeko in Aomori."

The voice files agreed.

The translations are kept separately from the original Japanese:

{
  "file": "COMMON",
  "variant": "en",
  "lines": {
    "3": "$C: S- Sorry, Sawatari... I know I haven't been able to come see you much lately",
    "4": "Honoka: It's fine... It's not like I miss you when you're not around or anything",
    "5": "Honoka: That's a lie... The truth is, I was a little lonely"
  }
}

Keeping these separate turned out to be one of those things that seemed like a technical detail at the time, but became really important.

The Japanese extraction is the source material. It can always be regenerated from the disc.

The English is just another layer on top.

That means a completely different translation, a different writing style, or even a full rewrite of the dialogue could exist alongside it without touching the original extraction.

The English is stored as plain text. The line wrapping is handled later when the game is built.

That turned out to be a very lucky decision for reasons that became obvious in the next step.

Step 4: English is too wide

The game assumes every character fits into a fixed square.

That works fine for Japanese.

It doesn’t for English.

The dialogue box supports 21 characters × 3 lines, and the renderer treats each character as the same width. So instead of flowing text, English gets forced into rigid blocks.

I wasn’t going to play through the entire game just to find overflows, so I built a previewer.

It renders the translated text using the game’s font and layout rules, then flags anything that spills over.

Every red section is text that no longer fits inside the original dialogue box.

The results weren’t great.

1,016 of 3,663 dialogue boxes overflowed - about 27% of the game.

At first glance, it looks like a translation problem.

It isn’t.

The issue is spacing.

The original renderer treated every character as a fixed-width square. That works perfectly when every character is roughly the same size.

Japanese:

すみません

English:

S o r r y

Each English letter is being given a full character cell. So a short word like “sorry” burns through five.

The game doesn’t need shorter text.

It needs variable-width characters.

Step 5: Making the letters narrower

Fixing this meant changing the game's own code.

Somewhere inside the Saturn executable was the routine responsible for drawing a character and moving the cursor ready for the next one.

So Claude disassembled the executable and started looking for it.

The disassembly revealed something important: the font was actually 24×12 pixels, stored in a different format than we first expected.

The original CRT test had appeared to confirm the earlier assumption. We had changed glyph data, rebuilt the disc, and watched the Saturn draw the result.

The chain worked, but the assumption underneath it was wrong.

The test proved we could change the font. It didn’t prove we fully understood the font.

Once the drawing code was understood, the fix was surprisingly small.

A handful of bytes changed how far the cursor moved after each character - moving half a character width instead of a full one - and allowed twice as many characters per line.

That almost solved it.

Loaded into the emulator, the letters were finally the right size and the spacing was correct.

But every line of dialogue still stopped halfway across the box.

The letters are right. The line isn't. The whole right half of that box is doing nothing.

The patch had changed how wide the letters were. It hadn't changed the code deciding where lines break.

That was still wrapping English at 21 characters — because 21 was the correct limit when the box could only hold 21.

It holds twice that now.

This is where storing the English without any line breaks paid off. It looked like a fussy detail at the time, but it meant the wrapping could be fixed during the build process instead of rewriting thousands of translated lines.

Same text, same box, using all of it.

The overflow measurement that previously showed 27% of boxes failing now shows 0.4%.

Fifteen boxes in the entire game still need attention, and those are simple text adjustments.

There was also one more useful discovery.

Could longer dialogue simply continue into another text box?

It turns out the game already knows how to do that.

Consecutive lines naturally chain into another box without needing any extra code.

So if a translation needs more space, the build process can split the line automatically.

Step 6: Translating all of it

With the last technical unknown out of the way, all that was left was volume.

Thirty-seven script files. Every line of dialogue, every choice menu, every location label.

Every batch went through a validator before it was allowed back in. Control codes counted, choice menus checked for the right number of options, nothing over the box budget.

The only Japanese left is 105 records belonging to the developers' own debug menu, which no player will ever see.

I want to be careful with that sentence, though, because for a while I was quietly reading it as "the game is translated".

It isn't. The script is the dialogue. The dialogue is not all of the text.

I'll come back to that, and I promise it's funnier than it sounds.

The problems that showed up at this scale were not the ones I expected.

Nobody warns you that English runs out of ways to giggle.

Twelve people share a handful of Japanese laugh sounds between them. By the seventh character, every natural English spelling had already been taken by somebody else.

"Heehee". "Hehe". "Heheh". "Hehehe". "Ahaha". "Haha". "Teehee".

Four translators working on the same person at the same time each invented a different laugh, and every one of them was a perfectly sensible decision made in isolation.

So there is now a laugh registry in the style guide. An actual table, of who laughs how, that has to be consulted before anyone is allowed to invent a new giggle.

I did not have "maintain a giggle database" on the list of things this project would involve.

Then there was the thing you can only notice by translating every route at once.

The entire premise of the game is a single unsigned letter that says "I want to see you", and the whole journey is about working out which of the twelve people sent it.

Every single route reveals its own person as the sender.

Two of them even share the same scene, word for word in the Japanese.

A player in 1998, finishing one route, would simply have learned that it was her. Reading all twelve at once is the only way to find out that the game's central mystery was never meant to have an answer.

Twenty-eight years, and possibly nobody noticed, because who would sit down and read all twelve?

Well. Quite.

Step 7: Two seconds of the opening movie

Then the translated script had to go back onto a disc, and the disc said no.

The English script is bigger than the Japanese one it replaced, and a 1998 disc has no spare room on it. The files sit end to end, in the order the game expects to find them, with nothing wasted anywhere.

Everything after the script had to shuffle along to make space, and even then it didn't quite fit.

So the opening movie gave up half a megabyte out of 33, which works out at roughly two seconds off the end of the intro.

I did not expect this project to involve cutting two seconds off a video to make room for words.

But it's a decent illustration of what a finished 1998 CD actually is. A fixed amount of physical space, entirely spent, by people who had no idea that somebody would be back twenty-eight years later asking for a bit more.

Sorry about your ending, whoever animated that.

Step 8: The game reviewed our translation

The first complete English build put a message on screen that we had not written.

「インフォ窓のメッセージが溢れました」

The info window's message has overflowed.

That is not our error handling.

That is the original developers' own assertion, left in the retail game, firing for what may well be the first time in its life.

The info window is the small box that displays location names. It was sized for Japanese labels of at most nine characters.

Ours included "Hiroshima Museum of Contemporary Art".

Thirty-six.

Ninety-four location labels got shortened. The longest is now twelve characters, along the lines of "Bot. Garden" and "Mt. Hakodate", so they read like map pins, which is roughly what the Japanese ones were doing anyway.

A debug check written in 1998 caught a defect in a translation written in 2026, and told us exactly what was wrong, in Japanese, the moment we broke it.

We had been carefully validating every line against the dialogue box, because that was the box we knew about.

Nobody had mentioned that the info window was a completely different and much smaller thing.

Nobody except the game, which had been sitting on that error message for twenty-eight years waiting for somebody to be daft enough to trigger it.

The debug message

Step 9: On the actual console

Everything up to this point had been proven on a computer.

The emulator is an enormous convenience. Rebuild the disc, reset, and you're looking at the result in seconds instead of burning for twenty minutes and walking across the room.

But it is still a program agreeing with another program.

The console is the part that doesn't care what you meant.

So: burn the rebuilt disc, carry it over to the Saturn, put it in, and try not to hover.

It works.

The script, in English, running off a disc, on the original 1998 hardware, on the same CRT that had beaten me at the name-entry screen a few weeks earlier.

Photographs of a television, because a screenshot wouldn't have felt like proof.

I have been staring at these for considerably longer than is reasonable.

Everything in this article is in those photographs. The handful of bytes that let letters be narrow. The script that came out as JSON and went back in

And then you press a button, and half the menu is still in Japanese.

Step 10: The text that isn't in the script

This is the bit I had wrong, and seeing it on a television is what corrected me.

The dialogue is translated but the dialogue was never the whole game.

Everything the player does between conversations is interface. Travelling to another city. Taking a part-time job to afford the train fare. Reading a girl's profile. Checking what day it is.

None of that text lives in the script files at all. It sits inside the game's executable as loose Japanese strings, scattered through the code that uses them.

There are 1,361 of them, about 13,659 Japanese characters.

I have translated seventeen.

Those seventeen are the name-entry screen, which I did on the first afternoon, because it was the screen that had stopped me and I was taking it personally.

Obviously I still needed to get past this screen, so hacked this together😂.

Everything else is untouched. The main menu. The job list, which is far more charming than it needed to be: apple orchard, ski resort, dishwashing, dock work, soba shop, takoyaki stand. Every girl's profile card, with her birthday, star sign, blood type and school.

And then there's the travel guide.

I didn't know there was a travel guide. Every location in the game has a written description attached to it, in the register of an actual guidebook. The one for the Saikai Bridge mentions that you can walk across it, but that people afraid of heights might prefer not to.

Nobody in 1998 had to write that. Somebody did anyway, for a dating sim, and then it sat there unread for nearly three decades until somebody who can't read Japanese counted it.

The dialogue, it turns out, was the easy text.

It lives in a container built to hold lines of varying length, so English can be whatever size it likes. That was the whole point of the round-trip test right at the beginning.

These strings have no container. Each one sits in a fixed hole in the executable with the next one immediately behind it, so a translation has to physically fit in the space the Japanese was using.

"Part-time Job" is considerably longer than アルバイト.

So what's left is a slot-by-slot fitting exercise across thirteen thousand characters of menus and travel writing.

Less glamorous than cracking a script format. Just as necessary.

If you want to try this yourself

There's one file on GitHub, and it's the one I'd actually point at:

https://github.com/joebuckle-dev/sentimental-graffiti-translation-claude

That's CLAUDE.md, the guide the whole project runs on. Not the tools, not the extracted script, not the translation, and definitely not the disc.

To be completely clear about that last part: there are no ROMs, no disc images, no BIOS files and no game data of any kind in there, and there never will be. It is a text file about how to think about a problem. Anyone wanting to do this to their own game would need their own disc and their own BIOS, which means owning the thing, which means finding it, which is its own adventure. Mine came from a folding table at a market.

The guide itself isn't really instructions so much as everything we got wrong, written down so it couldn't happen twice.

There's a table in it of mistaken conclusions. The disc that turned out not to be damaged. The USB port that was never the problem. The font format that was confidently recorded as verified and was wrong for a day and a half.

It is essentially a list of things I was sure about and shouldn't have been.

Which is probably the most transferable lesson here, and has very little to do with Sega Saturns.

It isn't finished

I should be straight about where this actually stands, because "I translated a game" is doing a lot of work in that headline.

The dialogue is done and it plays on the console. The menus, the profiles and the travel guide are all still in Japanese.

The translation itself has never been played end to end. Twenty-one thousand lines were checked by a validator, not by a human reading them in context, and I am fairly confident there are lines in there that are wrong, or stiff, or coming out of the wrong girl's mouth.

Two seconds of the opening movie are gone, and they're not coming back.

A stock Saturn still can't boot it at all, because the copy protection is physically stamped into a pressed disc and there's no way to write it. Mine needs a boot cart to get past that. Everything after that point is the game itself, running our data.

For the avoidance of doubt: I own the game. Both discs, bought from that market stall, sitting on the shelf next to the console. Everything here came off my own copy, and nothing is going anywhere else. This is a translation of a game I own, for the game I own, played on the hardware it shipped for.

And there is a whole second disc I haven't touched.

So it is playable in English, and it is not finished in English, both at the same time. I didn't expect that to be a distinction I'd have to make.

I bought this game because it was the only thing left on the table.

I still have no particular interest in dating sims, and I still can't read a word of Japanese.

But there is a Sega Saturn in the corner of the room playing a game in English that has never been in English, and I now understand every single layer of how it got there.

Which was not the plan. There was no plan.

I'll write part two when disc two is done.

Hardening an AI coding agent: the failures, and the code that fixed them

Joe Buckle — Fri, 31 Jul 2026 11:33:04 +0000

At Univoco we build retrieval-augmented assistants over a customer's own documentation. One of them is a coding agent that writes code for a proprietary document layout engine.

The useful thing about that engine is that there is effectively nothing about it on the public internet. The model cannot have learned it. Everything the agent knows comes from a private documentation index and the tools we give it to search that index.

That makes it an unusually clean laboratory. When memorisation is off the table, you get to watch the problem-solving process itself, and watch it fail.

It failed a lot, and the failures were rarely the ones I expected. What follows is the list, in the order we hit them, with the code that fixed each one.

#	Failure	Fix
1	Search tools are literal	filename matching, construct footprints
2	It searches the same thing forever	five layers of loop detection
3	One phrasing is a single point of failure	query inflation
4	Goal words vs. index words	a deterministic phrasebook
5	The retrieval tool invented APIs	self-auditing lookups
6	Deciding an API is invented	the checker stack
7	Findings get ignored	finish gates
8	The wording of a correction is load-bearing	the envelope and the silence trailer
9	A deletion that grew the file	take the generation out

1. Search tools are literal

The standard shape is fetch-then-answer: embed the question, pull the top-k chunks, hand them to the model. It works for a great many tasks, and it has one structural problem. The folder gets assembled before anyone has understood the problem, so you are deciding what might be useful before you know what is needed.

Giving the agent search tools of its own fixes the ordering and immediately produces a different class of bug, because search tools are literal.

Content search cannot see filenames

The agent searched every file in the project for json, found nothing, and offered to create a sample data file. sampleData.json was sitting in the project. The word it needed was on the front of the folder, not inside it.

for entry in project_paths.iter_project_files(ctx.project_dir, glob=glob):
    # A model hunting "the json"/"the xml" searches by TYPE, and content search can
    # never find a file whose content doesn't mention its own format (live failure:
    # sampleData.json was invisible to pattern "json" and the agent offered to create
    # a file that existed). Names are part of the searchable surface.
    if rx.search(entry["path"]):
        name_hits.append(f"{entry['path']} (filename match)")

A grep hit is a line, but the thing you edit is a construct

Asked to remove a feature, the agent searched, found the one matching line, deleted it, and declared success. The five lines above it declared and configured the same object.

So find_in_files now returns grep-style context plus an automatic variable footprint. A hit on layout.blockStart(box) also lists var box = new Block() and every property line of that variable. Only declared variables expand, so globals like layout never blow the output up.

The rule it encodes: the construct is all of its lines, not the one that matched.

Neither fix is clever. Both exist because something shipped.

Two layers of tools

There are two layers of tools, and the split explains several of the fixes that follow.

The base tools ship with the platform and are deliberately generic. list_files, read_file, find_in_files, write_file, edit_file, apply_edit, delete_lines: every customer gets them, and not one of them knows anything about anybody's domain.

The client tools are authored per customer and live in that customer's own directory. Here that means lookup_docs, resolve_topic, find_construct, check_symbols and dry_run, none of which mean anything without this engine's documentation and this engine's object model.

The mechanic that matters is that a client tool shadows a base tool of the same name. Two do. The client find_in_files adds something the generic one could not: when a literal search finds nothing, concept words like "block" or "the box" expand to the engine's real symbols, so the agent never has to translate a concept into code before it can search for it. And rag_lookup, the knowledge lookup itself, is shadowed too, which is what makes the fix in section 5 possible.

That split is the architecture. The generic layer is table stakes and everyone has it. The layer that knows what a "block" is called in this customer's world is the part that cannot be bought.

The two knowledge tools

These two need separating, because the gap between them drives a lot of what follows.

Tool	What it does	What it costs
`lookup_docs`	full-text search over the indexed documentation, returning the chunks that literally contain the term. No embeddings, no model, no generation.	milliseconds, and it cannot invent anything
`rag_lookup`	semantic retrieval, then a model composes an answer out of whatever came back	ten to thirty seconds, and the composing step can invent

The second one is what most people mean by RAG: retrieve some relevant text, hand it to a model, let the model write the answer. It is the right tool for "how do I build a table", and the wrong tool for "does tableColumnRule exist", which is the distinction the agent is told to route on:

TWO knowledge tools, and the choice matters. lookup_docs is ~1000x cheaper. Naming a symbol you already know is a FACT CHECK, not a research question.

That framing looks like a cost optimisation. It turns out to be a correctness one, for reasons section 5 gets to.

2. It searches the same thing forever

My favourite failure of the whole project:

The agent ran the same knowledge lookup twelve times in one turn, identical arguments each time, and the results themselves contained notes pointing out it had already asked. It made no difference.

The variant version was worse, and no exact-match check can catch it, because the arguments differ every time. One table turn ran 12 steps and 170k input tokens, and three of its five knowledge lookups re-confirmed the same three API names. Those names were already in its context, listed in the prompt fragment it had been handed at the start of the turn.

It was not searching. It was checking something it already knew, at ten to thirty seconds a time. The operator's note in the ledger puts it better than I can:

once it knows how to do stuff it shouldn't need to keep trying to find it at the rag end

A human has an internal sense of going in circles. The model does not, and telling it so in the tool output does nothing, because that text is just more context. Worse, on a provider without prompt caching the fixed ~11k of system prompt and tool schemas is re-sent every step, so cost is roughly fixed times steps plus transcript. Fewer steps is the only real lever, and a redundant lookup adds steps.

The fix is five layers, and the layering is the point: each layer only catches what the one before it structurally cannot see.

Layer 1: identical calls

A counter in the tool loop, tripping at _MAX_IDENTICAL_CALLS = 4. Past that the call is not executed, but it still has to be answered, because every tool call in the transcript needs a result.

# Stuck-loop detector: the same call over and over is fixation, not progress.
# Answer it with a stub (every tool_call must be answered) and head for the
# forced final answer instead of burning the rest of the budget.
key = (name, call.arguments or "")
call_counts[key] = call_counts.get(key, 0) + 1
if call_counts[key] > _MAX_IDENTICAL_CALLS:
    stuck = True
    messages.append({"role": "tool", "tool_call_id": call.id,
                     "content": f"[not run - you have made this exact call "
                                f"{call_counts[key]} times this turn]"})

Layer 2: rewordings within a turn

Reworded questions are distinct argument strings, so the identical-call counter never fires on them. Catching those needs a budget rather than a counter:

Constant	Value	Meaning
`_MAX_LOOKUPS_PER_TURN`	`12`	live lookups across all wordings
`_MAX_LIVE_LOOKUPS`	`2`	per identical query, then the turn cache answers

Layer 3: rewordings across turns and sessions

This one I like, because the obvious solution is wrong. You could embed every query and compare vectors. We do not, because that is a per-probe cost on the hot path for a problem that plain string handling solves.

Matching is deliberately embedding-free: normalise (lowercase, drop stopwords, light stemming with a double-consonant collapse so splitting to split), token-sort, then exact key hit or a difflib close match. Deterministic, stdlib, zero per-probe cost, and it catches exactly the reworded-not-rethought queries above. Genuinely different questions stay distinct.

The similarity threshold is 0.8, and the calibration comment is the reason I trust it:

the live pair whose only difference was the domain word scores 0.833
two genuinely different questions, create table vs create box, score about 0.73

The window between them is the sweet spot, and it was measured rather than guessed.

Layer 4: concurrent duplicates

Read-only calls get dispatched together, so three rewordings can all be in flight before any of them returns, and a dedupe that runs on completion never sees them. That needs a leader/follower collapse on the normalised query: the first runs, the rest wait and receive its answer.

Layer 5: do not ask for what you were already given

None of the above touches the failure this section opened with. A budget caps how much re-asking costs, but the agent re-confirming names it was handed at turn start should not be asking at all.

Writes to domain code are normally refused unless a knowledge lookup has run in that turn, which is what forces grounding. That gate was also what forced the redundant lookups: the agent had the names, but the rule did not know that.

So the prompt fragments now declare what they supply. Each construct fragment carries a provides list, generated rather than hand-written and scoped to one family, so the table fragment declares the table companions and not the rule ones. The task brief unions those into provided_symbols, and the gate exempts a write whose new X() constructors are all covered. A write reaching for a construct that was never delivered still grounds, and check_symbols remains the backstop before anything lands.

The general shape: a rule that forces good behaviour will also force pointless behaviour unless it can see what has already been satisfied.

The counter-lesson

This was the most useful thing in the section. The budget once cut a model off mid-way through correctly verifying eleven symbols it was about to use. It was doing exactly the right thing and we throttled it.

So budget is now charged per tool rather than per call:

_MAX_LOOKUPS_PER_TURN caps LIVE lookups across all wordings; cached serves and cheap tools are free (projects.budget_tools, default rag_lookup): a millisecond lookup_docs fact-check per symbol is diligence we demand, not cost to ration.

If your rate limit punishes the behaviour you are trying to encourage, the limit is wrong, not the behaviour.

One last piece of absurdity, included because it is a genuine class of bug and not just a funny story. An earlier nudge injected into the tool results said "look up the interface's REAL members". The model ran that sentence, verbatim, as a semantic search query. Twelve times.

Nudges now name an action, and only ever suggest exact-name lookups, never a phrase, because anything phrase-shaped in an agent's context is a candidate query.

3. Query inflation

One user phrasing is a single point of failure for recall. If the question happens to be worded unlike the corpus, retrieval returns nothing useful and the agent concludes the knowledge base is empty rather than that it asked badly.

So before the lookups run, a forced tool call reformulates the question into several deliberately different queries. Not paraphrases: different angles.

_QUERIES_TOOL = {
    "name": "search_queries",
    "description": (
        "Reformulate the question into optimised knowledge-base search queries using the right "
        "domain keywords / API terms. For a how-to / implementation / 'how do I ...' question, "
        "return 2-3 queries covering different angles (e.g. the core API/object, and a usage "
        "example). Return a single query ONLY for a trivial factual lookup (e.g. 'what is X')."
    ),
    ...
}

The user message that drives it is where the actual behaviour lives. Almost every clause is there to stop a specific degenerate output:

Generate {max_queries} search queries covering DIFFERENT angles so retrieval is thorough:
1. A focused how-to query for the specific task.
2. A BROAD keyword query - just the core nouns / concepts / terms (e.g.
   'user login session token'), to surface related material the specific query might miss.
3. If more are allowed, add a query aimed at a concrete example.
Do NOT return only a verbatim copy of the question. Only a trivial factual lookup gets a
single query.

Examples (apply the SAME pattern to the question's own domain):
- 'set up user authentication' -> ['how to set up user authentication',
  'authentication login session token configuration']
- 'export a report to CSV' -> ['how to export a report to CSV',
  'CSV export columns rows formatting']

The system line is one sentence and does most of the work:

You generate DISTINCT knowledge-base search queries using precise keywords / API terms. Never echo the question verbatim.

Without it you get the question back with the word order shuffled, which is the reworded-not-rethought problem from section 2, except now we are paying to generate it.

Three details that are not obvious

Each query carries a human label. The tool schema requires an about field alongside the query: "a short, natural human phrase for what this looks up, e.g. 'creating tables with borders'". That is not for retrieval, it is for the activity line the user watches. Generating the explanation in the same call as the query is free; deriving it afterwards is another model call or a bad heuristic.

The queries are not fanned out by code. They are injected back into the prompt as an instruction, so the agent still owns sequencing and can stop early.

def _guided_prompt(question, query_objs, adaptive=False):
    """Wrap a question with guidance to search each expanded query, then answer (or dig deeper)."""
    qlist = "; ".join(o["query"] for o in query_objs)
    tail = _ADAPTIVE_TAIL if adaptive else " Then answer fully."
    return f"{question}\n\n[Search the knowledge base for: {qlist}.{tail}]"

Expansion without a brake is just a licence to over-search, which is section 2 all over again. So _ADAPTIVE_TAIL ships with it:

Then judge the results: if they already answer the question, STOP and answer now, do not run extra, redundant, or tangential searches. Only if a genuine gap remains (or the results reference something specific you still need) search again for exactly that. Aim for the fewest searches that fully cover the question.

There is a ceiling of 6 regardless of configuration, and max_queries: -1 means the model decides within that ceiling.

One clarification, since the two get conflated: this is multi-query rewriting, not HyDE. No hypothetical answer document is generated. In a domain the model has never seen, a hypothetical document is a hallucination with extra steps, and embedding it just retrieves whatever is nearest to the invention.

4. The agent's words are not the index's words

This is the failure that cost the most and looked the least like a bug.

The agent phrases things by goal: "display the data in a table", "put the XML in a table". The corpus is indexed by the engine's own vocabulary: openStream, xmlDoc, selectNodes.

To a human expert those are obviously the same topic. To a retriever they have almost nothing in common, so the lookups returned nothing, the agent concluded the capability did not exist, and it reached for browser JavaScript instead.

Missing real content and inventing names is the expensive combination, because the invention then looks like a hallucination and you go and debug the wrong component.

Three layers, in increasing order of force.

A deterministic phrasebook

resolve_topic maps a goal to the real symbols, with no model involved at all:

The agent phrases requests by goal; the docs are indexed by object-model vocabulary. That mismatch made it MISS real content and INVENT names, spiralling. This reads the terminology datagraph (_topics.json) and hands back the real symbols, a note on the pattern, and the exact next lookup, so the agent searches truth instead of guessing.

Zero LLM, deterministic. Call it BEFORE rag_lookup when you don't already know the API names.

The map is mined rather than hand-written:

human phrasings come from query-pattern tags on the worked examples
symbols come from a regex over the example code, validated against the authoritative API dictionary
a curated overlay adds goal aliases for the high-value topics

Matching is stopword-filtered token overlap ranked by hit count. No embeddings anywhere in it.

A confidence ladder

Because a phrasebook that is confidently wrong is worse than none. The first version applied full force to any match at all:

# Confidence ladder (live miss: 'footer with page numbers' shared ONE word with the
# layers topic and got the full-strength 'do NOT invent' route, walling the agent off
# from the real constructs). A weak match is a HINT, not a law: strict only when the
# route explains at least half the goal's significant words, or 2+ of them.
coverage = top / max(1, len(_words(goal)))
if top < 2 and coverage < 0.5:
    return ("WEAK MATCH - the goal only loosely fits a mapped topic; treat this as a "
            "hint, NOT the answer:\n\n" ...)

A guardrail with a false positive does not fail neutrally. It actively steers the agent away from the answer, and it does it with authority.

A de-nouning rewrite

Users phrase goals in their own nouns ("display parts from an XML file grouped by category") and documentation indexes mechanisms ("load an XML file, iterate elements into table rows, heading per group"). One cheap call normalises the retrieval query before the lookup stack sees it, layered outermost so that dedupe and the persistent cache both key on the normalised form and hit more often.

The constraint on that rewriter is the interesting part:

#: Operations only, never API names: the rewriter half-knows jargon, and an invented
#: symbol in the query would poison retrieval with exactly the hallucination the
#: lookup exists to prevent. API discovery stays retrieval's job.

It ships disabled, incidentally. The doctrine version in the prompt does most of the work and the mechanical version is held in reserve for when doctrine leaks.

Where the phrasebook is applied matters more than the phrasebook. The terminology hint is prepended to every lookup result, whether or not the model asked for it, and whether or not retrieval returned anything, because a gap is exactly when the real names matter most.

5. The tool was the liar

The agent kept writing invented API calls into its work. It would be caught, it would look the correct ones up, and then it would do it again. Seven rounds of this in one session.

I assumed the model was inventing them. That is the famous failure mode and I never questioned it.

Then I read what the lookup tool had actually returned, rather than the model's account of what it returned.

The retrieval pipeline was generating the invented APIs. The style that answers conceptual questions composes an answer from retrieved examples, and that generation step had invented layout.tableCellEnd and Table.cellSpacing, neither of which exists, while asserting they came "directly from the example in the provided context".

That is not really a bug, it is what generation does. Ask an obliging model for a worked example and it will produce a worked example, and where the retrieved material runs thin it fills the gap with the most plausible name available. The word "lookup" was what fooled me: I had been treating an endpoint that ends in a generate step as though it returned facts, when the only thing separating it from the agent was that its inventions arrived with a citation attached.

The model was faithfully copying bad information from the one source it had every reason to trust. Worse, the loop was closed: the check flagged the file, the agent re-ran the same lookup to find the right answer, and got the same poison back.

The debugging rule that came out of it, because I lost days to not knowing it: read the actual tool results in the log, not the model's narration of them. A large proportion of what looked like model failures turned out to be a tool feeding it poison.

The fix is a client-side shadow that wraps the built-in lookup and audits its output with the same symbol checker the write path uses:

The examples style GENERATES its answers, and generation can invent APIs. A live run returned examples using APIs that do not exist, the agent wrote them, check_symbols flagged the file, the agent re-ran the same lookup and got the same poison: a perfect loop. This shadow breaks it at the source: run the built-in lookup, then run the SAME symbol audit the write path uses over the returned example, and hand the agent the answer and the corrections in one result, so it learns the example is partly wrong BEFORE writing it, together with the interface's real members to build from instead.

The second-order bug, which is the one worth stealing

The lookup cache had pickled the poison. Of 137 cached entries, 126 carried warning text. Every fuzzy cache hit was re-serving a known-bad answer, forever.

Flagged answers are now uncacheable, and the cache module has a rule list at the top that reads like scar tissue, because it is:

never cache failures or empties, because those are the answers most legitimately retried, and a wrong fuzzy hit on a cached "returned nothing" is poison
serve each entry at most once per turn, because a fuzzy match can be wrong and the corpus can change

One more thing, and I think it generalises

Once the guardrails were in, we probed the same grounded prompt across three model tiers, all with identical retrieval and identical checks:

Model tier	Grounded probe
frontier	5/5
mid	2/5
cheap	0/5

The verdict in the notes: the guardrails make a weak model SAFE (bounced), not GROUNDED.

A gate stack stops bad output reaching the user. It does not turn a model that cannot follow grounding instructions into one that can. Budget for the model tier; do not expect the scaffolding to buy it back.

6. Deciding that an API is invented

This is the check everything else depends on, and the first version was wrong in an instructive way. It asked an LLM to list the symbols in the agent's code, then full-text searched each one.

Every step leaned on the thing that was broken: an LLM summarising another LLM's hallucination, and a search whose only possible verdict was "not found", which tells an agent nothing about what to write instead, so it kept the invented call.

That second clause is the part I underestimated. "Not found" is not actionable. An agent that receives it has no better move available than the one it already made, so it makes it again.

The reframe: the documentation is complete and enumerable, so the question is decidable. Resolve what an object is, then ask whether the member exists on it.

layout.tableCellEnd                       ->  layout is Layout    ->  no such member  ->  INVENTED
var cell = new TableCell(); cell.start()  ->  cell is TableCell   ->  check `start`

Where the index comes from

"Complete and enumerable" is carrying a lot of weight in that paragraph, so here is where the index actually comes from. It is not hand-written, and that is the only reason the verdict is trustworthy. A hand-maintained list of what exists is just one more thing that can drift from what exists.

The member index falls out of the documentation's own structure. Every documented member is its own chunk, titled Interface-member, so parsing titles yields {interface: {members}} for the entire surface. Two patterns in the prose supply the rest of what the resolver needs:

Pattern in the docs	What it yields
`Interface-member` chunk titles	every member of every interface
`accessed using the global property "layout"`	`layout` resolves to `Layout`
`Parent interface: Block`	inherited members

The middle row is what makes receiver typing possible at all. The set of documented globals, the thing that turns layout.x into a decidable question rather than a guess, is extracted from a sentence the documentation happens to write about itself.

A second datagraph describes constructs rather than members: how each one is declared, and which calls open and close it. It is generated from the API definitions rather than curated, and the reason is the best argument in the codebase for never hand-writing ground truth:

Hand-curating it is error-prone: the first cut invented a List/listStart construct that does not exist.

We hallucinated an API while building the tool whose entire job is catching hallucinated APIs. The generator now derives the pairs mechanically: on a global-reachable interface, a <base>Start/<base>End pair whose Start takes a defined interface as its first argument. Pairs whose first argument is a primitive, like keepStart(string), are not constructs anyone instantiates, so they are skipped and reported rather than guessed at. Seven constructs survive that test.

An entry looks like this:

{
  "id": "table",
  "interface": "Table",
  "kind": "paired",
  "aliases": ["table", "grid"],
  "declare": "new Table",
  "start": "layout.tableStart",
  "end": "layout.tableEnd",
  "no_close": "Rows and cells end IMPLICITLY. There is NO tableCellEnd and NO tableRowEnd; those methods DO NOT EXIST."
}

That last field is the interesting one, because it is a negative fact. Some constructs close implicitly, so the end-calls a developer would reasonably expect are absent, and tableCellEnd is precisely what the model kept reaching for. The fact is authored once and read by both the checker and the terminology builder, so the two can never drift into disagreeing about it.

It also upgrades the finding. Instead of "no such member", which sends the agent hunting for a replacement that does not exist, the checker can say the call is unnecessary and the row closes on its own, so deleting it loses nothing. A checker that only knows what is absent can report; one that knows why it is absent can tell the agent what to do instead.

With the index in place, three further properties are what make the check survivable in production.

Precision over coverage

A member is only judged when the receiver's type is certain: a documented global, a new X(), or an interface named directly. A parameter, a function's return value, a nested property, all skipped.

a wrong accusation is worse than a miss: it sends the agent off to "fix" working code, and it teaches the user to ignore the check.

Eight check families, each of which exists because something landed

A member check alone is not enough, and every gap here was found in production rather than in design.

What landed	Why it slipped the previous checks	Check added
`engine.loadXML(...)`	undeclared receiver, so no type to resolve	object check
`loadXMLData()`	a bare call defined nowhere	function check
`const`, arrow functions	`node --check` passes what the target engine cannot run	syntax check (ES3)
`layout.blockEnd({ borderColor: 'red' })`	an invented key inside an object literal, not a dotted member	option check
`new ActiveXObject("Microsoft.XMLDOM")`	member check cannot type an unknown constructor	constructor check
`Object.keys`	valid JS, absent from an ES3-era engine	engine check
a referenced data file that does not exist	not a symbol at all	file check

The option check is the one I would not have thought of. An invented member on a dotted call is obvious. An invented key smuggled through an options object is invisible to a member check, and at runtime it is silently ignored, so the behaviour you promised the user simply never happens and nothing anywhere reports an error.

A checker's false positive is worse than a miss

We learned this twice, expensively.

Once by accident. A markdown fence in a knowledge-base snippet tripped a template-literal check and a real API was branded invented. The model then wrote that into its session memory: it learned that a real API does not exist, and carried that forward.

Once systematically. The target engine implements a genuine W3C DOM, and our browser-API denylist had been written on the assumption that DOM methods were browser-only. Of the 34 names on it, 23 were real APIs in this engine. The check had been flagging the agent's correct code, and it had helped cause the spiral in section 5.

Every new check now ships with a known-good-input case for both shapes it audits, whole files and fenced snippets.

The runtime layer

Static analysis has a hard limit: it must skip any receiver whose type is uncertain, which is precisely where dynamic code lives. So the last check generates a fake engine from the documentation index and actually runs the build against it.

The docs are COMPLETE and enumerable. So a fake engine can be GENERATED from them: every documented global and constructor exists; every documented member is a callable, chainable stub; touching an UNDOCUMENTED member on a typed object records an INVENTED finding. loadJS/importJS resolve real project files, so wiring is exercised for real.

It catches syntax errors and crashes, loads of files that do not exist, and invented members reached through dynamic flow: parameters and return values, exactly the receivers the static pass must skip. It runs once at the end of a turn, never per write, because one process launch per draft is cheap and one per write is not.

The limits are stated in the same docstring, which is the part I would want if I were adopting this:

What it can NOT prove: rendering. Data APIs return permissive stubs (no real XML), so loops may run zero iterations and zero-iteration paths go unexercised. This is a dry run, "the code executes against the documented surface", not a render test.

7. A finding that does not block is decoration

I built the checker, wired it in, and watched it work. It caught the invented APIs and reported them accurately, and nothing whatsoever happened as a result:

Told "these APIs do not exist, fix them now", the model wrote the user a prose essay ("you may need to rely on...") and finished the turn with the invented symbols still on disk. So findings also gate the END of the turn: a finish_gate the tool loop consults before accepting a final answer, which pushes the model back to work while anything is outstanding (same lesson as lookup_before_write: enforce it, don't ask for it).

The gate contract fits in one line:

gate(answer, asking=False) -> Optional[str]

Returning a string means the draft was not sent and that string goes back into the model's context. Returning None ships it.

Everything else is about the budget, and the budget is where the design decisions are.

Per-gate bounds compound

Gate	Own push budget
verify	2
bail	2
claim	1
checklist	1

Each is individually reasonable, and together they can bounce one turn six times, which is an unpredictable worst case for latency and cost. So there is a shared turn budget of 3 across every gate, and past it the composed gate goes silent and the turn ends with whatever honesty the pushes bought.

Which produced the next bug

Twice, every push was consumed by intermediate holds and a false final claim shipped unchallenged. Hence a reserve:

reserve_final keeps the last (user-visible) draft policed after the shared budget is spent mid-turn. Past max_pushes, up to reserve_final further pushes may be spent, but ONLY on NEW material: a draft whose text materially changed since the last hold, or new deeds. A verbatim doubled-down draft has already ignored this correction once, repeating it buys nothing, so it ends the turn as before.

Refuse before the write, not after

This was the single biggest behavioural improvement:

Auditing after the fact meant asking the model that just invented an API to un-invent it. It would rather explain, annotate, or stand by it. Refused content simply never becomes a file.

Police claims, not just code

The gate that catches the most is the simplest. Asked to split a file, the model created three modules and reported four:

Changes made:
  1. Created xmlLoader.js ...
  2. Created styledBox.js ...
  3. Created tableCreator.js ...
  4. Updated template.js to import and use the new files.

Step 4 never happened. It had planned it and narrated the plan in the past tense. The check is mechanical: past-tense verbs only, scoped to a single line, matched against ctx.files_changed, which every landed write funnels through.

The tense restriction is not incidental, and the comment explaining why earns its length:

# PAST TENSE ONLY, and deliberately so.
#   - "I will update template.js" is a plan, not a deed - the future tense must not trip this.
#   - The present tense describes what CODE does, not what the agent did: "template.js already
#     imports xmlLoader.js" and "createTable() creates a new Table" are statements of fact
#     about the project, and reading them as claims would gate honest, read-only answers.
# Only the past tense asserts "I did this", which is the thing the file system can contradict.

The scope restriction matters just as much: matching is per line, because a wider window lets one "Created" three bullets up capture every filename mentioned afterwards.

And the gate declares its own non-goal, which I think is why it works:

This checks only what the model VOLUNTEERS. It does not police silence... A false claim is wrong unconditionally; that is why this gate can be mechanical.

Two smaller rules that took real failures to learn:

Strike caps exist because the model can be wrong about how to fix something and must not be trapped re-trying forever.
Findings are split into introduced and inherited, after a user complained: "I only asked it to do 1 simple thing and it did it, but it also started talking about jsonData." Only a finding that a write in this turn introduced may hold the turn.

8. How the system talks to itself

Every gate above works by writing a message into the model's own context. Getting the wording of those messages right turned out to be as load-bearing as the logic that triggers them, and it is the part I have seen discussed least.

Start with the two constants that open and close every one of them.

The envelope

#: Opens every gate message. Injected corrections must not read as the user speaking, or the
#: model treats its own gated draft as user feedback.
ENVELOPE = "[automated pre-send check - NOT a message from the user] "

That is not a stylistic choice. Without the marker the model reads its own blocked draft as something the user saw and objected to, and then reasons about it as if a conversation had happened.

The silence trailer

#: Closes every gate message. Without it the model apologises to the user for a draft they never
#: saw ("You are correct. I apologise..."), explains itself, and asks permission, turning a silent
#: retry into a wall of confusing chat. The gate is plumbing and must stay invisible.
SILENCE = ("Do not apologise. Do not mention this check, your draft, or any mistake - the user "
           "never saw them and will not understand a reference to them. Do not ask permission: "
           "just do the work. Your reply must read as if you had done it right the first time. "
           # Live: a bail push made the model REWRITE a file smaller, deleting footer code
           # that had already passed the symbol check, and surrender in a comment instead.
           "Never respond to this check by deleting or stripping code that already passed "
           "its checks - fix FORWARD (add, complete, correct), never retreat.")

Every sentence in that string is a bug, and the last one is the strangest: a push meant to stop the agent giving up made it delete working, already-verified code and write an apology in a comment.

Correction pressure has a direction, and if you do not specify it, the model will sometimes satisfy you by removing the problem rather than solving it.

Both constants live in one module because they did not, once, and a hand-copied version quietly drifted out of sync.

Three of the messages, chosen for what each one teaches

Escalation on repetition - the second push stops being polite

SECOND refusal: you have made ZERO writes this turn, the file system is the ground truth
and it shows no change, yet your draft still reports work as done. It is false.

Do ONE of these, nothing else: (1) make the change now with a write tool and then
describe it, or (2) answer exactly what is true: that you changed nothing, and what you
found in the file instead.

The first time the model claims work it did not do, it gets a straightforward correction. This is what it gets if it doubles down.

Constraining what the model then tells the user - the budget message

[lookup budget exhausted - {live} knowledge lookups have run this turn] No further
lookups will run. Searching again, in any wording, returns this same message. STOP
searching and WRITE... In that final answer, if information is missing, say the SEARCH
BUDGET ran out, do NOT claim the knowledge base lacks it; the budget ending says nothing
about what the documentation contains.

The obvious message is "stop searching". The last clause took a live failure to learn: without it, our own rate limit came out of the agent's mouth as a factual claim about the customer's documentation. The agent had no way to know the difference, so we had to tell it.

A checker that argues - pre-empting the three moves the model actually makes

This is not "unverified", the documentation is COMPLETE, and these were checked against
every member of the interface. You invented them just now, most likely because a similar
name exists (tableStart exists, so tableCellEnd feels like it must). It does not. The
file will fail on the real engine.

Rewrite those calls using the real members listed above. Do NOT re-run the same
rag_lookup that produced this code; it will return the same wrong example. If you need a
pattern, rag_lookup a DIFFERENT, more specific goal. Do NOT annotate the call and move
on; do NOT keep it because it looks right. If the interface genuinely has no member for
what you intended, leave that part out and say so in your final answer, never leave a
call to a method that does not exist.

Note that it names the failed retrieval explicitly. Without that line the agent re-runs the lookup that produced the bad code in the first place, which is the closed loop from section 5.

The trust boundary between agents

When a plan is decomposed, each item runs as a bounded worker with its own context, and the worker's report comes back to the composer as a user message. Workers are told to end with what they built and what it exposes.

The composer does not believe the "exposes" line:

def exposures(path: str, content: str, answer: str) -> str:
    """One line describing what a landed file exposes - MECHANICAL over the file itself
    (the file is a fact; a model-authored "exposes" line is a claim). Falls back to the
    worker's own short answer for non-JS / no matches."""

That parenthesis is the whole article in one sentence. Everywhere the system has a choice between what a model says happened and what the file system says happened, it takes the file system, and where it cannot, it says so and degrades deliberately.

9. Some mistakes should not be catchable

The fix for this last one runs against every instinct: it is to take intelligence away, not add more.

Asked to remove a table, the agent removed one line, rewrote everything around it, and made the file bigger:

Attempt	Result
original	1527 bytes
"remove the table"	1653 bytes, reported as success
retry	1659 bytes, reported as success

The diagnosis had two layers, and both are worth knowing:

The removal instruction fell into a generative chunk-rewrite path, so "remove" became "rewrite this region, minus the thing".
The forced pre-write lookup for "how to remove a table" returned the documentation for creating one, because that is what the corpus has. That poisoned the intent: the agent came away believing you remove a table by calling tableEnd().

The fix was to take the generation out of that step entirely. The model reads the file and emits L<n>: DELETE markers, nothing else, and ordinary code drops those lines:

Delete-only pass: the model reads the file (or a where=-scoped region) and marks which lines belong to the thing being removed; they are dropped deterministically. It can neither rewrite nor add, so a removal can only shrink the file. This is where EVERY pure deletion goes instead of the generative chunk rewrite (which deleted one line and grew the file live).

The same file went from 1527 bytes to 678, with the right construct gone and the one next to it untouched.

What it all amounts to

Mechanism	Catches
filename matching, construct footprints	literal search missing the obvious
identical-call breaker, turn budget, fuzzy cache, in-flight collapse	fixation and rewording loops
query inflation with an adaptive brake	one phrasing being a single point of failure
`resolve_topic`, confidence ladder, de-nouning rewrite	goal words that miss an API-indexed corpus
self-auditing lookup shadow, uncacheable flagged answers	the retrieval tool inventing APIs
`check_symbols` (8 families) + the `dry_run` shim	invented APIs, statically and at runtime
composed finish gates, pre-write refusal, claim audit	findings being ignored, work claimed but not done
envelope, silence trailer, escalating pushes	the correction itself being misread or over-obeyed
delete-scan	a destructive edit that could grow the file

Across all of this, the model never got better. Every version ran on the same tier that failed the first time.

What changed was the space of things it could get away with: what it could search without a limit, what it could claim without evidence, what it could write without being refused, and what it could remove by generating rather than by marking.

That is unglamorous work, and it is almost all of the work.

Creating a web API with Lua using Nginx OpenResty

Joe Buckle — Sun, 02 Feb 2020 16:58:55 +0000

I'm a bit of an old fashioned, pragmatic developer and tend to stick with frameworks and patterns that I know are reliable - a 'stick in the mud' if you will.

This dive is deviating from my usual path.

I had never considered Lua to be a viable language to create a backend for a web application. Of course I'm not totally oblivious to the language itself - I do know it's used mostly for embedding in other software and game engines - but that's it really.

I came across the OpenResty project whilst on a learning journey of Nginx web server.

Writing code directly on the web server rather than passing the request to some interpreter is a very interesting concept for me.
Lua runs directly inside the Nginx worker which means a very small barrier between the webserver and the application code.
Additionally, Lua is known for being immensely fast compared to interpreted languages particularly when using the LuaJIT compiler.

Nginx is capable of handling a very high number of concurrent connections at a very low memory footprint.

However, there doesn't appear to be that much excitement about it.
It's been around for some time (at least since 2016) but it does appear to be lacking in developer adoption. Saying that, some big names are using it, including Cloudflare and Tumbler.

Despite that - the performance gain from this sort of backend has peaked my interest.

The easiest way (perhaps the only way) to get started with this is by installing OpenResty.
This provides the Nginx web server with the Lua module.

If you're familiar with Nginx this example will make sense to you. The location directive is being passed a Lua script as a file.
The output from that script is expected to be in JSON in this configuration:

location ~ ^/api(.*)$ {
  default_type 'text/json';
  add_header 'Content-Type' 'application/json';
  content_by_lua_file /etc/openresty/sites/api.lua; # < Points to Lua file
}

You can also pass arbitrary code into content_by_lua and content_by_lua_block.

Now, Lua is a reasonable simple language to pick up if you're used to Ruby or Python. I hadn't used it before but I was still able to quickly write a script that allows creation of API endpoints and parses the body and parameters.

There is an object you can access in Lua called ngx. This provides you with the data passed into Nginx from the request such as the body or the path etc...

Example:

ngx.var.request_method -- POST, GET.. whatever
ngx.req.get_body_data() -- The data passed in from the request
ngx.var.uri -- The request path

The script I wrote allows me to set the Method and Endpoints that are allowed.

--[[ api.lua --]]

-- Helper functions
function strSplit(delim,str)
    local t = {}

    for substr in string.gmatch(str, "[^".. delim.. "]*") do
        if substr ~= nil and string.len(substr) > 0 then
            table.insert(t,substr)
        end
    end

    return t
end

-- Read body being passed
-- Required for ngx.req.get_body_data()
ngx.req.read_body();
-- Parser for sending JSON back to the client
local cjson = require("cjson")
-- Strip the api/ bit from the request path
local reqPath = ngx.var.uri:gsub("api/", "");
-- Get the request method (POST, GET etc..)
local reqMethod = ngx.var.request_method
-- Parse the body data as JSON
local body = ngx.req.get_body_data() ==
        -- This is like a ternary statement for Lua
        -- It is saying if doesn't exist at least
        -- define as empty object
        nil and {} or cjson.decode(ngx.req.get_body_data());

Api = {}
Api.__index = Api
-- Declare API not yet responded
Api.responded = false;
-- Function for checking input from client
function Api.endpoint(method, path, callback)
    -- If API not already responded
    if Api.responded == false then
        -- KeyData = params passed in path
        local keyData = {}
        -- If this endpoint has params
        if string.find(path, "<(.-)>")
        then
            -- Split origin and passed path sections
            local splitPath = strSplit("/", path)
            local splitReqPath = strSplit("/", reqPath)
            -- Iterate over splitPath
            for i, k in pairs(splitPath) do
                -- If chunk contains <something>
                if string.find(k, "<(.-)>")
                then
                    -- Add to keyData
                    keyData[string.match(k, "%<(%a+)%>")] = splitReqPath[i]
                    -- Replace matches with default for validation
                    reqPath = string.gsub(reqPath, splitReqPath[i], k)
                end
            end
        end

        -- return false if path doesn't match anything
        if reqPath ~= path
        then
            return false;
        end
        -- return error if method not allowed
        if reqMethod ~= method
        then
            return ngx.say(
                cjson.encode({
                    error=500,
                    message="Method " .. reqMethod .. " not allowed"
                })
            )
        end

        -- Make sure we don't run this again
        Api.responded = true;

        -- return body if all OK
        body.keyData = keyData
        return callback(body);
    end

    return false;
end

Then I can call Api.endpoint() following to create my endpoints:

--[[ api.lua --]]
Api.endpoint('POST', '/test',
    function(body)
        return ngx.say(
            cjson.encode(
                {
                    method=method,
                    path=path,
                    body=body
                }
            )
        );
    end
)

Api.endpoint('GET', '/test/<id>/<name>',
    function(body)
        return ngx.say(
            cjson.encode(
                {
                    method=method,
                    path=path,
                    body=body,
                }
            )
        );
    end
)

So this has got me started. Obviously the logic inside these endpoints would be much more complex and usually have to connect to databases - all of which OpenResty supports.

One thing I did come across was the Lapis Framework created by Leafo who I know mostly from creating the PHP compilers for Less and SCSS.

Github for code the code here - https://github.com/bambattajb/openresty-api-example,

Embedded UI Components with Preact

Joe Buckle — Tue, 24 Dec 2019 10:34:14 +0000

As a front-end developer it's quite common to be tasked with creating a new UI component for an application.

You're given access to some backend API and you have to crack on and build a UI that sits somewhere inside the website as a widget, page or whatever.

The component is complex enough that you need to manage view states, so you'd quite like a system for that.
You also like using the cleaner ES6 syntax and you're already familiar with React and JSX and all that jazz.

Now, you don't have access to the applications overarching build tools - you just need to find a way of passing some reasonably sized and efficiently written code over to the application maintainers who will factor it into their monolith 😊.

I tend to see these UI components as their own small compartmentalised applications but I am fully conscious of the increased size when shipping seemingly unnecessary frameworks in packaged code.

So React is out of the question because it's > 30kb.

I eventually discovered Preact which claimed to come in at 3kb and supports all of the Virtual DOM and State Management features of React.

I totally signed up and was not disappointed. Just set up a Webpack build environment and away you go.

How is Preact so small?

React has its own Synthetic Event System for handling events and Preact uses the browsers native addEventListener.
This could lead to a user experience penalty.

React is trying very hard (30kb + hard) to ensure a consistent feel in whatever browser it's being run in; whereas Preact has chosen to use unpredictable browser-native event listeners for a massively reduced packaged size.

More on the 'subtle' differences here - Main Differences.

What's missing?

Well, not much!

Proptypes (they aren't used all the time anyway)
Children

Conclusion

Preact is a great solution to my problem but it does look as though it might come with some UI performance trade-offs leading to an inconsistent UX.

That said, if you chose to build and embedded UI component without a framework you most likely wouldn't create your own synthetic event system for seamlessly handling differences in native events.

Preact is totally winning for me in this context 👍

I love WordPress, but I don't like its ecosystem

Joe Buckle — Sat, 21 Dec 2019 15:24:21 +0000

Come gather round friends and I'll tell you a tale
Of when I claimed to be a WordPress guru
But was then made to use off-the-shelf themes and plugins
And that claim I sought to undo

I've been using WordPress to solve content management problems for clients in the digital publishing industry for around 10 years and know its core features pretty well having spent countless hours traversing through the codex and reading a plethora of problem-solving developer articles.

I am an advocate for the software and push it as a low-cost, easy to use, extensible and familiar content editing & publishing platform for digital publishing teams.

From a developer perspective the core provides a very good and efficient Query Engine, Database Abstraction Layer, Rest API Framework, User / Roles management and loads more.

It also provides a decent admin interface that is intuitive and adaptable enough that I can tailor it to specific business requirements.

What's not to like?

Well, last year I assumed the role of "WordPress Guru".

Up until then I had been using WordPress as a toolkit for building a unique set of features for my clients. I would be writing a lot of specific business logic around this toolkit just as you would a framework.

At the time of accepting this role I didn't know which sites I'd be working on and just assumed that, armed with this knowledge, I should be able to handle it right?

But I was wrong!

You see, there is, in my opinion, a "Dark Side" to WordPress - its Theme and Plugin culture.

It's not like I was unfamiliar with this and whenever I'd read up or hear of this culture I thought it was individuals or small businesses on small budgets looking to get a website up as cheap as possible and that it shouldn't apply to serious businesses or organisations.

I was wrong again!

I used the term "Dark Side" because I would be dealing with content managers that have a business requirement for something that is more flexible than the feature-restrictive theme or plugin. At some point this system was sold to them either by someone in-house or a developer that is now long gone.

The first problem is you have a client that doesn't understand "why" this feature is difficult to implement (they're right - it shouldn't be, if this system was built properly).
The second is the developer trying to shoehorn features into this system knowing it's not really fit for purpose thus making their ongoing maintenance work more difficult.

Sometimes you have to do things like "modify core code" of plugins only to learn of a "major security update" some weeks later that you now have to factor in.

Sites end up becoming a maintenance and security headache.

Lesson learned

I will never claim to be a "WordPress Guru" again as putting me in an environment where I have to build or extend sites that use WooThemes or Divi (for example) then I am lost!

Someone who is a "WordPress Guru" would understand the entire ecosystem of available WordPress products - not just its core system.

My advice for publishers is if you are looking at purchasing a Theme or Plugin to handle a feature you're needing make sure it does EVERYTHING you're ever going to need.

Otherwise consult a developer - there are some good quality plugins that allow themselves to be extended by using custom hooks and filters, perhaps enough so to fill the void in requirements.

DEV Community: Joe Buckle

Translating a Japanese Sega Saturn Game From 1998: The User Interface

The text that isn't in the script

The first UI mystery

When English exposes old assumptions

The plan was simple

The padding that wasn't padding

Finding space that isn't there

Creating space instead

Saving bytes with words

The screens that aren't menus

The limits you cannot negotiate

The finished interface

If you want to try this yourself

The part I had not counted on

Translating a 1998 Japanese Visual Novel on Sega Saturn I Bought by Accident

Step 1: Let Claude loose on the disc

Step 2: Find the letters

Step 3: Pull the script out into JSON

Step 4: English is too wide

Step 5: Making the letters narrower

Step 6: Translating all of it

Step 7: Two seconds of the opening movie

Step 8: The game reviewed our translation

Step 9: On the actual console

Step 10: The text that isn't in the script

If you want to try this yourself

It isn't finished

Hardening an AI coding agent: the failures, and the code that fixed them

Contents

1. Search tools are literal

Content search cannot see filenames

A grep hit is a line, but the thing you edit is a construct

Two layers of tools

The two knowledge tools

2. It searches the same thing forever

Layer 1: identical calls

Layer 2: rewordings within a turn

Layer 3: rewordings across turns and sessions

Layer 4: concurrent duplicates

Layer 5: do not ask for what you were already given

The counter-lesson

3. Query inflation

Three details that are not obvious

4. The agent's words are not the index's words

A deterministic phrasebook

A confidence ladder

A de-nouning rewrite

5. The tool was the liar

The second-order bug, which is the one worth stealing

One more thing, and I think it generalises

6. Deciding that an API is invented

Where the index comes from

Precision over coverage

Eight check families, each of which exists because something landed

A checker's false positive is worse than a miss

The runtime layer

7. A finding that does not block is decoration

Per-gate bounds compound

Which produced the next bug

Refuse before the write, not after

Police claims, not just code

8. How the system talks to itself

The envelope

The silence trailer

Three of the messages, chosen for what each one teaches

The trust boundary between agents

9. Some mistakes should not be catchable

What it all amounts to

Creating a web API with Lua using Nginx OpenResty

Embedded UI Components with Preact

How is Preact so small?

What's missing?

Conclusion

I love WordPress, but I don't like its ecosystem

What's not to like?

But I was wrong!

I was wrong again!

Lesson learned