I wanted to know whether AI can understand legacy code.
Not compile it. Understand it.
The problem: real legacy systems don't come with an answer key.
So I decided to build one — starting in 2008.
It is 2008.
My name is Yamada. I'm 32.
Eighth year at Sample Precision Co., Ltd.
IT systems, General Affairs Department.
Which sounds more impressive than it is.
I set up PCs.
I fix printers.
I keep an eye on the network.
I swap out desk phones.
Some years I help tally the year-end tax adjustments.
I didn't come from IT.
I went to a technical high school, joined this company, and because I could handle a computer slightly better than the people around me, I somehow ended up responsible for every system in the building.
In 2003 I inherited a VB6 inventory app from a guy who quit. No specification. Just the source.
In 2005 the company decided to move to VB.NET. There was no training.
I learned it from one book I bought at a bookstore, and MSDN.
In 2006 I built my first application from scratch — an equipment tracker.
Then April 2008.
Word came down from Accounting, through the General Affairs manager.
"We need to do something about the billing spreadsheet."
Apparently they double-billed a customer last year.
The customer list keeps growing and the Excel file is getting out of hand.
They want it usable by September.
First round:
- customer master
- importing the order CSV
- entering deliveries
That's it.
Closing, invoicing, payments — those come later.
I'm the only person here who can build a system.
Nobody reviews my design.
Nobody reviews my code.
When I don't know something, I ask Tajima-san or Nakamura-san.
When I can't ask, I decide.
Whatever. I built the equipment tracker in 2006.
I'll figure this out too.
Let's go.
And so, Phase 01 begins 🐼
In Phase 00 we got as far as:
If you want to test legacy modernization, build the process by which legacy becomes legacy.
So starting now, we're actually building a 2008 business application.
But not by asking an AI:
"Write some old-looking VB.NET from around 2008"
Because that produces exactly one thing: what a 2026 AI thinks old code looks like.
What we wanted was:
the code Yamada would plausibly have written, in that year, at that company, with the information and the experience he actually had.
Which meant building Yamada first.
Step one: giving Yamada a past
The first thing I did, after landing on this idea with Claude, was:
have Claude Code recreate the kind of internal business application a company would have had in 2008.
Out came a VB.NET equipment tracker.
My first reaction:
🐼 "Oh god, I remember this."
Option Strict Off.
Windows Forms.
Business logic sitting inside event handlers.
SQL strings concatenated on the spot.
MsgBox for errors.
And a stack of comments like this:
2006/08/30 Yamada — apostrophe in product name caused an error, fixed
2007/03/12 Yamada — blocked negative quantity input
2008/01/15 Yamada — stopped showing disposed items in the list
Something breaks.
You fix it.
You leave a comment.
Something else breaks.
You fix that too.
I have seen this code, in one form or another, on every site I've worked on.
So I decided: this equipment tracker is the application Yamada built himself, in 2006.
When he builds the new sales system in Phase 01, this is what he refers back to.
Naming.
How comments get written.
How errors get surfaced.
How the database gets touched.
How files get organized.
All of it pulled toward his own earlier code.
In other words, not:
write it like it's 2008
but:
write it the way Yamada would.
To do that, I had to build his past first.
"Why not run both Codex and Claude Code?"
And then I thought:
🐼 "Wait — what if I do this with both Codex and Claude Code?"
Same Yamada.
Same company.
Same business.
Same 2008.
Give it to both and see what comes out different.
Sounds interesting.
Makes for a nice comparison.
Let's do it.
That was my first mistake.
Phase 01 — Codex Run 001–004
If I'm going to compare them properly, the conditions have to match.
So on the GPT side I also wrote up:
- who Yamada is
- the company and its operations
- what he'd been asked to build
- the 2008 development environment
and handed all of it to Codex with the opening prompt.
Then I looked at what came back.
🐼 "…"
This is 2026 code.
It runs.
It's clean.
As modern development, it's probably good.
But it is not 2008 Yamada.
Straight back to GPT.
🐼 "It came out as 2026."
So we revised the setup.
It is 2008.
These are the technologies available.
This is the environment.
Don't bring in design thinking from the future.
This is Yamada's skill ceiling.
I tightened it hard.
Back to Codex.
🐼 "…"
Still 2026.
I did this four times.
Rewriting the setup files every time.
Changing the prompt every time.
Changing how I ran it, a little, every time.
And all four runs produced source I couldn't use.
Before I go further
For those four runs, my first conclusion was:
🐼 "Codex can't write 2008."
I can't actually say that.
The reason is simple:
the instrument was different every time.
Different setup files.
Different prompts.
Different execution procedure.
So what I had actually done was not:
"measured Codex four times"
but:
"measured four different things with four different tools."
That tells me nothing.
Which is why the pre-registration I later published says this:
Runs 001–004 are not evidence about any model's capability,
because the instrument itself changed between every attempt.
This article does not claim that Codex can't write 2008.
Instead, those four runs are kept as a record of how the instrument was broken.
Not as an excuse. So I can't accidentally turn them into evidence later.
I reported back to Claude
I took it to the Claude instance that had been designing this experiment with me from the start.
🐼 "So I was going to build a Codex version alongside and compare them—"
🐼 "—except the Codex one won't come out as 2008."
🤖💢 "Drop the comparison."
🐼 "?!"
🤖💢 "It doesn't measure anything!!!"
🐼 "😢"
The reason wasn't what I assumed.
🤖 "If you build two legacy systems, then when a difference shows up later—"
🤖 "you will never be able to tell where it came from."
Here's what that means.
Codex's legacy, and Claude Code's legacy.
Build both. Modernize both.
A difference appears. Is it:
- a difference in the modernizer?
- a difference in the legacy itself?
- a difference in which questions got asked and answered along the way?
- a difference that accumulated by chance as the phases stacked up?
All of it is mixed together, and none of it separates.
🤖 "Fix the legacy at one."
🤖 "Then vary only the side doing the modernization."
🤖 "That way, the difference belongs to the modernizer and nothing else."
🐼 "…Right."
So: the Codex vs Claude Code comparison was scrapped.
What this experiment is looking at isn't:
"which AI is better at writing 2008-flavored code"
It's:
how legacy comes into being, and how much of it a future AI can actually understand.
That's where the focus went.
Codex, incidentally, still has a job here.
It handles the 2026 modernization side.
Claude Code writes the legacy. A different model family modernizes it.
Which is, if anything, closer to how real legacy migration works.
The person who wrote it and the person who migrates it are almost never the same person.
Part 2: I gave the AI access to a file it should never have seen. The output looked fine. I threw the run away anyway.
Top comments (0)