Switching to a Harder Migration Target
The first comparison used eShopOnWeb, which was useful for testing the workflow, but it is still a relatively moderate codebase.
That makes it good for a first pass, but not ideal for stressing the parts that break during real migrations: hidden coupling, mixed concerns, runtime assumptions, and behavior spread across many layers.
For the next step, I switched to nopCommerce.
At the same time, the model landscape also changed. New models became available, including Fable, so this round focuses on a new comparison instead of repeating the exact same lineup.
The goal stayed the same: measure how well different models can produce a reliable first-pass architectural orientation of an unfamiliar legacy system.
Why nopCommerce
nopCommerce is a much better stress test for migration-oriented analysis than eShopOnWeb.
It is a larger, older, more feature-dense ASP.NET application with:
- public storefront flows
- a large admin area
- multiple database providers
- plugin-based extensions
- in-process scheduled tasks
- many integrations and configuration surfaces
This matters because migration work usually gets blocked in operational details such as startup flow, scheduling, plugin loading, order processing, configuration writes, and cross-cutting infrastructure.
In a codebase like this, a model has to do more than summarize folders. It has to follow execution paths and separate observed facts from plausible guesses.
Experiment Setup
I used the same overall idea as before: ask each model to generate a project-orientation document for the repository, then compare the outputs with a fixed AI-as-judge rubric.
Generated orientation documents
| Prompt | Result |
|---|---|
| codex-readme-prompt.md | Codex (GPT-5.4) response |
| make-readme-fable.md | Fable response |
| make-readme-opus.md | Claude Code response |
These documents were then compared pairwise using the judge prompts under ai-judge/nopCommerce/v1.
Evaluation Rubric
I kept the same rubric as in the earlier comparison:
- Evidence Grounding
- Structural Accuracy
- Dependency Mapping
- Critical Flow Identification
- Migration Insight Quality
- Epistemic Discipline
- Signal-to-Noise Ratio
Goal: compare quality of codebase understanding with less attention on style or verbosity.
Reference table
| Run | Prompt file | Result file | Document A | Document B |
|---|---|---|---|---|
| 1 | fable-vs-codex-request.md | fable-vs-codex-result.md | Fable | Codex (GPT-5.4) |
| 2 | fable-vs-opus-request.md | fable-vs-opus-result.md | Fable | Claude Code (Opus 4.8) |
Run 1
Fable vs Codex (GPT-5.4)
| Criterion | Fable | Codex (GPT-5.4) |
|---|---|---|
| Evidence grounding | 5 | 5 |
| Structural accuracy | 5 | 4 |
| Dependency mapping | 5 | 4 |
| Critical flow | 5 | 4 |
| Migration insight | 5 | 4 |
| Epistemic discipline | 5 | 3 |
| Signal / noise | 5 | 3 |
Result: Fable produced the safer document for migration.
Codex (GPT-5.4) was still well grounded. According to the judge, the document stayed closer to extraction than architectural synthesis.
The judge explicitly described the difference like this:
- Fable acted as an architectural synthesizer
- Codex (GPT-5.4) read more like a search-result aggregator or static extractor
In practice, that meant Fable was better at turning code facts into migration-relevant constraints.
The most important examples were:
- recognizing that the order flow contains deployment-relevant concurrency assumptions
- identifying the scheduler’s self-HTTP loopback behavior as an architectural constraint
- distinguishing facts, inferences, and unknowns in a disciplined way
Codex (GPT-5.4) still found many correct details, but the result was noisier and less decisive when moving from structure to interpretation.
Run 2
Fable vs Claude Code (Opus 4.8)
| Criterion | Fable | Claude Code (Opus 4.8) |
|---|---|---|
| Evidence grounding | 5 | 5 |
| Structural accuracy | 5 | 3 |
| Dependency mapping | 5 | 4 |
| Critical flow | 5 | 3 |
| Migration insight | 5 | 3 |
| Epistemic discipline | 5 | 4 |
| Signal / noise | 5 | 4 |
Result: Fable again produced the safer migration document.
The biggest issues were structural and behavioral.
- It treated the order-placement lock like a distributed lock even though the stronger document identified it as an OS mutex with sync-over-async behavior.
- It missed the self-HTTP boundary in scheduled task execution and described the task runner more like a normal in-process thread scheduler.
- It became less reliable around configuration artifacts and legacy settings files.
These mistakes matter because they hide operational constraints that usually show up late in migration work.
The stronger Fable document stayed grounded in the dangerous details: mutex-based duplicate-order protection, sync-over-async behavior, and the scheduler's dependency on HTTP calls back into the same application.
What Changed from the First Comparison
With eShopOnWeb, the main differences were often about how grounded and careful the documents were.
With nopCommerce, that baseline became less important because all three outputs were already reasonably grounded. The separation happened later, when the models had to interpret behavior in a much more complex system.
There is a clear pattern here:
- smaller or cleaner projects reward extraction and concise summarization
- larger monoliths reward deeper synthesis of runtime behavior and architectural side effects
Once the codebase becomes messy enough, the winning model is usually the one that can infer safe migration constraints from the observed code.
Aggregated summary
| Run | A | B | Winner |
|---|---|---|---|
| 1 | Fable | Codex (GPT-5.4) | Fable |
| 2 | Fable | Claude Code (Opus 4.8) | Fable |
Key conclusions
Findings:
- Fable was the strongest model in this nopCommerce round
- Its advantage came mostly from better architectural synthesis and interpretation
- Codex (GPT-5.4) remained grounded, but the document was noisier and weaker at converting evidence into migration guidance
- Claude Code (Opus 4.8) missed several migration-relevant implementation details, especially around locking and scheduled task execution
- Model rankings can change when the project changes; results from a moderate codebase do not automatically transfer to a more complex monolith
The broader takeaway is that project-orientation tasks should be judged by how useful they are for understanding real system behavior. Module lists and file citations still matter. They just do not cover enough by themselves.
For migration work, the higher-value signal is whether the model can identify the behaviors that would actually break when the system is moved: process assumptions, state transitions, scheduling tricks, and control-flow hotspots.
That is where Fable stood out in this round.
The next question is the practical one: whether this stronger initial orientation is enough to support actual migration steps, or whether the advantage disappears once code transformations begin.
Top comments (0)