An AI-generated replacement can compile, start and produce plausible output while changing a business rule. For a migration, the useful question is whether you can identify and explain each observable difference from the old implementation.
OpenAI’s October 19 research on low-resource programming languages distinguishes building new functionality from preserving existing behaviour. MultiLangBench asks models to build a SQLite-compatible engine across 20 languages. A separate evaluation migrates COBOL programs to other languages, with hidden tests requiring byte-for-byte parity on observable outputs. A migration passes only if all its tests pass.
OpenAI reports 86.6% accuracy for GPT-6.1 Sol at low reasoning effort on that migration evaluation and 97.9% at higher effort. Those are benchmark results, not a forecast for your repository. The announcement explicitly retains expert review and testing against production business rules and dependencies.
Here is a small migration workflow I would use to make those distinctions operational. It is a proposed implementation approach, not a reproduction of OpenAI’s evaluation or a tested production framework.
Start with an observable boundary
Choose a routine whose inputs, outputs and side effects you can isolate. A record-processing batch is a better first unit than an entire application with loosely understood integrations.
Write down:
- Input format, encoding and missing-value conventions.
- Output format, precision and ordering rules.
- Error codes and rejected-record behaviour.
- State changes, messages and other externally visible effects.
- Environment assumptions such as timezone, locale and dependency versions.
The point is to define what a comparison will cover. A matching output file says nothing about an unobserved message sent twice.
Separate parity from intentional changes
Create a fixture directory with input and expected-output files from an approved, isolated run of the old implementation. Use synthetic or appropriately protected data. Pin the old executable and its environment so that the reference result is reproducible.
Alongside the fixtures, keep a small decision ledger:
# Illustrative structure; identifiers and values are hypothetical.
case_id: positive_fraction_boundary
rule: retain_two_decimal_places
old_result: "12.34"
decision: preserve
approved_result: "12.34"
owner_role: financial_process_owner
reason: preserve_existing_contract
OpenAI’s article supplies the mechanism behind this example: COBOL fixed-point arithmetic discards excess decimal digits unless rounding is requested. For a hypothetical positive value of 12.349, discarding excess digits gives 12.34; rounding to the nearest hundredth gives 12.35.
A replacement that chooses the second result can run perfectly well. Whether that difference is a regression depends on the requirement. If the owner approves a rule change, record it separately and retain the old expected result as historical evidence.
Avoid quietly regenerating all expected files from the new implementation. That can turn the code being evaluated into the authority for its own correctness.
Make the first comparator deliberately strict
For a byte-oriented interface, a basic file comparison is useful. The following standard-library script compares two existing files and reports a mismatch. It does not execute either program, validate business correctness or prove migration completeness.
from pathlib import Path
import sys
def main():
if len(sys.argv) != 3:
raise SystemExit("Usage: compare.py expected.bin actual.bin")
expected = Path(sys.argv[1]).read_bytes()
actual = Path(sys.argv[2]).read_bytes()
if expected != actual:
first = next(
(i for i, (left, right) in enumerate(zip(expected, actual))
if left != right),
min(len(expected), len(actual)),
)
print(
f"Mismatch at byte {first}; "
f"expected length={len(expected)}, actual length={len(actual)}"
)
return 1
print("Outputs match byte for byte")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Start strict so differences become visible. If a format permits harmless differences, such as a field order that consumers explicitly do not depend on, define a narrow semantic comparison for that field. Keep the original output files. Broad whitespace stripping or sorting can hide changes that matter to a downstream consumer.
For the initial fixture set, include normal records and examples around the actual workflow’s boundaries: empty input, missing references, repeated records, date transitions and decimal precision. A passing set establishes evidence for those cases; it does not establish universal equivalence.
Inspect failures as part of the interface
Run the candidate in a sandbox with live writes disabled. Observe exit status, output files, state changes and attempted side effects. Use captured or simulated dependency responses to investigate retries and partial failures where appropriate.
Classify each discrepancy against the decision ledger: a candidate defect, an explicitly approved change, an unstable test environment or an unresolved requirement. Keep unresolved cases visible. An unexplained difference should not disappear behind a passing aggregate percentage.
Finally, measure generation effort separately from investigation, review and integration work. A lower API bill is one cost component. It does not tell you how much work remains to understand undocumented behaviour.
The first deliverable I would ask for is one isolated routine, a repeatable fixture set and a reviewed list of differences. That gives a team concrete evidence for deciding whether to expand the migration.
Top comments (1)
새 구현으로 기대값을 통째로 다시 만들지 말자는 경계가 특히 중요하네요. 출력 파일이 같아도 외부 알림이 두 번 나갈 수 있다는 예시처럼, 부수효과를 횟수와 순서로 함께 비교하면 관찰 범위가 더 분명해지겠습니다. 의도한 변경은 기존 기대값을 덮어쓰지 않고 승인 근거와 분리하는 구성도 좋습니다.