Modernizing a legacy RPG application in five safe steps, with a regression run after each one

"Modernize the RPG" is easy to say and risky to do. The programs that need it most are the ones that have run the business for twenty years, with rules nobody has written down anywhere except in the calculation specs.

The way I approach it is boring on purpose: many small steps, and after every step a regression run that must give exactly the same result as the legacy code. If the output differs, the step is not finished.

To show the method end to end, I built a small "legacy" accounts receivable application the way many shops still run it (fixed-format RPG, numbered indicators, the RPG cycle, O-specs, DDS files) and modernized it in five steps on PUB400, IBM i 7.5. Everything is on GitHub: dodikibmi/ibmi-modernization-showcase. All names and data are fictional.

The application and the safety net

Object	What it does
ARR200 (RPGLE)	posts each new payment to the customer's oldest open invoices; rejects amounts ≤ 0 and unknown customers; leftover money becomes unapplied credit; releases a customer from hold when the balance reaches zero
ARR300 (RPGLE)	aged trial balance driven by the RPG cycle: primary file, L1 control break, O-specs
ARC100 (CL)	the daily run: post payments, then print the report
ARCUSTP, ARINVP, ARINVL1, ARPAYP	DDS physical and logical files

The safety net is a small harness: load the same test data (customers, invoices and seven payments including the nasty ones: zero amount, unknown customer, overpayment), run the daily job, then save every row of the three files plus the spooled report to a text file. The legacy run is the baseline; each later stage must produce an identical file.

That one rule shaped every decision below.

Step 1: fixed format to fully free RPG, behaviour unchanged

Before:

     C     PYCUST        CHAIN     ARCUSTR                            90
     C     *IN90         IFEQ      *ON
     C                   MOVE      'E'           PYSTS
     C                   MOVEL     MSGCUS        PYMSG
     C                   UPDATE    ARPAYR
     C                   GOTO      ENDPST
     C                   ENDIF

After:

chain PYCUST ARCUSTR;
if not %found(ARCUSTP);
  rejectPayment(MSG_NO_CUSTOMER);
  return;
endif;

Six numbered indicators became %found, %eof and named variables, two GOTO/TAG became an early return from a procedure, and one subroutine became six procedures with parameters and local variables.

The payment program converted almost line by line. The report did not: fully free RPG has no I- or O-specs, so a cycle program has to be restructured into a read loop with a customer break and an externally described printer file. To prove that the new printer file prints exactly what the old O-specs printed, I generated both layouts from one table and compared the spooled output line by line.

The defect I kept on purpose. The legacy report prints an invoice of a customer that is missing from the master file as 100500 *UNKNOWN* (FICTIONAL). A failed CHAIN leaves the previous customer's name in the buffer, and MOVEL '*UNKNOWN*' only overwrites the first nine characters. Step 1 is a pure refactor, so I kept the bug (and marked it in the code). If you change structure and behaviour in the same step, a difference in the regression run no longer tells you which of the two caused it.

Step 2: DDS to SQL tables, without recompiling a single program

The new tables have long names, long column names (CUSTOMER_ID FOR COLUMN CUCUST), primary keys, CHECK constraints, and audit columns maintained by Db2 itself (ROW CHANGE TIMESTAMP, GENERATED ALWAYS AS (USER)), declared IMPLICITLY HIDDEN so the old programs never see them.

The hard requirement: the legacy programs and the step 1 programs must keep running without recompiling. A program compiled against a file checks the record format level identifier when it opens it; if it changed, you get CPF4131 until every program is recompiled. So each old file name comes back as an object over the new table, with the old record format, and the level identifier must be the same as before.

My first attempt used SQL indexes for all four files (CREATE INDEX ... RCDFMT ... ADD <columns>). Two of them got new level identifiers. The reason: an SQL index always places its key columns first in its record format, and in the invoice file the key is not the first field. For those two I used DDS logical files over the SQL table, which keep the original field order. The result (saved in test/out/level-ids.txt): four old level identifiers, four identical new ones, and both the legacy and the free programs ran against the new tables with output identical to the baseline.

Step 3: business rules in a service program, with unit tests, and the bug fixed

The rules that were buried in calculation specs moved into a NOMAIN service program with binder source and an explicit signature: validate a payment, how much of a payment goes to one invoice, when a customer leaves hold, days past due, the age bucket, the name to print.

None of these procedures does file I/O, so they can be tested without any test data. An iRPGUnit suite covers every rule, every bucket boundary (0/1, 30/31, 60/61, 90/91), a leap day, and the old *UNKNOWN* defect:

Success. 20 test cases, 28 assertions, 0 failure, 0 error.

Now the bug is fixed, in its own step. The regression run differs from the legacy output in exactly one line, the one that was wrong. Every payment, invoice and balance is still identical. That single-line diff is the best evidence I know that a fix did what it should and nothing else.

Step 4: native record I/O to embedded SQL, one transaction per payment

The legacy program had no commitment control. If it failed after updating an invoice but before updating the customer, the data was out of balance and someone fixed it by hand.

The SQL version works on the new tables by their long names and still takes every rule from the service program:

• a cursor over waiting payments, declared WITH HOLD so it survives each COMMIT;
• the customer row read WITH RS USE AND KEEP EXCLUSIVE LOCKS, then updated;
• open invoices from a cursor ORDER BY DUE_DATE FOR UPDATE, updated WHERE CURRENT OF;
• COMMIT after each payment; any SQL error means ROLLBACK and an escape message.

A regression run proves the result is the same. But a transaction is only proven when it rolls back, so I tested that too: another job held a lock on the second invoice of a customer. The run paid the first invoice in full, waited for the second, timed out and ended with an escape message. Afterwards the first invoice was still unpaid, the balance unchanged and the payment still waiting: the half-posted payment was rolled back. Once the lock was gone, the same command posted everything and the output matched the normal run.

Step 5: a REST API that reuses the RPG code

The last step adds a REST API in Python on PASE, using the standard library HTTP server and itoolkit. The point is that no business rule is written twice:

• the API validates a payment by calling the service program procedure directly from Python, so the API and the batch can never disagree;
• an accepted payment is inserted and then posted by the batch program itself, with its one-transaction-per-payment logic;
• age buckets come from the same procedure the report uses, all calls of one request batched into a single round trip.

The test sends the seven test payments one at a time through POST /api/payments. The status codes come back as expected (201 201 201 404 422 201 422), and the customers and invoices are identical to the step 4 batch run.

Before and after

	Legacy	Modernized
Numbered indicators	6 + L1, LR, 1P, OF	0
GOTO / TAG	2	0
Business rules	inside the calc specs of two programs	6 procedures, used by the batch and the API
Unit tests	none	20 test cases, 28 assertions
Database	DDS, no constraints	SQL tables, keys, CHECK constraints, audit columns, journaled
Programs recompiled for the DDS → SQL move	—	none
Failure halfway through a payment	data out of balance	rolled back
Known defect	prints the previous customer's name	fixed in its own step, covered by a test

What I would tell a team starting this

1. Build the regression harness first. Same test data, same run, every row and the spooled report saved as text. It costs a day and pays for itself in the first step.
2. Never change structure and behaviour in the same step. Keep the known bugs during the refactor, then fix each one in its own step with a test. A one-line diff is a proof; a fifty-line diff is a discussion.
3. Watch the level identifiers when you move DDS to SQL. An SQL index puts its keys first; where the keys are not the leading fields, a DDS logical file over the SQL table keeps the old format and saves you a recompile of every program.
4. Prove the rollback, not only the commit. Hold a lock from another job and watch the transaction undo itself.
5. New channels call the existing rules. The API is valuable because it reuses the RPG procedures and the batch program, not because it is written in Python.

The code, the test data and the saved output of every regression run are in the repository: github.com/dodikibmi/ibmi-modernization-showcase. I would be glad to hear how other shops structure this kind of work, especially the DDS to SQL move on files with many dependent programs.