What happens when you replace twenty years of ad-hoc scripts with one framework
Every IBM i shop I've ever worked with has the same thing tucked away somewhere: a folder of CL programs that do FTP.
Not one. Not two. Dozens.
Some of them were written when the current network admin was in high school. Some have the password sitting in clear text, one line below an FTP command. Some of them ran fine for fifteen years and nobody touched them because touching them is scarier than leaving them alone. A few of them died quietly overnight last Tuesday and nobody noticed until the morning report was missing.
That's what this project started from. Not a grand modernization strategy. Not a compliance directive. Just the growing feeling that what we had was a very polite kind of disaster waiting to happen.
The pile
If you've lived this, you already know the shape of it. But for anyone who hasn't, here's what the "pile" actually looks like in a typical IBM i shop:
- Clear-text passwords in CL source members. Anyone with read authority on the source file can see them.
- No audit trail. Who created the script? Who last changed the destination path? Nobody knows. You display the source and hope the comments are honest.
- No record of what ran when. If Tuesday's file didn't arrive, you're opening spool files and piecing together what happened from raw FTP output.
- No retries. If the remote server bounced you once, the job failed and you got an email at 3 a.m. — assuming the email worked.
- No recovery. If the FTP crashed mid-transfer, the production file was already half-overwritten. Good luck.
- No way to disable a broken interface without editing production source. So nobody disables anything.
Everyone knew. Everyone had a story about "that one time." Nobody had the time to fix it all.
Deciding to actually do it
I started with what I thought was a small fix. One of our FTP pulls from a mainframe kept failing on Sundays. I opened the CL, fixed the timeout, went to save, and saw the password in the source. Not hashed. Not referenced. Not pulled from a data area. Just sitting there in plain CL, in a source file that half the shop could read because that's how the whole library was set up fifteen years ago.
I fixed it. Then I opened the next CL. Same thing.
I looked at the list. Dozens of these things. Interfaces to a mainframe, to UNIX boxes, to a cloud SFTP gateway, to a vendor's SFTP. All with their own idea of what "error handling" meant. All with their own password, right there in the source.
You know that moment where you stop trying to fix the broken thing and start trying to replace the category of thing? That was that moment.
Why FTP, and not an API
The obvious question, if you don't live on IBM i: why keep FTP alive at all? Why not just replace it with REST calls, or a message queue, or literally anything from this decade?
Because I don't control the other end.
The counterparties on these transfers are a mainframe that's been running since before I started, a handful of UNIX boxes owned by other teams, a cloud SFTP gateway, and vendor systems whose integration options begin and end with "here's your SFTP login." None of them are going to stand up a REST API for me. Some of them can barely be asked to rotate a password. FTP/SFTP isn't a design choice on my side of the fence — it's the only door that exists on the other side.
So the goal was never "get off FTP." It was "stop treating FTP as a place where anything goes." Where I do control both ends — internal services, systems I own — I use APIs, and that's a different, easier problem. This project was specifically about the transfers where FTP is the only option and the only thing standing between "reliable" and "whatever that CL happened to do in 2011" is whether someone bothered to build a framework around it.
That distinction mattered more than anything else in scoping this. I wasn't competing with modern integration patterns. I was making the legacy pattern I was stuck with behave like one.
What these transfers actually move
It's worth being concrete about what's riding on this, because "FTP framework" sounds abstract until you see what's actually flowing through it.
The ERP itself runs on the IBM i. Around it sits a mainframe that's still the system of record for a slice of transactional data predating the ERP, a warehouse management system on its own platform, EDI exchanges with vendors and trading partners, and a handful of internal reporting feeds. None of these systems talk to each other natively — the FTP layer is the connective tissue. Nightly extracts pulled in for the ERP to consume. Outbound files pushed out for the warehouse system to pick up. EDI documents exchanged with partners. Inbound price and inventory files landing from vendors.
None of it is glamorous. All of it is load-bearing. If the inbound inventory feed doesn't land by 4 a.m., the warehouse is working from yesterday's numbers. If the outbound order file doesn't reach the vendor, nothing ships. This is the plumbing that makes otherwise-unrelated systems agree with each other.
That's what made "just harden the FTP layer" worth three weeks of evenings. It wasn't a nice-to-have. It was the seam where every other system touched every other system.
The approach
I wasn't trying to build a platform. I wasn't trying to win an architecture award. I wanted one thing: take all of this and consolidate it into a single framework that an operator can configure without touching source code, that logs everything, that encrypts credentials, and that doesn't silently half-overwrite production files when something goes wrong.
Three layers. That's it.
Maintenance — a subfile UI. Operators define transfers by filling in fields: host, port, remote path, local file, pre/post commands, retry counts, retention. Passwords go in once, encrypted.
Execution — one program that reads the config, acquires a lock, runs the transfer, validates what came back, logs everything, and updates status.
Housekeeping — scheduled retention cleanup. Archives age out automatically according to rules the operator set when they defined the interface.
A small set of objects overall. A handful of physical files for the data model. Two service programs (one for encryption, one for logging). Two main programs, a retention job, and a CL wrapper for operator inquiries, plus display and printer files and a key data area.
About three weeks of evenings, once the design settled. The hard part wasn't the code. The hard part was all the things I didn't realize needed to be solved.
Pre and post commands — the escape hatch
Not every interface is "connect, get file, disconnect." Some need something to happen right before or right after.
A vendor drops a zipped file — it needs unzipping after it lands, before the ERP job that reads it ever runs. A file needs archiving under a different name before the transfer starts, because a downstream job is still reading the old one. Or the transfer is really just step one, and step two is submitting the ERP load job the moment the data is safely in staging, instead of waiting on a fixed schedule.
Rather than hardcode any of that into the engine, each interface can define its own sequence of pre-commands and post-commands — plain CL, run through the same command executor, in whatever order the operator sequences them:
Pre-commands (before FTP): unzip inbound archive, rename prior file out of the way
Post-commands (after FTP): submit the ERP load job, notify downstream job scheduler
They're stored the same way the rest of the interface is configured — as sequenced rows against that interface — so adding a step doesn't mean touching a compiled program. It means adding a row.
This is the piece that turned the framework from "a way to move a file safely" into "a way to move a file safely and then make something happen because it arrived." That second part ended up mattering as much as the first.
The problems I didn't know I had
This is the part that genuinely surprised me.
I went in thinking: centralize the config, encrypt the passwords, add logging. Done.
What I didn't expect was how many failure modes you uncover the moment you have one central place to look at them.
Jobs that crashed and left the world inconsistent
The old scripts would copy from FTP output straight into production. If anything failed partway — network blip, disk full, somebody killing the job — you'd end up with a partial file in production and no clean way to tell what happened.
The fix wasn't complicated, but it changed everything: always archive the current production file to a timestamped member first, always FTP into a dedicated staging file (never production), and only copy from staging to production at the end, as the last step. If anything fails before the final copy, production is untouched.
[archive current] → [clear staging] → [FTP into staging]
→ [validate] → [atomic swap to production]
The existence of staging changed the shape of the whole program. It meant I could validate before committing. It meant I could detect empty files. It meant I could compare record counts to the last run. It meant I could reject a transfer that looked wrong before it ate the existing data.
Stripped down to the shape of it (generic names, not our actual field names), the execution engine looks roughly like this:
// Never touch production directly — everything lands in staging first
archMbr = %char(%date(): *iso0) + %char(%time(): *iso0);
CPYF FROMFILE(prodFile) TOFILE(archLib/archMbr) MBROPT(*ADD);
CLRPFM FILE(stgFile);
ftpGet(host: user: password: remotePath: stgFile);
if recordCount(stgFile) = 0 and not allowEmpty;
logAndHalt('E01: zero records received, expected data');
return;
endif;
varPct = ((lastRecCount - recordCount(stgFile)) / lastRecCount) * 100;
if varPct > maxVarPct;
logAndHalt('E02: record count dropped ' + %char(varPct) + '% vs last run');
return;
endif;
// Only now does production ever get touched
CPYF FROMFILE(stgFile) TOFILE(prodFile) MBROPT(*REPLACE);
Nothing exotic. The whole trick is ordering: archive first, validate in staging, and let production be the last thing written, not the first.
Jobs that died without anyone knowing
Every run now opens with an entry in a history table marked running. When it finishes clean, it gets flipped to complete. If it fails, it gets failed.
Which means: if I see a row stuck in running but the job isn't running anymore, something crashed. The next run of that interface detects the orphan, logs it, and moves on. No manual cleanup. No lingering locks. No mystery rows.
It's a tiny pattern — one column and three status values — but it's the difference between "silent failure" and "the system tells you what happened."
-- opening a run
UPDATE ftp_history
SET state = 'R', start_ts = CURRENT_TIMESTAMP
WHERE run_id = :runId;
-- closing it out cleanly
UPDATE ftp_history
SET state = 'C', end_ts = CURRENT_TIMESTAMP
WHERE run_id = :runId;
-- on startup, before doing anything else: find yesterday's ghosts
SELECT run_id, iface, start_ts
FROM ftp_history
WHERE state = 'R'
AND start_ts < CURRENT_TIMESTAMP - 1 HOUR;
-- anything this query returns didn't crash gracefully — it just crashed.
-- mark it 'F', log it, move on. No paging, no digging through spool files.
Jobs that retried forever
The old scripts didn't retry. If the server bounced you once, the whole job was a failure.
But sometimes the server just needs a second. So the new engine retries up to N times (per-interface, operator-configurable). Fine.
The interesting question was: what happens when retries run out?
Before, the job would just end in failure. But what if the operator is at their desk right now and could tell you something useful? What if they know the remote server is under maintenance and they want to skip this one and keep the rest of the batch going?
I wired it into the system operator message queue. When retries exhaust, the job sends an inquiry message: Retry / Skip / Cancel.
- Retry — reset the retry counter, try again.
- Skip — move on to the next record in the batch.
- Cancel — shut down the entire run.
It sounds like an old-school pattern because it is one. But for an operator-monitored batch window, it's perfect. The job doesn't die unnecessarily, and it doesn't loop forever either.
Interfaces that were broken for weeks
The old scripts had no concept of "this interface is broken, stop trying." If a vendor changed their SFTP credentials on a Monday and nobody told you, your job would fail every night at 2 a.m. for a month, sending 30 failure emails. And on day 32 someone would finally read one.
New rule: if an interface fails N times in a row consecutively (configurable per-interface), auto-disable it. Send the escalation email once, not thirty times. The operator has to explicitly re-enable it, which means they have to acknowledge what broke.
It's a small change. It ended the 3 a.m. email storm inside the first month.
Transfers that overwrote good data with bad data
This is the one I'm most proud of catching.
Every interface stores the record count from its last successful run. On the next run, after FTP completes but before production is overwritten, the engine compares the new count to the last one.
If the drop is greater than a configurable variance percentage (say, 50%), it halts and raises an error. Production is untouched. The operator gets called in to look at it.
The number of times this has caught an upstream batch that only wrote half its data to the output dataset… I'm not going to tell you, because it's embarrassing that this used to just happen silently.
The audit trail we didn't have
The maintenance UI writes an audit record for every add, change, delete, or restore — field-level, with before/after values, user, timestamp.
Password changes are always logged as masked asterisks. The actual value is never, ever written anywhere other than the encrypted field in the config table. Not in the audit log. Not in error messages. Not in the spool. Not in the job log. Not in emails.
(Credential handling is its own rabbit hole and its own article. For now: I use AES-256-CBC with the IBM i Cryptographic APIs, key stored in a data area with exclusive authority locked to the service program, and the plaintext exists in memory for exactly the duration of the login call and is cleared the moment login returns. If you want the details, see my next post.)
Operational shape, after
The whole system compiles in a documented order — physical files first, then the key data area, then the two service programs, the CL wrapper, and finally the three main programs. Operators run it with simple calls:
call the maintenance program // open the UI
call the FTP engine with (interface, code) // run one specific transfer
call the FTP engine with (interface, blank)// run all transfers for an interface
submit the FTP engine to a batch queue // scheduled batch window
submit the retention cleanup on a schedule // nightly housekeeping
No source changes. No recompile. New interface? Add a row. Change a destination path? Change a field. Disable one temporarily? Flip a flag. Restore yesterday's archive? Option from the maintenance screen, auto-logged to audit.
What I'd say to anyone thinking about this
If you're on IBM i and you've got a pile of FTP scripts like the one I had, I'd gently suggest that you have more technical debt than you realize, and less work than you think it will be to pay it off.
A few things I learned that were not obvious going in:
- Staging is the whole game. If production is never written to directly, half your failure modes stop being catastrophic and become "try again tomorrow."
- A history table with a state column is cheaper than a monitoring system. A running row that never became complete or failed is a crash. That's it. You just discovered yesterday's broken job this morning instead of Friday.
- Operators know things your program doesn't. A system-operator inquiry on exhausted retries sounds old-fashioned but is strictly better than either "fail silently" or "retry forever."
- Variance detection catches bad upstream jobs. Not just your own problems. Your upstream batch also has bugs. Stop letting its bugs corrupt your files.
- Auto-disable after N failures is a kindness. To you, to your team, to the operators who are tired of deleting 30 failure emails every Monday.
None of this is novel. Every pattern I used exists somewhere in the IBM i world, in some shop, written by somebody who already had the scar. The only thing that's new about this is that they're together, in one framework, running every FTP transfer we do.
What it feels like, afterwards
I'll tell you the thing I wasn't expecting.
After the rollout, the first week, we had a transfer fail. Remote server was down. The retry exhausted. The inquiry went out. Operator replied Skip. The run continued. The history table had one failed row and sixteen successful ones. The spool had a clean summary. The next morning, somebody logged in and saw the one red row, checked the remote status, clicked Retry from the maintenance screen, and it was done.
Nobody called me. Nobody sent a frantic email. No 3 a.m. page. The system told the operator what happened, they knew what to do, they did it.
That's what "modernization" actually looks like, I think, on IBM i. Not replacing the platform. Not rewriting everything in Java. Just taking the rough edges off the things that have been rough for twenty years, using the tools that are already there, and giving the people who run the business a reason to stop dreading their inbox.
Wrapping up
If you're in an IBM i shop and the word "FTP" gives you a mild anxiety response, you're not alone, and it's probably fixable. The platform has everything you need — SQL services, cryptographic APIs, subfile UIs, system-operator messaging, data areas, service programs. You don't need a new stack. You just need to decide the pile is the problem, and that one framework is the answer.
The credential encryption piece specifically — how I'm using the IBM i cryptographic APIs with AES-256-CBC, where the key lives, how it's rotated, and why you probably shouldn't write your own — is what I want to cover next. It's the part that should be easy and never quite is.
For now: if anyone from my old shops is reading this and you still have my old FTP scripts running somewhere, I'm sorry. I've done better since.
I am an IBM i practitioner working on modernization, integration, and warehouse systems.
Top comments (0)