We've got an old records system on an AS/400 that needed the data migrated. Short of writing a fully custom app to allow editing (due to the legal need to provide redaction capability forever), we needed some kind of front end. We have a robust document management system that provides all of the retention, security, search, redaction, and managed access features that we need. We looked at a ton of options, and this was the best fit.
So, I set about turning the data into a series of reports and feeding them into the system using their SDK. The first iteration just about killed the report server. At that point I learned how to generate reports locally on my batch machine, and how to do it in parallel. The new process resulted in the queues on the server being filled up, and the resources being saturated. Everything ground to a halt. My next lesson was in tuning the import process to not be so hard on the servers. The next hurdle was, in importing a couple of hundred million files, the SAN hosting the application's storage ran out of inodes. Every server hitting that storage location had a fit. There wasn't a lot I could do about that, but after the operations team cleared the issue, add that to the list of things I know to think about for the future.
I learned about a lot of neat things during that project. It gave me a ton of great lessons that I wouldn't have learned if stuff hadn't have broken.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
We've got an old records system on an AS/400 that needed the data migrated. Short of writing a fully custom app to allow editing (due to the legal need to provide redaction capability forever), we needed some kind of front end. We have a robust document management system that provides all of the retention, security, search, redaction, and managed access features that we need. We looked at a ton of options, and this was the best fit.
So, I set about turning the data into a series of reports and feeding them into the system using their SDK. The first iteration just about killed the report server. At that point I learned how to generate reports locally on my batch machine, and how to do it in parallel. The new process resulted in the queues on the server being filled up, and the resources being saturated. Everything ground to a halt. My next lesson was in tuning the import process to not be so hard on the servers. The next hurdle was, in importing a couple of hundred million files, the SAN hosting the application's storage ran out of inodes. Every server hitting that storage location had a fit. There wasn't a lot I could do about that, but after the operations team cleared the issue, add that to the list of things I know to think about for the future.
I learned about a lot of neat things during that project. It gave me a ton of great lessons that I wouldn't have learned if stuff hadn't have broken.