Entering this world as a Level 1 engineer, I soon realized that my ambition was greater than my knowledge - a humbling part of my growth. The first big project I took on was creating a snapshot of our production database cluster for use in lower environments. This process involved creating a copy of the production cluster into a temp DB, anonymizing customer data, updating api keys and passwords, and then generating a snapshot that was ready to be used by developers and QA testers. At the time, it was a massive undertaking, and I was eager to prove myself. Now, looking back, I can see there were many things I could have done better with how much I've learned since then. Could I have anticipated these improvements ahead of time? In some cases, yes, though I lacked the skills to implement it. But in others, it was experience that guided my progress.
The process I developed was functional, but fragile. Any small error would require manual intervention, which made the process less reliable. Still, it worked - albeit just barely. It was a delicate machine that could break down if someone so much as looked at it. During this period, our team began to focus heavily on a new project, and time/energy/motivation to return to this original project became scarce. Manual workarounds, while inefficient, were the most effective solution. But as the inefficiencies piled up, so did my mental list of tech debt. Finally, I reached a point where I had enough and decided to revisit my once biggest achievement turned my now biggest headache. I had a list of issues to work on:
- What to do with the temp DB's that were generated if the process failed overnight? At the time, I was just deleting them and restarting the process. But this was... not great.
- Next was the time it took to add the encryption key to the snapshot. Initially it took about 10 minutes, but over time, that ballooned to nearly 45 minutes. When I consulted AWS support, their suggestion was to "just increase the sleep." While this was a quick fix, it didn’t address the root of the problem.
- Sometimes the DB would reject the connection while it was running my laundry list of SQL queries. This would cause the job to return an error code and fail outright. Ugh.
- Most importantly, this job is shared between multiple applications. This was one of the main reasons why it was so fragile. Should I separate it out into a different job per application?
When I set out to revisit this project, I had an epiphany. It was like coming back to the video game you struggled with six months ago and beating it on the first try. Every day, you make small improvements that seem insignificant on a day-to-day basis, but over time they accumulate. When I started working on the project again I realized how much I had grown. Not just in my technical skills, but in my approach to problem-solving. Here's how I went about it:
- For the orphaned temp DB's: Reuse them! While I created some logic around the rest of the process to make it more resilient, if the job had failed and left around the temp DB's, just reuse them and start the job from the steps after they would get created. Hindsight is 20/20, etc, but I for whatever reason I couldn't see it my first time around.
- The encryption key time... Admittedly I still haven't found an answer for this. And AWS support was never able to provide me with a clear answer as to why this happened. In the end, I increased the sleep time to compensate for this mysterious time increase, but I pray in the future I will gain additional knowledge that will grant me an "aha!" moment if I come back around.
- I'm still unsure why the DB randomly rejects the SQL connection. But for now I've added a catch and retry. A good solution doesn't have to be complicated!
- After considering whether to split out the job for each application, I decided to stick to my guns and share the same job file between our three apps. I think splitting it up would be easier, but this gives me a self-enforced challenge to work into my development. And with my increased experience and skills, I was able to make it work.
Looking back, I can clearly see what my original implementation was lacking. But for a long time, other tasks and complacency with manual workarounds kept me from addressing them. Once I decided to revisit it, I was amazed at how quickly I was able to come up with solutions for my prior shortcomings. Just a few days of work and testing compared to the drawn-out struggle I had faced before. The experience was a powerful reminder of how much I had grown. Not just in my technical ability, but also in my confidence and approach to problem-solving. This project, which for a long time seemed so daunting, now serves as a marker of my progress.
In conclusion, my encouragement to you dear reader, is to revisit that old project you know could be better. The one you worked on so long ago. You may not realize it now, but once you start looking at how you can improve it you'd be surprised at how far you've come!
Top comments (2)
Thanks for sharing this Alex! I just started as a Level 1 engineer last month and I can't wait to look back on my first project in the future. This is a good reminder for me to make sure I do!
This article is a funny reminder to me, because today, with a little more seniority and experience, I can confidently say without fear that my first code samples that were released into production really do suck more now than I initially thought!
(Being serious, thanks for the article, it is a friendly reminder about being humble and learning from your past mistakes as a developer).