It was September 2024 and I was in way over my head.
I had just completed a DevOps Engineering program (offered by AltSchool Africa) with "moderate" confidence in the skills I had learned during the 12-month stint.
I heard of a job opening and thought "why not" and applied. I was given a take-home assessment that looked very similar to my final project at AltSchool. I was so excited. I thought I would crush it and move on to the interview stage.
I was SO WRONG.
🚧 The Task That Took Me Nearly a Year
Deploy a simple API on Google Kubernetes Engine (GKE) using Terraform to provision both the infrastructure and define the Kubernetes (K8s) deployments.
At AltSchool, I had defined K8s deployments using YAML manifests — like a normal person. I didn’t even know Terraform could handle K8s manifests. But I figured, how hard could it be? With some help from ChatGPT and a lot of grit, I could get the assessment done in a weekend.
Well… I did. After 10 months.
Important to note: I couldn't create a Google Cloud account, so I asked if I could use Microsoft Azure and I was permitted - so that's what I used.
🧠 What I Had to Learn (the Hard Way)
This assessment snowballed into one of the most difficult and rewarding technical challenges I’ve ever faced. Here’s are some of the rabbit holes I fell down:
- Configuring a virtual network (something Azure Kubernetes Service usually handles by default) and creating a NAT gateway for the cluster.
- Securing the K8s API server without using
authorized_ip_ranges(this forced me to rethink access for my chosen CI/CD tool that would runterraform applyanddestroy- GitHub Actions) - Learning what Microsoft Azure services could help with monitoring and how to configure them.
- Learning what K8s Network Policies were, what they protect, and why they matter.
- Learning how to issue a TLS certificate for my name.com domain (that I intended to use to make the API accessible publicly, abstracting its IP address) using Let’s Encrypt with DNS challenge — a mostly unpleasant experience.
- Figuring out how to keep any sensitive data (tokens and credentials) secure but accessible to my CI/CD pipeline using GitHub secrets, GitHub Actions workflow environment variables and a here-doc command?! (I just found out that was what it was called while writing the draft for this article)
- Writing reusable GitHub Actions workflow scripts to automate deployment and destruction — and learned just how fragile CI/CD can be.
- I had clashes with NGINX Ingress on the cluster way too often - network policy misconfigurations, ingress creation blocked by absent admission controller endpoints, among other issues.
- My name.com domain expired {frustrated screams} - so I had to pivot to using the ingress IP directly.
- And bash scripting. Quite a bit of bash scripting.
🎯 What Changed?
This project broke me down and rebuilt me. I came in thinking I had a decent handle on DevOps. I came out realizing how deep the field is — and how much more I still have to learn.
But here's the key: I did learn. Slowly. Painfully. Iteratively. And now, I know how to deploy a secure, observable, automated microservice stack from scratch — using tools I’d never even heard of when I started.
📝 Why I’m Writing This
I'm not fully done with the project (I'm kind of a perfectionist and there are bonus points in the assessment), but I have decided to write a collection of articles that try to detail the issues I encountered that I can remember and how I worked through them. I don't intend to win like a Pulitzer Prize or anything (though I wouldn't mind) but I genuinely hope my insights can help others battle similar challenges.
In the coming weeks, I’ll publish a series of articles breaking down:
- How to deploy Kubernetes workload using Terraform,
- My solution to securing an AKS cluster,
- Azure Networking and NAT Gateway configuration
- Securing the K8S API server (without using
authorized_ip_rangesblock) - Managing a Private Cluster
- K8S network policies
- How to automate deployments,
- Using GitHub Actions (for Continuous Integration and Deloyment) and Secrets (to securely pass sensitive tokens and credentials into Actions workflows),
- What not to do with NGINX Ingress (please learn from my pain),
- Monitoring with Azure Managed Prometheus & Grafana,
- Issuing TLS certificates via Let’s Encrypt DNS-01 + name.com’s API,
- Remote Backend Management for Terraform,
- And many other topics.
Here is the GitHub repository link for those that don't mind spoilers
🙌 Final Thoughts
No, I didn’t “crush” the take-home assessment. I never even submitted it. But I learned more in these 10 months than in 12 months at AltSchool — and I came out stronger, more capable, and more confident in my abilities.
If you're in over your head right now — keep going. You’re not alone. And you might be a lot closer than you think.
Stay tuned — the war stories (and lessons) are just beginning.
Top comments (0)