Every engineer knows the feeling. You finally fix the bug that's been blocking your ticket, you push it, and you sit there waiting for that beautif...
For further actions, you may consider blocking this person and/or reporting abuse
The message you sent to the account owner is the most underrated part of this post. Clear, says what's broken, what you already checked, and that your change is ready. A lot of people take years to learn to escalate like that.
Another "my code is fine, the pipeline isn't" classic in GitLab: protected CI/CD variables. If a secret is marked protected, it's only exposed to pipelines on protected branches and tags. Push the same commit to a feature branch and the job fails with an empty variable, often as a confusing auth error from Terraform or the cloud CLI rather than "variable missing". Worth checking whenever a job works on main and fails everywhere else.
And +1 on self-managed runners for the heavy jobs. That also keeps your Terraform credentials on infrastructure you control.
Thank you Kiell! Honestly, that message came from being too nervous to send anything vague 😅 I wanted whoever read it to know straight away that my change was ready and the only thing missing was compute.
The protected variables one is such a good catch, and it’s sneaky because the error doesn’t point you there at all.
Nervous turned out to be the right instinct here 😄 A message that says "the change is ready, the only thing missing is compute" is the one people can act on straight away. And yes, that protected-variables error sends you looking everywhere except the actual cause. Great write-up, Maame!
Thank you Kiell :D
I've lived this exact shape of failure on GitLab — the pipeline that fails with zero logs while your code is fine. In our case it was a stale group runner token, so no runner ever picked the job up and GitLab gave us nothing to grep, which taught me the same lesson you landed on: once there are no logs, the question stops being about code. Your point about usage quotas living on the account rather than the user is the one I'd tattoo on every junior's forehead: CI minutes are a shared resource, and only the group Owner can see them, so treat the quota like a budget you check in standup. One addition from my side: caching Terraform providers on the runner won't save you from an exhausted quota, but it makes every minute go further when you're re-running plans all day. Has your team turned on auto-cancel for redundant pipelines yet?
A stale runner token sounds even worse, Robert. At least I got a message. You just get a job that never starts. Treating the quota like a budget in standup is a great habit, and good point on caching providers. We haven’t set up auto-cancel yet, but it’s next on my list.
You are right that a silent never-start is worse than an error message, because there is nothing to search for and nothing to retry against. The check that saved me in GitLab was a small pre-flight job that validated the runner token and the quota before the real job queued, so a stale token failed loudly at the start instead of vanishing. When you add auto-cancel, will you cancel the whole pipeline or only the superseded job, so a slow but valid run does not get killed by its own retry?
Great write-up, Maame!
I really appreciate how you turned a frustrating internship experience into something other developers can learn from. One thing I’ve learned from managing CI infrastructure is that a failed pipeline doesn’t always mean failed code. Sometimes the problem is simply that the infrastructure never gave the job a chance to run. 😄
I’m particularly glad you mentioned self-managed runners. I’ve found them really useful for keeping CI workloads under my own control, although they do come with their own maintenance responsibilities.One small consideration with rules:changes: shared Terraform modules and dependencies need to be included in the rules too, otherwise important changes might accidentally skip validation. As for your question, one of the most frustrating situations is when a runner becomes unavailable while the code is perfectly fine. Everything is ready, but the pipeline just sits there waiting. 😂
Thanks for sharing your experience so openly. Keep learning and building!
Hi Mustafa, thank you so much! 😂😂 I was so stressed out in the beginning because I honestly thought I broke something. Seeing 5 red jobs and no logs at all, my first thought was “what did I just do?” Then I hit retry and got that “Status cannot transition via drop” error and it got even worse 😅
And thanks for the add up on rules:changes! That makes a lot of sense. If someone updates a shared module and the job only watches its own folder, the change could slip through without being checked. Will definitely keep that in mind and include the shared modules in the rules.
The runner one sounds just as frustrating, especially because nothing tells you it’s not your code. You just sit there wondering if it’s going to start or not 😂
Haha, I can imagine that moment! 😂 Five red jobs, no logs, and then GitLab throws an even scarier error at you. That’s enough to make anyone question their entire career for a few seconds! 😅
The funny thing about DevOps is that sometimes we spend more time figuring out why something didn’t run than debugging what actually ran. 😂
But you handled it well, Maame. You investigated, asked the right people, and turned the experience into something useful for others. That’s what matters.
Keep sharing these stories! I genuinely enjoy reading about real engineering experiences, especially the messy ones we all learn from. 😄
Thank you so much, Mustafa. Your comments really mean a lot and I learned something new from you too. I’ll definitely keep sharing the messy stories, there will be plenty more
Thanks again for reading and taking the time to share your experience! :)
Ayoub's line sums it up: once there are no logs, the question stops being what is wrong with my code and becomes who owns the thing that never started. Most people learn that much later than an internship, so writing it up this clearly is useful.
Two small things have helped on teams I have worked with. First, put the CI quota on the same alert list as disk space and certificate expiry, so the person with the Owner role gets a warning at 80 percent instead of a pipeline full of red with no logs. Second, keep a short line in the project README that says who owns the runners, the quota and the billing. It turns the message you had to write into a two minute job.
Your standup summary, what is done, what is blocking it and who is sorting it out, is also a good template for any blocked ticket. Did your team add a quota alert afterwards?
I think adding who owns the runners is the README is a brilliant idea. We did leave a quota alert afterwards so that hasn’t happened in a while
A failure with no logs is a different class of problem. The question stops being what's wrong with my code and becomes who owns the thing that never started. Finding that person was the real fix here, and nobody teaches it.
Exactly 💯
The "code is fine, pipeline is broken" case is so disorienting because there are no logs to start from. CI minutes running out silently is a sneaky one to catch.
Right?! That total silence is the absolute worst. When you're so used to scrolling through a wall of red logs to find your mistake, seeing nothing makes you question your whole setup (and your sanity for a second). Glad to know I'm not the only one who's gotten tripped up by the silent compute limit trap