Every engineer knows the feeling. You finally fix the bug that's been blocking your ticket, you push it, and you sit there waiting for that beautiful green tick.
That was me during my internship. My Terraform kept failing to create a database server because of a disk error. With some help from a senior engineer, I found the problem: my AMI filter was grabbing the wrong image, one that needed a bigger disk than I'd asked for. I fixed the filter, pushed the change and waited.
The pipeline didn't go green. It didn't really go red either. It didn't run at all.
Sometimes your pipeline is broken and your code is completely fine.
Here's what happened, how it got fixed, and what I'd do so it never catches you off guard.
1. Five Failed Jobs, Zero Logs
The pipeline page showed the whole thing as Failed. 5 out of 7 jobs, all red.
Normally that's fine. You click into a job, open the log, scroll to the red text and find your mistake.
This time there was nothing to read. The failure reason just said:
No more compute minutes available
No job log
No stack trace. No Terraform error. Nothing. The jobs never even started, so there was nothing to log.
If you ever see a pipeline full of failed jobs with no logs at all, check this first.
2. Retry Made It Look Even Worse
Of course I hit retry. (We all do.) Instead of running, GitLab threw this at the top of the page:
An error occurred while making the request.
Status cannot transition via "drop"
That one looks properly scary, like something is broken inside GitLab itself. As far as I can tell, it's just GitLab failing to update jobs that never got the chance to start. Same problem, more confusing message.
Retrying won't help until the minutes are back. You'll just get the same error every time. Once the minutes were topped up, one retry was all it took.
3. Compute Minutes Belong to the Account, Not You
Here's the part nobody tells you as a junior: you probably can't fix this yourself.
I told the senior engineer reviewing my change. She couldn't check the monthly quota either, so she pointed me to the colleague who manages our GitLab account. I found out later why: viewing a group's usage needs the Owner role, so a lot of developers can't even see how close they are to the limit.
So there's no point digging through your code. The fix is a message to the right person. Here's roughly what I sent:
Hi, our GitLab pipeline on the Terraform project has stopped running. It says we've run out of CI compute minutes for the month. Would you be able to check the quota when you get a chance? I've got a change ready, it just can't run until there's compute available.
He confirmed we'd run out and topped up the account.
The annoying part was the timing. Between waiting on a review and this, my ticket barely moved for days, and I had to explain at standup that my work was ready but nothing could run. If you're ever in that spot, keep it simple: what's done, what's blocking it, and who's sorting it out.
4. How to Check Before It Happens
If you're an Owner on the group, go to the group (not a subgroup), then Settings > Usage quotas. On a personal account, it's Usage quotas in the left sidebar. (GitLab docs)
A few things I wish I'd known:
- Usage resets every month, so this usually bites near the end of the month.
- GitLab renamed "CI/CD minutes" to "compute minutes" in version 16.1, so older Stack Overflow answers use the old name.
- If you're not an Owner, ask whoever is to keep an eye on it. Way easier than finding out mid-ticket like I did.
5. How to Make Your Minutes Last Longer
If I wanted to stop this from happening again, this is where I'd start.
Only run jobs when their files change. If your Terraform is split into components, there's no reason to plan the database when you only touched the load balancer. rules: changes: sorts that out:
plan:db:
script:
- terraform -chdir=db plan
rules:
- changes:
- db/**/*
Stop duplicate pipelines. A push to a branch with an open merge request can start two pipelines, one for the branch and one for the MR. A workflow: rules block at the top of .gitlab-ci.yml can make sure only one runs.
Cancel old pipelines when you push again. Pushed three fixes in five minutes? You don't need all three pipelines to finish. Mark jobs as interruptible: true and turn on auto-cancel for redundant pipelines in your CI/CD settings.
Cache what you download every time. Dependencies and Terraform providers get downloaded on every single run unless you cache them.
Make expensive jobs manual. If a job doesn't need to run on every push, when: manual is your friend.
Use your own runners for heavy work. According to GitLab's docs, project and group runners aren't affected by the compute quota. If your team already has a server, a self-managed runner can take the heavy jobs off your quota completely.
Final Thoughts
The funny thing is, my fix was right the whole time. The pipeline just had nothing to run on.
So I'm curious: what's the most annoying reason your pipeline has failed that had nothing to do with your code? Drop it in the comments. I want to collect them all 😅
Top comments (0)