DEV Community: Julien Danjou

Auto-Retry Is the Duct Tape of CI — Here’s Why We Still Need It

Julien Danjou — Tue, 13 May 2025 09:27:23 +0000

You push a PR. CI kicks off. Ten minutes later, red.

You check the logs.

It’s a test that fails every now and then — you’ve seen it flake before.

It times out, or throws a weird assertion. Maybe it’s infra. Maybe it’s racey. Maybe you don’t even care anymore.

You click “Retry.”

It passes. You merge. You move on.

The Next Day

The next day, someone else pings you.

“Hey, did your PR break this test? It’s failing for me now.”

You don’t know. It didn’t fail for you. At least not finally.

You just reran it until it turned green.

This is where a lot of teams end up with CI:

Half trust, half habit. A system that “works” — as long as you’re willing to babysit it.

Auto-retry is duct tape. And we all use it.
I’m not against auto-retry. In fact, I love it.

CI isn’t deterministic.

Jobs fail for reasons that have nothing to do with your code:

Docker pull limits
Cloud runner cold starts
Network hiccups
Flaky mocks
“It passed locally”
¯\(ツ)/¯

Auto-retry keeps velocity up when CI would otherwise block shipping.

It saves hours.

It lets engineers focus on the code — not the randomness around it.

But duct tape works best when you know where it’s holding things together.

When retry is invisible, confidence erodes

I’ve seen teams where retry was the fix for everything.

“Just rerun it, it’s always like that.”

Builds fail, builds get retried, builds go green — and no one remembers to look back. It becomes normal.

We stop asking why jobs fail, because the fix is one click away.

We trust the green check, even when we know it’s masking something flaky.

The team’s confidence in CI drops. But so does the pressure to improve it.

That’s the quiet cost of auto-retry: it prevents disaster… and hides decay.

We built CI Insights to make this visible

When we built CI Insights, we didn’t want to replace your CI.

We wanted to show you what’s really going on inside it.

That’s why one of the first things we added was retry awareness.

CI Insights shows you:

Which jobs pass after retry (flaky)
How often each job flakes
What your “retry recovery rate” looks like
Which jobs eat up the most rerun time
When your team is stuck in a silent feedback loop

It doesn’t stop jobs from failing.

But it tells you whether you’re fixing things… or just getting lucky.

Confidence isn’t just about test coverage

It’s about knowing that when CI says “green,” it means it.

And if it had to say “red” twice before that, someone should know.

CI Insights gives you a clear view of that — and lets you automate retry with context.

By job, by condition, by intent — not just “try again and pray.”

Want to know what your retry pattern really looks like?

➡️ Join the CI Insights beta

CI Failures Don’t Just Break Builds — They Break Focus

Julien Danjou — Wed, 07 May 2025 11:32:22 +0000

A few weeks ago, something broke in production.

No big deal — we've all been there. I did what I’ve done hundreds of times before:

Checked logs, wrote a patch, pushed to a new branch, and waited for CI.

Except CI failed. Not because of my change, but because someone had merged a PR 20 minutes earlier that broke main.

Now I'm stuck. The fix is ready, but I can't ship it without bypassing the tests or waiting for someone else to fix what they broke. From a clean engineering process point of view, this is a disaster.

So I face the same choice many of us do in this situation:

Force-merge the fix and cross my fingers, or hold off on fixing prod.

I clicked merge. Crossed my fingers. And hoped that while I fixed one problem, I didn’t just create two more.

CI doesn’t just fail. It fails at the worst possible moment.

You can build a beautiful, automated pipeline. You can configure your cache keys, lint your YAML, and badge your build.

But none of that helps you when:

A flaky test randomly fails a PR at 6:30 pm;
GitHub Actions hits a Docker pull rate limit again;
A CI step you don’t control starts failing globally.

And here’s the kicker: most of us rerun the job, get a green check, and move on. No root cause. No visibility. No long-term fix.

We’re debugging by ritual.

The real cost isn’t money — it’s momentum.

When CI breaks, you don’t lose dollars.

You lose flow.

You're in the middle of a fix. Or a feature. Or just trying to finish the last ticket of the sprint. CI breaks, and now you're digging through logs from a job you didn’t write for a failure you didn’t cause.

You rerun the job. Still red.

Rerun again. It’s green.

You merge, slightly less confident than before.

Your 20-minute bugfix becomes a 90-minute support fire. And you’re left wondering:

Did I actually fix the bug? Or did I just push the pain to Future Me?

Why do we tolerate this?

Because CI is treated like plumbing.

If it works, you don’t think about it. If it doesn’t, you patch it, rerun it, and carry on.

Nobody “owns” CI quality.

Nobody tracks flake rates in a dashboard.

Nobody budgets time for fixing transient failures.

And so it creeps.

What was a one-off rerun last week becomes standard practice. What was a reliable build becomes a minefield of red crosses and Slack pings.

So we started building CI Insights.

We didn’t want to replace CI. But we wanted answers:

What jobs are failing the most?
Are they flaky? Or actually broken?
Which tests slow us down the most?
What’s our actual lead time from PR to prod?
Why are reruns our default fix?

So we built something that watches your CI without changing it.

No YAML edits. No instrumentation. Real-time observability, and answers.

CI Insights tells us:

Which jobs flake
Which jobs got slower this month
How much time does our team spend rerunning things
How much merge delay CI is actually causing
What our deployment frequency looks like (DORA-style)

It’s like going from looking at raw logs… to having a dashboard that tells you why your team is grumpy.

What you can do today

Even without CI Insights, you can start spotting CI drift:

Track retry rates. If your reruns are increasing, something’s decaying.
Monitor merge delay. If jobs block PRs more than they fail, you have friction.
Surface flakes. Build a script that scans past failed-then-passed jobs.
Watch the slow creep. Job duration going up over time? That’s silent tech debt.

CI isn’t a tool. It’s a mirror.

And when it breaks, it reflects the messiest part of our engineering process: the stuff we patch instead of fix.

Want to see your own CI like this?

We’re opening up the CI Insights beta for GitHub Actions users. It’s free while in beta and built for engineers like us who just want their pipelines to work — or at least make sense when they don’t.

We’ll show you:

The flakiest jobs in your org
Job cost and rerun trends
Real-time CI status across repos
Slack alerts and auto-retries for the noisy stuff

👉 Join the beta

Or email me at jd@mergify.com — we’re always looking for developers to help shape CI observability.

Interview: The Performance of Python

Julien Danjou — Mon, 11 May 2020 11:40:56 +0000

Earlier this year, I was supposed to participate to dotPy, a one-day Python conference happening in Paris. This event has unfortunately been cancelled due to the COVID-19 pandemic.

Both Victor Stinner and me were supposed to attend that event. Victor had prepared a presentation about Python performances, while I was planning on talking about profiling.

Rather than being completely discouraged, Victor and I sat down (remotely) with Anne Laure from Behind the Code (a blog ran by Welcome to the Jungle, the organizers of the dotPy conference).

We discuss Python performance, profiling, speed, projects, problems, analysis, optimization and the GIL.

You can read the interview here.

Being in Charge

Julien Danjou — Fri, 17 Apr 2020 12:03:33 +0000

If you never heard of the 10x engineer myth, it's a pretty great concept. It boils down to the idea where an engineer could be 10x more efficient than a random engineer. I find this fantastically twisted.

Last week, I sat and chat with Alexis Monville in Le Podcast — a podcast that equips you to make positive change in your organization. We talked about that 10x Engineer myth, and from there we digressed on how to grow your career and handle the different aspects of it.

This was a very interesting exchange. Alexis is actually going to publish a new book next month (May 2020) entitled I am a Software Engineer and I am in charge.

Lucky me, this week, I had the chance to be able to read the book before everybody else — which means I actually read after our recording. I understood why Alexis said that a lot of what I was talking about during our podcast resonated with him. I send a detailed review of the book to Alexis and Michael if you're curious. I'm definitely recommending this book if you want to stop complaining about your job and start understanding how to pull the strings.

I wish I had this book available 10 years ago! 😅

One year of Mergify

Julien Danjou — Thu, 12 Mar 2020 08:25:00 +0000

It has been close to a year now that I've incorporated my new company, Mergify. I've been busy, and I barely wrote anything about it so far. Now is an excellent time to take a break and reflect a bit on what happened during those last 12 months.

What problem does Mergify solve?

Mergify is a powerful automation engine for GitHub pull requests. It allows you to automate everything — and especially merging. You write rules, and it handles the rest.

Example of rule matching returned in GitHub checks

For example, let's say you want your pull request to be merged, e.g., once your CI passes and the pull request has been approved. You just write such a rule, and our engine merges the pull request as soon as it's ready.

We also deal with more advanced use cases. For instance, we provide a merge queue so your pull requests are merged serially and tested by your CI one after another — avoiding any regression in your code.

Our goal is to make pull request management and automation easy. You can use your bot to trigger a rebase of your pull requests, or a backport to a different branch, just with a single comment.

Some people like to make bots talk to each other.

A New Adventure

Mergify is the first company that I ever started. I did run some personal businesses before, created non-profit organizations, built FOSS projects — but I never created a company from scratch, even less with an associate.

Indeed, I've chosen to build the company with my old friend Mehdi. We've known each others for 7 years now, and have worked together all that time on different open-source projects. Having worked with each other for so long has probably been a critical factor in the success of our venture so far.

I had little experience sharing the founding seats with someone, and tons of reading seemed to indicate that it would be a tough ride. Picking the right business partner(s) can be a hard task. Luckily, after working so much time together, Mehdi and I both know our strengths and weaknesses well enough to be able to circumvent them. 😅

On the other hand, we both have similar backgrounds as software engineers. That does not help to cover all the hats you need to wear when building a company. Over time, we found arrangements to cover most of those equally between us.

We don't have any magical advice to give on this. As in every relationship, communication is the key and the #1 factor of success.

Getting Users

I don't know if we got lucky, but we got users and customers pretty early. We used a few cooperative projects as guinea pigs first, and they were brave enough to try our service and give us feedback. No repository has been harmed during this first phase!

Then, as soon as we managed to get our application on the GitHub Marketplace, we saw a steady number of users coming to us.

This has been fantastic as it allowed us to get feedback rapidly. We set up a form asking users for feedback after they used Mergify for a couple of weeks. What we hear is that users were happy, that the documentation was confusing and that some features were buggy or missing. We planned all of those ideas as our future work in our roadmap, using the principles we described a few months ago.

We tried various strategies to get new users, but so far, organic growth has been our #1 way of onboarding new users. Like many small startups out there, we're not that good at marketing and executing strategies.

We provide our service for free for open-source projects We are now powering many organizations, such as Mozilla, Amazon Web Services, Ceph and Fedora.

Working with GitHub

Working with GitHub has been… complicated. When you build an application for a marketplace, your business is entirely dependent on the platform you develop for — both in terms of features and quality of service.

In our case, we hit quite many bugs with GitHub. Their support has mostly been fast to answer, but some significant issues are still opened months later. The truth is that the GitHub API could deserve more love and care from GitHub. For example, their GraphQL API is a work in progress for years and miss out many essential features.

We dealt and still deal with all those issues. It obviously impacts our operations and decreases our overall velocity. We regularly have to find new ways to sidestep GitHub limitations.

You have no idea how we wished for GitHub to be open-source. The idea of not having access to their code and understand how it works is so frustrating that we publish our engine as an open-source project. That allows all of our users to see how it works and even propose enhancements.

Automate all the way

We're a tiny startup, and we decided to bootstrap our company. We never took any funding. From the beginning, it has been clear to us that we had to think and act like we had no resources. We're built around a scarcity mindset. Every decision we make is based on the assumption that we basically are very limited in terms of money and time.

We basically act like any wrong choice we do could (virtually) kill the company. We only do what is essential, we ship fast, and we automate everything.

For example, we have built our whole operation about CI/CD systems, and pushing any new fix or feature in production is done in a matter of minutes. It's not uncommon for us to push a fix from our phone, just by reviewing some code or editing a file.

Growth

We're extremely happy with our steady growth and more users using our service. We now manage close to 30k repositories and merge 15k pull requests per month for our users.

That's a lot of mouse clicks saved!

If you want to try Mergify yourself, it's a single click log-in using your GitHub account. Check it out!