The most successful engineering teams typically keep their workflow durations between five to ten minutes on average, according to CircleCI's 2022 State of Software Delivery. The median is 3.7 minutes.
Looking at our team's slowest GitHub Actions using our GitHub App, our longest workflow is just under 13 minutes. This excludes local builds and workflows.
But it's not unusual to see build times measured in hours, depending on the stack you're using.
LinkedIn's engineering org previously wrote about the steps they took to decrease build times from 60 minutes to 15 minutes for their largest microservice, improving productivity and team happiness.
What's the longest build or workflow time you've ever encountered? What tools were you using / could you fix it?
Latest comments (28)
I am still waiting on my build from January 6th 1977 on my TRS 80 Model I (4K memory). I wrote code to calculate "The answer to life, the universe, and everything".
Although I heard there was some book about this with the answer, I am going to wait for my code, or the Vogon's, whichever comes first. ;)
Back then end of 80ies on IBM Main Frame changes to CPG (a German RPG derivate for "Transaction-Programming") CICS programs always needed 45-60 minutes of "compilation". No syntax-checking in the editor, so you had to be very thorough. Many years later when switching to Visual Basic for small applications my development style really degraded quickly into this hit-F5-and-see.
I worked on an application that was deployed as an entire virtual machine on top of cloud providers or, more often, ESX. There was no local development environment. We would spin up virtual machines to test and do QA on. The build itself took 10-15 minutes.
However, before being able to merge a PR we needed to run a set of precommit tests many of which couldn't be ran locally because they were testing other architectures. These tests ran on a locally hosted Jenkins instance. Even with them parallelized the test suite took 90 minutes and sometimes over 2 hours.
I've since worked in more modern development environments and couldn't go back. The cycles were absolutely brutal. We spent a ton of time making sure the test suites were stable, but we'd still have flaky tests or issues with infrastructure. It would sometimes take a full day to get a clean run.
Nine months. It took nine months to build a human out of a npm install child command. It was the worst nine months. Never again
I work with OSF with React. The build taking a list 5 minutes. The problem iss that every single line of code changed it's necessary to build again.
I use to work on a C++ project, wrapped in Java and .NET. The full build+test took about 10h, depending on the machine.
One day, we were pissed off fixing bugs in the same part of code again and again, so we wrote a test to test all the possible combination of values (only nominal ones, not even limit case or stress case). After a few hours of running, which seemed quite long, we had a look to progress to estimate how long it would takes.
This test would require more than 300 years to run... Yeah, combinatorial explosion...
We end up continuing fixing bugs case by case and add a test case when user report a bug...
Half an hour. Later I discovered my HDD was dead โ ๏ธ
Oh wow well back in the day (long long time ago!) I was doing 6800 assembler on Motorola Disk Operating Systems, it was a 2 pass macro assembler so needed to resolve the labels, macros, then compile and link so about 20 minutes each time.
That said even now I see plenty of jenkins pipelines that take 15-20mins to run.
If by "build" you mean a workflow in the CI server then we had an end-to-end test suite for PDF files processing which took something like five hours to complete and was run nightly.
about 40 minutes was my longest. it was a hyper ledger fabric blockchain project. and we automated it all the way through. generating crypto material, starting a complete network of needed services. extra node and golang services. then run and deploy the actual application on the blockchain and execute some test transactions.
A year if you consider at a certain previous employer we want to work, built it, it failed. It took maybe an hour a time and this would happen for 9/10 builds then we did this for over a year the whole process was known as "The Build" and it had an external agency maintaining it.. which is why we could never fix it try as we might
React build with test, 45 minutes. We now split up the fronted with webpack module federation and deploy it as a micro frontend
Well, I was compiling the Linux kernel on a standard machine in the late 1990ies and early 2000s (several times in fact)...
4 hours. I do have to mentioned that it was a final release and deployment of project ,so a little more than "build" was happening.
I used to work for a then startup as a college fresher, sharing roles of Q/A and customer support. My boss had send intimation email about down time. But downplayed the time it would take. He mentioned around one hour, but it took little over 4 hours, plus time it took to making network online. Which meant it was my worst night on customer support.
This was back in 2013. Now, they have moved from a monolithic dotnet codebase, to golang micro-services. Their staff consists of ivy league students, they haven't had a noticeable downtime in 5 years.
First of all there's a BIG difference between build and deploy stages.
I'm not convinved that using build times to measure "successful engineering teams" is a fair metric.
Having a build that lasts a subjective big amount of time is not necessarily bad.
i.e. you simply can't build and deploy a business production ready Next JS App in less than 4/5minutes. But even it lasts 1h it's not an issue. You can build and deploy when it's ready, which will last <2min more or less.
Things you can do to optimize the whole release process:
As the project grows and you need more dependencies, there's more code to evaluate, minify, bundle.... the more time it will take to build. It's ok, nothing wrong here.
On the other hand, having a pipeline that lasts more than 2 minutes to deploy is something you need to look at carefully and optimize it to reduce the downtime.
Optimizing Build and Deploy stages are a different world.
Apart from caching dependencies to speed up the build stage the only actions you can
And on the deployment side you can:
Disclaimer: I'm not much into infrastructure and this comment is propelled by my observations as lead dev and conversations with the Infra and DevOps guys. So If I miss something or I'm wrong on anything please tell me so I can learn more. ๐
This. I've seen enough insanely large Drupal projects with several gigs of database init dump import, cache warming, config and default content imports, XML imports and whatnot, that took up to 45 minutes until it they were considered built. The choice of framework and all code is actually sound, it just takes ages because they do a ton.
That's a great distinction to point out between the build and deploy stages. They can be optimized in different ways + have different expectations for how they'll run.
I used to work on a decently sized Gatsby site. Cutting out some dependencies and adding better caching took the final build from ~30 minutes to less than 10. They also added incremental builds to speed things up on smaller changes. The deploy step was always pretty fast -- unless it failed for some unknown reason...
ROLLBACK, ROLLBACK!
๐