Nobody really wants to see how sausage is made, do they? The process is messy, unglamorous, and usually skipped over in the final presentation (and oh boy, are we glad of it! 🤮🤮🤮)… but if you skip the "sausage-making", you never get the "full omelette"!
mmm, sausage...
...But we're not talking about breakfast foods today, we're talking about technology! In our world, most people would prefer to focus on shiny new features or huge cost savings stories rather than attend to the gritty details like testing, refactoring, and tooling. But without these "sausage-making" activities, the full DevOps "meal" (with all the acceleration and improvement that we crave) is always just out of reach.
My team at work recently went through this firsthand... so today we're going to explore an instance of our "sausage-making" journey to better Code Quality and then show you how it directly maps to the "omelette" of DevOps outcomes.
How the Sausage Was Made 🐖
Like many teams, we weren't very disciplined at first. Oh, don't get me wrong, we were checking all the required boxes:
- Our code was stored in the company's official Version Control System.
- Pull Requests were reviewed and approved according to Governance standards.
- Our changes were deployed by CI/CD.
- We had dependency analysis scans in our pipeline.
In Company terms, our DevOps practices were on track... dare I say, even a little bit progressive? BUT if we look closer, we start to see some cracks in the armor:
- Most of our PR reviews just functioned as a way to follow the Company rules. There were cases where the approval on several hundred lines of changes was granted within half a minute (in the olden days when we wrote on paper, we called this "Pencil-whipping" ✏️ Come at me, whippersnappers! 👴).
- Testing? Even the little bit that existed wasn't integrated into the pipeline. It ran only if you remembered to run it.
- Quality analysis tools? Ehhhh....
So yeah, we were moving fast, but... stuff broke sometimes. Every code push became a sort of loop:
- Get PR approval and push
- Uh oh, why did the deployment break?
- Fix errors, new PR with changes
- 🤞 GOTO 10
It was clear that our goal wasn't moving faster, but moving more sustainably. But to get there, we needed to make some sausage first. So we elected to make Code Quality a priority. Here's what that means to us:
Centralize Our Tooling 🎯
We started our journey by taking on a few research tasks to pick out a minimum set of tooling that we all would use across the team.
IMPORTANT
This isn't about heavy-handed governing of our workstation setups, but about coming to an agreement as a team... we identified a small handful of extensions and tools that we determined would be 'our stuff'.
For example: we all had linters in our VSCode installs, just... not the same one. And even though most of them are pretty solid, slight differences between rules meant you couldn't trust that because a change was fine in your editor, it would be fine for someone else.
How we decided which tools win:
- The tool has to be well-accepted in the industry.
- If it's open source, make sure it's not a 🧟 “zombie project” (it has recent activity in the project's repository).
- Able to implement within CI as well as locally in a developer's editor. This is critical: we should be able to run it locally and see the results, but we also want the pipeline to be able to stop if something bad slips through.
- It has to be almost zero extra work for a developer to use regularly. “Don't make seatbelts out of a cactus🌵.” When you're adopting quality tools that will potentially block progress, don't let them make the developer experience worse, or the team will revolt against them!
- It has to be customizable - if we have a team decision that we all agree should override industry standards, we should be able to easily reflect it in our guardrails.
There were of course some tools that are already provided by the company, such as Blackduck and SonarQube. Since those are either already required or roadmapped to be required, it was important to us to be able to implement those as well.
Implementing in the Pipeline 🛠️
Once we defined our list of quality tools, we made them run in our pipeline on ephemeral copies of our code... we build in containers that are created in the pipeline and will be destroyed when we're done with them; this forces us to be acutely aware of every dependency and quirk. As code moves through our environments, we should be scanning it regularly and aware of its status, so we can catch mistakes or security problems EARLY.
IMPORTANT: Implemented != Enforced
The #1 mistake that teams make when trying to improve their code quality is biting off more than they can chew. If you've lived in a world without quality checks for a long time, it's painful at first to bring the tools into your environment... make life easier on yourself by waiting to block merges until you have used the tooling a while and done some cleanup!
We found that SonarQube was already scanning (but not enforcing results) on every PR we submitted. While that seemed like it could be a quick win, it... wasn't. The results were ugly... we had a couple hundred security hotspots. We had a couple hundred code smells. And even though we already had a few unit tests in there, coverage wasn't being reported to Sonar because of the structure of our repository.
Security Hotspots are identified by Sonar as 'things that could potentially be exploitable'. These aren't vulnerabilities, per se: think of them like a Tornado Watch. The conditions for a twister exist, but we haven't seen one form yet.
Code Smells are a wide-ranging category of stuff that might be working just fine, but could be done more cleanly or carefully... "it just smells bad".
Code Coverage is a metric that defines the percentage of your code that's actually executed by your automated tests. This is important because I could very quickly write a unit test for my application and get it to pass, but if it doesn't check all the different branches and paths through my application, I'm not really any safer because of it. Covering as much of my code as possible with testing is desirable because it's a sign I've tested comprehensively.
Over the next few weeks, we worked on all these issues: First, we got the tooling to report reliable information. We had to custom-code a GitHub Actions workflow to handle our unit tests because it's a monorepo with multiple languages in play. Then we had to define a strategy for how those tests would be run, their results aggregated, and the final data delivered.
Visibility into this data sort of automatically uncovered our next task: We also had to WRITE an awful lot of tests to cover the code we had slung into production over the previous few years. And that led us on yet another side quest: a lot of our code wasn't designed for easy testing! It often required refactoring to make it easier to test, or to eliminate the smell/hotspot... so we refactored.
We lived with a simple mantra: Get. To. Passing.
Let me tell you though: there's something incredibly rewarding about seeing those metrics align with each new push. Pure dopamine!
We Haz Sausage Now 🐷
Today I'm pleased to share that we are down to 5 code smells (from 150), 0 security hotspots (from 130), and test coverage OVER 85% (from ZERO!). This work took us (in addition to our other day-to-day tasks) a month to complete. The race for quality has no finish line, as the motivational posters tell us... will we get this repo to look "perfect" in our scan? Maybe. Maybe not. But our stats look more respectable than ever!
A Quick Retrospective: How'd We Get Away with This? 🏆
So, Blink, your team completely turned around your AWS Platform's codebase in a month? That's… aggressive, dude. How?
I think there are several factors that enabled us:
Opportunistic approach to team capacity. We finished a couple larger tasks early in the summer and there were a few projects in the pipe that needed additional planning time, which left us with a little gap where our team had some margin. The lesson we should learn here is to provide a little space. Ask me about Queuing Theory and Side Project Time sometime (but come prepared for an earful 👂)!
Our Leadership was… motivated 😊. The need to clean up our code hit just about the same time as a new "Divisional Strategic Framework" was published - with lots of talk about a “Pillar of Operational Excellence”. Our management was already looking for things that we could use to explicitly define the pillar. Showing up after 2 weeks with a dashboard full of quality metrics that demonstrated the progress made in the past sprint was like visiting a dog kennel wearing a ribeye t-shirt! 🥩🐶
Leadership got out of the way and let us work. You might think this is a no-brainer, but… indulge me a moment! I want you to think about Goodhart's Law: “when a metric becomes a target, it ceases to be a good metric.” It would be extremely tempting for a manager to see that fancy dashboard and immediately start “setting goals” for us… to “drive the adoption”, if you will. But in doing so, they'd undermine the progress being made! I believe we succeeded because we were empowered to deliver better quality. And for those who worry that the team would “slack off” if left to our own devices, here's something for you to noodle on: Our team set a code coverage goal of 10%, and we hit 80. #intrinsicMotivation🎤💧.
Now Let's Talk About the DevOps Omelette 🍳
As I mentioned in the opening, nobody cares about how the sausage was made. They're more concerned with the important stuff... Security/Compliance remediations. Cost savings. New features. Customer requests. Nobody cares about your code coverage, Blink.
...or DO they???
To find out, let's get practical: Imagine that you have a finding you need remediated, and it's going to require a code change.
A Tale of Two Remediations
Without Quality Tooling ☹️
- Manually review the code to see if the proposed change is safe to make 🔍🧐
- Make the Change
- Debate with your Teammates. Figure out a “Safe Time” to deploy when you're likely to break in front of the smallest group of customers. Get signoffs from everyone in the company, provide blood sample, etc.
- Push it and 🤞🐇🍀🙏
With Quality Tooling 😎
- Make the Change
- Run the Test Suite 🤖✅
- LGTM, ship it 🚢🚢🚢
- Go Find Your Next Task
Do a little napkin-math on those two paths and I bet you'll see what I'm talking about... Having a robust test suite and well-implemented quality tools makes you faster.
The Big Picture: Quality as an Enabler of Velocity 🖼️
When people talk about trade-offs in software, "quality versus speed" is one of the first items that comes to mind. On the surface, it feels logical: spend less time testing and reviewing, and you'll ship faster. But our experience shows the opposite.
By investing in our code quality, we've created the conditions requisite for speed to exist. Here's how:
- Tests give us confidence. With a solid coverage baseline, we can change code without holding our breath. Regressions will be caught early.
- Automation reduces friction. Sonar and similar tools take debates out of the code review process. Instead of arguing about formatting or chasing down hidden issues, reviewers focus on design and functionality. The result is faster, cleaner merges.
- Automation also encourages consistency. "Did you remember to run the tests before you pushed?" Don't care. They'll be run by the CI. "Did you get sloppy because you were in a hurry?" Can't. Pipeline tools block sloppy mistakes and we hold ourselves to our own standards.
Remember: the goal is not to try and move faster, but to move sustainably. Speed without Quality is a sugar rush: you'll feel great for a sprint, then you crash hard as your bugs, rework, and tech debt pile up. Focusing on quality makes every acceleration reliable and repeatable.
The Breakfast Test 🥓🍳🥞
Sausage on its own isn't much to brag about. It's messy to make and not typically ordered in isolation. But when you fold that into an omelette, you have something tasty on your hands. It's satisfying… it's complete.
Code quality work? Same. By itself, adding tests or refactoring hotspots can feel like overhead. It doesn't demo well. It rarely makes a slide in an exec review. But if you do it alongside your feature work? Those investments make you better.
How's your omelette?
--blink
Top comments (0)