Are You Really Ready for CI/CD?

My first job out of college was for a large software company. We had major annual releases of our flagship product and minor releases every month or so. We followed a traditional software release cycle.

Our customers were used to it. They chose whether or not to take an update. They were in control. That was a little scary since we were the vendor — not them.

We decided to make a move toward CI/CD, turn the process on its head, and take a modern approach to software development. As soon as a product feature is complete and tested, push it into production.

Sound good? It is, but that’s just the tip of the iceberg. There is so much more that goes into providing a high-quality experience to your end-users.

Photo by NeONBRAND on Unsplash.

Set the Expectation

When moving from a traditional software release cycle to CI/CD, you might be surprised to find just as much pushback from people internally as you find externally from your customers.

People don’t like change. It’s human nature. Give someone a routine and they will be happy. Moving to continuous deployment is like telling somebody, “Everything you’ve done up to now is wrong. Please relearn all of it.”

Internal struggle

Inside your company, you’re going to get pushback from all sides of the org. They have to rework customer contracts. They don’t understand how something could make it to production without a full regression testing cycle. They don’t trust the developers to not break everything.

There are going to be hiccups. You have approximately a 0% chance of pulling it off without a hitch on your first try. That is OK. Set the expectation that processes are changing and you’re ironing out some of the details.

Client expectations

Your existing customer base is accustomed to you behaving a certain way, but you’re now going to change the way you operate to something “better.” You are going to have to prove to your customers that this new model is better than what you’re doing right now.

You must convince them this is going to be worth changing their process for. This is your chance to prove that defect turnaround gets exponentially quicker. There’s no more waiting for a specific date to see a fix. Once the change has been made, it’s in their hands.

Photo by Gary Chan on Unsplash.

Ditch the Word “Release”

When you switch to CI/CD, a standard release model goes out the window. Instead, you must focus on small incremental changes since every code push is going to production.

There are no more “releases” with this method of deployment. You aren’t going to be shipping an updated version of your product that has a year’s worth of changes in it anymore. Instead, you will be delivering tiny pieces of functionality, bug fixes, enhancements, and features daily.

The cadence doesn’t need to be set. Whenever the developer finishes the code and the QA approves it, ship it. That could mean once a week or it could mean five times a day. The idea is that when it’s ready, it’s available.

You will need to figure out how verbose you want to be with your “release notes.” Since you aren’t pushing out releases, do you need them anymore?

Some companies have a tech writer dedicated to the development team. With each code push, the tech writer will come up with a brief, user-friendly description of what changed. The description is pushed to the “What’s Updated” page, which is broken down by day.

There are many ways to do release notes, and none of them are better than others. Do what works best for your company.

Photo by Brett Jordan on Unsplash.

Emphasize Automated Tests

The breakneck speed of CI/CD doesn’t allow for a manual smoke test of your application with every change. It would be impossible to keep up with the changes.

To instill confidence both internally and externally to your organization, you must build up a suite of automated tests that run whenever a change is made. Automate the smoke tests you would have done manually. They are completed significantly faster. Plus, it removes the possibility of human error.

On my team, we run two sets of automated tests with every code change. First, we have a robust set of unit tests that exercise the code. They test the success and failure paths for every function in our codebase. If the unit tests fail, the build fails — which means our changes do not get deployed.

If the unit tests pass, we move on to integration tests. We have built a strong set of Postman collections that run through the standard business processes in our app. These test all the business flows our app supports and they are constantly evolving. Once again, if any of these tests fail, our build fails — and nothing gets deployed.

Only when all unit tests and integration tests pass do we know that we didn’t break backward compatibility or affect another component of the application adversely.

Automated tests are a “build once, test forever” investment.

With a little bit of upfront cost, you can guarantee that a specific business process will always work when you make changes in your app. When you make production pushes five times a day, having that kind of confidence is an absolute must.

Photo by Quino Al on Unsplash.

Know What to Do When Things Go Wrong

Let’s say a developer pushed a code change that caused a system outage. The automated tests hadn’t been updated to test the scenario, so it snuck by and made it into production.

Do you revert the change? Do you find a fix and push another change?

With CI/CD, especially with a microservice architecture, you need to adopt an “always forward, never back” mentality. Do your best to resolve issues with a code change that progresses your app forward instead of going back in time to an older commit.

Things will go wrong. They always do, and that’s OK. Having a strategy in place before moving to CI/CD will help curb the anxiety of the developers, support, and clients.

One of the best things you can do to help catch when things are about to go horribly wrong is to set up proactive monitoring.

You want to know there is a problem before your customers do.

There are many ways to do proactive monitoring, and it varies greatly depending on your architecture. One effective method that you can use no matter what architecture you have is something like a Postman Monitor.

Monitors run a set of tests throughout the day and alert you if anything goes wrong. You can make them run through security audits, business processes, or validate API contracts. All you need is an API.

Photo by La-Rel Easter on Unsplash.

Be Smart About How You Break Up the Work

The CI/CD mantra is small, incremental changes. We all know that not all new app features are small. Maybe you’re implementing a search feature for your application.

Adding a search requires a number of changes:

• Setting up Elasticsearch.
• Populating Elasticsearch indices.
• Creating a way to get new data added to the indices.
• Building the search endpoints.
• Providing a way to search in the UI.

With a traditional release cycle, you would simply build all these components and ship the final product. But that’s not how it works with CI/CD.

Each bullet point above is going to be a separate push to production. You will incrementally build the feature up over time, and the very last thing you do is provide a way to access it through the UI.

You can and should build your back-end functionality first so you can assemble automated tests and verify it works as expected. After you know it works, then you can show your users that the new feature exists.

To your customers, you just released a giant new feature. In actuality, it’s been there for days/weeks/months — just invisible. You’ve built it, tested it, automated it, and regression tested everything. Then you can hand over the keys.

Conclusion

As with most things, CI/CD is not as simple as it sounds. If you want to build secure, reliable software, you have to understand the ins and outs of what you’re getting into.

Set the expectations that things are changing beforehand. Don’t just rip off the Band-Aid.

Get into the small, incremental change mindset. Feel confident pushing to production multiple times a day.

Automate everything. Seriously, automate everything.

Put contingency plans in place. Have a process to troubleshoot, fix, test, and deploy a high-priority issue in minutes or hours — not days or weeks.

Be smart about how you build. Your app gets better little by little every day. Keep your changes small and focused. Put a development plan in place before you start coding.

CI/CD is absolutely worth it. After you get over the “everything is different” shock, you’ll wonder how you ever did development any other way. Fast turnaround on issues, features developed in the blink of an eye, and a clear view of your system’s health.

Good luck!

---

Like this content? Find the original article and plenty more on my blog!

Comments

[Vinicius Rio]

Hey man, thanks for this overview. CI/CD is a large jump in direction to become a better development and this is very difficult because disturb a cult that delivers a great thing. Thanks.