DEV Community: Dan Peach

Dogfooding At Armory

Dan Peach — Mon, 18 Apr 2022 15:46:16 +0000

We recently onboarded the main server component of our new software delivery platform — codenamed Borealis — onto Borealis itself. I’m going to talk a bit about Borealis, the experience of dogfooding at Armory, and what we hope to learn from it.

Borealis and Deploy Engine

The application we onboarded is called Deploy Engine. The name is uninspired and is out of place among the sea of ostensibly Grecian software systems — Kubernetes, Argo, Tekton, etc. — but is descriptive of its role within Borealis: it orchestrates software deployment strategies.

Using the Borealis CLI or API, a user can provide a high-level description of their deployment: First, I’d like 10% of production traffic to go to my new software version. Next, I’d like the deployment to wait thirty minutes and then I’d like to run a canary analysis of application metrics. Next, I’d like to increase traffic to 50% — and so on. Deploy Engine, in concert with a few other subsystems, orchestrates this kind of deployment workflow.

Dogfooding at Armory

In my experience, the mental state of even the most empathetic, thoughtful, and experienced software engineer is divorced from that of a user.

I have two analogies:

I am a very poor actor but during high school was required to act in classes and in friends’ short films. The experience of using software for the first time as a user — rather than as a developer — is a little bit like the watching a recording of yourself as a bad actor. It can feel stilted or inauthentic; the transitions from one moment to the next can be painful.
Experiencing software for the first time as a user can mirror the experience of sharing a favorite video or song with a friend. What you once experienced as beautiful, catchy, moving, or hilarious can suddenly feel trite, bland, or tasteless as you hear the song through the ears of your friend. It can be jarring or unpleasant; it can also teach you about your friend and their values.

Dogfooding is hard, painful work, but it’s one of the best ways to experience your software as your users will. It’s not a replacement for user feedback — we have biases and beliefs that inform how we experience our software that may be different than those of our customers — but it’s an essential part of the development lifecycle.

What we’ve learned

I’ll share two anecdotes about our experience of dogfooding Deploy Engine.

When we first tried to onboard Deploy Engine onto the Borealis platform, we hit a strange bug that we’d never seen in testing or development. I soon discovered a bug that’s a little embarrassing, a little funny, and fortunately won’t take long to describe.

Our build uses Kustomize to generate manifests for five Kubernetes resources. When we tried using Borealis to deploy for the first time, only two of the five resources were deployed. We had unit tests that ensured that this case would work, so I was mystified. After some digging, I found the answer.

Our API accepts both JSON-encoded and YAML-encoded Kubernetes manifests. We check the first 512 bytes of a manifest: if those bytes look like JSON, we interpret as JSON; if not, we interpret the manifest as YAML. Unfortunately, in the case where we decided the manifest is encoded as YAML, we only used the first 512 bytes – we threw the rest away! Oops.

Our unit tests checked that we could deploy multiple manifests, but we used unrealistically small manifests — smaller than 512 bytes. This is a bug I’m glad a customer never had to experience, and is a bug we likely wouldn’t have caught without dogfooding.

My second anecdote is also brief. Borealis has a UI that lists all recent deployments; Deploy Engine determines how those deployments are sorted. Working with the product team, we defined the sort based on our experience and what we thought would make sense to customers.

Once we started using that UI as users, we realized the sort didn’t make sense at all — it bothered us, and we decided to change it. That very same day, one of our first users asked us to update the sort on the deployments page; their suggested change was the exact change we had just implemented. We were able to go back to the customer and tell them that their fix would be ready by the end of the day.

Today all the services used by Borealis are deployed using Borealis. Additional teams throughout the company have also started using it as they’ve seen it improve. Teams across Armory are giving us feedback. Some of it is tough to hear, but we’ll catch bugs and improve the product in ways that our customers will love. I hope to give you an update on Borealis, our progress, and our dogfooding journey in a few months.

If you would like to try out Borealis, we are still accepting design partners, and I’d love to hear your feedback on it as well.

Thanks!

Dan

The post Dogfooding At Armory appeared first on Armory.

Reducing Downtime Risk with Blue-Green Deployments

Dan Peach — Tue, 15 Mar 2022 15:35:39 +0000

Many of the world’s most common and trusted applications- for both personal or business use – are now delivered as SaaS applications, whether on mobile, desktop, or other devices. End users assume these SaaS applications – whether paid for via subscription or invoked free on demand – to be available 24/7/365. Users trust application builders to provide a dependable and uninterrupted service experience. When software isn’t available, customers get frustrated and businesses’ reputations suffer.

Leading content platform and application software companies are increasingly using progressive software deployment strategies such as ‘blue-green’ to reduce the risk of downtime, enhance customer experience, and build brand loyalty. However, in many companies, there still exists the misconception that initiatives to improve quality and security translate into increased investment. Therefore, a business case for making the technology investments required to significantly reduce the mean time between service failures to avoid downtime will likely be required.

The Costs of Downtime

According to a 2019 report from Statista, the average cost of downtime can range from $100,000 to $5,000,000 per hour.

Content customers and software users are working all hours, locally and around the world. Therefore, there is no ‘good’ time to have an outage. They expect and/or require the applications to be available all the time.

A recent Accenture study found poor service quality is the top reason for customer churn. In addition to compromising customer experience, jeopardizing renewals, expansions and upsells, excessive downtime can trigger direct costs if service-level agreements (SLAs) are in place for operational performance. In addition, if a service is delivered to regulated industries like banking, insurance, and stock trading, outages may violate compliance requirements. Said violations can result in fines, penalties, or increased scrutiny from auditors and regulators. Resultant scrutiny is very expensive in terms of time, consuming resources that could be working on revenue generating activities instead.

There is also a sizable intangible cost to organizations that face continuous downtown incidents. Service breakdowns impact employee productivity, morale, and job satisfaction. This often results in higher employee turnover.

Implementing practices to achieve service uptime performance consistent with customer expectations is critical to sustained success. Blue Green deployments are a technique increasingly viewed as an operational best-practice for deploying applications reliably into production and ensuring minimal downtime.

Blue-Green Deployment

During a blue-green deployment, the system shifts traffic routing from the currently-deployed version to a new application version. This practice requires two concurrently-running application instances:

A current, or “blue,” version of the application
A new, or “green,” version of the application

Containerization and cloud-native development augment the blue-green deployment model: Kubernetes provides a cost-effective, flexible, and automated process of generating similar application instances, and label-based traffic routing provides the necessary foundation to switch between different application versions quickly.

I work on Armory’s new SaaS-based deployment offering, Project Borealis. We just added blue-green support. In Borealis we deploy the new version, then expose it to a preview or “dark” URL that the user can leverage for any needed validation. Once validated, they can issue an approval, and the new version will become live. The old version is kept around for instantaneous rollbacks until a final approval is issued, or a given amount of time has elapsed.

Application Updates

Let’s say a company has a running application with version V1 – this is the “blue” instance. When it’s time to deploy the next application version, V2, the company deploys without exposing this new application instance to production traffic – this is the “green” instance. Instead of receiving production traffic, it is exposed to a dark URL that can be used for manual or programmatic validation.

The process of deploying and configuring applications can be elaborate. Typically, teams organize the process using steps in runbooks. Execution times vary from a few minutes to a few hours.

The team tests each step without interrupting the blue application, eliminating any downtime. The blue application continues to serve customers until the green application is ready to go.

Next, a QA pipeline runs automated tests to validate the green version. The team can triage any issues without the stress of a ticking clock. Post validation, the DevOps team or an automated tool configures the load balancers to route new requests to the green version. Once all traffic has been redirected to the green application, the blue application can be repurposed, torn down, or kept alive for a quick rollback.

A blue-green deployment in Project Borealis

This process can readily be applied to stateful applications as well. However, if your application has a database, there is additional thinking to be done about how to handle roll forward and backward compatibility.

Application Rollbacks

Blue-green deployments significantly reduce the costs associated with a broken release. If a defect is detected during or after the deployment, the DevOps team can configure the load balancer to direct traffic to the still-running blue environment, as convenient as flipping a switch. Even if your application takes significant time to start, rollback is instant since the old version is still running.

Blue-green deployments are more complicated than downtime deployments. Companies must support these environments with release automation, testing, infrastructure provisioning, configuration management, and the associated CI/CD practices. Because the deployment logic is also significantly more complicated, such strategies are often impractical to adopt without leveraging a production-ready deployment solution such as Project Borealis.

The development teams must also invest the time to make these practices work for all teams that must collaborate to successfully ship software. Often, this requires some tooling standardization. While a custom-made continuous delivery process may work for some teams, it can be challenging to apply it to all teams in an enterprise. It can also be challenging to support and maintain an extensive, homegrown solution. For example, a new engineer may not find it easy to figure out how to use all the tools available for deployment, even with simple changes. Or if a connected tool or service is upgraded, it may require troubleshooting or code rewrites before it can work.

Armory Delivers Continuous Delivery

Armory provides enterprise-grade cloud-native continuous delivery, the next evolution of the open-source Spinnaker platform. With Armory, teams can accelerate the adoption of a blue-green model using:

Pipeline building based on Spinnaker
Role-based access control
Deployment compliance checks using policies
Infrastructure provisioning using Terraform
GitOps, secret management, and more

These features enable collaboration between the developer, QA, DevOps, and SecOps teams for building end-to-end deployment pipelines. New deployment pipelines can be executed in different phases like code tagging, application release, tests, security checks, deployment, and smoke tests. This results in a green environment being set up without all the usual manual work.

Armory provides a battle-tested, flexible pipeline management system with integrated CI tools, cloud providers, security policies, and third-party libraries. Pipelines can be fully automated or user interactions can be set up representing complex and customized delivery workflows to control the pipeline. Armory enables the creation of pipelines with stages to deliver software in support of any process or policy an organization requires. Teams can go from fully automated to manual checks at each gate.

In addition to Spinnaker, Armory recently announced a new intelligent deployment-as-a-service offering, Project Borealis, that supports blue-green strategies so that developers can focus on building great code rather than deploying it. My team recently added blue-green deployments to this SaaS offering and is helping our design partners to leverage it. A demo of blue-green in Project Borealis is available. If you’d like to try it out, spots are still available for additional design partners.

Next Steps

The blue-green deployment model can eliminate the risks and toil of application release. To reap the model’s benefits, different enterprise teams must collaborate closely and have a high level of readiness.

Armory provides a foundation for safe, reliable deployments using release and deployment pipelines with compliance checks. Its policy engine and external integrations reduce the deployment model’s overall complexity, enabling development teams to operate at peak efficiency.

Are you ready to reduce your downtime and keep your application users happy while saving deployment time and effort? Explore armory.io to learn how customers use the product to solve these deployment challenges and contact us for a demo and whitepaper to see how Armory can help you. If you’d like to try out Project Borealis, our new intelligent deployment as-a-service solution that makes advanced deployment strategies simple to adopt, you can sign up to learn more and become a design partner.

The post Reducing Downtime Risk with Blue-Green Deployments appeared first on Armory.