DEV Community: Chris Bui

Don't Couple Your Deployments

Chris Bui — Wed, 06 Mar 2024 14:33:00 +0000

You’ve made changes to your service and go to deploy it. Among your changes, you added a new SQS queue.

Your deployment pipeline applies the IaC changes to get you that new queue. Then, it updates your code.

Later, there’s a problem with your code, so you roll back. However, rolling back the commit will also revert your SQS queue and cause your IaC to destroy it, which you don’t want.

This is a contrived example, but I’ve seen variations of it so many times.

Another common example is Kubernetes deployments using kustomize. Typically, a directory with manifests in a repository is applied using kustomize in the pipeline.

Developers add configuration in the form of environment variables to the Deployment. A new service version is deployed and needs to be rolled back.

You want to revert the commit, but the environment variables must stay as the new version so you can’t. You can’t simply revert everything in all cases. For example, say the application’s database host changed because of an outside migration, and the old host has been torn down.

Just like with application code, don’t couple your deployments too tightly. Treat configuration as a separate deployment from application code.

Join the 80/20 DevOps Newsletter

If you're an engineering leader or developer, you should subscribe to my 80/20 DevOps Newsletter. Give me 1 minute of your day, and I'll teach you essential DevOps skills. I cover topics like Kubernetes, AWS, Infrastructure as Code, and more.

Don't Pick AWS us-east (if you can)

Chris Bui — Wed, 31 Jan 2024 01:56:27 +0000

After large AWS incidents, they post a post-mortem on this list.

As of today, there are 16 outages that AWS considers as significant.

Of these 16 outages, 8 were from the us-east regions. Four of those eight were in us-east-1.

If you were starting today and can afford it, I'd recommend using the us-west regions.

EC2 instances and everything in costs more in general in us-west. However, in the future, you might never need to engineer your workloads for multiple regions.

Summary of the AWS Lambda Service Event in Northern Virginia (US-EAST-1) Region, June 13, 2023
Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region, December 7, 2021
Summary of AWS Direct Connect Event in the Tokyo (AP-NORTHEAST-1) Region, September 2, 2021
Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region, November 25, 2020 
Summary of the Amazon EC2 and Amazon EBS Service Event in the Tokyo (AP-NORTHEAST-1) Region, August 23, 2019
Summary of the Amazon EC2 DNS Resolution Issues in the Asia Pacific (Seoul) Region (AP-NORTHEAST-2), November 22, 2018.
Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region, February 28, 2017.
Summary of the AWS Service Event in the Sydney Region, June 4, 2016.
Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region, September 20, 2015.
Summary of the Amazon EC2, Amazon EBS, and Amazon RDS Service Event in the EU West Region, August 7, 2014.
Summary of the Amazon SimpleDB Service Disruption, June 13, 2014.
Summary of the December 17th event in the South America Region (SA-EAST-1), December 17, 2013.
Summary of the December 24, 2012 Amazon ELB Service Event in the US-East Region, December 24, 2012.
Summary of the October 22, 2012 AWS Service Event in the US-East Region, October 22, 2012.
Summary of the AWS Service Event in the US East Region, June 29, 2012.
Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region, April 21, 2011

Sourced from AWS

Like what you've read?

Conways Law and Ownership

Chris Bui — Sun, 21 Jan 2024 00:43:49 +0000

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.
– Melvin E. Conway

Understanding Conway’s Law is a crucial aspect of navigating the complex landscape of modern software development, especially as teams become more specialized and distributed.

Here’s an example of Conway’s law: A company has a frontend team and a backend team, so their applications naturally have a clear boundary between the user interface and the data and business logic. This results in their application’s frontend and backend communicating over HTTP APIs. Concretely, this might mean they have a React.js application and many microservices that it talks to.

Conway’s law implies that your architecture will match your organization.

Ensuring you align your resources’ ownership with your organizational structure is important.

If you don’t get this right, the work will be inefficient and awkward.

For example, if you have an infrastructure team responsible for maintaining services’ Kubernetes deployment manifests, the Kubernetes deployment shouldn’t live within the services’ repository. It should live within a different repository that the infrastructure team owns.

Otherwise, the ownership model doesn’t align with the organization’s structure. There will be inefficiencies when the infrastructure team maintains the Kubernetes deployment manifests.

If you struggle with what team should own what at your company, align it with your company’s organizational structure. If that’s not the architecture you want, you’ll have to change the organization before changing the system’s design.

Conway’s Law can help us align our teams with our technology, making sure our work flows smoothly and our systems make sense.

Like what you've read?