Sometimes even your best ideas will fail to persuade. You can have evidence, examples, and enthusiasm, but still, you fall short of convincing others to give it a try. Today, I want to talk about another tool you can use; imagination1. Project yourself into the future where you are looking back at the problem after you solved it. You recount the steps you took along the way, and the problems you fixed, and then celebrate all that you've achieved.
In this article, I'll talk about the "PR/FAQ" and how you can use one to help your cause of "Going Serverless," "Strangling the Monolith" or, in this case, "Achieving Continuous Deployment." I think you'll find that the process of writing a PR/FAQ will clarify your thoughts and sharpen your arguments. Hopefully, it will help you put your idea into action.
Introducing the PR/FAQ
One of the more interesting things I learned during my time at Amazon is only tangentially related to engineering, but it has stuck with me: the PR/FAQ. Amazon didn't invent the Press Release / Frequently Asked Questions format, but they did popularize it, and they get credit for introducing it to me.
The PR/FAQ is essentially a press release from the future that describes the release of the new feature/product that you wish to begin in the present. The PR/FAQ is the starting point for discussions with leadership, engineering, marketing, and any relevant stakeholders. It presents your idea in a concise, easy-to-follow format that describes both the problems it solves and the value it brings.
While primarily utilized by Product Management, I've used the PR/FAQ to illustrate broad engineering change proposals. I like how it fosters conversation and allows those discussions to become part of the document via FAQs. I want to show one example of using a PR/FAQ that is similar to one I've created in the past.
I must note that, while this is inspired by real events and real companies, it is entirely fictitious (it's the future, after all).
The Situation
A large, publicly-traded company ("Acme") hires you to help them move from an expensive, error-prone, high-touch deployment process into something more resembling CI/CD. Your team ("Zodiac") builds multiple business-critical applications that take multiple days and multiple different team sign-offs to deploy.
Acme has a single AWS production account that all teams deploy into. The monthly spend on this account approaches $5MM/month and most of that comes from teams that are not yours. The Acme Operations team that controls the account is constantly worried about changes breaking the system. Some in leadership are interested in CI/CD but don't know how to get there. Many in engineering are interested in CI/CD but don't know how to get there. There is a QA team that is currently in charge of quality and a Product team that wants to know the minute a feature goes live.
Your goal is to take the Engineering pain away from your deploys while reducing risk for Operations. You also need to help Engineering take control of the whole software development lifecycle while supporting Product. Your PR/FAQ will include all these issues and wrap it in a happy ending that gives everyone involved a well-needed shot of confidence. Here's an example.
The Press Release (PR)
Zodiac Hits Continuous Delivery Milestone with Daily Deploys
Acme's Zodiac group adopts new practices to deliver features and fixes 10x faster
August 01, 2024. Zodiac is excited to announce improvements to its delivery pipeline that provide faster cycle times and safer deploys. This new practice "shifts left" the testing, infrastructure, and code management to allow deploys to happen in minutes instead of days. The engineering team can easily release bug fixes or add new features as a self-sufficient unit. Because of the many vertical silos of the previous process, releases were slow, error-prone, and expensive. With the new improvements, we have reduced the number of people involved in a release by 300%.
Releasing applications in 2023 was a repetitive task, with Architects manually creating/merging code branches, QA and Product manually testing both changes and existing functionality and then Operations taking action from hand-written instructions. By the time the release happened, the people who truly understood the changes (Engineering) were 7 days into their next cycle. In the event an issue was raised, they struggled to regain the context they had when they coded the change in the first place.
To give Engineering the self-sufficiency they need, the business needed risk mitigation. Working with Operations, the new per-team, per-environment AWS accounts provided both. Separate accounts provided a narrow blast radius and a clearly delineated area of responsibility for the team. The business also needed accountability, which is why the new process records every delivery in a central change log document, outlining who, what, why, and when. Finally, the adoption of a feature-flag framework allowed the team to decouple delivery from release. This gives the Product team full control of revealing the feature to customers while Engineering can focus on fast delivery.
Zodiac engineers benefit from this approach by following a few simple guidelines: write good tests, use feature flags, and put your JIRA ticket number in your commit message. That's it! The trunk-based release approach will, if all tests pass, deploy your changes to all environments. Product can then flip flags on/off as they see fit.
"This allows Engineering to own their changes without impacting what Operations is being asked to manage." said senior DevOps engineer, Josh Gordon. "The [AWS] account separation makes this possible."
"It was an eye-opener for us," said engineer Pat Wilson. "Knowing that nobody else was going to verify our code changes gave us tremendous incentive to write good tests."
If you'd like to introduce this to your team, speak to Stephen Mallory, Austen Heller, or Seth Orell to learn more about continuous delivery and how it can help your team succeed.
F.A.Q.s
What is a "self-sufficient" engineering team?
A team is self-sufficient when it is empowered to take all the actions necessary for it to create, release, and maintain software. This idea was succinctly captured by Amazon CTO Werner Vogels when he said "You build it, you run it".
What does "shift left" mean?
Shift Left is a practice intended to find and prevent defects early in the software process. These defects can be bugs, deployment issues, runtime performance, etc. In contrast, take a look at a traditional vertical separation of concerns: Engineering -> QA -> UAT -> Operations. Notice how the arrows go left to right as the new silo takes control. "Shift Left" aims to push responsibility leftward, to Engineering, for all things.
What is a "Feature Flag"?
Feature flags (or toggles) allow a team to modify system behavior without changing code. They separate the code deployment from the feature "reveal". This allows for new features to be delivered silently (or "darkly") and then turned on at an appropriate time without a separate release.
How many AWS Accounts are necessary?
The minimum "best practice" suggested by Amazon is one per team per environment.2 For example, the Zodiac team would have one AWS account dedicated to them for each environment supported. In addition, Many organizations also choose to assign a "personal" account to each engineer to encourage experimentation and exploration.
Who manages the multiple AWS Accounts?
Typically, this is an Operations area of control. Using tools like AWS Organizations and AWS Control Tower, Operations can create hierarchical account structures that match business units, create guardrail controls, and organize cost tracking. This can require a high degree of specialization that is often not found in Engineering.
Why is releasing more frequently preferred over less?
With our software-as-a-service platform (vs. desktop executable or firmware), we have the opportunity to release at any time. One of the great strengths of Agile is the ability to respond quickly. A quick release cycle means a bug fix takes less time to reach our customers. It also lowers the risk of any particular deployment. This has the additional positive side-effect of a hot context; engineers have their changes fresh in mind during deployment and can more quickly address problems should they arise. By automating the release tasks to make them fast, we also eliminate the errors that come along with a manual process.
Why was the old process error-prone?
The two primary sources of release errors were release branch management and the separation of those who make changes from those who deploy changes. In the first case, an architect would - for each product released - execute a series of manual steps to merge code both up and down from specific release branches. On the occasional merge conflict, this often required hand editing the files involved. Does the release contain the same code as the engineer intended? Probably (maybe). In the second case, the engineer's changes were interpreted by different silos of teams as it moved toward release: Development -> QA -> UAT/Product -> Operations. At each transition, more context was lost and more noise was introduced (have you ever played the "telephone" game?) Moving deployment responsibility up to Engineering - coupled with a trunk-based release pipeline - removes the common sources of release distortion.
How auditable is this system? Can't anyone just push anything?
This whole process relies heavily on both Trust and Accountability. We trust our engineers to do the right things. At the same time, we hold them accountable for the behavior of the system to our customers. Every code change is captured through our source-control process, including what was changed, who made the change, and when it happened. In addition, the engineer can enter extra information into the commit message to explain why she was making the change. One good practice is to include the JIRA ticket number here. Each release takes the commit history since the last release and records it in a release log. By automating the process, we now have a full trace of the What, When, Who, Why, and How of every deployment.
What role does QA/DevOps fit into this process?
DevOps is an ideology, not a team or a title. Every engineering team member should be seeking to implement DevOps principles. We want a team of generalists where everyone does everything. To this end, quality becomes an engineering concern. Traditionally, QA picks up the story when the engineer is "done" with it and writes tests to verify. With the new process, the engineer is responsible for the quality of the change he is making and writes his own tests. Someone from QA or DevOps is welcome to become an engineer on the team (I have examples of both to share) but we are looking to build the team with generalist engineers.
Further Reading
Robert (Munro) Monarch: PR FAQs for Product Documents
Martin Fowler: Continuous Delivery
Ian McAllister: Working Backwards Press Release Template and Example
AWS Whitepaper: Organizing workload-oriented OUs
-
There's nothing mystical about this approach; you are telling a story. A story is a powerful vehicle to concretize, to make real to someone, to express an idea. ↩
-
The optimal approach is one account PER SERVICE (or "Workload") per environment, where a single team might deploy many services. ↩
Top comments (0)