DEV Community: Jason Barry

The benefits of on-demand staging environments

Jason Barry — Mon, 19 Oct 2020 18:28:55 +0000

TL;DR: Test features in isolation. Test releases as a whole.

You know the drill: it’s the night before a release, and your dev team is scrambling to address last-minute feedback before deploying to production. Most of the feedback surrounds tiny changes that are trivial to fix -- but some of the feedback requires larger, substantial change. These long feedback loops reveal a more fundamental issue of communication breakdown. This should have been discussed earlier. Now you’re left wondering if it’s worth delaying the release to help ensure the stability of the build.

Sound familiar? I experienced this first-hand while working at a small (~6 engineers, ~2 designers, 1 PM) B2B SaaS startup. The product team would give feedback about the latest build on staging with only a day or two left in the sprint, causing a last-minute rush by the dev team to implement those changes. Corners were cut to meet deadlines, and hastily-written workarounds were rarely refactored, adding to the backlog of technical debt as the next sprint was already underway.

If you experience these symptoms during the development of your product, then chances are that your team doesn’t have the right tools at its disposal. It turned out that the product team left feedback late because they didn’t have a stable URL where they could verify UI/UX until changes were on our staging server.

We needed a way to get stakeholder feedback earlier in the development cycle -- well before features landed on staging.

What is a staging environment?

A staging environment is a replica of your production environment. There shouldn't be any significant differences between your staging environment and your live website, with one notable exception: the staging environment is private.

Think of staging as the ultimate quality assurance check, where you can make sure everything works in as close to launch conditions as possible. You'll catch bugs, avoid costly downtime, and give your developers the confidence to take risks.

How do you deploy to a staging environment?

The traditional deployment process to a single staging environment usually has four steps:

The developer builds and tests in a local environment, running on their machine.
Once they are satisfied with their work, the developer pushes their changes to a branch on the git repo, where they open a pull request. Continuous Integration runs automated tests — like linting, type checking, and unit tests — to catch any issues. If issues are found, the developer goes back to step
Once Continuous Integration passes, the developer's peers conduct a code review. This is necessary to gain consensus among your team that the proposed changes are greenlighted to be merged into the codebase. Not only should reviewers look at the code, but they should run the code to confirm that the code operates as intended. The developer goes back to step 1 to address any feedback left by reviewers.
Once the changes are deemed copacetic by automated tests and peer reviewers, the developer's branch is ready to be merged into the main branch. If Continuous Delivery is enabled, then merging to the main branch kicks off an automated task to deploy the changes in the pull request to your company's internal staging server.

Depending on how your DevOps team configures your git repo's webhooks, merging to the main branch may trigger additional steps. If your org follows the practices of Continuous Deployment, merging to master could roll the latest changes out to production automatically (hopefully behind a feature flag). Every DevOps team is different — check with yours to see which practice your company follows.

Why multiple staging environments instead of only one?

Instead of being constrained by all upcoming changes landing on a single server, you could have temporary, ephemeral environments that exist solely for the purposes of hosting the build of a given pull request. This is what we mean when we say that you should test features in isolation. Giving every branch in your git repo a dedicated environment is very freeing for both your developers and reviewers.

What you might not is realize that this comes with more benefits than just testing features.

Code review won’t block gathering product feedback

With a single staging server, your code won't be deployed until code review passes, and you won't have a link to share with other people on your team. If you want to double-check with a UI/UX designer that the animation you made is up to snuff, too bad — you're gated upon code review passing before you can Slack over a link.

If any stages above fail (red), you have to go back to square one. This can be costly if you’ve already performed code review, a time-consuming operation.

But UI/UX designers don't care about the quality of the code — so why should code review be a bottleneck for gathering UX feedback? Ideally, you'd be able to get their opinion before asking your dev peers to give your code a once-over. That way, you'll eliminate an extra speed bump when shipping your feature.

Ephemeral staging servers that spin up on demand allow for product feedback and code review to occur before (or at the same time) as code review. If there’s something that needs to be addressed, going back to square one isn’t very detrimental, since not much time has passed. You eliminate a speed bump by batching product feedback with code review, rather than having one be a dependency of the other.

Prevent an integration warzone

Sure, your new feature that you've worked hard on works now — but since a single staging environment is the landing pad for your entire engineering team's upcoming changes, a developer on your team may inadvertently push code that breaks what you wrote. Even now that you have an internal link you can share with a non-technical stakeholder, there are no guarantees that the link will continue to work, since it is actively being integrated with!

While a single staging environment is useful for testing a release as a whole, features should be tested in isolation. This means that an environment's purpose should be dedicated to testing the functionality of a single feature, rather than a collection of many features. Save testing the collection of features for your QA server, after you’ve already tested each feature individually.

Lower the barrier for running peers' code

Developers face a large amount of context switching when they are asked to review someone else's code. The process is unnecessarily interruptive, and looks something like this:

Stop what you’re doing
Stash any unstaged changes
Fetch from origin
Checkout their branch
Install dependencies (and potentially restart your development server)
➡️ Actually verify the new feature and provide feedback
Checkout your original branch
Pop your git stash
Reinstall dependencies (and potentially restart your development server)

With on-demand staging environments, on the other hand, the running branch is waiting for you to verify and provide feedback. Developers can jump into the deployment preview straight from the pull request without needing to checkout files, install dependencies, restart their development server, or change anything about their local environment.

Avoid redundant code reviews

The point of getting feedback is to make more implementation changes that address the issues raised in the feedback session. The problem is that this will require another code review! Even though the proposed changes the second time around are (hopefully) small, your reviewers will experience déjà vu because they are now going over the same lines of code for the same feature implementation.

With multiple staging environments at your disposal, you could spin up a preview environment for your designers to critique while your pull request is still open. You wouldn’t need to request a code review from your developer peers until your designer gives the thumbs up ahead of time.

More opportunities for testing

On-demand staging environments can be configured to be accessible behind a shared URL, which unlocks a wide range of testing and quality assurance opportunities.

Cross-browser testing: Ensure your product or website works perfectly on a wide range of browsers and devices.
Accessibility testing: Run accessibility tests to ensure your content is accessible to individuals with disabilities.
User testing: Create a pool of test users or use a crowd-testing platform to discover usability issues before you launch.

Avoid environment drift

Single staging environments are often configured once and left to decay, becoming a DevOps chore. This leads to drift between the staging and production environments, as similarities between the two decrease. This increases the risk of bugs appearing in either staging or production, which makes bugs difficult to reproduce.

Examples of these drifts include:

Environment and configuration files getting modified by unsuccessful staging deploys, leading to extraneous or incorrectly named settings.
Time-sensitive security vulnerability updates being applied to production in a rush, leaving staging several versions behind.

Whereas your production environment is under constant care and attention from the DevOps team, an infrequently used staging environment is lower on the priority list.
This can lead to an unreliable staging environment, which blocks your development team from launching new features.

More stakeholder engagement

In the past, stakeholders didn’t get to use the finished product until it was live. Instead, they got a brief scripted demo at the end of the sprint.

By giving stakeholders access to your staging environment, they can help with the quality assurance process and ensure that the development work completed meets their expectations. This saves time and prevents mistakes from accidentally shipping.

Increased developer productivity

With on-demand staging environments, developers can create as many staging environments as they require without asking DevOps to configure a new environment. Developers love it because they have the capability to spawn new servers, and DevOps loves it because they don’t need to field manual tasks of spinning up a server only for it to be taken down shortly afterwards.

Save your company time and money

Single staging environments are typically available 24/7 on a dedicated server or virtual machine. This is a waste of resources that could otherwise be used for production.

Furthermore, if you have several development teams, the single staging environment can become a bottleneck. Teams must wait for a staging environment to become available, waiting for other teams' quality assurance processes to complete. This could waste days of developer productivity and encourage teams to skip straight to production, losing all the benefits of an effective staging environment.

Each on-demand staging environment, on the other hand, is only online for the duration of the pull request. When the PR gets merged or closed, the environment gracefully shuts down, so you only pay for what you use.

Single vs. on-demand staging environments

Instead of the single staging environment decaying while it waits for someone to use it, an on-demand staging environment is created dynamically, triggered by a CI/CD pipeline.

This means developer teams can have access to staging whenever they need it, simply by opening a pull request. As you’d imagine, this boosts productivity and encourages developers to use staging environments, reducing the chances of costly production bugs. Once a developer is done with the staging environment, the staging environment is destroyed, along with any configuration, environment, or installation inconsistency.

	Single staging environment	On-demand staging environments
Developer self-serve	❌	✅
Low maintenance	❌	✅
Prevents redundant code reviews	❌	✅
Can test features in isolation	❌	✅
Can link to a build while pull request is still open	❌	✅
Number of environments	1	Unlimited
Cost	Pre-allocated	Allocated on demand and reclaimed automatically

Having said that, you should still have a staging server where you can do QA before rolling to production. This is where you should do a final spot-check to ensure that everything is working as expected before going live. The difference is that the purpose of this server is to test the release as a whole, rather than each feature on its own. How do the new features operate together? If there’s a bug found, which pull request and which lines of code introduced the bug? These questions are much easier to answer when features are tested while the pull request is still open.

Shorten cross-functional feedback loops with on-demand staging environments

As a front-end developer, being able to dynamically create a dedicated environment for an unlimited number of branches is a very powerful feeling. I can control my own network of machines without needing to ask for cycles from DevOps! The asynchronous nature of asking for feedback on a fleet of ephemeral environments is liberating -- I know that each deployment will be waiting for my reviewer exactly how I left it. And while I review my peers’ code, I can visually review the build alongside it without having to check out their branch locally.

It’s not only helpful to the developers on my team, but also helpful to the non-technical stakeholders too -- they get a lens into the engineering process, and can voice their opinions while the implementation details are still fresh in the developer’s mind.

If your organization is struggling with shipping features (on time) that meets everyone’s acceptance criteria, the solution may lie in tooling. Notice where and when the speed bumps occur -- handoff, development, code review, etc -- to help identify bottlenecks. Ask yourself whether your single staging server is a limiting factor during your team’s development processes.

Learn more

The easiest way to set up on-demand frontend staging environments is with FeaturePeek. FeaturePeek creates a deployment preview of your frontend every time you open a pull request. A drawer overlay is added on top that makes it easy for your reviewers to leave feedback by allowing them to leave comments, take screenshots with annotations, capture screen recordings, create tickets on popular bug-tracking platforms, and more.

FeaturePeek works with static sites and sites containerized with Docker. Best of all, you don't have to migrate to a new cloud ecosystem. Whether your company's website is hosted on AWS, Google Cloud, Heroku, Digital Ocean, whatever — since FeaturePeek is infrastructure-agnostic, you can use FeaturePeek no matter who your hosting provider is.

Try the 2-week free trial to see how FeaturePeek can improve your team's development productivity.

Where should your team review your front-end changes?

Jason Barry — Tue, 26 Nov 2019 12:12:03 +0000

You've just finished developing a new feature for your web app, and you're ready to get feedback from the rest of your team. You're pretty sure it's what your designer intended in their prototypes, although you had to take a few liberties – you did some guesswork when mockups for tablet breakpoints weren't supplied, and you had to come up with a workaround or two due to a technical constraint. You want to get some feedback on your work before rolling out the feature to production.

Engineers can leave feedback on GitHub or similar during code review, but this part of the product development lifecycle isn't accessible to those in non-code-savvy roles. What's the best way to open up your work to the rest of your team?

Let's explore some options. Ideally, we'd want a solution that:

Reduces the duration of feedback loops between cross-functional teams. You should be able to send a link for someone to review as soon as you are ready, not as soon as your internal infrastructure allows you to be ready.
Circumvents performing code review on the same feature twice. You shouldn't depend on your developer peers to finish reviewing your code before you're able to share your work with someone else on your team. If you receive feedback to make changes, your peers will have to do code review again for what is essentially the same feature.
Requires as little maintenance and code changes as possible. You shouldn't have to spend time building infrastructure to support a basic review process. If possible, you'd like to lean on an existing tool that has already solved this problem.

In person (or screensharing) 👎

While rolling up to your colleague's desk (or virtual desk if your team is remote) can clear up quick clarifying questions, it poses problems when trying to gather detailed critique. Common pitfalls include:

Scheduling common availability is difficult. What if your reviewer is in a meeting? Or working heads-down on an upcoming deliverable? You both need to have uninterruptible availability to get their feedback synchronously. Trying to get two people's schedule to sync up is too time-intensive and breaks their flow, especially in fast-moving startups.
You miss out widening your surface area of test cases. When you show someone a feature that you've coded up on your own machine, you are forgoing the opportunity to apply variance to it. How does it respond to production data? Does it require a font that is only installed locally? How does it behave when you're logged in under a different user account? You've tested responsiveness in Chrome DevTools — how about on a real device?
You become responsible for taking notes of reviewer feedback. You can jot reviewer feedback down while in person, or create a quick todo list, but ad-hoc review that exists outside of formal tools is undocumented and bound to slip through the cracks.

The productivity benefits of asynchronous communication are clear. You should avoid collecting feedback synchronously whenever possible.

Local dev server tunnel 👎

How about opening a tunnel to your local development server's running port? You could use ngrok to expose a port so that someone outside of your LAN could access your local dev server.

There are lots of problems with this approach:

Your machine must be powered on and online. As soon as you close your laptop, the bridge to your dev server is terminated.
You can't checkout another branch while the tunnel is live. If you wanted to show someone something on a specific branch, you'd have to stay on that branch. This means that you are blocked from working on anything else.
Your reviewers aren't getting a production build. Instead, what they're seeing is the development build, which isn't as performant as what eventually gets pushed to production.

Local build during code review 👎

As a developer, you're familiar with conducting code review when your coworkers open pull requests. The code may look correct... but does it work? A good way to test if it works is to run the code yourself. However, to do that on your own machine requires you to jump through a few hoops. The following flow should look all too familiar:

Stop what you're doing
Stash any unstaged changes
Fetch from origin
Checkout their branch
Install dependencies (and potentially restart your development server)
➡️ Actually verify the new feature and provide feedback
Checkout your original branch
Pop your git stash
Reinstall dependencies (and potentially restart your development server)

You can see the trouble with this — there are many steps you must accomplish before and after verifying that make the task a tedious effort. Most of the cognitive load of verifying someone else's feature on your machine comes from stopping what you're doing.

Furthermore, unless your entire team is fluent with git, this option is only available to developers. What about the non-developers on your team?

Staging or QA server 👎

Your continuous integration pipeline builds the latest changes from a protected branch in your git repo, and deploys the build to an internal server. It seems like a safe place to review new features, because it's a close representation of what production will look like when it eventually goes out the door. However, there are some problems with using this traditional method:

Feedback from reviewers causes more code review. When you ask someone for feedback on your work, they will most likely respond with at least some critique, which will require code changes to fix. Your developer peer must be asking themselves, "Didn't I already review this feature?" You should aim to have every new feature only go through code review once to avoid bottlenecks in your workflow.
Other developers could break your changes. The problem with this approach is that other developers on your team could inadvertently merge a change of theirs that breaks the functionality of your new feature. By passing around a link to an environment that gets fed from top-of-tree, you aren't guaranteeing that your feature is isolated. By the time your reviewer tests your new feature, a part of it (or the whole thing!) could not function as intended, even though it worked when it was on its own branch. Not only does this waste your reviewer's time, but it causes them to think that you incompetently didn't perform a basic test of your own work. A better option would be to guarantee that the code you push is the code your reviewer tests — this is what we mean when we say you should test features in isolation. A staging / QA environment is great for testing a whole release, not individual features.

Feature environments 👍

The best place to review your feature is on the feature branch itself. Ideally, as soon as you push your branch to your repo, your CI builds an image that is deployed to a dedicated host. Feature environments are automatic and ephemeral, orchestrated via webhooks from your repo.

Asynchronous. No scheduling calendars, syncing up for in-person walkthroughs, or tunneling to your local dev server.
Instant. No need to wait for unit tests or code review to pass before you can get feedback from designers, marketers, or other stakeholders on your team. No need to checkout branches and install dependencies just to verify your coworker's code. Shorten feedback loops with your reviewers so that you don't need to ask your developer peers to review the same feature multiple times.
Accurate. You can find more issues ahead of time by running your code on a machine that isn't your local dev box. Rest assured that what you see is what you will deploy, because your build comes from your CI server.
Isolated. Since the feature environment spins up from your branch, no one else will accidentally break your feature. Reviewers won't be upset that your feature clearly doesn't work.

Imagine every pull request in your repo having its own live, up-to-date environment. Links to the environment are posted as a comment in your pull request, and to a Slack channel. As soon as your feature gets merged, the environment gets torn down.

Feature environments as a service 👍👍

Developing the infrastructure in-house to support feature environments yourself takes developer time and ongoing maintenance. You should instead focus on building the product(s) that are core to your business.

FeaturePeek is a feature-environments-as-a-service platform that integrates with your GitHub organization. In just a few clicks, you can start building on-demand feature environments for every front-end pull request. On top of that, FeaturePeek overlays feedback and collaboration tools over every feature environment, so your reviewers can leave comments, file bugs, attach screen recordings, and more — all from the context of the environment itself.

We launched to solve this problem that we've seen so many small- to medium-sized companies encounter. Start a free trial today and let us know what you think.