What are environments, anyway?
Most software companies have their product deployed to a few different environments, which have names like local, dev, test, stage, and production. On my first day as a developer I was confused about why there were so many steps between a developer's computer and the app that customers used. It seemed inefficient and bureaucratic. I've come to appreciate those environments, though, and these days I break out in an ice-cold sweat if someone talks about deploying straight to production. If these terms are unfamiliar to you, or if you also wonder why applications need so many environments, stick around: I'll be giving you the low-down on each of them.
But first of all, what is an environment? It's a computer (real or virtual) which runs your software and which you control. If you use a Continous Integration tool (like VSTS, Travis, GitLab, or my personal favorite, TeamCity), you probably have access to a tidy list of environments alongside buttons that build the latest version of the project and deploy it to them. You may also have tasks you can run on each environment, like unit or end-to-end tests. Your own computer is an environment, referred to as local, and the server that provides customer access to your product is the production environment.
Let's talk about the purpose of these environments and best practices for each. My assumptions are that you're using Git and developing an app that runs on the web, but the same principles apply to other version control systems and platforms as well.
A tour of standard environments
Local
This is where all development work happens. Nobody but you has access to this environment, and thank goodness, because if you're anything like me it's usually broken. You use the Local environment to iterate on your own tasks and see the results of your code changes.
Best practices:
- Commit frequently.
- Merge from your team's master branch regularly.
- Run all of the project's automated tests before opening a pull request. You have tests, right?
- Avoid merging something back to master if you know it's broken. Sometimes it might make sense to do so, but be sure to notify your team ahead of time.
Dev
This is the cutting edge of development work. All the latest code lives here. It's a good place for developers to test new features against mocked data and verify that their code works well with everyone else's.
Best practices:
- Nobody outside of the product development and engineering department should have access to Dev. Not even QA.
- Dev should never be used for demoing the product. It's half-baked. It's totally acceptable for it to be broken at any moment; developers try not to break it, but that's only for the sake of their own productivity.
- Anyone who complains about Dev being broken should expect to be pelted with celery sticks and Cheez-Its.
- Dev should auto-deploy from the master branch every time there are new commits.
Test
This is the company's first chance to check for any problems that their developers and automated tests didn't catch.
Best practices:
- Nothing should be deployed to Test unless it has passed all of the project's automated tests and a manual sanity check.
- The company should focus its QA resources on the Test environment. A few days before every planned release, people who are intimately familiar with the product should begin regression testing there. From the moment testing begins until the code goes to Stage, the only new code deployed to Test should be bugfixes.
- When bugs are found on Test, they should be promptly reported to a project manager. Anyone and everyone can complain when Test is broken, but Test should not be used for demoing the product.
Stage
Stage is meant to simulate Production in as many ways as possible. It's one last line of defense against releasing bugs. Stage can also be used to demo new features and to train high-touch clients on upcoming changes to the application.
Best practices:
- Deploying an app to Stage should indicate that it has been approved by the company's QA team and it is believed to be bug-free.
- If your app works a certain way on Stage, you should be 99.9% certain it will work the same way on Production. One possible way to enforce this is to run Stage and Production side-by-side on the same infrastructure and switch from one to the other when it's time to release. For example, Microsoft Azure lets you deploy apps to "app slots" and swap any two slots via the command line. Similarly, Heroku lets you move apps from one environment to another using the pipeline interface.
- If possible, the Stage database should be overwritten with data from Production on a regular basis.
- Bugs on Stage should be prioritized similarly to Production bugs.
- Most of the time, Stage and Production should have exactly the same code. If a release is scheduled for the next few days, Stage may be ahead of Production.
Production (live)
Production is the only customer-facing environment. It's the reason why every other environment exists. As a company grows, Production tends to become more and more sacred. If Production is broken, customers will leave and the company's reputation for reliability will suffer. Anything that happens in Production is liable to end up in the news, for better or for worse.
Best practices:
- Don't break Production.
- You're gonna break Production. When you do, proactively notify your customers, be honest about the situation, fix it as soon as you can, and be prepared to make restitution to anyone who was affected.
- Don't skip QA on Production bugfixes. Sure, if the situation demands it, you can expedite or trim down your QA process. But do your best to make sure the fix doesn't break something elsewhere.
Final thoughts
These are standard environments for a large company. If your product hasn't gone public yet and development speed is more urgent than application stability, you probably only need a couple of these. As your user base grows and becomes less fault-tolerant, you can add environments to match the structure of your organization and the resources at your disposal. Each environment is another place where the right pair of eyes could prevent a bug from reaching Production and affecting users.
Top comments (10)
Excellent break down of environment usage. I'd like to note that different organizations have different name for each stage/s, however, the reasoning is still valid.
Some example
DEV -> INT -> SIT -> UAT -> PRD
LCL -> UAT -> PRD
LCL -> Staging -> Blue || Green PRD
Do not get caught up on the names, enforce separation of concern and responsibility.
Good breakdown, thanks for writing this.
Our team uses feature branches instead of committing to master, so the idea of "Dev should auto-deploy from the master branch every time there are new commits" is odd to me. But like was mentioned in another comment, applying the concepts in this article just means that we may have multiple dev environments: one for each feature branch.
About the Test and Stage environments...
Why is Stage the last line against bugs? The QA team works with the test environment. How will bugs be caught in Stage?
Similarly, only Stage was mentioned to have a Production-like database backing it. However, in my experience, many unexpected defects come from how the code interacts with odd production data. So the production-like data of Stage is crucial for catching bugs.
I'm a big believer in feature branches, though at some point they have to be merged into a master or dev branch, right?
I glossed over this, but although the QA team focuses on the Test environment, they should definitely spend some time vetting Stage before it goes to Prod. I agree that Production data is often wilder and weirder than test data, so testing against it can (depending on the product) be crucial to catching bugs. I worked at one place where every database (all the way down to developers' own local databases) would get refreshed with data from Production a few times per month. That introduced some overhead, but it was often helpful.
I agree with you, my organization recently had a problem that I think it could have been solved with a release branch, the problem now is that we need to integrate a new infrastructure and at leas in my company it is a big deal.
It's odd to me too. I'm more used to master representing production, and the test environments tracking a release branch. But the principle of separation is basically the same.
...Unless you've in a marketing agency where the idea of a "planned release" goes straight out the window.
Great post!
I am new to these concepts and I like to know some things!
Each enviroment (dev, test, stage) is a different computer (with its own database, its own code)?
Can I use a docker container for each of these envs?
If I use docker, can I use differents env in ONE docker container? Or I have to separate in several containers?
Sorry if the question are dumbs!
Thank you very much!
Hi Germán! Great questions.
Not necessarily. More than one environment could coexist on the same computer, but they'd need some separation (code in different folders, served on different ports if it's a web app).
Sure, there's no reason why you couldn't.
I don't know enough about docker to answer that one. I think it would difficult to configure like that, since you want the environments to act independently of each other.
This is all very true... unless you're embracing continuous deployment, in which case deploying straight to production is a core feature and a huge benefit.
Interesting, I've never been on a team that practiced continuous deployment. A few questions: Where does QA fit in? What about automated tests? How do you maintain confidence in your changes? How do you handle large, cross-team feature development? In your experience, does it work as well as as a multi-stage release cycle?