Holding a loaded rifle without knowing it

Two stories:

1. When I've served in the army a not-so-funny story happened to one of my friends:
   He played with his M16 because he was bored (It was the year 2000. no smartphones...). And in some weird way he ended up having his gun loaded without him knowing it.
   So how do I know it?
   Because an officer checked his gun... (lucky for us all).

2. When working with SQL SERVER a friend of mine (it's always friends. never me...) worked on his local copy of the database.
   Or that's what he thought he was doing.
   Actually he had two tabs opened inside SSMS:
   One representing his local enviroment.
   One representing our production.
   Can you guess what happened?

You are probably right:
The production database was deleted.

Well let's just FF 3 hours later when all the managers came back from home because nothing worked. And although it was on SQL AZURE which has great backup, he somehow managed to do all the steps which avoids the normal way of restoring to point in time.

We were so helpless that we tried to restore the data from the cache into the DB!
Finally the guys from Azure came to our help and gave us what we needed.

What did I learn from this?
Seems like less then I thought.
Because yesterday it almost happened to me again!

It's so easy to get into these places.

The only way is DISCIPLINE.No fancy tool will save you if you don't use it.

You set rules, and you understand deeply why they exist (In some proffessions they say: "Regulations are written in blood").

Try not to hold a loaded rifle, and if you must hold it- at least be aware of it.

Comments

[Eric Bishard]

Umm, that's why you set up permissions on production. And although I'm a developer... never give developers production rights.

[Philip Hallstrom]

In the world of devops, developers typically do have production rights. Still, it shouldn't be that easy to access them in the way OP mentions.

[Patrick Minton]

there's a difference between a developer having access to a production environment account that has production rights vs. giving the normal developer's normal account that he/she uses every day production rights.

I don't use sql server but there's definitely something fishy about the above situation. And this statement is straight up false:

The only way is DISCIPLINE.

Relying on developers to "be careful" is how that database got deleted in the first place. You are supposed to build systems with the expectation that humans will screw up, no matter how disciplined your team is.

[Theofanis Despoudis]

Exactly. Security 101.

[Denis Viklov]

Lel, would you like to say that you even don't fix bugs right on production servers via FTP?

[Hagai Luger]

Well, that depends.
If it's a client side issue - I prefer clicking F12 and changing through the dev console :-)

[Tim]

This is why for my Azure databases and others I've setup unique and separate admin credentials so there is only one user I can connect with for each environment. I also only use the Registered Servers option in Sql Server Management Studio and have my production servers color coded in red so it's easier to tell when I'm in the danger zone. I've done the same with my virtual machines where the desktop background is red for the production server. But I agree it takes discipline to not only set this up but to use it and not take shortcuts occasionally.

Some of the most important lessons we learn are also the more painful ones. :)

[Thomas H Jones II]

By no means idiot-proof (since they'll always build a better idiot), but, to help me avoid being the source of tragedy:

• Use different colors for dev xterms and prod xterms
• Use per-TTY shell-history files (so no accidents happen via lazy shell-history executions)
• Use different cloud-accounts to host dev and production services
  • If using web UI, only ever open dev and prod in specific browsers and/or profiles/sessions
  • If using CLI, make sure that each command is executed with a specific profile-name for each account and role
• If your CSP supports it, enable termination protection on prod resources
• Ensure use of "obvious" naming-patterns for dev and prod resources - both at the cloud-layer and within resources (instance hostnames; DB intance-names; etc.)

[Philip Hallstrom]

While it doesn't cover all use cases one trick I've learned over the years is to setup different color schemes and prompts between your environments.

For example, all my local terminal windows are a dark theme. All of my production terminals are a light theme. This visually helps remind me I'm somewhere different.

The terminal prompts also change to include the environment itself. They are prefixed with EDGE, STAGING, and PRODUCTION. Colors change too. EDGE is in green, STAGING is in yellow, PRODUCTION is in red. You can do the same thing for irb/pry/rails-console prompts.

Even the website itself. For EDGE and STAGING, throw a banner/blammy across the top in a way that it doesn't affect the layout (float it on top something). Some bar with enough color and labeling to help remind you where you are. If you don't see that, you're in PRODUCTION.

None of this would stop me from doing something stupid, but it helps keep me aware of just where I am.

[Vico]

i like the metaphor