DEV Community

Cover image for Create Chaos In Your Company Today, No I am Serious!

Create Chaos In Your Company Today, No I am Serious!

Joe Hobot on January 14, 2019

I am tagging this post as #productivity because I believe that is best way to describe something when you are in state of a chaos. Nothing fills y...
Collapse
 
darksmile92 profile image
Robin Kretzschmar

This is so true, we often don't take care of scenarios like what if the database is not available, if a certain validation fails, etc... and we need to make sure the user sees a reasonable message to let them know whats going on.

I think productivity fits to this topic :)

Collapse
 
damnjan profile image
Damnjan Jovanovic

A good reminder that we need to be ready/aware for/of possible chaos. I res about that recently in DevOps handbook. It really makes sense if you have many microservices running.

Collapse
 
joehobot profile image
Joe Hobot

Hvala.

See recently I got Kafka/Zookeeper on K8s running and things went smooth. Now I played that Chaos and went ahead and deleted random zk kf pods. Things were good until I found out about wait times and rebalancing etc. Now imagine introducing a lag of 1-2min in prod with say 500 pods. Disaster. :)

Collapse
 
adammckenna profile image
Adam McKenna

Hey there,

Some good points in this article, no doubt. I like the more abstract concept of intentionally putting your system in less-than-ideal circumstances, observing the outcomes and acting accordingly. It could help to deliver a really robust system.

I do think, though, that this post is very heavily aimed towards people who work in CI/CD or DevOps. As a web developer, there are many chaotic situations that I could intentionally trigger, but would be unable to resolve. Sometimes, the solution lies with a third party or is restricted by technology.

I think we need to take these principles, but keep them flexible and configure them to match a given discipline.

All in all - great article 😊

Collapse
 
joehobot profile image
Joe Hobot • Edited

True, this was more from DevOps prospective, however you could/can work along the DevOps (if you don't have own full access dev env) in where you could observe as to what happens if you shut off network , stop access to database, provide wrong username, dependencies on other apps and such.

When I do things like one above, I work with Devs because they help me provide better insights into app and or logs. So that Noc has a notification at least that an app restarted and healed it self by running script xyz that also restarted dependencies..

Thank you for commenting, much appreciated thoughts from different prospective..