I pushed an infinite loop into our deployment process which took down the site and got us rate limited and suspended by Heroku. It also caused elevated failure rates on their whole service enough that status.heroku.com had a yellow message about the situation.
Once removed a Tomcat server while trying to remove symlink. Fortunately was a internal server for CI that was being implemented.
And other time deleted a production database. Fortunately there was a morning backup and lost 3 articles
Umbraco Trainer and Certified Master @cogworks || Team Captain @wowschoolpl || #umbraCoffee http://umbra.coffee Co-Host || IT Consultant and Public Speaker
Umbraco Trainer and Certified Master @cogworks || Team Captain @wowschoolpl || #umbraCoffee http://umbra.coffee Co-Host || IT Consultant and Public Speaker
Two things, both on a large Wordpress installation:
While testing a migration of some custom tables, I deleted all our custom tables to do a clean run of those migrations. Unfortunately, I was in the SequelPro tab of the production database, not my local database.
I thought we were caching stuff in memcached, but the way you get Wordpress to use the cache is really hacky and I couldn't get it to work locally. So my typical workflow was to comment that file out, at which point Wordpress will start caching stuff to the database instead. Well, I accidentally pushed that commented file into production, so this large site with thousands of concurrent users started caching everything to the database instead of memcached. It didn't take long for MySQL queries to get exponentially longer due to millions of rows being queried on each request, bringing the entire site down.
One of my calculations worked backwards. The internal testing was manually triggering the fake passage of time by increasing the number of days, and it worked perfectly. As soon as it went to live I realised it was supposed to be x - y, not y - x and the whole thing ran backwards. read: it didn't work.
Umbraco Trainer and Certified Master @cogworks || Team Captain @wowschoolpl || #umbraCoffee http://umbra.coffee Co-Host || IT Consultant and Public Speaker
Umbraco Trainer and Certified Master @cogworks || Team Captain @wowschoolpl || #umbraCoffee http://umbra.coffee Co-Host || IT Consultant and Public Speaker
I pushed an infinite loop into our deployment process which took down the site and got us rate limited and suspended by Heroku. It also caused elevated failure rates on their whole service enough that status.heroku.com had a yellow message about the situation.
Once removed a Tomcat server while trying to remove symlink. Fortunately was a internal server for CI that was being implemented.
And other time deleted a production database. Fortunately there was a morning backup and lost 3 articles
First webmaster gig and first formal job. Learning FTP accidentally deleted a whole Wordpress site. Almost got fired on my first day.
Sounds like their screw-up to me! First day on the job and they have you "learning" FTP on the server with important sites on it?
Accident waiting to happen.
Yeah the company management and infrastructure were missing hahahaha
Hope the company had the proper source control etc. and it was just a matter of redeploy? Or not...? :)
No it didn't. Infrastructure was not a word in that company hahaha. I got lucky because there was another copy of it in an old HDD
Oh man, so it was fun! ;)
Two things, both on a large Wordpress installation:
One of my calculations worked backwards. The internal testing was manually triggering the fake passage of time by increasing the number of days, and it worked perfectly. As soon as it went to live I realised it was supposed to be x - y, not y - x and the whole thing ran backwards. read: it didn't work.
🙈
sudo rm -r -f /*
Oh no! How dare you?! ;)