loading...
markdown guide
 

I pushed an infinite loop into our deployment process which took down the site and got us rate limited and suspended by Heroku. It also caused elevated failure rates on their whole service enough that status.heroku.com had a yellow message about the situation.

 

Two things, both on a large Wordpress installation:

  1. While testing a migration of some custom tables, I deleted all our custom tables to do a clean run of those migrations. Unfortunately, I was in the SequelPro tab of the production database, not my local database.
  2. I thought we were caching stuff in memcached, but the way you get Wordpress to use the cache is really hacky and I couldn't get it to work locally. So my typical workflow was to comment that file out, at which point Wordpress will start caching stuff to the database instead. Well, I accidentally pushed that commented file into production, so this large site with thousands of concurrent users started caching everything to the database instead of memcached. It didn't take long for MySQL queries to get exponentially longer due to millions of rows being queried on each request, bringing the entire site down.
 

Once removed a Tomcat server while trying to remove symlink. Fortunately was a internal server for CI that was being implemented.
And other time deleted a production database. Fortunately there was a morning backup and lost 3 articles

 

One of my calculations worked backwards. The internal testing was manually triggering the fake passage of time by increasing the number of days, and it worked perfectly. As soon as it went to live I realised it was supposed to be x - y, not y - x and the whole thing ran backwards. read: it didn't work.

 
 

First webmaster gig and first formal job. Learning FTP accidentally deleted a whole Wordpress site. Almost got fired on my first day.

 

Sounds like their screw-up to me! First day on the job and they have you "learning" FTP on the server with important sites on it?

Accident waiting to happen.

 

Yeah the company management and infrastructure were missing hahahaha

 

Hope the company had the proper source control etc. and it was just a matter of redeploy? Or not...? :)

 

No it didn't. Infrastructure was not a word in that company hahaha. I got lucky because there was another copy of it in an old HDD

 

Forgot that the INT column type in MySQL only went up to 2147483647 - I was storing Facebook UID in that column for a fairly large-scale and very well known company's Facebook contest.

That was 6-ish years ago, pretty sure that's the last time I worked in that city.

 
 
Classic DEV Post from Feb 1

What is your morning routine?

So I wanted to try and do at least a post a week, and I didn't have much time to prepare one this wee...

Marcin Zajkowski profile image
Umbraco Trainer and Certified Master @cogworks || Team Captain @wowschoolpl || #umbraCoffee http://umbra.coffee Co-Host || IT Consultant and Public Speaker