loading...
Cover image for 10 Things You Can Do If You Have A Bad Deploy

10 Things You Can Do If You Have A Bad Deploy

flippedcoding profile image Milecia McG Updated on ・4 min read

Think about this. You just finished some updates to your website or project. Of course you're excited because you fixed a lot of bugs and you got to show a few people how great it was when you ran it locally.

If you're in a small company or you work for yourself, you might have to deploy those changes to the production server yourself. So you clean you your code, make all of your commits in Git, and deploy the changes that make you look like you can do anything.

Then it doesn't work. At all. Now you have angry people telling you to fix it and you're not sure what the problem is. Here's a few things you can check really fast to help figure out the issue:

Check the connection strings

It's nice when it's something easy like this. Make sure that you're connecting to the production database and not your development one. Depending on how you do your deployment, the connection string could be automatically updated to match the environment its in. That could mess up the string value and cause weird issues.

Check the CORS settings

If you have any calls to APIs that aren't connected to your site, you definitely need to check your CORS. The cross origin resource sharing settings you have could block any incoming data you expect. So if you’re using services like Google Maps, Stripe, or SendGrid make sure you quadruple check your CORS settings.

Check the username and password for everything

Sometimes it's the little things. Make sure you used the right username and password for everything your site uses. That includes the database, any API authentication, or internal authentication. It's best to check these simple things just to make sure you do waste time looking at other issues that aren't the problem.

Check the Publish procedure

Some people like to use build pipelines to run any last minute tests or checks and it's not a bad idea if you can do it. A part you can include in a pipeline is automatically publishing the new files once they pass all the tests. Make sure that this publish procedure references the right folders and files on both the client and server sides.

Check the published files

This might not be possible all the time, especially if you chunk files. Although if you have the option to look at the files it's a good way to find out if the right files were published. You can always check the date stamp, but sometimes it helps to see if the changes you made are actually there. Maybe you accidentally published the wrong branch of the site. It happens.

Check the appsettings.json

There's a chance that a value in here was changed during local testing and wasn't changed back before you deployed. The server root address will probably be different on your live site versus your local one. An IsEnabled value might need to be false instead of true for things to work on the live site. Just do a quick run through to make sure nothing looks unusual.

Check the config files

This is where a lot of the behind the screens rules are for your site. Is there a Rewrite action you're missing or anything else? You could have some routing that isn't quite right. Again, just go through everything and check the values. Also, if you have another project that you worked on similar to this, you can see what values it has.

Check the changes you made

Did you break it? It's not the most unlikely scenario. We all do crazy hacks to get stuff working locally that we would never purposely deploy. Check your code to see if you removed anything like that because that's most likely your problem. Even if it runs on your machine, just go through your changes and see if you did something odd.

Look for version differences

Is the front-end of your site still compatible with the back-end? Is the back-end still compatible with the server? Updates to libraries or entire frameworks can cause some nightmares because they take the longest to figure out. Save yourself a lot of time and swearing and check those version compatibilities.

Republish an older, working version the site

Until you can figure out why the site isn't working with your latest changes, it's ok to re-deploy an older, working version of the site. Anytime a site is down, the person who owns the site is losing money. You don't want that because then you'll lose money. Explain to your person that you'll just have to go back to the older version until you find the issue with the new one. They understand. Sometimes…

Do you have any horror stories of a project you KNOW worked, but it just wouldn't work when you deployed it? I definitely have a few.


Hey! You should follow me on Twitter because reasons: https://twitter.com/FlippedCoding

Posted on by:

flippedcoding profile

Milecia McG

@flippedcoding

Starting classes soon! | Software/Hardware Engineer | International tech speaker | Random inventor and slightly mad scientist with extra sauce

Discussion

pic
Editor guide
 

Excellent post with a helpful checklist. I think that automated Continuous Deployment process could help with most the list positions to check. It makes the deployment process repeatable. Additionally, we can use Azure Key Vault or HashiCorp Vault to store connection strings and other settings.

 

Thanks! I usually like to use build and release pipelines in Azure just to keep everything consistent (and so I can't forget anything). I haven't used a key vault before, but I think I'll start looking into it! 🙂

 

Thanks for this post. It's one thing to have a working development environment, or even staging environment. But Production releases, especially when they are mostly manual always make me double check things. I prefer to have a deployment checklist that I can walk myself through, no matter how well I know the system.

For Production I also like to have a roll-back procedure in place similar to the deployment procedure. If all else fails, you can always go back to a previous release. Some systems this is easier said than done.

Also, it's sometimes nice to let end users through some sort of notification, that the site or resources will be temporarily unavailable. I like to give myself more time than what I think, in case something does happen.

We were working on a release one time that had numerous database updates and migrations. Working with the DBA, we pushed through the changes, long story short, the schema changes were applied and absolutely nothing worked. It was about 24 hours in identifying numerous migration issues. Friday night release turned into a Saturday at the office.

 

The migrations and database updates always make me a little nervous. It just takes one little thing to mess up everything. We had someone updating stored procedures and they forgot to remove a DELETE statement they were using for testing AND they didn't change their connection to localhost or even dev... Yeah... There was a lot of panic going on when users couldn't access their accounts anymore. I feel your pain of Friday turning into Saturday.