Thanks for this very instructive article! Here are a couple of bash tricks you may use to slightly simplify your scripts:
Instead of using a retry loop, the random numbers can be generated with
echo$((floor + RANDOM %(range - floor +1)))
A very old trick used to avoid grepping-out grep:
grep"[n]ginx: worker process"
But note that you may be able to avoid grep if you can ask ps to display only the relevant processes. On Ubuntu, you may select the process that are owned by www-data:
I think there was a great learning experience in all of this, but here are some resources to consider researching if you haven't:
Ansible - makes managing remote servers easier, especially the bit about running a script remotely
Haproxy - reverse proxy to use instead of nginx, might be better at swapping backends gracefully
Kubernetes (potentially too heavy for this use, but handles basically everything you built here)
Also just a note from the outside about the port switching: your solution chooses a random port and checks if it is available. You could apply this logic to the original idea of swapping between 2 ports. Port A is the default, check if Port A is available. If it is, use it. If it isn't, use Port B. No state to keep track of, you're just dynamically choosing whether to use the port or to use an alternate one.
Thank you. That's helpful. I particularly like your idea about how to get rid of the random port generation. That would remove the need for cleaning up the old code (because I would only ever have two copies of it on the machine, instead of up to 1,000) and would open the path to quick rollbacks.
Jordan already provided a list of tools that can improve the process. I just want to add about several moments in your process:
if there is a daemon manager (service or systemctl), it's better to use it instead of nginx -s SIGNAL because the manager won't know anything about such reloads and won't be able to interact with the running process anymore.
checking when to remove the old code directory - since you know the old port, why don't just check if there are open connections to them? No connections - remove the code
the 1-second delay - when you do the switch, are you sure the new deploy is ready to be served? Perhaps it makes sense to make a readiness request before the switch? Anyway, you can use tcpdump (or anything similar) to record network traffic, with port/protocol filtering, and figure out to which backend (old or new port) is was made and failed to complete.
to run tasks in parallel, you can use parallel-ssh
speaking about one failed host - there are basically only two options if your scripts remove the old code and do it in parallel on several hosts. You can re-deploy for all or just for the failed host. Even rollback here is a re-deploy.
By the way, you can keep several old release folders to stop worrying about the right time to remove the previous release and to be able to revert your deployment (however you would need to track which is the previous one, but it is solvable)
Thanks for this very instructive article! Here are a couple of bash tricks you may use to slightly simplify your scripts:
But note that you may be able to avoid grep if you can ask ps to display only the relevant processes. On Ubuntu, you may select the process that are owned by www-data:
I think there was a great learning experience in all of this, but here are some resources to consider researching if you haven't:
Also just a note from the outside about the port switching: your solution chooses a random port and checks if it is available. You could apply this logic to the original idea of swapping between 2 ports. Port A is the default, check if Port A is available. If it is, use it. If it isn't, use Port B. No state to keep track of, you're just dynamically choosing whether to use the port or to use an alternate one.
Thank you. That's helpful. I particularly like your idea about how to get rid of the random port generation. That would remove the need for cleaning up the old code (because I would only ever have two copies of it on the machine, instead of up to 1,000) and would open the path to quick rollbacks.
That's a great point about the quick rollbacks. It would make the blue/green nature even better.
Did you check CapRover (caprover.com)? It's basically a self-hosted Heroku 🤷♂️. I like it. You can even manage a cluster with it.
Holy smokes! I haven't heard of that before. I gotta try it out. Thank you!
Jordan already provided a list of tools that can improve the process. I just want to add about several moments in your process:
Thanks for taking the time to read this post and to share your recommendations for improvements. They're helpful.
Part of your solution could be tightened up with this simple command.
setcap 'cap_net_bind_service=+ep' /usr/bin/nodeHere's the full story. How to Configure Node.js to Use Port 443
"Granting server access to well-known, privileged Linux system ports."
Great article it looks like you covered everything up.
I do have a question for you, how much time did spent to put all this together?
It took me 3-4 days to figure it out and code it up and then 1 day to write the blog post about it (because I was sick).