Essential Production Environment Tips from Real-World Experience

In the article, sharing some "Production environment" tips based on my practical experience.
I am trying to keep this blog raw, without modifying the article by chatgpt kind of LLM.

Tip 1: Always take Mongodb Backup everyday.

Once data is lost, data is just lost.
So take mongodb backup everyday.

Story: In the company i was working in, one day, the server had some issue because of it, were not able to access the data.
I have taken a backup everyday (for taking backup i did not taken permission from my boss, this is not so good idea and you don't follow this advice, instead take permission from boss, if boss gives permission, then take backup).
If i didn't taken the backup, then the company just have lost some months data around 3-4 month.

Tip 2: Make documentation

Create documentation.
Documentation can just be a plain text, markdown, screen recording, video recording of process.
If you write it down, its easy to copy paste it.
Documentation has saved my countless hours.

Story:
To setup a server, first time it had taken 4 days. I made basic documentation. The next time, i was just copy pasting the command and it just taken 4-5 hours for complete server setup.
I use docusaurus by Meta to create documentation.
Confluence is also a good tool for taking documentation.
(By docker and/or ansible like tool and some custom code, you may reduce time the time to some minutes for deployment).

List of topic you can make documentation:

1. Make documentation on how to setup server, install mongodb, redis, puppeteer, pm2, docker, node js, nginx, certbot for ssl etc. (if possible, you may use managed services like mongodb atlas, but self host mongodb is also fine).
2. Best practices software developer can use.
3. How to setup local computer environment.
4. Steps on How to take mongodb backup.
5. Some brief document on the functionality of website and what step are involved.
6. Write down all critical test.
7. And whatever things you feel is necessary.

But also don't document too many things, only important things.
Spend 2-3 days in a month for preparing or updating basic documentation.

Tip 3: Make a dev environment

Make a development environment of the website similar to production website.
First deploy the website to development environment, test it, then deploy to production environment.

In dev env, to not to allow email to send to all people, instead make a email list of some internal users, and only allow then to send mail.
Same for mobile, whatsapp, or any critical requirement also.

if(envKeys.SEND_EMAIL_TO_LIMITED_PEOPLE === '1'){
    if(!email include '[e-mail]') {
        // don't send
        return true;
    }
}

sendMails(emailObjArr);

Tip 4: Use managed services instead of self host or self developing (if possible)

Use reliable managed services which are in market for years like s3 (or r2 or digitalocean spaces), mongodb atlas, redis, auth solutions like firebase, supabase, clerk, wix for blogs, builder io for dynamic pages etc.
Many a time, it does not make sense to build the functionality from scratch.
Later, if your scale got too huge, either pay managed solution money or develop your own solution (if it make sense mathematically).

Tip 5: Test required thing before using a library or service.

Story 1st:

My requirement was create PDF and add images to pdf. Before using the library i created a separate project, then try to generate the pdf and added images.
In first library image function was not supported, so i was looking for another solution.
In another library, the image adding functionality was supported. So i taken it as final solution and implemented the same in the project. (Here try to keep the environment same or create a new branch to test).

Story 2nd:

The requirement was to change from cdn.example2.com to cdn.example.com, so what i did was, the company have many domains, instead of using primary domain, i used some other dummy domain and tested all things on how to change from cdn.example2.com to cdn.example.com. Once the testing is successful. Then i moved cdn.example2.com to cdn.example.com.

Story 3rd:

We also need to do internal testing.
For example, the requirement was to send marketing mail to 100k users.
First try to send mail to 10 or 20 internal users.
If that is successful, then only send to all user.
If the content is wrong, as you have send to some internal users, there is no issue.

Tip 6:

"Do not prepare for war in war, instead prepare before the war" - unknown.
Sometime, we need to apply this quote when we known on the website, traffic would be large. Many times, this situation is applicable to large companies or company who will have high traffic (like sale day that will come in once in year).
Or if cannot do it, then at least need to add a functionality to restrict signups or login.
"Anything that can go wrong will go wrong, and at the worst possible time" - Murphy's law.

Conclusion

These are some tips i learned from experience.
Based on the requirements, you may change the tips.
I may be correct in some stuff and also wrong.
Thanks for reading the article.