How should we keep our logs?

Since I have been dealing with a lot of log operations lately, I wanted to do detailed research on how to log the software. In order not to lose what I learned from these sources, I think it would be good to write an article and share what I learned with you. Then let's get started.
History of the Log
Syslog was developed as part of Sendmail in the 1980s. This has been adopted by other developers and has become the standard. Since then, many log libraries have matured and developed in our lives.
Advantages of logging
We may be keeping logs for many different purposes. Some advantages of keeping log records are as follows.

• To see the IP source of outgoing requests to our system
• Understanding the density of our system (User Interactions)
• Detecting system behaviors (System start times, stop times)
• Exploring performance statistics (request-response times)
• Detecting threats and vulnerabilities (Constantly failing user verifications)

Log Levels
The most difficult thing to encounter while keeping logs is at what level the logs should be recorded. When looking for a bug or just trying to get an idea of an app, it helps a lot to know what information we can expect to find in the logs. However, we will only know what to expect if we followed a rule when programming log levels. Here we will take a look at the most commonly used logging levels.
Fatal
Fatal is a serious problem. These are serious bugs that could possibly cause your app to crash. Someone should be alerted to fix it immediately, even in the middle of the night. There should be some kind of warning for Fatal events in the production environment.
Error
Error is a serious problem. Unlike Fatal, here it can be considered as a database connection within the application or an inability to access a service.
Be careful not to use the Error log level too generously, as this will add too much misinformation to the logs and reduce the importance of a single Error event. You don't want to be woken up in the middle of the night by something that might have waited until the next morning, do you?
Warn
We are now entering a more gray area. You should use the warn level to indicate that you have detected an unusual situation. Maybe you were trying to call a service and it failed several times before the automatic retry. It is unexpected and unusual, but it does no real harm, and it is not known whether the problem will persist or recur. When someone is available, they can investigate the issue.
Info
Info should be used to document state changes in the application or some objects within the application.
This information can be helpful to monitor what is actually happening in the system during development and sometimes even in the product environment.
Examples of using the Info level are:

• A new record (for example, a user) has been created or its information has changed.
• The status of a particular record (for example, an order) changed from "shipped" to "completed"
• A regularly running Job is started.

Debug
Debug specifies specific and detailed information used for debugging purposes. These logs help us move through the code step-by-step. Start, end, elapsed time, and parameters used for the query can be seen.
Trace
Trace is Very detailed information, used for development purposes only. It is used to provide the most information about a particular event. These logs help you to drill down to variable values. In the production environment, you can keep trace messages for a short time after deployment, but save these log statements temporarily, these records should be closed at the end.
Things to consider while keeping logs
Write meaningful log messages
Probably one of the most important issues is making sense of their logs. When a programmer creates a log, he usually tends to log the exception message directly. However, the important thing is to determine where and why the error originates at first glance.

Transaction failed
Record not found
User operation succeeds

There is nothing worse than this kind of log message. Without the appropriate comprehensible fields, this log message just makes your logs crowded.
A log message in the figure below is much more valuable than the message above.

Transaction failed: Phone 'abcabc' information is incorrect
Product with the id '12121' was not found
User 123456 successfully registered a cell phone.

For this reason, it is important that we write messages in a more readable and understandable way.
It will be easier and more understandable to write the data in the messages in a parsable format.

Transaction failed: Phone information is incorrect {User: 123456, Phone: abcabc}
Product with the id '12121' was not found {Product: 12121}
User successfully registered a cell phone {User: 123456, Phone: 5*********}

You can use a pattern to separate the logs from each other. For example, in the form of APP-S-SUB-CODE, respectively;
APP: Abbreviation for your app name
S: Severity with 1 letter (Ex. D: Debug, I: Info…)
SUB: The bottom of the application that the code is about
CODE: Numeric code specific to the error in question

**ABC-E-USR-111** - Transaction failed: Phone information is incorrect {User: 123456, Phone: abcabc}
**ABC-W-PRD-121** - Product with the id '12121' was not found {Product: 12121}
**ABC-I-USR-123** - User successfully registered a cell phone {User: 123456, Phone: 5*********}

Now, when we compare it with the first version, you can notice that our log records are more meaningful and understandable.
Write log messages in English
This may seem like a strange piece of advice, but I think English is better suited to the technical language. But if we look at it from a technical point of view. It means that your English messages will be recorded in logs with ASCII characters. If your message uses UTF-8 it may not be processed correctly, worst of all, it may become unreadable.
Do not create too many or too few logs
There is a correct balance for log records. Too many logs and it will be difficult to get any value from it. There will be a lot of confusion when manually browsing such logs. If there are too few logs, you run the risk of not being able to fix the problem. Troubleshooting is like solving a difficult puzzle, for which you need to have enough material.
Unfortunately, there is no magic rule when coding what to log.
For example, imagine you have an android robot and you ask him what he is doing today.
Scenario 1: I woke up, moved my head three inches to the right…
Scenario 2: I woke up, got into a taxi…
Usually, we don't need to know everything our robot does. But we might want to review Scenario 1 when it starts to malfunction.
Analyze the logs created when your application is in production and reduce or increase your log record number according to the problems found. Especially when troubleshooting, review the parts of the application where you want more detailed logs. Of course, this issue requires communication between the operation and the developer team.
Do not save sensitive information in the logs!!!
Log records are useful not only for debugging but also for compatibility and auditing. They can be an incredible source of information, as we generally have a tendency to log everything.
If our logs contain usernames, passwords, token information, or other sensitive information, this could lead to a security vulnerability. Therefore, we must consider log records as sensitive and keep them safe. We probably already know not to save the following information.

• credit card numbers
• ID numbers
• Passwords

However, we should not record the following information.

• Token
• Personal names
• Phone numbers
• Session information

Consequent
All in all, while logging is not a complicated task, there is a lot of finesse and balance to getting it right. Too little information will be insufficient. Too much information can be overwhelming if it is not relevant and not properly recorded.
Log records are not optional. Without sufficient logs, your application will be running in the dark.
By learning concepts such as log analysis, you will understand that logging can be much more than a troubleshooting aid. You can extract dormant information from your log files when needed. Armed with this type of information, you can make better decisions and even prevent problems from occurring. If that sounds like magic to you, it isn't. This is the skill you will gain by adopting log management.
Resources
Wikipedia - Syslog
Tom Hombergs - Tip: Use Logging Levels Consistently
Thilina Ashen Gamage - Enterprise Application Logging Best Practices (A Support Engineer's Perspective)
Dave Swersky - C# Logging best practices in 2021 with examples and tools
Priscill Orue - Python Logging Guide - Best Practices and Hands-on Examples
SentinelOne - Java Exceptions and How to Log Them Securely
SentinelOne - Logging Best Practices: The 13 You Should Know
Tomasz Nurkiewicz - 10 Tips for Proper Application Logging