When you have a service running somewhere you need to find out whether it is functioning correctly. Besides the possible tests, liveness checks, metrics, you can use application logging. But what makes an application log “Good”? In this blog we discuss 6 pointers on application logging to help you on your way!
1. Purpose is everything!
Before you start writing any log-output, first step back a little. Now ask yourself: where are you logging for? What is the goal and who is your intended audience? Do you intent to have a log which is targeted at application troubleshooting? Or is this a log used to capture all relevant events as part of an audit-log?
Once you have the purpose and the audience of the log defined, you can give it a go. Below you can find a couple of different purposes for logging. Of course, there are many more you could think of, but let’s just pick up a few:
Purpose case: logging for application maintenance
When you setup logging for your application maintenance, you should think at least of:
- Do you want to inform people about certain steps which are executed by the application?
- Which exceptions or errors in your application are actually worth logging in terms of faulty code paths? Does informing about them lead to better maintenance and/or business value?
- When is a certain state of the application worth waking up the standby for? There are various ways how you can resolve this. One of them is: use the difference between error level logging and warning level logging. error level would mean: get into action now, warning level could mean: when you are ready, pick it up. Many exceptions and errors that your code throws might not be worth picking up, and should not be logged as error or warning.
There are a lot of different things to consider. We often see applications throwing too many exceptions into the log, though the execution path through the exception still leads to the actual state which delivers value to a customer. Therefore, logging to help with your application maintenance goes beyond writing and storing logs: it starts with(re)designing your application control flows.
What can help here, is to ask yourself per user story: “what do you need to log to know that the functionality works correctly?” & “what do you need to log when the functionality fails?”
Purpose case: Audit Logs
Audit logs have very different requirements than application logs. In an audit log you need to capture every event that leads to the relevant state of change in your application or devices. This is rather different than the log for application maintenance: here the reason for an internal server error due to the lack of a database connection is less relevant. However, the steps your users went through in order to execute a transaction is key to the system.
Integrity of the audit log is important as the logs can serve as evidence in court if it comes to legal disputes about actions of a user or the system. This requires integrity protection of the audit log, as well as ensuring that all relevant actions are recorded. This works best if you design your logs for easy correlation: make sure you link the relevant logs from the start! This can be via a session-identifier, trace-IDs, et cetera.
Note that audit log messages require their own aggregation as well. So make sure that you format the messages in the same way.
Purpose case: Security Event Logs
Security event logs should be designed to inform the security team about any event they should be aware of. Ask the security team what they need from the logs: does this include the “happy flows” in your audit-logs? Dos this include all the error-states in your application logs? A security event log should be based on a threat model. What are you afraid of? Which scenario might be problematic to the security of the system and/or users?
2. Secrets in Logging
We often have to load secrets, such as credentials and/or cryptographic keys into our application. These secrets can be easily misused, when they fall into the wrong hands. Think of passwords to powerful administrative backends, or signing keys used to proof that a transaction came from the system to which the key belongs. In both cases, an attacker can easily do harm with them. Therefore, it is better not to put these kind of secrets in logging. After all: anybody with access to these logs might be able to leverage the secrets for a privilege escalation. A nice example of this can be found at challenge 8 of OWASP WrongSecrets.
3. Sensitive Data in Logging
Where passwords are an easy“no-go” for logging, there are quite a few items that could be“too sensitive” to log into your application logs without any additional obfuscation/encryption/masking:
- Financial, Health & Business information which should not be shared publicly
- Personal Identifiable Information as defined by the GDPR or other applicable laws and regulations.
- Application source code
- Any other type of information which is deemed too sensitive based on the underlying threatmodel.
Does that mean that you cannot log all of this information at all? Of course it should be, if relevant, part of your audit-log. But it might be a good idea to not share this in your application debug log.
4. Logging and Security
There are many security topics that are relevant if it comes to logging data. Let’s touch upon a few:
- Never trust external input. Never just insert raw data into your logs. Instead make sure that you only record data from trusted zones. Next: make sure you do proper encoding & data sanitization. WebGoat contains a nice challenge which shows what happens if you do not clean your data properly.
- Harden your setup. The recent events with Log4J show that we should ensure that our logging infrastructure is patched, hardened, and monitored.
- Protect integrity of your logs: Integrity of data becomes all the more relevant when the logs need to show what has happened. Audit-logs, transaction-logs, and such often need to be protected by either having signed/HMACed messages or using a WORM(Write Once, Read Many) storage solution.
- Protect the data. Logs should be protected at least as the same level as the data the process is dealing with. This means that you need to setup access management for logs, encrypt them at rest, et cetera.
5. Stacktraces and Logging
Stacktraces require special attention. When stacktraces encompass business objects in their descriptions, they can leak information which we just listed not to log. Therefore, always be careful with what you put in the context of an exception. Make sure that either your business objects, or your logging & exception handling configuration ensures that confidential information is cleared out during a stacktrace.
6. Actionable Logs(with context)
Make sure you don’t log just for the sake of logging. Try to understand what you need to record for which purpose, and keep it to that level. We often see log-items such as “Received incoming message”, and “starting processing message” in a row without any context of the message. This makes it hard to understand what is going on. It can help when you are debugging in order to see whether your received message is then being processed, but it does not help fellow developers after that. After all: the more log statements, the higher the risk that important events might be missed. That is why we recommend to prune your log messages(or lower their log-level) when you are done debugging.
Where to Go Next?
There are many more things you need to consider when designing your application logging, such as:
- What information is needed for higher level business goals?
- Which sources should you include to provide context to your logs? Think about the logs coming from your host-machines, cloud infrastructure, etc..
- Where do you store the logs over time?
For now: check your application logs and get in touch with the parties that rely on your logging information! Take the 6 pointers presented here and evaluate together: do the logs deliver the value you need? Is the application is designed in a sensible way when it comes to erroring and exception handling.
Read More:
- WebGoat 8: https://github.com/WebGoat/WebGoat
- OWASP cheatsheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
- Log4J:https://xebia.com/log4j-a-10-step-mitigation-plan/
- Wrongsecrets: https://github.com/commjoen/wrongsecrets
Top comments (0)