A primer on Enterprise Log Management Systems

This is a first article on series of articles regarding end to end log management in enterprise scenario. Feedback and suggestion are welcome.

Lets talk about Logs. Logging is a art balancing between right amount of data and least resource consumption. Logs are critical part of applications. Without logging it is impossible to find out root cause of issues. Logs are life saver when you are working on production environment where you don't have access to code. So log management is important. Log management consist of collecting, processing, storing the Logs. The log management system should be robust enough to handle logging failures without affecting application. It should be non-blocking asynchronous, so application performance should not affect. Also it should consume least amount of resources, so that it can be used in resource constrained edge devices.

Difference between logging and monitoring ?

Logging is managing data generated by application, infrastructure etc while monitoring is observing different metrics produced from system by human or automation tool.

whats difference between logs, metrics, and events ?

• Logs = Generally a log consist of date, time, application name, severity and log message itself. Logs also have predefined standard format. Example of log is anything that is written by application on stdout (console) or a log file.
• Metrics = Metrics are the set of values over a time period which is tagged using different properties or attributes which can help in filtering. Example of metric could be your marks in school with property like subject name or student name which can be helpful in filtering.
• Events = Events are actions or state changes that is recognized by software. Example of event will be starting/stopping an application or creating of a file etc. Although title says Log management system, It still manages logs, metrics and events although a bit differently.

How Enterprise Log management is different from regular log management?

Enterprise uses multiple platforms, technologies, cloud service providers and devices. Enterprise has to deal with lots of personal data in logs which on exposure could cost company a lots of reputational damage, client relationships as well as financial liabilities. At the same time Operations and Security teams want logs data to be available at one place. So that they can use it for different purposes such as error resolution, anomaly detection etc.

What are the steps and challenges associated with an enterprise log management system?

Log management can be broadly classified into 5 different steps. This steps and challenges associated with them are listed below.

1. Collect
   • deciding which logs are useful
   • collecting logs from different tools and platforms.
   • transporting logs securely to central location.
2. Process
   • Converting logs into standard/common format.
   • managing/applying uniform timestamp on logs. (If possible use NTP server.)
   • handling personal information in logs.
3. Store
   • storing logs in appropriate format.
   • access/manipulate logs in real time.
   • retrieve relevant logs using advance search engine to assist IT person.

What are the types of logs and how they are useful to us?

1. application logs : Logs generated by applications. This logs are immensely useful while debugging.
2. clients/endpoint logs : This logs provides us with demographic such as devices used to access the data, platforms used to access the application. So that we can test user experience on this devices.
3. system/server logs : This are the system logs where our application resides. This logs used to decide scaling of our applications. Here proper selection of metrics is the most important task.
4. authentication/authorization logs : This logs tells us who does what. This logs are most important from security perspective.
5. web traffic flow logs : Tells us about the flow of data in our systems. Very useful during network outages.

In next article, we will take a in depth look at what are the challenges faced while collecting logs and how to resolve them. We will also look at the best practices and tools for log collection.