I've just completed building Logs Analyzer library that provides functionalities to filter and analyze logs and in a way that doesn't affect the source application nor the original logs. The analyzer can be run either in the same application domain for simplicity or in a separate process so that it doesn't affect the original application from which the logs are being generated.
The idea behind the library is to facilitate troubleshooting applications issues and to monitor users' behaviors in dealing with the application. It also gives the ability to report the analyzed logs in more readable ways.
The library introduces the following three main functionalities:
Logs Filtering:
When the logs are huge or crowded with traces from different processes or application's areas they become hard to read which makes it helpful in some cases to report only part of the logs that are related to a specific feature or service. The user of the library can specify what logs they need to include in an output by specifying values for the components of each log.
The log's components that the filters can be applied on are: message, level, tags, extra parameters.
The following is part of the analyzer's configuration file that identifies two logs messages:
Logs Analyzing:
The library also supports applying analysis methods on the incoming logs that match defined filters (using Logs Filtering). The following are some of the supported methods:
1. Time Difference: this method expects to have two different logs as input, and the output will be the time difference between them whenever the second one is received.
2. Logs Frequency: it calculates the frequency at which a defined log message is being received.
3. Duplicate Log Detection: the analyzer will detect receiving a specific log message more than once.
4. Log Absence Detection: triggers an output if a specific log is not received within a configured interval in seconds.
5. Aggregate Function: triggers an output when a specific log is received along with an aggregated value calculated from a specific log parameter. The aggregate function used should be also specified in the configuration file.
The following is part of the library's configuration file for the supported analysis methods:
Logs Outputs:
The library also supports multiple ways of reporting the output of both Logs filters and analytics. The following are the supported output methods:
-
CSV files:
HTML files: the library has a simple HTML template page for logs but it can be replaced.
Text files.
Emails (this is still being validated).
Windows Popup messages:
General Notes:
- The logs analyzer can be enabled\ disabled from the associated configuration file without the need to restart the original application.
- While the analyzer library can be hosted within the domain of the source application, it can be also hosted by a standalone application for better isolation. To achieve that purpose it supports receiving logs using Named Pipes or Anonymous Pipes.
- Each analysis method can be associated with different output.
- Each analysis or filtering method can be configured to be run during a specific time.
- Emails and windows popups are used to quickly report a specific analysis output or issue, therefore they should be used when the expected results are not frequent or for exceptional cases.
- The analyzer also accepts having optional arguments with each log like tags and Key-Value parameters, and then they can be used in some of the supported analysis methods.
- While the analyzer can introduce useful reports and outputs, I don't recommend to fully depend on these reports as a complete substitution for building actual reports that depend on an official data source not the logs.
- It's also worth to mention that the analyzer methods are useful depending on the scenario you are monitoring and so not all methods will introduce helpful results for every log.
I've just published the library on github but I'm still continuously testing and validating it.
Top comments (0)