DEV Community

loading...

Are you Logging your applications RIGHT?

Akshat Sharma
Passionate about Engineering & Promoting Engineers
・4 min read

Log Management & Monitoring

This document is the baseline for setting up the strategy for Logging & Monitoring.

Table of Content

What to Log

Basics

  • Only log what Legislation allowed
    • Example GDPR
  • Maintaining Confidentiality | Only Public Records
    • Password & Encryption Keys
    • Payment Details
    • Detailed System Information etc
    • Information exposure through Error Messages
  • Sufficient Logging | NOT Logging Everything
    • CWE-779 Logging Excessive Data
  • Optimization Strategy: Right set of Training & Utilization
    • Not Determining what to monitor & how
    • Right Training & Documentation
    • Putting right practice for Alerting

Events to Log:

  • Authentication Events
    • All Success & Failure
    • Building a security policy
    • Access control violations
    • Incorrect Logins
    • Policy for Internal Employees
    • User Authentication(Failed/Reset/Successful) for all services including k8, Applications, Internal Systems
  • Authorization Events
    • Failure of Tokens & Internal Access Violation
  • Application Errors
    • All Application Errors
    • Startup & Shutdown Events
    • Configuration Changes
    • Application State Information
    • Input & Output validation(maintaining signal to noise ratio)

How to Log

Actionable's

  • Deciding where to record log files
    • Local Logs(Not a good practice)
    • Not to log locally, but if its required, log in separate partition & enable access control.
    • Access Control
    • Logging Remotely or Centralized Log-Server
      • Full Encryption
      • Access Control Mechanism
      • Integrity Checks for Log Files
      • Fail-over system
      • Setting up regular backups
      • Adding alerting
  • Format of log files
    • Metadata requirements
    • what has happened
    • when it happened
    • where it happened
      • source address
      • originating source(ipv4/ipv6)
    • who is responsible for the action
      • Logged on/Attempting User
      • Unique identifier
    • Standard Logging Format
    • Timestamp
    • Encoding - UTF8
    • Severity Levels(Standards)
      • 0 - Emergency - System crash
      • 1 - Alert - action must be taken
      • 2 - Critical - critical condition such as load
      • 3 - error - error conditions
      • 4 - warning
      • 5 - Notice
      • 6 - Informational
      • 7 - Debug
  • Handling Personal Data
    • Encrypting Personal Data
    • Pseudonymization(Privacy Enhancing Techniques)
    • Consulting Legal

Log Management

  • Log Aggregation
    • Consolidate Duplicate Events
    • Add Structure to the Docs
    • Remove sensitive Data through filters
    • Pseudonymization
    • Security
    • Input validation(XSS also possible in log files)
    • Encoding(Safeguard against injection attacks)
    • Filtering
  • Log Analyzing
    • Baselining
    • Anomaly Detection
    • Attack Signatures
  • Log Archiving

Security Aspects In Details

  • Information exposure through Error Messages
  • Unsuccessful login attempts should be monitored
  • Never, ever log credentials, passwords or any sensitive information.
  • Detecting Network Intrusion
  • ELK Login to be Employee should be strong & unique password to prevent
  • Following OWASP Top-10

Roles & Responsibilities

  • IT Leadership
    • Assigning responsibilities
    • Tools & Trainings
    • Incident response
  • Software/DevOps Lead
    • Defining what to log & monitor. (Generally this happens in after Planning phase)
    • Clear Documentation
    • Common Format of log files
    • Optimization or tweaking monitoring during operational phase
    • Maintaining the policy
    • Action on Log
    • Adding Logging/Monitoring to release notes & continuously monitoring behaviour in production
  • DevOps
    • Defining Response strategy
    • Setting up Infra & Enabling
    • Common Format of log files
    • Continuous Monitoring & Alerting
    • Verifying Pattern such as continuous login attempts
    • Defining Response strategy
    • Reporting policy violations
    • Generating regular reports
  • Legal Department
    • Validating the information logged & stored in database

Log types

  • Application logs[http request/response log, application level error
     {
     'id' : 'unique request id for tracing',
     'req' :{}, //Generated by Logger
     'res': {}, //Generated by Logger
     'level':'error',
     'message':'There is an error',
     'timestamp': '2016-06-12T05:00:00',
        'timezone': 'Pacific/Auckland',
     'context':'exception trace'
    }

Reference

  • db log [query,error]
     {
       "timestamp": "",
       "severity": "",
       "component": "",
       "context": "",
       "message": ""
       "dbStats" : {}, //Generated by Logger
       "serverStatus": {} //Generated by Logger 
     }
  • container level log
    {
       "container" : 
       {
         "name": ""
       },
       ecs : {}
    }

See the demo for reference

What to never log

  • Customer Confidential Information
  • Payment Records such as Credit Card
  • Banking Information

References

Discussion (0)