Are you Logging your applications RIGHT?

Log Management & Monitoring

This document is the baseline for setting up the strategy for Logging & Monitoring.

Table of Content

• What to Log
  • Basics
  • Events to Log
• How to Log
  • Actionable's
  • Log Management
  • Security Aspects In Details
• Roles & Responsibilities
  • Log types
  • What to never log
• Next ToDo’s
  • Select Logging Framework for Frontend/Backend/Infra etc
  • References

What to Log

Basics

• Only log what Legislation allowed
  • Example GDPR
• Maintaining Confidentiality | Only Public Records
  • Password & Encryption Keys
  • Payment Details
  • Detailed System Information etc
  • Information exposure through Error Messages
• Sufficient Logging | NOT Logging Everything
  • CWE-779 Logging Excessive Data
• Optimization Strategy: Right set of Training & Utilization
  • Not Determining what to monitor & how
  • Right Training & Documentation
  • Putting right practice for Alerting

Events to Log:

• Authentication Events
  • All Success & Failure
  • Building a security policy
  • Access control violations
  • Incorrect Logins
  • Policy for Internal Employees
  • User Authentication(Failed/Reset/Successful) for all services including k8, Applications, Internal Systems
• Authorization Events
  • Failure of Tokens & Internal Access Violation
• Application Errors
  • All Application Errors
  • Startup & Shutdown Events
  • Configuration Changes
  • Application State Information
  • Input & Output validation(maintaining signal to noise ratio)

How to Log

Actionable's

• Deciding where to record log files
  • Local Logs(Not a good practice)
  • Not to log locally, but if its required, log in separate partition & enable access control.
  • Access Control
  • Logging Remotely or Centralized Log-Server
    • Full Encryption
    • Access Control Mechanism
    • Integrity Checks for Log Files
    • Fail-over system
    • Setting up regular backups
    • Adding alerting
• Format of log files
  • Metadata requirements
  • what has happened
  • when it happened
    • timestamp with Timezone
    • source
    • destination
    • Synchronize Time Sources
  • where it happened
    • source address
    • originating source(ipv4/ipv6)
  • who is responsible for the action
    • Logged on/Attempting User
    • Unique identifier
  • Standard Logging Format
  • Timestamp
  • Encoding - UTF8
  • Severity Levels(Standards)
    • 0 - Emergency - System crash
    • 1 - Alert - action must be taken
    • 2 - Critical - critical condition such as load
    • 3 - error - error conditions
    • 4 - warning
    • 5 - Notice
    • 6 - Informational
    • 7 - Debug
• Handling Personal Data
  • Encrypting Personal Data
  • Pseudonymization(Privacy Enhancing Techniques)
  • Consulting Legal

Log Management

• Log Aggregation
  • Consolidate Duplicate Events
  • Add Structure to the Docs
  • Remove sensitive Data through filters
  • Pseudonymization
  • Security
  • Input validation(XSS also possible in log files)
  • Encoding(Safeguard against injection attacks)
  • Filtering
• Log Analyzing
  • Baselining
  • Anomaly Detection
  • Attack Signatures
• Log Archiving

Security Aspects In Details

• Information exposure through Error Messages
• Unsuccessful login attempts should be monitored
• Never, ever log credentials, passwords or any sensitive information.
• Detecting Network Intrusion
• ELK Login to be Employee should be strong & unique password to prevent
• Following OWASP Top-10

Roles & Responsibilities

• IT Leadership
  • Assigning responsibilities
  • Tools & Trainings
  • Incident response
• Software/DevOps Lead
  • Defining what to log & monitor. (Generally this happens in after Planning phase)
  • Clear Documentation
  • Common Format of log files
  • Optimization or tweaking monitoring during operational phase
  • Maintaining the policy
  • Action on Log
  • Adding Logging/Monitoring to release notes & continuously monitoring behaviour in production
• DevOps
  • Defining Response strategy
  • Setting up Infra & Enabling
  • Common Format of log files
  • Continuous Monitoring & Alerting
  • Verifying Pattern such as continuous login attempts
  • Defining Response strategy
  • Reporting policy violations
  • Generating regular reports
• Legal Department
  • Validating the information logged & stored in database

Log types

• Application logs[http request/response log, application level error

     {
     'id' : 'unique request id for tracing',
     'req' :{}, //Generated by Logger
     'res': {}, //Generated by Logger
     'level':'error',
     'message':'There is an error',
     'timestamp': '2016-06-12T05:00:00',
        'timezone': 'Pacific/Auckland',
     'context':'exception trace'
    }

Reference

• db log [query,error]

     {
       "timestamp": "",
       "severity": "",
       "component": "",
       "context": "",
       "message": ""
       "dbStats" : {}, //Generated by Logger
       "serverStatus": {} //Generated by Logger 
     }

• container level log

    {
       "container" : 
       {
         "name": ""
       },
       ecs : {}
    }

See the demo for reference

What to never log

• Customer Confidential Information
• Payment Records such as Credit Card
• Banking Information

References

• https://www.datadoghq.com/blog/monitoring-mean-stack-applications-with-datadog/
• Database Logs
  • https://docs.mongodb.com/manual/reference/command/serverStatus/#server-status-locks -https://docs.mongodb.com/manual/reference/command/dbStats/#dbcmd.dbStats
• K8 Logging
  • https://itnext.io/logging-best-practices-for-kubernetes-using-elasticsearch-fluent-bit-and-kibana-be9b7398dfee
• https://github.com/pimterry/loglevel
• https://medium.com/@davidmcintosh/winston-a-better-way-to-log-793ac19044c5
• https://stackoverflow.com/questions/5817738/how-to-use-log-levels-in-java
• https://blog.papertrailapp.com/best-practices-for-logging-in-nodejs/
• https://github.com/trentm/node-bunyan#readme
• https://medium.com/containerum/4-tools-to-monitor-your-kubernetes-cluster-efficiently-ceaf62818eea
• https://blog.coinbase.com/logs-metrics-and-the-evolution-of-observability-at-coinbase-13196b15edb7
• https://medium.com/easyread/understanding-about-rfc-3339-for-datetime-formatting-in-software-engineering-940aa5d5f68a
• https://webilicious.xyz/utilizing-winston-to-log-node-js-applications/
• Security
  • https://attack.mitre.org/
  • https://cwe.mitre.org/
  • https://cwe.mitre.org/data/definitions/778.html
  • https://cwe.mitre.org/data/definitions/223.html
  • https://cwe.mitre.org/data/definitions/209.html
  • https://cwe.mitre.org/data/definitions/779.html