DEV Community

Vivesh
Vivesh

Posted on

Exploring Logging Best Practices

#elasticsearch #elk #cloudskills #devops

The Essential Role of Logging in Software Systems

Logging is an essential component of any software system, providing critical insights into the behavior, performance, and issues of an application. Effective logging practices enable better debugging, monitoring, and security.

Why Logging Matters

  • Troubleshooting: Helps identify and diagnose issues in applications.
  • Performance Monitoring: Tracks the system’s performance and resource usage.
  • Audit and Compliance: Records user actions for security and compliance.
  • System Health: Provides visibility into system failures or irregularities.

Best Practices for Logging

1. Log at the Right Level

Logging levels help categorize the importance and purpose of log messages:

  • DEBUG: For development and troubleshooting (e.g., variable values, code flow).
  • INFO: General operational information (e.g., successful service starts).
  • WARN: Potential issues that don’t stop execution (e.g., deprecated APIs).
  • ERROR: Recoverable errors that impact functionality.
  • FATAL/CRITICAL: System-critical issues requiring immediate attention.
Example: 
DEBUG: Processing user input for login.
INFO: User successfully logged in.
WARN: API deprecated; consider updating to v2.
ERROR: Database connection failed. Retrying...
FATAL: Unable to start the application.
Enter fullscreen mode Exit fullscreen mode

2. Use Structured Logging

  • Log entries should follow a consistent format.
  • Structured logs are easier to parse, search, and analyze with tools like Elasticsearch, Splunk, or AWS CloudWatch.
Example: 

{
  "timestamp": "2024-12-19T10:45:00Z",
  "level": "INFO",
  "service": "auth-service",
  "message": "User logged in successfully",
  "userId": "12345",
  "ip": "192.168.1.1"
}
Enter fullscreen mode Exit fullscreen mode

3. Avoid Logging Sensitive Data

  • Redact or mask sensitive information like passwords, API keys, and personally identifiable information (PII).
  • Use encryption for logs that may contain sensitive data.
Example: 
WARN: Password not logged for security reasons.
Enter fullscreen mode Exit fullscreen mode

4. Centralize Logs

  • Use a centralized logging system to aggregate logs from all application components.
  • Tools for centralization:
    • Cloud-Based: AWS CloudWatch, Azure Monitor, or Google Cloud Logging.
    • Open-Source: ELK Stack (Elasticsearch, Logstash, Kibana), Fluentd, or Graylog.

5. Include Contextual Information

Logs should provide enough context to understand what happened:

  • Include user IDs, session IDs, request IDs, and timestamps.
  • Use unique request IDs for tracing logs across distributed systems.
Example: 
INFO: Processing order request. RequestID: 9876 UserID: 12345.
Enter fullscreen mode Exit fullscreen mode

6. Make Logs Machine and Human-Readable

  • Use structured formats like "J.S.O.N" for machines.
  • Add clear and descriptive messages for humans.

7. Implement Log Rotation

  • Avoid large log files by setting up rotation policies:
    • Compress old logs.
    • Delete logs after a specified retention period.
  • Use tools like Logrotate for managing log rotation.

8. Monitor Logs Actively

  • Implement automated monitoring for log patterns indicating issues (e.g., failed logins, high error rates).
  • Set up alerts to notify teams of critical issues.

9. Standardize Logging Across Services

  • Use a consistent logging library or framework across all services.
  • Define a shared logging schema for easier aggregation and analysis.

10. Test Your Logging System

  • Verify that logs provide the necessary information during troubleshooting.
  • Simulate outages or errors to ensure logs capture relevant details.

Common Logging Tools

  • Log Aggregation:
    • AWS CloudWatch, Google Cloud Logging, Elasticsearch.
  • Log Forwarding:
    • Fluentd, Logstash, Filebeat.
  • Log Analysis:
    • Kibana, Splunk, Grafana Loki.

Advanced Logging Concepts

  • Log Sampling: Reduces log volume by sampling less critical logs.
  • Distributed Tracing: Links logs from different services using tools like Jaeger or AWS X-Ray.
  • Anomaly Detection: Uses machine learning to detect unusual patterns in logs.

Task: Implement Centralized Logging for Applications Using the ELK Stack

Centralized logging helps aggregate logs from multiple sources, making it easier to search, analyze, and monitor application performance. The ELK Stack (Elasticsearch, Logstash, and Kibana) is a popular open-source solution for centralized logging.

Overview of ELK Stack Components

  1. Elasticsearch: A distributed search and analytics engine to store and index logs.
  2. Logstash: A data processing pipeline that ingests, transforms, and forwards logs to Elasticsearch.
  3. Kibana: A visualization and analytics platform for log data in Elasticsearch.

Steps to Implement Centralized Logging Using ELK Stack

1. Prerequisites

  • A Linux-based server for deploying ELK components (or use cloud-managed solutions like AWS Elasticsearch Service).
  • Sufficient memory and CPU for Elasticsearch (recommend 4GB+ RAM for small-scale use).
  • Applications configured to generate logs.

2. Install ELK Stack on a Linux Server

Step 1: Install Elasticsearch
  1. Add the Elasticsearch GPG key and repository: 

   wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
   echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
   sudo apt update
Enter fullscreen mode Exit fullscreen mode
  1. Install Elasticsearch: 

   sudo apt install elasticsearch
Enter fullscreen mode Exit fullscreen mode
  1. Start and enable Elasticsearch: 

   sudo systemctl start elasticsearch
   sudo systemctl enable elasticsearch
Enter fullscreen mode Exit fullscreen mode
Step 2: Install Logstash
  1. Install Logstash: sudo apt install logstash
  2. Verify installation: logstash --version
Step 3: Install Kibana
  1. Install Kibana: sudo apt install kibana
  2. Start and enable Kibana: sudo systemctl start kibana sudo systemctl enable kibana

3. Configure ELK Stack

Configure Elasticsearch
  • Edit the configuration file: 
  sudo nano /etc/elasticsearch/elasticsearch.yml
Enter fullscreen mode Exit fullscreen mode
  • Enable network binding for external access: 
  network.host: 0.0.0.0
Enter fullscreen mode Exit fullscreen mode
Configure Logstash
  1. Create a configuration file for Logstash: 
   sudo nano /etc/logstash/conf.d/logstash.conf
Enter fullscreen mode Exit fullscreen mode
  1. Define an input, filter, and output: 
   input {
       file {
           path => "/var/log/application/*.log"
           start_position => "beginning"
       }
   }

   filter {
       grok {
           match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{DATA:message}" }
       }
   }

   output {
       elasticsearch {
           hosts => ["http://localhost:9200"]
       }
       stdout { codec => rubydebug }
   }
Enter fullscreen mode Exit fullscreen mode
  1. Test and start Logstash: 
   sudo systemctl start logstash
   sudo systemctl enable logstash
Enter fullscreen mode Exit fullscreen mode
Configure Kibana
  • Edit Kibana configuration: 
  sudo nano /etc/kibana/kibana.yml
Enter fullscreen mode Exit fullscreen mode
  • Update the following settings: 
  server.host: "0.0.0.0"
  elasticsearch.hosts: ["http://localhost:9200"]
Enter fullscreen mode Exit fullscreen mode
  • Restart Kibana: 
  sudo systemctl restart kibana
Enter fullscreen mode Exit fullscreen mode

4. Set Up Log Forwarding

Install Filebeat
  1. Install Filebeat: 
   sudo apt install filebeat
Enter fullscreen mode Exit fullscreen mode
  1. Configure Filebeat to forward logs to Logstash: 
   sudo nano /etc/filebeat/filebeat.yml
Enter fullscreen mode Exit fullscreen mode

Add the following configuration:

   output.logstash:
       hosts: ["localhost:5044"]
Enter fullscreen mode Exit fullscreen mode
  1. Start Filebeat: 
   sudo systemctl start filebeat
   sudo systemctl enable filebeat
Enter fullscreen mode Exit fullscreen mode

5. Test the Setup

  1. Generate test logs in /var/log/application/.
  2. Check if logs are appearing in Kibana:
    • Access Kibana via http://<server-ip>:5601.
    • Create an index pattern in Kibana to view log data.

Additional Configurations

  • Retention Policy: Use Elasticsearch's ILM (Index Lifecycle Management) to manage log retention.
  • Security:
    • Enable authentication in Elasticsearch and Kibana.
    • Use HTTPS for secure log transport.
  • Scaling:
    • Use multiple nodes for Elasticsearch and Logstash for high availability.

Happy Logging!

Top comments (0)

Read next

techcreator_usa profile image

How does WebAssembly enhance web application performance?

Techcreator USA -

code42cate profile image

Day 13: Docker Multistage Builds

Jonas Scholz -

devops_descent profile image

Introduction to Amazon Simple Notification Service (SNS)

DevOps Descent -

code42cate profile image

Day 11: Docker Compose

Jonas Scholz -