Setting Up a Monitoring Stack with Nginx Logging Using Grafana, cAdvisor, Promtail, Prometheus, and Loki

In this guide, I’ll walk you through setting up a comprehensive monitoring and logging stack using Docker and Docker Compose. We'll integrate Nginx logging into Grafana, using Prometheus, Loki, Promtail, and cAdvisor. These tools help you track container resource usage and visualize metrics and logs in real-time.

Why This Stack?

• Grafana: Dashboarding tool for visualizing data from various sources.
• Loki: Lightweight log aggregation system.
• Promtail: Agent to ship logs to Loki.
• Prometheus: Monitoring system that collects and processes metrics.
• cAdvisor: Analyzes and exposes resource usage for running containers.

---

Step-by-Step Setup

1. Install Grafana on Debian/Ubuntu

Install Grafana using the following commands:

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana
sudo systemctl start grafana-server

Verify Grafana by opening http://localhost:3000 in your browser.

2. Install Loki and Promtail Using Docker

First, download the Loki and Promtail configuration files:

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Next, run the containers for Loki and Promtail:

docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml
docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml

3. Install Prometheus and cAdvisor

To monitor containers with Prometheus and cAdvisor, follow these steps:

wget https://raw.githubusercontent.com/prometheus/prometheus/main/documentation/examples/prometheus.yml -O prometheus.yml
docker run -d --name=prometheus -p 9090:9090 -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml
docker run -d --name=cadvisor -p 8080:8080 gcr.io/cadvisor/cadvisor:latest

4. Configure Nginx Logging

I set up Nginx to forward its logs to Promtail and visualize them in Grafana. Here's how you can do the same:

1. Install and run Nginx:
   

   sudo apt install nginx
   sudo systemctl start nginx
   

2. Add Nginx logs to Promtail:
   Update the Promtail configuration (promtail-config.yaml) to include the Nginx logs directory, typically /var/log/nginx/*.log.

3. Restart Promtail to apply the new configuration:
   

   docker restart promtail
   

5. Set Up Grafana Dashboards

1. Add Prometheus as a data source in Grafana.
2. Add Loki as a data source for log aggregation.
3. Create custom dashboards to visualize Nginx access and error logs, along with container metrics from cAdvisor and Prometheus.

Conclusion

This stack gives you a comprehensive monitoring setup with real-time metrics and logs for your Nginx server and Docker containers. You can monitor resource usage, analyze logs, and even set alerts for performance issues.

By leveraging Grafana's powerful visualization capabilities, you get full insights into your infrastructure, ensuring everything runs smoothly.