In my last post, I shared how I moved from blindly running commands to seeing my infrastructure breathe through metrics. But as every DevOps learner quickly discovers, metrics tell you there is a problem; logs tell you what the problem is.
Today, we’re giving our "Linux Health Sentinel" the ability to listen. We are adding centralised logging using Loki and Promtail.
The Concept: Metrics vs. Logs
If your server’s CPU spikes to 99%, Prometheus will show you a scary red line on a graph. That’s a metric. But why did it spike? Was it a brute-force SSH attack? A memory leak in a script?
To find out, you need the text records, the logs.
The Architecture
We are adding two new components to our existing setup:
- Loki (The Library): Lives on your laptop. It stores the logs and lets you search them.
- Promtail (The Spy): Lives on the Vagrant VM. It "tails" the log files (like tail -f) and ships them to Loki.
Prerequisites
- Ubuntu laptop with Grafana + Prometheus running
- Vagrant VM from Phase 1 (or any local VM of your choice)
- Basic networking between host and VM
Step 1: Setting up the Library (Loki)
On your control centre (laptop), we need to get Loki running. Loki is "Prometheus, but for logs." While some package managers have Loki, the safest and most consistent path is using the official binaries.
# Download and unzip Loki
wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64.zip
sudo apt update && sudo apt install unzip -y
unzip loki-linux-amd64.zip
chmod +x loki-linux-amd64
# Download the default config file
wget https://raw.githubusercontent.com/grafana/loki/main/cmd/loki/loki-local-config.yaml
Run Loki:
./loki-linux-amd64 -config.file=loki-local-config.yaml
PS: This setup is for local learning only and runs without authentication. Do not expose Loki directly to the internet.
Step 2: Deploying the Spy (Promtail)
Now, hop into your Vagrant VM. We need an agent to grab those system logs and send them over the network to your laptop.
- Install Promtail:
curl -O -L "https://github.com/grafana/loki/releases/download/v3.5.9/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
chmod a+x promtail-linux-amd64
- Configure Promtail: Download the basic config file to tell Promtail where to send the logs.
wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml
- Edit the Config: Change the clients URL to your laptop's IP address. Tip: Use hostname -I on your laptop to find the IP address your VM needs to talk to.
clients:
- url: http://<YOUR_LAPTOP_IP>:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
host: vagrant-vm
__path__: /var/log/*log
- Run Promtail:
./promtail-linux-amd64 -config.file=promtail-local-config.yaml
Step 3: Visualisation in Grafana
- Go to Grafana (localhost:3000).
- Add Data Source -> Select Loki.
- Set URL to http://localhost:3100. Click Save & Test.
- Go to the Explore tab (compass icon).
- Use the Label Browser to select job="varlogs" or host="vagrant-vm".
- Click Run Query.
Generating Test Logs
You should get some logs. We can also generate some noise on the VM:
sudo logger "Sentinel Test: Can you hear me, Grafana?"
sudo logger "Hello Loki, this is a test"
sudo logger "Sentinel Alert: Testing log flow to Grafana"
sudo logger -p user.err "Simulating a critical system error"
sudo logger "Hello Loki, this is test-2."
Don't expect a "Matrix-style" scrolling screen immediately! By default, Grafana shows a static snapshot. To see logs fly across your screen in real-time:
- Enable "Live" Mode: Look for the Live button in the top right of the Grafana UI.
- Adjust Auto-Refresh: Set the timer to 5s or 10s.
Conclusion
We have metrics, and we have logs. Our Sentinel is getting smarter. But we still have to look at the screen to know something is wrong.
Next, we’ll teach it to speak to us on Slack or Discord or email when it detects trouble, with Alerting.
Top comments (0)