Loki: Getting Started

The realm of open source log aggregators has seen significant growth over the years, with key players like ElasticSearch (2010) and Graylog (2016) dominating the field. More recently, Grafana Loki (2019) has emerged as a compelling addition to this ecosystem.

This article aims to facilitate a rapid onboarding process for log shipping using Promtail, log aggregation with Loki, and data visualization through Grafana. Collectively, this powerful trio of tools is commonly referred to as the PLG (Promtail-Loki-Grafana) stack.

The Server System

These instructions will work on Debian or Ubuntu system.

Using Vagrant (Virtualbox)

If you have Intel system, you can use Vagrant with Virtualbox to quickly bring up virtual servers on your workstation. Follow the instructions from respect sites to install these tools.

When ready, create a file called Vagrantfile with the following contents:

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu2204"
  config.vm.network "forwarded_port", guest: 3000, host: 3000, host_ip: "127.0.0.1"
  config.vm.network "forwarded_port", guest: 3100, host: 3100, host_ip: "127.0.0.1"
end

NOTE: For this setup, the configuration will utilize 3000 for Grafana and 3100 for Loki on the localhost. If any services are currently running on these ports, you'll have to either halt those services or modify the host: key in the provided above configuration to use an alternative port.

When ready, you can bring up the system and log into the system with the following:

vagrant up
vagrant ssh

Installing Loki

On your desired system, such as the Vagrant managed virtual machine above, you can run the following to install PLG:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key \
  https://apt.grafana.com/gpg.key

# add a repository for stable releases
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
  | sudo tee -a /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install -y promtail loki grafana

Verify Services are running

sudo service loki status
sudo service grafana-server status

If any of these services are stopped, you can run:

sudo service loki start
sudo service grafana-server start

The Client Workstation

Visualization

You can access the login page by opening a web browser on your system and navigating to http://127.0.0.1:3000. Simply use the default credentials, admin for both the username and password. Upon login, you will be prompted to set a new password for the admin account.

After logging in, you can click on Explore, Loki, and then select on of the logs by clicking on Label.

Ship More Logs with Promtail

When setting up a new system, there may not be many interesting logs initially. However, your host workstation system is likely to have accumulated a wealth of interesting logs over time.

If you are running the server as a virtual machine using the previously mentioned Vagrantfile configuration, you can easily ship logs from the laptop to localhost:3100, where Loki is up and running. This allows you to leverage the abundant logs from your host workstation for analysis and visualization.

You can install Promtail locally as well. If the host system is running Debian or Ubuntu, you can run this:

# add a repository for stable releases
sudo wget -q -O /usr/share/keyrings/grafana.key \
  https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
  | sudo tee -a /etc/apt/sources.list.d/grafana.list

# install promtail
sudo apt-get update && sudo apt-get install -y promtail

By default, the configuration file /etc/promtail/config.yml will set up Promtail to ship logs to the URL: http://localhost:3100/loki/api/v1/push. This configuration will function correctly if you are using Vagrant with Virtualbox to run Loki. However, if you are not running the Loki server locally, you will need to modify the address (currently set to localhost) in the configuration accordingly.

As an option you can modify the configuration with the following:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      host: my_workstation
      agent: promtail
      __path__: /var/log/*log

And later restart the service:

sudo systemctl restart promtail