What is & how did I setup Vector, Loki & Grafana in Linux Ubuntu?

#vector #loki #grafana #linux

Prerequisites:

Vector: It is observability pipeline that collects, transforms, routes logs, metrics, & traces from various sources to different destinations. It is lightweight, high-performance, & written in Rust.
Loki: Loki is a log aggregation system developed by Grafana Labs, similar to Prometheus but for logs. Loki can be used to collect application logs from multiple microservices & analyze them efficiently using Grafana dashboards.
Grafana: Grafana is an open-source analytics & monitoring platform used for visualizing & analyzing data from various sources. Grafana is commonly used to create monitoring dashboardds for system metrics, logs & application performance, integrating with Loki for log visualization.

How they work together?

Vector collects logs & metrics from applications, containers, or systems.
It processes & sends logs to Loki for storage & indexing.
Grafana queries and visualizes logs from Loki, providing insights through dashboards.

Why there is need of using such technologies?

Okay! lets imagine that there is an open book test in your college, when you start solving first question you need to search each and every page for the correct solution and there might not just a single book to find solution so it will be a tough for you to solve the test in stipulated time.

Now, let's bring in Vector, Loki & Grafana to help!

Think of Vector as a system that automatically adds bookmarks to important topics in all your books. Instead of flipping through every page, you can now quickly find the correct section based on the keywords of the question.

Now, let's say you have multiple books, but instead of searching through each one manually, you have a smart bookshelf(Loki) that groups books by subjects. For example:

Math problems go in the "Math" section.
C programming problems go in the "C" section.
Java programming problems go in the "Java" section.

So, when you need to find an answer, you don't search the whole bookshelf; you just pick the right section & look there!

Now, imagine you have a magic screen(Grafana) that, when you type your question, shows you exactly which book & page have the answer. You don't even need to check manually; you get the answer instantly!

This is exactly how logs & monitoring work in real-world applications. Instead of searching through millions of logs manually, these tools helps you find issues quickly, analyze data efficiently, & keep systems running smoothly!!

Setting up Vector, Loki & Grafana on Linux Ubuntu

Step 1: Setting up Vector

Install vector using following command:

 bash -c "$(curl -L https://setup.vector.dev)"
 sudo apt-get install vector

Verify if Vector is properly installed:
```
 vector --version
```
Start & check if the Vector is running in your local system using systemctl command
```
 sudo systemctl start vector
 sudo systemctl status vector
```

Run Vector using following command:

 sudo vector --config /path/to/vector.yaml

Note: Example configuration(vector.yaml)

# Vector Configuration

# Change this to use a non-default directory for Vector data storage:
# data_dir: "/var/lib/vector"

# Source configuration for reading logs
sources:
  my_source_id:
    type: file
    include:
      - /path/to/application.log
    read_from: beginning

# Transforms to parse logs
transforms:
  parse_file_logs:
    type: remap
    inputs: ["my_source_id"]
    source: |
      .timestamp = now()
      .message = string!(.message)
      .host = get_env_var!("HOSTNAME")
      .response_time = to_int!(.response_time)

# Sinks to output parsed logs
sinks:
  console:
    type: console
    inputs: ["parse_file_logs"]
    encoding:
      codec: json
      json:
        pretty: true
  loki:
    type: loki
    inputs: ["parse_file_logs"]
    endpoint: "http://localhost:3100"  # Change if Loki is running on another server
    Encoding:
      codec: json
      json:
        pretty: true
    labels:
      source: "vector"
      job: "application_logs"

# Vector's GraphQL API (disabled by default)
# Uncomment to enable API access at http://localhost:8686
# api:
#   enabled: true
#   address: "127.0.0.1:8686"

Step 2: Setting up Loki

Install software-properties-common

sudo apt install -y software-properties-common

Fetch grafana package key from grafana:

sudo apt-key adv --fetch-keys https://packages.grafana.com/gpg.key

Add grafana repository:

   sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

Install Loki using following command:

   sudo apt install loki

Start, enable & check if Loki is running in your system.

   sudo systemctl enable loki
   sudo systemctl start loki
   sudo systemctl status loki

Step 3: Setting up Grafana

Run the following command:

   sudo apt install -y grafana
   sudo systemctl enable --now grafana-server

Access Grafana
1. Open browser & navigate to: http://localhost:3000
2. Login with below credentials:
  - Username: admin
  - Password: admin
3. You will be asked to setup new password & then will be redirect to Grafana homepage.
Add Loki as a Data Source
1. Navigate to Configuration > Data Sources.
2. Select Loki
3. Enter http://localhost:3100
4. Click Save & Test