DEV Community

Stepan Vrany
Stepan Vrany

Posted on

Part III: Grafana

So now we have the Prometheus up and running and we even send some metrics there thanks to Grafana cloud agent. How to view these metrics in some more convenient way?

Yes, the answer is Grafana, the best in class tool for the job. And since we have a Kubernetes cluster - we're gonna run it inside this cluster. Grafana is not really resource-intensive beast and this will be also cheaper than managed services like Amazon Managed Grafana.


Grafana needs some persistent store for users, dashboards and other stuff. We can use Persistent Volume Claim but again - we want to keep our cluster lean. So our next choice is RDS in its smallest configuration.

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "random_password" "password" {
  length  = 32
  special = false

resource "aws_security_group" "this" {
  name   = "grafana_${var.environment}"
  vpc_id = var.vpc_id

resource "aws_security_group_rule" "egress_all" {
  type     = "egress"
  to_port  = 0
  protocol = "-1"
  cidr_blocks = [
  from_port         = 0
  security_group_id =

resource "aws_security_group_rule" "ingress_posgres" {
  type                     = "ingress"
  to_port                  = 5432
  protocol                 = "tcp"
  source_security_group_id = var.source_security_group_id
  from_port                = 5432
  security_group_id        =

resource "aws_db_subnet_group" "this" {
  name       = "grafana-${var.environment}"
  subnet_ids = var.subnet_ids

resource "aws_db_instance" "this" {
  availability_zone    = var.availability_zone
  allocated_storage    = 10
  engine               = "postgres"
  engine_version       = var.postgres_engine_version
  instance_class       = var.postgres_instance_class
  identifier           = "grafana-${var.environment}"
  db_name              = "grafana"
  username             = "root"
  password             = random_password.password.result
  skip_final_snapshot  = true
  db_subnet_group_name =
  vpc_security_group_ids = [,
Enter fullscreen mode Exit fullscreen mode

You can see here some similarities with the EC2 instance from the first chapter. Once again we've created a separate Security Group with the only ingress rule allowing the communication only from the Kubernetes cluster.

Single sign-on

Grafana supports variety of single sign-on options, in this writeup we're gonna use Gitlab but you can use Github, some generic OIDC solution or the internal database.

Installation of Grafana

Let's prepare values files for the Grafana Helm chart.

replicas: 1

    domain: grafana.<your domain name>
    root_url: https://grafana.<your domain name>/
    enforce_domain: true
    protocol: http
    enabled: false
    type: postgres
    host: <RDS endpoint from the previous steps>
    user: root
    password: <RDS password from the previous steps>
    name: grafana
    ssl_mode: require
    max_open_conn: 25
    max_idle_conn: 25
    enabled: true
    enabled: false
    enabled: false
    auto_assign_org: true
    enabled: true
    allow_sign_up: true
    client_id: <id from the Gitlab app>
    client_secret: <secret from the Gitlab app>
    scopes: read_api
    allowed_groups: <your Gitlab group>

  enabled: false
Enter fullscreen mode Exit fullscreen mode

Note the ingress.enabled property. In this particular infrastructure we're using Traefik so we don't need to create Ingress object. Instead I've created a simple IngressRoute resource:

kind: IngressRoute
  name: grafana
  namespace: grafana
    - websecure
  - match: Host(`grafana.<your domain name>`) && PathPrefix(`/`)
    kind: Rule
    priority: 1
    - name: grafana
      port: 80
      scheme: http
Enter fullscreen mode Exit fullscreen mode

If you're using Nginx ingress controller, then you just need to re-enable it in the values:

  enabled: true
  ingressClassName: nginx
  annotations: {} nginx
  labels: {}
  path: /
  pathType: Prefix
    - grafana.<your domain name>
Enter fullscreen mode Exit fullscreen mode

You know the drill I guess. Let's install this with simple Helm command:

helm repo add grafana
helm repo update
helm upgrade --install grafana grafana/grafana -n grafana -f values.yaml --version 6.21.4
Enter fullscreen mode Exit fullscreen mode

And that's it. For the administration tasks we just need one more thing - admin password:

kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Enter fullscreen mode Exit fullscreen mode

Adding Prometheus data source to Grafana

This is perhaps the most straightforward task, just log-in, open datasources section in settings and fill in the domain name we've defined in the first chapter.

Image description


And that's it! In the next chapter I'll show you how to forward alertmanager notifications to Telegram group. Stay tuned!

Top comments (0)