Part II: Lean Prometheus scraper

In the previous chapter of this series we've created a EC2 instance with Prometheus remote-write collector. So today we're gonna push there some metrics with Grafana cloud agent.

At the moment there's also a possibility to do this with a Prometheus in agent mode. However I did not test this setup yet so we're gonna stick to well-supported Grafana agent.

Kube state metrics

Personally I think that KSM are the most important metrics you need to monitor the cluster. They can show you restarting pods, pending pods and all the states we're trying to avoid somehow.

All you need to do is just install KSM with helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install ksm prometheus-community/kube-state-metrics \
  --set image.tag=v2.2.0 \
  --namespace grafana-agent \
  --create-namespace

Grafana cloud agent configuration

Now it's time to prepare configuration file for the scraper. Following snippet comes from the official documentation and it basically covers KSM, cadvisor and Kubelet.

kind: ConfigMap
metadata:
  name: grafana-agent
  namespace: grafana-agent
apiVersion: v1
data:
  agent.yaml: |    
    server:
      http_listen_port: 12345
    metrics:
      wal_directory: /tmp/grafana-agent-wal
      global:
        scrape_interval: 60s
        external_labels:
          cluster: cloud
      configs:
      - name: integrations
        remote_write:
        - url: http://p01.prometheus.local:9090/api/v1/write
        scrape_configs:
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: integrations/kubernetes/cadvisor
          kubernetes_sd_configs:
              - role: node
          metric_relabel_configs:
              - source_labels: [__name__]
                regex: container_network_transmit_packets_total|kubelet_certificate_manager_server_ttl_seconds|storage_operation_duration_seconds_bucket|node_namespace_pod_container:container_memory_swap|container_fs_writes_total|container_network_receive_bytes_total|kube_daemonset_status_desired_number_scheduled|cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits|rest_client_requests_total|node_namespace_pod_container:container_memory_working_set_bytes|kubernetes_build_info|kube_node_status_capacity|kubelet_pleg_relist_duration_seconds_bucket|kubelet_running_pods|storage_operation_errors_total|kubelet_running_containers|kube_daemonset_status_number_misscheduled|kube_job_failed|kube_statefulset_status_replicas|kube_job_status_succeeded|container_cpu_cfs_throttled_periods_total|kube_statefulset_status_update_revision|process_resident_memory_bytes|kubelet_pod_start_duration_seconds_count|kubelet_running_container_count|container_fs_writes_bytes_total|machine_memory_bytes|kubelet_cgroup_manager_duration_seconds_count|node_namespace_pod_container:container_memory_rss|kubelet_node_config_error|kubelet_runtime_operations_duration_seconds_bucket|kubelet_pleg_relist_interval_seconds_bucket|kube_job_spec_completions|kube_statefulset_status_current_revision|kube_statefulset_replicas|cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests|kubelet_node_name|kubelet_pod_worker_duration_seconds_bucket|go_goroutines|kubelet_volume_stats_capacity_bytes|kube_horizontalpodautoscaler_status_current_replicas|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate|kubelet_runtime_operations_errors_total|kube_daemonset_status_number_available|kube_deployment_status_replicas_available|up|storage_operation_duration_seconds_count|kube_daemonset_status_current_number_scheduled|kube_statefulset_status_replicas_updated|kube_node_status_condition|kube_node_status_allocatable|rest_client_request_duration_seconds_bucket|container_cpu_usage_seconds_total|namespace_workload_pod:kube_pod_owner:relabel|kubelet_pleg_relist_duration_seconds_count|kube_pod_owner|namespace_cpu:kube_pod_container_resource_requests:sum|kube_horizontalpodautoscaler_spec_max_replicas|kube_statefulset_status_replicas_ready|container_fs_reads_total|node_namespace_pod_container:container_memory_cache|container_network_transmit_packets_dropped_total|kubelet_volume_stats_inodes_used|kube_node_spec_taint|cluster:namespace:pod_memory:active:kube_pod_container_resource_requests|kube_pod_info|kubelet_cgroup_manager_duration_seconds_bucket|process_cpu_seconds_total|container_memory_cache|kube_statefulset_metadata_generation|kubelet_pod_worker_duration_seconds_count|volume_manager_total_volumes|namespace_cpu:kube_pod_container_resource_limits:sum|kube_deployment_metadata_generation|kube_replicaset_owner|container_memory_swap|kubelet_certificate_manager_client_ttl_seconds|kube_resourcequota|container_fs_reads_bytes_total|kubelet_runtime_operations_total|kube_horizontalpodautoscaler_status_desired_replicas|kube_pod_status_phase|kube_horizontalpodautoscaler_spec_min_replicas|kubelet_server_expiration_renew_errors|kube_pod_container_resource_limits|container_network_transmit_bytes_total|container_network_receive_packets_dropped_total|container_memory_working_set_bytes|kube_pod_container_status_waiting_reason|container_network_receive_packets_total|kube_namespace_created|namespace_workload_pod|kube_pod_container_resource_requests|kubelet_running_pod_count|namespace_memory:kube_pod_container_resource_limits:sum|kube_deployment_status_replicas_updated|kube_statefulset_status_observed_generation|kube_deployment_status_observed_generation|container_cpu_cfs_periods_total|node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile|kubelet_certificate_manager_client_expiration_renew_errors|cluster:namespace:pod_memory:active:kube_pod_container_resource_limits|kube_daemonset_updated_number_scheduled|kubelet_volume_stats_inodes|kube_node_info|kube_deployment_spec_replicas|container_memory_rss|namespace_memory:kube_pod_container_resource_requests:sum|kubelet_volume_stats_available_bytes
                action: keep
          relabel_configs:
              - replacement: kubernetes.default.svc.cluster.local:443
                target_label: __address__
              - regex: (.+)
                replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
                source_labels:
                  - __meta_kubernetes_node_name
                target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: integrations/kubernetes/kubelet
          kubernetes_sd_configs:
              - role: node
          metric_relabel_configs:
              - source_labels: [__name__]
                regex: container_network_transmit_packets_total|kubelet_certificate_manager_server_ttl_seconds|storage_operation_duration_seconds_bucket|node_namespace_pod_container:container_memory_swap|container_fs_writes_total|container_network_receive_bytes_total|kube_daemonset_status_desired_number_scheduled|cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits|rest_client_requests_total|node_namespace_pod_container:container_memory_working_set_bytes|kubernetes_build_info|kube_node_status_capacity|kubelet_pleg_relist_duration_seconds_bucket|kubelet_running_pods|storage_operation_errors_total|kubelet_running_containers|kube_daemonset_status_number_misscheduled|kube_job_failed|kube_statefulset_status_replicas|kube_job_status_succeeded|container_cpu_cfs_throttled_periods_total|kube_statefulset_status_update_revision|process_resident_memory_bytes|kubelet_pod_start_duration_seconds_count|kubelet_running_container_count|container_fs_writes_bytes_total|machine_memory_bytes|kubelet_cgroup_manager_duration_seconds_count|node_namespace_pod_container:container_memory_rss|kubelet_node_config_error|kubelet_runtime_operations_duration_seconds_bucket|kubelet_pleg_relist_interval_seconds_bucket|kube_job_spec_completions|kube_statefulset_status_current_revision|kube_statefulset_replicas|cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests|kubelet_node_name|kubelet_pod_worker_duration_seconds_bucket|go_goroutines|kubelet_volume_stats_capacity_bytes|kube_horizontalpodautoscaler_status_current_replicas|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate|kubelet_runtime_operations_errors_total|kube_daemonset_status_number_available|kube_deployment_status_replicas_available|up|storage_operation_duration_seconds_count|kube_daemonset_status_current_number_scheduled|kube_statefulset_status_replicas_updated|kube_node_status_condition|kube_node_status_allocatable|rest_client_request_duration_seconds_bucket|container_cpu_usage_seconds_total|namespace_workload_pod:kube_pod_owner:relabel|kubelet_pleg_relist_duration_seconds_count|kube_pod_owner|namespace_cpu:kube_pod_container_resource_requests:sum|kube_horizontalpodautoscaler_spec_max_replicas|kube_statefulset_status_replicas_ready|container_fs_reads_total|node_namespace_pod_container:container_memory_cache|container_network_transmit_packets_dropped_total|kubelet_volume_stats_inodes_used|kube_node_spec_taint|cluster:namespace:pod_memory:active:kube_pod_container_resource_requests|kube_pod_info|kubelet_cgroup_manager_duration_seconds_bucket|process_cpu_seconds_total|container_memory_cache|kube_statefulset_metadata_generation|kubelet_pod_worker_duration_seconds_count|volume_manager_total_volumes|namespace_cpu:kube_pod_container_resource_limits:sum|kube_deployment_metadata_generation|kube_replicaset_owner|container_memory_swap|kubelet_certificate_manager_client_ttl_seconds|kube_resourcequota|container_fs_reads_bytes_total|kubelet_runtime_operations_total|kube_horizontalpodautoscaler_status_desired_replicas|kube_pod_status_phase|kube_horizontalpodautoscaler_spec_min_replicas|kubelet_server_expiration_renew_errors|kube_pod_container_resource_limits|container_network_transmit_bytes_total|container_network_receive_packets_dropped_total|container_memory_working_set_bytes|kube_pod_container_status_waiting_reason|container_network_receive_packets_total|kube_namespace_created|namespace_workload_pod|kube_pod_container_resource_requests|kubelet_running_pod_count|namespace_memory:kube_pod_container_resource_limits:sum|kube_deployment_status_replicas_updated|kube_statefulset_status_observed_generation|kube_deployment_status_observed_generation|container_cpu_cfs_periods_total|node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile|kubelet_certificate_manager_client_expiration_renew_errors|cluster:namespace:pod_memory:active:kube_pod_container_resource_limits|kube_daemonset_updated_number_scheduled|kubelet_volume_stats_inodes|kube_node_info|kube_deployment_spec_replicas|container_memory_rss|namespace_memory:kube_pod_container_resource_requests:sum|kubelet_volume_stats_available_bytes
                action: keep
          relabel_configs:
              - replacement: kubernetes.default.svc.cluster.local:443
                target_label: __address__
              - regex: (.+)
                replacement: /api/v1/nodes/${1}/proxy/metrics
                source_labels:
                  - __meta_kubernetes_node_name
                target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - job_name: integrations/kubernetes/kube-state-metrics
          kubernetes_sd_configs:
              - role: service
          metric_relabel_configs:
              - source_labels: [__name__]
                regex: container_network_transmit_packets_total|kubelet_certificate_manager_server_ttl_seconds|storage_operation_duration_seconds_bucket|node_namespace_pod_container:container_memory_swap|container_fs_writes_total|container_network_receive_bytes_total|kube_daemonset_status_desired_number_scheduled|cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits|rest_client_requests_total|node_namespace_pod_container:container_memory_working_set_bytes|kubernetes_build_info|kube_node_status_capacity|kubelet_pleg_relist_duration_seconds_bucket|kubelet_running_pods|storage_operation_errors_total|kubelet_running_containers|kube_daemonset_status_number_misscheduled|kube_job_failed|kube_statefulset_status_replicas|kube_job_status_succeeded|container_cpu_cfs_throttled_periods_total|kube_statefulset_status_update_revision|process_resident_memory_bytes|kubelet_pod_start_duration_seconds_count|kubelet_running_container_count|container_fs_writes_bytes_total|machine_memory_bytes|kubelet_cgroup_manager_duration_seconds_count|node_namespace_pod_container:container_memory_rss|kubelet_node_config_error|kubelet_runtime_operations_duration_seconds_bucket|kubelet_pleg_relist_interval_seconds_bucket|kube_job_spec_completions|kube_statefulset_status_current_revision|kube_statefulset_replicas|cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests|kubelet_node_name|kubelet_pod_worker_duration_seconds_bucket|go_goroutines|kubelet_volume_stats_capacity_bytes|kube_horizontalpodautoscaler_status_current_replicas|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate|kubelet_runtime_operations_errors_total|kube_daemonset_status_number_available|kube_deployment_status_replicas_available|up|storage_operation_duration_seconds_count|kube_daemonset_status_current_number_scheduled|kube_statefulset_status_replicas_updated|kube_node_status_condition|kube_node_status_allocatable|rest_client_request_duration_seconds_bucket|container_cpu_usage_seconds_total|namespace_workload_pod:kube_pod_owner:relabel|kubelet_pleg_relist_duration_seconds_count|kube_pod_owner|namespace_cpu:kube_pod_container_resource_requests:sum|kube_horizontalpodautoscaler_spec_max_replicas|kube_statefulset_status_replicas_ready|container_fs_reads_total|node_namespace_pod_container:container_memory_cache|container_network_transmit_packets_dropped_total|kubelet_volume_stats_inodes_used|kube_node_spec_taint|cluster:namespace:pod_memory:active:kube_pod_container_resource_requests|kube_pod_info|kubelet_cgroup_manager_duration_seconds_bucket|process_cpu_seconds_total|container_memory_cache|kube_statefulset_metadata_generation|kubelet_pod_worker_duration_seconds_count|volume_manager_total_volumes|namespace_cpu:kube_pod_container_resource_limits:sum|kube_deployment_metadata_generation|kube_replicaset_owner|container_memory_swap|kubelet_certificate_manager_client_ttl_seconds|kube_resourcequota|container_fs_reads_bytes_total|kubelet_runtime_operations_total|kube_horizontalpodautoscaler_status_desired_replicas|kube_pod_status_phase|kube_horizontalpodautoscaler_spec_min_replicas|kubelet_server_expiration_renew_errors|kube_pod_container_resource_limits|container_network_transmit_bytes_total|container_network_receive_packets_dropped_total|container_memory_working_set_bytes|kube_pod_container_status_waiting_reason|container_network_receive_packets_total|kube_namespace_created|namespace_workload_pod|kube_pod_container_resource_requests|kubelet_running_pod_count|namespace_memory:kube_pod_container_resource_limits:sum|kube_deployment_status_replicas_updated|kube_statefulset_status_observed_generation|kube_deployment_status_observed_generation|container_cpu_cfs_periods_total|node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile|kubelet_certificate_manager_client_expiration_renew_errors|cluster:namespace:pod_memory:active:kube_pod_container_resource_limits|kube_daemonset_updated_number_scheduled|kubelet_volume_stats_inodes|kube_node_info|kube_deployment_spec_replicas|container_memory_rss|namespace_memory:kube_pod_container_resource_requests:sum|kubelet_volume_stats_available_bytes
                action: keep
          relabel_configs:
              - action: keep
                regex: ksm-kube-state-metrics
                source_labels:
                  - __meta_kubernetes_service_name

You can see that it's pretty much the same scheme as the Prometheus configuration has. So if you need to adjust some metrics - check the official Prometheus documentation.

Now you can just apply the configuration:

kubectl apply -f config.yaml

Grafana cloud agent deployment

When we have the configuration in place, we can proceed to the last bit - Deployment with the Grafana agent. Again, everything's described in the official documentation but this is what you need to do:

MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/main/production/kubernetes/agent-bare.yaml \
NAMESPACE=grafana-agent \
/bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install-bare.sh)" > deploy.yaml

and apply the manifest:

kubectl apply -f deploy.yaml

As the result we have 2 pods running in the grafana-agent namespace.

NAME                                      READY   STATUS    RESTARTS   AGE
grafana-agent-6f8b68fd6-hbnkr             1/1     Running   0          6d16h
ksm-kube-state-metrics-7d8f59c464-tb9st   1/1     Running   0          10d

Result

And guess what, now we have metrics stored in the Prometheus! If you're wondering why I communicate with localhost:9090 - check the previous chapter. This is AWS SSM port-forwarding functionality that is really useful for some basic debugging.

Wrap

So now we have the complete monitoring stack. But do we really need to use some strange port-forwarding to get there? Not really. The golden standard for the viewing of metrics is the Grafana and this is what we're gonna do in the next chapter. Stay tuned!