DEV Community: Chris

Per Service CPU Usage Monitoring - A Job For Prometheus & Grafana

Chris — Wed, 11 Nov 2026 04:31:47 +0000

Recently we had a situation where it would have been good to have had per service CPU usage graphed over time.

Some of our servers are run in a 'multi tenant' mode where we deliver a service to 20 different customers, with each customer using their own dedicated systemd service. Some of these customers use the service quite aggressively while others are well behaved.

Recently we restarted one of the services of an 'aggressive' customer in order to realize a new configuration and noticed that the overall CPU usage on the server dropped in half. This observation spawned a lot of questions...

Was this one service accounting for 50% of the overall CPU usage on the server?
Was the new configuration more efficient in some way?
Could it be coincidence that this service restart resulted in less CPU consumption?
Did the CPU usage for this particular service increase over time, or had it always been like this?

The server had a lot of CPU headroom, there was no immediate danger, yet it felt like there was a tiger in the bushes waiting for us. Suddenly we felt the need to increase our situational awareness.

Our monitoring server graphed total CPU usage, but had no notion of 'per process CPU usage'. If we had a way to visualize each customers CPU utilization it could help answer a lot of questions.

So! How to get this situational awareness? After discussing with our staff, searching the internet, and posing the question to our AI agents, a solution using Prometheus for data collection and Grafana for visualization was selected.

10,000 Foot View

A prometheus agent runs on the customer facing server that exposes per process related statistics
Prometheus collector runs on the monitoring server, polls the customer facing server and updates a data store
Grafana queries the prometheus datastore and renders time series graphs

Multi Tenant Server Provisioning

First we installed prometheus-process-exporter on the multi-tenant server. By default the prometheus-process-exporter service has no configuration. The only configuration related file installed by default is /etc/default/prometheus-process-exporter and looks like this:

# Set the command-line arguments to pass to the server.
# Due to shell escaping, to pass backslashes for regexes, you need to double
# them (\\d for \d). If running under systemd, you need to double them again
# (\\\\d to mean \d), and escape newlines too.
ARGS=""

We need to update this file to listen on a TCP socket and point it to a configuration file. The configuration file uses the yaml format. Here is an example we placed in /etc/prometheus/process-exporter.yml (the /etc/prometheus directory may need to be created):

process_names:
  - name: "{{.Cgroups}}"
    comm:
      - rust-widget

In our case every customer runs the same binary for their service, the binary name being rust-widget. The Cgroups argument will name each group after the seen service name, each systemd service gets its own group and children forked by it are associated with it.

Update the ARGS declaration in /etc/default/prometheus-process-exporter:

ARGS="--config.path=/etc/prometheus/process-exporter.yml --web.listen-address=10.10.10.10:9256"

Then restart the prometheus-process-exporter service to realize the change

Monitoring Server Provisioning

Next we need to configure the monitoring server to poll this new prometheus endpoint. To do this we'll add a 'job' to /etc/prometheus/prometheus.yml. This job goes in the scrape_configs section:

  - job_name: multi-tenant-1
    scrape_interval: 1m
    scrape_timeout: 1m
    static_configs:
      - targets: ['10.10.10.10:9256']
    metric_relabel_configs:
      - source_labels: [groupname]
        regex: '.*/([^/]+rust-widget)\.service.*'
        target_label: service
        replacement: '$1'

The metric_relabel_configs section is worth talking about. In our case all our customer facing services have a naming convention of <customer>_rust-widget. Since we are only interested in CPU usage information of these services we use a regex that matches that.

After changing this file restart prometheus to realize it. We can verify the state of the new endpoint by having a look at the prometheus web server. the default web server port for prometheus is 9090. The path that provides endpoint status is: http://127.0.0.1:9090/classic/targets. If prometheus is able to query the new endpoint and things are working well the endpoint State will be 'UP'.

Now, how to visualize this? Either create or edit an existing dashboard, then add a time series panel. Select the prometheus data source (we made the assumption that prometheus is already configured as a data source for the grafana instance). We used Claude to recommend a ProMQ query as most humans have little interest in learning this query language. Claude returned this for a per job graph and it worked as desired strait away:

sum by (service) (rate(namedprocess_namegroup_cpu_seconds_total{job="multi-tenant-1", service=~".+"}[5m]))

The resulting graph looks like this:

The next time we observe some odd change in CPU consumption we will be better positioned to know why. Reducing the tigers in the bushes one at a time...

Zero Day Protection - NGINX web server example

Chris — Thu, 05 Nov 2026 10:49:57 +0000

The recent PHP vulnerability got me thinking about how to prevent any kind of HTTP exploit be it known or not. In order for an attacker to exploit a weakness they must discover it. In order to discover an exploitable service the attacker much send a lot of probes. we see these probes on any web server that's open to the internet 24/7.

The web server at my home is constantly, shamelessly groped.

This inspired me to write a script that tails the web server log and takes action based on configurable attributes.

If we see three requests resulting in HTML status codes that have been designated 'not normal' for this web server, we signal Netfilter to block any traffic from the source IP of the request.
If we see any request from a HTTP user agent we've deemed unworthy we signal Netfilter to block any traffic from the source IP of the request. Seriously, would any human be using a user agent such as?:

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot"

Dropping the traffic at layer 3 appeals to me as we prevent any other buffoonery from this IP address. If we send and ICMP unreachable back, this consumes our resources. By dropping it the attacker knows not if it's just a slow connection or a non-existent server and will wait till it times out, consuming their resources.

The script goes like this:

Tail the log
Use a 'while' loop to process each new log message
The NGINX log was converted to a JSON format in order to have a predictable data structure. This is very helpful for scripts that ingest it. We realize the benefit of the data structure by loading log message with the python3 json module
Next we evaluate the log data:
- if the request was from a trusted network stop processing it, we are OK with any request from a trusted network.
- update a hash of clients that keeps track of each public IP address, status codes, and number of times the status code was returned.
- If the user agent contains a predefined term that's deemed unsavory signal Netfilter to block the public IP address of the request
- if the returned status codes is in a predefined list and exceeds a predefined number of occurrences block the public IP address of the request.

This routine looks like this:

while True:
    logLine = logTail.stdout.readline()
    logLine = logLine.decode(errors='ignore').strip()
    logHash = json.loads(logLine)
    clientIp = logHash['remote_addr']
    status = logHash['status']
    trust = checkAddress(clientIp)
    if trust == 'untrust':
        print('untrusted')
        if clientIp not in clientHash:
            clientHash[clientIp] = {}
        if status not in clientHash[clientIp]:
            clientHash[clientIp][status] = 0
        clientHash[clientIp][status] += 1
        if logHash['http_user_agent'] == '' or logHash['http_user_agent'] == '-':
            print('stupid user agent')
            doBlock(clientIp)
        for ua in unwantedAgents:
            if ua in logHash['http_user_agent']:
                print('blocking unwanted user agent')
                doBlock(clientIp)
        if status in blockableStatusCodes:
            if clientHash[clientIp][status] >= blockThreshold and clientIp not in blockedSet:
                print('strike ' + str(clientHash[clientIp][status]) + '!')
                doBlock(clientIp)
            else:
                print('strike ' + str(clientHash[clientIp][status]) + '!')

The complete script may be found here:
https://gist.github.com/netdsg/cc338230a46f8a01c1b25983d5935143

NGINX Log Taming - Structured Data & Statistical Analysis

Chris — Sun, 25 Oct 2026 00:42:54 +0000

Logs are great! They provide a lot of information that assists in understanding the current environment and provide insight when trying to solve problems. It seems every application has its own logging style. Some log to a file, some log to the system journal, some log to standard out, etc. In addition to the variance in where they log, they generally have their own logging format. Some attempt to log in plain English while some log using application specific codes that must be cross referenced in order to understand what is happening.

At some point logging collided with the concept of structured data and their was much rejoicing! Suddenly we could shelve our custom parsers, as proud of them as we were, and enjoy logs that have predictable key/value pairs. The most common way applications offer logs in a structured data format is via JSON output.

Not only are JSONified logs wonderful for scripts that ingest them, when we are hacking for our lives during an outage we can pipe the log file through a utility such as jq and find inspiration to keep living during dark times.

Now, on to the namesake of this article, NGINX! A lot of readers here may admin NGINX servers that produce giga bytes of logs every day. processing large logs for meaningful information can be challenging. NGINX has an option to change the logging format to JSON versus the difficult to read/parse default format. This maneuver involves creating a custom log_format and then associating it with the access_log.

The custom log_format definition must be in the /etc/nginx/nginx.conf file, while the access_log directive that invokes it may be in the same file for global effect, or in an a specific site file for more granular control of logs.

The log_format definition belongs in the http context of the /etc/nginx/nginx.conf file. Here is an example custom log_format that is named custom_json:

http {
    <-- snip -->

    log_format custom_json escape=json '{'
        '"time_local":"$time_local",'
        '"remote_addr":"$remote_addr",'
        '"host":"$host",'
        '"request":"$request",'
        '"status": "$status",'
        '"body_bytes_sent":"$body_bytes_sent",'
        '"request_time":"$request_time",'
        '"http_referrer":"$http_referer",'
        '"http_user_agent":"$http_user_agent"'
    '}';

    <-- snip -->
}

Next we invoke the custom parser, this can be defined in the http, server, or location contexts. Here is an invocation example:

access_log /var/log/nginx/access.log custom_json;
Once these changes have been made, restart the service, and have a look at the new log format. Now a line in the log file looks like this:

root@pagoda:/var/log/nginx# tail -1 access.log
{"time_local":"25/Oct/2026:08:30:50 +0900","remote_addr":"192.168.82.1","host":"activebit.net","request":"GET / HTTP/1.0","status": "200","body_bytes_sent":"65","request_time":"0.000","http_referrer":"","http_user_agent":""}

This is arguably harder on the eyes than the default format, but if we pipe it to a utility such as jq in renders in this way:

root@pagoda:/var/log/nginx# tail -1 access.log | jq .
{
  "time_local": "25/Oct/2026:08:30:50 +0900",
  "remote_addr": "192.168.82.1",
  "host": "activebit.net",
  "request": "GET / HTTP/1.0",
  "status": "200",
  "body_bytes_sent": "65",
  "request_time": "0.000",
  "http_referrer": "",
  "http_user_agent": ""
}

This is much easier to read and quickly understand what all the parts of the log message mean. Since it is structured data we do not have to write a custom parser that's prone to errors in order to programmatically analyze the data. Here is an example using python3 of how to quickly get statistics for a given key seen in the log file:

#!/usr/bin/env python3

import json
import sys 
from pprint import pprint

subjFile = sys.argv[1]
keyName = sys.argv[2]

def getFlatFile(ourFile):
    hook = open(ourFile, errors='ignore')
    raw = hook.read()
    hook.close()
    return raw.splitlines()

logList = getFlatFile(subjFile)

statHash = {}
linesProcessed = 0 
for l in logList:
    l = json.loads(l)
    linesProcessed += 1
    if l[keyName] not in statHash:
        statHash[l[keyName]] = 0 
    statHash[l[keyName]] += 1

print('stats for key: ' + keyName)
for seenKey in sorted(statHash):
    percent = round((statHash[seenKey] / linesProcessed) * 100)
    print(seenKey + ' occurances: ' + str(statHash[seenKey]) + ' percent: ' + str(percent) + '%')

This is an example of evaluating the 'status' key value:

root@pagoda:/home/chris/dev.to# ./json_log_eval.py /var/log/nginx/access.log status
stats for key: status
200 occurances: 61 percent: 29%
400 occurances: 11 percent: 5%
403 occurances: 9 percent: 4%
404 occurances: 126 percent: 60%
405 occurances: 1 percent: 0%
444 occurances: 1 percent: 0%

Fear No Log File!

Network Traffic Awareness

Chris — Thu, 22 Oct 2026 10:41:40 +0000

We have many connected devices such as phones, watches, tablets, laptops, cameras, refrigerators, air conditioners, etc; yet do we know who these devices are communicating with or how? Shouldn't we know? This train of thought inspired the 'Traffic Evaluator' project.

We can know what internet hosts our devices are communicating with and how.

Recording the network traffic and analyzing it reveals who our devices are communicating with. This also gives us insight into how much our ISP can know about us. Any organization in the 'network path' can easily harvest all this information.

In order to demonstrate the depth of information that is easily available through packet analysis we need a test point between the user and the internet. Providing a VPN tunnel with a respected protocol such as WireGuard allows this tool to have such a test point.

If you are interested the 'Traffic Evaluator' project is at https://activebit.net/te . This site allows a user to download and install a WireGuard tunnel configuration where all the traffic is under surveillance and a usage report can be generated at the users request. The report contains details such as:

Organizations the connected device is communicating with
Countries where the remote servers are located
Second Level Domain Summary
FQDNs seen in dns requests, TLS & QUIC Headers
Understanding of what protocols are in use

Peer Review - Another Dimension to Consider

Chris — Wed, 21 Oct 2026 08:38:31 +0000

When people think of peer review it is often associated with reducing risk for upcoming changes. While this is true there is another dimension that deserves consideration.

When I supported financial institutions and ISPs change control was very serious and peer review was part of that process. I found it very insightful as I learned much about my coworkers through the review. The questions someone poses reveals much more than their answers. The peer review process created an environment that allowed us to better understand our coworkers and suddenly we could realize and respect the depth of their experience, insight, and ingenuity in a real way. This process creates TEAM!

My path is somewhat opposite of others in that I moved from 'stable' organizations to startups. I found that change control and consequently peer review was not much wanted in startups. In some cases I found this rational as a fellow in a startup may be a one person department and conducting peer review with oneself is tiresome.

When there is more than one person working in the same space or concern startups often avoid peer review in the name of speed and agility; DO NOT SLOW ME DOWN!

I argue waiting a day will never kill a project and the respect that can be developed during the peer review process will in the long term be much more valuable than any 'shoot from the hip' change. Peer review is an opportunity to create a team in a natural way. Take advantage of this and become a stronger organization.