DEV Community: Ritom Puzari

See every API your server calls (and every call it receives) without changing a line of code

Ritom Puzari — Sat, 19 Sep 2026 21:45:20 +0000

Instrumenting an application for APM means picking a vendor SDK, adding it to every service, redeploying, and hoping the framework version is supported. There is an older trick that gets you most of the value with none of that: listen to the network interface.

What the host can see

A raw socket opened with socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL)) on Linux receives a copy of every frame that crosses any interface, the same tap tcpdump uses. On its own that is expensive on a busy box, so the first thing to do is attach a classic BPF program with setsockopt(SO_ATTACH_FILTER). The nine-instruction program that tcpdump -dd tcp prints accepts IPv4 and IPv6 frames whose next header is TCP and rejects everything else inside the kernel, before a single byte is copied to user space. The agent uses exactly that program, so UDP, ARP and ICMP never reach Python.

What remains is a stream of TCP segments. You do not need full stream reassembly to get useful data, because the interesting bytes sit at the start of a segment: an HTTP/1.x request line begins a client segment, a status line begins a server segment, and a TLS ClientHello is the first record on a fresh connection. Keying by the 4-tuple (source address, source port, destination address, destination port) and pairing a request with the next response on the reversed tuple gives you a latency sample per request. From that you can reconstruct:

Plain HTTP/1.x: method, path (query string stripped), Host, status code, bytes each way, and the time from the request segment to the first byte of the response. That is enough for per-endpoint counts, 4xx/5xx rates and p50/p95/max latency. Absolute-form request targets, which proxies use, are normalised to a path.
TLS: not the payload. The ClientHello (handshake type 1 inside a record of content type 22) carries the Server Name Indication extension (type 0) in clear text on TLS 1.2 and 1.3, so you still learn which host the server is talking to, on which port, how many connections, how many bytes each way, and the round-trip time from ClientHello to ServerHello. That is your outgoing dependency map: Stripe, your database host, an S3 endpoint, a third-party API nobody knew about. Encrypted ClientHello (ECH) hides SNI, but adoption on API endpoints is still rare.
Local services: traffic between your reverse proxy and your app on 127.0.0.1:8000 is usually plain HTTP, so you get full request-level detail for your own application even when the public side is TLS.

What it cannot see

HTTP/2 multiplexes streams inside one connection and is almost always inside TLS; without keys it shows up as an opaque flow with byte counts. gRPC likewise. Cleartext HTTP/2 (h2c) is binary-framed, so the request-line parser skips it too. For those you want either the reverse proxy's access log (nginx, Apache and Caddy formats, which the same agent tails from the standard paths) or an in-process SDK. Two more limits worth knowing: a request that spans several segments is only parsed if the request line and Host header are in the first 2 KB of the first segment, and keep-alive connections that pipeline requests are attributed by order, which is correct for HTTP/1.1 because responses must arrive in request order.

From packets to something useful

Raw requests are noise. The useful transformation is:

Normalise paths so cardinality stays bounded: /users/1234/orders/550e8400-e29b-41d4-a716-446655440000 becomes /users/{id}/orders/{uuid}. Purely numeric segments become {id}, UUIDs {uuid}, hex strings of 16 or more characters {hash}, long base64-looking segments {token}. Everything else is kept verbatim so /v1/users and /v2/users stay distinct.
Aggregate per reporting interval: count, 4xx and 5xx counts, and p50/p95/max latency per (direction, host, method, path). Percentiles are computed from the raw samples in that interval, not from averages of averages.
Keep a small sample of the slowest and failing requests with their timing, so an incident shows evidence, not just a number.
Roll up hourly for history, and build the service map from the outgoing side: this server talks to these hosts, this often, this slowly. The reverse-DNS name of each destination is resolved once and cached, so the map shows api.stripe.com, not an IP.

Doing it with the Vigil agent

The agent is a single Python file that uses only the standard library, so it runs on any Linux with Python 3.8 or newer. Installed as root it enables capture automatically; without root it still reports metrics, processes and connections from /proc/net/tcp, but not per-request detail:

curl -fsSL https://puzaricloud.in/install-server-agent.sh | sudo sh -s -- --token vgs_YOUR_TOKEN

The server's APIs tab then shows incoming endpoints with p95 and 5xx rates, the Network tab shows outgoing hosts with connections and bytes, and the Services tab draws the map. Rules on http_error_pct, http_p95_ms and new_destination open incidents, so a new outbound host appearing at 3 am pages you, which is one of the cheaper ways to notice a compromised box.

For request traces across services, drop in the Python or Node SDK; it reports to the same local agent on 127.0.0.1:9111 and links spans to the captured endpoints.

Privacy and safety notes

The parser reads the request line and the Host header, and the status line of the response. It never parses other headers or any body, so Authorization, Cookie, query strings and payloads are not in memory beyond the kernel buffer and never leave the machine. Only aggregates and a handful of slow-request samples (method, path, status, timing) are reported. If your compliance rules forbid even that, turn capture off per server and rely on access logs and SDK spans instead.

Originally published on the PuzariCloud engineering blog. Drafted with AI assistance and reviewed, edited and tested by the author, who builds Vigil by PuzariCloud, the monitoring service the examples use.

Self-hosting a full monitoring stack in one command: servers, logs, APM and status pages

Ritom Puzari — Sat, 19 Sep 2026 21:45:16 +0000

Teams self-host monitoring for two reasons: the data cannot leave the building, or the SaaS bill grew faster than the fleet. The catch is that "self-hosted monitoring" usually means five services. This post is about what those five are, what running them really costs, and how to collapse them into one install.

The usual stack and what each piece costs you

Need	Common choice	What you maintain
Host metrics	Prometheus + node_exporter	retention sizing, cardinality, remote storage once it grows
Dashboards + alerts	Grafana + Alertmanager	provisioning, dashboards as code, alert routing
Logs	Loki + Promtail	chunk storage, label discipline, query limits
Uptime and status page	Uptime Kuma	one more SQLite database to back up
Traces	Tempo or Jaeger	object storage and sampling

Each is good software. Together they are roughly a day to set up properly, an afternoon a month to keep patched, and one incident a quarter when a disk fills or an upgrade changes a config format. For a platform team that is fine. For two developers doing ops on the side, it is the reason monitoring quietly stops getting attention.

What a single install has to do to replace that

Collect host metrics, processes, containers and services with one agent that needs no configuration.
Ship logs from the same agent, with search and alerting on patterns.
Give you request-level visibility without instrumenting every service. Passive capture of HTTP on the host gets you per-endpoint counts, error rates and p95 latency with zero code changes; SDKs add traces where you want them.
Run outside-in checks (HTTP, TCP, DNS, TLS, cron heartbeats) and a status page.
Update itself safely, verify what it installs, and keep working when the licence server or the internet is unreachable.

The one-command version

Vigil's self-hosted edition installs like this on any Ubuntu, Debian, RHEL-family or Alpine host:

curl -fsSL https://puzaricloud.in/selfhosted/install.sh | sudo sh -s -- \
  --domain vigil.example.com --email you@example.com --license 'VGL1.…'

It installs Docker if missing, downloads the release bundle and its detached Ed25519 signature, and verifies the signature with openssl pkeyutl -verify -rawin against a public key embedded in the installer before extracting anything. Ed25519 was chosen over RSA because the signature is 64 bytes, verification is a single call in OpenSSL 1.1.1 or newer with no hash negotiation to get wrong, and the same key signs licence keys, so there is one root of trust to protect. The installer then writes a Caddyfile with automatic HTTPS (ACME through Let's Encrypt, or tls internal for private networks) and starts Postgres, a worker and two web replicas behind Caddy's load balancer with active health checks.

Two replicas are what make updates safe. vigil update downloads and verifies the new bundle, builds the image, then replaces the first web container and waits for its health check to pass before touching the second, so there is never a moment without a healthy backend. If the new version fails its health check within two minutes the previous image is retagged and both replicas roll back. vigil backup runs pg_dump in custom format plus the config files every six hours by cron and keeps 14 daily and 8 weekly archives.

For private networks, --internal-tls makes Caddy issue certificates from its own local CA and --no-phone-home keeps the instance fully offline. A licence key is a signed JSON payload (VGL1.<payload>.<signature>) that the instance verifies locally against the embedded public key, so the instance runs with no outbound connection at all. On first start each installation generates its own Ed25519 identity, and the licence that finally unlocks it is bound to that identity's fingerprint, which is what stops one key being copied across many hosts. When a key lapses there are 14 days of grace; after that the instance becomes read-only rather than stopping monitoring, because an expired licence should never be the reason you miss an outage.

You can inspect exactly what the installer does before running it: install.sh and its signature are published next to the public key.

Resource use

Postgres is the only stateful part. A fleet of 20 servers reporting every 30 seconds with logs enabled lands around 2 GB of database growth a month at the default retention.

When you should still run the big stack

If you need distributed tracing across dozens of services with high-cardinality labels, or you already have Prometheus expertise and dashboards you love, keep them. The one-install approach is for teams who want monitoring to be a thing they have, not a thing they run.

Get alerted before an SSL certificate expires (the three checks that actually matter)

Ritom Puzari — Sat, 19 Sep 2026 21:26:32 +0000

Let's Encrypt made certificates free and short-lived. It did not make them renew themselves. Renewal fails silently when a DNS record changes, a firewall rule blocks port 80, a cron job stops, or someone moves the site to a new server and forgets the hook. The certificate keeps working right up to the second it does not.

Check expiry from the shell

echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
  | openssl x509 -noout -dates -issuer -subject

Output includes notAfter=. Anything under 14 days on a Let's Encrypt certificate (90-day lifetime, renewed by certbot at 30 days remaining, twice a day) means dozens of renewal attempts have already failed. The -servername flag matters: it sets SNI, and without it a server hosting several sites returns its default certificate, which is a common source of false alarms in home-grown checks.

Check the chain, not just the leaf

Browsers accept a certificate only when they can build a chain to a trusted root. A server that sends the leaf without the intermediate works in Chrome (which caches intermediates) and fails in curl, in mobile apps and in every monitoring script:

curl -sSI https://example.com -o /dev/null && echo chain ok

If this fails with unable to get local issuer certificate while the browser is happy, the intermediate is missing. Chrome and Firefox fetch missing intermediates through the Authority Information Access extension or use a cache; OpenSSL-based clients do not. Fix it in the web server's ssl_certificate (nginx wants fullchain.pem, not cert.pem); in Caddy nothing is needed, because Caddy manages it. To see what the server actually sends, openssl s_client -showcerts prints every certificate in the handshake, and a chain of one is the tell.

Check the hostname

A certificate for www.example.com does not cover example.com unless it is in the Subject Alternative Names; since 2017 browsers ignore the Common Name entirely, so a certificate with only a CN fails everywhere modern. A wildcard *.example.com matches one label only: it covers api.example.com but not example.com and not v2.api.example.com. Redirects hide these gaps: the bare domain redirects to www before the browser complains, but a monitoring probe, an API client or a webhook hitting the bare host gets a hostname mismatch.

echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

Why "we have auto-renewal" is not a monitoring strategy

Auto-renewal is a process. Monitoring watches the outcome. The outcome is "the certificate the public sees expires on date X", and only an external check sees exactly what a visitor sees, including the wrong certificate served by a load balancer that still holds last year's file.

Getting a warning 14 days out

Three options, from most to least effort:

A cron job around the openssl command above that compares notAfter with date and posts to Slack. Works, breaks when the box that runs it is decommissioned.
A Prometheus blackbox_exporter probe with the probe_ssl_earliest_cert_expiry metric and an alert rule. Solid if you already run Prometheus.
An uptime monitor that checks the certificate as part of its HTTPS check. In Vigil every https monitor runs a dedicated TLS handshake every 12 hours, records the expiry, and alerts once per certificate when the days left cross the threshold you set per monitor (14 by default). Because the alert is keyed to the certificate's expiry date, a renewal resets it automatically and a certificate that keeps not renewing does not spam you. The chain and hostname problems above show up as a failed check with the exact openssl error attached, so you are not guessing.

You can run all three checks on any hostname right now without an account at /tools/ssl-check.

A short checklist for the next incident review

Is the expiry alert sent to a channel someone reads, not only to the ops mailbox nobody opens?
Does the alert fire at 14 days, when there is still time to debug renewal, rather than at 24 hours?
Are the bare domain, www, the API host and the status page all monitored separately? They can have four different certificates.
Does the monitoring probe run from outside your network? A probe on the same box sees a different picture from your customers.
Is OCSP stapling on, and is the stapled response fresh? A stale staple with Must-Staple set fails the handshake in Firefox even when the certificate itself is fine. openssl s_client -status shows the staple.

How to monitor PostgreSQL connections, cache hit ratio and replication lag (and get paged before it hurts)

Ritom Puzari — Sat, 19 Sep 2026 21:26:29 +0000

Most PostgreSQL outages announce themselves an hour early. Connections creep towards max_connections, the buffer cache starts missing, a replica falls behind. Nobody looks, because the dashboard that shows it lives in a tool nobody opens. This guide covers the four numbers worth alerting on, the queries behind them, and thresholds that page rarely but early.

1. Connections against max_connections

The single most common self-inflicted PostgreSQL outage is FATAL: sorry, too many clients already. It happens when an app pool grows, a migration script leaks connections, or someone runs pgbouncer in the wrong mode.

SELECT count(*) AS used,
       current_setting('max_connections')::int AS max,
       round(100.0 * count(*) / current_setting('max_connections')::int, 1) AS pct
FROM pg_stat_activity;

Alert at 90 % of the maximum. Warn at 75 % if you want a heads-up. Note that max_connections includes superuser_reserved_connections (3 by default), so ordinary roles hit the wall a few connections earlier than the number suggests. Look at the state column too: dozens of idle in transaction sessions mean an application is holding transactions open, which pins the xmin horizon so vacuum cannot reclaim dead tuples and, with a long enough hold, blocks DDL behind an ACCESS EXCLUSIVE lock queue.

2. Cache hit ratio

Postgres serves reads from shared_buffers when it can and from the OS page cache or disk when it cannot. A hit ratio that drops below 90 % on an OLTP database usually means the working set outgrew memory, a new query is scanning a big table, or somebody shrank the instance.

SELECT round(100.0 * sum(blks_hit) / nullif(sum(blks_hit) + sum(blks_read), 0), 2) AS cache_hit_pct
FROM pg_stat_database
WHERE datname = current_database();

These counters are cumulative since the last pg_stat_reset(), so a monitor has to diff consecutive readings and compute the ratio on the deltas. Alert when the rolling ratio stays under 90 % for several minutes rather than on a single reading. One caveat: blks_read counts reads from the OS page cache as misses, because Postgres cannot see the kernel cache. On a host with lots of free memory a "miss" may still be served from RAM, so pair this number with the host's disk read throughput before concluding the working set outgrew memory.

3. Replication lag

On a streaming replica:

SELECT CASE WHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN 0
       ELSE extract(epoch FROM now() - pg_last_xact_replay_timestamp()) END AS lag_seconds;

The CASE matters: when the replica has replayed everything it received, pg_last_xact_replay_timestamp() stops advancing on an idle primary, and the naive now() - replay_timestamp reports growing lag on a system that is perfectly in sync. On the primary, pg_stat_replication shows every replica's write_lag, flush_lag and replay_lag as intervals measured from WAL send time, which is the more precise view. Alert at 30 seconds for read replicas that serve traffic, longer for backup replicas. Lag that grows steadily is a replica that cannot keep up (usually single-threaded replay on a primary with many parallel writers); lag that spikes and recovers is usually a long transaction, a bulk load or a vacuum on the primary.

4. Active and waiting queries

SELECT count(*) FILTER (WHERE state = 'active') AS active,
       count(*) FILTER (WHERE wait_event_type = 'Lock') AS waiting_on_locks,
       count(*) FILTER (WHERE state = 'active' AND now() - query_start > interval '5 seconds') AS slow
FROM pg_stat_activity
WHERE backend_type = 'client backend';

A rising waiting_on_locks count is the earliest signal of a lock pile-up behind a migration. Alert when it exceeds a handful for more than a minute. wait_event_type and backend_type exist from PostgreSQL 10; on older servers drop the backend_type filter. pg_blocking_pids(pid) tells you which session is at the head of the queue, which is usually an ALTER TABLE waiting for a long-running SELECT.

Running these checks without an exporter

You can wrap the queries in a cron job that posts to a webhook, or run postgres_exporter and Prometheus. Both work; both are one more thing to maintain.

If a server already runs the Vigil agent, add a database entry to /etc/vigil-agent.json and restart it. The agent calls psql locally with the password passed through the environment, never on the command line, and reports the numbers above every interval:

{
  "databases": [
    {"system": "postgresql", "name": "main",
     "dsn": "postgresql://vigil_ro:PASSWORD@127.0.0.1:5432/app"}
  ]
}

Create a read-only role for it first. pg_monitor (PostgreSQL 10+) grants exactly the statistics views the queries above need, including the unredacted query column in pg_stat_activity, without any table access:

CREATE ROLE vigil_ro LOGIN PASSWORD 'PASSWORD';
GRANT pg_monitor TO vigil_ro;

vigil-server-agent --db-test prints what the collector sees so you can check the DSN before restarting. In the server's Databases tab you get connections, cache hit, transactions per second, replication lag, active and waiting queries, and the top statements when pg_stat_statements is installed. Rules for connections percentage, cache hit ratio and replication lag open incidents automatically, with the thresholds above as defaults.

Thresholds in one table

Metric	Warn	Page
Connections used	75 %	90 %
Cache hit ratio (rolling)	under 95 %	under 90 %
Replication lag	10 s	30 s
Sessions waiting on locks	3	10 for 1 minute
Idle in transaction	any older than 5 min	any older than 15 min

Tune them to your workload after a week of data, and make sure the alert goes to a channel someone reads at night.