DEV Community

Anand Rathnas
Anand Rathnas

Posted on Originally published at jo4.io

Scrape Prometheus Over the VPC, Not Through Cloudflare

#prometheus #devops #architecture #cloudflare

This article was originally published on Jo4 Blog.

We built three layers of defense for our Prometheus scrape. Two of them were unnecessary.

This isn't a "we found a bug" post. It's a "we designed the wrong architecture, then corrected it" post. The system worked. It was just doing far more than it needed to. Sharing it because the wrong path was the obvious-looking one, and the right path was sitting right there in our DO project the entire time.

The Original Setup

We have two droplets in production: jo4-server (Spring Boot API behind nginx) and jo4-impress (Prometheus + Grafana + Loki, our internal observability box). When we stood up Prometheus, the question was: how should it scrape /actuator/prometheus on the API?

The obvious answer — and the one we went with — was "the same way every other request reaches the API." Which means:

jo4-impress → Cloudflare edge → DO firewall → nginx → Spring Boot
              (WAF rule)        (CF CIDRs only)
Enter fullscreen mode Exit fullscreen mode

Three defense layers stacked on the public path:

  1. A Cloudflare WAF rule restricting /actuator/* to a tiny allowlist (our impress droplet's egress IP plus a couple of operator IPs).
  2. An OAuth2 client_credentials grant (RFC 6749 §4.4) on the actuator endpoints, scope metrics:read, enforced at the Spring Security layer.
  3. A DigitalOcean cloud firewall on jo4-server restricting inbound :443 to Cloudflare's published CIDR ranges, so nobody can bypass the WAF by hitting the origin directly.

It worked. The scrape ran every 30 seconds, dashboards lit up, alerts fired. We patted ourselves on the back for being thorough.

But every scrape was: TLS handshake with Cloudflare, WAF evaluation, second TLS handshake with nginx, OAuth token mint (cached, but still), HTTP round-trip back through the same chain. All to move ~80 KB of metrics between two boxes that were already neighbors.

The Realization

About a month in, I was running unrelated doctl commands and noticed both droplets in the same row of output:

$ doctl compute droplet list --format Name,PrivateIPv4,Region
ID           Name           Private IPv4    Region
123456789    jo4-server     10.108.0.3      nyc3
987654321    jo4-impress    10.108.0.5      nyc3
Enter fullscreen mode Exit fullscreen mode

Same region. Adjacent private IPs in the same /16. They had been in the same VPC the entire time — 10.108.0.0/16, the default DO VPC for nyc3.

DigitalOcean droplets in the same VPC can reach each other on private interfaces with no public exposure, no firewall hops, no encryption-in-transit requirement (the VPC is a private L2 segment). Cloudflare didn't need to be on the path at all. The DO firewall's CF-CIDR rule was protecting traffic that didn't have to traverse the public internet in the first place.

I had built a public path to talk between two machines on the same private network. That's the kind of thing you laugh at in someone else's design review.

The Simpler Architecture

The corrected topology:

jo4-impress (10.108.0.5) ──VPC──► jo4-server (10.108.0.3:8080)
                                  └─► OAuth still enforced at app layer
                                  └─► DO firewall: tcp/8080 from tag:impress
Enter fullscreen mode Exit fullscreen mode

One hop. No Cloudflare. No TLS termination. No nginx. Spring Boot listens on :8080 (it already did, for the nginx upstream), and Prometheus connects directly via the private IP.

The crucial check before pulling this trigger: does the OAuth gate still apply when the request arrives via the private interface, or was it only enforced because Cloudflare's WAF was upstream? This is the kind of question where "I'm pretty sure" gets you fired. We verified by hand:

# From inside jo4-impress, hit the private IP with no bearer:
$ curl -i http://10.108.0.3:8080/actuator/prometheus
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer ...
Enter fullscreen mode Exit fullscreen mode

Good. Spring Security doesn't care which interface the request came in on — the filter chain runs the same way for private and public traffic. The OAuth requirement is a property of the endpoint, not of the network path. The WAF rule had been belt-and-braces; the actual auth boundary was always at the app.

Token minting moved to the private IP too, since the token endpoint lives on the same Spring Boot app:

token_url: http://10.108.0.3:8080/oauth/token
Enter fullscreen mode Exit fullscreen mode

The Diff

The whole change is in prometheus.yml. Before:

- job_name: jo4-api
  metrics_path: /actuator/prometheus
  scheme: https
  oauth2:
    client_id: __PROMETHEUS_OAUTH_CLIENT_ID__
    client_secret_file: /etc/prometheus/oauth-client-secret
    token_url: https://jo4-api.jo4.io/oauth/token
    scopes:
      - metrics:read
  static_configs:
    - targets: ['jo4-api.jo4.io']
Enter fullscreen mode Exit fullscreen mode

After:

- job_name: jo4-api
  metrics_path: /actuator/prometheus
  scheme: http
  oauth2:
    client_id: __PROMETHEUS_OAUTH_CLIENT_ID__
    client_secret_file: /etc/prometheus/oauth-client-secret
    token_url: http://10.108.0.3:8080/oauth/token
    scopes:
      - metrics:read
  static_configs:
    - targets: ['10.108.0.3:8080']
Enter fullscreen mode Exit fullscreen mode

Three lines changed: scheme, token_url, and the target. The OAuth block is identical because the auth gate is identical — same client, same secret, same scope. Prometheus didn't know or care that the underlying network had been simplified out from under it.

http instead of https looks scary written down, but it's only scary if you treat the VPC like the public internet. DO's VPC is a private network segment between droplets in the same project and region; the threat model that justifies TLS on the open web (passive observers, route hijacks, captive portals) doesn't apply. We still terminate TLS at Cloudflare for everything that reaches users — this just isn't one of those things.

What Stays as Defense-In-Depth

We didn't strip the auth gate. Three layers remain, but now they're actually layered on top of a sound primary:

  • Primary boundary: VPC isolation. The scrape traffic physically cannot leave DO's nyc3 private network. There is no public path for this byte stream.
  • App-layer OAuth. Same metrics:read scope, same client_credentials grant. If somehow another droplet ended up in the VPC and tried to scrape, it would get a 401 without the bearer.
  • DO cloud firewall, tag-based. On jo4-server, we added tcp/8080 inbound from tag:impress. Tag-based rules follow droplets through rebuilds and IP changes — no hardcoded CIDR maintenance.
  • WAF rule + public exposure on :443. Kept for now. Not reached in practice (Prometheus doesn't hit it anymore), but the public actuator path still exists for emergency operator access. We'll probably retire it next quarter once we're confident nothing else depends on it.

The point isn't "fewer layers" for its own sake. The point is each remaining layer is on the actual path and earns its keep. Two of the original three were guarding against attacks on a public surface that didn't need to exist.

Lessons Learned

  • Check the topology before the rules. I added three security layers to a public path before asking whether the path needed to be public. The DO VPC was a one-line doctl query away. I just never thought to look.
  • Defense-in-depth is good. Defense-on-the-wrong-network-path is overhead. Every layer has a cost — config, latency, debuggability, things that break during outages. Layers should defend something real.
  • Verify the auth boundary is at the app, not the edge. Before deleting the public path, we curled the private IP without a bearer and confirmed the 401. If Spring Security had been relying on a Cloudflare-injected header (it wasn't, but it could have been), this refactor would have silently disabled our auth gate.
  • Tag-based firewall rules > IP-based rules. tcp/8080 from tag:impress survives droplet rebuilds. tcp/8080 from 10.108.0.5/32 survives until the next time the droplet's IP changes and you forget.
  • Be honest in the comments. The prometheus.yml comment block now narrates the actual topology and the actual primary boundary (VPC), with the auxiliary layers labeled as such. Future-me reading this in two years will know what's load-bearing and what's belt-and-braces.
  • It's fine to ship the wrong architecture and fix it. The wrong one worked. We had metrics, we had alerts. Nothing was broken. But "works" and "right" aren't the same thing, and shipping the simpler version when you notice the gap is a win, not an embarrassment.

When have you over-engineered a network path? What pushed you back to simpler? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Top comments (0)