DEV Community: Anand Rathnas

OAuth client_credentials in Spring Boot for Prometheus Scrapes

Anand Rathnas — Fri, 25 Sep 2026 06:00:33 +0000

We stood up Prometheus on a self-hosted DigitalOcean droplet and pointed it at our Spring Boot API's /actuator/prometheus endpoint. The obvious question came next: how does Prometheus authenticate?

Until that moment, our OAuth server had spoken exactly two grants — authorization_code (with PKCE) and refresh_token. Both assume a user behind the request. Zapier, Make, Pipedream, the Chrome extension, the MCP clients — every existing integration represents a person who clicked "Connect". Prometheus represents nobody. It's a process on a host scraping a metrics endpoint every 30 seconds. There is no consenting user.

The RFC has a grant for exactly this — client_credentials, §4.4. The client authenticates itself to the token endpoint with its credentials, gets back a bearer, and presents that bearer on subsequent calls. No user, no consent screen, no refresh token. Here's how we slotted it into an existing Spring Security stack without regressing a single user-bound integration.

Why client_credentials Fits

Three properties of the Prometheus scrape are worth being explicit about, because each maps directly to a §4.4 design choice:

There is no user. Prometheus is a daemon. It has no Auth0 identity, no email, no role assignments. Forcing it through authorization_code would mean creating a synthetic "Prometheus user" — a pattern that's been a security footgun every time we've seen it (service accounts that drift into having too many permissions because they're shaped like real users).

The credentials live on the server, rotatably. Prometheus reads its OAuth client_secret from a file on disk that the deployment script writes. Rotating means redeploying the secret file and restarting Prometheus — same operational model as any other server-side credential. No browser, no consent.

The blast radius is one endpoint. Prometheus has exactly one thing it's allowed to do: GET /actuator/prometheus. Not list URLs, not read analytics, not touch user data. The credential should be incapable of doing anything else even if it leaks.

client_credentials is the RFC-defined shape for all three. The wire flow is two HTTP calls: POST /oauth/token with HTTP Basic-Auth-encoded client credentials returns an access_token; every subsequent scrape carries Authorization: Bearer <access_token> until the token expires. Per §4.4.3 there is no refresh token — when the access token expires, the client re-authenticates by hitting /oauth/token again. Prometheus's built-in oauth2: block does this transparently.

Three Schema Relaxations

Before any of the Java could change, the database had to admit the shape of a userless token and the shape of a redirect-URI-less client. Liquibase changeset 356 made three modifications:

- changeSet:
    id: 356-add-oauth-client-credentials-grant
    author: anand
    changes:
      - addColumn:
          tableName: oauth_clients
          columns:
            - column:
                name: grant_types
                type: VARCHAR(100)
                defaultValue: "authorization_code,refresh_token"
                constraints:
                  nullable: false
                remarks: "Comma-separated RFC 6749 grant_type values the client is permitted to use."

      - dropNotNullConstraint:
          tableName: oauth_clients
          columnName: redirect_uris
          columnDataType: TEXT

      - dropNotNullConstraint:
          tableName: oauth_access_tokens
          columnName: user_id
          columnDataType: BIGINT

Each change has a specific reason:

oauth_clients.grant_types (NEW column). Previously, the grant type set was implicit — every client got AC + RT. Now each client carries the exact list of grants it's allowed to request. Existing rows get backfilled to "authorization_code,refresh_token" via the column default during the addColumn step, which is the property that keeps every existing integration green. Zapier's row, Make's row, both Pipedream rows, the Chrome extension's row — all get the same string they were implicitly assuming. The new CC-only Prometheus client gets "client_credentials" and nothing else.

oauth_clients.redirect_uris nullable. A CC-only client legitimately has no browser callback. Without this relaxation, every CC client registration would have to invent a fake URL just to satisfy a NOT NULL constraint that was modeling the wrong invariant. Enforcement now moves to the service layer — it's still required for authorization_code clients, but the rule is "required when grant_types includes AC", not "required always".

oauth_access_tokens.user_id nullable. A CC token represents the client, not a user. The foreign key to users(id) ON DELETE CASCADE stays — NULL becomes the signal for "service-principal token, no user lookup needed". Token validation branches on this: if userId is null, skip the user-load step.

The changeset uses addColumn and dropNotNullConstraint rather than raw SQL, which sidesteps the Liquibase PL/pgSQL footgun (folded-block YAML corrupting dollar quotes). All three changes are reversible and the rollback is provided explicitly.

Token Issuance

The token endpoint is a switch on grant_type. We added a third branch:

@Transactional
public TokenResponse issueClientCredentialsToken(String clientId, String clientSecret,
                                                 String requestedScope, String requestedResource) {
    OAuthClientEntity client = validateTokenEndpointClient(clientId, clientSecret);

    if (client.isPublicClient()) {
        throw new UrlService.AppException(ErrorCode.OAUTH_INVALID_CLIENT_SECRET,
                "client_credentials grant requires a confidential client");
    }

    if (!client.supportsGrant("client_credentials")) {
        throw new UrlService.AppException(ErrorCode.OAUTH_CLIENT_DISABLED,
                "Client is not authorized to use client_credentials grant");
    }

    // Default to the client's full allowed scope set if none requested; otherwise
    // intersect requested scopes with allowed (RFC 6749 §3.3 narrowing).
    Set<String> allowed = client.getScopesSet();
    Set<String> granted;
    if (requestedScope == null || requestedScope.isBlank()) {
        granted = allowed;
    } else {
        Set<String> requested = parseScopes(requestedScope);
        if (!allowed.containsAll(requested)) {
            throw new UrlService.AppException(ErrorCode.OAUTH_INVALID_SCOPE,
                    "One or more requested scopes are not granted to this client");
        }
        granted = requested;
    }
    String grantedScopeString = String.join(",", granted);

    // Mint access token only — no refresh token per RFC 6749 §4.4.3.
    String accessToken = generateToken(TOKEN_LENGTH);
    String accessTokenHash = hashToken(accessToken);
    long accessExpiresAt = System.currentTimeMillis() + (client.getAccessTokenLifetimeSeconds() * 1000L);

    OAuthAccessTokenEntity accessTokenEntity = OAuthAccessTokenEntity.builder()
            .tokenHash(accessTokenHash)
            .clientId(client.getId())
            .userId(null)  // CC tokens have no user
            .scopes(grantedScopeString)
            .expiresAt(accessExpiresAt)
            .tenantId(client.getTenantId())
            .audience(requestedResource)
            .slug(UUID.randomUUID().toString().replace("-", "").substring(0, 16))
            .build();

    accessTokenRepository.save(accessTokenEntity);
    // ...returns TokenResponse with no refresh_token
}

Three things this is doing right:

Public-client rejection. A public client (token_endpoint_auth_method = "none") is one that authenticates via PKCE only. CC has no PKCE story — its security model is "the client knows the secret". Public clients can never use CC, full stop. We reject before issuing.

Per-client grant gating. supportsGrant("client_credentials") checks the row's grant_types column. A client registered for AC + RT cannot suddenly switch grants on the wire. The grant set is part of the client identity, not a request-time choice.

Scope narrowing. RFC 6749 §3.3 says the authorization server MAY issue a narrower scope than requested, MUST NOT issue a wider one. We default to "narrowest of allowed and requested", and reject if the request asks for anything outside allowed.

The token gets persisted with userId = null — the column relaxation from migration 356 is what allows that row to exist at all. No refresh token row is created.

Authentication Subclass

The hardest design decision wasn't on the token endpoint. It was on the validation side. When a bearer comes in on /actuator/prometheus, the existing filter chain inspects the token and constructs a Spring Security Authentication object to attach to the request. The existing code expects every authentication to have a user:

// OAuthTokenAuthentication.java (existing — unchanged)
public OAuthTokenAuthentication(UserEntity user, OAuthAccessTokenEntity token, OAuthClientEntity client) {
    super(buildAuthorities(user, token));
    this.user = user;
    this.userId = user.getId();  // would NPE for CC tokens
    // ...
}

The naive fix is to make user nullable on OAuthTokenAuthentication and sprinkle null-checks across every consumer. We considered that and rejected it — the codebase has dozens of instanceof OAuthTokenAuthentication checks across AuthContext, OAuthScopeEnforcementFilter, Jo4McpTools, and elsewhere. Making user nullable on the existing type would mean auditing every single one of them and adding null guards, with the constant risk of missing one and getting an NPE in production six weeks later.

We introduced a separate authentication type instead:

@Getter
public class OAuthClientCredentialsAuthentication extends AbstractAuthenticationToken {

    private final OAuthAccessTokenEntity accessToken;
    private final OAuthClientEntity client;
    private final Long clientId;
    private final Set<String> scopes;

    public OAuthClientCredentialsAuthentication(OAuthAccessTokenEntity accessToken, OAuthClientEntity client) {
        super(buildAuthorities(accessToken));
        this.accessToken = accessToken;
        this.client = client;
        this.clientId = client.getId();
        this.scopes = accessToken.getScopesSet();
        setAuthenticated(true);
    }

    /**
     * The "principal" of a client_credentials token is the OAuth client itself.
     */
    @Override
    public Object getPrincipal() {
        return client;
    }
    // ...
}

What this buys us is type-driven correctness. Every existing instanceof OAuthTokenAuthentication check naturally evaluates false for a CC token. Code that asks "who is the user behind this request?" gets Optional.empty() from AuthContext.getActualUserOptional(). Controllers that call authContext.getCurrentUser() throw UNAUTHORIZED automatically. The MCP tools that filter by userId never even see CC requests. We didn't have to audit dozens of sites; the type system did it for us.

The validation filter branches once, at the boundary:

// inside OAuthTokenAuthenticationFilter, paraphrased
ValidatedToken validated = oauthService.validateAccessToken(bearer);
Authentication auth = validated.getUser() == null
    ? new OAuthClientCredentialsAuthentication(validated.getToken(), validated.getClient())
    : new OAuthTokenAuthentication(validated.getUser(), validated.getToken(), validated.getClient());
SecurityContextHolder.getContext().setAuthentication(auth);

validateAccessToken itself has exactly one change — the user lookup becomes conditional on userId != null. That's the entire blast radius on the validation side.

The Path Allowlist

Scope-based authorization gets us most of the way. The Prometheus client has metrics:read scope only — not read, not write. Endpoints requiring read/write reject it because the bearer lacks those scopes.

But "lacks the right scope" is one layer. We wanted defense-in-depth, because scope sets get edited in admin UIs by humans and a future "let's give the metrics client read to test something" five-second mistake should not silently grant CC tokens access to user data. So we added a second filter that runs immediately after the token filter:

@Slf4j
@Component
@Profile("auth0")
public class ClientCredentialsTokenGateFilter extends OncePerRequestFilter {

    private final AntPathMatcher pathMatcher = new AntPathMatcher();

    @Value("${app.security.cc-token-paths:/actuator/prometheus}")
    private String ccTokenPaths;

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
            throws ServletException, IOException {

        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        // Only act on CC tokens. Everything else (JWT, API key, user-bound OAuth, anonymous) passes through.
        if (!(auth instanceof OAuthClientCredentialsAuthentication ccAuth)) {
            filterChain.doFilter(request, response);
            return;
        }

        String requestPath = request.getRequestURI();
        List<String> allowed = Arrays.stream(ccTokenPaths.split(","))
                .map(String::trim)
                .filter(s -> !s.isEmpty())
                .toList();

        boolean pathAllowed = allowed.stream().anyMatch(p -> pathMatcher.match(p, requestPath));
        if (pathAllowed) {
            filterChain.doFilter(request, response);
            return;
        }

        log.warn("client_credentials token attempted to access non-allowlisted path: clientId={}, path={}, scopes={}",
                ccAuth.getClientId(), requestPath, ccAuth.getScopes());

        // RFC 6750 §3.1: insufficient_scope on a Bearer-protected resource.
        response.setStatus(HttpStatus.FORBIDDEN.value());
        response.setHeader("WWW-Authenticate",
                "Bearer error=\"insufficient_scope\", error_description=\"client_credentials tokens are not valid for this endpoint\"");
        response.getWriter().write(
                "{\"error\":\"insufficient_scope\",\"error_description\":\"client_credentials tokens are not valid for this endpoint\"}");
    }
}

A few details that matter:

The instanceof guard is the no-op short-circuit. JWT requests, API-key requests, user-bound OAuth requests, anonymous requests — all four pass through the filter with one type check and zero allocations. The cost is paid only on the small set of CC requests.

Config-driven allowlist. app.security.cc-token-paths is comma-separated and AntPathMatcher-evaluated. Adding a future endpoint (say, an alertmanager receiver) is a config edit, not a code change. The default — /actuator/prometheus — is the only thing CC tokens can reach out of the box:

# application-security.yaml
app:
  security:
    # Paths where client_credentials OAuth tokens are accepted. CSV; AntPathMatcher syntax.
    cc-token-paths: "/actuator/prometheus"

WWW-Authenticate: Bearer error="insufficient_scope" is the RFC 6750 §3.1 shape. Prometheus and other OAuth-aware clients can read this and surface a meaningful error, rather than getting a generic 403 with no clue what went wrong.

Never throws. Filters that throw can mask the underlying error or tear down the chain in surprising ways. This one always writes a response or delegates.

Wiring the Scrape

On the Prometheus side, the entire setup is one block in prometheus.yml:

scrape_configs:
  - job_name: jo4-api
    metrics_path: /actuator/prometheus
    scheme: http
    scrape_interval: 30s
    oauth2:
      client_id: __PROMETHEUS_OAUTH_CLIENT_ID__
      client_secret_file: /etc/prometheus/oauth-client-secret
      token_url: http://10.108.0.3:8080/oauth/token
      scopes:
        - metrics:read
    static_configs:
      - targets: ['10.108.0.3:8080']
        labels:
          service: alertstage
          env: prod

Notes from setting this up:

client_id inline, client_secret_file from disk. Per OAuth 2.0, the client ID is public — fine to bake into the config and sed-substitute at deploy time. The secret stays in a separate file with restrictive permissions, written by the bootstrap script from a GitHub Secret. Never co-located, never logged.

Private VPC URLs. token_url and the scrape target both use the DigitalOcean private IP 10.108.0.3. Traffic never leaves the VPC, which is the primary security boundary; the OAuth check is defense-in-depth on top of that. The token endpoint accepts plain HTTP only because it's on a private subnet; this would be HTTPS for any path that crosses the public internet.

scopes: ["metrics:read"]. Prometheus's oauth2: block converts this into the scope form field of the /oauth/token request. Our issueClientCredentialsToken narrows it against the client's allowed set — which happens to be just metrics:read — and stamps the result on the issued access token.

No refresh_token config. Prometheus knows §4.4.3 — when it sees a token response with no refresh_token, it just re-runs the token flow when the current access token nears expiry. No refresh logic on either side.

Testing It End-To-End

A curl sequence to verify the whole loop, against a freshly registered Prometheus client (client ID jo4_Xxxx, client secret captured at registration):

# 1. Token exchange — HTTP Basic-Auth-encoded client credentials.
ACCESS_TOKEN=$(curl -s -X POST https://jo4-api.jo4.io/oauth/token \
  -u "jo4_Xxxx:secret_XXXXXXXX" \
  -d "grant_type=client_credentials" \
  -d "scope=metrics:read" | jq -r .access_token)

# 2. Scrape — bearer on the request.
curl -i https://jo4-api.jo4.io/actuator/prometheus \
  -H "Authorization: Bearer $ACCESS_TOKEN"
# → HTTP/2 200
# → # HELP http_server_requests_seconds ...

# 3. Negative test — same token against a user endpoint.
curl -i https://jo4-api.jo4.io/api/v1/urls/me \
  -H "Authorization: Bearer $ACCESS_TOKEN"
# → HTTP/2 403
# → WWW-Authenticate: Bearer error="insufficient_scope", ...
# → {"error":"insufficient_scope","error_description":"client_credentials tokens are not valid for this endpoint"}

That third call is the critical one to verify. The same bearer, presented at a non-allowlisted path, must be rejected — even though the bearer is technically valid. If you see anything other than 403 + insufficient_scope there, the gate filter isn't wired correctly.

Lessons Learned

A new grant type is mostly a schema problem. The Java for client_credentials is small. The work was three column relaxations and an audit of every codepath that assumed userId was non-null.
A separate Authentication subclass beats nullable fields on the existing one. instanceof checks across the codebase become the type-system's audit for you. Sprinkling null-checks would have been an ongoing tax forever.
Always backfill new NOT NULL columns via a default. Adding grant_types NOT NULL would have failed on every existing row without defaultValue: "authorization_code,refresh_token". The default is what makes the migration zero-downtime and the rollout silent for every existing client.
Defense-in-depth means two independent layers, not one layer twice. Scope-based authorization handles "the client wasn't granted this scope". The path-allowlist filter handles "even if someone misconfigures the scope set tomorrow, this token still can't hit user data". Both layers must agree before access is granted.
WWW-Authenticate on 401/403 from a bearer-protected resource is RFC 6750 §3.1. Clients can react to error="insufficient_scope" programmatically. A naked 403 with no header forces humans into the loop for every failure.
No refresh token for CC, period. RFC 6749 §4.4.3. Prometheus and every other OAuth-aware client know how to re-run the token flow on expiry. Issuing a refresh token would be inventing capability the spec explicitly forbids.
Validate every existing integration before shipping. We scanned ten risk categories across the codebase before touching a line — Zapier, Make, Pipedream, Chrome extension, MCP clients, OIDC discovery, /oauth/userinfo, the scope filter, public endpoints, the previously-public /actuator/prometheus path. The migration changed default-value-backfilled columns; behavior changed for exactly one path, and it was the one we intended.

Adding CC grant to an existing AC flow? What schema migration bit you? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Bound Spring MVC's Async Executor or Pay for It Under Load

Anand Rathnas — Wed, 23 Sep 2026 05:43:18 +0000

Spring MVC's default async task executor spawns a brand new thread for every async request. No queue. No ceiling. No back-pressure. Spring itself logs a WARN on first use: "This executor is not suitable for production use under load." That's a load-bearing warning, and most teams scroll past it.

We learned the hard way and ended up wiring a bounded ThreadPoolTaskExecutor, a typed rejection path, and a Micrometer saturation gauge. Here's the whole arc.

What Spring MVC Defaults To

When a controller method returns a DeferredResult, Callable, WebAsyncTask, Flux, or Mono, Spring MVC needs an executor to run the async dispatch. If you don't configure one, you get SimpleAsyncTaskExecutor.

SimpleAsyncTaskExecutor does exactly what its name says: every submission spawns a fresh thread. There is no pool. There is no queue. There is no rejection policy, because there is no ceiling to reject against. Send it a thousand concurrent requests and it will cheerfully start a thousand threads, each with its own ~1 MB stack, and watch your container OOM.

Spring 6 logs this WARN on first use:

If you ignore the warning, the failure mode is exotic: latency stays fine until traffic crosses some invisible threshold, at which point the JVM either runs out of native threads (Linux pthread_create returns EAGAIN) or runs out of heap because every parked thread is holding stack frames and a Tomcat request scope.

Why That's Wrong For Real Apps

Three patterns push you onto this executor whether you realize it or not:

DeferredResult / Callable / WebAsyncTask returns. Anyone using the classic async-servlet pattern.
Flux / Mono returns from a Spring MVC controller. Spring MVC (not WebFlux) bridges reactive types onto the async-dispatch path. Every chunk emission round-trips through the executor.
SSE endpoints (SseEmitter). Long-lived streams that hold a dispatch thread for the entire lifetime of the connection.

We ship all three. Our analytics SSE stream in particular pins one async thread per subscriber for up to five minutes (our ASYNC_TIMEOUT). With SimpleAsyncTaskExecutor, 500 concurrent dashboard users would mean 500 unbounded threads with no warning sign before the container falls over.

Bounded Pool Setup

The fix is a real ThreadPoolTaskExecutor with explicit core, max, and queue settings. Here's the actual bean from WebMvcAsyncConfig.java:

@Bean(name = "mvcAsyncExecutor", destroyMethod = "shutdown")
public ThreadPoolTaskExecutor mvcAsyncExecutor() {
    ThreadPoolTaskExecutor exec = new ThreadPoolTaskExecutor();
    exec.setCorePoolSize(8);
    exec.setMaxPoolSize(200);
    // queueCapacity = 0 → Spring uses SynchronousQueue: direct hand-off, no buffering.
    exec.setQueueCapacity(0);
    exec.setThreadNamePrefix("jo4-mvc-async-");
    exec.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy());
    exec.setWaitForTasksToCompleteOnShutdown(true);
    exec.setAwaitTerminationSeconds(30);
    exec.initialize();
    return exec;
}

@Override
public void configureAsyncSupport(AsyncSupportConfigurer configurer) {
    configurer.setDefaultTimeout(ASYNC_TIMEOUT.toMillis());
    configurer.setTaskExecutor(mvcAsyncExecutor());
}

A few decisions worth calling out, because the obvious choices are wrong for long-lived async:

queueCapacity = 0. Spring's ThreadPoolTaskExecutor interprets queueCapacity = 0 as a SynchronousQueue — a direct hand-off with no buffering. The instinct is to "set a big queue, it's free." For short-lived tasks that's right. For SSE streams (which hold the thread for up to 5 minutes), a non-zero queue would park new connections waiting for an existing core thread to free — effectively "hang the client until someone else disconnects." With a 0-capacity queue, the pool grows from corePoolSize=8 toward maxPoolSize=200 as new subscribers arrive, and only true saturation hits the rejection handler.

AbortPolicy, not CallerRunsPolicy. This one we learned by pain. Our first attempt used CallerRunsPolicy (the seemingly-graceful "if the pool is full, run inline on the caller's thread"). Under load that pins the Tomcat request thread executing the task synchronously — which turned multi-minute Playwright teardowns into multi-minute hangs because the test runner's "close all contexts" call was racing pool saturation. Fail-fast with AbortPolicy is preferable: back-pressure surfaces as a RejectedExecutionException that the caller can translate into an HTTP 503, and the Tomcat thread is freed immediately for the next request.

waitForTasksToCompleteOnShutdown = true + awaitTerminationSeconds = 30. SSE streams need time to flush their final event and let the client see the close cleanly. Hard-killing the pool on shutdown means subscribers get a TCP reset instead of a proper stream-end signal.

destroyMethod = "shutdown". Pairs the bean lifecycle with the executor lifecycle so Spring's context shutdown drains the pool. Without it you leak threads on every context refresh.

Rejection With Meaning

AbortPolicy throws RejectedExecutionException. By default that propagates out of the async-dispatch machinery, hits the generic exception handler, and becomes a 500 Internal Server Error — which is a lie. It's not an internal error; it's capacity. Calling clients, CDN edges, and retry-aware HTTP libraries all behave differently against 500 vs 503: 500 says "give up, this is broken"; 503 says "back off and try again."

We give it a typed home in our ErrorCode enum:

SERVICE_SATURATED("SERVICE_SATURATED", "Server is temporarily at capacity. Please retry."),

And we catch it explicitly in GlobalExceptionHandler:

@ExceptionHandler(RejectedExecutionException.class)
public ResponseEntity<ResponseBody<EmptyResponse>> handleRejectedExecution(RejectedExecutionException ex) {
    String requestUri = getRequestUri();
    log.warn("Async executor saturated on {}: {}", requestUri, ex.getMessage());

    Fault fault = Fault.builder()
            .code(ErrorCode.SERVICE_SATURATED.getCode())
            .description(ErrorCode.SERVICE_SATURATED.getMessage())
            .build();

    ResponseBody<EmptyResponse> body = ResponseBody.<EmptyResponse>builder()
            .requestId(UUID.randomUUID().toString())
            .faults(List.of(fault))
            .build();

    return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
            .header(HttpHeaders.RETRY_AFTER, "5")
            .body(body);
}

Three things matter here. First, the response is 503 SERVICE_UNAVAILABLE — semantically correct for "we're at capacity," not "we're broken." Second, the Retry-After: 5 header is honored by fetch retry middlewares, the AWS SDK, and most CDN edges, which means callers self-throttle without us writing client code. Third, the structured fault carries SERVICE_SATURATED so the SPA can render a friendly "we're at capacity, try again" toast instead of a generic 5xx splash.

The mapping back to HTTP status lives in mapErrorCodeToStatus:

case SERVICE_SATURATED -> HttpStatus.SERVICE_UNAVAILABLE;

Without that explicit handler, the rejection would fall through to the catch-all Exception handler and become a 500. Same root cause, completely different operational signal.

Make It Observable

Bounded pools are useless if you can't see how close you are to the wall. We expose five gauges via a MeterBinder, scraped from /actuator/prometheus:

@Bean
public MeterBinder mvcAsyncExecutorMetrics(ThreadPoolTaskExecutor mvcAsyncExecutor) {
    return registry -> {
        Gauge.builder("mvc.async.executor.active",
                mvcAsyncExecutor, ThreadPoolTaskExecutor::getActiveCount)
            .description("Threads currently executing tasks")
            .baseUnit("threads")
            .register(registry);

        Gauge.builder("mvc.async.executor.pool.size",
                mvcAsyncExecutor, ThreadPoolTaskExecutor::getPoolSize)
            .description("Current pool size (grows from core toward max under load)")
            .baseUnit("threads")
            .register(registry);

        Gauge.builder("mvc.async.executor.pool.max",
                mvcAsyncExecutor, e -> (double) e.getMaxPoolSize())
            .description("Configured maxPoolSize ceiling")
            .baseUnit("threads")
            .register(registry);

        Gauge.builder("mvc.async.executor.queue.size",
                mvcAsyncExecutor, e -> {
                    ThreadPoolExecutor tpe = e.getThreadPoolExecutor();
                    return tpe == null ? 0 : tpe.getQueue().size();
                })
            .description("Queued tasks waiting for a thread")
            .register(registry);

        // The headline metric: active / max. 0.8 = alert; 1.0 = saturated.
        Gauge.builder("mvc.async.executor.saturation",
                mvcAsyncExecutor, e -> {
                    int max = e.getMaxPoolSize();
                    return max == 0 ? 0.0 : (double) e.getActiveCount() / max;
                })
            .description("Active-to-max ratio (0..1). >0.8 = capacity warning, 1.0 = rejecting")
            .register(registry);
    };
}

The first three (active, pool.size, pool.max) are diagnostic — useful in a Grafana dashboard to see how the pool actually grows. queue.size is included for forward compatibility; with queueCapacity = 0 it's always zero, but if we ever tune the queue we don't want to be the team that forgot to publish the gauge.

mvc.async.executor.saturation is the headline. It's a single derived ratio (active / max) bounded between 0 and 1, which means one alert threshold works regardless of how we resize the pool over time. Cardinality of one, semantics of "how close are we to rejecting" — exactly what a pager wants to wake you up about.

Alerts That Actually Mean Something

Two thresholds, both honest:

WARN when mvc_async_executor_saturation > 0.8 for 5 minutes. We're at 80% of the ceiling for long enough to mean it's not a spike. Likely time to bump maxPoolSize, investigate slow async handlers, or check if SSE clients are leaking connections.
CRIT when mvc_async_executor_saturation == 1.0. We're rejecting. SERVICE_SATURATED 503s are going out the door. Page on-call.

The warning level matters because saturation is a leading indicator. Once you're rejecting, customers have already seen 503s. Once you're at 80%, you have minutes to act before that happens — assuming you alert on it. The corollary: a Grafana dashboard with saturation as the headline tile (not request rate, not latency) tells you at a glance whether you're about to have a bad day.

We pair saturation with the Retry-After header on the 503 so even when we are rejecting, well-behaved clients smooth it out and we recover without a thundering herd hammering us the moment the pool drains.

Lessons Learned

SimpleAsyncTaskExecutor is a development convenience, not a production executor. If you return DeferredResult, Flux, Mono, or use SseEmitter from a Spring MVC controller, configure AsyncSupportConfigurer.setTaskExecutor with a real bounded pool. The default's WARN log message is correct.
Queue capacity is a function of task duration, not of "more is better." For sub-second tasks, a generous queue smooths spikes. For long-lived tasks like SSE, a queue silently hangs new connections behind old ones. Use queueCapacity = 0 + grow the pool to maxPoolSize.
AbortPolicy over CallerRunsPolicy for HTTP work. CallerRunsPolicy pins your Tomcat request threads under load and turns a capacity problem into a hung-process problem. Fail fast, return 503.
Map rejection to a typed error code, not a generic 500. RejectedExecutionException → SERVICE_SATURATED → HTTP 503 + Retry-After tells callers something actionable. 500 tells them nothing.
Saturation is the gauge worth alerting on. A derived active/max ratio is dimensionless, bounded 0..1, survives pool resizing, and gives you a single threshold to define "we're about to have a bad day."
The WARN threshold is the one that prevents pages. Alert at 0.8 with 5-minute hysteresis to get warned before you reject. Alert at 1.0 to know you're already rejecting. Both, not either.

What's your async-executor war story? Tail-latency, thread leaks, OOM? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

How One Spring Boot Default Killed Our SSE Endpoint Under Load

Anand Rathnas — Mon, 21 Sep 2026 06:02:31 +0000

Our staging environment had ten browser tabs open with the notification bell. The eleventh person to open the app got an API that just sat there. Not a 500. Not a timeout (yet). Just a silent stall on every endpoint, not only the SSE one. The pod looked healthy. CPU flat, memory fine, logs quiet.

We'd turned our notification bell into a Server-Sent Events stream while leaving one Spring Boot default in place. For a request-response API, that default is fine. For a 30-minute SSE connection, it's lethal: spring.jpa.open-in-view=true pins a Hikari connection per open tab for the entire lifetime of the stream.

Here's the bug, the math behind the symptom, and the one-line fix — plus the precondition the endpoint has to satisfy for the fix to be safe.

The Endpoint

The SSE endpoint lives in NotificationController.streamNotifications(). It returns a Flux<ServerSentEvent<NotificationStreamEvent>> to the client and stays open for 30 minutes, emitting the initial unread count, live updates from the event bus, and a heartbeat every 30 seconds:

@GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public ResponseEntity<Flux<ServerSentEvent<NotificationStreamEvent>>>
        streamNotifications() {
    authContext.requireScope("read");
    UserEntity user = authContext.getCurrentUser();
    Long userId = user.getId();

    ServerSentEvent<NotificationStreamEvent> initial = ServerSentEvent
        .<NotificationStreamEvent>builder()
        .event("unread-count")
        .data(NotificationStreamEvent.unreadCount(
            notificationService.getUnreadCount(userId)))
        .build();

    Flux<ServerSentEvent<NotificationStreamEvent>> events =
        notificationEventBus.subscribe(userId)
            .map(event -> ServerSentEvent.<NotificationStreamEvent>builder()
                .event(event.type()).data(event).build());

    Flux<ServerSentEvent<NotificationStreamEvent>> heartbeat =
        Flux.interval(HEARTBEAT_INTERVAL)
            .map(tick -> ServerSentEvent.<NotificationStreamEvent>builder()
                .comment("ping").build());

    Flux<ServerSentEvent<NotificationStreamEvent>> body = Flux.concat(
            Flux.just(initial), Flux.merge(events, heartbeat))
        .take(STREAM_LIFETIME);

    return ResponseEntity.ok().headers(headers).body(body);
}

Two numbers matter: STREAM_LIFETIME = Duration.ofMinutes(30) and HEARTBEAT_INTERVAL = Duration.ofSeconds(30). The stream stays open for half an hour. The client reconnects on close with a fresh JWT. The heartbeat keeps it alive through any proxy idle-timeout in between.

From the controller's perspective this looks innocuous. It's a Flux. It's reactive. It touches JPA once, synchronously, at the top, to seed the initial count. The Flux body that streams for the next 30 minutes never touches the database.

That's what we thought the runtime behavior was. The runtime had other ideas.

open-in-view: Spring's Helpful Default

spring.jpa.open-in-view=true is Spring Boot's default. It enables OSIV — Open Session In View. The mechanism is an interceptor that opens a Hibernate Session (and, through it, borrows a database connection from the Hikari pool) at the start of an HTTP request, and holds it open until the response is fully written.

For traditional MVC this is convenient. Lazy-loaded JPA associations still resolve when your Thymeleaf template or JSON serializer touches them after the controller returns. No LazyInitializationException — at the cost of holding a DB connection for the lifetime of the HTTP response.

For a request-response endpoint serving JSON in 50ms, "lifetime of the HTTP response" is 50ms. The connection round-trips back to the pool before the next request even arrives. Pool pressure is essentially zero and the convenience is large enough that the Spring team made it default-on.

Why It's Lethal for SSE

Server-Sent Events flips the lifetime assumption on its head.

The HTTP response on an SSE endpoint is not 50ms. It's the entire duration of the open stream — 30 minutes in our case. As long as the tab is open and the stream is live, the response is still being written one event at a time, and the servlet container hasn't released the request or the response.

OSIV doesn't know any of that. It just sees a request that's still in flight, so the session must still be open, so the connection it borrowed from Hikari must still be held.

One open SSE stream = one Hikari connection pinned for 30 minutes. The connection isn't being used. Nothing is running queries on it. It's reserved, sitting idle, unavailable to anyone else, until the response finally ends — at which point the user reconnects with a fresh JWT and the pinned-connection clock resets.

The Symptom Math

Hikari's default maximumPoolSize is 10. Spring Boot doesn't override it. Most apps don't either, because for a normal API the default is more than enough — connections cycle back in tens of milliseconds and you'd need hundreds of concurrent requests to feel any pressure.

Now run the math with OSIV + SSE:

Open SSE tabs	Hikari connections held	Connections free for other work
1	1	9
5	5	5
9	9	1
10	10	0
11	10	0 (request 11 waits)

The 11th SSE subscriber's request asks OSIV for a connection. There are none. Hikari blocks the request thread on the pool, waiting for one to free up. Nothing will free up for the next 29 minutes and 50 seconds. Eventually Hikari hits its connection-timeout (default 30 seconds) and throws SQLTransientConnectionException: HikariPool-1 - Connection is not available.

The cruel part: it's not just the SSE endpoint that hangs. Every endpoint that needs a DB connection — login, the click handler, the dashboard fetch — competes for the same pool. Ten pinned SSE tabs from a single demo session, and the whole API is wedged. The JVM is alive, nothing is crashing, everything is just queuing on an empty pool. You stare at flat CPU graphs for an hour and feel personally betrayed by your observability stack.

The Fix

One line in application-database.yaml:

spring:
  jpa:
    open-in-view: false

That disables the OSIV interceptor. Connections are no longer held for the lifetime of the HTTP response — they're borrowed when a transaction starts, returned when it ends. The 30-minute SSE stream now holds zero connections for the 30 minutes it's open. Hikari pool is unblocked. The 11th tab works. The 1000th tab works.

Spring's startup log even warns about OSIV being on by default, but it's easy to scroll past in a noisy log. On a normal MVC app it's not a problem worth acting on. On an app with long-lived reactive endpoints, it absolutely is.

The Precondition That Makes the Fix Safe

You can't just flip open-in-view: false and call it done. Turning OSIV off changes a semantic: lazy-loaded JPA associations outside an explicit transaction now throw LazyInitializationException. The endpoint has to be structured so it doesn't depend on that lazy-load convenience.

Our SSE endpoint already is, intentionally. Look at where it touches JPA:

ServerSentEvent<NotificationStreamEvent> initial = ServerSentEvent
    .<NotificationStreamEvent>builder()
    .event("unread-count")
    .data(NotificationStreamEvent.unreadCount(
        notificationService.getUnreadCount(userId)))   // <-- only JPA touch
    .build();

That one call. It runs synchronously at subscribe-time inside the @Transactional boundary of NotificationService.getUnreadCount(...). The transaction opens, borrows a connection, runs the count query, returns a long, commits, releases the connection. Total time: a few milliseconds.

After that, every event flowing through the Flux body — live events from notificationEventBus.subscribe(userId), the heartbeat pings, the close after 30 minutes — emits a plain in-memory NotificationStreamEvent record. No JPA. No lazy proxies. Nothing the persistence context needs to resolve.

The Javadoc on the endpoint spells out this contract explicitly:

That paragraph exists because the fix is one config flag away from being silently undone by a well-meaning future contributor who reads "Spring Boot default" and decides to align. The Javadoc is the load-bearing comment for the entire endpoint.

Lessons Learned

OSIV is a request-response convenience. It pays for itself in a 50ms JSON endpoint. It bankrupts you in a 30-minute SSE stream. The cost is "connection held for the lifetime of the HTTP response," and on a long-lived response that's the whole stream.
Hikari's default pool of 10 is generous for request-response and tiny for anything that pins connections. The default implicitly assumes you're returning connections quickly. If you're not, you usually only learn this in production at the eleventh concurrent subscriber.
Pool exhaustion makes everything hang, not just the slow endpoint. The DB pool is shared. SSE pinning it starves every other endpoint. The trail leads back to SSE only after you check pool stats.
spring.jpa.open-in-view: false is right for any app with long-lived reactive endpoints, but only safe if those endpoints don't depend on lazy-loading outside an explicit transaction. Audit first; don't flip the flag blind.
Document the precondition where the bug would re-enter. A one-paragraph "Pool contract" Javadoc on the endpoint is the difference between a fix that holds and one someone undoes in six months while "cleaning up config to match Spring defaults."
Reactive controllers on a servlet stack are a special kind of trap. Reactive types make you think in non-blocking lifetimes; the underlying servlet container is still tracking the HTTP request as in-flight for the full duration, and request-scoped infrastructure like OSIV happily plays along.

Has OSIV bitten you somewhere unexpected? Drop the symptom in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Grafana as Code: Surviving Destroy-Recreate With Provisioning Files

Anand Rathnas — Fri, 18 Sep 2026 05:43:16 +0000

Liquid syntax error: Unknown tag 'endraw'

Watchtower Tried to Upgrade Itself. It Didn't Come Back.

Anand Rathnas — Wed, 16 Sep 2026 05:50:05 +0000

On May 19 we pushed a fix to main. GitHub Actions built the image, tagged it :latest, shoved it into GHCR, and went home. Our droplet was supposed to pick that up within sixty seconds. It didn't. Not in sixty minutes, not in two days.

When we finally SSH'd in to ask why, the answer was in docker ps: our auto-update tool wasn't running. It had tried to update itself, succeeded at the "stop" half, and never made it to the "start" half. Nobody noticed for forty-eight hours because the API was up; only the deploy pipeline was silently dead.

This is the story of that incident and the two-line fix that makes it structurally impossible to repeat.

What Watchtower Is For

If you've never used Watchtower, it solves a small, useful problem: you push a new image to a registry, and Watchtower running on your host detects the new digest, pulls it, gracefully stops the old container, and starts a fresh one. No SSH, no docker compose pull && up -d, no human in the loop. For a tiny team on a single droplet, it's a five-line replacement for an entire CD pipeline. You point it at GHCR, give it the docker socket, and forget it exists.

Forgetting it exists is exactly the problem.

The Incident

Here's what we pieced together after the fact.

Our watchtower service was running :latest — the version everyone copy-pastes from blog posts. On May 19, upstream pushed a new :latest. Watchtower, doing its job, polled the registry, saw the new digest, and decided to update itself.

To update a container, Watchtower stops it, then starts the replacement. The replacement is launched by Watchtower. Which is the container being stopped.

You can see the deadlock from orbit. Watchtower issued docker stop against its own container. The Docker socket connection it was using to issue the "now start the new one" command got severed the instant the container died. The new container never launched. The host was left with no Watchtower at all — just the old API container, happily serving stale code, and no process polling the registry for new builds.

The comment we eventually added to docker-compose.yml captures it best:

Two days. We only caught it because someone noticed a bug fix that was definitely in main was definitely not in production.

Why It Couldn't Recover

Natural question: why didn't restart: unless-stopped save us? It's right there in the compose file.

restart policies fire when a container crashes — exits with a non-zero code, or the daemon restarts. They do not fire when a container is intentionally stopped via the API. From Docker's perspective, somebody asked Watchtower to stop. It stopped. Nothing to restart. The fact that the "somebody" was Watchtower itself — and that the next command in the sequence never ran because the issuer evaporated — is invisible to the daemon.

The deeper problem is that nothing was watching the watcher. Every other container on the host had Watchtower as its safety net. Watchtower had nothing. It was the bottom turtle, and when the bottom turtle quietly walks off, the whole stack just sits there.

The Two-Layer Fix

We fixed it twice, on purpose. The full Watchtower service now looks like this:

  # Watchtower - Auto-updates containers when new images are pushed to GHCR
  # Eliminates need for SSH-based deployments
  # NOTE: Using nickfedor/watchtower fork (containrrr/watchtower is archived
  # and incompatible with Docker 24+)
  #
  # IMPORTANT: image is PINNED, not :latest. On 2026-05-19 Watchtower
  # self-detected an updated :latest, tried to stop itself to upgrade, cut
  # its own Docker-socket connection mid-stop, and never restarted —
  # leaving this droplet on stale code for 2 days until manual intervention.
  # Bump this tag explicitly to upgrade Watchtower itself; do NOT track :latest.
  watchtower:
    image: nickfedor/watchtower:1.17.0
    container_name: jo4-watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json:ro
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_POLL_INTERVAL=60
      - WATCHTOWER_INCLUDE_STOPPED=false
      - WATCHTOWER_REVIVE_STOPPED=false
      - WATCHTOWER_ROLLING_RESTART=false
      # Opt-in mode: only watch containers labeled
      # com.centurylinklabs.watchtower.enable=true (i.e. jo4-api).
      - WATCHTOWER_LABEL_ENABLE=true
    restart: unless-stopped

And on the app service we want it to actually watch:

  app:
    image: ghcr.io/${GITHUB_REPOSITORY:-rathnasorg/jo4}/alertstage-api:${APP_VERSION:-latest}
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

Two layers.

Layer 1: Pin Watchtower's own image

image: nickfedor/watchtower:1.17.0. Not :latest. Not :stable. A specific tag we wrote down on purpose.

This single change makes the May 19 sequence impossible. When Watchtower asks "is there a newer image for nickfedor/watchtower:1.17.0?" the answer is, by definition, no. The only way Watchtower gets a new image is when we edit this file and bump the version — a human in the loop who can verify the new container actually came up.

(We also moved off containrrr/watchtower while here — that repo is archived and broke on Docker 24+. nickfedor/watchtower is the maintained fork.)

Layer 2: Opt-in label mode

WATCHTOWER_LABEL_ENABLE=true flips Watchtower from "watch every container on the host" to "watch only containers carrying the label com.centurylinklabs.watchtower.enable=true." We add that label to jo4-api. We do not add it to jo4-watchtower. The Watchtower container is now structurally invisible to itself.

Why Both Layers Matter

You could argue either layer alone is enough. Pin the image, no more updates, no more deadlock. Or use label-mode, Watchtower can't pick itself up regardless of tag. Pick one and move on.

We didn't, because each layer protects against a different human error six months from now.

Layer 1 fails the day somebody — maybe one of us, on autopilot — edits the compose file and changes 1.17.0 back to :latest. It's the kind of "cleanup" that looks reasonable in a PR diff. If layer 2 isn't there, that edit reintroduces the exact bug.

Layer 2 fails the day somebody adds the watchtower.enable=true label to the Watchtower container itself — maybe they're labelling everything for an inventory script, maybe a tutorial says "label all your containers." If layer 1 isn't there, Watchtower starts watching itself again.

Either failure alone is a foot. Both simultaneously, on the same PR, by the same engineer, is a foot wrapped in a kevlar boot. Defense in depth is cheap when the depth is two YAML lines.

Lessons For Any Auto-Update Tool

The Watchtower-specific story generalizes. Anything that auto-updates itself — your dependency bot, your CI runner image, your sidecar agent, your secrets-rotator — has the same shape of bug latent in it. Some patterns we now apply across the board:

Pin the version of anything that can mutate itself. Auto-update tools should not auto-update. Their version belongs in source control where a human has to type the new digit. :latest is for things being watched, not for things doing the watching.
Prefer opt-in over opt-out for blast radius. Default-watch-everything tools sweep up the watcher itself, your debug containers, your one-off docker run experiments. Opt-in lists are slightly more verbose and dramatically harder to misfire.
restart: unless-stopped is not a safety net for orchestration bugs. It catches crashes. It does not catch "I stopped myself on purpose and then died." If something in your stack has the authority to stop itself, give it a watchdog or accept it will walk off.
Silent CD failure is the worst kind. The API was up the whole time. Health checks green. Monitoring happy. The only symptom was "fixes don't reach production," which nobody catches until they go looking. Add an alert on "no successful deploy in N hours" — it would have paged us at hour 13 instead of hour 48.
Write the comment you'd want six months from now. The block we added to docker-compose.yml is the entire postmortem in five sentences. The next operator who looks at that file knows exactly why the version is pinned and what happens if they unpin it. More durable than any wiki page.

The footgun isn't gone — we still trust an external tool to mutate our running containers. But the specific path where the trust loops back on itself is closed off twice. That's enough.

Have you been bitten by a self-updating tool? What recovered you? Drop the war story in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Scrape Prometheus Over the VPC, Not Through Cloudflare

Anand Rathnas — Mon, 14 Sep 2026 05:58:35 +0000

We built three layers of defense for our Prometheus scrape. Two of them were unnecessary.

This isn't a "we found a bug" post. It's a "we designed the wrong architecture, then corrected it" post. The system worked. It was just doing far more than it needed to. Sharing it because the wrong path was the obvious-looking one, and the right path was sitting right there in our DO project the entire time.

The Original Setup

We have two droplets in production: jo4-server (Spring Boot API behind nginx) and jo4-impress (Prometheus + Grafana + Loki, our internal observability box). When we stood up Prometheus, the question was: how should it scrape /actuator/prometheus on the API?

The obvious answer — and the one we went with — was "the same way every other request reaches the API." Which means:

jo4-impress → Cloudflare edge → DO firewall → nginx → Spring Boot
              (WAF rule)        (CF CIDRs only)

Three defense layers stacked on the public path:

A Cloudflare WAF rule restricting /actuator/* to a tiny allowlist (our impress droplet's egress IP plus a couple of operator IPs).
An OAuth2 client_credentials grant (RFC 6749 §4.4) on the actuator endpoints, scope metrics:read, enforced at the Spring Security layer.
A DigitalOcean cloud firewall on jo4-server restricting inbound :443 to Cloudflare's published CIDR ranges, so nobody can bypass the WAF by hitting the origin directly.

It worked. The scrape ran every 30 seconds, dashboards lit up, alerts fired. We patted ourselves on the back for being thorough.

But every scrape was: TLS handshake with Cloudflare, WAF evaluation, second TLS handshake with nginx, OAuth token mint (cached, but still), HTTP round-trip back through the same chain. All to move ~80 KB of metrics between two boxes that were already neighbors.

The Realization

About a month in, I was running unrelated doctl commands and noticed both droplets in the same row of output:

$ doctl compute droplet list --format Name,PrivateIPv4,Region
ID           Name           Private IPv4    Region
123456789    jo4-server     10.108.0.3      nyc3
987654321    jo4-impress    10.108.0.5      nyc3

Same region. Adjacent private IPs in the same /16. They had been in the same VPC the entire time — 10.108.0.0/16, the default DO VPC for nyc3.

DigitalOcean droplets in the same VPC can reach each other on private interfaces with no public exposure, no firewall hops, no encryption-in-transit requirement (the VPC is a private L2 segment). Cloudflare didn't need to be on the path at all. The DO firewall's CF-CIDR rule was protecting traffic that didn't have to traverse the public internet in the first place.

I had built a public path to talk between two machines on the same private network. That's the kind of thing you laugh at in someone else's design review.

The Simpler Architecture

The corrected topology:

jo4-impress (10.108.0.5) ──VPC──► jo4-server (10.108.0.3:8080)
                                  └─► OAuth still enforced at app layer
                                  └─► DO firewall: tcp/8080 from tag:impress

One hop. No Cloudflare. No TLS termination. No nginx. Spring Boot listens on :8080 (it already did, for the nginx upstream), and Prometheus connects directly via the private IP.

The crucial check before pulling this trigger: does the OAuth gate still apply when the request arrives via the private interface, or was it only enforced because Cloudflare's WAF was upstream? This is the kind of question where "I'm pretty sure" gets you fired. We verified by hand:

# From inside jo4-impress, hit the private IP with no bearer:
$ curl -i http://10.108.0.3:8080/actuator/prometheus
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer ...

Good. Spring Security doesn't care which interface the request came in on — the filter chain runs the same way for private and public traffic. The OAuth requirement is a property of the endpoint, not of the network path. The WAF rule had been belt-and-braces; the actual auth boundary was always at the app.

Token minting moved to the private IP too, since the token endpoint lives on the same Spring Boot app:

token_url: http://10.108.0.3:8080/oauth/token

The Diff

The whole change is in prometheus.yml. Before:

- job_name: jo4-api
  metrics_path: /actuator/prometheus
  scheme: https
  oauth2:
    client_id: __PROMETHEUS_OAUTH_CLIENT_ID__
    client_secret_file: /etc/prometheus/oauth-client-secret
    token_url: https://jo4-api.jo4.io/oauth/token
    scopes:
      - metrics:read
  static_configs:
    - targets: ['jo4-api.jo4.io']

After:

- job_name: jo4-api
  metrics_path: /actuator/prometheus
  scheme: http
  oauth2:
    client_id: __PROMETHEUS_OAUTH_CLIENT_ID__
    client_secret_file: /etc/prometheus/oauth-client-secret
    token_url: http://10.108.0.3:8080/oauth/token
    scopes:
      - metrics:read
  static_configs:
    - targets: ['10.108.0.3:8080']

Three lines changed: scheme, token_url, and the target. The OAuth block is identical because the auth gate is identical — same client, same secret, same scope. Prometheus didn't know or care that the underlying network had been simplified out from under it.

http instead of https looks scary written down, but it's only scary if you treat the VPC like the public internet. DO's VPC is a private network segment between droplets in the same project and region; the threat model that justifies TLS on the open web (passive observers, route hijacks, captive portals) doesn't apply. We still terminate TLS at Cloudflare for everything that reaches users — this just isn't one of those things.

What Stays as Defense-In-Depth

We didn't strip the auth gate. Three layers remain, but now they're actually layered on top of a sound primary:

Primary boundary: VPC isolation. The scrape traffic physically cannot leave DO's nyc3 private network. There is no public path for this byte stream.
App-layer OAuth. Same metrics:read scope, same client_credentials grant. If somehow another droplet ended up in the VPC and tried to scrape, it would get a 401 without the bearer.
DO cloud firewall, tag-based. On jo4-server, we added tcp/8080 inbound from tag:impress. Tag-based rules follow droplets through rebuilds and IP changes — no hardcoded CIDR maintenance.
WAF rule + public exposure on :443. Kept for now. Not reached in practice (Prometheus doesn't hit it anymore), but the public actuator path still exists for emergency operator access. We'll probably retire it next quarter once we're confident nothing else depends on it.

The point isn't "fewer layers" for its own sake. The point is each remaining layer is on the actual path and earns its keep. Two of the original three were guarding against attacks on a public surface that didn't need to exist.

Lessons Learned

Check the topology before the rules. I added three security layers to a public path before asking whether the path needed to be public. The DO VPC was a one-line doctl query away. I just never thought to look.
Defense-in-depth is good. Defense-on-the-wrong-network-path is overhead. Every layer has a cost — config, latency, debuggability, things that break during outages. Layers should defend something real.
Verify the auth boundary is at the app, not the edge. Before deleting the public path, we curled the private IP without a bearer and confirmed the 401. If Spring Security had been relying on a Cloudflare-injected header (it wasn't, but it could have been), this refactor would have silently disabled our auth gate.
Tag-based firewall rules > IP-based rules. tcp/8080 from tag:impress survives droplet rebuilds. tcp/8080 from 10.108.0.5/32 survives until the next time the droplet's IP changes and you forget.
Be honest in the comments. The prometheus.yml comment block now narrates the actual topology and the actual primary boundary (VPC), with the auxiliary layers labeled as such. Future-me reading this in two years will know what's load-bearing and what's belt-and-braces.
It's fine to ship the wrong architecture and fix it. The wrong one worked. We had metrics, we had alerts. Nothing was broken. But "works" and "right" aren't the same thing, and shipping the simpler version when you notice the gap is a win, not an embarrassment.

When have you over-engineered a network path? What pushed you back to simpler? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Why Your Grafana 5xx-Rate Alert Spams DatasourceNoData Every 4 Hours

Anand Rathnas — Fri, 11 Sep 2026 05:45:05 +0000

Liquid syntax error: Unknown tag 'endraw'

Self-Healing Cloudflare CIDRs in DigitalOcean Firewalls

Anand Rathnas — Wed, 09 Sep 2026 05:50:35 +0000

Go grep your infra repo for 173.245.48.0/20. If that string is in a firewall rule, a Terraform module, or a doctl invocation, you have a time bomb. Cloudflare publishes the canonical list of edge IPs at two URLs, and they change. Not often — but often enough that a hardcoded copy quietly goes stale, and on the day a new CIDR ships, a slice of your real visitors gets dropped at the origin firewall with no log entry that obviously means "we're filtering them out."

We had this problem on the DigitalOcean firewall in front of our self-hosted monitoring droplet. The fix isn't complicated, but it has to be wired through the deploy step that actually applies the firewall, not a sidecar cron. Here's the pattern.

Why Hardcoded CIDRs Go Stale

Cloudflare's list at https://www.cloudflare.com/ips-v4 and https://www.cloudflare.com/ips-v6 is the source of truth for which edge IPs may legitimately terminate TLS for your zone. They publish about 15 v4 ranges and 7 v6 ranges at the time of writing. That list is stable for long stretches and then changes — a new range gets added when CF expands a POP, an old block gets retired in a network migration, sometimes a chunk shifts to a different aggregation prefix.

The standard hardcoded approach looks fine the day you ship it and decays silently. Six months in, somebody adds you to a new CF range, your DO firewall says "not on the list," and the request never makes it past the origin firewall. From the user's perspective: random subset of traffic, no useful error, retries sometimes work (because they hit a different CF edge on the way back), nothing in your application logs because the packet never reached your app. This is the worst class of bug — invisible to your observability stack because your observability stack is downstream of the failure.

The right fix is to make drift impossible by construction. Fetch the list live every time you write the firewall.

The Workflow Step

This is the actual step from our DigitalOcean-490-Impress-Bootstrap workflow. It runs every time we bootstrap or re-bootstrap the monitoring droplet, and the firewall rules are reconciled atomically against whatever Cloudflare published thirty seconds ago.

First the SSH base rule, sourced from a GitHub variable that holds operator home IPs:

FIREWALL_NAME="jo4-impress-firewall"
DROPLET_ID=$(doctl compute droplet list --format Name,ID --no-header \
  | awk -v n="$DROPLET_NAME" '$1==n {print $2}')

# Build base tcp/22 rule from operator allow-list (GH variable).
# Format `1.2.3.4/32,5.6.7.8/32` → `address:1.2.3.4/32,address:5.6.7.8/32`.
OPERATOR_SSH=$(echo "$HOME_IPS_FOR_SSH" | tr ',' '\n' | grep -v '^$' \
              | sed 's/^/address:/' | paste -sd, -)
if [ -z "$OPERATOR_SSH" ]; then
  echo "::error::HOME_IPS_FOR_SSH parsed to empty — refusing to deploy a no-SSH firewall (would lock everyone out)"
  exit 1
fi
INBOUND_22="protocol:tcp,ports:22,${OPERATOR_SSH}"

Now the live Cloudflare fetch and sanity check:

# Fetch live Cloudflare CIDRs for tcp/443. Fail loudly if either
# endpoint is unreachable — we'd rather block this workflow run
# than silently apply a half-empty allow-list.
CF_V4=$(curl -fsSL --max-time 10 https://www.cloudflare.com/ips-v4)
CF_V6=$(curl -fsSL --max-time 10 https://www.cloudflare.com/ips-v6)
if [ -z "$CF_V4" ] || [ -z "$CF_V6" ]; then
  echo "::error::Cloudflare published-IP fetch returned empty"
  exit 1
fi
V4_COUNT=$(echo "$CF_V4" | grep -c .)
V6_COUNT=$(echo "$CF_V6" | grep -c .)
if [ "$V4_COUNT" -lt 10 ] || [ "$V6_COUNT" -lt 5 ]; then
  echo "::error::Cloudflare CIDR fetch implausibly small: v4=$V4_COUNT v6=$V6_COUNT"
  exit 1
fi
echo "📡 Cloudflare published $V4_COUNT v4 + $V6_COUNT v6 ranges"
INBOUND_443=$(printf '%s\n%s\n' "$CF_V4" "$CF_V6" | grep -v '^$' \
              | sed 's/^/address:/' | paste -sd, -)
INBOUND_443="protocol:tcp,ports:443,${INBOUND_443}"

Two curl calls. A line count per family. A floor. Then a single comma-joined rule string in the format doctl wants. Five seconds of work in the runner, and the rule we're about to ship is sourced from CF's live publication rather than from somebody's memory of CF's publication on the day they wrote the workflow.

Finally the atomic apply:

# Always reconcile (cheap, idempotent). `firewall update` is
# atomic PUT — replaces all rules in one call, no remove-then-add
# gap. This also wipes any stale runner-IP rule from a prior
# run whose cleanup step failed.
echo "♻️  Reconciling rules on existing $FIREWALL_NAME ($FIREWALL_ID)..."
doctl compute firewall update "$FIREWALL_ID" \
  --name "$FIREWALL_NAME" \
  --inbound-rules "$INBOUND_22 $INBOUND_443" \
  --outbound-rules "$OUTBOUND" \
  --droplet-ids "$DROPLET_ID"

That's the whole thing. Every run, every bootstrap, every redeploy: the firewall converges on whatever CF published just now.

The Sanity Floor

The two-line floor is the only piece of this that needs defending, because it looks defensive-in-a-way-that-might-be-paranoid:

if [ "$V4_COUNT" -lt 10 ] || [ "$V6_COUNT" -lt 5 ]; then
  echo "::error::Cloudflare CIDR fetch implausibly small: v4=$V4_COUNT v6=$V6_COUNT"
  exit 1
fi

We know CF publishes ~15 v4 and ~7 v6 ranges. The floor of 10/5 is "well below the real number, well above zero." The threat model isn't "CF is hacked and publishes a tiny list" — it's much more pedestrian:

A transient blip at cloudflare.com returns a 200 with a partial body. curl -fsSL is happy because the status was 2xx. The CIDR list comes back truncated.
A future change to the publication format — a comment header, a JSON wrapper, an HTML maintenance page — survives curl and starts feeding garbage into our grep-and-sed pipeline.
A network hiccup mid-stream that delivers some lines and then EOFs cleanly.

Without the floor, any of those scenarios deploys a firewall with a one- or two-CIDR allow-list. Result: we just locked out 95% of legitimate Cloudflare traffic. The floor catches the "looks like a successful fetch but the content is wrong" failure mode, which is the only mode where this whole system fails dangerously. If CF actually shrinks their published list below 10 v4 ranges, somebody needs to look at it manually anyway, and the workflow failing is the right way to find out.

Why `doctl firewall update` Is Right

The other detail worth slowing down on is the choice of doctl compute firewall update over the more obvious-looking remove-rules + add-rules pair. The DO API's update endpoint is an atomic PUT — it replaces the entire rule set in one call. There is no window where the firewall has the old rules removed but the new rules not yet added.

The remove-then-add pattern has exactly that window. It might be 50ms. It might be 5 seconds if the API is slow. During that window, requests that should have been allowed by the new rules are denied because neither the old nor the new rules are in effect. With a high-traffic origin, "5 seconds of half-denied traffic" is a non-trivial number of failed user requests every time you reconcile.

The PUT semantics also give us a free side-effect: any stale ephemeral rule from a prior run gets wiped. Our workflow also temporarily adds the runner's public IP to the SSH rule for the duration of the job (so the runner can ssh into the droplet), then revokes it in an if: always() cleanup step at job end. If that cleanup step ever fails — doctl API blip, runner killed mid-step — the stale runner-IP entry would normally accumulate. With atomic PUT reconcile, the next run's firewall update overwrites whatever was there with a fresh rule set built from scratch. Self-healing, no janitor cron needed.

What This Replaces

The old pattern is one of:

Hardcoded list in the workflow. Decays. No alarm rings the day CF adds a new range. You find out from a customer.
A separate cron that updates the firewall daily. Adds a moving part. Has its own credentials, its own failure modes, its own observability gap. The cron breaks, you don't notice, you're back to the hardcoded-and-stale problem with extra complexity on top.
A Terraform module pinned to a CF data source. Closer, but only as fresh as your last terraform apply. If you only re-apply infra changes every few months, you've reintroduced the staleness window.

The pattern above sidesteps all of them: the firewall converges on CF's live list every time the bootstrap workflow runs, and the bootstrap workflow runs every time we change anything about the droplet. Drift is bounded by deploy cadence, not by anybody remembering to check.

There's nothing exotic here — two curl calls, a sanity check, one atomic doctl call — but the composition is the point. Live fetch + floor + PUT semantics gives you a firewall that's right by construction, not right by vigilance.

How do you keep upstream IP allow-lists fresh? What's worked for your cloud + edge provider? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Your Sentry DSN Is a GitHub Variable, Not a Secret

Anand Rathnas — Mon, 07 Sep 2026 05:53:31 +0000

Liquid syntax error: Unknown tag 'endraw'

Grafana 11 Provisions Alerts Before Dashboards — and That Breaks folderUid

Anand Rathnas — Fri, 04 Sep 2026 05:49:41 +0000

We rolled a tidy little change into our Grafana provisioning: pin the dashboard folder's UID so deep links don't break across redeploys. Restarted the container. Watched it exit 1 with this:

failed to create folder for provisioned dashboards" folder=jo4 ... err="a folder with the same name already exists in the current location"

That error makes no sense. We hadn't created any folder called jo4 anywhere else. The provisioning files in /etc/grafana/provisioning/dashboards/ are the only place the name appears. So who created the first one?

The answer turned out to be: Grafana did. Before our dashboard provider ran. And it's not what the docs imply.

What We Expected

The Grafana provisioning docs are organized in roughly the order we'd expect a fresh install to come up:

Datasources (so dashboards have something to query)
Dashboards (the things humans look at)
Alerting (rules that fire against dashboards/queries)

Our providers.yaml set the folder name and the folder UID, on the reasonable theory that pinning the UID protects external deep links from rotating after a rebuild:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    folderUid: jo4-folder           # <-- our innocent addition
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards/jo4

If dashboards are provisioned before alerting, this is fine. The dashboard provider creates a folder named jo4 with UID jo4-folder. Alerting provisioning runs later, references folder: jo4 by name, matches the existing one. Everyone goes home.

That is not what Grafana 11.2.0 actually does.

What Actually Happens

We spun up a sandbox container, dropped in the exact same provisioning tree, and tailed the logs. Here's the relevant window, milliseconds and all:

06:26:57.106  msg="Initialising datasources"
06:26:57.204  msg="starting to provision alerting"
06:26:57.322  msg="finished to provision alerting"
06:26:57.344  msg="starting to provision dashboards"
06:26:57.346  level=error msg="failed to create folder for provisioned dashboards"
              folder=jo4 ... err="a folder with the same name already exists in the current location"
06:26:57.346  msg="Stopped background service" service=*provisioning.ProvisioningServiceImpl
              reason="failed to provision dashboards"
06:26:57.346  level=fatal msg="Server shutdown" reason="..."

Three things to read off this:

Alerting provisions before dashboards. It's not even close — alerting starts 140ms before dashboards and finishes 22ms before dashboards even start. This is the opposite of how the docs are organized.
The alerting provisioner created the folder. Two milliseconds into dashboard provisioning, the error is already firing.
The whole provisioning service stops. Not "skip this dashboard provider and continue." It stops everything. The Grafana process then exits 1 because provisioning is a required background service. The container restart-loops.

Why

Once you accept the ordering, the rest falls out cleanly.

The alerting provisioner takes a folder field — a name, not a UID. There is no folderUid field accepted in alerting rule provisioning files. When alerting runs first and sees folder: jo4, it needs that folder to exist. It doesn't. So Grafana creates it, with a UID it generates on the spot (something like ffn7ba7vmp3i8a — looks random, is deterministic for this install).

Now the dashboard provisioner runs. Our providers.yaml says: "make sure there's a folder named jo4 with UID jo4-folder." Grafana does the name lookup first, finds an existing folder named jo4 — but its UID is ffn7ba7vmp3i8a, not jo4-folder. The provisioner's reconciliation logic decides this is a different folder and tries to create one with the pinned UID. The unique-name constraint on folders fires. Provisioning service stops. Container exits 1.

So folderUid in the dashboard provider is fundamentally incompatible with also having alerting rules in the same folder — at least under Grafana 11.2.0's current ordering — because alerting will have created the folder first, with its own auto-generated UID, and your pinned UID can never win the race.

The Fix

One removed line. The dashboard provider keeps folder: jo4 (the human-readable name) and drops folderUid entirely:

Before:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    folderUid: jo4-folder
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/jo4
      foldersFromFilesStructure: false

After:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/jo4
      foldersFromFilesStructure: false

That's it. With no folderUid, the dashboard provider does a name-only lookup, finds the folder alerting already created, reuses it, and provisions all the dashboards into it. The container comes up clean.

We also left a load-bearing comment in providers.yaml so the next person who sees a dangling UID in deep links and thinks "I'll just pin this" gets the full story before they break the deploy:

# IMPORTANT: do NOT set `folderUid`. Grafana 11.2.0 provisioning order is
# datasources → alerting → dashboards (verified empirically). Alerting
# provisioning auto-creates the folder by name (only field accepted) with a
# Grafana-generated UID. If we then pin a different folderUid here, Grafana
# tries to create a SECOND folder with the same name, errors, stops the
# provisioning service, and the container exits 1.

Verifying It Worked

Re-running the sandbox after removing folderUid, here's the timeline:

msg="starting to provision alerting"
msg="finished to provision alerting"
msg="starting to provision dashboards"
msg="finished to provision dashboards"

No error. Process stays up. Then we hit the Grafana API to see who owns the folder:

curl -s -u admin:admin http://localhost:3000/api/folders | jq '.[] | {title, uid}'

Output:

{ "title": "jo4", "uid": "ffn7ba7vmp3i8a" }

And the dashboards inside that folder all report folderUid: ffn7ba7vmp3i8a. So do the alerting rule groups. One folder, one UID, both provisioners pointing at it. Exactly what we wanted — we just don't get to choose the UID.

If you genuinely need a stable folder UID for deep links (we don't, but you might), the only path we've found that works is: let alerting auto-create it, read the resulting UID back via the API once after the first install, and codify it as an external constant. Don't try to pin it through providers.yaml.

Lessons Learned

Documentation ordering is not runtime ordering. The Grafana docs describe provisioning subsystems in the order you'd intuitively initialize them, but the actual startup sequence in 11.2.0 is datasources → alerting → dashboards. If something cross-references between subsystems, find out who runs first the empirical way — read the logs.
A unique-name constraint plus two creators is always a race. Whenever two independent provisioners can both produce the "same" named resource and only one of them lets you specify the UID, the one without UID control wins by going first. Your pinned UID has no path to victory.
Provisioning service failure = container failure. Grafana doesn't gracefully degrade when one provisioning file is bad. The whole service stops and the process exits. Treat your provisioning YAML with the same rigor as a database migration: it can take down the system.
folderUid in a dashboard provider is a foot-gun the moment you also have alerting in the same folder. Drop it. Let the folder be created by whoever runs first and look it up by name everywhere else.
Leave the comment. Future-you, or the next operator, will look at the removed folderUid and wonder why. A five-line comment in the YAML is cheaper than another two-hour debugging session.

Got bit by Grafana provisioning ordering? What was your symptom? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Your sed -i Edit Isn't Reaching the Container: The Bind-Mount Inode Trap

Anand Rathnas — Wed, 02 Sep 2026 05:35:45 +0000

We tweaked a scrape config in our Prometheus YAML, ran sed -i on the host, confirmed the file on disk had the new content, and waited for Prometheus to pick it up. It didn't. An hour later we were staring at a stat output that explained everything we'd ever misunderstood about bind mounts. If you've ever edited a file Docker mounted into a container and watched the container act like nothing happened, this post is for you.

The Symptom

Our Prometheus container has its config bind-mounted from the host:

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - /opt/impress/prometheus.yml:/etc/prometheus/prometheus.yml

We needed to add a scrape target. Easy:

sed -i 's/scrape_interval: 15s/scrape_interval: 30s/' /opt/impress/prometheus.yml
docker exec prometheus kill -HUP 1   # signal Prometheus to reload

The host file looked correct. Prometheus's /-/reload endpoint returned 200. But the runtime config in the UI still showed 15s. We re-ran the SIGHUP. Same. We checked Prometheus logs — it said it had reloaded the config. From its own file. Which apparently still said 15s.

That's the moment we stopped trusting our assumptions and asked the container what it was actually seeing.

Diagnosis: `stat` Tells the Truth

We compared the two views of "the same file":

# On the host
$ md5sum /opt/impress/prometheus.yml
a3f9c1e8...  /opt/impress/prometheus.yml

$ stat /opt/impress/prometheus.yml
  File: /opt/impress/prometheus.yml
  Size: 2847       Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 1835421     Links: 1

# Inside the container
$ docker exec prometheus md5sum /etc/prometheus/prometheus.yml
7b2d4f06...  /etc/prometheus/prometheus.yml

$ docker exec prometheus stat /etc/prometheus/prometheus.yml
  File: /etc/prometheus/prometheus.yml
  Size: 2791       Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 1835098     Links: 0

Two things jumped out:

Different MD5s. The container is reading different bytes than the host file contains.
Links: 0 inside the container. Zero hard links means no directory entry points at this inode anywhere on the filesystem. The file has been unlinked. The container is the only thing keeping the inode alive.

Different inode numbers (1835421 on the host, 1835098 in the container) confirmed it: these aren't the same file anymore. They were, when the container started. They aren't now. The container is holding open a ghost — an inode that exists only because it still has an open file descriptor pointing at it.

Why sed -i Is the Culprit

This is the part of sed -i nobody reads the man page about. sed -i does not edit in place. Despite the flag name, it does this:

Open the target file for reading.
Open a temp file in the same directory (sedXYZ or similar).
Stream the transformed content to the temp file.
rename(2) the temp file over the target name.

That rename(2) is the killer. The original inode (the one with the old content) is no longer reachable by name, but it doesn't get freed because the container has it open. The directory entry /opt/impress/prometheus.yml now points at a brand new inode with the new content. On the host that's the same path with new bytes — looks fine. Inside the container, nothing changed: the bind mount was set up at container start against the original inode, and bind mounts of single files are bound to inodes, not directory entries.

So the container keeps reading the ghost. Forever. Or until you restart it.

This is also why Links: 0. The original directory entry is gone (it now points to the new inode), so the old inode has zero references in the directory tree — but it has one open file descriptor inside the container, so the kernel keeps it alive.

The Fix

There are two correct ways out.

Option A: edit in place for real

You want to write into the same inode, not replace it. cat > file does exactly that — it opens the existing inode with O_TRUNC and writes new content. The inode number doesn't change. The bind mount still points at it. The container sees the update immediately:

# Build new content somewhere, then truncate-and-write the SAME inode
sed 's/scrape_interval: 15s/scrape_interval: 30s/' /opt/impress/prometheus.yml > /tmp/prom.new
cat /tmp/prom.new > /opt/impress/prometheus.yml
rm /tmp/prom.new

Confirm with stat that the inode number didn't change. If it did, you used a tool that does atomic-rename. Back up and try again.

Option B: restart the container

If the file change is paired with anything else (image bump, env-var change, a sibling config edit), just rebuild the container. This is what we did:

docker compose up -d --force-recreate prometheus

--force-recreate gives us a fully fresh container — new bind mounts against current inodes, plus any env-var or compose-file changes get picked up. Less surgical than cat > file, but bulletproof when you're not sure what else drifted.

docker compose restart prometheus also works for the inode problem alone, but it won't pick up env-var or compose-file changes — it restarts the existing container in place.

Tools That Have the Same Trap

sed -i is the famous one, but the trap is everywhere atomic-rename is the safe-write idiom:

mv newfile oldfile — literally rename(2). Same trap.
vim and emacs by default — both write to a backup/swap file and rename over the target, for crash safety. Configurable, but the defaults bite.
perl -i -pe '...' — same temp-file-and-rename dance as sed -i.
awk '...' file > tmp && mv tmp file — the explicit version of the same thing.
Most "safe save" code in editors and language tooling — gofmt -w, prettier --write, black, you name it. They rename for atomicity, which is correct for preventing half-written files on crash, and wrong for bind-mounted single files.

Anything that promises atomic writes is doing this. Atomic writes and bind-mounted single files are fundamentally incompatible.

The Bulletproof Workflow

After this incident we made two rules for editing bind-mounted config:

Prefer directory mounts over single-file mounts. A bind mount of a directory tracks the directory entries inside it, not specific inodes. Atomic-rename inside that directory works the way you expect. The cost is exposing sibling files to the container.
When a single-file mount is unavoidable, edit with cat > file. Build the new content in a tempfile somewhere outside the mounted path, then truncate-and-write the target. Verify with stat that the inode number is unchanged.

For our Prometheus case we kept the single-file mount (we don't want other files in /opt/impress visible to Prometheus) and added a tiny wrapper script that does the cat-redirect dance. No more sed -i on bind-mounted files. Ever.

Lessons Learned

sed -i is a lie. It's not in-place. It's "write a temp file and rename." That rename(2) creates a new inode and orphans the old one.
Bind mounts of single files are inode-bound. Whatever inode the directory entry pointed to at container-start time is the inode the container will hold forever. Replace that inode on the host and the container keeps the old one.
Links: 0 on a file inside a container is the smoking gun. It means the container is the last reference to an inode that's been unlinked from the filesystem. You're looking at a ghost.
md5sum on both sides of the mount is the fastest diagnostic. If host and container disagree, you've hit some flavor of this trap.
Atomic-rename is correct for safety, wrong for bind mounts. Use cat > file for in-place truncate-and-write, or mount the parent directory instead of the single file.
When in doubt, --force-recreate the container. It's the bigger hammer but it's reliable and it picks up env-var changes too.

Ever spent an hour chasing this? What was your tell? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Why pgrep Can't Detect cloud-init's Apt Races on DigitalOcean

Anand Rathnas — Mon, 31 Aug 2026 06:51:34 +0000

If you've ever written a wait loop that polls pgrep apt-get before running your own apt-get, congratulations: you have a race condition. We had one too. On DigitalOcean's Ubuntu 24.04 droplets, our cold-boot setup script would intermittently die on the apt frontend lock — even though we'd "carefully" waited for apt to be idle. Here's why our defensive wait loop was lying to us, and the one line that actually fixed it.

The Setup

We run a small observability droplet — Prometheus + Grafana behind nginx, called jo4-impress. It's bootstrapped from a GitHub Actions workflow that SCPs impress/do-setup/ to the freshly-provisioned droplet and runs setup-impress.sh. The script installs Docker via the convenience script (curl -fsSL https://get.docker.com | sh), then brings up the compose stack.

Because we knew Ubuntu's unattended-upgrades and the apt-daily timers fire shortly after boot, we put a pgrep-based wait loop in front of the Docker install. The idea: poll until no apt-get is running, then run our own. Belt-and-braces. What could go wrong.

The Problem

On a fresh droplet, the script intermittently failed with the classic:

E: Could not get lock /var/lib/dpkg/lock-frontend.
   It is held by process 2746 (apt-get)

The frustrating part: our wait loop reported "no apt-get running" two seconds before this error. We weren't ignoring it. We were checking. And yet PID 2746 — an apt-get we never saw — grabbed the lock the instant our curl get.docker.com | sh invoked its own apt-get update.

Here's the timeline pulled from /var/log/apt/history.log on the droplet (workflow run 26424379616):

Time (UTC)	Event
23:48:13	Droplet boots
23:49:01–27	`apt-get install ca-certificates gnupg curl …` (spawned by `install-do-agent`)
23:49:35–38	`apt-get install do-agent` (still inside `install-do-agent`)
23:49:42	Our pgrep wait loop concludes: "no apt-get running, proceeding"
23:49:43	`curl get.docker.com \
23:49:45	Lock contention error. Holder: PID 2746
23:49:48–49	{% raw %}`apt-get install droplet-agent` completes (this was PID 2746)

PID 2746 was spawned by install-dotty-agent, a DigitalOcean cloud-init vendor script. It started in the four-second gap between install-do-agent finishing and our curl | sh racing into apt-get update. Our pgrep loop caught the quiet moment between two vendor scripts and concluded — correctly, in the instant it sampled — that apt was idle. By the time we acted on that conclusion, the next vendor script had already launched.

The Two Sources of Contention

This is where it gets interesting. On a DigitalOcean Ubuntu 24.04 droplet at first boot, there are two completely independent things competing for the apt lock, and most defenses only address one of them.

(1) The systemd timer units. These are the ones every "first-boot apt fix" tutorial talks about:

unattended-upgrades.service
apt-daily.service, apt-daily.timer
apt-daily-upgrade.service, apt-daily-upgrade.timer

They fire on a randomized schedule shortly after boot. systemctl stop + systemctl mask handles them. This is the well-known half of the problem.

(2) cloud-init's modules-final → config-scripts_vendor stage. This is the half nobody warns you about. DigitalOcean ships two vendor scripts that cloud-init runs during this stage:

install-do-agent — installs the DigitalOcean monitoring agent
install-dotty-agent — installs droplet-agent (the web-console SSH bridge)

Both shell out to apt-get install … directly. They are not systemd units. They don't appear in systemctl list-units. Masking the apt timers does precisely nothing to them, because they were never running through those timers in the first place.

On the run captured above, install-do-agent ran 23:48:29–23:49:41 (71.8 seconds total) and install-dotty-agent ran 23:49:41–23:49:53 (12.2 seconds). Between them: a short window where pgrep apt-get returns empty — but cloud-init is very much still going to launch the next one.

Why the pgrep Loop Failed

The pgrep approach has a structural flaw, not a tuning flaw. Let's be explicit:

# The old wait loop (approximately)
for i in $(seq 1 60); do
  if ! pgrep -x apt-get >/dev/null; then
    echo "apt is free, proceeding"
    break
  fi
  sleep 5
done
apt-get install -y docker-ce  # races whatever cloud-init launches next

The check answers one question: is there an apt-get process running **right now? That's a point-in-time observation. It tells you nothing about whether something about to be launched will run an apt-get in the next few seconds.

Cloud-init's vendor-scripts stage is a sequence. While it's executing the sequence, the gaps between individual apt-get invocations are normal — script setup, package downloads, post-install hooks — and routinely large enough to make pgrep look clean. The wait loop is sampling a process state when it should be checking a higher-level lifecycle state: "is cloud-init done yet?"

The systemd-timer half of the problem made this look like a tuning issue ("just sleep longer", "increase the poll count"). It wasn't. No amount of polling helps when the next contender hasn't been spawned yet.

The Fix

Cloud-init ships a command that answers the actual question:

cloud-init status --wait

It blocks until cloud-init reaches its done state — vendor scripts inclusive. Once it returns, every install-*-agent invocation has completed and won't be spawning more apt-gets. We can install Docker without surprise contenders.

The diff that fixed setup-impress.sh:

- # Wait until any boot-time apt is finished.
- for i in $(seq 1 60); do
-   if ! pgrep -x apt-get >/dev/null; then
-     break
-   fi
-   sleep 5
- done
+ # Defense-in-depth: silence the systemd timer units...
+ APT_UNITS=(
+   unattended-upgrades.service
+   apt-daily.service apt-daily.timer
+   apt-daily-upgrade.service apt-daily-upgrade.timer
+ )
+ systemctl stop "${APT_UNITS[@]}" 2>/dev/null || true
+ systemctl mask "${APT_UNITS[@]}" 2>/dev/null || true
+
+ # Load-bearing: block until cloud-init's vendor scripts finish.
+ echo "⏳ Waiting for cloud-init to finish (DO vendor scripts release apt)..."
+ cloud-init status --wait
+
  echo "📦 Installing Docker via convenience script..."
  curl -fsSL https://get.docker.com | sh
  systemctl enable --now docker
+
+ # Lift the masks once Docker is in.
+ systemctl unmask "${APT_UNITS[@]}" 2>/dev/null || true
+ systemctl enable --now apt-daily.timer apt-daily-upgrade.timer 2>/dev/null || true
+ systemctl enable --now unattended-upgrades.service 2>/dev/null || true

Two things to notice. First: cloud-init status --wait is doing the load-bearing work. It's the one line that closes the race. Second: we still mask the apt timers, but only as defense-in-depth against the scenario where cloud-init has finished but a timer fires inside the Docker-install window. We re-enable them at the end so the droplet's normal patching cadence resumes.

The full context lives in impress/do-setup/setup-impress.sh, commented inline so the next person who reads it doesn't have to re-derive any of this.

Lessons Learned

pgrep answers "is X running now?" — not "will X run in the next ten seconds?" Any defensive wait built around pgrep has this blind spot. Use it for "wait for this specific PID to exit", not "wait for a class of process to be permanently quiet."
On Ubuntu 24.04 cloud images, there are two boot-time apt actors, not one. Systemd timers and cloud-init vendor scripts run independently. Masking the timers and ignoring cloud-init buys you the silence right up until cloud-init speaks.
cloud-init status --wait is the right primitive. It encodes the lifecycle question ("is cloud-init done?") that your script actually wants the answer to. No polling, no false positives, no off-by-a-few-seconds.
Belt-and-braces is fine; one of them has to actually be load-bearing. We kept the systemd masks because they cost us nothing and close a real (if narrower) window. But the masks alone never fixed the bug — cloud-init status --wait did.
When an intermittent bug has a four-second window, the logs from one bad run are worth more than a week of staring at the code. /var/log/apt/history.log on the failing droplet handed us the answer in the form of PID 2746 and an exact second. The fix took ten minutes once we had it.

Hit a similar cloud-init-vs-apt race? Drop the timeline in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

DEV Community: Anand Rathnas

OAuth client_credentials in Spring Boot for Prometheus Scrapes

Why client_credentials Fits

Three Schema Relaxations

Token Issuance

Authentication Subclass

The Path Allowlist

Wiring the Scrape

Testing It End-To-End

Lessons Learned

Bound Spring MVC's Async Executor or Pay for It Under Load

What Spring MVC Defaults To

Why That's Wrong For Real Apps

Bounded Pool Setup

Rejection With Meaning

Make It Observable

Alerts That Actually Mean Something

Lessons Learned

How One Spring Boot Default Killed Our SSE Endpoint Under Load

The Endpoint

open-in-view: Spring's Helpful Default

Why It's Lethal for SSE

The Symptom Math

The Fix

The Precondition That Makes the Fix Safe

Lessons Learned

Grafana as Code: Surviving Destroy-Recreate With Provisioning Files

Watchtower Tried to Upgrade Itself. It Didn't Come Back.

What Watchtower Is For

The Incident

Why It Couldn't Recover

The Two-Layer Fix

Layer 1: Pin Watchtower's own image

Layer 2: Opt-in label mode

Why Both Layers Matter

Lessons For Any Auto-Update Tool

Scrape Prometheus Over the VPC, Not Through Cloudflare

The Original Setup

The Realization

The Simpler Architecture

The Diff

What Stays as Defense-In-Depth

Lessons Learned

Why Your Grafana 5xx-Rate Alert Spams DatasourceNoData Every 4 Hours

Self-Healing Cloudflare CIDRs in DigitalOcean Firewalls

Why Hardcoded CIDRs Go Stale

The Workflow Step

The Sanity Floor

Why doctl firewall update Is Right

What This Replaces

Your Sentry DSN Is a GitHub Variable, Not a Secret

Grafana 11 Provisions Alerts Before Dashboards — and That Breaks folderUid

What We Expected

What Actually Happens

Why

The Fix

Verifying It Worked

Lessons Learned

Your sed -i Edit Isn't Reaching the Container: The Bind-Mount Inode Trap

The Symptom

Diagnosis: stat Tells the Truth

Why sed -i Is the Culprit

The Fix

Option A: edit in place for real

Option B: restart the container

Tools That Have the Same Trap

The Bulletproof Workflow

Lessons Learned

Why pgrep Can't Detect cloud-init's Apt Races on DigitalOcean

The Setup

The Problem

The Two Sources of Contention

Why the pgrep Loop Failed

The Fix

Lessons Learned

Why `doctl firewall update` Is Right

Diagnosis: `stat` Tells the Truth