DEV Community: Anand Rathnas

Reporting UI Errors Without Blowing Up Prometheus

Anand Rathnas — Fri, 16 Oct 2026 05:40:14 +0000

Liquid syntax error: Unknown tag 'endraw'

Alert on Brand-New Errors, Not Thresholds, with Redis

Anand Rathnas — Wed, 14 Oct 2026 05:58:00 +0000

Liquid syntax error: Unknown tag 'endraw'

A Vulnerability Disclosure Policy for a Solo SaaS

Anand Rathnas — Mon, 12 Oct 2026 07:09:40 +0000

Liquid syntax error: Unknown tag 'endraw'

Pacing Bulk Async Jobs Under an LLM Rate Limit

Anand Rathnas — Fri, 09 Oct 2026 07:16:03 +0000

Liquid syntax error: Unknown tag 'endraw'

Versioned Content Fingerprints for Change Detection

Anand Rathnas — Wed, 07 Oct 2026 07:02:52 +0000

Liquid syntax error: Unknown tag 'endraw'

Rate-Limit Failed Logins Without Locking Out Victims

Anand Rathnas — Wed, 30 Sep 2026 06:29:53 +0000

Liquid syntax error: Unknown tag 'endraw'

Closing the SSRF DNS-Rebinding Hole with a Custom Resolver

Anand Rathnas — Mon, 28 Sep 2026 06:33:13 +0000

Liquid syntax error: Unknown tag 'endraw'

OAuth client_credentials in Spring Boot for Prometheus Scrapes

Anand Rathnas — Fri, 25 Sep 2026 06:00:33 +0000

This article was originally published on Jo4 Blog.

We stood up Prometheus on a self-hosted DigitalOcean droplet and pointed it at our Spring Boot API's /actuator/prometheus endpoint. The obvious question came next: how does Prometheus authenticate?

Until that moment, our OAuth server had spoken exactly two grants — authorization_code (with PKCE) and refresh_token. Both assume a user behind the request. Zapier, Make, Pipedream, the Chrome extension, the MCP clients — every existing integration represents a person who clicked "Connect". Prometheus represents nobody. It's a process on a host scraping a metrics endpoint every 30 seconds. There is no consenting user.

The RFC has a grant for exactly this — client_credentials, §4.4. The client authenticates itself to the token endpoint with its credentials, gets back a bearer, and presents that bearer on subsequent calls. No user, no consent screen, no refresh token. Here's how we slotted it into an existing Spring Security stack without regressing a single user-bound integration.

Why client_credentials Fits

Three properties of the Prometheus scrape are worth being explicit about, because each maps directly to a §4.4 design choice:

There is no user. Prometheus is a daemon. It has no Auth0 identity, no email, no role assignments. Forcing it through authorization_code would mean creating a synthetic "Prometheus user" — a pattern that's been a security footgun every time we've seen it (service accounts that drift into having too many permissions because they're shaped like real users).

The credentials live on the server, rotatably. Prometheus reads its OAuth client_secret from a file on disk that the deployment script writes. Rotating means redeploying the secret file and restarting Prometheus — same operational model as any other server-side credential. No browser, no consent.

The blast radius is one endpoint. Prometheus has exactly one thing it's allowed to do: GET /actuator/prometheus. Not list URLs, not read analytics, not touch user data. The credential should be incapable of doing anything else even if it leaks.

client_credentials is the RFC-defined shape for all three. The wire flow is two HTTP calls: POST /oauth/token with HTTP Basic-Auth-encoded client credentials returns an access_token; every subsequent scrape carries Authorization: Bearer <access_token> until the token expires. Per §4.4.3 there is no refresh token — when the access token expires, the client re-authenticates by hitting /oauth/token again. Prometheus's built-in oauth2: block does this transparently.

Three Schema Relaxations

Before any of the Java could change, the database had to admit the shape of a userless token and the shape of a redirect-URI-less client. Liquibase changeset 356 made three modifications:

- changeSet:
    id: 356-add-oauth-client-credentials-grant
    author: anand
    changes:
      - addColumn:
          tableName: oauth_clients
          columns:
            - column:
                name: grant_types
                type: VARCHAR(100)
                defaultValue: "authorization_code,refresh_token"
                constraints:
                  nullable: false
                remarks: "Comma-separated RFC 6749 grant_type values the client is permitted to use."

      - dropNotNullConstraint:
          tableName: oauth_clients
          columnName: redirect_uris
          columnDataType: TEXT

      - dropNotNullConstraint:
          tableName: oauth_access_tokens
          columnName: user_id
          columnDataType: BIGINT

Each change has a specific reason:

oauth_clients.grant_types (NEW column). Previously, the grant type set was implicit — every client got AC + RT. Now each client carries the exact list of grants it's allowed to request. Existing rows get backfilled to "authorization_code,refresh_token" via the column default during the addColumn step, which is the property that keeps every existing integration green. Zapier's row, Make's row, both Pipedream rows, the Chrome extension's row — all get the same string they were implicitly assuming. The new CC-only Prometheus client gets "client_credentials" and nothing else.

oauth_clients.redirect_uris nullable. A CC-only client legitimately has no browser callback. Without this relaxation, every CC client registration would have to invent a fake URL just to satisfy a NOT NULL constraint that was modeling the wrong invariant. Enforcement now moves to the service layer — it's still required for authorization_code clients, but the rule is "required when grant_types includes AC", not "required always".

oauth_access_tokens.user_id nullable. A CC token represents the client, not a user. The foreign key to users(id) ON DELETE CASCADE stays — NULL becomes the signal for "service-principal token, no user lookup needed". Token validation branches on this: if userId is null, skip the user-load step.

The changeset uses addColumn and dropNotNullConstraint rather than raw SQL, which sidesteps the Liquibase PL/pgSQL footgun (folded-block YAML corrupting dollar quotes). All three changes are reversible and the rollback is provided explicitly.

Token Issuance

The token endpoint is a switch on grant_type. We added a third branch:

@Transactional
public TokenResponse issueClientCredentialsToken(String clientId, String clientSecret,
                                                 String requestedScope, String requestedResource) {
    OAuthClientEntity client = validateTokenEndpointClient(clientId, clientSecret);

    if (client.isPublicClient()) {
        throw new UrlService.AppException(ErrorCode.OAUTH_INVALID_CLIENT_SECRET,
                "client_credentials grant requires a confidential client");
    }

    if (!client.supportsGrant("client_credentials")) {
        throw new UrlService.AppException(ErrorCode.OAUTH_CLIENT_DISABLED,
                "Client is not authorized to use client_credentials grant");
    }

    // Default to the client's full allowed scope set if none requested; otherwise
    // intersect requested scopes with allowed (RFC 6749 §3.3 narrowing).
    Set<String> allowed = client.getScopesSet();
    Set<String> granted;
    if (requestedScope == null || requestedScope.isBlank()) {
        granted = allowed;
    } else {
        Set<String> requested = parseScopes(requestedScope);
        if (!allowed.containsAll(requested)) {
            throw new UrlService.AppException(ErrorCode.OAUTH_INVALID_SCOPE,
                    "One or more requested scopes are not granted to this client");
        }
        granted = requested;
    }
    String grantedScopeString = String.join(",", granted);

    // Mint access token only — no refresh token per RFC 6749 §4.4.3.
    String accessToken = generateToken(TOKEN_LENGTH);
    String accessTokenHash = hashToken(accessToken);
    long accessExpiresAt = System.currentTimeMillis() + (client.getAccessTokenLifetimeSeconds() * 1000L);

    OAuthAccessTokenEntity accessTokenEntity = OAuthAccessTokenEntity.builder()
            .tokenHash(accessTokenHash)
            .clientId(client.getId())
            .userId(null)  // CC tokens have no user
            .scopes(grantedScopeString)
            .expiresAt(accessExpiresAt)
            .tenantId(client.getTenantId())
            .audience(requestedResource)
            .slug(UUID.randomUUID().toString().replace("-", "").substring(0, 16))
            .build();

    accessTokenRepository.save(accessTokenEntity);
    // ...returns TokenResponse with no refresh_token
}

Three things this is doing right:

Public-client rejection. A public client (token_endpoint_auth_method = "none") is one that authenticates via PKCE only. CC has no PKCE story — its security model is "the client knows the secret". Public clients can never use CC, full stop. We reject before issuing.

Per-client grant gating. supportsGrant("client_credentials") checks the row's grant_types column. A client registered for AC + RT cannot suddenly switch grants on the wire. The grant set is part of the client identity, not a request-time choice.

Scope narrowing. RFC 6749 §3.3 says the authorization server MAY issue a narrower scope than requested, MUST NOT issue a wider one. We default to "narrowest of allowed and requested", and reject if the request asks for anything outside allowed.

The token gets persisted with userId = null — the column relaxation from migration 356 is what allows that row to exist at all. No refresh token row is created.

Authentication Subclass

The hardest design decision wasn't on the token endpoint. It was on the validation side. When a bearer comes in on /actuator/prometheus, the existing filter chain inspects the token and constructs a Spring Security Authentication object to attach to the request. The existing code expects every authentication to have a user:

// OAuthTokenAuthentication.java (existing — unchanged)
public OAuthTokenAuthentication(UserEntity user, OAuthAccessTokenEntity token, OAuthClientEntity client) {
    super(buildAuthorities(user, token));
    this.user = user;
    this.userId = user.getId();  // would NPE for CC tokens
    // ...
}

The naive fix is to make user nullable on OAuthTokenAuthentication and sprinkle null-checks across every consumer. We considered that and rejected it — the codebase has dozens of instanceof OAuthTokenAuthentication checks across AuthContext, OAuthScopeEnforcementFilter, Jo4McpTools, and elsewhere. Making user nullable on the existing type would mean auditing every single one of them and adding null guards, with the constant risk of missing one and getting an NPE in production six weeks later.

We introduced a separate authentication type instead:

@Getter
public class OAuthClientCredentialsAuthentication extends AbstractAuthenticationToken {

    private final OAuthAccessTokenEntity accessToken;
    private final OAuthClientEntity client;
    private final Long clientId;
    private final Set<String> scopes;

    public OAuthClientCredentialsAuthentication(OAuthAccessTokenEntity accessToken, OAuthClientEntity client) {
        super(buildAuthorities(accessToken));
        this.accessToken = accessToken;
        this.client = client;
        this.clientId = client.getId();
        this.scopes = accessToken.getScopesSet();
        setAuthenticated(true);
    }

    /**
     * The "principal" of a client_credentials token is the OAuth client itself.
     */
    @Override
    public Object getPrincipal() {
        return client;
    }
    // ...
}

What this buys us is type-driven correctness. Every existing instanceof OAuthTokenAuthentication check naturally evaluates false for a CC token. Code that asks "who is the user behind this request?" gets Optional.empty() from AuthContext.getActualUserOptional(). Controllers that call authContext.getCurrentUser() throw UNAUTHORIZED automatically. The MCP tools that filter by userId never even see CC requests. We didn't have to audit dozens of sites; the type system did it for us.

The validation filter branches once, at the boundary:

// inside OAuthTokenAuthenticationFilter, paraphrased
ValidatedToken validated = oauthService.validateAccessToken(bearer);
Authentication auth = validated.getUser() == null
    ? new OAuthClientCredentialsAuthentication(validated.getToken(), validated.getClient())
    : new OAuthTokenAuthentication(validated.getUser(), validated.getToken(), validated.getClient());
SecurityContextHolder.getContext().setAuthentication(auth);

validateAccessToken itself has exactly one change — the user lookup becomes conditional on userId != null. That's the entire blast radius on the validation side.

The Path Allowlist

Scope-based authorization gets us most of the way. The Prometheus client has metrics:read scope only — not read, not write. Endpoints requiring read/write reject it because the bearer lacks those scopes.

But "lacks the right scope" is one layer. We wanted defense-in-depth, because scope sets get edited in admin UIs by humans and a future "let's give the metrics client read to test something" five-second mistake should not silently grant CC tokens access to user data. So we added a second filter that runs immediately after the token filter:

@Slf4j
@Component
@Profile("auth0")
public class ClientCredentialsTokenGateFilter extends OncePerRequestFilter {

    private final AntPathMatcher pathMatcher = new AntPathMatcher();

    @Value("${app.security.cc-token-paths:/actuator/prometheus}")
    private String ccTokenPaths;

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
            throws ServletException, IOException {

        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        // Only act on CC tokens. Everything else (JWT, API key, user-bound OAuth, anonymous) passes through.
        if (!(auth instanceof OAuthClientCredentialsAuthentication ccAuth)) {
            filterChain.doFilter(request, response);
            return;
        }

        String requestPath = request.getRequestURI();
        List<String> allowed = Arrays.stream(ccTokenPaths.split(","))
                .map(String::trim)
                .filter(s -> !s.isEmpty())
                .toList();

        boolean pathAllowed = allowed.stream().anyMatch(p -> pathMatcher.match(p, requestPath));
        if (pathAllowed) {
            filterChain.doFilter(request, response);
            return;
        }

        log.warn("client_credentials token attempted to access non-allowlisted path: clientId={}, path={}, scopes={}",
                ccAuth.getClientId(), requestPath, ccAuth.getScopes());

        // RFC 6750 §3.1: insufficient_scope on a Bearer-protected resource.
        response.setStatus(HttpStatus.FORBIDDEN.value());
        response.setHeader("WWW-Authenticate",
                "Bearer error=\"insufficient_scope\", error_description=\"client_credentials tokens are not valid for this endpoint\"");
        response.getWriter().write(
                "{\"error\":\"insufficient_scope\",\"error_description\":\"client_credentials tokens are not valid for this endpoint\"}");
    }
}

A few details that matter:

The instanceof guard is the no-op short-circuit. JWT requests, API-key requests, user-bound OAuth requests, anonymous requests — all four pass through the filter with one type check and zero allocations. The cost is paid only on the small set of CC requests.

Config-driven allowlist. app.security.cc-token-paths is comma-separated and AntPathMatcher-evaluated. Adding a future endpoint (say, an alertmanager receiver) is a config edit, not a code change. The default — /actuator/prometheus — is the only thing CC tokens can reach out of the box:

# application-security.yaml
app:
  security:
    # Paths where client_credentials OAuth tokens are accepted. CSV; AntPathMatcher syntax.
    cc-token-paths: "/actuator/prometheus"

WWW-Authenticate: Bearer error="insufficient_scope" is the RFC 6750 §3.1 shape. Prometheus and other OAuth-aware clients can read this and surface a meaningful error, rather than getting a generic 403 with no clue what went wrong.

Never throws. Filters that throw can mask the underlying error or tear down the chain in surprising ways. This one always writes a response or delegates.

Wiring the Scrape

On the Prometheus side, the entire setup is one block in prometheus.yml:

scrape_configs:
  - job_name: jo4-api
    metrics_path: /actuator/prometheus
    scheme: http
    scrape_interval: 30s
    oauth2:
      client_id: __PROMETHEUS_OAUTH_CLIENT_ID__
      client_secret_file: /etc/prometheus/oauth-client-secret
      token_url: http://10.108.0.3:8080/oauth/token
      scopes:
        - metrics:read
    static_configs:
      - targets: ['10.108.0.3:8080']
        labels:
          service: alertstage
          env: prod

Notes from setting this up:

client_id inline, client_secret_file from disk. Per OAuth 2.0, the client ID is public — fine to bake into the config and sed-substitute at deploy time. The secret stays in a separate file with restrictive permissions, written by the bootstrap script from a GitHub Secret. Never co-located, never logged.

Private VPC URLs. token_url and the scrape target both use the DigitalOcean private IP 10.108.0.3. Traffic never leaves the VPC, which is the primary security boundary; the OAuth check is defense-in-depth on top of that. The token endpoint accepts plain HTTP only because it's on a private subnet; this would be HTTPS for any path that crosses the public internet.

scopes: ["metrics:read"]. Prometheus's oauth2: block converts this into the scope form field of the /oauth/token request. Our issueClientCredentialsToken narrows it against the client's allowed set — which happens to be just metrics:read — and stamps the result on the issued access token.

No refresh_token config. Prometheus knows §4.4.3 — when it sees a token response with no refresh_token, it just re-runs the token flow when the current access token nears expiry. No refresh logic on either side.

Testing It End-To-End

A curl sequence to verify the whole loop, against a freshly registered Prometheus client (client ID jo4_Xxxx, client secret captured at registration):

# 1. Token exchange — HTTP Basic-Auth-encoded client credentials.
ACCESS_TOKEN=$(curl -s -X POST https://jo4-api.jo4.io/oauth/token \
  -u "jo4_Xxxx:secret_XXXXXXXX" \
  -d "grant_type=client_credentials" \
  -d "scope=metrics:read" | jq -r .access_token)

# 2. Scrape — bearer on the request.
curl -i https://jo4-api.jo4.io/actuator/prometheus \
  -H "Authorization: Bearer $ACCESS_TOKEN"
# → HTTP/2 200
# → # HELP http_server_requests_seconds ...

# 3. Negative test — same token against a user endpoint.
curl -i https://jo4-api.jo4.io/api/v1/urls/me \
  -H "Authorization: Bearer $ACCESS_TOKEN"
# → HTTP/2 403
# → WWW-Authenticate: Bearer error="insufficient_scope", ...
# → {"error":"insufficient_scope","error_description":"client_credentials tokens are not valid for this endpoint"}

That third call is the critical one to verify. The same bearer, presented at a non-allowlisted path, must be rejected — even though the bearer is technically valid. If you see anything other than 403 + insufficient_scope there, the gate filter isn't wired correctly.

Lessons Learned

A new grant type is mostly a schema problem. The Java for client_credentials is small. The work was three column relaxations and an audit of every codepath that assumed userId was non-null.
A separate Authentication subclass beats nullable fields on the existing one. instanceof checks across the codebase become the type-system's audit for you. Sprinkling null-checks would have been an ongoing tax forever.
Always backfill new NOT NULL columns via a default. Adding grant_types NOT NULL would have failed on every existing row without defaultValue: "authorization_code,refresh_token". The default is what makes the migration zero-downtime and the rollout silent for every existing client.
Defense-in-depth means two independent layers, not one layer twice. Scope-based authorization handles "the client wasn't granted this scope". The path-allowlist filter handles "even if someone misconfigures the scope set tomorrow, this token still can't hit user data". Both layers must agree before access is granted.
WWW-Authenticate on 401/403 from a bearer-protected resource is RFC 6750 §3.1. Clients can react to error="insufficient_scope" programmatically. A naked 403 with no header forces humans into the loop for every failure.
No refresh token for CC, period. RFC 6749 §4.4.3. Prometheus and every other OAuth-aware client know how to re-run the token flow on expiry. Issuing a refresh token would be inventing capability the spec explicitly forbids.
Validate every existing integration before shipping. We scanned ten risk categories across the codebase before touching a line — Zapier, Make, Pipedream, Chrome extension, MCP clients, OIDC discovery, /oauth/userinfo, the scope filter, public endpoints, the previously-public /actuator/prometheus path. The migration changed default-value-backfilled columns; behavior changed for exactly one path, and it was the one we intended.

Adding CC grant to an existing AC flow? What schema migration bit you? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Bound Spring MVC's Async Executor or Pay for It Under Load

Anand Rathnas — Wed, 23 Sep 2026 05:43:18 +0000

This article was originally published on Jo4 Blog.

Spring MVC's default async task executor spawns a brand new thread for every async request. No queue. No ceiling. No back-pressure. Spring itself logs a WARN on first use: "This executor is not suitable for production use under load." That's a load-bearing warning, and most teams scroll past it.

We learned the hard way and ended up wiring a bounded ThreadPoolTaskExecutor, a typed rejection path, and a Micrometer saturation gauge. Here's the whole arc.

What Spring MVC Defaults To

When a controller method returns a DeferredResult, Callable, WebAsyncTask, Flux, or Mono, Spring MVC needs an executor to run the async dispatch. If you don't configure one, you get SimpleAsyncTaskExecutor.

SimpleAsyncTaskExecutor does exactly what its name says: every submission spawns a fresh thread. There is no pool. There is no queue. There is no rejection policy, because there is no ceiling to reject against. Send it a thousand concurrent requests and it will cheerfully start a thousand threads, each with its own ~1 MB stack, and watch your container OOM.

Spring 6 logs this WARN on first use:

This executor is not suitable for production use under load. Consider configuring a ThreadPoolTaskExecutor via WebMvcConfigurer.configureAsyncSupport.

If you ignore the warning, the failure mode is exotic: latency stays fine until traffic crosses some invisible threshold, at which point the JVM either runs out of native threads (Linux pthread_create returns EAGAIN) or runs out of heap because every parked thread is holding stack frames and a Tomcat request scope.

Why That's Wrong For Real Apps

Three patterns push you onto this executor whether you realize it or not:

DeferredResult / Callable / WebAsyncTask returns. Anyone using the classic async-servlet pattern.
Flux / Mono returns from a Spring MVC controller. Spring MVC (not WebFlux) bridges reactive types onto the async-dispatch path. Every chunk emission round-trips through the executor.
SSE endpoints (SseEmitter). Long-lived streams that hold a dispatch thread for the entire lifetime of the connection.

We ship all three. Our analytics SSE stream in particular pins one async thread per subscriber for up to five minutes (our ASYNC_TIMEOUT). With SimpleAsyncTaskExecutor, 500 concurrent dashboard users would mean 500 unbounded threads with no warning sign before the container falls over.

Bounded Pool Setup

The fix is a real ThreadPoolTaskExecutor with explicit core, max, and queue settings. Here's the actual bean from WebMvcAsyncConfig.java:

@Bean(name = "mvcAsyncExecutor", destroyMethod = "shutdown")
public ThreadPoolTaskExecutor mvcAsyncExecutor() {
    ThreadPoolTaskExecutor exec = new ThreadPoolTaskExecutor();
    exec.setCorePoolSize(8);
    exec.setMaxPoolSize(200);
    // queueCapacity = 0 → Spring uses SynchronousQueue: direct hand-off, no buffering.
    exec.setQueueCapacity(0);
    exec.setThreadNamePrefix("jo4-mvc-async-");
    exec.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy());
    exec.setWaitForTasksToCompleteOnShutdown(true);
    exec.setAwaitTerminationSeconds(30);
    exec.initialize();
    return exec;
}

@Override
public void configureAsyncSupport(AsyncSupportConfigurer configurer) {
    configurer.setDefaultTimeout(ASYNC_TIMEOUT.toMillis());
    configurer.setTaskExecutor(mvcAsyncExecutor());
}

A few decisions worth calling out, because the obvious choices are wrong for long-lived async:

queueCapacity = 0. Spring's ThreadPoolTaskExecutor interprets queueCapacity = 0 as a SynchronousQueue — a direct hand-off with no buffering. The instinct is to "set a big queue, it's free." For short-lived tasks that's right. For SSE streams (which hold the thread for up to 5 minutes), a non-zero queue would park new connections waiting for an existing core thread to free — effectively "hang the client until someone else disconnects." With a 0-capacity queue, the pool grows from corePoolSize=8 toward maxPoolSize=200 as new subscribers arrive, and only true saturation hits the rejection handler.

AbortPolicy, not CallerRunsPolicy. This one we learned by pain. Our first attempt used CallerRunsPolicy (the seemingly-graceful "if the pool is full, run inline on the caller's thread"). Under load that pins the Tomcat request thread executing the task synchronously — which turned multi-minute Playwright teardowns into multi-minute hangs because the test runner's "close all contexts" call was racing pool saturation. Fail-fast with AbortPolicy is preferable: back-pressure surfaces as a RejectedExecutionException that the caller can translate into an HTTP 503, and the Tomcat thread is freed immediately for the next request.

waitForTasksToCompleteOnShutdown = true + awaitTerminationSeconds = 30. SSE streams need time to flush their final event and let the client see the close cleanly. Hard-killing the pool on shutdown means subscribers get a TCP reset instead of a proper stream-end signal.

destroyMethod = "shutdown". Pairs the bean lifecycle with the executor lifecycle so Spring's context shutdown drains the pool. Without it you leak threads on every context refresh.

Rejection With Meaning

AbortPolicy throws RejectedExecutionException. By default that propagates out of the async-dispatch machinery, hits the generic exception handler, and becomes a 500 Internal Server Error — which is a lie. It's not an internal error; it's capacity. Calling clients, CDN edges, and retry-aware HTTP libraries all behave differently against 500 vs 503: 500 says "give up, this is broken"; 503 says "back off and try again."

We give it a typed home in our ErrorCode enum:

SERVICE_SATURATED("SERVICE_SATURATED", "Server is temporarily at capacity. Please retry."),

And we catch it explicitly in GlobalExceptionHandler:

@ExceptionHandler(RejectedExecutionException.class)
public ResponseEntity<ResponseBody<EmptyResponse>> handleRejectedExecution(RejectedExecutionException ex) {
    String requestUri = getRequestUri();
    log.warn("Async executor saturated on {}: {}", requestUri, ex.getMessage());

    Fault fault = Fault.builder()
            .code(ErrorCode.SERVICE_SATURATED.getCode())
            .description(ErrorCode.SERVICE_SATURATED.getMessage())
            .build();

    ResponseBody<EmptyResponse> body = ResponseBody.<EmptyResponse>builder()
            .requestId(UUID.randomUUID().toString())
            .faults(List.of(fault))
            .build();

    return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
            .header(HttpHeaders.RETRY_AFTER, "5")
            .body(body);
}

Three things matter here. First, the response is 503 SERVICE_UNAVAILABLE — semantically correct for "we're at capacity," not "we're broken." Second, the Retry-After: 5 header is honored by fetch retry middlewares, the AWS SDK, and most CDN edges, which means callers self-throttle without us writing client code. Third, the structured fault carries SERVICE_SATURATED so the SPA can render a friendly "we're at capacity, try again" toast instead of a generic 5xx splash.

The mapping back to HTTP status lives in mapErrorCodeToStatus:

case SERVICE_SATURATED -> HttpStatus.SERVICE_UNAVAILABLE;

Without that explicit handler, the rejection would fall through to the catch-all Exception handler and become a 500. Same root cause, completely different operational signal.

Make It Observable

Bounded pools are useless if you can't see how close you are to the wall. We expose five gauges via a MeterBinder, scraped from /actuator/prometheus:

@Bean
public MeterBinder mvcAsyncExecutorMetrics(ThreadPoolTaskExecutor mvcAsyncExecutor) {
    return registry -> {
        Gauge.builder("mvc.async.executor.active",
                mvcAsyncExecutor, ThreadPoolTaskExecutor::getActiveCount)
            .description("Threads currently executing tasks")
            .baseUnit("threads")
            .register(registry);

        Gauge.builder("mvc.async.executor.pool.size",
                mvcAsyncExecutor, ThreadPoolTaskExecutor::getPoolSize)
            .description("Current pool size (grows from core toward max under load)")
            .baseUnit("threads")
            .register(registry);

        Gauge.builder("mvc.async.executor.pool.max",
                mvcAsyncExecutor, e -> (double) e.getMaxPoolSize())
            .description("Configured maxPoolSize ceiling")
            .baseUnit("threads")
            .register(registry);

        Gauge.builder("mvc.async.executor.queue.size",
                mvcAsyncExecutor, e -> {
                    ThreadPoolExecutor tpe = e.getThreadPoolExecutor();
                    return tpe == null ? 0 : tpe.getQueue().size();
                })
            .description("Queued tasks waiting for a thread")
            .register(registry);

        // The headline metric: active / max. 0.8 = alert; 1.0 = saturated.
        Gauge.builder("mvc.async.executor.saturation",
                mvcAsyncExecutor, e -> {
                    int max = e.getMaxPoolSize();
                    return max == 0 ? 0.0 : (double) e.getActiveCount() / max;
                })
            .description("Active-to-max ratio (0..1). >0.8 = capacity warning, 1.0 = rejecting")
            .register(registry);
    };
}

The first three (active, pool.size, pool.max) are diagnostic — useful in a Grafana dashboard to see how the pool actually grows. queue.size is included for forward compatibility; with queueCapacity = 0 it's always zero, but if we ever tune the queue we don't want to be the team that forgot to publish the gauge.

mvc.async.executor.saturation is the headline. It's a single derived ratio (active / max) bounded between 0 and 1, which means one alert threshold works regardless of how we resize the pool over time. Cardinality of one, semantics of "how close are we to rejecting" — exactly what a pager wants to wake you up about.

Alerts That Actually Mean Something

Two thresholds, both honest:

WARN when mvc_async_executor_saturation > 0.8 for 5 minutes. We're at 80% of the ceiling for long enough to mean it's not a spike. Likely time to bump maxPoolSize, investigate slow async handlers, or check if SSE clients are leaking connections.
CRIT when mvc_async_executor_saturation == 1.0. We're rejecting. SERVICE_SATURATED 503s are going out the door. Page on-call.

The warning level matters because saturation is a leading indicator. Once you're rejecting, customers have already seen 503s. Once you're at 80%, you have minutes to act before that happens — assuming you alert on it. The corollary: a Grafana dashboard with saturation as the headline tile (not request rate, not latency) tells you at a glance whether you're about to have a bad day.

We pair saturation with the Retry-After header on the 503 so even when we are rejecting, well-behaved clients smooth it out and we recover without a thundering herd hammering us the moment the pool drains.

Lessons Learned

SimpleAsyncTaskExecutor is a development convenience, not a production executor. If you return DeferredResult, Flux, Mono, or use SseEmitter from a Spring MVC controller, configure AsyncSupportConfigurer.setTaskExecutor with a real bounded pool. The default's WARN log message is correct.
Queue capacity is a function of task duration, not of "more is better." For sub-second tasks, a generous queue smooths spikes. For long-lived tasks like SSE, a queue silently hangs new connections behind old ones. Use queueCapacity = 0 + grow the pool to maxPoolSize.
AbortPolicy over CallerRunsPolicy for HTTP work. CallerRunsPolicy pins your Tomcat request threads under load and turns a capacity problem into a hung-process problem. Fail fast, return 503.
Map rejection to a typed error code, not a generic 500. RejectedExecutionException → SERVICE_SATURATED → HTTP 503 + Retry-After tells callers something actionable. 500 tells them nothing.
Saturation is the gauge worth alerting on. A derived active/max ratio is dimensionless, bounded 0..1, survives pool resizing, and gives you a single threshold to define "we're about to have a bad day."
The WARN threshold is the one that prevents pages. Alert at 0.8 with 5-minute hysteresis to get warned before you reject. Alert at 1.0 to know you're already rejecting. Both, not either.

What's your async-executor war story? Tail-latency, thread leaks, OOM? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

How One Spring Boot Default Killed Our SSE Endpoint Under Load

Anand Rathnas — Mon, 21 Sep 2026 06:02:31 +0000

This article was originally published on Jo4 Blog.

Our staging environment had ten browser tabs open with the notification bell. The eleventh person to open the app got an API that just sat there. Not a 500. Not a timeout (yet). Just a silent stall on every endpoint, not only the SSE one. The pod looked healthy. CPU flat, memory fine, logs quiet.

We'd turned our notification bell into a Server-Sent Events stream while leaving one Spring Boot default in place. For a request-response API, that default is fine. For a 30-minute SSE connection, it's lethal: spring.jpa.open-in-view=true pins a Hikari connection per open tab for the entire lifetime of the stream.

Here's the bug, the math behind the symptom, and the one-line fix — plus the precondition the endpoint has to satisfy for the fix to be safe.

The Endpoint

The SSE endpoint lives in NotificationController.streamNotifications(). It returns a Flux<ServerSentEvent<NotificationStreamEvent>> to the client and stays open for 30 minutes, emitting the initial unread count, live updates from the event bus, and a heartbeat every 30 seconds:

@GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public ResponseEntity<Flux<ServerSentEvent<NotificationStreamEvent>>>
        streamNotifications() {
    authContext.requireScope("read");
    UserEntity user = authContext.getCurrentUser();
    Long userId = user.getId();

    ServerSentEvent<NotificationStreamEvent> initial = ServerSentEvent
        .<NotificationStreamEvent>builder()
        .event("unread-count")
        .data(NotificationStreamEvent.unreadCount(
            notificationService.getUnreadCount(userId)))
        .build();

    Flux<ServerSentEvent<NotificationStreamEvent>> events =
        notificationEventBus.subscribe(userId)
            .map(event -> ServerSentEvent.<NotificationStreamEvent>builder()
                .event(event.type()).data(event).build());

    Flux<ServerSentEvent<NotificationStreamEvent>> heartbeat =
        Flux.interval(HEARTBEAT_INTERVAL)
            .map(tick -> ServerSentEvent.<NotificationStreamEvent>builder()
                .comment("ping").build());

    Flux<ServerSentEvent<NotificationStreamEvent>> body = Flux.concat(
            Flux.just(initial), Flux.merge(events, heartbeat))
        .take(STREAM_LIFETIME);

    return ResponseEntity.ok().headers(headers).body(body);
}

Two numbers matter: STREAM_LIFETIME = Duration.ofMinutes(30) and HEARTBEAT_INTERVAL = Duration.ofSeconds(30). The stream stays open for half an hour. The client reconnects on close with a fresh JWT. The heartbeat keeps it alive through any proxy idle-timeout in between.

From the controller's perspective this looks innocuous. It's a Flux. It's reactive. It touches JPA once, synchronously, at the top, to seed the initial count. The Flux body that streams for the next 30 minutes never touches the database.

That's what we thought the runtime behavior was. The runtime had other ideas.

open-in-view: Spring's Helpful Default

spring.jpa.open-in-view=true is Spring Boot's default. It enables OSIV — Open Session In View. The mechanism is an interceptor that opens a Hibernate Session (and, through it, borrows a database connection from the Hikari pool) at the start of an HTTP request, and holds it open until the response is fully written.

For traditional MVC this is convenient. Lazy-loaded JPA associations still resolve when your Thymeleaf template or JSON serializer touches them after the controller returns. No LazyInitializationException — at the cost of holding a DB connection for the lifetime of the HTTP response.

For a request-response endpoint serving JSON in 50ms, "lifetime of the HTTP response" is 50ms. The connection round-trips back to the pool before the next request even arrives. Pool pressure is essentially zero and the convenience is large enough that the Spring team made it default-on.

Why It's Lethal for SSE

Server-Sent Events flips the lifetime assumption on its head.

The HTTP response on an SSE endpoint is not 50ms. It's the entire duration of the open stream — 30 minutes in our case. As long as the tab is open and the stream is live, the response is still being written one event at a time, and the servlet container hasn't released the request or the response.

OSIV doesn't know any of that. It just sees a request that's still in flight, so the session must still be open, so the connection it borrowed from Hikari must still be held.

One open SSE stream = one Hikari connection pinned for 30 minutes. The connection isn't being used. Nothing is running queries on it. It's reserved, sitting idle, unavailable to anyone else, until the response finally ends — at which point the user reconnects with a fresh JWT and the pinned-connection clock resets.

The Symptom Math

Hikari's default maximumPoolSize is 10. Spring Boot doesn't override it. Most apps don't either, because for a normal API the default is more than enough — connections cycle back in tens of milliseconds and you'd need hundreds of concurrent requests to feel any pressure.

Now run the math with OSIV + SSE:

Open SSE tabs	Hikari connections held	Connections free for other work
1	1	9
5	5	5
9	9	1
10	10	0
11	10	0 (request 11 waits)

The 11th SSE subscriber's request asks OSIV for a connection. There are none. Hikari blocks the request thread on the pool, waiting for one to free up. Nothing will free up for the next 29 minutes and 50 seconds. Eventually Hikari hits its connection-timeout (default 30 seconds) and throws SQLTransientConnectionException: HikariPool-1 - Connection is not available.

The cruel part: it's not just the SSE endpoint that hangs. Every endpoint that needs a DB connection — login, the click handler, the dashboard fetch — competes for the same pool. Ten pinned SSE tabs from a single demo session, and the whole API is wedged. The JVM is alive, nothing is crashing, everything is just queuing on an empty pool. You stare at flat CPU graphs for an hour and feel personally betrayed by your observability stack.

The Fix

One line in application-database.yaml:

spring:
  jpa:
    open-in-view: false

That disables the OSIV interceptor. Connections are no longer held for the lifetime of the HTTP response — they're borrowed when a transaction starts, returned when it ends. The 30-minute SSE stream now holds zero connections for the 30 minutes it's open. Hikari pool is unblocked. The 11th tab works. The 1000th tab works.

Spring's startup log even warns about OSIV being on by default, but it's easy to scroll past in a noisy log. On a normal MVC app it's not a problem worth acting on. On an app with long-lived reactive endpoints, it absolutely is.

The Precondition That Makes the Fix Safe

You can't just flip open-in-view: false and call it done. Turning OSIV off changes a semantic: lazy-loaded JPA associations outside an explicit transaction now throw LazyInitializationException. The endpoint has to be structured so it doesn't depend on that lazy-load convenience.

Our SSE endpoint already is, intentionally. Look at where it touches JPA:

ServerSentEvent<NotificationStreamEvent> initial = ServerSentEvent
    .<NotificationStreamEvent>builder()
    .event("unread-count")
    .data(NotificationStreamEvent.unreadCount(
        notificationService.getUnreadCount(userId)))   // <-- only JPA touch
    .build();

That one call. It runs synchronously at subscribe-time inside the @Transactional boundary of NotificationService.getUnreadCount(...). The transaction opens, borrows a connection, runs the count query, returns a long, commits, releases the connection. Total time: a few milliseconds.

After that, every event flowing through the Flux body — live events from notificationEventBus.subscribe(userId), the heartbeat pings, the close after 30 minutes — emits a plain in-memory NotificationStreamEvent record. No JPA. No lazy proxies. Nothing the persistence context needs to resolve.

The Javadoc on the endpoint spells out this contract explicitly:

Pool contract: this endpoint relies on spring.jpa.open-in-view=false (set in application-database.yaml). All JPA access happens at subscribe time via notificationService.getUnreadCount(...) inside its own transaction. The Flux body emits the in-memory NotificationStreamEvent record only — no JPA touch — so no Hikari connection is held for the STREAM_LIFETIME window. If OSIV is ever re-enabled, every open browser tab pins a connection until the stream closes, exhausting the pool at low concurrency. Do not re-enable OSIV.

That paragraph exists because the fix is one config flag away from being silently undone by a well-meaning future contributor who reads "Spring Boot default" and decides to align. The Javadoc is the load-bearing comment for the entire endpoint.

Lessons Learned

OSIV is a request-response convenience. It pays for itself in a 50ms JSON endpoint. It bankrupts you in a 30-minute SSE stream. The cost is "connection held for the lifetime of the HTTP response," and on a long-lived response that's the whole stream.
Hikari's default pool of 10 is generous for request-response and tiny for anything that pins connections. The default implicitly assumes you're returning connections quickly. If you're not, you usually only learn this in production at the eleventh concurrent subscriber.
Pool exhaustion makes everything hang, not just the slow endpoint. The DB pool is shared. SSE pinning it starves every other endpoint. The trail leads back to SSE only after you check pool stats.
spring.jpa.open-in-view: false is right for any app with long-lived reactive endpoints, but only safe if those endpoints don't depend on lazy-loading outside an explicit transaction. Audit first; don't flip the flag blind.
Document the precondition where the bug would re-enter. A one-paragraph "Pool contract" Javadoc on the endpoint is the difference between a fix that holds and one someone undoes in six months while "cleaning up config to match Spring defaults."
Reactive controllers on a servlet stack are a special kind of trap. Reactive types make you think in non-blocking lifetimes; the underlying servlet container is still tracking the HTTP request as in-flight for the full duration, and request-scoped infrastructure like OSIV happily plays along.

Has OSIV bitten you somewhere unexpected? Drop the symptom in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Grafana as Code: Surviving Destroy-Recreate With Provisioning Files

Anand Rathnas — Fri, 18 Sep 2026 05:43:16 +0000

Liquid syntax error: Unknown tag 'endraw'

Watchtower Tried to Upgrade Itself. It Didn't Come Back.

Anand Rathnas — Wed, 16 Sep 2026 05:50:05 +0000

This article was originally published on Jo4 Blog.

On May 19 we pushed a fix to main. GitHub Actions built the image, tagged it :latest, shoved it into GHCR, and went home. Our droplet was supposed to pick that up within sixty seconds. It didn't. Not in sixty minutes, not in two days.

When we finally SSH'd in to ask why, the answer was in docker ps: our auto-update tool wasn't running. It had tried to update itself, succeeded at the "stop" half, and never made it to the "start" half. Nobody noticed for forty-eight hours because the API was up; only the deploy pipeline was silently dead.

This is the story of that incident and the two-line fix that makes it structurally impossible to repeat.

What Watchtower Is For

If you've never used Watchtower, it solves a small, useful problem: you push a new image to a registry, and Watchtower running on your host detects the new digest, pulls it, gracefully stops the old container, and starts a fresh one. No SSH, no docker compose pull && up -d, no human in the loop. For a tiny team on a single droplet, it's a five-line replacement for an entire CD pipeline. You point it at GHCR, give it the docker socket, and forget it exists.

Forgetting it exists is exactly the problem.

The Incident

Here's what we pieced together after the fact.

Our watchtower service was running :latest — the version everyone copy-pastes from blog posts. On May 19, upstream pushed a new :latest. Watchtower, doing its job, polled the registry, saw the new digest, and decided to update itself.

To update a container, Watchtower stops it, then starts the replacement. The replacement is launched by Watchtower. Which is the container being stopped.

You can see the deadlock from orbit. Watchtower issued docker stop against its own container. The Docker socket connection it was using to issue the "now start the new one" command got severed the instant the container died. The new container never launched. The host was left with no Watchtower at all — just the old API container, happily serving stale code, and no process polling the registry for new builds.

The comment we eventually added to docker-compose.yml captures it best:

image is PINNED, not :latest. On 2026-05-19 Watchtower self-detected an updated :latest, tried to stop itself to upgrade, cut its own Docker-socket connection mid-stop, and never restarted — leaving this droplet on stale code for 2 days until manual intervention. Bump this tag explicitly to upgrade Watchtower itself; do NOT track :latest.

Two days. We only caught it because someone noticed a bug fix that was definitely in main was definitely not in production.

Why It Couldn't Recover

Natural question: why didn't restart: unless-stopped save us? It's right there in the compose file.

restart policies fire when a container crashes — exits with a non-zero code, or the daemon restarts. They do not fire when a container is intentionally stopped via the API. From Docker's perspective, somebody asked Watchtower to stop. It stopped. Nothing to restart. The fact that the "somebody" was Watchtower itself — and that the next command in the sequence never ran because the issuer evaporated — is invisible to the daemon.

The deeper problem is that nothing was watching the watcher. Every other container on the host had Watchtower as its safety net. Watchtower had nothing. It was the bottom turtle, and when the bottom turtle quietly walks off, the whole stack just sits there.

The Two-Layer Fix

We fixed it twice, on purpose. The full Watchtower service now looks like this:

  # Watchtower - Auto-updates containers when new images are pushed to GHCR
  # Eliminates need for SSH-based deployments
  # NOTE: Using nickfedor/watchtower fork (containrrr/watchtower is archived
  # and incompatible with Docker 24+)
  #
  # IMPORTANT: image is PINNED, not :latest. On 2026-05-19 Watchtower
  # self-detected an updated :latest, tried to stop itself to upgrade, cut
  # its own Docker-socket connection mid-stop, and never restarted —
  # leaving this droplet on stale code for 2 days until manual intervention.
  # Bump this tag explicitly to upgrade Watchtower itself; do NOT track :latest.
  watchtower:
    image: nickfedor/watchtower:1.17.0
    container_name: jo4-watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json:ro
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_POLL_INTERVAL=60
      - WATCHTOWER_INCLUDE_STOPPED=false
      - WATCHTOWER_REVIVE_STOPPED=false
      - WATCHTOWER_ROLLING_RESTART=false
      # Opt-in mode: only watch containers labeled
      # com.centurylinklabs.watchtower.enable=true (i.e. jo4-api).
      - WATCHTOWER_LABEL_ENABLE=true
    restart: unless-stopped

And on the app service we want it to actually watch:

  app:
    image: ghcr.io/${GITHUB_REPOSITORY:-rathnasorg/jo4}/alertstage-api:${APP_VERSION:-latest}
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

Two layers.

Layer 1: Pin Watchtower's own image

image: nickfedor/watchtower:1.17.0. Not :latest. Not :stable. A specific tag we wrote down on purpose.

This single change makes the May 19 sequence impossible. When Watchtower asks "is there a newer image for nickfedor/watchtower:1.17.0?" the answer is, by definition, no. The only way Watchtower gets a new image is when we edit this file and bump the version — a human in the loop who can verify the new container actually came up.

(We also moved off containrrr/watchtower while here — that repo is archived and broke on Docker 24+. nickfedor/watchtower is the maintained fork.)

Layer 2: Opt-in label mode

WATCHTOWER_LABEL_ENABLE=true flips Watchtower from "watch every container on the host" to "watch only containers carrying the label com.centurylinklabs.watchtower.enable=true." We add that label to jo4-api. We do not add it to jo4-watchtower. The Watchtower container is now structurally invisible to itself.

Why Both Layers Matter

You could argue either layer alone is enough. Pin the image, no more updates, no more deadlock. Or use label-mode, Watchtower can't pick itself up regardless of tag. Pick one and move on.

We didn't, because each layer protects against a different human error six months from now.

Layer 1 fails the day somebody — maybe one of us, on autopilot — edits the compose file and changes 1.17.0 back to :latest. It's the kind of "cleanup" that looks reasonable in a PR diff. If layer 2 isn't there, that edit reintroduces the exact bug.

Layer 2 fails the day somebody adds the watchtower.enable=true label to the Watchtower container itself — maybe they're labelling everything for an inventory script, maybe a tutorial says "label all your containers." If layer 1 isn't there, Watchtower starts watching itself again.

Either failure alone is a foot. Both simultaneously, on the same PR, by the same engineer, is a foot wrapped in a kevlar boot. Defense in depth is cheap when the depth is two YAML lines.

Lessons For Any Auto-Update Tool

The Watchtower-specific story generalizes. Anything that auto-updates itself — your dependency bot, your CI runner image, your sidecar agent, your secrets-rotator — has the same shape of bug latent in it. Some patterns we now apply across the board:

Pin the version of anything that can mutate itself. Auto-update tools should not auto-update. Their version belongs in source control where a human has to type the new digit. :latest is for things being watched, not for things doing the watching.
Prefer opt-in over opt-out for blast radius. Default-watch-everything tools sweep up the watcher itself, your debug containers, your one-off docker run experiments. Opt-in lists are slightly more verbose and dramatically harder to misfire.
restart: unless-stopped is not a safety net for orchestration bugs. It catches crashes. It does not catch "I stopped myself on purpose and then died." If something in your stack has the authority to stop itself, give it a watchdog or accept it will walk off.
Silent CD failure is the worst kind. The API was up the whole time. Health checks green. Monitoring happy. The only symptom was "fixes don't reach production," which nobody catches until they go looking. Add an alert on "no successful deploy in N hours" — it would have paged us at hour 13 instead of hour 48.
Write the comment you'd want six months from now. The block we added to docker-compose.yml is the entire postmortem in five sentences. The next operator who looks at that file knows exactly why the version is pinned and what happens if they unpin it. More durable than any wiki page.

The footgun isn't gone — we still trust an external tool to mutate our running containers. But the specific path where the trust loops back on itself is closed off twice. That's enough.

Have you been bitten by a self-updating tool? What recovered you? Drop the war story in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.