DEV Community: Jihed Ben Arfa

Your Kafka Event Was Published. Your Database Transaction Failed. Now What?

Jihed Ben Arfa — Tue, 15 Sep 2026 15:36:03 +0000

I still remember the first time this happened to me in production.

A customer order was created. The database said so. The event was supposed to reach Kafka so downstream services could react. Everything looked fine on the dashboard.

Two hours later, a support ticket: the order existed, but nothing had happened downstream. No provisioning. No notification. Nothing.

The database was right. Kafka had no idea the order ever existed.

That day, I learned about the dual-write problem the hard way.

The scenario that breaks

Here's the code everyone writes at first. It looks innocent:

@Transactional
public void createOrder(Order order) {
    orderRepository.save(order);
    kafkaTemplate.send("orders", order.toEvent());
}

The intention is clear: save the order, then tell the world about it.

The problem is that these two operations live in two different systems. The database has its own transaction. Kafka has its own delivery guarantees. Nothing binds them together.

So here's what can happen:

The database commit succeeds. Kafka is temporarily unavailable. The event is lost. Your database says the order exists. Your downstream services never learn about it.
Kafka accepts the message. The database transaction then fails on commit (constraint violation, connection drop, deadlock). The event is out. The order is not. You've announced something that never happened.

Both scenarios are silent. No exception reaches your logs in a way that tells you what really went wrong. You only find out when someone downstream asks why they're missing data.

The fixes that don't fix anything

The instinct is to wrap both operations in a try-catch. Or to publish after commit with TransactionSynchronizationManager. Or to use Kafka transactions.

Each of these helps in a narrow case. None of them solves the fundamental problem:

You cannot atomically write to your database and to Kafka. They are two separate systems.

Any solution that pretends otherwise is going to leak under load, during network partitions, or when a broker restarts at the wrong moment.

So the real question is not "how do I make them atomic." It's:

How do I make my system recover correctly when they inevitably disagree?

That's a different design problem. And it has a known answer.

Transactional Outbox

The idea is simple. You stop trying to talk to Kafka from inside your business logic. Instead, you write the event to your own database, in the same transaction as the business data. Then a separate process reads those events and publishes them.

Concretely:
Application ├── DB transaction │ ├── business data (order, subscriber, provisioning request) │ └── outbox event (what should be published) │ ▼ Outbox Relay (polling or CDC) │ ▼ Kafka

The state change and the intent to publish become atomic. If the transaction commits, the event is guaranteed to be in the outbox. If the transaction rolls back, the event disappears with it.

There is no window where the database says one thing and Kafka says another.

What the code actually looks like

Here's the pattern in a Spring Boot + JPA + Kafka setup. I'm using PostgreSQL, but this works the same on Oracle.

The outbox table:

CREATE TABLE outbox_event (
    id              UUID PRIMARY KEY,
    aggregate_type  VARCHAR(64)  NOT NULL,
    aggregate_id    VARCHAR(64)  NOT NULL,
    event_type      VARCHAR(128) NOT NULL,
    payload         JSONB        NOT NULL,
    created_at      TIMESTAMP    NOT NULL DEFAULT now(),
    published_at    TIMESTAMP,
    trace_parent    VARCHAR(64)
);

CREATE INDEX idx_outbox_unpublished
    ON outbox_event (created_at)
    WHERE published_at IS NULL;

Two details matter here. The partial index keeps the relay fast as the table grows. The trace_parent column is what lets you keep distributed tracing alive across the outbox boundary — more on that later.

Writing the business data and the event together:

@Transactional
public void createOrder(Order order) {
    orderRepository.save(order);

    OutboxEvent event = OutboxEvent.builder()
        .id(UUID.randomUUID())
        .aggregateType("Order")
        .aggregateId(order.getId().toString())
        .eventType("OrderCreated")
        .payload(toJson(order))
        .traceParent(currentTraceParent())
        .build();

    outboxRepository.save(event);
}

Notice what's not happening: no kafkaTemplate.send(). The method has no idea Kafka exists. That's the point.

The relay:

@Scheduled(fixedDelay = 500)
@Transactional
public void publishPendingEvents() {
    List<OutboxEvent> batch = outboxRepository
        .findTop100ByPublishedAtIsNullOrderByCreatedAtAsc();

    for (OutboxEvent event : batch) {
        kafkaTemplate.send("orders", event.getAggregateId(), event.getPayload())
            .whenComplete((result, ex) -> {
                if (ex == null) {
                    event.markPublished();
                }
            });
    }
}

Kafka down? The events pile up in the outbox. Nobody loses anything.

Kafka comes back? The next poll drains the backlog.

The business transaction has no idea any of this happened.

The parts nobody warns you about

The pattern is not complicated. What gets people is what happens next.

Ordering. If events for the same aggregate must be processed in order, your relay has to publish them in the order they were written. But if you scale the relay horizontally, two workers can grab overlapping batches and reorder things. The fix is SELECT ... FOR UPDATE SKIP LOCKED or partitioning by aggregate ID.

Consumer idempotence. Kafka is at-least-once. The outbox guarantees your event will eventually be published — but "eventually" might mean twice if the relay crashes between the publish and the markPublished(). Your consumer has to handle that. A processed_events table with a unique constraint on event_id is the cheapest answer.

Distributed tracing. This one bit me. When the HTTP request that triggered the order ends, the trace ends with it. The outbox relay runs later, in a different thread, possibly on a different instance. If you don't carry the trace context through the outbox table, your traces stop at the HTTP boundary.

That's why trace_parent is in the schema. The relay reads it and starts a new span linked to the original trace.

Table growth. The outbox is not a log you keep forever. Purge published events on a schedule — a daily job that deletes rows where published_at < now() - 7 days is fine for most systems.

Breaking Kafka on purpose

The moment the pattern clicked for me was when I deliberately broke Kafka in a test environment and watched the system recover.

Kill the broker mid-transaction. Watch the outbox fill up. Restart Kafka. Watch the relay drain it. The business data is already in the database. The events catch up. Nobody has to intervene.

That's the real test of the pattern. Not "does it work when everything is healthy." Anyone can write code that works when everything is healthy.

The test is: does it converge back to a correct state after Kafka, the database, or the network misbehaves?

If the answer is yes, the architecture is doing its job.

Where this comes from

I work in telecom provisioning. Number portability, 5G core migration, BSS integration. In this world, a provisioning request travels through six or seven systems, some of which are owned by other companies. The network doesn't always answer. A timeout doesn't mean failure. A retry can duplicate work.

The outbox pattern is one of the tools that makes this survivable. It doesn't eliminate failure. It makes failure recoverable.

I've packaged the pattern, along with Sagas, idempotent consumers, SLA timers, and reconciliation, in a runnable example:

→ bpmn-provisioning-patterns

The Outbox implementation is in the repo, with Kafka configured to be breakable on purpose. The tests show what happens when you kill the broker mid-transaction.

If you're building anything that writes to a database and publishes to a broker, take twenty minutes to read the code. Then try to break it. That's where the learning happens.

Next in this series: Kafka is at-least-once. Your business logic should be idempotent. I'll cover processed_events, unique constraints, safe replays, and how to design a consumer that can survive a duplicate delivery without corrupting state.

Why Keycloak Roles Don't Reach Spring Security

Jihed Ben Arfa — Tue, 25 Aug 2026 20:18:21 +0000

It happens to every team migrating to Keycloak. You've set up your realm, created roles, and assigned them to your users. You configure your Spring Boot application as an OAuth2 Resource Server. You log in, get a valid JWT, and hit your secured endpoint.

403 Forbidden.

You decode the JWT, and the roles are right there. Yet Spring Security acts as if your user has no privileges at all. Here's why this happens, why it often breaks only in production, and how to fix it cleanly.

The symptom: 403 despite a valid token

Let's say you secure an endpoint expecting an admin role:

@GetMapping("/api/admin")
@PreAuthorize("hasRole('admin')")
public String adminOnly() {
    return "Secret data";
}

You decode your Keycloak access token (e.g. using jwt.io) and see this payload:

{
  "realm\_access": {
    "roles": \[
      "admin",
      "user"
    ]
  },
  "resource\_access": {
    "my-client": {
      "roles": \[
        "manager"
      ]
    }
  },
  "preferred\_username": "jihed"
}

The role is right there. So why the 403?

Why it breaks: the GrantedAuthority prefix

Spring Security relies on a JwtAuthenticationConverter to translate the incoming JWT into an Authentication object. By default, Spring looks for a claim named scope or scp. If it finds one, it extracts the values and prefixes them with SCOPE\_.

If you want roles instead, you might have tried setting a custom claim name — but the default converter still expects a flat array of strings, and prefixes whatever it finds with ROLE\_.

Keycloak nests roles inside realm\_access.roles (realm-wide roles) and resource\_access.<client\_id>.roles (client-specific roles). Spring Security's default converter has no idea how to traverse that nested structure. It sees no roles, grants no authorities, and denies access.

The realm vs. resource trap

A common pitfall: fixing this for realm roles while ignoring client roles.

In development, teams often use realm roles because they're easier to assign globally. They write a quick converter that reads realm\_access.roles, and everything works.

Then comes production. Security policy dictates that roles must be scoped to the specific application (client roles) rather than the entire realm. The infrastructure team provisions client roles under resource\_access.my-client.roles. The application deploys, receives the new tokens, and fails with 403 because the custom converter only ever looked at the realm level.

The fix: a custom JwtAuthenticationConverter

To solve this robustly, you need a converter that extracts both realm and resource roles, applies the ROLE\_ prefix, and merges them into a single collection of GrantedAuthority objects:

import org.springframework.core.convert.converter.Converter;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.core.authority.SimpleGrantedAuthority;
import org.springframework.security.oauth2.jwt.Jwt;

import java.util.Collection;
import java.util.Map;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class KeycloakRoleConverter implements Converter<Jwt, Collection<GrantedAuthority>> {

    private final String clientId;

    public KeycloakRoleConverter(String clientId) {
        this.clientId = clientId;
    }

    @Override
    public Collection<GrantedAuthority> convert(Jwt jwt) {
        Stream<String> realmRoles = extractRealmRoles(jwt);
        Stream<String> clientRoles = extractClientRoles(jwt, this.clientId);

        return Stream.concat(realmRoles, clientRoles)
                .map(role -> new SimpleGrantedAuthority("ROLE\_" + role))
                .collect(Collectors.toSet());
    }

    @SuppressWarnings("unchecked")
    private Stream<String> extractRealmRoles(Jwt jwt) {
        Map<String, Object> realmAccess = jwt.getClaimAsMap("realm\_access");
        if (realmAccess == null || !realmAccess.containsKey("roles")) {
            return Stream.empty();
        }
        return ((Collection<String>) realmAccess.get("roles")).stream();
    }

    @SuppressWarnings("unchecked")
    private Stream<String> extractClientRoles(Jwt jwt, String clientId) {
        Map<String, Object> resourceAccess = jwt.getClaimAsMap("resource\_access");
        if (resourceAccess == null || !resourceAccess.containsKey(clientId)) {
            return Stream.empty();
        }
        Map<String, Object> clientAccess = (Map<String, Object>) resourceAccess.get(clientId);
        if (clientAccess == null || !clientAccess.containsKey("roles")) {
            return Stream.empty();
        }
        return ((Collection<String>) clientAccess.get("roles")).stream();
    }
}

Plug it into your security configuration:

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationConverter;
import org.springframework.security.web.SecurityFilterChain;

@Configuration
public class SecurityConfig {

    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        JwtAuthenticationConverter jwtConverter = new JwtAuthenticationConverter();
        // Pass your specific client ID here
        jwtConverter.setJwtGrantedAuthoritiesConverter(new KeycloakRoleConverter("my-client"));

        http.oauth2ResourceServer(oauth2 -> oauth2
                .jwt(jwt -> jwt.jwtAuthenticationConverter(jwtConverter))
        );

        http.authorizeHttpRequests(authz -> authz
                .requestMatchers("/api/admin").hasRole("admin")
                .anyRequest().authenticated()
        );

        return http.build();
    }
}

Returning clean errors: RFC 7807

When a 403 or 401 does happen, Spring Security's default behavior in a REST API isn't great — sometimes an empty body, sometimes an HTML error page. Modern APIs should return structured errors, and RFC 7807 (Problem Details for HTTP APIs) is the standard for that.

You can implement custom entry points to return properly formatted ProblemDetail objects:

import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.http.HttpStatus;
import org.springframework.http.ProblemDetail;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.security.web.access.AccessDeniedHandler;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.io.IOException;
import java.net.URI;

public class ProblemDetailAccessDeniedHandler implements AccessDeniedHandler {

    private final ObjectMapper objectMapper = new ObjectMapper();

    @Override
    public void handle(HttpServletRequest request, HttpServletResponse response, AccessDeniedException accessDeniedException) throws IOException {
        response.setStatus(HttpStatus.FORBIDDEN.value());
        response.setContentType("application/problem+json");

        ProblemDetail problem = ProblemDetail.forStatusAndDetail(HttpStatus.FORBIDDEN, "Access Denied");
        problem.setInstance(URI.create(request.getRequestURI()));

        objectMapper.writeValue(response.getOutputStream(), problem);
    }
}

Plug this into your HttpSecurity configuration under .exceptionHandling(), and your clients get clean, parsable JSON errors instead.

Verification

To make sure your mapping actually works, write an integration test with Testcontainers: spin up a real Keycloak instance, import a realm with your test users and roles, and issue a real request.

@Test
void shouldAllowAdminAccess() {
    String token = keycloak.getAccessToken("admin-user", "password");

    mockMvc.perform(get("/api/admin")
            .header("Authorization", "Bearer " + token))
            .andExpect(status().isOk());
}

Getting this out of your codebase

If you don't want to maintain this boilerplate in every microservice, I packaged this exact configuration into a Spring Boot starter: spring-keycloak-toolkit. It handles the realm and resource role mapping automatically and enforces RFC 7807 error responses for security exceptions.

It's not on Maven Central yet, but it's on JitPack, which builds straight from the GitHub tag:

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

<dependency>
    <groupId>com.github.jihedbfr-art</groupId>
    <artifactId>spring-keycloak-toolkit</artifactId>
    <version>v0.1.0</version>
</dependency>

Or clone it and run mvn install locally if you'd rather not pull from JitPack. Either way, set the client ID in your properties and the role mapping just works:

jihedapps:
  keycloak-toolkit:
    resource-id: my-client

Check the repo's README for the current release status if you're reading this later — a Central release may already be out by then.

---

Code and issues: github.com/jihedbfr-art/spring-keycloak-toolkit

Instrumenting a Keycloak SPI provider so the next silent JDBC bug doesn't need a code review to find it

Jihed Ben Arfa — Sun, 19 Jul 2026 18:08:13 +0000

Project: keycloak-spi-workbench — custom Keycloak SPI providers, one of them a read-only user storage federation (legacy-user-storage) that reads users from an existing JDBC table instead of forcing a big-bang migration into Keycloak's own store.

What happened first, no Sentry involved: LegacyUserStorageProvider#searchForUserByUserAttributeStream was calling getUserByUsername twice per invocation — once for a null check, once again to build the returned stream:

return getUserByUsername(realm, attrValue) != null
        ? Stream.of(getUserByUsername(realm, attrValue))
        : Stream.empty();

Two calls, two separate JDBC round trips, against a repository with no connection pooling (LegacyUserRepository opens a fresh Connection per query, on purpose — see the repo's README on why). Every username-attribute lookup, twice the DB load it needed. No exception, no wrong result, just silently double the work on every call. Caught by reading the method, not by any tool — fixed by storing the result once and reusing it, with a regression test that fails if it comes back.

That's exactly the kind of bug that's invisible until someone happens to read that method, and the reason I added Sentry next: catching this again shouldn't depend on a code review catching it first.

Where Sentry goes in: LegacyUserRepository's three query methods (findOneWhere, search, count) now open a child span under whatever transaction Keycloak's own request tracing has active — legacy_db.find_one, legacy_db.search — and search records how many rows came back:

private ISpan startSpan(String operation, String description) {
    ISpan parent = Sentry.getSpan();
    return parent != null ? parent.startChild(operation, description) : NoOpSpan.getInstance();
}

Falls back to NoOpSpan when there's no active transaction — never throws, never blocks a login on Sentry being reachable. Any SQLException also gets Sentry.captureException(e) before it's wrapped and rethrown, so a failing legacy database shows up as an event, not just a stack trace in a Keycloak log nobody's tailing.

The part that mattered more than the instrumentation itself: making it opt-in.

@Override
public void init(Config.Scope config) {
    String dsn = System.getenv("SENTRY_DSN");
    if (dsn != null && !dsn.isBlank() && !Sentry.isEnabled()) {
        Sentry.init(options -> {
            options.setDsn(dsn);
            options.setTracesSampleRate(1.0);
            options.setEnvironment(System.getenv().getOrDefault("SENTRY_ENVIRONMENT", "production"));
        });
    }
}

No SENTRY_DSN set, Sentry.init never runs, every span call resolves to NoOpSpan, every captureException call is a no-op. This is a provider other people install into their own Keycloak instance — it doesn't get to assume they want a Sentry account wired into their auth server, and it doesn't get to silently start sending data anywhere without an explicit environment variable making that choice.

Where this actually earns its keep: a per-user JDBC query duplicating itself is the boring, common case. The interesting one is a legacy database that's slow for one specific customer's data shape, or a driver that starts throwing intermittently under load — the search span's result_count and the per-query timing are exactly what would surface a slow query pattern before it shows up as "login is slow" tickets with nothing pointing at the actual repository call underneath.

The bug that only showed up because my test was too fast

Jihed Ben Arfa — Sun, 19 Jul 2026 17:59:34 +0000

A couple hundred milliseconds cost me an hour, and the annoying part is I'd written the fix for it before I even had the bug.

I was writing a Testcontainers deployment check for a custom Keycloak authenticator — boot a real Keycloak server with the built provider jar dropped into providers/, hit the admin REST API, confirm the provider shows up. First pass, I picked the wait strategy the way you'd expect:

.waitingFor(Wait.forLogMessage(".*Running the server in development mode.*", 1))

Keycloak prints that line during start-dev, so it felt like a reasonable "the server is up" signal. The test ran, and about one time in three it failed with a plain connection reset trying to hit the admin token endpoint. No stack trace pointing anywhere useful, just "connection refused," which is the least helpful error message a network client can give you.

The instinct is to blame Testcontainers, or Docker networking, or just add a Thread.sleep(2000) and move on. I've done that before and regretted it — a sleep just moves the flake to a slower CI runner instead of fixing it.

What was actually happening: that log line prints the moment Keycloak's startup sequence reaches "development mode," which is before the HTTP listener has finished binding and accepting connections. The gap is small — a couple hundred milliseconds most of the time — which is exactly why it only failed sometimes. Fast enough CI runner, you never see it. Slightly loaded runner, the test wins the race against the listener and gets a reset.

The fix was smaller than the investigation:

.waitingFor(Wait.forHttp("/realms/master")
        .forStatusCode(200)
        .withStartupTimeout(Duration.ofMinutes(3)))

Poll the actual endpoint you're about to call, instead of trusting a log line that tells you what the server is doing, not what it's ready for. Obvious in hindsight. It always is.

The part that stuck with me: a log-message wait strategy will pass code review without question, because it reads as "wait until the server says it's running" — which sounds exactly right. It just isn't the same claim as "wait until the server is accepting connections," and nothing about the code makes that distinction visible. You only find the gap between those two claims by having the test flake on you, ideally in CI where you can't step through it, which is the least convenient possible place to learn it.

Same class of bug, one level up: don't trust a stated intention, wait on the actual side effect you depend on. Applies well past Testcontainers — anywhere something logs "starting" before it's actually ready to be used.

Keycloak lets you pick a login option that's guaranteed to fail — here's the fix

Jihed Ben Arfa — Sun, 19 Jul 2026 17:59:32 +0000

Project: keycloak/keycloak — the identity and access management server. PR: #51021, fixing #46481.

The bug: if you write a custom Keycloak Authenticator that isn't a CredentialValidator — anything that isn't a password, OTP, WebAuthn, that kind of thing — and it requires a user to already be identified, Keycloak's "Try another way" screen will offer it as an option even when it isn't actually usable for that user. Pick it, and you get an immediate CREDENTIAL_SETUP_REQUIRED error. The option shouldn't have been on the screen in the first place.

Root cause, traced through two files in the services module:

AuthenticationSelectionResolver#addSimpleAuthenticationExecution decides what goes on the selection list. It only checks one thing:

Authenticator localAuthenticator = processor.getSession().getProvider(Authenticator.class, execution.getAuthenticator());
if (!(localAuthenticator instanceof CredentialValidator)) {
    nonCredentialExecutions.add(execution);
} else {
    ...
}

Not a CredentialValidator? Straight onto the list, no further questions. Nothing here checks authenticator.configuredFor(...) or the factory's isUserSetupAllowed().

There's a second, separate instance of the same gap. DefaultAuthenticationFlow.processSingleFlowExecutionModel has its own configuredFor/isUserSetupAllowed guard meant to catch exactly this — but only for requiresUser() == false authenticators, and only if they're a CredentialValidator:

if ((authUser != null) &&
        !authenticator.configuredFor(processor.getSession(), processor.getRealm(), authUser) &&
        !factory.isUserSetupAllowed() &&
        (authenticator instanceof CredentialValidator)) {
    throw new AuthenticationFlowException(...);
}

That instanceof CredentialValidator at the end means a plain authenticator in that exact shape runs authenticate() completely unguarded, even as a REQUIRED step — a code path that never goes through the selection list at all, so fixing the resolver alone wouldn't have caught it.

The fix: added isSelectableForCurrentUser to the resolver — authenticators that don't require a user pass straight through, the rest only get listed if they're configured for the current user or the factory allows self-setup on the spot. And dropped the instanceof CredentialValidator restriction in the flow's own guard, so it applies the same way the requiresUser() == true branch right above it already does.

Test: AuthenticationSelectionResolverTest, five cases covering the full matrix — no user yet, configured, not configured but self-setup allowed, not configured and not allowed, and authenticators that don't require a user at all (always selectable, regardless of the rest).

This one didn't need any special tooling — just reading the two files the original report pointed at and following the logic all the way to where it actually breaks, which is most of what real bug fixing is.

Testing Keycloak SPIs with Testcontainers — the part every tutorial skips

Jihed Ben Arfa — Sat, 18 Jul 2026 23:39:15 +0000

Most "here's how to write a custom Keycloak authenticator" posts end at mvn package. You get a class that implements Authenticator, a factory, a META-INF/services file, and a "drop the jar in providers/ and restart" instruction — and that's where the tutorial stops. Nobody shows you how to actually test the thing, which is a problem, because SPI development has a genuinely nasty feedback loop: package, copy the jar, restart Keycloak, click through a login flow by hand, repeat. I've spent enough evenings doing that manually on custom authenticators to want it gone.

The unit-testable part is easy. If your authenticator's logic is a pure function of some inputs (a user attribute, a client ID, a config value), you extract that into a method and mock the Keycloak model objects around it. That's normal JUnit + Mockito, nothing SPI-specific. The part that actually needs a real server is: does the jar deploy? Does META-INF/services point at the right class? Did you get a method signature wrong that only shows up as a stack trace on server startup? None of that shows up in a unit test, because a unit test never boots Keycloak.

Booting a real Keycloak in a test

Testcontainers makes this less painful than it sounds. You start the official image, copy your built jar straight into providers/ before the container starts, and let Keycloak boot normally:

@Container
private final GenericContainer<?> keycloak = new GenericContainer<>("quay.io/keycloak/keycloak:26.6")
        .withCopyFileToContainer(MountableFile.forHostPath(builtJarPath()), "/opt/keycloak/providers/my-spi.jar")
        .withEnv("KEYCLOAK_ADMIN", "admin")
        .withEnv("KEYCLOAK_ADMIN_PASSWORD", "admin")
        .withExposedPorts(8080)
        .withCommand("start-dev")
        .waitingFor(Wait.forHttp("/realms/master").forStatusCode(200));

The wait strategy is the one detail that'll cost you an hour if you get it wrong. My first pass used Wait.forLogMessage(".*Running the server in development mode.*", 1) — that line prints during startup, but before the HTTP listener is actually accepting connections. The test would then immediately try to hit the admin API and get a connection reset, because the server said "I'm running" a couple hundred milliseconds before it actually was. Polling the real endpoint with Wait.forHttp(...) instead of trusting a log line fixed it — obvious in hindsight, and exactly the kind of thing you only learn by having the test flake on you in CI.

Once the container's up, you hit the admin REST API the same way an admin console would:

HttpRequest providersRequest = HttpRequest.newBuilder()
        .uri(URI.create(baseUrl + "/admin/realms/master/authentication/authenticator-providers"))
        .header("Authorization", "Bearer " + accessToken)
        .GET()
        .build();

and assert your provider ID shows up in the list. That's it — that's the check that actually catches packaging mistakes: wrong META-INF/services entry, a factory that throws in init(), a missing dependency that only fails at classload time. None of that surfaces in a unit test.

Same idea, different provider type

I built a second provider — an event listener that publishes login events to Kafka — right after, and the same "don't mock the thing you're trying to prove works" logic applied to it, just with a different target. Mocking the KafkaProducer in a test would let a subtly wrong producer config (bad serializer, bad acks setting) pass silently. So instead of mocking, the integration test spins up a real Kafka broker via Testcontainers, constructs the provider with a real producer pointed at it, fires an event, and reads the message back with a real consumer. If the JSON shape is wrong, or the key/value serializers are misconfigured, the test catches it — a mock never would have.

The Maven wiring

One thing that trips people up: these integration tests need the packaged jar to already exist, so they can't run in the same phase as unit tests. Surefire runs unit tests at test (excluding anything named *IT.java), and Failsafe runs the integration tests at verify, after package has already produced the jar:

<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <excludes><exclude>**/*IT.java</exclude></excludes>
    </configuration>
</plugin>
<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <executions>
        <execution><goals><goal>integration-test</goal><goal>verify</goal></goals></execution>
    </executions>
</plugin>

mvn test for the fast feedback loop while you're writing the logic, mvn verify when you actually need to know the thing deploys — which is also what CI runs, so a green build actually means something.

None of this is complicated once it's set up. It's just work nobody writes about, because "here's a class that implements an interface" is a much shorter blog post than "here's how you prove the class actually works on a real server." The second one's the only one that matters once you're shipping.

Your Keycloak roles aren't working in Spring Security. Here's the actual reason.

Jihed Ben Arfa — Sat, 18 Jul 2026 23:39:14 +0000

If you've wired up a Spring Boot resource server behind Keycloak and your @PreAuthorize("hasRole('ADMIN')") is silently returning 403 for a user you can see has the ADMIN role in the admin console, you're not going crazy. This happens on basically every first Keycloak + Spring Security integration, and the reason is boring once you see it: Spring Security's default JWT converter has no idea Keycloak exists.

I've hit this on every Keycloak-secured backend I've built over the past couple of years — most recently on a multi-service Spring Boot 3 platform where I ended up owning the IAM layer, including custom Keycloak SPI authenticators. Same wiring problem, every time, until I finally pulled the fix out into a small library instead of copy-pasting it again.

The actual token shape

Spring Security's JwtAuthenticationConverter expects authorities to come from a scope (or scp) claim — a flat, space-separated string, OAuth2-style. That's the generic spec-y default. Keycloak doesn't do that. A Keycloak access token puts roles here instead:

{
  "realm_access": {
    "roles": ["offline_access", "uma_authorization"]
  },
  "resource_access": {
    "my-client": {
      "roles": ["ADMIN", "VIEWER"]
    }
  }
}

Realm roles under realm_access.roles, client roles nested under resource_access.<clientId>.roles. Spring Security never looks there. So JwtAuthenticationConverter runs, finds no scope claim, produces zero GrantedAuthority objects, and every hasRole(...) check quietly fails — no error, no stack trace, just "access denied" for a user who is very clearly in the right group.

The fix, the manual version

You write your own Converter<Jwt, AbstractAuthenticationToken> that reads both claims, flattens them into SimpleGrantedAuthority with a ROLE_ prefix (Spring Security's convention), and registers it on the JwtAuthenticationConverter:

public class KeycloakRealmRoleConverter implements Converter<Jwt, Collection<GrantedAuthority>> {

    @Override
    public Collection<GrantedAuthority> convert(Jwt jwt) {
        Map<String, Object> realmAccess = jwt.getClaim("realm_access");
        List<String> roles = realmAccess != null
                ? (List<String>) realmAccess.getOrDefault("roles", List.of())
                : List.of();

        return roles.stream()
                .map(role -> new SimpleGrantedAuthority("ROLE_" + role))
                .collect(Collectors.toList());
    }
}

Wire it into a JwtAuthenticationConverter, register that as a bean, and roles start working. That's maybe twenty lines and it's fine for a single service. The annoying part is you'll write this exact converter again on the next service, and the one after that, usually slightly differently each time (some people prefix with ROLE_, some don't, some forget client roles exist at all and only handle realm roles).

Where I landed

After the third time writing basically the same class, I pulled it into spring-keycloak-toolkit — an auto-configuration that registers the converter for you, handles both realm and client roles, and also fixes the second thing that bites you right after the first one: Spring Security's default 401/403 responses are empty bodies, which is useless if anything downstream (a frontend, an API gateway, a support engineer reading logs) needs to know why a request got rejected. The library adds RFC 7807 problem+json bodies for both cases instead.

What it deliberately does not do is touch your SecurityFilterChain or guess which endpoints should be public. I thought about auto-wiring that too, and decided against it — guessing your endpoint matchers wrong and failing silently is worse than making you write four lines of config yourself. A library that gets your security posture subtly wrong is more dangerous than one that does less.

If you're wiring up Keycloak with Spring Security and hitting this, the manual converter above will get you unstuck in five minutes. If you're doing it more than once, the library's on Maven via JitPack — link's in the repo.