DEV Community: Mohammed Ammer

Optimizing Keycloak Caches: Best Practices for Embedded and External Infinispan

Mohammed Ammer — Mon, 16 Sep 2024 15:14:36 +0000

Although setting up Keycloak is relatively straightforward, regardless of your infrastructure's complexity, optimizing its performance for your specific workload can be challenging.

One common approach is to use an external Infinispan with a database persistence to store sessions outside of Keycloak, at least until version 26 makes the user session persistence feature (introduced in Keycloak version 25) a permanent part of Keycloak, moving beyond its previous preview status.

In-Memory Cache

The biggest challenge many encounter with "in-memory" caches is managing memory usage. It's crucial for applications to have defined memory limits; otherwise, you risk facing endless issues, including memory overconsumption and frequent application crashes.

At this point, we can identify two types of caches:

External Infinispan Cache: This operates independently of Keycloak but is used by Keycloak as a remote cache.
Embedded Infinispan Cache: Managed within Keycloak, this cache allows data (e.g. sessions) to be shared across Keycloak nodes/containers, reducing the need for frequent reads from the external cache.

Let's start with the external Infinispan cache, as it's crucial for safeguarding sessions against any unexpected issues in the Keycloak cluster.

External Infinispan Cache

As mentioned earlier, using an external Infinispan cache solves part of the problem, but since it still stores data in memory, there's a risk of data loss. Adding persistent storage (e.g., a database) ensures the data remains safe.

Limit the memory size

With data persisted in the database, you can set a smaller memory limit for the Infinispan cache configuration to suit your cost-efficient resources. However, you must carefully assess the database capacity that matches your load, as less memory for cache means increased demand on the database.
See Configuring maximum count eviction from Infinispan Documentation



<distributed-cache>
  <memory max-count="10000" when-full="REMOVE"/>
</distributed-cache>

Avoid cache Preload

When an Infinispan instance starts, it preloads sessions from the database into the cache, even if the cache is not currently in use, to meet the max-count configured for each cache. To enable "lazy loading"—where Infinispan loads data only when needed—set the preload option to false. Additionally, ensure that the shared flag is set to true.
See Configuring Persistence Store from Infinispan Documentation



<distributed-cache  name="sessions">
    <persistence>
        <string-keyed-jdbc-store
            xmlns="urn:infinispan:config:store:jdbc:15.0" preload="false" shared="true" dialect="POSTGRES">
            <string-keyed-table prefix="EXT" create-on-start="true" drop-on-exit="false">
                <id-column name="id" type="VARCHAR(255)"/>
                <data-column name="data" type="BYTEA"/>
                <timestamp-column name="timestamp" type="BIGINT"/>
                <segment-column name="segment" type="INT"/>
            </string-keyed-table>
        </string-keyed-jdbc-store>
    </persistence>
</distributed-cache>

Disable the statistics

Unfortunately, querying statistics in Infinispan can lead to numerous executions of the following query:



SELECT COUNT(*) FROM "kc_sessions" WHERE timestamp < 0 OR timestamp > $1

As the amount of data in your sessions table increases, the performance of this query degrades. This is because the query is highly CPU-intensive for the database. If you're using AWS RDS, this could result in exhausting your CPU credits, causing your cluster to become unable to handle incoming traffic effectively.

Does this mean you won’t be able to monitor metrics for the sessions table? Not at all.

If you enable Keycloak metrics, you can still track the number of entries in the external Infinispan cache database through the vendor_cache_store_number_of_persisted_entries metric.

To disable statistics



<distributed-cache name="sessions" statistics="false">
</distributed-cache>

Limit owners

To conserve memory in your Infinispan instance, you can configure it to have the minimum number of owners for the sessions. As long as your database can handle I/O operations within an acceptable timeframe, this approach can be effective.



<distributed-cache name="sessions" owners="2">
</distributed-cache>

State transfer

State transfer is the process by which data is moved between nodes in a distributed cache cluster. This mechanism is essential for ensuring that all nodes in the cluster have a consistent view of the cache's data, especially when nodes are added, removed, or rebalanced.

As long as the preload is set to false, the initial state transfer doesn't really matter as no preload to be considered anyways.



<distributed-cache name="sessions">
    <state-transfer timeout="60000" await-initial-transfer="true"/>
</distributed-cache>

Embedded Infinispan Cache

The embedded Infinispan cache in Keycloak is an internal, local cache used to store session and configuration data to improve performance and reduce database load; however, improper configuration can lead to memory issues for Keycloak.

Limit the memory size

By default, the embedded Infinispan cache managed by Keycloak does not impose a limit on the number of stored sessions. This means that if sessions are configured to last for extended periods, such as 3 months or a year, they will remain in Keycloak until memory is exhausted. When memory becomes full, it can lead to issues.

Consider whether you truly need the embedded Infinispan cache if you already have an external cache in place. While I wouldn’t recommend disabling the embedded cache entirely, setting hard limits can help manage memory usage and prevent problems.

For example, if you're handling around 5,000 sessions at a time, setting a limit of 10,000 or even 50,000 sessions in memory should be reasonable. Once the embedded cache reaches its limit, it will evict sessions, but only from the embedded cache, not the external cache.

When a session is needed and has been evicted from the embedded cache, Keycloak will reload it from the external cache.

To ensure this setup works, you'll need to configure the embedded cache settings in Keycloak appropriately.



<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:15.0 http://www.infinispan.org/schemas/infinispan-config-15.0.xsd"
    xmlns="urn:infinispan:config:15.0">

    <cache-container name="keycloak">
        ...
        <distributed-cache name="sessions">
            <memory max-count="50000"/>
        </distributed-cache>
        ...
    </cache-container>
</infinispan>

Limit owners

Since sessions are stored in the external cache, which typically uses two owners, having just one owner in the embedded cache is usually sufficient. As long as the traffic to the external cache remains manageable for your system, setting the embedded cache to one owner per session should work well.



<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:15.0 http://www.infinispan.org/schemas/infinispan-config-15.0.xsd"
    xmlns="urn:infinispan:config:15.0">
    <cache-container name="keycloak">
        ...
        <distributed-cache name="sessions" owners="1">
        </distributed-cache>
        ...
    </cache-container>
</infinispan>

Keep eyes on Keycloak API performance

While this topic is a bit advanced, it could be quite relevant depending on your needs.

Consider a scenario where you have a custom user attribute, such as globalUserId, which serves as a primary key identifier for users within your ecosystem (distinct from the Keycloak USER ID - UUID).

In this case, if you need to perform operations on a Keycloak user, you would first need to retrieve the Keycloak User ID using your globalUserId. This process can introduce performance bottlenecks.

Keycloak uses JPA to manage its entities. When you use the official Keycloak API to query users by attributes with a request like:



POST /{realm}/users?q=globalUserId:value

the underlying database query executed is:



SELECT ue1_0.ID, ue1_0.CREATED_TIMESTAMP, ue1_0.EMAIL, ue1_0.EMAIL_CONSTRAINT, ue1_0.EMAIL_VERIFIED, ue1_0.ENABLED, ue1_0.FEDERATION_LINK, ue1_0.FIRST_NAME, ue1_0.LAST_NAME, ue1_0.NOT_BEFORE, ue1_0.REALM_ID, ue1_0.SERVICE_ACCOUNT_CLIENT_LINK, ue1_0.USERNAME 
FROM USER_ENTITY ue1_0 
LEFT JOIN USER_ATTRIBUTE a1_0 ON ue1_0.ID = a1_0.USER_ID 
WHERE a1_0.NAME = $1 
AND LOWER(a1_0.VALUE) = $2 
AND ue1_0.REALM_ID = $3 
ORDER BY ue1_0.USERNAME 
OFFSET $4 ROWS 
FETCH FIRST $5 ROWS ONLY

As shown, this query joins the USER_ENTITY table with the USER_ATTRIBUTE table to fetch the entire user entity. For a unique custom attribute with a small user base (fewer than a thousand users), this query can result in a latency of up to one second to retrieve user information, including the Keycloak UUID.

Unfortunately, Keycloak’s Admin API does not offer a way to query directly for the UUID from the USER_ATTRIBUTE table.

To address this, you might consider creating a custom Service Provider Interface (SPI) to build an endpoint that queries only the user attributes and returns the Keycloak user IDs associated with the queried attribute.

There are many resources available online for creating a custom API in Keycloak using SPI, specifically looking into RealmResourceProviderFactory. If you’d like a detailed guide on this process, feel free to ask for a blog post!

I hope you find it useful!

Understanding the 0.6-Second Detection Time for Full Outages

Mohammed Ammer — Sat, 14 Sep 2024 14:32:54 +0000

If you’ve explored the widely-read workbook on Site Reliability Engineering (SRE), you might have encountered the section on the five methods for alerting based on Service Level Objectives (SLOs) - Chapter 5. In the first method, which is the most basic and not generally recommended, an alert is triggered when the target error rate exceeds the SLO threshold. This is represented as "1: Target Error Rate ≥ SLO Threshold."

One aspect of this method is the claim that the detection time for a full outage is just 0.6 seconds. I found myself questioning where this 0.6-second figure comes from, as it seems calculated.

The error rate for alerting is designed to trigger as soon as possible, especially in the case of a full outage where the error rate would be 100%. So, why is the detection time cited as 0.6 seconds?

Despite extensive searching and effort to understand this, the explanation was not clear to me. I talked to @Ray to give some help and after all clear, I decided to write this blog post, hoping to clarify the concept in my own way.

In the book, it states:

if the SLO is 99.9% over 30 days, alert if the error rate over the previous 10 minutes is ≥ 0.1%:

- alert: HighErrorRate
  expr: job:slo_errors_per_request:ratio_rate10m{job="myjob"} >= 0.001

Assumptions

To grasp the 0.6-second detection time, let’s make two assumptions:

Alerts are evaluated in real-time.
We have a consistent rate of 100 events per second.

Question

From the graph provided, it’s evident that with an error rate of 1%, the detection time is around 1 minute.

Given that there is no fractional time involved, let's address the following question:

Why is the detection time 1 minute for an error rate of 1%?

The alert expression used is:

sum(rate(slo_errors[10m])) by (job)
/
sum(rate(slo_requests|10m])) by (job) > 0.001

This calculates a time window of 10 minutes.

So when:

1 second = 100 events
10 minutes = 600 seconds = 60,000 events

To achieve a 0.1% error rate (i.e. alert to fire), you need:

0.1% of the 60,000 events to fail in 10 minutes = 60 events

For an error rate of 1%:

1 failed event every 1 second To accumulate these 60 failed events and meet the 0.1% error rate threshold, it would take 60 seconds (1 minute).

For a full outage (100% error rate):

100 failed event every 1 second To accumulate these 60 failed events and meet the 0.1% error rate threshold, it would take 0.6 second!

Let us put it in equation

Detection Time Equation

Define Parameters:
- Event Rate (R): Number of events per second.
- Failure Rate Threshold (F): Error rate threshold in decimal form (e.g., 0.001 for 0.1%).
- Evaluation Window (W): Time window in seconds for the alerting calculation (e.g., 600 seconds for 10 minutes).
Calculate Total Events Needed for Threshold:
- Total events in the evaluation window = R * W
- Required number of failed events to hit the threshold = (R * W) * F
Determine Detection Time:
- Detection time (T) is the time required to accumulate the number of failed events necessary to meet the threshold.

The formula to calculate detection time is:

T = Required Failed Events / Failure Rate

Simplifying, the detection time can be given by:

T = ((R * W) * F) / R

Example

For instance, if the event rate is 100 events per second (R = 100), the error rate threshold is 0.1% (F = 0.001), and the evaluation window is 10 minutes (600 seconds) (W = 600):

Calculate Required Failed Events:
- Total events in 10 minutes = R * W = 100 * 600 = 60,000
- Required failed events = 60,000 * 0.001 = 60
Calculate Detection Time:
- Detection time = 60 / 100 = 0.6 seconds

Thus, for a full outage, where the failure rate is 100%, the detection time would be around 0.6 seconds.

I hope this explanation helps clear up the 0.6-second detection time!

Simplifying Keycloak Configuration with Terraform and Terragrunt

Mohammed Ammer — Sat, 04 May 2024 23:51:48 +0000

Keycloak, an open-source identity and access management solution, provides robust authentication and authorization services for modern applications. However, configuring Keycloak instances manually can be tedious and error-prone. In this blog post, we'll explore how to simplify Keycloak configuration using Terraform and Terragrunt, enabling infrastructure as code (IaC) practices for managing Keycloak realms, clients, users, and more.

Why Terraform and Terragrunt?

Terraform is an open-source tool that lets you manage your infrastructure as code. With Terraform, you can define your infrastructure in simple configuration files, and Terraform will automatically create, manage, and update your infrastructure according to those configurations.

Terragrunt is a thin wrapper for Terraform that provides extra tools for keeping your Terraform configurations DRY, managing remote state, and working with multiple Terraform modules. It helps you maintain clean, DRY, and repeatable Terraform code by providing extra functionality and best practices.

By leveraging both, we can have easy:

Infrastructure as Code (IaC): By defining Keycloak configuration as code, we can version control, review changes, and ensure consistency across environments.
Modularization: Modularize our Keycloak configuration, making it easier to manage complex setups.
State Management: Manage the state of our infrastructure, preventing configuration drift and ensuring that our infrastructure remains in the desired state.

Let us go!

Install Terraform and Terragrunt

Make sure you have Terraform and Terragrunt installed on your machine. You can find installation instructions on the official Terraform and Terragrunt documentation

Terraform: https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli
Terragrunt: https://terragrunt.gruntwork.io/docs/getting-started/install/

Project Structure

Creating a one-size-fits-all structure for a Terraform project can be challenging because it largely depends on the specific requirements of each project. Below is the structure I've found most suitable for organizing Keycloak components, concepts, and setup.

.
├── README.md                 <- The project overview
├── .tool-versions            <- Used tools versions (managed by asdf. see https://asdf-vm.com) 
├── README.md                 <- The project overview
├── modules                   <- Terraform modules
|   └── common
│       ├── provider.tf
│       └── variables.tf
│       └── output.tf
│       └── main.tf
│       └── README.md
│       └── docs/
|   └── clients
│       ├── provider.tf
│       └── variables.tf
│       └── output.tf
│       └── main.tf
│       └── README.md
│       └── docs/
|   └── my-realm
│       ├── provider.tf
│       └── variables.tf
│       └── output.tf
│       └── main.tf
│       └── README.md
│       └── docs/
|   └── other
│       ├── provider.tf
│       └── variables.tf
│       └── output.tf
│       └── main.tf
│       └── README.md
│       └── docs/
└── how-to                    <- Documentation
└── stage                     <- Terraform for environment stage
    ├── .terraform.lock.hcl   <- Terraform lock file
    └── terragrunt.hcl        <- Terragrunt file
    └── env.yaml              <- environment related variables
    └── main.tf               <- environment modules
└── prod                      <- Terraform for environment prod
    ├── .terraform.lock.hcl   <- Terraform lock file
    └── terragrunt.hcl        <- Terragrunt file
    └── env.yaml              <- environment related variables
    └── main.tf               <- environment modules

└── local                     <- Terraform for environment local
    ├── .terraform.lock.hcl   <- Terraform lock file
    └── terragrunt.hcl        <- Terragrunt file
    └── env.yaml              <- environment related variables
    └── main.tf               <- environment modules

In this project structure, I've included a modules directory containing a set of modules shared across all environments. Each module includes a main.tf file encapsulating the module's resources, along with input.tf, output.tf, and variable.tf files for easy configuration across different environments.

Common Module

Let's assume that in the common module, we configure realm events by using the jboss-logging event listener with some non-default configurations. Below is an example of how the main.tf file may look:



resource "keycloak_realm_events" "realm_events" {
  realm_id                     = var.realm_id
  events_enabled               = true
  events_expiration            = 1800
  admin_events_enabled         = true
  admin_events_details_enabled = true
  ]

  events_listeners = [
    "jboss-logging"
  ]
}

To include the realm_id variable, it must be defined in the variables.tf file as shown below:



variable "realm_id" {
  description = "Realm ID"
  type        = string
}

And we must configure the used providers in our module. In this case, the provider.tf will look like as:



terraform {
  required_providers {
    keycloak = {
      source = "mrparkers/keycloak"
    }
  }
}

Master Realm Module

In the realm-master module, the main.tf file should reference the common module. Here is an example of how the main.tf file may look:



data "keycloak_realm" "master" {
  realm = "master"
}

module "realm-master" {
  source = "../../modules/common"
  realm_id = data.keycloak_realm.master.id
}

Similarly, the provider.tf file should be configured as follows:



terraform {
  required_providers {
    keycloak = {
      source = "mrparkers/keycloak"
    }
  }
}

Local environment

In the main.tf file, we need to define the master realm in order to reference the realm-master module in our project.



# Define master realm
module "realm-master" {
  source = "../modules/realm-master"
}

Within each environment (e.g., prod, stage and local), there's an env.yaml file containing all the environment-specific variables.

For example, the env.yaml file for the local environment may look like this:



---
environment: local
url: http://localhost:8080/keycloak

And of course, don't forget to include the terragrunt-local.hcl file, which should be defined in the parent module.



include "root" {
  path = find_in_parent_folders("terragrunt-local.hcl")
}

Terragrunt configuration

As mentioned earlier, Terragrunt is highly beneficial for keeping your configuration Don't Repeat Yourself (DRY). For example, the terragrunt.hcl file may look like this:



# Generates the backend for all modules.
remote_state {
  backend = "s3"
  config  = {
    encrypt        = true
    key            = "keycloak/${path_relative_to_include()}/terraform.tfstate"
    region         = "<AWS REGION>"
    bucket         = "terraform-states"
    dynamodb_table = "terraform-lock"
  }
}

# Read the local "env.yaml" in every environment.
locals {
  vars        = yamldecode(file("${path_relative_to_include()}/env.yaml"))
  environment = local.vars.environment
  url         = local.vars.url
}

# Generate the "provider.tf" file for every module.
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<EOF
terraform {
  required_providers {
    keycloak = {
      source  = "mrparkers/keycloak"
      version = "4.4.0"
    }

    http = {
      source  = "hashicorp/http"
      version = "3.2.1"
    }
  }
}

data "http" "config" {
  url = "<INTERNAL CONFIGURATION URL>"
}

provider "keycloak" {
  client_id     = jsondecode(data.http.config.response_body).terraform-client-id
  client_secret = jsondecode(data.http.config.response_body).terraform-client-secret
  url           = "${local.url}"
}

EOF
}

# Generate the "backend.tf" file for every module.
generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite"
  contents  = <<EOF
terraform {
  backend "s3" {}
}
EOF
}

If you're familiar with Terragrunt, you'll find the terragrunt.hcl file above quite familiar, except for the http config part, which I'll describe later in section.

In remote_state, we use AWS S3 to store the state of our environment configurations. Additionally, we generate the backend and provider for every environment.

The locals block is mainly used to read the env.yaml file and assign the variables to local variables.

Lastly, the http config is responsible for making an HTTP call to wherever you securely store your environment configurations (at least the Terraform client ID and secret) to pass them to the keycloak provider.

The terragrunt-local.hcl file is similar to the terragrunt.hcl file, except for the remote state configuration, which isn't necessary in this case. The local file is primarily for testing your configuration on a local Keycloak cluster setup.

The Docker Compose

To run our Terraform configurations, we require a Keycloak cluster setup. In below, we use Docker Compose to start a Keycloak cluster, enabling us to run our local Terraform configurations seamlessly against it.



version: '3'

volumes:
  postgres_data:
      driver: local

services:
  postgres:
      image: postgres
      volumes:
        - postgres_data:/var/lib/postgresql/data
      environment:
        POSTGRES_DB: keycloak
        POSTGRES_USER: keycloak
        POSTGRES_PASSWORD: password
      ports:
        - "5432:5432"
  keycloak:
      image: quay.io/keycloak/keycloak:23.0.6
      environment:
        KC_DB_USERNAME: keycloak
        KC_DB_PASSWORD: password
        KC_DB_URL_HOST: postgres
        KC_DB: postgres
        KC_DB_SCHEMA: public
        KC_HTTP_RELATIVE_PATH: /keycloak
        KC_HOSTNAME_ADMIN: 127.0.0.1
        KC_HOSTNAME: localhost
        KEYCLOAK_ADMIN: admin
        KEYCLOAK_ADMIN_PASSWORD: admin
      command:
        - start-dev
      ports:
        - "8080:8080"
        - "8787:8787"
      depends_on:
        - postgres
  config_keycloak:
    image: ubuntu
    volumes:
      - ./keycloak-docker-config.sh:/opt/keycloak-docker-config.sh
    command: ./opt/keycloak-docker-config.sh
    depends_on:
      - keycloak

The keycloak-docker-config.sh script is primarily used to configure a Terraform client with admin privileges, which Terraform will use during its operations.



#!/bin/bash

apt update -y && apt -y install jq curl

until $(curl --output /dev/null --silent --head --fail http://keycloak:8080/keycloak); do
    printf '.'
    sleep 5
done

# Get access token
TOKEN=$( \
  curl -X POST \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "client_id=admin-cli" \
    -d "username=admin" \
    -d "password=admin" \
    -d "grant_type=password" \
    "http://keycloak:8080/keycloak/realms/master/protocol/openid-connect/token" | jq -r '.access_token')

# Create Terraform client (terraform/terraform)
curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -d '{"clientId": "terraform", "name": "terraform", "enabled": true, "publicClient": false, "secret": "terraform", "serviceAccountsEnabled": true}' \
  "http://keycloak:8080/keycloak/admin/realms/master/clients"

# Get the Terraform service account user ID
USER_ID=$( \
  curl -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${TOKEN}" \
    "http://keycloak:8080/keycloak/admin/realms/master/users?username=service-account-terraform" | jq -r '.[0].id')

# Get the admin role ID
ROLE_ID=$( \
  curl -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${TOKEN}" \
    "http://keycloak:8080/keycloak/admin/realms/master/roles" | jq -r '.[] | select(.name == "admin") | .id')

# Add the admin role to the Terraform service account user
curl -kv -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -d '[{"id":"'"${ROLE_ID}"'", "name":"admin"}]' \
  "http://keycloak:8080/keycloak/admin/realms/master/users/$USER_ID/role-mappings/realm"

To run it, open the terminal and run the below docker compose command



docker-compose up --build

After the docker-compose containers are up and running, navigate to http://localhost:8080/keycloak and log in using admin/admin as credentials. Ensure that the Terraform client is configured within the master realm.

Now it's time to run your Terraform local configurations. Open your terminal and execute the following commands:



# Navigate to the local environment
$ cd local 
# Ensure that Terraform-related files, including the auto-generated backend.tf and provider.tf, are removed
$ rm -r backend.tf provider.tf terraform.tfstate terraform.tfstate.backup .terraform.lock.hcl .terraform 
# Initialize Terraform to create all necessary files
$ terragrunt init --terragrunt-config terragrunt-local.hcl 
# Apply the Terraform configurations
$ terragrunt apply --terragrunt-config terragrunt-local.hcl

Now, open your web browser and go to http://localhost:8080/keycloak. Log in using your admin credentials and make sure your configurations are properly set up there!

I created a demo project on GitHub keycloak-terraform-demo

Conclusion

Organizing and managing Keycloak configurations with Terraform can greatly streamline your development process. By following the structure and steps outlined in this guide, you can efficiently set up and maintain your Keycloak environments, ensuring consistency and scalability across your projects. If you have any questions or suggestions, feel free to leave them in the comments below.

I hope you find it useful!

Mastering Spring Cloud Gateway Testing: Filters (part 2)

Mohammed Ammer — Wed, 24 Apr 2024 08:50:29 +0000

In Part 1, we dived into testing predicates efficiently. Now, let's continue in the same vein but shift our focus to filters. Just like predicates, filters play a crucial role in Spring Cloud Gateway routing by making changes to the request or the target backend.

In this post, we'll delve into writing effective integration tests for filters. These tests ensure that our custom filters perform precisely as intended, validating that they are accomplishing the specific tasks they were designed for.

Before going forward, I strongly recommend you go through Part 1 to leverage main concepts about Spring Cloud Gateway routing if you doubt about it.

Let us go!

For our demonstration, we've created AddLogisticProvidersHeaderGatewayFilterFactory. This function adds a new header containing shipping company information based on the client's IP location. That's all there is to it.

@Component
class AddLogisticProvidersHeaderGatewayFilterFactory(val countryService: CountryService, val logisticService: LogisticService) :
    AbstractGatewayFilterFactory<AddLogisticProvidersHeaderGatewayFilterFactory.Config>(
        Config::class.java
    ) {

    override fun apply(config: Config): GatewayFilter {
        return GatewayFilter { exchange, chain ->
            // Get the client IP address from the request
            val clientIP = exchange.request.headers.getFirst("X-Forwarded-For")
            val country = clientIP?.let { countryService.getCountry(it) }
            val shippingCompanies = country?.let { logisticService.getProviders(it) }
            shippingCompanies?.let {
                val request = exchange.request.mutate().header(config.headerName, it.joinToString(",")).build()
                return@GatewayFilter chain.filter(exchange.mutate().request(request).build())
            }
            chain.filter(exchange)

        }
    }

    class Config(
        val headerName: String,
    )
}

Testing strategy

Filters serve as individual units of logic that can modify both the request and/or response. Using our custom filter as an example, we anticipate modifying the request by adding a new header with calculated values.

By establishing an integration test specific to this filter, wherein we send various types of requests and validate whether the filter produces the expected output by adding or omitting the new header, we can ensure that the filter is functioning as intended.

Configuration

Based on the description above, here is how the Spring Cloud Gateway configuration for our integration test may look:

spring:
  cloud:
    gateway:
      routes:
        - id: route_all_to_oauth
          uri: http://localhost:${wiremock.server.port}
          predicates:
            - Path=/**
          filters:
            - name: AddLogisticProvidersHeader
              args:
                headerName: X-Shipping-Providers

In this configuration snippet, we use predicates to match all requests (as the specifics of the request aren't relevant for this test). However, in the filters section, we configure a single filter with the necessary settings. For the AddLogisticProvidersHeader filter, we specify the header name as X-Shipping-Providers.

Integration test

Similar to the predicates integration test, I utilized the Spock framework to handle my integration test. Below is the Integration Specification for the AddLogisticProvidersHeader filter.

@ActiveProfiles("AddLogisticProvidersHeaderGatewayFilterFactoryIntegrationSpec")
class AddLogisticProvidersHeaderGatewayFilterFactoryIntegrationSpec extends AbstractIntegrationSpec {


    public static final String SHIPPING_COMPANIES_RESULT = "shippingCompaniesResult"

    @Unroll
    def "given request from #country by ip #ip, expected shipping providers header existence #expectedHeaderExistence and expected header value #expectedHeaderValue"() {
        given:

        wireMockServer.stubFor(
                any(anyUrl()).atPriority(1)
                        .withHeader("X-Shipping-Providers", matching(".*"))
                        .willReturn(aResponse().withStatus(200).with {
                            if (expectedHeaderExistence) {
                                it.withHeader(SHIPPING_COMPANIES_RESULT, expectedHeaderValue)
                            }
                            return it
                        }
                        )
        )

        wireMockServer.stubFor(
                any(anyUrl()).atPriority(2)
                        .willReturn(aResponse().withStatus(200))
        )

        when:
        def request = webTestClient.get().uri {
            it.path("/")
            it.build()
        }
        if (ip != null) {
            request.header("X-Forwarded-For", ip)
        }

        def result = request.exchange()

        then:
        if (expectedHeaderExistence) {
            result.expectHeader().valueEquals(SHIPPING_COMPANIES_RESULT, expectedHeaderValue)
        } else {
            result.expectHeader().doesNotExist(SHIPPING_COMPANIES_RESULT)
        }

        where:
        country   | ip              | expectedHeaderExistence | expectedHeaderValue
        "France"  | "103.232.172.0" | true                    | "HERMES"
        "Germany" | "77.21.147.170" | true                    | "DHL,HERMES"
        "USA"     | "30.0.0.0"      | false                   | null
    }

}

In this test, we mock any request that has the header X-Shipping-Providers to return a header containing the same shipping companies as those added to the request header.

Subsequently, we validate whether the header was added to the request by checking for the existence of the header and confirming that it contains the expected value.

To try out the example by yourself, be sure to check out the accompanying GitHub repository here.

I hope you found it useful!

Mastering Spring Cloud Gateway Testing: Predicates (part 1)

Mohammed Ammer — Tue, 23 Apr 2024 23:52:59 +0000

Spring Cloud Gateway stands as a reactive HTTP gateway designed to streamline microservices communication through its routing, filtering, and integration features.

Within Spring Cloud Gateway, two pivotal components play key roles: Predicates and Filters. Before delving into these components and the challenge of testing them, it's essential to understand the core concept of routing, which serves as the primary function of any 'gateway'."

Routing

Routing is the process of directing traffic from one source to a specific destination. In the context of software development, routing refers to the mechanism by which requests are directed to the appropriate endpoints or resources within an application or system based on predefined rules or criteria.

Predicates

Predicates are conditions that define when and how a request should be routed. They adds flexibility to customize routing rules based on various factors such as headers, paths, parameters and even body. By leveraging predicates, we can efficiently control traffic flow and implement dynamic routing strategies.

Filters

Filters are interceptors that can modify incoming and outgoing HTTP requests and responses. Filters are applied to routes and can be customized to suit specific needs, giving the flexibility to modify the request, as well as the response, to enhance the functionality and behavior of the applications without tightly coupling code to individual services.

To summarize, when request goes through a Spring Cloud Gateway, the request goes into list of routes. Each route consists of list of predicates and filters. The request much match at most one route and the evaluation is based on getting acceptance of all defined predicates attached to the route. Once the route matched, the list of route's filters are executed in the same sequence as they were defined.

Let us go!

Route logic can become quite complex. This complexity is further even more when implementing custom predicates and filters. It depends on your business.

For our demo, we have two backend endpoints:

/v1/** endpoint:
- Designed for requests coming from specific Geo locations, such as countries.
- This endpoint requires the allowed shipping providers (logistics) to be included in the request header.
/v2/** endpoint:
- Designed to accept requests from any country.
- No special headers are required for requests to this endpoint.

Based on our above description, here is the Spring Cloud Gateway configuration may look like:



spring:
  cloud:
    gateway:
      routes:
        - id: route1
          uri: http://localhost:8080/
          predicates:
            - Path=/**
            - name: Geo
              args:
                countries: [ "DE", "FR" ]
          filters:
            - name: AddLogisticProvidersHeader
              args:
                headerName: X-Shipping-Providers
            - RewritePath=/(?<segment>.*), /v1/$\{segment}
        - id: route2
          uri: http://localhost:8080/
          predicates:
            - Path=/**
          filters:
            - RewritePath=/(?<segment>.*), /v2/$\{segment}

In the configurations above, when a request originates from Germany or France, our gateway:

Retrieves the logistics companies and adds them to the request header with the key X-Shipping-Providers.
Rewrites the request path to include the root path /v1/.
Forwards the modified request to the target URI.

For requests originating from other geographical locations, the gateway:

Appends /v2/ to the request path.
Forwards the modified request to the target URI.

Testing Predicates

Let's demonstrate what the GeoRoutePredicateFactory predicate might look like.



@Component
class GeoRoutePredicateFactory(val countryService: CountryService) :
    AbstractRoutePredicateFactory<GeoRoutePredicateFactory.Config>(Config::class.java) {

    override fun apply(config: Config): Predicate<ServerWebExchange>? {
        return GatewayPredicate { exchange ->
            val countries = config.countries.map { country -> country.uppercase(Locale.getDefault()) }
            if (countries.contains("ALL")) {
                return@GatewayPredicate true
            }

            // Get the client IP address from the request
            val clientIP = exchange.request.headers.getFirst("X-Forwarded-For") ?: "127.0.0.1"
            val country = clientIP.let { countryService.getCountry(clientIP) }
            // Check if at least one delivery option is available for the country
            countries.contains(country?.uppercase(Locale.getDefault()))

        }
    }

    class Config(val countries: List<String>)
}

Put simply, if all countries are allowed, the predicate returns true. If not, the predicate checks the client's request IP from the X-Forwarded-For header and returns true only if the request originates from one of the allowed countries.

Testing strategy

Predicates act as conditions for selecting a route for a request. If we can establish that a particular route is acceptable, it means that the predicates for that route have been satisfied. If there is only one predicate and the route is selected, it means the predicate evaluated to true; otherwise, it did not. Voilà!

Base mocks and Spring boot setup

To ensure the correctness of our predicates, relying solely on unit tests is insufficient. Predicates are part of a larger set of predicates and may rely on the Spring Cloud Integration framework to function properly, such as the exchange method.

Because of this, I advocate for using integration tests for the predicates (and filters, as we'll discuss later).

I love using the Spock framework for its simplicity, readability, and maintainability. That's why we use Spock to drive our integration tests.

To facilitate this, we create an abstract integration test Specification. Here, we initialize the webTestClient to make our REST calls and use WireMock to mock our target backends. This setup ensures that our integration tests are comprehensive and reliable.



@SpringBootTest(classes = [GatwaytestApplication])
@AutoConfigureWireMock(port = 0)
@AutoConfigureMockMvc
@ActiveProfiles("test")
abstract class AbstractIntegrationSpec extends Specification {

    @Autowired
    ObjectMapper objectMapper

    @Autowired
    WebTestClient webTestClient

    @Autowired
    WireMockServer wireMockServer

    def setup() {
        WireMock.reset()
    }
}

Base test predicates setup

To verify that the request is routed through the chosen route, we can revisit two important attributes of Spring Cloud Gateway:

GATEWAY_HANDLER_MAPPER_ATTR: This attribute should have the value RoutePredicateHandlerMapping when the request is being routed through the predicates.
GATEWAY_ROUTE_ATTR: This attribute should contain the name of the route that matches the request.

If we can capture the values of the two attributes and use them to validate whether the predicate behaves as expected, then we can utilize them effectively. To do this, we require the abstract predicate integration specification provided below as a foundation for our predicate integration specifications.



@Import(Config.class)
abstract class AbstractPredicateIntegrationSpec extends AbstractIntegrationSpec {

    protected static final String HANDLER_MAPPER_HEADER = "X-Gateway-Handler-Mapper-Class"
    protected static final String ROUTE_ID_HEADER = "X-Gateway-RouteDefinition-Id"

    @TestConfiguration(proxyBeanMethods = false)
    static class Config {

        @Bean
        @Order(500)
        GlobalFilter modifyResponseFilter() {
            return (exchange, chain) -> {
                String value = exchange.getAttributeOrDefault(GATEWAY_HANDLER_MAPPER_ATTR, "N/A");
                if (!exchange.getResponse().isCommitted()) {
                    exchange.getResponse().getHeaders().add(HANDLER_MAPPER_HEADER, value);
                }
                Route route = exchange.getAttributeOrDefault(GATEWAY_ROUTE_ATTR, null);
                if (route != null) {
                    if (!exchange.getResponse().isCommitted()) {
                        exchange.getResponse().getHeaders().add(ROUTE_ID_HEADER, route.getId());
                    }
                }
                return chain.filter(exchange);
            }
        }
    }
}

In the filter described above, we intercept the response and extract the values of two attributes: GATEWAY_HANDLER_MAPPER_ATTR and GATEWAY_ROUTE_ATTR. These values are then added to the response header using the names HANDLER_MAPPER_HEADER and ROUTE_ID_HEADER respectively.

Having this information allows us to assert the route ID, ensuring that the result of the predicate under test is as expected

Test GeoRoutePredicateFactory configuration

For our integration test to work, we need to define the spring cloud gateway routes. For that, we define a testing profile by the name of the predicate application-GeoRoutePredicateFactoryIntegrationSpec.yml and have two routes. one route by id route_based_target_predicate when the predicate actually passed, and route_others if the predicate doesn't passed and the request goes to a different route.
To enable our integration test, we must define the Spring Cloud Gateway routes. To achieve this, we create a testing profile named application-GeoRoutePredicateFactoryIntegrationSpec.yml. Within this profile, we define two routes:

route_based_target_predicate: This route is activated when the predicate successfully passes.
route_others: This route is activated if the predicate fails, and the request is directed to a different route.



spring:
  cloud:
    gateway:
      routes:
        - id: route_based_target_predicate
          uri: http://localhost:${wiremock.server.port}
          predicates:
            - Path=/**
            - name: Geo
              args:
                countries: [ "DE", "FR" ]
          filters:
            - PrefixPath=/prefix
        - id: route_others
          uri: http://localhost:${wiremock.server.port}
          predicates:
            - Path=/**
          filters:
            - PrefixPath=/prefix

GeoRoutePredicateFactory specification

We complete the integration testing setup by defining the integration test class for our predicate specification.



@ActiveProfiles("GeoRoutePredicateFactoryIntegrationSpec")
class GeoRoutePredicateFactoryIntegrationSpec extends AbstractPredicateIntegrationSpec {


    @Unroll
    def "given request from #country by ip #ip, expected route #expectedRoute"() {

        given:
        wireMockServer.stubFor(
                get(ANY).willReturn(aResponse().withStatus(200))
        )

        when:
        def request = webTestClient.get()
        if (ip != null) {
            request.header("X-Forwarded-For", ip)
        }


        def result = request.exchange()

        then:
        result.expectStatus().isEqualTo(200)
                .expectHeader().valueEquals(HANDLER_MAPPER_HEADER, RoutePredicateHandlerMapping.class.getSimpleName())
                .expectHeader().valueEquals(ROUTE_ID_HEADER, expectedRoute)

        where:
        country   | ip              | expectedRoute
        "Germany" | "77.21.147.170" | "route_based_target_predicate"
        "France"  | "103.232.172.0" | "route_based_target_predicate"
        "USA"     | "30.10.0.10"    | "route_others"
        null      | null            | "route_others"
    }
}

In our test setup, we define test cases in a "where" table containing the test data. The IP address serves as the input for our predicate logic. Based on our configuration, requests originating from Germany and France should be routed through route_based_target_predicate, and the decision is made based on the IP address of the incoming request.

For example, for IP addresses "77.21.147.170" and "103.232.172.0", we expect the route to be route_based_target_predicate as we anticipate the predicate to pass. Otherwise, the route should be route_others.

To verify this behavior, we assert that the value of the ROUTE_ID_HEADER matches the expectedRoute.

To try out the example by yourself, be sure to check out the accompanying GitHub repository here.

In part 2, I'll describe in details how you can master testing for filters.

I hope you found it useful!

Optimized Keycloak SPI Deployment using Maven Shade Plugin

Mohammed Ammer — Sun, 31 Mar 2024 22:27:38 +0000

When you develop Keycloak SPI, you probably add external dependencies (e.g. libraries). These dependencies are required to be in the classpath of Keycloak so your SPI can work as you expect.

To include the external libraries in Keycloak path, as mentioned in Keycloak Configuring providers:

Using third-party dependencies

When implementing a provider you might need to use some third-party dependency that is not available from the server distribution.
In this case, you should copy any additional dependency to the providers directory and run the build command. Once you do that, the server is going to make these additional dependencies available at runtime for any provider that depends on them.

To achieve that, you will probably need to download the JARs and copy it during building the Keycloak Docker Image.

This is a bit messy, especially when you have a new release and every time want to make sure to have the same dependencies versions.

To avoid all the hassle, I used Maven Shade Plugin.

This plugin provides the capability to package the artifact in an uber-jar, including its dependencies and to shade - i.e. rename - the packages of some of the dependencies.

It is also very useful when you've multiple SPIs but use different versions of same library (through the renaming feature).

A sample build configuration can be like:

...
<build>
    <plugins>
        <plugin>
            <artifactId>maven-shade-plugin</artifactId>
            <executions>
                <execution>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <phase>package</phase>
                </execution>
            </executions>
            <groupId>org.apache.maven.plugins</groupId>
            <version>${maven-shade.version}</version>
        </plugin>
    </plugins>
</build>
...

Follow the plugin documentation to configure the plugin by including/excluding/rename the dependencies.

That is all. I hope you find it useful.

Seamless Migration to Keycloak: Token Exchange

Mohammed Ammer — Sun, 31 Mar 2024 22:27:13 +0000

In previous post, we spoke about the migration of Refresh Token, where Token Exchange is playing a big role to have a seamless migration.

Keycloak is providing token exchange just by configuration, however, in our use case, we use custom identity provider due to our lack support to OpenID Connect.

In this article, I'll discuss an extension to our custom identity provider (OAuth 2.0) to support token exchange.

Enable external exchange

In our OauthServiceIdentityProvider, we override the supportsExternalExchange to return true.

public class OauthServiceIdentityProvider extends AbstractOAuth2IdentityProvider<OauthServiceIdentityProviderConfig>
{ 
...
   @Override
   protected boolean supportsExternalExchange()
   {
      return true;
   }
...
}

Exchange implementation

To implement the exchange logic, override the method exchangeExternalImpl.

In the below example, I added some validations then called the doGetFederatedIdentity(..) to get the federation. Different than the doGetFederatedIdentity in Brokering non-OIDC OAuth 2.0 identities, I use useSessionContent to decide from where we get the clientId. In case of token exchange, we get the clientId from the token claim token.getOtherClaims().get(TOKEN_CLAIM_CLIENT_ID)

@Override
protected BrokeredIdentityContext exchangeExternalImpl(EventBuilder event, MultivaluedMap < String, String > params) {
    final String subjectToken = params.getFirst(OAuth2Constants.SUBJECT_TOKEN);
    if (subjectToken == null) {
        event.detail(Details.REASON, OAuth2Constants.SUBJECT_TOKEN + " param unset");
        event.error(Errors.INVALID_TOKEN);
        throw new ErrorResponseException(OAuthErrorException.INVALID_TOKEN, "token not set", Response.Status.BAD_REQUEST);
    }
    String subjectTokenType = params.getFirst(OAuth2Constants.SUBJECT_TOKEN_TYPE);
    if (subjectTokenType == null) {
        subjectTokenType = OAuth2Constants.REFRESH_TOKEN_TYPE;
    }
    if (!OAuth2Constants.REFRESH_TOKEN_TYPE.equals(subjectTokenType)) {
        event.detail(Details.REASON, OAuth2Constants.SUBJECT_TOKEN_TYPE + " invalid");
        event.error(Errors.INVALID_TOKEN_TYPE);
        throw new ErrorResponseException(OAuthErrorException.INVALID_TOKEN, "invalid token type", Response.Status.BAD_REQUEST);
    }

    final String requestedType = params.getFirst(OAuth2Constants.REQUESTED_TOKEN_TYPE);
    if (requestedType != null && !requestedType.equals(OAuth2Constants.REFRESH_TOKEN_TYPE)) {
        event.detail(Details.REASON, "requested_token_type unsupported");
        event.error(Errors.INVALID_REQUEST);
        throw new ErrorResponseException(OAuthErrorException.INVALID_REQUEST, "invalid requested token type", Response.Status.BAD_REQUEST);
    }

    return doGetFederatedIdentity(subjectToken, false);

}

private BrokeredIdentityContext doGetFederatedIdentity(String accessToken, boolean useSessionContent) {
    if (accessToken == null) {
        throw new IdentityBrokerException("No token from server.");
    }

    JsonWebToken token;
    try {
        JWSInput jws = new JWSInput(accessToken);
        token = jws.readJsonContent(JsonWebToken.class);
        final String username = token.getSubject();
        String clientId;
        if (useSessionContent) {
            clientId = session.getContext().getClient().getName();
        } else {
            clientId = String.valueOf(token.getOtherClaims().get(TOKEN_CLAIM_CLIENT_ID));

        }

        final String userId = String.valueOf(token.getOtherClaims().get(TOKEN_CLAIM_USER_ID));
        final String userType = String.valueOf(token.getOtherClaims().get(TOKEN_CLAIM_USER_TYPE));

        final BrokeredIdentityContext user = new BrokeredIdentityContext(userId);
        user.setUsername(username);
        user.setIdpConfig(getConfig());
        user.setIdp(this);
        user.setUserAttribute(TOKEN_CLAIM_USER_ID, userId);
        user.getContextData().put(TOKEN_CLAIM_USER_TYPE, userType);

        return user;
    } catch (JWSInputException e) {
        throw new IdentityBrokerException("Invalid token", e);
    }
}

Add user attributes and mappings claims

Same as the authenticationFinished method, for the token exchange, we need to override the exchangeExternalComplete to enrich our user session with whatever needed add the required user attributes and session notes. Session notes can then be used later to map Keycloak token claims.

@Override
public void exchangeExternalComplete(UserSessionModel userSession, BrokeredIdentityContext context, MultivaluedMap < String, String > params) {
    final String value = (String) context.getContextData().get(TOKEN_CLAIM_USER_TYPE);
    if (StringUtil.isNotBlank(value)) {
        userSession.setNote(field, value);
    }

    String userId = context.getUserAttribute(TOKEN_CLAIM_USER_ID);
    if (userId != null && !userId.isBlank()) {
        userSession.getUser().setSingleAttribute(TOKEN_CLAIM_USER_ID, userId);
    } else {
        logger.warn("userId is null or blank");
    }
    super.exchangeExternalComplete(userSession, context, params);
}

By this article, I am done with our custom identity provider implementation to fully support OAuth 2.0 including token exchange.

Are you using Terraform to configure your Keycloak cluster? well, it is not easy to make it with Terraform when speaking about custom identity provider. In Custom Identity Providers in Keycloak with Terraform, I explained it in details to keep your platform Terraformed ;-)

I hope you find it useful.

Seamless Migration to Keycloak: Refresh token

Mohammed Ammer — Sun, 31 Mar 2024 22:27:08 +0000

In the previous post, we discussed the seamless migration of Authorization Code flow from Spring Authorization Server to Keycloak, also the JWKs merging to allow the resource servers validating the Keycloak tokens.

Now we go in further in the migration of the refresh token grant type, which "in most cases" has the most share in any identity server traffic.

Refresh Token

I gave an example of having a feature flag per client, where you control the client requests routing.

The same with refresh token requests. If the client is eligible for Keycloak, the next refresh token request must return an access token and refresh token from Keycloak (not the old identity server anymore).

Again, the user shouldn't be forced to re-login to be identified by Keycloak. This is exactly what is this article for.

Here is a sample refresh token request:

curl 'https://example.com/oauth/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Basic xxxxxxxxxxxxxxxxxx' \
--data-urlencode 'grant_type=refresh_token' \
--data-urlencode 'refresh_token=<REFRESH TOKEN>'

The routing flow

First, we define the pre-conditions for the refresh token routing flow to Keycloak.

The request is a token request (e.g. /oauth/token)
Keycloak is enabled for the client
The refresh token issued by the old identity provider

Once all pre-conditions met, the request should be routed to Keycloak.

Below diagram is to get an abstract overview about the flow:

Below sequence diagram is to describe in more details the flow:

Let us move on to discuss the configuration required in the gateway.

Gateway Configuration

The gateway configuration should look like below:

spring:
  cloud:
    gateway:
      routes:
        - id: refresh_token_target_kc
          uri: https://example.com
          predicates:
            - Path=/oauth/token**
            - name: KeycloakEnabled
              args:
                enabled: true
            - name: GrantType
              args:
                hints:
                  grant_type: [ "refresh_token" ]
            - name: JwtClaim
              args:
                hints:
                  claims:
                    iss: "old-identity-provider"
          filters:
            - oldIdpIntrospect
            - KcTokenExchangeModifyRequestBody
            - KcTokenChangeRequestUri

Path predicate

The path predicate is very straight forward. It is a built-in predicate to decide about the request path. We use it to decide if the request is a token request. If request path matches /oauth/token**, we move to the following predicate.

KeycloakEnabled predicate

Keycloak Enabled predicate is to check if target client is Keycloak enabled. Here we use our feature flags to check if Keycloak, realm and target client are enabled.

GrantType predicate

Grant Type predicate is to check if the token request grant type is refresh token.

JwtClaim predicate

Jwt Claim predicate is to check the JWT claims for the token sent by the client. The predicate here is to validate on the issuer. The iss should have the name of the old identity provider.

If all conditions are met, then the response token and refresh token must be issued by Keycloak. For that, the three filters oldIdpIntrospect, KcTokenExchangeModifyRequestBody and KcTokenChangeRequestUri are there.

Gateway Filters

oldIdpIntrospect Filter

The refresh token sent by the client must be valid and accepted by the old identity provider for the Keycloak to exchange it. To make that happen, I'll use OAuth 2.0 Token Introspection from the old identity provider to validate the refresh token, if the refresh token is active, then I call Keycloak to exchange the token.

You may need to implement the introspect endpoint in the old identity provider (for instance Spring Security OAuth) if not exist. Here is an example of introspect request:

curl --location --globoff 'https://example.com/oauth/introspect' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Basic <ENCODED_CREDENTIALS>' \
--data-urlencode 'token=<TOKEN>'

Token Exchange Filters

Although the client is known by Keycloak, the refresh token is unknown. Hence, if we route the traffic to Keycloak, the request will be rejected by default.

If you are into OAuth 2.0, the first thing may come to your mind is the OAuth 2.0 Token Exchange, where Keycloak exchanges the old identity provider token by an access token and refresh token issued by Keycloak.

Here is an example of a token exchange request:

curl --location --globoff 'https://example.com/realms/myrealm/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'client_id=<CLIENT_ID>' \
--data-urlencode 'client_secret=<CLIENT_SECRET>' \
--data-urlencode 'subject_token=<REFRESH_TOKEN>' \
--data-urlencode 'subject_issuer=<OLD_IDP_ISSUER>' \
--data-urlencode 'grant_type=urn:ietf:params:oauth:grant-type:token-exchange' \
--data-urlencode 'subject_token_type=urn:ietf:params:oauth:token-type:refresh_token' \
--data-urlencode 'requested_token_type=urn:ietf:params:oauth:token-type:refresh_token' \
--data-urlencode 'scope=<SCOPE>'

We created two filters for the Token Exchange to happen

KcTokenExchangeModifyRequestBody: To rewrite the request body to meet the token exchange request body.
KcTokenChangeRequestUri: To rewrite the URI to meet Keycloak token exchange URI.

Notes:

Keycloak has to be configured to have token exchange enabled for your client.
If you have custom identity provider SPI implemented as in Keycloak: Brokering non-OIDC OAuth 2.0 identities, you must override the related token exchange methods for the token exchange to work probably, especially if you will need to map claims from the old tokens to the new one. I have a full detailed article to guide you on how can you make it. See Seamless Migration to Keycloak: Token Exchange

I hope you find it useful.

Seamless Migration to Keycloak: Authorization Code Flow

Mohammed Ammer — Sun, 31 Mar 2024 22:27:04 +0000

ِAs mentioned earlier in my Insights into Transitioning from Spring Authorization Server, migrating from one identity provider to another is not an easy task, especially when you have wide range of clients.

Hard to set due dates for the clients' migration: For the client to comply to Keycloak schema, you might wait for so long (really long) until that happen.
Support old clients: The old versions of the apps will keep using the old identity provider hence you will need to keep maintaining the old identity provider as long as the older versions of the apps are supported.
Rollback: Rollback to use the old identity provider during the migration is not easy if the clients are to manage that. Especially in the early release, I prefer to have control to route the clients to either the old identity provider or Keycloak. In case of any failures (and until the issue get fixed), we have the flexibility to switch the user back to the old identity provider without any degradation to the user experience.
Avoid risk SLOs degradation for a non-functional change: Although migrating identity providers is a strategic decision and important to the organization when decided, I wouldn't accept customer experience degradation such move. Easy to switch back and forth helps to avoid that.

All above are enough reasons to aim for a seamless migration and for that to happen we need a Gateway.

Identity Provider Gateway

The idea here is to have a gateway service to intercept all traffic to the old identity provider, and based on configuration, the gateway routes the request to the proper identity provider. Remember, we talk about seamless migration where Keycloak brokering fits here. We talked about a custom identity provider to allow Keycloak to broker OAuth2.0 without OpenID Connect support. Check it out for more details.

In case of old identity provider, the gateway will just bypass the request, however if Keycloak is enabled for the target client and grant type, the gateway will either rewrite the URL, payload or reply with redirection to comply with Keycloak schemas.

Before we go in details, what about the resource servers?

Resource servers

Well, It depends on how resource servers deal with the bearer tokens and the identity provider. Resource servers may already validating the token early in level of gateway or the validation left for the resource server (e.g. backend service).
You may use the identity provider to validate every JWT or you use stateless JWTs where resource servers are validating on their own, independent on the identity provider.

Validate Keycloak JWTs

I'll assume here a stateless JWTs. In this case, the resource servers validate the token signature (using JSON Web Key Set (JWKS)) and expiration date.

For the resource servers to get the JWKS for Keycloak, the gateway intercept the requests to JWKS and combine the JWKS from Keycloak with the old identity provider one(s). Here, the resource servers have the JWKS for both identity providers and can validate the token signature regardless of the issuer.

Authorization Code flow

Although Authorization Code flow is complex compared to other OAuth2.0 flows but not really when it comes to migration.

The authorization code flow starts with the authorization request, where the client sends a request similar to:



https://authorization-server.com/oauth/authorize
?client_id=a17c21ed
&response_type=code
&state=5ca75bd30
&redirect_uri=https%3A%2F%2Fexample-app.com%2Fauth
&scope=photos

The request will go through group of predicates and filter(s) in our gateway service to decide about the request.

Prepare your Keycloak client

To make Keycloak ready to serve the clients, make sure that:

Keycloak has the same clients as the existing authorization server and secrets.
Give the clients same grant types as the existing authorization server.
Create same scopes
Map remote token claims to Keycloak token

Gateway Service

I use Spring Cloud Gateway to act as my identity gateway. To dynamically control the enabled realms or/and clients for Keycloak, below is the configuration you might need:



keycloak:
  enabled: true
  authorizeUrl: "https://auth.company.com/keycloak/realms/%s/protocol/openid-connect/auth"
  tokenUrl: "https://auth.company.com/keycloak/realms/%s/protocol/openid-connect/token"
  realms:
    - name: myrealm
      enabled: true
      clients:
        - name: app1
          enabled: false
        - name: app2
          enabled: true

where keycloak.enabled is a master flag to completely control the traffic to Keycloak. realms goes into the same direction.

We still need a configuration properties class to access it, here is it:



@ConfigurationProperties(prefix = "keycloak")
data class KeycloakProperties(
    val enabled: Boolean = false,
    val realms: List<Realm> = emptyList(),
    val tokenUrl: String = "",
    val authorizeUrl: String = ""
)

data class Realm(
    val name: String,
    val enabled: Boolean = false,
    val clients: List<Client> = emptyList()
)

data class Client(
    val name: String,
    val enabled: Boolean = false
)

Gateway Routes

Here is the gateway routes configuration:



spring:
  cloud:
    gateway:
      routes:
        - id: kc_authorize
          uri: https://auth.company.com
          predicates:
            - Path=/gatewayservice/oauth/authorize**
            - name: KeycloakEnabled
              args:
                enabled: true
            - CookieNegate=OAUTHSESSION,.*
            - QueryNegate=scope,kc_idp
          filters:
            - AuthorizeRedirectToKc
        - id: old_idp
          uri: https://auth.company.com
          predicates:
            - Path=/gatewayservice/oauth/**
          filters:
            - RewritePath=/(?<segment>.*), /oldidp/$\{segment}

Let us speak about every predicate and filter.

Path Predicate

The path predicate is very straight forward. It is a built-in predicate to decide about the request path. We use it to decide if the request is an authorization request. If request path matches /gatewayservice/oauth/authorize**, we move to the following predicate.

KeycloakEnabled Predicate

This predicate is to check if target client is Keycloak enabled. Here we use our feature flags to check if Keycloak, realm and target client are enabled.

CookieNegate Predicate

Due to the nature of the authorization code flow where authorization request shows twice. The 1st time when the client first trigger the flow and the 2nd time when the client gets the code from the identity server.
The cookie negate predicate is to make sure that the authorize request is the 1st one.



class CookieNegateRoutePredicateFactory : CookieRoutePredicateFactory() {
    override fun apply(config: Config): Predicate<ServerWebExchange> {
        return super.apply(config).negate()
    }
}

QueryNegate Predicate

Same as CookieNegate predicate but for query parameters. kc_idp is our defined scope for the Keycloak client in the brokered authorization server. If the request parameters has kc_idp scope, the request is brokered already and should be skipped.



class QueryNegateRoutePredicateFactory : QueryRoutePredicateFactory() {

    override fun apply(config: Config): Predicate<ServerWebExchange> {
        return super.apply(config).negate()
    }
}

AuthorizeRedirectToKc filter

Now, all predicates passed and the authorization request should be targeting Keycloak. The AuthorizeRedirectToKc is to map the request to the equivalent Keycloak URL in response Location header with HTTP status code 301.



class AuthorizeRedirectToKcGatewayFilterFactory(
    val keycloakService: KeycloakService
) : RedirectToGatewayFilterFactory() {

    override fun apply(config: Config): GatewayFilter {
        // Set the http status by FOUND and uri empty as it will be calculated and not possible to make it fixed through configuration
        config.apply {
            config.status = HttpStatus.FOUND.value().toString()
            config.url = ""
            config.isIncludeRequestParams = true
        }

        return super.apply(config)
    }

    override fun apply(httpStatus: HttpStatusHolder, uri: URI, isIncludedRequestParams: Boolean): GatewayFilter {
        return GatewayFilter { exchange, chain ->
            if (!exchange.response.isCommitted) {
                val kcAuthorizeUri = getKcAuthorizeUri(exchange)
                if (kcAuthorizeUri == null) {
                    logger.error { "Can't construct the keycloak authorize request from: <${exchange.request.uri}" }
                    ServerWebExchangeUtils.setResponseStatus(exchange, HttpStatusHolder(HttpStatus.BAD_REQUEST, null))
                    return@GatewayFilter exchange.response.setComplete()
                }
                return@GatewayFilter super.apply(httpStatus, kcAuthorizeUri, isIncludedRequestParams).filter(exchange, chain)
            }
            return@GatewayFilter Mono.empty<Void>()
        }
    }

    fun getKcAuthorizeUri(exchange: ServerWebExchange): URI? {
        val clientId = exchange.request.queryParams[CLIENT_ID]?.first() ?: return null
        return keycloakService.authorizeUrl(clientId)
    }

    companion object {
        const val CLIENT_ID = "client_id"
    }

}

In this post, we described in details how is the authorization code flow can be routed to Keycloak without any need from the client to change their implementation. In Seamless Migration to Keycloak: Refresh token, I describe how could we make it for the refresh token requests.

I hope you find it useful.

Securing Keycloak: Configuring Admin Access within Your Private Network

Mohammed Ammer — Sun, 31 Mar 2024 22:26:55 +0000

When it comes to administrative capabilities, Keycloak boasts a wealth of features that empower users to efficiently manage their system. Alongside a user-friendly web admin tool, Keycloak offers a robust REST API, enabling seamless programmatic control.

In this article, I'll discuss on how to prevent the public access to Keycloak admin.

For this, you need to decide about the public and private host for Keycloak. For instance, ingress will look like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak-ingress
  namespace: your-namespace
spec:
  rules:
  - host: internal.example.com
    http:
      paths:
      - path: /keycloak
        pathType: Prefix
        backend:
          service:
            name: keycloak
            port:
              number: 8080
  - host: external.example.com
    http:
      paths:
      - path: /keycloak
        pathType: Prefix
        backend:
          service:
            name: keycloak
            port:
              number: 8080

Then, in the deployment.yaml file, add environment variables as below:

      ...
      containers:
        - name: keycloak
          image: quay.io/keycloak/keycloak:24.0.2 # keycloak official docker image or your customised one
          env:
            - name: KC_HOSTNAME
              value: external.example.com
            - name: KC_HOSTNAME_ADMIN
              value: internal.example.com

Now, after you deploy Keycloak. Navigating https://external.example.com/keycloak/admin/ will redirect you automatically to https://internal.example.com/keycloak/admin/

You can still use web-proxy to control access to Keycloak if you've such requirements. I prefer to have to have a context path for Keycloak to facilitate that work. To configure it, you need to add below environment variables in deployment.yaml:

  KC_HOSTNAME_PATH: keycloak
  KC_HTTP_RELATIVE_PATH: /keycloak

That is all! I hope you find it useful.

Optimized Keycloak build

Mohammed Ammer — Sun, 31 Mar 2024 22:26:50 +0000

Keycloak provides a documentation on how is to configure Keycloak the proper way in Kubernetes. I wouldn't repeat the documentation so please make sure you go through it.

The most important part when said:

We recommend optimizing Keycloak to provide faster startup and better memory consumption before deploying Keycloak in a production environment. This section describes how to apply Keycloak optimizations for the best performance and runtime behavior - Optimize the startup

For that to happen, let us write our Dockerfile

The Dockerfile

Let us have a look to the entire Dockerfile before we go into each Dockefile layer.

ARG KC_VERSION

FROM quay.io/keycloak/keycloak:${KC_VERSION} AS builder

ARG KC_HEALTH_ENABLED=true
ARG KC_METRICS_ENABLED=true
ARG KC_DB=postgres
ARG KC_FEATURES
ARG KC_METRICS_SPI_VERSION
ARG KC_HOSTNAME_PATH=keycloak
ARG KC_HTTP_RELATIVE_PATH=/keycloak

ENV KC_VERSION=${KC_VERSION}
ENV KC_HEALTH_ENABLED=${KC_HEALTH_ENABLED}
ENV KC_DB=${KC_DB}
ENV KC_HOSTNAME_PATH=${KC_HOSTNAME_PATH}
ENV KC_HTTP_RELATIVE_PATH=${KC_HTTP_RELATIVE_PATH}
ENV KC_FEATURES=${KC_FEATURES}

ADD --chown=keycloak:keycloak https://github.com/aerogear/keycloak-metrics-spi/releases/download/${KC_METRICS_SPI_VERSION}/keycloak-metrics-spi-${KC_METRICS_SPI_VERSION}.jar /opt/keycloak/providers/keycloak-metrics-spi-${KC_METRICS_SPI_VERSION}.jar

RUN /opt/keycloak/bin/kc.sh build --cache-stack=kubernetes

# Installing additional RPM packages https://www.keycloak.org/server/containers
FROM registry.access.redhat.com/ubi9:9.3 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install -y curl-7.76.1 --installroot /mnt/rootfs  --releasever 9 --setopt install_weak_deps=false --nodocs && \
    dnf --installroot /mnt/rootfs clean all && \
    rpm --root /mnt/rootfs -e --nodeps setup

FROM quay.io/keycloak/keycloak:${KC_VERSION}

COPY --from=builder /opt/keycloak/ /opt/keycloak/
COPY --from=ubi-micro-build /mnt/rootfs /
COPY ./entrypoint.sh /deployments/entrypoint.sh

ENTRYPOINT ["/deployments/entrypoint.sh"]

Build Keycloak

For maintainability, parameterize the Keycloak version so it is easier from the build tool (CI/CD) to upgrade Keycloak without touching the Dockerfile. Below is the first stage in our multi-stage Dockerfile defined as builder

ARG KC_VERSION

FROM quay.io/keycloak/keycloak:${KC_VERSION} AS builder

Arguments

Here is the Dockerfile arguments.

ARG KC_HEALTH_ENABLED=true
ARG KC_METRICS_ENABLED=true
ARG KC_DB=postgres
ARG KC_FEATURES
ARG KC_METRICS_SPI_VERSION
ARG KC_HOSTNAME_PATH=keycloak
ARG KC_HTTP_RELATIVE_PATH=/keycloak

KC_HEALTH_ENABLED and KC_METRICS_ENABLED: Both are important for monitoring Keycloak. I enabled them by default.
KC_DB: For the target database provider. I consider postgres database.
KC_FEATURES: To control the enabling features in Keycloak from CI/CD.
KC_METRICS_SPI_VERSION: If KC_METRICS_ENABLED is true, the target version of the Keycloak Metrics SPI. Keycloak Metrics SPI is not an official metrics from Keycloak but a well known SPI. It worth to check and decide about it.
KC_HOSTNAME_PATH and KC_HTTP_RELATIVE_PATH are required to be added by the same name if you choose different context path for Keycloak than the default /. In my case, I choose /keycloak as context path.

Prepare the environment variables

ENV KC_VERSION=${KC_VERSION}
ENV KC_HEALTH_ENABLED=${KC_HEALTH_ENABLED}
ENV KC_DB=${KC_DB}
ENV KC_HOSTNAME_PATH=${KC_HOSTNAME_PATH}
ENV KC_HTTP_RELATIVE_PATH=${KC_HTTP_RELATIVE_PATH}
ENV KC_FEATURES=${KC_FEATURES}

Keycloak uses the environment variables during the build to read from and act based on it. In the above snippet, the required environment variables for our build added.

Prepare the SPIs

ADD --chown=keycloak:keycloak https://github.com/aerogear/keycloak-metrics-spi/releases/download/${KC_METRICS_SPI_VERSION}/keycloak-metrics-spi-${KC_METRICS_SPI_VERSION}.jar /opt/keycloak/providers/keycloak-metrics-spi-${KC_METRICS_SPI_VERSION}.jar

A example build step that downloads a JAR file from a URL and adds it to the providers directory

Build Keycloak

RUN /opt/keycloak/bin/kc.sh build --cache-stack=kubernetes

This is the most important step where we build Keycloak based on the environment variables we provided in the build step and cache stack as kubernetes.

Install additional RPM packages

# Installing additional RPM packages https://www.keycloak.org/server/containers
FROM registry.access.redhat.com/ubi9:9.3 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install -y curl-7.76.1 --installroot /mnt/rootfs  --releasever 9 --setopt install_weak_deps=false --nodocs && \
    dnf --installroot /mnt/rootfs clean all && \
    rpm --root /mnt/rootfs -e --nodeps setup

This step is not necessary, although if you need to have additional RPM packages in the container, you can install it as shown in the above example for installing curl.

Final image.

FROM quay.io/keycloak/keycloak:${KC_VERSION}

COPY --from=builder /opt/keycloak/ /opt/keycloak/
COPY --from=ubi-micro-build /mnt/rootfs /
COPY ./entrypoint.sh /deployments/entrypoint.sh

The final stage is our image where Keycloak build files are pre-installed and ready to start up.

From the builder stage, we copy the Keycloak build
From the ubi-micro-build stage, we copy the installed packages
entrypoint.sh is our custom entrypoint file to start Keycloak. In its basic form can be as simple as below:

exec /opt/keycloak/bin/kc.sh start --optimized

I prefer to have the entrypoint in separate file so later I have the flexibility to customize it without modifying the Dockerfile. Indeed I use additional scripts in the entrypoint. For instance, integrate Datadog with Keycloak. I can write a separate article to describe it ;)

Keycloak is an identity provider which provides the front-end for the users same as admins. Someone would think to narrow the exposed part of Keycloak to only what necessary public and keep everything else to be accessed only through VPN / Internal network.
In the next blog post, I describe how you can separate Keycloak public and private APIs/URLs for an enhanced security.

I hope you find it useful.

Seamless Migration to Keycloak: Insights into Transitioning from Spring Authorization Server

Mohammed Ammer — Sun, 31 Mar 2024 22:26:44 +0000

Transitioning between authorization solutions is often a strategic move. Here we dive into the journey from Spring Authorization Server to Keycloak – two prominent players in the realm of identity and access management (IAM)

In 2020, Spring decided to deprecate "Spring Security OAuth" and announced an End-of-Life as end of May 2022. Back then, we started to think about an alternative to Spring. Although Spring announced back the return of Spring Security OAuth in Spring Authorization Server, the earlier is poor in features compared to Keycloak and its strong community. There could be many reasons driven by evolving security needs, scalability requirements, or for enhanced user experience to device for Keycloak.

Starting on the journey of transitioning from Spring Authorization Server to Keycloak is quite the undertaking, isn't it? 👀
But fear not, because I'm here to guide you through it! Recognizing the complexity of this move, I've decided to break it down into a series of articles, tackling each challenge along the way to ensure a smooth transition 🤝.

You might be wondering, why go through all this trouble if both are based on Auth2.0 standard? Well, despite the popularity of both Spring Authorization Server and Keycloak, detailed migration guides are surprisingly scarce. Each has different URLs, slightly difference in payload and the most important is the missing OpenID Connect (OIDC) in Spring Security OAuth which makes it harder to have the easy move you expect (e.g. using the default OIDC broker in Keycloak). That's where I come in – to provide you with a roadmap for success 👨‍🎓

Before we dive into the nitty-gritty details, let's make sure we're all on the same page. Understanding the basics of Identity and Access Management (IAM) is key. If terms like OAuth 2.0, authorization code flow, refresh token flow, and OpenID Connect (OIDC) sound like a foreign language to you, Just Google it. I am sure you'll find many resources for that. Building a solid understanding of these fundamentals will set the stage for a smoother migration journey. So, are you ready to get started? Let's do this! 🚀

Getting started
When it comes to tackling big tasks like this, I'm a big advocate for starting small and taking things one step at a time. This approach allows for learning from mistakes and making necessary adjustments along the way – it's all about that iterative process!

So, where do we begin?
Well, let us assume the current authorization server as simple as below:

The authorization server depends on two services, one for the user profile and another that stores the user credentials.

It's crucial to maintain a seamless user experience during migration. Users shouldn't need to re-login for new tokens from Keycloak or transfer data between data sources. This applies to resource servers too.

To ensure this, we need a traffic-controlling gateway and careful management of client migration. Instead of big moves, migrating client by client minimizes the risk of full outages in case of failures.

Based on that, Here is -in abstract- the new design:

In below articles, I'll cover some topics to achieve a smooth migration:

Brokering non-OIDC OAuth 2.0 identities to solve absence of OIDC in Spring Security OAuth.
Authorization Code flow migration and retain OAuth Clients and Resource Servers Untouched
Refresh token migration and the same, OAuth Clients untouched.
The migration of refresh token grant type can't be completed without utilizing the Token Exchange in our OAuth 2.0 custom identity provider described in the above article.

Beside that, there are some selected topics where I found it very useful to share.

Optimized Keycloak build: Creating a suitable Docker file for Keycloak based on your setup
Ensure Keycloak stays within the internal network while exposing only its APIs publicly
Use maven shade plugin to include SPIs third-party libraries in the same SPI JAR
Use Terraform to configure a custom identity provider
For Datadog users, A guide to integrate Keycloak with Datadog

I hope it will help you!