DEV Community

Cover image for Sticky sessions with Apache APISIX - the demo
Nicolas Frankel for Apache APISIX

Posted on • Originally published at

Sticky sessions with Apache APISIX - the demo

Last week, we described the concept behind sticky sessions: you forward a request to the same upstream because there's context data associated with the session on that node. However, if necessary, you should replicate the data to other upstreams because this one might go down. In this post, we are going to illustrate it with a demo.

The overall design

Design options are limitless. I'll keep myself to a familiar stack, the JVM. Also, as mentioned in the previous post, one should only implement sticky sessions with session replication. The final design consists of two components: an Apache APISIX instance with sticky sessions configured and two JVM nodes running the same application with session replication.

The application uses the following:

Dependency Description
Spring Boot Eases the usage of Spring libraries
Spring MVC Allows offering HTTP endpoints
Thymeleaf View technology
Spring Session Offers an abstraction over session replication
Hazelcast (embedded) Implements session replication
Spring Security Binds an identity to a user session

The design looks like the following:


app1 and app2 are two instances of the same app; I didn't want to overcrowd the diagram with redundant data.

The heart of the application

The heart of the application is a session-scoped bean that wraps a counter, which can only be incremented:

public class Counter implements Serializable {              //1

    private int value = 0;

    public int getValue() {
        return value;

    public void incrementValue() {
Enter fullscreen mode Exit fullscreen mode
  1. Necessary for Hazelcast serialization to work

We can use this bean in the controller:

public class IndexController {

    private final Counter counter;

    public IndexController(Counter counter) {               //1
        this.counter = counter;

    public String index(Model model) {                      //2
        model.addAttribute("counter", counter.getValue());
        return "index";
Enter fullscreen mode Exit fullscreen mode
  1. Inject the session-scoped bean in the singleton controller thanks to Spring's magic
  2. When we send a GET request to the root, increment the counter value and pass it to the model

Finally, we display the bean's value on the Thymeleaf page:

<!DOCTYPE html>
<html xmlns:th="" lang="en">
<div th:text="${counter}">3</div>
Enter fullscreen mode Exit fullscreen mode

Configuring Spring Session with Hazelcast

Spring Session offers a filter that wraps the original HttpServletRequest to override the getSession() method. This method returns a specific Session implementation backed by the implementation configured with Spring Session, in our case, Hazelcast.

We need only a few tweaks to configure Spring Session with Hazelcast.

First, annotate the Spring Boot application class with the relevant annotation:

public class SessionApplication {
Enter fullscreen mode Exit fullscreen mode

Hazelcast requires a specific configuration as well. We can use XML, YAML, or code. Since it's a demo, I can choose whatever I want, so let's code it. Spring Boot requires either an Hazelcast object or a configuration object. The latter is enough:

public Config hazelcastConfig() {
  var config = new Config();
  var networkConfig = config.getNetworkConfig();
  networkConfig.setPort(0);                                                      //1
  networkConfig.getJoin().getAutoDetectionConfig().setEnabled(true);              //2
  var attributeConfig = new AttributeConfig()                                     //3
  config.getMapConfig(HazelcastIndexedSessionRepository.DEFAULT_SESSION_MAP_NAME) //3
          .addIndexConfig(new IndexConfig(
  var serializerConfig = new SerializerConfig();
  serializerConfig.setImplementation(new HazelcastSessionSerializer())            //3
  return config;
Enter fullscreen mode Exit fullscreen mode
  1. Choose a random port to avoid port conflict
  2. Allow Hazelcast to search for other instances and automagically form a cluster. It's going to be necessary when deployed as per our design
  3. Copy-pasted from the Spring Session documentation

Configuring Spring Security

Most Spring Session examples somehow use Spring Security, and though it's not strictly necessary, it makes the design easier. I want to explain why first.

One can think about sessions as a gigantic hash table. In regular applications, the key is the JSESSIONID cookie value, the value, another hash table of session data. However, the JSESSIONID is specific to the node. The app will give a different JSESSIONID if one uses another node. Since the key is different, there's no way to access the session data, even if the session data is shared across nodes. To prevent this loss, we need to come up with a different key. Spring Security allows using a principal (or the login name) as the session data key.

Here's how I set up a basic Spring Security configuration:

public SecurityFilterChain securityFilterChain(UserDetailsService service, HttpSecurity http) throws Exception {
  return http.userDetailsService(service)                                    //1
             .authorizeHttpRequests(authorize -> authorize.requestMatchers(
                 PathRequest.toStaticResources().atCommonLocations())        //2
                            .permitAll()                                     //2
                            .anyRequest().authenticated()                    //3
             ).formLogin(form -> form.permitAll()
                                     .defaultSuccessUrl("/")                 //4
Enter fullscreen mode Exit fullscreen mode
  1. The default in-memory user details service doesn't allow custom user details classes. I had to provide my own.
  2. Allow everybody to access static resources at "common" locations
  3. All other requests must be authenticated
  4. Allow everybody to access the authentication form
  5. Redirect to the root if successful, which maps the above controller

Putting our design to the test

Beside the counter, I want to display two additional pieces of data: the hostname and the logged-in user.

For the hostname, I add the following method to the controller:

private String hostname() throws UnknownHostException {
    return InetAddress.getLocalHost().getHostName();
Enter fullscreen mode Exit fullscreen mode

Displaying the logged-in user requires an additional dependency:

Enter fullscreen mode Exit fullscreen mode

On the page, it's straightforward:

<!DOCTYPE html>
<html xmlns:th=""
      xmlns:sec="">  <!--1-->
<td sec:authentication="principal.label">Me</td>                    <!--2-->
Enter fullscreen mode Exit fullscreen mode
  1. Add the sec namespace. It's not necessary but may help the IDE to help you
  2. Require the underlying UserDetail implementation to have a getLabel() method

Last but not least, we need to configure Apache APISIX with sticky sessions, as we saw last week:

  - uri: /*
        "webapp1:8080": 1
        "webapp2:8080": 1
      type: chash
      hash_on: cookie
      key: cookie_JSESSIONID
Enter fullscreen mode Exit fullscreen mode

Here's the design implemented on Docker Compose:

    image: apache/apisix:3.3.0-debian
      - ./apisix/config.yml:/usr/local/apisix/conf/config.yaml:ro
      - ./apisix/apisix.yml:/usr/local/apisix/conf/apisix.yaml:ro    #1
      - "9080:9080"                                                  #2
      - webapp1
      - webapp2
    build: ./webapp
    hostname: webapp1                                                #3
    build: ./webapp
    hostname: webapp2                                                #3
Enter fullscreen mode Exit fullscreen mode
  1. Use the previous configuration file
  2. Only expose the API Gateway to the outside world
  3. Set the hostname to display it on the page

We can log in using one of the two hard-coded accounts. I'm using john, with password john and label John Doe. Notice that Apache APISIX sets me on a node and keeps using the same if I refresh.

Screenshot from our app

We can try to log in with the other account (jane/jane) from a private window and check the counter starts from 0.

Now comes the fun part. Let's stop the node which should be hosting the session data, here webapp2 and refresh the page:

docker compose stop webapp2
Enter fullscreen mode Exit fullscreen mode

We can see exciting things in the logs. Apache APISIX can no longer find the webapp2, so it forwards the request to the other upstream that it knows, webapp1.

  1. The request is still authenticated; it goes through Spring Security
  2. The framework gets the principal out of the request
  3. It queries Spring Session
  4. And gets the correct counter value that Hazelcast replicated from the other node

The only side-effect is an increased latency because of Apache APISIX timeout. It's 5 seconds by default, but you can configure it to a lower value if needed.

When we start webapp2 again, everything works as expected again.


In this post, I described a possible setup for sticky sessions with Apache APISIX and replication involving the Spring ecosystem and Hazelcast. Many other options are available depending on your stack and framework of choices.

The complete source code for this post can be found on GitHub:

To go further:

Originally published at A Java Geek on July 2nd, 2023

Top comments (0)