DEV Community: Tomasz Kisiel

Securing CouchDB with Keycloak Behind Nginx Reverse Proxy – Part 1

Tomasz Kisiel — Thu, 20 Jun 2024 18:23:44 +0000

In the coming weeks, I will face the task of enriching the current CouchDB deployment in one of the projects using it for metadata storage with features like SSO integration and fine-grained access management. As the SSO service of the project uses Keycloak under the hood and I am relatively new to both Keycloak and CouchDB, I decided to make some proof of concept beforehand and share the results within this blog post series.

With the experiments done here, I aim to achieve three goals. First, securing access to the CouchDB instance using JWT authentication handler and Nginx as a reverse proxy. Second, providing a CLI utility that allows authenticating seamlessly using the OAuth2 authorization code flow with PKCE. And third, implementing the required solutions to maintain the authentication and authorization process for applications created and deployed with CouchApps.

Each of these goals will receive a dedicated blog post to address the given requirements and to create a proof of concept that can be further extended for production deployment.

Introduction

To simulate the production environment where the solutions should be implemented at the end of the process, I decided to use Docker and Bitnami containers as they are quick and easy to set up. The overall architecture will be composed of the following services:

Keycloak — an open-source identity and access management solution that provides user management and fine-grained authorization features.
Keycloak Config CLI — a utility to ensure the desired configuration state for a realm based on a JSON/YAML file.
PostgreSQL — a powerful, open-source object-relational database system that will be used as Keycloak’s data storage.
CouchDB — a document-oriented, open-source database, access to which we will secure using the OpenID Connect protocol offered by Keycloak.
Nginx — a web server that can also be used as a reverse proxy and load balancer. Technically, we will use the OpenResty distribution, which comes with a Lua just-in-time compiler, but I will often refer to it as Nginx either way.

The following diagram presents a visualization of the interactions between the parties involved in the whole process.

Setting Up the Environment

Skipping further discussion, let’s dive straight into the implementation of the docker-compose file. Within this paragraph, we will implement and explain each service one by one.

Starting with Keycloak, we can write the following YAML:



version: "3.9"

services:
  keycloak:
    image: "bitnami/keycloak:24.0.3"
    environment:
      KEYCLOAK_HTTP_PORT: "8080"
      KEYCLOAK_CREATE_ADMIN_USER: "true"
      KEYCLOAK_ADMIN: "admin"
      KEYCLOAK_PROXY: "edge"
      KEYCLOAK_ADMIN_PASSWORD: "admin"
      KEYCLOAK_DATABASE_HOST: "postgres"
      KEYCLOAK_DATABASE_USER: "postgres"
      KEYCLOAK_DATABASE_PASSWORD: "postgres"
      KEYCLOAK_DATABASE_NAME: "postgres"
      KEYCLOAK_DATABASE_PORT: "5432"
    depends_on:
      - "postgres"
    networks:
      - "cks-network"

networks:
  cks-network:
    driver: "bridge"

As mentioned previously, it is based on the one of Bitnami’s containers. Details about environment variables available to be set can be found on Bitnami’s GitHub. Here, we set up the default admin account and database credentials. Additionally, we set the proxy option to "edge" which basically means that communication with Keycloak will happen over HTTP and not over HTTPS protocol. This is acceptable as the Nginx reverse proxy will handle SSL for us.

This container depends on the PostgreSQL container, as Keycloak will use it as data storage, and belongs to the cks-network, the same as any other services we will add next.

For the next container, we will have the Keycloak Config CLI.



services:
  keycloak-config-cli:
    image: "bitnami/keycloak-config-cli:5.12.0"
    environment:
      KEYCLOAK_URL: "http://keycloak:8080"
      KEYCLOAK_USER: "admin"
      KEYCLOAK_PASSWORD: "admin"
      IMPORT_FILES_LOCATIONS: "/config/*"
    depends_on:
      - "keycloak"
    volumes:
      - "./keycloak/master.yaml:/config/master.yaml"
    networks:
      - "cks-network"

Again, we have Bitnami’s container described in detail on GitHub. What we need to set up here are basically environment variables related to Keycloak access and the config directory. We also specify that this container belongs to our default network and depends on the Keycloak container. Furthermore, we attach a volume here where the YAML file with realm configuration will be placed in the latter sections of this blog post.

Furthermore, we will have PostgreSQL, which in this case is strongly related to Keycloak as well.



services:
  postgres:
    image: "bitnami/postgresql:15.6.0"
    environment:
      POSTGRESQL_USERNAME: "postgres"
      POSTGRESQL_PASSWORD: "postgres"
      POSTGRESQL_DATABASE: "postgres"
    volumes:
      - "cks-postgres-data:/bitnami/postgresql"
    networks:
      - "cks-network"

volumes:
  cks-postgres-data:
    driver: "local"

Quite simple. Once more, it’s Bitnami’s container with default database access configuration stored in environment volumes. Additionally, we also have a volume attached to prevent data loss in case of container restart. Same network as previously.

As we’ve configured the first database, we can configure another one, so now it’s time for CouchDB.



services:
  couchdb:
    image: "bitnami/couchdb:3.3.3"
    environment:
      COUCHDB_USER: "admin"
      COUCHDB_PASSWORD: "admin"
      COUCHDB_SECRET: "top-secret"
      COUCHDB_BIND_ADDRESS: "0.0.0.0"
      COUCHDB_PORT_NUMBER: "5984"
    volumes:
      - "cks-couchdb-data:/bitnami/couchdb"
      - "./couchdb/10-config.ini:/opt/bitnami/couchdb/etc/local.d/10-config.ini"
    networks:
      - "cks-network"

volumes:
  cks-postgres-data:
    driver: "local"

One last time, we use Bitnami’s container version, the description of which can be found here. In environment variables, we have default admin credentials, secret used for cookie encryption, and startup config for CouchDB. Here, we have a persistent volume for data as well, and additionally, we also have a volume with a config file which will be described in detail later.

Last, but not least, there will be Nginx — our reverse proxy.



version: "3.9"

services:
  nginx:
    build: "./nginx"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "./nginx/certs:/opt/bitnami/openresty/nginx/conf/bitnami/certs:ro"
      - "./nginx/server_blocks:/opt/bitnami/openresty/nginx/conf/server_blocks:ro"
      - "./nginx/lua:/opt/bitnami/openresty/lua:ro"
    depends_on:
      - "couchdb"
      - "keycloak"
    networks:
      - "cks-network"

This service will also use the Bitnami container, but we will add a few packages there. Don’t worry about it now as we will cover it in a separate section. Nginx is also the single service that exposes ports so we can communicate with it. It depends on both the CouchDB and Keycloak services and belongs to the same network as all other containers. In volumes, we have separate directories attached for SSL certificates, server block definitions, and Lua scripts.

Now that the overall infrastructure is ready, we can focus on configuring individual services.

Configuring Keycloak

Theoretically, we could configure the entire Keycloak realm manually by clicking appropriate options in the GUI. However, I believe that posting all the screenshots here would not be practical as there would be a lot of them, and they may change over time. That’s why I decided to incorporate the Keycloak Config CLI. Thanks to this utility, we can store the configuration in a convenient YAML file that will be simple to present and describe.

In our application, we will have only one realm called “master”, and the initial setup looks as follows:



realm: "master"
attributes:
  frontendUrl: "https://auth.oblivio.localhost"

This is not very interesting but defines the realm name and the URL of the frontend application. We will later define an appropriate server block in Nginx to proxy this particular subdomain to the Keycloak container.

If you are wondering about the part with “oblivio”, it is nothing special. I just decided to name this application somehow and chose this particular word. It means “forgetfulness” or “loss of remembrance.”

The next part of our configuration is groups definitions. We will have one main group for CouchDB users with two subgroups for admins and regular users. Later, using attribute mapper, we will add appropriate roles for the access token so CouchDB can use it to properly identify user roles.



groups:
  - name: "couchdb"
    path: "/couchdb"
    subGroups:
      - name: "admins"
        path: "/couchdb/admins"
        attributes:
          _couchdb.roles:
            - "_admin"
      - name: "users"
        path: "/couchdb/users"
        attributes:
          _couchdb.roles:
            - "_user"

The attribute mentioned is called _couchdb.roles, and it is the default property name used by CouchDB to infer user roles from the access token, but it can also be changed to another value if needed.

Later, we have clients configuration. For now, we have only one client which will be used by Nginx to authorize access to CouchDB, but in the next part of the series, we will add one more.



clients:
  - clientId: "couchdb-proxy"
    name: "CouchDB Proxy"
    publicClient: "false"
    clientAuthenticatorType: "client-secret"
    secret: "32scbZbgGNSaVOAAuZHgYeTjdQrkfwTh"
    redirectUris:
      - "https://couchdb.oblivio.localhost/*"
    standardFlowEnabled: "true"
    directAccessGrantsEnabled: "false"
    optionalClientScopes:
      - "couchdb"
      - "profile"
      - "email"

This client type is confidential and has a secret set up so Nginx would be able to store it securely. The CouchDB Proxy client allows for only the authorization code flow known for OAuth2 and permits only redirection URIs to the specified subdomain where the CouchDB instance will be available.

Additionally, it comes with three optional client scopes. Email and profile scopes are shipped with the default Keycloak config, but the scope for CouchDB is custom, and we can define it with the following YAML.



clientScopes:
  - name: "couchdb"
    description: "CouchDB"
    protocol: "openid-connect"
    protocolMappers:
      - name: "couchdb-roles"
        protocol: "openid-connect"
        protocolMapper: "oidc-usermodel-attribute-mapper"
        config:
          user.attribute: "_couchdb.roles"
          claim.name: "_couchdb\\.roles"
          jsonType.label: "String"
          userinfo.token.claim: "true"
          access.token.claim: "true"
          id.token.claim: "false"
          multivalued: "true"
          aggregate.attrs: "true"

This scope contains a protocol mapper for OpenID Connect which defines to which property the attribute we added to the group will be mapped. Notice that the claim name contains two backslashes as otherwise the dot would mean that “roles” should be an object inside of the “_couchdb” property, which is not what we want. Backslashes prevent this behavior and store the whole string as one property name. As the user may have more than one role, we set this claim as a multivalued and aggregated attribute. At the end, we define in which tokens it should be present.

The last part of the configuration is the RSA key configuration that will be later used to sign generated tokens.



components:
  org.keycloak.keys.KeyProvider:
    - name: "rsa"
      providerId: "rsa"
      config:
        active: ["true"]
        enabled: ["true"]
        priority: ["1000"]
        algorithm: ["RS256"]
        kid: ["xvAsHaF2w0M1y9GG6bmFannhp9aFLKvHQRaAAb8gUYc"]
        privateKey: ["-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDM98i2/CFRiFYNlUtJ5ppUNyZUOa2+7SMnya3tzfrPOEVma6AJAMJ9YR2CL6SIkyz6q5RqnhQSXTzvPO9OasKuXLWtxpjVZRawCXoCciyaJTLe8qcmb6SOCOsjRSiGB1PaivJ/7NbCDiP6r8BxX4TXsYfdGW2EDBot+klxG6a+FObCA7KJ1bp/yPbgpP+mNyj7P8lG22E3USRjE3g8ag/J8b3UK+Azu1yBmdYAEPG1qz8q46tgF9qJiDo7QNDroRDLxoclypMsHJ3AIbJh5lquAl4uTALYMLI2foKJqXlc+JZ9tdTzxYg04R7SKuAcizdjZ2VccJNpGySGs5i8XguTAgMBAAECggEAOjpiQOmbpYfvumghPVtPmIEaWG8SVt0TUahPyvDrQZcg0BnfGu+mUOwX7/YM7eexrXy06x0BYr4uI2DSMxrNN6+KxVVX8beIHHZ0vOEmnpvWudOBfL/WpasO8bQh8QF/5uP2RDVKRVKzEfJ/3zVdjdEXYc5peEvf3BPwbTuHwRO4F69hgCZi8saBNXBinnOwQ8MSsUeA4RsC7+WcxygufBhNqjkqrYbpznkaZI5nrVdw7mb5E9KcOxbg0BUWnz141gPuUpu2O0iFiiAZoSlDtIwKCtdcvc2UJMYbXK9ORsIscRwP6b8T6O3Uhq1zkXQyjtLWbrfcpoNGGJ2udRFDEQKBgQD/UUVugyEOP98+S5e/2d2W+rk96HX7CKMQwj/ZA9NETugGAiD7fuUpbu3NzrAN7tsTXGOyArBEfdcXwjhvg52WhO9eOB/mcaiXEWVTk89ZrbcvZknDljKWM5zXOMvVZXAK6ci53jfVn5RLA2RIjK5mds1pLXIWgDHPrXFI7Nkb/wKBgQDNhA4/aaAyko8NiAWvk0hYZYQCFSH3YBl28lt90zzDjoZaQd/s8mpaRO4TY+KwFGBEznlFa28e1g0YzpZh13+V1ss4WT9NV/3tux2rblJWa2kmYbA/PeQDLHVfC/fq46T90uUW0wRIkc+nVTKG94Oo+tPUERjlmSJbzsjUDJLgbQKBgCXW9LRhSNfkzYBdEbuEXZwPwr6TIlE3QXutXmsabwhTrX2eeSbs8qfGYgY7mMon2V4wNjJexaMRB3zk8xpL5mI1h4huRwQPWk4xbNQLNxLydRDYVxxeuVabhaY8K7GP3CAx7+bkMWA+y2qmsQkzmHFlMCJjcuI0060U5pJJUBAfAoGALI30iMrdcBlV6hkTIn1Lsd5QQCNUucybuK3SJ/Ujt0Gu3uJpKXVkmS1Yb9u3yXShaklZATPJY2YEcNxYvd16S4HFjPHMR3hMFL38MK46K4IdybRkAVHpnMaGq5Rsqv+vRVfzUn9s7k6uNhjCW4BNitTWF6OdQilwyXaLE22magECgYAx1tGChvGQM3rYyJDA22ZNU4b+olc2bBJ0v45EX0unjGseuzPTKQRaGp8LqgByXcMuZqCCidsvlfrrz16hGHnsqPQFSV4ZL4D1pOKshmWhscLtF10FeC8z0QoJCNFaPuRkMCXyx+X+XjGsdfKRO/84z/7FfCI0t2QvvGWSnRpKUw==\n-----END PRIVATE KEY-----n"]
        certificate: ["-----BEGIN CERTIFICATE-----\nMIIDhzCCAm+gAwIBAgIUVv9/pUd6omHb9BhGmfZb/jVPmz4wDQYJKoZIhvcNAQELBQAwUzELMAkGA1UEBhMCUEwxEzARBgNVBAgMClNvbWUtU3RhdGUxHTAbBgNVBAcMFEdyZWF0ZXIgUG9sYW5kV2Fyc2F3MRAwDgYDVQQKDAdPYmxpdmlvMB4XDTI0MDUxNzE0NTkxNloXDTI2MDUxNzE0NTkxNlowUzELMAkGA1UEBhMCUEwxEzARBgNVBAgMClNvbWUtU3RhdGUxHTAbBgNVBAcMFEdyZWF0ZXIgUG9sYW5kV2Fyc2F3MRAwDgYDVQQKDAdPYmxpdmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzPfItvwhUYhWDZVLSeaaVDcmVDmtvu0jJ8mt7c36zzhFZmugCQDCfWEdgi+kiJMs+quUap4UEl087zzvTmrCrly1rcaY1WUWsAl6AnIsmiUy3vKnJm+kjgjrI0UohgdT2oryf+zWwg4j+q/AcV+E17GH3RlthAwaLfpJcRumvhTmwgOyidW6f8j24KT/pjco+z/JRtthN1EkYxN4PGoPyfG91CvgM7tcgZnWABDxtas/KuOrYBfaiYg6O0DQ66EQy8aHJcqTLBydwCGyYeZargJeLkwC2DCyNn6Cial5XPiWfbXU88WINOEe0irgHIs3Y2dlXHCTaRskhrOYvF4LkwIDAQABo1MwUTAdBgNVHQ4EFgQU9poNBVHTqAYUxc5c4naQhd2kOOswHwYDVR0jBBgwFoAU9poNBVHTqAYUxc5c4naQhd2kOOswDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANcDxRDTyi1VLSjA4DFm/s0aSYSRtiGJoYxCSjxW+IthzMDmV6kuI7c/n+O5gIOTBQ2gCF9evbVbFcF/nYq4zKo5WvCfrZ8Hekvjdm5TOSKMRGWaoydOsVsRPlvNN2q+iVFzmymPixWRblLzbYG1T0lRn6tLn2BKH0qkNUUg68ljA8qYgvulYo5FzSLB1KgZRjDyyDS5+IT/vr/M2H/4h1eCPdD2JROfxf4+3OKBXg5N2Y6DJ/mwNqe+8WGOLmaPDV6GaBVR8BcryYBohrEwYwouhqvNYsk5c1wLBS+k4T1PHC53I/9oGrdhX9jDQiHvQ2CzTp5e9rscbVr71nv03ug==\n-----END CERTIFICATE-----\n"]

I understand that at this point, you may feel uneasy about hard-coding the private key and secret in the earlier part. Don’t worry I feel the same way, but as this is mostly a proof of concept, I decided that this would be simpler at this point, so we will stick to this for now.

And that’s it, the whole configuration for the Keycloak realm. Later, we will also add a user to test it, but for now, we can move on to the CouchDB configuration.

Preparing CouchDB for SSO Integration

Since the CouchDB configuration is shorter than the previous one, I will simply put the entire config below and briefly describe it.



[couchdb]
uuid = 5f1a34cf3b35423690c2474a7527e2ff

[chttpd]
authentication_handlers = {chttpd_auth, jwt_authentication_handler}, {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler}
require_valid_user = false

[jwt_auth]
required_claims = exp, iat
roles_claim_path = _couchdb\.roles

[jwt_keys]
rsa:xvAsHaF2w0M1y9GG6bmFannhp9aFLKvHQRaAAb8gUYc = -----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzPfItvwhUYhWDZVLSeaaVDcmVDmtvu0jJ8mt7c36zzhFZmugCQDCfWEdgi+kiJMs+quUap4UEl087zzvTmrCrly1rcaY1WUWsAl6AnIsmiUy3vKnJm+kjgjrI0UohgdT2oryf+zWwg4j+q/AcV+E17GH3RlthAwaLfpJcRumvhTmwgOyidW6f8j24KT/pjco+z/JRtthN1EkYxN4PGoPyfG91CvgM7tcgZnWABDxtas/KuOrYBfaiYg6O0DQ66EQy8aHJcqTLBydwCGyYeZargJeLkwC2DCyNn6Cial5XPiWfbXU88WINOEe0irgHIs3Y2dlXHCTaRskhrOYvF4LkwIDAQAB\n-----END PUBLIC KEY-----\n

Moving from top to bottom, we have the UUID, which serves as the unique identifier for the instance. Later, we have a list of possible authentication handlers where we add the JWT authentication handler as it is not enabled by default. Because our users are no longer stored in the CouchDB instance, we also have to disable user validation.

Further, we have the JWT config where we set what claims are required and the path to the claim where the user’s roles are stored. Notice that here there is also a backslash which serves a similar purpose to the two backslashes described in the Keycloak configuration section. Lastly, we have the public key derived from the private key used to sign the access token so CouchDB could know that it can trust tokens provided from Keycloak.

Nginx as a Reverse Proxy

We are almost there, but before we are ready to test this solution, we have to configure one last thing — the reverse proxy. As this part will be relatively long, it will be split into three subsections for building the Docker image, configuring Nginx’s server blocks, and writing Lua scripts for authentication.

Building the Docker Image

The Bitnami image for OpenResty is good as is, but it lacks a few Lua packages that we will need for integration with Keycloak.



FROM bitnami/openresty:1.25.3-1

RUN opm get zmartzone/lua-resty-openidc
RUN opm get ledgetech/lua-resty-http
RUN opm get bungle/lua-resty-session=3.10

We are mostly interested in “zmartzone/lua-resty-openidc” as it implements the OpenID Connect Relying Party functionality, which we will benefit from for authorization with Keycloak. The following two packages are dependencies needed for the first one. As of the time of writing this post, the created solutions do not work with the “bungle/lua-resty-session” version newer than 3.10, so it is fixed to this version here.

Configuring Server Blocks

Now we can start configuring server blocks for our Nginx container. We will start simple with the following block.



server {
    server_name _;

    listen 80;
    listen [::]:80;

    return 301 https://$host$request_uri;
}

This part listens on port 80 for all domains and subdomains and redirects any request sent over HTTP to the secure version of the protocol. We will later generate self-signed certificates so we can use HTTPS on localhost.

Next, we have a server block for the Keycloak instance.



server {
    server_name auth.oblivio.localhost;

    listen 443 ssl;
    listen [::]:443 ssl;

    http2 on;

    ssl_certificate bitnami/certs/server.crt;
    ssl_certificate_key bitnami/certs/server.key;

    location / {
        proxy_pass http://keycloak:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

This part listens on port 443 for both IPv4 and IPv6 and handles requests sent to the subdomain “auth.oblivio.localhost”, as you may remember we set this address earlier as the frontend URL of Keycloak. There are also paths to SSL certificate and private key. We will generate them at the end of this section. And at the end, we handle all locations for this server to pass the request to the Keycloak container.

The last block here will be for the CouchDB.



server {
    server_name couchdb.oblivio.localhost;

    listen 443 ssl;
    listen [::]:443 ssl;

    resolver 127.0.0.11 valid=10s;

    http2 on;

    ssl_certificate bitnami/certs/server.crt;
    ssl_certificate_key bitnami/certs/server.key;

    location / {
        access_by_lua_file /opt/bitnami/openresty/lua/access.lua;
        proxy_pass http://couchdb:5984;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Once more, we have a similar setup. We configure the subdomain to handle, “couchdb.oblivio.localhost” in this case, ports to listen, and paths to SSL certificate and private key.

What’s different here is the “resolver” directive. It points to the special Docker DNS resolver, and it is needed because of subrequests that “zmartzone/lua-resty-openidc” will send for authentication purposes. Additionally, there is the “access_by_lua_file” directive, which points to the Lua script where we will create authentication logic in the next section.

Before moving on to the Lua part, let’s generate self-signed certificates using the “mkcert” utility. It’s relatively simple. This tool will also install this certificate in appropriate system directories so our browser can trust them.



mkcert -cert-file nginx/certs/server.crt -key-file nginx/certs/server.key oblivio.localhost \*.oblivio.localhost

We only have to define the path where the certificate and private key will be stored, as well as which domains it will protect. Quite simple, isn’t it?

Lua Script for Authentication

Here we will start by defining options for “zmartzone/lua-resty-openidc” that will match our specific use case.



local opts = {
    redirect_uri = "/callback",
    discovery = {
        issuer = "https://auth.oblivio.localhost/realms/master",
        authorization_endpoint = "https://auth.oblivio.localhost/realms/master/protocol/openid-connect/auth",
        end_session_endpoint = "https://auth.oblivio.localhost/realms/master/protocol/openid-connect/logout",
        token_endpoint = "http://keycloak:8080/realms/master/protocol/openid-connect/token",
        jwks_uri = "http://keycloak:8080/realms/master/protocol/openid-connect/certs",
        userinfo_endpoint = "http://keycloak:8080/realms/master/protocol/openid-connect/userinfo",
        revocation_endpoint = "http://keycloak:8080/realms/master/protocol/openid-connect/revoke",
        introspection_endpoint = "http://keycloak:8080/realms/master/protocol/openid-connect/token/introspect"
    },
    client_id = "couchdb-proxy",
    client_secret = "32scbZbgGNSaVOAAuZHgYeTjdQrkfwTh",
    scope = "openid couchdb",
    renew_access_token_on_expiry = true,
    access_token_expires_in = 60,
    accept_none_alg = false,
    accept_unsupported_alg = false,
    session_contents = {
        id_token = true,
        access_token = true,
        refresh_token = true
    }
}

There is quite a lot of them, but going up to bottom, you can see that first we defined the callback where the user should be redirected after successful authentication. In our case, it is a relative path to the current subdomain, which is “auth.oblivio.localhost”. Next, we have the URL endpoints for the OpenID Connect provider. Here you may notice that some of them are pointing directly to the container, while others use the whole subdomain. This depends on who will be actually using the given URL. If it is meant to be used by the browser, then we will go with the subdomain. If it is meant to be used by the library itself, we use the direct container address.

Later we have to choose which client we want to use and provide its secret, as well as the scopes that we want to use. We have the “couchdb” scope there, which will add the CouchDB roles based on the group configured in Keycloak to the access token. Further, we have token configuration like expiration time, whether it should be renewed when expired, and if unsupported algorithms are allowed.

At the end, we define session content, which means what information will be stored in the session. In our case, we want to have the ID Token, access token, and refresh token.

When we have options prepared, we can invoke the authentication function from “zmartzone/lua-resty-openidc” and benefit from it doing all the job for us.



local res, err = require("resty.openidc").authenticate(opts)

if err then
    ngx.status = 500
    ngx.say(err)
    ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
end

if res then
    ngx.req.set_header("Authorization", "Bearer " .. res.access_token)
end

It is quite simple. If the method invocation ends with an error, we also exit the whole process with an internal server error. Otherwise, if we have a successful response from the authentication provider, we add the authorization header of “Bearer” type with the access token returned by Keycloak to the request so CouchDB could use it to verify user identity.

Running and Testing the Setup

Whew — the configuration part is already behind us! Now we can start testing the solutions proposed. We can initiate the entire application with the following command:



docker compose up -d

Hopefully, if everything went well, our application should be already up and running. You can confirm it by navigating to the “https://auth.oblivio.localhost" address in your web browser. You should see the default Keycloak authentication page there.

As we are here, we can log in as the default administrator. We set its credentials in Docker environment variables, so now it is time to make use of them.

If you signed in successfully, you can go through all pages to check whether all options from the YAML file we prepared are present as expected here. What’s more, I would like you to also create a new user. We will try to sign in to CouchDB with its credentials.

To do this, go to the “Users” tab and click the “Add user” button. Here, click the “Email verified” toggle and set the user details according to your preference. Then, join this user to the “/couchdb/admins” group by clicking the “Join Groups” button and checking the correct group.

Click create and go to the “Credentials” tab. Then press the “Set password” button and create a password of your choice. For simplicity, disable the “Temporary” option.

Perfect! You just created a new user. Now open a private browser tab and go to “https://couchdb.oblivio.localhost" to check if you will be able to log in with its credentials to CouchDB. As you are not authenticated yet, you should be redirected to “https://auth.oblivio.localhost" where you have to provide the credentials you set previously.

Click the “Sign In” button, and you will be redirected to the CouchDB welcome page. There is only JSON with some basic information about the CouchDB instance, but from this place, you can go to “/_session” path where you can see information about the authenticated user or to “/_utils” to access Fauxton, which is a GUI for CouchDB management. From “/_session”, you should receive JSON similar to the one below:



{
  "ok":true,
  "userCtx":{
    "name":"287e5d17-4937-48b7-a6fe-cc2029c1cf68",
    "roles":["_admin"]
  },
  "info":{
    "authentication_handlers":["proxy","jwt","cookie","default"],
    "authenticated":"jwt"
  }
}

Here you can find out that you are authenticated as the user with the given UUID. This is a unique identifier assigned to the user by Keycloak. There is also information about user roles. If you were to join another group we created in Keycloak, you would see the “_user” role here. At the end, there is information about available authentication handlers and which one was used to authenticate the current user.

Summary

Yeah! We have completed the first part of the series. With this blog post, we have discovered that by using Keycloak, Nginx, Lua, and a bunch of configuration files, we are able to access the CouchDB instance with our own Single Sign-On managed by Keycloak. It was quite a long process, but in the end, we achieved what we set out to accomplish in this part. In the next parts, we will aim to extend this solution to also be able to access the CouchDB instance from the shell with our custom-made curl wrapper and to ensure that this solution is able to work with CouchApps as well.

If you need all the code from this article in one place, you can find it in my GitHub repository.

At this point, thank you for reading this article. I would love to hear your thoughts about this solution. Whether you work actively with CouchDB or Keycloak, can you spot weaknesses in this solution? Or maybe you would improve something? I would love to hear about it in the comments.

Don’t forget to check out my other articles for more tips and insights and other parts of this series when they are created. Happy hacking!

Get Rid of Tightly Coupled Modules and Circular Dependencies in NestJS

Tomasz Kisiel — Sun, 16 Jun 2024 15:26:52 +0000

NestJS is a great NodeJS framework that injects a lot of refreshment into the ecosystem of Node’s backend solutions. With its robust module system, it allows planning and building scalable architecture that contains modules responsible for wrapping related logic together. While working within a modular environment like this, you may sometimes encounter an issue of circular dependencies caused by tightly coupled modules. In most cases, you will recognize this as an error presented below.

[Nest] 2788 - 06/10/2023, 12:56:50 PM LOG [InjectorLogger] 
> Nest encountered an undefined dependency. 
> This may be due to a circular import or a missing dependency declaration. 
[Nest] 2788 - 06/10/2023, 12:56:50 PM ERROR [ExceptionHandler] 
> Nest can't resolve dependencies of the UserService (?). 
> Please make sure that the argument dependency at index [0] is available in the UserModule context.

This is caused by two services that depend on each other to perform their logic. In this example above, the PostService is a dependency of UserService, but also UserService is a dependency of PostService. This causes Nest’s dependency injection container to be unable to resolve this situation.

As per Nest’s documentation, you may, of course, use the forwardRef() function. However, this is only a temporary solution. If you don't truly solve the problem of tight coupling and circular dependencies, adding newer modules will become quite painful, and you will have to wrap most of your dependencies with the mentioned function.

In today’s post, I would like to suggest another solution, which, of course, may not be applicable to all cases. But even if it solves only half of your circular dependencies, this may be already a good step forward. So, before further ado, let’s examine an example problem and the proposed solution.

The problem

Let’s consider a not too simple application for writing blog posts. But, apart from just creating posts, we were required to implement a bunch of additional actions like sending notifications to the author’s followers, increasing the author’s reputation after post creation, configuring a payment gateway if the post is behind a paywall, and some other fancy features. At the end, we may end up with code similar to the following.

class PostService {
  constructor(
    private readonly postRepositiory: PostRepository,
    private readonly userService: UserService,
    private readonly reputationService: ReputationService,
    private readonly notificationService: NotificationService,
    private readonly trackingService: TrackingService,
    private readonly paymentService: PaymentService,
    private readonly moderationService: ModerationService 
  ) {}

  public createPost(args: CreatePostArgs): Post {
    const post = this.postRepository.create(args);

    this.reputationService.increaseReputation(args.userId);
    this.notificationService.notifyFollowersAboutPost(post);
    this.userService.updateUserActivity('post.created', post);
    this.trackingService.registerTrackable('post', post);
    this.moderationService.checkPostContentForViolations(post.content);    

    if (args.isPremium) {
      this.paymentService.chargeUserForPremiumPost(args.userId);
    }

    return post;
  }
}

This example is obviously made up, so please don’t pay too much attention to the details. I just want you to notice how many services PostService is dependent on and imagine that these services may also be dependent on PostService. For example, the notification service may require additional data from the post service to dispatch notifications, or the ModerationService may need to modify the post again via PostService after moderation. This is where circular dependencies occur.

So now that we have a grasp of the problem, let’s explore the solution that I want to propose.

The solution

The solution I want to propose for this problem is to use the well-known concept of Event-Driven Architecture. Instead of calling all subsequent actions from the createPost method, we will simply create the post there and emit an event. Then, any module interested in performing some action related to the event may do so without crossing its logical borders.

NestJS already comes with handy tools that we can use to benefit from events. If you don't yet have the package installed, you can simply add @nestjs/event-emitter to your application.

yarn add @nestjs/event-emitter

And when the package is installed, add the EventEmitterModule to the root module of your application.

import { Module } from '@nestjs/common';
import { EventEmitterModule } from '@nestjs/event-emitter';

@Module({
  imports: [
    EventEmitterModule.forRoot(),
    // ...
  ],
})
export class AppModule {}

When we have it ready, we may simply get rid of all PostService dependencies and replace them with only one — EventEmitter2. After this, we may also remove subsequent method invocations from the createPost method and instead emit an event with its key and payload. The payload may be defined as a separate interface or class if you wish, but here, for presentation purposes, I will just emit the created post as a payload.

class PostsService {
  constructor(private eventEmitter: EventEmitter2) {}

  public createPost(args: CreatePostArgs): Post {
    const post = this.db.posts.create({ ... });
    this.eventEmitter.emit('post.created', post);

    return post;
  }
}

The last thing we have to do is to add a listener to the modules that may be interested in this event. For example, in the notification module, we may have a listener as below. In the other modules, the code will be pretty much the same, just other services will be involved in event handling.

class NotificationsListener {
  constructor(
    private readonly notificationsService: NotificationsService
  ) {}

  @OnEvent('post.created')
  handlePostCreated(post: Post) {
    this.reputationService.notifyFollowersAboutPost(post);
  }
}

This way, the NotificationService and PostService are only loosely coupled now. The PostService is no longer dependent on NotificationService. We get rid of circular dependency here, yet we keep the functionality still working — Yay!

Summary

To sum up, in this article we explored the proposition of introducing events into your application to solve the issue of circular dependencies. This approach helps modules to stay within their borders yet react to actions performed in another module as well. Even though this way may not be applicable to all solutions it definitely may fix the one presented today.

I want to say thank you to all who read this article. I would love to hear your thoughts about this proposal and your ways of tackling circular dependencies in your applications, so feel free to share.

Don’t forget to check out my other articles for more tips and insights. Happy hacking!

CouchGO! — Enhancing CouchDB with Query Server Written in Go

Tomasz Kisiel — Sat, 15 Jun 2024 12:37:57 +0000

Over the past month, I’ve been actively working on proof-of-concept projects related to CouchDB, exploring its features and preparing for future tasks. During this period, I’ve gone through the CouchDB documentation multiple times to ensure I understand how everything works. While reading through the documentation, I came across a statement that despite CouchDB shipping with a default Query Server written in JavaScript, creating a custom implementation is relatively simple and custom solutions already exist in the wild.

I did some quick research and found implementations written in Python, Ruby, or Clojure. Since the whole implementation didn’t seem too long, I decided to experiment with CouchDB by trying to write my own custom Query Server. To do this, I chose Go as the language. I haven’t had much experience with this language before, except for using Go templates in Helm’s charts, but I wanted to try something new and thought this project would be a great opportunity for it.

Understanding the Query Server

Before starting work, I revisited the CouchDB documentation once more to understand how the Query Server actually works. According to the documentation, the high-level overview of the Query Server is quite simple:

The Query server is an external process that communicates with CouchDB via the JSON protocol over a stdio interface and handles all design function calls […].

The structure of the commands sent by CouchDB to the Query Server can be expressed as [<command>, <*arguments>] or ["ddoc", <design_doc_id>, [<subcommand>, <funcname>], [<argument1>, <argument2>, …]] in the case of design documents.

So basically, what I had to do was write an application capable of parsing this kind of JSON from STDIO, performing the expected operations, and returning responses as specified in the documentation. There was a lot of type casting involved to handle a wide range of commands in Go code. Specific details about each command can be found under the Query Server Protocol section of the documentation.

One problem I faced here was that the Query Server should be able to interpret and execute arbitrary code provided in design documents. Knowing that Go is a compiled language, I expected to be stuck at this point. Thankfully, I quickly found the Yeagi package, which is capable of interpreting Go code with ease. It allows creating a sandbox and controlling access to which packages can be imported in the interpreted code. In my case, I decided to expose only my package called couchgo, but other standard packages can be easily added as well.

Introducing CouchGO!

As a result of my work, an application called CouchGO! emerged. Although it follows the Query Server Protocol, it is not a one-to-one reimplementation of the JavaScript version as it has its own approaches to handling design document functions.

For example, in CouchGO!, there is no helper function like emit. To emit values, you simply return them from the map function. Additionally, each function in the design document follows the same pattern: it has only one argument, which is an object containing function-specific properties, and is supposed to return only one value as a result. This value doesn't have to be a primitive; depending on the function, it may be an object, a map, or even an error.

To start working with CouchGO!, you just need to download the executable binary from my GitHub repository, place it somewhere in the CouchDB instance, and add an environment variable that allows CouchDB to start the CouchGO! process.

For instance, if you place the couchgo executable into the /opt/couchdb/bin directory, you would add following environment variable to enable it to work.

export COUCHDB_QUERY_SERVER_GO="/opt/couchdb/bin/couchgo"

Writing Functions with CouchGO!

To gain a quick understanding of how to write functions with CouchGO!, let’s explore the following function interface:

func Func(args couchgo.FuncInput) couchgo.FuncOutput { ... }

Each function in CouchGO! will follow this pattern, where Func is replaced with the appropriate function name. Currently, CouchGO! supports the following function types:

Map
Reduce
Filter
Update
Validate (validate_doc_update)

Let’s examine an example design document that specifies a view with map and reduce functions, as well as a validate_doc_update function. Additionally, we need to specify that we are using Go as the language.

{
  "_id": "_design/ddoc-go",
  "views": {
    "view": {
      "map": "func Map(args couchgo.MapInput) couchgo.MapOutput {\n\tout := couchgo.MapOutput{}\n\tout = append(out, [2]interface{}{args.Doc[\"_id\"], 1})\n\tout = append(out, [2]interface{}{args.Doc[\"_id\"], 2})\n\tout = append(out, [2]interface{}{args.Doc[\"_id\"], 3})\n\t\n\treturn out\n}",
      "reduce": "func Reduce(args couchgo.ReduceInput) couchgo.ReduceOutput {\n\tout := 0.0\n\n\tfor _, value := range args.Values {\n\t\tout += value.(float64)\n\t}\n\n\treturn out\n}"
    }
  },
  "validate_doc_update": "func Validate(args couchgo.ValidateInput) couchgo.ValidateOutput {\n\tif args.NewDoc[\"type\"] == \"post\" {\n\t\tif args.NewDoc[\"title\"] == nil || args.NewDoc[\"content\"] == nil {\n\t\t\treturn couchgo.ForbiddenError{Message: \"Title and content are required\"}\n\t\t}\n\n\t\treturn nil\n\t}\n\n\tif args.NewDoc[\"type\"] == \"comment\" {\n\t\tif args.NewDoc[\"post\"] == nil || args.NewDoc[\"author\"] == nil || args.NewDoc[\"content\"] == nil {\n\t\t\treturn couchgo.ForbiddenError{Message: \"Post, author, and content are required\"}\n\t\t}\n\n\t\treturn nil\n\t}\n\n\tif args.NewDoc[\"type\"] == \"user\" {\n\t\tif args.NewDoc[\"username\"] == nil || args.NewDoc[\"email\"] == nil {\n\t\t\treturn couchgo.ForbiddenError{Message: \"Username and email are required\"}\n\t\t}\n\n\t\treturn nil\n\t}\n\n\treturn couchgo.ForbiddenError{Message: \"Invalid document type\"}\n}",
  "language": "go"
}

Now, let’s break down each function starting with the map function:

func Map(args couchgo.MapInput) couchgo.MapOutput {
  out := couchgo.MapOutput{}
  out = append(out, [2]interface{}{args.Doc["_id"], 1})
  out = append(out, [2]interface{}{args.Doc["_id"], 2})
  out = append(out, [2]interface{}{args.Doc["_id"], 3})

  return out
}

In CouchGO!, there is no emit function; instead, you return a slice of key-value tuples where both key and value can be of any type. The document object isn't directly passed to the function as in JavaScript; rather, it's wrapped in an object. The document itself is simply a hashmap of various values.

Next, let’s examine the reduce function:

func Reduce(args couchgo.ReduceInput) couchgo.ReduceOutput {
  out := 0.0
  for _, value := range args.Values {
    out += value.(float64)
  }
  return out
}

Similar to JavaScript, the reduce function in CouchGO! takes keys, values, and a rereduce parameter, all wrapped into a single object. This function should return a single value of any type that represents the result of the reduction operation.

Finally, let’s look at the Validate function, which corresponds to the validate_doc_update property:

func Validate(args couchgo.ValidateInput) couchgo.ValidateOutput {
  if args.NewDoc["type"] == "post" {
    if args.NewDoc["title"] == nil || args.NewDoc["content"] == nil {
      return couchgo.ForbiddenError{Message: "Title and content are required"}
    }

    return nil
  }

  if args.NewDoc["type"] == "comment" {
    if args.NewDoc["post"] == nil || args.NewDoc["author"] == nil || args.NewDoc["content"] == nil {
      return couchgo.ForbiddenError{Message: "Post, author, and content are required"}
    }

    return nil
  }

  return nil
}

In this function, we receive parameters such as the new document, old document, user context, and security object, all wrapped into one object passed as a function argument. Here, we’re expected to validate if the document can be updated and return an error if not. Similar to the JavaScript version, we can return two types of errors: ForbiddenError or UnauthorizedError. If the document can be updated, we should return nil.

For more detailed examples, they can be found in my GitHub repository. One important thing to note is that the function names are not arbitrary; they should always match the type of function they represent, such as Map, Reduce, Filter, etc.

CouchGO! Performance

Even though writing my own Query Server was a really fun experience, it wouldn’t make much sense if I didn’t compare it with existing solutions. So, I prepared a few simple tests in a Docker container to check how much faster CouchGO! can:

Index 100k documents (indexing in CouchDB means executing map functions from views)
Execute reduce function for 100k documents
Filter change feed for 100k documents
Perform update function for 1k requests

I seeded the database with the expected number of documents and measured response times or differentiated timestamp logs from the Docker container using dedicated shell scripts. The details of the implementation can be found in my GitHub repository. The results are presented in the table below.

Test	CouchGO!	CouchJS	Boost
Indexing	141.713s	421.529s	2.97x
Reducing	7672ms	15642ms	2.04x
Filtering	28.928s	80.594s	2.79x
Updating	7.742s	9.661s	1.25x

As you can see, the boost over the JavaScript implementation is significant: almost three times faster in the case of indexing, more than twice as fast for reduce and filter functions. The boost is relatively small for update functions, but still faster than JavaScript.

Conclusion

As the author of the documentation promised, writing a custom Query Server wasn’t that hard when following the Query Server Protocol. Even though CouchGO! lacks a few deprecated functions in general, it provides a significant boost over the JavaScript version even at this early stage of development. I believe there is still plenty of room for improvements.

If you need all the code from this article in one place, you can find it in my GitHub repository.

Thank you for reading this article. I would love to hear your thoughts about this solution. Would you use it with your CouchDB instance, or maybe you already use some custom-made Query Server? I would appreciate hearing about it in the comments.

Don’t forget to check out my other articles for more tips, insights, and other parts of this series as they are created. Happy hacking!