DEV Community: Matsuda

Adding Cognito Authentication to Self-Hosted Langfuse with AWS CDK

Matsuda — Sat, 15 Feb 2025 13:04:48 +0000

Langfuse provides built-in authentication using ID/password by default, but it also allows using other IdPs (NextAuth.js is used).

https://langfuse.com/self-hosting/authentication-and-sso

Although the default authentication method in Langfuse is ID/password, there are cases where integrating an external IdP is desirable for enhanced security.

In this post, I have added authentication using Amazon Cognito (hereafter referred to as Cognito) to Langfuse with AWS CDK, which was created in the previous post.

After configuration, you will be able to log in with Cognito as shown below:

Additionally, it is possible to disable the default authentication mechanism and enforce authentication via Cognito.

This post summarizes the key points of this setup.

The GitHub repository is available below:

https://github.com/mazyu36/langfuse-with-aws-cdk

Why Use an External IdP?

One of the main motivations for using an external IdP is to enhance security.

When using Langfuse's built-in authentication, users sign up via the following screen:

During sign-up, users enter their email addresses, but there is no additional verification step such as sending a verification code via email. As a result, accounts are created immediately, and users are redirected to the Langfuse dashboard.

This means that as long as someone has access, they can sign up and gain access right away, raising concerns about the misuse of API keys.

Another drawback is the limited configurability of authentication settings.

By using an external IdP, you can enforce email verification, restrict sign-up to pre-created users only, and flexibly customize authentication according to your requirements.

This is particularly useful when the application is widely available but you want to limit the users who can issue API keys.

Configuring Cognito

After creating a Cognito user pool and an app client, authentication can be enabled by setting the following environment variables in Langfuse Web (refer to the Langfuse documentation):

AUTH_COGNITO_CLIENT_ID: ID of the user pool client
AUTH_COGNITO_CLIENT_SECRET: Secret of the user pool client
AUTH_COGNITO_ISSUER: Set this to https://cognito-idp.{region}.amazonaws.com/{PoolId}. This value is also mentioned in the NextAuth.js documentation.
AUTH_DISABLE_USERNAME_PASSWORD: Setting this to true disables Langfuse’s built-in authentication. This is optional.
AUTH_COGNITO_ALLOW_ACCOUNT_LINKING: Setting this to true merges accounts if the same email address exists in Langfuse's built-in authentication. This is useful when migrating existing users to an external IdP. This setting is also optional.

Additionally, you must configure Cognito's app client with a callback URL set to https://{Langfuse domain}/api/auth/callback/cognito. Without this, authentication will not function correctly.

One important point to note is that Cognito requires callback URLs (except for localhost) to use https. Therefore, when implementing Cognito authentication in a self-hosted environment, HTTPS is mandatory.

Furthermore, you need to configure Cognito’s managed login (hosted UI) and domain settings.

The details of the CDK implementation will be covered later.

Architecture

Below is the overall architecture. Compared to the previous post, a Cognito UserPool and an associated ACM certificate have been added.

In this setup, a custom domain is used for Cognito’s authentication page, which internally creates a CloudFront distribution. Since ACM certificates must be issued in Virginia (us-east-1), deploying Langfuse in a different region results in a cross-region setup.

To handle this in CDK, the implementation is split into two stacks:

A stack specifically for Cognito’s custom domain ACM certificate (us-east-1 fixed)
A stack for all other resources (region can be any)

Ref: https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-add-custom-domain.html

CDK Implementation

The key implementation points are as follows:

Create a stack for us-east-1
Create Cognito’s UserPool, UserPool Client, and Domain
Configure Langfuse Web environment variables with Cognito information

1. Creating a Stack for `us-east-1`

A new stack is added to create the ACM certificate for Cognito’s custom domain. Since ACM certificates must be issued in us-east-1, the stack's region is fixed.

crossRegionReference is enabled to allow references across regions. This is a CDK-specific mechanism that utilizes SSM parameters. For more details, refer to the CDK documentation.

// Stack for creating ACM certificate
if (appConfig.enableCognitoAuth === true) {
  usEast1Stack = new UsEast1Stack(app, `UsEast1Stack-${envName}`, {
    env: {
      account: appConfig.env?.account ?? process.env.CDK_DEFAULT_ACCOUNT,
      region: 'us-east-1',  // Fixed to us-east-1
    },
    crossRegionReferences: true,  // Enable cross-region references
    domainConfig: appConfig.domainConfig,
  });
}

const stack = new LangfuseWithAwsCdkStack(app, `LangfuseWithAwsCdkStack-${envName}`, {
  env: {
    account: appConfig.env?.account ?? process.env.CDK_DEFAULT_ACCOUNT,
    region: appConfig.env?.region ?? process.env.CDK_DEFAULT_REGION,  // Region for deploying Langfuse
  },
  envName,
  hostName: appConfig.domainConfig?.hostName,
  domainName: appConfig.domainConfig?.domainName,
  crossRegionReferences: true,  // Enable cross-region references
  disableEmailPasswordAuth: appConfig.disableEmailPasswordAuth,
  certificateForCognito: usEast1Stack?.certificateForCognito,  // Pass ACM certificate
});

2. Creating Cognito UserPool, UserPool Client, and Domain

We will now define the resources related to Cognito. The implementation can be found in lib/constructs/auth/cognito-auth.ts.

First, let's create a user pool. There are no particular points to highlight here. Customize the authentication methods as needed based on authentication requirements.

this.userPool = new cognito.UserPool(this, 'UserPool', {
  accountRecovery: cognito.AccountRecovery.EMAIL_ONLY,
  signInAliases: {
    email: true,
  },
  standardAttributes: {
    email: {
      required: true,
      mutable: true,
    },
  },
  selfSignUpEnabled: true,
  userVerification: {
    emailSubject: 'Langfuse - Verify your new account',
  },
  removalPolicy: RemovalPolicy.DESTROY,
});

Next, let's create an app client. It is mandatory to specify the Callback URL and generate a client secret.

this.userPoolclient = this.userPool.addClient('CognitoClient', {
  authFlows: {
    userPassword: true,
    userSrp: true,
  },
  oAuth: {
    flows: {
      authorizationCodeGrant: true,
    },
    scopes: [cognito.OAuthScope.OPENID, cognito.OAuthScope.EMAIL],
    callbackUrls: [`${cdnLoadBalancer.url}/api/auth/callback/cognito`],  // Specify the Callback URL
  },
  generateSecret: true,  // Generate the client secret
});

Next, enable managed login for the Cognito authentication screen and set up a custom domain.

// Configure managed login
new cognito.CfnManagedLoginBranding(this, 'ManagedLoginBranding', {
  userPoolId: this.userPool.userPoolId,
  clientId: this.userPoolclient.userPoolClientId,
  useCognitoProvidedValues: true,
});

// Set up a custom domain for Cognito
const domain = this.userPool.addDomain('CognitoDomain', {
  customDomain: {
    domainName: `auth.${hostName}.${hostedZone.zoneName}`,
    certificate: certificateForCognito,
  },
  managedLoginVersion: cognito.ManagedLoginVersion.NEWER_MANAGED_LOGIN,
});

// **Wait for the A record of Langfuse to be generated**
domain.node.addDependency(cdnLoadBalancer.albARecord!);

// Generate a custom domain record
new route53.ARecord(this, 'CognitoARecord', {
  zone: hostedZone,
  recordName: `auth.${hostName}`,
  target: route53.RecordTarget.fromAlias(new targets.UserPoolDomainTarget(domain)),
});

One crucial point to be aware of is the following:

// **Wait for the A record of Langfuse to be generated**
domain.node.addDependency(cdnLoadBalancer.albARecord!);

Due to the specifications of Cognito's custom domains, a valid A record must exist for the parent domain. In this CDK implementation, the Cognito custom domain is designed to prepend auth. to the domain assigned to Langfuse.

Langfuse domain: langfuse.example.com (The domain can be configured in bin/app-config.ts)
Cognito custom domain: auth.langfuse.example.com

If a valid A record does not exist for the parent Langfuse domain when setting up the Cognito custom domain, the deployment will fail with an error.

Therefore, in the CDK implementation, we define a dependency to ensure that the A record for the Langfuse domain (specifically, the domain assigned to the ALB) is generated before setting up the Cognito custom domain.

This requirement is documented in the following AWS documentation:

A web domain that you own. Its parent domain must have a valid DNS A record. You can assign any value to this record. The parent may be the root of the domain, or a child domain that is one step up in the domain hierarchy. For example, if your custom domain is auth.xyz.example.com, Amazon Cognito must be able to resolve xyz.example.com to an IP address.

https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-add-custom-domain.html#cognito-user-pools-add-custom-domain-prereq

3. Configuring Cognito-related Information in Langfuse Web Environment Variables

Finally, configure the Cognito-related information as environment variables in Langfuse Web.

AUTH_COGNITO_CLIENT_ID: cognitoAuth.userPoolclient.userPoolClientId,
AUTH_COGNITO_CLIENT_SECRET: cognitoAuth.userPoolclient.userPoolClientSecret.unsafeUnwrap(),
AUTH_COGNITO_ISSUER: `https://cognito-idp.${Stack.of(this).region}.amazonaws.com/${cognitoAuth.userPool.userPoolId}`,

With this setup, authentication via Cognito is now enabled.

Verification

To enable Cognito authentication in Langfuse with AWS CDK, set enableCognitoAuth to true in bin/app-config.ts.

In the sample configuration in the repository, it is enabled for prod.

  prod: {
    // omit
    disableEmailPasswordAuth: true,  // Set to true to disable Langfuse's built-in authentication
    enableCognitoAuth: true,  // Enable Cognito authentication
  },

After applying these settings, deploy to the target environment. Since enabling Cognito authentication results in two separate stacks, the --all flag is required.

npx cdk deploy --context env=prod --all

Once the deployment is complete, access the Langfuse URL. You should see a Cognito login menu. Click on it.

With this implementation, users are redirected to Cognito's managed login authentication screen. It may take some time for the custom domain to propagate, so if you encounter an error, try again after a while. If authentication still fails, there may be a configuration issue.

Next, follow the user pool's settings to complete the signup process. In this implementation, after entering signup details, a verification code is sent to the specified email address, so complete the verification process.

Once signup is complete, users will be redirected to the Langfuse initial screen. From here on, authentication via Cognito will be available.

Conclusion

Introducing an external IdP can be a valuable way to enhance the security of a self-hosted Langfuse environment.
Give it a try!

Self-Hosting Langfuse v3 on AWS Using CDK

Matsuda — Thu, 06 Feb 2025 14:11:11 +0000

I created a CDK project to self-host the observability tool (OSS) Langfuse v3 on AWS.

https://github.com/mazyu36/langfuse-with-aws-cdk

In this article, I'll share the usage method, architecture, and troubleshooting know-how.

The following official documents are helpful for self-hosting:

Architecture

I've built Langfuse v3 on AWS. Essentially, it's a simple configuration that replaces Langfuse's architecture with managed services.

Langfuse Application (Web, Worker)

I've deployed the Langfuse container image to ECS on Fargate.

Service-to-service communication with ClickHouse, mentioned later, uses ECS Service Connect. Since Web and Worker act as clients (the requesting side) in service-to-service communication, Service Connect only needs to be defined with client settings.

If you want to reduce costs and just need basic communication, Service Discovery should also work fine.

According to the documentation, for production environments, the following is recommended:

All containers should have at least 2 CPUs and 4GB RAM
Langfuse Web should have 2 instances for redundancy

For production environments, we recommend to use at least 2 CPUs and 4 GB of RAM for all containers. You should have at least two instances of the Langfuse Web container for high availability. For auto-scaling, we recommend to add instances once the CPU utilization exceeds 50% on either container.

In the CDK implementation, you can configure whether to use Fargate Spot in lib/stack-config.ts. In development environments, you can use Fargate Spot to reduce costs.

ClickHouse - OLAP

ClickHouse is also running directly as a container image on ECS on Fargate. EFS is mounted for data persistence.

PostgreSQL - OLTP

Using Aurora Serverless v2.

In the CDK implementation, lib/stack-config.ts allows enabling Zero Capacity for cost reduction.

https://aws.amazon.com/blogs/database/introducing-scaling-to-0-capacity-with-amazon-aurora-serverless-v2

When enabled, the database stops after a certain period without connections. When attempting to connect during this state, it takes about 15 seconds to restart, so retries are necessary after a certain time. This is intended for development environments.

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless-v2-auto-pause.html#auto-pause-whynot

S3 - Blob Storage

Storage for storing events and traces. This uses S3 directly.

Cache/Queue

Redis/Valkey is needed for caching and asynchronous communication between Web and Worker. In the CDK project, ElastiCache is used, with Valkey as the engine for cost benefits.

There are several points to note about ElastiCache, which I'll explain in detail.

Use REDIS_STRING when using in-transit encryption

ElastiCache allows encryption settings for communication (TLS). In CDK, setting transitEncryptionMode to required allows only TLS communication.

    const cache = new elasticache.CfnReplicationGroup(this, 'Resource', {
      // omit
      transitEncryptionEnabled: true,
      transitEncryptionMode: 'required', // Allow only TLS communication. 'preferred' allows both TLS and non-TLS.
      // omit
      authToken: secret.secretValue.unsafeUnwrap(),
    });

When setting connection information for Langfuse Web/Server in environment variables, there are two ways:

Use REDIS_CONNECTION_STRING
Set REDIS_HOST, REDIS_PORT, REDIS_AUTH

https://langfuse.com/self-hosting/infrastructure/cache#configuration

When only TLS communication is allowed, currently, method 1 (REDIS_CONNECTION_STRING) must be used. Method 2 will not connect and won't output an error (I got stuck on this).

Here's the English translation:

(Updated February 15, 2025)
As of Langfuse v3.28.0 and later, TLS communication settings can now be configured using "2. Set REDIS_HOST, REDIS_PORT, REDIS_AUTH".

This can be enabled by setting the REDIS_TLS_ENABLED environment variable to true in the Langfuse Web/Worker environment. Since this eliminates the need to generate a REDIS_CONNECTION_STRING value, method 2 is generally preferred.

I have also updated the CDK implementation to use method 2. For specific implementation details, please refer to the environment variable configuration section below:

https://github.com/mazyu36/langfuse-with-aws-cdk/blob/main/lib/constructs/services/common-environment.ts#L57.

For versions prior to v3.28.0, you will need to use the REDIS_CONNECTION_STRING configuration as described in the following sections.

For TLS-only communication, set REDIS_CONNECTION_STRING with rediss://....

This is due to the implementation in Langfuse's packages/shared/src/server/redis/redis.ts.

Langfuse uses ioredis, and the instance creation is defined as follows:

  const instance = env.REDIS_CONNECTION_STRING
    ? new Redis(env.REDIS_CONNECTION_STRING, {
        ...defaultRedisOptions,
        ...additionalOptions,
      })
    : env.REDIS_HOST
      ? new Redis({
          host: String(env.REDIS_HOST),
          port: Number(env.REDIS_PORT),
          password: String(env.REDIS_AUTH),
           // No TLS configuration
          ...defaultRedisOptions,
          ...additionalOptions,
        })
      : null;

When using REDIS_HOST, individual host settings are configured, but the tls property is necessary for TLS communication (reference: ioredis documentation)

const redis = new Redis({
  host: "redis.my-service.com",
  tls: {}, // TLS configuration is necessary
});

However, this option is currently not provided by Langfuse. Therefore, using REDIS_HOST etc. makes TLS communication impossible.

On the other hand, ioredis allows TLS connection by defining connection information starting with rediss.

const redis = new Redis("rediss://redis.my-service.com");  // Specify TLS with rediss

In Langfuse, using REDIS_CONNECTION_STRING enables TLS communication.

Set noeviction

The parameter maxmemory-policy must be set to noeviction (no eviction setting).
This ensures that queue jobs are not removed from the cache.

You must set the parameter maxmemory-policy to noeviction to ensure that the queue jobs are not evicted from the cache.

[https://langfuse.com/self-hosting/infrastructure/cache#deployment-options:embed:cite]

To prevent jobs from accumulating infinitely, retry and job retention on failure are considered (example in IngestionQueue).

    IngestionQueue.instance = newRedis
      ? new Queue<TQueueJobTypes[QueueName.IngestionQueue]>(
          QueueName.IngestionQueue,
          {
            connection: newRedis,
            defaultJobOptions: {
              removeOnComplete: true,  // Remove job after success
              removeOnFail: 100_000, // Maximum number of failed job retentions
              attempts: 5,  // Number of retries
              backoff: {  // Retry with exponential backoff
                type: "exponential",
                delay: 5000,
              },
            },
          },
        )
      : null;

noeviction can be set in the parameter group in CDK:

    /**
     * We must set the parameter `maxmemory-policy` to `noeviction` to ensure that the queue jobs are not evicted from the cache.
     * @see https://langfuse.com/self-hosting/infrastructure/cache#deployment-options
     */
    const parameterGroup = new elasticache.CfnParameterGroup(this, 'RedisParameterGroup', {
      cacheParameterGroupFamily: 'valkey8',
      description: 'Custom parameter group for Langfuse ElastiCache',
      properties: {
        'maxmemory-policy': 'noeviction',  // here
      },
    });

Cluster mode and ElastiCache Serverless are not supported

At the time of writing, Langfuse Web/Worker does not support Redis/Valkey cluster mode. Therefore, ElastiCache cluster mode cannot be used.

Langfuse handles failovers between read-replicas, but does not support Redis cluster mode for now, i.e. there is no sharding support.

https://langfuse.com/self-hosting/infrastructure/cache#managed-redisvalkey-by-cloud-providers

This means ElastiCache Serverless also cannot be used currently. AWS documentation also states that ElastiCache Serverless runs in cluster mode.

ElastiCache Serverless runs Valkey, Memcached, or Redis OSS in cluster mode and is only compatible with clients that support TLS.

https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.corecomponents.html

If you try to use ElastiCache Serverless, you'll frequently encounter CROSSSLOT errors:

2025-02-04T09:09:03.525Z warn Redis connection error: CROSSSLOT Keys in request don't hash to the same slot

The cause is that keys are being operated on that hash to different slots.

In cluster mode, keys are calculated and stored in hash slots. When operating multiple keys, they must be in the same hash slot. A detailed explanation can be found in this article:

https://dev.to/inspector/resolved-crossslot-keys-error-in-redis-cluster-mode-enabled-3kec

Since the current implementation does not consider cluster mode, these errors occur. This cannot be resolved through infrastructure settings.

Usage

Please refer to the README for details. Below is a simple deployment method and the flow for verifying the operation.

CDK Configuration and Deployment

The CDK implementation allows parameters to be defined for each environment (dev/stg/prod), and the deployment target environment is specified as a context. The general deployment method is as follows:

Clone the repository and install the necessary libraries with npm ci.
Set the parameters for the corresponding environment in bin/app-config and lib/stack-config.
Deploy with npx cdk deploy --context env=ENV_NAME.

The deployment takes about 20 minutes. The URL is output in the Outputs.

 ✅  LangfuseWithAwsCdkStack-dev

✨  Deployment time: 1238.58s

Outputs:

# omit

LangfuseWithAwsCdkStack-dev.LangfuseURL = https://langfuse.example.com

# omit

✨  Total time: 1247.3s

Initial Setup

After opening the URL, first sign up by entering your email address and other details, then click Sign up.

If Aurora Serverless v2 Zero scaling is enabled and the database is in an idle state, you may encounter a DB error like the following (the error message is hard to read due to the color...). At this point, the DB will start to wake up from idle, so wait for a while (about 15-30 seconds) and then click Sign up again.

First, create an Organization by clicking New Organization.

Set the Organization name and click Create.

If you want to add members, configure them here. I will not set this up here, so click next.

Next, set the Project name. After setting, click create.

Finally, select API Keys and click Create new API Keys.

Once the Secret key and Public key are issued, make a note of them. This completes the initial setup.

Operation Verification

Perform operation verification using the issued API keys. Here, we use curl.

First, set the API keys and hostname as environment variables in an appropriate environment.

export LANGFUSE_SECRET_KEY="YOUR_SECRET_KEY"
export LANGFUSE_PUBLIC_KEY="YOUR_PUBLIC_KEY"
export LANGFUSE_HOST="YOUR_LANGFUSE_URL"

Next, execute the ingestion (trace ingestion) API with curl. Note that the content is dummy, so there is almost no actual data.

curl -X POST "$LANGFUSE_HOST/api/public/ingestion" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "batch": [
      {
        "type": "trace-create",
        "id": "'$(uuidgen)'",
        "timestamp": "'$(date -u +"%Y-%m-%dT%H:%M:%S.000Z")'",
        "metadata": null,
        "body": {
          "id": "'$(uuidgen)'",
          "name": "test",
          "timestamp": "'$(date -u +"%Y-%m-%dT%H:%M:%S.000Z")'",
          "public": false
        }
      }
    ],
    "metadata": null
  }'

If a 201 response is output, it is working correctly.

{
  "successes": [
    {
      "id": "523EFC5E-8BAC-485D-9CC3-C049B5F64FA4",
      "status": 201
    }
  ],
  "errors": []
}

You can check the traces in the browser under Tracing -> Traces.
If the executed data is included, the operation verification is complete 🎉

Troubleshooting Approach

I will summarize the actual troubleshooting steps I took during the construction.

When I was building, the Langfuse application started, but I struggled with issues such as ingestion (trace ingestion) and trace confirmation on the screen.

There is documentation available for troubleshooting during self-hosting.
https://langfuse.com/self-hosting/troubleshooting

Especially if ingestion is not working, the Missing Events After POST /api/public/ingestion section lists points to check.

Ingestion is done by implementing a program using the SDK or executing the API with the curl command mentioned earlier to ingest traces into Langfuse. If you cannot confirm it in the browser, the process is failing somewhere.

The general flow of ingestion is as follows:

When received via the web, upload the event to the bucket and set a job in the cache queue.
The worker picks up the job from the cache queue, reads the event from the bucket, and finally stores it in ClickHouse.
When accessed from the browser, the web retrieves the data from ClickHouse and displays it.

※You can get a feel for this by looking at packages/shared/src/server/ingestion/processEventBatch.ts and worker/src/queues/ingestionQueue.ts.

Based on the above process flow and documentation, I performed troubleshooting as follows:

Check the logs of the Web and Worker.
Check the bucket (S3).
Check the cache (ElastiCache).
Check ClickHouse.

I will describe what I mainly did in order.

1. Check the Logs of the Web and Worker

First, check the logs of the Web and Worker as the logs of Langfuse itself. If there is an error due to infrastructure layer configuration issues, the cause can be identified here.

However, at the time of writing, there were not many logs output, and the cause was often difficult to identify due to the asynchronous nature of the queue, etc.

2. Check the Bucket (S3)

When ingestion is performed, the event JSON is stored in S3. If the JSON does not exist in S3, it is likely that the connection between the Web and S3 is not working properly. Check the NW and permission settings.

3. Check the Cache (ElastiCache)

If the JSON is stored in S3, next check if the job is being stored and processed in ElastiCache.

Use valkey-cli for connection (if using Redis, redis-cli is better). The setup method is summarized in the following documentation.

https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/connect-tls.html

After setting up, connect to the cache with the following command.

$ valkey-cli -h Primary or Configuration Endpoint --tls -a 'your-password' -p 6379

Once connected, tail the logs with MONITOR.

$ MONITOR

If OK appears after executing the command and no logs are displayed, it is likely that the Web/Worker cannot connect to ElastiCache. Recheck the connection settings. If connected, some logs will be displayed.

Also, check the ingestion-queue as follows. If the Web and ElastiCache are connected and jobs are being added to the queue, something like the following should be displayed.

$ KEYS bull:ingestion-queue:*

# Result
1) "bull:ingestion-queue:2"
2) "bull:ingestion-queue:active"
3) "bull:ingestion-queue:3"
4) "bull:ingestion-queue:stalled"
5) "bull:ingestion-queue:3:lock"
6) "bull:ingestion-queue:meta"
7) "bull:ingestion-queue:id"
8) "bull:ingestion-queue:events"
9) "bull:ingestion-queue:1"
10) "bull:ingestion-queue:1:lock"
11) "bull:ingestion-queue:2:lock"
12) "bull:ingestion-queue:stalled-check"

When I was troubleshooting, the above was the case, but the traces were not displayed in the browser, so I dug deeper.

First, check if there are any failed jobs with the following command.

$ ZRANGE bull:ingestion-queue:failed 0 -1 WITHSCORES

# Result
1) "2"
2) "1738464474985"
3) "3"
4) "1738464474986"
5) "1"
6) "1738464474996"

Jobs with IDs 1~3 have failed. Let's look at the details of job ID 2.

$ HGETALL bull:ingestion-queue:2

# Result
1) "data"

# ... omit

17) "stacktrace"
18) "[\"Error: upstream request timeout\\n    at parseError (/app/node_modules/.pnpm/@clickhouse+client-common@1.4.0/node_modules/@clickhouse/client-common/dist/error/parse_error.js:37:39)\\n    at ClientRequest.onResponse 

# ...omit

Then, the error cause is upstream request timeout, and it is happening in the Clickhouse related part.
Therefore, it is expected that a timeout occurred when the Worker tried to request ClickHouse while processing the job.

In the issue I encountered, the cause was ultimately a missing security group setting. The inbound rule of the Fargate Service's security group for ClickHouse did not allow connections from the Worker's Fargate Service, so communication was not possible.

In this case, the Worker's logs did not output any errors, so the error content could not be identified without investigating ElastiCache.

4. Check ClickHouse

If everything seems fine on ElastiCache, check if data is stored in ClickHouse. First, connect to ClickHouse with ECS Exec.

aws ecs execute-command --cluster cluster-name \
    --task task-id \
    --container container-name \
    --interactive \
    --command "/bin/sh"

Then start the client with clickhouse-client. The following indicates a successful start.

# clickhouse-client
ClickHouse client version 25.1.2.3 (official build).
Connecting to localhost:9000 as user clickhouse.
Connected to ClickHouse server version 25.1.2.

Then, check if data is stored with SQL, just like with RDB.

USE default;
SHOW TABLES;
SELECT * FROM traces ORDER BY created_at DESC LIMIT 10;

Based on the results of checking the contents, you can suspect the following. However, in such cases, errors are likely to be output in the Web or Worker logs, so the issue might be identified at the 1. Check the Logs of the Web and Worker stage.

No data in ClickHouse: It is likely that the job processing results could not be stored, so the communication between the Worker and ClickHouse is suspicious.
Data exists in ClickHouse: If data exists but is not displayed in the browser, the communication between the Web and ClickHouse is suspicious.

Conclusion

If you find any issues or have suggestions for additions to the CDK project, please let me know!
I plan to work on CloudFront's VPC Origin support at some point in the near future.

ElastiCache Serverless: L2 CDK Construct Released in open-constructs

Matsuda — Fri, 17 Jan 2025 02:19:31 +0000

I contributed an L2 Construct for ElastiCache Serverless to open-constructs.

GitHub Repository: open-constructs/aws-cdk-library

I created this construct because ElastiCache Serverless seems likely to see increased demand, with features like Valkey support allowing usage from around $6 per month and Valkey 8.0 enabling faster scaling. It was also something I personally needed.

Get Started with Amazon ElastiCache for Valkey

Amazon ElastiCache version 8.0 for Valkey brings faster scaling and improved memory efficiency

In this article, I'll introduce how to use it.

Installing open-constructs

After creating your CDK project, simply run npm install:

$ npx cdk init --language=typescript
$ npm install @open-constructs/aws-cdk

Usage

Here's the general flow:

Create User for RBAC
Create UserGroup
Create ServerlessCache
Configure IAM policies (only if using IAM authentication)
Other useful features (connections, metrics)

Note that RBAC is only supported for Valkey and Redis. Therefore, steps 1 and 2 are not necessary when using Memcached.

ElastiCache RBAC Documentation

The following examples assume using Valkey.

1. Creating RBAC Users

First, create users for cache access control. There are three types: IAM authentication, password authentication, and no password authentication.

For IAM authentication users, use the IamUser class. With IAM authentication, the user ID and username must match, so the Construct only allows setting the ID (the username is automatically set to the same value as the user ID).

The accessString indicates permissions following Redis syntax. In this example, it allows all operations on all keys. Please refer to the documentation for details.

Access String Documentation

const user = new IamUser(this, 'User', {
  userId: 'my-iam-user',
  accessString: 'on ~* +@all',
});

For password authentication users, use the PasswordUser class. You can set up to two passwords, and the user ID and username can be set separately:

const user = new PasswordUser(this, 'User', {
  userId: 'my-user-id',
  userName: 'my-user-name',
  accessString: 'on ~* +@all',
  passwords: [
    cdk.SecretValue.unsafePlainText('strongPassword123'),
    cdk.SecretValue.unsafePlainText('backupPassword456'),
  ],
});

For users without password authentication, use the NoPasswordRequiredUser class. Note that this provides weaker security and should be used carefully:

const user = new NoPasswordRequiredUser(this, 'User', {
  userId: 'my-user-id',
  userName: 'my-user-name',
  accessString: 'on ~* +@all',
});

Important Note: The default User

This is perhaps the most complex aspect to understand.

ElastiCache requires that a user with the username default must be included in the UserGroup.

RBAC Usage Documentation

A user with the user ID default is created by AWS by default (no password authentication, full permissions). This user cannot be edited or deleted.

Additionally, usernames must be unique within a region.

Therefore, you need to provide a user with the username default and include it in the UserGroup using one of these methods:

Use the AWS-created default user (user ID and username are both default)
Create a new default user (user ID is not default, but username is default)

To use the former in your Construct, import it using the import method. For the latter, create a new default user with the username set to default. The latter approach is suitable if you want to customize the default user's permissions.

// Import the AWS-created default user
const defaultUser = NoPasswordRequiredUser.fromUserAttributes(this, 'DefaultUser', {
  // Both user ID and username must be 'default'
  userId: 'default',
  userName: 'default',
});

// Create a new default user
const newDefaultUser = new NoPasswordRequiredUser(this, 'NewDefaultUser', {
  // User ID must not be 'default'
  userId: 'new-default',
  // Username must be 'default'
  userName: 'default',
});

2. Creating UserGroup

Next, use the UserGroup class to create a logical group of users that will be associated with the cache:

declare const newDefaultUser: User;
declare const user: User;
declare const anotherUser: User;

const userGroup = new UserGroup(this, 'UserGroup', {
  // Register users. Must include a default user (username 'default')
  users: [newDefaultUser, user],
});

// You can also add users using the addUser method
userGroup.addUser(anotherUser);

3. Creating ServerlessCache

Finally, use the ServerlessCache class to create the cache and associate the user group:

declare const vpc: ec2.Vpc;
declare const userGroup: UserGroup;

const serverlessCache = new ServerlessCache(this, 'ServerlessCache', {
  engine: Engine.VALKEY,
  majorEngineVersion: MajorVersion.VER_8,
  serverlessCacheName: 'my-serverless-cache',
  vpc,
  // Associate the user group
  userGroup,
});

You can also configure CMK and snapshots. See the README for all available properties.

open-constructs ElastiCache README

Note that at the time of writing, the CacheUsageLimits property is not supported due to limitations in the aws-cdk-lib version used by open-constructs. I plan to add this property when it becomes available.

4. Configuring IAM Policies

If using IAM authentication users, you need to configure policies to allow connections:

IAM Authentication Documentation

The Construct provides a grantConnect method for easy permission setup:

declare const user: IamUser;
declare const serverlessCache: ServerlessCache;
declare const role: iam.Role;

// Grant "elasticache:Connect" permission
user.grantConnect(role);
serverlessCache.grantConnect(role);

5. Other Useful Features

You can easily configure security groups using connections:

declare const serverlessCache: ServerlessCache;
declare const instance: ec2.Instance;

// Allow connection from EC2 to cache on default port 6379
serverlessCache.connections.allowDefaultPortFrom(instance);

The Construct also makes it easy to create metrics and alarms. Dedicated methods are provided for BytesUsedForCache and ElastiCacheProcessingUnits metrics mentioned in the official documentation:

declare const serverlessCache: ServerlessCache;

// Get total bytes used for cache data (5-minute average)
const bytesUsedForCache = serverlessCache.metricBytesUsedForCache();

// Get total ElastiCacheProcessingUnits (ECPUs) consumed (5-minute average)
const elastiCacheProcessingUnits = serverlessCache.metricElastiCacheProcessingUnits();

// Create an alarm for ECPUs metric
elastiCacheProcessingUnits.createAlarm(this, 'ElastiCacheProcessingUnitsAlarm', {
  threshold: 50,
  evaluationPeriods: 1,
});

You can also define custom metrics using the metric method. For available serverless cache metrics, refer to:

declare const serverlessCache: ServerlessCache;

// Define CacheHits metric - total number of successful read operations (5-minute average)
const cacheHits = serverlessCache.metric('CacheHits', { statistic: 'sum' });

Conclusion

Thanks to the thorough review by the reviewers, I believe this has become a user-friendly Construct. Please give it a try!

DEV Community: Matsuda

Adding Cognito Authentication to Self-Hosted Langfuse with AWS CDK

Why Use an External IdP?

Configuring Cognito

Architecture

CDK Implementation

1. Creating a Stack for us-east-1

2. Creating Cognito UserPool, UserPool Client, and Domain

3. Configuring Cognito-related Information in Langfuse Web Environment Variables

Verification

Conclusion

Self-Hosting Langfuse v3 on AWS Using CDK

Architecture

Langfuse Application (Web, Worker)

ClickHouse - OLAP

PostgreSQL - OLTP

S3 - Blob Storage

Cache/Queue

Use REDIS_STRING when using in-transit encryption

Set noeviction

Cluster mode and ElastiCache Serverless are not supported

Usage

CDK Configuration and Deployment

Initial Setup

Operation Verification

Troubleshooting Approach

1. Check the Logs of the Web and Worker

2. Check the Bucket (S3)

3. Check the Cache (ElastiCache)

4. Check ClickHouse

Conclusion

ElastiCache Serverless: L2 CDK Construct Released in open-constructs

Installing open-constructs

Usage

1. Creating RBAC Users

Important Note: The default User

2. Creating UserGroup

3. Creating ServerlessCache

4. Configuring IAM Policies

5. Other Useful Features

Conclusion

1. Creating a Stack for `us-east-1`