DEV Community: Julian Michel

Automating secure CloudWatch Alarms with CDK Monitoring Constructs and encrypted SNS

Julian Michel — Sun, 30 Mar 2025 20:02:49 +0000

The AWS Well-Architected Framework describes how to implement cloud applications. For example, the Reliability Pillar advises architects and developers to set up proper monitoring, such as using CloudWatch Alarms. The Security Pillar defines how to build secure applications. Since both need to be combined in cloud applications, this blog post describes how to configure secure CloudWatch Alarms.

This blog post is based on AWS CDK as an Infrastructure as Code solution. The example project uses a Lambda function that will be enhanced with CloudWatch Alarms. The Alarms are created using cdk-monitoring-constructs, a construct that can automatically set up monitoring for CDK projects. To ensure a secure configuration, it uses cdk-nag, which throws error messages for insecure AWS resources.

Initial project setup

This example is based on a new TypeScript CDK project. First, install cdk-monitoring-constructs and cdk-nag to enable these features.

npm install cdk-monitoring-constructs
npm install cdk-nag

A simple Lambda function is added. For this Lambda function, we will create CloudWatch Alarms later.

new nodeLambda.NodejsFunction(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_22_X,
  entry: join(__dirname, '../src/my-lambda.ts'),
  handler: 'handler',
});

Monitoring configuration

The main setup is done for cdk-monitoring-constructs. The SNS Topic will send email notifications when an alarm is triggered.

const alarmTopic = new sns.Topic(this, 'AlarmTopic');
alarmTopic.addSubscription(new subscription.EmailSubscription('example@example.org'));

The MonitoringFacade defines the base configuration for cdk-monitoring-constructs. In this example, the CloudWatch dashboard isn't used, so createDashboard is set to false. The properties in alarmFactoryDefaults are set to very sensitive values - so alarms are triggered quickly.

const monitoring = new MonitoringFacade(this, "Monitoring", {
  dashboardFactory: new DefaultDashboardFactory(this, 'Dashboard', {
    createDashboard: false,
    dashboardNamePrefix: '',
  }),
  alarmFactoryDefaults: {
    alarmNamePrefix: 'MyApp',
    actionsEnabled: true,
    action: new SnsAlarmActionStrategy({
      onAlarmTopic: alarmTopic,
    }),
    evaluationPeriods: 1,
    datapointsToAlarm: 1,
  },
});

Now there are two options. It is possible to create alarms for a specific resource, such as the Lambda function.

monitoring.monitorLambdaFunction({
  lambdaFunction: myLambda,
  addFaultCountAlarm: {
    Critical: { maxErrorCount: 0, },
  },
});

Or, alarms can be created for all resource of a specific type created in the project. The configuration also defines the alarm metric and threshold.

monitoring.monitorScope(this, {
  lambda: {
    props: {
      addFaultCountAlarm: {
        Critical: { maxErrorCount: 0, },
      },
    },
  },
});

Check and improve security

It's time to verify that the application is secure. This is done be done using cdk-nag. This line in the CDK application (bin folder) enables the cdk-nag AWS Solutions Checks

cdk.Aspects.of(app).add(new AwsSolutionsChecks());

When running cdk synth, cdk-nag displays an error: The SNS theme does not require publishers to use SSL. So let's enable enforceSSL to have a secure configuration.

const alarmTopic = new sns.Topic(this, 'AlarmTopic', {
  enforceSSL: true,
});

To test the alarm, change the Lambda function so that it always fails, for example, upload invalid code. Then run the lambda function and wait for the alarm to become active. The alarm history will show state transitions and errors while the alarm action is triggered. The error messages indicates that it can't publish messages to the SNS topic.

Enabling enforceSSL has set a deny action for traffic without SSL encryption on the SNS Topic. CloudWatch can't publish messages anymore because no permissions were granted. So explicitly grant the publish permissions:

alarmTopic.grantPublish(new iam.ServicePrincipal('cloudwatch'));

After this change, SNS notifications will be sent to all subscribers. Let's increase the security even more. SNS Topics also support KMS encryption, so specify a KMS key as masterKey.

const alarmTopic = new sns.Topic(this, 'AlarmTopic', {
  enforceSSL: true,
  masterKey: kmsKey,
});

Alarm notifications will fail again, see the alarm history.

Similar to the previous error, permissions for the CloudWatch service are missing in the KMS Key. So let's grant the encrypt/decrypt permissions.

kmsKey.grantEncryptDecrypt(new iam.ServicePrincipal('cloudwatch'));

Another test shows that alarm notifications can be successfully sent to the KMS-encrypted SNS topic - everything works!

Triggering CloudWatch Alarms manually

Sometimes it is also helpful to trigger CloudWatch alarms manually. You don't need to implement a wrong lambda function to trigger an alarm, just use this CLI command.

aws cloudwatch set-alarm-state \
  --alarm-name "MyApp-MyFunction-Fault-Count-Critical" \
  --state-value ALARM --state-reason "Alarm test"

Instead of the ALARM value, you can also specify OK to reset the alarm.

Summary and lessons learned

You've learned how to create CloudWatch Alarms with cdk-monitoring-constructs - and how to implement a secure configuration:

Enforce SSL in the SNS topic and specify a KMS key.
Grant permissions for the CloudWatch service in the KMS topic and KMS key.

If you encounter problems with alarm actions, always check the alarm history in the CloudWatch console. It shows the error message that occurred when the alarm was delivered.

Embedding Q Business into existing applications

Julian Michel — Mon, 30 Dec 2024 16:42:25 +0000

Amazon Q Business has gotten some new features in the last few months. One of them is the ability to embed Q Business into other applications. This blog post gives some guidance on how this works and what to consider when adding Q Business to another application.

Why?

Often companies use other tools such as MS SharePoint or Confluence as their intranet. In case Q Business should not be treated as a separate tool, it can be integrated into existing applications. Users can then simply open a page that contains Q Business. For example, this screenshot shows the integration into a Confluence page.

How does it work?

First, you need to whitelist the site that embeds Q Business in the Q Business configuration. Open the new "Amazon Q embedded" configuration and add the URL of the site, such as https://julianmichel.de. Don't include the specific path and a trailing slash at the end.

Next, copy the URL of your Q Business application:

Then create an iFrame in the application and reference the URL you copied in the previous step. For example, in Confluence, search for the iFrame element.

Then paste the URL you copied earlier.

You can also do the same change in a plain HTML file by adding the iFrame element.

<iframe src="https://pgs3zrql.chat.qbusiness.us-east-1.on.aws/"
  title="QBusiness"></iframe>

For more details, see the Embedded Q Business section in the documentation.

Testing

You are now ready to open the Confluence page. Q Business will not load directly. Instead, it displays a login page. Press the Sign In-button.

It opens a new tab that handles the login process using the SSO capabilities in AWS IAM Identity Center. You can close this tab.

Next, it displays Q Business integrated into the Confluence page.

The integration into the custom HTML page also works. Again, the sign in page is displayed.

Once you have completed the sign in process, you will see Q Business integrated into the page.

Error handling

What to do if embedding Q Business doesn't work? If you get this error and Q Business doesn't load at all, check the URL you configured before. It is probably incorrect.

If the Grant Permission button doesn't work, check the output in Web Developer Tools.

For example, I received this error when accessing the page in Google Chrome's incognito mode: requestStorageAccess: Request denied because the embedded site has never been interacted with as a top-level context.

Root cause: Third-party cookies are blocked in incognito mode. I had to enable them and it worked.

Also check the iFrame sandbox attribute settings if used in the iFrame.

Whitelisting URLs with IaC

Of course, it makes sense to configure the URL of the embedded web packages using Infrastructure as Code. In an earlier post, I described how to deploy Q Business using AWS CDK. In the WebExperience resource, you can specify an array of URLs in the origins property. Unfortunately, the property isn't called embedded URLs or anything like that, so it's different from the naming in the AWS console.

new q.CfnWebExperience(this, 'WebExperience', {
  applicationId: application.attrApplicationId,
  roleArn: webExperienceRole.roleArn,
  origins: [
    'https://jumic-reinvent.atlassian.net',
    'https://julianmichel.de',
  ],
});

Summary

Embedded Q Business is a great feature for integrating Q Business into existing applications. From my point of view, it improves the user experience when users can access it in familiar applications.

Unfortunately, you have to log in to Q Business first. But this is not a design decision by AWS - it is related to web security standards.

If Q Business Embedded doesn't work out of the box, you can use the web developer tools to debug it.

How to create an AWS account without a debit/credit card (in Germany)

Julian Michel — Fri, 04 Oct 2024 20:26:11 +0000

When you create a new AWS account, you must enter a debit or credit card as your default payment method. You cannot create an AWS account without a debit or credit card, such as Mastercard or Visa. As of the end of July this year, it is now possible to sign up for a new AWS account in Germany without using a debit or credit card. Instead, you will need to link your AWS account to your bank account (see official announcement).

Why do we need this feature?

Credit cards are not widely used in Germany. Many people prefer the German girocard. Even in the young age group (18-27 years), only around 50 percent had a credit card in 2020.

Telling people in Germany that they should sign up for an AWS account could be difficult. Students can't sign up because they don't have a credit card, so they won't be trained in AWS.

I suppose this was one of the reasons why AWS now allows AWS accounts to be created with a bank account in Germany. For now, this feature is not available in other countries.

Why should you know about this (in Germany)?

If you are teaching AWS to German students or other people, you can tell them that this alternative payment method exists. This article shows how to set up an AWS account without a debit or credit card - you will understand the process and be able to tell others about it.

Select the payment method

Sign up for your AWS account as usual. On the Billing Information page, select Germany as your billing country. Once you select Germany, the form will display the additional payment method type "Bank Account". Select this option.

The form shows additional information about how AWS links your bank account. It uses a service called TrueLayer.

You are asked to open TrueLayer.

Link your bank account

Next, select your German bank from the drop-down list. You can't enter a foreign bank here.

The TrueLayer service explains what information is shared. TrueLayer also reads your account balance - but doesn't share it with AWS.

You will be directed to your bank account. Log in and confirm that TrueLayer is allowed to read the requested data.

Continue the registration process

After confirming access to your bank details, the information will be transferred to AWS and you will see the last four digits of your bank account number on the registration form. This confirms that your bank account has been successfully linked.

Everything else is the same as using a debit or credit card. Once your AWS account is created, you can also verify your bank account in the payment preferences.

Summary

It works - you can easily link your bank account. I was able to sign up for an AWS account without a debit or credit card. The process is straightforward, there is no time-consuming verification like with some other services.

How to publish custom cdk-nag rules and rule packs with Projen

Julian Michel — Mon, 23 Sep 2024 20:27:56 +0000

cdk-nag is a small tool for checking AWS CDK applications for (security) best practices. It provides rules and rule packs that can be applied to a CDK application. The rules are evaluated during cdk synth, which has the benefit of providing feedback to developers early in the development cycle. Similar capabilities can be implemented using the AWS Security Hub. However, cdk-nag has the advantage of finding issues before deployment.

cdk-nag provides five rules packs that can be used right out of the box:

AWS Solutions
HIPAA Security
NIST 800-53 rev 4
NIST 800-53 rev 5
PCI DSS 3.2.1

If one of these rule packs meets your organization's needs, you can directly use it. If you have custom rules, you can define your own rules and rule packs to make them available to all your projects. The documentation shows how to create them, but there is no information on how to publish them. So this blog post shows how to publish custom cdk-nag rules and rule packs.

Preconditions

You must have access to GitHub in order to create and use GitHub repositories. The Projen features used in this blog post are only available on GitHub.

To publish the cdk-nag ruleset, you must use an npm registry such as npmjs.com. In this blog post, the rule pack is published as a private npm package, which requires a paid npmjs account. If you are using a self-hosted npm registry, this is not a strict requirement.

Create a new Projen project

Projen is another open source project - it manages project configurations. Similar to AWS CDK, which creates CloudFormation templates, a Projen project is synthesized into project configurations, including GitHub actions, linter configuration, and so on.

cdk-nag rules and rule packs are published as a software library, so Projen provides everything needed to publish them. Also, cdk-nag itself uses Projen to publish new releases. The Projen type awscdk-construct contains everything needed to manage the project, including the GitHub workflows to publish new releases. With jsii it is also possible to publish releases in other programming languages like Java or Python. To create the new project, run this command:

npx projen new awscdk-construct

Add cdk-nag dependency

First, add cdk-nag as new dependency in file .projenrc.ts. Insert the following line:

peerDeps: ['cdk-nag'],

Then run npx projen to install it.

Implement a custom rule and rule pack

Now add one or more custom rules and a rule pack. See the documentation and the cdk-nag repository on GitHub for more information. As an example, use the following rule and rule pack.

Implement a rule that checks if SSE is enabled in S3 buckets. This is just an example - it probably makes more sense to use KMS encryption instead. Add a new file src/rules/S3DefaultEncryptionSSE.ts and paste the content:

import { parse } from 'path';
import { CfnResource, Stack } from 'aws-cdk-lib';
import { CfnBucket } from 'aws-cdk-lib/aws-s3';
import { NagRuleCompliance, NagRules } from 'cdk-nag/lib';

/**
 * S3 Buckets are encrypted S3 SSE encryption
 * @param node the CfnResource to check
 */
export default Object.defineProperty(
  (node: CfnResource): NagRuleCompliance => {
    if (node instanceof CfnBucket) {
      if (node.bucketEncryption == undefined) {
        return NagRuleCompliance.NON_COMPLIANT;
      }
      const encryption = Stack.of(node).resolve(node.bucketEncryption);
      if (encryption.serverSideEncryptionConfiguration == undefined) {
        return NagRuleCompliance.NON_COMPLIANT;
      }
      const sse = Stack.of(node).resolve(
        encryption.serverSideEncryptionConfiguration,
      );
      for (const rule of sse) {
        const defaultEncryption = Stack.of(node).resolve(
          rule.serverSideEncryptionByDefault,
        );
        if (defaultEncryption == undefined) {
          return NagRuleCompliance.NON_COMPLIANT;
        }
        const sseAlgorithm = NagRules.resolveIfPrimitive(
          node,
          defaultEncryption.sseAlgorithm,
        );
        if (sseAlgorithm != 'AES256') {
          return NagRuleCompliance.NON_COMPLIANT;
        }
      }
      return NagRuleCompliance.COMPLIANT;
    } else {
      return NagRuleCompliance.NOT_APPLICABLE;
    }
  },
  'name',
  { value: parse(__filename).name },
);

Also add a custom rule pack. This one contains a reference to the new rule and an existing rule from cdk-nag. Paste the contents into file src/MyCustomChecks.ts:

import { CfnResource } from 'aws-cdk-lib';
import { NagMessageLevel, NagPack, NagPackProps, rules } from 'cdk-nag';
import { IConstruct } from 'constructs';
import S3DefaultEncryptionSSE from './rules/S3DefaultEncryptionSSE';

export class MyCustomChecks extends NagPack {
  constructor(props?: NagPackProps) {
    super(props);
    this.packName = 'MyCustom';
  }
  public visit(node: IConstruct): void {
    if (node instanceof CfnResource) {

      this.applyRule({
        info: 'SSL not enabled in this bucket.',
        explanation: 'SSL must be enabled to encrypt traffic.',
        level: NagMessageLevel.ERROR,
        rule: rules.s3.S3BucketSSLRequestsOnly,
        node: node,
      });

      this.applyRule({
        info: 'SSE encryption not set for this bucket.',
        explanation: 'SSE encryption must be enabled to encrypt files in this bucket.',
        level: NagMessageLevel.ERROR,
        rule: S3DefaultEncryptionSSE,
        node: node,
      });

    }
  }
}

Replace the existing content in src/index.ts and export the two files created before.

export * from './MyCustomChecks';
export * from './rules/S3DefaultEncryptionSSE';

It is also recommended to add some test cases. First remove the generated default example test case by deleting file test/hello.test.ts. For the rule, add a new test case in file test/S3DefaultEncryptionSSE.test.ts

import { App, Stack } from 'aws-cdk-lib';
import { CfnBucket } from 'aws-cdk-lib/aws-s3';
import S3DefaultEncryptionSSE from '../src/rules/S3DefaultEncryptionSSE';

test('Non-Complient Bucket', () => {
  const app = new App();
  const stack = new Stack(app, 'Stack');
  const bucket = new CfnBucket(stack, 'Bucket', {
  });
  expect(S3DefaultEncryptionSSE(bucket)).toBe('Non-Compliant');
});

test('Complient Bucket', () => {
  const app = new App();
  const stack = new Stack(app, 'Stack');
  const bucket = new CfnBucket(stack, 'Bucket', {
    bucketEncryption: {
      serverSideEncryptionConfiguration: [
        {
          serverSideEncryptionByDefault: {
            sseAlgorithm: 'AES256',
          },
        },
      ],
    },
  });
  expect(S3DefaultEncryptionSSE(bucket)).toBe('Compliant');
});

To complete this step, run npx projen build to build and test the code. If successful, proceed to the next step.

Release to npmjs

To import the rule pack into another project, it must be published to npmjs. This will be done by Github actions generated by Projen. To release an internal library, add a scope to property name in .projenrc.ts. In my case, I added @jumic.

name: '@jumic/cdk-nag-custom-rules',

Also, enable npm release and set access level to RESTRICTED.

releaseToNpm: true,
npmAccess: NpmAccess.RESTRICTED,

At npmjs.com, open Access Tokens in the navigation and generate a new access token.

Enter a token name such as GitHub.

Define packages and scope.

As a result, npmjs will display the generated access token. This needs to be copied to GitHub.

On GitHub, go to the repository settings for your repository. Create a new secret.

Enter the name NPM_TOKEN, which is the default name Projen expects. Then paste the access token.

Once you have completed these configuration steps, commit your source code to the repository. The GitHub actions will start to build the project and publish it to npmjs.

Check the status on GitHub and make sure that the workflow is running successfully.

Also, go to the npmjs packages and check if the release is available.

If both checks pass, the custom cdk-nag rule package has been successfully released.

Adding your rule pack to a new CDK project

It's time to test the new rule pack. Create a new CDK project or use an existing CDK project. As this library was published as a private package, login to npmjs to access private packages:

npm login

Now you can install the dependency to your project:

npm install @jumic/cdk-nag-custom-rules

In the bin folder, add the new rule pack to the CDK application.

cdk.Aspects.of(app).add(new MyCustomChecks())

Then execute cdk synth and it will show all invalid resources.

In case that all checks are passed, you will see the standard CDK output.

Summary

Publishing cdk-nag rule packs with Projen works great. It is a very useful combination of tools for publishing custom cdk-nag rule packs. Once configured, changes are continuously published to the registry.

Using CDK in multiple programming languages? Just add more jsii targets to .projenrc.ts to publish packages for Python, Java, ... (see Projen documentation).

Full source code on GitHub

AWS SQS as an alternative to traditional application integration middleware

Julian Michel — Sun, 15 Sep 2024 22:28:55 +0000

Application integration is a common scenario in IT landscapes. A source system sends data that is consumed by a target system. There are various integration middleware that provide integration capabilities for third-party software. They implement features such as data quality checks or data conversation before it is sent to the target system.

Limitations of traditional integration middleware

As always, standard software is great for the use cases for which it was designed. Before working on AWS projects, I integrated shop floor data with ERP systems using traditional middleware. This software solution met all the business requirements. However, problems appeared when large amounts of data needed to be transferred. This particular software solution wasn't built for high traffic.

It was possible to implement some workarounds for this problem. However, it always felt like a workaround. For example, it was possible to create multiple parallel queues to process data in parallel. Since this was not a standard feature, each configuration had to be implemented multiple times. It worked, but it was not perfect.

How my first AWS training made an impact

When I took my first AWS trainings (AWS Cloud Practitioner and AWS Developer Associate), I was really impressed with the capabilities that AWS offers. One of them was the strong scalability capabilities - the feature that caused problems in some past projects. Since AWS SQS is a managed message queue, I thought it would be a good fit for my use case.

Implementation of a similar project with AWS

In one of my next projects, I had the opportunity to implement a similar project using AWS tools. A consumer sends messages to an SQS queue. An AWS Lambda function processes the data and posts it to an ERP system.

From a developer's perspective, this is not complex. However, it worked very well without the scalability issues. Multiple messages are processed in parallel. It scales very quickly when there are message spikes. If the target system isn't available, messages can be retried later. Basically, it is exactly the use case that SQS was built for - and it is high performance and resilient.

How to address complex requirements

For some use cases, special requirements had to be implemented.

First, for some scenarios, messages should be processed in order. SQS FIFO queues support this feature. By specifying the same message group ID, it is possible to process messages of that message group ID in sequence. In Smart Manufacturing applications this can be a production plant, a shop order or something similar.

Second, there was a need to limit the number of messages sent to the target system in parallel. AWS scales quickly, but some target systems have limited capacity. This is also possible in AWS. There is an attribute called maxConcurrency that limits the number of concurrent Lambda invocations when processing an SQS queue. When using CDK, this attribute can be specified when subscribing the Lambda function to the SQS queue:

lambdaFunction.addEventSource(new sources.SqsEventSource(queue, {
  maxConcurrency: 2,
}));

Summary

In comparison to trends like GenAI, integration applications are nothing really impressive. On the other hand, this example shows that basic services like SQS can be the foundation for IT applications and IT landscapes. IT applications can be cloud applications - or on-premises applications enhanced by cloud integration services.

Deploy Amazon Q Business with AWS CDK - example and best practices

Julian Michel — Sun, 11 Aug 2024 20:52:26 +0000

Amazon Q Business, like almost all other AWS services, can be configured using the AWS Console. To deploy it in production environments, you probably want to use Infrastructure as Code to avoid manual configuration tasks. This blog post shows how to deploy Amazon Q Business using AWS CDK, including a Confluence integration as a data source. For some basic understanding, take a look at the previous blog post where I configured everything manually and also described the prerequisites such as AWS IAM Identity Center.

Currently, only L1 constructs based on CloudFormation resource types are supported. Using the L1 construct is more complicated. However, I implemented the CDK code based on my previous manual configuration, which was very helpful.

You can also find the full source code in my GitHub repo.

Q Business CloudFormation resource types overview

CloudFormation provides six resource types for deploying Q Business:

AWS::QBusiness::Application - The main application
AWS::QBusiness::Index - The index that contains the knowledge imported from the data sources
AWS::QBusiness::Retriever - The retriever that reads data from the index for Q Business conversations
AWS::QBusiness::WebExperience - The frontend application
AWS::QBusiness::DataSource - Configuration of data sources
AWS::QBusiness::Plugin - Q Business plugins (e.g. Jira write access) - not used in this example

Most resources require IAM roles. In this blog post, we will first create the IAM role for each resource and then create the Q-Business resource.

Getting started with the AWS CDK project

This project is based on a new AWS CDK TypeScript project. In my sample project, I have added configuration variables for the AWS IAM Identity Center ARN (identityCenterInstanceArn) and the URL of the Confluence data source (confluenceHostUrl). Feel free to add more/other configuration variables - this is sufficient for my sample project.

File bin/cdk-q-business-confluence.ts contains values for these configuration options.



new CdkQBusinessConfluenceStack(
  app,
  'CdkQBusinessConfluenceStack',
  {
    identityCenterInstanceArn:
      'arn:aws:sso:::instance/ssoins-123467890abc',
    confluenceHostUrl: 'https://my-domain.atlassian.net/',
  },
);

These values are passed to the CDK Stack that is deployed (see filename lib/cdk-q-business-confluence-stack.ts).



interface QBusinessStackProps extends cdk.StackProps {
  identityCenterInstanceArn: string;
  confluenceHostUrl: string;
}

export class CdkQBusinessConfluenceStack extends cdk.Stack {
  constructor(
    scope: Construct,
    id: string,
    props: QBusinessStackProps,
  ) {
    super(scope, id, props);

Confluence authentication credentials

Q Business reads data from Confluence to add it to the index. The credentials are stored in a Secret Manager Secret, which is created first. The encryption key is only used for the secret - feel free to use it to encrypt data in Q Business itself (my cluster is currently unencrypted as it only contains test data generated for this purpose).

The source code doesn't contain the actual secrets - they have to be maintained manually in the Secrets Manager in the AWS console. If you also want to deploy your secrets/passwords with AWS CDK, use the cdk-sops-secrets library, which supports this in a secure way.



const encryptionKey = new kms.Key(
  this,
  'QBusinessKey',
  {
    alias: 'QBusinessKey',
    pendingWindow: cdk.Duration.days(7),
    removalPolicy: cdk.RemovalPolicy.DESTROY, // for testing only
  },
);

const secret = new secretsmanager.Secret(
  this,
  'Secret',
  {
    secretObjectValue: {
      username: cdk.SecretValue.unsafePlainText(
        'dummy value - please chanage manually after deployment',
      ),
      hostUrl: cdk.SecretValue.unsafePlainText(
        'dummy value - please chanage manually after deployment',
      ),
      password: cdk.SecretValue.unsafePlainText(
        'dummy value - please chanage manually after deployment',
      ),
    },
    encryptionKey,
  },
);

IAM policy and role for the Q Business application

The Q Business application mainly requires some CloudWatch Logs permissions.

How do I determine which permissions are required? I first configured Q Business manually. It provides options to automatically generate all required IAM roles. Then I transferred the permissions to the AWS CDK code. I recommend doing this for all IAM roles used in Q Business as long as no L2 Construct is available in AWS CDK.



const applicationPolicy = new iam.ManagedPolicy(
  this,
  'ApplicationPolicy',
  {
    statements: [
      new iam.PolicyStatement({
        sid: 'AmazonQApplicationPutMetricDataPermission',
        actions: ['cloudwatch:PutMetricData'],
        resources: ['*'],
        conditions: {
          StringEquals: {
            'cloudwatch:namespace': 'AWS/QBusiness',
          },
        },
      }),
      new iam.PolicyStatement({
        sid: 'AmazonQApplicationDescribeLogGroupsPermission',
        actions: ['logs:DescribeLogGroups'],
        resources: ['*'],
      }),
      new iam.PolicyStatement({
        sid: 'AmazonQApplicationCreateLogGroupPermission',
        actions: ['logs:CreateLogGroup'],
        resources: [
          `arn:aws:logs:${this.region}:${this.account}:log-group:/aws/qbusiness/*`,
        ],
      }),
      new iam.PolicyStatement({
        sid: 'AmazonQApplicationLogStreamPermission',
        actions: [
          'logs:DescribeLogStreams',
          'logs:CreateLogStream',
          'logs:PutLogEvents',
        ],
        resources: [
          `arn:aws:logs:${this.region}:${this.account}:log-group:/aws/qbusiness/*:log-stream:*`,
        ],
      }),
    ],
  },
);

const applicationRole = new iam.Role(
  this,
  'ApplicationRole',
  {
    assumedBy: new iam.ServicePrincipal(
      'qbusiness.amazonaws.com',
      {
        conditions: {
          StringEquals: {
            'aws:SourceAccount': this.account,
          },
          ArnLike: {
            'aws:SourceArn': `arn:aws:qbusiness:${this.region}:${this.account}:application/*`,
          },
        },
      },
    ),
    managedPolicies: [applicationPolicy],
  },
);

Q Business application

The Q Business application itself has only a few parameters. Always check which additional parameters are supported, e.g. you need to specify an encryption key if you want to encrypt data in Q Business.



const application = new q.CfnApplication(
  this,
  'Application',
  {
    displayName: 'CDK_QBusiness',
    identityCenterInstanceArn:
      props.identityCenterInstanceArn,
    roleArn: applicationRole.roleArn,
  },
);

Q Business index

The Q Business index stores the knowledge from the data sources. In the example, a starter index type is configured with one capacity unit.



const index = new q.CfnIndex(this, 'Index', {
  type: 'STARTER',
  capacityConfiguration: {
    units: 1,
  },
  applicationId: application.attrApplicationId,
  displayName: 'Index',
});

Q Business retriever

The Q Business retriever is used to read the data from the index when users interact with the Q Business. You must specify the index type (in this example NATIVE_INDEX instead of using Amazon Kendra) and a reference to the index itself.



new q.CfnRetriever(this, 'Retriever', {
  type: 'NATIVE_INDEX',
  applicationId: application.attrApplicationId,
  displayName: 'Retriever',
  configuration: {
    nativeIndexConfiguration: {
      indexId: index.attrIndexId,
    },
  },
});

IAM policy and role for the Q Business web experience

The Q Business web experience is the frontend for Q Business. It requires permissions to interact with Q Business and to use and create Q Business apps.



const principal = new iam.ServicePrincipal(
  'application.qbusiness.amazonaws.com',
  {
    conditions: {
      StringEquals: {
        'aws:SourceAccount': this.account,
      },
      ArnEquals: {
        'aws:SourceArn': application.attrApplicationArn,
      },
    },
  },
);

const webExperiencePolicy = new iam.ManagedPolicy(
  this,
  'WebExperiencePolicy',
  {
    statements: [
      new iam.PolicyStatement({
        sid: 'QBusinessConversationPermission',
        actions: [
          'qbusiness:Chat',
          'qbusiness:ChatSync',
          'qbusiness:ListMessages',
          'qbusiness:ListConversations',
          'qbusiness:DeleteConversation',
          'qbusiness:PutFeedback',
          'qbusiness:GetWebExperience',
          'qbusiness:GetApplication',
          'qbusiness:ListPlugins',
          'qbusiness:GetChatControlsConfiguration',
        ],
        resources: [application.attrApplicationArn],
      }),
      new iam.PolicyStatement({
        sid: 'QBusinessQAppsPermissions',
        actions: [
          'qapps:CreateQApp',
          'qapps:PredictProblemStatementFromConversation',
          'qapps:PredictQAppFromProblemStatement',
          'qapps:CopyQApp',
          'qapps:GetQApp',
          'qapps:ListQApps',
          'qapps:UpdateQApp',
          'qapps:DeleteQApp',
          'qapps:AssociateQAppWithUser',
          'qapps:DisassociateQAppFromUser',
          'qapps:ImportDocumentToQApp',
          'qapps:ImportDocumentToQAppSession',
          'qapps:CreateLibraryItem',
          'qapps:GetLibraryItem',
          'qapps:UpdateLibraryItem',
          'qapps:CreateLibraryItemReview',
          'qapps:ListLibraryItems',
          'qapps:CreateSubscriptionToken',
          'qapps:StartQAppSession',
          'qapps:StopQAppSession',
        ],
        resources: [application.attrApplicationArn],
      }),
    ],
  },
);

const webExperienceRole = new iam.Role(
  this,
  'WebExperienceRole',
  {
    assumedBy: principal,
    managedPolicies: [webExperiencePolicy],
  },
);

webExperienceRole.assumeRolePolicy?.addStatements(
  new iam.PolicyStatement({
    actions: ['sts:SetContext'],
    principals: [principal],
  }),
);

Q Business web experience

In the web experience, specify the IAM role. No additional configuration is required if you use the defaults.



new q.CfnWebExperience(this, 'WebExperience', {
  applicationId: application.attrApplicationId,
  roleArn: webExperienceRole.roleArn,
});

IAM policy and role for the Q Business Confluence Data Source

The data source configurations specify which source systems will be integrated with Q Business. For example, it requires permissions to create and delete documents in order to ingest the necessary data into Q Business.



const confluenceDataSourcePolicy =
  new iam.ManagedPolicy(
    this,
    'ConfluenceDataSourcePolicy',
    {
      statements: [
        new iam.PolicyStatement({
          sid: 'AllowsAmazonQToGetS3Objects',
          actions: ['s3:GetObject'],
          resources: ['arn:aws:s3:::bucket/*'],
          conditions: {
            StringEquals: {
              'aws:ResourceAccount': this.account,
            },
          },
        }),
        new iam.PolicyStatement({
          sid: 'AllowsAmazonQToGetSecret',
          actions: ['secretsmanager:GetSecretValue'],
          resources: [secret.secretArn],
        }),
        new iam.PolicyStatement({
          sid: 'AllowsAmazonQToDecryptSecret',
          actions: ['kms:Decrypt'],
          resources: [encryptionKey.keyArn],
          conditions: {
            StringLike: {
              'kms:ViaService': [
                'secretsmanager.*.amazonaws.com',
              ],
            },
          },
        }),
        new iam.PolicyStatement({
          sid: 'AllowsAmazonQToIngestDocuments',
          actions: [
            'qbusiness:BatchPutDocument',
            'qbusiness:BatchDeleteDocument',
          ],
          resources: [index.attrIndexArn],
        }),
        new iam.PolicyStatement({
          sid: 'AllowsAmazonQToIngestPrincipalMapping',
          actions: [
            'qbusiness:PutGroup',
            'qbusiness:CreateUser',
            'qbusiness:DeleteGroup',
            'qbusiness:UpdateUser',
            'qbusiness:ListGroups',
          ],
          resources: [
            application.attrApplicationArn,
            index.attrIndexArn,
            `${index.attrIndexArn}/data-source/*`,
          ],
        }),
      ],
    },
  );

const confluenceDataSourceRole = new iam.Role(
  this,
  'ConfluenceDataSourceRole',
  {
    assumedBy: new iam.ServicePrincipal(
      'qbusiness.amazonaws.com',
      {
        conditions: {
          StringEquals: {
            'aws:SourceAccount': this.account,
          },
          ArnEquals: {
            'aws:SourceArn':
              application.attrApplicationArn,
          },
        },
      },
    ),
    managedPolicies: [confluenceDataSourcePolicy],
  },

Q Business Confluence Data Source

The data source configuration must be repeated for each data source. It contains a display name, the role created earlier and a configuration (which is still empty).



new q.CfnDataSource(this, 'ConfluenceDataSource', {
  applicationId: application.attrApplicationId,
  displayName: 'ConfluenceDataSource',
  indexId: index.attrIndexId,
  configuration: {},
  roleArn: confluenceDataSourceRole.roleArn,
});

In my opinion, the configuration attribute is the most complex part of the documentation. It contains the configuration options that are specific to each data source. This means that each data source has its own set of attributes. The attributes are described in a JSON Schema (see CloudFormation documentation).

To get the specific JSON Schema, go to the list of available connectors and select the source system. In chapter "Using the API" you will find the specific JSON Schema, such as the Confluence JSON Schema.

I started writing the configuration with the help of the JSON Schema Validator to be sure that I will match the schema. For the basic fields I managed to specify all the information. When I got to the more complex repositoryConfigurations section with the field mappings, I was looking for a simpler solution.

I decided to use the CloudFormation IaC (infrastructure as code generator) generator for the first time. It can generate CloudFormation code based on manually created resources. I assumed that it would output the entire configuration including the field mappings. Unfortunately, the scan didn't show any Q Business resources because Q Business is not yet supported in the IaC generator.

Since the IaC generator doesn't work for this scenario, I tested the AWS CLI. First, go to the AWS console and open the data source you want to use as a template. The URL will contain three GUIDs that need to be specified in the CLI command:
https://us-east-1.console.aws.amazon.com/amazonq/business/applications/60935d06-f061-4b67-a7ef-6a3e748ed97b/indices/cab1f65f-1666-431c-836a-2e9f8d993c35/datasources/81675834-6e45-4698-aa46-f788b9809e92/details

Paste the GUIDs into the get-data-source command and run the command.



aws qbusiness get-data-source \
  --application-id 60935d06-f061-4b67-a7ef-6a3e748ed97b \
  --index cab1f65f-1666-431c-836a-2e9f8d993c35 \
  --data-source-id 81675834-6e45-4698-aa46-f788b9809e92

As a result, you get the full JSON configuration needed for the configuration (the screenshot only shows the first few lines).

Now I copied this configuration into the configuration attribute. I changed some of the values, such as a dynamic reference to the secret and the host URL. It looks complex, but it contains everything needed to deploy the data source.



new q.CfnDataSource(this, 'ConfluenceDataSource', {
  applicationId: application.attrApplicationId,
  displayName: 'ConfluenceDataSource',
  indexId: index.attrIndexId,
  configuration: {
    secretArn: secret.secretArn,
    syncMode: 'FORCED_FULL_CRAWL',
    enableIdentityCrawler: true,
    connectionConfiguration: {
      repositoryEndpointMetadata: {
        type: 'SAAS',
        hostUrl: props.confluenceHostUrl,
        authType: 'Basic',
      },
    },
    repositoryConfigurations: {
      space: {
        fieldMappings: [
          {
            dataSourceFieldName: 'itemType',
            indexFieldName: '_category',
            indexFieldType: 'STRING',
          },
          {
            dataSourceFieldName: 'url',
            indexFieldName: '_source_uri',
            indexFieldType: 'STRING',
          },
        ],
      },
      page: {
        fieldMappings: [
          {
            dataSourceFieldName: 'itemType',
            indexFieldName: '_category',
            indexFieldType: 'STRING',
          },
          {
            dataSourceFieldName: 'url',
            indexFieldName: '_source_uri',
            indexFieldType: 'STRING',
          },
          {
            dataSourceFieldName: 'author',
            indexFieldName: '_authors',
            indexFieldType: 'STRING_LIST',
          },
          {
            dataSourceFieldName: 'createdDate',
            indexFieldName: '_created_at',
            dateFieldFormat: "yyyy-MM-dd'T'HH:mm:ss'Z'",
            indexFieldType: 'DATE',
          },
          {
            dataSourceFieldName: 'modifiedDate',
            indexFieldName: '_last_updated_at',
            dateFieldFormat: "yyyy-MM-dd'T'HH:mm:ss'Z'",
            indexFieldType: 'DATE',
          },
        ],
      },
    },
    type: 'CONFLUENCEV2',
    additionalProperties: {
      isCrawlPageComment: false,
      exclusionUrlPatterns: [],
      inclusionFileTypePatterns: [],
      isCrawlBlog: false,
      blogTitleRegEX: [],
      proxyPort: '',
      inclusionUrlPatterns: [],
      attachmentTitleRegEX: [],
      includeSupportedFileType: false,
      isCrawlPage: true,
      fieldForUserId: 'uuid',
      commentTitleRegEX: [],
      exclusionSpaceKeyFilter: [],
      isCrawlBlogAttachment: false,
      isCrawlPersonalSpace: false,
      exclusionFileTypePatterns: [],
      isCrawlPageAttachment: false,
      inclusionSpaceKeyFilter: [],
      maxFileSizeInMegaBytes: '50',
      proxyHost: '',
      isCrawlArchivedSpace: false,
      isCrawlArchivedPage: false,
      isCrawlAcl: true,
      pageTitleRegEX: [],
      isCrawlBlogComment: false,
    },
  },
  roleArn: confluenceDataSourceRole.roleArn,
});

At first, I only used the attributes where I specified a value. After getting this error in the AWS console, I specified all attributes, including empty values or false boolean values.



We couldn't sync the following data source: 'ConfluenceDataSource',

at start time Aug 10, 2024, 5:05 PM GMT+2. Confluence Connector error

code: CNF-5133 Error message: The isCrawlPersonalSpace value is

invalid. Reason: isCrawlPersonalSpace should be a boolean value

true or false.

The deployment

The CloudFormation stack deployed successfully - the CloudFormation deployment took 21 minutes (destroying it again took 25 minutes).

I had to start synchronizing the data source manually because I didn't configure periodic synchronization. It is also necessary to assign users manually, as there is no CloudFormation resource for this.

Summary

After understanding the entire configuration, I was able to successfully deploy Q Business using AWS CDK. As a best practice, I can recommend to do the configuration manually for the first time and then use the IAM roles and the data source configuration created in the AWS console as a template. Especially the data source configuration, which is described in a JSON schema, is much easier if you can copy the values from the AWS CLI command. There are some reasonable default values there, such as mapping the source fields, which are hard to define on your own.

Can I use GenAI services to predict football results? - My experience during European Football Championship 2024

Julian Michel — Mon, 05 Aug 2024 19:42:15 +0000

During the European Football Championship 2024, we had a football betting game in my company. Since GenAI is useful for many use cases, I decided to predict the results using GenAI technologies. This blog post shows my experience and which AWS services are suitable for this use case.

Getting started with Amazon PartyRock

Amazon PartyRock is an Amazon Bedrock playground that's easy to use and free. So I decided to build a PartyRock app to predict football results.

GenAI models also have some basic knowledge of football. As I wanted to increase the accuracy of the predictions, I created a csv-file with all the results of the qualification round. I extracted the results from the German Wikipedia as it was in a tabular format that I could easily convert to a csv-file. The English Wikipedia page has more information. Since I only wanted to have the results, I decided to continue with the German Wikipedia page.

I ended up with a simple PartyRock application. It contains an upload field for the csv-file and two text fields to enter the teams playing in the next game:

In the prompt I told the model to predict the results. At first it didn't predict any results because the csv-file with the previous results didn't contain enough information. After changing the prompt again and telling the model to use it's own knowledge, I got real results. The last part of that sentence was important: "Use the data from [Previous results] and also your knowledge."

No more settings were needed - PartyRock started predicting the results.

Choosing the the best GenAI model

It's easy to compare the results of different GenAI models in PartyRock. You can simply add two text generation widgets with the same prompts and choose different models. I chose Claude 3 Sonnet and Jurassic-2 Ultra. PartyRock recognized that the Jurassic-2 model wasn't suitable for the complex task, so it chose Claude 3 Haiku instead, which is also displayed in the PartyRock app.

Predicting football results is a game of chance, so there's no model that's always right. The fact that PartyRock chose to use a different model instead of Jurassic-2 Ultra shows that they have different advantages, so it is always a good idea to test different models.

Bedrock

PartyRock is based on Bedrock, so it should be possible to get the same results directly in Bedrock. I entered a similar prompt in the Bedrock Playground available in the AWS console. Instead of uploading the csv-file to PartyRock, I copied the content into the prompt for Bedrock. As expected, Bedrock is able to predict football results.

If you want to implement a custom application for this use case, Bedrock can be the technical foundation as it provides the required functionality.

Bedrock Knowledge Bases

Knowledge Bases allow you to add custom documents to Bedrock. Instead of copying the previous result csv-file into the prompt, you can upload it to an S3 bucket. The content of the S3 bucket is then used to answer questions.

As a recommended option, Bedrock suggests creating a vector store based on Amazon OpenSearch Serverless. In my opinion, OpenSearch is very expensive, so I wanted to choose a cheaper option. First, I tested MongoDB Atlas, which has a free tier. However, the free tier doesn't include the features needed to use it as a vector store in Bedrock. Since Pinecone offers a free tier that can be used as a Bedrock vector store, I used it for this test.

Did Bedrock also predict football results with Knowlege Bases? No, unfortunately not. It tried to find information about matches between the two teams. However, it was unable to predict results because it didn't find enough specific information in the source files. I think Bedrock Knowledge Bases is very useful for retrieving information about documents - but it's not suitable for this use case.

Amazon Q Business

Amazon Q Business offers similar functionality. I uploaded the qualification results to a Confluence page with is configured as a data source in Q Business. The result was the same as with Bedrock Knowledge Bases. Q Business was very content centric. Since there was no information about the specific match, it didn't predict any results.

Prompt engineering in Amazon PartyRock

Prompt engineering was a task I had to pay attention to all the time. I had to change the prompt several times. After the first round I had to take into account the results of the previous round.

In the final rounds I had to tell the model that she also had to take into account additional time and the penalty shoot-out. This made the prompt more complex than before.

Evaluation

For the first week, I was on position one of the betting game for the European Football Championship 2024. At the end, I ended up at position 5 - probably better then predicting the results on my own as I'm not an sports expert.

The GenAI models often predict results such as 1:2 or 2:1 which is a common result - I also could use this strategy without GenAI on my own.

Different models resulted in different results. As I used two models, I had to decided which result I used in the game. So I always chose the result that made more sense to me.

As I was on the first position of the prediction game during the first days, I assume that it's easier to predict results when a better team plays against the weaker team. In the final rounds, both teams had similar skills, wo it was harder to predict the results.

Sometimes the model wasn't sure which result would be better, so it answered with two ideas. Also, I think some results after penalties are not realistic, like in this screenshot.

Summary

Predicting the football results for Euro 2024 was fun. Of course, if you are a sports expert, you will probably have better predictions. However, I don't think it was a bad idea to use GenAI as I got better results than other participants. I think this is not the primary use case for GenAI - but I was able to use it for predictions as well.

The important part for me: I learned a lot about GenAI in general and in AWS, and how to use prompt engineering for specific use cases.

How to build an enterprise chatbot with Amazon Q Business to integrate Confluence data

Julian Michel — Tue, 09 Jul 2024 20:31:06 +0000

GenAI tools like ChatGPT are now widely used - for private purposes. In an enterprise context, the requirements for building a GenAI chatbot are more complex. Source data must be securely integrated into the chatbot. The data must not be shared with other companies using the same GenAI platform of for training the model. User permissions must be supported so that restricted data is not shared with employees who should not have access to it.

Amazon Q Business is an AWS service that meets these requirements and enables businesses to easily build chatbots and GenAI assistants. Connectors are available to integrate various enterprise applications with Amazon Q Business. Confluence integration is one of the connectors available out of the box. It can import Confluence data into Q Business while respecting user-specific permissions.

This blog post shows how to integrate Confluence with Amazon Q Business and demonstrates that Q Business really respects permissions - it doesn't leak data to unauthorized users or employees.

Prerequisites

Amazon Q Business requires AWS IAM Identity Center for managed users. Therefore, set up IAM Identity Center first. The Identity Center instance must be set up in the same region as Q Business. Since Q Business is only available in the us-east-1 and us-west-2 regions, you must create AWS IAM Identity Center in one of these regions. If you have already configured IAM Identity Center in a different region, you must delete it and recreate it in one of the matching regions.

In addition, Q and the Confluence must use the same user data, otherwise the permissions won't work. Confluence can also be integrated with AWS IAM Identity Center, so you can use single sign-on for Q and Confluence. To do this, first set up AWS IAM Identity Center as the identity provider for Confluence.

Getting started: Create a Confluence API key

To integrate Confluence with Amazon Q Business, create an API key for one of your Admin users. They will have access to all content necessary to crawl the entire Confluence instance. For production use, you would use a technical user.

If you are using Confluence SaaS, go to the following URL to create a new API key: https://id.atlassian.com/manage-profile/security/api-tokens

Next, copy the API key.

Copy the API key to Secrets Manager

Q can use API keys that are stored in Secrets Manager. So create a new secret, select "other type of secret". Add the following attributes:

username: email of your confluence user
password: API key
hostUrl: Confluence URL

Enter a secret name that begins with QBusiness-. Only secrets that start with this name can be used in Q.

Create your Amazon Q Business application

Open Q Business in your AWS console and create a new application.

Enter the name of your application, such as AmazonQ, and then press "Next".

Configure the retrievers that will fetch data from the data source. If this is a proof of concept, select "Starter" and choose "1" unit.

Add the Confluence data source

Select the Confluence data source from the list of data sources.

Enter a name for the data source and specify the source type (cloud or on-premises) and URL.

Select "Basic authentication" and choose the secret you created earlier.

Let's create a new service role, which is a recommended option.

Select the sync scope - select "Pages" as the minimum option to transfer Confluence pages to Q.

Next, choose how often you want Q to sync your Confluence data. In the case of a proof of connect, "Run on demand" is sufficient to avoid costs.

User configuration

Select some AWS IAM Identity Center users who should have access to Q Business and select the subscription. Since there is a monthly fee after the free trial, check the pricing when selecting the subscription.

Now you are ready to use Amazon Q Business.

Prepare Confluence test data

To test the permissions feature in Amazon Q, create a Confluence page with restricted access. Only users who have access to the page in Confluence will be able to use the information in Q Business. In this example, only "User One" has access to the page.

Create a sample content for the new page.

Sync data source

Open the Q Business application, select the data source, and choose "Sync Now" to synchronize the data.

This will take a few minutes - Q Business will show the sync status in the meantime.

Debugging Amazon Q Business

In the "Sync run history" there is a link to view the sync log in CloudWatch Logs. Again, it takes a few minutes for the log to be available.

The log information is useful in case Q Business doesn't work as expected. In my first test, Q Business only worked for the first user (with API key). I was able to analyze the behavior in the Cloudwatch logs.

Q Business uses API calls like this: https://jumic-q.atlassian.net/wiki/rest/api/content/1310724/restriction/byOperation/read?&limit=200&start=0

In my case, the email attribute was empty because my first user was set up manually. I claimed it in the Confluence organization, then it worked.

Final test - Does Q Business handle permissions correctly?

First, let's test with "User Two". This user doesn't have access to the sample car information page.

Is it possible to leak the information in Q Business? No - Amazon Q responds that it can't find any relevant information.

Positive test - Can User One ask questions about the company car policy? Yes, it works because he is authorized in Confluence. Q Business works as expected.

Summary

Amazon Q Business with Confluence works really well. You can create a chatbot / AI assistant without programming - really cool. The security features are also great. It's good to see that Q Business doesn't leak information to unauthorized users. It only shows information when users are authorized in Confluence. In addition, Q Business also the source pages in their answers. That's useful for double-checking each result to make sure the results are correct.

Are there alternatives to Amazon Q Business? Atlassian has introduced its own GenAI functionality in Confluence. It's called Atlassian Intelligence. It can summarize pages or answer questions about the Confluence content. If you only need GenAI capabilities for Confluence, this feature might be sufficient.

Why still use Amazon Q Business? Q Business can connect to many other business applications such as MS SharePoint, MS Teams, Slack, databases or even S3 buckets. Users don't need to know in which source system the information is stored in. They can just ask Q Business, which uses information from all these source systems - which is more powerful.

Setting up AWS IAM Identity Center as an identity provider for Confluence

Julian Michel — Mon, 01 Jul 2024 05:05:41 +0000

AWS IAM Identity Center is a great tool for managing access to multiple AWS accounts in one centralized location. Users can assume roles in the AWS accounts they have access to and work in the AWS console or CLI.

It also supports single sign-on (SSO) capabilities to log in to some AWS services or third-party applications. One example of these applications is Confluence, which is widely used in enterprises. This blog post shows how to set up SSO in Confluence using AWS IAM Identity Center.

In what situations would you find this blog post useful?

You want to use Identity Center's SSO capabilities in Confluence.
You want to configure a source application for Amazon Q to test Q's security features (Q only uses information when a specific user has access to it)
You want to learn more about SSO and SAML.

In my case, I want to set up the Confluence integration to get familiar with Amazon Q. I chose the Confluence data sources because Confluence supports page-specific permissions, which should also be handled in Q. I'm using the trial version of Confluence for my tests and this blog post.

Getting started: AWS IAM Identity Center

I assume that AWS IAM Identity Center is already configured. To integrate Confluence as an application, open AWS IAM Identity Center in the AWS console and select "Applications" in the navigation. Then, add a new application.

AWS offers a catalog of out-of-the-box integrations for more than 300 applications. Confluence is one of them, so choose it.

Search for Confluence and select the result:

The next page provides a button to open step-by-step instructions for additional configuration assistance. Select this button to view these instructions. There are also options to change the name or description used in Identity Center.

In the step-by-step instructions, you will find application-specific instructions to configure the integration. For example it shows which values have to be configured in Confluence and in Identity Center. Download the certificate and copy the URL (both URLs are the same). You will need this information later again when setting up the identity provider in Confluence.

Now proceed with the necessary steps in Confluence. Later, we will need to perform some additional configurations in AWS IAM Identity Center.

Confluence: Adding a domain

Open the Atlassian Admin by navigating to the URL https://admin.atlassian.com/. In Atlassian Admin, you can manage the settings required for Identity Center integration.

First, add the domain used by your users. Accounts in your domain can become managed accounts, which means you can use the SSO capabilities of Identity Center.

Enter the domain name.

Before the domain can be associated, the ownership must be verified. I used DNS verification - feel free to use any of the other methods. For DNS validation, create the TXT record in the DNS management for your domain.

Ensure that the status is set to verified. If it is not verified, check the domain verification again. Next, select "Claim accounts" to automatically claim new accounts under this domain.

Use the recommended "Automatically claim" option to claim new accounts from your domain.

Now "claim setting" is set to "Automatically".

Confluence: Create a identity provider

In the next step, we will create a identity provider and link it to AWS IAM Identity Center. In the security settings, select "Identity providers" and use "Other provider" as there is no specific integration for Identity Center.

Confirm to start the free trial for Atlassian Guard to enable enterprise grade features.

Enter a name for the identity provider, e.g. "AWS IAM Identity Center".

Proceed with the SAML single sign-on integration.

Read the notes, then continue to the next step.

Now paste the certificate and URL you copied during the application configuration in AWS IAM Identity Center.

In case you didn't copy the values, you can display them again.

After you have configured the values in Confluence, the Confluence wizard will display two URLs that need to be copied to AWS IAM Identity Center.

In the Identity Center configuration, enter the URL of your Confluence instance and paste the two URLs you copied earlier.

Now continue in Confluence again. Select the previously created domain.

Stop the configuration wizard and save the SAML settings.

Confluence: Update authentication policy

As the final configuration step in Confluence, open the authentication policies and edit the newly created configuration (in my case, it is named "AWS IAM Identity Center").

Select the option "Enforce single sign-on".

AWS Identity Center: Assign users or groups

In Identity Center, add the users or groups that will be allowed to use the new application. I created a group and assigned all users to it.

Testing the integration

Log in to the AWS IAM Identity Center with one of the users. You should see the newly created Confluence application. Open the application.

Your browser will open a new window and you will be automatically signed in to Confluence.

Summary

If everything is set up correctly, the Confluence integration with AWS IAM Identity Center works well. The step-by-step instructions are useful, but read the documentation provided by Atlassian if you encounter any issues. Be careful when copying configuration values/URLs. Don't mix up the different URLs - this can cause errors and SSO won't work.

The integration of Confluence with AWS IAM Identity Center is just one example - many other applications can be integrated as well. AWS IAM Identity Center can also be used if you need a free SAML or OAuth 2.0 identity provider for software development or any other use case.

My personal AWS account setup - IAM Identity Center, temporary credentials and sandbox account

Julian Michel — Thu, 23 May 2024 21:26:29 +0000

I work on AWS projects every day and have access to internal and customer AWS accounts. But I also manage some personal AWS accounts. They are useful for several reasons:

I can run personal AWS applications, such as the smart home application I built.
I can test new features, e.g. it was very helpful to learn about the CDK pipeline when it was introduced. I was able to create the accounts as suggested in the documentation. Later I was able to apply my knowledge to the customer project.
If I have some permission issues, I can try to replicate the problem in my personal AWS accounts without any restrictions. I can try new services that are only available in organizational root accounts, such as AWS Identity Center.

Of course, I want to run the AWS account in a secure and convenient way. So I decided to manage the accounts as described in this article.

Account structure and AWS Organizations

Like a business, I created an AWS organization to manage my accounts. I'm using these accounts:

Root account, which owns the organization
Dev, Test, and Prod accounts to develop, test, and run my application
Pipeline account, which contains an AWS CodePipeline to deploy to Dev, Int, Prod
Sandbox account for testing new stuff (gets nuked after testing)

User management and authentication using AWS Identity Center

I'm sure AWS IAM user management was great when it first launched. Now, AWS IAM Identity Center has more features and is easier to use. For example, it provides an easy-to-use interface to access all accounts and roles. Or it has improved MFA capabilities, such as support for Apple's TouchID. That is why I chose AWS Identity Center. It is set up in the AWS Organizations root account and connects to all accounts in the organization.

When you open AWS IAM Identity Center, you can see the accounts and roles you can assume:

Temporary credentials and Leapp

AWS IAM Identity Center provides temporary credentials by default, which is a good security choice. It supports SSO integration with AWS CLI or manual download of access key, secret access key, and session token.

Personally, I prefer to use Leapp, a tool that supports secure cloud access in multi-account environments. Recently, the company behind Leapp announced the end of the commercial version. The open source version still exists and can be used.

Leapp displays all AWS accounts/roles that are configured in AWS Identity Center. If you select an account, the credentials can be used in the CLI. You can override the default profile so you don't have to pass a --profile parameter. Or, you can configure a name profile.

Nuking the sandbox account

When I'm learning a new AWS service or testing a complex scenario, I'm not always using infrastructure as code. In this case, it takes a long time to manually clean up an AWS account and delete all (expensive) resources.

To improve this process, I use aws-nuke, which automatically deletes all AWS resources in an AWS account. Be careful with this tool. The first time I used it, it also deleted the configurations required for IAM Identity Center. So I couldn't login anymore.

aws-nuke supports filters to exclude resources that should not deleted. I created filters to exclude the IAM Identity Center configuration and resources created during AWS CDK bootstrapping. I ended up with these filters:



presets:

  common:

    filters:

      IAMRole:

      - "OrganizationAccountAccessRole"

      - type: glob

        value: "cdk-hnb659fds-"

      - type: glob

        value: "AWSReservedSSO_AdministratorAccess_"

      IAMRolePolicyAttachment:

      - "OrganizationAccountAccessRole -> AdministratorAccess"

      - type: glob

        value: "AWSReservedSSO_AdministratorAccess_ -> AdministratorAccess"

      - type: glob

        value: "cdk-hnb659fds-"

      IAMRolePolicy:

      - type: glob

        value: "cdk-hnb659fds-"

      IAMSAMLProvider:

      - type: glob

        value: "arn:aws:iam:::saml-provider/AWSSSO__DO_NOT_DELETE"

      S3Bucket:

      - type: glob

        value: "s3://cdk-hnb659fds-assets"

      SSMParameter:

      - "/cdk-bootstrap/hnb659fds/version"

      CloudFormationStack:

      - CDKToolkit

      ECRRepository:

      - type: glob

        value: "Repository: cdk-hnb659fds-container-assets-*"

Summary

This setup works very well for me. AWS IAM Identity Center is great for managing users and logging into AWS accounts. Leapp is very helpful when using CLI credentials. However, it's not the only option as AWS IAM Identity Center offers other options as well. I regularly use the sandbox account when testing or learning about AWS services. By deleting everything with aws-nuke, I can easily start with an empty AWS account - and of course, I don't have to pay for unused resources.

AppSync Merged API – Our real project experience as part of our hackathon

Julian Michel — Thu, 29 Jun 2023 12:09:57 +0000

AWS released a new feature for GraphQL APIs in May: Multiple AppSync APIs can be merged into one overall API. Our AI Hackathon gave us the opportunity to test the new Merged API feature in a real-world project.

Why are Merged APIs such a useful feature?

When designing complex software, it is common to break the application into components or microservices. Each component has its own API so that it can be developed and deployed independently. However, when APIs are published or used in front ends, it makes sense to provide an aggregated API with services of all components. This is possible with the new Merged API feature.

How to create a Merged API using AWS CDK?

We first looked at the AppSync documentation in the AWS CDK to see how to create Merged APIs. Unfortunately, the documentation doesn’t include a section on Merged APIs. We checked the feature requests and pull requests on the AWS CDK GitHub project. However, since we did not find any information, we determined that Merged APIs are not currently supported by CDK L2 constructs. To get support for this feature in AWS CDK, we’ve created a feature request in the AWS CDK project.

So how did we create the Merged APIs?

CloudFormation already includes support for Merged APIs. The documentation describes the attributes and resources which are required. Using CDK L1 constructs, we could use these CloudFormation resources to create the Merged API.

First, we created the necessary IAM roles. Currently they don’t follow the principle of least privilege. When Merged APIs are supported in the CDK, this should also be considered.

const role = new Role(this, `merged-api-role-${props.namespace}`, {
  assumedBy: new ServicePrincipal('appsync.amazonaws.com'),
  roleName: `merged-api-role-${props.namespace}`,
});
role.addToPolicy(new PolicyStatement({
  resources: ['*'],
  actions: ['appsync:*'],
}));

const cwRole = new Role(this, 'MergedAPICWRole', {
  assumedBy: new ServicePrincipal('appsync.amazonaws.com'),
});

cwRole.addToPolicy(new PolicyStatement({
  actions: ['logs:CreateLogGroup',
    'logs:CreateLogStream',
    'logs:PutLogEvents'],
  resources: [`arn:aws:logs:${cdk.Stack.of(this).region}:${cdk.Stack.of(this).account}:*`],
}));

In the GraphQL API, the attributes apiType and mergedApiExecutionRoleArn are specific to Merged APIs. In addition, logging had to be explicitly configured, which requires more effort than L2 constructs.

const api = new CfnGraphQLApi(this, `merged-api-${props.namespace}`, {
  authenticationType: 'API_KEY',
  additionalAuthenticationProviders: [
    {
      authenticationType: 'AMAZON_COGNITO_USER_POOLS',
      userPoolConfig: {
        userPoolId: props.userPool.userPoolId,
        awsRegion: cdk.Stack.of(this).region,
      },
    },
  ],
  apiType: 'MERGED',
  name: cdk.Names.uniqueId(this),
  mergedApiExecutionRoleArn: role.roleArn,
  xrayEnabled: true,
  logConfig: {
    cloudWatchLogsRoleArn: cwRole.roleArn, fieldLogLevel: 'ALL',
  },
});

Each source API is assigned to the Merged API using a SourceApiAssociation.

new CfnSourceApiAssociation(this, name, {
  sourceApiIdentifier: graphqlApi.apiId,
  mergedApiIdentifier: mergedApi.attrApiId,
  sourceApiAssociationConfig: {
    mergeType: 'AUTO_MERGE',
  },
});

The first deployment failed. After specifying mergeType: 'AUTO_MERGE', we were able to create the Merged API.

How did we structure our project?

Each component is developed as separate CDK stack. An additional stack called MergedApiStack includes the Merged API. The source APIs are passed to the Merged API Stack using CDK Stack Props.

This allows us to publish a central AppSync API that includes API methods of all components.

Summary

Because of the missing CDK L2 constructs, it was difficult to create the Merged API. Otherwise, our experience with the Merged API has been very good. The Merged API is updated directly and allows us to have one central API for our application.

What’s next? We will likely implement an L2 construct to add support for Merged APIs in the AWS CDK. Then our hackathon and community work will also contribute to AWS CDK.

Who to reach out to?
Danielle, Matt, Johannes or myself.

Quickly recover from failures using automated SQS DLQ redrive

Julian Michel — Sat, 10 Jun 2023 10:04:04 +0000

Two years ago at re:Invent, AWS announced DLQ redrive to move messages from a dead letter queue (DLQ) to the original queue. Previously, this feature was only available using the AWS console. Now, DLQ redrive is available using AWS SDK and CLI.

The new feature provides the ability to automatically handle invalid messages in DLQs and increase the reliability in asynchronous applications.

This sample architecture includes a Lambda function that sends messages to a third party API. If the third party API becomes unavailable, the processing fails and the messages are sent to the DLQ. Previously, operation teams had to manually run the DLQ redrive in the AWS console. Lambda functions using AWS SDK can help to automate this process.

How can you automate this process? Implement the logic from the diagram:

The SQS queue sends failed messages to the DLQ
A time-based EventBridge event triggers a lambda function periodically (e.g. every 5 minutes)
The lambda function checks whether the DLQ contains failed messages. If there are failed messages, it calls a health check of the third party backend.
If the health check is successful, the lambda function starts the DLQ redrive using the AWS SDK method StartMessageMoveTask.

If the third party backend still fails, the process is repeated after five minutes until the DLQ is empty again.

For more details on the new API calls, read the official AWS blog post.