DEV Community: Joris Conijn

Fixing oversized artifacts AWS CDK Pipelines

Joris Conijn — Thu, 29 May 2025 17:12:13 +0000

I built a workload using AWS CDK, and the CodePipeline stopped working at the worst moment. It was right at the end of the sprint; we had done multiple deployments before. But at this moment, the moment you might recognize. When this last PR is released, we will have met our deadline! Kaboom, the pipeline stops working, and the reason for the failure is not related to the changes you made.

Some background

When you use CDK, you can write infrastructure as code using your favorite programming language. CDK comes with certain constructs that make your life easier. For example, you can easily create a CodePipeline that will fetch the code from your repository. It will then synthesize the CDK code into CloudFormation. The CodePipeline will then upload all assets to an S3 bucket, and the CloudFormation templates will be deployed to the target accounts. This CloudFormation template will contain references to the assets on S3. When the template is deployed, the assets are fetched from the S3 Bucket.

What has happened?

I ran into the problem that CodePipeline will pass an artifact from step to step. This artifact is a single zipped file, which makes it easy to pass it along the pipeline.

When CodePipeline wants to deploy the CloudFormation templates. It uploads the zipped file, and it knows where in the zipped file the template is located. However, all other assets are also in the zipfile; this could be the code that you want to run in Lambda functions or even Lambda Layers. You can imagine that when your workload starts to become more mature, there will be more assets in this zipfile.

Why is that a problem?

Well, the artifact has exceeded the maximum size! With CDK projects, this is something that you will run into sooner or later. Usually, you only add stuff to your projects and rarely remove things. And, as you already learned, this always happens at a moment that you don’t expect at all. And in my case, at the worst possible time, right before my deadline.

How can we fix this?

The answer is quite simple. As I already explained, the problem is with the size of the artifact being passed to the CloudFormation deploy action. We just have to reduce the size after all the assets in the artifact are uploaded to S3.

private stripAssets(pipeline: pipelines.CodePipeline) {
 const policies = [
   new iam.PolicyStatement({
     effect: iam.Effect.ALLOW,
     resources: [<code>arn:aws:s3:::${this.artifactBucket.bucketName}/*],
     actions: ['s3:PutObject'],
   }),
 ];

 if (this.artifactBucket.encryptionKey) {
   policies.push(
     new iam.PolicyStatement({
       effect: iam.Effect.ALLOW,
       resources: [this.artifactBucket.encryptionKey.keyArn],
       actions: ['kms:GenerateDataKey'],
     }),
   );
 }
 pipeline.addWave('BeforeStageDeploy', {
   pre: [
     new pipelines.CodeBuildStep('StripAssetsFromAssembly', {
       input: pipeline.cloudAssemblyFileSet,
       commands: [
         'S3_PATH=${CODEBUILD_SOURCE_VERSION#"arn:aws:s3:::"}',
         'ZIP_ARCHIVE=$(basename $S3_PATH)',
         'echo $S3_PATH',
         'echo $ZIP_ARCHIVE',
         'ls',
         'rm -rfv asset.*',
         'zip -r -q -A $ZIP_ARCHIVE *',
         'ls',
         'aws s3 cp $ZIP_ARCHIVE s3://$S3_PATH',
       ],
       rolePolicyStatements: policies,
     }),
   ],
 });
}

What happens in the code sample?

We make sure that we have the s3:PutObject rights on the artifact bucket. (We need to overwrite the existing artifact.)
We need the kms:GenerateDataKey rights. (Optional, I am using KMS encryption on the bucket.)
We add a CodeBuildStep that will consume the artifact.
The artifact is downloaded and unzipped by CodeBuild.
We figure out the origin of the artifact.
We remove all assets.
We will zip the artifact.
We will upload the artifact over the original.

Because we placed the CodeBuildStep in the pre-section it will be placed right before the deployment of the CloudFormation deploy action.

Conclusion

Hitting CodePipeline’s artifact limit can stall your sprint at the worst moment. By inserting a pre-deploy CodeBuildStep that strips the bulky assets after they’re safely uploaded to S3, you shrink the artifact without changing your workflow. This quick fix keeps your CDK pipeline flowing, your CloudFormation stacks deploying, and your release deadlines intact.

Photo by Ellie Burgin

The post Fixing oversized artifacts AWS CDK Pipelines appeared first on Xebia.

AWS CDK and the Hidden Risks to Least Privilege

Joris Conijn — Mon, 28 Apr 2025 09:28:38 +0000

Have we given up on the least privileged principle? Personally, I am a big fan of it. But let’s be honest, it can also be tough to follow the principle strictly. With the rise of CDK, it became even harder.

Why does CDK make it harder?

CDK is a great tool to use when you are developing your infrastructure. You can easily build resources using the level 1 and level 2 constructs. So far, so good. The problem lies within the level 2 constructs. They are somewhat opinionated about how you should use them. For example, when you want to store secrets in the Cloud, you create a secret in Secret Manager. This secret needs to be encrypted, so we will use KMS. This can easily be achieved through CDK:

const secret = new secretsmanager.Secret(this, 'Secret', {
 secretName: `MySecret`,
 description: 'My super Secret',
 encryptionKey: kmsKey,
 generateSecretString: {
   secretStringTemplate: JSON.stringify({ username: 'myUser' }),
   generateStringKey: 'password',
 },
});

As you can see in the code example. We are using a customer-managed KMS key. We do this because we want to control who can use the key for decryption. The fact that you pass the key into the Secret construct means that CDK will help you and adds a policy to your KMS key. The policy that gets added is the following:

{
 "Effect": "Allow",
 "Principal": {
   "AWS": "arn:aws:iam::000000000000:root"
 },
 "Action": [
   "kms:CreateGrant",
   "kms:Decrypt",
   "kms:DescribeKey",
   "kms:Encrypt",
   "kms:GenerateDataKey*",
   "kms:ReEncrypt*"
 ],
 "Resource": "*",
 "Condition": {
   "StringEquals": {
     "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
   }
 }
}

Those familiar with KMS policies will notice that any principal can now use the key if it is used through the secrets manager service. At first, you might think it is not that bad or convenient. But the problem with this is that any principal with an allow statement on the secretsmanager:GetSecretValue action will be able to read your secret.

It gets worse

You are storing the secret for a reason. You probably need to read it somewhere in your workload. To do that, you need to create a role and grant the role the rights to read the secret. This is easily done with CDK.

secret.grantRead(role)

But this simple statement again changes the KMS policy. It will add the following policy.

{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::000000000000:role/MyRole"
  },
  "Action": [
    "kms:Decrypt",
    "kms:Encrypt",
    "kms:GenerateDataKey*",
    "kms:ReEncrypt*"
  ],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
    }
  }
}

At least it’s specific to the role, and this is what you want. But remember the previous policy? It already allows all principals to use this key.

How to work around this?

Sadly, the only way around this is to use the level 1 constructs. These constructs are not opinionated and are a one-to-one mapping to the CloudFromation resources. With those level 1 constructs, you can specify the KMS key without changing the key policy for you. The flip side is that you do need to allow all principals who need to read the secret in your KMS key policy.

Conclusion

In summary, while CDK greatly simplifies infrastructure development, it can unintentionally weaken strict least privilege practices, especially around KMS key policies. You may need to step down to level 1 constructs to maintain tighter control, trading some convenience for the precision and security your workloads deserve.

Photo by AS Photography

The post AWS CDK and the Hidden Risks to Least Privilege appeared first on Xebia.

Optimizing OpenSearch Ingestion: Ensuring Reliability, Efficiency, and Cost Savings

Joris Conijn — Tue, 11 Mar 2025 12:15:38 +0000

Ingesting data into an OpenSearch cluster looks easy if you read the documentation. The truth is it is easy, but it all depends on how much you care about the data you are ingesting. Let me go one step back. Why do we even use OpenSearch? With the rise of AI, you also need a knowledge base. These knowledge bases can be hosted in OpenSearch. However, to use the OpenSearch database, you must also fill it out with data.

I recently was challenged to create a database for GenAI purposes. We had a dataset that needed to be loaded into OpenSearch. Amazon OpenSearch Ingestion sounds like a service that can help you with that.

Preparing the data

First of all, the data is never in the format that you want it to be. First, we will need to transform it into something that we can ingest. For this use-case, I used StepFunctions (but avoid costly loops) each document coming in will trigger an execution. In my case, I wanted to be able to search the document with semantic meaning. In order to do this efficiently, we will need to chunk the documents into smaller bits. This way, we can generate embeddings for each chunk, which can be used for semantic search. You can create a single file with all chunks, and each chunk will have the same metadata set; the only difference is the chunk text and chunk identification. Once we have this file, we will place it on S3, S3 will send an S3 Event Trigger to an SQS Queue, and the OpenSearch Ingestion pipeline will ingest it into the OpenSearch database.

So far so good

There is nothing wrong with the approach I just described. This is a textbook example coming from the AWS documentation pages. But an ingestion pipeline will cost you money even when you don’t ingest anything. For this reason, I developed a Lambda function that would stop the pipeline when no messages are in the queue. It will also start the pipeline when there are messages in the queue. This was a nice optimization from a cost perspective.

After some testing, we expected 10k documents in the system, but we only had around 5k. Where did the other documents go? The dead-letter queues were empty, and we did not have any log traces of any failures whatsoever… It turned out that when you stop the ingestion pipeline, all messages in the buffer are lost.

Acknowledgments to the rescue

You can argue a lot about the fact that when you stop a pipeline, the documents in the buffer are lost. But that is the reality that I needed to deal with. You can enable persistent buffering, which would at least help with the data loss part. But you still need to start the pipeline to continue the job. I looked more in the direction of SQS. The whole purpose of SQS is to queue messages and ensure they are processed. After some investigation, I noticed two options I wanted to share.

version: '2'
s3_pipeline:
  source:
    s3:
      workers: 10
      notification_type: sqs
      notification_source: s3
      codec:
        json: {}
      compression: none
      acknowledgments: true
      records_to_accumulate: 100
      on_error: retain_messages
      sqs:
        queue_url: https://sqs.eu-west-1.amazonaws.com/000000000000/MyQueue
        maximum_messages: 10
        visibility_timeout: 60s
        poll_delay: 0s
        visibility_duplication_protection: true
      aws:
        region: eu-west-1
        sts_role_arn: arn:aws:iam::000000000000:role/MyRole
  sink:
    - opensearch:
        serverless: false
        hosts:
          - https://vpcxxxxxx.eu-west-1.es.amazonaws.com
        index: documents-${/index}
        bulk_size: 5
        max_retries: 3
        dlq:
          s3:
            bucket: MyBucketName
            key_path_prefix: dlq/
            region: eu-west-1
            sts_role_arn: arn:aws:iam::000000000000:role/MyRole
        aws:
          region: eu-west-1
          sts_role_arn: arn:aws:iam::000000000000:role/MyRole
        document_id: ${/chunk_id}
        actions:
          - type: delete
            when: /operation == "delete"
          - type: update
            when: /operation == "update"
          - type: index
            when: /operation == "index"

The configuration options are acknowledgments and visibility_duplication_protection. The former will keep the message in the queue until it has been ingested into the sink, and the latter will ensure that the message stays in flight as long as it is in the buffer. These options will ensure that the SQS queue is used as intended. When a failure occurs, the message will reoccur in the queue and be retried. After x number of attempts, it will go to the dead-letter queue. These settings ensured that all 10k documents in the batch were ingested as expected.

The pipeline would still be stopped once the SQS Queue is empty. To prevent this from happening I also included the ApproximateNumberOfMessagesNotVisible next to the ApproximateNumberOfMessages metric.

Dealing with updates and deletes

I also wanted to share how we deal with updates and deletions. Ingestion of a new document is easy, but updates and deletions are different. When you have ingested a document with 15 chunks, and the updated version only has 12, you need to delete 3 and potentially update some chunks. For example, this might occur when you delete a paragraph in the middle of a text.

First, you need to know if the document has already been ingested. A search action on the OpenSearch database can do this. This search action returns all the chunks already present in the database. You can use logic to determine what chunk needs to be updated and which needs to be deleted. We add three additional metadata fields in each chunk: index, chunk_id, and operation.

The index field determines the index to which the document is being ingested. The chunck_id will become the internal document id, and the operation field will determine if the chunk needs to be indexed, updated, or deleted.

Conclusion

Ingesting data into OpenSearch sounds straightforward, but the real challenge lies in ensuring reliability, efficiency, and cost-effectiveness. While the textbook approach works, real-world scenarios often demand optimizations—like dynamically stopping the ingestion pipeline to save costs or leveraging SQS acknowledgments to guarantee message processing.

By implementing these adjustments, we achieved a more resilient ingestion process that ensures all documents make it into OpenSearch as expected. The combination of Step Functions for document transformation, SQS for reliable queuing, and metadata tagging for updates and deletions provided a scalable and maintainable solution. At the end of the day, ingestion isn’t just about getting data into the system—it’s about making sure the right data gets there, stays there, and can be efficiently queried when needed.

Photo by Andrea Piacquadio

The post Optimizing OpenSearch Ingestion: Ensuring Reliability, Efficiency, and Cost Savings appeared first on Xebia.

Cross-Stack RDS User Provisioning and Schema Migrations with AWS Lambda

Joris Conijn — Tue, 04 Mar 2025 13:59:53 +0000

Have you ever been in a situation where you want to provision or configure things cross-stack? Splitting these into logical stacks is always good when dealing with more complex environments. I already shared this in one of my previous blogs. But this also introduces a different problem!

Granting access to your database

When you use DynamoDB, you have to use IAM to grant access. However, other databases like MySQL also have an internal authentication method. You have a few options when you create the database in one stack and a different stack wants to access the database.

Use the credentials that you created at deployment time.
Use identity and access management (AWS IAM).
Create a local user per use-case.

Now, all options will work in the end, but they do have their pros and cons. I am a big fan of the least privileged principle. So, for me, using the credentials you created at deployment time is too privileged. With these credentials, you can create databases and tables. This also means that you can drop tables and delete databases. You can compare these credentials with the root credentials of a Linux system or the root account for your AWS account. The root account is there for a reason. That reason is not for day-to-day work. You should use those credentials for administrative tasks, not operational tasks.

You could use AWS IAM, and this will give us the ability to be more least privileged. This is because you need to create a user in the database. That user will receive grants on what it can do on the database and tables. You can create a read-only role if you only need to read from the database. If you also need to write, you can simply add that permission. Regardless of whether you go for option 2 or 3, you still need to provision the user and the permissions in the database.

Provisioning RDS users

After you created an RDS instance, you configured the database’s “root” credentials. The Secrets Manager should generate these credentials. Users should never have these credentials, but if they do, they are there just in case. Since we don’t want to use the root credentials, we need a user to access the database through our application. For this, we can use a provisioner lambda function. This lambda function creates the local users in the database. You can simply invoke the Lambda function as a custom resource using the same template as the RDS instance.

The Lambda function can retrieve the “root” credentials from Secrets Manager. It can use those credentials to connect to the database, create it, and load the initial schema. Next, it will create the user and the grants that the user needs. You don’t need to assign a password if you plan to use IAM. If you don’t want to use IAM, you should create the credentials in Secrets Manager and pass them dynamically into the custom resource. Afterward, your user is ready to use your application.

Provisioning RDS users cross-stack

But what if you have multiple stacks and want to use the RDS instance created in a different stack? The same principle applies. You still need to create the provisioner with the RDS instance. But instead of creating the custom resource in the template of the RDS instance, you create it in the other template. For this, you need the ARN of the Lambda function. You can use a naming schema or store the ARN in an SSM Parameter.

The advantage of this is that you can invoke the creation of database users in the template where they are used. Plus, if you make the provisioner smart enough, you can pass on the grants the user will need. This way, you encapsulate the user, the rights, and the application into the same template. This will reduce the maintenance load on your application and its infrastructure.

Schema migrations

One of the other advantages of having a provisioner is that you can also manage schema migrations. Mutating the schema can be invoked with the same ease as the user creation. This gives you the ability to perform these migrations during the CloudFormation deployments. You do need to make sure that the application is backward compatible.

Conclusion

Splitting infrastructure into multiple stacks keeps things organized, but it also introduces challenges like managing database access and schema changes. Using a Lambda function as a provisioner, you can automate user creation and schema migrations as part of your CloudFormation deployments. This approach keeps everything tightly scoped, ensures the least privilege principle, and reduces operational overhead. Whether you use IAM authentication or local users, making the provisioner smart enough to handle both ensures flexibility. With schema migrations baked into your deployments, you get a reliable and repeatable way to evolve your database without manual intervention.

Photo by Muhammed Ensar

The post Cross-Stack RDS User Provisioning and Schema Migrations with AWS Lambda appeared first on Xebia.

Securing S3 Downloads with ALB and Cognito Authentication

Joris Conijn — Fri, 28 Feb 2025 08:09:16 +0000

Securing an endpoint used to be hard. Nowadays, with the cloud, it’s quite easy. You only need to know how! Assume you have files on S3 that you like to share. You could make the object publicly available. This would allow your users to download the file using their browsers simply. If you need to scale it, you can add CloudFront. This would cache the content closer to your users, making sure that your users have the best performance. But what if you want to control who can download the file? For this, you will need authentication and authorization.

Authentication vs Authorization

Authentication is all about identifying who you are. First, we need to make sure that we know who the user is. Once we establish who the user is, we can see if the user can access the content. The latter is authorization. AWS has a service called Cognito that allows you to manage a pool of users. These users can originate from other sources like Google, Facebook, and your own identity provider. Or, if you don’t want to use identity providers, you can also create users directly in the user pool.

You can also create groups, and based on these groups, you can manage the authorization. For example, you could make a group called developers. All users within this group should be allowed to fetch the build report hosted on S3.

Building the endpoint

With the Cognito User Pool in place, we need a way to validate the user during the request. I am using an Application Load Balancer to invoke a Lambda function. This function can then check if the user can access the report. The logic is quite simple: if the user is part of the developer’s group, the user can read the report.

In this case, we can use the native Cognito integration of the application load balancer. What this will do is the following:

If the user is unauthenticated, navigate to the Cognito-hosted UI. If you use your identity provider, you will be redirected to your company’s login page. If you host the users from the user pool, a login form is shown to the user. After the user has logged a redirect, the user is now authenticated. The load balancer will now invoke the target group with the request.

We also want to check if the user is in the developer group. We will use a Lambda function to check this. The following code would do the trick:

import base64
import json
from typing import List

def decode(data: str) -> dict:
    return json.loads(base64.b64decode(data.split('.')[1]).decode('utf-8'))

def resolve_groups(groups: str) -> List[str]:
    return list(map(lambda group: group.strip(), groups[1:-1].split(',')))

def handler(event, context) -> dict:
    code = 403
    description = "403 Access Denied"
    body = "Access Denied"
    user = decode(event["headers"]["x-amzn-oidc-data"])
    groups = resolve_groups(user.get('custom:groups', '[]'))

    if 'developers' in groups:
        code = 200
        description = "200 OK"
        body = f"Hi {user.get('name')}, you should be able to download the report"

    return {
        "statusCode": code,
        "statusDescription": description,
        "isBase64Encoded": False,
        "headers": {"Content-Type": "json; charset=utf-8"},
        "body": body,
    }

The listener on the application load balancer and the user pool client can be configured as follows:

  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Port: 443
      Protocol: HTTPS
      Certificates:
        - CertificateArn: arn:aws:acm:eu-west-1:111122223333:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      DefaultActions:
        - AuthenticateCognitoConfig:
            OnUnauthenticatedRequest: authenticate
            Scope: openid
            UserPoolArn: arn:aws:cognito-idp:eu-west-1:111122223333:userpool/eu-west-1_xXXXxxxx
            UserPoolClientId: !Ref UserPoolClient
          Order: 1
          Type: authenticate-cognito
        - Order: 2
          TargetGroupArn:
            Ref: LambdaTarget
          Type: forward

  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      AllowedOAuthFlows:
        - code
      AllowedOAuthFlowsUserPoolClient: true
      AllowedOAuthScopes:
        - profile
        - phone
        - email
        - openid
        - aws.cognito.signin.user.admin
      CallbackURLs:
        - https://<MyDomainName>/oauth2/idpresponse
      ClientName: MyClient
      GenerateSecret: true
      LogoutURLs:
        - https://<MyDomainName>/logout
      SupportedIdentityProviders:
        - COGNITO
      UserPoolId: eu-west-1_xXXXxxxx

You also need to setup the lambda function as followed:

  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0
          Description: Allow all outbound traffic by default
          IpProtocol: "-1"
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          Description: Allow HTTPS access
          FromPort: 443
          IpProtocol: tcp
          ToPort: 443
      VpcId: !Ref VPCAsParameter

  LambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole

  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: my-bucket
        S3Key: path/to/code.zip
      Handler: index.handler
      Role: !GetAtt LambdaRole.Arn
      Runtime: python3.12
      VpcConfig:
        SecurityGroupIds:
          - !GetAtt LambdaSecurityGroup.GroupId
        SubnetIds:
          - !Ref Subnet1
          - !Ref Subnet2
          - !Ref Subnet3

  LambdaPermissions:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt LambdaFunction.Arn
      Principal: elasticloadbalancing.amazonaws.com

  LambdaTarget:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    DependsOn:
      - LambdaPermissions
    Properties:
      TargetType: lambda
      Targets:
        - Id: !GetAtt LambdaFunction.Arn

You will be prompted to log in when you navigate to the load balancer. Afterward, you can see that the lambda function was invoked. This is a very simple example, but the idea is that you can extend the lambda function with the logic you need. For example, you could create a pre-signed URL for the report on S3 and redirect the user directly to that URL. This will then download the report automatically.

Conclusion

Securing access to your S3 files doesn’t have to be complicated. By leveraging AWS Cognito, an Application Load Balancer, and a simple Lambda function, you can control exactly who gets access to your files—without exposing them to the public. With this setup, authentication is handled seamlessly, and authorization is as simple as checking group memberships. From here, you can expand the functionality further, such as generating pre-signed URLs for downloads or adding more granular permissions. The cloud makes it easy—you just need to know how!

Photo by Pixabay

The post Securing S3 Downloads with ALB and Cognito Authentication appeared first on Xebia.

Become a documentation ninja

Joris Conijn — Wed, 19 Feb 2025 07:34:42 +0000

Writing documentation sucks! But there are ways to make it easier and maybe even fun! In one of my previous blogs, I explained how you can embed your documentation in your pull requests and why you should consider doing it, too. In this blog, I want to go a bit deeper into the syntax you can use to improve your documentation pages on Confluence.

Syntax? You mean Markdown, right?

Yes, Markdown is the documentation syntax that Mark uses. However, it has some extensions that are still valid in Markdown format and allow you to use the macros from Confluence.

Linking to other pages

When you write documentation, you probably have different pages explaining different things. You can logically split areas, making it easier to navigate and read. But sometimes, you want to link from one page to another. The problem is that you don’t know the URL yet since that is a Confluence page. You could publish the page and look up the link, which kills the motivation to write documentation. Assume we have a page like this:

# My Confluence Page

My description of the page

And we want to link to this page. You can simply do this with the following syntax:

Read this on [My Confluence Page](ac:) or [read](ac:My Confluence Page) the page.

As you can see, you can link to the page title itself. Or, you can use a custom word, like “read” and link to the page. When mark uploads the page to Confluence the links will be resolved automatically.

Linking to Jira issues

Next to linking to pages, you will probably also refer to Jira issues. However, you could link to Jira issues using native markdown. When you use the Jira reference macro, the type, title, and status of the issue are shown. This gives immediate feedback to the reader on what the status is. What you need to do is to define a macro in the top of your markdown file and mention the issue as plain text. Here is an example:

<!-- Macro: MYJIRA-\d+
     Template: ac:jira:ticket
     Ticket: ${0} →

# My page that contains a Jira ticket reference

See task MYJIRA-123.

In this example, MYJIRA is the Jira project code. The macro definition matches everything that starts with MYJIRA- and is followed by a set of digits. You can then simply mention the issues in your markdown file, and they will be converted into Jira macros automatically.

Hide code snippets in collapsable macros

Especially when you write documentation, you may need to hide parts of the code snippets. One example that I use is for OS-dependent instructions. Assume you want to provide instructions on using your product on the command line. In that case, you could give instructions for Windows, Linux, and MacOS. To use this, you first need to define the macro. Then, use the macro to build the collapsable snippet.

<!-- Macro: :expand-start:(?s)(.*?):\n(.*?)\n:expand-end:
     Template: ac:expand
     Title: ${1}
     Body: ${2} -->
# My page with command line instructions

Before you start, you must ensure you set the environment variable:

:expand-start:Unix:

Unix instructions go here

:expand-end:

:expand-start:Windows:

Windows instructions go here

:expand-end:

Page properties and status badges

This is my favorite! This is such a powerful feature of Confluence. For those of you who are not familiar with this functionality, you define a table, which contains a set of keys and values. For example, you can specify an owner and a date in this table. You also need to set a label on the page. This is required for the report page. The label is used to make a list of pages that have this label. But the cool thing is that you can also display the values of the page properties.

As you can see in the screenshot above, we have three pages. The parent page holds the report, and the child pages are Project 1 and Project 2. However, it also shows the date, owner, and status badge. This is a great way to give a nice overview of projects.

To create the page properties on the child page, you need the following syntax.

<!-- Parent: Projects -->
<!-- Label: project -->
<!-- Macro: :in-progress:
     Template: ac:status
     Title: In Progress
     Color: Blue -->

# Project 1

<ac:structured-macro ac:name="details" ac:schema-version="1">
    <ac:rich-text-body>
        <table>
            <tbody>
                <tr><th><strong>Status</strong></th><td>:in-progress:</td></tr>
                <tr><th><strong>Owner</strong></th><td>Joris Conijn</td></tr>
                <tr><th><strong>Date</strong></th><td>2025-02-15</td></tr>
            </tbody>
        </table>
    </ac:rich-text-body>
</ac:structured-macro>

The Project 2 markdown file is similar to the one above. However, the parent page has a different definition.

# Projects

<!-- Include: ac:detailssummary
     SortBy: Title
     CQL: 'space = "Projects" and label = "project"' -->

In this example, my confluence space is called Projects. The Confluence Query Language (CQL) retrieves all pages in the Projects space that have the label project. A third requirement is that the page also needs to have the page properties defined. We did this on Projects 1 and 2, so the two projects are rendered in the table.

Conclusion

Writing documentation doesn’t have to be a chore. By leveraging Confluence macros within Markdown, you can enhance your documentation with dynamic links, Jira issue references, collapsible content, and structured page properties. These features make your documentation more interactive, easier to maintain, and more useful for your team.

By embedding these enhancements directly in your markdown files, you streamline the process of writing and updating documentation, reducing friction and improving adoption. So, next time you document a feature or process, try incorporating these techniques. You might find documentation efficient and enjoyable!

Photo by cottonbro studio

The post Become a documentation ninja appeared first on Xebia.

ECS Fargate Persistent Storage: EFS Access Points vs. Lambda Workarounds

Joris Conijn — Tue, 04 Feb 2025 21:33:12 +0000

When running a Docker container on ECS Fargate, persistent storage is often a necessity. I initially attempted to solve this by manually creating the required directory on EFS using a Lambda-backed custom resource. While this worked, it introduced unnecessary complexity. Through experimentation, I discovered a more elegant solution—using EFS access points. In this post, I'll walk through my journey, the challenges I faced, and how I ultimately simplified the setup with fewer resources and less maintenance.

Setting up EFS

When you start a container on ECS Fargate, you must define a TaskDefinition. This definition is used to start the container. Within this definition, you can define Volumes. These volumes can then be mounted in the container by specifying a mount point per container.

So far, so good! But I quickly ran into an issue. The task could not be started because the path on EFS did not yet exist. I tried to mount /my-task/data on /data in the container. However, the /my-task/data path does not exist on the EFS drive. The easy solution would be to hook it up to an EC2 instance and create the folder from there. You might think that the problem is solved, but this is not the solution to the underlying problem. As you could have read in my previous blog, I like to be able to spin up multiple versions of a stack. With this manual step in between, you can’t do that.

How about a custom resource?

We know we need to create a folder on the EFS drive. A Lambda function could do this, so I started implementing a custom resource. The idea is very simple: create a Lambda function that mounts the root of the EFS, and then the Lambda code would simply make the folder on EFS. We mark this custom resource as a dependency on the ECS Fargate service to ensure that the folder exists before we start the container and we are done.

I fully implemented this flow, and it worked perfectly. When you deploy your stack, the custom resource does its thing, and the container can be launched successfully.

Is there a better way?

When implementing the custom resource, I needed to create an access point. A Lambda function can only mount an EFS drive through an access point. This access point allows you to define a path on EFS and the user and group ID used to access this path. If you supply /my-lambda/ as the path, the function can only see the content of the my-lambda folder. This is great for protecting other paths from being accessed. From the lambda functions perspective, you don’t see the my-lambda folder you only see the content.

Why is this so important, you might ask? Well, this path is created on the EFS drive for you. I looked at the ECS documentation, and you can also use an access point for container volumes. Using the access point to define the /my-task/data, I set the uid and gid to the same ID the user executed in the container. We could run the container without the need for a custom resource, so I deleted the function from my template.

Conclusion

Experimentation often leads to better solutions. My initial approach—using a Lambda function to create the necessary EFS directory—worked but added unnecessary complexity. By leveraging EFS access points, I achieved the same result with a cleaner, more maintainable setup. This experience reinforced an important lesson: the simplest solution is often the best. When working with cloud infrastructure, it's always worth exploring built-in features before resorting to custom implementations.

Photo by Nataliya Vaitkevich

Streamlining Workflows with Feature Branches and Logical Stacks

Joris Conijn — Sun, 26 Jan 2025 14:10:55 +0000

Efficient collaboration and streamlined deployment processes are crucial in modern development workflows, especially for teams working on complex projects. Feature branches and stack-based development approaches offer powerful ways to isolate changes, test effectively, and ensure seamless integration. However, proper strategies can make managing resources, dependencies, and environments challenging. This blog explores how to optimize feature branch workflows, maintain encapsulated logical stacks, and apply best practices like resource naming to improve clarity, scalability, and cost-effectiveness.

Working with feature branches

I have a strong preference for encapsulating things into logical stacks. These stacks should have a minimal number of dependencies. And you should be able to deploy multiple instances. Let me first explain why you want to deploy multiple instances of the same stack. The answer is quite simple! Feature branches! If you work in a team with multiple people, you might work on different stories. Therefore, you don’t want to update the same environment with your changes. My change might break your change and vice versa. Detecting why something failed becomes more challenging in this case. By deploying your own environment, your changes will only impact your environment. When you are done, you can thoroughly test your changes before merging them into the main branch.

Working with multiple stacks

You can use multiple stacks, each with its area of responsibility, to keep things small and simple. However, this also introduces the danger of breaking encapsulation and introducing dependencies, such as an RDS database. You need to create it in a base stack and pass it to other stacks. This is needed because the components in the other stacks will need the endpoint to connect to it, creating a dependency.

This example applies to the more traditional lift and shift approaches. However, we always advise modernizing to reap all the benefits of the cloud. Why? Simple: In the example, we needed an RDS instance. If you create an RDS instance per stack, you will have multiple instances and must pay per RDS instance, driving up the cost. By switching to serverless, you pay for the usage. This will allow you to move the database into your stacks, removing the dependency. I explicitly say option because, in relational databases, it’s not always possible to split the database into two because of the nature of relational databases.

Use proper naming for your resources.

CloudFormation can handle naming resources for you. A naming schema follows this pattern: StackName + LogicalResourceId + RandomizedString. This ensures that you don’t end up with duplicate resource names. Some resources allow you to have duplicates, while others do not. So, having a guaranteed unique name is perfect from the perspective of having feature branches. But from a management perspective, this is bad. Most names are trimmed after several characters, and the generated long names are not that descriptive.

By introducing a prefix, you can have a clear name and still deploy your stacks multiple times. You simply pass the prefix as a parameter and prefix each resource name. So, for an IAM Role, that could be Joris-CheckoutProcess. The CheckoutProcess name describes what it is, a role used by, for example, a lambda function that processes the checkout. The prefix Joris makes it unique and provides information about who owns and created the resource. Using this pattern you avoid creating a scavenger hunt.

Conclusion

Adopting feature branches and logical stacks is an excellent way to foster efficient, error-free collaboration in team environments. You can enhance your development workflow by deploying isolated environments for individual changes, embracing serverless solutions to reduce dependencies, and implementing descriptive naming conventions. These practices streamline debugging and testing and ensure resource management remains clear and cost-efficient. Whether you’re modernizing applications or maintaining legacy systems, these strategies will help you harness the full potential of cloud-based development.

Photo by Min An

Stop organizing scavenger hunts in your cloud infrastructure

Joris Conijn — Sun, 19 Jan 2025 09:59:55 +0000

A CloudWatch alarm is triggered. Now what? I am not the first person to tell you that observability is essential to your cloud infrastructure. You are not done when you have set up CloudWatch alarms!

Who will act on those alarms?

Observing some metrics and raising an alarm if a certain threshold is breached is just the start! You must also consider who will act if this alarm is breached. In some cases, you might be able to automate your actions. For example, you could think of a high CPU load on your application servers. The CloudWatch alarm is then triggered, and you can scale up your cluster.

These auto-remediation actions work very well for all known issues that can go wrong. You can detect them and build remediation actions that you trigger based on these CloudWatch alarms.

But in some cases, you just need a human to jump online and figure out what happened. When this happens, you don’t want to start a scavenger hunt in your AWS account to find the problem. You want a clear starting point and, from there, guidance in the right direction.

CloudWatch Dashboards

CloudWatch dashboards can guide you in the right direction during production issues. For example, assume we have a SQS Queue that contains messages. A lambda function is processing these messages. If the process fails, it will retry, and after three attempts, it will deliver the message to the dead-letter queue.

When one or more messages are in the dead-letter queue, a CloudWatch alarm is triggered, and we know something went wrong in our application.

This alarm can trigger an SNS topic, and from there, you can reach an engineer who can investigate this issue. A dashboard that shows these components makes it easy to navigate to the issue.

Here, you can see the dead-letter queues from our application. MyLambda is in an alarm state, and on the right, you see a link that brings you directly to that function's LogGroup.

It turns out that someone pushed code with an exception. We fixed the code and started re-driving the messages in the dead-letter queue back to the original queue. The lambda function is now processing the messages as it should, and the alarm will go back to an OK state.

Conclusion

When you build cloud infrastructure, you also need to think about what can go wrong. Design for failure, build auto-remediation for all known failures and loop in a person when unknown issues appear. But be kind for your fellow engineer and don’t sent him on a scavenger hunt. Guide him to problem and give context about the failures in a CloudWatch dashboard.

Photo by Gill Heward.

Boost Your Productivity with awscurl: Simplifying IAM-Secured API Testing in AWS

Joris Conijn — Sat, 21 Dec 2024 13:43:35 +0000

With modern clouds, you can build awesome things. You can bring your ideas to life within the hour, enabling experimentation and driving innovation. One of the significant advantages of the cloud is that you get a lot of security controls out of the box. But these security controls can also block you from being productive!

How are these security controls blocking me?

When you use AWS, you can interact with it through the console, sdk, or cli. But these all have one thing in common. They all use the same set of APIs to perform the actions requested by the user. These APIs are protected, and how authentication and authorization are done through the service IAM. When you develop a workload or work on a PoC, you will also use the IAM service.

Assume you want to build an internal API and use the API Gateway service. Enabling IAM authentication on the methods you define is easy. You can then give the appropriate roles access to your API. The consumer of this API only needs to add the AWSSigv4 header, and as long as the role policy allows the invocation of the API, it will work.

Various programming languages can be used to add the signature. But what if you want to test the API from your local machine or the cloud shell from the console? In those cases, you have a shell, and generating the signature becomes much harder. This will impact your productivity.

How can we unblock ourselves?

As explained in the previous section, the challenge is to generate the signature based on the current payload. In the past, I used a simple Python script to perform these API calls, but that always took some time and energy to build. I also felt demotivated when I encountered such situations.

The problem is not the script that you need to write. The problem is that you need to do it repeatedly because each situation is different. This goes against one of my principles, which is to automate everything! There should be a solution to this problem, and the good news is there is one!

Igor Okulist created a small tool called awscurl. This tool allows you to perform a curl command that automatically signs your API call. For example, when we use the scenario described in the previous section, you can simply do the following command:

export AWS_PROFILE=my-profile
awscurl --region eu-west-1 --service execute-api -X GET https://my-url-to-my-api

The example assumes that you have a valid profile configured called my-profile. Then, you do the same command as the original curl command. But you only add a region and a service option to the command. For the comparison, the normal curl command would look like this:

curl -X GET https://my-url-to-my-api

But this command will result in the following error:

{"message": "Missing Authentication Token"}

The awscurl command will inject the Authorization header that looks something like this:

Authorization: AWS4-HMAC-SHA256 Credential=XXX, SignedHeaders=XXX, Signature=XXX

When you try this out yourself, you can add the -v option, which will print out the headers sent and received. You will see that the API now accepts the signature and responds with the actual results.

You can use it to perform any API call that supports sigv4, but for the majority of services, the AWS cli tool is the best tool for the job. But for your own APIs or calls to your OpenSearch cluster, this tool is definitely a timesaver.

Conclusion

When you develop an internal API or have an OpenSearch cluster that uses IAM for authentication and authorization, testing calls towards these endpoints can be challenging as you will need to create a signature. The awscurl cli tool helps you create this signature, allowing you to focus on your business value instead of building signatures for tests.

Photo by Pixabay

Relative Python imports in a Dockerized lambda function

Joris Conijn — Sat, 14 Dec 2024 14:08:02 +0000

Relative Python imports can be tricky for lambda functions. I wrote a blog on this 3 years ago. But recently, I ran into the same issue with Dockerized lambda functions. So, I figured it was time for a new blog!

You can follow along with the steps or look at the result directly on GitHub.

Project setup

Make sure you installed the AWS CDK cli.

brew install aws-cdk

Initialize the project:

cdk init app --language=typescript

Lambda setup

First we will need to create the file and folder structure:

mkdir -p lib/functions/hello-world/hello_world
touch lib/functions/hello-world/hello_world/__init__.py
touch lib/functions/hello-world/requirements.txt
touch lib/functions/hello-world/Dockerfile

Now you will need to fill the Dockerfile, like this:

FROM public.ecr.aws/lambda/python:3.12
COPY requirements.txt .
COPY hello_world ${LAMBDA_TASK_ROOT}/hello_world
RUN pip install --no-cache-dir -r requirements.txt
CMD ["hello_world.handler"]

We are using a Python base image that is based on Python 3.12. Next, we will copy in the requirements.txt file and the source code. We will install all dependencies listed in the requirements.txt file and make sure that the handler method is set as the CMD.

Next, we will need to fill our Python files with some code. In the __init__.py file, you can place the following content:

from typing import Dict, Any
# Example of the relative import
from .business_logic import business_logic


def handler(event: Dict[str, Any], context: Any) -> Dict[str, str]:
    # User the method that is imported based on a relative path
    name = business_logic(event)

    return {
        "Name": name,
        "Message": f"Hello {name}!",
    }

__all__ = [
    "handler"
]

Afterward, we will need to fill the business_logic.py file:

from typing import Dict, Any


def business_logic(event: Dict[str, Any]) -> str:
    return event.get("name", "World")

NOTE: The code used here could use multiple relative imports. This is possible because it is in a separate package. This example only shows one example in the __init__.py file. However, you can use multiple files here to improve the maintainability of your project.

For this example, I don't need any dependencies, so we can keep the requirements.txt file empty. I included it in this example to illustrate how you can include dependencies as well.

Create the Lambda function using IaC

Our folders and files are in place, so it is time to add the Lambda function to the CDK construct. You can simply add it like this:

    new lambda.Function(this, 'Function', {
      functionName: "hello-world",
      code: lambda.Code.fromAssetImage("lib/functions/hello-world", {
        platform: ecr_assets.Platform.LINUX_ARM64,
      }),
      runtime: lambda.Runtime.FROM_IMAGE,
      handler: lambda.Handler.FROM_IMAGE,
      architecture: lambda.Architecture.ARM_64,
      timeout: cdk.Duration.seconds(15),
      memorySize: 128,
    });

For this to work, you also need the following imports:

import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as ecr_assets from 'aws-cdk-lib/aws-ecr-assets';

Note that we make sure that the code directory points to the directory that contains the Dockerfile and that we select the ARM platform for both the code and the function itself.

Testing the lambda function locally

Fast feedback is important, so there might be cases where you need to run the container locally. For this, you first need to build the container:

docker build --platform linux/arm64 \
  -t hello-world:latest \
  -f ./lib/functions/hello-world/Dockerfile \
  ./lib/functions/hello-world

Note that this command can be executed from the project's root. Next, we need to make sure it's running before we can invoke it:

docker run --platform linux/arm64 -p 9000:8080 hello-world:latest

Afterwards, you can invoke the function as follows:

curl http://localhost:9000/2015-03-31/functions/function/invocations -d '{"name": "Joris"}'

Conclusion

Relative imports can be tricky! You need to place your code in a package. This allows you to do relative imports within your own package. This will enable cleaner code, as you can split responsibilities into multiple files, making it easier to manage and maintain.

Photo by Kaique Rocha

Avoid Costly Loops in AWS Step Functions

Joris Conijn — Sat, 12 Oct 2024 12:16:58 +0000

We all played around with AWS StepFunctions in our careers. It is a fantastic orchestration tool! But there are some scenarios where your cloud bill can explode in your face! In this blog, I will walk you through how you can end up in this situation and, maybe more importantly, how to avoid it!

Looping

When you look at this pattern, you might think, okay, we wait for 5 seconds and use lambda to query something. If the response is true, we are done; if not, we will wait again and check the status once more. The problem with this pattern is that the response might never be true, causing an endless loop, which will generate cost.

There are no endless loops in StepFunctions

If you find yourself in such an “endless” loop, AWS will stop executing once you have exceeded 25,000 events. Even if you implement it, it will eventually be stopped by AWS. But each loop has 3 transitions, and you must pay for these transactions. They are pretty cheap, but the problem lies within this loop, especially if this pattern is part of a map distribution that allows you to do 10,000 parallel executions.

Assume your StepFunction is invoked based on an S3 event trigger. We upload 100 files in our S3 bucket. This will trigger 100 executions resulting in 3,000,000 state transitions each 5 seconds until they exceed the 25,000 events.

You can already see that this will ramp up your AWS bill.

The callback pattern

You can solve this by implementing a callback pattern. It’s quite simple: You will place a message in an SQS Queue. You can then use any consumer to do the task at hand. When it’s done, the consumer notifies the step function execution and passes the result back.

This is a known pattern, but we design our systems for failure! The cool thing about a SQS Queue is its retry mechanism.

Now, we can design it as follows:

The StepFunction will place the task in the SQS Queue.
The Processor will receive a batch of messages
The function will process each message, and for each, it will:
1. Send a heartbeat to the StepFunction execution. This prevents the StepFunction from timing out.
2. If the message is invalid, we can fail the StepFunction execution directly.
Depending on the outcome, the function will:
1. When a failure occurs, the message will be kept in the queue.
2. This will trigger the retry mechanism based on the visibility settings on the queue.
3. When the message is processed successfully, we will:
4. Send a success to the StepFunction execution.
5. Remove the message from the queue.
When the message has reached the maximum number of retries, it will be delivered to the dead letter queue.
The Dead Letter Queue Processor will receive the message.
It will fail the StepFunction execution with some additional context.
Remove the message from the dead letter queue.

Why is this a good design? It accounts for failure! We left the actual processing job out of scope here. However, the lambda processing can fail for many reasons:

It can reach AWS Service limits.
It might depend on another external system that is unavailable.
You’re scaling so fast that the serverless components can’t handle the load yet.

You have a better chance of succeeding with the SQS Queue and the retry mechanism. If the retries are exhausted, the dead letter queue will mark the failure in the step function itself. This will keep all the execution information in a single place for you to analyze.

Conclusion

Avoid creating loops in your StepFunctions. Use the callback pattern instead! Design your flows to be robust and expect failures to happen!

Photo by Steve Johnson

The post Avoid Costly Loops in AWS Step Functions appeared first on Xebia.