DEV Community: Matheus das Mercês

AWS Lambda Durable Functions on Hexagonal Architecture: The Pattern You’ve Been Looking For

Matheus das Mercês — Wed, 25 Feb 2026 06:45:45 +0000

Yes, you read it right. When building serverless applications on AWS, one little thing seems to be forgotten in 2026: design patterns. And that's especially true when using Lambda Durable Functions and its new open-source Durable execution SDK.

And no, this is not another "Step Functions vs Lambda Durable Functions" comparison. In this article, we will not look back. We will explore how you can build a strong foundation for Durable Functions with Hexagonal Architecture, from a developer's perspective, and why this pattern might be the missing piece for building durable applications.

Introduction

At AWS re:Invent 2025, AWS introduced Lambda Durable Functions with an interesting premise: build like a monolith, deploy to microservices.

As an all-time big fan of the microservices approach, I have to admit: I got super excited that we can now build a Lambdalith without a guilty conscience.

A little over a year ago, I wrote an article explaining how to refactor a Lambdalith to microservices, using Hexagonal. What was considered before as an anti-pattern, it is fascinating that Durable Functions allows us now to do the opposite, but now, with the microservices benefits.

Plus, at first glance, it looked like an immediate replacement for AWS Step Functions, meant for developers. And that's what we've been seeing the community doing so far: comparing both services and exploring ways of migrating existing state machines to Durable Functions.

The new Durable execution SDK is powerful, and it can do pretty much everything that you already have available in Step Functions, but when building a Lambda function that handles orchestration, it is also easy to fall into the trap of building the well-known Lambda Bogeyman: spaghetti code, which makes the application hard to explain and evolve.

The problem isn't Durable Functions.
The problem isn't its SDK.
The problem is the lack of boundaries.

And if there is one thing that I learned with my Durable endeavors, that thing is: now, we need better ways of organizing the application code.

The old-fashioned way of building software

Lately, something keeps hammering my mind: we need, more than ever, principles.

Back when coding tools were nothing but a daydream, we used to think differently. Coding was the most important skill of a developer, and the ability to structure the code in a way that is, among other things, readable and testable. In object-oriented programming, the SOLID principles, for instance, remain a great starting point for designing clean software.

But SOLID alone was never the end goal.

You can absolutely apply SOLID principles when building with Lambda, but when orchestration becomes central, clear separation of concerns matters even more. That’s where Hexagonal Architecture comes in.

Hexagonal Architecture

Hexagonal Architecture, also known as "Ports and Adapters," offers a way to modularize your application so it can be more flexible and maintainable. By isolating the core business logic from external systems, this architecture promotes separation of concerns, where the application's core logic isn't tightly coupled to any specific technology or service.

Core Logic (Domain): The core contains the application's core business rules, completely isolated from the outer layers.
Ports: Defined interfaces that describe actions available to the core.
Adapters: Connect external systems to the application's core through ports, making it easy to switch out databases, API integrations, or other dependencies without impacting the core logic.

The real strength of Hexagonal Architecture is the boundaries it creates. It keeps business logic isolated, dependencies replaceable, and infrastructure concerns at the edges.

That becomes especially important when we introduce the Durable execution SDK. It brings powerful workflow capabilities, but it also introduces execution-specific mechanics that should be kept separate from the rest of your code.

Hexagonal Architecture doesn’t remove that complexity. It gives it a place to live.

Durable 🤝 Hexagonal

Durable Functions changed something important: the what and the how now live in the same place.

With regular Lambda functions, we mostly wrote the what: validate an order, process a payment, update a record. How it was executed wasn’t something we had to think much about, as this used to be part of our infrastructure code (A.K.A Step Functions). Plus, what was before split into microservices, now it can be part of a single monolith, as Durable Functions gives us "distributed system reliability".

More importantly, with the Durable execution SDK, the how(s) is part of the code. Parallel steps, maps, and child contexts all sit next to the business logic. That’s where it can get confusing.

Hexagonal Architecture is not a silver bullet, but it allows us to separate those concerns a bit. We can make the domain stay focused on what the system does (with a little bit of how). The workflow base layer handles how it runs. The adapters handle external calls.

Durable Functions gives us reliability in a monolith. Hexagonal keeps the structure clean. And when that happens, the SOLID principles can make sense again.

Inversion of control (IoC)

This is where things start to get really interesting.

"In software design, inversion of control (IoC) is a design principle in which custom-written portions of a computer program receive the flow of control from an external source (e.g., a framework). In procedural programming, a program's custom code calls reusable libraries to take care of generic tasks, but with inversion of control, it is the external code or framework that is in control and calls the custom code."
Source: https://en.wikipedia.org/wiki/Inversion_of_control.

In short, IoC is the practical tool that makes Hexagonal Architecture work. It's how you implement the D in SOLID, where abstractions should not depend on details. Details should depend on abstractions.

And how to leverage IoC when building with Lambda Durable Functions?

A concrete example

You are a developer. You need to build a data pipeline with the requirements to:

Ingest some data, transform it, and store it into a database.
Support more than one data source, so the ingestion and transformation code will be different depending on the data type.
Build the application in a modular way so that it's easier to evolve in the future with potential new data types.

After some conversations with your team, you decided to build it like a monolith, so that you don't have the cognitive overhead of splitting the application into microservices. Single code base, single deployment, super straightforward.

Of course, when someone asks how exactly you are going to design it, the universal engineering answer applies:

"It depends."

Although in fact it does depend, you are a great developer. You want to build a future-proof application, and despite all possibilities, you decided to:

Use Lambda Durable Functions. Build a monolith and use Parallel execution with automatic retries for each data type for reliability.
Leverage Hexagonal Architecture to keep the code structure clean.
Apply IoC (Inversion of Control) so that different ingestion and transformation code can be injected without modifying the orchestration logic, avoiding condition-heavy, tightly coupled code.

Now let’s look at what that means in practice.

It's all about code

To demonstrate, I have used InversifyJS, a library used to create inversion of control (IoC) container for TypeScript. An IoC container uses a class constructor to identify and inject its dependencies. While Hexagonal Architecture is particularly well-suited for typed languages, it is language-agnostic and can be implemented in any language or framework of your choice.

1️⃣ The IoC container: wiring behavior, not hard-coding it

The container defines multiple implementations for the same abstractions.
Each data source and mapper is bound by name, allowing us to switch behavior based on context instead of conditionals.

const container: Container = new Container();

// Bind multiple data source implementations with names
container.bind<IDataSource>(TYPES.DataSource).to(CustomerDataSource).whenNamed('customers');
container.bind<IDataSource>(TYPES.DataSource).to(ProductDataSource).whenNamed('products');
container.bind<IDataSource>(TYPES.DataSource).to(OrderDataSource).whenNamed('orders');

// Bind multiple data mappers implementations with names
container.bind<IDataMapper>(TYPES.DataMapper).to(CustomerDataMapper).whenNamed('customers');
container.bind<IDataMapper>(TYPES.DataMapper).to(ProductDataMapper).whenNamed('products');
container.bind<IDataMapper>(TYPES.DataMapper).to(OrderDataMapper).whenNamed('orders');

container
    .bind<Factory<{ dataSource: IDataSource; dataMapper: IDataMapper }, [string]>>(TYPES.DataSourceFactory)
    .toFactory((context: ResolutionContext) => {
        return (named: string) => {
            const dataSource: IDataSource = context.get<IDataSource>(TYPES.DataSource, {
                name: named,
            });

            const dataMapper = context.get<IDataMapper>(TYPES.DataMapper, {
                name: named,
            });
            return {
                dataSource,
                dataMapper
            };
        };
    });

Here, IoC allows us to inject different ingestion and transformation strategies without changing the workflow structure. No if (type === 'customers') spread all over the codebase.

2️⃣ The Lambda entrypoint: keeping Durable at the edge

This file acts as the Lambda entrypoint. It wraps the handler with Durable execution and delegates the actual logic to a resolved use case. With that, we have access to the DurableContext, and we use it later down the line.

import { withDurableExecution, DurableContext } from '@aws/durable-execution-sdk-js';
import { container, TYPES } from '../container/inversify.config';

export const durableFunction = withDurableExecution(async (event: any, context: DurableContext): 
Promise<any> => {
  const useCase = container.get<any>(TYPES.DurableFunction);
  return useCase.handler(event, context);
});

3️⃣ The workflow base layer: centralizing the initial orchestration mechanic

This abstract class centralizes the Durable parallel execution pattern. It takes an eventContexts array and runs the same workflow in parallel for each data type, with a parent context called execute-contexts-in-parallel.

{
  "eventContexts": ["customers", "products", "orders"]
}

import { DurableContext } from '@aws/durable-execution-sdk-js';

abstract class DurableParallelAbstractHandler {
  async handler(event: DurableFunctionEvent, context: DurableContext):
    Promise<DurableFunctionResponse> {
    try {
      const contextsToBeExecuted = event.eventContexts.map((eventContext) => ({
        name: eventContext,
        func: async (ctx: DurableContext) => {
          return await this.execute(eventContext, ctx)
        }
      }));

      const results = await context.parallel('execute-contexts-in-parallel', contextsToBeExecuted);

      console.log('Durable Function completed successfully');

      return {
        success: true,
        response: results,
        timestamp: new Date().toISOString(),
      };
    } catch (error: any) {
      console.error('Durable Function failed:', error);

      return {
        success: false,
        error: error.message,
        timestamp: new Date().toISOString(),
      };
    }
  }

  protected abstract execute(event: any, childContext: DurableContext): Promise<any>;
}

export default DurableParallelAbstractHandler;

Each value in eventContexts corresponds to a named binding in the IoC container (for example, customers resolves to CustomerDataSource + CustomerDataMapper). The workflow structure remains the same, only the injected behavior changes.

Because this class is abstract, it will be extended by other Lambdas that require the same parallel orchestration pattern. The parallel mechanics live in one place. Concrete implementations only need to provide the execute method.

4️⃣ The concrete Durable use case

This class implements the actual use case while inheriting orchestration mechanics from the base class.

Plus, it is not pure domain logic. It is the workflow use case layer.
Its responsibility is to coordinate execution, not to define how ingestion or transformation works.

@injectable()
class DurableFunction extends DurableParallelAbstractHandler {
    private dataFactoryInstance: { dataSource: IDataSource; dataMapper: IDataMapper };

    constructor(
        @inject(TYPES.DataSourceFactory) private dataSourceFactory: (named: string) => { dataSource: IDataSource; dataMapper: IDataMapper },
        @inject(TYPES.Storage) private storage: IStorage,
    ) {
        super();
    }

    async execute(dataSource: string, context: DurableContext): Promise<any> {
        const dataSourceType = dataSource || 'customers';

        await context.step('create-data-source-factory', async () => {
            this.dataFactoryInstance = this.dataSourceFactory(dataSourceType);
        });

        const rawResponse = await context.step(`fetch-data`, async () => await this.dataFactoryInstance.dataSource.fetch());

        await context.map(rawResponse, async (ctx, item, _index) => {
            ctx.runInChildContext(`process-item-${item.id}`, async (childCtx) => {
                let transformedData: DomainResponse;

                childCtx.step('transform-data', async () => {
                    transformedData = this.dataFactoryInstance.dataMapper.mapToDomain(item);
                    return transformedData;
                });

                childCtx.step('store-transformed-item', async () => {
                    const storageKey = `processed-data/${transformedData.entity.type}/${transformedData.entity.id}-${Date.now()}.json`;
                    await this.storage.put(storageKey, JSON.stringify(transformedData, null, 2));
                }
                );
            });
        });

        return {
            message: "Data processed and stored successfully",
        };
    }
}

export default DurableFunction;

What matters here is isolation. If tomorrow the ingestion logic changes for customers, or a new data type is introduced, the Durable workflow does not need to change, but only the injected implementation. The orchestration remains untouched.

For those who come after

What we built here is a monolith with distributed system reliability, by keeping a single codebase while letting Durable Functions handle the hard parts: parallel execution, retries, etc.

And the payoff shows up immediately in the Durable operations graph: you can literally see the top-level execute-contexts-in-parallel, from the abstract class, each branch running its own context (customers/products/…), and the steps and map iterations underneath.

A nice side-effect of this pattern is testing. You can test the container (IoC bindings) independently from workflow execution, and you don’t need to touch the orchestration layer when you change a mapper or data source.

IoC container test (short version):

import 'reflect-metadata';
import { container } from '../../../src/container/inversify.config';
import TYPES from '../../../src/container/types';
import IStorage from '../../../src/interfaces/storageIF';

describe('IoC container wiring', () => {
  it('resolves the data factory for a named context', () => {
    const factory = container.get<(name: string) => { dataSource: any; dataMapper: any }>(TYPES.DataSourceFactory);

    const { dataSource, dataMapper } = factory('products');

    expect(dataSource).toBeDefined();
    expect(typeof dataSource.fetch).toBe('function');

    expect(dataMapper).toBeDefined();
    expect(typeof dataMapper.mapToDomain).toBe('function');
  });

  it('resolves infrastructure providers', () => {
    const storage = container.get<IStorage>(TYPES.Storage);
    expect(storage).toBeDefined();
  });
});

And your Durable function test file, also in a short version, using the Durable Execution SDK JS Testing library:

import 'reflect-metadata';
import { LocalDurableTestRunner } from '@aws/durable-execution-sdk-js-testing';
import { container, TYPES } from '../../../../src/container/inversify.config';
import { durableFunction } from '../../../../src/example-app/handlers';
import { DurableFunctionEvent } from '../../../../src/example-app/durableAbstractHandler';
import IStorage from '../../../../src/interfaces/storageIF';
import LocalStorage from '../../adapters/storage/local/localStorage';

describe('Durable workflow', () => {
  let runner: LocalDurableTestRunner;

  beforeAll(async () => {
    process.env.ENVIRONMENT = 'test';
    (await container.rebind<IStorage>(TYPES.Storage)).to(LocalStorage).whenDefault();
    await LocalDurableTestRunner.setupTestEnvironment({ skipTime: true });
  });

  beforeEach(() => {
    runner = new LocalDurableTestRunner({ handlerFunction: durableFunction });
  });

  afterAll(async () => {
    await LocalDurableTestRunner.teardownTestEnvironment();
    delete process.env.ENVIRONMENT;
  });

  it('runs multiple data sources in parallel', async () => {
    const event: DurableFunctionEvent = { eventContexts: ['customers', 'products', 'orders'] };

    const execution = await runner.run({ payload: event });

    expect(execution.getStatus()).toBe('SUCCEEDED');
    expect(execution.getResult()?.success).toBe(true);
  });
});

The full example, including the complete test files, adapters, and the full hexagonal implementation, can be found here.

Conclusion

Lambda Durable Functions are more than a replacement for Step Functions. They are meant for a different kind of problem and a different kind of developer experience. When you choose Durable, you are choosing to write your workflows in code. And once orchestration lives in your codebase, boundaries become essential.

Durable Functions gives you distributed system reliability inside a monolith: parallel execution, maps, retries, etc. But reliability alone is not enough. Without clear organization, it’s easy to mix workflow mechanics with business logic and slowly create something hard to understand.

Also, Durable Functions work just as well for simple workflows as for complex ones. The difference is not in the workflow size or the number of steps, it’s in how you structure the code around it. Hexagonal Architecture helps keep things in place, while IoC helps keep dependencies clean.

Together, they allow you to build a Lambdalith that is reliable at runtime and maintainable in the long run.

The monolith vs microservices debate is still relevant. Durable Functions change how we approach it. You can build a great monolith, but only if you design it properly.

Durable Functions are powerful, but they require discipline. Design patterns were never optional; we just stopped talking about them. It doesn’t remove the need for architecture. It makes it impossible to ignore.

You Don’t Need a Construct for That: Best Practices for Serverless Infrastructure with AWS CDK Blueprints

Matheus das Mercês — Thu, 17 Jul 2025 07:29:16 +0000

If you use AWS CDK (Cloud Development Kit) to create infrastructure as code, you're probably familiar with Constructs and Aspects. Have you heard about CDK Blueprints? You can inject properties at L2 Constructs and apply best practices to all your resources at scale.

In this article, we are going to define best practices for a few resources by understanding the real meaning of CDK building blocks, see how easy it is to define Blueprints in CDK, explore the benefits of property injection, and understand why you don't need a Construct for applying best practices.

Introduction

"Everything in CDK is a Construct", but not everything needs to be a Construct.

No, this article is not only about CDK Constructs. What you're going to read briefly in the next session is a rough summary of the Constructs documentation, the way AWS wanted us to understand them, and understanding the semantics of it is going to make a big difference in your IaC (Infrastructure as Code) project.

For what would you need a Construct?

According to the documentation from CDK, Constructs are the basic building blocks of a CDK app, and they are a collection of one or more resources to be deployed via CloudFormation.

Moreover, Constructs are also defined in 3 levels (L1 for CloudFormation definition, L2 for a small layer of abstraction, and L3 for higher abstraction or often a collection of AWS resources). The documentation also explains that you can create your own Constructs for a specific pattern (example, S3 Bucket to SNS topic).

The purpose of them is to help you reach the desired state of your infrastructure as code by grouping resources and logic into ~basically~ classes.

Things you can do with Construct, but you don't need to (or should not?)

What you just read is all that CDK Constructs mean, according to AWS. But it's 2025, and we are creative people. Let's have a look at this example:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as logs from 'aws-cdk-lib/aws-logs';

export interface ValidatedLogGroupProps extends logs.LogGroupProps { }

export class ValidatedLogGroup extends Construct {
    public readonly logGroup: logs.LogGroup;

    constructor(scope: Construct, id: string, props: ValidatedLogGroupProps) {
        super(scope, id);

        if (props.retention !== logs.RetentionDays.TWO_WEEKS) {
            throw new Error('Retention must be set to TWO_WEEKS');
        }
        if (props.removalPolicy !== cdk.RemovalPolicy.DESTROY) {
            throw new Error('Removal policy must be set to DESTROY');
        }

        this.logGroup = new logs.LogGroup(this, 'LogGroup', props);
    }
}

Since you write CDK with, in this example, TypeScript, you can validate the Construct props and throw exceptions if they are not compliant with standards defined by you. In practice, this means:

You created a new layer of abstraction upon the L2 Construct of LogGroup, which contains default props from CDK Lib that might change between versions. This can lead to operational overhead when bumping versions.
You will not allow LogGroups to be created without a two-week retention (in this hypothetical situation, this configuration is mandatory).
You write it, you maintain it. The extra logic is going to be maintained by you, even though it is a small class.

Although this is possible, you need to ask yourself the questions: "Is it worth it? Are there other options? Am I misusing CDK Constructs?"

Gladly, we have alternatives

The beauty of imperative IaC is that you can do pretty much everything you want, preferably following the semantics of the tool.

Aspects

Hold on, we are almost there. Let's first see if Aspects help us validate props values.

The documentation starts explaining that Aspects are "a way to apply an operation to all constructs in a given scope, or it could verify something about the state of the constructs, such as making sure that all buckets are encrypted". Ok, things are getting more interesting.

The purpose of an Aspect is to modify resources at the CloudFormation level based on what was defined by a higher-level construct (L2) or throw exceptions if they deviate from a standard, pretty much what we tried to achieve with Constructs above. For example:

class BucketVersioningChecker implements IAspect {
  public visit(node: IConstruct): void {
    // See that we're dealing with a CfnBucket
    if (node instanceof s3.CfnBucket) {

      // Check for versioning property, exclude the case where the property
      // can be a token (IResolvable).
      if (!node.versioningConfiguration
        || (!Tokenization.isResolvable(node.versioningConfiguration)
            && node.versioningConfiguration.status !== 'Enabled')) {
        Annotations.of(node).addError('Bucket versioning is not enabled');
      }
    }
  }
}

// Later, apply to the stack
Aspects.of(stack).add(new BucketVersioningChecker());

Seems like we are getting there, but here are a few important points from the Aspects semantics:

You want to modify L1 Constructs (CloudFormation level)
You want to inspect (not modify) L2 Constructs props
You have to navigate the whole tree of resources and check one by one to inspect

Since we want to leverage the existing L2 LogGroup construct from CDK and modify the props to be compliant with your standards, we could use Aspects and modify the CloudFormation template after the fact - again, because it is possible does not mean it is the best way.

Blueprints

You know the drill: Let's see what the documentation says about CDK Blueprints.

"Use AWS CDK Blueprints to standardize and distribute L2 construct configurations across your organization. With Blueprints, you can ensure that AWS resources are configured consistently according to your organizational standards and best practices. For example, you can automatically enable encryption for all Amazon S3 buckets, apply specific logging configurations to all AWS Lambda functions, or enforce standard security rules for all security groups."

This is exactly what we are trying to achieve, and this is not me saying, but how AWS wants us to apply resource best practices.

Blueprints are powered by property injection, and each Blueprint is responsible for applying default properties to a specific L2 Construct when they are instantiated. We can think about security standards, cost optimization, compliance requirements, etc.

Let's get down to business and create some examples of Blueprints for a few resources:

CloudWatch Log Group

export class LogGroupPropsInjector implements IPropertyInjector {
    public readonly constructUniqueId: string;

    constructor() {
        this.constructUniqueId = LogGroup.PROPERTY_INJECTION_ID;
    }

    public inject(originalProps: LogGroupProps, _context: InjectionContext): LogGroupProps {
        return {
            ...originalProps,
            removalPolicy: RemovalPolicy.DESTROY,
            retention: RetentionDays.TWO_WEEKS
        };
    }
}

Lambda Function

export class MyNodejsFunctionPropsInjector implements IPropertyInjector {
    public readonly constructUniqueId: string;

    constructor() {
        this.constructUniqueId = NodejsFunction.PROPERTY_INJECTION_ID;
    }

    public inject(originalProps: FunctionProps, _context: InjectionContext): FunctionProps {
        return {
            ...originalProps,
            runtime: Runtime.NODEJS_22_X,
            memorySize: 128,
            timeout: Duration.seconds(30),
        };
    }
}

SQS Queue

export class MyQueuePropsInjector implements IPropertyInjector {
    public readonly constructUniqueId: string;

    constructor() {
        this.constructUniqueId = Queue.PROPERTY_INJECTION_ID;
    }

    public inject(originalProps: QueueProps, _context: InjectionContext): QueueProps {
        return {
            ...originalProps,
            visibilityTimeout: Duration.seconds(45),
            retentionPeriod: Duration.days(4),
        };
    }
}

Some extra remarks about CDK Blueprints:

Place default properties before …originalProps to allow overrides.
Place forced properties after …originalProps to prevent overrides.
Use CDK context to enable/disable injectors for testing.

The combination of the two

Since we are doing things "by the book" here, we also need to consider this sentence from the Blueprints docs:

"Blueprints are not a compliance enforcement mechanism. Developers can still override the defaults if needed. For strict compliance enforcement, consider using AWS CloudFormation Guard, Service Control Policies, or CDK Aspects in addition to Blueprints".

Even though it is possible to prevent overrides with Blueprints only (but remember the semantics!), you could combine it with Aspects in the following way:

With your Blueprints in place, your L2 constructs are going to have some properties injected during initialization.
If, for some reason, your Blueprints are not applied to the scope of a specific resource, you can have an Aspect as an extra layer of validation (Aspects inspection happens after construct initialization).
Your Aspect will then check if a specific property was not injected for a specific resource. You can throw an exception or add a warning message saying you could use a pre-existing Blueprint from your project.

Conclusion

In the realm of AWS CDK, while Constructs are fundamental building blocks for defining infrastructure as code, they are not always necessary for enforcing best practices. The introduction of CDK Blueprints offers a more streamlined and efficient approach to standardizing configurations across your organization. By leveraging property injection, Blueprints allow you to apply default properties to L2 Constructs, ensuring consistency in resource configurations without the need for additional abstraction layers.

Ultimately, the combination of Blueprints and Aspects allows for a balance between flexibility and compliance, enabling you to maintain best practices at scale while still allowing for necessary customizations. This approach aligns with AWS's vision of how infrastructure as code should be managed, providing a powerful toolkit for modern cloud infrastructure management.

References

Hands-On with Amazon Q Developer in GitHub (Preview): First Impressions

Matheus das Mercês — Mon, 05 May 2025 22:00:00 +0000

This week, Amazon released the preview of Amazon Q Developer in GitHub, allowing support across the software development lifecycle from coding, testing, and deploying to troubleshooting and modernizing applications.

I tested it, and I have to say—it's awesome! In this article, I will share my impressions and explain everything you can and can not do with the current version.

Introduction

Amazon Q Developer is a generative AI–powered assistant that helps developers and IT professionals build, operate, and transform software. It assists in writing, debugging, and optimizing code, as well as managing AWS resources and performing code transformations.

What's new?

The preview release of Amazon Q Developer in GitHub brings several new capabilities:

Feature development: Amazon Q Developer lets you generate code from natural language prompts, helping you build features, fix bugs, add tests, and refine logic directly within GitHub issues—saving time and reducing errors.
Code transformation: It modernizes outdated Java codebases by updating tech stacks and improving performance, all while maintaining original functionality and minimizing technical debt.
Code review: Amazon Q Developer automates code reviews in GitHub by analyzing pull requests, offering feedback, and suggesting fixes you can easily apply.

My experiment

Using Amazon Q Developer in GitHub feels just like assigning tasks to a real human being. It understands the context of repository issues, provides relevant code changes, and you can even interact with the assistant in the pull request. If you have a clear goal and you can give the proper instructions, Amazon Q Developer will most probably achieve it, of course, with some caveats.

To experiment with the agent, I have created two issues in a repository that uses Hexagonal Architecture (not like that matters, but I wanted to try somewhere as close as possible to a real-life repository). If you want to understand more about this design pattern, check out my blog post about it.

Creating a new issue

The first thing I wanted to try was to add a new adapter for SSM Parameter Store to my repository.

For this issue, I tried to be as much as clear as possible, but did not write much text. Remember, it is all about the prompt!

I selected "Amazon Q Developer Agent" under "Labels". For me, it felt like assigning this issue to a peer, but this time, not to a human but to an agent.

As soon as you add, you can see the update from the Amazon Q Developer agent:

In less than 2 minutes, I could see a new comment from the agent on the issue, and a link to a pull request!

Pull-request interaction

Amazon Q Developer created the pull request the way I usually do (and like): Clear description and link to which issue this PR resolves:

It also explains how to interact with the agent and how to request changes:

Reviewing the changes, there were only 4 files changed, so it was a very straightforward PR since this repository is not very complex.

The only thing I noticed was a violation of Hexagonal Architecture - the port (interface) created by Amazon Q Developer has "SSM" name on it, whereas it should not contain any service specifics in this layer. I was not expecting the agent to know this design pattern, so no big issue here. Then, I asked the agent to update it:

A few seconds after clicking on "Request changes", Amazon Q Developer added a comment to the PR:

And again, a couple of minutes later, voila:

Code review

Assigning the issue to Amazon Q Developer also gives it the freedom to review its code changes against security vulnerabilities:

And if something is found, Amazon Q Developer will suggest a change, and you can decide whether you accept or not:

Not perfect (yet)

I have to say, for its preview version, Amazon Q Developer in GitHub is awesome. There were a couple of things I personally would like to see improved in a later version, not in any specific order:

Code review for security issues and vulnerabilities happens on every iteration with the agent in a pull request, so you can eventually expect new suggestions every time you ask for a change.
Renaming files during your code review can be an issue for the agent. I asked a couple of times to rename/remove a file, but it can not understand the purpose of it really well.
Branch name can be an issue if you want your repository to follow branch name standards.
Source of issues now it is only GitHub. However, I wish to register issues in a third-party tool and have Amazon Q Developer pool them to GitHub somehow.

You can check both issues I've created for this experiment and its pull requests here.

Start using it now!

The first (and only) thing you need to do is install the Amazon Q Developer application in GitHub. There is no need to connect an AWS account.

You can choose to install it in all of your repositories or select the ones you want, and that is it! You can now create issues and assign them to Amazon Q Developer.

Conclusion

Amazon Q Developer in GitHub marks a significant step towards integrating AI into the software development lifecycle, not only in your IDE, but also in resolving issues defined in GitHub.

The interaction with the tool is great. If you are already used to GitHub issues and pull requests, using Amazon Q Developer will feel 100% natural to you.

While there are areas for improvement, it is good to remember that this is the preview mode, but the current capabilities are worth trying.

Have you tried it already? How was your experience? Leave a comment below, I would love to hear!

AWS Community Day Italy 2025: My Experience as an Attendee and Session Takeaways

Matheus das Mercês — Fri, 04 Apr 2025 08:49:42 +0000

I went to Italy to enjoy some holidays and my birthday, and it happened to be during the same week as the AWS Community Day Milan 2025, so of course, I did not miss the opportunity to attend - and the event was an absolute blast! Such a great atmosphere and amazing speakers, so it was a great opportunity to learn a lot and connect with like-minded AWS enthusiasts.

As someone who loves propagating knowledge, I've decided to write the key takeaways from my 3 favorite sessions and my thoughts about the event in general.

Welcome Booth

Right when entering the venue to claim my badge (I had to register a couple of days in advance) I got a bag with some cool AWS swags and information about the sponsors. More importantly, I received some vouchers for coffee. I mean, it is Italy, so the coffee is always good everywhere.

I got my double espresso, had a quick look at the space and seeing so many familiar faces from the international community, I immediately knew the day ahead was about to be awesome.

Keynotes

Kick-off

After breakfast and networking, Monica Colangelo (AWS Hero and event organizer) started the event by explaining the venue's logistics, the agenda, and so on. The information was very clear to me as an attendee, and it gave me the impression that the event was thoroughly planned and well-organized. I knew exactly what to expect, and the keynote speaker was about to get on stage!

Generative AI: tech du-jour or the next big thing?

The keynote speech was thrown by Massimo Re Ferrè, Director and Product Management at AWS.

It was such an interesting perspective when Massimo made me reflect on GenAI and LLMs throughout the whole session. He started the session by explaining when his "aha" moment was for GenAI - a simple Lambda@Edge function asked at the very beginning of ChatGPT. The code worked almost 100% and made him think "there is something here".

I could relate to Massimo as my "aha" moment was quite similar - a piece of code that before seemed impossible to write without a person, now an assistant would write within seconds - especially when Masismo explain the deepness of LLM nowadays, where "if you have a clear goal and ask the assistant, it will most probably achieve your goal" (think as a feature request).

"Where are these assistants going?" made me reflect on what I use LLM for nowadays. According to Massimo, LLM can make you go home early, and I agree. It is all about you letting "AI do the laundry and dishes for you to focus on art", not the opposite.

Approaching the end of the session, Massimo explores the risk management of GenAI. It can do a lot nowadays, so we have to be able to determine what benefits we want to extract out of it and what we are willing to give up to leverage the good parts of it.

If the mindset is not changed yet, we should. "The developer role will not disappear, it will change" and - the verb I like the most for this subject - evolve.

Sessions

Building Secure and Efficient SaaS Platforms on AWS Serverless

I have attended this session before in an AWS event in the Netherlands, delivered by Luciano Mammino and Guilherme Dalla Rosa, and I really liked it.

The session starts by explaining why multi-tenant architecture is important: It is mainly about optimizing resources and being more efficient by sharing that results across multiple customers.
The audience also has a good overview of the trade-offs between multi-tenant and single-tenant applications.

Then, they start a walkthrough in the solution built on AWS using API Gateway, an custom Lambda Authorizer, Cognito, and a DynamoDB table that is accessible through the gateway for the different tenants.

I like the idea of the "danger-zone" brought up by Luciano and Guilherme, when a tenant could possibly access data from a different tenant due to a bug, security issue, injection, or anything related. That happens because the differentiation between tenants happens directly in the query expression to DynamoDB and there is no distinguishing in the IAM role: the Lambda service role can freely query data on this table. "The Lambda has too much responsibility".

In my opinion, their solution is brilliant: they created a policy with well-defined permission boundaries, and, on the Lambda authorizer code, they narrowed down the permissions of this IAM Role based on the claims (tenant ID) in the token. This role now is assumed by the authorizer.

Now, ff there is a bug when tenant A is trying to access tenant B, it won't work anymore because the policy won't allow it. If there is a "noisy tenant", it is also possible to use an API Gateway usage plan to limit the number of requests for that specific tenant.

What I could extract from this session is that the example showed is a great example of how to apply least privilege permissions for a multi-tenant application without reinventing the wheel.

Event-Driven and serverless in world of IoT

The session, delivered by Jimmy Dahlqvist (AWS Hero), explored how event-driven and serverless architectures can address key challenges in IoT systems, including scalability, monitoring, latency, security, data volume, and cost per device.

Jimmy showed the audience a use case (a connecting entrance solution) where it must handle unpredictable traffic, long-running processing tasks, and strict budget constraints.

The initial architecture relied, amongst other components, on IoT rules for routing, small objects in S3, and debugging issues in DynamoDB due to query limitations. These bottlenecks led to a need for architectural changes, including improved components for routing, data storage, and debugging. Jimmy then started to dive deep into the architecture components, where he replaced some pieces of the solution to meet the requirements.

Jimmy concluded the session by summarizing the benefits of serverless and event-drive architectures in IoT environments, like, for example scalability, cost efficiency (pay-per-use model), and responsiveness of the solution.

Road to compliance: will your internal users hate your Platform Team?

Davide de Paolis, Engineering Manager of Platform Team, explained how the mindset shift between Software Engineers and Cloud Engineers can be hard and how to minimize the effects for the platform users.

When migrating to a more robust AWS Organization setup, there is a change of habit as well: now, engineers need to log in to multiple accounts and create multiple policies with SCPs, for example. He also explained the benefits of AWS Organization in a platform, which makes it easier to apply security controls, quotas, data isolation, cost allocation, etc.

When Davide navigates the audience through the mechanism of creating policies and enforcing tags on resource creation, for example, he created a good bridge to what was coming next: Enforce x inform.

More important than enforcing policies, communicating is crucial to make sure there is enough time for development teams to comply with the guardrails. Even more important: identifying who to inform. Davided showed a great example of an architecture of a solution responsible for identifying and notifying resource owners not compliant with the guardrails defined by the platform team.

The insightful idea of MVG (Minimum Viable Governance) brought up by Davide helps the team focus on what really matters, integrated based on feedback and changing priorities, and share goals and purpose. For him, a platform team can fail when they lose focus on the goals and create a disconnection between them and the development teams.

My key takeaway from this session is: A Platform Team, among other responsibilities, needs also to keep sharing knowledge, learning with each other, which takes time and patience, but it will create the desired connection with the development teams - without that, the "internal users will hate your platform team".

Honorable mentions

BuildersCard

Game time! Throughout the event, there was a game room, as shown above, where attendees could relax a bit and have fun between the sessions. The BuildersCards is an educational game that helps people understand how AWS services can work together to design well-architected applications. You don't necessarily need to have AWS knowledge or a technical background, although it is fun when you do because you can discuss the usage of the services during the game. It was also a great opportunity to meet new people in the community.

A Spotlight on Women Speakers

A special mention goes to the strong representation of women among the speakers, an encouraging step toward more inclusive conversations in our field. Hats off to the organizers for making it happen.

Conclusion

I have a bias to give my opinion about an AWS event in Italy, as I love the Italian culture and I am an AWS ~~nerd~~ enthusiastic, but I was happy to spend one day of my holidays in this event. Had so much fun and the sessions were great, so I really enjoyed the AWS Community Day Milan 2025. Connecting, sharing, and learning from the community is something that I truly enjoy doing.

Are Your AWS CloudTrail Costs Out of Control? Here’s Why

Matheus das Mercês — Mon, 17 Mar 2025 00:30:00 +0000

Are you happy with your CloudTrail bill? I asked that same question in my previous article about CloudWatch, and now, it is time to reflect on AWS CloudTrail.

In this article, I will explore possible reasons why you are overspending on CloudTrail, and discuss ways to keep the costs of CloudTrail well controlled.

Introduction

AWS CloudTrail is one of those services that delivers huge value to the AWS ecosystem while not spending tons of money. It enables auditing, security monitoring, and operational troubleshooting by tracking your user activity and API calls.

However, it is also one of those services that can get very expensive quite easily - especially because it tracks all activities in an AWS account/organization and generates audit logs.

Understanding how AWS charges you is a crucial step to avoid misconfiguration and keep compliance at the lowest price point.

The Serverless Paradigm

Yes, AWS CloudTrail is a serverless service. And that is because it is a fully managed service that automatically records AWS API activity and stores it in Amazon S3 or CloudWatch Logs. You don’t need to provision or manage servers - AWS handles everything behind the scenes.

Serverless is not just about the literal meaning of "Server-less" but also the principles behind it, for example, no infrastructure management and pay-per-use pricing. However, the trick part lies in the last one: the first copy of trails is free, but the second copy onwoards inccur charges, and this is where you might be spending quite some money.

CloudTrail pricing

Let's understand how AWS customers pay for CloudTrail. I want to focus on Trails, which is the subject of this article. On the CloudTrail pricing page, there are also details about Lake and Insights, which I will not cover here for now.

Do you want me to investigate and share insights about Lake or Insights for a future article? Let me know in the comments!

For management events delivered to S3, you pay $2.00 per 100,000 events delivered (after the first free copy)
For data events delivered to S3, you pay $0.10 per 100,000 events delivered
For network activity events delivered to S3, you pay $0.10 per 100,000 events delivered

NOTE: Amazon S3 charges apply and are not included in this analysis.

If you take a close look at the pricing and compare it to the trails in your AWS account, you can immediately see where the eventual high costs come from, so now, it is important to understand why.

Duplicated trails

The sentence I want you to reflect on from the pricing is "after the first free copy". What does that mean?

Let's imagine you set up organization trails in all member accounts. These trails go to an S3 bucket in a centralized logging account, for example. To deliver the same events to other destinations to allow different groups (for instance, developers, security, auditors, etc) to get their copy of these audit logs, you also created trails in the individual accounts. Although this is a valid use case, this can be costly - the first copy of these events is free of charge and you pay for the other ones - and that is why I call this duplicated trails. They generate a metric called PaidEventsRecorded, and this is how AWS charges you.

Still on the hypothetical AWS account, let's say this environment generates 5 million management events delivered to S3 per month:

The first trail, delivered to an S3 bucket that developers can access, is free of charge.
The second trail, delivered to an S3 bucket that the security team can access, will cost $100 (5,000,000 / 100,000 * $2.00 = $100)
The third trail, delivered to an S3 bucket that the auditors team can access, will cost $100 (5,000,000 / 100,000 * $2.00 = $100)

In total, these duplicated trails cost you $200. But is there something we can do in this scenario while still keeping compliance and the least privileges?

To avoid unnecessary costs from the PaidEventsRecorded metric, you can opt to remove the trails created in the specific accounts and keep the organization trail. By doing that, all the management events trail logs are still delivered to a centralized S3 bucket, but now you can control access per account with IAM roles.

For example, to give access to developers on their specific accounts within the organization, you can create an IAM role to be assumed by the developers from a specific account but only allow them to access the bucket prefix of the trails from their account. Your policy would look more or less like this:

{
    "Effect": "Allow"
    "Resource": "arn:aws:s3:::organization-trail-bucket/AWSLogs/OU_ID/ACCOUNT_ID/*",
    "Action": [
        "s3:Get*",
        "s3:HeadObject",
        "s3:List*",
        "s3:RestoreObject"
    ],
},
{
    "Effect": "Allow"
    "Resource": "arn:aws:s3:::organization-trail-bucket",
    "Action": "s3:ListBucket",
    "Condition": {
        "StringLike": {
            "s3:prefix": [
                "AWSLogs/OU_ID/ACCOUNT_ID/*"
            ]
        }
    },
}

By keeping the organization trail delivering logs to a centralized S3 bucket and controlling the access via IAM roles, you can eradicate the costs of having duplicated trails.

Data events

Slightly more complicated than the Management Events, you also pay for Data Events delivered to S3. Because there are no free copies (they always incur charges from the first copy) and the data events can generate a lot more audit logs, it's more difficult to reduce costs. They generate a metric called DataEventsRecorded, and this is how AWS charges you.

Nevertheless, data events are important for audit and compliance purposes. Of course, you can always negotiate with the audit team or the team responsible for those logs if they are necessary according to the policies defined by the organization.

AWS's advice is to "filter out AWS KMS or Amazon RDS Data API events by choosing Exclude AWS KMS events or Exclude Amazon RDS Data API events on the Create trail or Update trail pages". This can help you reduce the number of logs generated by data events.

Monitoring CloudTrail costs

Using the Cost Explorer console, you can get an overview of the PaidEventsRecorded and DataEventsRecorded metrics and how their costs increase/decrease over time.

You can select the metrics under the "Usage Type" filter:

Conclusion

In conclusion, managing AWS CloudTrail costs effectively requires understanding the pricing structure and identifying areas where expenses can be reduced.

By avoiding duplicated trails and utilizing IAM roles to control access to a centralized S3 bucket, you can eliminate unnecessary charges from multiple copies of management events.

For data events, consider filtering out less critical logs to reduce costs.

Regularly monitoring these expenses using tools like Cost Explorer can help you track and manage your CloudTrail spending more efficiently, ensuring compliance without overspending.

I would love to hear your thoughts! Let me know in the comments how your experience was with CloudTrail trails costs.

How To Mount an Amazon Elastic File System on Amazon CodeBuild From Another VPC

Matheus das Mercês — Mon, 03 Mar 2025 00:30:00 +0000

With CodeBuild, depending on the compute and environment type you configure, the runner offers a pretty good amount of disk space. However, after the build is completed, you lose the data in its disk because the storage is ephemeral. Mounting an Amazon EFS (Elastic File System) in the CodeBuild runner on the fly might be useful if you need to persist data across builds, run integration tests if the file system is used across multiple services, or manage large files efficiently.

In this quick how-to guide, I will show you how to mount an EFS on a CodeBuild ephemeral host that lives in a different VPC.

Prerequisites

Before mounting an EFS in the CodeBuild ephemeral host, make sure you take the following steps:

Make sure you have a way to access your CodeBuild runner (for example, via Systems Manager). This guide uses a self-hosted GitHub Actions CodeBuild runner. In a previous article, I shared how to set up a CodeBuild runner to self-host GitHub Actions in a large organization environment. Here is an example using CDK to show how that can be configured.
The CodeBuild project must be configured inside a VPC and have a security group assigned to it.
Both VPCs must be connected either via peering or Transit Gateway.
It is important that both VPCs are located in the same AWS region.
The CodeBuild project must be configured to run in privileged mode.
Make sure your local environment has the AWS CLI properly configured.

Step one: Configuring the EFS

If you haven't created the EFS yet, let's use the CLI to create a simple file system with the configuration needed for mounting on CodeBuild.

Create a new security group. Later, you will assign this security group to the newly created EFS.

aws ec2 create-security-group \
--group-name efs-example-sg \
--description "SG for the EFS mount target" \
--vpc-id vpc-id-example \

Write down the security group ID.

Create a new inbound rule for the new security group, allowing the CodeBuild runner security group.

aws ec2 authorize-security-group-ingress \
--group-id ID of the security group created for Amazon EFS mount target \
--protocol tcp \
--port 2049 \
--source-group ID of the security group of the CodeBuild runner \

Create a new file system with the name "ExampleFileSystem"

aws efs create-file-system \
    --performance-mode generalPurpose \
    --throughput-mode bursting \
    --encrypted \
    --tags Key=Name,Value=ExampleFileSystem

Write down the file system ID.

Create a mount target. The mount target must live in the same AZ as the CodeBuild runner and inside a private subnet.

aws efs create-mount-target \
--file-system-id file-system-id \
--subnet-id private-subnet-id \
--security-group ID-of-the security-group-created-for-mount-target \

NOTE: If you manage the CodeBuild runner configuration, you can create/modify your CodeBuild project to have the EFS mounted by default on every build. This reduces the complexity of having to manually mount it on every build. If that is your case, you can follow this guide from the AWS documentation and skip the following steps.
If the CodeBuild runner is managed by a DevOps/Platform team and you don't have control of its configuration or need to manually mount the EFS on the fly for any reason, you can continue with this guide.

Step two: Mounting the EFS

Make sure you have the DNS name of your file system as you follow the steps in this section. You can construct this DNS name using the following generic form:

file-system-id.efs.aws-region.amazonaws.com

Now, inside the CodeBuild runner (assuming you have a way to access it, as mentioned before), install the EFS utils as in the example below. Make sure you add the desired region in the /etc/amazon/efs/efs-utils.conf file:

sudo yum install -y amazon-efs-utils
sudo sed -i 's/#region = us-east-1/region = $REGION/' /etc/amazon/efs/efs-utils.conf

With the EFS mount helper installed, you can now mount the file system, pointing it to its DNS name. Make sure you add the IP address of the EFS in /etc/hosts, mapping the mount target IP address to your EFS file system's hostname:

sudo mkdir efs
EFS_IP_ADDR=$(aws efs describe-mount-targets --file-system-id $FILE_SYSTEM_ID --region $REGION | jq -r '.MountTargets[0].IpAddress')
echo "${EFS_IP_ADDR} $FILE_SYSTEM_DNS_NAME" | sudo tee -a /etc/hosts
sudo mount -t efs -o tls,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport $FILE_SISTEM_DNS_NAME efs/

Because both VPCs are connected via peering or a Transit Gateway, the CodeBuild runner can resolve the DNS name of the EFS.

Step three: Testing the EFS

To test the file system, you can run the following commands:

cd efs
sudo mkdir testing-efs
cd testing-efs
touch test.txt

Voila! Now, with you EFS mounted in the CodeBuild runner, this setup is particularly useful for sharing data across services or maintaining state between builds.

Is this a scenario you would use on your builds? Let me know in the comments!

Optimizing Amazon CloudWatch Costs for High-Traffic Lambda Functions with Advanced Logging Controls

Matheus das Mercês — Mon, 17 Feb 2025 00:30:00 +0000

Are you happy with your CloudWatch bill? If you have one or more high-traffic Lambda functions, you pay for their duration and number of requests per month. However, there is a chance these functions are also increasing your CloudWatch costs - and this is why you need Advanced Logging Controls.

In this article, I explain how to optimize CloudWatch costs while respecting compliance, and leveraging a simple AWS Systems Manager Automation Runbook to achieve full control of your logs.

Introduction

CloudWatch log groups have two storage classes and their costs vary from region to region, but according to the CoudWatch pricing page, in Ireland (eu-west-1):

Standard class, where you can have all log group features, like a subscription filter, for example, you pay $0.57 per GB.
Infrequent-Access class, where features are limited and you can not have a subscription filter in place and the data can only be queried using Log Insights, the cost is $0.285 per GB.

Knowing that you pay per GB for the data ingested into a log group, how can you determine the size of a message?

The short answer is doing that before sending it to CloudWatch is not very handy and there are situations where you can not or do not want to reduce its size. Instead, you can control what kind of data is ingested into log groups, when, and for how long - and this is where the benefits are.

It is all about bytes

Whenever your Lambda function sends logs to a log group, it uses the CloudWatch sub-feature of Collect (data ingestion). Under the hoods, your Lambda uses the PutLogEvents API from CloudWatch, generating a metric called DataProcessing-Bytes for the Standard Class and DataProcessingIA-Bytes for Infrequent-Access. Then, based on these two metrics, AWS creates your bill (more about monitoring these metrics later in this article).

To put this into perspective, let's imagine that you have a high-traffic Lambda function that executes 5 million times a day. I've separated some examples, including the number of bytes that message can generate and how much they can cost you at scale.

Application logs

Application logs are custom messages generated by your Lambda function. For instance, you want to log the event received, or debug messages you add to your code along the way, and so on.

Imagine that your high-traffic Lambda function handles API Gateway requests. For some reason, you want to log the event received. Let's have a look at an example of that payload:

{
  "resource": "/my/path",
  "path": "/my/path",
  "httpMethod": "GET",
  "headers": {
    "header1": "value1",
    "header2": "value1,value2"
  },
  "multiValueHeaders": {
    "header1": [
      "value1"
    ],
    "header2": [
      "value1",
      "value2"
    ]
  },
  "queryStringParameters": {
    "parameter1": "value1,value2",
    "parameter2": "value"
  },
  "multiValueQueryStringParameters": {
    "parameter1": [
      "value1",
      "value2"
    ],
    "parameter2": [
      "value"
    ]
  },
  "requestContext": {
    "accountId": "123456789012",
    "apiId": "id",
    "authorizer": {
      "claims": null,
      "scopes": null
    },
    "domainName": "id.execute-api.us-east-1.amazonaws.com",
    "domainPrefix": "id",
    "extendedRequestId": "request-id",
    "httpMethod": "GET",
    "identity": {
      "accessKey": null,
      "accountId": null,
      "caller": null,
      "cognitoAuthenticationProvider": null,
      "cognitoAuthenticationType": null,
      "cognitoIdentityId": null,
      "cognitoIdentityPoolId": null,
      "principalOrgId": null,
      "sourceIp": "IP",
      "user": null,
      "userAgent": "user-agent",
      "userArn": null,
      "clientCert": {
        "clientCertPem": "CERT_CONTENT",
        "subjectDN": "www.example.com",
        "issuerDN": "Example issuer",
        "serialNumber": "a1:a1:a1:a1:a1:a1:a1:a1:a1:a1:a1:a1:a1:a1:a1:a1",
        "validity": {
          "notBefore": "May 28 12:30:02 2019 GMT",
          "notAfter": "Aug  5 09:36:04 2021 GMT"
        }
      }
    },
    "path": "/my/path",
    "protocol": "HTTP/1.1",
    "requestId": "id=",
    "requestTime": "04/Mar/2020:19:15:17 +0000",
    "requestTimeEpoch": 1583349317135,
    "resourceId": null,
    "resourcePath": "/my/path",
    "stage": "$default"
  },
  "pathParameters": null,
  "stageVariables": null,
  "body": "Hello from Lambda!",
  "isBase64Encoded": false
}

This message generates 1.903 bytes. If this Lambda executes an average of 5 million a day, by the end of the month, the size in GB will be 274.66GB (1903 x 5000000 = 8.86GB a day, times 31 = 274.66GB).

Looking at the pricing for the Standard class, by the end of the month, this single log from a single Lambda cost you $156.55 (274.66GB x $0.57).

If you think this is an undesired cost - and somehow you want to keep logging these messages for debugging purposes, you need Advanced Logging Controls.

System logs

System logs are log messages generated by the Lambda service by default. For instance, Lambda reports data with the duration, billed duration, memory, and start and end time. Even if your Lambda code does not generate any application logs, by default, the system logs will always appear in your log group.

Let's have a look at a report message generated by a warm Lambda function in plain text (a cold Lambda has a different report message):

START RequestId: 1716e630-0997-4bd6-aae3-0f681ef1e69c Version: $LATEST
END RequestId: 1716e630-0997-4bd6-aae3-0f681ef1e69c
REPORT RequestId: 1716e630-0997-4bd6-aae3-0f681ef1e69c Duration: 1.80 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 68 MB

These logs created by one warm Lambda execution generate 262 bytes. If this Lambda executes an average of 5 million a day (not considering cold starts), by the end of the month, the size in GB will be 37.82GB (262 x 5000000 = 1.22GB a day, times 31 = 37.82GB).

Looking at the pricing for the Standard class, by the end of the month, this report messages from a single Lambda cost you $21.55 (37.82GB x $0.57).

Not a fortune, right? The important question to ask yourself is, do you need these log messages? Commonly, you would want to analyze the duration, and memory consumed for a Lambda to reduce costs related to Lambda execution, for example. But if you don't actively look at these messages, it is wise to change them to optimize your costs at scale.

Measuring the size of a log message

How can you calculate the size, in bytes, of a message sent to CloudWatch, as I did for the previous examples?

You can use the same query I've used in CloudWatch Log Insights:

FIELDS @message, ingested_bytes
#| filter @message like 'REPORT'
| STATS sum(strlen(@message)) AS ingested_bytes

ATTENTION: Querying data using Insights can also be expensive. For each GB of data scanned, you will pay $0.0057 (eu-west-1). Make sure you run this query in a log group that does not retain much data or only for a short time window.

Finding the perfect balance

There is always a balance between compliance and cost optimization. This is especially true in large organizations, where there might be constraints related to:

Retention period, as in how long the data must be stored to be compliant with the company's policies
The type of data ingested, which is more often related to the company's technical guidelines for specific AWS services

The important aspect of this trade-off is being able to negotiate. By clearly identifying these constraints alongside the analysis of cost reduction, you have better arguments to start a negotiation that benefits all parties involved.

Setting up a retention policy

Knowing how long you have to keep the data in a log group, you can start by setting up a retention policy for your log groups.

Less expensive than the DataProcessing-Bytes metric, there is another metric called TimedStoraged-BytesHrs metric, which is the amount of time that the data is stored - which AWS also charges you for. The cost for the Ireland region is $0.03 per GB compressed (0.15 compression ratio for each uncompressed byte).

Having a retention policy can help your application optimize some costs and also be more sustainable. Especially when using CDK (AWS Cloud Development Kit) as your IaC tool (Infrastructure as Code), when the default configuration from a log group is to never expire the data stored. If you don't use it after some period, it's best to set up a retention policy.

Below you can find an example, in TypeScript, of how to change the CDK default when creating a Log Group and setting the retention policy you define:

new LogGroup(this, 'MyLogGroup', {
  logGroupName: '/aws/lambda/advanced-logging-control',
  retention: RetentionDays.ONE_WEEK,
})

Instrumenting your code

One common mistake when developing a Lambda function using JavaScript (whether you use or not TypeScript features) is to log everything using the console.log method. In order to fully leverage Advanced Logging Controls and be able to manipulate the log level, it is important to use the correct methods when logging with JavaScript.

Something not very well-known is that, natively, the Lambda function logging configuration has 6 application log levels that you can define: INFO, WARN, ERROR, TRACE, DEBUG, and FATAL.

I have tested all JavaScript console methods when sending to a CloudWatch log group, and I separated a few examples from the console object documentation and how they match the Lambda application log-level configuration. Depending on the log level you set, different console methods will affect your CloudWatch log group. When using them properly, you make your Lambda function ready to switch the log level on the fly:

INFO level shows messages from all JavaScript console methods, except for console.debug.
WARN level shows messages from console.warn, console.assert, and console.error.
ERROR level shows messages from console.error only.
TRACE level shows messages from all JavaScript console methods.
DEBUG level shows messages from all JavaScript console methods.
FATAL level does not show any messages from any JavaScript console method.

Using the proper JavaScript console methods for specific situations not only makes your Lambda function ready to use Advanced Logging Controls but also helps your JavaScript comply with JavaScript's semantics.

Switching log level on the fly

Now, let's get down to business and see how to natively switch the log level of your Lambda function when needed.

Initial setup

I've created a very simple Lambda function with a few examples of using the JavaScript console methods:

export const handler = async (
    event: any,
): Promise<string> => {
    console.log('Received event:', JSON.stringify(event, null, 2));
    console.info('Info: Processing event');
    console.debug('Debug: Event details', event);
    console.warn('Warning: This is a sample warning message');
    console.error('Error: This is a sample error message');
    console.assert(false, 'Assert: This is a sample assert message');

    return 'Hello World!';
};

Using CDK, I set the System Log Level to WARN and the Application Log Level to ERROR:

new NodejsFunction(this, 'AdvancedLoggingControlFunction', {
  functionName: 'advanced-logging-control',
  entry: 'src/hello-world/handler.ts',
  handler: 'handler',
  //set application log level to ERROR and system log level to WARN
  applicationLogLevelV2: ApplicationLogLevel.ERROR,
  systemLogLevelV2: SystemLogLevel.WARN,
  //logging format must be set to JSON
  loggingFormat: LoggingFormat.JSON,
});

IMPORTANT: When changing the System Log Level and Application Log Level, the logging format must be set to JSON (defaults to Text).

This means that I am overriding CDK's default configuration for logging and reducing the messages sent to the CloudWatch log group. According to the explanation in the session above, now only console.error methods will show up in my CloudWatch log group.

If I execute this Lambda function, this is what I have in my CloudWatch Log Group:

Note that there are no System Log Messages (for instance, Lambda report) because I set the System Log Level to WARN, and only the console.error method affects the Log Group because the Application Log Level is ERROR only.

By only logging what you need, you can significantly reduce the costs of the DataProcessing-Bytes metric for a high-traffic Lambda Function and stop polluting your log group.

However, there are going to be situations where you want your Lambda function to be more verbose, to debug something, or to have extra information on your log group. Instead of always logging everything since the start, you might want to have the least detail possible and change it afterward, by updating your Logging Configuration to DEBUG (most detail) when needed:

aws lambda update-function-configuration \
  --function-name advanced-logging-control \
  --logging-config LogFormat=JSON,ApplicationLogLevel=DEBUG

NOTE: By not specifying the System Log Level in the parameters, AWS will automatically set it to INFO.

After the change, executing the Lambda function now gives us more messages in the log group:

As mentioned before, the DEBUG level will show messages from all console JavaScript methods.

Automating the process

Executing a CLI command every time you need to debug something is not very handy. More than that, you might want to execute this change in a more controlled and auditable way, especially when you do not have/must not have elevated privileges to do so.

To improve operational excellence, I have created a simple System Manager Automation Runbook, where this change can be executed in the AWS environment without the need to manually update the Lambda configuration using your IAM permissions. Instead, I have created an IAM role to be assumed by the SSM Automation Runbook. Plus, by doing that, the changes can be recorded by CloudTrail for compliance reasons.

More importantly, you want to analyze the Lambda function for a short period and reset it to the previous configuration to avoid extra costs with the DataProcessing-Bytes metric.

First of all, defining the IAM role to be assumed by the Automation:

const automationIamRole = new Role(this, 'AutomationIamRole', {
  assumedBy: new ServicePrincipal('ssm.amazonaws.com'),
});

automationIamRole.addToPolicy(
  new PolicyStatement({
    actions: [
      'lambda:GetFunctionConfiguration',
      'lambda:UpdateFunctionConfiguration',
    ],
    resources: [`arn:aws:lambda:${Aws.REGION}:${Aws.ACCOUNT_ID}:function:*`],
  })
);

NOTE: Achieving the least privileges can be a challenge because the Automation does not know upfront what Lambda function will be manipulated. Do you have an idea of how can this be solved? Let me know in the comments!

The Automation Runbook definition in CDK looks like this:

new CfnDocument(this, 'ModifyLambdaLogLevelDocument', {
  documentType: "Automation",
  name: 'ModifyLambdaLogLevelDocument',
  documentFormat: "YAML",
  updateMethod: "NewVersion",
  content: {
    schemaVersion: "0.3",
    description: "Modify the log level of a Lambda function temporarily. After 10 minutes, the log level will be reset to the original value.",
    assumeRole: automationIamRole.roleArn,
    parameters: {
      FunctionName: {
        type: "String",
        description: "The name of the Lambda Function",
      },
      LogLevel: {
        type: "String",
        description: "The log level to set",
        allowedValues: [
          "DEBUG",
          "INFO",
          "WARN",
        ],
      },
      Reason: {
        type: "String",
        description: "The reason for the change",
      },
    },
    mainSteps: [
      {
        name: "GetCurrentLoggingConfig",
        action: "aws:executeAwsApi",
        inputs: {
          Service: "Lambda",
          Api: "getFunctionConfiguration",
          FunctionName: "{{FunctionName}}",
        },
        outputs: [
          {
            Name: "CurrentLoggingConfig",
            Selector: "$.LoggingConfig",
            Type: "StringMap",
          },
        ],
      },
      {
        name: "ModifyLogLevel",
        action: "aws:executeAwsApi",
        inputs: {
          Service: "Lambda",
          Api: "updateFunctionConfiguration",
          FunctionName: "{{FunctionName}}",
          Description: "Update log level to {{LogLevel}}",
          LoggingConfig: {
            ApplicationLogLevel: "{{LogLevel}}",
            LogFormat: "JSON",
            SystemLogLevel: "{{LogLevel}}",
          },
        },
      },
      {
        name: "Wait10Minutes",
        action: "aws:sleep",
        inputs: {
          Duration: "PT10M",
        },
      },
      {
        name: "ResetLogLevel",
        action: "aws:executeAwsApi",
        inputs: {
          Service: "Lambda",
          Api: "updateFunctionConfiguration",
          FunctionName: "{{FunctionName}}",
          Description: "Reset log level to original value",
          LoggingConfig: "{{GetCurrentLoggingConfig.CurrentLoggingConfig}}",
        },
      }
    ]
  },
});

This document executes the following steps:

Get the current logging configuration for the provided Lambda function, saving it in a variable. This value will be used in the last step to reset the logging configuration to its original value.
Executes the API call to update the Lambda logging configuration with the provided log level. It accepts DEBUG, INFO, and WARN, which are more important levels to change on the fly.
Sleep for 10 minutes before changing it back. If 10 minutes is not enough time to debug, consider increasing the time or receiving the duration as a parameter in the document.
Reset the log level to its original value.

NOTE: It is also possible to add an approval step to be approved by one of your colleagues, although I have not included that in this example.

The full example, including the Lambda function and the SSM Automation Runbook can be found here.

Monitoring CloudWatch costs

Using the Cost Explorer console can give you an overview of the DataProcessing-Bytes and TimedStoraged-BytesHrs metrics and how their costs increase/decrease over time.

You can select, under the "Usage Type" filter both of the metrics:

Conclusion

Managing CloudWatch costs efficiently is crucial, especially for high-traffic Lambda functions. By implementing Advanced Logging Controls, you can significantly reduce unnecessary log ingestion and storage costs while maintaining compliance.

Key takeaways:

Monitor metrics like DataProcessing-Bytes and TimedStoraged-BytesHrs to track expenses.
Ensure logs are only stored for the necessary period to avoid excessive storage fees.
Leverage Lambda’s built-in log levels to filter out unnecessary logs and avoid polluting CloudWatch.
Utilize AWS Systems Manager Automation Runbooks to temporarily adjust log levels when debugging, without requiring constant manual intervention.
Use Cost Explorer to track trends and make informed decisions on further optimizations.

By adopting these practices, you can strike the right balance between compliance and cost efficiency, ensuring that your CloudWatch bills remain manageable while still providing the insights your applications need.

Building Scalable CI/CD Pipelines with Self-Hosted GitHub Actions on Amazon CodeBuild

Matheus das Mercês — Wed, 05 Feb 2025 00:00:00 +0000

Since Grady Booch introduced continuous integration (CI) in 1991, code changes can be tested and automatically integrated into a shared repository. Although CI/CD solutions have evolved quickly, the concept remains the same: you push, test, and deploy, ensuring the fast delivery of software updates.

In this post, you will discover how PostNL has designed fully serverless and centralized CI/CD pipelines in the AWS environment. These pipelines provide scalability, security, and convenience through services such as IAM and Amazon VPC.

Introduction

Nowadays, building your software on CI/CD pipelines is an easy task. Especially with tools like GitHub and Atlassian BitBucket, which offer hosted virtual machines to execute pipelines in a plug-and-play approach: you create a build file, define your build steps, and voila - your pipeline is running.

At PostNL, we use GitHub for collaboration and version control and we leverage the GitHub-hosted runners as our CI/CD tool, to run workflows for many use cases across the organization, like for instance deploying infrastructure and software, executing all kinds of tests and so on.

The AWS Center of Excellence (AWS CoE) team is dedicated to provisioning and establishing foundational infrastructure for PostNL development teams, to efficiently develop and deploy business-critical applications according to the best practices.

With that in mind, we have decided to design a solution that allows private communication between GitHub and AWS and supports architectures such as ARM64 in the pipelines, leveraging the Amazon CodeBuild feature that allows self-hosting GitHub action runners.

About GitHub-Hosted Runners

GitHub offers hosted virtual machines to run workflows. The Github-hosted runners are used by default across the PostNL organization.

Networking

By default, GitHub-hosted runners have access to the public internet. This means establishing communication with resources AWS, the traffic goes over the public internet. They are also shared across all GitHub customers.

However, things can get slightly complicated when a team needs, for example, to run integration tests against private resources inside an Amazon VPC, without having to expose these resources through a public Application Load Balancer or Amazon API Gateway.

Compute images

GitHub offers different types of runners for public and private repositories. Since all the repositories inside the PostNL Organization are private, teams are subject to use the supported runners for private repositories only. The list does not include, for example, the Linux Virtual Machine on ARM64 Architecture. This limits PostNL teams to executing builds using the Intel architecture only.

Note: According to GitHub documentation, the ARM64 Linux runner is in public preview and subject to change. It is only available for public repositories.

Why Self-Host GitHub Action Runners?

A self-hosted runner is a system that we deploy and manage to execute jobs from GitHub Actions. This approach offers more control of hardware, operating system, and software tools than GitHub-hosted runners provide to meet PostNL needs, for example:

Create custom hardware configurations with processing power or memory to run larger jobs;
Communicate with resources available on the internal network;
Choose a CPU architecture not offered by GitHub-hosted runners;

Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud. This also means that by self-hosting GitHub runners we are responsible for the cost of maintaining the runner system.

PostNL CCoE Shared Services

As the PostNL CCoE (Cloud Center of Excellence), our mission is to drive innovation and business agility for the PostNL engineering community and enable them to build cloud-native solutions. We do this by delivering a Landing Zone and facilitating knowledge sharing for teams to build, innovate, and own cloud solutions while delivering maximum customer value.

One of the practice areas under the CCoE structure is called Shared Services, managed by the AWS CoE team. In practice, the Shared Services is an AWS account inside the PostNL organization in AWS, responsible for offering business and engineering functions to PostNL development teams.

Network setup

At PostNL we are using a Transit Gateway to enable private network communications. All AWS accounts of the development teams are connected to two different routing tables on the Transit Gateway: one for production accounts and one for non-production accounts.

By providing the accounts with CIDR ranges coming from a big supernet we could prevent network traffic between production and non-production accounts creating just one blackhole route in the routing table containing the supernet. Since we wanted all teams to be able to access our Shared Services Account this account is attached to its routing table providing access to all production and non-production accounts.

For example, one of the services we provide in the Shared Services Account is the DNS service. For DNS we are using both public and private hosted zones. Although the public-hosted zones are authoritative, the private-hosted zones need to be reached from the PostNL internal DNS server so we created DNS resolver endpoints in the Shared Services Account.

For all the development team accounts we create a private hosted zone and associate that zone with all the VPCs of its workload and with the VPC of the Shared Services Account so the entries in all the private hosted zones can be retrieved from the Shared Services Account.

Because the Shared Services is at the organization level and can seamlessly communicate with the PostNL network in the AWS ecosystem, we have decided to design the Self-Hosted Runner in the Shard Services account, making it possible to process jobs for multiple repositories in the organization, while keeping security and compliance in the centralized CCoE environment.

The PostNL Self-Hosted Runner

The PostNL Self-Hosted Runner leverages the Amazon CodeBuild feature that allows self-hosting GitHub action runners. This solution provides:

A fully managed service to handle builds on ephemeral hosts.
No need for managing the underlying infrastructure, fully serverless.
Seamless integration with the PostNL internal network.
Full control over the runner's image and compute type supported by CodeBuild

Connection between GitHub and AWS

As the solution is offered and managed by the PostNL CCoE in the Shared Services account, the centralized approach enables development teams to execute builds on AWS without having to manage the CodeBuild project or the connection between AWS and GitHub.

This is a one-time configuration made in the Shared Services AWS account, creating a GitHub App connection for GitHub for Amazon CodeBuild. This is necessary to allow CodeBuild to execute jobs in the GitHub PostNL organization.

One CodeBuild project per team

By creating one CodeBuild project for each team, we can achieve the least privileges by using ABAC (Attribute-Based Access Control) based on the repository prefix, where only the GitHub repositories belonging to that specific team can run builds on their specific CodeBuild project. This is achieved by using the Filter Group of the CodeBuild source:

Since CodeBuild allows to assignment of at least one security group to the project, having one security group per team also makes it more fine-grained.

CodeBuild supports 5,000 build projects within a single region.

Leveraging existing IAM role permissions

CodeBuild assumes the existing OIDC (Open ID Connect) role configured for that specific GitHub repository.

Internal network communication

The solution also enables development teams to execute end-to-end and integration tests against resources in the internal network, leveraging the Shared Services Account network setup.

This can be achieved by using Security Group Referencing towards the security group of the CodeBuild project, adding an extra layer of security.

Customizing image and compute type

By default, the Self-Hosted CodeBuild projects are created using the image type of Linux ARM64 (not supported by standard GitHub-Hosted Runners) and compute type Medium (8 GB memory, 4 vCPUs).
When running builds, development teams have the freedom to customize the image and compute type by using all types supported by CodeBuild:

runs-on:
  - codebuild-<project-name>-${{ github.run_id }}-${{ github.run_attempt }}
  - image:<environment-type>-<image-identifier>
  - instance-size:<instance-size>

An example of how the CodeBuild project can be set up can be found here.

Next steps

The PostNL Self-Hosted Runner is currently in its first version, although it is already a reliable choice for running builds that require more power and connectivity across the organization, we want to evolve it into a more robust solution with a couple of ideas in mind:

Limit the size of the runners to a list pre-defined by the CCoE to the team's use case, since running very large builds can lead to cost increases. We can achieve that by creating, for example, a reusable GitHub Workflow to share across the PostNL organization.
Create our buildspec.yml file: since CodeBuild manages the build commands, we would have to manipulate the build steps (INSTALL, PRE_BUILD, and POST_BUILD) to install new packages/software needed for the builds at PostNL.

As CCoE, we want to be close to the development teams using the PostNL Self-Hosted Runner to gather feedback on how the solution can be improved.

Conclusion

The CCoE PostNL's implementation of Self-Hosted GitHub Action Runners using Amazon CodeBuild represents a significant advancement in our CI/CD pipeline capabilities.

By leveraging AWS's serverless infrastructure and integrating it with our internal network, we have achieved a scalable, secure, and efficient solution that meets the diverse needs of our development teams. This centralized approach not only enhances our ability to run complex builds and tests but also ensures compliance and security within the PostNL ecosystem.

Implementing Regression Tests for AWS Lambda with CDK Fine-Grained Assertions

Matheus das Mercês — Mon, 20 Jan 2025 00:30:00 +0000

When creating infrastructure on AWS using IaC (Infrastructure as Code) tools, knowing if the CloudFormation Template generated by the code has the expected definition is often challenging. There is also a need to ensure that the eventual changes introduce no unintended breaks to your infrastructure code.

In this article, I will explore why CDK (Cloud Development Kit) is a game-changer for testing IaC and how to leverage the CDK Fine-Grained Assertions approach in TypeScript to check the integrity of your Lambda functions definition before deploying your infrastructure.

Introduction

One of the biggest challenges of using IaC tools such as Serverless Framework, AWS SAM (Serverless Application Model), or Terraform is to make sure the output is correct and according to the standards defined by you or your organization, because these tools are YAML or JSON based.

This is especially true for Lambda functions, where you want to make sure that your infrastructure code is generating a Lambda function that has, for instance, the desired timeout time, memory size, and the required IAM permissions to serve its purpose.

Gladly, when using CDK as your IaC tool, it is possible to create assertions and let it fail in the CI if some configuration is deviated due to code change, for example. This approach is also known as Regression Tests.

CDK: A game changer for testing IaC

When AWS introduced CDK in 2019, the possibility of writing infrastructure using familiar programming languages such as TypeScript and Java sounded nice. However, I have to be honest - it did not catch my attention at first.

From https://docs.aws.amazon.com/cdk/v2/guide/home.html

I've had quite some experience with Serverless Framework and AWS SAM, and I was more in favor of using these tools because their use of YAML files provides a more explicit and declarative way to define infrastructure - less abstraction - compared to CDK.

It was only in November 2021 that I changed my mind: The CDK assertions library was announced and that was, for me, a true game changer: being able to write infrastructure in TypeScript was already great, but writing fine-grained tests towards infrastructure code really caught my eye.

Does that mean that CDK is better?

The short and cliche answer is: It depends. I personally still like YAML-based IaC tools for specific use cases, for instance:

Simple and/or small projects, when you need a more descriptive infrastructure definition with lower abstraction.
When you work in a team that is already familiar with YAML and has less knowledge of programming languages, reducing the learning curve.

Nowadays, for new projects that I know it is going to grow, and demand strict testing, I'd rather start with CDK especially due to the possibility of testing these resources before I even have to deploy them - and having full control of what is been tested.

Let's get down to business and explore how to implement regression tests with CDK fine-grained assertions.

CDK Fine-Grained Assertions

Assuming you have a code base with a CDK app, it is very easy to start writing assertions against the defined resources.

To demonstrate implementing regression tests in the CDK code, I've written a simple CDK app that creates a Lambda function, with a CloudWatch Log Group and an IAM Role with a defined IAM Policy, using TypeScript, as it's the language I am most familiar with.

I've created a test folder on the root of the project and a test file to start implementing the tests.
test/cdk-fine-grained-tests.test.ts

import { Template } from 'aws-cdk-lib/assertions';
import { App } from 'aws-cdk-lib';
import { CdkFineGrainedTestsStack } from '../lib/cdk-fine-grained-tests-stack';

describe('MyFunction Fine-Grained Tests', () => {
  const app = new App();
  const stack = new CdkFineGrainedTestsStack(app, 'CdkFineGrainedTestsStack', {});

  const template = Template.fromStack(stack);
});

Using the Template class from the CDK assertions library, I've created a "copy" of the CloudFormation template generated by the CDK stack.

After defining the main structure of the test file, we can start writing the assertions against the CloudFormatin template, to check the integrity of the configuration we want to include in the regression test. For the Lambda Function, it is important to check if the memory size does not get too big for its purpose, possibly incurring cost increases. The same goes for duration, architecture, and runtime.

it('should have created a lambda function with default configuration', () => {
    template.hasResourceProperties('AWS::Lambda::Function', {
      FunctionName: 'my-function',
      Handler: 'index.handler',
      Runtime: 'nodejs22.x',
      Architectures: ['arm64'],
      Timeout: 30,
      MemorySize: 128,
    });
  });

You can also check if the Lambda function's CloudWatch Log Group has the desired retention policy:

it('should have created a log group with the correct retention policy', () => {
  template.hasResourceProperties('AWS::Logs::LogGroup', {
    LogGroupName: '/aws/lambda/my-log-group',
    RetentionInDays: 7,
  });
});

And finally, ensuring the Lambda has the required IAM permissions to execute the actions:

it('should have created a iam service role with the lambda basic execution role', () => {
    template.hasResourceProperties('AWS::IAM::Role', {
      ManagedPolicyArns: [
        {
          'Fn::Join': [
            '',
            [
              'arn:',
              { Ref: 'AWS::Partition' },
              ':iam::aws:policy/service-role/AWSLambdaBasicExecutionRole',
            ],
          ],
        },
      ],
    });
});

it('should have created a iam policy with the correct permissions', () => {
    template.hasResourceProperties('AWS::IAM::Policy', {
      PolicyDocument: {
        Statement: [
          {
            Action: 's3:GetObject',
            Effect: 'Allow',
            Resource: 'arn:aws:s3:::my-bucket/*',
          },
        ],
      },
    });
});

You eventually would like to ensure that these properties remain untouched on new code changes unless this change is deliberated. If that is the case, the assertion also would need to be updated, otherwise the regression test will fail - and that is what we want.

The full example can be found here.

Executing the tests on the CI

After implementing the assertions in your code base, it is wise to execute the tests when you push changes to your branch.

For the example above, I've implemented a workflow to execute the tests in GitHub Actions on every push to the main branch:

name: Build and run tests
on:
    push:
        branches:
            - "main"

jobs:
    build-and-test:
      runs-on: ubuntu-latest
      permissions:
        id-token: write
        contents: read
      steps:
        - name: Checkout
          uses: actions/checkout@v4

        - name: Install dependencies
          id: install-dependencies
          run: |
            npm ci

        - name: Run CDK fine-grained tests
          id: cdk-fine-grained-tests
          run: |
            npm run test

This way you ensure that this step is executed before deployment, allowing early feedback about how the changes impacted your infrastructure.

Beyond Lambda Functions

Although this article is focused on Lambda Functions, CDK Fine-Grained Assertions can be used for any piece of infrastructure you want to implement regression tests.

Let's say that your CDK code creates an API Gateway. You could implement regression tests against, for example, a definition of a Gateway Response for the BAD_REQUEST exception as follows:

it('should have created a gateway response for bad request', () => {
   template.hasResourceProperties(
      'AWS::ApiGateway::GatewayResponse',
        {
          ResponseType: 'BAD_REQUEST_BODY',
          StatusCode: '400',
          ResponseTemplates: {
            'application/json':
              '{
               \n "message": "$context.error.validationErrorString"\n      
               }',
          },
        },
   );
});

As long as you know what resources your infrastructure has and what you would like to test, you have a lot of possibilities. That is because the assertions are made against the CloudFormation template generated by your CDK code.

Please refer to the CDK assertions library.

Conclusion

AWS CDK with Fine-Grained Assertions enables precise testing of infrastructure by validating the CloudFormation templates it generates. This approach ensures that resources, such as Lambda functions, adhere to specific configurations like memory size, timeout, and IAM permissions.

By integrating these tests into a CI/CD pipeline, you can catch unintended changes early and maintain confidence in your infrastructure code.

While YAML-based tools may suit simpler projects, CDK’s combination of programming flexibility and fine-grained testing makes it a nice choice for scalable and test-driven infrastructure development.

What are your thoughts? Let me know in the comments below, I would love to hear what you have to say!

The Difference Between Building on AWS and Making Ice Cream

Matheus das Mercês — Mon, 06 Jan 2025 07:32:31 +0000

What does building solutions on AWS have in common with making ice cream? On the surface, not much. Yet, if you dig deeper, they share some surprising parallels - and understanding these similarities can teach you a lot about building better software on AWS.

In this article, I will step into AWS's metaphorical ice cream shop and explore what separates quality from average.

Introduction

Imagine that you own an ice cream shop. You are not an ice cream wizard yet - you know one thing here and there, but you start to feel confident about what you do. The possibilities seem endless: you have easy access to every flavor, topping, and combination imaginable.

Meet Scoopy, an ice cream loving customer who just entered your shop. You want to show your worth, and Scoopy has high expectations.

After some time (there were so many options) Scoopy has decided. You deliver the pistachio flavor (my favorite one) in a bowl to Scoopy in just a couple of minutes, with just a few pistachio nuts on top, nothing too fancy. Scoopy takes a bite, smiles, and sits to enjoy the ice cream. He is happy. Mission accomplished.

Making great ice cream requires skill, creativity, and, most importantly, respect for what your customer wants if you want to give a good impression. It's similar to building scalable, efficient solutions on AWS - with one major difference.

To whom do you make ice cream?

You might think, "OK, I do not own my own company and am far from owning an ice cream shop. Then, to whom should I give a good impression with the solutions I build?"

Your customer isn’t just someone who walks into your ice cream shop. It’s anyone who interacts with the solutions you build. Whether you're working as part of a development team, answering to stakeholders, or creating internal tools, the customer is anyone who benefits from your work.

In an AWS context - or even in a generic development universe - your customer could be the end user of your website hosted on an S3 Bucket, the business team relying on your Lambda function to improve operations or even your colleagues who need your Step Function to automate their work. The key is understanding their needs, solving their problems, and delivering value.

Every line of code, every decision you make, is aimed at impressing and satisfying the customer—whoever they may be.

What makes your customer happy

One of my favorites Amazon Leadership Principles is Customer Obsession, the ability to "work vigorously to earn and keep customer trust". Customer Obsession is more than just a principle - it’s a mindset that ensures everything you do is aligned with delivering value to your customers.

Now, let’s reflect on a few characteristics of your ice cream that impressed your customer.

It took you around 2 minutes to prepare it. You put some high-quality Pistachio nuts on top, which was a personalized experience based on your customer needs, and that made the ice cream well presented. Scoopy did not want anything too fancy — and you respected that. He just wanted a nice Pistachio ice cream. Although you could have put some white chocolate on the top (let's be honest, that is a great match), that was not what Scope wanted.

You keep it consistent. Scoopy will not easily find similar ice cream elsewhere: you earned trust. Scoopy will come back for more.

These elements - personalization, attention to detail, quality, and consistency - are what leave a good impression on your customer, whether you're serving ice cream or building software on AWS.

The big difference

Building software on AWS has a lot in common with making ice cream—but there is one thing that differs completely: the unstoppable, unpredictable, time.

If you were in Scoopy's shoes, would you rather wait 2 minutes for an amazing creamy Pistachio ice cream or get an average frozen watery premade ice cream in 15 seconds?

The answer to that is clear. Scoopy was not looking for speed.

As much as you want to impress your customer, it’s important to set realistic expectations. In an ice cream shop, you can promise to Scoopy their ice cream in a couple of minutes because the process is straightforward and predictable. But when building solutions on AWS (or in any software development context) things are rarely that simple.

Here’s why it’s critical to stay realistic:

Building scalable, reliable, and secure solutions isn’t a quick task. It requires design, testing, and iteration. Rushing the process can lead to costly mistakes or unexpected results. Complexity takes time.
If a project will take weeks to deliver, be honest about it. Overpromising on speed might impress initially but will hurt your credibility in the long run. Transparency builds trust.
Customers appreciate timely delivery, but they value solutions that work even more. Prioritize getting it right over getting it done fast. Focus on quality, not speed.

Remember, delivering value to your customers is about focusing on quality and being honest about what’s achievable, and your customers will trust you more for it.

Conclusion

Building solutions on AWS and making ice cream might seem worlds apart, but they share a surprising amount in common. Both require creativity, attention to detail, and a focus on delivering value to the customer. Whether you're making the perfect ice cream bowl for Scoopy or designing an AWS-based application, success depends on your ability to combine the right tools and ingredients to build something that will make your customer happy - no matter what it takes.

But here’s the twist: the difference between building on AWS and making ice cream is that the ice cream is ready in 2 minutes.

What are your thoughts? Do you aim to impress your customers daily? Let me know in the comments below!

Three Steps to Build a More Cost-Effective Solution on AWS

Matheus das Mercês — Sat, 28 Dec 2024 16:51:59 +0000

When managing costs in AWS, the need for savings is often clear, but sometimes there's a lack of precise insight into how much needs to be saved within a specific timeframe. Whether you work for a company that wants you to respond to budget pressures or are looking for ways to optimize your environment, some daily reflections can help you speed up your cost-saving efforts and plant the seed for the future of your application.

In this post, I'll share three simple steps to help you build a more cost-effective solution on AWS.

Do you really need it?

During the AWS re:Invent 2024 keynote, Werner Vogels emphasized the importance of "simplexity" - the art of designing architectures that are powerful yet simple. "Simplexity" isn't just a fancy term; it's a mindset that drives innovation while avoiding unnecessary complexity.

Inspired by these principles, I've found one question to be a true game-changer: Do you really need it?

This question forces us to challenge assumptions and well-known architectures and focus on what truly matters. It aligns with the goals of keeping your architecture simple and driving sustainability and cost efficiency.

Step one: challenge decisions made in the past

How often have you encountered the response, "because we've always done it this way," when questioning why certain setups exist?

If you find yourself still using this rationale, it's time to reconsider - after all, it's 2024 (almost 2025!). Technology, particularly in the AWS ecosystem, evolves quickly every day. Continuously reevaluating these decisions is essential, both for the performance of your applications and for optimizing costs.

Imagine the following architecture diagram:

This AWS CloudTrail log aggregation solution centralizes logs from all member accounts in an AWS Organization into a single account. Each account sends its CloudTrail logs to both a CloudWatch Log Group and an S3 bucket in the central account. This setup provides centralized governance and storage for auditing and compliance purposes.

Looking at this diagram, I asked myself: do we really need to store the logs in both locations? A CloudWatch Log Group offers real-time monitoring capabilities, such as using Log Insights, while an S3 bucket provides long-term, cost-efficient storage that can be easily queried with Athena.

In this case, real-time monitoring through CloudWatch was not required. Challenging this past decision allowed us to eliminate CloudWatch log ingestion and retain only the S3 bucket - reducing unnecessary complexity.

The result of this simple change was a reduction of $4,038.81 per month in CloudWatch costs:

Simplifying the setup resulted in significant cost savings. The key takeaway is that past decisions may no longer be relevant today - it's essential to continuously challenge them.

Step two: challenge assumptions

Another common sentence people use is "because I think it is more secure". However, it's important to question whether this assumption still holds. What may have been considered correct in the past might no longer be the best approach today. Continuously reassessing decisions - based on facts - ensures you're adopting the most effective and efficient solutions for your current needs.

Let's have a look at another scenario in the same large AWS Organizations setup, where each member account had its own S3 Bucket used for CloudFormation deployment assets:

Do we really need to have an SSE-KMS encryption type with a Customer Managed Key in KMS? Reflecting on use cases for that: key rotation, centralized key management, and detailed access control over who can use the key. None of that applied to this scenario. We decided to remove the key and have SSE-S3 encryption type, where AWS owns and manages the encryption key.

The result of this simple change was a reduction of $2,040.43 per month in KMS costs:

This change allowed a simpler solution - while the data is still encrypted at rest, reducing maintainability since the KMS keys were removed. As a consequence, more cost reduction.

Regularly challenging assumptions is crucial for improving efficiency, reducing complexity, and optimizing costs. Continuous evaluation helps ensure that solutions evolve alongside technological advancements.

Step three: adopt "simplexity"

Cost efficiency in AWS is as much about avoiding unnecessary expenses as it is about building efficient systems - adopting "simplexity".

Reflecting on these steps daily can help you achieve quick wins that not only save some money today but also set your application up for a sustainable future.

Refactoring a Lambda Monolith to Microservices Using Hexagonal Architecture

Matheus das Mercês — Tue, 17 Dec 2024 10:35:07 +0000

In many applications, a single Lambda function contains all the application logic that handles all external events. The Lambda function sometimes acts as an orchestrator, handling different business workflows within complex logic. This approach has several drawbacks, like big package size, difficulty enforcing least privilege principles, and hard to test.

In this blog post, I will explain how to leverage the Hexagonal Architecture, also known as the "Ports and Adapters" approach, to refactor a Lambda monolith into microservices.

The big, complex, and tightly-coupled

AWS Lambda is incredibly easy to build and deploy, but it's also easy to fall into the trap of creating a "Lambda monolith". This approach bundles all your logic, processes, and dependencies into a single function that attempts to manage various events. For example, a Lambda monolith function would handle all API Gateway routes and integrate with all necessary downstream resources.

It becomes even more complex when the Lambda function acts as an orchestrator, handling different business workflows resulting in "spaghetti code" when many if-else statements are used.

These approaches are considered anti-patterns in Lambda-based applications and have several drawbacks, for instance:

Big Package Size: As logic grows, so does the size of your deployment package, which can slow down deployment times and the Lambda cold start time.
Hard to enforce the least privilege: It becomes challenging to assign least-privilege permissions, as one Lambda may require broad access to different resources.
Testing Complexities: Testing a large Lambda function is problematic because every modification impacts a broad range of code. Unit testing is difficult, and integration testing is even more challenging to build.

The preferred approach is to decompose the monolithic Lambda function into individual microservices, with each Lambda function dedicated to a single, well-defined task.

Decomposing the monolith

Before you enter Hexagonal Architecture, you need to think about how you would decompose your monolith. Moving from a monolithic Lambda to microservices is more than just splitting code into smaller parts; it's about strategically decomposing the monolith to create services that are independent, scalable, and maintainable. Without careful decomposition, you risk creating a set of microservices that are still tightly coupled or that mirror the same complexities and limitations of the original monolith.

My favorite approach is the "decompose by business capability". This means that each microservice is structured around a specific business capability or function, ensuring that each service corresponds to a particular area of the application's logic.

This method of decomposition is especially valuable in larger applications, where different business functions - such as product catalog management, order management, or delivery - have distinct lifecycles, data requirements, scaling needs, and sometimes distinct development teams. More importantly, you build a stable architecture since the business capabilities are relatively stable.

Strangling the monolith

With your business capabilities mapped to services, you need to think about where to start coding.

Instead of attempting a full rewrite, which can be risky and disruptive, the Strangler Pattern allows us to migrate functionality piece by piece. This technique involves incrementally replacing parts of the monolith with microservices, gradually "strangling" the monolith until it's fully decomposed.

For each business function that we pull out of the monolith, we create a new microservice that can handle it independently. Over time, as more services are extracted, the monolith becomes smaller until it's eventually replaced by a cohesive set of microservices.

This approach is ideal for modernizing applications in stages, reducing downtime and risks while moving toward a microservices architecture.

At this point, you probably already know where to start. It's time to dive a little bit deeper into Hexagonal Architecture.

Enter Hexagonal Architecture

Core Logic (Domain): The core contains the application's core business rules, completely isolated from the outer layers.
Ports: Defined interfaces that describe actions available to the core.
Adapters: Connect external systems to the application's core through ports, making it easy to switch out databases, API integrations, or other dependencies without impacting the core logic.

This approach allows us to design a Lambda setup where each function or service remains lean and single-purpose, eliminating the challenges of a monolithic structure. Let's look at how each layer of the Hexagonal can be implemented in a Lambda-based microservice.

Implementing the hexagonal layers

Each layer requires specific attention to refactor your Lambda application using Hexagonal Architecture. Here's a breakdown of the layers and how they work in practice.

To demonstrate implementing Hexagonal Architecture layers in code, I'll write an simple web-application backed by a Lambda function, which serves API Gateway requests and communicates with both DynamoDB and S3, using TypeScript, as it's the language I am most familiar with. While Hexagonal Architecture is particularly well-suited for typed languages, it is language-agnostic and can be implemented in any language or framework of your choice.
I have used InversifyJS, a library used to create inversion of control (IoC) container for TypeScript. An IoC container uses a class constructor to identify and inject its dependencies

Core Logic (Domain)

The core business rules reside here. The domain layer is completely isolated from AWS-specific code or other external dependencies, making it easy to test and modify.

import { injectable, inject } from "inversify";
import TYPES from "../../container/types";
import IRepository from "../../interfaces/repositoryIF";
import IStorage from "../../interfaces/storageIF";

@injectable()
class HelloWorld {
    constructor(
        @inject(TYPES.Repository) private repository: IRepository,
        @inject(TYPES.Storage) private storage: IStorage,
    ) {}

    async handler(_event: any): Promise<any> {
        //get data from storage
        const storageData = await this.storage.get("123");

        //update repository with new data
        await this.repository.update("dummy-id", {
            name: "dummy-name",
            age: 20,
            storageData,
        });
    }
}

export default HelloWorld;

Ports

The ports layer coordinates interactions between the domain and external services but doesn't contain direct calls to those services. It is agnostic about what the downstream service is called. In oriented-object programming, a port can be directly related to an interface.

export default interface IRepository {
    get(id: string): void;
    update(id: string, data: any): void;
}

export default interface IStorage {
    get(id: string): void;
}

Adapters

Inbound Adapters: These handle incoming requests, such as API Gateway events or other triggers, and pass them to the core logic. This is also the entry file for the Lambda Function.

import { APIGatewayProxyEvent, APIGatewayProxyHandler, APIGatewayProxyResult } from "aws-lambda";
import { container, TYPES } from "../container/inversify.config";
import HelloWorld from "./hello-world/helloWorld";

export const helloWorld: APIGatewayProxyHandler = async (
    event: APIGatewayProxyEvent,
  ): Promise<APIGatewayProxyResult> => {
    const lambda = container.get<HelloWorld>(
      TYPES.HelloWorld,
    );

    return lambda.handler(event);
};

Outbound Adapters: These manage outbound calls, like calls to databases (in this case, DynamoDB and S3) or third-party APIs, abstracting them through well-defined interfaces (ports).

import { DynamoDBClient, GetItemCommand, UpdateItemCommand } from '@aws-sdk/client-dynamodb';
import { injectable } from "inversify";
import IRepository from "../../../interfaces/repositoryIF";

@injectable()
class DynamoDbRepository implements IRepository {
    private client: DynamoDBClient;

    async update(id: string): Promise<void> {
        const updateItemCommand = new UpdateItemCommand({
            TableName: "example-table",
            Key: {
                id: { S: id },
            },
            UpdateExpression: "SET #name = :name",
            ExpressionAttributeNames: {
                "#name": "name",
            },
            ExpressionAttributeValues: {
                ":name": { S: "dummy-name" },
            },
        });

        await this.client.send(updateItemCommand);
    }

    async get(id: string): Promise<void> {
        const getItemCommand = new GetItemCommand({
            TableName: "example-table",
            Key: {
                id: { S: id },
            },
        });

        await this.client.send(getItemCommand);
    }
}

export default DynamoDbRepository;

import { GetObjectCommand, PutObjectCommand, S3Client } from '@aws-sdk/client-s3';
import { injectable } from "inversify";
import IStorage from "../../../interfaces/storageIF";

@injectable()
class S3Repository implements IStorage {
    private client = new S3Client({});

    async get(id: string): Promise<void> {
        const getItemCommand = new GetObjectCommand({
            Bucket: "example-bucket",
            Key: id,
        });

        await this.client.send(getItemCommand);
    }
}

export default S3Repository;

The full example can be found here.

By isolating each concern, the Lambda functions become much smaller, enabling us to enforce least-privilege permissions and streamline testing. Each function now interacts with its dependencies through well-defined interfaces, making it easier to manage and test independently. This is what the Hexagonal looks like in the example above:

The modular, scalable, and flexible

Using Hexagonal Architecture, it is possible to gradually decouple the monolith into smaller and independent microservices, for example:

This approach improves a Lambda-based application and brings many potential benefits:

Simplified Maintenance: Clear boundaries allow for focused modifications without fear of inadvertently breaking other parts of the application, optimizing package size, and reducing deployment and cold start times.
Clarity and Separation of Concerns: Each Lambda function has a single responsibility, making the codebase easier to read and navigate, and easy to enforce the least privileges principle.
Testing Efficiency: Testing becomes simpler, as services can be isolated and mocked cleanly.
Reusability: Since adapters are cleanly abstracted, they can be reused across multiple functions, reducing development redundancy.

It isn't all smooth sailing…

While this transition has numerous advantages, some challenges accompany the move to microservices using Hexagonal Architecture:

How to decouple: Breaking apart a monolithic Lambda into separate microservices requires careful thought about how to decouple services and deploy them effectively. Ensuring that each service is truly independent can be difficult.
Testing Overload: With multiple services and adapters, testing can grow exponentially. The question arises: where to draw the line with testing? The focus shifts to defining clear boundaries for the unit, integration, and end-to-end tests.
Repository Management: As the codebase grows, it's crucial to enforce strict practices against code duplication and maintain clear documentation.
Traceability: With the application composed of several microservices, it can become tricky to trace a request and log, instead of handling it as a single service, you need to make sure the request is been traced across all your microservices.

Final thoughts

Refactoring from a monolithic Lambda to a microservice-based Hexagonal Architecture setup involves a significant initial effort, but the rewards make it worthwhile:

Reusable Adapters: Once built, adapters can be used across multiple Lambda functions, making development faster and more consistent.
Simplified Testing: With a clear separation of concerns, testing becomes easier and more reliable. By abstracting dependencies, we eliminate the need for complex Jest mocks, focusing instead on testing business logic directly.
Enhanced Flexibility: Adapters provide a modular approach, so if there's a need to swap AWS services or add new integrations, the core logic remains unaffected. This adaptability allows for seamless changes without impacting the entire application.

Refactoring your Lambda setup with Hexagonal Architecture can transform a complex, tightly coupled monolith into a streamlined, testable, and flexible microservice ecosystem, providing a solid foundation for future growth.

How are your experiences with such a migration? Leave a comment below!