DEV Community: Omid Eidivandi

Amazon Bedrock Agentcore & System Design

Omid Eidivandi — Sat, 22 Nov 2025 01:17:39 +0000

The recent general availability of Amazon Bedrock Agentcore marks a significant milestone in the evolution of AI-powered applications on AWS. While Bedrock has already established itself as a leading platform for building and scaling generative AI solutions, Agentcore pushes the ecosystem further by offering a more unified runtime for orchestrating agents, managing memory, integrating tools, and enabling complex workflows. This GA release doesnt just expand AWSs AI portfolioit accelerates how quickly teams can design, deploy, and iterate on intelligent, autonomous systems.

Over the past months, a wave of technical content has emergeddeep dives into the Agentcore runtime, gateway integrations, memory persistence models, tool management, and more. These resources are incredibly valuable for understanding how to use the platform. But as the adoption curve steepens, its equally important to step back and look at the bigger picture: we are heading toward a world where distributed, specialized AI systems become the norm.

Agentcore makes it easier to build these components, but it also increases the risk of teams assembling complex systems without a solid architectural foundation. Without thoughtful system design, standardization, and distributed-systems thinking , organizations may unintentionally create fragile, siloed, or overly coupled AI ecosystems. The challengeand opportunitynow is to balance speed of development with robust design principles that ensure these emerging agent-based architectures remain scalable, observable, secure, and maintainable.

This article explores that broader context: what Agentcores GA really means for distributed system design, why standardization matters now more than ever, and how teams can avoid the architectural pitfalls that come with rapid innovation.

Distributed Design

Modern enterprise products are, at their core, distributed systems whether teams consciously design them that way or simply evolve into them over time. As organizations scale, their applications naturally break into interconnected services, workflows, and data domains that must communicate reliably. In this environment, boundaries matter : clear ownership, well-defined responsibilities, and stable interfaces determine whether a system grows sustainably or accumulates silent complexities that surface only during failures.

Distributed systems introduce challenges that are fundamentally different from writing business logic. Engineers may thoroughly understand the domain, the models, and the rules they implementyet still find themselves frustrated when the real difficulty emerges not from the logic itself, but from the interactions between components. Latency, partial failures, race conditions, version drift, inconsistent state, and unclear integration contracts can turn even well-designed services into unpredictable systems. This is often what makes distributed systems feel maddening: you can master the business logic and still get blindsided by emergent behavior rooted in architectural decisions.

These pitfalls are not signs of poor engineering but symptoms of systems where boundaries are blurred, responsibilities overlap, and integrations evolve informally rather than intentionally. Without deliberate system-design thinkingdefining boundaries, standardizing communication patterns, enforcing contracts, and anticipating failure modesdistributed architectures tend to drift. Over time, they accumulate complexity that becomes harder to reason about, harder to extend, and harder to trust.

In todays enterprise landscape, understanding distributed-systems principles is no longer optional. It is the foundation that ensures the entire product ecosystem remains resilient, observable, and adaptable as it grows.

Core Concepts of Distributed Systems

Distributed systems succeed or fail not because of any single component, but because of how those components interact, evolve, and respond to real-world conditions. Mastering a few foundational concepts helps teams design systems that stay reliable as they grow, adapt to change, and avoid the hidden complexity that often emerges at scale. The following principles offer a baseline for building resilient, predictable, and maintainable distributed architectures.

Communication: Communication is an unavoidable glue of distributed systems, and every interaction carries risks: latency, timeouts, and partial failures. Choosing clear patternssync, async, eventshelps manage uncertainty and keep components predictable.

Boundaries: Boundaries create separation of concerns, ensuring each service has a clear purpose. Well-defined boundaries reduce coupling and prevent hidden dependencies that make systems fragile and hard to evolve.

Specialization and Ownership: As systems grow, components must specialize to solve focused problems. Strong ownership ensures each one is maintained, monitored, and improved consistently, preventing drift and conflicting assumptions.

Contracts: Contracts define how components interactwhat they expect, provide, and guarantee. Strong, explicit contracts reduce integration surprises and make changes safer in evolving architectures.

Scalability: Scalability ensures systems can handle growth smoothly. Components should scale independently and avoid shared bottlenecks to prevent performance drops during spikes or expansion.

Unintelligent Distributed System

Traditional distributed systems are powerful but inherently unintelligent : every service is specialized, yet entirely dependent on human-designed logic, rules, and constraints. Their behavior reflects the limitations of the engineers who built themtheir time, cognitive bandwidth, domain knowledge, and ability to anticipate edge cases. Even well-architected systems can only operate within the boundaries explicitly encoded into them. They cannot adapt, reason, or make decisions beyond their predefined paths. This creates a landscape where services are efficient at what they were built for but rigid to change, reliant on manual intervention, and limited by the expertise available at design time.

Doctor vs Pharmacist

A helpful way to understand knowledge limitations in decision-making is to compare a doctor and a pharmacist. A pharmacist has deep, specialized expertise in medicationstheir composition, interactions, and safe usage. But when asked to diagnose a condition, their view is constrained; they lack the broader medical context required to interpret symptoms, evaluate risks, or consider alternative explanations. A doctor, on the other hand, combines a wide range of medical knowledge with diagnostic reasoning, allowing them to form a comprehensive picture before making a treatment decision. Both professionals are highly skilled, but the scope of their knowledge determines the quality and safety of their decisions. In complex systems, just like in healthcare, narrow expertise without broader understanding increases risk , while wider knowledge enables deeper analysis and more reliable outcomes.

Amazon Bedrock Agentcore

Amazon Bedrock Agentcore introduces a standardized foundation for building intelligent, agent-driven systems within an enterprise ecosystem. The Agentcore runtime provides a unified execution environment where agents can reason, call tools, manage memory, and orchestrate multi-step workflows without developers having to reimplement these patterns for every use case. It abstracts away the complexity of handling context, chaining operations, and maintaining state, enabling teams to focus on the logic that defines their domain rather than the mechanics of agent interaction.

Complementing the runtime, the Agentcore gateway streamlines communication between agents, services, and external systems. It acts as a controlled interface layermanaging routing, authentication, rate limits, and standardized behavior across all agent interactions. This consistency reduces integration overhead and ensures that agents can be safely exposed within larger architectures.

Together, the runtime and gateway dramatically accelerate product development. Instead of building custom orchestration layers, memory managers, or tool-integration pipelines, teams can assemble specialized AI-powered services much faster. The result is a shift from manual glue code and bespoke patterns to a repeatable platform where intelligent, domain-specific components can be created, tested, and deployed with greater speed and reliability.

E-Commerce System

To illustrate how Agentcore can accelerate intelligent system design, consider a typical e-commerce ecosystem. At its core are specialized services :

Product Service manages catalog information, attributes, and inventory.
Order Service handles shopping carts, checkout, and payment processing.
Shipping Service coordinates delivery options, tracking, and logistics.
Listing Page serves as the customer-facing interface for product search and ordering.

In a traditional distributed setup, these services communicate via well-defined APIs. The Listing Page queries the Product Service to display items, calls the Order Service to process purchases, and interacts with the Shipping Service to show delivery estimates. Each service is specialized and efficient within its domain, but interactions are linear and rigid , and the customer experience is constrained by the explicit logic programmed into each component.

Agentcore introduces the potential for agentic orchestration across these services. Agents can dynamically coordinate tasks, enrich queries, and reason over multiple services simultaneously. For example, a conversational search agent could interact with the Product, Order, and Shipping services to provide contextual recommendations, personalized bundling, or delivery-aware product suggestionsall in a single, seamless interaction. This approach transforms rigid service chains into adaptive, intelligent workflows , enabling faster innovation and more engaging user experiences.

Adopting an Agentic Approach with Agentcore

Agentcore enables an agentic layer over existing services using its runtime and gateway components. Heres how the e-commerce system can leverage them:

Conversation Search Chat UI Component Acts as the frontend interface for customers. It communicates with the Listing Agent via the Agentcore gateway , ensuring safe, standardized interaction with the agent layer without exposing backend complexity.
Listing Agent (Runtime) Maintains context and memory for the conversation, coordinates between agents, and manages multi-step workflows. It handles complex tasks like interpreting user queries, maintaining session state, and orchestrating data retrieval across the system.
Ordering Agent (Runtime) Interacts with both the Product and Shipping services to generate accurate, context-aware order recommendations. The runtime allows it to reason over inventory, pricing, and delivery constraints dynamically, providing qualified results to the Listing Agent.
Product Gateway Serves as a bridge between the Product Service and agents. Using OpenAPI-spec integration, the gateway ensures that requests from agents are routed correctly and consistently, without agents needing to manage low-level API details.
Shipping Gateway Functions similarly for the Shipping Service. It provides standardized access for agents, enabling safe queries and updates regarding delivery options, tracking, and logistics.

By combining Agentcore runtime for backend orchestration (Listing and Ordering Agents) with Agentcore gateways for safe, standardized access to existing APIs (Product and Shipping), the system gains a layer of intelligence on top of existing services. This setup enables conversational search, adaptive ordering workflows, and dynamic coordination between servicesall without modifying the underlying APIs or services.

Traditional Service Integrations

Traditional service integrations rely heavily on the foundational concepts of distributed systems :

Communication Teams coordinate API contracts, call patterns, and failure handling.
Boundaries Clear separation of responsibilities between services and teams prevents accidental coupling.
Specialization and Ownership Each team owns a service domain, ensuring consistent updates and reliability.
Contracts Deterministic API specifications define exactly how services interact.
Scalability Human planning ensures that services can grow sustainably, avoiding bottlenecks or cascading failures.

In these systems, half of the safety and stability comes from human oversight : collaboration, code reviews, discussions on edge cases, and shared understanding of trade-offs. Engineers act as the implicit safety net, catching inconsistencies, reasoning about system-wide effects, and maintaining trust between services.

Allowing agents to fully satisfy these integrations without careful supervision introduces significant risk. Unlike humans, agents may interpret, combine, or explore service interactions in unexpected ways , bypassing implicit constraints and assumptions. This can result in invalid requests, unpredictable workflows, or even cascading failuresputting the reliability and safety of the system at stake. Clear human-defined boundaries, specifications, and guardrails remain essential when introducing agentic layers into established service ecosystems.

Agentic Contract Pitfalls

In traditional distributed systems, contracts are deterministic : each service exposes a set of well-defined operations, with clear input and output formats. Consumers of these services rely on the documented capabilities, and trust comes from a shared, stable understanding of what each service can provide. Exploration and communication are tightly bound to this deterministic model, which minimizes surprises and ensures predictable behavior.

In an agentic system , the dynamics change. Agentsoften powered by foundation models (LLMs)attempt to reason, analyze, and construct the best match for a given task. This introduces uncertainty: even if the underlying services remain stable, the agents interpretation, reasoning, or creative combination of operations can produce unintended behaviors , which may propagate through the system. Without careful constraints, the agent itself becomes a potential single point of failure , capable of generating invalid requests, misusing service capabilities, or creating inconsistent workflows.

To avoid these pitfalls, well-defined service specifications are critical. Each service should provide:

Clean summarization and description a concise overview of what the service does and the intended use cases.
Examples illustrative input-output pairs showing correct usage.
Data formats and types clear definitions of fields, expected types, ranges, and constraints.
Possible values and enumerations to restrict agent actions to valid options.

By providing precise, structured, and machine-readable specifications , teams can guide agents to safely explore and reason over services, reducing the risk of unintentional behavior. Agentic contracts require both human clarity and machine interpretability to ensure that agents enhance workflows without introducing instability, while preserving the trust that traditional deterministic contracts provided.

Agentic Communication Pitfalls

In traditional distributed systems, communication is deterministic and predictable : services exchange well-defined requests and responses, and errors or retries are explicitly handled according to agreed protocols. Agents, however, introduce a fundamentally different mode of communication.

Agents can decide on their own how to handle situations , which may lead to unexpected behaviors. For example:

Ignoring contract priorities an agent may remove search criteria or skip fields without understanding their importance, violating implicit assumptions in the service contract.
Infinite retries in response to failures, an agent may loop endlessly, potentially overwhelming services or consuming unnecessary resources.
Unintended actions agents may execute operations that deviate from the original user request, misinterpreting intent or optimizing incorrectly.

These behaviors are non-deterministic and can propagate across the system, creating unpredictable workflows and system instability. Unlike human operators, agents lack an innate sense of context, risk, and priority unless explicitly constrained. Proper guardrails, monitoring, and well-defined service specifications are essential to prevent agentic communication from becoming a source of errors or cascading failures.

Agents Are Not Doctors

Agents, even when powered by advanced foundation models, cannot become truly specialized in a domain without a supporting agentic ecosystem. Unlike human experts, their reasoning depends entirely on the data and context they are provided. GenAI models come pre-trained on vast amounts of informationboth accurate and misleading, relevant and irrelevantwhich creates a noise-to-signal challenge. This information overload makes it difficult for agents to consistently focus on what truly matters, increasing the risk of misinterpretation, inconsistent decisions, or unsafe actions.

What is often missing is clean, structured, and meaningful data : well-defined decision rules, product and service specifications, clear contracts, and curated examples. Without these foundations, agents may produce plausible-sounding but incorrect results, overlook priorities, or misapply rules. In short, agents cannot become doctors : they lack the domain expertise and curated knowledge that human specialists acquire over years. Specialization requires not just intelligence, but a carefully constructed ecosystem that constrains, guides, and informs agent behaviorturning raw potential into reliable, context-aware expertise.

Conclusion

The adoption of an agentic approach is becoming increasingly evident, as enterprises look to leverage generative AI and intelligent agents to build more adaptive, responsive, and specialized systems. However, success depends on more than just deploying agents: it requires clean and optimized data, detailed product specifications, and precise service contracts. Teams must also focus on reusability and well-defined boundaries to maintain clarity, reduce coupling, and ensure predictable behavior. Rushing adoption without these foundations risks cascading, non-deterministic communication , which can compromise reliability and system integrity. Thoughtful design, structured data, and disciplined system practices remain essential to unlock the full potential of agentic architectures while maintaining control and trust.

AWS Lambda Extensions and Impact of Nodejs native fetch

Omid Eidivandi — Tue, 08 Apr 2025 15:17:32 +0000

Using the Lambda extensions necessitates establishing a connection between the process and the extension API via register() and next() calls. While this seems straightforward, there are caveats to details that can cause interest to be lost in the already initiated MicroVM and lead to the experiment of a new initialization. The following figure illustrates how the Lambda extension is involved during different lifecycle phases.

Telemetry Api Connection

When an extension handshakes with Telemetry Api a connection is established and kept alive during the lambda execution environment lifecycle. This allows the registered connection to be reused for future lambda invocations, hence the next() method calls.

Proactive Initialization

Proactive initialization is the way the aws lambda service helps to reduce cold starts by predicting the required load for any function based on the past data. While this is useful it can introduce some race conditions.

Node Native Fetch

Node Native fetch is an interesting option as it doesnt add unnecessary and external dependencies to the bundle and does the job well. The Fetch is a node Global Api allowing some standard and lightweight http communications, but it comes with its proper characteristics. Fetch uses the undici package which is a from-scratch implementation http-client. But fetch comes with a 5-minute timeout limit and this can not be modified like axios or equivalents.

Connection Timeout

Putting previous details together, in case of proactive initialization if the invocation reaches the lambda after a while ( explicitly 5 minutes ) the extension will experiment with a timeout and will crash. Thanks to the isolation of the extension and handler this will not fail the function invocation but will reinitiate the execution environment. This is on the paper case but Im not sure about that safety as my lambda functions had error metrics fed.

My implementation was from the exact Lambda sample in aws-samples repository here. Knowing that Amazon Q, Copilot, and the internet could not assist me, I jumped into reading all the docs and looked deeply into Fetch, undici, and extension code.

Resolving the problem

Finally, I found the undici globalDispather and looked at it to identify if it matches my needs. assuming, the lambda service must take care of the execution lifecycle, it sounds like a good candidate to resolve the extension problem.

The following snippet shows the simple implementation to tackle the timeout problem.

import { setGlobalDispatcher, Agent } from 'undici';

export const CONNECTION_TIMEOUT_MS = 60 * 60_000;

// Fetch is a global native api
// There is no possibility to configure the agent per request
// So we need to set the global agent configuration
// This is a workaround to globally set the agent configurationsetGlobalDispatcher(new Agent({ 
    connectTimeout: CONNECTION_TIMEOUT_MS,
    headersTimeout: CONNECTION_TIMEOUT_MS, 
    bodyTimeout: CONNECTION_TIMEOUT_MS, 
    keepAliveTimeout: CONNECTION_TIMEOUT_MS,
    keepAliveMaxTimeout: CONNECTION_TIMEOUT_MS
}));

Source Code

The source code example provided here https://github.com/XaaXaaX/aws-lambda-extensions

Conclusion

I wonder if no other AWS customer faced this issue and if so also wonder about how long the aws-samples example can stay as a blueprint for customers and introduce frustration. Im not arguing but showing my frustration for days and teams struggling from those error metrics.

I dont have more to add here, but Ill raise a PR on the provided example to prevent others have the same experience as me .

Enjoy Reading

Egress Rate Controlling in Distributed Systems (Part 2)

Omid Eidivandi — Fri, 28 Mar 2025 15:20:58 +0000

In the precedent part of this series, some scenarios of the producers impact on downstream service were explored. This part will focus on some patterns and practices in asynchronous design to reduce the impact on downstream service.

💡 This part focus on Asynchronous design and related patterns and practices

Real Scenarios

Before starting patterns and practices, lets look at a real scenario where the producer rate could impact downstream services.

Migrating Provisioned DDB to OnDemand

Recently, AWS reduced the cost of DynamoDb onDemand by 50%. This announcement led to many migration decisions. When we moved to On-Demand mode for one of our critical and core services, we recognized a higher processing latency and error rate in one of downstream services. While the overall services functionality was not impacted ( an old serverless system ), the delays and latency introduced by a higher rate of errors had a real business impact.

The above diagram demonstrates a typical design in many companies and works perfectly at some scale. Still, it can be dangerous when the processing rate exceeds the predicted rate. This design introduced significant OpenSearch queries latencies by migrating Dynamodb table to OnDemand.

Some metrics during investigations showed a higher rate of successful write requests compared to the provisioned mode before migration. In Provisioned mode, requests were throttled but succeeded after retries and an exponential backoff strategy for errors allowed balancing the instant peak to a wider time window by delaying it.

Fortunately ( or unfortunately ), experimenting with throttling and retrying for UpdateB allowed a decrease in the processing rate at a given time, helping downstream to continue processing or by giving downstream the chance of recovery. Migrating to OnDemand gave more capacity at initial migration to the producer (4 partitions 12000 RCU, 4000 WCU).

Too many changes increased the load on the downstream service, which in turn led to degraded performance of the downstream service using OpenSearch, the impact was due to the increased number of queries in OS internal buffer and, consequently, increased latency. These latencies lead to increased error rates for Lambda triggers. While not a real-world issue, using Bisect and a large batch size does increase the load on the downstream side.

Design Problems

Distributing many changes at the same time
Introducing the database changes instead of Concrete events representing the final and viable state at a given time (processing time)
Introducing Coupling between two specific services DDB stream and OpenSearch
Propagating Errors
Big BatchSize with Bisect On Failure

High-Level Async Design

In a Timely-Balanced scenario , the Count of instantaneous occurrences and Speed are reduced. Concurrency can consequently be improved but needs to be controlled at a layer based on possibilities.

In an Instantaneous scenario, the Count, speed, and concurrency are increased, which leads to more capacity needed on the downstream side.

While at some scale, Time-Balanced and Instantaneous scenarios produce some close functional results, at a larger scale, more challenges can be considered and refined intentionally. The following high-level diagram can better represent the challenge.

In Part 1 of this series related to Rate Controlling, four attributes were explored: Time Window , Concurrency , Processing Speed , and Count , but another important attribute, State was also under focus.

Producer & Consumer Integrations

While AWS provides many services that allow and simplify integrations in an Event-drive design, each service is purpose-built and has dedicated pros and cons. The following diagram showcases some available integrations. (The Lambdas Presented in this diagram are just to demonstrate a computing resource, but this can be a Fargate container or any other service with computing/processing capability)

💡 The Invocation Models and Service Integrations topic is deeply explained in Chapter 3 of Mastering Serverless Computing with AWS Lambda

In above diagram, three categories of integrations are discussed:

Streams
Pub/Sub
Queues

Streams

Streams such as DynamoDb streams or kinesis are ideal for near real-time data streaming, even in high-throughput scenarios with massive volumes of data. They guarantee ordering, provide means to manage the highest Write throughput requirements, and support handling large amounts of data.

Pub/Sub

Services providing Publ/Sub such as Event-Brdige or SNS have the capacity to distribute any single messages to 1 or many subscribers near realtime and offer high throughput allowing to manages high number of distinct messages, per design they operate in an fully asynchronous way which means they dont track the consumers processing status.

Queues

Queue systems such as SQS guarantee delivery and provide mechanisms for resiliency, load leveling, and decoupling. They provide simplified means to prevent message loss such as keeping messages in the queue till expirations and redrive policies.

General Recommendations

This section provides general guideline to better control the rate and load on downstreams and not focusing on design practices but adopting some level of practices to develop some safeguards.

Batching allows to reduce the rate of statelessness by providing related changes side by side that let the software code have more context to take decisions. Reducing the concurrency can be achieved in different way for each category but at the end a strict concurrency configuration such as ConcurrencyLimit with SQS, ReservedConcurrency with SNS/EB , or Partitioning with Streams allows reducing the execution rate and consequently the risk of overloading the downstream. Synthesizing messages allows being able to produce the final state at that given time by looking at a time-ordered batch of related records.

Source Code

💡 The whole examples can be found in source code GitHub repository

SQS integration

Describing how Amazon SQS works internally is out of scope, but at a high level the polling process of AWS lambda with SQS can be illustrated by following diagram

Aws lambda Asks for Messages and SQS returns available messages from sample nodes, and this is the SQS default behavior, By configuring a longer wait time than zero (default value for short polling), long polling will be applied and SQS will do a best effort to gather more messages from all nodes.

Batching: if a BatchSize specified this will improve the chance of proper batching, allowing more messages to be grouped and allows lambda to have more knowledge about state, while this state is yet distributed and not 100% localized, it will be useful and improve processing quality at some level. Using maxConcurrency option alongside batching allows to have more localized state by increasing the chance of having messages in the same batch. This can be configured using CDK as below.

const queue = new Queue(this, 'Queue', { 
    receiveMessageWaitTime: Duration.seconds(20),
    deadLetterQueue: { 
        maxReceiveCount: 3, 
        queue: new Queue(this, 'DeadLetterQueue') 
    }
});

lambdaFunction.addEventSource(new SqsEventSource(queue, {
    batchSize: 10,
    maxBatchingWindow: Duration.seconds(60),
    maxConcurrency: 2
}));

But Lambda Event Source Mapping provides another configuration option maxBatchingWindow that can be used allowing the lambda service to defer invocation up to 5 minutes while asking for more and more messages form SQS and a function is invoked when one of the following conditions is met as payload size reaches 6MB, Max Batching Window reaches its maximum value, or the Batch Size reaches its maximum value.

The lambda now has more context and can apply more control and synthesize messages. The following snippet shows how in typescript this can be achieved.

const records = (event.Records as SQSRecord[])
    .sort((a, b) => { return a.attributes.ApproximateFirstReceiveTimestamp.localeCompare(b.attributes.ApproximateFirstReceiveTimestamp); })
    .map((record) => { return { ...record, body: JSON.parse(record.body) }});

const grouped = Object.groupBy(records, (currentValue: recordType) => currentValue.body.subject);

const results: SQSRecord[] = [];

grouped.forEach((entry) => { 
    const entryValues = entry?.[1] as any[];
    let firstEvent = entryValues?.splice(0,1)?.[0];
    const initialType = firstEvent.body.type;

    entryValues.forEach((current: any) => { 
        const currentData = current.body.data; 
        const currentType = current.body.type; 

        if(initialType == 'user.signedup') { 
            if(currentType == 'user.profile_updated') { 
                firstEvent.body.data.profile = {
                    ...firstEvent.body.data.profile, 
                    age: currentData.age, 
                    nickname : currentData.nickname, 
                    prefered_channels: currentData.prefered_channels
                };
            } else if(currentType == 'user.unsubscribed') { 
                firstEvent = null;
            } 
        } 
    }); 
    results.push(firstEvent);
});

Sending two messages related to the same id will group them and by synthesizing will avoid distributing or doing unnecessary processes. The following json example represents two distinct events payload sent to SQS.

{ 
    "id": "dmPvPSlwFZOTOjefpu8m2",
    "time": "2025-03-23T17:13:32+00:00", 
    "source": "user.management", 
    "subject": "omid_unique_user_id", 
    "data": { 
        "name": "omid", 
        "profile": { 
            "age": 40 
        } 
    }, 
    "type": "user.signedup"
},{ 
    "id": "TAF5aEQeAh7GX1q8Cvam3", 
    "time": "2025-03-23T17:14:32+00:00", 
    "source": "user.management", 
    "subject": "omid_unique_user_id", 
    "data": { 
        "name": "omid", 
        "age": 43, 
        "nickname": "xaaxaax", 
        "prefered_channels": ["EMAIL"] 
    }, 
    "type": "user.profile_updated"
}

The Example will distribute a final and single user.signedup event representing a combination of both events.

{ 
    "id": "dmPvPSlwFZOTOjefpu8m2", 
    "time": "2025-03-23T17:13:32+00:00", 
    "source": "user.management", 
    "subject": "omid_unique_user_id", 
    "data": { 
        "name": "omid", 
        "profile": { 
            "age": 43, 
            "nickname": "xaaxaax", 
            "prefered_channels": ["EMAIL"] 
        } 
    }, 
    "type": "user.signedup"
}

While the above example provide some details to better control the rate and reducing the risk of overloading downstreams but this approach has no guarantee of control on top of single entities even if in above example the goal was achieved, this means it is possible to have two above messages in two distinct invocations while each invocation receives the respective batch but random entities.

Partitioning: While Batching can be useful but at some points two related messages can be presented in two distinct invocation ( execution environment ) even they are close in terms of time, Grouping related entities and guarantee that they will reach the same invocation (exp. user.signedup and user.profile_updated events related to same user). This can be configured using CDK as below.

const queue = new Queue(this, 'Queue', {
    receiveMessageWaitTime: Duration.seconds(20), 
    fifo: true, 
    deadLetterQueue: { 
        maxReceiveCount: 3, 
        queue: new Queue(this, 'DeadLetterQueue', { 
            fifo: true 
        })
    }
});

lambdaFunction.addEventSource(new SqsEventSource(queue, { 
    batchSize: 10
}));

Sending the previous message payloads presented in batching section, following SQS messages requests must be send to SQS, the presence of MessageGroupId guarantees that at a given time all messages for the same user ( MessageGroupId ) will be strictly received together.

 { 
    "Id": "Test-0001-2015-09-16T140731Z", 
    "MessageGroupId": "omid_user_id", 
    "MessageDeduplicationId": "dmPvPSlwFZOTOjefpu8m2", 
    "MessageBody": "{\r\n \"id\":\"dmPvPSlwFZOTOjefpu8m2\",\r\n \"subject\":\"omid_user_id\",\r\n \"time\":\"2025-03-23T17:13:32+00:00\",\r\n \"data\":{\r\n \"name\":\"omid\",\r\n \"profile\":{\r\n \"age\":34,\r\n \"nickname\":\"omid\"\r\n }\r\n },\r\n \"type\":\"user.signedup\"\r\n }" 
}, { 
    "Id": "Test-0002-2015-09-16T140930Z", 
    "MessageGroupId": "omid_user_id", 
    "MessageDeduplicationId": "2dokCjxmQvUk2brjLuus1", 
    "MessageBody": "{\r\n \"id\":\"2dokCjxmQvUk2brjLuus1\",\r\n \"subject\":\"omid_user_id\",\r\n \"time\":\"2025-03-23T17:14:32+00:00\",\r\n \"data\":{\r\n \"name\":\"omid\",\r\n \"age\":43,\r\n \"nickname\":\"xaaxaax\",\r\n \"prefered_channels\": [\"EMAIL\"]},\r\n \"type\":\"user.profile_updated\"\r\n }" 
}

This example apply a mix of Partitioning and Batching allowing to give more change to lambda have more consistent state which is managed by SQS internally.

DynamoDb / Kinesis Stream Integration

Before deep diving into Streams integrations, this section elaborate on how the integration works. The following diagram illustrates how Items, Item Collections, and stream records are organized.

Aws lambda Asks for List of shards ( aka. Partition in Dynamodb ), Now Lambda handle each shard process in isolation, and starts by getting the Shard Iterator ( aka. start pointer in streaming concept ), giving the iterator the stream returns the items in response. Knowing that each item collection ( items related to same PartitionKey ) is always located in a single Shard the lambda already benefit from having all related data as by default a single lambda execution is dedicated to each shard at a given time which means at least for a single PartitionKey there is a single running invocation.

Using AWS CDK, the integration can be refined to give more flexibility to make concrete decisions. The following snippet shows how the event source mapping can be configured.

lambdaFunction.addEventSource(new DynamoEventSource(table, {
      startingPosition: StartingPosition.LATEST,
      batchSize: 10,
      maxBatchingWindow: Duration.seconds(60),
      bisectBatchOnError: true,
      retryAttempts: 3,
      maxRecordAge: Duration.minutes(15),
      parallelizationFactor: 1,
}));

A batch of 10 records and a 60 seconds batching window is configured, allowing reception of more related records that consequently will allow to reduce the overload on downstreams.

Sending a new item to DynamoDb table and updating it

// New Item
{
    "PutRequest": {
        "Item": {
            "pk": {"S": "omid"},
            "sk": {"S": "t0000"},
            "nickname": {"S": "Omid"},
            "age": {"N": "30"},
            "prefered_channels": {"S": "EMAIL"},
            "status": {"S": "ACTIVE"},
        }
    }
}

// Update Item
{
    "PutRequest": {
        "Item": {
            "pk": {"S": "omid"},
            "sk": {"S": "t0000"},
            "nickname": {"S": "XaaXaaX"},
            "age": {"N": "43"},
            "prefered_channels": {"S": "EMAIL"},
            "status": {"S": "MDIFIED"},
        }
    }
}

Looking at handler code the grouping is done per PK and verification based on eventName being INSERT , MODIFY , or REMOVE.

  const grouped = Object.entries(
    Object.groupBy(records, (currentValue: DynamoDBRecord) => currentValue.dynamodb?.Keys?.pk.S || '')
  );

  const results: DynamoDbInternalRecord[] = [];

  grouped.forEach((entry) => {
    let key = entry?.[0];
    let entryValues = entry?.[1] as DynamoDBRecord[];
    let firstEvent = entryValues?.splice(0,1)?.[0];
    const initialType = firstEvent.eventName;

    let data: Record<string, any> = {};
    if(initialType == 'INSERT' || initialType == 'MODIFY')
      data = unmarshall(result.dynamodb?.NewImage as DynamoDbInternalRecord);
    else if(initialType == 'REMOVE')
      data = unmarshall(result.dynamodb?.OldImage as DynamoDbInternalRecord);

    entryValues.forEach((current: any) => {
      const currentType = current.eventName;
      let currentData: Record<string, any> = {};
      if(initialType == 'INSERT' || initialType == 'MODIFY')
        currentData = unmarshall(current.dynamodb?.NewImage as DynamoDbInternalRecord);
      else if(initialType == 'REMOVE')
         currentData = unmarshall(current.dynamodb?.OldImage as DynamoDbInternalRecord);

      if(initialType == 'INSERT') {
        if(currentType == 'MODIFY') {
          data.age = currentData.age;
          data.nickname = currentData.nickname;
          data.prefered_channels = currentData.prefered_channels;
        }
        else data = {};

      }
    });
    results.push(data);
  });

The printed result for the above records will be as below

{
    "pk": "omid",
    "sk": "t0000",
    "nickname": "XaaXaaX",
    "age": 43,
    "prefered_channels": "EMAIL",
    "status": "ACTIVE"
}

EventBridge / Sns Integration

SNS and EventBridge services will allow a Fanout design with multiple consumers (subscribers). SNS/EB, even though the event contract for SNS contains an array of records ( Records[] ), the lambda always receive a single message in the array ( ref in FAQ: here ). There is no possibility of refined batch processing and custom batching configuration that derives a higher level of concurrent execution of lambdas and a load without the possibility of control. The distribution is an asynchronous call, This means the Channel will not be aware of processing results in consumer side.

Reserved Concurrency : In cases of direct integration with SNS or Eventbridge, the solution to limit downstream impact is to apply reserved concurrency.
OnFailure Destination : Integrate a DLQ. The default behavior is 3 retries with default intervals between each invocation retry (unchangeable):
Load Leveling : Use an SQS as an intermediate layer to facilitate load balancing.

    const lambdaFunction = new NodejsFunction(this, 'LambdaZipFunction', {
      reservedConcurrentExecutions: 1,
      ...
    });

    const topic = new Topic(this, 'Topic');

    lambdaFunction.addEventSource(new SnsEventSource(topic, {
      deadLetterQueue: new Queue(this, 'DeadLetterQueue'),
    }));

Conclusion

Driving design decisions often lies on top of scalability and High-Throughput, While this is an ideal approach, this can become tricky and will impact the business process when boundaries are not well defined and communications are not well thought ( Which is often the case ). Keeping track of surroundings of a service while making such decisions is a must to have and will save time and effort in long term. While crossing out-of-control boundaries ( external system, partners, Saas, etc. ) is not a major case but when exists will cost companies effort, downtime and disruption.

Asynchronous designs give more power and control at communication boundaries but they come with some process oriented decisions by reducing concurrency as a safeguard but also speed for the sake of having more on the fly State consistency.

This part of series explored some details about typical async services and some integration details to control the load on downstream services. The next part of this series will focus on patterns and practices related to rate control in synchronous scenarios where the final product is highly coupled to an external service or downstream with limitations and constraints.

Egress Rate Controlling in Distributed Systems (Part 1)

Omid Eidivandi — Sun, 16 Mar 2025 16:39:40 +0000

Once upon a time, companies thought they could do everything in-houselike being the chef, the waiter, and the dishwasher at a restaurant. But soon, they realized their secret recipe for success was drowning in unimportant tasks like setting up email servers, building payment systems, and reinventing the wheel (which, spoiler alert, no one wants to do). So, they started outsourcing. They realized buying a pre-made pizza dough (API) is way more efficient than grinding the flour, kneading it, and praying it rises correctly every time. With APIs and SaaS, companies can focus on what they do bestserving their core business, while someone else worries about the sauce.

While focusing on core business is essential, external services often present challenges and highlight several considerations. Those services introduce quotas, limits, or constraints that vary based on Business plan, Architecture, or Infrastructure. Whatever the reason, any consumer must plan enough guardrails to isolate the internal process from external services, and those considerations differ based on the layer in the internal system that lives with that coupling.

Coupling Level

It is not avoidable to decouple all the external dependencies, and sometimes, a business needs to integrate some external service to gain complicated expertise such as payment, financing, insurance, etc. But at the same time, there is no way to persist all data for systems such as payment, and even for others such as insurance and financing, there is no possibility to predict all end-user interactions. The coupling can be established in two ways, Direct or Indirect.

Direct

Direct here does not mean Request / Response but more about how the end user product is coupled to the external systems. Coming back to the insurance example, the scenario can be described as:

A Product is available for a user.
The Product is eligible for N years insurance.
The User can modify the insurance default details.
The insurance price is computed per modified details.
The user makes the decision.

When a user changes the insurance duration, this results in a price change and the system needs to interact with the insurance partner service, which, brings the risk of introducing a new point of failure in the system.

Indirect

The above scenario mentions the default value for any single product, While these defaults are essential for the end user, they are often close to being static for a while. However, those values dont relate to the product but to the product category and characteristics. As an example, a Combined Refrigerator ( Frigo/Freezer) with an E-level energy class can have a higher insurance price than an A-class ( just assumptions ), Maybe that price can be lower for well-manufactured and popular brands like SIMENS, AEG, and BOSCH.

The above diagram illustrates a backend service interacting with the partner for product Creation/Modifications. This helps reduce the process coupling level to partner for all product detail page hits by persisting the defaults related to any product Category projected to all available products.

Coupling Layer

Focusing again on the articles topic, defining a rate-controlling strategy depends on the layer's position interacting with the external system, which drives the corresponding design and implementation patterns. Each layer offers distinct possibilities to solve the problem.

Client Side

When a client-side application, such as a Web or Mobile application, interacts with an external system directly or via a gateway proxy, any external system behavior will be replicated in the client app. This can be any latency, throttling, or internal failure problems. In case of rate issues, the client app will receive a 429 HTTP status code, indicating Too Many Requests. The client can retry the call and hope for a successful response. The problem with the client side is that there is no visibility about the overall rate of errors at a given time, and each client app runs with a local state.

Having millions of users means many users can find a single product details page, so they all need to interact with external insurance partners directly or via a gateway proxy. However, the important takeaway is that no single user knows about the overall load on the systems and how many retries are inflight to the external system.

Edge Side

If the process layer interacting with the external system is an edge layer, an edge-level state can help to keep track of the state, this allows to lead better the client apps and also avoids external system calls by applying some level of caching. But this layer adds a challenge of How zonal distributed edge locations can answer the regional rate challenge, while the rate is dedicated to a single regional communication boundary, many edge locations can send requests. However, this problem is many times less important than the client-side problem.

Earlier about the edge, the paragraph mentioned that the edge locations yet follow the same distributed state problems. While this is true, in the real world, Geographic insurance and pricing are definitively different, and this can be an unrealistic challenge except for some rare use cases.

Another option when using content delivery networks ( CDN ) is the Request Collapsing feature; it stands for serving the same origin response for many inflight requests, so the first request reaches the origin while the same requests at a given time are paused while waiting for the response to the first request. this reduces the origin load and, consequently, the external service load. ( The topic of CDN Request Collapsing is deeply discussed in Chapter 9 of Mastering Serverless Computing with AWS Lambda )

Server Side

The server layer is where all the autonomy and power reside. The server-side layer has access to computing power and is closer to the external system. It can be the best for keeping track of the rate limit's actual state and monitoring the actual consumed threshold. Using the server layer helps reduce the challenge of statelessness. However, this may be a more difficult layer for applying retries and exponential backoff of throttled requests, as this will increase the rate of suspended processor resources. The server layer allows synchronous or asynchronous interactions with the external systems. The choice of async or sync varies based on different constraints, and the Coupling level that can be Direct or Indirect.

Direct coupling choice will be synchronous as the end user product waits for the response, this means the interaction with the external service must be done before sending back the response.

All these assumptions can be real when a single container serves the backend service. However, it will suffer from the Concurrency challenge that will be explored later in this article and distributed state if the service is designed for high availability and scalability.

In the case of indirect coupling, an asynchronous backend process is a more flexible solution for interacting with an external system. Often, the async processes are on top of managed services such as SQS, Event Bridge, etc., and react to changes via messaging. This can be an ideal design, but the scaling capacity of managed services can become a point of reflection when interacting with external systems. This can be a real concern if the internal system is designed on top of Function as a service, in which the risk of high concurrency becomes a real concern.

Although the invocations are driven by pollers, each execution environment can receive a batch of records, giving more control to the consumer at the container level. However, the fact of batching by default is related to many factors such as speed, time window, and number of changes.

Processing Main Attributes

A variety of attributes can play a role in the rate of communication toward the external systems such as:

Time Window
Concurrency
Speed
Count

Time Window

A time window represents the time duration that a constraint applies, it is often a per-second metric, but there are services with per-minute or per-hour constraints. Whenever using a service with time window constraints, the base challenge will be, How do we adapt the internal rate in a dedicated time window based on external service constraints?

Concurrency

Concurrency is the ability to run many tasks simultaneously to increase some quality attributes such as Throughput and Scalability. A higher concurrency level can put more pressure on downstream, so crossing the rate constraints and thresholds. Controlling concurrency is a real challenge in distributed systems, and to achieve that control, a decrease in processing speed or throughput will be necessary. Another challenge in highly concurrent processes is state consistency, which can only be shared ( not done ) in a consistent way by context switching that is a complex process and, if achievable, will not allow the control of treatment but only sharing states by the complexity and cost of the real-time context switching process.

Speed

The speed represents how fast a system treats the demands. A higher speed means a higher level of statelessness, and achieving state means reducing the treatment speed. The system speed must be aligned with the overall business value, and there is no interest in driving fast while putting everyone in danger.

Count

The number of demands significantly impacts downstream services. Responding to a high number of demands is toward responsiveness, but how downstream services can handle that rate brings more design-related discussions. However, dealing with external systems becomes tricky because an external system has many more customers than our internal system. Sometimes, a subscription change and, sometimes, a move to another vendor can be the solution. Aligning two roadmaps is not achievable. Controlling the number of demands depends on the layer of coupling.

Trustable State

The rate controlling in distributed systems belongs to consistent, performant, and highly available persistent storage. Why those quality attributes are important?

Achieving a shared state consistently relates to many factors when dealing with distributed storage and will solve the most important challenge of controlling the State.

Availability: is important as unavailability in one node can lead to a stale state. so this will be important to fetch from the most consistent node or fetch and cache the state if any node is replaced. Yet, With all efforts, having a real-time trusted and available state is not an option if we talk about nano or milliseconds.

Performance: leads to lower latency, and lower latency means less overhead on storage, however, high latency will result in some shared state synchronization problems.

Consistency: The state must be consistent over different nodes in near real-time, updating the persisted state must be consistent for all subsequent reads.

DynamoDb, S3, and ElasticCache are some available options on AWS. Many other interesting options, such as Momento, can help achieve a distributed share state.

Conclusion

Designing distributed systems and business processes at scale leads to more inter-service communication needs, while in the best cases, the value is evident, there will be the worst cases that will lead to frustration because of the pressure a service puts on downstream services.

Some services add guardrails such as rate limiting to safeguard their infrastructure health, but this impacts the consumers and based on the presented coupling can impact the final products and users.

In this part of Rate Controlling in Distributed Systems , some concepts related to Rate Controlling, such as Coupling Level , Coupling Layer , Processing Main Attributes , and Trustable State, are explored to give an overview of how the rate is measured .

The new Part of the Series will explore some patterns and technical details to reduce the risk and control the rate in both Sync and Async designs.

Sidecar Pattern In Serverless Design

Omid Eidivandi — Sat, 01 Feb 2025 12:57:06 +0000

Observability becomes significantly more challenging when transitioning to distributed systems, particularly in Serverless architectures. While serverless design is beneficial for decomposition and scalability, its granular nature imposes challenges for observability. Therefore, it is important always to find ways to instrument the software without tightly coupling the instrumentation to the dedicated environment for core software processing.

This article explores two Serverless computing services and offers a straightforward method to centralize logging for Serverless applications:

Cloudwatch Subscription Filters
AWS Lambda Zip package with Extensions
AWS Lambda Custom Image with Extensions
AWS Lambda Web Adapter Image with Extensions
AWS Fargate with sidecar

About Provided Source Code

💡 The whole examples can be found in source code GitHub repository

The source code is designed as mono-repo using NX, and pnpm. The core package is a private lib that shares some central helpers with other modules. observability-core module is the prerequisite for all other modules and provides:

Lambda Extension Layer
ECR Repository
Base Container Image with Extension
Kinesis data Stream
IAM managed policy.

The dependencies are configured via nx.json file in root of the repository for the cdk and build targets. This will force the build and deployment of prerequisite module before the other modules.

{
   "targetDefaults":{
      "cdk":{
         "dependsOn":[
            {
               "projects":"@xaaxaax/observability-core",
               "target":"cdk",
               "params":"forward",
               "required":[
                  "projects",
                  "target"
               ]
            }
         ]
      },
      "build":{
         "dependsOn":[
            {
               "projects":"@xaaxaax/observability-core",
               "target":"build",
               "params":"forward",
               "required":[
                  "projects",
                  "target"
               ]
            }
         ]
      }
   }
}

The targets are defined in the root package.json file scripts section as below:

{
   "scripts":{
      "nx:build:all":"nx run-many --target=build --output-style static --skip-nx-cache",
      "nx:cdk:all":"nx run-many --target=cdk --output-style static --skip-nx-cache --require-approval never"
   }
}

💡 For simplicity the created functions are configured with function URLs and can be triggered easily. The only hint is a function gets triggered two times when triggering from the browser as an invocation will be for favicon.ico.

Cloudwatch Subscription Filters

AWS services like Lambda and Fargate have native integration with CloudWatch, but for critical workloads, the cost of log ingestion can become prohibitive. Depending on usage needs, CloudWatch logs can be utilized selectively, with different approaches available.

When using CloudWatch, Subscription Filters offer a way to forward logs to various destinations, including OpenSearch, Kinesis Data Streams, Amazon Data Firehose, or AWS Lambda.

In this section, CloudWatch Subscription Filters are used to stream logs to an Amazon Kinesis Data Stream for further processing and analysis.

The following code snippet showcases how to implement this log forwarding mechanism using AWS CDK:

    const logGropup = new LogGroup(this, 'LogGroup', {
      logGroupName: `/aws/lambda/${lambdaWithCloudwatch.functionName}`,
      retention: RetentionDays.ONE_DAY,
      removalPolicy: RemovalPolicy.DESTROY, });
    const logsDeliveryRole = new Role(this, `LogsDeliveryRole`, { 
      assumedBy: new ServicePrincipal('logs.amazonaws.com')
    }); 

    logGropup.addSubscriptionFilter('SubscriptionFilter', { 
      destination: new KinesisDestination(LogStream,{ 
         role: logsDeliveryRole 
     }),
     filterPattern: { 
         logPatternString: ' ', // this configure all logs to be filtered 
     } 
   }) 

   LogStream.grantWrite(logsDeliveryRole);

Invoking the lambda function will result sending log records in kinesis via cloudwatch subscription filters as shown is the following figure.

Lambda Telemetry Api

Lambda offers a Telemetry API, which will be excellent choice to capture function log records without using the cloudwatch logs. The logs received through the Telemetry API follow a straightforward format, as shown below.

{
   "time":"2025-01-30T00:00:00.000Z",
   "type":"function",
   "record":{
      "timestamp":"2025-01-30T00:00:09.429Z",
      "level":"INFO",
      "requestId":"79b4f56e-95b1-4643-9700-2807f4e68189",
      "message":"Log Message HERE"
   }
}

If The lambda LogFormat is TEXT the received format will be as the following snippet.

{
   "time":"2025-01-30T00:00:00.000Z",
   "type":"function",
   "record":"2025-01-30T00:00:09.429Z 79b4f56e-95b1-4643-9700-2807f4e68189 [INFO] Log Message HERE"
}

Lambda Extensions

For high-throughput applications, relying on CloudWatch Logs can lead to substantial costs. One way to mitigate this is by denying CloudWatch Logs permissions in the Lambda execution role. This prevents the Lambda service from sending logs to CloudWatch, and the consequent will be preventing the use of "Subscription Filters."

However, Lambda provides a Telemetry API that captures all logs, even when CloudWatch logging is disabled. By using the Extension API, you can subscribe to the Telemetry API and register for specific log categories, such as platform, function, or extension logs.

💡 The Source code repository provides the extension module here, that will be used for Lambda Zip package, Custom Image and Web Adapter image sections

The execution of extensions, whether as a standard ZIP package or a custom image, is managed by the Lambda runtime. The Lambda service scans the /opt/extensions directory and automatically executes any extensions found in that location.

For ZIP package deployments, this attachment occurs during the Lambda initialization phase, where the extensions path is constructed by aggregating all attached layers. However, for custom images, this structure must be manually set up during the container image build process.

This project generates two final assets from the same extension source, along with other previously mentioned resources. The extension itself is built using esbuild and bundled as JavaScript, with a post-build script handling the folder structure setup.

The provided final assets are:

A Lambda Layer
A ECR Base Container Image

Lambda Layer

For the layer the build process do all necessary steps. The only required step is to use CDK for creating the layer. The following snippet demonstrates the way to create a layer using CDK.

   const extension = new LayerVersion(this, 'kinesis-telemetry-api-extension', { 
      layerVersionName: `${props?.extensionName}`,
      code: Code.fromAsset(resolve(process.cwd(), `build`)),
      compatibleArchitectures: [
        Architecture.X86_64, 
        Architecture.ARM_64
      ], 
      compatibleRuntimes: [
         Runtime.NODEJS_20_X, 
         Runtime.NODEJS_22_X
      ], 
      description: props?.extensionName
   }); 
   // Exporting the Layer Arn to parameter store 
   new StringParameter(this, `LambdaExtensionArnParam`, { 
       parameterName: `/${props.contextVariables.stage}/${props.contextVariables.context}/telemetry/kinesis/extension/arn`,
       stringValue: extension.layerVersionArn
   });

The LayerVersion resource point to the build directory generated by build script, The underlying build folder structure is as below

- build 
   - extensions 
      - kinesis-telemetry-extension
   - kinesis-telemetry-extension
     - index.js

The kinesis-telemetry-extension file under extensions folder is an executable file that will be the entry point for lambda service to detect extension and execute it. The file name must be equal to the extension directory name for this example executable.

#!/bin/bash

set -euo pipefail
OWN_FILENAME="$(basename $0)"
LAMBDA_EXTENSION_NAME="$OWN_FILENAME"

echo "[extension:bash] launching ${LAMBDA_EXTENSION_NAME}"
exec "/opt/${LAMBDA_EXTENSION_NAME}/index.js"

Base Container Image

The base custom image with extension included is created using a Dockerfile. The Dockerfile simply use the built asset ( build folder ) contents and move it to the /opt/ directory in built image.

FROM node:22.13.1-slim

COPY build /opt/

WORKDIR /opt/extensions

The Image will be built and pushed to the ECR repository created via cdk which is a prerequisite for pushing the image to ECR. This is done using a post script.

{
   "name":"@xaaxaax/observability-core",
   ...
   "scripts":{
      "build:docker":"docker buildx build --platform linux/arm64 --no-cache -t $ECR_REPOSITORY:latest .",
      "postbuild:docker":"pnpm run build:docker:login && pnpm run build:docker:tag && pnpm run build:docker:push",
      "build:docker:login":"aws ecr get-login-password --region $REGION --profile admin@dev | docker login --username AWS --password-stdin $ECR_URI",
      "build:docker:tag":"docker tag $ECR_REPOSITORY:latest $ECR_URI/$ECR_REPOSITORY:latest",
      "build:docker:push":"docker push $ECR_URI/$ECR_REPOSITORY:latest",
      "cdk":"cdk --profile admin@dev --app 'tsx ./cdk/bin/app.ts' -c env=dev",
      "postcdk":"cross-env REGION=eu-west-1 ECR_URI=904233108557.dkr.ecr.eu-west-1.amazonaws.com ECR_REPOSITORY=lambda-telemetry-image pnpm run build:docker"
   }
   ...
}

Lambda Zip package with Extensions

Dealing with Zip lambda package is the simplest option by attaching the layer to the lambda function. but also giving the required permissions to the associated role for underlying infrastructure that the extension need to interact with that is kinesis data stream in the provided example.

The following CDK represents how the extension layer can be attached to the function and what are the required permissions.

    const extensionArn = StringParameter.fromStringParameterName( 
       this, 
       'extensionId', 
       `/${props.contextVariables.stage}/logs-collector-lambda-extension/telemetry/kinesis/extension/arn`
    ).stringValue;

    const managedPolicyArn = StringParameter.fromStringParameterName(
       this, 
       'policyName', 
       `/${props.contextVariables.stage}/logs-collector-lambda-extension/telemetry/kinesis/runtime/policy/arn`
    ).stringValue; 

    const functionRole = new Role(
       this, 
       'LambdaFunctionRole', { 
         assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
         managedPolicies: [
            ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole'),
            ManagedPolicy.fromManagedPolicyArn(this, 'managed-policy', managedPolicyArn)] }); 

    const lambdaFunction = new NodejsFunction(this, 'LambdaZipFunction', { 
        entry: resolve(process.cwd(), 'src/handler.ts'), 
        ... 
        bundling: { ... }, 
        layers: [
           LayerVersion.fromLayerVersionArn(this, 'ExtensionArn', extensionArn)
        ] 
    });

Lambda Custom Image With Extensions

The example custom images build a docker image base function from a Dockerfile based on provided base image with extension included.

The Dockerfile is based on both extension image and lambda nodejs 22 image provided by aws. The interesting part of AWS provided image is that it can be run locally and can be invoked for example using a curl command.

FROM 904233108557.dkr.ecr.eu-west-1.amazonaws.com/lambda-telemetry-image:latest AS extensions
FROM public.ecr.aws/lambda/nodejs:22

WORKDIR ${LAMBDA_TASK_ROOT}

COPY dist/* ./
COPY --from=extensions ./opt/ /opt/

CMD ["index.handler"]

The built asset will be copied to the /var/task path that can be accessed using LAMBDA_TASK_ROOT env variable and finally the CMD layer point to the handler inside index.js file.

💡 In the example the base image imageUri is hardcoded in Dockerfile but it can be parametrized using parameter store and passing as docker build ARGs,

Lambda Web Adapter Image With Extensions

While Lambda Adapter provides a custom runtime, it brings some particularity to the way the Dockerfile shall be used. As per LWA documentation and examples, the base image used is public.ecr.aws/docker/library/node:22.9.0-slim and not public.ecr.aws/lambda/nodejs:22 which means the lambda api interface no more can be used for local invocation, e.g using curl.

The example Dockerfile uses multiples stages

FROM 904233108557.dkr.ecr.eu-west-1.amazonaws.com/lambda-telemetry-image:latest AS extensions
FROM public.ecr.aws/awsguru/aws-lambda-adapter:0.9.0-aarch64 AS webadapter
FROM public.ecr.aws/docker/library/node:22.9.0-slim

WORKDIR ${LAMBDA_TASK_ROOT}

COPY dist/* ./
COPY --from=extensions ./opt/ /opt/
COPY --from=webadapter /lambda-adapter /opt/extensions/lambda-adapter

CMD ["node", "index.js"]

The image will use the extension base image alongside the lambda web adapter base image and use the contents of ./opt folder related to each extension. Also it uses the built asset of function code ( here a http server on default port of LWA 8080 ).

The particularity behavior for LWA in our example is the way function logs are received in extension. Only the function logs are under this unfortunate behavior and are not formatted as a valid json object but are treated as text event while the LambdaLogFormat is set as JSON. so the above official format is not working and element.record.message will result an undefined value. The following shows how the record is received which is a representation of Javascript object surrounded by double quotes.

{
   "time":"2025-01-29T21:24:33@.665Z",
   "type":"function",
   "record":"{ name: 'omid' }"
}

To resolve the problem, the extension is adopted to look at element.record if the element.record.message is value. But event the change is not sufficient as the received record is a double quoted JS object. So the log data must be formatted using JSON.stringify().

console.log(JSON.stringify( logObject ));

Fargate with Firelens sidecar

Fargate, as a serverless solution for running containers on demand, supports both short-lived and long-running tasks. Regardless of the use case, enabling containers to communicate and complement each others capabilities is essential for building scalable and efficient architectures.

In line with the examples in this article, this section demonstrates how to forward container logs to a central Kinesis Data Stream. To achieve this, the Fargate task can include a sidecar container responsible for collecting logs and forwarding them to the data stream.

The application container provides a Dockerfile as below

FROM --platform=linux/arm64 public.ecr.aws/docker/library/node:22-slim

COPY dist/* ./

CMD ["node", "index.js"]

But as mentioned above, there will be another Dockerfile for the log forwarder container that uses the FluentBit image provided by aws.

FROM amazon/aws-for-fluent-bit:latest

ADD container.conf /container.conf
ADD parsers.conf /parsers.conf

As shown in the Dockerfile there are two configuration files for Parsing and Container specific configurations such as Filtering , etc. the content of both files are as below

// parsers.conf file
[PARSER]
    Name    log_json
    Format  json

// container.conf file
[SERVICE]
    Parsers_File    parsers.conf

[FILTER]
    Name            parser
    Match           *
    Key_Name        log
    Parser          log_json

[FILTER]
    Name            grep
    Match           *
    Regex           app_name fargate-example-app

Let see how these are deployed and resources are created. AWS Cdk provides the L2 constructs that can be used to simplify the infra as code steps. The example uses the FargateTaskDefinition and FargateService constructs.

   const jobDefinition = new FargateTaskDefinition(this, 'JobDefinition', { 
      cpu: 256, 
      memoryLimitMiB: 512, 
      runtimePlatform: { 
        cpuArchitecture: CpuArchitecture.ARM64,
        operatingSystemFamily: OperatingSystemFamily.LINUX
      }, 
      taskRole: jobTaskRole, 
      executionRole: jobTaskExecutionRole
   });

After creating the base TaskDefinition, the app container and firelens router will be added as below

    jobDefinition.addContainer('Container', { 
      image: ContainerImage.fromAsset(join(process.cwd())), 
      logging: LogDrivers.firelens({ 
        options: { 
          Name: 'kinesis_streams', 
          region, 
          stream: props.streamName
        }
      })
    }); 

    jobDefinition.addFirelensLogRouter('LoggingContainer', { 
       image: ContainerImage.fromAsset(join(process.cwd(), 'fluent-bit')), 
       logging: LogDrivers.awsLogs({ 
          streamPrefix: 'logging',
          logGroup: new LogGroup(this, 'FireLensLogGroup', {
             logGroupName: `/ecs/${props.contextVariables.context}`, 
             retention: RetentionDays.ONE_DAY, 
             removalPolicy: RemovalPolicy.DESTROY
         })
      }),
      environment: { FLB_LOG_LEVEL: 'info' }, 
      firelensConfig: { 
        type: FirelensLogRouterType.FLUENTBIT, 
        options: { 
          configFileType: FirelensConfigFileType.FILE, 
          configFileValue: '/container.conf'
        }
      }
   });

A service shall be created to encapsulate a task consisting of two side by side containers. The is simple and straightforward.

   const service = new FargateService(this, 'Service', {
      cluster, 
      capacityProviderStrategies: capacityStrategy,
      desiredCount: 1,
      platformVersion: FargatePlatformVersion.VERSION1_4,
      propagateTags: PropagatedTagSource.TASK_DEFINITION, 
      taskDefinition: jobDefinition, 
      assignPublicIp: true, 
      vpcSubnets: { 
        subnets: vpc.publicSubnets 
      }, 
      securityGroups: [taskSecurityGroup]
   });

💡 For simplicity, the example allow assigning a public ip address to the task and put the service in public subnets, the reason that this is required is the FargatePlatformVersion.VERSION1_4 is under managed awsvpc and this is the simplest way to let fargate pull images from ECR. This is not recommended for production cases.

The Task role must have the kinesis PutRecords action permissions. Here the observability-core stack provides a managed policy that can be attached to the role.

  const managedPolicyArn = StringParameter.fromStringParameterName(
    this, 
    'ObservabilityManagedPolicy',
    `/${props.contextVariables.stage}/logs-collector-observability-core/telemetry/kinesis/runtime/policy/arn`
  ).stringValue; 

  const jobTaskRole = new Role(this, 'JobTaskRole', { 
     assumedBy: new ServicePrincipal('ecs-tasks.amazonaws.com'), 
     managedPolicies:[ 
        ManagedPolicy.fromManagedPolicyArn(this, 'TaskRoleManagedPolicy', managedPolicyArn)
     ]
 }); 

  const jobTaskExecutionRole = new Role(this, 'JobTaskExecutionRole', {
     assumedBy: new ServicePrincipal('ecs-tasks.amazonaws.com'),
     managedPolicies: [ 
         ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonECSTaskExecutionRolePolicy')
     ]
   });

After deploying attached IP to the created ENI can be used over http protocol. The logs will be sent to the Kinesis DS as shown in following screenshot.

💡 The Firelens has the same problem as LWA mentioned before, The log metadata object must be stringyfied. If the JS object is directly logged the same behavior will be provided as Web Adapter.

Conclusion

While Serverless offers a wide range of managed services that scale per needs, It is important to forget the shared responsibility that forces engineering teams to be engaged on their side. As part of software development the use of Processors capacity and Memory is the part under engineering teams ownership. This is not far from traditional software principals but is somehow forgotten by the fascinating nature of managed services.

Using Lambda extensions, multi container, or background processes is the way to apply processing isolation and achieve more trustable software which is running as a foreground process.

The article focuses on log aggregation to represent how decouple the critical processing from non critical ones via isolation and provides some examples to showcase the implementation in different scenarios.

Personal Use of AWS Organizations Using CDK

Omid Eidivandi — Tue, 05 Nov 2024 00:47:42 +0000

Over the years of using AWS, Ive invested a lot of effort into managing costs and maintaining a tidy account. However, I began creating new accounts and closing them once they were no longer needed. This approach has led to several challenges, including:

Complications in billing management
Inability to share credits
Challenges with quota management
Orphaned resources
A $1 charge for each account created
The need to create temporary emails and manage contact information

AWS Organization

AWS Organizations is a service from AWS designed to streamline the centralized management of accounts. It offers features such as account provisioning, centralized billing, access management, policy management, and enforcement of standards.

Organizational Unit

An Organizational Unit (OU) is a grouping of accounts that allows for the distribution of accounts based on specific contexts or needs. For example, you might create one OU for workload accounts (development, testing, production) and another for security or networking purposes.

This separation enables additional automation for the member accounts, such as bootstrapping or deploying infrastructure tailored to specific contexts.

AWS CDK and challenges

Onboarding a new organization using CDK initially appeared to be as straightforward as creating any other piece of infrastructure. However, as I began implementation, I encountered several gaps due to missing features in CloudFormation and the Dedicated Service API.

The AWS Organizations service requires trusted access, which is only available through the Organizations Service API.
Activating SSO with IAM Identity Center can only be done through a manual process in the console.
Creating accounts necessitates unique email addresses, which makes it frustrating to set up multiple Gmail or whatever accounts.
While setting up SES with Route 53 was relatively easy, the documentation was confusing and misleading.

Source Code

https://github.com/XaaXaaX/aws-cdk-organization-setup

Phase 1: Configuration

Since setting up an organization involves more details than an application, its important to consider the relevant configuration sections. The following ConfigType outlines the structure of the stack configuration.

export type Config = {
  contextVariables: ContextVariables;
  dns: DNSConfig;
  org: OrgConfg;
  sso:  SSOConfig;
}

DNS Config

DNSConfig should account for either importing an external HostedZone or creating a new one. By separating the types, we can enhance type safety and maintain better control within the stack.

type ExternalZone = { isExternal: true; hostedZoneId: string; }
type InternalZone = { isExternal: false; domainName: string; }
export type DNSConfig = (ExternalZone | InternalZone) & {
  mailExchangeDomainName: string;
};

Org Config

OrgConfig includes configuration attributes related to account creation and bootstrapping, as well as the activation of trusted services and parameter sharing.

export type OrgConfg = {
  memebers: {
    bootstrap?: boolean;
    accounts: {
      accountName: string;
    }[];
  };
  trustedAWSServices?: string[];
  crossAccountParametersSharing?: boolean;
};

SSO Config

SSOConfig features two distinct types: Ready and NotReady. This distinction allows for the creation of SSO and identity store groups, along with permission sets, after implementing the Click-Ops solution in the management account.

export type SSONotReady = { isReadyToDeploy: false; }
export type SSOReady = { 
  isReadyToDeploy: true; 
  ssoInstanceArn: string;
  identityStoreId: string; 
}
export type SSOConfig = SSONotReady | SSOReady;

HostedZone and Emailing

The management account can either own an existing HostedZone or create a new one for a domain name, and also set up the necessary components to receive emails. This is crucial for having unique email addresses for each account while still allowing them to be forwarded to a personal account.

The process for importing or creating the hosted zone is outlined below:

    let hostedZone: IHostedZone;
    if(dnsConfig.isExternal)
      hostedZone = HostedZone.fromHostedZoneId(this, 'HostedZone', dnsConfig.hostedZoneId);
    else {
      hostedZone = new HostedZone(this, 'HostedZone', { zoneName: dnsConfig.domainName, comment: 'Managed by CDK' });

      new StringParameter(this, 'HostedZoneId', {
        parameterName: `/${this.ENV}/${this.CONTEXT}/${dnsConfig.domainName}/hostedzone/id`,
        stringValue: hostedZone.hostedZoneId,
      })
    }

To enable email reception through the HostedZone, an MXRecord must be added. Note that the mailExchangeDomainName configuration can be either the same as the domain name (e.g., example.com) or a subdomain (e.g., mail.example.com).

  new MxRecord(this, 'MXRecord', {
      zone: hostedZone,
      values: [{
        hostName: `inbound-smtp.${this.REGION}.amazonaws.com`,
        priority: 10,
      }],
      recordName: dnsConfig.mailExchangeDomainName,
      deleteExisting: true,
    });

To receive emails using SES, you need to extend its capabilities through ReceiptRuleSet actions, as SES does not natively provide a mailbox option. To integrate SES, you must create an EmailIdentity, which should be of the Domain type, accomplished by providing the HostedZone.

    new EmailIdentity(this, 'Identity', {
      identity: Identity.publicHostedZone(HOSTED_ZONE),
    });

The ReceiveRuleSet allows you to establish rules for receiving emails and define actions to be taken. The actions specified in a rule will be executed in sequence. In this example, each incoming email will be stored in an S3 bucket, and a Lambda function will be triggered to process the stored content and forward it to other email servers.

const receiptRuleSet = new ReceiptRuleSet(this, 'MailReceivedRuleSet', {
        dropSpam: false,
        rules: [{
          tlsPolicy: TlsPolicy.REQUIRE,
          scanEnabled: false,
          enabled: true,
          recipients: [ HOSTED_ZONE.zoneName ],
          actions: [
            new S3({ bucket: deliveryBucket }),
            new Lambda({ function: receivedMailLambda }),
          ],
        }],
    });

Deploying the stack outlined above will create all the necessary resources for a functional domain name with email reception. However, if you attempt to send an email from your address, you will receive a postmaster response indicating that the email sending has failed.

inbound-smtp.eu-west-1.amazonaws.com has generated this error :

Requested action not taken: mailbox unavailable

💡 SES permits only one active ruleset at a time, so when you create a new ruleset, it will not be activated automatically.

To activate the ruleset, you can use AWS CDK custom resources with SDK calls, as shown below.


    const rulesetActivationSDKCall: AwsSdkCall = {
        service: 'SES',
        action: 'setActiveReceiptRuleSet',
        physicalResourceId: PhysicalResourceId.of('SesCustomResource'),
    };

    const setActiveReceiptRuleSetSdkCall: AwsSdkCall = {
      ...rulesetActivationSDKCall,
      parameters: { RuleSetName: receiptRuleSet.receiptRuleSetName }
    };
    const deleteReceiptRuleSetSdkCall: AwsSdkCall = rulesetActivationSDKCall;

    new AwsCustomResource(this, "setActiveReceiptRuleSetCustomResource", {
      onCreate: setActiveReceiptRuleSetSdkCall,
      onUpdate: setActiveReceiptRuleSetSdkCall,
      onDelete: deleteReceiptRuleSetSdkCall,
      logRetention: RetentionDays.ONE_WEEK,
      policy: AwsCustomResourcePolicy.fromStatements([
        new PolicyStatement({
          sid: 'SesCustomResourceSetActiveReceiptRuleSet',
          effect: Effect.ALLOW,
          actions: [
            'ses:SetActiveReceiptRuleSet',
            'ses:DeleteReceiptRuleSet',
          ],
          resources: ['*']
        }),
      ]),
    });

This solution lets to activate the created ruleset and the ses reception now works as expected.

💡 The provided example only trigger a lambda function that logs an ses event but you can implement your email forwarding if needed.

Organization and Accounts

The CDK for AWS Organizations only offers L1 constructs, but I find this approach simple and straightforward. In my opinion, adding L2 constructs might be unnecessary over-engineering, so Im fine with this CDK decision for now.

To create an Organization and an OU, the only parameters required are the FeatureSet of the Organization, which can be either CONSOLIDATED_BILLING or ALL.


    const orga =new CfnOrganization(this, 'Organization', { featureSet: 'ALL' });

    const orgUnit = new CfnOrganizationalUnit(this, 'OrganitationUnit', {
      name: `workloads${tempSuffix}`,
      parentId: orga.attrRootId
    });

    orgUnit.addDependency(orga);

The following snippets illustrate how to create accounts. In this example repository, the account list is provided through configuration, meaning the accounts parameter will be an array of objects in the format { accountName: string }. The created account IDs will be stored in the parameter store, although this may not be necessary for a personal organization setup.

ACCOUNTS.forEach((account: { accountName: string }) => {
      const awsAccount =new CfnAccount(this, `${account.accountName}Account`, {      
        accountName: `${account.accountName}${tempSuffix}`,
        email: `${account.accountName}${tempSuffix}@${DOMAIN_NAME}`,
        parentIds: [orgUnit.attrId],
      });

      const param = new StringParameter(this, `${account.accountName}AccountIdParam`, {
        stringValue: awsAccount.attrAccountId,
        description: `Account ID for ${awsAccount.accountName}`,
        parameterName: `/${this.ENV}/${this.CONTEXT}/${awsAccount.accountName}/account/id`,
      })
    });

To set up SSO using IAM Identity Center, it's essential to enable AWS Organization trusted access. Unfortunately, theres no way to activate this feature using CDK or CloudFormation, so we will once again rely on a Custom Resource. In this example, all services required for trusted access are specified through configuration (e.g., sso.amazonaws.com and servicequota.amazonaws.com).


    trustedServices.forEach((service: string) => {

      const identifier = service.replace('.', '');
      const enable: AwsSdkCall = {
        service: 'organizations',
        action: 'enableAWSServiceAccess',
        physicalResourceId: PhysicalResourceId.of(`OrgCustomResource${identifier}`),
        parameters: { ServicePrincipal: service },
      };

      const disable: AwsSdkCall = {
        service: 'organizations',
        action: 'disableAWSServiceAccess',
        physicalResourceId: PhysicalResourceId.of(`OrgCustomResource${identifier}`),
        parameters: { ServicePrincipal: service },
      };

      new AwsCustomResource(this, `AWSServiceAccessActivation${identifier}CustomResource`, {
        onCreate: enable,
        onUpdate: enable,
        onDelete: disable,
        logRetention: RetentionDays.ONE_WEEK,
        policy: AwsCustomResourcePolicy.fromStatements([
          new PolicyStatement({
            sid: 'OrgCustomResourceSetOrgAWSServiceActivation',
            effect: Effect.ALLOW,
            actions: [
              'organizations:enableAWSServiceAccess',
              'organizations:disableAWSServiceAccess',
            ],
            resources: ['*']
          }),
        ]),
      });
    })

SSO Setup

The provided example configuration uses a NotReady state by setting isReadyToDeploy = false, which prevents the CDK deploy from generating the SSO configuration, as the SSO instance has not yet been created. As mentioned earlier, this cannot be accomplished through automation or API calls; the only available API call for CreateInstance works solely for standalone accounts, not for Organization Management accounts.

Before changing the flag to true, you must go to the AWS Console in the management account, navigate to IAM Identity Center, and click the Enable button. After activation, youll need the SsoInstanceArn and IdentityStoreId, which should be set in the stack configuration file along with isReadyToDeploy=true.

Once these steps are completed, the CDK deploy will proceed to deploy the stack along with all associated groups and permission sets.

    const group = new CfnGroup(this, id, {
      displayName: `${id}`,
      description: `${id} Group`,
      identityStoreId,
    });

    const permissionSet = new CfnPermissionSet(this, `${id}PermissionSet`, {
      name: `${id}@${ENV}`,
      description: `${id}@${ENV}`,
      instanceArn: ssoInstanceArn,
      managedPolicies: managedPolicies,
      inlinePolicy: undefined,
      sessionDuration: Duration.hours(12).toIsoString(),
    });

    accounts.forEach((account) => {
      new CfnAssignment(this, `${id}Assignment`, {
        instanceArn: ssoInstanceArn,
        permissionSetArn: permissionSet.attrPermissionSetArn,
        principalId: group.attrGroupId,
        principalType: 'GROUP',
        targetId: account,
        targetType: 'AWS_ACCOUNT',
      });
    })

In the CDK snippet above, a group is created in the Identity Store, and a permission set is established for the SSO Instance. This group is assigned to each of the accounts created earlier. The code illustrates the Group Construct, which is utilized as shown below.

    // Org Accounts
    const developmentAccount = StringParameter.fromStringParameterName(this, 'AccountSecurity', `/${this.ENV}/${this.CONTEXT}/security_b/account/id`).stringValue; 

    //Managed Policies
    const adminManagedPolicy = ManagedPolicy.fromAwsManagedPolicyName('AdministratorAccess');
    const poweredUserManagedPolicy = ManagedPolicy.fromAwsManagedPolicyName('PowerUserAccess');
    const readonlyManagedPolicy = ManagedPolicy.fromAwsManagedPolicyName('ReadOnlyAccess');

    new Group(this, 'Admin', { 
      contextVariables: this.CONTEXT_VARIABLES,
      ssoInstanceArn: SSO_INSTANCE_ARN,
      identityStoreId: IDENTITY_STORE_ID,
      managedPolicies: [ adminManagedPolicy.managedPolicyArn ],
      accounts: [ developmentAccount ]
    });

    new Group(this, 'PowerUser', { 
      contextVariables: this.CONTEXT_VARIABLES,
      ssoInstanceArn: SSO_INSTANCE_ARN,
      identityStoreId: IDENTITY_STORE_ID,
      managedPolicies: [ poweredUserManagedPolicy.managedPolicyArn ],
      accounts: [ developmentAccount ]
    });

    new Group(this, 'Developer', { 
      contextVariables: this.CONTEXT_VARIABLES,
      ssoInstanceArn: SSO_INSTANCE_ARN,
      identityStoreId: IDENTITY_STORE_ID,
      managedPolicies: [ readonlyManagedPolicy.managedPolicyArn ],
      accounts: [ developmentAccount ]
    });

You can now create a user in IAM Identity Center and assign it to one or more groups. Next, navigate to the AWS Access Portal (accessible via the link from the Identity Center dashboard: https://d-123456788.awsapps.com/start), which will prompt you for login credentials.

Once logged in, the application page will allow you to select the appropriate group role and access it with the corresponding permissions.

Account Boostrap

It was an effective way to automate account creation and implement varying levels of security. However, an empty account requires several repetitive tasks before it can be considered usable. The example includes setting up GitHub OIDC and CDK Bootstrap to prepare the member accounts.

The bootstrap can be deactivated through configuration, and this will be verified in the organization stack as shown below.

   if( BOOTSTRAP ) {
      new Bootstrap(this, 'Bootstrap', { 
        contextVariables: props.contextVariables,
        regions: [ this.REGION ],
        organizationUnits: [ orgUnit ],
        types: {
          [BootstrapTypes.CDK]: { FileAssetsBucketKmsKeyId: 'AWS_MANAGED_KEY' },
          [BootstrapTypes.GitHub]: { Owner: gitHubConfig.owner, Repo: '*' }  
        }, 
      })
    }

The bootstrap construct creates a set of CloudFormation StackSets to allow the management account to bootstrap the member accounts, which will be triggered when accounts are created or updated. Unfortunately, the CDK does not offer a straightforward way to use CDK stacks for StackSets. After researching online, I found many solutions to be overly complicated, so I opted to stick with the readily available YAML templates found across the web (even though I probably wont look at or modify them). I'm fine with this approach.

    export enum BootstrapTypes {
        GitHub = 'oidc-github.yml',
        CDK = 'cdk-bootstrap-template.yml',
     }

    const { contextVariables: { stage: ENV, context: CONTEXT }, types: TYPES } = props;
    const tags =  Stack.of(this).tags.renderTags();

    Object.keys(TYPES).forEach((value: string, index: number) => {
      const typeIdentifier = value.replace('.yml', '').replace(/[^a-zA-Z]/g, '');
      const cfnParams = Object.entries(TYPES[value as unknown as BootstrapTypes])
        .map(([key, value]) => (
          { parameterKey: key, parameterValue: value } as CfnStackSet.ParameterProperty
        )); 

      new CfnStackSet(this, `BootstrapStackSet${typeIdentifier}`, {
        permissionModel: "SERVICE_MANAGED",
        stackSetName: `${CONTEXT}-bootstrap-${typeIdentifier}-${ENV}`,
        description: `Account bootstrap StackSet ${typeIdentifier}`,
        autoDeployment: { enabled: true, retainStacksOnAccountRemoval: false },
        capabilities: ["CAPABILITY_NAMED_IAM"],
        templateBody: readFileSync(join(process.cwd(), `/cdk/lib/orga/bootstrap/${value}`), 'utf8'),
        parameters: cfnParams,
        tags,
        operationPreferences: { failureToleranceCount: 1, maxConcurrentCount: 1 },
        stackInstancesGroup: [{
          regions: props.regions,
          deploymentTargets: {
            organizationalUnitIds: props.organizationUnits.map((ou: { attrId: string }) => ou.attrId), 
          },
        }],
      });
    });

Conslusion

For a long time, I had been trying to set up an organization, but since I couldn't find a working piece of code, I decided to dive into it myself. It was an exciting experience, tackling different challenges and solving them along the way. Kudos to AWS CDK for its flexibility!

Having an organization is a great way to experiment and easily tear things down afterward. When closing accounts, youll incur charges for 90 days after theyve been removed from the organization. However, after this 90-day period, the accounts will be permanently deleted. During this time, you can still recover the account, access it with limited permissions, and perform certain actions.

Conventional Use of AWS CDK

Omid Eidivandi — Mon, 04 Nov 2024 00:10:29 +0000

Infrastructure as code, a principal rule of agility and reliability, helps deliver configurable software by combining all different pieces into a single asset, such as Software code, Configuration, and infrastructure. However, this approach introduces some complexities and can become a bottleneck against one of the principal goals of IaC, agility.

AWS CDK, as an abstraction layer, offers more flexibility using code by overcoming the difficulties of structured cloud formation templating. However, this allowance of autonomy and flexibility can become an obstacle in the long term. Often, this becomes frustrating when applying some conventions at the enterprise level, when teams must change all stacks to respect some standard or convention. Also, facing changes in AWS services or deprecations are cases that need some extra effort and modification in all stacks which is theoretically not possible or takes a lot of time.

Configuration

When dealing with IaC, configurable software practices must be adopted to help have a central and easy-to-change stack. There are many ways to use configuration such as configuration files, ParameterStore, or CloudFormation Outputs. The choice of configuration depends on the nature of values, lifecycle, and dependencies.

Configuration File

A configuration file can be ideal for some internal and local configuration elements, the following snippet shows an example of a configuration file in Typescript, including some naming and static config values.

import { ContextVariables, EnvVariable } from "@type";

export type Config = {
  contextVariables: ContextVariables;
  org: { accounts: { accountName: string;}[];};
}

const defaultConfig: Config = {
  contextVariables: {
    context: `my-application`,
    stage: 'dev', 
    owner: 'operations',
    usage: 'EPHEMERAL',
  }
}

const getFinalConfig = (config: Partial<Config>): Config => {
  return {
    ...defaultConfig,
    contextVariables: {
      ...defaultConfig.contextVariables,
      ...config.contextVariables,
    },
    ...config
  }
}

export const getConfig = (stage: EnvVariable): Config => {
  switch (stage) {
    case 'test':
      return getFinalConfig({ contextVariables: { 
        ...defaultConfig.contextVariables, 
        stage: 'test', usage: 'PRODUCTION' } 
      });
    case 'prod':
      return getFinalConfig({ contextVariables: { 
        ...defaultConfig.contextVariables,
        stage: 'prod', usage: 'PRODUCTION' 
      }});
    case 'dev':
      return getFinalConfig({ contextVariables: { 
        ...defaultConfig.contextVariables, 
        stage: 'dev', usage: 'EPHEMERAL' 
      }});
    case 'sandbox':
      return getFinalConfig({ contextVariables: {
         ...defaultConfig.contextVariables,
         stage: 'sandbox', usage: 'POC' 
      }});
    default:
      return getFinalConfig({});
  }
};

The config can be later easily fetched in CDK app entry as below

const app = new cdk.App();
const environment = getEnv(app);
const config = getConfig(environment);

Parameter Store

Parameter Store is advantageous when using external configurations such as Application Load Balancer Listener Arn, VPC Id, etc. It is a good practice to use parameters for central configurations or unpredictable cross-stack ones.

Parameter Store as a solution can lead to complexities if:

Lack Of Conventional Naming for parameters.
Consumer stacks spread param fetch in different places.
Excessive use of Parameters

In the following snippet, the parameters fetch is done in the parent stack as passed to the nested stacks, either all nested stacks dont need all parameters.

export class AwsGatewayUsingLoadbalancerStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const albSecurityGroupId = StringParameter.fromStringParameterName(this, 'https-listener-securitygroupe-id', '/alb/securityGroupId').stringValue;
    const albHttpsListenerArn = StringParameter.fromStringParameterName(this, 'https-listener-arn',  '/alb/httpsListenerArn').stringValue;

    const securityGroup = SecurityGroup.fromSecurityGroupId(this, 'alb-security-group', albSecurityGroupId);
    const albListener = ApplicationListener.fromApplicationListenerAttributes(this, 'alb-listener', {
      listenerArn: albHttpsListenerArn,
      securityGroup: securityGroup,
    });

    const lambda = new NodejsFunction(this, 'example-lambda', {
      entry: join(process.cwd(), '/src/example.ts'),
      handler: 'handler',
      ...LambdaConfiguration
    });

    const target = new ApplicationTargetGroup(this, 'example-target-group', { targets: [ new LambdaTarget(lambda) ] });

    albListener.addTargetGroups('dosomething-target', {
      priority: 1,
      conditions: [
        ListenerCondition.hostHeaders(['myservice.example.com']),
        ListenerCondition.pathPatterns(['/v1/dosomthing']),
        ListenerCondition.httpRequestMethods([
          HttpMethod.POST,
          HttpMethod.OPTIONS,
        ]),
      ],
      targetGroups: [target],
    });
  }
}

The example fetches the parameters early and passes through when you need it. The advantage of this approach is that the parameters are side by side and in a single place, simplifying future changes and evolutions. another approach is the parameter is fetched once and shared, this means if you need multiple times the same parameter, the Api call to parameter store is done in a single place and only once.

Validation

As part of the Infrastructure as code, companies often apply conventions that help simplify governance, such as Available Stages ( dev, test, staging, prod, sandbox, etc. ) or Tagging ( stage, context, project, application, or domain).

Discovering noncompliant resources is one way of doing so, and this is a correct way of analyzing and finding noncompliant resources, but often, these discovered problems take a long time to fix and cause a lot of frustration.

A better approach is to think as an enabler, making the present and future life easy and trying to simplify the way teams must apply the regulations and conventional aspects.

Using CDK, a validator will be executed soon during Synthesizer execution. The following example validates the allowed stage configurations

export class WorkloadEnvValidator implements IValidation {
  constructor(private readonly variables: ContextVariables) {}

  public validate(): string[] {
    const errors: string[] = [];
    if(!(isEnvValid(this.variables.stage))) {
      errors.push(`Provided Stage value is not a valid environment. Must be one of: ${JSON.stringify(AvailableEnvs)}.`);
    }

    if(
      !['dev', 'sandbox'].includes(this.variables.stage) &&
      this.variables.usage !== 'PRODUCTION'
    ){
      errors.push(`Provided Stage value is not eligible to run ephemeral or experimental stacks.`);
    }

    return errors;
  }
}


// IsEnvValid allows Dev, Test, Prod and sandbox
//export const isEnvValid = (env: EnvVariable): env is EnvVariable =>
//  IsDevelopmentEnv(env) || 
//  IsTestingEnv(env) ||
//  IsProductionEnv(env) ||
//  IsSandboxEnv(env);

Running the synth using an unknown stage name will result in the following messages.

Aspects

We can use aspects to apply conventions and compliance-related tasks in an automated way, such as Tagging resources, Conventional Naming, etc. The following example shows a simple way of applying parameter naming.

export class ApplyParameterStoreNamingPolicyAspect implements IAspect {
  constructor(private readonly variables: ContextVariables) { }
  public visit(node: IConstruct): void {
    if (node instanceof CfnParameter) {
      const inspector = new TreeInspector();
      node.inspect(inspector);

      const name = inspector.attributes['aws:cdk:cloudformation:props']['name'].toString();
      if(name.startsWith(`/${this.variables.stage}/${this.variables.context}/`)) return;
        const cleanedName = name
            .replace(`${this.variables.stage}`, '')
            .replace(`${this.variables.context}`, '')
            .replace('//', '');
      node.addPropertyOverride('Name', `/${this.variables.stage}/${this.variables.context}${name}` );
      Annotations.of(node).addWarningV2(`${name}`, `Parameter Name should start with /${this.variables.stage}/${this.variables.context}, A managed fix is applied by renaming the parameter name but this can have consequences per usage, please apply the correct naming convention` );
    }
  }

This sample Aspect looks at parameter resources, and transforms the parameter name, and shows a warning in the terminal.

Centralization

There are cases where providing a framework or extensions can be useful, letting the teams use them when necessary, but applying compliance brings a lot of small pieces that can become error-prone to ask teams to apply explicitly or per need.

The best way of achieving the goal but also keeping the rate of change and effort as minimal as possible will be giving abstractions that help teams to achieve the goal with simplicity, this can be achieved with constructs but again creating constructs and applying them everywhere and in all services is not the best choice.

However, providing an abstraction of type Stack can be simple enough to propagate at the enterprise level. The example below shows how can be achieved.

import { ArnFormat, Aspects, Duration, NestedStack, NestedStackProps, Stack, StackProps, Tag } from "aws-cdk-lib";
import { Construct } from "constructs";
import { ApplyDestroyPolicyAspect, ApplyParameterStoreNamingPolicyAspect, ApplyTagsAspect } from "./aspects";
import { ContextVariablesValidator, WorkloadEnvValidator } from "./validators";
import { ContextVariables } from "../types/Context";
import { Rule, Schedule } from "aws-cdk-lib/aws-events";
import { LambdaFunction } from "aws-cdk-lib/aws-events-targets";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import { Runtime } from "aws-cdk-lib/aws-lambda";
import { join } from "path";
import { PolicyDocument, PolicyStatement, Role, ServicePrincipal } from "aws-cdk-lib/aws-iam";
export type EnforcedStackProps = StackProps & NestedStackProps & {
  contextVariables: ContextVariables;
}

export class EnforcedNestedStack extends NestedStack {
  protected readonly REGION: string;
  protected readonly ACCOUNT_ID: string;
  protected readonly ENV: string;
  protected readonly CONTEXT: string;
  protected readonly CONTEXT_VARIABLES: ContextVariables;

  constructor(scope: Construct, id: string, props: EnforcedStackProps) {
    super(scope, id, props);

    const { account: ACCOUNT_ID, region: REGION } = Stack.of(this);
    this.REGION = REGION;
    this.ACCOUNT_ID = ACCOUNT_ID;

    const { contextVariables: variables } = props;
    this.ENV = variables.stage;
    this.CONTEXT = variables.context;
    this.CONTEXT_VARIABLES = variables
  }
}
export class EnforcedStack extends Stack { 
  protected readonly REGION: string;
  protected readonly ACCOUNT_ID: string;
  protected readonly ENV: string;
  protected readonly CONTEXT: string;
  protected readonly CONTEXT_VARIABLES: ContextVariables;

  constructor(scope: Construct, id: string, props: EnforcedStackProps) {
    super(scope, id, props);

    const { account, region } = Stack.of(this);
    this.REGION = REGION;
    this.ACCOUNT_ID = ACCOUNT_ID;

    const { contextVariables } = props;
    this.ENV = contextVariables.stage;
    this.CONTEXT = contextVariables.context;
    this.CONTEXT_VARIABLES = contextVariables

    this.node.addValidation(new ContextVariablesValidator(contextVariables));
    this.node.addValidation(new WorkloadEnvValidator(contextVariables));
    Aspects.of(this).add(new AwsSolutionsChecks());

    if( contextVariables.usage !== 'PRODUCTION' ) 
      Aspects.of(this).add(new ApplyDestroyPolicyAspect());

    if( contextVariables.usage === 'EPHEMERAL' ) {

      const now = new Date(new Date().getTime() + 24 * 60 * 60 * 1000);

      const deleteFunction = new NodejsFunction(this, 'DeleteFunction', {
        handler: 'index.handler',
        runtime: Runtime.NODEJS_20_X,
        timeout: Duration.minutes(15),
        entry: join(process.cwd(), 'core/custom-resources', 'remove-stack.ts'),
        environment: {
          STACK_NAME: this.stackName,
        },
        role: new Role(this, 'DeleteFunctionRole', {
          assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
          inlinePolicies: {
            'CloudFormationPolicy': new PolicyDocument({
              statements: [
                new PolicyStatement({
                  actions: ['cloudformation:DeleteStack'],
                  resources: [
                    Stack.of(this).formatArn({
                      service: 'cloudformation',
                      resource: 'stack',
                      resourceName: `${this.stackName}/*`,
                      arnFormat: ArnFormat.SLASH_RESOURCE_NAME
                    }),
                  ]
                })
              ]
            })
          }
        })
      });

      new Rule(this, 'EphemeralRule', {
        schedule: Schedule.cron({
          minute: now.getMinutes().toString(),
          hour: now.getHours().toString(),
          day: now.getDate().toString(),
          month: now.getMonth().toString(),
          year:  now.getFullYear().toString(),
        }),
        targets: [ new LambdaFunction(deleteFunction)],
      });
    }

    Aspects.of(this).add(new ApplyTagsAspect({
      context: contextVariables.context,
      stage: contextVariables.stage,
      owner: contextVariables.owner,
      usage: contextVariables.usage,
    }));

    Aspects.of(this).add(new ApplyParameterStoreNamingPolicyAspect(contextVariables));

  }
}

The example applies all aspects and validations in a central place, also, as a specific detail, it uses one-time event bridge schedules to remove the ephemeral stacks after 24 hours.

Newsletter | Serverless Folks

Subscribe to Serverless Folks's newsletter.

serverlessfolks.com

Conclusion

Adopting IaC was a great evolution in how companies have been dealing with infrastructure, but as the rapidity of cloud infrastructure creation leads to a lot more amount of resources, this becomes important to enable teams to apply practices and help enterprises without becoming road-blockers or reducing the velocity of development teams.

AWS CDK is not only an object-oriented way of writing IaC but also provides many design patterns under the hood that can be extended to some requirements such as compliance, security, etc.

Lambda Code Execution Freeze/Thaw

Omid Eidivandi — Fri, 18 Oct 2024 15:50:44 +0000

AWS Lambda, a serverless computing service, enables code execution on demand while providing an isolated environment for each individual request. This design inherently ensures reliability, as any failure in one request does not affect others.

When working with Lambda, it's important to explore the intricacies of its execution environment and how it is managed. In this article, I will share my insights and observations, though some aspects may be open to debate.

Execution Environment

An execution environment is an isolated container (Micro-VM) that is launched on demand to handle incoming requests. Each environment processes one request at a time but can handle subsequent requests once the previous one is complete. If a new request arrives before the ongoing one finishes, an additional execution environment is created to manage it. The diagram below illustrates the lifecycle of an execution environment.

An Execution environment follows 3 phases: Initialization, Invocation, and shutdown

Init phase

The init includes initiating runtime, registering configured extensions, and downloading code packages which happen sequentially in respective order.

Invocation Phase

The Runtime, extensions, and function code will be invoked during the Invocation phase.

Shutdown Phase

The shutdown phase will shut down the runtime and send the shutdown signal to all extensions letting them clean up and finish the remaining work.

Reuse of Environment

An Execution Environment will be used for the subsequent requests, so the lambda will enter the shutdown phase if an execution environment does not receive any request for a period of time. The duration between the end of the invocation phase and the start of the shutdown phase is the idle duration when the Lambda service allows the reuse of that available environment.

AWS Lambda has a lot of interesting technical details. Lambda will not leave the execution environment running while there is no new demand, but the execution environment gets frozen and will be Thawed if it receives any new demand.

In the case of unintentional interruptions, the lambda will run the initialization as part of the next invocation, but this just seems to be a light init.

Code Execution

Lets first see how the code is executed for any request. When the first request is received, the lambda initiates the environment by running the top-level code. How the init phase behaves depends on the programming language used and how the code is written. Typically, when using NodeJs what happens during the initialization phase is like running the following command

> node index.js

The lambda service will then execute the function handler, when the execution is finished, the execution environment gets frozen. Thawing the execution environment i the tricky part. When lambda Freezes and the execution environment all background processes will be frozen and will be executed by thawing back the execution environment. But what about if the execution environment gets no more requests? The execution environment gets shut down and everything will be lost.

Freeze

When the execution environment gets frozen under the hood, the container gets into kinda hibernate state, when all resources get in a sleep state, like when a PC goes into hibernate mode, The address space of all running processes will be registered, allowing reconstruction of same state next.

Thaw

The thawing stage is part of runtime invoke when a container is reused as part of the invocation phase; the lambda service invokes the runtime, but this theoretically must be behind the Frozen background process reconstruction.

I could not find a response to what happens when the execution environment wakes up, but this must be when the container gets awake and not the runtime.

Try it out

The following example will create a lambda-based Api with two endpoints, one for awaited and another for non-awaited tasks to observe how lambda will behave in real time.

Both functions have the same code except the first use the await and the second non.

import { LambdaFunctionURLEvent, LambdaFunctionURLResult } from "aws-lambda";const delay = async (ms: number) => { return new Promise((resolve) => { setTimeout(resolve, ms); });}const Task = async (req: string, name: string, sleep: number) => { await delay(sleep); console.log(`${name} : `, req ); return { name: `${name}` };}export const handler = async (_event: LambdaFunctionURLEvent): Promise<LambdaFunctionURLResult> => { const resultA = await Task(_event.requestContext.requestId, "TaskA", 1000); const resultB = await Task(_event.requestContext.requestId, "TaskB", 2000); const resultC = await Task(_event.requestContext.requestId, "TaskC", 3000); const result = { resultA, resultB, resultC, }; return { statusCode: 200, body: JSON.stringify(result, null, 2), headers: { "Content-Type": "application/json", }, };}

Running the awaited function will give a response time of 6.XX seconds accumulating 1000, 2000, and 3000 milliseconds.

The non-awaited function has the same code but calls TaskB and TaskC without waiting.

const resultA = await Task(_event.requestContext.requestId, "TaskA", 1000);const resultB = Task(_event.requestContext.requestId, "TaskB", 2000);const resultC = Task(_event.requestContext.requestId, "TaskC", 3000);

Running the first request gives the following logs, showing that only TaskA is terminated and the log is present.

Running a second request results in the logs as below, the previous execution remaining tasks are executed as part of subsequent invocations.

But the interesting part is how long they took to log. The TaskB and TaskC are executed at the same time and ended instantly. The following image shows the response time as 1105 ms, which is normal for new invocation TaskA.

Looking at the Non-Awaited code previously, the TaskB and TaskC must take around 5000 ms together that is not the case. Looking at Billing Duration the Billed duration corresponds TaskA execution.

Real scenario

To trust better the hypothesis ( i did not ) and validate that there will be no tricky side effect, we gonna send a message to an SQS queue just to prove the idea behind the observations is real.

By running the first request there will be nothing fancy except a new message in the queue for TaskA.

Now, lets run another request and see what happens. Here, TaskB and TaskC just got included in the execution, and the new messages are present in the queue. the top 1 in the following screenshot is the first invocation message, and the three others are TaskB/TaskC of the first invocation + New Invocation TaskA.

In the above examples, the logs ingested were the logs after treatment. I was curious to see if I could observe how those function calls happen so i kept it simple by adding logs for the start and end of each treatment.

Here how Task method looks like

export const Task = async (req: string, name: string, sleep: number) => { console.log(`Starting ${name} : `, req ); await delay(sleep); console.log(`Waking up ${name} : `, req ); await client.send(new SendMessageCommand({ QueueUrl: process.env.QUEUE_URL, MessageBody: JSON.stringify({ req, name }), })); console.log(`End ${name} : `, req ); return { name: `${name}` };}

The Task method has three levels of Logs indicating the Starting, WakingUp, and End. According to the following logs, the first represents an execution that the frozen tasks are executed before the current execution, and the second one, TaskB woke up at the same time as the current invocation start ( CloudWatch log ordering is defaulted by time ). based on these observations frozen tasks are not executed at the function handler invoke phase but at runtime invoke apparently.

These observations can prove that the frozen processes will be run as soon as the execution environment is thawed, and this seems like a kind of decoupling ( this is only a hypothesis ) to prove the coupling or decoupling running some failures for TaskB and TaskC will illustrate better the internal state.

Simulating Failures

The Task Method will be changed to cover failing explicitly per Task name as following code snippet.

export const Task = async ( req: string, name: string, sleep: number, extendedProcess?: Function) => { console.log(`Starting ${name} : `, req ); await delay(sleep); console.log(`Waking up ${name} : `, req ); if( extendedProcess ){ extendedProcess(); } await client.send(new SendMessageCommand({ QueueUrl: process.env.QUEUE_URL, MessageBody: JSON.stringify({ req, name }), })); console.log(`End ${name} : `, req ); return { name: `${name}` };}

The extendedProcess is passed in TaskB initialization call as below

const extendedProcess = (name: string, req: string) => { console.log(`Failing ${name} : `, req ); throw new Error('TaskB Failed');}export const handler = async (_event: LambdaFunctionURLEvent): Promise<LambdaFunctionURLResult> => { ... const resultB = Task( _event.requestContext.requestId, "TaskB", 2000, extendedProcess("TaskB", _event.requestContext.requestId)); ...}

The first request will behave as before, but during the second execution, the previously frozen TaskB will result in the current execution interruption by throwing an UnresolvedPromiseRejection error. This proves that the Freeze/Thaw of incompleted tasks can be dangerous and breaks the AWS Lambda design principle of having isolated event-based processing. this approves a level of coupling that can become dangerous without careful implementation.

Tracking state

💡

This is just a game play not a way to implement production ready solutions

This experimentation pushed me to think about how the lambda can behave like a stateful container and act based on the state. To experiment with state tracking, a use case can be a task that will provide some state in the execution environment, which can be done using a variable outside the handler, but this time, I want to try not to keep only the state but defer the actions and see if possible.

How the idea behaves can be demonstrated by the following sequence diagram

The idea is that the un-waited task will check the execution environment state and and modify it. What was achived in this test :

Accumulting State in a Dictionary outside handler
Pushing accumulated items to an SQS when count reaches 10 item

But what about undesired situation , during the time i played i had some sort of timeout occasionally, while deep diving i discovered the dictionary state got empty after a timeout as per following screenshot from cloudwatch logs.

Conclusion

While this is cool to try and fail, this article just was a game around how lambda execution environment behaves, and the behavior seems like an hibernate , when again awaked, the process will come back and resuming, did you ever tried a hibernate while coping a huge folder from one drive to other ? this is the same behavior.

The final note is, a controlled and imperative programming model is many times more efficient that behavioral programming, actually for this case a better approach is using the lambda layer to push the logs but again this was a fun and i thought it is worth to share with the community.

Enjoy reading

You Are Not Saved By IaC

Omid Eidivandi — Sat, 28 Sep 2024 11:58:40 +0000

Technology exists to simplify human challenges, and as tech professionals, we must also leverage it to solve our own problems. One common area we deal with daily is Infrastructure as Code (IaC), raising frequent questions such as which tool is betterAWS CDK, CloudFormation, Serverless Framework, or Terraform?

However, we often overlook the foundational principles of IaC, like recovery, fast deployment, resiliency, and minimizing time-to-market (TTM). For instance, if you decide to implement a multi-regional failover, can it be deployed effortlessly? How quickly can you recover if your production environment, region, or accounts go down?

Common Roadblocks

To navigate the complexities of IaC, vigilance and discipline are essential. Lets explore some common roadblocks and how to address them effectively:

Historical manual interventions
Lack of configurable and parameterized code
Lost secrets that cannot be restored
Hard dependencies between stacks
Circular dependencies

Human Actions

Human intervention is a regular part of our jobsquick fixes to temporary issues often lead to future automation. Unfortunately, these "notes for later" sometimes get forgotten, turning into major pain points down the road. Identifying these recurring manual actions is key to saving time, effort, and frustration in the future.

Recommendations:

Use tags to identify automated resources.
Regularly explore untagged resources to detect those not yet automated.
Review generated IaC templates to find missing tags.
Foster a tech-driven culture within your team.

Configuration Shortcomings

One frequent issue in IaC is hardcoding variables in Stacks or Nested Stacks, which can complicate configuration management. Whether it's a queue name, topic name, or HTTP endpoint, manually searching through different IaC tools like CloudFormation or AWS CDK can slow you down. Centralizing all dependencies simplifies future changeslike shifting from "me.mycompany.com" to "me.mycompany.org"by allowing you to quickly locate and update configurations.

Losing Secrets

Managing secrets securely is crucial. While storing sensitive information like API keys or credentials in a secret manager is helpful, what happens if your account gets lost? The solution is to maintain a backup of all secrets outside the software environment, ideally in a dedicated vaultthis is more of an organizational best practice.

Managing Dependencies

Dependencies in IaC can be categorized into three levels:

Light Dependencies: Passed to the environment variables (e.g., Lambda), these wont break your deployment but could affect testing and runtime.
Soft Dependencies: Tied to infrastructure services but manageablelike subscribing to an SNS topic, though permission issues may arise from unautomated historical actions.
Hard Dependencies: These will prevent deployment if not properly handled. For example, an EventBridge rule may require an EventBus that isnt yet deployed. The key here is identifying priority stacks and documenting these relationships, often using dependency graphs or architecture diagrams.

Circular Dependencies

Over time, as requirements evolve, stacks can develop circular dependencies. Imagine planning a production release only to find that it fails due to a circular dependency between two stacks. For instance, Stack A may require a CloudFront distribution that needs an upstream domain name for CORS, but the record set is managed in another stackleading to a deadlock.

To avoid such issues, actively manage and mitigate circular dependencies. Divide stacks if needed or apply predictable naming conventions. For example, using "products.mycompany.com" instead of introducing direct dependencies between stacks can eliminate such problems.

Conclusion

By proactively addressing these common challenges, we can build more resilient, efficient, and scalable infrastructure, reducing downtime and increasing the speed of recovery when issues arise.

Enterprise Level Micro-frontend on AWS

Omid Eidivandi — Wed, 28 Aug 2024 23:48:04 +0000

The Primer
Boundary Definition Challenges
Rendering
Orchestration
Kernel / Shell
Shell Challenges
What we gonna build
Running the Example
Conclusion

Micro-frontends, a center of interest for some years now, have evolved rapidly and have become a great choice for achieving independent, autonomous, and well-defined teams and services that improve the velocity and agility of applications.

This article delves into different concepts and details regarding adopting micro-frontends, giving more clarity and vision about tradeoffs and the under-the-hood parts. To learn more about the impacts and goals of a MicroFrontend design, I highly recommend the Building Micro-Frontends book by Luca Mezzalira.

Building Micro-Frontends[Book]

What's the answer to today's increasingly complex web applications? Micro-frontends. Inspired by the microservices model, this approach lets you break interfaces into separate features managed by different teams of developers. … - Selection from Building Micro-Frontends [Book]

oreilly.com

The conception at the root of the proposal applies to achieve the above ideas, but when it comes to design and implementation, this leads to a lot of discussions and details to take into account and consider. choosing the right approach depends on the tradeoffs and the priorities that the business follows.

Micro-frontends are not far from the microservice design approach, as they apply the same goals fundamentally, but at the same time bring exact or partial complexities and challenges the microservices introduce.

The Primer

At high level, a micro-frontend architecture leads to answer achieving independent and autonomous collaboration to satisfy a final result asset within a distributed system.

In the above schema, the 3 services are deployed, maintained, and scale independently and will be composed together as a final result asset.

This is a simple, small and straightforward example but at the higher scale we need to think about:

Templating
Discovery
Routing
Composition

Boundary Definition Challenges

One principal challenge when designing Micro-Frontends is how to evaluate the boundaries of each MFE. Often having too many small distributed micro frontends seems to bring more flexibility and reusability opportunities, so an MFE can be composed in as many places as the client app needs. While having MFEs with small defined contexts brings the advantage of having self-managed and autonomous MFEs, but at the same time, adds more complexity at the Discovery & Routing part, also adds the communication chattiness into MFEs by calling the downstream services multiple times while the result asset will be in the same context. In the other side, having too big MFEs can improve some challenges small MFEs brings but gives less flexibility and opportunity of reusability, and will introduce the rendering performance issues, complexities and performance degradation at client side.

Having MFEs with a logical size based on the context is a key to success, at the same time it brings more logic at MFE side but avoid adding complexity at other parts of the system.

Some factors to consider while defining a micro-frontend context

Performance
Lifecycle
Co-existence
Reusability
Ownership
Clarity & Vision
Team Topology possibilities

Horizontal vs Vertical Splitting

The splitting refers to the fact of composing the MFEs on the horizontal or vertical axis.

In horizontal splitting the composition introduces a coupling like having a web app page dedicated to a micro-frontend in this case the components inside that given page have less opportunity to be reused in other contexts and pages. an example will be a widget that renders the active products for a given customer, while it can be shown in different pages like customer detail page or in the own customers admin page. Horizontal splitting leads to duplicated components, this seems ok but can be tricky when the number of reusable components raises or the standardisation and governance becomes a concern like applying design systems, complex business logic, etc.

In a vertical splitting the chance of reusability increases and a component has the chance of being reused in any other contexts and result assets. but it adds some layer of duplicated communication and data fetching but also at some points introduces useless decomposition.

When it comes to splitting a better approach is to pragmatically define the right boundary but also composition layer. often teams stands for applying MFE composition at a single layer being Web Server, Edge or shell. but the better approach seems to define right decomposition at server side while giving flexibility to future reuse but also right composition in shell ( kernel ).

In the above diagram having product catalog and insights at the same layer ( server side ) helps to reduce the unnecessary downstream service calls but adds some challenges such as cache configuration. So putting two Components in the same MFE must be considered when those components follow the same lifecycle.

This is important to think about different decision axis to adopt the right approach based on requirements and tradeoffs.

Rendering

The choice of rendering approach depends on functional requirements and technical possibilities. My journey began with front-end development using VBScript, where code ran on the client side. However, limitations such as database connectivity and security concerns led to a shift towards server-side development with ASP.NET, where interactions and rendering were handled on the server. As JavaScript technologies advanced, particularly with the introduction of Ajax, the drawbacks of server-side latency became apparent. This prompted a shift back to client-side rendering, allowing for partial or full application rendering on the client side, with DOM manipulation and updates occurring after API responses were received. This approach was well-suited to the era of limited server resources and networking constraints.

However, as server capabilities improved, particularly with the advent of cloud computing and rapid scaling, it became advantageous to offload processing back to the server. Today, server-side rendering (SSR) is a crucial front-end design decision, thanks to its ability to respond in milliseconds and scale efficiently. However, it's not always the optimal choice and should be considered within the context of the overall architecture.

Rendering choices are critical and involve several factors, including:

User Experience
Performance
Cost
Search Engine Optimization (SEO)

While other considerations like Separation of Concerns (SoC) and polyglot ecosystems are important, the ones listed above are the primary focus here. In the context of Micro-Frontends, various rendering strategies can be employed, such as Client-Side Rendering (CSR), Edge-Side Rendering (ESR), Server-Side Rendering (SSR), and Server-Side Generation (SSG), which is a variation of SSR.

Client Side Rendering

CSR applies rendering of HTML content in the browser using javascript dynamically, But with CSR a preliminary content is received from the server containing the javascript and CSS, further the execution of those javascript in client side will lead to content rendering by calling the required api or backend services. Before the script execution, the content will appear and will not be interactive.

To see more how the CSR and SSR behave this video by Scott Hanselman is a nice one.

Server Side Rendering

SSR is all about rendering the HTML content at server side before returning to the client (browser), The server request can be done at the page load or behind any action with the ability to trigger a request. The server response contains the content but also all necessary scripts that lead to have interactive content on the client side.

SSR can be applied in a variety of ways but listening to experts like Luca Mezzalira helps to capture enough and this video is a rich and detailed one

Server Side Generation

SSG is the technique of generating HTML assets in server side and let the client use the pre rendered assets, This is really close to the fact of using static assets but in SSG the asset content is HTML. The SSG is great if the number of required assets are limited but this can be tricky when dealing with millions of items. when using SSG it s important to consider how long and till when that assets is really required and estimate the resulting costs.

Edge Side Rendering

ESR helps to tackle some SSR challenges such as latency and performance, The ESR helps to respond to client request at Edge and benefit from the global Point of presence close to the user, aside of latency improvements by removing the full load on origin, ESR also helps to remove the load of processing on client side being a browser, mobile app or etc.. to achieve a more device friendly approach.

Orchestration

The Micro-Frontend orchestration is a pattern that is used to apply the composition of different micro frontends and makes a final result that represents a page or a representable and meaningful part of a UI. The orchestration can be done in different levels being Client side, Server side or Edge side. and also an intelligent mix of them can be applied.

This article fundamentally focuses on Server Side Rendering and demonstrates some challenges the teams face while designing SSR.

Kernel / Shell

As discussed earlier, while Micro Frontends (MFE) offer significant benefits to the overall system, they can also introduce cross-domain challenges and coordination issues among teams as they work to integrate distributed components into a cohesive solution. To address these complexities, a shell (or kernel) adds a layer of abstraction between the client and the distributed services, streamlining communication and simplifying the composition process, ultimately leading to a more flexible design.

Implementing a shell offers the advantage of simplifying the previously mentioned requirements. It facilitates easier discovery of services and their versions, resolves template details, routes requests to the appropriate service based on the resolved template, and ultimately handles the composition.

A shell (or kernel) is particularly valuable when managing various types of renderingwhether client-side, edge-side, server-side, or a combination of these approaches.

A shell can respond to the following requirements if applied:

Service Discovery
Template Discovery and rendering
Service deployment strategy ( Canary, Rolling, All-At-Once )
Error handling and fallbacks
Instrumenting for Observability

Templating

A template serves as a predefined framework that outlines the structure of the result set required by a client application, whether it's a browser or a mobile app. Consequently, the templating module must identify the appropriate template based on the request information received from the client. For instance, a template represents a page that includes various components such as scripts, meta tags, CSS references, and more.

At the first glance maybe templating is not really interesting approach but it helps to resolve a lot of frontend cross cutting concerns like shared scripts, design systems , and etc, without changing and deploying all MFEs or client app.

Discovery & Routing

Discovery is the process of locating the appropriate services based on identifiers, versions, and other factors. It provides a more granular approach to integrating services and the fundamental aspects of a distributed design with reduced complexity.

A high-level implementation of discovery would look like this:

The routing module is responsible for directing external and public service requests to the appropriate internal or private distributed services. An effective routing module takes into account the deployment strategy and the type of composition, whether server-side, client-side, or edge-side.

The diagram illustrates the routing process when server-side composition is used.

Routing can be handled either by the Web Server or the Shell. When the Shell manages routing, it performs lightweight mapping based on the incoming request and the associated micro-frontend. The Shell directs traffic to the appropriate service and conducts basic contextual and generic checks. More detailed routing, which requires domain-specific knowledge, is managed within the micro-frontend itself. This includes scenarios such as applying a specific version for an individual customer or a particular list of products.

Shell Challenges

When designing a shell the most important thing to consider is to design a shell agnostic of business details, often the teams start light and brings incrementally the complexity into shell for a single reason, and this is the shell is already there and the easiest way for change. But at longterm this will lead to have a higher risk of changes and distributed knowledge. why this is risky ? a shell can be central and need to be evolved by a wide range of teams, this becomes tricky in long run.

What we gonna build

The following diagram shows the application we are building for this article, we will build a simple react application providing a client application that interact with server side resources via a shell ( Kernel ) that provides the discovery & routing, and templating and a light Api to help the micro-frontends communicate together. Each micro-frontend provides the html in response which will be shown in our Web App.

Source Code

The example related to this part of series is Part-01 branch in following Github repository.

XaaXaaX / aws-microfrontend-ssr

Local development

To run the project locally first run the following command that uses the express to mount a local server from all handlers discovered based on configured paths

npm run local:start

The service configs are as below

const configs: ConfigSource[] = [
    { path: "accounts/signin/v1/", source: '@account/signin/index.ts' },
    { path: "bookmarks/v1/", source: '@bookmark/bookmarks/index.ts' },
    { path: "products/catalog/v1/", source: '@products/src/catalog/index.ts' },
];

The project needs to have the following section as part of tsconfig.json to dynamically import when the alias paths are used

"ts-node": {
    "require": ["tsconfig-paths/register"]
  },
  "require": [
      "ts-node/register/transpile-only"
  ],

Site

Run by default under the http://localhost:3000 it interacts with remote MFEs via this default but for local testing can be changed

after running local-lambda-invoke.ts a local server will be available server use…

View on GitHub

Bookmarks

The Bookmarks Micro frontend stack has the responsibility of returning the bookmarks for a given userid, the bookmarks MFE validates the presence of UserId parameter and returns the results for the corresponding user.

In bookmarks service, the bookmarks list MFE is vertically sliced, but all downstream service calls can be managed if required.

The bookmarks list mfe returns all bookmarks for given userid and returns an 4xx error if the userid not provided. also it applies the filtering of user bookmarks per product reference and name if applied.

The caching is applied per query string parameters including userid, ref, name. the following snippet represents the CDK example for bookmark service and related configuration.

The bookmarks distribution owns the caching requirements of bookmarks service, this is related to the way that each service owns and master its proper requirements.

this.Distribution = new Distribution(this, 'BookmarkDistribution', {
    ...
    defaultBehavior: {
        origin: new FunctionUrlOrigin(props.DefaultOriginListBookmarksFunctionUrl, {
            connectionAttempts: 3,
            connectionTimeout: Duration.seconds(1),
            keepaliveTimeout: Duration.seconds(5),
        }),
        allowedMethods: AllowedMethods.ALLOW_ALL,
        cachedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS,
        cachePolicy: new CachePolicy(this, 'BookmarksCachePolicy', {
            queryStringBehavior: CacheQueryStringBehavior.allowList(
                'userid',
                'ref',
                'name'
            ),
            defaultTtl: Duration.hours(1),
            minTtl: Duration.hours(0),
            maxTtl: Duration.hours(24),
        }),
        viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
        originRequestPolicy: OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER,
    },
});

const cfCfnDist = this.Distribution.node.defaultChild as CfnDistribution;

const bookmarksOriginAccessControl = new CfnOriginAccessControl(this, 'LambdaUrlOAC', {
    originAccessControlConfig: {
        name: `Bookmarks-Lambda-OAC`,
        originAccessControlOriginType: 'lambda',
        signingBehavior: 'no-override',
        signingProtocol: 'sigv4',
    }
});

cfCfnDist.addPropertyOverride(
    'DistributionConfig.Origins.0.OriginAccessControlId',
    bookmarksOriginAccessControl.getAtt('Id')
);

The cloudfront and function url integration is done using OAC Signature V4, the function url grants the permissions to let the calls only be authorized only from cloudfront.

this.FunctionUrl.grantInvokeUrl(new ServicePrincipal('cloudfront.amazonaws.com', {
     conditions: {
        ArnLike: {
            'aws:SourceArn': `arn:aws:cloudfront::${account}:distribution/XXXXXXXXX`,
        },
        StringEquals: { 'aws:SourceAccount': account},
      }
}));

Products

The Product service provides two distinct MFEs being catalog and Product Details Page ( PDP ), those are independent and isolated ones.

The product service follows same implementation and application architecture principles as bookmarks so vertically sliced and optimized for potential reusability needs. Here, the important note is possibility of sharing the repository layer, this is not an interest from programming perspective but more communication, as this let us to simply have multiple MFEs and apply composition for different components inside one MFE and reducing the number of network calls or Database calls.

The products service follows the same principals as bookmarks in terms of caching, security and uses also lambda function url for api invocations.

Web Application

The web application consists of Two pages , home page and product details page, the home page serves as a landing for bookmarks , and catalog micro frontends, while the PDP hosts only the product details micro frontend.

The Web app home page includes two MFEs , Product Catalog and bookmarks list, but this is a different case from the previous interests we had when optimizing to reduce network calls. The interest here applies more on organizational and bounded context.

Each micro-frontend is owned and runs by different team
Each one has its proper context so database or downstream service calls
There is no domain context based relation between them.

In this diagram the react app calls both MFEs in parallel and use the received results as html assets that live side by side. The website build assets will be uploaded on a dedicated bucket that will be as the default behavior of web app cloudfront. The web app cloudfront applies the required caching for static web site assets, but for dynamic routes as the ones related to home page calls, bookmarks and catalog, and product details page applies no caching but only forward the requests to the downstream MFE servies.

Bundling

To bundle the website app, the build is done using react-scripts build command, the bundle results will be deployed to the web app s3 bucket using CDK BucketDeployment construct.


new BucketDeployment(this, 'BucketDeployment', {
   sources: [ Source.asset(join(process.cwd(), '/front-app/website/build')) ],
   cacheControl: [CacheControl.fromString('max-age=1800,must-revalidate')],
   destinationBucket: frontStack.Bucket,
   distribution: ditribution.Distribution,
   distributionPaths: ['/*'],
});

The use of BucketDeployment is for sake of demonstration but in real projects the deployment process must be under a dedicated cicd pipeline with multiple stages including CI ( linting, testing, analysis, etc.) and CD.

Caching

The cloudfront behavior for the dynamic content will use a dedicated CachePolicy and OriginAcessPolicy applying no caching but forwarding all request QueryString parameters.

const dynamicContentCachePolicy = new CachePolicy(this, 'DynamicContentCachePolicy', {
    headerBehavior: CacheHeaderBehavior.none(),
    cookieBehavior: CacheHeaderBehavior.none(),
    queryStringBehavior: CacheHeaderBehavior.none(),
    defaultTtl: Duration.seconds(0),
    minTtl: Duration.seconds(0),
});

const dynamicContentOriginRequestPolicy = new OriginRequestPolicy(this, 'DynamicContentOriginRequestPolicy', {
    queryStringBehavior: OriginRequestQueryStringBehavior.all(),
    headerBehavior: OriginRequestHeaderBehavior.none(),
    cookieBehavior: OriginRequestCookieBehavior.none()
});

this.Distribution.addBehavior('api/v1/bookmarks/*', new HttpOrigin(props.BookmarkServiceDomainName), {
    allowedMethods: AllowedMethods.ALLOW_ALL,
    cachePolicy: dynamicContentCachePolicy,
    viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
    originRequestPolicy: dynamicContentOriginRequestPolicy
});

Rendering

The underlying code for the example website will be a simple react app, using useEffects react hook to fetch the data on page load.

useEffect(() => {
    async function RenderMFEs() {
      const promiseProcessSuccess = 'fulfilled';
      await Promise.allSettled([
        fetchMfe(Mfes.BOOKMARKS_LIST),
        fetchMfe(Mfes.PRODUCT_CATALOG),
      ]).then((results) => {
        results.forEach((result) => {
          if (result.status === 'rejected') console.error('HP error :', result.reason) });
        if (results[0].status === promiseProcessSuccess) setBookmarks(results[0].value);
        if (results[1].status === promiseProcessSuccess) setCatalog(results[1].value);

      });
    };

    if (!bookmarks || !catalog) 
      RenderMFEs();

  }, [bookmarks, catalog]);

The following code demonstrates the FetchMfe() function, which is called in useEffect react hook.


export const enum Mfes {
  'PRODUCT_DETAILS',
  'BOOKMARKS_LIST',
  'PRODUCT_CATALOG',
}
export const Paths: Record<Mfes, string> = {
  [Mfes.PRODUCT_DETAILS]: '/api/v1/products/details/',
  [Mfes.PRODUCT_CATALOG]: '/api/v1/products/catalog/',
  [Mfes.BOOKMARKS_LIST]: '/api/v1/bookmarks/',
}

export const fetchMfe = async (MFE: Mfes, host?: string): Promise<string>  => {

  const hostDomain = `${window.location.protocol}//${window.location.host}/`;

  let urlPath = '';
  urlPath = urlPath.concat(Paths[MFE]);

  const url = new URL(urlPath, hostDomain);

  const queryParameters = new URLSearchParams(window.location.search);
  if(queryParameters) url.search = queryParameters.toString();

  const res = await fetch(`${url.href}`);
  return await res.text();
}

Redirection

As per requirement the redirection of certain URLs are important for SEO and brand trustworthy on the public, this is the case when certain links are no more valuable or has no corresponding result ( ex. when a product get out of stock or a url is deleted permanently).

The following CDK shows how to use a CloudFront Function to apply simple redirection for specific paths. The CloudFront functions use a lightweight version of Javascript. This way we can easily apply the redirection on top of specific urls under a distribution behavior and let the crawler to accumulate the old url score and brings them to the new redirected url ( this is how google indexation works when the permanent redirection is applied)

Thanks to David Behroozi sharing the great and cost effective solution for redirection.

This sections use the mentioned solution for our dedicated purpose.

defaultBehavior: {
   allowedMethods: AllowedMethods.ALLOW_GET_HEAD,
   origin: webbucketOrigin,
   cachedMethods: AllowedMethods.ALLOW_GET_HEAD,
   cachePolicy: WebCachePolicy,
   viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
   functionAssociations: [
        {
           function: new Function(scope, `ProductRedirectFunctionViewerResponse`, {
                code: FunctionCode.fromFile({ filePath: `front-app/src/url-redirect.js`}),
                runtime: FunctionRuntime.JS_2_0 
           }),
           eventType: FunctionEventType.VIEWER_RESPONSE,
        },
   ]}

The CloudFront function will be triggered at viewer response CloudFront event source trigger stage and applies a 301 permanent redirection.

The solution uses the s3 user-defined metadata to register the redirection target link.

// @ts-ignore
function handler(event) {
  console.log(JSON.stringify(event, null, 2));
  const response = event.response,
        headers = response.headers,
        request = event.request;

  const header = 'x-amz-meta-location';

  if ( 
    'GET' == request.method &&  
    200 == response.statusCode && 
    headers[header] && 
    headers[header].value
  ) {
      headers.location = { value: headers[header].value };
      return {
        statusCode: 301,
        statusDescription: 'Moved Permanently',
        headers,
      };
  }
  return response;
}

The example uses BucketDeployment construct to upload a single file with corresponding metadata.

new BucketDeployment(this, 'RedirectionDeployment', {
   sources: [ Source.asset(join(process.cwd(), '/front-app/src/redirection-files')) ],
   metadata: {
      'location': `https://${ditribution.Distribution.distributionDomainName}/?category=ON_SOLD`,
   },
   destinationBucket: frontStack.Bucket,
   prune: false,
});

In this example all files in redirection-files folder will be uploaded with a location to the s3 bucket. but in real projects it can be any dedicated back-office generating files with corresponding metadata.

Shell

The shell is responsible for composing a template from different independent micro-frontends and returning the final result asset to the client app. The shell also applies some cross cutting concerns such as Authentication/Authorization, Graceful Degradation and Logging while acceding server side resources.

One of the most popular responsibilities of a shell is service (MFEs) discovery, a service discovery considers also the deployment strategy if applies, such as wighted , canary or Blue/Green.

A shell often impose an standard such as following example that can be integrated in a template as demonstrated below

<MfeTag
   src='...'
   error-handling='fallback'
   fallback='...'
   options='cors,auth,apikey,tls'
   timeout=1000
   passthrough='cookies=[...],query=[...],headers=[...]'
   strategy='canary'
/>

The above example represent a a simple custom HTML tag indicating under which constraints the MFEs communication shall be done.

in this example the shell will send a http call to the backend

Attribute	Description	Remarks
Src	The path to the MFE	This is often over HTTPS protocol but can be any other protocol such as TCP, WSS direct Lambda Request/Response invocation, StepFunctions sync execution, etc.
Erro-Handling	Indicates how will handle the overall template behavior in case of failures	The possible values are: Fail, Fallback, Degradation
Fallback	The fallback url to call if the src will be unavailable	This is often over HTTPS protocol but can be any other protocol such as TCP
Options	The options indicates standard cross cutting concerns related to this call	Possible values are: ApiKey, Authorization, Cors, mTLS, SignatureV4
Timeout	The MFE corresponding max waiting time for a response	This is in milliseconds
Passthrough	How the shell transfers the cookies , query string or headers to the MFE, if not present all parameters will be transited to the MFE	This can be customised based on the enterprise requirements
Strategy	This indicates the how the shell must consider the deployment strategy, Possible values are canary, bluegreen, weighted	The extra details per strategy will be fetched from discovery service , for example for canary the shell fetch the corresponding canary strategy like 10PercentPer5Minutes, To achieve an effective strategy the shell must be able to track state ( stateful )

In the next part of this series we will deep dive in the implementation details and design tradeoffs while considering a shell as a candidate to be user or not. The next part will focus on how to achieve a valuable shell and when consider applying a shell or adopt a simplified approach with less cognitive load but with a distributed responsibility at MFE sides per servie or Context.

Running the Example

To run the example locally, The backend MFEs must be ran on a local host server to let the local react website be functional. We use a simple way of running the typescript services locally over localhost( port 4242) to achieve local distributed MFEs and let the website communicate over localhost with MFEs.

The provided local server script is a simplified version of the script i used previously to run the backend lambda services with a minimum overhead and dependencies. A thanks to Zied Ben Tahar for giving the indications and helps to make this functional while we worked together at Aviv.

First, Lets deep a bit to see how this works. The script simply loop through an array of entrypoint and invoke the configured function which is exported.

const handleRoute = async (lambdaEntry: { entryPoint: string, handlerFn?: string, action?: (...args: any) => void }, req: Request, res: Response ) => {
   const module = await import(lambdaEntry.entryPoint);
   const lambdaFunctionHandler = module[lambdaEntry.handlerFn ?? "handler"];
   const result = await lambdaFunctionHandler(
       generateEvent(req), 
       createLambdaContextObjectFromContextPayload(req.body.context)
   );
   return res.send(result.body).end();
}

The entrypoints are the ts or tsx files, imported using dynamic import.

also the script uses the express to setup a local server using some defined middleware to use and the express router and provide all http verbs for each entrypoint.

const app = express();
app.use(cors());
app.use(express.json());
app.use(express.text());
app.use(express.raw());
app.use(express.urlencoded({ extended: true }));
const router = express.Router();

lambdasEntrypoints.forEach((lambdaEntry) => {
  router.all(`/${lambdaEntry.endpoint}/`, async (req: Request, res: Response) => {
    return handleRoute(lambdaEntry, req, res);
  });
});

The Entrypoints configs are configured as an array of entrypoints and are fetched using the glob package. The paths correspond to the corresponding MFEs real origin behavior to avoid changing the website code just for local testing.

export type ConfigSource = { path: string, source: string, handlerFn?: string, action?: (...args: any) => void };
const configs: ConfigSource[] = [
    { path: "api/v1/bookmarks/", source: 'micro-fronends/bookmarks/src/handlers/list/index.ts' },
    { path: "api/v1/products/catalog/", source: 'micro-fronends/products/src/handlers/catalog/index.ts' },
    { path: "api/v1/products/details/", source: 'micro-fronends/products/src/handlers/details/index.ts', action: (req: Request, res: Response) => { console.log(req); res.writeHead(302, {Location: `/api/v1/products/catalog/v1/?category=ON_SOLD`}).end();} },
];

export const lambdasEntrypoints = globSync(configs.map(src => src.source ?? './')  , { 
    ignore: [
        "**/**/node_modules/**",
        "**/**/*..test.ts",
        "**/**/*..spec.ts",
    ] }).map((entry: string) => {
        const config = configs.find(c => c.source.includes(entry));
        const entryPoint = join(process.cwd(), entry.split(path.sep).join(path.posix.sep)),
              lambdaName = entry
                .split(path.sep)
                .slice(-1)[0]
                .replace(".(ts|js)", ""),
              endpoint = config?.path,
              handlerFn = config?.handlerFn,
              action = config?.action;

        return { entryPoint, lambdaName, endpoint, handlerFn, action };
    });

The example local MFEs can be started using following comand

$ npm run local:start
------------ 
[Local λ debugger]: Local lambda invoke debug server is running at http://localhost:4242
[Local λ debugger]: Discovered 3 lambdas entrypoints
  [λ endpoint]: api/v1/bookmarks/
    [exported functions]: handler
  [λ endpoint]: api/v1/products/catalog/
    [exported functions]: handler
  [λ endpoint]: api/v1/products/details/
    [exported functions]: handler

The website can be ran using following command

$ npm run local:website

The package.json local:website script passes the local url using env variable and the Shared FetchMFE function look at this environment variable to decide where to look, principal host or the local.

{
   "local:website": "REACT_APP_LOCAL_URL=http://localhost:4242 npm run start --prefix front-app/website"
}

Addressing the navigation to [http://localhost:3000/?userid=HJ-HnhYul\_sm](http://localhost:3000/?userid=HJ-HnhYul_sm) url will fetch the user bookmarks and also the product catalog page as below

By adding the product reference as a query parameter the bookmarks and catalog will show only the corresponding references ( example http://localhost:3000/?userid=HJ-HnhYul_sm&ref=REF_2 )

The user auth will be as part of next article with shell implementation. the actual system pass all query string params to all the MFEs without considering the requirements of each MFE separately.

Conclusion

Applying a well architected Micro Frontend approach is highly similar to the micro services, it starts with curiosity, enthusiasm and rigor but can become complex and introduce the obstacles against the approach primer et goals.

Simplicity is a principal to success but putting the right part in the right place and defining the responsibilities and boundaries are the crucial things to achieve a long term decision and design.

In this part of series, some simple and straightforward parts were discovered by a focus on representing the overall design goals and possibilities while adopting Micro Frontends.

The next part will deep dive in Shell implementation and all corresponding modules such as Discovery, Templating, Routing, and putting all pieces together for achieving run and build time advantages.

Impersonation using AWS Congito

Omid Eidivandi — Mon, 29 Apr 2024 23:32:48 +0000

Security stands as a foundational element in software development, often taking center stage in architecture decisions and assessments. The approach to security, both in mindset and execution, can differ depending on factors like the intended usage scenario and the specific layers of the system, whether being public-facing, private, isolated, or serving as gateways. Additionally, a significant challenge arises in ensuring security while navigating the complexities of data ownership in multi-tenant software serving diverse customers. Prioritizing data isolation emerges as a crucial compromise, essential for the efficacy of the software solution.

However, in straightforward software systems, the notion of security can be distinct into two phases: Authentication and Authorization. Authentication facilitates the identification of users seeking interaction with the available services. At the same time, Authorization determines granular access permissions, enabling the system to either permit or deny access to the resources owned by the user.

At a high level, this diagram illustrates the functionality of Authentication (AuthN) and Authorization (AuthZ)

This diagram illustrates the public entry points to the internal ecosystem: the Web App, Mobile App, and Gateway. The initial interaction step involves authentication or signing in for users. Users can be real individuals or machines seeking access to services within the internal ecosystem. In the case of mobile or web apps, users typically sign in using account credentials, employing a username and password mechanism to obtain an access token. Gateways use credential grant standards such as authorization code, implicit, client credentials, or resource password (defined by IETF Authorization grant).

The diagram below illustrates the flow of the Authorization Protocol Flow.

Json Web Token ( JWT )

The JWT is often used as a security measure to fulfil security needs. However, for the sake of clarity, let's assume that ' JWT is not inherently Secure'.

The JWT consists of three main parts: the header, payload, and signature.

Header : Provides information about the token, including the algorithm used.
Payload : Contains the claims, such as email, expiration time, and user unique identifier (sub).
Signature : Uses a private key owned by the server to ensure the token's

JWT Attacks:

Signature Stripping: JWTs are signed and contain a signature as the third part of the token. Attackers attempt to recreate an unsigned token to gain unauthorized access. To counter this, the authorization server must validate the token signature as part of the validation process.
CSRF (Cross-Site Request Forgery): Attackers obtain a signed-in user token and attempt to submit requests to the server from another site. To mitigate this attack, it is advisable to avoid using persisted tokens like cookies unnecessarily. If session persistence is necessary, using short-lived tokens can help. Additionally, incorporating extra meta information such as a unique header in requests, generated previously by the server, adds an extra layer of security.
XSS (Cross-Site Scripting): This occurs when injected scripts reside in the browser and attempt to exploit and steal tokens from some legitimate storage, Often injecting using query strings or textboxes and sending requests using the user's logged-in session cookies or local storage. To mitigate XSS attacks, validating and sanitizing the received data on the server side is essential.

Learn about JWT and all related types like JWK, JWE here

OAuth and OIDC

Historically, OAuth 1 was introduced to apply a security mechanism to give access to parties for using resources on behalf of the resource owner without credential sharing, OAuth2 replaces the first protocol version and tries to safeguard server-side resources, either on behalf of the owner by initiating an access approval workflow or allowing the third party software gain access on behalf of the owner. However, a significant issue with OAuth was the sole responsibility of authorization and access control being with the authorization server. This resulted in a lack of fine-grained control on the application side, as the access token did not furnish adequate information for software to validate resource access and ownership.

OIDC, serving as an extended layer, sought to address this deficiency by introducing the ID token. Generated by the server, the ID token provides software with additional user information and metadata, thereby enhancing control and insight into user identities.

This is just a small part of history, learn more in this ebook by Okta: here

Amazon Cognito

Amazon Cognito stands out as an Identity and Access Management service offered by AWS, allowing you to incorporate a managed layer of security into your software. Amazon Cognito User Pools serve as the cornerstone for managing application-level security and adhering to the OAuth standard.

A user pool serves as a repository for application users (e.g., eidivandi@live.com as a user), with pricing based on Monthly Active Users (MAU). Within a user pool, one or more App Clients can be established. An app client represents an isolated client integration, not only ensuring application isolation but also managing Authentication (AuthN) and Authorization (AuthZ) flow isolation.

Cognito also introduces the concept of triggers, where specific actions within Cognito can invoke custom Lambda code, enhancing its extensibility. The diagram below outlines key actions and their corresponding triggers for any given action.

What we gonna build

In this article, our focus is on configuring an authorization server and implementing the impersonation feature to enable access to sub-accounts or customer resources for an already logged-in user. We'll walk through the setup process, following the structure outlined in the accompanying diagram.

The User authentication will be based on Email/Password credentials, allowing establishing a user session, and the impersonation will allow access to multiple tenant resources. This example will delve into the Password authentication flow and Custom authentication flow, demonstrating how they can work together synergistically. Additionally, it aims to offer insights into understanding Custom Authentication flows within Amazon Cognito more effectively.

Source Code

This article source code can be found on GitHub in the following link.

https://github.com/XaaXaaX/aws-cognito-impersonation

User Pool and App Client

Establishing a user pool and app client is straightforward, especially with the assistance of AWS CDK. The advantage of leveraging Cognito user pools lies in their simplicity for standard authentication and authorization procedures.

Cognito user pools offer various authentication flows, including Password, SRP, Admin Password, and Custom. In this article, we will use both the password flow for the Sign-In process and custom flow for impersonation and accessing resources on behalf of a tenant.

Below is a snippet demonstrating the CDK code to create the User pool, along with both password and custom flow based user pool clients:

this.userPool = new UserPool(this, 'MultiTenantUserPool', {
 removalPolicy: RemovalPolicy.DESTROY,
 accountRecovery: AccountRecovery.NONE,
 email: UserPoolEmail.withCognito('eidivandi@live.com'),
 selfSignUpEnabled: false,
 signInAliases: { email: true },
 autoVerify: { email: true },
 lambdaTriggers:{
   defineAuthChallenge: props.triggers.defineAuthChallenge,
   preTokenGeneration: props.triggers.preTokenGeneration,
   verifyAuthChallengeResponse: props.triggers.verifyAuthChallengeResponse,
 },
 customAttributes: {
  tenants: new StringAttribute({ mutable: true }),
 }
});
this.passwordAuthClient = new UserPoolClient(this, 'MultiTenantPasswordAuthClient', {
 userPool: this.userPool,
 generateSecret: false,
 idTokenValidity: Duration.minutes(5),
 accessTokenValidity: Duration.minutes(5),
 refreshTokenValidity: Duration.days(1),
 authFlows: { userPassword: true } 
});
this.secureAuthClient = new UserPoolClient(this, 'MultiTenantSecureAuthClient', {
 userPool: this.userPool,
 generateSecret: false,
 accessTokenValidity: Duration.minutes(5),
 refreshTokenValidity: Duration.hours(1),
 idTokenValidity: Duration.minutes(5),
 authFlows: { custom: true }
});

The UserPool is a source of authentication process and the users source of trust. The two UserPoolClient are the isolated boundaries for the different required auth flows. Having two separate UserPoolClient is just a preference but a single app client can manage different auth flows.

authFlows: { userPassword: true, custom: true}

Sign Up / Sign In

The sign up is managed by a lambda function using AdminCreateUser command. The function creates a user by generating a temporary password and user attributes including some default and reserved ones but also a custom attribute presenting a list of allowed tenants this user can interact with ( use of custom attribute is for article simplicity and can be done using any other type of storage). The function also forces the verification of email, this is done just for the sake of simplicity and to avoid changing the password behind the first signin while testing.

const UserAttributes = [{
 Name: 'family_name',
 Value: lastName 
},
{
 Name: 'given_name',
 Value: firstName 
},
{
 Name: 'email',
 Value: email 
},
{
 Name: 'email_verified',
 Value: 'true' 
}, 
{
 Name: 'name',
 Value: `${lastName} ${firstName}`
},
{
 Name: 'custom:tenants',
 Value: tenants?.join(',') 
}]; 
const adminCreateUserParams = {
 UserPoolId: process.env.COGNITO_USER_POOL_ID,
 Username: email,
 TemporaryPassword: generator.generate({ length: 10, numbers: true, symbols: true, strict: true, exclude: '&%#?+:/;', }),
 DesiredDeliveryMediums: ['EMAIL'],
 UserAttributes,
 ClientMetadata: { step: 'SignUp_CreateUser', } 
} satisfies AdminCreateUserCommandInput;

After signing up, an email will be sent with the temporary password, we use this email to signins, the signin function will use InitiateAuth command to authenticate and force the change password challenge by keeping the temporary password ( This step is just for demo and often in production the user will change the password, so the NEW-PASSWORD_REQUIRED challenge will be used to force the user to change the password )

const signInParams: InitiateAuthCommandInput = {
 AuthFlow: AuthFlowType.USER_PASSWORD_AUTH,
 ClientId: process.env.COGNITO_USER_POOL_CLIENT_ID,
 AuthParameters: { USERNAME: username, PASSWORD: password, },
 ClientMetadata: { step: 'Signin_InitAuth' }
};
const signinResponse = await client.send(new InitiateAuthCommand(signInParams));

The InitiateAuth response has a session that must be used to trigger the challenge as following snippet demonstrates.

if( signinResponse.ChallengeName === 'NEW_PASSWORD_REQUIRED' ) {
  const challengeParams: RespondToAuthChallengeCommandInput = {
    ChallengeName: 'NEW_PASSWORD_REQUIRED',
    ClientId: process.env.COGNITO_USER_POOL_CLIENT_ID,
    ChallengeResponses: {
      USERNAME: username,
      NEW_PASSWORD: password,
    },
    Session: signinResponse.Session,
    ClientMetadata: { step: 'Signin_Respond_Challenge' }
 };
 const challengeResponse = await client.send(new RespondToAuthChallengeCommand(challengeParams));
 authenticationResult = challengeResponse.AuthenticationResult; 
}

The challenge result will include the IdToken, AccessToken and RefreshToken and session that will be returned as the response.

Impersonation

The impersonation step is a custom Auth process including to step process, InitiateAuth and RespondToAuthChallenge command. before looking at how that behaves it is important to see how this custom flow works.

The user pool has 3 lambda trigger:

DefineAuthChallenge
VerifyAuthChallenge
PreTokenGeneration

Triggering the challenge can be done as per AWS Documentation here , when using a Custom auth flow, both the InitiateAuth and RespondToChallenge will trigger the DefineAuthChallenge with VerifyAuthChallengeResponse triggered between two invocations after the challenge verification passes and if the second DefineAuthChallenge invocation response indicates the generation of token the PreTokenGenration trigger will be invoked.

const initiateAuthCommandInput = new InitiateAuthCommand({
 AuthFlow: AuthFlowType.CUSTOM_AUTH,
 ClientId: clientId,
 AuthParameters: { USERNAME: email, },
 ClientMetadata: { step: 'Impersonation_InitAuth', tenant, }
});
const response = await client.send(initiateAuthCommandInput);
const challengeCommandInput = new RespondToAuthChallengeCommand({
 ChallengeName: 'CUSTOM_CHALLENGE',
 ClientId: clientId,
 ChallengeResponses: {
   USERNAME: response.ChallengeParameters?.USERNAME || '',
   ANSWER: 'impersonation',
 },
 ClientMetadata: {
   authFlow: 'impersonation',
   step: 'Impersonation_RespondToChallenge',
   tenant
 },
 Session: response.Session,
});
const challengeResponses = await client.send(challengeCommandInput);

Initiate Auth : DefineAuthChallenge

In the first step the DefineAuthChallenge trigger will receive an invocation with an event payload including the user attributes and an empty array of sessions. the sessions array represents the previous steps in auth challenge, for this first invocation the array is empty.This first invocation is triggered by InitiateAuth command.

{
 ...
 "triggerSource": "DefineAuthChallenge_Authentication",
 "request": {
   "userAttributes": {
     "sub": "e29524e4-f0a1-7018-7650-4bee0ca85f4b",
     "email_verified": "true",
     "cognito:user_status": "CONFIRMED",
     "name": "Hills Samir",
     "given_name": "Samir",
     "custom:tenants": "CUS-01,CUS-02",
     "family_name": "Hills",
     "email": "eidivandi@live.com"
  },
  "session": []
 },
 ...
}

The Lambda Function will orchestrates this steps as below and change the response elements.

if ( event.request.session.length === 0 ) {
  event.response.issueTokens = false;
  event.response.failAuthentication = false; 
  event.response.challengeName = 'IMPERSONATE_CHALLENGE';
} else if (...) {
   ....
} else {
   event.response.issueTokens = false; 
   event.response.failAuthentication = true;
}
return event;

Respond Auth Challenge: VerifyAuthChallenge

The RespondToAuthChallenge Command will initiate the rest of flow, and as first step to verify the previously initiated challenge by invoking the VerifyAuthChallenge trigger, this is where the validation verifies if the ChallengeAnswer corresponds to the response provided. As part of the validation, the verification of tenant parameters with the user's custom tenants attributes validates if interacting with this tenant resources is authorized for this user.

if (
   event.request.clientMetadata?.tenant &&
   event.request.userAttributes?.['custom:tenants']?.split(',')?.includes(event.request.clientMetadata?.tenant)
) {
 event.response.answerCorrect = 'impersonation' === event.request.challengeAnswer;
}
return event;

Respond Auth Challenge: DefineAuthChallenge

In the next step the DefineAuthChallenge will be invoked for a second time but the event will be partially different, including more information about previous session initiated and some metadata.

{
 ...
 "triggerSource": "DefineAuthChallenge_Authentication", 
 "request": {
   "userAttributes": {
     "sub": "e29524e4-f0a1-7018-7650-4bee0ca85f4b", 
     "email_verified": "true",
     "cognito:user_status": "CONFIRMED",
     "name": "Hills Samir",
     "given_name": "Samir",
     "custom:tenants": "CUS-01,CUS-02",
     "family_name": "Hills",
     "email": "eidivandi@live.com"
   },
   "session": [{
      "challengeName": "CUSTOM_CHALLENGE",
      "challengeResult": true,
      "challengeMetadata": null
   }],
   "clientMetadata": {
      "step": "Impersonation_RespondToChallenge",
      "authFlow": "impersonation",
      "tenant": "CUS-01"
    }
 },
 ...
}

Here the ClientMetadata is the only available option to pass the information between different triggers in the same challenge. this helps to share an ephemeral state between invocations. The function handler code will verify the session length and the challenge result has a truthy value.

if ( event.request.session.length === 0 ) {
  ... 
} else if ( event.request.session.length === 1 && event.request.session[0].challengeResult === true ) {
  event.response.issueTokens = true;
  event.response.failAuthentication = false;
} else { ... }

return event;

As part of the event response, the issueTokens and failAuthentication indicate if a token can be generated or not, or even failing the authentication.

Respond Auth Challenge: PreTokenGeneration

The next step will be the PreTokenGeneration to apply customization of token claims. The default token customization can be applied only to the idToken but advanced security options if activated will give the possibility of access token customization. ( Advanced Security Options applies extra cost )

let tenant; 
if( event.triggerSource == 'TokenGeneration_Authentication' && event.request.clientMetadata?.step == 'Impersonation_RespondToChallenge' ) { 
   tenant = event.request.clientMetadata?.tenant;
};
event.response = {
  claimsOverrideDetails: {
    claimsToAddOrOverride: {
     tenant
    },
    claimsToSuppress: [],
    groupOverrideDetails: {
      groupsToOverride: [],
      iamRolesToOverride: [],
      preferredRole: "",
    },
  }
};
return event;

The function verifies the trigger source and custom metadata provided in impersonation handler and injects the tenant claim in the token.

Authorizer

As per the diagram shown in What we gonna build section, the solution needs to have two layers of authorization. The first one is to validate the user sessions and authorize them to access protected endpoints, and the second one is to validate and authorize the impersonation token when trying to access the downstream services.

User Session

The first Authorizer is a CognitoAuthorizer attached to protected endpoints like impersonate endpoint. This can be achieved using aws CDK as below

const cognitoPasswordAuthorizer = new HttpUserPoolAuthorizer('CognitoPasswordUserPoolAuthorizer', props.cognito.userPool, {
   userPoolClients: [props.cognito.passwordAuthClient]
});
...
api.addRoutes({
  path: '/user/impersonate',
  methods: [HttpMethod.POST],
  integration: new HttpLambdaIntegration('ImpersonassionAuthFunctionIntegration', impersonassionAuthFunction),
  authorizer: cognitoPasswordAuthorizer
});

This authorizer will validate the signin token generated behind email and password authentication, to prevent unauthorized access to impersonation endpoint.

Impersonation

The impersonation authorier is a Lambda CustomAuthorizer integrating with ApiGateway and validates that the authorization token received is allowed to access the tenant resources. For this article's simplicity we use a single apigatway shared between downstream and authorization service but add a Lambda Authorizer attached at the route level for the downstream endpoint.

const customAuthorizer = new LambdaFunction(this, 'CustomAuthorizer', {
   entry: resolve(join(__dirname, '../../src/authorizer/handler.ts')),
   bundling: {
     banner: `import { createRequire } from 'module';const require = createRequire(import.meta.url);`,
   },
   environment: {
     COGNITO_USER_POOL_CLIENT_ID: props.cognito.secureAuthClient.userPoolClientId, 
     COGNITO_USER_POOL_ID: props.cognito.userPool.userPoolId,
     TABLE_NAME: props.table.tableName
    }
 });
 const cognitoImpersonationAuthorizer = new HttpLambdaAuthorizer('CognitoImpersonationAuthorizer', customAuthorizer , {});

The authorizer validates the Token against cognito userpool and client app.

const token = event.authorizationToken!.replace('Bearer ', '');
const decodedToken = decodeAndGetToken(token);
const user = await verifyToken(token, decodedToken.token_use, process.env.COGNITO_USER_POOL_CLIENT_ID!);

The verifyToken method uses the ' aws-jwt-verify' to validate the token using cognito userpool client.

const verifierParams = {
  userPoolId: process.env.COGNITO_USER_POOL_ID!,
  tokenUse: use as CognitoVerifyProperties['tokenUse'],
  clientId: process.env.COGNITO_USER_POOL_CLIENT_ID!,
};
const verifier = CognitoJwtVerifier.create(verifierParams); return verifier.verify(token, verifierParams);

The authorizer returns a policy allowing or denying access to invoke api gateway

return {
  principalId: 'user',
  policyDocument: {
    Version: '2012-10-17',
    Statement: [{
      Action: 'execute-api:Invoke',
      Effect: 'Allow',
      Resource: event.methodArn,
   }],
 },
};

The authorizer simply validates the sanity of the token and if it was originally owned by authorization server, but the fact of allowing that token to access tenant resources must be under the responsibility of downstream.

Lets say, that omid@gmail.com signed in and asked to impersonate the tenant CUS-01 now the generated token will be validated even if the requested resource to downstream is owned by CUS-02, this is the responsibility of the downstream service to validate if the authorized token corresponds well the CUS-02 tenant or not.

In the following section, we will see this in practice and deploy the sample solution.

Running the authorization server

https://github.com/XaaXaaX/aws-cognito-impersonation

The source code provides a brief README to follow and deploy the solution, and a postman collection and environment are also provided to simplify the testing purpose.

To start and test the solution lets start by signing up, Amazon Cognito will send an email containing a temporary password.

https://youtu.be/uMwp44P1H_o

After account creation, an email will be received containing the temporary password

Using this password the sign-in endpoint allows to establish a session using the password auth flow. Passing the received IdToken from the sign-in step as an authorization header lets the cognito authorizer identify the session and validate the access to impersonate api route.

https://youtu.be/RCkxXoix3fg

The Impersonation token will be generated by custom flow and must be passed along the call to the downstream, the custom lambda authorizer will validate the generated token, the responsibility of validating the resource is by downstream service to check the asked resource and validate it against the authorizer context provided.

In this example the downstream verifies the requested resource against authorization token context.

try {
   const eventBody = JSON.parse(event.body || '{}');
   if( eventBody.tenant !== event.requestContext.authorizer?.lambda.tenant ) {
     throw new Error('Tenant does not match');
   }
   return ActionResults.Success(eventBody);
} catch (e) {
    console.error(e);
    return ActionResults.InternalServerError({ message: e.message || e.name });
}

Conclusion

Security being part of our daily software development is not always simple such as providing an Api key or a JWT token, and when it gets complicated, this is important to think about shared responsibility. Understanding the Security foundation like AuthN, AuthZ and different components such as Client, Authorization server, and Resource Server will help to better give responsibility and scopes to each part of the communication flow.

Applying standards is not always free of effort and needs a higher level of deep understanding so looking at some resources like IETF helps to achieve a deeper vision and perspective.

Here some resources:

JWT ( rfc7519 ) : https://datatracker.ietf.org/doc/html/rfc7519#page-9
OAuth2 ( rfc6749 ) : https://datatracker.ietf.org/doc/html/rfc6749#page-7

Hope this article will be useful.

Experimenting Multiple triggers for Amazon SQS

Omid Eidivandi — Wed, 24 Apr 2024 00:58:36 +0000

AWS lambda is a core component of a wide range of software designs, as an advantage of lambda service, we can focus on its simple and efficient integration with other services in the AWS ecosystem. The Amazon SQS is an old-school member of this ecosystem that offers a highly scalable messaging queue service and integrates perfectly with AWS lambda.

Queue offering

The queuing services are used to decouple the software systems by their ability to act as an asynchronous transitional service to send messages from one system to another and let the producers and consumers be decoupled and independent.

In a queueing system, the producers write the messages to the queue, and the consumers fetch the messages from the queue process, and the message gets removed when no longer needed. by default, the queuing services keep the message as long as there is no removal of demand from consumers.

This design allows the producers and consumers to be independent and decoupled in terms of processing capacity, and manage better their internal state in an isolated way such as managing eventual failures, scaling, etc.

Event vs Job Consumer

Amazon SQS like any queuing service offers the basic functionality of message retrieval by consumers, this is often done by some sort of batch jobs or instances that periodically ask for messages. Any consumer can ask for removal at the end of processing. On the other hand, event-based consumption is more of a serverless concept that offers to send messages to the consumers per message availability, Amazon SQS being a serverless messaging service does not offer event-based message distribution unlike Amazon SNS and works with a job consumer approach.

Source Code

https://github.com/XaaXaaX/aws-sqs-lambda-message-filtering

AWS Lambda / SQS

AWS lambda integrates with SQS and this integration offers smooth event-based consumption but this impression is by the excellent way that AWS lambda manages this integration. How it works behind the scenes is that the Lambda service asks for a batch of visible messages in the queue and will manage the consumption and message lifecycle on its own, for sure the principal management of messages inside the queue is under SQS ownership like visibility timeout, delays, etc.

The lambda poller receives the messages from SQS, giving the desired maximum number of messages and the maximum wait time to let the messages be gathered.

Consequently, in lambda event source mapping, the filtering will be applied followed by a process of batching to prepare the batch of records per function configuration before invoking the function.

Filtering

As the following diagram illustrates, the filtering will be applied on the Lambda service side, and if the configured filtering can be applied the record will be considered to be passed to the lambda function. But in the case that the configured filtering can not be applied to the record what happens is that the lambda service discards the message to be processed but also considers it as a message to be deleted and it will remove that message from SQS.

If you are a fan of reading documentation like me you already know this, but I had a mental challenge to see how the SQS would behave if I used the SQS as a central queue in my system, and my reason was that in some central part of the system we can offload the events to multiple lambda consumers.

Logically the idea was that if I applied the filtering as each lambda listens to a different event each message could have a single consumer and this would not be against the recommendation and that helps me have central control of my events without having a monolith function to handle a significant amount of process or doing some sort of lambda based orchestration. The following diagram demonstrates the exact scenario.

As part of my tests, I sent 3 different message with different payloads

{ "isProductTransitEvent": "true", "productId": "123456", "deliveryId": "1"}

{ "isProductSynchroEvent": "true", "productId": "123456", "lotId": "12345"}

{ "isProductStockEvent": "true", "productId": "123456", "quantity": 10}

Only the first function received the messages, but sending more messages for 3rd payload results in randomly receiving messages but not all.

Another hypothesis was sending a single payload with different values for filtered fields, this time none of the second and third functions received their messages per filtering. trying to send 20 messages resulted in the same behavior as we experienced previously.

Lambda integration with SQS using Event source mapping at a high level can be presented as illustrated in the following sequence diagram, and the discarded events will be treated the same as successful messages.

The results ensure that the first trigger receives most of the time a large part of events, but what about if the records fail? what happens to the discarded events being part of the same batch of messages passed through the first trigger?

The answer is simple, In case of failure the discarded messages are deleted like in the success scenario, and the failed messages are retried, but the failed messages also after experiencing the error become visible after visibility timeout and will be reflected for another time, this leads potentially to loss the message in retry phase. as per the tests I did for this article, I never experienced a second retry for 10 messages.

Hope this can be useful ;)