DEV Community: Cláudio Filipe Lima Rapôso

Azure Cellular Architecture: Scaling with Cosmos DB Change Feed and Service Bus

Cláudio Filipe Lima Rapôso — Thu, 30 Apr 2026 12:25:41 +0000

1. Introduction

Scaling cloud-native systems often results in a massive, highly interconnected web of services where a single corrupted payload or regional brownout can trigger a global outage. Cellular Architecture mitigates this by decomposing the system into completely independent, isolated failure domains known as "cells." In this tutorial, you will build a highly resilient event-driven architecture using Microsoft Azure, combining the cellular pattern with an asynchronous workflow (Azure Cosmos DB, Azure Functions, and Azure Service Bus).

Instead of relying on a single monolithic data and compute plane, you will deploy multiple identical infrastructure stamps. A thin edge routing layer will inspect incoming requests and route them to the appropriate cell based on a partition key (such as a TenantId). If a localized infrastructure degradation or a "poison pill" event impacts one cell, the blast radius is strictly contained within that specific boundary, ensuring maximum availability for tenants residing in other cells. This pattern represents the ultimate fault isolation strategy for mission-critical enterprise systems on Azure.

2. Prerequisites

To successfully deploy this architecture, your environment must be equipped with the following:

An active Azure Subscription with Owner or User Access Administrator privileges to provision compute, database, messaging, and Role-Based Access Control (RBAC) resources.
The Azure CLI installed and authenticated locally (az login).
Terraform (version 1.3.0 or higher) installed and accessible from your terminal.
A solid understanding of the separation between a Control Plane (global routing state) and a Data Plane (the actual processing cells).
Familiarity with Domain-Driven Design (DDD) to accurately define the partition keys that will dictate cell placement without creating cross-boundary data dependencies.

3. Step-by-Step

Step 1: Defining the Data Plane Cell Module

What to do: Create a reusable Terraform module that encapsulates the entire event-driven pipeline (Resource Group, Cosmos DB, Service Bus, Storage, and Azure Functions). This module represents a single, mathematically isolated cell.

Why do it: The core tenet of cellular architecture is absolute repeatability and isolation. By packaging the data plane into a module, every cell becomes an identical, deterministic stamp of infrastructure. Provisioning a dedicated Cosmos DB account and Service Bus namespace per cell guarantees that a noisy neighbor or an overloaded queue in "Cell Alpha" consumes zero compute or IOPS from "Cell Beta."

Screenshot/Example: Save this configuration as modules/azure_cell/main.tf.

variable "cell_id" {
  description = "Unique identifier for the cell (e.g., alpha, beta)"
  type        = string
}

variable "location" {
  description = "Azure region for the cell"
  type        = string
  default     = "East US"
}

# Dedicated Resource Group for the Cell
resource "azurerm_resource_group" "cell_rg" {
  name     = "rg-cell-${var.cell_id}"
  location = var.location
}

# Isolated Cosmos DB Account for the Cell
resource "azurerm_cosmosdb_account" "cell_db" {
  name                = "cosmos-data-cell-${var.cell_id}"
  location            = azurerm_resource_group.cell_rg.location
  resource_group_name = azurerm_resource_group.cell_rg.name
  offer_type          = "Standard"
  kind                = "GlobalDocumentDB"

  consistency_policy { consistency_level = "Session" }
  geo_location { location = azurerm_resource_group.cell_rg.location, failover_priority = 0 }
}

# Isolated Service Bus Namespace for the Cell
resource "azurerm_servicebus_namespace" "cell_sb" {
  name                = "sb-namespace-cell-${var.cell_id}"
  location            = azurerm_resource_group.cell_rg.location
  resource_group_name = azurerm_resource_group.cell_rg.name
  sku                 = "Standard"
}

resource "azurerm_servicebus_topic" "cell_topic" {
  name         = "processing-topic"
  namespace_id = azurerm_servicebus_namespace.cell_sb.id
}

resource "azurerm_servicebus_subscription" "cell_sub" {
  name               = "processing-subscription"
  topic_id           = azurerm_servicebus_topic.cell_topic.id
  max_delivery_count = 5 # Built-in poison pill protection
}

# Note: In a complete module, you would also define the azurerm_linux_function_app 
# resources here for the Producer (Change Feed Trigger) and Consumer (Service Bus Trigger),
# utilizing SystemAssigned managed identities for secure communication within the cell.

Step 2: Instantiating the Independent Cells

What to do: In your root Terraform configuration, instantiate the cell module multiple times to create your physical failure domains.

Why do it: Deploying multiple instances generates the actual physical boundaries. You can assign high-tier enterprise tenants to their own dedicated cells or group smaller tenants into shared cells. Because the modules are parameterized, you can easily deploy cells across different Azure regions to combine cellular isolation with geographic high availability.

Screenshot/Example: Add this to your root main.tf.

module "cell_alpha" {
  source   = "./modules/azure_cell"
  cell_id  = "alpha"
  location = "East US 2"
}

module "cell_beta" {
  source   = "./modules/azure_cell"
  cell_id  = "beta"
  location = "West US 2"
}

module "cell_gamma" {
  source   = "./modules/azure_cell"
  cell_id  = "gamma"
  location = "North Europe"
}

Step 3: Building the Control Plane Mapping

What to do: Provision a highly available, globally distributed Cosmos DB account that acts as the Control Plane data store. This database maps your tenant identifiers to their assigned Cell ID.

Why do it: The routing layer must dynamically know where to forward incoming requests. This global mapping table is queried by the edge router. It must be decoupled from the data plane and highly available; if the routing map goes down, traffic cannot reach the perfectly healthy data plane cells behind it.

Screenshot/Example: Create a control_plane.tf file.

resource "azurerm_resource_group" "routing_rg" {
  name     = "rg-global-routing"
  location = "East US"
}

resource "azurerm_cosmosdb_account" "routing_db" {
  name                = "cosmos-global-tenant-map"
  location            = azurerm_resource_group.routing_rg.location
  resource_group_name = azurerm_resource_group.routing_rg.name
  offer_type          = "Standard"
  kind                = "GlobalDocumentDB"

  consistency_policy { consistency_level = "Eventual" } # Optimize for fast reads

  geo_location { location = "East US", failover_priority = 0 }
  geo_location { location = "West Europe", failover_priority = 1 }
}

resource "azurerm_cosmosdb_sql_database" "map_db" {
  name                = "RoutingDB"
  resource_group_name = azurerm_resource_group.routing_rg.name
  account_name        = azurerm_cosmosdb_account.routing_db.name
}

resource "azurerm_cosmosdb_sql_container" "tenant_map" {
  name                  = "TenantCellMap"
  resource_group_name   = azurerm_resource_group.routing_rg.name
  account_name          = azurerm_cosmosdb_account.routing_db.name
  database_name         = azurerm_cosmosdb_sql_database.map_db.name
  partition_key_path    = "/tenantId"
}

Step 4: Implementing the Thin Cell Router

What to do: Deploy an Azure Function App with an HTTP trigger acting as the "Cell Router". This function extracts the TenantId from the incoming HTTP payload, queries the global Cosmos DB mapping table to resolve the destination cell, and then forwards the payload to that specific cell's Cosmos DB.

Why do it: The cell router is the critical entry point and must remain computationally "thin." Complex business logic belongs strictly inside the data plane cells, not the router. By keeping the router simple, you minimize its execution time and failure modes. Once the router successfully writes the data to the target cell's Cosmos DB, the localized event-driven pipeline (Change Feed -> Producer Function -> Service Bus -> Consumer Function) takes over autonomously.

Screenshot/Example: Create a router.tf file.

# Storage Account and App Service Plan for the Router Function
resource "azurerm_storage_account" "router_storage" {
  name                     = "stglobalrouterfunc"
  resource_group_name      = azurerm_resource_group.routing_rg.name
  location                 = azurerm_resource_group.routing_rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_service_plan" "router_plan" {
  name                = "asp-router-plan"
  resource_group_name = azurerm_resource_group.routing_rg.name
  location            = azurerm_resource_group.routing_rg.location
  os_type             = "Linux"
  sku_name            = "Y1"
}

# The Thin Router Function
resource "azurerm_linux_function_app" "cell_router" {
  name                       = "func-global-edge-router"
  resource_group_name        = azurerm_resource_group.routing_rg.name
  location                   = azurerm_resource_group.routing_rg.location
  service_plan_id            = azurerm_service_plan.router_plan.id
  storage_account_name       = azurerm_storage_account.router_storage.name
  storage_account_access_key = azurerm_storage_account.router_storage.primary_access_key

  site_config { application_stack { node_version = "20" } }

  identity { type = "SystemAssigned" }
}

# RBAC: Allow Router to read the mapping table
resource "azurerm_role_assignment" "router_map_reader" {
  scope                = azurerm_cosmosdb_account.routing_db.id
  role_definition_name = "Cosmos DB Account Reader Role" # Note: Use custom data plane roles in production
  principal_id         = azurerm_linux_function_app.cell_router.identity[0].principal_id
}

4. Common Troubleshooting

Cross-Cell Dependencies: The most critical anti-pattern in cellular architecture is allowing cells to communicate with each other or share a backend data store. If "Cell Alpha" synchronously queries an API in "Cell Beta", the isolation boundary is compromised. Ensure strict domain boundaries; data required to process an event within a cell must reside entirely within that cell's Cosmos DB.
Control Plane Bottlenecks (Router Fatigue): The Cell Router represents a single point of failure. If the HTTP-triggered Azure Function queries the global Cosmos DB mapping table on every single request, you will encounter latency and potential RU/s (Request Unit) throttling. You must implement aggressive in-memory caching (e.g., using MemoryCache in C# or standard dictionary caching in Node.js) within the router's execution context so that tenant-to-cell mappings are resolved locally and instantly after the first lookup.
Poison Pills and Service Bus Dead-Lettering: If a malformed payload triggers the Cosmos DB Change Feed but crashes the Consumer Azure Function, the message will be returned to the Service Bus Subscription. Because this is a cellular architecture, this poison pill will only degrade the specific cell it entered. To resolve this automatically, the Terraform module is configured with max_delivery_count = 5. Service Bus will automatically move the failing message to the Dead-Letter Queue (DLQ) after 5 attempts, restoring the cell's processing health.

5. Conclusion

By embedding an asynchronous event-driven flow within a cellular architecture on Microsoft Azure, you have constructed a system optimized for extreme resilience and strictly controlled blast radiuses. The rigid separation between a global routing control plane and fully isolated data plane cells ensures that infrastructure failures, bad deployments, or tenant-specific traffic spikes remain mathematically contained. Utilizing Terraform modules to stamp out these cells guarantees environmental consistency and facilitates rapid horizontal scaling. As your system matures, consider enhancing the edge routing layer by placing Azure Front Door in front of your Cell Router Functions to provide global load balancing, WAF protection, and seamless traffic shifting during cell migrations or disaster recovery events.

AWS Cellular Architecture: Scaling Event-Driven Systems with DynamoDB, SNS, and SQS

Cláudio Filipe Lima Rapôso — Wed, 29 Apr 2026 15:00:02 +0000

1. Introduction

In the cloud-native era, scaling an architecture often leads to a massive, interconnected system where a single failure can cause a global outage. Cellular Architecture solves this by decomposing the system into independent, isolated failure domains called "cells." In this tutorial, you will build a highly resilient event-driven architecture by combining the cellular pattern with an asynchronous flow (DynamoDB Streams, AWS Lambda, Amazon SNS, and Amazon SQS).

Instead of a single monolithic data plane, you will deploy multiple identical infrastructure stamps (cells). A thin edge routing layer will inspect incoming requests and route them to the appropriate cell based on a partition key (such as a Tenant ID). If a poison pill event or a localized infrastructure degradation impacts one cell, the blast radius is strictly contained, ensuring maximum availability for tenants in other cells. This approach provides the ultimate fault isolation boundary for mission-critical distributed systems.

2. Prerequisites

To execute this deployment, your environment must be equipped with the following tools and foundational knowledge:

An active AWS account with administrative privileges to provision IAM, compute, database, and messaging resources.
Terraform (version 1.3.0 or higher) installed locally, alongside the AWS CLI authenticated with your credentials.
A clear understanding of the separation between the Control Plane (global routing state) and the Data Plane (the actual processing cells).
Familiarity with Domain-Driven Design (DDD) to correctly define the partition keys that will dictate cell placement without creating cross-boundary data dependencies.

3. Step-by-Step

Step 1: Defining the Data Plane Cell Module

What to do: Create a reusable Terraform module that encapsulates the entire event-driven pipeline (DynamoDB, SNS, SQS, and Lambdas). This module represents a single, self-contained cell.

Why do it: The core tenet of cellular architecture is repeatability. By packaging the data plane into a module, you ensure that every cell is an identical, deterministic stamp of infrastructure. This structural isolation guarantees that an overloaded SQS queue or a throttling DynamoDB table in Cell Alpha has absolutely zero impact on the compute resources in Cell Beta.

Screenshot/Example: Save this configuration as modules/event_driven_cell/main.tf.

variable "cell_id" {
  description = "Unique identifier for the cell (e.g., alpha, beta)"
  type        = string
}

# DynamoDB Table restricted to this specific cell
resource "aws_dynamodb_table" "cell_table" {
  name           = "app-data-cell-${var.cell_id}"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "id"

  attribute {
    name = "id"
    type = "S"
  }

  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"
}

# SNS Topic and SQS Queue strictly bound to this cell
resource "aws_sns_topic" "cell_topic" {
  name = "processing-topic-cell-${var.cell_id}"
}

resource "aws_sqs_queue" "cell_queue" {
  name = "processing-queue-cell-${var.cell_id}"
}

resource "aws_sns_topic_subscription" "cell_sub" {
  topic_arn = aws_sns_topic.cell_topic.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.cell_queue.arn
}

resource "aws_sqs_queue_policy" "cell_queue_policy" {
  queue_url = aws_sqs_queue.cell_queue.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = { Service = "sns.amazonaws.com" }
        Action    = "sqs:SendMessage"
        Resource  = aws_sqs_queue.cell_queue.arn
        Condition = {
          ArnEquals = { "aws:SourceArn" = aws_sns_topic.cell_topic.arn }
        }
      }
    ]
  })
}

# Producer Lambda Event Source Mapping
resource "aws_lambda_event_source_mapping" "dynamo_trigger" {
  event_source_arn  = aws_dynamodb_table.cell_table.stream_arn
  function_name     = aws_lambda_function.cell_producer.arn
  starting_position = "LATEST"
}

Step 2: Instantiating the Independent Cells

What to do: In your root Terraform configuration, instantiate the cell module multiple times to create your isolated failure domains.

Why do it: Deploying multiple instances generates the physical infrastructure boundaries. For example, assigning high-tier enterprise tenants to their own dedicated cells prevents noisy neighbor problems. While this deployment runs natively on AWS, establishing decoupled infrastructure stamps is the fundamental prerequisite for extending your architecture to active-active multi-cloud environments in the future.

Screenshot/Example: Add this to your root main.tf.

module "cell_alpha" {
  source  = "./modules/event_driven_cell"
  cell_id = "alpha"
}

module "cell_beta" {
  source  = "./modules/event_driven_cell"
  cell_id = "beta"
}

module "cell_gamma" {
  source  = "./modules/event_driven_cell"
  cell_id = "gamma"
}

Step 3: Building the Control Plane Mapping

What to do: Create a global DynamoDB table that acts as the Control Plane data store. This table maps your partition keys (e.g., Tenant ID) to their assigned Cell ID.

Why do it: The routing layer must know where to send incoming requests without utilizing hardcoded logic. This global mapping table is queried by the edge router dynamically. It must be highly available, as its failure would prevent traffic from reaching the perfectly healthy data plane cells behind it.

Screenshot/Example: Create a control_plane.tf file.

resource "aws_dynamodb_table" "tenant_routing_map" {
  name           = "global-tenant-cell-mapping"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "tenant_id"

  attribute {
    name = "tenant_id"
    type = "S"
  }

  tags = {
    Layer = "ControlPlane"
  }
}

Step 4: Implementing the Thin Cell Router

What to do: Deploy an Amazon API Gateway linked to a Lambda function. This function serves as the "Cell Router". It extracts the Tenant ID from the incoming HTTP payload, queries the global mapping table to resolve the destination cell, and then forwards the data to that specific cell's DynamoDB table.

Why do it: The cell router is the critical entry point and must remain computationally "thin". Any complex business logic belongs inside the cell, not the router. By keeping the router simple, you minimize its failure modes. Once the router successfully writes the initial data to the target cell's DynamoDB table, the localized event-driven pipeline (Streams -> Producer Lambda -> SNS -> SQS -> Consumer Lambda) takes over autonomously.

Screenshot/Example: Create a router.tf file.

# IAM Role granting the Router permission to read the map and write to ALL cells
resource "aws_iam_role" "router_role" {
  name = "edge-cell-router-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{ Action = "sts:AssumeRole", Effect = "Allow", Principal = { Service = "lambda.amazonaws.com" } }]
  })
}

resource "aws_iam_role_policy" "router_permissions" {
  role = aws_iam_role.router_role.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["dynamodb:GetItem"]
        Resource = aws_dynamodb_table.tenant_routing_map.arn
      },
      {
        Effect   = "Allow"
        Action   = ["dynamodb:PutItem"]
        Resource = "arn:aws:dynamodb:*:*:table/app-data-cell-*"
      }
    ]
  })
}

resource "aws_lambda_function" "cell_router" {
  function_name = "global-edge-router"
  handler       = "router.handler"
  runtime       = "nodejs20.x"
  role          = aws_iam_role.router_role.arn
  # Code package implementation omitted for brevity
}

4. Common Troubleshooting

Cross-Cell Dependencies and Shared State: The most destructive error in cellular architecture is allowing cells to communicate with each other or share a backend resource (like a single centralized RDS instance or S3 bucket). If Cell Alpha synchronously queries data from Cell Beta, the isolation boundary is broken. Ensure strict domain boundaries; data required by a cell must reside entirely within that cell.
Control Plane Bottlenecks (Router Fatigue): The Cell Router represents a single point of failure. If the Lambda router queries the global mapping table on every single request, you will encounter latency and DynamoDB read throttling. You must implement aggressive in-memory caching within the router's execution context so that tenant-to-cell mappings are resolved instantly locally.
Poison Pills and Asynchronous Failures: If a malformed payload triggers the DynamoDB Stream but crashes the Consumer Lambda, the message will cycle continuously in the SQS queue. Because this is a cellular architecture, this poison pill will only degrade the specific cell it entered. To resolve the localized issue, ensure your cell module includes a Dead Letter Queue (DLQ) to automatically eject failing messages after a set number of retries.

5. Conclusion

By embedding an event-driven flow within a cellular architecture, you have constructed a system optimized for extreme resilience and strictly controlled blast radiuses. The separation between a global routing control plane and isolated data plane cells ensures that infrastructure or deployment failures remain mathematically contained. Utilizing Terraform modules to stamp out these cells guarantees environmental consistency and facilitates rapid, linear horizontal scaling. As your system matures, consider enhancing the edge routing layer using Amazon Route 53 Application Recovery Controller (ARC) for advanced DNS-level cell shifting and disaster recovery automation.

Serverless Data Processing: Azure CosmoDB, Azure Functions, and Azure Service Bus

Cláudio Filipe Lima Rapôso — Wed, 29 Apr 2026 03:00:00 +0000

1. Introduction

An event-driven architecture using Azure Cosmos DB, Azure Functions, and Azure Service Bus is the enterprise standard for designing highly decoupled, scalable, and resilient systems in the Microsoft Cloud. In this tutorial, you will construct a complete, asynchronous data flow from scratch using Azure's analogous features to the AWS ecosystem.

The process begins when modifications occur in a Cosmos DB container, which automatically surfaces through the Cosmos DB Change Feed. This feed is continuously monitored by a producer Azure Function, which processes the change and publishes a message to a Service Bus Topic. The Service Bus Topic acts as a high-availability message router (similar to AWS SNS), performing a fan-out distribution of this notification to a Service Bus Subscription (acting as the queue, similar to AWS SQS). Finally, a second consumer Azure Function reads the messages from the subscription to perform the final business logic processing. By the end of this guide, you will be able to deploy this robust infrastructure using Terraform, adhering to Azure security best practices such as utilizing Managed Identities instead of vulnerable connection strings.

2. Prerequisites

Before starting the infrastructure build, ensure your development environment is prepared. You will need an active Azure subscription with permissions to provision compute, messaging, database, and Role-Based Access Control (RBAC) resources. The Azure Command-Line Interface (Azure CLI) must be installed and authenticated (az login) on your local machine. Additionally, Terraform (version 1.3.0 or higher) needs to be installed and globally accessible from your command terminal. Fundamental knowledge of Terraform's declarative syntax (HCL) and a basic understanding of Azure resource hierarchy (Resource Groups) are highly recommended. A modern code editor like VS Code with the HashiCorp Terraform extension will greatly facilitate your workflow.

3. Step-by-Step

Step 1: Initial Configuration and Azure Provider

What to do: Create the root Terraform configuration file to define the required Terraform version, establish the connection with the AzureRM provider, and create a foundational Resource Group.

Why do it: The azurerm provider block instructs Terraform on how to interact with the Azure Resource Manager API. In Azure, all resources must be logically grouped into a Resource Group. Deploying all interconnected services (Cosmos DB, Service Bus, Functions) into a single Resource Group makes it easier to manage their lifecycle, monitor costs, and delete the entire environment cleanly when no longer needed.

Example (main.tf):

terraform {
  required_version = ">= 1.3.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "main" {
  name     = "rg-eventdriven-architecture-prod"
  location = "East US"

  tags = {
    Environment = "Production"
    Project     = "EventDrivenArchitecture"
  }
}

Step 2: Creation of the Cosmos DB Account and Containers

What to do: Provision a Cosmos DB account with the SQL (Core) API, a database, and two containers: one for your application data and one "leases" container required to manage the state of the Change Feed.

Why do it: Cosmos DB is Azure's fully managed NoSQL database. Unlike AWS DynamoDB where Streams must be explicitly enabled, the Cosmos DB Change Feed is enabled by default. However, to consume it reliably with Azure Functions, you must create an auxiliary container called a "lease container." This container acts as a checkpoint system, allowing the producer function to keep track of which events it has already processed, ensuring reliable, ordered, and distributed event consumption.

Example (database.tf):

resource "azurerm_cosmosdb_account" "db_account" {
  name                = "cosmos-event-app-prod"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  offer_type          = "Standard"
  kind                = "GlobalDocumentDB"

  consistency_policy {
    consistency_level = "Session"
  }

  geo_location {
    location          = azurerm_resource_group.main.location
    failover_priority = 0
  }
}

resource "azurerm_cosmosdb_sql_database" "database" {
  name                = "ApplicationData"
  resource_group_name = azurerm_resource_group.main.name
  account_name        = azurerm_cosmosdb_account.db_account.name
}

# Main Data Container
resource "azurerm_cosmosdb_sql_container" "data_container" {
  name                  = "Items"
  resource_group_name   = azurerm_resource_group.main.name
  account_name          = azurerm_cosmosdb_account.db_account.name
  database_name         = azurerm_cosmosdb_sql_database.database.name
  partition_key_path    = "/id"
}

# Leases Container for Change Feed State
resource "azurerm_cosmosdb_sql_container" "leases_container" {
  name                  = "leases"
  resource_group_name   = azurerm_resource_group.main.name
  account_name          = azurerm_cosmosdb_account.db_account.name
  database_name         = azurerm_cosmosdb_sql_database.database.name
  partition_key_path    = "/id"
}

Step 3: Provisioning the Service Bus Topic and Subscription

What to do: Implement the Azure Service Bus Namespace, a Topic, and a Subscription. This maps perfectly to the AWS SNS/SQS fan-out pattern.

Why do it: A Service Bus Topic allows you to publish a single message that can be received by multiple independent Subscriptions. The Topic handles the routing (like SNS), while the Subscription acts as a dedicated, persistent queue for your consumer function (like SQS). This decouples your microservices and provides message retention in case the consumer function is temporarily unavailable.

Example (messaging.tf):

resource "azurerm_servicebus_namespace" "sb_namespace" {
  name                = "sb-event-routing-prod"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  sku                 = "Standard" # Required for Topics
}

resource "azurerm_servicebus_topic" "event_topic" {
  name         = "data-processing-topic"
  namespace_id = azurerm_servicebus_namespace.sb_namespace.id
}

resource "azurerm_servicebus_subscription" "event_subscription" {
  name               = "data-processing-subscription"
  topic_id           = azurerm_servicebus_topic.event_topic.id
  max_delivery_count = 10 # Send to Dead Letter Queue after 10 fails
}

Step 4: Deploying the Producer and Consumer Azure Functions

What to do: Create a Storage Account and an App Service Plan (Serverless Consumption tier) required to host Azure Functions. Then, deploy the two Linux Function Apps, ensuring that System Assigned Managed Identities are enabled for secure, passwordless authentication.

Why do it: Azure Functions require a backing Storage Account to manage triggers and function execution state. By enabling identity { type = "SystemAssigned" }, Azure automatically provisions a service principal in Microsoft Entra ID (formerly Azure AD) tied to the lifecycle of the Function. This is a crucial security best practice, allowing your functions to securely access Cosmos DB and Service Bus without hardcoding connection strings in your configuration.

Example (functions.tf):

resource "azurerm_storage_account" "func_storage" {
  name                     = "stfuncappdata123"
  resource_group_name      = azurerm_resource_group.main.name
  location                 = azurerm_resource_group.main.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_service_plan" "consumption_plan" {
  name                = "asp-serverless-plan"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  os_type             = "Linux"
  sku_name            = "Y1" # Dynamic Serverless Consumption
}

resource "azurerm_linux_function_app" "producer_func" {
  name                       = "func-cosmos-producer-prod"
  resource_group_name        = azurerm_resource_group.main.name
  location                   = azurerm_resource_group.main.location
  service_plan_id            = azurerm_service_plan.consumption_plan.id
  storage_account_name       = azurerm_storage_account.func_storage.name
  storage_account_access_key = azurerm_storage_account.func_storage.primary_access_key

  site_config {
    application_stack {
      node_version = "20"
    }
  }

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_linux_function_app" "consumer_func" {
  name                       = "func-sb-consumer-prod"
  resource_group_name        = azurerm_resource_group.main.name
  location                   = azurerm_resource_group.main.location
  service_plan_id            = azurerm_service_plan.consumption_plan.id
  storage_account_name       = azurerm_storage_account.func_storage.name
  storage_account_access_key = azurerm_storage_account.func_storage.primary_access_key

  site_config {
    application_stack {
      node_version = "20"
    }
  }

  identity {
    type = "SystemAssigned"
  }
}

Step 5: Applying Role-Based Access Control (RBAC) Triggers

What to do: Grant the specific IAM roles required for the Functions to interact with the database and messaging bus using Azure Role Assignments.

Why do it: While AWS uses IAM Policies attached to Event Source Mappings, Azure utilizes RBAC assignments. The Producer Function needs permission to read the Cosmos DB Change Feed and send messages to the Service Bus. The Consumer Function needs permission to receive messages from the Service Bus. Defining these natively in Terraform centralizes your security posture and enforces the principle of least privilege.

Example (rbac.tf):

data "azurerm_subscription" "current" {}

# Give Producer Function permission to Send to Service Bus
resource "azurerm_role_assignment" "producer_sb_sender" {
  scope                = azurerm_servicebus_namespace.sb_namespace.id
  role_definition_name = "Azure Service Bus Data Sender"
  principal_id         = azurerm_linux_function_app.producer_func.identity[0].principal_id
}

# Cosmos DB RBAC is custom, but for simplicity we grant Cosmos DB Account Reader 
# (Note: Data plane role assignments for Cosmos DB are often managed via azapi or CLI, 
# but this represents the structural intent)
resource "azurerm_role_assignment" "producer_cosmos_reader" {
  scope                = azurerm_cosmosdb_account.db_account.id
  role_definition_name = "DocumentDB Account Contributor" 
  principal_id         = azurerm_linux_function_app.producer_func.identity[0].principal_id
}

# Give Consumer Function permission to Read from Service Bus
resource "azurerm_role_assignment" "consumer_sb_receiver" {
  scope                = azurerm_servicebus_namespace.sb_namespace.id
  role_definition_name = "Azure Service Bus Data Receiver"
  principal_id         = azurerm_linux_function_app.consumer_func.identity[0].principal_id
}

4. Common Troubleshooting

Navigating cloud permissions and distributed architectures can present hurdles. Here are common issues and their solutions when working with this Azure stack:

Producer Function is not triggering on Cosmos DB changes: This usually indicates a problem with the leases container. Ensure that the leases container actually exists in your Cosmos DB database and that your Function's function.json bindings point to the correct database and collection names. Additionally, verify that your Function app settings contain the fully qualified URI to the Cosmos DB account if using Managed Identities.
Authorization rules or "Unauthorized" errors on Service Bus: If your consumer function fails to read the queue, ensure that the "Azure Service Bus Data Receiver" role has propagated. RBAC assignments in Azure can sometimes take 5-10 minutes to fully propagate globally. Avoid using older connection string bindings in your function code; ensure your code uses ServiceBusClient configured with DefaultAzureCredential().
Poison Messages and Infinite Loops: Just like AWS SQS, if your Consumer Function throws an unhandled exception, the Service Bus message is not marked as complete and will be retried. If this continues, it will cause an infinite loop. The max_delivery_count = 10 defined in Step 3 solves this by automatically moving the failing message to a dead-letter queue (DLQ) after 10 attempts, allowing your system to continue processing healthy messages.

5. Conclusion

In this tutorial, you transitioned architectural concepts across cloud providers, building an enterprise-grade, asynchronous infrastructure on Microsoft Azure. By utilizing Cosmos DB Change Feeds, Azure Functions, and Service Bus Topics/Subscriptions, you implemented a resilient Fan-Out event-driven system. Leveraging Terraform and System Assigned Managed Identities ensured that your infrastructure is not only automated and version-controlled but also inherently secure by eliminating the need for hardcoded credentials. As next steps, consider implementing Azure Application Insights for end-to-end distributed tracing across your functions, and placing your Storage Account behind an Azure Virtual Network (VNet) private endpoint for enhanced network isolation.

Serverless Data Processing: DynamoDB Streams, SNS, SQS, and Lambda

Cláudio Filipe Lima Rapôso — Tue, 28 Apr 2026 01:58:48 +0000

1. Introduction

An event-driven architecture utilizing DynamoDB Streams, AWS Lambda, Amazon SNS, and Amazon SQS is considered the gold standard for designing highly decoupled, scalable, and resilient systems in the cloud. In this tutorial, you will construct a complete, asynchronous data flow from scratch. The process begins when modifications occur in a DynamoDB table, which automatically triggers a data stream. This stream is continuously captured by a producer Lambda function, which processes the change and publishes a notification to an SNS topic. The SNS acts as a high-availability message router, performing a fan-out distribution of this notification to an SQS queue. Finally, a second consumer Lambda function reads the messages from the SQS queue to perform the final business logic processing. By the end of this guide, you will be able to deploy this robust infrastructure using Terraform alongside official AWS community modules. This approach is extremely useful because it ensures your infrastructure is version-controlled, highly maintainable, and inherently adopts security best practices, such as the principle of least privilege and fault tolerance.

2. Prerequisites

Before starting the infrastructure build, you must ensure that your development environment is properly prepared. You will need an active AWS account with administrative permissions to provision compute, messaging, database, and Identity and Access Management (IAM) resources. The AWS Command Line Interface (AWS CLI) must be installed and configured with your access credentials on your local machine. Additionally, Terraform (version 1.3.0 or higher) needs to be installed and globally accessible from your command terminal. Fundamental knowledge of Terraform's declarative syntax (HCL) and a basic understanding of how cloud messaging systems operate are highly recommended for the best learning experience. A modern code editor will greatly facilitate the creation and manipulation of the configuration files.

3. Step-by-Step

Step 1: Initial Configuration and AWS Provider

What to do: Create the root Terraform configuration file to define the required Terraform version and establish the connection with the AWS provider, explicitly specifying the region where the resources will be allocated.

Why do it: Terraform operates through plugins called providers. The AWS provider block instructs the tool on which API to use for authentication and resource provisioning. Declaring the region explicitly avoids unexpected behaviors and ensures that all interconnected services, such as DynamoDB and SNS, are deployed in the same geographical environment, significantly reducing the network latency of your architecture. Setting default tags also helps organize your resources for billing and management purposes.

Example (main.tf):

terraform {
  required_version = ">= 1.3.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
  default_tags {
    tags = {
      Environment = "Production"
      Project     = "EventDrivenArchitecture"
    }
  }
}

Step 2: Creation of the DynamoDB Table with Stream Enabled

What to do: Utilize the official DynamoDB Terraform module to create the main application table, ensuring that the Streams feature is activated with the view type configured to capture both new and old images of the modified items.

Why do it: The official module drastically simplifies the definition of partition keys and table attributes. Enabling the Stream is the foundational step of this entire architecture, as it acts as the initial trigger for our event pipeline. Setting the stream_view_type to NEW_AND_OLD_IMAGES guarantees that the consuming Lambda function will receive both the previous state of the modified item and its new state. This provides complete context, allowing your code to perform complex validations or delta comparisons during data processing.

Example (dynamodb.tf):

module "dynamodb_table" {
  source  = "terraform-aws-modules/dynamodb-table/aws"
  version = "~> 4.0"

  name           = "application-data-table"
  hash_key       = "id"
  billing_mode   = "PAY_PER_REQUEST"

  attributes = [
    {
      name = "id"
      type = "S"
    }
  ]

  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"
}

Step 3: Provisioning the SNS Topic and SQS Queue

What to do: Implement the official modules for Amazon SNS and Amazon SQS. You must configure the access policy of the SQS queue to securely allow the SNS topic to publish messages directly into it, and then create the subscription linking them together.

Why do it: Using SNS combined with SQS implements the robust "Fan-out" messaging pattern. SNS manages the immediate delivery and routing, while SQS ensures temporary storage, persistence, and message retention in case the consumer Lambda is temporarily unavailable. Utilizing the official modules abstracts away the complexity of manually crafting JSON IAM policies. By enabling create_queue_policy = true and providing the allowed ARNs, the SQS module automatically manages the IAM permissions required to receive payloads from SNS, keeping your security posture tight and error-free.

Example (messaging.tf):

module "sns_topic" {
  source  = "terraform-aws-modules/sns/aws"
  version = "~> 6.0"

  name = "data-processing-topic"
}

module "sqs_queue" {
  source  = "terraform-aws-modules/sqs/aws"
  version = "~> 4.0"

  name = "data-processing-queue"

  create_queue_policy = true
  queue_policy_statements = {
    sns_publish = {
      sid     = "AllowSNSPublish"
      actions = ["sqs:SendMessage"]
      principals = [
        {
          type        = "Service"
          identifiers = ["sns.amazonaws.com"]
        }
      ]
      conditions = [
        {
          test     = "ArnEquals"
          variable = "aws:SourceArn"
          values   = [module.sns_topic.topic_arn]
        }
      ]
    }
  }
}

resource "aws_sns_topic_subscription" "sns_to_sqs" {
  topic_arn = module.sns_topic.topic_arn
  protocol  = "sqs"
  endpoint  = module.sqs_queue.queue_arn
}

Step 4: Deploying the Producer and Consumer Lambda Functions

What to do: Configure two instances of the AWS Lambda module. The first function (Producer) requires strict permissions to publish messages to the SNS topic. The second function (Consumer) requires permissions to read and consume messages from the SQS queue.

Why do it: Lambda functions require highly granular and well-defined permissions (Policies) attached to their execution identity (Role). The Terraform Lambda module handles the packaging of your source code and the creation of these Roles simultaneously. By passing inline JSON policies directly into the module, we guarantee that each Lambda possesses strictly the minimum level of access necessary to perform its specific task. This prevents security vulnerabilities that arise from excessive or wildcard permissions.

Example (lambdas.tf):

module "lambda_producer" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.0"

  function_name = "dynamodb-stream-producer"
  handler       = "index.handler"
  runtime       = "nodejs20.x"
  source_path   = "../src/producer"

  environment_variables = {
    SNS_TOPIC_ARN = module.sns_topic.topic_arn
  }

  attach_policy_json = true
  policy_json = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["sns:Publish"]
        Effect   = "Allow"
        Resource = module.sns_topic.topic_arn
      }
    ]
  })
}

module "lambda_consumer" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.0"

  function_name = "sqs-message-consumer"
  handler       = "index.handler"
  runtime       = "nodejs20.x"
  source_path   = "../src/consumer"
}

Step 5: Mapping Event Sources to Triggers

What to do: Create the native aws_lambda_event_source_mapping resources to actively connect the DynamoDB Stream to the Producer Lambda, and the SQS Queue to the Consumer Lambda. You must also attach the AWS managed execution policies to the respective Lambda roles.

Why do it: While the Lambda module creates the compute resource, the "event source mapping" is the internal AWS polling service that actively reads the Stream or the Queue and invokes your function. This mapping mechanism requires specific operational policies (AWSLambdaDynamoDBExecutionRole and AWSLambdaSQSQueueExecutionRole) to function correctly. Integrating these components declaratively via Terraform centralizes the control of your end-to-end event flow, ensuring that infrastructure changes are applied consistently.

Example (triggers.tf):

# Attach Stream reading permissions to the Producer Lambda
resource "aws_iam_role_policy_attachment" "producer_dynamodb_policy" {
  role       = module.lambda_producer.lambda_role_name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaDynamoDBExecutionRole"
}

# Integrate DynamoDB Stream with Producer Lambda
resource "aws_lambda_event_source_mapping" "dynamodb_trigger" {
  event_source_arn  = module.dynamodb_table.dynamodb_table_stream_arn
  function_name     = module.lambda_producer.lambda_function_arn
  starting_position = "LATEST"

  depends_on = [aws_iam_role_policy_attachment.producer_dynamodb_policy]
}

# Attach SQS reading permissions to the Consumer Lambda
resource "aws_iam_role_policy_attachment" "consumer_sqs_policy" {
  role       = module.lambda_consumer.lambda_role_name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole"
}

# Integrate SQS with Consumer Lambda
resource "aws_lambda_event_source_mapping" "sqs_trigger" {
  event_source_arn = module.sqs_queue.queue_arn
  function_name    = module.lambda_consumer.lambda_function_arn
  batch_size       = 10

  depends_on = [aws_iam_role_policy_attachment.consumer_sqs_policy]
}

4. Common Troubleshooting

Even when following best practices, distributed systems can present challenges regarding network configuration and IAM permissions. Below are the most common problems you might encounter and how to efficiently solve them:

Producer Lambda is not invoked by DynamoDB: This almost always occurs due to IAM permission failures. The execution Role of the Lambda must possess the dynamodb:GetRecords, dynamodb:GetShardIterator, dynamodb:DescribeStream, and dynamodb:ListStreams permissions. Verify that the AWSLambdaDynamoDBExecutionRole policy was successfully attached in your Terraform state.
SQS Queue does not receive messages from the SNS Topic: The SNS subscription protocol dictates that the endpoint (the SQS queue) must explicitly authorize the reception of messages. Confirm that the SQS access policy contains a statement allowing the sqs:SendMessage action, with the principal set to the sns.amazonaws.com service, and a restrictive condition matching the exact ARN of your SNS topic. This prevents unauthorized cross-account access.
Duplicate processing or infinite retries in the Consumer Lambda: If your code within the consumer Lambda fails and throws an unhandled exception, the message will return to the SQS queue after the visibility timeout expires and will be reprocessed, creating an infinite loop of errors. To solve this, wrap your logic in try-catch blocks and implement a Dead Letter Queue (DLQ). Configure the maxReceiveCount property on your main SQS queue to safely route "poison pill" messages to the DLQ after a limited number of failed attempts.

5. Conclusion

In this tutorial, you have comprehensively understood and practically applied the creation of a highly scalable, asynchronous architecture on AWS. The combination of DynamoDB Streams, Lambda functions, SNS, and SQS forms a resilient, event-driven system where each isolated component performs its specific duty flawlessly. Utilizing Terraform with official modules ensured agility, security, and exceptional structural code organization. As next steps to evolve this ecosystem, it is highly recommended to implement KMS (Key Management Service) encryption for your SNS topics and SQS queues, build CI/CD pipelines to automate the provisioning process, and create specialized CloudWatch dashboards to monitor message age and function error metrics.

Practical Guide: Building an Active-Active Multicloud Cell-Based Architecture

Cláudio Filipe Lima Rapôso — Thu, 23 Apr 2026 03:00:00 +0000

1. Introduction

A multicloud cell-based architecture represents the pinnacle of fault isolation and vendor neutrality. By distributing autonomous "cells" across different cloud provider, such as placing Cell Alpha in AWS and Cell Beta in Azure, you eliminate the risk of a single cloud provider's regional or global outage taking down your entire application. At the end of this tutorial, you will understand how to provision a unified control plane that intelligently routes tenant traffic to entirely isolated data planes residing in disparate cloud ecosystems.

This architecture is highly valuable for organizations with extreme availability requirements and strict regulatory compliance mandates. It prevents vendor lock-in by enforcing an agnostic ingress layer and identical compute/state contracts across clouds. Building this requires mature infrastructure as code practices and a deep understanding of decoupled domain boundaries.

2. Prerequisites

To execute this architectural pattern, you must have the following tools and access configured:

Active accounts on both Amazon Web Services (AWS) and Microsoft Azure, with administrative credentials.
Terraform (version 1.0+) installed locally, with both hashicorp/aws and hashicorp/azurerm providers authenticated.
Python (version 3.11+) for writing the agnostic edge routing logic.
A registered domain name and access to a neutral Edge DNS provider (e.g., Cloudflare) to act as the global entry point.
Understanding of Domain-Driven Design (DDD) to ensure cell workloads are completely bounded and stateless.

3. Step-by-Step

A multicloud cellular architecture requires a strict separation between the "Control Plane" (which knows where tenants live) and the "Data Plane" (where the actual compute and storage happen). The sequence diagram below illustrates how an edge router evaluates the request and proxies it to the respective cloud provider.

3.1 Configuring the Multicloud Terraform Environment

What to do: Create a root Terraform configuration that initializes both the AWS and Azure providers simultaneously.
Why do it: To manage a true multicloud infrastructure, your CI/CD pipeline must be able to orchestrate state across both environments from a single source of truth, ensuring that the infrastructure contracts remain identical.

Example:
Create your main.tf file to authenticate both providers:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
  alias  = "primary"
}

provider "azurerm" {
  features {}
  subscription_id = var.azure_subscription_id
}

variable "azure_subscription_id" {
  type        = string
  description = "Azure Subscription ID for the multicloud landing zone"
}

3.2 Provisioning the AWS Cell (Cell Alpha)

What to do: Deploy the AWS-specific data plane, consisting of an API Gateway, Lambda, and DynamoDB.
Why do it: This establishes your first isolated failure domain. The AWS cell acts as a self-contained unit capable of serving its assigned tenants without any dependency on the central control plane or the Azure cell.

Example:
Append the AWS cell definition to your main.tf:

# AWS Cell Infrastructure (Simplified)
resource "aws_dynamodb_table" "aws_cell_state" {
  provider       = aws.primary
  name           = "multicloud-state-aws-alpha"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "id"

  attribute {
    name = "id"
    type = "S"
  }
}

resource "aws_lambda_function" "aws_cell_worker" {
  provider      = aws.primary
  filename      = "aws_worker.zip"
  function_name = "aws-worker-alpha"
  role          = aws_iam_role.aws_exec_role.arn
  handler       = "worker.handler"
  runtime       = "python3.11"
}

resource "aws_lambda_function_url" "aws_ingress" {
  provider           = aws.primary
  function_name      = aws_lambda_function.aws_cell_worker.function_name
  authorization_type = "NONE" # In production, restrict to Edge Router IPs/Tokens
}

output "aws_cell_endpoint" {
  value = aws_lambda_function_url.aws_ingress.function_url
}

3.3 Provisioning the Azure Cell (Cell Beta)

What to do: Deploy the Azure-specific data plane, consisting of an Azure Function and Cosmos DB.
Why do it: This creates the second failure domain in a completely separate physical network and control plane. If AWS experiences a major outage, the tenants assigned to the Azure cell experience zero degradation.

Example:
Append the Azure cell definition to your main.tf:

# Azure Cell Infrastructure (Simplified)
resource "azurerm_resource_group" "az_cell_rg" {
  name     = "rg-multicloud-beta"
  location = "East US"
}

resource "azurerm_cosmosdb_account" "az_cell_db" {
  name                = "db-multicloud-az-beta"
  location            = azurerm_resource_group.az_cell_rg.location
  resource_group_name = azurerm_resource_group.az_cell_rg.name
  offer_type          = "Standard"
  kind                = "GlobalDocumentDB"

  geo_location {
    location          = azurerm_resource_group.az_cell_rg.location
    failover_priority = 0
  }
  consistency_policy {
    consistency_level = "Session"
  }
}

resource "azurerm_storage_account" "az_storage" {
  name                     = "stmulticloudbeta"
  resource_group_name      = azurerm_resource_group.az_cell_rg.name
  location                 = azurerm_resource_group.az_cell_rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_service_plan" "az_plan" {
  name                = "plan-multicloud-beta"
  location            = azurerm_resource_group.az_cell_rg.location
  resource_group_name = azurerm_resource_group.az_cell_rg.name
  os_type             = "Linux"
  sku_name            = "Y1"
}

resource "azurerm_linux_function_app" "az_cell_worker" {
  name                       = "func-az-worker-beta"
  location                   = azurerm_resource_group.az_cell_rg.location
  resource_group_name        = azurerm_resource_group.az_cell_rg.name
  service_plan_id            = azurerm_service_plan.az_plan.id
  storage_account_name       = azurerm_storage_account.az_storage.name
  storage_account_access_key = azurerm_storage_account.az_storage.primary_access_key

  site_config {
    application_stack {
      python_version = "3.11"
    }
  }
}

output "az_cell_endpoint" {
  value = "https://${azurerm_linux_function_app.az_cell_worker.default_hostname}/api/process"
}

3.4 Developing the Agnostic Edge Router (Python)

What to do: Write the Python routing logic that will be deployed to a neutral edge layer (like Cloudflare Workers via Pyodide, or a dedicated edge Kubernetes cluster).
Why do it: You cannot host the global router inside AWS or Azure. If you host it in AWS and AWS goes down, your clients cannot reach the Azure cell because the router itself is dead. The routing layer must be cloud-agnostic.

Example:
Create global_router.py. This script intercepts the payload, queries the registry, and proxies the request across the internet backbone to the appropriate cloud.

import json
import urllib.request
import os

# The global registry maps tenant IDs to specific cloud cell endpoints.
# In production, this data is fetched from a distributed edge database.
GLOBAL_REGISTRY = {
    "T-100": {
        "provider": "aws",
        "url": os.getenv("AWS_CELL_ENDPOINT", "https://aws-api-mock.com/process")
    },
    "T-200": {
        "provider": "azure",
        "url": os.getenv("AZ_CELL_ENDPOINT", "https://az-api-mock.com/api/process")
    }
}

def route_request(request_body: str) -> dict:
    try:
        payload = json.loads(request_body)
        tenant_id = payload.get("tenant_id")

        if not tenant_id:
            return {"status": 400, "body": "Missing tenant_id"}

        # 1. Lookup tenant location in the registry
        cell_info = GLOBAL_REGISTRY.get(tenant_id)
        if not cell_info:
            return {"status": 404, "body": "Tenant not allocated to any cell"}

        target_url = cell_info["url"]
        provider = cell_info["provider"]

        # 2. Proxy request to the specific cloud provider
        req = urllib.request.Request(
            target_url,
            data=request_body.encode('utf-8'),
            headers={'Content-Type': 'application/json', 'X-Multicloud-Auth': 'EdgeToken123'}
        )

        with urllib.request.urlopen(req, timeout=5) as response:
            cell_response = response.read().decode('utf-8')

        # 3. Return payload to client with multicloud diagnostic headers
        return {
            "status": 200,
            "headers": {
                "X-Processed-By-Cloud": provider
            },
            "body": cell_response
        }

    except Exception as e:
        return {"status": 500, "body": f"Edge Routing Failure: {str(e)}"}

4. Common Troubleshooting

Operating a multicloud cellular architecture introduces extreme networking and data synchronization complexities. Be prepared to address the following challenges:

State Fragmentation and Global Reporting:
- Problem: Your business intelligence team cannot run global queries because data is split between AWS DynamoDB and Azure Cosmos DB.
- Solution: Do not attempt real-time federated queries across clouds. Implement a unified Data Lake (e.g., Snowflake or Databricks). Configure both the AWS Lambda and the Azure Function to asynchronously stream their state changes (Event Sourcing) into this central analytical repository.
Cross-Cloud Latency at the Routing Layer:
- Problem: If the neutral Edge Router has to query a database in AWS to find out that a tenant belongs in Azure, the latency overhead becomes unacceptable.
- Solution: The Tenant Registry must be cached at the edge. Utilize technologies like Cloudflare KV, Fastly Compute dictionaries, or Redis Enterprise Active-Active to ensure the routing metadata is geographically local to the Edge Router executing the Python script.
CI/CD Configuration Drift:
- Problem: Over time, developers add features to the AWS cell but forget to implement the exact Azure equivalent, breaking the "identical cell" contract.
- Solution: Enforce strict Hexagonal Architecture within your Python applications. The core domain logic must be a pure Python package shared across both clouds. The only differing code should be the specific I/O adapters (DynamoDB vs. Cosmos DB connectors).

5. Conclusion

By implementing a multicloud cell-based architecture, you have constructed the ultimate defense against cloud-provider lock-in and catastrophic outages. We used Terraform to simultaneously orchestrate isolated data planes in AWS and Azure, and developed an agnostic Python edge router to direct traffic based on tenant identity.

This model requires significant engineering maturity, but it guarantees that your infrastructure can scale infinitely horizontally while treating cloud providers as interchangeable utility commodities. As a next step, focus on automating the cross-cloud tenant migration process, allowing you to evacuate tenants from AWS to Azure in real-time if telemetry indicates a degradation in a specific provider's region.

Event-Driven Architecture on Azure vs AWS: Service Bus vs SNS/SQS

Cláudio Filipe Lima Rapôso — Tue, 21 Apr 2026 19:59:01 +0000

Your OrderService does six things when a customer clicks Place Order. It writes to the orders table, reserves inventory, charges the card, enqueues the shipping label, emails the receipt, and logs the analytics event. It does all of that inside one HTTP handler, in one transaction, on one box. When the payment gateway hiccups, the order fails. When the email provider rate-limits, the order fails. When inventory is slow, the order fails. The monolith tied six independent failure domains to one shared fate.

This article rebuilds that pipeline as an event-driven system on both Azure and AWS, and walks through where they genuinely differ - not where they superficially look the same. The reference scenario is e-commerce order processing with fan-out to Inventory, Payment, and Notification. Audience: intermediate-to-senior engineers who already read the vendor landing pages and want the parts those pages leave out.

The pattern before the products

Before naming any cloud service, fix the shape. What you want is publish/subscribe fan-out with durable per-consumer queues and per-consumer dead-letter queues. The producer emits one logical event - OrderPlaced - to a topic. The topic delivers a copy to one durable queue per consumer. Each consumer drains its own queue at its own pace, retries on its own schedule, and when it gives up, the message lands in its own DLQ - not a shared one.

That last part matters. A shared DLQ means a poisoned inventory message blocks the payment team's on-call from seeing their own poison. One queue per consumer, one DLQ per consumer, one retry budget per consumer. The blast radius of any single bad message is exactly one bounded context.

With the shape fixed, we can map it onto two clouds - one concern at a time.

Side by side, one concern at a time

Topic and subscription model

On AWS the topic and the queues are separate primitives. SNS publishes; SQS stores. You wire them together with a topic subscription and a queue policy. The queue holds your messages; the topic just fans them out. Two resource types per consumer.

On Azure Service Bus collapses that into one resource graph. A namespace contains a topic, and each consumer is a subscription on that topic. The subscription has a virtual queue behind it; you do not manage a separate queue resource. Fewer moving parts at the IaC layer, but less separation of concern - the topic and its subscribers share a lifecycle and a billing unit.

Queue semantics and DLQ

Both brokers are at-least-once. Consumers will see duplicates. No marketing slide changes that.

AWS pairs each SQS queue with an explicit DLQ via a redrive policy. maxReceiveCount is the threshold; the main queue holds the in-flight message until the consumer explicitly deletes it, controlled by visibility_timeout_seconds. That visibility timeout must exceed P99 handler latency, with margin. Set it too low and the broker redelivers while the first handler is still working - you get two concurrent handlers racing, and idempotency becomes load-bearing in a way you probably did not test.

Azure Service Bus bakes the DLQ into every subscription as $DeadLetterQueue. maxDeliveryCount plays the same role as maxReceiveCount. Service Bus also dead-letters on two classes of failure SQS does not know about: message TTL expiry and filter-evaluation exceptions. Those two extra DLQ triggers are real operational wins - expired or malformed messages do not vanish into metrics.

One common claim worth pushing back on: Service Bus gives exactly-once delivery. It does not. What it gives is duplicate detection within a bounded window (up to seven days) on a per-MessageId basis. That is a broker-side helper, not a semantic guarantee. Consumers must still be idempotent. Same story on SQS FIFO and its five-minute content-based dedup window.

Identity and data-plane auth

On AWS, consumers assume an IAM role from Lambda, ECS, or EKS. No access keys, no user credentials in config. The queue policy restricts senders to a specific topic ARN via an aws:SourceArn condition - without it, any SNS topic in your account can write to your queue. That is the classic confused-deputy footgun, and leaving the condition off is one of the most common rejection triggers in a real review.

On Azure, the equivalent of "no long-lived keys" is disableLocalAuth: true on the namespace, which kills SAS authentication entirely. All auth goes through AAD and Managed Identity. The right role is Service Bus Data Receiver scoped per subscription, not namespace-wide. Writers get Service Bus Data Sender on the topic. Scoping at the subscription level means a compromised notification consumer cannot read payment events - lateral movement is bounded by the RBAC scope.

Ordering

Both platforms can do ordering. Neither should do ordering by default.

On AWS, ordering means FIFO queues keyed by MessageGroupId. The cap is 300 messages/sec (3,000 with batching) per queue. That is a hard ceiling, not a soft throttle.

On Azure, ordering means SessionId on messages and requiresSession: true on the subscription. Order is preserved per session. Throughput is fine on Standard; partitioning on Premium pushes it higher. The cost is that session-pinning serialises a subscription - a slow consumer on one session stalls messages in that session until the lock clears.

If the domain does not require strict order, do not enable it. FIFO is a business decision, not an architectural default.

The trade-off matrix

This is the center of the article, not the closer. If you remember one thing, remember this table.

Dimension	AWS SNS + SQS	Azure Service Bus (Standard / Premium)
Primitive model	Topic (SNS) fans out to separate SQS queues - two resource types per consumer	Single namespace → topic → subscription - one resource graph
Cost model	Pay-per-request on both SNS publishes and SQS requests; no idle cost	Standard: pay-per-million operations + namespace hourly. Premium: fixed messaging units (predictable floor ≈ $670/MU/month at time of writing; verify current SKU pricing, it drifts)
Message ordering	Standard: none. FIFO: strict ordering by `MessageGroupId`, capped at 300 msgs/s (3,000 with batching)	Standard: ordering within a session (`SessionId`). Premium: same, higher throughput, partitioning supported
Delivery semantics	At-least-once. FIFO adds content-based dedup in a 5-minute window	At-least-once. PeekLock + duplicate detection up to 7 days. Broker-side helper; still needs idempotent consumers
Max message size	256 KB on both SNS and SQS. Workaround: claim-check via S3 + SQS Extended Client	Standard: 256 KB. Premium: 100 MB native
DLQ handling	Explicit SQS queue + redrive policy. `maxReceiveCount` threshold. Delivery-failure DLQ only	Implicit `$DeadLetterQueue` per subscription. `maxDeliveryCount` threshold. Also DLQs on TTL expiry and filter-eval errors
Filtering	SNS filter policies: JSON attribute match at subscription time	SQL filter and correlation filter per subscription - richer expression model
Ops surface	CloudWatch: `ApproximateNumberOfMessages`, `ApproximateAgeOfOldestMessage`. Alarm on DLQ depth > 0	Azure Monitor: `ActiveMessages`, `DeadletteredMessages`. Alarm on `DeadletteredMessages > 0`
Identity	IAM roles assumed by Lambda/ECS/EKS; SSE-KMS	AAD + Managed Identity; `disableLocalAuth=true` kills SAS
Network isolation	VPC Endpoints (Interface for SNS, Interface/Gateway for SQS)	Private Endpoint (Premium SKU for full VNet integration)
Throughput	SNS: millions of msgs/s. SQS standard: effectively unlimited. SQS FIFO: 300–3,000 msgs/s per queue	Standard: ~2,000 msgs/s per namespace as a working guideline. Premium: scales with messaging units (~1,000 msgs/s per MU)

One-line heuristic: If the workload demands >256 KB messages, strong ordering at high throughput, or VNet isolation with ordering, Service Bus Premium earns its cost. Otherwise - massive fan-out, idempotent consumers, per-request billing - SNS + SQS wins. Standard Service Bus is the baseline; Premium is an upgrade you justify, not a default.

The IaC - AWS

Production-grade Terraform. FinOps tags, SSE, redrive policies, and an aws:SourceArn condition on the queue policy. Keep all of it - each line carries a security or reliability property the cluster depends on.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40"
    }
  }
}

variable "project"        { type = string }
variable "environment"    { type = string }               # dev | stg | prd
variable "aws_region"     { type = string  default = "us-east-1" }
variable "consumers" {
  description = "Logical consumers that subscribe to the OrderPlaced topic."
  type        = set(string)
  default     = ["inventory", "payment", "notification"]
}
variable "max_receive_count" { type = number default = 5 } # redrive threshold

locals {
  name_prefix = "${var.project}-${var.environment}"
  tags = {
    Project     = var.project
    Environment = var.environment
    Workload    = "eda-orders"
    CostCenter  = "platform-events"
    ManagedBy   = "terraform"
  }
}

# --- Topic ----------------------------------------------------------------
resource "aws_sns_topic" "orders_placed" {
  name              = "${local.name_prefix}-orders-placed"
  kms_master_key_id = "alias/aws/sns"          # SSE at rest with AWS-managed CMK; swap to customer CMK for stricter tenants
  tags              = local.tags
}

# --- Queues + DLQs per consumer ------------------------------------------
resource "aws_sqs_queue" "dlq" {
  for_each                   = var.consumers
  name                       = "${local.name_prefix}-${each.key}-dlq"
  message_retention_seconds  = 1209600            # 14 days - max allowed, buys ops time
  kms_master_key_id          = "alias/aws/sqs"
  tags                       = merge(local.tags, { Role = "dlq", Consumer = each.key })
}

resource "aws_sqs_queue" "main" {
  for_each                   = var.consumers
  name                       = "${local.name_prefix}-${each.key}"
  visibility_timeout_seconds = 60                 # must exceed consumer max processing time
  message_retention_seconds  = 345600             # 4 days
  receive_wait_time_seconds  = 20                 # long polling - reduces empty-receive cost
  kms_master_key_id          = "alias/aws/sqs"
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq[each.key].arn
    maxReceiveCount     = var.max_receive_count
  })
  tags = merge(local.tags, { Role = "main", Consumer = each.key })
}

# --- Allow SNS to write to SQS -------------------------------------------
data "aws_iam_policy_document" "sns_to_sqs" {
  for_each = var.consumers
  statement {
    sid     = "AllowSNSDeliver"
    effect  = "Allow"
    actions = ["sqs:SendMessage"]
    principals {
      type        = "Service"
      identifiers = ["sns.amazonaws.com"]
    }
    resources = [aws_sqs_queue.main[each.key].arn]
    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_sns_topic.orders_placed.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "main" {
  for_each  = var.consumers
  queue_url = aws_sqs_queue.main[each.key].id
  policy    = data.aws_iam_policy_document.sns_to_sqs[each.key].json
}

resource "aws_sns_topic_subscription" "consumer" {
  for_each             = var.consumers
  topic_arn            = aws_sns_topic.orders_placed.arn
  protocol             = "sqs"
  endpoint             = aws_sqs_queue.main[each.key].arn
  raw_message_delivery = true                     # consumers parse the raw event, not the SNS envelope
}

# --- Consumer IAM role template (least-privilege) ------------------------
data "aws_iam_policy_document" "consumer_assume" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com", "ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "consumer" {
  for_each           = var.consumers
  name               = "${local.name_prefix}-${each.key}-consumer"
  assume_role_policy = data.aws_iam_policy_document.consumer_assume.json
  tags               = merge(local.tags, { Consumer = each.key })
}

data "aws_iam_policy_document" "consumer_sqs" {
  for_each = var.consumers
  statement {
    actions = [
      "sqs:ReceiveMessage",
      "sqs:DeleteMessage",
      "sqs:GetQueueAttributes",
      "sqs:ChangeMessageVisibility",
    ]
    resources = [aws_sqs_queue.main[each.key].arn]
  }
}

resource "aws_iam_role_policy" "consumer_sqs" {
  for_each = var.consumers
  role     = aws_iam_role.consumer[each.key].id
  policy   = data.aws_iam_policy_document.consumer_sqs[each.key].json
}

output "topic_arn" { value = aws_sns_topic.orders_placed.arn }
output "queues"    { value = { for k, q in aws_sqs_queue.main : k => q.arn } }
output "dlqs"      { value = { for k, q in aws_sqs_queue.dlq  : k => q.arn } }

The IaC - Azure

Same scenario, same tags, same maxDeliveryCount = 5. Note disableLocalAuth: true on the namespace and per-subscription RBAC at the bottom.

// Deployment scope: resourceGroup
targetScope = 'resourceGroup'

@description('Project short code, e.g. osecloud')
param project string

@allowed([ 'dev', 'stg', 'prd' ])
param environment string

@description('Azure region.')
param location string = resourceGroup().location

@description('Logical consumers that subscribe to the OrderPlaced topic.')
param consumers array = [ 'inventory', 'payment', 'notification' ]

@description('Consumer identity object IDs (managed identities that will receive Data Receiver role). Key must match a name in `consumers`.')
param consumerPrincipalIds object = {}

@description('Service Bus SKU. Use Premium when you need ordering at scale, VNet integration, or >1MB messages.')
@allowed([ 'Standard', 'Premium' ])
param skuName string = 'Standard'

var namePrefix = toLower('${project}-${environment}')
var tags = {
  project:     project
  environment: environment
  workload:    'eda-orders'
  costCenter:  'platform-events'
  managedBy:   'bicep'
}

// --- Namespace ------------------------------------------------------------
resource sbNamespace 'Microsoft.ServiceBus/namespaces@2022-10-01-preview' = {
  name:     '${namePrefix}-sb'
  location: location
  sku: {
    name: skuName
    tier: skuName
  }
  properties: {
    minimumTlsVersion: '1.2'
    publicNetworkAccess: 'Enabled' // set to 'Disabled' + private endpoint in prd
    disableLocalAuth: true         // force AAD/Managed Identity - no SAS keys
  }
  tags: tags
}

// --- Topic ----------------------------------------------------------------
resource topic 'Microsoft.ServiceBus/namespaces/topics@2022-10-01-preview' = {
  parent: sbNamespace
  name: 'orders-placed'
  properties: {
    defaultMessageTimeToLive: 'P14D'
    enableBatchedOperations: true
    supportOrdering: true          // preserves order within a session (partition) - only honoured with session-enabled subscriptions
  }
}

// --- Subscriptions + DLQ (DLQ is implicit per subscription) --------------
resource subs 'Microsoft.ServiceBus/namespaces/topics/subscriptions@2022-10-01-preview' = [for name in consumers: {
  parent: topic
  name:   '${name}-sub'
  properties: {
    deadLetteringOnMessageExpiration:    true
    deadLetteringOnFilterEvaluationExceptions: true
    maxDeliveryCount: 5             // redrive threshold → moves to $DeadLetterQueue
    lockDuration:     'PT1M'        // matches visibility-timeout in AWS terms
    defaultMessageTimeToLive: 'P4D'
    requiresSession: false          // set true for FIFO-per-session guarantees
  }
}]

// --- RBAC: Azure Service Bus Data Receiver on each subscription ----------
var dataReceiverRoleId = '4f6d3b9b-027b-4f4c-9142-0e5a2a2247e0' // Service Bus Data Receiver

resource rbacReceiver 'Microsoft.Authorization/roleAssignments@2022-04-01' = [for (name, i) in consumers: if (contains(consumerPrincipalIds, name)) {
  name:  guid(subs[i].id, consumerPrincipalIds[name], dataReceiverRoleId)
  scope: subs[i]
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', dataReceiverRoleId)
    principalId:      consumerPrincipalIds[name]
    principalType:    'ServicePrincipal'
  }
}]

output namespaceId       string = sbNamespace.id
output topicId           string = topic.id
output subscriptionNames array  = [for (name, i) in consumers: subs[i].name]

One caveat on the API version: 2022-10-01-preview is still labelled preview at the time of writing. If your platform team forbids preview API versions in production, pin to the latest stable GA release and re-test disableLocalAuth - its behaviour has shifted across API versions.

Seven architectural constraints

Treat these as acceptance criteria for any EDA pipeline you ship on either cloud. They are a punch list, not a wish list.

At-least-once is the default on both sides. Exactly-once is a consumer property - idempotent handlers plus a dedup store - not a broker guarantee. Broker-side windows (SQS FIFO 5 min, Service Bus duplicate detection up to 7 days) narrow the problem; they do not eliminate it.
Visibility timeout and lock duration must exceed P99 handler latency. If the broker redelivers while the first handler is still working, you double-process. Measure P99 under load, add headroom, and alert when handler duration approaches the timeout.
Ordering is a bet you pay for. FIFO caps AWS throughput at 300–3,000 msgs/s; sessions serialise Azure subscriptions. Turn it on only when the domain demands ordering - never as a safety blanket.
DLQs are not a graveyard. They need alerts (DLQ depth > 0 pages the on-call) and a documented replay procedure - SQS redrive or Service Bus $DeadLetterQueue receive-and-resubmit. A DLQ without a replay runbook is a silent leak.
Large messages are an anti-pattern. >256 KB on AWS implies claim-check via S3. Premium Service Bus supports 100 MB, but transport cost and consumer memory pressure still argue for claim-check at that size.
Tag every resource with project, environment, workload, costCenter, managedBy. Without those tags, FinOps cannot attribute spend and the platform team cannot enforce lifecycle policies. The snippets above carry the full set; do not strip them.
No SAS keys, no IAM user keys. Managed Identity on Azure, IAM roles on AWS. disableLocalAuth: true on the Service Bus namespace, aws:SourceArn condition on every SQS queue policy. Anything else is a long-lived credential waiting to leak.

The choice between SNS/SQS and Service Bus is rarely binary. Start on Standard Service Bus or SNS + SQS. Move to Premium or FIFO only when a constraint above - size, ordering, isolation - forces you there.

Practical Guide: Building a Cell-Based Architecture on Azure with Terraform and Python

Cláudio Filipe Lima Rapôso — Tue, 21 Apr 2026 15:15:00 +0000

1. Introduction

As cloud applications scale to serve global audiences, relying on a single centralized infrastructure stack introduces critical vulnerabilities. A localized failure or a noisy neighbor can trigger a systemic outage. Cell-Based Architecture mitigates this by partitioning the system into isolated, identical, and self-contained units called "cells." By routing specific tenants or users to dedicated cells, you constrain the blast radius of any degradation strictly to that cell, preserving the availability of the rest of the system.

By the end of this tutorial, you will be able to design and provision a cellular architecture on Microsoft Azure. We will utilize Azure Front Door as the global ingress, Azure Functions (Python) for dynamic traffic routing and compute, and Azure Cosmos DB for isolated state management. Mastering this pattern on Azure not only hardens your current workloads but also reinforces foundational multicloud principles, as the concepts of decoupled routing and state isolation are directly transferable across different cloud providers in a unified enterprise landscape.

2. Prerequisites

To execute the configurations and code in this tutorial, ensure you have the following tools and access levels:

An active Microsoft Azure subscription with Owner or Contributor permissions to create Resource Groups, Cosmos DB accounts, Azure Functions, and Azure Front Door.
Terraform CLI (version 1.0 or higher) installed locally for provisioning the Infrastructure as Code (IaC).
Python (version 3.9 or higher) installed locally, along with the Azure Functions Core Tools for local development and packaging.
Azure CLI installed and authenticated (az login) on your local environment.
Fundamental understanding of partitioning concepts and Terraform HCL syntax.

3. Step-by-Step

Before diving into the infrastructure code, let's visualize the execution flow. The sequence diagram below details how a global request is securely intercepted, evaluated, and forwarded to a strictly isolated cellular stack within Azure.

3.1 Defining the Cell Blueprint (Terraform Module)

What to do: Create a reusable Terraform module representing a single, isolated "Cell." This includes a dedicated Storage Account, Application Service Plan, Azure Function (Worker), and a Cosmos DB database.

Why do it: The primary rule of cellular architecture is absolute consistency across environments. By encapsulating the infrastructure in a Terraform module, you ensure that every generated cell is an exact replica, preventing configuration drift and simplifying horizontal scaling.

Example:
Create a directory named modules/cell and add a main.tf file.

# modules/cell/main.tf
variable "location" { type = string }
variable "resource_group_name" { type = string }
variable "cell_id" { type = string }

resource "azurerm_cosmosdb_account" "cell_db" {
  name                = "cosmos-cell-${var.cell_id}"
  location            = var.location
  resource_group_name = var.resource_group_name
  offer_type          = "Standard"
  kind                = "GlobalDocumentDB"

  consistency_policy {
    consistency_level = "Session"
  }

  geo_location {
    location          = var.location
    failover_priority = 0
  }
}

resource "azurerm_cosmosdb_sql_database" "cell_sqldb" {
  name                = "app-state"
  resource_group_name = var.resource_group_name
  account_name        = azurerm_cosmosdb_account.cell_db.name
}

resource "azurerm_storage_account" "cell_storage" {
  name                     = "stcell${var.cell_id}"
  resource_group_name      = var.resource_group_name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_service_plan" "cell_plan" {
  name                = "plan-cell-${var.cell_id}"
  location            = var.location
  resource_group_name = var.resource_group_name
  os_type             = "Linux"
  sku_name            = "Y1" # Consumption plan
}

resource "azurerm_linux_function_app" "cell_worker" {
  name                       = "func-worker-${var.cell_id}"
  location                   = var.location
  resource_group_name        = var.resource_group_name
  service_plan_id            = azurerm_service_plan.cell_plan.id
  storage_account_name       = azurerm_storage_account.cell_storage.name
  storage_account_access_key = azurerm_storage_account.cell_storage.primary_access_key

  site_config {
    application_stack {
      python_version = "3.11"
    }
  }

  app_settings = {
    "CELL_ID"             = var.cell_id
    "COSMOS_DB_ENDPOINT"  = azurerm_cosmosdb_account.cell_db.endpoint
  }
}

output "function_default_hostname" {
  value = azurerm_linux_function_app.cell_worker.default_hostname
}

3.2 Stamping Out Multiple Cells

What to do: In your root Terraform configuration, define the base resource group and iterate over a collection of identifiers to deploy multiple cells simultaneously.

Why do it: This translates the theoretical module into physical, isolated environments. Using a for_each loop allows you to effortlessly scale from two cells to fifty cells simply by updating a local variable, abstracting the complexity of managing parallel infrastructures.

Example:
In your root directory, create main.tf.

# main.tf
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "rg-cellular-architecture"
  location = "East US"
}

locals {
  cells = ["alpha", "beta"]
}

module "isolated_cells" {
  source   = "./modules/cell"
  for_each = toset(local.cells)

  cell_id             = each.key
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

3.3 Developing the Cell Router in Python

What to do: Write the Azure Function code in Python that will act as the global ingress router. It must inspect incoming HTTP requests, determine the appropriate tenant mapping, and proxy the request to the correct cell.

Why do it: The router abstracts the internal topology from the client. Applications interact with a single API endpoint, oblivious to the fact that their request is being routed to cell-alpha or cell-beta. This dynamic mapping is what allows you to perform live migrations of tenants between cells to balance load without downtime.

Example:
Create the Python code for your router function (__init__.py inside your Azure Function folder).

import logging
import json
import os
import urllib.request
import azure.functions as func

# In production, fetch this from Cosmos DB (Tenant Registry)
CELL_ENDPOINTS = {
    "alpha": os.environ.get("CELL_ALPHA_URL", "https://func-worker-alpha.azurewebsites.net"),
    "beta": os.environ.get("CELL_BETA_URL", "https://func-worker-beta.azurewebsites.net")
}

def get_target_cell(tenant_id: str) -> str:
    # Mocking a Cosmos DB registry lookup
    # A real implementation would query the global metadata database
    if tenant_id.startswith("A"):
        return "alpha"
    return "beta"

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Global Router processing a request.')

    try:
        req_body = req.get_json()
        tenant_id = req_body.get('tenant_id')

        if not tenant_id:
            return func.HttpResponse(
                "Missing partition key: tenant_id", 
                status_code=400
            )

        target_cell = get_target_cell(tenant_id)
        target_url = f"{CELL_ENDPOINTS[target_cell]}/api/process"

        # Proxy the request to the isolated cell
        data = json.dumps(req_body).encode('utf-8')
        proxy_req = urllib.request.Request(
            target_url, 
            data=data,
            headers={'Content-Type': 'application/json'}
        )

        with urllib.request.urlopen(proxy_req) as response:
            cell_response = response.read().decode('utf-8')

        return func.HttpResponse(
            json.dumps({
                "status": "success",
                "x_routed_to": target_cell,
                "data": json.loads(cell_response)
            }),
            mimetype="application/json",
            status_code=200
        )

    except Exception as e:
        logging.error(f"Routing error: {str(e)}")
        return func.HttpResponse("Internal Server Error", status_code=500)

3.4 Provisioning the Global Routing Layer

What to do: Deploy the central routing Azure Function and configure an Azure Front Door profile to sit in front of it.

Why do it: Azure Front Door acts as a secure, globally distributed entry point. It absorbs DDoS attacks at the edge, provides WAF capabilities, and ensures that the Global Router Function is protected from direct, unauthenticated internet exposure.

Example:
Append the routing infrastructure to your root main.tf.

# Shared storage for the Global Router
resource "azurerm_storage_account" "router_storage" {
  name                     = "stglobalrouter"
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_service_plan" "router_plan" {
  name                = "plan-global-router"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  os_type             = "Linux"
  sku_name            = "Y1"
}

resource "azurerm_linux_function_app" "global_router" {
  name                       = "func-global-router-gateway"
  location                   = azurerm_resource_group.rg.location
  resource_group_name        = azurerm_resource_group.rg.name
  service_plan_id            = azurerm_service_plan.router_plan.id
  storage_account_name       = azurerm_storage_account.router_storage.name
  storage_account_access_key = azurerm_storage_account.router_storage.primary_access_key

  site_config {
    application_stack {
      python_version = "3.11"
    }
  }

  app_settings = {
    "CELL_ALPHA_URL" = "https://${module.isolated_cells["alpha"].function_default_hostname}"
    "CELL_BETA_URL"  = "https://${module.isolated_cells["beta"].function_default_hostname}"
  }
}

Run terraform init, terraform plan, and terraform apply to deploy the entire multi-cell architecture.

4. Common Troubleshooting

Transitioning to a cellular architecture requires a shift in how you manage state and traffic. Here are common issues you may encounter:

Cosmos DB Throttling (HTTP 429) in a Specific Cell:
- Problem: One cell begins rejecting requests, while others operate normally. This usually indicates a "noisy neighbor" – a tenant whose workload has suddenly spiked, exhausting the Request Units (RUs) provisioned for that specific cell's database.
- Solution: Verify the metrics in Azure Monitor. If a tenant has outgrown the shared cell, you must execute a live migration. Update the Global Tenant Registry (Metadata DB) to point that specific tenant_id to a newly provisioned, dedicated cell, seamlessly redirecting their traffic.
Latency Overhead at the Routing Layer:
- Problem: Requests take significantly longer because they must pass through Front Door, the Router Function, and finally the Cell Function.
- Solution: The router logic must be aggressively optimized. Implement caching at the Global Router level using Azure Cache for Redis so the function doesn't need to query Cosmos DB for the tenant mapping on every single request.
Cold Starts in Azure Functions:
- Problem: The first request to a specific cell takes several seconds to execute.
- Solution: Since we used the Consumption Plan (Y1) for cost-effectiveness in this tutorial, cold starts are expected. For production workloads, switch the Application Service Plan to Premium (EP1 or higher) to keep instances pre-warmed, ensuring consistent low-latency responses.

5. Conclusion

Building a Cell-Based Architecture on Azure transforms monolithic vulnerabilities into contained, manageable units. By utilizing Terraform, we established a reproducible baseline for isolated environments, and implemented a Python routing layer to dynamically proxy traffic.

This decoupling ensures that massive traffic spikes or bad deployments are restricted to single domains. As you mature this architecture, focus on automating the "Tenant Migration" process—moving live data between Cosmos DB instances without downtime—and standardizing your Terraform modules to ensure this pattern can be rapidly adopted across multiple cloud environments.

Practical Guide: Building a Cell-Based Architecture on AWS with Terraform and Python

Cláudio Filipe Lima Rapôso — Tue, 21 Apr 2026 15:11:00 +0000

1. Introduction

In the cloud-native era, systems often reach a point where scaling a single massive architecture introduces unacceptable risks. A failure in a centralized component can result in a global outage, affecting all users simultaneously. Cell-Based Architecture solves this by dividing the system into multiple isolated, standalone, and identical instances called "cells." By placing users (or tenants) into specific cells, you drastically reduce the blast radius of any failure.

While this tutorial focuses on an AWS implementation, designing cellular strategies is a cornerstone of robust multicloud engineering. The principles of isolating state and routing traffic based on partition keys apply seamlessly across different providers, such as Microsoft Azure, ensuring your landing zones maintain high availability regardless of the underlying cloud.

By the end of this tutorial, you will understand how to provision a cellular infrastructure on AWS using Terraform. We will create a blueprint for a "cell," stamp out multiple instances of it, and build a Cell Router layer in Python to direct traffic to the correct isolated environment.

2. Prerequisites

To successfully implement this architectural pattern, you will need:

An active Amazon Web Services (AWS) account with administrative privileges.
Terraform installed locally (version 1.0 or higher) for Infrastructure as Code.
Python (version 3.9 or higher) to write the routing logic.
AWS credentials configured in your environment.
A foundational understanding of architectural decoupling and partition logic.

3. Step-by-Step

A Cell-Based Architecture introduces a "Thin Routing Layer" in front of your core infrastructure. The diagram below illustrates how an incoming request is evaluated and forwarded to a strictly isolated cellular stack.

3.1 Defining the Cell Blueprint (Terraform Module)

What to do: Create a reusable Terraform module that defines exactly what a single "Cell" looks like.
Why do it: The core tenet of cellular architecture is that every cell is identical. By using a Terraform module, you ensure that any updates to the infrastructure are applied uniformly across all isolated cells, preventing configuration drift.

Example:
Create a folder named modules/cell and add a main.tf file inside it. This blueprint contains an API Gateway, a Lambda function, and an isolated DynamoDB table.

# modules/cell/main.tf
variable "cell_id" {
  type        = string
  description = "Unique identifier for the cell (e.g., cell-1)"
}

resource "aws_dynamodb_table" "cell_state" {
  name           = "app-state-${var.cell_id}"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "id"

  attribute {
    name = "id"
    type = "S"
  }
}

resource "aws_lambda_function" "cell_compute" {
  filename      = "cell_worker.zip"
  function_name = "worker-${var.cell_id}"
  role          = aws_iam_role.cell_role.arn
  handler       = "worker.handler"
  runtime       = "python3.11"

  environment {
    variables = {
      CELL_ID    = var.cell_id
      TABLE_NAME = aws_dynamodb_table.cell_state.name
    }
  }
}

# (IAM Roles and API Gateway configurations for the cell would follow here)

3.2 Stamping Out Multiple Cells

What to do: In your root main.tf, iterate over a list of cell identifiers to provision multiple identical, isolated environments.
Why do it: This allows you to scale horizontally by adding new completely independent infrastructure stacks rather than increasing the size of a monolithic database or compute cluster.

Example:
In your root directory, create a main.tf to invoke the module.

# main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

locals {
  cells = ["cell-alpha", "cell-beta"]
}

module "isolated_cells" {
  source   = "./modules/cell"
  for_each = toset(local.cells)

  cell_id = each.key
}

3.3 Developing the Cell Router in Python

What to do: Write a Python function that acts as the ingress routing layer, determining which cell should handle a specific request based on a partition key.
Why do it: Clients should not need to know which cell they belong to. The router abstracts this complexity, allowing you to migrate tenants between cells behind the scenes without breaking client integrations.

Example:
Create a router.py file. This logic intercepts the request, hashes the partition key (e.g., tenant_id), and routes it.

import json
import hashlib
import urllib.request

# In a real environment, this mapping could be retrieved from a highly available edge key-value store.
CELL_ENDPOINTS = {
    "cell-alpha": "https://alpha.execute-api.us-east-1.amazonaws.com/prod",
    "cell-beta": "https://beta.execute-api.us-east-1.amazonaws.com/prod"
}

def get_target_cell(partition_key: str) -> str:
    """Consistently hashes the partition key to a specific cell."""
    hash_val = int(hashlib.md5(partition_key.encode('utf-8')).hexdigest(), 16)

    # Simple modulo distribution
    if hash_val % 2 == 0:
        return "cell-alpha"
    return "cell-beta"

def lambda_handler(event, context):
    try:
        body = json.loads(event.get('body', '{}'))
        tenant_id = body.get('tenant_id')

        if not tenant_id:
            return {"statusCode": 400, "body": "Missing partition key: tenant_id"}

        target_cell = get_target_cell(tenant_id)
        target_endpoint = CELL_ENDPOINTS[target_cell]

        # Forwarding the request to the isolated cell (Simplified for demonstration)
        req = urllib.request.Request(
            f"{target_endpoint}/process", 
            data=event.get('body').encode('utf-8'),
            headers={'Content-Type': 'application/json'}
        )

        with urllib.request.urlopen(req) as response:
            cell_response = response.read()

        return {
            "statusCode": 200,
            "body": json.dumps({
                "routed_to": target_cell,
                "cell_response": json.loads(cell_response)
            })
        }

    except Exception as e:
        return {"statusCode": 500, "body": str(e)}

3.4 Provisioning the Routing Layer

What to do: Add the routing layer to your root Terraform configuration to expose a single unified endpoint to your users.
Why do it: This centralizes ingress control. All external traffic hits the router, which then proxies the data over the AWS internal backbone to the respective cells, ensuring strict access control at the boundary.

Example:
Add this to your root main.tf:

data "archive_file" "router_zip" {
  type        = "zip"
  source_file = "router.py"
  output_path = "router.zip"
}

resource "aws_lambda_function" "cell_router" {
  filename      = data.archive_file.router_zip.output_path
  function_name = "GlobalCellRouter"
  role          = aws_iam_role.router_role.arn # (Assume basic execution role is created)
  handler       = "router.lambda_handler"
  runtime       = "python3.11"
}

resource "aws_lambda_function_url" "router_url" {
  function_name      = aws_lambda_function.cell_router.function_name
  authorization_type = "NONE"
}

output "global_entrypoint" {
  value       = aws_lambda_function_url.router_url.function_url
  description = "The single URL clients interact with."
}

4. Common Troubleshooting

Deploying cellular architectures shifts the complexity from infrastructure scaling to traffic routing and state management. Be prepared to handle these common challenges:

Partition Key Skew (Noisy Neighbors):
- Problem: One cell becomes overloaded while others are idle because a specific tenant_id generates 80% of the traffic.
- Solution: Monitor cell metrics closely. If a tenant outgrows a shared cell, you must implement a "tenant migration" process to move their data to an exclusive, single-tenant cell, updating the router's mapping logic accordingly.
Cross-Cell Data Aggregation:
- Problem: You need to generate a global report, but the data is split across multiple isolated DynamoDB tables.
- Solution: Do not query cells directly for global data. Implement an asynchronous data lake strategy where each cell streams its state changes (e.g., via DynamoDB Streams and Kinesis) to a central analytical data store.
Router Layer Bottlenecks:
- Problem: The Cell Router itself goes down, causing a global outage—exactly what cellular architecture tries to prevent.
- Solution: The routing layer must be incredibly thin and rely on highly resilient, geographically distributed edge services (like Amazon Route 53 or CloudFront/Lambda@Edge) rather than a single compute instance.

5. Conclusion

By implementing a Cell-Based Architecture, you establish definitive fault-isolation boundaries. We utilized Terraform to define a repeatable cell blueprint and created a thin Python routing layer to direct traffic dynamically.

This approach minimizes the blast radius of localized failures, making your systems inherently more resilient. As you expand on this concept, consider how this decoupled routing strategy translates across multicloud landscapes, allowing you to seamlessly route traffic between an AWS cell and an Azure cell based on performance, cost, or regulatory requirements.

Practical Guide: Building an Event-Driven Infrastructure on Microsoft Azure with Terraform and Python

Cláudio Filipe Lima Rapôso — Sun, 19 Apr 2026 13:57:33 +0000

1. Introduction

Event-driven architectures decouple system components, replacing direct synchronous communication with a highly scalable publish-subscribe model. By the end of this tutorial, you will be able to provision a complete event-based infrastructure on Microsoft Azure. This setup utilizes Azure Event Grid as the central event routing backbone, Azure Service Bus for reliable message queuing, and Azure Functions for serverless computational processing.

Mastering this topology is a structural requirement for modern software design. Isolating producers from consumers ensures that localized failures do not cascade through the system, enabling independent scaling of microservices. Furthermore, translating these concepts across different cloud providers strengthens a robust multicloud strategy, allowing you to map architectural patterns (like Event Bus -> Queue -> Serverless Compute) seamlessly into an Azure-based landing zone.

2. Prerequisites

To execute the configurations proposed in this guide, ensure you have the following prerequisites established:

An active Microsoft Azure account with permissions to create Resource Groups, Event Grid topics, Service Bus namespaces, and Function Apps.
Terraform installed locally (version 1.0 or higher) for provisioning the Infrastructure as Code (IaC).
Python (version 3.9 or higher) installed locally for developing the function logic.
The Azure CLI (az) installed and authenticated in your local environment.
Familiarity with terminal navigation and Terraform HCL syntax.

3. Step-by-Step

Before diving into the code, it is critical to visualize the event lifecycle. The sequence diagram below maps the flow of information across the provisioned Azure services.

3.1 Configuring the Provider and Resource Group

What to do: Define the Azure Resource Manager (azurerm) provider in Terraform and create a foundational Resource Group to logically group all infrastructure assets.

Why do it: Terraform requires provider definitions to authenticate and interact with the specific cloud API. The Resource Group is a mandatory Azure construct that controls the lifecycle and access management of the resources it contains.

Example:
Create a file named main.tf and add the following configuration:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "rg-event-driven-architecture"
  location = "East US"
}

3.2 Creating the Event Grid Topic

What to do: Provision a Custom Topic in Azure Event Grid.

Why do it: A Custom Topic serves as the dedicated endpoint where your applications publish business events. Isolating business events in a custom topic prevents mixing application logic with underlying Azure infrastructure events.

Example:
Append the following block to your main.tf:

resource "azurerm_eventgrid_topic" "custom_topic" {
  name                = "app-domain-events-topic"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

3.3 Provisioning the Service Bus Namespace and Queue

What to do: Create a Service Bus Namespace and a specific Queue within it to absorb incoming events.

Why do it: While Event Grid can push directly to an Azure Function, routing through a Service Bus Queue introduces a critical buffer. This ensures high availability, message durability, and prevents overwhelming the downstream compute service during traffic spikes.

Example:
Add the Service Bus configurations:

resource "azurerm_servicebus_namespace" "sb_namespace" {
  name                = "sb-event-driven-demo"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  sku                 = "Standard"
}

resource "azurerm_servicebus_queue" "event_queue" {
  name         = "event-processing-queue"
  namespace_id = azurerm_servicebus_namespace.sb_namespace.id
}

3.4 Developing the Azure Function in Python

What to do: Write the Python code using the Azure Functions v2 programming model to process messages arriving in the Service Bus Queue.

Why do it: The function represents the business logic reacting to the event. The v2 model utilizes decorators, providing a clean, concise way to define triggers and bindings directly in the code, automatically handling message deserialization.

Example:
Create a file named function_app.py in your project directory:

import logging
import json
import azure.functions as func

# Initialize the Function App
app = func.FunctionApp()

@app.service_bus_queue_trigger(
    arg_name="msg", 
    queue_name="event-processing-queue",
    connection="ServiceBusConnection"
)
def process_domain_event(msg: func.ServiceBusMessage):
    logger = logging.getLogger()
    logger.info("Initiating Service Bus event processing.")

    try:
        # Decode and load the message body
        msg_body = msg.get_body().decode('utf-8')
        event_payload = json.loads(msg_body)

        logger.info(f"Complete event payload: {json.dumps(event_payload, indent=2)}")

        # Event Grid schema typically encapsulates data in a 'data' field
        data = event_payload.get('data', {})
        order_id = data.get('order_id')

        logger.info(f"Successfully processed business operation for order: {order_id}")

    except json.JSONDecodeError:
        logger.error("Failed to decode message payload as JSON.")
    except Exception as e:
        logger.error(f"Unexpected error during processing: {str(e)}")

3.5 Provisioning the Compute Infrastructure for the Function

What to do: Define the Storage Account, Service Plan (Consumption), and the Linux Function App via Terraform, injecting the necessary connection strings as environment variables.

Why do it: Azure Functions require a backing storage account for state management and an execution plan to define scale and pricing. Injecting ServiceBusConnection securely links the compute tier to the messaging tier.

Example:
Add the compute resources to your main.tf:

resource "azurerm_storage_account" "sa" {
  name                     = "saeventdrivendemo123" # Must be globally unique
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_service_plan" "asp" {
  name                = "asp-event-driven"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  os_type             = "Linux"
  sku_name            = "Y1" # Serverless Consumption Plan
}

resource "azurerm_linux_function_app" "function_app" {
  name                       = "func-order-processor-app"
  resource_group_name        = azurerm_resource_group.rg.name
  location                   = azurerm_resource_group.rg.location
  service_plan_id            = azurerm_service_plan.asp.id
  storage_account_name       = azurerm_storage_account.sa.name
  storage_account_access_key = azurerm_storage_account.sa.primary_access_key

  site_config {
    application_stack {
      python_version = "3.11"
    }
  }

  app_settings = {
    "FUNCTIONS_WORKER_RUNTIME" = "python"
    "ServiceBusConnection"     = azurerm_servicebus_namespace.sb_namespace.default_primary_connection_string
  }
}

3.6 Creating the Event Grid Subscription (Routing Rule)

What to do: Configure an Event Subscription that filters events arriving at the Custom Topic and routes them to the Service Bus Queue.

Why do it: Advanced filtering ensures that only relevant events reach the compute tier, preventing unnecessary executions and lowering costs. This acts as the intelligent router in the architecture.

Example:
Complete the main.tf file with the routing subscription:

resource "azurerm_eventgrid_event_subscription" "queue_subscription" {
  name  = "route-order-created"
  scope = azurerm_eventgrid_topic.custom_topic.id

  service_bus_queue_endpoint_id = azurerm_servicebus_queue.event_queue.id

  advanced_filter {
    string_in {
      key    = "data.detail-type"
      values = ["OrderCreated"]
    }
  }
}

To deploy the infrastructure, run terraform init, terraform plan, and terraform apply. Note that while Terraform provisions the infrastructure, the actual deployment of the Python code is typically handled via Azure Functions Core Tools (func azure functionapp publish func-order-processor-app) or a CI/CD pipeline like GitHub Actions.

4. Common Troubleshooting

Deploying distributed systems can introduce integration challenges. Here are the most common issues and how to resolve them:

Service Bus Connection String Missing or Invalid:
Issue: The Azure Function fails to trigger, and logs show binding errors.
Solution: Verify the Application Settings in the Azure Portal for the Function App. Ensure ServiceBusConnection exactly matches the primary connection string of the Service Bus Namespace and is spelled correctly in the app.service_bus_queue_trigger decorator.
Event Grid Schema Mismatch:
Issue: Events are successfully published to the topic but never arrive in the Service Bus Queue.
Solution: Inspect the payload structure. Azure Event Grid requires a specific schema (id, subject, data, eventType, etc.). If you are filtering by data.detail-type in Terraform, ensure your published JSON explicitly contains a data object with a detail-type key matching "OrderCreated".
Storage Account Naming Conflicts:
Issue: Terraform fails during the terraform apply phase when creating the azurerm_storage_account.
Solution: Storage account names in Azure must be globally unique across all Azure customers, purely lowercase, and between 3 to 24 characters. Adjust the name attribute in the Terraform block to a highly unique string.

5. Conclusion

This tutorial established a resilient, decoupled architecture native to Microsoft Azure. By utilizing Terraform, we provisioned Event Grid for intelligent event routing, Service Bus for robust message queuing, and Azure Functions for scalable compute.

Implementing these patterns provides a clear parallel to other cloud ecosystems, reinforcing a strong foundation for designing cellular architectures and multicloud strategies. As a next step, explore implementing Dead-Letter Queues (DLQ) within the Service Bus to handle poison messages systematically, ensuring your distributed application remains robust even when faced with unprocessable data.

Practical Guide: Building an Event-Driven Infrastructure on AWS with Terraform and Python

Cláudio Filipe Lima Rapôso — Sun, 19 Apr 2026 13:52:58 +0000

1. Introduction

Event-driven architectures transform the way systems communicate by replacing direct synchronous calls with a highly decoupled publish-subscribe model. By the end of this tutorial, you will be able to provision a complete event-based infrastructure on AWS, using Amazon EventBridge as the central router, Amazon SQS for queuing, and AWS Lambda for computational processing.

This approach is fundamental for creating scalable and resilient systems. Decoupling ensures that a failure in one component does not take down the entire application, and it allows different parts of the system to scale independently. Although the technical focus here is AWS, mastering event routing and asynchronous processing is a core structural skill for strategies in multicloud environments, where similar architectural patterns are applied using corresponding services from each provider within a unified landing zone.

2. Prerequisites

To follow this tutorial and execute the proposed configurations, you need the following resources and prior knowledge:

An active Amazon Web Services (AWS) account with administrative permissions to create IAM, Lambda, SQS, and EventBridge resources.
Terraform installed on your local machine (version 1.0 or higher) for provisioning the Infrastructure as Code (IaC).
Python (version 3.9 or higher) installed locally to package the Lambda function code.
AWS credentials configured in your local environment (via AWS CLI or environment variables).
Basic knowledge of terminal navigation and Terraform HCL syntax.

3. Step-by-Step

Before writing the code, it is important to visualize the event lifecycle in our infrastructure. The sequence diagram below illustrates the flow of information between the provisioned services.

3.1 Configuring the Provider and Main File

What to do: Create the initial Terraform configuration file to define AWS as the cloud provider and establish the deployment region.

Why do it: Terraform needs to know which cloud API it will interact with and the default region to allocate resources. This centralizes the basic configuration and prepares the infrastructure state.

Example:
Create a file named main.tf and add the following block:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

3.2 Creating the Event Bus (EventBridge)

What to do: Provision a customized event bus in Amazon EventBridge.

Why do it: Although AWS provides a default bus, creating a custom bus for applications is an excellent domain isolation practice. This prevents mixing AWS infrastructure events with your application's business events.

Example:
Add to your main.tf file:

resource "aws_cloudwatch_event_bus" "custom_bus" {
  name = "app-domain-events"
}

3.3 Creating the SQS Queue for Load Absorption

What to do: Create a queue in Amazon SQS and configure the access policy to allow EventBridge to send messages to it.

Why do it: Sending events directly from EventBridge to Lambda can cause data loss in case of failures or traffic spikes. The SQS queue acts as a buffer, ensuring message delivery and allowing reprocessing in case of temporary errors.

Example:
Add the queue configurations and its respective policy:

resource "aws_sqs_queue" "event_queue" {
  name = "event-processing-queue"
}

resource "aws_sqs_queue_policy" "event_queue_policy" {
  queue_url = aws_sqs_queue.event_queue.id
  policy    = data.aws_iam_policy_document.sqs_policy.json
}

data "aws_iam_policy_document" "sqs_policy" {
  statement {
    effect    = "Allow"
    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.event_queue.arn]

    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_cloudwatch_event_rule.event_rule.arn]
    }
  }
}

3.4 Developing the Lambda Function in Python

What to do: Write the Python source code that will be executed when an event arrives in the queue.

Why do it: The Python code represents the business or computational logic that reacts to the event. In this example, we will iterate over the SQS queue records and log the details for visual validation of the architecture's functionality.

Example:
Create a file named processor.py in the same directory:

import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    logger.info("Starting processing of SQS event batch.")

    for record in event.get('Records', []):
        body_string = record.get('body', '{}')

        try:
            event_detail = json.loads(body_string)
            logger.info(f"Received event detail: {json.dumps(event_detail, indent=2)}")

            # Extracting the payload from EventBridge
            detail = event_detail.get('detail', {})
            order_id = detail.get('order_id')

            logger.info(f"Successfully processed order: {order_id}")

        except json.JSONDecodeError:
            logger.error("Error decoding message body as JSON.")

    return {
        'statusCode': 200,
        'body': json.dumps('Processing completed successfully')
    }

3.5 Provisioning Lambda with Terraform and Connecting to the Queue

What to do: Create the ZIP package of the Python code, define the Lambda function in Terraform, create the execution permission (IAM Role), and map the SQS queue as the Lambda event source.

Why do it: The infrastructure needs to allocate the code in AWS and provide the exact permissions so the Lambda can read messages from the queue and write logs to CloudWatch, following the principle of least privilege.

Example:
Add the following code to your main.tf:

data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "processor.py"
  output_path = "processor.zip"
}

resource "aws_iam_role" "lambda_exec_role" {
  name = "lambda_sqs_execution_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_basic_execution" {
  role       = aws_iam_role.lambda_exec_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

resource "aws_iam_role_policy_attachment" "lambda_sqs_execution" {
  role       = aws_iam_role.lambda_exec_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole"
}

resource "aws_lambda_function" "event_processor" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "OrderEventProcessor"
  role             = aws_iam_role.lambda_exec_role.arn
  handler          = "processor.lambda_handler"
  source_code_hash = data.archive_file.lambda_zip.output_file_base64sha256
  runtime          = "python3.11"
}

resource "aws_lambda_event_source_mapping" "sqs_trigger" {
  event_source_arn = aws_sqs_queue.event_queue.arn
  function_name    = aws_lambda_function.event_processor.arn
  batch_size       = 10
}

3.6 Creating the Routing Rule in EventBridge

What to do: Configure a rule that filters specific events published on the bus and directs them to the SQS queue.

Why do it: The bus can receive thousands of different events. Rules ensure that the target system (your SQS queue) receives only the events that matter to it, optimizing processing and reducing costs.

Example:
Complete the main.tf file with the routing configurations:

resource "aws_cloudwatch_event_rule" "event_rule" {
  name           = "capture-order-created"
  event_bus_name = aws_cloudwatch_event_bus.custom_bus.name
  description    = "Captures order created events"

  event_pattern = jsonencode({
    source      = ["com.mycompany.orders"]
    detail-type = ["OrderCreated"]
  })
}

resource "aws_cloudwatch_event_target" "sqs_target" {
  rule           = aws_cloudwatch_event_rule.event_rule.name
  event_bus_name = aws_cloudwatch_event_bus.custom_bus.name
  target_id      = "SendToSQS"
  arn            = aws_sqs_queue.event_queue.arn
}

To deploy the entire infrastructure, run the commands terraform init, terraform plan, and terraform apply in your terminal.

4. Common Troubleshooting

Even with a well-defined script, infrastructure provisioning can encounter obstacles. Below are the most frequent problems and their solutions:

Access Denied Permissions on the SQS Queue:
What is happening: EventBridge cannot deliver messages to the queue.
How to solve it: Check the aws_sqs_queue_policy. Ensure the EventBridge rule ARN is correct in the security condition. A misconfigured policy will result in the silent discard of events.
Lambda is not triggered by SQS messages:
What is happening: Messages arrive in the queue, but the Python function is not executed.
How to solve it: Validate if the IAM Role linked to the Lambda (lambda_sqs_execution_role) has the AWSLambdaSQSQueueExecutionRole policy properly attached. Lambda needs read permissions (sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes) to consume the data.
Event Pattern Incompatibility:
What is happening: You publish an event to the bus, but it does not appear in the queue.
How to solve it: Verify the accuracy of the injected JSON. The source and detail-type fields sent by the message producer must perfectly match the values stipulated in the event_pattern block of the Terraform rule. EventBridge is case-sensitive.

5. Conclusion

In this tutorial, we built a robust backbone for an event-driven architecture. We configured the structure in code using Terraform, establishing EventBridge as the intelligent traffic router, SQS as the fault-tolerance layer, and Python Lambda as the scalable processing unit.

Mastering this topology empowers you to integrate microservices safely and cleanly. As next steps, I recommend diving into the implementation of Dead Letter Queues (DLQ) for advanced handling of unresolved failures and exploring how this decoupling architectural pattern translates into the design of complex systems in multicloud scenarios.

Road to Golden Jacket: My Journey and Tips for the New AWS SOA-C03 Exam 🚀

Cláudio Filipe Lima Rapôso — Mon, 30 Mar 2026 16:15:52 +0000

When I sat down to take my AWS Certified SysOps Administrator - Associate exam on March 23, I had prepared for a broad range of operational topics. But when the test began, one thing became immediately clear: Amazon CloudWatch was everywhere.

It felt like every other scenario revolved around monitoring, analyzing logs, or troubleshooting performance bottlenecks. Now that AWS has officially transitioned to the new AWS Certified CloudOps Engineer - Associate (SOA-C03) exam, I realize that my heavy CloudWatch experience was actually the best preparation I could have asked for.

Here is a personal look at my exam experience, why CloudWatch is your best friend, and my top tips for anyone tackling the new CloudOps certification.

📈 The CloudWatch Marathon

During my exam on March 23, the monitoring and logging domain was a massive part of the test. I had to know exactly how to trigger alarms, collect logs, and automate responses.

If you are studying for the new SOA-C03 exam, this is even more critical. The domain has been expanded to "Monitoring, Logging, Analysis, Remediation, and Performance Optimization" and now carries the heaviest weight on the exam at 22%,.

To pass, you must understand how to:

Configure and manage the CloudWatch agent to collect metrics and logs not just from EC2 instances, but also from Amazon ECS and Amazon EKS clusters,.
Set up composite alarms and configure them to invoke AWS services directly or through Amazon EventBridge.
Automate remediation tasks using AWS Systems Manager Automation runbooks.

🔄 Bridging the Gap: SysOps to CloudOps

While my exam tested me heavily on traditional monitoring, the new "CloudOps" title reflects a modern shift in cloud operations. If I were taking the SOA-C03 today, here is what I would add to my study list to bridge the gap:

The Container Ecosystem: Containers are now officially in-scope. You need to understand Amazon Elastic Container Registry (ECR), Elastic Container Service (ECS), and Elastic Kubernetes Service (EKS),.
Infrastructure as Code (IaC): The new exam emphasizes creating and managing stacks using AWS CloudFormation and the AWS Cloud Development Kit (AWS CDK),.
Multi-Account Management: Operating in a single account is a thing of the past. You must know how to enforce compliance and manage security across multiple accounts using AWS Organizations, Service Control Policies (SCPs), and AWS Control Tower,.

💡 My Top Tips for Exam Day

Based on my experience in the hot seat, here are my top tips for crushing the exam:

Build a Real Monitoring Lab: Don't just read the docs. Spin up an EC2 instance, install the CloudWatch agent, and create a custom dashboard,. Set up an alarm to trigger an SNS notification or an EventBridge rule, so you see the automated flow in action.
Master Automation: Understand how to use AWS Systems Manager to automate operational processes and remediate issues,.
Eliminate the Noise: AWS questions are famously wordy and the distractors are designed to look plausible. Find the core requirement like "most cost-effective" or "highly available" and eliminate the answers that don't fit.
Embrace the AWS Well-Architected Framework: The exam tests your ability to support workloads according to these best practices,. Always think about how to optimize performance and reliability.

Taking this exam validated my practical skills and proved that I could maintain and secure workloads in the cloud. If you are preparing for the SOA-C03, dive deep into CloudWatch and embrace automation!

💡 Enjoyed it?

If you want to chat about AI, cloud, and architecture, follow me on my socials:

Social Media

I post technical content straight from the front lines. And whenever I discover a time-saving tool that gets the job done right, like this one, you'll hear about it too.

Have you taken the exam yet? Let me know your experience in the comments below! 👇

From Infrastructure to Product: Pipelines into a Profitable SaaS

Cláudio Filipe Lima Rapôso — Thu, 12 Mar 2026 19:12:31 +0000

Open Journal Systems (OJS) is the engine behind thousands of scientific publications worldwide. However, educational institutions, from high schools running literary magazines to universities managing peer-reviewed journals, share a massive pain point: they want to publish research, but they lack the internal IT resources to manage complex PHP monoliths, backups, and server security.

Blending my experience in the classroom with software architecture, I realized this is a massive opportunity. We can move beyond just "hosting" OJS and transform it into a fully automated, Zero-Touch B2B SaaS platform targeted globally at the education sector.

This article details the design, infrastructure, and product strategy to build a highly available, multi-tenant OJS SaaS on Amazon Web Services (AWS) using Terraform, Go, and GitHub Actions.

🏗️ 1. The Reference Architecture on AWS

To sell this globally, we cannot afford manual interventions or local server disks. We must adopt a distributed, stateless, and 100% managed topology.

Core Components:

Compute: Amazon ECS with AWS Fargate. We run the OJS Docker image in a serverless manner. We pay only for the CPU/RAM consumed, which is crucial for maximizing profit margins.
Database: Amazon RDS for MySQL. Fully managed, automated backups, and isolated data tiers.
File Persistence: OJS requires a persistent directory (files_dir) for articles and PDFs. We mount Amazon EFS (Elastic File System) directly to the ECS Task via the NFS protocol.

⚙️ 2. The "Zero-Touch" Provisioning Engine

To turn this infrastructure into a product, the entire customer lifecycle must be automated. Here is how the provisioning engine works:

The Storefront: A university administrator selects a plan on your landing page and checks out.
The Orchestrator (Golang API): For high concurrency and low footprint, a lightweight backend API built in Go listens for the payment gateway webhook (e.g., Stripe).
The Trigger: The Go API automatically calls the GitHub Actions API, passing the tenant_name, region, and tier as JSON payloads.
The Magic: GitHub Actions runs the Terraform scripts. Within minutes, the isolated AWS infrastructure is provisioned, Route53 updates the DNS (journal.university.edu), and the client receives an automated email with their admin credentials.

🌍 Level 1: The System Context Diagram (Business View)

Who it's for: Executives, Investors, University Directors, and non-technical Stakeholders.
What it shows: The "Big Picture." It places our OJS SaaS Platform at the center of the universe and focuses exclusively on answering two questions: Who uses the system? and What other systems do we talk to?

The Actors: It makes it clear that we have different user profiles (the Admin who pays, the Author who publishes, the Reader who consumes).
The External Boundaries: It shows that we didn't build a payment processor from scratch (we use Stripe), we didn't reinvent infrastructure pipelines (we use GitHub Actions), and we integrate with academic industry standards (Crossref).
The Value: Here, technology doesn't matter. There is no AWS, Go, or PHP in this diagram. The focus is purely on the business value flow.

Level 2: The Container Diagram (Solution/Cloud View)

Who it's for: Cloud Architects, DevOps Engineers, Tech Leads, and Security Teams.
What it shows: How the central system (the blue box from Level 1) is divided internally.
(Architectural Note: In the C4 Model, a "Container" means an independently deployable/runnable unit, not necessarily a Docker container, although in our case, they coincide).

Separation of Concerns: The division between the Storefront Web (React Front-end), the Provisioning API (our Go engine), and the Tenant Infrastructure (the client's OJS) is crystal clear.
Technology Choices: This is where the cloud "magic" appears. We show that OJS runs on ECS Fargate (Compute), uses Amazon RDS (Relational Database), and Amazon EFS (Network persistence for PDFs).
Communication Flow: It shows that the Admin doesn't access the database directly; the communication flows from the Browser to the Load Balancer, to the container, and only then to the database via port 3306.

🧩 Level 3: The Component Diagram (Engineering/Code View)

Who it's for: Software Developers (Backend).
What it shows: We take just one of the containers from Level 2 (in this case, our Go Provisioning API) and open the hood to see how the code is structured internally.

Domain Isolation: This is the perfect level to demonstrate Domain-Driven Design (DDD) principles. The Tenant Manager component represents the heart of our domain (where the business rules live).
Dependency Inversion: The diagram shows components like the GitHub API Client and the Metadata Repository. As architects, we want our domain (Tenant Manager) to not depend on specific infrastructure implementations, but rather on interfaces (ports and adapters). The visual flow shows how the business rule delegates the heavy I/O lifting to external clients.
The Webhook Lifecycle: It explains the exact path of the data: The request hits the Router, passes through validation in the Payment Handler, updates the database via the Metadata Repo, and finally triggers the infrastructure via the GitHub Client.

Here is the GitHub Actions workflow that your Go API triggers:

name: 'Zero-Touch SaaS Provisioning'
on:
  workflow_dispatch:
    inputs:
      tenant_name: { description: 'Tenant Name', required: true }
      environment: { description: 'Tier (basic/premium)', required: true }
      aws_region: { description: 'Target Region', required: true }

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ github.event.inputs.aws_region }}

      - name: Terraform Init & Apply
        run: |
          terraform init -backend-config="key=${{ github.event.inputs.tenant_name }}.tfstate"
          terraform apply -var-file="envs/${{ github.event.inputs.environment }}.tfvars.json" -auto-approve

🌍 3. Global Scaling and Data Compliance

Selling to the global education market means navigating strict data privacy laws. Because our infrastructure is codified with Terraform, deploying a compliant environment is just a matter of dynamically injecting the aws_region variable:

GDPR (Europe): If a German university signs up, the automation deploys to eu-central-1 (Frankfurt). The data never leaves Europe.
LGPD (Brazil): For Brazilian institutions, the automation targets sa-east-1 (São Paulo).
FERPA/HIPAA (USA): Medical universities publishing clinical trials can be hosted in isolated, dedicated environments in the US.

💎 4. Tiered Product Strategy (FinOps)

We can structure the SaaS offerings to capture both ends of the academic spectrum, leveraging AWS FinOps strategies:

Tier 1: High School & Small College Plan (High Volume)

Use Case: High school science fairs or small departmental newsletters.
Architecture: Shared ECS resources and a single shared RDS instance with logical separation (different schemas per school).
FinOps: We use Fargate Spot to utilize spare AWS compute capacity for up to a 70% discount. We also use EventBridge to scale the environments down to zero during weekends and school holidays.

Tier 2: University & Research Institute Plan (High Margin)

Use Case: Official university peer-reviewed journals indexing globally with Crossref and DOIs.
Architecture: Dedicated ECS Tasks, isolated RDS instances powered by Graviton (ARM) processors for better price/performance, and custom domain mapping.
Price Point: Premium recurring revenue with strict SLA guarantees and daily automated snapshots.

🐳 5. Dockerizing OJS

The Docker image remains very similar to any standard containerized environment, focusing heavily on the required PHP extensions:

FROM php:8.2-apache

# System dependencies and PHP extensions required for OJS on AWS
RUN apt-get update && apt-get install -y \
    libpng-dev libjpeg-dev libxml2-dev libzip-dev zlib1g-dev libicu-dev g++ \
    && docker-php-ext-configure gd --with-jpeg \
    && docker-php-ext-install gd mysqli opcache zip intl gettext

# Upload Optimizations
RUN { \
    echo 'opcache.memory_consumption=128'; \
    echo 'upload_max_filesize=64M'; \
    echo 'post_max_size=64M'; \
    } > /usr/local/etc/php/conf.d/ojs-optimization.ini

COPY . /var/www/html/
RUN chown -R www-data:www-data /var/www/html/

🛠️ 6. Infrastructure as Code (Terraform)

Governance across the AWS environment is managed via Terraform. The state (.tfstate) is securely stored in Amazon S3 with State Locking handled by DynamoDB.

Below is the critical snippet that connects ECS (Fargate) to EFS to persist the PDFs:

# 1. Amazon RDS MySQL
resource "aws_db_instance" "ojs_db" {
  identifier           = "ojs-db-${var.environment}"
  allocated_storage    = 20
  engine               = "mysql"
  engine_version       = "8.0"
  instance_class       = var.db_instance_class
  username             = var.db_user
  password             = var.db_password
  vpc_security_group_ids = [aws_security_group.rds_sg.id]
  db_subnet_group_name = aws_db_subnet_group.main.name
  skip_final_snapshot  = true
}

# 2. Amazon EFS for the files directory
resource "aws_efs_file_system" "ojs_files" {
  creation_token = "ojs-efs-${var.environment}"
  encrypted      = true
}

# 3. ECS Task Definition (Fargate) with EFS Volume
resource "aws_ecs_task_definition" "ojs_task" {
  family                   = "ojs-app-${var.environment}"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = var.fargate_cpu
  memory                   = var.fargate_memory

  container_definitions = jsonencode([{
    name  = "ojs-container"
    image = var.ecr_image_url
    mountPoints = [{
      sourceVolume  = "ojs-efs-volume"
      containerPath = "/var/www/ojsdata"
    }]
    environment = [
      { name = "MYSQL_HOST", value = aws_db_instance.ojs_db.address }
    ]
  }])

  volume {
    name = "ojs-efs-volume"
    efs_volume_configuration {
      file_system_id = aws_efs_file_system.ojs_files.id
      transit_encryption = "ENABLED"
    }
  }
}

Variable Injection (FinOps)

We define instance sizes via JSON (prod.tfvars.json) to control costs per environment:

{
  "environment": "prod",
  "region": "us-east-1",
  "fargate_cpu": "1024",
  "fargate_memory": "2048",
  "db_instance_class": "db.t4g.medium",
  "db_user": "ojsadmin"
}

🚀 7. CI/CD with GitHub Actions

The automated deployment pipeline checks out the code, configures AWS credentials, and updates the infrastructure via Terraform.

name: 'Deploy OJS AWS'
on:
  workflow_dispatch:
    inputs:
      tenant_name: { description: 'Tenant Name', required: true }
      environment: { description: 'Environment (dev/hom/prod)', required: true }

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Terraform Init (S3 Backend)
        run: |
          terraform init \
            -backend-config="bucket=ojs-terraform-state-central" \
            -backend-config="key=${{ github.event.inputs.tenant_name }}.${{ github.event.inputs.environment }}.tfstate" \
            -backend-config="dynamodb_table=terraform-lock"

      - name: Terraform Apply
        env:
          TF_VAR_db_password: ${{ secrets.DB_PASSWORD }}
        run: |
          terraform apply \
            -var-file="envs/${{ github.event.inputs.environment }}.tfvars.json" \
            -auto-approve

💰 8. FinOps: Cost Optimization on AWS

AWS provides fantastic mechanisms to slash costs in non-production environments:

Fargate Spot: In Dev/Hom environments, we configure the ECS Capacity Provider to use Fargate Spot, utilizing spare AWS compute capacity for up to a 70% discount.
Graviton (ARM): We use RDS instances powered by Graviton processors (db.t4g), which deliver better performance at a lower price point than the x86 architecture.
Scheduling Automation: We leverage Amazon EventBridge to trigger an AWS Lambda function that stops the RDS instance and scales the ECS Desired Count to 0 outside of regular business hours.

🎯 Conclusion

By bridging the gap between software architecture and the needs of the classroom, we can transform Open Journal Systems from a complex IT burden into a seamless, globally scalable SaaS product. The combination of AWS ECS Fargate for horizontal scalability, Go for rapid API orchestration, and Terraform for compliant IaC delivers a true, zero-touch EdTech platform.

Have you ever thought about productizing open-source software into a niche B2B SaaS? Share your thoughts and architectural approaches in the comments! 👇