DEV Community: Eslam Genedy

Stateless WebSockets? Yes — Using API Gateway + DynamoDB

Eslam Genedy — Tue, 21 Apr 2026 11:05:53 +0000

WebSockets and "stateless" feel like a contradiction. A WebSocket is a persistent, stateful connection — the server holds a socket open per client, and your broadcasting logic depends on knowing exactly which process owns which connection.

That's the assumption this article dismantles.

Using AWS API Gateway WebSocket API, DynamoDB, and Lambda, you can build real-time systems — chat, live dashboards, multiplayer games — where no server holds any connection state. Every handler is a stateless function. Any function can message any client. You can scale to zero and back without losing routing.

Let's build it.

The Problem With Traditional WebSocket Servers

A classic Node.js WebSocket server looks roughly like this:

// Traditional — stateful, process-bound
const clients = new Map(); // connectionId → socket

wss.on('connection', (socket) => {
  const id = generateId();
  clients.set(id, socket);

  socket.on('message', (data) => {
    // Broadcast to everyone — works ONLY if all sockets
    // are in THIS process
    clients.forEach((client) => {
      if (client.readyState === WebSocket.OPEN) {
        client.send(data);
      }
    });
  });

  socket.on('close', () => clients.delete(id));
});

This clients map is the problem. It lives in one process's memory. The moment you run two instances, a client connected to pod A can't be reached by code running on pod B. The solutions people reach for — sticky sessions, a Redis pub/sub layer, a shared socket server — all add operational complexity and reintroduce the statefulness you're trying to escape.

API Gateway's WebSocket support solves this at the infrastructure level.

How API Gateway WebSocket APIs Work

When a client connects to an API Gateway WebSocket endpoint, API Gateway:

Assigns a unique connectionId — a string like abc123== that identifies this socket globally
Invokes your Lambda on three route keys: $connect, $disconnect, and $default (or custom routes you define)
Holds the socket open itself — your Lambda runs and exits; the connection persists in API Gateway's infrastructure
Exposes a management endpoint you can POST to at any time to push a message to any connectionId

POST https://{api-id}.execute-api.{region}.amazonaws.com/{stage}/@connections/{connectionId}

Your Lambda doesn't hold any socket. It receives an event, does work, posts to the management endpoint, and exits. The connection state lives in API Gateway — external to your code.

This means your connectionId is the unit of state you need to manage. Store it somewhere queryable, and you can route messages from anywhere, at any time, without knowing which server (if any) is "holding" that connection.

That store is DynamoDB.

The Architecture

Client A ──────┐
Client B ──────┤──► API Gateway WebSocket ──► Lambda ($connect)   ──► DynamoDB (write connectionId)
Client C ──────┘                          ──► Lambda ($disconnect) ──► DynamoDB (delete connectionId)
                                          ──► Lambda ($default)    ──► read DynamoDB → POST to management API

Three Lambda functions. One DynamoDB table. No persistent servers.

Step 1 — The DynamoDB Table

Design the table around your query patterns. For a chat app with rooms:

Table: ws-connections

PK (partition key): connectionId   (string)  — e.g. "abc123=="
SK (sort key):      roomId         (string)  — e.g. "room#general"

GSI: roomId-index
  PK: roomId
  — lets you query "all connections in room X"

Additional attributes:
  userId     string
  connectedAt  number (Unix timestamp)
  ttl          number (Unix timestamp — auto-expire stale connections)

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('ws-connections')

def write_connection(connection_id: str, room_id: str, user_id: str, ttl: int):
    table.put_item(Item={
        'connectionId': connection_id,
        'roomId':       f'room#{room_id}',
        'userId':       user_id,
        'connectedAt':  int(time.time()),
        'ttl':          ttl   # DynamoDB TTL — auto-delete after session expires
    })

def delete_connection(connection_id: str, room_id: str):
    table.delete_item(Key={
        'connectionId': connection_id,
        'roomId':       f'room#{room_id}'
    })

def get_room_connections(room_id: str) -> list:
    response = table.query(
        IndexName='roomId-index',
        KeyConditionExpression='roomId = :rid',
        ExpressionAttributeValues={':rid': f'room#{room_id}'}
    )
    return [item['connectionId'] for item in response['Items']]

The TTL field is important. WebSocket clients disconnect without always firing $disconnect (browser tab crash, network drop). TTL ensures stale connection records expire automatically rather than accumulating and causing phantom-delivery errors.

Step 2 — The `$connect` Handler

Invoked when a client opens a WebSocket connection. Writes the connectionId to DynamoDB.

# handlers/connect.py
import json, os, time, boto3

dynamodb = boto3.resource('dynamodb')
table    = dynamodb.Table(os.environ['CONNECTIONS_TABLE'])

def handler(event, context):
    connection_id = event['requestContext']['connectionId']
    query_params  = event.get('queryStringParameters') or {}
    room_id       = query_params.get('room', 'general')
    user_id       = query_params.get('userId', 'anonymous')

    # TTL: 24 hours from now
    ttl = int(time.time()) + 86400

    table.put_item(Item={
        'connectionId': connection_id,
        'roomId':       f'room#{room_id}',
        'userId':       user_id,
        'connectedAt':  int(time.time()),
        'ttl':          ttl
    })

    return {'statusCode': 200}

The client connects like this:

const ws = new WebSocket(
  'wss://abc123.execute-api.us-east-1.amazonaws.com/prod' +
  '?room=general&userId=user_42'
);

Query string parameters arrive in event.queryStringParameters — a clean way to pass initial context without a separate HTTP call.

Step 3 — The `$disconnect` Handler

Invoked when the connection closes (cleanly). Removes the record from DynamoDB.

# handlers/disconnect.py
import os, boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource('dynamodb')
table    = dynamodb.Table(os.environ['CONNECTIONS_TABLE'])

def handler(event, context):
    connection_id = event['requestContext']['connectionId']

    # Query all rooms this connectionId appears in (a user might be in multiple)
    response = table.query(
        KeyConditionExpression=Key('connectionId').eq(connection_id)
    )

    with table.batch_writer() as batch:
        for item in response['Items']:
            batch.delete_item(Key={
                'connectionId': item['connectionId'],
                'roomId':       item['roomId']
            })

    return {'statusCode': 200}

Step 4 — The Message Handler (Broadcasting)

This is where the stateless magic becomes concrete. To broadcast a message to a room, this Lambda:

Reads all connectionIds for the room from DynamoDB
POSTs the message to each one via the API Gateway Management API
Handles stale connections gracefully (GoneException)

# handlers/message.py
import json, os, boto3
from boto3.dynamodb.conditions import Key
from botocore.exceptions import ClientError

dynamodb   = boto3.resource('dynamodb')
table      = dynamodb.Table(os.environ['CONNECTIONS_TABLE'])

def get_management_client(event):
    domain = event['requestContext']['domainName']
    stage  = event['requestContext']['stage']
    return boto3.client(
        'apigatewaymanagementapi',
        endpoint_url=f'https://{domain}/{stage}'
    )

def handler(event, context):
    connection_id = event['requestContext']['connectionId']
    body          = json.loads(event.get('body') or '{}')
    room_id       = body.get('room', 'general')
    message       = body.get('message', '')
    user_id       = body.get('userId', 'anonymous')

    # Fetch all connections in the room
    response = table.query(
        IndexName='roomId-index',
        KeyConditionExpression=Key('roomId').eq(f'room#{room_id}')
    )
    connections = response['Items']

    apigw = get_management_client(event)
    stale = []

    payload = json.dumps({
        'type':    'message',
        'room':    room_id,
        'userId':  user_id,
        'message': message
    })

    for conn in connections:
        cid = conn['connectionId']
        try:
            apigw.post_to_connection(
                ConnectionId=cid,
                Data=payload.encode('utf-8')
            )
        except ClientError as e:
            if e.response['Error']['Code'] == 'GoneException':
                # Connection no longer exists — queue for cleanup
                stale.append(cid)
            else:
                raise

    # Clean up stale connections that missed $disconnect
    if stale:
        with table.batch_writer() as batch:
            for cid in stale:
                batch.delete_item(Key={
                    'connectionId': cid,
                    'roomId':       f'room#{room_id}'
                })

    return {'statusCode': 200}

GoneException is the key error to handle. It means API Gateway has no record of that connectionId — the client disconnected without your $disconnect handler firing. When you catch it, delete the stale record so it doesn't keep appearing in future broadcasts.

Step 5 — Infrastructure with SAM / CloudFormation

# template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Globals:
  Function:
    Runtime: python3.12
    Timeout: 10
    Environment:
      Variables:
        CONNECTIONS_TABLE: !Ref ConnectionsTable

Resources:

  WebSocketApi:
    Type: AWS::ApiGatewayV2::Api
    Properties:
      Name: StatelessWsApi
      ProtocolType: WEBSOCKET
      RouteSelectionExpression: "$request.body.action"

  ConnectRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref WebSocketApi
      RouteKey: $connect
      Target: !Sub integrations/${ConnectIntegration}

  DisconnectRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref WebSocketApi
      RouteKey: $disconnect
      Target: !Sub integrations/${DisconnectIntegration}

  DefaultRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref WebSocketApi
      RouteKey: $default
      Target: !Sub integrations/${MessageIntegration}

  ConnectFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handlers/connect.handler
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref ConnectionsTable

  DisconnectFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handlers/disconnect.handler
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref ConnectionsTable

  MessageFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handlers/message.handler
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref ConnectionsTable
        - Statement:
            Effect: Allow
            Action: execute-api:ManageConnections
            Resource: !Sub arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${WebSocketApi}/*

  ConnectionsTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ws-connections
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: connectionId
          AttributeType: S
        - AttributeName: roomId
          AttributeType: S
      KeySchema:
        - AttributeName: connectionId
          KeyType: HASH
        - AttributeName: roomId
          KeyType: RANGE
      GlobalSecondaryIndexes:
        - IndexName: roomId-index
          KeySchema:
            - AttributeName: roomId
              KeyType: HASH
          Projection:
            ProjectionType: ALL
      TimeToLiveSpecification:
        AttributeName: ttl
        Enabled: true

Advanced: Broadcasting From Outside the WebSocket Context

One of the most powerful patterns this architecture unlocks: any service can push to any connected client, not just Lambda functions triggered by WebSocket messages.

A background job finishing a report, a payment service confirming a charge, a CI pipeline completing a build — any of them can push a real-time update to the right client by querying DynamoDB for the connectionId and posting to the management endpoint.

# billing_service/notify_client.py
# This service has no WebSocket context — it just knows the userId

import boto3, json, os
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource('dynamodb')
table    = dynamodb.Table('ws-connections')

def notify_user(user_id: str, payload: dict):
    """
    Find all active connections for a userId and push a message.
    Works from any service — no WebSocket server required.
    """
    response = table.query(
        IndexName='userId-index',  # add this GSI to support user-based lookups
        KeyConditionExpression=Key('userId').eq(user_id)
    )

    apigw = boto3.client(
        'apigatewaymanagementapi',
        endpoint_url=os.environ['APIGW_MANAGEMENT_URL']
    )

    message = json.dumps(payload).encode('utf-8')

    for item in response['Items']:
        try:
            apigw.post_to_connection(
                ConnectionId=item['connectionId'],
                Data=message
            )
        except apigw.exceptions.GoneException:
            table.delete_item(Key={
                'connectionId': item['connectionId'],
                'roomId':       item['roomId']
            })

This is the pattern that makes stateless WebSockets genuinely powerful for real-time apps. Your billing service, order service, or notification service can push updates to the browser the moment something happens — no polling, no dedicated WebSocket server, no shared state.

Client-Side: Handling Reconnection

Because the state is in DynamoDB and not in a server process, reconnection is clean. A new connectionId gets a new record; old records expire via TTL.

class ReconnectingWebSocket {
  constructor(url, options = {}) {
    this.url      = url;
    this.options  = options;
    this.attempts = 0;
    this.connect();
  }

  connect() {
    this.ws = new WebSocket(this.url);

    this.ws.onopen = () => {
      this.attempts = 0;
      console.log('Connected — new connectionId assigned by API Gateway');
      this.options.onOpen?.();
    };

    this.ws.onmessage = (event) => {
      this.options.onMessage?.(JSON.parse(event.data));
    };

    this.ws.onclose = () => {
      const delay = Math.min(1000 * 2 ** this.attempts, 30000);
      this.attempts++;
      console.log(`Reconnecting in ${delay}ms (attempt ${this.attempts})`);
      setTimeout(() => this.connect(), delay);
    };

    this.ws.onerror = (err) => {
      console.error('WebSocket error', err);
      this.ws.close();
    };
  }

  send(data) {
    if (this.ws.readyState === WebSocket.OPEN) {
      this.ws.send(JSON.stringify(data));
    }
  }
}

// Usage
const ws = new ReconnectingWebSocket(
  `wss://abc123.execute-api.us-east-1.amazonaws.com/prod?room=general&userId=${userId}`,
  {
    onOpen:    ()    => console.log('Ready'),
    onMessage: (msg) => renderMessage(msg)
  }
);

Exponential backoff with a 30s cap is the standard pattern. Each reconnect gets a fresh connectionId, and DynamoDB picks up the new record automatically.

DynamoDB Access Patterns Summary

Query	Access pattern	Index used
Write a new connection	`put_item`	Primary table
Delete on disconnect	`delete_item`	Primary table
Get all connections in a room	`query` by `roomId`	`roomId-index` GSI
Get all connections for a user	`query` by `userId`	`userId-index` GSI
Expire stale connections	TTL attribute	Automatic

What This Architecture Cannot Do

Be honest about the tradeoffs:

Latency budget per message. Each broadcast involves at least one DynamoDB read + N API Gateway POST calls. For a room with 50 connections, that's 51 DynamoDB operations per message. At DynamoDB's single-digit millisecond latency this is fine for chat and dashboards, but for sub-10ms gaming tick rates you'll want a different approach (Redis Pub/Sub, or purpose-built services like Ably or Pusher).

Lambda cold starts. The first invocation after a period of inactivity has a cold start penalty. For sporadic use cases this is fine. For a live dashboard that must respond instantly at all hours, use provisioned concurrency on the message handler Lambda.

Connection limits. API Gateway WebSocket APIs support up to 500 new connections per second per region (soft limit, can be raised). For very high-connection-rate scenarios — think a live event where 10,000 users join in 60 seconds — plan for this limit and request an increase in advance.

When To Use This Pattern

This architecture is a strong fit when:

You need real-time push but don't want to operate WebSocket servers
Your backend is already serverless or event-driven
Connection counts are in the tens of thousands, not millions
You want any downstream service to be able to push updates without coupling to a WebSocket layer
You're building chat, live notifications, collaborative editing, or live dashboards

It's a poor fit for:

High-frequency gaming (tick rates under 50ms)
Millions of concurrent connections (consider Ably, Pusher, or self-hosted at that scale)
Binary protocols (API Gateway WebSocket only supports text/JSON frames)

Recap

The stateless WebSocket pattern flips the traditional model:

Traditional	API Gateway + DynamoDB
Server holds socket in memory	API Gateway holds the socket
Broadcast requires shared process	Broadcast via management API from anywhere
Scale-out needs sticky sessions	Scale-out is trivial — no shared state
Connection loss = state loss	Connection loss = TTL cleanup
Ops burden: WebSocket server fleet	Ops burden: near zero

The connectionId is your external handle on what is otherwise an invisible infrastructure socket. DynamoDB is your routing table. Lambda is your logic. None of them need to know about each other — which is exactly what stateless means.

From Monolith State to Stateless Microservices: A Real-World Refactor

Eslam Genedy — Tue, 21 Apr 2026 10:59:12 +0000

Two years ago our team inherited Atlas — a B2B SaaS monolith with ~680,000 lines of Rails code, a decade of accumulated complexity, and a deployment pipeline that caused anxiety every Friday afternoon.

The symptoms were familiar:

Single PostgreSQL instance handling transactional and reporting load
In-process custom thread pool for background jobs
Singleton service objects maintaining application-level state
Session data stored in application memory behind a sticky-session load balancer
Deployments caused ~4 minutes of session loss for active users

Our goal wasn't a big-bang rewrite. It was a disciplined, iterative extraction — shipping features throughout the process while incrementally achieving statelessness.

Phase 1 — Audit: Finding State Where It Shouldn't Be

Before touching code, we needed a full inventory of where state lived. This sounds simple. It isn't.

Identifying Stateful Singletons

The most dangerous pattern we found: service objects instantiated once at boot and shared across all requests.

# config/initializers/rate_limiter.rb
RateLimiter = RateLimiterService.new(
  store: {},     # mutable hash in process memory
  window: 60,
  limit: 100
)

RateLimiterService held a mutable store hash accumulating per-user request counts. Under a single process: fine. Under autoscaling with multiple pods: each pod maintained independent counters, making rate limiting effectively broken.

We audited the entire codebase and found 23 such singletons across four categories:

Category	Count	Problem
Rate limiters / throttles	6	Per-process counters — ineffective at scale
Feature flag caches	4	Stale state — flags wouldn't propagate across pods
Third-party client pools	8	Connection state not shared, over-connecting
Tenant config aggregators	5	Mutations in one request leaked into others

The fix for every category followed the same principle: move state out of process memory into a shared external store.

# Before — process-local hash
RateLimiter = RateLimiterService.new(store: {})

# After — Redis-backed, shared across all pods
RateLimiter = RateLimiterService.new(
  store: RedisStore.new(
    client: Redis.current,
    namespace: "rate_limiter"
  )
)

The Sticky Session Problem

Atlas used Rails' default :cookie_store — but with a twist. A developer years earlier needed to store non-serializable ActiveRecord objects in the session. Rather than fix serialization, they mirrored part of the session server-side in a process-level hash, then configured the load balancer with IP-based sticky sessions to make sure users always hit the same pod.

The consequences:

Horizontal scaling didn't distribute load evenly
Pod restarts caused immediate session loss
Rolling deployments silently logged out active users

Fix part 1 — Migrate session storage to Redis:

# Gemfile
gem 'redis-session-store'

# config/initializers/session_store.rb
Rails.application.config.session_store :redis_session_store,
  key: '_atlas_session',
  redis: {
    expire_after: 2.hours,
    key_prefix: 'atlas:session:',
    client: Redis.current
  }

Fix part 2 — Eliminate non-serializable objects from session entirely:

# Before — storing the full ActiveRecord object
session[:current_user] = current_user

# After — store only the primitive ID, hydrate on each request
session[:current_user_id] = current_user.id

# In ApplicationController
def current_user
  @current_user ||= User.find_by(id: session[:current_user_id])
end

After this change we removed sticky sessions from the load balancer entirely. Session data became a first-class shared resource, not a pod-local artifact. We saw an immediate 18% reduction in p99 login latency by eliminating database joins that had been avoided via session caching with stale data.

Phase 2 — Extraction: Breaking State Ownership

With state inventoried and moved to external stores, we began service extraction using a strangler fig pattern — new services intercepted traffic slice by slice while the monolith continued running.

Choosing What to Extract First

We scored candidates across three dimensions:

Score = (Domain Clarity × Async Viability) / Shared State Coupling

High score = good first candidate. Notifications won.

Notifications had:

Clear bounded domain
No need for synchronous response
High enough volume to justify the infrastructure
A clean, definable event contract

The original monolith code was fully synchronous and blocking:

def complete_order(order)
  order.finalize!
  NotificationMailer.order_confirmation(order).deliver_now  # blocks thread
  WebhookService.deliver(:order_completed, order)           # blocks thread
  ActivityFeed.append(order.user, :order_completed, order)  # blocks thread
end

If the email provider was slow, order completion was slow. If a webhook endpoint timed out, the user waited. Classic incidental coupling through synchrony.

Phase 3 — SQS for Async State Updates

We replaced the synchronous calls with event publishing to SQS. The monolith's job changed: finalize the order, emit a fact, return.

The Event Bus Abstraction

We built a thin EventBus wrapper around the AWS SDK to keep the calling code clean:

# lib/event_bus.rb
class EventBus
  QUEUE_URLS = {
    "order.completed"      => ENV["SQS_ORDER_EVENTS_URL"],
    "user.registered"      => ENV["SQS_USER_EVENTS_URL"],
    "subscription.changed" => ENV["SQS_BILLING_EVENTS_URL"]
  }.freeze

  def self.publish(event_type, payload)
    url = QUEUE_URLS.fetch(event_type) do
      raise ArgumentError, "Unknown event type: #{event_type}"
    end

    sqs_client.send_message(
      queue_url:    url,
      message_body: JSON.generate({
        event:     event_type,
        payload:   payload,
        timestamp: Time.now.utc.iso8601,
        version:   "1.0"
      }),
      message_attributes: {
        "EventType" => {
          string_value: event_type,
          data_type:    "String"
        }
      }
    )
  end

  def self.sqs_client
    @sqs_client ||= Aws::SQS::Client.new(region: ENV["AWS_REGION"])
  end
end

The refactored order completion:

def complete_order(order)
  order.finalize!

  EventBus.publish("order.completed", {
    order_id:   order.id,
    user_id:    order.user_id,
    user_email: order.user.email,
    total:      order.total.to_s,
    currency:   order.currency,
    line_items: order.line_items.map { |li| { sku: li.sku, qty: li.qty } }
  })
end

The controller returns immediately. No email. No webhook. No feed update. All of those happen asynchronously, decoupled from the user's request thread.

The Notification Service Consumer

The extracted notification service is a standalone Python service (we used Python for new services — the team had strong Python ML tooling we wanted to leverage long-term). It polls SQS and processes events:

# notification_service/consumer.py
import boto3, json, logging
from handlers import order_handlers, user_handlers

logger = logging.getLogger(__name__)

HANDLER_MAP = {
    "order.completed":      order_handlers.handle_order_completed,
    "user.registered":      user_handlers.handle_user_registered,
    "subscription.changed": user_handlers.handle_subscription_changed,
}

def poll(queue_url: str, sqs_client):
    while True:
        response = sqs_client.receive_message(
            QueueUrl=queue_url,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20,          # long polling — reduces empty receives
            MessageAttributeNames=["EventType"]
        )

        for message in response.get("Messages", []):
            process_message(message, queue_url, sqs_client)

def process_message(message: dict, queue_url: str, sqs_client):
    try:
        body       = json.loads(message["Body"])
        event_type = body["event"]
        handler    = HANDLER_MAP.get(event_type)

        if not handler:
            logger.warning(f"No handler registered for event: {event_type}")
        else:
            handler(body["payload"])

        # Only delete the message if processing succeeded
        sqs_client.delete_message(
            QueueUrl=queue_url,
            ReceiptHandle=message["ReceiptHandle"]
        )

    except Exception as exc:
        # Do NOT delete — message returns to queue after visibility timeout
        # SQS dead-letter queue (DLQ) handles repeated failures
        logger.error(f"Failed to process message {message['MessageId']}: {exc}")

# notification_service/handlers/order_handlers.py
from clients.email_client import send_transactional_email
from clients.webhook_client import deliver_webhook
from clients.feed_client import append_feed_event

def handle_order_completed(payload: dict):
    send_transactional_email(
        template="order_confirmation",
        to=payload["user_email"],
        context={
            "order_id":   payload["order_id"],
            "total":      payload["total"],
            "currency":   payload["currency"],
            "line_items": payload["line_items"]
        }
    )

    deliver_webhook(
        event="order.completed",
        payload=payload
    )

    append_feed_event(
        user_id=payload["user_id"],
        event_type="order_completed",
        metadata={"order_id": payload["order_id"]}
    )

SQS Configuration: Dead-Letter Queue

Idempotency and failure handling are non-negotiable in async systems. We configured a DLQ for every event queue:

# terraform/sqs.tf

resource "aws_sqs_queue" "order_events" {
  name                       = "atlas-order-events"
  visibility_timeout_seconds = 30
  message_retention_seconds  = 86400   # 24 hours

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.order_events_dlq.arn
    maxReceiveCount     = 3            # 3 failed attempts → DLQ
  })
}

resource "aws_sqs_queue" "order_events_dlq" {
  name                      = "atlas-order-events-dlq"
  message_retention_seconds = 1209600  # 14 days — time to investigate
}

With maxReceiveCount = 3, a message that fails processing three times moves to the DLQ automatically. We set up CloudWatch alarms on DLQ depth so failures surface as alerts rather than silent data loss.

Phase 4 — Event-Driven Patterns Across Services

Once SQS was in place and the team had built muscle with it, we extended the pattern across the rest of the extraction effort. Three patterns proved especially valuable.

Pattern 1: Event Sourcing for Audit Trails

The billing domain previously had an updated_at column and a loosely maintained versions table. After extraction, every billing event became a first-class fact published to SQS and persisted in a dedicated event store:

# billing_service/event_store.py
import boto3, json
from datetime import datetime, timezone

dynamodb = boto3.resource("dynamodb")
table    = dynamodb.Table("atlas-billing-events")

def record_event(aggregate_id: str, event_type: str, payload: dict):
    timestamp = datetime.now(timezone.utc).isoformat()

    table.put_item(Item={
        "pk":         f"BILLING#{aggregate_id}",
        "sk":         f"EVENT#{timestamp}",
        "event_type": event_type,
        "payload":    payload,
        "version":    get_next_version(aggregate_id)
    })

def get_billing_history(aggregate_id: str) -> list:
    response = table.query(
        KeyConditionExpression="pk = :pk AND begins_with(sk, :prefix)",
        ExpressionAttributeValues={
            ":pk":     f"BILLING#{aggregate_id}",
            ":prefix": "EVENT#"
        },
        ScanIndexForward=True
    )
    return response["Items"]

This replaced what used to be a SELECT * FROM billing_audits WHERE account_id = ? query on the monolith's main database — which had become a serious bottleneck at reporting time.

Pattern 2: Saga for Distributed Transactions

Distributed systems don't have ACID transactions across service boundaries. When we extracted the subscription service, we had to handle the scenario: charge succeeds → subscription activation fails. We implemented a choreography-based saga using SQS + SNS:

charge.succeeded  ──► [Subscription Service] ──► subscription.activated
                                │
                           (on failure)
                                │
                                ▼
                       subscription.activation_failed
                                │
                                ▼
                       [Billing Service] ──► charge.reversed

# subscription_service/saga.py

def handle_charge_succeeded(payload: dict):
    account_id = payload["account_id"]
    plan_id    = payload["plan_id"]

    try:
        activate_subscription(account_id, plan_id)

        publish_event("subscription.activated", {
            "account_id": account_id,
            "plan_id":    plan_id,
            "charge_id":  payload["charge_id"]
        })

    except ActivationError as exc:
        # Compensating transaction — tell billing to reverse the charge
        publish_event("subscription.activation_failed", {
            "account_id": account_id,
            "charge_id":  payload["charge_id"],
            "reason":     str(exc)
        })

# billing_service/saga_handlers.py

def handle_activation_failed(payload: dict):
    charge_id = payload["charge_id"]
    reverse_charge(charge_id)

    publish_event("charge.reversed", {
        "charge_id":  charge_id,
        "account_id": payload["account_id"],
        "reason":     payload["reason"]
    })

The key insight: each service owns its own compensating logic. There's no central orchestrator that becomes a bottleneck or single point of failure.

Pattern 3: Read Model Projections (CQRS)

The reporting module was one of the last pieces consuming the monolith's main Postgres instance. We extracted it by building read-model projections — denormalized views rebuilt from events.

Each time an event arrived, a projection worker updated a read-optimized store:

# reporting_service/projections/revenue_projection.py

def on_order_completed(payload: dict):
    """
    Update the revenue projection whenever an order completes.
    This is eventually consistent — typically < 500ms behind real-time.
    """
    date_key = payload["timestamp"][:10]   # "2024-11-15"

    dynamodb.update_item(
        TableName="atlas-revenue-daily",
        Key={
            "date":     {"S": date_key},
            "currency": {"S": payload["currency"]}
        },
        UpdateExpression="ADD total_cents :amount, order_count :one",
        ExpressionAttributeValues={
            ":amount": {"N": str(payload["total_cents"])},
            ":one":    {"N": "1"}
        }
    )

The reporting API now reads from DynamoDB, never from Postgres. The main database load dropped by ~31% after this migration.

Results After 14 Months

Metric	Before	After
Deployment time	18 min (full downtime)	3 min (rolling, zero downtime)
Horizontal scaling	Not possible	Auto-scales to 12 pods
p99 API latency	1,840ms	290ms
Session-related incidents	~3/month	0 in last 6 months
Primary DB CPU (peak)	94%	41%
MTTR for notification failures	~2 hours	~8 minutes (DLQ alert → fix)

Lessons Worth Remembering

1. State audit before extraction.
Do not extract a service and discover later it secretly depends on shared mutable state. Build the inventory first. Every singleton, every in-memory store, every sticky-session dependency.

2. Make events self-contained.
An event payload should carry enough data that consumers never need to call back into the emitting service to process it. Avoid events like { "order_id": 123 } that force consumers to make a synchronous call to the monolith. Include the data the consumers need.

3. Idempotency is mandatory, not optional.
SQS delivers messages at least once. Consumers will see duplicates. Every handler must be safe to invoke multiple times with the same payload. Use a dedupe key:

def handle_order_completed(payload: dict):
    dedupe_key = f"notification:order_completed:{payload['order_id']}"

    if cache.exists(dedupe_key):
        return   # Already processed — safe to skip

    # ... process ...

    cache.set(dedupe_key, "1", ex=86400)  # 24-hour TTL

4. DLQs are your observability layer.
A growing DLQ is a canary. Alert on it aggressively. Every message in the DLQ represents a business event that did not complete — treat it with the same urgency as a failed database write.

5. Don't extract the hardest domain first.
Start with high-async-viability, low-shared-state domains (notifications, emails, audit logs). Build team confidence and tooling patterns before tackling the hard stuff (billing, auth, core domain logic).

The Code That Made the Biggest Difference

If you take one pattern from this article, take this one: the thin event bus wrapper with a strict schema contract.

# The monolith's entire async surface area lives here.
# Every service interaction is an event.
# Every event has a version field.
# Schema changes are additive, never breaking.

EventBus.publish("order.completed", {
  schema_version: "1.2",
  order_id:       order.id,
  # ... fields
})

Versioning events from day one saved us multiple breaking-change incidents. When the notification service needed a new field, we added it to the payload (non-breaking). When a field needed to change shape, we bumped the version and ran both handlers in parallel until consumers migrated.

The migration is ongoing — we still have three bounded contexts inside the monolith. But the hardest part wasn't the code. It was building shared mental models across the team about what "stateless" actually means in a distributed system, and why the discipline of event contracts pays for itself a hundred times over.

Building Serverless CI/CD Pipelines with AWS CodePipeline and CodeBuild

Eslam Genedy — Sat, 08 Feb 2025 20:34:57 +0000

In the world of serverless computing, Continuous Integration and Continuous Deployment (CI/CD) pipelines are essential for automating the deployment of Lambda functions and other serverless resources. AWS provides powerful tools like CodePipeline and CodeBuild to create robust CI/CD pipelines, while AWS SAM (Serverless Application Model) simplifies the deployment of serverless applications. In this article, we’ll dive deep into building serverless CI/CD pipelines, discuss best practices, and provide practical code samples.

Why Serverless CI/CD?

Serverless applications are highly dynamic, with frequent updates to Lambda functions, API Gateway configurations, and other resources. A well-designed CI/CD pipeline ensures:

Automated deployments: Reduce manual errors and speed up releases.
Consistency: Ensure that every deployment follows the same process.
Rollbacks: Quickly revert to a previous version in case of failures.
Testing: Automate unit tests, integration tests, and security checks.

Key Components of a Serverless CI/CD Pipeline

AWS CodePipeline: Orchestrates the stages of your CI/CD pipeline (e.g., source, build, deploy).
AWS CodeBuild: Compiles your code, runs tests, and packages your application.
AWS SAM: A framework for defining and deploying serverless applications.
AWS CloudFormation: Used by SAM to provision and manage resources.
AWS Lambda: The serverless compute service where your functions run.

Step 1: Setting Up CodePipeline and CodeBuild

1.1 Create a CodePipeline

Go to the CodePipeline console in AWS.
Click Create pipeline.
Name your pipeline (e.g., ServerlessPipeline).
Choose a source provider (e.g., GitHub, CodeCommit, or S3).
Connect your repository and select the branch to monitor.

1.2 Add a Build Stage with CodeBuild

In the pipeline creation wizard, add a Build stage.
Create a new CodeBuild project.
Configure the build environment:
- Choose a managed image (e.g., aws/codebuild/standard:5.0).
- Specify the runtime (e.g., Node.js, Python, etc.).
Add a buildspec.yml file to your repository to define the build steps.

Step 2: Automating Deployments with AWS SAM

AWS SAM simplifies the deployment of serverless applications by providing a shorthand syntax for defining resources like Lambda functions, API Gateway, and DynamoDB tables.

2.1 Create a SAM Template

Here’s an example template.yaml file for a simple serverless application:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.lambda_handler
      Runtime: python3.9
      CodeUri: .
      Events:
        HelloWorldApi:
          Type: Api
          Properties:
            Path: /hello
            Method: GET

2.2 Build and Deploy with SAM

In your buildspec.yml, define the build and deploy steps:

version: 0.2

phases:
  install:
    commands:
      - echo "Installing dependencies..."
      - pip install -r requirements.txt -t ./dependencies
  build:
    commands:
      - echo "Building the application..."
      - sam build
  post_build:
    commands:
      - echo "Deploying the application..."
      - sam deploy --no-confirm-changeset --no-fail-on-empty-changeset --stack-name my-serverless-app --capabilities CAPABILITY_IAM

Step 3: Handling Rollbacks and Testing

3.1 Automated Testing

Include testing in your pipeline to catch issues early. For example, add a test phase in your buildspec.yml:

phases:
  install:
    commands:
      - echo "Installing dependencies..."
      - pip install -r requirements.txt -t ./dependencies
  build:
    commands:
      - echo "Building the application..."
      - sam build
  test:
    commands:
      - echo "Running unit tests..."
      - python -m pytest tests/
  post_build:
    commands:
      - echo "Deploying the application..."
      - sam deploy --no-confirm-changeset --no-fail-on-empty-changeset --stack-name my-serverless-app --capabilities CAPABILITY_IAM

3.2 Rollbacks

AWS SAM and CloudFormation automatically handle rollbacks. If a deployment fails, CloudFormation reverts to the previous stable state. You can also configure alarms and automated rollbacks using AWS CloudWatch.

Step 4: Advanced Pipeline Features

4.1 Manual Approval Stage

Add a manual approval stage before deploying to production:

In the CodePipeline console, add a new stage (e.g., Approval).
Add an Approval action where a team member must approve the deployment.

4.2 Canary Deployments

Use AWS CodeDeploy to perform canary deployments for Lambda functions. This allows you to gradually roll out changes to a subset of users.

4.3 Environment-Specific Configurations

Use AWS Systems Manager Parameter Store or AWS Secrets Manager to manage environment-specific configurations (e.g., API keys, database credentials).

Best Practices for Serverless CI/CD Pipelines

Version Control Everything: Store your SAM templates, build scripts, and pipeline configurations in version control (e.g., GitHub).
Automate Testing: Include unit tests, integration tests, and security scans in your pipeline.
Monitor Deployments: Use CloudWatch alarms and dashboards to monitor the health of your deployments.
Use Infrastructure as Code (IaC): Define all resources (e.g., Lambda functions, API Gateway) using SAM or CloudFormation.
Optimize Build Times: Cache dependencies and use parallel builds to speed up the pipeline.

Code Samples

SAM Template (`template.yaml`)

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.lambda_handler
      Runtime: python3.9
      CodeUri: .
      Events:
        HelloWorldApi:
          Type: Api
          Properties:
            Path: /hello
            Method: GET

Build Specification (`buildspec.yml`)

version: 0.2

phases:
  install:
    commands:
      - echo "Installing dependencies..."
      - pip install -r requirements.txt -t ./dependencies
  build:
    commands:
      - echo "Building the application..."
      - sam build
  test:
    commands:
      - echo "Running unit tests..."
      - python -m pytest tests/
  post_build:
    commands:
      - echo "Deploying the application..."
      - sam deploy --no-confirm-changeset --no-fail-on-empty-changeset --stack-name my-serverless-app --capabilities CAPABILITY_IAM

Conclusion

Building serverless CI/CD pipelines with AWS CodePipeline and CodeBuild enables you to automate deployments, ensure consistency, and reduce manual errors. By leveraging AWS SAM, you can simplify the deployment of serverless applications and integrate best practices like automated testing and rollbacks.

With the steps and code samples provided in this article, you’re ready to create your own serverless CI/CD pipeline and take your serverless development to the next level. Happy building! 🚀

Processing Streaming Data and Fetching Files with AWS Lambda and Python

Eslam Genedy — Sat, 08 Feb 2025 20:28:28 +0000

AWS Lambda is a powerful serverless compute service that allows you to run code in response to events without managing servers. One of the most common use cases for Lambda is processing streaming data and fetching files from various sources, such as S3 buckets, HTTP endpoints, or databases. In this article, we’ll explore how to use AWS Lambda with Python to process streaming data and fetch different types of files, such as CSV, JSON, and images.

Use Case: Processing Streaming Data and Fetching Files

Imagine a scenario where you have a system that generates streaming data (e.g., logs, sensor data, or user activity). This data is stored in an S3 bucket, and you need to process it in real-time. Additionally, you may need to fetch and process files from external sources, such as APIs or databases.

Key Components:

AWS Lambda: For processing streaming data and fetching files.
Amazon S3: For storing streaming data and processed files.
Python: For writing the Lambda function logic.

Step 1: Set Up an S3 Bucket for Streaming Data

Go to the S3 console in AWS.
Create a new bucket, e.g., streaming-data-bucket.
Enable event notifications for the bucket to trigger a Lambda function whenever a new file is uploaded.

Step 2: Create a Lambda Function to Process Streaming Data

Go to the Lambda console.
Create a new function named processStreamingData.
Choose Python 3.x as the runtime.
Attach an IAM role with permissions to access S3 and other required services.

Here’s the Python code for the Lambda function:

import boto3
import json
import csv
import io

s3 = boto3.client('s3')

def lambda_handler(event, context):
    # Get the S3 bucket and object key from the event
    bucket_name = event['Records'][0]['s3']['bucket']['name']
    object_key = event['Records'][0]['s3']['object']['key']

    # Fetch the file from S3
    response = s3.get_object(Bucket=bucket_name, Key=object_key)
    file_content = response['Body'].read().decode('utf-8')

    # Process the file based on its type
    if object_key.endswith('.csv'):
        process_csv(file_content)
    elif object_key.endswith('.json'):
        process_json(file_content)
    elif object_key.endswith('.jpg') or object_key.endswith('.png'):
        process_image(bucket_name, object_key)
    else:
        print(f"Unsupported file type: {object_key}")

    return {
        'statusCode': 200,
        'body': json.dumps('File processed successfully')
    }

def process_csv(file_content):
    # Read CSV data
    csv_file = io.StringIO(file_content)
    csv_reader = csv.DictReader(csv_file)

    # Process each row
    for row in csv_reader:
        print(f"Processing CSV row: {row}")

def process_json(file_content):
    # Parse JSON data
    json_data = json.loads(file_content)

    # Process JSON data
    print(f"Processing JSON data: {json_data}")

def process_image(bucket_name, object_key):
    # Fetch the image from S3
    response = s3.get_object(Bucket=bucket_name, Key=object_key)
    image_content = response['Body'].read()

    # Example: Resize the image using a library like Pillow
    from PIL import Image
    import io

    image = Image.open(io.BytesIO(image_content))
    resized_image = image.resize((100, 100))

    # Save the resized image back to S3
    resized_image_bytes = io.BytesIO()
    resized_image.save(resized_image_bytes, format='JPEG')
    resized_image_bytes.seek(0)

    s3.put_object(
        Bucket=bucket_name,
        Key=f"resized/{object_key}",
        Body=resized_image_bytes
    )
    print(f"Resized image saved to S3: resized/{object_key}")

Step 3: Configure S3 Event Notifications

Go to the S3 console.
Select your bucket (streaming-data-bucket).
Go to the Properties tab and click Create event notification.
Set the event name to NewFileUpload.
Choose All object create events as the event type.
Send the event to the processStreamingData Lambda function.

Step 4: Fetching Files from External Sources

Sometimes, you may need to fetch files from external sources, such as APIs or databases. Here’s how you can modify the Lambda function to fetch and process files from an HTTP endpoint.

Example: Fetching a JSON File from an API

import requests

def fetch_json_from_api(url):
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        raise Exception(f"Failed to fetch JSON from {url}")

def lambda_handler(event, context):
    # Example: Fetch JSON data from an API
    api_url = "https://api.example.com/data.json"
    try:
        json_data = fetch_json_from_api(api_url)
        process_json(json_data)
    except Exception as e:
        print(f"Error fetching JSON: {e}")

    return {
        'statusCode': 200,
        'body': json.dumps('File fetched and processed successfully')
    }

Step 5: Testing the Lambda Function

Upload a CSV, JSON, or image file to your S3 bucket.
Check the CloudWatch logs to verify that the Lambda function processed the file correctly.
For external file fetching, ensure the Lambda function has internet access (configure a VPC with a NAT gateway if needed).

Best Practices for Processing Streaming Data and Files

Use Environment Variables: Store sensitive information like API URLs or credentials in environment variables.
Optimize Memory and Timeout: Adjust the Lambda function’s memory and timeout settings based on the size of the files being processed.
Handle Errors Gracefully: Implement error handling for network issues, invalid file formats, or unexpected data.
Leverage Layers: Use Lambda layers to include external libraries like Pillow for image processing.
Monitor with CloudWatch: Set up CloudWatch alarms and dashboards to monitor the performance of your Lambda function.

Conclusion

AWS Lambda, combined with Python, provides a flexible and scalable solution for processing streaming data and fetching files from various sources. Whether you’re handling CSV files, JSON data, or images, Lambda makes it easy to build event-driven workflows that scale automatically.

By following the steps and best practices outlined in this article, you can build robust serverless applications that process data in real-time and integrate seamlessly with other AWS services. Happy coding! 🚀

Introduction to Serverless Computing with AWS: A Deep Dive with Node.js

Eslam Genedy — Sat, 08 Feb 2025 20:17:28 +0000

Serverless computing is a game-changer for modern application development, and AWS Lambda is at the forefront of this revolution. By leveraging AWS Lambda, developers can build scalable, event-driven applications without worrying about server management. In this article, we’ll explore serverless computing with AWS services, focusing on AWS Lambda, API Gateway, and DynamoDB, with practical Node.js code samples.

What is AWS Lambda?

AWS Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources. You can use Lambda to build applications that respond quickly to new information, scale automatically, and only incur costs when your code is running.

Key Features of AWS Lambda:

Event-Driven: Lambda functions are triggered by events from AWS services like S3, DynamoDB, API Gateway, and more.
Automatic Scaling: Lambda automatically scales your application by running code in response to each trigger.
Pay-as-You-Go: You are charged only for the compute time your code consumes, measured in milliseconds.

AWS Services for Serverless Applications

To build a complete serverless application, you’ll often use a combination of AWS services:

AWS Lambda: For running your code.
API Gateway: To create RESTful APIs that trigger Lambda functions.
DynamoDB: A fully managed NoSQL database for storing application data.
IAM (Identity and Access Management): To manage permissions for your Lambda functions and other AWS resources.

Building a Serverless Application with AWS and Node.js

Let’s build a simple serverless application using AWS Lambda, API Gateway, and DynamoDB. The application will allow users to:

Create a new item in a DynamoDB table.
Retrieve all items from the DynamoDB table.

Prerequisites:

An AWS account.
AWS CLI installed and configured.
Node.js installed.

Step 1: Set Up a DynamoDB Table

Go to the DynamoDB console in AWS.
Click Create Table.
Name the table Items and set the primary key as id (String).
Click Create.

Step 2: Create an IAM Role for Lambda

Go to the IAM console.
Create a new role with the following permissions:
- AmazonDynamoDBFullAccess
- AWSLambdaBasicExecutionRole
Attach this role to your Lambda functions.

Step 3: Create a Lambda Function to Add Items to DynamoDB

Go to the Lambda console.
Click Create Function.
Name the function addItem.
Choose Node.js 18.x as the runtime.
Under Permissions, attach the IAM role you created earlier.

Here’s the Node.js code for the addItem function:

const AWS = require('aws-sdk');
const dynamoDb = new AWS.DynamoDB.DocumentClient();

exports.handler = async (event) => {
    const { id, name } = JSON.parse(event.body);

    const params = {
        TableName: 'Items',
        Item: { id, name },
    };

    try {
        await dynamoDb.put(params).promise();
        return {
            statusCode: 200,
            body: JSON.stringify({ message: 'Item added successfully' }),
        };
    } catch (error) {
        return {
            statusCode: 500,
            body: JSON.stringify({ error: 'Could not add item' }),
        };
    }
};

Step 4: Create a Lambda Function to Retrieve Items from DynamoDB

Create another Lambda function named getItems.
Use the same runtime and IAM role as before.

Here’s the Node.js code for the getItems function:

const AWS = require('aws-sdk');
const dynamoDb = new AWS.DynamoDB.DocumentClient();

exports.handler = async () => {
    const params = {
        TableName: 'Items',
    };

    try {
        const data = await dynamoDb.scan(params).promise();
        return {
            statusCode: 200,
            body: JSON.stringify(data.Items),
        };
    } catch (error) {
        return {
            statusCode: 500,
            body: JSON.stringify({ error: 'Could not retrieve items' }),
        };
    }
};

Step 5: Set Up API Gateway

Go to the API Gateway console.
Click Create API and choose HTTP API.
Add two routes:
- POST /items to trigger the addItem Lambda function.
- GET /items to trigger the getItems Lambda function.
Deploy the API to a stage (e.g., prod).

Step 6: Test the Application

You can use tools like Postman or curl to test your API.

Add an Item:

curl -X POST https://<api-id>.execute-api.<region>.amazonaws.com/prod/items \
-H "Content-Type: application/json" \
-d '{"id": "1", "name": "Sample Item"}'

Retrieve Items:

curl -X GET https://<api-id>.execute-api.<region>.amazonaws.com/prod/items

Advanced: Environment Variables and Logging

Environment Variables

You can configure environment variables in the Lambda console to store sensitive information like table names or API keys.

Logging

AWS Lambda automatically integrates with CloudWatch. Use console.log or console.error in your Node.js code to log messages:

console.log('Adding item to DynamoDB:', params);
console.error('Error adding item:', error);

Best Practices for Serverless Development

Keep Functions Small and Focused: Each Lambda function should perform a single task.
Use Environment Variables: Avoid hardcoding configuration values.
Monitor with CloudWatch: Set up alarms and dashboards to monitor your functions.
Optimize Cold Starts: Use provisioned concurrency for time-sensitive applications.
Secure Your APIs: Use IAM roles, API keys, or Cognito for authentication.

Conclusion

Serverless computing with AWS Lambda and Node.js enables developers to build scalable, cost-effective applications with minimal operational overhead. By combining AWS services like Lambda, API Gateway, and DynamoDB, you can create powerful serverless architectures that respond to events in real-time.

With the code samples and steps provided, you’re now equipped to start building your own serverless applications on AWS. Happy coding! 🚀

Managing Multi-Account AWS Environments: A Comparison of AWS Organizations, Control Tower, and AWS Config

Eslam Genedy — Mon, 01 Jan 2024 19:42:41 +0000

Introduction:

As organizations continue to scale their presence on Amazon Web Services (AWS), effective management of multiple AWS accounts becomes crucial. AWS provides several services to streamline this process, including AWS Organizations, AWS Control Tower, and AWS Config. In this article, we will explore each of these services, highlighting their features and differences, and offering insights into when to use each to best suit your organizational needs.

1. AWS Organizations:

AWS Organizations is a foundational service that enables you to consolidate multiple AWS accounts into an organization that you create and centrally manage. Key features of AWS Organizations include:

Consolidated Billing: AWS Organizations allows you to consolidate billing across all linked accounts, providing a single payment method for all AWS services used across the organization.
Organizational Units (OUs): You can organize accounts into OUs to reflect your organization's hierarchy, making it easier to apply policies and permissions at various levels.
Service Control Policies (SCPs): AWS Organizations allows the creation of SCPs to set fine-grained permissions, controlling which AWS services and actions are allowed or denied across your accounts.

2. AWS Control Tower:

AWS Control Tower builds upon AWS Organizations to simplify the process of setting up and governing a secure, multi-account AWS environment. It is particularly useful for organizations looking to establish a well-architected baseline. Key features of AWS Control Tower include:

Account Vending: Control Tower automates the process of creating new accounts, ensuring they adhere to best practices and organizational policies from the start.
Guardrails: Control Tower implements guardrails, which are pre-configured policies that help enforce security and compliance standards across all accounts within the organization.
Landing Zone: AWS Control Tower provides a Landing Zone, which is a set of best practices blueprints for setting up an environment using AWS Organizations, making it easier to get started with a secure, multi-account architecture.

3. AWS Config:

AWS Config, on the other hand, focuses on monitoring and assessing the configurations of AWS resources within an account. Key features of AWS Config include:

Configuration History: AWS Config maintains a detailed history of configuration changes, allowing you to understand how and when your resources were modified.
Config Rules: You can define and enforce rules to check the configuration of your AWS resources against desired settings. AWS Config can automatically remediate non-compliant resources.
Resource Relationships: AWS Config provides insights into the relationships between resources, helping you understand dependencies and potential impact of changes.

Comparison:

Scope of Management:
- AWS Organizations: Primarily focused on organizational and billing aspects.
- AWS Control Tower: Extends AWS Organizations, automating the setup of multi-account environments.
- AWS Config: Concentrates on monitoring and managing configurations within individual accounts.
Automation and Governance:
- AWS Organizations and Control Tower provide automated features for setting up and governing multi-account environments.
- AWS Config focuses on monitoring and assessing configurations but lacks the automated environment setup capabilities of Organizations and Control Tower.
Granularity of Control:
- AWS Organizations and Control Tower offer granular control through OUs and guardrails, respectively.
- AWS Config provides detailed insights into resource configurations but does not offer the same level of policy enforcement.
Change Monitoring:
- AWS Organizations and Control Tower focus on the initial setup and ongoing governance.
- AWS Config specializes in monitoring changes to resource configurations.

Conclusion:

AWS Organizations, AWS Control Tower, and AWS Config serve distinct purposes in managing multi-account AWS environments. Organizations and Control Tower are well-suited for organizations seeking automated setup and governance, while AWS Config is essential for monitoring and assessing resource configurations. Combining these services can provide a comprehensive solution, addressing both the initial setup and ongoing management of secure, scalable, and well-architected AWS environments. The choice between them depends on your organization's specific needs and priorities in managing its AWS infrastructure.

DEV Community: Eslam Genedy

Stateless WebSockets? Yes — Using API Gateway + DynamoDB

The Problem With Traditional WebSocket Servers

How API Gateway WebSocket APIs Work

The Architecture

Step 1 — The DynamoDB Table

Step 2 — The $connect Handler

Step 3 — The $disconnect Handler

Step 4 — The Message Handler (Broadcasting)

Step 5 — Infrastructure with SAM / CloudFormation

Advanced: Broadcasting From Outside the WebSocket Context

Client-Side: Handling Reconnection

DynamoDB Access Patterns Summary

What This Architecture Cannot Do

When To Use This Pattern

Recap

From Monolith State to Stateless Microservices: A Real-World Refactor

Phase 1 — Audit: Finding State Where It Shouldn't Be

Identifying Stateful Singletons

The Sticky Session Problem

Phase 2 — Extraction: Breaking State Ownership

Choosing What to Extract First

Phase 3 — SQS for Async State Updates

The Event Bus Abstraction

The Notification Service Consumer

SQS Configuration: Dead-Letter Queue

Phase 4 — Event-Driven Patterns Across Services

Pattern 1: Event Sourcing for Audit Trails

Pattern 2: Saga for Distributed Transactions

Pattern 3: Read Model Projections (CQRS)

Results After 14 Months

Lessons Worth Remembering

The Code That Made the Biggest Difference

Building Serverless CI/CD Pipelines with AWS CodePipeline and CodeBuild

Why Serverless CI/CD?

Key Components of a Serverless CI/CD Pipeline

Step 1: Setting Up CodePipeline and CodeBuild

1.1 Create a CodePipeline

1.2 Add a Build Stage with CodeBuild

Step 2: Automating Deployments with AWS SAM

2.1 Create a SAM Template

2.2 Build and Deploy with SAM

Step 3: Handling Rollbacks and Testing

3.1 Automated Testing

3.2 Rollbacks

Step 4: Advanced Pipeline Features

4.1 Manual Approval Stage

4.2 Canary Deployments

4.3 Environment-Specific Configurations

Best Practices for Serverless CI/CD Pipelines

Code Samples

SAM Template (template.yaml)

Build Specification (buildspec.yml)

Conclusion

Processing Streaming Data and Fetching Files with AWS Lambda and Python

Use Case: Processing Streaming Data and Fetching Files

Key Components:

Step 1: Set Up an S3 Bucket for Streaming Data

Step 2: Create a Lambda Function to Process Streaming Data

Step 3: Configure S3 Event Notifications

Step 4: Fetching Files from External Sources

Example: Fetching a JSON File from an API

Step 5: Testing the Lambda Function

Best Practices for Processing Streaming Data and Files

Conclusion

Introduction to Serverless Computing with AWS: A Deep Dive with Node.js

What is AWS Lambda?

Key Features of AWS Lambda:

AWS Services for Serverless Applications

Building a Serverless Application with AWS and Node.js

Prerequisites:

Step 1: Set Up a DynamoDB Table

Step 2: Create an IAM Role for Lambda

Step 3: Create a Lambda Function to Add Items to DynamoDB

Step 4: Create a Lambda Function to Retrieve Items from DynamoDB

Step 5: Set Up API Gateway

Step 6: Test the Application

Add an Item:

Retrieve Items:

Advanced: Environment Variables and Logging

Step 2 — The `$connect` Handler

Step 3 — The `$disconnect` Handler

SAM Template (`template.yaml`)

Build Specification (`buildspec.yml`)