DEV Community: necologicLabs

Real-Time Data Integration Techniques Using PostgreSQL Foreign Data Wrapper (FDW)

necologicLabs — Sun, 05 Jan 2025 09:12:00 +0000

Introduction

When operating relational databases on AWS, there may be situations where you need to exchange data between multiple databases. For example, you might have Aurora PostgreSQL as your source database containing a Products table, and you want to reference it in real time from a PostgreSQL database (the target) located in a different VPC. You then wish to perform a JOIN with the Transactions table on the target side.

No need for data duplication or ETL
Real-time access to the latest source data
Flexible use for historical aggregation and reporting

These requirements can be met using Foreign Data Wrapper (FDW). In this article, we’ll start by explaining what FDW is, then discuss its advantages and disadvantages, network configurations, and finally walk through the setup steps in order.

What Is Foreign Data Wrapper (FDW)?

Foreign Data Wrapper (FDW) is an extension feature of PostgreSQL that provides a mechanism allowing you to treat external data sources (e.g., tables from other databases) as if they were local tables.

In practice, you install an FDW extension (such as postgres_fdw), then configure “servers,” “user mappings,” and “foreign tables” to reference the tables located on a remote database.

Advantages and Disadvantages of Using FDW

Advantages

Real-Time Access

You can always retrieve the latest data from the source database. There’s no need for ETL batch processing, making it simple to leverage the most up-to-date information.
Improved Development Efficiency

By simply writing SQL JOIN statements, you can seamlessly integrate local and remote tables, simplifying application and data analysis implementations.
Reduced Data Duplication

Compared to a batch approach that duplicates data, FDW can help minimize storage costs and prevent unnecessary data redundancy.

Disadvantages

Performance Risks

Because access is remote, query response times can vary depending on network latency and the load on the source database.
High Dependency on the Source DB

If the source database experiences failures or heavy load, any queries passing through FDW will be directly affected.
Limited Functionality in Some Cases

Certain PostgreSQL features (like transaction management or triggers) may not be available on foreign tables.

Network Configuration Steps (VPC Peering or Transit Gateway)

Even if the source (Aurora PostgreSQL) and the target (PostgreSQL) are in the same account but reside in different VPCs, you need to configure one of the following methods to securely enable network connectivity:

VPC Peering
- Create a peering connection between the source VPC and the target VPC.
- Update the route tables in each VPC to include a route pointing to the peering connection for the relevant IP addresses.
- Open port 5432 (PostgreSQL port) in the security groups’ inbound/outbound rules.
AWS Transit Gateway
- If you need to connect multiple VPCs or multiple Regions from a single hub, use a Transit Gateway.
- Attach each VPC to the Transit Gateway and configure the necessary routes.
- Similarly, make sure security groups and NACLs allow traffic on the required ports.

In either case, be sure to configure the correct security groups to grant access to Aurora.

Step-by-Step Guide to Setting Up FDW

Below is a set of concrete steps assuming the following configuration:

Source: Aurora PostgreSQL (in a separate VPC)
- Endpoint: maindb-instance-1.ctqegcgyi6u1.us-east-1.rds.amazonaws.com
- Contains a Products table
Target: PostgreSQL (in a separate VPC)
- Endpoint: subdb-instance-1.ctqegcgyi6u1.us-east-1.rds.amazonaws.com
- Contains a Transactions table and needs to JOIN with Products

1. Preparation on the Aurora PostgreSQL (Source) Side

VPC Peering or Transit Gateway Configuration
- Ensure that the Aurora VPC and the target PostgreSQL VPC can communicate over the network.
- In the Aurora instance’s security group, allow inbound traffic on port 5432 from the target side.

Table Definition in the Source Database

CREATE TABLE Products (
  product_id UUID PRIMARY KEY,
  product_name TEXT NOT NULL,
  category TEXT NOT NULL,
  price NUMERIC(10, 2) NOT NULL,
  stock INTEGER NOT NULL
);

Assumes data is already populated.

2. Preparation on the PostgreSQL (Target) Side

Launch the Target PostgreSQL Instance
- This could be RDS for PostgreSQL, PostgreSQL on EC2, a container-based setup, etc.
- Make sure port 5432 is also open to accept connections.

Table Definition in the Target Database

CREATE TABLE Transactions (
  transaction_id UUID PRIMARY KEY,
  product_id UUID NOT NULL,
  quantity INTEGER NOT NULL,  
  total_price NUMERIC(10, 2) NOT NULL,
  transaction_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

This will be used in the future together with the foreign_products table (FDW) for JOIN operations.

Installing and Enabling FDW

On the target database (PostgreSQL), make postgres_fdw available for use:

-- Execute on the target database 
CREATE EXTENSION IF NOT EXISTS postgres_fdw;

Note: Depending on your managed service or PostgreSQL version, this extension may be disabled by default. You may need to enable it via the management console or similar interface.

Creating a Server Connection (CREATE SERVER)

CREATE SERVER source_aurora FOREIGN DATA WRAPPER postgres_fdw 
OPTIONS (   
    host 'maindb-instance-1.ctqegcgyi6u1.us-east-1.rds.amazonaws.com',
    port '5432',
    dbname 'postgres'  -- Name of the source database
);

Here, “source_aurora” is simply an identifier.

The “host” and “port” specify the Aurora connection details.

“dbname” is the source database name.

Creating a User Mapping (CREATE USER MAPPING)

Set up the authentication details needed to access the source via FDW:

CREATE USER MAPPING FOR target_user SERVER source_aurora  
OPTIONS (
    user 'aurora_user',
    password 'aurora_password'
);

Sample:

CREATE USER MAPPING FOR postgres SERVER source_aurora  
OPTIONS (  
    user 'postgres',   
    password 'xxx' 
 );

“FOR target_user” refers to a user present on the target PostgreSQL side.

“user” and “password” are valid credentials for Aurora PostgreSQL.

Creating a Foreign Table (CREATE FOREIGN TABLE)

Define the Products table from the Aurora source as a “foreign table” on the target:

CREATE FOREIGN TABLE foreign_products (
    product_id UUID,  
    product_name TEXT,  
    category TEXT,  
    price NUMERIC(10, 2), 
    stock INTEGER
) SERVER source_aurora 
OPTIONS (   
    schema_name 'public',  -- Schema in the source DB
    table_name 'products'  -- Table name in the source DB
);

This allows you to run a simple query, such as SELECT * FROM foreign_products; on the target database, which in turn retrieves data from the Products table in the remote Aurora database.

Tip: For PostgreSQL 9.5 and later, you can use the IMPORT FOREIGN SCHEMA feature to import multiple tables from the source schema at once, which is convenient when you need to work with numerous tables.

Confirming the Configuration

You can verify foreign table information using the following query:

SELECT 
    fs.srvname AS server_name, 
    fs.srvoptions AS server_options, 
    um.umuser AS user_id,
    um.umoptions AS user_options,    
    ft.foreign_table_name AS table_name, 
    ft.foreign_table_schema AS table_schema 
FROM    
    pg_foreign_server fs 
LEFT JOIN    
    pg_user_mappings um
ON 
    fs.oid = um.srvid  -- changed from umserver to srvid 
LEFT JOIN     
    information_schema.foreign_tables ft 
ON 
    fs.srvname = ft.foreign_server_name;

Sample Results

Attribute	Value
server_name	source_aurora
server_options	{host=maindb-instance-1.ctqegcgyi6u1.us-east-1.rds.amazonaws.com, port=5432, dbname=postgres}
user_id	16400
user_options	{user=postgres, password=xxx}
table_name	foreign_products
table_schema	public

Example of a JOIN Query

Once the configuration is set up correctly, you can easily join the target’s Transactions table with the remote foreign_products:

SELECT   
    t.transaction_id,  
    f.product_name,   
    f.category,    
    t.quantity,       
    t.total_price,      
    t.transaction_date  
FROM      
    transactions t 
JOIN   
    foreign_products f
ON    
    t.product_id = f.product_id
WHERE    
    f.category = 'Electronics';

Fields such as f.product_name and f.category come from the Aurora table in real time.
t.transaction_date is a local column in the target (Transactions) table.

This makes it straightforward to leverage the latest source database information on the target side.

Sample Results (Excerpt)

            transaction_id            | product_name |  category   | quantity | total_price |  transaction_date   
--------------------------------------+--------------+-------------+----------+-------------+---------------------
 f8b8b097-75ce-47e6-8fb8-42adbf3456fc | Growth       | Electronics |        4 |      421.16 | 2025-01-02 21:57:27
 8b8a1390-6f24-4d07-9239-adc5df9ab92e | Lose         | Electronics |        6 |      448.08 | 2025-01-03 03:29:54
 86386a4b-4eda-4ff4-9854-87b65d58e3aa | Without      | Electronics |        7 |     2537.57 | 2025-01-01 20:44:28
 203291b1-24af-443e-ad34-092151a99c61 | Wonder       | Electronics |        8 |     2098.00 | 2025-01-02 14:48:15
 fd75b3fa-f564-4599-9faf-c29dd42f82e3 | How          | Electronics |        3 |     1176.63 | 2025-01-02 15:22:55
 42a281a5-de30-4bf2-b528-bd3b942e35b6 | Pattern      | Electronics |        2 |      186.56 | 2025-01-01 06:54:58
 03a7c005-9f1c-46ca-b102-352dd0ed32ae | Win          | Electronics |        6 |     2955.60 | 2025-01-01 13:39:38
 6bc0bdd8-0d9e-47a8-853a-69ac235ca983 | Successful   | Electronics |        2 |      986.18 | 2025-01-02 23:59:29
 aaffdb6e-dbd1-4f33-a788-d62bdbf012bd | Every        | Electronics |        1 |      184.57 | 2025-01-02 07:48:06
 76f0a5db-297a-42fc-b8e6-d0e874031471 | Sing         | Electronics |        3 |     1426.11 | 2025-01-03 02:17:25
 b1dd0e57-4b49-413c-9514-aeb879200138 | Paper        | Electronics |        5 |      481.60 | 2025-01-01 04:53:51
 23f60790-6cb8-4c67-8c2c-60afe30289fb | Teacher      | Electronics |        6 |      429.30 | 2025-01-02 21:37:51
 638f7951-7a70-4da5-b87d-41e681143811 | Society      | Electronics |        3 |      302.22 | 2025-01-04 04:47:11

Performance

In PostgreSQL’s Foreign Data Wrapper (FDW), there is no built-in caching mechanism. Each time you access a foreign table, a query is sent to the remote database (the server configured for FDW) to retrieve the latest data.

Below are some performance measurements taken when retrieving data within the same AWS Region. The following query yielded these results:

(5987 rows) Time: 1094.468 ms (00:01.094)

Overall, this performance is quite good and can be considered practical for real-world use.

Tuning 1

PostgreSQL Foreign Data Wrapper (FDW) can push parts of query planning to the remote database to streamline communication. This is often referred to as query pushdown. Thanks to pushdown, not all data must be retrieved every time.

Pushdown Explained

FDW minimizes unnecessary data transfer by sending parts of the query to the remote database. For instance:

WHERE clauses
LIMIT
Some JOIN conditions
Queries that select only specific columns (SELECT column_name)

Query Optimization Tuning

Sample Query

To illustrate room for improvement, let’s consider a less optimal SQL query:

SELECT 
    p1.product_name,
    SUM(t.quantity) AS total_sold
FROM   
    transactions t 
JOIN (    
    SELECT 
        * 
    FROM
        foreign_products ) p1
ON
    t.product_id = p1.product_id 
WHERE 
    t.transaction_date >= '2022-01-01' 
AND  
    p1.category = 'Electronics'
AND  
    p1.product_id 
IN (
    SELECT
        product_id
    FROM
        foreign_products   
    WHERE 
        stock > 10
 )
GROUP BY
    p1.product_name;

(830 rows) Time: 531.426 ms

Although not terrible, it can be improved. Let’s analyze the query plan via EXPLAIN VERBOSE (sql)

Problem Analysis

Double Scanning of the Remote Table

The remote table foreign_products is being scanned twice:

First scan:

SELECT 
    product_id,
    product_name
FROM
    public.products 
WHERE
    ((category = 'Electronics'))

- Retrieves 6 rows for `category = 'Electronics'`.

Second scan:

SELECT
    product_id 
FROM 
    public.products 
WHERE
     ((stock > 10))

- Retrieves 683 rows for `stock > 10`.

Problem:

- The remote SQL is executed twice, increasing the communication cost with the remote database.
- Data is unnecessarily processed by a local Hash Join.

Unnecessary Local Processing

- There is a local Hash Join happening.
- Conditions like `category` and `stock` that could be handled remotely are instead processed locally, reducing efficiency.

Full Table Scan on the Local transactions Table

- The local `transactions` table is undergoing a Seq Scan:

    `Seq Scan on public.transactions t (cost=0.00..685.00 rows=29997 width=20)    Filter: (t.transaction_date >= '2022-01-01')`

- Impact:
    - Because there is no index on `transaction_date`, all 29,997 rows are being scanned.

Points for Improvement

Perform JOIN and condition filtering remotely

Apply conditions (category = 'Electronics' and stock > 10) on the remote database to reduce unnecessary local joins. For instance:

WITH filtered_products AS (
    SELECT
    product_id, 
    product_name   
    FROM       
    foreign_products    
    WHERE        
    category = 'Electronics'   
    AND         
    stock > 10 
)
SELECT     
    p.product_name,   
    SUM(t.quantity) AS total_sold 
FROM   
    transactions t 
JOIN      
    filtered_products p      
ON 
    t.product_id = p.product_id
WHERE    
    t.transaction_date >= '2022-01-01' 
GROUP BY
    p.product_name;

Add an Index to the Local Table

Create an index on transaction_date in the transactions table to avoid a full table scan.

Results

(830 rows) Time: 387.544 ms

This is roughly a 30% reduction in execution time.

From the query plan, you can see that remote access has been consolidated into a single call. While this example is not unique to foreign tables, inefficiencies in the query become more evident when using FDW. Hence, analyzing queries with EXPLAIN is crucial for avoiding performance issues.

Tuning 2 – Using Materialized Views for Caching

As another method of performance tuning, consider using caching. Since FDW does not provide built-in caching by default, you can use a Materialized View to cache data locally. This approach offloads frequent remote queries and complex processes from the remote database to the local environment, reducing load on the remote data source and further shortening query execution times.

Materialized View

Below is an example of caching the entire foreign_products table:

CREATE MATERIALIZED VIEW products AS SELECT * FROM foreign_products;

SQL

Next, use the materialized view to filter data. In the following query, only the reference to the table has been changed from foreign_products to the materialized view named products.

WITH filtered_products AS (
    SELECT    
        product_id, 
        product_name   
    FROM       
        products    
    WHERE         
        category = 'Electronics' 
    AND          
        stock > 10 
) 
SELECT    
    p.product_name, 
    SUM(t.quantity) AS total_sold
FROM     
    transactions t 
JOIN     
    filtered_products p 
ON 
    t.product_id = p.product_id
 WHERE 
    t.transaction_date >= '2022-01-01' 
GROUP BY 
    p.product_name;

(830 rows) Time: 353.511 ms

This yields roughly a 10% reduction in execution time. The exact effect will vary depending on data volume, query complexity, master database location, and how often the view is accessed. In this example, the entire table was cached, but caching just the subset of data you need may improve performance even more.

Because a materialized view will not reflect real-time data updates, its use depends on your requirements. For instance, in a batch workflow, you could refresh the materialized view (to get the latest data) before running the main process, then use the view locally for faster batch processing. This can be very effective.

Operational Points and Considerations

Performance Monitoring

- Since FDW is a remote connection, network latency may significantly impact queries.
- If you perform frequent JOIN operations, consider indexing and caching strategies on the source side as needed.

Load on the Source Database

- FDW directly references the latest data on the source. Repeatedly issuing heavy queries can affect the source’s performance.
- Monitor query frequency and complexity, and consider using materialized views if appropriate.

Permissions and Security

- In addition to VPC peering and security group settings, ensure proper roles and privileges are set at the database level to restrict unwanted access.

Combining Partial Data Migration

- If certain data does not need to be accessed in real time, using ETL or snapshots to bring it into the target might be faster.
- FDW is not a one-size-fits-all solution; combine various approaches based on the specific use case.

Dependency on PostgreSQL

- Because this approach depends on PostgreSQL features, it may constrain future system architecture decisions. Conduct sufficient reviews before committing to this design.

Summary

By using Foreign Data Wrapper (FDW), you can JOIN tables across Aurora PostgreSQL (source) and Aurora PostgreSQL (target) even if they reside in separate VPCs, making them appear as if they were part of the same database. Proper network configuration (VPC Peering or Transit Gateway) and basic FDW setup (server, user mapping, and foreign tables) enable real-time data access with minimal effort.

Advantages: Real-time access, enhanced development efficiency, reduced data duplication
Disadvantages: Potential performance risks, dependency on the source DB, limited functionality in certain scenarios

The key is to set up your network, security, and source-side optimizations with care. This approach is especially effective for scenarios where on-demand access to the latest data is required alongside seamless local table integration on the target side. Consider introducing it in your next project if it meets your requirements.

Mastering AWS Incident Management: Automating Responses with Systems Manager Incident Manager

necologicLabs — Fri, 03 Jan 2025 07:06:36 +0000

Overview

When handling increased error rates in AWS Lambda, categorizing errors and defining escalation paths is crucial. This guide demonstrates how to use AWS Systems Manager Incident Manager to automatically handle and escalate incidents effectively. The workflow involves collecting error details using Runbooks and notifying stakeholders through Amazon SNS.

Why Use AWS Systems Manager Incident Manager?

AWS Systems Manager Incident Manager provides centralized management for incident response within AWS environments. Key benefits include:

Native AWS Integration: Seamlessly integrates with services like Amazon CloudWatch, AWS Lambda, and Amazon EventBridge.
Runbook Automation: Facilitates automated or semi-automated workflows to troubleshoot and address incidents.
Multi-Channel Notifications: Supports notifications via Amazon SNS, Slack, and Amazon Chime.
Cost Efficiency: A viable alternative to commercial solutions for small-to-medium environments.

Limitations

For large-scale organizations requiring detailed reporting, complex team hierarchies, and multi-layer escalation flows, specialized tools like PagerDuty or ServiceNow may be more appropriate.

Architecture Overview

The architecture monitors AWS Lambda functions for errors using CloudWatch Alarms. Incident Manager automatically creates incidents and executes Runbooks for error handling and notifications.

Error Scenarios

Error A: Standard incident with email notifications.
Error B: Critical incident requiring SMS notifications and escalations.

CloudWatch Alarms are configured to distinguish between these error types, triggering specific incident responses accordingly.

Step-by-Step Configuration

Step 1: Create CloudWatch Alarms for Lambda Errors

Example Lambda Function:

import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    error_type = event.get("errorType")

    try:
        if error_type == "A":
            logger.error("Error A: A standard exception occurred.")
            raise Exception("Error A occurred")
        elif error_type == "B":
            logger.error("Error B: A critical runtime error occurred.")
            raise RuntimeError("Critical Error B occurred")
        else:
            logger.info("No error triggered.")
            return {"statusCode": 200, "body": "Success"}
    except Exception as e:
        logger.exception("An error occurred: %s", e)
        raise

Configure CloudWatch Metrics and Alarms:

Metrics Filters: Create filters for Error A and Error B.

Alarms: Link these filters to alarms with appropriate thresholds and periods.

Alarm Actions: Set up triggers to initiate Incident Manager workflows.

Step 2: Set Up Incident Manager

Enable Incident Manager:

- Navigate to the Incident Manager settings in the AWS Management Console and onboard your account.

Step 3: Configure Notification Contacts

Email: Notify administrators for Error A.

SMS: Notify stakeholders for Error B escalation.

Step 4: Define Escalation Plans

Error A: Email notification followed by SMS if unresolved.
Error B: Immediate SMS notification.

Step 5: Create a Runbook

Runbook Template:

schemaVersion: '0.3'
description: "Runbook for Lambda error handling."
parameters:
  AlarmName:
    type: String
    description: "Name of the CloudWatch Alarm"
  snsTopicArn:
    type: String
    description: "SNS Topic ARN for notifications"
mainSteps:
  - name: GatherErrorLogs
    action: aws:executeScript
    inputs:
      Runtime: python3.8
      Script: |
        def script_handler(events, context):
            return {"status": "Logs gathered"}
  - name: SendNotification
    action: aws:executeAwsApi
    inputs:
      Service: sns
      Api: Publish
      TopicArn: "{{ snsTopicArn }}"
      Message: "Lambda Errors detected. AlarmName: {{ AlarmName }}"

Step 6: Create Response Plans

Define separate response plans for Error A and Error B.
Link Runbooks and notification channels to each response plan.

Step 7: Link CloudWatch Alarms to Incident Manager

Edit alarm actions to trigger the corresponding Incident Manager response plans.

Demo

Commercial Tools Comparison

Feature	AWS Incident Manager	PagerDuty	ServiceNow
Cost Efficiency	High	Medium	Low
AWS Integration	Seamless	Limited	Limited
Escalation Flexibility	Moderate	High	High
Reporting and Analytics	Basic	Advanced	Advanced

Ideal Use Cases for AWS Incident Manager:

Small-to-medium environments with AWS-centric architectures.
Simple escalation and notification needs.
Cost-sensitive deployments.

Conclusion

AWS Systems Manager Incident Manager is a cost-effective tool for incident response in AWS-centric environments. While it lacks some advanced features of commercial solutions, it offers robust integration with AWS services and sufficient functionality for many use cases. Its ease of setup and low cost make it an attractive choice for small to medium-scale operations.

References

Distributed Tracing in Microservices: AWS X-Ray vs DataDog

necologicLabs — Sat, 28 Dec 2024 13:18:08 +0000

Overview

Microservice architectures significantly enhance scalability and development efficiency by dividing applications into smaller, independent services. However, they also introduce challenges such as visualizing service communication, monitoring performance, and identifying root causes of failures.

This article explores the importance of distributed tracing and provides a detailed comparison of two leading tools: AWS X-Ray and DataDog. Additionally, we demonstrate a practical example using a sample application flow: FunctionA → SQS → FunctionB, highlighting key points for implementation.

https://www.splunk.com/ja_jp/data-insider/what-is-distributed-tracing.html

Distributed Tracing in Microservices: AWS X-Ray vs DataDog
- Overview
- Table of Contents
- Introduction: The Importance of Distributed Tracing in Microservices
- Challenges of Debugging and Tracing in Microservices
- Why AWS X-Ray and DataDog?
- AWS X-Ray
- DataDog
- Feature Comparison
- Tracing Demo with a Sample Application
- 5.1 Architecture Overview
- 5.2 Key Points for Tracing with AWS X-Ray
- 5.3 Key Points for Tracing with DataDog
- Cost and Optimization
- AWS X-Ray
- DataDog
- Choosing the Right Tool
- Conclusion
- References

Introduction: The Importance of Distributed Tracing in Microservices

Microservices divide applications into smaller, manageable services to improve scalability and development efficiency. However, this architecture also presents challenges:

Visualizing Service Communication: Tracking how services interact and ensuring smooth communication.
Identifying Problem Areas: Quickly locating latency bottlenecks or errors between services.
Improving Overall Performance: Understanding bottlenecks to implement optimization strategies.

https://www.jaegertracing.io/docs/1.33/architecture

Distributed tracing addresses these issues by providing a clear view of the entire workflow across services.

Challenges of Debugging and Tracing in Microservices

Distributed tracing is essential due to these specific challenges in microservices:

Distributed Logging: Logs are spread across independent services, making collection and analysis complex.
Complexity in Debugging: Identifying failures in a system with multiple dependencies is more challenging.
Observability Requirements: Metrics, logs, and traces are necessary to provide a comprehensive view of the system’s health and performance.

Why AWS X-Ray and DataDog?

Among the many tracing tools available, AWS X-Ray and DataDog stand out for their capabilities and compatibility with various use cases.

AWS X-Ray

Tight Integration with AWS: Simplifies implementation with AWS services like Lambda, ECS, and Fargate.
Transparent Pricing: Easy to calculate costs alongside AWS resources.
Service Map Visualization: Provides a clear view of dependencies between services.

DataDog

Multi-Cloud and Hybrid Support: Works seamlessly across AWS, GCP, Azure, and on-premises environments.
Comprehensive Observability: Combines tracing, logging, and infrastructure monitoring into a single platform.
Highly Customizable Dashboards: Offers rich visualization with tag-based filtering.

Feature Comparison

Feature	AWS X-Ray	DataDog
Visualization	Service Map for dependency analysis	Advanced dashboards with service mapping
Instrumentation	SDK-based, manual or automatic	Automatic via agent or library integration
Supported Platforms	AWS-centric (EC2, Lambda, etc.)	Multi-cloud and on-premises
Metrics Integration	Works seamlessly with CloudWatch	Highly customizable external integrations
Log Management	CloudWatch Logs integration	Built-in log management (paid)
UI/Customization	Simple and functional	Highly customizable, modern UI

Tracing Demo with a Sample Application

5.1 Architecture Overview

We’ll use a simple serverless flow for this demonstration:

[API Gateway] → (FunctionA) → [SQS] → (FunctionB)

FunctionA: Receives requests from API Gateway and queues messages in SQS.
SQS: Processes messages asynchronously and triggers FunctionB.
FunctionB: Processes SQS messages, writes to a database, and logs results.

5.2 Key Points for Tracing with AWS X-Ray

Setup: Enable Active Tracing in Lambda and integrate X-Ray SDK for additional details.
Visualization: X-Ray’s Service Map displays API Gateway, Lambda, and SQS as connected components, showing processing times and errors.
Detailed Analysis: Identify cold starts and bottlenecks within the service chain.
Considerations: While great for AWS environments, X-Ray lacks support for multi-cloud setups.

5.3 Key Points for Tracing with DataDog

Setup: Use DataDog Lambda Library or agent for seamless integration.
Tags and Filtering: Tag services and requests with meaningful labels (e.g., environment or version).
Rich Dashboards: Use APM’s Service Map and built-in logs for comprehensive observability.

Considerations: Ideal for hybrid and multi-cloud systems but may involve higher costs.

Cost and Optimization

AWS X-Ray

Pay-per-Trace: Costs are proportional to the number of traces and data volume.
Unified with AWS Services: Simplifies budget management for AWS-only environments.

DataDog

Module-Based Pricing: Separate charges for APM, logging, and infrastructure monitoring.
Efficiency Gains: Reduces overhead by consolidating observability tools.

Choosing the Right Tool

Scenario	AWS X-Ray	DataDog
AWS-Centric Projects	Best for AWS-only architectures	Handles multi-cloud or hybrid environments
Serverless Workflows	Excellent with Lambda and API Gateway	Supports serverless and multi-cloud setups
Team Requirements	Works well for small teams	Suitable for DevOps/SRE teams with large setups
Budget Constraints	Cost-effective for AWS-only use cases	Flexible but potentially expensive

Conclusion

Distributed tracing is essential for visualizing complex workflows, identifying bottlenecks, and improving microservice performance. AWS X-Ray is an excellent choice for AWS-centric serverless projects due to its seamless integration and cost-effectiveness. DataDog, on the other hand, excels in hybrid or multi-cloud setups with its rich features and flexibility.

References

AWS SAM vs. Serverless Framework: Choosing the Right Framework for Serverless Development

necologicLabs — Fri, 27 Dec 2024 13:22:49 +0000

Introduction

The Challenge

As serverless development becomes more popular, many developers rely on frameworks like AWS SAM and Serverless Framework to build applications. However, deciding which tool to use can be challenging, especially when their functionalities overlap.

Purpose of This Article

This article compares AWS SAM and Serverless Framework, highlighting their features, use cases, and strengths. By the end, you’ll be able to choose the best tool for your project.

Background: The Growth of Serverless Architecture

Serverless architecture has become a compelling choice in recent years, offering numerous advantages.

1. The Evolution of Cloud Technology

Cloud technology has evolved through three significant phases:

Virtual Machines (VMs): Easy to migrate but require infrastructure management.
Containers: Lightweight virtualization enhancing portability and scalability.
Serverless: Offloads all infrastructure management to cloud providers, allowing developers to focus solely on application logic.

2. Business Drivers Behind Serverless Adoption

Enterprises prioritize:

Development Speed: Focus on building applications without managing runtime environments.
Cost Efficiency: Pay-as-you-go pricing minimizes wasted resources.
Scalability: Built-in autoscaling adjusts automatically to traffic demands.

3. Technical Benefits

Serverless architecture offers:

Automated Scaling: Handles concurrent executions seamlessly.
Improved Productivity: Developers focus on logic, not infrastructure.
Compatibility: Works well with modern architectures like microservices and event-driven systems.

4. Cloud Ecosystems

AWS Lambda, API Gateway, and DynamoDB exemplify how serverless ecosystems enable rapid development. Similar offerings from Google Cloud and Azure further validate serverless as a mainstream choice.

Why AWS SAM and Serverless Framework?

Key Reasons

Market Presence:
- AWS SAM: AWS’s official tool for serverless applications.
- Serverless Framework: A leading choice for multi-cloud environments.
Developer Support:
- AWS SAM integrates deeply with AWS resources.
- Serverless Framework provides extensive plugins and community-driven support.
Complementary Use Cases: These tools cater to distinct needs, making them excellent candidates for comparison.

Points of Comparison

Implementation and Development Efficiency: How each tool structures applications and supports productivity.
Deployment and Operational Flexibility: Differences in deployment workflows and constraints.
Integration with AWS Services: SAM’s AWS-centric features versus Serverless Framework’s flexibility.

Sample Application: Task Management

Overview

This sample application demonstrates:

DynamoDB: Manages task data (e.g., id, task, status).
API Gateway and Lambda: Provides REST endpoints.
DynamoDB Streams and SNS: Sends email notifications for task status updates.

API Endpoints

Method	URL	Description
GET	`/task?id=1`	Retrieves task by ID.
POST	`/task`	Updates or creates a task.

AWS SAM: AWS-Centric Power

Strengths

AWS Integration: Seamless support for AWS services like DynamoDB and SNS.
Robust Local Testing: High fidelity testing with sam local.
Best Use Case: Ideal for AWS-heavy environments.

Limitations

Limited support for non-AWS services.
Complex configurations may require familiarity with CloudFormation.

Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Description: >
  sam-task-app - A simple task management application with DynamoDB, API Gateway, and SNS.

Resources:
  TaskTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: TaskTable
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      BillingMode: PAY_PER_REQUEST
      StreamSpecification:
        StreamViewType: NEW_AND_OLD_IMAGES

  TaskApiFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: task-api/app.lambdaHandler
      Runtime: nodejs18.x
      CodeUri: task-api/
      Environment:
        Variables:
          TABLE_NAME: TaskTable
      Policies:
        - DynamoDBCrudPolicy:
            TableName: TaskTable
      Events:
        TaskApi:
          Type: Api
          Properties:
            Path: /task
            Method: any

  StreamHandlerFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: stream-handler/app.lambdaHandler
      Runtime: nodejs18.x
      CodeUri: stream-handler/
      Environment:
        Variables:
          TOPIC_ARN: !Ref NotificationTopic
      Policies:
        - Statement:
            Effect: Allow
            Action: sns:Publish
            Resource: !Ref NotificationTopic
      Events:
        StreamEvent:
          Type: DynamoDB
          Properties:
            Stream: !GetAtt TaskTable.StreamArn
            StartingPosition: TRIM_HORIZON

  NotificationTopic:
    Type: AWS::SNS::Topic
    Properties:
      Subscription:
        - Protocol: email
          Endpoint: xxx@gmail.com

Serverless Framework: Flexibility First

Strengths

Multi-Cloud Support: Works with AWS, Azure, GCP, and more.
Plugin Ecosystem: Highly customizable through plugins.
Best Use Case: Suitable for diverse, non-AWS-exclusive projects.

Limitations

Lacks built-in AWS-specific configurations like SAM’s Policies shortcuts.
May require extra setup for advanced AWS integrations.

Template

org: neclogoiclabs
app: sls-task-app
service: sls-test

package:
  individually: true
  exclude:
    - ./**

plugins:
  - serverless-plugin-common-excludes
  - serverless-plugin-include-dependencies
  - serverless-iam-roles-per-function

provider:
  name: aws
  runtime: nodejs18.x
  region: us-east-1

functions:
  apiHandler:
    handler: src/api/index.apiHandler
    package: 
      include:
        - src/api/**
    events:
      - http:
          path: task
          method: any
    iamRoleStatements:
      - Effect: Allow
        Action:
          - dynamodb:GetItem
          - dynamodb:PutItem
        Resource: 
          !GetAtt TaskTable.Arn

  streamHandler:
    handler: src/stream/index.streamHandler
    package: 
      include:
        - src/stream/**    
    environment:
      TOPIC_ARN: !Ref TaskNotificationTopic
    events:
      - stream:
          type: dynamodb
          arn:
            Fn::GetAtt: [TaskTable, StreamArn]
    iamRoleStatements:
      - Effect: Allow
        Action:
          - sns:Publish
        Resource:
          - !Ref TaskNotificationTopic
resources:
  Resources:
    TaskTable:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: TaskTable
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        BillingMode: PAY_PER_REQUEST
        StreamSpecification:
          StreamViewType: NEW_AND_OLD_IMAGES

    TaskNotificationTopic:
        Type: AWS::SNS::Topic
        Properties:
          TopicName: TaskNotificationTopic
          Subscription:
            - Protocol: email
              Endpoint: xxx@gmail.com

Conclusion

Both frameworks have unique strengths. Consider the following when choosing:

Choose AWS SAM for AWS-specific projects requiring deep service integration.
Choose Serverless Framework for multi-cloud scenarios or projects needing high customization.

References

Microservices Clean Architecture: Key Design Points and Migration Strategies

necologicLabs — Fri, 27 Dec 2024 13:13:24 +0000

In this article, we'll explore how to apply Clean Architecture principles when transitioning from a monolithic system to microservices. We'll address common challenges such as:

Designing adapter layers (for database access and external API calls)
Handling master data across multiple services
Dealing with code duplication
Managing performance impacts when table joins are no longer straightforward
Planning a phased approach to migrate from a monolith

Whether you're starting fresh with microservices or modernizing a legacy system, these insights will help you build a flexible, maintainable architecture.

1. Clean Architecture × Microservices Overview

1.1 What Is Clean Architecture?

Clean Architecture emphasizes separating business logic (the application core) from external elements (databases, UI, external APIs). At its heart:

Application Core (Use Cases, Entities): Encapsulates the essential business rules.
Adapters (or Interface/Infrastructure Layer): Handles external operations (DB queries, API calls, I/O) while shielding the core from infrastructure details.

1.2 What Is Microservices Architecture?

Microservices break a system into independently deployable services, each owning its own data store. Communication occurs via network calls (REST, gRPC, messaging, etc.). This approach increases development agility and scalability but also introduces concerns about data consistency, service orchestration, and operational overhead.

1.3 Why Combine the Two?

Clean Architecture: Strengthens each microservice internally so that core business logic remains independent of infrastructure details.
Microservices: Allows teams to scale and release features at the service level.

By mixing Clean Architecture with microservices, you maintain loose coupling and high scalability, while preserving clarity in your domain logic.

2. The Adapter Layer — DB Access & External API Calls

2.1 Role of the Adapter Layer

In Clean Architecture, you avoid pushing external dependencies directly into the core. Instead, all external I/O, such as database queries or API requests, happens in an adapter layer. The core interacts only with abstract interfaces (ports), making it easy to switch implementations.

2.2 DB Access Adapter

Each microservice typically manages its own database.
The adapter handles all CRUD operations, encapsulating details like SQL or ORM usage.
The core (use cases) relies on an interface that the DB adapter implements.

2.3 External API Integration Adapter

For cross-service or third-party API calls, the adapter manages protocols, request/response formats, and error handling (e.g., retries, circuit breakers).
This approach keeps the core’s business logic independent of external service details.

3. Master Data Handling — Turning Common Data into a Separate Service

3.1 What Is a Master Data Service?

When multiple services need the same reference data (e.g., country lists, category definitions), you can centralize it in a Master Data Service.

Ensures consistent data usage across services
Simplifies updates by localizing changes to one service

3.2 Pros and Cons

Pros
- Strong data consistency
- Single point for updates
Cons
- Potential single point of failure if not designed with redundancy
- Increased latency from external API calls

Mitigation Tips:

Use caching (e.g., Redis) to avoid excessive calls.

Implement circuit breakers and retries for resilience.

4. Code Duplication — Balancing DRY and Service Independence

4.1 The DRY Principle vs. Microservice Independence

Ideally, you reduce duplication by placing common logic into libraries or SDKs.
However, microservices often deliberately allow some duplication to keep services truly decoupled, each evolving at its own pace.

4.2 Minimizing Duplication

Create a Common Library
- Distribute shared API client code, data transformation logic, or utility functions.
Use an API Gateway or Service Mesh
- Centralize cross-cutting concerns like authentication, routing, or retries.
Leverage Shared Caching Strategies
- Provide a standardized approach so each service doesn’t reimplement the same patterns.

Table Example:

Method	Pros	Cons
Common Library (SDK)	Easy to apply DRY; single source of truth	Version conflicts across services
API Gateway	Simplifies client requests; reduces overhead	Gateway can become a single point of failure
Intentional Duplication	Each service can evolve independently	Fixes/updates must be applied in multiple places

5. Dealing with the Loss of Table Joins & Performance Impacts

5.1 From Monolith to Distributed Data

In a monolith, you can easily use JOIN queries. In microservices, each service owns its own data, making direct joins impossible. This demands alternative solutions for queries spanning multiple services.

5.2 Data Integration Patterns

CQRS (Command Query Responsibility Segregation)
- Separate read models from write models.
- Maintain a read-optimized view (materialized view or “query side”) using events or batch updates.
Data Aggregation Service
- A specialized service that fetches data from multiple services, combines it, and returns a single response.
- Frontend sees only one endpoint.
Caching
- Use Redis or in-memory caches to store frequently accessed, aggregated data for faster responses.

6. A Phased Migration from Monolith to Microservices

6.1 The Strangler Fig Pattern

Gradually replace parts of the legacy monolith with new microservices, leaving the old system running until it’s safe to switch over.

6.2 Step-by-Step Approach

API-Enable the Monolith
- Refactor direct DB calls into internal APIs.
Redirect to New Services
- As you introduce microservices, route specific functionality from the monolith to the new system.
Gradual Cutover
- Retire monolith components in stages, once the new services are stable and fully tested.

7. Summary & Key Takeaways

Apply Clean Architecture
- Keep your core business logic separate from infrastructure details with a well-defined adapter layer.
Centralize Common Data
- If multiple services rely on the same data, consider a Master Data Service plus caching and fault tolerance strategies.
Balance Code Duplication
- Be mindful of DRY but accept that some duplication may keep services independent and easier to maintain.
Address JOIN Limitations
- Adopt patterns like CQRS or a data aggregation service to replicate the convenience of table joins without merging databases.
Migrate in Phases
- Use the Strangler Fig pattern to introduce microservices incrementally, reducing risk and ensuring steady progress.

By combining microservices with Clean Architecture, you gain a scalable, maintainable solution that keeps domain logic clean. While you lose the simplicity of monolithic table joins, you gain flexibility, autonomy for each service, and the potential for more robust, scalable applications over the long run.

References

The Clean Architecture (by Uncle Bob) https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html
Microservices Patterns (by Chris Richardson) https://microservices.io/patterns/index.html
Strangler Fig Pattern (Martin Fowler) https://martinfowler.com/bliki/StranglerFigApplication.html
CQRS (Command Query Responsibility Segregation) – Martin Fowler https://martinfowler.com/bliki/CQRS.html

DEV Community: necologicLabs

Real-Time Data Integration Techniques Using PostgreSQL Foreign Data Wrapper (FDW)

Introduction

What Is Foreign Data Wrapper (FDW)?

Advantages and Disadvantages of Using FDW

Advantages

Disadvantages

Network Configuration Steps (VPC Peering or Transit Gateway)

Step-by-Step Guide to Setting Up FDW

1. Preparation on the Aurora PostgreSQL (Source) Side

2. Preparation on the PostgreSQL (Target) Side

Example of a JOIN Query

Sample Results (Excerpt)

Performance

Tuning 1

Pushdown Explained

Query Optimization Tuning

Sample Query

Problem Analysis

Points for Improvement

Results

Tuning 2 – Using Materialized Views for Caching

Materialized View

SQL

Mastering AWS Incident Management: Automating Responses with Systems Manager Incident Manager

Overview

Why Use AWS Systems Manager Incident Manager?

Limitations

Architecture Overview

Error Scenarios

Step-by-Step Configuration

Step 1: Create CloudWatch Alarms for Lambda Errors

Example Lambda Function:

Configure CloudWatch Metrics and Alarms:

Step 2: Set Up Incident Manager

Step 3: Configure Notification Contacts

Step 4: Define Escalation Plans

Step 5: Create a Runbook

Step 6: Create Response Plans

Step 7: Link CloudWatch Alarms to Incident Manager

Demo

Commercial Tools Comparison

Ideal Use Cases for AWS Incident Manager:

Conclusion

References

Distributed Tracing in Microservices: AWS X-Ray vs DataDog

Overview

Table of Contents

Introduction: The Importance of Distributed Tracing in Microservices

Challenges of Debugging and Tracing in Microservices

Why AWS X-Ray and DataDog?

AWS X-Ray

DataDog

Feature Comparison

Tracing Demo with a Sample Application

5.1 Architecture Overview

5.2 Key Points for Tracing with AWS X-Ray

5.3 Key Points for Tracing with DataDog

Cost and Optimization

AWS X-Ray

DataDog

Choosing the Right Tool

Conclusion

References

AWS SAM vs. Serverless Framework: Choosing the Right Framework for Serverless Development

Introduction

The Challenge

Purpose of This Article

Background: The Growth of Serverless Architecture

1. The Evolution of Cloud Technology

2. Business Drivers Behind Serverless Adoption

3. Technical Benefits

4. Cloud Ecosystems

Why AWS SAM and Serverless Framework?

Key Reasons

Points of Comparison

Sample Application: Task Management

Overview

API Endpoints

AWS SAM: AWS-Centric Power