Leveraging AWS API Destination to Trigger External APIs

Introduction

Last year, while developing an AWS solution to integrate Salesforce and Microsoft Dynamics NAV, I was tasked with designing a system to capture DynamoDB stream updates and send them to both platforms. During this process, I discovered AWS API Destinations, which provided a seamless way to securely invoke external APIs, simplifying API call management and authentication.

AWS EventBridge API Destinations allow users to configure rules that securely send events to external APIs. They improve reliability by automatically retrying failed requests based on built-in retry policies. This article explores how to use API Destinations, referencing a high-level architecture diagram.

High-Level Architecture explanation

The architecture consists of the following components:

• AWS DynamoDB: Acts as the primary data store.

• Stream Handler (AWS Lambda): This listens for changes in the DynamoDB stream and sends updates to EventBridge.

• EventBridge Event Bus: Routes events to API Destination.

• CloudWatch Logs: Captures logs for monitoring and debugging.

• API Destination: Acts as an intermediary to send requests to external APIs.

• DLQ (Dead Letter Queue): Captures failed event deliveries for further analysis.

• AWS Secret Manager: Manages credentials for secure API authentication.

• Connection: Defines the authorization method, credentials, and network connectivity for EventBridge to communicate securely with the external webhook.

• Webhook: The final external API endpoint that receives the triggered request.

Important AWS Considerations

• Only Public APIs Supported: API Destinations support only public domain names with publicly trusted certificates for HTTPS endpoints. Private APIs that are not publicly resolvable and mutual TLS (mTLS) authentication are not supported.

• Execution Timeout: Requests to API Destinations must complete within 5 seconds. If the target endpoint takes longer, EventBridge will time out the request and retry it according to the maximums set in the configured retry policy.

Use Cases for AWS API Destination

• Third-party Service Integration: You can easily integrate AWS event-driven architectures with SaaS platforms like Salesforce, Slack, or Stripe.

• Webhook-based Notifications: Automate event-driven notifications to external services.

• Cross-Cloud Communications: Bridge AWS events with APIs hosted on other cloud providers.

• Security and Compliance Monitoring: Send security logs to third-party SIEM tools for real-time analysis.

• Automated Data Synchronization: Push updates from AWS services to external databases or applications.

Best Practices for Using API Destination

• Optimize API Response Time: Since API Destination has a 5-second timeout, ensure that the external API is optimized for quick responses. Use caching or lightweight processing where possible.

• Utilize Dead Letter Queues (DLQ): Configure DLQs to capture failed API calls for debugging and retrying later.

• Monitor with CloudWatch Logs: Enable logging to track API calls, errors, and response times for troubleshooting and optimization.

• Leverage AWS Secrets Manager: Store API keys and authentication credentials securely instead of hardcoding them in API Destination configurations.

• Set Up Rate Limits: To prevent exceeding third-party API limits, use throttling mechanisms or request prioritization strategies.

• Use Exponential Backoff for Retries: Configure EventBridge retry policies to use exponential backoff, reducing the likelihood of overloading external APIs.

• Test with Mock APIs: Before integrating with a production API, use mock APIs to validate EventBridge configurations and request formatting.

• Implement Security Best Practices: Restrict access to API Destinations by using IAM roles and policies to prevent unauthorized access.

• Use Any HTTP Method Except CONNECT and TRACE: API Destination supports various HTTP methods, excluding CONNECT and TRACE.

• Leverage Input Transformers: Customize event payloads to match the expected parameters of the external API endpoint, ensuring seamless integration.

• Understand EventBridge Retry Behavior: Amazon EventBridge retries requests that receive HTTP error codes 401, 407, 409, 429, and any 5xx errors. It does not retry requests with 1xx, 2xx, 3xx, or most 4xx errors, except for the specified ones.

AWS CDK example

// Create a secret for client credentials
const clientSecret = new Secret(scope, 'SecretName', {
    secretName: 'secret-name',
});

// Create a secure connection for API authentication
const connection = new Connection(stack, 'ConnectionId', {
    connectionName: 'MyConnection',
    authorization: Authorization.oauth({
        authorizationEndpoint: "authorization-url",
        clientId: "client-id",
        clientSecret: clientSecret.secretValue,
        httpMethod: HttpMethod.POST,
        bodyParameters: {
            grant_type: "client_credentials",
        },
    }),
});

// Define the API Destination
const apiDestination = new ApiDestination(stack, 'ApiDestinationId', {
    apiDestinationName: 'ApiDestination',
    connection,
    endpoint: "endpoint-url",
    httpMethod: HttpMethod.POST,
    rateLimitPerSecond: 10,
});

// Create a Dead Letter Queue (DLQ) for failed events
const deadLetterQueue = new Queue(stack, 'DLQ');

// Define an EventBridge Rule
new Rule(stack, "EventBridgeRuleId", {
    ruleName: "EventBridgeRuleId",
    eventBus: new EventBus(stack, 'MyEventBus'),
    eventPattern: {
        source: ["eventSourceName"],
        detailType: ["eventDetailType"],
    },
    targets: [
        new ApiDestinationTarget(apiDestination, {
            deadLetterQueue,
            event: { detail: { key: "value" } },
            retryAttempts: 185,
            maxEventAge: Duration.hours(24),
        }),
    ],
});

Conclusion

AWS API Destination simplifies the process of securely triggering external APIs based on AWS events. Developers can build robust and scalable event-driven integrations with external services by leveraging managed authentication, retry policies, and monitoring capabilities. However, when designing solutions with API Destinations, it's essential to be mindful of execution time limits and supported API types.