Enhancing Workflow Efficiency with Direct HTTP API Calls in Step Functions

#aws #serverless #stepfunctions #architecture

Background
Logic
Calling API Authentication
System Overview
Solution
Retry mechanism
Limitation
Outcome

Background

Within the FSI (Financial Service Industry) company, there are 2000 traders who rely on real-time interest rate updates from an external third-party API endpoint to make informed decisions. It is crucial to closely monitor for errors and ensure the ability to retry.

Logic

Whenever the interest rate exceeds the threshold of 5.1, the traders' decision pool is promptly updated, enabling them to make informed choices. Otherwise, no action is taken. As the interest rate value constantly changes, it is essential to have a simple and efficient way to update it.

Calling API Authentication

To authenticate the API calls, we set up Amazon EventBridge and create authentication credentials. The API key is securely stored in AWS Secrets Manager.

System Overview

Step Functions
Orchestrate multi-step applications and microservices using visual workflows.
uses "state machines" to represent applications as a series of steps.

Step Functions -> Choice state
Allows a workflow to branch based on conditions or dynamic data. The workflow will continue down one of multiple paths based on the choice rules.

Step Functions -> Choice state -> Rules
Choice rules define the conditions that determine which path the workflow will take after a choice state. Rules can reference the output of previous steps, static conditions, or dynamic data to evaluate which path to follow.

Step Functions -> Error handling
Step Functions has features for retrying steps that fail, catching errors, and continuing or stopping the workflow based on error conditions.

EventBridge
Serverless event bus that connects applications together using data from applications and AWS services.

EventBridge -> Scheduler
Schedule regular events for EventBridge.

EventBridge -> API destinations
Send events to API endpoints.

Secrets Manager
Store and retrieve secrets securely without plaintext secrets.

Simple Notification Service (SNS)
Fully managed messaging service for application and person communication.

Lambda
Serverless compute service to run code without provisioning servers.

DynamoDB
Fully managed NoSQL database with single-digit millisecond performance.

Solution

Third-Party API Endpoint Retry:

Within the Step Function, we make HTTP calls to the endpoint and implement a retry mechanism. The system retries the call five times, with a 20-second interval between each attempt. In case of errors, a message is sent to an analysis error monitor dashboard, ensuring critical monitoring.

Lambda Concurrency Scaling Rate:

By scaling the lambda function's concurrency, we can achieve a rate of 1000 execution environment instances every 10 seconds.
This ensures that all 2000 traders' decision pools are updated within a 20-second timeframe, meeting critical requirements.

Real-Time Update of Traders' Decision Pool:

Leveraging Lambda functions, we implement a system with a currency of 2000 lambda functions.
This allows for real-time updates to be made across all traders' decision pools simultaneously, ensuring timely and accurate information.

Parameter Store Integration in Step Function:

To facilitate the ease of updating the threshold value (currently set at 5.1), we utilize choice rules within the Step Function. This enables us to store and modify the threshold value conveniently, ensuring flexibility and adaptability.

Retry mechanism

If there is an error in the HTTP call, the system will automatically retry.

Step Functions offer you the ability to handle errors by retrying, ensuring that your workflow stays on track and doesn't get derailed.
It allows you to easily restart workflows from the point of failure, skipping the successful steps that have already been completed.
This simplifies the recovery process by restarting only the necessary workflows.

Benefit
It helps you build resilient workflows, ensuring that your system can handle errors and recover smoothly.
You can configure an initial set of retries for errors that can be automatically resolved, all without any unnecessary delays.

Limitation

Your workflow execution should have fewer than 24,999 events.
If you exceed this limit, the event history of redriven executions will be appended to the existing event history.

Outcome

The implementation of a visualized dashboard enhances the workflow, making it easier, faster, and more user-friendly.
By directly calling the HTTP API, we significantly save development time and gain flexibility, enabling us to set up an efficient orchestration flow for the traders' decision pool.