Don't use SNS-SQS, Amazon Kinesis is a much better option.

Aakash Solanki — Mon, 30 Aug 2021 00:41:57 +0000

Before Kinesis, SNS with SQS was preferred for most use cases unless we don't need to guarantee FIFO on messages,
The main advantage of Kinesis is:
We can read the same message from multiple applications and re-read messages.

To manage spike load, it's pain in Kinesis,
we need to increase the number of shards where SQS is infinitely scalable.

Kinesis is preferred when data in the stream are related or not related to each other,
While SNS/SQS is preferred when data in the stream are not related to each other.

Pricing models are different,
SQS charges per message while Kinesis charges per hard per hour,
Total charges of SNS-SQS is higher than Kinesis,

Real-time analytics of data is only possible with Kinesis while SNS-SQS is good for batch messages for future processing.

Some points are missing, Please add those points or suggest me required changes in my post.

Algorithmic Trading Architecture on AWS using Zerodha kite API

Aakash Solanki — Fri, 27 Aug 2021 11:16:38 +0000

Please look into this architecture diagram, I'm using to automate trading signal,

Here I'm using Zerodha's Kite web socket API to fetch tick data of more than 200 securities,
To generate intraday trades using Redis in-memory database and to generate positional trades using dynamoDB database service,

Step 1
Lambda function cron job starts EC2 instance before market opening at every market day
Step 2
Python script connects with Zerodha kite web socket and starts data feeding
Step 3
Python script starts pushing data to a Redis in-memory database
Step 4
Python script resamples and pushes data from Redis to MongoDB, as per the intraday strategies after every fixed interval
Step 5
Lambda function resamples and pushes data from MongoDB to dynamoDB, as per the positional strategies after every fixed interval
Step 6
Python script generates an intraday trading signals and places orders to Zerodha
Step 7
After getting order confirmation from Zerodha, python script pushes order details to dynamoDB
Step 8
Lambda function gets positional trading data, generates a positional trading signals and places orders to Zerodha
Step 9
After getting order confirmation from Zerodha, Lambda function pushes order details to dynamoDB.
Step 10
Python script gets intraday order details from dynamoDB and updates the orders as per the current market price if required
Step 11
Python script updates current orders or cancel and place a new orders to Zerodha
Step 12
Lambda function gets current positional data and update positional orders as per current market price if required
Step 13
Lambda function updates current orders or cancel and place new orders to Zerodha
Step 14
Lambda function square-off intraday open positions, as per individual intraday strategy rules
Step 15
Lambda function square-off all intraday open positions in case of emergency
Step 16
Lambda function cron job starts step function after market closing
Step 17
Lambda function puts all data to S3 from MongoDB and dynamoDB
Step 18
Lambda function stops EC2 instance after complete all steps

Please suggest to me how can I improve this and a new important feature I can add to this.

DEV Community: Aakash Solanki

Don't use SNS-SQS, Amazon Kinesis is a much better option.

Algorithmic Trading Architecture on AWS using Zerodha kite API