DEV Community: Distinction Dev

Event driven architecture : Overview and comparison of AWS Messaging services

Bhavin Babariya — Wed, 03 Jul 2024 10:39:19 +0000

Overview of Event Driven Architecture ✨

Event-Driven Architecture (EDA) is a design paradigm where systems communicate and respond to events in real-time.

This architecture promotes loose coupling, scalability, and flexibility, as components are only connected through the events they produce and consume.

Event-Driven Architecture is widely used in systems requiring high responsiveness and real-time processing, such as financial trading platforms, IoT networks, and customer service applications.

In Event-Driven Architecture (EDA), the main components are Producer, Event Broker and Consumer :

Producer: This is the source that generates and emits events. Producers can be anything from applications, services, or devices that detect a change in state or trigger an event.
Event Broker: This intermediary handles the transmission of events from producers to consumers. It ensures decoupling by managing the distribution and routing of events, often providing features like event filtering, persistence, and scalability.
Consumer: Listens for and processes events received from the event broker. Consumers act on the event data, performing tasks such as updating systems, triggering workflows, or generating responses.

Event Driven Architecture Common Model

There are many model in EDA, but commonly Point to Point and Pub/Sub model is used.

1. Point to Point Model

The Point-to-Point model ensures reliable and direct communication between a single producer and a single consumer, enhancing transactional processing and delivery guarantees.

In this model, a producer sends messages to a specific queue. A consumer retrieves and processes these messages from the queue. The message broker manages the queue and ensures each message is delivered to only one consumer.

This model is particularly helpful for scenarios where each message needs to be processed by a single recipient, ensuring reliable message delivery and simplifying message routing.

Ex. SQS

2. Pub Sub Model in EDA

The Pub/Sub model is used in EDA to decouple producers and consumers, enhancing scalability and flexibility. It allows efficient, real-time communication by distributing messages through topics managed by an event broker.

In the Pub/Sub (Publish/Subscribe) model, a publisher sends messages to a specific topic. Consumers subscribe to this topic to receive and process the messages. The event broker manages the topics and ensures that messages from publishers are delivered to all subscribed consumers.

This is how the Pub/Sub model is helpful in making systems flexible and scalable by decoupling producers and consumers, allowing efficient real-time communication through managed topics.

🚀 AWS messaging services

AWS services useful in EDA include Amazon SQS, Amazon SNS, Amazon EventBridge, Amazon Kinesis, AWS MKS (Kafka), AWS Lambda, Amazon MQ (for Apache ActiveMQ and RabbitMQ), and many more.

Simple Queue Service (SQS)

AWS SQS is Queuing service that is useful in communication between Applications, microservices, and Distributed system.

Use Case of SQS ✴️

process asynchronous task
- Queues enable the processing of asynchronous tasks effectively. By using a queue, we can poll messages at any time, allowing for flexible task management and execution.
decoupling microservices
- When two services communicate via a queue, they are decoupled, eliminating direct dependencies. This allows each service to operate independently, enhancing system scalability and resilience.
batch processing
- SQS supports batch processing so we can do batch processing on queue messages and also optimise resource utilization.
job scheduling
- If we add messages to queue throughout days and want to process this all message’s at a time in day then we can schedule event and by polling mechanism we can do batch processing on all data which is inside queue.

Model / Mechanism

works on Pull mechanism.

Consumers

It supports only 1 consumer for 1 message.
we can also consume message by AWS SDK or aws lambda as well.

Supports Ordering Mechanism

Yes, it supports FIFO queue to process item in order.

Conditional Message Filtering

SQS doesn’t support conditional filtering mechanism for Message.

Encryption

supports message encryption using KMS

Throughput

SQS standard queue has unlimited throughput
SQS FIFO queue supports 3000 messages per second with batch processing or 300 message per second without batch processing.

Dead Letter Queue

Yes, it supports Dead Letter Queue

Pricing 🤑

$ 0.40 per 1 Million (Standard Queue)
$ 0.50 per 1 Million (FIFO Queue)
Data Outbound Charge
- $ 0.09 per GB

Simple Notification Service (SNS)

SNS (Simple Notification Service) is a fully managed pub/sub solution service offered by AWS. With SNS, users can create multiple topics and subscribers, and each topic can be connected with multiple subscribers.

Use Cases of SNS ✴️

Fan Out System
- Distribute a single message to multiple recipients efficiently.
Mobile Push Notifications
- Send real-time updates to mobile devices across various platforms.
System Monitoring Alerts
- Trigger alerts from monitoring tools based on specific events or thresholds.
Trigger Different Workflows
- Initiate diverse workflows by sending messages to various endpoints based on events.

Model / Mechanism

works on Push mechanism.

Consumers

It supports multiple consumer per message.
it supports Kinesis Data Firehose, Lambda, SQS, Email, HTTP/s, Application Notification and SMS as consumer.

Supports Ordering Mechanism

Yes, it supports FIFO topic to process item in order.

Conditional Message Filtering

SNS supports conditional filtering mechanism for Message.

Encryption

supports message encryption using KMS

Dead Letter Queue

Yes, it supports Dead Letter Queue

Throughput

SNS standard topic has near about unlimited throughput
SNS FIFO topic supports 300 messages per second or 10 MB message per second per topic.

Pricing 🤑

Standard Topic
- $ 0.50 per 1 M (Mobile Push Notification)
- $ 0.60 per 1 M (HTTP/s Request)
- $ 2.00 per 100,000 notifications
- No charge for SQS and Lambda
- $ 0.19 per 1 M Notification (Amazon Kinesis Data Firehose)
- Data Outbound Charge
  - $0.09 per GB
FIFO Topic
- Publish and publish batch API requests are $0.30 per 1 million and $0.017 per GB of payload data
- Subscription messages are $0.01 per 1 million and $0.001 per GB of payload data

Combined Use Case: SNS and SQS Integration 🤝

Fan-out with SQS:
- Use Case: Distributing a message to multiple queues for parallel processing.
- Example: An e-commerce platform needs to update inventory, process billing, and send a confirmation email when an order is placed. The order service publishes a message to an SNS topic, which then fans out to multiple SQS queues. Each queue is processed by different services responsible for inventory, billing, and email notifications, ensuring that the tasks are handled independently and concurrently.

Event Bridge

AWS EventBridge is a service that connects multiple AWS services based on events, facilitating event-driven architecture management. With EventBridge, you can send custom events from SaaS applications to an event bus, schedule tasks, and monitor AWS services. This enables seamless integration, automation, and real-time monitoring within your cloud environment.

Use Cases of Eventbridge ✴️

Building Serverless Event-Driven Architecture:
- AWS EventBridge allows the setup of a full event-driven infrastructure when using AWS services as both producers and consumers. AWS service events can act as sources, while AWS services can also be targets for event processing.
SaaS Integration with AWS Services:
- EventBridge supports custom events, enabling seamless SaaS integration. You can send custom events to an EventBridge event bus, facilitating communication between SaaS applications and AWS services.
Real-Time Monitoring and Alerting:
- EventBridge can monitor actions or events in real-time across various services. Based on these events, you can generate alerts or create CloudWatch logs, enhancing your system's observability and responsiveness.
Scheduling Tasks:
- EventBridge allows the scheduling of tasks using cron jobs or rate expressions. This enables you to automate the invocation of AWS services at specified times or intervals, ensuring timely execution of routine tasks.

Model / Mechanism

works on Event bus model.
uses Push mechanism to call target.

Consumers

It supports multiple consumer per rule.
We can set many aws services as target as well as also set HTTP/s endpoint if we want to call external API .
- Ex. invoke step function, start Glue Workflow.

Supports Ordering Mechanism

No, there are no order guarantee

Conditional Message Filtering

Event bridge supports event message filtering and transforming mechanism.
- We can define schema in schema registry and based on schema we can filter message.
We can use Eventbridge pipe to filter and transform data.

Encryption

doesn’t support message encryption using KMS

Archive and Event Replay

we can archive events and replay them later when it needed.

Throughput

Eventbridge has near about unlimited throughput for all aws service event.
In all regions, PutPartnerEvent (used by SaaS provider to write event in eventbus) has a soft limit of 1400 requests per second and 3,600 burst requests per second by default.

Pricing 🤑

EventBus
- Free for aws service event
- $1 per 1 million custom event or SaaS event or Cross account event.
  - 64 KB of chuck is considered as 1 Event. (If data of payload is 150 KB consider as 3 event)
EventPipe
- $0.40 per 1 million requests (event count after filter)
Event Replay
- $0.023 per GB for event storage.
- $0.10 per GB for archive processing.
Schema Registry
- Use of schema registry for aws and creating custom schema is free.
- $0.10 per million events (only discovery charge)

AWS Kinesis Datastream

Amazon Kinesis Data Streams (KDS) is used for real-time processing of streaming data at massive scale. when we need to process real time data processing & analytics at scale, Kinesis is useful.

Use cases of Kinesis ✴️

Real-Time Analytics:
- An e-commerce platform can use Kinesis Data Streams to capture and analyse clickstream data to understand user behaviour and personalise recommendations in real-time.
Log and Event Data Collection:
- We can use KDS to ingest and monitor application logs and system events to detect anomalies and react quickly to potential issues.
IoT Data Ingestion:
- Manufacturing companies can stream sensor data from IoT devices to monitor equipment health, predict maintenance needs, and optimise operations.
Financial Market Data Processing:
- Financial services can use KDS to process market data in real-time to detect trading opportunities and risks.

Model / Mechanism

works on Data Stream modal
works on Pull mechanism.

Consumers

AWS Lambda, AWS Kinesis DataStream, AWS Kinesis Data Analytics, AWS Kinesis Data Firehose, KCL (Kinesis Client Library)

Archive and Event Replay

Amazon Kinesis Data Streams' archive and replay features enable long-term data retention, fault recovery, and compliance by securely storing data in Amazon S3 and allowing for easy reprocessing.

Throughput

On Demand Mode
- Read Capacity : Handle upto 400 MB per second
- Write Capacity : Handle upto 200 MB/second and 2,00,000 records/second
Provisioned Mode ( for 1 shard only)
- Read Capacity : Maximum 2MB per second
- Write Capacity : 1 MB/second and 1,000 records/second

Drawbacks

Shard need to be managed manually.

Pricing 🤑

$ 0.015 per hour per shard (Provisioned Mode)
$0.04 per hour per shard (On Demand Mode)
$ 0.014 per 1 million PUT payload units
NOTE : There are other charges as well like Data Retention and data fan out etc..

Apache Kafka (AWS MKS)

Apache Kafka is a distributed, fault-tolerant, reliable, and durable streaming platform used for real-time data pipelines. Initially developed by LinkedIn, it later became open-source.

Kafka boasts high throughput and low latency, making it especially useful when data consistency and availability are crucial. It efficiently handles large volumes of data, enabling organizations to build robust, real-time data processing and analytics systems. Kafka's architecture supports horizontal scalability, ensuring that it can grow with the needs of the application.

Use Cases of Apache Kafka ✴️

Real-time analytics and monitoring
Event sourcing and event-driven architectures
Log aggregation and processing
Stream processing and transformation
Data integration and ETL (Extract, Transform, Load) pipelines

Model / Mechanism

Pub/sub Model

Consumers

AWS lambda,Kinesis Data Analytics, Kinesis Data Firehouse, EMR, Glue are connected to MSK directly while S3, DynamoDB, Redshift can be connected via Kafka Connect.

Supports Ordering Mechanism

Yes, there are order guarantee

Conditional Message Filtering

It doesn’t support message filtering at broker level, we have to handle filtering at consumer level.

Encryption

supports message encryption using KMS

Archive and Event Replay

AWS MSK retains all published messages for a configurable retention period.

Throughput

Kafka provides high throughput, also capable of handling large volumes of streaming data with low latency.

Pricing

Broker Instance charges
- $0.204 (price per hour for a kafka.m7g.large)
- $0.21 (price per hour for a kafka.m5.large)
Storage charge
- $0.10 (the price per GB-month in US East region)

📗 Very Important to know when to choose which service ?

Asynchronous job processing: Use Amazon SQS. Ideal for decoupling microservices and buffering requests.
- If numbers of event per second is lower or medium then SQS is always recommendable service
Sending notifications or invoking services: Use Amazon SNS. Perfect for sending notifications to multiple recipients or invoking services with pub/sub messaging.
Triggering services based on events: Use Amazon EventBridge. Best for integrating AWS services and custom applications through event-driven architectures.
- Eventbridge is highly recommend when SaaS integrate to AWS services is requirement.
Handling high request rates and event-driven data ingestion: Use Amazon Kinesis. Suitable for real-time data streaming and analytics with the ability to scale by adding shards.
- Kinesis is costly service compare to other service as it’s cost depends on number of active shards.
Live streaming and scalable, low-latency data pipelines: Use Amazon MSK (Managed Streaming for Apache Kafka). Excellent for building scalable, real-time data streaming applications with Apache Kafka.
- Apache Kafka is highly recommended when millions or billions of request occurs at time.

➕ Additional Considerations

Message Ordering and Deduplication: If you require strict message ordering and deduplication, consider using SQS FIFO Queues, Kinesis or Kafka.
Multiple Consumer Support: For scenarios where multiple consumers need to process the same stream of data, SNS , Kinesis, Kafka is preferred.
Complex Event Processing: For applications needing complex event processing and routing, EventBridge provides advanced capabilities for rule-based event handling.

Summary Table

Feature	SQS	SNS	Event bridge	Kinesis	Kafka (MKS)
Message Filtering	No	Yes	Yes	No	No
Order	Yes (FIFO Queue)	Yes(FIFO Topic)	No	Yes	Yes
Throughput	Low (FIFO)	Medium	Medium	High	High
Latency	Medium	Low	Low	Low	Low
Durability	High	High	High	High	High
Integration	AWS Services	AWS Services	AWS Services	Custom Applications & AWS Services	Custom Applications & AWS Lambda
SaaS support	No	No	Yes	No	No
Data Persistence	Yes	No	Yes	Yes	Yes
pricing	Low	Low	Low	High	High

Move aws resources from one stack to another cloudformation stack

Bhavin Babariya — Tue, 02 Jul 2024 06:18:00 +0000

Why do we need this?

The AWS CloudFormation resource limit is currently set at 500, although this size may increase with the introduction of new features in Application.
To accommodate this limitation, we must distribute all resources across various stacks.
Our approach involves isolating Lambda functions into a separate stack, while other resources such as S3 buckets and DynamoDB tables reside in an infra stack.
This is the reason why we need to import resources from the main stack into the infra stack.

Steps to move resources from one stack to another stack

Prerequisites

Apply 'DeletionPolicy: Retain' to all resources of the main stack

Applying 'DeletionPolicy: Retain' to all resources in the main stack ensures that when these resources are deleted during stack updates or deletions, they are retained rather than being deleted permanently.
This is particularly useful for resources that contain valuable data or configurations that need to be preserved even if they are no longer actively used.

Consider you have two cloudformation stack(which is generated by serverless framework): main and destination, and you want to import some resources from main to destination. Here are the steps to move resources from one stack to another stack without deleting the actual resources.

Copy AWS resources from the main cloudFormation stack and paste them into the destination cloudFormation stack.
Remove resources from the main stack and deploy the main stack.
Prepare another file named "resourcesToImport.txt" containing the AWS resource type, logical ID, and resource identifier.
Run a command to create an IMPORT changeset.
Execute a command to apply changeset which was created in the previous step.

1. Copy AWS resources from the main cloudFormation stack and paste them into the destination cloudFormation stack.

Copy destination stack cloudformation code into one file ( templateToImport.json)
Copy main stack resource’s ( which you want to import) cloudformation code and append them in destination stack code (templateToImport.json)

2. Remove resources from the main stack and deploy the main stack.

Now, remove all the resources which we want to import or we added into the destination stack in step 1 .
Redeploy main stack.

Now resources are not in any stack and also not deleted because resource’s deletionPolicy is set to Retain.

3. Prepare another file named "resourcesToImport.txt" containing the aws resource type, logical ID, and resource identifier.

Now, create One file named ‘resourcesToImport.txt’ and add ResourceType, LogicalResourceId and ResourceIdentifier for each resource which we want to import.

ResourceType will be the cloudformation resource type
LogicalResourceId will be the Logical Name of resource
ResourceIdentifier contains actual identifier of resource
- If resource is S3 bucket then value will be {"BucketName": ""}
- If resource is dynamodb table then value will be { "TableName": "ACTUAL_DYNAMODB_TABLE_NAME" }
- If resource is rest api then value will be { "RestApiId": "REST_API_ID" }

Example File :



[
  {
    "ResourceType": "AWS::S3::Bucket",
    "LogicalResourceId": "<LOGICAL_NAME_OF_BUCKET>",
    "ResourceIdentifier": {
      "BucketName": "<ACTUAL_NAME_OF_BUCKET>"
    }
  },
  {
    "ResourceType": "AWS::DynamoDB::Table",
    "LogicalResourceId": "<LOGICAL_NAME_OF_DYNAMODB_TABLE>",
    "ResourceIdentifier": {
      "TableName": "ACTUAL_NAME_OF_DYNAMODB_TABLE"
    }
  },
  {
    "ResourceType": "AWS::ApiGateway::RestApi",
    "LogicalResourceId": "<LOGICAL_NAME_OF_RESTAPI>",
    "ResourceIdentifier": {
      "RestApiId": "REST_API_ID"
    }
  }
]

Run a command to create IMPORT changeset

below command creates import changeset of resource



aws cloudformation create-change-set --stack-name <YOUR_STACK_NAME> --change-set-name <CHANGE_SET_NAME> --change-set-type IMPORT --resources-to-import file://resourcesToImport.txt --template-body file://templateToImport.json --capabilities CAPABILITY_NAMED_IAM --description "write here description" --profile <AWS_PROFILE>

5. Execute a command to apply the changeset.

below command executes the import changeset and resources will be move from main stack to destination stack 🥳



aws cloudformation execute-change-set --change-set-name <CHANGE_SET_NAME> --stack-name <YOUR_STACK_NAME> --profile <AWS_PROFILE>

👉 NOTE : Cloudformation doesn’t allow to import all types of resources. Few resources are not supported to import.

Below link contains all the resources which are allowed to import in cloudformation stack

Resource type support - AWS CloudFormation

Reference

Importing existing resources into a stack - AWS CloudFormation

React Rendering Guide

Biki — Thu, 01 Feb 2024 11:01:04 +0000

React has by far been the popular choice for writing User Interfaces, Performance is a important thing to consider in React as things can go out of the way, making your web app feel slow and user’s complaining things are taking time. When we talk about Performance in React, re-renders and their affect is very crucial to understand, How actually re-renders happen, how it flows through your app, what exactly happens during a re-render and why do they exist or we need them in the first.

All of these Question will help us to explore these topics, and to make it interactive and easy to grasp, lets become a detective , we would start out with a very common performance issue in our app, let see what’s going on and how we can try to fix the same, while trying we would be learning

What exactly a re-render is, why they exist?
Where does re-render initiate from.
How React tries to distributes re-renders through an app.
How moving state down can help us solve performance issues in our app

The Problem

Imagine you are a developer working on a react web app, this is a large production app with a lots of thing going on, you can assume it as a legacy codebase where it has been worked by other developer from the past and it has a huge customer base. You get your first ticket Assigned, you are exited and you got a simple task to warm you up, you need to create a Place order Page that opens a modal at the top of the app.

The code in abstract Looks like below and you get the idea where should the dialog be triggered

const App = () => {
 return(
   <div className="layout">
   <VerySlowComponent/>
   <SomeStuff />
   <OtherStuffAlsoComplicated/>
   </div>
   );
};

Seems quite any easy task , you must have done this a gazillion times, kidding probably more times than you can think

const App = () => {
// you would be addding some state
const [isOpen,setIsOpen] = useState(false)
 return(
   <div className="layout">
   // you need to add a button
   <Button onClick={() => setIsOpen(true)}>Place Order</Button>
   <VerySlowComponent/>
   <SomeStuff />
   <OtherStuffAlsoComplicated/>
   </div>
   );
};

Sprinkling some state that hold the dialog is open or closed. add the button which eventually triggers the state update onClick even. and the dialog renders only if the state variable value os true.

You did so and try to test it works, unfortunately you notice there is a slight delay to open that modal.

Many might feel that we need to bring the performance hooks like memo and callback to fix this, this techniques would feel meaningless to include here, there are more simpler things that we can use to solve this issue.

State Updates and Re-renders

Let's start from the beginning: the life of our component and the most important stages of it that we need to care about when we talk about performance. Those are: mounting, unmounting, and re-rendering.
When a component first appears on the screen, we call it mounting. This is when React creates this component's instance for the first time, initializes its state, runs its hooks, and appends elements to the DOM. The end result - we see whatever we render in this component on the screen.”

Unmounting is when React is directed that we no longer need a component, so it does a cleanup and unmounts the component essentially destroying the same, its associated state, hooks and eventually gets thrown out of the DOM elements as well.

Re-rendering means when React updates an existing component with some extra information. if we differentiate it between mounting, re-renders are less workaround since React will use the already existing component alive , doing all the necessary calculations for what changed, runs the hooks and eventually updating the existing DOM elements.

Lets see where Re-renders stem from, the origin point for re-renders is state. In React, when we use a Hook like useState,useReducer or any other state Management Tool like MobX, Redux etc. we introduce some piece of Data that would live with it and would be preserved throughout its lifecycle, if something happens that needs a response like user clicking a button, typing into an input or some external data is coming , the particular state would be updated with the new data.

Re-rendering is an important concept to grasp around react. This is what Allows React to update the component with the new Data and trigger hooks that depend on the same. Without Re-rendering, There will be no updates in React which would result in no interactivity, making the app completely static. so remember state update is the initial source for all renders in React bases app. if we take the example of the code

const App = () => {
const [isOpen,setIsOpen] = useState(false)

return(
  <Button onClick={() => setIsOpen(true)}>
    Place Order
  <Button/>
)
}

Here when We click on the button it triggers a re-render the setIsOpen state setter function is called, updating the state with new value from false to true. thus our App component re-renders as it holds the state.

As the state is updated, our App component renders, new data is delivered to component that depend on it. React does it by its own, it takes all the components that the initial component renders inside, re-renders those, then re-renders components nested inside of them, and so on until it reaches the end of the chain of components. it is recursive in Nature.

If you try to visualise a Typical React App as a Tree, every thing down from where the state update occurred will be rendered as well.

If you take a closer look at our App everything is rendering, which means those slow components are also part of it, when state changes

const App = () => {
 const [isOpen, setIsOpen] = useState(false);
 // everything that is returned here will be re-rendered when the state is updated”
“return (
   <div className="layout">      
    <Button onClick={() => setIsOpen(true)}>        
      Place Order      
    </Button>      
    {isOpen ? ( 
    <ModalDialog onClose={() => 
     setIsOpen(false)} />  
    ) : null}      
    <VerySlowComponent />     
    <BunchOfStuff />     
    <OtherStuffAlsoComplicated />    
  </div>  
);
};

As a result, the modal is not opening instantly, as React has to render everything before the our dialog can appear.

Remember, React will never Look up the Render Tree when the re-renders occurs. if the state was triggered somewhere in the middle, only components below it would go though re-render.
The only way for components at the bottom to affect the top is either to call the state update in those components or passing components as functions.

Bursting the re-renders Myth

We have Props in React, and you might have encountered the phrase “Component re-renders when its prop change”. it’s a popular misconceptions in React amongst a lot of people, many don’t doubt it and this is not true.
Normal React behavior is that if a state update is triggered, React will re-render all the nested components regardless of their props. And if a state update is not triggered, then changing props will be just "swallowed": React doesn't monitor them.”

In the context of re-renders, whether props have changed or not on a component matters only in one case: if the said component is wrapped in the React.memo higher-order component. Then, and only then, will React stop its natural chain of re-renders and first check the props. If none of the props change, then re-renders will stop there. If even one single prop changes, they will continue as usual.

Moving state down (The solution)

Now we have clarified how the whole re-rendering thing works in React, it’s time to apply the things we learned to the problem we are dealing and fix it, let have a look at the code where we use the modal state:

const App = () => {  
// our state is declared here  
  const [isOpen, setIsOpen] = useState(false);

  return (    
    <div className="layout">      
     {/* state is used here */}      
   <Button onClick={() => setIsOpen(true)}>        
      Place Order     
   </Button>      
{/* state is used here */}      
 {isOpen ? (        
      <ModalDialog onClose={() => 
setIsOpen(false)} />      ) 
   : null}      
<VerySlowComponent />      
<BunchOfStuff />      
<OtherStuffAlsoComplicated />    
</div>  
);
};

As we can see, it's relatively isolated: we use it only on the Button component and in ModalDialog itself. The rest of the code, all those very slow components, doesn't depend on it and therefore doesn't actually need to re-render when this state changes. “It's a classic example of what is called an unnecessary re-render.”

we could have wrapped the same in React.memo() which would prevent the re-render , but it has it own sets of nuances to dealt with. There is a better and simpler way, we can just extract the components that depends on the state , along with the state itself into a smaller isolated component like below

const ButtonWithModalDialog = () => {  
const [isOpen, setIsOpen] = useState(false);
  // render only Button and ModalDialog here  
 return (    
   <>      
    <Button onClick={() => setIsOpen(true)}>        
     Place Order      
   </Button>      
   {isOpen ? (        
   <ModalDialog onClose={() => 
   setIsOpen(false)} /> 
) : null}    
</>  
);
};

And then we can just add this newly created component of ours in the App component

const App = () => {  
   return (    
      <div className="layout">      
     {/* here it goes, component with the state inside */}     
     <ButtonWithModalDialog />      
     <VerySlowComponent />      
     <BunchOfStuff />      
     <OtherStuffAlsoComplicated />    
     </div>  
    );
   };

Updated change sandbox

Now, the state update when the Button is clicked is still triggered, and some components re- render because of it. But! It only happens with components inside the ButtonWithModalDialog component. And it’s just a button and our modal that should be rendered. The rest of out app is working as good.

We just created a new piece of branch inside our render tree and moved the state downwards thus our modal is appearing instantly as it should be, so we learned to fix this performance issue that looked big from the far but was a simple fix.

YAML Magic: Unveiling the Secrets of Aliases, Anchors, and Overrides!

Dhrumin Thakkar — Fri, 22 Dec 2023 05:48:56 +0000

Hey YAML enthusiasts! Ready to dive into the magical world of YAML and uncover some secrets that will make your data structures dance with joy? Buckle up, because today we're unraveling the mysteries of aliases, anchors, and overrides – the enchanting trio that adds a dash of wizardry to your YAML files!

Alias All the Things!

Let's kick things off with aliases. Imagine you have a YAML document with repetitive structures, and you wish there was a way to avoid redundancy. Enter aliases! They're like magical shortcuts for your YAML data.

heroes:
  - &superman
    name: Superman
    power: Flight

  - &batman
    name: Batman
    power: Intelligence

justice_league:
  - *superman
  - *batman

In this YAML snippet, we've defined two heroes, Superman and Batman, and then we've summoned them to join the Justice League using the * symbol followed by their alias. It's YAML's way of saying, "Hey, use the properties from this alias here too!" Simple, right?

Anchors: Your Data's Home Base

Now, let's talk anchors – the home base for your YAML structures. An anchor is like a bookmark for a specific point in your data, making it easy to reference and reuse.

countries:
  - &usa
    name: United States
    population: 331 million

  - &canada
    name: Canada
    population: 38 million

north_america:
  - *usa
  - *canada

In this example, we've anchored the USA and Canada in the countries list. Later, we effortlessly include them in the north_america list using the * symbol, just like with aliases. Anchors help keep your YAML DRY (Don't Repeat Yourself) and your data organized!

Overrides: A Splash of Individuality

Now, let's add a touch of individuality to our data with overrides. Sometimes, you want to reuse a structure but tweak certain properties. That's where overrides come in handy.

cities:
  - &nyc
    name: New York City
    population: 8.4 million

  - <<: *nyc
    borough: Manhattan

In this snippet, we've created an alias for New York City. Then, we've used the << syntax to override the properties and add a new one – the borough. This allows us to reuse the city's information while giving it a unique twist. Who said data can't have a personality?

Conclusion: YAML Sorcery Unleashed!

And there you have it – the magical trio of YAML: aliases, anchors, and overrides. With aliases, you can summon data effortlessly. Anchors serve as your data's home base, keeping everything neat and tidy. Overrides add a sprinkle of uniqueness to your structures.

So, next time you're working with YAML, don your wizard hat, sprinkle a bit of YAML dust, and let the magic of aliases, anchors, and overrides make your data dance with joy! 🎩✨ Happy YAML-ing!

How to deal with stuck ECS deployment

Himanshu Pant — Wed, 20 Dec 2023 06:28:03 +0000

One of the hardest problems that I’ve ever dealt with in my life is making ECS deployments with Cloudformation work and work nicely, which is hard for some reason that only AWS knows.

TLDR

If your cloudformation stack update containing ECS doesn’t finish in a reasonable amount of time(depends on previous deployment times and container image size), roughly 10-15 minutes. You have a problem with your service.

Why does it stay stuck?

Idk, it just does, this is one of those things that cloudformation(any deployment tool) really sucks at and CloudFormation waits for several hours before rolling back to a previous configuration. If the issue that's causing stack failure continues during stack rollback to a previous configuration, then the stack gets stuck in UPDATE_ROLLBACK_IN_PROGRESS status. Finally, the stack changes to UPDATE_ROLLBACK_FAILED status and that is DevOps hell.

What’s really happening though?

ECS ***********really*********** likes to be stable and if it can’t be stable then it won’t finish updating(it won’t tell cloudformation that the update finished) and it won’t rollback either, so the main goal is to get the service to reach stability and there can be many reasons for it not reaching stability, however that is beyond the scope of this article.

So what could’ve gone wrong?

Here are some common reasons why an Amazon ECS service can fail to launch new tasks:

Container image issues
A lack of necessary resources for launching tasks
A health check failure on a load balancer
Instance configuration or Amazon ECS container agent issues

An Amazon ECS service that fails to launch tasks causes AWS CloudFormation to get stuck in UPDATE_IN_PROGRESS status, and you can quick check this by going into the service and selecting deployments, and checking the status of the latest deployment.

So what do I even do now?

Delete the stack and recreate!

!https://media0.giphy.com/media/0bc1ObnZK6IO0FkMV8/giphy.gif?cid=7941fdc64gt9sxd9juwjb3gft1b2qq3zgnmk2oxdassm15n0&ep=v1_gifs_search&rid=giphy.gif&ct=g

The main goal is to get your cloudformation to say:- “Update Complete” and it can take the AWS CloudFormation stack several hours to stabilize. To stabilize your stack more quickly, just lie to it.

Disclaimer

The following resolution is intended to help you stabilize an AWS CloudFormation stack quickly without waiting for the stack to time out. The resolution isn't intended for production environments, as the Amazon ECS service is out of sync with the known state of AWS CloudFormation. To sync resources between your Amazon ECS service and the AWS CloudFormation stack, you must perform an error-free update on the stack.

Resolution

Change the desired task count of the Amazon ECS service to 0

Open the Amazon ECS console.
Choose your cluster.
Select the service, and then choose Update.
Set Number of tasks to 0, and then save the configuration.

So what do I then?

Get your service to run and stabilize, make sure the process can start.

S3 Multi-Part Upload: Part 2 Conclusion

Mitansh Gor — Sat, 18 Nov 2023 23:22:29 +0000

Howdy, my tech-savvy pals! 🌟 Remember our last rendezvous? We chatted about the multipart upload basics - the whole shebang! Today, get ready to roll up your sleeves because we're plunging into the deep end! 💦 We're talking all about those tricky low-level hurdles you might encounter while playing around with multipart stuff using Node.js + the Serverless framework. But hey, fear not! We're a dynamic duo, and together, we'll smash these challenges and soar to victory! 💪🚀

🌟 Remember our chat about multipart upload (Part 1)? It's like a three-step dance for your data! First, you start the upload party. Then, you groove through uploading the object parts. Finally, when all the parts are in place, you wrap up the multipart upload! 🚀📦

But before we take the plunge into each step's deep end, let's first secure our multipart permission set. Ready, set, go! 🏁

# serverless.yml
provider:
    ....
    iamRoleStatements:
            ......
        - Effect: Allow
          Action:
            - s3:GetObject
            - s3:PutObject
            - s3:AbortMultipartUpload
          Resource:
            - arn:aws:s3:::${Bucket}/*
        - Effect: Allow
          Action:
            - s3:ListBucketMultipartUploads
          Resource:
            - arn:aws:s3:::${Bucket}

🛠️ "PERMISSIONS ADDED!" 🛠️

Now that the permissions are in check, nothing's holding us back from diving deep into each thrilling step of the multipart saga! 💪🌊

Multipart Upload Initiation

Let's dive right into the exciting world of Multipart Upload Initiation with this code snippet! This magical piece of code helps us kickstart the multipart upload process for an S3 bucket and key. Remember, it's crucial to save the response (upload ID) this function gives us. This upload ID is like the secret key to the opened stream, and we'll need it for the rest of our multipart adventure! 🗝️✨

/**
 * Create a multipart upload for a given S3 bucket and key.
 *
 * @param {string} bucket - The S3 bucket name.
 * @param {string} key - The S3 object key.
 * @returns {Promise<string>} The upload ID for the multipart upload.
 * @throws {Error} If there's an issue with the multipart upload creation.
 */
const createMultipartUpload = async (bucket, key) => {
  if (typeof bucket !== 'string' || !bucket.trim()) {
    throw new Error('Invalid bucket name. Please provide a valid string value for the bucket.')
  }

  if (typeof key !== 'string' || !key.trim()) {
    throw new Error('Invalid object key. Please provide a valid string value for the key.')
  }

  try {
    const params = {
      Bucket: bucket,
      Key: key
    }

    // Use S3's createMultipartUpload with promise()
    const data = await S3.createMultipartUpload(params).promise()

    return data.UploadId
  } catch (error) {
    throw new Error(`Error creating multipart upload: ${error.message}`)
  }
}

Uploading Parts - The 'Part-y' Begins

Uploading chunks or parts might seem like a breeze, but here's the catch: the multipart upload has its own set of rules. One big rule? Chunk sizes must be greater than 5GB. But sometimes, we might encounter situations where certain chunks/parts are less than 5MB. 🤔😅

Handling these scenarios and validating them is crucial. Imagine, our multipart adventure encountering these unexpected smaller chunks - we need to be ready to address them! 💡📦💻

To tackle this challenge, we're diving into 'chunk mode'. Picture this: we've got an array of files fetched with their sizes from S3. Now, we're slicing and dicing this array into chunks, making sure each chunk has at least 5MiB of data. 🎲✂️
Check out the magic of chunkifying:

// Below is the chunk result of the list of metadata of files fetched from s3.
chunkifyArray = [
  [
    {
      fileName: 'A.csv',
      ContentLength: 1728033, // 1.64798069 MiB
    },
    {
      fileName: 'B.csv',
      ContentLength: 53326970, // 50.856561661 MiB
    },
  ],
  {
    fileName: 'C.csv',
    ContentLength: 21646619, // 20.643824577 MiB
  },
  [
    {
      fileName: 'D.csv',
      ContentLength: 1728033, // 1.64798069 MiB
    },
    {
      fileName: 'E.csv',
      ContentLength: 5226970, // 4.984827042 MiB
    },
  ],
]

See how we cleverly grouped those files into chunks? It's like solving a puzzle! 🧩💻 This way, we ensure each chunk meets our 5MB rule.

So, picture this: files A and B are like two peas in a pod, individually less than 5 MiB each, but when they team up, they surpass that mark. Meanwhile, file C is a lone ranger, confidently exceeding the 5 MiB mark all by itself. Then we've got files D and E, best buddies, also teaming up to go beyond that 5 MiB limit.

This clever strategy ensures our chunks are just the right size for this multipart upload adventure! 🚀🔍

Now, for files A and B, we're planning a little readStream party! 🎉📚 We'll grab the records from both files, blend them into one mighty string, and that fusion will become the uploadable part. Think of it as a superhero team-up! 💪🦸‍♂️ The same goes for the dynamic duo, files D and E.

But hey, file C is a solo act. We'll simply read its data and smoothly upload it via stream. 🌟💾

Imagine this snippet as our trusty guide to converting an array of files' metadata into chunks. Buckle up, we're diving into some code magic! ✨🚀


/**
 * Converts an array of file metadata into chunks based on a size threshold.
 *
 * @param {Object[]} data - Array of file metadata objects.
 * @param {number} THRESHOLDLIMIT_5MB - Size threshold for chunking.
 * @returns {Object[]} Array of chunks with grouped file metadata.
 */
const convertToChunks = async (data, THRESHOLDLIMIT_5MB) => {
  const chunkifyArray = [];
  let totalSize = 0;

  data.map((data) => {
    if (data.ContentLength) {
      totalSize += data.ContentLength;

      if (chunkifyArray.length === 0) {
        if (data.ContentLength >= THRESHOLDLIMIT_5MB) {
          chunkifyArray.push(data);
        } else {
          const obj = {
            content: [data],
            size: data.ContentLength,
          };
          chunkifyArray.push(obj);
        }
      } else {
        const currentRec = chunkifyArray[chunkifyArray.length - 1];

        if (
          currentRec.size === undefined ||
          (currentRec.size !== undefined && currentRec.size > THRESHOLDLIMIT_5MB)
        ) {
          if (data.ContentLength >= THRESHOLDLIMIT_5MB) {
            chunkifyArray.push(data);
          } else {
            const obj = {
              content: [data],
              size: data.ContentLength,
            };
            chunkifyArray.push(obj);
          }
        } else {
          // push into existing element of chunkifyArray
          chunkifyArray[chunkifyArray.length - 1].content.push(data);
          chunkifyArray[chunkifyArray.length - 1].size += data.ContentLength;
        }
      }
    }
  });

  return { chunkifyArray, totalSize } ;
};

Now that we've prepared our readStream data from individual files, it's time for the grand finale: uploading each chunk or part to our multipart stream. Enter our superhero function, uploadMultiPartHelper! 💪📤

/**
 * Uploads a part of a multipart upload to an S3 bucket.
 *
 * @param {Buffer | Uint8Array | string} body - The content of the part to upload.
 * @param {string} bucket - The name of the S3 bucket.
 * @param {string} key - The key (path) where the part will be stored in the bucket.
 * @param {number} partNumber - The part number for the multipart upload.
 * @param {string} uploadId - The ID of the multipart upload.
 * @returns {object} - The ETag and partNumber of the uploaded part.
 * @throws {Error} - If any validation or upload error occurs.
 */
const uploadMultiPartHelper = async (body, bucket, key, partNumber, uploadId) => {
  try {
    const params = {
      Body: body,
      Bucket: bucket,
      Key: key,
      PartNumber: partNumber,
      UploadId: uploadId
    }
    const data = await S3.uploadPart(params).promise()
    return {
      ETag: data.ETag,
      PartNumber: partNumber
    }
  } catch (error) {
    throw new Error(`Upload failed: ${error.message}`)
  }
}

With this uploadMultiPartHelper function ready to roll, our multipart upload strategy is almost complete! 🎉 But wait, there's a twist in the tale! What if the total size of all our files doesn't exceed the 5 MiB mark? 🤔 Let's tackle that scenario head-on with another code snippet:

/**
 * Validates the scenario where the total size of all files doesn't exceed 5 MiB.
 *
 * @param {Object[]} data - Array of file metadata objects.
 * @param {number} totalSize - The total size of all files.
 * @param {number} THRESHOLDLIMIT_5MB - Size threshold for validation.
 * @returns {boolean} Indicates whether the total size is under the threshold.
 */
const chunkDataWriteIntoStream = async (initalArrayOfFileMetaData, THRESHOLDLIMIT_5MB ) => {
  const { chunkifyArray, totalSize } = await convertToChunks( initalArrayOfFileMetaData, THRESHOLDLIMIT_5MB );
  if (totalSize < THRESHOLDLIMIT_5MB) {
    // If we're under the 5 MiB mark,
    // let's manually handle the upload using s3.upload() by combining all files into one.
  } else {
    // As the total size is bigger than 5MB, handle using multipart chunks upload...

    const uploadId = await createMultipartUpload(bucket, key);
        const respArr = []
    for (let i = 0; i < chunkifyArray.length; i++) {
      const partNumber = i + 1;
      const body = fetchCombinedRecordsFromMultipleFileObjects(chunkifyArray[i]); // create this function to fetch data from chunks.      
            // push the chunkifyArray[i] record into multipart stream having uploadID and partnumber 'i'.
            const uploadResponse = await uploadMultiPartHelper( body, bucket, key, partNumber, uploadId );
        respArr.push(uploadResponse)
        }
        return respArr 
        // respArr = [
        //    { 
      //      ETag: '',
      //      PartNumber: 0
      //    }
        //  ]
  }
};

Turbocharging Processing Speed

Hey, so you know that loop thing (for (let i = 0; i < chunkifyArray.length; i++) {) we've got above, running through our chunks? 🔄 In tech terms, it's kinda like a slowpoke when we're in a rush, especially with time limits like the 15-minute cap we've got in Lambda functions. ⏳

But guess what? We've got a secret recipe to speed things up! 🌟✨

Ingredient 1: Let's chop our chunkifyArray into smaller bits and use the power of promises to run those bits all at the same time! 🎉🔪🚀 Imagine it like a well-coordinated dance where multiple chunks perform their tasks simultaneously.
But in the world of Lambda functions, there's a limit on how much they can handle within that 15-minute timeframe. Based on some real-world testing and tinkering, Lambdas typically handle around 95 to 100 MB of files within that time span. 🕒📁

Now, imagine this: what if we've got larger files than that? 🤯📦 That's where Lambda might start feeling a bit overwhelmed, like trying to fit an elephant through a mouse hole! 🐘🕳️

Ingredient 2: Now, here's the cherry on top! Instead of relying on Lambda's time constraints, let's switch gears and implement this in Step Functions. It's like upgrading to a turbocharged engine for processing! 🏎️💨 By using Step Functions' Map state and iterating through each chunk in parallel, we're hitting the fast lane! And guess what? To turbo-boost the speed even more, we can set a MaxConcurrencysetting while configuring the Map step! 🌪️🔥🌐

With this added perspective, we're preparing a recipe for success that considers all the ingredients and ensures we handle any file size without breaking a sweat! 🌟🚀🔍

The Grande Finale - Completing the Multipart Upload

After uploading every relevant part, it's showtime! We call in the big guns with the "Complete Multipart Upload" action. 🎉

Here's where the magic happens: Amazon S3 takes all those parts, arranges them in ascending order by part number, and voilà! 🎩✨ A brand new object is born! 🌟🧬 It's like assembling the Avengers - each part plays a vital role in creating the ultimate superhero object!

But wait, there's a catch! 🤔📏 Your proposed upload should be larger than the minimum allowed object size. Each part has to be at least 5 MiB in size, except for the very last part. It's like ensuring each puzzle piece is big enough, fitting the puzzle guidelines! 🧩📐

Now, let's dive into this helper function, the secret sauce that makes all of this possible! 🍝✨

/**
 * Completes a multipart upload to an S3 bucket and returns the uploaded object's location.
 *
 * @param {string} bucket - The name of the S3 bucket.
 * @param {string} key - The key or path for the uploaded object.
 * @param {Array<{ ETag: string, PartNumber: number }>} partArray - An array of parts with ETag and PartNumber.
 * @param {string} uploadId - The unique upload identifier for the multipart upload.
 * @returns {Promise<string>} A Promise that resolves to the uploaded object's location.
 * @throws {Error} Throws an error if the upload fails.
 */
const completeMultiPartUpload = async (bucket, key, partArray, uploadId) => {
  try {
    const params = {
      Bucket: bucket,
      Key: key,
      MultipartUpload: {
        Parts: partArray
      },
      UploadId: uploadId
    }

    const data = await S3.completeMultipartUpload(params).promise()
    return data.Location
  } catch (error) {
    throw new Error(error.message)
  }
}

Aborting Multipart Uploads: When Things Go Awry

Ever wondered what happens if an error sneaks into the multipart process? Money talks, and in this case, it's about those unwanted charges! 💰💸

Here's the deal: if an error occurs between the multipart processes, the multipart stream remains open, and that means the billing continues. Yikes! 😱 It's like a stage curtain that should be closed after the show - it's gotta be done for the costs to stop! 🎭🚫

That's where the magic of aborting the multipart stream comes into play! 🌟✨

So, let's dive into the superhero function, the abortMultiPartHelper! This function performs the crucial task of aborting a multipart upload in an S3 bucket. It's like the emergency exit button for our multipart process! 🚀🛑

/**
 * Aborts a multipart upload in an S3 bucket.
 *
 * @param {string} bucket - The S3 bucket name.
 * @param {string} key - The S3 object key.
 * @param {string} uploadId - The upload ID of the multipart upload.
 * @returns {Promise<Object>} A promise that resolves with the response from the S3 service.
 * @throws {Error} If any validation fails or an error occurs during the operation.
 */
const abortMultiPartHelper = async (bucket, key, uploadId) => {
  // Validation checks for bucket, key, and uploadId
  // It's like checking the keys before opening the treasure chest! 🔑💰

  try {
    const params = {
      Bucket: bucket,
      Key: key,
      UploadId: uploadId
    }
    const data = await S3.abortMultipartUpload(params).promise()
    return data
  } catch (error) {
    throw new Error(`Error during abortMultiPartHelper: ${error.message}`)
  }
}

Remember, this function helps in preventing those unwanted charges by stopping the multipart upload in its tracks! 🛑💼 It's the safety net we need backstage to ensure everything runs smoothly. 🌟🔧

Best Practices: Aborting Multipart Streams Safely

let's talk about some genius moves to avoid those unexpected wallet withdrawals! 💰💸

Imagine this scenario: an open stream that's silently siphoning money from your pocket - not cool, right? As savvy backend devs, its always recommended to create two fantastic Lambda functions that act like financial guards when working with multipart! 🦸‍♂️🔒
1️⃣ The Specific Stream Terminator: This Lambda function is your go-to buddy! It's like having a specific key to shut down any particular multipart stream gone rogue! 🗝️🛑

/**
 * Aborts a specific multipart stream based on the provided uploadId.
 *
 * @param {string} bucket - The S3 bucket name.
 * @param {string} key - The S3 object key.
 * @param {string} uploadId - The unique ID of the multipart stream to be aborted.
 * @returns {Promise<Object>} Resolves with the response from S3.
 * @throws {Error} If any validation fails or an error occurs during the process.
 */
const abortSpecificStream = async (bucket, key, uploadId) => {
  try {
    const params = {
      Bucket: bucket,
      Key: key,
      UploadId: uploadId
    }
    const data = await S3.abortMultipartUpload(params).promise()
    return data 
  } catch (error) {
    throw new Error(`Error during abortSpecificStream: ${error.message}`)
  }
}

2️⃣ The Stream Terminator Deluxe: This Lambda function is your ultimate guardian! It's designed to sweep through and close any open multipart streams from the past. 🌪️🔒


/**
 * Lists in-progress multipart uploads on a specific bucket.
 *
 * @param {string} bucket - The S3 bucket name.
 * @returns {Promise<Object>} A promise that resolves with the response containing in-progress multipart uploads.
 * @throws {Error} If any validation fails or an error occurs during the operation.
 */
const listMultiPartUploads = async (bucket) => {
  try {
    const params = {
      Bucket: bucket
    };
    const data = await S3.listMultipartUploads(params).promise();
    return data;
  } catch (error) {
    throw new Error(`Error during listMultiPartUploads: ${error.message}`);
  }
};
/**
 * Aborts all open multipart uploads for a given S3 bucket.
 *
 * @param {string} bucket - The S3 bucket name.
 * @returns {Promise<Object[]>} A promise resolving to an array containing information about aborted uploads.
 * @throws {Error} Throws an error if the operation encounters any issues.
 */
module.exports.abortMultiPart = async (bucket) => {
  try {
    // Fetches information about open multipart uploads
    const data = await listMultiPartUploads(bucket);
    const output = [];

    // Iterates through each open upload and aborts it
    for (const obj of data) {
      const key = obj.Key;
      const uploadId = obj.UploadId;
      // Aborts the multipart upload
      const response = await abortMultiPartHelper(bucket, key, uploadId);
      // Records the abort response for the upload
      output.push(response);
    }

    return output; // Returns an array containing information about aborted uploads
  } catch (e) {
    throw new Error(e.message); // Throws an error if any issues occur during the operation
  }
}

That's the secret sauce! With these Lambda heroes on our side, we're safeguarding against any unwanted ongoing expenses. 🌟💼 Now, that's smart backend development! 🔧👨‍💻

We've reached the finish line, folks! 🏁🎉 Across these two blogs, we've dived deep into everything about multipart upload—theory, practice, highs, lows, you name it! Hope you had a blast and picked up some cool new tricks along the way! 🚀📚
Dear, data trailblazers! 🚀📊 Thanks a million for joining this thrilling data adventure! 🛡️🎸 As we navigate through the digital realm, remember to stay safe, keep chasing those data dreams, and always seek out new knowledge! 🌟✨

See you down the data highway, fellow voyagers! Until next time—farewell! 👋🌟

References :

AWS S3 Multipart : Part 1 Introduction

Mitansh Gor — Fri, 17 Nov 2023 21:19:03 +0000

Hey there, fellow data adventurers! 🌟 Let me whisk you away on a thrilling journey into the incredible world of multipart upload—a true superhero in the realm of data management! 💥

My Journey:

Imagine this: I stood face to face with colossal data mountains—CSV files nestled in an S3 bucket. My quest? To read them, transform their essence, merge their tales, and guide them back to their digital abode.

Approach 1: Picture me, utilizing S3 readStreams, reading file chunks in parallel, all to piece together a massive string for a swift return to S3. But alas! My valiant efforts met an unexpected foe—the string limit!(9,007,199,254,740,991 (2⁵³-1) characters) It abruptly halted my progress, leaving my data adventure in disarray.

Approach 2: Enter the game-changer—multipart upload! This genius approach not only banished the string limit issue but also slashed memory usage. It transformed my journey, enhancing speed and performance in handling and manipulating vast data! 🚀✨

Welcome to the world of multipart upload, where your colossal data finds new life in manageable parts—each one a hero in its own right! 🦸‍♂️✂️ Upload them at will, in any order! If one gets jittery and fumbles on its journey, no sweat! Send it back without spoiling the whole party. 🎉🔄

Then comes the grand finale! Amazon S3 swoops in like a leader, uniting these brave parts into the original, magnificent object. Ta-da! 🦸‍♂️✨

🌟Multipart upload unfolds a captivating three-step process for your data! : You kickstart the upload, then you proceed to upload the object parts, and once all the parts are in place, you wrap up the multipart upload. 🚀📦

Multipart Upload Initiation

It all begins with the initiation - sending that golden request to start your multipart upload. 📧 Amazon S3, our gracious host, responds with an exclusive upload ID, like a backstage pass to your data concert. 🎫💽 Remember, this upload ID is your VIP ticket; it's crucial for all future interactions with your upload party.

Uploading Parts - The 'Part-y' Begins

Now, onto the fun part - uploading those object parts! Each part gets its unique number, sort of like giving your data a ticket with a seat number in the cosmic theater. 🎭🎟️ You've got a range of numbers to choose from, 1 through 10,000 - go wild! And guess what? You don't have to follow a sequence; it's like a puzzle where every piece finds its place, even if it's out of order. 🧩

Just a heads up - if you upload a new part using the same number as a previous one, it's like hitting 'overwrite' on your old data. Be cautious with that power! 💥💾

The Grande Finale - Completing the Multipart Upload

🌟 Lights, Camera, Action - it's finale time! When the curtain falls on your multipart upload, Amazon S3 takes a bow, crafting a magnificent object. How? By neatly piecing together uploaded parts in order—like a puzzle forming, piece by piece! 🧩✨

Behind the scenes: Start a multipart upload, send those parts on an adventure, and now, the big decision! Hit "complete" or halt the show. Why? That's when billing for storing parts stops. Money talk! 💸💰

Here's the deal: Only after completion or stopping, Amazon S3 frees its hold on parts' storage—no more charges! It's like clearing the stage after the show—gotta be done! 🎭🚫💼

But if you hit "stop" midway? That's a wrap! No more uploading with that ID. The show's over. Some parts might wrap up or flub their dance after your call. Ensure all parts bow out before stopping for a tidy storage exit. 🎬🎶

Example

Alright, imagine you're about to upload a massive 100 GB file using multipart upload.

Step 1: The CreateMultipartUpload call kicks things off—kind of like saying, "Let the show begin!"

Step 2: Here's where the real action happens! We've got 1000 individual UploadPart calls, each flexing its muscles by handling a 100 MB part of that hefty 100 GB file. It's like breaking down the big act into smaller, more manageable dance routines. 🎶💃

Step 1002: The grand finale! The CompleteMultipartUpload call wraps up the show, tying all those amazing parts together into one spectacular performance! 🌟✨

And just like that, in a whirlwind of 1002 API calls, your colossal 100 GB file finds its way gracefully into the cloud! 🚀🌈

Why join this multipart upload bonanza?

Speed, my friends! Upload parts simultaneously and witness your data fly into the cloud faster than a superhero chasing villains! 🚀💨
And hold on tight! If network gremlins try to play spoilsport, fear not! Smaller parts mean a hiccup won't spoil the show—it's your data's safety net. 🛡️🌐
Pause, resume—no deadlines! Start the multipart shindig; it's all under your command! ⏸️⏩
The cherry on top? Begin the adventure without knowing the object's full size—talk about thrilling! 🌍 🎉

Pro tips for the multipart madness:

High-speed network? Dive into multipart upload for multi-lane data speedways! 🚗🚗🚗
Moody internet? Multipart uploads your shield! No need to restart; just focus on the parts that need a little extra love. 🌐
Oh, and by the way, check out these multipart upload core specifications:

Item Specification

Maximum object size 5 TiB

Maximum number of parts per upload 10,000

Part Numbers 1 to 10,000 (inclusive)

Part size 5 MiB to 5 GiB.

Item	Specification
Maximum object size	5 TiB
Maximum number of parts per upload	10,000
Part Numbers	1 to 10,000 (inclusive)
Part size	5 MiB to 5 GiB.

When life gives you lemonades (large data), make lemons (make small part and use multipart), leaving life scratching its head and saying, "Wait, what?" 🍋📦🚀

There you have it, amigos! Here's your high-flying, turbo-charged glimpse into the fantastic world of multipart upload—a sneak peek before we dive deeper into the data abyss! 🌟🚀

In this session, we've just scratched the surface, giving you a sneak peek into how multipart upload works. Picture it: splitting your data into smaller heroes, sending them off to the cloud, and reuniting them in a grand spectacle! 🦸‍♂️🌐

Stay tuned for the next blog where we'll dive deeper! We'll chat about the nitty-gritty, the low-level design, and even share some code snippets. It's like unlocking the secret codes to the data universe! 📜💻

So, until our paths cross again in the blogosphere, stay safe, keep rocking those data adventures, and never stop the quest for learning! 🛡️🎸📚

Catch you later, Until then—adieu! 👋🌟

References:

Productivity in the Modern World and the Role of AI — Cursor.sh

Mitansh Gor — Tue, 12 Sep 2023 06:50:39 +0000

Alright, folks, let's talk about productivity in this crazy, fast-paced world we live in. 🚀 You know, getting stuff done efficiently is the name of the game. It's like winning at life, really!

Now, imagine you're a programmer, and you've got a gazillion lines of code to deal with. 🤯 That's where our superhero, Artificial Intelligence (AI), swoops in to save the day! 🦸‍♂️

AI is like the Robin to your Batman. It can automate all those boring, repetitive tasks, crunch data like a champ, and even help you crack those mind-boggling problems. 🤖💥

In the coding universe, we've got nifty tools like Visual Studio Code and IntelliJ IDEA, Git, and Jira that make our lives easier. They help us write, debug, and manage code like a pro.

But wait, there's more! AI takes it up a notch. 🚀🔥

Let's checkoutCursor.sh 🎉 It's like having a coding buddy who's super smart. This bad boy uses AI magic to help you write better code, squash bugs faster than a bug zapper at a summer BBQ, and even teach you some new coding tricks. 🧙‍♂️✨

And here's the icing on the cake: Cursor.sh is so developer-friendly that if you're used to working with Visual Studio Code, it's like a seamless transition. The interface, UI, and UX are all familiar, so it's like your favorite old jacket – comfy and just right.

Oh, and when you download Cursor.sh, you'll need to do some configuration settings. It's like setting up your gaming console just the way you like it. You get to add all those configurations you had in Visual Studio Code, like extensions, themes, and custom settings. It's all about making you feel at home.

Cursor.sh uses the OpenAI GPT-3 API internally to generate responses based on the input provided. It uses the openai.ChatCompletion.create() method to generate responses in a conversational manner. The API takes a series of messages as input and returns a model-generated message as output.

But that's not all! You'll also need to manually set your GitHub or GitLab credentials. It's like unlocking the secret lair with your fingerprint – only you get in.

Cursor.sh cuts down the time you spend on debugging and decoding code, making you feel like a coding ninja. 🥋👾 Plus, it's perfect for beginners because it explains things in plain English.

And if you're a team player, Cursor.sh is your wingman, helping you collaborate like a pro. 🤝

So, folks, if you want to level up your coding game and make it more fun, give Cursor.sh a spin. It's not just a tool; it's your new coding BFF! 🤖❤️

Now, go out there and conquer the coding universe with Cursor.sh by your side! 💻🌌

Advantages

Code Understanding: So, you've got this magical thing called Cursor.sh that's like your coding BFF. It's like the Sherlock Holmes of code. It takes those cryptic code snippets and makes them as clear as day. No more head-scratching, folks!
Debugging Assistance: Ever played hide-and-seek with bugs in your code? It's a nightmare! But guess what? Cursor.sh is your bug-hunting sidekick. It sniffs out those pesky bugs and squashes 'em faster than you can say "debugging."
Code Generation: Need code in a hurry? Cursor.sh’s got your back! It's like having a code genie. You make a wish, and bam! It conjures up code snippets faster than you can say "Abracadabra."
Learning Tool: Are you a coding newbie? No worries! Cursor.sh is your coding tutor. It breaks down code like a kindergarten teacher explaining ABCs. Learning to code just got as easy as 1-2-3!
Collaboration: Teamwork makes the dream work, right? Cursor.sh is your virtual team-building coach. It's the glue that holds your team together, making those group projects a piece of cake.

Disadvantages

Limited Language Support: Picture this: Cursor.sh is your multilingual friend, but it's still learning some languages. So, for some coders, it might be like talking to your pet parrot—cool but not always useful.
Accuracy: AI is pretty smart, but it's not perfect. Sometimes, it might go "Oops!" and give you suggestions that make you scratch your head. It's like asking your GPS for directions and ending up in a cornfield.
Dependency: Don't be too clingy with Cursor.sh. Relying on it too much is like using training wheels forever—you'll forget how to ride the bike on your own. Keep those coding muscles flexed!

Use Cases

Code Review: Think of Cursor.sh as your code-savvy friend who gives brutally honest feedback. It spots issues, suggests improvements, and helps you level up your coding game.
Learning New Languages: Want to be a polyglot programmer? Cursor.sh is your language tutor. It helps you decipher the secret code of new languages and frameworks.
Debugging: Bugs are the vampires of coding, right? Cursor.sh is your trusty garlic necklace. It detects those blood-sucking bugs and protects your code.
Code Generation: Need code ASAP? Cursor.sh is your code vending machine. You punch in what you need, and it delivers fresh code snacks in seconds.
Documentation: Ever been lost in the jungle of code documentation? Cursor.sh is your guide with a machete. It helps you hack through the jungle and find your way.

Here I am sharing how easy it becomes to clean, optimize and document the below snippet of code

After cleaning and optimizing the code

Now lets try adding the JSDocs as well

So, in the grand finale, here's the deal with Cursor.sh: It's like having a coding sidekick on steroids! 🚀 It turbocharges your productivity, helping you conquer code like a pro. But hey, hold on to your coding hats, folks!

Cursor.sh isn't a superhero cape that magically makes all your coding problems disappear. 🦸‍♂️ It's more like a trusty toolbox filled with cool gadgets. It can save the day, but you still gotta be the coding hero.

Think of it this way: Cursor.sh is the turbo boost for your coding journey. 🚗💨 But remember, even with the turbo boost, you've got to steer the wheel and know the map. It's like having a GPS in your car—it's a game-changer, but you still gotta know how to drive.

So, as the world goes faster than a caffeine-fueled cheetah, tools like Cursor.sh are your secret weapon. 🌪️ They're not just fancy add-ons; they're the must-haves for modern coding.

Now, go out there, embrace the future, and let Cursor.sh be your coding co-pilot. 🤖🚁 It's not the destination; it's the epic journey of coding! 🌟🚀

OVER AND OUT.

Using an Assumed Role with AWS CLI

Tejas — Mon, 04 Sep 2023 03:57:33 +0000

Configuring Default Profile

To communicate with your AWS Resources and different AWS APIs, AWS provides its CLI commands and you can do so much work using the commands. To use these commands you need to get the profiles set. So let’s do this!

To configure a default AWS profile for your CLI, AWS provides a very handy command.

You can set a default profile by running this command:

    aws configure

Terminal prompts for AWS Access Key, Secret Key, Region, and Output Format. Configuring a default profile is useful if you are going to use that profile frequently.

After running this and providing appropriate values, your ~/.aws/config file will look like this:

[default]
region=ap-south-1
output=json

~/.aws/credentials file will look like below:

[default]
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Note: In Windows, the AWS directory will be located in C:/Users/<userid>/.aws

Configuring Named Profiles

AWS CLI can have multiple profiles so that you can switch between different profiles to work with different projects. You can add the profile name you want using the —-profile option.

For example, if you want to have a profile named dev_account, you can do that by running:

aws configure --profile dev_account

The same process would follow as configuring the default profile.

Great! You have configured a named profile for yourself🙌.

Now, your config file will look like this with multiple profiles configured:

[default]
region=ap-south-1
output=json
[profile dev_account]
region=us-east-1
output=json

and your credentials file will look like this:

[default]
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
[dev_account]
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Get the list of profiles by running the following in the terminal:

aws configure list-profiles

It will display profiles with the name you added now. Cheers🚀.

Configuring an IAM Role using CLI

You can also be having different roles having different permissions. So it is useful to configure them into CLI and to access the roles using commands.

For example, if you want a profile for a role named dev_account_readonly, you can do that by running:

aws configure --profile dev_account_readonly

The terminal prompts for the same configurations, but you can skip entering Access Key and Secret Access Key while creating a role profile, these credentials will be taken from the source profile we set ahead.
To set the required properties for the role profile, use the below commands and provide appropriate values:

aws configure set source_profile dev_account --profile dev_account_readonly
aws configure set role_arn ROLE_ARN --profile dev_account_readonly
aws configure set mfa_serial MFA_SERIAL_ARN --profile dev_account_readonly

Here, we set the source profile as the profile that has the permission to assume the role and IAM Role ARN. If the account is using Multi-Factor Authentication, then you also need to set the ARN of that MFA device.

After configuring the role your config file will be looking like this:

[default]
region=ap-south-1
output=json
[profile dev_account]
region=us-east-1
output=json
[profile dev_account_readonly]
region=us-east-1
output=json
source_profile=dev_account
role_arn=ROLE_ARN
mfa_serial=MFA_SERIAL_ARN

To verify if the role has been configured or not, you can do so by running below command:

aws iam get-role --role-name dev_account_readonly --profile dev_account_readonly

Replace the role and profile name with the names you set. All the information regarding the IAM Role will be returned in the JSON format.

That’s all you require for setting the profiles in your CLI and use the Assumed Role🎉!

Datadog-Cloudformation Integration using Serverless Framework

Jay Motka — Thu, 31 Aug 2023 09:39:14 +0000

🤓 Introduction

Datadog is a cloud monitoring service provider which helps to track and analyse your cloud services from a health and operations perspective and providing support for debugging issues.

Datadog provides different integrations with multiple cloud providers like AWS, GCP, and Azure. In this blog we will be seeing Datadog integration with AWS using one of the AWS Cloudformation service with the help of Serverless framework.

📄 Pre Requisites

You will need AWS and Datadog account and knowledge of AWS services like IAMRole, Cloudformation and Serverless framework.

Note: Here we are using Serverless framework with TypeScript and Serverless Compose to deploy multiple different stack.

🛠 Datadog Cloudformation Setup

In order to create Datadog Resources using Cloudformation, Datadog needs Cloudformation Extension Plugins to be activated, so our first step is to start with adding an activation stack to our Serverless application. These extension plugins needs IAM roles to be attached in order to execute the Cloudformation events. You can find more details regarding this in Datadog Amazon Cloudformation Documentation

Note: Here you can find a basic setup guide for Datadog-Cloudformation Integration

We'll create two service stack in the Serverless application, one for the Extensions Activation which contains Cloudformation extension activation (as we'll only deploy it once because we can not deploy it multiple times to activate the same resources) and one for the Infra Resources (To create different Datadog monitors and dashboards).

🧩 Setting Up the Extension Activation Stack

Let's create a IAM Role for the Role Delegation and Extensions Executions Role.

🧮 Setting Up the IAM Role

We are providing access to Datadog AWS account (464622532012) using Role Delegation to our IAMRole with necessary permissions.

DatadogExecutionRole: {
    Type: "AWS::IAM::Role",
    Properties: {
      Path: "/",
      RoleName: `DatadogAWSIntegrationRole`,
      Description: "IAM Role to integrate Datadog using Cloudformation",
      MaxSessionDuration: 8400,
      AssumeRolePolicyDocument: {
        Version: "2012-10-17",
        Statement: [
          {
            Effect: "Allow",
            Principal: {
              AWS: "arn:aws:iam::464622532012:root",
            },
            Action: "sts:AssumeRole",
            Condition: {
              StringEquals: {
                "sts:ExternalId": ["${ssm:/datadog/integration/externalId}"],
              },
            },
          },
          {
            Effect: "Allow",
            Principal: {
              Service: [
                "hooks.cloudformation.amazonaws.com",
                "resources.cloudformation.amazonaws.com",
              ],
            },
            Action: "sts:AssumeRole",
          },
        ],
      },
      Policies: [
        {
          PolicyName: `DatadogIntegrationRolePolicy`,
          PolicyDocument: {
            Version: "2012-10-17",
            Statement: [
              {
                Sid: "VisualEditor0",
                Effect: "Allow",
                Action: [
                  "health:DescribeAffectedEntities",
                  "budgets:ViewBudget",
                  "logs:DeleteSubscriptionFilter",
                  "states:ListStateMachines",
                  "states:DescribeStateMachine",
                  "tag:GetTagValues",
                  "logs:DescribeSubscriptionFilters",
                  "cloudtrail:GetTrailStatus",
                  "codedeploy:List*",
                  "es:ListTags",
                  "cloudwatch:Describe*",
                  "elasticmapreduce:Describe*",
                  "dynamodb:Describe*",
                  "rds:Describe*",
                  "route53:List*",
                  "elasticloadbalancing:Describe*",
                  "logs:FilterLogEvents",
                  "cloudfront:GetDistributionConfig",
                  "apigateway:GET",
                  "ses:Get*",
                  "cloudtrail:LookupEvents",
                  "fsx:ListTagsForResource",
                  "support:DescribeTrustedAdvisor*",
                  "events:CreateEventBus",
                  "lambda:List*",
                  "dynamodb:List*",
                  "s3:GetBucketNotification",
                  "cloudtrail:DescribeTrails",
                  "s3:PutBucketNotification",
                  "organizations:Describe*",
                  "organizations:List*",
                  "logs:PutSubscriptionFilter",
                  "fsx:DescribeFileSystems",
                  "kinesis:Describe*",
                  "codedeploy:BatchGet*",
                  "autoscaling:Describe*",
                  "s3:GetBucketTagging",
                  "tag:GetResources",
                  "logs:DescribeLogStreams",
                  "rds:List*",
                  "s3:GetBucketLogging",
                  "tag:GetTagKeys",
                  "elasticmapreduce:List*",
                  "backup:List*",
                  "sns:Publish",
                  "elasticache:List*",
                  "redshift:DescribeLoggingStatus",
                  "cloudwatch:Get*",
                  "support:RefreshTrustedAdvisorCheck",
                  "elasticfilesystem:DescribeFileSystems",
                  "es:DescribeElasticsearchDomains",
                  "xray:GetTraceSummaries",
                  "ecs:List*",
                  "health:DescribeEvents",
                  "sqs:ListQueues",
                  "logs:DescribeLogGroups",
                  "kinesis:List*",
                  "ecs:Describe*",
                  "health:DescribeEventDetails",
                  "cloudwatch:List*",
                  "logs:TestMetricFilter",
                  "elasticfilesystem:DescribeAccessPoints",
                  "elasticache:Describe*",
                  "sns:List*",
                  "xray:BatchGetTraces",
                  "ec2:Describe*",
                  "directconnect:Describe*",
                  "es:ListDomainNames",
                  "elasticfilesystem:DescribeTags",
                  "s3:ListAllMyBuckets",
                  "cloudfront:ListDistributions",
                  "redshift:DescribeClusters",
                  "s3:GetBucketLocation",
                  "lambda:GetPolicy",
                  "secretsmanager:CreateSecret",
                  "secretsmanager:DeleteSecret",
                  "secretsmanager:DescribeSecret",
                ],
                Resource: "*",
              },
            ],
          },
        },
      ],
    },
  }

Note: This role will work as extension execution role as well, if you need to have separate roles than you can create separate roles as well.

Now we'll create four AWS::CloudFormation::TypeActivation resources to attach to Activation Stack which will include.

Datadog::Dashboards::Dashboard - to create and manage Datadog Dashboards
Datadog::Monitors::Monitor - to create and manage Datadog Monitors
Datadog::Monitors::Downtime - to create and manage Monitors Downtime
Datadog::Integrations::AWS - to create integration between AWS and Datadog

Please refer to the Cloudformation Extension - Activation Guide on how to activate the Cloudformation extensions

If we deploy this Activation Stack we will be able to create the IAMRole along with the Cloudformation Extension Activation. (Note: We'll need to deploy this stack only once)

To use the activated extensions, you will need to set the Type Configurations for each extensions plugins using below code block. Also helpful if you are creating different stages and account for deployment with same AWS account.

aws cloudformation set-type-configuration --type-name "Datadog::Monitors::Monitor" --type RESOURCE --configuration-alias ConfigurationName --configuration '{"DatadogCredentials": {"ApiKey": "DATADOG_API_KEY_PROD", "ApplicationKey": "DATADOG_APPLICATION_KEY_PROD", "ApiURL": "https://api.datadoghq.eu" }}'

Now we move on to creating the Datadog Resources such as Integration, Monitors and Dashboards.

🏗 Creating Infra Stack

We'll create a different Serverless service stack to deploy the datadog resources.

⚙️ Datadog AWS Integration

Here you will need to provide your AWS Account ID so that Datadog account can do the integration between the two AWS accounts.

RoleName would be the same as we have created in the Activation Stack before.

Note:- ExternalIDSecretName should be unique in your account Secret Manager and will be used to store the external ID by the extension.

DatadogAWSIntegration: {
    Type: "Datadog::Integrations::AWS",
    Properties: {
      AccountID: "AWS_ACCOUNT_ID", // your AWS account ID
      ExternalIDSecretName: "DatadogIntegrationExternalID",
      RoleName: "DatadogAWSIntegrationRole",
    },
},

📊 Datadog Dashboards

Dashboards are helpful to monitor the services and different widgets for any services. Find more about Dashboards in Datadog Dashboards Documentation

Dashboard: {
    Type: "Datadog::Dashboards::Dashboard",
    Properties: {
      DashboardDefinition: "", // stringified object with  dashboard configurations and definitions.
    },
},

⏱ Datadog Monitor

Monitors are to create and capture events for any particular services. You can find more about monitors in Datadog Monitors Documentation

Monitor: {
  Type: "Datadog::Monitors::Monitor",
  Properties: {
    Type: "event-v2 alert", // type of the monitor
    Query: "defines the query of the monitor to capture specific triggered events",
    Name: "monitor-name",
    Message: "message body which will be sent to the communication channel",
    Tags: [], // array of tags assigned to the monitor
    Options: {
      NotifyAudit: false,
      Locked: false,
      EnableLogsSample: false,
      Thresholds: {
        Critical: 1,
        CriticalRecovery: 0.5,
        Warning: 0.75,
      },
      NewHostDelay: 300,
      NotifyNoData: false,
      IncludeTags: true,
      EscalationMessage: "",
    },
    Priority: 1, // priority level - critical
  }

Hope this has been a worthwhile reading! ✨

Furthermore you can explore many more solutions and services that Datadog provides like Logs Forwarding, Cloud Security, Tracing, etc.

Workaround with MFA and AWS-SDK to automate manual work

Tejas — Tue, 29 Aug 2023 05:51:22 +0000

Problem Statement

Recently, due to a misconfiguration in the source system, the ETL developed by the team failed and all the extracted data ended up in the retry mechanism because the ETL function was not expecting this kind of information. However, not all the information was supposed to be discarded, as some of it was genuine and had a customer impact. Therefore, it was not possible to avoid or discard all the information at once by deleting all records.

So, it ended up after to and fro discussion, that all the retry records (~2500) will need to be checked manually and delete which are not genuine and update retry count in DynamoDB Table which needs to be retried as failed records will happen in sequence for each type and was given strict instruction that you can’t just “delete files”.

All this information and data was stored in the AWS cloud (DynamoDB, S3). Therefore, to check all 2500 files one by one is very tedious and cumbersome process. So, after using some Sheldon brain, we came up with plan that we will create some code using TypeScript and AWS-SDK as we had access so, it can be automated. So, when we wrote a script, it came as blocker that due to Multi-Factor Authentication (MFA) it was not allowing us to process it as we had specific roles and profiles to be used. Thereby, so we figure it out a way, we can use a MFA and automated this manual work which will be in step-by-step in this blog.

Pre-requisite for the solution to work

Now that you have the problem at hand and you so want to have the solution but hold on! There are bunch of items you need to have configured to make it work for you.

AWS CLI: You need to have AWS CLI installed on your machine. This is required to set up the profile to use for AWS-SDK. Here’s the link to install AWS CLI on your system.
AWS Profile: After installing AWS CLI, AWS Profile needs to be configured in the CLI so that AWS-SDK can use it to make API calls. Here’s a guide on how to configure profiles in AWS CLI.
Note: Please make sure you have both config and credentials file inside your aws directory else AWS SDK would complain for not finding the config file.

The solution to make our lives easier

Okay, now that you know both the problems and pre-requisite to solve it, let’s dive deep into the codebase that’s doing all the magic.

First we need to have clients installed for DynamoDB, which will be used to make API calls to the table and other module to generate credentials. So, let’s do it:

yarn add @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb @aws-sdk/credential-providers

Cool, now you have got all the things required to make your lives easier. So let’s start writing some TypeScript.

First we need to generate the credentials to pass it to the DynamoDB client. So, let’s write the function that returns the credentials.

import { fromIni } from "@aws-sdk/credential-providers";

function getCreds() {
  try {
    return fromIni({ // returns the credentials using your AWS Profile
      profile: "aws-profile",
      mfaCodeProvider: async serial => {
        const mfaCode = await prompt(`Type your AWS MFA code: ${serial}`);
        return mfaCode;
      },
    });
  } catch (err) {
    console.error(err);
    throw err;
  }
}

What’s happening here?

fromIni function is reading your credentials and config files that are stored in your aws directory. It will have your access and secret keys and other necessary configuration. Mixes it with your MFA code and voila, you have the credentials to make the API calls for which you have access to.
Hmm, but how would you make the terminal ask you for the MFA code? For that NodeJS has an internal package called readline . Using this we will prompt for the MFA code. Let’s look into the prompt function:

import * as readline from "readline";

function prompt(query: string): Promise<string> {
    // first create the readline interface
  const rl = readline.createInterface({
      input: process.stdin,
    output: process.stdout,
  });

    // prompt to answer a question
  return new Promise(resolve =>
    rl.question(`${query}\n`, ans => {
      rl.close();
      resolve(ans);
    })
  );
}

Just there. Let’s add the functions to make API calls to DynamoDB table and update the records in bulk, which is not natively supported by DynamoDB.

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
  ExecuteStatementCommand,
  DynamoDBDocumentClient,
  UpdateCommand,
} from "@aws-sdk/lib-dynamodb";

const updateOrder = async (
    order: SomeSchema,
    docClient: DynamoDBDocumentClient
  ): Promise<void> => {
    const updateCommand = new UpdateCommand({
      TableName: "some-table-name",
      Key: {
        id: order.id,
      },
      // update retryCount and bucketName for each unique record
      UpdateExpression: "set retryCount = :x, bucketName = :y",
      ExpressionAttributeValues: {
        ":x": 0,
        ":y": "someS3BucketName",
      },
    });
    await docClient.send(updateCommand);
  };

async function fetchAndUpdateOrders(): Promise<void> {
  const client = new DynamoDBClient({
    credentials: getCreds(), // retrieve the credentials
  });
  const docClient = DynamoDBDocumentClient.from(client);
  const command = new ExecuteStatementCommand({
    Statement: 'SELECT * FROM "some-table-name"',
  });

  const data = await docClient.send(command); // store all the retrieved data in this variable
  const orders = data.Items as Array<SomeSchema>;
  console.log(`Found a total of ${orders.length} orders`);
  await Promise.all(
        // loop through all the orders and update them
    orders.map(order => {
      console.log(
        `Updating order with id: ${order.id}`
      );
      return updateOrder(order, docClient);
    })
  );
}

That’s it. You have the solution to your problem. Now just call the fetchAndUpdateOrders function and your problem’s gone forever(until something else breaks again!).

fetchAndUpdateOrders()
  .then(() => console.log("Finished updating the records!"))
  .catch(err => {
    console.error("Something bad happened!");
    console.error(err);
  });

At the end, you will have the whole solution looking like this:

import { fromIni } from "@aws-sdk/credential-providers";
import * as readline from "readline";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
  ExecuteStatementCommand,
  DynamoDBDocumentClient,
  UpdateCommand,
} from "@aws-sdk/lib-dynamodb";

function getCreds() {
  try {
    return fromIni({ // returns the credentials using your AWS Profile
      profile: "aws-profile",
      mfaCodeProvider: async serial => {
        const mfaCode = await prompt(`Type your AWS MFA code: ${serial}`);
        return mfaCode;
      },
    });
  } catch (err) {
    console.error(err);
    throw err;
  }
}

function prompt(query: string): Promise<string> {
    // first create the readline interface
  const rl = readline.createInterface({
      input: process.stdin,
    output: process.stdout,
  });

    // prompt to answer a question
  return new Promise(resolve =>
    rl.question(`${query}\n`, ans => {
      rl.close();
      resolve(ans);
    })
  );
}

const updateOrder = async (
    order: SomeSchema,
    docClient: DynamoDBDocumentClient
  ): Promise<void> => {
    const updateCommand = new UpdateCommand({
      TableName: "some-table-name",
      Key: {
        id: order.id,
      },
      // update retryCount and bucketName for each unique record
      UpdateExpression: "set retryCount = :x, bucketName = :y",
      ExpressionAttributeValues: {
        ":x": 0,
        ":y": "someS3BucketName",
      },
    });
    await docClient.send(updateCommand);
  };

async function fetchAndUpdateOrders(): Promise<void> {
  const client = new DynamoDBClient({
    credentials: getCreds(), // retrieve the credentials
  });
  const docClient = DynamoDBDocumentClient.from(client);
  const command = new ExecuteStatementCommand({
    Statement: 'SELECT * FROM "some-table-name"',
  });

  const data = await docClient.send(command); // store all the retrieved data in this variable
  const orders = data.Items as Array<SomeSchema>;
  console.log(`Found a total of ${orders.length} orders`);
  await Promise.all(
        // loop through all the orders and update them
    orders.map(order => {
      console.log(
        `Updating order with id: ${order.id}`
      );
      return updateOrder(order, docClient);
    })
  );
}

fetchAndUpdateOrders()
  .then(() => console.log("Finished updating the records!"))
  .catch(err => {
    console.error("Something bad happened!");
    console.error(err);
  });

Say hello to seamless bulk updates & goodbye to manual tweaks!

Cross account resource access in IAM

Pruthvi — Thu, 24 Aug 2023 07:19:48 +0000

For some AWS services, you can grant cross-account access to your resources using IAM. To do this, you can attach a resource policy directly to the resource that you want to share, or use a role as a proxy.

To share the resource directly, the resource that you want to share must support resource-based policies. Unlike an identity-based policy for a role, a resource-based policy specifies who (which principal) can access that resource.

The following list includes some of the AWS services that support resource-based policies. For a complete list of the growing number of AWS services that support attaching permission policies to resources instead of principals, see AWS services that work with IAM and look for the services that have Yes in the Resource Based column.

Amazon S3 buckets — The policy is attached to the bucket, but the policy controls access to both the bucket and the objects in it. For more information, see Access Control in the Amazon Simple Storage Service User Guide. In some cases, it may be best to use roles for cross-account access to Amazon S3. For more information, see the example walkthroughs in the Amazon Simple Storage Service User Guide.
Amazon Simple Notification Service (Amazon SNS) topics
Amazon Simple Queue Service (Amazon SQS) queues

Solution

Delegating AWS permissions in a resource-based policy

If a resource grants permissions to principals in your account, you can then delegate those permissions to specific IAM identities. Identities are users, groups of users, or roles in your account. You delegate permissions by attaching a policy to the identity. You can grant up to the maximum permissions that are allowed by the resource-owning account.

Important

In cross account access, a principal needs an Allow in the identity policy and the resource-based policy.

Assume that a resource-based policy allows all principals in your account full administrative access to a resource. Then you can delegate full access, read-only access, or any other partial access to principals in your AWS account. Alternatively, if the resource-based policy allows only list permissions, then you can delegate only list access. If you try to delegate more permissions than your account has, your principals will still have only list access.

For example, assume that you manage AccountA and AccountB. In AccountA, you have an Amazon SQS Queue named QueueA.

You attach a resource-based policy to QueueA that allows all principals in AccountB to perform SQS:SendMessage Action. They can send message to QueueA

Here we can use AWS::SQS::QueuePolicy or AWS::SQS::QueueInlinePolicy

Amazon SQS Queue Policy

The following sample is a queue policy that allows AWS account 444455556666(AccountA) to send and receive messages on queue queueA. You add the policy to the resources section of your template.
```
SampleSQSPolicy: 
  Type: AWS::SQS::QueuePolicy
  Properties:
        # 444455556666 :- AccountA Id 
    Queues: 
      - "https://sqs:us-east-2.amazonaws.com/444455556666/queueA"
    PolicyDocument: 
      Statement: 
        - Action: 
            - "SQS:SendMessage"
          Effect: "Allow"
          Resource: "arn:aws:sqs:us-east-2:444455556666:queueA"
          Principal:  
            AWS: 
              - "111122223333" # AccountB Id
```
AccountA gives AccountB gives access to QueueA by naming AccountB as a principal in the resource-based policy. As a result, AccountB is authorized to perform SQS:SendMessage action on QueueA, and the AccountB administrator can delegate access to its users in AccountB.
The AccountB root user has all of the permissions that are granted to the account. Therefore, the root user has full access to BucketA.
In AccountB, attach a policy such that any resource(ex. lambda) can perform SQS:SendMessage action.

iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "sqs:SendMessage"
        - "sqs:ReceiveMessage"
      Resource:
         - "arn:aws:sqs:us-east-2:444455556666:queueA"

For more information about complex uses of roles, such as granting access to applications and services, see Common scenarios for roles: Users, applications, and services.

Important

🚨 Give access only to entities you trust, and give the minimum level of access necessary. Whenever the trusted entity is another AWS account, any IAM principal can be granted access to your resource. The trusted AWS account can delegate access only to the extent that it has been granted access; it cannot delegate more access than the account itself has been granted.

More information about Principal in resource based policy:

AWS JSON policy elements: Principal - AWS Identity and Access Management

Reference:

Cross account resource access in IAM - AWS Identity and Access Management

AWS::SQS::QueuePolicy - AWS CloudFormation

AWS::SQS::QueueInlinePolicy - AWS CloudFormation

DEV Community: Distinction Dev

Event driven architecture : Overview and comparison of AWS Messaging services

In this Article

Overview of Event Driven Architecture ✨

Event Driven Architecture Common Model

1. Point to Point Model

2. Pub Sub Model in EDA

🚀 AWS messaging services

Simple Queue Service (SQS)

Use Case of SQS ✴️

Model / Mechanism

Consumers

Supports Ordering Mechanism

Conditional Message Filtering

Encryption

Throughput

Dead Letter Queue

Pricing 🤑

Simple Notification Service (SNS)

Use Cases of SNS ✴️

Model / Mechanism

Consumers

Supports Ordering Mechanism

Conditional Message Filtering

Encryption

Dead Letter Queue

Throughput

Pricing 🤑

Combined Use Case: SNS and SQS Integration 🤝

Event Bridge

Use Cases of Eventbridge ✴️

Model / Mechanism

Consumers

Supports Ordering Mechanism

Conditional Message Filtering

Encryption

Archive and Event Replay

Throughput

Pricing 🤑

AWS Kinesis Datastream

Use cases of Kinesis ✴️

Model / Mechanism

Consumers

Archive and Event Replay

Throughput

Drawbacks

Pricing 🤑

Apache Kafka (AWS MKS)

Use Cases of Apache Kafka ✴️

Model / Mechanism

Consumers

Supports Ordering Mechanism

Conditional Message Filtering

Encryption

Archive and Event Replay

Throughput

Pricing

📗 Very Important to know when to choose which service ?

➕ Additional Considerations

Summary Table

Move aws resources from one stack to another cloudformation stack

Why do we need this?

Steps to move resources from one stack to another stack

Prerequisites

1. Copy AWS resources from the main cloudFormation stack and paste them into the destination cloudFormation stack.

2. Remove resources from the main stack and deploy the main stack.

3. Prepare another file named "resourcesToImport.txt" containing the aws resource type, logical ID, and resource identifier.

5. Execute a command to apply the changeset.

Below link contains all the resources which are allowed to import in cloudformation stack

Reference

React Rendering Guide

The Problem

State Updates and Re-renders

Bursting the re-renders Myth

Moving state down (The solution)

YAML Magic: Unveiling the Secrets of Aliases, Anchors, and Overrides!

Alias All the Things!

Anchors: Your Data's Home Base

Overrides: A Splash of Individuality

Conclusion: YAML Sorcery Unleashed!