DEV Community: Aravind V

GenLearn - Your Personalized Learning Assistant!

Aravind V — Sun, 10 Mar 2024 05:25:59 +0000

🚀 Introducing: GenLearn – Your Personalized Learning Assistant! 📚
Built on PartyRock Playground during #partyrock-hackathon

GenLearn - Your Personalised Learning Assistant!

Get Started with GenLearn today and unlock your full learning potential! 🌐

App link
https://partyrock.aws/u/AravindVCyber/2N3IdneP6/GenLearn-Your-Personalised-Learning-Assistant!

Snapshot
https://partyrock.aws/u/AravindVCyber/2N3IdneisP6/GenLearn-Your-Personalised-Learning-Assistant!/snapshot/qDBzjmXhW

Try some experiment with this and share your snapshots or create your own app with Partyrock

Post in Community.AWS

Community | GenLearn - Your Personalised Learning Assistant!

GenLearn helps users learn, summarise, and present any topic with presentation slide notes in language of choice with links to Search, YouTube, Wikipedia, Images to unlock learning possibilities

community.aws

Do like and share my hackathon submission in DEVPOST

GenLearn - Personalised Learning Assistant | Devpost

GenLearn helps users learn, summarise, and present any topic with presentation slide notes in language of choice with links to Search, YouTube, Wikipedia, and Images to unlock learning possibilities

devpost.com

GenLearn

GenLearn helps users learn, summarise, and present any topic with presentation slide notes in language of choice with links to Search, YouTube, Wikipedia, and Images to unlock learning possibilities
Are you tired of spending countless hours scouring through textbooks and online resources to grasp a concept or prepare a presentation? Look no further! GenLearn is here to revolutionize your learning experience.
Say goodbye to tedious research and hello to efficient, personalised learning with GenLearn. Join us in shaping the future of education and empowerment. Let's learn smarter, together! 🌟

Problem Addressed:

In today's fast-paced world, individuals often struggle to find the time and resources needed to effectively learn and present information. Traditional methods of learning can be time-consuming and inefficient, leaving learners feeling overwhelmed and uninspired.
Solution:
GenLearn offers a comprehensive solution to this problem by leveraging generative technologies to create personalised learning materials and presentation slide notes. By harnessing the power of AI, users can quickly access curated content tailored to their learning preferences, saving time and effort.

Key Features:

Personalised Learning: Tailored learning materials and presentation slide notes on any topic.
Multi-Language Support: Content available in your preferred language for global accessibility.
Actionable Links: Direct access to Google Search, YouTube, Wikipedia, and Images for further exploration.
Efficiency: Streamlined learning process, allowing users to focus on understanding and application rather than searching for information.
Assistant: Bot is ready to clarify further on the topic requested

Interesting Hacks:

I also used some simple hacks to let the LLM know how to format the search links under different languages, and the results are amazing, with a few snippets of code like the below the LLM is able to understand how to generate links

import wikipedia

def generate_google_search_link (query):
    return f"https://www.google.com/search?q={'+'.join(query.split())}"

def generate_google_image_search_url (query):
    base_url = "https://www.google.com/search" 
    params = { "tbm": "isch", "q":query }
    url = base_url + "?" + "&".join([f"{key}={value}" for key, value in params.items()]) 
    return url 

def generate_youtube_links(query, num_links=1): 
    youtube_links = [] 
    for i in range(1, num_links + 1): 
        youtube_links.append(f"[YouTube Video {i}](https://www.youtube.com/results?search_query={'+'.join(query.split())}&page={i})") 
    return youtube_links 

def generate_wikipedia_links(query, num_links=1): 
    wikipedia_links = [] 
    search_results = wikipedia.search(query, results=num_links) 
    for result in search_results: 
         page_title = wikipedia.page(result).title 
         wikipedia_links.append(f"[{page_title}](https://en.wikipedia.org/wiki/{'_'.join(page_title.split())})") 
    return wikipedia_links

def generate_combo_links(query, google_links=1, youtube_links=1, wikipedia_links=1): 
    google_links = generate_google_search_link(query) 
    google_images_link = generate_google_image_search_url(query) 
    youtube_links = generate_youtube_links(query, youtube_links) 
    wikipedia_links = generate_wikipedia_links(query, wikipedia_links) 
    combo_links = { "Google Search": google_links, "Google Images": google_images_link, "YouTube Videos": youtube_links, "Wikipedia Pages": wikipedia_links } 
    return combo_links

suggested readings as hyperlinks

Notes on the developmental journey for the hackathon.

Inspiration

What it does

Whether you're a student, a professional, or a lifelong learner, GenLearn adapts to your needs, providing concise summaries and engaging content in your preferred language.
But that's not all! With actionable links to Google Search, YouTube, Wikipedia, and Images, you'll have a wealth of additional resources at your fingertips, enriching your learning journey like never before.
Say goodbye to tedious research and hello to efficient, personalised learning with GenLearn. Join us in shaping the future of education and empowerment. Let's learn smarter, together! 🌟

How we built it

Our app utilises cutting-edge generative solutions to create tailored learning materials and presentation slide notes on any topic you desire.
We have used Amazon partyrock playground to build this app, which helped us with easy access to generative llm models like Claude and Stability AI

Challenges we ran into

While connecting widget, we occasionally hand trouble in getting the required out in relevant format and under the current scope, eventually we iterated this using some prompt engineering to get around with the solution.
Had some challenges in getting the hyperlinks and eventually we mastered getting the links with the right model

Accomplishments that we're proud of

I think this is only the tip of the iceberg, if we could do this and help students with an interactive experience for students to learn and personalise the content in their language and summaries it with partyrock, it opens a lot of possibilities.

What we learned

When we started it is was giving output more similar to a usual ChatGpt query, but when we linked the various widgets, we managed to get a interactive experience. More over we started with text outputs, and later converted it to markdown and ensured to use emojis, to make the students feel cool. Besides which we went ahead and try to generate web search link to help student explore beyond and help opened the generated content to reference. Language translation would also be a great experience for students

What's next for Personalised Learning Assistant

I believe we could generate actionable links could help students reference and explore wide content from the web.
At the same time they could also be used to track user events to further interact with the playground
Also currently the playground does not support additional input component types, and input files/video. This could open more possibilities to make the interactive experience cool
It could as well in future support audio or video playback, and it could really make the learning experience engaging
Also the app is still using open pre-learned material, hopefully in future we could use RAG feeds to help industrial use cases make similar playground to benefits their users with relevant and appropriate data, more personalisation.

Other Features and Images:

Fun Facts for Users on Topic

Easy summary in the form of slides for presentation or notes
summarised notes in the form of slides 1

summarised notes in the form of slides 2

Brief explanation on the topic in the requested language

Casual image generation based on topic

Get Started with GenLearn today and unlock your full learning potential! 🌐

App link
https://partyrock.aws/u/AravindVCyber/2N3IdneP6/GenLearn-Your-Personalised-Learning-Assistant!

Snapshot
https://partyrock.aws/u/AravindVCyber/2N3IdneisP6/GenLearn-Your-Personalised-Learning-Assistant!/snapshot/qDBzjmXhW

Feel free to share any constructive feedback and reach me at twitter @Aravind_V7

Do like and share my hackathon submission in DEVPOST

https://devpost.com/software/personalised-learning-assistant-5vr28l

Code Reviewer using PartyRock

Aravind V — Thu, 16 Nov 2023 19:27:28 +0000

🚀 Exciting News! ❇ Introducing the Power of PartyRock's Interactive AI Tool - Code Reviewer! 🤖💻

Try the app I have build with PartyRock in a few hours: Code Reviewer
https://partyrock.aws/u/AravindVCyber/xLxhkm29u/Code-Reviewer

🌟 Unveiling a productive tool: Code Reviewer - an AI-driven, interactive, and generative tool that's transforming the way we review/refactor and optimize code in a single place! 🌐🔍

👨‍💻 Ever felt stuck during code reviews or while understanding completely new or complex code bases? Say hello to Code Reviewer! This incredible tool harnesses the prowess of PartyRock's advanced playground to leverage generative AI models, helping with the code review/refactoring process.

🔍 With Code Reviewer, dive into your codebase and witness the magic unfold:
Refactoring Recommendations: Get insightful suggestions for code improvements.

Unit Test Insights: Identify areas to add enhanced test coverage.
Vulnerability Detection: Unearth potential vulnerabilities in your code.
Code Smell Recognition: Sniff out areas that could use a cleanup.
Infer design patterns, data structures and algorithms and optimize for time and space complexity.

🧠 But that's not all! Code Reviewer goes beyond just identifying issues. It:
Provides actionable insights on enhancing maintainability and scalability.
Explains algorithms, data structures, and principles used within your code.
Ensures compliance with best practices like SOLID, DRY, and more!

⚡️ Seamlessly integrated with PartyRock, Code Reviewer is your ultimate coding companion, helping you write cleaner, more efficient, and secure and optimized code. 🚀

✨Besides this you can chat with the summary assistants to focus on your expectations on the code and get explanations.

✨
🌐 Elevate your coding experience today with Code Reviewer! Try it out and witness the AI-driven evolution in code optimization! 🚀💡

Try the app I have build with PartyRock in a few hours: Code Reviewer
https://partyrock.aws/u/AravindVCyber/xLxhkm29u/Code-Reviewer

Enjoy building more 🚀 with #PartyRock

Know more about PartyRock at https://lnkd.in/ghwy7rns

AI #CodeReviewer #Innovation #CodeOptimization #AIinCoding #TechInnovation #aws #awscommunity

Limitations and Solutions to consider while using SQS ️

Aravind V — Sun, 18 Dec 2022 09:22:44 +0000

Amazon Simple Queue Service (SQS) is a fully managed, distributed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. While SQS is a powerful and reliable service, it does have some limitations that you should consider when using it. In this article, we will explore some of these limitations and discuss potential solutions that you can consider to overcome them.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

Check out lots more coming direct to your mailbox by joining my free newsletter

🔗 Exploring Serverless @ substack

Exploring Serverless | Aravind Vadamalaimuthu | Substack

Exploring Serverless. Click to read Exploring Serverless, by Aravind Vadamalaimuthu, a Substack publication. Launched 2 years ago.

exploringserverless.substack.com

Please find notable limitations in using SQS services below:

Message size limit,
Message retention period,
Visibility timeout,
Delivery guarantees,
Throughput limits,
Regional availability

In this article, we will discuss these limitations in more detail and provide potential solutions that you can consider to overcome them. By understanding these limitations and implementing appropriate solutions, you can ensure that SQS is an effective and reliable message queuing service for your applications.

Message size limit: 🪂

Each message in an SQS queue has a maximum size limit of 256 KB. If you need to send larger messages, you may need to use a different service, such as Amazon SNS or Amazon Kinesis.

Solutions to use large message size 🐳

To overcome the message size limit of Amazon Simple Queue Service (SQS), you can use the following approaches:

Split the message into smaller chunks:

If your message is larger than the size limit, you can split it into smaller chunks and send multiple messages for each chunk. You can then have the consumer reassemble the chunks into the original message. This approach can be used for messages of any size, as long as you have the necessary logic to split and reassemble the message.

import * as AWS from 'aws-sdk';

// Set up an SQS client
const sqs = new AWS.SQS();

// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';

// Define the maximum chunk size
const chunkSize = 200000; // 200 KB

// Split the message into chunks
const chunks = [];
for (let i = 0; i < largeMessage.length; i += chunkSize) {
  chunks.push(largeMessage.slice(i, i + chunkSize));
}

// Send a message for each chunk
for (const chunk of chunks) {
  sqs.sendMessage({ QueueUrl: queueUrl, MessageBody: chunk }, (err) => {
    if (err) {
      console.error(err);
    }
  });
}

This example uses a loop to split the largeMessage string into chunks of chunkSize bytes and then sends a message for each chunk using the sendMessage method of the SQS client.

To reassemble the message on the consumer side, you can use a similar approach to receive and process the messages in a loop:

import * as AWS from 'aws-sdk';

// Set up an SQS client
const sqs = new AWS.SQS();

// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';

// Receive and process messages from the queue
while (true) {
  sqs.receiveMessage({ QueueUrl: queueUrl, MaxNumberOfMessages: 10 }, (err, data) => {
    if (err) {
      console.error(err);
      return;
    }
    for (const message of data.Messages) {
      // Concatenate the message body with the reconstructed message
      reconstructedMessage += message.Body;
      // Delete the message from the queue
      sqs.deleteMessage({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }, (delErr) => {
        if (delErr) {
          console.error(delErr);
        }
      });
    }
  })
}

Using message attributes:

SQS allows you to attach up to 10 message attributes to each message, which can be used to store additional metadata or data. You can use message attributes to store parts of a larger message and then have the consumer retrieve and reassemble the message using the attributes. This approach is limited to a total of 1 KB of message attributes per message.

Amazon Simple Queue Service (SQS) allows you to attach up to 10 message attributes to each message, which can be used to store additional metadata or data. You can use message attributes to store parts of a larger message and then have the consumer retrieve and reassemble the message using the attributes. This approach is limited to a total of 1 KB of message attributes per message.

You can use message attributes to send a large message using the AWS SDK for JavaScript (AWS.SQS):

import * as AWS from 'aws-sdk';

// Set up an SQS client
const sqs = new AWS.SQS();

// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';

// Define the maximum chunk size
const chunkSize = 900; // 900 bytes

// Split the message into chunks
const chunks = [];
for (let i = 0; i < largeMessage.length; i += chunkSize) {
  chunks.push(largeMessage.slice(i, i + chunkSize));
}

// Send a message for each chunk
for (let i = 0; i < chunks.length; i++) {
  const messageAttributes = {
    chunkNumber: { DataType: 'Number', StringValue: i.toString() }
  };
  sqs.sendMessage({ QueueUrl: queueUrl, MessageBody: chunks[i], MessageAttributes: messageAttributes }, (err) => {
    if (err) {
      console.error(err);
    }
  });
}

This example uses a loop to split the largeMessage string into chunks of chunkSize bytes and then sends a message for each chunk using the sendMessage method of the SQS client. The message attributes include a chunkNumber attribute that indicates the order of the chunk in the original message.

To receive a large message using Amazon Simple Queue Service (SQS), please follow the below example

import * as AWS from 'aws-sdk';

// Set up an SQS client
const sqs = new AWS.SQS();

// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';

// Receive and process messages from the queue
while (true) {
  sqs.receiveMessage({ QueueUrl: queueUrl, MaxNumberOfMessages: 10 }, (err, data) => {
    if (err) {
      console.error(err);
      return;
    }
    for (const message of data.Messages) {
      // Get the chunk number from the message attributes
      const chunkNumber = parseInt(message.MessageAttributes.chunkNumber.StringValue, 10);
      // Store the chunk in the reconstructed message array
      reconstructedMessage[chunkNumber] = message.Body;
      // Delete the message from the queue
      sqs.deleteMessage({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }, (delErr) => {
        if (delErr) {
          console.error(delErr);
        }
      });
    }
  });
}

// Reassemble the message from the chunks
const reconstructedMessage = reconstructedMessage.join('');

This example uses a loop to receive and process messages from the queue using the receiveMessage method of the SQS client. For each message, it gets the chunkNumber attribute from the message attributes and stores the message body in the corresponding position in the reconstructedMessage array.

Once all the messages have been received and processed, the reconstructedMessage array can be joined to form the original message.

If you need to send larger messages, you can use a different service that supports larger message sizes, such as Amazon SNS or Amazon Kinesis. These services offer various options for sending large payloads, such as message batching, compression, and chunking.

Two popular options are Amazon Simple Notification Service (SNS) and Amazon Kinesis. These services offer various options for sending large payloads, such as message batching, compression, and chunking.

Using SNS service:

You can send a large message using Amazon SNS and with the AWS SDK for JavaScript (AWS.SNS):

import * as AWS from 'aws-sdk';

// Set up an SNS client
const sns = new AWS.SNS();

// Define the topic ARN
const topicArn = 'arn:aws:sns:ap-south-1:71717177117171:my-topic';

// Send the message
sns.publish({ TopicArn: topicArn, Message: largeMessage }, (err) => {
  if (err) {
    console.error(err);
  }
});

This example uses the publish method of the SNS client to send the largeMessage string to the specified topic. SNS supports message sizes up to 2 MB.

To receive and process the message on the consumer side, you can set up a subscription to the SNS topic and specify a delivery mechanism, such as an SQS queue or an HTTP/S endpoint. The consumer can then receive and process the messages as usual.

You can set up an SQS queue as a subscription to an SNS topic:


import * as AWS from 'aws-sdk';

// Set up an SNS client
const sns = new AWS.SNS();

// Define the topic ARN and queue URL
const topicArn = 'arn:aws:sns:ap-south-1:71717177117171:my-topic';
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';

// Create the subscription
sns.subscribe({
  Protocol: 'sqs',
  TopicArn: topicArn,
  Endpoint: queueUrl
}, (err) => {
  if (err) {
    console.error(err);
  }
});

This example uses the subscribe method of the SNS client to create a subscription to the specified SNS topic, using the SQS queue as the delivery mechanism.

To receive and process the messages from the SQS queue, you can use the same approach as in the previous examples, using the receiveMessage and deleteMessage methods of the SQS client.

To set up an HTTP or HTTPS endpoint as a subscription for an SQS queue, you will need to create an Amazon Simple Notification Service (SNS) topic and then configure the topic to send notifications to the HTTP or HTTPS endpoint.

Here is an overview of the process:

Create an SNS topic and choose "HTTP/S" as the protocol for the endpoint.
Provide the URL for the HTTP or HTTPS endpoint as the endpoint address.
Create an SQS queue and choose "SNS" as the protocol for the subscription.
Choose the SNS topic that you created in step 1 as the topic for the subscription.

Once you have set up the subscription, any messages that are sent to the SQS queue will be forwarded to the HTTP or HTTPS endpoint as a POST request. The body of the request will contain a JSON object with details about the message, including the message body and any metadata.

It's important to note that the HTTP or HTTPS endpoint must be able to process the POST request and handle the message promptly. If the endpoint is unable to process the message, it may be retried multiple times before the message is considered to have failed delivery.

import * as cdk from 'aws-cdk-lib';
import * as sqs from 'aws-cdk-lib-sqs';

export class MyStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create an SQS queue
    const queue = new sqs.SqsQueue(this, 'MyQueue', {
      visibilityTimeout: cdk.Duration.seconds(300)
    });

    // Create an HTTP subscription for the queue
    const subscription = new sqs.HttpSubscription(this, 'MySubscription', {
      queue,
      endpoint: 'https://example.com/endpoint'
    });
  }
}

Using Kinesis service:

Amazon Web Services (AWS) Kinesis is a fully managed streaming data platform that enables real-time processing of streaming data at scale. It is designed to handle high volume and velocity data streams, such as clickstreams, log files, and social media feeds.

In this case, the SQS queue is being used to decouple Kinesis. This means that the system that is producing the data stream (e.g. a log generator or social media platform) can send the data to the Kinesis stream without needing to directly communicate with the consumer of the data (e.g. an analytics system or machine learning model). Instead, the producer can simply add a message to the SQS queue, which will trigger further processing to send the data to the Kinesis stream. This decoupling can make it easier to scale and maintain the system, as the producer and consumer can operate independently without needing to directly communicate with each other.

import * as cdk from 'aws-cdk-lib';
import * as kinesis from '@aws-cdk/aws-kinesis';
import * as sqs from '@aws-cdk/aws-sqs';

export class MyStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create a Kinesis stream
    const stream = new kinesis.Stream(this, 'MyStream', {
      shardCount: 1
    });

    // Create an SQS queue
    const queue = new sqs.Queue(this, 'MyQueue', {
      visibilityTimeout: cdk.Duration.seconds(300)
    });

    // Send data to the Kinesis stream and post a message to the SQS queue once it has been sent
    stream.grantWrite(queue);
  }
}

This code creates a new Kinesis stream with a single shard and an SQS queue with a visibility timeout of 300 seconds. It then grants the queue write permissions to the stream, which allows it to send data to the stream. When data is added to the stream, it will trigger further processing to send the data to any consumers that are subscribed to the stream. The message that is posted to the SQS queue will contain information about the data that was sent to the stream, such as the partition key and the sequence number. This can be used to track the progress of the data through the system.

import { Kinesis } from 'aws-sdk';

const kinesis = new Kinesis();

export async function handler(event: any) {
  console.log(`Received event: ${JSON.stringify(event, null, 2)}`);

  const message = event.Records[0].body;
  const data = JSON.parse(message);

  // Get the shard iterator for the specified shard and sequence number
  const result = await kinesis.getShardIterator({
    StreamName: process.env.STREAM_NAME,
    ShardId: data.ShardId,
    ShardIteratorType: 'AT_SEQUENCE_NUMBER',
    StartingSequenceNumber: data.SequenceNumber
  }).promise();

  const shardIterator = result.ShardIterator;

  // Read data from the stream using the shard iterator
  let records = await kinesis.getRecords({
    ShardIterator: shardIterator,
    Limit: 1
  }).promise();

  while (records.Records.length > 0) {
    // Process the records that were read from the stream
    for (const record of records.Records) {
      if (record.PartitionKey === data.PartitionKey) {
        // This record has the desired partition key, process it
        // ...
      }
    }

    // Get the next shard iterator
    shardIterator = records.NextShardIterator;

    // Read the next batch of records
    records = await kinesis.getRecords({
      ShardIterator: shardIterator,
      Limit: 1
    }).promise();
  }

  // Delete the message from the queue once it has been processed
  const deleteResult = await sqs.deleteMessage({
    QueueUrl: process.env.QUEUE_URL,
    ReceiptHandle: event.Records[0].receiptHandle
  }).promise();

  console.log(`Deleted message: ${JSON.stringify(deleteResult, null, 2)}`);
}

This code reads a single record from the stream using the ShardIterator from the message and the getRecords method of the Kinesis client. It then processes the data that was read from the stream. Finally, it deletes the message from the queue using the deleteMessage method of the SQS client. This will mark the message as processed and remove it from the queue.

Message retention period: 🪂

SQS queues can retain messages for a maximum of 14 days. If you need to retain messages for a longer period, you may need to use a different service, such as Amazon S3 or Amazon Glacier.

The message retention period of an Amazon Simple Queue Service (SQS) queue determines how long a message is retained in the queue before it is automatically deleted. The default retention period is 4 days, and the maximum retention period is 14 days.

You can set the message retention period when you create a queue or update an existing queue using the SetQueueAttributes action of the SQS API or the corresponding method of the AWS SDKs.

import * as AWS from 'aws-sdk';

// Set up an SQS client
const sqs = new AWS.SQS();

// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';

// Set the message retention period to 7 days
sqs.setQueueAttributes({
  QueueUrl: queueUrl,
  Attributes: {
    MessageRetentionPeriod: '604800' // 7 days in seconds
  }
}, (err) => {
  if (err) {
    console.error(err);
  }
});

If you need to retain messages for a longer period, you can use a different service, such as Amazon S3 or Amazon Glacier. These services offer long-term storage options that can store messages indefinitely or for a specified period.

You can use Amazon S3 to store messages from an SQS queue for long-term retention:

import * as AWS from 'aws-sdk';

// Set up an SQS client
const sqs = new AWS.SQS();

// Set up an S3 client
const s3 = new AWS.S3();

// Define the queue URL and bucket name
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
const bucketName = 'my-s3-bucket';

// Receive and process messages from the queue
while (true) {
  sqs.receiveMessage({ QueueUrl: queueUrl, MaxNumberOfMessages: 10 }, (err, data) => {
    if (err) {
      console.error(err);
      return;
    }
    for (const message of data.Messages) {
      // Store the message in S3
      s3.putObject({
        Bucket: bucketName,
        Key: message.MessageId,
        Body: message.Body
      }, (putErr) => {
        if (putErr) {
          console.error(putErr);
          return;
        }
        // Delete the message from the queue
         sqs.deleteMessage({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }, (delErr) => {
        if (delErr) {
          console.error(delErr);
        }
      });
      }
    }
    })
}

This example uses a loop to receive and process messages from the queue using the receiveMessage method of the SQS client. For each message, it stores the message body in an S3 bucket using the putObject\ method of the S3 client and then deletes the message from the queue using the deleteMessage method of the SQS client.

To retrieve the messages from S3, you can use the getObject method of the S3 client. You can also set up lifecycle policies on the S3 bucket to automatically delete or transition the messages to Amazon Glacier after a specified period.

Visibility timeout: 🪂

When a message is received from an SQS queue, it becomes hidden from other consumers for a specified period, known as the visibility timeout. If the consumer fails to process the message within this timeout, the message will become visible again and may be processed by another consumer. This can lead to duplicate processing if not handled properly.

The visibility timeout of an Amazon Simple Queue Service (SQS) queue is the amount of time that a message is hidden from other consumers when it is being processed. This is to prevent other consumers from processing the same message simultaneously, which can lead to duplicate processing of the message.

The visibility timeout is specified in seconds and has a maximum value of 12 hours (43200 seconds). The default value is 30 seconds.

The visibility timeout of an Amazon Simple Queue Service (SQS) queue is designed to prevent multiple consumers from processing the same message simultaneously, which can lead to duplicate processing of the message. However, there may be situations where you need to extend the visibility timeout of a message to allow more time for processing.

To extend the visibility timeout of a message in an SQS queue, you can use the ChangeMessageVisibility action of the SQS API or the corresponding method of the AWS SDKs.

You can extend the visibility timeout of a message by 5 minutes (300 seconds) using the AWS SDK for JavaScript (AWS.SQS):

import * as AWS from 'aws-sdk';

// Set up an SQS client
const sqs = new AWS.SQS();

// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';

// Receive and process messages from the queue
while (true) {
  sqs.receiveMessage({ QueueUrl: queueUrl, MaxNumberOfMessages: 10 }, (err, data) => {
    if (err) {
      console.error(err);
      return;
    }
    for (const message of data.Messages) {
      // Process the message
      processMessage(message, (processErr) => {
        if (processErr) {
          console.error(processErr);
          // Extend the visibility timeout by 5 minutes
          sqs.changeMessageVisibility({
            QueueUrl: queueUrl,
            ReceiptHandle: message.ReceiptHandle,
            VisibilityTimeout: 300
          }, (visErr) => {
            if (visErr) {
              console.error(visErr);
            }
          });
        } else {
          // Delete the message from the queue
          sqs.deleteMessage({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }, (delErr) => {
            if (delErr) {
              console.error(delErr);
            }
          });
        }
      });
    }
  });
}

This example uses a loop to receive and process messages from the queue using the receiveMessage method of the SQS client. For each message, it calls the processMessage function to process the message. If the processMessage function returns an error, it means that the message processing failed, and the visibility timeout of the message is extended by 5 minutes using the changeMessageVisibility method of the SQS client. If the processMessage function succeeds, the message is deleted from the queue using the deleteMessage method of the SQS client.

Delivery guarantees: 🪂

SQS is a best-effort delivery service, which means that it does not guarantee the order of messages or that every message will be delivered. If you need stronger delivery guarantees, you may need to use a different service, such as Amazon SNS or Amazon Kinesis.

Amazon Simple Queue Service (SQS) provides different delivery guarantees to ensure the reliability and availability of message delivery.

Here are the main delivery guarantees provided by SQS:

At least once delivery: This guarantee ensures that each message is delivered at least once, but it may be delivered more than once in certain circumstances, such as when the consumer fails to process the message or when the visibility timeout is extended. This is the default delivery guarantee for SQS queues.

At least once delivery is a delivery guarantee provided by Amazon Simple Queue Service (SQS) that ensures that each message is delivered at least once, but it may be delivered more than once in certain circumstances. This is the default delivery guarantee for SQS queues.

At least once delivery: 🐳

Here is how the "at least once delivery" guarantee works in SQS:

When you send a message to an SQS queue, the message is stored in the queue and made available to consumers.

When a consumer retrieves a message from the queue using the receiveMessage the action of the SQS API or the corresponding method of the AWS SDKs, the message is hidden from other consumers for a specified period known as the visibility timeout.

During the visibility timeout, the consumer processes the message and then sends a request to delete the message from the queue using the deleteMessage the action of the SQS API or the corresponding method of the AWS SDKs.

If the consumer successfully deletes the message from the queue, the message is removed from the queue and is not delivered again.

If the consumer fails to delete the message from the queue, the message becomes visible to other consumers again after the visibility timeout expires, and the process is repeated until the message is successfully deleted from the queue.

This means that a message may be delivered more than once in the following circumstances:

The consumer fails to process the message and does not delete it from the queue before the visibility timeout expires. The consumer extends the visibility timeout of the message using the changeMessageVisibility the action of the SQS API or the corresponding method of the AWS SDKs. To avoid these issues, it is important to design your consumer application to process the message and delete it from the queue promptly, and to handle duplicate messages if they occur.

At most once delivery: 🐳

This guarantee ensures that each message is delivered at most once, but it may not be delivered at all in certain circumstances, such as when the consumer is unable to process the message. To use this delivery guarantee, you can use the SQS FIFO (First-In-First-Out) queue, which is designed to prevent duplicates and guarantee the order of delivery.

A FIFO (First-In, First-Out) queue is a type of Amazon Simple Queue Service (SQS) queue that preserves the order of messages and ensure that each message is processed and deleted exactly once.

At most once delivery is a delivery guarantee provided by Amazon Simple Queue Service (SQS) that ensures that each message is delivered at most once, but it may not be delivered at all in certain circumstances. To use this delivery guarantee, you can use the SQS FIFO (First-In-First-Out) queue, which is designed to prevent duplicates and guarantee the order of delivery.

Here is how the "at most once delivery" guarantee works in an SQS FIFO queue:

When you send a message to an SQS FIFO queue, the message is stored in the queue and made available to consumers.

If the consumer successfully deletes the message from the queue, the message is removed from the queue and is not delivered again.

This means that a message may not be delivered at all in the following circumstances:

Exactly once delivery: 🐳

This guarantee ensures that each message is delivered exactly once, but it requires additional effort to implement and may not be suitable for all use cases. To use this delivery guarantee, you can use the SQS FIFO (First-In-First-Out) queue in combination with a distributed transaction system, such as Amazon DynamoDB or Amazon Aurora, to store the message processing state and coordinate the processing of the message.

Exactly once delivery is a delivery guarantee that ensures that each message is delivered exactly once. This guarantee requires additional effort to implement and may not be suitable for all use cases. To use this delivery guarantee, you can use the SQS FIFO (First-In-First-Out) queue in combination with a distributed transaction system, such as Amazon DynamoDB or Amazon Aurora, to store the message processing state and coordinate the processing of the message.

You can implement "exactly once delivery" using an SQS FIFO queue and Amazon DynamoDB:

When you send a message to an SQS FIFO queue, the message is stored in the queue and made available to consumers.

When a consumer retrieves a message from the queue using the receiveMessage the action of the SQS API or the corresponding method of the AWS SDKs, the consumer stores the message processing state in a DynamoDB table using the putItem action of the DynamoDB API or the corresponding method of the AWS SDKs.

The consumer processes the message and then sends a request to delete the message from the queue using the deleteMessage the action of the SQS API or the corresponding method of the AWS SDKs.

If the deleteMessage request succeeds, the consumer removes the message processing state from the DynamoDB table using the deleteItem action of the DynamoDB API or the corresponding method of the AWS SDKs.

If the deleteMessage the request fails, the consumer retrieves the message processing state from the DynamoDB table using the getItem action of the DynamoDB API or the corresponding method of the AWS SDKs, and checks whether the message has already been processed. If the message has already been processed, the consumer removes the message processing state from the DynamoDB table and does not process the message again. If the message has not been processed, the consumer processes the message and repeats the deleteMessage request until it succeeds.

This example uses DynamoDB to store the message processing state and coordinate the processing of the message, which ensures that the message is delivered exactly once.

Throughput limits: 🪂

SQS has limits on the number of requests per second that can be made to a queue, as well as the maximum number of inflight messages (messages that have been received by a consumer but have not yet been deleted). If you exceed these limits, you can design your consumers to handle the workload more efficiently or consider using a higher-throughput queue type (such as an Amazon SQS FIFO queue).

Workarounds 🐳

Optimize your consumer design: 🪴

One way to avoid exceeding throughput limits is to design your consumers to handle the workload efficiently. This may involve using techniques such as batch processing, multithreading, or asynchronous processing to reduce the number of requests made to the queue.

Use a higher-throughput queue type: SQS offers two types of queues: 🪴

standard and FIFO (First-In, First-Out). Standard queues have higher throughput limits but do not guarantee message order, while FIFO queues have lower throughput limits but guarantee message order. If message order is not critical for your use case, you may be able to increase your throughput by using a standard queue.

Monitor your usage and adjust as needed: 🪴

You can use Amazon CloudWatch to monitor your queue usage and adjust your workload as needed to stay within the limits. For example, if you are consistently approaching the inflight message limit, you may need to increase the number of consumers processing messages from the queue.

Use Auto scaling:

If you are using Amazon Elastic Container Service (ECS) or Amazon EC2 to run your consumers, you can use Auto scaling to adjust the number of consumers based on the workload. This can help you to optimize your usage of SQS and avoid exceeding throughput limits.

Regional availability: 🪂

SQS is available in multiple regions, but a queue can only be accessed from within the same region it was created in. If you need to access a queue from another region, you can set up a cross-region replication configuration or use a service such as AWS Global Accelerator to access the queue.

Workarounds: 🐳

Choose the right region: 🪴

When creating a queue, you should choose the region that is closest to the consumers that will be accessing the queue. This will help to minimize latencies and improve performance.

Use cross-region replication: 🪴

If you need to access a queue from a different region, you can set up a cross-region replication configuration to replicate messages to a secondary queue in the destination region. This allows you to access the queue from multiple regions while maintaining a consistent copy of the messages.

import * as cdk from 'aws-cdk-lib';
import * as sqs from '@aws-cdk/aws-sqs';

export class MyStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create the source queue
    const sourceQueue = new sqs.Queue(this, 'SourceQueue', {
      queueName: 'my-queue',
      redrivePolicy: {
        deadLetterQueue: {
          queueUrl: 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue-replica'
        },
        maxReceiveCount: 10
      }
    });

    // Create the target queue in a different region
    const targetQueue = new sqs.Queue(this, 'TargetQueue', {
      queueName: 'my-queue-replica',
      region: 'ap-south-1'
    });
  }
}

This will set up the source queue to send messages that it can't process to the target queue in the specified region. The deadLetterQueue property of the redrivePolicy specifies the target queue, and the maxReceiveCount the property specifies the maximum number of times a message can be delivered to the source queue before it is sent to the dead-letter queue.

Keep in mind that cross-region replication for SQS is based on the DLQ feature. When a message is sent to the DLQ, it means that the message couldn't be processed by the source queue. You can configure the number of times a message can be delivered to the source queue before it is sent to the DLQ using the maxReceiveCount parameter in the RedrivePolicy.

Use AWS Global Accelerator: 🪴

AWS Global Accelerator allows you to access resources in different regions using static IP addresses. This can be used to access an SQS queue from a different region without the need for cross-region replication.

AWS Global Accelerator is a network service that routes traffic to the optimal Amazon Web Services (AWS) Region for lower latency and higher performance. It uses static anycast IP addresses to route traffic to the optimal AWS Region, and it includes health checking to ensure that traffic is routed to healthy endpoints.

import * as cdk from 'aws-cdk-lib';
import * as globalaccelerator from '@aws-cdk/aws-globalaccelerator';
import * as sqs from '@aws-cdk/aws-sqs';

export class MyStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create a new Global Accelerator
    const accelerator = new globalaccelerator.GlobalAccelerator(this, 'Accelerator', {
      acceleratorType: globalaccelerator.AcceleratorType.STATIC,
      enabled: true,
      regions: [globalaccelerator.Regions.US_EAST_1, globalaccelerator.Regions.US_WEST_2]
    });

    // Create an SQS queue
    const queue = new sqs.Queue(this, 'Queue', {
      visibilityTimeout: cdk.Duration.seconds(300)
    });

    // Create an accelerator endpoint for the queue
    const endpoint = new globalaccelerator.AcceleratorEndpoint(this, 'Endpoint', {
      accelerator,
      port: 80,
      protocol: globalaccelerator.Protocol.TCP,
      resource: queue
    });

    // Create a listener for the endpoint
    new globalaccelerator.Listener(this, 'Listener', {
      accelerator,
      port: 80,
      protocol: globalaccelerator.Protocol.TCP,
      endpointGroups: [{
        endpoints: [endpoint]
      }]
    });
  }
}

Cost: 🪂

Using SQS can incur costs, including charges for the number of requests made, the number of messages sent and received, and the amount of data transferred. You will need to consider these costs and optimize your usage as needed.

SQS is charged based on the number of requests and the volume of data transferred.

Here are the main factors that affect the cost of using SQS:

Number of requests:

SQS charges a request fee for each action performed on a queue, such as sending a message, receiving a message, or deleting a message. The request fees vary depending on the type of queue and the region in which the queue is located. For example, the request fees for an SQS Standard queue in the US East (N. Virginia) region are $0.00000040 per request for sending a message and $0.0000004 per request for receiving and deleting a message.

Data transfer:

SQS charges for data transfer based on the amount of data transferred in and out of the service. The data transfer fees vary depending on the region in which the queue is located. For example, the data transfer fees for an SQS Standard queue in the US East (N. Virginia) region are $0.09 per GB for data transferred in and $0.01 per GB for data transferred out.

Number of messages:

SQS charges a fee for the number of messages stored in a queue, which is based on the volume of data stored in the queue. The number of messages fee varies depending on the type of queue and the region in which the queue is located. For example, the number of messages fee for an SQS Standard queue in the US East (N. Virginia) region is $0.00000040 per 10,000 messages.

Visibility timeout:

SQS charges a fee for the number of seconds that a message is in the queue and not visible to consumers. The visibility timeout fee varies depending on the type of queue and the region in which the queue is located. For example, the visibility timeout fee for an SQS Standard queue in the US East (N. Virginia) region is $0.00000160 per 10,000 seconds.

Optimization on Cost: 🐳

There are several ways you can reduce the cost of using Amazon Simple Queue Service (SQS) by optimizing the usage of the service:

Use the appropriate queue type: 🪴

SQS offers different queue types that have different characteristics and pricing models. For example, the SQS Standard queue provides best-effort delivery at the lowest cost, while the SQS FIFO (First-In-First-Out) queue provides guaranteed delivery and order at a higher cost. Choose the queue type that best fits your requirements and optimize your usage accordingly.

Optimize the number of requests: 🪴

SQS charges a request fee for each action performed on a queue, such as sending a message, receiving a message, or deleting a message. To reduce the cost of requests, you can batch requests together using the sendMessageBatch and deleteMessageBatch actions of the SQS API or the corresponding methods of the AWS SDKs, which allows you to send or delete multiple messages in a single request.

Optimize the data transfer: 🪴

SQS charges for data transfer based on the amount of data transferred in and out of the service. To reduce the cost of data transfer, you can minimize the size of the messages you send and receive, and compress the data if possible. You can also use Amazon CloudWatch to monitor the data transfer volume and identify opportunities to optimize your usage.

Optimize the number of messages: 🪴

SQS charges a fee for the number of messages stored in a queue, which is based on the volume of data stored in the queue. To reduce the cost of storing messages, you can delete messages from the queue as soon as they are processed, and use the SQS Dead Letter Queue feature to store and analyze failed messages.

Optimize the visibility timeout: 🪴

SQS charges a fee for the number of seconds that a message is in the queue and not visible to consumers. To reduce the cost of the visibility timeout, you can set a shorter visibility timeout for messages that can be processed quickly, and a longer visibility timeout for messages that require more time to process. You can also use the changeMessageVisibility the action of the SQS API or the corresponding method of the AWS SDKs to extend the visibility timeout of a message if necessary.

These are some of the main limitations of SQS to consider. It is important to carefully evaluate your specific needs and requirements to determine if SQS is the right solution for your use case.

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

Also free to comment/review if you find something which could be wrong or explained in a better way, I am open to your thoughts and also ready to clarify so that our readers are getting the best content.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

Check out lots more coming direct to your mailbox by joining my free newsletter

🔗 Exploring Serverless @ substack

Exploring Serverless | Aravind Vadamalaimuthu | Substack

Exploring Serverless. Click to read Exploring Serverless, by Aravind Vadamalaimuthu, a Substack publication. Launched 2 years ago.

exploringserverless.substack.com

🏄‍♂️ Stacks-Dash 🤹‍♂️ - A console for monitoring multi account / cross region cloudformation deployments using Amplify Studio

Aravind V — Sat, 01 Oct 2022 19:44:31 +0000

Realtime notifications using aws appsync subscriptions

Delighted to share my full stack project as part of the current amplify hackathon in hashnode, do read the full article and share your thoughts.

Before we go further let me clarify that this tool is more or less an extension to my early project with AWS CDK called event-forwarder which is publicly available.

For more details on that check out the below links to understand from where I sourced the data.

aravindvcyber / event-forwarder

event-forwarder

Event Forwarder 🎢

Basically, I named this to stress the fact that we would be routing and consuming various aws event bridge events (especially the newly available AWS cloud formation events) to various channels which would help developers and teams effortlessly track and monitor the deployments happening in a multi-region and multi-account ecosystem.

Note: I have published the initial blog of this, highly recommended to check that out first before you try to do hands on.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 [dev to @aravindvcyber](so grant the sqs:SendMessage implicitly while the subscription is created.)

I believe a lot of use cases will come soon for now I am starting with the below one.

Starting with Cloudformation events to post Slack notifications effortless from multi-region and even cross accounts to never miss your/peers/ci initiated AWS cloudformation deployments on stacks and resources besides that it could also…

View on GitHub

🔁 Original post at 🔗 Dev Post aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder

🔁 Reposted at 🔗 dev to @aravindvcyber aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder

About this project 🚧

Here I will be trying to build a web-console to monitor the various deployments happening via cloud formation cross-region and in multiple accounts using a single page application built using amplify studio and Figma. The cloud formation events are sourced from my other public project (discussed above in the links). Hope this helps someone as well.

Plan 🏄

In this application, I will be showing how I managed to forward the cloud formation events from multiple accounts across regions into a single web console with some deep links to identify the status of the various deployments in real time.

So here I will extend the event forwarder project by generating dynamodb streams to put data into my custom data model (GraphQL + Dynamodb) defined using a lambda handler.

Hence in simple words, the live data will be generated by dynamodb streams coming out of the event-forwarder master DB, which is temporarily used to send slack notifications.

We will extend this by creating a new stack that will be consuming the data dynamodb streams using a consumer lambda which puts the processed information to dynamodb, later we planned to move this via the graphql API using eventbridge integration.

As you can see that the external stack now emits events which would be used as the primary and the direct dynamodb integration will be obsolete.

Final Architecture 🗽

Here you could find that the external eventbridge events are filtered and buffered with sqs and then connected to stacksDash function which we will be creating.

Further the eventbridge and sqs besides custom gql resolvers will be created and glue using amplify add custom feature which can help us use cdk stacks to provision the additional backends using amplify cli.

Data Modelling in Amplify Studio⛷️

Here I have shown how I created a data model and set authorization rules to configure Cognito user pools as the default mode besides letting the owners access the data in the system for finer control of the data. Besides this, I have enabled the apiKey-based authentication to be used mostly for backend data inserting and processing for finer control

Lambda created using amplify cli 🏄‍♂️

I quickly created a lambda (amplify add function) with the necessary graphql endpoint and key environment variables automatically inserted it also can have the event payload defined to quickly test using the amplify mock function. This lambda can now be used to perform mutations to the data model, and in the future, I have plans to connect the dynamodb streams via this lambda.

Authentication Enabled with Signup 🤸

It is very easy to set up authentication in amplify studio.
I managed to spin up an email verification-based signup and was also able to set up groups to help with my real account and a sandbox profile for test and reviewing profiles with mock data.

Signup is much easier 🤼

I can easily create new users from the studio as well as from the app, which will automatically create a Cognito user in the Cognito user pool with the required information I required. For the sandbox account, I managed to directly activate it from the studio.

Also, the OIDC mode is very much promising, and I hope I can quickly spin up that soon in the future to use social logins.

Since as you may know this project is only a demonstration and cannot be directly used by public users like the blog or other content-sharing sites, I stick to email-based authentication for now.

Integration with social providers for authentication 🤺

Here I have used google oauth to sign and access the application, so that I could use the app seamlessly in any device I have already signed in the browser securely and also stop remembering multiple accounts and credentials.

Prototyping the UI design with Figma 🏌️

Now it is time to do some real design and prototyping in Figma, I managed to learn this last 2 weeks and I could create rich UI components as shown below. It is very much helpful as I could visually design and get the pixel-perfect components in my React front end through amplify studio.

Figma made this project work scale and be consistent, and I hope more such integration will be there in amplify studio soon.

Studio UI Configuration 🏋️

From amplify studio, I managed to sync my Figma and commit the new changes to my components. Besides this cool features like creating the collection, configuring the properties, creating a slot for UI elements, and binding the data are much useful to get this into a useable mode when I pull this using amplify pull

UI workflow 🤺

Okay, let us see what we have got this web console.

Sandbox Home Page with Mocked data 💪

Using the credentials, for the sandbox user, I have created. I was able to log in to the home page easily by skipping the verification step which I confirmed in amplify studio itself for this user.

Real Account data demo 🧗‍♂️

Okay now it is time to walk you through my real account, and for the data generation it is automatically taken care of as I build and deploy amplify again and again(Since it uses cloud formation to provision resources), it generates the data for me and now I can access them from a single dashboard and as shown in this clip below.

Search and Filters 🏊‍♂️

Here we have a sidebar that acts like a master filter that will refine the collection as we select them one after the other. It was a bit tricky since I have to prepare the icon and selectable into groups of frames and then dynamically link them to the state using in react. And finally, I managed to filter them as per my selection.

More the collection component bootstrapped from the amplify studio is also amazing with a search bar and pagination controls.

Deeplinks to aws console 🏋️‍♀️

This project could have more features as I continuously refine it while I explore new features and implement them in the future.

Here specifically I was looking for buttons that could deep link me to the AWS console to the specific resource and it worked as expected as shown below. Hope it is helpful for someone. The rest is more of experiments to try out amplify studio and cli.

Deep link to all CFN events 🏋️‍♀️

Selecting any stack level event will also show a full list of all the resource level events associated with the full lifecycle of the CFN stacks as shown in the below video. This is achieved by creating a custom graphql resolver which could access the existing dynamodb table (this is the master dynamodb table in the event forwarder project) beyond the one which is directly created by this tool.

Using custom resource in amplify it is very easily to leverage and extend the existing resources to work seamless across your application.

Appsync makes it easier with GraphQL 🚣‍♀️

I managed to test my graphql endpoint in my studio of choice in apollo sandbox as well as used Cognito user ids to test the user level access in the aws appsync console easily throughout the project work.

GraphQL Transformer v2

Also the new graphql transfer v2 for amplify is really a simple and cool way to design and model the schema, capabilities regards to indexs and auth are tried part of this solution.

Also I also created a custom VTL based resolver using CDK which help me with easier hack whenever I need a new resolver for my specific query needs.

Using amplify add custom you can create a cloudformation or cdk resource. I created a custom resolver using cdk as shown below with VTL based template.

const retVal: AmplifyDependentResourcesAttributes =
      AmplifyHelpers.addResourceDependency(
        this,
        amplifyResourceProps.category,
        amplifyResourceProps.resourceName,
        [
          {
            category: "api",
            resourceName: "StacksDash",
          },
        ]
      );

    const resolver = new appsync.CfnResolver(
      this,
      "custom-resolver-custQuery",
      {

        apiId: cdk.Fn.ref(retVal.api.StacksDash.GraphQLAPIIdOutput),
        fieldName: "custQuery",
        typeName: "Query", // Query | Mutation | Subscription
        requestMappingTemplate: appsync.MappingTemplate.fromFile(
          path.join(__dirname, "..", "Query.custQuery.req.vtl")
        ).renderTemplate(),
        responseMappingTemplate: appsync.MappingTemplate.fromFile(
          path.join(__dirname, "..", "Query.custQuery.res.vtl")
        ).renderTemplate(),
        dataSourceName: "DeploymentsTable", // DataSource name
      }
    );

    resolver.applyRemovalPolicy(RemovalPolicy.DESTROY);

Sample Simple VTL template for request and response below

{
    "version": "2017-02-28",
    "operation": "GetItem",
    "key": {
        "id": $util.dynamodb.toDynamoDBJson($ctx.args.id),
    }
}


#set($ctx.result = $util.parseJson($ctx.result))

$util.toJson($ctx.result)

Appsync subscriptions for live sync 🚀

Besides that amplify api also comes with auto-generated mutations and subscriptions which are very much helpful to update the data items in the react dom as and when the cfn deployments events are emitted with any necessary page reload or refresh.

Notification banners built with figma components with variants 🚣‍♀️

The above appsync notifications also powers these handy banners displayed to identify the distinct updates without refreshing the screen in the user browser with amplify studio it is seamless to bring these updates then and there

AWS Amplify Hosting 🚣‍♀️

Finally hosting the frontend is the missing piece and glad to full fill this with CI/CD approach using amplify hosting.

Also this gave really cool features to lock and secure my website with authentication for endpoint and synthetic canary to monitor the endpoint.

Branch level and global access control for hosting

This would definitely help me secure my app as I release it to test and for my own use privately as well.

Synthetic Canaries for continuous monitoring

Previews for pull requests to main branch

Conclusion 🚵‍♂️

Due to time constraints, and steep learning sessions, I have to complete regards to Figma and amplify cli and studio, in a matter of few weekends to get my first amplify project. Hope you like it and I am open to reviews and suggestions. And I also hope I could explore new features and implement them to make this more useful in the future.

And I am still cleaning up the sandbox account, once it is ready I will be to share it for someone to inspect and check it in action. Or feel free to comment or connect with me to help you with the necessary access for the limited preview before I publish the working model.

For more details on that check out the below links to understand from where I sourced the data for cloud formation events.

aravindvcyber / event-forwarder

event-forwarder

Event Forwarder 🎢

Note: I have published the initial blog of this, highly recommended to check that out first before you try to do hands on.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 [dev to @aravindvcyber](so grant the sqs:SendMessage implicitly while the subscription is created.)

I believe a lot of use cases will come soon for now I am starting with the below one.

Starting with Cloudformation events to post Slack notifications effortless from multi-region and even cross accounts to never miss your/peers/ci initiated AWS cloudformation deployments on stacks and resources besides that it could also…

View on GitHub

🔁 Original project at 🔗 Dev Post aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder

🔁 Reposted project at 🔗 dev to @aravindvcyber aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder

Nonetheless, it is a great opportunity to learn and build and demo to everyone what I built using amplify studio.

🎉 Thanks for supporting! 🙏 and do follow and share this series for more such articles.

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post aws-cdk-101-projects-stacks-dash-a-console-for-monitoring-multi-account-cross-region-cloudformation-deployments-using-amplify-studio

🔁 Reposted at 🔗 dev to @aravindvcyber aws-builders/stacks-dash-a-console-for-monitoring-multi-account-cross-region-cloudformation-deployments-using-amplify-studio-1ik8

💥 AWS CDK 101 - 💫 Dynamodb Local setup and integrating with sam invoke

Aravind V — Sun, 18 Sep 2022 17:44:37 +0000

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 Original post at 🔗 Dev Post aws-cdk-101-sam-local-to-test-and-debug-lambda-function

🔁 Reposted at 🔗 dev to @aravindvcyber aws-cdk-101-sam-local-to-test-and-debug-lambda-function-1afj

Also, we have started to develop an open source project which we would be using to play around with refracting the architecture as well as learn CDK stuff at the same time we will provide something useful for our community. Find more about this discussed in the article below.

🔁 Original project post at 🔗 Dev Post aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder

🔁 Reposted project post at 🔗 dev to aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder-1m0m

event-forwarder Github repo

Dynamodb local with sam invoke 🍡

Earlier in our article, we have seen how to set up local integration testing of the lambda using sam invoke. Dynamodb local will not only be a cool integration setup to inspect in locally, but also it can help to understand the dynamodb data model using NoSQL workbench.

So why do we need now ❔

In our previous article, we mentioned that we have sam local to test the function locally. But we don't have the local dynamodb. We set up that brings that into the same docker network in this article to fix the below error.

Thank you fully the error slack hook post the exception clearly to my slack

Docker composes dynamodb local 🌟

Let us use docker compose to spin up dynamodb local

Dynamodb local container 🤡

I prefer to use the docker mode of setting up dynamodb local since it was very much predictable and can also use docker compose yml file alongside.

Here in this snippet of yaml file below, you can find a simple dynamodb docker container bound to cloud network and local volume ddb-data which has been already pre-created.

version: "3.5"

services:
  dynamo:
    container_name: local-ddb
    image: amazon/dynamodb-local
    networks:
      - cloud
    ports:
      - "8000:8000"
    volumes:
      - ddb-data:/home/dynamodblocal
    working_dir: /home/dynamodblocal
    command: "-jar DynamoDBLocal.jar -sharedDb -dbPath ."

networks:
  cloud:
    external: true

volumes:
  ddb-data:
    external: true

Create the docker network 💦

Here let us define a docker network to be shared by our local cloud containers.

docker network create -d bridge cloud

Create the docker volume 🐟

Here let us create a new docker volume and mount it to our local folder

docker volume create \
        --driver local \
        --opt type=none \
        --opt o=bind \
        --opt device=*******Your Local Volume folder*********** \
        ddb-data

Start and Stop dynamodb local 🌱

Navigate to the directory you have defined the docker compose file. Open the terminal and run the below command to start it.

docker-compose up -d

And to stop the container run the below command

docker-compose down

Validate the ddb connection local 🌴

Run the below command in the local terminal to make sure the port is correct as per your docker-compose you have defined above.

aws dynamodb list-tables --endpoint-url http://localhost:8000

You may get some results like the below.

{
  "TableNames": [

  ]
}

Create local table 🌾

Before that if the table already exists you can delete it as shown below.

aws dynamodb delete-table --table-name eventStores9 --endpoint-url http://localhost:8000

Net us create the table by copying the schema from the cloud table by running the below describe-table command

aws dynamodb describe-table --table-name eventStores9 --profile av > eventStores9.json

Note in the above step, I have not used the endpoint but rather used my AWS named profile av to connect. In your case, you can have your own and write the output to the file eventStores9.json

Now run the below command using the file created above (note it is expected to throw error)

aws dynamodb create-table --cli-input-json file://eventStores9.json --endpoint-url http://localhost:8000

When you get the above error, you simply have to remove those nodes from the JSON file you are using and the cleanup version of the file is already checkin to the local of this project to help you identify.

And then you can run the same command again, if it does not throw any error it should create the table and will give the description of the table as output.

aws dynamodb create-table --cli-input-json file://eventStores9.json --endpoint-url http://localhost:8000

Validate by listing the tables locally 🌸

aws dynamodb list-tables --endpoint-url http://localhost:8000

You may get some results like the below.

{
  "TableNames": [
    "eventStores9"
  ]
}

And there you have it your local dynamodb table is ready to help you.

NoSQL Workbench 🐲

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.settingup.html

You may also try this tool which can help you model dynamodb much faster and even push it to the cloud or local connection. However, this tool does not support LSI in the data modeling stage. Hope it is available soon. If it supports LSI, we would have directly imported the cloud formation template without parameters to create the schema in the tool itself. I tried it but is ignoring the LSI. So that is why I used the AWS CLI to generate the table.

Still, this tool can help us with the Operation Builder feature which will connect to the dynamodb tables directly and help with a range of other features and even help with some code snippets as well to design.

Create a new database connection 🐐

Inspect the metadata and LSI and GSI definitions

With this interface, you can directly query, inspect and manipulate the local table data during testing and debugging.

Further, you can also use LSI and GSI in your operations seamlessly.

Code changes to the function 🍃

Code changes to the function to enable local ddb connection. Just like how we identified AWS_SAM_LOCAL, we can do the same here as well in the dynamodb-util.ts file.

Take note of the name of the host it is the container name we defined in docker-compose yaml

if(process.env.AWS_SAM_LOCAL) {
  options.endpoint = 'http://local-ddb:8000'
}

const dynamo = AWSXRay.captureAWSClient(new DynamoDB(options));

Run script changes 🌲

Here we will readily know that sam CLI scripts are part of the run scripts as we demonstrated in the previous article. Here we need to include the docker network while using sam local invoke to be able to connect to used other

{
  "sam:i": "sam local invoke $npm_config_fname --docker-network cloud -e $npm_config_event | jq .",
    "sam:i:debug": "sam local invoke $npm_config_fname --docker-network cloud -d 9999 -e $npm_config_event  --log-file logs/sam-debug-logs.txt --debug | jq .",
}

This should help to connect the function with local dynamodb as expected.

Post testing events results in dynamodb 🌰

I have run all the test events I have prepared in the last article in events and I can see the results in the local table.

npm run t:1:1 -- --event=events/event_stack_create_complete.json
npm run t:1:1 -- --event=events/event_stack_delete_complete.json
npm run t:1:1 -- --event=events/event_stack_update_complete.json
npm run t:1:1 -- --event=events/event_resouce_create_complete.json
npm run t:1:1 -- --event=events/event_drift_detection_complete.json

Scan dynamodb items

Normal Query

Query using LSI_STATUS

Debugging the data in vscode 🌿

npm run d:1:func1 -- --event=events/event_stack_create_complete.json

Verify that it is passing through the sam local script section when it is true and it resets the dynamodb endpoint.

With this approach, we will be able to monitor data moving in and out of the dynamodb table in local

If you don't like to use js debugging when using typescript with webpack you can add the below to your launch config attach configuration sourceMapPathOverrides as discussed in the previous article.

{
"sourceMapPathOverrides": {
    "meteor://💻ap{workspaceFolder}/*",
    "webpack:///./{workspaceFolder}/node_*",
    "webpack://?:{workspaceFolder}/*"
}

Then directly you can place break points in the typescript files and you can cleanly debug them as follows.

Sample typescript debugging screenshots

Sample typescript debugging screenshots with a specific point of interest

Conclusion ⛲

This will be extremely useful when you repeatedly iterate and make code changes to the processor and you could use the test event to test it swiftly using local dynamodb setup and integration.

We will be talking about more similar engineering concepts as we refactor and refine the event forwarder project. Keep following for similar posts on engineering with IaC primarily using AWS CDK and Serverless.

Also, feel free to contribute to the progress of the below solution with your comments, and issues, maybe you can also do a pr if you feel it can help our community.

event-forwarder Github repo

🔁 Original project post at 🔗 Dev Post aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder

🔁 Reposted project post at 🔗 dev to aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder-1m0m

⏭ We have our next article in serverless and IaC, do check out

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post aws-cdk-101-dynamodb-local-setup-and-integrating-with-sam-invoke

🔁 Reposted at 🔗 dev to @aravindvcyber aws-cdk-101-dynamodb-local-setup-and-integrating-with-sam-invoke-527f

🐋 AWS CDK 101 - 🦋 SAM local to test and debug lambda function

Aravind V — Sun, 18 Sep 2022 08:57:24 +0000

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 🔗 Dev Post aws-cdk-101-send-message-across-accounts-using-sns-topic-and-sqs

🔁 🔗 dev to aws-cdk-101-send-message-across-accounts-using-sns-topic-and-sqs-h2i

🔁 Original project post at 🔗 Dev Post aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder

🔁 Reposted project post at 🔗 dev to aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder-1m0m

event-forwarder Github repo

Integration testing lambda 🍡

Earlier in our article, we have seen how to use Eventbridge and SNS to forward over messages to remote stacks as well. Now let us make one more addition to our stack by setting up local integration testing of the lambda using sam local.

We need to get this since after performing numerous iterations with cdk deploy and validating the cloud watch logs, i wanted to find something which i could leverage to be capable of find the bugs prior to release.

Sam local 🤡

Using Sam local will be able to set up sam local docker to test our lambda processor, though someone of you may not find it useful. But you may be making changes to the code in repeated iterations. Now and then you may need to hot-swap or force deploy the changes and based on the assets it will unnecessarily lead to the below scenarios if you are not interested with

you could also check in your test events rather than using in lambda console.
you can try to run the lambda in a similar environment (like arm64) and runtime (different node versions) even if you don't have that locally
Inspect the environment information and debug log traces for lambda container
Monitor the system usage and audit the memory composition of the layers used
This can also help you use pull remote layers and bind with local layers to simulate it.
temporarily your network could be offline and you could not test.
increase asset file/zip uploads
KMS keys used for cross-account deploy buckets
need to commit for the pipeline to deploy event for minor changes
more importantly you may need to wait for the deployment/pipeline to complete

To start with we will be first discussing how to set up sam local.

SAM CLI Installation 🤖

For this, we need to install sam cli

Installing the AWS SAM CLI on Linux
Installing the AWS SAM CLI on Windows
Installing the AWS SAM CLI on macOS

I prefer the homebrew option if you have that installed already.

brew tap aws/tap
brew install aws-sam-cli
sam --version

Json output format 👀

For better formatting, I use to have | jq . towards the end of certain npm run scripts. If you need this u need to install check out the below link
jq download

Again I do it with homebrew

brew install jq

Add the below launch configuration 🦴

In vscode, you can also add the below launch config to the launch configurations in the file .vscode/launch.json. This will enable the js map link to the source as well so that we could follow the code in vscode with js map errors in debug console.

 {
    "name": "Attach to SAM CLI",
    "type": "node",
    "request": "attach",
    "address": "localhost",
    "port": 9999,
    "localRoot": "${workspaceRoot}/dist/lambda/stack-event-processor",
    "remoteRoot": "/var/task",
    "protocol": "inspector",
    "stopOnEntry": false
}

If you don't like to use js debugging when using typescript with webpack you can add the below to your launch config attach configuration sourceMapPathOverrides as discussed above.

{
"sourceMapPathOverrides": {
    "meteor://💻ap{workspaceFolder}/*",
    "webpack:///./{workspaceFolder}/node_*",
    "webpack://?:{workspaceFolder}/*"
}

Define the stack template in the package config 🐔

Here I have defined a random stack template name for the remote stack npm_package_config_remoteTemplate since it has generated stack id ./cdk.out/RemoteEventRouterStack-ap-south-1-456.template.json.
You can update it to the correct one as per your synth.

  "config": {
    "stackName": "EventForwarderStack",
    "cfnTemplateProcessor": "./cdk.out/EventForwarderStack.template.json",
    "remoteTemplate": "./cdk.out/RemoteEventRouterStack-ap-south-1-456.template.json"
  },

Sample Events 🐥

I have placed sample events in the events folder to run the function. This can grow based on needs. Here a dummy cloudformation event is stringified inside the body of the dummy sqs event payload.

events

Detecting sam local setup locally 🦉

You can add the below code into the handler to choice the right flow and use sam local seamlessly.

const handle = async function (event: SQSEvent) {

  if(process.env.AWS_SAM_LOCAL) {
    logger.info("running with sam local:", { event });
  }

Note since we are using webpack to bundle to the dist folder we will using this our local and we could not link to typescript, so we have place a similar break point in the js file to linkable by the debugger

Else add add the sourceMapPathOverrides property to the launch config as discussed above for webpack.

How to run 🦚

As already advised the config and the AWS named profile and package config has to be updated as per your preference and then you can run the below scripts.

Build script with sam template 🐢

{
    "sam:b:1": "sam build --docker-network cloud -t $npm_package_config_cfnTemplateProcessor --profile av --region ap-south-1 ",
    "sam:b:2": "sam build --docker-network cloud -t $npm_package_config_remoteTemplate --profile av-s --region ap-south-1 ",
}

Run scripts for testing 🐜

Note for testing you have to provide the event payload in the below format as follows.

Directly invoking the function when the sam template is already validated

npm run t:1:func1 -- --event=events/event_stack_delete_complete.json

or always build with sam before testing

npm run t:1:1 -- --event=events/event_stack_delete_complete.json

{
    "sam:i": "sam local invoke $npm_config_fname --docker-network cloud -e $npm_config_event | jq .",
    "t:1:func1": "npm run sam:i --fname=stackEventProcessor",
    "t:1:func2": "npm run sam:i --fname=failedMessageLogger",
    "t:2:func1": "npm run sam:i --fname=failedMessageAggregator",
    "t:1:1": "npm run sam:b:1 && npm run t:1:func1",
    "t:1:2": "npm run sam:b:1 && npm run t:1:func2",
    "t:2:1": "npm run sam:b:2 && npm run t:2:func1"
}

run scripts for debugging 🌸

Note for debugging you have to provide the event payload in the below format as follows.

--log-file logs/sam-debug-logs.txt is used to move the cloudwatch logs to my current folder you may remove that if you prefer. Also when the log file is enabled you won't get the web socket link for debugging and you only get the below notification. Once you find it you can switch to vscode to attach to SAM CLI process using our config defined above.

--debug option is used to capture more verbose environment-related insights and trace beyond your function logic, feel free to remove it if you don't need them

In these scripts, you may additionally find that we are synthesizing and building before debugging to make sure the latest code changes are into sam local.

npm run d:1:func1 -- --event=events/event_stack_delete_complete.json

{

    "sam:i:debug": "sam local invoke $npm_config_fname --docker-network cloud -d 9999 -e $npm_config_event  --log-file logs/sam-debug-logs.txt --debug | jq .",
    "d:1:func1": "npm run cdk:synth:all && npm run sam:b:1 && npm run sam:i:debug --fname=stackEventProcessor",
    "d:1:func2": "npm run cdk:synth:all && npm run sam:b:1 && npm run sam:i:debug --fname=failedMessageLogger",
    "d:2:func2": "npm run cdk:synth:all && npm run sam:b:2 && npm run sam:i:debug --fname=failedMessageAggregator",

}

Debugging in vscode 🌹

As discussed earlier we made a break point in the js file under the dist folder to catch via the debugger.

Inspect Docker container while debugging 💐

Understand system processes and usage

memory requirements of layers

lambda container environment variables

Conclusion ⛲

This will be extremely useful when you repeatedly iterate and make code changes to the processor and you could use the test event to test it swiftly. Watch out for our next article on local dynamodb setup and integration.

Also, feel free to contribute to the progress of the below solution with your comments, and issues, maybe you can also do a pr if you feel it can help our community.

event-forwarder Github repo

🔁 Original project post at 🔗 Dev Post aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder

🔁 Reposted project post at 🔗 dev to aws-cdk-101-projects-cdk-stackresourcedrift-events-forwarded-to-cool-slack-posts-event-forwarder-1m0m

⏭ We have our next article in serverless and IaC, do check out

https://dev.to/aws-builders/aws-cdk-101-dynamodb-local-setup-and-integrating-with-sam-invoke-527f

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post aws-cdk-101-sam-local-to-test-and-debug-lambda-function

🔁 Reposted at 🔗 dev to @aravindvcyber aws-cdk-101-sam-local-to-test-and-debug-lambda-function-1afj

🐬 AWS CDK 101 -🐠 Send message across accounts using SNS topic and SQS

Aravind V — Sat, 10 Sep 2022 06:43:54 +0000

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 Original previous post at 🔗 Dev Post

🔁 Reposted the previous post at 🔗 dev to @aravindvcyber

🔁 Original project post at 🔗 Dev Post

🔁 Reposted project post at 🔗 dev to @aravindvcyber

event-forwarder Github repo

Cross Account sendMessage 🍡

Earlier in our article, we have seen how to use custom Eventbridge and SQS by configuring an event rule and target which shifts the messages to the sqs queue and extended the same to remote stacks as well. Now let us make one more addition to our stack by retrieving the dlq messages from the remote stack across regions to our processor region.

Original post at 🔗 Dev Post

Reposted at 🔗 dev to @aravindvcyber

To start with we will be first discussing how to start polling the messages from the dlq using a lambda processor.

Before that let us set up a lambda layer that will have our external dependencies necessary for logging and monitoring.



export const generateLayerVersion = (
  scope: Construct,
  layerName: string,
  props: Partial<LayerVersion>
): LayerVersion => {
  return new LayerVersion(scope, layerName, {
    ...defaultLayerProps,
    code: Code.fromAsset(join(__dirname, "..", "layers", layerName)),
    ...props,
  });
};

const powertoolsSDK = generateLayerVersion(this, "powertoolsSDK", {});

exportOutput(this, "powertoolsSDKArn", powertoolsSDK.layerVersionArn);

Lambda processor definition 🪴

Here you can find the definition of the lambda function which will be used to poll messages from dlq and push to SNS topic.



 const failedMessageAggregator = new Function(
      this,
      "failedMessageAggregator",
      {
        code: Code.fromAsset("dist/lambda/failed-message-aggregator"),
        handler: "failed-message-aggregator.handler",
        ...commonLambdaProps,
        functionName: "failedMessageAggregator",
        layers: [powertoolsSDK],
        environment: {
          TOPIC_ARN: remoteStackEventTargetDlqSns.topicArn,
          TZ: config.get("timeZone"),
          LOCALE: config.get("locale"),
        },
      }
    );

failedMessageAggregator.applyRemovalPolicy(RemovalPolicy.DESTROY);

Lambda handler code 🌷

The full and latest code should be found in the git hub repo below.

failed-message-aggregator.ts



class Lambda implements LambdaInterface {

  @tracer.captureMethod()
  private async processSQSRecord (rec: SQSRecord)  {
    logger.info("Fetching DLQ message:", {rec});
    const params: PublishInput = {
      Message: rec.body,
      Subject: "Forwarding event message to SNS topic",
      TopicArn: process.env.TOPIC_ARN,
    };
    const snsResult: PublishResponse = await sns.publish(params).promise();
    logger.info("Success", { params, snsResult });
  }

  public async handler(event: SQSEvent) {
    try {
      await Promise.all(
        event.Records.map(async (rec: SQSRecord) => {
          await this.processSQSRecord(rec);
        })
      );
      return {
        statusCode: 200,
        headers: { "Content-Type": "text/json" },
        body: {
          EventsReceived: [...event.Records].length,
        },
      };
    } catch (error) {
      logger.error("Error", { error });
      return {
        statusCode: 400,
        headers: { "Content-Type": "text/json" },
        body: {
          EventsReceived: [...event.Records].length,
          Error: error
        },
      };
    }
  };

}

Event Source mapping DLQ to lambda 🌳

Here we will map the remote dlq to trigger the lambda which we have built above.



failedMessageAggregator.addEventSource(
      new SqsEventSource(remoteStackEventTargetDlq.queue, {
        batchSize: 10,
        maxBatchingWindow: Duration.seconds(20),
      })
);

SNS topic to push to subscribers 🦚

This topic will be used to receive messages from the lambda and push into relevant subscriber channels. Here we will subscribe this to common dlq in the processor stack.



const remoteStackEventTargetDlqSns = new Topic(
      this,
      "remoteStackEventTargetDlqSns",
      {
        displayName: "remoteStackEventTargetDlqSns",
        topicName: "remoteStackEventTargetDlqSns",
      }
);

remoteStackEventTargetDlqSns.applyRemovalPolicy(RemovalPolicy.DESTROY);

exportOutput(
      this,
      "remoteStackEventTargetDlqSnsArn",
      remoteStackEventTargetDlqSns.topicArn
);

Granting access to lambda to Send Message 🍋

Now will be grant access to the lambda function to send messages as the producer.




remoteStackEventTargetDlqSns.grantPublish(failedMessageAggregator);

Two-way handshake to link SNS to SQS 🥬

With regards to sns and sqs in different account it is essential to set up the two-way handshake for this there have to be two actions allowed one at each end.

sns:Subscribe in remote topic
sqs:SendMessage in consumer queue (subscriber)

Remote stack configurations

Granting access to processor account to subscribe

Here we will be granting access to processor account resources to subscribe to this topic as follows.



remoteStackEventTargetDlqSns.addToResourcePolicy(
      new PolicyStatement({
        sid: "Cross Account Access to subscribe",
        effect: Effect.ALLOW,
        principals: [new AccountPrincipal(targetAccount)],
        actions: ["sns:Subscribe"],
        resources: [remoteStackEventTargetDlqSns.topicArn],
      })
);

Processor stack configurations 🏝️




remoteAccounts.map((account: string) => {
      remoteRegions.map((region: string) => {

        // Here we will be adding the reference and the subscription
    });
});

Referencing to the remote topic

In the processor stack, we will be getting the reference to the relevant topics as follows.



const remoteStackEventTargetDlqSns = Topic.fromTopicArn(
    this,
    `remoteStackEventTargetDlqSns-${region}-${account}`,
    `arn:aws:sns:${region}:${account}:remoteStackEventTargetDlqSns`
);

Subscribing to the remote topic

Here we will be subscribing to the processor region dlq to receive the messages from the remote region SNS topic as follows.

Note it is highly recommended to subscribe from the consumer stack so that the subscription gets auto-confirmed, else there will be another confirmation step you may need to do from the console or confirmation message to do that yourself.



const subProps: SqsSubscriptionProps = {
          rawMessageDelivery: true,
};

remoteStackEventTargetDlqSns.addSubscription(
    new aws_sns_subscriptions.SqsSubscription(
      stackEventTargetDlq.queue,
      subProps
    )
);

The above subscription setup from the processor stack also grants the sqs:SendMessage implicitly while the subscription is created.

Conclusion ⛲

With this approach just like how we pooled the remote cfn events to a common event bridge across regions and accounts, we are also able to get the remote dlq events to a common dlq. These messages in dlq can be inspected without switching to another region or account, which the maintainer doesn't have any access.

This will be extremely useful when you build similar event-driven solutions.

Also, feel free to contribute to the progress of the below solution with your comments, and issues, maybe you can also do a pr if you feel it can help our community.

event-forwarder Github repo

🔁 Original project post at 🔗 Dev Post

🔁 Reposted project post at 🔗 dev to @aravindvcyber

⏭ We have our next article in serverless and IaC, do check out

https://dev.to/aws-builders/aws-cdk-101-sam-local-to-test-and-debug-lambda-function-1afj

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

🍉 AWS CDK 101 -🥒 Cross region putEvents across accounts using Eventbridge for Event-Forwarder project

Aravind V — Sun, 04 Sep 2022 19:41:43 +0000

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 Original previous post at 🔗 Dev Post

🔁 Reposted the previous post at 🔗 dev to @aravindvcyber

Also, we have started to develop an open source project which we would be using to play around with refracting the architecture as well as learn cdk stuff at the same time we will provide something useful for our community. Find more about this discussed in the article below.

🔁 Original project post at 🔗 Dev Post

🔁 Reposted project post at 🔗 dev to @aravindvcyber

event-forwarder Github repo

Cross Account PutEvents 🍡

Earlier in our article, we have seen how to use custom Eventbridge and SQS by configuring an event rule and target which shifts the messages to sqs queue.

Original post at 🔗 Dev Post

Reposted at 🔗 dev to @aravindvcyber

But in this article, we will be using the default event bus only since we are trying to source the cloud formation events for the above-mentioned project. To start with we will be first discussing how to make the event bridge events be routed to an sqs queue in our main processor stack.

First, let us get the reference to the default target bus in the processor region

export const getDefaultBus = (
  scope: Construct,
  region: string,
  account: string
): IEventBus => {
  return EventBus.fromEventBusArn(
    scope,
    `DefaultBus-${region}-${account}`,
    `arn:aws:events:${region}:${account}:event-bus/default`
  );
};

Then we will create a target queue, a processor queue for lambda trigger, and processor dlq. Since we have to use the queue and dlq construct multiple times, let us refract them to some utils below.

export const dlqQueueProps = {
  retentionPeriod: Duration.days(7),
  removalPolicy: RemovalPolicy.DESTROY,
};

export const normalQueueProps = {
  retentionPeriod: Duration.days(1),
  removalPolicy: RemovalPolicy.DESTROY,
  deliveryDelay: Duration.seconds(1),
  visibilityTimeout: Duration.minutes(3),
};

export const generateDLQ = (scope: Construct, queueName: string): Queue => {
  return new Queue(scope, queueName, {
    ...dlqQueueProps,
    queueName,
  });
};

export const generateQueue = (
  scope: Construct,
  queueName: string,
  deadLetterQueue: DeadLetterQueue
): Queue => {
  return new Queue(scope, queueName, {
    ...normalQueueProps,
    queueName,
    deadLetterQueue,
  });
};

Let provision these necessary queues now

const stackEventProcessorQueue = generateQueue(
      this,
      "stackEventProcessorQueue",
      stackEventProcessorQueueDLQ
);

const stackEventTargetDlq: DeadLetterQueue = {
      queue: generateDLQ(this, "stackEventTargetDlq"),
      maxReceiveCount: 100,
};

const stackEventProcessorQueueDLQ: DeadLetterQueue = {
      queue: generateDLQ(this, "stackEventProcessorQueueDLQ"),
      maxReceiveCount: 100,
};

Event Rule 🍻

Let us now source these cloud formation events from the current region

const stackEventsRule = new Rule(this, "stack-events-rule", {
      eventPattern: {
        detail: {
          "stack-id": [{ exists: true }],
        },
        source: ["aws.cloudformation"],
      },
      eventBus,
});

And please remember this event pattern can be very extended to filter the relevant stack events only or source from all stacks like me and filter in the upcoming steps.

Event Rule target ☕

Now let us use the target dlq and processor queue to build our event rule target to wire up to the event rule which is defined above.

const stackEventTarget = new aws_events_targets.SqsQueue(
      stackEventProcessorQueue,
      {
        retryAttempts: 3,
        deadLetterQueue: stackEventTargetDlq.queue,
      }
);

stackEventsRule.addTarget(stackEventTarget);

From then on the processor queue will take over the events as per the projects we discussed above and we are not interested in that in this article.

Multi-region or multi-account cloud formation events processing 🧃

What we have just seen is that we can source the events from the current region. But what if we may need to pull the events from another region or even another account? That is when we use the other stack part of the project which we will put into every account-region pair and source the cloud formation events necessary for our project.

TargetBus cross-account policy 🥢

const remoteAccounts = config.get("remoteAccounts").split(",");
    const remoteRegions = config.get("remoteRegions").split(",");

    remoteAccounts.map((account: string) => {
      remoteRegions.map((region: string) => {

        new CfnEventBusPolicy(this, `CrossAccountPolicy-${region}-${account}`, {
          action: "events:PutEvents",
          eventBusName: eventBus.eventBusName,
          principal: account,
          statementId: `Accept-PutEvents-From-${region}-${account}`,
          // condition: {

          // }
        });
      });
    });

Here I have used an array of remote accounts or remote regions to be used allowed to forward events to the processor stack besides that you can be a little more restrictive by adding some conditions as per your requirement for security regions or to reduce unnecessary or unwanted resources accessing our target default bus.

Single region used for processing and storage 🏖️

And why we do why, this is because of multiple reasons.

Need not deploy the complex processing setup in every region or accounts
Also this also means we are also not starving our environment much our solution scales up reaching the limits of the respective services
Easily to inspect the processing stack in a particular region with xray, cloudwatch, and we could have a single dashboard with cloudwatch metrics and it will be easier for maintainable
Lightweight remote stacks are portable and we could very easily provision them in multiple regions or accounts with less footprint.

Remote stack configuration 🏜️

Remote Eventbus Rule

With the similar approach used earlier, we could extend this further in this remote stack by building an event bus rule and event bus target with a failed event queue

const targetAccount = config.get("account");
const targetRegion = config.get("region")
const targetBus = getDefaultBus(this, targetRegion, targetAccount);
const remoteBus = getDefaultBus(this, props.region, props.account)
const remoteStackEventsRule = new Rule(this, "remote-stack-events-rule", {
  eventPattern: {
    detail: {
      "stack-id": [{ exists: true }],
    },
    source: ["aws.cloudformation"],
  },
  eventBus: remoteBus,
});

Remote Eventbus Target

Adding a dlq to the target like the processor stack once again but now we use an eventbus as the target which is the processor region default bus that we have referenced. This should be able to connect and push messages even across regions into even different accounts. Since we have already granted this using the policy above. Also, the event bus fails it puts into a regional dlq.

const remoteStackEventTargetDlq: DeadLetterQueue = {
  queue: generateDLQ(this, "remoteStackEventTargetDlq"),
  maxReceiveCount: 100,
}
const remoteStackEventTarget = new aws_events_targets.EventBus(targetBus, {
  deadLetterQueue: remoteStackEventTargetDlq.queue,
})
remoteStackEventsRule.addTarget(remoteStackEventTarget);

Thus we can demonstrate how we would be able to pool cloud formation events across the region and even from different accounts. This will be extremely useful when you build similar event-driven solutions.

Also feel free to contribute to the progress of the below solution with your comments, issues, maybe you can also do a pr if you feel if can help our community.

event-forwarder Github repo

🔁 Original project post at 🔗 Dev Post

🔁 Reposted project post at 🔗 dev to @aravindvcyber

⏭ We have our next article in serverless and IaC, do check out

https://dev.to/aws-builders/aws-cdk-101-send-message-across-accounts-using-sns-topic-and-sqs-h2i

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

🤹‍♂️ AWS CDK 101 Projects - 🏄‍♂️ CDK Stack/Resource/Drift events forwarded to cool Slack posts - Event-Forwarder

Aravind V — Mon, 29 Aug 2022 05:03:00 +0000

Delighted to share my first post as AWS Community Builder, do read the full and share your thoughts.

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in my previous series on AWS CDK 101 which gives exposure to working from beginner to intermediate level.

If in case missed my previous article, do find it with the below links.

🔁 Original previous series post at 🔗 Dev Post

🔁 Reposted the previous series post at 🔗 dev to @aravindvcyber

In this new series article, we will be talking about new advanced serverless projects series built with AWS CDK using typescript.

This will be also my first series as AWS Community Builder in the developer's tools category.

Without my ado, let me start briefing about this first project I just put it into the public domain for this awesome aws community folks as an opportunity to collaborate and refine this tool which was just started to learn by playing in the cloud solutions.

Expectations 🏗️

This solution is one which I feel could not only help the developer but also their peers along with Leads, Ops, and architects to track the deployment of Cloud formation solutions in AWS cloud and monitor and track any mutations continuously using slack channels and yes in the future drive actions and discussions from the posts to have a seamless developer experience across multi-region and cross-account deployments.

This could very much help the developer/team monitor the deployments of both personal and organizational accounts with less footprint cost-effectively.

Starting with Cloudformation events to post Slack notifications effortless from multi-region and even cross accounts to never miss your/peers/ci initiated aws cloudformation deployments on stacks and resources besides that it could also notify drift events.

Architecture in short 🏗️

The backbone of this stack is based on the new feature released for the availability of cloud formation events in the default bus in the AWS Eventbridge.

These events are of three types.

Stack level Events
Resource level Events
Drift detection Events

This project stack has three components as elaborated below.

Event Forwarder Data Stack This has the dynamodb schema and pre-created indexes
Remote Event Router Stack which is deployed into one or many regions across accounts forwarding specific eventbridge events (Cloudformation Events specifically) into the specific target default EventBus (in the below stack) from the current default EventBus (source) by making use of event bridge rules.
Event Forwarder Stack which lives in a single region for event ingestion and transformation to various delivery channels (Slack is the first channel) from the default EventBus (target).

Prerequisites 🏗️

1) At least single region where you could cdk deploy with necessary privileges to spin up the resources such as lambda, sqs, eventbridge rules, and targets with access to cloudwatch and xray

Make use of AWS free tier benefits for personal use or request access sandbox account from your org to try this out.

1) At least one Custom Slack App with incoming webhook generated and configured to post to necessary slack channels

Want to experiment create a free slack workspace (Recommended), creating a slack app with an incoming webhook, or generating an incoming webhook in a slack app from your organization admin? There are multiple tutorials to get this or DM me for clarifications.

Create Slack App

Setup Incoming Webhook

Project repo in GitHub 🎢

I have fully shared the code base that I have built into this public repository which now available for your review and learning or even implement this into your account to catch up with the deployments happening across the interested stacks direct into your Organizational or Private Slack workspace. Private Slack workspaces are free to set up and use like me and you could create a simple slack app with an incoming webhook and direct the messages to your channel of choice.

event-forwarder Github repo

Use cases which emit Cloudformation Events 🎢

CDK deployment via terminal or even CDK destroy these are visible for the developer but may not be saved anywhere except in the AWS console view. Your peers are also not aware of these deployments happening until they check the console. Sending this to slack channels will drive greater involvements from the team.

Direct cloudformation console level changes/deletes

Resource level mutations can also be tracked with drift detection but only one person knows what happened and could not easily correlate
Someone edits the Cloud formation template using designer and updates the stack with/without changeset

Or yourself/team can have other parallel IaC framework like terraform making some changes via terminals
Even your CI/CD pipeline does deployments to AWS, you need not check them to understand if the deployments are completed rather slack post will reach your channel in seconds consistently across the included environments
Forbidden environments where you don't have access to checkout the events and resources created and may know if it had failed or does not know the resource names and you could not check this in the console.

Possibilities 🎢

Quite frankly this is my second weekend on this project, I believe that this could be very much improved when I receive feedback from the community and fine-tune and extend the solution in future.

Also I have not restricted the cfn events to any specific stack, since data generation was key to continuously build this solution. Make use of the event bridge rule filtering pattern and matchers to get the interest events only to avoid too much noise in your channels by simply adding more refined patterns.

const stackEventsRule = new Rule(this, "stack-events-rule", {
      eventPattern: {
        detail: {
          "stack-id": [{ exists: true }],
        },
        source: ["aws.cloudformation"],
      },
      eventBus,
    });

Extracted Utils 🎢

I have made used most of the generic parts of this solution are refracted into separated util functions which could help to simply the solution and may be reuse them in your own project work.

Slack Utils
Console Utils
Dynamodb Utils
Data model to interact with dynamodb
Cfn events definition and may be more soon.

Checkout the config folder 🎢

This project is designed to have the various developer expectations and limits we may have so I have made myself aware of those and included most of the configurations below as config.

The default.json and test.json are from GitHub with dummy fields make sure you create local.json and production.json overriding the necessary fields of choice.

{
    "account": "123",   //Primary account to host the solution
    "region": "ap-south-1", //Primary region to host the solution
    "slackhook": "", //You need to update your slack app incoming webhook as the primary delivery channel for your users
    "errorslackhook": "",//You need to update your slack app incoming webhook as the error delivery channel to notify the stack maintainers in your environment to get the errors as and when they happen and need to dive into cloudwatch logs to figure out most of the issues.
    "eventStore": "eventStores9",//This is some new dynamodb table name of your choice
    "remoteRegions":"us-east-1,us-east-2", //Your secondary regions to monitor
    "remoteAccounts":"123", //Your secondary accounts to monitor
    "logLevel":"WARN", //Use WARN for less verbose and use INFO for the verbose cloudwatch logs for the main processor handler
    "perPostEventCount": "10",//This is used to limit max N events in a single slack post, this is mainly to make sure we don't hit limit on a single slack post
    "dynamodbQueryPagingLimit": "15",//This is used to have limited the read units on a single API call to not throttle your dynamodb if you are using provisioned RCU.
    "logRetentionDays": "14",//Retention period for the primary handler cloudwatch logs
    "deleteNotified": "false", //Setting this to true will automatically delete data from dynamoDb once the event are notified. But I highly recommend you to have this false to understand the data which gets generated, which could help you with other integrations or possibilities in this monitoring
    "timeZone": "Asia/Kolkata", //custom time zone for time
    "locale": "en-US" //necessary locale
}

Slack Posts Results

Never miss anything happening to your cloudformation stacks since you will be always notified in your respective slack channel.

Drift detection notifications

Stack deletion notifications

Stack/Resource creation/update notifications

Even exception are sent to slack

This can be directly reported to the maintainers without waiting to dig into the cloudwatch logs to identify and track the issue.

And all these cool things about this slack post it is having a rich format that could be customized at the source and it also include a deep link to the resources provisioned besides the stack-level links to various console screens.

Note these console links will only work if you have already logged into your respective AWS account in the browser where you open them and if you have the necessary privileges to make sure that security even for production environments is honored when you share this across various members of your teams for any follow ups in any slack threads.

Dynamodb to store data

Here in this article, we choose to use dynamodb not only as an ad-hoc data store. I believe the data generated will trigger further insights and expand the possibilities of this solution. Also I have used provisioned RCUs and WCUs to make use of my free tier benefits and as well set throttling, it is also recommend try with on-demand mode and pay as your usage.

Indexes created for critical data lookups

Sample DB item

{
 "stackId": "arn:aws:cloudformation:ap-south-1:575066707855:stack/EventForwarderStack/279c3120-1f91-11ed-a6ef-022d5cdd6444",
 "time": 1661700649000,
 "account": "575066707855",
 "clientRequestToken": "",
 "detail": "{\"stack-id\":\"arn:aws:cloudformation:ap-south-1:575066707855:stack/EventForwarderStack/279c3120-1f91-11ed-a6ef-022d5cdd6444\",\"stack-drift-detection-id\":\"651434b0-26e6-11ed-817d-06bb5d2f96a6\",\"status-details\":{\"stack-drift-status\":\"\",\"detection-status\":\"DETECTION_IN_PROGRESS\"},\"drift-detection-details\":{\"drifted-stack-resource-count\":-1}}",
 "detectionStatus": "DETECTION_IN_PROGRESS",
 "driftDetectionDetails": "{\"drifted-stack-resource-count\":-1}",
 "eventId": "2892c708-d38d-9a9a-115e-c981c1d3d62f",
 "logicalResourceId": "",
 "notified": "false",
 "notifiedTime": 1661700651498,
 "physicalResourceId": "",
 "region": "ap-south-1",
 "resources": [
  "arn:aws:cloudformation:ap-south-1:575066707855:stack/EventForwarderStack/279c3120-1f91-11ed-a6ef-022d5cdd6444"
 ],
 "resourceType": "",
 "status": "DETECTION_IN_PROGRESS",
 "statusReason": "",
 "type": "CloudFormation Drift Detection Status Change"
}

Engineering Aspects 🎢

I would like to discuss this further in the upcoming articles and we refine this stack and refine. There is always opportunities to improve and perfect.

Besides this, I made this repo public now, and I am sure our creative community will like to request features/issues, help patch and contribute and refractor this solution further to feed our discussions in our engineering articles soon beside me.

This project is open to your generous contributions if you feel it will help other developers and also you could also solve the issues or bring new features with a Pull request.

event-forwarder Github repo

Github Workflow for CI and testing 💥

Added Github Workflow to validate the new changes.

⏭ We have our next article in serverless, do check out

https://dev.to/aws-builders/aws-cdk-101-cross-region-putevents-across-accounts-using-eventbridge-for-event-forwarder-project-1de2

If in case missed my previous article, do find it with the below links.

🔁 Original previous series post at 🔗 Dev Post

🔁 Reposted the previous series post at 🔗 dev to @aravindvcyber

🎉 Thanks for supporting! 🙏 and do follow and share this series for more such articles.

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

🤹‍♂ AWS CDK 101 - 🏄 GraphQL Mutations using AppSync with dynamodb

Aravind V — Sat, 04 Jun 2022 18:33:39 +0000

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 Original previous post at 🔗 Dev Post

🔁 Reposted the previous post at 🔗 dev to @aravindvcyber

In this article, we will be introducing a data access layer as a wrapper on top of our dynamodb table. Here specifically we have chosen graphql using AWS appsync to perform mutations in our dynamodb table.

Construction 🏗️

Let us start by updating our previous file lib/appsync-stack.ts for our new stack.

Schema definition for Mutations 🎢

You may also use the AWS console to update the schema before we update in CDK asset files when we know for sure.

Here you can find the mutations part which we have appended to our schema file which we created in our previous article above.

input CreateMessagesTableInput {
    createdAt: Int!
    messageId: String!
    event: AWSJSON
}

input UpdateMessagesTableInput {
    createdAt: Int!
    messageId: String!
    event: AWSJSON
}

input DeleteMessagesTableInput {
    createdAt: Int!
    messageId: String!
}

type Mutation {
    createMessagesTable(input: CreateMessagesTableInput!): MessagesTable
    updateMessagesTable(input: UpdateMessagesTableInput!): MessagesTable
    deleteMessagesTable(input: DeleteMessagesTableInput!): MessagesTable
}

Dynamodb connection as a data source 🛶

Here we have already directly integrated dynamodb API with graphql as a data source.

Additionally we are updating our earlier article by making use of the import value of table arn to get the table name directly from the previous stack exports.

Exporting table arn from another stack 🐆

new CfnOutput(this, "MessagesTableExport", {
      value: messages.tableArn,
      exportName: 'MessagesTableArn'
    })

Importing table arn in this stack 🐮

const messages = Table.fromTableArn(
      this,
      "MessagesTableImport",
      Fn.importValue("MessagesTableArn")
    );

const MessagesDS = AppSyncApi.addDynamoDbDataSource("MessagesDataSource", messages);

VTL Mapping template 🛩️

Here we need to use VTL (Velocity Template Language) to transform/manipulate our request and the response we send/receive from the below resolvers. Using this can be a good strategy as this can be used in many places not only in appsync and API gateway.

You may also use the AWS console to test these transformations using the sample payload from logs before we update in CDK asset files.

Create message resolver 🚠

Here you can also look into the Xray trace to understand when these blocks are utilized for createMessagesTable resolver

MessagesDS.createResolver({
      typeName: "Mutation",
      fieldName: "createMessagesTable",
      requestMappingTemplate: MappingTemplate.fromFile(
        "assets/createMessagesTableRequest.vtl"
      ),
      responseMappingTemplate: MappingTemplate.fromFile(
        "assets/createMessagesTableResponse.vtl"
      ),
    });

createMessagesTableRequest VTL template 🌟

{
  "version": "2017-02-28",
  "operation": "PutItem",
  "key": {
    "messageId": $util.dynamodb.toDynamoDBJson($ctx.args.input.messageId),
    "createdAt": $util.dynamodb.toDynamoDBJson($ctx.args.input.createdAt),
  },
  "attributeValues": $util.dynamodb.toMapValuesJson($ctx.args.input),
  "condition": {
    "expression": "attribute_not_exists(#messageId) AND attribute_not_exists(#createdAt)",
    "expressionNames": {
      "#messageId": "messageId",
      "#createdAt": "createdAt",
    },
  },
}

createMessagesTableResponse VTL template ⛱️


$util.toJson($context.result)

Delete message resolver 🤡

Here you can also look into the Xray trace to understand when these blocks are utilized for deleteMessagesTable resolver

MessagesDS.createResolver({
      typeName: "Mutation",
      fieldName: "deleteMessagesTable",
      requestMappingTemplate: MappingTemplate.fromFile(
        "assets/deleteMessagesTableRequest.vtl"
      ),
      responseMappingTemplate: MappingTemplate.fromFile(
        "assets/deleteMessagesTableResponse.vtl"
      ),
    });

deleteMessagesTableRequest VTL template 🎈

{
  "version": "2017-02-28",
  "operation": "DeleteItem",
  "key": {
    "messageId": $util.dynamodb.toDynamoDBJson($ctx.args.input.messageId),
    "createdAt": $util.dynamodb.toDynamoDBJson($ctx.args.input.createdAt),
  },
}

deleteMessagesTableResponse VTL template 🎣

$util.toJson($context.result)

Update messages resolver 🦄

Here you can also look into the xray trace to understand when these blocks are utilized for updateMessagesTable resolver

MessagesDS.createResolver({
      typeName: "Mutation",
      fieldName: "updateMessagesTable",
      requestMappingTemplate: MappingTemplate.fromFile(
        "assets/updateMessagesTableRequest.vtl"
      ),
      responseMappingTemplate: MappingTemplate.fromFile(
        "assets/updateMessagesTableResponse.vtl"
      ),
    });

updateMessagesTableRequest VTL template 🐿️

This update logic may look complex but just remember that while updating we need to take care of updated attributes, removed attributes, and new attributes.

{
  "version": "2017-02-28",
  "operation": "UpdateItem",
  "key": {
    "messageId": $util.dynamodb.toDynamoDBJson($ctx.args.input.messageId),
    "createdAt": $util.dynamodb.toDynamoDBJson($ctx.args.input.createdAt),
  },

  #set( $expNames  = {} )
  #set( $expValues = {} )
  #set( $expSet = {} )
  #set( $expAdd = {} )
  #set( $expRemove = [] )

  ## Looping through each argument, except keys **
  #foreach( $entry in $util.map.copyAndRemoveAllKeys($ctx.args.input, ["messageId", "createdAt"]).entrySet() )
    #if( $util.isNull($entry.value) )
      #set( $discard = ${expRemove.add("#${entry.key}")} )
      $!{expNames.put("#${entry.key}", "${entry.key}")}
    #else
      $!{expSet.put("#${entry.key}", ":${entry.key}")}
      $!{expNames.put("#${entry.key}", "${entry.key}")}
      $!{expValues.put(":${entry.key}", $util.dynamodb.toDynamoDB($entry.value))}
    #end
  #end

## Updating existing attributes **
  #set( $expression = "" )
  #if( !${expSet.isEmpty()} )
    #set( $expression = "SET" )
    #foreach( $entry in $expSet.entrySet() )
      #set( $expression = "${expression} ${entry.key} = ${entry.value}" )
      #if ( $foreach.hasNext )
        #set( $expression = "${expression}," )
      #end
    #end
  #end

  ## Adding new attributes **
  #if( !${expAdd.isEmpty()} )
    #set( $expression = "${expression} ADD" )
    #foreach( $entry in $expAdd.entrySet() )
      #set( $expression = "${expression} ${entry.key} ${entry.value}" )
      #if ( $foreach.hasNext )
        #set( $expression = "${expression}," )
      #end
    #end
  #end

  ## Removing unwanted attributes **
  #if( !${expRemove.isEmpty()} )
    #set( $expression = "${expression} REMOVE" )

    #foreach( $entry in $expRemove )
      #set( $expression = "${expression} ${entry}" )
      #if ( $foreach.hasNext )
        #set( $expression = "${expression}," )
      #end
    #end
  #end

  ## Final update expression **
  "update": {
    "expression": "${expression}",
    #if( !${expNames.isEmpty()} )
      "expressionNames": $utils.toJson($expNames),
    #end
    #if( !${expValues.isEmpty()} )
      "expressionValues": $utils.toJson($expValues),
    #end
  },

  "condition": {
    "expression": "attribute_exists(#messageId) AND attribute_exists(#createdAt)",
    "expressionNames": {
      "#messageId": "messageId",
      "#createdAt": "createdAt",
    },
  }
}

updateMessagesTableResponse VTL template 🐁

$util.toJson($context.result)

Client to explore graphQl ❄️

Appsync Explorer Queries ♨️

In the AWS console, you can navigate the appsync and start querying. One advantage you have here is that we have cloud watch and tracing logs readily available if in case you want to check.

Apollo Studio Queries 🏕️

But normally prefer the dark mode in apollo graphql studio, you may also try it out if you prefer that, maybe we would get that in the AWS console as well someday.

We will be refining this in our coming articles to achieve optimum speed and resource utilization.

We will be adding more connections to our stack and making it more usable in the upcoming articles by creating new constructs, so do consider following and subscribing to my newsletter.

⏭ We have our next article in serverless, do check out

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

🤹‍♂ AWS CDK 101 - 🏄 GraphQL Mutations using AppSync with dynamodb

Checkout more in my page https://t.co/CuYxnKr0Ig #Serverless #typescript #graphql #awscdk #dynamodb #aws https://t.co/3y1zUMIyFk
— Aravind V (@Aravind_V7) June 4, 2022

🏄 AWS CDK 101 - 🐲 GraphQL using AppSync with dynamodb

Aravind V — Mon, 30 May 2022 05:42:44 +0000

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 Original previous post at 🔗 Dev Post

🔁 Reposted the previous post at 🔗 dev to @aravindvcyber

In this article, we will be introducing a data access layer as a wrapper on top of our dynamodb table. Here specifically we have chosen graphql using AWS appsync to perform basic list items and get an item from dynamodb.

Construction 🏗️

Let us start by creating a new file lib/appsync-stack.ts for our new stack.

Imports used in this new stack ⛩️

Here we have imported the below objects to help us in our definition

import { Duration, Expiration, Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { Table } from 'aws-cdk-lib/aws-dynamodb';
import { GraphqlApi, MappingTemplate, Schema, FieldLogLevel, AuthorizationType } from 'aws-lib-cdk/aws-appsync-alpha';

Appsync construct with the definition ⛺

A new stack is created which is used to define our appsync endpoint as shown below.

Here we will be getting a default API key with an expiration of 7 days so that we can rotate this periodically. Besides that, we have logged our API sufficiently to understand the background process much better.


export class GqlStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const AppSyncApi = new GraphqlApi(this, 'gqlApi', {
      name: 'gqlApi',
      schema: Schema.fromAsset('assets/messages-schema.gql'),
      xrayEnabled: true,
      logConfig: {
            excludeVerboseContent: false,
            fieldLogLevel: FieldLogLevel.ALL,
      },
      authorizationConfig: {
          defaultAuthorization: {
              authorizationType: AuthorizationType.API_KEY,
              apiKeyConfig: {
                  name: 'default-api-key',
                  description: 'default-api-key-description',
                  expires: Expiration.after(Duration.days(7))
              }
          }
      }
    })

  }
}

Schema definition file 🎢

You may also use the AWS console to update the schema before we update in CDK asset files when we know for sure.

type Message {
    message: AWSJSON!
}

type MessagesTable {
    createdAt: AWSTimestamp!
    messageId: String!
    event: Message
}

type MessagesTableConnection {
    items: [MessagesTable]
    nextToken: String
    scannedCount: Int
}

type Query {
    getMessage(messageId: String!, createdAt: AWSTimestamp!): MessagesTable
    listMessages(filter: TableMessagesTableFilterInput, limit: Int, nextToken: String): MessagesTableConnection
}

input TableAWSTimestampFilterInput {
    ne: AWSTimestamp
    eq: AWSTimestamp
    le: AWSTimestamp
    lt: AWSTimestamp
    ge: AWSTimestamp
    gt: AWSTimestamp
    contains: AWSTimestamp
    notContains: AWSTimestamp
    between: [AWSTimestamp]
}

input TableMessagesTableFilterInput {
    createdAt: TableAWSTimestampFilterInput
    messageId: TableStringFilterInput
}

input TableStringFilterInput {
    ne: String
    eq: String
    le: String
    lt: String
    ge: String
    gt: String
    contains: String
    notContains: String
    between: [String]
    beginsWith: String
}

Dynamodb connection as a data source 🛶

Here we are directly integrating dynamodb API with graphql as a data source

const messages = Table.fromTableName(this,'MessagesTableImport', 'MessagesTable');

const MessagesDS = AppSyncApi.addDynamoDbDataSource("MessagesDataSource", messages);

VTL Mapping template 🛩️

You may also use the AWS console to test these transformations using the sample payload from logs before we update in CDK asset files.

Get message resolver 🚠

Here you can also look into xray trace to understand when these blocks are utilized for getMessage resolver

MessagesDS.createResolver({
      typeName: 'Query',
      fieldName: 'getMessage',
      requestMappingTemplate: MappingTemplate.fromFile('assets/getMessageRequest.vtl'),
      responseMappingTemplate: MappingTemplate.fromFile('assets/getMessageResponse.vtl'),
})

getMessageRequest VTL template 🌟

{
    "version": "2017-02-28",
    "operation": "GetItem",
    "key": {
        "messageId": $util.dynamodb.toDynamoDBJson($ctx.args.messageId),
        "createdAt": $util.dynamodb.toDynamoDBJson($ctx.args.createdAt)
    }
}

getMessageResponse VTL template ⛱️


#set($ctx.result.event = $util.parseJson($ctx.result.event))

$util.toJson($ctx.result)

List messages resolver 🤡

Here you can also look into xray trace to understand when these blocks are utilized for listMessages resolver

MessagesDS.createResolver({
        typeName: 'Query',
        fieldName: 'listMessages',
        requestMappingTemplate: MappingTemplate.fromFile('assets/listMessagesRequest.vtl'),
        responseMappingTemplate: MappingTemplate.fromFile('assets/listMessagesResponse.vtl'),
})

listMessagesRequest VTL template 🎈

{
  "version": "2017-02-28",
  "operation": "Scan",
  "filter": #if($context.args.filter) $util.transform.toDynamoDBFilterExpression($ctx.args.filter) #else null #end,
  "limit": $util.defaultIfNull($ctx.args.limit, 20),
  "nextToken": $util.toJson($util.defaultIfNullOrEmpty($ctx.args.nextToken, null)),
}

listMessagesResponse VTL template 🎣

#set($children = [])
#foreach($item in $ctx.result.items)
  #set($item.event = $util.parseJson($item.event))
  $util.qr($children.add($item))
#end
#set($ctx.result.items = $children)
$util.toJson($ctx.result)

Client to explore graphQl ❄️

Appsync Explorer Queries ♨️

In the AWS console, you can navigate the appsync and start querying. One advantage you have here is that we have cloud watch and tracing logs readily available if in case you want to check.

Apollo Studio Queries 🏕️

But normally prefer the dark mode in apollo graphql studio, you may also try it out if you prefer that, maybe we would get that in the AWS console as well someday.

Conclusion 💫

In this exercise, we have tried only the get and list operations to differentiate the effectiveness of the scan and query operation.

scannedCount value in the above results shows the cost associated with list operation as the table gets bigger. I have removed the filter variable to show you what are the scanned records as shown below.

We will be refining this in our coming articles to achieve optimum speed and resource utilization.

We will be adding more connections to our stack and making it more usable in the upcoming articles by creating new constructs, so do consider following and subscribing to my newsletter.

⏭ We have our next article in serverless, do check out

https://dev.to/aravindvcyber/aws-cdk-101-graphql-mutations-using-appsync-with-dynamodb-9n0

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

🏄 AWS CDK 101 - 🐲 GraphQL using AppSync with dynamodb
{ by @Aravind_V7 }

Checkout more such in my page https://t.co/CuYxnKr0Ig https://t.co/phzOKFlXXO #aws #typescript #graphql #dynamodb #awscdk https://t.co/yeqeLZQt8A
— Aravind V (@Aravind_V7) May 30, 2022

💥 AWS CDK 101 - 🐣 Event source mapping with Cfn property override

Aravind V — Tue, 24 May 2022 04:21:13 +0000

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 Original previous post at 🔗 Dev Post

🔁 Reposted the previous post at 🔗 dev to @aravindvcyber

In this article, we will be making a simple change in our CDK stack which eventually helps us with more optimization and control in processing streams. The high-level configuration which we have made to connect dynamodb streams event source with lambda in the last two articles will be optimized using event source mapping L2 construct with simple override with its cfn L1 construct as well.

Why we need this refactoring 🪲

In our last article, you can find that we are using dynamodb streams from stgMessages to delete objects from S3. The current configuration does not filter or distinguish the event based on the event name at the source, instead, we invoke the lambda unnecessary even for events other than REMOVE like INSERT and we end up, making unwanted batch invocations, which could have otherwise been avoided.

Benefits we achieve 🌋

In a previous couple of articles we directly interfaced with our dynamodb streams without any filter and our lambda is processing every single one of them and optionally choosing business logic based on the CRUD operation which leads to more number of stream records processed by the lambda eventually.
The above scenario can be avoided by filtering using the eventName in the dynamodb stream record and selectively using only the most appropriate record needed to be processed in our specific lambda handler.
Here we not only get a chance to filter with CRUD operation, but also you can filter with the other fields which are part of the record object, this will help us filter these messages to the fullest.
One more thing we achieve here would be that we can offload the filtering from inside the lambda to the invocation source mapping itself and hence time and cost are saved.

Necessary imports in the stack 📙

Here we will be making use of the below imports in our stack file.


import {
  CfnEventSourceMapping,
  EventSourceMapping,
  StartingPosition,
} from "aws-cdk-lib/aws-lambda";

Previous event source mappings 🌷

Earlier we have used something like the below to configure the source to the lambda. This does not have the filtering enabled and all the streams are used to invoke the lambda, which leads to unwanted executions.

The below L2 construct does not yet provide a provision to add the filter and hopefully, we can expect this soon.


const stgTableSourceProps: DynamoEventSourceProps = {
    startingPosition: lambda.StartingPosition.LATEST,
    batchSize:100,
    maxBatchingWindow: Duration.seconds(60),
}
stgMessageStreamFunc.addEventSource(new DynamoEventSource(stgMessages, stgTableSourceProps));

Generic event source mapping construct 🌰

Due to the shortcomings discussed above, we are going to try another generic L2 construct.

We will replace this with a more generic event source mapping construct as shown below for both the lambda handlers used in the previous 2 articles.
Here we have added a few changes besides filtering.

startingPosition is changed to StartingPosition.TRIM_HORIZON to get the oldest item first.
bisectBatchOnError will split the record chunk into half and retry each half so that we can identify and isolate the problematic record from the other records while processing them inside the handler.

const msgDeleteEventSourceMap = new EventSourceMapping(this, "msgDeleteEventSourceMap", {
      target: messageStreamFunc,
      eventSourceArn: messages.tableStreamArn,
      batchSize: 100,
      maxBatchingWindow: Duration.seconds(60),
      startingPosition: StartingPosition.TRIM_HORIZON,
      bisectBatchOnError: true, 
});

msgDeleteEventSourceMap.applyRemovalPolicy(RemovalPolicy.DESTROY);


const stgMsgDeleteEventSourceMap = new EventSourceMapping(this, "stgMsgDeleteEventSourceMap", {
      target: stgMessageStreamFunc,
      eventSourceArn: stgMessages.tableStreamArn,
      batchSize: 100,
      maxBatchingWindow: Duration.seconds(60),
      startingPosition: StartingPosition.TRIM_HORIZON,
      bisectBatchOnError: true, 
});

stgMsgDeleteEventSourceMap.applyRemovalPolicy(RemovalPolicy.DESTROY);

Granting stream read to handler functions 🍄

In the last two articles, we have not performed this explicitly, since the dynamodb event source addition to the lambda construct, automatically applied for the necessary permissions. Whereas in this case, we may have to grant the privileges as shown below.


messages.grantStreamRead(messageStreamFunc);


stgMessages.grantStreamRead(stgMessageStreamFunc);

Cloudformation property override 🌼

Currently, we could not add the filter criteria as this EventSourceMapping L2 construct directly, so here we have to access its L1 construct and add a property override as shown below for both the lambda handlers.

Once do note that there is much scope in the filtering pattern used here where we could include more fields to achieve the desired effect and it is not only limited to eventName only.

const cfnMsgDeleteEventSourceMap = msgDeleteEventSourceMap.node.defaultChild as CfnEventSourceMapping;

cfnMsgDeleteEventSourceMap.addPropertyOverride("FilterCriteria", {
      Filters: [
        {
          Pattern: JSON.stringify({
            eventName: ["INSERT"],
          }),
        },
      ],
});


const cfnStgMsgDeleteEventSourceMap = stgMsgDeleteEventSourceMap.node.defaultChild as CfnEventSourceMapping;

cfnStgMsgDeleteEventSourceMap.addPropertyOverride("FilterCriteria", {
      Filters: [
        {
          Pattern: JSON.stringify({
            eventName: ["REMOVE"],
          }),
        },
      ],
});

Issue while deploying the stack 🍭

You may occasionally get an error failure to deploy the stack, by then it is necessary for us to manually delete the event source mapping in the lambda either with the AWS console or using the AWS CLI as shown below.

The stack named CommonEventStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The event source arn (" arn:aws:dynamodb:ap-south-1:*****:table/stgMessagesTable/stream/2022-05-21T03:39:54.488 ") and function (" CommonEventStack-stgMessageStreamFuncF95CB7BF-877BwMFlctoc ") provided mapping already exists. Please update or delete the existing mapping with UUID 43eccb79-7a6f-4fd0-82f1-c176cff317da

AWS console to delete the mapping 🎏

AWS cli to delete mapping 🍣

Verify the event source mapping if it is existing as shown below

aws lambda delete-event-source-mapping --uuid ********

Then we can delete it by executing the below command

aws lambda delete-event-source-mapping --uuid ********

Conclusion 🍤

Here we have demonstrated two things as shown below.

Ability to filter streams inside the generic event source mapping which is not only limited to dynamodb stream, this has a lot of other applications to be used similarly for kinesis streams, SQS source, and a lot more sources.
Additionally we have learned how we can use the L1 construct for an L2 resource and override certain properties, this will have a lot of use cases beyond this example.

We will be adding more connections to our stack and making it more usable in the upcoming articles by creating new constructs, so do consider following and subscribing to my newsletter.

⏭ We have our next article in serverless, do check out

https://dev.to/aravindvcyber/aws-cdk-101-graphql-using-appsync-with-dynamodb-16dd

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

🏄 AWS CDK 101 - 🐲 GraphQL using AppSync with dynamodb
{ by @Aravind_V7 }

Checkout more such in my page https://t.co/CuYxnKr0Ig https://t.co/phzOKFlXXO #aws #typescript #graphql #dynamodb #awscdk https://t.co/yeqeLZQt8A
— Aravind V (@Aravind_V7) May 30, 2022