DEV Community: arszen123

Building an offer notification service on AWS

arszen123 — Sat, 26 Feb 2022 10:18:20 +0000

Let's build a simple web crawler application on AWS, that sends a notification when there is a special offer for a product.

I love peanut butter, and whenever there is an offer for it I would like to stock up some. So I thought I will create a simple application that sends me a notification whenever there is an offer for this product. What I need, to create this application is:

The product URL,
And the exact location where the offer is located on the webpage, so I can extract this information. For this product, the offer is located in the #pap-banner-text-value HTML element.

Probably you want to get notified by different things, so you can customize the application logic. Perhaps you would like to get notifications when a GPU/XBOX/PS4 is back in stock or you want to pull data from an API and send notifications based on predefined criteria. It's up to you what the application sends notifications about.

Architecture

The AWS services we will use to create this application are:

AWS EventBridge - For scheduling lambda function invocations
AWS Lambda - For crawling the website and publishing messages to an SNS topic
AWS SNS - For sending email notifications

We'll also use NodeJS for the Lambda function and Serverless for managing the infrastructure and deploying the application.

Setting up the development environment

First, we have to install the Serverless CLI.

npm install -g serverless

After installation, we have to configure the credentials. If you don't have it, you can generate new credentials on the AWS console.

serverless config credentials --provider aws --key 1234 --secret 5678

After we have installed the CLI, we can create a new project.

serverless create --template aws-nodejs --path offer-notification-application

The above command creates a skeleton project with a serverless.yml file, where we'll define our infrastructure, and a handler.js where we'll implement our Lambda function.

Implementation

Based on the above architecture diagram, we'll have a Lambda function, which will be invoked every day. This Lambda function will fetch the target website content, and whenever an offer is found it publishes a message to an SNS topic. Because our Lambda function will publish to an SNS topic, therefore it must have the necessary permissions to do this.

We also need an SNS topic where the offers will be published and an email subscriber which needs to be notified whenever a new message is published on this topic.

The following serverless.yml file describes the above-mentioned architecture.

We also need a .env file with an EMAIL environment variable in it, set to the desired email address where we want to receive the notifications.

Now, that we have defined our infrastructure, we can move on and write our application logic.

We should create a new src directory and move the handler.js file under the newly created directory, to make it better structured.

What our Lambda function needs to do is fetch the target website content and check whether there is an interesting offer. For this, we need some additional packages to fetch the site (axios) and parse the HTML content (cheerio), so let's install them with the following command.

npm install axios cheerio

Now we have everything to implement the application core logic. For this product, it will look like the following.

As you can see the fetchOffer function fetches the website content, and as we already know that the offer is located in the #pap-banner-text-value HTML element, with cheerio we can easily extract the content of it.

Because I only want to get notifications when the offer is something like 2 FOR 1 or 30 % off, I need to check if the offer matches with one of the regular expressions.

In the handler function, the fetchOffer function is invoked, and whenever it returns an offer it will be published to the SNS topic.

It's very simple, isn't it?

Deployment

Now we can deploy the application to AWS with a single command.

serverless deploy

After it's successfully deployed, we should receive a confirmation email to the configured email address. After we have confirmed the subscription, we'll receive an email from every message published on that SNS topic.

To remove the deployed application run the following command.

serverless remove

Testing

We can easily test the deployed application manually by invoking the Lambda function with the following command.

serverless invoke --function crawl

If there is an offer on the site, we should receive an email about the offer.

Summary

For creating this application we have used Serverless, to define the infrastructure and deploy our application. We also used AWS Lambda for running our code, scheduled events for triggering the Lambda function invocation, and SNS for sending email notifications to subscribers. As you have seen it was very easy to implement and deploy this application to AWS with Serverless.

You can check out the repository on GitHub.

GitHub Copilot in flight

arszen123 — Thu, 09 Dec 2021 15:30:32 +0000

GitHub Copilot in flight

GitHub Copilot?

If you haven't heard about GitHub Copilot yet, it is an AI-powered code assistant which can suggest whole lines and entire functions of code inside your favorite IDE.

Copilot in flight

Just when I started to create a simple compiler in TypeScript, I got access to GitHub Copilot. I got excited and instantly integrated it into VSCode and started using it. My first thought was it will only slow me down because I have to interpret and correct every single line of code it suggests. Soon I realized it made the development faster than I could have imagined.

The first thing I tried is to make Copilot suggest me a complete implementation of a CircularBuffer utility class. But it only suggested a simple Buffer class, which implementation was incorrect and I had to fix it manually.

Probably this was a bigger challenge for it, so I decided to only use it for smaller tasks like function implementations and unit test writing. In the beginning, it still suggested some irrelevant and incorrect code, but as the project grew the suggestion got better.

My experience is that it adapts to the developer/project, so you can't trust it 100% (as it is noted on their website). For example, at first, I started writing unit tests like this:

And soon it started suggesting similar code. Imagine a test case where I want to test a more complex expression, like this: 10 * (4 + 16) / 10. It would be messy, and unreadable. So I refactored the test cases to assert the generated syntax tree.

You can see that it's much easier to read this test case than the previous one. Copilot quickly adapted to this and started suggesting relevant code, to make the development faster.

Another example is when I wanted to parse a simple expression with a single function call for the first time.

At first sight, this code suggestion looks good, but it does not fit entirely well into my project. First, there is no CallNode, only a FunctionCallNode (It was defined before I started writing this test case). Also, the node first parameter should be a token and not another node. You can see a possible implementation below.

After this test case was implemented, Copilot started suggesting correct implementations for similar test cases. Below is a nested function call test case suggested by Copilot.

I also tried to observe Copilot during the core components implementation. So, I gave it a task to suggest a function call parser, and to my surprise, it suggested a nice code snippet.

But there is some problem with the above code,

It should create a new FunctionCallNode and not a VarNode ,
If the current token is not a function identifier, it should forward the control and try to parse an expression,
The parsed function must have one and only one parameter, and it is a big limitation.

As you can see in the above implementation the previously mentioned errors are fixed and the parsed function calls can have zero or more parameters.

Conclusion

For me, Copilot was very helpful for some tedious work, especially when I had to write almost the same code over and over (ex. token definitions, specific node types), and it was also very useful for writing unit tests. For the core components implementation, I mostly used it as an autocomplete to type faster, and not as an assistant who suggests whole implementations.

In my opinion, Copilot adapts to the project/developer, therefore it does not make anyone a better developer, but it can make a good developer faster. We still have to understand the suggested code and correct any mistake it makes, but it looks very promising.

Thank you for reading my first ever article, I hope you enjoyed it.