DEV Community: Nabin Adhikari

AWS Simple Email Service Caveat

Nabin Adhikari — Fri, 17 Jan 2020 04:55:43 +0000

Amazon Web Services (AWS) has a ton of features for almost anything a developer wants to do. AWS has one for sending emails called Simple Email Service SES. AWS SES can be used to send a text-based and template-based email.

In this post, I want to talk about one particular caveat by sending a template-based email using AWS SES. It is what Amazon says "Rendering-Failure", which happens when variables in the template don't match with provided template data. Let's see what that statement means.

Let's say we have the following template (Template name: WelcomeEmail) that we want to use.

<div>
  <h1>Welcome {full_name}!</h1>
  <p>{message}</p>
</div>

If we were to provide data for this template, it would look something like this.

const templateData = {full_name: "Nabin Adhikari", message: "Australia is burning"}

We can use the following Javascript code from lambda to send this email and hopefully you'd receive it.

await this.ses.sendTemplatedEmail({
  Destination: { ToAddresses: ["you@example.com"], },
  Source: "me@example.com",
  Template: "WelcomeEmail",
  TemplateData: JSON.stringify(templateData)
}).promise();

For more information on how to send templated-based email using AWS SES, please read this - https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-personalized-email-api.html.

The Problem

Now is the time to talk about the caveat SES has. See how template and template data have the same number variables and exactly the same variables in the example above? That is why you would probably receive the email. However, what do you think will happen if you miss any variables or added new variables unknown to the template? You might have already guessed. Yes, the email might not be sent. Now you say, yes we can catch the error and will process it later. But here is the caveat, SES would say it succeed. It is frustrating to ensure all of these data match with the template when you have many email templates and SES API saying success.

Yes Nabin, that sucks, what can be done about this? Well, in the link I have attached for SES email above, it also talks about the solution. If you don't want to feel abandoned after you have done everything the article says and still no success, please continue with this post.

The Solution

The inconsistencies between template variables and template data is called "Rendering-Failure" by AWS, which can be notified through AWS Simple Notification Service (SNS). Let's go through this to get notified if any of "Rendering-Failure" occurs while sending the template-based email.

The first thing we need for this is an SNS topic and subscribe to it. Let's create a topic first. Go to Amazon SNS dashboard and to Topics from the left side menu. Click a button to create a new topic and give it a meaningful name. I will name it ses-failure. Next, we need to subscribe to it. For this, go to the Subscriptions tab in newly created topic details and click on the button labeled "Create Subscription" to starting creating a subscription. You need to choose a protocol that is suitable for you, email is fine for me. If you also choose email, enter your email address as Endpoint and hit the "Create Subscription" button. You have to confirm the subscription by clicking on a link sent to your email address. Now, we should have a topic and a subscription as in the image below.

Next, we need to create a configuration set in AWS SES. First, go to the SES dashboard and click Configuration Sets from the left side menu. You should see a big button to create a new configuration set. Click on it and give it a meaningful name and click the Create Configuration Set button. I'll name it rendering-failure-cs.

Next, you need to edit that configuration to add a destination. You should see a Select a Destination Type dropdown, click on it and choose SNS. This would open a dialogue with form as below.

You need to leave Enabled enabled, give a meaningful name, check Rendering Failure from Event types options, choose the SNS topic name you created earlier (mine was ses-failure), and submit. You should see it as below.

This is all we need to do as per AWS documentation. However, there is one final step to do for all of these steps to work.

The Final Step

The final step is to add the created configuration set name while sending an email. My configuration set name is rendering-failure-cs, you should remember yours. Do not confuse this name with Added SNS Event destination name, because I did. Add ConfigurationSetName: "rendering-failure-cs" parameter while sending email as shown in screenshot below.

Now if you miss any variable data or added extra unintentionally, you'd receive an email with missing or extra variable or the reason behind rendering failure.

Finally

AWS has great resources and tons of services, but every now and then I got stuck in between because of incomplete documentation if not missing documentation. I had spent quite a bit of time to make this thing working and I wanted to share with you so that you don't have to. Thank you for reading this far. I would really appreciate if you provide any feedback or suggestions.

Cover Image from Unsplash. https://unsplash.com/photos/gClIPKFrRjE

Amazon Simple Queue Service with NodeJs (Amazon SQS)

Nabin Adhikari — Sun, 24 Nov 2019 22:23:10 +0000

As we all know Amazon is providing countless services to make our (as developer) life easier. Simple Queue Service or aka Amazon SQS is boon for microservice or serverless application.

Need for SQS will be realized mostly when one part of the service works very quickly and another at a different pace. For example, You have a job that sends a daily email to your subscribers. It can get hundreds or thousands of email addresses to send emails to in a matter of milliseconds. However, sending emails is a bit slower process. SQS comes handy in this scenario. Using SQS, your first job could send email addresses to queue, a lambda function is triggered whenever there is a new message in the queue and get the email address and finally send it.

This article is divided into three sections. In section one we setup AWS and create two functions with minimum code and deploy them to AWS. In section two, we would actually work on creating an SQS queue and sending messages. In the last section, we work on functionTwo to make it receive and process queue messages.

If you just want to see the code, click Here

Section One - Setup

Let's now talk about how can we implement SQS using NodeJs. In this article, we will be using the Serverless Framework, which makes it easier to get started and deploy our application stack to AWS.

First, install the Serverless Framework globally using this command in your terminal. And configure your AWS account using your key and secret key.

# Install serverless
npm install -g serverless
# Configure AWS
serverless config credentials --provider aws --key <your_key> --secret <your_secret_key>

Next, let's create our sample app using serverless.

serverless create --template aws-nodejs-typescript --path sqs-demo

This should create a project inside the sqs-demo folder with the following files.

We need to install all the NPM packages in this project. So let's go to that folder and install them. Following command be used, if you are unsure.

# Change the current directory to project directory
cd sqs-demo
# Install required NPM packages
npm install

We will also need aws-sdk to send a message. Please install it as developer dependencies using the following command.

npm i -D aws-sdk @types/aws-sdk

To test sending and receiving a message with SQS, we'd need two lambda functions. One to send a message and another to receive and process it. Let's go to handler.ts file and delete everything to start fresh.

Let's create two functions functionOne and functionTwo, where functionOne will send messages to SQS and functionTwo would receive and process that message. At this point, your handler.ts should have these lines of code.

// handler.ts
import { Handler, SQSHandler } from 'aws-lambda';
import 'source-map-support/register';

export const funtionOne: Handler = async (_event, _context) => {
  console.log("I am funtionOne");
}

export const funtionTwo: SQSHandler = async (_event, _context) => {
  console.log("I am funtionTwo");
}

We need to put these functions into the serverless.yml file in order for Serverless to create them. Simply remove the hello function under the functions section and add these two. We also need to give them SQS permission. serverless.yml file after adding these two functions would look like this.

service:
  name: sqs-demo

custom:
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true

plugins:
  - serverless-webpack

provider:
  name: aws
  runtime: nodejs12.x
  region: ap-southeast-2
  apiGateway:
    minimumCompressionSize: 1024 # Enable gzip compression for responses > 1 KB
  environment:
    AWS_NODEJS_CONNECTION_REUSE_ENABLED: 1
  # SQS Permission given
  iamManagedPolicies:
  - 'arn:aws:iam::aws:policy/AmazonSQSFullAccess'

functions:
  funtionOne:
    handler: handler.funtionOne
  funtionTwo:
    handler: handler.funtionTwo

You can deploy these functions using Serverless easily. Just put this command.

serverless deploy

This should create two functions in AWS Lambda. Now we move into section two, where we actually start working on the SQS side.

Section Two - SQS Setup + Send Message

Let's login to AWS Console and search for Simple Queue Service and go there. Chose the appropriate region on the top right corner of the webpage and hit Get Started Now button. This should start the process of creating a new queue.

Enter the queue name you want to give it, I'd put sqs-demo. Now there are two types of the queue to choose from between Standard Queue and FIFO Queue. You can read the description to understand them better, but for this demo purpose, we are going with Standard Queue. Now hit the Quick-Create Queue button to create it.

Now the queue should be created and ready for use. Your page should look like this. Please note the ARN and URL, we will need it later.

Now let's enable our functionOne to send a message. To achieve this we need an SQS package from aws-sdk. Simply import it at the top of this file import {SQS} from 'aws-sdk and add the following code in the functionOne. I will explain it later.

try {
    const sqs = new SQS({
      region: 'ap-southeast-2'
    });
    const response = await sqs.sendMessage({
      MessageBody: 'hello world',
      QueueUrl: '<your queue url>'
    }).promise();
    console.log(`Message put on queue`, response);
  } catch (e) {
    console.log('Exception on queue', e);
  }

In this snippet, we are creating a new SQS object providing our region (mine is ap-southeast-2). Then we are using its sendMessage method to send our message to the queue. We'd need to provide MessageBody and QueueUrl, which we saved from earlier steps. Next, deploy it using the serverless deploy command.

Now we test our functionOne function. Go to AWS Lambda service and look for functionOne in Functions tab and click on its name to go to its detail page. On top right corner there is Test button, click on it, enter any name if asked for event name and hit Test. It should display execution result and there should be the console of the successful message sent as below.

Now if we go to SQS console, we should be seeing Messages available with the higher count.

At this point, we have successfully send a message to the queue. Now we work on functionTwo to receive this message from the queue and process it.

Section Three- SQS Receive and Process Message

First of all, let's add SQS event trigger for functionTwo. To do this we need to edit serverless.yml file under functionTwo section, we need to add the following after its handler.

events:
      - sqs: <your sqs queue arn number copied earlier>

Now the function should look like this.

Next, change the functionTwo to log all Records from the event object. You can replace functionTwo with the following code.

export const funtionTwo: SQSHandler = async (_event, _context) => {
  console.log("I am function two");
  for (let message of _event.Records) {
    console.log(`Handling message: ${message}`);
  }
}

All this functionTwo is doing is logging the message to console. You can change it to send an email or whatever you require the queue for. After this let's deploy it one last time.

As soon as your new functionTwo is deployed, it will start processing those messages in the queue and wait for new incoming messages.

You can also check logs for functionTwo. Mine is as below.

Overall, it is quite easy to create a new Amazon Simple Queue Service Queue, put message init and trigger another lambda function to pick those messages and process them.

You can find the complete project in GitHub

I would really appreciate your feedback.

Thank you for reading.

AWS Cognito and Twitter

Nabin Adhikari — Mon, 26 Aug 2019 05:58:28 +0000

Hi All,

I am in need of using AWS Cognito to support Email/Password, Facebook, Google, and Twitter Authentication with Dotnet Core 2.1 Web Api.

I am using JwtBearer Authentication and is automatically supported all but Twitter. Has anyone done similar projects before or anyone has related experience, I'd love to hear from you.

Thanks in advance,
Nabin

Setup Express API Test with Mocha and Chai using Typescript

Nabin Adhikari — Sat, 06 Jul 2019 02:31:28 +0000

Today, I'll be explaining how we can easily set up the testing environment
for the development of an Express API app. In this setup, we will be using Mocha and Chai with Typescript.

So, Let's get started.

First things first. Let's get our basic project setup. We will be using NPM here, but you can use Yarn as well (if you prefer it).

$ npm init -y

This should create a package.json file in your current working directory. Next, we install express, body-parser, and mongoose for creating a meaningful API using the following command.

npm i -S express body-parser mongoose

Now let's install all the dependencies only we care about (devDependencies).

npm i -D typescript chai mocha nodemon ts-node supertest

Let's install types for all our dependencies as well. Anyway, How good is typescript without types?

npm i -D @types/node @types/mocha @types/express @types/chai @types/supertest @types/mongoose

Setup Typescript

Now let's setup typescript. Just create tsconfig.json file at the root directory and copy the following config.

{
    "compilerOptions": {
        "target": "es2017",
        "module": "commonjs",
        "outDir": "dist",
        "sourceMap": true
    },
    "include": [
        "src/**/*.ts"
    ],
    "exclude": [
        "node_modules",
        "test/**/*.ts"
    ]
}

Test script

In our package.json file let's add a script test-dev to run it continuously every time we make any changes in our codebase. Simply put the following script in the scripts section of package.json.

"test-dev": "nodemon --watch . --ext ts --exec \"mocha -r ts-node/register test/**/*.test.ts\""

What this does is ask nodemon to watch all of the folders and run Mocha if it finds any file changes.

Now, let's create a test directory in the root directory and create index.test.ts inside it, then add the following scripts.

import {expect} from 'chai';
describe("Index Test", () => {
    it('should always pass', function () {
        expect(true).to.equal(true);
    });
});

Next, execute npm run test-dev to start the test. You should see one test suite passing. Congratulation, you have successfully created your test.

Now, let's add a couple of tests first to test our express app.

it('should POST /api/todo v2', async function () {
    const res = await request(app)
        .post('/api/todo').send({todo: "first todo"});
    expect(res.status).to.equal(200);
    expect(res.body).not.to.be.empty;
    expect(res.body.data).not.to.be.empty;
    expect(res.body.data).to.be.an("object");
    expect(res.body.error).to.be.empty;
});
it('should GET /api/todo', async function () {
    const res = await request(app).get('/api/todo');
    expect(res.status).to.equal(200);
    expect(res.body).not.to.be.empty;
    expect(res.body.data).not.to.be.empty;
    expect(res.body.data).to.be.an("array");
    expect(res.body.error).to.be.empty;
});

Curious where request and app comes from? Import them like this.

import app from '../src/app';
import {agent as request} from 'supertest';

These couple of tests are testing GET and POST for route /api/todo. We know we haven't created express yet, which we will create next. If you have tests running you should see a big error complaining about everything. We aim to fix those error by the end of this post.

Setup Express App

We create index.ts file inside src folder and add the following scripts.

import * as express from 'express';
import {Todo} from "./models";
import * as mongoose from 'mongoose';
import {json} from 'body-parser';

const app = express();
app.use(json());

if (mongoose.connection.readyState === 0) {
    mongoose.connect('mongodb://localhost:32768/test')
        .then(() => console.log('Db connected...'))
        .catch(e => console.log("Db connection error", e));
}
app.get('/api/todo', async (req, res) => {
    try {
        const todos = await Todo.find({});
        res.status(200).json({error: null, data: todos});
    } catch (e) {
        res.status(500).json({error: e.message, data: null});
    }
});
app.post('/api/todo', async (req, res) => {
    try {
        const todo = await new Todo({todo: req.body.todo}).save();
        res.status(200).json({error: null, data: todo});
    } catch (e) {
        res.status(500).json({error: e.message, todo: null});
    }
});
export default app;

It might look huge file to copy, but essentially all this is doing is creating an express with a single route /api/todo for GET and POST method to create a todo (POST) and return all of the todos (GET).

If you are curious where the Todo comes from, it is a mongoose model create in Todo.ts with following contents.

import * as mongoose from 'mongoose';
const Todo = mongoose.model('todo',
    new mongoose.Schema({
            todo: String,
            complete: {
                type: Boolean,
                default: false
            }
        }
    )
);
export default Todo;

This way, we can follow Test Driven Development (TDD) approach. I've written this post to help others get started quickly, as it took me quite a bit of time to figure this out. Please feel free to correct me if I've mistaken about anything.

Optional Chaining JavaScript / TypeScript

Nabin Adhikari — Wed, 11 Jul 2018 12:50:57 +0000

While working with JavaScript or Typescript, one of my most dreadful experience is reading property of object since there is no optional chaining (please correct me if I'm wrong). Obviously, we can have multiple if/else statement for safe property extraction, I think there should be a better clean approach.

Then, I started searching for such libraries and they work very well except they were lacking one most important feature (for me) to give fallback value. That is why I started thinking of creating npm module to solve this problem as my first npm module ever so that I could help other developers as well.

Today, I've created safechainjs, which exactly does what I wanted and I am very excited to publish my first ever npm modules. I'd be very honored for any feeback or suggestion here.

It is very easy to get started. For e.g.

$ npm install --save safechainjs

For a object like this:

const obj = {
    name: {
        firstName: "Albert",
        lastName: "Einstein"
    },
    dob: "1897/03/14",
    wife: ['Maric', 'Elsa']
};

We can easily and safely pull out properties using safechainjs like this.

const safechain = require('safechainjs');
const fname = safechain(obj, 'name', 'firstName', 'Unknown'); // Albert
const address = safechain(obj, 'address', 'postcode', 0); // 0
const wives = safechain(obj, 'wife', []); // ['Maric', 'Elsa']

Minimum three parameters are needed for function to sucessfully work.

First argument is the object to get property from
Multiple arguments to grab property in multiple level
Last argument is fallback value to be returned incase of missing request property

I'd really appreciate if you use it and provide any suggestions or feedback.
Happy JS
Nabin Adhikari