DEV Community: Lex Ned

Writing AWS lambdas with Architect

Lex Ned — Tue, 11 Aug 2020 07:48:53 +0000

It was one of those days when you ought to write just a lambda. The code and tests were easy to write. The infrastructure was needlessly complicated. After searching for a solution, I found the Architect project. It provides a framework for writing lambdas. You have to focus on the code while the infrastructure is automated by it.

I wanted to experiment with this technology, so I went with building a lambda that would send notifications when someone negatively mentioned me on Twitter. Not a Twitter user, but I found the concept fun. You can find the project at https://github.com/thelexned/tell-me-when-they-are-angry.

The idea is relatively simple. I ought to write a lambda that gets the latest twitter mentions for my Twitter account. Next, I should analyze the mentions for negative sentiments. When I encounter negative comments, I have to send a notification with these tweets.

I will not discuss here the lambda's code. I will concentrate on how Architect works and how I used it for building this project.

Architect
Installing the Architect CLI and the AWS SDK would be the first step for working with Architect.

npm install -g @architect/architect
npm install -g aws-sdk

I created the app.arc file.

@app
tell-me-when-they-are-angry

@scheduled
tweet-sentiment-analyzer rate(1 hour)

@tables
mentions
  mentionId *String

@events
they-are-angry

Here I'm defining the app name, two lambdas, and one DynamoDB table. The scheduled lambda runs every hour and contains the main logic. The events lambda is present for provisioning the notification infrastructure. Its output is the notification data. The Architect framework created it.

For working with Twitter's API, you need to create an application on their developer platform. Doing this will get you a set of tokens you can use for accessing the API. You can use the Twitter tokens for setting the environment variables for the lambda.

arc env [ENVIRONMENT] TWITTER_CONSUMER_KEY [VALUE]
arc env [ENVIRONMENT] TWITTER_CONSUMER_SECRET [VALUE]
arc env [ENVIRONMENT] TWITTER_ACCESS_TOKEN_KEY [VALUE]
arc env [ENVIRONMENT] TWITTER_ACCESS_TOKEN_SECRET [VALUE]

It generates a file called .arc-env. Make sure you don't version this file as it may contain sensitive data. Also, beware of writing to it directly. It will not work. You have to run the arc env commands for setting the environment variables.

The Twitter commands mentioned earlier have an ENVIRONMENT parameter and a VALUE parameter. The VALUE setting should get replaced by the values you have from Twitter. The ENVIRONMENT parameter can have one of three values: testing, staging, or production. The testing environment is used by the local Architect sandbox and the other two for deploying to AWS.

Assuming you have everything set up and your lambda code in place, you can deploy the app.

arc deploy [ENVIRONMENT]

The deployment is going to create CloudFormation templates for the infrastructure. It then uploads and provisions the CloudFormation stacks. If you are using the project I built, you can go to SNS after the deployment and subscribe to the topic created. Then wait for someone to mention you negatively on Twitter.

App structure
Architect apps have a predefined structure. They need an app.arc file for the app definitions and an src directory for the lambdas. You might also have a directory inside src called shared. Here you can add the code you want to share inside your lambdas. Also, place the package.json file inside the lambda directory for installing the required modules.

The project I built has the following structure:

.
├── app.arc
├── LICENSE.md
├── README.md
└── src
    ├── events
    │   └── they-are-angry
    │       └── index.js
    └── scheduled
        └── tweet-sentiment-analyzer
            ├── handler.js
            ├── handler.test.js
            ├── index.js
            ├── jest.config.js
            ├── jest.setup.js
            ├── mapper
            │   ├── tweets-mapper.js
            │   └── tweets-mapper.test.js
            ├── package.json
            ├── package-lock.json
            ├── repository
            │   ├── mentions-repository.js
            │   ├── mentions-repository.test.js
            │   ├── tweets-repository.js
            │   └── tweets-repository.test.js
            └── service
                ├── notifications-publisher.js
                ├── notifications-publisher.test.js
                ├── sentiment-analyzer.js
                └── sentiment-analyzer.test.js

You can notice that the tests are in the src directory. I'm not too fond of this, but I didn't find a way to add them somewhere else. I had this issue because I chose to mock the lambda's dependencies. If I used the sandbox, it would have worked. I didn't fancy that I couldn't assert anything on the sandbox, though.

Issues
During development, I struggled with an issue that prevented the deployment from working.

Unable to upload artifact ./src/scheduled/tweet-sentiment-analyzer referenced by CodeUri parameter of TweetSentimentAnalyzer resource.
ZIP does not support timestamps before 1980

It occurred because one of the dependencies in node_modules had the directory creation timestamp set to a timestamp that's before 1980. For fixing this, I had to add a hack to package.json.

"scripts": {
  "install": "find . -mtime +10950 -exec touch {} \\;",
}

The install script changes the timestamp for the files on the local directory when running npm install. The timestamps need to be fixed on every install because Architect runs the install command before deployments.

The 3 steps of publishing to npm

Lex Ned — Mon, 13 Jul 2020 20:12:37 +0000

A few days ago, I built a CLI for a key-value store. Now I had to share it with the world. For that, I used npm. And It was incredibly easy.

STEP 1
Register at https://www.npmjs.com/signup

STEP 2

npm login

STEP 3

npm publish

That's it. Now, don't forget to publish your module again after making changes. Also, don't forget to bump the version in package.json before publishing.

How to write a CLI in node.js

Lex Ned — Sun, 12 Jul 2020 06:38:17 +0000

I never created a CLI in node.js before. I wanted to build something useful but easy to implement. I don't remember how I came up with writing a CLI for a key-value store. It seemed this would be a great small project for learning.
Now that I knew what to do, I had to find a name for it. All I could come up with is key-value-persist. This name is uninspiring, but it does the job. It is descriptive enough. Maybe I should have added the "cli" suffix to it?

Starting out

I like it when I know what to do from the get-go. I feel this starts building momentum.

npm init -y

I now had the project initialized. Next, I had to investigate what node.js modules to use. It turns out that "commander" is one of the most used for building CLIs.

npm install --save commander

How about the key-value store? It was time to search on npm for a solution. That's how I found "keyv". It is a key-value store with a simple interface and multiple storage options. Exactly what I needed.

npm install --save keyv
npm install --save @keyv/sqlite

I decided to go with the SQLite storage for simplicity.
I also wanted to test the implementation, so I installed jest.

npm install --save-dev jest

Project structure

At first, I just had a file that contained simple logic.

const commander = require('commander');
const commandPackage = require('../package.json');
const Keyv = require('keyv');

commander
    .version(commandPackage.version)
    .description(commandPackage.description)
    .usage('[options]')
    .option('-n, --namespace <namespece>', 'add key value pair to namespace', 'local')
    .option('-s, --set <key> <value>', 'set value for key')
    .option('-g, --get <key>', 'get value for key')
    .option('-d, --delete <key>', 'delete key value pair')
;

commander.parse(process.argv);
const keyv = new Keyv(`sqlite://${__dirname}/data.sqlite`, {namespace: commander.namespace});
keyv.set('test', 'val').then(() => {
    keyv.get('test').then((val) => {
        console.log(val);
    });
});

As you can see, I did not integrate the data persisting with the CLI. I wanted to know if they worked on their own. I could figure out the integration later.
After verifying that these node.js modules can do the job, I wondered how to structure the project. I had two things to take care of: the CLI and the data persistence. That's how I came up with the directory structure of the project.

.
├── src
│   ├── command
│   └── data-persistence
└── test
    ├── command
    └── data-persistence

Building the CLI

Building the CLI was similar to what the "commander" documentation was describing. I only wrapped the functionality in a new object. You know, for when you want to change the node.js module responsible for the CLI.

const commander = require('commander');
const commandPackage = require('../../package.json');

function Command() {
    const command = new commander.Command()
    command
        .version(commandPackage.version)
        .description(commandPackage.description)
        .usage('[options]')
        .arguments('<key> <value>')
        .option('-s, --set <key> <value>', 'set value for key')
        .option('-g, --get <key>', 'get value for key')
        .option('-d, --delete <key>', 'delete key value pair')
    ;

    this.command = command;
}

Command.prototype.parse = function (args) {
    this.command.parse(args);
}

module.exports = {
    Command
}

I instantiated the "commander" in the constructor, defined the command options, and exposed a method for parsing the command arguments.

Then I had to create the data persister. I wrote methods for getting, setting, and deleting data.

const Keyv = require('keyv');

function Persister() {
    this.keyv = new Keyv(`sqlite://${__dirname}/../../data/data.sqlite`);
}

Persister.prototype.set = function(key, value) {
    return this.keyv.set(key, value);
}

Persister.prototype.get = function (key) {
    return this.keyv.get(key);
}

Persister.prototype.delete = function(key) {
    return this.keyv.delete(key);
}

module.exports = {
    Persister
}

Then I had to make the command work with the persister. I had to call the proper action in the persister given a command option.

const {Persister} = require('./data-persistence/persister');
const {Command} = require('./command/command');

const command = new Command();
const persister = new Persister();
command.parse(process.argv);

At this point, I did not have a way to find what option and what key-value pair I sent to the command. I had to add the missing methods to the command object.

Command.prototype.isGetCommand = function () {
    return !!this.command.get;
}

Command.prototype.isSetCommand = function () {
    return !!this.command.set;
}

Command.prototype.isDeleteCommand = function () {
    return !!this.command.delete;
}

Command.prototype.getKey = function () {
    if (this.isGetCommand()) {
        return this.command.get;
    }

    if (this.isSetCommand()) {
        return this.command.set;
    }

    if (this.isDeleteCommand()) {
        return this.command.delete;
    }

    throw new Error('The key is not defined');
}

Command.prototype.getValue = function () {
    return this.command.args.length !== 0 ? this.command.args[0] : "";
}

Next, I could add the logic that called the persister based on a command option.

if (command.isGetCommand()) {
    persister.get(command.getKey()).then((value) => {
        if (value) {
            process.stdout.write(`${value}\n`);
        }
    });
}

if (command.isSetCommand()) {
    persister.set(command.getKey(), command.getValue());
}

if (command.isDeleteCommand()) {
    persister.delete(command.getKey());
}

I had almost everything working. Next, I wanted to show the help information. It was for when the command options were not valid.

Command.prototype.isCommand = function () {
    return this.isGetCommand() ||
        this.isSetCommand() ||
        this.isDeleteCommand();
}

Command.prototype.showHelp = function () {
    this.command.help();
}

The main file was getting bigger. I did not like how it turned out. I decided to extract this functionality to a separate object. That's how I came up with the command-runner object.

function CommandRunner(command, persister) {
    this.command = command;
    this.persister = persister;
}

CommandRunner.prototype.run = function (args) {
    this.command.parse(args);

    if (!this.command.isCommand()) {
        this.command.showHelp();
    }

    if (this.command.isGetCommand()) {
        this.persister.get(this.command.getKey()).then((value) => {
            if (value) {
                process.stdout.write(`${value}\n`);
            }
        });
    }

    if (this.command.isSetCommand()) {
        this.persister.set(this.command.getKey(), this.command.getValue());
    }

    if (this.command.isDeleteCommand()) {
        this.persister.delete(this.command.getKey());
    }
}

module.exports = {
    CommandRunner
}

I'm passing the command and the persister to it. I took this decision for easier testing. It also permits changing the implementation for the command and persister objects without changing the integration part. Now my main file was simpler.

const {Persister} = require('./data-persistence/persister');
const {Command} = require('./command/command');
const {CommandRunner} = require('./command/command-runner');

const command = new Command();
const persister = new Persister();
const runner = new CommandRunner(command, persister);
runner.run(process.argv);

Testing

I decided to write unit tests only. I did not want to complicate things. I did not want to create a test database just for creating integration tests.
When writing tests, I had two issues. One was that the "commander" module was exiting the process on certain occasions. The other one was that I had to capture the command output. In both cases, I used jest spies.

const {Command} = require('../../src/command/command');

describe("Command", () => {
    describe("#parse", () => {

        test("parses valid options", () => {
            const consoleErrorSpy = jest.spyOn(console, 'error').mockImplementation();
            const command = new Command();
            command.parse(['-g', 'test-key']);
            expect(consoleErrorSpy).toHaveBeenCalledTimes(0);
        });

        test("exits with error on non existent option", () => {
            const consoleErrorSpy = jest.spyOn(console, 'error').mockImplementation();
            const processExitSpy = jest.spyOn(process, 'exit').mockImplementation();

            const command = new Command();
            command.parse([
                    'app',
                    'kvp',
                    '-b'
                ]
            );
            expect(consoleErrorSpy).toHaveBeenCalledTimes(1);
            expect(consoleErrorSpy).toHaveBeenCalledWith("error: unknown option '-b'");
            expect(processExitSpy).toHaveBeenCalledTimes(1);
            expect(processExitSpy).toHaveBeenCalledWith(1);
        });

        test("exits with error on non existent option argument", () => {
            const consoleErrorSpy = jest.spyOn(console, 'error').mockImplementation();
            const processExitSpy = jest.spyOn(process, 'exit').mockImplementation();

            const command = new Command();
            command.parse([
                    'app',
                    'kvp',
                    '-g'
                ]
            );
            expect(consoleErrorSpy).toHaveBeenCalledTimes(1);
            expect(consoleErrorSpy).toHaveBeenCalledWith("error: option '-g, --get <key>' argument missing");
            expect(processExitSpy).toHaveBeenCalledTimes(1);
            expect(processExitSpy).toHaveBeenCalledWith(1);
        });
    });
});

The remaining tests don't introduce new concepts. I will not present them here. You can check them out at https://github.com/thelexned/key-value-persist.

Installing the command globally

I wrote the app and the tests. Now I had to find a way to install it globally. It seems that npm has this functionality. But before installing it, I had to add a bin attribute to the package.json file. For this, I wrote a script that would execute the CLI's main file.

#!/usr/bin/env node
require('../src/index.js');

Then I added the bin attribute to package.json.

"bin": {
  "kvp": "./bin/kvp"
}

The only thing left was to install the CLI globally.

npm link

I could now run the CLI from anywhere.

kvp --help

TLDR

It might take you less time reading the code https://github.com/thelexned/key-value-persist.