DEV Community: Tathagata

Moving your cron jobs to AWS lambda with SAM cli

Tathagata — Sun, 31 Mar 2019 02:35:59 +0000

Every now and then, you need to a script to run at regular interval and do something interesting. Before the advent of serverless computing, cron jobs were the defacto way of doing this. But now we have several options that are more cost effective. ECS scheduled tasks and lambdas are two serverless alternatives for task scheduling in AWS. Depending on your workload you can decide which option would be the right choice for you. Lambdas, with 15-minute runtime limit are ideal for small workloads. Let's explore how we can get this working.

Install SAM cli

AWS Sam cli is a tool with an adorable chipmunk as mascot. It simplifies building and deploying lambdas.

pip install --user aws-sam-cli

Create a project

sam init --name lambda-cron --runtime python3.7
[+] Initializing project structure...

Project generated: ./lambda-cron

Steps you can take next within the project folder
===================================================
[*] Invoke Function: sam local invoke HelloWorldFunction --event event.json
[*] Start API Gateway locally: sam local start-api

Read lambda-cron/README.md for further instructions

[*] Project initialization is now complete

Lets take a look at the generated code

tree lambda-cron/
lambda-cron/
|-- README.md
|-- event.json
|-- hello_world
|   |-- __init__.py
|   |-- __pycache__
|   |   |-- __init__.cpython-37.pyc
|   |   `-- app.cpython-37.pyc
|   |-- app.py
|   `-- requirements.txt
|-- template.yaml
`-- tests
    `-- unit
        |-- __init__.py
        |-- __pycache__
        |   |-- __init__.cpython-37.pyc
        |   `-- test_handler.cpython-37.pyc
        `-- test_handler.py

5 directories, 12 files

This might seem an overkill of artifacts compared to a line in your cron tab. But if you think you are getting rid of the entire server, this doesn't look that unreasonable. Let's take a look at the non-obvious files.

event.json

sam's scaffolding is targets the most popular lambda use case - a function servicing an API gateway endpoint. We can ignore this.

app.py

This file is of primary importance, as this is where your code goes. By default, lambda_handler is the name of the function which you populate. For now, we will just limit it to a printing a string.

def lambda_handler(event, context):
    print("Do something interesting!")

requirements.txt

Standard stuff - your package dependencies go here.

template.yaml

This file is responsible for orchestrating the serverless-ness. It declaratively specifies what components make up your stack, where the stack is your entire software architecture deployed as one unit in AWS CloudFromation. If this is not magic, I don't know what is. I am a big fan of CloudFormation.

Note this file was generated to build a lambda function that gets triggered by an HTTP request to the API gateway endpoint. But our goal is to run it at scheduled intervals. Take a look at the section which describes the events.

      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

We need to replace it with the schedule information. Sam template supports schedule expression in both cron syntax and a simpler rate expression.

Every minute using rate

      Events:
        HelloWorld:
          Type: Schedule
          Properties:
            Schedule: rate(1 minute)

Every minute using cron

      Events:
        HelloWorld:
          Type: Schedule
          Properties:
            Schedule: cron(1 * * * ? *)

test_handler.py

This is pre-generated unittest stubs for pytest. You can install pytest (pip install pytest) and run pytest like usual.

Build

With your tests passing, it is now time to do the builds

cd lambda-cron/
t lambda-cron $ sam build --use-container
2019-03-30 19:42:24 Starting Build inside a container
2019-03-30 19:42:24 Building resource 'HelloWorldFunction'

Fetching lambci/lambda:build-python3.7 Docker container image......
2019-03-30 19:42:25 Mounting /Users/t/projects/python/playground/lambda-deep-dive/lambda-cron/hello_world as /tmp/samcli/source:ro inside runtime container

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Note the first time you run this, it will download the docker image it needs for running.

Local testing

One of the best functionality that sam offers is the ability to test your code locally in a docker container. This makes quick iteration possible without having to acutally deploy on AWS lambda environment.

sam local invoke --no-event -t template.yaml
2019-03-30 21:01:43 Found credentials in shared credentials file: ~/.aws/credentials
2019-03-30 21:01:43 Invoking app.lambda_handler (python3.7)

Fetching lambci/lambda:python3.7 Docker container image....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2019-03-30 21:02:53 Mounting /Users/t/projects/python/playground/lambda-deep-dive/lambda-cron/hello_world as /var/task:ro inside runtime container
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
Do something intersting!
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72
REPORT RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72  Duration: 16.52 ms  Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 22 MB
null

Package your code

Let's create a bucket and pass it to the packaging command. Note we are passing a file called packaged.yaml which is the generated cloud formation template that we will be using in the next command to deploy it.

aws s3 mb s3://lambda-cron-bucket
sam package --s3-bucket lambda-cron-bucket --output-template-file packaged.yaml

Deploy

Finally deploy your cron job. You'll need to provide a name for your stack and tell cloudformation explicitly that you are ok with the creation of IAM related resources by passing the CAPABILITY_IAM flag.

aws cloudformation deploy --template-file /absolute/path/packaged.yaml --stack-name lambda-cron --capabilities CAPABILITY_IAM

Validate

Lets take a look if our cron is executing every minute.

sam logs -n HelloWorldFunction --stack-name lambda-cron --tail

You should see something like the following:

2019-03-30 20:35:19 Found credentials in shared credentials file: ~/.aws/credentials
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.613000 START RequestId: c2c8a898-dfab-450f-944b-d43d77741c35 Version: $LATEST
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.618000 Do something intersting!
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.638000 END RequestId: c2c8a898-dfab-450f-944b-d43d77741c35
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.638000 REPORT RequestId: c2c8a898-dfab-450f-944b-d43d77741c35  Duration: 24.83 ms  Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 63 MB
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.272000 START RequestId: 93645e34-850b-47b0-a4a3-00d10e502057 Version: $LATEST
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.278000 Do something intersting!
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.298000 END 
...

Congratulations on your first serverless cron job!

What's with the cover picture?

Keeping with the theme of my other dev.to posts, it has nothing to do with the post. It's view of our beautiful city, Chicago, from my balcony.

Raspberry Pi on your TV

Tathagata — Sat, 09 Feb 2019 17:42:35 +0000

I had a raspberry pi that was gathering dust for a few months. Earlier last month I hooked it up to a usb keyboard, mouse and a monitor and got the WiFi set up. But with all the messy wires and the dedicated real estate for the screen, it defeated the purpose of being a pocket size computer.
Last weekend I finally got it to a setup that I am happy with. Optional bluetooth keyboard and mouse, so no messy wires.

Using the TV as a monitor

The idiot box is a vice of mine that sucks up all the productive time. Since it has a dedicated place already in my living room already, plugged it to the TV. So fun that I have a linux machine hidden behind all the trinkets on TV cabinet!

A small settings change was needed so that I can instantly switch over to the pi when I turn on my TV or switch over from other HDMI inputs. Edit the lightdm configuration /etc/lightdm/lightdm.conf as root and uncomment the line xserver-command=X to xserver-command=X -s 0 dpms. Save and reboot.

Phone Apps to replace mouse and keyboard

Terminal emulator

Termius - A beautiful terminal emulator that is fast, feature packed and best of all - free for my purposes. Just install, connect to the raspberry pi’s up address and boom you are on the shell. It amazes me how beautifully the ncurses screens renders on Termius!

Side note: Fing is a nice little app to find the up addresses of devices on your network.

GUI access with Phone/Ipad

Install VNC Viewer on your iPhone or iPad. On the pi, install realvnc.

sudo apt-get update 
sudo apt-get install realvnc-vnc-server
reboot

After rebooting sudo raspi-config > Advanced Options > VNC Enable. 👍

That’s it! You a running Linux desktop on your TV that you can use with your phone or tablet. So theoretically, I can go from Netflix n chill to a linux desktop, with the flick of a button on the remote.

(Optional) Bluetooth Keyboard & Mouse

While realVNC works better than I expected from the free app, it wouldn't be bad to throw in a bluetooth keyboard and mouse to the mix. My Magic Keyboard and Mouse are small and wireless - would probably be useful at times.

sudo bluetoothctl
scan on
connect ma:ca:dd:re:ss:00
pair ma:ca:dd:re:ss:00 
remove ma:ca:dd:re:ss:00

Tip: Here's how to find the mac address of your bluetooth device, if you have them connected to your mac: click on the apple menu , hold down option key, click on System information and click on Bluetooth.

Wait. What does the cover picture have to do with this post?

Not a single thing. This is a picture I took at Sorrento. ❤️ Italy, can't wait to go back!

Ecto drop Postgres database on AWS RDS

Tathagata — Thu, 24 Jan 2019 06:59:03 +0000

Lets say during your development cycles, you have screwed your postgres database and the easiest way to get out of the mess is to drop the database and restart fresh. In this example we will look at a Phoenix project that uses a Postgres database.

Warning

To make it clear, dropping the database is a bad idea if you haven't taken snapshots. Dropping deletes all data; so do it only in non-prod.

Problem: Dropping a database fails due to immortal rdsadmin

mix do ecto.drop, ecto.create, ecto.migrate

This nicely works on your local postgres instance.
However, if the database is on AWS RDS, you'll be greeted by this interesting message.

$ mix ecto.drop
** (Mix) The database for SuperPetStore.Repo couldn't be dropped, reason given: ERROR:  55006: database "petdb" is being accessed by other users
DETAIL:  There is 1 other session using the database.
LOCATION:  dropdb, dbcommands.c:840

If you look at the number of connections

> select * from pg_stat_activity;

Sure enough, there will be one persistent connection from rdsadmin. It is AWS RDS user that takes care of backup, updates and maintenance. This connection gets in the way of dropping the db and starting afresh.

> use postgres;
You are now connected to database "postgres" as user "petuser"
Time: 0.078s
postgres> drop database petdb;
You're about to run a destructive command.
Do you want to proceed? (y/n): y
Your call!
database "petdb" is being accessed by other users
DETAIL:  There is 1 other session using the database.

Time: 5.031s (5 seconds), executed in: 5.030s (5 seconds)
postgres>

pg_terminate_backend

select pg_terminate_backend(pid) from pg_stat_activity where datname='petdb';drop database pipeops;
You're about to run a destructive command.
Do you want to proceed? (y/n): y
Your call!
+------------------------+
| pg_terminate_backend   |
|------------------------|
| True                   |
+------------------------+
SELECT 1
DROP DATABASE
Time: 0.118s

First pg_terminate_backend allows you to terminate the rdsadmin account and shoving in the drop statement in the same line gets the job done before rdsadmin spawns up and connects to petdb.

With the database gone, you can now run ecto to create the db afresh!

Wait. What does the picture above have to do with this post?

Not a single thing. This is a picture I took at Siena. ❤️ Italy, can't wait to go back!

Docker cross repository push

Tathagata — Thu, 24 Jan 2019 05:38:10 +0000

I just found dev.to and immediately fell in love with the retro 🤣 look! Instead of putting in much thought, I started brain dumping - mainly about things that did not work on the first go. No effort has been made to be helpful - typos, errors, misinformation may lay ahead. Welcome to the Internet!

Problem: Need to push a docker image from one AWS account to another

ECR is the docker repository as a service provided by AWS. To have better separation of your environment, you would typically have a prod account and one or more non-prod accounts. Your CI pipeline would be running in your non-prod account allowing you to rapidly iterate and build new images. Once your testing cycle completes, you would want to promote this image to the production account. Below are the set of commands needed to promote your release candidate from non-prod to production.

Login to your docker registry of source account

aws ecr --profile dev --region us-east-1 get-login --no-include-email | bash

Pretty straight forward. Make a cli call which returns the docker login command which is piped to execute in bash. dev is the aws profile for the account which contains the docker image.

CI produced docker image

docker pull 123456789012.dkr.ecr.us-east-1.amazonaws.com/img:aa567e02ae7477a6b022da30fa50137a252dd143

In 123456789012.dkr.ecr.us-east-1.amazonaws.com, 123456789012 stands for the account number of aws account where our docker image lives.
Lets say you have done your testing and you are now satisfied to promote this to prod.

Switch repository and try to push

aws ecr --profile prod --region eu-west-1 get-login --no-include-email | bash

Try to push

docker push 234567890121.dkr.ecr.us-east-1.amazonaws.com/img:ab77ad7b362d7552bc2c7a69f2650a5b50bcdcc1

And ...

The push refers to repository [234567890121.dkr.ecr.us-east-1.amazonaws.com/img]
tag does not exist: 234567890121.dkr.ecr.us-east-1.amazonaws.com/img:aa567e02ae7477a6b022da30fa50137a252dd143

Uh oh forgot to tag

This was not very obvious to me. But docker, awesome as it is, gives you the nice error message above telling exactly where things are going wrong. To push to the prod registry, you'll need to tag the image accordingly.

docker tag 123456789012.dkr.ecr.us-east-1.amazonaws.com/img:aa567e02ae7477a6b022da30fa50137a252dd143 234567890121.dkr.ecr.us-east-1.amazonaws.com/img:aa567e02ae7477a6b022da30fa50137a252dd143

Push away

Finally push the image and you are done.

docker push 234567890121.dkr.ecr.us-east-1.amazonaws.com/img:ab77ad7b362d7552bc2c7a69f2650a5b50bcdcc1

Congratulations, you have promoted the image!

Wait. What does the picture above got to do with this post?

Not a single thing. This is a picture I took at Siena. ❤️Italy, can't wait to go back!