Using Docker on Lambda for Postgres to S3 Backups

Intro

If you don't care about any context you can just skip down to the Ship it section.

I've never written a tech blog post before, but I figured out how to leverage AWS Lambda's newly announced container image support (Dec 1, 2020) to back up a database I'm maintaining and it wasn't as straightforward as I'd hoped, so I figured I'd write something on it.

For context, as a cost-cutting measure™, I have just a single EC2 instance with some docker containers running my application, and that same EC2 instance houses my database! Sorry if you hate it!

If you also didn't feel like dealing with the costs and complexities of RDS & snapshotting and just want a nice way to back up your data, you're in the right place!

My reasons

1. I have 2- count 'em- 2 users for an app I made and I want to make sure their data is less ephemeral than the reliability of a database-on-EC2-instance setup would suggest. I imagine this scales to databases much larger than what I have, but I haven't tested it out personally. I suppose you're up against the max timeout (15 minutes), and the max memory capacity (10GB) of a Lambda execution.
2. It's super cheap- for S3, all you're paying for is the storage, since data transfer into S3 is free, and you shouldn't be exporting from S3 very often. And of course Lambda is very cheap- I've calculated that for my setup, backing up an admittedly very small amount of data once every hour, it will cost approximately $0.05/mo.
3. It's super configurable- you can snapshot as frequently or infrequently as you'd like, and I guess if you wanted to you could backup only certain tables or something- I don't know I'm not super familiar with RDS' snapshotting capabilities, maybe they're similar.
4. I can pull my production data down for local testing/manipulating very easily! It's just a single docker command!

High-Level

1. For the actual database backups I adapted a Docker-based script I found on GitHub here.
2. For running that script I'm using AWS Lambda's new "Container Image" option. I borrowed the Dockerfile from the Building a Custom Image for Python section of Amazon's announcement to help configure things in a way that'd make sense.
3. For triggering the Lambda on a cron I'm using Amazon EventBridge. I've never heard of it before but it was really easy to create a rule that just says, "run this Lambda once an hour," so I'm recommending it.
4. I'm storing the SQL dump files in a private S3 bucket with a retention policy of 14 days.

Let's go lower now

So in my head I was like, "Oh cool I can just chuck this Dockerfile on a Lambda and run a command on an image with some environment variables once an hour and we're golden."

It doesn't really work that way though.

Lambda requires that your Docker image's entrypoint be a function, in some programming language, that gets called when triggered.

So rather than just being able to trigger a docker run command (which could in my case run a script, backup.sh) like so:

docker run schickling/postgres-backup-s3

It's more that you're setting up a Docker environment for a program (in my case a Python program), that'll have an entrypoint function, that'll run whatever you need to run (again, in my case backup.sh).

What's that entrypoint function look like? Pretty simple:

import json
import subprocess

def handler(event, context):
    print("Received event: " + json.dumps(event, indent=2))
    subprocess.run("sh backup.sh".split(" "))
    print("Process complete.")
    return 0

The mangling of the Dockerfile they provide as an example in the Building a Custom Image for Python section of Amazon's announcement, to look more like the Dockerfile of the Postgres to S3 backup script was the more complicated part. I'll let you take a look at that what that final Dockerfile looks like here.

Some other gotchas

AWS Environment variables

By far the most annoying part of this whole thing was finding some GitHub issue comment that mentioned that Lambdas automatically set the AWS_SESSION_TOKEN and AWS_SECURITY_TOKEN environment variables, and it turns out it was causing a hard-to-track-down error in the backup script's invocation of the aws client along the lines of:

An error occurred (InvalidToken) when calling the PutObject operation: The provided token is malformed or otherwise invalid.

If all this article accomplishes is that someone stumbles upon this section after Googling that error, I will consider it astoudingly successful.

Anyway, I just had to edit the backup.sh file to add these two lines and the complaining stopped:

unset AWS_SECURITY_TOKEN
unset AWS_SESSION_TOKEN

Writing to files

For some reason Lambda didn't like that the backup.sh script was writing to a file. After a couple minutes of researching with no luck, I decided to change:

echo "Creating dump of ${POSTGRES_DATABASE} database from ${POSTGRES_HOST}..."
pg_dump $POSTGRES_HOST_OPTS $POSTGRES_DATABASE | gzip > dump.sql.gz

echo "Uploading dump to $S3_BUCKET"
cat dump.sql.gz | aws $AWS_ARGS s3 cp - s3://$S3_BUCKET/$S3_PREFIX/${POSTGRES_DATABASE}_$(date +"%Y-%m-%dT%H:%M:%SZ").sql.gz || exit 2

to:

echo "Creating dump of ${POSTGRES_DATABASE} database from ${POSTGRES_HOST} and uploading dump to ${S3_BUCKET}..."

pg_dump $POSTGRES_HOST_OPTS $POSTGRES_DATABASE | gzip | aws $AWS_ARGS s3 cp - s3://$S3_BUCKET/$S3_PREFIX/${POSTGRES_DATABASE}_$(date +"%Y-%m-%dT%H:%M:%SZ").sql.gz || exit 2

There might be a better way around this but I couldn't find one, so here we are, just piping away.

Lambda timeouts

The Lamba I had was timing out by default after 3 seconds. Make sure you jack that up in the function configuration's General Configuration.

Test it out locally

Testing this out locally is really easy because the example Dockerfile that Amazon provided in their announcement has "Lambda Runtime Interface Emulator" support built in.

Build the image:

docker build -t db-backup -f db-backup.Dockerfile .

Run it in Terminal Window 1:

docker run \
    -e POSTGRES_DATABASE=<POSTGRES_DATABASE>ms
    -e POSTGRES_HOST=<POSTGRES_HOST> \
    -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> \
    -e POSTGRES_USER=<POSTGRES_USER> \
    -e S3_ACCESS_KEY_ID=<S3_ACCESS_KEY_ID> \
    -e S3_BUCKET=<S3_BUCKET> \
    -e S3_REGION=<S3_REGION> \
    -e S3_PREFIX=<S3_PREFIX> \
    -e S3_SECRET_ACCESS_KEY=<S3_SECRET_ACCESS_KEY> \
    -p 9000:8080 \
    db-backup:latest

Trigger it in Terminal Window 2:

curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

Ship it

Sure, the right™ way to ship it can be debated by whoever, but the simplest way is probably:

1. Clone the code:
   

   git clone https://github.com/cd17822/lambda-s3-pg-backup.git
   

2. Build the Docker image:
   

   docker build -t db-backup -f db-backup.Dockerfile
   

3. Tag the built image to be pushed to a private ECR repository:
   

   docker tag db-backup <AWS_ACCOUNT_ID>.dkr.ecr.<ECR_REGION>.amazonaws.com/<ECR_REPO_NAME>:latest
   

4. Push the image up to ECR:
   

   docker push <AWS_ACCOUNT_ID>.dkr.ecr.<ECR_REGION>.amazonaws.com/<ECR_REPO_NAME>:latest
   

5. Create a private S3 bucket that we'll be storing the backups in (I'd also recommend setting up a retention policy unless you want to keep around these files forever).

6. Create a Lambda function by selecting Create Function in the Lambda Console.

7. Select Container Image, name it whatever you want, find the Docker image in ECR in Browse Images, leave everything else as default and finally select Create Function.

8. Scroll down to Environment variables and set values for the following environment variables:
   

   POSTGRES_DATABASE
   POSTGRES_HOST
   POSTGRES_PASSWORD
   POSTGRES_USER
   S3_ACCESS_KEY_ID
   S3_BUCKET
   S3_PREFIX
   S3_SECRET_ACCESS_KEY
   

9. Scroll down further and make sure you edit the General Configuration such that the Timeout is bumped up to something like 5 minutes.

10. At this point you can select Test on the top right and check to make sure your function's working.

11. Finally, you can set up a scheduled trigger by selecting Add trigger in the Designer. I'd recommend setting up a simple EventBridge trigger that runs on a cron (cron(13 * * * *)) or with a set frequency (rate(1 hour)).

Restoring from backup

You could set up a Lambda to restore your database from backup that's triggered by emailing a photo of yourself crying to an unsolicited email address using AWS Computer Vision, but for the sake of this article I figured I'd just include the easy way to do it.

In the same repo that the backup script is in lies a restore script. It's hosted on DockerHub making it really easy to pull and run locally:

docker pull schickling/postgres-restore-s3
docker run -e S3_ACCESS_KEY_ID=key -e S3_SECRET_ACCESS_KEY=secret -e S3_BUCKET=my-bucket -e S3_PREFIX=backup -e POSTGRES_DATABASE=dbname -e POSTGRES_USER=user -e POSTGRES_PASSWORD=password -e POSTGRES_HOST=localhost schickling/postgres-restore-s3