Deploy Jupyter Notebook to AWS Lambda

🐙 GitHub

More often, I find myself opening Jupyter Notebook when facing a math or algorithmic problem. Maybe it is also true for you. At first, you think like: “I’ll make only the research/visualization part in a notebook and then move everything to plain python.” Yet after some time, you end up finishing the algorithm in Jupyter Notebook. A few years earlier, when there were no ”cloud lambdas,” you will end up moving code somewhere. Hopefully, these days, it is possible to deploy your function written in Jupiter Notebook in less than a minute.

AWS and Terraform

We will use Terraform to define infrastructure as code, so we end up making less DevOps routine. And of course, you need to have an AWS account.

Steps

1. Make lambda function ready to deploy.
2. Create infrastructure with AWS Lambda and API Gateway.
3. Test our function.
4. Create a bash script for deployment.

Function Ready for Deploy

At first, we need to add handler function to our notebook that will receive a request and return the result of calculations.

gist:167b2f197bbb56ccea3843da7dbdf68d

As we could see, the function parses JSON, run algorithm_on_steroids, and then return the result in JSON format. Next, we need to create a file with the name libs.txt. There we will add names of all third-party libs we use(numpy, matplotlib, pandas, …). Then we add a bash script that will create a .zip file ready to be deployed to lambda. What script does:

1. Create a directory.
2. Convert the notebook to the python file and move it to the directory.
3. Run pip install for each lib specified in libs.txt.
4. Zip folder.

AWS Lambda + API Gateway

To use AWS Lambda, we need to create infrastructure. It includes:

1. S3 to store the function.
2. Lambda that will run the function.
3. API Gateway to communicate with the function.

For simplicity, we will omit best practices and put everything in one terraform file. But before we could run it, we need to specify credentials in the file. Special attention to this piece of the file:

gist:5772a3d448c7b3de399acb3b9c539605

Create Lambda and Test It!

Now we could open a terminal in the directory with the lambda and all files mentioned earlier.

gist:4a4cd7a87b6325c2deda24a0de5d17a4

Isn't it magic? Three commands and we have deployed function.After running terraform apply at the end of the output, we will see a green line with the URL. It is the URL we could use to run the function. Let’s test it by making a POST request.

```shell{promptUser: radzion}
curl --request POST --data '{"a": 3, "b": 4}' /function

OK, but what if we make changes to the function and want to see a new version deployed? Let’s write a script for deployment.

`gist:3c9af5d6a4cdaea8f52712f4eac21be3`

Now we could run it by typing:



```shell{promptUser: radzion}
. ./deploy.sh tf-lambda tf-lambdas function.zip

Conclusion

In this post, we've made the automation tool for deploying the Jupyter Notebook function. It is not ideal. For example, it would be good to add a script that will automatically find third-party libs in the notebook and add them to libs.txt, but this is out of the scope of this post. To delete everything we made from AWS, we can run terraform destroy. And poof — everything deleted from AWS. I hope this post was useful to you!