DEV Community: Vadym Khodak

Creating AWS resources using Boto3 for deploying Django project

Vadym Khodak — Sun, 12 Sep 2021 13:57:02 +0000

This is a bonus blog post to Django with AWS Lambda series. If you are a Python developer and you need to create AWS resources but you don't want to learn Terraform or use AWS Management Console this blog post is for you.

Boto3 Introduction

We will use Boto3 - the AWS SDK for Python. Boto3 makes it easy to integrate your Python application, library, or script with AWS services including Amazon S3, Amazon EC2, Amazon DynamoDB, and more.

Boto3 has two levels of APIs:

Client (or "low-level") APIs provide one-to-one mappings to the underlying HTTP API operations.
Resource APIs hide explicit network calls but instead provide resource objects and collections to access attributes and perform actions.

Let's start codding

First, we need to install boto3 python library:

pip install boto3

I used version 1.18.40 of boto3 which was the latest version on the day of writing this post. It is always better to use the latest version of boto3 as the AWS team is actively working on this library.

Second (optional), we can install python-dotenv python library to read environment variables from .env file:

pip install python-dotenv

This step is optional, but I do recommend using environment variables for any sensitive or configurable parameters like passwords, AWS credentials, and such.

Third (if you decided to use environment variables), we should create .env file with the following variables:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION_NAME=

DEFAULT_VPC_ID=
DB_INSTANCE_IDENTIFIER=
RDS_DB_NAME=
RDS_USERNAME=
RDS_PASSWORD=
S3_BUCKET_NAME=

You can find your DEFAULT_VPC_ID using AWS Management Console

Fourth, we need to create Python file where we will write our code to create all the necessary AWS resources for deploying a Django project on AWS Lambda.

touch django_aws_resources.py

Using Boto3 - the AWS SDK for Python

First, we need to import all libraries we are going to use, load the environment variables, and set up some helpful variables.

import json # to dump Python object with S3 bucket policy to JSON string
import os  # to get the necessary environment variables  
import time  # to wait until AWS RDS instance will be created
from datetime import datetime  # to generate a unique timestamp for unique CallerReference

import boto3
from dotenv import load_dotenv

load_dotenv()

S3_BUCKET_NAME = os.environ["S3_BUCKET_NAME"]
REGION_NAME = os.getenv("AWS_REGION_NAME") or "us-east-1"

Second, we should create a SecurityGroup and add inbound and outbound rules:

ec2_resource = boto3.resource("ec2", region_name=REGION_NAME)

security_group = ec2_resource.create_security_group(
    Description="sg-for-lambdas",
    GroupName="django-rds-security-group",
    VpcId=os.environ["DEFAULT_VPC_ID"],
    TagSpecifications=[
        {
            "ResourceType": "security-group",
            "Tags": [
                {"Key": "Name", "Value": "django-demo-rds-security-group"},
            ],
        },
    ],
    DryRun=False,
)

_ = security_group.authorize_egress(
    DryRun=False,
    IpPermissions=[
        {
            "FromPort": 0,
            "IpProtocol": "-1",
            "IpRanges": [],
            "Ipv6Ranges": [
                {"CidrIpv6": "::/0", "Description": "allow all (demo only)"},
            ],
            "PrefixListIds": [],
            "ToPort": 0,
        },
    ],
    TagSpecifications=[
        {
            "ResourceType": "security-group-rule",
            "Tags": [
                {"Key": "Name", "Value": "egress rule"},
            ],
        },
    ],
)

_ = security_group.authorize_ingress(
    DryRun=False,
    IpPermissions=[
        {
            "FromPort": 0,
            "IpProtocol": "-1",
            "IpRanges": [
                {"CidrIp": "0.0.0.0/0", "Description": "allow all (demo only)"},
            ],
            "Ipv6Ranges": [
                {"CidrIpv6": "::/0", "Description": "allow all (demo only)"},
            ],
            "PrefixListIds": [],
            "ToPort": 0,
        },
    ],
    TagSpecifications=[
        {
            "ResourceType": "security-group-rule",
            "Tags": [
                {"Key": "Name", "Value": "ingress rule"},
            ],
        },
    ],
)

Third, we need to create an RDS instance with Postgres engine:

rds_client = boto3.client("rds", region_name=REGION_NAME)
_ = rds_client.create_db_instance(
    DBName=os.environ["RDS_DB_NAME"],
    DBInstanceIdentifier=os.environ["DB_INSTANCE_IDENTIFIER"],
    AllocatedStorage=20,
    DBInstanceClass="db.t2.micro",
    Engine="postgres",
    EngineVersion="12.5",
    MasterUsername=os.environ["RDS_USERNAME"],
    MasterUserPassword=os.environ["RDS_PASSWORD"],
    VpcSecurityGroupIds=[security_group.id],
    Tags=[{"Key": "name", "Value": "django_demo_rds"}],
)

Fourth, we should create an S3 bucket for static files:

s3_client = boto3.client("s3", region_name=REGION_NAME)
_ = s3_client.create_bucket(
    ACL="private",
    Bucket=S3_BUCKET_NAME,
)

Fifth, we need to create a CloudFront Origin Access Identity and update the S3 bucket policy to allow a CloudFront distribution serving static files from the bucket:

cloudfront_client = boto3.client("cloudfront", region_name=REGION_NAME)

response = cloudfront_client.create_cloud_front_origin_access_identity(
    CloudFrontOriginAccessIdentityConfig={
        "CallerReference": str(datetime.utcnow().timestamp()),
        "Comment": f'access-identity-{S3_BUCKET_NAME}.s3.amazonaws.com"',
    }
)
origin_access_identity_id = response["CloudFrontOriginAccessIdentity"]["Id"]

s3_resource = boto3.resource("s3")
bucket_policy = s3_resource.BucketPolicy(S3_BUCKET_NAME)
_ = bucket_policy.put(
    Policy=json.dumps(
        {
            "Version": "2008-10-17",
            "Statement": [
                {
                    "Sid": "1",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": f"arn:aws:iam::cloudfront:user/CloudFront "
                        f"Origin Access Identity {origin_access_identity_id}",
                    },
                    "Action": "s3:GetObject",
                    "Resource": f"arn:aws:s3:::{S3_BUCKET_NAME}/*",
                }
            ],
        }
    ),
)

Sixth, we need to create a CloudFront Distribution to serve static files from the S3 bucket

cloudfront_distribution_response = cloudfront_client.create_distribution(
    DistributionConfig={
        "CallerReference": str(datetime.utcnow().timestamp()),
        "Origins": {
            "Quantity": 1,
            "Items": [
                {
                    "Id": S3_BUCKET_NAME,
                    "DomainName": f"{S3_BUCKET_NAME}.s3.amazonaws.com",
                    "S3OriginConfig": {
                        "OriginAccessIdentity": f"origin-access-identity/cloudfront/{origin_access_identity_id}"
                    },
                },
            ],
        },
        "Restrictions": {"GeoRestriction": {"RestrictionType": "none", "Quantity": 0}},
        "ViewerCertificate": {
            "CloudFrontDefaultCertificate": True,
        },
        "DefaultCacheBehavior": {
            "TargetOriginId": S3_BUCKET_NAME,
            "Compress": True,
            "ViewerProtocolPolicy": "allow-all",
            "AllowedMethods": {
                "Quantity": 3,
                "Items": ["GET", "HEAD", "OPTIONS"],
                "CachedMethods": {
                    "Quantity": 2,
                    "Items": ["GET", "HEAD"],
                },
            },
            "ForwardedValues": {
                "QueryString": False,
                "Cookies": {
                    "Forward": "none",
                },
            },
            "MinTTL": 0,
            "DefaultTTL": 3600,
            "MaxTTL": 86400,
        },
        "Enabled": True,
        "IsIPV6Enabled": True,
        "DefaultRootObject": "index.html",
        "Comment": "Django React static distribution",
    }
)

Then, we can wait until the RDS instance will be created to get the Database Host Name for Django project configurations:

while True:
    rds_db_instances = rds_client.describe_db_instances(
        DBInstanceIdentifier="string",
        Filters=[
            {"Name": "db-instance-id", "Values": [os.environ["DB_INSTANCE_IDENTIFIER"]]},
        ],
        MaxRecords=20,
    )

    if rds_db_instances["DBInstances"][0].get("Endpoint"):
        db_host_name = rds_db_instances["DBInstances"][0]["Endpoint"]["Address"]
        break
    time.sleep(100)

Finally, we can print Database Host Name and CloudFront Distribution Domain Name to use them in Django project configurations:

print(f"Database Host Name: {db_host_name}")
print(f"CloudFront Domain Name: {cloudfront_distribution_response['Distribution']['DomainName']}")

Note, that this is an example of configuring AWS resources. Your production configuration can be different.

Final Words

Today we saw one more way of how to prepare AWS infrastructure for a Django project. You can find a full version of the code showed here in this GitHub repository.

Don't forget to follow me on Twitter @vadim_khodak or on LinkedIn so you do not miss the next posts.

Deploying a Django project on AWS Lambda using Serverless (Part 4)

Vadym Khodak — Thu, 19 Aug 2021 08:28:16 +0000

As I promised in my previous blog post Deploying a Django project on AWS Lambda using Serverless (Part 3), I'd like to show you how to add a React.JS client to a Django project and deploy it with Django on AWS Lambda using Serverless.

BLUF

Django framework allows you to build a client using Django templates, but there are a lot of cases when this is not enough. Business requirements for the client-side of your application could require adding more complex logic on the client-side. In these cases, we will not be able to solve business problems without a JavaScript web framework (React.JS, Vue.JS, Angular, etc). I'd like to show you how to build a simple React.JS client and integrate it with a Django project using axios library on the client-side and Django REST Framework on the server-side.

With this approach, I will build a React.JS client with a domain name of CloudFront distribution as a PUBLIC_URL and store it on AWS S3 bucket together with Django static files. Then, I add the built index.html file with React.JS app to Django templates folder and deploy it with the Django project on AWS Lambda.

Getting started

I've already described how to create a Django project and deploy it on AWS Lambda using Serverless in my first blog post of this series. I will use that project to get started.

Let's go through this process step by step:

Clone the Django repository I used in the first part of these series and go to the cloned repository:

git clone https://github.com/vadym-khodak/django-aws-lambda
cd django-aws-lambda

Follow instructions from this blog post to run the Django server (install requirements, configure environment variables using .env, apply migrations, create a superuser, collect static, run the server).

If everything works, you are ready to start working on the client.

Add React.JS client

In this part, I'll show how to create a simple React.JS client and integrate it with our Django project, but I'm sure that you can easily use Vue.JS (I'm not familiar with Angular) as steps pretty much the same.

There are many options to create React.JS client. I'm using react-scripts in this example.

Install react-scripts using npm (Node package manager)

npm install -g react-scripts

Use react-scripts to initialize a React.JS client

npm init react-app client

Check that React.JS client was built successfully

cd client
npm run start

It will open your browser on localhost port 3000 and you will see a page like this:

Update Django project configuration

Let's update our Django configuration to render index.html file with React.JS client:

Add CLIENT_DIR environment variable to .env file:

CLIENT_DIR="client/build"

Update django_aws_lambda/urls.py file with the following code:

from django.contrib import admin
from django.urls import path, include
from django.views.generic import TemplateView


urlpatterns = [
    path('admin/', admin.site.urls),
    path('hello/', include('hello.urls')),
    path('', TemplateView.as_view(template_name="index.html"), {'resource': ''}),
    path('<path:resource>', TemplateView.as_view(template_name="index.html")),
]

Update STATICFILES_DIRS in django_aws_lambda/settings.py

STATICFILES_DIRS = [
    str(ROOT_DIR / env('CLIENT_DIR', default='client/build')),
    str(ROOT_DIR / 'static'),
]

Check that our Django project can run the React.JS client

Build production React.JS client locally the following commands:

cd client
export PUBLIC_URL=static/
npm run build
cd ..

Collect static files by running the following command:

python manage.py collectstatic --no-input

Run this Django project locally (make sure that you have already applied migrations and created a superuser):

python manage.py runserver

Go to your browser, open this URL http://localhost:8000 and you will see a page like this:

It looks the same as we saw before by running npm run start in client folder. There are just a couple of differences - now it is run on port 8000 and it is run by our Django web server.

Make the React.JS client talk to the Django server.

First, we need to create an API endpoint to return some data from the server to the client. The easiest way to build REST API in Django is using Django REST framework project.

Install Django REST framework and add it to requirements.txt file

pip install djangorestframework

Create a new Django app called users by running the following command:

python manage.py startapp users

Update users/views.py file with the following code:

from rest_framework import status
from rest_framework.decorators import api_view
from rest_framework.response import Response


@api_view(["GET"])
def get_current_user(request):
    return Response(status=status.HTTP_200_OK, data={"username": request.user.username})

This is an endpoint that returns Response object with username (if the user is not authorized it will return an empty string as username)

Update users/urls.py file with the following code:

from django.urls import path

from .views import get_current_user

app_name = "users"
urlpatterns = [
    path("me/", view=get_current_user, name="get-current-user"),
]

Update our Django project configuration

Update django_aws_lambda/urls.py file with the following code:

from django.contrib import admin
from django.urls import path, include
from django.views.generic import TemplateView


urlpatterns = [
    path('admin/', admin.site.urls),
    path('hello/', include('hello.urls')),
    path('api/users/', include('users.urls')),
    path('', TemplateView.as_view(template_name="index.html"), {'resource': ''}),
    path('<path:resource>', TemplateView.as_view(template_name="index.html")),
]

Update INSTALLED_APPS in django_aws_lambda/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'hello',
    'rest_framework',
    'users',
]

Modifying React.JS client to send requests to the Django server

Install axios library:

cd client
npm install axios -S

-S flag will add axios library to package.json file

Update client/src/App.js file with the following code:

import { useEffect, useState } from 'react';
import axios from 'axios';
import logo from './logo.svg';
import './App.css';

function App() {
  const loadUserDetails = async () => {
    const response = await axios.get('/api/users/me/');
    return response.data;
  };
  const [userData, setUserData] = useState(false);
  useEffect(() => {
    loadUserDetails().then((payload) => {
      setUserData(payload);
    });
  }, []);

  return (
    <div className="App">
      <header className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <p>
          Edit <code>src/App.js</code> and save to reload.
        </p>
        <a
          className="App-link"
          href="https://reactjs.org"
          target="_blank"
          rel="noopener noreferrer"
        >
          Learn React
        </a>
        <h1>Hello, World!</h1>
        <h2>I'm {(userData && userData.username) || 'Unknown User'}</h2>
      </header>
    </div>
  );
}

export default App;

Build production optimized client by running the following command:

export PUBLIC_URL=static/
npm run build
cd ..

Collect static files by running the following command:

python manage.py collectstatic --no-input

Run this Django project locally:

python manage.py runserver

Go to your browser, open this URL http://localhost:8000 and you will see a page like this:

But if you authorize using your superuser username and password you will see a page like this:

Deploying on AWS Lambda using Serverless

Prepare AWS infrastructure

I've already described how to prepare AWS infrastructure in my previous blog posts, so you can use one of the following approaches:

Prepare AWS infrastructure manually as it is described in Deploying a Django project on AWS Lambda using Serverless (Part 3) blog post
Prepare AWS infrastructure automatically using terraform as it is described in Deploying a Django project on AWS Lambda using Serverless (Part 2) blog post

Update Serverless configuration

Add client folder to package.exclude to exclude it from deployment

Update URL in `client/src/App.js` file to be able to send requests to production server

    const response = await axios.get('/production/api/users/me/');

Use Docker for deploying your Django project to AWS Lambda using Serverless

Prepare Amazon Linux 2 docker image with all the necessary dependencies:

docker run -it -v $(pwd):/root/src/ -v /Users/<your-username>/.aws:/root/.aws amazonlinux:latest bash
# install the necessary Unix dependencies:
yum install sudo -y
sudo yum install -y gcc openssl-devel bzip2-devel libffi-devel wget tar sqlite-devel gcc-c++ make
# install node.js version 14:
curl -sL https://rpm.nodesource.com/setup_14.x | sudo -E bash - 
sudo yum install -y nodejs
# install Python 3.8.7:
cd /opt
sudo wget https://www.python.org/ftp/python/3.8.7/Python-3.8.7.tgz
sudo tar xzf Python-3.8.7.tgz
cd Python-3.8.7
sudo ./configure --enable-optimizations
sudo make altinstall
sudo rm -f /opt/Python-3.8.7.tgz
# create python and pip aliases:
alias python='python3.8'
alias pip='pip3.8'
# update pip and setuptools:
pip install --upgrade pip setuptools
# install serverless:
npm install -g serverless
# move to project directory
cd /root/src/
# install requirements inside docker container:
pip install -r requirements.txt
# set the necessary environment variables
export DJANGO_SETTINGS_MODULE=django_react_aws_lambda.production
export AWS_ACCESS_KEY_ID=<your-aws-access-key-id>
export AWS_SECRET_ACCESS_KEY=<your-aws-secret-access-key>
# migrate database changes
python manage.py migrate
# create a superuser in the database
python manage.py createsuperuser
# build React.JS client for AWS Lambda
cd client
npm install 
export PUBLIC_URL="https://<your-cloud-front-distribution>.cloudfront.net/static/"
npm run build
# copy `index.html` from `client/build` to `templates`
cp build/index.html ../templates/index.html
cd ..
# collect static files to AWS S3 bucket
python manage.py collectstatic  --no-input
# install serverless packages from package.json
npm install
# deploy your Django project to AWS Lambda using Serverless
serverless deploy -s production

Now, your Django Project with React.JS client will be available at this URL:
https://<some-id>.execute-api.<your-aws-region>.amazonaws.com/production

Without authorized user:

With authorized user:

How it works

When you go to the Django project URL on your browser it will go to AWS API Gateway that will trigger AWS Lambda function with the Django WSGI server. The Django server will render index.html with React.JS app, the browser will use the domain name of the CloudFront distribution to get React.JS client from the AWS S3 bucket.

Final Words

In this blog post, we saw how to add the React.JS client to the Django project and deploy them on AWS Lambda using Serverless. There is a link to the GitHub repository (GitLab copy) with the code shown in this blog post.

It is the final blog post in this series. I showed just one of many ways to deploy a Django project on AWS Lambda, prepare AWS infrastructure, and add a React.JS client. You can find many other ways to do the same, it is up to you which approach to use.

Don't forget to follow me on Twitter @vadim_khodak or on LinkedIn so you do not miss the next posts.

Deploying a Django project on AWS Lambda using Serverless (Part 3)

Vadym Khodak — Thu, 20 May 2021 11:01:06 +0000

To follow up my previous blog post I decided to show you how to update existing and create new AWS resources for a Django project using Terraform (infrastructure-as-code).

Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.

Terraform is a powerful tool that helps to define your AWS infrastructure as code. Using it, you can:

Have a single source of truth for infrastructure definition and configuration.
Easily replicate AWS infrastructure for other environments or from one AWS account to another one.
Add the necessary updates to your AWS resources.
Store all changes for your AWS resources in a version control system and vet them through a code review process.
Define AWS infrastructure in a human-readable way.

In my previous blog post, you can see how to prepare AWS infrastructure manually via AWS console, now, let's take a look at how we can use Terraform to prepare all the necessary AWS resources for a Django project.

Install and initialize Terraform CLI

Step 1: Install Terraform CLI

macOS:

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Linux (Ubuntu/Debian)

sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform

Check Terraform CLI installation:

terraform --help

The output should be like that:

Usage: terraform [-version] [-help] <command> [args]

The available commands for execution are listed below.
The most common, useful commands are shown first, followed by
less common or more advanced commands. If you're just getting
started with Terraform, stick with the common commands. For the
other commands, please read the help and docs before usage.

Step 2: Initialize your working directory using terraform CLI

Go to your project and create terraform folder

cd django-aws-lambda
mkdir terraform
cd terraform

Create main.tf file inside terraform folder with the following lines:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.37"
    }
  }
  required_version = ">= 0.15.0"
}

provider "aws" {
  region     = "us-east-1"
}

Initialize Terraform CLI

terraform init

The output should be like that:

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Import and update existing AWS resources using Terraform CLI

Step 3: Add a VPC resource and import it

Create vpc.tf file inside terraform folder with the following code:

resource "aws_vpc" "default" {
  cidr_block           = "172.31.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
  instance_tenancy     = "default"
}

Export your AWS credentials:

export AWS_ACCESS_KEY_ID=<YOUR_ AWS_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_ AWS_SECRET_ACCESS_KEY>

Import existing default VPC:

terraform import aws_vpc.default <your-vpc-id>

You should go to your AWS console to get your VPC id (make sure that you use the right region)

Step 4: Add a Security Groups resource and import it

Create security_groups.tf file inside terraform folder with the following code:

resource "aws_security_group" "default" {
  name        = "default"
  description = "default VPC security group"
  vpc_id      = aws_vpc.default.id

  ingress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    security_groups = []
    self            = true
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

}

Import existing default Security Group:

terraform import aws_security_group.default <your-security-group-id>

Step 5: Check infrastructure changes and apply them

Run terraform plan command to check changes to be applied for your AWS infrastructure. If there is nothing to change your output should be like this:

aws_vpc.default: Refreshing state... [id=vpc-<some-id>]
aws_security_group.default: Refreshing state... [id=sg-<some-id>]

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your configuration and the remote system(s). As a result, there are no actions to take.

Add inbound and outbound rules for PostgreSQL database in security_groups.tf (add the following lines inside aws_security_groups resource)

  ingress {
    from_port = 5432
    to_port   = 5432
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  egress {
    from_port        = 5432
    to_port          = 5432
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

Run terraform plan command to check changes again. Now, the output should look like that:

aws_vpc.default: Refreshing state... [id=vpc-<some-id>]
aws_security_group.default: Refreshing state... [id=sg-<some-id>]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_security_group.default will be updated in-place
  ~ resource "aws_security_group" "default" {
      ~ egress                 = [
          - {
              - cidr_blocks      = [
                  - "0.0.0.0/0",
                ]
              - description      = ""
              - from_port        = 0
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "-1"
              - security_groups  = []
              - self             = false
              - to_port          = 0
            },
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 5432
              + ipv6_cidr_blocks = [
                  + "::/0",
                ]
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 5432
            },
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = null
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
        id                     = "sg-5d23015f"
      ~ ingress                = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 5432
              + ipv6_cidr_blocks = [
                  + "::/0",
                ]
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 5432
            },
          - {
              - cidr_blocks      = []
              - description      = ""
              - from_port        = 0
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "-1"
              - security_groups  = []
              - self             = true
              - to_port          = 0
            },
          + {
              + cidr_blocks      = []
              + description      = null
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = true
              + to_port          = 0
            },
        ]
        name                   = "default"
        tags                   = {}
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

You can see in this output that existing rules are going to be deleted and new rules are going to be added. If it is what you expected you are ready to apply all these changes.

Run terraform apply command to apply all infrastructure changes to resources in your AWS account. It will show you all changes again and will ask you to confirm your changes before applying them.

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Create new AWS resources using Terraform CLI

Step 6: Create variables.tf and outputs.tf files

Create variables.tf file inside terraform folder with the following lines:

variable "REGION" {
  type    = string
  default = "us-east-1"
}

variable "RDS_DB_NAME" {
  type = string
}

variable "RDS_USERNAME" {
  type = string
}

variable "RDS_PASSWORD" {
  type = string
}

variable "S3_ORIGIN_ID" {
  type    = string
  default = "django-react-static-assets"
}

Update main.tf file to use AWS region from environment variable:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.37"
    }
  }
  required_version = ">= 0.15.0"
}

provider "aws" {
  region     = var.REGION
}

Terraform variables allow us to use environment variables to protect secrets (like passwords) and apply the same infrastructure for different stages, regions, etc.

You should export these variables in your OS with adding TF_VAR_ before variable names, for example:

export TF_VAR_REGION="us-east-1"
export TF_VAR_RDS_DB_NAME="django_aws"
export TF_VAR_RDS_USERNAME="<your-database-master-username>"
export TF_VAR_RDS_PASSWORD="<your-database-master-password>"
export TF_VAR_S3_ORIGIN_ID="<your-s3-bucket-name>"

Create outputs.tf file inside terraform folder with the following lines:

output "cf_domain" {
  value = aws_cloudfront_distribution.static_distribution.domain_name
}
output "db_domain" {
  value = aws_db_instance.django-aws.address
}
output "exec_lambda_role" {
  value = aws_iam_role.exec_lambda.arn
}
output "security_group_id" {
  value = aws_security_group.default.id
}

Step 6: Create an RDS resource

Create rds.tf file inside terraform folder with the following lines:

resource "aws_db_instance" "django-aws" {
  identifier                 = "django-aws"
  allocated_storage          = 20
  storage_type               = "gp2"
  engine                     = "postgres"
  engine_version             = "12.5"
  instance_class             = "db.t2.micro"
  name                       = var.RDS_DB_NAME
  username                   = "postgres"
  password                   = var.RDS_PASSWORD
  port                       = 5432
  publicly_accessible        = true
  availability_zone          = "us-east-1a"
  security_group_names       = []
  vpc_security_group_ids     = [aws_security_group.default.id]
  parameter_group_name       = "default.postgres12"
  multi_az                   = false
  backup_retention_period    = 0
  backup_window              = "10:05-10:35"
  maintenance_window         = "sun:07:47-sun:08:17"
  auto_minor_version_upgrade = false
  copy_tags_to_snapshot      = true
  skip_final_snapshot        = true
}

Step 7: Create S3 resources

Create s3.tf file inside terraform folder with the following lines to create S3 bucket, block public access, and S3 bucket policy resources:

resource "aws_s3_bucket" "static-assets-django-react" {
  bucket = var.S3_ORIGIN_ID
  acl = "public-read"
  force_destroy = true
}

resource "aws_s3_bucket_public_access_block" "public_access_block" {
  bucket = aws_s3_bucket.static-assets-django-react.id

  ignore_public_acls = false
  restrict_public_buckets = false
  block_public_acls   = false
  block_public_policy = false
}

resource "aws_s3_bucket_policy" "app_static" {
  bucket = aws_s3_bucket.static-assets-django-react.id
  policy = jsonencode({
    Version = "2008-10-17"
    Id      = "PolicyForCloudFrontPrivateContent"
    Statement = [
      {
        Sid    = "1"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ${aws_cloudfront_origin_access_identity.origin_access_identity.id}"
        },
        Action   = "s3:GetObject",
        Resource = "${aws_s3_bucket.static-assets-django-react.arn}/*"
      }
    ]
  })
}

Step 8: Create CloudFront resources

Create cloud_front.tf file inside terraform folder with the following lines to create CloudFront distribution, and CloudFront Origin Access Identity resources:

resource "aws_cloudfront_origin_access_identity" "origin_access_identity" {
  comment = "access-identity-${var.S3_ORIGIN_ID}.s3.amazonaws.com"
}

resource "aws_cloudfront_distribution" "static_distribution" {
  origin {
    domain_name = aws_s3_bucket.static-assets-django-react.bucket_regional_domain_name
    origin_id   = var.S3_ORIGIN_ID
    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.origin_access_identity.cloudfront_access_identity_path
    }
  }

  enabled             = true
  is_ipv6_enabled     = true
  comment             = "Some Comment"

  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD", "OPTIONS"]
    cached_methods   = ["GET", "HEAD"]
    compress         = false
    target_origin_id = "S3-${var.S3_ORIGIN_ID}"

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 0
    max_ttl                = 0
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    cloudfront_default_certificate = true
  }
}

Step 9: Create IAM resources

Create iam file inside terraform folder with the following lines to create IAM Role, IAM Policy, and Attach Policy to the Role resources:

resource "aws_iam_role" "exec_lambda" {
  name = "exec_lambda"
  path = "/"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = [
            "lambda.amazonaws.com"
          ]
        },
        Action = "sts:AssumeRole"
      },
    ]
  })
}
resource "aws_iam_policy" "exec_lambda_policy" {
  name        = "exec_lambda_policy"
  description = "Lambda policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow",
        Action = [
          "lambda:InvokeFunction",
        ],
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "attach_role" {
  role       = aws_iam_role.exec_lambda.name
  policy_arn = aws_iam_policy.exec_lambda_policy.arn
}

Step 10: Apply all changes to create the necessary AWS resources

Run terraform plan to check changes that will be applied.
Run terraform apply to create all the necessary resources.

As a result, you will have a long output with all logs and custom output we configured in outputs.tf:

You should use these outputs in your Django and Serverless configurations.

Final Advice

I'd like to highlight that this is just an example, so your production configuration can be different. You need to be sure that your infrastructure configuration satisfies your security policy.

There is some advice:

Use a version control system to track changes in your infrastructure and for Code Review.
Don't store your terraform state in GitHub, GitLab, Bitbucket, etc.
Configure an S3 bucket as a backend for terraform, to enable collaboration:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.37"
    }
  }
  required_version = ">= 0.15.0"

  backend "s3" {
    region         = var.REGION
    bucket         = var.BECKEND_BUCKET_NAME
    key            = var.STATE_KEY
    dynamodb_table = var.DYNAMO_DB_TABLE
    encrypt        = true
  }
}

provider "aws" {
  region = var.REGION
}

Final words

I was inspired by Mykyta Protsenko's talk "Infrastructure-as-code: evolving tools vs core principles" to write this blog post. Thanks a lot to Mykyta for all your help and support.

I'll show how to add a simple React.JS client and serve it from an S3 bucket using a CloudFront distribution in my next blog post. Follow me on Twitter @vadim_khodak or on LinkedIn so you do not miss the next posts.

Deploying a Django project on AWS Lambda using Serverless (Part 2)

Vadym Khodak — Mon, 10 May 2021 11:40:59 +0000

After my previous blog post Deploy Django App on AWS Lambda using Serverless (Part 1), people started asking me about AWS infrastructure for Django projects, so I decided to share my experience with this.

It is not obvious (for people who don't have enough experience with AWS resources) how to create and configure all the necessary AWS resources to deploy a Django project on AWS Lambda using Serverless.

Here are a list of ways of how to do this:

manually via the AWS console
automatically using Terraform
automatically using AWS SDK

In this blog post, I will show you how to do it manually via AWS console.

Update configuration for existing AWS resource and create new ones

Step 1: Create your own AWS account (if you don't have one).
Here is a link to a manual for creating and activating an AWS account.

Step 2: Go to the AWS Management Console

Step 3: Select your region. In my case, it is US East (N. Virginia)us-east-1

Step 4: Update Security Group with rules for AWS RDS service (PostgreSQL)

Type EC2 in the search bar and click on the EC2 service in the search results

Choose the Security Groups option in the Network & Security section

Click on your security group id

Click on Edit inbound rules

Click on Add rule. Then, select the PostgreSQL option in the Type column. Next, choose the Anywhere option in the Source column. Finally, click on the Save rules button

Go to the Outbound rules tab and add the same rules that were described in the previous section

Step 5: Create an IAM role

Type IAM in the search bar and click on the IAM service in the search results

Click on Roles in the IAM side bar or in the IAM dashboard

Click on the Create Role button

Select AWS service, Lambda, and click on the Next: Permissions button

Type Lambda in the search bar, select the AWSLambda_FullAccess policy, click on the Next: Tags button

Click on the Next: Review button

Type Role name, Role description, and click on the Create role button

Click on the created role name

Click on the Copy Role ARN button

Next, you should add the role ARN to the Serverless configuration directly or using environment variables (for example .env file)

ROLE=arn:aws:iam::<your-aws-account-id>:role/exec_lambda

Step 6: Create S3 buckets for static assets and deployment

Type S3 in the search bar and click on the S3 service in the search results

Click on the Create bucket button

Type Bucket name, select your AWS region, unselect Block all public access and click on the Create bucket button

Then, you should repeat all the steps mentioned above to create an S3 bucket for deployment.

Next, you should add your bucket names to your Django and Serverless configurations directly or using environment variables (for example .env file)

AWS_STORAGE_BUCKET_NAME='django-react-static-assets'
DEPLOYMENT_BUCKET='django-react-deployments'

Step 7: Create a CloudFront distribution

Type CloudFront in the search bar and click on the CloudFront service in the search results

Click on the Create Distribution button

Click on the Get started button

Select your S3 bucket

Select: Yes for Restrict Bucket Access, Create a New Identity for Origin Access Identity, Yes, Update Bucket Policy for Grant Read Permissions on Bucket, HTTP and HTTPS for Viewer Protocol Policy

Type "some comment" (optional) and click on the Create Distribution button

Go to the distributions list and copy the Domain Name

Then, you should add CloudFront distribution Domain Name to your Django and Serverless configurations directly or using environment variables (for example .env file)

AWS_S3_CDN_DOMAIN="<domain-id>.cloudfront.net"

Step 8: Create RDS

Type RDS in the search bar and click on the RDS service in the search results

Click on the Create database button

Select Standard create, PostgreSQL, and Version (in my example, it is PostgreSQL 12.5-R1)

Select Free Tier as a Template, fill in DB instance identifier, Master username, Master password, Confirm password

Select Burstable classes (includes t classes) and db.t2.micro for DB instance class, General Purpose (SSD) as Storage type, and 20 as Allocated storage, unselect Enable storage autoscaling

Skip the Availability & durability section
Configure Connectivity:
- Select your default VPC as Virtual private cloud (VPC) > After a database is created, you can't change the VPC selection.
- Select default as Subnet group
- Select Yes for Public access
- Select Choose existing for VPC security group, and select default for Existing VPC security groups section
- Select Availability Zone (in my example us-east-1a)

Select Password authentication or any other you want to use as Database authentication options, unselect all check boxes in Additional configuration and type Initial database name (in my example, it is django_aws)

Click on the Create database button

Go to the RDS dashboard and click on Databases in the RDS side bar or on DB instances

Click on the created database identifier

Copy the database endpoint, the subnets, and the security groups

Then, you should add this info to your Django and Serverless configurations directly or using environment variables (for example .env file)

DB_HOST='django-aws.<db-domain-id>.us-east-1.rds.amazonaws.com'
DB_USER='<your-master-db-user>'
DB_PASSWORD='<password-for-your-master-db-user>'
DB_NAME='<your-db-name>'
SECURITY_GROUPS=sg-<security-group-id>
SUBNETS=subnet-<subnet-id>,subnet-<subnet-id>,subnet-<subnet-id>,subnet-<subnet-id>,subnet-<subnet-id>,subnet-<subnet-id>

NOTE: This is just an example of AWS configuration I use in my example. You may use your own configuration.

Automate managing your AWS infrastructure

I showed you how to configure all the necessary AWS resources for a Django project manually using the AWS Management Console. There are some ways to automate this process. I'll show how to manage AWS resources using Terraform (infrastructure as code) in my next blog post. Follow me on Twitter @vadim_khodak or on LinkedIn so you do not miss the next posts.

Deploying a Django project on AWS Lambda using Serverless (Part 1)

Vadym Khodak — Fri, 16 Apr 2021 06:46:04 +0000

BLUF

As a follow-up to a post where we looked at the most common questions about Django in the Cloud, now, I'd like to help you deploy your Django App on Amazon Web Services and make you more independent from other developers like DevOps and CloudOps Engineers. There are many options for doing that but I'd like to show one of them and I hope that in the end you will be able to deploy your Django App on AWS Lambda using Serverless.

I was motivated by Daniil Bratchenko's article Don’t Let Software Vendors Dictate Your Business Processes to start writing this blog post.

It is so hard to find Software that will fit all your business processes as all companies are unique. This is why many companies have decided to set up dedicated teams building Software for their specific business processes and needs. From my personal point of view, Django App on AWS Lambda using Serverless is a good solution for cases like that.

Also, you can use this approach for prototyping your projects running them at their early stage.

There are a few advantages and disadvantages of using this approach.

Advantages of using AWS Lambdas:

cost (AWS Lambda is cheaper comparing to AWS EC2);
simplicity in running and maintaining;
scalability;
quick deployment.

the disadvantages:

AWS Lambda requires some extra time to run your App;
size limit for deployment package;
API Gateway limitation (30-sec timeout, 6 Mb response body size);
it might cost more than AWS EC2 if there are too many requests.

Prepare AWS infrastructure

Probably, you are aware of a variety of AWS services required for web applications. In order to deploy a Django project on AWS Lambdas you should prepare your AWS infrastructure.
There is a list of AWS services I use for my Django project:

Lambdas to run our wsgi application
API Gateway to handle HTTP request and send them to Lambdas
S3 buckets for Lambda deployments and storing static files
CloudFront distribution for serving static files from S3 bucket
RDS for a database (I use Postgres)
VPC with subnets
EC2 for Security Groups
IAM for roles and policies
CloudWatch for logs

AWS Lambdas, API Gateway will be created automatically by Serverless. I will try to walk you though the process of creating all the necessary AWS resources in my following blog posts.

Create a Django project

Django startproject command allows us to create a simple Django project, in addition to that, there are some great Cookiecutter projects that can help you start your project easily (For example Cookiecutter Django). I use default django-admin startproject cli command in this example.

pip install django
django-admin startproject django_aws_lambda

Configure requirements

There are many options to store your project requirements, for example requirements.txt, Pipfile, pyproject.toml. You can one of these options. I'm using requirements.txt here.

create requirements.txt file in a root directory of the project
add the following libraries to requirements.txt file:

boto3==1.17.17
Collectfast==2.2.0
Django==3.1.7
django-environ==0.4.5
psycopg2-binary==2.8.6
Werkzeug==1.0.1

create and activate virtual environments

Choose your preferred tool for managing virtual environments (like conda, pyenv, virtualenv, etc.)

install requirements

pip install -r requirements.txt

Create `hello` Django app

create app using startapp Django command

python manage.py startapp hello

create templates folder

mkdir templates

create index.html file in templates folder with the following lines:

{% load static %}

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Greeting</title>
</head>
<body>
<div>
  <h1>Hello {{ name }}</h1>
<img src="{% static 'django.jpeg' %}" alt="Django" style="width: 20%">
</div>
</body>
</html>

create folder static in the root directory of the project

mkdir static

add an image file to static folder, for example django.jpeg
update hello/views.py

from django.shortcuts import render


# Create your views here.
def hello(request, resource=None):

    return render(request, "index.html", {"name": resource or 'World'})

Configure environments variables:

create .env file in the root directory of the project
configure the following variables:

STAGE='production'
DB_HOST=<your database host>
DB_USER=<your database user name>
DB_PASSWORD=<your database password>
DB_NAME=<your database name>
DJANGO_SECRET_KEY=<some django secret key>
AWS_S3_CDN_DOMAIN=<your Cloud Front distribution, like: `<distribution id>.cloudfront.net`>
AWS_S3_REGION_NAME=<your AWS region>
AWS_STORAGE_BUCKET_NAME=<AWS s3 bucket for static files with punlic policies>
DEPLOYMENT_BUCKET=<AWS s3 bucket for deployment>
AWS_KEY_ID=<your AWS Key Id>
AWS_SECRET=<your AWS Secret>
DJANGO_ADMIN_URL=<Django admin url>
DJANGO_ALLOWED_HOSTS=<list of allowed hosts separated by coma>

Create configuration for local development and production

update settings.py in django_aws_lambda folder with the following lines:

"""
Django settings for django_aws_lambda project.

Generated by 'django-admin startproject' using Django 1.11.29.

For more information on this file, see
https://docs.djangoproject.com/en/1.11/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.11/ref/settings/
"""

from pathlib import Path

import environ

ROOT_DIR = Path(__file__).resolve(strict=True).parent.parent
env = environ.Env()
READ_DOT_ENV_FILE = env.bool('DJANGO_READ_DOT_ENV_FILE', default=True)
if READ_DOT_ENV_FILE:
    env.read_env(str(ROOT_DIR / '.env'))

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = env('DJANGO_SECRET_KEY', default='<some-secured-key>')
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False

ALLOWED_HOSTS = env.list('DJANGO_ALLOWED_HOSTS', default=['127.0.0.1', 'localhost'])
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'hello',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'django_aws_lambda.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            str(ROOT_DIR / 'templates'),
            str(ROOT_DIR / 'staticfiles'),
        ],
        'OPTIONS': {
            'loaders': [
                'django.template.loaders.filesystem.Loader',
                'django.template.loaders.app_directories.Loader',
            ],
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.template.context_processors.i18n',
                'django.template.context_processors.media',
                'django.template.context_processors.static',
                'django.template.context_processors.tz',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'django_aws_lambda.wsgi.application'


# Database
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': ROOT_DIR / "db.sqlite3",
    }
}


# Password validation
# https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/1.11/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.11/howto/static-files/

STATIC_ROOT = str(ROOT_DIR / 'staticfiles')
STATIC_URL = '/static/'

STATICFILES_DIRS = [str(ROOT_DIR / 'static')]
STATICFILES_FINDERS = [
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
]
MEDIA_ROOT = str(ROOT_DIR / 'media')
MEDIA_URL = '/media/'
ADMIN_URL = env('DJANGO_ADMIN_URL')

create local.py and production.py files inside django_aws_lambda folder on the same level as settings.py
add the following lines to local.py:

from .settings import *  # noqa


DEBUG = True

add the following lines to production.py:

from .settings import *  # noqa

DEBUG = False
DATABASES["default"] = {
    'ENGINE': 'django.db.backends.postgresql',
    'NAME': env("DB_NAME"),
    'USER': env("DB_USER"),
    'PASSWORD': env("DB_PASSWORD"),
    'HOST': env("DB_HOST"),
    'PORT': '5432',
}
DATABASES["default"]["ATOMIC_REQUESTS"] = True  # noqa F405
DATABASES["default"]["CONN_MAX_AGE"] = env.int("CONN_MAX_AGE", default=60)  # noqa F405
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
SECURE_SSL_REDIRECT = env.bool("DJANGO_SECURE_SSL_REDIRECT", default=False)
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SECURE_HSTS_SECONDS = 60
SECURE_HSTS_INCLUDE_SUBDOMAINS = env.bool("DJANGO_SECURE_HSTS_INCLUDE_SUBDOMAINS", default=True)
SECURE_HSTS_PRELOAD = env.bool("DJANGO_SECURE_HSTS_PRELOAD", default=True)
SECURE_CONTENT_TYPE_NOSNIFF = env.bool("DJANGO_SECURE_CONTENT_TYPE_NOSNIFF", default=True)
INSTALLED_APPS += ["storages"]  # noqa F405
AWS_KEY = env("AWS_KEY_ID")
AWS_SECRET = env("AWS_SECRET")
AWS_STORAGE_BUCKET_NAME = env("AWS_STORAGE_BUCKET_NAME")
AWS_QUERYSTRING_AUTH = False
_AWS_EXPIRY = 60 * 60 * 24 * 7
AWS_S3_OBJECT_PARAMETERS = {"CacheControl": f"max-age={_AWS_EXPIRY}, s-maxage={_AWS_EXPIRY}, must-revalidate"}
AWS_S3_REGION_NAME = env("AWS_S3_REGION_NAME", default=None)
AWS_S3_CUSTOM_DOMAIN = env("DJANGO_AWS_S3_CUSTOM_DOMAIN", default=None)
aws_s3_domain = AWS_S3_CUSTOM_DOMAIN or f"{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com"
STATICFILES_STORAGE = "django_aws_lambda.utils.StaticRootS3Boto3Storage"
COLLECTFAST_STRATEGY = "collectfast.strategies.boto3.Boto3Strategy"
STATIC_URL = f"https://{aws_s3_domain}/static/"
DEFAULT_FILE_STORAGE = "django_aws_lambda.utils.MediaRootS3Boto3Storage"
MEDIA_URL = f"https://{aws_s3_domain}/media/"
MEDIAFILES_LOCATION = "/media"

STATICFILES_LOCATION = "/static"
TEMPLATES[-1]["OPTIONS"]["loaders"] = [  # type: ignore[index] # noqa F405
    (
        "django.template.loaders.cached.Loader",
        [
            "django.template.loaders.filesystem.Loader",
            "django.template.loaders.app_directories.Loader",
        ],
    )
]

update wsgi.py file in django_aws_lambda folder with the following lines:

"""
WSGI config for django_aws_lambda project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/1.11/howto/deployment/wsgi/
"""

"""
WSGI config for django_aws_lambda project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/3.0/howto/deployment/wsgi/
"""

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_aws_lambda.production')

application = get_wsgi_application()

update urls.py file in django_aws_lambda folder with the following lines:

"""django_aws_lambda URL Configuration

The `urlpatterns` list routes URLs to views. For more information please see:
    https://docs.djangoproject.com/en/3.0/topics/http/urls/
Examples:
Function views
    1. Add an import:  from my_app import views
    2. Add a URL to urlpatterns:  path('', views.home, name='home')
Class-based views
    1. Add an import:  from other_app.views import Home
    2. Add a URL to urlpatterns:  path('', Home.as_view(), name='home')
Including another URLconf
    1. Import the include() function: from django.urls import include, path
    2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import path

from hello.views import hello

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', hello),
    path('<path:resource>', hello),
]

update manage.py with the following lines:

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys


def main():
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_aws_lambda.production')
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == '__main__':
    main()

create a folder utils inside django_aws_lambda
create storages.py file inside utils folder with the following lines:

from storages.backends.s3boto3 import S3Boto3Storage


class StaticRootS3Boto3Storage(S3Boto3Storage):
    location = "static"
    default_acl = "public-read"


class MediaRootS3Boto3Storage(S3Boto3Storage):
    location = "media"
    file_overwrite = False

Run Django project locally

set environment variable with a path to Django local configuration file

export DJANGO_SETTINGS_MODULE=django_aws_lambda.local

migrate database changes

python manage.py migrate

create a superuser in the database

python manage.py createsuperuser

Then provide a username, user email, password, and confirm the password

collect static files

python manage.py collectstatic

run server locally

python manage.py runserver

go to http://127.0.0.1:8000/ and you will see this:
go to http://127.0.0.1:8000/Dev.to and you will see this:

Create serverless configuration

initialize npm:

npm init

install serverless

npm install -g serverless

install serverless plugins

npm install -P serverless-dotenv-plugin
npm install -P serverless-prune-plugin
npm install -P serverless-python-requirements
npm install -P serverless-wsgi

create serverless.yaml file with the following configuration:

service: django-aws-lambda

plugins:
  - serverless-dotenv-plugin
  - serverless-prune-plugin
  - serverless-python-requirements
  - serverless-wsgi
useDotenv: true

custom:
  dotenv:
    logging: false
  pythonRequirements:
    dockerizePip: non-linux
    zip: true
    fileName: requirements.txt
  stage: ${env:STAGE}
  wsgi:
    app: django_aws_lambda.wsgi.application
    packRequirements: false
  prune:
    automatic: true
    number: 3

functions:
  - app:
      handler: wsgi_handler.handler
      events:
        - http: ANY /
        - http: ANY /{proxy+}
      timeout: 30

provider:
  name: aws
  role: arn:aws:iam::<role_id>:role/<role_name>
  profile: <your-profile-name>  # make sure that you configured aws profile using `aws configure --profile <your-profile-name>`
  region: us-east-1
  runtime: python3.8
  versionFunctions: false
  stage: ${env:STAGE}
  timeout: 60
  vpc:
    securityGroupIds:
      - <your-security-group-id>
      - <your-security-group-id>
    subnetIds:
      - <your-subnet-id>
      - <your-subnet-id>
  deploymentBucket:
    name: ${env:DEPLOYMENT_BUCKET}
  apiGateway:
    shouldStartNameWithService: true
  lambdaHashingVersion: 20201221

package:
  individually:
    true
  exclude:
    - .env
    - .git/**
    - .github/**
    - .serverless/**
    - static/**
    - .cache/**
    - .pytest_cache/**
    - node_modules/**

Use Docker for deploying your Django project to AWS Lambda using Serverless

run Amazon Linux 2 docker image:

docker run -it -v $(pwd):/root/src/ -v /Users/<your_user>/.aws:/root/.aws amazonlinux:latest bash

install the necessary Unix dependencies:

yum install sudo -y
sudo yum install -y gcc openssl-devel bzip2-devel libffi-devel wget tar sqlite-devel gcc-c++ make

install node.js version 14:

curl -sL https://rpm.nodesource.com/setup_14.x | sudo -E bash - 
sudo yum install -y nodejs

install Python 3.8.7:

cd /opt
sudo wget https://www.python.org/ftp/python/3.8.7/Python-3.8.7.tgz
sudo tar xzf Python-3.8.7.tgz
cd Python-3.8.7
sudo ./configure --enable-optimizations
sudo make altinstall
sudo rm -f /opt/Python-3.8.7.tgz

create python and pip aliases:

alias python='python3.8'
alias pip='pip3.8'

update pip and setuptools:

pip install --upgrade pip setuptools

install serverless:

npm install -g serverless

move to project directory

cd /root/src/

install requirements inside docker container:

pip install -r requirements.txt

set environment variable with a path to django production configuration file

export DJANGO_SETTINGS_MODULE=django_aws_lambda.production

migrate database changes

python manage.py migrate

create a superuser in the database

python manage.py createsuperuser

Then provide a username, user email, password, and confirm the password

collect static files to AWS S3 bucket

python manage.py collectstatic

If you get NoCredentialsError from botocore you should add to environment variables AWS_PROFILE:

export AWS_PROFILE=<your-aws-profile-name>

install serverless packages from package.json

npm install

deploy your Django project to AWS Lambda using Serverless

serverless deploy -s production

Your response will look like that:

Serverless: Adding Python requirements helper to ....
Serverless: Generated requirements from /root/src/requirements.txt in /root/src/.serverless/requirements.txt...
Serverless: Installing requirements from /root/.cache/serverless-python-requirements/ ...
Serverless: Using download cache directory /root/.cache/serverless-python-requirements/downloadCacheslspyc
Serverless: Running ...
Serverless: Zipping required Python packages for ....
Serverless: Using Python specified in "runtime": python3.8
Serverless: Packaging Python WSGI handler...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Removing Python requirements helper from ....
Serverless: Injecting required Python packages to package...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service app.zip file to S3 (60.48 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..........
Serverless: Stack update finished...
Service Information
service: <your-serverless-service-name>
stage: production
region: <your-aws-region>
stack: <your-serverless-service-name>-pronduction
resources: 8
api keys:
  None
endpoints:
  ANY - https://<some-id>.execute-api.<your-aws-region>.amazonaws.com/production
  ANY - https://<some-id>.execute-api.<your-aws-region>.amazonaws.com/production/{proxy+}
functions:
  app: <your-serverless-service-name>-production-app
layers:
  None
Serverless: Prune: Running post-deployment pruning
Serverless: Prune: Querying for deployed function versions
Serverless: Prune: <your-serverless-service-name>-production-app has 3 additional versions published and 0 aliases, 0 versions selected for deletion
Serverless: Prune: Pruning complete.
Serverless: Removing old service artifacts from S3...

**************************************************************************************************************************************
Serverless: Announcing Metrics, CI/CD, Secrets and more built into Serverless Framework. Run "serverless login" to activate for free..
**************************************************************************************************************************************

Now, your Django Project will be available at this URL:
https://<some-id>.execute-api.<your-aws-region>.amazonaws.com/production

Congratulations!

Here is a link to a GitHub repository with the code shown in this blog post.

If you want to learn more about Django projects on AWS Lambdas follow me on Twitter (@vadim_khodak) I plan to write a post showing how to create all the necessary AWS resources for this Django project, how to add React.js client to the Django project and more.

DEV Community: Vadym Khodak

Creating AWS resources using Boto3 for deploying Django project

Boto3 Introduction

Let's start codding

Using Boto3 - the AWS SDK for Python

Final Words

Deploying a Django project on AWS Lambda using Serverless (Part 4)

BLUF

Getting started

Add React.JS client

Update Django project configuration

Check that our Django project can run the React.JS client

Make the React.JS client talk to the Django server.

Modifying React.JS client to send requests to the Django server

Deploying on AWS Lambda using Serverless

Prepare AWS infrastructure

Update Serverless configuration

Update URL in client/src/App.js file to be able to send requests to production server

Use Docker for deploying your Django project to AWS Lambda using Serverless

How it works

Final Words

Deploying a Django project on AWS Lambda using Serverless (Part 3)

Install and initialize Terraform CLI

Import and update existing AWS resources using Terraform CLI

Create new AWS resources using Terraform CLI

Final Advice

Final words

Deploying a Django project on AWS Lambda using Serverless (Part 2)

Update configuration for existing AWS resource and create new ones

Automate managing your AWS infrastructure

Deploying a Django project on AWS Lambda using Serverless (Part 1)

BLUF

Prepare AWS infrastructure

Create a Django project

Configure requirements

Create hello Django app

Configure environments variables:

Create configuration for local development and production

Run Django project locally

Create serverless configuration

Use Docker for deploying your Django project to AWS Lambda using Serverless

Top 6 questions people ask about Django Apps in a Cloud

Where can I host Django for free?

Which Cloud solution is the best for Django?

For example:

How to deploy a Django application to Pythonanywhere?

How to deploy a Django application to Heroku?

How to deploy a Django application to Google Cloud?

How to deploy a Django application on AWS?

Summary

Update URL in `client/src/App.js` file to be able to send requests to production server

Create `hello` Django app