DEV Community: João Marcos

Build a GitLab CI/CD pipeline to deploy a Django app to AWS Lambda

João Marcos — Fri, 22 Dec 2023 22:47:19 +0000

CI and CD stand for Continuous Integration and Continuous Delivery. CI refers to a modern software development practice of incrementally and frequently integrating code changes into a shared source code repository. Automated jobs build and test the application to ensure the code changes being merged are reliable. The CD process is then responsible for quickly delivering the changes into the specified environments.

In my last post you learned how to deploy a Django app to an AWS Lambda function using Serverless Framework. This post will use the same project available in GitHub. Now we’ll create a pipeline to run unit tests, build and deploy the app automatically whenever a new merge request is merged into the staging branch of the project. So, the first step is to create the staging branch, you can run the command bellow in a terminal window in the root folder of the project:



git checkout -b staging
git push origin staging

The first command creates and switches to the new branch, the second command pushes it to the server. staging will be our main branch, all feature branches must be created from it. Create the the branch we will work on, I’ll call it create-cicd-pipeline:



git checkout -b create-cicd-pipeline

The `.gitlab-ci.yml` file

To use GitLab CI/CD, we start by creating a .gitlab-ci.yml file at the root of our project. This file will contain the configuration for our CI/CD pipelines.

Create the file and add the code bellow:



stages:
  - test

"Server Tests":
  image: python:3.9-slim
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"  && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "staging"
  before_script:
    - pip install --upgrade pip
    - pip install -r requirements/dev.txt
  script:
    - pytest -v

The code above crates a pipeline with one job named “Server Tests”. The job belongs to the “test” stage and will run in the python:3.9-slim Docker image.

`stages`

Stages can contain groups of jobs. The order of the items in stages defines the execution order of the jobs. Important to know about stages:

Jobs in the same stage run in parallel
Jobs in the next stage run after the jobs from the previous stage complete successfully

`before_script`, `script`

These sections define the commands that will run during job execution. If any of those script commands return a non-zero exit code (indicating an error or failure), the job fails and subsequent commands do not execute.

`rules`

It is used to determine whether a job should be included (run) or excluded (not run) from a pipeline. Rules are evaluated when the pipeline is created. When a match is found, the job is either included or excluded from the pipeline depending on the configuration.

The rules:if clause specifies when to add a job to a pipeline. If the if statement is true, then the job is added. In our if statement we’re using two predefined CI/CD variables.

Predefined CI/CD variables

GitLab CI/CD makes a set of predefined variables available in every GitLab CI/CD pipeline. Those variables can be used in pipeline configuration and job scripts. To determine the execution of this job, we are using the following variables in the rules:if clause:

CI_PIPELINE_SOURCE

How the pipeline was triggered. This variable can take on a few different values, such as push, web, api, among others. When a merge request is created or updated, its value becomes merge_request.

CI_MERGE_REQUEST_TARGET_BRANCH_NAME

The target branch name of the merge request.

Now that you understand the rules clause and the variables we’re using in the if statement, you can guess when this job is going to be run, right?

The “Server Tests” job will be added to the pipeline every time a Merge Request, whose target branch is “staging”, is created or updated (when you push more commits to the MR’s source branch, for example).

There’s one last thing missing before we can create our first MR and run the pipeline — The database!

The `service` clause

A service is an additional container that your pipeline can create. The service will be available to your first container. The two containers will have access to one another and will be able to communicate when the job is running. You specify the service’s image by using the services keyword.

The most common use case for this clause is to run a database container. It’s easier and faster to use an existing image and run it as an additional container than to install PostgreSQL, for example, each time the pipeline is run.

Let’s create our PostgreSQL service and pass custom environment variables to the containers so our application can connect to the database:



stages:
  - test

"Server Tests":
  image: python:3.9-slim
  stage: test
  services:
    - postgres:15.4-alpine
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"  && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "staging"
  before_script:
    - pip install --upgrade pip
    - pip install -r requirements/dev.txt
  script:
    - pytest -v
  variables:
    PGDATA: /pgdata
    POSTGRES_USER: secret
    POSTGRES_PASSWORD: secret
    POSTGRES_DB: django_serverless
    DB_HOST: postgres
    DB_NAME: django_serverless
    DB_USER: secret
    DB_PASSWORD: secret
    DB_PORT: 5432

The custom environment variables are defined in the variables clause. The variables starting with POSTGRES_ will be used by the PostgreSQL container to set up the database. The ones starting with DB_ will be used by the Django application to connect to the database. The app’s connection configuration is located in the myapi/settings.py file. Search for the DATABASE constant.

Commit and push the changes to GitLab so we can create our first MR.

Create a Merge Request on GitLab

On your project’s home page, navigate to the left-side menu and click on “Code” then select “Merge Requests”. Choose the source and target branches as shown in the image bellow, and click on “Compare branches and continue”.

In the next page you can just click on “Create merge Request”.

The pipeline is automatically triggered after the MR is created. You can click on the pipeline ID to check the stages and jobs:

You can also click on the jobs to see the detailed execution logs.

The Deploy Job

Create the “Deploy Staging” job after the “Server Tests” one in the .gitlab-ci.yml:



"Deploy Staging":
  image: node:16-bullseye
  stage: deploy
  environment: staging
  rules:
    - if: $CI_COMMIT_REF_NAME == "staging" &&  $CI_PIPELINE_SOURCE == "push"
  before_script:
    - apt-get update && apt-get install -y python3-pip
    - npm install -g serverless
    - npm install
    - touch .env
    - echo "STATIC_FILES_BUCKET_NAME=$STATIC_FILES_BUCKET_NAME">>.env
    - echo "AWS_REGION_NAME=$AWS_REGION_NAME">>.env
    - echo "DB_NAME=$DB_NAME">>.env
    - echo "DB_USER=$DB_USER">>.env
    - echo "DB_PASSWORD=$DB_PASSWORD">>.env
    - echo "DB_HOST=$DB_HOST">>.env
    - echo "DB_PORT=$DB_PORT">>.env
  script:
    - sls deploy --verbose
    - sls wsgi manage --command "collectstatic --noinput"

In this job we’re using the node:16-bullseye docker image. Bullseye is the Debian release codename, it means that the image is based on version 11 of Debian. I chose this version because it already comes with python3.9 pre-installed. That’s the same python version of the runtime of the Lambda function defined in the serverless.yml file of our project.

This job will run only when new code is pushed to the staging branch.

In the before_script section, we’re installing the dependencies (pip, Serverless Framework and plugins) and creating the .env file that our application needs. But where did the variables, such as $STATIC_FILES_BUCKET_NAME and $DB_NAME come from?

GitLab CI/CD variables

There are a few ways to define custom environment variables to be used in pipeline’s jobs. In the “Server Test” job you learned how to set environment variables for a specific job by using the keyword variables.

This method is not a good choice if you need to declare variables that will take on sensitive values, like $DB_PASSWORD, for example. Such variables should be stored in the settings in the UI, not in the .gitlab-ci.yml file.

To define CI/CD variables in the UI:

Go to the project’s Settings > CI/CD and expand the Variables section.
Select Add variable and fill in the details:
- Key: Must be one line, with no spaces, using only letters, numbers, or _.
- Value: No limitations.
- Type: Variable (default) or File.
- Environment scope: Optional. All, or specific environments.
- Protect variable Optional. If selected, the variable is only available in pipelines that run on protected branches or protected tags.
- Mask variable Optional. If selected, the variable’s Value is masked in job logs.

These are the variables I added:

Besides the variables I referenced in the before_script clause of the deploy job, I’ve included AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. This will ensure that the machine running the jobs has access to my AWS account.

Don’t forget to mask the variables with the most sensitive data, such as the database password and the AWS secret access key.

Environments

Environments are used define deployment locations for code. On GitLab you can create multiple environments that align with your workflow, such as integration, staging, beta, and production, among others. Custom variables assigned to specific environments, like “staging”, remain inaccessible from pipelines running in a different environment, such as “production”, for example.

To create an environment for your project in the UI:

Select Operate > Environments.
Select Create an environment.
Type a name and a URL (optional) for the environment
Click on the “Save” button.

Deploying to Staging

Before our first deploy, we’ll add one more if clause to the rules section of the “Server Tests” job:



- if: $CI_COMMIT_REF_NAME == "staging" &&  $CI_PIPELINE_SOURCE == "push"

It’s the same if we have in the “Deploy Staging” job. This will make the tests job run again before the deploy job is executed.

The final version of the .gitlab-ci.yml file:



stages:
  - test
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/pip-cache"

cache:
  key: pip-cache-$CI_COMMIT_REF_SLUG
  paths:
    - $PIP_CACHE_DIR

"Server Tests":
  image: python:3.9-slim
  stage: test
  services:
    - postgres:15.4-alpine
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"  && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "staging"
    - if: $CI_COMMIT_REF_NAME == "staging" &&  $CI_PIPELINE_SOURCE == "push"
  before_script:
    - pip install --upgrade pip
    - pip install -r requirements/dev.txt
  script:
    - pytest -v
  variables:
    PGDATA: /pgdata
    POSTGRES_USER: secret
    POSTGRES_PASSWORD: secret
    POSTGRES_DB: django_serverless
    DB_HOST: postgres
    DB_NAME: django_serverless
    DB_USER: secret
    DB_PASSWORD: secret
    DB_PORT: 5432

"Deploy Staging":
  image: node:16-bullseye
  stage: deploy
  environment: staging
  rules:
    - if: $CI_COMMIT_REF_NAME == "staging" &&  $CI_PIPELINE_SOURCE == "push"
  before_script:
    - apt-get update && apt-get install -y python3-pip
    - npm install -g serverless
    - npm install
    - touch .env
    - echo "STATIC_FILES_BUCKET_NAME=$STATIC_FILES_BUCKET_NAME">>.env
    - echo "AWS_REGION_NAME=$AWS_REGION_NAME">>.env
    - echo "DB_NAME=$DB_NAME">>.env
    - echo "DB_USER=$DB_USER">>.env
    - echo "DB_PASSWORD=$DB_PASSWORD">>.env
    - echo "DB_HOST=$DB_HOST">>.env
    - echo "DB_PORT=$DB_PORT">>.env
  script:
    - sls deploy --verbose
    - sls wsgi manage --command "collectstatic --noinput"

Commit and push the updates to GitLab. The “Server Tests” job will run again, after it finishes successfully, you click on the “Merge” button.

Conclusion

A CI/CD pipeline brings many advantages to an application. The automation of the tests and deploys speeds up the development cycle, enabling quicker delivery of features and updates to your users while increasing consistency and reliability.

See you next time!

Deploy a Django application to AWS Lambda using Serverless Framework

João Marcos — Thu, 14 Dec 2023 21:34:35 +0000

If you have a Django application and want an easy and fast way to deploy it to the AWS Lambda service, you can use the Serverless Framework. It helps you deploy AWS Lambda functions with the necessary resources.

The instructions in the Framework’s documentation are very clear. If you haven’t installed it yet, you can follow the steps outlined in this link to do so.

For this demonstration, I developed a simple Django application. It has only one model and leverages DRF (Django REST Framework) to provide a REST API. You can check out all the code used in this post by accessing this GitHub repository.

Creating the Serverless service

To create a new Serverless service you can run the sls create command and pass a template as a parameter. You can see a list of template examples in the official sls repository. As I write this post, there isn’t a Django template available in this list. Therefore, we are going to create a serverless.yml file in the root directory of our project and manually write the code we need in it.



service: django-serverless
frameworkVersion: '3'
useDotenv: true

provider:
  name: aws
  region: ${env:AWS_REGION_NAME}
  runtime: python3.9
  stage: ${opt:stage, 'stg'}
  timeout: 30
  memorySize: 2048

The code above have defined the name of our service and some basic configuration (timeout, memory allocation, AWS region) of the lambda function that will be created when we first deploy our application.

Serverless’ Plugins

A Serverless plugin is custom JavaScript code that extends the Serverless Framework with new features. To deploy our Django application we will need to install a few plugins and you’ll learn to do that in the next sections.

Plugin: `serverless-wsgi`

Django’s primary deployment platform is WSGI (Web Server Gateway Interface), which is a specification that describes how a web server communicates with web applications.

API Gateway, which is going to receive the requests to our application and invoke the Lambda function, does not natively support WSGI-based applications. We need to convert API Gateway requests to and from standard WSGI request.

To accomplish this task, we’ll install our first plugin: serverless-wsgi.



sls plugin install -n serverless-wsgi

After you run this command in the project's root directory via the terminal, you’ll notice that a new section was created within our serverless.yml file:



plugins:
  - serverless-wsgi

All plugins we install will be declared in this section.

Now we can define the creation of our API Gateway and Lambda function:



functions:
  api:
    handler: wsgi_handler.handler
    events:
      - http: ANY /
      - http: ANY /{proxy+}

The API Gateway will act as a transparent proxy, passing all requests it receives directly to the lambda function. Our Django application will be responsible for matching them with the appropriate endpoint. The serverless-wsgi plugin requires the function to have wsgi_handler.handler set as the Lambda handler.

The plugin also requires extra configuration and the place to add that is under the custom section in serverless.yml



custom:
  wsgi:
    app: myapi.wsgi.application

As I mentioned before, Django is configured for WSGI by default. The application callable, which the application server uses to communicate with our code, is created when you run the Django’s startproject command. By default, its’s path is set to <project_name>.wsgi.application.

Plugin: `serverless-python-requirements`

We need to bundle the dependencies our application requires to work properly. This plugin does this and make them available in PYTHONPATH.



sls plugin install -n serverless-python-requirements

Again, the command will add the plugin to the plugins section of our serverless.yml file.That’s all that’s needed for basic use, python dependencies specified in requirements.txt will be bundled when you run the command to deploy the app.

You might have noticed that the Django app I’m using for this example has a folder requirements with two files: dev.txt and prod.txt. In this case, we’ll need to explicitly point out the file that will be read by the plugin. We do that by editing the custom section again:



custom:
  wsgi:
    app: myapi.wsgi.application
  pythonRequirements:
    fileName: requirements/prod.txt

Plugin: `serverless-dotenv-plugin`

This plugin will automatically load the variables stored in the .env file into the Lambda function as environment variables.



sls plugin install -n serverless-dotenv-plugin

Packaging

Serverless Framework packages up our code into a zip file. You can run sls package to build and save the deployment artifact in the service’s .serverless/ directory.

Some files and/or folders that are in our app’s directory are not needed in the lambda function. To prevent them from getting bundled up during the deploy we’ll use the configuration bellow:



package:
  exclude:
    - venv/**
    - __pycache__/*
    - node_modules/**
    - README.md
    - pytest.ini
    - conftest.py
    - .venv
    - .venv.example
    - package.json
    - package-lock.json

This is how our serverless.yml file looks like at this point:



service: django-serverless
frameworkVersion: '3'
useDotenv: true

provider:
  name: aws
  region: ${env:AWS_REGION_NAME}
  runtime: python3.9
  stage: ${opt:stage, 'stg'}
  timeout: 30
  memorySize: 2048

functions:
  api:
    handler: wsgi_handler.handler
    events:
      - http: ANY /
      - http: ANY /{proxy+}

plugins:
  - serverless-wsgi
  - serverless-python-requirements
  - serverless-dotenv-plugin

custom:
  wsgi:
    app: myapi.wsgi.application
  pythonRequirements:
    fileName: requirements/prod.txt

package:
  exclude:
    - venv/**
    - __pycache__/*
    - node_modules/**
    - README.md
    - pytest.ini
    - conftest.py
    - .venv
    - .venv.example
    - package.json
    - package-lock.json

Deployment

To make our first deployment we need a way to authenticate with our AWS account. I’ll use an Access Key from an IAM User that I’ve created in my AWS account. You can learn more about this here.

Once you have the “secret” and “id” values of your credentials, create the following env vars in the terminal window where the deploy command will be run:



export AWS_ACCESS_KEY_ID="your_key_id_value"
export AWS_SECRET_ACCESS_KEY="your_secret_key_value"

Now you can run:



sls deploy

The deploy command bundles up the application and creates a CloudFormation stack that manages all the resources within the AWS account. If you’re familiar with CloudFormation, exploring the stack info will provide deeper insights into the underlying infrastructure.

The Lambda Function

Now that the the Lambda function was created and is triggered by an API Gateway, we can access our API with the link provided by the deploy command.

The application is working, but it looks weird. Something is missing…

The static files

Websites generally need to serve additional files such as images, JavaScript, or CSS. In Django, we refer to these files as “static files”. Learn more.

There are a few different approaches we can employ to serve the static files. A commonly used tactic is to serve files from a cloud storage provider like AWS’ S3 and that’s the method we will implement.

Let’s configure Django to use a custom file storage backend to integrate with S3. The django-stogages python lib provides exactly what we need, you can install it using pip:

The Django app I’m using as example already has this lib in the requirements file. You’ll also notice that the changes required in settings.py are already implemented.



pip install django-storages

Then add add the code bellow to the myapi/settings.py file.



STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
    },
    "staticfiles": {
        "BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
    },
}

AWS_STORAGE_BUCKET_NAME = os.environ.get("STATIC_FILES_BUCKET_NAME")
AWS_S3_REGION_NAME = os.environ.get("AWS_REGION_NAME")
AWS_QUERYSTRING_AUTH = False

Setting AWS_QUERYSTRING_AUTH to False removes query parameter authentication from generated URLs, which we don’t need because we’ll use a public bucket. The name of the bucket to which Django will send the static files and the AWS region will be read from environment variables. Add those values to your .env file:



STATIC_FILES_BUCKET_NAME=YOUR_BUCKET_NAME_HERE
AWS_REGION_NAME=us-east-1
DB_NAME=
DB_USER=postgres
DB_PASSWORD=
DB_HOST=
DB_PORT=5432

The next step is to create the S3 bucket that will store and serve the static files. Using the Serverless Framework, we can define AWS infrastructure resources we need and easily deploy them. Those resources can be defined in a property titled resources in serverless.yml. What goes in this property is raw CloudFormation template syntax in YAML. So let’s create a public S3 bucket:



resources:
  Resources:
    StaticFilesBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${env:STATIC_FILES_BUCKET_NAME}
        PublicAccessBlockConfiguration:
          BlockPublicAcls: false
          BlockPublicPolicy: false
          IgnorePublicAcls: false
          RestrictPublicBuckets: false

    StaticFilesBucketPolicy:
      Type: AWS::S3::BucketPolicy
      Properties:
        Bucket:
          Ref: StaticFilesBucket
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Sid: PublicReadGetObject
              Action: "s3:GetObject"
              Effect: Allow
              Principal: "*"
              Resource: "arn:aws:s3:::${env:STATIC_FILES_BUCKET_NAME}/*"

The Lambda function needs permission to upload the static files to the bucket. This permission can be set via IAM Role, which Serverless Framework automatically creates for our service. We can modify this Role to add permissions to the code running in our function as we need them. Add the code bellow under the provider property:



iam:
    role:
      statements:
        - Effect: Allow
          Action:
            - s3:*
          Resource:
            - arn:aws:s3:::${env:STATIC_FILES_BUCKET_NAME}
            - arn:aws:s3:::${env:STATIC_FILES_BUCKET_NAME}/*

This is the final version of our serverless.yml file:



service: django-serverless
frameworkVersion: '3'
useDotenv: true

provider:
  name: aws
  region: ${env:AWS_REGION_NAME}
  runtime: python3.9
  stage: ${opt:stage, 'stg'}
  timeout: 30
  memorySize: 2048
  iam:
    role:
      statements:
        - Effect: Allow
          Action:
            - s3:*
          Resource:
            - arn:aws:s3:::${env:STATIC_FILES_BUCKET_NAME}
            - arn:aws:s3:::${env:STATIC_FILES_BUCKET_NAME}/*

functions:
  api:
    handler: wsgi_handler.handler
    events:
      - http: ANY /
      - http: ANY /{proxy+}

plugins:
  - serverless-wsgi
  - serverless-python-requirements
  - serverless-dotenv-plugin

custom:
  wsgi:
    app: myapi.wsgi.application
  pythonRequirements:
    fileName: requirements/prod.txt

package:
  exclude:
    - venv/**
    - __pycache__/*
    - node_modules/**
    - README.md
    - pytest.ini
    - conftest.py
    - .venv
    - .venv.example
    - package.json
    - package-lock.json

resources:
  Resources:
    StaticFilesBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${env:STATIC_FILES_BUCKET_NAME}
        PublicAccessBlockConfiguration:
          BlockPublicAcls: false
          BlockPublicPolicy: false
          IgnorePublicAcls: false
          RestrictPublicBuckets: false

    StaticFilesBucketPolicy:
      Type: AWS::S3::BucketPolicy
      Properties:
        Bucket:
          Ref: StaticFilesBucket
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Sid: PublicReadGetObject
              Action: "s3:GetObject"
              Effect: Allow
              Principal: "*"
              Resource: "arn:aws:s3:::${env:STATIC_FILES_BUCKET_NAME}/*"

Run sls deploy again to update the stack. You can access the S3 page of your AWS account to verify the bucket created. If the deploy command was successful, you should be able to see 2 buckets: the one explicitly created by us and another implicitly generated by Serverless Framework to store the deployment artifacts of our application.

Now we only need to run Django’s collectstatic command to upload the static files to the bucket. Fortunately, serverless-wsgi plugin already provides an easy way for remote execution of Django management commands, the wsgi manage command. Run in your terminal window:



sls wsgi manage --command "collectstatic --noinput"

Run the migrations:



sls wsgi manage --command "migrate"

You can access /api/books to interact with the app using DRF’s Browsable API.

Create a `package.json` file

It is important to have a package.json file to track the dependencies our project has. In our case, those dependencies are the Serverless' plugins we installed. To create the file you can run npm init. The dependencies installed in node_modules will be read and used to create the file.



{

  "name": "django-serverless",

  "version": "1.0.0",

  "description": "",

  "dependencies": {

    "serverless-wsgi": "^3.0.3",

    "serverless-dotenv-plugin": "^6.0.0",

    "serverless-python-requirements": "^6.0.1"

  },

  "devDependencies": {},

  "repository": {

    "type": "git",

    "url": "git+https://github.com/joao-marcos/django-serverless.git"

  },

  "author": "João Marcos"

}

What’s Next?

In the next blog post we’ll build a pipeline using GitLab CI to automate deployments whenever new code is pushed to the repository. See you next time!