DEV Community: Daiquiri Team

How to successfully hold a demonstration for an IoT product without any hardware

Volodymyr Ryzhkov — Mon, 17 Oct 2022 14:49:45 +0000

Why is this necessary? Several teams often work on the product, and sometimes one of the teams does not have time to complete its part. It was our case: we were developing the web and back-end parts of the IoT system, and the hardware team on the client's side did not meet the deadline.

At the same time, the product owner has already scheduled demonstrations with customers, which he would not like to postpone because some meetings can't be scheduled a second time, or the impression will be spoiled. But we found a way to get out of this situation and meet all the deadlines with little effort.

About the product

Our client's company manufactures smart scales. In our collaboration, they were trying to optimize the process of controlling the storage of products in the sales halls of supermarkets, and the idea of their product was as follows:

Demonstration process

A test shelf had to be installed in one of the supermarkets for a successful demonstration of the system. It was also needed so that our team could test all the subsystems that should work with the shelf.

According to the demonstration plan, the client had to show the process of adding a new store and its employees to the system and adding shelves to the selected store. Real-time shelf data was displayed on a separate dashboard, and informative reports on shelf performance could also be created and viewed. In case of deviation of temperature or insufficient quantity of goods, employees were notified and were able to solve the problem with the shelf.

Problem solving

What ways out of the situation did we have? We could do nothing because the responsibility for the failure of the demo was not on our side. At the same time, delaying the project delivery date is always unpleasant. We will send the invoice later, there are risks that we will have to rework some tasks later, and the team members have already lost the context and are working on the tasks of other projects. Even more critical that our client got into an unpleasant situation because, without a shelf, it was almost impossible to hold a demonstration successfully.

So yeah, we just decided to create a software equivalent of a physical shelf that would do all the same things as a real shelf when finished.

We started with requirements development because we had to consider many details regarding the work of the shelf. First, we thought about how customers and employees will interact with the shelf. They can accidentally lean on the shelf, which can cause the system to get false statistics. You can also forget to close the refrigerator door, and the temperature will start to rise. After working through possible user scenarios, we decided that the shelf, in this case, could be replaced by a separate (or not separate) application that would be able to easily add a large number of new shelves and send data about the operation of each shelf. We immediately informed the client about this idea and that when we do it, it will be possible to run a demo using our test application only. It will be as relevant as possible to how our admin panel will work with real shelves.

The work took us a couple of days of teamwork. We simulated the possibility of turning off the shelf or losing connection with it with a probability of 1% every minute. Also, every minute a random amount of goods was "taken" from the shelf, and if the shelf remained less than 30% loaded, with a probability of 50% it was filled again. The typical storage temperature range of the product was from 2 to 6 degrees Celsius, so every minute we increased the temperature with a probability of 5%, and when the temperature exceeded the specified range, the employee received a notification. We also considered the possibility of sending false indicators, which the system had to filter and not take into account in the general work statistics.

With the help of the software shelf, our team tested all the necessary subsystems and demonstrated the operation of the product to the client. A few days later, our client successfully held the demonstration to their customers and fully validated the product hypotheses. When the physical shelf was ready after two weeks, we added it to the system. We ensured that our software counterpart worked similarly to the physical shelf, which again emphasized the correctness of the chosen solution to the problem.

Conclusion

We at Daiquiri Team believe that the development team must be aware of the client's tasks and problems, even if they are outside the competence of the developers. This is how the team gets enough background to understand the entire client's problem and be able to solve it completely. Saying that you can't continue work because third-party system components are not ready is not expedient. Components can always be replaced with another solution.

As a developer, I truly believe that all developers should care about the end customer of the product, at least partly. I participated in many projects, and I feel it from real life, so if you want to work with me on your IoT project or any other projects — write me directly or contact us on the website, and we will be glad to discuss and solve your problem.

How to build the SSR web application and mobile apps from the same code base

Yevhen Bondar — Sun, 25 Sep 2022 10:03:17 +0000

Server-side rendering improves initial page load speed. The browser receives pre-rendered HTML from the server, so it takes less time to build the page.

Also, SSR sites play better with SEO. Search engine crawlers grab the site's HTML content without executing JS code. Without SSR, search engines will not index page content and will not find internal links.

NuxtJS has a Server-side rendering feature:

Server-side rendering (SSR), is the ability of an application to contribute by displaying the web-page on the server instead of rendering it in the browser. Server-side sends a fully rendered page to the client.

But you cannot use SSR when you build a static site for the mobile application. Here I'll show how to create an SSR web and cross-platform mobile application from the same code base using CapacitorJS and NuxtJS.

Building the app

As an example, we will build a simple app to show local time in Kyiv.

To test SSR, we need to get data from the API server. We will receive data from worldtimeapi.org API using the useFetch method.

Let's start with the application from the previous guide.

Remove app.vue file and add pages/index.vue with the following content:



<template>
  <div>
    <h1>🇺🇦 Kyiv Time</h1>

    <p>
      🕥 Current time: <b>{{ timeData.datetime }}</b>
      <a @click="$nuxt.refresh()" href="">🔄</a>
    </p>
  </div>
</template>

<script setup>
const { data: timeData } = await useFetch('https://worldtimeapi.org/api/timezone/Europe/Kiev')

</script>

Here we have:

Retrieving current time in const timeData variable.
🕥 displaying time on the page.
🔄 refresh button.

Run yarn dev and check http://localhost:3000/ what we have:

Let's look at the page source code:

In NuxtJS, Server-side rendering is enabled by default. But what will we get if we create a static site for the mobile build? Run yarn generate && yarn preview and recheck http://localhost:3000/.

If you try to refresh the page, you will see that time doesn't change. NuxtJS retrieved time from the server during static site generation and saved it in the pre-rendered HTML page.

Let's add ssr: false to the nuxt.config.ts



import { defineNuxtConfig } from 'nuxt'

// https://v3.nuxtjs.org/api/configuration/nuxt.config
export default defineNuxtConfig({
    ssr: false,
})

and rebuild the site with yarn generate && yarn preview.

Now NuxtJS runs HTTP requests on the client side. There is no datetime in the page sources, and datetime changes every time you refresh a page.

So, we must build a web application with SSR and a mobile application in client-only mode. Let's use an environment flag MOBILE_BUILD: we will build a web application with MOBILE_BUILD=0 and a mobile application with MOBILE_BUILD=1.

You can use a .env file to store environment variables, and I'll go with export MOBILE_BUILD=1.

Change nuxt.config.ts:



import { defineNuxtConfig } from 'nuxt'

// https://v3.nuxtjs.org/api/configuration/nuxt.config
export default defineNuxtConfig({
    ssr: !process.env.MOBILE_BUILD,
})

Run yarn generate && yarn preview and verify that the application works in client-only mode.

Finally, let's build and check the Android application:



yarn cap sync
yarn cap open android

Run the application on an emulator and check that the refresh button works.

🎉 Congratulations, it works (on my laptop, at least 🙂).

Makefile for your Django project

Yevhen Bondar — Sun, 18 Sep 2022 16:52:41 +0000

During the development process, you often need to run some commands in your terminal: create migrations, run tests, linters, etc. Usually, you execute these commands regularly.

It's helpful to have shortcuts for such commands. And even better to share them among other developers on the project. For this goal, I use Makefile. Here is the list of useful commands for a Django project.

Installing dependencies

I'm not a big fan of Poetry. It has some problems with Dependabot and doesn't play with Renovate at all. I use pip-tools with separate requirements.in and requirements-dev.in for local and prod environments.

pip-install-dev:
    pip install --upgrade pip pip-tools
    pip-sync requirements.txt requirements-dev.txt

pip-install:
    pip install --upgrade pip pip-tools
    pip-sync requirements.txt

pip-update:
    pip install --upgrade pip pip-tools
    pip-compile requirements.in
    pip-compile requirements-dev.in
    pip-sync requirements.txt requirements-dev.txt

Commands:

pip-install-dev: Installs all requirements, including local. It is the main command for installing the project's dependencies, so I put this command at the top.
pip-install - I don't use this command often, but it could be helpful to run your code with production dependencies only.
pip-update Updates your requirements.txt and requirements-dev.txt after you add a new package to the requirements.in or requirements-dev.in.

Running the project

server:
    python manage.py migrate && python manage.py runserver

worker:
    python -m celery -A project_name worker --loglevel info

beat:
    python -m celery -A project_name beat --loglevel info

That is an obvious one. The commands to run local web server and Celery. Also, I like to apply new migrations automatically when I run the server.

Running linters

lint:
    flake8 palyanytsya
    mypy palyanytsya

black:
    python -m black palyanytsya

cleanimports:
    isort .
    autoflake -r -i --remove-all-unused-imports --ignore-init-module-imports project_name

clean-lint: cleanimports black lint

checkmigrations:
    python manage.py makemigrations --check --no-input --dry-run

Commands:

lint: Runs flake8 linter and mypy type checker.
black: Automatic code formatting with Black.
cleanimports: runs isort and removes unused imports with Autoflake. Be sure to set up profile=black in isort settings to avoid conflicts with Black.
clean-lint: run all the stuff above. You can run this command before committing to formatting your code properly.
checkmigrations: prevents you from committing model changes without migrations. Really cool stuff!

Also, I use make lint && make checkmigrations in the CI pipeline and in the git pre-commit hook. You can also create a command for setting up such a hook:

install-hooks:
    echo "make lint && make checkmigrations" > .git/hooks/pre-commit && chmod 777 .git/hooks/pre-commit

Running tests

test:
    pytest -n 4 -x

Runs pytest in multiprocess mode using pytest-xdist.

Compiling messages

messages:
    python manage.py makemessages --all --ignore=venv --extension html,py
    python manage.py translate_messages -s en -l uk -l es -u
    python manage.py compilemessages --ignore venv

Collects all string literals for translation. Also, I use Django Autotranslate to translate phrases using Google Translate automatically.

The end

As long as the project goes on, new local commands appear. Maybe, you need to shell into the dev server to run some python code. Or download a fresh database dump (without sensitive data) for local bug reproducing. It's better to keep this knowledge in the codebase, not in your head. Also, you won't need to explain how to do exactly the same operation to your colleagues.

For more complex tasks, you can even create a shell script and call it from Makefile. So, it would be easier to find this new command for other devs.

Creating a custom page in Django Admin

Yevhen Bondar — Sat, 10 Sep 2022 10:13:06 +0000

Django is a great web framework for fast development. Django Admin allows you to manage your data without creating your own views.

The primary use case of the Admin is CRUD operations. You can populate your local database with test data or change some real data in the production environment.

Sometimes, you need some custom interfaces to perform your routines on the project. Fortunately, Django Admin has a lot of room for customization. In this guide, I'll show how to create a custom detail page in Django Admin.

Basic setup

Let's start from scratch. I'll use python3.9 in this guide. If you are not interested in the basic details, you can jump to the custom Order page part.

First, create a virtual environment and install Django. Also, we need Pillow for Django's ImageField:

python3.9 -m venv venv 
. ./venv/bin/activate
pip install django==4.1.1 Pillow==9.2.0

Then, create a new Django project and product app inside.

python -m django startproject django_admin_example .
python -m django startapp products

It will be a simple shop application with Product, Order, and OrderItem models. Order can have several OrderItem and belongs to some User. Each OrderItem contains the Product and count—quantity of the Product in the Order.

Add these models in products/models.py:

from django.contrib.auth import get_user_model
from django.db import models

User = get_user_model()


class Product(models.Model):
    title = models.CharField(max_length=255)
    price = models.IntegerField()
    image = models.ImageField()

    def __str__(self):
        return self.title


class Order(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    created = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"{self.user} {self.created.date()}"


class OrderItem(models.Model):
    order = models.ForeignKey(Order, on_delete=models.CASCADE)
    product = models.ForeignKey(Product, on_delete=models.CASCADE)
    count = models.PositiveIntegerField()

    def __str__(self):
        return f"{self.product} x{self.count}"

Let's include the product app in django_admin_example/settings.py. Also, we need to set MEDIA_URL and MEDIA_ROOT for file uploading:

INSTALLED_APPS = [
    # Other apps
    "products",
]

...

MEDIA_URL = 'media/'
MEDIA_ROOT = BASE_DIR / 'media/'

Now we can create and run migrations:

python manage.py makemigrations
python manage.py migrate

We are done with the models. Moving to the Admin part. Create a products/admin.py with the following content:

from django.contrib import admin
from products.models import Product, OrderItem, Order


@admin.register(Product)
class ProductAdmin(admin.ModelAdmin):
    list_display = ['title', 'price']


class OrderItemInline(admin.TabularInline):
    model = OrderItem

@admin.register(Order)
class OrderAdmin(admin.ModelAdmin):
    list_display = ['user', 'created']
    inlines = [OrderItemInline]


@admin.register(OrderItem)
class OrderItemAdmin(admin.ModelAdmin):
    list_display = ['order', 'product', 'count']

We created OrderItemInline to change order items on the Order form.

Next, we need to add a URL configuration for the admin site in urls.py. Also, we want to add media file URLs to serve images:

from django.contrib import admin
from django.urls import path
from django.conf import settings
from django.conf.urls.static import static

urlpatterns = [
    path('admin/', admin.site.urls),
    *static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT),
]

Finally, we need a superuser to login into the admin:

$ python manage.py createsuperuser
Username (leave blank to use 'user'): admin
Email address: admin@test.com
Password: 
Password (again): 
Superuser created successfully.

Run python manage.py runserver, go to the http://127.0.0.1:8000/admin/, and log in with the admin user.

Now, it's time to fill our database with some dummy data!

Let's add a customer user;

Some product to our shop;

And create an order for the customer.

Custom Order Page

Now we are ready to create a page for order summary. We want customer information, a list of ordered products, and a total order sum on this page.

Let's start with an empty template in products/templates/admin/products/order/detail.html. Django Admin enforces this structure for the templates: admin/APP_NAME/MODEL_NAME/some_template.html. So, all custom templates for the Order model will be in this folder.

Then, let's create an OrderDetailView in admin.py:

from django.contrib.auth.mixins import PermissionRequiredMixin
from django.views.generic.detail import DetailView
from products.models import Order

class OrderDetailView(PermissionRequiredMixin, DetailView):
    permission_required = "products.view_order"
    template_name = "admin/products/order/detail.html"
    model = Order

Add this view to the OrderAdmin.get_urls, create the detail column and add it to the list_display:

from django.contrib import admin
from django.urls import path, reverse
from django.utils.html import format_html
from products.models import Order


@admin.register(Order)
class OrderAdmin(admin.ModelAdmin):
    list_display = ['user', 'created', 'detail']
    inlines = [OrderItemInline]

    def get_urls(self):
        return [
            path(
                "<pk>/detail",
                self.admin_site.admin_view(OrderDetailView.as_view()),
                name=f"products_order_detail",
            ),
            *super().get_urls(),
        ]

    def detail(self, obj: Order) -> str:
        url = reverse("admin:products_order_detail", args=[obj.pk])
        return format_html(f'<a href="{url}">📝</a>')

OrderDetailView will accept pk as an argument. Also, we wrap the view into admin_site.admin_view. This wrapper checks that user is logged in, user.is_staff=True, and enforces CSRF validation.

Let's check how it looks:

Creating a template

Now the detail link leads to an empty page. Let's create a real one.

As docs says:

If you want to use the admin layout, extend from admin/base_site.html:

Add this code to the detail.html and check the page in your browser:

{% extends 'admin/base_site.html' %}

{% block content %}

    <h1>Order by {{ object.user }} {{ object.created.date }}</h1>

    <dl>
        <dt>Name:</dt>
        <dd>{{ object.user.get_full_name }}</dd>
        <dt>Email:</dt>
        <dd>{{ object.user.email }}</dd>
    </dl>

{% endblock %}

The page looks like a Django Page, but we are missing some components on the page: breadcrumbs, sidebar, and change password/logout buttons. You can check admin/base.html and admin/change_form.html to see how they are implemented.

First, we need a context for the template. Let's override a OrderDetailView.get_context method:

class OrderDetailView(PermissionRequiredMixin, DetailView):
    ...

    def get_context_data(self, **kwargs):
        return {
            **super().get_context_data(**kwargs),
            **admin.site.each_context(self.request),
            "opts": self.model._meta,
        }

This will be enough for the sidebar and change password/logout buttons. Now, let's add the breadcrumbs block to the detail.html. We can take the breadcrumbs block from the admin/change_form.html and modify it like this:

{% load i18n admin_urls %}

{% block breadcrumbs %}
    <div class="breadcrumbs">
        <a href="{% url 'admin:index' %}">{% translate 'Home' %}</a>
        &rsaquo; <a href="{% url 'admin:app_list' app_label=opts.app_label %}">{{ opts.app_config.verbose_name }}</a>
        &rsaquo; <a href="{% url opts|admin_urlname:'changelist' %}">{{ opts.verbose_name_plural|capfirst }}</a>
        &rsaquo; {{ object }}
    </div>
{% endblock %}

Finally, let's add a product table. We need a total amount of Order, so create a total_amount method for OrderItem and Order:

class Order(models.Model):
    ...

    @property
    def total_amount(self):
        return sum([item.total_amount for item in self.orderitem_set.all()])

class OrderItem(models.Model):
    ...

    @property
    def total_amount(self):
        return self.product.price * self.count

Add the table and the total to the detail.html:

{% block content %} 
    ...

    <table style="width: 100%; margin-bottom: 10px;">
        <thead>
        <tr>
            <td></td>
            <td>Product</td>
            <td>Price</td>
            <td>Count</td>
            <td>Total</td>
        </tr>
        </thead>
        <tbody>
        {% for item in object.orderitem_set.all %}
            <tr>
                <td><img src="{{ item.product.image.url }}" height="50"></td>
                <td>{{ item.product.title }}</td>
                <td>${{ item.product.price }}</td>
                <td>x{{ item.count }}</td>
                <td>${{ item.total_amount }}</td>
            </tr>
        {% endfor %}
        </tbody>
    </table>

    <p style="font-size: 18px;text-align: end;padding-right: 60px;">
        Total: ${{ object.total_amount }}
    </p>

{% endblock %}

Check the final result:

The end

Congratulations! We added a custom page to the Django Admin. This is a way you can customize the Admin Django according to your project needs.

You can find the source code here.

If you need to build a custom web application, check out our website or connect with me directly on LinkedIn.

How to create Android and iOS apps from the NuxtJS application using CapacitorJS

Yevhen Bondar — Wed, 31 Aug 2022 15:02:08 +0000

In this guide, I will show how to wrap an existing NuxtJS web application into Android and iOS mobile apps using CapacitorJS

Capacitor is an open-source native runtime for building Web Native apps. Create cross-platform iOS, Android, and Progressive Web Apps with JavaScript, HTML, and CSS.

CapacitorJS runs your static site on a local browser and shows it to the user. Also, it has some interface between native and web contexts for accessing native features.

But why do you need this if users can access your website from mobile browsers?

It's more useful for users. If a user uses your web app frequently, it's easier for them just to click on your app icon than go to the browser and search for your app bookmark or type your app domain name. Both Android and iOS have the feature "Add website to home screen". But a separate native app provides better UX.
Distribution channel. Users can search for your app in AppStore and PlayMarket. So, it's better to be present on all popular platforms to reach more users.
Native features. Maybe, your web app doesn't need them. But after you create a mobile version of your app, you can change your mind. At least you can add notifications by FCM.

I think we are done with the "why?" part. Let's go to the "how?" part.

Creating dummy NuxtJS application

To simplify this guide, I'll create a new Nuxt application. Feel free to jump to the Installing CapacitorJS section if you already have one.

So, let's create a nuxt3 application using the official guide. I'll use NodeJS v16 and Yarn v1.22 for installation.

Open a terminal and run:



$ yarn add global nuxi
$ yarn nuxi init nuxt-mobile
$ cd nuxt-mobile
$ yarn install

Here we installed nuxi Nuxt Command Line Interface, created a new project nuxt-mobile, and installed dependencies from package.json. You can replace nuxt-mobile with any name.

Next, let's change app.vue. Replace the existing file with the following one:



<template>
  <div>
    <h1>Nuxt Mobile</h1>

    <p>This is a basic example of a mobile app built with NuxtJS and CapacitorJS</p>
  </div>
</template>

Now, let's see what we got. Run the development server:



$ yarn dev -o

You will be navigated to the browser. Here you will see your web application.

It's pretty empty, but it will be enough to show how to wrap a web application into mobile apps.

Installing CapacitorJS

Now, let's add CapacitorJS to the application. I'll follow instructions from the official guide:

In the root of your app, install Capacitor's main npm depdencies: the core JavaScript runtime and the command line interface (CLI).



$ yarn add @capacitor/core
$ yarn add --dev @capacitor/cli

Then, initialize Capacitor using the CLI questionnaire:



$ yarn cap init

The CLI will ask you questions about your package. You can pick any Name and Package (I used NuxtMobile and com.mycompany.nuxtmobile values). Then you need to set the Web asset directory as dist. This is a destination on NuxtJS static site. The CLI will save these answers into capacitor.config.json and check the file content.

After the Capacitor core runtime is installed, you can install the Android and iOS platforms.



$ yarn add @capacitor/android @capacitor/ios

CapacitorJS uses a static version of the site. So, we need to generate it:



$ yarn generate

Now we have a static site so that we can initialize iOS and Android applications:



$ yarn cap add android
$ yarn cap add ios

These commands create native applications in android and ios folders. They clone web content from the dist directory to ios/App/App/public and android/app/src/main/assets/public directories.

Every time you change your site, you need to run yarn generate to generate a new static site and npx cap sync to update the native apps.

Running Android app

Let's run the Android application. First, download and install AndroidStudio.

After you finish, return to the console and run yarn cap open android. This will open android folder in AndroidStudio. Here you can change your Android files and run the application.

Let's add an emulator to run. Go to the "Tools" -> "Device Manager" and click "Create Device."

First, you need to pick "Hardware device." The main difference between different devices is a screen size and availability of PlayMarket. I'll pick the Pixel 2 because it has PlayMarket and a 5.0" screen size.

Then, you need to choose the Android OS image. I will use Android API 33.

Finally, click 'Next' and 'Finish'. After this, you will see a new device on the 'Devices' dropdown.

Now, you can choose your device and click 'Run' to build and start the application:

Android Studio will start the emulator, build, install and run your application:

We are done with Android; let's move to the iOS part.

Running iOS app

To build the application for iOS, you need a MacBook. If you have Linux or Windows, you can skip this part.

First, you need XCode to build and run the iOS application. You can get it on AppStore.

After installation, return to the console and run yarn cap open ios. This command will open the ios folder as an XCode project. Here you can change iOS application files and settings; build and run the application.

Now, we need to pick an emulator device to run the application. I'll use the iPod touch.

Then click 'Run'. XCode will build the app and run it on the device.

We are done with the iOS part.

The end

Congratulations! In this guide, we turn a new NuxtJS web application into iOS and Android mobile applications. It's a very basic web app so we wouldn't upload it to stores :) But if you already have a full-featured web application, you can turn it into mobile apps and upload them to PlayMarket and AppStore!

You can find the source code of the project here.

If you need to build a web or mobile app prototype, check out our website or connect with me directly on LinkedIn.

Deploying Django Application on AWS with Terraform. ECS Autoscaling

Yevhen Bondar — Tue, 23 Aug 2022 13:18:22 +0000

This is the 7th part of the "Deploying Django Application on AWS with Terraform" guide. You can check out the previous steps here:

In this part, we'll make our Django web application scalable using ECS Autoscaling.

Autoscaling is the ability to increase or decrease the number of running instances. It allows you to handle traffic spikes and save money for low intensive periods of time.

When you enable autoscaling for ECS service, AWS creates Cloudwatch alarms to determine whether we need to add a new instance or remove a redundant one.

Let's see how it works in practice.

ECS Autoscaling configuration

First, create a new autoscale.tf with the following content:

resource "aws_appautoscaling_target" "prod_backend_web" {
  max_capacity       = 5
  min_capacity       = 1
  resource_id        = "service/${aws_ecs_cluster.prod.name}/${aws_ecs_service.prod_backend_web.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

resource "aws_appautoscaling_policy" "prod_backend_web_cpu" {
  name               = "prod-backend-web-cpu"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.prod_backend_web.resource_id
  scalable_dimension = aws_appautoscaling_target.prod_backend_web.scalable_dimension
  service_namespace  = aws_appautoscaling_target.prod_backend_web.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
    target_value = 80
  }

  depends_on = [aws_appautoscaling_target.prod_backend_web]
}

Here we defined:

AWS AppAutoscaling Target. We want to scale the prod_backend_web service by instance count dimension from 1 to 5.
AWS AppAutoscaling Policy. We want to scale prod_backend_web up when the ECSServiceAverageCPUUtilization metric exceeds 80%.

We are ready to apply changes, but first, let's think about load balancer health checks.

Load Balancer Health Checks

Now we have a quire aggressive health checks. If the container fails to respond twice with a timeout of 2 seconds, Load Balancer considers this container unhealthy and removes them.

It could be an okay solution for the little amount of traffic. But if many requests reach the container and CPU usage goes up to 100%, the container will fail to respond to Health Checks. So, Load Balancer kills them, and we will face an even worse situation: there will be no containers to handle the traffic at all

The possible solution is to increase health checks timeout and unhealthy_threshold. Thus, there will be more possibility for overloaded containers to survive.

I think it's not a perfect solution, but it will work for this test. If you know a more elegant way to keep overloaded containers running, feel free to leave a comment.

Go to the load_balancer.tf and increase unhealthy_threshold, timeout, and interval parameters.

# Target group for backend web application
resource "aws_lb_target_group" "prod_backend" {
  ...

  health_check {
    ...
    unhealthy_threshold = 5
    timeout             = 29
    interval            = 30
    ...
  }
}

Let's apply our changes and check them at the AWS console.

CloudWatch Alarms

First, go to the ECS console and check the autoscaling policy for the prod_backend_web ECS Service. Select prod ECS cluster, select prod-backend-web service and click "Update". Pass to the step "Set Auto Scaling" and click on the prod-backend-web-cpu autoscaling policy.

Here we see that autoscaling becomes effective when average CPU utilization reaches 80%. But what is the condition for scaling down? Let's check CloudWatch alarms associated with this autoscaling policy.

Go to the CloudWatch console and look at the alarms.

Here we see that we scale up when the average CPU load exceeds 80% during 3+ minutes. And, we scale down when the average CPU load goes less than 72% for 15 minutes.

Such specific numbers, but how can we adjust them to our case? For this, you need to create and use custom metrics for alarms with customized_metric_specification param in aws_appautoscaling_policy.

Also, you can change AlarmHigh and AlarmLow metrics manually in the console. It's not a preferable way to create a repeatable setup, but it's okay for our test. So, I'll change the AlarmLow metric to 50% and 10 minutes.

Stress Testing

Let's move to the tests. I'll use ApacheBenchmark for stress testing. This tool can send a lot of requests to our service, so the CPU load goes up.

First, ensure that now the web service has only one container running.

Also, you need to increase the limit of open files with ulimit -n 10000.

Now we are ready to run the benchmark. We'll use the health-check URL for this test:

$ ab -n 100000 -c 1000 https://api.example53.xyz/health/

Where -c 1000 concurrent number of requests, -n 100000 is the total number of requests.

Check the CloudWatch metrics and ECS Service for the next 10-15 minutes.

First, you should see the CPU spike in the charts. After 3 minutes, ECS autoscale starts to spawn new instances.

Then, the average CPU drops below 80%. There were 3 ECS tasks at this moment of time.

After some time, CPU load exceeds 80% again, and ECS autoscale creates the 4th instance. You can see them on the ECS console.

So, scale up works; let's check scale down. Stop ApacheBenchmark and wait for 10-15 minutes to wait for scale down.

You'll see how CPU load drops to zero and ECS scales down the web service to 1 instance.

Recheck the ECS console to ensure that we have only one web task running:

So, scale down works too. Let's commit and push our changes to the infrastructure repository.

The end

Congratulations! In this part, we added ECS autoscaling for the web service. We increase Health Check timeout and period to prevent killing overloaded containers. Then, we run a stress test and verify that number of instances increases when CPU load goes up and decreases when CPU load goes down.

You can find the source code of backend and infrastructure projects here and here.

If you need technical consulting on your project, check out our website or connect with me directly on LinkedIn.

Deploying Django Application on AWS with Terraform. Connecting to Amazon S3

Yevhen Bondar — Wed, 17 Aug 2022 09:28:54 +0000

This is the sixth part of the "Deploying Django Application on AWS with Terraform" guide. You can check out the previous steps here:

In this step, we are going to:

Create an S3 bucket for storing media files.
Add S3 storage in the Django project.
Add image upload in the Django admin.
Add a S3 Terraform backend.

About S3

Why do we actually need S3 storage?

Why do we need S3 storage?
When a user uploads some file to the web server, Django, by default, saves this file in the filesystem. As far as we can have multiple containers for web ECS service, we can run into the situation when container web_1 creates the user file, and container web_2 tries to read them. Moreover, when ECS recreated container web_1, we will lose the user file.

So, now we have stateful behavior. To achieve a stateless behavior, we need to store user files in common global storage. S3 is the storage we are looking for.

In this case, container web_1 creates a new file in an S3 bucket, and container web_2 can read this file.

Here is an S3 description from AWS:

Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can store and protect any amount of data for virtually any use case, such as data lakes, cloud-native applications, and mobile apps. With cost-effective storage classes and easy-to-use management features, you can optimize costs, organize data, and configure fine-tuned access controls to meet specific business, organizational, and compliance requirements.

File upload to local storage

But first, let's add a file upload in our Django application.

To work with S3, we need to install django-storages and boto3 packages. And for handling images in Django, we need the Pillow library. Add them to requirements.txt and run pip install -r requirements.txt.

boto3==1.24.45
django_storages==1.13.1
pillow==9.1.0

But for now, we'll work with Django default FileSystemStorage.

Let's create a new Django application photos with a Photo model with image ImageField.

Navigate to the django-aws-backend project and create a new Django application with python manage.py startapp photos.

Then add the photos application in settings.py at the bottom of INSTALLED_APPS and specify MEDIA_URL and MEDIA_ROOT settings:

INSTALLED_APPS = [
    ...
    'photos.apps.PhotosConfig',
]

MEDIA_URL = "/media/"
MEDIA_ROOT = BASE_DIR / "media"

Now, we are ready to create a Photo model in photos/models.py:

from django.db import models


class Photo(models.Model):
    title = models.CharField("Title", max_length=255)
    image = models.ImageField("Image", upload_to="photos/")

To allow management of Photo objects in the admin panel, add PhotoAdmin class in photos/admin.py

from django.contrib import admin

from photos.models import Photo


@admin.register(Photo)
class PhotoAdmin(admin.ModelAdmin):
    list_display = ["title"]

Finally, to serve media files by the Django development server, add this code to the django_aws/urls.py:

from django_aws.settings import DEBUG, MEDIA_URL, MEDIA_ROOT
from django.conf.urls.static import static

...


if DEBUG:
    urlpatterns += static(MEDIA_URL, document_root=MEDIA_ROOT)

All the necessary code is in place. So we can make and apply migrations.

$ python manage.py makemigrations photos      
Migrations for 'photos':
  photos/migrations/0001_initial.py
    - Create model Photo
$ python manage.py migrate              
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, photos, sessions
Running migrations:
  Applying photos.0001_initial... OK

Now, let's try to create a new Photo in Django admin. Run python manage.py runserver and go to the admin panel http://127.0.0.1:8000/admin/photos/photo/. Click "Add Photo", enter a title, and pick any image from your filesystem. Then, click "Save and continue editing".

Django will save an image to the local filesystem. Verify that you can click on the "current image" and see it in the browser. You can also check it in the django-aws-backend/media/photos folder.

Django S3 settings

We've successfully uploaded an image to the local filesystem. I prefer to use FileSystemStorage for local development. It allows me to work and run tests without the internet connection.

But for the production environment, we need to provide S3 settings. So let's add to settings.py the following settings:

DEFAULT_FILE_STORAGE = env(
    "DEFAULT_FILE_STORAGE", default="django.core.files.storage.FileSystemStorage"
)
AWS_ACCESS_KEY_ID = env("AWS_ACCESS_KEY_ID", default="")
AWS_SECRET_ACCESS_KEY = env("AWS_SECRET_ACCESS_KEY", default="")
AWS_STORAGE_BUCKET_NAME = env("AWS_STORAGE_BUCKET_NAME", default="")
AWS_S3_REGION_NAME = env("AWS_S3_REGION_NAME", default="")
AWS_S3_ENDPOINT_URL = env("AWS_S3_ENDPOINT_URL", default="")
AWS_S3_FILE_OVERWRITE = False

We defined:

DEFAULT_FILE_STORAGE. We keep FileSystemStorage for local development and set S3Boto3Storage for production.
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY - AWS access key, as a string. We'll create a separate user and access key for accessing s3 storage.
AWS_STORAGE_BUCKET_NAME - the name of S3 bucket.
AWS_S3_REGION_NAME - AWS region. I'll use the same us-east-2 region; you can specify yours.
AWS_S3_ENDPOINT_URL - URL for connecting to S3.
AWS_S3_FILE_OVERWRITE - If the file with the specified name already exists, django-storages will append extra characters.

We are done with the Django part. The application is ready to work with S3. Commit and push the changes, and ensure that CI/CD is passed successfully.

Creating S3 bucket

Move to the Terraform project. Let's create an S3 bucket. First, we need to specify the bucket name in variables.tf. The name should be unique in your AWS region.

# S3

variable "prod_media_bucket" {
  description = "S3 Bucket for production media files"
  default = "prod-media-427861343"
}

Then, create a new s3.tf file in the django-aws-infrastructure project with the following content and run terraform apply:

resource "aws_s3_bucket" "prod_media" {
  bucket = var.prod_media_bucket
  acl    = "public-read"

  cors_rule {
    allowed_headers = ["*"]
    allowed_methods = ["GET", "HEAD"]
    allowed_origins = ["*"]
    expose_headers  = ["ETag"]
    max_age_seconds = 3000
  }

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid = "PublicReadGetObject"
        Principal = "*"
        Action = [
          "s3:GetObject",
        ]
        Effect   = "Allow"
        Resource = [
          "arn:aws:s3:::${var.prod_media_bucket}",
          "arn:aws:s3:::${var.prod_media_bucket}/*"
        ]
      },
    ]
  })
}


resource "aws_iam_user" "prod_media_bucket" {
  name = "prod-media-bucket"
}

resource "aws_iam_user_policy" "prod_media_bucket" {
  user = aws_iam_user.prod_media_bucket.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "s3:*",
        ]
        Effect = "Allow"
        Resource = [
          "arn:aws:s3:::${var.prod_media_bucket}",
          "arn:aws:s3:::${var.prod_media_bucket}/*"
        ]
      },
    ]
  })
}

resource "aws_iam_access_key" "prod_media_bucket" {
  user = aws_iam_user.prod_media_bucket.name
}

Here we created:

A new S3 bucket. We allowed public access to all files in this bucket and specified CORS rules to allow the browser to load files for any origin (and for our domain as well).
A new IAM user with the IAM policy to connect to the S3 bucket.
An IAM Access Key for the user to use in the Django application.

Let's check our bucket in AWS S3 Console:

Now, we are ready to pass S3 credentials to the ECS services. Add variables in ecs.tf:

...
locals {
  container_vars = {
    ...

    s3_media_bucket         = var.prod_media_bucket
    s3_access_key           = aws_iam_access_key.prod_media_bucket.id
    s3_secret_key           = aws_iam_access_key.prod_media_bucket.secret
  }
}
...

Use these variables for defining environment variables in templates/backend_container.json.tpl and apply changes with terraform apply:

[
  {
    ...
    "environment": [
      ...
      {
        "name": "DEFAULT_FILE_STORAGE",
        "value": "storages.backends.s3boto3.S3Boto3Storage"
      },
      {
        "name": "AWS_ACCESS_KEY_ID",
        "value": "${s3_access_key}"
      },
      {
        "name": "AWS_SECRET_ACCESS_KEY",
        "value": "${s3_secret_key}"
      },
      {
        "name": "AWS_STORAGE_BUCKET_NAME",
        "value": "${s3_media_bucket}"
      },
      {
        "name": "AWS_S3_REGION_NAME",
        "value": "${region}"
      },
      {
       "name": "AWS_S3_ENDPOINT_URL",
       "value": "https://${s3_media_bucket}.s3.${region}.amazonaws.com/"
      }
    ],
    ...
  }
]

Wait for the deployment new version of ECS services. Then, go to the admin panel of the deployed application and try to add a new photo. Fill the fields and click "Save and continue editing." Then click the current image URL to ensure your photo is uploaded to S3 correctly.

Also, check your file in the S3 console:

So, we successfully added file uploading to S3 for our Django application. Now the application is stateless so that we can scale without problems.

The last feature I want to add to our Terraform project is an S3 backend.

Terraform S3 backend

Why do we actually need an S3 backend for Terraform?

Now Terraform stores the state of our infrastructure locally in the terraform.tfstate file. So, you can apply changes from your local machine only and cannot collaborate with other developers. Also, if you decide to add CI/CD to the Terraform project, your CD runner will have no information about the current infrastructure state.

The solution to these problems is a remote backend. Terraform will store terraform.tfstate file in S3 bucket. So, you can apply changes from different environments, including CD runners.

So, let's go to the AWS S3 Console and create an S3 bucket in your region. You can choose any name for your bucket, and I'll go with terraform-427861343. Be sure to disable public access so that nobody can know your infrastructure state. Also, it's better to enable versioning on this bucket. It allows you to roll back to the previous version in case of some problems.

Now, we can specify the S3 backend in providers.tf:

terraform {
  ...

  backend "s3" {
    bucket = "terraform-427861343"
    key    = "terraform"
    region = "us-east-2"
  }
}

and initialize the provider with terraform init.

Terraform will ask if you want to copy the existing state to the new backend. If you say no, Terraform will create an empty state in the S3 backend. We want to preserve the state of the infrastructure, so you need to respond yes.

$ terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes


Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
...

So, let's check that we've migrated successfully. Run terraform apply to verify that our infrastructure matches the configuration. You should see the message

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Also, check your S3 bucket. You should see a new terraform file here.

So, we successfully move our Terraform state to S3. Commit and push the changes in the Terraform project.

The end

Congratulations! In this part, we added image uploading to S3 for our Django application. Also, we moved Terraform backend to S3. If you want more insight into application state, resource usage, and changes, you can use Spacelift Terraform provider. Check how Spacelift can improve your workflow here.

You can find the source code of backend and infrastructure projects here and here.

If you need technical consulting on your project, check out our website or connect with me directly on LinkedIn.

How to become a better Django Developer by doing a pet project?

Yevhenii Kosmak — Tue, 16 Aug 2022 16:23:44 +0000

My name is Zhenya Kosmak. Now I work as a product manager, but I have 3+ years of quite serious commercial experience using Django. And even now, I strive to code for at least 10 hours weekly. I deeply believe the best way to master any framework, Django as well, is to build a truly big pet project. So a few bits of my recent pet project experience.

The pet project's name is Palianytsia. It's a tool for enriching Ukrainian vocabulary and, generally, studying Ukrainian. Now, 2 months after project public launch, we have 200+ DAU (daily active users) across all clients (web app, iOS, and Android apps) and a sufficient audience (10K+ followers on Instagram). The back-end of this product is entirely on Django.

It must be noted that a great developer in the modern world couldn’t be imagined as a single player. Many tough challenges show up, particularly in teamwork. So it would be a rather more quality experience if you start your pet project with a team containing competent people for such roles as product owner, designer, and project manager.

Your pet project should solve a sufficient number of most standard back-end tasks. The tasks have to be real, not synthetical. Here are examples of how you can set them up.

Main feature

In our case, it was a search mechanism on top of ElasticSearch. We have a database of sentences from different sources, and the task of this API endpoint is to return the most relevant sentences for each query. We have additional aspects on this. The sentences must show different contexts in which the user's input was used. Also, the search results should come from various sources in order to include principally different ways to use the search phrase.

This task teaches you how to work with ElasticSearch, and customize its ordering. Also, you practice using DRF to build quality API endpoints here. If you want to obtain excellent documentation of such endpoints, you might use drf-spectacular or else as well.

Data collection

Django developer must be a Python professional as well. Not all the everyday tasks are covered with some Django libraries, but they still have to be solved. HTTP scraping is one of the most standard ones.

We needed to collect articles from 40+ websites to provide users with comprehensive examples of each word or phrase usage. So we've built some abstract parser classes for most routine needs:

BaseParser. We define general parsing logic here.
BaseFeedPaginationParser. We use it to collect article URLs when the website's feeds are paginated, i.e., the URLs of the feed look like https://example.com/feed/<page>.
BaseFeedCrawlerParser. On some sites, there were no paginated URLs; they had only infinite scroll or "Next page" URLs on each page. This parser works with such websites. Also, we use it for Wikipedia.
BaseArticleParser. This guy is used for collecting the content of each article.

Such class hierarchy was a definitely productive solution. After finishing work on these classes, developing a parser was mainly like writing a config file. For example:

class DetectorMediaBaseFeedParser(BaseFeedParser):
    start_date: datetime.date

    def feed_urls(self) -> List[str]:
        return [
            f"{self.base_url}archive/{d.strftime('%Y-%m-%d')}/"
            for d in date_range(self.start_date, today())
        ]

class DetectorMediaBaseArticleParser(BaseArticleParser):
    paragraphs_selector = "#artelem > p"

    def extract_paragraphs(self, dom: HtmlElement) -> List[str]:
        # Current markup
        paragraphs = super().extract_paragraphs(dom)
        if paragraphs:
            return paragraphs

        # Old br-only markup
        content = cssselect_one(dom, "#artelem")
        paragraphs = split_br_into_paragraphs_lxml(content) if content else []
        return paragraphs

You can only guess what a mess we could get if we developed each parser separately. It's also an excellent practice for requests, asyncio, or anything else you prefer to get the data from websites.

When you’ve made 40 similar scrapers from scratch…

This task also teaches you how to orchestrate those parsers. We had to gather millions of articles based on our estimates, and we wanted to be able to collect all data from all sites in 24 hours. So we have to use a tasks queue to ensure the solution is efficient and scalable. The most common approach on the Django stack is to use celery, but we prefer dramatiq as the most lightweight and still productive one. Bringing such a solution to a stable and scalable level is also a very effective way to improve your coding skills.

Last but not least, we need a state here. We have to save each gathered URL and article content after scraping, so we wouldn't lose the work results if our server crashes. And the content is stored in S3 Wasabi cloud storage. Here is your practice with PostgreSQL or any other RDBMS you'd like, and S3 as well.

Data processing

Okay, we have articles; what's next? We need a database of Ukrainian sentences split properly, with filtered artifacts (such as BB-code parts or broken HTML, which occurs on most websites), normalized (we don't need repeated spaces or else). As input, we have a PostgreSQL table with articles; each article is stored on S3 storage and split into paragraphs. So we've got such tasks:

Split a paragraph into a list of sentences. The job looks easier than it is. No way it's something like paragraph.split('.'), it's way harder. For some direction, you can look at this StackOverflow question.
Skip all paragraphs with mostly non-Ukrainian content.
Filter out trash content.

Let's add some 3rd-party tools

Sometimes when you want to say something in one language, you can only recall the word or phrase in another. So let's add Google Translate so that our users would be able to enter queries using any language, and we will search for translation in our database.

It's a nice opportunity to practice using dependencies (google-cloud-translate), storing credentials, and even developing a cache mechanic. The last one would be helpful to decrease the number of requests sent to the API so that we will spend less. By the way, if you send less than 500K characters monthly, you wouldn't even be charged with Google Cloud cause it's free under that limit.

And one more tricky task. The user searches a query. Should you or shouldn't you translate it? There are several ways to solve this issue. We decided to go this way: we stem and normalize all the words from the query, then we find all of them in our dictionary of Ukrainian words. We definitely shouldn't go for a translation if we found all of them. Else way, we better get it.

User-related mechanics

We need authentication, authorization, registration, password recovery — all that boring stuff.

It's a good chance to practice django-allauth, dj-rest-auth, djangorestframework-simplejwt. Remember such things:

Users shouldn't be logged out for no reason, so cookies shouldn't expire in one hour,
Their credentials must be safe, so that should be HTTP-only cookies and data kept secure on the server side,
All errors must be processed intuitively and logically, so the front-end part will be easy to develop.

Developing 3rd-party authentications is also a good idea for new technical experience. Adding Google or Facebook as sign-up methods might be a more intriguing task than you'd imagine.

Should we do some automated tests?

This many tests we’ve made!

Of course, we should! But please don't do it like your home assignment at university. Choose the most vulnerable parts of projects for future developer mistakes and cover them up. In our case, we made such:

ElasticSearch test for disk usage, RAM, and response speed. Actually, we did this before all else in this project as a part of the proof-of-concept stage. We filled the ElasticSearch DB with dummy data and used ApacheBench, and some write-only Python scripts to test the performance on the production-level server.
Performance tests on the primary endpoint to ensure we'll endure the load. These tests were added to the stable code base so that we could rerun the test when needed.
Unit tests for parsers, text processing mechanics, and string utilities. So that we can be sure that our technical solutions would be changed only in a conscious way.

Bottom line

Let's summarize. On such a pet project, you can effectively learn:

General business logic development. When you have a real-world task, the coding experience becomes more fluent.
Work with mainly used databases. Any project relies on them, so if your task is challenging enough, you may learn those well.
Task queues. They become vital just as you encounter any IO-bound, CPU-bound or high-loaded objective. Choose high restrictions, and you learn it well for sure.
3rd-parties. They are an integral part of any product. The more you learn, the better you're ready for the next ones.
Network libraries. Not to say vital, but an omnipresent piece of knowledge you'd definitely need. Any pet project will acquire your attention on this point.
Tests. Not homework, but real salvation of doing error twice.

It's almost the same as spending a year on a commercial project. But you can do it yourself, and if you are patient and stubborn enough, you can master it. So think about it :)

And the last thing. If you are thinking about something bigger than a pet project, you can talk with me directly or check out our cases on Daiquiri Team.

Deploying Django Application on AWS with Terraform. Setting up Celery and SQS

Yevhen Bondar — Fri, 12 Aug 2022 12:31:00 +0000

This is the fifth part of the "Deploying Django Application on AWS with Terraform" guide. You can check out the previous steps here:

In this step, we are going to:

Add a Celery + SQS setup for local development.
Create a periodic task using Celery Beat.
Add create an SQS instance on AWS.
Deploy worker and beat ECS services on AWS.

About Celery and SQS

As docs says:

Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling.

So, you can run your long-running, CPU-bound, IO-bound tasks in separate docker containers. Your web server can schedule some tasks, and Celery will pick and execute them.

Web server and Celery communicate via some backend. It could be Redis or RabbitMQ. But we use AWS as a cloud provider. So, we can use the SQS backend. Celery docs says:

If you already integrate tightly with AWS, and are familiar with SQS, it presents a great option as a broker. It is extremely scalable and completely managed, and manages task delegation similarly to RabbitMQ.

You can check more info about SQS here.

Local development setup

Running SQS locally

SQS is a managed solution by AWS. And we could create a separate SQS instance for development purposes. But I want to run all things locally to be able to work without the Internet connection. Also, it's a good practice to run unit and integration tests without Internet access.

To run SQS locally, we will use softwaremill/elasticmq-native Docker image.

Go to the django-aws-backend folder and add a new service to docker-compose.yml in your Django project:



...

services:
  ...

  sqs:
    image: "softwaremill/elasticmq-native:latest"
    ports:
      - "9324:9324"
      - "9325:9325"

Run docker-compose up -d to run the SQS container. Then check http://127.0.0.1:9325/ in your browser to see the SQS management panel.

Also, check http://127.0.0.1:9324/ URL to ensure that SQS API is working. You will see an error XML output like this:



<ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/">
      <Error>
        <Type>Sender</Type>
        <Code>MissingAction</Code>
        <Message>MissingAction; see the SQS docs.</Message>
        <Detail/>
      </Error>
      <RequestId>00000000-0000-0000-0000-000000000000</RequestId>
    </ErrorResponse>

Now, we are ready to add Celery to our Django project.

Adding Celery to Django project

Let's add the Celery package to requirements.txt and run pip install -r requirements.txt. Be sure you activated venv before.



celery[sqs]==5.2.6

Then, create a new file django_aws/celery.py with the following content:



import os

from celery import Celery

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_aws.settings")

app = Celery("django_aws")

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object("django.conf:settings", namespace="CELERY")

# Load task modules from all registered Django apps.
app.autodiscover_tasks()

and add to the django_aws/settings.py these lines:



CELERY_BROKER_URL = env("CELERY_BROKER_URL", default="sqs://localhost:9324")
CELERY_TASK_DEFAULT_QUEUE = env("CELERY_TASK_DEFAULT_QUEUE", default="default")
CELERY_BROKER_TRANSPORT_OPTIONS = {
    "region": env("AWS_REGION", default="us-east-1")
}

Here we initialized the Celery app in django_aws/celery.py. We will use this app to specify and schedule tasks for Celery. Also, we provided connection parameters in django_aws/settings.py. As default values, we set our local setup. For production, we can pass parameters via environment variables.

Now, we are ready to create and run our first task. Let's create django_aws/tasks.py with the following code:



import logging
import time

from django_aws import celery


@celery.app.task()
def web_task() -> None:
    logging.info("Starting web task...")
    time.sleep(10)
    logging.info("Done web task.")

The web_task will run for 10 seconds and put messages in the log stream at the start and the end of execution.

Now, we need to add a way to add this task to the queue. Let's create a django_aws/views.py with the following view:



from django.http import HttpResponse
from django_aws.tasks import web_task


def create_web_task(request):
    web_task.delay()
    return HttpResponse("Task added")

Add this view to urls.py:



...
from django_aws import views

urlpatterns = [
    ...
    path('create-task', views.create_web_task),
]

...

Now, if we hit URL http://127.0.0.1:8000/create-task, the create_web_task view will add a new task to the local SQS. Start the local web server with python manage.py runserver, hit this URL several times, and look at the SQS admin page http://127.0.0.1:9325/.

So, we successfully add tasks to the queue. Now, let's execute them with celery. Run celery -A django_aws worker --loglevel info to start the worker process. The worker will immediately pick tasks from the queue and execute them:

Stop the celery process.

If you run into the problem ImportError: The curl client requires the pycurl library, check out my post on StackOverflow

Also, we need to add some libraries to Dockerfile for compiling the pycurl in the docker image. Replace Dockerfile with the next one:



FROM python:3.10-slim-buster

EXPOSE 8000

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update  \
    && apt-get --no-install-recommends install -y \
        build-essential \
        libssl-dev \
        libcurl4-openssl-dev \
    && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir --upgrade pip
RUN pip install gunicorn==20.1.0

COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt

WORKDIR /app
COPY . /app

RUN ./manage.py collectstatic --noinput

Adding Celery beat

Now, let's create a periodic task using Celery Beat. We will add a simple task like create_web_task and schedule it for execution once a minute. For this, let's add beat_task to tasks.py:



@celery.app.task()
def beat_task() -> None:
    logging.info("Starting beat task...")
    time.sleep(10)
    logging.info("Done beat task.")

Then, add the CELERY_BEAT_SCHEDULE setting in settings.py:



from datetime import timedelta
...
CELERY_BEAT_SCHEDULE = {
    "beat_task": {
        "task": "django_aws.tasks.beat_task",
        "schedule": timedelta(minutes=1),
    },
}

and run the beat process celery -A django_aws beat --loglevel info. Every minute beat process adds a new task to SQS. Check http://127.0.0.1:9325/ to see them.

Wait for several tasks in queue, stop the beat process and run the worker again celery -A django_aws worker --loglevel info. The worker will process beat_task tasks, and you will see the logs:



[2022-08-04 11:13:59,088: INFO/MainProcess] Task django_aws.tasks.beat_task[4189aa07-b75e-4743-94e0-2a0c3b84443a] received
[2022-08-04 11:13:59,089: INFO/MainProcess] Task django_aws.tasks.beat_task[0de67363-2e2a-421c-9630-1c6c7c685382] received
[2022-08-04 11:13:59,095: INFO/ForkPoolWorker-1] Starting beat task...
[2022-08-04 11:13:59,095: INFO/ForkPoolWorker-8] Starting beat task...
[2022-08-04 11:14:09,096: INFO/ForkPoolWorker-1] Done beat task.
[2022-08-04 11:14:09,096: INFO/ForkPoolWorker-8] Done beat task.
[2022-08-04 11:14:09,097: INFO/ForkPoolWorker-1] Task django_aws.tasks.beat_task[0de67363-2e2a-421c-9630-1c6c7c685382] succeeded in 10.002475121000316s: None
[2022-08-04 11:14:09,097: INFO/ForkPoolWorker-8] Task django_aws.tasks.beat_task[4189aa07-b75e-4743-94e0-2a0c3b84443a] succeeded in 10.002584206999018s: None

So, we successfully run Celery worker and beat processes using local SQS. Let's add the celerybeat-schedule file to .gitignore, commit and push our changes. Ensure that CI/CD passed successfully.

We are done with Django part, let's move to the AWS.

Deploying to AWS

Creating AWS SQS instance and user

Move to the django-aws-infrastructure folder, create a sqs.tf file with the following content, and run terraform apply.



resource "aws_sqs_queue" "prod" {
  name                      = "prod-queue"
  receive_wait_time_seconds = 10
  tags = {
    Environment = "production"
  }
}

resource "aws_iam_user" "prod_sqs" {
  name = "prod-sqs-user"
}

resource "aws_iam_user_policy" "prod_sqs" {
  user = aws_iam_user.prod_sqs.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "sqs:*",
        ]
        Effect   = "Allow"
        Resource = "arn:aws:sqs:*:*:*"
      },
    ]
  })
}

resource "aws_iam_access_key" "prod_sqs" {
  user = aws_iam_user.prod_sqs.name
}

Here we've created a new SQS instance, a new IAM user, granted access to this SQS instance to this user, and created an IAM Access Key to give access to SQS from Django application. Let's look at the new instance in AWS console:

Activating a region

Now, let's create an ECS for the Celery worker and beat.

First, let's "activate" our region on AWS. For some reason, AWS doesn't allow you to create more than 2 ECS containers in the region. You need to create an EC2 instance in this region to remove this limit. Let's do it manually in EC2 Console. Be sure that you use your AWS region.

Click "Launch Instance"

Pick any name for your instance. I'll go with Test Server

Scroll down to the "Key pair" card and pick "Proceed without a key pair". We won't connect to this server, so we don't need one. Then click "Launch Instance" to create a new instance in your region.

AWS will soon create an instance. Go to the Instances tab in EC2 Console and verify that you have 1 "Running" instance.

After that, you can terminate the instance because we don't need it. Pick the instance, click the "Instance state", then "Terminate instance", and confirm termination. AWS will permanently remove your EC2 instance.

So, now we can create more than two ECS containers. Let's continue creating Celery ECS.

Running a Celery via ECS

Now, let's define our Celery ECS service. First, add new variables in ecs.tf:



locals {
  container_vars = {
    ...

    sqs_access_key = aws_iam_access_key.prod_sqs.id
    sqs_secret_key = aws_iam_access_key.prod_sqs.secret
    sqs_name = aws_sqs_queue.prod.name
  }
}

and pass it to containers in backend_container.json.tpl:



[
  {
    ...
    "environment": [
      ...
      {
        "name": "AWS_REGION",
        "value": "${region}"
      },
      {
        "name": "CELERY_BROKER_URL",
        "value": "sqs://${urlencode(sqs_access_key)}:${urlencode(sqs_secret_key)}@"
      },
      {
        "name": "CELERY_TASK_DEFAULT_QUEUE",
        "value": "${sqs_name}"
      }
    ],
    ...
  }
]

So, we passed SQS credentials to ECS services. Then, add the following content in ecs.tf and run terraform apply:



...

# Cloudwatch Logs
...

resource "aws_cloudwatch_log_stream" "prod_backend_worker" {
  name           = "prod-backend-worker"
  log_group_name = aws_cloudwatch_log_group.prod_backend.name
}

resource "aws_cloudwatch_log_stream" "prod_backend_beat" {
  name           = "prod-backend-worker"
  log_group_name = aws_cloudwatch_log_group.prod_backend.name
}

...

# Worker

resource "aws_ecs_task_definition" "prod_backend_worker" {
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 256
  memory                   = 512

  family = "backend-worker"
  container_definitions = templatefile(
    "templates/backend_container.json.tpl",
    merge(
      local.container_vars,
      {
        name       = "prod-backend-worker"
        command    = ["celery", "-A", "django_aws", "worker", "--loglevel", "info"]
        log_stream = aws_cloudwatch_log_stream.prod_backend_worker.name
      },
    )
  )
  depends_on = [aws_sqs_queue.prod, aws_db_instance.prod]
  execution_role_arn = aws_iam_role.ecs_task_execution.arn
  task_role_arn      = aws_iam_role.prod_backend_task.arn
}

resource "aws_ecs_service" "prod_backend_worker" {
  name                               = "prod-backend-worker"
  cluster                            = aws_ecs_cluster.prod.id
  task_definition                    = aws_ecs_task_definition.prod_backend_worker.arn
  desired_count                      = 2
  deployment_minimum_healthy_percent = 50
  deployment_maximum_percent         = 200
  launch_type                        = "FARGATE"
  scheduling_strategy                = "REPLICA"
  enable_execute_command             = true

  network_configuration {
    security_groups  = [aws_security_group.prod_ecs_backend.id]
    subnets          = [aws_subnet.prod_private_1.id, aws_subnet.prod_private_2.id]
    assign_public_ip = false
  }
}

# Beat

resource "aws_ecs_task_definition" "prod_backend_beat" {
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 256
  memory                   = 512

  family = "backend-beat"
  container_definitions = templatefile(
    "templates/backend_container.json.tpl",
    merge(
      local.container_vars,
      {
        name       = "prod-backend-beat"
        command    = ["celery", "-A", "django_aws", "beat", "--loglevel", "info"]
        log_stream = aws_cloudwatch_log_stream.prod_backend_beat.name
      },
    )
  )
  depends_on = [aws_sqs_queue.prod, aws_db_instance.prod]
  execution_role_arn = aws_iam_role.ecs_task_execution.arn
  task_role_arn      = aws_iam_role.prod_backend_task.arn
}

resource "aws_ecs_service" "prod_backend_beat" {
  name                               = "prod-backend-beat"
  cluster                            = aws_ecs_cluster.prod.id
  task_definition                    = aws_ecs_task_definition.prod_backend_beat.arn
  desired_count                      = 1
  deployment_minimum_healthy_percent = 50
  deployment_maximum_percent         = 200
  launch_type                        = "FARGATE"
  scheduling_strategy                = "REPLICA"
  enable_execute_command             = true

  network_configuration {
    security_groups  = [aws_security_group.prod_ecs_backend.id]
    subnets          = [aws_subnet.prod_private_1.id, aws_subnet.prod_private_2.id]
    assign_public_ip = false
  }
}

Here we created:

Cloudwatch Logs streams for worker and beat.
Worker ECS task definition and ECS service. We specified desired_count=2 to show how multiple workers can run for the same queue. In the future we will scale worker ECS depending on CPU load.
Beat ECS task definition and ECS service. Here we specified desired_count=1 because we don't want to schedule duplicates for periodic tasks.

Let's check our services in the ECS console.

Here are our worker and beat service:

Here are worker and beat tasks. You can see that ECS creates two tasks for the worker service and only one task for the beat service:

Here are worker logs. For now, we see only beat tasks in logs:

Let's add a new task from the web. Hit https://api.example53.xyz/create-task URL (replace a domain with your one). You should see a 'Task added' message in response. Then, return to ECS worker logs, and pick the '30s' interval to see the most recent log events. You should see 'Starting web task' and 'Done web task' messages in the logs.

So, we successfully run ECS for worker and beat processes and ensure that both web and beat Celery tasks are executed successfully.

We are done with the infrastructure repo so that you can commit and push the changes.

Updating deploy

There is still one more task. To ensure that we will update our ECS services with every deployment, we need to modify our ./scripts/deploy.sh. Let's add the same instruction as for the web service:



...

echo "Updating web..."
aws ecs update-service --cluster prod --service prod-backend-web --force-new-deployment  --query "service.serviceName"  --output json
echo "Updating worker..."
aws ecs update-service --cluster prod --service prod-backend-worker --force-new-deployment  --query "service.serviceName"  --output json
echo "Updating beat..."
aws ecs update-service --cluster prod --service prod-backend-beat --force-new-deployment  --query "service.serviceName"  --output json

echo "Done!"

So, we will force a new deployment for the worker and beat services on ECS with every push.

Commit and push changes. Wait for CI/CD and check your services in ECS Console. After some time, new tasks will arise:

You can schedule more web tasks and see them in logs to ensure that things work as expected.

The end

Congratulations! We've successfully created an AWS SQS instance and added Celery worker + beat services to ECS. Our Django application can run long-living tasks in the background worker process.

You can find the source code of backend and infrastructure projects here and here.

If you need technical consulting on your project, check out our website or connect with me directly on LinkedIn.

Deploying Django Application on AWS with Terraform. Namecheap Domain + SSL

Yevhen Bondar — Tue, 09 Aug 2022 11:01:00 +0000

In previous steps, we've deployed Django with AWS ECS, connected it to the PostgreSQL RDS, and set up GitLab CI/CD

In this step, we are going to:

Connect Namecheap domain to Route53 DNS zone.
Create an SSL certificate with Certificate Manager.
Reroute HTTP traffic to HTTPS and disable ALB host for Django application.
Add /health/ route for Health Checks.

Setting up Namecheap API

I already have a domain name on Namecheap. So I choose to connect the Namecheap domain to an AWS Route53 zone. But you can register domain with AWS Route53.

First, let's enable API access for Namecheap. Look through this guide to receive APIKey and add your IP to the whitelist.

Second, add the Namecheap provider to Terraform project. Add to the provider.tf file following code:

terraform {
  required_providers {
    namecheap = {
      source  = "namecheap/namecheap"
      version = ">= 2.0.0"
    }
  }
}

provider "namecheap" {
  user_name   = var.namecheap_api_username
  api_user    = var.namecheap_api_username
  api_key     = var.namecheap_api_key
  use_sandbox = false
}

In variables.tf add:

# Namecheap
variable "namecheap_api_username" {
  description = "Namecheap APIUsername"
}
variable "namecheap_api_key" {
  description = "Namecheap APIKey"
}

Also add TF_VAR_namecheap_api_username and TF_VAR_namecheap_api_key variables to .env to provide values to the corresponding Terraform variables.

TF_VAR_namecheap_api_username=YOUR_API_USERNAME
TF_VAR_namecheap_api_key=YOUR_API_KEY

Import .env variables with export $(cat .env | xargs) and run terraform init to add a Namecheap provider to the project.

Connecting Domain to AWS

Now, let's create a Route53 zone for the Namecheap domain and set up AWS nameservers. Thus, all DNS queries will be routed to the AWS Route53 nameservers, and we can manage DNS records from the AWS Route53 zone.

Add to the variables.tf following code:

# Domains
variable "prod_base_domain" {
  description = "Base domain for production"
  default = "example53.xyz"
}
variable "prod_backend_domain" {
  description = "Backend web domain for production"
  default = "api.example53.xyz"
}

Add a route53.tf file:

resource "aws_route53_zone" "prod" {
  name = var.prod_base_domain
}

resource "namecheap_domain_records" "prod" {
  domain = var.prod_base_domain
  mode   = "OVERWRITE"

  nameservers = [
    aws_route53_zone.prod.name_servers[0],
    aws_route53_zone.prod.name_servers[1],
    aws_route53_zone.prod.name_servers[2],
    aws_route53_zone.prod.name_servers[3],
  ]
}

Run terraform apply. Check nameservers on Namecheap:

Creating SSL Certificate

Now, let's create an SSL certificate and set up DNS A record for api.example53.xyz domain.

Add to the route53.tf following code:

...

resource "aws_acm_certificate" "prod_backend" {
  domain_name       = var.prod_backend_domain
  validation_method = "DNS"
}

resource "aws_route53_record" "prod_backend_certificate_validation" {
  for_each = {
    for dvo in aws_acm_certificate.prod_backend.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = aws_route53_zone.prod.zone_id
}

resource "aws_acm_certificate_validation" "prod_backend" {
  certificate_arn         = aws_acm_certificate.prod_backend.arn
  validation_record_fqdns = [for record in aws_route53_record.prod_backend_certificate_validation : record.fqdn]
}

resource "aws_route53_record" "prod_backend_a" {
  zone_id = aws_route53_zone.prod.zone_id
  name    = var.prod_backend_domain
  type    = "A"

  alias {
    name                   = aws_lb.prod.dns_name
    zone_id                = aws_lb.prod.zone_id
    evaluate_target_health = true
  }
}

Here we are going to create a new SSL certificate for api.example53.xyz, validate the SSL certificate via DNS CNAME record, and add DNS A record to Load Balancer.

Apply changes with terraform apply and wait for certificate validation. Usually, it takes up to several minutes. But in some cases, it can take several hours. You can check more info here.

Redirecting HTTP to HTTPS

Now let's use the issued SSL certificate to enable HTTPS. Replace the resource "aws_lb_listener" "prod_http" block in the load_balancer.tf with the following code:

# Target listener for http:80
resource "aws_lb_listener" "prod_http" {
  load_balancer_arn = aws_lb.prod.id
  port              = "80"
  protocol          = "HTTP"
  depends_on        = [aws_lb_target_group.prod_backend]

  default_action {
    type = "redirect"
    redirect {
      port        = "443"
      protocol    = "HTTPS"
      status_code = "HTTP_301"
    }
  }
}

# Target listener for https:443
resource "aws_alb_listener" "prod_https" {
  load_balancer_arn = aws_lb.prod.id
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-2016-08"
  depends_on        = [aws_lb_target_group.prod_backend]

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.prod_backend.arn
  }

  certificate_arn = aws_acm_certificate_validation.prod_backend.certificate_arn
}

Here we redirect unsecured HTTP traffic to HTTPS and add a listener for the HTTPS port. Apply changes and check https://api.example53.xyz URL. You should see Django starting page.

Setting up the ALLOWED_HOSTS variable

Now, let's provide the ALLOWED_HOSTS setting to the Django app. It's important to prevent HTTP Host header attacks. So, Django Application should only accept our domain api.example53.xyz in the host header.

Now Django accepts any domain, for example, Load Balancer's domain. Visit https://prod-1222631842.us-east-2.elb.amazonaws.com to check this fact. You can ignore the warning about an invalid SSL Certificate and see that Django responds to this host.

Also, let's disable a Debug mode and remove the SECRET_KEY value from the code to improve security. Add the TF_VAR_prod_backend_secret_key variable with a random generated value to the .env, run export $(cat .env | xargs), and specify this var in variables.tf:

variable "prod_backend_secret_key" {
  description = "production Django's SECRET_KEY"
}

Next, pass the domain name and SECRET_KEY in ecs.tf, set up SECRET_KEY, DEBUG, and ALLOWED_HOSTS variables in backend_container.json.tpl and apply changes:

locals {
  container_vars = {
    ...

    domain = var.prod_backend_domain
    secret_key = var.prod_backend_secret_key
  }
}

"environment": [
  ...
  {
    "name": "SECRET_KEY",
    "value": "${secret_key}"
  },
  {
    "name": "DEBUG",
    "value": "off"
  },
  {
    "name": "ALLOWED_HOSTS",
    "value": "${domain}"
  }
],

Now we have all necessary environment variables on ECS. Move to the Django app and change settings.py:

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = env("SECRET_KEY", default="ewfi83f2ofee3398fh2ofno24f")

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = env("DEBUG", cast=bool, default=True)

ALLOWED_HOSTS = env("ALLOWED_HOSTS", cast=list, default=["*"])

Here we receive SECRET_KEY, DEBUG, and ALLOWED_HOSTS variables from env variables. We provide default SECRET_KEY to allow running the application locally without specifying SECRET_KEY in the .env file.

Health Check

All user's requests would have Host header api.example53.xyz. But, we also have health check requests from a load balancer.

AWS load balancers can automatically check our container's health. If the container responds correctly, the load balancer considers that target is healthy. Otherwise, the target will be marked as unhealthy. Load balancer routes traffic to healthy targets only. Thus, user requests wouldn't hit unhealthy containers.

For HTTP or HTTPS health check requests, the host header contains the IP address of the load balancer node and the listener port, not the IP address of the target and the health check port.

We don't know the Load Balancer IP address. Also, this IP can be changed after some time. Therefore, we cannot add the Load Balancer host to the ALLOWED_HOSTS.

The solution is to write a custom middleware that returns a successful response before the host checking in the SecurityMiddleware.

First, go to the infrastructure, change the health check URL in load_balancer.tf to /health/, and apply changes:

resource "aws_lb_target_group" "prod_backend" {
  ...
  health_check {
    path = "/health/"
    ...
  }
}

Return to the Django project and create django_aws/middleware.py:

from django.http import HttpResponse
from django.db import connection


def health_check_middleware(get_response):
    def middleware(request):
        # Health-check request
        if request.path == "/health/":
            # Check DB connection is healthy
            with connection.cursor() as cursor:
                cursor.execute("SELECT 1")

            return HttpResponse("Healthy!")

        # Regular requests
        return get_response(request)

    return middleware

Add this middleware to the settings.py before the SecurityMiddleware:

MIDDLEWARE = [
    'django_aws.middleware.health_check_middleware',
    'django.middleware.security.SecurityMiddleware',
    ...
]

Run python manage.py runserver and check 127.0.0.1:8000/health/ URL in your browser. You should see the text response Healthy!.

Commit and push changes, wait for the pipeline and check the Load Balancer domain again https://prod-57218461274.us-east-2.elb.amazonaws.com/. Now, we get a Bad Request error. Also, we didn't see a traceback or other debug information, so we can be sure that the debug mode is disabled.

Also, navigate to https://prod-57218461274.us-east-2.elb.amazonaws.com/health/ in your browser to check health_check_middleware. We get the Healthy! response. So, the Load Balancer will be able to check containers' health without providing the correct Host header.

Congratulations! We've successfully set up a domain name, created health checks, disabled the debug mode, and removed SECRET_KEY value from the source code. Do not forget to push infrastructure code to GitLab.

You can find the source code of backend and infrastructure projects here and here.

If you need technical consulting on your project, check out our website or connect with me directly on LinkedIn.

Deploying Django Application on AWS with Terraform. GitLab CI/CD

Yevhen Bondar — Wed, 03 Aug 2022 07:07:00 +0000

In previous parts, we've deployed the Django web application to ECS and connected PostgreSQL to it. But now, we have to deploy application changes manually. In this part, we are going to automate this process with the following steps:

Create a GitLab group and projects for the backend and the infrastructure repositories.
Add the test CI/CD stage to run tests.
Add the build CI/CD stage to build a docker image and push it to the ECR.
Add the deploy CI/CD stage to update the application on AWS.

Creating Projects

Let's start with creating a GitLab group and projects. Go to gitlab.com and register an account.

Then create a GitLab group. A group is like a folder for projects. You can configure shared settings like access policy and CI variables for them.

Create projects for Django and Terraform in this group. Be sure to remove the "Initialize repository with a README" option to create a clean repository.

Also, add your ssh key to SSH Keys section. It allows you to access your projects via Git.

Now, let's push both of our repositories to GitLab.

# Push backend
cd ../django-aws-backend
git remote add origin git@gitlab.com:django-aws/django-aws-backend 
git push --set-upstream origin main

# Push infrastructure
cd ../django-aws-infrastructure
git remote add origin git@gitlab.com:django-aws/django-aws-infrastructure 
git push --set-upstream origin main

Check your GitLab projects in a browser and verify that the push is successful.

Stage: Test

Now let's add unit tests check to the Django project. Django's unit tests use a Python standard library module unittest. See more info about testing a Django application here.

Go to the Django project, activate venv and start a PostgreSQL Docker container:

$ cd ../django-aws-backend
$ . ./venv/bin/activate
$ docker-compose up -d

Let's create django_aws/tests.py and add a simple test to check DB connection:

from django.test import TestCase
from django.db import connection


class DbConnectionTestCase(TestCase):
    def test_db_connection(self):
        self.assertTrue(connection.is_usable())

Run python manage.py test locally:

$ python manage.py test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.010s

OK
Destroying test database for alias 'default'...

Now, let's add this check on GitLab's side. Take a look here if you have no idea what GitLab CI is. Generally, GitLab will run code specified in .gitlab-ci.yml on every push in the repository.

Create .gitlab-ci.yml file with following content:

image: python:3.10

stages:
  - test

variables:
  POSTGRES_PASSWORD: password
  POSTGRES_DB: django_aws
  DATABASE_URL: postgres://postgres:password@postgres:5432/django_aws

test:
  services:
    - postgres:14.2
  cache:
    key:
      files:
        - requirements.txt
      prefix: ${CI_JOB_NAME}
    paths:
      - venv
      - .cache/pip
  stage: test
  script:
    - python -m venv venv
    - . venv/bin/activate
    - pip install --upgrade pip
    - pip install -r requirements.txt
    - python manage.py test

Explanation:

image: python:3.10. By default, GitLab runs CI/CD pipeline on shared runners hosted by GitLab using the docker executor. Here we specify docker image for the executor.
stages:. A pipeline can have several stages. Now we have only the test stage. In the future, we will have three stages: test, build, and deploy. GitLab will execute stages in the specified order.
services: postgres:14.2. GitLab will run PostgreSQL in a separate container during the test stage. So, our Django application will be able to run DB-related tests.
variables: POSTGRES_PASSWORD, POSTGRES_DB DATABASE_URL - environment variables for both Django and PostgreSQL docker containers.
cache: key:. Here we cache the result of the pip install command to speed up pipeline executions. If the key files: requirements.txt hasn't changed since the last run GitLab will download the cache for venv and .cache/pip directories.
script: ... is commands to execute. GitLab executes commands one by one. If some command returns a non-zero exit code, GitLab will interrupt pipeline execution and mark it Failed. If all commands have been executed successfully, GitLab marks the current stage as Successful and goes to the next stage.

Now, push the changes and take a look at CI/CD tab of your GitLab Django project.

$ git add .
$ git commit -m "add gitlab-ci; add test"
$ git push

Stage: Build

Tests are passed, so we move on to the build stage. At this stage, we need to connect our GitLab account with our AWS account to grant GitLab access to the ECR repository. For this, we'll create a separate AWS user gitlab with limited permissions. Let's go to the AWS IAM console and create a new user.

You can find instructions on how to create an AWS user here

Add to this user AmazonEC2ContainerRegistryPowerUser policy to enable read and write permission to any ECR repository on this account. Proceed to the final step of user creation and save ACCESS_KEY_ID and SECRET_ACCESS_KEY.

Now go to the GitLab group settings and add AWS_ACCOUNT_ID, AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID, and AWS_DEFAULT_REGION variables. GitLab runner will use these credentials for AWS CLI calls.

Make sure to mask sensitive variables AWS_ACCOUNT_ID, AWS_SECRET_ACCESS_KEY, and AWS_ACCESS_KEY_ID to hide their values in GitLab logs. Take a look at the AWS deployment documentation page for more information.

Then add to the .gitlab-ci.yml build block:

stages:
  - test
  - build

variables:
  ...
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""
  AWS_REGISTRY_URL: "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${CI_PROJECT_NAME}:latest"

test:
  ...

build:
  stage: build
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  services:
    - docker:20.10-dind
  before_script:
    - aws ecr get-login-password | docker login --username AWS --password-stdin $AWS_REGISTRY_URL
  script:
    - docker pull $AWS_REGISTRY_URL || true
    - docker build --cache-from $AWS_REGISTRY_URL -t $AWS_REGISTRY_URL .
    - docker push $AWS_REGISTRY_URL
  only:
    - main

Explanation:

image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest. This image allows using AWS CLI commands during CI/CD.
services: docker:20.10-dind. We use docker-in-docker to build the docker image inside the docker executor. Docker starts as a separate service, so we can access the docker daemon from the GitLab job.
variables: DOCKER_HOST, DOCKER_TLS_CERTDIR. Specify the path to the docker-in-docker daemon and disable TLS connection.
AWS_REGISTRY_URL. Construct the ECR URL from the project name. Ensure that the names of the ECR repo and Gitlab project are the same.
before_script: aws ecr get-login-password. Log in to ECR.
docker pull $AWS_REGISTRY_URL || true. Pull actual image to use it as cache
docker push $AWS_REGISTRY_URL. Upload image to ECR.
only: main. We'll build a docker image for main branch only.

Commit your changes and check a CI/CD pipeline:

$ git add .
$ git commit -m "add build stage"
$ git push

Stage: Deploy

Build stage passed, now we'll deploy container from ECR to ECS.

But let's start with running migrations. We want to run migrations as separate ECS task to avoid any side effects on web application. Go to the infrastructure project and make following changes in ecs.tf and apply changes:

# Production cluster
...

locals {
  container_vars = {
    region = var.region

    image     = aws_ecr_repository.backend.repository_url
    log_group = aws_cloudwatch_log_group.prod_backend.name

    rds_db_name  = var.prod_rds_db_name
    rds_username = var.prod_rds_username
    rds_password = var.prod_rds_password
    rds_hostname = aws_db_instance.prod.address
  }
}

# Backend web task definition and service
resource "aws_ecs_task_definition" "prod_backend_web" {
  ...
  container_definitions = templatefile(
    "templates/backend_container.json.tpl",
    merge(
      local.container_vars,
      {
        name       = "prod-backend-web"
        command    = ["gunicorn", "-w", "3", "-b", ":8000", "django_aws.wsgi:application"]
        log_stream = aws_cloudwatch_log_stream.prod_backend_web.name
      },
    )
  )
  ...
}
...
# Cloudwatch Logs
...
resource "aws_cloudwatch_log_stream" "prod_backend_migrations" {
  name           = "prod-backend-migrations"
  log_group_name = aws_cloudwatch_log_group.prod_backend.name
}

# Migrations

resource "aws_ecs_task_definition" "prod_backend_migration" {
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 256
  memory                   = 512

  family = "backend-migration"
  container_definitions = templatefile(
    "templates/backend_container.json.tpl",
    merge(
      local.container_vars,
      {
        name       = "prod-backend-migration"
        command    = ["python", "manage.py", "migrate"]
        log_stream = aws_cloudwatch_log_stream.prod_backend_migrations.name
      },
    )
  )
  depends_on         = [aws_db_instance.prod]
  execution_role_arn = aws_iam_role.ecs_task_execution.arn
  task_role_arn      = aws_iam_role.prod_backend_task.arn
}

Here we've moved the same variables for migration and web containers in container_vars. Then we've created a separate task definition and log stream for the migration container. Now we can the start task with a specified task definition to apply migrations for every release.

Now let's create a deployment script. Return the Django project and create scripts/deploy.sh file

cd ../django-aws-backend
mkdir scripts 
touch scripts/deploy.sh 
chmod 777 scripts/deploy.sh

with the following content:

#!/bin/bash

set -e

# Collect ECS_GROUP_ID and PRIVATE_SUBNET_ID for running migrations
echo "Collecting data..."
ECS_GROUP_ID=$(aws ec2 describe-security-groups --filters Name=group-name,Values=prod-ecs-backend --query "SecurityGroups[*][GroupId]" --output text)
PRIVATE_SUBNET_ID=$(aws ec2 describe-subnets  --filters "Name=tag:Name,Values=prod-private-1" --query "Subnets[*][SubnetId]"  --output text)

echo "Running migration task..."
# Construct NETWORK_CONFIGURATON to run migtaion task 
NETWORK_CONFIGURATON="{\"awsvpcConfiguration\": {\"subnets\": [\"${PRIVATE_SUBNET_ID}\"], \"securityGroups\": [\"${ECS_GROUP_ID}\"],\"assignPublicIp\": \"DISABLED\"}}"
# Start migration task
MIGRATION_TASK_ARN=$(aws ecs run-task --cluster prod --task-definition backend-migration --count 1 --launch-type FARGATE --network-configuration "${NETWORK_CONFIGURATON}" --query 'tasks[*][taskArn]' --output text)
echo "Task ${MIGRATION_TASK_ARN} running..."
# Wait migration task to complete
aws ecs wait tasks-stopped --cluster prod --tasks "${MIGRATION_TASK_ARN}"

echo "Updating web..."
# Updating web service
aws ecs update-service --cluster prod --service prod-backend-web --force-new-deployment  --query "service.serviceName"  --output json

echo "Done!"

You can check the docs for run-task, wait tasks-stopped, and update-service commands.

Run this script locally to check it:

$ ./scripts/deploy.sh
Collecting data...
Running migration task...
Task arn:aws:ecs:us-east-2:947134793474:task/prod/dcc06d7ec1ac4bb69bba445565eddf8b running...
Updating web...
"prod-backend-web"
Done!

We've successfully run this script by the admin user. GitLab CI/CD will use the gitlab user, so we need to grant all required permissions.

Let's create a new policy gitlab-deploy-ecs, and add it to the gitlab user. Go to the IAM Console, select the "Users" tab and click on the gitlab user.

Next, click on "Add inline policy" and add the JSON policy definition. You need to use your AWS_ACCOUNT_ID instead of 947134793474 number.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ecs:UpdateService",
                "ecs:DescribeTasks"
            ],
            "Resource": ["*"]
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": [
                "arn:aws:iam::947134793474:role/ecs-task-execution",
                "arn:aws:iam::947134793474:role/prod-backend-task"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "ecs:RunTask",
            "Resource": "arn:aws:ecs:us-east-2:947134793474:task-definition/backend-migration*"
        }
    ]
}

Explaining policies:

ec2:DescribeSubnets, ec2:DescribeSecurityGroups — for "Collecting data..." stage.
ecs:RunTask, iam:PassRole — for running a migration ECS task.
ecs:DescribeTasks — for waiting a migration ECS task ends.
ecs:UpdateService — for updating the ECS Django web application.

Click "Review Policy", name the policy gitlab-ecs-deploy and click "Create Policy". Now, the gitlab user will be able to execute the deploy.sh script.

Add deploy stage to .gitlab-ci.yml

...

stages:
  - test
  - build
  - deploy

...

deploy:
  stage: deploy
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - ./scripts/deploy.sh
  only:
    - main

At the deploy stage, we simply run deploy.sh script. We use the aws-base image to have access to AWS CLI commands.

Finally, let's add some changes to Django to see that our application will be updated automatically. Let's change the Django Admin header text. Add this line to django_aws/urls.py:

admin.site.site_header = "Django AWS Admin Panel"

Now commit your changes and see how's your pipeline going.

$ git add .
$ git commit -m "add deploy stage"
$ git push

Check your admin page prod-57218461274.us-east-2.elb.amazonaws.com/admin and see a new title. It can take some time to update the ECS service.

Also, do not forget to push infrastructure code to GitLab.

Congratulations! We've successfully set up CI/CD with GitLab for our web Django web application. Now we can commit our code to the main branch, and GitLab CI/CD will automatically test, build and deploy it on the AWS.

But the prod-57218461274.us-east-2.elb.amazonaws.com domain looks not user-friendly :) Also, we need to secure the connection between a user and the Django application with an SSL certificate. In the next part, we'll connect the Namecheap domain to AWS and set up an SSL certificate for them.

You can find the source code of backend and infrastructure projects here and here.

If you need technical consulting on your project, check out our website or connect with me directly on LinkedIn.

Deploying Django Application on AWS with Terraform. Connecting PostgreSQL RDS

Yevhen Bondar — Fri, 29 Jul 2022 09:50:00 +0000

In the previous part of this guide, we deployed the Django web application on AWS ECS.

In this part, we'll create a PostgreSQL RDS instance on AWS, connect it to the Django ECS task and enable access to Django Admin. We choose PostgreSQL because it supports complex SQL queries, MVCC, good performance for concurrent queries, and has a large community. As AWS says:

PostgreSQL has become the preferred open source relational database for many enterprise developers and start-ups, powering leading business and mobile applications. Amazon RDS makes it easy to set up, operate, and scale PostgreSQL deployments in the cloud.

Add PostgreSQL to Django project

Go to the django-aws-backend folder and activate virtual environment: cd ../django-aws-backend && . ./venv/bin/activate.

First, let's set up PostgreSQL via Docker for local development. Create docker-compose.yml with the following content:

version: "3.1"

services:
  postgres:
    image: "postgres:14"
    ports:
      - "5433:5432"
    environment:
      POSTGRES_PASSWORD: "postgres"
      POSTGRES_DB: "django_aws"

Then run docker-compose up -d to start a container. Now, let's connect Django to PostgreSQL.

We use a non-standard 5433 port to avoid potential conflict with your local PostgreSQL

Next, we need to add pip packages: psycopg2-binary for working with PostgreSQL and django-environ to retrieve PostgreSQL connection string from environment. Also, we'll add the WhiteNoise package to serve static files for Django Admin.

Add to the requirements.txt file these packages and run pip install -r requirements.txt:

django-environ==0.8.1
psycopg2-binary==2.9.3
whitenoise==6.1.0

Now, change settings.py. First, let's load env variables. We try to read the .env file in the project root if it exists.

import environ

BASE_DIR = Path(__file__).resolve().parent.parent

env = environ.Env()
env_path = BASE_DIR / ".env"
if env_path.is_file():
    environ.Env.read_env(env_file=str(env_path))

Now we can use environment variables in settings.py. Let's provide DATABASES from env:

DATABASES = {
    'default': env.db(default="postgresql://postgres:postgres@127.0.0.1:5433/django_aws")
}

We've set a default value for the DATABASE_URL to allow running the Django project locally without specifying this variable in the .env file.

Third, add the WhiteNoiseMiddleware to serve static files and specify the STATIC_ROOT variable.

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    "whitenoise.middleware.WhiteNoiseMiddleware",
    ...
]

...

STATIC_URL = '/static/'
STATIC_ROOT = BASE_DIR / "static"

Also, add the RUN ./manage.py collectstatic --noinput line to the bottom of Dockerfile to collect static files in the static folder.

Apply migrations, create a superuser and start a web server:

(venv) $ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
...
(venv) $ python manage.py createsuperuser
Username: admin
Email address: 
Password: 
Password (again): 
Superuser created 
(venv) $ python manage.py runserver
...
Starting development server at http://127.0.0.1:8000/

Go to http://127.0.0.1:8000/admin/ and sign in with your user.

Finally, let's commit changes, build and push the docker image.

$ git add . 
$ git commit -m "add postgresql, environ and static files serving"
$ docker build . -t 947134793474.dkr.ecr.us-east-2.amazonaws.com/django-aws-backend:latest
$ docker push 947134793474.dkr.ecr.us-east-2.amazonaws.com/django-aws-backend:latest

We are done with the Django part. Next, we create a PostgreSQL instance on AWS and connect it to the ECS task.

Creating RDS

AWS provides the Relational Database Service to run a PostgreSQL. Now, we'll describe the RDS setup with Terraform. Go to the Terraform folder ../django-aws-infrastructure and add a rds.tf file:

resource "aws_db_subnet_group" "prod" {
  name       = "prod"
  subnet_ids = [aws_subnet.prod_private_1.id, aws_subnet.prod_private_2.id]
}

resource "aws_db_instance" "prod" {
  identifier              = "prod"
  db_name                 = var.prod_rds_db_name
  username                = var.prod_rds_username
  password                = var.prod_rds_password
  port                    = "5432"
  engine                  = "postgres"
  engine_version          = "14.2"
  instance_class          = var.prod_rds_instance_class
  allocated_storage       = "20"
  storage_encrypted       = false
  vpc_security_group_ids  = [aws_security_group.rds_prod.id]
  db_subnet_group_name    = aws_db_subnet_group.prod.name
  multi_az                = false
  storage_type            = "gp2"
  publicly_accessible     = false
  backup_retention_period = 5
  skip_final_snapshot     = true
}

# RDS Security Group (traffic ECS -> RDS)
resource "aws_security_group" "rds_prod" {
  name        = "rds-prod"
  vpc_id      = aws_vpc.prod.id

  ingress {
    protocol        = "tcp"
    from_port       = "5432"
    to_port         = "5432"
    security_groups = [aws_security_group.prod_ecs_backend.id]
  }

  egress {
    protocol    = "-1"
    from_port   = 0
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Here we create an RDS instance, a DB subnet group, and a Security Group to allow incoming traffic from ECS 5432 port only. We'll specify db_name, username, password, and instance_class in the variables.tf:

# rds

variable "prod_rds_db_name" {
  description = "RDS database name"
  default     = "django_aws"
}
variable "prod_rds_username" {
  description = "RDS database username"
  default     = "django_aws"
}
variable "prod_rds_password" {
  description = "postgres password for production DB"
}
variable "prod_rds_instance_class" {
  description = "RDS instance type"
  default     = "db.t4g.micro"
}

Notice that we haven't provided a default value for the prod_rds_password variable to prevent committing the database password to the repository. Terraform will ask you for a variable value if you try to apply these changes.

$ terraform apply             
var.prod_rds_password
  postgres password for production DB

  Enter a value:

It's not convenient and error-prone to type a password every time. Gladly, Terraform can read variable values from the environment. Create .env file with TF_VAR_prod_rds_password=YOUR_PASSWORD variable. Interrupt password prompting, run export $(cat .env | xargs) to load .env variable, and rerun terraform apply. Now Terraform picks the password from the .env file and creates an RDS PostgreSQL instance. Check it in the AWS RDS console.

Connecting to ECS

Let's connect the RDS instance to ECS tasks.

Add to backend_container.json.tpl environment var DATABASE_URL

[
  {
    ...
    "environment": [
      {
        "name": "DATABASE_URL",
        "value": "postgresql://${rds_username}:${rds_password}@${rds_hostname}:5432/${rds_db_name}"
      }
    ],
    ...
  }
]

Add RDS variables to the prod_backend_web task definition in the ecs.tf file:

resource "aws_ecs_task_definition" "prod_backend_web" {
  ...

  container_definitions = templatefile(
    "templates/backend_container.json.tpl",
    {
      ...

      rds_db_name  = var.prod_rds_db_name
      rds_username = var.prod_rds_username
      rds_password = var.prod_rds_password
      rds_hostname = aws_db_instance.prod.address
    },
  )
  ...
}

Let's apply changes and update the ECS service with the new task definition. Run terraform apply, stop current task via web console and wait for the new task to arise.

Now, go to the admin URL on the load balancer hostname and try to log in with random credentials. You should get the relation "auth_user" does not exist error. This error means that the Django application successfully connected to PostgreSQL, but no migrations was run.

Running migrations

Now, we need to run migrations and create a superuser. But how can we do it? Our infrastructure has no EC2 instances to connect via SSH and run this command. The solution is ECS Exec.

With Amazon ECS Exec, you can directly interact with containers without needing to first interact with the host container operating system, open inbound ports, or manage SSH keys.

First, we need to install the Session Manager plugin. For macOS, you can use brew install session-manager-plugin. For other platforms, check this link.

Next, we need to provide an IAM policy to the prod_backend_task role and enable execute_command for the ECS service. Add this code to the ecs.tf and apply changes:

resource "aws_ecs_service" "prod_backend_web" {
  ...
  enable_execute_command             = true
  ...
}

resource "aws_iam_role" "prod_backend_task" {
  ...

  inline_policy {
    name = "prod-backend-task-ssmmessages"
    policy = jsonencode({
      Version = "2012-10-17"
      Statement = [
        {
          Action   = [
            "ssmmessages:CreateControlChannel",
            "ssmmessages:CreateDataChannel",
            "ssmmessages:OpenControlChannel",
            "ssmmessages:OpenDataChannel",
          ]
          Effect   = "Allow"
          Resource = "*"
        },
      ]
    })
  }
}

Stop the old ECS task in web console and wait for the new task.

Then let's retrieve the new task_id via CLI and run aws esc execute-command. Create a new file touch backend_web_shell.sh && chmod 777 backend_web_shell.sh with the content:

#!/bin/bash

TASK_ID=$(aws ecs list-tasks --cluster prod --service-name prod-backend-web  --query 'taskArns[0]' --output text  | awk '{split($0,a,"/"); print a[3]}')
aws ecs execute-command --task $TASK_ID --command "bash" --interactive --cluster prod --region us-east-2

Run ./backend_web_shell.sh to shell into task. If you have some troubles, please check Amazon ECS Exec Checker. This script checks that your CLI environment and ECS cluster/task are ready for ECS Exec.

Run migrations and create a superuser from the task's console:

$ ./manage.py migrate
$ ./manage.py createsuperuser 
Username: admin
Email address: 
Password: 
Password (again): 
Superuser created

After this, check your deployment admin URL and try to sign in with provided credentials.

Congratulations! We've successfully connected PostgreSQL to the ECS service. Now you can commit changes in the Terraform project and move to the next part.

In the next part we'll set up CI/CD with GitLab.

You can find the source code of backend and infrastructure projects here and here.

If you need technical consulting on your project, check out our website or connect with me directly on LinkedIn.