DEV Community: wassef ben ahmed

Deploying Django in Production.

wassef ben ahmed — Sun, 23 Jun 2024 20:40:38 +0000

This 5-step tutorial will guide you through deploying a Django application using Gunicorn behind a reverse proxy (such as Nginx).

Its important to note that this greatly depends on the needs of your project, but for most people, this is intended to be a minimal/initial setup. This configuration ensures that your application is served efficiently and securely, straight to production 🚀.

Prerequisites

Before starting, ensure you have the following:

A domain name. (duh)
A server with root access (e.g. an Ubuntu server).
A Django application ready for deployment. (eg. being under /opt/project_name)
A virtual environment. (eg. being under /opt/project_name/venv)
Basic knowledge of Linux command-line operations.
Nginx installed on your server.

Step 0: Setting Up Your Domain and DNS

Before deploying your Django application, ensure your domain points to your server and DNS settings are correctly configured. Here’s a brief guide on how to do this:

Domain Registration:
- Register your domain with a domain registrar.
Obtain Server IP Address:
- Get the public IP address of your server. This is the address that clients will use to access your application.
DNS Configuration:
- Log in to your domain registrar’s control panel and find the DNS settings for your domain.
- Add an A record to point your domain to your server's IP address:
  - Type: A
  - Name: @ (this represents your domain, e.g., your_domain.com)
  - Value: [Your Server's IP Address]
  - TTL: Default or 3600 seconds (1 hour)
- If you want to use www.your_domain.com, add a CNAME record:
  - Type: CNAME
  - Name: www
  - Value: your_domain.com
  - TTL: Default or 3600 seconds (1 hour)
Propagation Time:
- DNS changes may take some time to propagate (up to 48 hours, but usually within a few hours). You can use tools like whatsmydns.net to check the propagation status.
Verify DNS Settings:
- After DNS propagation, you should be able to verify your domain points to your server by using tools like ping or dig:
```
 ping your_domain.com
```

You should see responses from your server's IP address.

Firewall and Security Group Configuration:
- Ensure your server's firewall and any cloud provider security groups allow traffic on ports 80 (HTTP) and 443 (HTTPS).

Once your DNS is set up and pointing to your server, you can proceed with the deployment steps outlined in the tutorial. This setup ensures that visitors accessing your domain will be directed to your server, where Nginx and Gunicorn will handle the requests.

Step 1: Install Gunicorn

First, activate your virtual environment and install Gunicorn:

source /opt/project_name/venv/bin/activate
python3 -m pip install gunicorn

Step 2: Configure Gunicorn

Create a Gunicorn configuration file (optional but recommended) to manage settings like worker processes. Create a file named gunicorn_config.py in your project directory:

# gunicorn_config.py

bind = "127.0.0.1:8000"
workers = 3

Step 3: Adjust Django Settings

Update your Django settings to handle the proxy setup correctly. Edit your settings.py file to include the following:

# settings.py

# Static files (CSS, JavaScript, Images)
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')

# Media files (Uploaded by users)
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

# Specifies a list of valid host/domain names for the Django site, providing protection against HTTP Host header attacks.
ALLOWED_HOSTS = ['your_domain.com', 'www.your_domain.com', 'another_domain.com']

# Tells Django to use the X-Forwarded-Host header from the proxy, allowing it to know the original host requested by the client.
USE_X_FORWARDED_HOST = True
# Tells Django to use the X-Forwarded-Port header from the proxy, indicating the port number used by the client.
USE_X_FORWARDED_PORT = True
# Instructs Django to trust the X-Forwarded-Proto header, which is set by the proxy server, to determine whether the request is secure (HTTPS).
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

# Forces all HTTP requests to be redirected to HTTPS.
SECURE_SSL_REDIRECT = True

# Ensures that the CSRF cookie is only sent over HTTPS connections.
CSRF_COOKIE_SECURE = True
# Ensures that the session cookie is only sent over HTTPS connections.
SESSION_COOKIE_SECURE = True

# Enables HTTP Strict Transport Security (HSTS) for the specified duration (in seconds), forcing browsers to only connect via HTTPS.
SECURE_HSTS_SECONDS = 31536000
# Applies HSTS policy to all subdomains.
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
# Allows the domain to be included in browsers' HSTS preload list, ensuring maximum protection.
SECURE_HSTS_PRELOAD = True

# Enables the X-Content-Type-Options header, preventing browsers from MIME-sniffing a response away from the declared content-type.
SECURE_CONTENT_TYPE_NOSNIFF = True
# Controls the information sent in the Referer header, improving privacy and security by not sending the referrer from HTTPS to HTTP.
SECURE_REFERRER_POLICY = 'no-referrer-when-downgrade'

# SECURE_BROWSER_XSS_FILTER = True
# [Deprecated on 4.0]
# Enables the X-XSS-Protection header, which tells browsers to block detected cross-site scripting (XSS) attacks.

Collect Static Files

Django comes with a handy command to collect all static files into the directory specified in STATIC_ROOT.

python3 -m /opt/project_name/manage.py collectstatic

Step 4: Create a Systemd Service for Gunicorn

Create a systemd service file to manage the Gunicorn process. Create a file named gunicorn.service in /etc/systemd/system/:

# /etc/systemd/system/gunicorn.service

[Unit]
Description=gunicorn daemon
After=network.target

[Service]
User=yourusername
Group=www-data
WorkingDirectory=/opt/project_name
ExecStart=/opt/project_name/venv/bin/gunicorn --config /opt/gunicorn_config.py project_name.wsgi:application

[Install]
WantedBy=multi-user.target

Replace /opt/project_name, /opt/project_name/venv, and project_name with your actual project paths and names.

Reload the systemd daemon and start the Gunicorn service:

sudo systemctl daemon-reload
sudo systemctl start gunicorn
sudo systemctl enable gunicorn

Step 5: Configure Nginx

Create an Nginx configuration file to proxy requests to Gunicorn. Create a file named project_name in /etc/nginx/sites-available/:

# /etc/nginx/sites-available/project_name

server {
    listen 80;
    server_name your_domain.com www.your_domain.com another_domain.com;

    location = /favicon.ico { access_log off; log_not_found off; }

    location /static/ {
        alias /opt/project_name/static/;  # Must be the same as STATIC_ROOT
    }

    location /media/ {
        alias /opt/project_name/media/;  # Must be the same as MEDIA_ROOT
        access_log /var/log/nginx/media_access.log;
        error_log /var/log/nginx/media_error.log;
    }

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://127.0.0.1:8000;

        access_log /var/log/nginx/django_access.log;
        error_log /var/log/nginx/django_error.log;
    }

    listen [::]:80;
}

Create a symbolic link to enable the site, and ensure the static and media directories, as well as the log directories, have the correct permissions.

You need to make sure that the user running Nginx has the necessary read/write permissions.

# Create directories if they don't exist
sudo mkdir -p /opt/project_name/static /opt/project_name/media /var/log/nginx

# Set ownership (assuming nginx user and group) and permissions
sudo chown -R nginx:nginx /opt/project_name/static /opt/project_name/media /var/log/nginx
sudo chmod -R 755 /opt/project_name/static /opt/project_name/media /var/log/nginx

sudo ln -s /etc/nginx/sites-available/project_name /etc/nginx/sites-enabled

Test the Nginx configuration and restart the service:

sudo nginx -t
sudo systemctl restart nginx

Step 6: Configure HTTPS

It's highly recommended to secure your application with HTTPS. Use Certbot to obtain a free SSL certificate:

sudo apt-get install certbot python3-certbot-nginx
sudo certbot --nginx -d your_domain.com -d www.your_domain.com -d another_domain.com

Follow the prompts to configure SSL.

Conclusion

Your Django application is now deployed using Gunicorn behind an Nginx reverse proxy.

However, there are several ways you could customize and enhance this deployment:

Load Balancing: If you anticipate high traffic, consider setting up multiple Gunicorn instances behind a load balancer to distribute the load evenly.
Containerization: Using Docker to containerize your application can simplify deployment and provide greater consistency across different environments.
Advanced Security Configurations: Additional security measures such as setting up a Web Application Firewall (WAF), regular security audits, and implementing stricter Content Security Policies (CSP) can further protect your application.

Be sure to explore these options and refer to the official documentation for Gunicorn, Nginx, and Django for further customization and advanced configurations. By continuously refining your deployment setup, you can ensure your Django application remains secure, efficient, and scalable.

Ship Slightly Better Microservices.

wassef ben ahmed — Mon, 04 Sep 2023 12:58:33 +0000

Introducing env_should_be

In the fast-paced oblate spheroid we call home, building microservices and multi-environment deployments, ensuring the integrity, security, and compatibility of containers is a monumental challenge. Battling with microservices and incremental development, I've seen firsthand the complexities and frustrations that arise with maintaining consistency across different deployment environments. I was trying to solve this problem and, in doing so, created env_should_be—a PyPI package that might change your approach to building containers… (We’ll see about that!)

The Motivation Behind env_should_be

Microservices architecture, with its flexibility and scalability, has become the standard for modern software development. However, it also brings a new set of challenges, one of which is ensuring that containers are running in environments that are compatible and secure. Traditional methods often involve extensive coordination between developers and operations teams, leading to bottlenecks, configurations, and security vulnerabilities.

The need for a more efficient, automated, and developer-friendly approach became evident. env_should_be was created to address these challenges by allowing developers to define deployment environment requirements using simple JSON or YAML descriptions. This empowers developers to specify the exact conditions under which a container can run, mitigating compatibility issues and enforcing security standards seamlessly… This environment description can either be one or multiple files and can also be a combination for each type of environment…

env_should_be in Action

So, how does env_should_be work, and what sets it apart? At its core, env_should_be is a command-line interface (CLI) tool that you can effortlessly integrate into your container deployment pipelines. It takes deployment environment descriptions in JSON or YAML format as input, allowing you to specify requirements for various environment variables, file paths, and even execution callbacks…. It also can be shipped as part of the actual container or be used as a pipeline stage, to validate the environment before deploying…

Environment Descriptions

Imagine you have a flask microservice that relies on a database and cache, and you want to ensure that certain environment variables meet specific criteria each time before the app starts, Here's an example YAML description:

DB_USER:
  length: 8
  regex: "^[a-zA-Z0-9]+$"
DB_PASSWORD:
  length: 12
  regex: "^(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9])(?=.*[!@#\\$%\\^&\\*])(?=.{8,})"
DB_HOST:
  option:
    - localhost
    - 127.0.0.1
DB_PORT:
  length: 4
  regex: "^[0-9]+$"
  is_int: True
  is_float: False
CACHE: # sure that it has the same value across whatever env?
  constant: "redis://container_name:6379/1"
APP_ENV:
  option:
    - dev
    - prod
  required: false # only get's validated if it is set (not None)

then a compose file:

version: '3'
services:
  app:
    build:
      context: .
    expose:
      - 5000
    environment:
    command: >
      sh -c '
        env_should_be -d "/app/descriptions/app.json" && \
        flask run --host=0.0.0.0 --port 5000'

env_should_be will check these conditions before allowing the container to run. If any condition is not met, it can either block the container from running or using the -fs argument, let it start and log the errors, depending on your preference…

Callback Scripts

But that's not all—env_should_be also provides the ability to execute callback scripts whenever a container attempts to start with an environment that doesn't match the defined description. This feature is incredibly powerful because it enables you to take immediate corrective actions or trigger notifications when something is amiss.

Why env_should_be Matters

env_should_be offers a multifaceted solution to the challenges of microservices and multi-environment deployments:

1. Consistency and Compatibility

By specifying environmental requirements, it ensures that containers only run in environments that can support them. This eliminates compatibility issues, reducing the likelihood of runtime errors and downtime.

2. Security and Compliance

It empowers you to enforce security standards by defining strict criteria for environment variables. You can rest assured that your microservices are operating in a secure environment, mitigating potential security vulnerabilities.

3. Developer-Operations Collaboration

With it, developers can define deployment requirements independently, reducing the need for constant coordination with operations teams. This fosters a more efficient and autonomous development workflow.

4. Error Handling and Notifications

Error-handling options allow you to choose whether to block containers or let them start with errors logged. Additionally, the ability to trigger callback scripts ensures swift responses to deployment issues.

Getting Started

To get started, you can easily install it via PyPI and integrate it into your Docker containers.

# I believe you are using a venv, right?  
python -m pip install env_should_be

full list of possible descriptions:

(
  "boolean",
  "length",
  "min_length",
  "max_length",
  "regex",
  "option",
  "constant",
  "is_int",
  "is_str",
  "is_float",
  "is_number",
  "is_greater_than_eq",
  "is_lower_than_eq",
  "is_http",
  "is_https",
  "is_ipv4",
  "is_ipv6",
  "is_email",
  "is_uuid",
)

Feel free, to try it/ break it, and open PR.

source code

latest version

The Concept Of Virtualization (hypervisor)

wassef ben ahmed — Thu, 13 May 2021 20:06:04 +0000

There are 7 primary types of virtualization, Hypervisor being the topic of this article.

We'll start by taking a look at what a server looks like without Virtualization.

We'll start by taking a look at what a server looks like without virtualization. In this example, our server is working with the physical hardware that it has, it's running a Windows OS that is serving a website over HTTP. As simple as this configuration might be we can run to some limitations along the way:

our OS is tied to hardware, which means it is not portable and we can't easily move that OS and the application that is installed on it to another piece of hardware if we have some sort of issue or if wanted to upgrade the specs due to more/less traffic.
the hardware might be underutilized because of the wrong estimation of what we need and in the context of cloud computing, this will cause a higher bill.

Let's now look at virtualization how it can help us solve some of these issues.

So with a virtualization stack, we have a server with an extra layer of abstraction.

this layer is able to present virtualized hardware to the operating system, in that case, we call it a virtual machine or VM for short. The VM has an allocation of resources, maybe the server has 32GB of RAM but we just give 8GB the VM, also a certain amount of CPU power, disk space, and an allocation of shared networking... There is no physical hardware here it's virtualized and presented by the hypervisor but the OS doesn't know that, it just sees hardware.

the cool part about this is being able to run multiple VMs on the same physical hardware, another benefit is that any instance of these VMs can easily be moved to another server if it failed, went down, or if you wanted to perform some upgrades on hardware.

useful resources:

running Hyper-V in EC2

Amazon Machine Images (AWS AMI)

What virtualization software does EC2 use?

Open Source Software Licensing.

wassef ben ahmed — Sat, 17 Oct 2020 17:53:03 +0000

The use of open-sourced code is inevitable. Whatever you may be working on, you will likely make use of code snippets from the web or some library that implements a sub-dependency relying on other people’s code.

Is it risky to use open source software:

Some open-source licenses generally allow the use of their components as long as you maintain any copyright notices. But if you use a code with a restrictive license in your proprietary software, you might be legally obligated to release your software under the same license.

What are the different types of open-source licenses:

Public domain: It means that anyone can modify and use the software without any restrictions

Permissive: contains minimal requirements about how the software can be modified or redistributed. Also known as “Apache-style” or “BSD style” another common variant is the MIT License.

LGPL: The GNU Lesser General Public License allows you to link to open source libraries in your software. If you simply compile or link an LGPL-licensed library with your own code, you can release your application under any license you want, even a proprietary license.

Copyleft: also known as reciprocal licenses or restrictive licenses, allow you to modify the code and distribute new works based on it, as long as you meet the requirements for redistribution under the same license.

What kind of risks are there using open-source licenses:

Low-risk: Permissive licenses are considered low risk because it’s easy to meet their reuse requirements: Usually, you just have to retain the copyright notice, but you don’t have to expose your source code. Examples are the Apache and MIT Licenses.

Medium-risk: Semi-permissive licenses sometimes referred to as limited licenses, weak copyleft licenses, or simple copyleft licenses, are considered a medium risk because if you modify the code, you have to release the modifications, but not your whole application, under the same license. Different licenses define “modification” differently. Examples are the Mozilla and the Eclipse Public Licenses.

High-risk: Restrictive licenses carry a great deal of legal risk. If you use a component with one of these licenses, you might be legally obligated to release your entire application code. Examples are the GNU GPL and GNU LGPL.

should you learn THE CLOUD?

wassef ben ahmed — Sun, 11 Oct 2020 20:07:25 +0000

keeping-up with technologies is very hectic, but bear with me, and let’s see if you should invest your time in it? or maybe not?

What is the cloud, I hear you ask! It’s 2020 everyone has heard of the term, but it seems like people don’t really know what it means, and they have weird expectations about what it can do. Let’s start with a simple definition.

cloud computing: the delivery of on-demand computing services, from applications to storage and processing power over the internet, and on a pay-as-you-go basis.

do we really need it?

Yes, we DO! Whoever you may be, running your own startup, software developer, or just a user of a cloud application ( Gmail, Google Drive, Google Docs). Everyone will benefit time and money. Though I can nerd-out and keep ranting about the endless possibilities of cloud computing, I’m keeping it short so you don't spend the day on an article.

Are people really switching to the cloud?

Cloud-based apps are expanding by the day. Their number has nearly tripled between 2013 and 2016, from 545 to 1427 different services and revenue from the sector is expected to grow in adoption to 56% by 2020, jumping to $331 billion by 2022 (from $175.8 billion in 2018).

should you learn it?

Maybe, It all comes back to what you are interested to build. As it has not fully dominated the field yet, it’s not a must, but it will be replacing a lot of jobs and creating new ones in the near future, so I think that early adopters we’ll have great leverage. If you have the time, I strongly recommend that you take the jump.

Top cloud providers (2020):

1. Amazon Web Services.

2. Microsoft Azure.

3. Google Cloud Platform.

Git terms for dummies!

wassef ben ahmed — Wed, 05 Aug 2020 02:55:51 +0000

Mid journey toward learning to code you came across the term GIT whether it is mentioned in a tutorial, or you've seen it on every job posting out there. This is the article for you, I'll try to explain the most common terms used and give you a glimpse of what this tool has to offer.

GIT :

Git is a version control system, it’s simply a tool for programmers to coordinate their work, providing a better workflow for teams.

Repository :

Repository is a treasury for your code.
A repository consists of a folder containing files of code / images / text …

Remote :

Remote repository is a version of your project on the internet that is hosted on a web-based version control like GitHub.

Branching :

Branching out or creating a new branch is having another copy of your code on the same repository and is used to give developers the freedom of experimenting and working on new features without the fear of altering the main project.

Merging :

Merging is combining multiple branches of the project on the main branch that is commonly referred to as master.

Merge conflicts :

Merge conflicts happens after trying to merge two branches caused by having concurrent changes in the same file that git doesn't know how to deal with.

The 3 git states :

Modified: A tracked file has been altered.
Staging: Files that are ready to be committed.
Committed: A file the has been added to the last commit, committing is creating an image of the repository. Every commit has a hash to identify it, the author’s name, date, and a commit message.

What is POSIX?

wassef ben ahmed — Tue, 21 Jul 2020 23:27:43 +0000

It sure sounds like a cool name!
POSIX stands for Portable Operating System Interface and is a family of standards, specified by the IEEE designed to facilitate application portability across UNIX based systems.

If you wrote your programs to rely on POSIX standards, you'll be able to port them easily among a large family of Unix derivatives.

Honorable mentions of UNIX SYSTEMS :

Some of the standards POSIX defines :

1.C API

Greatly extends STANDARD C with things like:

file operations: mkdir, dirname, symlink, readlink, link.
networking: socket().
memory management: mmap, mlock, mprotect, advise. Those APIs also determine the underlying system concepts on which they depend, forcing you to conform.

2.CLI utilities

E.g: cd, ls, echo ...
Many utilities are direct shell front ends for a corresponding C API functions and some are implemented by Bash as built-ins.

3.Shell language

Support for the shell language.
E.g :

a = 5;
echo "$a"; #to print the variable a.

4.Program exit status

ANSI C says 0 or EXIT_SUCCESS for success, EXIT_FAILURE for failure.
what POSIX adds:

126: command found but not executable.
127: command not found.
>128: terminated by a signal.

5.Regular expression

Support for the two types of regex (BRE/ERE).
E.g :

echo 'a.1' | grep -E 'a.[[:digit:]]' 
#search for a regex from the piped input

6.Filenames

/ is the path separator.
NUL cannot be used.
. is cwd, .. is parent.
can only contain a-zA-Z0-9._-

What is a Load Balancer?

wassef ben ahmed — Thu, 09 Jul 2020 21:18:14 +0000

Why?

imagine that you've worked on a startup website or a web app for a client and it started to get 100 visits per month. one year in, and the number of visits drastically increased to the point that the client called you complaining about the app's performance, so being the great engineer you are, you decided to upgrade the server's specs hoping it will solve the response time problem and it worked!

But what if the server hits 3000 visits per minute and costumers started getting a "502 bad gateway" and upgrading the server specs isn't an option being too expensive or that you are already using the maxed out specs the hosting service could offer.

Hosting services list :
1. Bluehost.
2. HostGator.
3. InMotion.

Upgrading the specs or what is called vertical scaling is sometimes accepted if used on a database server being that you need more storage or a better CPU to process data ...

But what if, we still run into errors even after the upgrade?
many factors may come into play even with a maxed-out server, the CPU handling the threads of different requests simultaneously will cause it to prioritize some of them in favor of others causing unwanted/unexpected behaviors...

Imagine being in a restaurant that has 10 clients and a waiter that can't handle all incoming requests, so we replaced him with a four-armed waiter but that also didn't solve the problem being that, he takes a lot of trips to the kitchen and a client may order 4 plates that will occupy all 4 hands.

Clearly the solution isn't surgically adding more hand to the waiter but in hiring another!
He'll take some load off the first and we'll be distributing the pressure on both.
That's what we're calling :

horizontal scaling :
adding more servers instead of maxing out the main server, which will consume a bit of the resources in each server making it easy to just add more, in a rush hour, and costing less than a gigantic server.

To apply horizontal scaling we need to deploy the code like we'ed normally do but on multiple servers (let's say 10)
Having the code duplicated on multiple places with NGINX as a load balancer, it will distribute the incoming requests on our 10 servers