DEV Community: Sean

Project #3: Create a CI/CD pipeline for an existing Django project

Sean — Sat, 25 Feb 2023 19:22:35 +0000

I finished this project here taking a project from testdriven.io and then re-writing it to use GitHub Actions to deploy to my Linode server.

I have a previous Django project I built about a year ago and a particular way I like to structure my projects. So I'll be re-writing the actions to work with that project.

Project #2: Docker/Django project from testdriven.io (status: in-progress)

Sean — Mon, 19 Dec 2022 02:20:38 +0000

I started a course from Test-Driven Development with Django, Django REST Framework, and Docker

Status: Complete

2023-02-25
I haven't updated this in a while but for a good reason: I was actually doing the work and I can happily report that I have completed this project. According to my logs, I've put in 30+ hours into it and have learned a lot.

I took the TDD Django Project from testdriven and built a CI pipeline which:

Built a CI version of my latest code commits and ran tests and linting
Once those tests passed, it built a production docker image and pushed it up to GitHub Container Registry
If that job was successful, another job pulled down the latest images from GitHub Container Registry and copied the secret .env file + production docker-compose file and built the production image in Linode and deployed

In the end I needed 3 guides to complete this project:

2023-01-17
After getting my first build to run, I was beating my head trying to run the tests. I am trusting my gut in everything and decided to do a GitHub Actions course on Udemy. I'll go through this over the next two weeks so that I can really understand Actions. I'm not trying to piece together a buncha random StackOverflow articles. I'm doing this legit.

2023-01-12
I've been putting in time this week. In my project, added flake8/black/isort which are great packages I will use for my projects in the future. It got to a point where the tutorial wanted me to sign-up for GitLab + upload my image to Docker Hub but I wanted to use GitHub Actions + GitHub Container Registry. I watched a video from Nana Techworld and after bashing my head against the wall for an hour I finally got a passing build (though it is incomplete)!

2022-12-24
Worked on this about 1.5 hours today. It's looking good but it's quite Heroku specific, which I have no plans of using for future projects. Merry Christmas to all

2022-12-18
It's going pretty smoothly right now, I've put roughly 2.5 hours into it. The project that we build ends up deploying to Heroku but when I'm done with the project I'll try to adapt it to deploy to my Linode.

Project #1: Deploy a Docker Container on Linode (status: Abandoned)

Sean — Tue, 22 Nov 2022 03:10:29 +0000

Status: Abandoned

Background

A couple of years ago, I worked on a project that dockerized a Django application that displayed a basic 'Upload' form and uploaded the files within the same container

https://github.com/drivelous/docker-ubuntu18.04-django3.0.2

It has a local and production build with the production build firing up an nginx service.

That repo is the product of customizing code that I wrote when I followed this tutorial: https://testdriven.io/blog/dockerizing-django-with-postgres-gunicorn-and-nginx/

Progress Log
2022-12-10
I did everything but couldn't do this. I did a final ~2 hour session trying to get it to work but decided to start a new tutorial from https://testdriven.io/

Since it's a paid course, I know that the code will work. At some point in time, I will return back to this after my knowledge of Docker and Docker Compose gets better.

2022-11-28
I've put in a few hours this past week and while I've learned a bit, ultimately, I ended up posting on StackOverflow to see if anyone could help me out -- https://stackoverflow.com/questions/74606942/docker-nginx-service-cant-communicate-with-django-service-and-returning-bad-req

I also just want to note that this nginx/django guide is great: https://mattsegal.dev/nginx-django-reverse-proxy-config.html

2022-11-22
Still having issues but inching a bit closer.

I realize now that using an official nginx Docker image reroutes all the logs that would normally be in /var/log/nginx/ to either stdout/stderr. This is useful so that outside of the container I can run docker logs <container_name> but since I was looking for debug level logs it didn't help much -- I know my server is returning 400, but why?

So in nginx.conf I added the line below and rebuild the image

upstream polls_django {
    server web:8000;
}

error_log /var/log/nginx/error_foo.log debug;  <--
...

When running docker exec -it <container_name> /bin/sh I took a peak at error_foo.log and saw a lot of Operation timed out requests

2022/11/23 01:34:39 [info] 7#7: *11 client timed out (110: Operation timed out) while waiting for request, client: xx.xx.xxx.xx, server: 0.0.0.0:80
2022/11/23 01:38:40 [info] 7#7: *15 client timed out (110: Operation timed out) while waiting for request, client: xx.xx.xxx.xx, server: 0.0.0.0:80

I had a hunch it was because my environment variables .env.prod didn't have my IP in DJANGO_ALLOWED_HOSTS (which I thought would explain why it works locally but not in production) but that wasn't it.

I have a hunch that something is amiss. Looking at the web application logs, there are no requests that made it to Django. nginx is not able to reach it for some reason.

I've worked an hour on this. I gotta come back to it later on.

2022-11-21
Locally, running the production command, I can get it to work

docker-compose -f docker-compose.prod.yml up --build

Local:

But when I run it on my Linode, I'm getting Bad Request 400.

All I can see in my logs is the 400

$ docker logs docker-ubuntu1804-django302_nginx_1
$ [22/Nov/2022:02:58:25 +0000] "GET / HTTP/1.1" 400 154 "-"...

I've tried getting a shell into nginx and then running cat on the nginx error.log but it freezes

$ docker exec -it docker-ubuntu1804-django302_nginx_1 /bin/sh
$ [nginx] cd /var/log/nginx
$ [nginx] cat error.log
--> (stuck hanging here)

I'm still trying to figure out what's wrong
I can't seem to find the error logs within my container

Project List

Sean — Thu, 17 Nov 2022 00:25:13 +0000

Projects

Deploy my old Docker/Django/Ubuntu container on Linode (Project #1) -- In Progress

Final Project
Build photo uploader written with Django/React. This project will:

Have a basic CI/CD build which automatically deploys changes to main after tests pass
Have a development/production build
Log exceptions in Sentry

Hi everyone, I'm Sean

Sean — Thu, 17 Nov 2022 00:18:30 +0000

I'm starting this blog to outline my journey of learning DevOps.

I've been a professional Software Engineer for 7+ years, mostly building and maintaining full stack web applications written in Django/React. While my skills have been improving writing application logic, actually deploying my apps to the web has always been a huge weakness of mine. I want to change that and write about the learning process here.

I've done a few tutorials in the past (namely Docker Mastery and nginx fundamentals) and of course the various build pipelines at my job are a part of the engineering department's every day essentials, however, I know that really learning the ins-and-outs of DevOps requires designing and building a difficult but neat project that will challenge me while keeping my interest.

My ultimate project is a photo uploading application using Django/React but with all the bells and whistles of a production application. On the way, I will be building some intermediate projects. You can find them in the entry, which I will update.

If you would like to reach me, send me an email at:
sean AT (github handle) DOT com

Questions

Sean — Mon, 14 Nov 2022 16:49:59 +0000

I'm building a photo uploader with all the DevOps bells and whistles. I am using this post to host all my questions. Once I answer them, I'll add the answers to them.

As of November 14, 2022, the way I'm thinking of building it is:

TODO: create a diagram and show here

React application

How should I serve it?
Should I deploy it to a server running nginx? And why?
Should I serve it as a single staticfile hosted on a bucket?
Should I use Docker?

Django application

How do I add data monitoring?
I will use Docker to serve this application. What base image should I use?
I have an application server + a task queue. How do I get these two to talk each other?
How do I spin up new instances and have the load balancer know about them?
How do I spin up new instances in response to data monitoring?
What should I use to manage the database?
How do I get my local build to upload images to one bucket and production build to upload to another?

Misc

How do I manage environment variables in production?
When I push changes to main, how do I create a hook to deploy my latest changes?

Docker

How do I expose my docker container to the outside world?