DEV Community: Adyaksa W

Easy Manual Setup HTTPS In Nginx

Adyaksa W — Wed, 12 Jan 2022 13:59:34 +0000

If you’re lazy like me, you would use certbot to create certificates for HTTPS. This is my preferred method because it’s simple, fast, and can automatically renew a certificate when it expired. Unfortunately, the world ain’t all sunshine and rainbows. There are some occasions where certbot can’t immediately create a certificate, such as:

The server is behind a VPN. Usually, there is a proxy that is already set up. If this is not the case for you, then I hope you know the list of things where the proxy should be set up.
The web server is behind a secured TLD, such as .app. This means you can’t access it on HTTP protocol and certbot can’t create a certification. This is because, on default, certbot uses the HTTP-01 challenge, which means this challenge would fail on the webserver.

If you’re here because of the first reason, then I recommend configuring the proxy for the certbot. Because sooner or later you will need to configure it for automatic certificate renewal. Your future self will thank you if you do this hard work now.

In my case, I have some websites that use the .app domain. Because of this, I need to learn how to set up a certificate manually. For the explanation, I will skip over the certificate creation because it’s already generated in my domain provider. If you want to refresh your understanding of HTTPS, you can check this helpful comic from HowHTTPSWorks.

So the first step is getting your certificate. In my case, I used porkbun as my provider. There will 4 files in the given zip: domain.cert.pem, intermediate.cert.pem, private.key.pem, public.key.pem. If you’re used to certbot file naming like me, this mapping will helps you:

domain.cert.pem (Porkbun) -> fullchain.pem (certbot) -> ssl_certificate (nginx)
intermediate.cert.pem (Porkbun) -> chain.pem (certbot) ->ssl_trusted_certificate(nginx)
private.key.pem (Porkbun) -> privkey.pem (certbot) -> ssl_certificate_key
public.key.pem (Porkbun) -> cert.pem (certbot)

If you want to understand these files more, you can check these StackOverflow discussions:

Now you need to move this file to your server. We can just use SCP to move it like this:

scp artilearn.app-ssl-bundle host@ipaddress:~/

The SSL certificate is successfully copied to our remote server. Now we need to put our website on the server. For this tutorial purpose, we will just use a dummy HTML file just to test our SSL.

The last step is to configure the Nginx config file. We only need to configure 4 things:

The port where our website will be (Line 2)
SSL Certificate for the website (Line 3–5)
Root folder of our website (Line 6)
Domain name of our website (Line 7)

Now restart the Nginx service with sudo systemctl restart nginx. When you check https://artilearn.app/dummy.html, our HTML file will successfully appear with the correct certificate!

Now our certificate has been successfully set up. But don’t forget this certificate is manually set up, and we haven’t configured any automatic renewal. But how should we do it? Well, we can go to the first sentence in this article: Being lazy and using certbot. We can do this now because our website can now be accessed from the HTTPS URL, which HTTP-01 challenge approves this method. Now just go to the certbot site and follow the simple step there!

Hello, I’m Adyaksa, and I write about software development and my language learning experience. I’m planning to release a weekly blog about something that I find interesting while working on my side projects. If you’re interested, you can follow me to keep updated about it!

This article is crossposted from my Medium account. You can follow it to get my latest article earlier.

Automate Your Flask Deployment to VPS Using Github Actions

Adyaksa W — Mon, 03 Jan 2022 02:46:39 +0000

(Cover Image by Joshua Sortino on Unsplash)

In the project lnprice that I have mentioned in my previous article, the backend deployment is done manually like this:

Push code to Github
Pull code in VPS
Redeploy application

After dozens of deployments, I have enough. I decide to learn how to automate this. For some context, I already have experience in using Github Actions to automate my deployment. The reason that I’m not doing it in this project is that I have never done it in a Flask environment. I just want to prioritize the application first before any automation. So in this article, I will describe the step that I have done to deploy my backend on the VPS automatically. All the code can be accessed in my Github repository

1. Setup Pipenv & Dependency

To have an easier time automating deployment, we need to have a simple dependency management system. So we need to install Pipenv in our project to handle our dependency and then install our dependency. We can do it like this:

pip install pipenv
pipenv install flask
pipenv shell

By running pipenv shell , we initiate a virtual environment in this project that has the Flask framework.

2. Create Flask Application

The next step is creating a simple Flask application. We can use the sample on the Flask homepage. But in my case, I changed it a bit to use a different port so we don’t accidentally use an already used port.

Now run flask run and go to localhost:5001 , there should be a ‘Hello World’ on that page.

3. Create Docker File

In my opinion, the easiest and painless way to deploy our application is to use Docker. We can definitely not use it and still be able to automate our deployment. But because there is so much possible pain that could appear when not using Docker, so I would really suggest using it.

Now, before touching Docker, we need to talk about how to deploy Flask to production. Like Django, Flask needs a WSGI server to deploy the application. There are 2 common WSGI server: UWSGI and Gunicorn. I won’t go into detail what is the difference between them. This article will use Gunicorn as the WSGI server.
GitHub - tiangolo/meinheld-gunicorn-flask-docker: Docker image with Meinheld and Gunicorn for Flask…

Now, someone already created a docker image for the Gunicorn-Flask combination. But in the context of this tutorial, that image needs modification so we are better off creating it from scratch. Here is the created Dockerfile.

Why do we use requirements.txt instead of using Pipfile directly? Here, I just want to treat Pipenv as some ‘dependency organizer’. I don’t want to use Pipenv as a virtual environment tool. You should be able to use the dependency from Pipfile directly and use Pipenv to access the dependency with some modification to the code.

After creating the Dockerfile, we can build and run the Dockerfile by running the following commands:

docker build -t 1-flask-deployment .
docker run -dp 5001:5001 1-flask-deployment

Try to run docker container ls , the 1-flask-deployment container should be up and running and can be accessed from localhost:5001 .

4. Setup Github Runner

Before creating the Github Actions file, we need to configure our own Github Runner. We need to place the Runner on the same server as the server where we want to place our production code. For this step, we can just follow the steps that are outlined in the official Github Runner Documentation

5. Create Github Actions

Time to pay off the hard part of creating the Dockerfile! Because we already know the step to run the container, now we just need to repeat that step in the Github Actions.

In this code, I configured the Github Actions so it will run only when the code is pushed to master (Line 3–6) and running in my own Virtual Machine (Line 10).

After pushing the deploy.yml, try to call curl localhost:5001 from inside the VPS. The ‘Hello World’ text should appear like when we call it in our local system. This means that the Github Runner is working, and the deployment automation is working!

References:

Pipenv Basic: https://pipenv.pypa.io/en/latest/basics/
Flask Quick Start: https://flask.palletsprojects.com/en/2.0.x/quickstart/
Pipenv docker: https://pythonspeed.com/articles/pipenv-docker/
Gunicorn Flask: https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-ubuntu-18-04

Why You Should Use VueJS

Adyaksa W — Mon, 27 Dec 2021 06:04:14 +0000

In the current framework trend for frontend, there is 3 mainstream that we commonly know: React, Vue, and Angular. In my recent projects, when there is a need to write frontend applications, I always used Vue. I just love using Vue.

From VueJS Official Site

Why? Well, first of all, I'm not hardcore enough to learn many things just for a simple project. I want simplicity. So for this reason, I excluded Angular. Now it comes down to React and Vue. Here comes my second reason: I love Vue syntax.

First of all, the file structure is quite simple yet separated beautifully. If you never touched Vue before, here is a snippet of basic Vue syntax

**<template>**
  <h1>Hello {{name}}</h1>
**</template>
<script>**
export default {
  data() {
    return {
      name: 'Adyaksa',
    }
  }
}
**</script>
<style>**
h1 {
  color: red;
}
**</style>**

So Vue file structure is divided into 3 sections: template, script, and style. The combination will form a Vue Component. The template is where the HTML structure is described. All heavy lifting is placed in the script section, where we can put all normal frontend scripts here in addition to Vue-specific scripts such as component lifecycle. And then the last section is where we put our CSS for the code.

One thing that I have experienced when using React is that when your team doesn't have a clear formatting guideline, it's harder to find specific code that you need. Moreover, when you have many components with their own specific styling, you will have an enormous number of files that you have. But when we're using Vue, all the HTML, CSS, and JS are combined in 1 class with a specific order that is already defined. Because of this, we know where each section is located in the file and we have an easier time finding what we need. This is also described in Vue docs: "What About Separation of Concerns?"

And then the second one is what makes creating HTML in Vue fun: Directives. Imagine that you want to create a list based on value from array arrayList . You can easily do it by adding v-for directives like this:

<li **v-for="item in arrayList"**> {{ item }} </li>

Hey, what's so fun about it? Well, imagine that you want to create something more complex such as displaying the ranking of an item with its attributes. By using this, we can just add v-for directives to easily access all the attributes of the item. And there are many more neat directives such as v-if, v-show, v-model etc.

But it's not all fun and game. Like all languages, VueJS readability would suffer at a more complex project. Its code structure also doesn't help, with every bit of code stuffed in a file. But still, I think this is a small price to use this fun language.

Hello, I'm Adyaksa, and I write about software development and my language learning experience. I'm planning to release a weekly blog about something that I find interesting while working on my side projects. If you're interested, you can follow me to keep updated about it!

What I Learn From Trying Selenium on VPS

Adyaksa W — Mon, 20 Dec 2021 08:39:51 +0000

Currently, I'm exploring various ideas for side projects. One of them needs some automated scraping that runs as a CRON job. Because I'm used to scrapping with Python using Selenium, I immediately choose them and connect them with Flask. There is 1 little problem: My previous scraping experience is only on a local machine. So I need to learn how to do scraping on the server and add a CRON job on top of that. Because it's quite a common use case, there should be many tutorials about it and make it easier when I need to troubleshoot, right?

Oh god, how wrong I was.

Okay, so before I started building this, I already researched a bit about 3 common libraries for scraping in Python with their advantage and disadvantage. To recap it a bit from what I learned:

Selenium: Can handle javascript-heavy websites, pass more scrapper detection than other alternatives, but so much slower than other libraries.
BeautifulSoup + requests: Simplest solution (I used requests on my previous project, so I only need to learn how to parse HTML using BeautifulSoup, which is quite easy too).
Scrapy: So much functionality and faster than Selenium, but quite inflexible than other solutions.

Because some websites that I need to scrape have some scrapper detection and are quite javascript-heavy, I don't see the need to use another library. I also don't need to update this CRON job too often, so slow performance doesn't matter to me.

In total, it took me 1 day to finish the prototype for this side project. I tested the scrapper locally and it works. It's slow, but expected. Satisfied with the result, I try to start the scraping on the VPS.

It broke. Okay no worry, I can fix this.

The error keeps changing, and it's getting more obscure.

I spent a few days after that to debug it, and my final solution is "Restart and hope for the best".

By Luca Bravo from Unsplash

There is so much hardship that comes with scraping on the VPS.

First, VPS doesn't have GUI, so we need to configure the server or the Selenium so it can still run without GUI. I found 2 solutions regarding this, using virtual display or running Selenium with a headlessbrowser.

I tried the virtual display approach, but the error that came up is still the same. When I changed to the headless solution, the error changes, so I choose this solution.

Second, the program frequently crashes.

To handle this, I used WebDriverWait so the driver can wait until the browser finishes loading. I also added various try except in risky lines that have a high chance to create an error.

Lastly, the driver frequently disconnects.

This is the hardest problem for me. What I found out is that ChromeDriver is very unstable compared to GeckoDriver. It successfully reduces the disconnection, but still not zero. So I need to run the scrapper in batch and restart every time the driver disconnected.

As a result, I changed my approach. I used the second scraping method (BeautifulSoup + requests) for website that can handle it. For the rest of the website, I used Selenium and wait patiently.
From this project, I learned that scheduled scraping on VPS is hard and time-consuming. Moreover, the problem that I encountered doesn't include when the target website blocks your scraping attempt. To handle this, there is a need to configure my scrapper so it doesn't look like it came from a scraping program. But because I'm just doing this side project for fun, I just change a few basic configurations. If in the future I need to have another scraping on VPS, I probably will invest a bit on a few proxies to change the requester's IP or even use web scraping services such as ScraperAPI.

But if not really needed, I don't want to do scraping on VPS again.

If you're curious about the side project that I'm building, you can check it out on this page. This is a simple price aggregation website for light novels. I'm planning to release a weekly blog about something that I find interesting while working on my side projects, and this article is one of them.