DEV Community: Daniel Kang

Taking a peak at Django - Part 1

Daniel Kang — Thu, 18 Jun 2020 02:47:03 +0000

Many people might think that python is not for the web, but that's actually not the case. We can use Python to make a backend server with the help of Django and Flask. Today, we are going to look over the basic installation process of Django and a quick guide on writing your first Django app.

For this project, I will use goormIDE and upload my container link down below so anyone could check out the code and see how it works.

I'm going to set up my project using goormIDE's python stack (which already has python and Django installed), but for those who are using another IDE, feel free to see my previous tutorial on how to set up a Python development environment: https://dev.to/daniel2231/a-python-ide-that-you-can-use-anywhere-26cg

You can also directly use the Django stack on the right side of the Python stack, and skip this part of the tutorial.

For Django, all you have to do (if you have Python installed) is to type the following commands on your terminal.

$ python -m pip install Django

If you want to check whether if you already have Django installed, use the following commands.

python -m django --version

Let's begin!

Setting up

Assuming you already have Django installed, lets begin with making a new Django project. Type this on your terminal.

django-admin startproject myblog

You can change example to anything, preferably your project name. For this tutorial, I'm going to name my project 'myblog'.

This will create an 'myblog' project directory that will contain the necessary files you are going to need. The default project directory will look like this.

myblog/
    manage.py
    myblog/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Let's go over the files.

myblog/ is your root folder for your project. Django doesn't care about this file. You can rename this to whatever you want.

manage.py will help you create applications, start your development web server, and help you work with databases.

/myblog/ ,that is inside your project folder will be your actual Python package for your project. You will need to use its name when trying to import anything inside it (e.g. Instagram.urls).

__init__.py will instruct Python to treat this directory as a Python package.

settings.py will contain your website's information. This is where you will store the location of our static files, databse configuration settings, and register any application that you create.

urls.py is where you store the URL declarations for this Django project.

asgi.py is an entry point for ASGI-compatible web servers to server your project.

wsgi.py is used to help your Django application communicate with the webserver.

Now, let's try running your server. Before we run anything, we must set up goormIDE so that we can run our project in a custom url goorm provides us with.

On your settings.py, edit this part:

ALLOWED_HOSTS = []

to this:

ALLOWED_HOSTS = ['*']

Go to Project > Running URL and Port and copy the url that is provided. If the port is not on 80, change the port to 80.

Run the following code in your terminal.

python manage.py runserver 0:80

Now try accessing the custom goorm link. If successful, You will see this screen:

Part 2 Coming soon...

How to crawl a website in just 5 minutes

Daniel Kang — Mon, 15 Jun 2020 09:09:25 +0000

Have you ever had that time when you want to just get the information that you want from a website automatically, without going inside the website? What we will make today is a crawler bot that will crawl, or get information for you automatically from any website that you want. This might be useful for those who wants to get updated information from the website, or get data from a website without all those copy and paste hasle.

What we are going to use

In this tutorial, we are going to use Scrapy, an open source python framework for extracting data from websites. For more information on Scrapy, I recommend visiting their official website. https://scrapy.org/

For our development environment, I am going to use goormIDE so that I can share my container so that others can see. I'll share the container link with the completed code down below. You are free to use any development environment that you wish.

Scrapy in 5 minutes

Setting up the container

For those who wish to use goormIDE, you can check out my previous guide on how to set up a Python development environment in goormIDE here: https://dev.to/daniel2231/a-python-ide-that-you-can-use-anywhere-26cg

Installing Scrapy

Now, lets install scrapy. Write the following command in the command line.

pip install scrapy

Making a new spider

After you are done installing scrapy, we are going to use scrapy to make a scrapy project. On your parent directory (or wherever you want to put your project), run the following command.

scrapy startproject scraper

You can change scraper to any name that you want.

The default project directory will look like this:

scraper/
   scrapy.cfg             # Deploy configuration file
   scraper/               # Project's Python module
      __init__.py         
      items.py            # Project items definition file
      middlewares.py      # Project middewares file
      pipelines.py        # Project pipelines file
      settings.py         # Project settings file
      spiders/            # A directory for your spiders
         __init__.py

Now, lets get inside our spiders folder.

cd scraper/spiders/

Making a new spider

A Spider is a set of classes which contains instructions on how a certain site or sites will be scraped.
"Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site (or, in some cases, a group of sites)." (source: https://docs.scrapy.org/en/latest/topics/spiders.html)

We will now make a new spider that will crawl this site: https://www.tiobe.com/tiobe-index/
What I want to get from this site is the ranking of each programming language and its ratings.

Note: Before you crawl a site, make sure to check their policy on crawling. Some sites might have restrictions on what you can crawl or not. For more clear instructions, check a site's robots.txt file (You can check for them like this: www.sitename.com/robots.txt) Check this article out for more information: https://www.cloudflare.com/learning/bots/what-is-robots.txt/

What we will make is a crawler that will get the rank, name of programming language, and ratings from the website.

To make a new spider, enter the following commands:

scrapy genspider crawlbot www.tiobe.com/tiobe-index

To break up, scrapy genspider creates a new spider for us, crawlbot is the name of the spider, and the link will tell the spider where we will crawl the information.

Finding out what to crawl

Inside your spider folder, you will see that a new spider has been created.

What we must do now is to find out how we are going to tell the scraper what must be crawled.

Go to the website: https://www.tiobe.com/tiobe-index/

Right click and select inspect.

Here, we can see that the information that we need is stored inside a table.

When we open the table body tag, we can see that the information is stored inside each table row, as a form of table data.

There are two basic ways to tell our crawler what to crawl. The first way is to tell the crawler what class attribute the tag has. The second way is to get the Xpath of the tag that we would like to crawl. We are going to use the second method.

To get a tag Xpath, simply right click on the tag (on the developer console) and select Copy, Copy Xpath.

When I copy the Xpath of the rating, I get:

//*[@id="top20"]/tbody/tr[1]/td[1]

Now, remove the number behind tr and add /text() at the end of the Xpath.

//*[@id="top20"]/tbody/tr/td[1]/text()

What I just did tells the crawler that I will get the text of every first table data of every table row.

Now, lets get the Xpath of the ratings, programming language, and rank.

Rank: //*[@id="top20"]/tbody/tr/td[1]/text()
Programming language: //*[@id="top20"]/tbody/tr/td[4]/text()
Ratings: //*[@id="top20"]/tbody/tr/td[5]/text()

Now, open your crawlbot spider and add the following codes:

# -*- coding: utf-8 -*-
import scrapy


class CrawlbotSpider(scrapy.Spider):
    name = 'crawlbot'
    allowed_domains = ['https://www.tiobe.com/tiobe-index/']
    start_urls = ['https://www.tiobe.com/tiobe-index/']

    def parse(self, response):
        rank = response.xpath('//*[@id="top20"]/tbody/tr/td[1]/text()').extract()
        name = response.xpath('//*[@id="top20"]/tbody/tr/td[4]/text()').extract()
        ratings = response.xpath('//*[@id="top20"]/tbody/tr/td[5]/text()').extract()

        for item in zip(rank, name, ratings):
            scraped_info = {
                'Rank': item[0].strip(),
                'Name': item[1].strip(),
                'Ratings': item[2].strip()
            }
            yield scraped_info

Let's break this code up. First, we are telling the crawler where to crawl by providing a url link. Then, we make a variable where we will store the crawled data. Don't forget to put .extract() behind!
Lastly, we are returning each crawled data by using yield scraped_info.

We are now done!
Let's try running the crawler.

scrapy crawl crawlbot

We can see that the crawler gets all the data that we need from the website. However, it is a bit inconvenient to see the data in our terminal. Let's tell our crawler to export the data in JSON format.

Export to JSON

Go to settings.py on your project directory and add this two lines of code:

FEED_FORMAT = "json"
FEED_URI = "rank.json"

Now, run the crawler again and you will see that all the data you crawled will show up in rank.json.

That's it! I'll leave my container link down below for those who wants to see the code.

Container link: https://goor.me/KqvR6

That's all folks! Happy coding!

How to upgrade your Python version

Daniel Kang — Tue, 09 Jun 2020 03:10:22 +0000

On my latest post, I introduced a way to make a python IDE that you can access from anywhere. On this post, I will go over on how we can install the latest python 3.8.3 on our IDE.

For those who haven't read the post before this, I recommend you to check them out: https://dev.to/daniel2231/a-python-ide-that-you-can-use-anywhere-26cg

In this tutorial I will use an IDE called goormIDE, but you can follow this tutorial on any IDE that you wish. However, you must be using a linux based terminal if you want to follow this tutorial.

Personally, I would recommend using apt to install python as it is faster and less cumbersome.

Installing the latest Python from Source

First, update your package lists and install the packages required.

sudo apt update
sudo apt install libffi-dev libsqlite3-dev wget libbz2-dev build-essential libssl-dev libreadline-dev zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev

You will be prompt to install the packages. Press y to continue.

Now, download the python version you want. In this tutorial, we are going to install python3.8.3

wget https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz

Extract the downloaded package using the command below.

tar -xf Python-3.8.3.tgz

Now, after you are done extracting the installation file, switch to the python source directory and run the following command.

Move to python source directory

cd Python-3.8.3

Then run command:

./configure --enable-optimizations

After that, lets start the python build by using the following command.

make -j 8

Now, install the python binaries.

sudo make altinstall

Once that's finished, check if your python is installed correctly.

python3.8 --version

Yup, its correctly installed. 👍

Installing the latest Python using Apt

This
First of, run the following commands to update your package list and install the prerequisites.

sudo apt update
sudo apt install software-properties-common

Next, add deadsnakes PPA to your system's resource list. PPAs are Personal Packages Archives that lets users upload Ubuntu source packages to be built and published as apt repositories.

sudo add-apt-repository ppa:deadsnakes/ppa

You will get prompted to press Enter. Press Enter to continue.

Press [ENTER] to continue or Ctrl-c to cancel adding it.

After you are done, install the python you desire. In this tutorial, we will be installing python 3.8

sudo apt install python3.8

After installation, verify if python is installed correctly.

python3.8 --version

Presto! We got it!

A python IDE that you can use anywhere

Daniel Kang — Wed, 03 Jun 2020 10:17:55 +0000

As a student who travels a lot (pub - campus - cafe - home), I tend to carry around my macbook air because its light (and I hate heavy things). However, macbook air isn't really a powerful machine and it often drives me mad when the code I'm trying to run is too heavy to run.

I've been trying a lot of things to solve this problem, but recently I found that an online IDE might help run my codes from my laptop.

So I did some quick research and found some products that might help.

glitch
goormIDE
Repl.it
Visual Studio Online
paiza.io

Now, all these IDEs were ok, but the one that felt easy to use and setup was goormIDE. So today, I would like to write a short tutorial on how to setup a python development environment in goormIDE.

Registration

First, we need to sign up for a free account.
Link to goormIDE: https://ide.goorm.io/

Now click Sign Up, and complete the signup process. (It doesn't take long)

After registration, you will be redirected to your dashboard.

Setup

Now, lets try making a new container where we will install everything we need to work with python. Click the create new container button (as this is a free tier account, you can only make 5 containers)

Write your container name, select a region, and select the python stack. Now, if we see the bottom of the page, we can see that with a single click, all the things that you will need for python development is installed.

Now click the create button on the top right of the page. After a bit of time, a popup will prompt you if you would like to run your container. Click run container.

And presto! You got your own online python IDE! Now, lets check if we got all the stuff we need installed.

I'm going to use tensorflow, so I will only check if python and tensorflow is installed.
Lets first try checking if python is correctly installed.

python --version

Great!

Now lets try tensorflow.

python3 -c 'import tensorflow as tf; print(tf.__version__)'

Good. Everything is in place.

Now, you can connect to your container from anywhere, which is great for me as I don't have to buy a new laptop.

That's all for now, I will post further tutorials as I get used to the new IDE I will be using for a while.