DEV Community: Niklas Tiede

How to Cheaply Host Your Full-Stack Side-Project: Serving it From Home

Niklas Tiede — Sat, 27 May 2023 20:01:51 +0000

Frontend projects can be hosted for free. But when you're creating a full-stack app your provider needs to process requests, making database calls etc. Hosting providers offer this hardly.

So when no one offers this for free what else can we do? Build it ourselves!

Content

Hardware
Docker
Dynamic DNS
Port Forwarding
My Home Server Setup

1. Hardware

To start, you'll need a computer that can act as a server. You don't necessarily need a powerful machine for this, especially if your app isn't too resource-intensive. An old desktop or a Raspberry Pi can work great for small-scale projects.

Make sure your server has a reliable internet connection. It should ideally be wired directly into your router to maximize uptime and minimize latency. Furthermore, you need a docker compatible OS. You can install it via a bootable USB stick.

2. Docker

Next, we're going to use Docker, an open-source platform that simplifies deploying applications within software containers. Containers package up the code, configs, and dependencies into isolated units, making the app easy to run on any platform that supports Docker.

To containerize our application we build a docker image using a Dockerfile.

FROM openjdk:17-jdk-alpine
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

We deploy the container with a docker run command or using docker-compose.

3. Dynamic DNS

To make your server accessible from the internet, we deal with the public IP address that your ISP assigns, which changes regularly. The solution is Dynamic DNS (DDNS) - a service that keeps a DNS record up-to-date with your changing IP.

There are several DDNS providers to choose from. After selection, configure it by installing their software on your server to regularly check your public IP and inform the DDNS service if it has changed.

4. Port Forwarding

With a DDNS hostname pointing to your public IP, you'd want your app to be accessible via a user-friendly domain name. Purchase a domain, and set up a CNAME record pointing to your DDNS hostname.

The next step is port forwarding, a router feature that directs incoming traffic on specific ports to your server. For instance, if your app runs on port 8080, set up port forwarding to direct traffic coming in on this port to your server.

You've now got a home server hosting your full-stack app. Remember, this is a simplified guide - consider security, data safety, and home internet reliability when hosting critical applications at home.

5. My Home Server Setup

To wrap things up, I want to share a real-life implementation of everything we've covered in this article. I have set up a home server (UM-560 Mini PC) running Ubuntu Server, and Docker, and deployed my IMDb Clone project that you can access here.

Here's a quick overview of the setup:

UM560 Home Server: This is a powerful, compact mini PC, which makes it an excellent choice for a home server. It runs Ubuntu Server, a robust, scalable, and secure platform for deploying applications.

IMDb Clone: This is the application that's been hosted on the server. It's a full-fledged web application that gives users access to a vast database of movie information, just like the real IMDb. For more information visit github.com/niklastiede/imdb-clone

Docker: I also tried docker-swarm but due to poor compatibility with ElasticSearch I stuck with Docker-Compose
DynDNS: My domain provider NameCheap provides also dynamic DNS, I use ddclient as a DynDNS client on the server
Port Forwarding: I have a FritzBox Router on which I opened port 443

Hosting this app on my home server was a journey full of learning and excitement. It required a fair amount of configuration and tweaking, but the end result was well worth the effort.

Remember, this didn't happen overnight. It required a good understanding of the technologies involved, a fair bit of troubleshooting, and a lot of patience. But once everything was up and running, the sense of accomplishment was immense.

I hope my story inspires you and reassures you that setting up your own home server for your side projects is entirely doable. Just take it one step at a time, keep learning, and most importantly, enjoy the process. 😃

Here is the Java Backend to my IMDb Clone!

Niklas Tiede — Sun, 23 Oct 2022 19:29:09 +0000

Heya fellows!

As someone who graduated in chemistry and had never any contact with software development in an professional environment I had to grasp all knowledge I could get from open-source projects on Github to become a professional developer.

And for me it was not that easy to find projects from which I could learn. People often suggest building a movie database app but I was not able to find a good real-world example on Github. So here I present a nice IMDb Clone!

What does this project differentiate from other real-world example apps? It comes with the original IMDb dataset of ~9 million movies shipped!

What Functionality does this Backend provide?
How to Rebuild this Project
Testing some Endpoints
What's the Data Model under the Hood?
Topics to learn from this Project

1. What Functionality does this Backend provide?

user can search for movies from the original IMDb dataset (with Elasticsearch as Search Engine and images saved with MinIO!)
user registration with email confirmation and JWT security
user can rate movies (CRUD by MySQL)
user can add movies to their watchlist
user can comment movies and have conversations

2. How to Rebuild this Project

The best way to learn is to get our hands on. Firstly, we just have to download and build the repository:

git clone https://github.com/NiklasTiede/imdb-clone.git

Before we can run the application we have to pull and run the docker image of the data base. This saves us the extra effort of adding credentials to config files, creating database schemes, importing the dataset and indexing the table.

cd infrastructure/deployment/development
docker-compose up -d

Be aware: the docker containers need 5-20 min before you can connect to it, because the MySQL container has to import the data from the .csv file first! You can also remove the -d (detach) flag to see when the container is ready. Then we can load our gradle project, let it download dependencies if not already done and run it!

3. Testing some Endpoints

I added .http-files which make it very simple to execute requests against the endpoints! Here's an example if we want to search for a movie (substring search).

We get a paged reponse back.

{
  "content": [
    {
      "id": 2872718,
      "primaryTitle": "Nightcrawler",
      "originalTitle": "Nightcrawler",
      "startYear": 2014,
      "endYear": null,
      "runtimeMinutes": 117,
      "movieGenre": [
        "THRILLER",
        "DRAMA",
        "CRIME"
      ],
      "movieType": "MOVIE",
      "imdbRating": 7.8,
      "imdbRatingCount": 528339,
      "adult": false,
      "rating": null,
      "ratingCount": 0
    }
  ],
  "page": 0,
  "size": 1,
  "totalElements": 30,
  "totalPages": 30,
  "last": false
}

If we want to access secured endpoints we have to create an account. The first account created will automatically have Admin role permissions.

Email confirmation is not necessary, it is turned off by default. But you can activate it by setting the following property from false to true.

spring.mail.properties.mail.smtp.starttls.enable=false

Now we can login.

A Json Web Token will be returned which we then use to authenticate if necessary.

{
  "accessToken": "eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiIxIiwiaWF0IjoxNjY2NTQxNTEyLCJleHAiOjE2NjkxMzm1MTJ9.HKlIagNPjHH-GITb_hMzcS4bH0jSMZGrjolw_buneIpm7MIYiN-42nLMqkj7ulRYqXv4LFfWwsNEMGFIJim30w",
  "tokenType": "Bearer"
}

Now lets make a request to a secured endpoint. Only users can rate movies. So lets rate a movie.

The rating of the movie is stored in database.

You can go on and check out also some other endpoints!

4. What's the Data Model under the Hood?

Here's an UML diagram of the underlying data model.

The two most important tables of the data model are the account- and movie-table. The watchlist and rating tables are using a composite primary key: each user can rate or put a movie an his watchlist only once. For the email confirmation and the password reset we need a verification token which is persisted in the verification_token table.

5. Topics to Learn from this Project

I learned a lot by writing this project. Here are some interesting topics you can find in this project.

handling Exceptions with @ControllerAdvice
mapping with Mapstruct
validation with javax constraints
multiple Enums: Bitvalue vs. extra Table
entities based on Composite Primary Key
scheduling Jobs
spring boot Profiles
custom Validation Annotations
Entity-Relationship Models
importing Data (csv-file) into MySQL
configure CORS Policy
column Indexing

...and many more! Feel free to fork/clone the repository and discover it!

I put a lot of effort into this project and will continue working on it. I would really appreciate if you give my Github repository a star! 😃

I wish you all a nice day!

Publishing Python Apps at Anaconda and further Automation

Niklas Tiede — Sun, 09 May 2021 17:37:16 +0000

Heya fellows,

The code of this post can be found on Github (see here).

When a setup.py file was already created, it's pretty simple to go a step further and create an Anaconda package. Anaconda packages store the necessary metadata within a meta.yaml file but when a setup.py file was already created its data can be easily imported by jinja2 templating.

{% set data = load_setup_py_data() %}

package:
  name: 'tihttp'
  version: {{ data['version'] }}

source:
  path: ..

build:
  number: 0
  entry_points:
    {% for entry_point in data['entry_points']['console_scripts'] %}
    - {{ entry_point }}
    {% endfor %}
  script: python -m pip install --no-deps --ignore-installed .

requirements:
  build:
    - python
    - pip
    - setuptools

  run:
    - python
    {% for dep in data['install_requires'] %}
    - {{ dep.lower() }}
    {% endfor %}

test:
  imports:
    - {{ data['name'] }}
  source_files:
    - tests
  requires:
    {% for test_dep in data['extras_require']['dev'] %}
    - {{ test_dep.lower() }}
    {% endfor %}
  commands:
    - pytest tests

about:
  home: {{ data['url'] }}
  license: {{ data['license'] }}
  summary: {{ data['description'] }}
  doc_source_url: {{ data['url'] + '/blob/master/README.md' }}

Now we can build the package.

$ conda build .

The generated file tihttp-0.1.0-py37_0.tar.bz2 can be found within the anaconda3/conda-build/linux-64 directory. To upload the package to the anaconda repository we have to register and then we can use the anaconda upload command. The package was uploaded to your channel and is now ready for distribution! 😙

$ anaconda upload home/niklas/anaconda3/conda-build/linux-64/tihttp-0.1.0-py37_0.tar.bz2

$ conda install -c niklastiede tihttp

But building and uploading packages for different Python interpreter versions and different operating systems is tedious. This work can be automated by a bash script:

#!/bin/bash

export PATH=~/anaconda3/bin:$PATH
pkg='tihttp'
array=( 3.6 3.7 3.8 3.9 )


# delete old built packages
if [[ -d $HOME/conda-bld/ ]]; then
    rm -r $HOME/conda-bld/
fi
for i in $HOME/anaconda3/conda-bld/linux-64/$pkg*; do
    echo $i
    rm $i
done
echo "Deleting old conda packages done!"


# building conda packages
for i in "${array[@]}"
do
    echo $i
    conda build --py $i .
done
echo "Building conda packages done!"


# converting conda packages to other platforms
platforms=( osx-64 linux-32 linux-64 win-32 win-64 )
for file in $HOME/anaconda3/conda-bld/linux-64/$pkg*; do
    echo $file
    conda convert --platform all $file  -o $HOME/conda-bld/
    for platform in "${platforms[@]}"
    do
        conda convert --platform $platform $file  -o $HOME/conda-bld/
    done
done
echo "converting packages to other platforms done!"


# uploading packages
find $HOME/conda-bld/**/$pkg*.tar.bz2 | while read file
do
    anaconda upload $file
done
echo "Uploading conda packages done!"

Instead of automating the build process locally we could instead use this nice github action Publish Conda! Here's the code snippet for the workflow:

name: publish_conda

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: publish-to-conda
      uses: fcakyon/conda-publish-action@v1.3
      with:
        subdir: 'conda'
        anacondatoken: ${{ secrets.ANACONDA_TOKEN }}
        platforms: 'win osx linux'

The world of anaconda does a great service to the data science community and I hope that this post decreases the barrier for publishing things at anaconda. 😄 I wish you all a great time!

How to Release and Publish Python Apps at PyPI

Niklas Tiede — Sun, 09 May 2021 11:33:49 +0000

Heya fellows,

The code of this post can be found on Github (see here).

As stated in previous posts our project can now be downloaded from Github and installed via pip locally. But Python packages are usually installed from the Python Packaging Index (PyPI) with a simple name easy to remember.

So how do we release our project so others can download it conveniently? We make sure that our main line is in a releasable state and then we

Tag our project and stick thereby to a versioning scheme
Release it on Github
Release it on PyPI

Git Tags and Release at Github
Publishing at PyPI
Automated Release with Github Actions

1. Git Tags and Release at Github

Depending on the kind/size of a project developers use Git branching workflows when working as a team but for smaller projects it's sufficient to work on the main line and create only branches when introducing features that might affect the stability of the project. This way we keep the main line always in a releasable state.

Then we can then make a new release as we gathered enough fixes/features. Developers pick either semantic versioning or calendar versioning as versioning scheme. We use the former and call our first release v0.1.0. We point to the commit we want to release by tagging it.

We can use lightweight or annotated tags which contain extra metadata. I like to use lightweight tags. The git tag command tags by default the last commit. We can also tag older commits when specifying the commit ID (example: git tag v1.1.0 c52e686). In this case we will tag the last commit and push it into the remote repo.

$ git tag v0.1.0                # git tag <tagname>
$ git tag                       # returns a list of all tags
$ git push origin v0.1.0

Now we browse our way to the Github repository and create a release. We go to Releases 🠲 Create a new release, and type the tag we wanna release. We can add binaries if we release projects which use compiled languages or write release notes. I like to document the changes of the release in a CHANGELOG.md file.

At last, we publish the release. In the next section we will package our project and publish it at the Python Packaging Index (PyPI).

2. Publishing at PyPI

To publish a python project at PyPI we have to package it by generating a source distribution (.tar.gz) and a wheel distribution (.whl).

$ pip install wheel

$ python setup.py sdist bdist_wheel

The generated files can be found within the dist directory. Now we have to register at PyPI before we can upload our project. For testing purposes it's convenient to upload the project to TestPyPI first. So, let's do that.

$ pip install twine

$ twine upload --repository testpypi dist/*

Now let's test installing tihttp.

$ pip install -i https://test.pypi.org/pypi/ --extra-index-url https://pypi.org/simple tihttp

Our tools works perfectly fine!

$ tihttp -H google.com

And now that we are sure that everything works, it's time to upload things to PyPI.

$ twine upload --repository pypi dist/*

$ pip install tihttp

Keep in mind that you can upload a specific semantic version just once to PyPI, thats why it's important to test uploading the version when just starting out the deployment process for a new project. Otherwise you have to bump the release version up and repeat the process (in that case the world will not end either).

3. Automated Release with Github Actions

Both processes (Github release and publishing at PyPI) can be automated using Github actions. The automation here is not that powerful for a project of this size but still I REALLY like Github actions and wanna let keep you in mind how nice it is! 😀

The instructions for the workflows are stored as yaml-file within the .github/workflows/ directory of the project. Tags can be automatically released using the github action gh-release.

name: GH Release

on:
  push:
    tags:
      - 'v*.*.*'

jobs:
  github-release:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - name: Release
      uses: softprops/action-gh-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

As you can see each major, minor or patch tag increment will create a new release.

At last I wanna show you a more powerful workflow for uploading releases automatically to PyPI.

name: Upload Python Package

on:
  release:
    types: [created]

jobs:
  pypi-release:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install setuptools wheel twine
    - name: Build and publish
      env:
        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
      run: |
        python setup.py sdist bdist_wheel
        twine upload dist/*

To store the credentials open your repository on Github and then go to settings 🠲 secrets 🠲new repository secret then type in the name (for instance PYPI_USERNAME) and its value.

Ok, I hope this post had some value to you. I wish you all a great day! 🤠

How to Write Documentation in the Python World!

Niklas Tiede — Wed, 05 May 2021 18:39:50 +0000

This is part 7 of the multi-part series "The Evolution of a Script". The code of this post can be found on Github (see here).

Clean and Self Descriptive Code
Comments, Docstrings and Git Commit Messages
README Files
Sphinx Documentation

In this post I wanna talk about a few dimensions of how to document a project in the python world. This makes it easier for the user or fellow developers to understand the project as a whole or separate parts of it. We will start at the lowest level and move toward higher abstraction levels.

1. Clean and Self Descriptive Code

The lowest level of documentation is the code itself. The goal is to create readable and reusable software. This can be achieved by adhering to a few principles:

Meaningful, pronounceable and consistent naming of variables/functions/classes.
Single-responsibility principle: Each function fulfills only 1 purpose, same applies for classes/modules on a higher abstraction level.
Use static typing for bigger projects, enforce type checking with mypy. Dynamic languages like Python make object identification difficult.
Avoid reinventing the wheel and make good use of Python's standard library. A good developer uses existing code strategically to his advantage.
Stick to a consistent style, I like Googles styleguide for Python.
Is the code elegant and pleasing? Listen to your intuition, your subconscious will point to the right things.

2. Docstrings, Comments and Git Commit Messages

>>> import math
>>> math.__doc__
'This module is always available.  It provides access to the'
'mathematical functions defined by the C standard.'

Docstrings are string literals which occur as first statement in a function, class, method or module definition. The docstrings become the __doc__ special attribute of that object. They're used to explain the general purpose of an object whereas comments explain smaller parts of the code. Comments are used to explain non-obvious portions of the code. Docstrings are surrounded by """triple quotes""" and divided into one-line or multi-line docstrings whereas comments starts with # at the beginning.

def my_function():
    """This is a docstring"""
    return None

# docstring of the function
my_function.__doc__

# displays documentation of the function
help(my_function)

There are many docstring formats available. Most commonly used are NumPy, PyDoc and Googles docstring style. It's a good idea to stick with a format which supports the Python documentation generator Sphinx. This generates a part of your documentation automatically from your docstrings. The last section of this post will show how to generate and host documentation with Sphinx and readthedocs.

A convenient VSCode extension is the Python Docstring Generator to facilitate the creation of docstrings.

def multiply(x: int, y: int) -> int:
    """[summary]

    Args:
        x (int): [description]
        y (int): [description]

    Returns:
        int: [description]
    """
    return x * y

It detects parameters automatically and you just have to fill out the marked fields. It uses the Google style by default.

Another source of documentation is available if the project has a Git history. A good git history gives you information about the reason for each code change. You can supercharge the git capabilities of VSCode by GitLens and you will see each commit message next to the code it was committed to.

Furthermore, I like to use gitmoji to make the reading of my commit messages visually more appealing and force myself to commit only code changes which fall into one category of the gitmojis.

3. README Files

A nicely written README is the first document the visitor of a project will see. Here's a screenshot of THELOUNGE, an IRC client for self-hosting:

What a beautiful README! What does it make so good?

It has a visually appealing and memorable logo which is compatible with Github's light and dark theme by using a transparent .png picture.
A concise self-description.
Badges from shields.io which visualize the quality of the project.
Links to documentation.
An example, here screenshot of the application when running.
A list of contained features.

Often README files also contain instructions for the installation or a tutorial on how to use it. The READMEs file format is .md (markdown) or .rst (reStructuredText). Sometimes it's also a good idea to provide examples to the user for certain, common use cases. Projects with a data scientific background tend to use jupyter notebooks (.ipynb) to demonstrate the capabilities of the project. Other projects use plain python files for demonstration purposes.

4. Sphinx Documentation

For smaller projects the README.md can be sufficient whereas projects like libraries benefit from a more extensive hosted technical documentation. I will show you here how simple it is to create your own freely hosted documentation with Sphinx, readthedocs and Github Pages!

OK, let's create some documentation! Sphinx is the tool that will help us to simplify this process.

$ pip install sphinx

$ mkdir docs
$ cd docs

Sphinx will ask us a couple of questions:

$ sphinx-quickstart

> Separate source and build directories (y/n) [n]: y
> Project name: tinyHTTPie
> Author name(s): Niklas Tiede
> Project release []: 0.1.0
> Project language [en]: enter

To create the documentation we have to use the make html command within the docs directory. This creates the HTML of our documentation.

$ cd ..
$ make html

If we open the index.html file with the browser via live server we can see how it will look like. But its appearance is pretty puristic yet. Therefore we use the popular RTD theme to give it a professional look. We install the theme...

$ pip install sphinx_rtd_theme

...and customize the conf.py file. We add the following lines:

import sphinx_rtd_theme

extensions = ["sphinx_rtd_theme",]
pygments_style = "sphinx"
version = '0.1.0'
html_theme = 'sphinx_rtd_theme'

And render again.

$ make html

Now it looks way better! Ok, next we wanna write some more content. The documentation should be written in the reStructuredText (.rst) syntax. Here's a nice cheat sheet. A .rst previewer in your IDE will speed up things. I've added some documentation about tinyHTTPie, see index.rst, install.rst and tutorial.rst. The last step is to publish our documentation. We have to register at readthedocs.org and let it hook our repository.

Voilà! A nice documentation was created! I hope you see how easy it is to setup such a good-looking documentation and that documentation has so many interesting aspects we're typically not aware of! 😃

Tools to automate Python Tests

Niklas Tiede — Tue, 04 May 2021 14:09:27 +0000

Heya fellows,

The code of this post can be found on Github (see here).

When projects grow a good test suite gives you confidence that new code you add don't cause parts of the application to break. It improves the maintainability of the project. The complexity of small projects is low and tests are only necessary when the size of the project increases. But for the sake of this tutorial we will write a small test to demonstrate the usage of pytest, tox and github actions.

We will store the tests within a separate folder. Here's the current structure of the project.

├── LICENSE
├── README.md
├── setup.py
├── tihttp.py
├── tests
│  ├── __init__.py
│  └── test_tihttp.py

We place a test_tihttp.py file within the tests folder. It will contain the test suite. The main() function which contains the logic for triggering the flags has to be imported from the tihttp.py file.

from tihttp import main

def test_GET_body(capsys):
    main(["-B", "http://jsonplaceholder.typicode.com/todos?userId=1"])
    captured = capsys.readouterr()
    result = captured.out
    with open("tests/jsonplaceholder.json", "r") as f:
        output = f.read()
    assert result == output

We compare the response of a GET request to an API with the expected JSON data. We place a jsonplaceholder.json file into the same folder which contains our expected output. Then we install pytest and let it execute the test.

$ pip install pytest
$ pytest

Our test passes. To increase the tests verbosity -v is a useful flag. Furthermore, I like to use the -s flag to see the captured output. Next we add pytest to our extra requirements in the setup.py file.

extras_require={
    'dev': [
        'pytest', # pip install tihttp[dev]
    ],
},

This gives us the possibility to install extra dependencies (testing, linting tools etc.) easily by adding a [dev] to the package name.

$ pip install .[dev]            # local install
$ pip install tihttp[dev]       # remote install, PyPI repo

We tested all of this with python 3.7.3. But how does our application behave when executed on a different interpreter version? So let's test it against different Python versions! We use tox. It let's us run tests in multiple virtual envs.

$ pip install tox

Tox needs a recipe to know which virtualenv/commands to create/execute. This recipe is named tox.ini.

[tox]
envlist = py36,py37,py38,py39

[testenv]
deps =
    pytest
commands =
    pytest

If some of the Python interpreters are missing on your system, install them from the deadsnakes archive:

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt install python3.5 python3.6 python3.7 python3.8 python3.9

Now let's test across different interpreters!

$ tox

If you wanna test against a specific environment or execute only one file, then type:

$ tox -e py38
$ tox -e py38 -- test/main_test.py   # executes only a single test

Ok, we did the test locally, but when working in a team using continuous integration is pretty convenient. We set a integrate.yaml file up within a .github/workflows directory to tell github actions what jobs to execute. The following github actions file will test across different platforms and Python versions.

name: Python package

on: [push]

jobs:
  run:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        python-version: [3.6, 3.7, 3.8, 3.9]
        os: [ubuntu-latest, macos-latest, windows-latest]
    steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Cache pip
        uses: actions/cache@v2
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-
            ${{ runner.os }}-
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install flake8 pytest pytest-cov
          pip install -r requirements.txt
      - name: Lint with flake8
        run: |
          # stop the build if there are Python syntax errors or undefined names
          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
          # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
      - name: Test with pytest
        run: |
          pytest

Don't be intimidated by the length of this job. It's just illustrating how powerful Github workflows can be. 🥰

Distribute a Python App via Setup.py File

Niklas Tiede — Tue, 04 May 2021 09:08:58 +0000

Heya fellows,

The code of this post can be found on Github (see here).

We have seen that it's tedious and unclear how to distribute a project with a bash script. So I'll show you in this post the way to go: using a setup.py file!

We have to create a setup.py file, put it into the root directory and then we can use pip to install our project locally. This installs the project into the active virtual environment. But how does a setup.py file looks like? Here's a template containing the most important key-value pairs:

import setuptools

setuptools.setup(
    name="tihttp",
    version="0.1.0",

    package_dir={"": "src"},
    packages=setuptools.find_packages(where="src"),

    install_requires=[
        "requests>=2.21",
    ],

    entry_points={
        "console_scripts":
        ["tihttp=tihttp.main:run_main"]
        },
)

The package_dir and packages values are necessary when a source layout is used, so that setup.py knows where to look for the packages. We used here a source layout to show these key-values and how they affect the entry_points section. The entry_points section tells pip where the command line tool is invoked from.

The command (in this case tihttp) is mapped to the starting point of the command line tool (here the run_main() function). Now tihttp can be installed and the command should return a proper help interface.

$ pip install .

$ tihttp --help

Another big advantage of this setup is that pip allows to install packages in editable mode:

$ pip install -e .

Your project is then installed into your currently activated virtual environment and every change to the code immediately affects this installed dependency. This allows to continue developing while testing the package comfortably. Furthermore, the setup.py file allows to download and install your project directly from github.

$ pip install https://github.com/NiklasTiede/tinyHTTPie/archive/5-Distributing-by-Setup-File.zip

Direct installations from github can take time especially for projects which are bigger (large git history) and use non-Python languages. Therefore PyPI and Anaconda are usually the preferred places to go for installing packages.

I want to note that many developers in the python community store project metadata and settings used for linting/testing within a setup.cfg file instead of a setup.py file but this is a topic for another post.

Distribute a Python App via Bash Script

Niklas Tiede — Mon, 03 May 2021 08:09:59 +0000

Heya fellows,

This is part 4 of the multi-part series "The Evolution of a Script". The code of this post can be found on Github (see here).

We created a useful tool, but wouldn't it be nice if we could make it accessible to others? So let's share it! At first, we have to make a snapshot of our virtual environment.

$ pip freeze > requirements.txt

Thereafter we create a new Git repository and push it to Github!

$ echo "# tinyHTTPie" >> README.md
$ git init
$ git add .
$ git commit -m "first commit"
$ git branch -M main
$ git remote add origin https://github.com/NiklasTiede/tinyHTTPie.git
$ git push -u origin main

Now others will be able to download our project using the clone command:

$ git clone https://github.com/NiklasTiede/tinyHTTPie.git

Now they have to rebuild it by recreating the virtual environment, extracting the Python interpreter path, adding a shebang line to the script and an alias to .bashrc. Quite laborious. A bash script automates this process!

I store all my scripts within a MyScripts folder so the bash script will create it.

if [[ ! -d ~/MyScripts ]]; then
    mkdir ~/MyScripts
    folder1=~/MyScripts
    echo "${folder1} was generated."
fi
if [[ ! -d ~/MyScripts/tinyHTTPie ]]; then
    mkdir ~/MyScripts/tinyHTTPie
    folder2=~/MyScripts/tinyHTTPie
    echo "${folder2} was generated."
fi

Then the tihttp.py and requirements.txt will be copied.

PWD=$(pwd)
REQUIREMENTS="$PWD/requirements.txt"
SCRIPT="$PWD/tihttp.py"
TARGET=~/MyScripts/tinyHTTPie

cp "$REQUIREMENTS" "$TARGET"
cp "$SCRIPT" "$TARGET"
cd ~/MyScripts/tinyhttp

The Python interpreter version is checked.

PYVER=$(python --version)
CURRENT_PY_VERSION=${PYVER:7:11}
testvercomp ${CURRENT_PY_VERSION} 3.0.0 '>'
if [[ $(echo $CURRENT_PY_VERSION | grep '2.'*) ]]; then
    echo "first digit is 2, you're python version is too low"
elif [[ $(echo $CURRENT_PY_VERSION | grep '3.'*) ]]; then
    echo "first digit is 3, you're python versions fulfills the requirements!"
fi

pipenv will be installed if it is not yet installed.

if [[ ! $(pipenv) ]]; then
    echo "pipenv not existent, so installing via pip..."
    pip install pipenv
    echo "...pipenv was installed?"
fi

The virtual environment will be rebuilt.

pipenv install -r requirements.txt

The shebang line is added.

pybin=$(pipenv --py)
echo -e "#\!${pybin}\n\n$(cat tinyhttp.py.py)" > tinyhttp.py

The shebang is checked.

EXIS_HEADLINE=$(head -n 1 tinyhttp.py)
EXP_HEADLINE="#!${pybin}"
if [[ $EXIS_HEADLINE == $EXP_HEADLINE && ${#EXIS_HEADLINE} == ${#EXP_HEADLINE} ]]; then
    echo "shebang line was added correctly!"
elif [[ $EXIS_HEADLINE != $EXP_HEADLINE || ${#EXIS_HEADLINE} != ${#EXP_HEADLINE} ]]; then
    echo "shebang was NOT added correctly!"
fi

File permissions are given.

chmod +x ~/MyScripts/tiniHTTP/tiny_HTTPie_clone.py

The script adds the alias to the configuration file.

SHELL=$(printenv SHELL)
MYSHELL=${SHELL:5:10}
ALIASFILE=~/.bashrc

if [[ $MYSHELL  == "zsh" ]]; then
    echo "You're default shell is ZSH."
    if [[ -e ~/.aliases ]]; then
        ALIASFILE=~/.aliases
        echo "You're storing your files within the .aliases file."
    elif [[ -e ~/.zshrc ]]; then
        ALIASFILE=~/.zshrc
        echo "You're storing your files within the .zshrc file."
    fi
elif [[ $MYSHELL  == "bash" ]]; then
    echo "You're default shell is BASH."
    if [[ -e ~/.aliases ]]; then
        ALIASFILE=~/.aliases
        echo "You're storing your files within the .aliases file."
    elif [[ -e ~/.bashrc ]]; then
        ALIASFILE=~/.bashrc
        echo "You're storing your files within the .bashrc file."
    fi
fi

And thats it! Puh, pretty much work and it's still pretty prone to errors (adding the script to the PATH instead of using an alias would make things less error prone). But there's a better solution to distribute packages in a standardized way: writing a setup.py file!

Create a Python Command Line App with Argparse

Niklas Tiede — Mon, 03 May 2021 08:07:03 +0000

Heya fellows,

This is part 3 of the multi-part series "The Evolution of a Script". The code of this post can be found on Github (see here).

One of the big benefits of using argparse is that it generates a usage help (a --help flag) automatically. It's common practice to separate the events which are triggered by flags from the creation of the argumentparser. Here's a argparse template I often use.

import argparse

def main(argv=None):

    parser = build_parser()
    args = parser.parse_args(argv)

    if args.arg_name:
        pass

    return 0

def build_parser():
    parser = argparse.ArgumentParser()

    # positional/optional arguments:
    parser.add_argument()

    return parser

def run_main():
    try:
        sys.exit(main())
    except Exception as e:
        sys.stderr.write(e)
        sys.exit(1)

if __name__ == '__main__':
    run_main()

If you're wondering where the arguments from outside are passed into the script: the default value of the argv identifier is None. Argparse knows then internally that it has to use the arguments from outside.

When converting our script into an argparse tool we can throw all the logic away which informs the user about incorrect usage. All this is now done by argparse! We wrap argparse around our script:

#!/usr/bin/python

import argparse
import requests
import collections
import sys


def main(argv=None):
    parser = build_parser()
    args = parser.parse_args(argv)

    url = ''

    if args.URL:
        url = args.URL

    if ('http://' or 'https://' or 'http://www.' or 'https://www.') not in url:
        if url[:4] == 'www.':
            url = url[4:]
        url = 'http://' + url

    try:
        resp = requests.get(url)
    except requests.exceptions.RequestException as e:
        print(f'Response Failed.')
        return 1

    header = dict(collections.OrderedDict(resp.headers))
    body = resp.text

    for section in sorted(header.items()):
        print(f"{section[0]}: {section[1]}")
    print()
    print(body)
    return 0


def build_parser():
    parser = argparse.ArgumentParser(
        prog='tihttp',
        description='A tiny HTTP client for sending GET requests.'
    )

    parser.add_argument('URL',action='store',)

    return parser


def run_main():
    try:
        sys.exit(main())
    except Exception as e:
        sys.stderr.write(e)
        sys.exit(1)


if __name__ == '__main__':
    run_main()

Adding the -H, -B flags and the boolean logic returns a more sophisticated version of our script:

#!/usr/bin/python

import argparse
import requests
import collections
import sys


def main(argv=None):
    parser = build_parser()
    args = parser.parse_args(argv)

    url = ''

    if args.URL:
        url = args.URL

    if ('http://' or 'https://' or 'http://www.' or 'https://www.') not in url:
        if url[:4] == 'www.':
            url = url[4:]
        url = 'http://' + url

    try:
        resp = requests.get(url)
    except requests.exceptions.RequestException as e:
        print(f'Response Failed.')
        return 1

    header = dict(collections.OrderedDict(resp.headers))
    body = resp.text

    if args.body and not args.header:
        print(body)
        return 0

    if args.header and not args.body:
        for section in sorted(header.items()):
            print(f"{section[0]}: {section[1]}")
        return 0

    if (args.header and args.body) or (not args.header and not args.body):
        for section in sorted(header.items()):
            print(f"{section[0]}: {section[1]}")
        print()
        print(body)
        return 0

    return 1


def build_parser():
    parser = argparse.ArgumentParser(
        prog='tihttp',
        description=f'A tiny HTTP client for sending GET requests.'
    )

    # positional arguments:
    parser.add_argument(
    'URL',
    action='store',
    )

    # optional arguments:
    parser.add_argument(
    '-H',
    '--header-only',
    dest='header',
    action='store_true',
    help='Prints only the header of the Response.')

    parser.add_argument(
    '-B',
    '--body-only',
    dest='body',
    action='store_true',
    help='Prints only the body of the Response.')
    return parser


def run_main():
    try:
        sys.exit(main())
    except Exception as e:
        sys.stderr.write(e)
        sys.exit(1)


if __name__ == '__main__':
    run_main()

Adding the help flag returns now a nicely formatted usage help!

$ tihttp --help

usage: tihttp [-h] [-H] [-B] URL

A tiny HTTP client for sending GET and POST requests.

positional arguments:
  URL

optional arguments:
  -h, --help         show this help message and exit
  -H, --header-only  Prints only the header of the Response.
  -B, --body-only    Prints only the body of the Response.

Create a Python Command Line App with Sys Module

Niklas Tiede — Mon, 03 May 2021 08:04:04 +0000

Heya fellows,

This is part 2 of the multi-part series "The Evolution of a Script". The code of this post can be found on Github (see here).

The sys modules sys.argv is a list that gives us access to all command line arguments passed to the Python script. This gives us the ability to make our script more flexible. We can now change the URL we would like to request by passing an argument to the script. The first element of sys.argv (sys.argv[0]) is the scripts name.

#!/usr/bin/python

import requests
import collections
import sys

url = sys.argv[1]

r = requests.get(url)

header = dict(collections.OrderedDict(resp.headers))
body = resp.text

for section in sorted(header.items()):
    print(f"{section[0]}: {section[1]}")

To test our changes we type:

$ tihttp https://the-coding-lab.com/

Success! Our tools feels now way more like a real command line app!

Ok, next let's make our tool more user friendly by improving error handling. We're used to enter a URL without schema, but every browser uses the HTTP method as its default scheme. So this is what we would like to implement for our tool.

#!/usr/bin/python

import requests
import collections

url = sys.argv[1]

if ('http://' or 'https://' or 'http://www.' or 'https://www.') not in url:
    if url[:4] == 'www.':
        url = url[4:]
    url = 'http://' + url

resp = requests.get(url)

header = dict(collections.OrderedDict(resp.headers))
body = resp.text

for section in sorted(header.items()):
    print(f"{section[0]}: {section[1]}")

Ok. But even if the URL was typed correctly several errors when communicating with the server can happen. The status code (3 digit number) returned will give us more information about why the error occurred. The first digit represents the class. The responses are grouped in 5 classes.

Status Code	Description	Class
200	Success	Successful Responses
301	Moved Permanently	Redirects
302	Moved Temporarily	Redirects
304	Not modified	Redirects
400	Bad request	Client Error
401	Unauthorized	Client Error
403	Forbidden	Client Error
404	Not found	Client Error
500	Internal Server Error	Server Error

These errors are handled using Python's exceptions.

#!/usr/bin/python

import requests
import collections
import sys


input_url = sys.argv[1]

if 'http://www.' and 'https://www.' not in input_url:
    if input_url[:4] == 'www.':
        input_url = input_url[4:]
    input_url = 'http://www.' + input_url

try:
    r = requests.get(url)
except requests.exceptions.RequestException as e:
    print(f'Response Failed.')

header = dict(collections.OrderedDict(resp.headers))
body = resp.text

for section in sorted(header.items()):
    print(f"{section[0]}: {section[1]}")

So good, so far. But what if a user of the tool types too many arguments or no URL at all accidentally? We have to advise our script to handle these use cases! Boolean logic solves this.

#!/usr/bin/python

import requests
import collections
import sys

arg_array = sys.argv[1:]
input_url = ''
body_bool, header_bool = False, False

if len(arg_array) > 1:
    print('Too many arguments.')
    sys.exit(0)

if len(arg_array) == 1:
    input_url = arg_array[0]

if not input_url:
    print('No URL was given.')
    sys.exit(0)

if 'http://www.' and 'https://www.' not in input_url:
    if input_url[:4] == 'www.':
        input_url = input_url[4:]
    input_url = 'http://www.' + input_url

try:
    r = requests.get(input_url)
except requests.exceptions.RequestException as e:
    print(f'Response Failed.')
header = dict(collections.OrderedDict(resp.headers))
body = resp.text

for section in sorted(header.items()):
    print(f"{section[0]}: {section[1]}")

Now let's add some options. It would be nice if the user could decide if we he wants to print only the header or the body of the response. Usually command line tools use flags to give options. We will use a -H flag to display only the header and -B flag to display only the body.

#!/usr/bin/python

import requests
import collections
import sys

arg_array, input_url = sys.argv[1:], ''
body_bool, header_bool = False, False

if '-b' in arg_array:
    arg_array.remove('-b')
    body_bool = True

if '-h' in arg_array:
    arg_array.remove('-h')
    header_bool = True

if len(arg_array) > 1:
    print('Too many arguments')
    sys.exit(0)

if len(arg_array) == 1:
    input_url = arg_array[0]

if not input_url:
    print('No URL was given')
    sys.exit(0)

if 'http://www.' and 'https://www.' not in input_url:
    if input_url[:4] == 'www.':
        input_url = input_url[4:]
    input_url = 'http://www.' + input_url

try:
    r = requests.get(input_url)
except requests.exceptions.RequestException as e:
    print(f'Response Failed.')
header = dict(collections.OrderedDict(resp.headers))
body = resp.text

if body_bool and not header_bool:
    print(body)

if header_bool and not body_bool:
    for section in sorted(header.items()):
        print(f"{section[0]}: {section[1]}")

if (body_bool and header_bool) or (not body_bool and not header_bool):
    for section in sorted(header.items()):
        print(f"{section[0]}: {section[1]}")
    print()
    print(body)

Some testing proves that we can use now two options:

$ tihttp -H https://the-coding-lab.com/
$ tihttp -B https://the-coding-lab.com/

We see that it can become quite tedious to add more functionality when limiting ourself to thesys module. It needs alot of boolean logic. But Python has a library only for creating command line interfaces: argparse!

Aspects of Writing Solid Python Applications

Niklas Tiede — Wed, 28 Apr 2021 09:03:28 +0000

Heya fellows,

as developers we're always lacking time during the day, so we have to make smart decisions and use our time efficiently when turning ideas to code. In general it's a good strategy to build a minimal viable product at first to avoid wasting time in writing unnecessary features.

But nevertheless even a minimal viable product should match some standards to attract users and other developers joining the project. In this series I will shed some light on several aspects of writing solid Python applications. We will create a little HTTP client and thereby we will:

Turn a script into a cli app with sys/argparse
Compare shell script installation vs. setup.py file
Write tests with pytest, tox and github actions
Use sphinx to create documentation
Publish our app at PyPI and Anaconda

1. Writing a Simple Script

First we will create a simple script which can perform GET requests. I keep all my scripts within a MyScripts folder. Then I create a project folder, and a virtual environment using pipenv.

$ cd ~/MyScripts
$ mkdir tinyHTTPie
$ cd tinyHTTPie

$ pipenv --python 3.7
$ pipenv shell

Then we have to create our project file and install dependencies:

$ touch tihttp.py
$ pip install requests

Let's write our script. We want to print the header of the response. I used the OrderedDict object imported from the collections library to sort the keys of the dictionary.

import requests
import collections

resp = requests.get('https://the-coding-lab.com/')

header = dict(collections.OrderedDict(resp.headers))
body = resp.text

for section in sorted(header.items()):
    print(f"{section[0]}: {section[1]}")

When we execute the script...

$ python tihttp.py

...the metadata of the request are returned successfully.

Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 124
Cache-Control: max-age=600
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 5764
Content-Type: text/html; charset=utf-8
Date: Tue, 16 Feb 2021 14:39:59 GMT
...

But when you realize that a script of yours is pretty useful, you might think "How could I turn this script into an easy-to-use command?". At first, you have to add a shebang line to stick the scripts virtual environment to it permanently. The which python command or pipenv --py will return the path to the Python interpreter of the virtual environment. It will look like:

/home/niklas/.local/share/virtualenvs/tinyHTTPie-iqhOkNUA/bin/python ~/tinyHTTPie/tihttp.py

Now this path has to be added to the head of the script as a shebang line. We can do this from the command line.

$ pybin=$(pipenv --py)
$ echo -e "#\!${pybin}\n\n$(cat tihttp.py)" > tihttp.py

The head of the file should look like this:

#!/home/niklas/.local/share/virtualenvs/tiny-HTTPie-iqhOkNUA/bin/python

import requests
...

To make the script executable we have to give permissions. Then we can just type the path to the script and it will be interpreted by the specified Python interpreter.

$ chmod +x tiny_HTTPie_clone.py
$ ./tihttp.py

Giving the path an alias will make it easier for us to remember/use our tool. To allow us to use the alias in every new shell session, we add it to the .bashrc file (or wherever you're storing your aliases).

$ echo "\nalias tihttp='~/MyScripts/tiny_HTTPie_clone.py'" >> ~/.bashrc
$ source ~/.bashrc

Now you can call your script in each new shell session no matter where you are on the filesystem.

$ tihttp

But there is still a problem: every time our tool sends a GET request to a website we have to change the URL by opening the file, changing the URL, saving and executing it. Would it not be nicer to add the URL as argument to our tihttp command? Python's sys module from the standard library comes here to our rescue!

Managing Environments with Conda and pip

Niklas Tiede — Tue, 27 Apr 2021 21:37:40 +0000

Heya fellows,

When dealing with dependencies the Python world offers several tools to manage these. It's important to deal with Python as well as non-Python packages, to handle different Python versions and to ensure a good isolation. Tools like pipenv and poetry are pretty popular for these purposes but conda has proven to be the tool of choice within the scientific community over the years. 💪

Conda im combination with pip is what I like to use on a daily basis. Conda handles non-Python code especially well and therefore libraries like RDKit (cheminformatics library) are only available in the conda ecosystem. It's a great resource for data scientific work.

Here I wanna show you some basic commands of conda and pip I'm using to manage my dependencies.

At first you have to install Anaconda on your system if it's not already installed. In the following we will install pandas, matplotlib and jupyter lab so we have a nice environment for data scientific work.

Creating Environments and Installing Packages

At first we create our new environment:

$ conda create -n my-new-environment python=3.8

Here we name the environment my-new-environment and use Python version 3.8. Conda will collect package metadata and create our new environment. By activating our new environment...

$ conda activate my-new-environment

...we leave condas base environment and enter our newly created environment. Then we can install the dependencies we need.

$ conda install matplotlib pandas

Jupyter lab is offered via the conda-forge channel, a popular, well-maintained channel, not via the default channel. That's why we have to provide the channels name:

$ conda install -c conda-forge jupyterlab

Or we add the channel right away to condas configuration so that the conda-forge repo is searched on top of the default channel when installing packages. Conda-forge is a pretty popular and well-maintained channel and therefore we will add it here permanently.

$ conda config --add channels conda-forge

You can see that conda-forge was added to your ~./condarc file.

ssl_verify: true
channels:
  - conda-forge
  - defaults

Some of condas commands are similar those of pip. To list the packages within your current environment we type:

$ conda list

If you want to search through all anaconda repositories for a package, then use this command:

$ anaconda search <package_name>

Managing Multiple Environments

But how do we deal with multiple environments? Let's clone the environment we created in the first place:

$ conda create --clone my-new-environment --name my-clone

When typing conda env list we see the two environments we created are next to the base environment.

$ conda env list

# conda environments:
#
base                   /home/niklas/anaconda3
my-new-environment     /home/niklas/anaconda3/envs/my-new-environment
my-clone               /home/niklas/anaconda3/envs/my-clone

We can switch the environment by just activating the new environment.

$ conda activate my-clone

To check which environment is currently active we can type conda info. BTW: Frameworks like Oh-My-Zsh show the active environment within the terminal which makes life easier.

Ok, now lets clean up things and delete the cloned environment.

$ conda deactivate 
$ conda env remove --name my-clone

Conda caches all installed packages within the 'pkgs' directory, so from time to time I clean it up to free up space.

$ conda clean -h

Sharing Environments

Developers are no one-man armies. We share our work to other developers/engineers and they should be able to reproduce our work. Conda is able to export the environments dependencies as a yaml file and recreate the environment from this file.

$ conda env export --name my-new-environment > environment.yaml

Now all dependencies can be found in the environment.yaml file. This file can be used by someone else to recreate the environment. When no --file flag value is provided conda searches for a environment.yaml file in the current working directory and tries to recreate the environment from this file.

$ conda env create              
$ conda env create --file environment-dev.yaml

But don't feel forced to only use conda packages when managing dependencies with conda! I also use conda when managing projects where I install only dependencies from pypi.org. Just as reminder here are the commands for storing/recreating pip dependencies.

$ pip freeze > requirements.txt   
$ pip install -r requirements.txt

Conda makes switching between different environments so easy and you have always access to conda packages. Unfortunately, condas speed for creating environments and installing conda packages is not outstanding but nonetheless it's good enough that I prefer it over other dependancy manager 🙂. If you want to see how to upload a project to Anaconda, take a look into my article 'Publishing at Anaconda'.

Ok that's it, I hope you I could arouse your curiosity for Anaconda. Thanks for your attention and have a nice day! 🙂

All conda commands I'm using can be found in my Linux Cheat Sheet.

DEV Community: Niklas Tiede

How to Cheaply Host Your Full-Stack Side-Project: Serving it From Home

Content

1. Hardware

2. Docker

3. Dynamic DNS

4. Port Forwarding

5. My Home Server Setup

Here is the Java Backend to my IMDb Clone!

Contents

1. What Functionality does this Backend provide?

2. How to Rebuild this Project

3. Testing some Endpoints

4. What's the Data Model under the Hood?

5. Topics to Learn from this Project

Publishing Python Apps at Anaconda and further Automation

How to Release and Publish Python Apps at PyPI

Contents

1. Git Tags and Release at Github

2. Publishing at PyPI

3. Automated Release with Github Actions

How to Write Documentation in the Python World!

Contents

1. Clean and Self Descriptive Code

2. Docstrings, Comments and Git Commit Messages

3. README Files

4. Sphinx Documentation

Tools to automate Python Tests

Distribute a Python App via Setup.py File

Distribute a Python App via Bash Script

Create a Python Command Line App with Argparse

Create a Python Command Line App with Sys Module

Aspects of Writing Solid Python Applications

1. Writing a Simple Script

Managing Environments with Conda and pip

Creating Environments and Installing Packages

Managing Multiple Environments

Sharing Environments