DEV Community: Can Kahraman

Flask configuration

Can Kahraman — Tue, 25 Jul 2023 15:27:47 +0000

Configuration hell

Sometimes it's the trivial parts that confuses me. They are being portrayed as "don't spend too much time on it". However a structured configuration might really save you some headaches.

After realizing that I had a few harmless config differences on my prod env, being accumulated from different sources like docker-compose, or dockerfile and env vars, I have been meaning to change my flask app's config structure for some time now and I finally found time to do it because I started to assume things. And it's dangerous if you are starting to assume things. Having a single source of truth for your config variables is important to me. Otherwise you get lost in the config sea, trying to understand which config variable gets overwritten/ignored by which file.

Before this refactor I have been using python-dotenv and a python file called config.py with multiple separated classes such as BaseConfig, DevConfig and ProdConfig etc. to manage different configurations for different environments. At first I knew what I was doing, but at some point, it started to get complicated and I began to mix things up and what belonged where.

How to choose the right config file format?

There are several configuration file formats out there, such as JSON, YAML, TOML, ini or .env files and similar. As you can see we don't have the perfect solution, and it looks like different library developers have different ideas regarding how the perfect configuration file should look like.

For one thing, JSON files are missing comments, which sounded like a deal-breaker for me initially. TOML files become overly verbose when it comes to too much nested data. Don't know about YAML, maybe it's the choice for you. There are different opinions on each one, search for the one that suits you.

Personally the decisions I'm taking ultimately come down to how simple the tool is. Start with the simple and try out more complicated ones as you need them. Therefore I went with JSON, even though it doesn't allow comments. That's fine. If I really crave for comments, maybe I can think of another solution then. But as I said, stick to your simple solution unless you absolutely can't do without.

Previously on "Configuration Hell"

I had the following structure when it came to my configuration. I'm using the factory pattern in flask to make the process of creating an app with different configurations easier. This method also simplifies unit testing immensely.

# app/__init__.py

from flask import Flask

def create_app(config_type: str = "DevConfig") -> Flask:
    app = Flask(__name__)

    # Load the configuration.
    app.config.from_object(f"config.{config_type}")
    ...

    return app

# config.py

class Config:
    """
    Common settings for all environments.
    """
    STATIC_FOLDER = pathlib.Path(__file__).parent / "app" / "static"
    UPLOAD_FOLDER = STATIC_FOLDER / "uploads"
    PDF_FOLDER = STATIC_FOLDER / "pdfs"
    THUMBNAILS_FOLDER = STATIC_FOLDER / "thumbnails"
    MAX_CONTENT_LENGTH = 16 * 1000 * 1000
    TESTING = False
    ...

class DevConfig(Config):
    SQLALCHEMY_DATABASE_URI = os.environ.get("SQLALCHEMY_DATABASE_URI", "mysql+pymysql://user:pwd@mariadb:3306/mydatabase")
    ...

class ProdConfig(Config):
    SQLALCHEMY_DATABASE_URI = os.environ["SQLALCHEMY_DATABASE_URI"]
    ...

class TestConfig(Config):
    TESTING = True
    SQLALCHEMY_DATABASE_URI = os.environ.get("SQLALCHEMY_DATABASE_URI", "sqlite:///")
    ...

The Config class holds the common configuration variables that won't change from env to env. And the following classes respectively, hold configuration variables that are specific to their environments.

The confusion I'm having here is, when do I need to feed the env variables? When I'm developing on dev, there could be some sane defaults. When I'm deploying to prod, those sane defaults should definitely be ignored because they are specific to dev environment. I also want the application to yell at me when I'm on production and I haven't supplied the database connection string for instance. There mustn't be any default values there and it has to be supplied conciously. There are also several different config possibilities depending on whether I'm working with docker or not. I want to see each of them clearly and detect the differences with a single glance, instead of looking under docker-compose, or looking under aws task definition or several locations at the same time, trying to merge them in my head.

Refactor

In order to make things clear, I created an instance folder to hold all of my json configurations. This folder will include configurations for all the environments, that is dev.json, prod.json, docker-dev.json, docker-prod.json, test.json and finally an example file called config.json, which should behave as a template for all the other files.

We will commit the config.json file to the version control system because it doesn't contain any sensitive information in it, just some fake data. Additionally, I also committed the test.json into the VC because it doesn't contain secrets either in my case. I use it to run my unit/integration tests on github actions.

Now my instance folder looks like this:

And my factory function looks like this:

# app/__init__.py

def create_app(env: Optional[str] = "prod") -> Flask:
    """Create flask app with the given configuration."""
    app = Flask(__name__, instance_relative_config=True)

    # Load the configuration.
    app.config.from_object("config.Config")
    app.config.from_file(f"{env}.json", load=json.load)

Here is my config.py file. This file will be here to hold some logic for configuration. I don't want to hardcode file paths, I want them to be relative to the config.py in the repo.

# config.py

import pathlib


class Config:
    # Upload folder under static.
    STATIC_FOLDER = pathlib.Path(__file__).parent / "app" / "static"
    UPLOAD_FOLDER = STATIC_FOLDER / "uploads"
    PDF_FOLDER = STATIC_FOLDER / "pdfs"
    THUMBNAILS_FOLDER = STATIC_FOLDER / "thumbnails"
    MAX_CONTENT_LENGTH = 16 * 1000 * 1000

    # Enable this in order to echo all the sql commands to the db.
    # SQLALCHEMY_ECHO = True

This is the instance/config.json file that behaves as an example template for all other config files.

{
  "SECRET_KEY": "verysecret",
  "PERMANENT_SESSION_LIFETIME": 1800,
  "GOTENBERG_URL": "http://gotenberg:3000/forms/libreoffice/convert",
  "SQLALCHEMY_DATABASE_URI": "mysql+pymysql://user:pwd@localhost:3306/mydatabase",
  "ADOBE_PDF_EMBED_API_KEY": "",
  "ALLOWED_EXTENSIONS": [
    "txt",
    "pdf",
    "png",
    "jpg",
    "jpeg",
    "gif",
    "ppt",
    "csv",
    "html",
    "xls",
    "xlsx",
    "docx",
    "doc"
  ],
  "TESTING": false,
  "WTF_CSRF_ENABLED": true,
  "MAIL_USERNAME": "example@mail.com",
  "MAIL_PASSWORD": "password",
  "MAIL_SERVER": "smtp.server.com",
  "MAIL_PORT": 587,
  "MAIL_USE_TLS": true,
  "MAIL_USE_SSL": false,
  "MAIL_DEFAULT_SENDER": "example@mail.com",
  "SQLALCHEMY_ECHO": false
}

Gunicorn

By having a separate config file for each env, now I can see exactly what kind of differences I have between the environments. When I want to fire up an app that specifically should use a configuration file called docker-prod.json, that file should exist and it should contain all of the needed vars under it. This file is the single source of truth for that environment.

CMD gunicorn --bind 0.0.0.0:5000 "app:create_app(env='docker-prod')"

Some struggles with passing json config to the docker container

Since I'm building my container in the github actions, there's no way for me to include my docker-prod.json into the container directly. Therefore I have discovered a simple trick, that enables me to encode my config file completely into a single base64 string (which in turn saves me from including all of my env variables one-by-one), put it under github secrets and pull/decode it from there during the build process.

In the job that I'm currently building my container, I have the following logic:

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    environment: production

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Decode docker-prod.json
        run: echo -n "${{ secrets.DOCKER_PROD_JSON }}" | base64 --decode > $GITHUB_WORKSPACE/instance/docker-prod.json

        ...

What happens here? The checkout actions checks-out the source code and decodes the long base64 string from github secrets, putting it under instance/docker-prod.json. Eventually when the application starts, I have my config file right where it needs to be without checking it in under version control.

Conclusion

Alright, this is huge for me. Discovering the above trick allowed me to host my prod config free of charge on github secrets. I'm not sure how much it costs to store your secrets in AWS Secret Manager, but yeah...

Now just by looking at different config files I have in my instance folder, I instantly see what database connection string I'm using, what email credentials are there, what different api keys I'm making use of in different environments. I don't need to go through multiple config files and try to guess which one supersedes the other.

Notes to self

Maybe this is a naive approach, maybe not. Perhaps you will find some useful points here, perhaps you'll just shrug and say "meh, pretty basic stuff". I'm writing this article for self-document purposes, which should guide me to better practices along the way. What I learned, how I learned, how I applied, how it turned out in the end.

I'll stick to this for a while and see how it works out.

Visualize Git

Can Kahraman — Mon, 21 Jun 2021 16:23:40 +0000

Dangit Git

Yep, sometimes git can be complicated, but that does not mean necessarily it can not be understood unless you are a git expert. You all probably have used git at some point in your development stages and familiar with the basics. Well today I am here to show you the peculiar aspects of this tool and how we can deal with trouble.

Visualize stuff baby

Below representation will be our visual work space.

This is our repository and it has several commits and branches in it. First we have an initial commit (root of our tree) at the top, after which we have 3 commits (added contributors, improvements and refactor), whose parents are the same. As you see, we also have 3 branches, master, feature1, and feature2. We also have HEAD which we will talk about in a bit.
In order to make it more understandable, I will show you how visual representation changes while we execute the corresponding git commands. By doing so, it is going to allow us to notice the relationship between the commands and the bahavior of git.

What is HEAD and HEAD~ etc. ?

We will move HEAD from time to time in our visual graph, but what is this magical special word to begin with ? Let me explain, HEAD is nothing more than a pointer in your git workflow, it just marks where you are. That's it. Simply accept HEAD as an arrow pointing to your location on a map. Now in order to mark other locations, we might need a reference point right ? That is for instance, if I need to explain someone where the commit refactor is, I can just say go 2 commits above HEAD and there you will find it. Thus HEAD~1 is like saying "the commit above HEAD" and similarly HEAD~2 is equal to saying "the commit 2 steps above HEAD". By the way HEAD~ is the same as HEAD~1 so no fancy stuff there, we programmers are lazy you know.

In order to prove our point let us execute git checkout feature2.

HEAD moved to point to the feature2, huh well that was simple enough. Get back to master now, we will continue.

What is `--soft` and `--hard`

These are amongst the most common misunderstood concepts when starting to learn git. I will simply give an illustration to make it more comprehensible. Imagine you are building a tower of cards, like when you were a child and you have a vicious cat pondering around. Suddenly your mom summons you to dinner and you have to go. If you take your tower of cards with you, that is a soft choice. You don't want your cat to destroy your progress, therefore you take the tower (your changes) with you. On the contrary, if you decide to leave it as is, that vicious cat will definitely tear it down, that's a hard choice. This hard choice will indefinitely discard all the changes you have done, so use it carefully.

In summary:

git reset --soft HEAD~ - move back one commit and keep your changes
git reset --hard HEAD~ - move back one commit and discard all changes

1 - Modify commit ?

You might sometimes commit a work-in-progress and immediately realize you forgot to add something to the commit.

git add - add the missing/modified file(s) you forgot to add
git commit --amend --no-edit - alter the previous commit (remove --no-edit if you want to modify the commit message)

This will replace your commit with a fresh new one. Now let's see what changed in our visual graph. Oh look at that, only modification was to the hash of our commit. That means we changed the contents of this commit.

2 - Committed to master instead of a new branch ?

We were working casually and finished our work, only to commit to master instead of another branch. Ah, a silly mistake indeed. You can see our mistake in the graph below. I assure you it has a trivial fix.

git branch feature3 - create a new branch at your current position
git reset HEAD~ --hard - take back master to the previous commit
git checkout feature3 - switch to your new branch

Let's look at our graph. Okay seems like we now have a branch called feature3 and we successfully recovered our master to where it belongs. HEAD is pointing to the feature3 branch and that's what we expect from the last action we took above.

3 - Committed to the wrong branch !

You made a commit to a wrong branch ? This is a very common scenario and it has an easy fix, unless you have pushed your changes (please tell me you did not). If you haven't pushed your changes yet, hurrraaaay. If you have already pushed your changes, well we'll deal with that later. Let's clear the easier one first.

git reset HEAD~ --soft - take yourself back to the previous commit and keep your changes
git stash - stash your changes so we can switch branches

git checkout feature2 - switch to another branch
git stash pop - unpack your changes from stash

git add . - add your files and commit them normally
git commit -m "fixed"

4 - I want to revert this specific commit from 10 days before

One of those moments where you need to take back a change from daaaaays ago right ? Git has you covered.

git log - find the hash of that commit you want to revert
git revert <hash>

This action creates a revert commit, meaning it does NOT modify any of your previous commits and just creates a new commit with the exact opposite file modifications. This is cool. You can see the reverse commit below, a whole new commit.

5 - Oh dang, I did something terribly wrong, I lost my commit !

This one is nasty. It is best we visualize this one in order to understand what's coming step by step, because my friend, most of the beginners are afraid of GETTING DETACHED. Ok jokes aside, seriously when I first encountered a similar scenario where I had to go into detached state and did something wrong, I felt terribly lost, even though I was not. That's why we will spend extra effort to make it more transparent.

Imagine checking out a specific commit, in this case refactor.

$ git checkout c5b64
Note: checking out 'c5b647'.

You are in 'detached HEAD' state...

Ooookayyy, what does that even mean ? Keep it cool and beware, pretty much nothing is unrecoverable in git except that previously mentioned hard reset or maybe deleting your folder altogether and ending this agony. Detached HEAD state means you are pointing to a commit directly, instead of pointing to a branch. HUUUH and that's it ? YEAH ! EXPERIMENT TIME ! Since we are pointing to a lonely commit, let us change something and commit it and see what happens.

Seems like we are still in that evil detached HEAD state. Let's assume we took such an action while not knowing the bits and pieces of git and switched back to master via git checkout master. Clearly we now know, git checkout master in its most basic form is like saying "Go ahead and just point to master".

You see, we checked out some unknown commit somewhere in space, made some changes and committed it with the name of "some changes after refactor" and moved back to master, am I correct ? Now, attention please, since you don't know the hash of that "some changes after refactor" commit and it does not have an explicit branch attached to it, you have no way of knowing if it's there. Hence, you think after all your developments and improvements, you lost it all and it's time to start from ground zero. Stop right there because git reflog is coming for rescue.

We may inspect the output of this command:

$ git reflog
5b35f6d HEAD@{1}: pull ...
ca92d15 HEAD@{2}: ...
759dab1 HEAD@{3}: commit (merge): ...
065e269 HEAD@{4}: commit: ...
f357606 HEAD@{5}: commit: ...
9u7b45 HEAD@{6}: checkout: moving from master to 9u7b45d272867b63d54f96d4aa57f8ecc479cd0

That 9u7b45 HEAD@{6}: checkout: moving from master to 9u7b45d272867b63d54f96d4aa57f8ecc479cd0 should give you a pretty rough idea of what you have done, that is, you have moved to a branchless commit in the past. There it is now, your precious little commit that was drifting in space without any label on it. Now go there and put a label on it, or that is to say, create a branch, whatever.

I AM DONE WITH GIT !

If you are really done with this insane version control stuff that I have explained, here is a final solution :D

cd ..
sudo rm -r ducking-git-repo-dir
git clone https://some.github.url/ducking-git-repo-dir.git
cd ducking-git-repo-dir

Proceeds to copy the working folder to -> "/home/git_bender/last_last_final_version_of_git_tutorial/"

Inspired from Dangitgit.

Want to compress your pdf files ?

Can Kahraman — Tue, 25 May 2021 14:41:23 +0000

Are you tired of scanning images and trying to shrink them under 25 mbs just so you can send them via email ? Look no further, I am here to save you from this trouble. (that kinda rhymed.)

This basic shell script uses ghostscript to compress your scanned pdfs significantly. Just yesterday I scanned 100 pages of documents and it was over 90 mbs. I searched for a way to compress them under 25 mbs and voila. Here I was with only 6 mbs of pdfs. Much wow. Such compression.

Why we need it,

Fast.
Ez.
No need to go online. Especially no need to upload company top-secret documents to any website you see on the Internet.
Possible huge compression without notable loss.

Command usage:

./shrinkpdf.sh in.pdf out.pdf

Don't forget to install ghostscript.

If you are in a situation exactly like me where there are 100 pdf files under one folder that you want to compress altogether, a simple for loop will suffice.

import os
import subprocess
import sys

infolder = sys.argv[1]
outfolder = "compressed_" + infolder
os.mkdir(outfolder)

for f in os.listdir(infolder):
    subprocess.run(["./shrinkpdf.sh", os.path.join(infolder, f), os.path.join(outfolder, f)])

Run this python script,

python shrink.py pdfs/

and all of your pdfs will be put under compressed_pdfs/.

http://www.alfredklomp.com/programming/shrinkpdf/

All the relevant details and more, inside. I humbly wanted to let you know such a useful tool exists.

Some thoughts about productivity

Can Kahraman — Mon, 24 May 2021 22:25:58 +0000

For the last couple of years, I have been in the Linux space and mostly tried to find the best distro, best file manager, best terminal etc. I was caught up in the loop of finding the best programs to handle a job. This approach, unless controlled, usually leads to low productivity in the long term.

How so you might ask ? Well, this behavior is really relatable for some people. When I am talking about this subject, some of you might probably say, "Oh yes I have been there". In Linux world, there is even a term for this. Ricing.

Ricing means tweaking the configuration of your working environment to suit your personal needs. At particular level, one might consider this useful, and it indeed is. There are some workflow approaches, shortcuts and others that helps the developer access files and folders faster, learn by muscle memory and perform some set of actions quicker. Past that particular level and you are now in the over-ricing zone. Where you are essentially tweaking every aspect of each program, work environment to your needs, but at the same time you are never satisfied so you keep tweaking and changing settings, configs and what not. It reduces your productivity and at the end of the day when you look back, you realize you have done nothing but tweaking some stuff to find the best work environment.

This is what I call the loop (referring to the Bandersnatch here). You are in the loop. You need to get out and you need to get sh*t done. Don't worry about using the best tool out there, don't try to find the absolute efficient program. Just get it done.

This is in actuality a paradox. The paradox goes as such, "The more choices you have, the less satisfied you are". Seems about right when you think for a bit.