DEV Community: Serhat Teker

Automate Python Virtual Environment with a Script

Serhat Teker — Fri, 01 Apr 2022 00:00:00 +0000

Issue

There are 2 things that annoy me every time when I create a python virtual environment.

NOTE

Below instructions are related to the python's default venv module.

If you are using any other version/virtual environment management tools,
like Conda, Poetry, Pyenv, below doesn't apply.

The first one is that I have to type every time:

$ python -m venv ./.venv
$ . ./.venv/bin/activate
# or
# $ source ./.venv/bin/activate

The second thing is that after installing any package I'm getting below pip
version WARNING:

WARNING: You are using pip version 21.1.1; however, version 22.0.4 is available. You should consider upgrading via the '/path/to/project/.venv/bin/python3.8 -m pip install --upgrade pip' command.

It is an expected behavior, 'cause python -m venv calls python -m ensurepip to install pip.

And there is no way to globally upgrade since pip is installed with bundled version which is almost out of date. You can check it like below:

>>> import ensurepip
>>> ensurepip.version()
'21.1.1'

Even though you can't upgrade to the latest version with below commands as mentioned in ensurepip doc:

$ python -m ensurepip --upgrade

Because ensurepip will only install the bundled version even with the --upgrade option.

There is no official option to update the bundled pip and setuptools, yet.

Solution

Okay, since we are developers, we always find our ways and produce our solutions that suits our needs.

My solution comes from a shell script. You can put it in your shell configuration file, like .zshrc, .bashrc, config.fish.

With below function, when I enter ve command, it will create my virtual environment — if it doesn't exist, activate it and upgrade to the latest pip version.

# 0. If not already in virtualenv:
# 0.1. If virtualenv already exists activate it,
# 0.2. If not create it with global packages, update pip then activate it
# 1. If already in virtualenv: just give info
#
# Usage:
# $ ve
# or
# $ ve python3.9
# or
# $ ve python3.9 ./.venv-diff
ve() {
    local py=${1:-python3.8}
    local venv="${2:-./.venv}"

    local bin="${venv}/bin/activate"

    # If not already in virtualenv
    # $VIRTUAL_ENV is being set from $venv/bin/activate script
      if [ -z "${VIRTUAL_ENV}" ]; then
        if [ ! -d ${venv} ]; then
            echo "Creating and activating virtual environment ${venv}"
            ${py} -m venv ${venv} --system-site-package
            echo "export PYTHON=${py}" >> ${bin}    # overwrite ${python} on .zshenv
            source ${bin}
            echo "Upgrading pip"
            ${py} -m pip install --upgrade pip
        else
            echo "Virtual environment  ${venv} already exists, activating..."
            source ${bin}
        fi
    else
        echo "Already in a virtual environment!"
    fi
}

This function may seem complex, so let me walk through it line by line.

$1 is the first argument, the desired python version, $2 is the second argument which is virtual environment's name/directory, provided to the function ve(). If no arguments provided function will use default values: python3.8 and .venv. You can these default values according to your needs.

ve() {
    local py=${1:-python3.8}
    local venv="${2:-./.venv}"

    local bin="${venv}/bin/activate"
    ...
}

You can provide positional arguments like:

$ ve python3.9 .venv2

After creating local variables, if we are already in a virtual environment, it just gives "already in a virtual environment" message. It decides this whether we're in a virtual environment by checking if there is a environment variable called $VIRTUAL_ENV exported from activate — .venv/bin/activate, script.

ve() {
    ...

    if [ -z "${VIRTUAL_ENV}" ]; then
        ...
    else
        echo "Already in a virtual environment!"
    fi
}

If we're not in a virtual environment, it will control if .venv directory exists. If the directory exists, it will be activated:

ve() {
    ...

    if [ -z "${VIRTUAL_ENV}" ]; then
        if [ ! -d ${venv} ]; then
            ...
        else
            echo "Virtual environment  ${venv} already exists, activating..."
            source ${bin}
        fi

    else
        ...
    fi
}

However, if .venv directory does not exist, it will:

Create virtual environment .venv with global packages included (line 7),
Add exporting desired python version environment variable into activate script (line 8)
Activate the virtual environment (line 9)
Upgrade the current pip version to the latest one (line 11)

ve() {
    ...

    if [ -z "${VIRTUAL_ENV}" ]; then
        if [ ! -d ${venv} ]; then
            echo "Creating and activating virtual environment ${venv}"
            ${py} -m venv ${venv} --system-site-package
            echo "export PYTHON=${py}" >> ${bin}    # overwrite ${python} on .zshenv
            source ${bin}
            echo "Upgrading pip"
            ${py} -m pip install --upgrade pip
        else
            ...
        fi

    else
        ...
    fi
}

You probably don't need line 8, I'm adding this, because using this environment variable in my other aliases/functions/binaries.

Conclusion

In the end, with this shell script we solved our annoying bugs and make us more productive again.

You can find this script in this gist or in function from my dotfiles repo.

All done!

Changelog

2022-05-10 : Added gist and dotfiles link references

surepip

Django REST Filtering Tutorial - Filter with Current User (Part II)

Serhat Teker — Thu, 24 Mar 2022 00:00:00 +0000

Entrée

We've seen in part I how to setup our repo, sending HTTP requests and some fundamental concepts about Django REST filtering.

In this second part we will limit our views/querysets/objects with user, more precisely with current authenticated user.

The authors list endpoint — /authors/, displays all items regardless of whether they belong to current user or not.

$ http :8000/authors

[
    {
        "first_name": "Name1",
        "full_name": "Name1 Surname1",
        "id": 1,
        "last_name": "Surname1",
        "user": 1
    },
    {
        "first_name": "Name2",
        "full_name": "Name2 Surname2",
        "id": 2,
        "last_name": "Surname2",
        "user": null
    },
    {
        "first_name": "Name3",
        "full_name": "Name3 Surname3",
        "id": 3,
        "last_name": "Surname3",
        "user": 3
    }
]

How can we change that behavior in order to return items only owned by current authenticated user?

NOTE

You can find the code we're working on this tutorial series in this repository: django-rest-filtering-tutorial

You don't need to download and setup the repository in order to follow this series, however I'd like to encorauge to do so, since while you play you learn better as always.

Current User

This is one common filter usage which is filtering the queryset to ensure only relevant results to the current authenticated user who makes the request are returned.

For this purpose we will override default get_queryset as we've seen in default filtering in detail.

Let's filter authors list endpoint with the current user:

from rest_framework import generics

from src.authors.models import Author
from src.authors.serializers import AuthorSerializer


class AuthorListView(generics.ListAPIView):
    serializer_class = AuthorSerializer

    def get_queryset(self):
        """
        This view returns a list of all the authors for the currently
        authenticated user.

        Returns empyt list if user Anonymous
        """
        user = self.request.user

        if not user.is_anonymous:
            return Author.objects.filter(user=user)

        return Author.objects.none()

NOTE

In our repo we use ModelViewSet for authors endpoint. You can update it like above.

You can look at related AuthorSerializer and Author model in the repo, however for easy access I put them there as well. In order to examine them just expand the below section.

AuthorSerializer and Author model

AuthorSerializer from src/authors/serializers.py:

# src/authors/serializers.py
from rest_framework import serializers

from src.authors.models import Author


class AuthorSerializer(serializers.ModelSerializer):

    class Meta:
        model = Author
        fields = ("id", "user", "first_name", "last_name", "full_name")
        read_only_fields = ("full_name",)

Author model from src/authors/model.py:

# src/authors/model.py
from django.db import models
from django.conf import settings

USER = settings.AUTH_USER_MODEL


class Author(models.Model):
    """
    Author entity

    Provides first_name and last_name, since he/she can write unter a Pen Name
    """
    user = models.ForeignKey(to=USER, on_delete=models.DO_NOTHING, blank=True, null=True)
    first_name = models.CharField(max_length=255)
    last_name = models.CharField(max_length=255)

    def __str__(self):
        return f"{self.first_name} {self.last_name}"

    @property
    def full_name(self):
        return f"{self.first_name} {self.last_name}"

Now we can make a successful request by including the username and password of one of the users we created in populate_db.py:

$ http -a "User1:1234" ":8000/authors/"

[
    {
        "first_name": "Name1",
        "full_name": "Name1 Surname1",
        "id": 1,
        "last_name": "Surname1",
        "user": 1
    },
]

As you can recognize now it returns only one object instead of three.

And if you we make a request as anonymous or unauthenticated user, we will get:

$ http ":8000/authors/"

[]

User Authentication and Permission

It's usually better idea to separate authentication and permission layer/logic. For this goal you can use authentication_classes and permission_classes on per-view or per-viewset basis;

from rest_framework import generics
from rest_framework.authentication import SessionAuthentication, BasicAuthentication
from rest_framework.permissions import IsAuthenticated

from src.authors.models import Author
from src.authors.serializers import AuthorSerializer


class AuthorListView(generics.ListAPIView):
    serializer_class = AuthorSerializer
    authentication_classes = (SessionAuthentication, BasicAuthentication)
    permission_classes = (IsAuthenticated,)

    def get_queryset(self):
        """
        This view returns a list of all the authors for the currently
        authenticated user.
        """
        user = self.request.user

        return Author.objects.filter(user=user)

Another method is setting the default authentication globally, using the DEFAULT_AUTHENTICATION_CLASSES setting:

# settings.py
REST_FRAMEWORK = {
    "DEFAULT_AUTHENTICATION_CLASSES": [
        "rest_framework.authentication.BasicAuthentication",
        "rest_framework.authentication.SessionAuthentication",
    ]
}

INFO

In the repo these are disabled, you can set them globally if you comment out the related lines at the end of the setting.py file.

And now our view will look cleaner like this:

# src/authors/views.py
from rest_framework import generics

from src.authors.models import Author
from src.authors.serializers import AuthorSerializer


class AuthorListView(generics.ListAPIView):
    serializer_class = AuthorSerializer

    def get_queryset(self):
        """
        This view returns a list of all the authors for the currently
        authenticated user.
        """
        return Author.objects.filter(user=self.request.user)

For more info about authentication and permissions you can look at:

Custom FilterBackend

There is another method which is that we can also provide our own generic filtering backend.

To do so override BaseFilterBackend class, and override the .filter_queryset method. The method should return a new, filtered queryset.

As well as allowing clients to perform searches and filtering, generic filter backends can be useful to restrict which objects should be visible to any given request or user.

Let's write IsOwnerFilterBackend for our current user logic:

# src/authors/filters.py
from django_filters import rest_framework as filters


class IsOwnerFilterBackend(filters.BaseFilterBackend):
    """
    Filter that only allows users to get their own objects
    """
    def filter_queryset(self, request, queryset, view):
        return queryset.filter(owner=request.user)

Add this filter to our AuthorListView:

# src/authors/views.py
from rest_framework import generics
from django_filters import rest_framework as filters

from src.authors.models import Author
from src.authors.serializers import AuthorSerializer
from src.authors.filters import IsOwnerFilterBackend


class AuthorListView(generics.ListAPIView):
    serializer_class = AuthorSerializer
    queryset = Author.objects.all()
    filter_backends = (IsOwnerFilterBackend,)
    filterset_fields = ("user",)

Now if we send request we only receive related authors to the user.

One side note to this method; as we've already seen above, we can achieve the same behavior by overriding get_queryset() on the view, but using a filter backend allows us to more easily add this restriction to multiple views, or to apply it across the entire API.

We can also make this filter backend as our global default backend:

# settings.py
REST_FRAMEWORK = {
    "DEFAULT_FILTER_BACKENDS": ["src.authors.filters.IsOwnerFilterBackend"],
}

Conclusion

In this tutorial we learned how to filter the queryset to ensure only relevant results to the current authenticated user who makes the request are returned.

See you in the next part of this tutorial series.

All done!

Django REST Filtering Tutorial - Intro (Part I)

Serhat Teker — Thu, 17 Mar 2022 00:00:00 +0000

Entrée

In this tutorial I'd like to aim to collect some of Django REST filtering concepts and common examples that I use and teach usually. Some of them can be found in official Django REST documentation, however I'm going to walk by a sample repository.

The repo consists as less code as possible which implies it's not designed according to best practices such as api versioning (api/v1), The Twelve Factor App etc.

NOTE

You can find the code we're working on this tutorial series in this repository: django-rest-filtering-tutorial

You don't need to download and setup the repository in order to follow this series, however I'd like to encorauge to do so, since while you play you learn better as always.

Setup Repo

As mentioned above you don't need to setup the repo however if you like to, —I suggest as well, run below commands:

# Create virtual environment named .venv
$ python -m venv ./.venv
# Activate the virtual environment
$ source ./.venv/bin/activate
# Install python packages
$ python -m pip install -r requirements.txt
# Create db, make migrations and seed database with some dummy data
$ python populate_db.py
# Run development server
$ python manage.py runserver 8000
# or
# ./manage.py runserver 8000

INFO

You don't need to give 8000 port since it's 8default port but I always prefer verbosity.

Also another option is that you can use gnu make and simply run:

$ make start

This make target will run all commands above together so it will setup the repository and run development server.

Sending HTTP Requests

I'm going to use httpie for HTTP requests since it's more human-usable —or let's say more dev-friendly; it provides colorful syntax highlighting (like pretty JSON outputs), persistent sessions, capability of downloading files etc. and it's written in Python but you can use curl as well:

$ http GET http://localhost:8000/authors/
# or with less keystroke, it's same as above
$ http :8000/authors/
# curl equivalent
$ curl http://localhost:8000/authors/

For more info you can look at httpie doc.

Let's start with some basics and send http requests to endpoints.

In order to get all authors, run:

$ http :8000/authors/

HTTP/1.1 200 OK
Allow: GET, POST, HEAD, OPTIONS
Content-Length: 165
Content-Type: application/json
Date: Fri, 17 Mar 2022 11:03:37 GMT
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.8.11
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

[
    {
        "first_name": "Name1",
        "full_name": "Name1 Surname1",
        "id": 1,
        "last_name": "Surname1",
        "user": null
    },
    {
        "first_name": "Name2",
        "full_name": "Name2 Surname2",
        "id": 2,
        "last_name": "Surname2",
        "user": null
    },
    {
        "first_name": "Name2",
        "full_name": "Name2 Surname2",
        "id": 3,
        "last_name": "Surname2",
        "user": 3
    }
]

Mind the slash (/) suffix after authors; if you don't append it you will get:

$ http :8000/authors

HTTP/1.1 301 Moved Permanently
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Fri, 17 Mar 2022 11:05:35 GMT
Location: /authors/
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.8.11
X-Content-Type-Options: nosniff

You can change this behavior by adding --follow argument as a flag or as a config in httpie's config file which by default located in ~/.config/httpie/config.json. As a command flag:

$ http :8000/authors --follow

As a config:

// ~/.config/httpie/config.json
{
    "__meta__": {
        "about": "HTTPie configuration file",
        "help": "https://httpie.org/doc#config",
        "httpie": "1.0.3"
    },
    "default_options": [
        "--timeout=300",
        "follow"
    ]
}

Django controls this behavior by APPEND_SLASH setting:

APPEND_SLASH

Default: True

When set to True, if the request URL does not match any of the patterns in the URLconf and it doesn’t end in a slash, an HTTP redirect is issued to the same URL with a slash appended. Note that the redirect may cause any data submitted in a POST request to be lost.

The APPEND_SLASH setting is only used if CommonMiddleware is installed (see Middleware). See also PREPEND_WWW.

Also if you'd like to omit header part of the response provide --body flag.

Default Filtering

Default filter for Django REST is no filter, meaning that it returns all objects from the database.

In other words if you try to filter the endpoint with some parameters it will return all objects together just like it did above:

$ http ":8000/authors/?id=1"

[
    {
        "first_name": "Name1",
        "full_name": "Name1 Surname1",
        "id": 1,
        "last_name": "Surname1",
        "user": null
    },
    {
        "first_name": "Name2",
        "full_name": "Name2 Surname2",
        "id": 2,
        "last_name": "Surname2",
        "user": null
    },
    {
        "first_name": "Name2",
        "full_name": "Name2 Surname2",
        "id": 3,
        "last_name": "Surname2",
        "user": 3
    }
]

For better understanding let's look at the responsible get_queryset method from the source code:

# rest_framework/generics.py
def get_queryset(self):
    """
    Get the list of items for this view.
    This must be an iterable, and may be a queryset.
    Defaults to using `self.queryset`.

    This method should always be used rather than accessing `self.queryset`
    directly, as `self.queryset` gets evaluated only once, and those results
    are cached for all subsequent requests.

    You may want to override this if you need to provide different
    querysets depending on the incoming request.

    (Eg. return a list of items that is specific to the user)
    """
    assert self.queryset is not None, (
        "'%s' should either include a `queryset` attribute, "
        "or override the `get_queryset()` method."
        % self.__class__.__name__
    )

    queryset = self.queryset
    if isinstance(queryset, QuerySet):
        # Ensure queryset is re-evaluated on each request.
        queryset = queryset.all()
    return queryset

As you can see, it just returns queryset attribute from the related view which in our case is Article.objects.all():

# src/articles/view.py
from rest_framework import viewsets

from src.articles.models import Article
from src.articles.serializers import ArticleSerializer


class ArticleViewSet(viewsets.ModelViewSet):
    serializer_class = ArticleSerializer
    queryset = Article.objects.all()

So as an example if you change queryset attribute to Article.objects.filter(content__icontains="3"),

# src/articles/view.py
from rest_framework import viewsets

from src.articles.models import Article
from src.articles.serializers import ArticleSerializer


class ArticleViewSet(viewsets.ModelViewSet):
    serializer_class = ArticleSerializer
    queryset = Article.objects.filter(content__icontains="3")

it only will return Fake Article3 article:

$ http ":8000/authors/"
# or
$ http ":8000/authors/?id=1"

[
    {
        "author": 3,
        "content": "Fake Content3",
        "id": 3,
        "regions": [],
        "title": "Fake Article3"
    }
]

At the same time you may realize you can directly change the get_queryset method and throw the queryset attribute:

# src/articles/views.py
class ArticleViewSet(viewsets.ModelViewSet):
    serializer_class = ArticleSerializer

    def get_queryset(self):
        return Article.objects.filter(content__icontains="3")

Conclusion

That's all for this part of the series. In the next parts you will explore more in depth example and cases.

All done!

How to Install Deb Packages

Serhat Teker — Thu, 10 Mar 2022 00:00:00 +0000

.deb is the installation package format used by all Debian based distributions like Ubuntu, Kali , Linux Mint , Pop!_OS etc. and of course Debian itself.

In this post, I will explain how to install .deb files on Ubuntu. The same instructions apply for any other Debian based distribution as mentioned above.

The Ubuntu repositories contain thousands of deb packages that can be installed either from the Ubuntu Software Center or from the command line using the apt.

However many applications are not included in the Ubuntu or any 3rd party repositories. Another reason that you'd like to install a local deb package would be their version on default package manager could be old and you'd like to get new version of them.

Those unofficial applications can be downloaded from the developer’s websites or git repositories and installed manually. But be careful when installing .deb packages from unofficial sources especially if it's not open source.

Downloading Deb Package

For demonstration let's download Vscode IDE via wget:

$ wget -O 'vscode-stable.deb' 'https://code.visualstudio.com/sha/download?build=stable&os=linux-deb-x64'

As you may guess from query paramaters this command will download 64 bit architecture of Vscode deb package from official Vscode repository as named vscode-stable.deb in your current working directory.

Installing Deb Package

When it comes to installing deb packages there are command-line or graphical user interface options.

From the command line you have several tools at your disposal as well. In the following sections, I will show you how to use apt, and dpkg utilities to install deb packages.

dpkg

dpkg is a low-level package manager for Debian based systems. Use the -i or --install flag to install deb packages with dpkg.

# First update package indices
$ sudo apt update
# Install via dpkg
$ sudo dpkg -i ./vscode-stable.deb

Unlike apt, dpkg doesn’t resolve dependencies. If you get any dependency errors when installing deb packages, you can use the following apt command to resolve and install all package dependencies:

$ sudo apt install -f

apt

apt is a command-line utility for installing, updating, removing, in short managing deb packages on Ubuntu, Debian, and related Linux distributions. It was introduced in Ubuntu 14.04 and combines the most commonly used commands from apt-get and apt-cache.

The apt package manager will resolve and install all the package dependencies by itself, so it's better to use apt instead of dpkg.

To install local deb packages with apt you need to provide the full path to the deb file. If the file is located in your current working directory instead of typing the absolute path, you should prepend ./ before the package
name. Otherwise, apt will try to retrieve and install the package from Ubuntu’s repositories.

Let's assume you are in the same directory after wgetting Vscode deb package, then just simply run:

# First update package indices
$ sudo apt update
# Install via apt
$ sudo apt install ./vscode-stable.deb

Conclusion

We saw how to install local deb packages on a Debian based distribution —as an example Ubuntu. When installing deb packages prefer using apt since it resolves and install all the package dependencies.

All done!

Python Interactive Shell Interfaces

Serhat Teker — Thu, 03 Mar 2022 00:00:00 +0000

The Python Interactive Shell is an interactive interpreter that can execute Python commands.

You can invoke it from your default shell — zsh, bash, fish etc.:

$ python
# or for a specific python version
$ python3.9

Also you can run interactive shell with a module/file like below:

$ python -i somefile.py

Use Case

Thanks python, however I can't say I enjoy your default shell. I need more capable, colorful interactive shell like IPython, bpython.

It's easy to achieve this in Django. It's well documented in django-shell:

# IPython
$ django-admin -i ipython
# bpython
$ django-admin -i bpython

Well, how can we achieve it in Python?

Solution

Python's way is kind of similar; you invoke it with some module using -m flag:

For IPython run:

$ python -m IPython -i
# with module/file
$ python -m IPython -i somefile.py

For bpython run:

$ python -m bpython -i
# with module/file
$ python -m bpython -i somefile.py

NOTE
In order to be able to use IPython and/or bpython first you have to install them:
$ python -m pip install ipython
$ python -m pip install bpython

Yes, it's that easy and now we can take advantage of amazing features like below:

In-line syntax highlighting
Readline-like auto-complete with suggestions displayed as you type
Expected parameter list for any Python function, etc.

Probably it would be better to have them as aliases:

# .aliases
alias pi='python -m IPython -i'
alias pib='python -m bpython -i'

All done!

Unknown Vim Trick

Serhat Teker — Thu, 24 Feb 2022 00:00:00 +0000

One of beautiful thing about vim is that you can learn something even after using 5+ years. Even though usual vim learning curve characterized like below:

This is the command I've been looking for a long time. I accidentally saw it on one of the ThePrimeagen.

Entrée

Let's assume that I'm on the line 5 and want to update "_cache_update_cache" on the line 30, to "_cache_update_current".


...


...



class UpdateCacheMiddleware(MiddlewareMixin):
    """
    Response-phase cache middleware that updates the cache if the response is
    cacheable.
    Must be used as part of the two-part update/fetch cache middleware.
    UpdateCacheMiddleware must be the first piece of middleware in MIDDLEWARE
    so that it'll get called last during the response phase.
    """

    def __init__(self, get_response):
        super().__init__(get_response)
        self.cache_timeout = settings.CACHE_MIDDLEWARE_SECONDS
        self.page_timeout = None
        self.key_prefix = settings.CACHE_MIDDLEWARE_KEY_PREFIX
        self.cache_alias = settings.CACHE_MIDDLEWARE_ALIAS

    @property
    def cache(self):
        return caches[self.cache_alias]

    def _should_update_cache(self, request, response):
        return hasattr(request, "_cache_update_cache") and request._cache_update_cache

...

What I was doing: press 25jf"ci" on Normal mode, which means:

25j : down 25 line
f" : find the first occurrence of "
ci" : change inside of "

and then write _cache_update_current.

The annoying part is you have to press additional f" every time. For a normal user it probably means nothing, but for a vim user it means a lot, since "Real Vim ninjas count every keystroke"¹

Therefore I was using a macro and key binding for this. Until now.

Trick

The trick is just press ci"; you don't have to press additional f" (for an unknown reason it works which shouldn't). So our above command is just 25jci" now.

And it works with usual text objects like {, [, *, ' etc.

All done!

https://www.vimgolf.com ↩

Change Default Git Init Branch

Serhat Teker — Thu, 17 Feb 2022 00:00:00 +0000

As a developer my default branch is usually develop for my personal projects. (I don't want to deep dive into details of Git Branching strategies like Trunk-Based Development Git Flow (Feature Based Development) etc. maybe later in another post.)

So I want to make my default branch name other than master for the first commit.

And my common workflow looks like this:

$ git init && git commit --allow-empty -m "Initial commit"
$ git checkout -b develop
$ git push -u origin

So the question is: Is there any another way to change the default first branch name that git init sets?

Solution

Yes, there is, but you need minimum Git version 2.28.
Just run:

$ git config --global init.defaultBranch develop

Or as usual you can add this directly into your global config file:

# ~/.gitconfig
# ~/.config/git/config
# or
# ${XDG_CONFIG_HOME}/git/config

...

[init]
    # >=2.28.0
    defaultBranch = develop

...

Now whenever I initiate a git repo my default branch will be develop and this will save a lot of keystroke.

All done!

Tmux New Session

Serhat Teker — Thu, 10 Feb 2022 00:00:00 +0000

Entrée

I keep forgetting how to create a new tmux session.

Is it this one?

$ tmux new
sessions should be nested with care, unset $TMUX to force

No.

Is it this one?

$ tmux new-session
sessions should be nested with care, unset $TMUX to force

No.

$ tmux new-session -d
# from tmux cli, prefix/ctrl+b
# :new -d

No error? Good. Check session with prefix + s (ctrl + b + s). Ok it seems this works.

However this gives a strange session name like 29. What is it? I though the ultimate answer to the universe was 42.

Solution

Here is the complete command to create a new tmux session:

$ tmux new-session -d -s "new" -c ~/path/to/some/dir -n "main"

This command will create a session named new, with a window named main in ~/path/to/some/dir as working directory.

It would probably better to run it from function with some args for more customization;

# .zshrc/.bashrc


# Crate and attach a new session with args
# defaults: session named "new" in ${HOME} as working dir with window named "main"
#
# Usage:
# $ tnew
# $ tnew remote ~/path/to/dir
# $ tnew remote ~/path/to/dir scripts
tnew() {
    local session_name="${1:-new}"
    local session_dir=${2:-~/}
    local session_window_name="${3:-main}"

    tmux new-session \
        -d \
        -s ${session_name} \
        -c ${session_dir} \
        -n ${session_window_name}
}

As described in function's doc you can use this function as below:

$ tnew
$ tnew remote ~/path/to/dir
$ tnew remote ~/path/to/dir scripts

Now I can peacefully forget (again) creating new tmux session, all I need to
remember: tnew.

All done!

How to Run Multiple Long Commands in Docker Compose

Serhat Teker — Thu, 03 Feb 2022 00:00:00 +0000

Use Case

I want to run multiple/long commands for my service(s) in my docker-compose file, like below:

---
# docker-compose.yaml

services:
  server: &server
    build:
      context: .
      dockerfile: ./compose/staging/server/Dockerfile
    image:  someproject_slug_staging_server:staging_shortsha_unixtime
    depends_on:
      - postgres
    env_file:
      - ./.envs/.staging/.server
      - ./.envs/.staging/.postgres
    command: python /app/manage.py collectstatic --no-input
    command: python /app/manage.py migrate
    command: gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app

  postgres:
    build:
      context: .
      dockerfile: ./compose/staging/postgres/Dockerfile
    image:  someproject_slug_staging_postgres:staging_shortsha_unixtime
    volumes:
      - staging_postgres_data:/var/lib/postgresql/data:Z
      - staging_postgres_data_backups:/backups:z
    env_file:
      - ./.envs/.staging/.postgres

However it won't work since it's not possible.

Solutions

Script File

One solution is that putting all your commands in a script file -let's say start.sh, put it into your image, build with it and call it directly.

As an example here is a docker compose file:

---
# staging.yaml

version: '3.7'

volumes:
  staging_postgres_data: {}
  staging_postgres_data_backups: {}
  staging_traefik: {}
  staging_redis: {}

services:
  server: &server
    build:
      context: .
      dockerfile: ./compose/staging/server/Dockerfile
    image:  someproject_slug_server:staging_shortsha_unixtime
    depends_on:
      - postgres
      - redis
    env_file:
      - ./.envs/.staging/.server
      - ./.envs/.staging/.postgres
      - ./.envs/.staging/.redis
    command: /start

  postgres:
    build:
      context: .
      dockerfile: ./compose/staging/postgres/Dockerfile
    image:  someproject_slug_postgres:staging_shortsha_unixtime
    volumes:
      - staging_postgres_data:/var/lib/postgresql/data:Z
      - staging_postgres_data_backups:/backups:z
    env_file:
      - ./.envs/.staging/.postgres

  traefik:
    build:
      context: .
      dockerfile: ./compose/staging/traefik/Dockerfile
    image:  someproject_slug_traefik:staging_shortsha_unixtime
    depends_on:
      - server
    volumes:
      - staging_traefik:/etc/traefik/acme:z
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
      - "0.0.0.0:5555:5555"

  redis:
    image: docker.io/bitnami/redis:6.2
    env_file:
      - ./.envs/.staging/.redis
    ports:
      - "6379:6379" # WARNING: In production never bind 0.0.0.0 wildcard adress of the host
    volumes:
      - staging_redis:/bitnami/redis/data:z

  celeryworker:
    <<: *server
    command: /start-celeryworker

  celerybeat:
    <<: *server
    command: /start-celerybeat

  celeryflower:
    <<: *server
    command: /start-celeryflower

  aws-backup:
    build:
      context: .
      dockerfile: ./compose/staging/aws/Dockerfile
    env_file:
      - ./.envs/.staging/.server
    volumes:
      - staging_postgres_data_backups:/backups:z

WARNING

Don't bind your host's 0.0.0.0 wildcard adress in production environment. (Actually instead of deploying via docker-composein production you should consider to use more reliable, and robust distributed systems like Kubernetes but this is a topic for another post.)

6379:6379 means 0.0.0.0:6379:6379, so you open your redis instance to whole open world. And if you don't use AUTH or don't take required security precaution -like TLS etc. you may get some headache. So use it only if you
know the related consequences.

For more redis security topic look at this: Redis Security

And as an example here is content of start from server service:

#!/usr/env/bin bash
# start

set -o errexit
set -o pipefail
set -o nounset


python /app/manage.py collectstatic --noinput
python /app/manage.py migrate   # WARNING: Not a good idea for prod
gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app

WARNING

Migration is not a command you want to run every container start but since it's staging environment let's assume it's OK.

Multiline Scalar

Running in a script file is OK however maybe it's local or staging environment which means you need to debug and tweak these commands one by one; maybe some flags are not working etc. and you don't want to re-build the image every time.

How are we gonna do this?

The answer lines in the multiline capability of yaml syntax:

command: >
  bash -c "python /app/manage.py collectstatic --noinput
  && python /app/manage.py migrate
  && gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app"

This is called Folded Style. The indention in each line will be ignored. A line break (\n) will be inserted at the end.

Or you can use Literal Style:

command:
  - /bin/bash
  - -c
  - |
  python /app/manage.py collectstatic --noinput
  python /app/manage.py migrate
  gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app

This turns every newline within the string into a literal newline, and adds one at the end.

Folded Style

Key: >
  this is my very very very
  long command

Output:

this is my very very very long command\n

Mind new line-\n (end of line-EOL)

Literal Style

Key: |
  this is my very very very
  long command

Output:

this is my very very very\nlong command\n

Mind new lines-\n, and compare to it above output.

Fore more detailed style and yaml syntax you can look at this amazing explanation in this link.

These were the solutions for running multiple or long commands in your docker-compose file.

All done!

Merge Upstream Pull Request into Your Fork

Serhat Teker — Thu, 27 Jan 2022 00:00:00 +0000

Use Case

I have forked a repo, I've made some changes and my fork and upstream diverged.

In the meantime someone sent a PR to upstream and this PR is still open or rejected/closed etc by maintainers. However I'd like to apply the changes that PR proposes into my fork.

Solutions

One Time

If this is a one time thing, then you can get with fetch:

$ git fetch upstream pull/42/head

Then you can merge, cherry-pick etc. these PR changes.

Or you can pull directly:

$ git pull upstream pull/42/head

INFO

As you may see, above scripts assumes your remote upstream repo named upstream and 42 is the PR number you'd like to apply, change them accordingly.

Git Alias

If you regularly will apply PR*s then you can add related functions as *aliases in your local or global config, a.k.a one of the below:

${YOUR_REPO}/.git/config (local)
~/.gitconfig (global)
${XDG_CONFIG_HOME}/git/config (global)

# ${REPO}/.git/config
# ~/.gitconfig
# ${XDG_CONFIG_HOME}/git/config
[alias]
    gfpr = "!f() { git fetch upstream refs/pull/$1/head:pr/$1; } ; f"
    gppr = "!f() { git pull upstream pull/$1/head; } ; f"

Then fetch all *PR*s.

$ git fetch upstream

To check out a particular PR #42:

$ git checkout pr/42
Branch pr/42 set up to track remote branch pr/42 from upstream.
Switched to a new branch 'pr/42'

Local gitconfig

You can append fetch config in your local config.

# .git/config
[remote "upstream"]
    fetch = +refs/heads/*:refs/remotes/origin/*
    url = git@github.com:joyent/node.git
    fetch = +refs/pull/*/head:refs/remotes/origin/pr/*

The rest is the same as above.

With Script

You can append above related fetch config with below script:

#!/usr/bin/env bash
#
# git-fetch-pr.sh

git remote -v | grep fetch | grep github | \
    while read remote url _; do
        if ! git config --get-all "remote.$remote.fetch" | grep -q refs/pull
        then
            git config --add "remote.$remote.fetch" \
                '+refs/pull/*/head:refs/remotes/'"$remote"'/pull/*'
        fi
    done

The rest is the same.

All done!

Resources:

Convert Strings To Be URL Safe

Serhat Teker — Thu, 20 Jan 2022 00:00:00 +0000

In order to convert your strings to be URL safe you can use python's built-in urllib:

>>> from urllib import parse
>>> string_raw = "Test: Hey! this is an example string"
>>> parse.quote(string_raw)
'Test%3A%20Hey%21%20This%20is%20an%20example%20string'

quote() replaces special characters in string using the %xx escape. Letters, digits, and the characters _.-~ are never touched. By default, this function is intended for converting the path section of a URL.

If you want to use your strings as query strings you can use quote_plus:

>>> from urllib import parse
>>> string_raw = "Test: Hey! this is an example string"
>>> parse.quote(string_raw)
'Test%3A+Hey%21+This+is+an+example+string'

quote_plus() is like quote() but also replace spaces with plus signs. It is required for building HTML form values when building up a query string to go into a URL.

For more info you can look at official python doc: quote, quote_plus

All done!

List Files Tracked by Git

Serhat Teker — Thu, 13 Jan 2022 00:00:00 +0000

All current files

In order to show all of the tracked files that have been committed on the current branch, run:

$ git ls-tree --full-tree --name-only -r HEAD

--full-tree makes the command run as if you were in the repo's root directory.
-r recurses into subdirectories. Combined with --full-tree this gives you all committed, tracked files.
--name-only removes SHA / permission info for when you just want the file paths.
HEAD specifies which branch you want the list of tracked, committed files for. HEAD is the pointer for the commit you have checked out currently.

And the output will be like:

...
static/img/logopal.png
static/img/logos/bitcoin.png
static/img/logos/bmc-button.svg
static/img/logos/django-logo.png
static/img/logos/git-logo-full.png
static/img/logos/git-logo-red.png
static/img/logos/git-logo-white.png
static/img/logos/neovim-logo.png
static/img/logos/python-logo.png
static/img/logos/python-logo2.png
static/img/logos/python-logo3.png
static/img/logos/python-logo4.png
static/img/logos/python-logo5.png
static/img/logos/python-logo6.png
static/img/logos/python-logo8.svg
static/img/logos/vim-logo.png
static/img/python3.7-default.jpg
static/img/safari-pinned-tab.svg
...

If you want to show the files on the master branch, replace HEAD with master:

$ git ls-tree --full-tree --name-only -r master

Probably you also want a alias for that:

alias gtf='git ls-tree --full-tree --name-only -r HEAD'

All files including deleted

If you want to list all of files that ever existed including deleted files, run below:

$ git log --pretty=format: --name-only --diff-filter=A | sort - | sed '/^$/d'

All done!

References:

stackoverflow