DEV Community: Mikołaj Buchwald

Generating SSH key – the best way

Mikołaj Buchwald — Tue, 04 Jul 2023 09:59:23 +0000

tl;rd:

ssh-keygen -t rsa -b 2048 -C "user@institution" -f ~/.ssh/institution-user

where:

user is your identifier, e.g., "joe"
institution is your workplace, or an indication that it is your personal key, e.g., "buchcorp", "personal", etc.

An additional step is to rename the private key to have a *.pem extension:

mv ~/.ssh/institution-user ~/.ssh/institution-user.pem

I don't know the security standpoint/standards, but from the user experience/usability point of view I am a great fan of file extensions in general.

Important ingredients of a perfect `ssh-keygen` command

There are numerous tutorials on the web how to generate a pair of SSH keys. However, I didn't find a single one that covers all the arguments/parameters that in my opinion are essential when generating an SSH key pair via a terminal. What are these essential ingredients of a perfect ssh-keygen command? Here they are:

Comment –- needles to say, the comment doesn't seem that important, unless you are faced with an impossible task of disentangling a multiple keys sitting in authorized_keys for months, if not years. Good luck guessing which key belong to who then :) In other words, always put a meaningful text that will allow others (as well as yourself) to identify the particular key as yours (unless you are a hacker, or whatever :D )
Filename -- you want your private and public keys to have a meaningful name right away –- else they can get lost in a number of the default id_rsa files, or worse –- you will overwrite an important key. Please note that I explicitly mention the path to the file: -f ~/.ssh/key-name (instead of specifying just the file name).

The command

Here is the example command using the above-mentioned characteristics:

ssh-keygen -t rsa -b 2048 -C "user@institution" -f ~/.ssh/institution-user

where:

user is your identifier, e.g., "joe"
institution is your workplace, or an indication that it is your personal key, e.g., "buchcorp", "personal", etc.

mv ~/.ssh/institution-user ~/.ssh/institution-user.pem

I don't know the security standpoint/standards, but from the user experience/usability point of view I am a great fan of file extensions in general.

Sources

Bonus: see how to manage multiple SSH keys in my other article: https://dev.to/mikolajbuchwald/git-via-ssh-multiple-keys-management-3bkm

Cover image: photo by Michael Dziedzic on Unsplash

Conda equivalent of `pipenv run`

Mikołaj Buchwald — Wed, 14 Jun 2023 13:54:24 +0000

If you wish to run a single command using a Python environment created with conda (i.e., you don't want to activate the environment first), you can use the following command:

conda run -n your_env_name some_command

E.g.:

conda run -n webprojects python --version

or:

conda run -n datascience jupyter lab

Make sure that you are using the -n argument; pipenv does not require specifying an "optional" argument, hence it took me some time to get this working (i.e., it works just like pipenv run jupyter lab).

References:

Cover Photo by Thomas Oldenburger on Unsplash

Running Bash commands with Python

Mikołaj Buchwald — Tue, 25 Apr 2023 20:54:36 +0000

When plain Bash is not enough

A while ago I read an excellent introductory Bash tutorial on Dev.to by Sahil Bondre.

It is a really great tutorial on shell scripting basics, but I remember when I was beginning my adventure with shell scripting, and when I was (over)fascinated with Bash. In time, when my neuroimaging scripts started becoming more and more complex, I slowly began loosing control over my code!

Many times I felt like that:

The Python era

That was about that time when I discovered, that at some point, when more complicated logical expressions were needed (more conditionals, loops, and functions), it’s worth switching from Bash to Python. Hopefully, with Python you can wrap (almost) any bash command. And it is as simple as that:

import subprocess as sp

bash_command = "ls ~/"
print("\nBash command:\n%s\n" % bash_command)

process = sp.Popen(bash_command, stdout=sp.PIPE, shell=True)
output, error = process.communicate()

print("\nBash command output:\n%s\n" % output)
print("Bash command error:\n%s\n" % error)

(It works both for Linux and macOS.)

Note the additional prints, as it is easier to debug the command before it is run. Additionally, you can see the output of the command (just like for a normal bash shell).

The above example was inspired by this StackOverflow answer.

It turned out that Python syntax is more readable and easier to maintain, yay! 🙂

Take `~/` message

Don't stick with a plain bash for too long — often all you really needed is Python! Nevertheless, for a very simple scripts, you can of course always make a use of a good ol’ Bash ;)

See also my original post from 2020 (now it is already 2023 :) , but the knowledge is still a very much up-to-date): https://www.mindyourdata.org/posts/when-plain-bash-is-not-enough/

Cover photo source: Photo by Abhishek rana on Unsplash

Initializing a `git` repository with `git-lfs` (large file storage)

Mikołaj Buchwald — Wed, 18 Jan 2023 14:53:25 +0000

As it happens, you would like to initialize a git repository with git-lfs (large file storage) functionality to store some really large files. For someone a file of a 10MB size may seem large, for somebody else it would have to be as 1GB in size, but regardless of that, this tutorial explains how to efficiently initialize a repository with git-lfs.

Prerequisites

First, make sure that you have git lfs installed (how-to install for: macOS, Linux). In short, for macOS: brew install git-lfs.

Initializing `git-lfs`

Assuming that you have not initialized your repository with the "normal" git command yet, run the following commands:

git init
git lfs install

For the sake of the example (to show you some difficulties you may encounter), let's assume that the files that you wish to add to git-lfs are stored in two separate directories (e.g., data and supplementary_materials). The tree of your repository looks somehow like that:

.
├── ...
├── README.md
├── data
│   ├── data1.png
│   ├── data2.zip
│   └── label.txt
├── images
│   └── some_pict.png
└── supplementary_materials
    ├── first_subdir
    │   └── somefile.png 
    ├── second_subdir
    │   └── somefile.png 
    └── third_subdir
        └── somefile.zip

First, some basic stuff about synchronizing with your remote repository:

git remote add origin https://github.com/mikbuch/your-repository.git

Then:

git branch -M main
git pull origin main

Now, there are basically two ways of adding files to git-LFS.

The first (recommended) method

Create .gitattributes file:

vim .gitattributes

and add the following lines:

data/**/* filter=lfs diff=lfs merge=lfs -text
supplementary_materials/**/* filter=lfs diff=lfs merge=lfs -text

Note! In our example we have two separate directories that you wish to add to git-lfs in which you have some subdirectories. (1) The syntax has to be exactly like the one in the code above, or else you won't add all files in all subdirectories. (2) If you have several directories, from the point of view of your current working directory, you have to list them all, one by one.

The second (not recommended) method

Use the following commands to add directory and all files in this directory and its subdirectories:

git lfs track data/**/*
git lfs track supplementary_materials/**/*

Note! The command git lfs track supplementary_materials/** will not be sufficient to add all files in all subdirectories.

Why this method is not recommended?: It will list/add all tracked files to your .gitattributes (one by one!), e.g.:

➜  git:(main) ✗ cat .gitattributes

data/data1.png filter=lfs diff=lfs merge=lfs -text
data/data2.zip filter=lfs diff=lfs merge=lfs -text
data/label.txt filter=lfs diff=lfs merge=lfs -text
...
supplementary_materials/first_subdir/somefile.png filter=lfs diff=lfs merge=lfs -text
supplementary_materials/second_subdir/somefile.png filter=lfs diff=lfs merge=lfs -text
supplementary_materials/third_subdir/somefile.zip filter=lfs diff=lfs merge=lfs -text
...

What you would like to have in your .gitattributes is just:

data/**/* filter=lfs diff=lfs merge=lfs -text
supplementary_materials/**/* filter=lfs diff=lfs merge=lfs -text

It is much more clearer approach. See the first (recommended) method described above.

Preview the `.gitattributes` file

cat .gitattributes

Add all files to your repository:

git add --all

Show which files are tracked with (`git lfs ls-files`)

git lfs ls-files

Note! git lfs ls-files works only after you have added the files to the repository with git add.

Show which files were added to the repository (regardless of whether LFS or not):

git status

Make sure that the files are tracked with git-LFS:

du -sh .git
du -sh .git/lfs

Note on `GitHub`

If you are using GitHub, please make sure that you are using the proper SSH file:

cat ~/.ssh/config

In GitHub, also make sure that your key is added to you profile (not just one repository!). See the output of a command:

ssh -T -ai ~/.ssh/my-key.pem git@github.com

Source: https://superuser.com/a/232406/950943

If you used this key for one repository only you will see somethink like this:

Hi mikbuch/your-repository! You've successfully authenticated, but GitHub does not provide shell access.

You should see output like this:

Hi mikbuch! You've successfully authenticated, but GitHub does not provide shell access.

Committing and pushing to the server

Commit the files (both LFS and "normal"):

git commit -a -m "Initializing commit with git LFS"

Finally, you can push your reporitory to the remote server:

git push -u origin main

In order to reset (if needed/optional) during the process:

rm -rf .git
rm .gitattributes

And that's it, I hope you will find this tutorial useful, have a nice day!

Additional resources

045 Introduction to Git LFS (Large File Storage) by Dan Gitschooldude: https://youtu.be/xPFLAAhuGy0
Example use: https://github.com/git-lfs/git-lfs/wiki/Installation#debian-and-ubuntu

Source for the cover photo image: Photo by Wesley Tingey on Unsplash

EasySSH - GUI host manager for Linux

Mikołaj Buchwald — Wed, 03 Aug 2022 20:43:00 +0000

If you like to have your virtual servers catalogued and available quickly with SSH, a decent GUI for host manager may come in handy.

On macOS I absolutely love using Core Shell, but I struggled a bit to find a decent Linux (Ubuntu) alternative for that. Until I came across this post and review by Jack Wallen at linux.com, and I finally found out about EasySSH!

EasySSH is a decent piece of software, and it has all functionalities I have been looking for.

Configuring a host:

Connecting:

For the review and more screenshots please see the post I linked above.

Installing

The best part is that EasySSH that it is available via apt, so installing it is as simple as:

sudo apt-get -y install easyssh

Give it a try and let me know what you think in the comments!

Source of the graphics in the cover: Photo by Erol Ahmed on Unsplash

Conda tip from a `requirements.txt` type of person

Mikołaj Buchwald — Sat, 23 Jul 2022 13:39:32 +0000

I am a requirements.txt (venv/pipenv) type of person. I have my reasons for that, but I won't mention these reasons here (well, technically, problems such as the one described in this post is one of these reasons).

At some point, I finally need to get familiar with conda. The first surprise: installing conda on linux is not as simple as sudo apt-get install ... or sudo pip install .... You need to download a bash installer and run it. But I was cool with that. After choosing some installation options, and adding conda to the PATH (it was also not mentioned explicitly in the installation guides that you have to do it [1][2]) I was ready to go.

In the repository of a project I wanted to start working on, I had an environment.yml file (which is an equivalent of a requirements.py). Now comes the most tricky part:

The official documentation says:

Create the environment from the environment.yml file:

conda env create -f environment.yml

Ok, so I did run this command, but I got the following error:

CondaValueError: prefix already exists: /home/user/anaconda3

That is unfortunate. Anyways, I googled the error, and here's what I found:

Conda bug: CondaValueError: prefix already exists: /home/user/anaconda2

Long story short, if your .yml file have an empty name field, you have to explicitly specify environment name when creating the environment with conda command, like that:

conda env create -f environment.yml -n environmentName

I followed this advice and my environment has been successfully created.

To sum up, my tip, as a requiements.txt type of person, is that conda is less "user friendly" in the sense that you have to be more aware what is going on, in terms of command execution and naming of the environments. To compare this to the pipenv's behavior: by default pipenv creates a directory with your virtual environment in the ~/local/share/virtualenvs/ which comprises of the name of the directory where you invoke pipenv command, and a hash. So, the entry level is lower, you just run the command and it works, but then you have to know where the environment is located (you get that information also in the pipenv's output log). Conda, on the other hand, is harder to use because of the more enigmatic error information, and the conventions you have to be aware of, before even running the command.

Git via SSH - multiple keys management

Mikołaj Buchwald — Thu, 23 Jun 2022 21:11:27 +0000

In order to use git over ssh, you need ssh keys. When you want to use multiple keys for different domains, you have to know where and how to configure it. I always forget where this configuration file is stored, so here I make a short note about it. Just edit ~/.ssh/config, e.g.:

vim ~/.ssh/config

Then put there something like:

Host github.com
 HostName github.com
 IdentityFile ~/.ssh/id_rsa_github
Host gitlab.com
 HostName gitlab.com
 IdentityFile ~/.ssh/id_rsa_gitlab

Et violà!

Source: https://superuser.com/a/232406/950943

Edit: Troubleshooting: On AWS' EC2 machines you may get an Bad owner or permissions on /home/ec2-user/.ssh/config error when trying to clone a repository, e.g.:

[ec2-user@ip-my-ip ~]$ git clone git@gitlab.com:mikbuch/my-repo-name.git
Cloning into 'my-repo-name'...
Bad owner or permissions on /home/ec2-user/.ssh/config
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

You have to make sure that you have 400 permission on your ~/.ssh/config file, i.e.:

[ec2-user@ip-my-ip ~]$ ls -la ~/.ssh/
total 12
drwx------ 2 ec2-user ec2-user   73 Dec 29 19:51 .
drwx------ 7 ec2-user ec2-user  175 Dec 29 19:52 ..
-r-------- 1 ec2-user ec2-user 1831 Dec 29 19:50 my-deployment-key.pem
-rw------- 1 ec2-user ec2-user  957 Dec 20 20:25 authorized_keys
-rw-rw-r-- 1 ec2-user ec2-user   81 Dec 29 19:51 config

Use the following command to change the permissions:

[ec2-user@ip-my-ip ~]$ chmod 400 ~/.ssh/config

See the result:

[ec2-user@ip-my-ip ~]$ ls -la ~/.ssh/
total 16
drwx------ 2 ec2-user ec2-user   92 Dec 29 19:53 .
drwx------ 8 ec2-user ec2-user  196 Dec 29 19:53 ..
-r-------- 1 ec2-user ec2-user 1831 Dec 29 19:50 my-deployment-key.pem
-rw------- 1 ec2-user ec2-user  957 Dec 20 20:25 authorized_keys
-r-------- 1 ec2-user ec2-user   81 Dec 29 19:51 config
-rw-r--r-- 1 ec2-user ec2-user  207 Dec 29 19:53 known_hosts

Now the clonning should work:

[ec2-user@ip-my-ip ~]$ git clone git@gitlab.com:mikbuch/my-repo-name.git
Cloning into 'my-repo-name'...
The authenticity of host 'gitlab.com (2606:4700:90:0:f22e:fbec:5bed:a9b9)' can't be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.
ECDSA key fingerprint is MD5:f1:d0:fb:46:73:7a:70:92:5a:ab:5d:ef:43:e2:1c:35.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gitlab.com,2606:4700:90:0:f22e:fbec:5bed:a9b9' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 287, done.
remote: Counting objects: 100% (206/206), done.
remote: Compressing objects: 100% (169/169), done.
remote: Total 287 (delta 37), reused 200 (delta 36), pack-reused 81
Receiving objects: 100% (287/287), 1.25 MiB | 3.19 MiB/s, done.
Resolving deltas: 100% (64/64), done.

How to update Firefox on Ubuntu Linux

Mikołaj Buchwald — Sun, 12 Jun 2022 11:54:06 +0000

tldr;

# Backup the old version
sudo mv /opt/firefox /opt/firefox_bak

# Untar the new Firefox
sudo tar xjf firefox-*.tar.bz2 -C /opt

# Remove the tar from Downloads
rm -r ~/Downloads/firefox-*.tar.bz2

I installed Firefox on Kubuntu with tarball (no snap nor flatpak):

I went to: https://www.mozilla.org/en-US/firefox/new/ , and downloaded the tarball
sudo tar xjf ~/Downloads/firefox-*.tar.bz2 -C /opt/
sudo ln -s /opt/firefox/firefox /usr/bin/firefox
rm -r ~/Downloads/firefox-*.tar.bz2

When it came to updating Firefox (I got a pop-up in top right corner of the application), I downloaded a new version (used to have 100.x, now I downloaded 101.x), and I used the following commands:

# Backup the old version
sudo mv /opt/firefox /opt/firefox_bak

# Untar the new Firefox
sudo tar xjf firefox-*.tar.bz2 -C /opt

# Remove the tar from Downloads
rm -r /Downloads/firefox-*.tar.bz2

Graphic from: https://en.wikipedia.org/wiki/Red_fox

Integrating Django framework with Matomo web analytics platform

Mikołaj Buchwald — Tue, 05 Apr 2022 12:42:33 +0000

Recently I wanted to integrate a web application we are working on with Matomo analytics functionalities. Django web framework was used in this application on the backend side. It turned out that there was a package for integrating Matomo with Django -- injecting JavaScript via Django's template mechanism. However, this package was from 2013 (9yo), and it was written for piwik -- the former name of the Matomo.

So I decided to refresh the package and I created django-matomo. The module is available at pip, with:

pip install django-matomo

PyPi website of the package

The source code is available at the django-matomo GitHub repository

I contacted the Matomo Team, and they agreed to update the official support for the integration for Django to django-matomo! It is now listed at their website: https://matomo.org/integrate/

There is also a post I created on Django forums, if anyone have questions regarding the integration on the Django side.

Credits for the django-piwik (legacy package) goes to its creator Raphael Jasjukaitis.

Drawing a rectangle in OpenCV Python

Mikołaj Buchwald — Sun, 27 Mar 2022 22:19:37 +0000

How to draw a rectangle in OpenCV Python

A simplistic example for beginners on how to draw a rectangle using OpenCV in Python. Example as a Google Colaboratory notebook.

Part of my answer to FB post at Python's group.

Python Django `dbshell`

Mikołaj Buchwald — Fri, 25 Mar 2022 12:08:54 +0000

Python Django's dbshell is a very useful tool for SQL database debugging when working on a Django application.

To run SQL shell interactively:

python manage.py dbshell

You should see:

SQLite version 3.32.3 2020-06-18 14:16:19
Enter ".help" for usage hints.
sqlite>

Useful Django sqlite commands:

.help
.tables

An example SQL command:

SELECT * FROM django_migrations;

In order to see the names of the columns (the header) of the table:

.header on
.mode column
pragma table_info('django_migrations');

Example output:

sqlite> pragma table_info('django_migrations');
cid         name        type        notnull     dflt_value  pk        
----------  ----------  ----------  ----------  ----------  ----------
0           id          integer     1                       1         
1           app         varchar(25  1                       0         
2           name        varchar(25  1                       0         
3           applied     datetime    1                       0