DEV Community: João Vitor

Keep man output in the screen

João Vitor — Tue, 28 Dec 2021 17:31:26 +0000

Reading man man I found out that the pager used by man can be overridden.

 -P pager, --pager=pager
              Specify which output pager to use.  By default, man uses
              less, falling back to cat if less is not found or is not
              executable.  This option overrides the $MANPAGER
              environment variable, which in turn overrides the $PAGER
              environment variable.  It is not used in conjunction with
              -f or -k.

              The value may be a simple command name or a command with
              arguments, and may use shell quoting (backslashes, single
              quotes, or double quotes).  It may not use pipes to
              connect multiple commands; if you need that, use a wrapper
              script, which may take the file to display either as an
              argument or on standard input.

MANPAGER variable

export MANPAGER="less -FirSwX"

And after closing man you can still read the man page that you were reading.

Git user config per path

João Vitor — Tue, 22 Jun 2021 19:13:25 +0000

TIL git config core.sshCommand

TIL here that git provides a config for core.sshCommand.

I was already splitting my personal and work git user configuration by path with this setting in my ~/.gitconfig.

So adding the block below simplified my .ssh/config and my cloning/pushing.

[core]
       sshCommand = "ssh -i ~/.ssh/id_ed25519_personal -F /dev/null"

~/.gitconfig

[include]
    path = ~/dotfiles/gituser_work.include
[includeIf "gitdir/i:~/dotfiles/"]
    path = ~/dotfiles/gituser_personal.include

gituser_work.include

[user]
       name = Your Full Name
       email = your@work-email.com

gituser_personal.include

[user]
       name = Your Full Name
       email = your@personal-email.com

[core]
       sshCommand = "ssh -i ~/.ssh/id_ed25519_personal -F /dev/null"

Clone github repos in parallel

João Vitor — Sat, 13 Feb 2021 16:10:18 +0000

It is possible to clone a list of github repositories in parallel.

Given that the file is called your_organization.txt.

cd ~/github-orgs/your-organization
cat your_organization.txt | parallel  -j 25 'clone-missing.sh {}; echo job {#} completed {};'

You can generate a list of repositories of a github user or a github organization with these helper scripts:

Clone azure repos in parallel

João Vitor — Sat, 13 Feb 2021 16:01:04 +0000

Azure repos are hosted inside devops projects.

You can list azure projects with

az devops project list --detect true

To list repositories in azure from one specific az devops project

az repos list --project project_id

To grab the repo list from multiple projects in parallel and save it to a file:

org=$(basename $(pwd)); az devops project list --detect true | jq -r '.value[]|.id' | parallel -j 10 "az repos list --project {} | jq -r '.[]|.sshUrl'" | tee ${org}.txt

As the repo ssh url is different from the github one the clone-missing.sh script had to be changed in a not compatible way.

Difference

github ssh url: git@github.com:${githuborg}/${repo}.git
azure repo url: git@ssh.dev.azure.com:v3/${devopsorg}/${devopsproject}/${repo}

New azure-clone-missing.sh command added to the parallel_commands to handle this url difference.

Cloning azure repositories in parallel

Given that you created the org.txt file the command below clones only the missing repositories.

cd ~/azure-orgs/azuredevopsorg/
org=$(basename $(pwd)); cat ${org}.txt | parallel  -j 25 'azure-clone-missing.sh {}; echo job {#} completed {};'

If you want to exclude some repositories from your clone.

PS change the regex that excludes the repo.
grep -Ev '(repo1|repo2)' ${org}.txt

cd ~/azure-orgs/azuredevopsorg/
org=$(basename $(pwd)); grep -Ev '(repo1|repo2)' ${org}.txt | parallel  -j 25 'azure-clone-missing.sh {}; echo job {#} completed {};'

Documentation reference

Update git repos in parallel

João Vitor — Thu, 29 Oct 2020 13:54:46 +0000

Are you living in a microservices world?
Do you need to keep multiple git repositories in sync?

How much time would it take to update 63 repositories?

update-repos.sh

The update-repos.sh script uses gnu-parallel and execute the git pull --rebase and git fetch --prune in parallel.

Update repositories in parallel

cd ~/github-orgs/PacktPublishing
time update-repos.sh

Output:

[truncated]
job 56 completed ./Develop-and-Operate-Microservices-on-Kubernetes
Already up to date.
job 55 completed ./DevOps-Puppet-Docker-and-Kubernetes
Already up to date.
job 62 completed ./-Kubernetes-for-Absolute-Beginners---Hands-on
[truncated]

real    0m9,083s
user    0m6,321s
sys     0m2,668s

If you do that synchronously it would have taken near 4 minutes.

real    4m53,583s
user    0m6,480s
sys     0m4,776s

Using gnu parallel to save time

João Vitor — Thu, 29 Oct 2020 05:52:36 +0000

As a developer I'm constantly reading code, specially from github.

Often I choose to clone repos from some user or organization and I've been using parallel to save me some time cloning and updating code.

List repos

Clone missing repos

clone-missing.sh

Update repos from a directory in parallel

update-repos.sh

There are more details in parallel_commands readme.

The workflows / scripts defined below assumes that you already have a GITHUB_TOKEN environment variable capable of listing github repositories and that you have the parallel_commands/bin in your PATH.

update-repos.sh is the one that I most use daily.

It does a git pull --rebase and git fetch --prune in all repositories inside a directory in parallel. This can save you and your team lots of time.

Offline studing

PacktPublishing and oreillymedia have lots of quality repos to be used in studies.

Imagine that you want to study kubernetes and clone all repositories from PacktPublishing that contains kubernetes in the name.

Filtering kubernetes from PacktPublishing shows 63 repositories. Cloning 63 repositories by hand would be at least slow.

Here is an nicer way of doing this:

1- Create a list of repos from PacktPublishing

You can use the static file in this gist.

mkdir -p ~/github-orgs/PacktPublishing
cd ~/github-orgs/PacktPublishing
curl -s -L -o PacktPublishing.txt https://gist.githubusercontent.com/joaovitor/e1658abc0946ec9e5528c533f2f502a8/raw/88df9a43c2f9253d686b4ca6eb45bf50c047678b/PacktPublishing.txt

Or get it with the github-print-organization-repos.sh.

This one take a while PacktPublishing has more than 6000 repositories. github-print-organization-repos.sh makes around 600 github api calls to generate the file with the list of repos.

mkdir -p ~/github-orgs/PacktPublishing
cd ~/github-orgs/PacktPublishing
gh_owner=$(basename $(pwd)); github-print-organization-repos.sh ${gh_owner} ${gh_owner}.txt

2- Clone the missing repos that match kubernetes

cd ~/github-orgs/PacktPublishing
gh_owner=$(basename $(pwd)); grep -Ei kubernetes ${gh_owner}.txt | parallel  -j 25 'clone-missing.sh {}; echo job {#} completed {};'

3- Study offline

fd instead of find

João Vitor — Fri, 06 Dec 2019 12:44:46 +0000

I've been using fd instead of find locally.

It describe itself as "A simple, fast and user-friendly alternative to 'find'".

From their readme.

Features

Convenient syntax: fd PATTERN instead of find -iname '*PATTERN*'.
Colorized terminal output (similar to ls).
It's fast (see benchmarks).
Smart case: the search is case-insensitive by default. It switches to case-sensitive if the pattern contains an uppercase character*.
Ignores hidden directories and files, by default.
Ignores patterns from your .gitignore, by default.
Regular expressions.
Unicode-awareness.
The command name is 50% shorter* than find :-).
Parallel command execution with a syntax similar to GNU Parallel.

Try it out.

rg instead of grep

João Vitor — Fri, 06 Dec 2019 12:18:15 +0000

ripgrep aka rg provides great defaults to search inside files.

It automatically skip .git directories and files ignored by git reducing the amount of data being searched.

These are sane defaults to have and can safe you a lot of development time.

Searching for the same word inside the same directory gives me this.

With rg

time rg -li 'pattern'

real    0m17.514s
user    0m15.610s
sys     0m8.822s

With grep

time grep -ilR 'pattern'

real    1m9.520s
user    0m29.768s
sys     0m8.352s

exa instead of ls

João Vitor — Fri, 06 Dec 2019 12:07:08 +0000

exa describe itself as the "A modern version of ‘ls’. https://the.exa.website/".

I'm using exa instead of ls locally.

Here are my aliases making this official.

alias l='exa'
alias la='exa -a'
alias ll='exa -lah'
alias ls='exa --color=auto'

Try it out.

bat instead of cat

João Vitor — Fri, 06 Dec 2019 12:03:42 +0000

I replaced my cat in linux with bat.
I'm using this alias in my shell environment.

alias cat='bat --style=plain'

bat gives me syntax highlight, pagination, search in the file being printed and more.

Give it a try.