DEV Community

Cover image for CLI tools you won't be able to live without 🔧
Alicia Sykes
Alicia Sykes

Posted on • Updated on

CLI tools you won't be able to live without 🔧

As developers, we spend a lot of our time in the terminal. There's a lot of helpful CLI tools, which can make your life in the command line easier, faster and generally more fun.

This post outlines my top 50 must-have CLI tools, which I've come to rely on. If there's anything I'm missing - do let me know in the comments :)

At the end of the article, I've included some scripts to help you automate the installation and updating of these tools on various systems/ distros.

↕️ Contents (click to expand)


thefuck - Auto-correct miss-typed commands

thefuck is one of those utilities you won't be able to live without once you've tried it. Whenever you mis-type a command and get an error, just run fuck and it'll auto-correct it. Use up/down to choose a correction, or just run fuck --yeah to just execute the most likely immediately.


View thefuck on GitHub Author nvbn Written in Python

# MacOS (via Homebrew)
brew install thefuck

# Arch Linux
sudo pacman -S thefuck

# FreeBSD
pkg install thefuck
Enter fullscreen mode Exit fullscreen mode

zoxide - Easy navigation (better cd)

z lets you jump to any directory without needing to remember or specify its full path. It remembers which directories you've visited, so you can jump around quickly - you don't even need to type the full folder name. It also has an interactive selection option, using fzf so you can live-filter directory results


View zoxide on GitHub Author ajeetdsouza Written in Rust

# MacOS (via Homebrew)
brew install zoxide

# Arch Linux
sudo pacman -S zoxide

# Debian / Ubuntu
sudo apt install zoxide

# FreeBSD
pkg install zoxide

# Other (via Rust Creates)
cargo install zoxide --locked
Enter fullscreen mode Exit fullscreen mode

tldr - Community-maintained docs (better man)

tldr is a huge collection of community-maintained man pages. Unlike traditional man pages, they're summarized, contain useful usage examples and nicely colourized for easy reading


View tldr on GitHub Author tldr-pages

# MacOS (via Homebrew)
brew install tldr

# Other (via NPM)
npm install -g tldr
Enter fullscreen mode Exit fullscreen mode

scc - Count lines of code (better cloc)

scc gives you a breakdown of number of lines of code written in each language for a specific directory. It also shows some fun stats, like estimated cost to develop and complexity info. It's incredibly fast, very accurate and has support for a wide range of languages


View scc on GitHub Author boyter Written in Go

# MacOS (via Homebrew)
brew install scc

# Other (via go)
go install
Enter fullscreen mode Exit fullscreen mode

exa - Listing Files (better ls)

exa is a modern Rust-based replacement for the ls command, for listing files. It can display file-type icons, colors, file/folder info and has several output formats - tree, grid or list


View exa on GitHub Author ogham Written in Rust

# MacOS (via Homebrew)
brew install exa

# Arch Linux
sudo pacman -S exa

# Debian / Ubuntu
sudo apt install exa
Enter fullscreen mode Exit fullscreen mode

duf - Disk Usage (better df)

duf is great for showing info about mounted disks and checking free space. It produces a clear and colorful output, and includes options for sorting and customizing results.


View duf on GitHub Author muesli Written in Go

# MacOS (via Homebrew)
brew install duf

# Arch Linux
sudo pacman -S duf

# Debian / Ubuntu
sudo apt install duf

# FreeBSD
pkg install duf
Enter fullscreen mode Exit fullscreen mode

aria2 - Download Utility (better wget)

aria2 is a lightweight, multi-protocol, resuming download utility for HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink, with support for controlling via an RPC interface. It's incredibly feature rich, and has tons of options. There's also ziahamza/webui-aria2 - a nice web interface companion.


View aria2 on GitHub Author aria2 Written in C++

# MacOS (via Homebrew)
brew install aria2

# Arch Linux
sudo pacman -S aria2

# Debian / Ubuntu
sudo apt install aria2
Enter fullscreen mode Exit fullscreen mode

bat - Reading Files (better cat)

bat is a clone of cat with syntax highlighting and git integration. Written in Rust, it's very performant, and has several options for customizing output and theming. There's support for automatic piping and file concatenation


View bat on GitHub Author sharkdp Written in Rust

# MacOS (via Homebrew)
brew install bat

# Arch Linux
sudo pacman -S bat

# Debian / Ubuntu
sudo apt install bat
Enter fullscreen mode Exit fullscreen mode

diff-so-fancy - File Comparisons (better diff)

diff-so-fancy gives you better looking diffs for comparing strings, files, directories and git changes. The change highlighting makes spotting changes much easier, and you can customize the output layout and colors


View diff-so-fancy on GitHub Author so-fancy Written in Perl

# MacOS (via Homebrew)
brew install diff-so-fancy

# Arch Linux
sudo pacman -S diff-so-fancy

# Debian / Ubuntu
sudo apt install diff-so-fancy
Enter fullscreen mode Exit fullscreen mode

entr - Watch for changes

entr lets you run an arbitrary command whenever file changes. You can pass a file, directory, symlink or regex to specify which files it should watch. It's really useful for automatically rebuilding projects, reacting to logs, automated testing, etc. Unlike similar projects, it uses kqueue(2) or inotify(7) to avoid polling, and improve performance


View entr on GitHub Author eradman Written in C

# MacOS (via Homebrew)
brew install entr

# Arch Linux
sudo pacman -S entr

# Debian / Ubuntu
sudo apt install entr
Enter fullscreen mode Exit fullscreen mode

exiftool - Reading + writing metadata

ExifTool is handy utility for reading, writing, stripping and creating meta information for a wide variety of file types. Never accidentally leak your location when sharing a photo again!


View exiftool on GitHub Author exiftool Written in Perl

fdupes - Duplicate file finder

jdupes is used for identifying and/or deleting duplicate files within specified directories. It's useful for freeing up disk space when you've got two or more identical files


View jdupes on GitHub Author jbruchon Written in C

fzf - Fuzzy file finder (better find)

fzf is an extremely powerful, and easy to use fuzzy file finder and filtering tool. It lets you search for a string or pattern across files. fzf also has plugins available for most shells and IDEs, for showing instant results while searching. This post by Alexey Samoshkin highlights some of it's use cases.


View fzf on GitHub Author junegunn Written in Go

# MacOS (via Homebrew)
brew install fzf

# Arch Linux
sudo pacman -S fzf

# Debian / Ubuntu
sudo apt install fzf
Enter fullscreen mode Exit fullscreen mode

hyperfine - Command benchmarking

hyperfine makes it easy to accurately benchmark and compare arbitrary commands or scripts. It takes care of warm-up runs, clearing the cache for accurate results and preventing interference from other programs. It can also export results as raw data and generate charts.


View hyperfine on GitHub Author sharkdp Written in Rust

# MacOS (via Homebrew)
brew install hyperfine

# Arch Linux
sudo pacman -S hyperfine

# Debian / Ubuntu
sudo apt install hyperfine
Enter fullscreen mode Exit fullscreen mode

just - Modern command runner (better make)

just is similar to make but with some nice additions. It let's you group your projects commands together into recopies, which can be easily listed and run. Supports aliases, positional arguments, different shells, dot env integration, string interprulation, and pretty much everything else you could need

View just on GitHub Author casey Written in Rust

# MacOS (via Homebrew)
brew install just

# Arch Linux
sudo pacman -S just

# Debian / Ubuntu
sudo apt install just
Enter fullscreen mode Exit fullscreen mode

jq - JSON processor

jq is like sed, but for JSON - you can use it to slice and filter and map and transform structured data with ease. It can be used to write complex queries to extract or manipulate JSON data. There's also a jq playground that you can use to try it out, or formulate queries with live feedback

View jq on GitHub Author stedolan Written in C

most - Multi-window scroll pager (better less)

most is a pager, for reading through long files or command outputs. most supports multi-windows and has the option to not wrap text

Author Jed Written in S-Lang

procs - Process viewer (better ps)

procs is an easy to navigate process viewer, it has colored highlighting, makes sorting and searching for processes easy, has tree view and updates in real-time


View procs on GitHub Author dalance Written in Rust

rip - Deletion tool (better rm)

rip is a safe, ergonomic and performant deletion tool. It let's you intuitively remove files and directories, and easily restore deleted files


View rip on GitHub Author nivekuil Written in Rust

ripgrep - Search within files (better grep)

ripgrep is a line-oriented search tool that recursively searches the current directory for a regex pattern. It can ignore the contents of .gitignore and skip binary files. It's able to search within compressed archives, or only search specific extension, and understands files using various encoding methods


View ripgrep on GitHub Author BurntSushi Written in Rust

rsync - Fast, incremental file transfer

rsync lets you copy large files locally or to or from remote hosts or external drives. It can be used to keep files across multiple locations synced, and is perfect for creating, updating and restoring backups

View rsync on GitHub Author WayneD Written in C

sd - Find and replace (better sed)

sd is an easy, fast and intuitive find and replace tool, based on string literals. It can be executed on a file, an entire directory, or any piped text


View sd on GitHub Author chmln Written in Rust

tre - Directory hierarchy (better tree)

tre outputs a tree stye list of files for your current or a specified directory, with colors. When running with the -e option, it numbers each item, and creates a temporary alias that you can use to quickly jump to that location


View tre on GitHub Author dduan Written in Rust

xsel - Access the clipboard

xsel let's you read and write to the X Selection clipboard via the command line. It's useful for piping command output to the clipboard, or a copied data into a command

View xsel on GitHub Author kfish Written in C

CLI Monitoring and Performance Apps

bandwhich - Bandwidth utilization monitor

Show bandwidth usage, connection information, outgoing hosts and DNS queries in real-time


View bandwhich on GitHub Author imsnif Written in Rust

ctop - Container metrics and monitoring

Like top, but for monitoring resource usage for running (Docker and runC) containers. It shows real-time CPU, memory and network bandwidth as well as the name, status and ID of each container. There's also a built-in log viewer, and options to manage (stop, start, exec, etc) containers


View ctop on GitHub Author bcicen Written in Go

bpytop - Resource monitoring (better htop)

bpytop is a fast, interactive, visual resource monitor. It shows top running processes, recent CPU, mem, disk and network history. From the interface you can navigate, sort and search - there's also support for custom color themes


View bpytop on GitHub Author aristocratos Written in Python

glances - Resource monitor + web and API

glances is another resource monitor, but with a different feature set. It includes a fully responsive web view, a REST API and historical monitoring. It's easily extendable, and can be integrated with other services


View glances on GitHub Author nicolargo Written in Python

gping - Interactive ping tool (better ping)

gping can run ping tests on multiple hosts, while showing results in real-time graph. It can also be used to monitor execution time, when used with the --cmd flag


View gping on GitHub Author orf Written in Rust

dua-cli - Disk usage analyzer and monitor (better du)

dua-cli let's you interactively view used and available disk space for each mounted drive, and makes freeing up storage easy


View dua-cli on GitHub Author Byron Written in Rust

speedtest-cli - Command line speed test utility

speedtest-cli just runs an internet speed test, via - but straight from the terminal :)


View speedtest-cli on GitHub Author sivel Written in Python

dog - DNS lookup client (better dig)

dog is an easy-to-use DNS lookup client, with support for DoT and DoH, nicely coloured outputs and the option to emit JSON


View dog on GitHub Author ogham Written in Rust

CLI Productivity Apps

Surf the web, play music, check emails, manage calendars, read the news and more, all without leaving the terminal!

browsh - CLI web browser

browsh is a fully interactive, real-time, and modern text-based browser rendered to TTYs and browsers. It supports both mouse and keyboard navigation, and is surprisingly feature rich for a purely terminal based application. It also mitigates battery drain issues that plague modern browsers, and with support for MoSH, you can experience faster load times due to reduced bandwidth


View browsh on GitHub Author browsh-org Written in JavaScript

buku - Bookmark manager

buku is a terminal-based bookmark manager, with tons of configuration, storage and usage options. There's also an optional web UI and browser plugin, for accessing your bookmarks outside of the terminal


View buku on GitHub Author jarun Written in Python

cmus - Music browser / player

cmus is terminal music player, controlled with keyboard shortcuts. It has support for a wide range of audio formats and codecs, and allows organising tracks into playlists and applying playback settings


View cmus on GitHub Author cmus Written in C

cointop - Track crypto prices

cointop show current crypto prices, and track the price history of your portfolio. Supports price alerts, historical charts, currency conversion, fuzzy searching, and much more. You can try the demo via the web at, or by running ssh


View cointop on GitHub Author cointop-sh Written in Go

ddgr - Search the web from the terminal

ddgr is like googler, but for DuckDuckGo. It's fast, clean and easy, with support for instant answers, search completion, search bangs, and advanced search. It respects your privacy by default, and also has HTTPS proxy support, and works with Tor


View ddgr on GitHub Author jarun Written in Python

micro - Code editor (better nano)

micro is an easy to use, fast and extendable code editor with mouse support. Since it's packaged into a single binary, installation is as simple as curl | bash


View micro on GitHub Author zyedidia Written in Go

khal - Calendar client

khal is a terminal calendar app, which shows upcoming events, month and agenda views. You can sync it with any CalDAV calendar, and add, edit and remove events directly


View khal on GitHub Author pimutils Written in Python

mutt - Email client

mut is a classic, a terminal based mail client for sending, reading and managing emails. It supports all mainstream email protocols and mailbox formats, allows for attachments, BCC/CC, threads, mailing lists and delivery status notifications


View mutt on GitHub Author muttmua Written in C

newsboat - RSS / ATOM news reader

newsboat is an RSS feed reader and aggregator, for reading the news, blogs and following updates directly from the terminal


View newsboat on GitHub Author newsboat Written in C++

rclone - Manage cloud storage

rclone is a handy utility for syncing files and folders to various cloud storage providers. It can be either invoked directly from the command line, or easily integrated into a script as a replacement for heavy desktop sync apps

View rclone on GitHub Author rclone Written in Go

taskwarrior - Todo + task management

task is a CLI task management/ todo app. It's both simple and unobtrusive, but also incredibly powerful and scalable, with advanced organisation and query features built in. There's also a lot (700+!) of extra plugins for extending it's functionality and integrating with third-party services


View taskwarrior on GitHub Author GothenburgBitFactory Written in C++

tuir - Terminal UI for Reddit

tuir is a great one if you want to look like you're working, while actually browsing Reddit! It's got intuitive keybindings, custom themes, and can render images and multi-media content too. There's also haxor for hacker news


View tuir on GitLab Author ajak Written in Python

CLI Dev Suits

httpie - HTTP / API testing testing client

httpie is a HTTP client, for testing, debugging and using APIs. It supports everything you'd expect - HTTPS, proxies, authentication, custom headers, persistent sessions, JSON parsing. Usage is simple with an expressive syntax and colourized output. Like other HTTP clients (Postman, Hopscotch, Insomnia, etc) HTTPie also includes a web UI


View httpie on GitHub Author httpie Written in Python

lazydocker - Full Docker management app

lazydocker is a Docker management app, that lets you view all containers and images, manage their state, read logs, check resource usage, restart/ rebuild, analyse layers, prune unused containers, images and volumes, and so much more. It saves you from needing remember, type and chain multiple Docker commands.


View lazydocker on GitHub Author jesseduffield Written in Go

lazygit - Full Git management app

lazygit is a visual git client, on the command line. Easily add, commit and puch files, resolve conflicts, compare diffs, manage logs, and do complex operations like squashes and rewinds. There's keybindings for everything, colors, and it's easily configurable and extenable


View lazygit on GitHub Author jesseduffield Written in Go

kdash - Kubernetes dashboard app

kdash is an all-in-one Kubernetes management tool. View node metrics, watch resources, stream container logs, analyse contexts and manage nodes, pods and namespaces

View kdash on GitHub Author kdash-rs Written in Rust

gdp-dashboard - Visual GDP debugger

gdp-dashboard adds a visual element to the GNU Debugger, for debugging C and C++ programs. Easily analyse memory, step through breakpoints, and view registers


View gdb-dashboard on GitHub Author cyrus-and Written in Python

CLI External Sercvices

ngrok - Reverse proxy for sharing localhost

ngrok safely* exposes your localhost to the internet behind a unique URL. This lets you share what you're working on with you're remote colleagues, in real-time. Usage is very simple, but it's also got a lot of advanced features for things like authentication, webhooks, firewalls, traffic inspection, custom/ wildcard domains and much more


View ngrok on GitHub Author inconshreveable Written in Go

tmate - Share a terminal session via internet

tmate let's you instantly share a live terminal session with someone elsewhere in the world. It works across different systems, supports access control/ auth, can be self-hosted, and has all the features of Tmux

View tmate on GitHub Author tmate-io Written in C

asciinema - Recording + sharing terminal sessions

asciinema is very useful for easily recording, sharing and embedding a terminal session. Great to showcase something you've built, or to show the command-line steps for a tutorial. Unlike screenrecording videos, the user can copy-paste the content, change the theme on the fly and control playback

View asciinema on GitHub Author asciinema Written in Python

navi - Interactive cheat sheet

navi allows you to browse through cheatsheets and execute commands. Suggested values for arguments are dynamically displayed in a list. Type less, reduce mistakes and save yourself from having to memorise thousands of commands. It integrates with tldr and to get content, but you can also import other cheatsheets, or even write your own

View navi on GitHub Author denisidoro Written in Rust - Fast file sharing

transfer makes uploading and sharing files really easy, directly from the command line. It's free, supports encryption, gives you a unique URL, and can also be self-hosted.
I've written a Bash helper function to make usage a bit easier, you can find it here or try it out by running bash <(curl -L -s


View on GitHub Author dutchcoders Written in Go

surge - Deploy a site in seconds

surge is a free static hosting provider, that you can deploy to directly from the terminal in a single command, just run surge from within your dist directory! It supports custom domains, auto SSL certs, pushState support, cross-origin resource support - and it's free!

surge-sh-example-usage - Check the weather is a service that displays the weather in a format that's digestible in the command line. Just run curl or curl to try it out. There's URL parameters to customise what data is returned, as well as the format


View on GitHub Author chubin Written in Python


cowsay - Have an ASCII cow say your message

cowsay is a configurable talking cow. It's based off the original by Tony Monroe


View cowsay on GitHub Author piuccio Written in JavaScript

figlet - Output text as big ASCII art text

figlet outputs text as ASCII art


View figlet on GitHub Author cmatsuoka Written in C

lolcat - Make console output raibow colored

lolcat makes any text passed to it rainbow coloured


View lolcat on GitHub Author busyloop Written in Ruby

neofetch - Show system data and ditstro info

neofetch prints distro and system info (so you can flex that you use Arch btw on r/unixporn)


View neofetch on GitHub Author dylanaraps Written in Bash

As an example, I'm using cowsay, figlet, lolcat and neofetch to create a custom time-based MOTD shown to the user when they first log in. It greets them by their name, shows server info and time, date, weather and IP. Here's the source code.


Installations and Management

Most of us have a core set of CLI apps and utils that we rely upon. Setting up a new machine, and individually installing each program would get tiresome very quickly. So the task of installing and updating your terminal apps is the perfect candidate for automation. Here are some example scripts I've written, which can be easily dropped into your dotfiles or just run independently to ensure you're never missing an app.

For MacOS users, the easiest method is using Homebrew. Just create a Brewfile (with touch ~/.Brewfile), then list each of your apps, and run brew bundle. You can keep your package list backed up, by putting it in a Git repo. Here's an example one, to get you started:

On Linux, you usually want to use the native package manager (e.g. pacman, apt). As an example, here's a script to install the above apps on Arch Linux systems

Desktop apps on Linux can be managed in a similar way, via Flatpak. Again, here's an example script :)


... So that's it - a list of handy CLI apps, and a method for installing and keeping them up-to-date across your systems.

Hopefully some of these will be useful to some of you :)

I'd love to hear what you're favourite CLI apps are, let me know in the comments below!

Additional Info

What wasn't included

  • This list doesn't include the basics, like Vim, Tmux, Ranger, ZSH, Git, etc - which you're likely already using
  • I've also not included anything too niche, or only specific to a small number of users
  • Nothing system-specific, or that isn't cross-platform (Linux/ Unix, MacOS) is included
  • And I've not included apps which relate to the terminal, but are not CLI apps (like terminal emulators)
  • For most of the projects listed, there's a plethora of alternatives that achieve similar things, for brevity those also weren't included


Huge kudos to the authors, and communities behind each of these apps. Without them and their hard work, our life in the command line would be much less awesome. Where possible, I've tried to credit the authors, but if I've missed any - let me know below, and I'll push an update


What have I missed? I'd love to hear your favourite CLI apps, especially if there's something awesome that I've missed!

I'd also like to hear your thoughts and suggestions - I'm always looking to improve :)


Here's the script that I made to generate the author, language and GitHub star badges:

Repo Badge Generator

GitHub logo Lissy93 / repo-badge-maker

🛡️ Very, very quick script to generate repo badges for blog posts

🛡️ Repo Badge Generator

Quick script to generate embedable badges for a repository


I made this to quickly generate the project links, for this post: CLI tools you won't be able to live without 🔧

Just a simple script to quickly generate embedable markdown badges to show repository info.
Data is fetched from the GitHub API, and badges are served up using the API.
Includes a badge for + link to: the author, the repo + star count, and the language used.
Paste in the URL to a repo, hit 'Generate', and results will be copied to your clipboard.


It's just a static HTML page, so serve it with any web server, CDN or static host.
There's a demo hosted on GH Pages, at:

Report an Issue

Found something that's not working? Open an issue, and describe the problem, steps to reproduce alond with…

View on GitHub

View repo-badge-maker on GitHub Author Lissy93

Find More

If you were enjoying this, I recommend also checking out:

If you're new to the command line, then The Art of Command Line by Joshua Levy is an excellent resource, as is the Bash Guide by Adnan Ahmed.

And if you are looking for inspiration, you'll love r/unixporn ⚡

If you like this kind of stuff,
consider following for more :)
Follow Lissy93 on GitHubFollow Lissy_Sykes on Twitter

Top comments (106)

gbhorwood profile image
grant horwood

this is awesome and i fully plan to go over a bunch of these this weekend!

i would also suggest toilet for this list. figlet foo | toilet -f term --gay gives you a nice oh-my-zsh-style headline.

mariamarsh profile image
Maria 🍦 Marshmallow

Very comprehensive list, thank you so much, Alicia! Here are a few more command line tools that you might find useful:

awscli: is a command line interface for Amazon Web Services. It provides commands for a variety of AWS services, including EC2, S3, and RDS.

mycli: is a command line interface for MySQL that provides auto-completion and syntax highlighting.

ncdu: a disk usage analyzer that can help you quickly identify which files and directories are taking up the most space on your file system.

atinypixel profile image
Aziz Kaukawala • Edited

Awesome! 😍

Would like to add:

  • Micro: a modern and intuitive terminal-based text editor (better nano)! 😉
  • Forx: a command line tool for checking exchange rates between currencies, both crypto and fiat.
lissy93 profile image
Alicia Sykes

Micro is awesome - thanks for sharing!! I've added it to the list :)
I'm usually a Neovim user, but I really enjoyed trying out Micro, and it looks like it'd be ideal for simple tasks.

atinypixel profile image
Aziz Kaukawala

Indeed, micro makes editing config files in terminal a breeze. ✨

Thank you!

moraym profile image
Moray Macdonald

Great list! Looks like I'm going to spend most of today installing things instead of working...

My contribution to the list would be autojump. It learns all the directories you frequently cd into so you can quickly jump to them later. e.g. j Doc will take me straight to /home/moray/Documents/ from wherever I was.

proteusiq profile image
Prayson Wilfred Daniel

Nice. There is z on the list that does the same 😂

raibtoffoletto profile image
Raí B. Toffoletto

Awesome list!! 🔥 There are many I didn't know about.

I'd just add tmux to the mix. I can't live without it.

Also for the fun, cmatrix is a nice effect to have in your terminal when you are afk... specially piped to lolcat 😅

sgtnasty profile image
Ronaldo Nascimento

This list is actually useful, has most of the tools I use plus a few new ones. It explains about the tool, has screenshots and direct link to source. Very cool indeed, I would like to add the bitwarden command line clint with fzf are a great pair.

cubiclesocial profile image

This is an interesting list. I'm not usually a fan of listicles (article + list = listicle) since they are rarely carefully curated and someone basically throws the list together to fulfill some quota. You obviously spent a great deal of time putting this together, including figuring out what tool they replace. Very nice.

Two observations/thoughts:

  1. speedtest-cli is fine as long as you understand that the results might not be realistic. Ookla, the organization behind, may have the largest selection of donated speed test servers globally, but ISPs are fully aware of their existence. Some ISPs even go out of their way to lie to their customers such as remove bandwidth caps for connections to the known, public list of Ookla speed test servers so that you get the advertised speeds you are paying for in just that one specific use-case. Ookla also sells cobranded versions of their speed testing widget to the ISPs that cater to each ISP's whims/desires, which I think is pretty sus. That said, something that does speed testing is better than nothing. Speed testing tools are also useful for the datacenter. For example, DigitalOcean advertises a minimum 1Gps link speed but I've seen burst rates up to 3Gps even for their cheapest VPS servers.

  2. From the current librsync GitHub repo:

The rsync algorithm is different from most differencing algorithms because it does not require the presence of the two files to calculate the delta. Instead, it requires a set of checksums of each block of one file, which together form a signature for that file. Blocks at any position in the other file which have the same checksum are likely to be identical, and whatever remains is the difference.

The bolded portion is the important phrase that makes both librsync and anything built on it likely to introduce data corruption over time. There is no guarantee that the same checksum of a portion of a file means that the stored data is actually identical. I have personally witnessed significant data corruption due to the underlying algorithm used in librsync. rsync and librsync are, in my experience, unsuitable for the purposes that they claim to be suitable for.

lissy93 profile image
Alicia Sykes

Thanks @cubiclesocial - I wasn't actually aware of that librsync issue, I've not experienced corruption myself, but it's very good to be aware of. On the topic, is there anything similar that you recommend for incremental file transfers?
I have tried unison in the past, but since it works in a similar way, I'd imagine it would also be susceptible to this issue, especially when dealing with large data sets. For backups specifically, possibly restic could be an option?

cubiclesocial profile image

Possibly. For backups, I use my own 3rd generation software called Cloud Backup. I backup my servers via a Remoted API Server + Cloud Storage Server instance. Been running that combo for years with only the occasional hiccup with network connectivity that usually clears itself up by re-running the backup automatically until it succeeds. Whenever I've needed to retrieve something, it's there and ready for use and can be pulled back onto the system within a few seconds from the backup. The setup handles multiple GB of transfers daily across multiple systems. I even use Cloud Backup when I need to migrate between *NIX systems because it faithfully preserves timestamps, owner, group, privileges, symlinks, etc.

Cloud Backup, Cloud Storage Server, and Remoted API Server were written before I came up with "question-answer" CLI interfaces. As a result, initial setup is really awkward and overly complex. So I can't really recommend using what I use for that reason. I'll eventually get around to fixing that problem.

I also have a very large drive attached to a cheap-o mini PC that is firewalled onto its own VLAN that I pay for a single user license of Backblaze (approx $7/mo). I push backup data to the Cloud Storage Server instance running on that system a couple of different ways. The Backblaze client software then picks up everything on the drive and puts it in the cloud. Basically, this setup gives me unlimited online backup storage for all my computer systems (instead of paying for cloud storage for each system). The key to saving money with Backblaze is to dump everything onto one system with at least one large external attached drive over USB. In my case, by using Cloud Backup, Backblaze just gets compressed, encrypted data blobs but someone could go a lot simpler than my setup and just dump straight files onto a similar setup for cheap, "unlimited" cloud storage (unlimited = as much locally attached storage as can be afforded). I'm a penny pinching fiend. I wish Backblaze had a Linux client but I suspect they don't because they don't want people to abuse their system any more than it currently gets abused.

The DIY alternative to paying for online cloud storage is to setup a backup system with Cloud Storage Server at a friend's house and use Remoted API Server on any public facing VPS to allow their home IP to freely roam and also to not have to worry about router/firewall rules. Then point Cloud Backup at the running Remoted API Server. Once setup, the bonus with this approach is that restoring everything from the backup takes a fraction of the time it would take over the Internet (especially when restoring multi-TB of data): Drive to the friend's house, pick up the equipment, drive home, adjust the configs to point at the local network, restore everything locally, revert the configs, drive back, put the equipment back in place. Fully restored in mere hours instead of days or weeks. Buy the friend a pizza or coffee to celebrate.

ahmed_onour profile image
Ahmed Onour

This list is the best list I ever found on tools 🔥

cadnav profile image
</cadnav> • Edited

I literally just created this account so I could comment on on your munificence ; and I literally had to Google synonyms of generosity to find the word munificence. Seriously, this is mind blowing - more so because I avidly search for such tools but have never heard of several of these. Anyway, thank you SO VERY MUCH- I’ll see you guys around soon…we are all now pupils of the chosen one, Alicia

lissy93 profile image
Alicia Sykes

Thank you so much @ ☺️☺️

afolarin profile image
Amos Folarin

You should definitely add fd (i.e. fd-find, fdfind on Ubuntu) to the list -- a nicer syntaxed find command with some nifty extras.

billcosta profile image
Bill Costa • Edited

Lots of great tools. Thanks!

There were a few apps on your list that I had already discovered, installed, tried a few times, and forgot about. It was good to be reminded so I can look at them again. It can take a bit of effort to make a new tool part of your muscle memory, no matter how useful.

For me one indispensable tool, that now finally flows off my finger tips, is ack -- a grep-like source code search tool. Most of the time I can just type in a snippet of what I'm looking for, and it quickly finds that bit of code I wrote for some other project, but I just can't remember which one. The defaults settings are so well chosen, I rarely need to provide any options.

praful profile image

Thank you for writing this. You may have done something unique on the internet: written an article that more than lives up to its title!

More power to your elbow :)

urbanisierung profile image

Thanks for this list! fzf is great!

wadecodez profile image
Wade Zimmerman

jq is super underrated

kbilleter profile image

gron is pretty useful too -- especially if you don't want to work through the structure (eg. gron json | grep 'thing' | gron -u)

tamirazrab profile image

List made me to go back to linux on my main machine ditching windows and all the games. I'd surely enjoy the perks I get in return. TY! 😄

lissy93 profile image
Alicia Sykes

If you do prefer Windows, then all these utils should work in WSL :)

A few weeks ago I used yuk7/ArchWSL on Terminal, and the experience was surprisingly pleasant. With the Ubuntu image, you can also run GUI apps, which is pretty neat.

tamirazrab profile image

Thanks I was planning to dual boot, but WSL might save me some time surely.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.