DEV Community: Kevin Cui

Use Burp Suite Community Edition Like A Pro

Kevin Cui — Sun, 01 May 2022 11:44:59 +0000

You should have noticed the pun in this post title 😉. That's right, some powerful Burp Suite extensions can make the free community edition as good as the Burp Pro version. Knowing how to use those extensions will make you feel like a pro.

Turbo Intruder

Let's start with Turbo Intruder. It's one of the most (if not the most) popular Burp extensions. Basically, anyone who knew Burp extensions should have heard the name of Turbo Intruder.

The Burp Suite community edition comes with a "lite" version of Burp Intruder. The first impression of this community edition Intruder is: It has really low RPS (Request Per Second). The time you waited for finishing a 1000-request brute-forcing attack can make you grow out some gray hair, slow and anxious. Meanwhile, the Burp Pro version provides a decent speed Burp Intruder, which is suitable for daily usage.

However, Turbo Intruder is a beast on another level. It's fast and furious. To be honest, I never hit once its full speed and I don't know how fast it can truly go...

Another highlight of the Turbo Intruder is the customization. It supports Python scripts, which makes it flexible. You could modify any parameters as you want or add additional steps in the script to fit your needs. Personally, I find it's pretty handy for testing race conditions.

Turbo Intruder will let you forget about the Burp Intruder in the Burp Pro version.

Logger++

After a session of testing, you may already have collected hundreds of requests and responses in Logger or HTTP history. Sometimes, you need to search for a specific request with a certain parameter in the request URI path or a specific request with a certain value in the response... It's clearly not an option to go through all the requests and check them one by one. Missing the search function is a big deal in the Burp Suite community edition.

Logger++ provides enhanced search functionality. Among all the features in Logger++, I find its grep tool is the most useful feature. As the name suggested, you could "grep" logs with matched patterns:

If you are using Logger++ already, I'm pretty sure you won't open Burp Logger anymore, and you won't mind using the search functionality in the Burp Pro version either.

Interactsh Collaborator

The Burp Collaborator is a killer tool in the Burp Pro version. It's valuable for testing OOB (Out-of-band), or generating temporary email addresses...

Don't be sad yet if you cannot use Collaborator in the community edition. Because if you know interactsh-collaborator extension, you won't miss a thing.

Interactsh Collaborator provides a good alternative, based on interactsh-client. It offers the same functionalities as the Burp Collaborator does:

LazyCSRF

Last but not least, another feature in Burp Pro you may need is "Generate CSRF PoC". LazyCSRF is an extension to fulfill this purpose with some additional enhancements:

The purpose of this post is not to demotivate you from purchasing the Burp Pro license. Contrarily, if you love Burp Suite and you're able to pay for the Pro license, then please own it properly like a pro.

Here is a list of the mentioned extensions in the post and their GitHub repositories:

Turbo Intruder: https://github.com/PortSwigger/turbo-intruder
Logger++: https://github.com/nccgroup/LoggerPlusPlus
Interactsh-collaborator: https://github.com/wdahlenburg/interactsh-collaborator
LazyCSRF: https://github.com/tkmru/lazyCSRF

Three jq Tips You May Need to Know

Kevin Cui — Tue, 03 Nov 2020 13:50:55 +0000

jq is famously known as a powerful command-line JSON parser. Besides performing as a parser, jq has many other handy usages as well. Here, I'd like to share 3 jq tips you may need to know.

JSON Prettier

When we curl API to fetch JSON response with raw data. It certainly looks messy:

$ curl https://api.spacexdata.com/v4/launches/latest
{"fairings":{"reused":false,"recovery_attempt":true,"recovered":null,"ships":["5ea6ed2e080df4000697c907","5ea6ed2e080df4000697c908"]},..."id":"5ef6a2e70059c33cee4a8293"}

Now let's toss raw data into jq and make it look prettier:

$ curl https://api.spacexdata.com/v4/launches/latest | jq
{
  "fairings": {
    "reused": false,
    "recovery_attempt": true,
    "recovered": null,
    "ships": [
      "5ea6ed2e080df4000697c907",
      "5ea6ed2e080df4000697c908"
    ]
  },
  ...
  "id": "5ef6a2e70059c33cee4a8293"
}

In the opposite way, when we got some pretty-printed JSON data and want to concatenate them to a one-line string, using jq -c:

$ curl https://api.spacexdata.com/v4/launches/latest | jq > example.json

$ cat example.json
{
  "fairings": {
    "reused": false,
    "recovery_attempt": true,
    "recovered": null,
    "ships": [
      "5ea6ed2e080df4000697c907",
      "5ea6ed2e080df4000697c908"
    ]
  },
  ...
  "id": "5ef6a2e70059c33cee4a8293"
}

$ cat example.json | jq -c
{"fairings":{"reused":false,"recovery_attempt":true,"recovered":null,"ships":["5ea6ed2e080df4000697c907","5ea6ed2e080df4000697c908"]},..."id":"5ef6a2e70059c33cee4a8293"}

JSON Validator

jq can be served as a simple JSON validator, for instance:

$ echo '{"fairings":{"reused":false,, "recovery_attempt":true}}' | jq
parse error: Expected value before ',' at line 1, column 29

Because its output doesn't give much information to pinpoint directly the error in an obvious way, it's better to use a proper validator for some complex cases. But for some quick checks to validate JSON data, jq can be considered as a suitable choice.

JSON Pathfinder

Analyzing JSON files and finding the correct path syntax to a certain value sometimes can be troublesome. Many times, I was punched hard by the frustration of trial and error. Until one day, this magic jq command line brings me the light: jq -r 'paths(scalars) as $p | "." + ([([$p[] | tostring] | join(".")), (getpath($p) | tojson)] | join(": "))'.

Let's use the same curl data from the previous example, in this case, I want to get the path syntax of ships ids:

$ curl https://api.spacexdata.com/v4/launches/latest \
| jq -r 'paths(scalars) as $p | "." + ([([$p[] | tostring] | join(".")), (getpath($p) | tojson)] | join(": "))' \
| grep '.ships'
.fairings.ships.0: "5ea6ed2e080df4000697c907"
.fairings.ships.1: "5ea6ed2e080df4000697c908"
.ships.0: "5ea6ed2f080df4000697c910"
.ships.1: "5ea6ed2f080df4000697c90b"
.ships.2: "5ea6ed2e080df4000697c907"
.ships.3: "5ea6ed2e080df4000697c908"

From the output, now I know the right path syntax is .ships[].

As you can see from this example, this magic jq command line will print each path and value pair from the input JSON. Combing with grep command, it's sufficient to be used as a handy JSON pathfinder.

If you do know other hot jq tips, do not hesitate to share them with me.

Git Alias -- Did You Git It?

Kevin Cui — Sat, 04 Jul 2020 07:02:36 +0000

Git alias is powerful and yet simple to set up. I'd like to share my journey in configuring git alias for my personal use. Now let's dig it and git it!

Day 1: Why git alias?

I was using Tig for some time. It's good but not great. The problems I got during the time:

It takes time to memorize the default key bindings. Since I'm using Colemak keyboard layout, obviously, the default navigation key bindings need to be adjusted accordingly. Therefore, it takes me more time to customize them.
Tig doesn't support customized diff formatter, like my current favorite delta. An open issue from 2016 gives me the fearful feeling that Tig wouldn't officially support this feature in the foreseeable future.
Switching between different views in Tig makes the workflow complex and slow. For example, to add an untracked file: run tig, press s to switch to status view, move up and down to select an untracked file, press u to add it. Well, in this case, why using Tig instead of simply typing git add <untracked_file>?

At some point, I decided to drop Tig and tried to search for other tools to assist my git workflow.

Don't get me wrong. Tig is a decent tool. However, my needs are just picky and I'm just too lazy 🤷.

After trying this and that, surprisingly, I find out that git alias is one ideal solution:

Git alias is supported by default, no need to install any 3rd-party dependencies.
Git alias is relatively easy to customize. Knowing git command and put an alias for it, that's all about.
Git alias is illegally powerful once you know how to use ! to run the external command! Yes, its existence is a crime.

Day 2: start with [alias]

The official way to add git alias is to edit ~/.gitconfig file:

Open ~/.gitconfig
Add [alias] section
Put a bunch of aliases
Save and close file, done!

After this change, my alias setup looks like this:

[alias]
  a = add
  c = commit
  ca = commit --amend
  ci = commit -m
  cm = commit -m
  co = checkout
  di = diff
  p = push origin
  pl = pull
  rsthrd = reset --hard origin/master
  s = status

Those aliases cover all my basic needs of git workflow. For example, to add an untracked file, now I can type: git a <untracked_file>.

By using one letter alias for most commonly used git commands saves my time and gives me pleasures. Yeah, you know that typing fewer letters feels amazing!

Day 3: go external !

As you may know, fzf is an extremely powerful tool (yep, it should be illegal as well). Imagine that git alias x fzf, what a dream team-up!

By using ! symbol can make this dream come true. ! allows running external commands with the alias, including git command itself. So many possibilities can be achieved by combining external commands.

For example, I add an alias ac, known as add modified files, to add modified file:

[alias]
  ...
  ac = ! git add $(git diff --name-only --diff-filter=M | fzf -m --bind "ctrl-a:select-all")

Video: Link

The steps in this video are:

git s to show unstaged changes
git ac to list modified files in fzf, ctrl-a to select all and enter to add them
git s to show the result that all modified files are added and staged

Day 4: go smooth

To smooth git workflow, I tweak fzf key binding to configure command execution when selection is made in fzf.

For example, I create an alias ds, known as diff staged files, to show changes in staged files and after pressing enter, it goes to commit process directly:

[alias]
  ...
  ds = ! git diff --name-only --cached | fzf --preview "git diff --cached -- {} | delta" --bind "enter:abort+execute(git commit)"

Video: Link

The steps in this video are:

git ds to show all staged files
Check changes by switching files listed in fzf preview screen
Press enter to go to commit
Type commit message and save it

As you may also notice in the ds alias, I'm now able to use delta to format git diff in fzf preview screen, by piping diff to it.

Until now, all the main problems I have with Tig are solved by git alias. I'd be happier than ever.

Day 5: go dynamic

For some reason, I'd like to pass the file name as an argument to a git command bound with an alias.

For example, I'd like to see certain change in a file, when and who made it, and all related changes in the same commit. It involves git blame command with the file name as an argument. Without doubt, it's possible to accomplish it with git alias by using sh -c to pass arguments:

[alias]
  ...
  b = ! sh -c \"git blame $1 | fzf --bind 'enter:abort+execute(git diff-tree -p {1} | delta)'\"

Video: Link

⚠️ Pay attention to escape single quotes ' or/and double quotes " inside sh -c if needed.

An alternative and more complex method is to bind a function, like ! f() {...}; f. Once you dig deeper in this way, you will certainly find the answers for all your needs.

Day 6: go flexible

Pursuing the ambition of customizing as many aliases as possible to perfectly suit my needs, the aliases are getting complicated with many duplicated and redundant codes. It's hard to maintain in the configuration file like this:

[alias]
  ac = ! git add $(git diff --name-only --diff-filter=M | fzf -0 -m --reverse --height=40% --bind tab:down,btab:up,ctrl-j:up,ctrl-k:down,change:top,alt-space:toggle,ctrl-a:select-all,ctrl-u:deselect-all --cycle)
  b = ! sh -c \"git blame $1 | fzf -0 --ansi --bind tab:down,btab:up,ctrl-j:up,ctrl-k:down,change:top,alt-space:toggle,ctrl-a:select-all,ctrl-u:deselect-all --bind 'enter:abort+execute(git diff-tree -p {1} | delta --theme=iceberg)' --cycle\"
  ds = ! git diff --name-only --cached | fzf -0 --ansi --preview-window=right:80:wrap --preview 'git diff --cached -- {} | delta --theme=iceberg' --bind tab:down,btab:up,ctrl-j:up,ctrl-k:down,change:top,alt-space:toggle,ctrl-a:select-all,ctrl-u:deselect-all --bind alt-k:preview-down,alt-j:preview-up,alt-h:preview-page-down,alt-l:preview-page-up --bind 'enter:abort+execute(git commit)' --cycle

For instance, if I want to update fzf preview screen key bindings for all aliases, I have to edit all aliases one by one. It's a bummer.

Luckily, git alias can be edited via command line:

~$ git config --global alias.<alias_name> <command>

This creates the possibility to let Bash script handle flexibly the alias modifications. I can define some variables for the common parts in the aliases, which permits that the modifications in one location will affect all aliases at once.

Here is a sample of Bash script:

#!/usr/bin/env bash
# remove all aliases
git config --global --remove-section alias

# set variables
FZF_KEYBINDING="tab:down,btab:up,ctrl-j:up,ctrl-k:down,change:top,alt-space:toggle,ctrl-a:select-all,ctrl-u:deselect-all"
FZF_PREVIEW_KEYBINDING="alt-k:preview-down,alt-j:preview-up,alt-h:preview-page-down,alt-l:preview-page-up"
DIFF_CMD="delta --theme=iceberg"
GIT_DIFF_NAME="git diff --name-only"
GIT_DIFF_NAME_CACHED="$GIT_DIFF_NAME --cached"
GIT_DIFF_TREE="git diff-tree -p {1}"

# define alias
git config --global alias.ac "! git add \$($GIT_DIFF_NAME --diff-filter=M | fzf -0 -m --reverse --height=40% --bind $FZF_KEYBINDING --cycle)"
git config --global alias.b '! sh -c "git blame $1 | fzf -0 --ansi --bind '"$FZF_KEYBINDING"' --bind '\''enter:abort+execute('"$GIT_DIFF_TREE"' | '"$DIFF_CMD"')'\'' --cycle"'
git config --global alias.ds "! $GIT_DIFF_NAME_CACHED | fzf -0 --ansi --preview-window=right:80:wrap --preview 'git diff --cached -- {} | $DIFF_CMD' --bind $FZF_KEYBINDING --bind $FZF_PREVIEW_KEYBINDING --bind 'enter:abort+execute(git commit)' --cycle"

⚠️ Pay attention to escape $ symbol by \$, in case $ symbol indicates the variable used in alias.

In the example above, if I want to add more key bindings for fzf preview screen, I'll modify FZF_PREVIEW_KEYBINDING then run the Bash script. All aliases will be updated accordingly and beautifully.

For more details and examples, go checkout addalias.sh.

Day 7: go extreme

After 6 days hands-on of git alias configuration, it's time to lay down holding a beer and enjoy the efficiency brought by git alias? Wait a minute here, I still have some small problems with the current setup:

Certain alias is conflicted with default git command. For example, git am is already used by git-am, and it's impossible to overwrite git command by git alias.
Typing 3-letter git to initial git command every time is fatigue.

It's time to push the limit and go extreme. To solve those problems, the idea is to create a function in ~/.zshrc, like:

...
g () {
    if [[ "$1" == "am" ]]; then
        git ac
    else
        git "$@"
    fi
}
...

Now I'm able to use am as add modified files, way easier to remember than ac. If on a rare occasion I want to use git-am, I'll just simply type git am.

Meanwhile, using g instead of git reduces the time of typing 2 letters. What a pleasure for a lazy person like me!

Sum-up

Git alias is amazing.
Git alias with ! to execute external command is powerful.
Git alias combined with other commands like fzf and sh -c is matchless.
Git alias created by Bash script is flexible.
Shell alias/function gives wings to git alias.

Did you git alias as well? What's your favorite git alias? Share it with me and let me learn from you.

Twenty Months With Colemak

Kevin Cui — Fri, 19 Apr 2019 08:51:23 +0000

Time truly flies. It has been 20 months since I switched from QWERTY to Colemak keyboard layout. Am I now able to type on the Colemak keyboard as fast as the time flies? Certainly not. My typing speed 20 months ago was 75 WPM on average with QWERTY, now it's 85 WPM on average with Colemak. I can feel the improvement and that's why I never look back to QWERTY again.

Besides things mentioned in One Week with Colemak, are there somethings special happened during these 20 months?

Remap Caps Lock as...?

Caps Lock key is a huge waste on the standard keyboard. It locates in such important position but with less functional benefit. Some remap it as Escape key; some remap it as BackSpace key...

My favorite remapping solution is: Press Caps Lock once, it sends "Escape" key event; Hold Caps Lock, it sends "Control" key event. So, one Caps Lock key can act as Escape and Control combined together.

How to achieve this key binding on Linux?

Act as Control when press and hold: HHKB keyboard by default puts Control key on the usual Caps Lock position, which is great for me, so no tweak is needed. For other keyboards, I usually use xmodmap to map Caps Lock key as Left Control key.
Act as Escape when single press: xcape can configure one key to act as other key when pressed and released:

xcape -t 300 -e 'Control_L=Escape'

How to achieve this key binding on macOS?

Karabiner is a great tool to archive the same key binding conception on macOS. Karabiner supports customized json configuration file. Here is my configuration file.

...
"from": {
    "key_code": "caps_lock",
    "modifiers": {
        "optional": [ "any" ]
    }
},
"to": [{
    "key_code": "right_control"
}],
"to_if_alone": [{
    "key_code": "escape"
}],
...

Some small troubles in the real world

When you use some uncommon tools in the real world, you know you have to prepare some small troubles that may come to you every day:

QWERTY is literally everywhere: Accept the reality and enforce the muscle memory of QWERTY layout as well. Otherwise, like me, I'm extremely clumsy when typing on QWERTY.
Pair programming or testing on my machine: Remember to switch keyboard and key binding settings on the fly, depending on the person.
"This guy is a weirdo.": So what?

Overall, I'm super happy with Colemak for now. I have no plan to learn another keyboard layout. I guess I will stick with Colemak for years. Maybe after some years, I'll write another recap post about Colemak. I shall call it "Twenty Years With Colemak". 😉

Build A Killer Note-taking System

Kevin Cui — Sun, 07 Apr 2019 15:26:05 +0000

The proverb says, "The palest ink is better than the best memory". Note-taking is certainly an essential part of knowledge learning, information collecting and retaining. In the fast-changing digital era, pen, ink and paper may not be able to fulfill the important job of note-taking anymore. So, what's a killer note-taking system nowadays?

My requirements?

My dream system must be Easy, Efficient and Flexible:

Know where & how to write notes easily without hesitations
Find right notes & roll back to old changes efficiently without hassles
Convert notes to other formats and share them flexibly without troubles

Build it!

Note format

I love taking notes with markdown. Why markdown?

It's clean for writing and reading
It's efficient, almost like writing plain texts
It's flexible to convert to any other formats

Note taking & organization app

I like doing everything in the terminal, so naturally, Vim is the best note-taking editor for me.

For organizing notes, in the beginning, I set up different folders for different notes. But it's not quite close to what I want. Especially, searching note is not a straightforward task, it needs to find or grep for a while. Then I wrote a handy script pynote for helping note organization, it worked and only at "it works" level... Recently, I switched to the amazing Notational Velocity for Vim, a plugin to port famous Notational Velocity into Vim. The whole note-taking experience never becomes so smooth! 💆

Note version-control system

It's easy. Git is the only reasonable option I can think of. ¯\_(ツ)_/¯

git init, make note changes, git add, git commit or git revert... All the fancy version control things are already familiar under my hands.

Note auto backup system

Google Drive? OneDrive? Dropbox? Nah, they are not really Git-compatible by default. However, I got inspired by SparkleShare, an open-source file sync app powered by Git, it adds file changes and pushes them to Github or GitLab automatically.

I may achieve the same functionality by using a simple script. The idea is to run fswatch in the background and keep monitoring file changes. When file changes are happening, fswatch will trigger Git commands to add and push changes automatically.

Here is a snippet of Bash script:

#!/usr/bin/env bash
...
function notesync() {
  fswatch -r -0 -d "$1" -e '\.git' -m "inotify_monitor" --event=Created --event=Updated --event=Removed | xargs -0 -n 1 zsh -c "source ~/.zshrc; cd $1; git add -A; git commit -m 'Update file'; git push" &
}
notesync "<folder_path>"
notesync "<another_folder_path>"
...

fswatch part is to monitor file changes, filter out correct file change events and send the signal:

fswatch -r -0 -d "$1" -e '\.git' -m "inotify_monitor" --event=Created --event=Updated --event=Removed | ... &

-r: watch subdirectories recursively
-0: pass NUL as delimiter (no idea why it's need.....)
-d: directory to monitor, the target folder full of notes
-e: exude paths matching regex, here I don't want to trigger Git commands when any changes are happening in .git
-m: monitor, type fswatch -M to see available monitors, more details of all available monitors: here
--event: filter event by event flag, mainly using events related to file changes, all available events: here
&: last but not least, don't forget it at the end, because it provides the ability to run fswatch command in background and detach the process

xargs part is to get signal from fswatch and execute specific commands:

... | xargs -0 -n 1 ...

-0: input items are terminated by null, not whitespace (no idea why it's needed...)
-n: maximum argument, 1 it is

zsh command part is to run Git commands:

... zsh -c "source ~/.zshrc; cd $1; git add -A; git commit -m 'Update file'; git push" ...

zsh -c: execute following commands in zsh
source ~/.zshrc: all my SSH keys are declared in .zshrc file, this command will load configurations from .zshrc file. Otherwise, push to remove Git repository may get permission denied
git add -A...: Git commands to add changes and push to remote repository. Some optimization ideas: 1) git pull first, in case it's a shared repository; 2) Put some specific texts in commit messages, maybe also including file names

Sumup

This is my killer note-taking system. It's easy, efficient and flexible:

Note format: Markdown
Note taking & organization app: Vim + Notational Velocity for Vim
Note version-control system: Git
Note auto backup system: fswatch + GitHub

So far I'm quite happy with this system. I enjoy using it not only for taking some private notes with my dirty secrets, but also for sharing notes with colleagues in my day-to-day work, like notes for exploratory testing. Oh, I almost forget, guess what I'm using now for wring this blog post?

How does your note-taking system look like? Share with me your awesome ideas?

Replace 8 Frequently Used Linux Commands!?

Kevin Cui — Sun, 02 Sep 2018 15:21:08 +0000

Almost a decade ago, I started to touch the command-line interface (CLI). I’m now a nerdy CLI lover and I’m so obsessed by its efficiency. Therefore, I’m always trying out new command-line commands. From time to time, some commands are replaced by other improved ones in my daily workflow. Especially for some frequently used commands, it’s worth to find better alternatives.

Here is a list of improved commands I’d like to share with you:

cd? z, up and fzf!

z

z is a very popular alternative command for jumping around. z doesn’t require any additional configurations. Although it works out of the box, you need just a little bit of patience. Because z has a “magic” algorithm based on “frequency” which can learn from the frequent vested directories after your cd around for a while.

An easy and quick example how to z:

~$ pwd
/home/foo/bar/one/two/three

~$ z tw
~$ pwd
/home/foo/bar/one/two

~$ z foo
~$ pwd
/home/foo

~$ z th
~$ pwd
/home/foo/bar/one/two/three

Want to know how “frequency” score looks like for all directories? Simply run command z -l will give you a list.

Want to clean some/all saved directories? Simply modify $HOME/.z.

up

up is an interesting command to replace cd .. and this ridicules one cd ../../../../

~$ pwd
/home/foo/bar/one/two/three

~$ up
~$ pwd
/home/foo/bar/one/two

~$ up 2
~$ pwd
/home/foo/bar

~$ up h
~$ pwd
/home

fzf

fzf is a versatile command, which can be combined in other commands. With the help of its amazing searchability, it can be used to “cd into the selected directory”:

fzf example

ls? exa!

exa aims to be the modem replacement for ls. In my opinion, it did it. exa is really easy to use. Its options are straightforward:

Here are my aliases:

alias ls='exa -s mod --git'
alias ll='exa -l -s mod --git --time-style=long-iso'

More options to tweak exa: exa options

cat? bat!

bat is a cat clone with syntax highlighting, line numbers and automatic paging (using less as the default pager). To display the contents of a file using bat looks like:

It’s possible to customize the theme of syntax highlighting. It supports Sublime Text syntax format. Check these default themes to have some inspirations: bat themes

grep? rg!

rg, ripgrep, is fast & furious. It has almost the same parameters of grep command, with improved output style and speed.

One main usage of grep for me is to find out which files contain some specific contents. An example to compare grep and ripgrep:

If you’re interested in how fast rg can be? Check this comparison: rg vs. grep vs. ag. vs. ack… Not sure if it’s still the case now. But overall, ripgrep is fast enough for my daily use.

find? fd and fzf!

fd

fd is a replacement for find focusing on user-friendly, with simplifying options of find. For example, I want to edit a file but I don’t remember what’s its full path actually is. Usually, I need to find its full path first and then edit it:

~$ find . -ipath "*how*exp*" -name "index.html"
./2017/04/17/how-i-do-exploratory-testing/index.html
~$ vi ./2017/04/17/how-i-do-exploratory-testing/index.html

Since I know the result will be only one file found, I can combine vi inside find command as well:

~$ find . -ipath "*how*exp*" -name "index.html" -exec vi {} \;

With fd, the command is more readable:

~$ fd -p 'how.*exp.*/index'
2017/04/17/how-i-do-exploratory-testing/index.html

fzf

fzf has an extremely powerful fuzzy searchability. It can nest finding steps directly with another command. I bind the key ctrl-t to enable fzf. As the same example as above, but with fzf, it’s a completely different story:

~$ vi (press ctrl-t)

Another fzf example

An alias to combine vim and fzf:

alias vif='vim $(fzf)'

kill? fkill!

fkill, its f stands for “fabulously” (It’s definitely not the word I’m thinking in my mind 😉). Sometimes, kill some zombie processes is necessary. fkill can search a process and kill it interactively. It makes killing stylish:

fkill example

watch? loop!

“UNIX’s missing loop command.” loop is designed to fill the gap. It works well to replace watch:

# monitor current directory and print file list every 6 seconds

# watch
~$ watch -n 6 ls -l

# loop
~$ loop -e 6s 'ls -l'

Moreover, loop has some awesome options to determine when to stop watching:

...
-c, --until-contains <until_contains> Keep going until the output contains this string
-r, --until-error <until_error> Keep going until the command exit status is non-zero, or the value given
-m, --until-match <until_match> Keep going until the output matches this regular expression
-t, --until-time <until_time> Keep going until a future time, ex. "2018-04-20 04:20:00" (Times in UTC.)
...

Example: stop watching when stop-now file is created and then open it in vim:

~$ loop -e 6s 'clear;date;ls -l' -c 'stop-now'; vi stop-now

loop example

Actually, using loop to replace for command is also promising and full of pleasure:

# create 10 markdown files: 0.md, 1.md ... 9.md

# for
~$ for ((i=0; i < 10; i++)); do touch ${i}.md; done

# loop
~$ loop -n 10 'touch ${COUNT}.md'

man? tldr, pet and how2!

tldr

tldr is a simplified collection of man pages. It returns some command snippets with practical examples:

pet

pet is a CLI snippet manager. It’s a handy tool to save some commands snippets which are easily forgotten. The idea is to use pet building own reference manuals.

how2

how2 takes another approach to search for the right command: it queries questions and answers from stackoverflow!

how2 example

In case the displayed answer is not what exactly you want, how2 provides the interactive mode to select in a list of all relative questions and answers.

From now on, we can get confident to “save the world”!

Share this list if you think it’s a good read for other CLI nerds. 💝

Drop me a message if you have other commands to recommend. ❤️

3 mitmproxy Tips You Might Not Know About

Kevin Cui — Sat, 28 Jul 2018 13:13:04 +0000

^{Photo by Ashley Batz}

mitmproxy is a powerful proxy tool for debugging network requests and responses. It’s my favorite proxy tool and becoming one of my daily essential tools for working. I would like to share with you 3 mitmproxy tips you might not know about.

First things first, a small checklist:

If these questions are way too easy for you and your answers are “Yes! Yes! Yes!”, then continue reading. Otherwise, please click and read the links above, and see you here later.

Tips 1: Custom configuration and key binding

One of the advantages of mitmproxy is highly customizable. It’s very continent to set up once configurations that every time mitmproxy loads them automatically when it starts. All the available handy options are in the mitmproxy document. Here is my ~/.mitmproxy/config.yaml:

console_palette: "dark"
console_palette_transparent: True
console_mouse: False
console_focus_follow: True
ignore_hosts: []

As I’m using colemak keyboard layout, j and k keys are opposite for the navigation of up and down in mitmproxy. I have to change these 2 keys to make navigation smooth. Before, to change key bindings in old versions, I have to hack around code in defaultkeys.py. Since mitmproxy v4.0, custom key binding is finally officially supported! Here are my key bindings in ~/.mitmproxy/keys.yaml:

-
  key: j
  ctx: ["global"]
  cmd: console.nav.up
-
  key: k
  ctx: ["global"]
  cmd: console.nav.down
-
  key: l
  ctx: ["flowlist"]
  cmd: console.nav.select

To know more about which command to use in cmd field, type K (shift + k) in mitmproxy. A list of current key bindings will show up. More information about key binging configuration is in the mitmproxy document.

Tips 2: Knowing client connection status when a filter is applied

It happens quite often to me: when I applied all filters in advance and then tried to connect a client to mitmproxy. However, there is nothing show up in the flow list. How do I know if it’s because the client is not connected to the proxy successfully or no requests are matching the filter?

A usual but not good way: Well, I have to remove the filter to see all requests. If I can see requests from the client, then I will again apply the filter…

An easy way is to add | .* in the filter, which will reveal all incoming requests. Yep, regex magic:

mitmproxy filter on

Another easy way is to simply type E (shift+e), which will open events logs. It shows the client connection status. Use q to quit events view:

mitmproxy events view

Tips 3: Using mitmproxy as a mock server

The killer feature of mitmproxy is scripting. Using python scripts, all requests and responses are fully controlled and changeable on the fly. Besides that, mitmproxy is super robust to handle multiple clients simultaneously connected to one host. These features make mitmproxy a decent mock server. Therefore, I use mitmproxy a lot as a mock server.

For example, a script for the purpose of rewriting and mocking response body, mitm-rewrite:

Map http://example.com/pass to test_pass.json (terminal on the bottom left)
Create mock response data in test_pass.json (terminal on the bottom right)
Visit http://example.com/pass on the client side
The initial response is now replaced by the one in test_pass.json (terminal on top)

If you want to know more useful scripts, I have a mitm-scripts collection to check out. If you want to make your own scripts, there are some examples.

Didn’t try mitmproxy yet? Get it a try. 😉
Know other mitmproxy tricks? Tell me more. 😍

One Week with Colemak

Kevin Cui — Sun, 27 Aug 2017 08:07:30 +0000

Last weekend, I decided to switch from QWERTY to Colemak keyboard layout. Did I suffer a lot since then? Of course no, well, maybe a little bit. After “one-week intensive typing training”, actually I quite enjoy the fresh feeling of typing with Colemak. I’m pretty happy that now I am able to type on Colemak keyboard layout with the average speed 50 WPM. I would say that it’s good enough for daily life. If you happen to be interested in switching to Colemak, here are my short notes for every day of last week and how my “intensive” training looks like.

D1 - Saturday

Time Spent: 1 hour
Typing Speed: 15 WPM

Plan? Just do it!

There are some different approaches to help people switching to Colemak progressively. One interesting approach is called Tarmak (Transitional Colemak) learning layouts. Tarmak has 4 layouts in total, each of the 4 Tarmak layouts changes 3–4 new keys from the previous step until you’ve made the full transition to Colemak. ^[1]

In deed, it sounds like a good plan. But I didn’t go that way, I chose a crazy approach: go cold turkey. I get rid of QWERTY and use Colemak immediately. Because I believe that I could handle the change to Colemak completely at once. At the end, it’s all about training muscle memory.

Remember the layout

First things first, remember the layout. To be familiar with Colemak layout, I spent 30 minutes on typing alphabet “abcdefgh…” repeatedly, until I am 100% sure that all 26 letters are in their correct positions.

D2 - Sunday

Time spent: 3 hours (3 1-hour sessions)
Typing speed: 25 WPM

Brain takes full control

After I memorized the layout, the real “intensive” training began. The main site I used is Speed Typing Online. One article after another, I keep typing. Meanwhile, I have to let my brain know exactly what I need to do and then my fingers would be forced to do it. This process of erasing muscle memory is slow. My brain needs to be highly focused in order to control each single movement of my fingers, and to prevent these movements initialized by old muscle memory.

Pinky is weaky

One difficulty I did realize is that my right little finger is weaker than my other fingers. Before with QWERTY, it takes control of letter “P”, which appearance frequency is way more lower than the letter “O”. Now with Colemak, it has to take care of letter “O”. This sudden increase of heavy work load on the shoulder of right pinky finger makes it clumsy. I can really feel the “stop” moment when letter “O” is needed. So I have to focus on training pinky finger, even more.

D3 - Monday

Time spent: 1 hour training + 8 hours working
Typing speed: 28 WPM

Productivity?

As I decided to go cold turkey and I don’t want a detour, I changed the keyboard layout on my working laptop to Colemak as well. I remember my first and second login attempts were failed, after that, nothing was really struggling. To be honest, the typing speed around 25 WPM is enough for daily work, but the efficiency is… meh… Especially, when replying someone’s questions in chat app, I feel sorry to let people on another side waiting a bit longer than usual. Besides that, when I was coding/scripting, I felt that I have more time for thinking, which might be a good thing.

Smartphone out!

As the inventor of Colemak mentioned, Colemak isn’t recommended for smartphones as it increases finger travel and error rate compared to QWERTY. On smartphones I would recommend gesture typing on a QWERTY layout. ^[2] Furthermore, the muscle memory of typing on smartphone keyboard is not the same. Therefore, I will still keep using QWERTY layout in smartphone until one day another “advanced” keyboard layout invented for smartphone.

D4 - Tuesday

Time spent: 1 hour training + N hours working + N hours imagining
Typing speed: 30 WPM

Use imagination

I found a magic way to perform typing training without a physical keyboard: using my imaginations. For instance, when I am in public transport, I’d like to listen to podcasts. From the earphones I heard the words, then I put my hands in the air and move fingers to type these words. Others who were watching at me may think that I could be a pianist, or probably a madman.

D5 - Wednesday

Time spent: 1 hour training + N hours working + N hours imagining
Typing speed: 40 WPM

Targeted training

It’s time to have more targeted training of some common suffixes. In theory, if I could type them unconsciously as a reflection, my typing speed would increase a lot. So I made a list of suffixes to focus on:

-ly, -ing, -ion, -er, -or, -ment, -ous, -ful, -able

On the other hand, I found that I made many mistakes on top row letters “f”, “g”, “l” and “y”. I keep in my mind to pay attention to them: be accurate first, then speed up.

Moreover, I started to learn typing systematically by following the lessons on typing.com. I skipped the entire beginner lessons and some lessons designed for QWERTY only.

D6 - Thursday

Time spent: 1 hour training + N hours working + N hours imagining
Typing speed: 50 WPM

Repetition, Repetition, Repetition…

Every day, I am better at typing than yesterday. I felt my muscle memory is improved little by little: now I don’t need to think about the layout ahead! It means that I can think other stuff in my mind while typing letters on the keyboard. It’s definitely a good sign, which tastes so sweet. It’s the power of repetition and the outcome of never-give-up.

D7 - Friday

Time spent: 1 hour training + N hours working + N hours imagining
Typing speed: 52 WPM

Time to have fun

After these days’ practices, I know I am on the right track to achieve the success. Although it’s important as my daily routine to keep using Speed Typing Online and following basic lesson on typing.com, learning typing can also be full of fun, without the boredness of repetition. It’s time to discover something new and have some fun:

TypingCAT: It’s an ordinary typing site. But its UI is super clean and comfortable. It will make you feel awesome when you are typing.
Ratatype typing test: Instead of the usual time-box approach, this site requires you to finish the whole article and then shows your score. Their typing materials are really good. They normally have 50-100 words, with the mix of letters, numbers and punctuations.
typeracer: A typing competition is presented as a form of car racing game. It supports Multiplayer “racing” together. Its UI is okish and its material is so-so. But if you want to see how is your performance under stress, it’s a good game to play.

D(n) - In the future…

1. Use Caps Lock as…?

Colemak maps Caps Lock key to Backspace. It makes sense to have another Backspace on the left hand to balance its usage in two hands. But so far I don’t use it at all. Maybe I will map it to Esc, which is useful in Vim.

2. Get rid of bad habits

I still need to get rid of some bad habits of using QWERTY:

Always use left thumb to tap Space bar
Always use left pinky to tap Shift

I know how to do it correctly, but… Fine, I will do it correctly in the future.

3. Colemak x Vim

New layout for Vim is a tough challenge. I had several oops-moment which are caused by “gg” vs. “dd”. But I don’t think I will remap keys to make the original Vim style alike. I prefer to relearn it with Colemak from scratch. Here is the Colemak/vim cheat sheet ^[3]. I know it will take a while to fully operate Vim with Colemak.

4. Get “QWERTY mode” back?

As a side effect, the muscle memory of QWERTY is completely lost. I cannot type with QWERTY layout anymore. Should I train myself to get QWERTY mode back in the future? I guess no ¯_(ツ)_/¯

[1] Øystein Bech Gadmar: How does Tarmak work?

[2] Shai Coleman: Is Colemak suitable for smartphones?

[3] Aimee Rivers: “Colemak and vim”: But what about h/j/k/l?

Why I Switch to Colemak Keyboard Layout?

Kevin Cui — Sat, 19 Aug 2017 18:48:03 +0000

Colemak, pronounced /‘ko:lmæk/ (Coal-Mac), is a keyboard layout based on QWERTY. It’s developed by Shai Coleman and released in 2006. Colemak keyboard layout looks like:

q w f p g j l u y ;
   a r s t d h n e i o
    z x c v b k m , . /

What’s special about Colemak? Why do I decide to switch from QWERTY to Colemak?

Reason 1: Discard QWERTY

As we all known, QWERTY was originally designed for the typewriter to solve the problem of the type bars colliding with each other and jamming. The idea behind is to arrange the keys with the most common letters in hard to reach spots, to slow typists down and try to avoid typewriter jams. ^[1]

If you’re interested in more stories of QWERTY, The Curse of QWERTY is a nice article to read.

Since the mechanical typewriters vanished from historical stage and all the modern computers don’t have “mechanical sticking” problem, why we still need to bear the pain of the inefficiency of QWERTY? It’s time to let QWERTY be retired with honor. Let it go!

Reason 2: I want a challenge

When I was 14 years old, I started to learn to type on a typewriter in school. Yes, a real typewriter and it’s amazing! I can still recall the comment when type bars hit on paper, the sound and the feeling are so satisfying. At the end of the semester, I successfully passed typing exam with the speed of 70 WPM. Now it has been 15 years, my speed stays around 75 WPM. For me, it’s obviously not improved at all…

As a guy who is eager for something new and obsessed with efficiency, I don’t believe that 75 WPM is my limit. I’m afraid that QWERTY itself is the obstacle. Maybe an alternative keyboard would unchain my ability. That’s why I want a challenge to see how fast I’m able to type.

Reason 3: Efficiency is the key

Colemak is designed to be ergonomic, comfortable and fast. ^[2]

According to carpalx analysis, Colemak has a quite low typing effort among 10 keyboard layouts. The carpalx detailed analysis ^[3] shows:

Colemak has a great home row usage 74%, compared to QWERTY 34% and Dvorak 71%
Colemak bottom row usage is low at 9%, the same as Dvorak, which is better than QWERTY 15%
Colemak is more balanced in hand use, with a 6% preference for the right hand. Dvorak has 14% for the right and QWERTY 15% for the left

There is an app to analyze the different keyboard layouts. It’s called Keyboard Layout Analyzer. I’m curious how the result would be. So I tried to input:

A recent BBC article
George R. R. Martin Biography
One of my python script
My blog index.html
One of my blog post plain text

The result of these 5 rounds: Colemak scores the highest in each round.

High home row usage, less finger travel distance, and balanced left-right hands… These advantages of Colemak make it as one of the best keyboard layouts for improving efficiency and ergonomics.

Reason 4: Easy to learn

I remember 5 years ago, I planned to switch to Dvorak. But I was totally failed after 1-week struggling. Because the transition phrase from QWERTY to Dvorak is really hard, I was not good enough to use only Dvorak at work. Therefore, it required me to memorize 2 different keyboard layouts at the same time, which was possible for my brain but impossible for my muscles. I was kidding at that time: I need at least a 2-months vacation to train Dvorak without any productivity.

To learn Colemak, I feel it would be a different story. Colemak is based on QWERTY layout, which means it’s built without the terrifying prospect of changing everything. The Colemak layout is reasonably similar to QWERTY with just 17 different keys as opposed to Dvorak’s 33, and only two letters P and E that move from one hand to the other, whereas with Dvorak it’s 22. ^[4]

Colemak is rather easy to learn. It took me only 30 min to memorize Colemak keyboard layout without any frustration. After 1 day training, now I’m already used to it. I estimate that a 2-days weekend should be enough for me to have a reasonable typing speed. It wouldn’t cause my huge headache to use Colemak at work next Monday.

These tantalizing reasons give me the motivation to switch to Colemak. I guess I have to face this challenge and I will be busy with training next week. Challenge is on and change is now.

Happy typing Colemak!

[1] HowStuffWorks: Why are the keys arranged the way they are on a QWERTY keyboard?

[2] Shai Coleman: Colemak advantages

[3] Carpalx: Colemak typing effort

[4] Eric Limer: Why I’m Learning To Type All Over Again