DEV Community: Mario Mazo

Kinesis Advantage for elixir and ruby developers

Mario Mazo — Mon, 18 Jan 2021 22:03:41 +0000

Keyboard remapping for elixir and ruby development on doom emacs or vim

I got the kinesis advantage 2 keyboard some time ago I have to say is money well spent. My wrist pain is now minimal. As I spend my days doing go and elixir I have fine tuned my keyboard to increase my productivity. Here I will share the layout I use while developing in elixir using doom emacs. But all the information here might as well apply to ruby and vim.

For this I have create a layer named e_querty.txt that way all I have to do is Progm+e to switch to elixir mode. The layer looks as follows:

thumb keys layout

I use a Mac so Im using the mac naming for the keys but you get the idea.

So I replicated Ctrl, Option, Command and Shift on both sides of the keyboard because as a doom emacs user with evil mode (vim style navigation) I need to have access to all letters and symbols while pressing Ctrl, Shift etc. and having them in only one side made some combos really hard to execute.
So at the end the thumbkeys look as the on the following picture

Key remapping

Numbers

So as I rarely use numbers in my normal workflow I decided to invert the numbers into the symbols (Shift+) to speed up navigation in emacs/vim. When I need a number I can use Shift+<symbol> to get the number.

Other keys

Since elixir is snake case I inverted - so now by default I get _
| is way more used than \ so also inverted those.
: is more used than ; so inverted as well
" more used than '... inverted
Esc replaced CapsLock for easy escaping

Layout file

Here is my e_qwerty.txt just drop it in you keyboards active directory and you will get the mapping

[scroll]>[shutdn]
{1}>{speed5}{-rshift}{1}{+rshift}
{rshift}{1}>{speed5}{1}
{lshift}{1}>{speed5}{1}
{2}>{speed5}{-rshift}{2}{+rshift}
{rshift}{2}>{speed5}{2}
{lshift}{2}>{speed5}{2}
{3}>{speed5}{-rshift}{3}{+rshift}
{rshift}{3}>{speed5}{3}
{lshift}{3}>{speed5}{3}
{4}>{speed5}{-rshift}{4}{+rshift}
{rshift}{4}>{speed5}{4}
{lshift}{4}>{speed5}{4}
{5}>{speed5}{-rshift}{5}{+rshift}
{rshift}{5}>{speed5}{5}
{lshift}{5}>{speed5}{5}
{6}>{speed5}{-lshift}{6}{+lshift}
{rshift}{6}>{speed5}{6}
{lshift}{6}>{speed5}{6}
{7}>{speed5}{-lshift}{7}{+lshift}
{rshift}{7}>{speed5}{7}
{lshift}{7}>{speed5}{7}
{8}>{speed5}{-lshift}{8}{+lshift}
{rshift}{8}>{speed5}{8}
{lshift}{8}>{speed5}{8}
{9}>{speed5}{-lshift}{9}{+lshift}
{rshift}{9}>{speed5}{9}
{lshift}{9}>{speed5}{9}
{0}>{speed5}{-lshift}{0}{+lshift}
{rshift}{0}>{speed5}{0}
{lshift}{0}>{speed5}{0}
{hyphen}>{speed5}{-lshift}{hyphen}{+lshift}
{lshift}{hyphen}>{speed5}{hyphen}
{rshift}{hyphen}>{speed5}{hyphen}
{\}>{speed5}{-lshift}{\}{+lshift}
{lshift}{\}>{speed5}{\}
{rshift}{\}>{speed5}{\}
[caps]>[escape]
[lctrl]>[lwin]
[rwin]>[lalt]
[rctrl]>[rwin]
{;}>{speed5}{-lshift}{;}{+lshift}
{rshift}{;}>{speed5}{;}
{lshift}{;}>{speed5}{;}
{'}>{speed5}{-lshift}{'}{+lshift}
{lshift}{'}>{speed5}{'}
{rshift}{'}>{speed5}{'}
[home]>[rctrl]
[pup]>[rctrl]
[end]>[lshift]
[pdown]>[rshift]
[kp-lctrl]>[lwin]
[kp-rwin]>[rctrl]
[kp-rctrl]>[rwin]

closing remarks

why 4 shifts? no idea I have no use for those "external" shifts so I keep them there
Got some tips for the kinesis advantage, emacs/vim or elixir/ruby productivity? just comment

Elixir and DooM Emacs

Mario Mazo — Fri, 15 Jan 2021 20:14:29 +0000

Genesis

Recently I gave up on configuring vim and spacevim over and over, and I decided to try emacs... a piece of tech I have avoided for over 20 years. My goal was to have something that worked out of the box, with all the nice stuff that vim provides. After some searching I found doom emacs so I installed emacs for the first time in my live in my 20+ years of *nix, ops and programming career.

So first things first; doom emacs is freaking amazing, now I feel bad for all the time I wasted tuning vim and vscode, with doom 99% of the things worked out of the box, some minor relearning was needed, but was some love at first sight kind of thing, only thing left was having full integration with my favourite programming language elixir to have a nice IDE like experience

Nice soap opera dude! Where is the elixir-ls part?

So I began reading how to integrate elixir and I found a lot of tutorials but seems like doom moves fast and almost of all them were outdated, did not work and were quite long. So here is they simplest way i have found to integrate elixir-ls with doom emacs and turned out it was super simple.

Elixir-ls in doom emacs

install elixir-ls

git clone https://github.com/elixir-lsp/elixir-ls.git ~/.elixir-ls
cd ~/.elixir-ls
mix deps.get
MIX_ENV=prod mix compile
MIX_ENV=prod mix elixir_ls.release -o release

add elixir-ls to your path

Add elixir-ls to your PATH
in your shell rc, in my case .bashrc or .zshrc

export PATH=$PATH:$HOME/.elixir-ls/release

add your env variables to doom

I use a mac so the emacs.app starts isolated and the PATH is not the same as the one in your shell. Also i almost never start emacs from the command line so the path to the elixir-ls has to be in your doom env variables

~/.emacs.d/bin/doom env

you can validate that the elixir-ls path is in ~/.emacs.d/.local/env

activite lsp and elixir with lsp support

In the config.el uncomment lsp, tree-sitter and replace elixir as follows

tree-sitter
lsp
(elixir +lsp +tree-sitter)

sync doom

~/.emacs.d/bin/doom sync

restart emacs... profit
you should see a little 🚀 (rocket) in your status bar that means is connected to the lsp server, and you get auto-complete and all other good stuff.

troubleshooting

you can use SPC b B and search for a buffer *lsp-log* to see error messages

Closing notes

This gives the basic set up to start working, you can customize the lsp with lsp-ui but I dont know how and Im not that interested in fine tuning every single piece of it. This is enough to get me up and running.

Also make sure you keep your elixir-ls up to date. And if you have multiple erlang/OTP versions is very likely you will need a compiled elixir-lsp to each specific version.

If you know useful cool tricks for doom+elixir please do share them! Im kind of new in this :D

Mutual exclusion using go and consul

Mario Mazo — Thu, 13 Sep 2018 09:38:11 +0000

Mutually exclusive workers using golang and consul

Implementing mutually exclusive workers is fun and useful. In this post I will show how to do a naive implementation (DO NOT use this in production) of this pattern using client side leader election with consul.

The reason I chose consul for this task is because it was already available in our infrastructure, but the same could be achieved with different tools, for example AWS dynamoDB. Basically you just need a Key-Value store that support locks.

All the code snippets in the post are taken from a sample github repo where you can find the full implementation.

The problem

We had a job server that each hour would connect to a postgreSQL database read some tasks then it would proceed the execute the tasks created in the last hour, and then it would write back to the database if the task was succefully finished.

The problem with this is that it doesn't scale. The jobs began to take longer and longer and if machine crashed all tasks were not executed

The solution - Mutually exclusive workers (Distributed Locks)

We knew we needed to run multiple instances of the job server. But each task could only be executed once. So we decided to use distributed locks with consul. After a quick search we realized that implenting the leader election algorithm was the best solution for us.

So the whole concept is quite simple. All job servers get a list of tasks to be executed, then they iterate over the tasks list, get a lock on the task
so all other job servers skip that task, and move on to the next task. This way we can have multiple nodes working at the same all working on different tasks.

In the image we see that serverA locks task1 and executes the task sending it down that pipe with a solid arrow. ServerB and ServerC have dashed arrows what I mean is that task1 could not be executed by those 2 servers. Later on serverB would lock another task and so on.

Client side leader election

The leader election algorithm is quite simple its just two steps:

Create a session in consul
Try to put a Lock on that session. If you succeed you are leader if not... well you are not leader

Step 1: Creating the session

First we will create a small wrapper function around the session creation. So
we always create a session no matter if we can work on a specific task or not.

func (ec *exclusiveWorker) createSession() error {
    sessinConf := &api.SessionEntry{
        TTL:      ec.sessionTimeout,
        Behavior: "delete",
    }

    sessionID, _, err := ec.client.Session().Create(sessinConf, nil)
    if err != nil {
        return err
    }

    fmt.Println("sessionID:", sessionID)
    ec.sessionID = sessionID
    return nil
}

We will pass to configuration parameters to the go consul client

TTL: This is the time out for the session. After this time has passed, consul will execute the behavior. I have set 15sfor this example
Behavior: This delete behavior means that after the TTL has been reached the session is deleted and the Key associated with it

If you want to better understand how behaviors work please read the official consul sessions documentation. But the main take is this:

If the release behavior is being used, any of the locks held in association with the session are released, and the ModifyIndex of the key is incremented. Alternatively, if the delete behavior is used, the key corresponding to any of the held locks is simply deleted. This can be used to create ephemeral entries that are automatically deleted by Consul.

Step 2: Acquire the session

Once we have a session we try to aquire the session. If we succed put a Lock on it and return success. We are leaders. If we failt to acquire the lock it means that the task is already being executed by another server. Here is were the mutual exlusitivy happens.

Again we are going to wrap this in a simple funcion for easy handling

func (ec *exclusiveWorker) acquireSession() (bool, error) {
    KVpair := &api.KVPair{
        Key:     ec.key,
        Value:   []byte(ec.sessionID),
        Session: ec.sessionID,
    }

    aquired, _, err := ec.client.KV().Acquire(KVpair, nil)
    return aquired, err
}

We are going to need to pass 3 values to the consul client

Key: This is the indentifier of the tasks. To follow conventions I chose service/<TASK_NAME>/leader but could be anything that better fit your needs. I really thought about using service/<APP_NAME>/<TASK_NAME>.
Value: This is really unimportant when you are using consul for distributed locks. I chose the current sessionID (which is nothing but an UUID) for easy debugging. But could well be rambo or goku
Session: is the ID of the session we are going to try to Lock

Step 3:

There is no step 3. That's all is there for the leader election part. But we still need a couple of helper functions we are going to need to make this work in a more realistic way.

Destroy session

Once our task is done we have to be nice and release the lock. We could wait for the TTL and beahvior to kick in, but that's not nice. So let's implement
a basic destroy session function we can call when our task is finished, and all we need is the sessionID.

func (ec *exclusiveWorker) destroySession() error {
    _, err := ec.client.Session().Destroy(ec.sessionID, nil)
    if err != nil {
        erroMsg := fmt.Sprintf("ERROR cannot delete key %s: %s", ec.key, err)
        return errors.New(erroMsg)
    }

    return nil
}

Renew session

If the tasks is taking more than the TTL the session and key are deleted by the behavior. If this happens a different server could lock exactly the same task and you would execute 2 times the same task which is what we are trying to avoid.

So we need to renew the session constanly to avoid triggering the behavior. The consul client comes with a handy function RenewPeriodic() that does exactly that. So lets write the wrapper:

func (ec *exclusiveWorker) renewSession(doneChan <-chan struct{}) error {
    err := ec.client.Session().RenewPeriodic(ec.sessionTimeout, ec.sessionID, nil, doneChan)
    if err != nil {
        return err
    }
    return nil
}

Here we need 3 things:

sessionTimeout: This is the original TTL. The client will use this to refresh the session each TTL/2
sessionID: Id of the session we want to renew
doneChan: Is a channel we use the signal that we need to keep renewing the session or if we close the channel we mean that we are done with the task and we don't need to renew the session anymore

Demo

In the accompanying github repo There is a fully working implementation of this. It's also very simple and intended for learning purporses.
You will need a work go installation to be able to compile the code and docker with docker-compose to be able to run a consul server. So lets see the demo:
First launch consul

$ docker-compose up

Then open 2 terminals and in the first run the code and you should see something like this:

$ go run main.go
sessionID: bac7cf19-285e-9907-98ad-e8189a07cbd9
I can work. YAY!!!
Starting to work

you now can check the web interface of consul http://localhost:8500/ui/dc1/kv
to verify that the keys are created, locked and destroyed either by TTL, finishing the task or interrupting the task.

in the second one if you run the code you can see the code exiting while the task is executed

$ go run main.go
sessionID: d0f26b95-11cb-236c-bba7-601441f2ae74
I can NOT work. YAY!!!
$

if you interrupt the task by doing Ctrl+C you should see the cleanup happening

$ go run main.go
sessionID: 4685d391-251d-9f6d-1c2c-5ab6fdbd9f98
I can work. YAY!!!
Starting to work
^C2018/09/13 11:16:04 Job interrupted. Cleaning up

if you try to connect right after the task is finised you will notice you cannot conect. This is due to lock-delay which is documented in the sessions section of the consul documentation.

Final thoughts

I invite you to check the github repo as the code there is full of notes about implementation that could be useful for a real implementation.

Also there are things I did not implement like stop() or Discovering the Leader, but implementation should be simple. Feel free to submit a pull requres.