DEV Community: protium

An interpreted language you can try in my terminal website

protium — Thu, 17 Nov 2022 18:20:30 +0000

A few weeks ago I started reading Writing An Interpreter In Go and
while reading it and implementing the interpreter a question popped up: what if I can run this
interpreter in the terminal in my website?

To recap, a few months ago I decided to transform my personal website into
a terminal (read more about it here).
In today's post I'll give an overview of the whole implementation, but first a preview:

You can try it by typing simia in the terminal here

The book: a great resource for learning about interpreters

Although this post is not a book review, I'd like to say that it was a really nice reading.
You can learn from the basics of tokenization to the evaluation of the parsed input.
The author does a great job guiding the reader through all the concepts with concrete examples.
If you want to learn about how interpreters and REPL work this book is definitely a great start!

Extending the Monkey language

In the book, the author introduces us to the Monkey language.
Its syntax is quite simple and that motivated me to add more expressions and syntax, essentially extending
the language to my desire. I wanted to create a language that would combine syntax from go, rust, and
elixir. And that's how simia was born.
I'm still working on it but so far I have added support for:

Range expressions
for-loop expressions along with the infix operator in. e.g.

for a in 1..10 { 
  log(a); 
}

Support for boolean condition in for-loop

for a > 0 { 
  log(a); 
}

Parentheses are optional for if and for-loop expressions
Support for elixir's pipe operator |>

8 |> factorial() |> add(100)

Re-assign variables with = operator

let foo = "bar";
foo = "1234";

And in my TODO list:

Support modules
Support pub key
Add more built-ins for arrays, strings, and maps
Support mut key and control mutability

Overall, I want simia to be some sort of modern functional programming language.

Loading the interpreter in the browser: WebAssembly

If you are not familiar with wasm, here a quote from https://webassembly.org/

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine.
Wasm is designed as a portable compilation target for programming languages, enabling deployment on
the web for client and server applications.

Basically it allows us to convert the simia interpreter (written with golang) into a set of instructions that the browser
can understand and execute.

Doing this in golang was really simple thanks to the syscall/js
package and the arch wasm when compiling the interpreter.
Let's see a simple example:

// main.go

package main

import (
    "syscall/js"
)

func main() {
    done := make(chan struct{}, 0)
    js.Global().Set("wasmHash", js.FuncOf(hash))
    <-done
}

func hello(this js.Value, args []js.Value) any {
    return "hello " + args[0].String()
}

Notice the channel we keep waiting for input. This is necessary because the wasm module it is an application
and not a library. So this application should keep running and not exit as soon as js.Global()... is executed.

Now we compile the app to wasm:

GOOS=js GOARCH=wasm go build -o hello.wasm ./main.go

In order to load our wasm we need to use the wasm tools go provides

cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .

and now the final bit, we load the wasm from js and use our wonderful hello function:

<html>
<head>
    <meta charset="utf-8"/>
    <script src="wasm_exec.js"></script>
    <script>
        const go = new Go();
        WebAssembly.instantiateStreaming(fetch("hello.wasm"), go.importObject).then((result) => {
            go.run(result.instance);
            console.log(window.hello("universe")); // hello universe
        });
    </script>
</head>
<body>
</body>

And that's it, serving the file and opening it in a browser should show the result in the developer console.
I took exactly the same steps to compile and use the simia interpreter in the browser.
The wasm module exports to objects:

simia: a function that evaluates simia code and returns the result
simia_version: the version of the interpreter

Caveats

All was going beautifully but I had to step back to think about a problem: simia has a built-in function
called log which prints values to the stdout using the fmt package.
When compiled to wasm, the fmt.Println will use console.log as a buffer, so any log instruction will be print
into the browser console.

The solution is to modify the built-in to use a buffer and then pass some sort of buffer from javascript,
so then I can use that buffer to print to the terminal element. I will probably will solve this during the
weekend, but if you have a better idea please let me know in the comments.

Extending my website terminal

When I came up with the idea of adding the REPL into my website, I took a look at the existing code in my website
and identified the requirements needed to make this happen:

The wasm module can be loaded on demand, i.e. the first time the user calls simia from the terminal
Commands should be able take the std in and out.
The REPL will use the wasm module to eval the user input
ctrl-c should "kill" the process and return to the shell

In other to implement this requirements I have prepared a few refactors:

Improve error handling, e.g. "command missing", "wrong arguments", etc
Improve key presses handling and extract common functionality
Introduce processId. When a command execution returns a process Id it means that the main function, let's call it shell, should stop processing the input. Once the command exits it should call a callback to restore the main function.

That's it, I can say that now my terminal has a fully working simia REPL.

Outro

I had a lot of fun writing the interpreter and loading it into my terminal website, I definitely learned
a lot of things in the process. I think writing an interpreter is quite a good practice for any developer.

All the changes I made to my website were introduced in this PR

What else would you add to the language or the terminal?
I'm planning to do some sort of treasure hunt in my website, stay tuned!

👽

Github template for Golang services

protium — Sun, 19 Jun 2022 20:26:51 +0000

As a weekend project I created a github template that can be very handy for creating go services with relational databases.
Let's take a look at what is included.

Task Runner

For many years, GNU make has been my to-go tool to run rules and tasks for any sort of project. It is
fairly simple to use but it can also become complex as some rules might require to execute external tools or
even declare bash functions with in a rule definition. I must admit that the developer experience can
be rough for those who haven't used make before.

Developer experience is a very important topic to me and I decided to use this project to find a reliable
alternative to make. And that's how I found Task:

Task is a task runner / build tool that aims to be simpler and easier to use than, for example, GNU Make.

After checking some examples and its API docs, I got convinced I should give it a try.
Although I'm not a fan of yaml I found some neat features that I prefer over make:

Import env variables

Makefile

include .env
$(eval export $(shell sed -ne 's/ *#.*$$//; /./ s/=.*$$// p' .env))

Taskfile

dotenv: ['.env']

Showing help

Although make doesn't create a help command, there is a very common pattern to define one:

help: ## print this help
    @awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) | sort

Task provides a list of available commands when running task -l, but you need to add a desc field to each command:

run:
  desc: Run go app
  cmds:
    - go run cmd/main.go

CLI args

When working with make you can pass arguments/variables a target, for instance

hello:
  @echo $(name)

To pass the argument you would call it like make hello name=dev. This can become tedious when passing multiple args to the inner command.
A simple hack to allow cli args is to filter out those args from the make goals and also avoid errors when
targets are not found:

# Filter out make goals from CLI args
args = $(filter-out $@,$(MAKECMDGOALS))
# Do nothing if target not found
%:
    @:
hello:
  @echo $(args)

It works fine unless the arguments have a value that matches the name of any target.

On the other hand, task provides a simple template variable with the arguments CLI_ARGS

hello:
  cmds:
    - echo {{.CLI_ARGS}}

The command can then be called as task hello -- world

Verbose file targets definition

Although make file targets are not difficult to understand, I think task syntax is easier to understand
for a new dev. Let's compare them:

pkg/db/%.go: db/queries/%.sql
    @docker run --rm -v ${CURDIR}:/src -w /src kjconroy/sqlc generate

Taskfile

db-gen:
  desc: Generate queries code using Sqlc
  cmds:
    - docker run --rm -v $pwd:/src -w /src kjconroy/sqlc generate
  sources:
    - db/queries/*.sql
  generates:
    - pkg/db/*

So far task seems to be a very good alternative to make, at least for my personal use case.

Folder structure

The folder structure of this template is based on folder structures I've seen across many go repos,
which I really like:

- `cmd/`: app entry points
- `db/`:
  - `migrations/`: SQL migrations files
  - `queries/`: SQL query files used by `sqlc`
- `pkg/`: app sources
  - `db/`: code generated by `sqlc`

Database

As mentioned before, this template is for a go service with a relational database.
As a personal preference I chose postgres as db engine.

Schema

I have recently adopted the practice of defining my db schemas using dbml
and generating the sql code using their CLI tool. Example:
This schema

Table users {
  id uuid [pk]
  user_name text [not null, unique]
  password_hash bytea [not null]
  created_at timestamptz [not null, default: `now() at time zone 'utc'`]
  updated_at timestamptz [not null, default: `now() at time zone 'utc'`]
}

Will generate this SQL code:

CREATE TABLE "users" (
  "id" uuid PRIMARY KEY,
  "user_name" text UNIQUE NOT NULL,
  "password_hash" bytea NOT NULL,
  "created_at" timestamptz NOT NULL DEFAULT (now() at time zone 'utc'),
  "updated_at" timestamptz NOT NULL DEFAULT (now() at time zone 'utc')
);

Queries and migrations

After having worked with different ORMs (goent, gorm) and plain go sql code, I find that having a
middle ground is always the most versatile option. This middle ground is about having control over raw SQL
queries and the ability to generate code to run all those queries and represent models in code.

For this matter I chose sqlc which offers a good amount of features like
support for different dbs and db drivers. Regarding db migrations, I prefer to run them isolated from the code. There are countless tools to manage
migrations but recently I've been sticking with go-migrate
which also provides a go library in case I want to integrate the migrations into the code.

To run both tools, I've created tasks that will use their docker images.
Alternatively, there could be a task to install them into the system.

Docker

The template provides a multi-stage Dockerfile. It uses the official golang:1.18 image for building and
a scrath image to copy the binaries.

A docker-compose file can be used to get a db instance and run migrations on it. In this cases the migrations
service waits for the postgres service to be ready, so we only need to run docker-compose up -d.

CI

A github actions workflow is provided to run go fmt, vet, test and gosec.
An initial configuration for dependabot is also provided.

That's it, go take a look at the repo here.

Thanks for reading 👽

Creating a Text-based UI with rust

protium — Thu, 02 Jun 2022 20:48:08 +0000

Intro

Continuing with my last project rq, I recently started to work on
this project card: Implement interactive prompt.
Let's have a look how can we implement a text-based UI with rust.

The ANSI Standard

Before we dive into the wonderful world of terminal emulators and text-based interfaces,
we need to understand what the ANSI escape sequences are:

ANSI escape sequences are a standard for in-band signaling to control cursor location, color, font styling, and other options on video text terminals and terminal emulators.

Source: https://en.wikipedia.org/wiki/ANSI_escape_code

By using ANSI sequences we can tell the terminal emulator how to render text, change the cursor position,
clear the screen, etc. In my last post
I used these sequences to implement a very basic shell for my profile website.

Let's see some examples. Head to your favorite terminal emulator (I use kitty):

echo "\x1b[35;47mANSI? \x1b[0m\x1b[1;32mSI"

Analysis:

\x1b: this is the hexadecimal for ESC (escape). It starts the escape sequence
[: is the control sequence introducer
35;47m: the SGR (Select Graphic Rendition) where 35 sets the foreground color to magenta and 47 sets the background to white. m is the final character specified by the standard
\x1b[0m: it's a special sequence to reset or turn all attributes off
\x1b[1;32m: Sets foreground to green and add the bold attribute

Lets try this in rust:

println!("\x1b[35;47mANSI? \x1b[0m\x1b[1;32mSI");

It should have the same result as with the echo command above.

Another example:

println!("\x1b[2J");

According to the standard [2J erases entire display.

Designing the text-based interface

As a loyal user of (neo)vim, I love the concept of buffers. I want my app to have 2 buffers,
one for the list of HTTP requests and one for the current response:

Let's start from scratch. To enter into the "interactive" mode, we need to clear the whole screen.
For that we will use the following escape sequences:

println!("\r\x1b[2J\r\x1b[H");

\r, carriage return. We go all the way to the left
As explained before, [2J clears the entire screen
[H moves the cursor to home, or the corner top-left of the terminal (col 1, row 1).

Nice, an empty screen.
Next I want to draw the requests the program loaded from the http file
(for context: I have published this article
about how to parse HTTP request with rust). Based on the design, the selected request should print the
HTTP method in green, URL in white and body in orange. If the request is selected, the whole request line should be green.
Let's write a helper function:

fn draw_request(req: &HttpRequest) -> String {
  let mut request_line = format!("{} {} HTTP/{}", colorize(Color::Green, req.method), req.url, req.version);
  if !req.header.is_empty() {
    request_line = format!("{}\r\n{:?}\r\n", request_line, req.headers);
  }
  if !req.body.is_empty() {
    request_line = format!("{}\n{}", request_line, colorize(Color::Orange, req.body.as_str()));
  }
}

If you followed along my last post, you may have an idea of what the colorize function looks like.

Now we are able to print all the request. Since we will use > to visually mark which request is
selected, we need to cater some space for it, hence we will print two empty spaces before each request start:

for req in http_file.requests {
  println!("  {}\n", draw(&req));
}

// go home 
println!("\x1b[H");
// print the cursor >
print!("\x1b[32m>");

Results:

User interaction

The user should be able to navigate between requests with the arrow keys. To achieve this we will need:

Track the current cursor position (column and row)
Move the cursor down or up according to the user input
Redraw the interface (clean the screen and draw)

We can compare this flow to how a game engine works: we evaluate changes and use the engine to draw the game
in its current state.

Since I want to finish this project in less than a month, I have decided to take a look at the different
options in rust for building a TUI. There are a few options that I can use as a TUI "engine".

I opted for tui-rs which is being used by some of my favorite
terminal apps like bottom. I headed over the examples
and used the list example as a base.

After a small refactor:

fn ui<B: Backend>(f: &mut Frame<B>, app: &mut App) {
  // Split the screen into 2 vertical portions
  let chunks = Layout::default()
      .direction(Direction::Horizontal)
      .constraints([Constraint::Percentage(50), Constraint::Percentage(50)].as_ref())
      .split(f.size());

  let items: Vec<ListItem> = app
      .requests
      .iter()
      .map(|req| ListItem::new(draw_request(req)))
      .collect();

  let mut list_block = Block::default().borders(Borders::ALL).title("Requests");
  let list = List::new(items)
      .highlight_style(
          Style::default()
              .add_modifier(Modifier::BOLD)
              .fg(Color::Green),
      )
      .highlight_symbol("> ");

  f.render_stateful_widget(list.block(list_block), chunks[0], &mut app.list);
}

So we are at the same point as when using ANSI escape sequences to render the UI.
This time we can easily implement a key handler and deal with the user interaction:

match event::read()? {
    Event::Key(key) => match key.code {
        KeyCode::Down => app.next(),
        KeyCode::Up => app.previous(),
        _ => {}
    },
}

where app is an instance of the App struct:

struct App {
    list: ListState, // from tui-rs
    requests: Vec<HttpRequest>,
    response_buffer: String,
}

impl App {
    fn next(&mut self) {
        let mut i = self.list.selected().unwrap() + 1;
        if i >= self.requests.len() {
            i = 0;
        }

        self.list.select(Some(i));
    }

    fn previous(&mut self) {
        let mut i = self.list.selected().unwrap();
        if i == 0 {
            i = self.requests.len() - 1;
        } else {
            i -= 1;
        }
        self.list.select(Some(i));
    }
}

Sweet! Now it is possible to navigate between requests:

Sending HTTP requests

Once the app is interactive and we can select a request, we can implement a handler for the Enter key as follows:

// impl App ...
fn selected_request(&self) -> HttpRequest {
    self.requests[self.list.selected().unwrap()].clone()
}

//....

KeyCode::Enter => {
  let req = app.seleted_request();
  let data = match rq_core::request::execute(&req).await {
    Ok(r) => r,
    Err(e) => e.to_string(),
  };
  app.response_buffer = data;
}

I have implemented the execute function in the last post, you can take a look at the source here.

The UI is practically finished: the user can navigate with arrow keys and hit Enter to execute a request.

Bonus: Multi-threading

The problem we are facing now is that, even though the request is executed async with tokio, it is
blocking the main thread because we are awaiting it. If we send a slow request, the UI won't receive updates nor
key events until the request is done.

A proper solution would be to spawn a new thread to perform the request and assign the response to the app response_buffer once it finishes.
If you have worked before with a compiled language like C++ or C#, you will know that we need to access to
the app variable in a thread-safe way by using a mutex (mutual exclusion object):
In short, we want to block any other thread from accessing to the app state at the same time that we want to change it.

Furthermore, we also need to take into account the lifetime checks made by the rust compiler.
If we simply move a variable into a new thread, its ownership is moved and thus the variable is killed once
the thread finishes. The solution? use a pointer to "leak" the app variable into another thread.

Without entering into details that are out of the scope of this article, I will use an Atomically Reference Counted,
from the rust standard library, to create a shared pointer of a Mutex. The multi-thread implementation looks like:

// app_arc: Arc<tokio::sync::Mutex<App>>

KeyCode::Enter => {
  // create a new reference
  let app_arc = app_arc.clone();
  tokio::spawn(async move {
      // the ownership of app_arc was now moved into this thread
      let data = match rq_core::request::execute(&req).await {
          Ok(r) => r,
          Err(e) => e.to_string(),
      };
      // here we use the tokio Mutex to lock our variable since we want to write data into it
      let app = app_arc.lock().await;
      app.response_buffer = data;
      // at this point the thread exists and app_arc is killed.
      // app is also killed, unlocking it.
  });
}

The result:

As you can see, we can navigate while a request is being executed. The UI keeps responsive. Perfect!

This is it for today, I hope you learned something new.

You can check out the repo here.
If you would like to collaborate with this project there are some TODOs over here.

Thanks for reading!

Other articles:

My profile website is now a terminal
An HTTP request parser with rust and pest.rs
My dotfiles: https://github.com/protiumx/.dotfiles
My blog source: https://github.com/protiumx/blog

👽

My profile website is now a terminal

protium — Wed, 25 May 2022 12:51:03 +0000

When I was younger I used to think that my profile website would be a really cool, fully featured website, with shiny colors and animations; built with the latest cutting edge frontend technology...
Turns out that the older I get, the more I prefer a simple terminal. No UI, just text and commands.

The last time I updated my profile website, it looked like this:

It was already pretty minimalistic, right? But not enough. Now my profile website is just a terminal:

Let's see how this was possible.

A pragmatic approach

A few days ago I was shaping this idea on my head and found this cool library: xterms. It's been used by a lot of apps, VS Code being among them. I decided to give it a try to see how complex could it be, so I headed to the docs and started adding the code to my website. As you can see the docs are pretty good, they are surely autogenerated from TS docs but this is good because it means the code itself is well documented.

Before starting coding I set a few requirements:

I don't want to use npm modules. I want my website source to be simple and minimal
I want to make use of javascript modules which are supported by all (relevant) browsers
The terminal commands should be abstract to allow me to remove or add commands at will with a few changes

Then, how do I install xtermjs without using npm? The solution is simple, I host the files. I extracted the files from the npm packages with this command



npm v xterm dist.tarball | xargs curl | tar -xz

and moved package/lib/xterm.js into app/

To use javascript modules, I just needed to import the main.js file as module



<script type="module" src="/app/main.js"></script>

Terminal Commands

Although not using typescript let's say that the terminal commands implement the following interface



interface Command {
  id: string;
  description: string;
  usage: string;
  args: number;
  run: (terminal: Terminal) => Promise<void>;
}

Then we need a command runner that will parse the user input



interface CommandRunner {
    (term: Terminal, userInput: string) => Promise<boolean>;
}

The runner will return false if a command was not found.
Let's now define 1 command:



const lsCommand =   {
  id: "ls",
  description: 'list files',
  usage: '[usage]: ls filename'
  args: 0,
  async run(term, args) {
    for (const file of files) {
      term.write(file.name + '\t\t');
    }
  },
};

Now that we shaped the command, we can think of handling user input.

Terminal basic functionality

The terminal should support:

It should show a prompt
ctrl + l: should clear the terminal
ctrl + c: should send a SIGINT
enter: should run a command from the current user input

The terminal should also handle common errors:

command not found
command with wrong arguments

With this in mind we can start handling the user input.

xterm provides a onKey event which receives a handler function ({ key, domEvent }) => void, so we receive an event per each key press done by the user. This means that we need to track the user input and add each key as a char. When the user presses enter we should evaluate the input we have so far. Pretty straigt forward



let userInput = '';
if (ev.keyCode == 13) {
  await runCommand(term, userInput);
  userInput = '';
  prompt(term);
} else {
  term.write(key);
  userInput += key;
}

NOTE: xterm doesn't render the user input, so we need to do it when it makes sense (not enter, not an arrow key, etc)

Handling the clear-screen can be implemented as



if (ev.ctrlKey && ev.key === 'l') {
  term.clear();
  return;
}

and the SIGINT



if (ev.ctrlKey && ev.key === 'c') {
  prompt(term);
  userInput = '';
  return;
}

At this point we have a pretty basic working terminal, so let's add some more commands

Basic commands

What are the most known commands? For my terminal I want to be able to use cat, ls, rm, exit. But remember that this terminal is actually my profile website, so they should make sense in that context. So I decided the terminal should have a file system, where files are shaped like



interface File {
  name: string;
  content: string;
}

Example



const files = [{ name: "about.md", content: "once upon a time"}];

With this in mind, cat will print the file content, ls will print each file's name and rm will delete the file from the array.

For the exit command we can just close the window from javascript: window.close().

Going further

I have decided that I wanted to have a file named blog.md which should contain my last 5 posts.
To fetch this info, I used the RSS feed xml file generated by hugo for my blog. All I need to do is to fetch the file, parse the xml document and get the title and links of each post:



export async function fecthLastPosts() {
  const res = await fetch('/blog/index.xml');
  const text = await res.text();
  const parser = new DOMParser();
  const xmlDoc = parser.parseFromString(text,"text/xml");
  const posts = xmlDoc.getElementsByTagName('item');
  const lastPosts = [];
  for (let i = 0; i < 5; i++) {
    const title = posts[i].getElementsByTagName('title')[0].childNodes[0].nodeValue;
    const link = posts[i].getElementsByTagName('link')[0].childNodes[0].nodeValue;
    lastPosts.push(title + `\r\n${link}\r\n`);
  }

  files[0].content = lastPosts.join('\n');
}

Now cat blog.md prints my last 5 posts, and thanks to the web link addon of xterm each link is clickeable. Noice.
But why stopping here? Every hackerman terminal should have a whoami command. So this command will just print information about my self.

Also, cool web apps contain photos of cats, so I decided to write a randc command what will open a random photo of a cat.
For this I found this amazing rest API



  {
    id: "randc",
    description: 'get a random cat photo',
    args: 0,
    async run(term, args) {
      term.writeln('getting a cato...');
      const res = await fetch('https://cataas.com/cat?json=true');
      if (!res.ok) {
        term.writeln(`[error] no catos today :( -- ${res.statusText}`));
      }  else {
        const { url } = await res.json();
        term.writeln(colorize(TermColors.Green, 'opening cato...'));
        await sleep(1000);
        window.open('https://cataas.com' + url);
      }
    },
  },

The result:

I think this should do it for a profile terminal. I'm very satisfied with the simplicity of it and the commands I have implemented.
I'll problaly add more commands in the future and also implement streams , just for fun.

What command would you add to your profile terminal?
Go have some fun with it: https://protiumx.dev

Update:

I have refactored the project structure to improve readability and make it more generic.
It also loads your command history from the local storage. All the changes can be seen here: https://github.com/protiumx/protiumx.github.io/pull/1

Update 2:

rm supports glob pattern

Update 3:

added man command
added uname command

An HTTP request parser with rust and pest.rs

protium — Mon, 23 May 2022 21:43:05 +0000

Introduction

I happened to have learned about parsing expression grammars a few days ago and got really excited about writing my own grammar.
As I was missing this VS Code extension when working on neovim, an idea popped up:
What if I write an HTTP grammar and an execute request from an interactive prompt?

Preview:

Preparation

For our grammar, we need to make sure we understand the HTTP standard defined here.



Request = Request-Line ; Section 5.1
        *(( general-header        ; Section 4.5
         | request-header         ; Section 5.3
         | entity-header ) CRLF)  ; Section 7.1
        CRLF
        [ message-body ]          ; Section 4.3

Let's breakout the definition

Request

A Request is conformed by a Request-Line optionally followed by headers and a new line (a Carriage Return and Line Feed) then another new line. After the headers we find the message-body.

Request-Line



Request-Line = Method SP Request-URI SP HTTP-Version CRLF

Here we should fine a Method followed by a white space. Then the Request-URI followed by a white space. Finally the HTTP-Version.

An example of a request can be GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.\r\n (note the end of the line)

Request-Header

Defined as



equest-header = Accept                   ; Section 14.1
      | Accept-Charset           ; Section 14.2
      | Accept-Encoding          ; Section 14.3
      | Accept-Language          ; Section 14.4
      | Authorization            ; Section 14.8
      | Expect                   ; Section 14.20
      | From                     ; Section 14.22
      | Host                     ; Section 14.23
      | If-Match                 ; Section 14.24
      | If-Modified-Since        ; Section 14.25
      | If-None-Match            ; Section 14.26
      | If-Range                 ; Section 14.27
      | If-Unmodified-Since      ; Section 14.28
      | Max-Forwards             ; Section 14.31
      | Proxy-Authorization      ; Section 14.34
      | Range                    ; Section 14.35
      | Referer                  ; Section 14.36
      | TE                       ; Section 14.39
      | User-Agent               ; Section 14.43

let's open one of those sections to check the format.



Authorization = "Authorization" ":" credentials

Basically this header is the word Authorization followed by : and then the credentials.

Message-Body



message-body = entity-body
                    | <entity-body encoded as per Transfer-Encoding>

For simplicity let's assume that the body can be anything that is not a request line nor a header.

Now we have enough information to write our http grammar, more specific a set of rules.

Pest

To write our grammar we will use pest. From the website:

pest is a general purpose parser written in Rust with a focus on accessibility, correctness, and performance. It uses parsing expression grammars (or PEG) as input, which are similar in spirit to regular expressions, but which offer the enhanced expressivity needed to parse complex languages.

Sounds great!

Pest has its own syntax for writing grammar rules. A rule is defined as follows:



my_rule = { ... }

Let's create a new rule my_rule= { "test" }. This rule will match the word test inside an input. So if we want to parse a file that only contains "test" or new lines we need to define the shape of a line with the delimiters SOI and EOI (start and end of input respectively)



testy_file = {
    SOI
    ~ (my_rule | NEWLINE)*
    ~ EOI
}

Let's test this out in the online grammar editor:

The parser has identified a testy_file that contains to matches for the rule my_rule. So far so good.

HTTP Grammar

Now we are ready for writing our grammar (are we?). We need to write this grammar as close as possible to the standard, since it's defined in a similar way, with rules. We can imagine something like this



request = { 
    request_line ~
    headers? ~
    NEWLINE ~
    body?
}

Our request rule says: a request_line optionally followed by headers followed by a new line, then optionally followed by the body. With optionally I make reference to the ? mark which is a repetition operator (those who are familiar with regex expressions should understand this right away) and means that something can occur zero or one times.
For our request_line we will match the standard as follows:



request_line = _{ method ~ " "+ ~ uri ~ " "+ ~ "HTTP/" ~ version ~ NEWLINE }

We are talking about a method followed by a blank space then the uri then another blank space then the http version followed by a NEWLINE. NEWLINE is a pest built-in rule that matches "\n" | "\r\n" | "\r".

Let's break the request line.

For method we just need to match any of the http methods defined in the standard



Method = "OPTIONS"                ; Section 9.2
        | "GET"                    ; Section 9.3
        | "HEAD"                   ; Section 9.4
        | "POST"                   ; Section 9.5
        | "PUT"                    ; Section 9.6
        | "DELETE"                 ; Section 9.7
        | "TRACE"                  ; Section 9.8
        | "CONNECT"                ; Section 9.9

Therefore our method rule should be



method = { ("GET" | "DELETE" | "POST" | "PUT" | ...) }

Here our rule says: match the exact word GET or DELETE or ... you get the idea.

Next, for simplicity purpuses, we will say that the uri is anything that is not a blank space



uri = { (!whitespace ~ ANY)+ }
whitespace = _{ " " | "\t" }

Note: here we also defined that the white space can be a space or a tab. Note that _ means that the rule is silent; it does not produce tokens.

The HTTP version goes like:



version = { (ASCII_DIGIT | ".")+ }

Meaning: any digit or a dot character. In our request_line the version is defined as the word "HTTP/" followed by the version.

Finally, we require a NEWLINE at the end of our request line.

Let's test this out on the editor.

Perfect! Note that the whitespace rule does not produce a token, we only see method, uri and version. Let's continue with the headers.



header = { header_name ~ ":" ~ whitespace ~ header_value ~ NEWLINE }
header_name = { (!(":" | NEWLINE) ~ ANY)+ }
header_value = { (!NEWLINE ~ ANY)+ }

In here we say that a header_name is anything except for : , because this character is a header separator, it delimits the header name. We also don't want a new line, we will see why bellow.
The name is followed by the separator : and then 1 white space.
The header_value is anything except for a new line, because the new line delimits a header. After the value we encounter a new line.

Let's test this again:

Here we can se that the parser matches 1 header as { header_name: "auth", header_value: "token" }. Did you notice that it also say headers > header ? That is because we also want a rule that can matches a 1 or more header rules. We define this rule as:



headers = { header+ }

This way the pest parser will produce a token headers with a list of header matches.

Lastly the body:



body = { ANY+ }

The body is anything after the headers

So far our grammar can parse 1 request from the input. Coming back to the original idea for this post, we want to be able to parse multiple http requests from a file. But here we have a conflict because our body rule will match anything after the headers, and that anything can be another request.
To solve this problem we need the help of a ✨ delimiter ✨

The .http file syntax

To delimit each http request in our file we will make use of 3 # symbols (as the VS Code rest client does).
Let's go add this to our grammar:



delimiter = { "#"{3} ~ NEWLINE+ }

Our delimiter is exactly 3 number signs followed by 1 or more new lines. Then we can say that the body is anything except for a delimiter:



body = { (!delimiter ~ ANY)+ }

Let's test this:

Nice! So finally we can define what our http file should look like:



file = { SOI ~ (delimiter | request)* ~ EOI}

Our http file is composed by delimiters or requests, 0 or more of them.
We have completed our http grammar, now it's time to get rusty

Parsing http files

We will refer to this example for setting up the base code



extern crate pest;
#[macro_use]
extern crate pest_derive;

use pest::Parser;

#[derive(Parser)]
#[grammar = "grammar.pest"]
struct HttpParser;

We will use the following file for testing



GET https://protiumx.github.io HTTP/1.1
authorization: token

###

POST https://rq-rust.free.beeceptor.com/api HTTP/1.1

{
  "hello": "hola"
}

Note: I set up a mock api in https://beeceptor.com/. It might not be available by the time you are reading this. But you can use any endpoint that accepts POST.

Let's go ahead an parse the file



fn main() {
    let unparsed_file = fs::read_to_string("test.http").expect("cannot read file");

    let file = HttpParser::parse(Rule::file, &unparsed_file)
        .expect("unsuccessful parse") // unwrap the parse result
        .next().unwrap(); // get and unwrap the `file` rule; never fails

        println!("{:#?}", file);
}

we get:



Pair {
    rule: file,
    span: Span {
        str: "GET https://protiumx.github.io HTTP/1.1\nauthorzation: token\n\n###\n\nPOST https://rq-rust.free.beeceptor.com/api HTTP/1.1\n\n{\n  \"hello\": \"hola\"\n}\n",
        start: 0,
        end: 142,
    },
    inner: [
        Pair {
            rule: request,
            span: Span {
                str: "GET https://protiumx.github.io HTTP/1.1\nauthorzation: token\n\n",
                start: 0,
                end: 61,
            },
            inner: [
                Pair {
                    rule: method,
                    span: Span {
                        str: "GET",
                        start: 0,
                        end: 3,
                    },
                    inner: [],
                },
...

The Pair struct has a pretty clear structure and we could use to print each inner Pair. But we want to get rusty so we will put some effort.

We will define an HttpFile struct. This struct will contain a vector of HttpRequest.
Let's define them:



use std::collections::HashMap;

#[derive(Debug)]
pub enum HttpMethod {
    Get,
    Post,
    Put,
    Delete,
    // ...
}

#[derive(Debug)]
pub struct HttpRequest {
    pub method: HttpMethod,
    pub url: String,
    pub version: String,
    pub headers: HashMap<String, String>,
    pub body: String,
}

#[derive(Debug)]
pub struct HttpFile {
    pub requests: Vec<HttpRequest>,
}

And to get even more rusty we will implement the TryFrom trait to each struct. Let's implement it for the HttpFile



impl<'i> TryFrom<Pair<'i, Rule>> for HttpFile {
    type Error = Error<Rule>;

    fn try_from(pair: Pair<Rule>) -> Result<Self, Self::Error> {
        let iterator = pair.into_inner();
        let mut requests = vec![];
        for item in iterator {
            match item.as_rule() {
                Rule::EOI => {
                    break;
                }
                Rule::request => {
                    requests.push(item.try_into()?);
                }
                _ => {}
            }
        }
        Ok(Self { requests })
    }
}

Note: Pair<'i, Rule> has the lifetime of the input.
As we know, the file rule can contain 0 or more of (delimiter | request). This means that in its content we can find: delimiter, request or End of File.
Here we will try to parse each request, so we need the TryFrom trait for the HttpRequest struct



impl<'i> TryFrom<Pair<'i, Rule>> for HttpRequest {
    type Error = Error<Rule>;

    fn try_from(pair: Pair<'i, Rule>) -> Result<Self, Self::Error> {
        let mut iterator = pair.into_inner();
        let mut ret = Self {
            method: iterator.next().unwrap().try_into()?,
            url: iterator.next().unwrap().as_str().to_string(),
            version: iterator.next().unwrap().as_str().to_string(),
            headers: HashMap::new(),
            body: String::new(),
        };

        for item in iterator {
            match item.as_rule() {
                Rule::headers => {
                    ret.parse_headers(item.into_inner());
                }
                Rule::body => {
                    ret.body = item.as_str().to_string();
                }
                _ => {
                    unreachable!();
                }
            }
        }

        Ok(ret)
    }
}

Each request can have 5 inner matches: method, url, version, headers and body.
The first 3 do not have inner rules, so we can just extract them as &str. After this, the iterator can only have headers or body pairs. For the headers we perform a similar operation:



impl HttpRequest {
    fn parse_headers(&mut self, pairs: Pairs<Rule>) {
        for item in pairs {
            let mut kv = item.into_inner();
            let key = kv.next().unwrap().as_str().to_string();
            let value = kv.next().unwrap().as_str().to_string();
            self.headers.insert(key, value);
        }
    }
}

Grapes! We are pretty much done with the parser. Now let's use the parsed content to make the requests.

reqwest

After a quick check of the available rust http client libraries I opted for reqwest. It has a pretty simple API and it seems to be among the most used libraries for this matters. But I'm a bit concerned about all its dependencies so I might try ureq later.

For this part I just followed the examples on the docs website and ended up with this code:



#[derive(Default)]
pub struct HttpClient {
    pub client: reqwest::blocking::Client,
}

impl HttpClient {
    pub fn new() -> Self {
        let mut headers = header::HeaderMap::new();
        headers.insert(
            header::CONTENT_TYPE,
            header::HeaderValue::from_static("application/json"),
        );
        headers.insert(
            header::ACCEPT,
            header::HeaderValue::from_static("application/json"),
        );

        let client = reqwest::blocking::Client::builder()
            .timeout(Duration::from_secs(10))
            .default_headers(headers)
            .no_gzip()
            .build()
            .unwrap();
        HttpClient { client }
    }

    pub fn execute(&self, req: &HttpRequest) -> Result<(), Box<dyn std::error::Error>> {
        let request = self
            .client
            .request(Method::from_str(req.method.to_string().as_str())?, &req.url);

        let headers: header::HeaderMap = (&req.headers).try_into()?;

        let body = req.body.clone();
        let res = request.headers(headers).body(body).send()?;

        println!("{}\n", res.text()?);

        Ok(())
    }
}

What we can see in here:

Use the ClientBuilder to add some default headers and config for the request
Parse our HttpMethod into reqwest::Method. Note: HttpMethod implements the Display trait.
Parse our HashMap<String, String of headers into reqwest::header::HeaderMap. This is possible because it implements the TryFrom<HashMap<String, String>> trait.
Finally clone the body and send the request. We output the body of the request as text (this consumes the body)

Note: We are not trimming the body, perhaps we should.

No we are ready for the last part: the interactive prompt

Lazyness kicks in

At this point I felt a bit lazy to implement the prompt so I searched for existing solutions, one of them being inquire. It has a pretty straight forward API, so to show an interactive select we just need this lines:



    let http_file = parse(&file_content).unwrap();

    let request = Select::new("Select requests to execute:", http_file.requests)
        .prompt()
        .unwrap();

    let client = HttpClient::new();
    client.execute(&request)?;

Note that the HttpRequest implements the Display trait.

The result:

Gorgeous.

Unintroduction

This project turned out to be quite fun! I ended up naming it rq and you can see its source here

My goal is to have a full interactive prompt that allows you to execute many request without existing the prompt mode.
If you would like to collaborate I have created a few TODO cards here.

That's it. Thanks for reading 👽

Others posts:

Automate an articles section in your github.io page

protium — Fri, 13 May 2022 17:26:41 +0000

Today I wanted to update my github page to show a list of my
last Medium articles. I ended up automating it.

First approach

My first attempt was fetching the RSS feed using the URL medium.com/feed/@username and parse
the xml document. But I was hit with a charming CORS error.
Now what?

I remembered that I'm using a github action to update a similar section on my github profile.
So, what if I tell this action that the README file is called index.html?

blog-post-workflow

The github action supports a readme_path parameter. After
a quick dive in its source code I noticed that this file could be anything, not necessarily a markdown file. Problem solved!

Let's add the articles section into the html:

<section class="content__articles">
  <p>&gt; last published articles:</p>

  <ul>
    <!-- BLOG-POST-LIST:START -->
    <!-- BLOG-POST-LIST:END -->
  </ul>
</section>

Since the action is intended for markdown files, the default template for the posts links is [title](url) but we need html code.
Luckily the github action also provides a template param. So let's change the template to generate list items:

<li><a href="$url" target="_blank">$title</a></li>$newline

That's it!

Our workflow configuration should be:

name: Update Medium articles
on:
  schedule:
    - cron: '0 0 * * 0' # Runs once a week
  workflow_dispatch:

jobs:
  update-articles-section:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: gautamkrishnar/blog-post-workflow@master
        env:
          GITHUB_TOKEN: $
        with:
          feed_list: "https://medium.com/feed/@protiumx"
          readme_path: "https://raw.githubusercontent.com/protiumx/blog/main/articles/004/index.html"
          template: '<li><a href="$url" target="_blank">$title</a></li>$newline'

Check the file here.
NOTE: I added the workflow_dispatch trigger so I can trigger the action from the github UI.

That's all. I few lines of code and we added an automated section for our github page (or any page).

Publish your blog articles everywhere with this github action

👽

kitty + zsh + powerlevel10k = ✨ aesthetics ✨

protium — Mon, 09 May 2022 19:49:40 +0000

Who doesn't like a good looking terminal and useful layouts and keymaps?
This has always been my goal since I started using linux as main OS.

I have used lots of different terminals, plugins and settings over the years but I think now
I achieved the glory with my current setup:

Let's have a look to the configuration.

What do I use?

Terminal: Kitty
Shell: zsh
zsh config management: Oh my Zsh!
zsh theme: powerlevel10k
editor: neovim

Configuration

Terminal

To achieve a clean look in the terminal, I changed the opacity of kitty, removed the window title bar,
and changed margins. Here the configuration needed:



background_opacity 0.76
draw_minimal_borders yes
window_padding_width 2
window_border_width 0
hide_window_decorations yes
titlebar-only yes
active_border_color none

I have also added changes for the tabs bar in order to make it look minimalist:



tab_bar_edge top
tab_bar_background none
tab_bar_style powerline
tab_powerline_style slanted
tab_title_template "{fmt.fg.c2c2c2}{title}"

active_tab_title_template "{fmt.fg._fff}{title}"
active_tab_foreground #fff
active_tab_font_style bold-italic
active_tab_background #8631B4

inactive_tab_foreground #c2c2c2
inactive_tab_background #8631B4

In order to move quick between splits I added the following mappings:



map cmd+shift+up neighboring_window up
map cmd+shift+left neighboring_window left
map cmd+shift+right neighboring_window right
map cmd+shift+down neighboring_window down

powerlevel10k

There is not much to mention here since I have used the powerlevel10k script to configure it.
Once you install powerlevel10k it should init the configuration wizard. If you want to re-configure
do so by running p10k configure in your terminal.

You can check my powerlevel10k config file here.

neovim

Plugins:

vim-airline

In this case, I got rid of many highlights background colors



highlight LineNr ctermbg=none
highlight Normal ctermbg=none
highlight NonText ctermbg=none
highlight SignColumn ctermbg=none
highlight VertSplit ctermbg=none ctermfg=98 cterm=none

I have also setup a color column to show a visual limit at 100 chars



set colorcolumn=100
highlight ColorColumn ctermbg=93

My vim-airline config goes like this:



let g:airline_theme='selenized_bw'

" Show git branch
let g:airline#extensions#branch#enabled=1

let g:airline#extensions#hunks#enabled=0
let g:airline_powerline_fonts=1
let g:airline_detect_spell=0

" Short version for modes
let g:airline_mode_map = {
      \ '__'     : '-',
      \ 'c'      : 'C',
      \ 'i'      : 'I',
      \ 'ic'     : 'I',
      \ 'ix'     : 'I',
      \ 'n'      : 'N',
      \ 'multi'  : 'M',
      \ 'ni'     : 'N',
      \ 'no'     : 'N',
      \ 'R'      : 'R',
      \ 'Rv'     : 'R',
      \ 's'      : 'S',
      \ 'S'      : 'S',
      \ ''     : 'S',
      \ 't'      : 'T',
      \ 'v'      : 'V',
      \ 'V'      : 'V',
      \ ''     : 'V',
      \ }

That's it! The setup is fairly simple and the results are quite good.

You can have a look at all my config and dotfiles in here.

PS: this article has its source on github.
I'm using a github action to publish it to different platforms. Read more
here

👽

blogpub@v0.4.1: host media on github for your blog posts

protium — Sun, 20 Feb 2022 17:56:37 +0000

blogpub@v0.4.1: host media on github for your blog posts

Hey everyone, this is just a small update on the blogpub action.

Relative paths

In release v0.4.1 I added support for relative
paths which means you can now use github to host the media you use in your blog posts.
How does it work? Just use relative paths (to the markdown file) and you are ready to go.
E.g.

The action replaces all relative paths with the raw content github urls.
In the example .magic.gif is replaced with https://raw.githubusercontent.com/protiumx/blog/articles/blog-002/magic.gif.
Check out the code here.

That's all!

What other feature would you find useful for this action?
Let me know in the comments.

Publish your blog articles everywhere with this github action

protium — Sun, 16 Jan 2022 14:35:02 +0000

Long ago I made this comment in this article:

This is a really good idea with great potential. 
Imagine a standarized API for different blogs. 
You can automatize publishing and editing, multiple collaborators.
And use a git provider as unique source of content.

And you can also make your git repo as some sort of blog.
I'll use it for my next posts for sure.

Thanks!

It was an idea that I had left on the back of my head and I didn't come back to it because I wasn't writing articles actively. But this has changed in the last months and I have published 6 articles during December. So I decided to revisit the idea and finally develop it.

blogpub

blogpub is a github action developed with typescript with the purpose of allowing you to use a github repository as source of truth for your blog posts. The action supports:

Automatically publishing to Medium and DEV.to
Templating
Metadata/config section (published, tags, title, etc)

In fact, this article was automatically published by blogpub and you can see it's source in this folder.

Template Support

Sometimes we want to add different content to our blog posts depending on the platform, for instance for dev.to I want to use liquid tags, which medium doesn't support.
For this purpose I have used handlebars to make use of conditionals. Example:

{{#if medium}}
This is only for Medium
{{/if}}
{{#if devto}}
This is only for DEV.to
{{/if}}

For the first release of the action, the template context contains:

{
  medium: boolean;
  devto: boolean;
}

Article metadata

I have added support for a metadata section (similar to dev.to) where we can specify the following attributes:

title: [string] The title of the article. If not specified, the first H1 heading will be used.
description: [string] Description for dev.to API.
tags: [string] Comma separated tags. Note: Medium allows up to 5 tags whereas Dev.to only 4.
license: [string] Medium license type. Refer to Medium API Docs. Default: public-domain
published: [boolean]. Default: true

Example:

---
title: "First blogpub test"
tags: test, ci
---
# I'm using `blogpub`

Adopting CI/CD practices for blogging

Imagine the following scenario:

Write an article
Create PR
CI checks the spelling
Your peers review the article and propose changes or approve it
Merge the PR to main and it gets published everywhere
Use send tweet action to promote your new article.

I can imagine this workflow in a company where different colleagues write articles together. I like the idea that we could also automatize tweets, this is why blogpub outputs the URLs of the article after publishing it.

Usage

Ideally you want to run this action for every push to your main branch. This setup should be enough.

name: 'publish'

on:
  push:
    branches:
      - 'main'
    paths:
      - 'articles/*'
jobs:
  publish:
    name: publish new article
    runs-on: ubuntu-latest    
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: blogpub
        uses: protiumx/blogpub@v0.2.3
        with:
          articles_folder: articles
          devto_api_key: ${{secrets.DEV}}
          gh_token: ${{secrets.GH}}
          medium_token: ${{secrets.MEDIUM}}
          medium_user_id: <user_id>

Few things to take into account:

We only want to run the action for files inside a folder.
You need to setup repository secrets with your tokens and api keys.

Check this blog source to see this in action.

What is missing?

For the first release I just wanted to be able to publish the articles in Medium and DEV.to. But there are a few features that would be handy:

Uploading images: Imagine also versioning images and then uploading those images automatically
Support for updates: If you want to make a correction, it should be possible to update the source and it should get reflected in all the platform the article was published to.
Platform specific configuration: maybe we want different tags per platform
Support multiple articles: at the moment blogpub publishes only 1 article per run.

What else would you like to have? Let me know in the comments!

That's it!
As usual, any help is well received and I have a TODO list if you would like to collaborate with this project.

protiumx / blogpub

Github action to publish your blog articles from Markdown to Medium or Dev.to

blogpub

Github action to publish your blog articles to Medium or Dev.to using their respective REST APIs The action searches for markdown files in the commit of the push event and uses first md file that finds.

Updating articles

Currently it's not supported to update the articles on the different platforms. If the markdown file found in the push event already exists on the commit before, the action will skip it. This avoids publishing the article again.

Pre-requisites

In order to interact with both platforms API's you will need:

Medium integration token
Dev.to API Key
Github access token with reading permissions

Usage

The action will grab an markdown file from a push event to a branch. The following workflow configuration will publish articles that are committed to the main branch:

on:
  push:
    branches:
      - 'main'
    paths:
      - 'articles/*'

Note…

View on GitHub

🤖

autoenv: relax, ansible will setup your dev environment for you

protium — Thu, 30 Dec 2021 16:54:59 +0000

Yesterday I published: Bash + GNU Stow: take a walk while your new macbook is being set up. I was already taking a look at ansible so I decided to finish my playbooks to match exactly what my dotfiles script does.

So today we are going to learn how to automate our dev environment with ansible.

Bootstrapping

Similar to my dotfiles script, we need to install necessary dependencies before running the playbooks. For that we need these dependencies:

xcode
brew
git, python

Putting into code:

bootstrap() {
  info "Bootstraping..."
  info "Installing xcode"
  install_xcode

  info "Installing HomeBrew"
  install_homebrew

  info "Installing python3"
  install_brew_formula "python3"

  info "Installing git"
  install_brew_formula "git"

  PATH="/usr/local/bin:$(/opt/homebrew/bin/python3 -m site --user-base)/bin:$PATH"
  export PATH

  info "Installing pip"
  curl https://bootstrap.pypa.io/get-pip.py | python3
}

Installing ansible

I'm using virtualenv to fetch all ansible dependencies, without installing them globally.

python3 -m pip install --user virtualenv
virtualenv venv
. venv/bin/activate

info "Installing Ansible"
pip install ansible

info "Setting up Ansible"
ansible-galaxy collection install -r setup/requirements.yml

Requirements are:

- name: community.crypto
- name: community.general

The collection community.crypto will be used to generate SSH keys.

At this point we have all our requirements. Let's write the playbook.

autoenv playbook

To replicate my .dotfiles commands, I have set up the following tasks:

[+] Installing all the apps, `brew` packages and casks

For this we can use the community.general. We can define some list variables with all the apps/packages/apps we use. Then we execute as follows:

- name: Install Homebrew formulas
  community.general.homebrew:
    name: "{{ homebrew['formulas'] }}"
  tags: [packages, homebrew_formulas]

Same for taps and casks.
For App store apps we use mas and ansible.builtin.command to run a shell command in loop:

- name: Install app store apps
  ansible.builtin.command: "mas install {{ item }}"
  loop: "{{ homebrew['mas'] }}"
  tags: [packages, mac_app_store]

[+] Writing all my macOS settings

Here we will need community.general.osx_defaults. We need to define a list with settings, having domain, key, value and type for each setting. E.g.

- domain: com.apple.TimeMachine
  key: DoNotOfferNewDisksForBackup
  name: Disable prompting to use new exteral drives as Time Machine volume
  type: bool
  value: 'true'

After defining all our settings, this task is defined as follows:

- name: Set macOS default settings
  community.general.osx_defaults:
    domain: "{{ item['domain'] }}"
    key: "{{ item['key'] }}"
    type: "{{ item['type'] | default(omit) }}"
    value: "{{ item['value'] }}"
  loop: "{{ defaults }}"
  tags: system_settings

[+] Stowing dotfiles

I've seen many setups with ansible using the file copy functionality to manag dotfiles, but this is not suitable for me. I prefer using stow so any change on the symlinks can be pushed to the git repo.

I opted for using the same bash script from my dotfiles repo and call the script as a command:

- name: Install Dotfiles
  ansible.builtin.command: sh ~/.autoenv/dotfiles/install.sh
  tags: dotfiles

[+] Setting `vs code` as default for all the source code extensions

For this I just needed to set a list variable with all the extension and then loop into a shell command:

- name: Set VSCode as default editor
  ansible.builtin.shell: |
    local exts=("{{ fileExtensions | join(' ') }}")
    for ext in $exts; do
    duti -s com.microsoft.VSCode $ext all
    done
    exit 0
  tags: settings

[+] Installing `vim-plug`

Same as before, another shell command will do the trick.
But here I also wanted to check if vim-plug was already installed. And for that ansible provides a file stat api and conditional tasks using the when keyword.

- name: Check vim-plugged file
  stat:
    path: ~/.local/share/nvim/site/autoload/plug.vim
    register: vim_plug

- name: Install neovim plugin manager
  ansible.builtin.shell: sh -c 'curl -fLo $HOME/.local/share/nvim/site/autoload/plug.vim --create-dirs https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim'
  when: not vim_plug.stat.exists

[+] Generating SSH keys

As mentioned before, here we make use of the community.crypto collection to generate a ssh key.

- name: Create SSH keys
  community.crypto.openssh_keypair:
    path: "{{ ansible_user_dir }}/.ssh/id_{{ item }}"
    passphrase: "{{ ssh_passphrase }}"
    type: "{{ item }}"
    size: 4096
    comment: "{{ ansible_user_id }}@{{ ansible_hostname }} {{ ansible_date_time['date'] }}"
  loop: "{{ ssh_key_types_to_generate | split(',') }}"
  loop_control:
    label: "{{ item }}"

Note: ssh_passphrase and ssh_key_types_to_generate are variables.

[+] Uploading keys to github

In this task I want to upload my ssh key to github. For that I need a github token and use the community.general collection:

- name: Register SSH key with Github
  vars:
    github_keys:
      - "{{ ansible_user_dir }}/.ssh/id_ed25519.pub"
    pubkey: "{{ lookup('first_found', github_keys, errors='ignore') }}"
  community.general.github_key:
    name: "{{ ansible_user_id }}@{{ ansible_hostname }}"
    pubkey: "{{ lookup('file', pubkey) }}"
    state: present
    token: "{{ github['personal_token'] }}"
  tags: github

[+] Installing pip packages

To install our pip packages we can use the ansible.buitin.pip command and declare a list variable with the packages to install:

- name: Install Python packages
  ansible.builtin.pip:
    executable: "{{ pip }}"
    name: "{{ pypi_packages }}"
    extra_args: --user
  tags: [core, python]

That's it.

Post install

So I had a few things left off that weren't suitable to run with ansible since they need some interaction: running rustup-init, installing nvim plugins and restarting the system. Basically:

nvim +PlugInstall +qall

if ! hash rustc &>/dev/null; then
  info "Triggering Rust-up"
  rustup-init
fi

info "Done"
info "System must restart. Restart?"

select yn in "y" "n"; do
  case $yn in
      y ) sudo shutdown -r now; break;;
      n ) exit;;
  esac
done

We are completely done 🤖

Conclusions

I decided to try ansible out of curiosity to manage my dev environment. It is a very robust and versatile solution but it feels like an overkill for this task. I definitely like to set configuration files in yaml but my configurations/variables inside bash files are not too bloated to justify the switch (though I wrote the whole thing for ansible already). For now I'd prefer to continue using my bash+stow solution.

What do you think? Would you rather use ansible?

protiumx / autoenv

Setup new dev machines with Ansible

autoenv

My ansible playbooks to setup macOS laptos with a development environment.

Running

export GITHUB_TOKEN="<- token ->"
curl -sO https://raw.githubusercontent.com/protiumx/autoenv/main/autoenv

The script will install the initial requirements:

xCode
Hombrew
Git
Python3 and PIP
Virtual env
Ansible

From there, ansible takes over with the autoenv playbook. When ansible is done, the post-install script run commands that are not suitable for ansible.

Github Token

ansible will upload the ssh key to github, for that you need to export a GITHUB_TOKEN before running the scripts.

Customization

Most of the customizable configs reside on the grou-vars definitions. You can check all the system settings, brew packages/casks and app store apps that will be installed.

View on GitHub

👽

Bash + GNU Stow: take a walk while your new macbook is being set up

protium — Wed, 29 Dec 2021 14:06:43 +0000

I'm addicted to automation. I said it. Okay?

Today we are going to learn how we can automatize all our macOS settings, apps and homebrew packages using bash and stow (and a few other cli tools).

Intro

As developers we might encounter ourselves setting up a new macOS environment quite often (I had to do it 2 times so far this year). We tend to accumulate dotfiles pretty easily and keeping them in sync across multiple machines becomes painful.
And not only that. We also accumulate a galaxy of different packages, tools and apps that we use on daily bases to work.

Oh and all the macOS settings that we surely need? our poor souls.

In order to reduce the pain and enjoy life we can make use of different tools and our coding skills to automatize all this process.

The objective

When I receive a new Macbook at a new job, all I want to do is:

Login in the app store
Download and run a magic script
Go out and enjoy the sun while the macbook is being set up

Bootstraping

So first we need a few tools to start setting up everything. As we already know xcode, brew and git are a must. Here we start with:

xcode-select --install
sudo xcodebuild -license accept

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

brew install git

Brew taps, casks and formulas

We want to install all the packages and apps we need.
To list all your currently installed packages (formulae) and apps (casks) you can run:

brew list

we also need to list all the taps (Third-Party Repositories) we use:

brew tap

Perfect. With this information we can code some for loops to iterate on the taps, casks and formulas:

apply_brew_taps() {
  local tap_packages=$*
  for tap in $tap_packages; do
    if brew tap | grep "$tap" > /dev/null; then
      warn "Tap $tap is already applied"
    else
      brew tap "$tap"
  done
}

install_brew_formulas() {
  local formulas=$*
  for formula in $formulas; do
    if brew list --formula | grep "$formula" > /dev/null; then
      warn "Formula $formula is already installed"
    else
      info "Installing package < $formula >"
      brew install "$formula"
    fi
  done
}

install_brew_casks() {
  local casks=$*
  for cask in $casks; do
    if brew list --casks | grep "$cask" > /dev/null; then
      warn "Cask $cask is already installed"
    else
      info "Installing cask < $cask >"
      brew install --cask "$cask"
    fi
  done
}

Note: in order to keep our scripts some how idempotent, we want to check if the formulas/casks/taps are already installed.

These functions will take a list and run brew install. How do we use them?

# apply the taps first
taps=(homebrew/cask)
apply_brew_taps "${taps[@]}"

# install casks
apps=(firefox docker)
install_brew_casks "${apps[@]}"

# install formulas
packages=(curl go)
install_brew_formulas "${packages[@]}"

OK. But some of my apps are not available in hombrew 😡

Installing App Store apps

For all the apps that cannot be found as brew casks we can use mas (Mac App Store cli).
This cli tool needs the IDs of the apps. To check the apps you have installed simply run:

mas list

If not all your apps are listed you can also search for them and get their IDs:

mas search Goodnotes

So again, writing some pretty code:

masApps=(
  "937984704"   # Amphetamine
  "1444383602"  # Good Notes 5
  "768053424"   # Gappling (svg viewer)
)

install_masApps() {
  info "Installing App Store apps..."
  for app in $masApps; do
    mas install $app
  done
}

Neat!

So far we have installed all our apps and cli tools.
Let's see how we can automatize our macOS settings.

`defaults`: macOS settings from the terminal

This tool will help us to read and write settings. For that, you need to know the domain, key and type of the setting. Usually a google/duckduckgo/ecosia search should help you to find this information. If not, defaults provides some commands to help you search for the settings.
Let's say I want to change some finder setting. First, find its domain:

defaults domains | grep finder

Read all the settings available for finder

defaults read com.apple.finder

I want to change ShowExternalHardDrivesOnDesktop but I don't know its type

defaults read-type com.apple.finder ShowExternalHardDrivesOnDesktop

Type is boolean. So finally we can set this to false

  defaults write com.apple.finder ShowExternalHardDrivesOnDesktop -bool false

(You can see all my settings in my dotfiles repo,link below)

Lastly, and this is a personal preference, I want to set vscode as default application for all my source code files. To do so we will use duti.

code_as_default_text_editor() {
  local extensions=(
    ".c"
    ".cpp"
    ".js"
    ".jsx"
    ".ts"
    ".tsx"
    ".json"
    ".md"
    ".sql"
    ".html"
    ".css"
    ".scss"
    ".sass"
    ".py"
    ".sum"
    ".rs"
    ".go"
    ".sh"
    ".log"
    ".toml"
    ".yml"
    ".yaml"
    "public.plain-text"
    "public.unix-executable"
    "public.data"
  )

  for ext in $extensions; do
    duti -s com.microsoft.VSCode $ext all
  done
}

Settings: check.

Managing dotfiles with GNU Stow

stow is simply a symlink manager that let us define a folder structure (package) for our dotfiles and then symlink them following that structure. So let's say one of your dotfiles is ~/.config/kitty/kitty.conf, this will be our kitty package. In order to place this file in the correct path, we define the package as as

dotfiles/kitty/.config/kitty/kitty.conf

(assuming all our dotfiles are in a folder called dotfiles)
We can simulate stow to see the resulting symlinks

cd dotfiles
stow -nSv kitty

If the output looks correct, we symlink the package with

stow --target $HOME kitty

Note: we want to use stow only for files, since a folder like .config will always contain many other files/folders, we don't want to symlink this folder. Therefor we need to first remove existing files (the ones we want to stow) and check that the needed directories exist. So, some lovely code for that:

stow_dotfiles() {
  local files=(
    ".zprofile"
    ".gitconfig"
    ".zshrc"
    ".vimrc"
  )
  local folders=(
    ".config/nvim"
    ".config/kitty"
  )
  info "Removing existing config files"
  for f in $files; do
    rm -f "$HOME/$f" || true
  done

  for d in $folders; do
    rm -rf "$HOME/$d" || true
    mkdir -p "$HOME/$d"
  done

  local dotfiles="kitty nvim"
  info "Stowing: $dotfiles"
  stow -d stow --verbose 1 --target $HOME $dotfiles
}

So far so good, all our dotfiles are versioned and any change on them can be pushed and synced in all our macOS machines with git.
What else did I want to do?
...

Setting up Oh My Zsh

I use oh my zsh along with powerlevel10k and this is how my terminal looks

pretty, right?

So I need to install this scripts on my system, for which I have more lovely code:

install_oh_my_zsh() {
  if [[ ! -f ~/.zshrc ]]; then
    info "Installing oh my zsh..."
    ZSH=~/.oh-my-zsh ZSH_DISABLE_COMPFIX=true sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

    chmod 744 ~/.oh-my-zsh/oh-my-zsh.sh

    info "Installing powerlevel10k"
    git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ~/.oh-my-zsh/custom/themes/powerlevel10k
  else
    warn "oh-my-zsh already installed"
  fi
}

Note: my powerlevel10k configuration is part of my dotfiles. If you don't have any, you can run p10k configure and follow the configuration process.

Installing (Neo)Vim plugins

Lastly, I also want to automatically install all my neovim plugins. First I need to install the plugin manager (I use vim-plug) and then I want to install all the plugins defined on my dotfiles

install_neovim() {
  info "Installing NeoVim"
  install_brew_formulas neovim

  info "Installing Vim Plugged"
  sh -c 'curl -fLo "${XDG_DATA_HOME:-$HOME/.local/share}"/nvim/site/autoload/plug.vim --create-dirs https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim'
}

# Install plugins and quit
nvim +PlugInstall +qall

Eureka. My dev environment is (almost) fully automated. So coming back to the objectives, this is the magic script:

curl -sO https://raw.githubusercontent.com/protiumx/.dotfiles/main/dotfiles

That's it. My life quality has been improved by 2Π%.

And here my dotfiles repo

protiumx / .dotfiles

My automatic dev machine setup scripts and configs

dotfiles

Set up dev environment in a macOS machine. This script installs all the packages and apps I use, stow my dotfiles and sets all my preffered macOS configurations.

Check my Medium article.

Installing

Run the dotfiles script:

curl -sO https://raw.githubusercontent.com/protiumx/.dotfiles/main/dotfiles

Reusing

In order to reuse these scripts, here a summary of files you can change/adapt to your needs:

scripts/packages.sh: all the homebrew taps and packages to install
scripts/fonts.sh: homebrew fonts to install
scripts/apps.sh: homebrew casks to install
scripts/osx.sh: macOS settings
scripts/config.sh: general settings and dotfiles

Testing Stow

To double checks that the dot files will be correctly linked, you can use stow to simulate the result. E.g.

stow -nSv vim

View on GitHub

Note: I have added a lot of read -p on my script to debug each section but they can all be removed so there is no need for interaction. Prompting for the sudo password could be than an enable it for the rest of command.

What I'm missing?

configuration for different apps like clipy. defaults show some configuration but not all of them.
some macOS config like disable autocapitalization don't seem to be available through defaults

I'd love to hear some ideas to improve this current setup!
Currently I'm checking ansible so stay tuned for a possible upgrade.

👽

Your new pretty and minimalist resume with LaTex

protium — Tue, 28 Dec 2021 02:01:06 +0000

Hi everyone, today we are going to take a look at how we can create a pretty resume using LaTex.

Minimalist and concise

When writing a résumé we always want to achieve a minimalist and concise document to easily catch the eyes of any recruiter/company. To do so, we want to organize its content into 2 columns, having a good spacing for all the content to fit without losing readability. And here is where LaTex comes to the rescue.

I split my resume into: Skills (33%) and Work Experience (66%) columns.

\begin{minipage}[t]{0.33\textwidth} 

\begin{minipage}[t]{0.66\textwidth}

Preview

Before describing each section, let's have a look at the final result

# Name Section

This is a command that takes 2 parameters: first name and last name. The last name has a thicker font and uses the accent color.

\namesection{Alan}{Turing}

# Skills

Here you want to list all your skills and additional information. As you can see in the preview, this column has several sections: Info, Education, Course Work, Skills, and Languages.

\ Info

\section{Info}

Here we place general information: location, date of birth, contact information, links. I think having plain urls looks pretty ugly so I decided to follow the following format: {domain}//{user_name}. This way it's easier to have an idea of what the link is about. E.g.

\textbf{Dev.to://}\href{https://dev.to/protium}{protium}\\

\ Education

\section{Education}

For this section I opted for the following format, where each entry is a subsection

{University Acronym}
{University Name}
{Degree}
{Year} | {State, Country}

E.g.

\subsection{FaMAF}
\location{National University Of Córdoba}
\descript{BSc. in Computer Science \\
BSc. in Physics}
\location{
2017-2019 | Córdoba, Argentina
}

\ Course work

\section{Course Work}

A list of all the courses you have finished or certifications you have obtained so far. Here you want to add a link to your certificate if possible (Coursera, Udemy, EdX, etc). If there is no certificate you could add a link to the course syllabus.

\ Skills

\section{Skills}

For this section, I wanted to list all the tools/languages I know and have experience with, for which I added different \subsections:

Programming: all the languages I have experience with
Infrastructure: all the infra tools I have experience with
Storage: all the storage solutions I have experience with
Familiar With: all the technologies I have used for personal projects or PoC at different companies.

\ Languages

\section{Languages}

The last section is dedicated to the languages you speak, with information about your level. In my case I have used the Common European Framework of Reference. E.g.

\subsection{Languages}
\textbf{Spanish:} Native\\
\textbf{English:} C1\\

# Work Experience

This is, of course, the main section of our resume. Over the years I learned that a good work experience description should provide the following information:

Company Name and Website
Your Position
Month and year
Location
Very small description of the company/product
What where your main tasks/accomplishments in the company. (Here is the real deal so let's get into more details in the next section).
Tech Stack: the company's tech stack. Be careful about not revealing sensitive information.

# Describing your position at company X

I have seen a lot of resumes during my career and I think most of the people fail at describing what they've done while working at company X. So to make it easier, let's list what a good description should have:

Achievements: what you have achieved while working in company X? Did you create something amazing? As example, you could write: "I refactored the search algorithm which led to an improvement of 70% in performance". It must be something you feel proud about.
What tasks did you enjoy at this position? "I created a PoC to add rust to our event processor"; "I created new tools for automatize deployments"
Some trivial tasks: "Prepared migration from Vue to React". Maybe you didn't enjoy these tasks but they were part of your tasks at company X.

E.g.

\runsubsection{\href{https://cool-00.example/}{Cool Company 00}}
\descript{| Senior Software Engineer}
\location{\ello{\textbf{Aug 2021 – Present} | Munich, Germany | Cool platform}}
\vspace{\topsep}
\begin{tightemize}
\item Developed a cool algorithm that increased the speed of transactions by 70\%
\item Migrated cloud to AWS
\item {\bf Tech Stack:} NodeJS, ReactJS, Typescript, JWT, Web Sockets, Jest, ExpressJS, Go, RabbitMQ, Temporal, Kubernetes, PostgreSQL, GCloud, Github Actions, Argo CD, Sentry\newline
\end{tightemize}

# Fonts

Ideally, you want to choose fonts that are easy to read and do not look too fancy. For this project I have used 2 google fonts in order to:

have a geek font for the Skills, locations, positions.
have a "normal" font for the jobs descriptions.

You can easily change these fonts in the resume.cls file.

\setmainfont[Color=primary, Path = fonts/maven-pro/, BoldFont=MavenPro-SemiBold]{MavenPro-Regular}

# Colors

Colors are defined as follow

\definecolor{primary}{HTML}{2b2b2b} 
\definecolor{headings}{HTML}{6A6A6A}
\definecolor{subheadings}{HTML}{333333}
\definecolor{links}{HTML}{33AFFF}

Note that links is our accent color.

That's it!
This post ended up being longer than I expected. I hope you found something useful here and I'm looking forward to see your customizations!

👽

protiumx / dev.resume

Your new pretty and minimalist resume

dev.resume

Your new pretty and minimalist resume. Based on: https://github.com/deedydas/Deedy-Resume

Preview

Fonts

Main font: MavenPro
Skills font: SourceCodePro

You can change the fonts as you want. I recommend taking a look at google fonts

Colors

You can change any color in the class definition.

primary: #2b2b2b
headings: #6A6A6A
subheadings: #333333
links: #33AFFF

Commands

\namesection: The resume header for your name. It expect to arguments: first name and last name
\runsubsection: For company name
\location: Location of the company

Compiling

You need a xelatex compiler. I used overleaf to edit and compile my resume.

Sponsorship

If you find this project useful you can support my work with:

View on GitHub

DEV Community: protium

An interpreted language you can try in my terminal website

The book: a great resource for learning about interpreters

Extending the Monkey language

Loading the interpreter in the browser: WebAssembly

Caveats

Extending my website terminal

Outro

Github template for Golang services

Task Runner

Import env variables

Showing help

CLI args

Verbose file targets definition

Folder structure

Database

Schema

Queries and migrations

Docker

CI

Creating a Text-based UI with rust

Intro

The ANSI Standard

Designing the text-based interface

User interaction

Sending HTTP requests

Bonus: Multi-threading

My profile website is now a terminal

A pragmatic approach

Terminal Commands

Terminal basic functionality

Basic commands

Going further

Update:

Update 2:

Update 3:

An HTTP request parser with rust and pest.rs

Introduction

Preparation

Request

Request-Line

Request-Header

Message-Body

Pest

HTTP Grammar

The .http file syntax

Parsing http files

reqwest

Lazyness kicks in

Unintroduction

Automate an articles section in your github.io page

First approach

blog-post-workflow

kitty + zsh + powerlevel10k = ✨ aesthetics ✨

What do I use?

Configuration

Terminal

powerlevel10k

neovim

blogpub@v0.4.1: host media on github for your blog posts

blogpub@v0.4.1: host media on github for your blog posts

Relative paths

Publish your blog articles everywhere with this github action

blogpub

Template Support

Article metadata

Adopting CI/CD practices for blogging

Usage

What is missing?

protiumx / blogpub

Github action to publish your blog articles from Markdown to Medium or Dev.to

blogpub

Updating articles

Pre-requisites

Usage

autoenv: relax, ansible will setup your dev environment for you

Bootstrapping

Installing ansible

autoenv playbook

[+] Installing all the apps, brew packages and casks

[+] Installing all the apps, `brew` packages and casks

[+] Setting `vs code` as default for all the source code extensions

[+] Installing `vim-plug`

`defaults`: macOS settings from the terminal