DEV Community: Erick 🙃 Navarro

Telegram bot to preview social media content

Erick 🙃 Navarro — Mon, 03 Feb 2025 23:27:15 +0000

I usually share some links of Twitter(it is known as X now but I still call it Twitter) in Telegram, but for some reason the previews doesn't work well most of the times.

One solution was to edit a Twitter link so I can use vxtwitter.com instead of twitter.com, this provides better previews but doing it that way is too manual so I made a bot to detect when I share a Twitter link and edit the message to use vxtwitter.com instead.

The bot uses the library telethon, it's a really easy to use python library. Also it has great documentation.

This is the complete code.

from telethon import TelegramClient, events

# get credentials from https://my.telegram.org, under API Development
# section, then fill the following section
api_id = 123
api_hash = "api hash"

# The first parameter is the .session file name (absolute paths allowed)
client = TelegramClient("my-bot", api_id, api_hash)

# outgoing=True is important so we only listen to our own messages,
# otherwise this will listen to all the messages received, direct
# messages and group messages
@client.on(events.NewMessage(outgoing=True))
async def rewrite_twitter_links(event):
    if "x.com" in event.raw_text:
        reply_text = event.raw_text.replace("x.com", "vxtwitter.com")

        # we must use the correct object to be able to edit the
        # message, otherwise we'll get an error
        if event.is_group:
            source = event.chat_id
        else:
            source = event.sender_id

        await client.edit_message(
            source, event.message.id, reply_text, link_preview=True
        )

client.start()
client.run_until_disconnected()

The code is simple, it connect to Telegram API using a "personal bot", this means it will act on your behalf. It will detect when the message has a x.com link and it will replace it(making an edit) with vxtwitter.com so it will be have a nice preview :)

This can be extended to do another nice things, for example: TikTok doesn't allow to see videos in mobile at least you have their app installed. In case someone send me a TikTok link I automatically respond with a preview message using vm.vxtiktok.com so I can see the video inside Telegram without having to install the app :).

The code to do that is:

@client.on(events.NewMessage(incoming=True))
async def rewrite_tiktok_links(event):
    # ignore group messages
    if event.is_group:
        return

    if "vm.tiktok.com" in event.raw_text:
        await event.reply(event.raw_text.replace("tiktok.com", "vxtiktok.com"))

In this case we respond only to a direct messages, that's why the early return when event.is_group is true.

Add D2 support to Hugo

Erick 🙃 Navarro — Thu, 13 Jun 2024 12:49:06 +0000

D2 is declarative language to generate diagrams, it's like mermaid on steroids, it has a cli so it's easy to use.

Hugo doesn't support it at the moment of writing this, there is an open issue where the conversation is being done.

So in the meantime official support is added we're going to make our own integration. It will have 2 parts:

A little HTTP server that receives the d2 code and return a resulting SVG
A custom hugo code block render hook

A simple HTTP server in `go`

This server only uses standard library and do the follow:

receive d2 code in a POST payload.
call d2 cli and get its std output.
return this output as response body.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os/exec"
)

func handleRenderRequest(w http.ResponseWriter, r *http.Request) {
    requestBody, err := io.ReadAll(r.Body)
    if err != nil {
        w.WriteHeader(http.StatusBadRequest)
        return
    }

    defer r.Body.Close()

    output, err := renderText(string(requestBody))

    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    fmt.Fprintf(w, output)
}

func renderText(content string) (string, error) {
    // d2 -, will render the text received in stdin
    command := exec.Command("d2", "-")
    command.Stdin = bytes.NewBuffer([]byte(content))
    output, err := command.Output()
    if err != nil {
        return "", err
    }
    return string(output), nil
}

func main() {
    http.HandleFunc("POST /render", handleRenderRequest)
    http.ListenAndServe(":8080", nil)
}

We can also use d2 as a package but using it as a CLI is less code.

Now if we send a request the following request:

POST http://localhost:8080/render
shape: sequence_diagram
alice -> bob: What does it mean\nto be well-adjusted?
bob -> alice: The ability to play bridge or\ngolf as if they were games.

Custom code block render hook

Hugo allow us to define custom code block render hooks, we're going to define a custom one using d2 hook so when we define the following code it will call our server and insert its resulting SVG.

D2 code here

Now we need to create a new file in layouts/_default/_markup/render-codeblock-d2.html and put the following code:

{{- $renderHookName := "d2" }}

{{- $inner := trim .Inner "\n\r" }}
{{- $position := .Position }}

{{- $apiEndpoint := "http://localhost:8080/render" }}

{{- $opts := dict "method" "post" "body" $inner }}
{{- with resources.GetRemote $apiEndpoint $opts }}
  {{- with .Err }}
    {{- errorf "The %q code block render hook was unable to get the remote diagram. See %s. %s" $renderHookName $position . }}
  {{- else }}
      <div style="width: 600px">
    {{ .Content | safeHTML }}
      </div>
  {{- end }}
{{- else }}
  {{- errorf "The %q code block render hook was unable to get the remote diagram. See %s" $renderHookName $position }}
{{- end }}

This code uses resources.GetRemote to make a POST request to our server and then insert the response content as part of the document.

The render will only be done on build time so we don't need to have the render server always on

Conclusion

Hugo allows us to add custom features to our site and go allows us to accomplish task using on only standard library.

Enjoy! 🎉

Mermaid preview using xwidget browser

Erick 🙃 Navarro — Wed, 05 Jun 2024 14:19:05 +0000

Mermaid.js is a great tool to make diagrams in plain text, I use it a lot and I wanted to have a way to see previews of the code I was writing.
There are some options to do that but they require to have a mermaid-cli installed, which requires nodejs as well.

Emacs has a built-in webkit browser, in case it was compiled with --with-xwidgets flag, and mermaid run on js so it should be possible to just run the code I want in the browser and see it there.

This function makes the magic, it just take a region previously selected, which have the mermaid code, create a temp file and write some HTML there(including our mermaid code).

(defun my/preview-mermaid ()
  "Render region inside a webit embebed browser."
  (interactive)
  (unless (region-active-p)
    (user-error "Select a region first"))
  (let* ((path (concat (make-temp-file (temporary-file-directory)) ".html"))
         (mermaid-code (buffer-substring-no-properties (region-beginning) (region-end))))
    (save-excursion
      (with-temp-buffer
        (insert "<body>
  <pre class=\"mermaid\">")
        (insert mermaid-code)
        ;; js script copied from mermaid documentation
        (insert "</pre>
  <script type=\"module\">
    import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.esm.min.mjs';
    mermaid.initialize({ startOnLoad: true });
  </script>
</body>")
        (write-file path)))
    (xwidget-webkit-browse-url (format "file://%s" path))))

Demo

Moving to Emacs Tree Sitter Modes

Erick 🙃 Navarro — Wed, 22 May 2024 04:30:15 +0000

I'm currently using emacs 30 and this version has support for more languages using the new ts-modes, I've been using emacs-tree-sitter package since Emacs added dynamic modules feature.

Now using emacs 30 I give a try to use only ts-modes and maybe delete some external packages I was using.

How to install grammars

Emacs doesn't have a built-in way to install automatically most common grammar so we're going to use a package that can handle that, we use tree-sitter-langs package, which is a repository for grammars.

When we install it it download and install all the available grammars in its own directory but to be used by emacs ts-modes we need to put them inside /.emacs.d/tree-sitter directory. I write a helper function to accomplish this:

(defun my/copy-grammars-to-emacs-tree-sitter-dir ()
  "Copy tree-sitter grammar files to native Emacs dir."
  (interactive)
  (let* ((files (directory-files (tree-sitter-langs--bin-dir) nil "\\.dylib$")))
    (dolist (grammar-file files)
      (copy-file (concat (tree-sitter-langs--bin-dir) grammar-file) (concat (expand-file-name user-emacs-directory) "tree-sitter/" "libtree-sitter-" grammar-file) t)
      (message "%s grammar files copied" (length files)))))

The migration

I use straight.el to manage my packages so I'm going to use it to configure built-in ts-modes, by now I only migrated a few modes.

For elixir:

(use-package elixir-ts-mode
  :straight (:type built-in)
  :mode (("\\.ex\\'" . elixir-ts-mode)
         ("\\.exs\\'" . elixir-ts-mode)
         ("\\mix.lock\\'" . elixir-ts-mode)))

For docker:

(use-package dockerfile-ts-mode
  :straight (:type built-in)
  :defer t
  :mode (("\\Dockerfile\\'" . dockerfile-ts-mode)
         ("\\.dockerignore\\'" . dockerfile-ts-mode)))

For typescript

(use-package typescript-ts-mode
  :straight (:type built-in)
  :defer t
  :mode "\\.tsx?\\'")

For toml

(use-package toml-ts-mode
  :straight (:type built-in)
  :mode "\\.toml\\'"
  :defer t)

For yaml

(use-package yaml-ts-mode
  :straight (:type built-in)
  :mode "\\.ya?ml\\'")

With this now I have 5 dependencies less in my configuration :).

Conclusion

I'm in the process to migrate others modes like python-mode, go-mode and so on but in some cases syntax highlighting is not the best. For example shell-script-mode has better highlight than bash-ts-mode. I'm been using these "new" ts-modes for a few weeks and everything is working well.

Bonus

I also updated a package I use to run ispell on text nodes to support built-in tree-sitter support. Now it supports tree-sitter.el and treesit packages.

Simple OBS Client in Elixir

Erick 🙃 Navarro — Sun, 12 May 2024 19:49:50 +0000

Let's write a simple module to control OBS using websockets.

OBS already have a websocket server which can accept many command to control its features. We're going to use that websocket server to make some actions like:

Change scene
Apply source filters
Many other things

Basic configuration

First we need to install a websocket client, fresh is a easy to use client that works on top on mint.

Let's add the following code to our mix.exs file.

     {:earmark, "~> 1.4"},
+    {:fresh, "~> 0.4.4"}
   ]
 end

Now we need to define a module that use all the functionality of fresh, like the following code:

defmodule MyApp.OBS do
  use Fresh

  def handle_connect(_status, _headers, state) do
    payload = %{"op" => 1, "d" => %{rpcVersion: 1}}
    {:reply, {:text, Jason.encode!(payload)}, state}
  end
end

At this point this just connect to the websocket and responds with a specific message {"op": 1, "d": {"rpcVersion": 1}}, this is required to "authenticate" against OBS websocket server. This according to OBS websocket protocol. In case we set up a password in our OBS settings we need to modify this response.

Now we need to start our client, we can do it manually or using application.ex like the following code:

 @impl true
 def start(_type, _args) do
   children = [
+    # OBS websocket client
+    {MyApp.OBS,
+     uri: "ws://localhost:4455", state: nil, opts: [name: {:local, :obs_websocket_client}]}
   ]
 end

Here is where we define 2 things:

the location of the websocket server, if we are running OBS in the same machine we can use ws://localhost:4455, this is the default address that OBS uses.
the "name" of the process that will be used later to send messages to. In this case is :obs_websocket_client. The client is just an erlang process so we need some way to send messages to it.

Controlling OBS

Now that we have a working websocket client we can send messages to OBS.

We are going to implement a function to change the scene in OBS, to do that we need to use the SetCurrentProgramScene action.

Let's add 3 functions to our OBS module.

+def change_scene(scene_name) do
+  send_message("SetCurrentProgramScene", %{"sceneName" => scene_name})
+end

+defp send_message(type, data) do
+  payload = %{
+    "op" => 6,
+    "d" => %{
+      # generate a random identifier, we can use any other module
+      "requestId" => Ecto.UUID.generate(),
+      "requestType" => type,
+      "requestData" => data
+    }
+  }

+  send(:obs_websocket_client, {:send, Jason.encode!(payload)})
+end

+def handle_info({:send, message}, state) do
+  {:reply, [{:text, message}], state}
+end

send_message/2, receive a type and a map, it will use these to prepare a payload which will be send to our websocket client. We're using send/2 because our client is just another process, which have the name defined previously in application.ex: :obs_websocket_client.
handle_info/2, this will receive the message and send it to the websocket server, because OBS protocol just use plain text we use a {:text, message} response.
change_scene/1 this just reuse the previously defined functions. We send the SetCurrentProgramScence message with a specific payload. Now we can just call MyApp.OBS.change_scene("gaming") and it will be changed.

Enjoy 🎉.

Switching from helm to vertico and friends

Erick 🙃 Navarro — Fri, 01 Sep 2023 05:23:06 +0000

I've been using helm for many years, since I started to use emacs itself. It was great because it is a "all in one" tool, very convenient when starting in emacs world.

The problem(?)

Actually there is a not problem perse, just maybe the "startup time", I had the load deferred so the first time I call some function from helm there was a little "wait", nothing really bad but a little annoying

Motivation

Just wanted to trying new things 😅

The new stuff

There are new tools that are gaining popularity because of their approach: "just do one thing", these tools are:

vertico: UI for completion, based on minibuffer, it also has a option to use a separate buffer, just like helm
marginalia: Enhance the minibuffer with more context about the information shown
orderless: completion style, add fuzzy search to filter between the data shown by vertico
consult: commands for search and navigation
embark: runs commands given a context at point

All of these tools rely on emacs builtin functionality as much as possible and try to adhere to the APIs of different builtin components, that's a nice approach :)

The migration

Before migrating I needed to confirm that there were a replacement for every(or at least more of them) feature I use in my workflow, these features are:

Fuzzy search in a project

I was using helm-ls-git, which works on top oh helm and also put at the top all the files that have been modified, it uses git to get that information, it very useful because it give more context when searching for a file.

I haven't found(yet) something that replace helm-ls-git but for fuzzy search I just can call project-find-file, which is builtin in emacs.

Search and replace in a whole project

For this I was using helm-ag, it allow to search into project files content, make another filter using helm and also create an editable buffer with the results, once all the edits are done we can modify all the matched files with a single command.

This can be done using embark-export function, which export all the content in the minibuffer and create a new buffer and use wgrep to have a editable buffer, that's a lot of manual steps, in helm-ag it was more easier so I wrote some code to have the same experience

(defun my/grep-edit-results ()
  "Export results using `embark-export' and activate `wgrep'.
This only runs for ripgrep results"
  (interactive)
  (when (cl-search "Ripgrep" (buffer-string))
    ;; we use `run-at-time' to ensure all of these steps
    ;; will be executed in order
    (run-at-time 0 nil #'embark-export)
    (run-at-time 0 nil #'wgrep-change-to-wgrep-mode)
    (run-at-time 0 nil #'evil-normal-state)))

(define-key minibuffer-mode-map (kbd "C-c C-e") #'my/grep-edit-results)

This can be executed from the minibuffer and only when it is a Ripgrep execution, this way I have the same as I used to with helm-ag.

Navigate the kill ring

Many times I delete stuff from many buffers and put them together in a new buffer, to do that I rely on the kill-ring history, helm has a function helm-show-kill-ring with allow to fuzzy search in the history and then paste the selected item into the current buffer. consult has an option to that too consult-yank-from-kill-ring. It is not show the options as helm but it does the work.

Fuzzy search inside a buffer

For this I used helm-swoop, consult already has a function that do the same, it is consult-line

Backup files

helm-backup is great, it create a backup of your current file on every save and store it in a git repository, it allow to look for all the versions using helm, I wanted to still use it but I don't want to have helm installed just for one package, fortunately the author has another package git-backup which has all the functionality for backups has no dependency on helm, with this I was able to reproduce what I have and with some elisp code.

(defvar my/backup-dir (expand-file-name "~/.git-backup"))

(defun my/git-backup-versioning ()
  "Save a version of the current file."
  (unless (featurep 'git-backup)
    (require 'git-backup))
  (git-backup-version-file (executable-find "git") my/backup-dir '() (buffer-file-name)))

(defun my/git-backup-run-action (command commit-hash)
  "Execute COMMAND with COMMIT-HASH using another defaults arguments."
  (apply command `(,(executable-find "git") ,my/backup-dir ,commit-hash ,(buffer-file-name))))

(defun my/git-backup ()
  "Navigate in versions of the current file."
  (interactive)
  (unless (featurep 'git-backup)
    (require 'git-backup))
  ;; for some reason an extra space after `%h|' is required to avoid an error when
  ;; the shell command is executed
  (let* ((candidates (git-backup-list-file-change-time (executable-find "git") my/backup-dir "%cI|%h| %ar" (buffer-file-name)))
         (selection (completing-read "Pick revision: " candidates))
         (commit-hash (nth 1 (string-split selection "|")))
         (action (completing-read "Choose action: " '("diff" "new buffer" "replace current buffer"))))
    (cond ((string-equal action "diff") (my/git-backup-run-action 'git-backup-create-ediff commit-hash))
          ((string-equal action "new buffer") (my/git-backup-run-action 'git-backup-open-in-new-buffer commit-hash))
          ((string-equal action "replace current buffer") (my/git-backup-run-action 'git-backup-replace-current-buffer commit-hash))
          (t (message "Not valid option")))))

(use-package git-backup
  :ensure t
  :hook (after-save . my/git-backup-versioning))

Final thoughts

For the time this post was written I've been using the new setup for just a few days, so far the experience was good, it feels a simple setup now and gave me some ideas for another features that I'd like to develop.

Also you can see the diff after the migration in my dotfiles

Special thanks to @oantolin, creator or embark, for helping me with some doubts in the telegram channel of emacs in Spanish emacs-es

Run ispell on text nodes using tree sitter

Erick 🙃 Navarro — Sun, 26 Jun 2022 19:28:34 +0000

tree-sitter is a great tool to have an incremental syntax tree of our code, in emacs it can be used to add syntax highlighting instead of using the regular regex based highlighting system. There are other use cases for this and now we're going to use it to build a simple tool to run ispell using the content of a text node, of course we can select the text manually but it will be easier and fancy to do it in a programmatic way using the syntax tree generated by tree-sitter.

Emacs doesn't have support by default for tree-sitter so we need to install it, the following code will do it using use-package:

(use-package tree-sitter
  :ensure t
  :hook
  ;; enable highlight using tree-sitter instead of regex based system
  (tree-sitter-after-on . tree-sitter-hl-mode)
  :config
  (global-tree-sitter-mode))

(use-package tree-sitter-langs
  :ensure t)

To build this tool we need two things:

Find a way to check if given our current position we are in a string node, it can be a literal string, a multiple line string, a comment, etc.
Call ispell pragmatically using the position of a tree-sitter node.

Using tree-sitter to get node at the current position

tree-sitter package has some functions we can use, tree-sitter-node-at-pos will give us the nearest node of a given type for our current position, so if we run (tree-sitter-node-at-post 'string (point)) if will return, in case it exists, a string node otherwise it will return nil, using this we can check for any possible "string" values, in a programming language we can have string, comment and other elements which have text that need a spell check.

Because tree-sitter use a specific grammar for every programming language, a "string" element can have different names, for example in python we have string but in go we have interpreted_string_literal, we can check this values by running M-x tree-sitter-debug-mode from a buffer using the language we want to know their "text" element names.

Now we have to define a list with all the supported languages we want to use:

(setq tree-sitter-text-grammar-mapping '((python-mode . (string comment))
                                         (go-mode . (interpreted_string_literal comment))
                                         (js-mode . (string template_string comment))
                                         (elixir-mode . (string comment))))

Here we only support 4 languages, but it easy to add more, these are the ones I use more often.

Now with the following function we can use the previous defined list of languages and extract a valid text node at current position:

(defun get-text-node-at-point ()
  "Get valid node for the current major mode using `tree-sitter-text-grammar-mapping'"
  (let* ((types (alist-get major-mode tree-sitter-text-grammar-mapping))
         ;; get string nodes from all the available nodes at the current point
         (matches (seq-map (lambda (x) (tree-sitter-node-at-pos x (point) t)) types))
         (filtered-matches (remove-if (lambda (x) (eq nil x)) matches)))
    ;; get first valid match
    (if filtered-matches
        (car filtered-matches))))

Call ispell using a tree-sitter text node

We can use ispell-region to run ispell over a specific region, this function receives the start and end positions of a region so we need to extract those values from our tree-sitter text node.

tsc-node-start-position and tsc-node-end-position can be used for this:

(defun run-ispell-on-node (node)
  "Run ispell over the text of the received `node'"
  (ispell-region (tsc-node-start-position node) (tsc-node-end-position node)))

Putting all together

Now we can combine these two functions and assign it to a keybinding:

(defun run-ispell-at-point ()
  "Run ispell at current point if there is a text node."
  (interactive)
  (let ((node (get-text-node-at-point)))
    (if node
        (run-ispell-on-node node))))

(global-set-key (kbd "C-x C-s") 'run-ispell-at-point)

Now when we run C-x C-s, if we are at a text node, ispell will run and check the spelling of that node.

Enjoy.

Auto build and publish emacs org configuration as a website

Erick 🙃 Navarro — Sun, 04 Apr 2021 16:32:08 +0000

Having our emacs configuration in an org file is great, it allow us to have it more organized and easy to read, but org files have more features and one of them is the ability to be exported to different formats like HTML, PDF, markdown and so on. So what if we export our emacs configuration to HTML and then publish it in a website? 🤯

It probably doesn't have any real utility but it would be nice to have a exclusive web page to show our emacs config to our friends :)

We can do this in two ways:

Manually, we can export the org file using the regular exportation feature of org-mode and then upload the resulting HTML somewhere
Automatically, our configuration will be rendered and published into a website every time we push some changes to our dotfiles repository

Let's define what we need to do to have the automatic way:

Have a script that render our org config file
Run this script in a CI so it can be run every time we push some changes
Push the rendered HTML to an extra repository
Activate Github Pages in the extra repository, so we can have an url where we can see the resulting website

Let's assume we have this structure in our dotfiles repository:

├── emacs
│   ├── config.org
│   └── init.el
└── scripts

This is a "regular" structure for a dotfiles repository, the extra scripts folder will be used later. Now let's deep into how it will work.

Automate the org file rendering

We need to create to files and put them inside scripts folder:

render-emacs-config-to-html.sh, this will render our config file and place the resulting HTML file inside scripts/output/index.html
org-render-html-minimal.el, this is a minimal config file to be able to render org into html, it load the required packages and make some basic configuration

Let's explore first scripts/org-render-html-minimal.el

(require 'package)

(setq package-archives '(("gnu" . "https://elpa.gnu.org/packages/")
                         ("melpa" . "https://melpa.org/packages/")))

(package-initialize)

(package-refresh-contents)

;; this is required to highlight code blocks properly
(package-install 'htmlize)

(require 'org)

(require 'htmlize)

;; For some reason the default value `inline-css' doesn't apply syntax highlighting correctly
;; in the resulting html file so we need to change the value to `css'
(setq org-html-htmlize-output-type 'css)

To export code blocks correctly we need the package htmlize, this package is available in MELPA so we need to configure MELPA and then install it from there.

Now let's check scripts/render-emacs-config-to-html.sh

#!/bin/sh

# read the docs based theme
echo "#+SETUPFILE: https://raw.githubusercontent.com/fniessen/org-html-themes/master/org/theme-readtheorg.setup" > index.org
cat ../emacs/config.org >> index.org

emacs index.org --batch -Q --load org-render-html-minimal.el -f org-html-export-to-html --kill

# output will be the directory uploaded to the render repository so we have to put all the resulting files inside that folder
mkdir output
mv index.html output/

What are we doing here?

We basically create a new org file called index.org and put a setup configuration file in it. You can avoid this step if you put this line directly in your config file, in this case we're using one of the themes available in this repository, there is more themes available in this other repository so you can choose the one you like the most.

Now we need to run emacs with our previously defined configuration org-render-html-minimal.el and tell it to render our index.org.

And finally we put the resulting index.html inside output folder. This folder will be used later.

Using Github actions to build and publish the rendered config

We're going to use a Github action called push-directory-to-another-repository, this action allow us to commit and push changes in another repository. Some configuration is required to use this action:

Create a extra repository

This extra repository will be used to host our rendered config file, in my case this repository is erickgnavar/emacs-config, we also need to activate Github Pages in this repository and set it up to use master branch

The url generated, erickgnavar.github.io/emacs-config in my case, is where our rendered config file will be published.

Create a personal token

To be able to push changes into the new repository we have to create a personal access token, this can be made in account settings, this token should have the repo scoped activated.

Configure a secret variable in our dotfiles repository

The Github action needs a secret variable called API_TOKEN_GITHUB, this variable allow the action to push changes into the new repository, we can create it by going to repository/settings/secrets/New repository secret

Configure Github action config file

Finally we have to create a file .github/workflows/ci.yml with the following content:

name: CI

on:
  push:
    branches: [ master ]

jobs:
  build-emacs-config-page:
    runs-on: ubuntu-latest
    container: alpine:3.13.4
    steps:
      - uses: actions/checkout@v2
      - name: Install emacs
        run: apk --update add emacs
      - name: Render config into html
        run: cd scripts && sh render-emacs-config-to-html.sh
      - name: Pushes to destination repository
        uses: cpina/github-action-push-to-another-repository@cp_instead_of_deleting
        env:
          API_TOKEN_GITHUB: ${{ secrets.API_TOKEN_GITHUB }}
        with:
          source-directory: 'scripts/output'
          destination-github-username: 'YOUR_GITHUB_USERNAME'
          destination-repository-name: 'YOUR_NEW_REPOSITORY_NAME'
          user-email: bot@emacs.bot

This action config file make some things:

Install emacs so we can run it to render our config file
Render our config file using the script render-emacs-config-to-html.sh we previously defined
Take the content of scripts/output, commit and push it into our destination repository, this is why we need to move the resulting HTML file into output folder
And finally it calls github-action-push-to-another-repository action which will do all the git stuff required to push the changes

Now every time we push changes to our dotfiles repository this action will push the rendered config file to our destination repository, the commits will look like this:

And when we enter to the url generated from Github pages, erickgnavar.github.io/emacs-config in my case, we can see our configuration rendered:

Enjoy 🎉

Easy deploy of docker based projects

Erick 🙃 Navarro — Sat, 03 Apr 2021 03:57:24 +0000

I have a personal server where I run some projects, some of them written in python, elixir and other technologies so having to deal with specific installation of any of these technologies is not an ideal workflow, to fix this I use docker and all of them are deployed using docker-compose, they're connected to a single PostgreSQL server and they're behind the same web server.

Running all of these projects in this way it's easier to maintain and in case something happens with the server I can re deploy everything in a easy way. Let's take a look to these tools and how they work together.

Let's assume we have the following requirements:

Deploy a django application
Deploy a phoenix application
Each application needs a PostgreSQL database
Both applications should be behind a web server and being accessed over HTTPS
All of these should run in the same server

To solve this we're going to:

Set up a linux server
Install PostgreSQL
Configure a web server which will handle incoming traffic and SSL termination
Run our applications inside Docker containers

Setting up a server

If you already have a server you can skip this section.

We first need a server which can run docker, most linux distros can be used for this but in this case we'll be using Ubuntu Server, if you don't have a server yet you can use any of these referral links to get some credit when you create your account:

Digital Ocean this will get you $100 on credits to be used in 2 months
Hetzner this will get you 20€ on credits, this provider has cheaper prices than Digital Ocean but it only have data centers in Europe
Linode This is not a referral link but you can get $100 on credits, AFAIK it doesn't expire

Once you get a server it's recommended to make some basic configurations like updating packages, setup a firewall, etc. You can follow this Linode guide to secure your server.

After that you need to install docker, to do that you can follow the official documentation, this have specific instructions for you linux distribution.

Once we have docker and docker-compose installed we can follow this guide.

Installing PostgreSQL in our host machine

We're going to use a unique instance of PostgreSQL installed in the host machine, this way we can share the resources used by PostgreSQL with all the applications that we're going to deploy, we just need to create new users and databases for each one of the applications.

First let's install PostgreSQL with:

sudo apt install postgresql-server

We need to login with postgres user so we can be able to enter to a psql session. We can do it with:

sudo su - postgres

Now we can open a psql session and create the databases and users for our applications:

postgres=# CREATE USER django WITH ENCRYPTED PASSWORD 'secret';

postgres=# CREATE DATABASE django WITH OWNER django;

And we do the same for our phoenix application:

postgres=# CREATE USER phoenix WITH ENCRYPTED PASSWORD 'secret';

postgres=# CREATE DATABASE phoenix WITH OWNER phoenix;

Configuring Caddy as a reverse proxy

Caddy is a "new" web server written in Go that have 2 main features that make it a good option for simpler deployments:

Simpler configuration file
Free auto configured SSL certificates, using Let's Encrypt service, and automatic renewals

If we were using for example Nginx we have to deal with HTTPS certificates by ourselves, installing certbot, and also have to configure some way to renew the certificates, Let's Encrypt issues certificates that expire after 3 months.

Let's define our domains django.domain.com and phoenix.domain.com which will send traffic to their specific applications.

Our django application needs that Caddy serves the static files so we define file_server option and tell caddy where are our static files, we also tell Caddy to send the traffic to port 8000 where our application is listening.

django.domain.com {
    root * /opt/django

    @notStatic {
        not path /static/*
    }

    reverse_proxy @notStatic localhost:8000
    file_server
}

Our phoenix application will serve static files by itself so we just need to define the reverse_proxy directive to be able to send the traffic to port 4000

phoenix.domain.com {
    reverse_proxy localhost:4000
}

Now when we reload our caddy server with sudo systemctl reload caddy it will get the SSL certificates and internally will check if they still valid, otherwise it will renew them.

Running our projects with docker-compose

Docker compose is a tool that allow us to define different docker services in a easier way using a yaml file.

We're going to configure our two projects using docker-compose but we first need their docker images so let's build them.

Let's clone our projects(both are in the same repository, just in different folders), build the images and then publish them on a registry.

This can be made in a separate machine because once the images are pushed to a remote registry they can be downloaded in our server.

cd simple-django-project-with-docker
docker build -t registry.mycompany.com/django:v1 .
docker push registry.mycompany.com/django:v1

cd simple-phoenix-project-with-docker
docker build -t registry.mycompany.com/phoenix:v1 .
docker push registry.mycompany.com/phoenix:v1

You can use docker hub to push your images or use Gitlab registry in case you want free private images.

Django application

Let's create a folder in /opt/django and put the following code into a docker-compose.yml file.

version: "2"
services:
  web:
    image: registry.mycompany.com/django:v1
    restart: always
    network_mode: host
    environment:
      ALLOWED_HOSTS: "django.domain.com"
      DEBUG: "0"
      DATABASE_URL: "postgres://django:secret@localhost:5432/django"
      DJANGO_SETTINGS_MODULE: "config.settings"
      SECRET_KEY: "a 32 long secret key"
    volumes:
      - ./static:/app/static
    ports:
      - "127.0.0.1:8000:8000"

The static folder will be used by Caddy to server static files.

Phoenix application

Now for our phoenix application let's create a folder /opt/phoenix and put the following code into a docker-compose.yml file.

version: "2"
services:
  web:
    image: registry.mycompany.com/phoenix:v1
    restart: always
    network_mode: host
    environment:
      DATABASE_URL: "postgres://phoenix:secret@localhost:5432/phoenix"
      MIX_ENV: prod
      HOST: "phoenix.domain.com"
      SECRET_KEY_BASE: "a 32 long secret key"
    ports:
      - "127.0.0.1:4000:4000"

Because we're running PostgreSQL in our host machine instead of a docker container we have to use network_mode: host, this allow us to access postgres just pointing to localhost.

Deploying our projects

Once we have the docker-compose.yml files configured we can go inside each project folder and run:

docker-compose up -d

For the django application we also have to run these commands, these are specific of django deployment process.

# Run database migrations
docker-compose exec -T web python manage.py migrate

# Collect all static files and place them in our STATIC_ROOT folder which will be served by Caddy
docker-compose exec -T web python manage.py collectstatic --no-input

Deploying new changes

Because we're using docker, when we need to update changes we just need to update their Docker images and restart their services. Some technologies can have differences in their deployment process but the basic idea is the same.

Let's see how it could be for our two example applications.

Django application

When we update a django application we need to run some extra commands like migrate, collectstatic, etc. We can follow these steps to run them inside the docker container:

docker pull NEW_DJANGO_IMAGE

sed -i "s/image.*/image:\ NEW_DJANGO_IMAGE/" docker-compose.yml

docker-compose up -d --force-recreate

docker-compose exec -T web python manage.py migrate

docker-compose exec -T web python manage.py collectstatic --no-input

We're pulling the new image from our registry, updating the image value in our docker-compose.yml file, restart the service (it will use the new image now) and then we can execute migrate and collectstatic commands

Phoenix application

For the phoenix application we're going to follow almost the same process with just one difference, we don't need to run migrations in a separate step because they will run when the application starts, this is defined in the phoenix docker image itself.

So we just need to pull the new image, update it in docker-compose.yml file and then restart the service, the final script will be:

docker pull NEW_PHOENIX_IMAGE

sed -i "s/image.*/image:\ NEW_PHOENIX_IMAGE/" docker-compose.yml

docker-compose up -d --force-recreate

Conclusion

Having a central PostgreSQL instance and a central web server(Caddy), both in the host machine instead of inside a container allow us to manage them easily and also allow us to share these common services alongside the many applications that we are running in our server.

Using compilation mode to run all the things

Erick 🙃 Navarro — Thu, 01 Apr 2021 16:44:22 +0000

Compilation mode is a major mode that allow us to run a command and see its output result in a special buffer, this resulting buffer show the errors and allow us to navigate through them, you can check the documentation for more details.

It's a "simple mode" but it can be used for many things like compile and run a program, run tests, and so on.

Usage

Interactive

compile is an interactive function so we can call it with M-x compile and enter the command we want to execute.

In the following example we're compiling this blog using hugo binary:

From code

compile is a emacs-lisp function so we call it from our code, we just need to take care about the default-directory when we call it, for example if we call it from lib/hello.ex buffer, default-directory will be lib and in some cases we want to use our project root, or a different directory, to run our command.

To fix this we need to setup default-directory before we call compile, for example let's build a custom function to run hlint in our entire project and then show its results in a compilation buffer:

(defun my/run-hlint ()
  "Run  hlint over the current project."
  (interactive)
  (let ((default-directory (projectile-project-root)))
    (compile "hlint .")))

In this case we're setting up default-directory with our project root (using projectile to get the root) and then when we call compile it will take default-directory correctly.

Some tweaks

These modifications to default behaviour of compilation-mode should be made after the mode was loaded so we need to use with-eval-after-load otherwise these changes won't be applied correctly.

Evil-mode

The compilation buffer has some preset key bindings that conflict with evil-mode, for example when we press g in a compilation buffer this will re-run the command, but this key binding is also used by evil-mode, to fix this we can disable the default key binding with:

(with-eval-after-load 'compile
  (define-key compilation-mode-map (kbd "g") nil)
  (define-key compilation-mode-map (kbd "r") 'recompile)
  (define-key compilation-mode-map (kbd "h") nil))

In this case h key binding is also disabled (also used by evil) and r is remapped to recompile for easy access now that we disabled g default key binding.

Follow compilation output

By default compilation-mode doesn't follow the output of the command so if our command result has a large output we'll need to scroll manually, to fix this we can change compilation-scroll-output to t

(with-eval-after-load 'compile
  ;; set cursor to follow compilation output
  (setq compilation-scroll-output t))

Enable ANSI colors

Some tools show results with colors for easy reading but compilation-mode won't show them by default, you can make them look better with:

(require 'ansi-color)

(defun colorize-compilation-buffer ()
  (let ((inhibit-read-only t))
    (ansi-color-apply-on-region (point-min) (point-max))))

(add-hook 'compilation-filter-hook 'colorize-compilation-buffer)

This was taken from this Stack Overflow answer

Re run compilation from another buffer

When we're making changes in our code we want to re-run our compilation process right after we save the changes but to do this we have to move to the compilation buffer to be able to re-run the compilation, a better approach to do this could be just call recompile by using a key binding, I use evil-leader to make this:

(evil-leader/set-key "R" 'recompile)

But it can be attached to any key binding, for example:

(global-set-key (kbd "C-c C-r") 'recompile)

Bonus: run parrot mode animation when a compilation is successful

I configured parrot-mode to animate the little parrot every time the compilation process is a success, to make this we need a small function that check if it was a success and then we need to attach it to 'compilation-finish-functions, this is a variable defined in compilation-mode.

(defun my/parrot-animate-when-compile-success (buffer result)
  (if (string-match "^finished" result)
      (parrot-start-animation)))

(use-package parrot
  :ensure t
  :config
  (parrot-mode)
  (add-to-list 'compilation-finish-functions 'my/parrot-animate-when-compile-success))

Conclusion

As we can see compilation-mode is a simple but powerful mode that allow us to build our own tools, we can create an automatic build system in case there is not something already existing for the technology we're using or if we just want to run some tasks in an easier way.

Things I wish I knew when I started in elixir development

Erick 🙃 Navarro — Mon, 01 Mar 2021 14:54:15 +0000

These are some things I commonly use in my elixir development workflow that might be interesting for someone.

Managing multiple versions of elixir and erlang

When you have to work in more than one project at time that could probably means you have to handle different elixir and erlang versions so installing the default version that your OS provides won't be helpful. Here is where asdf shines to solves this problem, asdf allow us to have different versions of elixir, erlang and other languages in the same machine so we can easily switch between them.

In macOS you can install it with brew install asdf and then follow the instructions that the installer prints out to set up the PATH, more info in its Github page https://github.com/asdf-vm/asdf

Now we have asdf installed we need to install the plugins to handle erlang and elixir, we can install them with:

asdf plugin add erlang
asdf plugin add elixir

Once we have those installed we need to define which versions we're going to use in our project, there is more than one way to do that:

Using environment variables

We can set up the required versions by defining environment variables with the ASDF_ prefix so if we need version elixir 1.10 we need to define the variable ASDF_ELIXIR_VERSION with the value 1.10 the same applies for erlang or other programming languages as well.

For example we can define variables for elixir and erlang as the example below:

export ASDF_ELIXIR_VERSION=1.10.3-otp-22
export ASDF_ERLANG_VERSION=22.3

To handle environment variables in a easy way we can use direnv, it allows to define environment variables in a file .envrc and it will loaded automatically as soon as we enter to out project folder.

File based config file

asdf allow us to define a .tool-versions file where we can put all the versions needed for our project, we can define one as the example below:

erlang 23.0.2
elixir 1.10.4-otp-23

Creating a new project

Because we have first to define the versions used in a project we can't just run mix new my_app because asdf doesn't know yet which versions we want. To do this we have 2 options:

Define global versions of elixir and erlang using for example asdf global elixir 1.9.0 and the same for erlang asdf global erlang 22.3 and then we can execute mix new my_app normally
Define the versions just for the mix new command execution, for example ASDF_ELIXIR_VERSION=1.9.0 ASDF_ERLANG_VERSION=22.3 mix new my_app, this way we don't affect the global scope and then we can define these same versions inside the created project.

I like the second one because I don't have to change the global version each time I want to create a new project and I can easily access to that command from bash history.

Notes about erlang compilation

asdf uses kerl under the hood to handle erlang compilation and when we are installing a new version it will ask for a java installation 😕, to avoid this behaviour we can define the following environment variable:

export KERL_CONFIGURE_OPTIONS="--disable-debug --without-javac"

Ecto database url

If we are dealing with databases in our project we will probably be using Ecto. And Ecto allow us to define database credentials in two ways, the first one is define them separately as the example below:

config :my_app, Repo,
  database: "ecto_simple",
  username: "postgres",
  password: "postgres",
  hostname: "localhost"

And the second one is using a unique parameter:

config :my_app, Repo,
  url: "postgres://postgres:postgres@localhost/ecto_simple"

This is my favorite option for these reasons:

Just one value to maintain
This format is also accepted in psql, for example we can execute psql postgres://postgres:postgres@localhost/ecto_simple and we're connected to the database. I just discovered this a few weeks ago 😅
We can change credentials for example when we're running a mix command just prepending the value DATABASE_URL=postgres://postgres:postgres@localhost/test_db mix something in the case we're loading it from an environment variable

Then if you have the connection url in a variable called DATABASE_URL, using direnv of course 😉, you can just execute psql $DATABASE_URL to database session.

Using iex

Enable shell history

A cool feature of elixir is iex, you can load modules, recompile them and so on, but when sometimes we execute "large" pieces of code or some cases that we're trying out to understand the code or something else and when we have to restart the session we lost all the history 😢, we can avoid this by adding a flag -kernel shell_history enabled in the environment variable ERL_AFLAGS before we start our iex session. I just put following code in my .zshrc to have it enabled for all my projects:

export ERL_AFLAGS="-kernel shell_history enabled"

Preload aliases

Another thing than could be annoying to deal with is aliasing a large module name, for example if we have MyApp.Contexts.Authentication.User and we are using this module pretty often it could be easier to have it already loaded when we start a iex session, we can make this by defining a .iex.exs file in the project root with the desired aliases, for example:

alias MyApp.Contexts.Authentication.User

And now when we start a new iex session we will have that module aliased from the beginning so we can use User.whatever without a problem.

Keep in mind that even if we can make an alias(a module name is just an atom) when we starting a session using just iex we cannot access to its functions. We need to start our iex session using iex -S mix

Recompiling modules

Within a iex sessions we can recompile a module just writing r module_name and if the want to recompile the whole project we can execute recompile, this is useful when we are making some changes in the code and we need to test it right away with all the values that we already had defined. It's also called "REPL based development" and it's most used with lisp based programming languages but having iex in elixir we can use those nice features as well.

Mix tasks

These are tasks that mix can run, duhh.. But we can create them and use them in our projects. For example maybe we are debugging some code and we don't want to execute a long process(business process) instead of that we can just extract some function calls and execute them from a mix task using existing data. We can create a mix task with the following code:

defmodule Mix.Tasks.Foo do
  @moduledoc false
  use Mix.Task

  def run(_args) do
    Application.ensure_all_started(:my_app)
    IO.puts("runnning...")
  end
end

We have to name this file foo.ex and place it inside lib folder and now we can run mix foo and we'll get a running... message.

I use this many times, actually I have some defined tasks in many projects than I reuse to debug some workflows.

I know that we "should" be defining the cases that we are debugging in a test, run it and then try to fix the code and then run the tests again but this way works for me so I'm OK with that 🙃

Working with local third party libraries

In some cases we could found some weird behaviour, a bug of just want to know a little more deep about how a third party library works. In that case it could be difficult to setup a local version of a library that we use in our project.

I remember using just pip install -e path_to_library in python and just starting to changing the library code.

In elixir when we want to install a local version of a library we can specify the path of it in the mix.exs file, for example:

defmodule MyApp.MixProject do
  use Mix.Project

  def project() do
    [
      app: :my_app,
      version: "0.0.1",
      elixir: "~> 1.0",
      deps: deps(),
    ]
  end

  def application() do
    []
  end

  defp deps() do
    [
      {:ecto, "~> 2.0"},
      {:postgrex, "~> 0.8.1"},
      {:ecto_sql, path: "ecto_sql_local_path"}
    ]
  end
end

In this case we're telling our project to install ecto_sql from the given path, this will work but just the first time because it will load and compile ecto_sql at the beginning and then when we're making some changes in the code placed in ecto_sql_local_path these changes won't be recompiled automatically because mix is only watching for changes inside our project. In this case we can force to recompile some modules by using for example r Ecto.Migrator from within an iex session but if we are modifying more modules it would be tedious to recompile manually every one of them, for this case we can define a Recompiler module that make this work for us, name it as you want, this will contains:

defmodule Recompiler do
  def run do
    modules_to_recompile = [
      Ecto.Migrator,
      Ecto.SomeOtherModule
    ]

    for module <- modules_to_recompile do
      IEx.Helpers.r(module)
    end
  end
end

We can place this module somewhere inside our lib folder and when we call Recompiler.run from within a iex session it will recompile all the defined modules.

Activate python virtualenv automatically with direnv

Erick 🙃 Navarro — Mon, 01 Mar 2021 05:06:47 +0000

direnv is a tool to set up automatically environment variables as soon as we enter in a directory that contains a .envrc file. We can use this feature to activate our virtualenvs as well.

Let's see what happens when we activate manually a virtualenv with source ./env/bin/activate:

A new new environment variable called VIRTUAL_ENV is exported.
The path is updated to include the bin directory inside our virtualenv this is made to allow us to point to the correct python installation and run cli interfaces exposed by the dependencies we have installed.

Because all the "magic" about activating a virtualenv is basically configuring some environment variables we can do it automatically using direnv.

Let's assume we have a virtualenv installed in the path /Users/erick/.virtualenvs/demo, the virtualenv is located inside ~/.environments because I'm using virtualenvwrapper but it can be in any other location. Now we can use this location to configure our .envrc file as the following:

export VIRTUAL_ENV=/Users/erick/.virtualenvs/demo
export PATH=/Users/erick/.virtualenvs/demo/bin:$PATH

Now when we enter our project folder the virtualenv will be "activated" automatically and when we leave the project folder it will be "deactivated".

Also when we use this method is easiest for our editor(emacs in my case) to recognize the current python installation and be able to run tests, execute files, etc.

DEV Community: Erick 🙃 Navarro

Telegram bot to preview social media content

Add D2 support to Hugo

A simple HTTP server in go

Custom code block render hook

Conclusion

Mermaid preview using xwidget browser

Demo

Moving to Emacs Tree Sitter Modes

How to install grammars

The migration

Conclusion

Bonus

Simple OBS Client in Elixir

Basic configuration

Controlling OBS

Switching from helm to vertico and friends

The problem(?)

Motivation

The new stuff

The migration

Fuzzy search in a project

Search and replace in a whole project

Navigate the kill ring

Fuzzy search inside a buffer

Backup files

Final thoughts

Run ispell on text nodes using tree sitter

Using tree-sitter to get node at the current position

Call ispell using a tree-sitter text node

Putting all together

Auto build and publish emacs org configuration as a website

Automate the org file rendering

Using Github actions to build and publish the rendered config

Create a extra repository

Create a personal token

Configure a secret variable in our dotfiles repository

Configure Github action config file

Easy deploy of docker based projects

Setting up a server

Installing PostgreSQL in our host machine

Configuring Caddy as a reverse proxy

Running our projects with docker-compose

Django application

Phoenix application

Deploying our projects

Deploying new changes

Django application

Phoenix application

Conclusion

Using compilation mode to run all the things

Usage

Interactive

From code

Some tweaks

Evil-mode

Follow compilation output

Enable ANSI colors

Re run compilation from another buffer

Bonus: run parrot mode animation when a compilation is successful

Conclusion

Things I wish I knew when I started in elixir development

Managing multiple versions of elixir and erlang

Using environment variables

File based config file

Creating a new project

Notes about erlang compilation

Ecto database url

Using iex

Enable shell history

Preload aliases

Recompiling modules

Mix tasks

Working with local third party libraries

Activate python virtualenv automatically with direnv

A simple HTTP server in `go`