DEV Community: Ayman Bagabas

Self-hosted Soft Serve

Ayman Bagabas — Fri, 28 Apr 2023 00:00:00 +0000

Soft Serve is a self-hostable Git server for the command-line. It supports Git over HTTP(s), SSH, and the Git Protocol. Soft Serve also comes with a simple straight-forward user management interface for teams.

In this post, we will go through how to set up your Soft Serve instance. This includes setting up SSH access, HTTPS using Certbot, and how to manage your Soft Serve instance.

Prerequisites

In this post, we are assuming that you have a basic knowledge of networking, a general understanding of how to use Linux and the command-line, and are comfortable using git commands.

You will need:

A server to run Soft Serve on.
A domain name to access your server (optional).

If you’re using a cloud provider, make sure you have the right access settings before proceeding i.e. access tokens. Running Soft Serve locally or on-premise will vary depending on your setup. This post will only cover setting up Soft Serve on the host server using Systemd, reroute OpenSSH traffic, and set up Certbot for HTTPS.

We will be using a virtual machine running Ubuntu 22.04 hosted on the cloud. Many cloud providers provide virtual machine services. DigitalOcean calls them Droplets. EC2 if you’re using AWS.

Note: make sure you enable access to the server and add any firewall rules. Soft Serve uses ports 23231/tcp (SSH), 23232/tcp (HTTP), and 9418/tcp (Git). We will reconfigure Soft Serve to run on port 22/tcp (SSH) and 443/tcp (HTTPS), then use port 2200/tcp for OpenSSH shell access.

Setting Up Soft Serve

We will start by installing Soft Serve from Charm’s APT repository, setting up a Systemd service, getting a Let’s Encrypt certificate using Certbot, and lastly, reconfiguring OpenSSH to access the shell on an alternate port (since Soft Serve will be using the default SSH port). Let the fun begin!

Installing Soft Serve

Soft Serve and all other Charm tools can be installed via APT/RPM repositories. Check out the installation section for more options. Since we’re using Ubuntu, we can install Soft Serve from the Charm APT repository:

# Retrieve and import repository key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL <https://repo.charm.sh/apt/gpg.key> | sudo gpg --dearmor -o /etc/apt/keyrings/charm.gpg
# Add APT repository source
echo "deb [signed-by=/etc/apt/keyrings/charm.gpg] <https://repo.charm.sh/apt/> * *" | sudo tee /etc/apt/sources.list.d/charm.list
# Install Soft Serve & git
sudo apt update && sudo apt install soft-serve git

You’re all set! You should now be able to run the soft binary.

soft --version

Now that we have Soft Serve installed, let’s run it locally.

soft


Soft Serve is a self-hostable Git server for the command line.

Usage:
  soft [command]

Available Commands:
  help Help about any command
  serve Start the server

Flags:
  -h, --help help for soft
  -v, --version version for soft

Use "soft [command] --help" for more information about a command.

To start the server, we can run soft serve. This will create a data directory that will store the repositories and database.

2023-04-28 server: Starting Git daemon addr=:9418
2023-04-28 server: Starting HTTP server addr=:23232
2023-04-28 server: Starting SSH server addr=:23231
2023-04-28 server: Starting Stats server addr=:23233

Well, well, we now have a running Soft Serve instance! But, this would be tedious to run each time our server restarts. Luckily, systemd can help us start the process on boot.

Systemd

Create a file under /etc/systemd/system/soft-serve.service and put your Systemd service configuration. Here we will be running Soft Serve as root to simplify things. We will place the server’s data under /var/local/lib/soft-serve. Make sure you have added your SSH authorized key to the SOFT_SERVE_INITIAL_ADMIN_KEYS environment variables. You can remove this later once the “admin” user is created and has your key.

For a full list of Soft Serve server settings and environment variables refer to the Server Settings section.

[Unit]
Description=Soft Serve git server 🍦
Documentation=https://github.com/charmbracelet/soft-serve
Requires=network-online.target
After=network-online.target

[Service]
Type=simple
Restart=always
RestartSec=1
ExecStartPre=mkdir -p /var/local/lib/soft-serve
ExecStart=/usr/bin/soft serve
Environment=SOFT_SERVE_DATA_PATH=/var/local/lib/soft-serve
Environment=SOFT_SERVE_INITIAL_ADMIN_KEYS='ssh-ed25519 AAAAC3NzaC1lZDI1...'

[Install]
WantedBy=multi-user.target

Now, reload Systemd configuration, enable and start the service.

sudo systemctl daemon-reload
sudo systemctl enable soft-serve.service
sudo systemctl start soft-serve.service

We can check the logs using journalctl -u soft-serve.service.

Tip: add -f flag to “tail” the logs as they appear. Useful when using tmux to keep an eye on logs journalctl -fu soft-serve.service.

HTTPS Certificate

To be able to use HTTPS in Soft Serve, we will need to set up TLS certificates so that Soft Serve can use an encrypted connection to communicate with the world. We will be using Certbot to issue us Let’s Encrypt certificates. Following Certbot instructions, we will choose Other and Ubuntu 20 for the instruction options.

Note: if you’re not using a domain name, you won’t be able to issue an HTTPS certificate since Let’s Encrypt doesn’t allow the use of bare IP addresses. ZeroSSL is a great alternative that supports bare IP addresses.

Now, make sure you have updated your DNS records to point your server’s IP address to your custom domain. This is typically done using a A record. This will vary depending on your DNS domain provider. We will be using git.example.com to demonstrate issuing a certificate for the subdomain git.

Install the certbot cli tool using snapd to issue the certificate for our domain.

# Stop Soft Serve
sudo systemctl stop soft-serve.service
# Install certbot from snapd
sudo snap install --classic certbot
# Issue certificate
sudo certbot certonly --standalone
# Enter your email address
# Agree for terms of service
# Enter your domain(s): git.example.com

Voilà, we have an HTTPS certificate!

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/git.example.com/fullchain.pem
Key is saved at: /etc/letsencrypt/live/git.example.com/privkey.pem
This certificate expires on 2023-07-28.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt: <https://letsencrypt.org/donate>
 * Donating to EFF: <https://eff.org/donate-le>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Once we have our certificate, we will need to update our Soft Serve configuration to use them and point to our new https:// address.

Server Configuration

Soft Serve stores its server configuration in the config.yaml file under the data directory. By default, it uses the relative path data as a data directory. You can override this by defining a SOFT_SERVE_DATA_PATH environment variable (as seen above in the systemd service file). This means that our config.yaml file lives under /var/local/lib/soft-serve since we have that as our data directory path.

To use HTTPS default port (443), we have to tell Soft Serve about our Let’s Encrypt certificates. And since we are using a custom domain now, we need to update the server’s public URL. This is the address that will you will be using to manage user access and git clone repositories. Lastly, we will make OpenSSH use a different port, so we can still have shell access on our remote host.

Info: you can override configuration settings using environment variables. For example, to override the server’s name add SOFT_SERVE_NAME=’Git Melon’ to your soft-serve.service file.

Let’s edit the file and see what’s in there 🤔

# Make sure you have $EDITOR defined
# Use vim <3
# export EDITOR=vim
sudo $EDITOR /var/local/lib/soft-serve/config.yaml

Here, we can see the default server configurations.

Change the ssh.listen_addr to use the default SSH port (22).
Use ssh://git.example.com as our ssh.public_url (this will be used for clones over SSH e.g. git clone git@git.example.com:repo.git).
Change the http.listen_addr to use HTTPS default port (443).
Update http.public_url to point to use https:// and point to our custom domain https://git.example.com.
Set http.tls_key_path and http.tls_cert_path to use the generated Let’s Encrypt certificates.

The final configurations look like this:

# Soft Serve Server configurations

# The name of the server.
# This is the name that will be displayed in the UI.
name: "Soft Serve"

# Log format to use. Valid values are "json", "logfmt", and "text".
log_format: "text"

# The SSH server configuration.
ssh:
  # The address on which the SSH server will listen.
  listen_addr: ":22"

  # The public URL of the SSH server.
  # This is the address that will be used to clone repositories.
  public_url: "ssh://git.example.com"

  # The path to the SSH server's private key.
  key_path: "ssh/soft_serve_host"

  # The path to the server's client private key.
  # This key will be used to authenticate the server to make git requests to
  # ssh remotes.
  client_key_path: "ssh/soft_serve_client_ed25519"

  # The maximum number of seconds a connection can take.
  # A value of 0 means no timeout.
  max_timeout: 0

  # The number of seconds a connection can be idle before it is closed.
  idle_timeout: 0

# The Git daemon configuration.
git:
  # The address on which the Git daemon will listen.
  listen_addr: ":9418"

  # The maximum number of seconds a connection can take.
  # A value of 0 means no timeout.
  max_timeout: 0

  # The number of seconds a connection can be idle before it is closed.
  idle_timeout: 3

  # The maximum number of concurrent connections.
  max_connections: 32

# The HTTP server configuration.
http:
  # The address on which the HTTP server will listen.
  listen_addr: ":443"

  # The path to the TLS private key.
  tls_key_path: "/etc/letsencrypt/live/git.example.com/privkey.pem"

  # The path to the TLS certificate.
  tls_cert_path: "/etc/letsencrypt/live/git.example.com/fullchain.pem"

  # The public URL of the HTTP server.
  # This is the address that will be used to clone repositories.
  # Make sure to use https:// if you are using TLS.
  public_url: "<https://git.example.com>"

# The stats server configuration.
stats:
  # The address on which the stats server will listen.
  listen_addr: "localhost:23233"
# Additional admin keys.
#initial_admin_keys:
# - "ssh-rsa AAAAB3NzaC1yc2..."

This looks good so far. Now before we start Soft Serve up again, we need to change the port that OpenSSH uses. This is specified in /etc/ssh/sshd_config.

sudo $EDITOR /etc/ssh/sshd_config
# Uncomment `#Port 22`
# Change `Port 22` to `Port 2200`
# Or use the power of `sed` :)
sudo sed -i 's/^#Port 22/Port 2200/g' /etc/ssh/sshd_config
# Restart sshd
sudo systemctl restart sshd

You can now access your server’s shell on port 2200. Try it out: ssh -i <my-precious-key> -p 2200 git.example.com.

Now, let’s start our Soft Serve server again.

sudo systemctl start soft-serve.service

View logs using journalctl -fu soft-serve.service. Verify the server is indeed running on :22 and :443.

Apr 28 20:33:39 ip-172-31-84-249 soft[7594]: 2023-04-28 server: Starting Git daemon addr=:9418
Apr 28 20:33:39 ip-172-31-84-249 soft[7594]: 2023-04-28 server: Starting HTTP server addr=:443
Apr 28 20:33:39 ip-172-31-84-249 soft[7594]: 2023-04-28 server: Starting SSH server addr=:22
Apr 28 20:33:39 ip-172-31-84-249 soft[7594]: 2023-04-28 server: Starting Stats server addr=localhost:23233

Manage Soft Serve

Now that we successfully set up our Soft Serve server, we want to manage users, server access, and repositories. Since we added our authorized key as an environment variable above, we should be able to see all the admin commands when ssh -i <my-precious-key> git.example.com help.

Soft Serve is a self-hostable Git server for the command line.

Usage:
  ssh git.example.com [command]

Available Commands:
  help Help about any command
  info Show your info
  pubkey Manage your public keys
  repo Manage repositories
  set-username Set your username
  settings Manage server settings
  user Manage users

Flags:
  -h, --help help for this command

Use "ssh git.example.com [command] --help" for more information about a command.

Users and Access

You can manage users using the user command. For example, Soft Serve creates a default admin user on the first run that uses the keys defined in SOFT_SERVE_INITIAL_ADMIN_KEYS. Let’s verify that using the info command.

$ ssh -i <my-precious-key> git.example.com info
Username: admin
Admin: true
Public keys:
  ssh-ed25519 AAAAC3NzaC1lZDI1...

We can add more keys using the pubkey command.

ssh -i <my-precious-key> git.example.com pubkey add ssh-rsa AAAAB3NzaC1yc2...

Use the user command to create more users. Add -a to mark user as admin.

ssh -i <my-precious-key> git.example.com user create lemon -a
ssh -i <my-precious-key> git.example.com user add-pubkey lemon ssh-rsa AAAAB3NzaC1yc2...

To change the current username, use the set-username command.

ssh -i <my-precious-key> git.example.com set-username melon

Repositories

Using the repo command, you can create, delete, import, and manage repository settings. You can add/remove collaborators using repo collab. Let’s go through an example of creating a new repository, pushing code, and adding a new collaborator who can access the repo.

First, we will create a new repository.

ssh -i <my-precious-key> git.example.com repo create hula-hoop

If you ssh into the server (without any arguments) you should see the TUI and the new repository.

Push to Repository

Now, let’s push some files to the repository.

# Clone the repository
git clone git@git.example.com:hula-hoop.git
cd hula-hoop
# Change default branch
git branch -M main
# Add content
$EDITOR README.md
git add README.md
git commit -m "Add README.md"
git push origin main

Collaborators

Let’s add the user lemon that we created earlier as a collaborator. The command add takes a repository name and a username as arguments. repo collab add REPOSITORY USERNAME.

# Add user as a collaborator
ssh -i <my-precious-key> git.example.com repo collab add hula-hoop lemon
# List repository collaborators
ssh -i <my-precious-key> git.example.com repo collab list hula-hoop

Nested Repositories

Soft Serve also supports nested repositories, you can create repositories with any arbitrary path. Go wild!

ssh -i <my-precious-key> git.example.com repo create my/super/nested/new
ssh -i <my-precious-key> git.example.com repo create my/super/nested/new/repository

Mirrors

You can also import repositories from any public remote. Use the repo import command.

ssh -i <my-precious-key> git.example.com repo import soft-serve <https://github.com/charmbracelet/soft-serve>

Use --mirror or -m to mark the repository as a pull mirror.

Metadata

In Soft Serve, a repository has different properties. Repositories can be hidden, private, or mirrored. Repositories can also have their own descriptions and a project name different from the repository’s name. For example, we want the repository we imported above to be presented as Soft Serve rather than soft-serve. To do so, let’s set the repository project name.

ssh -i <my-precious-key> git.example.com repo project-name soft-serve 'Soft Serve'

Let’s also add a description to this repository.

ssh -i <my-precious-key> git.example.com repo description soft-serve 'The hackable self-hosted Git server!'

For more info on repository commands try repo help.

What’s Next?

That’s it for now. Check out Soft Serve’s README for more information on how to use Soft Serve.

Till next time, happy hacking!

Simple Vim Session Management

Ayman Bagabas — Thu, 13 Apr 2023 00:00:00 +0000

Recently I switched back to using Neovim after being a VSCode user for a while. One of the things I miss in VSCode is the session management that comes bundled in. My brain became wired to doing things like code <projectDir> to open the editor with all files loaded just like the way I left them. Luckily in Vim, you can use :mksession to create sessions.

The Problem

Just like VSCode, I want Neovim to save the session file under the project root directory. Specifically in .nvim/session.vim. I want Neovim to detect whether the opened file is a directory, and based on that, load the session file if exists. Before exiting Neovim, save the session file under the project root directory.

You see, I tried different plugins including persistence.nvim, persisted.nvim, auto-session, sessions.nvim, and project.nvim. None of them achieved what I wanted.

How?

Neovim supports sessions. You can use :mksession to create session files, and :source <file> or nvim -S <file> to restore a session. And with the sessionoptions option, we can tell Neovim exactly what to save in the session file. Simple just like that. Pairing that with autocmds on startup and exit we can get what we want. Read more about this :h session.

The Config

First, we need to tell Neovim what to save in the session file. We do so using the sessionoptions option. In your init.lua add the list of session options you want to save and restore:

vim.opt.sessionoptions = "buffers,curdir,folds,help,tabpages,winsize,winpos,terminal,localoptions"

Here I’m choosing almost everything 🙂 Make sure to avoid adding options to the list since it will interfere with other plugins you might have. Read more about this :h sessionoptions.

Now, we need to add out autocmds on VimEnter and VimLeave.

-- Simple session management on directory open
-- Here we check if the opened file is a directory.
-- Then we load the sessionfile if exists.
-- And lastly, we delete the extra directory buffer.
vim.api.nvim_create_autocmd("VimEnter", {
  callback = function(data)
    -- buffer is a directory
    local isdirectory = vim.fn.isdirectory(data.file) == 1
    if not isdirectory then
      return
    end

    -- source session.vim if it exists
    local sessionfile = vim.fn.resolve(data.file .. "/.nvim/session.vim")
    if vim.fn.filereadable(sessionfile) == 1 then
      vim.cmd("source " .. sessionfile)
    end

    -- wipe the directory buffer
    vim.cmd("bw " .. data.buf)
  end,
  nested = true,
})

-- Check if we are in the project root.
-- If we are, save the session file.
vim.api.nvim_create_autocmd("VimLeave", {
  callback = function()
    local isproject = false
    for _, root in ipairs({ ".git", ".hg", ".bzr", ".svn", "Makefile", "package.json", "go.mod" }) do
      if vim.fn.isdirectory(root) == 1 then
        isproject = true
        break
      end
    end

    -- only save session if we are in a project root
    if not isproject then
      return
    end

    local sessionfile = ".nvim/session.vim"
    vim.fn.mkdir(".nvim", "p")
    vim.cmd("mksession! " .. sessionfile)
  end,
})

Bonus

When using git, it can be annoying to see the .nvim directory in every project you have. You can ignore the file by adding it to a ~/.gitignore and exclude the file contents from git.

echo '.nvim' >> '~/.gitignore'
git config --global core.excludesFile '~/.gitignore'

That’s it, hope this helps 😃.

no title

Ayman Bagabas — Thu, 02 Jun 2022 16:11:13 +0000

Blue Mountain

📍Blue Mountain Summit Loop

Import Notion Pages to Jekyll

Ayman Bagabas — Tue, 29 Mar 2022 00:00:00 +0000

Lately, I started using Notion for note-taking, to-do lists, and now for writing blog posts. Notion makes it simple to combine all those activities in a unified pretty interface. Using Jekyll, Github Pages, and Github Actions, I was able to import my Notion Blog database posts into Jekyll using a Github workflow that runs twice a day.

What is a Notion Database?

Notion databases are smart tables that can hold a collection of pages with customizable properties and multiple layouts. Think of it as smart spreadsheets on steroids.

The Layout

Jekyll uses YAML front matter to specify post title, date, tags, and other properties. Using Notion databases properties, we can map Jekyll YAML front matter into a simple Notion Database. Each entry in the database corresponds to a Jekyll post with its respective front matter properties. The script will use the date specified as the published date for the post, it will default to the entry creation date if one was not specified. If you want a page to be published, simply check the “Publish” checkbox.

The Script

Writing the importer script was a breeze using Notion JavaScript SDK and souvikinator/notion-to-md. The SDK pulls the entries in the database, then the converter converts each Notion page into Markdown ready to be used in Jekyll.

To use the script, first, you have to create a new Notion integration that can access your Blog page database. Simply go to settings, choose integrations, click on “develop your own integration” and create a new integration with the proper scopes.

After creating that, you will get an integration key that would be used later with the script for it to work. But before that, you will need to invite your newly created integration bot to the Notion page that has the database that you want to use. Simply click the “share” button on the database page and invite the integration you just created. It will have the same name you specified when you created the integration.

This is how the script works:

// Create a Notion client from an environment variable
const notion = new Client({
    auth: process.env.NOTION_TOKEN,
});

// Query the database and filter out unpublished entries
const response = await notion.databases.query({
        database_id: process.env.DATABASE_ID,
        filter: {
            property: "Publish",
            checkbox: {
                equals: true
            }
        }
    })

// Iterate over the results
for (const r of response.results) {
  // build the post front matter
  // convert the page to markdown
  // write it to disk
}

Here, we’re using 2 environment variables to store the Notion integration token and the database ID. You can find the database ID in the Notion page URL. https://www.notion.so/<database_id>?v=<long_hash>

You can find the importer script here.

Periodically Import Content

Using Github Actions, we can create a workflow that runs periodically every hour to run the script and then publish the contents using Github Pages. This depends on your setup, for me, I’m hosting my website on Github Pages just because of its convenience.

The workflow runs the script, then commits all the changes back to the repository. This then triggers Github Pages to create a deployment of the website and publish the changes. Here is the workflow I’m currently using:

name: Jekyll importer

on:
  push:
  schedule:
    - cron: "0 */1 * * *"

jobs:
  importer:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@master

      - uses: actions/setup-node@v2
        with:
          node-version: "17"

      - run: npm install

      - run: node _scripts/notion-import.js
        env:
          NOTION_TOKEN: $
          DATABASE_ID: $

      - uses: stefanzweifel/git-auto-commit-action@v4
        env:
          GITHUB_TOKEN: $
        with:
          commit_message: Update Importer posts
          branch: master
          commit_user_name: importer-bot 🤖
          commit_user_email: actions@github.com
          commit_author: importer-bot 🤖 <actions@github.com>

Here, I’m running the workflow every hour and using Github Secrets to store the Notion token and database ID, then use those as environment variables when running the script.

That’s it for now! Go on and write your next post!

Nyan Cat Over SSH

Ayman Bagabas — Fri, 25 Mar 2022 00:00:00 +0000

Everyone knows the Nyan Cat meme that started in 2011. The meme started from a YouTube video that merged a Japanese pop song with an animated cartoon cat. Then a Telnet Nyan Cat server was created by klange/nyancat. And now, in 2022, after 11 years of nyaning, it got ported to SSH! Try it out here ssh dir.charm.sh -p2226

The Why

Well, to put it simply, why not port it to SSH! SSH is safer and a more modern protocol than Telnet is.

The How

For this, I went with charmbracelet/wish, which makes it easier to build SSH servers in Golang. Given it already has a middleware for the awesome charmbracelet/bubbletea TUI framework, this just made the port much easier to implement. On top of that, I used charmbracelet/lipgloss to do the styling and coloring.

Implementation

Nyan Cat C implementation consists of multiple frames that then get translated to ANSI256 colors. Each character in these frames represents a color which then gets rendered as a background of 2 whitespace characters within a 90-millisecond interval (the default).

All that is done after the program determines the terminal size and whether or not to crop the frames before they are being rendered. Thankfully, I didn’t need to do any of those calculations, thanks to NARKOZ port of Nyan Cat in Golang.

Bubbletea is a framework based on The ELM Architecture which basically means there are 3 main components, a model struct, an update function, and a view function. In this case, the model will have the terminal size (number of rows and columns), the program start time, and the frame index that should be rendered next. The update function waits for the 90-millisecond interval to pass before it can increment the frame index in the model. Which then triggers the view function to draw the contents in the terminal window.

SSH Server

Here comes charmbracelet/wish into play. It makes creating SSH applications in Golang super easy. Just wrap your application in a Wish middleware that handles SSH operations like commands, public key or password access, users, etc.

The server works by receiving requests from clients, reading the client’s terminal window size, creating a Bubbletea program for that session with the given window size, and finally rendering the program to the SSH session.

Where To Find?

You can find this project on Github aymanbagabas/nyancatsh. You could also install it using go install github.com/aymanbagabas/nyancatsh@latest. If you want the non-ssh version that only uses Bubbletea go install github.com/aymanbagabas/nyancatsh/cmd/nyancat@latest

Credits

This wouldn’t happen without these awesome projects:

Run native C/C++ code in the browser using Emscripten

Ayman Bagabas — Wed, 18 Nov 2020 00:00:00 +0000

While working on WMIDumpper, a simple tool that analyzes ACPI WMI blocks, I had to figure out how to implement bmfdec in JavaScript. My first thought was to port it to JavaScript and put in the time and effort to rewrite ~1500 lines of C code in JS. But then a light bulb went on in my head, WebAssembly! A quick search showed that Emscripten is exactly what I need. It can compile C/C++ native code into WebAssembly and run it on the web.

Building native code into WebAssembly/JS

I cloned bmfdec and followed Emscripten documentation on how to compile the project into wasm. Building bmfdec was straightforward using Emscripten compiler.

emcc bmfdec/bmf2mof.c -o bmf2mof.js

There was one problem though, bmfdec was written to take input from stdin as a binary file. I had to modify bmfdec.c and change the main() function to take a buffer instead of reading binary from stdin and called it parse_data().

int parse_data(uint8_t *pin, ssize_t lin) {
  static char pout[0x40000];
  int lout;
  if (lin < 0) {
    fprintf(stderr, "Failed to read data: %s\n", strerror(errno));
    return 1;
  } else if (lin == sizeof(pin)) {
    fprintf(stderr, "Failed to read data: %s\n", strerror(EFBIG));
    return 1;
  }
  if (lin <= 16 || ((uint32_t*)pin)[0] != 0x424D4F46 || ((uint32_t*)pin)[1] != 0x01 || ((uint32_t*)pin)[2] != (uint32_t)lin-16 || ((uint32_t*)pin)[3] > sizeof(pout)) {
    fprintf(stderr, "Invalid input\n");
    return 1;
  }
  lout = ((uint32_t*)pin)[3];
  if (ds_dec((char *)pin+16, lin-16, pout, lout, 0) != lout) {
    fprintf(stderr, "Decompress failed\n");
    return 1;
  }
  return process_data(pout, lout);
}

Exporting native functions to JavaScript

To be able to call C functions within JavaScript, we have to compile bmf2mof with some extra flags to modularize, and export symbols to the JS output file.

Using MODULARIZE compiler flag makes the generated JavaScript modular where you can use promises and require() in Node. EXPORT_NAME='bmf2mof' compiler flag changes the exported module name, in this case, it would be named bmf2mof(). WASM=1 specifies that we want a wasm output. And finally "EXPORTED_FUNCTIONS=['_parse_data']" exports the function parse_data from the C code. We also want to optimize the output JS code so we will use -O2.

emcc bmfdec/bmf2mof.c -s "EXPORTED_FUNCTIONS=['_parse_data']" -s "MODULARIZE=1" -s "EXPORT_NAME='bmf2mof'" -s "WASM=1" -O2 -o bmf2mof.js

Now the generated bmf2mof.js will have a _parse_data function that maps to the C function and can be called from the JavaScript code.

const bmf2mof = require('bmf2mof.js')

const buf = new Uint8Array([0x46, 0x4F, 0x4D, 0x42, ..., 0x20, 0xEC, 0xFF, 0x0F])

bmf2mof().then(instance => {
    function arrayToPtr(array) {
        var ptr = instance._malloc(array.length)
        instance.HEAPU8.set(array, ptr)
        return ptr
    }

    instance._parse_data(arrayToPtr(buf), buf.length)
})

Everything works as expected, running the code above node [filename].js outputs the data to stdout. However, if we want to use this in the browser, the output would go to the console. So we have to figure out a way to catch the output and redirect it to where we want, in this case, it would be a textarea in WMIDumpper.

Emscripten documentation was a bit lacking when it comes to this, the generated JS file, so I took the time to try and understand what it does. It turned out that the generated JS defines some default functions to handle the mapping from C/C++ to JS. For example, it defines a printErr function that binds to console.warn meaning C/C++ code that prints to stderr would use console.warn in the JS. See Create the Module object for more technical details.

The generated JS takes an object that overrides the default module functions. To redirect our output from stdout to the textarea in the HTML, all we need to do is define our own print function.

const textarea = document.getElementById('textarea')

bmf2mof({
    print: function (text) {
        textarea.value += text + '\n'
    }
}).then(instance => ...)

And of course, we need to source the generated JS script in the HTML before we can use this code in the browser.

Reference

You can find the WebAssembly forked version of bmfdec at bmf2mof.wasm.

WMIDumpper really is just a web clone of wmidump and bmfdec so kudos to iksaif and pali for their awesome work.

Dynamic DNS using DDclient

Ayman Bagabas — Sat, 16 Feb 2019 00:00:00 +0000

DDclient is a service used to update dynamic DNS entries on many services. It is useful if you need a DDNS client that can work with pretty much any DNS service. Most distributions provide DDclient in their official repositories. It is available on Debian-based systems, Fedora, Archlinux, and many more.

In my case, I need DDclient for my NextcloudPi server that runs on a Raspberry Pi 3B hooked up to a storage device. This server gives me access to my files anywhere anytime as long as it has a working internet connection. I’m using a free DNS service from Dynu and in their website, they go in details of how you would set up dynamic DNS using DDclient. It turns out that they use a dyndns2 protocol by www.dyndns.com to provide this service.

Now after installing DDclient, the client can configure it under /etc/ddclient.conf where you can set the update interval, server address (in this case dynu.com), username and password, etc.

protocol=dyndns2
pid=/var/run/ddclient.pid
ssl=yes
use=web, web=checkip.dynu.com/, web-skip='IP Address'
server=api.dynu.com
login=[YOUR_DYNU_USERNAME]
password='[YOUR_PASSWORD]'
[YOUR_DNS_ADDRESS]

As you can see I’m using dynu.com server. Different DNS services would have different configurations.

Writing a Chip-8 emulator

Ayman Bagabas — Mon, 17 Sep 2018 00:00:00 +0000

If you ever played retro games on modern computers, then you probably know what an emulator is. Chip-8 is an interpreted programming language that was created originally by Joseph Weisbecker. Chip-8 programs get interpreted by a virtual machine. It offers a very simple monochrome graphics and uses a 4Kb of memory. It has the “8” part because all the system’s components, like CPU registers, have a size of 8 bits or 1 byte.

What is an Emulator?

The idea of an emulator is the same across any operating system. You basically try to translate the instructions of one system to another. The process of translation is done in three main parts:

Fetching opcodes
Decoding opcodes
Executing opcodes

These three steps happen in the CPU which is the main component of a system. The first step is to read operation codes from the program, then the CPU tries to decode these codes and then executes them accordingly.

Chip-8

Space Invaders

Before you get very excited, you really have to understand how the system works and behaves. Get familiar with your binary and hexadecimal conversions. A hex viewer may come handy when debugging a program.

CPU and Memory

Chip-8 is a simple system that has 16 CPU registers, each takes information up to 8 bits (1 byte). The program counter, I register, opcode placeholder, and a stack pointer, all have a size of 16 bits (2 bytes). Memory is a 4Kb memory where the first 512Kb is reserved for the interpreter which, this makes most programs written for the Chip-8 start at location 512. A minimal of 16 level stack pointer is required and it is used to store return locations from the program counter register.

Timers

There are two timers both countdown from 60 to 0. Delay timer is used for program events and its value can be set and read. And a sound timer which plays a beep whenever it reaches 0. After every operation execution, both timers get subtracted by 1.

Input

The Chip-8 uses 16 keys of input, (0x0-0xF) which are usually mapped to:

1 2 3 C 1 2 3 4
4 5 6 D --\ Q W E R
7 8 9 E --/ A S D F
A 0 B F Z X C V

Usually ‘2’, ‘4’, ‘8’, and ‘6’ are used for directions.

Graphics

The Chip-8 has a 32x64 pixels monochrome display. It draws graphics on the screen using sprites. “A sprite is a group of bytes which are a binary representation of the desired picture.” And they may take up to 15 bytes. Chip-8 provides a set of predefined sprites for representing hexadecimal digits from 0 to F. For example:

  "1" | Binary | Hex
------+----------+-----
  * | 00100000 | 0x20
 ** | 01100000 | 0x60
  * | 00100000 | 0x20
  * | 00100000 | 0x20
 *** | 01110000 | 0x70

  "F" | Binary | Hex
------+----------+-----
**** | 11110000 | 0xF0
* | 10000000 | 0x80
**** | 11110000 | 0xF0
* | 10000000 | 0x80
* | 10000000 | 0x80

These sprites should be stored in the reserved interpreter area 0-512 of memory.

Implementation

Start with the big picture, how does your computer works? After you boot your computer it starts initializing components and devices. This includes inputs, outputs, graphics, CPU, etc. Then it loads the system and starts executing instructions sequentially. The main loop should look something like this:


#include // Chip-8 system
#include // Input
#include // Graphics

int main(int argc, char **argv) {
    initChip8();
    initInput();
    initGraphics();

    loadROM("INVADERS");

    while(systeIsRunning) {
        // Execute instruction
        executeOP();

        // Refresh display if flag is set
        if (drawFlag)
            drawGraphics();

        // Play beep if flag is set
        if (playSound)
            playBeep();

        // Set input and set keys states
        setInput();
    }
    return 0;
}

Chip-8 should implement ways to load and execute instructions:


void initChip8() {
    // initialize system
    // memory
    // registers
    // stack
    // graphics
}

void loadROM(file) {
    // read file into memory
    // starting from location 512Kb
    // or 0x200 in hex
}

void executeOP() {
    // fetch opcode
    opcode = (memory[pc] << 8) + memory[pc + 1];
    // decode opcode
    switch (opcode & 0xF000) {
        // execute opcode
        case 0x0:
            if (opcode == 0x00E0)
                clearDisplay();
            elseif (opcode == 0x00EE)
                // return from subroutine
        case 0x1:
        .
        .
        .
        case 0xF:
    }

    // timers
    if (delay_timer > 0)
        --delay_timer;
    if (sound_timer > 0) {
        if (sound_timer == 1)
            playSound();
        --sound_timer;
    }
}

Input and graphics depend on the libraries you will be using. I went with SDL2 for these two since it is cross-platform, very well known, and has a lot of documentation. You can refer to my Chip-8 implementation here.

What next?

Well, I really learned a lot from this project. Next, I want to extend this project to have a terminal based UI using ncurses. Or maybe work on a more complex system like the NES?

References

Arch Linux on Matebook X Pro

Ayman Bagabas — Mon, 23 Jul 2018 00:00:00 +0000

Recently, I got a new laptop, Huawei Matebook X Pro. It has an i7 8th Gen. Intel CPU, 16Gib of RAM, 512Gib of SSD storage, Nvidia MX150 GPU with 2Gib of DRAM, and a beautiful HiDPI and touchscreen. It also comes with a fingerprint sensor. The first thing I did was installing Arch Linux, because well I am a Linux user! In the end, everything was working properly except:

~~Two out of four speakers.~~ See UPDATE1 down below
The fingerprint sensor.
~~Some keys in the Fn row.~~ See UPDATE2

The installation was very straightforward and like every other Arch Linux installation process. I followed their installation guide. After the installation is complete, everything was working properly except for some minor issues.

Installation

First, disable secure boot from the BIOS menu by pressing F2 while booting. Then have your Arch Linux installation media ready, then you can access the boot menu by holding F12 while booting. Once you boot into Arch Linux live boot image, you will notice that the font is too small, you can change that to use a large font with setfont latarcyrheb-sun32, or you could use any other font you like.

Multi-lib repositories

You probably want to enable multi-lib repos to support 32-bit software. Just uncomment that in /etc/pacman.conf or add the following:

[multilib]
Include = /etc/pacman.d/mirrorlist

Grub

I used Grub for the bootloader. Obviously, you want to use Grub for UEFI systems. For the ESP location, I had mine set to /boot/efi just to follow other Linux distors approach. Because of the HiDPI screen that comes with this laptop, Grub would very tiny to see, a quick fix is to set the GRUB_GFXMODE variable to something like 1600x1200x32. The available values can be fetched from Grub command line by executing videoinfo. Edit your /etc/default/grub file to include these lines:

GRUB_GFXMODE=1600x1200x32
GRUB_GFXPAYLOAD_LINUX=keep

If you are dual-booting you should install the package os-prober to make Grub detect Windows partitions. don’t forget to regenerate grub.cfg using:

sudo grub-mkconfig -o /boot/grub/grub.cfg

After Installation

Because I have a dual-boot setup with Windows, I ran into a little problem after partitioning Windows partition. However, it was easily fixed using ntfs-3g package which includes the tool ntfsfix. After you complete the installation and boot into Arch, just run the tool with -b to fix bad sectors and -d to clear dirty flag:

sudo ntfsfix -b -d /dev/nvme0n1p3

In my case, my Windows partition was /dev/nvme0n1p3, you should change that based on your partition name. You can use lsblk or blkid to list all your partitions.

AUR helper

One of Arch Linux beauties is AUR, where you can easily get any Linux package installed with ease. I used the tool aurman which is IMO one of the safest ones out there. Here is the full list of AUR helpers. To install aurman:

git clone https://aur.archlinux.org/aurman.git
cd aurman
makepkg -si

Desktop Environment

Since this device comes with a touch-enabled and HiDPI screen, I decided to go with Gnome because it supports these two things pretty well.

sudo pacman -S gnome gnome-extra

This would automatically install Gnome with Wayland. Wayland doesn’t require any further configuration or drivers.

Nvidia driver & Bumblebee

The MBXP comes with Intel UHD Graphics 620 and NVIDIA Geforce MX150. If you are planning to use the Nvidia card for gaming, rendering, or anything your best two options are using Prime technology with Nouveau (open source NVIDIA driver), or use NVIDIA proprietary driver with Bumblebee. I decided to go with the later because it offers better performance. Install Bumblebee

sudo pacman -S bumblebee bbswitch nvidia mesa acpi_call lib32-virtualgl lib32-nvidia-utils

Enable Bumblebee service and add user to Bumblebee group:

sudo systemctl enable bumblebeed.service
sudo gpasswd -a $USER bumblebee

You probably need to Enable NVIDIA card during shutdown to avoid issues with using it. Now we need to tell Bumblebee to use bbswitch for card switching. Edit /etc/bumblebee/bumblebee.conf to include this:

[driver-nvidia]
PMMethod=bbswitch

Sometimes Bumblebee doesn’t detect the card which results in an error. You need to define the card BusID in /etc/bumblebee/xorg.conf.nvidia, just uncomment the line where it has “BusID” and set it to the actual device ID. You can get that using lspci. In my case, it was “PCI:01:00:0”.

Power Saving

I was able to get a great 8-10 hours of battery with low brightness. I am using TLP for managing power saving.

TLP

Install TLP:

sudo pacman -S tlp tlp-rdw

Enable TLP:

sudo systemctl enable tlp.service
sudo systemctl enable tlp-sleep.service
sudo systemctl mask systemd-rfkill.service
sudo systemctl mask systemd-rfkill.socket
sudo systemctl enable NetworkManager-dispatcher.service

And since TLP enables NetworkManager by default, there is no need to enable that. You want to set TLP default mode to the battery. Edit /etc/default/tlp to have these settings:

# Operation mode when no power supply can be detected: AC, BAT.
TLP_DEFAULT_MODE=BAT

# Operation mode select: 0=depend on power source, 1=always use TLP_DEFAULT_MODE
TLP_PERSISTENT_DEFAULT=1

Also since we are using Bumblebee with Nvidia you have to make sure that TLP doesn’t enable power management for the card which can break auto switching with Bumblebee. You should avoid using powertop --auto-tune since it enables power management resulting in breaking Bumblebee. Make sure you have these lines in /etc/default/tlp to exclude the Nvidia card from power management:

RUNTIME_PM_BLACKLIST="01:00.0"
RUNTIME_PM_DRIVER_BLACKLIST="amdgpu nouveau nvidia radeon"

Again make sure that you have the correct bus id for your card.

Audio

In /etc/modprobe.d/audio_powersave.conf add to enable audio power saving:

options snd_hda_intel power_save=1

WiFi

Since this laptop comes with Intel Wireless 8265, we can use iwlwifi power saving options. Add these options in /etc/modprobe.d/iwlwifi.conf:

options iwlwifi power_save=1 d0i3_disable=0 uapsd_disable=0
options iwldvm force_cam=0

Intel GPU

Power saving options for Intel GPU, just stick this line in /etc/modprobe.d/i915.conf.

options i915 enable_guc=3 enable_fbc=1

Suspend and Hibernate

Personally, I prefer hibernating my machine whenever I do not use it for a long time, that is why I use suspend-then-hibernate. Suspend to RAM works out of the box, but hibernate requires some work. First, make sure you have either a swap partition or swap file. I went with swap file just because it does not require partitioning. According to Redhat Recommended System Swap Space, 1.5 of system RAM is the recommended amount of swap for hibernation which is 24Gb in this case.

sudo fallocate -l 24G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Now define that within /etc/fstab for auto mounting:

/swapfile none swap defaults 0 0

In order for hibernate to work, you have to define where the system should look for the resume image in your Linux partition. You can define that in the kernel parameter in your bootloader resume=UUID=ce6dd35a-08d5-4b49-a46c-eff1de8937ce. Here I am using partition UUID, you can get that with sudo blkid. Since I am using a swap file, I also have to define a resume_offset=645120 which is the location of the swap file in the partition. You can get that using sudo filefrag -v /swapfile. Finally, add resume hook after udev and i915 module in /etc/mkinitcpio.conf. Regenerate initramfs sudo mkinitcpio -P.

MISC

Add pcie_aspm=force kernel parameter to enable ASPM (Active State Power Management).
Disable watchdog, add blacklist iTCO_wdt to /etc/modprobe.d/nowatchdog.conf.
Take a look at Improving Performance and Power Management.
Regenerate initramfs sudo mkinitcpio -P.
Regenerate grub.cfg sudo grub-mkconfig -o /boot/grub/grub.cfg.
Install Plymouth.
Enable auto-brightness aurman -S iio-sensor-proxy.

Files

Config files used in this post etc.zip.

UPDATE 1 - fix sound

You can fix the sound issue with hdajackretask which is part of alsa-tools package then follow this picture and click on “Install boot override”:

HDAJackReTask

You might need to set “connectivity” to “internal” to get it working. Finally, recreate your initramfs sudo mkinitcpio -P and reboot.

UPDATE 2

Missing hotkeys micmute, wlan, and pc manager now work using this driver. It will be part of linux 4.21 along with the speakers fix and micmute LED.
Some people reported slow network connection with the above settings. To fix that, drop uapsd_disable=0 from /etc/modprobe.d/iwlwifi.conf.
Also if you were using full disk encryption, don’t forget to add the speakers fix firmware files to /etc/mkiniticpio.conf like this: FILES=(/usr/lib/firmware/hda-jack-retask.fw).

Suspend then hibernate in systemd 239

Ayman Bagabas — Wed, 18 Jul 2018 00:00:00 +0000

In systemd 239, they have added a new service that handles suspending then hibernating after a given amount of time. This is easier than using external scripts since it comes built-in with this version of systemd. You can check systemd version with systemctl --version.

First, you have to define the delay time before the system wakes up and go into hibernation and that should be defined in /etc/systemd/sleep.conf

[Sleep]
HibernateDelaySec=15min

Here, what we care about is the last line HibernateDelaySec where you can define delayed time. As you see, I have it set to 15 minutes after suspending.

Lastly, we need to override the default suspend to execute suspend-then-hibernate instead of regular suspend:

ln -s /usr/lib/systemd/system/systemd-suspend-then-hibernate.service /etc/systemd/system/systemd-suspend.service

This will make systemd executes suspend-then-hibernate instead of suspend every time suspend is invoked.

Reference

systemd-sleep.conf

systemd-suspend-then-hibernate.service

Minesweeper using HTML5

Ayman Bagabas — Sun, 13 May 2018 00:00:00 +0000

Obviously, everyone was born before the year 2000 knows the famous classic game ‘Minesweeper’. To me, I knew this game when I was little at the time where Windows XP was ruling everywhere. It is funny because at that time I did not know exactly how the game is played. That time the game was some kind of a luck game to me where you try to eliminate all the squares except the ones with mines until you either win or lose 😆.

Until recently, a friend pointed out that they saw a video of a guy who made a fully perfect AI to solve the game. I liked the idea so I decided to make one. I started by learning how the game works and it turned out to be very simple. There is a certain number of mines in the game and the player has to discover where these mines are to win the game. Every square has a weight that shows how many mines they are within the surrounding 8 squares of that particular square. If the weight was zero the game will reveal all the surrounding squares except the ones with the mines, otherwise, it will reveal the square itself. Although things did not go very well with the idea, I ended up making the game only.

How does it work?

With the help of HTML5 canvas element, you can draw geometric objects. We can achieve this dynamically using JavaScript. First, we create a canvas element in the page:

<canvas id="example" width="500" height="500">Any text here will get displayed if the browser does not support HTML5 canvas</canvas>

Then we use JavaScript to create objects and control their properties:

var canvas = document.getElementById("example");
var context = canvas.getContext("2d");
context.beginPath();
context.fillStyle = 'red';
context.fillRect(0, 0, 10, 10);
context.closePath();

This will create a square with side equals to 10 and a position of (0, 0). Easy and simple.

Grub

Minesweeper

Coming from an Object-Oriented Programming mentality, I wanted the ability to create classes just because I am used to it this way 😃. I was surprised when I knew that you can mimic creating classes in JavaScript .

I started by questioning what attributes each square has? And I came up with these: isMine, isFlagged, isDown, x, y, and weight. isMine tells if a square is a mine. isFlagged is when a square is being flagged or marked. isDown if a square is revealed. x and y hold the location in the game. weight is a number greater than zero where it holds how many mines within its surrounding squares. With these attributes I came with this JS class:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

class square {
    constructor(x, y) {
        this.x = x;
        this.y = y;
        this.isDown = false;
        this.isMine = false;
        this.isDown = false;
        this.weight = 0;
    }
    get x() { return this._x; }
    set x(value) { this._x = value; }
    get y() { return this._y; }
    set y(value) { this._y = value; }
    get weight() { return this._weight; }
    set weight(value) { this._weight = value; }
    get isDown() { return this._isDown; }
    set isDown(value) {
        this._isDown = value;
        if (this.isMine) {
            end = true;
        }
    }
    get isMine() { return this._isMine; }
    set isMine(value) { this._isMine = value; }
    get isFlagged() { return this._isFlagged; }
    set isFlagged(value) {
        this._isFlagged = value;
        if (value)
            numberofmines = (numberofmines <= 0) ? 0: numberofmines - 1;
        else
            numberofmines++;
        updateHeader();
    }
}

The game does not have a loop, it contains two main functions. The first one is called to initialize the game environment like setting mouse event handler. Here is where all the magic happens, whenever the user clicks the first click the timer starts and it passes the click to the game. If it was inside the game frame where the squares are located it will register a click for that square. Otherwise, it will start a new game. Which is the second function, the previous function is only called once. This one is called to clear all the previous data in the game and create a start a new game. Obviously, this is done with the help of other functions to make it a little more organized.

Source code

The source code is available at JSMinesweeper or you can try it out here .

Sync Google calendar using Vdirsyncer and Orage

Ayman Bagabas — Sun, 08 Apr 2018 00:00:00 +0000

For a long time I wanted to have all my calendars and todo lists synchronized with my current desktop setup. Currently, I am using XFCE software, like Thunar, with Openbox to manage everyday stuff. Yes, I know with Gnome DE you can achieve that easily, but Gnome has a HUGE package dependencies. XFCE on the other hand is very light-weight and simple.

The other wonderful piece of software is vdirsyncer. It supports CalDAV protocol which is also supported by Google calendar. With simple configurations we can synchronize Google calendar locally. Then we can use Orage to view/modify our calendar files. Orage is a time-managing application that can manage your calendars, appointments, alarms, and todo lists.

First, make sure we have all the needed packages. Orage and Vdirsyncer should exist in most distros official repositories. In Archlinux, the installation can be done with sudo pacman -S vdirsyncer orage. It would be similar for other distros as well. Ubuntu/Debian sudo apt-get install vdirsyncer orage.

Orage highlighting events

Here you can see how orage highlights dates with events attached to them. For example, April 1st is Easter Sundy. That was pulled from my Google United States holidays calendar.

Orage displaying events

Vdirsyncer setup

First, we start by configuring Vdirsyncer to sync Google calendars. My configuration file looks like this:

[general]
status_path = "~/.vdirsyncer/status/"

[pair calendar]
a = "google_calendar"
b = "local_calendar"
collections = ["from a", "from b"]
metadata = ["color"]

[storage local_calendar]
type = "singlefile"
path = "~/.calendars/%s.ics"

[storage google_calendar]
type = "google_calendar"
token_file = "~/.vdirsyncer/google_token"
client_id = "CLIENT_ID"
client_secret = "CLIENT_SECRET"

Here, we specified were the status for Vdirsyncer should be located. Mine is located under ~/.vdirsyncer/status. Then we tell Vdirsyncer to create a pair of calendars for bidirectional syncing, where a is the actual Google calendar, and b is the local files pulled from the cloud. Then we define the two calendars as storages.

storage local_calendar, here we specify what the type of this storage which is “singlefile” meaning Vdirsyncer would have a single file for each calendar that you have.
storage google_calendar, here where it gets interested. The type here is “google_calendar” which means use Google cloud to fetch the calendars. With this type you have to specify a token file, client id, and client secret. All of these are essential for synchronizing.

Now we have to enable CalDAV API for our Google account.

Go here and create a new project. This is required so that you can enable CalDAV API for that project.
Click on “Enable APIs and Services” or on the left side click on Library, then search for “CalDAV” and enable it.
Now we need to get our credentials, client secret and id, for the API. Click on “create credentials” then choose “CalDAV API” for your API and select “Other” for Application type. Click next and choose a name e.g. “vdirsyncer” then click continue.
You will get your client id, stick it into your configuration file.
We still missing the client secret. Click done after you get the client id then click on the credentials name and get the client secret from there.

Now we have Vdirsyncer config setup, we need to authorize it to access our Google account.

vdirsyncer discover calendar

This will try to authorize the pair ‘calendar’ defined in the config file. A browser window or a link should pop up to complete the authorization and that would create the access token defined in the config. Now we can synchronize Google calendar by

vdirsyncer sync

Orage setup

Now we need to tell Orage where are the calendar ics files so that we can view them in Orage. Run Orage then go to File -> Exchange data, and add all the pulled ics files from ~/.calendar. Notice the read only check mark, if you want to modify your Google calendar you have to make sure to uncheck that when you add your calendar file which is the same as your Google account name “google@gmail.com.ics”. You should see all your calendars events highlighted within Orage. You can play with Orage settings to hide the window borders and tray icon. When double click on a date an events window should pop up then you can add events, todo, etc, but make sure you select your Google account file from the top bar in order to have it synchronized.

Using Cron to synchronize periodically

You can add a cron to automate the sync command. crontab -e

MAILTO=""
@hourly vdirsyncer sync

This will sync calendars every hour. Make sure you have a cron service running. I am using “cronie” for my setup and with Archlinux, I had to enable the cronie service for that to happen.

sudo systemctl enable cronie
sudo systemctl start cronie