DEV Community: Gaveen Prabhasara

Understanding the Linux Boot Sequence

Gaveen Prabhasara — Sat, 21 Oct 2023 00:00:00 +0000

In my previous post on the Asura Linux blog, I introduced Asura Linux as an Open Source project to experiment with different ways of building and running modern general-purpose Linux systems—and to share what we learn along the way. I explained why the Asura Linux Project exists and what I was thinking about the immediate future. If you haven't seen it, please have a look.

In my excitement about introducing the project, I promised to publish the second blog post about the Linux boot process shortly. If you are following me (or Asura Linux) on the fediverse, you may have seen a couple of updates since then. In summary, I suffered from multiple back-to-back health issues related to a viral flu. Whenever I wasn't in bed, I still had to catch up with work, even though I was in no condition to work. Since I'm finally feeling close enough to normal, let's begin without further ado.

When I started using Linux, I didn't feel I grasped what was going on with my computer until I could make sense of the long swaths of text that flew by during the boot time. While most of the latter part of the boot sequence showed me the statuses of starting system services, I could see there were different stages to the boot sequence.

The boot sequence is the first user experience (UX) when interacting with a computer. It starts by powering up the hardware, giving the firmware the control, which ultimately hands over the boot process to the software in the computer before a user can interact with their productive work. From both technical and UX perspectives, the boot sequence marks a good starting point to discuss running Linux.

This post discusses the Linux boot sequence specifics under three scenarios: the past, present, and future. Therefore, let's start by understanding the overall generic process along with a few key concepts.

A Generic Boot Sequence

Power On
- The computer motherboard and the rest of the essential electronics are powered up.
Firmware Initialization
- The firmware (usually stored in a chip) starts running.
- The firmware makes sure the essential hardware components are functional.
  - If there are errors, they are indicated (e.g., via sounds, lights, and messages)
- The firmware then tries to read a bootable device based on the configured boot order.
Reading the Boot Device
- A boot device (usually a specific section) is read.
- This location contains either a bootloader or further instructions to locate one.
Bootloader
- A bootloader is a program configured to know how to locate and load an Operating System (OS).
- Sometimes, a bootloader can point to another bootloader instead of an OS.
Loading the Operating System
- The final-stage bootloader program loads the essential parts of the OS into the memory, adhering to any special instructions given.
- It also locates and makes the file systems available for the OS.
Operating System Initialization
- The OS usually hands the rest of the boot sequence to a supervisor program that knows how to start and manage software processes.
- The supervisor program then starts/stops systems and application services as configured.
- Finally, the user sees a login prompt when the login service is ready.

The boot sequence can be considered done when the 'init' system completes starting/stopping necessary background services and finally provides the user with a login prompt.

Usually, the essential parts of an OS loaded in step 4 include the following.

Kernel
Initial RAM File System

The Kernel is the core part of the OS that communicates with the hardware and provides management functions of resources such as tasks, I/O, and memory. Therefore, the kernel is a bare minimum requirement to boot a Linux system. For the convenience of the boot process, it is usually included in a compressed image file that knows how to uncompress and load itself into the system memory. The kernel image file usually carries a name starting with vmlinuz- and is located in the /boot directory.

The Initial RAM File System (initramfs or initrd) is an archive file that contains the minimum operating environment necessary to successfully mount the root file system, and further boot instructions. It is also located usually in the /boot directory.

Next, let's dive into the three scenarios we mentioned above now that we know the stages of the Linux boot sequence. We'll skim over the generic stages and fill in the details relevant to the specifics of each scenario.

Boot Process in the Past (BIOS, SysV Init)

Firmware Initialization (BIOS)
- Most older computers (and even current ones for compatibility reasons) support a type of firmware known as BIOS.
  - BIOS stands for Basic Input/Output System.
- The BIOS firmware has a self-diagnostic state known as Power-on Self-test (POST).
  - POST is responsible for verifying the integrity of BIOS and initializing a few essential system components such as CPU, RAM, and I/O controllers.
- After initialization, BIOS looks for a bootable device based on the boot order specified in its settings (e.g., floppy disk, hard disk, CD-ROM).
Reading the Boot Device
- The BIOS-based systems usually support a disk partitioning scheme known as MBR.
- A bootable device is formatted to have a Master Boot Record (MBR), the first 512 bytes of a drive reserved for specific information such as the Partition Table of the particular disk and a few boot instructions.
- The limited space in the Master Boot Record (MBR) is usually insufficient to hold all the instructions to locate and boot an operating system. Therefore, the burden of continuing the boot process is passed to a secondary-stage bootloader stored elsewhere on a disk.
Bootloader
- Many bootloader programs such as GRUB/GRUB2 and LILO can boot a BIOS-based Linux system.
- When the final-stage bootloader is ready, its configuration looks for a few specific files on the disk needed to boot an operating system.
Loading the Operating System
- The kernel and the initial RAM disk images are loaded into the memory.
- The tools and the environment included in the initial RAM file system are then used to mount the root file system of the computer.
Operating System Initialization
- Many Unix-like systems historically support a version of SysV init (and later a spiritual successor named Upstart), configurable using init scripts.
- The init system manages the initialization of user-space services, mounting other file systems, setting up networking, and more.
- SysV init also had a concept of run levels that were presets of configurations for different cases (e.g., single-user mode, multi-user mode, multi-user mode with networking, multi-user mode with networking and GUI, etc.).
- Finally, the user sees a login prompt when the login service is ready.

A setup like this continues to work, still supported by many current Linux distros and computer hardware. However, as expected from an area of technology as active as general-purpose computing, there are multiple new alternatives with broader features.

One such alternative is UEFI. It's a more modern alternative to BIOS developed by an industry consortium. UEFI is increasingly becoming the preferred firmware mode, even when BIOS is available as a fallback.

On the Linux side, systemd has been mass adopted as the default init system by most Linux distributions roughly over the past decade.

Boot Process in the Present (UEFI, systemd)

Firmware Initialization (UEFI)
- Most newer computers support a type of firmware known as UEFI.
  - UEFI stands for Unified Extensible Firmware Interface.
- Instead of a POST stage, UEFI has what is known as the "SEC" (Security) phase and the "PEI" (Pre-EFI Initialization) phase. Once these two stages are completed, the boot process is handed over to the DXE (Driver eXecution Environment) phase and BDS (Boot Device Selection) phase, which makes sure the system's hardware is in a functional state and attempts to locate and boot from the boot device.
- UEFI looks for a valid EFI system partition based on the boot order to boot from.
Reading the Boot Device
- UEFI-based systems usually support a disk partitioning scheme known as GUID Partition Table (GPT). However, they also support MBR partitioned disks via the Compatibility Support Module (CSM).
- The EFI System Partition (ESP) is a small FAT32 partition with a specific directory structure that contains executable EFI binaries (e.g., bootloader), boot configuration, UEFI drivers, and kernel images. The EFI partition can be shared by multiple operating systems when dual-booting.
- The bootloader for a UEFI system is usually an executable EFI program signed by the respective OS vendor. This is where Linux diverges from the other OSs.
  - Unlike Windows or macOS, Linux doesn't have a single vendor. This means Linux distributions may have different signing processes for the components used during the boot. To overcome this ideological difference, the Linux community came up with what's known as a 'shim'.
  - The shim is a pre-compiled, pre-signed EFI program that Linux distributions can use to handle the Secure Boot process. Instead of loading the operating system, the shim loads another bootloader (e.g., GRUB2) preferred by the distro.
  - Instead of relying on the UEFI firmware to validate GRUB2's signature, the shim verifies it using the trusted Machine Owner Keys (MOKs) generated during the installation. Then, the shim executes the signed bootloader (e.g., GRUB2) executable.
Bootloader
- GRUB2 bootloader then loads the Linux kernel.
- As with the bootloader, the kernel (and its modules) must also be signed for the Secure Boot to work. The shim can verify the kernel's signature before allowing it to execute, using its list of Machine Owner Keys (MOKs).
Loading the Operating System
- The kernel and the initial RAM disk images are loaded into the memory.
- The tools and the environment included in the initial RAM file system are then used to mount the root file system of the computer.
Operating System Initialization
- The first userspace¹ program (i.e., systemd) is started once the root file system is mounted.
- systemd service takes over, initializing userspace services and eventually leading to a login prompt or graphical interface.
- Finally, the user sees a login prompt when the login service is ready.

While this setup is an improvement over the past scenario, there's still room for improvement. For example, most Linux distros have to rely on the shim for Secure Boot, which then has to cryptographically verify several things using keys generated locally on each computer.

Now that we are caught up to the current status of the Linux process, I want to draw you a picture of a conceptual future boot process. However, we need to understand a few additional concepts to get there.

Unified Kernel Image (UKI)

In the previous scenarios, we discussed separate kernel and initramfs images. At the core of our future scenario is what is known as a Unified Kernel Image (UKI). A UKI is a Linux kernel image, an initramfs, and a few other optional components rolled into a single UEFI program that can be loaded by the UEFI firmware or a bootloader. Since a UKI contains everything needed to boot an OS, our boot sequence and the amount of cryptographic verification we need to do become simpler.

Trusted Platform Module (TPM) and Platform Configuration Registers (PCRs)

Another feature in most modern computers is the Trusted Platform Module (TPM) chip. It can either be a discrete or an integrated hardware component that's likely already included in your computer. The TPM provides a way to generate cryptographic keys, securely store secrets, and validate hardware and software states.

The Platform Configuration Registers (PCRs) are a set of registers within the TPM chip. They can be used to store measurements (e.g., hash values) of software and hardware states, which can then be used to verify the system integrity. There are usually at least 24 PCRs owned by the firmware (i.e., UEFI). These PCRs remain sealed and can be unsealed under specific conditions during specific stages of the boot process to ensure a Secure Boot.

Discoverable Partitions

If we start to go through our generic boot sequence, you might notice we still need to read the disks to locate our UKI. The Discoverable Partitions Specification (DPS)² by the UAPI Group provides a mechanism to use the Universally Unique Identifiers (UUIDs) of a GPT-partitioned disk (or a disk image) to enable automatic discovery of the disk partitions and their intended mount points.

In addition, a block-level verification mechanism such as dm-verity can be used to verify the integrity of the disk partitions/images during the boot process.

A Potential Future Boot Process

Firmware Initialization (UEFI)
Bootloading
- The disk partitions are automatically discovered based on the GPT partition UUIDs.
- The EFI System Partition (ESP) is read, and the UKI is started to be loaded.
- The UKI signed by the respective Linux distro is verified and loaded.
- The root file system is discovered, integrity verified, and mounted.
Operating System Initialization
- The first userspace program (i.e., systemd) is started and takes over, initializing userspace services and eventually leading to a login prompt or graphical interface.
- Finally, the user sees a login prompt when the login service is ready.

I'm sure you'd agree this new scenario seems like an improvement over what we've seen earlier. What we briefly talked about is mostly a hypothetical scenario. However, some distros such as GNOME OS and carbonOS have already started to implement parts of the specifications necessary to realize it.

This scenario and much more beyond is detailed in the blog post "Brave New Trusted Boot World" by Lennart Poettering. Subsequently, the points in his blog post have become part of the collaboration scope of the UAPI Group.

Initially, I wanted to include more details into our future scenario. For example, I wanted to discuss how the System Extensions (UAPI Group) specification enables powerful constructs for building Linux systems. Ultimately, I decided against further delaying an already delayed post. But I'm sure we'll get into a detailed post about System and Configuration Extension somewhere down the line.

I still haven't decided what the next post will be about. Since I'm currently thinking about two topics, I think it might be one of them. These are:

generating Unified Kernel Images (UKIs)
alternative Privilege Escalation mechanisms (e.g., sudo)

I won't go as far as to tempt the universe by indicating a timeline. But it shouldn't be too long. In the meantime, please message on fediverse or by email while I wonder if I should enable comments.

For safety and efficiency, operating systems are designed to run user programs in a separate area (user-space) so they can't directly interfere with or crash the core system functions (system-space). ↩
The Discoverable Partitions Specification (DPS) - UAPI Group Specifications ↩

My Vim Story

Gaveen Prabhasara — Mon, 09 Mar 2020 18:06:20 +0000

Once you have used 'notepad.exe', you have used all text editors—they said. A text editor is a text editor is a text editor—they said. After the first few years of running Linux/Unix professionally, I kind of subconsciously agreed with this idea, eventhough I knew Vim—I thought I knew Vim. But when other people who actually knew how to use Vim used it, Vim still looked like magic.

If you have worked in a command line at some point, I am sure you can appreciate the wisdom of learning how to use a text editing program properly. On Unix/Linux command lines, text is king. Therefore, learning how to manipulate text effectively goes a long way for your productivity. If you know how to open a file and edit its content, that enables you to configure your system too.

The text editing program I prefer for these kind of work is Vim. It is also the text editor I can pretty much expect to find in any environment I log into—laptops and servers alike. Vim is available on multiple operating systems including Linux—which is my OS of choice.

There are other text editors I occasionally use such as "gedit" and "Apostrophe". But, most of the time, when I am typing something, writing code, or editing a configuration file, I am using vim.

Like many before me, I started my text editor journey by running vim, promptly feeling lost, then failing to exit vim, and finally resigning to reboot the computer instead. Thus, I initially became an emacs user—primarily because nano was not well-known at that point. Eventually, I learned the bare minimum of vim use, just enough to do basic text editing.

What changed my mind about Vim was actually investing some time to learn it. Vim has a notorious learning curve. After years of already working on Linux, I only spent a decent amount of time to learn Vim and customize it to my liking somewhere around 2009. I can say without a doubt that I ended up liking Vim a lot—it definitely helped my productivity. It is powerful and versatile—and more importantly, did I mention I like it?

Back in the day, my vim setup was a jumble of config files and a bunch of plugins dropped into a directory until it croaked reasonably the way I wanted. It was done out of necessity because vim did not have decent package management for plugins at that point. Later when Pathogen came along, I jumped at the opportunity to organize vimfiles better. While Pathogen made it easier to organize vimfiles and manage plugins, it did not have a inbuilt way of tracking changes from multiple plugin sources over time.

At this point I was already using git and I was familiar with the relatively new git submodules feature. I felt git submodules could be used to manage individual plugins effectively. I looked around to see if anyone had already thought of that. Eventhough I am pretty sure others may have thought about it—to the best of my knowledge—no one had publicly shared a similar setup at the time. So I did it using submodules—which was a humble brag for me. These past versions of my vim configuration files are no longer there. An archived, somewhat-later version is there in a git repo though.

Since then, the most obvious change in how I maintain my vim set up had been the switch to vim-plug for plugin management. The excellent vim-plug eliminated the need to juggle git submodules and made the whole process quite painless.

The sections below will describe the vim configuration I am using. I will try to keep this post reasonably updated in future as well, in the hope it might be useful to someone—just like it helped me to read what others were doing.

Few things to note:

These should work with both regular vim or neovim.
When describing below, I have grouped some relevant sections together for convenience. They may be spread around in the actual .vimrc file.
Latest version of my vim configuration can be found in the .vimrc of the git repo, not this post.

A Few Useful Vim Concepts

Before we begin, I would also like to quickly introduce you to a couple of key vim concepts I wish someone had explained to me much earlier.

Vim has multiple modes. Vim is what you call a 'modal' editor, which simply means vim has a few modes to work with.

As an analogy, think of a smartphone these days. There is a limited number of ways you can interact with it physically, but you have seemingly boundless amount of things you can do with it. For example, simple tap on the screen behaves differently based on which mode you are in. The singular touch input in this example can mean different interactions such as waking up the screen, unlocking, launching camera app, repositioning focal point, focusing on something, taking a photo, sharing a photo with someone, liking someone's photo in another app, etc. Based on the context (or mode, if you will), the same physical thing you do, can give different results.

Vim is somewhat like that. There is an Insert Mode where you can input text (i.g., typing) into the editor. There is a Visual Mode where you can select a section of text so that you visually track which selection of text you are taking actions on (e.g., copy selected text). Then there is the Normal Mode where you do everything else, such as moving around and manipulating text (e.g., cut, copy, paste, delete, etc.).

You can learn more about these modes and how you can enter/exit them, you can check the Vim Wikibook.

Editing with Vim is like programming your text. Which is a roundabout way of saying that Vim has shortcut keys/key sequences. You can ask Vim to do simple actions or combine multiple instructions to do more complex actions, which can be intuitive once you learn the basics.

Since you are limited to interacting with a software using inputs devices in your computer, there are bound to be limits to what you can input. Vim works around this by having shortcut keys or key sequences. Notice how I said key 'sequences' not 'combinations'? Vim usually have shortcuts defined in such a way that you can press keys followed by other keys rather than press all of at once.

For example, pressing the letter d within the normal mode tells vim to 'delete' the next character. If you want to delete the next 5 character, the instruction becomes 5d. While dw deletes the next word, and 3dw deletes the next 3 words; dd deletes the entire current line of text. 7dd deletes 7 lines starting with the current. You get the idea.

If we take this a little further, you can say things like "delete text up to the next occurrence of character , by pressing dt, in the normal mode of course. These actions can be repeated by pressing ., again while in normal mode. Want to change the text instead of deleting it? Replace the 'd' with a 'c' (e.g., ct,). Want to just select the text visually instead of both delete or change? Use 'v' instead of 'd' or 'c' (e.g., vt,). You can also throw directionality into the mix by using h, j, k, l as arrow key alternatives.

The good thing is, you do not have to remember everything. You can build up the set of instructions with basic building blocks, entering one key press at a time. Sort of like conducting an orchestra, or like programming your text editing. Pretty neat, right?

You can define your own shortcuts. Vim allows you to define your own shortcuts in a couple of different ways, some of which you will see below.

The key is that Vim allows you to prefix your keyboard shortcut sequences with a special key (called leader key), so that your own shortcuts will not conflict with inbuilt vim shortcuts.

Personally, I prefer to set shortcuts in what I think as "the vim way". To explain it a bit further, I prefer to set shortcuts that do not break default vim features, at least logically adjacent to their inbuilt counterparts if there are any, and do not hamper muscle memory in the long run. You can see for yourself if the configuration below adhere to this within reason.

Vim buffers are different editing spaces, whereas vim tab pages are not. 'Vim buffers' are not to be confused with 'vim tab pages' or 'vim windows'.

A buffer in Vim is an editing space in memory. You can have a file already opened there, or you can save to a file later, or you can discard it. You can have multiple buffers open in a Vim instance.

A window in Vim is a view into a buffer. You can have a single window with a single buffer, or you can have multiple views (e.g., as horizontal/vertical screen splits, as standalone windows) into the same buffer. For example, you can split your screen area into two windows and view two different locations of a file you have opened. Just remember, the "window" mentioned here is a vim window (i.e., usually visible as a full-screen or a split-screen area)—not necessarily a graphical window like you are used to.

A tab page in Vim is a way to organized vim windows. Which means each tab page can have multiple vim windows (e.g., split windows). Therefore, you can have the same buffer viewed in more than one tab page too.

I know this can be confusing. You can see if the the following summary helps. Vim help summarizes this as:

A buffer is the in-memory text of a file.

A window is a viewport on a buffer.

A tab page is a collection of windows.

Trying to use tab pages as different file editing spaces is a common mistake that can lead to confusing errors. If you are not sure about what to use, try buffers first since they are the primitive you need as editing spaces and you view them anyway in 'windows'. If you think you need 'tab pages', you can introduce them into your workflow later.

Vim is extensible with plugins. You can add features/functions not inbuilt by using plugins, which are programs written in vimscript or other programming languages. You will see below which plugins I am using.

To summarize these key concepts:

Vim has multiple modes
Editing with Vim is like programming your text
You can define your own shortcuts
Vim buffers are different editing spaces, whereas vim tab pages are not
Vim is extensible with plugins

Explaining My Configuration

These are built around the default vim installation environment found in the Fedora-family of Linux distributions (i.e., including Fedora*, CentOS, and Red Hat Enterprise Linux) because those are my usual work environments.

My .vimrc starts with some basic settings.

With the comments accompanying them, these should be pretty self-explanatory. Most of these so some basic convenience-oriented settings or improve visual cues.

I should however mention the set autowrite. According to the vim documentation, autowrite can save the file on certain actions, even if you had not explicitly asked to save (i.e., "write" in vim terminology).

I like to avoid surprises like this as much as possible. However, autowrite is one of these very few exceptions I have left because it makes up in convenience. For example, it avoids the annoying 'unsaved content warning' when switching between buffers. If you are not sure about this, you should probably avoid setting (i.e., enabling) it.

Next up is something that actually has an effect on the text you will be editing—how the tab character/key is handled.

I know how polarizing "Tabs vs Spaces" debate can be. Therefore, I will not get into it. I used to be in 'tabs' camp in the very beginning, but I switched to using spaces instead of tabs where I can, mainly for the purpose of getting a reasonably consistent visual experience across different environments.

I prefer to have 4 spaces per tab because most of the languages I work with these days (e.g., Rust) play nicely with that. The smarttab and bs settings help with backspace among other things when you use spaces instead of tabs. If you do not set these, using backspace to delete all the expanded tab characters would become tedious.

I also have set automatically enter spaces when I press the tab key with set expandtab. This is another potential pitfall you should note. If you do not want to expand into spaces, then avoid setting expandtab.

Then there are some tools (e.g., Python and Go) which do not always handle spaces well when they expect tabs. For those you need to set exceptions. I have set a few exceptions for tab handling later in the configuration.

The comments here are pretty self explanatory. Something to remember is if you set ignorecase, case sensitivity is ignores across everywhere patters are used. This can lead to tricky situations when you use pattern search and replace type of actions. Therefore, I have left it disabled.

Most of the settings here are on non-printable characters. Therefore, they do not have an effect on the text you are editing, only acts as visual cues while you are in the editor.

The exception here is autoindent, which actually indents text automatically based on the indentation of the previous line. You can disable this by explicitly using noautoindent (i.e., disable temporarily by entering :set noautoindent in normal mode, or adding set noautoindent to .vimrc).

You do not need to use a mouse with vim—in fact, it is discouraged. But if you want to do useful things like scrolling at your speed while doing a visual inspection, and then click to go to somewhere, you can. While I love using vim, I also am not militant about being keyboard-only. There are times when adding a mouse into the mix—or more accurately, not rendering the mouse/touchpad useless—can be helpful when working with vim. This is why I enable mouse too.

Vim also has a GUI version, usually called gvim. It comes with an interface similar usual GUI programs, complete with things like toolbars, menubars, scrollbars, etc.

When I use gvim, I prefer to have a clean and simple windows, somewhat emulating what the terminal version looks like. If you prefer the GUI elements you can skip these settings.

Again, these settings a explained in the comments.

Finding the name used by your preferred font and setting it permanently can be a little tricky. One easy way to do that is to:

First, in normal mode of gvim, give :set guifont=* and hit enter. It will open the font selection dialog of your environment. The font you set this way is only temporary (i.e., until you close vim).
Once you make the preferred selection of font and size, in normal mode, give :set guifont and hit enter. It will show the used name of the font. You can use the name later in your .vimrc to set the font permanently. Note that you could use the escape character \ before special characters such as spaces in the font name.

In the example above, I have added "Inconsolata-g" font in "Medium" typeface at the size "11" points. If you want to set any other monospace font like Fira Mono, Source Code Pro, DejaVu Mono, Hack, etc., you can do so with guifont. If you use vim in a terminal instead of gvim, then the font is inherited through terminal settings.

Next comes a very important setting, eventhough it is just one line.

As explained earlier, Vim lets you define your own shortcuts without risking conflicts with defaults, using a value called a "leader"—referred in .vimrc as <leader>. By default, the leader key is set to "\". Due to the ease of reach, I prefer to set the leader key to be ",". Therefore, when you see something like <leader>y it translated to ,y in my setup.

Would this not conflict with inserting a comma?—you might wonder. There will be no conflict because you type in (i.e., insert) commas of your text in insert mode while you invoke shortcuts in normal mode.

Next is another interesting part—plugin management. Vim also has an inbuilt plugin manager since version 8. However, I have not looked enough into it. Since my current setup works really well, I did not have a need to change it either.

I use vim-plug developed by Junegunn Choi, as the plugin manager. It is written in vim script, thus do not need external runtimes or complex manual installation steps. With 'vim-plug' installing and keeping plugins up-to-date is a breeze.

Plugins are defined between the 'vim-plug' invocations of call plug#begin() and call plug#end(). I pass the parameter '~/.vim/bundle' to call plug#begin() so that it know where to install my plugins. In this case, the parameter is a directory called bundle inside the main vim files directory ~/.vim. I have it there because it is a leftover from the days when I used a different plugin manager.

I am not going to go one by one in this plugin list. However, as you can see above, I have grouped my plugins into a few areas such as:

"Augment Vim behavior"
"Augment UI elements"
"Augment filetype handling and syntax highlighting"
"Add utility"
"Add colorschemes"

Since all of my plugins are added via GitHub, you can find out more by visiting the relevant repository pages. To get the URL of each plugin repository page, append the name in the plugin definition after "https://github.com/". For example, "easymotion/vim-easymotion" repository can be found at "https://github.com/easymotion/vim-easymotion".

Some of the plugins here either need to be configured or can be configured. Some of the features enabled by these plugins, along with the shortcuts they introduce, are documented in the README.md file of the git repo.

I am again not going to explain the above any further because they are pretty self-explanatory with the help of included comments.

One thing to note is the section where I have included tab behaviour exceptions (e.g., for Ruby and Go) as I mentioned earlier. Some other exceptions are handled automatically by file type plugins.

If you look closely as the sections of the same .vimrc file I have been explaining so far, you will notice it has a syntax. As I mentioned earlier, Vim has an inbuilt scripting language. You can use this language to write custom functions which can make things easier for you by scripting them.

In the above example, there is a function with the name of "DiffWithSaved()" which combines inbuilt features of vim to find if there has been an changes since you last saved. If there are, it will present the differences between the last saved state and current state in a view similar to the Unix diff. Finally, it can be invoked with a command named "DiffSaved".

I have defined a few more functions as well. Please note that ToggleFullScreen() function has an external dependency, an external program named wmctrl.

While these functions define what to do, they are not automatically invoked. The way I have chosen to call them is via either shortcut keys or function keys.

You can see that in the above, DiffSaved and QuickfixToggle() are configured to be invokes with <leader>? and <leader>q respectively. As per my configuration, this translated into pressing ,? and ,q respectively.

I have mapped some useful shortcuts to deal with cut/copy/paste with clipboard in gvim. These complement the inbuilt x, y, p for cut/copy/paste respectively, but for external clipboard (e.g., OS clipboard). I also have <leader>v (i.e., ,v) to select the text that was pasted into vim, right after it was pasted.

In addition to custom shortcut keys, I have also mapped some of the function keys (e.g., F1 - F12 keys in the keyboard) to do useful things.

You might notice that the convention I use is: function key mappings only change editor environment, whereas anything that affect the actual text are mapped as shortcuts with a "leader" prefix.

In the above, I have essentially remapped F1 key as another ESC key to avoid potential accidental F1 key presses when reaching ESC. F4 toggles spellchecking, F7 uses junegunn/limelight.vim plugin, and F11 calls the above mentioned ToggleFullScreen() function. The rest should be self-explanatory.

I also have a shortcut for when you edit a file away, but forgot you needed sudo privileges to save it. Instead of pulling your hair out and starting from scratch, there are things you can do. The following makes one of those things even more convenient to use (i.e., instead of using :w to save, type :w!! to save with 'sudo' privileges).

I mentioned earlier that buffers are the way to go when you need to keep multiple files open. The following section of the settings map a few shortcut keys to make switching between buffers faster.

In addition, I have also configured the plugin buftabs here. It shows which files you have opened at the bottom of the vim window. You can not point and click it to switch between buffers, but rather provides a minimalist indication of opened buffers and statuses.

Finally, we have the exciting part—colorscheme! The first two lines termguicolors and background set two values explicitly rather than try to derive them from the environment.

My termguicolors setting explicitly says that the terminal you are using supports GUI colors. It does in my case because most of the time I use GNOME Terminal which comes with Fedora. If your terminal does not support GUI colors, then do not set this.

My background setting is to make my terminal background dark. Like many people who need to stare at a screen for a long time, I too prefer dark themes whenever I have the option. Some colorschemes select between dark and light versions based on this value, which is why I have made 'dark' explicit. If you prefer light, you can set it to 'light' too.

If you do not need either of these two, then you should not set them because some colorschemes would look weird if you set the values different from what they are expecting or capable of handling.

With those out of the way, I finally set a colorscheme. My colorscheme of choice have changed overtime. These days, I prefer a theme called "One" which has both a light and dark version (picked based one the background setting).

The end result should look something like this (screenshot on Imgur).

That is about it for this post. Since it is already long, I did not try to include further details such as what the plugins do or which shortcuts they introduce. If you would like to hear about these details—or just wanted to give me your comments—please leave a comment, or let me know via email or @s/DMs.

Originally posted at: https://gaveen.me/2020/02/my-vim-story/

My updated vimfiles can be found at:

gaveen / vimfiles

My Vimfiles

Vimfiles

This is my current vimfiles. Plugins are managed with vim-plug. These should work with vim/gvim and neovim.

This is not intended as a distribution, as I may change it depending on the work I'm doing. Plus, I pretty much only use this under Linux. But, you are more than welcome to use my vim files to explore. I have tried to keep my .vimrc file well commended for your convenience.

How to use

If you still want to use these vimfiles, this is how you can do so.

First, you'll need to clone this repository.
$ git clone https://github.com/gaveen/vimfiles.git

Then, you could create symbolic links to the .vim directory and .vimrc file. Assuming the location where you cloned the repository is ~/Apps/vimfiles, you can do something like:
$ ln -s ~/Apps/vimfiles/.vim ~/.vim
$ ln -s ~/Apps/vimfiles/.vimrc ~/.vimrc

Finally, launch vim and install the plugins. (Note:…

View on GitHub

I don't believe in Kubernetes

Gaveen Prabhasara — Tue, 03 Dec 2019 00:00:12 +0000

I have always been a believer in Cloud Computing. Even when big executives of the world were calling it a fad, I believed in the cloud—or its promise. It was not so much an achievement to be had—rather as with to so many other people in tech—it spoke to me from a place of understanding. We may not have called it "cloud", but when Amazon opened the gates of their Web Services to everyone, we saw good things we recognized with and we realized it was an opportunity to build things better.

Cloud Computing was about getting good IT infrastructure accessible to the rest of us. Finally, we did not need to be a sizeable enterprise or have a small army of ops and devs just to have the necessary infrastructure in place. It was empowering, it was enabling. When cloud users could solve their problems conveniently, people interested in cloud infrastructure could enjoy building them—and in the process, change many professions and how the industry worked.

After achieving the cloud, our mission should have been to make cloud computing accessible to more and more people. We dreamt of people having on-demand access to all the IT resources they want at an ever reducing cost, being able to use technology easily and conveniently to solve problems they needed solving. That, should have been our mission.

Instead, here we are today left fixated on things like Kubernetes¹ and trying to emulate how giant companies—so called hyperscalers—like Google and Facebook operate their infrastructure. Instead of trying to get people closer to solving their problems, we are effectively erecting newer barriers in complexity.

To be clear, I am not trying to diss Kubernetes. I like Kubernetes—with certain caveats. It is a system with many components and—as with any complex system—comes with inherent operational complexities. It is an interesting system catering specific needs. It has its place. More accurately, Kubernetes is a tool for cloud builders, not cloud consumers.

A problem with the current Cloud Native narrative is that a lot of companies (and in turn, a significant amount of eco system resources), who would otherwise be working with a consumer focus are busy with building tools for cloud builders—and then try to wrestle these into submission as cloud consumer technologies. You do not have to take my word for it—you can try deploying your next application to production on Kubernetes.

What happened to bringing the best technology conveniences to the rest of us? Whatever happened to harnessing the power of IT so that it stops being an inhibitor and becomes the great enabling force that it can be? What happened to—may Cthulhu have mercy on my soul—powering "digital transformation"?

When the most activity in tech is taking place on a tangent, it is refreshing to see ventures like Oxide Computer Company daring to pause and ask fundamental questions again. In their case—at least from what I can gather at this point—they are tying to bring the advantages of hyperscale IT infrastructure because the hardware industry seem to have forgotten the rest of us in favor of the hyperscalers.

As for me, I want modern, secure, cost-effective computing to be delivered to more and more people because our industry seem to have forgotten that along with the fixation on hyperscale. I do of course believe in the existence of Kubernetes. I just do not believe it is the thing that carries the mission forward—at least not in the current form. This definitely is something I would like to work on. It had always been my interest. At different times, in different places, in different forms—it had always been the mission.

PSA: Containers ≠ Docker ≠ Kubernetes ↩

The future of networking is software

Gaveen Prabhasara — Wed, 02 Oct 2019 17:53:03 +0000

Edit: The original post maybe edited since publishing here. Please check the original post if you're reading this much later after the published date.

I believe the future of networking is software. To be a little more specific on my thoughts, I believe the future of networking in the data center is software-based. Water is wet, and on that note, thank you for coming to my TED talk. And, quick PS: clouds are also hosted in data centers.

While what I just mentioned maybe obvious to say—given how pretty much everything has its future in software—you would be surprised at the state of the networking industry if you came to it as an outsider. The cloud has already happened and compute has embraced it with both arms,... and legs, and tentacles, and whatever else you have available. Storage is also not too far behind. Cloud-scale storage technologies are becoming closer and closer to being commoditized¹. Network—in a way—is the last frontier in the data center not yet fully subscribing to cloud ethos.

Things are certainly changing—more on that later. But, networking vendors are still trying to sell you pricey switches, they are only just now getting serious about DevOps, and their idea of a software-defined network (SDN) is to run a bunch of software on top of their existing traditional network solution stack (i.e., ASIC-driven expensive boxes + mostly-blackbox NOS/firmware + ridiculous licensing + exorbitant professional services + consultancy that makes sure you stay with them).

Forgive me for sounding cynical, I am not. I think things should be better though. To that end, let me share a few things and sort-of-predictions I believe of the future, at least at this point in time. However—fair disclaimer—I may have a vested interest in this game. Therefore, take whatever beyond this point as however you will.

I believe the future of networking—at least, networking in the data center—is in software. Networking in general is gradually heading that way. Telco industry has learned from what happened with the "cloud fad", and are trying to move towards NFV², SDN³, and SD-WAN⁴ already. But let me focus on the data center (and in turn, cloud), which has been my work domain for a while.

I believe that the enablers of this shift will be multi-fold. For example network virtualization⁵ and network functions virtualization² are precursors, and they have already been here for a while. Also, network equipment supporting alternative NOSs have been here for a while. For example, some brand vendors like Mellanox and Dell EMC—as well as whitebox switch vendors—support multiple OSs such as generic Linux, Cumulus, Open Network Linux, SONiC, along their own OSs on their switches.

I believe the next wave of convergence will incorporate networking into already converged compute and storage. For the lack of a better term—and in line with the buzzword-based naming (e.g., CI⁶/HCI⁷)—let us call it Fully-Converged Infrastructure (FCI). If and when this becomes a reality, it is likely that it will co-exist with existing types of infrastructure—similar to how HCI/CI still remain somewhat of niche markets. However, if this comes to pass with significant enough adoption, it will be a salient indicator that the balance has been tipped from hardware to software.

In any case, I believe there will be a significant shift in direction from specialized hardware-based switch/network appliances to software-based networking on generic computing platforms.

I am not sure how the gap will be bridged exactly. Perhaps, x86 server-based networking will become more powerful to catch up to ASIC/FPGA-accelerated networking. Perhaps, network appliance hardware will become generic open platforms. Perhaps, ASICs and FPGAs will become increasingly generic till they become commodity components in generic computing. Perhaps, it will be somewhere in between.

We already see some precursors in the evolution of the hardware aspect. For example, x86-based switches are now commonplace, while some ASICs are becoming more and more in the territory of generic platforms⁸.

Software in the networking industry on the other hand, had been a little more nimble. While proprietary software stacks of traditional vendors have been evolving further, there also has been an interest to consolidate effort on the common components. For example, Linux Foundation Networking and Open Networking Foundation has received contribution from even the bigger network vendors such as Cisco, Arista, Juniper, and Mellanox. This has resulted in the development or improvement of interesting open source technologies such as DPDK, P4, VPP, SONiC, Stratum, etc. Open source-based as well as proprietary vendor OSs⁹ also have been available in the market.

The physical networking (e.g., fiber/copper infrastructure, optical transceivers, cables, connectors, etc.) components are likely to remain and continue to evolve in traditional / proprietary supply chains as they involve physical fabrication, manufacturing, and other logistics.

But the non-physical side of networking space (e.g., which contains processing, forwarding, routing/switching decisions, protocols, virtual endpoints, segmentation / micro-segmentation, policy frameworks, telemetry / observability, etc.) will get much more interesting. I am not saying traditional networking equipment will become obsolete in a hurry. But I believe building networks with generic computing platforms that are at least akin to x86 servers of today—perhaps augmented with FPGAs or ASICs—will at least become a mainstream option to ops teams. These networking infastructure will largely be handled by the software they are running.

With cloud computing¹⁰ becoming a mainstay—along with all the shifts in thinking it brings—networking is due for a re-imagining. While the body of domain knowledge, standards, abstractions, and even expertise can be reused, I believe that the traditional thinking such as the ones that involve believing in different switch boxes for different places¹¹ in the network is—IMHO—antiquated and only serve to drive network vendor sales.

Application-aware Layer 4 - Layer 7 software systems are already starting to do pretty-cool things¹². However, Layer 2 - Layer 3 networking that enables them—as well can further empower them—are yet to catch up to the future that is already dawning.

And, I believe it is going to be awesome to build.

e.g., Ceph, MinIO, etc. ↩
Network Functions Virtualization (e.g., OPNFV) ↩
Software-Defined Networking ↩
SD-WAN ↩
Network Virtualization (e.g., VMware NSX) ↩
Converged Infrastructure ↩
Hyperconverged Infrastructure ↩
Competing vendors like Arista and Cisco use the same Broadcom ASIC silicon series (e.g., Tomahawk, Trident, Jericho) in some of their comparable switches. ↩
e.g., Cumulus Networks, Big Switch Networks, and Pluribus Networks ↩
Cloud Computing and its inner aspects such as public/hybrid clouds, Cloud Native Infrastructure, etc. has become a mainstay in the industry, and continue to influence the thinking in adjacent areas of technology. ↩
Either talking in terms of older core/aggregation/access switch classification or the newer spine/leaf classification, is still hinged on the fact that these switches are usually built for different purposes with different limitations and capabilities. However, if everything is software, running on sufficient compute power, and necessary physical connectivity, there will not be much technical justification why one of your switches cannot be a core switch while another is orders of magnitude more expensive than the previous one. ↩
Layer 4 - 7 software options include cloud-native software such as Cilium, Service Mesh software (e.g., Envoy/Istio, and Linkerd), etc. ↩