DEV Community: Anna

Building Debian packages from source in bootstrapped Debian

Anna — Thu, 17 Apr 2025 15:35:38 +0000

If you’re a curious user - especially one who’s into customizing your Debian - you’ve probably run into this situation at least once: you discover a cool software, but it’s either not available in your Debian’s package repositories, or it is there, but the version is outdated.

Newer versions often bring in completely new features that you actually want to use. When I think of Debian Stable, a few examples come to mind.

Terminal emulator Alacritty, for instance. The version in the Debian Stable repo is dated: Alacritty of this version uses a .yml config file, but newer versions have switched to .toml.

Or btop, the version in Debian Stable doesn’t support NVIDIA GPU monitoring, while the current version does.

And then there's Hyprland, a window manager that’s only available in Debian Unstable and Testing (Trixie). On Debian Stable, it’s not even in the repos at all.

This article is dedicated to a terminal-based file manager yazi that is super mega cool, but it’s not available in any of Debian’s repositories - not in Stable, not in Testing, not even in Unstable.

NB! Building applications from source isn’t necessarily hard in terms of the actual process, but making them work, and more importantly, making sure they don’t break your system, is another story.

Small utility apps that are meant to run in user space (i.e. they don’t touch anything kernel related, like modules, are generally less risky in terms of stability of your Debian afterwards). The main problem is that there’s no guarantee they’ll work properly due to dependencies. The more dependencies an app has, the higher the risk that it won’t work at all.

Also, be very careful when a piece of software wants to downgrade some packages as part of its dependency list. I strongly recommend avoiding that. Downgrading system packages can mess up your system a lot.

Before building anything from source, always evaluate potential security risks. Don’t rush into compiling some random app that has minimal activity on GitHub - especially without reviewing the source code carefully.

NB2! I’m currently using Debian Testing (aka Trixie), which is to become Debian 13, I hope soon. If you’re trying to get Yazi working on Debian Stable, I can’t guarantee it’ll work due to dependency versions.

What’s so particular about building software from source in a debootstrapped Debian, and what does that even mean?

If you’ve ever developed something in Python or used Node.js then this will probably make sense to you. You might be familiar with Python virtual environments, or how Node.js projects usually live in their own directories with all the installed libraries.

When you’re working inside a Python environment and need some libraries, you install them into that isolated environment. Move to another project? Create a new environment, install only what you need, and that’s it. No system-wide Python mess. Same story with Node.js, you usually don’t install JS libraries globally, they just live in the project folder. Clean, contained.

Now, Debian bootstrapping offers a kind of similar idea but in the context of system-level stuff.

Let’s say you want to install Yazi, which is written in Rust. You’ll need Rust installed on your system to build it. Or maybe you want to build Ly display manager, written in Zig—you’ll need to install the Zig compiler. Other programs written in C? You’ll need a C compiler and a bunch of build-time libraries.

But here’s the key point: software often has one set of requirements to build and a completely different set of requirements to run.

For example, with Yazi, you only need Rust to build it. Once that’s done, Rust isn’t required to run the app. So, what do you do? Install Rust, build the thing in 10 minutes, and then… what? Leave Rust on your system? Delete it? What if the software needed tons of other build-time tools that you’ll never use again?

That’s where debootstrap comes in. It lets you create a sort of mini Debian environment:a clean, isolated system within your system. It’s not a full second Debian install, but it is like a mini mirror of your Debian system where you can install/build/test whatever you want without messing up your main OS. It’s like a virtual "project" space for your system-level experiments.

debootstrap is a tool which will install a Debian base system into a subdirectory of another, already installed system. It doesn't require an installation CD, just access to a Debian repository. It can also be installed and run from another operating system, so, for instance, you can use debootstrap to install Debian onto an unused partition from a running Gentoo system. It can also be used to create a rootfs for a machine of a different architecture, which is known as "cross-debootstrapping". (Debian Wiki: Debootstrap)

First, you need to install debootstrap package:

$ sudo apt install debootstrap

Then, I think it is convenient to elevate your user and execute commands as root user ($ changes for # in code snippets, meaning that these commands are run by root user)

First, I create a directory where my bootstrapped Debian will reside:

# mkdir /trixie-chroot
## If you use Debian Stable:
# mkdir /stable-chroot

## And then, I "deboostrap" Debian into this directory
## NB! I use Trixie, and I Debootstrap Trixie!!!

# debootstrap trixie /trixie-chroot http://deb.debian.org/debian/
## If you use Debian Stable:
# debootstrap stable /stable-chroot http://deb.debian.org/debian/

As I mentioned, bootstrapped Debian is not a fully functional system and cannot function outside of the "main" Debian on which it resides. That is due to the fact that it is exactly your main OS sharing with bootstrapped Debian some its components in order to enable a simulation of a complete runtime environment inside the bootstrapped Debian, allowing it to behave like a real system.

This is achieved by mounting pseudo filesystems to bootstrapped Debian, or more correctly, chroot.

chroot on Unix-like operating systems is an operation that changes the apparent root directory for the current running process and its children. (Debain Wiki: chroot)

It may sounds complicated and scary, but actually it is just a couple of commands mount.

First thing to do is to mount the proc filesystem inside the chroot (/trixie-chroot in my case). The proc filesystem provides access to kernel and process information. Many system tools (like ps, top, etc.) rely on /proc to function correctly. Without it tools inside the chroot won’t be able to access process info or kernel parameters.

Second, is to mount the sysfs filesystem to /trixie-chroot. sysfs exposes kernel devices and attributes. It's required by many parts of the system like udev, systemd, etc. Without it hardware-related commands (or anything interacting with kernel devices) may not work inside the chroot.

Third, you need to provide information to chroot Debian about DNS nameservers, especially if you have some custom configuration. Otherwise you will not be able reach internet from chroot and install packages.

All these complex sounding things can be done with 3 commands:

# mount proc /trixie-chroot/proc -t proc
# mount sysfs /trixie-chroot/sys -t sysfs
# cp /etc/hosts /trixie-chroot/etc/hosts

Now, you can "login" into bootstrapped Debian!

# chroot /trixie-chroot /bin/bash

Now, we can start building Yazi from source! I’ll be following the official installation guide...
...with one interesting modification you might find useful for other Rust-native apps.

But let’s get started.

First, it's important to note that Yazi has various dependencies: because as a terminal file manager, it can do things like preview images and different file types right inside the terminal! And of course, this kind of functionality requires some additional software to be installed.

For me, the dependencies of Yazi looked pretty familiar. You can check them one by one if you’re unsure.
So why am I installing them into a Debian chroot if I don't actually plan to use it?
Well, I'm not that advanced when it comes to Rust apps. I’ve got more experience building from source using make, and if you’ve ever done that, you probably know about the ./configure step—it scans your system and sets the right parameters for the build.

What often happens is, if something’s missing on your system, that is not required by any core feature of to be built software, these features are just get skipped during build processes, because requirements for them are not satisfied. I’m not sure if Rust works exactly the same way, but either way, it’s not a big deal: I’m planning to destroy this chroot after installation anyway.

chroot # apt update
chroot # apt install ffmpeg 7zip jq poppler-utils fd-find ripgrep fzf zoxide imagemagick

Here comes NB for Debian Stable users:

Note that these dependencies are quite old on some Debian/Ubuntu versions and may cause Yazi to malfunction. In that case, you will need to manually build them from the latest source. (Yazi Installation guide)

As the next step, I need to install Rust, of course:

chroot # apt install curl
chroot # curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
chroot # rustup update

And after this... I diverge from the official installation guide. Actually, this project is awesome, and installing it is super easy—to the point that it will literally leave only Rust installed on your system afterwards. Here's the command from the installation guide -
cargo install --locked yazi-fm yazi-cli. As you can see, this installs two packages directly on your system.

However, in my chroot setup, this is kind of useless, because if I run that command, it’ll install those packages inside the chrooted Debian, which I obviously won’t be using.

But here’s the cool part: Rust has this amazing crate called cargo-deb.

Debian packages from Cargo projects
This is a Cargo helper command which automatically creates binary Debian packages (.deb) from Cargo projects. (Rust Docs: cargo-deb)

Now you can probably guess what my goal is : I want to get two .deb files: one for yazi-fm and one for yazi-cli. Then I can just copy them over to my main system and install them with dpkg. Simple and clean!

To do that, first I need to install cargo-deb from crates.io.

chroot # rustup update  
chroot # apt install build-essentials # gcc compiler
chroot # cargo install cargo-deb

After this, I clone the Yazi project's GitHub repo:

chroot # git clone https://github.com/sxyazi/yazi.git
chroot # cd yazi

And then ... I just instruct Rust to build from source 2 .deb files:

chroot # cargo deb -p yazi-fm --locked
chroot # cargo deb -p yazi-cli --locked

After these two process are completed 2 .deb files are placed into ./target/debian directory of yazi root directory.

What is left is to copy these two files to the main system and install 2 packages from them using dpkg.

chroot # exit
# exit 
$ cd # teleporting to home directory of my regular user
$ sudo cp /trixie-chroot/root/yazi/target/debian/yazi-cli_25.4.8-1_amd64.deb .
$ sudo cp /trixie-chroot/root/yazi/target/debian/yazi-fm_25.4.8-1_amd64.deb .

$ sudo dpkg -i yazi-fm_25.4.8-1_amd64.deb
$ sudo dpkg -i yazi-cli_25.4.8-1_amd64.deb

Voila!

Command to launch Yazi File manager: yazi
Command to launch Yazi CLI: ya

Where to Start in Web Development: Ignoring Learning HTTP(S), URLs, DNS, IP, SSL Will Have Consequences...

Anna — Fri, 21 Mar 2025 18:49:50 +0000

This is the second article in a series on Web Development. If you’re just starting out, I highly recommend reading the first part first. In that part I illustrated and explained the DOM (Document Object Model); in this article, I’ll use the term DOM assuming you’re already familiar with it.

This isn’t my first article on networking. From my profile stats, I’ve noticed that topics around networking get few readers, so I can guess that now you might be thinking, "I can skip this for now—I don’t really need it for starting with web dev". Well, here’s what I believe might happen on your web development learning path if you postpone learning the things listed in the title:

CRUD Applications, the Database part will be pretty tough.

Improper database client configuration, struggle with setting up secure connections and handling data flows properly.
HTML Forms & HTTP methods mess-ups

Sensitive data unintentionally spawning in URL because of GET method in form tag? Plus, in general, handling user inputs may be confusing.
Try/Catch Blocks in JS fetch functions

When you fetch data using JS, you’re dealing with network requests and responses. If you do not understand how those work, you will be confused by what you’re actually catching in an error.
Debugging fetching functions

Without understanding HTTP language or the request-response cycle, you will spend extra time trying to understand why your frontend does not display what it should. Let's not mention the situationships when some try to fetch something from remote servers pointing to the localhost.
Hosting your web apps (portfolio, pet projects, etc.)

Eventually, you’ll want to put your work online. Without networking fundamentals, you might not know even where to start.
Routing in your web apps

If you don’t understand how URLs translate into requests and how servers respond, you might struggle with design of your app's routing schemes.
APIs

Web development revolves around APIs. APIs are mostly about communication via network.
Dockerizing your apps

I noticed the urge to containerize anything that is not yet containerized even on early stages of development, but containerization involves such things as port mappings, virtual networking etc. You might also struggle with linking containers together or exposing your app's parts properly.

I hope I have convinced you somehow to invest your time in learning networking fundamentals.

In the previous part’s conclusion, I showed this scheme visualizing on high level how browsers work:

As you can see in that diagram, the "query" from the browser’s address bar plays a key role: by using that "query", your browser’s networking layer sends a request and receives a response from the correct server. That response contains some data (for example, .html, .css, and .js files) that gets rendered—based on these files the Document Object Model is constructed by the browser—and then visualized.

In this article, I’ll focus on what’s happening in the browser’s networking layer and also explain what servers are and what role they play (Spoiler: "server" is not always about a backend!).

Here is the roadmap:

Browser's Address Bar
URIs and URLs
HTTP Protocol
1. Internet Protocol and IP addresses
2. Transmission Control Protocol (TCP)
3. TCP/HTTP Network traffic
4. Understanding Ports
5. DNS
6. TCP in action: 3-way handshake
HTTPS
1. Little extra info: multiple IP addresses and horizontal scaling of web apps
2. About encryption
3. TLS/SSL certificates
4. Secure communication channel

1. Browser's Address Bar

Modern browsers are quite powerful, and you might not even notice that when you type something in the address bar, you’re effectively using it like a search engine (default search engine of your browser). You can drop in any text, and the browser takes that text and forwards it to your default search engine—all without you really realizing.

But if you’ve been using browsers and the internet for a long time, you might remember it wasn’t always like this. You couldn’t just throw any random text into the address bar and expect the browser to figure it out. Back then, you had to first go to the search engine’s website (e.g., Google or Bing) and then type your search query there.

For demonstration purposes, I’ll use a silly "query". My browser is Brave, and its default search engine is Brave Search. I type this into the address bar: difference between .com and .dev. This is what I see:

What if I type the same "query" without spaces, and even worse, with dots between words - difference.between.comand.dev? It can happen—maybe you’ve done it when typing quickly on your phone and missed the spaces. Such "query" results in:

Error example: DNS address could not be resolved

Jump back to DNS section

So here comes the error: the browser definitely did not process both queries the same way. The first query was treated as a string that got passed to the search engine, but the second query (with dots) made the browser do something else instead of just processing it along as a search query.

If I double-click on the address bar in both cases, here’s what I see:

https://differencebetween.com.and.dev/ — which leads to an error page
https://search.brave.com/search?q=difference+between+.com+and+.dev&source=desktop&summary=1&conversation=1dd960092fc9678fb88e64

Those are the "queries" the browser processed. I keep putting "query" in quotes (" ") because the technically correct term is a URI, and the actual "query" part is:
?q=difference+between+.com+and+.dev&source=desktop&summary=1&conversation=1dd960092fc9678fb88e64. That’s part of what I typed into the browser’s address bar. Let’s start with what a URI is.

2. URIs and URLs

Uniform Resource Identifiers (URI) are used to identify "resources" on the web. URIs are commonly used as targets of HTTP requests, in which case the URI represents a location for a physical resource, such as a document, a photo, binary data.
The most common type of URI is a Uniform Resource Locator (URL), which is known as the web address.
A Uniform Resource Name (URN) is a URI that identifies a resource by name in a particular namespace. (MDN web docs: URIs)

A URL (Uniform Resource Locator) is renown term not only by developers but also by just users. The point is that a URL ∈ URI (is a subset of), meaning a URI is broader (every URL is a URI, but not every URI is a URL). A URL is a specific type of URI that not only identifies a resource but also provides the information needed to retrieve it (such as its network location and protocol).

In the previous article, I gave examples of opening files from my PC using this URI:

file:///home/lalala/Projects/DEVTO/webdev/randomPDF.pdf

This is a URI, but not exactly a URL—it’s a bit ambiguous because it could be called a URL if you consider it a locator for a file resource. However, it is common to use “file URI” to emphasize that it accesses a local file rather than a resource over HTTP/HTTPS.

By contrast,

https://search.brave.com/search?q=difference+between+.com+and+.dev&source=desktop&summary=1&conversation=1dd960092fc9678fb88e64

is definitely both a URI and a URL.

First, I want to "decompose" one illustrative URL - http://www.example.com:80/path/to/myfile.html?key1=value1&key2=value2#SomewhereInTheDocument to demonstrate all its functional parts, which can help in understanding how it works—this is a key moment:

_1. http:// is the scheme of the URL, indicating which protocol the browser must use

www.example.com is the host name of the URI, indicating which Web server is being requested. Here, we use a domain name. It is also possible to directly use an IP address, but because it is less convenient, it is rare to do so, unless the server doesn't have a registered domain name

:80 is the port of the URL, indicating the technical "gate" used to access the resources on the web server. It is usually omitted if the web server uses the standard ports of the HTTP protocol (80 for HTTP and 443 for HTTPS) to grant access to its resources. Otherwise, it is mandatory.

/path/to/myfile.html is the path of the URL, indicating the location of the resource on the web server. In the early days of the Web, this was an actual directory path to a physical location on the web server. Nowadays, web servers usually abstract this to an arbitrary location.

?key1=value1&key2=value2 is the query of the URL, which are extra parameters provided to the web server. The parameters are a list of key/value pairs prefixed by the ? symbol, and separated with the & symbol. These can be used to provide additional context about the resource being requested._

Let's detangle it all one by one. I'm starting with the protocol.

In a URL, anything before :// is a protocol. When I was opening local files from my PC, I used the file:// protocol to access them. However, the most common and widespread protocol—and the one you'll 100% be dealing with—is HTTP(S). So it makes sense to explain it.

3. HTTP Protocol

HTTP: Hypertext Transfer Protocol (HTTP) is an application protocol that defines a language for clients and servers to speak to each other. This is like the language you use to order your goods. (MDN web docs: How the web works)

The browser acts a client, and it communicates with servers using the HTTP language (If it is indicated in the URL, i.e. when it starts with http(s)://). This language per se is a language of verbs—actions with self-explanatory names like GET, PUT, DELETE, POST, TRACE, and others. The browser sends one of these verbs to an address indicated in the URL (URL addresses I will cover later), and if that address is "valid", the server found on this address sends a response.

I want to clarify that, in this article, a server is simply a computational device—like the PC or notebook you’re using, but without a monitor or GUI. For simplicity, just think of it that way for now.

When (if) the server responds to the browser’s request, it does so using HTTP, as it was contacted via this protocol. If the request sent by browser got rejected by server for any reason, HTTP ensures that a client (browser) anyway receives a response. That’s very important (for your JS try catch blocks :D). HTTP responses are more complex than requests (verbs) — the status codes are a crucial part of any HTTP response.

_HTTP response status codes indicate whether a specific HTTP request has been successfully completed. Responses are grouped in five classes:

Informational responses (100 – 199)

Successful responses (200 – 299)

Redirection messages (300 – 399)

Client error responses (400 – 499)

Server error responses (500 – 599) (MDN web docs: URIs)_

Now, let me show you some nerd fun that will help you to understand in deep how HTTP protocol works. Of course you can learn all the status codes, methods of HTTP language, but I think it is not enough if HTTP protocol remains a block box for you, especially when it comes to the debugging of your web apps.

The difficulty in understanding networking concepts is because it's kinda all happening under the hood – you don't see any of the communication statuses or responses in the raw format, until you get an error. This is true if you're a user; but if you're a dev, you will be intact even more with networking stuff. However, even if you're a dev and were avoiding networking by all means, I'll show you what's inside this black box.

Let's start with the fact that the HTTP protocol sits on top of another protocol, TCP.

The Transmission Control Protocol (TCP) is one of the main protocols of the Internet protocol suite. It originated in the initial network implementation in which it complemented the Internet Protocol (IP). Therefore, the entire suite is commonly referred to as TCP/IP. TCP provides reliable, ordered, and error-checked delivery of a stream of octets (bytes) between applications running on hosts communicating via an IP network (Wikipedia:TCP)

Here is the chain of protocols:

HTTP Protocol <-- TCP Protocol <-- IP Protocol

Let's start from the bottom up.

3.1 Internet Protocol and IP addresses

IP (Internet Protocol) has the task of delivering packets from the source host to the destination host solely based on the IP addresses in the packet headers. For this purpose, IP defines packet structures that encapsulate the data to be delivered. It also defines addressing methods that are used to label the datagram with source and destination information (Wikipedia:IP).

IP may be quite familiar to you, especially when it comes to the term "IP address". The device from which you're reading this article has an IP address, and my PC, from which I'm writing this article, also has an IP address. DEV.TO, as a portal, has an IP address as well. Even though your device and my PC do not communicate directly, both our devices are communicating with DEV.TO.

There are actually two versions of the IP protocol: IPv4 and IPv6. As stated in the Wikipedia quote above, the IP protocol is about communication between the source and destination.

Try to think of packets of data as envelopes with some info—Internet Protocol is mostly about the structure of addresses. Each envelope with a letter (data) inside has a to (destination) and a from (source).

Now, what about IP versions? Let's say that we lived in the old days when people resided in houses rather than apartments (one house=one household). The addressing system was quite different—just the street name, house number, city, and country. That system worked fine. But then, as time went on, globalization and urbanization began, and more people started living in cities. It can't be that only one family lives in a huge house, so houses started to be divided into apartments (one house=many households). Now, if a letter arrives with just the street, city, country, and house number, it will not find the correct Destinee. A new "addressing" mechanism had to be invented.

Image source

That's exactly what happened with IPv4 (Internet Protocol version 4). It was created when the Internet was a far less "global"—few had access to it—and it worked fine. But look at us in 2025: each of us usually has at least two devices that need to be connected to the Internet, and many of us don't live alone but with family, so the numbers multiply. Here’s the thing: IPv4 addresses are mathematically limited—the number of unique addresses available is around 4 billion. If you calculate only for private use and individuals, it is misleading because think also about all big companies with many servers that use unique IP addresses if they're connected to the global Internet.

So, a newer version of IP was born—IPv6 (Internet Protocol version 6). This version of the protocol completely changes the address structure and syntax, providing an enormous pool of unique addresses so that every server, every personal device can have its own unique address.

The problem is that IPv6 has not yet been widely adopted, and most of the Internet is still on IPv4.

So you may think there's a problem: network packets (the envelopes from the analogy above) are getting lost, because the addresses are "duplicated"... but that's not true. Various mechanisms somehow solve the issue of IPv4 addresses scarcity, and the core mechanism is NAT—Network Address Translation. I won't go into detail here, but if you genuinely want to advance your understandings in networking, you can read my other articles on networking: this, and this.

In the scope of this article, I just want to explain the key point of how NAT affects your devices and any web app you will ever develop and host. Your devices don't have unique IPv4 addresses when connecting to the internet. Firstly, you never connect directly to the internet from your devices unless you have a device directly attached to a fiber cable connected in a wired way to the global net. Yes, if you think the global internet is wireless, you're mistaken. The global internet is entirely wired, except for the internet provided by satellites. So, how do you access the internet from your devices? Mostly wirelessly, I imagine—you connect to a Wi-Fi router. And if you've ever handled internet provision at your home, you might have seen the operator arrive at your house with a thin cable, your to be Wi-Fi router and connecting it with a wire! Another alternative is using a SIM card in your router or mobile phone, which receives signals "wirelessly".

While cellular towers broadcast wireless signals to phones, themselves they rely on a wired or high-capacity link to connect to the rest of the carrier’s network. In the case of Wi-Fi routers, fiber cable is physically brought to your internet provider's establishment, which is connected by cable—first to the city, then to the country, and then to the rest of the world.

Anyway, that was just FYI. What's important for web development is this: at your home setup your Wi-Fi router has a unique IPv4 address*. (* stands for the fact that first, most probably, this unique ipv4 address is not as long lasting as the classical physical address of a building - it is constantly changing - usually much more than once a day; second, there is such a thing as CGNAT and I leave it for you to check what is it). Your devices connected to Wi-Fi router don't have unique IPv4 addresses each. If you check your PC's IP address, you'll see something like 192.168.x.x.

I see this this:
(If you're on Mac or Windows, check the web on how to find your IP address information.)

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 ....
    ...
    inet 127.0.0.1/8 scope host lo
    ...
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
    ...
    inet 192.168.8.9/24 brd 192.168.8.255 
    ...

There are two things you see in my output: lo and eno1: 192.168.8.9

eno1 is the network interface—or the point of connection for my PC. It's the Ethernet cable connected to my Wi-Fi router, and that's its address. What is lo? It's the loopback interface. You probably know it well—it's the famous localhost - 127.0.0.1.

My Wi-Fi router has a public IPv4 address. Also, it is creating a local network (my home network; LAN - local area network) that spans as long as the signals of my router reach. Any device that can connect to my Wi-Fi router becomes a part of this network. This network is a default private network, and it uses IP addresses reserved for private usage. So, my PC has the IPv4 address 192.168.8.9, my phone - 192.168.8.10, my laptop - 192.168.8.5 and so on. This default local network can accommodate up to 254 devices simultaneously connected to my router. With the NAT mechanism, my Wi-Fi router translates all the requests and -responses sent to global net from each device connected to it, granting them access to the internet (e.g request: I go to google from my PC to find a picture of kitty and then I download it).

So, how is this all related to web dev?

You cannot directly host anything you develop on a private IP address if you want to expose it to the global net.
Anything!!! Absolutely anything you see on the web—any page you manage to open, any existing services, anything at all—resides/is stored on a "device" with an IP address.
Whatever you visit on the web from a browser: A. exposes your external (e.g the router’s public IP) IP address (little remark: VPNs and proxies can mask your IP :-)) B. exposes its IP address to your browser.
The alfa-numerical part following http(s):// in a URL (roughly speaking, "the name" of a website) can be presented as an IP - e.g you can generally replace http(s)://example.com with http(s)://<ip_address> and access the site that way, however, some sites have domain-specific configurations, so entering an IP might not always load the intended site, but the point is that a domain name ultimately resolves to an IP (on domains later on!!).

Okay, basta with IP. I introduced key information about IP addresses:

Next up: TCP.

3.2 Transmission Control Protocol (TCP)

In my analogy with envelopes and letters, the Transmission Control Protocol plays the role of a delivery service. Thanks to IP, envelopes (network packets with data) have a standardized addressing system, so if they're delivered responsibly, the data will arrive from sender to recipient. This is where TCP comes into picture: its job is to deliver envelopes (network packets with data) in a reliable way, ensuring they reach their destination.

TCP isn’t the only protocol on top of IP. There’s also UDP (User Datagram Protocol), but I’ll leave it aside for now. This meme fits well to explain the difference:

Anyway, since HTTP sits on top of TCP, let’s concentrate on TCP rather than on other protocols.

To illustrate TCP in action, as well as HTTP and HTTPS, I’ll share the results of some network packets capturing. By the way, continuing my envelope analogy: the HTTP protocol is like the language of the letter (data) inside the envelope (network packets). The sender writes the letter in a language, and the recipient responds in the same language.

When a letter (data) is sent in the HTTP "language", anyone nosy who intercepts the envelope can open it and read the letter (data) clearly, because it’s not encrypted. But if the letter (data) is written in HTTPS "language", this nosy someone will only see some general information and then abracadabra instead of the actual content.

HTTPS is a secure version of HTTP that stops bad people from reading your data while it is being transported. On the modern web, pretty much every server uses HTTPS, so if you don't include it explicitly, the browser assumes that is what you are using and adds it for you.

Anyway, I promised some nerd fun, and by that, I mean that I will capture network packets that travelling in and out of my PC to show you what is inside.

3.3 TCP/HTTP Network traffic

For HTTP network traffic, I’ll use this site: http://httpforever.com/.

For HTTPS traffic, I’ll use the URL of my DEV.TO profile as an example: https://dev.to/dev-charodeyka.

The tool I’ll use to capture network packets is Wireshark (it has a GUI). I could do the same with tcpdump, but for demonstration purposes, I want something visual.

I’ll also be using a software called curl:

curl - transfer a URL
curl is a tool for transferring data from or to a server. It supports HTTP and HTTPS protocols.

In this setup, curl will act as the client in the typical client–server relationship. In web dev a browser is usually the client in such a setup. The command I’ll use is curl with a verbosity flag (-v), so I can show you all the details.

Curl output

$ curl -v http://httpforever.com/
* Host httpforever.com:80 was resolved.
* IPv6: 2604:a880:4:1d0::1f1:2000
* IPv4: 146.190.62.39
*   Trying [2604:a880:4:1d0::1f1:2000]:80...
* Immediate connect fail for 2604:a880:4:1d0::1f1:2000: Network is unreachable
*   Trying 146.190.62.39:80...
* Connected to httpforever.com (146.190.62.39) port 80
* using HTTP/1.x
> GET / HTTP/1.1
> Host: httpforever.com
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Server: nginx/1.18.0 (Ubuntu)
< Date: Mon, 17 Mar 2025 21:00:08 GMT
< Content-Type: text/html
< Content-Length: 5124
< Connection: keep-alive
< Referrer-Policy: strict-origin-when-cross-origin
< X-Content-Type-Options: nosniff
...
<
<!DOCTYPE HTML>
<html>
    <head>
        ....
</html>

Back to explanation

Hear me out! What you’re seeing is exactly the same thing that happens under the hood when I visit this website from my browser. It might be confusing—like, how can the website (the "front end") also be about servers?

Well, that confusion arises because there’s a common understanding that "frontend" refers to something to be run in a browser (client-facing), and "backend" is the stuff on servers. But actually... everything runs on servers. Hard-to-swallow pill: if you want to self-host your future web apps and avoid paying for web hosting or PaaS, welcome to servers and the Linux world.

The backend and frontend parts of a web app can reside on the same physical machine (or the same cloud instance). However, on the software level, they are separated. Frontend is running on web servers (software, "server" comes from the fact that this software is serving something).

This line from curl output shows which web server the http://httpforever.com/’s frontend is running on:

< Server: nginx/1.18.0 (Ubuntu)

The physical server has Ubuntu OS and is running the Nginx. Physical server's IP is:

* IPv4: 146.190.62.39
default HTTP port is 80 so we see this:
*   Trying 146.190.62.39:80...

3.4 Understanding Ports

Now, about ports (:80), even if it’s outside the main scope of this article, I’ll explain with an analogy so you understand what a port is.

Imagine you have a house. To enter, you have a door. If you remove the door, you can’t enter the house. If you don’t know where the door is, you also can’t get in. Let’s expand on that: maybe you have a door for you and your family, a little door for your cat, and a big door on the ground floor for your car. If you try to drive your car through the cat door, it won’t work, and if the cat goes through the garage door, it ends up somewhere it doesn’t need to be.

It’s the same concept with ports. HTTPS traffic uses port 443, and HTTP uses port 80 by default. You may face the need to open other ports on your PC/server, as many services have their own default ports—like databases. All these ports need to be open if you want external access for a given service. And it’s also about what requests or messages you’re pushing through a port. Returning to my analogy: the cat enters the house through its little door looking for food. If it walked into the garage, it wouldn’t find its feeder. Same idea with software. For example, Mongo DataBase’s default port is 27017. If you try to send curl requests there expecting an HTTP response, it won’t work, because MongoDB can’t respond in that way.

Before I move on to the same command for an HTTPS site, I want to explain the first line of the curl output:

Host httpforever.com:80 was resolved.

3.5 DNS

You can have a look again on the error that I encountered when I dropped in the browser's search bar an erroneous URL: https://differencebetween.com.and.dev/ DNS error

Basically, the browser displayed an error because I was trying to reach an abracadabra web address, differencebetween.com.and.dev, using the https protocol. The browser couldn’t find a server at that "address" because it simply doesn’t exist. But how did the browser figure that out? Where did the browser "look up" that "address"? Does your browser have some kind of "address book" containing all the existing "addresses" on the web? Of course not. Every browser indeed checks an "address book" of the entire Internet, but it does so via the network – it doesn’t store it locally.

Before I give a name to this "address book of the Internet", I want to explain what is in this book, gracefully connecting all the pieces of information I shared above. However, I guess you can already guess what a browser needs to find a server... an IP address, of course! Take another look at the curl output:

* Host httpforever.com:80 was resolved.
* IPv6: 2604:a880:4:1d0::1f1:2000
* IPv4: 146.190.62.39
*   Trying [2604:a880:4:1d0::1f1:2000]:80...
* Immediate connect fail for 2604:a880:4:1d0::1f1:2000: Network is unreachable
*   Trying 146.190.62.39:80...
* Connected to httpforever.com (146.190.62.39) port 80

First thing that happened after curl request was sent to http://httpforever.com was that this web address was resolved. Resolved de-facto means that the client (curl in this case, though the same is completely true for browsers) received the information that httpforever.com == 146.190.62.39 (IPv4) and httpforever.com == 2604:a880:4:1d0::1f1:2000 (IPv6). The browser immediately tried to send the HTTP request to the first IP it found, which was the IPv6 address. But as I mentioned, the Internet and many servers are still only on IPv4, so the connection failed... then it quickly started to send the request to the IPv4 address, and the connection was successful!

So, where do browsers get a "dictionary" that maps human-readable addresses to IP addresses? Browsers get it from DNS (Domain Name System).

The Domain Name System (DNS) is the phonebook of the Internet. Humans access information online through domain names, like nytimes.com or espn.com. Web browsers interact through Internet Protocol (IP) addresses. DNS translates domain names to IP addresses so browsers can load Internet resources. (Cloudflare: What is DNS?)

I won’t go into too much detail on DNS, it is simple per se in what it is for, but it is not that simple to configure from the standpoint of networking configuration. Here is the schematic representation on how your browser reaches Domain Name System:

URL example: https://www.cloudflare.com/learning/dns/what-is-dns/
[Browser] --> (Browser's internal DNS cache)
                    |
                    v
        Found IP of www.cloudflare.com?
        |                |
        No               Yes ---> resolved, sending request...
        |
        v
[Operating System of PC/server] --> (Local DNS cache & configuration)
                    |
                    v
        Found IP of www.cloudflare.com?
        |                |
        No               Yes ---> resolved, sending request...
        |
        v
[Router/Modem] -->Forwards DNS query
               Forwards where? Depends on configuration..
                │             OR               |
                v                              v
  [  External Public DNS           [Internet Service Provider's 
   (e.g. Google, 8.8.8.8)]                 (ISP) DNS] 
                |             OR               |
                v                              v
               Found IP of www.cloudflare.com?
                |                |
                No               Yes ---> resolved, sending request...
                |
                v
         www.cloudflare.com does not exists.

The key takeaway is that a "human" address in a URL is just for people. Underneath, there’s always an IP address. And as a first step, browsers try to look up the "translation" of the provided address into an IP. If it fails, that’s the end of the query and results in an unavoidable error. Another takeaway is that you can't, just out of the blue, decide that your server with IP 123.124.135.5 (for example) will have a domain name like coolest-site.eu by specifying it in your server’s configurations. I hope that’s pretty obvious.

3.6 TCP in action: 3-way handshake

Anyway, here is the nerd fun!!! Haha, I called it nerd fun because once my colleague told me that I need to have a social life after I mentioned that I observed my home lab's network packets for hours to examine a peculiar network anomaly.

As I mentioned, I use Wireshark for monitoring network traffic on my PC and with this tool I can capture all the network packets that pass through it. If I start capturing everything, the output gets flooded in seconds, because any website I open in my browser ends up in the stats table. Currently, I have my PC connected to only one network, and I will capture packets traveling via this network. I will filter out only the packets for HTTP example site, http://httpforever.com, as I already know the IP after the first launch of curl I can use this network packets filter: ip.addr==146.190.62.39 and tcp.port==80. So I start capturing with an active packet filter in Wireshark, and then I just open http://httpforever.com in my browser.

Here is what I see:

That’s just what happened under the hood for one simple action: I tried to load the landing page of http://httpforever.com.

Let’s decompose it. First, for each network packet (each row in the table above = network packet) there’s the source and destination. The source of the first packet is the private IP address of my PC, as the communication was started by my PC (specifically, by my browser). The destination of the first network packet is the IP of the site http://httpforever.com. In the Info column, as a first thing you see the ports 40500 → 80. Yes, 40500 is the port used on my PC during this communication - it’s an ephemeral port, which is basically a byproduct of NAT applied by my router. The first thing after ports in the first row, column Info (No. 7) you see is the abbreviature SYN — SYN (Synchronize) is the first step in making a handshake, so it’s like extending your hand to someone else to initiate a handshake.

Next, the server where http://httpforever.com's frontend runs, sees the handshake request from my browser and acknowledges it, which is the 2nd step of handshake - SYN-ACK (Synchronize-Acknowledge) (No. 8 line in the table above).

Next, my browser sends an ACK to confirm the server’s SYN-ACK, and that completes the handshake (No. 9 line in the table above). At this point, a reliable connection is established, and data transmission can begin.

As you can see, the row No. 10 in the table informs that the network packet with HTTP GET method request was sent - data exchange between my browser and the server of http://httpforever.com has started -after the handshake, my browser immediately requests what it needs. The server acknowledges the GET request and starts sending data (PSH) (rows No. 11 - 13 in the table above).

Notice there are more than one transfers of different lengths before the line No.18 that contains HTTP response 200 - OK, meaning, that first GET request from my browser was completed. That is because, servers servers over HTTP never send everything requested in one big mega data transfer. For example, if you wanted to download a 10 GB video, the website's server from which you are downloading it wouldn’t just dump the entire file at once—that would overwhelm your network connection. Instead, HTTP network packets are chunked, and the full content is split across those packets. How the data is spitted in network packets by size, is there some rule? There are limits on the server side and on your PC. In fact, without realizing it, you’ve already seen your PC’s limit per network packet if you connect your PC to the Internet via ethernet cable - by default it is 1500 bytes, and this value is called MTU - Maximum Transmission unit. Here is the recap of $ ip a command, output of which contains mtu value:

$ ip a
...
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
    ...
    inet 192.168.8.249/24 brd 192.168.8.255 
    ...

Some packets can for some reasons not arrive to client from the first attempt, however, TCP will do its best to ensure the delivery of everything, that's why it is considered as a reliable protocol. And that is why I called it in my analogy with envelopes and letters as a delivery service. 3 way handshakes, and reattempting of sending network packets if something goes wrong.

Image source; this picture is used to stress out the fact that TCP acts as a "reliable delivery service" for network packets - not just dropping them "by the door", but requiring the "signature" as a confirmation of delivery

NB! the Wireshark output may be a bit not accurate for visualizing how network packets are split up due to the fact that Wireshark takes data segments and reassembles them into the complete application-level message. However, I wanted to demonstrate that network traffic is more complex that it seems and it is segmented.

However, this is very important to know that data travels in chunks when requested—and never gets dumped all in one bulk. It is key for cases when you have to fetch/process huge amounts of data with your JavaScript/TypeScript code (for example, NDJSON). You can take advantage of streaming approaches instead of fetching an entire file at once without streaming and nuking your PC's RAM while trying to process it. Plus, all the "download status bars" are exactly about tracking the chunks of data that arrive and comparing how much is left to the total size of the data being downloaded.

So, I showed you nerd fun with HTTP network traffic - to help you understand how HTTP works, and what happens when you try to open a website from your browser. Now, what's left to discuss is HTTPS.

4. HTTPS

If you look again at the output of a request sent by curl to website via HTTP, the most important part is the last section (which I horribly shortened), but here is the point: after all the info about HTTP communication, there’s actual data in HTML format - all the elements that will be displayed in your browser after it will reconstruct and render DOM based on this html data.

Also, if you look at line No.18 in the Wireshark table from above, you can see that the content type sent over as HTML to my browser was (text/html).

Using Wireshark, I can actually follow each network packet. So, if I select the packet that was carrying the data, I can investigate it—and I see the content in plain text. Here it is:

Getting ahead, here’s what I see if I capture network traffic for an HTTPS site: in Wireshark, I start tracking TCP port 443 and the IP of the DEV.TO website, start capturing packets, go to https://dev.to/dev-charodeyka in my browser, identify any packet with data in Wireshark table, and put my nose there, trying to see the data:

Abracadabra! The real content, html data is hidden. In the process of transfer, data transferred via HTTPS—and not HTTP—is encrypted. So if someone tries to capture it and sniff the content, they won’t understand what it is.

How is that possible? I’ll demonstrate by repeating the same procedure I did for the HTTP website—I’ll send a curl to the HTTPS site - https://dev.to/dev-charodeyka.

$ curl -v https://dev.to/dev-charodeyka
0* Host dev.to:443 was resolved.
* IPv6: (none)
* IPv4: 151.101.66.217, 151.101.2.217, 151.101.194.217, 151.101.130.217
0*   Trying 151.101.66.217:443...
* GnuTLS ciphers: NORMAL:-ARCFOUR-128:-CTYPE-ALL:+CTYPE-X509:-VERS-SSL3.0
* ALPN: curl offers h2,http/1.1
* found 152 certificates in /etc/ssl/certs/ca-certificates.crt
* found 456 certificates in /etc/ssl/certs
* SSL connection using TLS1.2 / ECDHE_RSA_CHACHA20_POLY1305
*   server certificate verification OK
*   server certificate status verification SKIPPED
*   common name: dev.to (matched)
*   server certificate expiration date OK
*   server certificate activation date OK
*   certificate public key: RSA
*   certificate version: #3
*   subject: CN=dev.to
*   start date: Tue, 07 Jan 2025 22:00:10 GMT
*   expire date: Sun, 08 Feb 2026 22:00:09 GMT
*   issuer: C=BE,O=GlobalSign nv-sa,CN=GlobalSign Atlas R3 DV TLS CA 2024 Q4
* ALPN: server accepted h2
* Connected to dev.to (151.101.66.217) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://dev.to/dev-charodeyka
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: dev.to]
* [HTTP/2] [1] [:path: /dev-charodeyka]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
> GET /dev-charodeyka HTTP/2
> Host: dev.to
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< server: Cowboy
....
< strict-transport-security: max-age=31557600
< content-length: 145678
<
<!DOCTYPE html>
<html lang="en">
  <head>
  ....
</html>

Again, same logic. Now, I hope you can read the output.
First, the dev.to address was resolved into an IP address:

Host dev.to:443 was resolved
* IPv4: 151.101.66.217, 151.101.2.217, 151.101.194.217, 151.101.130.217

However, not one, but many. Why?

4.1 Little extra info: multiple IP addresses horizontal scaling of web apps

Well, I told you networking stuff is cool, and now, dear reader, if you've read this far, you'll also learn about load balancing and horizontal/vertical scaling.

Small web applications that don’t have many visitors can run even on a Raspberry Pi (a tiny PC acting as a server). Indeed, they can even run on something like a phone or any device with limited computational power. However, when a website is pretty well-known - and it's not just a basic blog with zero interactivity, but more like an online shop or DEV.TO - one single server may not be enough to ensure that each user has nice smooth experience browsing it. And it's not only about server hardware specs like super-mega RAM or many many CPU cores; it's also about the network, haha. Network transmission speed does not multiply with number of cpu/ram.

There are two ways of dealing with the hardware resources deficit of a server where web app is running: horizontal and vertical scaling. Let’s say you created a web app and hosted it on a Raspberry Pi. Then you added some cool backend algorithms that do something for your users, so even though you only have a few users, the algorithms demand computational power. You upgrade your Raspberry Pi to something more powerful with more RAM and a better CPU. One important detail is that your website was coolest.site.eu, mapped to the IP 123.124.6.8 - the IP address of that Raspberry Pi. You replace the Raspberry Pi, move your app to the new device, and also bind the SAME IP to the new device. This is vertical scaling.

But what if your web app doesn’t do anything too computationally heavy, yet you have thousands of users accessing it simultaneously? Sure, you could replace your Raspberry Pi with a super-mega server, but that might not be the best use of resources. Instead, you could buy four more Raspberry Pi, duplicate your web app on 3 of them, and thus end up with four different IP addresses—because they can’t all share one IP. Then you’d take 4th Raspberry Pi, install any load balancing software on it (like Nginx), and use it as a "sorter" of requests sent by browsers of your website users when they try to access it. For example, if a thousand users try to reach your website, they won’t all slam into one single Raspberry Pi. Instead, their browsers’ requests land on Load balancer's Raspberry Pi first. It sees how busy each Raspberry Pi is and directs traffic to whichever Pi is least busy at that moment. That’s horizontal scaling, and what load balancer does is balance the load among all available servers.

Okay, next. As I mentioned before, HTTPS is the secure version of the HTTP protocol. If you look at the end of the curl output I shared above, once again we see HTML data in plain text—but only after the browser successfully establishes communication with one of dev.to’s servers that hosts the frontend code. That’s why in the Wireshark screenshot, when I tried to put my nose in a HTTPS network packet, I saw weird symbols instead of plain HTML. It’s because the data was encrypted.

4.2 About encryption

If I send you the message 12334 145 672 849, like this, you won’t understand it—unless you know we have a decryption map such as 1=h, 2=e, 3=l, 4=o, 5=w, 6=a, 7=r, 8=y, 9=u, which transforms the message into "hello how are you". You can then respond using the same "encryption key". Of course, this is a very simplified example. In real encryption scenarios, there are usually two keys: public and private.

In my simple example, public key will be containing this info: 1 -> ?, 2 -> ?, 3 -> ?, etc. From public key you only get partial information about encryption—you know how to encrypt the message - turn letters into numbers.
Private key is the secret part that completes the map—only the holder of the private key can reliably invert the numbers into letters.
The public key is indeed public, and it helps a client figure out how data will be encrypted by server. And where is the public key stored when we talk about servers that run websites and exposes them via https protocol? In a small document called a certificate.

Returning to the curl output:

0*   Trying 151.101.66.217:443...
* GnuTLS ciphers: NORMAL:-ARCFOUR-128:-CTYPE-ALL:+CTYPE-X509:-VERS-SSL3.0
* ALPN: curl offers h2,http/1.1
* found 152 certificates in /etc/ssl/certs/ca-certificates.crt
* found 456 certificates in /etc/ssl/certs
* SSL connection using TLS1.2 / ECDHE_RSA_CHACHA20_POLY1305
*   server certificate verification OK
*   server certificate status verification SKIPPED
*   common name: dev.to (matched)
*   server certificate expiration date OK
*   server certificate activation date OK
*   certificate public key: RSA
*   certificate version: #3
*   subject: CN=dev.to
*   start date: Tue, 07 Jan 2025 22:00:10 GMT
*   expire date: Sun, 08 Feb 2026 22:00:09 GMT
*   issuer: C=BE,O=GlobalSign nv-sa,CN=GlobalSign Atlas R3 DV TLS CA 2024 Q4
* ALPN: server accepted h2
* Connected to dev.to (151.101.66.217) port 443

Here, you see a certificate verification step because client (curl, in this case, but the same is for browser) found certificates. Where do they come from, and how does the browser (or curl) verify them?

4.3 TLS/SSL certificates

Well, these certificates were created by dev.to’s server admins. First, two keys were generated on the server: a public and a private key. The certificate includes the server’s public key and some extra data (domain name, organization info, etc.). The dev.to admins sent this certificate to a Certificate Authority (CA) to have it signed. In dev.to’s case, I can see the certificate was signed by CA called GlobalSign:

*   issuer: C=BE,O=GlobalSign nv-sa,CN=GlobalSign Atlas R3 DV TLS CA

That CA verified that dev.to actually owns the domain and then issued a signed certificate if everything was legitimate about this server and domain. The signed certificate is sent back to the dev.to server. After this, dev.to's server is equipped with its private key (secret) and its public key that got embedded in the signed certificate (this is not secret). This signed certificate is TLS/SSL certificate

When I try to access https://dev.to/dev-charodeyka via a browser:

I. My browser starts a connection via the Secure Socket Layer (SSL) protocol to the dev.to server:

* SSL connection using TLS1.2 / ECDHE_RSA_CHACHA20_POLY1305

II. The dev.to server sends its signed certificate (which contains the public key) to my browser.
III. My browser verifies the certificate (checks expiration, domain match, CA trust, etc.). You can see that checking process in the output:

*   server certificate verification OK
*   server certificate status verification SKIPPED
*   common name: dev.to (matched)
*   server certificate expiration date OK
*   server certificate activation date OK
*   certificate public key: RSA
*   certificate version: #3
*   subject: CN=dev.to
*   start date: Tue, 07 Jan 2025 22:00:10 GMT
*   expire date: Sun, 08 Feb 2026 22:00:09 GMT
*   issuer: C=BE,O=GlobalSign nv-sa,CN=GlobalSign Atlas R3 DV TLS CA 2024 Q4
* ALPN: server accepted h2

IV. After the certificate is verified and approved by my browser, the dev.to server and my browser establish a secure channel. Think of it like going into a private room so nobody else can hear what you're discussing with another person. In digital communication, that means all messages are encrypted in such a way that only the sender and recipient can understand them.

4.4 Secure communication channel

To create this secure channel, both my browser and dev.to server need to agree on encryption and decryption keys:

1) They agree on the TLS (Transport Layer Security) version and cipher format:

* SSL connection using TLS1.2 / ECDHE_RSA_CHACHA20_POLY1305

2) The dev.to server and my browser each create "ephemeral" key pairs for this specific session that my browser initialized.
3) They exchange the public parts of these ephemeral keys (encrypted with or signed by the dev.to server’s private key to ensure authenticity).
4) A shared secret (the session key) is computed on both ends - my browser and dev.to server.

Now, my browser and the dev.to server each have the same session key. All the HTML, images, and everything else is encrypted with this session key—and it’s unique to this session.

Finally, once this secure channel is established, my browser sends encrypted HTTP requests and receives encrypted responses (like the HTML code for rendering my dev.to profile).

You can compare all these steps to the earlier HTTP Wireshark table—there, everything was visible in plain text. Under HTTPS, it’s all far not that clear. No one intercepting the packets from "outside" of the secured channel can read the data unless they managed to "steal" a secret key.

Well... I guess that’s all for networking fundamentals in the context of web development. This article got quite long.

The next (and concluding) part of this series will be mostly centered around the backend side of web development and JavaScript, but from a particular perspective - here is a teaser: can you use JavaScript the same you use Python?

Summarizing:

Don’t ignore the networking and servers. Even if you’re aiming to become a CSS/HTML/JS guru who can create amazing UIs, remember that understanding how servers and networking work is crucial. Otherwise, your frontend code could unintentionally breach security and expose sensitive data. Web dev isn’t just about pretty visuals; it’s also about safe and efficient communication between clients and servers.
HTTPS is more complex than HTTP; it’s not simply a "version 2" of HTTP. Developing on HTTP and localhost won’t behave exactly the same as when you switch to HTTPS in production. Secure cookies, CORS, and encrypted data transfers all tax your code once communication passes from HTTP to HTTPS.
Get familiar with how ports work and how to monitor them. You don’t want to open a bazillion ports in your web app by repeatedly opening up database clients or other services each on a new port.
Understanding URLs is vital for setting up routing in your web apps, especially when you deal with protected endpoints that require authorization.
Understanding IP addresses is just basic for any IT field. It will prevent you from sending your friend a URL to evaluate your cool web app that has as address your home network’s 192.168.x.x address.

A privacy-aware bonus that may encourage you to explore in-depth networking and DNS:

If you’re a privacy-obsessed and always use a VPN to surf the web like an untraceable ninja, ensure that DNS queries are never forwarded to your ISP’s DNS server. If they are, you are a traceable ninja.

Where to Start in Web Development: Your Browser as Your First IDE

Anna — Sun, 09 Mar 2025 22:50:15 +0000

The reality of starting web development may be very confusing in the beginning. There are so many resources, courses, tutorials, and so on—but how do you choose the right one? Moreover, chances are that right from the start you’ll bump into React/Angular/Vue, because they are very popular frameworks, and it might seem like all web development can be done only with one of these frameworks and it begins with picking a "right" framework.

Plus, a lot of tutorials begin by setting up VS Code as the IDE (Integrated Development Environment). You start using VS Code, learn how to write .html and .css files, do some basic JavaScript scripting, and then—for simplicity—you install the VS Code Live Server. You click here, click there, maybe even run npm run... and—boom—your first web app appears in the browser. Enjoy!

But wait—what’s actually going on? localhost? Some port (:3000)? The browser complaining about an unsecured HTTP connection? JavaScript seems weird, and it’s not clear how or where to use it, or what the use cases are. How is the browser displaying your scripts so nicely? What is Node.js, and how did installing it on your PC allow the browser to execute JS scripts? So many questions might come up...

The worst thing you can do is leave these questions unanswered. As soon as you continue learning without truly understanding, these blind spots keep piling up. Eventually, you might realize you can’t develop anything based on your own ideas—at least not without following a step-by-step tutorial—because you don’t really know where to start.

In general, if you’re just discovering web development field and want to begin somewhere, starting with any of frameworks (React, Svelte, Vue and Angular are frameworks)—will only confuse you, because they’re definitely not the tools to start with web development.

Even though, of course, HTML, CSS, and JavaScript (further - JS) are the fundamentals of web development, but don't worry—this article isn't about that you just need to first master them and you will become web dev.

I am assuming that if you want to do web development, your final aim is to learn how to create web applications or websites—basically something that will be running in a browser.

And here is the first question:

How browsers work? What happens "behind the hood" when you open your browser and homepage shows on your screen? How does your browser manage to display all the web sites you visit?

You may be aware of the fact that most of the web applications have a "backend" that is residing on some mysterious "servers".

And here is the second question:

What exactly are the "backend" and "server," and how does a browser communicate with a "server"? Which code is considered "frontend," and which is "backend"? What determines whether some code belongs to one or the other?

If you cannot answer these question, that's "where" you must start in web development—by finding the answer to them.

In this series of articles I will provide the answers to these questions illustrating them with hands on code examples. I will be writing code in a very direct way—no IDEs, no frameworks, no extra stuff. Just a browser, my PC's Linux OS, and basic JS and HTML. Because I also want to show you that you can do many things with just basic tools and that only your imagination is the limit. You don't need React, Nginx, Express, Angular, VSCode Live Server or something else to start trying some web development.

So, let's start!

I’ve decided to write a set of articles on web development, rather than pack everything I want to share into one very long piece. This first article in the series—which you’re currently reading—will focus on using the browser as your powerful tool for web development. Here’s a roadmap:

Building confidence with a browser
1. Open random files in your browser to see what happens
2. Browser's Developer Tools as an IDE (Integrated Development Environment)
How does a browser work?
1. What is the document?
2. Browser's rendering engine
3. Document Object Model - DOM
4. Browser's JavaScript Interpreter

The second part of this web development series focuses on the browser's networking layer, specifically on URLs, the HTTP protocol, and a few must-know networking concepts in general. Although the networking field in IT isn’t simple, it’s a tough pill you have to swallow - if your goal is to get your web applications to a production-ready stage—rather than leaving them on your PC in some project folder, you must understand networks.

You might question this statement, especially if you’ve already explored the two well-known branches of web development—backend and frontend—and decided that frontend is what you want to do. However, even if you’ve chosen to focus on one, that doesn’t mean you can completely ignore the other. You still need at least the basic concepts and an understanding of how both "branches" work—frontend and backend. The third part of this set of articles will be dedicated to "backend" with a focus on what makes a code "frontend" or "backend".

NB! I use Linux, more precisely Debian, and I have an allergy to any other OS. So, if you use Windows or macOS, you might need to google how to do the same things on your system. Anyway, whatever I do in this article is not related in any way to the type of Operational System.

1. Building confidence with a browser

If before web development you were just a user of your preferred browser, that golden time is over—now it's your working horse if you're aiming to be a front-end developer. As a user, you might be very picky about browsers, choosing one that suits you. But the moment you decide to do front-end, hehe, soon you will install all the most common browsers for testing purposes—at least one per engine group they use (yes, even Microsoft Edge). I'll explain why later.

So, the browser... Now, I am writing this article in the browser. In a tab I am working I see in the address bar something like this:

https://dev.to/dev-charodeyka/where-to-start-in-web-development-bla-bla/edit

And if I go to any other webpage, I'll see something like:

https://some-site/home

You may have noticed that you can use your browser not only just to view web sites, but also to open images, PDF files, and other types of files. For example, I don't have any PDF reader on my system, so I use the browser to view PDF files if needed.

1.1 Open random files in your browser to see what happens

Okay, so I open LibreOffice, drop some random text, and save it as a PDF on my PC and... I open it in the browser by typing in the address bar: file:///home/lalala/Projects/DEVTO/webdev/randomPDF.pdf.

Little remark: pay attention to how a local file stored on my PC is opened in the browser vs how websites are opened— "file:///home/lalala/..." vs "https://some-site/home..". Do you notice the same syntax? The key part to notice is: https:// and file:// (third / in file:///home/lalala/... belongs to the absolute path of the file). It is important, I will return to this later in the article.

And here is what I see when I open a randomPDF.pdf file in my Browser:

What if I want to view a text file with the same content instead of a PDF?

# first, I create it 
$ vim randomTXT.txt
Hellow!
I am a TXT file!
I am visualized in a browser!

Then I paste this into browser's address bar to open text file: file:///home/lalala/Projects/DEVTO/webdev/randomTXT.txt

As you can see, the text is just text—there’s no bigger font for a title line, because the TXT format does not provide any way to format/style the text... but a Hyper*Text **Markup **L*anguage does—let’s create one!

# first, I create it 
$ vim randomHTML.html
<h1>Hellow!</h1>
<h3>I am a TXT file!</h3>
<p>I am visualized in a browser!</p>

I paste this into browser's address bar: file:///home/lalala/Projects/DEVTO/webdev/randomHTML.html
Here is the result:

Let's add some spice - styling of one word with Cascading Style Sheets - a style sheet language used for specifying the presentation and styling of a document written in a markup language:

$ vim randomHTML.html
<h1>Hellow!</h1>
<h3>I am a random <span style="color: red;">HTML</span> file!</h3>
<p>I am visualized in a browser!</p>

Result:

I demonstrated HTML and CSS in action. If I want to display more text or change the color of some words, is modifying randomHTML.html file and then refresh the browser's page to see the changes is the only way to do it? Heh, of course not.

1.2 Browser's Developer Tools as an IDE (Integrated Development Environment)

First, I have to open the developer tools (hereafter, "dev tools") in my browser. If you're not sure how to open dev tools, you can easily find instructions online. A common method is to right-click anywhere on the page and select "Inspect" from the drop-down menu; however, this option may not always be available, as some websites block it. In that case, you can launch the dev tools from your browser's control panel, where you also can check a specific keyboard shortcut they are usually tied to.

This is how my chromium based browser Brave dev tools look like (the quality of screenshot is not the best due to the automatic compression of DEVTO attachments):

Development tools of my Brave browsers are grouped into Elements, Console, Sources, Network, Performance, Memory, Application, Security. I add/delete/modify HTML elements from Elements Tab.

Using dev tools I can add/delete/modify HTML elements and CSS styles for them "on the fly". Of course dev tools are a pretty powerful tool and are not meant only for these silly modifications, but I just wanted to demonstrate how you can easily experiment with HTML and CSS directly from a browser.

However, keep in mind that whatever I do with the browser’s dev tools doesn’t affect the original .html file that I opened in the browser—all changes will be lost if I don’t save them into a separate file. The same is true for any web page you inspect with dev tools and eventually modify something - of course it does not affect permanently the original page in any way.

Okay, I played a bit with HTML and CSS from Elements tab of dev tools. There’s another interesting tab in these tools called Console, which is an interactive shell where I can write some JS code and execute it!

Some silly "print" of the sum of two numbers and a couple of evaluations of expressions:

//JS code I ran in the Browser's console:
const a = 2
const b = 3
a>b
a===b
a<b
console.log(a+b)

And you know what? Why not add a new HTML element to the displayed .html file's content opened in a browser with JS? Because JS is definitely capable of it:

//JS code I used to add a new text to my HTML document:

//first, I add <div>
const newRandomDiv = document.createElement('div');
//then, I add child of <div>, <p> with some text
newRandomDiv.innerHTML = '<p>Hellow, I was added by JavaScript!!!</p>';
//my HTML document has the <body>, so I append new element as a child of it
document.body.appendChild(newRandomDiv);

However, you might ask: what is the document? I opened the Console of dev tools and haven’t declared any document variable or anything like that—where does it come from? I guess it’s obvious that the document in question is related to the HTML file that’s open in the tab:

2. How does a browser work?

On the screenshot above, you can see that I used the browser’s JavaScript interactive shell to "print" the document. From the output it is clear that it contains HTML elements from the original .html file I opened in the browser, plus some other elements that I added manually by directly editing the HTML structure in the dev tools and also using JavaScript from the browser. However, what was not originally in the randomHTML.html file that I opened are the tags like <html>,<head> and <body>.

The <html> tag represents the root of an HTML document. The tag defines the document's body.
The <body> element contains all the contents of an HTML document, such as headings, paragraphs, images, hyperlinks, tables, lists, etc.

2.1 What is the `document`?

Everything, that is enclosed between <html></html> tags is the document. And, roughly speaking, that is what web development (front-end part) is about: working on this document - bending it, mangling it, shaping it to make it look how you want. When I opened randomHTML.html file with the browser I loaded this file into the browser, and its content became a part of document object.

When an HTML document is loaded into a web browser, it becomes a document object.
The document object is the root node of the HTML document.
The document object is a property of the window object.
The document object is accessed with: window.document or just document (W3Schools: HTML DOM Documents)

What about the TXT and PDF files that I opened before? Well, their content also become a part of document objects, when I open them in the browser.

This is the .txt file opened in my browser when I inspect the tab's content with dev tools:

<html>
  <head>
    <meta name="color-scheme" content="light dark">
  </head>
  <body>
    <pre style="word-wrap: break-word; white-space: pre-wrap;">Hellow!
     I am a random TXT file!
     I am visualized in a browser!
    </pre>
  </body>
</html>

You may notice how interestingly the raw text file content was "wrapped" into an HTML document object by my browser. It is not uncommon for modern browsers to create a minimal HTML "wrapper" for a content that a browser was requested to display and that it managed to recognize.

In the plain text file, there was no any styling, like the first line in bold and enlarged font. But content of the .txt file has the line breaks. They were preserved by the browser by attaching CSS rules for word-breaking, white space, and word wrapping!

Let's have a look on the document object structure of PDF file opened in the browser:

<html>
  <head>
  </head>
  <body style="height: 100%; width: 100%; overflow: hidden; margin:0px; background-color: rgb(82, 86, 89);">
    <embed name="DDDBE1725B1DA9C2C0B9CFA699CCB3B9" style="position:absolute; left: 0; top: 0;" width="100%" height="100%" src="about:blank" type="application/pdf" internalid="DDDBE1725B1DA9C2C0B9CFA699CCB3B9">
  </body>
</html>

What is similar to a .txt file is that the browser’s PDF viewer is embedded in a minimal HTML document. While with a text file I was able to see the actual file's content directly and even modify it right from the browser, with a PDF file it's not the same. This kind of embedding is actually a byproduct of how my browser handles and renders PDF files.

Can I mess it up with how PDF file is displayed using dev tools even if it is originally PDF file? Sure:

Here is JS code I used:

const redThingy = document.createElement('div');
redThingy.style.backgroundColor = 'red';

redThingy.style.width = '400px';
redThingy.style.height = '400px';

redThingy.style.position = 'absolute';
redThingy.style.top = '50%';
redThingy.style.left = '50%';
redThingy.style.transform = 'translate(-50%, -50%)';

redThingy.innerHTML = '<p>Hello, I was added by JavaScript to <span style="color: yellow;">mess up</span> PDF!!!</p>';

document.body.appendChild(redThingy);

However, obviously, this red box with text didn’t affect the PDF file itself in any way. What my modifications actually affected was just how my browser rendered the opened PDF file.

I guess it’s time to summarize the point of all of these experiments.

As you can see, a browser is a powerful tool. It's not just an "app" to open websites with added features like a photo viewer or PDF viewer.

2.2 Browser's rendering engine

Any browser has its own engine that renders what you see on its User Interface (UI). Roughly speaking, to render anything a browser uses instructions in a markup language—HTML.

Different browsers work differently. However, let me explain the process slightly generalizing:

When you enter a query in the browser's search bar (a URL), networking layer of the browser delivers your "request" to a destination point that is retrieved from URL and bring back to browser a response which could be any type of file or data. Some file types can’t be displayed directly (for example, if the file is corrupted or has an unrecognized format), in which case the browser may display an error or prompt you to download it instead.
If the received data is recognized as HTML, the browser parses it to build a Document Object Model (DOM) tree. This process involves reading the HTML and turning it into a structured hierarchy of nodes that represent the page’s elements. If the received data isn’t HTML but the browser can still display it (like plain text or PDF), the browser will process it in a specialized way. For plain text files, most browsers apply a minimal HTML “wrapper”; for PDFs, the browser’s PDF viewer is embedded in within an HTML context, so browser can position it, handle scrolling, zoom, and so on.
The browser then creates or updates a render tree, which is the combination of the DOM and the CSSOM (the CSS Object Model). The layout step figures out the exact positions and sizes of all elements on the page.
Finally, the browser paints (or “rasterizes”) the render tree onto your screen. This is what you actually see in the browser’s viewport.

There are many variables in the rendering process. Every element in the DOM tree is an object that the browser needs to render by calculating its position (this is where responsive design comes in, since an element’s positioning depends heavily on the available space on the user’s device), appearance, and any dynamic changes. The DOM tree isn’t static; the browser continually re-renders it as elements move or change appearance—like those animations you see on some websites.

2.3 Document Object Model - DOM

I want to elaborate more on the DOM tree.

The backbone of any HTML document is its tags. In the Document Object Model (DOM), every HTML tag is an object. Nested tags are considered “children” of the enclosing tag, and even the text inside a tag is treated as an object. The DOM represents HTML as a tree structure of tags. (Source)

<html>
  <head>
  </head>
  <body>
    <h1>...</h1>
    <div>...</div>
    <...>...</...>
  </body>
</html>

You can notice the nested structure of HTML document: each tag is an object, and the nesting forms a tree-like hierarchy.

I'm repeating this: if you're aiming to be a front-end developer, the DOM is everything. In front-end development, almost everything ultimately revolves around DOM manipulation. I've shown how to manipulate the DOM using browser's dev tools—adding HTML elements, applying CSS, and most importantly, doing it through JS. Because in front-end development, JS is primarily about manipulating the objects in the DOM tree, which are (at their core) HTML elements.

All these DOM objects are accessible via JavaScript. I’ve only shown a small part, and it might have seemed very easy. Well, it was easy because I was working directly with those DOM objects—there were only a few, and everything was clear.

But remember this: the JS libraries you use to "simplify" certain features—and especially any framework (React, Angular, Vue, etc., though each to a different extent)—actually take you one step further away from direct manipulation of the DOM objects.

Frameworks abstract direct DOM manipulation, so you write your code in a more declarative way. Behind the scene, any framework still manipulates the DOM — but you interact with a framework’s abstractions instead of directly selecting or updating DOM elements.

The key point is that these abstractions are not "bad" - they make development more efficient and your code more maintainable, as long as you have a basic understanding of how the DOM works. If you rely on abstractions without knowing what’s happening underneath, you are cooked. No fancy framework will make you a good developer if you don’t at least keep the DOM tree in mind whenever you manipulate it with JS.

I’ve shown how to do manipulate DOM directly in the browser—not to promote this style of coding (obviously, no one codes like this for a full project). I just wanted to show you what your browser is truly capable of. Everything I did was handled entirely by the browser—there was no server behind it, no external tools, just the browser.

2.4 Browser's JavaScript Interpreter

And this leads to a very important point: I was able to execute JavaScript in the browser because browsers have a built-in JavaScript interpreter. Think about it—if you’re a Python developer, before you could run Python code, you had to install Python on your PC (unless you're on Linux, where it's usually pre-installed). That’s because Python isn't machine code, so your PC can’t understand it without an interpreter. The same goes for JavaScript. The key point is that modern browsers come with a JavaScript interpreter embedded in their engine.

Sounds great, right? But here’s the cornerstone: different browsers have slightly different JavaScript interpreters inside their engines. For basic, standard JavaScript usage, this isn’t a big deal. But when you get into non-standard usage, things get trickier. And every external library you add to your code potentially brings you slightly closer to using a JS in "non-standard" way that may be not supported by some browsers. And these browsers may be the favorite browser of potential users of your web applications :).

To conclude, here is a very generalized scheme of how browsers work:

In the next article of this series on Web Development, I’ll elaborate on the format of the "query" you see in the browser’s search bar, as well as the role of "servers".

This query format is very important—I’d even call it crucial— because the routing in your future web apps result in these "queries".

Jumping ahead, I should mention that "query" is actually a bit of an oversimplification of what appears in the address bar, because the correct technical term is URI (Uniform Resource Identifier). The true query is just one part of a URI, and URIs are closely tied to the HTTP protocol in web development, which in turn is tied to networking.

In the next article, I’ll do my best to simplify and clarify all these concepts!

Summarizing the main points:

Browsers have a built-in JS interpreter and they do run JS code.
The browser’s JS interpreter is not the same as your PC (or server)’s JS interpreter—they’re different. Keep this in mind.
The internal components of browsers are complex, and each browser implements them differently. In my humble opinion, that’s what makes the frontend part harder than the backend in modern architectures.
You need to understand both backend and frontend to be a good web developer—you can’t just master one and completely ignore the other.
In front-end development, almost everything ultimately revolves around DOM manipulation (= you have to understand the concept of DOM as clear as possible)
If "network" for you only means "the Wi-Fi at home", you must invest your time in studying networking.

Virtualization on Debian with libvirt&QEMU&KVM — Networking beyond "default": Forwarding Incoming Connections to NAT'ed network

Anna — Mon, 27 Jan 2025 23:30:02 +0000

This is the second part of "Networking Beyond "Default". All must-have theory was explained in detail in the previous part, this article will be very practical.

If your understanding of these concepts is very wobbly, I strongly recommend reading the previous article:

nftables rules that perform network address translation
The structure of IPv4 addresses and CIDR
Physical vs Virtual Network Interfaces
Virtual network switches a.k.a bridges

Terminology I will use in this article:

Host – This refers to your PC, on which you set up virtualization and create virtual machines.
Guest – This is any virtual machine you create. VM/VMs – A virtual machine/Virtual Machines.
LAN – Local Area Network managed by my WiFi Router provided by ISP(Internet service Provider)
NAT – Network Address Translation.
DEFAULT network – the Libvirt's virtual network that virtual machines are connected to by default (gets started with sudo virsh net-start default)
Packet - a network packet is a formatted unit of data carried by a network.

As I have spitted the content on virtualization into 3+ articles, I’ll start with a quick overview of my current setup and the goals I want to achieve with different network configurations.

My virtual machines are created under qemu:///system, which means I run all virsh commands with sudo. Using sudo privileges allows me to configure the network as I need, whereas qemu:///session has very limited permissions in hardware virtualization (for more details, check this article).
My host machine runs Debian, with nftables managing network traffic and firewalld serving as the firewall management tool.
I have a virtual machine running MongoDB (VM name: deb-mongo). This VM was created in the most basic way and is attached to the DEFAULT libvirt network. Here’s what that entails:

The deb-mongo VM can communicate with other VMs connected to the same DEFAULT network, which operates in the NAT mode.
The VM can communicate with the host in both directions—whether the host initiates the connection (e.g., via SSH) or the VM initiates it.
The VM has access to the internet, allowing me to install MongoDB, update system packages, use ping, curl, wget, and so on.
No device on my local home network (the same network the host is on) can access this VM directly, and of course, nothing from the outside my home LAN can reach it either.

What I want to achieve, for demonstration purposes, is to make the deb-mongo VM reachable from my laptop, which is on the same local home network as the host (My Desktop PC). I want to be able to connect to MongoDB running on the VM using the MongoDB Compass GUI client on my laptop.

In the previous article, I analyzed libvirt's DEFAULT network in detail and broke its configuration down into small pieces. If you were paying close attention, you would have already understood which nftables rules block connections from anything other than the host machine to the VMs connected to DEFAULT network.

In the scope of this article I plan to enable access to the VM from devices connected to my LAN by modifying nftables rules of DEFAULT network, plus configure port forwarding on the host.

This option is viable, however, it is about turning a well-defined libvirt’s DEFAULT network into something different, which it was not meant to be.

I wrote this tutorial for educational purposes as I noticed that the questions about how to do it can be found on all possible forums for different distros. Personally, I do not use this network configuration in my home lab, and later in this article it will become clear why.

At some point, it’s better to define a new virtual network from scratch and define rules and NAT for it, rather than modify libvirt’s DEFAULT one, if you are very “NATophile”.

However, I can assume, that you are not, just maybe you think that is the only option to make VMs reachable for other devices connected to LAN. But it is not true. In the next article I will cover more suitable network configuration for the use case when VMs should be accessible members of LAN.

Now, let's begin with DEFAULT network tweaking!

Here is the roadmap for this article:

➀ DEFAULT libvirt network

➀.➀ virbrN and vnetN - what do they do?
➀.➁ About DHCP

➁ Configuring static IPv4 address on VM

➁.➀ Shrinking DHCP range
➁.➁ Modifying /etc/network/interfaces

➂ Connecting to a VM from Another Device on the LAN using the Port Forwarding Method:

➂.➀ What is the Port Forwarding?
➂.➁ Firewalld and open ports
➂.➂ Forwarding ports on the host with firewalld
➂.➃ Adding a rule to nftables ruleset to accept inbound network traffic to the VM
➂.➄ Considerable drawbacks of this method

➀ DEFAULT libvirt network

First, I start the DEFAULT network, then I start the VM deb-mongo and enter its console.

# ip a serves to show when virtual network interfaces appear
$ ip a
1: lo
 ...
2: eno1:
    inet 192.168.1.X/24 brd 
3: wlx123456789kl:
    inet 192.168.1.Y/24

$ sudo virsh net-start default

$ ip a
1: lo
 ...
2: eno1:
    inet 192.168.1.X/24 brd 
3: wlx123456789kl:
    inet 192.168.1.Y/24
4: virbr0:
    inet 192.168.122.1/24

$ sudo virsh start deb-mongo

$ ip a
1: lo
 ...
2: eno1:
    inet 192.168.1.X/24 brd 
3: wlx123456789kl:
    inet 192.168.1.Y/24
4: virbr0:
    inet 192.168.122.1/24
5: vnet0: 
    inet ???

$ sudo virsh console deb-mongo

➀.➀ `virbrN` and `vnetN` - what do they do?

First, I want to schematize what these virbr0 and vnet0 network interfaces. They weren’t there initially (ip link/ip a before sudo virsh net-start default did not contain them in outputs), but virbr0 showed up after I started the DEFAULT network, and vnet0 appeared when I started the deb-mongo VM.

virbr0 is the virtual network switch, and while its name suggests it is a bridge (and bridges are generally used to connect different networks), in the case of the DEFAULT libvirt network, virbr0 doesn't bridge anything - it does not bridge the DEFAULT network (192.168.122.0/24) to any of the networks the host is connected to. In a more physical sense, no physical network interface is plugged into this virbr0. All network traffic between VM and host, between VM and the outside world is governed by NAT (Network Address Translation) configuration.

This is partially why no VM connected to the DEFAULT network can be reached from other devices on your local home network, which the host is connected to. The other reason lies in the rules that define NAT, which essentially translate and isolate the internal network traffic.

vnet0 is a network TUN device. TUN/TAP devices are kernel virtual network devices. Being network devices supported entirely in software, they differ from ordinary network devices which are backed by physical network adapters.

The guest VM will have an associated tun device created with a name of vnetN, which can also be overridden with the element. The tun device will be attached to the bridge.This provides the guest VM full incoming & outgoing net access just like a physical machine. (Libvirt:Network Interfaces)

Let's move on.

If you check above the latest ip a output from my host PC, you'll notice there is no IPv4 address associated with the vnet0 interface, which is the virtual network interface of my VM deb-mongo. So, to proceed, I need to check the IP address of the VM directly from the VM itself.

I want to connect to the MongoDB instance running on this VM using the MongoDB Compass GUI client installed on my host machine. To do this, the first step is finding the IP address of the VM.

user-mongo@deb-mongo:~$ ip a
1: lo: 
...
2: enp1s0: 
    inet 192.168.122.196/24 brd 192.168.122.255

That's quite a strange IP address, especially considering this is the second VM created in this network. 192.168.122.1 is the address used by virbr0, which acts as the router/gateway for this DEFAULT network (I'll elaborate more on it later). So why is my VM's IP .196 instead of something like .2 or .3? Where did it get this address? The answer is that it got it from DHCP - the Dynamic Host Configuration Protocol.

➀.➁ About DHCP

When you set up any network, any device connecting to this network needs to have certain information, such as the IP-address of its interface, the IP-address of at least one domain name server, and the IP-address of a server in the LAN that serves as a router to the internet. In the manual setup you have to type in this information for each client anew. With the Dynamic Host Configuration Protocol (DHCP) the computers can do that automatically for you. (Source)

Each virtual network switch can be given a range of IP addresses, to be provided to guests through DHCP.
Libvirt uses a program, dnsmasq, for this. An instance of dnsmasq is automatically configured and started by libvirt for each virtual network switch needing it. (Libvirt: Network Address Translation-NAT)

As stated in the libvirt documentation quoted above, I expect to find in the network configuration the range of IPs that the DHCP server is configured to assign. Indeed:

$ sudo virsh net-dumpxml default

<network>
  <name>default</name>
  <uuid>2becd6d0-5a0f-4b45-afff-5a518370fc8c</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='MA:C:AD:DD:RE:SS'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

DHCP automatically assigns IPs to the connected to DEFAULT network virtual machines from the range 192.168.122.2 - 192.168.122.254, based on availability rather than strictly sequential allocation. That’s why my VM ended up with .196.

What DHCP does respect when assigning IPs is that it keeps track of what’s happening in the network it manages. It won’t assign an IP that’s already currently occupied to new VMs joining the network. However, unless configured otherwise, DHCP won’t intervene if two VMs somehow end up with the same IP, and that can create a mess.

Why would this happen, you might ask? Well, it’s not uncommon to change the DHCP mode from dynamic IP allocation to static IPs (manually assigned). With the default setup, DHCP assigns dynamic addresses, so when a device connects to the network, it gets an IP like X. But if it disconnects and reconnects later, it might be assigned a completely different IP like Y, which can be very inconvenient for certain services that need to be accessed remotely.

➁ Configuring static IPv4 address on a VM

NB! It’s not being said for sure that DHCP is some crazy guy randomly assigning IP addresses by just looking at new connections and completely ignoring whether a device (like a VM's network interface) was already connected before. In fact, it’s very common for the IP address assigned once to persist. This is because the VM’s network interface, even if virtual, has a consistent and unchanging MAC address - so, it can be identified.

Anytime the VM reconnects to the same network, it will 99.9% get the same IP address as it was assigned the first time. However, sometimes certain factors can cause DHCP to reassign a different address.

On top of that, you may want to bring some order to your VMs, especially if there’s a logical structure to them, and you’d like more control over their network configuration. Assigning static IP addresses can help with this. While static IPs introduce some network fragility, they also improve network security, as you can create firewall rules, nftables rules, etc., based on the fixed IPs.

Of course, you can still set up such rules with DHCP-assigned dynamic IPs. However, the problem arises if the VM’s IP address changes for any reason. In that case, all the carefully imposed network traffic rules would collapse, and the setup would need to be reconfigured.

Moreover, knowing how to configure VM's network interface in the way you want it to is very nice skill to have, IMHO.

To manually assign a static IP to a VM, there are two things to respect: the range of network - I cannot assign IP address out of range; unique IP address - no any other VMs should have the same IP.

➁.➀ Shrinking DHCP range

To prevent, from the start, the possibility that DHCP assigns to a new VM an IP address I manually glued to one of the VMs by reconfiguring its network interface, I have to modify the DEFAULT network configuration and shrink the DHCP range—the set of available addresses that DHCP can use to assign.

Currently, this range occupies all the addresses available in the 192.168.122.0/24 network—all 254 addresses, excluding those reserved for the gateway/router (virbr0) and the broadcast address. By shrinking this range, I ensure there are IPs left outside the DHCP range that I can assign manually to specific VMs without interference.

#to prevent problems with VM connectivity
$ sudo virsh destroy deb-mongo

$ sudo virsh net-edit default
#I replace this:
<network>
...
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
---> <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>
#with:
<network>
...
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
---> <range start='192.168.122.101' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

# restart DEFAULT network
$ sudo virsh net-destroy default
$ sudo virsh net-start default

# check that changes came in power:
$ sudo virsh net-dumpxml default

➁.➁ Modifying `/etc/network/interfaces`

As it was found out earlier, the current IP of deb-mongo VM is 192.168.122.196. To change this IP I will have to modify configurations of network interface stored in /etc/network/interfaces

user-mongo@deb-mongo:~$ sudo vim.tiny /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug enp1s0
iface enp1s0 inet dhcp <--here is what I need to change

# I comment out the line: #iface enp1s0 inet dhcp and replace it with:
iface enp1s0 inet static

With this line (iface enp1s0 inet static), I just switched the mode of interface configuration from dhcp to static, which means I now have to configure it manually. My goal is to assign an IP address of my choice, which can be set using the address field. However, if you were attentive to the previous section, you’d know that DHCP doesn’t just assign an IP address—it also provides the device (the network interface of the VM) with crucial information about the network it’s connecting to. More specifically, it provides details such as the size of the network (netmask) and the IP address of the gateway. Since I switched to static mode, I need to manually provide the following details in the configuration: address, gateway, and netmask.

address 192.168.122.10 is the IP address I want this VM to have persistently. You can choose any number outside the range .

Information about the gateway and netmask is taken from this part of the network configuration:

  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.101' end='192.168.122.254'/>
    </dhcp>
  </ip>

Gateway is ip address='192.168.122.1', netmask is netmask='255.255.255.0'. With the netmask, it’s pretty straightforward—it corresponds to the CIDR notation of the network. For example, /24 equals 255.255.255.0, with the last 0 indicating the part of the IP address range that is available for assignment. This defines the size of the network.

But what about the gateway? Remember the previous article's explanation about network switches? In the DEFAULT network, this job is handled by virbr0. All VMs connected to this virtual network switch become part of the same network, and they are allowed to communicate with each other (as per the nftables rules set by libvirt).

It’s not just that by connecting to the same virtual network switch, VMs can magically communicate directly with one another. No, there is a network switch in between EACH COMMUNICATION, acting like a post office. When you want to send a letter to someone, you write the letter, put the destination address on it, and bring it to the post office—you don’t usually travel to the recipient’s address and deliver it manually. The post office handles everything, ensuring the letter is delivered.

The same logic applies to the network switch, and in the case of the DEFAULT network, this role is performed by virbr0, which acts as the gateway. Gateways, as a technical term, are broader than just routers or switches because they can perform a variety of tasks. It is important for the VM to know where to find the gateway in order to communicate via it with other network devices and with the outside world (if allowed). Regardless, the gateway (virbr0) will decide how to handle any communication sent from the VMs connected to it. virbr0 IP address is 192.168.122.1 (can validate with ip a).

This information ends up in the configuration file /etc/network/interfaces:

....
....
#iface enp1s0 inet dhcp
iface enp1s0 inet static
  address 192.168.122.10
  netmask 255.255.255.0
  gateway 192.168.122.1

Then, I restart networking systemd service:

user-mongo@deb-mongo:~$ sudo systemctl restart networking
user-mongo@deb-momgo:~$ ip a
1: lo: 
....
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP ...
    link/ether ...
    inet 192.168.122.10/24 brd 192.168.122.255

# to check if nothing went messed up:
user-mongo@deb-momgo:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=112 time=17.4 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=112 time=17.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=112 time=16.5 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms

#try to update system packages
user-mongo@deb-momgo:~$ sudo apt update && upgrade
...
Fetched 485 kB in 1s (901 kB/s)
...
All packages are up to date.

#try to ssh from HOST MACHINE
ssh user-mongo@192.168.122.10
...
Are you sure you want to continue connecting (yes/no/[fingerprint])?
#It is requesting you again to accept the fingerprint, because you change "address" of a VM
#Anyway ssh is successful

NB! If you experience any problems with VM's connectivity to the internet, try the following steps:

Destroy the network with sudo virsh net-destroy default;
stop the VM with sudo virsh destroy <vm-name>; restart the libvirtd service with sudo systemctl restart libvirtd; start the network with sudo virsh net-start default; start the VM again with sudo virsh start <vm-name>. Restarted VM should not have problems with connectivity. If connectivity still doesn’t work after these steps, then something got messed up wrong with the configuration.

So now, I should be able to connect to the MongoDB running on my VM deb-mongo from the host using the MongoDB Compass GUI client.

For demonstration purposes, I didn’t configure any sophisticated RBAC for the database, so all I need to connect is the IP address and the port through which the database is accessible. Here is the MongoDB configuration on the VM:

$ user-mongo@deb-mongo:~$ cat /etc/mongof.conf
...
# network interfaces
net:
  port: 27017
  bindIp: 192.168.122.10
...

Voila:

Now, the objective is to modify configurations of the DEFAULT network in such a way that I can connect to MongoDB on the VM from a laptop that is connected via Wi-Fi to the same local home network as my PC (host).

➂ Connecting to a VM from Another Device on the LAN using the Port Forwarding Method:

This is what Port Forwarding method is about:

As I’ve pointed out more than once, 192.168.122.10 belongs to a different network than 192.168.1.0/24 (my home's LAN). virbr0 does not bridge any physical interface of my host PC that is connected to the LAN with the DEFAULT network. As a result, the VM’s address is unreachable for any device connected to the LAN.

However, the devices connected to my home's LAN can communicate between themselves.

➂.➀ What is the Port Forwarding?

If you are a developer and work on remote servers using VSCode as IDE, most probably you’ve already used port forwarding indirectly. For example, when you’re developing something like a web app during its early stages, it’s common for the app’s components—backend or frontend—to run on the server’s localhost using specific development ports. However, when you connect to this server via VSCode and update the code, you can see in the browser the preliminary results on your development machine (let's call it laptop), not on the server’s browser.

Here’s the point: let’s say your frontend app is running on the server’s localhost at port 5173. If you try to access it directly from your laptop’s browser, you won’t see anything. That’s because the app is running on the server, not your laptop. Without port forwarding, the server’s localhost is inaccessible to your laptop.

What happens, often with just two clicks—and sometimes even automatically in VSCode—is port forwarding. This forwards the server’s port (e.g., 5173) to a port on your laptop. As a result, you can open the browser on your laptop and see your app at an address like http://localhost:5173. What you’re actually seeing is the server’s localhost:5173, forwarded to your laptop.

So, I’m about to do something similar, bu with port forwarding on the host to the VM deb-mongo. On my host machine, there’s no MongoDB installed, but it is running on the VM. I want to use the host machine as a kind of layover "airport" for the network packets traveling from my laptop to the VM to reach the database (the MongoDB Compass GUI client on my laptop exchanges packets with the MongoDB server and translates the packet payloads into the visualizations I see).

The layover airport analogy fits perfectly here. In real life, layovers are a common practice, and the most interesting parallel is with visas. For instance, let’s say you’re traveling to Brazil from your home country. Based on a bilateral agreement between your country and Brazil, you don’t need a visa to enter Brazil. However, your flight has a layover in the UK, and to enter the UK, you do need a visa. Here’s where it gets interesting: if you’re just transiting through the UK and stay within the airport's transit zone, you don’t need a true UK visa (let's leave aside transition visa). As long as you don’t exit the airport, everything works perfectly. But if you try to leave the transit zone, UK authorities will block you because you’re not permitted to enter UK without a true visa.

Similarly, in networking, my host machine acts as the "layover airport." Packets from my laptop need to travel through this transit point (the host) to reach the MongoDB instance running on the VM. As long as the host is correctly configured to forward packets (like a transit zone in an airport), everything flows smoothly. If, however, the laptop tries to directly access the VM (bypassing the host’s forwarding), the packets will fail because they are not permitted to "enter".

Anyway, why do I speak so much about permissions? Because if you use firewalld or any other firewall that’s up and running, they usually protect your device from unauthorized access. By default, all ports on your host are closed for NEW connections. However, ESTABLISHED and RELATED connections are often allowed, meaning that network packets can navigate through them. ESTABLISHED means the connection was initiated by the device itself (remember the distinction between inbound vs outbound traffic from the previous article?).

What I want to do is use a random port, 12345, on my host machine as a "dummy port" that my laptop can connect via LAN to using the TCP protocol. I want that all network packets sent by my laptop to my host's IP:12345 will be redirected to the VM deb-mongo, specifically to MongoDB’s default port, 27017, and the same for network packets with response from MongoDB.

➂.➁ Firewalld and closed ports

Now, for demonstration purposes, I’ll show you that a connection to this port (12345) on my host is not allowed by firewalld. I’m using the netcat-openbsd package to perform this simulation. It’s installed on both my laptop and my PC (the host).

On the host, I start listening on port 12345. This is just a random listener; there isn’t any service running on this port. It’s not like the case with MongoDB, where MongoDB listens on its default port and expects specific syntax that it can understand. With netcat, I can send any nonsense via TCP, and the listener will still accept it without validation—because it’s just a raw socket tool.

On my host I start a listener on port 12345

$ nc -lv 12345
Listening on 0.0.0.0 12345

From the laptop I try to send a message to the host private IP, port 12345:

$ echo "hellow from the laptop" | nc -v 192.168.1.106 12345
nc: connect to 192.168.1.106 1235 (tcp) failed: No route to host!

The connection fails. I start by investigating the firewalld rules because I’m sure it’s responsible for blocking the connection. And indeed:

$ sudo firewall-cmd --query-port=12345/tcp
no

This confirms that port 12345 is not open (read: no any device can connect to my host via this port). Next, I control which zones are being used for my network interfaces. firewalld applies rules based on zones. However, explaining zones in detail is outside the scope of this article (you can check the documentation here).

$ sudo firewall-cmd --get-active-zones
libvirt
  interfaces: virbr0
public (default)
  interfaces: wlx123456789kl

Not the best idea to modify firewall rules for the public zone, so I switch to the home zone, which is actually the technically correct one because my host is connected to my home's LAN.

$ sudo firewall-cmd --zone=home --change-interface=wlx123456789kl
success
$ sudo firewall-cmd --zone=home --query-port=12345/tcp
no

This port 12345 is also closed in the home zone, so I could just open it with sudo firewall-cmd --zone=home --add-port=12345/tcp. But I want to open it only to be accessible by the private IP address of my laptop. So I just add a rich rule instead.

Remember! Avoid, whenever you can, any wobbly rules for accepting something. Do not open ports to the whole world. Always try to minimize access and add precise, restrictive, and well-defined permissive rules.

$ sudo firewall-cmd --zone=home --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="tcp" port="12345" accept'
success

And voilà: the message from my laptop successfully arrived at my host, port 12345:

$ sudo nc -lv 12345
Listening on 0.0.0.0 12345
Connection received on LAPTOP-1234567.station XXXXX
hellow from laptop!

NB! If you are unfamiliar with firewalld, you might not have noticed that I didn’t add these rules permanently. This means that when I reboot or reload the firewalld service, these rules will be lost—which is exactly what I wanted.

For port forwarding, the rule is different and doesn’t require keeping the port open on the host which will be forwarded. This is where my earlier example with visas is valid—packets that are simply transiting through my host don’t need permission from the firewall because they don’t actually "enter" my host at the end.

So, I’ve demonstrated and imposed these rules purely for educational purposes, and now I’ll flush them with sudo firewall-cmd --reload, as they’re no longer needed.

➂.➂ Forwarding ports on the host with `firewalld`

Now, I’ll forward the host’s port 12345 to the VM’s deb-mongo default MongoDB port 27017. I’ll do this by adding a rule with firewalld.

According to the firewalld documentation, this is how port forwarding should be done:

[--permanent] [--zone=zone] [--permanent] [--policy=policy] --add-forward-port=port=portid[-portid]:proto=protocol[:toport=portid[-portid]][:toaddr=address[/mask]] [--timeout=timeval]

As I flushed the rules from the example above, I first need to repeat the process of changing the zone for my WiFi interface from public to home. Once that’s done, I can apply the port forwarding rule. NB! I do not add --permanent option to the commands, but if you plan to keep these firewalld configurations, you will need to specify this option.

$ sudo firewall-cmd --zone=home --change-interface=wlx123456789kl
success
# THE RULE FOR PORT FORWARDING
$ sudo firewall-cmd --zone=home --add-forward-port=port=12345:proto=tcp:toport=27017:toaddr=192.168.122.10
success

So, can I connect now from my laptop to MongoDB running on the VM deb-mongo? No. However, half the job is done.

In the previous article, I described in very detailed way why any inbound traffic is blocked by rules specified in nftables that are placed there by libvirt. For the DEFAULT network, these NAT rules are configured to isolate VMs, allowing them to connect outbound but blocking any inbound traffic from external devices (except Host, of course). This is why the connection still fails.

➂.➃ Adding a rule to `nftables` ruleset to accept inbound network traffic to the VM

To fix this, I’ll need to adjust these nftables rules. To see ALL the active nftables rules, you can use the command sudo nft list ruleset. To see only rules related to the DEFAULT network, you can specify the table and its family. Table name is libvirt_network, and its family is ip which stands for IPv4 - rules in this table are applicable ONLY to IPv4 network traffic! I prefer to use sudo nft -a list table ip libvirt_network.
The -a option adds a handle to each rule. This handle acts like an index, allowing me to reference and manipulate any specific rule without having to retype the entire rule.

$ sudo nft -a list table ip libvirt_network
...
table ip libvirt_network { # handle 6
    chain forward { # handle 1
        type filter hook forward priority filter; policy accept;
        counter packets 728 bytes 637074 jump guest_cross # handle 7
        counter packets 728 bytes 637074 jump guest_input # handle 5
        counter packets 207 bytes 14639 jump guest_output # handle 3
    }

    chain guest_output { # handle 2
        ip saddr 192.168.122.0/24 iif "virbr0" counter packets 1 bytes 76 accept # handle 55
        iif "virbr0" counter packets 0 bytes 0 reject # handle 52
    }

    chain guest_input { # handle 4
        oif "virbr0" ip daddr 192.168.122.0/24 ct state established,related counter packets 1 bytes 76 accept # handle 56
        oif "virbr0" counter packets 9 bytes 468 reject # handle 53
    }

    chain guest_cross { # handle 6
        iif "virbr0" oif "virbr0" counter packets 0 bytes 0 accept # handle 54
    }

    chain guest_nat { # handle 8
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 192.168.122.0/24 ip daddr 224.0.0.0/24 counter packets 0 bytes 0 return # handle 61
        ip saddr 192.168.122.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return # handle 60
        meta l4proto tcp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 # handle 59
        meta l4proto udp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 1 bytes 76 masquerade to :1024-65535 # handle 58
        ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade # handle 57
    }
}
...

If you’ve already dag all the internet searching for an answer about how to add port forwarding to the DEFAULT libvirt network in NAT mode, or how to use the port forwarding method to access NAT'ed virtual networks, I imagine that your attention is attached to the chain guest_nat in the displayed above libvirt_network table, because it handles NAT.

However, this is misleading! The rule that is actually blocking inbound connections to the VM (deb-mongo, in my case) from a laptop connected to the LAN is not there - the blocking rule is in the chain guest_input!

chain guest_input {
          oif "virbr0" ip daddr 192.168.122.0/24 ct state established,related counter packets 1 bytes 76 accept # handle 56
          oif "virbr0" counter packets 9 bytes 468 reject # handle 53
}

These two rules, #56 and #53, are the culprits behind the issue. Rule #56 accepts ONLY the traffic with destination address in the DEFAULT network 192.168.122.0/24 which is part of ESTABLISHED and RELATED network connections. This is why connections between the host and the VMs work seamlessly, as those are ESTABLISHED or RELATED.
Similarly, outbound traffic from any VM on this network (e.g., downloading packages or connecting to external resources) also falls under this rule.

However, when I try to connect to any VM on this network (192.168.122.0/24) from my laptop, which is on the LAN (a "neighbor" of the host), this traffic is a part of NEW connection. This is where Rule #53 comes into picture and REJECTS any traffic that is part of NEW connections targeting the virtual network devices.

The intuitive solution is to modify rule #56, and using its handle, I can do something like this :

$ sudo nft replace rule libvirt_network guest_input handle 56 oif "virbr0" ip daddr 192.168.122.0/24 ct state new,established,related counter accept

HOWEVER! Remember, it’s wiser to avoid adding overly permissive rules. So, instead, I’ll specify that traffic that is part of NEW connections AND:
A) ONLY from the private IP of my laptop
B) ONLY to the VM deb-mongo
c) ONLY to the port 27017 of the VM deb-mongo
should be accepted, while the rest should fall into rule #53—the rejection rule.

To do this, I need to add a new rule to handle NEW traffic.

Please NOTE! Rule #56, which accepts traffic from ESTABLISHED and RELATED connections, is the first in the chain guest_input. Rule #53, which rejects anything that is not accepted by Rule #56, is the second rule in the chain. If you add a new rule using sudo nft add rule ..., it will be appended to the end of the chain. This means it will never participate in filtering traffic from NEW connections because all these packets will already be rejected by Rule #53. The order of rules in chains matters!

Instead of using sudo nft add rule..., I will use sudo nft insert rule .... This command always inserts the rule at the top of the chain, which is not always ideal for different network logic but, in this case, is completely fine.

$ sudo nft insert rule libvirt_network guest_input \
    oif "virbr0" ip saddr 192.168.1.10 ip daddr 192.168.122.10 tcp dport 27017 \
    ct state new counter accept

VOILA! the connection on my laptop is installed right away after this new rule is added:

As you may notice, if you are following my steps, nothing needs to be restarted with sudo virsh- nor the DEFAULT network, nor the VM, nor the systemd networking service on the host, nor the same service on the VM. That’s the superpower of network traffic management and mangling.

As I mentioned earlier, virbr0 is not connected in any way to any physical network interfaces of your host. All the connectivity is managed through the nftables ruleset in the libvirt_network table for IPv4 traffic.

➂.➄ Considerable drawbacks of this method

In the end, to enable the Port Forwarding on the NAT'ed DEFAULT network it’s just two commands one: to add the firewalld rule to impose port forwarding on the HOST. Second command is to add a rule to the guest_input chain in the libvirt_netwrok ip table in nftables ruleset.

Quite elegant and simple, no? So, what’s the drawback then, besides the complexity of nftables rules for people completely unfamiliar with them? WELL… the drawback is the persistence of this configuration—or, to be more accurate, zero persistence.

When I added the port forwarding rule with firewalld, I noted that I didn’t make it permanent, but you can preserve the rules by using the --permanent option. What about the nftables rule I added? How do you preserve it? No way.

If you remember, at the start, I showed you (using ip link/ip a outputs) that virbr0 doesn’t exist as an interface before you start the DEFAULT network that is based on it. The same logic applies to the nftables rules related to this virtual interface. If you list all the rules when the DEFAULT network is down, there will be no libvirt_network table in nftables ruleset. But the moment you start the DEFAULT network, the table immediately appears—it is added automatically by libvirt itself.

So, any custom rules for the DEFAULT network you add to nftables will be lost the moment this network stops and restarts. This is the main drawback of this otherwise clean and straightforward approach.

There is a workaround, though—a script that can automate the re-adding of the nftables rule. Even though it’s still a workaround, it is presented in the libvirt documentation on how to forward connections for the DEFAULT network.

However, there’s an important note: the documentation’s example is written for iptables, not nftables.

Here’s a tiny reminder: you cannot just drop iptables rules out of the blue on your Debian system if you have nftables up and running. At best, the rules will simply not work. At worst, you will mess up all the network traffic on your host :3.

Here is this script:

#!/bin/bash

# IMPORTANT: Change the "VM NAME" string to match your actual VM Name.
# In order to create rules to other VMs, just duplicate the below block and configure
# it accordingly.
if [ "${1}" = "VM NAME" ]; then

   # Update the following variables to fit your setup
   GUEST_IP=
   GUEST_PORT=
   HOST_PORT=

   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
    /sbin/iptables -D FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
    /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
    /sbin/iptables -I FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
    /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi

This workaround is actually a hook script— libvirt's hooks are quite handy for specific system management needs. These hooks can trigger various scripts based on VM-related events like start, shutdown, etc. You can definitely create a similar hook for adding the nftables rule, and it would even be much shorter than this monstrosity of iptables rules.

However, I will not be doing it, nor will I use this configuration for my home lab. That’s because the DEFAULT network is designed for a different purpose—it’s meant to provide a plug-and-play or out-of-the-box experience for new VMs. You create a VM, start it, and everything works from the networking side without additional configuration.

I prefer to keep it that way. For accessibility to the VMs from my LAN devices, I use other network configurations that are better suited for this purpose.

Virtualization on Debian with libvirt&QEMU&KVM — Networking beyond "default": Must Have Concepts to Start

Anna — Thu, 23 Jan 2025 00:12:56 +0000

Initially, I planned to publish all how-to s for advanced network configurations for virtual machines in one article. However, I am unstoppable when it comes to writing, and the article became quite long. So, I decided to split the theory from the hands-on configurations, XML/Shell scripting, etc.

You might think you don’t need to read this, but I strongly discourage you from jumping directly to the next article and blindly copying my configurations without understanding what you’re doing.

NB! I’m not a network engineer, nor have I taken courses on this. I’m actually just a dev who’s just passionate about Debian and spends way too much time on my PC. So, please don’t throw slippers at me if I end up sharing any misleading information. I’m simply documenting and describing some of my experiments in this field.

Terminology I will use in this article, sometimes in acronym form – I try to avoid it, but sometimes it just happens automatically:

Host – This refers to your PC, on which you set up virtualization and create virtual machines.
Guest – This is any virtual machine you create. VM/VMs – A virtual machine/Virtual Machines.
ISP – Internet Service Provider.
LAN – Local Area Network.
NAT – Network Address Translation.
DEFAULT network – This refers to the Libvirt's virtual network that virtual machines are connected to by default, within the scope of qemu:///system. If you do not understand the difference between qemu:///system and qemu:///session and how to switch between them, please refer to the previous article. In this article I will be creating/configuring VMs that are in the scope of qemu:///system.
Packet - A network packet is a formatted unit of data carried by a network.

When it comes to networking, it’s really hard to decide where to even start explaining. To avoid turning this article into some kind of networking handbook—which I don’t have the expertise to write anyway—I’ll probably have to skip over some fundamental concepts.

However, I’ll do my best to simplify things and provide schemes for the main concepts so you can (hopefully!) follow along.

Here is the road-map for this article:

➀ About Public IP address and LAN

➀.➀ Inbound vs Outbound traffic

➁ Understanding your LAN and private IP address(s) of your host machine

➁.➀ Network Interfaces: general information
➁.➁ Network Interfaces: MAC addresses
➁.➂ IPv4 vs IPv6 addresses
➁.➃ IPv4 address ranges reserved for private networks
➁.➄ IPv4 addresses structure and CIDR

➂ Libvirt's DEFAULT Network: about NAT mode

➂.➀ About virtual network switches
➂.➁ Libvirt wants iptables, Debian has nftables: what to do?
➂.➂ DEFAULT Libvirt's network: what's under the hood?
➂.➃ Nftables rulesets: tables and chains explained
➂.➄ About how NAT works

Let's start!

I’ve already written some explanations of how your home network works, about public IP address, and ports in this article.

Here is the schematic representation of the most common setup of local "home" network with WiFi router playing the central role in it:

➀ About Public IP address and LAN

The most important takeaway from this scheme is that all your devices—whether they connect to your Wi-Fi router wirelessly or physically (via Ethernet cable)—are part of a local network, your home network. This LAN (Local Area Network) is created and managed by your Wi-Fi router.

Most likely*, only your router has a public IP address! Your devices do not (I wrote most likely because if you’ve configured your Home Lab to use IPv6, the story is quite different. But I’m guessing you wouldn’t be reading this article if you were capable of doing so! :) ) Instead, each device connected to your Wi-Fi network is assigned a private IP address—one private address per device. If, for some reason, something gets messed up with your router and it assigns the same IP address to two devices, those devices will start having problems accessing the internet (they will start losing some packets - for example, pinging will be unstable, with some percentage of packets lost).

You can check your public IP address using website like DNS Leak Test, which will show you IP address with which you visited this site (what is DNS and DNS "leaking" I will explain later in this article). Your Public IP address is discovered by servers hosting the websites you visit. For example, when I upload an image here in articles, the Dev.to servers see the request coming from my public IP address that’s pushing the data.

➀.➀ Inbound vs Outbound traffic

For now, what you know for sure is that a Wi-Fi router is the magic box that allows you to access internet resources from your devices. This is quite obvious because you paid for exactly that. But how does it actually work?

In the previous part of this series, I mentioned that I couldn’t host anything on my PC - a website - without tweaking the router. In general, when someone tries to reach you via your public IP address, they send you packets with some data/requests. These packets first reach your Wi-Fi router as it is the one having public IP address, not your devices connected to WiFi! What happens next depends on your router’s configuration.

Will it drop the packets, effectively blocking them, or will it redirect them to the appropriate member of your local network (for example, your PC)? If it does redirect, the router uses the private IP address (as he knows all the private addresses of connected devices) of the target device (your PC) to forward the packets. Router will also have to forward the packets to the correct port. However, for all of this to happen, the router needs to be explicitly configured. By default, most Home-purpose routers will just block incoming traffic, and the packets will simply be dropped. This is why I cannot host any website on my PC without modifying the configurations of WiFi router.

However, I can download whatever I want by default. When I download something, it comes to my computer in form of packets with data as well. And the stuff that I am downloading is coming in, not out. So, it is also a sort of incoming traffic, and it does not get blocked by WiFi router at all.

The key difference in these two examples is who initializes the communication. When you download something—i.e using wget—it’s your device that sends the request, initializing the connection. The server responds, sends packets with requested data, they arrive to your WiFi router, it redirects them to the device that made a "request" - and that’s why it works - WiFi router does not impede this process, because it is outbound traffic.

Inbound traffic, on the other hand, refers to situations where the connection isn’t initiated by your device—like when uninvited guests show up at the door of your house. In those cases, the firewall rules of your router sends them away. These rules protect your local home network from unexpected or unwanted traffic.

In the scope of this article on virtualization, the focus will be on networking "locally". It won’t be about creating networks that make your VMs publicly accessible, vulnerable, or anything of that sort. I won’t touch my router configurations, and all configurations will be done on networks that operate behind a Wi-Fi router.

➁ Understanding your LAN and private IP address(s) of your host machine

As I mentioned, the Wi-Fi router creates a LAN (local area network). Any devices you connect to the Wi-Fi join this LAN, and their communication—both with each other (if configured so) and with the outside world (web) —is managed by the Wi-Fi router.

The first thing to do is to get familiar with the private IP addresses of your host machine - what do they mean, why they are like this, how to manage them.

If you’re using ONLY Network Manager to handle your network connections... well, put it aside for now. Your new best friend is the iproute2. It should already be installed by default on Debian. The thing is, when you install Network Manager, it can start conflicting with iproute2 configurations. However, on Debian, this is partially resolved by the default setup, where Network Manager only manages wireless network interfaces.

➁.➀ Network Interfaces: general information

Now, let’s take a look at what’s going on with the networks on my host machine.

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> ...
....
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> ... state UP ....
   ....
3: wlxXXXXXX43643754XX: <BROADCAST,MULTICAST,UP,LOWER_UP> ... state UP 
   ...
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> .... state DOWN ...
    ....

What I have:

lo: This is the loopback interface. It’s a virtual network interface used by my OS to communicate with itself. It’s always there—no need to touch it! It won’t participate in the networking configurations I’ll be covering in this article.

eno1: This is my network interface that comes with the Ethernet cable that physically links my PC to the Wi-Fi router. *This guy will be playing the key role in the networking setups for my VMs.
*

wlxXXXXXX43643754XX: This is the wireless network interface that comes from my USB Wi-Fi adapter. In my setup, it has higher priority - this interface is used for communications with WiFi router and within the LAN. However, this wireless network interface WILL NOT participate in SOME networking setup (bridged networking) for the virtualization process because it’s problematic. Wireless network devices use different drivers, different protocols, and it isn’t trivial to make them to participate in bridged networks.

virbr0: The last guy in the list, which is currently DOWN, is a virtual bridge. It’s managed by libvirt/qemu. It is created when the libvirt daemon is first installed and started. However, it remains down unless any libvirt's network that uses this network interface starts (i.e sudo virsh net-start default).

Let's look into more details about each network interface:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> ...
....
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> ... state UP ....
   link/ether 12:ab:c3:d4:ef:56
   inet 192.168.1.X/24 brd 192.168.1.255
   inet6 fe80::278e:1234:l678:123/64 
3: wlxXXXXXX43643754XX: <BROADCAST,MULTICAST,UP,LOWER_UP> ... state UP 
   link/ether 98:zy:xb:67:kl:34
   inet 192.168.1.Y/24 brd 192.168.1.255
   inet6 fe80::kre7:b5b5:z8y6:987/64 
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> .... state DOWN ...
    link/ether 77:mo:5g:k9:r0:46
    inet 192.168.122.1/24 brd 192.168.122.255

In this article section, I will explain what are these values: link/ether, inet and inet6.

➁.➁ Network Interfaces: MAC addresses (`link/ether`)

First, I want to point your attention to the fact that eno1 and wlxXXXXXX43643754XX Network Interfaces are providing the connection to the same network, my home local network (LAN), managed by my Wi-Fi router. Just the origin of eno1 is the cable (Ethernet), and the origin of wlxXXXXXX43643754XX is the Wi-Fi USB adapter, so a wireless connection. Why do they have different IP addresses even if they are "attached" to the same network?

All other devices connected to my house Wi-Fi have private IP addresses - like the laptop at 192.168.1.A, the phone at 192.168.1.B, the PlayStation at 192.168.1.C, etc. So why does my single desktop PC have two addresses in the same network? Does this cause confusion?

My PC can connect to a home local network (and to any network that is not virtualized by the PC itself=existing only "inside" the PC) through a NIC—Network Interface Card/Contrtoller. This is a hardware component that is often integrated into the motherboard.

If you’re using a laptop that doesn’t have a port for an Ethernet cable, it most likely doesn’t have a NIC either. So, even if you imagine buying something like a USB-to-Lightning-to-Type-C adapter just to connect an Ethernet cable to port of Type C, it won’t work. This is because it’s not just about the "shape" of the cable, but the capability to process the type of communication that comes through it - and this capability is granted by NIC.

The solution is to purchase a proper USB hub with a built-in NIC and Ethernet port. The laptop example is a good one because it shows that an Ethernet cable is not the cornerstone of connectivity to networks. Such laptops indeed connect wirelessly, if a network of interest allows so. In this case, they do not use a default NIC but rather a WNIC—a Wireless Network Interface Card.

A WNIC in a desktop computer often needs to be purchased separately—either as a PCIe network card (to be attached to the motherboard's PCI slot) or as part of a USB Wi-Fi adapter.

In my case, I have them both, integrated NIC and WNIC of WiFi USB adapter:

#list of Ethernet controllers
$ lspci | grep -i ethernet
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Controller
#list of WNIC
$ lsusb
Bus 002 Device 002: ID 1234:5678 TP-Link 802.11ac NIC

My Wi-Fi router perceives connection from each (W)NICs as a separate initialization of a connection and assigns a distinct private IP address to each. That’s why my PC, which has two network interfaces, ends up with two IP addresses—one for each (W)NIC. The router cannot identify that both NICs physically belong to the same device (my PC) because the physical identifiers are the NICs themselves (not a CPU, not a motherboard ecc)!

Each NIC has its own unique MAC address (Media Access Control address), which serves as the unique permanent identifier. Not only network interfaces, but every piece of network-connected hardware, MUST HAVE a unique MAC address in a network. This is different from an IP address because a MAC address is permanent: every NIC has one and only one MAC address, hardcoded by the manufacturer (though, sometimes it is possible to change it). You can see the MAC addresses of your network interfaces using ip link or ip a commands—in the lines labeled link/ether.

While MAC addresses are permanent, an IP address is temporary and can change quite often - every time a device connects to a network, it is possible that WiFi router can assign to it a new private IP address (the probability of it depends on the network configuration and the device’s own settings).

You can think of a MAC address as a device’s permanent name, while an IP address is more like instructions for other devices on how to communicate with it. For example, your device might always be MAC number A, but at any given moment, it can be located at IP address B or IP Address C ecc.

Now that I’ve covered MAC addresses, it’s time to explain IP addresses.

➁.➂ IPv4 vs IPv6 addresses and NAT

To see the IP addresses of your PC's network interface(s) (roughly speaking Private IP address(es) of your PC) you can execute ip a command. You can scroll up and take another look at my output of this command. Now you know that the link/ether field tells you the MAC address(es) of network interfaces. If you’re familiar with IP addresses, you can guess that inet field exactly contains IP address information (the address though which other devices on the same network can communicate with PC). However, 'inet' field is not the one that shows you IP address of a network interface. inet6 field (if you have it) shows you another address and it is also IP address (even if it looks more like a MAC address)!

My PC's eno1 interface (Ethernet) has inet 192.168.1.X/24 and inet6 fe80::278e:1234:l678:123/64; wlxXXXXXX43643754XX interface (wireless) has inet 192.168.1.Y/24 and inet6 fe80::kre7:b5b5:z8y6:987/64; virbr0 virtual network interface managed by libvirt has only inet 192.168.122.1/24.

Both inet and inet6 fields of ip a output contain completely valid IP addresses - it’s not like what you see in inet6 field is just an encrypted, altered, or transformed version of inet field value, nor does it indicate a completely different network. The value of inet field is the IPv4 address, while inet6 displays the IPv6 address.

IP stands for Internet Protocol and it has two major versions: IPv4 and IPv6. IPv6 is the newer, more advanced version, offering enhanced features, greater capabilities, and significant potential for addressing future network demands.

In a previous article, I already touched on the topic of IPv4 vs. IPv6 differences. I’ll share the most relevant takeaway as it relates to this article:

A unique public IP address is a scarce resource! Actually, a unique public IPv4 address is in deficit. Internet Protocol version 4 (IPv4) forms the foundation of most Global Internet traffic today. An IP Address represented under IPv4 is composed of four sets of numbers ranging from 0 to 255, separated by periods(.).
If you do the straightforward math - total four numbers in an IPv4 address; each number can be in range between 0 and 255 (256 possible values) - 256 * 256 * 256 * 256 = 4,294,967,296 total addresses.

The first distinction is that IPv6 addresses are not just numerical; they are alphanumeric. IPv6 can provide many more unique addresses than the IPv4 system (340 trillion trillion trillion unique addresses vs 4 billion). But this is not the only difference. IPv6, as a protocol, is more modern and from the beginning was designed to be more secure.

It is not like IPv4 was designed without security in mind, just when it was created, the internet had far fewer users, and the number of devices worldwide was significantly lower. Let me illustrate how it affects the security mechanisms.

Security mechanisms, such as encryption, inevitably introduce additional computational load. For example, imagine my PC is communicating with your PC using an encryption mechanism. All the packets we exchange are heavily encrypted, and our devices communicate using public IP addresses, which act like the destination addresses for our data packets (like mailing letters). My device with its public IP address communicates with your device at its public IP address.

This scenario demonstrates the end-to-end principle of communication: my device can communicate with yours securely, with security mechanisms implemented directly in the communicating end nodes (our PCs, that have encryption/decryption keys). The intermediary nodes (like gateways and routers) don’t take part in our secure communication.

So, the Gateways and Routers are bad? Not at all! However, in this example, gateways and routers can A) introduce computational overhead B) make some encryption mechanisms impossible.

Getting closer to the topic... This idealized scenario of end-to-end communication is possible today, but only if we use IPv6 and our devices are properly configured to support it. Why? Because IPv6 adreesses poll is really huuuge, so every device can have a public IPv6 address, enabling direct communication between them without needing intermediary translation.

Each packet that we exchange in my example as any network packet consists of control information (headers) and user data (payload). Control information provides data for delivering the payload (e.g., source and destination network addresses, error detection codes, or sequencing information).

So, in case of IPv4. My PC sends to my WiFi router a packet that it is meant for you. But my PC does not have a Public IP address! So the source IP address in the headers is abracadabra for the web scope and it needs to be_ translated_, according to some rules. This job is done by NAT - Network Address Translation mechanism of my WiFi router. And here it is why this intermediary node in our communication is a baddie - NAT introduces processing overhead because it rewrites packet headers for every packet that passes through the router. And this happens not only on my side, but also on your side!

Some encryption techniques are meant for end-to-end communication only with consistent IP addresses - so they are not possible if NAT (in its default form) is in the middle.

With IPv4, gateways, routers become unavoidable due to the scarcity of public IPv4 addresses, and so does NAT. I started introducing NAT right here; however, I will elaborate more on this in the next sections, as the libvirt's DEFAULT network is based on the NAT mechanism.

_A little thought experiment: how do you think the internet would be if IPv6 were fully adopted and replaced IPv4? What would happen to the prices for internet service? Would the speed of the internet change? PS. you can check out what is CGNAT :) _

In this article, I will use IPv4 addresses for any network configuration. Unfortunately, Internet Protocol version 4 (IPv4) still forms the foundation of most global internet traffic today. Plus, as I stated earlier, this article is focused on creating a local network rather than configuring any VM to be remotely reachable from the outside.

The scope of IPv4 reserved addresses for private networks is quite large for a home setup, so let's move to it.

➁.➃ IPv4 address ranges reserved for private networks

According to standards set forth in Internet Engineering Task Force (IETF) document RFC-1918 , the following IPv4 address ranges are reserved by the IANA for private internets, and are not publicly routable on the global internet:
10.0.0.0/8 IP addresses: 10.0.0.0 – 10.255.255.255
172.16.0.0/12 IP addresses: 172.16.0.0 – 172.31.255.255
192.168.0.0/16 IP addresses: 192.168.0.0 – 192.168.255.255
Note that only a portion of the “172” and the “192” address ranges are designated for private use. The remaining addresses are considered “public,” and thus are routable on the global Internet.(Source)

Key takeaway is that there is set of IPv4 IP addresses that are reserved for private networks. The reservation ensures that there will be no conflicts with global - public IP addresses.

➁.➄ IPv4 addresses structure and CIDR

All IPv4 IP addresses has the same structure - 4 numbers divided by dots: x.x.x.x. The trailing slash with a number after it is not a part of IP address, it is CIDR (Classless Inter-Domain Routing) notation.

Take a detailed look at this again:

10.0.0.0/8 IP addresses: 10.0.0.0 – 10.255.255.255
172.16.0.0/12 IP addresses: 172.16.0.0 – 172.31.255.255
192.168.0.0/16 IP addresses: 192.168.0.0 – 192.168.255.255

The largest network range (with the most IP addresses) is represented by 10.0.0.0/8. The CIDR number after the slash (e.g., "/8") is not random; it refers to the number of bits allocated for the network portion of the IP address.

A "/8" means that the first 8 bits are reserved for the network, leaving 24 bits for the host part. This allows for 16,777,216 available addresses for devices (hosts) within the network. The smaller the CIDR number (like "/8"), the larger the number of available IP addresses, because fewer bits are used for the network portion, and more are left for devices.

The IP IPv4 address is 32 bits long, divided into four groups (octets) of 8 bits each. For example, 192.168.1.0 is written as:

192    .    168    .     1    .    0
11000000.10101000.00000001.00000000

Each number is an 8-bit block (each block represents one of the four octets). The CIDR prefix tells how many of these 32 bits are used for the network portion.

Lets return to my outputs from ip a:

eno1: inet 192.168.1.X/24 brd 192.168.1.255
wlxXXXXXX43643754XX: inet 192.168.1.Y/24 brd 192.168.1.255
virbr0: inet 192.168.122.1/24 brd 192.168.122.255

eno1 and wlxXXXXXX43643754XX network interfaces connect my PC to the local network managed by my WiFi router. Router assigned them private IP address 192.168.1.X and 192.168.1.Y. These IP addresses were vacant, so they were assigned by router's DHCP (Dynamic Host Configuration Protocol). CIDR /24 tells me that in this local network there are 256 "spots" for devices:

/24 means the first 24 bits are for the network. This leaves 8 bits for the host (the devices within the network). Network bits: 24 bits; host bits: 8 bits; 2^8 = 256 possible IP addresses in this network (but in practice, some are reserved for special purposes).

The virbr0 network interface is a different case. First, it is managed by libvirt/QEMU, as I mentioned, not by my WiFi router, and it is a virtual bridge. The Network to which this network interface connects also has a CIDR of /24, meaning there are 256 available addresses.

However, remember the last information I shared in the previous part of this series on virtualization? RECAP: I created a VM connected to the DEFAULT network, and I mentioned that even though it can reach the internet, no device connected to my local home network can access this VM via SSH, ping, or anything else (except HOST!). This is because any 192.168.122.X address is not part of my local home's 192.168.1.X/24 network! The 192.168.1.X/24 range covers addresses from 192.168.1.0 to 192.168.1.255, but any 192.168.122.X address is outside of that range. As a result, there’s no connection, no communication.

After this long introductory session, let’s get back to virtualization with virsh, QEMU, and KVM, focusing on network configurations.

➂ Libvirt's DEFAULT Network: about NAT mode

libvirt uses the concept of a virtual network switch. The network interface you saw in my ip a outputs in the previous section, called virbr0, is nothing more than a virtual network switch automatically created and managed by libvirt.

➂.➀ About virtual network switches

I don’t know if you’re familiar with physical network switches, but these guys look like this (not so friendly for trypophobic folks, hehe):

Getting closer to understanding networks: my PC has an Ethernet cable that’s plugged into the Wi-Fi router. But what if I plug it into my laptop instead? Haha, I can’t do that because my laptop doesn’t have an Ethernet port. However, if I could, that connection would create a small network of two devices (my PC and my laptop), allowing them to communicate with each other.

But what if I wanted to attach something else—a second laptop? It would be pretty hard to do with the scarcity of Ethernet ports (even if both laptops had one Ethernet port).

If I had a network switch, though, I could plug in all the devices I wanted—up to the number of available ports on the switch. The network switch would handle the communication between them by forwarding packets between connected devices!

So, the virtual network switch works kinda the same way, just for VMs. The default virbr0 virtual network switch is used when VMs are connect to the DEFAULT network. This virtual network switch enables them to communicate easily with each other.

Here is the DEFAULT network:

$ sudo virsh net-list --all

 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   no          yes

This DEFAULT network operates in Network Address Translation (NAT) mode (the one I introduced above discussing IPv4 vs IPv6).

By default, a virtual network switch operates in NAT mode (using IP masquerading rather than SNAT or DNAT).
This means any guests connected through it, use the host IP address for communication to the outside world. Computers external to the host can't initiate communications to the guests inside, when the virtual network switch is operating in NAT mode. (Libvirt: Virtual Networking)

DEFAULT network in the NAT modeLibvirt documentation

Libvirt's documentation specifies, that the NAT is set up using iptables rules.

And here we arrive at another important player in the whole networking process – iptables. It is often perceived juts as a tool that protects your system from unauthorized access/traffic based on some rules. However, the reality is that it is much, much more than that, as it can also be used for traffic manipulation, forwarding, NAT (Network Address Translation), and more.

Iptables provides packet filtering, network address translation (NAT) and other packet mangling.
NOTE: iptables was replaced by nftables starting in Debian 10 Buster. (Debian Wiki: iptables)

I am using Debian Sid, so I definitely have nftables and not the legacy iptables.

Is nftables just a modern version of iptables with fixed vulnerabilities, faster performance, and so on, but still it is just theiptables under the hood? NO. They are two different frameworks designed to do the same job—'mangling' network traffic. However, think of 'different frameworks' in this context as you would if you have experience with Python: it's like TensorFlow and PyTorch. In web development, it's like React and Angular. You cannot write a neural network using PyTorch and then expect to just copy-paste the network source code into TensorFlow and have it work. The same goes for nftables and iptables. They are different, with different syntax, and different logic, especially when it comes to IPv6 traffic.

It's better not to mix the two, thinking, 'Oh, for an issue Y I'll write and add some rules in iptables, but then for the issue X I found a tutorial for nftables, so I'll add rules this way'. (Played Witcher 3? Remember what happened to Geralt when he was courting both Triss and Yennefer? Well, the same can happen to your network traffic if you start playing around with different tools for network traffic management)

However, even if you do so (use both the nftables and the legacy iptables tool at the same time), Debian has you covered in a certain way. First, the iptables utility is not installed on the system by default. If it is installed, the iptables utility will, by default, use the nftables backend. But, again:

Should I mix nftables and iptables/ebtables/arptables rulesets?
No, unless you know what you are doing. (Debian Wiki: nftables)

➂.➁ Libvirt wants `iptables`, Debian has `nftables`: what to do?

As I mentioned before, libvirt's DEFAULT network functions in NAT mode, and this NAT mode is defined using iptables rules. So, when you installed libvirt tools, it most probably pulled in iptables as a dependency. Indeed:

$ aptitude why iptables
i   libvirt-daemon-system          Depends libvirt-daemon-driver-nwfilter (= 10.10.0-3)
i A libvirt-daemon-driver-nwfilter Depends iptables

But I already discouraged you from using iptables, hehe. So, what's the plan? First, most likely, you don't even have nftables up and running yet :D. Because if you did, and it was running as a systemd service, you would have encountered some troubles starting VMs with the DEFAULT network. I'll show you why:

#if for some reason you do not have nftables installed:
# $ sudo apt install nftables
$ sudo systemctl status nftables
#is it active? No? Then, start it
$ sudo systemctl startnftables
# and ENABLE it so it will start on the Boot
$ sudo systemctl enable nftables.service

FYI: Take a look at the nftables ruleset "in use" using command sudo nft list ruleset when nftables.service is stopped and when it's started (or before starting it and after). Compare the two and try to find libvirt's NAT configuration :).

#And now I bring UP DEFAULt libvirt network
$ sudo virsh net-start default
error: Failed to start network default
error: internal error: Failed to apply firewall command 'nft -ae insert rule ip libvirt_network guest_output iif virbr0 counter reject': Error: Could not process rule: No such file or directory
insert rule ip libvirt_network guest_output iif virbr0 counter reject

OOPS! I broke everything. Libvirt wants iptables, not nftables.
Whatever wants Libvirt, but in Russian there is saying: "eat what is given". There’s a legit solution to make libvirt work well with nftables. And it’s not just some workaround.

Maybe you are familiar with UFW—Uncomplicated Firewall—this tool is built on top of iptables, making it easier to manage iptables rules. You write simplified rules, and it translates them into iptables rules in the underground. A similar interface also exists for nftables! It’s called firewalld.

First, let’s clarify why this isn’t just a workaround that adds extra software to your system just to make something work for VMs:"

You should consider using a wrapper instead of writing your own firewalling scripts. It is recommended to run firewalld, which integrates pretty well into the system. See also https://firewalld.org/ (Debian Wiki: nftables)

And...:

The firewalld software takes control of all the firewalling setup in your system, so you don't have to know all the details of what is happening in the underground. There are many other system components that can integrate with firewalld, like NetworkManager, libvirt, podman, fail2ban, docker, etc.(Debian Wiki: nftables)

So, firewalld, besides all its perks in firewall configuration and network traffic management, is the bro that will make libvirt function correctly with nftables. It will ensure that all the NAT rules, written with love by the libvirt devs, are active. That means the DEFAULT network based on NAT rules will work again.

$ sudo apt install firewalld
$ sudo systemctl start firewalld
$ sudo systemctl enable firewalld

#have a look on the new ruleset firewalld brought with it to nftables
$ sudo nft list ruleset

What about libvirt's ruleset for NAT?

$ sudo nft list ruleset | grep libvirt

#nothing 

#now i start DEFAULT network
$ sudo virsh net-start default
Network default started
$ sudo nft list ruleset | grep libvirt
        iifname "virbr0" jump mangle_PRE_libvirt
        iifname "virbr0" jump nat_PRE_libvirt
        iifname "virbr0" oifname "virbr0" jump nat_POST_libvirt
...

$ sudo nft list tables

table inet filter
table inet firewalld
table ip libvirt_network
table ip6 libvirt_network

So, there's a separate "table" for libvirt_network. This table is managed by libvirt, and libvirt fills this table with rules when its virtual network switch becomes active (virbr0). This happens only if at least one network using the virtual bridge virbr0 is started.

➂.➂ DEFAULT Libvirt's network: what's under the hood?

As I mentioned before, the libvirt DEFAULT network's NAT is set up using iptables rules. And in my case Libvirt is forced to use what is available on my Debian - nftables. So, if I explore existing rules in nftables, I for sure should find there Libvirt's default network rules, according to which traffic circulate between VMs and FROM VMs TO the "outside world".

And here's how the network magic happens for VMs: when they interact with each other and with the host, and access the internet to fetch whatever they're commanded to:

$ sudo nft -a list ruleset
table ip libvirt_network { # handle 6
    chain forward { # handle 1
        type filter hook forward priority filter; policy accept;
        counter packets 0 bytes 0 jump guest_cross # handle 7
        counter packets 0 bytes 0 jump guest_input # handle 5
        counter packets 0 bytes 0 jump guest_output # handle 3
    }

    chain guest_output { # handle 2
        ip saddr 192.168.122.0/24 iif "virbr0" counter packets 0 bytes 0 accept # handle 13
        iif "virbr0" counter packets 0 bytes 0 reject # handle 10
    }

    chain guest_input { # handle 4
        oif "virbr0" ip daddr 192.168.122.0/24 ct state established,related counter packets 0 bytes 0 accept # handle 14
        oif "virbr0" counter packets 0 bytes 0 reject # handle 11
    }

    chain guest_cross { # handle 6
        iif "virbr0" oif "virbr0" counter packets 0 bytes 0 accept # handle 12
    }

    chain guest_nat { # handle 8
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 192.168.122.0/24 ip daddr 224.0.0.0/24 counter packets 1 bytes 40 return # handle 21
        ip saddr 192.168.122.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return # handle 20
        meta l4proto tcp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 # handle 19
        meta l4proto udp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 # handle 18
        ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade # handle 17
    }
}

➂.➃ Nftables rulesets: Libvirt Network's table and chains explained

Let’s break it down:

First, libvirt_network rules have their own table. In nftables, tables act as "containers" within the overall ruleset (to see the full list: sudo nft list ruleset). Tables contain chains, sets, maps, flowtables, and stateful objects.

Each table belongs to exactly one family. If you want to apply a specific set of rules to some network traffic, you must first define the table by specifying its type. The type determines that only traffic of this specific type will be filtered by the rules in that table.

For example, the table ip libvirt_network rules only filter IPv4 traffic/packets because its table is assigned the ip family:

table ip libvirt_network

A chain is essentially a list of rules. In the case of libvirt_network ruleset, everything starts with the forward chain, which is responsible for filtering incoming traffic packets.

table ip libvirt_network {
    chain forward { #rules}
        chain guest_output {#rules}
    chain guest_input {#rules}
        chain guest_cross {#rules}
    chain guest_nat {#rules}
}

  ______         A |-----------------|F
 |packet|  --->  C |  chain forward  |I ---|
  ¯¯¯¯¯¯         C |  type: filter   |L    | 
                 E |  hook: forward  |T    |
                 P |  policy: accept |E    |
                 T |-----------------|R    |
                     -----packet---->      | 
                                           |
                                Routing decision:
                     <---?---  | |----?----> 
             |-----------------| | -----------------|
             |                 | ?                  |
|------------------------|     | |    |------------------------| 
|    chain guest_input   |     | V    |   chain guest_output   |    
|    inbound packets     |     |      |    outbound packets    |
|   destined for guests  |     |      |       from guests      |   
|           ?            |     |      |             ?          |
|------------------------|     |      |------------------------| 
            |                  |                    |  
|-----------|------------|     |      |-------------|----------|  
|                        |     |      |                        | 
+ACCEPT             REJECT-    |      +ACCEPT             REJECT-
+If:              Anything-    |      +If:              Anything-
+outgoing             else-    |      +source IP:           else-
+interface:               -    |      +192.168.122.0/24         -
+virbr0                   -    |      +                         - 
+                         -    |      +incoming                 -
+connection               -    |      +interface:               -
+state:                   -    |      +virbr0                   -     
+established OR           -    |      +++++++++++++++++++++++++++ 
+related                  -    | 
+                         -    |      |--------------------------|
+destination:             -    |      |    chain guest_cross     | 
+in 192.168.122.0/24      -    |------|    traffic between       |
+++++++++++++++++++++++++++           |    guests on virbr0      |                                            
                                      |             ?            |        
                                      |--------------------------| 
                                      +ACCEPT              REJECT-
                                      +If:                No rule-
                                      +incoming interface:       -
                                      +virbr0                    - 
                                      +                          - 
                                      +outgoing interface:       -
                                      +virbr0                    -
                                      ++++++++++++++++++++++++++++ 

 ##########################################################  
 #                Routing decision is made!               #
 #                            IF:                         #
 #                 outbound packets from guests           #
 ##########################################################
    :==========:      ______                      :==========:
    :guest VM  : --> |packet|                     : Other VM : 
    :==========:      ¯¯¯¯¯¯                      :==========:
                        |                                  ^ 
                        V                                  |
 |------------------------|    |--IF traffic FROM:      NO NAT
 |                        |    |                    (NO masquerade)
 |   chain guest_nat      |--->| guest subnet TO:          ^
 |   type: nat            |    | (224.0.0.0/24) OR --------|
 |   hook: postrouting    |    | (255.255.255.255)
 |   priority: srcnat;    |    |==================================
 |   policy: accept;      |--->| IF traffic FROM:   
 |                        |    | guest subnet TO: ---------|
 | -----------------------|    | DIFFERENT subnet          V
                               |           MASQUERADE traffic (NAT)
                                                           |
                                          INTERNET <--------

I hope this ASCII scheme brought some clarity to how VMs attached to the DEFAULT libvirt's network communicate with the host, with each other, and with the outside world. Here’s some more info.

A network packet pops up on the virbr0 interface and then gets filtered (by chain forward) and routed or dropped - based on the nftables rules.

For example, when one VM sends a packet to another VM (if both are connected to DEFAULT network, it’s routed to that VM. The rules in chain guest_cross are triggered—there’s no rejection, so the packet is delivered without any special mangling, since it goes from virbr0 to virbr0 network interface. Chain guest_nat also participates, but NAT does not apply here because it’s an internal connection, and the rules say that when it’s from the same subnet to the same subnet, there’s no NAT.

Another example: the host sends a packet to a VM. This triggers the chain guest_input. The connection tracking state is established in this case (the VM has already initiated a connection to the host). And here it is! This is why you cannot connect to the VM remotely (even from another device connected to the same local home network as host and configured port forwarding!)—it will be new connection, not established!

Also, if you try to connect to the VM from a laptop on the same Wi-Fi as the host, it gets dropped because the VM is on a different network (remember, 192.168.122.x is not a part of 192.168.1.0/24 network!). But you configured port forwarding? Still nope. Packets will be dropped. Because again, it’s a new connection.

Yet another example: the VM wants to run sudo apt update && sudo apt upgrade. It needs to fetch data from the Debian repositories. This is governed by chain guest_output, so the rules are triggered and they don’t cause the drop of packets. Then the guest_nat chain is activated because traffic is going from the guest subnet to an external subnet. Without NAT, it wouldn’t work at all—the VM network is different from your local home network (WiFi), so they don’t directly communicate. They’re isolated from each other, and virbr0 doesn’t bridge these two networks!

To summarize: there are only two possible outcomes for the network traffic you initialize - FROM and TO your virtual machines that are attached to the DEFAULT network—it will either reach its destination (ACCEPTED) or be dropped (REJECTED).

I hope it’s clear when it comes to host-to-VM, VM-to-host traffic, and communication between VMs. 1) There are no specific nftables network filtering/mangling rules blocking it. 2) These communications are possible by the specifically configured and virtualized virbr0 virtual network switch.

However, HOW does traffic TO the internet—and especially FROM other networks (like your home LAN) work — can still be challenging to understand. NAT (Network Address Translation) might still appear like a black box. So, in the final section of this article, I’ll try to simplify things and explain it in detail.

Understanding NAT is crucial for all IPv4 traffic, as NAT has become a widespread networking configuration. This is due to the inherent limitations IPv4 faces in the modern world, making NAT a kind of symbiotic solution for IPv4 networks.

➂.➄ About how NAT works

Let’s get back to the NAT chain of the libvirt_network ip table. This chain doesn’t just apply filtering rules to network traffic—it handles postrouting rules, meaning it sees all packets after routing, right before they leave the local system.

For example, imagine your VM wants to connect to the internet to update itself when you run sudo apt update. This command fetches the data about the versions of your system’s packages against the repository versions. The packets from the VM are sent to http://deb.debian.org/debian/ (the Debian package repo source link).

These packets (fetching request) end up on virbr0, which acts like an airport for them. It decides if these “passengers” (packets) need to take an “international flight” (outside the guest VM network) or a “domestic flight” (within the guest VM network).

NB! I will cover DNS servers and gateways that act as routers in detail in the next article! For now, let’s simplify things (even though it’s not fully technically accurate) and say that the virbr0 virtual network switch handles this somehow, so I can focus on explaining NAT (otherwise I will never finish this article T_T).

Let's get back to the postrouting rules specified in the guest_nat chain and disaamble them to see how they worl

$ sudo nft -a list ruleset
...
chain guest_nat { # handle 8
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 192.168.122.0/24 ip daddr 224.0.0.0/24 counter packets 1 bytes 40 return # handle 21
        ip saddr 192.168.122.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return # handle 20
        meta l4proto tcp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 # handle 19
        meta l4proto udp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 # handle 18
        ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade # handle 17
    }
}
..

Handles are used natively by nftables to make it easier to reference specific rules when you need to modify or transform them, so they’re quite handy for me right now—I don’t have to retype rules.

Handles #20 and #21 aren’t what I’m looking for to illustrate NAT in action. These handles specifically ignore packets traveling within the same guest network.

But handles #19, #18, and #17 are exactly where NAT is in action! These three handles are categorized based on the communication protocol in use—TCP (#19), UDP (#18), and all other protocols (#17).

All three handles process traffic in the same way. The key word in all these rules is masquerade.

The term masquerade hints what happens to the source address of the network traffic—it gets "masked".

Let’s follow the network packets from a VM that executed sudo apt update. First, since network communication of this type uses the TCP protocol, I have to look into the rule of handle #19:

ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 -->  
--> This is like a programming `if` condition:
If the **s**ource **addr**ess (saddr) is within the 192.168.122.0/24 subnet 
AND the **d**estination **addr**ess (daddr) is outside of this subnet:
     masquerade to :1024-65535

Masquerade replaces the original source address with something else. In this case, it replaces it with the private IP address of the host. More specifically, it replace it with the private IP of the network interface with the highest priority (if there’s more than one, as in your case).

And what’s with the numbers after the colon? Those are ports—specifically the ephemeral port range (1024-65535). Why such a big range? And aren’t these ports busy on the host?

This port range is the ephemeral port range, which is designed to avoid conflicts with privileged ports (0–1023). Privileged ports are reserved for well-known system services, so this range ensures the masqueraded traffic doesn’t interfere with them.

Oh I left alone the packets from sudo apt update. Here they are: after masquerading:

The packets originally departed from the VM at 192.168.122.10 on port 80 (since it’s HTTP traffic)
They reached virbr0 (the virtual network switch).
virbr0 routed them to go outside the DEFAULT network (where VM belongs to).
Before leaving, they were masqueraded: each packet’s header was modified: the source address was replaced from 192.168.122.10:80 → to 192.168.1.5:12345. Here, 192.168.1.5 is the private IP address of the host (my PC), and 12345 is an ephemeral port assigned dynamically for this communication.

Here is the peculiar schema:

Source of Original Image

And that’s how NAT works. That’s all. Hahaha, just kidding.

That’s how the FIRST NAT worked. Now, these poor little packets have to "embark" on yet another "plane". They’re off to their next layover, where they’ll get NAT'd (masqueraded) AGAIN— new masks for everyone! This time, the layover is at the WiFi router.

The NAT’d packets leave the host with source address 192.168.1.5:12345. They arrive to my Wi-Fi router on the LAN side.
Wi-Fi Router NAT

The WiFi router sees packets from 192.168.1.5:1234 → to 123.123.7.132:80 (I am sooo bad, I do not know the IP address of Debian Servers & lazy to check).

NAT rewrites the source IP AGAIN from 192.168.1.5 to my router’s public IP (111.111.111.111) and change the port to another ephemeral one (54321).

NB! This is important! This is the cornerstone of weak routers (from the point of hardware):

Network packets don’t just travel to the Debian servers for a one-way trip; it’s always a round trip! At some point, response packets will come back. In the case of success, the VM expects to receive information about the versions of the packages it requested, to figure out what’s outdated.

Now, with all this masquerading, we end up in a situation much like those in movies—a classic masquerade ball drama (someone gets kissed because they were mistaken for someone else under their mask). To prevent these kinds of situationships, the WiFi router steps in as the responsible dude in charge. The router keeps track of who is going where, and if something comes back from "there," it makes sure it gets sent to the right who. To do this, router keeps updated a sort of table (thankfully it’s .xlsx). This table looks something like this:

|Private IP    | Public IP      | Source Port | Destination Port |
|192.168.1.5   | 123.193.7.162  | 54321       | 80               |
|192.168.1.20  | 111.153.6.132  | 35465       | 443              |
|192.168.1.11  | 125.161.7.162  | 34564       | 27017            |
|192.168.1.16  | 3.193.5.132    | 1224        | 22               |
.......................
#This table can be quite long if many devices are connected to the same WiFi and actively use Internet!

And then, the router keeps this information... in the table.

Here’s where an important point comes in: this table can grow very quickly. If your router is fast and powerful, no big deal—but if you cheaped out on it, you might run into trouble. A router that can’t handle more than a certain number of rows (based on its hardware limitations) will start to slow down significantly as the table grows beyond.

Again I left alone travelling network packets. Here they are:

The packets departed to the Debian server with my public IP and an ephemeral port: 111.111.111.111:54321.

The Debian server responds, and the response arrives from 123.123.7.132:80 to the router. The router then checks its table and says, "Ah, here it is!" It finds the matching entry in its tracking table and figures out which device the response is meant for. At this point, the router rewrites the headers of the packet AGAIN, removing its public IP address (111.111.111.111) and restoring the private IP and port from the table. Finally, it sends the packet back to my PC (the host) at 192.168.1.5:12345.

The host sees in the packets the destination 192.168.1.5:12345, checks its NAT table from the guest_nat chain, and recognizes this was originally from 192.168.122.10:80.

It rewrites AGAIN the destination from 192.168.1.5:12345 back to the VM’s IP and port: 192.168.122.10:80.

The packet is finally forwarded to the VM at 192.168.122.10:80.
The VM sees a response from http://deb.debian.org/debian/ to 192.168.122.10:80, completes the TCP handshake, and gets the repository data.

And now, that’s truly that's all! This is how NAT works. Enjoying IPv4? Still thinking IPv6 is difficult and scary?

For a little fun (and for little support for my emotional state after writing this article), try counting and writing in the comments how many times the network packets from the VM, headed to the internet, were rewritten along the way :).

Now after this TINY introduction to networking, it is time to get to the hands-on configurations. How can I make virtual machines accessible from other devices connected to the same LAN as the host—or even remotely? While I won’t be covering remote access here, making VMs accessible from other devices on the same home network can be quite convenient.

For example, I have a pretty powerful Desktop PC in terms of specs, and maybe sometimes I’m lazy and want to work from my laptop, which is like a toy in comparison to my PC. When I say I am lazy, I mean I don't want to sit properly at my desk; I want to loaf on the sofa with my laptop. However, I still want to use the resources of my PC. I can, of course, connect via SSH to my host machine, but it may be that I just want to connect to the VM with MongoDB to do something or check on it.

There are three main ways to do so:

Port forwarding & Custom NAT
Bridged networking (aka "shared physical device")
PCI Passthrough of host network devices

All of them I will cover in the next article.

Virtualization on Debian with virsh&QEMU&KVM — Installation of virtualization tools and first VM creation

Anna — Sun, 12 Jan 2025 13:00:56 +0000

In this article, I will not cover the basics of virtualization—what it is and when you might need it. This article is for those who are more or less familiar with the concept, but don’t know how to get started with it on Debian.

Here’s the road map for this article:

➀ Virtualization and self-hosting
➁ Virtualization as a tool for resource-Constrained application development
➂ Virtualization on Debian: how it works

➂.➀ CPU virtualization support
➂.➁ Hypervisor

➃ KVM & QEMU
➄ Libvirt
➅ Validation of virtualization tools installation

➅.➀ Important! qemu:///system vs qemu:///session

➆ Creation of first VM:

➆.➀ OS image file
➆.➁ Preparing storage
➆.➂ VM creation with virt-install

➇ Let the Networking begin!

➇.➀ Userspace (SLIRP or passt) connection
➇.➁ NAT forwarding (aka "virtual networks")

First, I will introduce my use case - why do I need virtual machines on my personal PC.

Long story short, I’m an "on-premise" girl (read "not very pro with cloud infrastructures"). I have an experience with dealing with on-prem infrastructure, and now I’m facing the need to deploy SOMEWHERE my personal project—a web app—so it sees the real world and real world sees it. And no, this isn’t a static website; it has a backend and a database. And hypothetically, some components will need horizontal scaling in the future.

➀ Virtualization and self-hosting

My personal PC isn’t bad at all in terms of specs—perfectly capable for development purposes and even, theoretically, for serving all the needs of my small app in production. However, hosting anything exposed to the web on a personal PC is out of the question. If your first thought is that the only obstacle is my PC needing to run 24/7, it is not about it. Using a machine that has some personal data for hosting of something exposed to the web is a VERY BAD IDEA. If you don’t understand why, you can check out this article.

If I create Virtual Machine(s) on my personal machine, and configure very well the networks, does that solve the problem? NO! And here’s why. The main impediment to hosting anything from "home"—even if you bought a proper server for it—is your router and internet provider. Is this about internet speed? Nope. It’s about...your public IP address.

Let's say you move into a new house, and it doesn’t have Wi-Fi. So you contact the internet providers in your city, check the prices, select the most advantageous offer, and… sign the contract. If you’re just an average user and didn’t specify otherwise, the contract gives you “internet for your house” from the chosen provider.

A technician arrives at the scheduled appointment, brings some cables, and a plastic box—which is the router. They deal with the cables, connect them to the router, hand you a manual along with the Wi-Fi network name and password, and voilà! You can connect all your home devices and enjoy browsing the web.

When you just use the internet, you’re most likely never even aware of your public IP address. But chances are, it’s not static at all. It changes periodically, and this is done by your internet provider—because that’s how they often manage their clients with “home” use.

When it comes to hosting something, like a website, even if you buy a domain name like my-cool-site.it, how will people find it? Who will bind YOUR PC WITH CODE of this site (where all the site’s needs and dependencies reside) to that domain? Domain name of your web app needs to be resolved in such a way that the correct IP address behind it is revealed.

Theoretically, you don’t even need to buy a domain name; your site can work perfectly fine with just an IP address like https://12.34.56.78/home. But it’s not a top if you want your site to be searchable on Google and not just accessible by people who already have the link.

If your internet provider changes your public IP address periodically, it’s like frequently moving houses. People trying to send you letters would still send them to your old address unless you keep updating them, and the letters would never reach you. The same logic applies to the hosting having a dynamic IP. You could, of course, manually update everything and rebind the domain to your new public IP address, but that’s hardly convenient.

If you want to have a static public IP address, you should contact your internet provider and find out the conditions under which you can get it. It will probably come with an increased payment for internet service. Is sticking a certain IP address to your Wi-Fi router that hard and does it require extra effort to keep it like this to cover "technical" costs? No. The increased payment is not even very related to the fact that the need for fixed public IP address can hint that the internet access is for business use, so it’s about earning something, and therefore why not to charge you more. Well... it’s because a unique public IP address is a scarce resource! Actually, a unique public IPv4 address is in deficit. Internet Protocol version 4 (IPv4) forms the foundation of most Global Internet traffic today. An IP Address represented under IPv4 is composed of four sets of numbers ranging from 0 to 255, separated by periods(.).

If you do the straightforward math - total four numbers in an IPv4 address; each number can be in range between 0 and 255 (256 possible values) - 256 * 256 * 256 * 256 = 4,294,967,296 total addresses.

So here, on the market for internet service, a basic economic rule comes into play: demand is growing with increasing digitalization around the world, but the supply is restricted by the very nature (mathematical) of the good (unique IPv4 address), so the prices for this good are increasing. In the next article, I will cover more details on IPv4, explain a bit about IPv6 (the solution for this IP deficit situationship), and also cover some interesting aspects of networking that are consequences of this IPv4 address deficit (NAT).

Plus, another obstacle for self-hosting is a router, provided by your internet provider. They often have very restricting measures in terms of incoming https/s traffic (it gets blocked), and those restrictions (thankfully) will impede any hosting attempts. I say thankfully, because if you truly do not understand how it works, it is better that these restriction, firewall rules, are up and protecting you.

However, keep in mind, hosting on the **same network **you use for any personal device is not a perfect idea if you are unable to configure all the security mechanisms, firewalls, and configure networks properly.

Summing this up, currently, it is not an option for me to "selfhost".

➁ Virtualization as a tool for resource-Constrained application development

So, virtualization is not a solution for my problem with deployment of the web-app. Then where it can be deployed? The cloud. I can choose a cloud provider, rent the instances that match my app's needs, configure them, and deploy my app. Simple, right? Well, not so fast—because every instance, every service, comes with a price. And those prices... For someone like me, who’s built a pretty powerful PC for around $1,000, seeing cloud pricing for "little server" instances can be a bit confusing. To give you an idea, you can explore pricing on AWS using their calculator. I’ll share some screenshots of EC2 instance pricing:

2 vCPUs, 4 GB of RAM, and storage for an additional cost. All yours for around $30 per month if you want to host something that has a server side operations.

Actually, 2 vCPUs, not CPUs. vCPUs stand for virtual CPUs because they aren’t real physical CPUs—they’re virtualized. And an EC2 instances are essentially just virtual machines.

Now as we are back to the word virtualization, let’s talk about my use case—my needs. When it comes to the development of my app on my PC, even though I could install everything needed directly (since I’m always using the same stack as a developer), it’s far from optimal. Why clutter my PC with installations os stuff like Nginx and MongoDB, leaving them hanging around unnecessarily when the project is finished?

To keep everything tidy for development purposes, virtual machines hosted on my PC is the great solution. However, the real issue with development directly on my PC is this: when I develop on a machine with 20 CPU cores of the latest generation, 64 GB of RAM, and 12 GB of GPU memory, how can I be sure that what I’ve developed will actually run on a small EC2 instances? Or more importantly, how can I evaluate the resource requirements for my app in general? (let's leave code-based evaluation aside for now)

This is where virtualization will really help me. I can evaluate my code’s performance right from the start by creating VMs with small resources attached and placing my app's components there!

Note for the Dockerists/Dockerphiles/Containerphiles

I can already foresee the "Gosh, just learn Docker—it’s easy! Developing on bare metal is dinosauric; containerization is the key!" argument. I have no doubt Docker can handle everything. In fact, I personally enjoy Docker Swarm quite a bit (can’t say the same for Kubernetes, though).

However, let’s not forget that Docker has under a virtualization technology. And as I mentioned earlier, EC2 instances are nothing more than virtual machines. So, when you spin up an EC2 instance, you’re essentially getting a VM—a virtual layer. Then, when you install Docker on top of that, you’re adding… yet another virtual layer! And all of this is happening on a modest machine with just a few CPUs and some RAM.

You know what happens when you pile on more and more virtualization layers? They take you farther and farther away from the bare-metal performance of the hardware.

And Kubernetes for small apps? That’s like using a bazooka to kill a fly. Sure, I know Docker apps can be deployed in various ways on AWS (not only on top of EC2), but that’s not the point. My small-scope web app doesn’t need any of the "perks" Docker van bring.

"with Docker, my app can run everywhere"—because it’s no longer tied to OS. But I don’t plan to run my app anywhere except on Debian. I know how my app component's VMs work; I will set them up myself and I will know exactly what’s there.
"Docker provides an isolated environment" Sure, but isolated from what? Separate VMs already provide plenty of isolation.

As for bundling and isolating software of different components, and managing version conflicts. If it is your primary need for Docker even in small projects...Naughty, naughty - did you give up on pure TypeScript/Python and relied on external libraries a lot? Not my case, by the way.

Why would one follow the containerization hype just because everyone else is doing it?

That said, I’m not completely throwing Docker out of my stack. But for me, dockerization is something I’ll consider only when everything else is ready.

Moreover, Docker/Containerization is as same easy as virtualization/Virtual Machines. Different sintaxis, some different concepts, but the logic behind is more or less the same. Docker is easy when it comes to setting up everything in a default way, but if you need something more advanced, most probably you will get very frustrated if you do not know anything about virtualization of hardware and virtual machines. Docker is not a rocket science at all for those who have some experience with Virtual Machines.

So let's start with virtualization on Debian!

➂ Virtualization on Debian: how it works

Virtualization process is happening under the "instructions" of your PC's physical CPU, so it is important that your CPU is supporting it. Yes, virtual machines can access (if allowed so) various hardware components of your PC, but it is exactly the CPU that is responsible for isolation of process running on guest VMs from the host (your physical PC). If your CPU supports the virtualization, first, it needs to be enabled on your PC:

➂.➀ CPU virtualization support

To know if you have virtualization support enabled, you can check if the relevant flag is enabled with grep. If the following command for your processor returns some text, you already have virtualization support enabled:
For Intel processors you can execute grep vmx /proc/cpuinfo to check for Intel's Virtual Machine Extensions.
For AMD processors you can execute grep svm /proc/cpuinfo to check for AMD's Secure Virtual Machine. (The Debian Administrator's Handbook: Virtualization)

In my case:

#this command is counting the number of times 'vmx flags' is mentioned in the output of /proc/cpuinfo. It is equal to 20, meaning that all my CPU cores support virtualization (I have 20 cores totale)
$ egrep -c '(vmx flags)' /proc/cpuinfo
20
#additional command
$ lscpu | grep Virtualization
Virtualization:       VT-x

If in your case the output of command grep vmx /proc/cpuinfo is empty, but your CPU is quite modern and is supposed to support virtualization, you’ll have to enter the BIOS during boot and enable it there. The steps to follow in BIOS are something like described in this guide. The interface of your BIOS depends on the brand of your motherboard, so if you are lost you’ll need to check instructions on how to enable virtualization on your PC in web.

All the CPU cores are ready to virtualize something! Who starts?

➂.➁ Hypervisor

A hypervisor! hypervisor is a bit generic term:

A hypervisor, also known as a virtual machine monitor (VMM) or virtualizer, is a type of computer software, firmware or hardware that creates and runs virtual machines. A computer on which a hypervisor runs one or more virtual machines is called a host machine, and each virtual machine is called a guest machine. (Wikipedia)

There are different types of hypervisors. To simplify their perception, they can be divided into two types (left and right images of the scheme below):

The second type of hypervisors might be familiar to you if you’ve ever used VirtualBox. It runs on top on Windows, just like any other app on Windows. On the other hand, Type 1 hypervisors run directly on bare metal. They often have their own OS, specifically tuned for virtualization purposes. And they are often used for enterprise scope.

I included Proxmox in the category of hypervisors type 2, because it comes as a Debian-based OS. To use it, you’ll need to replace your current desktop Debian with Proxmox OS. By the way, Proxmox is pretty great—easy to use and functionality-rich. I use this hypervisor for work, and I find it awesome. But it is not fully technically correct that Proxmox is hypervisor of type 1, as it is based on KVM&QEMU.

Let's get to KVM, that is schematized in the middle on the image above. Is it a hypervisor? Well... yes and no. The term "hypervisor" is generic, so you could call it that. But technically, KVM is a Linux kernel module. You don’t have to build it yourself—it comes shipped with the Linux kernel that is core part of your Debian, just like other kernel modules (for example, drivers).

In this article, I’ll be using KVM to set up virtualization tools on my PC. A popular alternative to KVM on Debian is Xen. Xen is a truly Type 1 hypervisor, even though it can also run alongside Debian OS for personal use.

Xen is a “paravirtualization” solution. It introduces a thin abstraction layer, called a “hypervisor”, between the hardware and the upper systems; this acts as a referee that controls access to hardware from the virtual machines. (The Debian Administrator's Handbook: Virtualization)

As Xen runs between the hardware and the upper systems it qualifies as a Type 1 hypervisor. VMware ESXi is another example of a Type 1 hypervisor.

I’ll be using a KVM-based virtualization setup instead of Xen—just a personal preference.

➃ KVM & QEMU

But what exactly is KVM, besides being a kernel module?

The Kernel Virtual Machine, or KVM, is a full virtualization solution for Linux on x86 (64-bit included) and ARM hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, which provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko. (Debian Wiki: KVM)

While KVM is providing most of the infrastructure that can be used by a virtualizer, but it is not a virtualizer by itself. Actual control for the virtualization is handled by a QEMU-based application.
Unlike other virtualization systems, KVM was merged into the Linux kernel right from the start. Its developers chose to take advantage of the processor instruction sets dedicated to virtualization (Intel-VT and AMD-V), which keeps KVM lightweight, elegant and not resource-hungry. The counterpart, of course, is that KVM doesn't work on any computer but only on those with appropriate processors.
Unlike such tools as VirtualBox, KVM itself doesn't include any user-interface for creating and managing virtual machines.(The Debian Administrator's Handbook: Virtualization)

Before, I showed how to check if the virtualization is enabled on your PC, the following command will show you if you have KVM kernel module and it can be used:

# checking for presence of KVM kernel modules
$ lsmod | grep kvm
kvm_intel             327680  0
kvm                   983040  1 kvm_intel
#Additional check with cpu-checker package:
$ sudo apt install cpu-checker 
$ sudo kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

As the Debian Manual on virtualization stated in the quote above KVM alone goes alongside with qemu for virualization porocesses.

QEMU (stands for Quick Emulator) is a generic and open source machine emulator and virtualizer.
When used as a machine emulator, QEMU can run OSes and programs made for one machine (e.g. an ARM board) on a different machine (e.g. your own PC). By using dynamic translation, it achieves very good performance.
When used as a virtualizer, QEMU achieves near native performance by executing the guest code directly on the host CPU. QEMU supports virtualization when executing under the Xen hypervisor or using the KVM kernel module in Linux. When using KVM, QEMU can virtualize x86, server and embedded PowerPC, 64-bit POWER, S390, 32-bit and 64-bit ARM, and MIPS guests. (QEMU Wiki)

Is it possible to virtualize using only KVM? THEORETICALLY yes, but KVM has neither GUI nor CLI, so one has to write code in C in order to virtualize something, but KVM alone will not emulate virtual CPUs or virtual RAM.
Can you use QEMU without KVM? Yes. QEMU alone can emulate a full system with its built-in binary translator Tiny Code Generator (TCG), which is purely emulated (= compute-intensive) and overall performance of fully emulated system can be slow. Thus, using QEMU without an accelerator is inefficient and generally best for experimental purposes (e.g if your CPU has an architecture A, but you’re curious about exploring how it all works on CPU architecture B). To “accelerate” the emulated system that run on the same architecture to the host’s one QEMU is using accelerators; and KVM is one of them. However, QEMU can use alternative accelerators like XEN (QEMU: Virtualisation Accelerators).

QEMU is not a software package that comes pre-installed on Debian, so you’ll need to install it manually. And here’s where confusion can arise. If you Google around, you’ll most likely find something like this for Debian-based systems sudo apt install qemu-kvm virt-manager bridge-utils. At first glance, this seems fine— you actually need QEMU to work with KVM. But here’s the tricky part: qemu-kvm isn’t even a real package. It’s a virtual package, which actually point to something else:

For my case, it’s fine because I plan to have all my guests using this architecture. But if you want to use a different architecture for your guest VMs, qemu-kvmwill bring in redundant package. There are other packages that will install QEMU besides qemu-system-x86, like qemu-system-arm, qemu-system-misc, qemu-system-ppc and qemu-system, which will bring you dependencies to virtualize/emulate various architectures with qemu.

I will install QEMU in this way:

$ sudo apt install qemu-system-x86

To shape your choice a bit, according to QEMU documentation on virtualization with KVM:

QEMU can make use of KVM when running a target architecture that is the same as the host architecture. For instance, when running qemu-system-x86 on an x86 compatible processor, you can take advantage of the KVM acceleration — giving you benefit for your host and your guest system (QEMU: features KVM)

Technically, if you try to create a guest with a CPU architecture different from your host machine’s CPU, KVM won’t be used, because, remember, QEMU can fully emulate machines (I haven’t tested this myself, though).

QEMU is installed, KVM is ready to virtualize, so everything is technically set up. However, the QEMU CLI syntax is far from simple and pretty particular. I would prefer to use a syntax which is more familiar to me. And this is where Libvirt will help me.

➄ Libvirt

Libvirt is collection of software that provides a convenient way to manage virtual machines and other virtualization functionality, such as storage and network interface management.
An primary goal of libvirt is to provide a single way to manage multiple different virtualization providers/hypervisors. No need to learn the hypervisor specific tools! (Libvirt FAQ)

Libvirt is a bundle of software that includes an API library, a daemon (libvirtd), and a command line utility (virsh).

Libvirt tools for management of virtual machines are 'virsh', 'virt-manager', and 'virt-install', which are all built around libvirt functionality.

virt-manager is a GUI tool for creation and management of VMs entirely through a graphical user interface (GUI) <-- can be a viable option in the beginning.
virt-install is a CLI tool that enables the creation and management of VMs via commands and parameters. If created VMs are supposed to have a display and graphical sessions, they can be accessed with virt-viewer (for a display).
virsh is a command-line utility that can do a lot of stuff - staring from very simple tasks like VM creation up to advanced virtualization. virsh works tightly with XML configuration files, that can be used to configure domains, virtual machine specs, networks ecc. Virsh gives an option to connect to existing VMs remotely via SSH <--I will be using this tool.

I install libvirt with the following command:

$ sudo apt install libvirt-daemon-system
$ systemctl status libvirtd
● libvirtd.service - libvirt legacy monolithic daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-01-10 22:39:29 CET; 2min 17s ago
#if not enabled in your case:
# $ sudo systemctl enable libvirtd

In this way, I will have libvirt-clients installed as well, as it is a dependency of this package.

Everything needed is supposed to be installed for now. First, I want to validate that everything is OK, and then I can proceed with first VM creation.

➅ Validation of installed virtualization tools

$ virt-host-validate
  QEMU: Checking for hardware virtualization  : PASS
  QEMU: Checking if device '/dev/kvm' exists  : PASS
  QEMU: Checking if device '/dev/kvm' is accessible : PASS
  ...
$ virsh version
Compiled against library: libvirt 10.10.0
Using library: libvirt 10.10.0
Using API: QEMU 10.10.0
Running hypervisor: QEMU 9.2.0

#command to check, which guest machines you can emulate with QEMU features you have installed:
$ virsh capabilities
...
  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'> <---
      <wordsize>32</wordsize> <---
      <emulator>/usr/bin/qemu-system-i386</emulator>
      ...
      <domain type='qemu'/> <---
      <domain type='kvm'/>  <---
    </arch>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'> <---
      <wordsize>64</wordsize> <---
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      ...
      <domain type='qemu'/> <---
      <domain type='kvm'/>  <---
    </arch>
  </guest>

➅.➀ Important! qemu:///system vs qemu:///session

Here is the command I want you to pay attention to:

$ virsh uri
qemu:///session

$ sudo virsh uri
qemu:///system

As you can see, there’s a difference between running virsh with sudo or without it. When you run virsh with sudo, it connects to the system libvirtd service, the one launched by systemd. libvirtd is running as root, so has access to all host resources. Daemon config is in /etc/libvirt, VM logs and other bits are stored in /var/lib/libvirt.
On the contrary, if you run virsh without sudo, it connects to qemu:///session, that is a session libvirtd service running as the app user, the daemon is auto-launched if it's not already running. libvirt and all VMs run as the user. All config and logs and disk images are stored in $HOME directory of a user. This means each user has their own qemu:///session VMs, separate from all other users. Details are taken from here.

If, for some reason, your output of virsh uri is empty, you can connect manually. And if you mess up between the session or system, you’ll be informed about it in the output.

$ virsh
virsh # connect qemu:///system
==== AUTHENTICATING FOR org.libvirt.unix.manage ====
System policy prevents management of local virtualized systems

#The correct way if you want to connect to user-space session:
virsh # connect qemu:///session
#The correct way if you want to connect to system wide session:
$ sudo virsh
virsh # connect qemu:///system

THIS INFO IS VERY IMPORTANT:

With qemu:///session, libvirtd and VMs run as your unprivileged user. This integrates better with desktop use cases since permissions aren't an issue, no root password is required, and each user has their own separate pool of VMs.
However because nothing in the chain is privileged, any VM setup tasks that need host admin privileges aren't an option. Unfortunately this includes most general purpose networking options.
The default qemu network mode when running unprivleged is usermode networking (or SLIRP). This is an IP stack implemented in userspace. This has many drawbacks: the VM can not easily be accessed by the outside world, the VM can talk to the outside world but only over a limited number of networking protocols, and it's very slow. (Source)

If this quote does not tell you much, here is the key takeaway: qemu:///session integrates better with desktop use cases. Any VM setup tasks that need host admin privileges aren't an option. This means that your VMs in the scope of qemu:///session will have ONLY general purpose networking options.

➆ Creation of first VM

NB! For the demonstration purposes, I will be creating first VM in the scope of qemu:///session. In this way, I will be able to demonstrate the constraints of created VM. Then, I will show you, how to move create VM under qemu:///system.

➆.➀ OS image file

To create my first virtual machine, which will, of course, be Debian Stable (Bookworm), I need an .iso file. I’ll go for the minimal netinstall image to keep the system tidy and install later only tools I will need.

$ cd #to teleport to $HOME directory
$ mkdir -p .local/share/libvirt/images/
$ cd .local/share/libvirt/images/
$ wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.9.0-amd64-netinst.iso

➆.➁ Preparing storage

Then, I want to specify the storage device I plan to use for all my virtual machines (as they will share portions of it). Since I manage all storage devices on my PC using LVM, the first step is to create a new logical volume using the available free space in my existing logical volume group.

$ sudo vgs
  VG            #PV #LV #SN Attr   VSize    VFree
  MY-vg       1   5   0 wz--n- <372.53g <129.02g

How will my virtual machines use the storage space I plan to create for them? Each machine will have one or more virtual disks, and these virtual disks are essentially "disk images." As everything is a file on Linux OS, these disk images are files. And they can have different formats. The .qcow format is a file format for disk image files used by QEMU. Its updated version, .qcow2, offers better optimization to the original .qcow. I can also create disk images for VMs in the .raw format. However, .qcow2 is generally more space-efficient and can be snapshot-ed and compressed.

So the task is the following: I need to create a .qcow2 disk image for my to be created VM. And I want to use available space in my logical volume group.

There are two options:

I can create a fairly large new logical volume to provide space for multiple virtual machines. In this case, I need to place a file system on top of the new logical volume, mount it, and then create .qcow2 disk images over it. Why? Because a logical volume without a file system is a single, contiguous block of storage. A single .qcow2 can occupy the entire block device, but there’s no mechanism to store multiple files on the same device unless a file system is present.
The second option is to create separate logical volumes sized to the needs of each virtual machine, with each logical volume fully allocated to a single .qcow2 image. By the way, you can also use physical partitions for your VMs, as logical volumes are just my preferred method of managing storage space. However, even if under each .qcow2 image there is its "personal" logical volume, this doesn’t mean you can expand the size of .qcow2 image by expanding
logical volume. No, not at all. If my VM runs out of space, I’ll need to attach a new "virtual disk", create an additional logical volume, create new .qcow2 image over it.... This quickly becomes a mess.

So, I prefer the first option: a single logical volume as a kind of storage pool for all my virtual machines disk images.

If your understanding of LVM terminology is a bit wobbly and you still confuse logical volume groups with logical volumes, I recommend to read this article.

#I create new logical volume with name 'virt-machines' inside of the existing volume group
$ sudo lvcreate -L 100G -n virt-machines MY-vg
# I create filesystem on top of it 
$ sudo mkfs.ext4 /dev/MY-vg/virt-machines
# I create a mounting point for it
$ sudo mkdir -p /mnt/virt-machines
# I mount it
$ sudo mount /dev/MY-vg/virt-machines /mnt/virt-machines
# i add automounting option on boot with by modifying /etc/fstab
$ sudo vim.tiny /etc/fstab
# I add this line 
/dev/mapper/MY--vg-virt--machines /mnt/virt-machines ext4 defaults 0 0
# to validate syntax:
$ sudo mount -a

Now, I can create a .qcow2 disk image in this directory. Since I plan to create and run VMs in my user space, I’ve given ownership of this directory to my user to avoid any permission issues later.

# I create a .qcow2 disk image
$ sudo qemu-img create -f qcow2 /mnt/virt-machines/deb-nginx.qcow2 10G

I have an .iso file from which the VM will boot (which will be attached as a virtual CDROM), and I have a virtual disk where the new system will be installed. Now, I just need to create a VM and allocate CPU cores and RAM to it. For the first VM creation, I will use virt-install instead of virsh to demonstrate the logic, and then proceed with XML configuration explanations. The virt-install CLI is part of the virtinst package and is not included in the libvirt-clients package. It needs to be installed separately.

➆.➂ VM creation with `virt-install`

I will be using the default options of the virt-install command, with two exceptions: --graphics none and --extra-args='console=ttyS0'. My VMs don’t need any graphical interface as they will not have display servers; I will access them via the console. Debian offers not only a graphical installer but also a terminal user interface (TUI) installer, which will guide through the installation process.

$ sudo apt install virtinst
$ virt-install \
  --connect qemu:///session \
  --name deb-nginx \
  --ram 4096 \
  --vcpus 2 \
  --disk path=/mnt/virt-machines/deb-nginx.qcow2,size=10 \
  --location $HOME/.local/share/libvirt/images/debian-12.8.iso \
  --os-variant debian12 \
  --graphics none \
  --extra-args='console=ttyS0'

If you encounter an error:

Traceback (most recent call last):
  File "/usr/bin/virt-install", line 6, in <module>
    from virtinst import virtinstall
  File "/usr/share/virt-manager/virtinst/__init__.py", line 8, in <module>
    import gi
ModuleNotFoundError: No module named 'gi'

Check if you are currently not in any active Python environment. I use anaconda, so the base conda environment is always activated. I just deactivate it with conda deactivate command before executing virt-install command.
You will see this:

And then installation of Debian (in TUI only) should pop up.

After installation is finished, VM is rebooted, I close active console ad reconnect with virsh.

$ virsh --connect qemu:///session console deb-nginx

Actually, everything is ready, VM is usable, so technically I can try to go into SSH...

➇ Let the Networking begin!

To SSH into the VM, I need to know the private IP address it was assigned (and I believe it was, as I expect some default network to have been configured and the VM joined it during the creation process via QEMU). I will leave the details about how SSH works from a networking perspective for now and will cover it in the next article of this virtualization series.

To find out IP of created VM:

root@deb-nginx:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN 
....
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether MA:CA:DD:RE:SS:VM brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp1s0
       valid_lft 77570sec preferred_lft 77570sec
    inet6 XXXXXXXXXXXXXX/64 scope site dynamic mngtmpaddr
       valid_lft 86291sec preferred_lft 14291sec
    inet6 XXXXXXXXXXXXXXXX/64 scope link
       valid_lft forever preferred_lft forever

The network interface is enp1s0, and the IPv4 address is 10.0.2.15. So, let's try ssh!

#from Host machine!
$ ssh 10.0.2.15

Nothing!

➇.➀ Userspace (SLIRP or passt) connection

However, as I mentioned earlier, qemu:///session VMs are primarily intended for desktop use, such as trying out a new distro. The network used under qemu:///session is somewhat primitive and restrictive—it does not allow incoming connections to the VMs and cannot be properly modified. For instance, you cannot configure more sophisticated network settings without sudo privileges to create network components like bridges, change their states, etc.

I can check which network is configured for this VM, and in general, I can review the full configuration of created VM. When I used virt-install, I simply passed some options during the creation process to specify how I wanted my VM, and those parameters were translated into configuration file. This file is much more detailed and "technical" than the option list I provided when I was creating VM. QEMU thoroughly translated my requirements into technical specifications, allocated the necessary hardware, and configured other components for my VM to work. The configuration format used by virsh for almost everything is XML.

$ virsh dumpxml deb-nginx

<domain type='kvm' id='3'>
  <name>deb-nginx</name>
  <uuid>0057aa74-392a-4b4b-ac89-8557d7b9312d</uuid>
  ...
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu> <--interesting
  <os>
    <type arch='x86_64' machine='pc-q35-9.2'>hvm</type>
  </os>
  <features>
   ...
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'/> <--interesting
   ...
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' discard='unmap'/>
      <source file='/mnt/virt-machines/deb-nginx.qcow2' index='2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    ....
--------------> HERE IT IS, NETWORK INTERFACE <-------------------
    <interface type='user'>
      <mac address='MA:CA:DD:RE:SS:VM'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    ....
 ---> O! Mouse and keyboard: <---------
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    ....
</domain>

Created VM has a network interface, only one. But in XML configuration file I see that this interface has type "user" <interface type='user'>, and normally the type should be "network".

I can check for existing alternatives (other network interfaces). Under qemu:///session, as expected, there is nothing:

$ virsh net-list --all

 Name   State   Autostart   Persistent
----------------------------------------

However, it seems that this userspace network thingy can now be configured quite extensively, because newer versions of libvirt have introduced more advanced features and options:

Since 9.0.0 an alternate backend implementation of the user interface type can be selected by setting the interface's subelement type attribute to passt. In this case, the passt transport (https://passt.top) is used. Similar to SLIRP, passt has an internal DHCP server that provides a requesting guest with one ipv4 and one ipv6 address; it then uses userspace proxies and a separate network namespace to provide outgoing UDP/TCP/ICMP sessions, and optionally redirect incoming traffic destined for the host toward the guest instead.(Libvirt: Userspace connection

However, configuration of userspace connection is beyond the scope of this article (and the next article on networking as well). For my use case, I don’t actually need port forwarding—I need something different.

➇.➁ NAT forwarding (aka "virtual networks")

However, qemu:///system has one default interface (NAT):

 $ sudo virsh net-list --all

 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   no          yes

So, I just recreate a VM into qemu:///system scope and this VM will be using by default this 'default' network.

First, I have to destroy and undefine VM in qemu:///session.

$ virsh destroy deb-nginx
$ virsh undefine deb-nginx
#(optionally, recreare disk image)
$ sudo rm /mnt/virt-machines/deb-nginx.qcow2
$ sudo qemu-img create -f qcow2 /mnt/virt-machines/deb-nginx.qcow2 10G
#IMPORTANT! Start dafault network if it is not started yet
$ sudo virsh net-start default

$ sudo virt-install \
  --connect qemu:///system \
  --name test \
  --ram 4096 \
  --vcpus 2 \
  --disk path=/mnt/virt-machines/deb-nginx.qcow2,size=10 \
  --location /var/lib/libvirt/images/debian-12.9.iso \
  --os-variant debian12 --graphics none \
--extra-args='console=ttyS0'

Please note where I placed the ISO file. If you place it inside /etc/libvirt/in some folder, as it might seem like the right place, you could encounter a weird and misleading error, such as:
error: internal error cannot load AppArmor profile 'libvirt-9cb01efc-ed3b-ff8e-4de5-7227d311dd15'.

If you put the ISO file somewhere under your $HOME directory, you might see a warning like this:
WARNING /home/..../debian-12.8.iso may not be accessible by the hypervisor. You will need to grant the 'libvirt-qemu' user search permissions for the following directories: ['/home/...', '/home/....local', '/home/.../.local/share'].

Both errors are related to the fact that the VM creation process cannot access the ISO file.

Meanwhile I proceed with new Installation via TUI. I setup LVM on this VM, and I put /var on separate logical volume, because this VM is meant for NGINX and NGINX can be very talkative in its logs, especially if configured badly. If you do not know how to do it, refer to this article. I also installed SSH server, so I can ssh into this VM from host.

NB if you have ufw up! During installation, Debian should auto-configure the network. If it fails and you see this:

...disable ufw temporarily for the process of installation, then enable it again afterward. It's not the best solution, the best approach is to adjust ufw rules so it doesn't block DHCP requests and DNS resolution, which libvirt uses to configure the VM's network through the default NAT setup.

When Installation is completed, I reopen the console with virsh and login into the freshly created VM. First, lets I check connectivity, disccover the IP address, try to ssh from host:

$ sudo virsh console deb-nginx
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=112 time=15.5 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=112 time=17.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=112 time=16.3 ms
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP>
.....
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether XXXXXXXXXXXXXXXX brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.125/24 brd 192.168.122.255 scope global dynamic enp1s0
       valid_lft 3217sec preferred_lft 3217sec
    inet6 XXXXXXXXXXXXXXXX scope link
       valid_lft forever preferred_lft forever

So, the IP is 192.168.122.125.
*NB! If you try to execute ssh 192.168.122.125 from the host, the login will fail because you will automatically be attempting to ssh as root, and root login via ssh is disabled by default on Debian.
*

I do:

$ ssh user@192.168.122.125
->yes
user@192.168.122.125's password:

So, what can I do from this VM from the network standpoint:

I can access the Internet (e.g., run apt update and apt upgrade).
I can SSH into this VM from the host machine.

What I cannot do:

I have a laptop connected to the same home network as my PC (via WiFi). Can I SSH into this VM from the laptop? No. This is why:

$ ip a
.....
5. virbr0 ...
  inet 192.168.122.0/24

You may have many questions and few answers about network configuration, but this article is already quite long, so I’ll move the networking setups to the second part of this series

Debian 12 … is amazing! How to: Create your custom codehouse #6 [Giving Voice to Debian: Wireless Audio Devices configuration]

Anna — Mon, 30 Dec 2024 12:34:07 +0000

In the previous article, I demonstrated all the steps required to set up a custom UI from scratch, including the display server, window manager, status bar and additional useful apps. Additionally, I shared my choices for fundamental apps, such as a terminal emulator and a browser.

Now, it’s time to complete the setup. What I configured previously was primarily focused on output devices—like the monitor, which handles what you see—and input devices like the keyboard and mouse, which allow interaction with applications. However, input and output audio devices remain untouched (in my case, a Marshall wireless Bluetooth headset with both a microphone for input and speakers for output).

My input/output audio device is only one—a Marshall IV headset (which I use mostly in wireless Bluetooth mode, but it also has a wired option). I don’t have separate speakers or a microphone. So, the first thing I need to ensure is that my Bluetooth dongle is working properly. I have written a detailed article dedicated to troubleshooting Wi-Fi and Bluetooth devices, so if your Bluetooth device gives you any troubles, please refer to that article.

My Bluetooth dongle works perfectly on Debian—truly plug and play. There’s no need for configuration, installation of drivers/firmware, or any tweaks; it just works. This Bluetooth dongle is from the brand Edimax, purchased on Amazon for around $10 (Edimax BT-8500). It supports the Bluetooth 5.0 protocol.

Therefore, my Debian system is ready to be configured to use my headset seamlessly for audio playback and, if needed, microphone input.

Here’s the road map for this article:

➀ Sound on your PC: where it starts (hardware-side)?

➁ Sound on your PC: where it starts (software-side)?

➂ ALSA, PipeWire or Pulseaudio: what is better*? (*trick question)

➃ Bluetooth devices management tool

➄ PipeWire vs Pulseaudio

➅ PipeWire: Installation

➆ About XDG Desktop Portal

➇ Pipewire audio profiles: Understanding the difference between device profile headset-head-unit and a2dp

➀ Sound on your PC: where it starts?

Let’s start by understanding what is the crucial component of your PC which determines whether you can enjoy any audio or not. It’s not just about having speakers/microphone, a headset, or anything else that can actually play/register sound. Your PC needs a hardware piece capable of processing audio at a very low level. That job is done by sound cards.

A sound card is a computer expansion card that can input and output sound under program control. (Source)

If you have a laptop or a PC—or if you built a PC manually and you are worried that you might have forgotten some piece—there’s nothing to worry about in 99% of cases; your machine has a sound card. Modern machines typically have on-board sound cards integrated into the motherboard.

Zoomed area of sound card on motherboard with 3 audio jacks

Depending on your motherboard’s specifications, the quality of the integrated sound card will vary, and that directly affects the quality of the sound your PC can produce or record. Some people who work with professional audio may find the capacities of integrated sound cards unsatisfactory, so they might opt for an additional sound card that can be attached via PCIe or USB (not all motherboards have PCIe slots for sound cards, though). However, for everyday use, an integrated sound card is usually very sufficient.

$ lspci -d ::0403 #::0403 here means Audio device subclass
00:1f.3 Audio device: Intel Corporation Raptor Lake High Definition Audio Controller (rev 11)
01:00.1 Audio device: NVIDIA Corporation GA106 High Definition Audio Controller (rev a1)

The first audio device is my motherboard’s integrated sound card. But what about NVIDIA Corporation High Definition Audio Controller? Why is it there? Can I use it?

Nvidia High Definition Audio allows NVIDIA GPU to transmit audio to any display (monitor or TV) with built-in speakers, provided it has an audio-capable connector (most commonly HDMI or DisplayPort). By using Nvidia HD Audio, you can enjoy various uncompressed and lossless audio formats, which can significantly enhance your overall listening experience—especially if you’re using modern displays with integrated speakers.

However, my monitor is pretty simple and it does not have any audio device. So, I will be using on-board soundcard.

➁ Sound on your PC: where it starts (software-side)?

As with any hardware device, a sound card needs firmware and drivers to function. On Debian, you usually don’t have to worry about these, except in cases where something is messed up with the kernel modules—this can often happen if Debian is running on a virtual machine. Yes, like most drivers, sound card drivers are part of Linux kernel, so it’s the Linux kernel that provides them to your Debian.

$ lspci -k -d ::0403
00:1f.3 Audio device: Intel Corporation Raptor Lake High Definition Audio Controller (rev 11)
    DeviceName: Intel HD Audio
    Subsystem: ASUSTeK Computer Inc. Raptor Lake High Definition Audio Controller
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel, snd_soc_avs, snd_sof_pci_intel_tgl
01:00.1 Audio device: NVIDIA Corporation GA106 High Definition Audio Controller (rev a1)
    Subsystem: ASUSTeK Computer Inc. GA106 High Definition Audio Controller
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel

Everything is alright with sound card's driver in my case - snd_hda_intel driver is present and in use.

➂ ALSA, PipeWire or Pulseaudio: what is better*? (trick question)

I have a sound card, I have drivers and firmware, and I have an audio output device—my headset connected via cable (let’s leave aside the Bluetooth management part for now).

So, I go to YouTube to listen to my favourite music band, Public Memory...

Well, I can’t hear anything (I can’t provide a screenshot as proof, though XD). So, what am I missing?

If you’ve ever Googled/troubleshooted about audio on Linux, most probably you’ve encountered the three names ALSA, PulseAudio, and PipeWire, regardless of your distro. Maybe, these names can be confusing. Are they alternatives to each other, do you need all of them, or which one is better? That’s why I gave this section its title. I want to answer these questions by exploring why I can’t listen to Public Memory - which software I am missing?

ALSA (Advanced Linux Sound Architecture)

_ Advanced Linux Sound Architecture (ALSA) is a software framework and part of the Linux kernel that provides an application programming interface (API) for sound card device drivers.
Put simply, ALSA can be divided into two components: The kernel API that provides access to your sound card for higher-level sound servers and applications, and a userspace library that provides more general functions (like effects, mixing, routing, etc.)
There is no way to "replace" ALSA, with regards to the kernel API. Previously, there was also OSS (Open Sound System), but that's been deprecated for nearly 20 years. The same is not true of ALSA's userspace library, which can be replaced. (Debian Wiki: ALSA)_

Here we are! The first answer is: ALSA is a must-have. No ALSA = no audio. Moreover, you already have it on Debian, because it is the part of the Linux kernel, the same as sound card drivers. So, since I have the Linux kernel, I should already have ALSA installed.

Indeed, I have it:

# I check for presense of package that contains the ALSA userspace library and its standard plugins, as well as the required configuration files:
$ dpkg -l | grep libasound2
ii  libasound2:amd64                     1.2.8-1+b1                           amd64        shared library for ALSA applications
ii  libasound2-data                      1.2.8-1                              all          Configuration files and profiles for ALSA drivers

Why I cannot listen to Public Memory, then?? T-T.

Well, if you haven’t been reading this series of articles from the first part, you might not know that when I installed my Debian, I used a super minimal installation: Debian Stable with only the standard system utilities. This setup is usually perfect for server environments, and the last thing you’d typically do on a server is play music. So, I do have ALSA, I have the drivers, and everything is ready for me, but it needs to be initialised. By default, it’s not initialised because Debian (if you choose this option during installation) doesn’t run/initialise anything you didn’t explicitly request (and I love it).

The initialization is quite simple: sudo alsactl init. However, I cannot use it because this command is part of the alsa-utils package, which is not installed by default on my setup and is not included in the standard system utilities. Therefore, I need to install it first.

$ sudo apt install alsa-utils
$ sudo alsactl init
Found hardware: "HDA-Intel" "Realtek ALC897" "HDA:10ec0897,104387fb,00100402" "0x1043" "0x87fb"
Hardware is initialized using a generic method
Found hardware: "HDA-Intel" "Nvidia GPU 9f HDMI/DP" "HDA:10de009f,1043881d,00100100" "0x1043" "0x881d"
Hardware is initialized using a generic method

Aaaannd magic! Public Memory divine music is in my ears!

Is ALSA restrictive in some way? Does it work only for one audio stream at a time?

For development purposes, I have multiple browsers installed, so I opened another Public Memory song in a different browser and played it simultaneously. It works—I hear the overlaying audio of both songs.

I think it’s time to set up Bluetooth management tools and try listening to songs using my wireless headset.

Note: Regarding the question posed in the section title, I haven’t fully explored it yet. For now, it’s clear that ALSA is the key to audio input/output working on Debian, and it’s functioning well for little tests I performed. I’ll leave advanced tests—like using the microphone, testing apps for calls, etc.—for later, as my priority now is setting up Bluetooth to use my headset wirelessly.

➃ Bluetooth devices management tool

I have a Bluetooth adapter (dongle Edimax BT-8500) and a Bluetooth device (Marshall IV headset).
Basically, I just need to install the bluetooth metapackage.

Package: bluetooth (5.66-1+deb12u2)
This package provides all of the different plugins supported by the Bluez bluetooth stack

$ sudo apt install bluetooth

I won't be using any GUI for now; I'll use command-line tool bluetoothctl instead. First, I'll check if the bluetooth service is running.

$ systemctl status bluetooth
# if it is not running
$ sudo systemctl start bluetooth

Then, I run bluetoothctl and enter its CLI

$ bluetoothctl
[bluetooth]# power on
# i set the agent to handle pairing and set it to default mode:
[bluetooth]# agent on
[bluetooth]# default-agent
# start scanning devices
[bluetooth]# scan on
[NEW] Device XX:XX:XX:XX:XX:XX MAJOR IV
[bluetooth]# scan off
#numbers you see is unique MAC address of your audio device
# once I have the MAC address of my Marshall headset, I can initiate pairing:
# I PUT INTO PAIRING MODE MY MARSHALL HEADSET
[bluetooth]# pair XX:XX:XX:XX:XX:XX
Attempting to pair with 1C:6E:4C:84:CF:A7
[CHG] Device XX:XX:XX:XX:XX:XX Connected: yes
[CHG] Device XX:XX:XX:XX:XX:XX Bonded: yes
...
[CHG] Device XX:XX:XX:XX:XX:XX ServicesResolved: yes
[CHG] Device XX:XX:XX:XX:XX:XX Paired: yes
Pairing successful
[CHG] Device XX:XX:XX:XX:XX:XX ServicesResolved: no
[CHG] Device XX:XX:XX:XX:XX:XX Connected: no
[bluetooth]# scan off
##Hmm, pairing is successful, but connection is no...
[bluetooth]# connect XX:XX:XX:XX:XX:XX
Attempting to connect to XX:XX:XX:XX:XX:XX
Failed to connect: org.bluez.Error.Failed br-connection-profile-unavailable

The error isn’t very self-explanatory. Something seems off. Bluez's corresponding error description still not entirely clear:

Failed to find connectable services or the target service.

However, nothing to worry, this isn’t really about troubleshooting or debugging. I just wanted to gently point you back to the unanswered question in the previous section - is having only ALSA enough?

First off, there are two guides for Debian on Bluetooth usage: one for non-audio Bluetooth devices, and another specifically for Bluetooth audio devices. The second one, the right one for my case, explicitly states:

Pre-configuration
In short: To connect to a given device, you need Bluetooth hardware on your PC (either built-in, or in the form of a USB dongle), the Bluez daemon, and a compatible audio server (either PulseAudio or PipeWire). Alternatively Bluetooth ALSA available since Debian 12 bookworm allows to avoid running of a high-level sound server. (Debian Wiki: Bluetooth Audio)

So, there are two options here:

You stick with ALSA only, using an additional package Bluetooth ALSA. In this case, you don’t need PulseAudio or PipeWire for Bluetooth audio devices to work.
Both PulseAudio and PipeWire, as mentioned above, are audio servers.

However, it may still be unclear why I can't simply use ALSA if it worked for wired music playback, and what's missing from my system. The answer is that, without additional software, the Bluetooth codec used by my Marshall headset isn't supported.

A codec determines how Bluetooth transmits from the source device to your headphones. It encodes and decodes digital audio data into a specific format. In an ideal world, a high-fidelity signal would be possible at the minimum specified bit rate, resulting in the least amount of space and bandwidth required for storage and transmission. Lower bitrates actually mean better compression but often mean worse sound quality, a high bitrate usually means better sound quality and worse compression. (Source)

There are more advanced codecs and less advanced ones. However, the use of codecs is somewhat bilateral. I mean, the choice I mentioned above—that you can go for just an additional package like Bluetooth ALSA or choose one of the sound servers—can be partially guided by the support of audio codecs out of the box. In this case, the winner is PipeWire. However, if your audio input/output doesn't support advanced codecs, there's no way to utilize them.
Interestingly, despite using Bluetooth 5.0, the my Marshall Major IV supports only the default SBC audio codec. This codec is adequate and the connection is stable, however it’s not ideal (source).

So, actually for my setup just additional package for bluettohb audio devices support for ALSA is viable version:

The bluez-alsa-utilspackage supports the LDAC and SBC codecs. AAC support is not compiled into the official package, because the required library is only available in the non-free repository.
This will also automatically configure a "bluez-alsa" systemd service, and install the proper ALSA configuration files to glue everything together. No further steps should be necessary. If you run into issues after installation though, a reboot might help. (Source)

However, I would prefer to use a sound server on my setup.

➄ PipeWire vs Pulseaudio

Before comparing these two audio servers, it would be helpful to first understand what they do and why they’re needed except for the Bluetooth connection of audio devices.

In the previous article of this series, I installed and configured a display server as the first step in building my custom UI. The reason for this is that all graphical applications "live" on the display server, which handles all inputs and outputs. Moreover, there is an inter-process communication, the desktop bus, used by applications to communicate with each other, creating a kind of "graphical ecosystem".

A similar parallel can be drawn for sound servers. While ALSA is sufficient for many use cases, you might need a sound server for more advanced and dynamic audio usage. This includes not just playing music but also audio and video streaming, sharing screens with audio during calls, and leveraging advanced audio technologies and codecs. For most of these scenarios, a sound server becomes essential.

The difference between PulseAudio and PipeWire is somewhat analogous to the difference between the X11 display protocol and Wayland. The latter is a much newer technology, designed and developed to replace PulseAudio while addressing all of its shortcomings.

There’s nothing inherently wrong with PulseAudio—it’s neither abandoned nor outdated. However, PipeWire is more modern and, in some cases, offers out-of-the-box support for advanced audio devices without the need for tweaks or extra configuration. This isn’t relevant in my case since, as I mentioned, my Marshall Major IV headset only supports the very basic SBC Bluetooth codec. Nonetheless, I’ve decided to go with PipeWire.

PipeWire is a server and API for handling multimedia on Linux. Its most common use is for Wayland and Flatpak applications to implement screensharing, remote desktop, and other forms of audio and video routing between different pieces of software. Per the official FAQ, "you can think of it as a multimedia routing layer on top of the drivers that applications and libraries can use."
As opposed to PulseAudio's focus on consumer audio and JACK's focus on professional audio, PipeWire aims to work for all users at all levels. Among other techniques, PipeWire achieves this with its ability to dynamically switch between different buffer sizes, for adapting to the different latency requirements of different audio applications.
In Debian 12, PipeWire 0.3.65 is available, and is considerably more reliable, and is a comfortable drop-in replacement for many use-cases. PipeWire is the default sound server with GNOME Desktop. (Debian Wiki: PipeWire)

➅ PipeWire: Installation

In the previous article, while configuring the BSPWM window manager and display server, it was necessary to set up D-Bus for the X session. This is because, when initiating the graphical user interface with startx from the console, an X session managed by X11 is started, and it exists with BSPWM which executes all configurations specified in its setup.

Similarly, PipeWire requires a session manager to function effectively.

Two session managers are now available. The one pulled in by default (wireplumber) is the one recommended by pipewire's developers. The other one (pipewire-media-session) is primitive, and is best when using PipeWire just for its basic functionality like screensharing. When using PipeWire as your system's sound server, the maintainer recommends installing the more advanced WirePlumber instead. This command will install WirePlumber while removing the old session manager:_

So, I just need to install WirePlumber, and as a dependency, it will bring PipeWire to my system. There is this recommendation from Debian developers:

It is recommended to install the metapackage pipewire-audio which depends on wireplumber (the recommended session manager), pipewire-pulse (to replace PulseAudio), pipewire-alsa (ALSA) and libspa-0.2-bluetooth (for Bluetooth support). Moreover, installing this metapackage will remove pulseaudioto prevent any conflicts between both sound server. (Source)

*NB! The Debian 12 Bookworm stable repository has quite outdated versions of the PipeWire sound server (v. 0.3.65) and its dependencies. Since PipeWire is young and evolving quickly, it may be important to have a more up-to-date version installed. So, the option to install it from Bookworm backports may be viable (v. 1.2.5), and later on, I will show you how. *

However, even if you decide to go for it, make sure to read the text below and check the logs, because otherwise, you might not understand the XDG Desktop Portal. A version of WirePlumber and PipeWire from Bookworm backports doesn't seem to complain about it, but trust me, you'll most likely need the XDG Desktop Portal."

$ sudo apt install pipewire-audio
$ systemctl --user start wireplumber

Now, I’ll check if everything is okay with wireplumber service:

$ systemctl --user status  wireplumber
● wireplumber.service - Multimedia Service Session Manager
     Loaded: loaded (/usr/lib/systemd/user/wireplumber.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-12-30 13:04:27 CET; 13s ago
....

Dec 30 13:04:27 wonderland systemd[1017]: Started wireplumber.service - Multimedia Service Session Manager.
Dec 30 13:04:27 wonderland wireplumber[6190]: Can't find org.freedesktop.portal.Desktop. Is xdg-desktop-portal running?
Dec 30 13:04:27 wonderland wireplumber[6190]: found session bus but no portal
Dec 30 13:04:27 wonderland wireplumber[6190]: Failed to set scheduler settings: Operation not permitted
Dec 30 13:04:27 wonderland wireplumber[6190]: SPA handle 'api.libcamera.enum.manager' could not be loaded; is it installed?
Dec 30 13:04:27 wonderland wireplumber[6190]: PipeWire's libcamera SPA missing or broken. libcamera not supported.
Dec 30 13:04:28 wonderland wireplumber[6190]: Trying to use legacy bluez5 API for LE Audio - only A2DP will be supported. Please upgrade bluez5.

Wireplumber is running, but there are a couple of things that aren't okay. Let's go through them one by one with the errors.

wireplumber: Failed to set scheduler settings: Operation not permitted

This is definitely something about permissions. It might be related to the fact that PipeWire version available on Debian Bookworm is low, so the permission issue is most likely due to the absence of a specific user group (pipewire) created with the necessary permissions when PipeWire is installed. The default group is used, but PipeWire needs more. Personally, I would leave it as is and only debug if something really doesn't work.

PipeWire's libcamera SPA missing or broken. libcamera not supported.

This is just about a PipeWire plugin, and depending on whether you need it or not, you can install it. I don't need it, so I’ll leave PipeWire to complain about the missing plugin. Here is the package for this plugin.

Trying to use legacy bluez5 API for LE Audio - only A2DP will be supported. Please upgrade bluez5
Well, here I am completely okay with it since I don't have devices that use legacy codecs. A2DP is exactly what I need:

Advanced Audio Distribution Profile (A2DP)
A standard for how Bluetooth devices that can stream high-quality audio to remote devices. This is most commonly used for linking wireless headphones and speakers to your PC. (Source)

What’s left is quite interesting, and it’s something you should pay attention to:

Can't find org.freedesktop.portal.Desktop. Is xdg-desktop-portal running?
found session bus but no portal

pipewire service reports the same issue in slightly different manner:

$ systemctl --user  status pipewire
● pipewire.service - PipeWire Multimedia Service
     Loaded: loaded (/usr/lib/systemd/user/pipewire.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-12-30 13:04:27 CET; 12min ago
TriggeredBy: ● pipewire.socket
....

Dec 30 13:04:27 wonderland systemd[1017]: Started pipewire.service - PipeWire Multimedia Service.
Dec 30 13:04:27 wonderland pipewire[6189]: mod.rt: Can't find org.freedesktop.portal.Desktop. Is xdg-desktop-portal running?
Dec 30 13:04:27 wonderland pipewire[6189]: mod.rt: found session bus but no portal
alisa@wonderland:~$

So, DBus User Message Bus is running correctly (I dedicated a very detailed attention to configuring it in the previous article). But something related to the desktop is missing...

What even is this?

➆ About XDG Desktop Portal

XDG Desktop Portal
A portal frontend service for Flatpak and other desktop containment frameworks.
xdg-desktop-portal works by exposing a series of D-Bus interfaces known as portals under a well-known name (org.freedesktop.portal.Desktop) and object path (/org/freedesktop/portal/desktop). (Source)

Mostly, it’s needed for containerized software like Flatpaks, as the quote above says. However, it’s not only for this, and even if you don’t use Flatpak, you could still run into problems. A very trivial case could be with a browser like Mozilla or any other when you try to upload something, and the file chooser dialog is needed. It won’t work without the portal.

Portals were designed for use with applications sandboxed through Flatpak, but any application can use portals to provide uniform access to features independent of desktops and toolkits. This is commonly used, for example, to allow screen sharing on Wayland via PipeWire, or to use file open and save dialogs on Firefox that use the same toolkit as your current desktop environment. (Arch Wiki: XDG Desktop Portal)

XDG Desktop Portal can be installed with sudo apt install xdg-desktop-portal on Debian. HOWEVER, THE COMMON MISTAKE IS TO INSTALL ONLY THIS PACKAGE. XDG Desktop Portal needs at least one backend (can have also more). There are different backends available on Debian Bookworm:

xdg-desktop-portal-gnome
xdg-desktop-portal-gtk
xdg-desktop-portal-kde
xdg-desktop-portal-wlr

I will proceed with installation of xdg-desktop-portal first.

Package: xdg-desktop-portal
desktop integration portal for Flatpak and Snap
xdg-desktop-portal provides a portal frontend service for Flatpak, Snap, and possibly other desktop containment/sandboxing frameworks. This service is made available to the sandboxed application, and provides mediated D-Bus interfaces for file access, URI opening, printing and similar desktop integration features. (Source)

$ sudo apt install xdg-desktop-portal
$ systemctl --user start xdg-desktop-portal
$ systemctl --user status xdg-desktop-portal
● xdg-desktop-portal.service - Portal service
     Loaded: loaded (/usr/lib/systemd/user/xdg-desktop-portal.service; static)
     Active: active (running) since Mon 2024-12-30 13:29:28 CET; 1s ago
 ....

Dec 30 13:29:28 wonderland systemd[1017]: Starting xdg-desktop-portal.service - Portal service...
Dec 30 13:29:28 wonderland xdg-desktop-por[7132]: No skeleton to export
Dec 30 13:29:28 wonderland systemd[1017]: Started xdg-desktop-portal.service - Portal service.

$ systemctl --user  restart wireplumber

$ systemctl --user  status wireplumber
● wireplumber.service - Multimedia Service Session Manager
     Loaded: loaded (/usr/lib/systemd/user/wireplumber.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-12-30 13:29:46 CET; 7s ago
   ...

Dec 30 13:29:46 wonderland systemd[1017]: Started wireplumber.service - Multimedia Service Session Manager.
Dec 30 13:29:46 wonderland wireplumber[7157]: Failed to set scheduler settings: Operation not permitted
Dec 30 13:29:46 wonderland wireplumber[7157]: SPA handle 'api.libcamera.enum.manager' could not be loaded; is it installed?
Dec 30 13:29:46 wonderland wireplumber[7157]: PipeWire's libcamera SPA missing or broken. libcamera not supported.
Dec 30 13:29:47 wonderland wireplumber[7157]: Trying to use legacy bluez5 API for LE Audio - only A2DP will be supported. Please upgrade bluez5.

WirePlumber just wanted the XDG Desktop Portal available, without the specific backend. So, after installing the XDG Desktop Portal and starting it as a service, the errors in the WirePlumber and PipeWire services related to the XDG portal disappeared (after restarting both of them).

However, the XDG Desktop Portal service itself is reporting that no backend was found: xdg-desktop-portal[7132]: No skeleton to export.

I already mentioned that even if you don’t use Flatpaks, you can still encounter problems with other apps, like browsers. The backend is easy to choose when you have the default DE: if it’s GNOME, then xdg-desktop-portal-gnome; if it’s KDE, then xdg-desktop-portal-kde. These are usually installed automatically if you have installed one of default DEs. I would install the GTK backend (xdg-desktop-portal-gtk). Why? For example, Firefox depends on GTK, and I also use GIMP, which depends on GTK. In general, GTK will be completely enough for me. But please note, some Flatpaks depend on other services that require a specific backend, like GNOME for XDG Desktop Portal. If a Flatpak doesn’t work with GTK, you’ll have to check the logs to figure out what it needs and install it if necessary, possibly another backend.

$ sudo apt install xdg-desktop-portal-gtk

NB! You need to understand this sequence in order to decide how and when to launch the aforementioned services (wireplumber and xdg-dektop-portal).

The wireplumber service manages the sound server pipewire sessions.
If you enable it with systemctl --user enable wireplumber, it will start automatically on boot.
However, you cannot enable xdg-desktop-portal service in the same way! Moreover, even if you can do it manually, keep in mind that it will result in an error without the DISPLAY environment variable, just like dunst in the previous article.

So, I do not enable wireplumber because it will need to be restarted after xdg-desktop-portal is launched. Instead, I modify the bspwmrc config, so it handles it for me.

I add these lines (systemctl --user start xdg-dektop-portal,
systemctl --user restart wireplumber and systemctl --user restart pipewire):

$ sudo vim.tiny /etc/xdg/bspwm/bspwmrc
#! /bin/sh

#SXHKD Launching
pgrep -x sxhkd > /dev/null || sxhkd -c "/etc/xdg/bspwm/sxhkdrc" &

#DISPLAY env import
systemctl --user import-environment DISPLAY

#Starting XDG Desktop portal
systemctl --user start xdg-dektop-portal

#ReStarting Wireplumber
systemctl --user restart wireplumber
systemctl --user restart pipewire

#DUNST Launching
systemctl --user start dunst

#POLYBAR Launching
polybar-msg cmd quit

echo "---" | tee -a /tmp/polybar.log
polybar 2>&1 | tee -a /tmp/polybar.log & disown


bspc monitor -d I II III IV V

bspc config border_width         2
bspc config window_gap          12

bspc config split_ratio          0.52
bspc config borderless_monocle   true
bspc config gapless_monocle      true

You can do sudo reboot and then startx to check the success of modifications with:

systemctl --user status wireplumber systemctl --user status xdg-desktop-portal and systemctl --user status pipewire. Here is what I have:

Everything seems alrighty!
Time to test Bluetooth!

$ bluetoothctl
[bluetooth]# scan on
[NEW] Device XX:XX:XX:XX:XX:XX MAJOR IV
[bluetooth]# pair XX:XX:XX:XX:XX:XX
Attempting to pair with XX:XX:XX:XX:XX:XX
[CHG] Device XX:XX:XX:XX:XX:XX Connected: yes
[CHG] Device XX:XX:XX:XX:XX:XX Bonded: yes
Pairing successful
[bluetooth]# connect XX:XX:XX:XX:XX:XX
Attempting to connect to XX:XX:XX:XX:XX:XX
Connection successful
[MAJOR IV]# scan off

Connection successful! Start enjoying Public Memory - Zig Zag.

Now I can use the WirePlumber CLI to check if the device is registered there.

➇ Pipewire audio profiles: Understanding the difference between device profile headset-head-unit and a2dp

$ wpctl status
PipeWire 'pipewire-0' [0.3.65, alisa@wonderland, cookie:1612476836]
 └─ Clients:
 ...

Audio
 ├─ Devices:
 │      39. GA106 High Definition Audio Controller [alsa]
 │      40. Built-in Audio                      [alsa]
 │      51. MAJOR IV                            [bluez5]
 │
 ├─ Sinks:
 │      41. Built-in Audio Digital Stereo (IEC958) [vol: 0.40]
 │  *   52. MAJOR IV                            [vol: 0.60]
 │
 ├─ Sink endpoints:
 │
 ├─ Sources:
 │      42. Built-in Audio Analog Stereo        [vol: 1.00]
 │
 ├─ Source endpoints:
 │
 └─ Streams:

...

Sinks are audio output devices, and sources are audio input devices. As you can see, wpctl status shows that my Marshall headset is registered as a sink (so, output) and not as a source, so I can't use the microphone of my headset.

wpctl is similar to pactl, which is the PulseAudio CLI. However, you can use it with PipeWire without installing the PulseAudio server—just the utilities. So, I’ll do that because pactl is easier to use for me. pactl list output will help me explain why my headset is attached only as an output device.pactl is part of the pulseaudio-utils package.

$ sudo apt install pulseaudio-utils 
$ pactl list
Card #52
    Name: bluez_card.XX_XX_XX_XX_XX_XX
    Driver: module-bluez5-device.c
    Owner Module: n/a
    Properties:
                ....
        api.bluez5.connection = "connected"
        api.bluez5.device = ""
        bluez5.auto-connect = "[ hfp_hf hsp_hs a2dp_sink ]"
                ...
        device.alias = "MAJOR IV"
        device.api = "bluez5"
        device.bus = "bluetooth"
        device.description = "MAJOR IV"
        device.form_factor = "headphone"
                ....
    Profiles:
        off: Off (sinks: 0, sources: 0, priority: 0, available: yes)
        a2dp-sink: High Fidelity Playback (A2DP Sink) (sinks: 1, sources: 0, priority: 16, available: yes)
        headset-head-unit: Headset Head Unit (HSP/HFP) (sinks: 1, sources: 1, priority: 1, available: yes)
        a2dp-sink-sbc: High Fidelity Playback (A2DP Sink, codec SBC) (sinks: 1, sources: 0, priority: 18, available: yes)
        a2dp-sink-sbc_xq: High Fidelity Playback (A2DP Sink, codec SBC-XQ) (sinks: 1, sources: 0, priority: 17, available: yes)
        headset-head-unit-cvsd: Headset Head Unit (HSP/HFP, codec CVSD) (sinks: 1, sources: 1, priority: 2, available: yes)
        headset-head-unit-msbc: Headset Head Unit (HSP/HFP, codec mSBC) (sinks: 1, sources: 1, priority: 3, available: yes)
    Active Profile: a2dp-sink-sbc

Input is quite long, but you just have to find the card of your audio device. What’s important in the output are the profiles. For my Marshall IV headset, the available profiles (which my Marshall supports, and as I mentioned above, it’s not the highest-end codec it supports in terms of quality) are a2dp-sink (only output), a2dp-sink-sbc (only output), a2dp-sink-sbc_xq (only output), headset-head-unit (both input and output) and headset-head-unit-xvsd (both input and output).

Active Profile: a2dp-sink-sbc

a2dp-sink-sbc: High Fidelity Playback (A2DP Sink, codec SBC) (sinks: 1, sources: 0, priority: 18, available: yes)

You can see that there are no sources for this profile, only sinks! So, THIS IS THE REASON WHY I can use my Marshall only as an output device.

However, I am not bound to only one codec, automatically selected. I can switch it.

$ pactl set-card-profile bluez_card.XX_XX_XX_XX_XX_XX headset-head-unit
#I can change it back
$ pactl set-card-profile bluez_card.XX_XX_XX_XX_XX_XX a2dp-sink-sbc

NB! When I use the profile headset-head-unit to use the microphone, the sound quality is quite horrid. I guess this is connected to the maturity of PipeWire and the overall performance of Bluetooth audio devices on Linux, which is evolving rapidly but still has room for improvement. You can tweak the configuration, and if I find a solution to improve the audio quality, I will definitely share it. Most likely, it will be related to the configuration of PipeWire

(Optional)

Upgrading the PipeWire sound server version could be a partial remedy. For Debian Bookworm, the way to upgrade it is through the Bookworm backports:

#add bookworm-backports repo
$ sudo vim.tiny /etc/apt/sources.list
deb http://deb.debian.org/debian/ bookworm-backports main contrib non-free
$ sudo apt update
$ sudo apt install --reinstall -t bookworm backports pipewire-audio

That's all! See you in the next, conclusive part, which will be dedicated to having fun with Polybar, shell cuxtomization and installing some cool apps.

"Why is it, when something happens, it is always you TWO?"- troubleshooting Bluetooth and Wi-Fi devices on Debian 12

Anna — Tue, 24 Dec 2024 00:30:40 +0000

Hardware support on Linux OSs is one of the most "sensitive" topics for new users transitioning from Windows. Troubleshooting can be quite difficult, as most hardware works in a plug-and-play manner on Windows, leading its users to not have any idea about how exactly hardware works even approximately.

I want to make one important point clear from the start: labeling Linux as a badly-done because of some hardware issues you’ve encountered and concluding that Windows is better is a VERY WRONG MINDSET TO START WITH. First of all, Windows is not a free OS. While it’s not crazily expensive for personal use, the costs escalate significantly in enterprise setups. Windows OS is developed and maintained by people, whoa are paid for it, while most Linux OSs are developed and maintained by the community. Please respect their work and remain polite and humble in the discussions — no one guaranteed you that everything will work perfectly out of the box.

Moreover, I want to explain another point: some troubles with Linux hardware support don’t arise because Linux developers are not skilled enough or don’t care. In fact, it’s quite the opposite. The issue with hardware support is that a VERY large share of hardware manufacturers simply don’t care about Linux users. They only ship drivers and firmware for Windows and provide warranties for that platform. For Linux, it's not just that they provide a driver with no warranty—in the most cases, there’s nothing at all. That’s half of the problem. The other half is that their firmware and drivers are often closed-source. It’s one thing to have open-source code that was written for Windows and it is available for everyone in an official Git repository, so Linux devs can just adapt this code for Linux, and another entirely when Linux developers have to reverse-engineer the closed code and then rewrite it for Linux OS.

This article will be dedicated to two problematic devices: Wi-Fi USB sticks and Bluetooth dongles. Support for Wi-Fi and Bluetooth devices is, I think, one of the most discussed topics on forums, Reddit threads, etc., across all Linux distros. I know that laptop touchscreens also cause troubles sometimes, but I don’t have a device with such functionality to write about it. However, what I do have is a Wi-Fi USB stick that doesn’t work out of the box and Bluetooth dongles that also don’t work out of the box.

Returning to the statement above about manufacturers not caring about users, a good example is Realtek. Chances are, you'll have to deal with them, because no matter the brand of your Wi-Fi device, most likely the core of it will be Realtek (chipset). You’re welcome to visit their website and try to download a driver that you need (Spoiler: UX of this site is -1000000/10, and for someone without a solid understanding of hardware, it will seem like reading Arabic—even if the site is in English).

However, I have written this article to shed light on many details related to hardware support on Linux, and I will show you how I get non-functioning Wi-Fi and Bluetooth devices to work with some tweaking.

The main objective of this article is to provide an explanation of logic behind of how hardware works on Linux, not to make something work at all costs, especially if it risks the stability of the system or involves installing questionable packages. There will be none of that in this article.

Here’s the road map for this article:

➀ Hardware Support: The Role of the Linux Kernel vs. Debian OS

➁ Clarifying the Difference Between Firmware and Drivers

➂ Troubleshooting Step #1: How to Identify the Root Cause – Missing Firmware or Missing Driver?

➃ Troubleshooting Step #2: Leveraging Debian repository's components to fetch missing firmware drivers

➄ "Why is it, when something happens, it is always you TWO?" - An Answer.

➅ Troubleshooting Step #3: Fetching missing WiFi drivers: Upgrading Kernel vs Manual installation

➆ Troubleshooting Step #4: Fetching missing Bluetooth firmware from web vs Replacing Bluetooth dongle

➀ Hardware Support: The Role of the Linux Kernel vs. Debian OS

You may have noticed that the title states this article will be about troubleshooting Bluetooth and Wi-Fi devices on Debian, but in the introduction, I spoke about Linux in general. You might wonder why. Debian has unfairly earned the reputation of being an OS that isn’t for beginners, especially regarding the fact that getting some hardware to work always requires tweaking (which is certainly not true starting from Debian 12).

It often takes the heat for the fact that Ubuntu—a Debian-based distro—supports something while Debian does not. People question how this is possible and throw accusations like, "Debian is bad," and so on.

And here’s the crucial point: hardware drivers are NOT THE RESPONSIBILITY OF THE OS in most cases. Hardware drivers are parts of the Linux kernel, and more specifically, they are kernel modules.

However, for the proper functioning of hardware on the OS, not only drivers are needed, but also firmware. Despite the common mixing of terms (when problems related to drivers being reported as missing firmware, and vice versa), Firmware ≠ Drivers!.

➁ Clarifying the Difference Between Firmware and Drivers

Firmware refers to embedded software which controls electronic devices. Well-defined boundaries between firmware and software do not exist, as both terms cover some of the same code. Typically, the term firmware deals with low-level operations in a device, without which the device would be completely non-functional (Source)

The key takeaway about firmware is that without it, hardware is completely non-functional. Without a correct firmware, it’s just a piece of microchip, and no driver will be able to send the correct signal to it. When we’re talking about USB devices—like Bluetooth and Wi-Fi dongles—once connected to USB ports, if the situation with their firmware is very bad, they will be detected as just unknown USB devices. This can happen with improperly produced hardware purchased from no-name brands or from unsuccessful replicas of existing hardware from renowned brands.

Device drivers are available as kernel modules (in most cases).

Kernel modules are pieces of code that can be loaded and unloaded into the kernel upon demand. They extend the functionality of the kernel without the need to reboot the system. For example, one type of module is the device driver, which allows the kernel to access hardware connected to the system. Without modules, we would have to build monolithic kernels and add new functionality directly into the kernel image. Besides having larger kernels, this has the disadvantage of requiring us to rebuild and reboot the kernel every time we want new functionality (Source).

So, now you know that drivers and firmware are not the same thing. The first step in troubleshooting is to understand what exactly the problem is with your non-functioning device—missing firmware or missing driver. Once identified, you can move on to resolving the issue.

➂ Troubleshooting Step #1: How to Identify the Root Cause – Missing Firmware or Missing Driver?

The troublesome devices that you will see in this article (TP-Link Archer T3U Plus WiFi USB Adapter and ASUSTek Broadcom BCM20702A0 Bluetooth dongle) I use solely for the purposes of this article; they are quite dinosauric and were originally purchased without any thought regarding their Linux compatibility. Keep that in mind, as I will elaborate more on this later. The truth is that far from all devices need troubleshooting on Linux—there are plenty of Bluetooth and Wi-Fi USB devices that work in plug-and-play mode, meaning no additional configuration is needed beyond plugging them into the USB port (I will share the lists later on in the article).

The first command that will give me an overview of the plugged-in devices is:

$ lsusb
# I will not be providing the list of full devices attached to my PC, just the 2 that are in question for this article
# Bluetooth dongle:
Bus 001 Device 009: ID 0b05:17cb ASUSTek Computer, Inc. Broadcom BCM20702A0 Bluetooth
# Wi-Fi USB stick
Bus 001 Device 003: ID 2357:0138 TP-Link 802.11ac NIC

How do I know that these devices are not functioning? With the Wi-Fi device, it's easy. When I run a command to list available network interfaces, I do not see a wireless network interface, which should be presented, because previous command demonstrated that my WiFi USB device is not broken, it is attached and identified correctly by Debian.

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> .... state UNKNOWN 
    ......
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> ..... state UP .....

I also have Network Manager installed, which handles my wireless network interface. When I try to activate a Wi-Fi connection using nmtui, I see that the list of available connections is empty:

With Bluetooth dongle, instead, I see a quick log pop up when booting into my Debian—an error related to Bluetooth. These boot logs are usually displayed very quickly, and if they’re not critical for the boot process, your system will boot anyway. However, you can (and should!) always review all the logs if during the booting process you saw some warning/error message. You can do it using the dmesg command.

$ sudo dmesg  #| grep Bluetooth

Here is the log, which is very self-explanatory as to why the Bluetooth dongle does not work properly:

Okay, for the Bluetooth dongle, the first troubleshooting step is done—the problem is with the missing firmware.

However, dmesg did not give me any error messages regarding the Wi-Fi USB stick. I just see that it was correctly identified (the same output as from lsusb), so I have to continue with further investigation.

There are two commands that can give you more technical details about your connected USB devices:

$ usb-devices
# Output for Broadcom Bluetooth dongle:
T:  Bus=01 Lev=01 Prnt=03 Port=07 Cnt=01 Dev#=  9 Spd=12  MxCh= 0
D:  ....
P:  Vendor=....
S:  Manufacturer=Broadcom Corp
S:  Product=BCM20702A0
S:  SerialNumber=5CF37094AE3A
C:  ....
I:  If#= 0 Alt= 0 #EPs= 3 Cls=ff(vend.) Sub=01 Prot=01 Driver=btusb
...
...
...
# Output for TP-Link WiFi USB stick:
T:  Bus=01 Lev=01 Prnt=02 Port=02 Cnt=01 Dev#=  3 Spd=480 MxCh= 0
D:  ....
P:  Vendor=.....
S:  Manufacturer=Realtek
S:  Product=802.11ac NIC
S:  SerialNumber=123456
C:  ....
I:  If#= 0 Alt= 0 #EPs= 5 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none)
....
....
....

So, the Bluetooth Broadcom dongle has missing firmware, but the driver (Driver=btusb) is loaded for it. Meanwhile, the TP-Link Realtek Wi-Fi Adapter shows Driver=(none), meaning the Wi-Fi USB stick has a missing driver.

$ lsusb -v
#outputs of this command are quite long for each USB device
# output for TP-Link Wi-Fi USB stick:
Bus 001 Device 003: ID 2357:0138 TP-Link 802.11ac NIC
Couldn't open device, some information will be missing
idVendor           0x2357 TP-Link
idProduct          0x0138
iManufacturer           1 Realtek
iProduct                2 802.11ac NIC
#output for Broadcom Bluetooth:
Bus 001 Device 009: ID 0b05:17cb ASUSTek Computer, Inc. Broadcom BCM20702A0 Bluetooth
Couldn't open device, some information will be missing
idVendor           0x0b05 ASUSTek Computer, Inc.
idProduct          0x17cb Broadcom BCM20702A0 Bluetooth
iManufacturer           1 Broadcom Corp
iProduct                2 BCM20702A0

From the lsusb -v, I also pasted details about the product ID, vendor, and manufacturer, as this information is very important for later if none of the other methods work and I have to manually search for a driver for the Wi-Fi and firmware for the Bluetooth.

However, before rushing into digging the internet for manually fetching drivers and firmware—which is the least optimal solution, as it is not the "Debian way" at all and could break your system or create a security breach if the driver/firmware is acquired from a "questionable" source—it's important to consider alternatives. If you're not familiar with the "Debian way" of administering your system and don't understand the risks of breaking your system, I advise you to read this article. For security-related threats when installing anything on your system, you can read this.

However, Debian offers a reliable potential solution for missing firmware and drivers directly from the Debian package repositories!

➃ Troubleshooting Step #2: Leveraging Debian repository's components to fetch missing firmware drivers

Before we start with this troubleshooting step, I want to make it clear where firmware and Linux drivers come from to your OS. Let’s start with firmware.

Historically, firmware would be built into the device's ROM or Flash memory, but more and more often, a firmware image has to be loaded into the device RAM by a device driver during device initialization. (Source)

The problem is, as I mentioned before, that many hardware manufacturers target ONLY Windows users, so the firmware shipped together with the drivers will be for Windows. And Linux users?

Free software based systems such as Debian depend on the cooperation between manufacturers and developers to produce and maintain quality drivers and firmware. Drivers and firmware are what determine if, and how well, your hardware works.
Non-free drivers and firmware are produced by entities refusing or unable to cooperate with the free software community. With non-free drivers and firmware support is often unavailable or severely constrained. For instance features are often left out, bugs go unfixed, and what support does exist from the manufacture may be fleeting.

Most of the firmware comes bundled with the Linux kernel in Linux distros. And often we have to praise the brilliant minds of Linux/Debain developers, when manufacturers refuse to cooperate. However, firmware can end up in your system not only as a part of Linux Kernel. This brings us back to Debian and its package repositories.

What about drivers? Well, it's the similar scenario:

Most of WWAN and WLAN dongles (USB devices) have their proprietary Windows drivers onboard. When plugged in for the first time, they act like a flash storage and start installing the Windows driver from there. If the driver is installed, it makes the storage device disappear and a new device, mainly composite (e.g. with modem ports), shows up (Source).

As I mentioned before, drivers are, in most cases, shipped with the Linux kernel. Device drivers are kernel modules, which are essentially reworked and adapted Windows drivers for the Linux kernel, in cases where the manufacturer didn't bother releasing drivers for Linux.

When you install Debian, it comes with the Linux kernel, which is actually a large package (and indeed it is handled as a package on your Debian) called linux-image-* (in my case, for example, linux-image-amd64). This package is fetched from the Debian repository, not from the Linux upstream. That's why the kernel version may vary. For instance, Debian Bookworm stable comes with kernel version 6.1.x, while the current latest release of the Linux kernel is 6.13. The Linux kernel package on Debian comes with many built-in kernel modules (drivers) that serve most of your hardware. Generally, if the Linux kernel doesn't have a kernel module (device driver) for a particular hardware device, Debian won't have it either. However, keep in mind the kernel version. The most accurate way to say the previous phrase would be that if the Linux kernel of version 6.1.x doesn't have some drivers, Debian Stable will not have them either. If you don’t understand why Debian won’t just upgrade the kernel to the latest version, then I recommend reading this article to better understand the Debian distro.

However, there are exceptions, and Debian doesn't always lack drivers that the Linux kernel doesn't have. A famous example is the Nvidia driver. It’s a proprietary driver for Nvidia GPUs that can be installed from the Debian repository, while the Linux kernel only offers the open-source Nouveau driver for Nvidia graphics cards. Another example is iwlwifi.

So, let's start with first attempt of troubleshooting. The probability of getting a kernel module-driver for my Realtek Wi-Fi USB from any Debian repository component is quite low— iwlwifi, mentioned earlier, is for Intel Wi-Fi adapters, not Realtek. However, the missing firmware can be fetched with a much higher chance!

Troubleshooting Step #2: execution

Debian, as a distro, has more than one package repository from which it can install and update packages and software. If you don’t understand what a package repository is and which repositories Debian has, I strongly recommend this article and this article. Debian has one main repository for each of its releases, and this repository also has repository components, one of which is specifically called non-free-firmware. The term "non-free" doesn’t mean you have to pay someone somewhere to use the software from there. It means that this component contains proprietary, closed-source software that has been carefully analyzed and rewritten for Debian OS to function well and harmoniously with the rest of the system, ensuring maximum possible support for various hardware devices.

Here is the list of repository components:

main consists of DFSG-compliant packages, which do not rely on software outside this area to operate. These are the only packages considered part of the Debian distribution.
contrib packages contain DFSG-compliant software, but have dependencies not in main (possibly packaged for Debian in non-free).
non-free contains software that does not comply with the DFSG.
non-free-firmware provides firmware that is needed by some hardware, but does not comply with the DFSG.(SourcesList — Debian Wiki).

So, the troubleshooting is very simple—you just have to ensure that apt can fetch packages from these repository components. To do so, you'll probably need to modify the list of sources for apt. Just to remind you, I am using the Debian Stable release, not Sid or Trixie.

$ cat /etc/apt/sources.list
# this is what I have:
#first, main repository of Debian stable
deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian/ bookworm main non-free-firmware
#repository for security updates
deb http://security.debian.org/debian-security bookworm-security main non-free-firmware
deb-src http://security.debian.org/debian-security bookworm-security main non-free-firmware
# repository bookworm-updates, to get updates before a point release is made;
# see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
deb http://deb.debian.org/debian/ bookworm-updates main non-free-firmware
deb-src http://deb.debian.org/debian/ bookworm-updates main non-free-firmware

What you need to look at is the first line, related to the main repository of your Debian release. If it, for some reason, only has the main component and contrib, you will need to add non-free and non-free-firmware: deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware.

After modifying sources.list, you'll need to run sudo apt update so that apt becomes aware of the updated sources list.

After this, I run:

$ sudo apt install firmware-misc-nonfree
#for double check:
$ sudo apt install firmware-realtek

The firmware-misc-nonfree package will bring this firmware. I can see the following files:

/lib/firmware/brcm/BCM-0a5c-6410.hcd
/lib/firmware/brcm/BCM-0bb4-0306.hcd

However, what the Broadcom Bluetooth dongle was looking for was a different version of firmware: BCM-0b05-17cb.hcd.

To inspect which firmware is on your OS, you just can inspect the contents of /lib/firmware. If I go into the subfolder /lib/firmware/brcm, I can see the existing firmware. In my case, However, none of them matches the firmware version that appeared in the dmesg error.

If you want to take a look at the existing device drivers, they are in a different location - /lib/modules/$(uname -r)/kernel/drivers/. uname -r command will show you the version of kernel in use by your Debian. You can find more than one kernel version in /lib/modules if you regularly update your System and Kernel.

Bluetooth drivers can be found in /lib/modules/$(uname -r)/kernel/drivers/bluetooth. Available Realtek Wi-Fi drivers are in /lib/modules/$(uname -r)/kernel/drivers/net/wireless/realtek/rtlwifi/

Anyway, the problem with both my troublesome devices still persists. There are Realtek drivers, but it seems none of them is suitable for my Realtek Wi-Fi device, and the same goes for the firmware for my Bluetooth dongle. So, I have to continue troubleshooting. As I mentioned earlier, what’s left for me is to fetch the driver and firmware from somewhere outside of Debian repositories.

Before proceeding with the last-resort troubleshooting solution, I want to remind you of what I said about these devices. They were acquired without any thought of their compatibility with Linux, and that’s where the problem starts. It’s not that Debian is a bad OS or Linux is a bad system—it’s just that both the manufacturer AND vendor of BOTH devices don’t care about Linux users. However, not all devices are like this on Linux OSs. I’m not saying that there are MANY other manufacturers that are better at supporting Linux (Broadcom and Realtek are very big players on the market of hardware, so the chances are high, that many pieces you buy, will be manufactured by them), but rather that MANY their devices already have drivers and firmware provided by Linux software engineers, just you were unlucky to get devices that are unsupported YET or ALREADY. Not many still, but there are also manufacturers that genuinely care about Linux users, and I believe they deserve your attention when you are about to buy a new hardware device.

The following section will be dedicated to existing Wi-Fi and Bluetooth devices that can work on Linux as plug-and-play. The reason you're reading this article is that your devices are not one of them, and most likely, you didn’t know how to choose the right devices when you bought them. It’s important to understand what exactly the support for a piece of hardware on Linux depends on (spoiler: NOT A BRAND/MODEL NAME).

➄ "Why is it, when something happens, it is always you TWO?" - An Answer.

Why exactly can Bluetooth and Wi-Fi hardware together cause some troubles? Do they have something in common? YES. If you didn’t know, Bluetooth is nothing else but a network—small, but still a network—a PAN (personal area network):

Bluetooth is a short-range wireless technology standard that is used for exchanging data between fixed and mobile devices over short distances and building personal area networks (PANs) (Source)

When you have a laptop, you have much less control over its hardware components compared to a PC (desktop setup), especially if you’ve built it yourself by selecting and assembling each part.

In terms of Wi-Fi and Bluetooth, it’s common to go for Bluetooth dongles or USB Wi-Fi sticks with antennas, or even dongles for Wi-Fi. But why are Wi-Fi cards so often overlooked and forgotten? These cards are essentially PCIe devices—in other words, just cards you attach to your motherboard.

Sure, they occupy a PCIe slot, which can be seen as a drawback. However, dongles occupy USB slots, and Wi-Fi PCIe cards don’t take up the precious PCIe slot reserved for your fancy GPU. Despite this, they’re still often avoided.

I bring this up because network cards highlight why Wi-Fi and Bluetooth issues often go hand in hand. If you look up network cards, you’ll find that most of them are "2-in-1" devices—combining both Wi-Fi and Bluetooth functionality.

Why Wi-Fi PCIe cards are often bundled with Bluetooth?
The primary reason is shared chipsets. Wi-Fi and Bluetooth are frequently integrated into a single chipset because both technologies use similar radio frequency (RF) components and they operate in the 2.4 GHz band. Combining them on a single card eliminates the need to duplicate components. Both Wi-Fi and Bluetooth require antennas to transmit and receive signals. By bundling the two, the same antenna(s) can often serve both purposes, reducing hardware requirements and saving space.

So, is a network PCIe card the panacea for all problems related to the lack of support for Bluetooth or Wi-Fi on Debian? NO. If you look at the image above, you’ll notice the antennas, metallic parts, and that green microcircuit board... that’s the guy! Whether it works out-of-the-box when attached to your motherboard, or requires extensive tweaking (and possibly still doesn’t work), depends entirely on it.

In the image above, the green microcircuit board belongs to Intel and is labeled with the model AX210. The crucial point is that whether a Wi-Fi or Bluetooth device works on Debian depends entirely on the available software support (drivers and firmware) for the chipsets used in those devices.

Please read the information from Debian developers, that I quoted below. You will most likely be confused by all the combinations of numbers and letters that are chipset names, but I will untangle it for you using the example of my Wi-Fi USB device later on in this article!

A WiFi device operates on an electronic chip called a "chipset". We can find the same chipset in several different devices. Consequently, the driver/module for one chipset will work for all wireless devices using that chipset.
Currently there are only a few modern wifi chipsets readily available that work with free software systems. For USB wifi devices this list includes the Realtek RTL8187B chipset (802.11G) and the Atheros AR9170 chipset (802.11N). For Mini PCIe all cards with an Atheros chipset are supported.
Wifi has always been a problem for free software users. USB Wifi cards are becoming less free. With the older 802.11G standard many USB wifi cards had free drivers and did not require non-free firmware. With 802.11N there are only a couple chipsets on the market, from Atheros, which are completely free.
One company which specializes in free software and sells 802.11N USB wifi cards, ThinkPenguin.com, has indicated the availability of free software supported 802.11N USB wifi cards is disappearing. Solving this problem will require more demand than currently exists. Next time you purchase a piece of hardware ask yourself if it is free software compatible. (Debian Wiki: WiFi)

Now that you are fully equipped with the theory behind hardware support on Linux/Debian, I will proceed with troubleshooting my USB Wi-Fi adapter.

➅ Troubleshooting Step #3: Fetching missing WiFi drivers: Upgrading Kernel vs Manual installation

I hope it’s clear now that there is no "safe for Linux" brand or manufacturer from which you can buy hardware—specifically Wi-Fi adapters—that will just work when you plug it in on Debian because it is THAT brand and THAT manufacturer. The quote I shared above may be confusing because it talks about chipsets with some alphanumerical names.

The support on Debian depending on the chipset makes it more complicated to choose the correct Wi-Fi device if you’re planning to buy one with the goal of making it work seamlessly on your Debian system. However, it’s not that bad. Remember, in the world of Linux, you always have a community that has your back.

Identifying the chipset and it's driver

What information do I have for now about my Wi-Fi USB adapter?
These are the details about my devices that I extracted with lsusb -v:

Device: ID 2357:0138 TP-Link 802.11ac NIC

idVendor 0x2357 TP-Link
idProduct 0x0138
iManufacturer 1 Realtek
iProduct 2 802.11ac NIC

Here’s the cornerstone—not all devices from the brand TP-Link are unsupported by Debian. It depends on the manufacturer of the chipset used in the TP-Link device, which in my case is Realtek. Not all Wi-Fi devices manufactured by Realtek will work seamlessly on Debian, because it depends on the specific chipset used. For my Wi-Fi adapter, it is known which wireless standards the chipset uses: it is 802.11ac. But chipset name still remains unknown.

That’s all. Oh, actually, look at the device ID numbers—they are meaningful, especially 0138, which is the product ID. I could technically find this on the web, specifically on the TP-Link site, as I definitely know the vendor.

However, I know the exact model of my Wi-Fi adapter—it is written on the metallic part of the USB stick, burned there, so I know it’s the TP-Link Archer T3U Plus.

Maybe, if you don’t recall the model name of your device that’s causing you problems and there’s no way to retrieve it, and you don’t have it written anywhere... well, you can try searching by the vendor ID and product ID first here and also here.

My Wi-Fi adapter is absent in both links—ID 0138 is not present. Indeed, there is an Archer T3U listed in the first link, but not the Plus version..

However, let’s look at the example of the Archer T3U first link:

012d Archer T3U [Realtek RTL8812BU]

The first part of this troubleshooting step is to find exactly this info—RTL8812BU. Does it remind you of something? No? Here’s a refresher:

$ ls /lib/modules/$(uname -r)/kernel/drivers/net/wireless/realtek/rtlwifi/
#this is the output of this command, that list all available drivers on my Debian
btcoexist  rtl8192c   rtl8192cu  rtl8192ee  rtl8723ae  rtl8723com  rtl_pci.ko  rtlwifi.ko
rtl8188ee  rtl8192ce  rtl8192de  rtl8192se  rtl8723be  rtl8821ae   rtl_usb.ko

And none of these present drivers fits my Wi-Fi Adapter! So, what do I need, and why don’t they fit?

Let’s start with the first one: rtl8192cu.

Running sudo modinfo rtl8192cu will show me the details about which device this driver is suitable for.

The key thing that is important for my troubleshooting—and actually gives a hint as to why this driver is not suitable for my device—is this line:
description: Realtek 8192C/8188C 802.11n USB wireless

My Wi-Fi adapter uses 802.11ac wireless standard. For more details, check here: IEEE 802.11ac on Wikipedia.

I can enrich modinfo output by checking the Realtek website, specifically the page dedicated to Wi-Fi device chipsets (there is also a page for combined Wi-Fi and Bluetooth). Here’s the link: Realtek Wi-Fi Chipsets.

Why rtl8192cu is not my Wi-Fi adapter's chipset based on the Realtek description?

General Description:
The Realtek RTL8811CU-CG is a highly integrated single-chip that supports 1-stream 802.11ac, bla bla bla...

I know that my Wi-Fi adapter supports 2 streams.

Host interface:
USB 2.0 for WLAN and BT controller.
I know that my Wi-Fi adapter uses USB 3.0.

So, I can continue exploring the Realtek list of chipsets. There’s no need to examine other existing drivers with modinfo because they definitely won’t fit.

I could say that I examined the Realtek list of chipsets and matched the parameters, hehe. Actually, I did out of curiosity, but in reality, I just Googled my model to find which drivers I need. However, the Realtek information card for this chipset looks exactly like the one in my Wi-Fi adapter.

Product Search: RTL8812BU

So, to make my Wi-Fi adapter work I need the rtl8812bu driver, and it is definitely currently missing on my Debian. Don’t worry if your device is more complicated and you find it difficult to figure out which driver you need, I have an amazing GitHub repository for you, which can be considered as a bible of a Wi-Fi adapters support on Linux.

I strongly recommend to check USB WiFi Adapter Information for Linux and USB WiFi adapters that are supported with Linux in-kernel drivers and USB WiFi adapters with Linux out-of-kernel drivers.

I started with USB WiFi adapters with Linux out-of-kernel drivers since my current Debian kernel definitely does not have the driver I need—rtl8812bu. However, I found some interesting information on this link:

chipsets - rtl8812bu and rtl8822bu - AC1200 - USB 3
As of kernel 6.2, the above chipsets have an in-kernel driver. It is located in the rtw88 in-kernel driver. I invite all to test the new in-kernel driver and use it if it meets your needs.

I know that my kernel version is inferior to this, however here is the preciese info:

$ uname -r
6.1.0-28-amd64

Troubleshooting Step #3: implementation

So, here is where the troubleshooting starts—I have two choices:

upgrade the kernel
install the kernel module manually using the code kindly provided by the gentleman who maintains this Wi-Fi adapters compatibility "bible": Linux Driver for USB WiFi Adapters that use the RTL8812BU and RTL8822BU Chipsets.

I will use both methods just for the purpose of this article.

Personally, I would go for a kernel upgrade, and I do this on my personal setup. However, the issue is that Debian Stable ships with kernel 6.1.x, so I actually have the maximum version available from the Debian stable main package repository. The unique, quite safe way to update the kernel is using Debian backports. If you don’t know what that is, you can read this article.

What are the risks of this method (you may also face them)? I have NVIDIA drivers installed. They are installed from the Debian stable repo. Nvidia drivers require kernel headers, the version of which should match the kernel version. So, I will have to upgrade those from the backports as well. Additionally, MAYBE I will need to reinstall the Nvidia driver from the Debian backports repository. (If all of this doesn’t make sense to you and you don’t understand what I’m talking about, I recommend you read this article.

The second option is manual installation. What are the drawbacks? I don’t like this method because it’s less the Debian way, and that’s all. Want to know my policy on it? I’d rather buy a new, compatible plug-and-play Wi-Fi USB adapter—or actually, not even that, but a PCIe network card. For me, it’s not worth installing a kernel module manually and potentially destabilizing my system, even slightly. But this is just my point of view. I’ll let you decide, as I will show you both examples. Please note, my concerns are not related to the developer/maintainer of driver I will be manually installing, he is doing a tremendously great job, it is just me, I manage my system in a particular way. By the way, I use a snapshotting tool, so I can try both. If you don’t know anything about snapshotting, I recommend this article. Using snapshot tool can make the procedure of troubleshooting much less problematic.

Method 1: upgrading kernel version from bookworm-backports repo

I start with a kernel upgrade from Bookworm backports.

The first thing to do is to make sure that the Bookworm backports repo is enabled and accessible for your apt to fetch packages:

$ sudo vim /etc/apt/sources.list
#append these lines:
#bookworm-backports
deb http://deb.debian.org/debian/ bookworm-backports main contrib non-free

$ sudo apt update

If you're worried that Debian will now update everything from there when you run sudo apt update, don’t worry—it won’t do this unless you explicitly ask for it.

Next, I search for the available kernel versions in the bookworm-backports repo:

$ apt search linux-image*
# I cannot install linux kernel 6.2, the minimum version available is 6.9, I am fine with it
linux-image-6.9.7+bpo-amd64/stable-backports 6.9.7-1~bpo12+1 amd64
  Linux 6.9 for 64-bit PCs (signed)
$ sudo apt install -t bookworm-backports linux-image-6.9.7+bpo-amd64
#if you want to fetch the latest available version from bookworm-backports use
# sudo apt install -t bookworm-backports linux-image-amd64
#immediately after I install Linux headers as well
$ sudo apt install -t bookworm-backports linux-headers-6.9.10+bpo-amd64
$ sudo reboot

During the installation of the kernel headers, I got these messages on logs:

dkms: running auto installation service for kernel 6.9.10+bpo-amd64.
Sign command: /lib/modules/6.9.10+bpo-amd64/build/scripts/sign-file
Signing key: /var/lib/shim-signed/mok/MOK.priv
Public certificate (MOK): /var/lib/shim-signed/mok/MOK.der
nvidia-current.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.9.10+bpo-amd64/updates/dkms/

nvidia-...................
.....

DKMS is rebuilding the NVIDIA kernel modules, which was triggered by the kernel upgrade. That’s quite good and convenient—it seems that I don’t need to reinstall the NVIDIA drivers. Moreover, DKMS is rebuilding the NVIDIA kernel modules signing them after for UEFI Secure Boot, because I have it enabled on my machine, and everything is going as configured. If you also use Secure Boot on your PC but have only a vague idea about it, I recommend reading this article.

The first thing I do after rebooting is run sudo apt autoremove to remove the old kernel packages.

The NVIDIA drivers are in place, as I can see from the nvidia-smi output. uname -r displays expected version of kernel in use.

I use sudo nmtui to activate Wi-Fi connection, and this time I have the complete list of detected Wi-Fi networks.

Let’s check if the wireless interface is present in the list of network interfaces and is UP.

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> ...state UNKNOWN ...
   .....
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> .... state UP ....
    ....
3: wlx984827dd37bb: <NO-CARRIER,BROADCAST,MULTICAST,UP> ...state UP ...

Voila!
Additional commands to control drivers in use:

$ usb-devices
T:  Bus=01 Lev=01 Prnt=02 Port=02 Cnt=01 Dev#=  3 Spd=480 MxCh= 0
D:  Ver= 2.10 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=2357 ProdID=0138 Rev=02.10
S:  Manufacturer=Realtek
S:  Product=802.11ac NIC
S:  SerialNumber=123456
C:  #Ifs= 1 Cfg#= 1 Atr=80 MxPwr=500mA
I:  If#= 0 Alt= 0 #EPs= 5 Cls=ff(vend.) Sub=ff Prot=ff Driver=rtw_8822bu
.... 
$ ls /lib/modules/$(uname -r)/kernel/drivers/net/wireless/realtek/rtw88
rtw88_8723de.ko.xz  rtw88_8821c.ko.xz   rtw88_8822b.ko.xz   rtw88_8822c.ko.xz   rtw88_pci.ko.xz
rtw88_8723d.ko.xz   rtw88_8821cs.ko.xz  rtw88_8822bs.ko.xz  rtw88_8822cs.ko.xz  rtw88_sdio.ko.xz
rtw88_8723du.ko.xz  rtw88_8821cu.ko.xz  rtw88_8822bu.ko.xz  rtw88_8822cu.ko.xz  rtw88_usb.ko.xz
rtw88_8821ce.ko.xz  rtw88_8822be.ko.xz  rtw88_8822ce.ko.xz  rtw88_core.ko.xz

Manual installation/building of Realtek driver.

So far, I am satisfied. I have a well-configured system, my NVIDIA drivers were built with DKMS, and I also have Secure Boot enabled and configured on my system. Actually, if your Debian setup wasn’t configured as well and precisely, some things could go wrong—especially with the NVIDIA drivers. That’s why I will also show another method for manually installing the driver for Realtek as an alternative without doing anything with your system's kernel version.

I am precisely following this guide:

$ sudo apt update
$ sudo apt upgrade
# First, my Debian should have installed some packages. (actually, this is what i do not like about manual installation of drivers, it is not installation but rather building, so it requires to have a lot of software to be installed for building

Mandatory packages: gcc make bc kernel-headers build-essential git
Highly recommended packages: dkms rfkill iw ip

$ sudo apt install -y linux-headers-$(uname -r) build-essential bc dkms git libelf-dev rfkill iw 
# in my case it will install this
#The following NEW packages will be installed:
  #bc git git-man iw libelf-dev liberror-perl rfkill zlib1g-dev
# Create a directory to hold the downloaded driver
mkdir -p ~/src
# Move to the newly created directory
cd ~/src
# Download the driver
git clone https://github.com/morrownr/88x2bu-20210702.git
# Move to the newly created driver directory
cd ~/src/88x2bu-20210702
# Run the installation script (install-driver.sh)
sudo sh install-driver.sh
# When I was prompted to edit the config, I chose "no" and then I chose "yes" to reboot.

After rebooting, the results are exactly the same as the first method's ones, so I will not repeat the proves.

The problem with Wi-Fi USB Adapter is resolved. The one that is left is with the Bluetooth Broadcom device.

➆ Troubleshooting Step #4: Fetching missing Bluetooth firmware vs Replacing Bluetooth dongle

I’ll start again by analyzing what I have on my device (output of lsusb -v).

Bus 001 Device 009: ID 0b05:17cb ASUSTek Computer, Inc. Broadcom BCM20702A0 Bluetooth
idVendor           0x0b05 ASUSTek Computer, Inc.
idProduct          0x17cb Broadcom BCM20702A0 Bluetooth
iManufacturer           1 Broadcom Corp
iProduct                2 BCM20702A0

I checked the details about my Bluetooth device by ID on the previously shared link:
This time, my device is present:
17cb Broadcom BCM20702A0 Bluetooth

But this time, I do not need a driver; I need firmware. Again, I turn to community support, as the Debian repository component non-free-firmware could not provide the required Bluetooth Broadcom firmware.

The firmware that dmesg reported my Bluetooth dongle is searching for is BCM-0b05-17cb.hcd. Here is the GitHub repository from where I can get it.

Will I install the firmware from there? No. Is it because I don’t trust the maintainer? No. It’s because I don’t trust the old firmware from Broadcom. Why? Because the repository maintainer generously shared this information:

Recently several vulnerabilities have been discovered in the Bluetooth stack such as CVE-2018-5383, CVE-2019-9506 (KNOB), CVE-2020-10135 (BIAS) and more. Since Broadcom has stopped active support for its consumer devices, your system may be subject to security risks. You will have to use these devices at your own risk. As a repository maintainer, I cannot provide security fixes.(Source)

Am I sure that the firmware my Bluetooth dongle is seeking is not suffering from reported vulnerabilities? I can’t know for sure. But knowing that this dongle is quite dinosauric, the chances are that it could be. So, the solution I see for me is to replace it with another Bluetooth dongle.

My alternative Bluetooth dongle was purchased recently and works perfectly on Debian—truly plug and play. There’s no need for configuration, installation of drivers/firmware, or any tweaks; it just works. This Bluetooth dongle is from the brand Edimax, purchased on Amazon for around $10 (Edimax BT-8500). It supports the Bluetooth 5.0 protocol.

Here are the outputs of the commands:

$ lsusb
Bus 001 Device 009: ID 7392:c611 Edimax Technology Co., Ltd Edimax Bluetooth Adapter

$ sudo dmesg
 1178.132034] Bluetooth: hci1: RTL: examining hci_ver=0a hci_rev=000b lmp_ver=0a lmp_subver=8761
[ 1178.133013] Bluetooth: hci1: RTL: rom_version status=0 version=1
[ 1178.133024] Bluetooth: hci1: RTL: loading rtl_bt/rtl8761bu_fw.bin
[ 1178.144494] Bluetooth: hci1: RTL: loading rtl_bt/rtl8761bu_config.bin
[ 1178.202622] Bluetooth: hci1: RTL: cfg_sz 6, total sz 27814
[ 1178.342160] Bluetooth: hci1: RTL: fw version 0x09a98a6b

Even if it may seem unpleasant to you that I opted for such a solution, that is my choice. I would never compromise my system just to make a device, priced no more than $20, work. However, I’ve shared the GitHub repo, and if you have the same devices, you are free to use it at your own risk.

I’ve achieved the objective of this article: I resolved the problems with my troublesome hardware - Bluetooth Dongle and Wi-Fi USB adapter. The notable solution was with the Broadcom Bluetooth issue. Really, no such inexpensive devices are worth compromising your system for. If your Wi-Fi and Bluetooth hardware pieces are very hard to troubleshoot, just consider replacing them. I’ve shared links, and I will repeat them here. Always remember, if you can, support vendors that care about Linux users, even if sometimes they are a bit overpriced.

Feel free to ask the questions on Reddit, and feel free to share with Linux community any Wi-Fi & Bluetooth device that worked seamlessly on your Debian/Linux setup!

Debian 12/13 … is amazing! How to: Create your custom codehouse #5 [From Console only to Custom Graphical User Interface]

Anna — Sun, 22 Dec 2024 19:59:19 +0000

Author's Note: This article was originally written for Debian 12 Stable (Bookworm). However, with the release of Debian 13 approaching -and since I’m currently using Debian Testing (Trixie) - I’ll be sharing screenshots and configs from my own setup. Things might look slightly different on your end. That said, some aspects may vary significantly (or even a lot), and I’ll make sure to point those out.

Customization is one of those things where, if you want your setup to look cool, you sometimes need cutting-edge versions of software.
Most of the content here will still work on Debian Stable. Just keep in mind that some minor functionalities in certain apps might not.

Building a Custom Minimalistic UI on Debian: From Bare Display Server to Fully Functional User Interface with BSPWM and Polybar.

In the previous parts of this Debian 12 series of articles (4 parts), I covered many details about the Debian distribution — what Debian offers, how it packages software, how to install Debian, and how to administer your Debian system in a secure way. If you enjoyed it so far and I’ve sparked your curiosity about Debian, let’s move on to the most interesting part - building Debian custom UI!

In parts 3A and 3B, you may have noticed that all the screenshots I provided are just white text on a black screen. That’s because I installed a very minimal version of Debian that, at the beginning, had only the standard system utilities and no way to run any GUI applications, even if I wanted to. Why? Was something wrong with the graphics drivers? Nope. I have only one video card - NVIDIA (my Intel CPU doesn’t have an integrated video card), I did have Nouveau drivers for the NVIDIA GPU from the start.

By the way, I installed the proprietary NVIDIA drivers. I’ve described this process in detail in this article (I couldn’t integrate it here because it would make this article incredibly long).

If you’re familiar with Linux systems, you may know about distro flavors. They’re essentially different Desktop Environments (DEs) bundled with the distro of your choice. So, if you download an .iso with a flavour, and install the system, you’ll have the Desktop Environment you picked. The most famous ones are GNOME, XFCE, and KDE Plasma. However, during installation I did not have any of them installed on my OS, so all I have is this “black" terminal. But are these well-known Desktop Environments are the only way to make your OS more user-friendly and get a UI? NOPE. Otherwise, I wouldn’t have written this article.

Will I install one of these Desktop Environments on my Debian setup and then show you how to customize it? NOPE. Why? Because it’s no fun. Also, it will bring me bloatware (in this case, software I will never use). To explain myself better, no matter how much you customize GNOME, GNOME is still GNOME. There’s no way to make it perfectly fit your needs while stripping away everything you NEVER use. The only DE I personally tolerate is XFCE. The rest? Meh. Basta. I deal with Windows on my work PC, eight hours a day, five days a week—enough of these cute "click-click" UIs.

For me, the most precious part of the process of Debian customization is that it brings you really close to the OS itself. Are you learning Rust? Curious about writing applications for Linux OSs (or even Windows)? But you’ve never gone far beyond GNOME and for you it is a magic box? Bad news: chances are, you won’t go far beyond web apps either.

Let's a bit leave for now UI, and I instead of writing a bullet list of this article content will show you schematically what this article for. So here is how I see basic needs of a user like me - the one who uses Debian for software development purposes.

What do we have here: input and output devices. As inputs, I have the keyboard and mouse, and also the microphone on the headset. For output, there’s the monitor, and for music or calls, the headset itself. For simplicity, I draw the mouse and keyboard using a wired connection. My mouse is wireless, but it’s important to note that if your keyboard and mouse use a dongle, they usually don’t rely on Bluetooth—they use radio frequency (RF) technology to communicate with their receivers (the dongles), so on the scheme they are not connected in any way to the Bluetooth dongle.

My keyboard is an amazing AKKO keyboard. Unfortunately, on Linux, it works only in wired mode, but I’m completely fine with that. On the other hand, my Marshall headset connects via Bluetooth. My monitor receives its output from the NVIDIA graphical processing unit.

NB! I mention the brands of the stuff I use not as an advertisement, but to show what works for me.

So, the goal is to install everything needed on my Debian to make all my input/output devices work flawlessly and to configure a custom Graphical Interface. Let's start with the latter.

I have made the decision to split the broad topic "Building a Custom Minimalistic UI on Debian: From Bare Display Server to Fully Functional Setup" due to the large amount of content. The first part, which you are reading now, focuses on the fundamental components installation and configuration. Here’s the roadmap:

Understanding how UI and Desktop Environments work on Debian
Display Servers, Display Communication Protocols and Windows Managers
Installing and configuring BSPWM Windows Manager

➃ Installing Browser (Brave browser) and Terminal Emulator (Alacritty)

➄ About Desktop Bus and inter-process communication (IPC)

➅ Installing Status Bar utility (polybar), File Manager (thunar), App Launcher (dmenu), Notification Daemon (dunst).

The following sub-part will focus on configuring Bluetooth audio devices and Wi-Fi.

The final article in this Debian series will be dedicated to customization of the status bar (Polybar). Moreover, I will install additional software that is unintegral part of my system's setup. By the end of that article, you’ll be fully equipped to diverge from my configuration files and create an even better UI of your taste—where only your imagination sets the limits!

Let's start!

1. Understanding how UI and Desktop Environments work on Debian

I don’t know if you’re familiar with web development, but let me use it as an abstract example to explain how Desktop Environments (DE) work and how the GUI of applications function on your Debian.

_Imagine you visit a website DEV.TO, and you read one of my articles. You like it and decide to give it a unicorn as a reaction. You click on the unicorn, and it appears in the list of reactions under my article. That’s what you see on "your side".

But behind the scenes—on the server side—your click is actually an event. This event triggers a piece of code on the server. The logic might look something like this:

If 'unicorn' is clicked → increment the counter for 'unicorn' reactions on my article (identified by its unique ID).
Update the database → write the new number of unicorns to the DB for the article 'idX'.
So, anytime another person opens my article, they’ll see the updated number of unicorns already given to it, which will be fetched from statistics DB._

Here’s your explanation refined and tied to Debian apps, keeping your tone and style:

How is this related to any Debian app? It’s exactly the same. Take File Explorer, for example. You open this app, explore the contents of your directories, search for a file, or maybe move a file from one directory to another. You do all of this with a mouse via the GUI, perhaps using drag-and-drop. But behind all of this, there are commands being executed! These commands are similar to the ones you could easily run in the terminal yourself: ls, mv, cp, find, and so on.

The GUI acts as a layer that simplifies these operations, making them accessible through clicks, drags, and buttons. But at its core, the GUI is just executing the system commands. It’s like a web app sending events to the server—behind the visuals, there’s logic and execution happening.

The Desktop Environments, feature-rich GNOME for example, however still under have commands, even though the complexity of them is pretty high. When you double click on the icon of and App GNOME opens a window on the screen with this app, you can expand it or you can collapse it or you can drag it to the corner of the screen - what a magic! Does GNOME sends some "signals" to GPU to re-render the screen really quickly when something changes? NO. Here is the schematic representation of the "parties" involved into your System UI.

Firmware and GPU drivers are out of the scope of these article, VERY DETAILED explanation about how firmware and drivers work on Debian can be found here and here NB! this scheme is accurate only for X (X11) communication protocol, for Wayland protocol is quite different, about their differences read below.

GNOME DE has several key components. The Window Manager is responsible for the functionality of floating windows—allowing you to drag, collapse, or expand them. The Compositor takes care of how these windows look, handling things like transparency, animations, and borders.

However, what exists outside of GNOME (or any other DE) is the Display Server. All Desktop Environments depend on the display server. It’s the display server that makes the fundamental difference between a terminal-only system and a system with a graphical UI.

2. Display Communication Protocols, Display Servers and Windows Managers

About Display Servers and Display Communication Protocols

When it comes to display stuff, there are two major "display technologies" used by almost all Linux operating systems, and the same applies to Debian. Their names are the X Window System (aka X11 or just X) and Wayland. To reduce confusion with terminology related to X-* stuff: the X Window System is an X display communication protocol, the latest version of which is 11, so it’s also called X11.

In the X Window System, the X Server itself does not give the user the capability of managing windows that have been opened. Instead, this job is delegated to a program called a window manager.
Regarding Wayland, the job is delegated to display server called a compositor or compositing window manager. (WindowManager - Debian Wiki)

There is a quite of confusion with all these terms related to X11 and Wayland, display communication protocols, display servers. Let's take a browser as an example to see the difference. FYI, there are terminal-based browsers! However, let’s focus on a classic browser with GUI features like interactive tabs, status bars, etc.

When you launch a browser—even if you launch it from the terminal (yes, on Linux, you don’t need to click an icon or launch the program from a menu; you can launch it just fine from the terminal with a command i.e. brave-browser)—it should appear on your screen. Based on where you click and what you do, the image on your screen changes immediately.

Obviously, some code handles this process, communicating your inputs and determining what should be displayed as outputs. In other words, there’s a display communication protocol. And this is where X11 and Wayland diverge—they ARE completely different protocols!

Wayland is a communication protocol that specifies the communication between a display server and its clients, as well as a C library implementation of that protocol. A display server using the Wayland protocol is called a Wayland compositor, because it additionally performs the task of a compositing window manager. (Source)

The X Window System core protocol is the base protocol of the X Window System, which is a networked windowing system for bitmap displays used to build graphical user interfaces on Unix, Unix-like, and other operating systems. The X Window System is based on a client–server model: a single server controls the input/output hardware, such as the screen, the keyboard, and the mouse; all application programs act as clients, interacting with the user and with the other clients via the server. (Source)

The key takeaway from these two quotes is NOT ONLY that the Wayland and X Window System protocols are different, BUT that they run on entirely different display servers. The X.Org Server is designed to run on the X11 communication protocol. Meanwhile, in case of Wayland, display servers are called the Wayland compositors and run upon the Wayland protocol.

However, to give you a clearer idea of how display servers and display communication protocols are distinct and separate things, let me inform you about existence of XWayland. XWayland is a set of patches applied to the X.Org server codebase that allow an X server to run on top of the Wayland protocol. These patches are developed and maintained by the Wayland developers to provide compatibility with X11 applications during the transition to Wayland.

A bit of background on Wayland: it’s much younger than X11, which was developed over 40 years ago. While X11 has been updated over time, it still struggles with an old-style architecture, legacy code, etc. Wayland, on the other hand, was created completely outside the X11 ecosystem, including X.Org display servers, so it has no dependency on that old technology.

However, Wayland isn’t brand new—it wasn’t released just last year. Its current status shows increasing adoption. Both GNOME and KDE Plasma desktop environments have transitioned to Wayland. Debian supports it seamlessly:

Xorg is the default X Window server since Debian 4.0 (etch). For Debian 10 and later, the default human interface protocol is Wayland. (source)

For a long time, the bottleneck was Nvidia’s proprietary drivers, but that issue has been resolved—Nvidia creates less and less troubles to Wayland based systems with which driver version update.

The display server communicates with its clients over the display server protocol, a communications protocol. The display server is a key component in any graphical user interface, specifically the windowing system.

So, here’s the answer to the rhetorical question I posed earlier in this article (After I installed a very minimal version of Debian that, at the beginning, had only the standard system utilities, why there is no way I can run any GUI applications?): After a minimal installation of Debian, the critical component missing is the Display Server. That’s why I can only use the terminal until I install a display server on the system.

Therefore, I have to proceed with installation of a display server. And after all this praise for Wayland—and maybe you've seen it hyped on Reddit and other forums— I will go for the X11 communication protocol and the X.Org Server. The reason? The key software I plan to use, bspwm, requires it.

BSPWM is a tiling window manager. I’ll update the previous diagram to visualize how it fits into the system.

So, BSPWM in my setup will be doing the job of windows management - open the windows for the apps when I launch any and will arrange them for me. What does it mean tiling?

In the context of window managers, tiling refers to a method of organizing and displaying application windows on a screen in a non-overlapping, grid-like manner. Unlike traditional stacking window managers (like those used in GNOME, KDE), where windows can overlap and need to be manually resized or moved, a tiling window manager automatically arranges windows to make the best use of available screen space.

bspwm arranges windows as the leaves of a full binary tree.

bspwm is very much in line with the "Debian way" of software—it doesn’t try to absorb a lot of functionality into itself. Instead, some functionality is delegated to other software. For example, in the diagram above, you can see picom above bspwm. picom is a compositor responsible for adding extra effects to the windows managed by bspwm, like transparency, animations, etc. However, bspwm can run perfectly fine without it!

What bspwm cannot work without is the sxhkd software. This is because bspwm doesn’t handle any keyboard or pointer inputs on its own. sxhkd is needed to translate keyboard and pointer events into bspc invocations. bspc is a program that sends messages to bspwm via its socket.

Display server: Xorg

Installing Xorg is as simple as running sudo apt install xserver-xorg-core. However, if you previously installed NVIDIA drivers using the nvidia-driver package (my case), it might already be installed. Below is the output of the command used to check if this package is already installed (dpkg -l | grep xserver-xorg*):

The nvidia-driver package pulled in xserver-xorg-video-nvidia as a dependency, which, in turn, brought the xserver-xorg-core package. Here is the code if your system does not have these packages installed:

$ dpkg -l | grep xserver-xorg*
#if output is empty:
#the X11 server itself without drivers and utilities:
$ sudo apt install xserver-xorg-core
#alternative for "full" xorg server installation:
#sudo apt install xorg

Note that you won't have the startx command if you install just x11 server itself, and therefore you will not be able to start a graphical display right after installation. startx command comes with the package xinit.

$ sudo apt install xinit

After this, if I execute startx, Xorg display server starts!

NB! If you cannot launch the X server with startx and your error says something like (EE) no screens/output found (EE), it’s most likely an issue with the NVIDIA not writing anything properly into the Xorg configuration file after installation. That happened to me once on Debian Sid. The solution from the Debian Wiki: NVIDIA Proprietary Driver:

As the NVIDIA driver is not autodetected by Xorg, a configuration file is required to be supplied. Modern Debian packages for the NVIDIA driver should not require you to do anything listed here as they handle this automatically during installation, but if you run into issues, or are using a much older version of Debian, you may try going through these steps.
Automatic:
Install the nvidia-xconfig package, then run sudo nvidia-xconfig. It will automatically generate a Xorg configuration file at /etc/X11/xorg.conf.

This is how my screen changes:

You can see poorly displayed nvidia-smi output because the terminal is a bit weird—it’s not using all the available space. By the way, you can see some colors in the terminal area. The default terminal emulator that is used is xterm. It has its own "window", though! But there’s no window manager yet to handle it more properly. Here’s proof that it’s not the fault of a badly configured Xorg server not understanding my screen size: when I run xrandr, I see the correct resolution for my monitor.

3. Installing and configuring BSPWM Windows Manager

For convenience, to proceed with the bspwm installation, I return to the terminal space outside the display server. However, there’s no stopx command to stop the X server started with startx. So, I just have to kill the X server process. When I run ps aux | grep X, I can retrieve the PID (process ID) and kill it with kill :

$ kill 1191

BSPWM installation:

$ sudo apt install bspwm
#this will install also sxhkd package

Now your window manager is installed. However, if you run bspwm in the terminal, it won't work, and if you start the display server, you'll most likely see a black screen. This happens because the window manager is present but not properly configured. You can try it anyway—this will teach you how to open a new TTY session using shortcut keys if something goes wrong in future. For me, it's Ctrl+Alt+F_X_ (F2, F3, F4, F5 ecc - each F key opens new terminal session, or if they're already in use, I can switch between them). In this way you do not loose control over your machine and do not have to use emergency power off.

bspwm configuration: configuration directory

So, let's configure bspwm and its ally sxhkd! This is done via creating configuration files for them. But where?

The default configuration file is $XDG_CONFIG_HOME/bspwm/bspwmrc: this is simply a shell script that calls bspc. (Source: BSPWM GitHub page)

$XDG_CONFIG_HOME is an environmental variable, so you can check its value by running echo $XDG_CONFIG_HOME. I’m quite sure the output will most likely be empty. Why? First of all, what is XDG?

XDG originally stood for "X Desktop Group," but now it stands for "Cross-Desktop Group." When it comes to desktop environments like GNOME, XFCE, KDE Plasma, etc., you can actually have more than one installed on your system, and you can switch between them. However, their developers need to be somewhat coordinated—especially regarding configuration file directories—otherwise, the system can become a mess. This is where freedesktop.org steps in to help.

freedesktop.org hosts the development of free and open source software, focused on interoperability and shared technology for open-source graphical and desktop systems. We do not ourselves produce a desktop, but we aim to help others to do so. (Source)

And as part of this interoperability effort, there is a document called the XDG Base Directory Specification.

In this specification, you’ll find more details about $XDG_CONFIG_HOME:

$XDG_CONFIG_HOME defines the base directory relative to which user-specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used. (Source)

If you have not set $XDG_CONFIG_HOME (i.e., the output of echo $XDG_CONFIG_HOME is empty), bspwm will search for its configuration in $HOME/.config/bspwm/. Your user's HOME directory should be set by default as it is one of the standard variables in Linux environments, and you can check it by running echo $HOME. (When you run cd, it brings you to your home directory exactly because of this variable.)

I want to draw your attention to the last two outputs from the screenshot. Here’s what I did: I logged in as the root user, and as you can see, the root user’s HOME directory is different!

Why does this matter? Well, if you put BSPWM's configuration files in your current user’s home directory, other users won’t have access to the nicely configured UI (using bspwm's and other tools' configuration files I’ll show you later). The root user won’t have access either.

So, what happens if you don’t set $XDG_CONFIG_HOME and place the BSPWM and SXHKD configuration files in .config within your user’s home directory (/home/<your-username>/.config)? Here’s what will happen:

When you log in as your default user and start the X display server, you’ll see your configured setup.
When you log in as root and start the display server, you’ll see a black screen.
When you log in as any other user, they’ll also see a black screen.

The solution to avoid these situations—if you truly need a shared configuration—is to place the configuration files somewhere outside of any single user’s home directory.

When it comes to X display server configurations, there’s a standardized location for this: /etc/xdg. By placing the configuration files there, they become accessible system-wide for all users (for !reading!; a user without sudo privileges cannot modify them.

If you really need a setup for multiple users, you can place all configurations I mention below in /etc/xdg/.. repeating the directory structure I will use. Both sxhkd and bspwm have option "-c" when they are executed. This option is meant to point to the "custom" location of configuration files. Both software is searching for location of their config by interrogating environmental variable XDG_CONFIG_HOME. If this variable is not set, the fallback default directory for config is $HOME/.config/.... Even though it is possible to set XDG_CONFIG_HOME environmental variable to point to /etc/xdg/ directory, I would rather not, due to the fact that many apps will try to write something there, if during installation this $XDG_CONFIG_HOME variable is detected.

bspwm configuration: configuration files

After bspwm installation you can find the examples of sxhkd's and bspwm's default configurations that can be used as a starting point for your custom configurations. The both configuration files can be found in /usr/share/doc/bspwm/examples.

Here, I want to introduce some slang related to what I’m about to do. The process of customizing Linux OSs is often called "Linux ricing". This term is widely used on Reddit, especially in a subreddit dedicated to it, where people share their setups. Often, when a setup is particularly cool, you’ll see comments from others asking to share the "dot files".

The term "dot files" actually refers to sharing the configurations used to achieve a specific UI setup. However, "dot files" (or more correctly, DotFiles) is not just slang—there’s even a Debian wiki page dedicated to DotFiles.

Why DotFiles are called this? I mentioned earlier, the default path for BSPWM configs is $HOME/.config, which is a folder, not a file. But its name starts with a dot (.). If you navigate (cd) to your home directory and list the contents with ls, you’ll see just regular files and directories. However, if you run ls -a, you’ll see much more, especially you will see files and directories starting with a (.).

To avoid making this article overly long, I’ll also share my "DotFiles" on GitHub.

Let's proceed with BSPWM configuration. You will have to copy "examples" of configuration to the directory where BSPWM will be searching it. If you are fine that configuration will work only for your current user you have to create subdirectories in $HOME/.config and place configuration files there in a correct way. I will be keeping configuration files in /etc/xdg.

$ cd (this will teleport you to your $HOME directory)
# you can double check
$ pwd 
$ ls -a 
#if you do not see .config directory in the output, you will have to create it
$ mkdir .config
#create a directory for bspwm config
$ mkdir .config/bspwm
#create a SEPARATE directory for sxhkd config
$ mkdir .config/sxhkd
# copy configuration files there:
$ cp /usr/share/doc/bspwm/examples/bspwmrc ./config/bspwm/
$ cp /usr/share/doc/bspwm/examples/sxhkdrc ./config/sxhkd/

Now, I can start modifying them. If you’ve chosen to keep BSPWM configuration accessible for all users and placed them outside of user's home directory (in /etc/xdg), you will have to use sudo— elevated privileges —to modify these files that are in /etc directory. If you keep your configs in $HOME/.config, you don’t need sudo!

This is the example of bspwmrc configuration:

I’ll start by modifying it.

$ cd .config/bspwm
$ sudo vim.tiny bspwmrc
#Let's see what is there:
#----------First Line-----------#
pgrep -x sxhkd > /dev/null || sxhkd &
# this command ensures that sxhkd is running along bspwm
#pgrep -x searches for processes sxhkd, if sxhkd is already running, pgrep exits with a success status
#> /dev/null discards the standard output of pgrep. This means the command doesn’t clutter your terminal with process IDs.
#|| - a logical OR operator. The command after the || only runs if previous command, pgrep, did not find sxhkd process. In such case, it starts sxhkd & in the background (& means run in the background).
#As I will not be keeping configiration files in the default directory, I have to inform sxhkd about where its configuration file is when it is launched:
#pgrep -x sxhkd > /dev/null || sxhkd -c "/etc/xdg/bspwm/sxhkdrc" &
#----------Second Line-----------#
bspc monitor -d I II III ...
# this line sets up "or workspaces", you can perceive them as tabs in the browser. You can navigate between them and have various applications opened. To avoid cluttering, I stop on 5 workspaces. There are Roman numbers by default, but you can name them as you want, 1,2,3 ecc or A,B,C... If you install fonts with icons later, you can use even Icons!
# I just trim everything after V.
bspc monitor -d I II III IV V
#NB! If you have more than one monitors, you can use separate bspc monitor commands to assign different sets of workspaces to each monitor.
#bspc monitor HDMI-1 -d I II III
#bspc monitor XX-1 -d IV V VI

#----------Lines 3-7 -----------#
bspc config ...
#these lines are about default configurations about arrangements of windows, for now i leave them as they are

#----------Lines 8-12 -----------#
bspc rule -a ...
#these lines set some specific rules for programs. I comment them out to keep the syntaxis for later, but I definitely do not need these exact rules:
#bspc rule -a ...
#bspc rule -a ...
#bspc rule -a ...
# ...

Next, I’ll move on to modifying sxhkdrc. In this configuration file, I need to set the hotkey combinations that will handle opening programs for me.

4. Installing Browser (Brave browser) and Terminal Emulator (Alacritty)

To start, the most crucial program I need is a terminal emulator. There are 2 cool Terminal emulators, Kitty and Alacritty. I will be installing Kitty. I also need a browser, and my favorite is Brave Browser, so I’ll have to install them first.

NB! Alacritty can run ONLY on a GPU, so if it is a problem for some reason, consider Kitty or another option like Terminator.

Debian 12 Stable vs Debian Testing Alert!
Current version of Kitty: version 0.41.1
Kitty version in bookworm repo: 0.26.5-5

Kitty version in trixie repo: 0.39.1-1

Current version of Alacritty: 0.15.1
Alacritty version in bookworm repo: 0.11.0-4
Alacritty version in trixie repo: 0.15.1-1

The difference between Alacritty from bookworm repo vs trixie repo is quite notable, because alacritty of version 0.11 uses .yaml configuration files and alacritty of version 0.15 .toml configuration file.

On Debian stable to get Alacritty version is 0.15, you will have to build Alacritty from source.

In install Kitty:

$ sudo apt install kitty
$ kitty -v
kitty 0.39.1 created by Kovid Goyal

To install Brave Browser I execute a single command:

# $ sudo apt install curl

$ curl -fsS https://dl.brave.com/install.sh | sh

Now, I need to go to the sxhkd configuration file, sxhkdrc, and add the hotkey shortcuts that will handle opening windows with these two apps for me!

$ cd
$ sudo vim.tiny .config/sxhkd/sxhkdrc 
# I replace urxvt terminal emulator with kitty
super + Return 
   kitty
# I add a new shortcut for Brave Browser:
super + b
   brave-browser
#A bit on syntaxis: 
# - "super" is "Windows" key on your keyboard, in Linux it is called super
# - "+" stands for which key you will press together with "super" key
# - "Return" is Enter key
# - "b" is just a letter of my choice, notice, it is minuscle
# - indented command - this command should be exactly the same that is in use when you launch any app from terminal. If for launching apps before you used only Desktop icons, you will have to google a bit to find the exact commands.

Everything is almost ready! All that is left is just to tell X display server and X11 communication protocol which windows manager they have to start! To do this, you have to create another configuration file .xinitrc. This file should be in your home directory, not in .config, just in $HOME directory.

# "teleport" to your home directory
$ cd
# check for the existence of .xinitrc file
$ ls -a
#if it is not there (most probably), you will have to create one
$ touch .xinitrc

Okay, now the .xinitrc file is created, but it is empty. Inside, I need to add just a couple of lines to point to launching the bspwm window manager.

This file will be automatically used by the startx command, which starts a graphical X session.

NB! In classical desktop environments like GNOME, KDE, etc., the process of starting a graphical session is handled by a Display Manager. A Display Manager is the graphical UI application that you see with a login form prompting you to enter the username of the user you want to log in as and the corresponding password. For example:

Ubuntu Display Manager

However, Display Managers are not always GUI-based; sometimes they are TUI (Text User Interface):

Ly Display manager

The very famous choice for custom UI setups is LightDM, which provides a minimalistic GUI for logging in on the go. However, a Display Manager is optional and doesn’t add an extra layer of security. You can simply log in from the terminal and then use the startx command to start an X graphical session. However, there is one thing that Display Managers handle automatically, which you will need to configure manually...

To command X11 to execute bspwm when startx command is launched is just exec bspwm. However, to make all your applications with GUI to work properly, there should be another player involved in this. And this other participant is D-Bus a.k.a Desktop Bus.

5. About Desktop Bus and inter-process communication (IPC)

D-Bus is a message bus system, a simple way for applications to talk to one another. In addition to inter-process communication, D-Bus helps coordinate process lifecycle; it makes it simple and reliable to code a "single instance" application or daemon, and to launch applications and daemons on demand when their services are needed.

D-Bus per se is a daemon, meaning it is a process that runs in the background. On Debian 12, it is managed by systemd. The D-Bus daemon should be preinstalled and up and running; you can check it with systemctl status dbus. However, there is quite different output if you run systemctl --user status dbus:

So, by default, D-Bus System Message Bus is up and running, while D-Bus User Message Bus is not running!

Will your bspwm setup work even like this? Yes. Will you be able to launch applications, yes, most probably. However, here is the log of Brave Browser when I simply started X session with exec bspwm:

See: Brave is complaining (in the left terminal window, you can see many repeating errors related to bus: ERRROR:bus.cc: Failed to connect to the bus), but running. For many applications it is crucial to have D-Bus Message Bus up and running for the X session they run in, so they will not function properly because they use it to communicate with other apps!

Having the D-Bus message bus running for the session is not trivial to configure. As I mentioned, the system D-Bus message bus is up and running right from the start of the boot process because it is one of the crucial systemd services.

Each session you start, when configured properly, must have a DBUS_SESSION_BUS_ADDRESS. When you boot into a TTY (like in my case, without a GUI environment), the session is started, and systemd handles it. So, I see this:

I logged in as my user, alisa, and by default, Debian has the dbus-user-session package installed. As a result, I have the DBUS_SESSION_BUS_ADDRESS variable set, which is critical. Without this environment variable, applications you start are unaware of the D-Bus session. So, the objective is to have this variable set (echo $DBUS_SESSION_BUS_ADDRESS should not be empty!).

When I switch to the root user with su -, which does not retain any user environment variables, you can see that there is no D-Bus address. However, if I use su without the -, the situation is different. Please note that this is a weak example because such a method of logging in is not very thorough and is usually used for other purposes. In general, systemd user space and D-Bus User Message Bus configuration is quite complicated, so my explanation here may provide misleading information.

When you start an X server display with startx, you are starting a graphical session, and it behaves differently. It does not require a login, so systemd does not automatically start a D-Bus session for it.

You might think that after entering the graphical session, you can manually run systemctl --user start dbus to start the D-Bus session. But this will not work because it’s not that simple. When you start it manually, it runs, and a socket is created. However, systemd behaves differently for users, and you will need to perform various additional steps to make it work properly, leveraging only systemd capabilities and the dbus-user-session package.

Since it is a bit quite of the scope of this article, I just recommend reading more on this topic for a deeper understanding, if you are curious. You can start from here.

If you follow a guide like this one, you’ll end up with a user D-Bus user session shared across all sessions and users. While functional, this approach can be considered suboptimal and not exactly the "Debian way," even though many desktop environments choose this route. For more details, you can refer to this communication.

However, if you're not keen on tweaking systemdservices and socket configurations, even here, Debian has you covered! The objective is to start the D-Bus user message bus for a to be initialized BSPWM graphical session (with exec bspwm). There’s a command-line utility that handles this: dbus-launch. This utility is part of the Debian dbus-x11package.

Package: dbus-x11
simple interprocess messaging system (X11 deps)
D-Bus is a message bus, used for sending messages between applications. Conceptually, it fits somewhere in between raw sockets and CORBA in terms of complexity.
This package contains the dbus-launch utility, which automatically launches one D-Bus session bus per X11 display per user. If the dbus-user-session package is also installed, it takes precedence over this package. Source

I install it:

$ sudo apt install dbus-x11

If you have googled around bspwm installation and configuration, you may have encountered the line you should add to your .xinitrc to start bspwm when graphical session is launched:
exec dbus-launch --exit-with-x11 bspwm. This, however, is not entirely correct, as it should be exec dbus-launch --exit-with-session bspwm.

I modify the .xinitrc file

$ cd
$ vim.tiny .xinitrc
 exec dbus-launch --exit-with-session bspwm 
#if you decided to place the configuration of bspwm in /etc/xdg, you do need to specify path to configuration file!
# exec dbus-launch --exit-with-session bspwm -c "/etc/xdg/bspwm/bspwmrc"

dbus-launch --exit-with-session bspwm works well for me because I include only the BSPWM launch command in .xinitrc. All other programs that are part of the session and run in the background are started from the BSPWM configuration.

However, if you want to start them separately, you need a slightly different command: exec dbus-launch --exit-with-session ~/.xinitrc.

In my case, to ensure complete tidiness, i go a bit further, and add the check for D-Bus User Bus already running:

$ cd
$ vim .xinitrc
 if [ -z "$DBUS_SESSION_BUS_ADDRESS" ]; then
    exec dbus-launch --exit-with-session bspwm
 else
    exit 0
 fi

Finally, it's time to try it! In terminal I execute the command startx:

You can see that the previous DBus-related error, “ERROR: bus.cc: Failed to connect to the bus” has been resolved. However, you might notice other errors—there are some related to kwalletd. It’s happening because Brave Browser tries to launch KDE Wallet Manager to use it a security mechanism for saved passwords storage. KDE Wallet Manager is part of the KDE Plasma desktop environment, and since I don’t have KDE Plasma installed, I don’t have the wallet manager either.

That said, Brave still runs fine without it. If you’d like to use KDE Wallet Manager, it’s easy to install with sudo apt install kwalletmanager. Don't worry it will not install you KDE Plasma DE, even though you will see a quite long list of stuff to be installed into your system.

At the next startx session, you’ll most likely be prompted to configure the manager by either setting a password or pointing to an existing GPG key (you need to generate that beforehand, details about how to do it are here).

However, KWallet may not work properly or might not pop up at all if there’s another missing configuration—the XDG desktop portal. I’ll discuss the XDG Desktop Portal in my next article, when I’ll be setting up the sound server to configure audio devices. It’s important for the sound server, so I decided to move that explanation to the next part, so I can show you more logs.

For now, for my standards everything is running as it should, and the base functionality is ready! Now, I just need to proceed with installing additional software to complete my custom UI setup!

6. Installing Status Bar utility (polybar), File Manager (thunar), App Launcher (dmenu) and Notification Daemon (dunst).

Status Bar: Polybar

First, I need a status bar. In the context of a desktop environment, a status bar refers to:

For this I will be using Polybar, that is installed with:

$ sudo apt install polybar

Polybar is a highly customizable tool, though it comes with some limitations. For example, you cannot use SVG icons directly, but you can use fonts that support icons and include characters from those fonts. By default, this is how the status bar will look if you use default configuration:

There is the detailed documentation on customization: Polybar Wiki. Like BSPWM and SXHKD, Polybar is customizable through a configuration file:

By default, polybar will load the config file from ~/.config/polybar/config.ini, /etc/xdg/polybar/config.ini, or /etc/polybar/config.ini depending on which it finds first.
If you do not specify the name of the bar and your config file only contains a single bar, polybar will display that bar. Otherwise you have to explicitly specify bar name. (Source)

After installation, the default configuration file can be found at /etc/polybar/config.ini. For now, I’ll leave the default configuration as it is. I plan to modify it in the next article, focusing on the main functionality I want to add or adjust. This includes status bar buttons for managing Wi-Fi connections, sound volume, Bluetooth connections, and display backlight.

In fact, I plan to have two Polybar status bars—one at the top and another at the bottom of the screen. You can create objects like icons or even simple text labels (as buttons) on Polybar and attach shell scripts to them. These scripts will define the functionality triggered by button presses, mouse clicks, arrow keys, or even the mouse wheel!

Notification daemon: dunst

A notification daemon handles the display of notifications, such as error messages or warnings.

Dunst is a highly configurable and lightweight notification daemon. It can be configured to display various useful notifications, such as messages from email, social media, and more. You can install it with:

$ sudo apt install dunst

The default configuration will be placed in /etc/xdg/dunst/dunstrc. If you prefer to keep all configurations in $HOME/.config, you’ll need to copy it there using:

$ cd
$ mkdir .config/dunst
$ cp /etc/xdg/dunst/dunstrc $HOME/.config/dunst

How to launch Dunst?

If you've ever seen some dotfiles of other people who use Dunst in their Linux ricing setups, maybe you noticed that they just add the line dunst & to their bspwmrc config. If you copied the config files to $HOME/.config/dunst, that's fine, dunst will find its config there. However, if you kept the config files where they were originally placed in /etc/xdg/dunst/dunstrc (like I do), you need to tell Dunst to look for them there with -conf argument (if you did not set explicitly $XDG_HOME_CONFIG to point to /etc/xdg).

However, there is something more than configuration. Spoiler: If you just add dunst & to your bspwmrc after instalaltion, DUNST WILL NOT WORK, and you will most likely not notice it (it will just silently go in error, if you do not set for it logging). Now I will explain why.

First, Dunst and Polybar are different. Polybar runs in the background, and so does Dunst. But Dunst is a notification daemon, meaning it's a service, that can be managed by systemd like other daemons. You can manage it with systemctl, and it should be available so that apps needing to notify you about something will have Dunst up and running when your X server is running. Let me show you some logs from the console before I start the display server with startx.

If you try to start dunst from console (no graphical session), it will go in error:

$ systemctl --user start dunst
...
Dec 30 11:07:49 wonderland systemd[946]: Starting dunst.service - Dunst notification daemon...
Dec 30 11:07:49 wonderland dunst[1061]: WARNING: Cannot open X11 display.
Dec 30 11:07:49 wonderland dunst[1061]: ERROR: [  get_x11_output:0065] Couldn't initialize X11 output. Aborting...
Dec 30 11:07:49 wonderland systemd[946]: dunst.service: Main process exited, code=killed, status=5/TRAP
Dec 30 11:07:49 wonderland systemd[946]: dunst.service: Failed with result 'signal'.
Dec 30 11:07:49 wonderland systemd[946]: Failed to start dunst.service - Dunst notification daemon.

$ dunst
WARNING: Cannot open X11 display.
ERROR: [  get_x11_output:0065] Couldn't initialize X11 output. Aborting...

So, the error is explicit: it cannot find the X11 output, which is pretty logical because the X server is not running and graphical X session was not started yet.

Okay, I start the server and graphical session with startx, and then I restart dunst:

$ systemctl --user restart dunst
Job for dunst.service failed because a fatal signal was delivered to the control process.
See "systemctl --user status dunst.service" and "journalctl --user -xeu dunst.service" for details.

Same error!

But as you can see, $DISPLAY is not empty when I echo it. So, what’s the problem?

The issue is that I tried to start Dunst before without X11, and with the unset DISPLAY variable. So even when I restarted it now, from active graphical session, it goes into an error. How can I fix this?

I need to make sure the DISPLAY environment variable is available for systemd user services before I restart Dunst. I can import this environment variable for systemd user services with:

$ systemctl --user import-environment DISPLAY
$ systemctl --user restart dunst

Voila:

So, the important part about starting Dunst is that before you can start it, you need to import the DISPLAY environment variable using:

$ systemctl --user import-environment DISPLAY

When you start the graphical session with startx from the console, the DISPLAY variable is not available there initially. After graphical X session starts, it becomes available, but Dunst doesn't automatically get informed about it.

I prefer to launch Dunst using systemctl rather than dunst &. If you placed all configurations in /etc/xdg rather than in $HOME/.config, to launch Dunst with systemctl you will also need to pass the path to the config file. I don’t have to do this as I've placed the Dunst configuration in $HOME/.config/dunst/dunstrc.

You can find the configuration files for user-space services managed by systemd in /usr/lib/systemd/user

$ ls /usr/lib/systemd/user
$ sudo vim.tint /usr/lib/systemd/user/dunst.service
[Unit]
Description=Dunst notification daemon
Documentation=man:dunst(1)
PartOf=graphical-session.target

[Service]
Type=dbus
BusName=org.freedesktop.Notifications
ExecStart=/usr/bin/dunst -conf /etc/xdg/dunst/dunstrc

I added the argument to point to the config file:
ExecStart=/usr/bin/dunst -conf /etc/xdg/dunst/dunstrc
Now I can add both Polybar and Dunst to the bspwmrc so they are launched automatically by bpc. I will need to modify the BSPWM configuration file.

$ cd 
$ vim.tiny .config/bspwmrc #you do not need sudo if you are in $HOME
#I add this lines after pgrep -x sxhkd....

#DISPLAY env import
systemctl --user import-environment DISPLAY

#DUNST Launching
systemctl --user start dunst

#POLYBAR Launching
# terminate in "elegant" way all running polybarS, if there are some (alternative to kill)
polybar-msg cmd quit
# require logging of polybar
echo "---" | tee -a /tmp/polybar.log
# launch polybar in background and start logging. As for now I am using default polybar config, the command polybar will automatically fetch it, if the configuration is placed in one of the places, where polybar searches it (in my case it is in /etc/polybar/config.ini)

polybar 2>&1 | tee -a /tmp/polybar.log & disown

XDG desktop portal

XDG Desktop Portal
A portal frontend service for Flatpak and other desktop containment frameworks.
xdg-desktop-portal works by exposing a series of D-Bus interfaces known as portals under a well-known name (org.freedesktop.portal.Desktop) and object path (/org/freedesktop/portal/desktop). (Source)

Portals were designed for use with applications sandboxed through Flatpak, but any application can use portals to provide uniform access to features independent of desktops and toolkits. This is commonly used, for example, to allow screen sharing on Wayland via PipeWire, or to use file open and save dialogs on Firefox that use the same toolkit as your current desktop environment. (Arch Wiki: XDG Desktop Portal)

All well-known Desktop environments like KDE Plasma, Gnome, XFCE etc. use XDG portals, however, what is important is that XDG Desktop Portal has a backend and different DEs use different backends.

When an application sends a request to the portal, it is handled by xdg-desktop-portal, which then forwards it to a backend implementation. This allows implementations to provide suitable user interfaces that fit into the user's desktop environments, and access environment-specific APIs for requests like opening a URI or recording the screen. Multiple backends can be installed and used at the same time.(Arch Wiki: XDG Desktop Portal Backends)

Each XDG portal backend is based on a specific Toolkit. For example GNOME's desktop portal backend is based on GTK v.4; while KDE Plasma's backend is based on Qt 6. And this is actually why you cannot customize DEs in a similar way. For example, you cannot install a Theme for KDE Plasma 6 and try to apply it to GNOME, it will just not work. Same even for KDE Plasma 5 and KDE Plasma 6 - first uses a Qt 5 and the latter one Qt 6.

Anyway, with BSPWM, what should you use? Well, there is a generic backend for custom DEs that is based on GTK v.3. This GTK 3.0 Toolkit will enable you later to apply Themes for your DE! For example, if you like Dracula Theme, you will be able to apply it DE-wide : colors, fonts ecc.

However, remember, XDG desktop portal with at least one backend is not just a question of whether you will be able to apply custom themes or not later, it is about providing applications the means for their proper functioning!

XDG Desktop Portal can be installed with sudo apt install xdg-desktop-portal on Debian. The common mistake is to install only this package. This package does not bring you any "default" backend mentioned above.

I proceed with installation of xdg-desktop-portal first.

Package: xdg-desktop-portal
desktop integration portal for Flatpak and Snap
xdg-desktop-portal provides a portal frontend service for Flatpak, Snap, and possibly other desktop containment/sandboxing frameworks. This service is made available to the sandboxed application, and provides mediated D-Bus interfaces for file access, URI opening, printing and similar desktop integration features. (Source)

$ sudo apt install xdg-desktop-portal
$ systemctl --user start xdg-desktop-portal
$ systemctl --user status xdg-desktop-portal
● xdg-desktop-portal.service - Portal service
     Loaded: loaded (/usr/lib/systemd/user/xdg-desktop-portal.service; static)
     Active: active (running) since Mon 2024-12-30 13:29:28 CET; 1s ago
 ....

Mar 30 13:29:28 wonderland systemd[1017]: Starting xdg-desktop-portal.service - Portal service...
Mar 30 13:29:28 wonderland xdg-desktop-por[7132]: No skeleton to export
Mar 30 13:29:28 wonderland systemd[1017]: Started xdg-desktop-portal.service - Portal service.

As you can see, the XDG Desktop Portal service is reporting that no backend was found: xdg-desktop-portal[7132]: No skeleton to export.

I install "generic" backend xdg-desktop-portal-gtk that uses GTK 3 Toolkit.

$ sudo apt install xdg-desktop-portal-gtk
# you can check that the warning about Skeleton is gone
$ systemctl --user restart xdg-desktop-portal
$ systemctl --user status xdg-desktop-portal

In general, GTK will be completely enough for my setup. But please note, some apps, especially Flatpaks depend on other services that require a specific backend, like GNOME's one. If a Flatpak doesn’t work with GTK generic backend, you’ll have to install another backend. As I mentioned above, multiple backends can be installed and used at the same time. XDG desktop portal will redirect app's requests to the right backend.

File manager

yazi

$ sudo apt install ffmpeg 7zip jq poppler-utils fd-find ripgrep fzf zoxide imagemagick

sudo apt install debootstrap

File Manager: Thunar

File Manager is the guy that you open to explore what's on your PC, find files, move/copy files and interact with files from USB pen drives:

I use Thunar, that can be installed with

sudo apt install thunar

App Launcher: dmenu

App Launcher is a tool that displays installed applications, allowing you to launch them with a mouse click. Additionally, it provides a search utility for quickly finding the app you’re looking for.

In classic desktop environments, an app launcher typically looks something like this—for example, GNOME's app launcher:

I am using dmenu. This app launcher is much more minimalistic compared to the GNOME app launcher displayed above. It won’t display all the icons or other extras, but it will allow you to search efficiently and launch apps with a click.

I prefer this minimalist approach because I rarely use app launcher anyway. Most of the time, I launch apps from the command line or by using SXHKD keybindings, which I configure for the apps I frequently use. dmenu can be installed with this command:

sudo apt install suckless-tools

I need to add the Thunar and dmenu to the SXHKD configuration to bind hotkeys for launching them.

$ cd /etc/xdg/bspwm
# if you decided to keep configurations in $HOME/.config:
#$ cd 
#$ cd .config/sxhkd
$ sudo vim sxhkdrc #you do not need sudo if you are in $HOME
#I add this lines only
# file manger
super + f
      thunar
#dmenu is already there, if you kept the default config!
# super + @space
#      dmenu_run

Here we are! Now, if I run startx from terminal, all my setup starts:

On the top: status bar Polybar; on the right: Thunar, Brave Browser

On the top, overlaying polybar status bar: app launcher -dmenu

VOILA!

Debian 12: NVIDIA Drivers Installation

Anna — Sun, 01 Dec 2024 20:05:39 +0000

NVIDIA, with the rise in popularity of Large Language Models (LLMs), has firmly secured its position as the leader in the GPU market, leaving AMD GPUs far behind. However, for Linux users, installing NVIDIA GPU's drivers is still troublesome. In this article, I’ll cover key concepts about NVIDIA drivers and will show an easy way to install them on Debian OS. This article is a long-read, however, understanding how driver-related things work on Linux will make your user experience much better.

Speaking about LLMs, other AI models, neural networks, etc. - if you're here because your final goal is to run them on your GPU, you're on the right track. Installing the NVIDIA drivers is the FIRST STEP toward enabling your algorithms to run on your NVIDIA GPU.

BUT. It’s only the step 1/2. The next essential step is installing the CUDA Toolkit. Don’t be misled: just having the drivers installed is not enough to actually use your GPU with Compute Unified Device Architecture (CUDA).

The confusion often comes from NVIDIA’s own documentation. Sometimes docs refer to the drivers as "CUDA" something - for example, just try to make sense of the first three compatibility tables here). Also, after successfully installing the NVIDIA drivers when you run nvidia-smi and see a "CUDA Version: 1y.x" there, it’s easy to think "cool, it’s all set up—2-in-1, done!" But nope - that’s just the CUDA "user-space" driver. It’s not the actual CUDA toolkit your algorithms need to perform GPU-based computations.

NVIDIA Docs Hub: CUDA Compatibility

Here’s the roadmap of this article:

1 Linux kernel & Co .ko: Understanding what are the drivers on Debian OS and how they are related to Linux kernel.

1.1 Drivers = Kernel Modules
1.2 NVIDIA drivers ≈ nvidia*.ko + libcuda.so; libcuda.so != CUDA Toolkit!

2 NVIDIA documentation & NVIDIA recommended drivers

2.1 NVIDIA OpenED Source of their Drivers? O_o

3 NVIDIA drivers installation: "Debian" way:

3.1 Installation Step 1: Add the contrib and non-free repository components to the list of sources for apt
3.2 Installation Step 2: Check if your OS is booted with Secure Boot
3.3 Installation Step 3: Check if your system uses dracut
3.4 Installation Step 4: Choose a flavour - "proprietary" vs "open"
3.5 DKMS is your bro when it comes to drivers
3.6 About Linux Headers
3.7 nvidia-kernel-dkms OR nvidia-open-kernel-dkms <-- install kernel space components of NVIDIA drivers; libcuda1 and other libraries (lib*) <-- install user-space components of NVIDIA drivers, a.k.a CUDA drivers

4 How and when to update NVIDIA drivers

My GPU model: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate]

UPD: Added a section about NVIDIA drivers installation on Debian Trixie (testing) and about newest NVIDIA drivers installation following the NVIDIA guide (with precaution measures)

1. Linux kernel & Co .ko

When you first install an OS using a graphical installer, you might notice the graphics of Installer look a bit rough or stretched, kind of like this:

After installation is finished and you boot for the first time, everything usually looks fine in terms of resolution. That’s because, during the installation of Debian or any other OS, the Linux Kernel with default drivers (including drivers for Graphical Output) are installed.

You boot from a pen drive with .iso installer to install Debian OS →
at the early stages of installation the Linux kernel is getting installed (as no OS can operate without it) →
the Kernel doesn’t come alone - it has kernel modules, which are essentially the Linux drivers for your PC's hardware pieces.

During installation, the installer doesn’t connect to the Linux kernel source repository (upstream, containing the most recent existing version of Linux kernel) to fetch the kernel for your system; instead, it connects to its own repository, where the Linux kernel is packed, optimised, and tuned to work harmoniously with the system. In case of Debian:

The kernels in Debian are distributed in binary form, built from the Debian kernel source. It is important to recognize that Debian kernel source may be (and in most cases is) different from the upstream (or "pristine") kernel source, distributed from www.kernel.org and its mirrors. Due to licensing restrictions, unclear license information, or failure to comply with the Debian Free Software Guidelines (DFSG), parts of the kernel are removed in order to distribute the source in the main section of the Debian archive. (Source)

Key takeaway:

Your OS's kernel has kernel modules and they are providing your OS with possibility to use various hardware components of your PC, because drivers per se are kernel modules.

1.1 Drivers = Kernel Modules

You can explore the drivers available on your system by running ls /lib/modules/$(uname -r)/kernel/drivers/ ($(uname -r) part retrieves the exact version of the kernel currently in use by your system; there might be multiple kernel versions installed, especially if you regularly update your system packages (some updates bring also new version of Linux Kernel).

When I run lspci -k | grep -A 3 NVIDIA, I can see that currently for my NVIDIA video card, the kernel driver in use is Nouveau. By default, Debian provides Nouveau drivers for NVIDIA graphics cards because they are open-source and non-proprietary, unlike NVIDIA’s official drivers.

Is it because Debian devs are obsessed with some anti-capitalist or whatsoever views so they pack their system only with open source "free" software? Nope. The point in sticking to open source software is that open source software can be tested and integrated to an OS much easier than closed-source software. Its behaviour, impact on other OS’s components is much more predictable because source code is open.

I mentioned that Debian "provides" Nouveau drivers (not only them, ofc), and I’d like to elaborate on word "provide". On Debian, the Linux kernel is not just a standalone "file/object" pulled during installation - it is a package. This package is called linux-image-*, where the asterisk represents a specific target architecture build of the kernel package, such as cloud, amd64, rt, and the version of this package. This kernel package linux-image-*brings with itself default kernel modules, like the Nouveau driver (nouveau kernel module), to ensure hardware compatibility out of the box. You can actually check which package provides a specific kernel module with dpkg-query command. In case of nouveau driver for NVIDIA video card the command will be:
dpkg-query -S /lib/modules/$(uname -r)/kernel/drivers/gpu/drm/nouveau/nouveau.ko

Even though there’s nothing wrong with Nouveau drivers for NVIDIA GPU — you can use them and enjoy very good performance (many Linux users prefer them over NVIDIA’s proprietary drivers). However, if you plan to use your NVIDIA GPU for AI, machine learning, or LLMs, you will absolutely need NVIDIA’s proprietary drivers.

1.2 NVIDIA drivers ≈ nvidia*.ko + libcuda.so

As I mentioned above, the NVIDIA drivers are kind of the foundation on top of which the CUDA Toolkit will sit. The CUDA Toolkit is what actually provides your algorithms with access to GPU-based computations. However, installing the CUDA Toolkit is the next step—it comes AFTER installing the NVIDIA drivers.

This section is dedicated to kernel modules; when it comes to installing drivers for something on Linux, it’s technically about installing (a.k.a building) kernel modules. You can even "physically" find them on your system. They have the .ko extension (.ko stands for Kernel Object) and live in the kernel-related directories, typically under /lib/modules/$(uname -r)/ (remember, uname -r displays your system's kernel version).

However, in the case of NVIDIA driver installation, it’s not just about installing the nvidia*.ko modules (yep, there’s more than one). There’s another important piece that works closely with those NVIDIA kernel modules: the user-space libraries (NB! it is not about users of PC, it is about stuff outside kernel-space). The most important one is libcuda.so (.so stands for Shared Object). And again—please don’t get confused. Yes, it’s called "CUDA", but it’s not the CUDA Toolkit.

The key takeaway here is that NVIDIA driver installation should be done sensibly - you want the NVIDIA .ko kernel modules and the NVIDIA .so libraries to communicate properly. If you install the drivers in a messed up way - like making multiple installation attempts without properly cleaning up after failed ones - things can easily get twisted and broken.

NVIDIA drivers ≈ nvidia*.ko + libcuda.so
BUT!
libcuda.so != CUDA Toolkit

2. NVIDIA documentation & NVIDIA recommended drivers

In my experience, when it comes to installing NVIDIA drivers, the most common approach people take is going straight to the official NVIDIA website, finding the driver that matches their GPU model, downloading SOMETHING (it is never a kernel module, driver file or something close - it is usually an installer script, file with extension .run), and then trying to execute downloaded file to install drivers.

And it’s usually at that step the frustration starts. Even though the installer provides a TUI (text-based user interface) and tries to guide you through, it starts throwing all kinds of technical prompts you have to manage. If you don’t fully understand what the installer is actually asking, it can be devastating. Don't worry I got you covered.

If you go to the official NVIDIA site on the Download The Official NVIDIA Drivers | NVIDIA page, you’ll be prompted to manually enter your NVIDIA device model. Here’s what’s suggested for my NVIDIA GeForce video card:

The second screenshot shows only 2 items from a long list of 10+ "recommended certified versions." The issue is that they aren’t ordered by version—570.x.x.x is followed by 535.x.x.x, and then it jumps back to 550.x.x.x. The confusion starts right here.

Let’s say I just pick the first one in the list, which has a nice description:

What is an NVIDIA Recommended Driver?
This driver meets the quality levels applied to Windows drivers that pass testing in Windows Hardware Quality Labs (WHQL), therefore providing the same attention to driver reliability, robustness, and performance for non-Windows operating systems (e.g., Linux).

Well that's nice that it was tested in some Windows Hardware Quality Labs, but I am searching for Linux drivers...well...whatever. If I click on the View button, I’m redirected to the Download page. There, you’ll see three tabs—click on Additional Information, and that’s where the installation instructions are hiding!

Installation instructions: Once you have downloaded the driver, change to the directory containing the driver package and install the driver by running, as root, sh ./NVIDIA-Linux-x86_64-570.133.07.run

Seems very easy-peasy-lemon-squisy. But what is that .run stuff?

NVIDIA describes its .run installation file as a helper script for installation of NVIDIA drivers, as it can help you to select the correct driver version:

If you are not sure, NVIDIA provides a new detection helper script to help guide you on which driver to pick. For more information, see the Using the installation helper script section later in this post. (Source)

On Linux systems, a .run file is typically either a single binary executable or a shell script that includes a binary blob that can be installed (source). Essentially, any .run file acts as an installer script that will make system-wide changes, since it specifies that it should be executed with root permissions. As I mentioned earlier, drivers are kernel modules, so this installer will build an additional kernel module for your OS. But how exactly will be built this new kernel module(s?) - it will be just binaries, or DKMS will build a new kernel module for my system following the instructions? (If you do not understand the difference, don't worry I will cover it later in this article)

Anyway, if you don’t have a clear answer to question above, that's reason enough NOT to execute this .run file on your Debian system with root privileges without a second thought.

This isn’t about the security risks with NVIDIA drivers or the possibility of some embedded nasty stuff—it’s about your Debian stability + the future maintenance of installed drivers. Plus, it will be much more difficult to debug/uninstall this stuff if something goes wrong.

Moreover, a first paragraph of "Additional information" section on the Driver download page states this:

Note that many Linux distributions provide their own packages of the NVIDIA Linux Graphics Driver in the distribution's native package management format. This may interact better with the rest of your distribution's framework, and you may want to use this rather than NVIDIA's official package. (Source: NVIDIA driver details)

And as you can guess, in this article I will show how to install drivers as Debian's native package. However, I have added a bonus section at the end of this article where I show how I install drivers with precaution measures using NVIDIA installer .run script.

2.1 NVIDIA OpenED Source of their Drivers? O_o

For a long time, NVIDIA drivers were renown for their very close-source-ness and even their license states clearly:

You may not reverse engineer, decompile, or disassemble the SOFTWARE provided in binary form, nor attempt in any other manner to obtain source code of such SOFTWARE.

However, there’s an interesting thing — in July 2024, NVIDIA published an article on their site titled "NVIDIA Transitions Fully Towards Open-Source GPU Kernel Modules".

Apparently, this transition didn’t just start in 2024 — they actually began it back in 2022. Here’s the link to the article.

At first glance, you might think: "Wow, cool — now it must be easier to install drivers, as Linux OS can integrated them easier to their codebase".

However, it seems that it’s not like NVIDIA published the source code of their proprietary kernel modules (NVIDIA drivers) — at least from what I understand. Instead, they created a separate open-source development driver branch, where the community can contribute and all that. Here is the GitHub repository: NVIDIA/open-gpu-kernel-modules.

As I mentioned, as a bonus section to this article I will show you the process of NVIDIA drivers installation following NVIDIA documentation. However, luckily, there is much easier and more "comprehensible" way to install NVIDIA drivers - Debian way.

3. NVIDIA drivers installation: "Debian" way

The procedure for installing NVIDIA drivers is covered in detail in NvidiaGraphicsDrivers - Debian Wiki. I mentioned the "Debian way" of drivers installation, and actually it is a broad term related not only to drivers installation, and if you’re using a Debian distro, I recommend familiarizing yourself with this term. You can find many details here.

In the documentation, the section you need is "Debian 12 Bookworm" (I’m assuming you’re using this version of Debian, the latest stable one). The version of the NVIDIA drivers installed by following that guide will be 535.183.01. Yes, this version is older than the "official" driver available on NVIDIA’s website (currently, it is 570.x.x), which I mentioned earlier. However, this is a trade-off: you sacrifice a bit in terms of the driver’s newness, but you gain system stability and ease of installation.

The "Debian way" of installing NVIDIA drivers is about installing them as a couple of packages using apt package manager. However, as I mentioned, NVIDIA’s software is proprietary and historically close-source, so Debian keeps this type of third-party software in a separate component of its package repositories.

If you’re unfamiliar with how Debian packages its software, apt for you is a sort of "black box", and the sources.list file seems completely incomprehensible, I highly recommend checking out these articles before proceeding: about Debian releases, about Debian software installation - repositories and repository components.

The package(s) containing the NVIDIA drivers is located in the "non-free" component of Debian’s package repositories. The term "non-free" doesn’t mean you need to pay to use the packages found there; rather, it refers to the nature of the software, as it includes closed-source code that isn’t publicly accessible. By default, "non-free" component is excluded from the list of sources that apt uses to fetch and install packages onto your system.

Debian Way Installation Step 1: Add the `contrib` and `non-free` repository components to the list of sources for `apt`:

$ sudo vim /etc/apt/sources.list
#add contrib and non-free components and check that non-free-firmware is also listed:
deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware
#this step is important: you have to make apt aware of changes made to this file:
$ sudo apt update

NB! If you have your kernel package linux-image-* installed not from Debian _bookworm repo but from Debian backports repo, you will need to install NVIDIA drivers also from there! This is due to the various dependencies linked to kernel and its version of NVIDIA drivers package!_.

Debian Way Installation Step 2: Check if your OS is booted with Secure Boot.

Rule of the thumb - if you use a Dual Boot with Windows 11, high chances is that Secure Boot is enabled!. To be sure, run sudo mokutil --sb-state. If you see this:

$ sudo mokutil --sb-state
SecureBoot enabled

If you do not understand what the Secure Boot is about, you will need to study it. I wrote the detailed article about Secure Boot and Debian, specifically from the perspective of how it impacts kernel modules and NVIDIA drivers. As an alternative, you always can find all answers in Debian Documentation - SecureBoot - Debian Wiki.

NB! If you have Secure Boot enabled and proceed with the following steps, your installed NVIDIA drivers will NEVER load until you have them signed. The process of kernel modules signing for Secure Boot setups is out of the scope for this article!

If you don’t know how to sign kernel modules for UEFI, I strongly recommend reading the Debian documentation or the article I mentioned. Once you’ve learned how to handle Secure Boot and have new kernel modules signed, you can return here to continue with the installation steps. FYI, in this article, I am installing drivers on my system that has Secure Boot enabled.

Debian Way Installation Step 3: Check if your system uses `dracut`.

Dracut is the low-level tool that is used to create an initial image used by the kernel for preloading the block device modules (such as IDE, SCSI or RAID) which are needed to access the root filesystem, mounting the root filesystem and booting into the real system (Source). This "job" can be done not only by dracut, but also by initramfs-tools. You can find out "who" does this job for your system with dpkg -l | grep -E 'dracut|initramfs-tools'. If you see in the output dracut, you have to:

Make a dracut configuration file /etc/dracut.conf.d/10-nvidia.conf (you can actually name it anything you like, as long as it ends in .conf) with the following contents:

install_items+=" /etc/modprobe.d/nvidia-blacklists-nouveau.conf /etc/modprobe.d/nvidia.conf /etc/modprobe.d/nvidia-options.conf "

Note the spaces between quotes and characters.
The modprobe.d files referenced here will be added by the nvidia-driver package. #### Debian Way Installation Step 4: Choose a flavour - "proprietary" vs "open"

I mentioned the fact that there is ongoing transition of NVIDIA fully towards open-source GPU kernel modules. It is cool, no doubts, but in this transition period for us, users, this just adds more confusion. Here is the Debian Documentation on NVIDIA drivers installation:

Debian Documentation for Debian Stable

Which "flavour" should you choose? If before there was question: which version should you install? Now the question is: which flavour (open source/proprietary) and which version should you install?

If you're an open-source software supporter, the choice might seem pretty obvious. But at the same time, it’s worth noting that this relatively new open-source flavor of the NVIDIA driver stack is still, well... relatively new compared to the proprietary version we've been using for years. That might (or might not) mean it's less mature and potentially more prone to bugs or issues.

However, in practice, as newer driver versions come out and you get your hands on more modern GPUs, you might find that you actually do not have a choice anyway.

In NVIDIA's article about their transition to open-source kernel modules, I found two important diagrams:

Before CUDA Toolkit 12.6

Previously, using the open-source GPU kernel modules would mean that you could not use the top-level metapackage. You would have had to install the distro-specific NVIDIA driver open package along with the cuda-toolkit-X-Y package of your choice.
Beginning with the CUDA 12.6 release, the flow effectively switches places (Source)

After the CUDA Toolkit 12.6 release

CUDA Toolkit is the thingy which should be installed AFTER NVIDIA drivers installation on your system. and CUDA Toolkit version suitable for your system depends fully on the version of CUDA driver version (remember this aforementioned libcuda.so?). Here is the compatibility matrix:

CUDA Toolkit and Corresponding Driver Versions

So, according to this table, if you install NVIDIA drivers of version >=560.x.x, you must install "open" flavour if you want to install on top of them CUDA Toolkit.

Little spoiler: version 535.x of "open" falvour drivers doesn’t work on my GPU even if it is supposed to.

I say it is supposed to work, because my GPU device is in the list of supported devices by this "open" flavour drivers. How to understand if "open" flavour is an option for your GPU device?

You need to check the compatibility of your GPU with the NVIDIA's open GPU kernel modules - there’s a table here with all supported models (f you scroll down, you’ll see it).

However, just finding your GPU model name in the table isn’t enough. For example, my GPU — an NVIDIA GeForce RTX 3060 — is listed five or more times. So what you should actually be checking is:

In the below table, if three IDs are listed, the first is the PCI Device ID, the second is the PCI Subsystem Vendor ID, and the third is the PCI Subsystem Device ID.

So first, I had to find the right parameters - at least the first one- the PCI Device ID. Because for my card, in all the mentions in that table, there's only one number that mentioned - the PCI Device ID.

Here is how to identify both Vendor and Device ID:

$ lspci -nn | grep NVIDIA
06:0b:0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)

From the output above, 10de is the Vendor ID and 2504 the Device ID.

And, apparently my GPU is in the table of compatible devices:

NB! You CANNOT install both flavours - "just to test" (to switch flavour later on you will have to purge the installed one), of course, and I want to explain some stuff about what is the software level difference in-depth between these two flavours (even if installation commands look somehow alike).

NB! DO NOT EXECUTE THESE COMMANDS ONE AFTER ANOTHER, READ THE EXPLANATION BELOW FIRST, IF YOU REALLY DON'T CARE EXECUTE ANY OF THEM, BUT NOT BOTH.

To install "proprietary" flavor these commands should be run:

$ sudo apt install nvidia-driver firmware-misc-nonfree

To install "open" flavor, there is an additional package to install - nvidia-open-kernel-dkms:

$ sudo apt install nvidia-open-kernel-dkms nvidia-driver firmware-misc-nonfree

You can see that both commands install the nvidia-driver package and firmware-misc-nonfree. The latter one is related to firmware, not the driver. Don’t confuse the two - firmware and drivers are not the same thing. However, that’s not really the point here.

What actually makes the difference when installing a specific flavour is the order of the commands — and I’ll show you why. It all comes down to the dependencies each package brings in.

Debian Way Installation: DKMS is your bro when it comes to drivers

First, I want to elaborate on nvidia-open-kernel-dkms package — specifically the dkms part of this package name.

DKMS is your bro when it comes to Linux drivers (kernel modules).

Dynamic Kernel Module Support (DKMS) is a framework which allows kernel modules to be dynamically built for each kernel on your system in a simplified and organized fashion. (Source)

It is exactly DKMS that takes care of automatically rebuilding registered kernel modules (read drivers) whenever you install a different Linux kernel.

If you install some proprietary drivers on Debian without DKMS is not ideal. What’s actually happening under the hood during installation is that standalone kernel modules (those .ko files) are compiled as-is.

Leaving stability aside for a second, the first major inconvenience you’ll run some time after installation into is this: you’ll have to reinstall (rebuild) them manually every time there’s a major kernel update. And if you have Secure Boot enabled on your system, you’ll also need to sign them all manually. Every time.

So yes — DKMS is the bro when it comes to installing extra drivers (kernel modules). It builds them for you in an organized way.

NB! For experienced folk: I know, that I am generalising — I’m aware that sometimes DKMS can turn into the pure evil and nuke stuff, especially because it’s way too eager to rebuild stuff the moment it sees kernel headers updated.
However, IMHO, for regular desktop/home use on stable systems like Debian, DKMS is still bro in the end.

Chances are your OS doesn’t have the dkms package installed — meaning, no DKMS tool at all. As far as I know, DKMS doesn’t come preinstalled.

DKMS relies heavily on Linux headers, so I want to explain what those are.

Debian Way Installation Step: About Linux headers

If you're using your Debian system the "Debian way", you're probably installing packages mostly from the official Debian repositories. Since these packages (including the kernel) come from Debian’s repos, they’re built and tested to work well together.

The Linux kernel itself comes in the linux-image-* package. After installation, you get a bunch of preinstalled drivers — which are essentially kernel modules — that make various pieces of your hardware work: Wi-Fi sticks, Bluetooth headsets, etc.

These preinstalled drivers are actually part of the upstream Linux kernel. They are separate kernel modules, sure, but they communicate with the kernel seamlessly because they’re written and optimized to do exactly that.

However, when it comes to external, third-party kernel modules (third-party means that the source code was written by non Debian/Linux devs)— for them, your kernel is somehow a black box. These modules can’t just "communicate" to it directly. That’s where Linux headers come in.

Linux headers can be roughly described as the programming interface needed to interact with the Linux kernel. If a module needs to communicate with the kernel, headers are required to build and install it — and to make it work at all.

So here's a basic schematic of how it all fits together:

DKMS<-->linux-headers* package<-->Kernel (linux-image-* package)<--> Your Debian OS

So when you install the package nvidia-open-kernel-dkms, it must pull in headers and dkms as dependencies. You can explore the full dependency chain here.

`nvidia-kernel-dkms` OR `nvidia-open-kernel-dkms` <-- install kernel space components of NVIDIA drivers; `libcuda1` and other libraries (`lib*`) <-- install user-space components of NVIDIA drivers, a.k.a CUDA drivers

But now you might be wondering:
"If I want to install the proprietary flavour instead, will I just get some sad, standalone kernel modules built without DKMS?"

The answer is NO. nvidia-driver package as dependencies shall bring you something like nvidia-kernel-dkms (notice, non open).

$ sudo apt install --simulate nvidia-driver 

Installing:                     
  nvidia-driver

Installing dependencies:
  dkms                     libnvcuvid1                libnvidia-rtcore          nvidia-persistenced
  firmware-nvidia-gsp      libnvidia-allocator1       nvidia-alternative        nvidia-settings
  glx-alternative-mesa     libnvidia-cfg1             nvidia-driver-bin         nvidia-smi
  glx-alternative-nvidia   libnvidia-egl-gbm1         nvidia-driver-libs        nvidia-support
  glx-diversions           libnvidia-egl-wayland1     nvidia-egl-common         nvidia-suspend-common
  libcuda1                 libnvidia-eglcore          nvidia-egl-icd            nvidia-vdpau-driver
  libegl-nvidia0           libnvidia-encode1          nvidia-installer-cleanup  nvidia-vulkan-common
  libgl1-nvidia-glvnd-glx  libnvidia-glcore           nvidia-kernel-common      nvidia-vulkan-icd
  libgles-nvidia1          libnvidia-glvkspirv        nvidia-kernel-dkms        update-glx
  libgles-nvidia2          libnvidia-ml1              nvidia-kernel-support     xserver-xorg-video-nvidia
  libgles1                 libnvidia-pkcs11-openssl3  nvidia-legacy-check
  libglx-nvidia0           libnvidia-ptxjitcompiler1  nvidia-modprobe

Suggested packages:
  menu  nvidia-cuda-mps

Let me give you a comparison. Let's say you want to install the Brave browser, and you already have Firefox installed. When you installed Firefox, it brought in a bunch of dependencies needed for a browser to function on your freshly installed Debian. Brave is also a browser — and even though it definitely has its own unique dependencies (maybe fewer, maybe more), it may still rely on some of the low-level stuff that Firefox already brought in. So when you install Brave, it might skip installing those dependencies because they’re already there.

It can also go the other way: Brave might be able to use low-level software A or low-level software B to function, and it just happens to find the low-level software B Firefox brought in. So Brave uses that instead of pulling in the low-level software A. But if you had installed Brave first, it would have brought in low-level software A. :3

Same logic applies here with the NVIDIA drivers.

If the nvidia-driver package doesn't find any existing NVIDIA kernel modules already installed, it will go ahead and install everything — including the kernel-level stuff.

But if you already installed the kernel modules beforehand (through nvidia-open-kernel-dkms), and then you install nvidia-driver, it will bring in just the user-space libraries.

$ sudo apt install --simulate nvidia-open-kernel-dkms nvidia-driver

Installing:                     
  nvidia-driver  nvidia-open-kernel-dkms

Installing dependencies:
  dkms                     libnvcuvid1                libnvidia-rtcore            nvidia-settings
  firmware-nvidia-gsp      libnvidia-allocator1       nvidia-alternative          nvidia-smi
  glx-alternative-mesa     libnvidia-cfg1             nvidia-driver-bin           nvidia-support
  glx-alternative-nvidia   libnvidia-egl-gbm1         nvidia-driver-libs          nvidia-suspend-common
  glx-diversions           libnvidia-egl-wayland1     nvidia-egl-common           nvidia-vdpau-driver
  libcuda1                 libnvidia-eglcore          nvidia-egl-icd              nvidia-vulkan-common
  libegl-nvidia0           libnvidia-encode1          nvidia-installer-cleanup    nvidia-vulkan-icd
  libgl1-nvidia-glvnd-glx  libnvidia-glcore           nvidia-kernel-common        update-glx
  libgles-nvidia1          libnvidia-glvkspirv        nvidia-legacy-check         xserver-xorg-video-nvidia
  libgles-nvidia2          libnvidia-ml1              nvidia-modprobe
  libgles1                 libnvidia-pkcs11-openssl3  nvidia-open-kernel-support
  libglx-nvidia0           libnvidia-ptxjitcompiler1  nvidia-persistenced

Suggested packages:
  menu  nvidia-cuda-mps  nvidia-kernel-dkms  | nvidia-kernel-source  | nvidia-open-kernel-source

And no, this has nothing to do with your user’s home folder or /home or anything like that. This goes much deeper — we're talking about memory spaces: kernel space vs user space.

The key point here is that in this context, kernel space always comes first — the kernel modules are the base layer. The user-space components (like libraries and tools) attach themselves to what's already there in kernel space.

_NB! Potential culprit if you have installed NVIDIA drivers but they do not work - your NVIDIA kernel modules versions should be aligned with your nvidia-driver version, you can do analysis with dpkg -l or apt list - - installed and grep nvidia

So, actually, your flavour choice all boils down to two packages — only one of which will be installed and used on your Debian:
nvidia-kernel-dkms or (its dependency alternative) nvidia-open-kernel-dkms.

Installing "proprietary" flavour, version 535.x.y - from LTS branch

I have installed "proprietary" flavor because as I mentioned even though my GPU is in compatibility matrix with "open" flavor of 535.x. driver version does not work. I will show you error I get later.

Before dealing with anything related to drivers I ALWAY create a snapshot of my system. I use Timeshift tool for this. You can check this my article on snapshotting with Timeshift.

To purge all existing nvidia-packages installed on your system you can try something like that:

# to list nvidia-related packages installed previously with apt
$ apt list --installed | grep nvidia
$ sudo apt --purge remove '*nvidia*'
$ sudo apt autoremove

Installation:

$ sudo apt install nvidia-driver firmware-misc-nonfree
# NB!!!If your linux-image-* package is from bookworm backports, you must install nvidia-driver package from this repo to avoid broken dependencies error!

If you did not blacklist nouveau drivers before installation, at certain point during installation you will see a warning message in the terminal (likely on the blue background):

Conflicting nouveau kernel module loaded
The free nouveau kernel module is currently loaded and conflicts with the non-free nvidia kernel module
The easiest way to fix this is to reboot the machine once the installation has finished

So, you just need to press Enter to proceed, the solution is the reboot - once installation has finished, reboot your system (sudo reboot).

When you are booted again, run nvidia-smi command.
This is my output:

If you see something similar (and no errors), congratulations! You have successfully installed the NVIDIA drivers. For additional confirmation, you can list your PCI devices and the kernel modules in use (loaded drivers) by running lspci -k | grep -A 3 NVIDIA.

Here are NVIDIA kernel modules built by DKMS with love for you:

$ sudo modinfo -n nvidia
/lib/modules/6.12.21-amd64/updates/dkms/nvidia.ko

$ ls /lib/modules/6.12.21-amd64/updates/dkms/
nvidia-drm.ko  nvidia.ko  nvidia-modeset.ko  nvidia-peermem.ko  nvidia-uvm.ko

$ ls /usr/lib/x86_64-linux-gnu/ | grep libcuda
libcudadebugger.so.1
libcudadebugger.so.535.183.01
libcuda.so
libcuda.so.1
libcuda.so.535.183.01

4. How and when to update NVIDIA drivers

This section is not about NVIDIA driver VERSION updates, because even other Debian releases (Unstable, Testing) offer the SAME major NVIDIA driver version—535.x—just like Bookworm repo. So if you’re not planning to downgrade (which is another story entirely), I don’t really see any "debian source" for VERSION update.

However, I do want to discuss a different kind of NVIDIA driver update—not about the version, but about the kernel module update itself.

If this question still did not pop up in your mind, I will anticipate it: "Since NVIDIA drivers are kernel modules, what happens to them then when Debian kernel version updates?"

It can be quite common that when you start using Debian, one of the first things you do is install NVIDIA drivers. But eventually, if you’re using Debian Stable release, you might encounter the need for a more major kernel version—for example, in my case, it was for Wi-Fi USB adapter drivers. So you learn a bit about backports and update the kernel from there - not just small regular updates from Debian Stable repositories (like from 6.1.10 to 6.1.11), but literally jumping from kernel v6.1.x to v6.9.x or even v6.12.x. That’s a major jump, and there’s a good chance your NVIDIA driver won’t work properly. Why? Because of Linux headers.

As I mentioned, headers are really important for NVIDIA drivers to work on your system. They serve as an API or interface that the NVIDIA driver uses to communicate with the kernel. When you update the kernel with sudo apt install -t bookworm-backports linux-image-amd64 (to install the newest kernel version from backports), it will only install the kernel and not the headers! You have to install the headers manually, and they must match your kernel version exactly. So if you install linux-image-amd64 from backports, you also have to install the headers in the same way (linux-headers-amd64). If you use a more detailed command pointing to the exact version of linux-image* package (e.g. sudo apt install -t bookworm-backports linux-image-6.9.7+bpo-amd64), the headers should match it (e.g. sudo apt install -t bookworm-backports linux-headers-6.9.7+bpo-amd64)—they’re always paired.

Here’s the trick with NVIDIA driver updates: if you installed them the Debian way, they were built with DKMS. This nice tool will rebuild (read "update") your NVIDIA drivers, and the trigger for that is installing the linux headers. During the final steps of their installation, you’ll see in log that DKMS is rebuilding stuff (and not just NVIDIA drivers, but also other kernel modules-drivers you installed in a way that they were built with DKMS).

So, this is yet another reason why it’s better to install NVIDIA drivers in the Debian way, or at least make sure that whatever you install manually is built with DKMS and not just scrapped together with some script.

However, keep in mind, that a VERY major kernel update (for example, when kernel gets updated from version 6.x to 7.x) may require installing a newer NVIDIA driver, as changes to the kernel APIs may not be compatible with the existing driver version.

5. BONUS SECTIONS

5.1 Installing driver in NVIDIA suggested way (with `.run` file)

I downloaded my "driver" (.run file) for my GPU card model from this section of NVIDIA Official website using:

$ wget https://us.download.nvidia.com/XFree86/Linux-x86_64/570.133.07/NVIDIA-Linux-x86_64-570.133.07.run

The search prompt where you can enter your OS, GPU model, etc., is here.

A link to the README instructions is available under the Additional Information tab on the download page of a driver.

Chapter 4 of this README is dedicated to installation of NVIDIA driver with this .run executable.

The .run file is a self-extracting archive. When executed, it extracts the contents of the archive and runs the contained nvidia-installer utility, which provides an interactive interface to walk you through the installation.

And here is about how Nvidia driver installation will communicate with your kernel:

When the installer is run, it will check your system for the required kernel sources and compile the kernel interface. You must have the source code for your kernel installed for compilation to work. On most systems, this means that you will need to locate and install the correct kernel-source, kernel-headers, or kernel-devel package; on some distributions, no additional packages are required.

In a default Debian setup (the set of packages you get pre-installed), you’ll only have the package linux-image-* (which contains kernel itself and default kernel modules-drivers, as it was mentioned above)—without the kernel headers or developer libraries preinstalled.

So, naturally, the installation process for NVIDIA drivers requires having Linux headers. However, Linux headers aren’t the only requirement for installation. And the README file contains all the additional instructions.

The important section to pay attention to—which can cause a very unpleasant experience for your PC management—is if this .run script does not find DKMS (Dynamic Kernel Module Support) installed on your system:

The installer will check for the presence of DKMS on your system. If DKMS is found, you will be given the option of registering the kernel module with DKMS. (README)

If this .run script doesn’t find it—will it prompt you to install it, give a warning, fail, or install it for you? I really have no idea.

So, if the NVIDIA proprietary .run installer doesn’t find DKMS installed on your system (dpkg -l| grep dkms), it will probably compile just standalone kernel modules for each component of their drivers—meaning you’ll have to rebuild them manually at every major kernel update. And if you have Secure Boot enabled on your system, you’ll also need to sign them all manually.

So, first I install DKMS and Linux headers. The dkms package itself pulls in the linux-headers-* package as a dependency. However, be cautious! If you've made any modifications to your kernel (like an update or whatever), make sure to double-check that your Linux headers version match the exact version of your kernel — down to every number and dot.

For example, in my case, my kernel version is:

$ uname -r
6.1.9-32-amd64

I install linux-headers-* in this way to ensure the exact match with my kernel version:

$ sudo apt install linux-headers-$(uname -r)
# then, I install DKMS
$ sudo apt install dkms
# I double check the version of linux headers
$ dpkg -l | grep linux-headers*

(If you see more than one version of headers, it’s probably because you updated the kernel in the past — most likely automatically with sudo apt update && sudo apt upgrade. The most important thing is that in the output you see headers with version that corresponds exactly to the current version of the kernel. You can actually remove older versions of the Linux headers used for previous kernel versions if you don’t need them anymore.)

I downloaded the .run file, and now I’m executing it, following these mini instructions:

Installation instructions: Once you have downloaded the driver, change to the directory containing the driver package and install the driver by running, as root, sh ./NVIDIA-Linux-x86_64-570.133.07.run (https://www.nvidia.com/en-us/drivers/details/242273/)

$ sudo sh ./NVIDIA-Linux-x86_64-570.133.07.run

The first thing I am shown in Terminal UI is the choice of flavour again, and as I mentioned, the "open" flavour from the Debian repo didn’t work. Eh, let’s give it a try here!

Okay, right after first complaint: it needs a C compiler (gcc), so I install it with:

$ sudo apt install build-essential
# to check:
$ gcc -v

Nice, as next, it complains about the nouveau driver currently in use by my system:

I press --> Ok

I press --> Yes

There are some scary words in these commands if you’ve never dealt with it before, but it’s not as bad as it sounds. What the NVIDIA installer is asking for is to first unload the nouveau kernel module. This isn’t just because it’s installed and present — it’s actually in use, meaning your NVIDIA GPU is currently serving it, and it can’t be responsive to NVIDIA drivers.

modprobe configurations are essentially files that get dropped into /etc/modprobe.d/. You’ll get one or two text files with a few simple lines — probably something like blacklist nouveau. Then, the initramfs will be updated (rebuilt) so that it knows not to load nouveau on the next boot, but rather the NVIDIA drivers for the kernel.

The configuration files in the /etc/modprobe.d directory will be read during the next boot, and the blacklisting will be respected.

So actually it tells you what I explained above;
If you want to revert changes later you can just cancel these written files from /etc/modprobe.d

I press -->

I press --> (as an alternative I could do Abort and reboot but I am lazy)

Installer starts to build kernel modules...

--------------You can skip all stuff related to keys, signing if your setup does not have Secure Boot Enabled!-------------------------

I press --> < Sign the kernel module >

I press --> (I have generated a keypaor for signing and enrolled MOK key)

Full path to private key:

Full path to public key:

----- Secure Boot related prompts finsihed ------------

Next, warning about 32-bit libraries:

Next, warning about the fact that some user-space library cannot be installed:

Then, I press --> "Rebuild initramfs"

Okay, NVIDIA kernel modules are built.
Only now, the installation of user-space libraries starts:

Then, I select that I do want it writes X configuration file (If you use Wayland, select No!):

Finally, Installation is completed.

I am informed that I have to do reboot.

Well, apparently it works!

Here is the nvidia-smi output (though, with some strange log messages (?):

$ nvidia-smi
Mon Apr 14 18:47:38 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:01:00.0  On |                  N/A |
|  0%   28C    P8             11W /  170W |     845MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Summarizing: If you don’t fully understand these instructions and the steps involved, then installing NVIDIA drivers this way might not be for you. You can, of course, experiment (snapshotting tools are great for these kinds of experiments).

5.2 Installing NVIDIA drivers on Debian Testing with KDE Plasma that is running on Nouveau drivers.

If you didn't know, the KDE Plasma Desktop Environment can run on both the X display server and the Wayland display server. If you prefer the X display server and use it, this section is not for you. You just install the drivers, reboot, and your KDE X session should be intact (the session selector is in the bottom left corner on the login menu). The NVIDIA drivers installation process blacklists the Nouveau drivers, so after rebooting, the KDE desktop environment should start with the NVIDIA drivers.

However, if you use Wayland and stick with it, you'll need some extra configuration. Right after installing the NVIDIA drivers, your KDE Wayland session might break temporarily until you make a couple of modifications. It's not critical - for example, I installed the NVIDIA drivers, rebooted, and my Wayland session failed. I switched to an X session, made the necessary modifications, rebooted, and after that, my KDE Wayland session started smoothly.

Here is some info about one of my setups with KDE:

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux trixie/sid"
NAME="Debian GNU/Linux"
VERSION_CODENAME=trixie
# My KDE setup is on Debian Testing (Debian 13!)
$ plasmashell --version
plasmashell 6.3.4
$ nvidia-smi
| NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8  
# yep I am a baddy, I installed drivers in NVIDIA recommended way :3

So, I installed NVIDIA drivers, rebooted and logged into KDE X session (Wayland session was not starting). Then, I just followed steps from Debian documentation on NVIDIA drivers installation, section on Wayland support.

$ sudo vim /etc/default/grub.d/nvidia-modeset.cfg
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX nvidia-drm.modeset=1"

$ sudo update-grub

$ sudo apt install nvidia-suspend-common

$ sudo systemctl enable nvidia-suspend.service
$ sudo systemctl enable nvidia-hibernate.service
# This command will fail if you did not reboot after nvidia drivers installation and nouveau drivers are still in use by your Debian
$ sudo systemctl enable nvidia-resume.service

$ cat /proc/driver/nvidia/params | grep PreserveVideoMemoryAllocations
PreserveVideoMemoryAllocations: 1

If this parameter is set to zero, you should be able to override it by adding a configuration into modprobe.d (assuming the file doesn't already exist):

$ sudo vim /etc/modprobe.d/nvidia-power-management.conf
options nvidia NVreg_PreserveVideoMemoryAllocations=1

$ sudo reboot

After reboot, Voila!

Debian Secure Boot: To be, or not to be, that is the question!

Anna — Thu, 28 Nov 2024 22:30:14 +0000

While all my articles are about Debian, this article can still be useful for other Linux OSs since it’s focused on the logic behind Secure Boot.

NB! In this article, I will show how to install NVIDIA drivers on a setup with Secure Boot enabled. However, I will not go into details about the NVIDIA driver installation peculiarities. I have published the separate article on this topic. NVIDIA drivers in this article serve as an example of how to install kernel modules signed for Secure Boot.

🔮🧙‍♀️: If you’re here reading this article, it’s because:
A. You’ve encountered a huuuuugeeee problem with your Linux OS because of Secure Boot (most probably your problem is goodbye 🫡 nvidia kernel module (a.k.a driver) after booting with Secure Boot enabled).
B. You’re a security freak and firmly believe that if there’s a security mechanism, it is absolutely UNACCEPTABLE not to use it on your OS! <--- I am here.

Here are some alternative scenarios that might lead you to start googling "How to Enable Secure Boot on OS X (Debian, I hope)":

So, you’re using your OS, enjoying it, 🦄🌈… and then something happens that pushes you to start digging through forums about Secure Boot. Two possible scenarios:

You’re using a dual boot setup with Windows and recently upgraded to Windows 11. Windows 11's minimum requirements include Secure Boot. As a result, you disable Secure Boot in the BIOS to log into your Debian system, then re-enable it in the BIOS to log back into Windows. And you are fed up doing this.
To install and play a video game (like Valorant, FIFA), your system needs Secure Boot enabled. In this case, Secure Boot acts as part of the game’s anti-cheat mechanisms.

Anyway, I guess you have your reasons to read this article, so here is the roadmap:

1 Understanding what Secure Boot actually is:

1.1 What is UEFI
1.2 UEFI vs BIOS
1.3 Secure Boot: myths

2 What are Kernel Modules and their origins: Linux Kernel Upstream, DKMS and third party

3 Secure Boot Setup:

3.1 Generating your own signatures
3.2 Enabling Secure Boot
3.3 About MOK and how to manage it

4 Configuring DKMS to automatically sign kernel modules

4.1 "Outsource" signing keys generation to DKMS
4.2 Creating and Enrolling your own Machine Owner Key; pointing DKMS to use it

5 Using your MOK to sign installed kernel modules manually

1. What is Secure Boot?

UEFI Secure Boot (SB) is a verification mechanism for ensuring that code launched by a computer's UEFI firmware is trusted. It is designed to protect a system against malicious code being loaded and executed early in the boot process, before the operating system has been loaded. (Source)

Let's decompose it. First, what is UEFI?

1.1 About UEFI

UEFI stands for Unified Extensible Firmware Interface.

Unified ...

If you have more than one OS on your PC and not even in dual boot (when more than ONE OSs reside on the same storage device disk), you will still see in BIOS* ONLY ONE storage device to boot from. That is because UEFI has one bootable partition (ESP) ** for the whole PC**. Usually, it is created when you install the first OS. And then if you install something on additional disk —the booting will be done from the first disk where the firstly installed OS resides. So, UEFI should be intelligent enough about loading firmware based on the choice where you want to boot. For example, UEFI will have to load one set of firmware for booting Debian and another for Windows.

...Firmware Interface.

So it is about firmware, as you can guess. Why do we talk about firmware? Is it UEFI who messes up your Bluetooth or Wi-Fi dongles? No, it is not that.

First, a quick reminder: what is firmware? Firmware refers to embedded software which controls electronic devices. So your PC is just a bunch of electronic devices — hardware — CPU, motherboard, video card, SSD disks, some ports like USB ports with something, or ports for cables. You plug in your PC, so all these guys get energy to run and do something... but do what?

When you press the power-on button, all these guys do not just start running and doing something. First, of course, they are having a kind of brain—it is the CPU. But what should the CPU do? Okay, you power-on your PC. If you press some combination of keys that depends on your motherboard brand, you GIVE INSTRUCTION to enter BIOS*. Then in BIOS*, you give instruction from which storage device (disk) to boot.

Okay, you selected to boot from SSD1 where you know you have your Debian OS. You exit BIOS* and wait for booting to finish. But what is going on there behind? WHO is instructing what to do, which hardware to use, and the sequence of actions which should be performed to bring UP your OS? Your OS has a bootloader which is stored in /boot directory/partition. So UEFI just executes the "instructions" found there AND provides an access to firmware to employ hardware for this boot process.

I guess you know that the core of the Linux OS is the Linux kernel. Once launched by UEFI, it takes control over UEFI and makes the rest to boot your OS. Key word related to Secure Boot here is "once launched". If UEFI refuses to execute/launch the kernel of your OS, you are not booting anywhere.

Important to know that when you just freshly install Debian, it has ONLY the kernel gently and with love packed for you by Debian devs. BUT eventually, when you start to perform post-installation configuration... you very probably will start installing some kernel modules that are like _additions to your system's Linux kernel that ALLOW you to use some hardware IN A DIFFERENT way from the default one defined by Linux kernel. Sometimes it is about allowing you to use a hardware piece in general, because it is not possible for some reason within default Linux kernel._

When you install such additional kernel modules, they should also be launched during the booting process together with your OS's kernel.

Coming back to the argument of this article, it is UEFI Secure Boot who can completely block your booting process by refusing to load your OS's kernel, or in the most common scenario, it just refuses to load kernel modules that you have installed additionally.

1.2 UEFI vs BIOS

If you noticed earlier, whenever I mentioned BIOS, I was adding a ** * **. Now, I’d like to explain why, to avoid any confusion. You see, the term "BIOS" has become a widely used word, often associated with a specific tool that you access mainly to change the booting process—like booting from an installation medium with a new OS or changing the boot order. Over time, the word "BIOS" stuck and is now commonly used in this context. However, from a technical standpoint, it’s not accurate anymore to call this "startup tool" BIOS. BIOS stands for Basic Input/Output System.

The BIOS in older PCs initializes and tests the system hardware components (power-on self-test or POST for short), and loads a boot loader from a mass storage device which then initializes a kernel. (Source)

However, BIOS is a legacy (term used for an "outdated" software that is still in use) for years. If you’re using a relatively modern machine, what you’re actually using is UEFI—the successor to BIOS, designed to overcome its technical limitations. Some operating systems, like Windows 11, have completely dropped support for BIOS firmware and won’t run on older hardware that relies on it. Additionally, all Intel platforms have also discontinued support for BIOS entirely. (source).

When you access the UEFI BIOS on your PC, you can manage via minimalistic graphical user interface (GUI) rather than the terminal-like interface that legacy BIOS had. You can navigate using a mouse, and there might even be graphical metrics displaying your PC's health and performance. At first glance, you might think UEFI is just the "bad guy" who slapped a GUI onto the old BIOS while introducing complexities like Secure Boot or the frustration of a single boot partition (ESP) that impedes you to have "all parts" of OS together on one disk including bootloader - for example, when running multiple operating systems on one PC, UEFI rules out that all bootloaders are stored on the same disk, where the first OS you installed resides.

However, that negative perception of UEFI isn’t accurate. Legacy BIOS, being a pretty old technology, had significant limitations that made it unsuitable for modern hardware. For instance, it couldn’t handle large storage devices (beyond 2TB), relied on MBR instead of GPT partitioning, and made recovery efforts more challenging when things went wrong. BIOS simply couldn’t meet the demands of modern PCs. So, even if it is possible for you to stick with BIOS, there’s really no reason to cling to such an outdated technology.

By now, you’ve probably realized that Secure Boot is an exclusive feature of UEFI—it’s simply not possible with BIOS.

1.3 Secure Boot: myths

First, a more technical elaboration on what Secure Boot actually is:

Secure Boot (SB) works using cryptographic checksums and signatures. Each program that is loaded by the firmware includes a signature and a checksum, and before allowing execution the firmware will verify that the program is trusted by validating the checksum and the signature. When SB is enabled on a system, any attempt to execute an untrusted program will not be allowed. This stops unexpected / unauthorised code from running in the UEFI environment. (SecureBoot - Debian Wiki)

Myth 1: Secure Boot is a sneaky tool developed by Microsoft to block the use and spread of Linux OS.

This perspective can be found all over Google, and the myth might even seem plausible if you explore your UEFI BIOS settings to enable Secure Boot. For instance, on my ASUS motherboard, enabling Secure Boot requires changing the OS type to "Windows OS." By default, it’s set to "Other OS," which essentially means Secure Boot is disabled.

[How to enable Secure Boot on ASUS motherboard](https://www.asus.com/support/faq/1049829/)

Even if I do not use Windows on my PC, my OS type is set to "Windows OS" to have Secure boot enabled. How does Windows OS or Microsoft enter into the booting process when you enable Secure Boot? I am continuing to quote the Debian documentation on Secure Boot:

Most x86 hardware comes from the factory pre-loaded with Microsoft keys. This means the firmware on these systems will trust binaries that are signed by Microsoft. Most modern systems will ship with SB enabled - they will not run any unsigned code by default. Starting with Debian version 10 ("Buster"), Debian supports UEFI secure boot by employing a small UEFI loader called shim which is signed by Microsoft and embeds Debian's signing keys. This allows Debian to sign its own binaries without requiring further signatures from Microsoft. Older Debian versions did not support secure boot, so users had to disable secure boot in their machine's firmware configuration prior to installing those versions. (SecureBoot - Debian Wiki)

Debian developers dealt with Microsoft signatures and Debian OS has its UEFI bootloader signed. This means you don’t need to deal with Microsoft to register anything or go through any registration/login ecc processes.

If you’re using the default Debian installer and an "official" Debian Linux kernel (from Debian package repos), Secure Boot shouldn’t pose any problems for booting. However, if your OS doesn’t support Secure Boot or its UEFI loader isn’t signed with Microsoft’s certificates—or if you’re running a custom kernel—this will obstacle the booting process until you have these custom components "signed". However, again, this doesn’t mean that Microsoft is anyhow involved in such signing process.

Secure Boot is about signatures in general, ensuring that only trusted and verified software can run during the boot process.

UEFI Secure Boot is not an attempt by Microsoft to lock Linux out of the PC market here; SB is a security measure to protect against malware during early system boot. Microsoft act as a Certification Authority (CA) for SB, and they will sign programs on behalf of other trusted organisations so that their programs will also run. There are certain identification requirements that organisations have to meet here, and code has to be audited for safety. But these are not too difficult to achieve. (SecureBoot - Debian Wiki)

Myth 2: Secure Boot is a security measure you should only consider enabling and configuring if your personal setup is at considerable risk of being physically accessed by unauthorized individuals with the intent to access your data with workaround by booting from external medium or infect your system with malicious kernel/kernel components.

While this statement is partially true—because Secure Boot does indeed protect against physical threats—it’s not limited to that. As I mentioned earlier, one of the reasons you might have found this article is because your NVIDIA drivers aren’t loading after enabling Secure Boot. So, how does that relate to physical access? No one physically implanted those drivers into your system, I assume that you installed them normally while your OS was running. Yet, they’re still being blocked from loading.

That’s exactly what Secure Boot is about: verifying that everything loaded during the boot process (and sometimes even after) is trusted and properly signed. It’s not just about protecting against unauthorized physical access.

Think of it like going to a bank to request a new product. If the bank is reputable, that’s one thing, but in some cases, a bank can overwhelm you with a pile of documents to sign just to get the desirable bank's product. Buried within those documents—perhaps on page N of the nth document—might be a clause stating you agree to an additional product, like a credit card with a sky-high interest rate. The moment you sign, that agreement with the bank takes effect. So, it’s always a good idea to read the details carefully before signing anything.

Secure Boot works in a similar way. UEFI will only load kernels or kernel modules that are signed—anything else gets blocked. Disabling Secure Boot, on the other hand, is like walking into a bank where all their products take effect automatically, no signatures required, just because you asked for them. This lack of "signing step" opens the door to risks.

Above, I’ve mentioned the term kernel modules several times in relation to NVIDIA drivers. But are all your system's drivers kernel modules act and installed in the same way? No. And this is where a potential security gap arises, one that Secure Boot can help protect against. If you’re not sure what exactly you’re installing at certain moment—whether it’s just application software or a kernel module—Secure Boot can step in as a protection layer.

With NVIDIA drivers, it’s a one-way street: you need them to function, so whether they are kernel modules or not, you’re going to install them. But let’s look at another example: imagine your Bluetooth, touch screen, or some other hardware on your laptop isn’t working on a Linux OS. You hit the forums, dig through solutions, and start following instructions: “install this,” “run this command,” “clone this repo,” “build it with make,” and so on. If you don’t fully understand what you’re doing, you’re exposing your system to risks.

Secure Boot won’t protect you from malicious applications you might install, but it will protect your system from malicious kernel additions. If a suspicious or unsigned kernel module tries to load, Secure Boot will block it—end of story.

So, this refusal triggered by Secure Boot can act as a flag, signaling that what you have installed is a kernel module. This gives you the chance to investigate further, understand what you actually installed, and make an informed decision about whether to sign it and enable it or leave it blocked for your safety.

2. What are Kernel Modules and their origins: Linux Kernel Upstream, DKMS and third party

Understanding what are the Kernel Modules and where they are coming from is the key in understanding how to make all your OS's components compliant for Secure Boot so they do not fall apart on booting step.

Repeating: when it comes to hardware devices, it’s about drivers, not just software or apps. And drivers on Linux are kernel modules:

Linux device drivers come in the form of kernel modules - object files which may be loaded into the running kernel to extend its functionality. (Debian: Managing the kernel modules)

In case of Debian, where do these drivers come from? Who adds them to the OS?

_Under Debian, the module can be installed from three different kind of sources:

Upstream Linux kernel modules: Those are shipped in the linux-image-* kernel packages.

Extra modules, that's aren't in the upstream Linux kernel. Those are usually built using dkms. The available modules can be listed by running apt rdepends dkms.

Others, like third party, proprietary and other or binary blobs modules... You should not install such modules on your system except when you have no other choice. (Modules - Debian Wiki)_

Most of your PC's hardware relies on drivers from the upstream Linux kernel modules, which are included in the linux-image-* Debian package, which is installed during OS installation, has a version as any other packages and can be upgraded.

Them, there's the DKMS source of kernel modules. NVIDIA drivers, for example, fall into this category. They are third-party kernel modules built using DKMS (Dynamic Kernel Module Support), so they aren’t included directly in the upstream kernel, but neither they are just binary blobs. This makes them part of the second category, not the third, in the list of driver sources I mentioned above.

What is DKMS?

DKMS (Dynamic Kernel Module Support Framework) is a framework designed to allow individual kernel modules to be upgraded without changing the whole kernel. It is also very easy to rebuild modules as you upgrade kernels (Debian - Details of package dkms in bookworm).

Kernel modules in this category can be found in Debian repos, however maybe not always in main component of repo, but also in such components as non-free, contrib and non-free-firmware. (If you do not know how Debian ships its software and you do not know what are Debian repos and how to manage them you can check these articles 1, 2). In most cases, you can find drivers that can be build with DKMS in Debian repos.

Third-party binaries that build kernel modules for you may come on your horizon when some hardware pieces of your machine don’t work on your Debian setup, and drivers aren’t available in Debian’s package repositories. This often happens with Intel wireless adapters, Realtek Wi-Fi chipsets, Bluetooth devices, input devices like keyboards or mice, USB storage devices, and so on.

However, when a hardware piece does not work/function well on your PC, it’s important to understand this difference: driver ≠ firmware! If firmware is missing on your OS, Debian usually informs you (Warning message in dmesg, journalctl, or when you update your kernel. If Debian informs you, that means that it did not manage to fetch it from any repo you have enabled for apt. You may need to add additional repositories (like bookworm-backports) or add components to the stable package repo (contrib, non-free, non-free-firmware). As a last resort, you can fetch a missing firmware directly from the Linux repository(example of available firmware for Realtek WiFi chipsets). Firmware is tightly connected to the hardware device itself, essentially controlling the small controllers and metal chips within the hardware. The firmware is called upon by the driver when the device is in use.

So, if you have firmware but no driver for the device, you won’t be able to use it. Conversely, if there’s a working driver but it can’t find the correct firmware—or the firmware in use is faulty—the device won’t function properly.

For Secure Boot, you don’t need to sign firmware (.fw files), but kernel modules-drivers (.ko files) installed separately (that are not shipped with your OS from the start) do require signing. This is because firmware is invoked by kernel modules, and if those modules are untrusted or not loaded by UEFI, the firmware becomes irrelevant—no one calls or uses it.

With that said, let me conclude about third-party binaries that build kernel modules: use them only in cases of absolute necessity. Don’t resort to them to make some dubious Aliexpress USB “Super Mega Ultra Cool SSD 1,000,000 TB USB3.0” or a random WiFi/Bluetooth USB device work, do not pull something weird on your OS to play with LED colours of your keyboard following a first found guide. While it’s fine for experimental purposes, such as learning how hardware integrates with an OS, I strongly advise against using these solutions for long-term purposes. If something goes wrong, at best, you risk breaking your OS; at worst, you expose you breach security of your system.

3. Secure Boot Setup

So, now that I’ve hopefully explained the details about Secure Boot, I can proceed with setting it up. In the end, it all comes down to ensuring that your system’s kernel and kernel modules are signed—either with Microsoft signatures or manually using your own signature. A signature, in the context of Secure Boot, essentially is a key.

3.1 Generating your own signatures

If you’re a software developer, you may already be familiar with the concept of keys. For example, authentication on GitHub uses ssh keys, or when you need to secure your app’s traffic over the HTTPS protocol, this involves generation of certificates and keys. Similarly, some applications with high-security access use this method of authentication, where access to certain resources is allowed only from specific devices. In such cases, you, the owner of device (PC), generate a key pair—public and private keys—and share the public key with the provider (app's owner). This key is then enrolled in their authentication system, allowing the server to use your shared public key to authenticate your private key when you connect. Your private key always remains only on your device (PC); it should never be shared and must be stored securely. But if you store it securely, there needs to be a system that manages your keys. Otherwise, when a request is sent, it’s not like the authentication mechanism of an app X will search through all the directories on your PC to find a matching private key. Instead, keys need to be created, registered, and enrolled in a structured way.

On Linux OSs, for Secure Boot all "signatures"-keys are managed by shim.

shim is a simple software package that is designed to work as a first-stage bootloader on UEFI systems.
A key part of the shim design is to allow users to control their own systems. The distro CA key is built in to the shim binary itself, but there is also an extra database of keys that can be managed by the user, the so-called Machine Owner Key (MOK for short).(SecureBoot - Debian Wiki)

Decomposing it: Machine is your PC. Machine Owner is you. If you’re using a Linux distro that managed to obtain a Microsoft signature (like Debian OS), your system’s bootloader is already signed (with distro CA key) and compatible with Secure Boot—but not by you, the machine owner. This means that the kernel and the kernel modules that came as part of your distro by default are already signed and you will be booted by Secure Boot with no problem.

However, any kernel module you install or build manually after the initial installation must be signed by you, using your own key—the Machine Owner Key.

3.2 Enabling Secure Boot

Secure Boot is possible only if it is enabled and only on UEFI BIOS. To check this, go to your BIOS* before booting into any installed OS. Look at the title—if you see "UEFI," you’re good to go. However, if your BIOS* has a terminal-like, scary interface and you only see "BIOS" in the titles—sorry, this is the end of the story for you. No Secure Boot.

Anyway, I’ll assume you have a UEFI BIOS. How to enable Secure Boot depends on the brand of your motherboard, as each manufacturer designs it differently. My motherboard is ASUS, so I simply searched online for "enable Secure Boot ASUS motherboard." For me, the guide is this one.

Remember, if you’re using a custom kernel or a distro that doesn’t support Secure Boot by default, you won’t be able to boot once you enabled Secure Boot. Unfortunately, I can’t provide a guide for that case. I’m a Debian girl and I don’t use any other distro (except for Arch ho-ho). However, if you are in that situation, the idea is to do the steps I will describe below before enabling Secure Boot (not only that, however, you will have to deal also with GRUB). Once everything is signed for UEFI Secure Boot, you can enable it in BIOS*.

Disclaimer: For the next steps, I’m not reinventing the bicycle—I’m simply following the Debian Guide on Secure Boot.

3.3 About MOK and how to manage it

Let’s align our starting point: there are two possible scenarios:

Scenario 1. You have a fresh install and are about to install some drivers—a kernel module, most likely NVIDIA drivers. In this case, you have two options. Remember the DKMS source for your kernel modules? You can configure DKMS to sign all modules it builds a prescindere (by default). Once that’s done, any module built with DKMS—like NVIDIA drivers on Debian—will already be signed.
Scenario 2. You already have NVIDIA drivers installed, but they don’t work because of Secure Boot. Here, you have two ways to fix this:

Configure DKMS to sign your modules and then reinstall the drivers.
Manually sign the kernel modules that are already installed.

I will cover both scenarios.

After enabling Secure Boot in UEFI BIOS, boot into your Debian and check if Secure Boot enabled:

$ sudo mokutil --sb-state
SecureBoot enabled

You can check the list of already enrolled keys with command sudo mokutil --list-enrolled

If you haven't try to do something with key enrollment before, you will most probably see two enrolled keys: one is the Debian key, and the other is the Debian DKMS module signing key. However, even if this key appears in the list, it does not mean that it will automatically be used to sign any additional kernel modules you are about to build (or install). While there is functionality to export the enrolled MOKs for manipulation, using mokutil --export will export all enrolled keys in .der format, and a .der key is similar to .pub keys - they are "public" by nature. mokutil --export will never be able to extract for you private keys that were paired with .der keys. And without a private key, matching to any extracted .der key you will not be able to sign anything.

Therefore, you will need to generate a new MOK for manually signing additional kernel modules you are about to install. By the way, the keys currently enrolled in UEFI, if you did not enroll anything before, are not MOK keys—they are owned by Debian, not you.

In this article, I will be installing the nvidia-driver package from the Debian bookworm repository (this package installs "proprietary" flavour of drivers). NVIDIA drivers are kernel modules, and they require a signature to be loaded under Secure Boot. If you install NVIDIA drivers in an adequate way NVIDIA kernel modules will be built using DKMS (for example if you install them with nvidia-driver Debian package).

4. Secure Boot Scenario 1 - Fresh install of NVIDIA drivers; configuring DKMS to automatically sign kernel modules.

If you have been following the official Debian guide on NVIDIA drivers installation, you have probably come across the note about Secure Boot. If you clicked on link with "detailed instaructions", you were redirected to the page about Secure Boot and DKMS. However, if you are here reading this article, it probably didn’t work for you.

I installed my Debian system using the netinstall .iso with minimal installation, including only the standard system utilities. I had the kernel and the default kernel modules installed. However, I did not have the kernel headers installed, nor did I have DKMS installed.

To check if your system has the dkms package installed, use the following command:

$ sudo dpkg -l | grep dkms

Even if the output of this command is empty, when you will run 'sudo apt install nvidia-driver', this package as a dependency shall bring for you also dkms package.

4.1 "Outsource" signing keys generation to DKMS

So, dkms package will be installed as a dependency AS ONE OF THE STEPS of nvidia-driver package installation. Right after its installation, dkms will be building NVIDIA kernel modules for you (because it knows how to do it and it will be instructed to do so during installation process). And then, when kernel modules are built...DKMS will sign them for you using its own signing keys...which are not enrolled into your MOK manager and WILL not be acknowledged by UEFI at the next boot.

NB! DKMS signs the kernel modules it builds automatically upon the installation/building completion. If you don’t explicitly tell DKMS (via its config file) to use a specific keypair, DKMS will generate for itself a keypair automatically after building THE FIRST kernel module on your system. HOWEVER, of course, DKMS won’t be able to enroll these generated keys into your PC’s list of MOK-registered keys!

Where does DKMS place his automatically generated keys? It places them in the /var/lib/dkms/ directory. This is not some sacred info, you can see were DKMS searches for keys in its config file by cat /etc/dkms/framework.conf (mok_signing_key and mok_certificate fields). You can verify that right after installing the dkms package, this directory will be empty. However, if you install a driver like the nvidia-driver package - which pulls in DKMS as a dependency - and then, after installation, if you list the contents of /var/lib/dkms using ls, you’ll see the keys DKMS used to sign the new modules. In general, if you’re attentive to the installation logs, you’ll also spot an entry indicating that DKMS is signing the built (installed) kernel modules.

And here is the cornerstone. When DKMS generates a key pair automatically, it’s not exactly creative with naming—it defaults to something like DKMS Signing Key. The issue is, if you run multiple OSes (like different Debians) across different disks, or you frequently install/reinstall your OS for a fresh start, the list of enrolled MOKs can get very cluttered. And cleaning that up later is a pain—especially if the names of different keys are all the same and don’t explain anything. NB! 1. If you purge your OS from disk, destroy all partitions, install on top of it new OS, none of these will remove already enrolled keys related to purged OS from MOK list. 2. You cannot attach reinstalled system to the enrolled keys of previous OS. Private key gone (which is very probable if you wipe out entire OS from disk to reinstall it) = no signing/validating is possible).

So, my personal choice is to keep control over key generation and use keypair that I generate and label manually, rather than letting DKMS keep creating new ones and enrolling them each time.

However, if you prefer minimal effort and OK with using keys generated by DKMS, you just have to enroll public key of generated key pair:

$ sudo mokutil --import /var/lib/dkms/mok.pub # prompts for one-time password
$ sudo mokutil --list-new # recheck your key will be prompted on next boot

<rebooting machine then enters MOK manager EFI utility: enroll MOK, continue, confirm, enter password, reboot>

$ sudo dmesg | grep cert # verify your key is loaded

4.2 Creating and Enrolling your own Machine Owner Key; pointing DKMS to use it

In order to prevent DKMS to generate its own signing keys and keep everything tidy on my system with comprehensive labels, before installing any kernel module for first time on my system (usually it is NVIDIA), I first install dkms and point it to keys it should use for signing.

First, I create my MOK - Machine Owner Key:

$ su - #if you are logged
# mkdir -p /var/lib/shim-signed/mok/
# cd /var/lib/shim-signed/mok/
# openssl req -nodes -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER -out MOK.der -days 36500 -subj "/CN=My Name/"
# openssl x509 -inform der -in MOK.der -out MOK.pem

Then, I enroll this newly generated key, so it becomes acknowledged by shim and then by UEFI after reboot:
NB! store securely input password, you will need it later!

# exit 
$ sudo mokutil --import /var/lib/shim-signed/mok/MOK.der # prompts for one-time password
$ sudo reboot

At the reboot, the device firmware should launch it's MOK manager and prompt you to review the new key and confirm it's enrollment (Press any key and then select "Enroll MOK" option, then "continue", then "yes", insert the password you have chosen when you enrolled the key with mokutil --import, then "Reboot"). After this, when you boot again, you can verify:

$ sudo mokutil --list-enrolled
$ sudo mokutil --test-key /var/lib/shim-signed/mok/MOK.der
/var/lib/shim-signed/mok/MOK.der is already enrolled

Now, I have key that I want is getting used by DKMS. Instead of waiting for when dkms will be installed as a dependency during ongoing installation of NVIDIA drivers, I anticipate it to configure DKMS so it uses for signing the keys I want.

NB! If you tweaked somehow version of kernel on Debian - most common scenario, when you upgraded the version of kernel on Debian Stable from backports repo, you HAVE TO ensure that the version of kernel and kernel headers are aligned! DKMS relies heavily on your system's kernel headers to build modules for your kernel. If version of headers is misaligned with version of kernel, modules will be built in wrong way and most probably will result to be broken and not working.

# your system's kernel version
$ uname -r 
# version of present kernel headers
$ dpkg -l | grep linux-headers-*
# if in the output you do not see linux-headers of the precisely matching version to the output of uname -r, you have to install them!
# you may also have more than one version of linux-headers,
# that is dure to the fact that anytime your OS kernel version is updated, 
# the old version of kernel is removed with sudo apt auptoremove,
# but headers may remain
# to install linux-headers that match precisely your system's current kernel version:
$ sudo apt install linux-headers-$(uname -r)

Installing dkms:

$ sudo apt install dkms
#if you installed linux-image-* from bookworm-backports(kernel version >6.1.x)
#sudo apt install -t bookworm-backports dkms

Configuring dkms to use a specific keypair for signing:

Go to /etc/dkms and run ls—you should see the file framework.conf. Use cat to view its contents, and you will find the mok_signed_key and mok_certificate fields pointing to /var/lib/dkms/... This directory is the default directory where DKMS searches for keys to sign modules and it is the directory where it will place automatically generated own keys if it finds it empty and if it is configured to look there.

So, the point is to modify DKMS config file, so it looks for keys in directory where I places my MOK keypair. You simply need to modify these lines, and point dkms to existing and enrolled MOK key pair:

ls /etc/dkms
cat /etc/dkms/framework.conf
sudo vim /etc/dkms/framework.conf
#delete '#' in the beginning of the lines!
mok_signing_key="/var/lib/shim-signed/mok/MOK.priv" 
mok_certificate="/var/lib/shim-signed/mok/MOK.der"

Now, you can proceed with NVIDIA drivers installation:

$ sudo apt install nvidia-driver
$ sudo reboot

After reboot:

$ nvidia-smi
$ modinfo nvidia-current

VOILA!

5 Secure Boot Scenario 2 - NVIDIA drivers are already installed; using your MOK to sign installed kernel modules

If you have already installed NVIDIA drivers and they worked perfectly before enabling Secure Boot but stopped working afterward, you have to repeat the steps mentioned above - generate MOK key-pair and enroll this MOK. Then, if you want that in future all kernel modules built with DKMS are automatically signed, you have to add path to your key pair into the DKMS configuration file. However, if you have installed NVIDIA drivers before, this DKMS config documentation will not have a reverse effect—kernel modules that have already been built and installed will not be signed. You have to sign those kernel modules manually, or reinstalled (rebuild) them. Additionally, if for some reason you plan to install kernel modules that will not be built with DKMS, they will also remain unsigned, and you will have to sign them manually.

For this example, I have installed NVIDIA drivers, but I did not sign anything. Therefore, after I boot, NVIDIA drivers are not available.
I see errors in the dmesg logs from the boot process: in lsmod, I do not see any loaded NVIDIA kernel module, and nvidia-smi shows an error:

However, the drivers are definitely installed! All kernel modules are located in /lib/modules. If you periodically update your kernel version and install a newer version, you will see more than one subdirectory there. I am using the latest kernel, and what I need to check for are updates to this kernel—because when I installed the NVIDIA kernel modules, and since I built the NVIDIA kernel with DKMS, I go to the /dkms subdirectory. Here they are! My NVIDIA driver kernel modules:

So, these kernel modules I have to sign, to make them work under Secure Boot. This is the method indicated by the Debian developers:

#first, you need to create these kernel package-related variables:
$ VERSION="$(uname -r)"
$ SHORT_VERSION="$(uname -r | cut -d . -f 1-2)"
$ MODULES_DIR=/lib/modules/$VERSION
$ KBUILD_DIR=/usr/lib/linux-kbuild-$SHORT_VERSION
$ cd "$MODULES_DIR/updates/dkms" # For dkms packages

#you have to use passphrase you used for enrollment of MOK
$ echo -n "Passphrase for the private key: "
$ read -s KBUILD_SIGN_PIN
$ export KBUILD_SIGN_PIN
#simple bash loop to sign all kernel modules found in the current directory
$ find -name \*.ko | while read i; do sudo --preserve-env=KBUILD_SIGN_PIN "$KBUILD_DIR"/scripts/sign-file sha256 /var/lib/shim-signed/mok/MOK.priv /var/lib/shim-signed/mok/MOK.der "$i" || break; done
#or you can sign modules below the current directory one by one:
$ sudo --preserve-env=KBUILD_SIGN_PIN "$KBUILD_DIR"/scripts/sign-file sha256 /var/lib/shim-signed/mok/MOK.priv /var/lib/shim-signed/mok/MOK.der nvidia-curernt.ko
...
$ sudo update-initramfs -k all -u
$ sudo reboot

Here is my result:

VOILA!

Summarizing (my personal advices):

🦄 Do not neglect this security mechanism; it is neither as scary nor as hard to configure as it may seem.
🦄 Do not disable Secure Boot carelessly.
🦄 Secure Boot protects you not only during the boot process but beyond.
🦄 Adapting your system to be compatible with Secure Boot is simply a matter of OR enrolling (making it visible to UEFI) a signing key automatically generated by DKMS OR generating your Machine Owner Key (MOK), enrolling it, and then using it to sign your kernel modules or show the software that builds them for you (DKMS) where your enrolled keys are.
🦄 If you are a software developer and still find Secure Boot challenging or restrictive, I advise to reconsider it. Adapting your system for UEFI Secure Boot is an excellent exercise and an opportunity to deepen your understanding of keys, certificates, and key-based signatures—concepts you will undoubtedly encounter frequently in your developer career.

Using Timeshift for System's Snapshots and Recovery on Debian 12 via Command Line

Anna — Wed, 13 Nov 2024 18:39:51 +0000

Timeshift for Linux is an application that provides functionality similar to the System Restore feature in Windows and the Time Machine tool in Mac OS. Timeshift protects your system by taking incremental snapshots of the file system at regular intervals. These snapshots can be restored at a later date to undo all changes to the system.(Timeshift GitHub)

When exploring a new distro, it’s always helpful to have the option to undo changes, especially after installing or configuring something substantial that significantly alters your system. For example, installing NVIDIA drivers modifies multiple components of your system, and simply uninstalling the drivers doesn’t always revert everything to the exact state it was in before driver's installation.

Snapshots ≠ Backups

It’s important to note that, although snapshots are sometimes referred to as backups—even Debian documentation lists Timeshift as a backup tool - BackupAndRecovery-Debian Wiki—I somewhat disagree with this classification.

The maintainers of Timeshift also make this distinction clear:

Timeshift is similar to applications like rsnapshot, BackInTime and TimeVault but with different goals. It is designed to protect only system files and settings. User files such as documents, pictures and music are excluded. This ensures that your files remain unchanged when you restore your system to an earlier date. (Source)

If you need a true backup tool, you should look elsewhere or manually back up periodically just some important data to cloud storage or physical media/storage device.

While many people use snapshots as backup tools—and they work 90% of the time—in the other 10%, they can fail. Holding onto snapshots for too long, especially as the system changes significantly, can lead to problems. This isn't an issue specific to Timeshift, but it can happen if, for example, you take a snapshot, then make major modifications to your storage setup (such as reconfiguring LVM, partitions, changing filesystems). In such cases, it’s best to delete old snapshots and create a fresh one after successful modifications.

I think that probability to encounter such issues with Timeshift is quite low, but in my experience with VMware snapshots, I ran into trouble for precisely this reason.

After taking a snapshot of a Virtual Machine (VM), a logical volume has been extended and additional RAID storage has been added to this VM.The snapshot afterward has not been deleted and then a write-intensive task has been launched, which eventually has triggered an I/O error on the added RAID storage. It was a bit of a "situationship" since the RAID was specifically set up to ensure data integrity. Fortunately, no data was lost, but the Virtual Machine got crazy, caught between the restrictive snapshot (designed to allow a full rollback in VMware) and the newly evolved storage configuration—especially with the repetitive write process on the RAID system.

For more details about Snaphshots vs Backups, you can read this thread on Reddit. And I proceed with timeshift installation, because I need exactly a snapshot tool, not a backup tool.

Timeshift GUI - a little drawback

Timeshift has a graphical user interface, which, for some, may be a strong advantage, while for others, it might be less appealing (I, for example, prefer terminal or command line interfaces).

The timeshift package depends on lib-gtk-3.0 (Debian -- details of package timeshift in Bookworm). lib-gtk is the GTK graphical user interface library, and this dependency makes timeshift a suboptimal solution for machines primarily used as servers. However, I use it on my personal PC, and although I can’t recall any other package in my setup that uses GTK, this tool’s reliability outweighs the drawback of having GTK installed. But if this is a concern for you, you may want to consider a different tool for snaphots. Here’s the original response (a bit dated) from the creator of Timeshift:

Timeshift has dependencies on GTK3 libraries so you must have that installed even if you have not installed a desktop. During installation it will install all dependencies it requires.
Separating the codebase into 2 separate packages (with and without Gtk dependencies) involves a lot of work.

I’m not sure if the current maintainers of Timeshift have plans to separate Timeshift into GUI-only and CLI/TUI-only versions, or if this might have already been done. I haven’t found any information on it.

Preparing "storage device" for Timeshift

My system is partitioned with LVM, and I have a volume group that holds all my system’s logical volumes (called wonderland-vg). First, I want to check if there’s any free space in this volume group. If not, I’ll need to expand it.

Most tutorials and guides focus on how to configure Timeshift where to store snapshots via the GUI. However, my system currently doesn’t have a DE nor even a display server (technically, I am on headless server, that means that I cannot use mouse). Timeshift’s approach to snapshot destinations is based on storage devices (timeshift’s config file expecting UUID) rather than directories, which makes sense—storing snapshots on the external storage device is useful in case the system breaks. So first, I’ll prepare a separate logical partition—a logical volume—for my future snapshots.

#these commands give me stats about my volume group
$ sudo vgdisplay
$ sudo vgs

I have around 160 GB free in wonderland-vg volume group, so I do not need to expand it:

I proceed with creation of new logical volume and mounting it to /timeshift directory:

# create logical volume with name timeshift in volume group wonderland-vg
$ sudo lvcreate -L 20G -n timeshift wonderland-vg
#verify creation
$ sudo lvdisplay
# create a filesystem (all my system is ext4, choose any you use
$ sudo mkfs.ext4 /dev/wonderland-vg/timeshift

Installing Timeshift

Installation of timeshift in Debian is quite straightforward: just run sudo apt install timeshift.

Oh my, that’s quite a lot of dependencies!
My current Debian setup is very close to a server setup, so I have very few things installed. Surprisingly, it doesn’t seem to pull in the display server Xorg, which is actually pretty good, especially for people using Wayland. However, there are still quite of some X11 packages installed.

Configuring Timeshift

The configuration of timeshift with command line is done mostly via configuration file. In /etc/timeshift/, you can find the default config - default.json. If you had already tried to create a snapshot with sudo timeshift --create, you will find in that directory also timeshift.json file. If you want to have control over the choice of storage device where your snapshots will be stored before launching timeshift --create for the first time, you have to modify /etc/timeshift/default.json. If you want to change the storage device where timeshift already have placed some snapshot(s), you have to modify /etc/timeshift/timeshift.json.

In order to point timeshift to use a specific storage device, you have to provide its UUID. If you run sudo blkid, you’ll see the UUIDs of all your storage devices and can pick one where you are willing to place your snapshots.

If you are with the setup that you have control over your mouse, the task becomes quite simple - run sudo blkid, identify storage device (it can be a logical volume, there is no requirement for storage to be physical partition, it can be logical partition as well!). Then you copy corresponding UUID and you will have to paste it into timeshift configuration file (/etc/timeshift/default.json or /etc/timeshift/timeshift.json) in the field "backup_device_uuid" (between backquotes).

NB! If you are on headless server and you cannot just easily select a text from terminal, here is the workaround:

# /dev/mapper/wonderland--vg-timeshift is a logical volume where I want to store my snapshots
# with this command I extract UUID and store it into a temporary file
$ sudo blkid -s UUID -o value /dev/mapper/wonderland--vg-timeshift > /tmp/uuid-snapshots.txt
# Then, I open the `timeshift` configuration file to modify.
# I do it before running timeshift for the first time, so I have only `default.json` as a config file
$ sudo vim.tiny /etc/timeshift/default.json
{
  "backup_device_uuid" : "|",
  ...
}
# I place a cursor between backquotes (where | is)
# in normal vim mode, I type:
:r /tmp/uuid-snapshots.txt
# this command will place the content of file where your coursor is, if needed, modify it a bit, so it is placed correctly
# Save the file!

NB! IF you run sudo timeshift --create for the first time after installation **it will not prompt you to select the storage, but instead it will pick one on its own. In my experience, it selects the first physical partition of a disk where your OS resides (for example, /dev/sdb1); the problem that often it's a /boot partition, so it's definitely not the best place to store snapshots. In this case, interrupt the execution (Ctrl +C) and go to modify/etc/timeshift/timeshift.json (it should be created after the first launch) - either manually provide the correct UUID of the desirable storage or remove automatically set UUIDs (backup_device_uuid and parent_device_uuid fields). Once you’ve done that, restart the command, and it should prompt you to select the storage. However, timeshift has already placed its directory to the storage (which he picked by itself) after the first launch. It may have sense to go and wipe /timeshift directory from there after configuring timeshift to store snapshots on desirable storage device**.

I think the GUI would give more comprehensive control over device selection.

If using Timeshift's GUI is not an option, you can schedule backups using cron jobs or changing false to true in configuration file for schedule_monthly, schedule_weekly, schedule_daily, schedule_hourly (NB! I did not test it!). I don't enable automatic snapshots because, as I mentioned, they’re not true backups, and I don’t need constant snapshotting. I’ll take snapshots manually when necessary (e.g., before major system updates or modifications).

If you're planning to set up automatic snapshots, you should also check the configuration file for the snapshot retention policy.

For instance, if you schedule snapshots to run daily, even if they’re incremental, there’s no point in keeping all of them indefinitely (because they occupy storage's space). You need to pay particular attention to the values of these fields in the configuration file: count_monthly, count_weekly, count_daily, count_hourly.

Keep in mind that manually deleting snapshots via command line is a bad idea. Snapshots are incremental: only the first snapshot contains a "full" snapshot, while subsequent snapshots are just deltas (differences) from that first snapshot. If you manually delete the first (or "parent") snapshot, the others become unusable. However, if you use the retention policies and Timeshift commands, like sudo timeshift --delete --snapshot '<name/timestamp from sudo timeshift --list>', your snapshots will be managed correctly.

Another configuration field to check is exclude. By default all user's home directory is excluded from snapshotting.

NB! If you’re experimenting with different Desktop Environments, browsers, terminal emulators, or shells, remember that anything related to the customization, display stuff typically has configuration files in your $HOME (/home/your_username) directory—often in $HOME/.config. If your /home directory is excluded from snapshotting (field exclude in /etc/timeshift/timeshift.json), when you install for instance KDE -> sudo timeshift --restore -> install GNOME, all of KDE’s config files will remain in $HOME and $HOME/.config. If you do many experiments withs such things, it will inevitably lead to clutter of $HOME directory over time (remember, that ls is not sufficient to see all content, try at least ls -a).

Testing Timeshift

I’ve already created a couple of backups, and since they’re incremental, only the first one was large and took more time to create. Let’s see if this tool can handle reverting a kernel version update and restore the system from "incremental" snapshot (not the first, complete one).

To see how timeshift organizes its stuff/files/snapshots, you can mount the storage device which is used by timeshift with:

# create a directory - mounting point
$ sudo mkdir /timeshift
# mount there newly created logical partition
$ sudo mount /dev/wonderland-vg/timeshift /timeshift
# to see what is inside
$ ls -a /timeshift

This is directory structure that was created automatically by timeshift for itself:

My system's kernel version before kernel upgrade from bookworm-backports repo:

My system's kernel version after kernel upgrade from bookworm-backports repo:

I have 2 snapshots and I will be using the latest one.

$ sudo timeshift --list
$ sudo timeshift --restore --snapshot <timestamp/name form the list>

Timeshift will ask me whether I want to reinstall GRUB. Since the change to my system was a kernel upgrade, I confirm. Then, I’m asked where GRUB should be installed.

After the system state and settings are retrieved from the snapshot and ready for rollback, the system reboots.

NB: If you use BTRFS, don't miss the opportunity to take advantage of its excellent restore capabilities—they are fully supported in Timeshift:

It is strongly recommended to use BTRFS snapshots on systems that are installed on BTRFS partition. BTRFS snapshots are perfect byte-for-byte copies of the system. Nothing is excluded. BTRFS snapshots can be created and restored in seconds, and have very low overhead in terms of disk space.(Timeshift GitHub)

My system's kernel version after restore:

UPD on testing:

I’ve noticed that sometimes when I do a rollback using timeshift restore and then reboot, the restored system’s wireless network interface (Wi-Fi) ends up down. NetworkManager can’t resolve it—it’s not just the Wi-Fi gets disconnected, but the entire wireless interface disappearing (ip a does not list it at all). However, doing a clean reboot after the restore fixes this issue without any problem.

Summarizing:

Timeshift's Pros:
🦄 Continuously maintained and widely-used snapshotting tool
🦄 Easy management via command line
🦄 Incremental snapshots
🦄 Clearly separates snapshots from backups
🦄 Has an active community
Timeshift's Cons:
🫏 No-GUI (CLI-only) installation is not possible
🫏 Brings many dependencies related to the Xorg display server, which may not be the best for server setups or systems with Wayland
🫏 Limited guides on CLI-only usage, even though it's possible
🫏 Storage destination is bound by UUID, which may be confusing
🫏 Limited identifiers in storage selection when creating the first snapshot, making it harder to choose the correct storage device

DEV Community: Anna

Building Debian packages from source in bootstrapped Debian

Where to Start in Web Development: Ignoring Learning HTTP(S), URLs, DNS, IP, SSL Will Have Consequences...

1. Browser's Address Bar

2. URIs and URLs

3. HTTP Protocol

3.1 Internet Protocol and IP addresses

3.2 Transmission Control Protocol (TCP)

3.3 TCP/HTTP Network traffic

3.4 Understanding Ports

3.5 DNS

3.6 TCP in action: 3-way handshake

4. HTTPS

4.1 Little extra info: multiple IP addresses horizontal scaling of web apps

4.2 About encryption

4.3 TLS/SSL certificates

4.4 Secure communication channel

Where to Start in Web Development: Your Browser as Your First IDE

1. Building confidence with a browser

1.1 Open random files in your browser to see what happens

1.2 Browser's Developer Tools as an IDE (Integrated Development Environment)

2. How does a browser work?

2.1 What is the document?

2.2 Browser's rendering engine

2.3 Document Object Model - DOM

2.4 Browser's JavaScript Interpreter

Virtualization on Debian with libvirt&QEMU&KVM — Networking beyond "default": Forwarding Incoming Connections to NAT'ed network

➀ DEFAULT libvirt network

➀.➀ virbrN and vnetN - what do they do?

➀.➁ About DHCP

➁ Configuring static IPv4 address on a VM

➁.➀ Shrinking DHCP range

➁.➁ Modifying /etc/network/interfaces

➂ Connecting to a VM from Another Device on the LAN using the Port Forwarding Method:

➂.➀ What is the Port Forwarding?

➂.➁ Firewalld and closed ports

➂.➂ Forwarding ports on the host with firewalld

➂.➃ Adding a rule to nftables ruleset to accept inbound network traffic to the VM

➂.➄ Considerable drawbacks of this method

Virtualization on Debian with libvirt&QEMU&KVM — Networking beyond "default": Must Have Concepts to Start

➀ About Public IP address and LAN

➀.➀ Inbound vs Outbound traffic

➁ Understanding your LAN and private IP address(s) of your host machine

➁.➀ Network Interfaces: general information

➁.➁ Network Interfaces: MAC addresses (link/ether)

➁.➂ IPv4 vs IPv6 addresses and NAT

➁.➃ IPv4 address ranges reserved for private networks

➁.➄ IPv4 addresses structure and CIDR

➂ Libvirt's DEFAULT Network: about NAT mode

➂.➀ About virtual network switches

➂.➁ Libvirt wants iptables, Debian has nftables: what to do?

➂.➂ DEFAULT Libvirt's network: what's under the hood?

➂.➃ Nftables rulesets: Libvirt Network's table and chains explained

➂.➄ About how NAT works

Virtualization on Debian with virsh&QEMU&KVM — Installation of virtualization tools and first VM creation

➀ Virtualization and self-hosting

➁ Virtualization as a tool for resource-Constrained application development

Note for the Dockerists/Dockerphiles/Containerphiles

➂ Virtualization on Debian: how it works

➂.➀ CPU virtualization support

➂.➁ Hypervisor

➃ KVM & QEMU

➄ Libvirt

➅ Validation of installed virtualization tools

➅.➀ Important! qemu:///system vs qemu:///session

➆ Creation of first VM

➆.➀ OS image file

➆.➁ Preparing storage

➆.➂ VM creation with virt-install

➇ Let the Networking begin!

➇.➀ Userspace (SLIRP or passt) connection

➇.➁ NAT forwarding (aka "virtual networks")

Debian 12 … is amazing! How to: Create your custom codehouse #6 [Giving Voice to Debian: Wireless Audio Devices configuration]

➀ Sound on your PC: where it starts?

➁ Sound on your PC: where it starts (software-side)?

➂ ALSA, PipeWire or Pulseaudio: what is better*? (trick question)

ALSA (Advanced Linux Sound Architecture)

➃ Bluetooth devices management tool

➄ PipeWire vs Pulseaudio

➅ PipeWire: Installation

2.1 What is the `document`?

➀.➀ `virbrN` and `vnetN` - what do they do?

➁.➁ Modifying `/etc/network/interfaces`

➂.➂ Forwarding ports on the host with `firewalld`

➂.➃ Adding a rule to `nftables` ruleset to accept inbound network traffic to the VM

➁.➁ Network Interfaces: MAC addresses (`link/ether`)

➂.➁ Libvirt wants `iptables`, Debian has `nftables`: what to do?

➆.➂ VM creation with `virt-install`

Debian Way Installation Step 1: Add the `contrib` and `non-free` repository components to the list of sources for `apt`:

Debian Way Installation Step 3: Check if your system uses `dracut`.

`nvidia-kernel-dkms` OR `nvidia-open-kernel-dkms` <-- install kernel space components of NVIDIA drivers; `libcuda1` and other libraries (`lib*`) <-- install user-space components of NVIDIA drivers, a.k.a CUDA drivers

5.1 Installing driver in NVIDIA suggested way (with `.run` file)