DEV Community: David Foley

libvirt network forwarding inconsistencies

David Foley — Tue, 17 Feb 2026 22:44:00 +0000

In an INCUS container on a KVM guest, ping commands are successful when executed on the container, but not when the are executed from the host on the other end.

TL;DR

Change the libvirt virtual bridge forwarding mode to "open"

The Problem

Consider a scenario where there is a group of physical servers connected to a core router. On one of the servers a KVM guest is running, which in turn is a host for some INCUS containers. Various virtual bridges, firewalls and subnets are implicated in providing connectivity to the INCUS containers. Everything is verified to be configured correctly from the virtualisation to the routing.

A given INCUS guest can initiaite successful ICMP pings to:

KVM guests on the same physical host
Physical hosts external to it's own.
The core router

However, ICMP pings intiated on any device external to the container's own physical server do not succeed. In other words: ping command are successful when executed on the container, but not when the are executed from the host on the other end.

This is an inconsistent result and completely unintuitive.

The root cause can be attributed to the libvirt virtual network forwarding mode and a misunderstanding about what mode was appropriate for my environment. I was using libvirt's forward mode "route", where "guest network traffic [is] forwarded to the physical network via the host's IP routing stack, but without having NAT applied.". The documentation does not mention that libivrt firewall rules are added which interferes with the traffic described above.

The Solution

The solution is to use libvirt's "open" forwarding mode: "As with mode='route', guest network traffic will be forwarded to the physical network via the host's IP routing stack, but there will be no firewall rules added to either enable or prevent any of this traffic."

Step 1: Open the editor the XML definition for your virtual network bridge

On your KVM host, run the following command to begin editing the XML definition for your virtual network bridge.

sudo virsh net-edit kvm_virbr0

Explanation:

virsh is the command to manipulate libvirt
net-edit is the sub-command to edit networks managed libvirt
kvm_virbr0 is the name of the virtual network bridge to which my KVM guests connect to. Adjust where appropriate and your network name differs.

Step 2: Change the forwarding mode

Edit the XML definition so that the forward mode = "open" as in the this example:

<network> <name>kvm_virbr0</name> <uuid>345a0dcc-4f89-4f7f-xcv3-160d1145837c</uuid> <forward mode='open'/>

Save the file and close the editor.

Step 3: Restart the virtual network bridge

There is much advice on how to do this across the internet, the only reliable method I have found is to restart the computer.

Credits

Photo by Jaime Street on Unsplash

This article was first posted at https://www.dfoley.ie/blog/libvirt-network-forwarding-inconsistencies #indieweb #posse

Raspberry Pi Pico NOT flashing with arduino-cli

David Foley — Tue, 12 Sep 2023 14:28:47 +0000

While flashing a Raspberry Pi Pico (RP2040) from a Raspberry Pi 3 I got the error "No drive to deploy", there's a relatively simple fix if hard to find on the internet.

Error Flashing the Device

While using Earle Philhower's RP2040 board package in the arduino-cli to flash a Raspberry Pi Pico (RP2040) from a Raspberry Pi 3 (Raspberry Pi OS Lite, buster) I got the following error:

Converting to uf2, output size: 516096, start address: 0x2000
No drive to deploy.
An error occurred while uploading the sketch

The board was accessible via serial, the arduino-cli was correctly rebooting the RP2040 to where an external drive was appearing at /dev/sda.

I could find nothing on the internet except this bug report, however there was no effective solution proposed there. Wondering whether it was the board package or Raspberry Pi OS Lite, I decided that trying the official Arduino Core for mbed enabled devices might help.

I encountered the same difficulty. Looking around on the web, I find a Arduino forum thread which had the solution.

The solution: post_install.sh

After installing official Arduino Core for mbed enabled devices before you attempt to use it the first time you must run the shell script sudo ~/.arduino15/packages/arduino/hardware/mbed_rp2040/2.5.2/post_install.sh

Credits

Photo by dhehaivan on Unsplash

This article was first posted at https://www.dfoley.ie/blog/raspberry-pi-pico-not-flashing-with-arduino-cli #indieweb #posse

dnf packaging errors for MariaDB and borgbackup

David Foley — Mon, 11 Sep 2023 11:50:47 +0000

I recently encountered an error on a database server when using dnf (yum) on CentOS 8 Stream. Dependencies for MariaDB and borgbackup created a problem and there was an unusual fix which I hadn't previously been aware of.

TL;DR

Set module_hotfixes=true in MariaDB repo definition
Enabled powertools repository as recommended by Fedora EPEL team

The Problem

When I went to use the dnf package manager, I get the following error:

[david@mariadb0 ~]$ sudo dnf update
Last metadata expiration check: 12:00:00 ago on Mon 01 Jan 1970 00:00:00 UTC.
Error: 
 Problem 1: cannot install the best update candidate for package borgbackup-1.1.16-1.el8.x86_64
  - nothing provides python3.6dist(packaging) needed by borgbackup-1.1.17-1.el8.x86_64
 Problem 2: package MariaDB-shared-10.5.9-1.el8.x86_64 requires MariaDB-common, but none of the providers can be installed
  - cannot install the best update candidate for package mariadb-connector-c-3.1.11-2.el8_3.x86_64
  - package MariaDB-common-10.5.7-1.el8.x86_64 is filtered out by modular filtering
  - package MariaDB-common-10.5.8-1.el8.x86_64 is filtered out by modular filtering
  - package MariaDB-common-10.5.9-1.el8.x86_64 is filtered out by modular filtering
 Problem 3: package MariaDB-shared-10.5.9-1.el8.x86_64 requires MariaDB-common, but none of the providers can be installed
  - cannot install the best update candidate for package mariadb-connector-c-config-3.1.11-2.el8_3.noarch
  - package MariaDB-common-10.5.7-1.el8.x86_64 is filtered out by modular filtering
  - package MariaDB-common-10.5.8-1.el8.x86_64 is filtered out by modular filtering
  - package MariaDB-common-10.5.9-1.el8.x86_64 is filtered out by modular filtering
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

The host hasn't been touched recently, this error arose spontaneously.

It turns out there are two issues here: a) Default package filtering is leading to MariaDB's dependencies not being resolved; b) The Fedora EPEL repository was not setup correctly, as a result borgbackup's dependencies cannot be resolved. Fortunately there are some easy fixes, however it was hard to track this down on the web, hence this blog post.

Fixing MariaDB

This bug was previously identified by the MariaDB community and I found a suggested fix in the bug report. The steps are as follows:

Edit the MariaDB repository definition: sudo vi /etc/yum.repos.d/MariaDB.repo. Add module_hotfixes=true to the end of the file
Run sudo dnf update --best --allowerasing

I have updated a stack exchange thread related to this with the advice.

Fixing borgbackup

This error arose because I did not pay attention to the advice for setting up the Fedora EPEL repository. Execute the following and then try dnf again:

dnf install 'dnf-command(config-manager)'
dnf config-manager --set-enabled powertools

This article was first posted at https://www.dfoley.ie/blog/dnf-packaging-errors-for-mariadb-and-borgbackup #indieweb #posse

Activate partial LV after drive failure

David Foley — Sun, 10 Sep 2023 14:11:29 +0000

I recently noticed a drive in my home NAS was pre-fail according to S.M.A.R.T. diagnostics, when I replaced the drive I found it difficult to mount the remaining drives in the LVM VG to recover data.

TL;DR

Use command lvchange -ay --partial example_logical_volume to bring a LVM LV with failed disks

The Problem

The LVM volume that the drive was a member of had three drives. The majority of the data in the group was backed up redundantly to a local and cloud drive using borg backup. There was about 900GB of which was not backed up because it was deemed to be unimportant, certainly not important enough to warrant the cost of backing it up to a cloud server, as well as locally, like the other data. However since the the failing drive was still online I tried to save as much of the 900GB as possible.

I first copied off as much as I could to a spare drive, which was quite successful. During this process, the faulty drive failed completely. This wasn't a problem as I had saved the 900GB and had the backups. The backups had never actually been used, so I did a borg check to ensure they were consistent. Despite that I was still a little nervous and I wanted to access the two good drives in the LVM volume to pull off the remaining data. I knew this was possible but had never done it before and found it very difficult to find information on the web. Which is why I am writing this small blog post.

In the end it turned out to be quite simple, all that was required was the use of the --partial flag when using lvchange. So the command I executed was lvchange -ay --partial example_logical_volume. This brought the LV back up and I was able to access the data on the two remaining disks. See the lvchange manpage) for more information

This article was first posted at https://www.dfoley.ie/blog/activate-partial-lv #indieweb #posse

Website not working in any browser on iOS

David Foley — Sun, 26 Jul 2020 23:23:03 +0000

This week I encountered the most bizzare support request: "Is it possible that our website would not load on iPhones?". Don't be silly, "that's not a thing" I told the customer ... It appears I was wrong.

TL;DR

Every browser on iOS behaved the same and gave vague feedback
Curl gave the same error, but provided far better feedback
The error was caused by a web server behind a reverse proxy sending out an Upgrade: h2 header which is not permitted in the HTTP/2 protocol (RFC7540)
To fix the problem, remove Protocols h2 from the Apache config

The Error

Every browser (Safari, Chrome & Firefox) on iOS (iPhone and iPad) completely failed when loading the website. At the same time the site was accessible and functionting normaly on all browsers on Mac, Linux, Windows and Android. The error on iOS presented slightly differently depending on which browser was being used, some gave vague feedback, others gave none.

Safari and Firefox (iOS)

When the URL was input into the address bar the browser indicated it was loading the site, then quickly failed and closed the tab. There was no error feedback whatsoever.

Chrome (iOS)

When the URL was input into the address bar the browser indicates it is loading the site, then quickly fails and shows the following error page with the vague error description "ERR_FAILED":

Investigation

The website in question was a WordPress site running on a LAMP stack in an LXC container, all requests to the site were being routed through an NGINX reverse proxy. The configutation was managed by Ansible and the same roles had been applied to a number of other hosts. These were also exhibiting the same error which meant I had the same error on over 20 websites and applications. There was a lot of diversity in the group of containers affected, so this quickly allowed me to focus on a small set of commonanlities amongst the sites that were being affected.

It was very difficult to know where to start becuase there was such little feedback on the error, so I started looking at the raw HTTP conversations using Firefox, this yielded nothing. So I then went to the command line and ran curl. BINGO, I got the the following error: curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1). When I appended -v, I got the following:

*   Trying 116.202.000.000:443...
* TCP_NODELAY set
* Connected to site.example.com (116.202.000.000) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=site.example.com
*  start date: May 23 09:25:56 2020 GMT
*  expire date: Aug 21 09:25:56 2020 GMT
*  subjectAltName: host "site.example.com" matched cert's "site.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ea761971d0)
> GET / HTTP/2
> Host: site.example.com
> User-Agent: curl/7.65.3
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [h2]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host site.example.com left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

So it turns out the error was caused by an invalid header being sent to the client. The server advertises that the connection can be upgraded to HTTP/2 despite the fact the connection has already been established using HTTP/2. The protocol as defined in RFC7540 does not permit this:

8.1.2.2. Connection-Specific Header Fields

HTTP/2 does not use the Connection header field to indicate
connection-specific header fields; in this protocol, connection-
specific metadata is conveyed by other means. An endpoint MUST NOT
generate an HTTP/2 message containing connection-specific header
fields; any message containing connection-specific header fields MUST
be treated as malformed (Section 8.1.2.6).

The only exception to this is the TE header field, which MAY be
present in an HTTP/2 request; when it is, it MUST NOT contain any
value other than "trailers".

This means that an intermediary transforming an HTTP/1.x message to
HTTP/2 will need to remove any header fields nominated by the
Connection header field, along with the Connection header field
itself. Such intermediaries SHOULD also remove other connection-
specific header fields, such as Keep-Alive, Proxy-Connection,
Transfer-Encoding, and Upgrade, even if they are not nominated by the
Connection header field.

Note: HTTP/2 purposefully does not support upgrade to another
protocol. The handshake methods described in Section 3 are
believed sufficient to negotiate the use of alternative protocols.

The root cause, "How did this invalid header get sent to the client?"

In my setup an NGINX reverse proxy recevied requests from the public internet and then forwarded them to Apache instances running in LXC containers on the same host. While the connection between the clients browser and the NGINX reverse proxy was over HTTPS and used HTTP/2, the connection onwards to the Apache server used neither. So when an Ansible role, which assumed that TLS should be setup on all Apache servers, inserted Protocols h2 into the Apache config it created the scenario described above and fell fowl of RFC7540.

The fix

Don't advertised HTTP/2 twice, remove Protocols h2 from the Apache config.

Further work

Understand why the three web browsers from different vendors had exactly the same error on iOS and why the shared their approach to the protocal error with curl.

This article was first posted at https://www.dfoley.ie/blog/website-not-working-all-browsers-ios #indieweb #posse

DEV Community: David Foley

libvirt network forwarding inconsistencies

TL;DR

The Problem

The Solution

Step 1: Open the editor the XML definition for your virtual network bridge

Step 2: Change the forwarding mode

Step 3: Restart the virtual network bridge

Links

Credits

Raspberry Pi Pico NOT flashing with arduino-cli

Error Flashing the Device

The solution: post_install.sh

Links

Credits

dnf packaging errors for MariaDB and borgbackup

TL;DR

The Problem

Fixing MariaDB

Fixing borgbackup

Activate partial LV after drive failure

TL;DR

The Problem

Website not working in any browser on iOS

TL;DR

The Error

Safari and Firefox (iOS)

Chrome (iOS)

Investigation

The root cause, "How did this invalid header get sent to the client?"

The fix

Further work