DEV Community: Daniel Holth

Configuring Tvheadend for ATSC in North America

Daniel Holth — Fri, 31 Jan 2025 23:22:04 +0000

Here's how I configured Tvheadend for ATSC (North America)

Install Tvheadend. I would up using the Fedora packages and not the container, after spending too much time debugging containerization issues.
dmesg showed error messages related to the firmware. The USB stick's firmware was available in Ubuntu but not in Fedora. The same model card can have different tuners, but my Hauppauge WinTV HVR-850 required sudo wget -O /lib/firmware/dvb-fe-xc5000-1.6.114.fw https://linuxtv.org/downloads/firmware/dvb-fe-xc5000-1.6.114.fw per the linuxtv.org website.
Enable TCP 9981, 9982 through the firewall.

In tvheadend's configuration interface,

In TV Adapters, enable the ATSC-T side of your adapter. Always press Save.
In Networks, add and enable an ATSC-T network perhaps named after your city. Use United States: us-ATSC-center-frequencies-8VSB-062009. The ones with a suffix omit no-longer-used TV frequencies, don't worry about w_scan, we only have 35 to worry about.
Go back to TV Adapters. Add your new network to the enabled ATSC-T adapter's Networks dropdown.
Force Scan on the Networks tab. The system will check each ATSC channel for signal, and populate found channels on Services.
On Services, choose Map All Services

Tvheadend will gradually populate the program guide for channels that provide one. It's possible to watch the channels that don't appear in the guide by switching to them.

Hopefully now you have Tvheadend showing signs of life, and can configure it to work with a more pleasant frontend interface like Kodi.

Self-updating Containers on Linux with Quadlet aka podman-system-generator

Daniel Holth — Wed, 30 Oct 2024 18:20:26 +0000

If you are using a modern Linux system like Fedora 40, you may want to configure systemd which manages services, and podman to run containers as an alternative to Docker. This setup makes it possible to run rootless containers as a normal user without worrying about giving the container global permissions.

systemd uses .service files to control what to run. For example, samba, a file server, has /usr/lib/systemd/system/samba.service

[Unit]
Description=Samba AD Daemon
Documentation=man:samba(8) man:samba(7) man:smb.conf(5)
Wants=network-online.target
After=network.target network-online.target

[Service]
Type=notify
PIDFile=/run/samba.pid
LimitNOFILE=16384
EnvironmentFile=-/etc/sysconfig/samba
ExecStart=/usr/sbin/samba --foreground --no-process-group $SAMBAOPTIONS
ExecReload=/bin/kill -HUP $MAINPID
Environment=KRB5CCNAME=FILE:/run/samba/krb5cc_samba

[Install]
WantedBy=multi-user.target

We want our container to start automatically on boot and restart when needed, so we need a systemd .service file. In the old days we would create a container using podman and then manually put a .service file for that container somewhere.

This works for a while but our service references an exact container. When we want to update the container, we have to re-write the service file to reference the update. On my system the service file referenced the new container in one place and the old one in the other, I forgot to update both, and systemd kept restarting my container.

Six months later, I discovered Quadlet.

Quadlet is a generator to alleviate the burden of manually writing .service files, one of many found in /usr/lib/systemd/system-generators:

# ls -1 /usr/lib/systemd/system-generators
kdump-dep-generator.sh
nfs-server-generator
podman-system-generator
rpc-pipefs-generator
selinux-autorelabel-generator.sh
systemd-bless-boot-generator
systemd-cryptsetup-generator
systemd-debug-generator
systemd-fstab-generator
systemd-getty-generator
systemd-gpt-auto-generator
systemd-hibernate-resume-generator
systemd-integritysetup-generator
systemd-rc-local-generator
systemd-run-generator
systemd-system-update-generator
systemd-sysv-generator
systemd-veritysetup-generator
zram-generator

Generators convert higer-level configuration to systemd early on during startup, when we call systemctl daemon-reload, or when we call systemctl --user daemon-reload for rootless services.

Examples

Mosquitto is a simple message broker I've chosen to run as root. After placing the .container file in the right place and running systemctl daemon-reload (or rebooting), we can run systemctl start mosquitto. All the podman commands to pull, run docker.io/library/eclipse-mosquitto:latest are taken care of for us.

# cat /etc/containers/systemd/mosquitto.container
[Container]
ContainerName=mosquitto
HostName=hass
Image=docker.io/library/eclipse-mosquitto:latest
Volume=/etc/mosquitto:/mosquitto/config
Volume=/mosquitto/data
Volume=/mosquitto/log
# mqtt
PublishPort=1883:1883

The service winds up in /etc/containers/systemd/mosquitto.container:

# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
#
# unifi.container
[X-Container]
ContainerName=mosquitto
HostName=hass
Image=docker.io/library/eclipse-mosquitto:latest
Volume=/etc/mosquitto:/mosquitto/config
Volume=/mosquitto/data
Volume=/mosquitto/log
# mqtt
PublishPort=1883:1883

[Unit]
Wants=network-online.target
After=network-online.target
SourcePath=/etc/containers/systemd/mosquitto.container
RequiresMountsFor=%t/containers
RequiresMountsFor=/etc/mosquitto

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=mosquitto --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d -v /etc/mosquitto:/mosquitto/config -v /mosquitto/data -v /mosquitto/log --publish 1883:1883 --hostname hass docker.io/library/eclipse-mosquitto:latest

Rootless Containers

The rootless .container files go into ~/.config/containers/systemd. I'm using one to run Home Assistant.

$ find .* -name \*.container
.config/containers/systemd/homeassistant.container

My container file looks like this. I also gave it permission to /dev/ttyACM0 for a ZigBee radio, and CAP_NET_RAW,CAP_NET_BIND_SERVICE to attempt to allow it to bind to any port, and monitor the network for new devices.

$ cat homeassistant.container
[Unit]
Description=Homeassistant

[Container]
AddCapability=CAP_NET_RAW,CAP_NET_BIND_SERVICE
AddDevice=/dev/ttyACM0
ContainerName=homeassistant
Environment=TZ=America/New_York
Image=ghcr.io/home-assistant/home-assistant:latest
Network=host
Volume=/home/me/hass/config:/config
PodmanArgs=--privileged --group-add=keep-groups
# GroupAdd=keep-groups # future release? https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html

[Install]
WantedBy=default.target

The generated service file winds up in /run/<uid>/systemd. Now an up-to-date Home Assistant container runs as my user when I start my Fedora server.

/run/user/1000/systemd$ cat generator/*
# Automatically generated by /usr/lib/systemd/user-generators/podman-user-generator
#
[Unit]
Wants=network-online.target
After=network-online.target
Description=Homeassistant
SourcePath=/home/me/.config/containers/systemd/homeassistant.container
RequiresMountsFor=%t/containers
RequiresMountsFor=/home/me/hass/config

[X-Container]
AddCapability=CAP_NET_RAW,CAP_NET_BIND_SERVICE
AddDevice=/dev/ttyACM0
ContainerName=homeassistant
Environment=TZ=America/New_York
Image=ghcr.io/home-assistant/home-assistant:latest
Network=host
Volume=/home/me/hass/config:/config
PodmanArgs=--privileged --group-add=keep-groups

# GroupAdd=keep-groups # future release? https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html

[Install]
WantedBy=default.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=homeassistant --cidfile=%t/%N.cid --replace --rm --cgroups=split --network=host --sdnotify=conmon -d --device=/dev/ttyACM0 --cap-add=cap_net_raw,cap_net_bind_service -v /home/me/hass/config:/config --env TZ=America/New_York --privileged --group-add=keep-groups ghcr.io/home-assistant/home-assistant:latest

Conclusion

systemd has many features including user-mode process management, powerful cron-like service timers, and now container management features that long-term Linux users may not be familiar with. After struggling through the sometimes-spotty documentation of this newish feature, we've managed to create a maintainable set of containerized services on our home Linux server.

Using nginx as a SSL offloading proxy to MQTT

Daniel Holth — Sat, 12 Oct 2024 20:06:38 +0000

nginx's stream_proxy and stream_ssl modules can be used to add tls/ssl support to mosquitto or any tcp server.

This can be useful because mosquitto only supports certain certificate types. The mbedTLS stack I was using on a microcontroller board wasn't able to understand my certificate loaded into mosquitto; but it was able to make https:// requests to nginx on the same server with the same certificate. Our nginx server supports, and has permission to read, these certificates.

For MQTT we will proxy tcp directly and we will not use http. Edit /etc/nginx/nginx.conf to add include /etc/nginx/conf.d/*.stream; at the top level, as /etc/nginx/conf.d/*.conf is loaded into the http section.

On Fedora 40, dnf install nginx-mod-stream to get the module.

/etc/nginx/conf.d/01-mqtt.stream::

stream {
        upstream backend {
                hash $remote_addr consistent;
                server localhost:1883;
        }
        server {
                listen       8883 ssl;
                listen       [::]:8883 ssl;
                server_name  hass;

                ssl_certificate "/etc/letsencrypt/live/hass/fullchain.pem";
                ssl_certificate_key "/etc/letsencrypt/live/hass/privkey.pem";
                ssl_session_cache shared:SSLB:1m;
                ssl_session_timeout  10m;
                ssl_ciphers PROFILE=SYSTEM;
                ssl_prefer_server_ciphers on;

                proxy_connect_timeout 1s;
                proxy_timeout 10m; # is default
                proxy_pass backend;
        }
}

After a restart, nginx listens on the mqtt-tls port 8883, handles encryption, and forwards plain text to localhost:1883.

For mosquitto, the only configuration needed is

password_file /mosquitto/config/pwfile.conf
listener 1883
# or listener 1883 0.0.0.0 to listen on all interfaces

How we reduced conda's index fetch bandwidth by 99%

Daniel Holth — Sun, 23 Apr 2023 19:20:39 +0000

The new conda 23.3.1 release from March, 2023 includes an --experimental=jlap flag or experimental: ["jlap"] .condarc setting that can reduce repdata.json fetch bandwidth by orders of magnitude. This is how we developed conda's new incremental repodata feature.

Conda is a cross-platform, language-agnostic binary package manager that includes a constraint solver to choose compatible sets of packages. Before conda can install a package, it downloads information about all available packages. This allows the solver to make global decisions about which packages to install. The time and bandwidth spent downloading this metadata can be significant, but we have improved this in conda 23.3.1. By enabling the experimental: ["jlap"] feature in .condarc, conda users can see more than a 99% reduction in index fetch bandwith.

Idea

Traditionally, conda tries to fetch each channel's entire repodata.json or smaller current_repodata.json every time the cache expires, typically between 30 seconds to 20 minutes from the last remote request. If the channel has not changed this is a quick 304 Not Modified, but very active channels will change several times an hour. Any change, usually only a few added packages, requires the user to re-download the entire index; most conda users will be familiar with this process. My manager said one of Anaconda's customers wanted a better way to track changes in our repository and I became interested in solving the problem.

I began to pursue a solution based on computing patches between successive versions of repodata.json to let users download only the changes, once they had an initial, complete copy of the index in their cache.

Initial Prototypes

I chose RFC 6902 JSON Patch, a generic patch format for .json. JSON Patch shows the logical difference between two .json files instead of comparing the files textually, saving space by discarding formatting.

I wrote a Rust implementation of a .json patchset format based on a single .json file with an array of patches.

The Rust implementation helped to simplify the format and show that it could be language independent. In Python, we might have included optional or multiple-typed "string or null" fields without thinking about it. In Rust, "mandatory, always a single type" fields are easiest to specify. I was surprised to find that this change simplified the Python code as well.

Experimentation showed that PyPy was faster than Rust for comparing two large repodata.json; CPython's json.loads and json.dumps also perform shockingly well. Stopped development on the Rust implementation.

I began writing a formal specification based on array-of-patches style.

Specification

I wrote a Conda Enhancmenent Proposal (CEP) for a new json lines based .jlap format. The earlier array-of-patches format has the same problem as repodata.json but in miniature, since the client has to download every patch each time. In contrast, the new .jlap system appends patches to the end of a file. It is designed to fetch new patches from a growing file using HTTP Range requests. With this system, update bandwidth is proportional to the amount of change that has happened since last time.

The .jlap format allows clients to fetch the newest patches and only the newest patches with a single HTTP Range request. It consists of a leading checksum, any number of lines of patches, one per line in the JSON Lines format, and a trailing checksum.

The checksums are constructed in such a way that the trailing checksum can be re-verified without re-reading (or retaining) the beginning of the file, if the client remembers an intermediate checksum. The trailing checksum is used to make sure the rest of the remote file was not changed.

When repodata.json changes, the server wil truncate the metadata line, appending new patches, a new metadata line and a new trailing checksum.

We needed patch data to test this system. I wrote a web service that would create that data. The service checked repodata.json every five minutes, compared the current and previous versions, updated a patch file and hosted it separately from the main repository.

The initial demonstration used a repodata.json proxy, fetching patches from repodata.fly.dev while forwarding package requests to the upstream server. The user points conda at the proxy server instead of the default channel. Another prototype adds a similar proxy into conda's requests-based HTTP backend, but duplicates an extra local cache on top of the existing cache.

The .jlap format is generic over JSON. The underlying checksum/range request system is generic for any growing line-based file. Consider adapting it if you have a similar problem.

Server Side Improvements

I became the conda-index maintainer, rewriting it from conda-build index to a new standalone package. We solved speed problems updating large repositories like conda-forge and defaults. This involvement made it easy to control the server-side data so that we could improve the client and server together.

Team Move

I became a full-time member of the conda team, having transferred from Anaconda's packaging team. I slowly began understanding conda's internals enough to be able to produce a solution that could integrate with conda's existing cache, instead of a local caching proxy.

Implementation of zstandard-compressed repodata.json, parallel downloads

On November 6, a community member noticed that fetching repodata.json with on-the-fly server compression was slower than fetching it uncompressed, if your connection was faster than the remote server's gzip compressor. We merged server-side repodata.json.zst support into conda-index on November 14.

November's conda release included parallel package downloads and extraction, a speed improvement that makes a difference proportional to your latency to the package server.

Ship `repodata.json.zst`

We shipped zstd-compressed repodata to conda-forge and defaults on December 15.

Refactor cache

January's 23.1.0 conda release included a refactor of its cache that was important for incremental repodata.json support. Instead of inlining cache metadata into a modified repodata.json, conda stores unmodified repodata.json in its cache, and stores cache metadata in a separate file. We avoid having to reserialize repodata.json in many cases and can preserve its orginal content and formatting.

Wolf Vollprecht submitted a draft CEP to standardize the cache format between conda and mamba. We continue to converge on a shared format.

Ship `repodata.jlap` incremental repodata

March's 23.3.1 conda release shipped support for repodata.jlap under the --experimental=jlap flag. This feature also includes support for repodata.json.zst with a fallback to repodata.json if unavailable.

When the cache is empty, conda will try to download repodata.json.zst. This file is much faster to download and decompress compared to Content-Encoding: gzip and is slightly smaller.

When the cache is primed, conda will look for repodata.jlap. It will download the entire file, apply any relevant patches (comparing to a content hash of repodata.json) and remember the length of the patch file.

On subsequent fetches, conda will use a HTTP Range request to download new bytes, if any, added to repodata.jlap, and apply new patches.

Conclusion

This screen capture of a conda-forge/noarch search shows that were able to download a single update to the channel in 1464 bytes for what would have otherwise have been a 10358799-byte download of the complete index. The patch size is proportional to the amount of change that has happened since the last time you ran conda.

When --experimental=jlap is enabled, frequent conda users will see much faster index updates, especially when bandwidth is limited.

certbot hooks for gandi.net v5 DNS API

Daniel Holth — Sun, 13 Nov 2022 01:53:34 +0000

I wanted to use gandi.net's API to update DNS for certbot/letsencrypt. I wasn't able to get TSIG (a standard DNS update method) to work. There is a plugin for gandi's API, but I wanted to make my own.

First, I went to https://account.gandi.net/en/users/USERNAME/security to generate an API key.

I saved the key as HTTP headers in /etc/letsencrypt/gandi-api-key.txt and chmod 600 gandi-api-key.txt to keep it secret:

Authorization: Apikey SECRET-API-KEY
Content-Type: application/json

and created two hooks:

/etc/letsencrypt/gandi_hook.py:

#!/usr/bin/env python3
import os
import json
import subprocess
import time

domain = os.environ["CERTBOT_DOMAIN"]  # "domain.example.com"
domain, zone, tld = domain.split(".")
zone = ".".join((zone, tld))

command = f"curl -H @gandi-api-key.txt https://api.gandi.net/v5/livedns/domains/{zone}/records/_acme-challenge.{domain}/TXT -X PUT -d".split()

payload = json.dumps(
    {"rrset_ttl": 300, "rrset_values": [os.environ["CERTBOT_VALIDATION"]]}
)

subprocess.run(command + [payload], cwd="/etc/letsencrypt", check=True)

# https://www.gandi.net/en-US/domain/dns '250 ms'
time.sleep(1)

and the cleanup script, /etc/letsencrypt/gandi_cleanup.py:

#!/usr/bin/env python3
import os
import json
import subprocess
import time

domain = os.environ["CERTBOT_DOMAIN"]  # "fedora.monotreme.club"
domain, zone, tld = domain.split(".")
zone = ".".join((zone, tld))

command = f"curl -H @gandi-api-key.txt https://api.gandi.net/v5/livedns/domains/{zone}/records/_acme-challenge.{domain}/TXT -X DELETE".split()

subprocess.run(command, cwd="/etc/letsencrypt", check=True)

chmod +x gandi_hook.py gandi_cleanup.py

To run certbot, certbot certonly --manual --preferred-challenge=dns --manual-auth-hook /etc/letsencrypt/gandi_hook.py --manual-cleanup-hook /etc/letsencrypt/gandi_cleanup.py -d test.monotreme.club --test-cert.

Modern Linux includes a systemd timer to automatically renew the certificates. Enable it with systemctl enable certbot-renew.timer and systemctl start certbot-renew.timer.

Unlike cron, the systemd timer has a random start time to avoid DDOS'ing letsencrypt's servers.

[Timer]
OnCalendar=*-*-* 00/12:00:00
RandomizedDelaySec=12hours
Persistent=true

Using sql.js-httpvfs with browser

DEV Community: Daniel Holth

Configuring Tvheadend for ATSC in North America

Self-updating Containers on Linux with Quadlet aka podman-system-generator

Examples

Rootless Containers

Conclusion

Using nginx as a SSL offloading proxy to MQTT

How we reduced conda's index fetch bandwidth by 99%

Idea

Initial Prototypes

Specification

Server Side Improvements

Team Move

Implementation of zstandard-compressed repodata.json, parallel downloads

Ship repodata.json.zst

Refactor cache

Ship repodata.jlap incremental repodata

Conclusion

certbot hooks for gandi.net v5 DNS API

Ship `repodata.json.zst`

Ship `repodata.jlap` incremental repodata