DEV Community: Charles Hartmann

Test your Compose ViewModels in milliseconds: fakes, runTest, the Main dispatcher and Turbine

Charles Hartmann — Sat, 31 Oct 2026 14:00:57 +0000

The ViewModel is where a Compose screen's behavior lives: what the screen shows while loading, what happens when the user saves, which error appears when the network fails. It is plain Kotlin and runs on the JVM, so its tests take milliseconds and need no emulator. That makes it the best-value test in an Android app. This post tests a small note editor's ViewModel with a fake repository, coroutine test dispatchers, Turbine and SavedStateHandle.

The code under test

The editor loads a note by id, lets the user change the title and body, and saves. Its state is a single StateFlow<EditorUiState>, and the important parts look like this:

class EditorViewModel(
    private val repository: NotesRepository,
    private val savedStateHandle: SavedStateHandle,
) : ViewModel() {

    private val noteId: Long? = savedStateHandle.get<Long>("noteId")
    private val _state = MutableStateFlow(EditorUiState())
    val state: StateFlow<EditorUiState> = _state.asStateFlow()

    init {
        viewModelScope.launch {
            val note = noteId?.let { repository.getNote(it) }
            _state.update {
                it.copy(
                    title = savedStateHandle["draftTitle"] ?: note?.title.orEmpty(),
                    body = note?.body.orEmpty(),
                    isLoading = false,
                )
            }
        }
    }

    fun save() {
        val current = _state.value
        if (current.title.isBlank()) {
            _state.update { it.copy(error = "Title is required") }
            return
        }
        viewModelScope.launch {
            _state.update { it.copy(isSaving = true) }
            when (val result = repository.save(noteId, current.title, current.body)) {
                is SaveResult.Ok -> _state.update { it.copy(isSaving = false, saved = true) }
                is SaveResult.Failed -> _state.update {
                    it.copy(isSaving = false, error = result.message)
                }
            }
        }
    }
    // onTitleChange() also writes savedStateHandle["draftTitle"]
}

NotesRepository is an interface. That one decision makes the ViewModel testable: a test can hand it anything that implements the interface.

Fakes, not mocks, by default

A mock (MockK, for example) is configured per test: you tell it what to return and afterwards verify which calls were made. A fake is a small, real implementation written by hand, usually backed by a map in memory. Google's testing guidance recommends fakes as the default test double, for practical reasons:

A fake behaves like the real thing, so tests check behavior ("after saving, the note is in the repository") rather than implementation details ("save was called once with these arguments").
A fake is written once and reused by every test. Mocks are configured again in every test.
Refactoring the ViewModel does not break tests that use a fake, as long as the behavior stays the same.

Keep mocks for the edges: a third-party class you cannot implement easily, or a check that something was not called.

class FakeNotesRepository : NotesRepository {
    private val notes = mutableMapOf<Long, Note>()
    private var nextId = 1L
    var failNextSave: String? = null
    var saveDelayMs = 0L

    fun seed(vararg seed: Note) {
        seed.forEach { notes[it.id] = it }
        nextId = (notes.keys.maxOrNull() ?: 0L) + 1
    }

    override suspend fun getNote(id: Long): Note? = notes[id]

    override suspend fun save(id: Long?, title: String, body: String): SaveResult {
        delay(saveDelayMs)                 // virtual time in tests
        failNextSave?.let { failNextSave = null; return SaveResult.Failed(it) }
        val note = Note(id ?: nextId++, title, body)
        notes[note.id] = note
        return SaveResult.Ok(note.id)
    }

    fun all(): List<Note> = notes.values.toList()
}

Step 1: replace the Main dispatcher

viewModelScope launches on Dispatchers.Main. On the JVM there is no Android main looper, so a test that creates this ViewModel fails with an error about the Main dispatcher missing. The fix is Dispatchers.setMain and Dispatchers.resetMain from kotlinx-coroutines-test, and the Android Developers coroutine testing guide recommends wrapping them in a JUnit rule so no test forgets:

class MainDispatcherRule(
    val testDispatcher: TestDispatcher = UnconfinedTestDispatcher(),
) : TestWatcher() {
    override fun starting(description: Description) {
        Dispatchers.setMain(testDispatcher)
    }

    override fun finished(description: Description) {
        Dispatchers.resetMain()
    }
}

Every ViewModel test then starts with one line: @get:Rule val mainDispatcherRule = MainDispatcherRule().

Step 2: understand runTest and the two dispatchers

runTest runs the test body as a coroutine with virtual time: a delay(5_000) inside it completes instantly, while the test still sees five seconds pass. There are two test dispatchers, and the difference matters:

StandardTestDispatcher queues new coroutines. They do not run until the test yields, with advanceUntilIdle(), advanceTimeBy(), runCurrent() or by suspending. It is the default in runTest, and it lets you check intermediate states such as "saving is in progress".
UnconfinedTestDispatcher starts new coroutines eagerly until their first suspension. Simple tests read shorter, but you cannot observe the state between "launched" and "finished".

When MainDispatcherRule installs a test dispatcher as Main, runTest reuses its scheduler, so advanceUntilIdle() in the test also drives coroutines launched in viewModelScope.

Step 3: the first tests

class EditorViewModelTest {

    @get:Rule val mainDispatcherRule = MainDispatcherRule(StandardTestDispatcher())

    private val repository = FakeNotesRepository()

    private fun viewModel(noteId: Long? = null) = EditorViewModel(
        repository,
        SavedStateHandle(mapOf("noteId" to noteId)),
    )

    @Test fun saving_blank_note_shows_error_and_saves_nothing() = runTest {
        val vm = viewModel()
        advanceUntilIdle()

        vm.onTitleChange("   ")
        vm.save()
        advanceUntilIdle()

        assertThat(vm.state.value.error).isEqualTo("Title is required")
        assertThat(repository.all()).isEmpty()
    }

    @Test fun saving_shows_progress_then_marks_saved() = runTest {
        repository.saveDelayMs = 2_000
        val vm = viewModel()
        advanceUntilIdle()
        vm.onTitleChange("Groceries")

        vm.save()
        runCurrent()                       // the save starts, then suspends in delay()
        assertThat(vm.state.value.isSaving).isTrue()

        advanceUntilIdle()                 // two virtual seconds pass instantly
        assertThat(vm.state.value.saved).isTrue()
    }
}

The classic mistake is asserting before advancing. With StandardTestDispatcher, the coroutine launched in init has not run when the constructor returns, so a test that reads state.value.title straight away sees the default empty title and fails, or worse, passes for the wrong reason.

Step 4: sequences of states with Turbine

When the order of states is the behavior, Turbine collects the flow and lets you await each item:

@Test fun save_emits_saving_then_saved() = runTest {
    repository.saveDelayMs = 1_000         // a suspension between the two updates
    val vm = viewModel()
    advanceUntilIdle()
    vm.onTitleChange("Packing list")

    vm.state.test {
        assertThat(awaitItem().isSaving).isFalse()   // current state
        vm.save()
        assertThat(awaitItem().isSaving).isTrue()
        assertThat(awaitItem().saved).isTrue()
        cancelAndIgnoreRemainingEvents()
    }
}

StateFlow is conflated: if two updates happen before the collector runs, it sees only the latest, and equal consecutive values are dropped. A test that expects a state that was immediately replaced waits and times out. That is why the fake has a delay: the suspension between "saving" and "saved" lets the collector see both. Assert on the states that matter, and make sure something suspends between them.

The stateIn trap

A flow exposed with stateIn(viewModelScope, SharingStarted.WhileSubscribed(5_000), emptyList()) runs its upstream only while someone collects it. Reading .value in a test with nobody collecting returns the initial empty list forever. Collect it in backgroundScope, which runTest cancels for you at the end:

backgroundScope.launch(UnconfinedTestDispatcher(testScheduler)) {
    vm.notes.collect()
}

Process death, at the ViewModel level

SavedStateHandle has a public constructor that takes a map, so you can simulate the system recreating the ViewModel: write a value through the first instance, build a second one with a handle holding the same values, and assert the state comes back.

val restored = EditorViewModel(repository, SavedStateHandle(mapOf(
    "noteId" to null,
    "draftTitle" to handle.get<String>("draftTitle"),
)))
advanceUntilIdle()
assertThat(restored.state.value.title).isEqualTo("Half-typed")

Checklist

The ViewModel depends on interfaces, and each has a reusable fake.
A MainDispatcherRule replaces Dispatchers.Main in every ViewModel test.
Tests advance (advanceUntilIdle(), runCurrent()) before asserting.
Turbine tests have a real suspension between the states they expect.
stateIn flows are collected in backgroundScope before reading .value.
Saved state is tested by rebuilding the ViewModel from a SavedStateHandle.

This is adapted from my book Test It Before They Do (Android the Right Way, Book 8). The free sample is here and the full book is here.

Want the week's homelab articles, videos and one tip from the books in your inbox every Sunday? Get the free weekly email.

Drafted with AI help and edited by the author; based on his book.

Real HTTPS for your homelab Kubernetes apps without opening a port: k3s Traefik, cert-manager and a DNS-01 wildcard

Charles Hartmann — Fri, 30 Oct 2026 14:05:06 +0000

Every web app in a homelab cluster should answer at a proper hostname with a certificate your browser trusts, and you should not have to forward a single port from the internet to get there. On k3s you already have most of the pieces: the Traefik ingress controller ships with it. Add cert-manager, one DNS-01 issuer and one wildcard certificate, and every app you route gets real HTTPS automatically.

The trap

There are two traps, one on each side.

The first is living with certificate warnings: self-signed certificates on every app, a browser you have trained to click "proceed anyway", and a password manager that refuses to autofill. That habit is exactly what an attacker on your network would rely on.

The second is fixing it the obvious way: opening port 80 to the internet so Let's Encrypt can run an HTTP-01 challenge. That publishes your lab to every scanner on the internet for the sake of a padlock icon.

The DNS-01 challenge avoids both. You prove you control the domain by creating a DNS record through your DNS provider's API. Nothing on your network has to be reachable from outside.

How a request reaches a pod

When something does not load, walk this path in order:

Your browser asks LAN DNS for paperless.home.example.com and gets the private address of Traefik's LoadBalancer Service (for example 192.168.40.200, handed out by MetalLB).
The node that holds that address receives the connection and hands it to a Traefik pod.
Traefik terminates TLS, matches the hostname against its routes, and forwards the request to the app's ClusterIP Service.
The Service sends it to one of the app's ready pods.

Step 1: check the Traefik that k3s installed

k3s installs Traefik from a bundled Helm chart into kube-system and registers an IngressClass called traefik. Because it is the only ingress controller in the cluster, it picks up standard Ingress objects without being told.

kubectl get ingressclass
kubectl -n kube-system get deploy,svc traefik
kubectl -n kube-system get helmchart traefik -o yaml | grep -i version

Change Traefik's settings with a HelmChartConfig named traefik, never by editing the generated chart manifest. One gotcha when you copy older snippets: the chart version k3s ships moved the Service type from service.type to service.spec.type, and the k3s documentation warns that the old key is silently ignored.

While you are there, redirect plain HTTP to HTTPS everywhere:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      web:
        http:
          redirections:
            entryPoint:
              to: websecure
              scheme: https
              permanent: true

kubectl apply -f traefik-config.yaml
kubectl -n kube-system get pods -l app.kubernetes.io/name=traefik -w

k3s re-runs the chart with the merged values and Traefik restarts.

A note if you are following an older tutorial that installs ingress-nginx first: in November 2025 Kubernetes SIG Network and the Security Response Committee announced the retirement of ingress-nginx, with best-effort maintenance until March 2026 and no releases or security fixes after that. k3s already gives you Traefik. Translate those guides' Ingress objects to the traefik class, and review any nginx.ingress.kubernetes.io/* annotations, because Traefik ignores them.

Step 2: install cert-manager

cert-manager watches Certificate objects, gets certificates from an issuer such as Let's Encrypt, stores them in Secrets and renews them before they expire. The project publishes its Helm chart in an OCI registry (v1.21.3 at the time of writing; check for the current release):

helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.3 \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true
kubectl -n cert-manager get pods

crds.enabled=true lets the chart install and upgrade cert-manager's CustomResourceDefinitions.

Step 3: a DNS-01 issuer

You need a domain you own, hosted at a DNS provider cert-manager can talk to. It supports several directly (Cloudflare, Route 53, Google Cloud DNS, Azure DNS, DigitalOcean and others) and many more through webhooks. With Cloudflare, follow cert-manager's documentation: create an API token, not the global API key, with Zone - DNS - Edit and Zone - Zone - Read, limited to your zone.

kubectl -n cert-manager create secret generic cloudflare-api-token \
  --from-literal=api-token='PASTE-THE-TOKEN'

Then a ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: you@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Create a second issuer named letsencrypt-staging with the server https://acme-staging-v02.api.letsencrypt.org/directory. Staging has much more generous rate limits, so get everything working there first. The token secret lives in the cert-manager namespace because, for a ClusterIssuer, cert-manager looks for referenced secrets in its own namespace by default.

Step 4: one wildcard for every internal name

You could request one certificate per app. A single wildcard for *.home.example.com is simpler and has a privacy benefit: every certificate Let's Encrypt issues is published in public Certificate Transparency logs, and a wildcard publishes *.home.example.com instead of a list of every app you run. Wildcards can only be issued through DNS-01, which is one more reason to use it.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-home
  namespace: kube-system
spec:
  secretName: wildcard-home-tls
  issuerRef:
    name: letsencrypt-staging        # switch to letsencrypt-prod when it works
    kind: ClusterIssuer
  dnsNames:
    - "home.example.com"
    - "*.home.example.com"

kubectl apply -f wildcard.yaml
kubectl -n kube-system get certificate wildcard-home -w     # READY becomes True
kubectl describe certificaterequest -n kube-system          # if it does not

Once the staging certificate shows Ready, change issuerRef.name to letsencrypt-prod, apply again, and delete the old secret so cert-manager issues a fresh one.

Step 5: make it Traefik's default certificate

The certificate lives in kube-system, but an Ingress can only reference Secrets in its own namespace. Instead of copying the secret into every namespace, make it Traefik's default. Traefik serves the default certificate for any TLS connection that no more specific certificate matches:

apiVersion: traefik.io/v1alpha1
kind: TLSStore
metadata:
  name: default
  namespace: kube-system
spec:
  defaultCertificate:
    secretName: wildcard-home-tls

Now any Ingress under home.example.com that asks for TLS gets the wildcard automatically, with no secretName at all:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
  namespace: whoami
spec:
  tls:
    - hosts: ["whoami.home.example.com"]
  rules:
    - host: whoami.home.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: whoami
                port:
                  number: 80

Plain Ingress is a fine choice here. The k3s documentation states that the traditional Ingress API is still supported and not planned for deprecation, and most Helm charts still generate it.

Keep internal names internal

DNS-01 means the certificate is public but the service is not. Nothing in public DNS points at your lab; the only public record cert-manager creates is a temporary TXT record during issuance. *.home.example.com resolves only through your LAN resolver's override to a private address, so from outside your network the names do not resolve and the address is unreachable.

For access from outside the house, use a VPN instead of a port forward. Once your phone is connected, it uses the LAN resolver and every app works exactly as it does at home.

Checklist

kubectl get ingressclass shows traefik, and Traefik settings live in one HelmChartConfig.
Plain HTTP redirects to HTTPS.
cert-manager is installed with crds.enabled=true.
The DNS provider credential is a scoped API token, stored in the cert-manager namespace.
Staging and production ClusterIssuers exist, and you proved the setup on staging first.
One wildcard Certificate is Ready and set as Traefik's default in a TLSStore.
No port is forwarded from the internet; remote access goes through a VPN.

This is adapted from my book Kubernetes Without the Overwhelm (Homelab the Right Way, Book 12). The free sample is here and the full book is here.

Want the week's homelab articles, videos and one tip from the books in your inbox every Sunday? Get the free weekly email.

Drafted with AI help and edited by the author; based on his book.

Your app starts slow after every update: generate a Baseline Profile and prove it is applied

Charles Hartmann — Fri, 23 Oct 2026 14:21:08 +0000

A Baseline Profile is the single change most likely to make an existing Compose app start faster and scroll more smoothly without touching its logic. It is a list of the classes and methods your app needs on its most important paths, shipped inside the app, so the runtime can compile them ahead of time on install instead of discovering them slowly on the user's phone. This post generates one with the Baseline Profile Gradle plugin and then proves, three ways, that it is applied.

The trap

Your Kotlin compiles to DEX bytecode. The Android Runtime (ART) can interpret it, JIT-compile hot methods while the app runs, or run machine code compiled ahead of time (AOT). ART does not AOT-compile your whole app on install. It records which methods were used into a profile and compiles them later, when the device is idle and charging.

That works eventually, and "eventually" is the problem. After every install and every update, users start in the slow path until the device has recorded enough and found a quiet moment to compile it. Google Play's Cloud Profiles help later installs of a release, but they need time and users to build up, so they do nothing for the first users of each new version.

Jetpack libraries, Compose included, ship their own profiles, which is why a plain Compose app already benefits a little. Library profiles only cover library code, though. Your screens, ViewModels and repositories are in nobody's profile but yours. Google's documentation states that Baseline Profiles improve code execution speed by about 30% from the first launch for the code paths they include; your own gain depends on your app, and Macrobenchmark tells you what it is.

Step 1: the generator module

The Baseline Profile Gradle plugin (androidx.baselineprofile, 1.5.0) does the plumbing. Android Studio's Baseline Profile Generator module template (File > New > New Module, Android Studio Iguana or newer, AGP 8.2 or newer) writes this for you. The generator is a com.android.test module that points at the app:

// baselineprofile/build.gradle.kts
plugins {
    id("com.android.test")
    id("org.jetbrains.kotlin.android")
    id("androidx.baselineprofile")
}
android {
    namespace = "com.example.fieldnotes.baselineprofile"
    compileSdk = 36
    defaultConfig {
        minSdk = 28
        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
    }
    targetProjectPath = ":app"
}
dependencies {
    implementation("androidx.benchmark:benchmark-macro-junit4:1.5.0")
    implementation("androidx.test.uiautomator:uiautomator:2.4.0")
}

Generation needs root on the device, except on Android 13 (API 33) and newer, where Macrobenchmark can collect profiles without root. A Gradle-managed device with systemImageSource = "aosp" works everywhere because AOSP images allow root.

The app module applies the same plugin, depends on the generator, and includes ProfileInstaller:

// app/build.gradle.kts
plugins {
    id("com.android.application")
    id("org.jetbrains.kotlin.android")
    id("androidx.baselineprofile")
}
dependencies {
    implementation("androidx.profileinstaller:profileinstaller:1.4.1")
    baselineProfile(project(":baselineprofile"))
}
baselineProfile {
    saveInSrc = true      // commit the generated profile
    mergeIntoMain = true  // one profile for all variants
}

ProfileInstaller matters for installs that do not come from Google Play: sideloaded builds, internal testing installs and other stores. There, Play is not around to hand the profile to ART, so ProfileInstaller writes it on the app's first launch.

Step 2: write the journeys

The generator is an instrumented test with BaselineProfileRule. It drives the app through the paths users take every day, and the rule records every class and method that ran:

@RunWith(AndroidJUnit4::class)
@LargeTest
class BaselineProfileGenerator {
    @get:Rule val rule = BaselineProfileRule()

    @Test fun generate() = rule.collect(
        packageName = "com.example.fieldnotes",
        includeInStartupProfile = true,
    ) {
        pressHome()
        startActivityAndWait()
        device.wait(Until.hasObject(By.res("notes_list")), 5_000)

        val list = device.findObject(By.res("notes_list"))
        list.setGestureMargin(device.displayWidth / 5)
        list.fling(Direction.DOWN)
        device.waitForIdle()

        device.findObject(By.res("note_row")).click()
        device.wait(Until.hasObject(By.res("editor_body")), 5_000)
        device.pressBack()
    }
}

The test finds elements by resource id, so in Compose expose test tags as resource ids once near the root with Modifier.semantics { testTagsAsResourceId = true } and tag the elements the generator needs with Modifier.testTag("notes_list").

Good journeys follow a few rules. Cover what users actually do, most common first. Wait for content, because if the journey moves on before the list loads, the list's code is not in the profile. Keep it deterministic with local or seeded data rather than a live server. And never sign in to real accounts from a generator.

Step 3: generate and commit

Run the plugin's task, not the test directly:

./gradlew :app:generateBaselineProfile

The plugin builds the app, runs the generator and copies the result into your sources:

app/src/main/generated/baselineProfiles/baseline-prof.txt
app/src/main/generated/baselineProfiles/startup-prof.txt

Commit both. Every release build compiles them into the app, and you will find the binary form under assets/dexopt/baseline.prof in the AAB or APK. The rules inside are plain text: HSPLcom/example/fieldnotes/ui/NotesViewModel;->loadNotes()V, where H means hot, S startup and P post-startup.

The second file is a Startup Profile, produced because of includeInStartupProfile = true. It tells R8 to put every class needed at startup into the primary DEX file. It needs R8 enabled for release (isMinifyEnabled = true) and DEX layout optimization, which is on by default from AGP 8.3.

Step 4: prove it is applied

A profile that does not reach the device is just a file in a repository.

Benchmark it. Compare a fresh install with no ahead-of-time compilation against one that requires your profile and fails loudly if the app has none:

@Test fun startupNoCompilation() = startup(CompilationMode.None())

@Test fun startupBaselineProfile() = startup(
    CompilationMode.Partial(baselineProfileMode = BaselineProfileMode.Require))

Run both on the same physical device and compare the median timeToInitialDisplayMs.

Check the device by hand:

PKG=com.example.fieldnotes
adb install app-release.apk
adb shell am broadcast \
  -a androidx.profileinstaller.action.INSTALL_PROFILE \
  $PKG/androidx.profileinstaller.ProfileInstallReceiver
adb shell am force-stop $PKG
adb shell cmd package compile -f -m speed-profile $PKG
adb shell dumpsys package dexopt | grep -A 1 $PKG

The broadcast prints "1" on success, and the dexopt status should read speed-profile.

Ask from inside the app. ProfileVerifier.getCompilationStatusAsync() reports whether the profile was installed and compiled (isCompiledWithProfile, hasProfileEnqueuedForCompilation()). Log it in internal builds; "enqueued" is normal right after an install from outside Play.

Checklist

The generator module applies androidx.baselineprofile and targets :app.
The app depends on ProfileInstaller and on the generator through baselineProfile(...).
Journeys cover launch, the main list, opening an item, and wait for content.
baseline-prof.txt and startup-prof.txt are committed.
None versus Partial(Require) shows a measured gap on a real device.
dumpsys package dexopt reads speed-profile.
The profile is regenerated after screen redesigns, major Compose upgrades and before releases.

This is adapted from my book Fast by Default (Android the Right Way, Book 7). The free sample is here and the full book is here.

Want the week's homelab articles, videos and one tip from the books in your inbox every Sunday? Get the free weekly email.

Drafted with AI help and edited by the author; based on his book.

Snapshots on a schedule, replicas on another box: sanoid and syncoid pull replication for ZFS

Charles Hartmann — Thu, 22 Oct 2026 14:05:51 +0000

A ZFS snapshot is a read-only picture of a dataset at one instant. It takes about a second to create and costs no space at first. Snapshots on a schedule, thinned automatically and copied to a second machine, are the single biggest upgrade most homelab storage can get. This post builds exactly that with two tools from the same project: sanoid takes and prunes snapshots, syncoid replicates them.

The trap

Homelab storage usually fails in one of two opposite ways. The first lab has no snapshots at all, so an accidental rm -rf on a share is permanent. The second takes snapshots every fifteen minutes and never prunes them: ten thousand of them, a pool that is mysteriously full, and a zfs list -t snapshot that takes a minute to answer.

Both labs also believe the snapshots are their backup. They are not. A snapshot lives in the same pool as the data. If the pool dies (two disks in a RAIDZ1, a controller that writes garbage, a mistaken zpool destroy), the snapshots die with it. Snapshots protect against changes: deletions, overwrites, bad upgrades, ransomware on a share. Backups protect against loss. You need both, and replication to a second box gets you most of the way.

Why snapshots are cheap

ZFS never overwrites data in place. When a block changes, the new version goes to a new location and the dataset's pointers move to it. A snapshot simply keeps the old pointers, so until something changes the snapshot and the live dataset share every block.

That also tells you where snapshot space goes. A snapshot grows as the live data moves away from it: delete a 20 GB folder and the space is not freed until every snapshot that still references those blocks is destroyed. When a pool fills up unexpectedly, this command sorts snapshots by the space only they hold, and the culprit is usually at the bottom:

zfs list -t snapshot -o name,used -s used

Step 1: a retention policy with sanoid

Debian and Proxmox VE package sanoid, and the package ships a systemd timer that runs it every 15 minutes:

apt install sanoid
systemctl enable --now sanoid.timer
systemctl list-timers | grep sanoid

The policy lives in /etc/sanoid/sanoid.conf: datasets at the top, templates at the bottom. A trimmed version:

[tank/documents]
    use_template = important
[tank/shares]
    use_template = standard
    recursive = yes
[tank/media]
    use_template = light

[template_important]
    frequently = 0
    hourly = 48
    daily = 30
    weekly = 8
    monthly = 12
    yearly = 0
    autosnap = yes
    autoprune = yes

[template_standard]
    hourly = 24
    daily = 14
    weekly = 4
    monthly = 3
    autosnap = yes
    autoprune = yes

[template_light]
    hourly = 0
    daily = 7
    weekly = 0
    monthly = 0
    autosnap = yes
    autoprune = yes

Each dataset section names a template, and recursive = yes applies it to every child dataset. The numbers are how many snapshots of each kind to keep. Recent history is dense (48 hourlies) and older history is sparse (12 monthlies), so about a hundred snapshots cover a full year, and sanoid deletes each one when it falls off the end of its rung.

Anything you leave out comes from /etc/sanoid/sanoid.defaults.conf, which you should not edit, so set every interval explicitly in your own templates.

Test the policy without touching anything, then take a round of snapshots and look:

sanoid --cron --readonly --verbose      # simulate
sanoid --take-snapshots --verbose       # take snapshots now
zfs list -t snapshot -o name,used -s creation tank/documents | tail -5

One rule if the box also runs Proxmox guests: pick one snapshot owner per dataset. Proxmox's own VM snapshots and storage replication create and delete snapshots on the same zvols. If Proxmox replication or vzdump with PBS already covers your guests, leave tank/vm out of sanoid, or keep its policy very light.

Step 2: how incremental replication works

zfs send turns a snapshot into a stream of bytes; zfs receive turns that stream back into a dataset on another pool. The first send is full. Every later send is incremental: only the blocks that changed between two snapshots travel. Both sides must keep the last common snapshot, or the next incremental has nothing to build on.

By hand it looks like this, with nas and backup as SSH host aliases with key-based login:

zfs snapshot tank/documents@rep-1
zfs send tank/documents@rep-1 | ssh backup zfs receive -u backup/documents
zfs snapshot tank/documents@rep-2
zfs send -i @rep-1 tank/documents@rep-2 | ssh backup zfs receive -u backup/documents

zfs receive -u does not mount the received dataset, which avoids mountpoint clashes on the target. Tracking the last common snapshot for every dataset, every night, is the part nobody keeps up with. syncoid does it for you: it finds the newest common snapshot, sends the incremental, resumes interrupted transfers when both sides support it, and cleans up its own snapshots.

Step 3: pull, do not push

A backup box that pulls from the NAS is safer than a NAS that pushes to the backup box. If the NAS is compromised, an attacker holding its SSH key can delete everything on a push target. With pull replication, the NAS has no credentials for the backup box at all.

On the backup box, a small script run by cron at 01:30:

cat > /usr/local/sbin/pull-replicas <<'EOF'
#!/bin/sh
S="/usr/sbin/syncoid --recursive --no-sync-snap --quiet"
$S nas:tank/photos    backup/photos
$S nas:tank/documents backup/documents
EOF
chmod +x /usr/local/sbin/pull-replicas
echo '30 1 * * * root /usr/local/sbin/pull-replicas' > /etc/cron.d/pull-replicas

By default syncoid creates its own short-lived syncoid_ snapshot for each run. --no-sync-snap tells it to use the snapshots sanoid already took, which keeps the target's history identical to the source. --recursive includes child datasets. For an encrypted dataset, add --sendoptions=w for a raw send: the blocks travel and are stored still encrypted, and the backup box never needs the key. Keep every passphrase somewhere that does not depend on the pool it unlocks.

The received datasets still need pruning. On the backup box, give them a template that never takes snapshots and only prunes what replication brings in:

[backup/photos]
    use_template = backup
    recursive = yes
[backup/documents]
    use_template = backup
    recursive = yes

[template_backup]
    autosnap = no
    autoprune = yes
    hourly = 48
    daily = 30
    weekly = 8
    monthly = 12

For tighter security, replicate as a non-root user with ZFS permission delegation: on the NAS, zfs allow -u replicator send,hold,snapshot tank, and run syncoid on the backup box with --no-privilege-elevation. The receiving user needs create,mount,receive on the target dataset. Test the delegation once by hand before relying on it.

A TrueNAS box works as the target too. It speaks standard ZFS over SSH, so the same syncoid commands work; create the destination parent dataset in the TrueNAS UI first and let TrueNAS own the retention for those datasets.

Step 4: prove it

Replication you have never restored from is a hope, not a plan. Once a month, restore one real file from the replica and compare checksums from both machines:

# On the backup box
zfs clone backup/documents@autosnap_2026-10-16_00:00:02_daily backup/restore-test
zfs set mountpoint=/mnt/restore-test backup/restore-test
sha256sum /mnt/restore-test/taxes/2025.pdf
ssh nas sha256sum /tank/documents/taxes/2025.pdf
zfs destroy backup/restore-test

Matching checksums are the proof.

Checklist

Every dataset that matters has a sanoid template, and tank/vm has exactly one snapshot owner.
sanoid.timer is enabled and zfs list -t snapshot shows recent autosnaps.
A second box receives incremental replication, ideally by pulling.
Encrypted datasets replicate raw, and the keys are stored off the pool.
The backup box prunes with a backup template (autosnap = no).
You have restored one file from the replica and compared checksums.

This is adapted from my book Store It Right (Homelab the Right Way, Book 11). The free sample is here and the full book is here.

Want the week's homelab articles, videos and one tip from the books in your inbox every Sunday? Get the free weekly email.

Drafted with AI help and edited by the author; based on his book.

Build a Proxmox cloud-init template from the official Debian 13 image (and clone VMs that configure themselves)

Charles Hartmann — Tue, 20 Oct 2026 14:02:31 +0000

Every VM in a rebuildable lab starts from the same place: a template made from a distribution's official cloud image. Cloud images are small, minimal installs built to be configured on first boot by cloud-init, which reads its settings from a little drive that Proxmox attaches to the VM. That is how a fresh clone wakes up with the right hostname, your user, your SSH key and its address, without anyone touching a console.

Pick the image

Debian publishes cloud images at cloud.debian.org. At the time of writing (October 2026) the latest folder for trixie (Debian 13) holds, among others:

debian-13-genericcloud-amd64.qcow2: for cloud and virtual machines, with a smaller set of hardware drivers. This is the one for Proxmox VMs.
debian-13-generic-amd64.qcow2: the same system with a wider set of drivers, for odd hardware.

Ubuntu publishes its own cloud images at cloud-images.ubuntu.com, and the Proxmox documentation uses one in its example. The commands below are the same for either.

Build the template by hand, once

Run these on the Proxmox host. The steps follow the Proxmox VE Cloud-Init Support documentation; VM ID 9000 and the storage names are examples.

cd /var/lib/vz/template/iso
wget https://cloud.debian.org/images/cloud/trixie/latest/\
debian-13-genericcloud-amd64.qcow2

qm create 9000 --name debian13-tmpl --memory 2048 \
  --cores 2 --net0 virtio,bridge=vmbr0 \
  --scsihw virtio-scsi-pci --ostype l26 --agent enabled=1
qm set 9000 --scsi0 local-lvm:0,import-from=\
/var/lib/vz/template/iso/debian-13-genericcloud-amd64.qcow2
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot order=scsi0
qm set 9000 --serial0 socket --vga serial0

What each step does:

qm create makes an empty VM with a VirtIO network card on bridge vmbr0 and the VirtIO SCSI controller that cloud images expect.
import-from copies the downloaded image into a new disk on local-lvm and attaches it as scsi0.
--ide2 ...:cloudinit adds the small cloud-init drive. Proxmox writes the user, network and SSH settings onto it.
--boot order=scsi0 boots straight from the disk, skipping the CD-ROM check.
The serial console is what many cloud images expect as their display. The Proxmox docs note that if a particular image does not work with it, you switch back to the default display.

Set the cloud-init defaults

Settings made on the template are inherited by every clone, so put the parts that are the same everywhere here:

qm set 9000 --ciuser admin
qm set 9000 --sshkeys /root/admin_ed25519.pub
qm set 9000 --ipconfig0 ip=dhcp
qm set 9000 --searchdomain home.example.com \
  --nameserver 192.168.20.1

--sshkeys takes a file with one public key per line. Use keys, not --cipassword; the Proxmox documentation itself says passwords are generally not recommended here.

Add your own first-boot steps

The most common extra is the QEMU guest agent: the Debian cloud image does not include it, and without it Proxmox (and any tooling that asks Proxmox) cannot read the VM's address or shut it down cleanly.

The --cicustom option lets you supply your own files in place of the generated ones. It accepts user, network, meta and vendor files. Replacing user throws away the user and SSH key settings above, so add a vendor file instead: cloud-init merges it with the generated user data.

Custom files must live on a storage that allows the snippets content type. On the default local storage that is /var/lib/vz/snippets/. Enable the content type under Datacenter > Storage > local > Content, then create the file:

#cloud-config
# /var/lib/vz/snippets/debian-vendor.yaml
package_update: true
packages:
  - qemu-guest-agent
runcmd:
  - systemctl enable --now qemu-guest-agent

Attach it to the template and look at what a clone will receive:

qm set 9000 --cicustom "vendor=local:snippets/debian-vendor.yaml"
qm cloudinit dump 9000 user

Watch out: the Proxmox docs point out that custom cloud-init files must be available on every node a VM may migrate to, or the VM will not start there. In a cluster, use shared storage for snippets or copy the file to every node.

Turn it into a template

qm template 9000

A template cannot be started; it can only be cloned. That is the point: nobody logs into it, so it never drifts.

Clone and personalize

qm clone 9000 120 --name docker01
qm set 120 --ipconfig0 ip=192.168.20.120/24,gw=192.168.20.1
qm disk resize 120 scsi0 20G
qm start 120

After a minute, ssh admin@192.168.20.120 should work with your key. The cloud image's disk is small, which is why the clone's disk is grown with qm disk resize; cloud-init grows the root filesystem to fill it on boot.

Linked or full clones

When you clone a template, Proxmox tries to make a linked clone by default: the new disk only stores the differences from the template's disk. It is fast and small. The catch is that the clone depends on the template; you cannot delete the template while linked clones exist, and not every storage type supports them. Pass --full to get an independent copy.

For a rebuildable lab, full clones are usually the better default. The template will be rebuilt whenever a new Debian point release arrives, and full clones do not care.

Checklist

Debian 13 genericcloud image imported, cloud-init drive added, serial console set.
User, SSH key, DNS and search domain set on the template, not on each clone.
A vendor snippet installs qemu-guest-agent; qm cloudinit dump shows it merged in.
qm template run; clones made with --full.
The snippet is kept in your repository; the copy on the Proxmox host is just a deployment of it.

This is adapted from my book Automate Your Homelab (Homelab the Right Way, Book 9). The free sample is here and the full book is here.

Want the week's homelab articles, videos and one tip from the books in your inbox every Sunday? Get the free weekly email.

Drafted with AI help and edited by the author; based on his book.

Promtail is end of life: ship your homelab's logs to Loki with Grafana Alloy instead

Charles Hartmann — Mon, 19 Oct 2026 14:03:43 +0000

The default homelab log strategy is SSH and journalctl, one machine at a time, after the fact. By the time you look, the container was recreated and its logs went with it, or the journal on a small root disk already rotated.

The second trap is building a log stack from an old tutorial. Many still use Promtail, which reached end of life on March 2, 2026. Grafana's documentation states that it receives no further updates and that future development happens in Grafana Alloy, so new setups should start with Alloy.

How the pieces fit

Loki stores logs. Unlike a full-text search engine it indexes only a small set of labels (host, unit, container) and stores the log lines compressed in chunks, which keeps it light enough for a homelab. Alloy is the agent: it reads the journal and Docker's container logs on each host, attaches labels, and pushes the lines to Loki. Grafana queries Loki with LogQL.

Install Alloy on each host

Alloy runs as a native service, which makes reading the journal straightforward. The steps follow Grafana's Debian instructions (Alloy 1.20 at the time of writing, October 2026). On the Proxmox host and on each Docker host:

apt install gpg
mkdir -p /etc/apt/keyrings
wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] \
https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list
apt-get update && apt-get install alloy
usermod -aG adm,systemd-journal alloy

The last line matters: Alloy's documentation warns that without membership in adm and systemd-journal the journal source starts without an error and collects nothing. On Docker hosts, also add the alloy user to the docker group so it can read container logs through the socket (with the usual caveat that the socket means full control of the host).

The configuration

It lives in /etc/alloy/config.alloy and is a set of components wired together: sources forward to a writer.

// /etc/alloy/config.alloy on docker1
loki.write "default" {
  endpoint {
    url = "http://192.168.10.20:3100/loki/api/v1/push"
  }
}

// systemd journal, with unit and level as labels
loki.relabel "journal" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label = "unit"
  }
  rule {
    source_labels = ["__journal_priority_keyword"]
    target_label = "level"
  }
}

loki.source.journal "host" {
  forward_to    = [loki.write.default.receiver]
  relabel_rules = loki.relabel.journal.rules
  labels        = {host = "docker1", job = "journal"}
}

// Docker containers, labelled by container name
discovery.docker "local" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "containers" {
  targets = discovery.docker.local.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label = "container"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  forward_to = [loki.write.default.receiver]
  labels     = {host = "docker1", job = "docker"}
}

On the Proxmox host, keep the loki.write and journal blocks and drop the Docker ones; the journal there carries pvedaemon, pveproxy, pvestatd, backup jobs and the kernel. Restart with systemctl restart alloy and check systemctl status alloy. Alloy also serves a debugging UI on port 12345 when started with a listen address, which shows each component and whether it is healthy.

Labels: few and boring

Loki is fast because its index is small. Every distinct combination of label values creates a new stream, and too many streams make Loki slow and heavy. Keep labels to things with a handful of values that you filter by: host, job, unit, container, level. Never turn request IDs, user names, IP addresses or full paths into labels; search for them in the line instead.

Querying with LogQL

In Grafana, open Explore, pick the Loki data source and start with a label selector, then filter lines:

{host="pve1", unit="pvedaemon.service"}
{job="docker", container="immich_server"} |= "error"
{job="journal", level=~"err|crit|alert|emerg"}
sum by (host) (count_over_time({level="err"}[1h]))

The first shows the Proxmox API daemon's log. The second shows only lines containing "error" from one container. The third shows every high-priority journal line across the lab. The last turns logs into a metric: errors per host per hour, which makes a good panel on an overview dashboard.

Don't forget the local copies

Loki gives you a searchable copy, but Docker still keeps container logs on the host, and with the default json-file driver they grow without limit unless configured. Set limits in /etc/docker/daemon.json on each Docker host and restart Docker (existing containers pick it up when recreated):

{
    "log-driver": "json-file",
    "log-opts": { "max-size": "10m", "max-file": "3" }
}

On the hosts themselves, cap the journal too: SystemMaxUse=500M in /etc/systemd/journald.conf, then systemctl restart systemd-journald. Loki is now the long memory; local logs only need to cover a few days.

Checklist

Alloy, not Promtail, on every host.
The alloy user is in adm and systemd-journal (and docker on Docker hosts).
Labels: host, job, unit, container, level. Nothing high-cardinality.
Docker's json-file logs capped in daemon.json; the journal capped with SystemMaxUse.
A Logs panel under the graphs on your host dashboard, so a spike and its log lines sit together.

This is adapted from my book Watch Your Homelab (Homelab the Right Way, Book 7). The free sample is here and the full book is here.

Want the week's homelab articles, videos and one tip from the books in your inbox every Sunday? Get the free weekly email.

Drafted with AI help and edited by the author; based on his book.

Why your 2-node Proxmox cluster freezes (and the QDevice fix that is not a third server)

Charles Hartmann — Mon, 19 Oct 2026 02:49:29 +0000

Two Proxmox nodes feel like redundancy. One box dies, the other carries on. That is the picture most people have when they join a second machine to the cluster.

Out of the box, it is the opposite: two machines that can each stop the other from working. This post covers why that happens, the small fix that solves it, and the two related traps that make a cluster look haunted.

How quorum works, in one paragraph

A Proxmox cluster uses Corosync to agree on who is alive. Each node gets one vote, and the cluster is quorate when more than half the votes are present. Without quorum, the cluster filesystem at /etc/pve goes read-only. You can't start guests, change configs or create backups, and nothing in the UI saves.

The trap: 1 vote out of 2 is not a majority

Now do the arithmetic for two nodes. Lose either one, and the survivor holds 1 vote out of 2. That is exactly half, and half is not more than half. So the survivor freezes.

That is the part that catches people. The node that is still up, still powered and still on the network is the one that stops working, because it has no way to know whether the other node is dead or just unreachable. Refusing to write is the safe choice for the cluster, and it is a very annoying choice for you.

You can see the vote count at any time:

pvecm status

The fix: a third vote that isn't a third server

A QDevice is a tiny external voter. It can run on a Raspberry Pi, a NAS container or any always-on Linux box that isn't one of the cluster nodes. It gives the cluster a third vote without a third hypervisor.

With three votes, losing one node leaves 2 out of 3. That is more than half, so the survivor stays quorate and keeps working.

The setup is three installs and one command:

# On the external box (Debian/Ubuntu/Raspberry Pi OS):
apt install corosync-qnetd

# On EVERY cluster node:
apt install corosync-qdevice

# On ONE cluster node (root SSH to the QDevice box must work):
pvecm qdevice setup 192.168.10.5

# Check: you should see 3 total votes and "Qdevice" listed
pvecm status

Replace 192.168.10.5 with the address of your own external box.

Gear mentioned (As an Amazon Associate I earn from qualifying purchases. Affiliate links.) Raspberry Pi · All the gear the books recommend

Where the QDevice must not live

Don't run the QDevice as a VM or container on one of the cluster nodes. It is tempting, because the hardware is already there. But if that node goes down, it takes its own vote and the QDevice's vote with it, and you are back to 1 vote out of 3 on the survivor. Separate hardware is the whole point.

QDevice placement	One node dies
VM or container on a cluster node	That node takes two votes with it; no quorum
Separate always-on box	The third vote survives; the other node keeps quorum

The related trap: a node on Wi-Fi that keeps flapping

Even with enough votes, a cluster can look broken if one member keeps dropping in and out. The usual cause is Wi-Fi.

Corosync is extremely sensitive to latency and jitter. It wants a stable, low-latency link: single-digit milliseconds, consistently. Wi-Fi delivers great average speeds and terrible consistency. A microwave, a neighbor's channel change or a roaming decision adds 50 to 300 ms of jitter for a few seconds. Corosync sees missed tokens, decides the node is gone, and the cluster reshuffles. A few seconds later it is back. Then it is gone again.

The symptoms look random:

The node shows a red X, then turns green again
/etc/pve is briefly read-only
Backup jobs fail with "cluster not ready"
The logs fill with lines like [TOTEM] A processor failed, forming new configuration

Check for it:

journalctl -u corosync --since "24 hours ago" | \
  grep -Ei "failed|new membership|retransmit"
corosync-cfgtool -s        # link status per node
ping -c 100 -i 0.2 <other-node> | tail -2   # look at mdev/max, not avg

Look at the mdev and max numbers from the ping, not the average. A good average with a big max is exactly the jitter Corosync hates.

The fix isn't a Corosync tuning trick. It's a cable. If the node physically can't be wired, use a powerline or MoCA adapter (still better than Wi-Fi), or take it out of the cluster and run it standalone. A standalone node you manage separately is far less trouble than a cluster member that drops out twice an hour.

If your nodes have two network ports, give Corosync a second link (link1) on a separate network or a direct cable between the nodes. Corosync fails over between links automatically.

Gear mentioned (As an Amazon Associate I earn from qualifying purchases. Affiliate links.) Powerline or MoCA Ethernet adapter · All the gear the books recommend

When you're stuck without quorum right now

If one node of two is dead and you need the survivor working immediately, there is an emergency lever:

pvecm expected 1

This tells the survivor to accept being alone. It is an emergency lever, not a fix. Only use it when you are sure the other node is truly off, because two nodes that each think they are the whole cluster can write conflicting configs. Once the crisis is over, the real fix is still the QDevice.

Scripts: wait for quorum instead of failing halfway

If you script changes on a cluster with a flaky member, check quorum before each write and retry, instead of letting the script die partway through a batch:

until pvecm status | grep -q "Quorate:.*Yes"; do
  echo "waiting for quorum..."; sleep 10
done
pct destroy 123 --purge

The loop waits until pvecm status reports the cluster as quorate, then runs the write. Wrap each destructive or config-changing step this way and a brief membership blip costs you a pause, not a half-finished job.

Checklist

Every cluster node is wired, ideally on the same switch
Two-node clusters have a QDevice on separate hardware, never on a cluster node
pvecm status shows the expected vote count (3 for two nodes plus a QDevice)
Corosync logs show no membership changes over the past day
Nodes with two ports give Corosync a second link (link1)
Scripts check for quorum before each write
You know the pvecm expected 1 lever, and when not to pull it

Drafted with AI help. I edited it, and it's based on my own setup.

This post is adapted from Chapter 2 of my ebook Homelab the Right Way: Proxmox, OPNsense & Home Assistant. The free sample is a full chapter; the whole book covers the cluster build, storage, backups and the network around it. All 18 of my homelab and Android ebooks are in one $89 bundle (code RIGHTWAY20 takes 20% off through Oct 31). Get the free weekly email: https://chartmann1590.github.io/homelab-the-right-way/newsletter/

The 60-second version is on YouTube: https://youtube.com/shorts/v00ZPYLWcfM

Podcast, episode 1: The green check lies — backups, networking, self-hosting and a turtle

Charles Hartmann — Mon, 19 Oct 2026 00:57:10 +0000

Three AI hosts, real homelab answers. The first episode of the Homelab the Right Way Podcast is out: Ep. 1: The green check lies — backups, networking, self-hosting and a turtle (46 minutes, published October 18, 2026, free).

Disclosure first: Maya, Sam and Theo are fictional AI hosts. Their voices are AI-generated, the script was written with AI help and edited by a human, and every fact comes from the Homelab the Right Way books and blog. AI hosts, human-edited, facts from the books.

What the episode covers

Backups: the green check lies. Why the green check on a Proxmox backup job proves almost nothing, the 3-2-1 plan, keep-daily=7, the backup job that silently skips new VMs, and the restore drills that tell you whether you actually have a backup.
Storage. RAID is not a backup, the NFS share that fills your root disk, and sanoid + syncoid pull replication for ZFS.
Networking. What a /24 actually means, and why a 2-node Proxmox cluster freezes (plus the QDevice fix).
Self-hosting. The traps around Immich, Nextcloud and Vaultwarden.
Home Assistant. The motion light that turns off on you, and the Zigbee fix that beats any setting.
The turtle tank. The ESP32 wire that can fry the board, and the DS18B20 readings that are not temperatures.

Listen

Episode page: https://redcircle.com/shows/homelab-the-right-way/episodes/55cb28a3-0a54-4fc3-952d-70a2fedea93f
Show page: https://redcircle.com/shows/homelab-the-right-way
RSS feed (paste it into any podcast app): https://feeds.redcircle.com/b40a097c-964d-4d5c-820f-a74bbe389dc7
Show notes, chapters and every article mentioned: https://chartmann1590.github.io/homelab-the-right-way/posts/podcast-ep-1-the-green-check-lies/

Spotify and Apple Podcasts listings are not live yet; the RSS feed works in any app today.

Gear mentioned in the episode: the TP-Link TL-SG105 5-port gigabit switch and the Kasa Smart Plug Ultra Mini EP10. As an Amazon Associate the show earns from qualifying purchases; the links are in the show notes.

Originally published on the Homelab the Right Way blog: https://chartmann1590.github.io/homelab-the-right-way/posts/podcast-ep-1-the-green-check-lies/

Your ufw rules don't cover your Docker ports (and the log-rotation setting to make before your first container)

Charles Hartmann — Sun, 18 Oct 2026 14:01:37 +0000

Two things bite almost every new Docker host in a homelab. One is a security surprise. The other is a disk that fills up a few weeks later. Both are a five-minute fix if you do them before the first real container.

The firewall surprise

Docker's own documentation on packet filtering says it plainly: Docker and ufw are incompatible. When you publish a container's port, Docker routes the traffic in the nat table, before it reaches the INPUT chain that ufw uses. Your "deny all" rule never sees it.

So this does not do what it looks like it does:

sudo ufw default deny incoming
docker run -d -p 9000:9000 some/admin-tool

Port 9000 is open to anyone who can reach the host, ufw or not.

The fix is a habit, not a firewall rule

Publish as little as possible. On a well-laid-out Docker host only the reverse proxy publishes ports (80 and 443). Every other container is reached through the proxy on an internal Docker network and has no ports: section at all.
Bind admin ports to localhost. If a tool must publish a port for you alone, write 127.0.0.1:9000:9000 and reach it through an SSH tunnel:

# Reach a localhost-only admin port from your laptop
ssh -L 9000:127.0.0.1:9000 admin@192.168.20.10

Filter at the network edge. Your router decides which VLANs can reach the Docker VM at all. Rules for traffic to containers can also go in Docker's DOCKER-USER chain; the "Packet filtering and firewalls" page in Docker's docs covers it.

A compose file that follows this looks like:

services:
  app:
    image: some/app:1.2
    networks: [proxy]
    # no ports: the reverse proxy reaches it on the proxy network
  admin:
    image: some/admin-tool:3.1
    ports:
      - "127.0.0.1:9000:9000"
networks:
  proxy:
    external: true

Set log rotation before the first real container

By default Docker uses the json-file logging driver and performs no log rotation. A chatty container can fill the disk with logs. Docker's docs suggest either the local driver or rotation options for json-file. Set them once in /etc/docker/daemon.json:

{
    "log-driver": "json-file",
    "log-opts": {
      "max-size": "10m",
      "max-file": "3"
    }
}

Note the quotes around "3": log options in daemon.json must be strings. Restart Docker to apply:

sudo systemctl restart docker
docker info --format '{{.LoggingDriver}}'

Common mistake: changing daemon.json and expecting old containers to pick it up. Logging options apply to containers created after the change. Recreate the running ones:

docker compose up -d --force-recreate

One more thing about the docker group

The post-install page warns that the docker group grants root-level privileges: anyone in it can start a container that mounts / from the host. On a single-admin homelab VM that is usually acceptable. Never add a service account or a shared login to it, and anything that can talk to /var/run/docker.sock controls the host, so mount the socket read-only where the tool allows it and only into tools you trust.

Checklist

Only the reverse proxy publishes ports (80 and 443).
Admin tools are bound to 127.0.0.1 or sit behind the proxy.
VLAN rules on the router decide who can reach the Docker VM.
Log rotation is set in /etc/docker/daemon.json, and existing containers were recreated.
Nobody but you is in the docker group.

This is adapted from my book Containers Without the Chaos (Homelab the Right Way, Book 8). The free sample is here and the full book is here.

Drafted with AI help and edited by the author; based on his book.

MediaPipe LLM Inference is maintenance-only: start on LiteRT-LM (and add this ABI check)

Charles Hartmann — Sat, 17 Oct 2026 14:01:22 +0000

Running a language model on the phone itself is the most private option you have and the most demanding. The model file is large, it needs memory while it runs, and it turns battery into words. In return, the feature works in airplane mode on most recent 64-bit phones, and you choose the model.

LiteRT-LM replaces the MediaPipe LLM Inference API

For the last couple of years the usual way to do this was MediaPipe's LLM Inference API. Google's MediaPipe documentation now carries a notice that the LLM Inference API (Android, iOS and Web) is in maintenance-only mode, that new features and optimizations are focused on LiteRT-LM, and that projects should migrate.

New code should start on LiteRT-LM.

LiteRT-LM is Google's runtime for language models on top of LiteRT. Its Kotlin API is small:

an Engine loads a .litertlm model
a Conversation holds the chat state
sendMessageAsync streams the reply as a Kotlin Flow

The current release on Google Maven at the time of writing (October 2026) is 0.18.0, still below 1.0:

dependencies {
    implementation("com.google.ai.edge.litertlm:litertlm-android:0.18.0")
}

Getting a model

LiteRT-LM reads .litertlm files. Google points to the LiteRT Community organization on Hugging Face for ready-made ones, including Gemma builds.

The author ships Android apps on Google Play as Hartmann Studios, and his ollama-android-client keeps a curated catalog in LocalModelCatalog.kt. Its rule: prefer the smallest CPU-friendly quantization (int4 or q8) over f32 "so downloads stay reasonable on mobile data and the model actually fits in RAM". Two entries from that file:

Catalog entry	File	Approx. size
Gemma 3 270M IT (q8)	`gemma3-270m-it-q8.litertlm`	304 MB
Gemma 3 1B IT (int4)	`gemma3-1b-it-int4.litertlm`	584 MB

For the rest of this article, assume the file is in your app's private storage and you have its path.

Engine and conversation

The documentation is explicit that engine.initialize() can take a significant amount of time (it gives "up to 10 seconds" as an example) and should run on a background thread or coroutine. Load the engine once and keep it. Create a new conversation per chat.

class LocalLlm(private val context: Context, private val modelPath: String) {
    private var engine: Engine? = null
    private val lock = Mutex()

    suspend fun engine(): Engine = lock.withLock {
        engine ?: withContext(Dispatchers.Default) {
            Engine(EngineConfig(
                modelPath = modelPath,
                backend = Backend.CPU(),
                cacheDir = context.cacheDir.path, // speeds up later loads
            )).also { it.initialize() }
        }.also { engine = it }
    }

    fun release() { engine?.close(); engine = null }
}

A conversation takes an optional system instruction, earlier messages and sampling settings:

val config = ConversationConfig(
    systemInstruction = Contents.of("You are a concise, friendly assistant."),
    initialMessages = listOf(
        Message.user("What is a homelab?"),
        Message.model("A few computers at home that run your own services."),
    ),
    samplerConfig = SamplerConfig(topK = 10, topP = 0.95, temperature = 0.8),
)

Streaming a reply as a Flow

sendMessageAsync(text) without a callback returns a Flow<Message>, which the documentation recommends for coroutine users. In the book this sits behind a small TextGenerator interface so the UI does not care which backend answers:

override fun generate(messages: List<ChatMessage>): Flow<String> = flow {
    val history = messages.dropLast(1).filter { it.role != Role.System }
        .map {
            if (it.role == Role.User) Message.user(it.text)
            else Message.model(it.text)
        }
    val system = messages.firstOrNull { it.role == Role.System }?.text
    val config = ConversationConfig(
        systemInstruction = system?.let { Contents.of(it) },
        initialMessages = history,
    )
    llm.engine().createConversation(config).use { conversation ->
        conversation.sendMessageAsync(messages.last().text)
            .collect { emit(it.toString()) }
    }
}.flowOn(Dispatchers.Default)

Use flowOn, not withContext, inside a flow { } builder. ollama-android-client's LiteRtChatService.kt documents why in a comment: a flow { } builder cannot emit from a different dispatcher than the one that collected it, and wrapping the body in withContext throws "Flow invariant is violated" at runtime.

The ABI check is not optional

This is the part that comes from a real crash report.

LiteRtChatService.kt notes that the litertlm-android AAR versions it uses only ship native libraries for arm64-v8a and x86_64, with no 32-bit ARM build. Phones without those ABIs threw UnsatisfiedLinkError the moment the native library loaded.

The fix has two parts. First, check Build.SUPPORTED_ABIS up front:

override suspend fun availability(): Availability =
    if (Build.SUPPORTED_ABIS.any { it == "arm64-v8a" || it == "x86_64" })
        Availability.Ready
    else Availability.Unavailable("This phone's processor is not supported")

Second, catch Throwable rather than Exception around engine creation. UnsatisfiedLinkError is an Error, not an Exception, and it slipped past the old catch.

GPU first, CPU as the safety net

The GPU backend is usually faster for language models, but it does not work on every phone or with every model. To use it, the LiteRT-LM Android guide says to declare the OpenCL-related native libraries in the manifest, marked as not required:

<application>
    <uses-native-library android:name="libvndksupport.so" android:required="false"/>
    <uses-native-library android:name="libOpenCL.so" android:required="false"/>
</application>

Then try GPU and fall back to CPU. ollama-android-client builds a list of backend attempts and, inside each, a list of history lengths to try: the full compacted history, then half, then one exchange, then none. Only when every rung fails does the user see an error.

Fitting the conversation into the context window

A language model on a phone has a small context window, and the conversation grows with every message. When the history no longer fits, generation fails. ollama-android-client trims history to a token budget before every request:

It estimates tokens cheaply as characters divided by four, plus a small overhead per message.
It allows history to use 55 percent of the model's token limit, leaving the rest for the new message and the reply.
It keeps the newest exchanges that fit, then adds the very first exchange back if there is room, then up to a couple of older exchanges that share keywords with the new message.

The chat itself stays in Room. Only the copy sent to the model is trimmed.

Releasing the model

A loaded model holds a large amount of native memory. Close the conversation when a chat ends (the use block above does this) and close the engine when the user leaves the AI part of the app or the system signals memory pressure:

class AssistantApp : Application() {
    lateinit var localLlm: LocalLlm

    override fun onTrimMemory(level: Int) {
        super.onTrimMemory(level)
        if (level >= ComponentCallbacks2.TRIM_MEMORY_BACKGROUND) {
            localLlm.release()
        }
    }
}

Checklist

New code on LiteRT-LM, not the MediaPipe LLM Inference API.
engine.initialize() off the main thread, once.
flowOn, never withContext, inside flow { }.
Check Build.SUPPORTED_ABIS for arm64-v8a or x86_64 before loading.
Catch Throwable around engine creation.
GPU first, CPU fallback, with the manifest entries marked not required.
Release the engine on memory pressure.

This is adapted from my book On-Device AI for Android (Android the Right Way, Book 5). The free sample is here and the full book is here.

Drafted with AI help and edited by the author; based on his book.

Ten quick edits should not queue ten syncs: unique work in WorkManager, done right

Charles Hartmann — Fri, 16 Oct 2026 17:47:13 +0000

The user saves a note on the subway and puts the phone in their pocket. Android may kill your process a minute later. The note still has to reach the server when the train comes up for air, maybe in an hour, maybe tomorrow. That is persistent work, and on Android persistent work means WorkManager. Google's offline-first guide says as much: draining an offline queue is persistent work that is often delegated to WorkManager.

WorkManager 2.12.0 is the stable release at the time of writing (October 2026). It stores every request in its own database, so scheduled work survives process death and reboots. The scheduling part is easy. The part that goes wrong is how many syncs you schedule, and what the worker says when one fails.

The trap: one sync per save

The obvious design is to enqueue a SyncWorker from the repository every time a row is written. It works in the emulator. Then a user edits a note ten times in a minute, and ten syncs are queued. Each one pushes the same dirty rows, each one pulls the same pages, and two of them can run at the same time against the same database. Add a retry on a flaky connection and the queue grows faster than it drains.

The fix is three decisions, all of them one line each: make the push unique work, pick the right policy for duplicates, and return Result.retry() only for errors that can actually clear up on their own.

Fix 1: unique work with the right policy

enqueueUniqueWork (one-time) and enqueueUniquePeriodicWork (periodic) take a name and a policy for what to do when unfinished work with that name already exists. For one-time work, ExistingWorkPolicy has four options:

REPLACE cancels the existing work and enqueues the new request.
KEEP ignores the new request and keeps the existing one.
APPEND adds the new work to the end of the existing chain.
APPEND_OR_REPLACE appends unless the existing chain has failed or been cancelled, in which case it starts fresh.

For a push, KEEP is the right one. If a sync is already queued, it will pick up every dirty row when it runs, including the one just saved, so the new request adds nothing.

object SyncScheduler {
    private const val PUSH = "notes-push"
    private const val PERIODIC = "notes-periodic"

    fun requestPush(context: Context, wifiOnly: Boolean) {
        val request = OneTimeWorkRequestBuilder<SyncWorker>()
            .setConstraints(syncConstraints(wifiOnly))
            .setBackoffCriteria(BackoffPolicy.EXPONENTIAL, 30, TimeUnit.SECONDS)
            .build()
        WorkManager.getInstance(context)
            .enqueueUniqueWork(PUSH, ExistingWorkPolicy.KEEP, request)
    }

    fun schedulePeriodic(context: Context, wifiOnly: Boolean) {
        val request = PeriodicWorkRequestBuilder<SyncWorker>(6, TimeUnit.HOURS)
            .setConstraints(syncConstraints(wifiOnly))
            .build()
        WorkManager.getInstance(context)
            .enqueueUniquePeriodicWork(PERIODIC, ExistingPeriodicWorkPolicy.KEEP, request)
    }
}

Call schedulePeriodic from Application.onCreate(); with KEEP it is harmless to call on every launch. Periodic work has a floor: the minimum repeat interval is 15 minutes, the same as JobScheduler. A flex interval lets it run within a window at the end of each period rather than at an exact moment.

Fix 2: constraints that match the user's settings

Constraints make WorkManager wait for the right conditions. For sync, a connected network is usually enough, plus the user's Wi-Fi-only setting:

fun syncConstraints(wifiOnly: Boolean) = Constraints.Builder()
    .setRequiredNetworkType(if (wifiOnly) NetworkType.UNMETERED else NetworkType.CONNECTED)
    .setRequiresBatteryNotLow(true)
    .build()

One consequence is easy to miss: if a constraint stops being met while the worker runs, WorkManager stops the worker and retries it later. Your doWork must be safe to stop at any suspension point, which is one more reason every sync step should be idempotent.

Fix 3: retry only what can succeed later

When a worker returns Result.retry(), WorkManager waits for the backoff delay and runs it again. Result.failure() stops it, and fails everything chained after it. Deciding which errors are transient (timeouts, HTTP 5xx, 429) and which are permanent (401 after a token refresh, 400 for a malformed payload) is the most important line of sync code you will write:

class SyncWorker(
    context: Context,
    params: WorkerParameters,
    private val sync: SyncEngine,
) : CoroutineWorker(context, params) {
    override suspend fun doWork(): Result = when (sync.syncOnce()) {
        SyncOutcome.Done -> Result.success()
        SyncOutcome.Transient -> if (runAttemptCount < 5) Result.retry() else Result.failure()
        SyncOutcome.Permanent -> Result.failure()
    }
}

The backoff numbers come from the WorkManager guide: the delay can be no less than 10 seconds, the policy is LINEAR or EXPONENTIAL, and the default is EXPONENTIAL with a 30-second delay. That is the exponential backoff the offline-first guide recommends for network reads and queued writes, for free: your code only classifies the failure and returns retry(), with no sleep loop of its own. Cap attempts with runAttemptCount, as above, so a permanently broken request does not retry forever.

A worker with constructor parameters beyond context and params needs a WorkerFactory. With Hilt, annotate it with @HiltWorker and use HiltWorkerFactory; without DI, register a custom factory through Configuration.Provider in your Application and disable the default initializer in the manifest.

Chains, expedited work and the foreground rules

Work can run in sequence, each step's output becoming the next step's input. A chain is the natural shape for "upload attachments, then push notes, then pull":

WorkManager.getInstance(context)
    .beginUniqueWork("full-sync", ExistingWorkPolicy.KEEP, uploadAttachments)
    .then(pushNotes)
    .then(pullNotes)
    .enqueue()

The chaining guide's failure rule: when a step fails or is cancelled, every dependent step is marked failed or cancelled too. Keep chains short and make each step meaningful on its own, so a failure tells you something.

When the user is waiting (they tapped "Sync now"), mark the request expedited with setExpedited(OutOfQuotaPolicy.RUN_AS_NON_EXPEDITED_WORK_REQUEST), so an exhausted quota degrades to normal work instead of dropping it. On Android versions before 12, WorkManager may run expedited work as a foreground service, so any worker used with setExpedited must implement getForegroundInfo(); leaving it out can crash on older platform versions.

For a long first sync, a CoroutineWorker can call setForeground() to run as a foreground service with an ongoing notification. If your app targets Android 14 (API 34) or higher, you must specify a foreground service type for every long-running worker, declare it in the manifest by merging it into WorkManager's own service, and request the matching permission (FOREGROUND_SERVICE_DATA_SYNC for sync). Starting with Android 16, the guide notes, long-running workers can exhaust your app's job quota. For most apps the honest design is a sync that does a bounded amount of work per run and lets the next run continue, rather than one heroic foreground session.

Show the state from the repository, not the worker

To display "Syncing..." or "Last sync failed", observe the unique work's state as a Flow with getWorkInfosForUniqueWorkFlow("notes-push") and map RUNNING, ENQUEUED and FAILED to UI states. The worker itself never touches the UI.

Checklist

The push is unique work named once, enqueued with ExistingWorkPolicy.KEEP.
Periodic sync is unique too, scheduled from Application.onCreate(), interval 15 minutes or more.
Constraints read the user's Wi-Fi-only setting; doWork is safe to stop at any suspension point.
retry() only for timeouts, 5xx and 429; failure() for 400 and 401 after a refresh.
Attempts capped with runAttemptCount; backoff left to WorkManager.
Chains short; a failed step fails its dependents, and that is intended.
Expedited workers implement getForegroundInfo(); long-running workers declare a foreground service type when targeting API 34+.
Every sync step is idempotent, because a stopped worker runs it again.

A sync that outlives the app is not the clever part. The clever part is making sure only one of them runs, and that it gives up at the right moment.

Originally published on the Homelab the Right Way blog. This article is based on the ebook Data That Survives; every book in the series has a free sample. Written with AI help and edited by me.

Is Jellyfin really using your GPU? A six-rung ladder to prove it on Proxmox

Charles Hartmann — Fri, 16 Oct 2026 17:46:47 +0000

Most of the time a media server just sends the file. Sometimes the app on the other end cannot play it: a 4K HEVC film on an old TV, a high-bitrate file on a phone on mobile data, subtitles the app cannot draw itself. Then the server converts the video while you watch. That is transcoding, and on the CPU it is slow, hot and often not fast enough for real time. An Intel integrated GPU has dedicated video engines for exactly this, and the Jellyfin documentation recommends QSV (Intel Quick Sync Video) on mainstream Intel GPUs. Getting it to work inside a Proxmox container is a short list of steps. Knowing that it works is a separate job, and the one most guides skip.

The trap: a dropdown that says QSV

Set Hardware acceleration to Intel QuickSync in the Jellyfin dashboard and the page looks done. It is not. If the container cannot open the render device, or the group number is wrong, or the source codec is not ticked for hardware decoding, Jellyfin quietly falls back to software. The film still plays. The CPU does the work. Nothing in the player tells you.

The fix is a ladder: six checks, each ruling out one failure, climbed in order. If a rung fails, fix it before climbing further.

Two things first. Intel processors whose model ends in "F" have no integrated GPU, and the Jellyfin docs say not to use them for this. And the old recipe of a privileged container with lxc.cgroup2.devices.allow lines is kept in the Jellyfin docs only for Proxmox VE 7 and older. On Proxmox VE 8 and 9 the container stays unprivileged.

Rung 1: the host has a render device

On the Proxmox host:

ls -l /dev/dri
getent group render

You should see renderD128 (and card0 or similar) owned by the render or video group. If /dev/dri is missing, the host kernel has no driver for the GPU, and nothing below will work until it does.

Rung 2: the device reaches the container with the right group

Proxmox VE 8 and newer can pass a device into a container through the web interface, and the Jellyfin documentation recommends exactly this route. The detail that matters is the group id, and it is the container's group id, not the host's. Inside the container:

# inside the media container (CT 110)
getent group render
# render:x:993:      <- your number may differ

On some systems the group is video or input instead of render; use whichever exists. Then on the host, in the container's Resources tab, choose Add > Device Passthrough, enter /dev/dri/renderD128 as the device path, and set the GID in the advanced options to the number you just found. The same thing from the command line, using the dev[n] option from the Proxmox documentation:

# on the Proxmox host
pct set 110 --dev0 /dev/dri/renderD128,gid=993
pct reboot 110

After the restart, ls -l /dev/dri inside the container should show renderD128 owned by the render group. The container is still unprivileged and the host keeps using the GPU. One catch from the Jellyfin docs: you must be logged in to Proxmox as root to add device passthrough; if the option is unavailable, that is why.

If Jellyfin runs in a VM instead, the whole GPU moves into it with PCI passthrough, and the host, and every other guest, lose it.

Rung 3: Docker hands the device to Jellyfin

The official Jellyfin image already contains Intel's user-mode media drivers and the OpenCL runtime, so nothing needs installing inside it. Docker only has to pass the device through and add the group that owns it, with the number from rung 2:

# /opt/stacks/jellyfin/compose.yaml (GPU part)
services:
  jellyfin:
    image: jellyfin/jellyfin:12.2
    user: 1000:1000
    group_add:
      - "993"          # gid of render inside the LXC
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128

If you have two GPUs, the second one is usually renderD129.

Rung 4: the drivers open the GPU from inside

Before touching any Jellyfin setting, confirm the container can actually use the device. Both commands come from the Jellyfin documentation:

docker exec -it jellyfin /usr/lib/jellyfin-ffmpeg/vainfo
docker exec -it jellyfin /usr/lib/jellyfin-ffmpeg/ffmpeg -v verbose \
  -init_hw_device vaapi=va -init_hw_device opencl@va

vainfo should name the Intel iHD driver and list the supported profiles and entry points. The second command should list an Intel OpenCL device. If you get failed to initialize display or a permission error, the group in group_add does not match the group that owns /dev/dri/renderD128 inside the container. Fix the number and recreate the container.

Rung 5: the dashboard settings match your GPU

Now the transcoding settings in the Jellyfin dashboard:

Hardware acceleration: Intel QuickSync (QSV), or VA-API if you chose that.
Device: leave /dev/dri/renderD128 unless you use a second GPU.
Hardware decoding: tick only the codecs your GPU supports, as the docs instruct. The vainfo output and the codec tables in the Jellyfin Intel guide tell you which. Older iGPUs cannot decode HEVC 10-bit or AV1 in hardware; AV1 decoding starts with Gen 12 (11th Gen Core) graphics.
Enable hardware encoding.
Turn on tone mapping, so HDR sources look right on SDR screens. Both tone-mapping methods need a GPU with HEVC 10-bit decoding.

Rung 6: watch the GPU do the work

A setting that says "QSV" proves nothing. Make Jellyfin transcode and watch the GPU.

Play a film in the web client and force a transcode by picking a lower quality in the player settings.
In the dashboard, the active session should show that it is transcoding, not direct playing.
On the machine that owns the GPU (the Proxmox host for a container, the VM for a VM), install and run intel_gpu_top:

apt update && apt install -y intel-gpu-tools
intel_gpu_top

The Jellyfin guide explains the engine rows: Video is the QSV decoder or encoder at work, VideoEnhance is QSV VPP processing, Render/3D covers OpenCL workloads such as tone mapping. During a hardware transcode the Video row should be busy. If every row sits at zero while the CPU climbs in top, the transcode is running in software: go back down the ladder.

When it still uses the CPU

Subtitles burned in. Image-based subtitles force a burn-in. It is still a video transcode, so make sure it is accelerated, and prefer text subtitles the client can render itself.
Codec not ticked. Jellyfin then decodes in software even when encoding on the GPU.
Wrong device after a reboot. With more than one GPU, render node numbers can change. Pin the right one in both the passthrough and the compose file.

Checklist

CPU model does not end in "F"; Quick Sync confirmed on Intel's spec page.
/dev/dri/renderD128 exists on the host.
Render group gid read inside the container; device passed with that gid; container still unprivileged.
Jellyfin compose file has the device under devices and the gid under group_add.
vainfo shows the iHD driver; the OpenCL check finds an Intel device.
QSV selected with only supported codecs ticked; hardware encoding and tone mapping on.
A forced transcode shows the Video engine busy in intel_gpu_top.
After a host kernel update, vainfo and a forced transcode checked again.

Once this works, most of your playback should not need it at all: direct play, where the server just sends the file, is the goal, and transcoding is the safety net underneath it.

Originally published on the Homelab the Right Way blog. This article is based on the ebook Stream Your Own Library; every book in the series has a free sample. Written with AI help and edited by me.