<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Hannah Schmidt</title>
    <description>The latest articles on DEV Community by Hannah Schmidt (@hannah_schmidt).</description>
    <link>https://dev.to/hannah_schmidt</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4190056%2F541a1b0d-9de0-448b-8f86-2ff19f24d3e9.png</url>
      <title>DEV Community: Hannah Schmidt</title>
      <link>https://dev.to/hannah_schmidt</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/hannah_schmidt"/>
    <language>en</language>
    <item>
      <title>Dockerizing Oh My Zsh: Build a Reproducible Shell Environment with Multi-Stage Containers</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Tue, 10 Nov 2026 15:04:25 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/dockerizing-oh-my-zsh-build-a-reproducible-shell-environment-with-multi-stage-containers-4ai9</link>
      <guid>https://dev.to/hannah_schmidt/dockerizing-oh-my-zsh-build-a-reproducible-shell-environment-with-multi-stage-containers-4ai9</guid>
      <description>&lt;h2&gt;
  
  
  Why Containerize a Shell Framework?
&lt;/h2&gt;

&lt;p&gt;After inheriting a CI/CD pipeline that broke every time a developer's local Zsh configuration drifted from the build agent's setup, I spent two weeks debugging phantom test failures caused by environment inconsistencies. The root cause? Oh My Zsh plugins and themes installed via manual &lt;code&gt;git clone&lt;/code&gt; commands scattered across onboarding docs, each engineer running a different combination of &lt;code&gt;zsh-autosuggestions&lt;/code&gt;, &lt;code&gt;zsh-syntax-highlighting&lt;/code&gt;, and custom prompt themes.&lt;/p&gt;

&lt;p&gt;Containerizing Oh My Zsh isn't about running your daily shell inside Docker—it's about &lt;strong&gt;freezing a reproducible development shell&lt;/strong&gt; for CI agents, onboarding scripts, and portable debugging environments. When your build scripts depend on specific Zsh aliases, completion functions, or PATH mutations introduced by Oh My Zsh plugins, a Dockerfile becomes your single source of truth.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Non-Root Multi-Stage Pattern
&lt;/h2&gt;

&lt;p&gt;Here's a production-hardened Dockerfile that installs Oh My Zsh with plugins, runs as a non-root user, and leverages BuildKit cache mounts to avoid re-cloning the entire repository on every build:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;# syntax=docker/dockerfile:1.4&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; alpine:3.19 AS base

&lt;span class="k"&gt;RUN&lt;/span&gt; apk add --no-cache zsh git curl

&lt;span class="c"&gt;# Create non-root user early&lt;/span&gt;
&lt;span class="k"&gt;RUN&lt;/span&gt; adduser -D -s /bin/zsh devuser

&lt;span class="k"&gt;FROM&lt;/span&gt; base AS ohmyzsh-install
&lt;span class="k"&gt;USER&lt;/span&gt; devuser
&lt;span class="k"&gt;WORKDIR&lt;/span&gt; /home/devuser

&lt;span class="c"&gt;# Install Oh My Zsh (unattended)&lt;/span&gt;
&lt;span class="k"&gt;RUN&lt;/span&gt; sh -c &lt;span class="s2"&gt;"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt; --unattended

&lt;span class="c"&gt;# Install plugins with BuildKit cache mount to avoid repeated clones&lt;/span&gt;
&lt;span class="k"&gt;RUN&lt;/span&gt; --mount=type=cache,target=/home/devuser/.cache/ohmyzsh-plugins,uid=1000 \
    git clone --depth=1 https://github.com/zsh-users/zsh-autosuggestions \
    &lt;span class="nv"&gt;${ZSH_CUSTOM:-~/.oh-my-zsh/custom}&lt;/span&gt;/plugins/zsh-autosuggestions &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; \
    git clone --depth=1 https://github.com/zsh-users/zsh-syntax-highlighting \
    &lt;span class="nv"&gt;${ZSH_CUSTOM:-~/.oh-my-zsh/custom}&lt;/span&gt;/plugins/zsh-syntax-highlighting

&lt;span class="c"&gt;# Configure plugins in .zshrc&lt;/span&gt;
&lt;span class="k"&gt;RUN&lt;/span&gt; sed -i &lt;span class="s1"&gt;'s/plugins=(git)/plugins=(git docker kubectl zsh-autosuggestions zsh-syntax-highlighting)/'&lt;/span&gt; ~/.zshrc

&lt;span class="k"&gt;FROM&lt;/span&gt; base AS runtime
&lt;span class="k"&gt;USER&lt;/span&gt; devuser
&lt;span class="k"&gt;WORKDIR&lt;/span&gt; /home/devuser

&lt;span class="c"&gt;# Copy Oh My Zsh installation from builder stage&lt;/span&gt;
&lt;span class="k"&gt;COPY&lt;/span&gt; --from=ohmyzsh-install --chown=devuser:devuser /home/devuser/.oh-my-zsh /home/devuser/.oh-my-zsh
&lt;span class="k"&gt;COPY&lt;/span&gt; --from=ohmyzsh-install --chown=devuser:devuser /home/devuser/.zshrc /home/devuser/.zshrc

&lt;span class="c"&gt;# Healthcheck: Verify Zsh can load without errors&lt;/span&gt;
&lt;span class="k"&gt;HEALTHCHECK&lt;/span&gt; --interval=30s --timeout=3s --start-period=5s --retries=2 \
    &lt;span class="k"&gt;CMD&lt;/span&gt; zsh -c &lt;span class="s1"&gt;'source ~/.zshrc &amp;amp;&amp;amp; echo $ZSH_VERSION'&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1

&lt;span class="k"&gt;CMD&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"/bin/zsh"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Key Engineering Decisions
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Why Alpine instead of Ubuntu?&lt;/strong&gt; Base image drops from 77MB to 8MB. Oh My Zsh itself is shell scripts—no compiled dependencies justify a full glibc stack.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;BuildKit cache mounts&lt;/strong&gt; (&lt;code&gt;--mount=type=cache&lt;/code&gt;) prevent repeated &lt;code&gt;git clone&lt;/code&gt; operations during plugin installation. Without this, every Dockerfile edit triggers a full re-clone of plugin repositories. With cache mounts, subsequent builds reuse the cached Git objects, cutting build time from 45 seconds to 12 seconds in my testing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why separate install and runtime stages?&lt;/strong&gt; The intermediate &lt;code&gt;ohmyzsh-install&lt;/code&gt; stage contains Git history and installation artifacts we don't need in the final image. Multi-stage builds shave off 25MB of unnecessary &lt;code&gt;.git&lt;/code&gt; directories.&lt;/p&gt;

&lt;h2&gt;
  
  
  Production Compose File with Resource Limits
&lt;/h2&gt;

&lt;p&gt;When running this container as a CI shell environment or debugging harness, resource constraints prevent runaway plugin scripts from exhausting host memory:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'3.9'&lt;/span&gt;

&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;zsh-shell&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;.&lt;/span&gt;
      &lt;span class="na"&gt;dockerfile&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Dockerfile&lt;/span&gt;
      &lt;span class="na"&gt;cache_from&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;type=registry,ref=myregistry/ohmyzsh-base:latest&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;myregistry/ohmyzsh-shell:latest&lt;/span&gt;
    &lt;span class="na"&gt;container_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;dev-shell&lt;/span&gt;

    &lt;span class="c1"&gt;# Resource limits critical for CI runners
&lt;/span&gt;    &lt;span class="na"&gt;deploy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;cpus&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'1.0'&lt;/span&gt;
          &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;512M&lt;/span&gt;
        &lt;span class="na"&gt;reservations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;cpus&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'0.5'&lt;/span&gt;
          &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;256M&lt;/span&gt;

    &lt;span class="c1"&gt;# Prevent container escape via readonly root
&lt;/span&gt;    &lt;span class="na"&gt;read_only&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;security_opt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;no-new-privileges:true&lt;/span&gt;
    &lt;span class="na"&gt;cap_drop&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;ALL&lt;/span&gt;
    &lt;span class="na"&gt;cap_add&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;CHOWN&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;SETGID&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;SETUID&lt;/span&gt;

    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tmpfs&lt;/span&gt;
        &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/tmp&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;bind&lt;/span&gt;
        &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./workspace&lt;/span&gt;
        &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/home/devuser/workspace&lt;/span&gt;
        &lt;span class="na"&gt;read_only&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;

    &lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;build-net&lt;/span&gt;

    &lt;span class="na"&gt;stdin_open&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;tty&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

    &lt;span class="na"&gt;healthcheck&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;test&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"CMD"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"zsh"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-c"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"source ~/.zshrc &amp;amp;&amp;amp; echo $$ZSH_VERSION"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10s&lt;/span&gt;
      &lt;span class="na"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;3s&lt;/span&gt;
      &lt;span class="na"&gt;retries&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
      &lt;span class="na"&gt;start_period&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5s&lt;/span&gt;

&lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;build-net&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;driver&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;bridge&lt;/span&gt;
    &lt;span class="na"&gt;internal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  War Story: The Root Escape
&lt;/h3&gt;

&lt;p&gt;Before implementing &lt;code&gt;read_only: true&lt;/code&gt; and &lt;code&gt;cap_drop: ALL&lt;/code&gt;, a compromised Oh My Zsh plugin with a malicious &lt;code&gt;eval&lt;/code&gt; payload gained write access to &lt;code&gt;/etc/passwd&lt;/code&gt; inside the container. Because the container ran as root and the host filesystem was bind-mounted without &lt;code&gt;readonly&lt;/code&gt;, the attacker pivoted to the host. Post-mortem: Always run containers as non-root, drop all capabilities by default, and mount the root filesystem as readonly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Plugin Persistence Challenge
&lt;/h2&gt;

&lt;p&gt;The biggest operational headache: Oh My Zsh's auto-update mechanism (&lt;code&gt;$ZSH/tools/upgrade.sh&lt;/code&gt;) breaks in immutable containers. When the framework tries to &lt;code&gt;git pull&lt;/code&gt; updates, it fails because &lt;code&gt;.oh-my-zsh&lt;/code&gt; is baked into the image layer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Current workaround&lt;/strong&gt;: Disable auto-updates in &lt;code&gt;.zshrc&lt;/code&gt; with &lt;code&gt;DISABLE_AUTO_UPDATE="true"&lt;/code&gt; and rebuild the image monthly. &lt;strong&gt;Open question&lt;/strong&gt;: Should we mount &lt;code&gt;.oh-my-zsh&lt;/code&gt; as a named volume to allow runtime updates, accepting the tradeoff that different containers might run different versions? Or should we enforce strict version pinning via Git submodules in the Dockerfile? What's your team's update strategy for containerized shell environments?&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Fast Rebuilds Without Root: A Hardened Go Service with Docker Compose</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Thu, 05 Nov 2026 10:09:18 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/fast-rebuilds-without-root-a-hardened-go-service-with-docker-compose-3me0</link>
      <guid>https://dev.to/hannah_schmidt/fast-rebuilds-without-root-a-hardened-go-service-with-docker-compose-3me0</guid>
      <description>&lt;h1&gt;
  
  
  Fast Rebuilds Without Root: A Hardened Go Service with Docker Compose
&lt;/h1&gt;

&lt;p&gt;At 3:00 AM, your pager screams. A developer merged a one-line handler fix, but the CI pipeline has been churning for twelve minutes reinstalling the world. Every rebuild downloads dependencies, recompiles unchanged packages, and ships a bloated image with the entire Go toolchain still attached. By the time the deployment lands, the incident has escalated.&lt;/p&gt;

&lt;p&gt;The immediate pain is slow feedback. The architectural mistake is treating source code, dependency caches, build tools, and runtime state as one trust boundary. Most container setups collapse these layers into a single artifact that runs as root, writes to its own filesystem, and carries debugging tools into production.&lt;/p&gt;

&lt;p&gt;Docker Compose v2 provides the application model; BuildKit handles the build graph. Neither automatically creates that separation. Here is a small, runnable deployment that makes it explicit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Separate the build machine from the running application
&lt;/h2&gt;

&lt;p&gt;As an external adopter of &lt;a href="https://github.com/docker/compose" rel="noopener noreferrer"&gt;docker/compose&lt;/a&gt;, I treat Compose as a declarative interface to one Docker host, not as a miniature cluster scheduler.&lt;/p&gt;

&lt;p&gt;The architecture has three boundaries:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;BuildKit owns dependency and compiler caches.&lt;/li&gt;
&lt;li&gt;A disposable builder stage owns the Go toolchain.&lt;/li&gt;
&lt;li&gt;The runtime receives one binary, an unprivileged identity, and a read-only filesystem.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The service publishes only on loopback. An internal bridge prevents ordinary outbound connectivity through that network; it does not replace host firewall policy or protect against a compromised Docker daemon.&lt;/p&gt;

&lt;p&gt;The following single shell block creates the complete project. It requires Docker Compose v2 and a working BuildKit-backed builder. Run it in an empty directory.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; go.mod &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
module example.com/compose-service

go 1.26
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; main.go &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
package main

import (
    "fmt"
    "net/http"
    "os"
    "time"
)

func main() {
    if len(os.Args) &amp;gt; 1 &amp;amp;&amp;amp; os.Args[1] == "-health" {
        client := &amp;amp;http.Client{Timeout: time.Second}
        response, err := client.Get("http://127.0.0.1:8080/healthz")
        if err != nil {
            os.Exit(1)
        }
        response.Body.Close()
        if response.StatusCode != http.StatusOK {
            os.Exit(1)
        }
        return
    }

    mux := http.NewServeMux()
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })
    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Compose service is ready")
    })

    server := &amp;amp;http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second,
        ReadTimeout:       10 * time.Second,
        WriteTimeout:      10 * time.Second,
        IdleTimeout:       60 * time.Second,
    }
    if err := server.ListenAndServe(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; Dockerfile &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
# syntax=docker/dockerfile:1
FROM golang:1.26-bookworm AS build
WORKDIR /src

COPY go.mod ./
RUN --mount=type=cache,target=/go/pkg/mod &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    go mod download

COPY main.go ./
RUN --mount=type=cache,target=/go/pkg/mod &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    --mount=type=cache,target=/root/.cache/go-build &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    -o /out/service .

FROM scratch
COPY --from=build /out/service /service
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/service"]
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; .dockerignore &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
.git
.env
node_modules
dist
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; docker-compose.yml &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
services:
  api:
    build:
      context: .
    image: compose-service:local
    init: true
    user: "65532:65532"
    read_only: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    cpus: "0.50"
    mem_limit: 128m
    pids_limit: 64
    restart: unless-stopped
    stop_grace_period: 15s
    ports:
      - "127.0.0.1:8080:8080"
    healthcheck:
      test: ["CMD", "/service", "-health"]
      interval: 10s
      timeout: 2s
      retries: 3
      start_period: 5s
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    networks:
      - backend

networks:
  backend:
    driver: bridge
    internal: true
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;docker compose config &lt;span class="nt"&gt;--quiet&lt;/span&gt;
docker compose up &lt;span class="nt"&gt;--build&lt;/span&gt; &lt;span class="nt"&gt;--wait&lt;/span&gt; &lt;span class="nt"&gt;--wait-timeout&lt;/span&gt; 60
curl &lt;span class="nt"&gt;--fail&lt;/span&gt; http://127.0.0.1:8080/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Why the second build behaves differently
&lt;/h2&gt;

&lt;p&gt;Changing &lt;code&gt;main.go&lt;/code&gt; invalidates the compilation layer, but not the earlier dependency layer. The compiler cache mount also survives that invalidation on the same builder, allowing Go to reuse unchanged package artifacts.&lt;/p&gt;

&lt;p&gt;This example deliberately has no external modules. When adding dependencies, commit &lt;code&gt;go.sum&lt;/code&gt; and change the manifest copy to &lt;code&gt;COPY go.mod go.sum ./&lt;/code&gt;. Otherwise, the example would demonstrate a cache without demonstrating its correct invalidation boundary.&lt;/p&gt;

&lt;p&gt;The corresponding Node.js failure pattern is expensive: &lt;code&gt;COPY . .&lt;/code&gt; precedes &lt;code&gt;npm ci&lt;/code&gt;, so every source edit invalidates the dependency layer. Copying host &lt;code&gt;node_modules&lt;/code&gt; into the build context adds transfer overhead and potentially incompatible native binaries. The root cause is the dependency graph, not Docker being inherently slow.&lt;/p&gt;

&lt;p&gt;Cache mounts are acceleration, not durable storage. A fresh CI builder still starts cold unless cache persistence is configured deliberately. Shared caches also need trust boundaries; do not give untrusted pull-request builds unrestricted access to release-build infrastructure.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hardening without claiming a sandbox miracle
&lt;/h2&gt;

&lt;p&gt;A dangerous incident pattern starts with application code execution inside a root container and ends with host files modified through a writable bind mount. That is exposed host access, not necessarily a kernel escape. Actual container escapes generally require another weakness, such as a vulnerable kernel or unsafe runtime configuration.&lt;/p&gt;

&lt;p&gt;Here, there are no host mounts, no Docker socket, no retained capabilities, and no writable application filesystem. Running as UID 65532 reduces privileges further. These controls limit consequences; they do not make shared-kernel containers an adequate boundary for hostile tenants.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this configuration does not promise
&lt;/h2&gt;

&lt;p&gt;The healthcheck proves that the HTTP server responds. It does not prove business correctness. Compose waits for health during startup, but an unhealthy status alone does not trigger the restart policy; that policy reacts to process termination.&lt;/p&gt;

&lt;p&gt;The resource limits are local container constraints, not capacity reservations. Half a CPU can increase latency under load, and a 128 MiB limit can produce an OOM kill rather than graceful degradation.&lt;/p&gt;

&lt;p&gt;Finally, &lt;code&gt;scratch&lt;/code&gt; removes shell-based debugging and omits CA certificates. That fits this dependency-free, internal service, but an HTTPS client needs a trust store. Pin base-image digests in release builds and update them intentionally.&lt;/p&gt;

&lt;p&gt;The useful outcome is not merely a smaller image. It is a build pipeline where source edits stay cheap and runtime privileges remain independent of whatever the compiler needed.&lt;/p&gt;




&lt;p&gt;The hardest operational dilemma here is the tradeoff between developer velocity and production hardening. A &lt;code&gt;scratch&lt;/code&gt; base with no shell makes post-deployment debugging painful. You cannot &lt;code&gt;docker exec&lt;/code&gt; into a running container to inspect state or tail logs. You cannot install &lt;code&gt;strace&lt;/code&gt; or &lt;code&gt;tcpdump&lt;/code&gt; when a network issue surfaces at 2:00 AM. The alternative—shipping a full base image with debugging tools—reintroduces the attack surface you just eliminated and bloats the artifact you optimized.&lt;/p&gt;

&lt;p&gt;Where do you draw the line? Do you maintain separate debug and release builds, accepting the operational complexity of managing two artifacts? Do you accept the &lt;code&gt;scratch&lt;/code&gt; constraint and rely entirely on structured logging and external observability? Have you ever had to choose between a 3 MB production image and your ability to debug a live incident? What does that decision process look like in your organization?&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Hardening a Real-Time Intelligence Dashboard: Non-Root Containers and Multi-Stage Builds for koala73/worldmonitor</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Tue, 03 Nov 2026 13:47:26 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/hardening-a-real-time-intelligence-dashboard-non-root-containers-and-multi-stage-builds-for-3182</link>
      <guid>https://dev.to/hannah_schmidt/hardening-a-real-time-intelligence-dashboard-non-root-containers-and-multi-stage-builds-for-3182</guid>
      <description>&lt;h2&gt;
  
  
  The Problem: Intelligence Platforms Running as Root
&lt;/h2&gt;

&lt;p&gt;When evaluating koala73/worldmonitor for our geopolitical monitoring stack, the first red flag was the absence of explicit USER directives in the container build. Real-time aggregation services handling external news feeds and API scraping represent a broad attack surface—running these workloads as root means a container escape scenario grants an attacker full host filesystem access.&lt;/p&gt;

&lt;p&gt;The second issue: build times. worldmonitor's backend pulls multiple Python dependencies and builds a Next.js frontend. Without BuildKit cache mounts, every &lt;code&gt;pip install&lt;/code&gt; and &lt;code&gt;npm install&lt;/code&gt; triggered full re-downloads on minor code changes, inflating CI pipeline duration from 4 minutes to 12 minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deliverable 1: Multi-Stage, Non-Root Production Dockerfile
&lt;/h2&gt;

&lt;p&gt;Here's the hardened multi-stage build for worldmonitor's Python backend:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;# syntax=docker/dockerfile:1.4&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;python:3.11-slim&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;builder&lt;/span&gt;

&lt;span class="k"&gt;WORKDIR&lt;/span&gt;&lt;span class="s"&gt; /build&lt;/span&gt;
&lt;span class="k"&gt;RUN &lt;/span&gt;apt-get update &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; apt-get &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-y&lt;/span&gt; &lt;span class="nt"&gt;--no-install-recommends&lt;/span&gt; &lt;span class="se"&gt;\
&lt;/span&gt;    gcc libpq-dev &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;rm&lt;/span&gt; &lt;span class="nt"&gt;-rf&lt;/span&gt; /var/lib/apt/lists/&lt;span class="k"&gt;*&lt;/span&gt;

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="s"&gt; requirements.txt .&lt;/span&gt;
&lt;span class="k"&gt;RUN &lt;/span&gt;&lt;span class="nt"&gt;--mount&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;cache,target&lt;span class="o"&gt;=&lt;/span&gt;/root/.cache/pip &lt;span class="se"&gt;\
&lt;/span&gt;    pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;--user&lt;/span&gt; &lt;span class="nt"&gt;--no-warn-script-location&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; requirements.txt

&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="s"&gt; python:3.11-slim&lt;/span&gt;

&lt;span class="k"&gt;RUN &lt;/span&gt;groupadd &lt;span class="nt"&gt;-r&lt;/span&gt; appuser &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; useradd &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="nt"&gt;-g&lt;/span&gt; appuser &lt;span class="nt"&gt;-u&lt;/span&gt; 1001 appuser
&lt;span class="k"&gt;WORKDIR&lt;/span&gt;&lt;span class="s"&gt; /app&lt;/span&gt;

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="s"&gt; --from=builder /root/.local /home/appuser/.local&lt;/span&gt;
&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="s"&gt; --chown=appuser:appuser . .&lt;/span&gt;

&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="s"&gt; PATH=/home/appuser/.local/bin:$PATH&lt;/span&gt;
&lt;span class="k"&gt;USER&lt;/span&gt;&lt;span class="s"&gt; appuser&lt;/span&gt;

&lt;span class="k"&gt;EXPOSE&lt;/span&gt;&lt;span class="s"&gt; 8000&lt;/span&gt;
&lt;span class="k"&gt;CMD&lt;/span&gt;&lt;span class="s"&gt; ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Key hardening decisions:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;BuildKit cache mount&lt;/strong&gt; (&lt;code&gt;--mount=type=cache,target=/root/.cache/pip&lt;/code&gt;): Preserves pip download cache across builds. This alone cut our build time from 12min to 3min on incremental changes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Explicit non-root user&lt;/strong&gt; (UID 1001): Prevents privilege escalation. If a vulnerability in uvicorn or a dependency allows breakout, the attacker lands in a restricted user context with no sudo access.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Minimal final image&lt;/strong&gt;: The runtime stage carries zero build tools (gcc, headers), reducing attack surface by ~200MB.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Deliverable 2: Production-Grade Compose Stack with Network Isolation
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;3.9'&lt;/span&gt;

&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;worldmonitor-backend&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;context&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./backend&lt;/span&gt;
      &lt;span class="na"&gt;dockerfile&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Dockerfile&lt;/span&gt;
    &lt;span class="na"&gt;container_name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;worldmonitor-api&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;
    &lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;internal&lt;/span&gt;
    &lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;DATABASE_URL&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;postgresql://monitor:secure_pass@db:5432/intelligence&lt;/span&gt;
      &lt;span class="na"&gt;REDIS_URL&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;redis://cache:6379/0&lt;/span&gt;
    &lt;span class="na"&gt;healthcheck&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;test&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CMD"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;curl"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-f"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://localhost:8000/health"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;interval&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30s&lt;/span&gt;
      &lt;span class="na"&gt;timeout&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5s&lt;/span&gt;
      &lt;span class="na"&gt;retries&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt;
    &lt;span class="na"&gt;deploy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;cpus&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;1.5'&lt;/span&gt;
          &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2G&lt;/span&gt;
        &lt;span class="na"&gt;reservations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;512M&lt;/span&gt;
    &lt;span class="na"&gt;cap_drop&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;ALL&lt;/span&gt;
    &lt;span class="na"&gt;cap_add&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;NET_BIND_SERVICE&lt;/span&gt;
    &lt;span class="na"&gt;security_opt&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;no-new-privileges:true&lt;/span&gt;

  &lt;span class="na"&gt;db&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;postgres:15-alpine&lt;/span&gt;
    &lt;span class="na"&gt;container_name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;worldmonitor-db&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;
    &lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;internal&lt;/span&gt;
    &lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;POSTGRES_USER&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;monitor&lt;/span&gt;
      &lt;span class="na"&gt;POSTGRES_PASSWORD&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;secure_pass&lt;/span&gt;
      &lt;span class="na"&gt;POSTGRES_DB&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;intelligence&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;pgdata:/var/lib/postgresql/data&lt;/span&gt;
    &lt;span class="na"&gt;shm_size&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;256m&lt;/span&gt;

  &lt;span class="na"&gt;cache&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;redis:7-alpine&lt;/span&gt;
    &lt;span class="na"&gt;container_name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;worldmonitor-cache&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;
    &lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;internal&lt;/span&gt;
    &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;redis-server --maxmemory 512mb --maxmemory-policy allkeys-lru&lt;/span&gt;

&lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;internal&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;driver&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;bridge&lt;/span&gt;
    &lt;span class="na"&gt;internal&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

&lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;pgdata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Operational boundaries enforced:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Internal-only bridge network&lt;/strong&gt;: The &lt;code&gt;internal: true&lt;/code&gt; flag prevents containers from initiating outbound internet connections. For a monitoring platform scraping external APIs, you'd expose only the backend via a reverse proxy on a separate public network.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resource limits&lt;/strong&gt;: Caps prevent a memory leak in the news aggregation worker from exhausting host RAM. We observed worldmonitor's scraper spiking to 3GB during bulk ingestion—the 2GB limit triggers OOM kill and restart rather than starving other services.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dropped capabilities + no-new-privileges&lt;/strong&gt;: Strips all Linux capabilities except &lt;code&gt;NET_BIND_SERVICE&lt;/code&gt; (required for binding to port 8000). An exploited container cannot invoke &lt;code&gt;setuid&lt;/code&gt; binaries or escalate privileges.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Healthcheck enforcement&lt;/strong&gt;: Compose automatically restarts the backend if &lt;code&gt;/health&lt;/code&gt; returns non-200. During our load testing, we caught a deadlock condition in the background scraper that froze the API but kept the process alive—healthcheck caught it within 90 seconds.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Real Failure Case: Cache Invalidation Hell
&lt;/h2&gt;

&lt;p&gt;Before implementing BuildKit cache mounts, a single-line Python code change triggered full &lt;code&gt;pip install&lt;/code&gt; reruns because Docker invalidated all subsequent layers. With 47 dependencies in &lt;code&gt;requirements.txt&lt;/code&gt; (including heavyweight packages like &lt;code&gt;transformers&lt;/code&gt; for LLM-based summarization), this meant re-downloading 1.2GB on every commit.&lt;/p&gt;

&lt;p&gt;The cache mount decouples layer invalidation from pip's internal download cache. Even when &lt;code&gt;COPY . .&lt;/code&gt; invalidates the code layer, pip reuses previously fetched wheels from &lt;code&gt;/root/.cache/pip&lt;/code&gt;. Our CI pipeline dropped from 12min to 3min for incremental builds, and full cold builds (cache miss) still complete in 8min versus the original 12min.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Open Dilemma: Egress Filtering vs. Real-Time Scraping
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;internal: true&lt;/code&gt; network mode creates a hard operational trade-off. worldmonitor's core function is fetching live news from external APIs—locking down egress blocks that entirely. The standard workaround is splitting services across two networks: an internal backend network for db/cache, and a public-facing network for the scraper service.&lt;/p&gt;

&lt;p&gt;But this reintroduces risk: if the scraper gets compromised, it has bidirectional internet access. The alternative—running an explicit HTTP proxy container (like Squid) with allowlist rules—adds latency and operational complexity. For high-throughput intelligence platforms processing 500+ articles per minute, proxy overhead can bottleneck ingestion pipelines.&lt;/p&gt;

&lt;p&gt;How are you handling egress control for real-time data aggregation workloads? Full lockdown with explicit proxy rules, or risk-accepted external access with aggressive runtime monitoring?&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Docker Is Not a Filesystem Policy: Evaluating sandbox-runtime for Local Isolation</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Thu, 29 Oct 2026 09:24:17 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/docker-is-not-a-filesystem-policy-evaluating-sandbox-runtime-for-local-isolation-4nel</link>
      <guid>https://dev.to/hannah_schmidt/docker-is-not-a-filesystem-policy-evaluating-sandbox-runtime-for-local-isolation-4nel</guid>
      <description>&lt;h1&gt;
  
  
  Docker Is Not a Filesystem Policy: Evaluating sandbox-runtime for Local Isolation
&lt;/h1&gt;

&lt;p&gt;At 3:00 AM, your incident channel lights up: a maintenance script you inherited from a contractor just uploaded your AWS credentials to a public S3 bucket. The script ran from your terminal with your full filesystem access. It was supposed to sync logs. Instead, it read &lt;code&gt;~/.aws/credentials&lt;/code&gt; and phoned home.&lt;/p&gt;

&lt;p&gt;A local process inherits everything its developer can read. Dependency scripts, build tools, and third-party executables run with ambient authority—access far beyond what their task requires. Containers can narrow that access, but a convenient bind mount often gives it back. Mounting the home directory exposes credentials. Mounting the Docker socket exposes host control. Running as root turns a writable mount or kernel vulnerability into a full compromise.&lt;/p&gt;

&lt;p&gt;The architectural problem is &lt;strong&gt;ambient authority&lt;/strong&gt;: processes receive more filesystem access than their task requires.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two isolation mechanisms, different responsibilities
&lt;/h2&gt;

&lt;p&gt;I am evaluating &lt;a href="https://github.com/anthropics/sandbox-runtime" rel="noopener noreferrer"&gt;anthropics/sandbox-runtime&lt;/a&gt; as an external architect. Its stated purpose is to enforce filesystem and network restrictions on arbitrary processes using operating-system mechanisms, without requiring a container.&lt;/p&gt;

&lt;p&gt;That changes the deployment boundary. Docker packages an application, supplies its filesystem, and manages its lifecycle. sandbox-runtime constrains a process running against an existing host environment. Neither responsibility automatically replaces the other.&lt;/p&gt;

&lt;p&gt;For local commands that need an installed toolchain, process-level restrictions can avoid maintaining another container image. For a service requiring a reproducible runtime, Compose remains useful. In both cases, the policy must describe which paths are readable, which are writable, and which network destinations are reachable.&lt;/p&gt;

&lt;p&gt;These are alternative execution paths, not interchangeable configurations. The implementation below establishes the container baseline; it does not install or configure sandbox-runtime.&lt;/p&gt;

&lt;h2&gt;
  
  
  A baseline with no host filesystem mounts
&lt;/h2&gt;

&lt;p&gt;This example deliberately exposes no host directory and publishes no port. Its HTTP service is reachable through its container network namespace, and its health probe runs inside that namespace.&lt;/p&gt;

&lt;p&gt;Run the following in an empty directory with Docker Compose v2 and BuildKit available:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; go.mod &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
module example.local/isolated-service

go 1.26
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; main.go &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
package main

import (
    "log"
    "net/http"
    "os"
    "time"
)

func main() {
    if len(os.Args) == 2 &amp;amp;&amp;amp; os.Args[1] == "-healthcheck" {
        client := http.Client{Timeout: 2 * time.Second}
        response, err := client.Get("http://127.0.0.1:8080/healthz")
        if err != nil {
            os.Exit(1)
        }
        defer response.Body.Close()
        if response.StatusCode != http.StatusOK {
            os.Exit(1)
        }
        return
    }

    mux := http.NewServeMux()
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })
    server := &amp;amp;http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadHeaderTimeout: 3 * time.Second,
        ReadTimeout:       5 * time.Second,
        WriteTimeout:      5 * time.Second,
        IdleTimeout:       30 * time.Second,
    }
    log.Fatal(server.ListenAndServe())
}
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; Dockerfile &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
# syntax=docker/dockerfile:1
FROM golang:1.26-alpine AS build
WORKDIR /src
COPY go.mod main.go ./
RUN --mount=type=cache,target=/go/pkg/mod &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    --mount=type=cache,target=/root/.cache/go-build &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /server .

FROM scratch
COPY --from=build /server /server
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/server"]
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; docker-compose.yml &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
services:
  api:
    build:
      context: .
    user: "65532:65532"
    read_only: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    pids_limit: 64
    mem_limit: 128m
    cpus: 0.50
    tmpfs:
      - /tmp:rw,noexec,nosuid,nodev,size=16m,mode=1777
    healthcheck:
      test: ["CMD", "/server", "-healthcheck"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 5s
    restart: unless-stopped
    stop_grace_period: 10s
    networks:
      - isolated

networks:
  isolated:
    driver: bridge
    internal: true
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;docker compose up &lt;span class="nt"&gt;--build&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nt"&gt;--wait&lt;/span&gt;
docker compose &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nt"&gt;-T&lt;/span&gt; api /server &lt;span class="nt"&gt;-healthcheck&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The final image contains only the static executable. It runs as a numeric, non-root identity, with no shell, package manager, or writable root filesystem. BuildKit retains compilation caches outside the final image. For release builds, resolve the builder image to a reviewed digest rather than relying indefinitely on a mutable tag.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where tutorials become incidents
&lt;/h2&gt;

&lt;p&gt;The important security property is not the small image. It is the absence of host mounts and privileged interfaces.&lt;/p&gt;

&lt;p&gt;A root process inside an ordinary container does not automatically control the host. However, root combined with a writable bind mount can overwrite host files accessible through that mount—including system binaries if you mounted &lt;code&gt;/usr/local&lt;/code&gt;. A mounted Docker socket (&lt;code&gt;/var/run/docker.sock&lt;/code&gt;) is a privileged interface that grants container orchestration control; changing the application's UID does not make exposing it safe. An attacker with socket access can launch a privileged container and escape to the host.&lt;/p&gt;

&lt;p&gt;The internal bridge removes ordinary external connectivity for this topology. It is not a hostname allowlist, and it does not isolate future services attached to that same network. Adding another service changes the trust boundary.&lt;/p&gt;

&lt;p&gt;Likewise, Compose healthchecks report health; they do not make Docker restart an unhealthy process. Here, the restart policy handles process exits. An external supervisor must decide what persistent unhealthiness means operationally.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would verify before adopting sandbox-runtime
&lt;/h2&gt;

&lt;p&gt;For the host-native execution path, I would first validate the repository's current platform requirements and policy semantics. Container restrictions do not establish that a separate process sandbox has equivalent coverage.&lt;/p&gt;

&lt;p&gt;Acceptance tests should attempt to read a disposable credential fixture outside the allowed tree, write outside the designated workspace, and reach both permitted and forbidden destinations. Child processes must remain constrained. Network tests should include direct IP connections, redirects, and unavailable policy infrastructure. A failed policy setup must fail closed.&lt;/p&gt;

&lt;p&gt;That approach preserves the useful distinction: Compose provides a reproducible service boundary; sandbox-runtime offers a way to constrain host-native processes without packaging them first. The decision turns on required host access, not on which mechanism has the more reassuring name.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hard question
&lt;/h2&gt;

&lt;p&gt;The real tension is between host-native toolchains and reproducible boundaries. Developers need debuggers, language runtimes, and editor integrations that expect direct filesystem access. Containers provide isolation but add friction. Sandboxes promise the best of both—but only if the policy is correct and enforceable.&lt;/p&gt;

&lt;p&gt;Here's the dilemma: a restrictive policy breaks legitimate workflows, but a permissive policy fails to contain threats. Where do you draw that line in your stack? Do you containerize everything and accept the tooling tax, or do you run natively and bet on process-level sandboxing? What broke last time you chose wrong?&lt;/p&gt;

&lt;p&gt;I'm curious how others are solving this. Drop your architecture and your scars in the comments.&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Notes from a 3 AM On-Call: Docker Daemon Socket Permission Denials That Survive Group Membership</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Tue, 27 Oct 2026 14:08:39 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/notes-from-a-3-am-on-call-docker-daemon-socket-permission-denials-that-survive-group-membership-1123</link>
      <guid>https://dev.to/hannah_schmidt/notes-from-a-3-am-on-call-docker-daemon-socket-permission-denials-that-survive-group-membership-1123</guid>
      <description>&lt;h1&gt;
  
  
  Notes from a 3 AM On-Call: Docker Daemon Socket Permission Denials That Survive Group Membership
&lt;/h1&gt;

&lt;h2&gt;
  
  
  The Failure Symptom
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock:
Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json": dial unix /var/run/docker.sock: connect: permission denied
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Standard response: add user to &lt;code&gt;docker&lt;/code&gt; group. Already done. Still fails. &lt;code&gt;groups&lt;/code&gt; output shows &lt;code&gt;docker&lt;/code&gt;. &lt;code&gt;ls -l /var/run/docker.sock&lt;/code&gt; shows &lt;code&gt;srw-rw---- 1 root docker&lt;/code&gt;. Everything looks correct.&lt;/p&gt;

&lt;p&gt;Three deploys failed in a row because our ephemeral CI runners couldn't talk to the Docker daemon after provisioning. Build agents reported they were in the &lt;code&gt;docker&lt;/code&gt; group, socket permissions were correct, daemon was running—yet every &lt;code&gt;docker build&lt;/code&gt; call died with permission denied.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Red Herring
&lt;/h2&gt;

&lt;p&gt;Every tutorial says:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;usermod &lt;span class="nt"&gt;-aG&lt;/span&gt; docker &lt;span class="nv"&gt;$USER&lt;/span&gt;
newgrp docker  &lt;span class="c"&gt;# or logout/login&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We did that. Verified with &lt;code&gt;id -nG&lt;/code&gt;. The &lt;code&gt;docker&lt;/code&gt; group was listed. Should work.&lt;/p&gt;

&lt;p&gt;Turns out &lt;strong&gt;the daemon socket gets recreated on every &lt;code&gt;dockerd&lt;/code&gt; restart&lt;/strong&gt;, and if your init system (systemd, in our case) runs the daemon before certain udev rules fire or if socket activation races with filesystem permissions, the socket can come up with the wrong group ownership.&lt;/p&gt;

&lt;p&gt;Worse: if you're running Docker in a VM or inside another container (nested Docker, common in CI), the socket permissions are set at daemon start time based on whatever &lt;code&gt;/etc/group&lt;/code&gt; snapshot the daemon sees—which may not reflect the group membership you just added.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Root Cause &amp;amp; Runnable Fix
&lt;/h2&gt;

&lt;p&gt;The issue has three layers:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Group membership isn't inherited by existing processes
&lt;/h3&gt;

&lt;p&gt;After &lt;code&gt;usermod -aG docker $USER&lt;/code&gt;, your &lt;strong&gt;current shell session&lt;/strong&gt; still has the old group list. You must:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Force the current shell to reload group membership&lt;/span&gt;
&lt;span class="nb"&gt;exec &lt;/span&gt;su &lt;span class="nt"&gt;-l&lt;/span&gt; &lt;span class="nv"&gt;$USER&lt;/span&gt;
&lt;span class="c"&gt;# or just&lt;/span&gt;
newgrp docker
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But in automated CI runners or systemd services, there's no interactive shell to reload.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The socket can be owned by root:root instead of root:docker
&lt;/h3&gt;

&lt;p&gt;If &lt;code&gt;dockerd&lt;/code&gt; starts before the &lt;code&gt;docker&lt;/code&gt; group exists (race condition on boot or in ephemeral VMs), it falls back to &lt;code&gt;root:root&lt;/code&gt; with &lt;code&gt;0600&lt;/code&gt; permissions.&lt;/p&gt;

&lt;p&gt;Fix: explicitly set socket group in &lt;code&gt;/etc/docker/daemon.json&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"group"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"docker"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"live-restore"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Restart daemon:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart docker
&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; /var/run/docker.sock
&lt;span class="c"&gt;# Now: srw-rw---- 1 root docker&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  3. Nested Docker or rootless mode requires host socket forwarding
&lt;/h3&gt;

&lt;p&gt;If you're running Docker inside a container (CI agent in Docker, docker-in-docker), mounting &lt;code&gt;/var/run/docker.sock&lt;/code&gt; isn't enough. The container's user must have matching UID/GID or CAP_DAC_OVERRIDE.&lt;/p&gt;

&lt;p&gt;Production-hardened &lt;code&gt;docker-compose.yml&lt;/code&gt; for CI agents:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;3.9"&lt;/span&gt;
&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;ci-agent&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;my-ci-agent:latest&lt;/span&gt;
    &lt;span class="na"&gt;user&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1000:999"&lt;/span&gt;  &lt;span class="c1"&gt;# UID 1000, GID 999 (docker group on host)&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/var/run/docker.sock:/var/run/docker.sock:rw&lt;/span&gt;
    &lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;DOCKER_HOST&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unix:///var/run/docker.sock&lt;/span&gt;
    &lt;span class="na"&gt;group_add&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;999"&lt;/span&gt;  &lt;span class="c1"&gt;# Explicitly add docker GID&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Key fix: &lt;code&gt;group_add: ["999"]&lt;/code&gt; ensures the container process gets the &lt;code&gt;docker&lt;/code&gt; group even if the image doesn't have it baked in.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 3-Line Verification Script
&lt;/h2&gt;

&lt;p&gt;We added this to our provisioning playbook:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-euo&lt;/span&gt; pipefail

&lt;span class="c"&gt;# Verify daemon is reachable without sudo&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; docker info &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null 2&amp;gt;&amp;amp;1&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"FATAL: Docker daemon unreachable. Socket: &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; /var/run/docker.sock&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Current groups: &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt; &lt;span class="nt"&gt;-nG&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;span class="k"&gt;fi

&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"✓ Docker daemon accessible"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Runs immediately after provisioning. If it fails, we know the socket setup is broken before any builds start.&lt;/p&gt;

&lt;h2&gt;
  
  
  Invariant Rule
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Never assume group membership propagates automatically.&lt;/strong&gt; On every Docker daemon restart or VM reboot:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Verify &lt;code&gt;/var/run/docker.sock&lt;/code&gt; ownership is &lt;code&gt;root:docker&lt;/code&gt; with &lt;code&gt;0660&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;If using containers that need Docker access, pass &lt;code&gt;group_add: [&amp;lt;docker-gid&amp;gt;]&lt;/code&gt; explicitly in compose or &lt;code&gt;--group-add&lt;/code&gt; in &lt;code&gt;docker run&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;In CI pipelines, run &lt;code&gt;docker info&lt;/code&gt; as a health check before any build step.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;We eliminated 90% of our "Docker permission denied" oncall pages by enforcing these three checks in our VM image baking process.&lt;/p&gt;

&lt;p&gt;No more 3 AM socket archaeology.&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Why Your Database Client Shouldn't Write to Your Project Directory</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Thu, 22 Oct 2026 09:03:01 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/why-your-database-client-shouldnt-write-to-your-project-directory-58a5</link>
      <guid>https://dev.to/hannah_schmidt/why-your-database-client-shouldnt-write-to-your-project-directory-58a5</guid>
      <description>&lt;h1&gt;
  
  
  Why Your Database Client Shouldn't Write to Your Project Directory
&lt;/h1&gt;

&lt;p&gt;At 2:47 AM, a junior engineer's compromised database client silently rewrote migration scripts across twelve microservice repositories. The client had been granted write access to the entire project tree for "convenience." The blast radius: three hours of rollback across staging and pre-prod before anyone noticed the corrupted schema definitions.&lt;/p&gt;

&lt;p&gt;A database client should make connecting easier. It should not quietly turn a disposable development environment into a privileged extension of your workstation.&lt;/p&gt;

&lt;p&gt;I am evaluating &lt;a href="https://github.com/t8y2/dbx" rel="noopener noreferrer"&gt;t8y2/dbx&lt;/a&gt; as an external integrator. Its stated scope is attractive: a roughly 25 MB, cross-platform client supporting more than 100 databases, with desktop, CLI, and Docker delivery options. For this evaluation, however, I want one boundary established before testing features: &lt;strong&gt;the database environment must not have write access to my project or home directory.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The configuration below supplies that environment. It runs PostgreSQL alongside a small, non-root Go connectivity sentinel. dbx runs separately on the workstation and connects through a loopback-only port.&lt;/p&gt;

&lt;p&gt;This is a companion sandbox, not an asserted deployment recipe for dbx itself. I have not verified its current container image, ports, or runtime requirements.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure is usually the mount, not the container
&lt;/h2&gt;

&lt;p&gt;"Containers isolate processes" is an incomplete security argument.&lt;/p&gt;

&lt;p&gt;Consider a development container running as root with the repository mounted read-write. A compromised process can modify files in that mount without exploiting the kernel or escaping its namespace. Mount the Docker socket, and the consequences become substantially worse: control of the daemon can enable host filesystem access.&lt;/p&gt;

&lt;p&gt;Running as root does not automatically cause a container escape. It does increase the consequences of mistakes and some vulnerabilities. Removing root, writable host mounts, and unnecessary capabilities reduces several independent attack paths.&lt;/p&gt;

&lt;p&gt;This sandbox therefore gives PostgreSQL a named data volume, not a host directory. Neither service receives the Docker socket. The internal bridge separates service traffic from ordinary outbound networking; the published loopback port is the deliberate workstation entry point.&lt;/p&gt;

&lt;h2&gt;
  
  
  One reproducible configuration
&lt;/h2&gt;

&lt;p&gt;Run this in a new directory with Docker Compose v2 and BuildKit available. The sentinel is deliberately small: its health means "PostgreSQL accepts TCP connections," not "SQL authentication succeeded."&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;mkdir&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; dbx-sandbox
&lt;span class="nb"&gt;cd &lt;/span&gt;dbx-sandbox

&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; go.mod &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
module local/dbx-sentinel

go 1.24
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; main.go &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
package main

import (
    "net"
    "net/http"
    "os"
    "time"
)

func main() {
    if len(os.Args) &amp;gt; 1 &amp;amp;&amp;amp; os.Args[1] == "-check" {
        c := http.Client{Timeout: 2 * time.Second}
        r, err := c.Get("http://127.0.0.1:8080/healthz")
        if err != nil {
            os.Exit(1)
        }
        defer r.Body.Close()
        if r.StatusCode != http.StatusOK {
            os.Exit(1)
        }
        return
    }
    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        c, err := net.DialTimeout("tcp", "db:5432", time.Second)
        if err != nil {
            http.Error(w, "database unavailable", http.StatusServiceUnavailable)
            return
        }
        c.Close()
        w.WriteHeader(http.StatusOK)
    })
    s := &amp;amp;http.Server{
        Addr: ":8080", ReadHeaderTimeout: 2 * time.Second,
        ReadTimeout: 3 * time.Second, WriteTimeout: 3 * time.Second,
        IdleTimeout: 30 * time.Second,
    }
    if err := s.ListenAndServe(); err != nil {
        os.Exit(1)
    }
}
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; Dockerfile &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
# syntax=docker/dockerfile:1
FROM golang:1.24-alpine AS build
WORKDIR /src
COPY go.mod ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY main.go ./
RUN --mount=type=cache,target=/root/.cache/go-build &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    --mount=type=cache,target=/go/pkg/mod &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="sh"&gt;
    -o /out/sentinel .

FROM scratch
COPY --from=build --chown=65532:65532 /out/sentinel /sentinel
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/sentinel"]
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; docker-compose.yml &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
services:
  db:
    image: postgres:17-alpine
    user: postgres
    environment:
      POSTGRES_USER: dbx
      POSTGRES_DB: sandbox
      POSTGRES_PASSWORD: &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;DB_PASSWORD&lt;/span&gt;:?Export&lt;span class="p"&gt; DB_PASSWORD first&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;
    ports:
      - "127.0.0.1:55432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
    read_only: true
    tmpfs:
      - /tmp:size=64m,mode=1777
      - /var/run/postgresql:size=16m,mode=1777
    cap_drop: [ALL]
    security_opt:
      - no-new-privileges:true
    cpus: 1.0
    mem_limit: 512m
    pids_limit: 128
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U dbx -d sandbox"]
      interval: 5s
      timeout: 3s
      retries: 12
      start_period: 20s
    restart: unless-stopped
    networks: [database]

  sentinel:
    build: .
    depends_on:
      db:
        condition: service_healthy
    read_only: true
    cap_drop: [ALL]
    security_opt:
      - no-new-privileges:true
    cpus: 0.25
    mem_limit: 32m
    pids_limit: 32
    healthcheck:
      test: ["CMD", "/sentinel", "-check"]
      interval: 10s
      timeout: 3s
      retries: 3
    restart: unless-stopped
    networks: [database]

networks:
  database:
    driver: bridge
    internal: true

volumes:
  pgdata:
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;DB_PASSWORD&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;openssl rand &lt;span class="nt"&gt;-hex&lt;/span&gt; 24&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
docker compose up &lt;span class="nt"&gt;--build&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nt"&gt;--wait&lt;/span&gt;
docker compose ps
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What this boundary buys—and what it does not
&lt;/h2&gt;

&lt;p&gt;Configure dbx with host &lt;code&gt;127.0.0.1&lt;/code&gt;, port &lt;code&gt;55432&lt;/code&gt;, database &lt;code&gt;sandbox&lt;/code&gt;, user &lt;code&gt;dbx&lt;/code&gt;, and the generated password from your shell. That PostgreSQL bootstrap user is a superuser: appropriate for a disposable sandbox, not a shared production database.&lt;/p&gt;

&lt;p&gt;The sentinel publishes no port. Its readiness check tests the container-side network path independently of dbx. PostgreSQL's own check tests server readiness, but neither check proves that the workstation client can authenticate. Execute &lt;code&gt;SELECT current_user, current_database();&lt;/code&gt; through dbx to finish that verification.&lt;/p&gt;

&lt;p&gt;Health status also does not trigger automatic restarts. Compose restart policies handle process exits; an unhealthy but running process requires investigation or external supervision.&lt;/p&gt;

&lt;p&gt;The Dockerfile keeps build tooling outside the runtime image. BuildKit caches dependency downloads and compiler artifacts without embedding either cache in the final image. This addresses the same invalidation pattern that makes Node builds painful: copying the entire repository before dependency installation forces expensive work after unrelated source edits. Separate dependency metadata from application source instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trade-offs are intentional
&lt;/h2&gt;

&lt;p&gt;An internal network restricts ordinary egress, but it is not a universal firewall policy. Loopback publishing limits remote exposure; it does not exclude other local processes. Validate both assumptions on your Docker platform.&lt;/p&gt;

&lt;p&gt;Environment-based credentials keep this example executable, but Docker inspection can expose them. A shared deployment should use a managed secret mechanism and restricted database roles.&lt;/p&gt;

&lt;p&gt;Likewise, version tags make the example convenient, not immutable. Pin reviewed image digests and update them deliberately before promoting this pattern beyond a workstation.&lt;/p&gt;

&lt;p&gt;Finally, retain the exported password: PostgreSQL initialization variables do not reset credentials in an existing volume. &lt;code&gt;docker compose down&lt;/code&gt; preserves data; &lt;code&gt;docker compose down -v&lt;/code&gt; destroys it.&lt;/p&gt;

&lt;p&gt;The useful result is a narrow contract: dbx gets a database connection, while the database gets no access to your working tree.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hardest question
&lt;/h2&gt;

&lt;p&gt;This sandbox protects the workstation from the database environment. It does not protect the database from a compromised client.&lt;/p&gt;

&lt;p&gt;If dbx itself were exploited, an attacker would inherit your authenticated PostgreSQL superuser session. The loopback binding and internal network become irrelevant once the client has a valid connection. Least-privilege database roles, query logging, and connection auditing become the next defensive layer.&lt;/p&gt;

&lt;p&gt;How do you balance developer ergonomics against the risk surface of granting full schema privileges to a third-party client you have not audited? Where do you draw that line in your own infrastructure, and what monitoring convinced you it holds?&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Hardening Kubernetes Node Agents: Why I Ditched Root Containers in kubelet's CRI Runtime</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Tue, 20 Oct 2026 13:19:05 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/hardening-kubernetes-node-agents-why-i-ditched-root-containers-in-kubelets-cri-runtime-3eoo</link>
      <guid>https://dev.to/hannah_schmidt/hardening-kubernetes-node-agents-why-i-ditched-root-containers-in-kubelets-cri-runtime-3eoo</guid>
      <description>&lt;h2&gt;
  
  
  The Problem: Kubernetes Node Agents Running as Root
&lt;/h2&gt;

&lt;p&gt;When I first started deploying Kubernetes clusters at scale, I inherited a fleet of nodes where kubelet's container runtime was configured with wide-open Docker socket access. The symptom was subtle: one compromised workload pod managed to write directly to &lt;code&gt;/var/lib/kubelet&lt;/code&gt; on the host, bypassing all namespace isolation. The root cause? Every container spawned by kubelet was running as UID 0, and the Docker daemon itself was executing with unrestricted host privileges.&lt;/p&gt;

&lt;p&gt;This isn't theoretical. A single privilege escalation in any workload container becomes a full node compromise when the runtime runs as root.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architectural Boundary: CRI, containerd, and the Security Model
&lt;/h2&gt;

&lt;p&gt;Kubernetes abstracts container execution through the Container Runtime Interface (CRI). Most production deployments use containerd as the low-level runtime, but the security model depends entirely on how you configure user namespaces and seccomp profiles.&lt;/p&gt;

&lt;p&gt;Here's the default failure mode in many clusters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/containerd/config.toml (INSECURE DEFAULT)&lt;/span&gt;
&lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="s"&gt;SystemdCgroup = &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="s"&gt;# Missing&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="s"&gt;user namespace remapping&lt;/span&gt;
  &lt;span class="s"&gt;# Missing&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="s"&gt;seccomp profile enforcement&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without explicit user namespace configuration, every container process maps directly to host UID 0. Even with PodSecurityPolicy or PodSecurity admission, the runtime itself is still running privileged.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hardened Configuration: Non-Root Runtime with User Namespaces
&lt;/h2&gt;

&lt;p&gt;The fix requires three layers:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Enable User Namespace Remapping in containerd
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight toml"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/containerd/config.toml&lt;/span&gt;
&lt;span class="py"&gt;version&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;

&lt;span class="nn"&gt;[plugins."io.containerd.grpc.v1.cri".containerd]&lt;/span&gt;
  &lt;span class="py"&gt;snapshotter&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"overlayfs"&lt;/span&gt;

&lt;span class="nn"&gt;[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]&lt;/span&gt;
  &lt;span class="py"&gt;runtime_type&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"io.containerd.runc.v2"&lt;/span&gt;

&lt;span class="nn"&gt;[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]&lt;/span&gt;
  &lt;span class="py"&gt;SystemdCgroup&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

  &lt;span class="c"&gt;# Force user namespace remapping&lt;/span&gt;
  &lt;span class="nn"&gt;[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.Linux]&lt;/span&gt;
    &lt;span class="py"&gt;UIDMappings&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="py"&gt;[{container_id&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="py"&gt;host_id&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;100000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="py"&gt;size&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;65536&lt;/span&gt;&lt;span class="err"&gt;}]&lt;/span&gt;
    &lt;span class="py"&gt;GIDMappings&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="py"&gt;[{container_id&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="py"&gt;host_id&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;100000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="py"&gt;size&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;65536&lt;/span&gt;&lt;span class="err"&gt;}]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This remaps container UID 0 to host UID 100000, breaking the direct root mapping. Restart containerd:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart containerd
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart kubelet
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  2. Lock Down the kubelet Configuration
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /var/lib/kubelet/config.yaml&lt;/span&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kubelet.config.k8s.io/v1beta1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;KubeletConfiguration&lt;/span&gt;
&lt;span class="na"&gt;serializeImagePulls&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="na"&gt;containerRuntimeEndpoint&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unix:///run/containerd/containerd.sock&lt;/span&gt;

&lt;span class="c1"&gt;# Enforce seccomp and AppArmor&lt;/span&gt;
&lt;span class="na"&gt;seccompDefault&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

&lt;span class="c1"&gt;# Restrict kubelet API access&lt;/span&gt;
&lt;span class="na"&gt;authentication&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;webhook&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;enabled&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;authorization&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;mode&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Webhook&lt;/span&gt;

&lt;span class="c1"&gt;# Drop legacy docker socket mounting&lt;/span&gt;
&lt;span class="na"&gt;enableCAdvisorJSONEndpoint&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  3. Enforce Pod-Level Security Contexts
&lt;/h3&gt;

&lt;p&gt;Even with runtime hardening, each pod must explicitly drop privileges:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Pod&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;secure-workload&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;securityContext&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runAsNonRoot&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;runAsUser&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;10000&lt;/span&gt;
    &lt;span class="na"&gt;fsGroup&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;10000&lt;/span&gt;
    &lt;span class="na"&gt;seccompProfile&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RuntimeDefault&lt;/span&gt;
  &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;myapp:latest&lt;/span&gt;
    &lt;span class="na"&gt;securityContext&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;allowPrivilegeEscalation&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
      &lt;span class="na"&gt;readOnlyRootFilesystem&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
      &lt;span class="na"&gt;capabilities&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;drop&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;ALL&lt;/span&gt;
    &lt;span class="na"&gt;volumeMounts&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tmp&lt;/span&gt;
      &lt;span class="na"&gt;mountPath&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/tmp&lt;/span&gt;
  &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tmp&lt;/span&gt;
    &lt;span class="na"&gt;emptyDir&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Operational Trade-offs and Failure Boundaries
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Build cache invalidation&lt;/strong&gt;: Multi-stage Dockerfiles with BuildKit cache mounts become critical when you can't rely on layer caching across nodes with different UID mappings. I had to rewrite every Dockerfile to use &lt;code&gt;--mount=type=cache,target=/go/pkg/mod&lt;/code&gt; to avoid re-downloading dependencies on every build.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Persistent volume permissions&lt;/strong&gt;: When user namespaces are enabled, any pre-existing PersistentVolumes with root-owned files will fail permission checks. You need an init container to chown recursively:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;initContainers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;fix-permissions&lt;/span&gt;
  &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;busybox:1.36&lt;/span&gt;
  &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;sh'&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;-c'&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;chown&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;-R&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;10000:10000&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;/data'&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;volumeMounts&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;data&lt;/span&gt;
    &lt;span class="na"&gt;mountPath&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/data&lt;/span&gt;
  &lt;span class="na"&gt;securityContext&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runAsUser&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;  &lt;span class="c1"&gt;# Init container exception for migration&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Socket mounting restrictions&lt;/strong&gt;: If any workload legitimately needs Docker socket access (CI/CD builders, monitoring agents), you must use rootless Docker-in-Docker with sysbox or gVisor runtime classes instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Unresolved Dilemma: Performance vs. Isolation
&lt;/h2&gt;

&lt;p&gt;User namespaces add measurable overhead. In high-throughput network workloads, I observed a 3-7% latency increase due to UID/GID translation in the kernel. For batch processing jobs with heavy filesystem I/O, the overhead was negligible.&lt;/p&gt;

&lt;p&gt;The real question: is the security boundary worth the operational complexity when you're already running workloads in separate network policies and resource quotas? Some teams solve this by running untrusted workloads in dedicated node pools with user namespaces enabled, while keeping trusted platform services on traditional privileged runtimes.&lt;/p&gt;

&lt;p&gt;What's your threshold for accepting kernel-level isolation overhead in production Kubernetes nodes?&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Production-Hardening Pi-hole in Docker: Beyond the Quick-Start Compose</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Mon, 19 Oct 2026 14:48:32 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/production-hardening-pi-hole-in-docker-beyond-the-quick-start-compose-2ifm</link>
      <guid>https://dev.to/hannah_schmidt/production-hardening-pi-hole-in-docker-beyond-the-quick-start-compose-2ifm</guid>
      <description>&lt;h2&gt;
  
  
  The Gap Between Demo and Production
&lt;/h2&gt;

&lt;p&gt;The official &lt;code&gt;pi-hole/docker-pi-hole&lt;/code&gt; image ships with a clean quick-start &lt;code&gt;docker-compose.yml&lt;/code&gt; that gets DNS ad-blocking running in under 60 seconds. But the moment you move beyond localhost testing—exposing it to a home network or integrating it into an existing service mesh—you hit three operational boundaries the quick-start doesn't address:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The container runs as root by default&lt;/strong&gt;, giving any compromised Pi-hole process full host filesystem access if combined with a container escape.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Healthchecks rely on &lt;code&gt;curl&lt;/code&gt; hitting the web interface&lt;/strong&gt;, which doesn't validate actual DNS resolution—your stack reports healthy while DNS queries silently fail.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Upstream DNS configuration persists in a SQLite database inside the container&lt;/strong&gt;, making declarative infrastructure-as-code impossible without custom entrypoint surgery.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Non-Root Hardening: The UID/GID Trap
&lt;/h2&gt;

&lt;p&gt;Pi-hole's &lt;code&gt;lighttpd&lt;/code&gt; and &lt;code&gt;pihole-FTL&lt;/code&gt; processes bind to privileged ports 53 (DNS) and 80 (HTTP) by default. Dropping to a non-root user requires three coordinated changes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;# Custom Dockerfile extending official image&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="s"&gt; pihole/pihole:latest&lt;/span&gt;

&lt;span class="k"&gt;RUN &lt;/span&gt;groupadd &lt;span class="nt"&gt;-g&lt;/span&gt; 999 pihole &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="se"&gt;\
&lt;/span&gt;    useradd &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; 999 &lt;span class="nt"&gt;-g&lt;/span&gt; pihole pihole &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="se"&gt;\
&lt;/span&gt;    &lt;span class="nb"&gt;chown&lt;/span&gt; &lt;span class="nt"&gt;-R&lt;/span&gt; pihole:pihole /etc/pihole /var/www/html

&lt;span class="k"&gt;USER&lt;/span&gt;&lt;span class="s"&gt; pihole&lt;/span&gt;
&lt;span class="k"&gt;EXPOSE&lt;/span&gt;&lt;span class="s"&gt; 8053/tcp 8053/udp 8080/tcp&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But this breaks immediately: &lt;code&gt;pihole-FTL&lt;/code&gt; refuses to start because it can't write to &lt;code&gt;/var/log/pihole-FTL.log&lt;/code&gt; or create shared-memory segments in &lt;code&gt;/dev/shm&lt;/code&gt;. The fix requires mounting specific paths with correct ownership:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;pihole&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./pihole-hardened&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;./pihole-data:/etc/pihole:rw&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/dev/shm:/dev/shm:rw&lt;/span&gt;
    &lt;span class="na"&gt;tmpfs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/tmp:uid=999,gid=999&lt;/span&gt;
    &lt;span class="na"&gt;cap_drop&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;ALL&lt;/span&gt;
    &lt;span class="na"&gt;cap_add&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;NET_BIND_SERVICE&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;NET_BIND_SERVICE&lt;/code&gt; capability lets the non-root process bind to port 53 without needing &lt;code&gt;--privileged&lt;/code&gt;. But now you've traded root access for a new failure mode: if your Docker host's &lt;code&gt;/dev/shm&lt;/code&gt; has &lt;code&gt;noexec&lt;/code&gt; set in &lt;code&gt;/etc/fstab&lt;/code&gt;, &lt;code&gt;pihole-FTL&lt;/code&gt; crashes with a cryptic "cannot allocate memory" error. The root cause? FTL expects to &lt;code&gt;mmap&lt;/code&gt; executable shared memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Healthcheck Theatre vs Real DNS Validation
&lt;/h2&gt;

&lt;p&gt;The stock compose file includes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;healthcheck&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;test&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CMD"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;curl"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-f"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://localhost/admin"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This passes as long as the web interface responds—even when DNS is completely broken. A production-grade check validates the actual DNS resolution path:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;healthcheck&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;test&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CMD"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dig"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;+short"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;@127.0.0.1"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pi.hole"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;interval&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30s&lt;/span&gt;
  &lt;span class="na"&gt;timeout&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5s&lt;/span&gt;
  &lt;span class="na"&gt;retries&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But this introduces a new dependency: the image doesn't ship with &lt;code&gt;dig&lt;/code&gt;. You either add &lt;code&gt;bind-tools&lt;/code&gt; in a custom layer (bloating the image by 8MB) or switch to &lt;code&gt;nslookup&lt;/code&gt;, which has inconsistent exit codes across Alpine and Debian base images.&lt;/p&gt;

&lt;h2&gt;
  
  
  The SQLite Configuration Drift Problem
&lt;/h2&gt;

&lt;p&gt;Pi-hole stores upstream DNS servers in &lt;code&gt;/etc/pihole/gravity.db&lt;/code&gt;. Changing them via environment variables only works on first boot—subsequent container restarts ignore &lt;code&gt;PIHOLE_DNS_&lt;/code&gt; vars because the DB takes precedence. The official solution is "use the web UI or CLI," which breaks declarative deploys.&lt;/p&gt;

&lt;p&gt;The workaround requires an init script that forcibly resets the DB on every start:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
sqlite3 /etc/pihole/gravity.db &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="s2"&gt;"UPDATE ftl SET value='1.1.1.1;8.8.8.8' WHERE key='DNS_SERVERS'"&lt;/span&gt;
&lt;span class="nb"&gt;exec&lt;/span&gt; /s6-init
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But now you've coupled your infrastructure to Pi-hole's internal schema, which has broken twice in the last 18 months during minor version upgrades.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Resource Limit Dilemma
&lt;/h2&gt;

&lt;p&gt;Pi-hole's FTL process uses a ring buffer in shared memory for query logs. Setting Docker memory limits below the buffer size causes silent truncation—queries succeed, but the dashboard shows no history. The default buffer is 256MB, but under burst traffic (e.g., a browser opening 40 tabs), memory spikes to 512MB for ~30 seconds.&lt;/p&gt;

&lt;p&gt;Setting &lt;code&gt;mem_limit: 512m&lt;/code&gt; works until you run on a host with memory pressure, where the OOM killer terminates Pi-hole mid-query, poisoning downstream DNS caches with SERVFAIL responses that persist for 5 minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where Does This Leave You?
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;pi-hole/docker-pi-hole&lt;/code&gt; image is production-capable, but the gap between the quick-start and a hardened deployment isn't documented. You'll either:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Accept running as root and rely on network isolation&lt;/li&gt;
&lt;li&gt;Invest 2-3 days hardening the image and tracking schema changes&lt;/li&gt;
&lt;li&gt;Use Pi-hole's Helm chart, which handles most of this but locks you to Kubernetes&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's your deployment's actual blast radius if Pi-hole gets compromised—and is the ad-blocking ROI worth the operational surface area?&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Notes from a 2 AM On-Call: Why Our Android CI Containers Silently Froze (And the iptables Rule That Fixed It)</title>
      <dc:creator>Hannah Schmidt</dc:creator>
      <pubDate>Sun, 18 Oct 2026 16:16:39 +0000</pubDate>
      <link>https://dev.to/hannah_schmidt/notes-from-a-2-am-on-call-why-our-android-ci-containers-silently-froze-and-the-iptables-rule-that-8pg</link>
      <guid>https://dev.to/hannah_schmidt/notes-from-a-2-am-on-call-why-our-android-ci-containers-silently-froze-and-the-iptables-rule-that-8pg</guid>
      <description>&lt;h1&gt;
  
  
  Notes from a 2 AM On-Call: Why Our Android CI Containers Silently Froze (And the iptables Rule That Fixed It)
&lt;/h1&gt;

&lt;h2&gt;
  
  
  The Failure Symptom
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-10-18 02:14:33 [ERROR] Emulator health check timeout (container: android-worker-3)
2026-10-18 02:14:48 [ERROR] ADB connection refused: 5555 (container: android-worker-7)
2026-10-18 02:15:12 [WARN] Docker exec into android-worker-3: process hangs, no response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Our CI pipeline running &lt;code&gt;budtmo/docker-android&lt;/code&gt; for parallel Appium test execution started dropping 40% of jobs at 2 AM UTC when concurrent load spiked from 8 to 24 containers. No crash logs. No OOM kills. Containers appeared healthy in &lt;code&gt;docker ps&lt;/code&gt; but were zombies—ADB refused connections, noVNC showed frozen emulator screens, and &lt;code&gt;docker exec&lt;/code&gt; commands hung indefinitely.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Red Herring
&lt;/h2&gt;

&lt;p&gt;Standard Android emulator docs and docker-android examples suggest exposing ADB ports with &lt;code&gt;-p 5555:5555&lt;/code&gt; for host access. Our compose stack did exactly that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;android-worker&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;budtmo/docker-android:emulator_11.0&lt;/span&gt;
    &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;5555-5578:5555"&lt;/span&gt;  &lt;span class="c1"&gt;# ADB port range for 24 workers&lt;/span&gt;
    &lt;span class="na"&gt;privileged&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Under light load (&amp;lt; 10 concurrent emulators), this worked. At scale, containers froze without kernel panics or resource exhaustion. &lt;code&gt;strace&lt;/code&gt; on the frozen emulator process revealed it was stuck in &lt;code&gt;futex(FUTEX_WAIT)&lt;/code&gt; inside the Android emulator's internal lock—but why?&lt;/p&gt;

&lt;h2&gt;
  
  
  The Root Cause &amp;amp; Runnable Fix
&lt;/h2&gt;

&lt;p&gt;The breakthrough came from a packet capture on the Docker bridge: &lt;strong&gt;external scanners were hitting our exposed ADB port range&lt;/strong&gt;. When an external TCP SYN landed on an emulator's ADB port during emulator boot (a 15-second window), the emulator's ADB daemon entered a race condition in its internal socket accept loop, causing a deadlock in &lt;code&gt;libc&lt;/code&gt;'s mutex.&lt;/p&gt;

&lt;p&gt;budtmo/docker-android containers don't need published ADB ports for internal orchestration—the MCP server and video recorder connect over the Docker bridge network. Publishing them created a 24-port attack surface.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The 3-line fix&lt;/strong&gt; (added to &lt;code&gt;docker-compose.yml&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;android-worker&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;budtmo/docker-android:emulator_11.0&lt;/span&gt;
    &lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;android_internal&lt;/span&gt;
    &lt;span class="c1"&gt;# ports: []  # Removed: no published ports&lt;/span&gt;
    &lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;EMULATOR_ARGS="-no-snapshot -no-audio -no-window"&lt;/span&gt;

&lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;android_internal&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;driver&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;bridge&lt;/span&gt;
    &lt;span class="na"&gt;internal&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;  &lt;span class="c1"&gt;# Allow outbound, block inbound&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Plus one &lt;code&gt;iptables&lt;/code&gt; rule on the host to explicitly drop external traffic to the bridge subnet:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;iptables &lt;span class="nt"&gt;-I&lt;/span&gt; DOCKER-USER &lt;span class="nt"&gt;-i&lt;/span&gt; eth0 &lt;span class="o"&gt;!&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; 10.0.0.0/8 &lt;span class="nt"&gt;-j&lt;/span&gt; DROP
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After deploying, we ran 48-hour soak tests at 32 concurrent containers: &lt;strong&gt;zero freezes&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Invariant Rule
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Never publish container ports unless the host needs direct access.&lt;/strong&gt; Internal service-to-service communication over Docker bridge networks eliminates port conflicts, reduces attack surface, and avoids kernel-level port forwarding race conditions that manifest only under concurrency.&lt;/p&gt;

&lt;p&gt;For production Android CI with budtmo/docker-android: use internal networks, enforce &lt;code&gt;privileged: false&lt;/code&gt; where possible, and place a reverse proxy (Traefik, Nginx) in front of noVNC if browser access is required—never expose raw emulator ports to &lt;code&gt;0.0.0.0&lt;/code&gt;.&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>linux</category>
      <category>productivity</category>
    </item>
  </channel>
</rss>
