DEV Community: Alex Darbyshire

The Autonomous Writer: Experimenting with Self-Evolving System Prompt and Context

Alex Darbyshire — Sat, 21 Mar 2026 10:41:00 +0000

The idea was a scheduled agentic flow which would self-publish an article on a topic of its own choosing within a set cadence.

The result: somewhere around one to ‘one and a half’ articles a week, posted to https://theautonomouswriter.com

Idea to Implementation

On Sunday morning I semi-legibly wrote the following in my notebook.

Handwritten notes outlining the autonomous writer concept

I snapped a pic with my phone and passed it to Gemini Pro 3.0 with the prompt ‘OCR this then start speccing out the design with me’ (sic).

After a few iterations, we arrived at a design spec. I got off the phone and onto Fedora, init'd a git repo and dropped Gemini's spec into Claude CLI (running Opus 4.6). Claude wrote this design spec to file Design Spec, having re-written Gemini's original.

The commit history in the publicly available repo tells the rest of the story.

A few highlights:

JSON based memory in the repo avoiding need for relational/noSQL/vector DBs
a bit of a voice, a system prompt with foundational principles and influences (not quite Asimov's three laws)
the ability to edit its own system prompt, excepting its foundational principles (hopefully)

Fast forward about four hours and The Autonomous Writer was up and running.

Tech Stack

The hosting stack ended up being very similar to this site.

And on the agentic side:

Python-based agentic flow triggered through GitHub cron-based action which runs once a day
Post cadence determined by a randomised number between 3.5 and 5.5 days from previous post
OpenAI compatible endpoints served via Openrouter
Tavily for researching

Feature Roadmap

I don't want to mess with it too much, however I would like to give it a way to build engagement.

Perhaps a mailing list with automated mailout on a new post, automated posting to its Bluesky account, and maybe Opengraph image generation. All set and forget style.

Internet and Environmental Pollution

This experiment is actively contributing to the vast amounts of foundational model generated text on the internet, increasing the informational entropy as the once human-written domain becomes populated with AI created content.

Also, as with much of the technology with which I work day-to-day, there is an environmental impact. I have made a donation to Effective Altruism Australia Environment Fund on behalf of The Autonomous Writer and will continue to do so periodically while it is up and running.

Final thought

I am interested to watch how this evolves.

Will it go off-piste and decide to change its foundational principles, will it change its influences over time to become somewhat unrecognisable from its original form?

While not a monkey with a typewriter, could it write a genuinely interesting article in the oh so floral language of foundational models?

An Ode to the Occasional Proxy: Dynamic HTTP Forward Proxy Configuration in Windows, WSL/Ubuntu, and Docker

Alex Darbyshire — Thu, 17 Jul 2025 23:52:00 +0000

You may find yourself... supporting developers working behind a non-transparent forward proxy which is implemented in some environments and not others.

And you may ask yourself, "How do I work this?" - with a nod to the Talking Heads, whose "Once in a Lifetime" seems strangely appropriate when dealing with proxy configurations which work sometimes but not others.

In Windows, a host of CLI tools commonly used in development don't inherit Windows proxy settings. Tools like curl, wget, npm, pip, go modules, Composer, etc. Similar behaviour can be seen with many of the underlying request libraries (notably excepting .NET's HttpClient).

Proxy environment variables HTTP_PROXY, HTTPS_PROXY, NO_PROXY, FTP_PROXY, ALL_PROXY and their uncapitalised variants can help with this, but they can also introduce quirks, particularly when you use a proxy in one environment and not another on the same machine.

For example, a laptop moving between office network and home network. Supporting users of varying technical knowledge using say Python occasionally can get interesting: 'Hey, have you tried this Python script that will make your job easier?'... "It doesn't install half the time, I give up."

In both Windows and Linux, the implementation of proxy environment variables and proxy configuration settings isn't as standardised as one would like. See this post from GitLab: We need to talk: Can we standardize NO_PROXY? - it does a great job giving the lay of the land.

In summary: NO_PROXY behaves differently (asterisk VS leading-dot VS maybe accepts CIDR notation), usage and precedence of capitalised and non-capitalised varies, and some programs may ignore proxy environment variables entirely opting for their own custom config.

From an ops point of view, there are alternative implementations such as a transparent proxy. That is a forward, intercepting proxy which operates without client-side configuration. This potetentially makes a juicy target for bad actors (all your HTTP traffic eggs in one basket).

These notes are for the person who doesn't have the scope to implement a transparent proxy, and the person who is working without local admin (a prudent measure in many environments). Of course, get the blessing of your ops teams before messing around with network configurations, especially on other machines which they will have to support.

From a security perspective in an operational network environment, there are good reasons for having a forward proxy. For example, install a Trusted Root Certificate on each user's machine via GPO and a proxy allows one to Man In The Middle SSL traffic for inspection, filtering and blocking by security software.

This helps immeasurably in preventing threats, and as is often the case with great power, brings along a host of regulatory and compliance requirements, great responsibility.

Overall Approaches

Add more proxies:
Set up a local proxy which points to the internal proxy with a failover to direct access, and configure proxy settings once via environment variables. If the proxy you are working behind requires say Kerberos or NTLM auth - this is the only approach for many of the tools (some tools have specific config for these auth methods, it starts to get finicky).

I do enjoy the touch of irony to this solution in that we add another proxy to solve proxy issues. This approach has the advantage of working across existing processes when moving between environments. It has the cons of a single point of failure, adding another bit of software and added complexity when trying to access from within containers (due to 127.0.0.1 being the container itself).

Shell startup scripts:
Use shell startup scripts to set and unset proxy environment variables based on environment. This only works for non-authenticated proxies.

There are trade-offs: adding a bunch of logic to startup scripts isn't ideal, shell startup time is slowed, and if you're doing this on other users' machines, there can be complications around when they actually open shells. Also, if you're working across larger WANs, you may need to check several proxy IPs. And, key is that existing processes won't receive the set or unset environment variables, meaning restarting programs is required.

A variation on this is adding a service that periodically updates environment variables, though this can potentially disassociate the idea that processes have the env vars they were spawned with.

Another variation is to create a script which the user runs manually when they change environment, although this removes any semblance of dynamism.

Which approach? I have tended towards the second option, mainly as I implement it on others' machines in addition to my own. The devs and dev environment is in WSL/Ubuntu using Bash startup scripts with a bit of hackery - I leave the Windows env vars alone as it limits the potential for breaking business-as-usual type work. I might go the first option if I was looking to solve this on my machine only, and of course there would be no other viable option if dealing with authenticated proxies.

Proxy Chaining a Local Forward Proxy

A few of the relevant tools in this space:

CNTLM - supports NTLM upstream, not Kerberos
TinyProxy - small footprint
Squid - open-source enterprise grade, widely adopted
Privoxy - geared towards privacy/filtering
kpx - has failover, supports NTLM and Kerberos upstream
px - supports NTLM and Kerberos upstream
Proxifier - commercial license, requires admin to install network hook as it intercepts traffic. Notably this allows it to proxy regardless of env vars or indeed support in the program being proxied
Escobar - supports NTLM and Kerberos upstream
Fiddler - supports NTLM and Kerberos upstream. Debugging/modifying/inspecting focus
Charles - supports NTLM and Kerberos upstream. Debugging/inspecting focus

Of these, CNTLM and kpx handle failover and upstream auth based on a skim of the docs. CNTLM has been around since 2007 and is written in C but doesn't support Kerberos. Apparently Microsoft is deprecating NTLM.

kpx is written in Go, supports NTLM and Kerberos, and under active dev for about a year at time of writing.

We'll use kpx in our example implementation.

Note that if deployed on Windows host and accessed from WSL: using default settings at the time of writing, the proxy won't be available at 127.0.0.1 from within WSL (as it refers to the WSL instance itself, not the Windows host). You can grab the IP of the Windows host.

kpx Implementation

Download the binary for your chosen system from the kpx releases page. Or, pull the repo and go run cli/main.go

Co-locate a kpx.yaml config file with the binary.

bind: 127.0.0.1
port: 7777
debug: true # This console logs all requests - for demo purposes, would turn it off for day-to-day

proxies:
  our_work:
    type: anonymous
    host: your-proxy 
    port: 3128

rules:
  - host: "*"
    proxy: our_work,direct # this CSV defines the failover chain, in our case try 'our_work' first and then fall back to direct

Worth noting, in the current release (v1.11.0) the failover chain is attempted for each request. Also, there was an error related to use of 'direct' in the failover (rules[].host.proxy) - I submitted a pull request to address this.

Proxy Environment Variables for 'Local Proxy' approach

For the local proxy approach, set your HTTP_PROXY, HTTPS_PROXY, FTP_PROXY and their lowercase variants to http://127.0.0.1:7777 once in the User space for Windows (via PowerShell [Environment]::SetEnvironmentVariable("KEY", "VALUE", "User") or Settings Edit environment variables for your account) and/or Ubuntu/WSL (via /etc/environment). Since we failover to direct, NO_PROXY is optional.

With NO_PROXY, be aware asterisks, wildcards and CIDR masks may or may not work depending on the underlying program (refer back to We need to talk: No Proxy)

Docker Local Proxy Caveat

127.0.0.1 in a Docker container won't provide access to the Linux host by design, so our local proxy won't be available in containers using this IP.

There are a few workarounds at time of writing. They introduce a more environment-specific quirks (ugh). See Stackoverflow From inside of a Docker container, how do I connect to the localhost of the machine? [closed].

One could say create the proxy in a container, use macvlan to give it its own routeable IP (a bit icky), and then use ~/.docker/config.json to set docker specific proxy env vars using the proxy container's IP (and override the config.json env vars in the proxy container itself to avoid a loop).

Startup Script - PowerShell Profile

Adding the below script to a PowerShell profile will dynamically set proxy environment variables to those declared in the script.

The testing logic opted for here is Do we get an OK response from google via the proxy within 1 second?.

We could do this some loops and less lines, opted for explicit and verbose.

# === POWERSHELL PROXY SETUP SCRIPT START ===
# Version: 1.0 - HTTP Proxy Auto-Configuration Script
# Generated: 2025-07-17
$proxyUrl = "http://your-proxy:8080"
$noProxy = "localhost,127.0.0.1,.local,.mycorporatedomain.com,*.mycorporatedomain.com"

function Set-ProxyEnvironmentVariables {
    param($ProxyUrl, $NoProxyList)

    Write-Host "Setting proxy environment variables..." # both cases for compatibility

    # Set proxy environment variables for User
    [Environment]::SetEnvironmentVariable("HTTP_PROXY", $ProxyUrl, "User")
    [Environment]::SetEnvironmentVariable("HTTPS_PROXY", $ProxyUrl, "User")
    [Environment]::SetEnvironmentVariable("http_proxy", $ProxyUrl, "User")
    [Environment]::SetEnvironmentVariable("https_proxy", $ProxyUrl, "User")
    [Environment]::SetEnvironmentVariable("NO_PROXY", $NoProxyList, "User")
    [Environment]::SetEnvironmentVariable("no_proxy", $NoProxyList, "User")

    # Set proxy environment variables for current process (in case someone uses the shell)
    [Environment]::SetEnvironmentVariable("HTTP_PROXY", $ProxyUrl, "Process")
    [Environment]::SetEnvironmentVariable("HTTPS_PROXY", $ProxyUrl, "Process")
    [Environment]::SetEnvironmentVariable("http_proxy", $ProxyUrl, "Process")
    [Environment]::SetEnvironmentVariable("https_proxy", $ProxyUrl, "Process")
    [Environment]::SetEnvironmentVariable("NO_PROXY", $NoProxyList, "Process")
    [Environment]::SetEnvironmentVariable("no_proxy", $NoProxyList, "Process")

    Write-Host "Proxy environment variables set successfully. Restart any processes reliant on them."
}

function Clear-ProxyEnvironmentVariables {
    Write-Host "Clearing proxy environment variables..."

    # Clear all proxy-related variables
    $variablesToClear = @("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "NO_PROXY", "no_proxy")

    foreach ($variable in $variablesToClear) {
        [Environment]::SetEnvironmentVariable($variable, $null, "User")
        [Environment]::SetEnvironmentVariable($variable, $null, "Process")
    }

    Write-Host "Proxy environment variables cleared. Restart any processes reliant on them."
}

function Test-ProxyConnection {
    param($ProxyUrl)

    try {
        Write-Host "Testing proxy connection..."
        $response = Invoke-WebRequest -Uri "https://www.google.com" -Proxy $ProxyUrl -TimeoutSec 1 -UseBasicParsing
        return $response.StatusCode -eq 200
    } catch {
        Write-Host "Proxy connection test failed: $($_.Exception.Message)"
        return $false
    }
}

# Main logic
if (Test-ProxyConnection -ProxyUrl $proxyUrl) {
    Set-ProxyEnvironmentVariables -ProxyUrl $proxyUrl -NoProxyList $noProxy
} else {
    Clear-ProxyEnvironmentVariables
}
# === POWERSHELL PROXY SETUP SCRIPT END ===

PowerShell's user profile startup script varies depending on version, PowerShell 7+ is at %userprofile%\Documents\PowerShell\profile.ps1

One could also alter the Windows proxy settings themselves in this script by altering the relevant registry keys.

#Enable
Set-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name ProxyEnable -Value 1 

#Disable
Set-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name ProxyEnable -Value 0

Startup Script - Bash via /etc/bash.bashrc (Ubuntu)

Adding the below to /etc/bash.bashrc will dynamically set proxy env vars to those declared in the script.

We use /etc/bash.bashrc instead of /etc/profile as we want to set this for interactive and non-interactive shells. We use the global startup scripts instead of user-based ones as we want to set the env vars for all users (which for these machines tends to be a single user). Can't imagine needing to apply this to a multi-user machine. We don't use /etc/environment as this applies on boot.

The script uses jq for manipulating JSON, make sure it is installed sudo apt update && sudo apt install jq

As with the PowerShell variant, this uses the testing logic Do we get an OK response from Google via the proxy within 1 second?

Note: This script could be much simpler if we opted to not handle Docker Daemon or warn about sudoers env vars.

# === PROXY SETUP SCRIPT START ===
# Version: 1.0 - HTTP Proxy Auto-Configuration Script
# Generated: 2025-07-17

proxy_url="http://your-proxy:3128"
no_proxy="localhost,127.0.0.1,.local,.mycorporatedomain.com,*.mycorporatedomain.com"

# Function to check and warn missing sudoers configuration for proxy environment variables
check_sudoers_proxy_config() {
    if ! grep -q "env_keep.*HTTP_PROXY" /etc/sudoers 2>/dev/null; then
        echo "Warning: Proxy environment variables may not be preserved with sudo commands, e.g. sudo apt install"
        echo "Consider adding this line to /etc/sudoers using 'sudo visudo':"
        echo "Defaults env_keep += \"HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy\""
    fi
}

# Function to handle docker daemon config and restart if required
update_docker_daemon_proxy() {
    local action="$1"  # "set" or "unset"
    local proxy_url="$2"
    local no_proxy="$3"

    # Only proceed if Docker is installed
    command -v docker > /dev/null || return

    # Ensure daemon.json exists
    if [ ! -f /etc/docker/daemon.json ]; then
        echo "Creating /etc/docker/daemon.json..."
        echo '{}' | sudo tee /etc/docker/daemon.json > /dev/null
        sudo chown root:root /etc/docker/daemon.json
        sudo chmod 644 /etc/docker/daemon.json
    fi

    local needs_update=false

    if [ "$action" = "set" ]; then
        local current_http_proxy=$(jq -r '.proxies["http-proxy"] // empty' /etc/docker/daemon.json)
        local current_https_proxy=$(jq -r '.proxies["https-proxy"] // empty' /etc/docker/daemon.json)
        local current_no_proxy=$(jq -r '.proxies["no-proxy"] // empty' /etc/docker/daemon.json)

        if [ "$current_http_proxy" != "$proxy_url" ] || [ "$current_https_proxy" != "$proxy_url" ] || [ "$current_no_proxy" != "$no_proxy" ]; then
            needs_update=true
        fi 

    else
        local has_proxies=$(jq -r 'has("proxies")' /etc/docker/daemon.json 2>/dev/null)
        [ "$has_proxies" = "true" ] && needs_update=true
    fi

    if [ "$needs_update" = "true" ]; then
        sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.backup

        if [ "$action" = "set" ]; then
            jq ". + {\"proxies\": {\"http-proxy\": \"$proxy_url\", \"https-proxy\": \"$proxy_url\", \"no-proxy\": \"$no_proxy\"}}" /etc/docker/daemon.json > /tmp/daemon.json 2>/dev/null
        else
            jq 'del(.proxies)' /etc/docker/daemon.json > /tmp/daemon.json 2>/dev/null
        fi

        if [ $? -eq 0 ]; then
            sudo mv /tmp/daemon.json /etc/docker/daemon.json
            sudo systemctl restart docker 2>/dev/null || echo "Warning: Could not restart Docker daemon"
            echo "Docker daemon proxy configuration ${action}"
        else
            echo "Warning: Failed to update Docker daemon configuration"
            rm -f /tmp/daemon.json
        fi
    fi
}

# Function to test and set proxy
setup_proxy() {

    if curl -s --connect-timeout 1 --proxy "$proxy_url" https://www.google.com > /dev/null 2>&1; then
        # Set environment variables
        export HTTP_PROXY="$proxy_url"
        export HTTPS_PROXY="$proxy_url"
        export NO_PROXY="$no_proxy"
        export http_proxy="$proxy_url"
        export https_proxy="$proxy_url"
        export no_proxy="$no_proxy"

        update_docker_daemon_proxy "set" "$proxy_url" "$no_proxy"

        # Set PEAR proxy if installed
        command -v pear > /dev/null && pear config-set http_proxy "$proxy_url" 2>/dev/null

        echo -e "Proxy environment configured\n"
    else
        # Unset environment variables
        unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy

        update_docker_daemon_proxy "unset"

        # Unconfigure PEAR proxy if installed
        command -v pear > /dev/null && pear config-set http_proxy "" 2>/dev/null

        echo -e "No proxy detected, proxy configuration cleared\n"
    fi
}

# Run on shell startup
setup_proxy
check_sudoers_proxy_config
# === PROXY SETUP SCRIPT END ===

Notes on specific usages

Docker (modern)

docker run and docker build pass in proxy env vars from the process they were executed in automatically
docker pull uses the Docker Daemon config in /etc/docker/daemon.json. This settings affects the FROM line when building Dockerfiles (as it is effectively a docker pull)
Dockerfiles - in most cases I have been able to avoid baking in references to proxies or env vars in images (with exception of PEAR/PECL)

An alternate approach to process env vars is using ~/.docker/config.json - I feel having the process level env vars is better (unless using the local proxy approach in which case setting specific in-container proxy env vars would help work around pointing to local).

The startup script restarting the docker daemon will stop and start containers causing env vars to be propagated.

Note: Older versions of Docker did not do as much automatically with proxy env vars.

Another Note: the proxy config settings of daemon.json and config.json differ - your go-to LLM will most likely serve you up settings structured for config.json.

Third Note: Docker Daemon technically does respect proxy environment variables, however the process is usually managed by the init system (systemd in Ubuntu's case) which requires environment variables to be specifically set within a service conf or service override conf. To have a look at the environment variables of the dockerd process, see sudo cat /proc/$(pgrep dockerd)/environ | tr '\0' '\n'

kind

Tear down and reinstantiate when moving between environments to allow env vars to propagate (after restarting Docker Daemon):

kind delete cluster
kind create cluster

Tilt

When installed using the standard install script, the image builds are called using runc as root and don't get the env vars propagated.

Tiltfiles use Starlark configuration language, one can dynamically populate a config object based on the env vars and then pass that object as build_args. This avoids the issue of missing/null value error when trying to use the env vars directly in the Tiltfile build_args.

# In your Tiltfile
proxy_config = {}
proxy_vars = ['HTTP_PROXY', 'HTTPS_PROXY', 'NO_PROXY']

for var in proxy_vars:
    value = os.getenv(var)
    if value:
        proxy_config[var] = value

docker_build('myapp', '.', build_args=proxy_config)

apt Notes

apt is typically run using sudo, and for good reason env vars do not get passed to sudo.

To whitelist proxy environment variables for sudo, add this to /etc/sudoers using visudo:

Defaults env_keep += "HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy"

This allows the proxy environment variables to be passed through to elevated commands.

The alternative here is to script setting/unsetting proxy settings in /etc/apt/apt.conf.

PEAR and PECL notes

These don't respect the proxy environment variables, need to specifically tell PEAR to use the proxy. Yes, some of us still use PECL on occasion in PHP development, haha.

Direct command to set PEAR proxy:

pear config-set http_proxy http://your-proxy:3128

In Dockerfiles that include PEAR/PECL installs:

# Set PEAR proxy using environment variable (or default to no proxy), PECL shares these settings
ARG HTTP_PROXY="''" 
RUN pear config-set http_proxy $HTTP_PROXY
RUN pecl install your-package

.NET 7.0 & 8.0 Windows Notes

The out-of-the-box requests library HttpClient in .NET respects Windows proxy settings. It stops respecting these settings as soon as one of the env vars is set, so correctly setting HTTP_PROXY and missing NO_PROXY could cause issues when accessing local resources.

The default behaviour can be altered in the code.

Keep aware of the env vars being applied for a specific process - e.g. your instance of Rider may be executing programs using a specific version PowerShell which has a different startup script to the one you expect.

.NET 7.0 & 8.0 no_proxy syntax

Wildcards are not *.myexcludeddomain.com, this can come as a surprise if you derived your exclusions from say your Windows proxy settings which does support asterisk based wildcards. Wildcard format is leading dot .myexcludeddomain.com

Line from relevant lib: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpEnvironmentProxy.cs#L251

Framework 4.8 Notes

By default, programs compiled to Framework 4.8 respect Windows proxy settings. They do not respect proxy environment variables.

VSCode and IntelliJ Proxy Behaviour

VSCode

Windows: VSCode inherits system proxy settings from Windows Internet Options by default. Changes to system proxy settings require restarting VSCode to take effect, as the Electron framework caches network configuration at startup.

Ubuntu: VSCode respects standard proxy environment variables (HTTP_PROXY, HTTPS_PROXY, NO_PROXY). When moving between proxy and non-proxy environments, VSCode must be restarted for the new environment variables to be recognised by the underlying Electron process.

Extensions that make network calls (like language servers, Git integration, or marketplace access) inherit these same proxy settings. However, some extensions may implement their own HTTP clients that don't respect the inherited configuration.

Manual proxy configuration is available via File > Preferences > Settings → search "proxy", though environment variables typically provide better automation for dynamic environments.

IntelliJ (Phpstorm, Rider, Pycharm, IDEA, etc)

Windows: IntelliJ detects and uses Windows system proxy settings automatically. Like VSCode, changes to system proxy configuration require an IDE restart to take effect, as the JVM caches network settings during initialisation.

Ubuntu: IntelliJ respects proxy environment variables when launching from a terminal that has them set. Moving between environments requires restarting the IDE to pick up new environment variable values.

Manual configuration is available via File > Settings > Appearance & Behavior > System Settings > HTTP Proxy. The "Auto-detect proxy settings" option works well on Windows but may be less reliable on Linux systems.

Plugin installations, version control operations, and built-in HTTP clients all inherit the configured proxy settings. However, some plugins that bundle their own HTTP libraries may require additional configuration or may otherwise override these settings. Continue.dev in IntelliJ at time of writing, I am looking at you... (it's open source so if I want it fixed, I can look to a mirror)

Debugging

Testing Proxy Connectivity

Use curl to test proxy connectivity:

# Test direct connection
curl -I https://www.google.com

# Test via proxy
curl -I --proxy http://your-proxy:8080 https://www.google.com

# Test with authentication
curl -I --proxy-user username:password --proxy http://your-proxy:8080 https://www.google.com

Setting Up a Test Environment

For testing at home, simulate a proxy environment:

Block direct internet access on test machine (via firewall rules or router):
Deploy Squid proxy using Docker on another box:

   docker run -d --name squid-proxy \
     -p 3128:3128 \
     ubuntu/squid:latest

Test your proxy configuration with the blocked environment.

This lets you figure it all out at your own leisure on your own gear.

Conclusion

Dealing with non-transparent forward proxies can be one of the 'fun ones' in work environments.

I feel like it is a good one to write about as there isn't an ideal solution at the development level. There are trade-offs, complexity and a touch of cognitive load whichever approach you take.

The aim is to improve the developer experience (while maintaining secure practices).

Same as it ever was

Dynamic Hostnames in Nginx: Docker, Kubernetes, and Azure Container Apps

Alex Darbyshire — Fri, 04 Jul 2025 23:52:00 +0000

Nginx fails to start when upstream services aren't available at startup time, showing host not found in upstream. This post focuses on fixing this issue in Docker first, then extends the solution to Kubernetes and Azure Container Apps using environment variables to construct dynamic hostnames (like my-rg-${ENVIRONMENT}-api) which adapt to different deployment environments.

The error occurs because Nginx resolves all hostnames at startup, not request time. This fail-fast behaviour is reasonable in many environments, but problematic in container orchestration where services start in unpredictable order or may not always be available by design.

Usually I would use Kubernetes ingress annotations, Kustomize, Helm and/or Namespaces to handle this - Azure Container Apps requires a different approach. I came across this when implementing separate environments for development, staging and production, and wanting to name the containers in a way that was descriptive within the Azure Sub.

Complete working example: nginx-docker-k8s-aca-example repository demonstrates this solution across all three platforms.

Why This Matters

Nginx is designed to fail fast if upstream hosts aren't available at startup. This behavior creates problems in some container environments.

As Sandro Keil explains, Nginx's default behavior prevents containers from starting when upstream services aren't yet available.

Docker Upstream Availability Problem

When you see this error in Docker:

nginx: [emerg] host not found in upstream

The problem happens when nginx tries to resolve upstream hostnames at startup:

# This will fail at startup if my-rg-local-api isn't available yet
upstream api_backend {
    server my-rg-local-api:80;
}

server {
    location / {
        proxy_pass http://api_backend;
    }
}

The solution is to use variables with proxy_pass, which defers DNS resolution until request time:

# This allows nginx to start even if upstream isn't available
server {
    location / {
        set $api_upstream "my-rg-local-api";
        proxy_pass http://$api_upstream:80;
    }
}

Important Note: When using variables with proxy_pass, you must configure a resolver. The resolver doesn't use search domains from /etc/resolv.conf like system tools, requiring explicit DNS configuration.

Docker Solution: Variables + Resolver

The solution has two parts:

Use variables to defer DNS resolution until request time
Configure a DNS resolver for variable resolution

First, enable nginx's built-in DNS resolver detection in your Dockerfile:

# Enable local DNS resolver (makes DNS IP from resolv.conf available as env var)
ENV NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1

Then configure Nginx using the automatically detected resolver:

# Use nginx's automatically detected DNS resolver
resolver ${NGINX_LOCAL_RESOLVERS} valid=10s ipv6=off;

server {
    listen 80;

    location / {
        # Variables defer DNS resolution until request time
        set $api_upstream "my-rg-${ENVIRONMENT}-api";
        proxy_pass http://$api_upstream:80;
    }
}

The key points:

Variables with proxy_pass defer DNS resolution from startup to request time
NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1 automatically extracts DNS resolver from /etc/resolv.conf
${NGINX_LOCAL_RESOLVERS} uses the automatically detected resolver IP
valid=10s ensures DNS entries expire quickly (helpful if services restart)
envsubst can substitute environment variables in templates for different environments

Kubernetes Extension

The Docker solution works until you deploy to Kubernetes. There you'll see:

unexpected DNS response for my-rg-local-api

Kubernetes introduces an additional problem - the resolvers configured in Nginx don't automatically use the search domains from the host's /etc/resolv.conf file.

Azure Container Apps Challenges

Moving to Azure Container Apps introduces yet another layer of DNS challenges:

"Error 404 - This Container App is stopped or does not exist."

The issue with Azure Container Apps is that the DNS configuration doesn't provide working hostnames by default. While container apps are accessible through DNS, the format is unique and requires special handling:

Container Apps have a specific internal domain naming pattern
You must use the .internal. prefix with the container app environment DNS suffix
Without this exact format, services won't be able to communicate with each other

This is why our solution explicitly uses .internal.${CONTAINER_APP_ENV_DNS_SUFFIX} for ACA environments.

The internal DNS within the Azure Container Apps environment does not show the internal part of the FQDN when using nslookup. And notably, using say curl with the service name alone works fine - suggesting there is something going on behind the scenes.

Universal Environment-Aware Solution

A solution that works across all three environments:

1. Domain Suffix Detection Script

Create a startup script that detects the environment type:

#!/bin/sh
# /docker-entrypoint.d/17-detect-k8s-and-aca.sh

RESOLV_CONF="/etc/resolv.conf"
export DOMAIN_SUFFIX=""

# Check if we're in a Kubernetes environment and set up environment variables
if [ -f "$RESOLV_CONF" ] && grep -q "cluster.local" "$RESOLV_CONF"; then
    # Extract the first search domain that includes cluster.local
    K8S_SEARCH_DOMAIN=$(grep "^search" "$RESOLV_CONF" | head -n1 | awk '{print $2}')
    export DOMAIN_SUFFIX=".${K8S_SEARCH_DOMAIN}"
fi

# Override domain suffix if running in Azure Container Apps
# Set HTTP protocol based on Azure Container Apps (ACA manages internal TLS)
if [ -n "$CONTAINER_APP_ENV_DNS_SUFFIX" ]; then
    export DOMAIN_SUFFIX=".internal.${CONTAINER_APP_ENV_DNS_SUFFIX}"
    export UPSTREAM_PROTOCOL="https"
else
    export UPSTREAM_PROTOCOL="http"
fi

# Monkey patch our dynamically determined env vars into our default template
envsubst '$DOMAIN_SUFFIX $UPSTREAM_PROTOCOL' < /etc/nginx/templates/default.conf.template > /tmp/default.conf.template.tmp
mv /tmp/default.conf.template.tmp /etc/nginx/templates/default.conf.template

The envsubst command substitutes environment variables in the template file, allowing our configuration to adapt to different environments without code changes.

2. Nginx Config Template

# Dynamic resolver from container environment (nginx automatically sets NGINX_LOCAL_RESOLVERS)
resolver ${NGINX_LOCAL_RESOLVERS} valid=10s ipv6=off;

server {
    listen 80;
    server_name _;

    # Common proxy headers (required by Azure Container Apps)
    proxy_ssl_server_name on;
    proxy_http_version 1.1;

    location / {
        set $frontend_upstream "my-rg-${ENVIRONMENT}-frontend${DOMAIN_SUFFIX}";
        proxy_pass ${UPSTREAM_PROTOCOL}://$frontend_upstream;
    }

    location /api {
        set $api_upstream "my-rg-${ENVIRONMENT}-api${DOMAIN_SUFFIX}";
        proxy_pass ${UPSTREAM_PROTOCOL}://$api_upstream/;
    }
}

3. Dockerfile Configuration

FROM nginx:alpine

COPY ./templates/default.conf.template /etc/nginx/templates/
COPY ./docker-entrypoint.d/17-detect-k8s-and-aca.sh /docker-entrypoint.d/

# Enable local DNS resolver (makes DNS IP from resolv.conf available as env var)
ENV NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1

# Document our intended local port
EXPOSE 80

What it looks like (Tilt/Kind)

Browser showing successful responses from both frontend (/) and API (/api) endpoints through the nginx proxy.

Environment-Specific Notes

Docker

Nginx's built-in NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1 automatically extracts DNS resolver from /etc/resolv.conf for compatibility
Variables with proxy_pass allow nginx to start before upstream services are available
This approach works well when dealing with multiple services that need to be proxied
Example docker-compose.yml setup:

  services:
    nginx:
      build: .
      ports:
        - "80:80"
      environment:
        - ENVIRONMENT=local

    api-service:
      image: strm/helloworld-http
      container_name: my-rg-local-api

Kubernetes

Script detects K8s by looking for cluster.local in /etc/resolv.conf
Automatically appends the correct namespace and domain suffix, allowing the same configuration to work across different namespaces without modification
Set variables through Pod spec:

  env:
    - name: ENVIRONMENT
      value: "local"

Azure Container Apps

Uses special format: .internal.${CONTAINER_APP_ENV_DNS_SUFFIX} which is required to fix DNS hostname resolution
The .internal. prefix is essential - without it, service-to-service communication fails
Automatically detects Azure environment and sets protocol to HTTPS (other environments use HTTP)
Requires proxy_ssl_server_name on and proxy_http_version 1.1
Set env vars in Azure portal or with Infrastructure as Code

Alternative Proxies

Traefik, Envoy, or Caddy likely solve one of these Out Of The Box.

References

Conclusion

This addresses Nginx's dynamic hostname resolution challenges across platforms:

For Docker users - properly resolves dynamic hostnames using the container's DNS resolver
For Kubernetes users - automatically handles search domains and service discovery without manual configuration
For Azure Container Apps - adapts to the special internal domain format required by the platform