Headless Chromium gives up on HTTP basic auth. A local proxy injects the header instead.

A dashboard I run sits behind nginx basic auth. The page is a small SPA: an HTML shell, then JavaScript that fetches a JSON index and builds the feed. I wanted a real browser to load it, because a screenshot of the rendered page catches layout problems that reading the JSON never will.

The first attempt produced no image at all. Not a blank page, not a login dialog.

chromium --headless=new --disable-gpu --no-sandbox \
  --window-size=800,600 --screenshot=page.png http://127.0.0.1/

Page load failed: net::ERR_INVALID_AUTH_CREDENTIALS
ls: cannot access 'page.png': No such file or directory

A plain curl against the same address answers 401 with the body of the challenge, so the origin is doing exactly what it should:

HTTP/1.1 401 Unauthorized
Server: nginx/1.24.0 (Ubuntu)
WWW-Authenticate: Basic realm="Restricted area"
Cache-Control: no-cache, no-store, must-revalidate, max-age=0

The server offers a realm and waits for an answer. Headless has no window in which to show the prompt, so the navigation ends as a failed load instead of as a page displaying a login box. The error name makes it worse than it is: ERR_INVALID_AUTH_CREDENTIALS suggests bad credentials were sent, and none were sent at all.

Two failures wear the same log line

Before reaching for a workaround, it helps to know which of two very different things you are looking at. Same URL, same moment, only the User-Agent differs:

curl -s -o /dev/null -w '%{http_code}\n' -A "Python-urllib/3.14" https://example.test/
# 403
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/140.0" https://example.test/
# 401

403 means something in front of the application refused the request, and on Cloudflare-fronted hosts a bare script User-Agent is enough to earn it. 401 means the origin received the request and asked for credentials. A script that only prints "fetch failed" cannot tell these apart, and guessing costs an afternoon: you go looking for the wrong password while a WAF is answering.

Log three things on every failed request and the ambiguity disappears: the status code, the Server header, and the User-Agent you sent. The proxy below sends a browser User-Agent for the same reason.

Read the credential from a file, inject the header locally

The fix that worked was to stop teaching the browser about authentication and put a small HTTP proxy in front of the origin. The proxy adds the Authorization header itself, so Chromium talks to a server that never asks for anything. This is the relevant slice of a 90 line script, with its original names:

import base64
import http.server
import pathlib
import socketserver
import urllib.error
import urllib.request

ORIGEM = "http://127.0.0.1:80"
CRED = pathlib.Path.home() / ".site_auth"
PORT = 8912


def credencial():
    """Le ~/.site_auth sem ecoar o valor."""
    for linha in CRED.read_text(encoding="utf-8").splitlines():
        linha = linha.strip()
        if linha.startswith("SITE_AUTH="):
            v = linha.split("=", 1)[1].strip().strip("'\"")
            return base64.b64encode(v.encode()).decode()
    raise SystemExit("SITE_AUTH nao encontrado em ~/.site_auth")


AUTH = credencial()


class Proxy(http.server.BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_GET(self):
        req = urllib.request.Request(ORIGEM + self.path)
        req.add_header("Authorization", "Basic " + AUTH)
        req.add_header("User-Agent",
                       "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                       "(KHTML, like Gecko) Chrome/140.0 Safari/537.36")
        try:
            with urllib.request.urlopen(req, timeout=20) as r:
                corpo = r.read()
                self.send_response(r.status)
                self.send_header("Content-Type",
                                 r.headers.get("Content-Type", "text/html"))
                self.send_header("Content-Length", str(len(corpo)))
                self.end_headers()
                self.wfile.write(corpo)
        except urllib.error.HTTPError as e:
            corpo = e.read()
            self.send_response(e.code)
            self.send_header("Content-Length", str(len(corpo)))
            self.end_headers()
            self.wfile.write(corpo)

Four details in that code carry the weight:

• The credential is read from a file at startup and never printed, so the value does not land in shell history or in a process listing.
• base64.b64encode of user:pass is the entire content of a Basic header. There is no hashing step and nothing to negotiate.
• The origin status code and Content-Type are forwarded, so a 404 through the proxy is still a 404 and a PDF arrives as a PDF. A proxy that renders everything as 200 text/html hides the bug you were testing for.
• socketserver.ThreadingTCPServer binds to 127.0.0.1 only. A credential-injecting proxy listening on 0.0.0.0 is an open door with your password on the handle.

With it running, the same page returns:

curl -s -o /dev/null -w '%{http_code} %{content_type} %{size_download}\n' http://127.0.0.1:8912/
# 200 text/html 10013

What a 200 through the proxy actually proves

It proves the transport works. It says nothing about the page.

The publication route of that SPA answers 200 with 3789 bytes, and those 3789 bytes contain zero references to the media the publication claims to have. The HTML is a shell; publicacao.js fetches the JSON afterwards and injects the images into the DOM. A curl of an SPA route can look perfectly healthy while the page renders an error state in the browser.

So the browser is not ceremony here, it is the only observer that sees what the user sees. The assertions have to live on the DOM: count the media nodes, read naturalWidth on each image, click the zoom button and confirm the dialog opened. That is what the proxy unlocks, and it is the reason to keep the proxy in the loop instead of stopping at curl.

Lazy images report a width of zero

One trap makes honest DOM assertions fail for the wrong reason. loading="lazy" on an image below the fold means the browser has not decoded it, and naturalWidth reads 0 until the image enters the viewport. A test that checks width without scrolling first reports every image after the first screen as broken.

Fix it in the test, before measuring:

for (const img of document.querySelectorAll('.media-item img')) {
  img.loading = 'eager';
  await new Promise(resolve => img.complete || img.addEventListener('load', resolve, {once: true}));
}

The same rule applies to the network waterfall: a lazily loaded asset produces no entry in performance timing until you scroll to it, so "no request was made" and "the request happens later" look identical from a distance.

Stop the proxy when the proof is in

The proxy holds the connection open while it waits on the origin, and it holds your automation with it. In my case the browser call in front of the proxied page ran until timeout and returned nothing, twice, before I noticed the proxy was the common factor. It is a diagnostic tool for one page at a time, not part of the test harness.

The order that keeps you honest:

1. curl first for the status, the content type and the byte count. That is the cheap proof, and it needs no browser.
2. Start the proxy, load the page, assert on the DOM, capture what you need.
3. Kill the proxy as soon as the assertions pass, then rerun the same check against the origin to confirm the failure you were investigating still reproduces.

If the protected environment is yours, there are cheaper options worth weighing first. A session cookie set by the test itself, or a --user-data-dir profile you log into once by hand, avoids the proxy entirely and keeps the browser's own auth state. The proxy earns its place when the environment is not yours to configure and the credential exists only as a static header.

Um detalhe que vale para quem trabalha com isso em português: o mesmo proxy serve para conferir se o texto publicado chegou íntegro no servidor, e foi assim que descobri que uma publicação estava exibindo o caminho de um arquivo no lugar do corpo. Se você já perdeu tempo com uma página que abre no navegador comum e morre no headless, conta nos comentários como resolveu.