DEV Community: Jonas Francisco de Lima Filho

My screenshot was 1080x1350. The page was 1722 pixels tall.

Jonas Francisco de Lima Filho — Mon, 09 Nov 2026 20:07:20 +0000

A piece of artwork went live last month without the contact line at the bottom. Phone, handle and site were in the HTML, the CSS placed them, the browser laid them out, and the file that got published did not contain them. Three checks ran before publishing, and all three passed.

What the check asked

The pipeline was short: render a local HTML file, screenshot it, verify it. Verification asked two questions. Is the image 1080 by 1350? It is. Does each element sit inside its own box, with no overlap and no text against the edge? Yes.

Nobody asked whether the file contained the whole page. The image had the dimensions from the request, and the layout assertions had run against a live DOM whose document was 1722 pixels tall.

A screenshot request describes a rectangle

I rebuilt the case with a minimal page: one flex column, 1080 wide, 90 pixels of lateral padding, nine paragraphs and a footer with a 3px rule above it. The design height was 1350. The content did not fit, and a flex column grows instead of overflowing, so the document ended at 1722. Nothing in the CSS was wrong. A page can be taller than the box you drew for it.

Measuring before capturing:

JSON.stringify({
  doc: document.documentElement.scrollHeight,
  viewport: window.innerHeight,
  footer: Math.round(document.querySelector('.rodape').getBoundingClientRect().bottom)
})

Output on that page:

{"doc":1722,"viewport":1350,"footer":1652}

The class is rodape, footer in Portuguese, because the test page came out of a Portuguese pipeline.

The footer bottom sits at 1652. Its rule, the three pixels that separate contact data from the body, sits at y=1590. The document was 372 pixels taller than the box in the request.

Two files with the right size and the same missing block

chrome --headless=new --screenshot --window-size=1080,1350 produced a 1080 by 1350 PNG. Last row containing ink: y=1262. Everything from 1262 down is white, 87 pixels of it, which reads as bottom padding. The rule at 1590 is absent, the footer is absent.

The same page captured over CDP at exactly 1080 by 1350 produced another 1080 by 1350 PNG. Last row with ink: 1349, zero white. The footer is absent too, and this file carries no visual hint of loss, because the crop lands in the middle of a paragraph and the image looks full bleed.

Two files, identical dimensions, identical gap, different reasons to believe nothing is wrong. The first looks like a design with breathing room. The second looks finished.

The size check was comparing the request to itself

The verification read the PNG dimensions and compared them to the constant it had passed in. Both numbers came from the same place, so the check could only fail if the image library were broken. The layout assertions had a similar problem: they read getBoundingClientRect() from a 1722 pixel document and confirmed each box sat where the CSS said. They were right about the layout and silent about the file.

A check is worth something when its reference comes from outside the thing being checked. Here the reference had to be the document height, read after layout and before capture.

Capture at the height you measured

Two changes fixed it, and neither is about the tool.

First, read scrollHeight after the load and pass that number to the capture. Second, assert that the element you care about is inside the captured box, using its own coordinates:

altura = doc["doc"]
assert doc["footer"] <= altura, f"footer fora da captura: {doc['footer']} > {altura}"

With 1722 passed in, the full page came back: last row with ink 1650, 71 pixels of white under it, the 3px rule at 1590, contact line present. The same request through the CLI at 1722 also worked, which puts the bug in the number rather than in the binary.

That detail matters for the diagnosis. I had blamed the tool for months.

What CDP adds, and the two details that bite

Emulation.setDeviceMetricsOverride fixes the viewport at the exact size of the piece before navigation, so the page lays out against the box you intend instead of the default window. Page.captureScreenshot with captureBeyondViewport: true returns the whole document, including what sits under the fold.

ws_envia(s, {"id": 2, "method": "Emulation.setDeviceMetricsOverride",
             "params": {"width": w, "height": h,
                        "deviceScaleFactor": 1, "mobile": False}})
ws_envia(s, {"id": 4, "method": "Page.captureScreenshot",
             "params": {"format": "png", "captureBeyondViewport": True}})

Because the capture can exceed the requested size, my script crops it back:

if (aw, ah) != (w, h):
    im.crop((0, 0, w, h)).save(png)
    print(f"recortado de {aw}x{ah} para {w}x{h}")

Today that line printed recortado de 1080x1722 para 1080x1350. It is a failure message wearing an informational tone, and the crop it reports is exactly what removed the footer. A shrink deserves an exception, not a print.

The second detail is the debugging port. Two captures on the same port collide, because the previous Chrome has not released the socket yet (TIME_WAIT) and the second connection fails with an error that says nothing about ports. Deriving the port from the pid removed it: porta = 9401 + (os.getpid() % 400).

One more, measured twice today: --screenshot clips at the window height instead of stretching the image. My own script's docstring said it stretched. Rulers drawn at y=200, 600 and 1000 landed on those exact rows in both tools, and a ruler at 1800 did not appear at all in the CLI output, which is a crop and not a rescale. The fix worked while the explanation was wrong, and that is a reason to re-measure your own comments.

The shape of this bug

Every version of it looks the same: an artifact carries the coordinate system of the request, and the check reads coordinates from the request. Image dimension against requested dimension. Body length against expected length. Row count against the count in the config.

The cheapest defence is one independent reading per artifact, taken from the source that produced it: scrollHeight for a page, the element rectangle for a component, the actual byte length for a payload. It costs a line of code and it fails loudly, which is what the three green checks were missing.

Measured on Chromium 145.0.7632.0, headless=new, one machine, November 2026. The clipping behaviour of --screenshot has changed between versions, so pin your own numbers before quoting mine.

Se o seu pipeline também publica imagem sem conferir se o rodapé entrou no arquivo, comenta aqui embaixo o que a sua verificação mede. Aprendo mais com o caso dos outros do que com o meu, e respondo todo mundo.

Half of my corrupted text was pure ASCII, and my character check passed it

Jonas Francisco de Lima Filho — Fri, 06 Nov 2026 20:29:13 +0000

I generate long documents as files and publish them with a script: reports, proposals, article bodies. The publisher never reads the sentences. It checks that the media paths exist and that the body is not empty, then puts the page live.

That gap is where two different text corruptions survived for weeks. One of them is invisible to the check most people write first.

The defect that looks like damage

The first time, a file came back with words fused together. marinhoprofundo where two words belonged, branco,.grid where a table header used to be. In the same document, a stretch of characters from no Latin alphabet sat in the middle of a sentence.

This form announces itself. You scan the file, see the damage, and fix the source.

The cause was not the publisher. It was the tool that wrote the file. Long text with accents and punctuation passes through it in chunks, and occasionally a chunk returns with pieces glued or replaced. By then the publisher had already accepted the file, the validator had passed it, and the card was live with the broken text.

The defect that looks clean

The second form does not look like damage, because every byte in it is seven-bit ASCII.

I found three cases in my own documents. A list item read agendamento, CRM,|Issue tracker: a comma jammed against a pipe, and an English label right after it. Another line read Três Interactions, a Portuguese word followed by the English one. A third read e é priced como tal, an English verb inside a Portuguese sentence.

Run this against those three lines:

import re
re.findall(r"[^\x00-\x7F]", texto)

You get an empty list. Nothing sits outside the ASCII range, so any check built on unusual characters returns clean on a document a reader will notice in the first paragraph. The defect is lexical, and no character class catches a wrong word.

Nothing downstream complains

The publishing CLI does not parse prose. My validator compares the manifest against the publication JSON, which means it compares media counts, body length and ids. Both sides agreed, because both were describing structure.

A structural check cannot see that a sentence contains the wrong language. It can only tell you the sentence exists.

That is why the gate has to inspect the text itself, in the step before publishing, and not the record after it.

Three checks in one file

The gate I use is a single Python file, run before the publish command. It exits 1 on any defect, so it chains with && and the publish never runs on a bad source.

These are the checks, abridged from the real file:

OK = set("áàâãäéêíóôõöúüçÁÀÂÃÄÉÊÍÓÔÕÖÚÜÇ…")
ESTRANGEIRAS = re.compile(r"\b(issue|tracker|interactions|priced|checklist|starter)\b", re.I)
PIPE_PROSA = re.compile(r"\S\|\S")

t = pathlib.Path(sys.argv[1]).read_text()
prosa = re.sub(r"```

.*?

```", " ", t, flags=re.S)
defeitos = []

estranhos = sorted({c for c in re.findall(r"[^\x00-\x7F]", t) if c not in OK})
if estranhos:
    defeitos.append(f"caracteres corrompidos: {estranhos}")

if ESTRANGEIRAS.search(prosa):
    defeitos.append("palavra de outro idioma no meio da prosa")

if PIPE_PROSA.search(prosa) and not _tem_tabela(prosa):
    defeitos.append("pipe no meio da prosa")

The first check handles the visible form. It collects every non-ASCII character and compares against the set of accented letters and punctuation that belong in Portuguese.

The second check handles the invisible form. It is a short list of English words that showed up glued into my prose. The list stays short on purpose. Terms that Brazilian tech writing uses legitimately, such as bot, lead, setup, dashboard, status and deploy, are not on it. A gate that cries wolf gets ignored, and the ignoring generalizes to the checks that work.

The third check uses a punctuation mark as evidence. A pipe is a table separator in markdown and never appears inside running prose. When one appeared mid-sentence, it marked the exact span where the text had been spliced.

Two guards that keep the gate trusted

A checker is only useful while you believe its output, so both guards here exist to remove false positives.

The first strips fenced code blocks before the prose checks. An ffmpeg filter containing adelay=0|0 is correct syntax and failed the pipe check, which taught me to skim past failures. The character check still runs over the entire text, code included, because corruption inside a code block is a defect too.

The second detects a legitimate markdown table by looking for a separator line such as ---|--- under a line with pipes. Without it, a real table with pipes in its cells fails the gate for being what it is.

The allowed character set was tuned the same way. A correct text containing ± and → failed as corrupted on a measurement article about color. Those symbols have no place in a Portuguese document, so the fix was to write "35 degrees to each side" and a colon instead of an arrow. When a checker flags correct text, either the checker changes or the style does, and pretending it was fine is how the whole gate loses credibility.

Where the damage is born

The gate runs on the source file, before publishing. Correcting the JSON afterwards costs a second deploy, and until that deploy the card sits live with the wrong text. One full cycle of that was enough to make the ordering permanent.

Two practices came out of the same class of defect, where the wrong value is plausible instead of obviously wrong.

Tables with numbers that leave the house are built from data. I write the rows as a list of tuples, derive the total with sum(), and assert it against the number typed into the prose. When the assert fails, the column is wrong, and I learn it on my machine instead of in a client conversation.

Documents that exist in two formats come from one object. A markdown body and a PDF generated from the same structure stay in sync by construction. Two hand-written versions diverge in an accent or a figure, both look readable, and they reach different recipients.

Para quem publica em português

Se você gera texto longo com ferramenta e publica por script, um gate de alguns segundos vale mais que reler o arquivo uma vez. A releitura acontece quando você lembra, o gate acontece em toda publicação, e ele acusa o defeito antes de o card ir ao ar. Me conta nos comentários qual foi a corrupção mais estranha que já passou pelo seu pipeline, porque esse tipo de caso raramente aparece documentado.

Six still PNGs, six narration clips, and the ffmpeg pass that keeps them in sync

Jonas Francisco de Lima Filho — Wed, 04 Nov 2026 20:23:39 +0000

Six still PNGs, six narration clips, and the ffmpeg pass that keeps them in sync

A vertical video with narration usually means a timeline editor and someone dragging clips until the words land on the right frame. I had an SSH session on a two-core ARM machine, six PNGs exported at 1080x1920, and ffmpeg.

The result is a 47.9 second reel: six typographic frames, each with one sentence of Portuguese narration. The pipeline is 103 lines of Python calling ffmpeg as a subprocess, with no dependencies beyond ffmpeg, ffprobe and a TTS CLI. Here is what it does, and which decisions the machine forced.

The frames were designed as still images

Every scene is a static frame rendered elsewhere into a PNG at exactly 1080x1920, 8-bit RGB. No animation, no zoom. On a phone held at arm's length, text that moves has to be chased, and each of these frames holds a sentence that the narration finishes in about seven seconds.

That removes most of the usual complexity: no timeline, no transitions, no partial renders. A scene is one image that stays for a fixed number of seconds, and the only open question is how many.

The hardware mattered. Two cores, aarch64, no GPU. libx264 at veryfast is fine for a slideshow and would not be fine for anything with motion. I measured one segment while writing this: 8.44 seconds of video, 211 frames, 5.8 seconds of wall clock, 38 KB on disk and 32 kb/s, because a still frame has nothing to spend bits on.

The narration sets the timing

The obvious approach is to fix five seconds per scene and squeeze the audio to fit, which makes speech sound rushed in one place and padded in another. Doing it the other way is easier: synthesize the narration first, one MP3 per scene, then ask ffprobe how long each file is. That number is the timeline.

BASE = Path("/home/ubuntu/hermes_tools/pecas_plano_fde")
FPS = 25
VOZ = "pt-BR-AntonioNeural"
EDGE = "/home/ubuntu/hermes_tools/voz/venv/bin/edge-tts"

def dur(caminho):
    saida = subprocess.run(
        ["ffprobe", "-v", "error", "-show_entries", "format=duration",
         "-of", "default=noprint_wrappers=1:nokey=1", str(caminho)],
        capture_output=True, text=True, check=True).stdout.strip()
    return float(saida)

The six clips measured 7.944, 7.080, 6.888, 7.392, 7.128 and 8.472 seconds: 98 words in 44.904 seconds of speech, about 131 words per minute in Portuguese at the default rate. Nothing there was estimated. If a sentence grows, its scene grows, and no other scene compensates.

The narration comes from the edge-tts CLI in a virtualenv, voice pt-BR-AntonioNeural, rate +0%. Separate files instead of one long track is what keeps the rest simple: each one can be measured, replaced and reused on its own.

Each frame becomes a clip of its own length

for i, d in enumerate(duracoes, start=1):
    png = BASE / f"reel-cena{i}-1080x1920.png"
    seg = destino / f"seg{i}.mp4"
    seg_dur = round(d + 0.5, 2)
    subprocess.run([
        "ffmpeg", "-y", "-loop", "1", "-t", str(seg_dur), "-i", str(png),
        "-vf", "scale=1080:1920:flags=lanczos,format=yuv420p",
        "-r", str(FPS), "-c:v", "libx264", "-preset", "veryfast", "-crf", "20",
        "-pix_fmt", "yuv420p", str(seg)
    ], check=True, capture_output=True)

scale with lanczos does nothing on an input already at 1080x1920, and stays in the chain so it survives a frame exported at another size. format=yuv420p is the pixel format phones decode without argument, and it has to reach the final file. -crf 20 with veryfast is a compromise for two cores; on a machine with headroom, medium would shave bytes nobody would notice.

The + 0.5 is breathing room. Without it the next sentence lands on the exact frame where the previous one ended, and the cut feels clipped even with correct timings.

Padding the audio so each line stays with its frame

Concatenating the six MP3s gives 44.9 seconds while the video is 47.9. Muxing those directly means the audio falls behind, and by the fifth scene the voice describes something that left the screen seconds earlier. Padding each narration to the duration of its own segment removes the accumulation.

entradas, filtros = [], []
for i, (_, seg_dur) in enumerate(segmentos, start=1):
    entradas += ["-i", str(destino / f"narracao{i}.mp3")]
    filtros.append(f"[{i - 1}:a]adelay=0|0,apad=whole_dur={seg_dur}[a{i - 1}]")
concat_in = "".join(f"[a{i}]" for i in range(len(segmentos)))
filtros.append(f"{concat_in}concat=n={len(segmentos)}:v=0:a=1[aout]")

apad=whole_dur=N extends each stream with silence to N seconds, the number computed in the earlier loop. The adelay filter at the front of the chain carries a zero delay and changes nothing audible; it stays there for scenes that need a lead-in before the voice starts. For padding alone, apad is enough.

subprocess.run(["ffmpeg", "-y", "-i", str(video), "-i", str(audio),
                "-map", "0:v:0", "-map", "1:a:0", "-c:v", "copy", "-c:a", "aac",
                "-shortest", "-movflags", "+faststart", str(final)], check=True)

-c:v copy is why this step costs almost nothing: the segments were already encoded and joined by the concat demuxer, so the muxer only interleaves streams. +faststart moves the index to the front of the file, which matters when it is played from a phone over a network.

What the container reports

video: h264, 1080x1920, 25/1 fps, 1195 frames
audio: aac, 24 kHz, mono
duration: 47.9 seconds, reported the same for both streams

The arithmetic matches the file: 44.904 seconds of speech plus six half-second pads is 47.904. The audio stays at 24 kHz mono because that is what the TTS produced; resampling to 44.1 kHz would add bytes and no information, and Instagram re-encodes it anyway.

What that probe answers is whether video and audio agree on duration. When they differ, either a segment was written at a different length than the one used for its pad, or the concat dropped frames at a join. Both show up before anything is uploaded, and neither is visible while watching the file on a laptop at half size.

The silence you hear is longer than the padding you set

I ran a two-scene version of the pipeline while writing and measured the join with silencedetect at -45 dB and 0.25 seconds. The padding was 0.5 seconds and the measured silence at the cut was 1.52 seconds.

The extra comes from the narration files. TTS output carries a little silence at its own start and end, and apad adds on top of it. The same run flagged pauses of about 0.9 seconds inside the speech, so the clips have internal gaps of their own. What you set is a constant; what a listener perceives is that constant plus the edges of two files.

Padding therefore has to be tuned by measuring the join. Half a second worked for six scenes of about seven seconds, where the sentence still holds attention. For a fast cut it would be too much.

What I would change

The docstring of the script says 0.7 seconds of padding. The loop uses 0.5. Nothing broke, but the comment is already wrong, and comments drift faster than the code they describe.

I would also write the durations to a JSON file before encoding instead of recomputing them in the same pass. The script is one long function, so an encode that fails halfway re-synthesizes every narration clip when it restarts, and synthesis is the slow step that touches the network. Splitting them turns a failed render into a five-second retry.

The rest holds. Six frames, six clips, one filtergraph and a format check at the end, with no GUI, no GPU and no cloud render service.

Se esse tipo de detalhe ajuda no seu projeto, deixa um comentário contando o que você já quebrou tentando sincronizar áudio com imagem.

Bisecting a request body to find why the API returned 403

Jonas Francisco de Lima Filho — Mon, 02 Nov 2026 20:05:25 +0000

I publish articles here from a small script, so the draft exists before I ever open the editor. Last week that script got a 403 back and no article. The same token read the account one second earlier, and the same request went through afterwards with a body that said less. The fix came from splitting the article in half, not from rewriting it.

A 403 with an HTML body is not an authentication problem

The response body started with <!DOCTYPE html> and contained the words "Attention Required!". That is not the shape of this API's errors. Rejected calls come back as JSON, with a message field, a status code and something you can act on. An HTML page in the body means the request stopped in front of the application.

Two more facts from the same minute: reading the article list with the same credentials returned 200 and a full payload, and the failing call had been working for weeks. The token was valid, the route existed, and the decision was made somewhere else.

The habit is cheap to build. On any 4xx, log three things before changing a single character of the input: the status code, the content-type header and the first 200 bytes of the body. When you see HTML, you are no longer debugging your client.

Rule out size with one control payload

The obvious suspect was the size of the request. The article was about 9 KB of markdown, the earlier ones were smaller, and a 403 without a message invites superstition.

So I sent a body of the same order with content I did not care about: repeated paragraphs of plain text, no code, no links, about 8.4 KB, on a draft. It returned an id. The real article failed at roughly 9 KB. That result is weak on its own, because 8.4 and 9 are close enough to sit on either side of a limit.

The bisect settled it. One of the four parts was 2 KB and still failed, so no size threshold explains the pattern. The variable had to be content.

Split by paragraph, and keep the wrapper

The script that did the split is short. It cuts the body into four blocks of whole paragraphs, leaves the markdown intact and writes one payload per part:

import json
import pathlib
import sys


def partes(texto, n=4):
    """Divide em n blocos de paragrafos, sem cortar frase no meio."""
    paras = [p for p in texto.split("\n\n") if p.strip()]
    total = sum(len(p) for p in paras)
    alvo, atual, saida, bloco = total / n, 0, [], []
    for p in paras:
        bloco.append(p)
        atual += len(p)
        if atual >= alvo and len(saida) < n - 1:
            saida.append("\n\n".join(bloco))
            bloco, atual = [], 0
    if bloco:
        saida.append("\n\n".join(bloco))
    return saida


corpo = pathlib.Path(sys.argv[1]).read_text(encoding="utf-8")
for i, bloco in enumerate(partes(corpo), 1):
    destino = pathlib.Path(f"parte-{i:02d}.json")
    destino.write_text(
        json.dumps({"article": {"body_markdown": bloco}}), encoding="utf-8"
    )
    print(destino.name, len(bloco.encode("utf-8")), "bytes")

Each part goes out as its own update to the same article id. That means the draft ends up holding whatever the last successful part wrote, which is harmless on a draft and is the reason to run this test on a draft instead of on something already published.

Part two failed, parts one, three and four passed, and the failing part was the only one with code in it.

The trigger was one inline command

I split part two again, then the first half of part two, and ended up with a single paragraph. It contained an inline code span: a shell request line, followed by a URL in backticks, in the shape of something a reader could paste and run. Replacing that construction with a plain sentence describing the same request made the whole article post on the first attempt.

What I can state with confidence is the experiment, not the reason. Two bodies differed by that one construction and only one was accepted. I cannot see the rule that scored it, and there is a decent motive behind such a rule, since a body full of paste-ready commands with live URLs is exactly what a spammer would send. The article still needed to describe a real request, and the sentence version did that without shipping the command.

This is where technical prose and content filters disagree. Command blocks are what makes a post useful to the next person. The compromise that worked: keep commands inside fenced blocks, keep a URL out of the same inline span as the command name, and test on a draft where a scrambled body costs nothing.

The wrapper is part of the request

Once I dropped to the raw HTTP route to send parts, each payload has to carry the object explicitly:

{"article": {"body_markdown": "..."}}

Sending a bare object returns 422 with "article param must be a JSON object". The integration tool I normally use builds that wrapper itself, and its schema output does not show it, which is why the requirement only appears when you bypass the tool. Two layers, and forgetting the outer one produces an error that reads like a problem with your data.

The field the tool never sends

The tool that updates an article accepts nine keys. A tenth, ai_disclosure_level, is refused with Unknown key. The route accepts it, with four values: not_disclosed, no_ai, some_ai and fully_autonomous.

That matters because the platform asks authors to disclose machine involvement, and a tool schema that cannot express the field makes compliance look impossible. It is not. One PUT to the article with that single field is enough.

The check worth keeping: a 200 does not prove the field was stored. Read the article back and confirm two values in the response, ai_disclosure_level and ai_disclosure_label. On the most recent post here the pair came back as some_ai and "AI-assisted". A response that shows not_disclosed means the call succeeded and wrote nothing, which is the failure mode to watch for, because the status code looks identical.

The checklist I keep now

On a 4xx, read the content type and the first 200 bytes before touching the input.
Send a control payload of similar size with content you know is harmless, before rewriting anything.
Bisect by paragraph, with the wrapper on every part, on a draft.
Read back every field you set. The status code confirms transmission, not storage.
There is no delete for articles on this API, so a draft created by mistake stays on the account. Reuse its id with an update instead of creating a second one.

None of this is specific to one platform. Any proxy that inspects request bodies can reject a payload for what it says, and the diagnosis repeats: exclude the boring variables first, then halve the input until the failing piece is small enough to read.

Um recado curto para quem escreve em português: o método serve igual, e a parte que mais economiza tempo é ler o status e o tipo de conteúdo antes de mexer no texto. Se você já perdeu uma tarde com um erro que parecia autenticação e não era, conta nos comentários o que ele era.

Headless Chromium gives up on HTTP basic auth. A local proxy injects the header instead.

Jonas Francisco de Lima Filho — Sat, 31 Oct 2026 02:11:33 +0000

A dashboard I run sits behind nginx basic auth. The page is a small SPA: an HTML shell, then JavaScript that fetches a JSON index and builds the feed. I wanted a real browser to load it, because a screenshot of the rendered page catches layout problems that reading the JSON never will.

The first attempt produced no image at all. Not a blank page, not a login dialog.

chromium --headless=new --disable-gpu --no-sandbox \
  --window-size=800,600 --screenshot=page.png http://127.0.0.1/

Page load failed: net::ERR_INVALID_AUTH_CREDENTIALS
ls: cannot access 'page.png': No such file or directory

A plain curl against the same address answers 401 with the body of the challenge, so the origin is doing exactly what it should:

HTTP/1.1 401 Unauthorized
Server: nginx/1.24.0 (Ubuntu)
WWW-Authenticate: Basic realm="Restricted area"
Cache-Control: no-cache, no-store, must-revalidate, max-age=0

The server offers a realm and waits for an answer. Headless has no window in which to show the prompt, so the navigation ends as a failed load instead of as a page displaying a login box. The error name makes it worse than it is: ERR_INVALID_AUTH_CREDENTIALS suggests bad credentials were sent, and none were sent at all.

Two failures wear the same log line

Before reaching for a workaround, it helps to know which of two very different things you are looking at. Same URL, same moment, only the User-Agent differs:

curl -s -o /dev/null -w '%{http_code}\n' -A "Python-urllib/3.14" https://example.test/
# 403
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/140.0" https://example.test/
# 401

403 means something in front of the application refused the request, and on Cloudflare-fronted hosts a bare script User-Agent is enough to earn it. 401 means the origin received the request and asked for credentials. A script that only prints "fetch failed" cannot tell these apart, and guessing costs an afternoon: you go looking for the wrong password while a WAF is answering.

Log three things on every failed request and the ambiguity disappears: the status code, the Server header, and the User-Agent you sent. The proxy below sends a browser User-Agent for the same reason.

Read the credential from a file, inject the header locally

The fix that worked was to stop teaching the browser about authentication and put a small HTTP proxy in front of the origin. The proxy adds the Authorization header itself, so Chromium talks to a server that never asks for anything. This is the relevant slice of a 90 line script, with its original names:

import base64
import http.server
import pathlib
import socketserver
import urllib.error
import urllib.request

ORIGEM = "http://127.0.0.1:80"
CRED = pathlib.Path.home() / ".site_auth"
PORT = 8912


def credencial():
    """Le ~/.site_auth sem ecoar o valor."""
    for linha in CRED.read_text(encoding="utf-8").splitlines():
        linha = linha.strip()
        if linha.startswith("SITE_AUTH="):
            v = linha.split("=", 1)[1].strip().strip("'\"")
            return base64.b64encode(v.encode()).decode()
    raise SystemExit("SITE_AUTH nao encontrado em ~/.site_auth")


AUTH = credencial()


class Proxy(http.server.BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_GET(self):
        req = urllib.request.Request(ORIGEM + self.path)
        req.add_header("Authorization", "Basic " + AUTH)
        req.add_header("User-Agent",
                       "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                       "(KHTML, like Gecko) Chrome/140.0 Safari/537.36")
        try:
            with urllib.request.urlopen(req, timeout=20) as r:
                corpo = r.read()
                self.send_response(r.status)
                self.send_header("Content-Type",
                                 r.headers.get("Content-Type", "text/html"))
                self.send_header("Content-Length", str(len(corpo)))
                self.end_headers()
                self.wfile.write(corpo)
        except urllib.error.HTTPError as e:
            corpo = e.read()
            self.send_response(e.code)
            self.send_header("Content-Length", str(len(corpo)))
            self.end_headers()
            self.wfile.write(corpo)

Four details in that code carry the weight:

The credential is read from a file at startup and never printed, so the value does not land in shell history or in a process listing.
base64.b64encode of user:pass is the entire content of a Basic header. There is no hashing step and nothing to negotiate.
The origin status code and Content-Type are forwarded, so a 404 through the proxy is still a 404 and a PDF arrives as a PDF. A proxy that renders everything as 200 text/html hides the bug you were testing for.
socketserver.ThreadingTCPServer binds to 127.0.0.1 only. A credential-injecting proxy listening on 0.0.0.0 is an open door with your password on the handle.

With it running, the same page returns:

curl -s -o /dev/null -w '%{http_code} %{content_type} %{size_download}\n' http://127.0.0.1:8912/
# 200 text/html 10013

What a 200 through the proxy actually proves

It proves the transport works. It says nothing about the page.

The publication route of that SPA answers 200 with 3789 bytes, and those 3789 bytes contain zero references to the media the publication claims to have. The HTML is a shell; publicacao.js fetches the JSON afterwards and injects the images into the DOM. A curl of an SPA route can look perfectly healthy while the page renders an error state in the browser.

So the browser is not ceremony here, it is the only observer that sees what the user sees. The assertions have to live on the DOM: count the media nodes, read naturalWidth on each image, click the zoom button and confirm the dialog opened. That is what the proxy unlocks, and it is the reason to keep the proxy in the loop instead of stopping at curl.

Lazy images report a width of zero

One trap makes honest DOM assertions fail for the wrong reason. loading="lazy" on an image below the fold means the browser has not decoded it, and naturalWidth reads 0 until the image enters the viewport. A test that checks width without scrolling first reports every image after the first screen as broken.

Fix it in the test, before measuring:

for (const img of document.querySelectorAll('.media-item img')) {
  img.loading = 'eager';
  await new Promise(resolve => img.complete || img.addEventListener('load', resolve, {once: true}));
}

The same rule applies to the network waterfall: a lazily loaded asset produces no entry in performance timing until you scroll to it, so "no request was made" and "the request happens later" look identical from a distance.

Stop the proxy when the proof is in

The proxy holds the connection open while it waits on the origin, and it holds your automation with it. In my case the browser call in front of the proxied page ran until timeout and returned nothing, twice, before I noticed the proxy was the common factor. It is a diagnostic tool for one page at a time, not part of the test harness.

The order that keeps you honest:

curl first for the status, the content type and the byte count. That is the cheap proof, and it needs no browser.
Start the proxy, load the page, assert on the DOM, capture what you need.
Kill the proxy as soon as the assertions pass, then rerun the same check against the origin to confirm the failure you were investigating still reproduces.

If the protected environment is yours, there are cheaper options worth weighing first. A session cookie set by the test itself, or a --user-data-dir profile you log into once by hand, avoids the proxy entirely and keeps the browser's own auth state. The proxy earns its place when the environment is not yours to configure and the credential exists only as a static header.

Um detalhe que vale para quem trabalha com isso em português: o mesmo proxy serve para conferir se o texto publicado chegou íntegro no servidor, e foi assim que descobri que uma publicação estava exibindo o caminho de um arquivo no lugar do corpo. Se você já perdeu tempo com uma página que abre no navegador comum e morre no headless, conta nos comentários como resolveu.

JSON ids above 2^53 arrive rounded, and a reviver only fixes the leaf

Jonas Francisco de Lima Filho — Wed, 28 Oct 2026 20:09:15 +0000

A JSON integer above 2^53 does not survive a round trip through JSON.parse. It comes back as a different number, with the same type, no exception and no warning. A record that matched a row in the database stops matching, and the code consuming it has no signal that anything moved.

The measurements below ran on Node v26.12.1. They locate the boundary, show where a JSON.parse reviver helps, and show the two places where it does not.

The smallest case that fails

const parsed = JSON.parse('{"id": 9007199254740993}').id;

console.log(parsed);                                  // 9007199254740992
console.log(typeof parsed);                           // "number"
console.log(String(parsed) === "9007199254740993");   // false

No exception, no deprecation warning, no field flagged as suspicious. The value is off by one and the type is still number, so every typeof check in the codebase keeps approving it. The same literal written directly in source code is also already wrong before it reaches JSON:

console.log(String(9007199254740993)); // "9007199254740992"

2^53 is the boundary, and above it the step is 2

A JavaScript number is an IEEE 754 double with 53 bits of mantissa. The largest integer that survives a round trip is Number.MAX_SAFE_INTEGER, which is 9007199254740991. The next power of two, 9007199254740992, is still exactly representable, but it is no longer safe:

console.log(Number.MAX_SAFE_INTEGER);        // 9007199254740991
console.log(2 ** 53);                        // 9007199254740992
console.log(Number.isSafeInteger(2 ** 53));  // false

Above 2^53 the distance between two representable numbers is 2, so even integers are exact and odd integers are not. An odd id lands on a neighbour, and the parser has no way to know it did.

I generated 1000 random integers above 2^53 and compared each with the value returned by JSON.parse. Two runs:

run 1: 505 unchanged, 495 changed, and 505 of the samples were even while 495 were odd;
run 2: 500 unchanged, 500 changed, with the same even and odd counts.

The parity match is the mechanism showing itself. Even input stays, odd input moves. The "changed" number in a report should not be read as a fixed percentage of your data, because it depends on whether your ids are even or odd.

Two different ids can become one key

Rounded values do not only disagree with the database. They also collide with each other, and a collision inside a Set or a Map merges records that the source system kept separate.

const a = JSON.parse('{"id": 9007199254740992}').id;
const b = JSON.parse('{"id": 9007199254740993}').id;

console.log(a === b); // true

I sent 1000 consecutive integers starting at 2^53 and counted how many distinct values survived:

1000 consecutive ids above 2^53 produced 501 distinct values, so 499 keys were lost;
1000 consecutive ids below 2^53 produced 1000 distinct values.

Deduplication, Map lookups and "have I seen this record" checks all assume one id per record. Above 2^53 that assumption is untrue, and nothing in the standard library objects.

A reviver fixes the leaf, not the container

Modern V8 passes a third argument to the reviver function: a context object whose source holds the original text of the value being revived. That text has not been through the double, so it can be turned into a BigInt without loss.

const exato = JSON.parse('{"id": 9007199254740993}', (chave, valor, contexto) =>
  chave === "id" && contexto && contexto.source ? BigInt(contexto.source) : valor
);

console.log(String(exato.id)); // "9007199254740993"

What matters is what the reviver receives. I logged every call for one payload:

JSON.parse('{"id": 9007199254740993, "ids": [9007199254740992, 9007199254740993]}',
  (k, v, ctx) => { console.log(k, typeof v, ctx && ctx.source); return v; });

The output, in order:

key id, type number, source "9007199254740993";
key 0, type number, source "9007199254740992";
key 1, type number, source "9007199254740993";
key ids, type object, no source;
key "" (the root object), type object, no source.

Leaves carry the raw text, including elements inside an array. Containers do not. That ordering is why a reviver written at the wrong level looks like it runs and does nothing.

A reviver placed at the array level shows the trap:

// wrong level: v is already [9007199254740992, 9007199254740992]
const tarde = JSON.parse('{"ids": [9007199254740992, 9007199254740993]}', (k, v) =>
  Array.isArray(v) ? v.map(String) : v
);

console.log(tarde.ids); // ["9007199254740992", "9007199254740992"]

The strings are produced, the two values are identical, and the bug survives with a cast applied to it. The children are revived before the parent, so the fix has to sit where the primitive still carries its source: k === "id", or typeof v === "number" && ctx.source.length > 15, or an explicit list of keys you know are identifiers.

What the reviver does not fix

Three limits came out of the same measurement, and they decide whether the reviver is worth using at all.

BigInt breaks the way back. A BigInt id cannot go through JSON.stringify without help. JSON.stringify({id: 9007199254740993n}) throws TypeError: Do not know how to serialize a BigInt. You need a replacer that converts to a decimal string, and then the payload is no longer the same shape it arrived in.

A value already parsed stays rounded. The reviver runs during parsing. If your HTTP client, your SDK or your framework parsed the body first, a second parse of the reserialized text cannot recover anything. This is the common case in application code: the client hands you an object and the raw text is gone.

Ids nested in arrays of objects need a leaf rule, not a container rule. The reviver sees the array element with its source, so a rule that inspects the key path works, but a rule that inspects the parent does not.

The fix I would pick first: keep ids as strings

Everything above is a consequence of accepting identifiers as JSON numbers. A quoted id round trips exactly through JSON.parse:

console.log(JSON.parse('{"id": "9007199254740993"}').id); // "9007199254740993"

The string form trades one comparison operator for the entire failure mode: it does not round, it does not collide, it does not depend on parity, and it survives any number of parse and stringify passes. A 64 bit id space reaches about 1.8e19, roughly 2000 times the safe integer limit, so a generator using the full range crosses 2^53 as it grows.

I have also seen schemas disagree about the type of the same identifier: one tool definition types the field as a string, another as an integer, and a strict validator rejects the call when you send the wrong one. Handling ids as strings on both sides ends the guessing, and the string survives the boundary that the number does not.

If a numeric id has to stay numeric for a legacy consumer, do the conversion at the edge, right after reading the response, and keep the string next to it for logging. The moment a rounded number reaches a log line, you lose the ability to tell whether the mismatch is in your code or in the data.

The test that catches this

A round trip test with the fixture {id: 42} passes on any implementation, including one that would break on the first real id. The fixture is what has to change:

const acima = "9007199254740993";
const volta = JSON.parse(`{"id": ${acima}}`, (k, v, ctx) =>
  k === "id" && ctx && ctx.source ? BigInt(ctx.source) : v
);
console.assert(String(volta.id) === acima, "id perdeu precisão no parse");

Two rules make that test worth having. The fixture has to sit above 2^53, because below it the bug cannot appear. And the same test has to fail on the unfixed parser, otherwise you are asserting that your test data is small, not that the parser is correct. I ran the control on purpose: the same payload through plain JSON.parse returns a value different from acima.

When an id mismatch shows up again, print the raw response body before printing the parsed object. The parsed object is the evidence that misleads, because it looks exactly like a valid record.

Se você já perdeu tempo comparando um id no log com o mesmo id no banco e encontrou números diferentes, conta nos comentários como descobriu a causa. Respondo em português.

Your asset URL returned 200. It was serving index.html.

Jonas Francisco de Lima Filho — Mon, 26 Oct 2026 20:29:31 +0000

Your asset URL returned 200. It was serving index.html.

I run a small private ops portal on nginx that holds the output of my scheduled jobs: reports, generated images, PDFs, drafts. A Python CLI called publicar copies each file into media/<publication-id>/, writes a JSON record that points at it, and rebuilds a manifest the feed reads. Upload verification, for a while, was one line of curl asking for the status code. It came back 200 and I moved on.

Then I asked the server a more precise question and got two different answers in the same minute, for two paths on the same host:

/media/20261026-1926-o-id-chega-arredondado-26-10-16h/01-slide-1-e6347173.png
  200  content-type: image/png    hash matches the local file

/media/pasta-que-nao-existe/relatorio.pdf
  200  content-type: text/html    hash does not match anything

The second path never existed. No PDF was ever generated with that name, no directory was created for it. It answered 200 because it was answering with the application shell.

Where the 200 comes from

The portal is a single-page app, and the vhost has the line that every single-page app needs:

location / {
    try_files $uri $uri/ /index.html;
}

That fallback is correct for routes. It is what lets /publicacao.html?id=abc render instead of returning 404 when the router owns the URL. The same line also decides what happens to a file request nobody planned for: the file is missing, so nginx serves index.html with status 200.

Matching in nginx runs by location, and my vhost has extension-specific blocks. A missing .png was correctly a 404, and so was a missing .json, because both extensions have their own location and nginx serves them from disk. Video, PDF, zip and CSV had no location at all. Those requests fell into location /, took the fallback, and returned HTML under a 200.

That is why the failure looked intermittent: identical code, identical record, different answer depending on the extension. A <video> element pointed at a missing .mp4 receives HTML and shows a broken player. A download link saves a file called index.html.

Immutable caching makes the filename part of the contract

Measured on the same request that returned the real image:

cache-control: public, max-age=31536000, immutable

One year, and immutable tells the browser not to revalidate even on a reload. That header is a promise about the bytes at that URL, and the only way to keep it is to make sure the bytes at that URL never change. So the publisher names every file with a hash of its own content:

bruto = src.read_bytes()
digest = hashlib.sha256(bruto).hexdigest()[:8]
nome = f"{indice:02d}-{slug(src.stem, 32)}-{digest}{ext}"

The numeric prefix is for ordering inside the publication page. The hash is what identifies the file. Replace the cover image and the URL changes, so the new bytes reach everybody immediately, including the reader with the old page still open. Name it by position instead, 01-capa.png, and the old image keeps being served for a year to anyone who already cached it. I lost a favicon that way once: a 404 got cached at the edge for a year, which is the same mistake in the other direction.

Updates append, they do not replace

--atualizar on this CLI starts from what the record already has:

midias = list(anterior.get("midias", []))
for spec in (args.midia or []) + (args.adicionar_midia or []):
    midias.append(copiar_midia(origem, pub_id, len(midias) + 1, legenda, tipo))

Appending is the behaviour I want when a report gets its PDF attached an hour after the text went out. It is the wrong behaviour for a correction. Ask to replace slide two of a six-slide piece and the record ends up holding both versions: eleven entries, five of them the piece you thought you had replaced, and the page renders all of them in order.

The deduplication rule follows from the naming scheme. Two paths that share the content hash are the same image, because the hash was computed from the bytes. Filter by the numeric prefix and it works once: the next update renumbers everything, so the prefix you excluded no longer points at the copy you meant to drop.

The config in the repository is not the config on the server

There is a script in the repo that adds the missing piece. It inserts a /media/ location with try_files $uri =404 and the immutable header, idempotently, and it is correct code.

grep -c "/media/" /etc/nginx/sites-available/central_operacoes
0

Zero. The generator lives in the repository, the nginx config lives in /etc, and nothing compares the two. Git shows a clean tree, the script passes a quick read, and the server keeps answering with the configuration it loaded weeks ago. An /etc file outside version control has no diff to review, so the verification has to read the running system: nginx -T prints the configuration nginx actually parsed, including everything pulled in by include, which is more than the file you edited.

add_header does not inherit

The same location would have carried the cache header and the 404, and it would have introduced a quieter problem. In nginx, a location that declares its own add_header stops inheriting the headers set at the server level. No warning, no error, and the headers simply stop appearing on responses from that location. HSTS disappears, along with the X-Content-Type-Options and X-Frame-Options lines I set once and stopped thinking about.

The generator includes the shared security snippet for exactly that reason:

location ^~ /media/ {
    add_header Cache-Control "public, max-age=31536000, immutable" always;
    include /etc/nginx/snippets/central_operacoes_security.conf;
    etag on;
    expires 1y;
    try_files $uri =404;
}

Any new location that sets a header needs that include. The mistake is invisible in a screenshot and obvious in a security scan, which is a gap that can last a long time.

One function that answers the real question

A status code is a claim by the server. The bytes are the fact. This function checks both, plus the identity of the file, and it is what told me the PDF path was fake:

def check(url, local):
    """Confirms the URL serves the local file, not a fallback page."""
    base = ["curl", "-s", "-u", AUTH, "--resolve", "agentjj.com.br:443:127.0.0.1"]
    meta = subprocess.run(base + ["-o", "/dev/null", "-w", "%{http_code} %{content_type}", url],
                          capture_output=True, text=True, check=True).stdout.split()
    status, ctype = meta[0], meta[1] if len(meta) > 1 else ""
    corpo = subprocess.run(base + [url], capture_output=True, check=True).stdout
    igual = hashlib.sha256(corpo).hexdigest() == hashlib.sha256(open(local, "rb").read()).hexdigest()
    return status, ctype, igual

Run against the two paths, it returns ('200', 'image/png', True) and ('200', 'text/html', False). The first tuple says the file is there and it is the file I uploaded. The second says the server answered successfully with something else entirely.

The three fields earn their place. The status tells you the request was routed. The content type tells you what kind of thing came back, which is where a fallback page gives itself away. The hash tells you it is your file and not a stale copy under the same name. Checking only the first is the version of this test that passes while the page is broken, and I have shipped that version more than once.

What I would check first next time

Read the content type before trusting a 200, especially on a server with a single-page fallback. Confirm the running configuration instead of the file you wrote, because a generator that never executed looks exactly like a generator that worked. And when caching is immutable, treat the filename as part of the API, since the URL has to change when the content changes.

Se esse tipo de armadilha aparece no seu projeto, comenta aí qual foi: servidor que responde 200 com página de erro, cache que serve arquivo velho, ou configuração que só existe no repositório. Gosto de comparar com o que já quebrou por aqui.

Four bugs my publisher's validator could not see

Jonas Francisco de Lima Filho — Fri, 23 Oct 2026 21:26:23 +0000

Four bugs my publisher's validator could not see

I publish everything I build to a small private ops portal: scheduled job reports, generated images, deploys, drafts. One Python CLI called publicar writes a JSON file per publication into data/publications/, rebuilds the feed manifest in data/index.json, and has a validar subcommand that is supposed to stop a broken record from going live.

Between October 1 and October 6, seven publications went out with a file path where the article body should have been. The validator ran on every one of them and printed OK. The feed showed cards that opened a page whose entire content was /tmp/corpo.md.

This is what those failures had in common, and the check that now catches them.

The flag never looked at its input

--markdown takes either a literal string or - for stdin. A file path is also a literal string. That is the whole implementation:

markdown = args.markdown
if markdown == "-":
    markdown = sys.stdin.read()

I had been typing --markdown /tmp/corpo.md, expecting the CLI to read the file because that is what most tools do. It stored the path. Eighteen characters, exit code 0, no warning. The fix belongs in the validator, not in the flag, because changing the flag's meaning would silently break the calls that already pipe through stdin.

Agreement between two files is not correctness

The validator compared the manifest against the per-publication JSON. Both were produced from the same corpo field, so both contained the path, so they agreed, so the check passed. A check that compares a value to a copy of itself is a check on the copying code. It says nothing about whether the value is right.

What made this detectable was an independent statement about the shape of a body. A body has line breaks. A body is usually longer than a path. A body rarely ends in .md. The current check encodes exactly those three properties and nothing else:

corpo = (p.get("corpo") or "").strip()
if corpo and "\n" not in corpo and len(corpo) < 300 \
        and (corpo.startswith(("/", "~/")) or corpo.startswith(".")) \
        and re.search(r"\.(md|markdown|txt|html)$", corpo):
    problemas.append(
        f"{f.name}: corpo e um caminho de arquivo, nao o texto "
        f"({corpo}) — reidrate com: publicar publicar --atualizar "
        f"{pid} --markdown - < {corpo}")

Two things matter more than the regex. First, the error message carries the repair command, so a failing run tells me what to type. Second, the check is deliberately narrow. A conservative rule that fires on real defects beats a clever rule that fires on files with no newline for legitimate reasons.

I verified it in both directions: fed it a record with a path body and watched it fail, then hydrated the same record with the real text and watched it pass. A validator I have only seen pass is a validator I have not tested.

`--tags a,b,c` became one tag with commas in it

--tags is action="append", so the argument arrives as a list. The old code assumed a string and split on commas only when the type matched. It also appended to whatever tags the record already had, which is the behaviour I wanted for a second call. The result was a chip in the feed that read projeto,cliente,enrique as a single label.

"tags": sorted(set(
    [t for bruto in (args.tags or []) for t in bruto.split(",")]
    + anterior.get("tags", []))),

The quiet part is the + anterior.get("tags", []). Update operations in this CLI add to lists, they do not replace them. Tags and media accumulate. Links are rewritten from scratch, because I decided a link list is a statement of the record's current state rather than a log. Two different merge policies in the same function, chosen per field. Every one of them needs to be written down, because neither is discoverable from the flag name.

A type string that nothing validated, and three readers that broke silently

Media entries carry tipo. I typed ::imagem once, in Portuguese, out of habit. The CLI accepted it, wrote it into the JSON, and three separate consumers broke, each in a different and silent way:

resumo_card builds the feed card and picked the cover by comparing tipo == "image". It found nothing, returned capa: None, and the feed served a 404 for the card image.
The publication page's JavaScript fell through to its default branch and rendered the file name as text instead of an image.
The media viewer filtered by type, got an empty group, and did nothing when the thumbnail was clicked.

publicar validar passed all three times. It had checked that the media file existed and that the manifest matched the JSON. Both were true. The manifest had recorded "no cover" and the JSON had recorded "no cover", and consistency was the only property being measured.

The fix normalises in one place and accepts both spellings at the boundary:

# O tipo da midia pode vir em ingles ("image", valor que o `--midia`
# grava por padrao) ou em portugues ("imagem", que tambem aceita).
# Aceitar so "image" deixava a capa em None sem nenhum aviso: o
# manifesto ficava coerente com o JSON e o `publicar validar` nao
# reclamava, enquanto o card do feed servia um 404 em silencio.
for m in mids:
    if m.get("tipo") in ("image", "imagem"):
        capa = m["path"]
        break

Accepting both is the smaller half of the lesson. The larger half is that an enumeration written by hand should be normalised at the moment of entry, not matched at the moment of use, because every consumer that pattern-matches on it is a place the match can be wrong.

Update appended, so correcting a piece doubled it

A carousel went from five slides to six. I ran the update flags again with the new set of images. The publication ended with eleven media: six new and five from the old run. The next correction made twelve.

The fix is not "filter the old files out", because the numeric prefix in each stored file name is rewritten on every run, so a filter based on position works once and fails the second time. The reliable key was already in the file name, appended by the CLI when it copies media into the publication folder: a content hash. 07-slide-1-2f78bcbb.jpg and 01-slide-1-2f78bcbb.jpg are the same image. Two paths sharing a hash are one media, regardless of their prefixes.

What the checks look like now

Every table above reduces to a question I ask before trusting a green run:

Does the check measure the property I care about, or does it compare two artifacts produced by the same code path? If the manifest and the record are both generated from one field, agreement between them proves nothing about the field.
Have I seen this check fail on a genuinely broken input? A check whose only observed output is OK has an unknown failure mode.
Is the enum normalised where it enters, or matched where it is used? The second option multiplies the number of places that can disagree.
Which fields does an update add to, and which does it replace? Write the answer next to the flag.

The one that cost the most time was the first. I had built a validator, I ran it before every deploy, and it was measuring consistency the entire time. Consistency is the cheapest property to verify and the least useful one to be sure about.

If you have built a publishing pipeline of your own and hit a similar case, I would like to hear which invariant you ended up checking. Meantime, the whole system is mine, small, and under a scheduler, so the failures above are all reproducible rather than hypothetical.

Se você mantém um pipeline de publicação parecido e prefere português, deixa um comentário que eu respondo em português. Esse texto saiu de um CLI que publica relatórios, imagens e deploys num portal próprio, e a parte de validação foi a que me ensinou mais.

Three ways a sync job reports success while reading nothing

Jonas Francisco de Lima Filho — Wed, 21 Oct 2026 21:13:30 +0000

I keep a weekly job that copies tasks from ClickUp into my own dashboard. It is a small Python wrapper around a CLI that reaches an integration platform, and it publishes a report with the lists and tasks it found. On October 19 it published 12 tasks from 3 lists, and that number happened to be correct.

The reason I went looking was different. The job went green every week, and green was the only thing I was reading. What follows is what I measured when I stopped trusting the exit code.

Every failure exits 0

The wrapper is the usual twelve lines: run the CLI, check the return code, parse stdout as JSON, return the dict.

proc = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
if proc.returncode != 0:
    raise RuntimeError(f"exit {proc.returncode}: {proc.stderr[:300]}")
return json.loads(proc.stdout)

That code assumes the process fails when the call fails. In this CLI it does not. I sent a task-tracker workspace ID as a JSON number instead of a string and got this, with exit code 0:

{"successful": false,
 "error": "Input validation failed for CLICKUP_GET_SPACES.\nSchema: /home/ubuntu/.composio/tool_definitions/CLICKUP_GET_SPACES.json\n- team_id: Instance type \"number\" is invalid. Expected \"string\"."}

A slug that does not exist behaves the same way: exit 0, "successful": false, "slug": "ToolRouterV2_ToolNotFound". The failure is a field inside the JSON, not a process state.

So the wrapper raises on nothing and hands the error object to the next function as if it were a result. What happens next depends on the caller. Where the caller indexes resp["data"], the run dies with KeyError: 'data', which is a message about my key and not about the call that failed. Where the caller guards with if resp.get("successful"), the run finishes green having read zero of whatever it wanted. Neither outcome is a report about the failed call.

The envelope changes shape when the payload gets big

Same tool family, different day. Listing 100 files from Google Drive returns them inline, under data.files. Listing 200 returns this instead:

{"successful": true, "storedInFile": true, "tokenCount": 17598,
 "sizeBytes": 70390,
 "outputFilePath": "/home/ubuntu/.hermes/cache/scratch/composio/adhoc_143ac9a280f4/GOOGLEDRIVE_LIST_FILES_OUTPUT_e603b2a0.json"}

The 200 files are real. They are inside that file, together with a nextPageToken, and the response that reached my code has no data key at all. Code written against the small response does resp.get("data", {}).get("files", []) and gets [], with no exception and no log line. The wrapper was not wrong about JSON. It was wrong about which JSON.

I did not chase the exact byte threshold where the platform starts spilling to a file. The code has to handle both shapes, so the threshold changes nothing. The habit that did change something: after every large listing, print the length of the list I just built, and compare it with what the response says it contains.

A conditional on the success flag is a silent skip

This is real code from the sync job, and it looks defensive:

resp = run_composio("CLICKUP_GET_FOLDERLESS_LISTS", {"space_id": SPACE_ID})
if resp.get("successful"):
    all_lists.extend(resp["data"].get("lists", []))

resp = run_composio("CLICKUP_GET_FOLDERS", {"space_id": SPACE_ID})
if resp.get("successful"):
    for folder in resp["data"].get("folders", []):
        all_lists.extend(folder.get("lists", []))

Lists live in two places, so the job asks twice and merges. If the second call fails, the job does not stop. It syncs the folderless lists, writes a report with one list instead of three, marks the run ok, and the dashboard shows a plausible number. Nothing about that page lets me tell a quiet week from a lost API call.

The failure that did leave a record in my dashboard was the opposite kind: exit 7 in 0.004 seconds, with an empty body. The script never started, so a wrapper caught it and published a red entry with the log path. Loud failures get recorded. Quiet ones get published as data.

A read that feeds a report should fail loudly. Skipping is correct only when the missing thing is genuinely optional, and then the code should say why in a comment, because the next person reading that conditional cannot tell the two apart.

The same number arrives as the wrong type

The workspace ID is 90171594651. In the API it is a string, and in my payload it went out as a number, which is why the error above names team_id and both types.

I had written it without quotes because it looks like an ID and IDs are numbers. The schema was on disk the whole time, one command away:

composio execute CLICKUP_GET_SPACES --get-schema

The lesson I keep relearning is to read the schema before the first call, not after the third failure. The related one: when a validation error names the field and the two types it wants, that message is better than anything I will discover by rewriting the payload by guess.

One more quirk of this CLI, since it shaped the wrapper: -d is mandatory even for a tool that takes no arguments, so the payload is -d '{}'. Leaving it out is an argument error, which is one of the few failures that does come back with a non-zero code.

The wrapper, with four checks

Each check raises with the tool slug in the message, so a failure names the call and not just the job:

def run_tool(slug, payload):
    proc = subprocess.run(
        ["composio", "execute", slug, "-d", json.dumps(payload)],
        capture_output=True, text=True, timeout=120,
    )
    if proc.returncode != 0:
        raise RuntimeError(f"{slug}: exit {proc.returncode}: {proc.stderr[:300]}")
    try:
        resp = json.loads(proc.stdout)
    except json.JSONDecodeError:
        raise RuntimeError(f"{slug}: non-JSON stdout: {proc.stdout[:300]}")
    if resp.get("successful") is not True:
        raise RuntimeError(f"{slug}: {resp.get('error')}")
    if resp.get("storedInFile"):
        resp = json.loads(Path(resp["outputFilePath"]).read_text())
        if resp.get("successful") is not True:
            raise RuntimeError(f"{slug}: {resp.get('error')}")
    return resp.get("data") or {}

The shape check belongs to the caller, because only the caller knows what a correct answer looks like:

def items(data, key, slug):
    value = data.get(key)
    if not isinstance(value, list):
        raise RuntimeError(f"{slug}: expected {key!r} to be a list, got {type(value).__name__}")
    return value

Two lines there carry most of the value. Resolving storedInFile before anyone reads data keeps the envelope invisible to the rest of the code. Making the shape check raise on a missing key turns "zero items" into a sentence that says which tool and which field were wrong.

Test it with wrong input

A wrapper like this is only proven by the failures it catches, so the two cases I keep are a slug that does not exist and an ID sent with the correct value in the wrong type. Both come back with exit 0 and successful: false. A wrapper that raises on those two is reading the answer and not the process. If both pass against a wrapper that only looks at the exit code, the tests are wrong, not the wrapper.

The third case is a large listing, which is the one people skip because the small fixture works. Point the wrapper at any response big enough to spill to disk and assert that the item count matches what the response reports.

Se você mantém rotina agendada batendo em API de terceiro, o ponto que vale discussão é onde o seu wrapper decide que a resposta é aceitável. Escrevo em português também, então deixa um comentário aqui embaixo que eu respondo.

Your vision model cannot see a broken layout. Measure the pixels instead.

Jonas Francisco de Lima Filho — Mon, 19 Oct 2026 20:34:12 +0000

I generate the social cards for my projects as HTML and CSS, screenshot them with headless Chromium, and publish them without a person looking at each slide first. That last part is the reason I needed a gate: something has to decide whether a card is safe to publish, and it has to be cheap enough to run on every piece, every day.

The obvious candidate was vision. Screenshot the card, send it to a multimodal model, ask whether any text is cut, overlapping or missing. Multimodal models read text well and describe screenshots well, so this looks like a solved problem.

It did not survive contact with the pipeline. What follows are the failures, and the check that replaced them.

Asking a vision model conditions the answer

The first version asked, literally: "is any text cut off in this image?" The model said yes on pieces where nothing was cut. I measured those margins by hand and they were comfortable.

The same question fails in the other direction. When the layout really is broken, a leading question pushes the model toward a confident sentence with a location attached, and the location is often wrong. A model that produces plausible coordinates is worse than one that says it cannot tell, because the coordinates end up in a report and the report gets trusted.

What changed my mind was not a benchmark. It was a card that passed the vision check and shipped with the footer missing, phone number included. The measurement below caught the same class of defect on the next run, in two seconds, with the row number and no adjectives.

Ink is any pixel that differs from the background

The naive pixel check fails on a light theme, and it fails quietly. Define ink as "light pixels" and a cream background becomes ink: every row reports text at y=12, a batch of cards all fail, and nothing is actually wrong.

The rule that held up separates three things.

Ink is any pixel whose color differs from the piece's background, sampled at a corner the design never uses for content.
Decoration is separated from content by distance to that background: 0 to 24 is background, 25 to 49 is halo or glow, 50 and above is content.
Stroke thickness separates a grid line from a glyph. A rule is one pixel wide. A letter never is.

from PIL import Image

img = Image.open(path).convert("RGB")
w, h = img.size
bg = img.getpixel((4, 4))

def distance(px):
    return max(abs(px[i] - bg[i]) for i in range(3))

def kind(px):
    d = distance(px)
    if d < 25:
        return "background"
    if d < 50:
        return "halo"
    return "content"

Those thresholds are not universal. They come from one palette, and they age the moment the palette changes. Recalibrate when the design changes, or the check keeps passing while measuring the wrong layer.

A cut is local, so measure every line

A card can be perfect everywhere except one line, which is exactly the case a whole-image average hides. Group the rows that contain content into bands, then take the leftmost and rightmost ink inside each band.

def bands(mask, h, gap=30):
    rows = [y for y in range(h) if any(mask[y])]
    out, current = [], []
    for y in rows:
        if current and y - current[-1] > gap:
            out.append(current)
            current = []
        current.append(y)
    if current:
        out.append(current)
    return out

For each band, min_x and max_x. If min_x < 6 or max_x > w - 6, the card fails and the report names the band. Six pixels is the tolerance. Below that, a glyph is touching the frame, and touching the frame on a phone means clipped.

Chromium's `--screenshot` does not capture a tall page

This one cost an hour and it is not a layout bug. A 1080 by 1350 card came out with a white strip at the bottom. Below y=1232 there was no ink at all, and the footer carrying the phone number, the handle and the site had vanished.

The DOM was correct. The CSS was correct. Every assertion against the DOM passed, because the DOM was never the problem. chromium --screenshot fits the page into the window and stretches the result, so the capture is fake exactly where the content sits lowest.

The fix is to drive Chromium over the DevTools protocol, set the real metrics, and ask for the region beyond the viewport:

cdp("Emulation.setDeviceMetricsOverride",
    width=1080, height=1350, deviceScaleFactor=1, mobile=False)
shot = cdp("Page.captureScreenshot", format="png", captureBeyondViewport=True)

Then measure the output by pixel. Not by eye, and not by asking a model.

The failure I did not expect: a test that passes for the wrong reason

The gate that checks reading order failed four times before it was right.

Fixed positions for the numbers. Broke the first time the layout moved.
Height only. Found two of the three numbers and called the order valid.
Height plus width, no control. Indistinguishable from a function that always returns sorted.
Height plus width with a control. The control glued the bands into one block because it used a 24 pixel gap while the grouping constant was 30, the order came out empty, and an empty order passed.

That fourth one is the useful story. The control existed, it ran, and it proved nothing, because the control has to produce the wrong answer for a wrong input. Both paths also have to call the same function. If the control has its own copy of the logic, you are testing the fixture.

Text in the frame has to be explicit data

The corners of each card carry the state, the badge and the slide number. All three passed their checks while the content was wrong, because the state was derived from the last word of the badge. A piece labelled "GOOGLE CLOUD" reported its state as "CLOUD", another reported "LADO", and no check could see it since the value was already wrong before it arrived.

Deriving one field from another saves a line of code and creates a class of bug the verifier cannot catch. Frame text is data. Write it as data.

What the gate looks like now

Every piece runs the same order: render, measure ink, check margins per line, check reading order, fingerprint the footer region. Measurement decides. A vision model is at most a second opinion, and when it is used, the prompt describes what to look at instead of asking whether something is broken.

The rule I took from all of this: if a check can only ever say yes, it is not a check. Feed it a card that is genuinely broken and watch it fail, because on the day it is wrong, nobody else is looking at the image either.

Se você está montando um pipeline parecido e prefere português, deixa um comentário que eu respondo em português. Esse texto saiu de um sistema que gera, verifica e publica arte de carrossel sem revisão humana, e a parte de verificação foi a que me ensinou mais.

DEV Community: Jonas Francisco de Lima Filho

My screenshot was 1080x1350. The page was 1722 pixels tall.

What the check asked

A screenshot request describes a rectangle

Two files with the right size and the same missing block

The size check was comparing the request to itself

Capture at the height you measured

What CDP adds, and the two details that bite

The shape of this bug

Half of my corrupted text was pure ASCII, and my character check passed it

The defect that looks like damage

The defect that looks clean

Nothing downstream complains

Three checks in one file

Two guards that keep the gate trusted

Where the damage is born

Para quem publica em português

Six still PNGs, six narration clips, and the ffmpeg pass that keeps them in sync

Six still PNGs, six narration clips, and the ffmpeg pass that keeps them in sync

The frames were designed as still images

The narration sets the timing

Each frame becomes a clip of its own length

Padding the audio so each line stays with its frame

What the container reports

The silence you hear is longer than the padding you set

What I would change

Bisecting a request body to find why the API returned 403

A 403 with an HTML body is not an authentication problem

Rule out size with one control payload

Split by paragraph, and keep the wrapper

The trigger was one inline command

The wrapper is part of the request

The field the tool never sends

The checklist I keep now

Headless Chromium gives up on HTTP basic auth. A local proxy injects the header instead.

Two failures wear the same log line

Read the credential from a file, inject the header locally

What a 200 through the proxy actually proves

Lazy images report a width of zero

Stop the proxy when the proof is in

JSON ids above 2^53 arrive rounded, and a reviver only fixes the leaf

The smallest case that fails

2^53 is the boundary, and above it the step is 2

Two different ids can become one key

A reviver fixes the leaf, not the container

What the reviver does not fix

The fix I would pick first: keep ids as strings

The test that catches this

Your asset URL returned 200. It was serving index.html.

Your asset URL returned 200. It was serving index.html.

Where the 200 comes from

Immutable caching makes the filename part of the contract

Updates append, they do not replace

The config in the repository is not the config on the server

add_header does not inherit

One function that answers the real question

What I would check first next time

Four bugs my publisher's validator could not see

Four bugs my publisher's validator could not see

The flag never looked at its input

Agreement between two files is not correctness

--tags a,b,c became one tag with commas in it

A type string that nothing validated, and three readers that broke silently

Update appended, so correcting a piece doubled it

What the checks look like now

Three ways a sync job reports success while reading nothing

Every failure exits 0

The envelope changes shape when the payload gets big

A conditional on the success flag is a silent skip

The same number arrives as the wrong type

The wrapper, with four checks

Test it with wrong input

Your vision model cannot see a broken layout. Measure the pixels instead.

Asking a vision model conditions the answer

Ink is any pixel that differs from the background

A cut is local, so measure every line

Chromium's --screenshot does not capture a tall page

The failure I did not expect: a test that passes for the wrong reason

Text in the frame has to be explicit data

What the gate looks like now

`--tags a,b,c` became one tag with commas in it

Chromium's `--screenshot` does not capture a tall page