DEV Community: Serhat

I measured what 300 websites actually serve to AI agents

Serhat — Sat, 29 Aug 2026 16:08:49 +0000

People keep saying that sites treat cryptographically signed AI agents differently, or that they hide extra text aimed at models. Both claims get repeated a lot. Neither seemed to have been measured.

So I built a tool and pointed it at 300 public sites.

What I expected to find

Two things, mainly.

First, that sites would treat a signed agent differently from an unsigned one. There is a standard for this now, HTTP Message Signatures with the web-bot-auth drafts, and Cloudflare shipped support for it. A bot signs its requests with a private key, publishes the public key at a well-known URL, and the site can verify who is actually knocking.

Second, that some sites would be hiding text for models. Invisible spans, text present in the machine-facing response but absent from what a person sees.

Neither one showed up.

Signed identity changed almost nothing. Across 300 sites there were three cases where a signed request was treated differently from an identical unsigned one, and in all three the signed request was treated worse. Never better.

Hidden machine-only text turned up on one page out of 281. It belongs to firecrawl.dev, a 1px transparent span that tells AI agents which onboarding file to read. Real, but rare, and not adversarial.

The standard works. Nobody is using it to decide anything yet.

What I did not expect

The measurement that produced a number on every single page was cost.

The median page costs a machine reader 2592 tokens. That is cl100k_base, measured on extracted text, not raw HTML. Script and style tags are not counted, because no agent sends those to a model.

About half of that is not body content. Roughly a fifth is navigation, header and footer. The rest is repeated text.

Seven of the 300 sites serve a markdown variant to machine readers. Those pages deliver the same content at a median of 962 tokens, about a third of the cost.

The part that bothered me

Six of those seven markdown variants are missing Vary: Accept.

That header tells caches that the response depends on what the client asked for. Without it, a cache sitting in front can store the markdown version and hand it to a browser, or store the HTML and hand it to an agent. It fails quietly and nobody notices.

Three of the seven also drop real body content from the markdown version. Not navigation, actual page copy. All three are on the same platform.

Then I pointed the tool at Cloudflare's own documentation page for the markdown feature.

It has both problems. Missing Vary: Accept, and the markdown version is missing the heading "How to enable" along with the paragraph underneath it.

That one surprised me.

Why this is harder to measure than it looks

Here is the part I did not plan for.

My first attempt compared one unsigned request to one signed request on each site's homepage. Sensible enough. Except that when I fetched the same homepage twice with identical unsigned requests, 8 out of 30 sites returned different content.

Twenty seven percent. Product cards rotating. Timestamps ticking over. Tracking parameters on links.

So a naive signed-versus-unsigned comparison is mostly measuring the page moving underneath you. Any result you get is noise wearing a costume.

What fixed it: measure an interior content page instead of a homepage, fetch the unsigned request three times, and define a stable core as the blocks that are identical across all three. Compare the signed request only against that core.

That brought baseline instability from 27 percent to under 4.

If you take one thing from this, take that. Anyone measuring identity effects on live pages without controlling for the page changing under them is publishing noise.

Things I got wrong

The first cost figures counted raw HTML. Script, style and attributes went in as if a reader sent them to a model. Withdrawn.

The first markdown figure was 6/7 for content loss. Most of what it counted was navigation chrome, which is exactly what a markdown conversion is supposed to drop. Corrected to 3/7.

The injection scan first reported 58 findings. Almost all of them were HTML comments, tooltips, and related-post rails. I went through all 190 dropped findings by hand before tightening the rules, because a detector tuned by looking at its own output is not a measurement.

One of four challenge findings did not reproduce on a second fetch. It is not reported as a finding.

All of that is listed in the writeup with what each number was, what it is now, and why it changed.

The tool

Go, single static binary, four runtime dependencies, no model calls, no API key, nothing leaves your machine. MIT.

It fetches a page under several identities, including one that signs its requests properly, and tells you what each one received. It does not grade sites.

content-parity check https://example.com/

Repo: https://github.com/Zulwatha/content-parity

Full numbers and method: https://github.com/Zulwatha/content-parity/blob/main/docs/results.md

If you can break the methodology, I would rather hear it than not.

I got tired of running Redis for one background job, so I built a SQLite job queue

Serhat — Sun, 02 Aug 2026 10:47:22 +0000

Every side project I've shipped in the last few years has hit the same wall at roughly the same point. The app works, it runs on a single cheap VPS, and then I need to send an email after the request returns. Or resize an image. Or run something every night at 3am.

The standard answer in Node land is BullMQ, which means Redis. And Redis is great. But now my "one small app on one small server" has a second service to install, monitor, secure, and remember exists when the box reboots. All so I can send an email five seconds later.

The third time I caught myself provisioning Redis for a queue that would see maybe two hundred jobs a day, I stopped and wrote vardiya instead.

The idea

If your whole app lives on one machine, you already have a durable, transactional store sitting right there: the disk. SQLite in WAL mode handles concurrent readers and a single writer perfectly well, and a job queue is mostly just "one writer claims rows atomically."

So vardiya keeps the entire queue in a SQLite file. One runtime dependency (better-sqlite3). Cron parsing, backoff math, and id generation are all in-house, so there's no dependency tree to audit.

import { Vardiya } from "vardiya";

const v = new Vardiya({ databasePath: "./jobs.sqlite" });
await v.init();

await v.enqueue("email", { to: "a@b.com" }, { delayMs: 5_000 });

const worker = v.createWorker({ concurrency: 4 });
worker.process("email", async (job) => sendEmail(job.payload));
await worker.start();

That's the whole setup. No broker URL, no connection retries, no second process. When your app dies, the file is still there, and so are your jobs.

What it actually covers

I didn't want a toy that falls over the moment you use it seriously, so the boring-but-necessary parts are in:

Retries with fixed or exponential backoff, optional jitter, and a dead-letter state when maxAttempts runs out
Delayed jobs, priorities, and custom jobId for dedup when your producer might fire twice
Repeatable jobs via 5-field cron (plus @hourly, @daily, @weekly, @monthly)
Heartbeats and stalled-job reclaim, so a crashed worker doesn't strand jobs in "active" forever
Atomic claim via a single UPDATE ... RETURNING, so multiple worker processes on the same file won't grab the same job
Typed events for everything (job:completed, job:failed, job:dead, ...)

On my machine the bench does around 13k enqueues/sec and roughly 5k processed jobs/sec end-to-end. Your disk will disagree with my disk, so run npm run bench yourself before quoting numbers.

The honest part

vardiya is at-least-once, and I say so in the README instead of burying it. A job can run twice if a worker dies after doing the work but before recording completion. Every crash-safe queue has this property; the ones that claim exactly-once are just moving the problem into your side effects. Write idempotent handlers, use jobId as a dedup token, and you're fine.

Also: SQLite is a single-writer database. Multiple processes on one host sharing a local file works. A fleet of app servers fighting over a network mount will not, and I'm not going to pretend otherwise. If you need many machines pulling from one logical queue, BullMQ or pg-boss are the right tools and I'd use them myself. Same if you want dashboards, rate-limit groups, or sandboxed processors. Those are product features other projects spent years on, and vardiya is deliberately just the queue core.

Why "vardiya"?

It's Turkish for "work shift." Workers clocking in, picking up jobs, clocking out. It felt right.

Try it

npm install vardiya

Requires Node >=22. The repo has examples, the full API reference, and a comparison table against BullMQ / pg-boss / bee-queue if you want the details: github.com/Zulwatha/vardiya

It's MIT licensed and young. Issues and PRs are very welcome, especially war stories from anyone who has pushed SQLite queues harder than I have. And if the "one VPS, no Redis" shape matches your app, I'd genuinely like to hear whether it holds up for you.