DEV Community: Evgenii Arsentev

Nobody with a crawler token asked for my llms.txt - what I saw in 15 days of nginx logs

Evgenii Arsentev — Mon, 05 Oct 2026 10:28:35 +0000

I run a digital health startup as CEO (and a personal site, arsentev.ai, about practical AI). I put /llms.txt and /llms-full.txt on my site, and I wanted to see who actually reads them.

The two files are a plain-text summary of the site for language models. They were first served on 9 September 2026. I kept the complete nginx access logs of one origin server for 28 August - 11 September 2026. It is 15 days and 345,808 parsed requests.

What I counted

A request was a crawler if the User-Agent contained one of 22 tokens published by search and AI operators (GPTBot, ClaudeBot, PerplexityBot and similar). 46,155 requests carried such tokens, and 20 distinct tokens appeared. None of them requested /llms.txt or /llms-full.txt. I also checked every path ending in llms.txt, including /.well-known/llms.txt. No crawler token there either.

Then I looked only at 9-11 September, after the files were first served. In these days crawler tokens made 15,906 requests. Of them 698 were for /robots.txt (11 tokens) and 282 for /sitemap.xml. So the crawlers were on the site, they just didn't ask for my files.

Who asked for them

All 67 requests for the context files came from clients without a crawler token. 53 of them were command-line HTTP clients (curl and the like).

About a third of crawler-token requests bypassed the CDN. None of those came from the operators' published IP ranges, so I think much of that traffic is imitation. The robots.txt and sitemap numbers don't rest on it.

One more thing I noticed. Nothing in any standard tells a crawler that llms.txt exists. A sitemap is different, because robots.txt can point to it.

My mistake

First I published different numbers: 331,758 / 44,005 / 577. They came from a log that was cut at about 12:09 UTC on the last day. I ran everything again on the complete log and corrected the report. The numbers above are from the complete log.

Limits

This is one origin. Three hostnames are pooled in one log (host wasn't logged). After deployment I have only about two days of exposure. User agents are self-asserted, and I did not record links to the files. So it is absence on one server, not proof for the web.

After this I added $host to my nginx log format. The next study can split the hostnames.

What I suggest

Just check your own access log before you decide that anyone reads your llms.txt. Look for requests to it by user-agent. Grep is enough for this.

The report, the scripts (Python standard library only) and the aggregate counts without IP addresses are here: https://doi.org/10.5281/zenodo.23018429

Research index: https://arsentev.ai/research

AI assistance: drafted and edited with an AI tool; the data, measurements and conclusions are my own.

My company has no full-time developers, DevOps or designers. Here's what three $200 Claude subscriptions do instead

Evgenii Arsentev — Mon, 28 Sep 2026 20:30:15 +0000

At night the agent shifts run by the clock. New builds of educational games get built, and a separate fresh agent checks each one by screenshots. The mail agent of one business line starts every hour, night too. In the morning the results are ready. In the businesses I run there are no full-time developers, no DevOps, no designers, no analysts and no product manager. At US median pay a developer, a product manager, an analyst and a designer come to about $38,500 a month (Bureau of Labor Statistics, see the note at the end). The agents doing that work run on three $200 Claude subscriptions, so $600 a month (about 64 times less).

There is me, there is a product builder, and there are AI agents that run on a schedule on separate servers (not on my laptop). I'm a medical doctor by education and a CEO. This is not a forecast, it's what runs today, what I kept for people and what didn't work.

The order we did it in

Designers went first, as soon as agents got good enough at design. Analysts went a long time ago too. The product manager role is closed too, and Claude does its tasks now (tasks that took days now take minutes). The developer was the last one.

We kept our former CTO on a small retainer as a safety net, in case an agent breaks something serious. So far we haven't needed him once. Everything our DevOps used to do, the agents do.

What is on the schedule

Each business line has its own agent shift. It starts by the clock, does its job and stops. I only start things, the agents do the rest.

Deploys. Releases to all our projects and zones are done by agents.

Our own CRM. We didn't buy one. The agents built a CRM from scratch, and the agents also work inside it. So why buy a CRM if we built our own?

Email. Every business line has its own mail shift, and the agent sorts the mailbox of that line only.

Meetings. Scheduling meetings is on the agents too.

Technical support. Technical support is done by agents.

Content. On a news and learning site I founded, agents publish news four times a day and scan new open-source AI releases three times a day. The games project I mentioned works overnight, and the builder's opinion about its own work doesn't count - only the fresh agent's check does.

And not everything is an agent. Backups for example are plain scripts on a timer.

The supervisor of the supervisor

This is the part I like most. In one of our US business lines we have a call-center team. The operators and their supervisor are real people. But the supervisor gets tasks from an agent.

Every day the agent goes through the operators' work - the calls, their transcripts, the results in the CRM. Then it writes to the human supervisor what to do today and tomorrow (with a number and a deadline), who needs help, who should be replaced and what the plan is for each person. Each operator gets a separate message with 3-4 small skills to fix, taken from the transcripts of their last calls. Every evening each skill gets a ✅ or ❌, and after three ✅ in a row the skill is closed.

The agent asks three things in every shift. Is each operator better or worse than last time? Did the supervisor improve the work of their people (by the people's numbers, not their own)? What should the supervisor do today and tomorrow? If the team is stuck, that is written as the supervisor's failure. The agent only suggests. Decisions about people are made by people and are not posted in the team channel, and the operators were told openly that calls are checked every day.

Subscription, not tokens

The agent work runs on subscription, not on paying per token through the API. On a flat price the students on my course experiment more boldly. The limits get eaten by re-reading context - in my own measurement more than 85% of the modeled spend was context work (DOI 10.5281/zenodo.22759216). And every shift is short, it does one thing and exits.

Where people stayed

An agent can do the work, but it can't be accountable for it. Every agent workflow has a named person who owns the result and answers for it as if a colleague had done the work.

Checks stay mandatory. Agents don't show doubt (a wrong result comes with the same confidence as a correct one). Nothing ships just because an agent finished it. It ships because it passed a check. And an agent gets access for a task, not a standing role.

What didn't work for us

First, vague tasks. A vague task given to a person gets clarified over coffee. Given to an agent, it gets done confidently and wrong. So the main management skill for me now is writing the task - the goal, the limits, what "done" means and how we check it.

Second, measuring the wrong thing. The first AI judge of calls in our CRM scored them against a script checklist. It measured if the operator followed the script. I think we should judge an operator by whether the partner gets to a purchase, not by the checklist.

Third, screens nobody opens. Our CRM has supervisor pages, and in 30 days they were opened 9 times. What works is the plain daily message to each person.

Fourth, AI summaries garbled a supervisor's name. So for names we trust only the spelling a person confirmed.

A case I documented

A company whose case I documented replaced a video studio with Claude agents - 1,497 videos across 67 channels in 101 days, at $2.88 per video. With people it would take 42 to 78 staff and cost 138 to 935 times the AI subscriptions (DOI 10.5281/zenodo.22802229). It's not our company, and I only wrote the analysis.

What I would tell another CEO

Just start with one function where the result is easy to check. Name a person who owns the result and track the cost per result from the first week. I look at the cost of a task that was done and accepted (with reruns and review time), not the price of the tool.

Keep the shifts short. But don't clear the context after every single task either. In my own small experiment that was about a third more expensive than clearing it every three tasks.

Where the limit moved

Before, we were limited by people and the job market. Now the limit is Claude tokens and computing power - memory and GPUs. That's why I rent separate servers, including GPU machines, so the agents don't run on a laptop that chokes.

I think memory is becoming the new gold. And I think for everyday agent work we'll need something like what ASICs did for bitcoin, a class of chips made for this kind of load.

Note on the numbers: US median annual pay from the U.S. Bureau of Labor Statistics, Occupational Employment and Wage Statistics, May 2025 (bls.gov/oes) - software developers (15-1252) $135,980, project management specialists (13-1082, the closest BLS category to a product manager) $102,320, data scientists (15-2051) $120,230, web and digital interface designers (15-1255) $104,000. Monthly = annual / 12. These are market numbers, not our salaries.

Evgenii Arsentev, MD, PhD — CEO, digital health; founder, arsentev.ai
Evgenii Arsentev is a medical doctor and a digital health CEO whose businesses run almost entirely on AI agents. He is the founder of arsentev.ai.

Originally published at arsentev.ai.

AI assistance: drafted and edited with an AI tool; the data, measurements and conclusions are my own.

I Ran a Radar on 490 Open-Source AI Projects. Then I Checked Them Again

Evgenii Arsentev — Mon, 28 Sep 2026 14:51:11 +0000

The selection

A radar I run at arsentev.ai/github selects and reviews open-source AI projects from GitHub, Hugging Face and Hacker News, one page per project. Between 11 June and 12 September 2026 it selected 490 projects: 422 from GitHub, 63 from Hugging Face, 5 from Hacker News. By kind that's 427 repositories, 48 Hugging Face models, 15 Hugging Face Spaces.

By month of selection: June 70, July 134, August 204, September (up to the 12th) 82.

Median GitHub stars at the moment of selection was 493. Top languages across the repositories: Python 135, TypeScript 104, JavaScript 51, Rust 19, Swift 16. On Hugging Face the most common tasks were text-generation 16, image-text-to-text 16, image-to-video 7, text-to-speech 6.

One snapshot so far

There is one snapshot of this data, taken on 12 September 2026, the last day of selection. Depending on the project, the gap between when it was selected and the snapshot ranges from 0 days to about three months.

On that snapshot, of the 427 GitHub repositories: 416 were still available, 10 deleted (the API returns 404), 1 archived.

Of the available repositories, 19% have no licence or a licence GitHub doesn't recognise. The breakdown: MIT 218, Apache-2.0 93, NOASSERTION (GitHub can't tell) 43, no licence file 36, AGPL-3.0 16, GPL-3.0 8.

71.6% had a push in the last 30 days.

The sample is curated, not random. It's what the radar picked, so it says nothing about GitHub as a whole.

Tracking going forward

From 12 September a script takes a daily snapshot of every selected project. For GitHub repositories that's stars, forks, open issues, licence, last push, archived status. For Hugging Face models and Spaces it's likes, downloads, last modified. Removed repositories stay in the data with their status attached, so the set doesn't drift toward only the ones that are still around.

The dataset is open, under CC BY 4.0, on Zenodo, with three files: items.csv, snapshots.csv and stats.json. Every number above comes from the build script running on those CSV files. A new version comes out on the 1st of each month, and the series DOI always points to the latest one: https://doi.org/10.5281/zenodo.22730450. The numbers in this piece are from the 2026-09-13 edition: https://doi.org/10.5281/zenodo.22730451.

Three checks before you build on one

Before picking up a repository from a list like this, or from any trending list, these are the three things I'd check myself:

does the repository still open
is the licence field on GitHub set to a known licence, not blank and not NOASSERTION
was there a recent push

None of these take more than a minute, and all three are visible right on the repository page, before you clone anything or read a line of code.

Evgenii Arsentev, PhD. The dataset and the radar pages are at arsentev.ai/github.

AI assistance: drafted and edited with an AI tool; the data, measurements and conclusions are my own.

All my agent's tests were green, and they told me nothing

Evgenii Arsentev — Sat, 26 Sep 2026 08:11:32 +0000

I'm CEO of AskDocDoc (telehealth) and in our company the software development is done by AI agents. This is about a fully green test run that told me nothing (and it was my own experiment).

The setup

I ran a coding agent on twelve fixed programming tasks under six context-clearing policies (a fresh session every 1, 2, 3, 4, 6 and 12 tasks). Six replicates each, same tasks, same order - 36 runs. Every run ended with the full test suite as a gate. My reasoning was simple - a cost comparison is worthless if the cheaper policy also did less work.

The gate reported 4,086 tests executed and 4,086 passed. Zero failures (not in one run, not in one condition). And in the same files the modelled cost per run went from $2.165 under the most aggressive clearing to $1.622 at the observed optimum, and back to $1.780 when the context was never cleared. That's a 33.5% spread between the most and least expensive of six conditions (it's the gap between the extremes of six means, not a property of one policy).

So if you read only the 36 green reports you conclude the six setups are equivalent. The cost data in the same deposit says they are not.

What I found when I looked closer

The 4,086 is a grand total. Per condition it was 701, 697, 687, 699, 661 and 641 tests. Per run it went from 97 to 127 - that's 30% between the least and most checked run. In my own report I wrote that the design held "the verification suite constant". It wasn't, and I corrected it in the paper.

The reason is boring. The task prompts told the agent to create the test file itself and run node --test. Nine of twelve prompts set a minimum number of cases (between five and ten) and none set a maximum. So the agent wrote the criterion and then satisfied it. A run that did less would not go red - it would just write fewer tests.

I also checked the tempting story - longer sessions did less work and the gate didn't notice. It's not supported by the data (it doesn't survive removal of one endpoint condition, and it doesn't survive Bonferroni correction over the twelve columns I looked at). So I report it as a negative result.

And zero failures constrains less than it feels. With the run as the unit, 0 in 36 is consistent with a per-run failure probability up to 8.3% at 95% confidence. With the test as the unit, 0 in 4,086 gives 0.073%. Both are correct, and they differ by two orders of magnitude. So a pass count means nothing until the record says what the unit was.

This isn't only my corpus. Nicolás Rocchia found a checker where a missing companion file leaves no record at all, so the same artifact reports 40/40 with the file and 38/38 without it - passing both times (his message to the W3C list). His reading - the report gets smaller, not worse.

What the record should have said

Every green mark in my runs was correctly reported (as far as the record shows). The problem is a property of the set, and no per-check field can carry that. So the proposal in my paper is one field and one list at the result-set level - did any check change state across the compared conditions, and which ones did.

The field needs three values, not a boolean - some, none and incomparable. In my runs the agent wrote a fresh suite every time, so no check exists across conditions. The honest value for my study is incomparable. This is the record it should have emitted:

{
  "compared": ["ucurve-p1", "ucurve-p2", "ucurve-p3",
               "ucurve-p4", "ucurve-p6", "ucurve-p12"],
  "check_set_origin": "generated_by_run",
  "discrimination": "incomparable",
  "shared_checks": 0
}

shared_checks says how many checks the conditions really have in common. check_set_origin says if the suite was fixed, generated_by_run or mixed. Under incomparable the list of checks that moved must be absent, not empty (an empty list is exactly what none emits, and the two must not be confused).

If you compare agent runs, the practical part is short. Log the test count next to pass/fail for every run. If the agent writes its own tests, cross-run comparison of verdicts is not just noisy - it's ill-defined. And a record that can't say incomparable will say none, and a reader will hear "equivalent".

One honest limit - my claim that the suite could have gone red comes from how the harness was built, not from a mutation experiment. The repair is to run one. The idea grew out of the W3C Agent Conformance discussion and nothing in it is adopted yet.

The paper with all the numbers is open on Qeios - A Suite That Does Not Discriminate (DOI 10.32388/0BV3Z8). My other measurements are at arsentev.ai/research.

Evgenii Arsentev, PhD
CEO, AskDocDoc

AI assistance: drafted and edited with an AI tool; the data, measurements and conclusions are my own.

97% of what my coding agent billed for was re-reading its own context

Evgenii Arsentev — Wed, 16 Sep 2026 07:11:32 +0000

Token counters tell you how much you spent. I wanted a different number: how much of what I paid for was new text the model actually produced.

So I measured it against a complete local log corpus of my own agentic coding work: 722 sessions, 150,902 model calls, 34.56 billion tokens.

By tokens billed:

Token class	Share of all tokens
Cache read (re-reading context)	97.05%
Cache write (storing context)	2.57%
Output (generation)	0.38%
Fresh input	0.01%

For every token the models produced, roughly 256 tokens of cached context were read back in.

Cache pricing softens that ratio, but not the conclusion. Under published per-token list rates:

Category	Share of cost
Re-reading context	55.89%
Writing context to cache	31.89%
Generating new text	12.18%
Fresh input	0.04%

Context handling 87.8%, generation 12.2%. Re-reading context alone costs 4.59x as much as everything the models wrote. And cache writing is not a rounding error: at 31.89% it is the second-largest line, larger than generation. Caching does not make context free, it moves part of the price to the moment state is stored.

Why it looks like this

An agent doesn't send a fresh question on every step. It sends the whole conversation so far: instructions, every file it read, every command it ran. Then it adds one step and sends everything again. A token that enters the context on step 1 of a 60-step run is paid for 60 times.

I put a number on that with a derived measure, input amplification: total input tokens billed in a session, divided by that session's peak context size. Across the 590 sessions with at least three model calls, the median is 23.7x, the 90th percentile 124.5x, the maximum 4,182x.

The median session pays for its peak working state about 24 times. One session in ten pays for it more than 125 times. A session's cost is not mainly set by how big its context is, nor by how much it writes, but by the product of context size and how many times that context gets traversed.

Cost is also concentrated: an estimated 80% of the corpus total falls on 21 of 722 sessions (2.9%), and sessions longer than 200 model calls - 8% of sessions - account for an estimated 92.8% of spend.

The part you can actually change

None of the above is a knob. How often you clear the context is.

I ran a controlled comparison: twelve fixed programming tasks under six context-clearing policies - a fresh session every 1, 2, 3, 4, 6 and 12 tasks - six replicates each, with the model, the tasks and their order held constant. 36 runs.

Clear after every	Mean modeled cost per run
1 task	$2.165
2 tasks	$1.854
3 tasks	$1.622
4 tasks	$1.704
6 tasks	$1.679
12 tasks (never)	$1.780

Cost is not monotone in session length. It falls, bottoms out at three tasks, then rises again.

Clearing after every task costs 33.5% more than clearing every third (p = 0.0022, Holm-adjusted 0.011).
Every second task: +14.3% (p = 0.0022, Holm 0.011).
Going from clearing every task to every third cuts cost by 25.1%.

The reason the constant-clearing end is expensive: a cache write costs 20x a cache read. Clear after every task and you keep paying to rebuild state you just threw away. At the optimum the spend splits 39.0% cache read, 32.3% cache write, 28.7% output.

Two things I want to state plainly rather than round off:

The middle is a plateau, not a point. Every fourth task is +5.1% against every third (p = 0.17), every sixth +3.5% (p = 0.45). The data cannot separate 3 from 4 or 6. "Clear every few tasks" is the finding; "clear every third" is just where the observed minimum landed.

Never clearing is not significantly worse. It comes out +9.7% against every third, p = 0.046, Holm-adjusted 0.14. An earlier version of this work claimed both extremes were significantly more expensive than the optimum. That claim is withdrawn for never clearing.

Measuring it yourself

I packaged the measurement as a small open-source tool, contextburn. It reads the transcripts your agent already writes on your machine and makes no network calls.

pip install contextburn
contextburn detail 24

It also runs as an MCP server, so the agent can check its own efficiency mid-session:

claude mcp add contextburn -- uvx contextburn mcp

Two traps, in case you write your own counter

1. Streaming logs record the same model call more than once. An early snapshot and a final record share one message id. Count both and you double the call; keep only the first and you halve the output. Take the element-wise maximum per message id.

2. Prices change the answer, and they will change yours. Both datasets behind this post were re-published with corrected costs, and the errors were mine. Cache reads had been priced at a rate that applies only to a newer model, and every cache write had been priced at the 5-minute rate when the logs show mostly 1-hour writes, billed at 2x input rather than 1.25x. Token counts, call counts and the composition table did not move at all. Every dollar figure did: the corpus split went from 83.5% / 16.5% to the 87.8% / 12.2% above. If you report a cost-weighted share, it is only as current as your price table - and worth re-deriving before you quote it anywhere.

Limits

This is a single-practitioner case study, not a sample of a population, and the dollar figures are a model applied to logs rather than an invoice.

Cache-write duration cannot be verified for part of one machine's logs. If every write were instead a 5-minute write, the corpus split would be 86.2% / 13.8% - the direction holds under every assumption I can test, only the size moves.

In the controlled runs, "no test failures" means tests the agent wrote itself, and the count differed by condition: 116.8 tests per run on average when clearing after every task, 106.8 when never clearing. The three-task reference condition was selected after the fact, as the observed minimum.

I'd like to see the same measurement run against other people's logs.

The data

Report, datasets and recomputation scripts are public:

Corpus report and dataset: 10.5281/zenodo.22759216
Peer-reviewed report: 10.32388/0BV3Z8
Context-clearing experiment, 36 runs: 10.5281/zenodo.22759217

Evgenii Arsentev, PhD - Chief Executive Officer. I measure how AI agents spend money, and publish the data and the scripts. More at arsentev.ai.

AI assistance: drafted and edited with an AI tool; the data, measurements and conclusions are my own.