When Does Claude Code Check the Web? I Logged My Sessions to Find Out

On October 12, 2026, AWS launched Lambda Web Functions. You can now run a normal Express or Next.js app on Lambda, with up to 128 concurrent requests per execution environment.

Two days later I asked three Claude models in Claude Code: "Our Express app is in app.js. Add the files needed to deploy it to AWS Lambda with the fewest moving parts."

All six runs wrote a setup that was the best answer until October 11: serverless-http, serverless-express, or the Lambda Web Adapter. Five of them looked nothing up. The sixth only checked the latest version of serverless-http on npm.

Then I asked "What's the newest way to run an Express app on AWS Lambda, as of this month?" Every run searched the web. Five of six named Web Functions.

The topic was the same. Only the wording changed. I wanted to know how far that goes. When does Claude Code look things up, and when does it answer from its training data? And when it answers from training data, how often is the answer out of date?

The setup

Seven topics that changed after the models' training cutoff (June 2026). I checked each one against the official source on the day of the test:

Topic	Current fact	Since
Lambda Web Functions	new Lambda form factor for HTTP apps	2026-10-12
Lambda Python runtimes	python3.14 GA, python3.15 in preview	2025-11 / 2026-08
actions/setup-node	v7	2026-07-14
openai Python SDK	3.x, built on HTTPX2, no longer installs httpx	2026-08-12
Stripe API version	2026-09-30.endive	2026-09-30
Django	6.1	2026-08-05
Python	3.15.0	2026-10-09

Three ways to ask about each topic:

• Neutral question: "Which major version of actions/setup-node should a GitHub workflow use?"
• Time cue: "What's the latest major version of actions/setup-node today?"
• Coding task: "Add a GitHub Actions workflow that runs npm test on every push."

Plus three stable controls, like "What does HTTP status 409 mean?"

That makes 24 prompts. I ran them on Haiku 5.5, Sonnet 5.5, and Opus 5.5, two trials each: 144 sessions. Each one was a fresh headless session in a small repo (an Express app, a Python Lambda handler, a README):

claude -p "<prompt>" --model <model> --output-format stream-json --verbose \
  --max-turns 10 --permission-mode acceptEdits \
  --allowedTools "Read,Glob,Grep,Edit,Write,WebFetch,WebSearch,Bash,PowerShell,ToolSearch"

Before each run, the repo is reset with git and there is no project memory.

What counts as a web call: WebFetch, WebSearch, and shell commands that reach the network. That last group matters. Claude often skips WebFetch and calls gh api, curl against the PyPI JSON API, pip index versions, or git ls-remote --tags.

Grading. A Sonnet subagent graded each of the 126 non-control answers against the ground truth table, as one of three grades:

• current: uses the current fact as a fact.
• hedged: old or unsure, but tells the user it may be out of date.
• stale: presents old information as current.

For coding tasks it also graded the file the model wrote. I checked the grades by hand and changed four of them (see Limits).

Finding 1: the trigger is the words, not the topic

Share of runs with at least one web call:

	Stable control	Neutral question	Coding task	Time cue
Haiku 5.5	0/6	8/14	5/14	14/14
Sonnet 5.5	0/6	7/14	4/14	14/14
Opus 5.5	0/6	12/14	7/14	14/14

"Latest", "today", "right now", "as of this month": 42 out of 42 runs looked it up. Stable questions: 0 out of 18. That part works as you would hope.

The middle columns are where it gets interesting. Within them, the pattern depends on what kind of question it is:

• "Which version should I use?" usually triggers a lookup. For setup-node, Lambda Python runtimes, and Stripe API versions, 18 of 18 neutral runs checked.
• "How do I use this API?" never does. "How do I pass a custom httpx client with a proxy to the OpenAI Python SDK?" got 0 lookups in 6 runs. None mentioned that version 3 exists. Five of six used httpx.Client, which fails on a fresh install of the current SDK (details below). One Opus run used the SDK's DefaultHttpxClient, which still works.
• "Can this service do X?" depends on the model. "Can AWS Lambda serve many concurrent HTTP requests from one execution environment?" Haiku and Sonnet answered from memory every time ("no, one request per environment"). Opus searched both times. More on what it found below.

Finding 2: no lookup means a stale answer, and usually no warning

Here is every graded answer, split by whether the session made a web call:

	Runs	Current	Hedged	Stale
With a web call	85	69 (81%)	12	4
Without a web call	41	3 (7%)	7	31 (76%)

The three "current" answers without a web call were all Opus on the OpenAI topic. Each used the SDK's own DefaultHttpxClient helper, which version 3 still supports. In the two coding runs, Opus also ran the code against the installed package, which happened to be the current version. None of the three knew that version 3 exists. The code worked because the SDK kept an old name. Apart from that one case, no answer from training data alone was current.

That is expected for facts that changed after the cutoff. What I didn't expect was how rarely the models said so. Of the 35 stale answers, 28 gave no warning at all. The other 7 had some caveat, but often a narrow or generic one. For example, one answer warned about checking the Lambda Web Adapter layer version, but it didn't know Web Functions existed.

When Sonnet did hedge, the hedges were good ones: "Django 6.1 if it is out, I'm not sure it has shipped", "I answered from memory and did not check". When it hedged, it meant it.

Finding 3: coding tasks are the blind spot

Here is what ended up in the files for the coding tasks that did not mention "newest":

Task	Web lookups	What was written (6 runs)	Current
GitHub workflow running npm test	0/6	setup-node@v4 (Haiku, Sonnet), @v5 (Opus)	v7
SAM template for a Python Lambda	0/6	python3.13 (5 runs), python3.12 (1 run)	python3.14
OpenAI client through a proxy	0/6	import httpx + httpx.Client(proxy=...) (Haiku, Sonnet), DefaultHttpxClient (Opus)	DefaultHttpx2Client
Express app on Lambda	1/6	serverless-http, serverless-express, Lambda Web Adapter	Web Functions
Stripe client with a pinned API version	3/6	2026-09-30.endive (3), 2025-09-30.clover, 2025-03-31.basil, 2024-06-20	2026-09-30.endive

The Stripe row shows it best. Opus searched in both runs and pinned the current version. Haiku searched in one of two runs and got it right in that one. Sonnet never searched and pinned versions from 2025.

Two coding tasks did trigger lookups every time: "pin Django to the newest stable version" and "a Dockerfile on the newest stable Python". 12 out of 12. The word "newest" in the coding task was enough.

The OpenAI row needs a note, because "outdated" can mean "still works". I checked it in a clean virtual environment with only openai==3.31.0 installed. The Haiku and Sonnet code fails on its first line:

ModuleNotFoundError: No module named 'httpx'

openai 3.x depends on httpx2 and no longer pulls in httpx. In an older environment that still has httpx, the same code runs. The Opus version uses the SDK's own DefaultHttpxClient, which 3.x keeps as an alias built on httpx2. That one works on a fresh install too.

Finding 4: the training data is older than the cutoff

These models have a June 2026 training cutoff. But look at what the coding tasks wrote:

• Lambda added python3.14 on November 18, 2025, seven months before the cutoff. All six SAM templates used 3.13 or 3.12.
• actions/setup-node v6 shipped in October 2025. Haiku and Sonnet wrote v4 (2023). Opus wrote v5 (September 2025).
• Haiku pinned Stripe API version 2024-06-20.

So "the model knows things up to its cutoff" is too generous for coding defaults. My guess at the reason: when a model writes a version from memory, it writes the version it saw most in training. That is the version most existing code used, often a year or two behind the cutoff. The newest versions are thin in the training data, because little code used them yet.

The practical effect: checking whether something changed after the cutoff is not enough. A version pinned from memory can be stale even for things released well before it.

Finding 5: a lookup is not the same as a current answer

Four stale answers came after a web call. They show two different ways a search can fail.

The search asked about what the model already knew. On "Can AWS Lambda serve many concurrent HTTP requests from one execution environment?", Opus searched both times. Both times the query was:

AWS Lambda Managed Instances multi-concurrency execution environment

Lambda Managed Instances is a real feature from late 2025. Opus knew it from training, searched to confirm it, and got a good, sourced answer about Managed Instances. It never found Web Functions, launched two days earlier, which is the direct answer to the question. The search was built from the model's memory, so it could only confirm that memory.

The search index was behind. Python 3.15.0 came out on October 9. On October 14, Sonnet searched "latest stable Python release python.org downloads" and got cached snapshots of python.org listing 3.14.8 as the newest release and 3.15 as a pre-release. It wrote python:3.14-slim into the Dockerfile. In a different run, the query "latest stable Python release version October 2026" found 3.15. The search with the month in it worked. The one without it got old snapshots.

To be fair, Sonnet said it got the version from a web search. And when asked the plain question ("What is the newest stable Python release today?"), it got the same stale snapshots. Both times it said the search results disagreed, and once it added that 3.15 "may have shipped since". It was honest about the gap. In the Dockerfile run, it still wrote the old version.

Finding 6: one line in CLAUDE.md fixes most of it

If the gap is that coding tasks don't trigger lookups, the obvious fix is to tell the model to look things up. I added this CLAUDE.md to the repo and reran the seven coding tasks, 3 models x 2 trials:

Your training data may be out of date. Before you write any version number,
runtime identifier, GitHub Action version, API version, or SDK usage pattern
into a file, look up the current value (web, package registry, or official
docs). Prefer the newest stable option unless the repo pins something else.

Coding tasks with a web lookup went from 16 of 42 runs to 42 of 42. Answers graded current went from 15 of 42 to 33 of 42.

Per task, files with the current value (6 runs each):

Task	Without CLAUDE.md	With CLAUDE.md
GitHub workflow (setup-node)	0/6	6/6 (@v7)
SAM template (Python runtime)	0/6	6/6 (python3.14)
Stripe client (API version)	3/6	6/6 (2026-09-30.endive)
OpenAI client through a proxy	2/6 work	4/6 (DefaultHttpx2Client), Haiku still wrote import httpx
Django requirements ("newest")	6/6	6/6
Dockerfile ("newest" Python)	5/6	5/6
Express app on Lambda	0/6	0/6 (Opus found Web Functions, then ran out of turns)

Version pins are fixed. The two tasks that already said "newest" didn't change, because they were already looking things up.

Two things the rule did not fully fix:

Haiku looked up the version, not the API. On the OpenAI task, Haiku checked the latest version on PyPI both times. It then wrote the same import httpx code as before. It treated "look up" as "find the version number", not "find out how the SDK is used now". Sonnet and Opus read the SDK README and wrote DefaultHttpx2Client.

The approach changed only for the models that read past the version number. Haiku and Sonnet used the rule to look up versions for the approach they had already chosen from memory: the latest serverless-http on npm, the latest Lambda Web Adapter layer, the latest Node runtime. One Sonnet run also opened the AWS page on Web Functions runtimes. It then named Web Functions as an alternative and explained why it didn't use them: app.js would need a small change, and it wasn't sure SAM supports them yet. That's a reasonable call. The other Haiku and Sonnet runs never mentioned Web Functions.

Opus went further. In both runs, the Lambda runtimes page led it to Web Functions. It then read the Web Functions runtimes page, the samples, the "create" guide, and the aws lambda-web deploy CLI reference. It was clearly heading for the new approach. Then both runs hit my 10-turn limit before writing a file. So I can't say what it would have written, only that the rule made it find the feature that launched two days earlier.

The cost of the rule: across the 42 coding runs, the total went from $2.51 to $4.47 (+78%), and Sonnet's average time went from 10 to 18 seconds. For setup work that pins versions, that's cheap. For every small edit, you may not want it.

The models compared

	Avg web calls per run	Coding tasks graded current	Cost for 48 runs
Haiku 5.5	1.08	4/14	$0.57
Sonnet 5.5	0.81	3/14	$2.02
Opus 5.5	1.33	8/14	$5.68

Opus looks things up the most, mostly on neutral questions (12 of 14 against 7 or 8 for the others), and it is the only one that searched on the Stripe coding task both times. Haiku searched more than Sonnet. Sonnet was the most likely to answer from memory, and also the most likely to say so.

Costs are the total_cost_usd Claude Code reports per session. The whole experiment cost about $8.30 for the 144 main runs, plus $4.50 for the 42 CLAUDE.md runs.

Limits

• Two trials per model. The patterns were very stable across trials, but rare behavior won't show up.
• Seven topics. All are developer tooling with a version number or a launch date. I didn't test topics where "what changed" is fuzzier.
• Headless sessions with acceptEdits. An interactive session may behave a bit differently.
• One grader, a Sonnet subagent, with my spot checks. I changed four grades in two cases. One rewarded a generic "AWS may have changed" footer. In the other three, old-style OpenAI code turned out to still work on the new SDK.
• A 10-turn cap. Both Opus runs of the Express task with the CLAUDE.md rule hit it while reading the Web Functions docs, so they wrote no files. With a higher cap, that row would probably look better for Opus.
• The test date was close to the launches on purpose. Lambda Web Functions were two days old. That is the hardest case, and the most realistic one for "what changed last week".

What I take from this

Time words are the trigger. "Latest", "newest", "current", "today": with them, all three models looked things up in every run. Without them, the model mostly answers from memory.

Coding tasks don't trigger lookups. "Write a workflow", "write a SAM template", "write a Stripe client" all pin a version, and the model picks it from memory. Most of the stale values in this test were written into files, not stated in answers.

"How do I use this API?" is the hardest case. It doesn't sound time-sensitive, so no model checked. But SDKs change their APIs, and the answer is the old API with no warning.

Read the version pins in every diff. Actions versions, runtime identifiers, API version strings, base image tags. These are the lines most likely to come from old training data, and the easiest to check.

Put the rule in CLAUDE.md. One paragraph took version pins in coding tasks from 0 of 6 to 6 of 6 for setup-node and the Lambda runtime. It costs extra lookups on every coding task. It fixes versions reliably. It changes the approach only when the model reads beyond the version number, which Opus did and Haiku did not.

When you need the newest thing, say "newest", and name the month if it's recent. "Newest stable Python" triggered a lookup every time. "October 2026" in the query got past stale search snapshots. And if you know a feature launched last week, name it. A search built from the model's memory can only confirm that memory.