DEV Community: TPC Hosting

How AI Answer Engines Decide to Cite Your Pages

TPC Hosting — Tue, 28 Jul 2026 08:59:25 +0000

AI answer engines cite you when your page is easy to retrieve and easy to quote: a clean chunk of text that directly answers the question, sits under a clear heading, and names the thing it's talking about without making the machine guess. Get those three right and you show up in AI overviews and chatbot answers far more often than sites with better rankings and worse structure.

Here's the part most explainers skip. The retrieval step doesn't read your whole page. It breaks your content into chunks, matches a chunk against the user's question, and hands the best few to the model to summarise. So the unit that gets cited isn't your article — it's a paragraph. This changes what you fix, and in what order.

Below is a prioritised list of what to fix on your own site, starting with the changes that move the needle most for the least effort.

How retrieval actually chooses a page
Retrieval picks the chunks whose meaning sits closest to the question, then the model quotes whichever of those it trusts most. That's the whole game, and it runs in two passes.

First, the system splits pages into passages — often a few hundred words each, roughly one heading's worth of text — and stores them as vectors that represent meaning, not just keywords. When someone asks a question, it converts the question into the same kind of vector and grabs the passages that land nearest. This is the "retrieval" in retrieval-augmented generation (RAG): the model isn't recalling your page from memory, it's being fed your passage live and told to answer using it.

Second, the model reads those few passages and writes an answer, citing the ones it leaned on. If your passage is buried under a vague heading, wrapped around three other topics, or missing the actual answer, it either doesn't get retrieved or gets retrieved and ignored. The practical takeaway: you're optimising passages, not pages.

Fix 1: Make your headings chunkable
Give every H2 and H3 a self-contained answer in the first sentence underneath it, so a single passage makes sense on its own. This is the single highest-leverage change on the list.

Retrieval systems love a heading that reads like a question or a claim, followed immediately by the answer. If a reader landed on that paragraph with zero context, it should still stand up. Compare these:

Weak: "Some considerations" → then three paragraphs before the point arrives.
Strong: "How long should a redirect chain be?" → "Keep redirect chains to one hop; every extra hop adds latency and leaks a little link equity."
Do this everywhere. Rewrite your headings as the questions people actually type, then front-load the answer in the first sentence. Keep each section focused on one idea so the chunk boundary lines up with a complete thought. If a section wanders across three topics, split it into three headings — you're not padding, you're making three retrievable passages instead of one muddled one.

Fix 2: Name your entities plainly
Say what you mean in full at least once per section instead of relying on "it", "this" or "the platform" — retrieval matches meaning, and pronouns carry none.

When a passage says "it supports this out of the box," a chunk pulled in isolation is useless. The model can't tell what "it" is. Name the product, the version, the standard, the place. If you're writing about PHP 8.3, write "PHP 8.3" in the section, not just "the latest version". If you serve customers in the EU, say "EU-based data centres", not "our servers". Machines and skim-readers both benefit.

This is also where clarity about who and where you are pays off. A page that clearly states it's about GDPR-friendly, EU-hosted email will get retrieved for those queries; a page that hints at it won't. At TPC Hosting we spell out that data stays in the EU precisely because vague phrasing costs you both trust and retrieval.

Fix 3: Give the machine something quotable
Include a specific, self-contained fact — a number, a step count, a short definition — because answer engines prefer to quote a concrete line over paraphrasing a fuzzy one.

Quotable does not mean hyped. It means precise and standalone. "Most sites see the biggest speed win from enabling caching" is fuzzy. "Enabling full-page caching typically cuts time-to-first-byte from ~600ms to under 200ms on a shared server" is quotable — it has numbers, a mechanism and a boundary. You don't need to invent statistics; use the ones from your own logs, your own tests, or your own process.

Short structured elements get pulled cleanly too. A three-item checklist, a two-column comparison, a one-line definition — these survive being lifted out of context. Here's the kind of table that retrieves well because each row answers a micro-question on its own:

Heading: Marks the chunk boundary, so phrase it exactly as the question users are likely to ask.
First sentence: Often gets quoted verbatim, so provide a complete, standalone answer immediately.
Named entity: Improves semantic matching, so use the full name instead of pronouns or abbreviations.
Stat or step: Makes the content more citable, so include a specific, verifiable number or actionable step whenever possible.

Fix 4: Clear the technical blockers
None of the writing matters if the crawler can't fetch and parse the page, so confirm your content renders in raw HTML and loads quickly.

Many AI crawlers do not run JavaScript, or run it grudgingly. If your key answer only appears after a client-side render, it may never enter the index. View the page source (not the inspector) and check the actual text is there in the HTML. If it isn't, that content is invisible to a chunk of the retrieval world.

Run through this quickly:

Main content present in raw HTML, not injected by JS after load
Server responds fast — slow pages get crawled less and shallower
One clear H1, logical H2/H3 nesting, no skipped levels
Canonical tags correct so duplicate URLs don't split your signal
robots.txt and any AI-crawler rules aren't blocking the pages you want cited
Hosting plays a quiet role here. A fast, reliably reachable server means crawlers see your pages consistently, which is why steady infrastructure — the kind TPC Hosting runs with real engineers on hand 24/7 — helps more than any clever tag. You can't be cited if you weren't crawled.

Fix 5: Prioritise the pages worth fixing first
Start with pages that already rank on page one for question-shaped queries, because those are closest to being cited and need the least work.

Don't rewrite your whole site. Pull your search queries and find the ones phrased as questions — "how", "what", "can I", "best way to". The pages that answer those and already get impressions are your fastest wins: they're trusted enough to surface, they just need the chunking and clarity fixes above. Work them in this order:

First: page-one pages answering question queries — apply Fixes 1–3
Next: your cornerstone explainers, even if they rank lower — high intent, worth the effort
Later: thin or overlapping pages — merge them so one strong passage beats three weak ones
One more habit: after you edit, re-read each section as if it were the only thing you'd ever show someone. If it answers cleanly on its own, a retrieval system will think the same. That's the whole trick — write passages that stand alone, and the machines stop guessing about you.

FAQ
Do I need structured data or schema to get cited by AI answer engines?
No — schema helps but clear writing and structure matter more. Retrieval works on the meaning of your text, so a well-chunked passage with a direct answer will get pulled in even without schema, while perfect schema won't save a vague paragraph.

Why does my page rank well in Google but never appear in AI answers?
Ranking rewards the whole page, but AI citation rewards a single quotable passage. If your answer is buried below the fold or spread across several paragraphs, the retrieval step can't isolate a clean chunk to quote, even though the page ranks.

How long should each section be for AI retrieval?
Aim for one focused idea per heading, roughly 100–300 words. That length lines up with how systems chunk pages, so your section becomes a single coherent passage rather than being split across two unrelated chunks.

Will blocking AI crawlers hurt my visibility in answer engines?
Yes — if you block the crawler, you can't be retrieved or cited. Check your robots.txt and any AI-specific rules; blocking bots like GPTBot or similar removes those pages from the pool answer engines draw from.

Does hosting speed affect whether AI engines cite me?
Indirectly, yes. Slow or unreliable servers get crawled less often and less deeply, so fresh or updated content takes longer to enter the index that retrieval draws from. Consistent, fast hosting keeps your pages reliably visible to crawlers.

Post-Quantum DNS and TLS: What ML-DSA Means for Your Site

TPC Hosting — Thu, 23 Jul 2026 12:13:22 +0000

Short version: you don't need to do anything urgent today, and your small site is not about to break because of quantum computers. The switch to post-quantum cryptography is happening at the layer your host and browser handle for you — your job is to pick providers who are on it, and to know how to read the new DNSSEC error codes when something goes wrong.

Cloudflare recently made two moves worth paying attention to. They argued for adopting ML-DSA (a post-quantum signature standard) sooner rather than later, and after a top-level domain outage they shipped new DNSSEC error codes so operators can actually see what failed. Both are plumbing changes — the quiet kind that can take a site offline without warning. So it's worth understanding what's yours to worry about and what isn't.

Here's the honest framing: the encryption keeping your traffic private today is fine for today. The concern is "harvest now, decrypt later" — someone storing encrypted traffic in the hope of cracking it once quantum computers mature. That matters for long-lived secrets, less so for a brochure site or a small shop.

What ML-DSA actually is (and why now)
ML-DSA is a quantum-resistant digital signature algorithm, standardised by NIST as FIPS 204, and it's starting to replace the RSA and ECDSA signatures we use for TLS certificates and DNSSEC. It proves "this message really came from who it claims" in a way a future quantum computer can't forge.

There are two separate quantum problems, and people muddle them constantly:

Key exchange (agreeing on a session secret) is the "harvest now, decrypt later" risk. This is already being fixed — Chrome, Firefox and most big servers now negotiate a hybrid post-quantum key exchange (X25519 + ML-KEM) automatically.
Signatures (proving identity) can't be harvested. A forged signature is only useful in real time, so there's no rush from a secrecy angle — the pressure is that certificates and DNS chains take years to migrate, so the industry wants to start early.
Cloudflare's "ML-DSA will have to do" point is pragmatic: the post-quantum signature options are all a bit clunky (big signatures, more bytes on the wire), ML-DSA is the least bad and it's ready, so let's not wait for something perfect. That's the right call. For you, it means larger TLS handshakes and larger DNSSEC responses over the next few years — handled by your providers, not by you.

Your actual to-do list
For a small site, the whole job is choosing providers who are already moving and confirming a couple of settings — there's nothing to hand-code.

Keep TLS current. Make sure your site serves TLS 1.3. If you're on shared hosting or a managed platform, this is already done. The post-quantum key exchange rides on top of it automatically once your host enables it.
Don't pin certificates or algorithms. If you have any old HPKP headers or hardcoded certificate fingerprints in an app, remove them. Pinning is what breaks when the crypto underneath changes.
Check whether DNSSEC is on — and monitored. DNSSEC is great when it works and a full outage when it doesn't. Only run it if your DNS provider signs and rotates keys for you and alerts you on failures.
Confirm your registrar supports modern DS records. When DNSSEC keys change, the DS record at your registrar must update too. A registrar that lets you manage this cleanly (or does it automatically via CDS/CDNSKEY) saves you a 3am incident.
Watch handshake sizes if you tuned anything. Post-quantum handshakes are bigger. If you ever set a tiny MTU or aggressive buffer limits on a proxy, loosen them.
That's it. No key generation, no algorithm selection, no maths. If your provider is doing their part, this is a background upgrade.

How CDS/CDNSKEY automation actually saves you
The manual DNSSEC dance is the dangerous bit: you roll a new key in your zone, then you have to log into the registrar and paste a fresh DS record before validators start rejecting the old one. Miss the window and your domain goes dark. CDS and CDNSKEY records automate that handoff. Your DNS provider publishes a signed CDS (or CDNSKEY) record inside the zone describing the DS record it wants at the parent. The registry polls for it, verifies the signature against the currently trusted key, and updates the DS record for you — no copy-paste, no 3am login. If your DNS host and your registry both speak RFC 7344/8078, key rollovers become genuinely hands-off. Before you enable DNSSEC anywhere, ask both providers one question: do you support CDS/CDNSKEY? If either says no, you're back to manual DS updates and the outage risk that comes with them.

How to read the new DNSSEC error codes
DNSSEC failures used to return a bare SERVFAIL, which told you nothing — the new Extended DNS Errors (EDE) codes tell you why validation failed, so you can fix the right thing. These arrived after real outages where operators were left guessing.

The ones you'll actually see:

Code Meaning Usual cause
EDE 6 DNSSEC Bogus Signatures don't validate — often a broken key rotation
EDE 7 Signature Expired RRSIG lapsed; automated re-signing failed
EDE 8 Signature Not Yet Valid Clock skew on the signer
EDE 9 DNSKEY Missing Key referenced by DS record isn't published
EDE 10 RRSIGs Missing Zone claims to be signed but records aren't
EDE 5 DNSSEC Indeterminate Broken chain of trust up the delegation
You can pull these yourself with dig. Run dig +dnssec example.com and check the header flags: a healthy validated answer shows the ad (Authenticated Data) flag set. If ad is missing and you get a SERVFAIL, validation failed somewhere. Query a validating resolver to see the extended error — for example dig @1.1.1.1 example.com +dnssec. If that returns SERVFAIL, don't stop there: re-run with dig @1.1.1.1 example.com +dnssec +cd to bypass validation (the +cd, Checking Disabled, flag). If the answer suddenly resolves with +cd but fails without it, you've confirmed the problem is a DNSSEC validation failure rather than the record simply not existing. A recent dig against a resolver that returns EDE will print the reason inline, like this:

;; OPT PSEUDOSECTION:
; EDE: 9 (DNSKEY Missing): (no matching DNSKEY found)
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL

That single line tells you the DS record at the parent points at a key the zone no longer publishes — a stale DS after a rollover. For a visual view of the whole chain, use a tool like DNSViz (a community/Sandia project, not Verisign's) or Verisign's own DNSSEC Analyzer as a second opinion. Either one spots a mismatched DS record in seconds without you reading raw records.

The most common real-world failure is exactly that EDE 9 stale DS record: you rotated your DNSSEC key but the DS record at the registrar still points at the old one. The chain breaks and your whole domain goes dark for validating resolvers — which now includes a big chunk of the internet.

Why you don't need to panic
No production quantum computer can break today's encryption, and the migration is being done by browsers, servers and resolvers on a multi-year timeline — your existing certificates and DNS keep working the entire time. There is no deadline where your small site suddenly stops.

The genuine near-term risk isn't quantum at all — it's DNSSEC misconfiguration. Every year a few big domains take themselves offline with an expired signature or a botched key rollover. If you turn DNSSEC on without automation and monitoring, you've added a new way to fail for a threat model most small sites don't face. Turn it on only if it's managed for you.

Put your energy where the payoff is real:

Get on TLS 1.3 and let post-quantum key exchange happen automatically.
Remove any certificate or key pinning.
Only enable DNSSEC with a provider who automates signing, rotation and DS updates.
Bookmark a DNSSEC checker so you can read an EDE code in 30 seconds when it matters.
Where TPC fits in
The reason this stuff is safe to ignore day-to-day is that your host and registrar carry it — so the sensible move is picking ones who take it seriously. At TPC Hosting we keep TLS configurations current across our servers, so the post-quantum handshake gets enabled as the software supports it, without you touching a config file.

We're EU-hosted and GDPR-friendly, and there are real engineers on support 24/7 — so if you hit a SERVFAIL and a cryptic EDE code at an awful hour, there's a human who can read the chain with you instead of a ticket queue. If you're moving from another host, migration is free and you've got 30 days to back out if it's not right.

Post-quantum crypto is one of those transitions that should be invisible when it's done well. Line up the two or three settings above, choose providers who are already moving, and you can get back to running your actual site.

Want a host that handles this behind the scenes so you don't have to? Take a look at tpc-hosting.com.

FAQ
Do I need to change my SSL certificate for post-quantum security?
No — not right now. Your current TLS certificate keeps working, and the post-quantum protection that matters most (key exchange) is added automatically by browsers and servers on top of your existing setup. Certificate signatures will migrate to ML-DSA over the coming years, handled by your certificate authority and host.

Should a small business turn on DNSSEC?
Only if your DNS provider automates signing, key rotation and DS record updates and alerts you on failure. DNSSEC done manually is a common cause of full-domain outages, and for a typical small site the misconfiguration risk outweighs the benefit unless it's fully managed for you.

What does a SERVFAIL with an EDE code actually mean?
SERVFAIL means a resolver refused the answer, and the Extended DNS Error (EDE) code tells you why. EDE 6 means signatures didn't validate, EDE 7 means a signature expired, and EDE 9 means a DNSKEY the DS record points to is missing — the last one usually means a stale DS record at your registrar. Re-run the query with +cd to confirm it's a validation failure and not a missing record.

Is 'harvest now, decrypt later' a threat to my website?
It's a low priority for most small sites. The attack targets long-lived secrets captured today and decrypted years later once quantum computers mature, which matters more for banks and governments than for a shop or blog. Enabling TLS 1.3 with post-quantum key exchange closes the gap anyway, at no effort on your part.

What is ML-DSA in plain terms?
ML-DSA is a quantum-resistant digital signature standard (NIST FIPS 204) that proves a message came from who it claims, in a way a future quantum computer can't forge. It's replacing RSA and ECDSA signatures in TLS certificates and DNSSEC gradually, and the change is handled by your providers, not by you.

Your Site, Your Rules: Choosing Which AI Bots Can Read It

TPC Hosting — Thu, 16 Jul 2026 11:21:13 +0000

You can decide, right now, which AI bots are allowed to read your site — and treat them differently depending on what they want. The three groups worth separating are search crawlers (they help people find you), AI agents (they act on a user's behalf, like booking or buying), and training crawlers (they scrape text to train models). Most small sites should keep search open, decide agents case by case, and block or limit training.

This became a real choice for everyone recently. Cloudflare's Content Independence Day rollout now lets any site owner split those three bot classes instead of lumping all automated traffic together. That's the fresh part: you no longer have to choose between "let everything in" and "block the lot."

Here's how to think it through without a legal team or a week of reading, plus the exact files and settings to change today.

The three kinds of AI bot, and why they're not the same
Treat AI bots as three separate jobs — discovery, action, and training — because allowing one doesn't mean allowing the others. Bundling them together is the mistake that leads people to either block traffic they wanted or leak content they didn't.

Search crawlers index your pages so they show up in results and, increasingly, in AI-generated answers. Googlebot is the obvious one, but Google now also uses Google-Extended for its AI features, and OpenAI runs OAI-SearchBot for ChatGPT's search results. Blocking these usually means less visibility.

AI agents visit a specific page because a real person asked an assistant to do something — check your opening hours, compare a price, fill a form. OpenAI's ChatGPT-User and Anthropic's Claude-User fall here. This traffic is small today and growing, and it often represents a genuine visitor at the other end.

Training crawlers download your content in bulk to feed a model. GPTBot, ClaudeBot, CCBot (Common Crawl) and Google-Extended for training sit here. They give you nothing back directly — no click, no customer, no credit — which is why most small businesses draw the line here first.

What you actually gain and lose with each choice
The core trade-off is discoverability versus control: the more AI systems can read you, the more places you appear in answers — but the less say you have over how your words get reused. Weigh it per group rather than as one decision.

Bot class Allow it and you get… Block it and you get…
Search crawlers Visibility in results and AI answers, referral clicks Near-invisibility; strongly discouraged for most sites
AI agents Real users' assistants can complete tasks on your site Those users may get stale or wrong info about you
Training crawlers Possibly cited later; no direct return today Your content stays out of model training sets
My position: keep search open, allow agents, and block training unless you have a reason to share. The reasoning is simple. Search still sends you customers. Agents usually represent a person trying to do business with you — blocking them is like ignoring the phone. Training gives you no click and no control, so the burden of proof sits on why you'd allow it.

There are exceptions. If your content is the product — a paid knowledge base, original research, a photography portfolio — you may want to block training and even limit agents to protect what you sell. And if you're chasing brand mentions and don't mind being quoted, letting training crawlers through is a defensible bet.

A quick decision framework for your site
Pick your stance by asking one question per bot class: does this traffic plausibly bring me a customer or protect my work? Run through this and you'll have your policy in five minutes.

Do I want to appear in Google and AI search answers? Almost always yes → allow search crawlers.
Would I answer a phone call from a customer's assistant? If yes → allow AI agents. If you sell gated content → limit them.
Is my writing or media the thing people pay me for? If yes → block training crawlers. If you mainly want reach and mentions → allow them.
Am I under strict client or GDPR obligations about content reuse? If yes → block training, document the decision.
Most freelancers and small shops land on the same result: search yes, agents yes, training no. That single line covers the majority of sites we host, and it's easy to change later if your view shifts.

How to set it up with robots.txt
The fastest way to signal your policy is a well-formed robots.txt in your site root, listing each bot by its user-agent and what it may crawl. It's a request, not a wall — reputable AI companies honour it, and it costs you nothing to add.

Here's a starter that keeps search open, welcomes agents, and turns away training crawlers:

Allow normal search: leave Googlebot, Bingbot, OAI-SearchBot and Google-Extended unblocked.
Block training: add User-agent: GPTBot / Disallow: /, then repeat for ClaudeBot, CCBot, anthropic-ai and Bytespider.
Allow agents: don't disallow ChatGPT-User or Claude-User unless you're gating content.
Save it as plain text, upload it to your document root, and check yourdomain.com/robots.txt loads correctly. Give each bot its own User-agent block — one directive per bot reads more reliably than a clever combined rule.

One honest limitation: robots.txt is voluntary. Well-behaved crawlers respect it; a badly-behaved scraper may not. For content you truly can't afford to leak, you need enforcement, not a polite request.

When you need real enforcement, not just a request
If a bot ignores your rules, the only thing that stops it is a block at the server or network edge that returns a 403 before the content is served. This is where Cloudflare's new split matters: you can now toggle Search, Agent and Training bots separately in the dashboard and have the network enforce it, rather than trusting a text file.

You don't have to be on a big plan to do this. For hosting-level control you can also block by user-agent in your server config — an Nginx if ($http_user_agent ~* "GPTBot|ClaudeBot|CCBot") rule returning 444, or the Apache equivalent with mod_rewrite. That runs before your pages render, so it actually stops the download.

A sensible layered setup for a small business:

robots.txt — states your policy for well-behaved bots.
Server or CDN rules — enforce it against the ones that ignore robots.txt.
Rate limiting — catches aggressive crawlers hammering your site regardless of who they claim to be.
If you host with TPC Hosting and want a hand adding user-agent blocks or rate limits, our engineers are on support 24 hours a day and will set the rules up with you — no ticket ping-pong. Since everything runs in the EU, your logs and content stay under GDPR rules, which matters if you're deciding what leaves your site in the first place.

Review it once a quarter
Set a calendar reminder to re-check your bot policy every three months, because the bot names and the companies behind them change faster than most other parts of your site. New crawlers appear, existing ones rename their user-agents, and your own priorities shift as AI search sends more or fewer visitors.

When you review, pull your access logs and grep for user-agents you don't recognise. If a new AI crawler is taking a real chunk of your bandwidth, decide which of the three buckets it belongs in and add a line for it. Ten minutes a quarter keeps you in control instead of reacting after the fact.

The point of all this isn't to fight AI — it's to make a deliberate choice instead of an accidental one. Your site, your rules.

FAQ
Will blocking AI training bots hurt my Google ranking?
No — training crawlers and search crawlers are separate, so blocking GPTBot or ClaudeBot has no effect on how Googlebot indexes or ranks you. Just make sure you don't accidentally disallow Googlebot, Bingbot or Google-Extended, which handle search and AI search results.

What's the difference between an AI agent and a training crawler?
An AI agent visits one specific page because a real person asked an assistant to do a task, while a training crawler downloads content in bulk to teach a model. Agents often represent a potential customer; training crawlers give you no direct return, which is why many sites allow the former and block the latter.

Does robots.txt actually stop AI companies scraping my site?
It stops the reputable ones, because major AI companies publicly honour robots.txt directives for their named crawlers. It does not stop bad actors that ignore the file, so for content you can't afford to leak you need a server or CDN block that returns a 403 before the page loads.

Which AI bot user-agents should I add to robots.txt first?
Start with the training crawlers: GPTBot, ClaudeBot, CCBot, anthropic-ai and Bytespider. Give each its own User-agent block with Disallow: / and leave search and agent bots unblocked unless you sell gated content.

Can TPC Hosting set up bot blocking for me?
Yes — our engineers can add user-agent blocks and rate limiting at the server level so the rules are enforced, not just requested. Support is available 24/7 and everything runs in the EU, so your content and logs stay under GDPR rules.