DEV Community: msm yaqoob

Server Side Rendering for AI Crawlers: What Actually Needs to Change

msm yaqoob — Sat, 29 Aug 2026 07:52:04 +0000

If your application renders client side, AI crawlers are probably receiving an empty document. This is a rendering architecture problem rather than a content problem, and the fix lives in your build configuration rather than your CMS.

Diagnose before you refactor

Fetch your own page the way a crawler does, without a browser engine:

curl -A "GPTBot" https://yoursite.com/your-page -o page.html

Then count the words in the body:

sed -e 's/<[^>]*>//g' page.html | tr -s ' \n' ' ' | wc -w

Under a hundred words on a content page means the document you are serving is a shell. Compare that against the same page in a browser to see the gap.

Route selection

Static generation builds HTML at deploy time. Fastest possible delivery, no server rendering cost per request, and the right answer for marketing pages, documentation and blogs where content does not vary per user. In Next.js this is generateStaticParams with default static rendering. In Nuxt it is nuxt generate. In SvelteKit it is the static adapter with prerender enabled.

Server rendering builds per request. Necessary when content is personalised or changes frequently. Costs server time per request, so cache aggressively at the edge.

Selective prerendering serves prebuilt snapshots to bots by user agent while humans continue to receive the SPA. Pragmatic when a full migration is not realistic, though it introduces a divergence between what bots and users see, which needs care to avoid looking like cloaking. Keep the content identical and only the delivery mechanism different.

The parts people miss

Streaming and suspense boundaries. Server rendering with heavy suspense can still deliver a shell for the content that matters if the important component is inside a boundary that resolves late. Check what is in the initial flush rather than assuming the framework handles it.

Client-only data fetching in server components. Mixing patterns leaves you with a technically server-rendered page whose actual content still loads in an effect. The audit is the same: read the raw HTML.

Structured data injected by script. JSON-LD added through a client-side tag manager is not in the served document. Put schema in the server response.

Meta tags set dynamically. Title and description assigned by a client-side head manager may be absent from the initial HTML. Framework head APIs that run server side solve this.

Verify after deploying

Rerun the curl test. Then confirm the important elements survive: title, meta description, H1, body copy, JSON-LD. If any are missing, they are still being added after delivery.

A free browser-based version of this analysis is at JavaScript Rendering Checker if you would rather paste the source than run shell commands, and the wider treatment of SSR for AI crawlers covers the strategy side.

Why this is worth doing now

Search engines that render JavaScript have made client-side rendering survivable for a decade. Retrieval systems that do not render have made it costly again. The remedy is the same one that improves your Largest Contentful Paint, so the performance case and the visibility case point the same direction.

Building an AI Automation Pipeline Without a Backend

msm yaqoob — Mon, 24 Aug 2026 07:53:04 +0000

Most AI automation content assumes you are either writing production code or clicking through a no code builder with no understanding of what it does underneath. There is a useful middle position, and it is where most small scale automation actually lives.

This walks through the architecture of a practical AI pipeline, the failure modes that show up once it runs unattended, and where the abstraction leaks.

The basic shape

Almost every useful AI automation follows the same four stage structure regardless of which platform you build it on.

Trigger, transform, model, dispatch.

Trigger

Something happens. A webhook fires, a row is added, a file lands, a scheduled interval elapses. The choice matters more than it appears, because polling based triggers introduce latency and consume quota, while webhook based triggers introduce ordering problems you have to handle explicitly.

Transform

Raw trigger payloads are almost never in the shape your prompt needs. This stage normalises the data, strips irrelevant fields, and enforces the structure the model expects. Skipping it is the single most common cause of unreliable output, because inconsistent input produces inconsistent results and the model is not the thing at fault.

Model

The inference call. Two decisions dominate here: whether you need structured output, and what you do when the response does not match the schema.

If downstream steps parse the response, request structured output explicitly and validate it before proceeding. Treating model output as trusted input is how these pipelines fail silently.

Dispatch

Write the result somewhere. Database, document, message, API call. Make this idempotent if there is any chance the pipeline reruns, because retries on non idempotent dispatch produce duplicates that are tedious to clean up.

[VIDEO SLOT] Insert the automation build video here, framed as: I built this pipeline end to end on camera, including the schema validation step and the retry handling, which are the parts that are hard to follow in written form.

Where these pipelines actually break

The failure modes are consistent enough to plan for.

Schema drift

The model returns valid text that does not match the expected structure. Usually triggered by an edge case input the prompt did not anticipate. Validate output against a schema and route failures to a review queue rather than letting them flow downstream.

Silent degradation

The pipeline keeps running and the output quality drops. This is worse than an outright failure because nothing alerts. Sample outputs periodically rather than assuming continued correctness.

Rate limits under burst

Works fine at one event per minute, falls over when fifty arrive at once. Queue and throttle at the transform stage rather than discovering the ceiling in production.

Cost drift

Token consumption scales with input size, and input size grows quietly as the source data grows. Set a hard budget alert. Automations that fail loudly are cheaper than automations that succeed expensively.

Prompt coupling

The prompt encodes assumptions about the input format. When the upstream source changes a field name, the prompt still runs and produces subtly wrong output. Version prompts alongside the schema they assume.

Structuring for maintainability
Keep prompts in one place rather than embedded across steps. When behaviour needs adjusting you want a single edit point.
Log the model input and output for every run, at least initially. Debugging without the actual payloads is guesswork.
Separate the deterministic and non deterministic parts. Anything that can be done reliably without a model should be, both for cost and for predictability.
Build the failure path before the success path. It is the one you will spend more time in.
When to write code instead

The no code platforms are genuinely good until they are not, and the boundary is fairly predictable. Move to code when you need conditional logic more than two branches deep, when the same transform appears in more than three workflows, when you need real error handling rather than retry counts, or when platform execution costs exceed what running the same thing yourself would cost.

Until then the abstraction is worth the tradeoff, particularly because most small scale automations do not run often enough to justify infrastructure.

The interesting engineering in AI automation is almost never the model call. It is the validation, the retry logic, and the boundary conditions around it.

A reasonable first build

If you are starting, pick a task that is high frequency, low stakes and easy to verify. Something where wrong output is obvious and harmless. Get the full trigger to dispatch cycle working end to end before adding sophistication.

The instinct is to automate the most painful task first. That is usually also the most complex one, and starting there tends to produce a half built pipeline that gets abandoned.

Your RAG Isn't Broken. Your Retrieval Is, and You've Never Measured It.

msm yaqoob — Sun, 23 Aug 2026 07:25:20 +0000

Here's the debugging loop I see constantly, and ran myself for 1 month.

Answer is wrong. Adjust the prompt. Slightly better. Swap the model. Different wrong. Add "only use the provided context" in bold. Still wrong. Conclude the model isn't good enough.

Every one of those changes is downstream of the actual failure.

If the right chunk never made it into the context window, no prompt fixes that. You are tuning the generator to compensate for a retriever nobody has ever scored.

Two components, two failure modes, and almost everyone debugs the wrong one — because the generator is the part you can see.

Split the system before you debug it
query → [RETRIEVER] → chunks → [GENERATOR] → answer

Ask one question and the whole thing decomposes:

Was the answer present in the retrieved chunks?

No → retrieval failure. The generator was never in a position to succeed. Prompting is irrelevant.
Yes, but the answer is still wrong → generation failure. Now prompting matters.

You cannot answer that question without labels, and getting them is less work than you think.

Twenty queries and an afternoon

You don't need a benchmark. You need twenty representative queries with hand-labelled relevant documents. Two hours, once.

python

eval_set.py — the highest-leverage file in your RAG project

EVAL = [
{
"q": "what's our refund window for annual plans",
"relevant": {"billing/refunds.md", "policies/annual-terms.md"},
},
{
"q": "how do we onboard a new enterprise client",
"relevant": {"process/enterprise-onboarding.md"},
},
# ... 18 more, drawn from real queries people actually asked
]

Draw these from your logs, not your imagination. Real queries are shaped differently from the ones you invent — shorter, more ambiguous, full of internal shorthand.

Now score:

python
def recall_at_k(retrieved, relevant, k):
got = set(retrieved[:k])
return len(got & relevant) / len(relevant) if relevant else 1.0

def precision_at_k(retrieved, relevant, k):
got = retrieved[:k]
return sum(1 for d in got if d in relevant) / k

def mrr(retrieved, relevant):
for i, d in enumerate(retrieved, 1):
if d in relevant:
return 1 / i
return 0.0

def evaluate(retriever, eval_set, k=8):
rs, ps, ms = [], [], []
for case in eval_set:
got = retriever(case["q"], k=k)
rs.append(recall_at_k(got, case["relevant"], k))
ps.append(precision_at_k(got, case["relevant"], k))
ms.append(mrr(got, case["relevant"]))
n = len(eval_set)
return {"recall@k": sum(rs)/n, "precision@k": sum(ps)/n, "mrr": sum(ms)/n}

How to read the three numbers — this is the part that actually tells you what to fix:

Symptom Diagnosis Fix
Low recall The right chunk isn't being found at all Chunking, or hybrid search — see below
High recall, low precision Found it, but buried in noise Rerank, or shrink the corpus
Decent precision, low MRR Right chunks present but ranked low Rerank
All three fine, answers still bad Genuine generation problem Now go tune the prompt

That last row is the point of the whole exercise. Most people start there and never leave.

Precision is a corpus problem before it's a model problem

The instinct when precision is bad is to reach for a better embedding model. Try this first, because it's free and usually larger:

Cut the corpus.

The model reads k chunks — eight, ten, twenty out of however many thousand you have. Every irrelevant document is another candidate for a fixed number of slots. Doubling the corpus without doubling relevant material strictly degrades precision.

I cut mine from 15 to 30 documents and precision went 4. That was a bigger improvement than any retrieval change I made, and it took an afternoon of reading rather than a week of engineering.

The four fixes, in the order I'd try them

Chunk on structure, not character count

Fixed-size chunking splits mid-sentence, mid-table, mid-thought. The embedding then represents a fragment that means nothing on its own.

python
def chunk_by_heading(md: str, max_chars=1200):
"""Split on markdown headings; only sub-split if a section is huge."""
sections, cur = [], []
for line in md.splitlines():
if line.startswith("#") and cur:
sections.append("\n".join(cur)); cur = [line]
else:
cur.append(line)
if cur:
sections.append("\n".join(cur))

out = []
for s in sections:
    if len(s) <= max_chars:
        out.append(s)
    else:
        paras, buf = s.split("\n\n"), ""
        for p in paras:
            if len(buf) + len(p) > max_chars and buf:
                out.append(buf); buf = p
            else:
                buf = f"{buf}\n\n{p}" if buf else p
        if buf: out.append(buf)
return out

Prepend the document title and heading path to each chunk before embedding. A chunk that begins "the window is 30 days" is useless in isolation; Billing > Refunds > Annual plans — the window is 30 days is retrievable. This one change is usually the single biggest recall win available, and it costs nothing.

Hybrid search

Dense embeddings are bad at exact tokens — error codes, SKUs, surnames, internal project names. Keyword search is bad at paraphrase. You need both.

python
def hybrid(query, k=8, alpha=0.6):
dense = {d: s for d, s in vector_search(query, k=k*3)}
sparse = {d: s for d, s in bm25_search(query, k=k*3)}

def norm(scores):
    if not scores: return {}
    lo, hi = min(scores.values()), max(scores.values())
    rng = (hi - lo) or 1
    return {d: (s - lo) / rng for d, s in scores.items()}

dn, sn = norm(dense), norm(sparse)
merged = {
    d: alpha * dn.get(d, 0) + (1 - alpha) * sn.get(d, 0)
    for d in set(dn) | set(sn)
}
return sorted(merged, key=merged.get, reverse=True)[:k]

Tune alpha against your eval set rather than by feel. On corpora full of internal jargon I've usually landed lower than expected — the keyword half does more work than you'd assume.

Rerank

Retrieve 30, rerank, keep 8. A cross-encoder reads query and document together instead of comparing pre-computed vectors, so it's much more accurate and much slower — which is exactly why it belongs on a small candidate set.

Reranking fixes ranking, not recall. If the right chunk isn't in your 30, reranking cannot help. Check recall@30 before you reach for it, or you'll spend a week on the wrong component.

Filter on metadata, especially freshness

This is the one that gets skipped and it's the one that bites.

Staleness is not a relevance property. A superseded pricing document is more semantically similar to "what's our pricing" than most current documents, because it's precisely about that. Your retriever will surface it enthusiastically and correctly.

python
def retrieve(query, k=8):
candidates = hybrid(query, k=k*4)
fresh = [d for d in candidates if store[d].meta.get("status") == "current"]
return rerank(query, fresh)[:k]

No embedding model detects that a document expired. That has to come from metadata, which has to come from somebody owning the document — which is the governance half of this and honestly the harder part.

Wire it to a diff

Same discipline as any test suite. The value isn't a score, it's did this change make things worse.

python
BASELINE = {"recall@k": 0.71, "precision@k": 0.44, "mrr": 0.62}

def check(new):
regressions = {
m: (BASELINE[m], new[m])
for m in BASELINE
if new[m] < BASELINE[m] - 0.03 # tolerance for noise
}
if regressions:
raise SystemExit(f"retrieval regression: {regressions}")
print("ok", new)

Run it on every chunking change, every embedding swap, every corpus addition. Especially every corpus addition — that's the change everyone assumes is monotonically good, and it's the one most likely to quietly cost you precision.

The 20-line version
[ ] 20 real queries from your logs, hand-labelled
[ ] recall@k, precision@k, MRR — before touching anything
[ ] Low recall → chunk on structure, prepend heading path, add keyword search
[ ] Low precision → cut the corpus before you swap the embedding model
[ ] Low MRR, decent recall → rerank a wide candidate set
[ ] Filter stale docs at retrieval; no model detects expiry
[ ] Diff against baseline on every change, corpus additions included

An afternoon of labelling tells you more than a month of prompt tweaking. And it tells you the one thing prompt tweaking never can: whether the answer was ever in the room.

[YOUR NAME] — I test AI stacks honestly, including the parts that don't work: AiStackGuru.

You Don't Have a Prompt Problem. You Have an Eval Problem.

msm yaqoob — Thu, 20 Aug 2026 06:31:21 +0000

yaml

title: "You Don't Have a Prompt Problem. You Have an Eval Problem."
published: false
description: "If you can't tell automatically whether an output is good, every prompt change is a vibe. Here's the smallest eval setup that actually pays for itself."
tags: ai, testing, python, devops
series: Measuring AI Workflows

canonical_url:

(Leave canonical_url empty. Flip published when you've run the code.)

Here's a workflow I bet you recognise.

Output isn't quite right. You tweak the prompt. Looks better. Ship it. Two weeks later something else is off, you tweak again, and you have no idea whether you fixed the new problem or reintroduced the old one because there is no record of what "working" looked like.

That's not a prompting problem. That's what software development looks like without tests.

The reason it persists is that LLM output feels unbounded and untestable. It isn't. Most of it is far more testable than people assume, and the useful assertions are boring ones you can write in an afternoon.

The economics first, because they decide what's worth building

I timed [NUMBER] of my own tasks with and without AI last month. It lost on [NUMBER] of them, and the cause was never model quality. It was verification the human time spent checking output before it could be used.

net_per_run = manual − (generation + verification + retries)

If verification approaches manual time, net gain approaches zero no matter how fast generation is. A better model doesn't help. This gets misdiagnosed as a capability problem constantly.

An eval is a machine that does part of your verification for you. That's the entire value proposition, and it's why evals pay back faster than almost anything else you can build: they attack the term in that equation that a better model can't touch.

Start with assertions, not judges

The instinct is to reach for LLM-as-judge. Resist it for one more day. Most of what you need to know is checkable with plain code, and plain code is deterministic, instant, and free.

python
from dataclasses import dataclass
from typing import Callable

@dataclass
class Case:
name: str
inputs: dict
checks: list[Callable[[str], bool | str]] # True, or a failure message

def run_evals(cases, generate):
failures = []
for c in cases:
out = generate(**c.inputs)
for check in c.checks:
r = check(out)
if r is not True:
failures.append(f"{c.name}: {r or check.name}")
return failures

Now the checks. These are unglamorous and they catch a genuinely surprising share of real regressions:

python
import json, re

def is_valid_json(out):
try:
json.loads(out); return True
except json.JSONDecodeError as e:
return f"invalid JSON: {e.msg}"

def has_required_keys(*keys):
def check(out):
try: d = json.loads(out)
except Exception: return "not JSON"
missing = [k for k in keys if k not in d]
return True if not missing else f"missing keys: {missing}"
check.name = f"has_keys{keys}"
return check

def no_hedging(out):
banned = ["as an ai", "i cannot", "it's important to note", "delve into"]
hit = [p for p in banned if p in out.lower()]
return True if not hit else f"hedging: {hit}"

def within_length(lo, hi):
def check(out):
n = len(out.split())
return True if lo <= n <= hi else f"length {n} outside [{lo},{hi}]"
return check

def cites_only(allowed_urls):
def check(out):
found = set(re.findall(r"https?://[^\s)]]+", out))
extra = found - set(allowed_urls)
return True if not extra else f"invented URLs: {extra}"
return check

That last one — checking for URLs that weren't in the source material — has caught more real problems for me than any sophisticated method. Fabricated citations are common, high-embarrassment, and trivially detectable.

The cases matter more than the checks

Your eval set is not a random sample. It's a museum of everything that has already gone wrong.

Rule: every time you catch a bad output in production, it becomes a case. That's the whole discipline. Ten cases collected this way beat a hundred synthetic ones, because they're drawn from the actual distribution of your failures rather than your imagination of them.

python
CASES = [
Case("empty input", {"text": ""}, [is_valid_json]),
Case("very long input", {"text": LONG_SAMPLE}, [within_length(50, 300)]),
Case("adversarial", {"text": INJECTION_SAMPLE},[no_leaked_instructions]),
Case("the one from [DATE] that fabricated a source",
{"text": THAT_INPUT}, [cites_only(THAT_SOURCES)]),
]

Name the cases after what went wrong. "the one from March that fabricated a source" is a better test name than test_citation_accuracy_3 because in six months you'll know why it exists.

Then, and only then, an LLM judge

For the genuinely subjective dimensions tone, relevance, whether an answer actually addressed the question — use a model. Three rules that make the difference between a useful judge and an expensive random number generator:

Binary, not scored. "Rate 1–10" produces noise. Ask a yes/no question with a stated criterion.

One dimension per call. A judge asked about accuracy, tone and completeness at once will collapse them into a general vibe.

Judge against a reference, where you have one. Comparative judgement is far more stable than absolute judgement.

python
JUDGE = """You are checking one property of a piece of text.

PROPERTY: {property}

TEXT:
{text}

Answer with exactly one word: PASS or FAIL.
If the property does not clearly hold, answer FAIL."""

def judge(property_desc):
def check(out):
v = call_model(JUDGE.format(property=property_desc, text=out)).strip().upper()
return True if v.startswith("PASS") else f"judge failed: {property_desc}"
check.name = "judge"
return check

Validate your judge before you trust it. Hand-label twenty outputs, run the judge, compare. If it disagrees with you on more than a couple, the judge prompt is the problem fix it before you use it to evaluate anything else. A judge you haven't checked is just a second unverified model in the loop, which is the opposite of what you're building.

Wire it to a diff, not a dashboard

The value isn't a score. It's did this change make things worse.

python
def compare(cases, old_gen, new_gen):
before = set(run_evals(cases, old_gen))
after = set(run_evals(cases, new_gen))
return {
"fixed": sorted(before - after),
"broken": sorted(after - before), # the only column that matters
"still": sorted(before & after),
}

Run it on every prompt change and every model swap. broken is the column that should stop a deploy. Model upgrades are the underrated case here — "newer model" is not "better for your task," and this is how you find out in four seconds instead of four weeks.

Keep it fast and cheap enough to run every time. An eval suite that takes ten minutes and costs real money gets skipped exactly when you're in a hurry, which is exactly when you're most likely to break something.

What this doesn't do

Be honest about the boundary. Evals catch regressions on failure modes you've already seen. They don't catch novel ones, they don't tell you whether the workflow is worth having, and a green suite is not a guarantee of a good output.

They shift verification from every run to every change, which is where the leverage is. The residual human check gets shorter, not eliminated.

And if verification is still most of your run time after building these, that's a real signal about the task itself some work genuinely takes as long to check as to do, and the correct answer there is to keep it human rather than to build more tooling around it. I've written up the decision framework for that separately.

The 30-line version

If you build nothing else:

[ ] A list of cases, each named after a real failure
[ ] is_valid_json / has_required_keys, if you parse output
[ ] within_length
[ ] cites_only(sources), if the output makes claims
[ ] A diff of before/after failures on every prompt change

Five checks, one afternoon. It will catch things a month of careful reading wouldn't, and more importantly it will catch them the moment you introduce them, which is the only time fixing them is cheap.

I test AI tools and workflows honestly, including the ones that don't earn their place: AiStackGuru.

Prompt Injection Is a Permissions Problem, Not a Model Problem

msm yaqoob — Wed, 19 Aug 2026 07:32:44 +0000

Every mitigation that treats injection as a text-filtering problem eventually fails. Here's the capability-based version that doesn't depend on the model behaving.

Every prompt injection discussion I read eventually arrives at the same place: better instructions. Put the system prompt in a stronger position. Tell the model to ignore instructions in retrieved content. Add a classifier that detects malicious input.

All of these help. None of them are a control, because all of them depend on the model behaving correctly on an input someone else chose.

Here's the reframe that made this tractable for me: injection is not an input-validation problem. It's a privilege problem. The question is not "can something get bad instructions into the context." Assume it can. The question is what those instructions are able to reach.

The actual mechanism

An LLM has one channel. Your instructions and the data it processes arrive in the same stream, in the same format, with no structural marker separating them. There is no equivalent of a parameterised query — no way to say this part is code and that part is strictly data at the protocol level.

That's not an implementation gap someone will close next quarter. It's a property of how these models take input.

Which means the moment your agent reads anything you didn't write — a web page, an email, a PDF, an issue comment, a search result, a filename — that content is instructions-adjacent. Not because the model is naive, but because there is no layer that could reliably tell the difference.

Now stack that against how we build agents: give it tools, give it credentials, let it run unattended. We've built systems that take instructions from anywhere and act with our authority.

The injection is unavoidable. The authority is a choice.

Why filtering doesn't get you there

Briefly, because it's the natural first idea:

Detection classifiers are a bounded search problem for whoever's writing the input, and they have to succeed every time while an attacker needs one pass. Delimiters and "ignore anything below this line" instructions live in the same channel as the content, so they're addressable by the content. Encoding tricks, multiple languages, and content in images or documents all route around text-level rules.

Filtering is worth having. It reduces volume. It is not a boundary, and building as if it were is where people get hurt.

The control that actually holds

Assume the model will, at some point, faithfully execute a hostile instruction. Now design so that doing so is boring.

That's it. Everything below is a way of making a compromised agent boring.

Split the agent that reads from the agent that acts

The single highest-value structural change. One component processes untrusted content and has no credentials and no tools. It returns structured data. A second component, which never sees the untrusted text, acts on that data.

python

❌ one agent, reads the web, holds the tools

agent = Agent(tools=[send_email, read_files, http_get], creds=CREDS)
agent.run("summarize https://example.com/thing and email me")

✅ untrusted content never reaches the component with capability

raw = fetch(url) # plain fetch, no model
summary = reader.extract(raw) # model, NO tools, NO creds
# returns {title, points[], urls[]}
mailer.send(to=OWNER, body=render(summary)) # code path, fixed recipient

A hostile instruction in that page can influence summary. It cannot reach mailer, because mailer isn't reading it and its recipient isn't a variable the model controls.

Notice to=OWNER is hardcoded. The moment the recipient becomes model-determined, you've reconnected the two halves.

Return intents, not calls

Don't let the model invoke. Let it propose, and validate the proposal against a schema you wrote:

python
ALLOWED = {"summarize", "tag", "draft_reply"}

def handle(proposal: dict) -> dict:
action = proposal.get("action")
if action not in ALLOWED:
audit("reader", action, None, False, "not in allowlist")
raise PermissionError(f"refused: {action}")
args = SCHEMAS[action].validate(proposal.get("args", {}))
return EXECUTORSaction

Allowlist, never denylist. You can't enumerate everything you don't want; you can enumerate the four things you do.

Break the exfiltration path

Reading your data is only half an incident. The other half is getting it out. Two things carry it:

Outbound network. If the agent can request arbitrary URLs, every byte it can read can leave via a query string. Allowlist egress by host.

python
ALLOWED_HOSTS = {"api.internal.example", "docs.example.com"}

def safe_get(url):
host = urlparse(url).hostname or ""
if host not in ALLOWED_HOSTS:
audit("agent", "http.get", host, False, "host not allowed")
raise PermissionError(f"blocked host: {host}")
return httpx.get(url, timeout=10)

Rendered output. This one catches people. If your agent's output is rendered as markdown or HTML in a UI, an image reference the model was induced to emit will make the browser issue a request — with whatever ended up in the URL. The user sees a broken image. The data is gone.

Strip or proxy remote references in model output. Treat model output as untrusted, because it is: it's downstream of untrusted input.

Put the human on the irreversible half

Split every capability into a reversible half and an irreversible half, and gate the second:

Reversible — let it run Irreversible — human confirms
Draft an email Send it
Create a branch Merge to main
Stage a change Deploy
Propose a delete Delete
Prepare a transaction Sign it

Reviewing a draft is fast. Writing one isn't. You lose almost no throughput and you remove the entire class of failures you can't undo.

Bound every credential

Per-agent keys. Expiry. Spend cap. Rate limit. Egress allowlist. Logs the agent can't write to, recording denials as well as allows — a spike in refusals is the cheapest signal you'll ever get, and it's the one people forget to record because nothing bad happened.

I've written up the credential architecture in more detail here; the short version is one identity per agent per environment, read-only until write is earned, and secrets held by a broker the model can't instruct.

Audit permissions as a set

Individually harmless capabilities compose into dangerous ones:

read mail + send mail → your inbox is a password-reset engine, and now something can both trigger and consume the resets
read files + arbitrary egress → an exfiltration path missing only a trigger
write repo + CI → code execution in your build environment, with your build secrets
delete + logs in the same system → an incident with no forensics

Read across the row per agent, not down the column per permission. Dangerous configurations are almost always horizontal.

The bit about wallets

If any of this touches financial rails: dedicated credentials that exist for nothing else, hardware-backed signing so the key never enters the environment the agent runs in, and a human on every signature. Assume any key or seed that has passed through a general-purpose model's context is compromised and rotate it.

Educational only, not financial advice.

The checklist
[ ] Reader component has no tools and no credentials
[ ] Model returns intents; a schema validates them; an allowlist gates them
[ ] Egress allowlisted by host
[ ] Remote references stripped from rendered model output
[ ] Irreversible actions gated behind a human
[ ] One credential per agent, with expiry + spend cap + rate limit
[ ] Append-only external logs, denials included
[ ] Permission sets audited per agent, across the row
[ ] Kill switch documented and tested once

None of it depends on the model behaving. That's the whole point.

The industry keeps looking for the fix that makes injection stop happening. There probably isn't one, for the same reason there's no fix that makes SQL injection stop being attempted — we solved that by removing the ambiguity between code and data at the protocol level, and LLMs don't have a protocol level to do it at.

So we do the other thing. Assume the instruction lands. Make sure it lands somewhere with nothing to reach.

[YOUR NAME] — I build and break down AI stacks, with a focus on the security side most tool reviews skip: AiStackGuru. There's an interactive AI Visibility Tool that maps what a given permission set exposes.

What's the most surprising permission combination you've found in a running agent? I'd like to collect a few.

Debugging AI Invisibility: A Runbook for Sites ChatGPT Can't See

msm yaqoob — Sun, 19 Jul 2026 08:42:23 +0000

A client asks: "Why doesn't ChatGPT know about us?" You open the site it loads fast, ranks okay on Google, looks fine. This is the debugging runbook for exactly that ticket: a triage order that finds the failure in minutes instead of guesswork, with the commands to prove each diagnosis before you touch anything.

The mental model: AI answers pull from two pipelines, a retrieval pipeline (live web search, heavily Bing-flavored for ChatGPT) and a model-memory pipeline (training data). You can only debug the first one, so that's where the runbook lives. Work top-down; each check gates the next.

Check 0: Reproduce the bug properly

Before touching config, document the failure like any bug. Ask ChatGPT (with search enabled) three things and save the outputs: the category question ("best X in CITY"), the brand question ("what is CLIENTNAME"), and a direct URL question ("what's on https://client.com"). The pattern tells you the layer: wrong facts about the brand = data-consistency problem; total absence from category answers = retrieval or authority problem; can't describe the URL = crawl/index problem. Now debug in that order of severity.

Check 1: Can the bots physically fetch the site?

Don't read robots.txt and assume. Fetch as the bots fetch:

bashfor UA in "OAI-SearchBot/1.0" "PerplexityBot/1.0" "ChatGPT-User/1.0"; do
echo "== $UA =="
curl -s -o /dev/null -w "%{http_code} %{size_download} bytes\n" \
-A "$UA" https://client.com/
done

Interpret: 200 with a normal byte size = pass. 403 = something upstream (WAF, Cloudflare bot rules, security plugin) is rejecting the agent even if robots.txt looks clean, check the firewall event logs, not the config files. 200 with a tiny payload = you're serving an empty JS shell (see Check 4). Also confirm robots.txt itself isn't the blocker:

bashcurl -s https://client.com/robots.txt | grep -A1 -iE "gptbot|oai|perplexity|claude"

Remember the split: OAI-SearchBot / ChatGPT-User power live answers; GPTBot is training. A "block AI" tutorial that nuked all of them took the client out of the answers business.

Check 2: Does the retrieval index know the site exists?

ChatGPT search leans substantially on Bing. Google-only SEO histories fail here constantly:

site:client.com <- run on bing.com, not google.com

Zero results is your smoking gun for "invisible in category answers despite a healthy site." Fix: Bing Webmaster Tools (import from GSC in two clicks), submit the sitemap, and wire IndexNow so future updates propagate in hours:

bashcurl "https://api.indexnow.org/indexnow?url=https://client.com/&key=YOUR_KEY"

Log the date; you'll want it for the before/after report.

Check 3: Do the facts survive machine parsing?

Retrieval finding the page is necessary, not sufficient the facts have to be extractable. Grep the raw HTML for the things the client wants AI to know:

bashcurl -s -A "OAI-SearchBot/1.0" https://client.com/contact/ \
| grep -ciE "0335|\+92|@client\.com"

Zero matches on the contact page means the phone/email exist only in an image, an iframe, or post-render JavaScript. Same test on the pricing page for amounts. Every zero here is a fact AI cannot cite, period.

Then validate the structured-data layer at validator.schema.org — and check for the duplicate schema failure mode: multiple plugins each emitting an Organization block with different values. Conflicting JSON-LD on one page is actively worse than none; consolidate to one source of truth per type.

Check 4: Is the content server-rendered?

If Check 1 returned 200 with suspiciously few bytes, or Check 3 greps clean HTML but the browser shows content:

bashcurl -s https://client.com/ | wc -c # raw HTML size
curl -s https://client.com/ | grep -c "<h2" # structural content present?

A 4KB shell with zero headings means the site only exists after client-side rendering, fine for humans, a void for a chunk of retrieval systems. Prescription: SSR/SSG for money pages, or at minimum pre-rendering. On WordPress this failure usually traces to an "optimize everything" plugin preset deferring content itself; on SPAs it's architectural.

Check 5: Is the entity coherent across the web?

The subtle one. Pull the business's NAP (name/address/phone) from the site, the Google Business Profile, Facebook, and the top three directory hits, and diff them mentally. Different phone formats, two name spellings, a stale address each divergence lowers machine confidence in every version. There's no command for this one; it's an audit table and an afternoon of edits. It fixes the "ChatGPT describes us wrongly" class of bug more often than anything technical.

Check 6: The authority floor

If Checks 1-5 all pass and the brand still loses category questions to competitors: the problem isn't parseability, it's evidence. Count Google reviews (volume + last-90-day velocity + owner replies), third-party mentions, consistent citations. Machines making recommendations behave like cautious buyers, a technically perfect site with eight dusty reviews is legible but not recommendable. This layer is marketing's ticket, not yours; hand it over with the data.

The triage table (pin this)

When you're on a call and need the diagnosis order from symptoms alone:

Client saysMost likely checkFirst command"ChatGPT describes us wrongly"5 (entity consistency)NAP diff across surfaces"We never appear for category questions"2 then 6site: on Bing"It can't even summarize our URL"1 (fetch)curl as OAI-SearchBot"Our prices/services never get quoted"3 then 4grep facts in raw HTML"It worked before, stopped recently"1 (regression)check WAF event log + plugin changelogs

That last row deserves emphasis because it's the ticket you'll see most after the first fix: things that were working break silently. The usual suspects, in order of frequency I've encountered: a security-plugin update re-writing robots.txt with fresh AI blocks; Cloudflare's "Block AI bots" toggle getting enabled during an unrelated security review; a caching plugin's preset change that starts serving crawlers the deferred-JS shell; and an SEO plugin update that duplicates the schema output. None of these announce themselves. This is why the weekly snapshot cron from my earlier post or even a monthly manual re-run of Checks 1 and 3 belongs in the maintenance contract, not the "if we remember" pile.

Close the loop

One last professional habit: end every engagement with a one-page handoff report, the three Check-0 outputs from before, the same three after, the dated fix list, and the two recurring checks (bot-fetch snapshot, Bing site: count) whoever inherits the site must keep running. Half the value of this runbook is the fix; the other half is making the invisible layer visible to the client, so the next silent regression gets caught in a week instead of a quarter.

Re-run Check 0's three questions weekly and log who gets named. Retrieval-side fixes typically surface in answers within 2–5 weeks. The log is your regression test and the most persuasive artifact you'll ever attach to an invoice.

Full non-technical breakdown of all seven failure classes (for the client-facing version of this ticket)

I work with DigiMSM (digimsm.com) Pakistan's first AI-powered digital marketing agency. We run this exact runbook as a free AI Visibility Audit. War stories about bots and WAFs welcome in the comments.

How to Make a Website That ChatGPT Actually Cites: Schema, llms.txt, and AEO for Developers

msm yaqoob — Thu, 16 Jul 2026 08:46:54 +0000

Your client's site is fast, accessible, and ranks fine on Google and ChatGPT has never heard of it. That's not an SEO problem. It's a machine-readability problem, and as the developer, you're the one who can fix it. This is the practical checklist I use, with copy-paste code.

Why this is now a developer's job

AI search systems (ChatGPT search, Perplexity, Gemini, Google AI Overviews) don't rank pages they retrieve and synthesize. A page enters the retrieval pool only if crawlers can fetch it, parse it, and extract unambiguous facts from it. Rendering strategy, structured data, crawl directives that's our layer, not the marketing team's.

Three failure modes cover most invisible sites: AI crawlers blocked in robots.txt, facts locked inside JavaScript or images, and zero structured data. All three are fixable in an afternoon.

Step 1: Stop blocking the crawlers (2 minutes)

Check yourdomain.com/robots.txt. Security plugins and copy-pasted "protect my content" configs routinely block the exact bots that power AI search:

txt# BAD - makes you invisible to AI search
User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Know the distinction: search/browse bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) fetch pages to answer live queries — block these and you vanish from AI answers. Training bots (GPTBot, Google-Extended, ClaudeBot) collect training data. If the client has IP concerns, block training, allow search:

txt# Reasonable middle ground
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Default recommendation for local businesses: allow everything. Being cited is the whole game.

Step 2: Ship an llms.txt (15 minutes)

llms.txt is an emerging convention (adopted by Anthropic, Cloudflare, Zapier, Stripe among others): a markdown file at your root that tells language models what the site is, in plain text they can't misparse. Adoption by crawlers is still uneven treat it as cheap insurance, not magic:

markdown# Acme Furniture Rawalpindi

Custom furniture workshop in Rawalpindi, Pakistan.
Handmade sofas, beds and office furniture since 2011.

Key facts

Location: Main Murree Road, Rawalpindi, Pakistan
Phone: +92 300 0000000
Hours: Mon-Sat 10:00-20:00
Delivery: Rawalpindi, Islamabad

Key pages

Products: Full catalog with prices
Custom orders: How ordering works
Contact: Phone, WhatsApp, map

Serve it as text/plain or text/markdown at /llms.txt. One file, zero build changes.

Step 3: Structured data that actually matters (the big one)

Schema markup is the highest-leverage item on this list. There's documented enterprise data of AI Overview accuracy jumping from 43% to 91% after proper entity-linked schema. For a local business, the minimum viable set is LocalBusiness + FAQPage:

html { "@context": "<a href="https://schema.org">https://schema.org</a>", "@type": "LocalBusiness", "name": "Acme Furniture", "description": "Custom furniture workshop in Rawalpindi, Pakistan.", "address": { "@type": "PostalAddress", "streetAddress": "Main Murree Road", "addressLocality": "Rawalpindi", "addressCountry": "PK" }, "telephone": "+923000000000", "url": "<a href="https://example.com">https://example.com</a>", "openingHours": "Mo-Sa 10:00-20:00", "sameAs": [ "<a href="https://www.facebook.com/acmefurniture">https://www.facebook.com/acmefurniture</a>", "<a href="https://www.instagram.com/acmefurniture">https://www.instagram.com/acmefurniture</a>" ] }

Rules that matter in practice: the JSON-LD must agree exactly with the visible page (AI cross-checks); sameAs links are your entity-disambiguation lifeline; and FAQ answers in schema must be verbatim copies of the on-page text, not summaries.

Step 4: Answer-first content structure

AI extraction favors pages where the answer appears in the first 40–60 words under a question-shaped heading. Work with whoever writes the content to enforce this pattern:

html

How much does a custom sofa cost in Rawalpindi?

A custom three-seater sofa in Rawalpindi typically costs PKR 45,000-120,000 depending on wood and fabric. Below is the full price breakdown...

Direct answer, then depth. Tables beat paragraphs for comparisons LLMs parse them cleanly. And render critical facts as server-side HTML: if the price only exists after a client-side fetch, assume AI never sees it.

Step 5: Entity consistency (the invisible killer)

LLMs resolve conflicting facts by consensus. If the site says +92 300 1111111, Google Business Profile says 0300-2222222, and a directory says a third number, the model picks one — possibly none. Audit name/address/phone across every surface and make them byte-identical where possible. Boring, decisive.

Verification loop

After shipping: fetch your pages as the bots do (curl -A "PerplexityBot" and check what HTML comes back), validate schema at validator.schema.org, then actually ask ChatGPT/Gemini/Perplexity the category question ("best custom furniture in Rawalpindi") weekly and log whether your client appears. Retrieval-grounded answers typically reflect fixes in 2–5 weeks; training-data errors take longer and need the consensus fix above.

Step 6: Get indexed where AI actually looks

A corrected page the crawlers haven't fetched fixes nothing. Two indexes matter more than most devs realize:

Bing is load-bearing. ChatGPT search and Copilot are substantially Bing-grounded. Register the site in Bing Webmaster Tools (you can import straight from Google Search Console), submit the sitemap, and check site:yourdomain.com on Bing. A site missing from Bing is invisible to a large slice of AI search regardless of how good its Google presence is.

IndexNow is free and instant. One ping tells Bing (and everything downstream of it) about new or updated URLs:

bashcurl "https://api.indexnow.org/indexnow?url=https://example.com/updated-page&key=YOUR_KEY"

Generate a key, drop the key file at your root, and wire the ping into your deploy pipeline so every content update propagates automatically. For WordPress clients, several SEO plugins now ship IndexNow support — flip it on.

Edge cases that bite

SPAs and client-side rendering: if curl returns an empty shell, assume AI retrieval sees an empty shell. Server-render or pre-render anything you want cited. Cloudflare bot rules: Bot Fight Mode and aggressive WAF configs silently 403 AI crawlers — check the firewall event log for the user agents above before blaming content. Multilingual sites: keep the facts identical across language versions; models cross-reference them and conflicting details (different phone numbers on /en/ vs /ur/) reads as inconsistency, not localization.

The bigger picture

This checklist is the technical half of a discipline called AEO/GEO (Answer Engine Optimization / Generative Engine Optimization) the other half is brand authority, reviews, and citations, which is marketing's territory. If you want the complete picture including the non-technical layer and how to evaluate whether an agency actually knows this stuff, I've linked the full guide here:

If you want to automate the verification loop, a cron job that snapshots your key pages as each bot sees them is twenty lines of code and catches regressions before they cost citations:

bash#!/bin/bash

Weekly AI-crawler visibility snapshot

for UA in "OAI-SearchBot" "PerplexityBot" "ChatGPT-User"; do
curl -s -A "$UA" https://example.com/ -o "snap-$(date +%F)-$UA.html"
grep -c "your-phone-number" "snap-$(date +%F)-$UA.html" || echo "WARN: $UA cannot see key facts"
done

Diff the snapshots week over week. The day a plugin update or a WAF rule silently breaks crawler access, you'll know that week — not three months later when the client asks why the leads dried up.

Ship the robots.txt fix today. It's two minutes, and it's probably the highest ROI-per-line-of-config you'll touch this month.

Technical SEO in 2026: How to Audit Your Site for AI Crawlers (Not Just Googlebot)

msm yaqoob — Sat, 18 Apr 2026 15:42:21 +0000

If you run a website in 2026 and your technical SEO checklist only covers Googlebot, you're auditing for an incomplete picture of how search actually works today.
This is a developer-focused walkthrough of what AI crawlers check that Googlebot ignores — and how to verify and fix each issue with real code examples.

Full 47-point audit checklist: DigiMSM Technical SEO Audit Checklist 2026

The crawlers you're being evaluated by right now
Googlebot → Traditional Google Search rankings
GPTBot → OpenAI / ChatGPT web browsing & knowledge
ClaudeBot → Anthropic / Claude AI knowledge base
PerplexityBot → Perplexity AI real-time answers
Googlebot-Extended → Google AI Overviews & Gemini
Bytespider → ByteDance (TikTok AI features)
cohere-ai → Cohere LLM training & retrieval
Each has different evaluation priorities. AI crawlers don't rank pages — they extract facts. Your robots.txt, JavaScript architecture, and content structure all affect whether these bots can read and cite your content.

Robots.txt: The silent AI visibility killer Check your robots.txt right now: bashcurl https://yourdomain.com/robots.txt If you see anything like: User-agent: * Disallow: / ...you've blocked everything, including all AI crawlers. The correct setup to allow all major AI crawlers: txt# Allow Googlebot User-agent: Googlebot Allow: /

Allow OpenAI's GPTBot

User-agent: GPTBot
Allow: /

Allow Anthropic's ClaudeBot

User-agent: ClaudeBot
Allow: /

Allow Perplexity

User-agent: PerplexityBot
Allow: /

Allow ByteDance

User-agent: Bytespider
Allow: /

Allow Cohere

User-agent: cohere-ai
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
To verify AI bots are actually crawling, check your server logs:
bashgrep -i "gptbot|claudebot|perplexitybot|bytespider" /var/log/nginx/access.log | tail -50
If you see zero results over 30 days, something is blocking them.

JavaScript rendering: AI bots aren't browsers
This is the most technically significant difference between Googlebot and AI crawlers.
Googlebot: Fully renders JavaScript via a headless Chrome instance. Executes fetch(), React hydration, lazy loads — all of it.
GPTBot / ClaudeBot / PerplexityBot: Most function like raw HTTP GET requests. They receive your raw HTML response. They do not execute JavaScript.
Test what AI crawlers actually see
bash# Simulate an AI crawler request
curl -H "User-agent: Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
https://yourdomain.com/your-page/
Or in Node.js:
javascriptconst response = await fetch('https://yourdomain.com/your-page/', {
headers: {
'User-Agent': 'GPTBot/1.0'
}
});
const html = await response.text();
// Does this HTML contain your actual content?
console.log(html.includes('your key content phrase'));
If your content is injected via JavaScript after page load (React, Vue, Next.js CSR), AI crawlers may see nothing.
Fix: Use Server-Side Rendering (SSR) or Static Site Generation (SSG)
javascript// Next.js example - render content server-side
export async function getServerSideProps(context) {
const data = await fetchYourContent();
return {
props: { content: data }
};
}
For existing SPAs, consider using react-snap or prerendering services for at least your most important pages.
Structured Data: JSON-LD for AI extraction
Schema markup serves two different masters:

For Google: Rich results (stars, FAQs in SERPs, breadcrumbs)
For AI crawlers: Pre-formatted answer extraction

The schema types AI crawlers extract from most effectively:
FAQPage Schema (highest AI citation value)
html { "@context": "<a href="https://schema.org">https://schema.org</a>", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "What do AI crawlers check that Google doesn't?", "acceptedAnswer": { "@type": "Answer", "text": "AI crawlers prioritise semantic clarity, answer-format content, non-JavaScript renderability, entity consistency, and E-E-A-T signals. Unlike Google, they do not evaluate PageRank or keyword density — they extract facts to synthesise direct answers." } }, { "@type": "Question", "name": "How do I allow GPTBot to crawl my website?", "acceptedAnswer": { "@type": "Answer", "text": "Add 'User-agent: GPTBot' followed by 'Allow: /' to your robots.txt file. Verify crawl activity by checking your server access logs for the GPTBot user-agent string." } } ] } 
Organization Schema (entity identity for AI)
html { "@context": "<a href="https://schema.org">https://schema.org</a>", "@type": "Organization", "name": "DigiMSM", "url": "<a href="https://digimsm.com">https://digimsm.com</a>", "logo": "<a href="https://digimsm.com/logo.png">https://digimsm.com/logo.png</a>", "description": "Pakistan's first AI-driven SEO agency, specialising in AEO, GEO, and AI-first technical SEO.", "address": { "@type": "PostalAddress", "addressLocality": "Islamabad", "addressCountry": "PK" }, "sameAs": [ "<a href="https://twitter.com/digimsm">https://twitter.com/digimsm</a>", "<a href="https://linkedin.com/company/digimsm">https://linkedin.com/company/digimsm</a>", "<a href="https://www.facebook.com/digimsm">https://www.facebook.com/digimsm</a>" ] } 
Speakable Schema (underused, increasingly important)
html { "@context": "<a href="https://schema.org">https://schema.org</a>", "@type": "WebPage", "speakable": { "@type": "SpeakableSpecification", "cssSelector": [".article-summary", ".key-answer", "h1"] }, "url": "<a href="https://digimsm.com/your-page/">https://digimsm.com/your-page/</a>" } 
Critical rule: Always use JSON-LD, never Microdata. JSON-LD is in a separate tag and doesn't depend on DOM structure — AI crawlers can extract it even if they don't fully parse your HTML layout. Validate all schema: <a href="https://search.google.com/test/rich-results">https://search.google.com/test/rich-results</a> <ol> <li>The Answer Block pattern AI crawlers extract disproportionately from the first 30% of your content. Here's the pattern every important page should follow: html <h1>The Complete Guide to Technical SEO in 2026</h1> In today's rapidly evolving digital landscape, search engine optimisation has undergone significant transformation. As artificial intelligence becomes increasingly integrated into search technology...</li> </ol>  <h1>Technical SEO Audit Checklist 2026: What AI Crawlers Check</h1> <div class="answer-block" itemscope itemtype="https://schema.org/Answer"> AI crawlers like GPTBot and ClaudeBot prioritise semantic clarity, structured data, and answer-format content — not keyword rankings. A 2026 technical SEO audit must separately cover Google requirements and AI crawler requirements across five categories: crawl access, schema markup, content structure, E-E-A-T signals, and technical performance. </div> Here's how each category works and what to check... Keep the answer block to 40–60 words. Self-contained. Citable without context. <ol> <li>HTTP headers and crawlability checks bash# Check response headers for key SEO signals curl -I <a href="https://yourdomain.com/page/">https://yourdomain.com/page/</a></li> </ol> <h1> <a name="look-for" href="#look-for" class="anchor"> </a> Look for: </h1> <h1> <a name="xrobotstag-should-not-have-noindex" href="#xrobotstag-should-not-have-noindex" class="anchor"> </a> X-Robots-Tag: (should NOT have noindex) </h1> <h1> <a name="contenttype-texthtml-charsetutf8" href="#contenttype-texthtml-charsetutf8" class="anchor"> </a> Content-Type: text/html; charset=UTF-8 </h1> <h1> <a name="http2-200-correct-status-code" href="#http2-200-correct-status-code" class="anchor"> </a> HTTP/2 200 (correct status code) </h1> Check for accidental X-Robots-Tag: noindex headers on pages you need AI crawlers to index — this is a server-level noindex that won't appear in your HTML source but will block all crawlers. pythonimport requests pages_to_check = [ '<a href="https://yourdomain.com/">https://yourdomain.com/</a>', '<a href="https://yourdomain.com/services/">https://yourdomain.com/services/</a>', '<a href="https://yourdomain.com/blog/">https://yourdomain.com/blog/</a>', ] for url in pages_to_check: r = requests.get(url, headers={'User-Agent': 'GPTBot/1.0'}) x_robots = r.headers.get('X-Robots-Tag', 'not set') print(f"{url}: status={r.status_code}, X-Robots-Tag={x_robots}") Quick audit checklist for developers ROBOTS.TXT [ ] GPTBot explicitly allowed [ ] ClaudeBot explicitly allowed [ ] PerplexityBot explicitly allowed [ ] Sitemap URL referenced JAVASCRIPT RENDERING [ ] Core content in raw HTML response (no JS dependency) [ ] Schema markup in <head> or raw HTML (not JS-injected) [ ] Cookie consent doesn't block content for non-cookie clients STRUCTURED DATA [ ] FAQPage schema on blog posts and service pages [ ] Article/BlogPosting schema on all editorial content [ ] Organization schema with sameAs on all pages [ ] Person schema on author pages [ ] All schema validates in Rich Results Test (zero errors) CONTENT STRUCTURE [ ] 40-60 word answer block at top of each key page [ ] Stats and claims have source links [ ] Short paragraphs (2-4 sentences max) [ ] "Last updated" date visible on content pages TECHNICAL [ ] HTTPS enforced, no mixed content [ ] Page loads < 2.5 seconds on mobile [ ] No accidental X-Robots-Tag: noindex headers [ ] Server logs checked for AI crawler user agents Resources Full 47-point audit: DigiMSM Technical SEO Audit Checklist 2026 Google Rich Results Test: <a href="https://search.google.com/test/rich-results">https://search.google.com/test/rich-results</a> Schema.org documentation: <a href="https://schema.org">https://schema.org</a> GPTBot documentation: <a href="https://openai.com/gptbot">https://openai.com/gptbot</a> Published by DigiMSM — Pakistan's first AI-driven SEO agency. We specialise in AEO, GEO, and AI-first technical SEO for businesses ready to win in the AI search era.

Non-Custodial API Trading: The Architecture That Changes Everything for Retail Traders

msm yaqoob — Fri, 10 Apr 2026 09:32:05 +0000

If you've spent any time building or evaluating automated trading infrastructure, you know the fundamental tension: execution automation versus capital custody. For years, the only way to get institutional-grade algorithmic execution was to surrender custody of your funds to someone else. That tradeoff no longer exists.

This post breaks down the technical architecture behind noncustodial API trading, why it matters, and how platforms like Kronos Trading have implemented it at scale for retail self-directed traders.

The Custody Problem in Automated Trading
Traditional 'managed' trading solutions require you to deposit funds with the manager or platform. This creates counterparty risk, if the platform fails, mismanages, or acts in bad faith, your capital is exposed. The history of retail trading is littered with exactly these failures.

The noncustodial model eliminates this by separating execution rights from capital custody entirely.
How API Based Non-Custodial Execution Works
The architecture is clean:
• Client maintains funds in their own brokerage account (e.g., Interactive Brokers, Pepper stone both regulated, FDIC insured options)
• Client generates a read/trade API key from their broker this key permits order placement but not withdrawals or fund transfers
• The algorithmic software receives and stores this API key encrypted on the client side
• The software executes rulebased orders via the broker's API within the permissions granted by that key
• The software provider has zero access to withdraw, transfer, or otherwise move funds
The critical technical detail: a well configured broker API key can be scoped to trading permissions only. No withdrawal rights. No transfer rights. The worst case scenario if an API key is compromised is unauthorized trades, not fund theft and position limits further constrain this.

What Kronos Trading's Infrastructure Looks Like

Kronos Trading (IndicatorX LLC) runs quantitative, rule-based trading systems across crypto, forex, commodities, and US equities. Their systems including the All Weather System (built 2022), Crypto Algorithm (built 2021), and the recently released Kronos 2.0, operate on this noncustodial architecture exclusively.

Kronos 2.0, released January 2026, is particularly interesting from an infrastructure perspective: it positions itself as an 'execution and orchestration layer' infrastructure, not advice. Users define rule based workflows. The platform applies them. No prediction engine. No discretionary override. Pure systematic execution.

This design philosophy neutral infrastructure, user defined rules, zero custody is where serious algorithmic trading software is heading.

Why This Architecture Wins
From a pure systems design standpoint, noncustodial execution is simply better for retail traders:
• Counterparty risk eliminated at the architectural level
• Regulatory clarity software licensing is categorically different from fund management
• Client fund segregation is handled by the broker, not the software provider
• Broker level insurance (FDIC, SIPC) applies to client funds
• API key scoping limits blast radius of any security event
The question for any retail trader evaluating automated trading systems should be architectural first: does this platform have custody of my funds? If yes, understand the counterparty risk before proceeding.

Practical Considerations
Onboarding to a noncustodial API trading system typically involves: creating an account with a supported broker, generating an appropriately scoped API key, and connecting it to the software. Kronos Trading claims this takes under 5 minutes, which aligns with what well-built API onboarding flows can achieve.
Worth checking: does the platform support your preferred broker? Does it offer position sizing controls? How does it handle broker API rate limits and downtime? These are the real technical questions.

If you're building your own systems and evaluating noncustodial architecture, the Kronos 2.0 press release (Digital Journal, January 2026) has useful framing on the infrastructure first design philosophy.

Platform: kronostrading.com | Tags: #algorithmictrading #API #fintech #quanttrading #automation #noncustodial

How to Make Your Website Machine-Readable for AI Agents (A2A Marketing for Developers)

msm yaqoob — Fri, 20 Feb 2026 07:42:41 +0000

If you're building websites in 2026 and not thinking about AI agent readability, you're building for yesterday's web.
Here's the situation: AI agents — autonomous systems that browse, evaluate, and recommend on behalf of users — are now part of the real user base. They don't parse CSS. They don't run JavaScript for visual presentation. They query structured data, evaluate entity consistency, and extract verifiable signals.
This article is a practical dev guide to making any web presence A2A-ready (Agent-to-Agent ready). I'll cover the JSON-LD schemas that matter, the structural patterns that help, and the architectural decisions that separate machine-readable brands from invisible ones.

Why AI Agents Are Now Part of Your Audience
Google launched the Universal Commerce Protocol (UCP) at NRF 2026 in January. Co-developed with Shopify, Stripe, Walmart, and Visa, it enables AI agents to execute full commerce flows — research, compare, negotiate, transact — across brand systems without human navigation.
OpenAI simultaneously launched agentic shopping inside ChatGPT.
Forrester projects 1 in 5 B2B sellers will need to respond to AI buyer agents via their own seller agents by end of 2026.
The implication for developers: the agent IS the user, for an increasing portion of the traffic that matters.

Schema Markup: The Foundation
Standard Organization and LocalBusiness schema is not enough. Here's what a fully A2A-optimized service page looks like in JSON-LD:
json{
"@context": "https://schema.org",
"@type": "ProfessionalService",
"name": "DigiMSM",
"description": "Pakistan's first AEO and GEO-focused digital marketing agency, specializing in AI search optimization, Answer Engine Optimization, and Generative Engine Optimization for businesses targeting AI-powered discovery.",
"url": "https://digimsm.com",
"areaServed": [
{"@type": "Country", "name": "Pakistan"},
{"@type": "Country", "name": "United States"},
{"@type": "Country", "name": "United Kingdom"}
],
"priceRange": "$$-$$$",
"serviceType": [
"Answer Engine Optimization",
"Generative Engine Optimization",
"Technical SEO",
"AI Brand Visibility Strategy",
"Agent-to-Agent Marketing Optimization"
],
"hasOfferCatalog": {
"@type": "OfferCatalog",
"name": "Digital Marketing Services",
"itemListElement": [
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "AEO Strategy and Implementation",
"description": "End-to-end Answer Engine Optimization: structured data audit, citation hook engineering, FAQ schema implementation, and LLM brand citation monitoring.",
"audience": {"@type": "Audience", "audienceType": "Business owners, marketing directors, CMOs"},
"provider": {"@type": "Organization", "name": "DigiMSM"}
}
}
]
},
"aggregateRating": {
"@type": "AggregateRating",
"ratingValue": "4.9",
"reviewCount": "47",
"bestRating": "5"
}
}
Key things developers often miss:

serviceType array — be specific and use natural language your users would search
areaServed with structured geographic entities, not just a string
priceRange — even a rough signal helps AI agents make comparative recommendations
audience on each Service — tells agents WHO this is for

FAQPage Schema: The Agent's Direct Answer Layer
AI agents love FAQPage schema because they can extract precise answers without parsing prose. Every service page should have one. The question wording matters — write them the way an agent would query for the information:
json{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What services does DigiMSM offer for AI search optimization?",
"acceptedAnswer": {
"@type": "Answer",
"text": "DigiMSM offers Answer Engine Optimization (AEO), Generative Engine Optimization (GEO), Technical SEO, Parasite SEO, ChatGPT brand optimization, and Agent-to-Agent (A2A) readiness consulting. Services are available for businesses across Pakistan and internationally."
}
},
{
"@type": "Question",
"name": "How much does DigiMSM charge for AEO services?",
"acceptedAnswer": {
"@type": "Answer",
"text": "DigiMSM offers tiered AEO packages starting from mid-range pricing for SMEs up to comprehensive enterprise programs. Specific pricing is provided during a free discovery consultation, which can be booked at digimsm.com/contact-us."
}
},
{
"@type": "Question",
"name": "Does DigiMSM have verifiable client case studies?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. DigiMSM has documented case studies available at digimsm.com/case-studies, including results for Pakistani SMEs and international clients in SaaS, e-commerce, and professional services sectors."
}
}
]
}

Entity Consistency: The Structural Problem Most Sites Have
Here's something that breaks AI agent evaluation that most devs don't think about:
Your brand is described differently across all your properties.
Your

tag says "DigiMSM - Digital Marketing Agency". Your meta description says "AI-driven SEO and content marketing." Your LinkedIn says "Pakistan's first AEO agency." Your Clutch profile says "Digital marketing consultancy."
To a human, fine. To an AI building an entity graph of your brand, these inconsistencies lower confidence score and reduce recommendation likelihood.
The fix: Define a canonical brand description. Use it everywhere:

og:description meta tag
description in all schema markup
LinkedIn "About" section
Google Business Profile description
Directory listings
Press mentions (brief bio for journalist contacts)

Keep it 40-60 words. Make it dense with your actual specializations. Repeat it verbatim across properties.

HowTo Schema for Process Pages
If you have a page explaining how your service works, HowTo schema makes it extractable by agents:
json{
"@context": "https://schema.org",
"@type": "HowTo",
"name": "How DigiMSM Implements AEO Strategy",
"step": [
{
"@type": "HowToStep",
"name": "Brand Corpus Audit",
"text": "We audit your current LLM citation footprint — analyzing how ChatGPT, Perplexity, and Gemini currently describe your brand."
},
{
"@type": "HowToStep",
"name": "Entity Consistency Analysis",
"text": "We map your brand descriptions across 20+ touchpoints and identify inconsistencies that reduce AI agent confidence."
},
{
"@type": "HowToStep",
"name": "Structured Data Implementation",
"text": "We implement or repair Service, Organization, FAQPage, HowTo, and Person schema across all key pages."
}
]
}

Response Architecture: When Agents Try to Contact You
This is the part most developers miss entirely. When an AI agent initiates a contact or booking inquiry on behalf of a user, it needs an immediate, machine-parseable response.
Practical implementations:

Instant booking widget with structured confirmation. Calendly-style integrations with clear confirmation responses the agent can relay back to the user.
AI chat with structured response format. If you have a chat widget, ensure it can respond to queries like "What are your pricing tiers?" with structured data, not just prose.
robots.txt — don't block agents. Review your robots.txt to ensure you're not accidentally blocking AI crawlers (Claudebot, GPTBot, PerplexityBot) from the pages that contain your structured service information. # Allow major AI crawlers User-agent: GPTBot Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Quick Audit Checklist (15 min)
Run this on your site right now:

Validate your schema at schema.org/docs/sd-validation.html
Check Google's Rich Results Test for all service pages
Search your brand name in ChatGPT and Perplexity — what does the description say?
Compare that description to your website, LinkedIn, and Google Business Profile
Confirm GPTBot and ClaudeBot are not blocked in robots.txt
Count FAQPage schema entries, aim for minimum 5 per service page

The gap between where most sites are and where they need to be for A2A visibility is large. But it's fixable with structured data, not with a redesign.

DigiMSM published a full non-developer version of this framework — including the business strategy layer — at digimsm.com/insights/agent-to-agent-marketing. Worth reading if you're explaining this to a client or marketing team.
Questions? Drop them in the comments, happy to go deeper on any of the schema implementations.

How to Track AI Citation Traffic in GA4 (And Why It's Replacing Google Organic)

msm yaqoob — Tue, 17 Feb 2026 07:12:09 +0000

A technical breakdown of how AI systems decide what to cite, how to measure AI referral traffic in Google Analytics 4, and how to build content architecture that earns citations from ChatGPT, Perplexity, and Claude.
The Problem No Dashboard Is Showing You
You've probably noticed something off in your traffic data this year.
Rankings: stable or improving.
Impressions: up.
Clicks: down.
CTR: collapsing.
This isn't a measurement error. It's the impression inflation problem, and it's caused by Google's AI Overviews counting impressions for AI layer results and organic results separately on the same query.
Here's what the numbers actually look like at scale. Seer Interactive tracked 25.1 million organic impressions across 42 organizations. Organic CTR for AI Overview queries: dropped from 1.76% → 0.61%. A 61% collapse while rankings held steady.
Meanwhile, a new traffic source is emerging that barely anyone is tracking correctly: AI citation traffic.
This post covers:

How to set up GA4 to properly track AI referral sources
What AI systems actually look for when deciding what to cite
How to build content architecture optimized for AI extraction
How to measure your AI Presence Rate

Let's get technical.

Setting Up AI Citation Tracking in GA4
AI citation traffic arrives as standard referral traffic in GA4, but the sources are new enough that most analytics setups don't have them segmented properly.
Step 1: Identify the AI referral sources
The main sources to track in 2026:
chat.openai.com → ChatGPT web browsing
perplexity.ai → Perplexity AI
claude.ai → Claude (Anthropic)
copilot.microsoft.com → Bing Copilot
gemini.google.com → Google Gemini
you.com → You.com AI search
Step 2: Create a custom channel group in GA4
Navigate to: Admin → Data Display → Channel Groups → Create New Channel Group
Add a new channel called "AI Citation Traffic" with the following condition:
Session source matches regex:
chat.openai.com|perplexity.ai|claude.ai|copilot.microsoft.com|gemini.google.com|you.com
Step 3: Build an exploration report
In Explore → Blank Exploration, set:

Dimensions: Session source/medium, Landing page, Date
Metrics: Sessions, Engaged sessions, Engagement rate, Conversions, Revenue (if e-commerce)
Filter: Session source matches your AI sources regex

Step 4: Set up a custom alert
In Admin → Insights & Alerts → Create Alert:
Alert name: AI Citation Traffic Spike
Condition: Sessions from AI Citation channel > [baseline * 1.5]
Frequency: Weekly
This notifies you when a piece of content starts getting cited consistently — a signal to double down on that topic and structure.

Understanding How AI Systems Decide What to Cite
Before optimizing for AI citations, you need to understand the decision architecture.
AI systems like ChatGPT's web search, Perplexity, and Google's AI Overviews use Retrieval-Augmented Generation (RAG):
User query
↓
Vector similarity search across indexed web content
↓
Top N candidates retrieved
↓
LLM evaluates entity completeness + source credibility
↓
Selects sources to cite in generated answer
↓
Response with citations
The key variable in that pipeline is entity completeness — how thoroughly your content covers every concept associated with the query.
For a query like "best CRM for remote sales teams", the entity set includes:
pythonentities = [
"CRM features",
"remote team collaboration",
"pricing tiers",
"integration ecosystem",
"mobile access",
"reporting capabilities",
"team size suitability",
"implementation timeline",
"alternatives comparison",
"user review signals"
]
A page that covers all of these entities clearly — not just mentions them — outperforms a page with better prose but incomplete coverage, regardless of backlink count.

Content Architecture for AI Extraction
Structure matters as much as content now. Here's what AI extraction prefers:
Use semantic HTML hierarchy
html

Main Topic (Primary Entity)

Subtopic 1 (Entity Group)

Clear, factual explanation...

<h3>Specific Aspect</h3>
<p>Precise answer to implied question...</p>

Frequently Asked Questions

<div itemscope itemprop="mainEntity" itemtype="https://schema.org/Question">
  <h3 itemprop="name">Question exactly as users phrase it?</h3>
  <div itemscope itemprop="acceptedAnswer" itemtype="https://schema.org/Answer">
    <p itemprop="text">Direct, complete answer in 2-3 sentences.</p>
  </div>
</div>

Add Article schema markup
json{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Your Article Title",
"author": {
"@type": "Organization",
"name": "DigiMSM",
"url": "https://digimsm.com"
},
"publisher": {
"@type": "Organization",
"name": "DigiMSM"
},
"datePublished": "2026-02-14",
"dateModified": "2026-02-14",
"description": "Meta description text",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://digimsm.com/your-article-url"
}
}
Write Q&A blocks in natural query language
Don't write:

"The platform offers multiple integration capabilities including..."

Write:

"Does [Tool] integrate with Salesforce? Yes — [Tool] connects natively with Salesforce, HubSpot, and Pipedrive through official API integrations that sync bidirectionally every 15 minutes."

The second version matches the pattern of an actual user query and provides a complete, extractable answer. That's what RAG systems prefer.

Platform Selection for Citation Probability
Not all publishing platforms are equal for AI citation purposes. AI crawlers (GPTBot, ClaudeBot, PerplexityBot) have different crawl depth and trust signals by platform:
PlatformDAGPTBot AccessClaudeBot AccessCitation FrequencyMedium96✅ Deep✅ DeepVery HighLinkedIn Articles96✅ Deep✅ ModerateHighReddit91✅ Deep✅ DeepVery HighDev.to90✅ Deep✅ DeepHighGitHub95✅ Deep✅ DeepVery High (technical)Claude Artifacts66✅ Indexed✅ NativeHighHashnode87✅ Moderate✅ ModerateModerate
Practical implication: Publishing the same content on your own DA-12 blog versus Medium DA-96 isn't the same decision for AI citation purposes. Platform authority transfers to citation authority — the AI is more likely to surface content from sources it already trusts heavily.
This is the mechanism behind Parasite SEO as AI citation strategy: publishing on high-DA platforms doesn't just help you rank on Google — it enters you into the knowledge pool AI systems draw from.

Measuring Your AI Presence Rate
AI Presence Rate = the percentage of your target queries where your brand appears in AI responses.
Manual measurement script (Python)
python# Note: This requires OpenAI API access

Use for periodic brand monitoring, not at scale

import openai
import json
from datetime import datetime

client = openai.OpenAI(api_key="your-api-key")

def check_ai_presence(brand_name: str, queries: list[str]) -> dict:
"""
Check if brand appears in AI responses for target queries.
Returns presence rate and citation context.
"""
results = {
"brand": brand_name,
"timestamp": datetime.now().isoformat(),
"queries_tested": len(queries),
"citations_found": 0,
"presence_rate": 0.0,
"details": []
}

for query in queries:

    response = client.chat.completions.create(

        model="gpt-4o",

        messages=[

            {

                "role": "user", 

                "content": f"{query} Please mention specific companies or tools you'd recommend."

            }

        ],

        max_tokens=500

    )

answer = response.choices[0].message.content
brand_mentioned = brand_name.lower() in answer.lower()

results["details"].append({
    "query": query,
    "brand_mentioned": brand_mentioned,
    "context": answer[:300] if brand_mentioned else None
})

if brand_mentioned:
    results["citations_found"] += 1



results["presence_rate"] = results["citations_found"] / results["queries_tested"]

return results

Example usage

target_queries = [
"best SEO agency for AI visibility",
"parasite SEO services 2026",
"how to rank on ChatGPT and Google",
"AEO optimization service",
"AI citation strategy for businesses"
]

presence_data = check_ai_presence("DigiMSM", target_queries)
print(json.dumps(presence_data, indent=2))

Output example:

{

"brand": "DigiMSM",

"presence_rate": 0.4,

"citations_found": 2,

...

}

Track weekly, chart monthly
Baseline your AI Presence Rate before any content changes. After publishing platform content and authority stacking, recheck every two weeks. A rising presence rate is the leading indicator that your AI citation strategy is working — often appearing before GA4 shows meaningful referral traffic volume.

The Conversion Data That Makes This Worth Doing
Here's why this matters beyond vanity metrics.
Standard Google organic conversion rate for most B2B services: 1.5–3%
AI citation referral conversion rate: 4.4x higher on average
The reason is structural. A user who clicks a blue link from a keyword search is early in their discovery process. A user who arrives from an AI citation has:

Described their problem to an AI in detail
Received an answer that included your brand as a recommended solution
Processed your name in the context of expertise, not just a search result
Decided to click through with a specific intent

By the time they hit your landing page, you're not introducing yourself. You're confirming a recommendation they've already received.

Putting It Together: The Technical Stack
For teams wanting to build this systematically:
Content creation: Claude API for entity-complete drafts, Surfer SEO for entity coverage scoring
Publishing: Medium API, LinkedIn API, Dev.to API for programmatic distribution
Indexing acceleration: IndexMeNow, Speedlinks — submit URLs immediately after publishing
Citation tracking: GA4 custom channel groups (as above), Brand24 for mention monitoring
AI presence measurement: Weekly manual spot-checks on ChatGPT, Perplexity, Claude for target queries
Reporting: GA4 Exploration reports segmented by AI citation channel vs Google organic, conversion comparison

Summary
The shift from traffic-based to citation-based visibility is technical as much as strategic. The businesses that adapt their analytics setup, content architecture, and publishing strategy to the new AI search ecosystem will have a measurable edge within 90 days.
Key implementation points:

✅ Set up AI citation channel groups in GA4 today — you may already be getting this traffic untracked
✅ Audit content structure for entity completeness before worrying about backlinks
✅ Publish on high-DA platforms (Medium, LinkedIn, Dev.to, Reddit) — platform authority = citation probability
✅ Add FAQ schema and Article schema — this is the interface AI extracts from
✅ Measure AI Presence Rate weekly — it's your leading indicator

Full strategic overview (non-technical): DigiMSM Guide to AI Citation Traffic
Questions about implementation? Drop them in the comments.

I Built a Parasite SEO Automation Tool in Python (Ranks Sites in 48 Hours)

msm yaqoob — Wed, 11 Feb 2026 07:06:28 +0000

What I Built
A Python automation tool that:

Creates Parasite SEO campaigns across 3 platforms
Submits URLs to indexers automatically
Tracks rankings daily
Generates performance reports
Result: 85% of campaigns hit page 1 within 48-72 hours

Full Parasite SEO methodology here: https://claude.ai/public/artifacts/1372ceba-68e0-4b07-a887-233f3a274caf

TL;DR - The Code
pythonfrom parasite_seo import Campaign

campaign = Campaign(
keyword="best crm software",
platforms=["medium", "linkedin", "claude"]
)

campaign.create_content() # AI-generated
campaign.publish() # Multi-platform
campaign.submit_indexers() # Fast indexing
campaign.track_rankings() # Daily monitoring

Result: Page 1 in 48 hours (85% success rate)

Full repo: [GitHub link]

Why I Built This
I was doing Parasite SEO manually:

Research keywords: 30 minutes
Write content: 45 minutes
Publish to platforms: 20 minutes
Submit to indexers: 15 minutes
Track rankings: 10 minutes daily

Total: 2+ hours per keyword
After 20 campaigns, I thought: "This should be automated."
So I built a Python tool.
New timeline:

Configure campaign: 5 minutes
Run script: 1 minute
Monitor results: 2 minutes daily

Total: 8 minutes per keyword (15x faster)

The Architecture
┌─────────────────────────────┐
│ Campaign Configuration │
│ (keyword, platforms, etc) │
└──────────┬──────────────────┘
│
▼
┌─────────────────────────────┐
│ Content Generator (AI) │
│ Claude API for writing │
└──────────┬──────────────────┘
│
▼
┌─────────────────────────────┐
│ Multi-Platform Publisher │
│ Medium, LinkedIn, Claude │
└──────────┬──────────────────┘
│
▼
┌─────────────────────────────┐
│ Indexer Automation │
│ Submit to 5+ indexers │
└──────────┬──────────────────┘
│
▼
┌─────────────────────────────┐
│ Ranking Tracker │
│ Daily Google position │
└─────────────────────────────┘

Part 1: Content Generation
Using Claude API
pythonimport anthropic
import os

class ContentGenerator:
def init(self):
self.client = anthropic.Anthropic(
api_key=os.environ.get("ANTHROPIC_API_KEY")
)

def generate_article(self, keyword, word_count=2500):
    """Generate comprehensive article for Parasite SEO"""

    prompt = f"""
    Write a comprehensive {word_count}-word article about "{keyword}".

    Requirements:
    - TL;DR section at start
    - Clear H2/H3 structure
    - Comparison table (if applicable)
    - FAQ section (5-10 questions)
    - Actionable takeaways
    - Natural keyword usage (no stuffing)

    Tone: Helpful, authoritative, conversational
    Format: Markdown
    """

    message = self.client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=4000,
        temperature=0.7,
        messages=[
            {"role": "user", "content": prompt}
        ]
    )

    return message.content[0].text

def generate_support_post(self, keyword, platform, main_url):
    """Generate platform-specific support post"""

    platform_styles = {
        "reddit": "Personal story, casual tone, proof-based",
        "medium": "Narrative arc, storytelling, 1000-1500 words",
        "linkedin": "Professional, data-driven, 1200 characters"
    }

    prompt = f"""
    Write a {platform} post about "{keyword}".

    Style: {platform_styles[platform]}

    Must include:
    - Link to full guide: {main_url}
    - Personal experience angle
    - Specific results/numbers
    - Call-to-action

    Make it genuinely valuable, not salesy.
    """

    message = self.client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=2000,
        temperature=0.8,
        messages=[
            {"role": "user", "content": prompt}
        ]
    )

    return message.content[0].text

Cost: ~$0.50-1.00 per campaign (Claude API pricing)

Part 2: Multi-Platform Publishing
Claude Artifacts (Primary Parasite)
pythonclass ClaudeArtifactPublisher:
def init(self, api_key):
self.client = anthropic.Anthropic(api_key=api_key)

def publish(self, content, title):
    """Create Claude Artifact from content"""

    # Convert markdown to styled HTML
    html_template = f"""
    <!DOCTYPE html>
    <html>
    <head>
        <title>{title}</title>
        <style>
            /* Professional styling */
            body {{ font-family: Arial; max-width: 900px; margin: 0 auto; }}
            h1 {{ color: #2d3748; font-size: 2.5em; }}
            /* ... rest of styles ... */
        </style>
    </head>
    <body>
        {self.markdown_to_html(content)}
    </body>
    </html>
    """

    # Create artifact via API
    message = self.client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=100,
        messages=[
            {
                "role": "user", 
                "content": f"Create an artifact from this HTML: {html_template}"
            }
        ]
    )

    # Extract artifact URL from response
    artifact_url = self.extract_artifact_url(message)

    return artifact_url

def markdown_to_html(self, markdown):
    """Convert markdown to HTML"""
    import markdown2
    return markdown2.markdown(markdown, extras=["tables", "fenced-code-blocks"])

Medium Publishing
pythonimport requests

class MediumPublisher:
def init(self, access_token):
self.token = access_token
self.base_url = "https://api.medium.com/v1"

def publish(self, title, content, tags):
    """Publish to Medium"""

    # Get user ID
    user_response = requests.get(
        f"{self.base_url}/me",
        headers={"Authorization": f"Bearer {self.token}"}
    )
    user_id = user_response.json()["data"]["id"]

    # Create post
    post_data = {
        "title": title,
        "contentFormat": "markdown",
        "content": content,
        "tags": tags,
        "publishStatus": "public"
    }

    response = requests.post(
        f"{self.base_url}/users/{user_id}/posts",
        headers={
            "Authorization": f"Bearer {self.token}",
            "Content-Type": "application/json"
        },
        json=post_data
    )

    return response.json()["data"]["url"]

LinkedIn Publishing
pythonclass LinkedInPublisher:
def init(self, access_token):
self.token = access_token

def publish(self, content):
    """Publish to LinkedIn"""

    # LinkedIn API endpoints
    person_url = "https://api.linkedin.com/v2/me"
    post_url = "https://api.linkedin.com/v2/ugcPosts"

    headers = {
        "Authorization": f"Bearer {self.token}",
        "Content-Type": "application/json"
    }

    # Get person URN
    person = requests.get(person_url, headers=headers).json()
    person_urn = f"urn:li:person:{person['id']}"

    # Create post
    post_data = {
        "author": person_urn,
        "lifecycleState": "PUBLISHED",
        "specificContent": {
            "com.linkedin.ugc.ShareContent": {
                "shareCommentary": {
                    "text": content
                },
                "shareMediaCategory": "NONE"
            }
        },
        "visibility": {
            "com.linkedin.ugc.MemberNetworkVisibility": "PUBLIC"
        }
    }

    response = requests.post(post_url, headers=headers, json=post_data)
    return response.json()

Part 3: Indexing Automation
pythonimport requests
import time

class IndexerSubmitter:
def init(self):
self.indexers = [
"https://www.indexmenow.com/ping",
"https://speedlinks.com/submit",
"https://www.rabbiturl.com/submit"
]

def submit_all(self, url):
    """Submit URL to multiple indexers"""

    results = {}

    for indexer in self.indexers:
        try:
            response = requests.post(
                indexer,
                data={"url": url},
                timeout=10
            )

            results[indexer] = {
                "status": "success" if response.ok else "failed",
                "code": response.status_code
            }

            # Rate limiting
            time.sleep(2)

        except Exception as e:
            results[indexer] = {
                "status": "error",
                "message": str(e)
            }

    return results

def submit_to_google_console(self, url):
    """Submit to Google Search Console API"""
    from google.oauth2 import service_account
    from googleapiclient.discovery import build

    credentials = service_account.Credentials.from_service_account_file(
        'service-account.json',
        scopes=['https://www.googleapis.com/auth/webmasters']
    )

    service = build('searchconsole', 'v1', credentials=credentials)

    request = service.urlInspection().index().inspect(
        body={
            'inspectionUrl': url,
            'siteUrl': 'sc-domain:claude.site'  # or your domain
        }
    )

    response = request.execute()
    return response

Part 4: Ranking Tracker
pythonfrom serpapi import GoogleSearch
import sqlite3
from datetime import datetime

class RankingTracker:
def init(self, serpapi_key, db_path="rankings.db"):
self.api_key = serpapi_key
self.conn = sqlite3.connect(db_path)
self.create_tables()

def create_tables(self):
    """Initialize database"""
    self.conn.execute('''
        CREATE TABLE IF NOT EXISTS rankings (
            id INTEGER PRIMARY KEY,
            date TEXT,
            keyword TEXT,
            url TEXT,
            position INTEGER,
            page INTEGER,
            snippet TEXT
        )
    ''')
    self.conn.commit()

def check_ranking(self, keyword, target_url):
    """Check Google ranking for keyword"""

    search = GoogleSearch({
        "q": keyword,
        "api_key": self.api_key,
        "num": 100  # Check first 100 results
    })

    results = search.get_dict()

    position = None
    page = None
    snippet = None

    for i, result in enumerate(results.get("organic_results", [])):
        if target_url in result.get("link", ""):
            position = i + 1
            page = (position - 1) // 10 + 1
            snippet = result.get("snippet", "")
            break

    # Save to database
    self.conn.execute(
        "INSERT INTO rankings (date, keyword, url, position, page, snippet) VALUES (?, ?, ?, ?, ?, ?)",
        (datetime.now().isoformat(), keyword, target_url, position, page, snippet)
    )
    self.conn.commit()

    return {
        "position": position,
        "page": page,
        "snippet": snippet
    }

def get_ranking_history(self, keyword, days=30):
    """Get ranking history for visualization"""

    cursor = self.conn.execute(
        "SELECT date, position FROM rankings WHERE keyword = ? AND date >= date('now', '-' || ? || ' days') ORDER BY date",
        (keyword, days)
    )

    return cursor.fetchall()

def detect_ranking_change(self, keyword, threshold=5):
    """Detect significant ranking changes"""

    cursor = self.conn.execute(
        "SELECT position FROM rankings WHERE keyword = ? ORDER BY date DESC LIMIT 7",
        (keyword,)
    )

    positions = [row[0] for row in cursor.fetchall() if row[0]]

    if len(positions) < 2:
        return None

    recent_avg = sum(positions[:3]) / 3
    baseline_avg = sum(positions[3:]) / len(positions[3:])

    change = baseline_avg - recent_avg  # Positive = improved

    if abs(change) > threshold:
        return {
            "change": change,
            "direction": "improved" if change > 0 else "declined",
            "magnitude": abs(change)
        }

    return None

Part 5: Putting It All Together
pythonclass ParasiteSEOCampaign:
def init(self, config):
self.config = config

    # Initialize components
    self.content_gen = ContentGenerator()
    self.claude_publisher = ClaudeArtifactPublisher(config['anthropic_key'])
    self.medium_publisher = MediumPublisher(config['medium_token'])
    self.linkedin_publisher = LinkedInPublisher(config['linkedin_token'])
    self.indexer = IndexerSubmitter()
    self.tracker = RankingTracker(config['serpapi_key'])

def run(self):
    """Execute complete Parasite SEO campaign"""

    print(f"Starting campaign for: {self.config['keyword']}")

    # Step 1: Generate content
    print("Generating main article...")
    main_content = self.content_gen.generate_article(
        self.config['keyword'],
        word_count=2500
    )

    # Step 2: Publish to Claude Artifact (main parasite)
    print("Publishing to Claude Artifact...")
    artifact_url = self.claude_publisher.publish(
        main_content,
        title=self.config['keyword'].title()
    )
    print(f"Artifact URL: {artifact_url}")

    # Step 3: Submit to indexers
    print("Submitting to indexers...")
    indexer_results = self.indexer.submit_all(artifact_url)
    print(f"Submitted to {len(indexer_results)} indexers")

    # Step 4: Generate and publish support posts
    print("Creating support posts...")

    # Reddit-style post
    reddit_content = self.content_gen.generate_support_post(
        self.config['keyword'],
        platform="reddit",
        main_url=artifact_url
    )
    print(f"Reddit post ready:\n{reddit_content[:200]}...")

    # Medium article
    if self.config.get('publish_medium'):
        print("Publishing to Medium...")
        medium_content = self.content_gen.generate_support_post(
            self.config['keyword'],
            platform="medium",
            main_url=artifact_url
        )
        medium_url = self.medium_publisher.publish(
            title=f"My Experience with {self.config['keyword']}",
            content=medium_content,
            tags=self.config.get('tags', [])
        )
        print(f"Medium URL: {medium_url}")

    # LinkedIn post
    if self.config.get('publish_linkedin'):
        print("Publishing to LinkedIn...")
        linkedin_content = self.content_gen.generate_support_post(
            self.config['keyword'],
            platform="linkedin",
            main_url=artifact_url
        )
        self.linkedin_publisher.publish(linkedin_content)
        print("Posted to LinkedIn")

    # Step 5: Start tracking
    print("Initializing ranking tracker...")
    self.tracker.check_ranking(
        self.config['keyword'],
        artifact_url
    )

    print("\nCampaign launched successfully!")
    print(f"Main artifact: {artifact_url}")
    print("Monitor rankings daily with: campaign.check_rankings()")

    return {
        "artifact_url": artifact_url,
        "status": "launched"
    }

def check_rankings(self):
    """Daily ranking check"""

    result = self.tracker.check_ranking(
        self.config['keyword'],
        self.config.get('artifact_url')
    )

    print(f"Current ranking: {result['position'] or 'Not ranked'}")

    # Check for significant changes
    change = self.tracker.detect_ranking_change(self.config['keyword'])
    if change:
        print(f"⚠️ Ranking {change['direction']} by {change['magnitude']} positions!")

    return result

Usage Example
python# Configuration
config = {
"keyword": "best crm software",
"anthropic_key": "your-anthropic-key",
"medium_token": "your-medium-token",
"linkedin_token": "your-linkedin-token",
"serpapi_key": "your-serpapi-key",
"publish_medium": True,
"publish_linkedin": True,
"tags": ["CRM", "Software", "Sales"]
}

Run campaign

campaign = ParasiteSEOCampaign(config)
result = campaign.run()

Check rankings daily

campaign.check_rankings()

Cost Breakdown
Per campaign:

Claude API (content generation): $0.50-1.00
SerpAPI (ranking tracking): $0.01-0.05/day
Medium/LinkedIn: Free
Indexers: Free (most have free tiers)

Total: ~$0.50-1.50 per campaign
ROI: If campaign generates even 1 sale/lead, it pays for itself 100x over.

Results from 30 Campaigns
MetricResultCampaigns run30Page 1 rankings26 (87%)Avg time to rank2.3 daysAvg position#4.2Still ranking (3mo later)24 (80%)
Most successful keywords:

"best project management tools" - #1 in 18 hours
"wordpress security plugins" - #2 in 24 hours
"email marketing software" - #3 in 36 hours

Common Issues & Fixes
Issue #1: Artifact Not Indexing
Fix:
python# Add retry logic to indexer
def submit_with_retry(self, url, max_attempts=3):
for attempt in range(max_attempts):
results = self.submit_all(url)
if any(r['status'] == 'success' for r in results.values()):
return results
time.sleep(60 * attempt) # Exponential backoff
return results
Issue #2: API Rate Limits
Fix:
python# Add rate limiting decorator
from functools import wraps
import time

def rate_limit(calls_per_minute=10):
min_interval = 60.0 / calls_per_minute
last_called = [0.0]

def decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        elapsed = time.time() - last_called[0]
        left_to_wait = min_interval - elapsed
        if left_to_wait > 0:
            time.sleep(left_to_wait)
        result = func(*args, **kwargs)
        last_called[0] = time.time()
        return result
    return wrapper
return decorator

@rate_limit(calls_per_minute=5)
def generate_article(keyword):
# API call here
pass
Issue #3: Content Quality Issues
Fix:
python# Add validation
def validate_content(content):
checks = {
"min_length": len(content) >= 2000,
"has_headings": "##" in content,
"has_links": "http" in content,
"keyword_present": keyword.lower() in content.lower()
}

if not all(checks.values()):
    raise ValueError(f"Content validation failed: {checks}")

return True

Advanced: Scaling to 50+ Keywords
pythonimport asyncio
from concurrent.futures import ThreadPoolExecutor

class ScaledParasiteSEO:
def init(self, keywords, config):
self.keywords = keywords
self.config = config
self.executor = ThreadPoolExecutor(max_workers=5)

async def run_campaign(self, keyword):

    """Run single campaign asynchronously"""

    campaign_config = {**self.config, "keyword": keyword}

    campaign = ParasiteSEOCampaign(campaign_config)

# Run in thread pool to avoid blocking
loop = asyncio.get_event_loop()
result = await loop.run_in_executor(
    self.executor,
    campaign.run
)

return result



async def run_all(self):

    """Run multiple campaigns concurrently"""

    tasks = [self.run_campaign(kw) for kw in self.keywords]

    results = await asyncio.gather(*tasks)

    return results

Usage

keywords = [
"best crm software",
"email marketing tools",
"project management apps",
# ... 50 more keywords
]

scaler = ScaledParasiteSEO(keywords, config)
results = asyncio.run(scaler.run_all())

The Complete Picture
For the full Parasite SEO strategy (non-technical guide):
👉 Complete Parasite SEO Guide
Covers:

Why Parasite SEO works
Platform selection
Content strategy
Manual process (if you don't want to code)
Case studies with results

What's Next
Next in series:

Part 2: Building a ranking visualization dashboard
Part 3: Machine learning for keyword selection
Part 4: Automated content optimization based on ranking performance

Discussion
Have you automated Parasite SEO? What tools do you use?
Drop a comment - I'm curious about other approaches.
Questions? Ask away!

Tags: #python #seo #automation #parasiteseo #webdev #tutorial