DEV Community: Wahib EL KHADIRI

I Taught an Open-Source SEO Tool to Check Whether Your Site Is Readable by AI Agents

Wahib EL KHADIRI — Tue, 21 Jul 2026 23:08:35 +0000

Most SEO audit tools still ask the questions search engines cared about ten years ago: is your title tag the right length, do your images have alt text, is your sitemap valid. Those still matter. But a growing chunk of how people find content now goes through an AI assistant instead of a search results page, and almost no audit tool checks whether your site is even reachable by one.

I picked up an open issue on open-seo (an open-source Semrush/Ahrefs alternative) to close that gap, and it turned into a good lesson in scoping a feature for someone else's codebase instead of your own.

The actual question

"Is this site AI-readable?" breaks down into concrete, checkable things:

Does robots.txt block the crawlers that build AI search indexes and answer live user requests?
Does the site have an llms.txt — the emerging convention that gives an AI assistant a clean map of your important pages?
Does the site advertise a Markdown alternate for its pages, so an assistant can read clean content instead of parsing full HTML?

Straightforward to state. The interesting part was not flagging noise.

Lesson 1: not every robots.txt block is a signal

A site with Disallow: / for every crawler made a deliberate, site-wide choice. Reporting "GPTBot is blocked!" on a site that blocks everything isn't useful information — it's noise dressed up as an insight. So the check only flags an AI agent when it's treated worse than the generic rules:

const rootUrl = `${origin}/`;
if ((robots.isAllowed(rootUrl, GENERIC_PROBE_AGENT) ?? true) === false) {
  return []; // whole site is closed — not an AI-specific signal
}

I also grouped agents by why they visit instead of firing one issue per bot. GPTBot and ClaudeBot are training crawlers; blocking them is often an intentional content policy and barely worth a heads-up. ChatGPT-User and Claude-User fetch a page live because an actual human asked an assistant about it right now — blocking those is a much bigger deal, because you're breaking a request in progress, not opting out of a training set. Same signal, completely different severity, so they're separate issue types with separate default severities instead of one generic "AI crawler blocked" bucket.

Lesson 2: absence isn't always an error

llms.txt and Markdown alternates are both new enough that almost nobody has them yet. If the check flagged every single audited site for "missing Markdown alternates," it would train users to ignore the audit tool's warnings entirely — the boy-who-cried-wolf failure mode of any linter. Both checks are info severity with copy that says, explicitly, "this is common, not an error." The one genuinely broken state — a site that serves llms.txt but violates its one hard structural requirement (starts with an H1) — is the only case that gets bumped to warning.

Lesson 3: match the codebase's own constraints, not your own instincts

My first instinct for "does this page have a Markdown alternate" was to persist a new boolean column on the crawled-pages table. Then I actually looked at how the project's crawl pipeline works: full page data lives in a database table, but only a slim summary (title, status code, a few other fields) survives between crawl phases as durable workflow state — link lists and everything else deliberately get dropped to keep memory bounded on 10,000-page crawls.

Adding a persisted column meant a schema migration, which meant touching both the SQLite and Postgres schema paths this project supports side by side — a much bigger, riskier diff for a feature that doesn't actually need to survive past the audit run. So instead the flag rides through the same transient summary object the pipeline already carries between phases, and the finding — not the raw per-page flag — is the only thing that gets persisted, as a single site-level issue row. Same pattern the project already used for the file-based llms.txt check; I just extended it instead of inventing a new one.

Lesson 4: extend the open PR, don't fragment the review

I'd already opened a PR for the robots.txt and llms.txt checks. Before anyone had reviewed it, I built the Markdown-alternates check too — and instead of opening a third PR, I pushed a second commit onto the same branch. The maintainer hasn't looked at any of it yet, so there was no in-flight review to disrupt, and one PR that closes three-quarters of the original issue is easier to review than three PRs that each close a quarter of it and reference each other.

Result

19 unit tests, no schema changes, no new external API calls beyond one extra fetch for llms.txt. The PR's open here if you want to see the actual diff: every-app/open-seo#122.

every-app / open-seo

Open source alternative to Semrush and Ahrefs

OpenSEO

Open source alternative to Semrush and Ahrefs

OpenSEO is an SEO tool for the people. If tools like Semrush or Ahrefs are too expensive or bloated, OpenSEO is a pay-as-you-go alternative that you actually control.

All-in-one SEO tool for you and your AI agent.

Connect with any agent like Claude Code, OpenClaw or Hermes. We have pre-built skills, but you can build your own to tailor OpenSEO to your needs.

Hosted Version

Try OpenSEO for free on our website. If you want to support the project, a hosted subscription is $10/month.

openseo.so

Why use OpenSEO?

Best in class MCP and AI Skills.
Modern, simple UI.
- Focused workflows instead of a bloated, complex SEO suite.
No subscriptions.
- Bring your own DataForSEO API key and pay only for what you use.
Fork and vibe code your own custom tool.

Main SEO Workflows

Keyword research
Rank tracking
Competitor Insights
Backlinks
Site…

View on GitHub

The code itself was the easy part. The actual work was reading how someone else already solved similar problems in their codebase before writing a single line — the "how do I avoid noise" and "how do I fit this project's existing shape" questions mattered more than the crawler-detection logic itself.

I Shipped My First Rust Release, and CI Turned Red Twice in 20 Minutes

Wahib EL KHADIRI — Tue, 21 Jul 2026 22:43:39 +0000

I shipped the first public release of my Rust project last night. Within about twenty minutes, CI turned red twice. Here's what broke, how I found it, and the two calls I made under a bit of time pressure.

The project

AgentOS is a runtime layer for AI agents — the infrastructure around an agent, not the agent logic itself: supervised lifecycle, a gRPC message bus, a health endpoint, an SSE event stream, a secrets vault, and deterministic time-travel replay (every LLM/tool exchange gets journaled, so a run can be replayed offline with no API cost, and forked into alternate timelines from any checkpoint).

It's a 10-crate Rust workspace, MIT/Apache-2.0. I'd been building it privately for a while. Last night I flipped the repo to public and tagged the first alpha release. That's when things got interesting.

Failure #1: a clippy lint that didn't exist last month

The repo had been private, so CI hadn't run against a fresh clippy in a while. The moment I pushed a commit after going public, this showed up:

error: you seem to want to iterate on a map's values
   --> crates/bus/src/grpc.rs:631:37
    |
631 |         for (_agent_id, senders) in subs.iter() {
    |                                     ^^^^^^^^^^^
    |
    = help: use the corresponding method
631 -         for (_agent_id, senders) in subs.iter() {
631 +         for senders in subs.values() {

clippy::for_kv_map, denied by -D warnings in CI. Fair catch — the code was iterating a HashMap and discarding the key on every iteration, which is exactly what .values() is for. One-line fix:

// before
for (_agent_id, senders) in subs.iter() {

// after
for senders in subs.values() {

Pushed, CI green again. This one was boring, which is the best kind of bug to have.

Failure #2: the release matrix, at 12:33am

The bigger one showed up when I tagged v0.1.0-alpha. The release workflow builds five targets in parallel — Linux x64, Linux arm64, macOS Intel, macOS Apple Silicon, Windows — and packages each into a binary release with checksums.

Four went green. Linux arm64 died with:

error: failed to run custom build command for `openssl-sys v0.9.116`
...
Could not find directory of OpenSSL installation
$HOST = x86_64-unknown-linux-gnu
$TARGET = aarch64-unknown-linux-gnu

openssl-sys needs a real OpenSSL installation (headers + libs) for the target architecture to link against, and cross-compiling from an x86_64 GitHub Actions runner to arm64 doesn't have that available without extra setup. Not a code bug — a missing piece of cross-compilation infrastructure.

I had three honest options at that point:

Block the whole release until arm64 cross-compiling was solved properly
Vendor OpenSSL and wire up a cross-linker for the arm64 target under time pressure, late at night, for a release I hadn't tested that way before
Ship the four targets that work, drop arm64 from this release, and track the real fix as its own issue

I went with option 3. Four platforms is still a real alpha release that covers the overwhelming majority of contributors trying it out. Blocking on a cross-compilation edge case for a Linux architecture almost nobody in the alpha audience runs felt like optimizing for the wrong thing at midnight. And silently vendoring a rushed fix into a security-relevant dependency (TLS) is exactly the kind of shortcut that comes back to bite you.

So: pulled the target from the matrix, re-tagged, watched the four remaining builds go green, and filed the real fix as its own issue with three candidate approaches laid out (the best one is probably migrating off openssl-sys to rustls entirely, which would remove the system OpenSSL dependency for every target, not just arm64 — cross-compilation problems like this are usually a sign the dependency choice needs revisiting, not just the CI config).

The one check I'm glad I ran before either of this

Before any of the above, I'd actually run the README's own quickstart end to end — cargo build --workspace, then run the example agent — specifically to catch the gap between "the README claims this works" and "this actually works." It did, and the real terminal output (supervisor spawning the agent, health server on :8080, gRPC bus on :50051, an SSE stream on :8081) is now in the README instead of a hypothetical example. I also downloaded the actual released Windows binary afterward and ran agentOS.exe --version against it — small thing, but it's the difference between "the release workflow exited 0" and "a stranger who downloads this file gets a working program."

Neither bug was catastrophic. Both were the kind of thing that's mildly stressful in the moment and completely mundane in hindsight — which is most of what real engineering work actually looks like, release-day war stories included.

If you want to poke at it, the repo's here: github.com/WAHIB-EL-KHADIRI/AgentOS. There are a handful of labeled good first issues if anyone wants to dig into a self-contained piece of it — I'm around to answer questions on any of them.

WAHIB-EL-KHADIRI / AgentOS

Runtime infrastructure for AI agents — lifecycle, supervision, secrets, and deterministic time-travel replay. Built in Rust.

AgentOS - Runtime Infrastructure for AI Agents

AgentOS is an open-source runtime layer for AI agents. It focuses on the infrastructure around agents: lifecycle, supervision, messaging, state, secrets observability, and trace replay.

Created by WAHIB EL KHADIRI — founder and architect.

Most agent frameworks help you build an agent workflow. AgentOS focuses on what happens after that workflow needs to run as a long-lived process, fail clearly restart carefully, and be inspected after the fact.

See it run

Real output from a fresh clone — no API key required to bring the runtime up:

$ cargo run -p agentos-cli -- run --agent examples/simple_agent.toml
INFO agentos_kernel::supervisor: agent spawned and running agent_id=agent_simple_agent state=Running name=simple-agent
INFO agentos_kernel::agent: agent loop started agent_id=agent_simple_agent
INFO agentos_kernel::events: system event emitted event=agent.spawned seq=0
INFO agentOS::run: AgentOS runtime started agent_id=agent_simple_agent host=127.0.0.1 http_port=8080 grpc_port=50051 sse_port=8081
INFO agentos_kernel::health: health server listening on 127.0.0.1:8080
INFO agentos_bus::grpc: gRPC bus server listening on 127.0.0.1:50051

…

View on GitHub