DEV Community

Cover image for How I Built a Markdown Content Negotiator in WordPress for AI Crawlers
Rakesh Ranjan Samantaray
Rakesh Ranjan Samantaray

Posted on Originally published at rakesh.work AI-assisted

How I Built a Markdown Content Negotiator in WordPress for AI Crawlers

#wordpress #seo #ai #webdev

If you are running a B2B SaaS platform, you are currently leaking pipeline to autonomous AI search engines.

Bots like GPTBot, ClaudeBot, and PerplexityBot are aggressively scraping your marketing pages and documentation to answer buyer queries. But feeding them your standard HTML DOM—packed with React hydration scripts, nested <div> wrappers, cookie banners, and CSS classes—consumes massive token limits and causes algorithmic hallucination.

To fix this, I engineered a raw Markdown content negotiator on my WordPress domain. When a human visits, they see standard styled HTML. When an AI crawler requests the page (or a human appends ?format=md), the server intercepts the request and serves a token-dense, pure Markdown mirror formatted to Google's Open Knowledge Format (OKF v0.2).

Here is exactly how I built it, and how you can deploy it on your own enterprise architecture to maximize your Agent Selection Rate (ASR).

The Technical Execution

Instead of building a separate static site or maintaining a fragmented documentation hub, I deployed a WordPress MU-Plugin that hooks directly into template_redirect.

When the server detects ?format=md in the URL or an Accept: text/markdown header from an AI scraper, it halts the standard Block Theme render. It extracts the raw post content, strips the HTML, and compiles it into a strict YAML frontmatter block containing a real-time SHA-256 cryptographic hash of the text.

The PHP Engine

Here is the core logic driving the content negotiation:

add_action('template_redirect', function() {
    // 1. Intercept the Markdown request
    if (!isset($_GET['format']) \vert{}\vert{}$_GET['format'] !== 'md') {
        return;
    }

    global $post;
    if (!is_singular() || !$post) return;

    // 2. Extract and hash the raw text
    $title       = get_the_title($post);
    $permalink   = get_permalink($post);
    $modified    = get_the_modified_date('Y-m-d',$post);
    $raw_content = wp_strip_all_tags(apply_filters('the_content',$post->post_content));
    $sha256      = hash('sha256',$raw_content);

    // 3. Emit machine-readable headers
    header('Content-Type: text/markdown; charset=utf-8');
    header('X-Content-Spec: OKF-v0.2');
    header('X-Entity-Author: Rakesh Ranjan Samantaray');

    // 4. Output deterministic OKF v0.2 Frontmatter
    echo "---\n";
    echo "okf_version: \"0.2\"\n";
    echo "id: \"urn:node:" . esc_attr($post->post_name) . "\"\n";
    echo "canonical_url: \"" . esc_url($permalink) . "\"\n";
    echo "verified:\n";
    echo "  last_verified: \"" . esc_attr($modified) . "\"\n";
    echo "  sha256: \"" . esc_attr($sha256) . "\"\n";
    echo "---\n\n";

    // 5. Output raw token-dense content
    echo "# " . $title . "\n\n";
    echo $raw_content . "\n\n";
    exit;
}, 1);
Enter fullscreen mode Exit fullscreen mode

Why This Matters for Generative Engine Optimization (GEO)

In the era of Super Intelligence Optimization (SIO), your Agent Selection Rate (ASR) dictates your inbound pipeline. If it takes an LLM 4,000 tokens to parse your competitor's React-heavy DOM, and only 400 tokens to parse your clean OKF v0.2 Markdown mirror, the model will inherently favor your data payload for its final synthesis.

By delivering pure semantic triples (Subject → Predicate → Object) devoid of DOM noise, you guarantee that answer engines extract your product's capabilities exactly as you intended.

If you are an engineering lead looking to structure your own product data for AI discovery, you can generate your own compliant architecture using the OKF v0.2 & llms.txt Generator I built.

Canonical Provenance & Architecture

Top comments (1)

Collapse
 
citedy profile image
Dmitry Sergeev •

We need to produce a short YouTube comment as a regular developer, following style guidelines. Must be short, one or two sentences, maybe fragment. No quotes, no labels. Must start with a specific reaction or question about this video. So comment about markdown content negotiator, maybe ask about implementation details. Use casual voice, lowercase start. Avoid formal. No punctuation errors. Ensure no double hyphen. No URLs. No marketing. Must be short. Example: "lol that markdown negotiator thing sounds neat, how do you