<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Owen King</title>
    <description>The latest articles on DEV Community by Owen King (@owen_king_e3d7d83a33c46c8).</description>
    <link>https://dev.to/owen_king_e3d7d83a33c46c8</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4137371%2F74993d7a-703c-4515-9098-df89a5b3811e.png</url>
      <title>DEV Community: Owen King</title>
      <link>https://dev.to/owen_king_e3d7d83a33c46c8</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/owen_king_e3d7d83a33c46c8"/>
    <language>en</language>
    <item>
      <title>HTML to Markdown to embeddings: a RAG ingestion pipeline in ~50 lines of JavaScript</title>
      <dc:creator>Owen King</dc:creator>
      <pubDate>Sat, 26 Sep 2026 08:35:25 +0000</pubDate>
      <link>https://dev.to/owen_king_e3d7d83a33c46c8/html-to-markdown-to-embeddings-a-rag-ingestion-pipeline-in-50-lines-of-javascript-3mon</link>
      <guid>https://dev.to/owen_king_e3d7d83a33c46c8/html-to-markdown-to-embeddings-a-rag-ingestion-pipeline-in-50-lines-of-javascript-3mon</guid>
      <description>&lt;p&gt;Every RAG system has the same boring first mile: fetch a page, strip the HTML noise, split the text, embed the chunks, put the vectors somewhere useful. Most tutorials spend 300 words on BeautifulSoup selectors. Here is the version where the first mile is one API call, in about 50 lines.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 1: URL to clean Markdown + chunks + embeddings, in one call&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;API&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https://rag-scrape-api.owerryking.workers.dev&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;API&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/scrape&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;POST&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Content-Type&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https://example.com/pricing&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;chunkSize&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;embed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;markdown&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;chunks&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="c1"&gt;// chunks: [{ text, embedding }] - heading-aligned, bge-small (384-dim)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No Cheerio, no readability heuristics, no "how do I split on token boundaries" - the chunks are aligned to heading boundaries (which matters: a chunk that starts mid-section retrieves badly), and each one arrives with its embedding already computed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 2: upsert into your vector store&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;upsert&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;hash&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;text&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="na"&gt;vector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;embedding&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https://example.com/pricing&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That loop looks identical for Chroma, Qdrant, LanceDB, Pinecone or a local array. Because chunks are heading-aligned, your citations point at coherent sections instead of arbitrary 500-token windows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 3: change-aware re-indexing (the part everyone skips)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Re-embedding an unchanged page wastes money and time. Pass a content hash and skip work you have already done:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;r2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;API&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/scrape&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;POST&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Content-Type&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;embed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;ifNoneHash&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;lastHash&lt;/span&gt; &lt;span class="p"&gt;}),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;notModified&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;r2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;notModified&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// nothing changed, zero embedding cost&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run that nightly over your sources and only genuinely updated pages cost you anything.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 4: whole documentation sites&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;crawl&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;API&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/crawl&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;POST&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Content-Type&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https://docs.example.com&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;maxPages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt; &lt;span class="p"&gt;}),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;llmsTxt&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;crawl&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="c1"&gt;// pages: same shape as /scrape, already chunked if you ask&lt;/span&gt;
&lt;span class="c1"&gt;// llmsTxt: bonus - a ready-to-serve llms.txt for the site&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same-host BFS, capped at 100 pages per call, one quota unit per successfully crawled page.&lt;/p&gt;

&lt;p&gt;The whole pipeline - fetch, clean, chunk, embed, refresh - collapses to four fetch calls. The interesting work (retrieval quality, prompts, evals) starts after the first mile ends.&lt;/p&gt;

&lt;p&gt;Free tier: 50 URLs/month, no card. OpenAPI spec: &lt;a href="https://rag-scrape-api.owerryking.workers.dev/openapi.json" rel="noopener noreferrer"&gt;https://rag-scrape-api.owerryking.workers.dev/openapi.json&lt;/a&gt;&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>tutorial</category>
      <category>ai</category>
      <category>api</category>
    </item>
    <item>
      <title>Generate an llms.txt for any website in 30 seconds (free, no signup)</title>
      <dc:creator>Owen King</dc:creator>
      <pubDate>Sat, 26 Sep 2026 08:34:10 +0000</pubDate>
      <link>https://dev.to/owen_king_e3d7d83a33c46c8/generate-an-llmstxt-for-any-website-in-30-seconds-free-no-signup-2462</link>
      <guid>https://dev.to/owen_king_e3d7d83a33c46c8/generate-an-llmstxt-for-any-website-in-30-seconds-free-no-signup-2462</guid>
      <description>&lt;p&gt;Google reads robots.txt. AI assistants increasingly read llms.txt - a simple markdown file at the root of your site that tells language models what is worth reading, in the format they prefer.&lt;/p&gt;

&lt;p&gt;If you are running a docs site, a blog, or a product, here is the short version of why it matters: AI tools and agents increasingly crawl the web looking for content to quote and cite. A good llms.txt gives them a curated, structured summary instead of forcing them to guess from navigation menus and sitemap XML. The spec lives at llmstxt.org, and early adopters report their content appearing more accurately in AI-generated answers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What a good llms.txt contains&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One-line description of the site&lt;/li&gt;
&lt;li&gt;Links to your key pages with short markdown descriptions&lt;/li&gt;
&lt;li&gt;Optional: the full content of important pages inline&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Generating one in 30 seconds (free, no signup)&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Go to &lt;a href="https://rag-scrape-api.owerryking.workers.dev/llms-txt-generator" rel="noopener noreferrer"&gt;https://rag-scrape-api.owerryking.workers.dev/llms-txt-generator&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Enter your site URL&lt;/li&gt;
&lt;li&gt;Copy or download the generated llms.txt and host it at yoursite.com/llms.txt&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The generator crawls your site (up to 20 pages), pulls the titles and descriptions, and formats everything per the llmstxt.org spec. No account needed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Prefer automation?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;There is an API route for that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;POST https://rag-scrape-api.owerryking.workers.dev/llms-txt
{ "url": "https://yoursite.com", "maxPages": 20 }
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It returns ready-to-serve llms.txt content. Wire it to a cron job and your file stays current as your site changes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Three tips&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Run the generator after your sitemap exists - crawlers follow sitemaps and internal links&lt;/li&gt;
&lt;li&gt;Heavily JavaScript-rendered sites may need the crawl to start from a content-rich entry page&lt;/li&gt;
&lt;li&gt;Keep it under ~100 links; it is a menu for a model, not a second sitemap&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is also a full free URL-to-Markdown converter at /convert if you want to see exactly what an AI sees when it reads your pages.&lt;/p&gt;

&lt;p&gt;Free tier (50 requests/month, no card): &lt;a href="https://rag-scrape-api.owerryking.workers.dev" rel="noopener noreferrer"&gt;https://rag-scrape-api.owerryking.workers.dev&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>seo</category>
      <category>llm</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>An AI agent paid me $9 with no human involved - here's the full architecture</title>
      <dc:creator>Owen King</dc:creator>
      <pubDate>Sat, 26 Sep 2026 08:34:09 +0000</pubDate>
      <link>https://dev.to/owen_king_e3d7d83a33c46c8/an-ai-agent-paid-me-9-with-no-human-involved-heres-the-full-architecture-5ejc</link>
      <guid>https://dev.to/owen_king_e3d7d83a33c46c8/an-ai-agent-paid-me-9-with-no-human-involved-heres-the-full-architecture-5ejc</guid>
      <description>&lt;p&gt;Last week my API got its first payment from a customer who has never filled in a form, cannot click a checkout button, and does not own a credit card. It was an AI agent. It sent 9 USDC on Base, claimed its upgrade through an API endpoint, and started working. No human touched anything.&lt;/p&gt;

&lt;p&gt;I run RagScrape — a URL-to-Markdown API for RAG pipelines (free tier: 50 URLs/month, no signup). Like every SaaS, my upgrade path assumed a human: quota exceeded, please visit the pricing page. For agents that is a dead end. They cannot pay with cards, and most API owners respond by hard-blocking them. I went the other way and made agents a first-class customer.&lt;/p&gt;

&lt;p&gt;The design has four parts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. A quota wall that sells&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When an agent exhausts its free quota, it does not get an HTML pricing page. It gets a machine-readable 402 response with an upgrade object:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"upgrade"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"checkout"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"POST /create-checkout (a human completes payment)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"crypto"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Send 9+ USDC on Base to 0x... then POST /crypto/claim {txHash} - instant, no human"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agents read llms.txt and OpenAPI documents the way humans read homepages, so the same instructions live in both. Your 402 response is your storefront for machines.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Crypto where agents actually live&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Humans pay with cards. Agents pay with wallets. USDC on Base means roughly one-cent fees, dollar-stable value, and settlement in seconds. There is no payment processor to onboard with, no subscription engine to build - the payment itself is the invoice, and renewal is just another payment plus claim.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Trust nothing, verify on-chain&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The agent claims its payment by posting a transaction hash. My server then pulls the transaction receipt from Base's public RPC endpoint and checks four things: the USDC contract emitted a Transfer event, the recipient is my address, the amount covers the price, and the transaction succeeded. There is no webhook to secure because the blockchain is the webhook.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Idempotency and expiry&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Every transaction hash can be claimed exactly once - a registry in Workers KV makes double-claiming impossible. Credits last 31 days, and a daily cron downgrades lapsed keys back to the free tier. Nobody accumulates ghost subscriptions.&lt;/p&gt;

&lt;p&gt;The part that surprised me: the whole thing was about 200 lines of code. The hard part was not payments. It was writing documentation a machine can act on - llms.txt, an OpenAPI spec with real examples, and error messages that say what to do next instead of "unauthorized".&lt;/p&gt;

&lt;p&gt;If you are building APIs in the agent era, assume your next customer is software. Give it a wall it can read, a rail it can use, and a receipt it can prove.&lt;/p&gt;

&lt;p&gt;Free tier (50 URLs/mo, no card): &lt;a href="https://rag-scrape-api.owerryking.workers.dev" rel="noopener noreferrer"&gt;https://rag-scrape-api.owerryking.workers.dev&lt;/a&gt; - and yes, agents can pay it in USDC.&lt;/p&gt;

</description>
      <category>showdev</category>
      <category>ai</category>
      <category>crypto</category>
      <category>api</category>
    </item>
    <item>
      <title>Stop Splitting by 500 Tokens: Why Heading-Aligned Chunking Wins RAG</title>
      <dc:creator>Owen King</dc:creator>
      <pubDate>Tue, 22 Sep 2026 10:15:24 +0000</pubDate>
      <link>https://dev.to/owen_king_e3d7d83a33c46c8/stop-splitting-by-500-tokens-why-heading-aligned-chunking-wins-rag-1cc0</link>
      <guid>https://dev.to/owen_king_e3d7d83a33c46c8/stop-splitting-by-500-tokens-why-heading-aligned-chunking-wins-rag-1cc0</guid>
      <description>&lt;p&gt;Most RAG pipelines fail before retrieval even starts—because arbitrary token splitters blindly slice sentences, code blocks, and context in half.&lt;/p&gt;

&lt;p&gt;When you split text strictly every 500 or 1,000 tokens, boundaries land anywhere: midway through an explanation, inside an HTML table, or cleanly separating an &lt;code&gt;H3&lt;/code&gt; subhead from the critical steps listed below it. Your vector search ends up indexing orphan fragments. The result? Your LLM hallucinates, not because it lacks reasoning, but because its retrieved context was severed from its source meaning.&lt;/p&gt;

&lt;p&gt;Humans structure knowledge hierarchically. Headings define logical scopes, prerequisites, and intent. &lt;/p&gt;

&lt;p&gt;Heading-aligned chunking honors this structure. By isolating content strictly along logical sections (&lt;code&gt;#&lt;/code&gt;, &lt;code&gt;##&lt;/code&gt;, &lt;code&gt;###&lt;/code&gt;), every chunk represents a complete, cohesive thought. You eliminate boundary bleed, retain table integrity, and give your embedding models dense, unambiguous semantic targets to index.&lt;/p&gt;

&lt;p&gt;RagScrape makes this painless. Instead of assembling brittle web scrapers, HTML cleaners, and custom chunking scripts, you can turn any public URL into heading-aligned, token-slimmed Markdown in a single API call:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST https://rag-scrape-api.owerryking.workers.dev/scrape &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{
    "url": "https://docs.stripe.com/api",
    "chunk": true,
    "stripLinks": true,
    "embed": true
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With &lt;code&gt;chunk: true&lt;/code&gt;, you get cleanly partitioned sections mapped to the author's original structure. Add &lt;code&gt;embed: true&lt;/code&gt;, and RagScrape returns native 384-dimensional embeddings for each chunk alongside token-optimized text—omitting noisy links and images with zero extra infrastructure on your end.&lt;/p&gt;

&lt;p&gt;Better chunks mean better embeddings, higher-precision retrieval, and fewer wasted tokens in your prompt window.&lt;/p&gt;

&lt;p&gt;Try it free (no signup): &lt;a href="https://rag-scrape-api.owerryking.workers.dev/convert" rel="noopener noreferrer"&gt;https://rag-scrape-api.owerryking.workers.dev/convert&lt;/a&gt; — free API key: 50 requests/month. Full docs: &lt;a href="https://rag-scrape-api.owerryking.workers.dev/llms-full.txt" rel="noopener noreferrer"&gt;https://rag-scrape-api.owerryking.workers.dev/llms-full.txt&lt;/a&gt;&lt;/p&gt;

</description>
      <category>rag</category>
      <category>ai</category>
      <category>llm</category>
      <category>api</category>
    </item>
  </channel>
</rss>
