<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Rabii · Pandapi</title>
    <description>The latest articles on DEV Community by Rabii · Pandapi (@rabii_slimane_22e39df3bdd).</description>
    <link>https://dev.to/rabii_slimane_22e39df3bdd</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4241898%2F34981e22-2962-44e1-a56a-cb7339ffa669.png</url>
      <title>DEV Community: Rabii · Pandapi</title>
      <link>https://dev.to/rabii_slimane_22e39df3bdd</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/rabii_slimane_22e39df3bdd"/>
    <language>en</language>
    <item>
      <title>One Python script for government tenders from SAM.gov, TED, the UK, Canada and the World Bank</title>
      <dc:creator>Rabii · Pandapi</dc:creator>
      <pubDate>Mon, 16 Nov 2026 01:41:27 +0000</pubDate>
      <link>https://dev.to/rabii_slimane_22e39df3bdd/one-python-script-for-government-tenders-from-samgov-ted-the-uk-canada-and-the-world-bank-1n0b</link>
      <guid>https://dev.to/rabii_slimane_22e39df3bdd/one-python-script-for-government-tenders-from-samgov-ted-the-uk-canada-and-the-world-bank-1n0b</guid>
      <description>&lt;p&gt;If you sell to governments, the work starts with finding the tender. Then you find out every country publishes them somewhere different, in a different format, with a different search that matches different things.&lt;/p&gt;

&lt;p&gt;I wanted one list: every open cybersecurity tender from the last two weeks, from the US, the EU, the UK, Canada and World Bank projects, with a deadline at least a week away. Here's the free way, where it hurts, and what I ended up using.&lt;/p&gt;

&lt;h2&gt;
  
  
  The free way: call each portal yourself
&lt;/h2&gt;

&lt;p&gt;All of these portals have official open-data APIs. Some are pleasant to use and some are not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TED (EU)&lt;/strong&gt; is the nicest one. You don't need a key, and the expert query language is powerful:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"https://api.ted.europa.eu/v3/notices/search"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"query"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'(notice-title ~ "cybersecurity" OR description-proc ~ "cybersecurity") AND publication-date&amp;gt;=20261102'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"fields"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"notice-title"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"buyer-name"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"deadline-receipt-tender-date-lot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"buyer-country"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"limit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"notices"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"notice-title"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"eng"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"buyer-name"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What comes back is multilingual on purpose. &lt;code&gt;buyer-name&lt;/code&gt; is &lt;code&gt;{"fra": ["Infrabel sa"], "nld": ["Infrabel nv"]}&lt;/code&gt;, deadlines are per lot, and values sit in different fields depending on the notice form. That's fine, it's just another parser to write.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SAM.gov (US)&lt;/strong&gt; needs an API key from your SAM.gov account. A personal key for a non-federal user with no role is limited to &lt;a href="https://open.gsa.gov/api/get-opportunities-public-api/" rel="noopener noreferrer"&gt;10 requests a day&lt;/a&gt;. Its keyword search also matches attachments and the standard contract clauses that almost every defense notice carries. In my test window, &lt;strong&gt;164 of 234&lt;/strong&gt; SAM.gov notices matched "cybersecurity" only because of boilerplate like "applicable cybersecurity standards" deep inside the clause list, and they weren't cybersecurity work at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The UK&lt;/strong&gt; has two systems: Find a Tender (above threshold, OCDS JSON) and Contracts Finder (below threshold, a different API). Find a Tender rate-limits hard. Ask for too much and you get HTTP 429 with "try again in 2 minutes".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;CanadaBuys&lt;/strong&gt; publishes a daily open-data file rather than a search API. The &lt;strong&gt;World Bank&lt;/strong&gt; has its own procurement notices API.&lt;/p&gt;

&lt;p&gt;That makes five APIs, five schemas, several date formats and four kinds of industry code (CPV, NAICS, UNSPSC, GSIN), and you still have to dedupe, convert currencies and filter on deadlines. If you only care about one country, use its API directly. It's free and you'll learn the data.&lt;/p&gt;

&lt;h2&gt;
  
  
  The lazy way: one feed
&lt;/h2&gt;

&lt;p&gt;I make a feed on Apify that does that plumbing. Disclosure: I make this at Pandapi.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# pip install "apify-client&amp;gt;=3"&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"blueberry_rainstorm/public-tenders-feed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"keywords"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"cybersecurity"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"penetration testing"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"postedWithinDays"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"minDaysLeft"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;default_dataset_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'{t["countryName"]:&amp;lt;15} {str(t["daysLeft"] or "-"):&amp;gt;3}d  {t["valueEur"] or "":&amp;gt;10}  {t["title"][:70]}'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It returned &lt;strong&gt;22 notices from 4 of the 6 portals&lt;/strong&gt; (TED 11, World Bank 6, SAM.gov 3, CanadaBuys 2; the UK had none that matched). Every notice has the same fields: title, buyer, country, deadline, days left, value in the original currency plus USD and EUR, classification codes and the link to the official notice. A few I'd actually look at:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Buyer&lt;/th&gt;
&lt;th&gt;Country&lt;/th&gt;
&lt;th&gt;What&lt;/th&gt;
&lt;th&gt;Deadline&lt;/th&gt;
&lt;th&gt;Value (EUR)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;ENISA (EU Agency for Cybersecurity)&lt;/td&gt;
&lt;td&gt;Greece&lt;/td&gt;
&lt;td&gt;Security assessment services&lt;/td&gt;
&lt;td&gt;14 Dec&lt;/td&gt;
&lt;td&gt;3,000,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HIQA&lt;/td&gt;
&lt;td&gt;Ireland&lt;/td&gt;
&lt;td&gt;Continuous pen test tool suite and services&lt;/td&gt;
&lt;td&gt;18 Dec&lt;/td&gt;
&lt;td&gt;550,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Aeroporti di Puglia&lt;/td&gt;
&lt;td&gt;Italy&lt;/td&gt;
&lt;td&gt;Cybersecurity monitoring services&lt;/td&gt;
&lt;td&gt;21 Dec&lt;/td&gt;
&lt;td&gt;600,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;EU Agency for the Space Programme&lt;/td&gt;
&lt;td&gt;Spain&lt;/td&gt;
&lt;td&gt;PKI L2/L3 maintenance&lt;/td&gt;
&lt;td&gt;16 Dec&lt;/td&gt;
&lt;td&gt;1,100,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Danmarks Nationalbank&lt;/td&gt;
&lt;td&gt;Denmark&lt;/td&gt;
&lt;td&gt;Framework for cybersecurity advisory and assessment&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;td&gt;1,471,454&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ministry of Telecom (World Bank project)&lt;/td&gt;
&lt;td&gt;Palestinian Territories&lt;/td&gt;
&lt;td&gt;Cybersecurity maturity assessment&lt;/td&gt;
&lt;td&gt;12 Dec&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two things I care about in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;SAM.gov boilerplate is filtered.&lt;/strong&gt; By default a SAM.gov notice is kept only if the keyword is in the title, the buyer, or the opening of the real description with the clause lists removed. That's how 234 notices became 3. If you really want deep mentions too, set &lt;code&gt;strictKeywordMatch&lt;/code&gt; to &lt;code&gt;false&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;matchedInTitle&lt;/code&gt;&lt;/strong&gt; marks the strongest hits. Sort on it to get a short list for the bid/no-bid meeting.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What still needs a human
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Some notices mention your keyword only as one requirement in a bigger project. A Romanian substation rebuild and a Canadian drone procurement both list cybersecurity in their scope. They're correct matches, but they aren't cybersecurity tenders. Read the titles.&lt;/li&gt;
&lt;li&gt;English keywords find many non-English EU notices because TED translates titles, but not all of them. A Croatian SOC tender written only in Croatian didn't match. For the EU, CPV codes are more precise: &lt;code&gt;"cpvCodes": ["72220000", "79417000"]&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Some TED notices publish no deadline or value at the first stage. The feed keeps them so you don't miss them, and shows the fields as empty.&lt;/li&gt;
&lt;li&gt;The UK's Find a Tender rate limit makes runs that include it take a few minutes instead of seconds.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Cost
&lt;/h2&gt;

&lt;p&gt;$2 per 1,000 notices, no start fee, and nothing is charged for notices that are filtered out. This run returned 22 notices, so it cost &lt;strong&gt;$0.044&lt;/strong&gt;. With &lt;strong&gt;Only new notices&lt;/strong&gt; on and a daily schedule, a 30-notices-a-day alert costs about $0.06 a day, and a free Apify account's $5 monthly credit covers that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which one should you use?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;One country, or you want to learn the data: call that portal's API directly. TED's is a pleasure to use.&lt;/li&gt;
&lt;li&gt;Several countries, a daily alert into Slack, email or a sheet, or a feed for an AI bid-triage agent: a combined feed saves the parsers. Mine is here: &lt;a href="https://apify.com/blueberry_rainstorm/public-tenders-feed" rel="noopener noreferrer"&gt;Government Tenders &amp;amp; Public Procurement API&lt;/a&gt;. There are others on Apify Store, so compare them on your own keywords.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;More runnable examples are in &lt;a href="https://github.com/riaa3102/pandapi-examples" rel="noopener noreferrer"&gt;pandapi-examples&lt;/a&gt; on GitHub.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Disclosure: I make this at Pandapi. 🐼&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>opendata</category>
      <category>dataengineering</category>
    </item>
    <item>
      <title>Turn a docs site into RAG-ready Markdown chunks in 5 minutes (no scraper code)</title>
      <dc:creator>Rabii · Pandapi</dc:creator>
      <pubDate>Thu, 12 Nov 2026 23:06:57 +0000</pubDate>
      <link>https://dev.to/rabii_slimane_22e39df3bdd/turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code-59j</link>
      <guid>https://dev.to/rabii_slimane_22e39df3bdd/turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code-59j</guid>
      <description>&lt;p&gt;You want a chatbot that answers questions about a library's documentation. The hard part is rarely the LLM. It is getting the docs out of HTML and into clean, well-cut chunks: no navigation menus, no cookie banners, no footers repeated on every page, and chunks that keep track of which section they came from.&lt;/p&gt;

&lt;p&gt;This walkthrough takes the FastAPI documentation, converts it to Markdown, cuts it into ~400-token chunks with their heading path, and loads them into a local vector store. Total code: about 25 lines of Python.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Python 3.9+&lt;/li&gt;
&lt;li&gt;A free Apify account (the free plan's monthly credit covers this tutorial many times over)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;pip install apify-client chromadb&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We use the &lt;a href="https://apify.com/blueberry_rainstorm/web-page-to-markdown" rel="noopener noreferrer"&gt;Web Page to Markdown&lt;/a&gt; Actor for the conversion. Disclosure: I built it. It costs $1 per 1,000 pages converted, and failed pages are free. Everything below also works with any tool that gives you clean Markdown plus heading-aware chunks; this one just does both in one call.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Point it at the sitemap
&lt;/h2&gt;

&lt;p&gt;Most docs sites publish a &lt;code&gt;sitemap.xml&lt;/code&gt;. Giving the sitemap instead of a list of URLs means new pages are picked up automatically the next time you run it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"blueberry_rainstorm/web-page-to-markdown"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                       &lt;span class="c1"&gt;# only the sitemap this time&lt;/span&gt;
    &lt;span class="s2"&gt;"sitemapUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://fastapi.tiangolo.com/sitemap.xml"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxPages"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                   &lt;span class="c1"&gt;# keep the first test small&lt;/span&gt;
    &lt;span class="s2"&gt;"contentMode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"main"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;            &lt;span class="c1"&gt;# drop menus, sidebars and footers&lt;/span&gt;
    &lt;span class="s2"&gt;"chunkSize"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                 &lt;span class="c1"&gt;# ~tokens per chunk&lt;/span&gt;
    &lt;span class="s2"&gt;"chunkOverlap"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;pages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"pages"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On a test run with &lt;code&gt;maxPages: 5&lt;/code&gt;, the five pages came back in 2.4 seconds without a browser, because the FastAPI docs are static HTML. JavaScript-only pages are detected and re-rendered in Chrome automatically.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Look at what came back
&lt;/h2&gt;

&lt;p&gt;Each page is one item with metadata, the full Markdown, and the chunks:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://fastapi.tiangolo.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"FastAPI - FastAPI"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"FastAPI framework, high performance, easy to learn, fast to code, ready for production"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"language"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"en"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"canonicalUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://fastapi.tiangolo.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"wordCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2134&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"approxTokens"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4139&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"renderedWithBrowser"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"# FastAPI - FastAPI\n\n*FastAPI framework, high performance, easy to learn..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"chunks"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"index"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"headingPath"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"FastAPI - FastAPI"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"approxTokens"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two fields matter most for RAG:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;headingPath&lt;/code&gt;&lt;/strong&gt; (for example &lt;code&gt;Concurrency and async / await - FastAPI &amp;gt; In a hurry?&lt;/code&gt;). Store it as metadata, and prepend it to the chunk text before embedding. A chunk that starts "Use &lt;code&gt;async def&lt;/code&gt; when..." is much easier to retrieve when the embedding also knows it sits under "Concurrency and async / await".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;approxTokens&lt;/code&gt;&lt;/strong&gt; lets you check the budget before you embed: on our 5-page test the pages ranged from 103 to 5,428 tokens and became 89 chunks in total.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Load the chunks into a vector store
&lt;/h2&gt;

&lt;p&gt;Chroma runs locally and embeds with a small default model, so there is nothing else to sign up for.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;chromadb&lt;/span&gt;

&lt;span class="n"&gt;db&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;chromadb&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;PersistentClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"./fastapi-index"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;col&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_or_create_collection&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"fastapi-docs"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;                       &lt;span class="c1"&gt;# failed pages are listed, not charged&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"chunks"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'{page["url"]}#{chunk["index"]}'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'{chunk["headingPath"]}\n\n{chunk["text"]}'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;metas&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                      &lt;span class="s2"&gt;"section"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"headingPath"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;upsert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;documents&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metadatas&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;metas&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"chunks indexed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Using the page URL plus the chunk index as the id makes re-runs safe: &lt;code&gt;upsert&lt;/code&gt; replaces chunks instead of duplicating them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: Ask it something
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;hits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query_texts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"When should I use async def instead of def?"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n_results&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;hits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"metadatas"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"section"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pass the top chunks, with their URLs, to the LLM of your choice and ask it to cite the URL. Because each chunk carries its source page and section, the answer can link back to the exact part of the docs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tips from running this on real sites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Main content vs full page.&lt;/strong&gt; &lt;code&gt;"contentMode": "main"&lt;/code&gt; is right for docs, blogs and articles. For homepages, product pages and listings, use &lt;code&gt;"full"&lt;/code&gt; so nothing important is cut.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Strip site-specific clutter&lt;/strong&gt; with &lt;code&gt;removeSelectors&lt;/code&gt;, for example &lt;code&gt;.related-posts, #comments, .edit-this-page&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep it fresh.&lt;/strong&gt; Schedule the run weekly in Apify and re-run the upsert. Only pages that exist get re-embedded, and the ids stay stable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Respect the site.&lt;/strong&gt; Use it on documentation you are allowed to reuse (your own, or open-source docs under a permissive license), and keep &lt;code&gt;maxConcurrency&lt;/code&gt; modest on small sites.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Cost of this tutorial
&lt;/h2&gt;

&lt;p&gt;50 pages at $1 per 1,000 pages is $0.05. The Chroma part runs on your machine for free.&lt;/p&gt;




&lt;p&gt;If you try this on another docs site and something comes out messy (tables, code blocks, odd navigation), tell me which site in the Actor's Issues tab and I will look at it.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>llm</category>
      <category>python</category>
      <category>webscraping</category>
    </item>
  </channel>
</rss>
