Give your LLM live web search: query top results clean page Markdown (no search API key)

#python #ai #llm #rag

LLMs are stuck at their training cutoff. The usual fix is to give them a web search step: run a query, take the top results, read the pages, and put the relevant text in the prompt. Most tutorials stop at "get a search API key", which means a paid account with Google, Bing or Brave and their rate limits.

Here's a way to do the whole loop (search → fetch pages → clean Markdown) in one call, with no search API key.

What one run returns

Query: best open source vector database 2026, top 5 results, with page content. This is the real output (Markdown trimmed):

[
  {
    "query": "best open source vector database 2026",
    "engine": "duckduckgo",
    "position": 1,
    "title": "9 Best Open Source Vector Databases in 2026 (Tested) - Geekflare",
    "url": "https://geekflare.com/open-source/vector-database/",
    "domain": "geekflare.com",
    "contentSuccess": true,
    "tokensApprox": 200,
    "markdown": "| Qdrant | Yes — sparse and dense vectors fused in one query | ..."
  },
  {
    "position": 5,
    "title": "7 Best Open-Source Vector Databases in 2026",
    "domain": "krunalkanojiya.com",
    "markdown": "Qdrant is the best balanced open-source vector database for most teams starting a dedicated retrieval service in 2026. It combines ..."
  }
]

Every result has its rank, title, URL and domain. With fetchPageContent on, it also has the page body as Markdown with boilerplate stripped, plus a token estimate. That's exactly what you want to paste into a prompt with citations.

The grounding loop in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

def web_search(query, n=5, chars=3000):
    run = client.actor("get_anything/ai-web-search-scraper").call(run_input={
        "queries": [query],
        "engine": "duckduckgo",
        "maxResults": n,
        "fetchPageContent": True,
        "maxCharsPerPage": chars,      # keep the prompt within budget
    })
    return [r for r in client.dataset(run["defaultDatasetId"]).iterate_items() if r.get("contentSuccess")]

def build_context(results):
    return "\n\n".join(
        f"[{r['position']}] {r['title']} ({r['url']})\n{r['markdown']}" for r in results
    )

results = web_search("best open source vector database 2026")
context = build_context(results)

prompt = f"""Answer using only the sources below and cite them like [1].

{context}

Question: Which open-source vector database should a small team start with in 2026, and why?"""
# send `prompt` to the model you use (Claude, GPT, a local Llama...)

A few tips from using it:

Budget with maxCharsPerPage. Five pages × 3,000 characters is about 4k tokens, which fits comfortably in any modern context window.
Keep the citation index. Numbering sources by position lets the model cite [1]–[5], and you can show the links in your UI.
Several queries per run. queries takes a list, so you can fan out a question into 3–4 phrasings in one run and de-duplicate by URL.
DuckDuckGo is the default engine and works without extra setup. Google is available too, behind a residential proxy.

Or skip the code: let the assistant search for itself (MCP)

If you use Claude, ChatGPT, Cursor or another MCP client, add the scraper as a tool through the Apify MCP server:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=get_anything/ai-web-search-scraper" } } }

Now you can ask "Search the web for this week's news on X and summarise the top 5 sources with links", and the assistant runs the search and reads the pages itself. Setup guide: Connect any Apify scraper to Claude or ChatGPT in 2 minutes.

When you already have the URLs

If you don't need the search step (say you have a list of docs pages or articles), use the sister tool that only converts URLs to clean Markdown and can crawl a whole site: LLM Web Scraper.

Cost

Pay per result, no subscription. See the Pricing tab on the Actor page. The Apify free plan's monthly credit is enough to try it properly.

Try it: AI Web Search Scraper on Apify

What are you using for grounding today, and where does it break? I'm curious which sites give people the most trouble.

DEV Community