Give Your AI Chatbot Real-Time Web Data with GcrawlAI

gcrawl ai — Fri, 09 Oct 2026 10:02:36 +0000

Language models have a training cutoff, so they guess when asked about prices, news or product changes. Give your chatbot a web tool and it can look things up, read the page and answer with a source. This guide adds that tool in a few lines.

Expose GcrawlAI as a tool

Define one function the model can call. It searches, reads the top results and returns clean text.

`python
import os, requests

API = "https://api.gcrawlai.com/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['GCRAWLAI_API_KEY']}"}

def gcrawl(endpoint, payload):
r = requests.post(f"{API}/{endpoint}", json=payload, headers=HEADERS, timeout=60)
r.raise_for_status()
return r.json()

def web_lookup(query, k=3):
hits = gcrawl("search", {"query": query, "limit": k})["results"]
out = []
for h in hits:
md = gcrawl("scrape", {"url": h["url"], "formats": ["markdown"]})["markdown"]
out.append({"url": h["url"], "text": md[:3000]})return out

Decide when to search

Not every message needs the web. Let the model call the tool only for fresh or factual questions.

Search for prices, availability, news and recent releases
Skip search for greetings, coding help and general knowledge
Scrape a specific URL directly when the user pastes a link

Ground the answer and cite

Pass the returned text back to the model and instruct it to answer only from that text. Show the URLs to the user so they can verify.

python def chat(user_msg): if needs_web(user_msg): # simple classifier or model tool-call ctx = web_lookup(user_msg) return ask_llm(f"Answer from this context only and cite URLs:\n{ctx}\n\nQ: {user_msg}") return ask_llm(user_msg)

Note: Cache results for a few minutes. Repeated questions then cost nothing and respond faster.

Combine with your own knowledge

Use your indexed docs from the onboarding assistant for product questions and live web search for everything else. For long multi-step questions, hand over to a deep research agent.

DEV Community: gcrawl ai

Give Your AI Chatbot Real-Time Web Data with GcrawlAI