ChatGPT can now call your own tools through custom MCP connectors — including web scraping. But there is a catch the marketing pages skip: connectors must be remote servers, so a local tool like CrawlForge cannot be pasted in directly. This is the honest version: what is actually possible, why a wrapper is needed, and the exact bridge to build.
Table of Contents
- What ChatGPT Connectors Are
- Which Plans Can Use Them
- The Transport Catch: Remote Only
- Why CrawlForge Needs a Wrapper
- Build the Bridge
- Add the Connector in ChatGPT
- Auth and Safety
- A Simpler Alternative
What ChatGPT Connectors Are
ChatGPT supports custom MCP connectors — renamed "apps" in December 2025, so the UI now says "Apps & Connectors." Through Developer mode, you connect an external MCP server and ChatGPT calls its tools mid-conversation, asking you to confirm before any write action. Same Model Context Protocol that powers web scraping in Claude — different client. Developer mode is explicitly a beta.
Which Plans Can Use Them
Per OpenAI's plan table, adding a custom MCP connector is available on Plus, Pro, Business, Enterprise, and Edu — not Free or Go. Full write-action support is rolling out most broadly to Business, Enterprise, and Edu. If you only need ChatGPT to read scraped data, the read-only path below is enough.
The Transport Catch: Remote Only
This trips people up. A ChatGPT connector must be a remote MCP server reachable over HTTPS (SSE or Streamable HTTP transport). You paste a URL; you do not point it at a command on your machine. That rules out local stdio servers — the kind you install with npx. To use one, host it publicly or tunnel a local server via ngrok or Cloudflare Tunnel.
There is also a naming rule: ChatGPT's deep research / company-knowledge paths require two read-only tools named search and fetch with a specific schema. Full Developer mode allows arbitrary tools, so that constraint applies only to the deep-research path.
Why CrawlForge Needs a Wrapper
CrawlForge ships as a local stdio MCP server (via npx) plus a REST API at https://www.crawlforge.dev/api/v1/tools/. Neither is a remote MCP URL, and its tools are named search_web, fetch_url, and extract_content — not the search/fetch pair deep research expects. So you cannot paste CrawlForge straight into ChatGPT today. The practical path is a thin remote MCP wrapper — about 30 lines.
Build the Bridge
FastMCP (Python) is the quickest way to stand up a remote MCP server exposing the search + fetch tools ChatGPT wants. Each calls CrawlForge's REST API with your cf_live_ key in the X-API-Key header:
server.py — the full bridge
import os
import httpx
from fastmcp import FastMCP
mcp = FastMCP("CrawlForge Bridge")
BASE = "https://www.crawlforge.dev/api/v1/tools"
HEADERS = {"X-API-Key": os.environ["CRAWLFORGE_API_KEY"]}
@mcp.tool()
async def search(query: str) -> list[dict]:
"""Search the web. Returns id/title/url results for ChatGPT."""
async with httpx.AsyncClient(timeout=30) as client:
r = await client.post(f"{BASE}/search_web", headers=HEADERS,
json={"query": query, "limit": 10})
results = r.json().get("results", [])
return [{"id": x["link"], "title": x["title"], "url": x["link"]} for x in results]
@mcp.tool()
async def fetch(id: str) -> dict:
"""Fetch full page content by id (the URL) for ChatGPT."""
async with httpx.AsyncClient(timeout=30) as client:
r = await client.post(f"{BASE}/extract_content", headers=HEADERS,
json={"url": id})
data = r.json()
return {"id": id, "title": data.get("title", id),
"text": data.get("content", ""), "url": id}
if __name__ == "__main__":
mcp.run(transport="http", host="0.0.0.0", port=8000)
Run it and expose it over HTTPS. For a quick test, tunnel your local port:
pip install fastmcp httpx
export CRAWLFORGE_API_KEY="cf_live_your_key_here"
python server.py
# in another terminal:
ngrok http 8000
For Developer mode you can skip the search/fetch naming and map tools one-to-one to CrawlForge — expose scrape_structured, stealth_mode, or deep_research directly. Same pattern.
Add the Connector in ChatGPT
- Settings → Apps & Connectors → Advanced → enable Developer mode.
- Apps & Connectors → Create.
- Paste your public HTTPS MCP URL (e.g. your ngrok URL plus
/mcp), name it, choose an auth method. - Confirm the "I trust this application" checkbox.
Your search and fetch tools appear. In a chat, select the connector and ask ChatGPT to research a topic — it calls search, then fetches the best results through CrawlForge.
Auth and Safety
Connectors authenticate with none (public) or OAuth — there is no API-key-header option in the UI, which is why the wrapper holds your CrawlForge key server-side. ChatGPT confirms before write actions, and you can inspect each call before approving.
Take OpenAI's warnings seriously: only connect servers you trust. Custom connectors increase risk, including prompt injection, and a model mistake on a write action could destroy or leak data. A read-only scraping bridge is low-risk; lock it down with OAuth before sharing.
A Simpler Alternative
If you would rather not host anything, use CrawlForge from code with the OpenAI Agents SDK or Responses API — no remote server required. See CrawlForge with the OpenAI Agents SDK.
Top comments (0)