Drive ChatGPT's web app with a browser automation library such as Playwright, or call a scraping API that runs that browser for you. Both give you what ChatGPT shows a visitor, and that is a different result from what the OpenAI API returns for the same prompt.
- A short Playwright script works, with one condition. On 16 November 2026 a headed Chromium got ChatGPT's logged-out page and a full answer with its links. A headless Chromium got Cloudflare's "Just a moment..." page with HTTP 403.
-
The page has few stable hooks. The answer's elements carry generated class names such as
x78zum5. The fixed hooks we found were a form field name and twoaria-labelvalues. - The API and the web app cite different pages. A 2026 preprint found that the Responses API and Google Search together recover only about a quarter of the URLs the web app cites.
The steps for the browser route:
- Install Playwright and its Chromium:
pip install playwright, thenplaywright install chromium. - Open
https://chatgpt.com/in a headed browser. - Type the prompt into
textarea[name="prompt"]and click the button labelled "Send message". - Wait for
section[aria-label="DIL content"], then wait until its text stops changing. - Read the answer text and its links, and store them with the time and the country of your exit IP.
How do I get ChatGPT responses programmatically without using the OpenAI API?
There are two routes, and both read the same page, ChatGPT's web interface. With your own browser automation you run the browsers, proxies and parser. With a scraping API the vendor runs them and you send one HTTP request.
This is the browser route. We ran it on 16 November 2026 against the logged-out page:
import time
from playwright.sync_api import sync_playwright
PROMPT = "What are the best project management tools for remote teams?"
with sync_playwright() as p:
browser = p.chromium.launch(headless=False) # headless got a 403
page = browser.new_page()
page.goto("https://chatgpt.com/")
page.locator('textarea[name="prompt"]').fill(PROMPT)
page.locator('button[aria-label="Send message"]').click()
answer = page.locator('section[aria-label="DIL content"]').last
answer.wait_for(timeout=120_000)
# The answer streams in. Wait until the text stops changing.
text, stable = "", 0
while stable < 3:
time.sleep(2)
current = answer.inner_text()
stable = stable + 1 if current == text else 0
text = current
links = answer.locator("a[href^='http']").evaluate_all(
"els => els.map(e => e.href)"
)
print(text[:500])
print(links)
browser.close()
It printed the answer (Asana, ClickUp, monday.com, Notion, Jira and Trello) and six links, each tagged utm_source=chatgpt.com. Three things about this page matter for the code:
- The logged-out page served a compact layout: a
textareanamedpromptand a form that posts to/unauth-mweb/conversation. A signed-in session gets a different layout with different selectors. - The answer has no
data-attribute that names it. Thearia-label"DIL content" was the only label on its section, and the elements inside it use generated class names. - The links are the inline links of the answer. The script does not read the sources panel or the model name, which need more parsing.
What does the browser route take at scale?
One answer on one laptop is the easy part. A schedule of prompts in several countries adds these parts, and you keep each one running:
| Part | What it takes |
|---|---|
| Browsers | One Chromium per concurrent prompt. Headed mode needs a display, or a virtual one such as Xvfb on a server |
| Proxies | An exit IP in each country you sample, because the answer and its links follow the location |
| The bot check | Cloudflare sits in front of chatgpt.com. In our test it stopped headless Chromium with a 403 and let a headed one through. Cloudflare documents its bot detection engines, and the result can change with any update |
| Selectors | The three hooks in the script are labels and a field name. A layout change removes them without an error, and the script then waits until it times out |
| Monitoring | Alert on the share of empty or timed-out answers, not only on exceptions |
The public record of the old wrappers shows where the work goes. pyChatGPT moved from plain HTTP to undetected_chromedriver to get past Cloudflare, and its repository is now archived.
If you need a few hundred answers a month from one country, the browser route can be the cheaper one. Count the hours you spend on fixes before you compare it with a per-answer price.
Which scraping APIs run the browser for you?
Several vendors sell ChatGPT's web answers as an API. Each request below is from the vendor's own page, read on 16 November 2026:
| Vendor | Request | What comes back |
|---|---|---|
| Bright Data ChatGPT Scraper API | A list of {"url": "https://chatgpt.com/", "prompt": ..., "country": ...}
|
Answer text, answer HTML, citations, attached links, recommendations, country |
| Oxylabs ChatGPT scraper | {"source": "chatgpt", "prompt": ..., "parse": true} |
markdown_text and citations
|
| cloro |
POST /v1/monitor/chatgpt with prompt and country
|
text, markdown, sources, citationPills and model
|
All three route each request through the country you set and hand back JSON. They differ in the fields, in whether they return the model ChatGPT chose, and in how they bill: per record, per result or per credit. Compare them on a sample of your own prompts.
Why not use the OpenAI API with web search?
The OpenAI API is the right tool when you want a model's output. It is a weak stand-in for what ChatGPT tells its users, and two independent studies measured the gap.
The preprint by Jiang and others (arXiv, 11 October 2026) recorded citations from signed-in ChatGPT web sessions across model versions 5.2 to 5.5, then ran the same questions through the Responses API with web search and through Google. Combined, the two recovered only about one quarter of the URLs the web app cited.
Researchers in Hamburg compared the web interface and the API on more than 24,000 German news answers over five weeks, as The Decoder reported. The API leaned on encyclopedias, with Wikipedia at almost 15% of its citations. welt.de and bild.de made up about 13% of web references and 2% of API citations.
Which route should you pick?
| Your case | Route |
|---|---|
| You want the model's answer in your own product | The OpenAI API |
| A few hundred answers a month, one country, and you already run browsers | Your own Playwright script |
| Several countries, a schedule, or the sources panel and model name parsed | A scraping API |
What are the limits?
Both routes see what a logged-out visitor sees. Anything that needs a signed-in session is out of reach. If ChatGPT asks for a login in one country, requests from that country fail until it stops. ChatGPT picks the model for each request, so the same prompt can get a different model on the next run.
The browser test ran once, from one machine in one country, on 16 November 2026. The study figures are from the linked pages; the preprint is not peer-reviewed, and it used signed-in sessions. This post is published by cloro, one of the three APIs in the table above.
Top comments (0)