DEV Community

Cover image for How to get the UI answers of ChatGPT through an API (if you want what users see, not the OpenAI API output)
Ricardo Batista
Ricardo Batista

Posted on Originally published at cloro.dev AI-assisted

How to get the UI answers of ChatGPT through an API (if you want what users see, not the OpenAI API output)

#chatgpt #api #python #webscraping

To get the UI answers of ChatGPT through an API, call a service that runs your prompt in the ChatGPT web interface and returns the rendered answer as JSON. The OpenAI API does not do this. It returns model output, without the web search, source citations, shopping cards and entity panels that the web interface adds. Three routes return the UI answer: a managed ChatGPT endpoint such as cloro, a scraping platform such as Apify or Bright Data, and a Playwright stack that you run yourself.

With a managed endpoint the job is one POST request. cloro's endpoint is POST https://api.cloro.dev/v1/monitor/chatgpt. You send a prompt and a country. The response carries text, sources, citationPills and entities, plus markdown, searchQueries and shoppingCards when you ask for them. If you need model output for your own application, the OpenAI API is the correct tool and you do not need a scraper.

Key takeaways

  • The OpenAI API and the ChatGPT web UI are different surfaces. Citations, shopping cards, brand entities and the search queries ChatGPT issued exist only in the web UI.
  • At 1,000 queries a day, the source comparison estimates $100 to $300 a month for a managed API and $980 to $2,140 a month for DIY Playwright.
  • DIY is the better choice with zero budget, a strong engineering team and 8 to 15 hours a month for maintenance.

Steps with a managed endpoint

  1. Get an API key from the provider.
  2. Send a POST request to https://api.cloro.dev/v1/monitor/chatgpt with prompt and country.
  3. Set the include flags for the optional fields that you need.
  4. Set the client timeout to 300 seconds or more.
  5. Read the parsed fields from result in the JSON response.

Why does the OpenAI API return a different answer than the ChatGPT UI?

The chat.completions API and the chatgpt.com web interface use different stacks. The API answers from a base model. The web interface does more work before the answer reaches the screen:

  • It breaks the prompt into sub-queries (query fan-out).
  • It runs a web search against OpenAI's search index.
  • It adds citations from the search results.
  • For shopping prompts, it adds a product card with merchant offers.

Two measurements show the size of the difference. Across roughly 2,500 prompts, we found that ChatGPT grounded 98.4% of its answers in live sources, at 14.1 sources each. A second measurement on 2026-08-19 gave 78.9% and 12.7 sources. And Surfer's analysis reports only about 20% overlap between API responses and the answers that users see in the ChatGPT UI. That figure is Surfer's and was not reproduced here.

Is there an API that returns what ChatGPT actually shows real users?

Yes. A ChatGPT scraper API runs the prompt in the consumer product and returns the answer that a user sees. OpenAI does not offer one. Its developer API (platform.openai.com) returns model completions, with no source pills, shopping cards or ads.

Four data types exist only in the web UI: sources and citations, shopping cards with merchant offers, brand entities, and the sub-queries that ChatGPT issued before it answered.

How do I get ChatGPT responses programmatically without using the OpenAI API?

There are four routes. The cost column is an estimate at 1,000 queries a day from cloro's comparison of 8 tools, with proxies, CAPTCHA solving and engineer time included.

Route Examples What you still build Estimated cost per month
Managed ChatGPT endpoint cloro Nothing. Fields arrive parsed. $100 to $300
Scraping platform Apify actors, Bright Data Scraping Browser Cookie refresh (Apify) or the parsing layer (Bright Data) $280 to $900
Browser infrastructure or anti-bot API Browserbase, Browserless, ScrapingBee, ZenRows Stream assembly, citation parsing, selectors $449 to $1,040
DIY framework Playwright with stealth plugins All of it $980 to $2,140

The DIY route needs five components:

  • A stealth-patched Playwright Chromium. Vanilla Playwright is detected on the first navigation.
  • A residential proxy with a sticky session. Datacenter IPs get a CAPTCHA on the first prompt.
  • A network interceptor for backend-api/f/conversation. Only that URL carries the answer stream.
  • A completion check that waits for the [DONE] line in the stream.
  • A step that clicks the sources button and reads the flyout after the answer completes. Citations are not in the stream.

ChatGPT sends the answer as Server-Sent Events. The first function a DIY scraper needs is the parser for that stream:

import json
from typing import List, Dict, Any

def extract_raw_response(input_string: str) -> List[Dict[str, Any]]:
    """Parse ChatGPT's Server-Sent Events stream."""
    json_objects = []

    # Split by lines that start with "data: "
    lines = input_string.split("\n")

    for line in lines:
        # Skip empty lines and non-data lines
        if not line.strip() or not line.startswith("data: "):
            continue

        # Remove "data: " prefix
        json_str = line[6:].strip()

        # Skip special markers like [DONE]
        if json_str == "[DONE]":
            continue

        # Try to parse as JSON
        try:
            json_obj = json.loads(json_str)

            # Only include if it's a dictionary (object), not string or other types
            if isinstance(json_obj, dict):
                json_objects.append(json_obj)
        except json.JSONDecodeError:
            # Skip invalid JSON
            continue

    return json_objects
Enter fullscreen mode Exit fullscreen mode

The complete Playwright class, the citation extractor and the shopping-card parser are in the full DIY guide.

How do you call an API for the ChatGPT UI answer?

This request comes from the public chatgpt-scraper README:

import requests

payload = {
    'prompt': 'best project management software for remote teams',
    'country': 'US',
    'include': {'markdown': True, 'searchQueries': True},
}

response = requests.post(
    'https://api.cloro.dev/v1/monitor/chatgpt',
    headers={'Authorization': 'Bearer YOUR_API_KEY'},
    json=payload,
)

print(response.json())
Enter fullscreen mode Exit fullscreen mode

The same call with cURL, this time with shopping cards:

curl -X POST https://api.cloro.dev/v1/monitor/chatgpt \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "best running shoes for flat feet", "country": "US", "include": {"shopping": true}}'
Enter fullscreen mode Exit fullscreen mode
Parameter Description Default
prompt (required) The query or question, 1 to 10,000 characters none
country (required) Country code for localized results (US, GB, DE) none
state US state code none
include.markdown Return the answer as Markdown false
include.html Return a URL to the full HTML, which expires after 24 hours false
include.rawResponse Return the unparsed upstream payload false
include.searchQueries Return the query fan-out terms false
include.shopping Return shopping cards and inline products false
include.ads Return sponsored blocks false
disableWebSearch Return the answer ChatGPT gives without a forced web search false

The response:

{
  "success": true,
  "result": {
    "text": "For remote teams, the strongest options are...",
    "citationPills": [{ "citationPillId": 0, "label": "Asana", "url": "https://asana.com", "domain": "asana.com" }],
    "searchQueries": ["best project management software 2026", "asana vs monday remote teams"],
    "markdown": "For remote teams, the strongest options are **Asana**..."
  }
}
Enter fullscreen mode Exit fullscreen mode
Field in result Content
text, markdown The answer as plain text and as Markdown
sources Each cited URL with position, url, label and description
citationPills The inline citation chips. Entries share a citationPillId when one pill cites several sources.
entities The products, brands and concepts that the answer names, each with a type
searchQueries The search terms that ChatGPT ran
shoppingCards Product carousels. Each one groups products[] with price and merchant offers.
inlineProducts Products in the answer text, separate from the carousels
ads Sponsored blocks
map Business and location entries with rating, reviews and address
rawResponse The unparsed upstream payload

Three details affect a production client:

  • Timeout. The README recommends 300 seconds or more, because failed attempts are retried on the server before the response returns. A typical response takes 30 to 45 seconds.
  • Search queries. The README states that searchQueries comes back empty on ChatGPT's default mobile-web interface. The legacy: true option asks for the interface that returns it, on a best-effort basis.
  • Batches. For thousands of prompts a day, use POST /v1/async/task with taskType: "CHATGPT" and receive the results by webhook.

The price list has two ChatGPT rows. "ChatGPT (web search)" is the base request at 5 credits. "ChatGPT (full response)" is 7 credits: the base request plus one flat charge of 2 credits for any of include.rawResponse, include.searchQueries, include.ads and include.shopping. These are async prices. A synchronous request to /v1/monitor/chatgpt adds 2 credits, so the two sample requests above cost 9 credits each. The Hobby plan is $100 a month for 250,000 credits. A panel of 1,000 prompts a day uses about 210,000 credits a month at 7 credits, or about 150,000 at 5 credits. The Free plan is 500 credits a month.

What should I use instead of building on unofficial ChatGPT API wrappers?

Use a route that someone maintains against ChatGPT's changes, or budget for that maintenance yourself. Code that depends on ChatGPT's private web endpoint breaks when the endpoint changes. The backend-api/f/conversation path changed twice in 2025. The source guide also records two changes to the stream chunk shape in one year, and the shopping-card event moved from a flat products array to a nested offers block. OpenAI's CSS class names also change between deploys, so class selectors break roughly weekly.

API routers such as LiteLLM and OpenRouter are a different category. They wrap the official API, so they return the API answer and not the UI answer.

The contract matters too. In a 2023 thread on OpenAI's developer forum, users report that the Terms of Use bar data extraction by scraping other than through the API. That risk is highest for an account that accepted the terms. Read the current OpenAI Terms of Use before you start. This is not legal advice.

When is DIY or the OpenAI API the better choice?

  • The OpenAI API is the better choice when your application needs generated text. A scraper endpoint is built for monitoring. A typical answer takes 30 to 45 seconds and each query is billed, which is expensive for casual use.
  • DIY Playwright is the better choice with zero budget and a strong engineering team. It is free, open source and has no vendor lock-in. Plan for 8 to 15 engineer hours a month.
  • A managed endpoint fits recurring monitoring: brand mentions, citation tracking and the same prompt from several countries. At 500,000 to 1 million requests a month, the source guide estimates $4,800 to $14,100 a month for an in-house stack, and engineer time is the largest part.

How these figures were collected

We measured grounding on roughly 2,500 prompts for each of six AI engines. For the tool comparison, we ran 30 queries through chatgpt.com with each of 8 tools and scored six criteria: auth handling, stream assembly, citation extraction, anti-bot survival, selector stability and cost per query.

The monthly costs are estimates. They assume engineer time at $100 an hour, a CAPTCHA solver at $2 per 1,000 challenges on 5% of requests, and mobile residential proxies at $10 per GB. The request, the parameters and the response fields come from the public README and were not changed.

This tutorial is published by cloro, which is one of the options in it. The cloro cost row and field list are self-reported. The 20% overlap figure belongs to Surfer.

Sources and full guides

Top comments (0)