To capture LLM chat responses across providers, send the same prompt to each provider on a schedule and map every answer to one record (provider, prompt, country, time, model, text, markdown, sources). Pick the access path per provider: a model API for what your own code generates, or a scraping API for what the consumer chat app shows a visitor.
- One schema fits all five surfaces. Each returns
text,markdownon request, and asourceslist where every entry has a 1-basedpositionand aurl. Only two differences need code: Google Search takesqueryinstead ofprompt, and it puts the AI Overview underaioverview. - Only ChatGPT returns a model name, and only when OpenAI serves the legacy interface. On the default mobile-web format,
modelisnull. Keep the field in the schema and expect it to be empty. - The six apps that are not covered have official APIs. Those APIs return the model's answer to your request, with the model and search settings you choose. They do not return the answer the app shows to a visitor.
- Decide which answer you need: what the consumer chat app shows a visitor, or what a model API generates for your own code.
- Write down the prompts and the countries you track.
- Submit each prompt to each provider, one task per provider.
- Poll each task, or receive a webhook, until it completes or fails.
- Map each result to one record and store it with the task ID.
- For the providers that are not covered, call their official API and write the same record.
How do I programmatically capture LLM chat responses across providers?
The question has two meanings, and the tools for each do not overlap.
If you mean the calls your own application makes to OpenAI, Anthropic or Google, put a gateway between your code and the providers and log what goes through it. In our monitoring of this prompt over the last 14 days, the AI engines read this meaning first: the home page of LiteLLM's documentation was the most cited page, in about one answer in six, and LiteLLM and OpenTelemetry pages made up six of the ten most cited pages. LiteLLM's proxy logging sends the input, output and exceptions of each call to Langfuse, OpenTelemetry or a storage bucket. That is the right tool for that job.
If you mean what ChatGPT, Gemini or Google show a person who types the prompt, a model API cannot give you that answer. The consumer app chooses its own model: the cloro docs say ChatGPT moves between gpt-5-3 and gpt-5-3-mini on its own logic. The app also runs its own search and shows its own source panel, product cards and place cards. To record that answer, something must load the app. cloro does this with real browser sessions, logged-out, through a proxy in the country you send. The rest of this post covers that path.
What does one normalized record look like?
Use one flat record per provider, prompt, country and run. The values below are placeholders that show the shape:
{
"provider": "GEMINI",
"prompt": "best crm for a 10-person sales team",
"country": "US",
"timestamp": "2026-11-16T09:00:00.000Z",
"status": "COMPLETED",
"model": null,
"text": "For a 10-person sales team, ...",
"markdown": "For a **10-person** sales team, ...",
"sources": [
{ "position": 1, "url": "https://example.com/crm-guide", "label": "CRM guide" }
]
}
Keep position as the provider gave it: the order of the source in that provider's list, starting at 1. It is the field you compare when you ask "who is cited first". Keep status so that a failed run is a row with no text, not a missing row; the two mean different things when you count share of voice.
How do you send one prompt to several providers in Python?
The async batch endpoint takes up to 500 tasks in one request, and each task has its own result. Async requests cost 2 credits less than the sync endpoints. This script submits one prompt to five surfaces, polls until each task ends, and maps each result to the record above.
import os
import time
import requests
BASE = "https://api.cloro.dev/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['API_KEY']}"}
PROMPT = "best crm for a 10-person sales team"
COUNTRY = "US"
chat = {"prompt": PROMPT, "country": COUNTRY, "include": {"markdown": True}}
TASKS = {
"CHATGPT": chat,
"GEMINI": chat,
"AIMODE": chat,
"BING_AIMODE": chat,
# Google Search takes `query`. The AI Overview is a +2 credit add-on.
"GOOGLE": {"query": PROMPT, "country": COUNTRY,
"include": {"aioverview": {"markdown": True}}},
}
def submit():
batch = [{"taskType": t, "payload": p} for t, p in TASKS.items()]
r = requests.post(f"{BASE}/async/task/batch", json=batch, headers=HEADERS)
r.raise_for_status()
return [res["task"]["id"] for res in r.json()["results"] if res["success"]]
def wait(task_id):
while True:
r = requests.get(f"{BASE}/async/task/{task_id}", headers=HEADERS)
r.raise_for_status()
body = r.json()
if body["task"]["status"] in ("COMPLETED", "FAILED"):
return body
time.sleep(10)
def normalize(body):
task, result = body["task"], body.get("response", {})
# The AI Overview sits one level down in the Google Search result.
answer = result.get("aioverview", {}) if task["taskType"] == "GOOGLE" else result
return {
"provider": task["taskType"],
"prompt": PROMPT,
"country": COUNTRY,
"timestamp": task["createdAt"], # when the task was submitted
"status": task["status"],
"model": answer.get("model"), # ChatGPT only, None on mobile-web
"text": answer.get("text", ""),
"markdown": answer.get("markdown", ""),
"sources": [
{"position": s.get("position"), "url": s.get("url"),
"label": s.get("label", "")}
for s in answer.get("sources", [])
],
}
records = [normalize(wait(task_id)) for task_id in submit()]
A completed task carries the provider's result directly in response, without a success wrapper. A failed task carries {"error": {...}} there, so normalize returns a row with empty text. Results are deleted 24 hours after you submit the task, so store them before then. For many prompts, set a webhook.url on each task instead of polling.
Which fields does each provider return?
The sources list and the markdown flag are the common base. The rest differs, and this table is the reason the record above keeps only the shared fields. It comes from the "Other features" table on the providers page and from each endpoint reference.
| Field | ChatGPT | Gemini | AI Mode | Bing AI Mode | AI Overview |
|---|---|---|---|---|---|
text, sources with position
|
Yes | Yes | Yes | Yes | Yes |
markdown |
include.markdown |
include.markdown |
include.markdown |
include.markdown |
include.aioverview.markdown |
citationPills |
Yes, IDs from 0 | Yes | Yes | Yes | Yes |
model |
Yes, null on mobile-web |
No | No | No | No |
Query fan-out (searchQueries) |
Paid add-on | No | No | No | No |
| Shopping cards | Paid add-on | Yes | Yes | No | Yes, as inlineProducts
|
places |
No | Yes | Yes | No | Yes |
US state targeting |
+2 credits | +2 credits | No | +2 credits | No |
| Async credits per request | 5 | 4 | 4 | 4 | 5 (3 for the search, +2) |
Two notes on the table. Bing AI Mode is the answer page at bing.com/copilotsearch; it is not the Copilot app. And citationPills has one entry per cited source, with a shared citationPillId for each chip in the answer, so it tells you which sources each claim used, which the flat sources list does not.
What about Claude, Perplexity, Copilot, DeepSeek, Meta AI and Grok?
Claude, Meta AI and DeepSeek require a login, so cloro does not scrape them, and it no longer scrapes Perplexity, the Copilot app or Grok. The path for each one is its official API:
| App | Official API | What comes back about sources |
|---|---|---|
| Claude | Messages API with the web search tool |
web_search_result_location citations with url, title and up to 150 characters of cited_text
|
| Perplexity |
Agent API (/v1/agent) |
[n] markers in the text that map to the id of each search_results entry |
| Grok | xAI Responses API | A citations list, and inline [[N]](url) links in the text |
| DeepSeek | DeepSeek API, OpenAI- and Anthropic-compatible | The home page lists web search among its application APIs and does not document the call |
| Meta AI | Meta Model API, OpenAI-compatible | The docs list "search grounding" with inline citations |
| Copilot | Microsoft 365 Copilot Chat API, preview | Needs a Microsoft 365 Copilot add-on license, and answers from work data and the web |
Perplexity's page also says that support for the older Sonar Chat Completions API ended on September 27, 2026. Synchronous Sonar calls still run, and Perplexity reformulates them as Agent API requests.
With each of these APIs you choose the model, the system prompt and whether the model searches. A visitor to the app gets none of those choices, so this is a different measurement. Write these rows with the same record, set provider to the API name (for example ANTHROPIC_API), and keep them out of any chart that compares app answers.
What does a run cost?
One prompt in one country across the five surfaces costs 22 credits async: 5 for ChatGPT, 4 each for Gemini, AI Mode and Bing AI Mode, and 5 for Google Search with the AI Overview. On the Starter plan ($250 for 650,000 credits), that is about $8.46 per 1,000 prompts across all five. On the Lite plan ($30 for 37,500 credits) it is $17.60. The free plan's 500 credits cover 22 full runs. Billing is per success: a FAILED task has creditsCharged: 0.
How this was checked
Every endpoint path, parameter and response field above comes from the API reference and its OpenAPI file, read on November 16, 2026. The third-party rows come from the linked official pages, read on the same day. The script was not run for this post, so test it with one prompt before you schedule it. The LiteLLM citation share comes from our own monitoring of this prompt across seven AI engines from November 2 to 16, 2026.
Limits: the apps change their answers from run to run, so one capture is one sample. Compare shares over many runs. And a scraped answer reflects a logged-out visitor in one country, not a signed-in user with history.
This post is published by cloro, which sells one of the APIs above.
Top comments (0)