DEV Community

Cover image for How do I programmatically capture LLM chat responses across providers?
Ricardo Batista for cloro

Posted on

How do I programmatically capture LLM chat responses across providers?

To capture LLM chat responses across providers, send the same prompt to each provider on a schedule and map every answer to one record (provider, prompt, country, time, model, text, markdown, sources). Pick the access path per provider: a model API for what your own code generates, or a scraping API for what the consumer chat app shows a visitor.

  • One schema fits all five surfaces. Each returns text, markdown on request, and a sources list where every entry has a 1-based position and a url. Only two differences need code: Google Search takes query instead of prompt, and it puts the AI Overview under aioverview.
  • Only ChatGPT returns a model name, and only when OpenAI serves the legacy interface. On the default mobile-web format, model is null. Keep the field in the schema and expect it to be empty.
  • The six apps that are not covered have official APIs. Those APIs return the model's answer to your request, with the model and search settings you choose. They do not return the answer the app shows to a visitor.
  1. Decide which answer you need: what the consumer chat app shows a visitor, or what a model API generates for your own code.
  2. Write down the prompts and the countries you track.
  3. Submit each prompt to each provider, one task per provider.
  4. Poll each task, or receive a webhook, until it completes or fails.
  5. Map each result to one record and store it with the task ID.
  6. For the providers that are not covered, call their official API and write the same record.

How do I programmatically capture LLM chat responses across providers?

The question has two meanings, and the tools for each do not overlap.

If you mean the calls your own application makes to OpenAI, Anthropic or Google, put a gateway between your code and the providers and log what goes through it. In our monitoring of this prompt over the last 14 days, the AI engines read this meaning first: the home page of LiteLLM's documentation was the most cited page, in about one answer in six, and LiteLLM and OpenTelemetry pages made up six of the ten most cited pages. LiteLLM's proxy logging sends the input, output and exceptions of each call to Langfuse, OpenTelemetry or a storage bucket. That is the right tool for that job.

If you mean what ChatGPT, Gemini or Google show a person who types the prompt, a model API cannot give you that answer. The consumer app chooses its own model: the cloro docs say ChatGPT moves between gpt-5-3 and gpt-5-3-mini on its own logic. The app also runs its own search and shows its own source panel, product cards and place cards. To record that answer, something must load the app. cloro does this with real browser sessions, logged-out, through a proxy in the country you send. The rest of this post covers that path.

What does one normalized record look like?

Use one flat record per provider, prompt, country and run. The values below are placeholders that show the shape:

{
  "provider": "GEMINI",
  "prompt": "best crm for a 10-person sales team",
  "country": "US",
  "timestamp": "2026-11-16T09:00:00.000Z",
  "status": "COMPLETED",
  "model": null,
  "text": "For a 10-person sales team, ...",
  "markdown": "For a **10-person** sales team, ...",
  "sources": [
    { "position": 1, "url": "https://example.com/crm-guide", "label": "CRM guide" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Keep position as the provider gave it: the order of the source in that provider's list, starting at 1. It is the field you compare when you ask "who is cited first". Keep status so that a failed run is a row with no text, not a missing row; the two mean different things when you count share of voice.

How do you send one prompt to several providers in Python?

The async batch endpoint takes up to 500 tasks in one request, and each task has its own result. Async requests cost 2 credits less than the sync endpoints. This script submits one prompt to five surfaces, polls until each task ends, and maps each result to the record above.

import os
import time

import requests

BASE = "https://api.cloro.dev/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['API_KEY']}"}
PROMPT = "best crm for a 10-person sales team"
COUNTRY = "US"

chat = {"prompt": PROMPT, "country": COUNTRY, "include": {"markdown": True}}
TASKS = {
    "CHATGPT": chat,
    "GEMINI": chat,
    "AIMODE": chat,
    "BING_AIMODE": chat,
    # Google Search takes `query`. The AI Overview is a +2 credit add-on.
    "GOOGLE": {"query": PROMPT, "country": COUNTRY,
               "include": {"aioverview": {"markdown": True}}},
}


def submit():
    batch = [{"taskType": t, "payload": p} for t, p in TASKS.items()]
    r = requests.post(f"{BASE}/async/task/batch", json=batch, headers=HEADERS)
    r.raise_for_status()
    return [res["task"]["id"] for res in r.json()["results"] if res["success"]]


def wait(task_id):
    while True:
        r = requests.get(f"{BASE}/async/task/{task_id}", headers=HEADERS)
        r.raise_for_status()
        body = r.json()
        if body["task"]["status"] in ("COMPLETED", "FAILED"):
            return body
        time.sleep(10)


def normalize(body):
    task, result = body["task"], body.get("response", {})
    # The AI Overview sits one level down in the Google Search result.
    answer = result.get("aioverview", {}) if task["taskType"] == "GOOGLE" else result
    return {
        "provider": task["taskType"],
        "prompt": PROMPT,
        "country": COUNTRY,
        "timestamp": task["createdAt"],  # when the task was submitted
        "status": task["status"],
        "model": answer.get("model"),    # ChatGPT only, None on mobile-web
        "text": answer.get("text", ""),
        "markdown": answer.get("markdown", ""),
        "sources": [
            {"position": s.get("position"), "url": s.get("url"),
             "label": s.get("label", "")}
            for s in answer.get("sources", [])
        ],
    }


records = [normalize(wait(task_id)) for task_id in submit()]
Enter fullscreen mode Exit fullscreen mode

A completed task carries the provider's result directly in response, without a success wrapper. A failed task carries {"error": {...}} there, so normalize returns a row with empty text. Results are deleted 24 hours after you submit the task, so store them before then. For many prompts, set a webhook.url on each task instead of polling.

Which fields does each provider return?

The sources list and the markdown flag are the common base. The rest differs, and this table is the reason the record above keeps only the shared fields. It comes from the "Other features" table on the providers page and from each endpoint reference.

Field ChatGPT Gemini AI Mode Bing AI Mode AI Overview
text, sources with position Yes Yes Yes Yes Yes
markdown include.markdown include.markdown include.markdown include.markdown include.aioverview.markdown
citationPills Yes, IDs from 0 Yes Yes Yes Yes
model Yes, null on mobile-web No No No No
Query fan-out (searchQueries) Paid add-on No No No No
Shopping cards Paid add-on Yes Yes No Yes, as inlineProducts
places No Yes Yes No Yes
US state targeting +2 credits +2 credits No +2 credits No
Async credits per request 5 4 4 4 5 (3 for the search, +2)

Two notes on the table. Bing AI Mode is the answer page at bing.com/copilotsearch; it is not the Copilot app. And citationPills has one entry per cited source, with a shared citationPillId for each chip in the answer, so it tells you which sources each claim used, which the flat sources list does not.

What about Claude, Perplexity, Copilot, DeepSeek, Meta AI and Grok?

Claude, Meta AI and DeepSeek require a login, so cloro does not scrape them, and it no longer scrapes Perplexity, the Copilot app or Grok. The path for each one is its official API:

App Official API What comes back about sources
Claude Messages API with the web search tool web_search_result_location citations with url, title and up to 150 characters of cited_text
Perplexity Agent API (/v1/agent) [n] markers in the text that map to the id of each search_results entry
Grok xAI Responses API A citations list, and inline [[N]](url) links in the text
DeepSeek DeepSeek API, OpenAI- and Anthropic-compatible The home page lists web search among its application APIs and does not document the call
Meta AI Meta Model API, OpenAI-compatible The docs list "search grounding" with inline citations
Copilot Microsoft 365 Copilot Chat API, preview Needs a Microsoft 365 Copilot add-on license, and answers from work data and the web

Perplexity's page also says that support for the older Sonar Chat Completions API ended on September 27, 2026. Synchronous Sonar calls still run, and Perplexity reformulates them as Agent API requests.

With each of these APIs you choose the model, the system prompt and whether the model searches. A visitor to the app gets none of those choices, so this is a different measurement. Write these rows with the same record, set provider to the API name (for example ANTHROPIC_API), and keep them out of any chart that compares app answers.

What does a run cost?

One prompt in one country across the five surfaces costs 22 credits async: 5 for ChatGPT, 4 each for Gemini, AI Mode and Bing AI Mode, and 5 for Google Search with the AI Overview. On the Starter plan ($250 for 650,000 credits), that is about $8.46 per 1,000 prompts across all five. On the Lite plan ($30 for 37,500 credits) it is $17.60. The free plan's 500 credits cover 22 full runs. Billing is per success: a FAILED task has creditsCharged: 0.

How this was checked

Every endpoint path, parameter and response field above comes from the API reference and its OpenAPI file, read on November 16, 2026. The third-party rows come from the linked official pages, read on the same day. The script was not run for this post, so test it with one prompt before you schedule it. The LiteLLM citation share comes from our own monitoring of this prompt across seven AI engines from November 2 to 16, 2026.

Limits: the apps change their answers from run to run, so one capture is one sample. Compare shares over many runs. And a scraped answer reflects a logged-out visitor in one country, not a signed-in user with history.

This post is published by cloro, which sells one of the APIs above.

Top comments (0)