DEV Community

Elowen
Elowen

Posted on

Log Search Sources So AI Agent Answers Can Be Audited Later

A search-aware AI agent should not only return an answer.

It should also preserve enough context to explain where that answer came from.

If the agent called a search tool, what query did it use? When did it search? Which results were returned? Which sources were passed into the model? Which ones were ignored?

Without that trace, debugging becomes guesswork.

This post shows a small pattern for logging search sources as part of an AI agent answer trace. A SERP API such as TalorData SERP API can provide structured search result input; your agent workflow should preserve the search context around that input.

What the trace should answer

A useful search trace should help answer:

  • What did the agent search?
  • When did it search?
  • Which search result data was returned?
  • Which URLs were used as context?
  • Which result titles and snippets were visible to the model?
  • Was the source fresh enough for the task?
  • Can the answer be reviewed later?

This is not only for compliance or audit.

It also helps day-to-day debugging.

A minimal trace shape

Start with a simple object:

{
  "answer_id": "ans_2026_07_14_001",
  "user_prompt": "What changed in search results for this topic?",
  "search_trace": {
    "query": "serp api seo monitoring",
    "searched_at": "2026-07-14T09:00:00Z",
    "location": "United States",
    "results_used": [
      {
        "position": 1,
        "title": "Example result title",
        "url": "https://example.com/page",
        "snippet": "Short result summary"
      }
    ]
  },
  "answer_summary": "Short answer generated by the agent"
}
Enter fullscreen mode Exit fullscreen mode

You can expand this later, but the first version should preserve query, time, location, and source URLs.

Keep raw and selected sources separate

The search tool may return many results.

The model may only use a few.

Track both when possible:

{
  "raw_result_count": 10,
  "selected_result_count": 3,
  "selected_urls": [
    "https://example.com/page-a",
    "https://example.com/page-b",
    "https://example.com/page-c"
  ]
}
Enter fullscreen mode Exit fullscreen mode

That distinction matters.

If the answer is weak, you need to know whether the search results were poor, the selection step was poor, or the final generation step was poor.

A small Python example

This example is intentionally generic. Replace endpoint, auth, parameters, and response fields with the real documentation for your SERP API provider.

import json
import os
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

import requests


SERP_API_ENDPOINT = os.environ["SERP_API_ENDPOINT"]
SERP_API_KEY = os.environ["SERP_API_KEY"]
TRACE_DIR = Path("agent_traces")


def fetch_search_results(query: str, location: str) -> dict[str, Any]:
    response = requests.get(
        SERP_API_ENDPOINT,
        headers={"Authorization": f"Bearer {SERP_API_KEY}"},
        params={
            "q": query,
            "location": location,
        },
        timeout=30,
    )
    response.raise_for_status()
    return response.json()
Enter fullscreen mode Exit fullscreen mode

Normalize the result before putting it into the trace:

def normalize_results(raw: dict[str, Any]) -> list[dict[str, Any]]:
    results = []

    # Adjust this path to match your SERP API response schema.
    for index, item in enumerate(raw.get("organic_results", [])[:10], start=1):
        results.append(
            {
                "position": item.get("position", index),
                "title": item.get("title"),
                "url": item.get("link") or item.get("url"),
                "snippet": item.get("snippet"),
            }
        )

    return results
Enter fullscreen mode Exit fullscreen mode

Create an answer trace:

def build_answer_trace(
    answer_id: str,
    user_prompt: str,
    query: str,
    location: str,
    normalized_results: list[dict[str, Any]],
    answer_summary: str,
) -> dict[str, Any]:
    return {
        "answer_id": answer_id,
        "user_prompt": user_prompt,
        "search_trace": {
            "query": query,
            "searched_at": datetime.now(timezone.utc).isoformat(),
            "location": location,
            "results_used": normalized_results[:5],
            "raw_result_count": len(normalized_results),
        },
        "answer_summary": answer_summary,
    }
Enter fullscreen mode Exit fullscreen mode

Save it as JSON:

def save_trace(trace: dict[str, Any]) -> Path:
    TRACE_DIR.mkdir(exist_ok=True)
    path = TRACE_DIR / f"{trace['answer_id']}.json"

    with path.open("w", encoding="utf-8") as file:
        json.dump(trace, file, indent=2, ensure_ascii=False)

    return path
Enter fullscreen mode Exit fullscreen mode

What to log first

For a first version, log these fields:

  • answer ID
  • user prompt or prompt ID
  • search query
  • search timestamp
  • location or market
  • result URLs
  • titles and snippets
  • selected sources
  • answer summary
  • model or prompt version if relevant

Do not wait until the full observability system exists.

A basic trace is still much better than no trace.

Where the SERP API fits

The SERP API provides the structured search input.

The answer trace preserves how that input moved through the agent workflow.

For teams building search-aware AI agents, TalorData can provide the search result data layer. The agent application should still own trace storage, source selection records, and answer review workflows.

Final note

If you are building AI agents with search tools, what do you log today: search query, returned sources, selected sources, prompt version, or final answer only?

Top comments (0)