DEV Community

Peyton Green
Peyton Green

Posted on

Python AI Agent Observability Without the SaaS Tax

#ai #agents #python #observability

Earlier this year, a Claude Code caching bug went semi-viral: a subtle cache-key mismatch was silently making requests miss the prompt cache, sending some sessions' API bills 10-20x higher than expected — with nothing in a normal workflow that would have surfaced it. It got tracked as a GitHub issue (#40524), cross-posted to X by Alex Volkov, and people are still writing it up months later. Nobody caught it by watching their code. People caught it by watching their bill.

That's the exact failure mode this toolkit exists to catch before it reaches your bill.

You built a Python AI agent. It works in development. Then you put it in front of real traffic and the questions start:

  • What did that session actually cost?
  • Why did it fail on that specific request?
  • Which tool call is slow?
  • Did the prompt change I shipped last week break anything?

If you've tried to answer these questions, you know the standard path: sign up for Langfuse, or Arize, or Helicone, or Weights & Biases, or one of the twelve other platforms that want to be your LLM observability layer. Most of them are good products. Most of them also cost $200-500/month and take an afternoon to integrate.

There's a lighter path. Here's a set of stdlib-only Python files that answer the same questions locally, with no external service required.

What you actually need from observability

Most developers building AI agents need four things:

  1. Trace logging — what happened in each step, how long it took, what went in and came out
  2. Cost tracking — how much did this session cost, which calls are expensive
  3. Error monitoring — when is something failing, alert me before users notice
  4. Regression testing — did my prompt change break behavior

That's it. You don't need distributed tracing. You don't need a real-time dashboard. You need these four things to work without falling over.

Trace logging: JSONL to a local file

The simplest thing that works: append a JSON record after each LLM call.

from trace_logger import AgentTraceLogger

logger = AgentTraceLogger("traces.jsonl", agent_name="search-agent")

with logger.trace("call_llm", input_text=prompt) as t:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )
    output = response.choices[0].message.content
    t.set_output(output)
    t.set_tokens(
        prompt_tokens=response.usage.prompt_tokens,
        completion_tokens=response.usage.completion_tokens,
        model="gpt-4o",
    )
Enter fullscreen mode Exit fullscreen mode

Each with logger.trace(...) block writes one line to traces.jsonl:

{"trace_id": "search-agent.call_llm.1743070800000", "agent_name": "search-agent", "step_name": "call_llm", "started_at": "2026-03-27T10:00:00Z", "ended_at": "2026-03-27T10:00:01Z", "latency_ms": 1247.3, "input_text": "search for Python asyncio patterns", "output_text": "Here are the key asyncio patterns...", "prompt_tokens": 45, "completion_tokens": 312, "total_tokens": 357, "model": "gpt-4o", "success": true}
Enter fullscreen mode Exit fullscreen mode

It also works as a decorator:

@logger.traced("summarize", model="gpt-4o-mini")
def summarize(text: str) -> str:
    return client.chat.completions.create(...).choices[0].message.content
Enter fullscreen mode Exit fullscreen mode

Stats over all traces:

print(logger.stats())
# {'total_traces': 847, 'error_count': 12, 'error_rate': 0.0142,
#  'avg_latency_ms': 1823.4, 'p95_latency_ms': 4201.0,
#  'step_breakdown': {'call_llm': 603, 'extract': 180, 'summarize': 64}}
Enter fullscreen mode Exit fullscreen mode

No database. No external service. A JSONL file you can grep, jq, or read into pandas.

Cost tracking: SQLite so restarts don't lose history

For costs, you want something that persists. JSONL is fine for traces, but cost data benefits from queries.

from cost_tracker import CostTracker

tracker = CostTracker("costs.db", session_id="user-abc-123")

# After each call:
tracker.record(
    model="gpt-4o",
    prompt_tokens=response.usage.prompt_tokens,
    completion_tokens=response.usage.completion_tokens,
    step_name="extract_entities",
)

# Session summary:
print(tracker.session_summary())
Enter fullscreen mode Exit fullscreen mode

Output:

{
  'session_id': 'user-abc-123',
  'total_cost_usd': 0.00612,
  'total_tokens': {'prompt_tokens': 1847, 'completion_tokens': 612, 'total_tokens': 2459},
  'breakdown': [
    {'model': 'gpt-4o', 'step': 'extract_entities', 'calls': 3, 'total_tokens': 1230, 'cost_usd': 0.00405},
    {'model': 'gpt-4o-mini', 'step': 'classify', 'calls': 5, 'total_tokens': 1229, 'cost_usd': 0.00207}
  ]
}
Enter fullscreen mode Exit fullscreen mode

The tracker ships with a pricing table for OpenAI, Anthropic, and Gemini models. Update it with one call:

tracker.add_model_pricing("my-fine-tuned-model", prompt_per_1m=5.00, completion_per_1m=15.00)
Enter fullscreen mode Exit fullscreen mode

Set a cost threshold to catch runaway sessions:

tracker = CostTracker("costs.db", session_id=session_id, alert_threshold_usd=0.50)

# Raises CostThresholdExceeded if session crosses $0.50
Enter fullscreen mode Exit fullscreen mode

Error monitoring: rolling error rate with webhook alerts

A rolling error rate is more useful than a total error count. If your last 100 steps have 20 failures, something is wrong — even if you've had 10,000 successful steps before.

from error_monitor import ErrorMonitor

monitor = ErrorMonitor(
    state_file="monitor_state.json",
    webhook_url=os.environ.get("SLACK_WEBHOOK_URL"),
    error_rate_threshold=0.15,  # alert when >15% of recent steps fail
    window_size=100,
    alert_cooldown_seconds=300,  # don't re-alert for 5 minutes
    agent_name="search-agent",
)

# In your agent loop:
try:
    result = run_tool_call(...)
    monitor.record_success("tool_call")
except ToolCallError as e:
    monitor.record_error("tool_call", error=e)
    # handle or re-raise

# Health check endpoint:
@app.get("/health")
def health():
    return monitor.health()
Enter fullscreen mode Exit fullscreen mode

When error rate exceeds the threshold, the monitor POSTs to your webhook:

[search-agent] Error rate alert: 18.0% errors in last 100 steps (threshold: 15.0%)
Enter fullscreen mode Exit fullscreen mode

Works with Slack, Discord, or any service that accepts a JSON POST with a text field. State persists to JSON — the window resets on restart, but alert history doesn't.

Regression testing: catch prompt changes before users do

The most common source of silent breakage in agent development: you change a prompt, the output looks reasonable, but it now fails a specific case that worked before.

from eval_harness import EvalHarness

harness = EvalHarness(agent_fn=my_agent, name="search-agent-v2")

harness.add_case(
    name="basic_factual",
    input="What is the capital of France?",
    expected="Paris",
)
harness.add_case(
    name="handles_empty_input",
    input="",
    judge=lambda output: len(output) > 0,  # should still respond
)
harness.add_case(
    name="cites_sources",
    input="What are the key asyncio patterns in Python?",
    judge=lambda output: any(word in output.lower() for word in ["asyncio", "async", "await"]),
)

report = harness.run(verbose=True)
print(report.summary())

# In CI:
report.assert_pass_rate(0.90)  # raises AssertionError if <90% of cases pass
Enter fullscreen mode Exit fullscreen mode

Load test cases from a file:

# test_cases.json
[
    {"name": "factual_1", "input": "Capital of France?", "expected": "Paris"},
    {"name": "factual_2", "input": "Capital of Germany?", "expected": "Berlin"}
]

harness.add_cases_from_file("test_cases.json")
Enter fullscreen mode Exit fullscreen mode

This isn't a replacement for promptfoo or DeepEval — those tools run thousands of variants with LLM-as-judge scoring. This is for the lightweight version: a set of known-good test cases you run on every deploy.

Optional: sync to Langfuse

If you want a web UI for browsing traces, Langfuse is worth the extra setup. It's open-source, self-hostable, and has a free cloud tier.

from langfuse_adapter import LangfuseAdapter

adapter = LangfuseAdapter.from_env()  # reads LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY

# Sync the last 50 traces you haven't sent yet:
adapter.sync_all(logger, limit=50)
Enter fullscreen mode Exit fullscreen mode

The adapter tracks which traces it's already synced and skips them on future calls.

Export for analysis

When you want to understand patterns across sessions:

python export.py report --traces agent_traces.jsonl --costs agent_costs.db
Enter fullscreen mode Exit fullscreen mode

Output:

Trace Summary (agent_traces.jsonl)
  Total traces:    847
  Errors:          12 (1.42%)
  Avg latency:     1823ms
  Total tokens:    847,392
  Top steps:
    call_llm: 603
    extract: 180
    classify: 64

Cost Summary (agent_costs.db)
  Total calls:     847
  Total cost:      $3.421847
  By session: ...
  By model: ...
Enter fullscreen mode Exit fullscreen mode

Or export to CSV for pandas:

python export.py traces --input agent_traces.jsonl --output traces.csv
Enter fullscreen mode Exit fullscreen mode

What this isn't

This toolkit is intentionally limited. It won't:

  • Trace requests across multiple services (use OpenTelemetry for distributed tracing)
  • Provide a real-time dashboard (use Langfuse, Grafana, or similar)
  • Run thousands of eval variants (use promptfoo or DeepEval for that)

It's local-first observability for a single Python agent. You get actionable data without giving it to a SaaS platform.

Getting started

The five core files (trace_logger, cost_tracker, error_monitor, eval_harness, export) are stdlib only — drop them into your project directory and import. No pip install required.

If you want the Langfuse integration: pip install langfuse.
If you want Parquet export: pip install pandas pyarrow.

The Python Agent Observability Toolkit includes all six files plus a README with wiring examples.

If you're at the point where you're asking "what did that session actually cost?" — that's the right moment to add this. It takes 30 minutes to wire in, and you'll have answers instead of guesses.

If you found this useful, the AI Dev Toolkit has 272 prompts for the other 90% of the AI development workflow — architecture decisions, code review, debugging, and more.

Top comments (0)