Measure Prompt Cache Usage with an OpenAI-Compatible Python Client

Prompt caching is easy to describe and easy to measure incorrectly.

A second request is not automatically proof of a cache hit. The model, provider, prompt length, request shape, and cache lifetime can all affect the result. The safest workflow is to send a controlled pair of requests and inspect the complete usage object returned by the API.

This tutorial uses RouterBase’s OpenAI-compatible chat completions endpoint.

Prerequisites

• Python 3.10+
• The current openai Python package
• A RouterBase API key
• A model ID copied from the live catalog at GET https://routerbase.com/v1/models

Store both values as environment variables:

export ROUTERBASE_API_KEY="sk-rb-..."
export ROUTERBASE_MODEL="your-current-model-id"

Do not hard-code production credentials in source code.

Install the SDK

pip install openai

Send two controlled requests

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ROUTERBASE_API_KEY"],
    base_url="https://routerbase.com/v1",
)

model = os.environ["ROUTERBASE_MODEL"]

shared_context = """
You are reviewing an API operations handbook.
Focus on error classification, tenant isolation, and observability.
Return one concise engineering recommendation.
""" * 80

def run(label: str) -> None:
    response = client.chat.completions.create(
        model=model,
        user="prompt-cache-observability-demo",
        messages=[
            {"role": "system", "content": shared_context},
            {"role": "user", "content": "What should the team log for each request?"},
        ],
    )

    usage = response.usage.model_dump() if response.usage else {}
    print(f"\n{label}")
    print(json.dumps(usage, indent=2))

run("first request")
run("second request")

The repeated system content and stable user value make the pair easier to compare. They do not guarantee that the second request will be cached.

What to inspect

Look for the cache-related fields exposed in the response usage, including cached_tokens where available. RouterBase’s current documentation also describes provider-native cache read and cache creation fields.

Compare them with:

• total input tokens
• model ID
• request latency
• task success
• cost per successful task

If the cache field is missing or zero, keep the result. It is evidence, not an error to hide. Recheck the current model, provider behavior, prompt shape, and documentation before drawing a conclusion.

Production notes

Use a stable end-user identifier so cache isolation matches your tenancy model. Never reuse one user identifier across unrelated customers merely to chase a higher hit rate.

Also avoid measuring only one request pair. A useful production view groups cache usage by model, prompt family, tenant, and task outcome over time.

Source of truth

RouterBase documents cache behavior and response fields here:

https://docs.routerbase.com/api-reference/chat-completions.md

For current model IDs, use the live catalog:

https://routerbase.com/v1/models

RouterBase: https://routerbase.com/?utm_source=devto&utm_medium=article&utm_campaign=prompt_cache_observability

Comments

[Luis Cruz]

I appreciate the emphasis on inspecting the complete usage object returned by the API to accurately measure prompt cache usage, as simply relying on a second request being a cache hit can be misleading due to various factors like model, provider, and prompt length. The example code using the OpenAI-compatible Python client is also helpful in demonstrating how to send controlled requests and compare the results. One important consideration that stands out is the need to use a stable end-user identifier for cache isolation, which is crucial for maintaining tenancy models in production environments. How do you handle cases where the cache field is missing or zero, and what additional steps would you recommend for troubleshooting cache behavior in such scenarios?