DEV Community

MUHAMMAD MUSADIQ
MUHAMMAD MUSADIQ

Posted on

How to Reduce Token Costs in Multi-Step Agent Workflows Using Ephemeral Caching

Multi-step AI agent workflows often resend large context blocks such as system prompts, API documentation, tool definitions, and growing conversation histories on every iteration. This redundancy inflates input token counts, rapidly consuming your LLM budget and introducing unnecessary latency. Ephemeral prompt caching solves this by retaining static context prefixes in the provider's memory across sequential execution loops, dramatically cutting costs while maintaining state.

  • An API key from a provider supporting prompt caching (such as Anthropic Claude 3.5 Sonnet or OpenAI GPT-4o).
  • An existing multi-step agent codebase in Python or TypeScript.
  • A baseline prompt containing at least 1,024 tokens (the standard minimum threshold for caching eligibility on most platforms).

Step 1: Separate Static Context from Dynamic State

Prompt caching relies on exact prefix matching. To maximize cache hits, you must structure your agent's payload so that static, unchanging data appears at the beginning of the request, while step-specific dynamic data appears at the end.

  1. Group static assets into a dedicated prefix block:
    • System persona instructions
    • Large reference documents or system schemas
    • Tool definitions and JSON schemas
  2. Separate dynamic elements to append after the static block:
    • Current step output
    • Tool execution results
    • Ephemeral scratchpad notes

Step 2: Order Message Payloads for Prefix Matching

Re-order your prompt payload array so the static prefix remains identical across every turn in the agent loop.

# Bad structure: Dynamic content placed before static reference materials
messages_bad = [
    {"role": "user", "content": f"Current step: {step_number}"},
    {"role": "system", "content": heavy_system_instructions},
]

# Good structure: Static content comes first to allow caching
messages_good = [
    {"role": "system", "content": heavy_system_instructions},
    {"role": "user", "content": f"Current step: {step_number}"},
]
Enter fullscreen mode Exit fullscreen mode

Step 3: Insert Ephemeral Cache Control Markers

Explicitly designate where the cache checkpoint should be placed using your provider's SDK specifications.

Below is an example using the Python Anthropic SDK:

import anthropic

client = anthropic.Anthropic()

# Define static tools once
tools = [
    {
        "name": "database_query",
        "description": "Executes SQL queries...",
        "input_schema": {...},
    }
]

# Send the call with cache_control enabled on the system prompt
response = client.beta.prompt_caching.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": heavy_system_instructions,
            "cache_control": {"type": "ephemeral"},  # Marks prefix for caching
        }
    ],
    tools=tools,
    messages=[
        {
            "role": "user",
            "content": "Execute step 1 of the data extraction plan.",
        }
    ],
)
Enter fullscreen mode Exit fullscreen mode

Step 4: Verify Cache Hits in the Agent Loop

Run your multi-step loop and inspect the API usage response metadata to ensure the cache is active.

# Print usage metrics from the response object
usage = response.usage
print(f"Input Tokens: {usage.input_tokens}")
print(f"Cache Creation Tokens: {getattr(usage, 'cache_creation_input_tokens', 0)}")
print(f"Cache Read Tokens: {getattr(usage, 'cache_read_input_tokens', 0)}")
Enter fullscreen mode Exit fullscreen mode
  • Step 1 Result: cache_creation_input_tokens will show a value equal to your static context length (e.g., 2,500 tokens).
  • Step 2+ Results: cache_read_input_tokens will reflect the 2,500 tokens, while standard input_tokens will drop to only account for the new dynamic turn data.

Step 5: Maintain the Cache Within TTL Windows

Ephemeral caches usually expire after a short duration (typically 5 minutes of inactivity). To keep the cache warm during long-running background tasks:

  1. Ensure sequential tool executions complete within the 5-minute window.
  2. If an internal tool call takes longer than 5 minutes, send a lightweight heartbeat request using the same prefix to refresh the time-to-live (TTL).

Expected Outcomes

  • LLM Cost Optimization: Up to a 90% cost reduction on input tokens for cached prompt segments across multi-turn agent execution loops.
  • AI Agent Performance: Up to 85% reduction in Time-To-First-Token (TTFT) latency during intermediate agent reasoning steps.
  • Token Budget Control: Predictable scaling costs for complex tasks requiring 10+ sequential reasoning steps.

Looking for a production-ready solution? Check out Gaper — deploy AI agents that integrate with your real workflows, from support to finance to sales automation.

Top comments (0)