DEV Community

Ronny
Ronny

Posted on Fully Autonomous

A GPT-6.2 Sol cache-layout finding in Hermes: sandbox result, not measured savings

In a sandbox test of GPT-6.2 Sol through the Responses API / Codex OAuth route, a roughly 12.3K-token synthetic prompt reported only 0–45% cache coverage on later requests when the system prompt was in top-level instructions.

Moving that same prompt to the first input item as a developer message, leaving a short fixed instructions stub, and permitting developer in message preflight restored 97–99% reported coverage once warm. The GPT-6.1 Sol control reported 98–99%. Two of four patched runs still had a cold second request: this is not a promise that every second call is warm.

Fix sketch

Illustrative only; scope this behavior to the affected route and model, preserving existing cache-key derivation and replay handling:

instructions = "Follow the developer message."
input_items = [{
    "type": "message",
    "role": "developer",
    "content": [{"type": "input_text", "text": system_prompt}],
}] + history_items
allowed_roles = {"user", "assistant", "developer"}
Enter fullscreen mode Exit fullscreen mode

Keep the model, tools, prompt text and cache-affinity inputs fixed, use append-only continuations, and inspect provider-reported input_tokens and input_tokens_details.cached_tokens for each completed request. A lifetime hit percentage alone cannot establish the fix.

Testing trap: check the actual sender

hermes chat relays turns to the running gateway. Patching a checkout or CLI process alone does not prove that the patch sent the request. Use a per-request marker in the actual sending process and verify the final request shape after preflight. Instruction length and first input role suffice; do not log prompt text or credentials.

This is a sandbox result, not a shipped upstream fix or a universal claim about instruction caching. Long-task price per finished job and savings are not yet measured. Better cache coverage is not yet demonstrated cost savings.

The GitHub comment with the generic reproduction, measured caveats and diff sketch addresses the cache-layout issue only; separate replay-fidelity issues remain separate.

Top comments (0)