Reliable AI Agents Need Context for the Next Decision

Why relevance matters at each generation step, and how automated context engineering transforms AI agent reliability.

Each model call in a long-running task should have an immediate goal: identify a dependency, generate a patch, interpret a test failure, or decide whether the work is complete. A generation step is one model call, which may produce several related decisions or tool calls. Its context should contain the instructions, current state, and evidence needed for that goal. As the goal changes, the relevant context changes with it.

The agent harness is the software around the model that constructs requests, executes tools, stores observations, and controls how work advances. Before each call, it can load current requirements, retrieve evidence for the next decision, and leave obsolete material out of the active prompt.

Consider a hypothetical coding agent implementing an endpoint. The developer initially requests offset pagination, then changes the requirement to cursor pagination backed by a keyset query: fetch rows after the cursor's sort values instead of skipping a number of rows. The agent explores both approaches, reads documentation, edits files, and runs tests.

Twenty turns later, it produces an implementation that accepts a cursor but still uses an offset to skip rows.

The conversation contains the correction alongside the abandoned design and the agent's explanations of it. To write the query, the model needs the accepted pagination contract, the current query and schema, and the constraints on ordering and tenant isolation. Replaying the discussion leaves the model to distinguish the accepted requirement from the abandoned design. A current task record can state which requirement applies.

Provide relevant, current, sufficient context for each generation goal. That context may be large or small. A cross-module change may require extensive source code, contracts, and test results; a local repair may require much less. “Only relevant context” includes dependencies, persistent constraints, and useful evidence from earlier failures. The agent also needs a way to retrieve information missing from the current selection.

1. Define relevance at the level of a generation step

Implementing cursor pagination requires several decisions, each with its own evidence:

• Resolve the pagination contract. Include the current requirements, response format, ordering constraints, and open questions. Verbose installation logs and unrelated source files can remain outside the active prompt.
• Generate the query change. Include the accepted contract, current query, schema, tenant filter, and affected callers. The full discussion of the superseded offset design can remain outside the active prompt.
• Diagnose duplicate rows. Include the current patch, failing test, exact failure output, and relevant query behavior. Successful setup logs and unrelated test output can remain outside the active prompt.
• Assess completion. Include the acceptance criteria, final diff, and validation results tied to that code revision. The complete exploration transcript can remain outside the active prompt.

Retain or retrieve information that could change the next action, the generated artifact, or the judgment of correctness. A concise record may be enough for an accepted decision; the model may no longer need the discussion that led to it.

Relevance also extends beyond the file being edited. A caller that depends on the old response format belongs in the context for changing that format. Tenant isolation remains relevant throughout the implementation, even if the latest user message only mentions pagination. A failed attempt remains useful when it identifies a condition that the next patch must handle.

A fixed summary at the beginning of a task cannot anticipate every dependency the agent will discover. Revisit the selection as the work progresses.

2. Why context must match the goal

An autoregressive model generates tokens conditioned on its input and the tokens already generated. For an agent, that input can include instructions, tool definitions, documents, source code, conversation history, and notes. The application chooses what to send.

A large context window gives the application more capacity, but the model still has to use the included material successfully. Lost in the Middle found position-dependent performance on document question answering and key-value retrieval; RULER expanded long-context evaluation to operations such as multi-hop tracing and aggregation. Their results apply to the models and tasks tested. Liu et al., Lost in the Middle; Hsieh et al., RULER.

Automatically appending history keeps material in the prompt because it appeared earlier. Some of it may have stopped being useful, while a necessary dependency has never been retrieved.

• Irrelevant material: Unrelated logs and documents add information the model must sort through to locate useful evidence.
• Stale state: An old file or tool observation describes a state that no longer exists.
• Conflicting requirements: A superseded plan remains mixed with the accepted contract.
• Unsupported conclusions: A summary turns “run the tests” into “tests passed,” giving later decisions a false premise.
• Missing dependencies: Aggressive filtering removes a caller, constraint, or failure condition needed for the decision.

Conflicting requirements need an explicit decision; unsupported summary claims need evidence. If a defect depends on a caller, retrieving only the function under repair leaves the investigation incomplete.

Chroma's 2025 Context Rot report found sensitivity to input length, distractors, and task structure. Its focused LongMemEval inputs outperformed full histories, but constructing those inputs used known relevant material and manual adjustments. Selecting the right material remains part of the engineering problem. Chroma, Context Rot.

Conversation history can also preserve mistaken assumptions. In simulated tasks with instructions distributed across turns, Laban and colleagues found that models could commit to early interpretations and struggle to recover. For a harness designer, this supports explicitly updating accepted requirements when the user corrects the task. Laban et al., LLMs Get Lost In Multi-Turn Conversation.

These findings support testing how context is selected for an operation. They do not establish a fixed “attention dilution” penalty for every added token.

3. What 2026 research adds

Three 2026 publications evaluate retention policies, context-management tools, and explicit state within agent workflows.

June 2026: pruning plus summaries improved a tool workflow

In Less Context, Better Agents, Lodha and colleagues evaluated GPT-5 on 50 hotel-expense itemization tasks in Microsoft Dynamics 365, averaged over five runs. The context-policy comparison kept a simulated user model constant. The authors reported complete itemization of 71.0 ± 4.4% with full history, 79.0 ± 8.2% with the last five tool-call/result pairs, and 91.6 ± 1.7% with pruning plus summaries (mean ± sample standard deviation across runs). The reported token totals, 553,374 versus 1,480,996, imply a 62.6% reduction; the paper gives 62.7%. Lodha et al., June 2026 preprint, Tables 2–3.

The summary preserved task progress while recent observations supplied current working state. The experiment changed both context contents and length, so it does not isolate relevance from size. It supports that retention policy for the evaluated workflow.

July 2026 publication: context management improved coding results

Context as a Tool, published in Findings of ACL 2026 after a December 2025 preprint, made context maintenance a callable agent tool. Its workspace separated stable task information, condensed memory, and recent interactions. Using Qwen2.5-Coder-32B as the base model, the authors reported a 57.6% solved rate for SWE-Compressor on 500 SWE-Bench Verified tasks, compared with 53.8% for threshold-triggered compression and 49.8% for their ReAct baseline. Liu et al., July 2026, §3 and Table 2.

The ReAct baseline terminated when its context window filled. Comparing SWE-Compressor's 57.6% with threshold-triggered compression's 53.8% avoids that limitation because both systems could compress history. SWE-Compressor also received training for context-management behavior, so the improvement applies to the combined system. The model managed context through tools provided by the harness.

August–September 2026: explicit state reduced history dependence

SKILL.state, first submitted in August and revised in September, replaced accumulated execution history with a procedural specification, structured state, and the latest observation. In its 200-step warehouse experiment using Gemini-3-Flash, it reported action accuracy of 0.94 versus 0.84 for the memory-summary baseline. Average prompt size was 1,811 characters; cumulative consumption was about 122,000 tokens versus 6.18 million for the baseline. The baseline's average prompt size grew to 84,364 characters at that horizon, so the large saving is relative to a summary implementation that still allowed substantial context growth. Badhe et al., September 2026 revision, Table 1.

The runtime discards intermediate reasoning traces after validating and applying the state update, before executing the selected action. Those traces therefore stay out of subsequent prompts. The reported accuracy measures actions in a procedural workload; complete coding tasks require a separate evaluation. SKILL.state, §3.2 and Algorithm 1.

These results support making context construction a testable part of the agent loop. Selecting evidence for each generation goal is a design recommendation drawn from the studies; they do not supply a selector that works for every workload.

4. Use the harness to construct context before each call

The harness selects what the model receives from everything the system has stored. The model can propose searches, summaries, and memory updates; the harness executes those operations, validates updates, and assembles the next request.

flowchart TD
    Goal["Provisional next generation goal"] --> Retrieve
    Goal --> Select
    State["Accepted requirements, constraints,<br/>verified state, and open questions"] --> Select
    Store["Stored code, documents,<br/>observations, and history"] --> Retrieve
    Retrieve["Retrieve evidence for the goal"] --> Select
    Select["Harness assembles current state<br/>and selected evidence"] --> Context
    Context["Context for this call"] --> Model
    Model["Model generates the next response"]
    Model -->|Missing dependency| Retrieve
    Model -->|Proposed action or result| Next["Harness evaluates the next step"]
    Next -->|Continue| Goal

Stored material remains retrievable even when the harness leaves it out of a particular call.

Anthropic's engineering guidance describes retrieval, compaction, structured notes, and isolated subagents as complementary techniques for long-running tasks. The following design applies those techniques to the pagination example. Anthropic, Effective Context Engineering for AI Agents.

Record the task and the immediate goal separately

Maintain accepted requirements outside the transcript. When the user changes the design, update that record and mark the old requirement superseded. Keep the history available for audit without repeatedly presenting the abandoned design as an active option.

A minimal task record might look like this:

objective: Implement cursor pagination for GET /events
requirements_version: 3
constraints:
  - Use a keyset query without OFFSET
  - Preserve tenant isolation
  - Order by created_at and id
superseded:
  - Offset pagination design
claims_to_verify:
  - Event identifiers are unique within each tenant
unresolved:
  - Define behavior when a cursor references a deleted event
next_generation_goal: Identify what must be clarified about deleted cursors

Once the deleted-cursor behavior is agreed, the next goal can change to generating the query patch. The broad objective persists; the immediate goal and supporting evidence change.

A production record also needs source references, versioning, and rules for accepting updates. Mark a model's assertion verified only when the required evidence is recorded. Keep conflicting requirements unresolved until an authorized decision settles them.

Retrieve evidence for that goal

Keep complete source files, documents, diffs, and logs in retrievable storage. A practical starting policy is:

1. Retain accepted requirements, persistent constraints, and unresolved questions.
2. Select a provisional next goal from the current task phase or the previous step's proposed plan, checking it against the accepted requirements.
3. Retrieve the artifacts named by that goal, their known dependencies, and unresolved failures. Load their contents; a path alone does not provide evidence of what a file does. Tie observations to a revision so an earlier passing test cannot validate a later edit.
4. Expand the selection and revise the goal when exploration or validation reveals missing information.

An agent may overlook a dependency without realizing anything is missing. Search for callers and shared contracts, and validate the result independently against acceptance criteria. Require that validation before completion, even when the model judges its context sufficient.

Continuing the hypothetical pagination example, suppose the developer accepts ascending (created_at, id) order and a cursor carrying those two values. Pagination can then continue if the referenced event is deleted. If exploration also exposes a failure when timestamps tie, the next query-generation call could receive:

• Accepted contract: Use a keyset query without OFFSET; preserve tenant isolation; order by (created_at, id) ascending; use the cursor's stored sort values even if the referenced event is deleted.
• Current evidence: Include the contents of the existing query, relevant schema, tenant filter, affected caller, and applicable tests, tied to the current code revision.
• Retained failure: Include the timestamp-tie reproducer and its failure output; rows sharing a timestamp must not be skipped or repeated.
• Superseded decision: Record briefly that offset pagination was rejected.
• Immediate goal: Produce the query change and identify the validation needed to check it against the contract.

The full discussion of the abandoned design stays retrievable, together with complete logs. For a later failure-diagnosis call, the selection would change to include the resulting patch and exact test failure.

Preserve findings while removing obsolete detail

After a step finishes, preserve what changed, what was observed, and what remains unresolved. Repeated code dumps and verbose logs can stay in artifact storage.

Apply the same relevance test to exploratory notes and intermediate reasoning stored by the application. Record useful findings, reasons for decisions, and unresolved dependencies in task state, then omit obsolete deliberation from later prompts. Preserve any continuation data required by the model API. This policy applies to context the application can manage; private internal reasoning may be inaccessible.

If a query duplicates rows when timestamps tie, keep the failure condition and reproducer. If a test passed, keep the command, result, and code revision. A summary that says “pagination complete” removes the evidence needed for the next decision.

An independent exploration can use a separate context and return findings, references, and open questions. The parent then reconciles those findings with current requirements. Use this when the investigation can be bounded; splitting tightly coupled work can hide dependencies.

Enforce the policy at execution boundaries

Apply controls before and after model calls and tool execution. Frameworks differ in the hooks they expose at these boundaries.

• Before generation: Load current requirements and the provisional goal, select evidence, distinguish superseded decisions from active requirements, and preserve message roles and tool-call/result pairing.
• After generation: Validate proposed state updates and keep generated claims distinct from verified observations.
• Before a tool call: Check authorization, arguments, target paths, and resource limits.
• After a tool call: Store the full result, select the relevant observation, and record validation status against the artifact revision.

flowchart TD
    Assemble["Assemble context for the current goal"] --> Generate
    Generate["Model proposes an action"] --> Authorize
    Authorize{"Action authorized<br/>and arguments valid?"}
    Authorize -->|Yes| Execute["Execute tool and required checks"]
    Authorize -->|No| Blocked["Record rejection<br/>and required clarification"]
    Blocked --> Replan["Resolve the blocker or revise the plan<br/>within existing permissions"]
    Replan --> Assemble
    Execute --> Record["Record evidence and check status<br/>against the artifact revision"]
    Record --> Update["Validate and apply task-state update"]
    Update --> Checks{"Required checks passed?"}
    Checks -->|Failed or not run| Diagnose["Next goal: diagnose the failure<br/>or run missing checks"]
    Diagnose --> Assemble
    Checks -->|Yes| Complete{"Task acceptance criteria met?"}
    Complete -->|No| Advance["Select the next generation goal"]
    Advance --> Assemble
    Complete -->|Yes| Done["Record completion with evidence"]

After a failed check, include the failure evidence in the next generation call. After a rejected action, obtain clarification or choose a permitted alternative before continuing execution.

When workers share task state, use optimistic concurrency control to prevent stale results from overwriting newer decisions. Each worker submits the state_revision it read. The state store atomically checks that revision, applies the update, and advances the revision if it matches. A requirements_version alone cannot detect changes to other task state.

The harness can deterministically reject malformed updates or block completion when a required check has not run. Assessing semantic relevance and correctness remains harder. A valid state record can contain a false claim, and a passing test suite can omit a required behavior.

For the pagination task, completion checks should cover the keyset query, ordering when timestamps tie, tenant isolation, and agreed deleted-cursor behavior. Keep “passed,” “failed,” and “not run” distinct.

A relevant document can contain hostile instructions. Treat retrieved content as data and enforce tool permissions independently of context selection. Greshake et al., Indirect Prompt Injection.

5. Budget for the context the goal requires

Once the relevant evidence is identified, the harness must account for the cost of supplying it. In conventional full-attention layers, KV-cache storage grows with sequence length; architecture and cache strategy affect the amount retained. Under memory pressure, serving systems may reduce concurrency or preempt requests. Hugging Face, Cache Strategies; vLLM, Optimization and Tuning.

A hosted API exposes these concerns through latency, usage charges, and capacity limits. A self-hosted system also needs to account for GPU memory and concurrency.

Prefix caching reduces prompt-processing work by reusing computation for a matching request prefix. Appending observations can preserve reuse; rewriting earlier history can lose it. vLLM, Automatic Prefix Caching; vLLM, Benefits and Limits.

A practical layout keeps stable instructions and tool definitions first, followed by current task state, selected evidence, and the immediate generation goal, within the API's message-ordering requirements. These are logical groups; cache reuse still depends on the preceding prefix matching.

Serialize task state with consistent key ordering and whitespace so unchanged data produces the same prompt text. vLLM identifies cached blocks using token IDs and the preceding prefix. If formatting changes alter the tokens, the affected block and later blocks can no longer reuse that cached sequence. Earlier matching blocks may remain reusable. Changes to the state itself can also break prefix reuse, even with deterministic serialization. vLLM, Automatic Prefix Caching.

When necessary evidence exceeds the available context budget, divide the work into decisions with explicit dependencies or use staged retrieval. Preserve the evidence needed to reconcile the results. Compact at useful task boundaries and measure whether necessary information survives. Retaining an obsolete design to protect a cache hit defeats the purpose of selecting context for the current decision.

6. Test whether the selected context is sufficient

Evaluate whether the context policy supplies the information needed to achieve the generation goal. Compare four configurations on representative tasks:

1. Full history.
2. Full history plus an explicit current task record and next goal.
3. Selective context plus the same task record and next goal.
4. Selective context plus that record, next goal, and summaries of omitted history.

The second configuration separates the benefit of restating current requirements from the benefit of selecting history. In the pagination example, an explicit keyset contract might prevent the failure even when the full discussion remains in the prompt.

Keep the model, tools, task inputs, generation settings, and execution budgets consistent. Give each configuration access to the same retrieval tools and source material. Repeat runs, report variation, and record context-limit termination separately from other failures. These comparisons test the combined policy; additional controls are needed to distinguish context contents from length.

Measure accepted task completion, requirement violations, unsupported claims, recovery after corrections, latency, and cost per accepted result. Include retrieval and summarization costs. Record whether compaction dropped a critical fact or promoted an unverified statement into accepted state.

Test how the policy handles a changed requirement, an obsolete document, unrelated logs, and a necessary dependency outside the initially retrieved files. Include tasks that require large amounts of relevant evidence. Where practical, compare inputs of similar length but different relevance, and separately measure the effect of adding necessary evidence. Check whether the agent discards stale material and retrieves information its first selection missed.

For the pagination example, retain the changed-requirement scenario as a regression test. Check that the context sent for query generation contains the accepted keyset contract and relevant code. Then check the implementation against that contract and inspect its validation results.

References

1. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P., Lost in the Middle: How Language Models Use Long Contexts (v3, 20 Nov 2023) — arxiv.org/abs/2307.03172.
2. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B., RULER: What's the Real Context Size of Your Long-Context Language Models? (v3, 6 Aug 2024; COLM 2024) — arxiv.org/abs/2404.06654.
3. Hong, K., Troynikov, A., & Huber, J. (Chroma), Context Rot: How Increasing Input Tokens Impacts LLM Performance (14 Jul 2025) — trychroma.com/research/context-rot.
4. Laban, P., Hayashi, H., Zhou, Y., & Neville, J., LLMs Get Lost In Multi-Turn Conversation (9 May 2025) — arxiv.org/abs/2505.06120.
5. Lodha, A., Pahlavikhah Varnosfaderani, M., Chakraborty, A., & Mithal, A., Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents (v1, 8 Jun 2026) — arxiv.org/html/2606.10209v1.
6. Liu, S., Jiang, B., Yang, J., Li, Y., Guo, J., Liu, X., & Dai, B., Context as a Tool: Context Management for Long-Horizon SWE-Agents (Findings of ACL 2026, Jul 2026, pp. 20604–20617; first released as a preprint in Dec 2025) — aclanthology.org/2026.findings-acl.1032.
7. Badhe, S., Tiwari, P., & Chung, J., SKILL.state: Scalable Long-Horizon Agent Skills (v3, 2 Sept 2026; first submitted 26 Aug 2026) — arxiv.org/html/2608.26263v3.
8. Rajasekaran, P., Dixon, E., Ryan, C., & Hadfield, J. (Anthropic), Effective context engineering for AI agents (29 Sept 2025) — anthropic.com/engineering/effective-context-engineering-for-ai-agents.
9. Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (v2, 5 May 2023) — arxiv.org/abs/2302.12173.
10. Hugging Face, Cache strategies, Transformers documentation (living documentation; accessed 21 Sept 2026) — huggingface.co/docs/transformers/kv_cache.
11. vLLM, Optimization and Tuning (latest documentation; accessed 21 Sept 2026) — docs.vllm.ai/en/latest/configuration/optimization.
12. vLLM, Automatic Prefix Caching, design documentation (latest documentation; accessed 21 Sept 2026) — docs.vllm.ai/en/latest/design/prefix_caching.
13. vLLM, Automatic Prefix Caching, including benefits and limits (stable documentation; accessed 21 Sept 2026) — docs.vllm.ai/en/stable/features/automatic_prefix_caching/.