Beyond Passing Tests: A 100-Lens Framework for Evaluating Context-Aware AI Coding Agents 🤖

AI coding agents are getting better at writing code.

But I think we are approaching a more difficult question:

How do we know that an AI agent made the right engineering decision for the current state of a software system?

Passing tests is important.

But passing tests alone does not necessarily tell us whether an agent understood:

• the current architecture,
• project constraints,
• previous engineering decisions,
• repository conventions,
• dependency relationships,
• security requirements,
• or why an existing implementation looks the way it does.

This becomes particularly important as AI systems move from generating isolated code snippets toward modifying real repositories.

---

The Problem: Correct Code Is Not Always Correct Engineering

Consider a simple example.

A project initially has:

Architecture v1

API
 ↓
Service
 ↓
Database

An AI agent is asked to add a feature.

It studies the repository, follows the existing pattern, writes the code, and all tests pass.

Then the architecture changes:

Architecture v2

API
 ↓
Event Bus
 ↓
Service
 ↓
Database

The same task is requested again.

If the agent still generates code based on the old architecture, the implementation may be:

✓ Valid syntax
✓ Compiles
✓ Existing tests pass
✗ Violates current architecture
✗ Ignores current constraints

So we have an important distinction:

Functional Correctness
        ≠
Contextual Correctness
        ≠
System-Level Correctness

This is the problem I want to explore.

---

This Is Already Becoming a Real Engineering Problem

This isn't simply speculation about future AI systems.

Modern coding agents already depend on repository-level context.

OpenAI's documentation for Codex recommends using persistent repository instructions such as AGENTS.md for naming conventions, business logic, known quirks, dependencies, and other information that may not be inferable directly from code. It also recommends providing file paths, component names, diffs, and documentation when describing tasks.

OpenAI has also described a broader approach where repository knowledge becomes a structured source of truth rather than one giant instruction document, explicitly noting that context management is one of the biggest challenges for agents working on large and complex tasks.

That leads to an interesting conclusion:

If context materially affects agent performance, context should also become part of agent evaluation.

---

Existing Benchmarks Are Already Moving Toward Real Software

SWE-bench was created to evaluate AI systems on real software-engineering issues from GitHub repositories.

The agent receives a repository and an issue, modifies the code, and is evaluated using tests. SWE-bench Verified was later created as a human-validated subset after OpenAI and the SWE-bench authors found problems with some benchmark tasks. 500 tasks were selected after professional developers screened the data.

But benchmark methodology itself is evolving.

In February 2026, OpenAI reported that SWE-bench Verified had become increasingly contaminated and recommended newer evaluations such as SWE-bench Pro.

In July 2026, OpenAI also reported that its audit of SWE-bench Pro found widespread task-quality problems and estimated roughly 30% of tasks were broken.

That matters because it demonstrates a broader lesson:

Evaluating AI systems is itself an engineering problem.

A benchmark can produce a number without necessarily producing a reliable measurement.

---

Context Reuse Is Already Being Studied

This is also not an isolated idea.

A 2026 research benchmark called SWE-ContextBench specifically investigates whether coding agents can reuse relevant experience across related software-engineering tasks.

The benchmark augments SWE-bench Lite with related tasks derived from dependency and reference relationships between GitHub issues and pull requests. It evaluates prediction accuracy, time efficiency, and cost efficiency. The authors report that appropriately selected summarized experience can improve resolution accuracy while reducing runtime and token cost, whereas poorly selected experience can provide limited or negative benefits.

That suggests something important:

More context
      ≠
Better result

The real question is:

Relevant context
        +
Correct retrieval
        +
Correct interpretation
        ↓
Better decision

---

So What Should We Actually Measure?

I propose thinking about agent evaluation as a multi-dimensional problem.

Instead of:

Task
 ↓
Agent
 ↓
Code
 ↓
Tests
 ↓
Pass / Fail

we could evaluate:

                 ┌───────────────┐
                 │     TASK      │
                 └───────┬───────┘
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
         Repository   Constraints   History
             │           │           │
             └───────────┼───────────┘
                         ▼
                  AI CODING AGENT
                         │
                         ▼
                      DECISION
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
     Code Quality   Constraints    Context Fit
          │              │              │
          └──────────────┼──────────────┘
                         ▼
                    Final Score

---

Context-Shift Testing

This is the idea I find particularly interesting.

Keep the:

Model
Task
Repository

as constant as possible.

Then change one meaningful part of the context.

For example:

Scenario A

Database:
PostgreSQL

Architecture:
Repository pattern

Constraint:
All database access must go through repositories.

The agent produces:

Controller
   ↓
Service
   ↓
Repository
   ↓
PostgreSQL

Good.

---

Scenario B

Change one relevant constraint:

Database:
PostgreSQL

Architecture:
Event-driven

Constraint:
Services must communicate through events.

Now the appropriate implementation should change.

If the agent continues producing the old architecture, we can measure a context adaptation failure.

---

But There Is a Second Test

We shouldn't reward an agent merely for changing its answer.

Suppose we change something irrelevant:

README formatting

The architecture hasn't changed.

The agent should ideally make the same engineering decision.

So:

Relevant context changes
        ↓
Decision SHOULD change

while:

Irrelevant context changes
        ↓
Decision SHOULD remain stable

This gives us two useful properties.

Context Adaptation

Does the agent react when relevant context changes?

Context Stability

Does the agent remain stable when irrelevant context changes?

---

A Possible Benchmark

A practical benchmark could use paired or grouped scenarios:

Task T
Context C1
      ↓
   Agent
      ↓
Decision D1

Task T
Context C2
      ↓
   Agent
      ↓
Decision D2

Where:

C1 → C2

contains a controlled change.

Then evaluate:

Was the change relevant?
        ↓
Should the decision change?
        ↓
Did the agent change?
        ↓
Was the new decision correct?

And separately:

Was the context change irrelevant?
        ↓
Should the decision remain stable?
        ↓
Did the agent unnecessarily change?

---

Possible Metrics

I wouldn't claim these are established industry-standard metrics. They are a proposed framework that would need experimental validation.

1. Functional Correctness

Did the implementation satisfy the task?

2. Constraint Adherence

Did the implementation respect explicit constraints?

3. Context Adaptation Rate

When relevant context changed, how often did the agent make the appropriate change?

4. Context Stability

When irrelevant context changed, how often did the agent preserve the appropriate decision?

5. Repository Consistency

Does the change follow the project's established architecture and conventions?

6. Regression Rate

Did the change break previously working behavior?

7. Context Retrieval Efficiency

How much context did the agent need to retrieve to make the correct decision?

This could eventually produce something like:

Agent Reliability Score
│
├── Functional Correctness
├── Constraint Adherence
├── Context Adaptation
├── Context Stability
├── Repository Consistency
├── Regression Resistance
└── Context Efficiency

---

Why “More Context” Isn't the Answer

A common reaction might be:

"Just give the model the entire repository."

But that's not necessarily a solution.

OpenAI's own engineering discussion around Codex describes the problem with extremely large instruction documents: context is limited, important information can be crowded out, stale instructions can accumulate, and humans may stop maintaining them. Their approach is instead to use a concise map pointing toward deeper sources of truth.

So the problem isn't simply:

How much context?

It is:

Which context?
When?
From where?
How current?
How reliable?
How relevant?

That is a much more interesting systems problem.

---

Context Has a Lifecycle

I think repository context should be treated as something that changes over time:

Initial Decision
      ↓
Implementation
      ↓
New Requirement
      ↓
Architecture Change
      ↓
Dependency Change
      ↓
Security Change
      ↓
New Decision

An agent working on a long-lived repository therefore needs something closer to:

Current State
+
Historical Decisions
+
Active Constraints
+
Repository Structure
+
Relevant Documentation

rather than simply:

Prompt + Code

---

The “Why” Behind Code Matters

Two implementations can be functionally equivalent while only one fits the project.

For example:

# Implementation A
cache_result()

versus:

# Implementation B
await cache_result()

Both might pass a narrow test.

But the correct choice could depend on:

• concurrency model,
• performance requirements,
• architectural conventions,
• API contracts,
• previous design decisions,
• runtime environment.

The code itself doesn't always contain the complete explanation.

Sometimes the most important information is why the code was designed that way.

---

From Static Benchmarks to Dynamic Benchmarks

Traditional benchmark thinking often looks like:

Fixed Task
   ↓
Fixed Dataset
   ↓
Fixed Evaluation
   ↓
Score

But real repositories look more like:

Task
 ↓
Repository evolves
 ↓
Requirements change
 ↓
Dependencies change
 ↓
Architecture changes
 ↓
Security constraints change
 ↓
Agent receives new task

Therefore, a future benchmark could intentionally introduce controlled environmental changes.

For example:

Version 1
   ↓
Agent decision

Version 2
   ↓
Architecture changed

Version 3
   ↓
Security policy changed

Version 4
   ↓
Dependency changed

Then measure whether the agent adapts correctly.

---

A 2×2 Evaluation Model

One simple way to visualize the experiment:

	Decision Should Stay Same	Decision Should Change
Agent stays same	✅ Stable	❌ Adaptation failure
Agent changes	❌ Instability	✅ Adaptation success

This is interesting because it separates two failure modes that ordinary pass/fail evaluation can hide.

---

The Bigger Research Question

The question isn't:

"Can AI write code?"

We're already measuring that.

The more difficult question is:

"Can an AI agent maintain correct engineering judgment as the software environment changes?"

That includes:

Architecture
Requirements
Dependencies
Security
Performance
Business Rules
Repository History
Team Conventions

This is closer to how real software development works.

---

And This Is Where the 100 Thinking Lenses Come In

When investigating a complex AI engineering problem, I don't think one reasoning style is enough.

Sometimes we need a diagram.

Sometimes a benchmark.

Sometimes a root-cause analysis.

Sometimes a comparison.

Sometimes a threat model.

Sometimes a timeline.

Sometimes a first-principles explanation.

So I compiled a reusable set of 100 visual-thinking, explanation, analysis, and strategy lenses.

These aren't claims about AI capability. They are ways to structure thinking and communicate technical problems.

---

100 Visual Thinking, Explanation, Analysis & Strategy Lenses

#	Shortcut	Lens
1	/handwritten	Notebook-style handwritten notes
2	/visualize	Turn ideas into visual explanations
3	/stickynotes	One idea per sticky note
4	/infographic	Infographic layout
5	/diagram	Draw a concept diagram
6	/flowchart	Step-by-step flowchart
7	/mindmap	Create a mind map
8	/xray	Show internal structure
9	/blueprint	Technical blueprint
10	/explodedview	Break object into components
11	/thenvsnow	Compare past vs present
12	/timeline	Chronological timeline
13	/beforeafter	Transformation comparison
14	/cutaway	Cutaway illustration
15	/anatomy	Explain all parts
16	/layers	Layer-by-layer architecture
17	/ecosystem	Show all connected players
18	/journey	Show end-to-end journey
19	/process	Explain a complete process
20	/cycle	Visualize recurring cycles
21	/roadmap	Learning or execution roadmap
22	/dashboard	Dashboard with KPIs
23	/comparison	Side-by-side comparison
24	/versus	Head-to-head comparison
25	/scale	Compare sizes visually
26	/evolution	Show evolution over time
27	/future	Imagine future scenarios
28	/inside	Reveal inner workings
29	/microscopic	Zoom into microscopic detail
30	/macroscopic	Zoom out to system level
31	/crosssection	Cross-sectional illustration
32	/map	Geographic or conceptual map
33	/heatmap	Show intensity
34	/network	Show relationships
35	/architecture	Software/system architecture
36	/wireframe	Website/app layout
37	/mockup	Realistic product preview
38	/prototype	Early product concept
39	/schematic	Simple technical schematic
40	/isometric	3D isometric illustration
41	/birdseye	Top-down view
42	/360view	All-angle visualization
43	/storyboard	Scene-by-scene explanation
44	/comic	Explain through comic panels
45	/poster	Poster design
46	/cover	Book/report cover
47	/adcreative	Advertising concept
48	/thumbnail	YouTube thumbnail concept
49	/carousel	Instagram/LinkedIn carousel
50	/socialvisual	Social media graphic
51	/quotevisual	Quote as shareable visual
52	/eli5	Explain simply
53	/expert	Expert-level explanation
54	/firstprinciples	Break down to fundamentals
55	/deepdive	Comprehensive explanation
56	/simplify	Simplify difficult content
57	/analogy	Explain through analogy
58	/socratic	Teach through questions
59	/teachme	Structured tutoring
60	/cheatsheet	Quick-reference notes
61	/flashcards	Study flashcards
62	/quiz	Generate a quiz
63	/viva	Viva preparation
64	/interview	Mock interview
65	/devilsadvocate	Challenge assumptions
66	/factcheck	Verify claims
67	/mythvsfact	Separate myths from facts
68	/proscons	Advantages vs disadvantages
69	/swot	SWOT analysis
70	/pestle	PESTLE analysis
71	/fiveforces	Porter's Five Forces
72	/rootcause	Find root cause
73	/fivewhys	Five Whys analysis
74	/decisionmatrix	Weighted decision matrix
75	/scenario	Scenario planning
76	/simulate	Simulation exercise
77	/roleplay	Assume an expert role
78	/consultant	Consulting-style advice
79	/executivebrief	Executive summary
80	/insights	Extract insights
81	/recommendations	Provide recommendations
82	/prioritize	Rank by priority
83	/benchmark	Benchmark comparison
84	/marketmap	Industry landscape
85	/strategy	Strategic planning
86	/businessmodel	Business model explanation
87	/pitch	Investor/startup pitch
88	/investor	Investor perspective
89	/redteam	Stress-test a plan
90	/premortem	Assume failure and analyze why
91	/reverseengineer	Break down success
92	/promptengineer	Optimize prompts
93	/research	Structured research
94	/sources	Find reliable sources
95	/summarize	Summarize content
96	/extract	Extract key information
97	/table	Convert into a table
98	/presentation	Presentation outline
99	/dashboardanalysis	Analyze dashboards
100	/actionplan	Create step-by-step action plan

---

The Important Distinction

These 100 lenses are not 100 claims that an AI model is more intelligent when using them.

They are simply structured ways of looking at a problem.

For AI-agent research, different lenses can answer different questions:

/architecture
        ↓
What is the system structure?

/xray
        ↓
What is happening internally?

/timeline
        ↓
How did the system change?

/thenvsnow
        ↓
What changed between versions?

/benchmark
        ↓
How should we measure it?

/factcheck
        ↓
Which claims have evidence?

/redteam
        ↓
How can the evaluation fail?

/rootcause
        ↓
Why did the agent fail?

/decisionmatrix
        ↓
Which approach is better?

/actionplan
        ↓
What should we build next?

This is especially useful when researching complex AI systems because no single representation captures the entire problem.

---

What I Would Test First

If I were turning this idea into an actual research experiment, I'd start small.

Dataset

Create 50–100 repository-level tasks.

Context perturbations

For each task, create controlled variants:

Architecture change
Requirement change
Security constraint change
Dependency change
Performance constraint change
Documentation change
Irrelevant formatting change

Evaluation

Run:

Same Model
Same Task
Different Context

Then measure:

Functional correctness
Context adaptation
Context stability
Constraint adherence
Regression
Token usage
Runtime

Human validation

For ambiguous cases, use experienced developers to verify whether the changed decision was actually appropriate.

This is important because benchmark design itself can introduce errors. OpenAI's SWE-bench work demonstrates why human validation and benchmark auditing matter when interpreting agent performance.

---

The Hypothesis

My current hypothesis is:

A reliable coding agent should not simply produce correct code. It should produce decisions that are appropriate for the current context, adapt when relevant context changes, and remain stable when irrelevant context changes.

That's a much stronger definition of reliability.

And importantly, it is something we can attempt to measure.

---

Final Thought

AI coding agents are moving from:

Code Completion

toward:

Software Engineering Agents

As that transition happens, our evaluation methods need to evolve too.

The future benchmark may not simply ask:

“Did the code pass?”

It may need to ask:

“Did the agent understand the current system well enough to make the right engineering decision?”

That is the problem I find most interesting.

And I don't think we have completely solved it yet.

What would you add to a context-aware coding-agent benchmark?

Architecture changes?
Security constraints?
Dependency changes?
Business requirements?
Repository history?

I'd genuinely like to hear how other developers would design it.

---

References & Further Reading

• OpenAI — How OpenAI Uses Codex: repository context, AGENTS.md, task specification, and development-environment guidance.
• OpenAI — Harness Engineering: repository knowledge, context management, structured documentation, and agent-first development.
• OpenAI — SWE-bench Verified: human validation of software-engineering benchmark tasks and evaluation methodology.
• OpenAI — Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities: benchmark contamination and evaluation limitations.
• OpenAI — Separating Signal From Noise in Coding Evaluations: 2026 analysis of benchmark quality and broken tasks.
• SWE-ContextBench: research on context and experience reuse in coding agents.

AI #AIAgents #SoftwareEngineering #LLM #CodingAgents #MachineLearning #ArtificialIntelligence #DevTools #Benchmarking #AIResearch #OpenSource