Your AI agent says 'done' but nothing works: an open-source framework to fix it

I've been working with AI agents in production (Claude Code, Codex, Cursor) for over a year. The pattern is always the same:1. You ask the agent for something2. It says "done"3. It doesn't work4. You switch models5. Still doesn't workThe problem isn't the model. It's the environment.## The gapThere's a huge gap between "demo that works" and "system that produces consistent results." Models are capable, but they fail because of:- Vague specifications: the agent can only guess- No verification: says "done" without running tests- No state: every session starts from scratch- No observability: you don't know the cost or if quality drops## The solution: 6 progressive tiersI published ia-engineer-framework, an open-source framework (MIT) that closes this gap with 6 layers:### Tier 1: Harness (10 min)Templates you copy to your project. CLAUDE.md with instructions, feature_list.json with machine-readable features, init.sh to verify the environment.

bashcp -r ia-engineer-framework/harness/ your-project/# Edit CLAUDE.md with your commands# Edit feature_list.json with your features# Run your agent — it reads the files automatically

Tier 2: Eval (30 min)JSONL datasets with scenarios and expected answers. A runner that tests against any LLM. A scorer that measures accuracy. A gate that blocks CI if it drops below the threshold.

bashpython3 run-evals.py --dataset prompting-basics.jsonl --output results.jsonlpython3 score-evals.py --results results.jsonl# Accuracy: 87.5% — PASS

Tier 3: Observe (1h)Logger that records 10 fields per interaction (tokens, cost, latency, tools, errors). Cost tracker that aggregates by session/model/day. Importable Grafana dashboard.### Tier 4: Hooks (15 min)Claude Code automation: pre-commit lint, post-task tests, session cost guard, security scan.### Tier 5: Patterns (reading)8 reference documents: diagnostic loop, multi-session, graph engineering, circuit breaker, HITL, model routing, prompt versioning, agent testing.### Tier 6: CI/CD (1h)GitHub Action that runs evals on every PR. Quality gate that blocks merge if they fail.## Diagnostic Loop: the most useful patternWhen your agent fails, don't switch models. Diagnose the layer:1. Specification — was the task clear?2. Context — did the agent have the info?3. Environment — correct dependencies?4. Verification — were there tests?5. State — did it remember previous work?Fix the layer that failed. Repeat. After 3-5 rounds your environment is robust. Without spending more money.## Comparison with Learn Harness Engineering (11.3K stars)| | LHE | ia-engineer-framework ||---|---|---|| Scope | Harness only | Harness + eval + observe + CI || Eval pipeline | No | Runner + scorer + CI gate || Observability | No | Logger + cost tracker + alerts || CI/CD | No | GitHub Action + quality gate || Format | Course | Copy-paste to production || Lock-in | Specific | Claude Code, Codex, Cursor |## Principles1. Automation over AI — if a test solves it, don't use an LLM2. Verifiable > smart — 8 eval scenarios > 1 "perfect" prompt3. Copy-paste to production — every file works standalone4. Progressive — Tier 1 in 10 min, Tier 6 when you have a team5. No lock-in — Claude Code, Codex, Cursor, anything## Get started in 5 minutes

bashgit clone https://github.com/pedri77/ia-engineer-framework.gitcp -r ia-engineer-framework/harness/ your-project/

Edit CLAUDE.md and feature_list.json with your data. Run your agent.GitHub: pedri77/ia-engineer-frameworkMIT. Use, modify, distribute freely.---This is part of IAcademy, an applied AI academy for real work.