DEV Community: David Van Assche (S.L)

Measuring What Your AI Learned: Epistemic Vectors in Practice

David Van Assche (S.L) — Mon, 13 Apr 2026 17:45:28 +0000

Part 2 of the Epistemic AI series. In Part 1, we defined the problem: AI tools don't track what they know. Here, we make it measurable.

When we talk about "what the AI knows," we're not being metaphorical. Knowledge has structure, and that structure is measurable — not perfectly, but well enough to catch the failures that matter.

The 13 Epistemic Vectors

Empirica tracks 13 dimensions of an AI's knowledge state. Not as a gimmick — each vector maps to a specific class of failure you've seen in practice:

vectors = {
    "know":        0.65,  # Domain understanding
    "uncertainty": 0.35,  # What I DON'T know (explicit!)
    "context":     0.70,  # Surrounding state awareness
    "clarity":     0.80,  # How clear the path forward is
    "coherence":   0.75,  # Internal consistency
    "signal":      0.60,  # Information quality vs noise
    "density":     0.55,  # Relevant knowledge per unit context
    "state":       0.70,  # Current system/project state
    "change":      0.40,  # Amount of change made
    "completion":  0.30,  # Progress toward goal
    "impact":      0.65,  # Significance of work
    "engagement":  0.85,  # How actively working the problem
    "do":          0.70,  # Ability to execute
}

Why 13? Because we kept finding failure modes that weren't captured:

know without uncertainty = overconfident AI
clarity without signal = clear path built on noise
completion without change = claiming done but nothing happened
engagement without do = actively spinning without capability

Each pair creates a tension that prevents gaming. You can't claim high know while uncertainty is also high — the measurement catches the contradiction.

The Transaction Lifecycle

Vectors aren't static. They change as the AI works. The epistemic transaction is the measurement window:

PREFLIGHT → [investigate] → CHECK → [implement] → POSTFLIGHT

PREFLIGHT: Declare Your Baseline

Before starting work, the AI declares what it thinks it knows:

empirica preflight-submit - << 'EOF'
{
  "task_context": "Implement JWT auth middleware",
  "vectors": {
    "know": 0.45,
    "uncertainty": 0.40,
    "context": 0.60,
    "clarity": 0.50
  },
  "reasoning": "Read the route definitions but haven't explored 
    the middleware chain yet. Moderate context from project structure."
}
EOF

This is the starting measurement. It's a prediction: "Here's how well I think I understand this before investigating."

Investigation Phase (Noetic)

The AI reads code, searches patterns, builds understanding. Everything it discovers gets logged:

# What you learned
empirica finding-log --finding "Auth middleware uses Express next() 
  pattern at routes/auth.js:45" --impact 0.5

# What you don't know
empirica unknown-log --unknown "How are user roles differentiated? 
  No role field in JWT payload schema."

# What didn't work
empirica deadend-log --approach "Tried passport.js integration"   --why-failed "Too heavy for JWT-only auth, would add 12 dependencies"

These aren't just notes — they're grounded evidence that the calibration system uses to verify self-assessments.

CHECK: Gate the Transition

empirica check-submit - << 'EOF'
{
  "vectors": {
    "know": 0.82,
    "uncertainty": 0.15,
    "context": 0.85,
    "clarity": 0.88
  },
  "reasoning": "Investigated middleware chain, understand JWT flow, 
    found role definitions in JWT claims. Ready to implement."
}
EOF

The system evaluates: did the vectors change in a way that's consistent with the evidence logged? If the AI claims know: 0.82 but logged zero findings and zero unknowns, that's a rushed assessment — the gate catches it.

This is the critical insight: you can't skip investigation and go straight to acting. The measurement forces understanding before execution.

POSTFLIGHT: Measure the Learning

After implementation:

empirica postflight-submit - << 'EOF'
{
  "vectors": {
    "know": 0.90,
    "uncertainty": 0.08,
    "change": 0.80,
    "completion": 1.0
  },
  "reasoning": "Auth middleware implemented with role guards. 
    Unit tests passing. Learned about Express 5 async changes."
}
EOF

The delta between PREFLIGHT and POSTFLIGHT is the learning:

know:        0.45 → 0.90  (+0.45)  # Learned a lot
uncertainty: 0.40 → 0.08  (-0.32)  # Resolved most unknowns
change:      0.00 → 0.80  (+0.80)  # Made substantial changes
completion:  0.00 → 1.00  (+1.00)  # Goal met

This delta IS the measurement. Over time, you can see:

Does the AI consistently overestimate its starting knowledge?
Does it underestimate uncertainty?
Do its estimates get more accurate across sessions?

Grounded Verification: The Part That Keeps It Honest

Self-assessment alone is self-serving. The grounded verification layer compares the AI's claims against deterministic evidence:

# AI claims: know=0.90, change=0.80
# Grounded evidence:
evidence = {
    "test_results": {"passed": 42, "failed": 3},     # 3 failures!
    "ruff_violations": 2,                              # lint issues
    "git_diff_lines": 156,                            # real change metric
    "findings_logged": 5,                              # investigation breadth
    "unknowns_resolved": 3,                            # learning evidence
}

# Grounded calibration:
# - test failures → know is probably ~0.75, not 0.90
# - git diff confirms change=0.80 is reasonable
# - 5 findings + 3 resolved unknowns → investigation was real

The calibration score measures the distance between self-assessment and grounded evidence. A score of 0.0 means perfect calibration. In practice, we see scores of 0.10-0.30 — the AI is usually overconfident, and the grounded layer catches it.

What This Looks Like in Practice

Here's a real POSTFLIGHT from an Empirica session (editing for clarity):

Calibration score: 0.134
Grounded coverage: 69.2%

Gaps:
  know:        overestimate by 0.33  (claimed 0.82, evidence shows 0.49)
  uncertainty: underestimate by 0.13 (claimed 0.15, evidence shows 0.28)
  coherence:   underestimate by 0.20 (claimed 0.75, evidence shows 0.95)

Sources: artifacts, codebase_model, prose_quality, 
         document_metrics, source_quality, action_verification
Sources failed: []  (all evidence collectors healthy)

The AI was overestimating its knowledge and underestimating its uncertainty — the most common pattern. But now we can see it, which means we can correct for it in the next transaction.

Try It

pip install empirica
cd your-project
empirica project-init
empirica setup-claude-code

# Start a measured session:
empirica session-create --ai-id claude-code
# → Opens transaction, gates investigation before action

The framework is open source, the measurement is real, and the calibration improves over time. Not because the model gets better — because the measurement infrastructure makes overconfidence visible.

Next in the series: **Part 3 — Grounded Calibration vs Self-Assessment* — why the AI's self-report is structurally unreliable and how deterministic evidence changes the game.*

Empirica on GitHub | Part 1: Your AI Doesn't Know What It Doesn't Know

Your AI Doesn't Know What It Doesn't Know — And That's the Biggest Problem in AI Tooling

David Van Assche (S.L) — Mon, 13 Apr 2026 17:45:27 +0000

"The most dangerous thing isn't an AI that's wrong. It's an AI that's wrong and confident about it."

Every developer working with AI agents has hit this wall: your tool says something with absolute confidence, and it's completely wrong. Not because the model is bad — because nothing in the system tracks what it actually knows versus what it's guessing.

This is the epistemic gap, and it's the single biggest unsolved problem in AI developer tooling.

The Problem: Confidence Without Calibration

When you use Claude, ChatGPT, or any LLM-based tool:

It never says "I'm 60% sure about this"
It doesn't distinguish between "I read this in the codebase" and "I'm inferring this from patterns"
After a long conversation, it loses track of what it verified versus what it assumed
When context compresses, learned insights vanish silently

This isn't a model problem. GPT-5 won't fix it. Claude Opus 5 won't fix it. It's a measurement problem at the infrastructure layer.

What Actually Happens in Practice

You ask your AI to update the auth middleware. It says "Done!" with 100% confidence. But:

Did it check if JWT was already configured? Maybe.
Did it verify the session store compatibility? Probably not.
Will it remember this decision next session? No.
Did it investigate before acting, or just pattern-match? You'll never know.

The AI doesn't track:

What it investigated versus what it assumed
Which assumptions turned out to be wrong
What it learned that should persist across sessions
How its confidence should change based on evidence

Why This Matters More Than You Think

If you're building AI-assisted workflows, this gap compounds:

No learning curve. Your AI makes the same mistakes on day 100 that it made on day 1, because nothing measures whether its predictions improve.
Invisible context loss. When conversations compact (Claude Code, Cursor, etc. all do this), the AI loses track of what it verified. It re-assumes things it already checked.
Sycophancy masquerading as agreement. When you push back on a wrong answer, the AI often just agrees with you — not because you're right, but because agreement is the path of least resistance. Without calibration, there's no mechanism to distinguish "user is right, I should update" from "user is insistent, I should capitulate."
No grounded verification. The AI self-reports its confidence. Nobody checks. It's like a student grading their own exam.

What Epistemic Measurement Looks Like

Imagine if your AI tooling tracked 13 dimensions of its own knowledge state:

Vector	What It Measures
know	How well it understands the domain
uncertainty	What it DOESN'T know (explicit)
context	Understanding of surrounding state
clarity	How clear the path forward is
signal	Quality of information vs noise
change	Amount of change made
completion	Progress toward current goal

And imagine it measured these at three points:

PREFLIGHT: "Here's what I think I know before starting"
CHECK: "Here's what I learned during investigation — am I ready to act?"
POSTFLIGHT: "Here's what I actually learned and changed"

The delta between PREFLIGHT and POSTFLIGHT IS the learning. Not a vibe. A measurement.

The Grounded Calibration Loop

Self-assessment alone is sycophantic. What you actually need is a comparison between what the AI claims to know and what deterministic evidence shows:

AI self-assessment: know = 0.85, uncertainty = 0.10
Grounded evidence (test results, linter, git diff): know = 0.62, uncertainty = 0.35
Calibration gap: overestimating know by 0.23, underestimating uncertainty by 0.25
Adjustment signal: "Be more cautious with know estimates in future transactions"

The grounded evidence comes from deterministic services — test results, linter output, git metrics, documentation coverage — things that don't lie. When the AI says "I know this codebase well" but the test suite shows 3 failures in the module it just edited, the gap is measurable.

This is what calibration means: the distance between what you claim to know and what the evidence shows. Over time, this distance should shrink. If it doesn't, the AI isn't getting better — it's just getting more confident.

This Isn't Theory — It's Infrastructure

We've been building this measurement layer as an open-source framework called Empirica. It's a Python CLI that hooks into Claude Code (and any LLM tool that supports hooks) to:

Track epistemic vectors across sessions
Gate actions behind investigation (you can't write code until you've demonstrated understanding)
Verify self-assessments against deterministic evidence
Persist learning across context compaction
Measure calibration drift over time

It's not a wrapper or a prompt. It's measurement infrastructure that makes the epistemic gap visible and closes it over time.

Getting Started

Prerequisites: Python 3.10+, a project with a git repo, and optionally Claude Code for the full hook integration.

# Install Empirica
pip install empirica

# Initialize tracking in your project
cd your-project
empirica project-init

# If using Claude Code, wire up the hooks:
empirica setup-claude-code

That's it. From this point, every Claude Code conversation in this project is measured — PREFLIGHT declares baseline knowledge, CHECK gates the transition from investigation to action, and POSTFLIGHT captures what was actually learned. The Sentinel (an automated gate) ensures investigation happens before implementation.

Without Claude Code, you can still use the CLI directly to track any AI workflow:

# Declare what you know before starting
empirica preflight-submit - <<< '{"vectors": {"know": 0.5, "uncertainty": 0.4}, "reasoning": "Starting auth investigation"}'

# Log what you discover
empirica finding-log --finding "JWT middleware uses Express next() pattern" --impact 0.5

# Measure what you learned
empirica postflight-submit - <<< '{"vectors": {"know": 0.85, "uncertainty": 0.1}, "reasoning": "Auth flow fully understood"}'

What's Next in This Series

This is Part 1 of a series on epistemic AI — making AI tools that actually know what they know:

Part 2: Measuring What Your AI Learned — epistemic vectors in practice
Part 3: Grounded Calibration vs Self-Assessment — why self-reporting fails
Part 4: Adding Epistemic Hooks to Your Workflow — integration tutorial
Part 5: The Voice Layer — how AI learns your communication patterns

Each article will have runnable code, real measurements, and honest assessments of what works and what doesn't. Because that's the whole point — if you're not honest about uncertainty, you're just building a more eloquent liar.

Empirica is open source (MIT) and under active development. We're a small team in Vienna building measurement infrastructure for AI. If this resonates, check us out on GitHub or follow this series for the deep dives.

Why Your AI Agent Needs Memory That Decays (and How Qdrant Makes It Work)

David Van Assche (S.L) — Fri, 06 Mar 2026 13:30:22 +0000

I've been building an open-source epistemic measurement framework called Empirica, and one of the core challenges I ran into early on was memory — not the "stuff vectors in a database and retrieve them" kind, but memory that actually behaves like memory. Things fade. Patterns strengthen with repetition. A dead-end from three weeks ago should still surface when the AI is about to walk into the same wall, but a finding from a one-off debugging session probably shouldn't carry the same weight six months later.

That's where Qdrant comes in, and I want to share how we're using it because it's a fairly different use case from the typical RAG setup.

The problem with flat retrieval

Most RAG implementations treat memory as a flat store — embed a chunk, retrieve by similarity, done. That works for document Q&A, but it falls apart when you need temporal awareness. An AI agent working across sessions and projects needs to know not just what was discovered, but when, how confident we were, and whether that knowledge is still valid.

Think about how your own memory works — you don't recall every detail of every workday equally. The time you accidentally dropped the production database? That stays vivid. The routine PR you reviewed last Tuesday? Already fading. That asymmetry is functional, not a bug.

Two memory types, one vector store

We use Qdrant for two distinct memory layers:

Eidetic memory — facts with confidence scores. These are discrete epistemic artifacts: findings ("the auth system uses JWT refresh with 15min expiry"), dead-ends ("tried migrating to async but the ORM doesn't support it"), decisions ("chose SQLite over Postgres because single-user, no server needed"), mistakes ("forgot to check null on the config reload path"). Each carries a confidence score that gets challenged when new evidence contradicts it — a finding's confidence drops if a related finding surfaces that undermines it. Think of it as an immune system: findings are antigens, lessons are antibodies.

Episodic memory — session narratives with temporal decay. These capture the arc of a work session: what was the AI investigating, what did it learn, how did its confidence change from start to finish. Episodic memories naturally decay over time — a session from yesterday is more relevant than one from last month, unless the pattern keeps repeating, in which case it strengthens instead of fading.

Both live in Qdrant as separate collections per project, which gives us clean isolation and lets us do cross-project pattern discovery when we need it.

The retrieval side — Noetic RAG

I've been calling this approach "Noetic RAG" — retrieval augmented generation on the thinking, not just the artifacts. When an AI agent starts a new session, we don't just load documents. We load:

Dead-ends that match the current task (so it doesn't repeat failed approaches)
Mistake patterns with prevention strategies
Decisions and their rationale (so it understands why things are the way they are)
Episodic arcs from similar sessions (temporal context)
Cross-project patterns (if the same anti-pattern appeared in project A, surface it in project B)

The similarity search here isn't just cosine distance on the task description — it's filtered by recency, weighted by confidence, and scoped by project (with optional global reach for cross-project learnings).

What this looks like in practice

# Focused search: eidetic facts + episodic session arcs
empirica project-search --project-id <ID> --task "auth token rotation"

# Full search: all collections
empirica project-search --project-id <ID> --task "auth token rotation" --type all

# Include cross-project patterns
empirica project-search --project-id <ID> --task "auth token rotation" --global

When context compacts (and it will — Claude Code's 200k window fills up fast), the bootstrap reloads ~800 tokens of epistemically ranked context instead of trying to reconstruct everything from scratch. Findings, unknowns, active goals, architectural decisions — weighted by confidence and recency.

The temporal dimension

This is the part that makes Qdrant particularly well-suited. We store timestamps and decay parameters as payload fields, and filter on them at query time. A dead-end from yesterday with high confidence outranks a finding from last month with medium confidence. But a pattern that's been confirmed three times across two projects? That climbs in relevance regardless of age.

The decay isn't a fixed curve — it's modulated by reinforcement. Every time a pattern re-emerges, its effective age resets. Qdrant's payload filtering makes this efficient: we can do the temporal math at query time without re-embedding anything.

Why this matters beyond the obvious

The real value isn't just "AI remembers things" — it's that the memory is epistemically grounded. Every artifact has uncertainty quantification. Every session has calibration data (how accurate was the AI's self-assessment compared to objective evidence like test results and code quality metrics). The memory doesn't just tell you what happened — it tells you how much to trust what happened.

After 5,600+ measured transactions, the calibration data shows AI agents consistently overestimate their own confidence by 20-40%. Having memory that carries that calibration forward means the system gets more honest over time, not just more knowledgeable.

Try it

Empirica is MIT licensed and open source. If you're building anything where AI agents need to remember across sessions — especially if temporal awareness matters — the prosodic/episodic/eidetic architecture might be worth looking at.

GitHub: github.com/Nubaeon/empirica
Website: getempirica.com
Install: pip install empirica

Happy to answer questions about the Qdrant integration or the broader noetic RAG architecture.

The best (free - cheap) AI friendly Cli and Coding environments

David Van Assche (S.L) — Fri, 26 Sep 2025 17:01:41 +0000

With so many LLM providers and coding environments, how do you choose the right one for your next project? We all want the "best" model, but what we really need is the one that's the most reliable, the most cost-effective, and the most suited for our workflow. This guide breaks down the real-world performance, pricing, and hidden costs of the top LLM providers and CLI environments, from freemium to enterprise. We'll go beyond the marketing claims and give you the data you need to make an informed decision.

CLI and Code-Focused Environments (Sorted by Cost)

Tier 1: Free & Open-Source (Cost is just API Tokens / Free Tier Access)

Cursor CLI
- Cost Model: Free. Relies on the user's API key (OpenAI, Anthropic, etc.).
- Use Case & Features: An editor and CLI environment built around a code-aware AI. Ideal for developers who want maximum control over the model and are happy to manage their own API costs.
Qwen Code
- Cost Model: Free tier with 2,000 requests per day and a 60 RPM limit.
- Use Case & Features: A coding agent focused on tool calling and environment interaction. Offers a generous free tier for developers on a budget, perfect for experimenting with agentic workflows.
GitHub Copilot CLI
- Cost Model: Free Tier Available. New "Copilot Free" tier offers 2,000 code completions and 50 premium requests per month. Students, teachers, and open-source maintainers get Copilot Pro for free.
- Use Case & Features: Agent-powered, GitHub-native tool that executes coding tasks. This is the new, more powerful agentic Copilot CLI, replacing the older gh-copilot extension.

Tier 2: Freemium & Free-for-Individual (Generous Free Access)

Gemini Code Assist
- Cost Model: **Free for individuals (permanently).
- ** Access to higher daily limits is available through a subscription to Google AI Pro ($19.99/month), which often includes an
- extended free trial for 12 months for students in eligible regions.
- Use Case & Features: An AI-first coding assistant integrated directly into major IDEs and the terminal (Gemini CLI). The individual version is a highly generous, no-cost option.
Warp Code
- Cost Model: Freemium. Includes 150 free AI requests per month. Paid plans start around
- $15/user/month for teams.
- Use Case & Features: A complete agentic development environment that unifies the terminal, editor, and AI features. Known for its speed, local indexing, and multi-model orchestration.
Atlassian Rovodev
- Cost Model: Free tier with an Atlassian Cloud account. Quotas are based on "AI credits" tied to paid Jira/Confluence plans.
- Use Case & Features: Integrated with the Atlassian ecosystem, focusing on developer tasks within project management (Jira) and documentation (Confluence). Best for teams already on the Atlassian stack.

Tier 3: Subscription Required (Paid Access to High-End Models)

Codex CLI (OpenAI)
- Cost Model: Included with ChatGPT paid plans (Plus, Pro, Business, Enterprise), starting at $20/user/month (ChatGPT Plus).
- Use Case & Features: A comprehensive software engineering agent powered by models like GPT-5-Codex. It works in the terminal, IDE, and cloud, using tools, tracking progress with a to-do list, and supporting multi-modal input.
Claude Code (Anthropic)
- Cost Model: Likely included with a paid Claude Pro or higher subscription. Uses Anthropic's latest models (e.g., Sonnet/Opus).
- Use Case & Features: An agentic coding partner focused on extended thinking and complex, multi-step tasks. It uses planning modes and creates project memory files (CLAUDE.md) for deep context management.
Grok CLI (xAI)
- Cost Model: Requires an X Premium+ subscription, which is roughly $ 30 - $40/month for consumer access, or an API plan for token-based billing.
- Use Case & Features: Distinguished by its focus on real-time data integration (from the X platform) and its "rebellious streak." Best for projects requiring up-to-the-minute data integration alongside coding tasks.

LLM and Inference Provider Cross-Reference

OpenAI
- Freemium / Entry Level Cost: Free tier with a $5.00 credit (valid for 3 months).
- Pricing Model: Pay-per-token; different rates for different models (e.g., GPT-4o is more expensive than GPT-3.5).
- Reliability & Throughput: High. Offers tiered usage limits that scale with spend. Known for robust infrastructure but can have occasional downtime.
Anthropic
- Freemium / Entry Level Cost: Free tier with a $10.00 credit (valid for a limited period).
- Pricing Model: Pay-per-token. Haiku model is the most cost-effective.
- Reliability & Throughput: High. Free tier has rate limits (e.g., 5 RPM, 20K TPM on Haiku), which are sufficient for development and experimentation.
Deepseek
- Freemium / Entry Level Cost: Free tier with some trial credits.
- Pricing Model: Pay-per-token, with separate rates for input (cache hit/miss) and output tokens.
- Reliability & Throughput: Generally good, but may not have the same global infrastructure as larger providers. Good for cost-sensitive projects.
Qwen (Dashscope)
- Freemium / Entry Level Cost: Free tier with 2,000 requests per day and a 60 RPM limit.
- Pricing Model: Pay-per-token after free tier. The Qwen-Flash model is very cheap for simple tasks.
- Reliability & Throughput: Good. The free tier is generous for personal projects and offers a great way to test the model's capabilities.
Fireworks AI
- Freemium / Entry Level Cost: $50 monthly spend limit with a valid payment method.
- Pricing Model: Pay-per-token. Very competitive rates for various open-source models like Deepseek and Qwen.
- Reliability & Throughput: Very high. Known for its speed and low latency. The free tier is well-suited for experimentation and small-scale applications before committing to a higher spend tier.
Amazon Bedrock
- Freemium / Entry Level Cost: Pay-as-you-go, no upfront cost. Some models have a free trial period.
- Pricing Model: Complex. On-demand, provisioned throughput, and commitment-based pricing. Pricing varies significantly by model.
- Reliability & Throughput: Extremely High. Backed by AWS's robust infrastructure, offering high reliability and the ability to scale. Best for production use cases where you need a consistent throughput.
Hugging Face
- Freemium / Entry Level Cost: Free tier for inference endpoints on some models.
- Pricing Model: Pay-per-hour for dedicated inference endpoints.
- Reliability & Throughput: Reliability depends on the model's popularity and the infrastructure supporting it. The free tier can have high latency due to a queuing system.
OpenRouter
- Freemium / Entry Level Cost: Pay-per-token. You only pay for what you use.
- Pricing Model: Aggregator. Provides access to many models (including OpenAI and Anthropic) on a single API key, often at competitive rates.
- Reliability & Throughput: Varies by model, but generally high. The platform manages the back-end complexity of multiple models, making it a great entry point for comparing different models.