DEV Community: Kwansub Yun

AI-SLOP Detector v3.5.0 — Every Claim, Verified Against Source Code

Kwansub Yun — Wed, 15 Apr 2026 06:19:37 +0000

I published a LinkedIn post about AI-SLOP Detector's self-calibration system and download numbers. Someone asked the reasonable question: "Can you actually back that up?"

Yes. Here's the source.

This isn't a feature announcement. It's a line-by-line audit of seven claims against the actual codebase. Every VERDICT links to a real file and real line numbers. The repo is public — go check it yourself.

What was claimed

Claim	Verdict
Every scan is recorded	✅ TRUE
Repeat scans become calibration signal	✅ TRUE
Updates only when signal is strong enough	✅ TRUE
Visible policy artifact (`.slopconfig.yaml`)	✅ TRUE
Explicit numeric limits govern calibration	✅ TRUE
Detects empty/stub/phantom/disconnected code	✅ TRUE
~1.4K downloads last week	✅ TRUE

All seven. No fabrications. No inflated numbers. Here's the proof.

Claim 1: "Every scan is recorded"

Source: src/slop_detector/history.py, lines 116–180

def record(self, file_analysis, git_commit=None, git_branch=None, project_id=None) -> None:

Auto-invoked on every CLI run. The only opt-out is --no-history. Each scan writes to SQLite at ~/.slop-detector/history.db and stores:

deficit_score, ldr_score, inflation_score, ddc_usage_ratio
n_critical_patterns, fired_rules
git_commit, git_branch, project_id

Schema is now at v5, auto-migrated on startup through every release from v2.9.0 to v3.5.0.

VERDICT: TRUE. The record() call is real. The schema is versioned. The behavior is not optional.

Claim 2: "Every re-scan becomes signal"

Source: src/slop_detector/history.py, lines 221–246

def count_files_with_multiple_runs(self, project_id=None) -> int:
    # Only files scanned >= 2 times count as calibration events
    SELECT file_path FROM history GROUP BY file_path HAVING COUNT(*) >= 2

Source: src/slop_detector/ml/self_calibrator.py, lines 301–309

def _extract_events(self, project_id=None):
    rows = self._load_history(project_id=project_id)
    by_file = self._group_runs_by_file(rows)

Single-scan files produce no calibration events. Only repeat scans generate improvement or fp_candidate labels. The threshold is hardcoded in SQL, not assumed.

VERDICT: TRUE. The repeat-scan requirement is enforced at the query level, not in documentation.

Claim 3: "Updates only when the signal is strong enough"

Source: src/slop_detector/ml/self_calibrator.py, lines 37–54 (constants) and 251–262 (enforcement)

CONFIDENCE_GAP: float = 0.10   # min gap between #1 and #2 candidate
MIN_IMPROVEMENTS: int = 5       # improvement events required
MIN_FP_CANDIDATES: int = 5      # fp_candidate events required

Gate 1 — confidence gap check (line 251):

if result.confidence_gap < CONFIDENCE_GAP:
    result.status = "insufficient_data"
    result.message = (
        f"Confidence gap {result.confidence_gap:.4f} < {CONFIDENCE_GAP}. "
        f"Candidates are too close — need more history data for reliable calibration."
    )
    return result  # NO UPDATE APPLIED

Gate 2 — score delta check (line 262):

if current_score - winner_score < 0.02:
    result.status = "no_change"  # also does not apply

Two independent guards. Both must pass before any weight update applies.

VERDICT: TRUE. Ambiguous signal is rejected twice before touching configuration.

Claim 4: "Leaves behind a visible policy every time it changes"

Source: src/slop_detector/ml/self_calibrator.py, docstring line 17–18

Return CalibrationResult; optionally write to .slopconfig.yaml via --apply-calibration

When --apply-calibration is passed and status == "ok", optimal weights are written to .slopconfig.yaml. Plain-text YAML. Human-readable. Git-versionable. Every calibration change is a diff.

VERDICT: TRUE. The policy artifact is explicit. You can git blame it.

Claim 5: "Explicit limits govern calibration"

Source: src/slop_detector/ml/self_calibrator.py, lines 37–54

MIN_W: float = 0.10             # minimum allowed weight per dimension
MAX_W: float = 0.65             # maximum allowed weight per dimension
MAX_PURITY_WEIGHT: float = 0.25 # purity ceiling
DOMAIN_TOLERANCE: float = 0.15  # max per-dimension deviation from domain anchor
DOMAIN_DRIFT_LIMIT: float = 0.25 # warn when optimal weight drifts this far
GRID_STEP: int = 20             # 0.05 increment resolution

No ML model. No learned bounds. Every constraint is a named constant with a comment explaining why it exists. The calibration space is a bounded grid, not an open optimization landscape.

VERDICT: TRUE. Every limit is auditable. Nothing is opaque.

Claim 6: "Detects empty implementations, phantom dependencies, disconnected pipelines"

These are the three canonical defect patterns AI code generation produces at scale. Each has a dedicated module.

Defect class	Implementation
Empty/stub functions	`src/slop_detector/metrics/ldr.py` — LDRCalculator detects `pass`, `...`, `raise NotImplementedError`, `TODO`
Phantom/unused imports	`src/slop_detector/metrics/hallucination_deps.py` — AST-based import vs usage analysis via `HallucinatedDependency` dataclass
Disconnected pipelines	`src/slop_detector/metrics/ddc.py` — DDC (Declared Dependency Completeness) usage ratio
Function clone clusters	`src/slop_detector/patterns/python_advanced.py` — Jensen-Shannon Divergence on 30-dim AST histograms, JSD < 0.05 = clone

The clone detection is worth noting. JSD on AST histograms catches structural duplication that string similarity misses entirely. LLMs produce a lot of this — same function logic, slightly renamed.

VERDICT: TRUE. Each defect class has a named module with a working implementation.

Claim 7: "~1.4K downloads in the past week"

Source: pypistats.org API (mirrors=false), queried 2026-04-15

last_week:  1,407  (mirrors excluded — actual pip install traffic)
last_month: 1,787
last_day:   83

"~1.4K" is within 0.5% of 1,407. Mirrors excluded means bot traffic is stripped — these are real install invocations.

VERDICT: TRUE. Verified against pypistats in real time. The number is not rounded up.

Why this format exists

Most open-source project posts make claims. Few back them up with file paths and line numbers.

That gap is the same problem AI-SLOP Detector is built to close. AI-generated code makes claims too — functions that look complete, imports that look used, pipelines that look connected. Static analysis finds the gap between what the code says and what it does.

This post applies the same standard to the project's own marketing copy. If a claim can be verified, it should be. If it can't, it shouldn't be made.

The codebase is public: github.com/flamehaven01/AI-SLOP-Detector

Pull requests welcome. Audits welcome more.

Verified by static code analysis + pypistats API, 2026-04-15

It Gets Smarter Every Scan: AI-SLOP Detector v3.5.0 and the Self-Calibration Loop

Kwansub Yun — Mon, 13 Apr 2026 16:15:49 +0000

Previously: 🔻v3.1.0 — Three Formula Refinements and the Adversarial Tester That Found Them ·
🔻v2.9.0/v2.9.1 — The Tool That Turned On Itself

By late 2025, everyone was building with AI. A weekend was enough to launch a SaaS app, and by Monday it was already on Product Hunt. The code looked finished, the UI worked, and the demo landed. That was also the problem.

In 2026, some of the consequences started arriving in public. Exposed databases, weak security boundaries, brittle automation, and production systems that looked polished enough to ship but had clearly not been understood at the level their surface confidence implied. Not every one of those failures belongs to static analysis, and it would be too easy to pretend otherwise. But many of them still point to the same upstream condition: code that looks complete long before it deserves trust.

That is the layer this release is about.

The breach is the headline. The review gap is the story.

A missing security rule is not the same thing as a stubbed auth function. A runtime-only bug is not the same thing as a phantom import.

A broken architecture is not the same thing as a buzzword-heavy helper. These are different failure classes, and any serious tool has to respect that difference.

What they often share, though, is the review environment that let them through. AI increased output volume, increased speed, and increased surface polish.

Review depth did not increase with it. That matters because AI-generated code has a very recognizable habit: it often looks complete before it is complete.

It compiles. It passes tests. It sounds like it knows what it is doing. Then you open the function.

def calculate_quality_score(data: Dict[str, Any]) -> float:
    """
    Advanced multi-dimensional quality assessment using
    proprietary algorithms with statistical normalization,
    entropy-based weighting, and dynamic threshold calibration.
    Returns a score between 0 and 100.
    """
    # TODO: implement the actual algorithm
    return 85.0

This is not noisy code. It is confident emptiness. In an analytics path, it becomes false certainty. In a payment path, it becomes a defect. In an auth path, it becomes risk.

The issue is not that AI writes ugly code. The issue is that AI reliably produces code that is structurally plausible while functionally thin.

That is a narrower claim than “AI is dangerous,” but it is also far more useful.

We ran into this ourselves

This did not begin as a theory about other people’s repos. It began when we found a flaw in our own scoring model. Back in v2.8.0, we discovered that our formula was accidentally rewarding spaghetti code.

A large god function could sometimes look healthier than a small clean function because complexity was dividing the penalty instead of amplifying it.

That was backwards, so the math changed.

AI-SLOP Detector now evaluates four dimensions:

LDR for logic density
Inflation for jargon density relative to real logic
DDC for dependency usage rather than dependency presence
Purity for critical structural defects that should drag the whole score down

These are combined with a weighted geometric mean, not an arithmetic average.

Why that matters:

one strong-looking axis should not be able to hide a collapsed one
a polished docstring should not rescue empty logic
if one important dimension fails, the whole score should feel it

That is the scoring philosophy underneath the tool. But even that was not enough.

Static analyzers have a threshold problem

Take a perfectly legitimate ML helper:

def prepare_training_batch(
    raw_samples: List[Dict[str, Any]],
    tokenizer: PreTrainedTokenizer,
    max_length: int = 512,
) -> Dict[str, torch.Tensor]:
    """
    Tokenize and pad samples for transformer training.
    Handles attention mask generation and HuggingFace
    tokenizer conventions for batch encoding.
    """
    return tokenizer(
        [s["text"] for s in raw_samples],
        padding=True,
        truncation=True,
        max_length=max_length,
        return_tensors="pt",
    )

There is nothing wrong with this code. But a generic detector may still overreact, because terms like tokenizer, attention mask, and HuggingFace can look suspicious if the analyzer does not understand the domain it is scanning. In a real ML codebase, those terms are normal. In a CRUD backend, some of them may be genuine anomaly signals.

That is the threshold problem. The same threshold can be wrong in one codebase and exactly right in another. A universal threshold sounds elegant, but real repositories are local. They have habits, idioms, and boilerplate that are legitimate inside one domain and suspicious inside another.

So the next problem became obvious: the tool had to learn the project it was scanning. That is the real center of v3.5.0.

What AI-SLOP Detector actually does

AI-SLOP Detector is a static analyzer built to catch a specific defect class that shows up repeatedly in AI-generated code: unimplemented stubs, disconnected pipelines, phantom imports, clone-shaped emptiness, placeholder-heavy production paths, and jargon inflation that outruns the actual logic. It is not a style linter, not a full security scanner, and not a runtime verifier. It is a detector for structural hollowness.

That distinction matters because it keeps the claim honest. The tool is not trying to solve every production risk. It is trying to catch one layer that becomes more expensive as AI output scales faster than human review.

pip install ai-slop-detector
slop-detector --init
slop-detector --project.

The workflow is the product story

What makes this release interesting is that it is not just “more patterns” or “more language support.” It is a workflow story.

The detector now has a real loop. It scans the file, classifies its role, computes the 4D score, applies structural pattern penalties, and writes the result to history. Then, once enough repeated scans exist, it revisits that history, extracts behavioral signals, tunes the weights inside bounded domain-aware limits, updates the configuration, and keeps scanning. That is the release.

That final stretch is what changed this from “detector upgrade” into “adaptive detector.” The tool no longer only evaluates code. It also learns from what happens after evaluation.

Self-calibration is the real headline

Every scan is recorded to a local SQLite history database. That history is not just there for reporting. It becomes the signal surface for the next tuning step. Once enough repeated scans accumulate, the detector begins asking a simple question: when this file was flagged, what happened next?

That produces two behavior-derived event types. An improvement event means the file was flagged, later changed, and its deficit dropped meaningfully. A false-positive candidate means the file was flagged, then scanned again with the same content and little meaningful score movement.

That difference is more important than it sounds. A lot of “self-improving” systems quietly learn from their own outputs. They mark something suspicious, then later use that same judgment as the truth signal for tuning. The system becomes better at agreeing with itself. That is not calibration. That is self-imitation with cleaner packaging.

v3.5.0 tries to avoid that trap. Its labels are not taken from the scoring formula. They are inferred from developer behavior around repeated scans. The formula says, “this looks suspicious.” The next run reveals whether a human treated that suspicion as real.

That signal is not perfect. An unchanged file is not always a false positive. It may be legacy code, low priority, or simply out of scope. But it is still a healthier signal than teaching the formula to imitate its own prior outputs.

What the loop actually looks like

The loop is not mystical. It is mechanical. Repeated scans accumulate, improvement and likely-FP events are extracted, candidate weight sets are evaluated, the search is bounded around the project’s current domain anchor, and if a strong enough winner appears, the config gets updated. If a calibrated weight drifts too far from the domain anchor, the system emits a warning.

This is what makes the title true. It gets smarter every scan, not because a hidden model is hallucinating taste, but because repeated use creates a bounded feedback loop. That is much less magical, and much more trustworthy.

Why `--init` matters more now

There is another reason the calibration story works better in v3.5.0. The detector no longer starts from a generic nowhere. --init now performs domain-aware bootstrap, detects the likely project type, and seeds the starting weights accordingly. That means calibration starts near the right neighborhood instead of wandering across the whole map.

That improves the first week of use, not just the tenth. And that matters, because bad first impressions kill adaptive tools. If the detector is only smart after a month of annoying you, it will never survive long enough to get smart.

Good initialization is not a convenience feature. It is part of whether the loop can gather clean signal at all.

JS, TS, and Go are not side quests

v3.5.0 also expands analysis coverage to Go, JS, JSX, TS, and TSX. That is useful on its own, but the deeper significance is architectural. Structurally hollow AI-generated code is not a Python-only phenomenon. If the detector’s long-term direction is project-local calibration rather than one-size-fits-all scoring, then wider language support is not a side feature. It is the natural expansion of the same idea.

Different languages. Same review gap. Same loop.

The honest boundary

This tool still does not close every gap. It will not fix missing infrastructure controls, catch every runtime bug, prove the architecture is correct, or replace security review. A clean structural profile is not proof of safety.

What it can do is narrow one expensive blind spot: the distance between code that looks finished and code that carries enough actual logic to deserve confidence. That is a smaller claim than “AI risk solved,” but it is also the kind of claim that survives production better.

Why this matters now

AI has made software generation dramatically cheaper. It has not made understanding cheaper. That difference is where governance debt begins to accumulate.

If teams can now generate far more code than they can truly review, then the review stack needs tools that operate below style and above syntax. Not tools that ask whether the code is pretty, but tools that ask whether the implementation carries enough substance for the confidence wrapped around it.

That is the space AI-SLOP Detector is trying to occupy. Not the whole problem. Just one layer that became impossible to ignore.

Quick start

pip install ai-slop-detector
cd my-project/
slop-detector --init
slop-detector --project .

Fix what clearly deserves fixing. Leave legitimate idioms alone. Then keep scanning. If the loop is doing its job, the next pass should know your codebase a little better than the first one did.

GitHub: flamehaven01/AI-SLOP-Detector

Can AI Review Physics? Yes — That Is Why We Built SPAR

Kwansub Yun — Sun, 12 Apr 2026 09:34:43 +0000

A standalone review framework for checking whether outputs deserve the claims attached to them.

Most review systems answer a familiar question:

Did the system still produce the expected output?

SPAR is built for a narrower and more dangerous one:

Does the output still deserve the claim attached to it?

That is the split. Not reliability alone, but admissibility.

In practical terms, admissibility means claim-worthiness: whether a result justifies the interpretation, governance status, or scientific statement built on top of it.

A system can be reliable and inadmissible at the same time.

A physics engine can compute beta_G_norm, return zero, pass regression, and stay green across the whole pipeline. The report can still say the background is admissible. But if the function producing beta_G_norm is a stub that always returns zero, the output is stable while the claim attached to it is false.

That is not hypothetical. It is one concrete class of review failure SPAR was designed to surface.

What SPAR Is

SPAR (Sovereign Physics Autonomous Review) is a deterministic framework for claim-aware review.

It does not replace unit tests. It does not replace regression benchmarks. It does not replace scoring systems. It reviews a different object:

the output
the claim attached to that output
the implementation state behind it
the maturity state that should travel with it

SPAR started inside Flamehaven-TOE, an open physics simulation and AI governance engine. It has since been extracted into a standalone open-source framework:

github.com/flamehaven01/SPAR-Framework

The framework includes a generic review kernel, explicit score and verdict policy, registry-backed review surfaces, and a first domain adapter for physics. Physics is the first adapter and the first domain where this review model was stress-tested. It is not the limit of the framework.

The core idea is simpler than the name: an output can pass while the claim attached to it drifts.

Why Ordinary Review Is Not Enough

Ordinary review is usually shallow by necessity. It asks questions like:

did the code run
did the output shape stay valid
did the score remain within bounds
did regression remain green

Those are necessary checks. They are not always enough.

The failure SPAR cares about is not, in the first instance, a crash. It is not even always a wrong number. It is this:

the code executes
the output looks plausible
the tests pass
and the interpretation is still overstated or structurally false

That failure can appear in several ways: a placeholder implementation returns stable-looking values; a maturity registry stays stale after the implementation improves; a score looks smooth before its epistemic basis is strong enough to justify the interpretation attached to it; an approximation gets reported as closure.

None of these failures is spectacular. That is exactly why they are easy to miss.

A Minimal Divergence

The clearest way to see the difference is in review form.

{
  "ordinary_regression": {
    "kernel_exec": "PASS",
    "output_contract": "PASS",
    "test_suite_status": "GREEN"
  },
  "spar_review": {
    "layer_a": "CONSISTENT",
    "layer_b": "CONSISTENT",
    "layer_c": "GAP_STATE_MISMATCH",
    "finding": "Implementation path is genuine; registry classification remains stale",
    "required_action": "RECLASSIFY"
  }
}

Ordinary regression says the system still works. SPAR says the system may no longer be describing its own computation truthfully.

SPAR is not "tests, but harsher." It can produce a different review outcome even when ordinary regression remains green. In this case, the required action is not rejection. It is reclassification. That is not the same as testing harder. It is reviewing a different object.

The Core Mismatch Classes

SPAR treats three mismatch classes as first-class review objects.

Anchor mismatch — the output conflicts with a declared analytical or contractual anchor.

Interpretation mismatch — the report language claims more than the implementation state justifies.

Maturity mismatch — the implementation, registry, and outward-facing claim have drifted out of sync.

Ordinary review mostly checks whether a system still passes. SPAR checks whether the result is still being described honestly.

The Three-Layer Structure

Layer A — Anchor Consistency

Layer A checks whether output agrees with a declared analytical or contractual anchor. The expected value is not "whatever the engine produced last time." It is "what the declared contract says must appear for this background, under this formulation."

# Flat Minkowski: beta residuals must vanish
status = CONSISTENT if abs(beta_G(output)) <= 1e-4 else ANOMALY

# de Sitter: admissibility gate must FAIL
# A PASS here indicates a gate defect, not a success.

Layer A tests agreement with a declared reference contract — not truth in some unconstrained universal sense. Analytical anchors depend on regime, normalization, and formulation. That distinction matters. Still, the engineering value is clear: reliability can remain intact while anchor-consistency fails. A Layer A anomaly means the output contradicts the contract the system claims to be using.

Layer B — Interpretation Validity

Layer B checks whether the interpretation attached to the output stays within declared scope. This layer is deterministic — it does not rely on a free-form LLM judge. It uses explicit rule tables over structured runtime artifacts, maturity states, and report text.

Typical checks: does the report claim full closure while the path is still heuristic or partial; is a bounded approximation being described as exact; is an environment-conditional bridge being written up as universal; are overclaim phrases appearing where runtime state does not support them.

Layer B does not eliminate semantic ambiguity. What it does is narrow the problem from "solve rhetoric in general" to "enforce explicit admissibility contracts against declared model states." That makes it auditable. Not complete. Auditable.

Layer C — Existence and Maturity Probes

Layer C asks what kind of implementation produced the result: genuine, approximate, gapped, environment-conditional, or research-only.

This is where SPAR becomes especially different from ordinary review. It does not merely score outputs. It checks the ontological status of the path that produced them. A result from a known-limited path is not the same thing as a result from a genuine path. A research probe is not production-grade closure. A dependency-bound bridge is not a universal capability. Those distinctions change what the output is allowed to claim.

Why the Registry Matters

A framework like this needs more than score outputs. It needs structured state that can travel with the result.

ARCHITECTURE_GAPS = {
    "C4_sidrce_omega": (
        "SIDRCE Omega (v4.5+): chi-squared Gaussian model. "
        "score = exp(-chi2/2), chi2 = (||beta||/tol)^2. "
        "Derived from zero-centered Gaussian likelihood. "
        "GAP: tolerance values are calibration parameters with "
        "no first-principles derivation of their exact magnitude."
    ),
    "C8_rg_linearized": (
        "RG flow: linearized dilaton ODE only. Metric does NOT flow. "
        "APPROXIMATION: valid only for small perturbations "
        "around a fixed background geometry."
    ),
}

A machine-readable registry turns caveat prose into runtime surface. That lets review results carry explicit maturity labels — open, partial, closed, environment_conditional, research_only — rather than vague prose. Without that surface, approximation and closure collapse into the same sentence.

Scoring Policy Is Explicit

SPAR keeps score policy visible. No hidden learned weights.

SCORE_TABLE = {
    "ANOMALY":       -15,   # contradicts an analytical anchor
    "FAIL":          -10,   # review-layer failure
    "GAP":            -5,   # honest gap disclosure
    "WARN":           -3,   # bounded concern
    "APPROXIMATION":  -2,   # known simplification
}

def journal_verdict(score, layer_a):
    if count_anomalies(layer_a) >= 2:  return "REJECT"
    if score >= 85:                    return "ACCEPT"
    if score >= 70:                    return "MINOR REVISION"
    if score >= 50:                    return "MAJOR REVISION"
    return "REJECT"

These are not laws of nature. They are review policy. A hidden learned scorer may feel sophisticated. An explicit policy is easier to inspect, debate, and change.

Two or more Layer A anomalies trigger unconditional REJECT regardless of total score. Mathematical contract failures are not averaged away by cleaner signals elsewhere.

A Concrete Example: The Omega Score Transition

Flamehaven-TOE's primary governance metric, SIDRCE Omega, once relied on a large arbitrary multiplicative constant applied to the raw residual. The outputs looked stable. Nothing felt obviously broken.

SPAR still flagged it. Stability was not the right question. The stronger question was whether the formula justified the interpretation being attached to it. A free scaling constant with no physical derivation is not the same thing as a physically motivated model.

The formula was replaced with a chi-squared Gaussian construction:

gs_score = math.exp(-0.5 * (beta_result.beta_G_norm / tol_G) ** 2)
b_score  = math.exp(-0.5 * (beta_result.beta_B_norm  / tol_B) ** 2)
si_score = math.exp(-0.5 * (beta_result.beta_Phi_norm / tol_Phi) ** 2)

omega_physics = (gs_score * b_score * si_score) ** (1/3)

That change matters because it introduces a reversible relation to the underlying residual:

||beta|| = tol * sqrt(-2 * ln(score))

Given a reported score, the residual is recoverable. The score is no longer just a presentation layer. It encodes a falsifiable relation to the quantity beneath it.

SPAR did not respond by declaring the problem solved. It updated the classification precisely: the formula ceased to be arbitrary, but the remaining gap shifted to the tolerance scales, which are still calibration parameters. That is a narrower and more honest claim than either "arbitrary" or "fully resolved."

That is exactly the kind of distinction SPAR is built to review.

Quick Start

pip install -e .[dev]

from spar_framework.engine import run_review
from spar_domain_physics.runtime import get_review_runtime

runtime = get_review_runtime()

result = run_review(
    runtime=runtime,
    subject={
        "beta_G_norm": 0.0,
        "beta_B_norm": 0.0,
        "beta_Phi_norm": 0.0,
        "sidrce_omega": 1.0,
        "eft_m_kk_gev": 1.0e16,
        "ricci_norm": 0.02,
    },
    source="flat minkowski",
    gate="PASS",
    report_text="Bounded report text.",
)

print(result.verdict)
print(result.score)
print(result.model_registry_snapshot["total_models"])

The review result carries more than pass/fail: a verdict, a score, and a maturity-aware review surface. That surface is what makes the output governable rather than just evaluable.

Where This Framework Fits

Physics remains the first adapter and strongest early testbed. The review pattern is broader.

It fits anywhere outputs can pass while the attached claim can drift: scientific computing pipelines, PDE and simulation workflows, scientific ML surrogates, inverse and calibration models, AI code review, model governance, regulated analytics and reporting.

That does not mean every team needs the full framework. Often the first useful step is smaller.

A Lightweight Adoption Path

Level 1 — Claim Check. Add three explicit questions to an existing workflow: What is the output actually claiming? Does that claim match the implementation state? Is this result exact, approximate, partial, or heuristic? Most teams can do this immediately with no new tooling.

Level 2 — Maturity Labels. Attach state labels to results: heuristic, partial, closed, environment_conditional. A small registry. Already a meaningful step beyond ordinary review.

Level 3 — Full SPAR. Layer A anchor consistency, Layer B interpretation validity, Layer C existence and maturity probes, registry-backed snapshots, explicit score and verdict policy.

SPAR can be used as a review habit before it is adopted as a full framework.

What SPAR Does Not Do

SPAR does not provide a universal truth engine, free-form LLM judging in the core, domain contracts inside the generic kernel, or certainty about whether a scientific claim is true in all possible senses.

SPAR is not a machine for declaring truth. Its narrower goal is to make claim drift reviewable.

Reliability Is Not Enough

Reliability asks whether a system produces stable, repeatable outputs. Admissibility asks whether those outputs deserve the meanings attached to them.

A stub that always returns zero can be reliable. A heuristic threshold can be reliable. A smoothly calibrated score can be reliable. None of those facts alone makes the resulting claim justified.

Current AI and scientific tooling are already better at measuring reliability than admissibility. That asymmetry is understandable — reliability is easier to benchmark, easier to automate, easier to ship in CI. But admissibility is where silent approximations, overstated claims, and maturity mismatches accumulate.

SPAR is one working answer to that problem. Not a universal answer. A technical one.

It turns implementation state, maturity state, analytical anchoring, and scope honesty into review objects that can travel with the result.

That is why the architecture may matter outside the domain that produced it.

Repository: github.com/flamehaven01/SPAR-Framework

Docs: What Is SPAR · Admissibility · Physics as the Proof Case · Use Cases

AI SLOP Detector v3.1: Three Formula Refinements and the Adversarial Tester That Found Them

Kwansub Yun — Thu, 09 Apr 2026 05:29:57 +0000

We shipped v2.9.0 with a scoring engine we trusted. We ran tests. Everything passed.

Then we built a tool specifically designed to find cases where the score was less precise than it could be — and it found three.

This is the story of v3.1.0. And the patch that followed six hours later.

Glossary — internal terminology used throughout this post

Term	What it means
Deficit score	The final output of the scorer. 0 = structurally clean, 100 = critical. Derived as `100 × (1 - GQG)`.
GQG	Geometric Quality Gate. A weighted geometric mean of LDR, Inflation quality, DDC, and Purity. The single formula the scorer evaluates.
LDR	Logic Density Ratio. Ratio of executable logic lines to total lines. Low LDR = file is mostly stubs, blanks, or comments.
Inflation	Metric that flags jargon-heavy docstrings unsupported by actual code complexity. A 2-line function with a 30-line docstring using 12 buzzwords scores badly.
DDC	Dead/Duplicate Code ratio. Tracks unreachable paths, copy-pasted blocks, phantom imports.
Purity	Pattern hit rate. How many structural anti-patterns (god functions, stub returns, nested complexity) fire on the file.
Cyclomatic Complexity (CC)	Count of independent code paths. A straight-line function = CC 1. Each `if`, `for`, `while`, `except` adds 1.
fhval	flamehaven-validator. An external tool that interrogates the scorer from outside the codebase. Its purpose is to catch cases where internal test consistency masquerades as correctness.
SPAR	Subcommand of fhval. Adversarial regression loop with three layers. Tests whether the scorer measures what it claims to measure.
JSD	Jensen-Shannon Divergence. A symmetric, bounded (0–1) measure of divergence between two probability distributions. Used here to compare AST node-type histograms between functions.
AST	Abstract Syntax Tree. The parsed structure of source code. An `if` statement, a `return`, a function call each become typed nodes.
function_clone_cluster	New pattern in v3.1.0. Detects files where many functions share near-identical AST structure — the fragmented god function evasion pattern.
placeholder_variable_naming	New pattern in v3.1.0. Detects vocabulary-clean code with zero semantic content: single-letter parameter floods, sequential numbered variables.
AM/GM gap	Core refinement in v3.1.0. The calibrator used an arithmetic mean (simpler approximation); the scorer uses a geometric mean (the precise target formula). Aligning them closes a ~5-7pt estimation gap on uneven files.

Quick context

AI SLOP Detector is a static analyzer that measures structural code quality — not style, not formatting. It scores each file across four dimensions and assigns a deficit between 0 (clean) and 100 (critical):

Dimension	What it measures
LDR	Ratio of executable logic to total lines
Inflation	Jargon, docstring bloat, unsupported claims
DDC	Unreachable paths, copy-pasted blocks
Purity	Pattern hit rate (stubs, god functions, etc.)

These four numbers feed a single formula — a weighted geometric mean — called the GQG. The output is the deficit score: 100 × (1 - GQG).

The calibrator's job is to find the best weights for that formula by searching over thousands of known cases.

Before v3.1.0: the self-scan

We don't ship a version without running the detector against itself. Before cutting v3.1.0, we ran v3.0.3 — a structural debt reduction pass on the three highest-deficit files in the codebase.

Self-scan before: avg_deficit=23.57, 15 deficit files, status=suspicious
Self-scan after:  avg_deficit=20.33, 12 deficit files, status=clean

analysis/cross_file.py dropped from 70.3 to 28.7 (critical → clean). ci_gate.py from 69.3 to 22.3. cli.py from 68.4 to 20.9. The fixes were mechanical: extracted nested closures to private methods, replaced if/elif/else dispatch chains with dict dispatch, removed re-declared constants.

The point is not that these numbers are good. It's that the tool had to earn its own PASS before we shipped the version that refines the formula. Shipping a scoring engine while your own codebase sits at suspicious would have been its own kind of slop.

The adversarial tester: fhval SPAR

In a previous post we described fhval — flamehaven-validator. The core concern: when every tool in an ecosystem is built by the same person against the same baseline, internal consistency can masquerade as correctness. Passing your own tests proves nothing about whether your tests are asking the right questions.

For v3.1.0 we added a spar subcommand — an adversarial regression loop that interrogates the scorer from the outside. Running SPAR against the v3.0.x scorer:

SPAR score: 55 / 100  [FAIL]

Layer A anomalies:
  A3 stub_class_8_methods     expected >= 30  got 20.0  [ANOMALY]
  A4 fragmented_god_function  expected >= 10  got  0.0  [ANOMALY]
  A5 vocab_clean_meaningless  expected >=  8  got  0.0  [ANOMALY]

Layer C blind spots:
  C2 inflation_blindspot      [BLIND_SPOT]
  C3 ddc_annotation_gap       [BLIND_SPOT]

Three gaps. Two documented scope limits. Score: 55 FAIL.

Each gap pointed at a specific detection weakness. The SPAR methodology itself — how Layer A/B/C work, why adversarial ground truth is hard to author from inside the codebase — is a separate topic covered in tomorrow's post. Here we focus on what the gaps told us and what we changed.

Refinement 1: The calibrator and scorer were using different formulas

The scorer computes a weighted geometric mean. The calibrator — which finds optimal weights — was computing a weighted arithmetic mean as its optimization target.

Those are not the same thing, and for a quality gate, the difference is structural.

Consider a file with three dimension scores: LDR=0.9 (good), inflation_quality=0.1 (very bad), DDC=0.8 (good).

Formula	Calculation	Result	Deficit
Arithmetic mean	(0.9 + 0.1 + 0.8) / 3	0.60	40
Geometric mean	(0.9 × 0.1 × 0.8) ^ (1/3)	0.42	58

The arithmetic mean gives deficit=40. The geometric mean gives deficit=58. The gap is 18 points — not rounding, but structural. The geometric mean amplifies weak dimensions because one bad score pulls the entire product down. The arithmetic mean averages over them.

The scorer uses the geometric mean for good reason: a file with excellent LDR but zero actual logic (all docstrings) should not score deficit=30. It should score much higher. The formula enforces that.

The first-generation calibrator used an arithmetic mean as a simpler starting approximation. So it was finding weights that minimize error against a different objective than the scorer actually computes. The result: roughly 5–7 point underestimation on files with uneven dimension profiles — which are precisely the target of this tool.

The AM ≥ GM inequality means the calibrator's scores were always optimistic. For balanced files (all dimensions similar) the gap is small and harmless. For uneven files, it was systematic — and those are the cases that matter most.

Refinement:

# Before (calibrator _recompute_deficit)
quality = (w_ldr * ldr + w_inflation * (1 - inflation_n) + w_ddc * ddc) / total_w

# After — mirrors the scorer's GQG formula exactly
from math import exp, log
gqg = exp(
    (
        w_ldr * log(max(1e-4, ldr))
        + w_inflation * log(max(1e-4, 1.0 - inflation_n))
        + w_ddc * log(max(1e-4, ddc))
    )
    / total_w
)
deficit = min(100.0, 100.0 * (1.0 - gqg))

This is why SPAR anomaly A3 (stub_class_8_methods) jumped from deficit 20.0 to 40.0: the stub class had heavily uneven dimensions, and the geometric mean scored it correctly once the calibrator was trained against the right target.

Refinement 2: The complexity modifier had a dead zone at the common end

The inflation metric applies a complexity modifier to penalize functions that are simultaneously simple and jargon-heavy — a common pattern in AI-generated code: a two-line function surrounded by an elaborate docstring.

The first-generation modifier formula:

# Before
complexity_modifier = max(1.0, 1.0 + (avg_complexity - 3.0) / 10.0)

For CC=1: 1.0 + (1-3)/10 = 0.8 → max(1.0, 0.8) = 1.0
For CC=2: 1.0 + (2-3)/10 = 0.9 → max(1.0, 0.9) = 1.0
For CC=3: 1.0 + (3-3)/10 = 1.0 → max(1.0, 1.0) = 1.0

CC=1, 2, and 3 all received the same modifier: 1.0. This meant simple functions — the three most common complexity levels — paid no complexity premium on inflation, regardless of how jargon-heavy they were. The modifier only activated from CC=4 upward.

Simple jargon-heavy functions are the most common AI code signature. The formula was least sensitive precisely where it needed to be most sensitive.

# After — CC=1 is the baseline, not CC=3
complexity_modifier = max(1.0, 1.0 + (avg_complexity - 1.0) / 10.0)

Now CC=2 gets a 1.10× modifier, CC=3 gets 1.20×. The penalty scales from the simplest meaningful function upward.

Refinement 3: Purity weight was documented but not connected

The GQG formula includes a purity dimension:

# Before
w_pur = 0.10  # hardcoded constant
final_score = gqg_score * (1.0 - w_pur * purity_penalty)

.slopconfig.yaml had a weights.purity field. The calibrator's weight search had a purity parameter. Neither was connected to this constant — users could configure weights.purity: 0.20 and nothing would change.

# After
w_pur = weights.get("purity", 0.10)  # default unchanged; now configurable

One line. The config surface now matches the implementation.

Two new detection patterns

Stub evasion: empty container returns

The existing return_constant_stub pattern caught return True, return 0, return "string" — but not return {}, return [], return (), return set(). These are equally common stub patterns in class skeletons:

class DataProcessor:
    def get_results(self) -> dict:
        return {}  # was not flagged before

    def list_items(self) -> list:
        return []  # was not flagged before

Both are now caught by return_constant_stub and interface_only_class.

Fragmented god function: AST clone detection

SPAR anomaly A4 was a file with 12 one-liner helper functions:

def _compute_r1(x): return x * 1.1
def _compute_r2(x): return x * 1.2
def _compute_r3(x): return x * 1.3
# ... through r12

Each function individually looks clean: low complexity, no nesting, short. No single function exceeds any per-function threshold. But collectively, this is a decomposed god function — a large computation split into structurally identical fragments that evade per-function gates.

The new pattern: function_clone_cluster.

How it works. For each file, build a 30-dimensional histogram of AST node types for every function: how many If nodes, Return nodes, Call nodes, BinOp nodes, and so on. The histogram is normalized to a probability distribution. Then compute pairwise Jensen-Shannon Divergence between all function pairs. JSD is bounded between 0 and 1. Two functions with near-identical AST structure produce JSD close to 0.

Functions with JSD < 0.05 get an edge in a graph. BFS finds connected components. The largest component is the clone cluster.

Thresholds:
  >= 6 functions in cluster: CRITICAL
  >= 4 functions in cluster: HIGH

Why JSD and not simpler metrics. Cosine similarity or Euclidean distance on raw histograms don't handle sparse distributions well — short functions have mostly empty histograms, and small absolute differences dominate. JSD compares distributions rather than raw vectors, stable when most histogram dimensions are near zero. It also has an upper bound of 1, which makes the 0.05 threshold interpretable rather than dataset-dependent.

The JSD threshold (0.05) was calibrated against the internal test corpus. It will produce false positives on files with many similar utility functions — for example, a large set of _validate_field_X() validators that are structurally identical by design. Adjust via --config if needed.

Placeholder variable naming (v1.0)

SPAR anomaly A5 was vocabulary-clean code with zero semantic content:

def aggregate(a, b, c, d, e, f, g):
    r1 = a + b
    r2 = r1 * c
    r3 = r2 - d
    # ... through r12
    return r12

No buzzwords. No docstring bloat. Every traditional linter passes this. The new placeholder_variable_naming pattern applies two checks:

Single-letter parameter density: 5 or more single-letter parameters (excluding self, cls, _) → HIGH.
Sequential numbered variables: a run of 8 or more → HIGH; 4 or more → MEDIUM.

This is v1.0: it detects naming style, not semantic quality. Known false positive zone: scientific and math libraries legitimately use single-letter conventions (x, y, z, mu, sigma). Suppress with domain_overrides in .slopconfig.yaml.

SPAR result after v3.1.0

SPAR score: 85 / 100  [PASS]

Layer A: 5/5 anchors consistent
Layer B: 4 documented limitations (no regressions)
Layer C: C2 inflation_blindspot [BLIND_SPOT — known scope limit]
         C3 ddc_annotation_gap  [BLIND_SPOT — known scope limit]

55 → 85 PASS.

The two remaining blind spots are not gaps to close — they're the documented scope limits of static analysis: a tool that reads AST cannot determine whether arithmetic is semantically meaningful, or whether annotation-heavy imports serve a real runtime purpose. Those require a different class of model. Documenting the ceiling is part of the job.

The full SPAR methodology — how Layer A/B/C work, why Layer A ground truth is hard to author from inside the codebase, and what "validating the validator" means in practice — is covered in tomorrow's post.

v3.1.1: the self-inspection patch

v3.1.0 and v3.1.1 shipped on the same day. The clone detection pattern introduced in v3.1.0 had a visibility gap: function_clone_cluster fired in the Issues section but produced no signal in the Core Metrics table. A community issue caught it within hours.

But before cutting v3.1.1, we ran the tool against itself — and the new patterns found something:

placeholder.py    deficit: 70.3  [CRITICAL_DEFICIT]
python_advanced.py  deficit: 74.0  [CRITICAL_DEFICIT]

Both files are part of the detection engine itself. Root cause: check_node methods with cyclomatic complexity 20–31, caused by compound boolean logic that had accumulated across releases. The tool was flagging its own pattern implementations as having the exact complexity problems it was designed to detect.

We extracted four module-level helpers in placeholder.py (_strip_docstring, _has_abstractmethod, _empty_container_repr, _is_placeholder_stmt) and added _make_god_issue() and _collect_numbered_vars() to python_advanced.py. Each check_node method went from 20–70 lines to 8–15. The detector earned its own PASS before shipping the patch.

placeholder.py      70.3 → 43.7  [CRITICAL → SUSPICIOUS]
python_advanced.py  74.0 → 66.7  [CRITICAL → INFLATED_SIGNAL]

Additional v3.1.1 refinements:

Clone Detection row added to Core Metrics table (CRITICAL/PASS at a glance).
Table style unified to box.ROUNDED across all project output (was mixing three styles).
VS Code extension: extractJson() strips [INFO] log lines before JSON.parse — previously caused silent parse failures when CLI log output appeared alongside JSON. Workspace analysis replaced with a QuickPick list of deficit files sorted by score; clicking opens the file in the editor.

If you installed 3.1.0, upgrade to 3.1.1 before using clone detection in CI.

How this fits alongside existing tools

Tool	Approach	What it sees that others don't
Semgrep	Pattern-matching on AST	Rule violations you've pre-authored
SonarQube	Cognitive complexity, duplication, coverage	Complexity, coverage gaps — not structural emptiness
Radon	Cyclomatic complexity	Raw CC values; used internally by AI SLOP Detector
Bandit	Security rules	Security vulnerabilities
mutmut / cosmic-ray	Mutation testing	Whether your test suite catches real bugs
AI SLOP Detector	Metric-based structural analysis	Docstring theater, stub pipelines, fragmented logic, phantom imports

The key gap: a file can be fully SonarQube-clean while containing zero actual logic — all stubs, all docstrings, all type annotations. Cognitive complexity doesn't measure whether the complexity is real. LDR does. Inflation does.

The complementary tool here is mutation testing. SPAR tests whether the scorer measures what it claims. Mutation testing tests whether your tests catch what they claim to catch. Both are adversarial approaches to the meta-problem: how do you validate the validator?

Score evolution

If you're running AI SLOP Detector on an existing project, upgrading to 3.1.x will change your scores. The formula alignment in Refinement 1 increases deficit on files with uneven dimension profiles, typically by 3–8 points. This is not drift — it's the scorer becoming more precise in the region where it matters most. Files that were borderline suspicious may move into inflated_signal. Check your CI threshold after upgrading.

Previous scores were valid estimates produced by the first-generation model. v3.1.x scores are tighter estimates with better sensitivity where dimensions are uneven — which is precisely the profile of AI-generated code.

Honest limitations

function_clone_cluster threshold (JSD < 0.05) was calibrated against the internal test corpus. It will fire false positives on legitimate utility function clusters. Adjust via --config.
placeholder_variable_naming v1.0 has no semantic context. def distance(x, y, z) is legitimate; the pattern doesn't know that.
SPAR score 85 means five ground truth anchors pass and eight of ten Layer C probes hold. The space of evasion patterns is open-ended. More in tomorrow's SPAR post.
The Layer A corpus is internally authored. External adversarial contributions would make it stronger.

Install / upgrade

pip install ai-slop-detector==3.1.1
# or
pip install --upgrade ai-slop-detector

VS Code extension: search "AI SLOP Detector" in Extensions, or install from VSIX:

code --install-extension vscode-slop-detector-3.1.1.vsix

# Scan a project
slop-detector --project ./your-project

# Machine-readable output
slop-detector --project ./your-project --json | jq '.file_results[] | {file: .file_path, deficit: .deficit_score}'

GitHub: flamehaven01/AI-SLOP-Detector

Previous posts in this series:

My AI Maintainer Kept Making Wrong Calls. So I Made It Report Its State Before Touching Anything.

Kwansub Yun — Tue, 07 Apr 2026 17:24:43 +0000

🔎 Glossary: terms used in this article

🔸 MICA (Memory Invocation & Context Archive): A governance schema for AI context management. Defines how context should be structured, trusted, scored, and handed off across sessions.

🔸 memory_injection: A MICA operational mode. The archive is updated after each maintenance session and read by the next AI session to compensate for session amnesia.

🔸 Session Report Format: The structured opening output the model must produce at session start — declaring archive version, self-test result, drift status, and active invariants — before any repository-level work begins.

🔸 Self-Test Policy: Machine-evaluable checks that validate the archive against the real project state: file existence, hash integrity, and invocation protocol presence.

🔸 Drift Response Policy: The schema-level declaration of how hash mismatches and missing files are handled. Different failure classes carry different response actions.

🔸 Design Invariant: A structured governance rule with identity, severity, and statement. Not a style guideline. A constraint the model cannot rationalize past.

🔸 Provenance Registry: The record of tracked files with SHA256 hashes. The basis for drift detection.

🔸 Deviation Log: The audit trail of formal exceptions to design invariants. Empty means no exceptions have been logged — not that no judgment calls were made.

1. What Part 5 Left Open

Part 5 placed MICA inside the context engineering landscape and drew one boundary: MICA is not a collection system. It begins after collection ends. Its job is to govern what enters the session, what remains authoritative, and how the model proves it actually loaded the governed archive at all.

That answer was correct. It was also still abstract.

This post comes down from that framing. It shows what MICA looks like when it is actually running — not as a concept, but as a protocol inside a real project.

The project is flamehaven.space, a Next.js B2B site maintained by a solo operator using a MICA package in memory_injection mode. Everything shown here is drawn from the live archive. Values that would expose internal configuration are anonymized; structure and behavior are real.

2. The Session Opening Report

Every MICA session in memory_injection mode begins with a declared output before any work starts. The archive specifies the required format:

[SESSION READY]
Archive: flamehaven-space-maintainer v0.2.0
Self-test: PASS (3 checks -- ST-001, ST-002, ST-003)
Drift: no hash mismatch detected
Active invariants: DI-001 (critical), DI-002 (critical) + 24 others loaded
Gate: PASS -- proceeding with maintenance

This is not a courtesy summary. It is a gate. The archive field session_report_format.gate_block_on is set to critical_self_test_failure — meaning the model must declare its load state before it is permitted to make any repository-level changes.

The format is specified in the archive itself:

json
"session_report_format": {
  "trigger": "session_start",
  "required_fields": ["archive_version", "self_test", "drift_status", "active_invariants", "gate"],
  "format_template": "[SESSION READY]\nArchive: {archive_version}\nSelf-test: {self_test}\nDrift: {drift_status}\nActive invariants: {active_invariants}\nGate: {gate}"
}

The model does not decide what to declare. The archive tells it what a valid session opening looks like.

3. What the Self-Test Actually Checks

The self_test_policy runs on session_start and pre_handoff. Three checks matter here:

ST-001 (provenance_sha256_format, severity: error) — verifies that provenance hashes in the registry match the expected format. A malformed hash means the file fingerprint is untrustworthy.
ST-002 (provenance_file_exists, severity: warning) — verifies that files listed in the provenance registry actually exist on disk. A missing file is not a formatting error; it is a ghost reference.
ST-003 (invocation_pattern_present, severity: error) — verifies that the invocation protocol is declared and readable. If the model cannot confirm how it was loaded, it cannot confirm the session is in a governed state.

Failure behavior is set per-check. The overall on_failure policy for this archive is warn_continue — the session proceeds, but the failure is surfaced explicitly in the opening report.

This is a deliberate calibration. A site maintenance session that blocks hard on every provenance warning would be too brittle for solo operation. The severity ladder reflects the actual cost of each failure mode, not a theoretical maximum.

4. What Drift Detection Looks Like in Practice

The archive's drift_response_policy is minimal by design:

"drift_response_policy": {
  "on_hash_mismatch": "warn_continue",
  "on_file_missing": "warn_block",
  "reminder_after_change": true,
  "inline_sync_required": false
}

The distinction between warn_continue and warn_block is operationally significant.

A hash mismatch means a tracked file changed — which happens legitimately during ordinary maintenance. The model surfaces the mismatch and continues.

A file that has gone missing entirely is a different failure class. The model blocks and requires operator acknowledgment before proceeding.

reminder_after_change: true means the archive instructs the model to remind the operator to refresh the provenance registry and artifact manifest before minting the next archive version. This is not automated enforcement. It is memory injection: the archive tells the next session what the previous session should have reminded the operator to do.

The deviation_log in v0.2.0 is empty. That is not a sign the system has never been used. It means no deviations have been formally logged yet — which is itself a state the archive captures.

5. What Happens When a Deployment Changes Something

A concrete scenario: the operator ships a writing refresh mode change — switching from automatic ISR to manual operator-triggered revalidation. Three files change: next.config.ts, a helper .bat script, and the playbook.

On the next session open:

Self-test runs. ST-002 may flag if the helper script path is not in the provenance registry yet.
Drift check runs. Hash mismatches fire for the changed files. on_hash_mismatch: warn_continue — the session proceeds.
The opening report surfaces the mismatch:

[SESSION READY]
Archive: flamehaven-space-maintainer v0.2.0
Self-test: PASS with warnings (ST-002: update-writing-now.bat not in provenance registry)
Drift: hash mismatch on next.config.ts, flamehaven-space-maintainer-playbook.v0.2.0.md
Active invariants: DI-001 (critical), DI-002 (critical) + 24 others
Gate: PASS WITH WARNINGS -- operator review required

The operator now has a concrete decision surface before touching anything: what changed, what the model knows about, and what it does not.

The model then follows the change process defined in the playbook: identify the canonical subsystem touched, patch the smallest coherent surface, run build and audit, verify route-level behavior, then update README, MICA, or playbook if the change alters maintainer truth.

At the end of the session, if a new archive version is minted, the synchronization rule is explicit: file name, project.version, and the archive handoff marker must be updated in the same change. Not sequentially. The same change.

This is the operational point of drift reporting: not merely to announce that something changed, but to force the model and the operator to see the same changed surface before any new work proceeds.

6. What the Design Invariants Actually Govern

The archive carries 26 design invariants. The first six establish the perimeter everything else operates within:

DI-001 (critical): Flamehaven is positioned as a governance-first, founder-led B2B AI systems practice, not a generic agency or AI wrapper shop.
DI-002 (critical): Primary conversion surface is the main domain flamehaven.space, not Medium, Substack, DEV.to, or LinkedIn.
DI-003 (high): Writing detail pages are authoritative artifacts linked to projects, selected work, contact, and SEO canonical ownership.
DI-004 (high): Selected Work must distinguish public, private, and internal systems without broken public links or placeholder copy.
DI-005 (high): Legacy WordPress-era routes must redirect away from obsolete agency messaging.
DI-006 (high): Operational choices favor deterministic behavior, inspectability, and maintenance continuity over decorative complexity.

These are not style guidelines. They are session-blocking constraints. An AI that proposes converting Selected Work to a live-fetch real-time surface is violating DI-006. An AI that treats cross-posting as the canonical publishing path is violating DI-002. An AI that leaves a legacy route live because a redirect “seems unnecessary” is violating DI-005.

The invariants exist so the model cannot rationalize its way past the operator's architectural intent — even across sessions, even with a new model instance that has never seen the project before.

7. What MICA Cannot Do Here

The deviation_log is empty because no formal deviation has been logged. But there have been judgment calls.

One example is the writing hero image fallback logic. The site had to support both modern Notion block structures and legacy imported posts with a different nesting format. That decision did not begin as an invariant. It began as a session-level judgment call, became a playbook rule, and only then became stable maintainer truth.

That path matters.

MICA does not automate the step from “we discussed this and made a call” to “this is now a governed constraint.” It provides the place to put the result. The operator decides what rises to the level of an invariant, what remains a lesson in the playbook, and what disappears when the session ends.

That boundary — what gets governed, what gets remembered, what gets lost — is not a gap in MICA. It is a design decision made with every archive update.

8. What Part 7 Will Address

Part 6 showed what MICA looks like in operation inside a single maintenance agent. The structure holds. The protocol runs. The session report is predictable.

But that is still the easier case.

The project being governed was a site: a relatively stable artifact, maintained by one operator, where the main problem was making sure the model did not forget what already mattered.

Part 7 moves into a harder setting.

The governed project is now a tool that runs inside AI workflows itself. That changes the governance problem. The issue is no longer only session amnesia. It is iterative accumulation: what the system learns across cycles, what becomes authoritative, what remains provisional, and what must be carried forward without allowing drift to harden into false memory.

Part 7 is that case.

Named decision from this post: A session report is not a polite summary. It is a hard gate. The model must declare — in the exact format dictated by the archive — what it loaded, what tests passed, and what drift it detected, before it is allowed to touch the repository.

MICA is part of the Flamehaven governance-first AI systems practice. Schema, technical report, and production instance: flamehaven.space. Open-source tooling: AI-SLOP-Detector. All schema references follow the v0.2.0 standard unless a specific earlier version is named.

How Auditing 10 Bio-AI Repositories Shaped STEM-AI

Kwansub Yun — Mon, 30 Mar 2026 12:07:20 +0000

Reading path:
This post is part of a series.
(1) STEM-AI introduction — what the framework is and why we built it
(2) Technical audit report — full findings across 10 repositories
(3) Narrative summary — what those findings actually mean

What Text Could See — and What Code Revealed

In March 2026, we ran STEM-AI against 10 high-visibility open-source bio/medical AI repositories.

The framework did what it was designed to do. It surfaced missing disclaimers, absent CI, weak reproducibility signals, and public-facing governance gaps. Those findings mattered, and the scores were directionally right.

But when we reviewed the audits more carefully, one pattern kept appearing: some of the most consequential failures were not visible in the artifact surface at all. They only became obvious when we looked directly at the code.

This function lives inside a repository presented as an AI-driven drug discovery workflow.

def generate_analogues(self, seed_smiles: str, count: int = 3):
    """
    Mocks a generative model (like REINVENT).
    In a real app, this would call a PyTorch model.
    """
    # Simple string manipulation for demo purposes
    analogues = []
    for i in range(count):
        if "C" in seed_smiles:
            new_smi = seed_smiles.replace("C", "C(C)", 1) if i == 0 else seed_smiles + "F"
            analogues.append(new_smi)
        else:
            analogues.append(seed_smiles + "C")
    return analogues

It does not generate molecules. It appends characters.

SMILES (Simplified Molecular Input Line Entry System) is a strict notation for molecular structure. A valid SMILES string encodes real geometry and bonding. Appending C produces a syntactically valid string that represents no real compound. The function runs without error, returns a list, and the pipeline continues downstream.

Our framework scored this repository T0. Correctly. But not because it saw this function.

It scored T0 because the README was missing disclaimers. The CI was absent. Reproducibility was undocumented. Text-path evaluation is designed to measure exactly that. It did.

The audit result was correct. The evidence surface had room to go deeper.

Running the audits showed us what code-path evaluation could add on top.

What Code Access Makes Visible

The drug discovery example was not unusual.

CellAgent's pipeline ends with this call:

# main.py, line 153
final_result = planner.generate_final_result(results)

The method exists. Its body is pass. The pipeline completes without error and produces nothing. A text audit reading the README would have no way to know this.

BioAgents includes a rate limiter for external API calls:

// rateLimiter.ts, lines 62-68
if (process.env.USE_JOB_QUEUE === 'true') {
    await this.rateLimiter.consume(userId);
} else {
    // Job queue disabled - skip rate limiting for direct calls
    return next();
}

USE_JOB_QUEUE defaults to false in .env.example. Every default deployment has rate limiting disabled. The function name implies protection. In default operation, there is none.

The pattern across all three:

the code looks governed.
the behavior tells a different story.
That story is only visible when you read the code.

Text scores and code behavior can diverge. Knowing where and how they diverge is the next layer of evidence worth capturing.

Four Directions the Audits Opened

Reviewing all ten audits, we identified four areas where code-path evaluation could extend what text auditing already does well.

Direction 1: Clinical exposure is visible in imports, not just in README text.

A repository importing pharmacogenomics allele tables has clinical exposure regardless of what its README says. Detecting that dependency at the import level — rather than waiting for a disclaimer — lets the framework flag exposure earlier.

The key distinction is severity: a direct pharmacogenomics import (CYP2D6, CPIC) signals live patient-facing risk and is classified CA-DIRECT.

A general-purpose medical imaging library like pydicom or MONAI is classified CA-INDIRECT — research-use exposure, not necessarily a live clinical output path. The import alone does not determine clinical risk; the classification tier does.

Direction 2: Not all clinical proximity is the same.

A live pharmacogenomics dosage tool and a README roadmap note about a future ClinVar integration are not equivalent risks. Differentiating them — live output vs. research context vs. planned feature — makes the evaluation more precise and makes the accountability expectations more appropriate.

Direction 3: Scoring stability is worth measuring directly.

We ran Stage 1 on one repository in multiple passes. The results ranged across 28 points on the same input. Overlapping trigger conditions between hype-detection items are one contributing factor.

LLM runtime stochasticity is another — the exact split between the two is still under measurement. Adding explicit discrimination examples — what exact phrasing triggers each item, what does not — makes the scoring surface cleaner and reduces the most obvious sources of variance.

Direction 4: Code-path behavior deserves its own scan layer.

A fail-open pattern is a control path that appears to enforce a constraint but defaults to bypassing it. The BioAgents rate limiter above is the example. In a clinical output path, a silent pass-through is not graceful degradation. It is an untraced result that looks like a real one. Building a dedicated scan for these patterns adds a check that text auditing was never meant to provide.

These four directions came directly from running the audits. The scores across the 10 repositories remain as published. Code-path evaluation is what the framework can now add on top of them.

What v1.0.6 Added — Carried Forward in v1.1.2

These changes were introduced in v1.0.6 and are carried forward in the current internal v1.1.2 package. They extend the framework's evidence surface into code-level behavior. Calibration is ongoing.

1. Two Evidence Paths, Not One

We narrowed one of the biggest divergence points by splitting evaluation into a text path and a code path.

The text path works as before: read the README, CHANGELOG, and public posts, score against the rubric. Always available regardless of access to the repository.
The code path activates when the audit has a local clone. It runs through Claude Code, Codex CLI, Gemini CLI, or Copilot CLI. Claims are not interpreted. They are measured. A README that says "IRB-approved data" earns no points for the statement. Points require a provenance artifact in the code.

When code confirms the README, that is a positive signal. When it contradicts it, that contradiction is the finding.

2. Clinical Dependency Detection

At the start of every local audit, a scan script reads Python imports and README keywords. It classifies the result into one of three severity levels:

# ca_detection_scan.sh -- pharmacogenomics section

# CA-DIRECT: live patient-facing output risk
check_import "CPIC|cpic|PharmGx|pharmacogenomic" \
  "Pharmacogenomics (CPIC/PharmGx)" "CA-DIRECT"

check_import "DPYD|CYP2D6|CYP2C19|allele" \
  "Pharmacogene alleles" "CA-DIRECT"

# CA-INDIRECT: research-use clinical exposure
check_import "import pydicom|from pydicom" \
  "pydicom (DICOM imaging)" "CA-INDIRECT"

check_import "import monai|from monai" \
  "MONAI (medical AI)" "CA-INDIRECT"

Accountability requirements follow the actual clinical proximity of the code. Not the aspirational proximity of the roadmap. A roadmap mention without active implementation is treated as CA-PLANNED rather than collapsed into the same bucket as live clinical output.

The pattern matching is against import statements and function names, not comment text. False positive calibration is still in progress.

3. Code Integrity Scanning (C1-C4)

A second scan handles four code-level checks:

hardcoded credentials (C1)
unpinned dependencies (C2)
clinical-path stubs (C3)
fail-open exception handlers (C4).

The C4 check targets the BioAgents-style pattern. Searching for clinical keywords on the except: line misses most real cases. Clinical context lives in function names and surrounding code. The scan uses a two-pass approach: first identify files with clinical-domain context, then find silent exception handlers within those files.

A silent except: pass in a clinical-context file is a trust-surface failure. The scan makes it visible without requiring a reviewer to read every exception block manually.

4. Discrimination Examples

To reduce the 28-point variance, we added explicit examples for each hype-detection item: what exact phrasing triggers it, what does not, and what the documented edge cases are.

The goal is to reduce obvious scoring drift enough that the same repository is no longer interpreted as two different trust surfaces across different auditors or different LLMs. That goal is not yet verified. The discrimination examples are the primary mechanism toward it.

The Three Questions Now Have a Fourth

In the first post, we described three questions the framework was built around:

Did the repository describe its limits honestly?

Did public communication remain consistent with those limits?

Did the codebase show evidence of maintenance and biological responsibility?

Running 10 real audits pointed toward a fourth:

4. Does the code actually do what the documentation says — and where it diverges, is that divergence visible and traceable?

That fourth question is what the audit outputs kept surfacing. A function name that sounds real. A pipeline that looks complete. An output that is plausible. An implementation that is a stub, or a control path that silently bypasses its own constraint.

The first three questions can be answered by reading. The fourth requires looking at the code.

What the Framework Added — and What Stays the Same

The first post ended with this:

"STEM-AI is meant to support serious review, not replace it."

That has not changed. Every report carries a non-removable disclaimer: LLM-generated audit, not a regulatory determination, not clinical certification. Every report carries an expiry date. The minimum threshold for supervised pilot consideration is still T3. None of the March 2026 repositories reached it.

What the audits added is narrower — broader evidence coverage on top of an already-working foundation:

A verifiable artifact shifts the accountability surface — it does not eliminate the possibility of falsification. The framework treats its presence as a necessary condition, not a sufficient one.

What the framework gained is that the evidence it counts now extends beyond what authors say about their own code.

Three Directions Still Ahead

Automated re-audit on repository changes. A score from three months ago may not describe the same repository. The trajectory signal measures issue close rate and release frequency across consecutive 90-day windows. It is a partial answer. A CI-triggered re-audit path is the logical next step.

The denominator problem. Zero of 10 repositories reached T3. This may accurately describe the ecosystem's current state. It may also reflect calibration issues in the upper tiers. Distinguishing between the two requires before-and-after auditing of repositories that have received systematic governance remediation.

The Stage 2 redistribution question. Most audits have no cross-platform consistency data. When that data is unavailable, the framework redistributes Stage 2's weight equally between documentation quality and engineering accountability.

For repositories with clinical-direct exposure, a well-written README can then compensate for weak code accountability. A guardrail flags this condition. The current redistribution rule is explicit but not yet final — it remains one of the framework's open calibration questions.

If there are open-source bio-AI repositories you think should be >audited next, drop them in the comments. Bonus if they claim clinical >relevance, drug discovery, or medical reasoning.

STEM-AI v1.1.2 — Trust Evaluation Framework for Medical AI Repositories.

"Code works. But does the author care about the patient?"

Everyone Was Talking About Context Engineering. Nobody Had Solved Governance.

Kwansub Yun — Wed, 25 Mar 2026 09:29:42 +0000

Disclosure: This article was written by the author with AI assistance for editing. All technical content, architecture decisions, and design rationale are the author's own. #ABotWroteThis

Glossary: terms used in this article

🔸 MICA (Memory Invocation & Context Archive): A governance schema for AI context management. Defines how context should be structured, trusted, scored, and handed off across sessions.

🔸 Fail-Closed Gate: An admission rule that excludes a context item if it fails a required threshold — regardless of its score on other dimensions. No exceptions. Introduced in v0.1.7.

🔸 README-as-Protocol: The pattern in which an AI session's natural behavior of reading the README first is formalized as the primary invocation mechanism. No installation required. Introduced in v0.1.8.

🔸 Invocation Protocol: The schema-level declaration of how a MICA archive reaches an AI session — and how the session confirms it was loaded. Formalized as a required field in v0.1.8.

🔸 Session Report Format: The structured opening report the model must produce at session start to confirm the archive was loaded. Required in v0.1.8.

🔸 Design Invariant Entry: A structured governance rule with identity, rule text, and severity. Replaced plain string invariants in v0.1.8.

🔸 Self-Test Policy: Machine-evaluable checks that validate the archive against the real project state — file existence, hash integrity, and README sync. Required in v0.1.8.

🔸 Playbook: The operator-facing discipline layer that sits outside the schema. The schema enforces structure; the Playbook enforces judgment.

🔸 Context Engineering: The practice of shaping what the model sees, in what order, with what boundaries, and under what assumptions — not just what you ask it, but what it actually knows at runtime.

🔸 CTX: A collection-first context packaging approach that gathers relevant workspace material and delivers it to the model. In this article, CTX represents the collection layer of context engineering — answering, “What does the AI see?”

1. What Parts 1 Through 4 Actually Established

The first four parts of this series already narrowed the problem considerably.

Part 1 defined the failure mode.
The issue was not that long-running AI work needed “more prompt.” The issue was that a model can only act on what it actually knows right now, and most project context systems still treat that as a document problem instead of a governance problem.
Part 2 established the first hard boundary.
A schema can exist, and the model can still have no reliable way to know it exists. That is the difference between a document and a constraint.
Part 3 moved governance into the schema.
Provenance, deviations, and semantic rules could no longer remain outside the system in READMEs, comments, or team habits. They had to become machine-readable structure.
Part 4 made that structure operative.
The model already treated the README as its natural entry surface. Once that behavior was declared as an invocation protocol, the schema stopped being a passive archive and became a runtime contract.

That progression defines what MICA is actually trying to solve.

It is not trying to be “better prompting.”

It is not trying to be “more retrieval.”

It is trying to answer a narrower question:

How does governed context reach the model, under declared rules, with confirmable load and auditable change?

That is the bridge into the broader landscape.

2. Context Engineering Was Never Just Prompting

One clarification matters before going further.

Context engineering is not just prompt writing.

At the broadest level, it is the practice of shaping what the model sees, in what order, with what boundaries, and under what assumptions. Prompts are one part of that. Retrieval is another. File selection, memory handoff, system instructions, and workspace state all belong to the same larger question:

What does the model actually know right now?

By that definition, MICA is part of context engineering.

Not because it retrieves context.

Not because it packs more tokens into a window.

But because it governs which context is allowed to shape the session, under what trust conditions, and with what record when those conditions are tested.

That distinction matters, because most of the field has focused on a different layer.

3. The Conversation Was Already Happening

Context engineering is not a new idea, and MICA does not claim to have invented the conversation.

The term was amplified by Andrej Karpathy and others, but the underlying practice — designing what the model sees, not just what you ask it — had already been emerging in serious AI work.

Collection-first tools already existed. CTX is a useful example of that layer: it gathers relevant workspace material and delivers it to the model without manual copy-paste. It answers an important question well:

What does the AI see?

At the same time, some of the sharper practitioner writing was already moving beyond collection alone. One such example was an OpenAI Developer Community post by Serge Liatko, “Prompt Engineering Is Dead, and Context Engineering Is Already Obsolete: Why the Future Is Automated Workflow Architecture with LLMs.”

The value of that piece was not the author's status, but the precision of the problem it named: manually maintained context eventually reaches its ceiling.

Once system state changes faster than humans can keep context aligned, the real question is no longer just how to collect context, but how to automate its ownership, maintenance, and validation as the system evolves.

That was an important move forward.

But one layer still remained underdefined.

4. The Missing Layer Was Already Visible

The same missing layer had already shown up elsewhere.

In Your Agentic Stack Has Two Layers. It Needs Three, I argued that the usual stack had matured around two strong layers:

MCP / tool calls — how the agent talks to systems
agent skills — what the agent can do

But something was still missing above both:

the layer that decides whether the agent should do it, under what constraints, and toward what end

That is the layer of intent, authority, and governance.

The same problem appears in context engineering.

A context pipeline can be excellent at retrieval and still be weak at governance. It can gather the right files, summarize the right notes, and deliver the right-looking material — and still fail to answer:

what is authoritative?
what is provisional?
what must never be violated?
what changed since last time?
how does the session prove it loaded the governed archive at all?

Those are not collection questions.

They are governance questions.

5. Where CTX Stops and MICA Begins

CTX solves the collection problem.

It answers:

What does the AI see?

That is a necessary layer. Without it, context management collapses back into manual copy-paste, repeated explanation, and fragile session startup.pro

But it does not answer the next set of questions:

Where did this context item come from, and can that claim be verified?
What happens when a file changes between sessions?
Which constraints are non-negotiable, and what is the consequence when they are violated?
Who approved the last change to the archive, and can that decision be audited?
When the session begins, how does the system confirm the archive was actually loaded?

Those are not retrieval questions.

They are governance questions.

That is the narrow but consequential difference.

CTX collects context and delivers it.

MICA governs what trust that context carries, what invariants it must not violate, what happens when it changes, and how the session proves that the governed archive was actually loaded.

One answers:

What does the AI see?

The other answers:

Under what rules does the AI operate — and what is the record when those rules are tested?

They are different layers. Neither replaces the other.

6. The Part That Was Still Open

A recurring theme in serious context-engineering discussion is that context cannot remain a hand-curated artifact forever. It has to become a function of system state.

But that still leaves one question open:

Who owns the specification for each step's input — and how is this versioned, tested, and audited as requirements shift?

MICA is a concrete, working answer to that gap.

Not the only possible answer. Not necessarily the final one. But a real one.

Its claim is not that context engineering needed to be invented.

Its claim is that context engineering still needed a governance layer with at least four properties:

machine-addressable invariants
versioned and auditable change records
self-tests against the real project state
declared invocation with confirmable session load

That is the layer MICA was built to supply.

7. What Governance Actually Means Here

Governance is an overloaded word. In this context it means something specific.

Provenance

Every context item must declare where it came from in a way that can be checked.

Auditability

Changes to the archive are recorded when they happen, not reconstructed later from memory.

Invariant enforcement

Constraints are not vague README prose. They are structured entries with identity and severity.

Self-testing

The archive is checked against the real project state, not only against its own internal shape.

Invocation confirmation

The model does not silently ignore the archive. Session start requires a structured acknowledgment that the governed archive was loaded.

None of these are abstract principles in MICA.

They are structural requirements in a running schema.

8. What This Is Not

It is worth being precise about the boundary.

MICA does not generate context automatically from the codebase. That is not its job. Collection-first systems already exist, and they are valuable. MICA governs what happens once context has been identified.

MICA does not replace human judgment. A schema can require structure, audit trail, drift response, and self-tests. It cannot eliminate operator discipline. That is why the boundary between schema and playbook matters.

MICA is also not a finished system. Parts 1 through 4 of this series were explicit about what each version got wrong, what each version corrected, and what remained unresolved.

That design history is part of the claim, not an embarrassment to it.

9. The Actual Claim

The actual claim is not that MICA solves all of context engineering.

It is this:

A small operation can already run a governed AI context system — with verifiable provenance, deviation audit trail, structured invariants, self-testing, and declared invocation — without waiting for future tooling that does not yet exist.

That claim is demonstrated by the design history already covered in this series:

v0.1.0 made scoring implementable
v0.1.5 brought governance structure into the schema
v0.1.7 made scoring fail-closed
v0.1.8 made invocation declared and confirmable
v0.1.8.1 clarified the remaining runtime ambiguities

And in practice, governance at runtime can look as concrete as this:

[SESSION READY]
Gate: PASS (self-tests: 7/7) | Track: A,B
Critical Invariants: 3/3 | Deviations: 0

If a critical check fails, the session does not proceed. That is what governance looks like when it becomes operative.

What can be said now is narrower and more solid: the gap is real, collection-first systems solve one side of it, and MICA addresses the governance side. The conversation about what comes after context engineering was already happening; MICA is one concrete answer to that part of it.

10. What Comes Next

Part 4 ended with a specific question:

Where does MICA sit in the context engineering landscape that already existed around it?

This post is the answer.

It sits inside context engineering — but not at the collection layer.

It does not compete with retrieval-first systems by trying to collect more files, pack more tokens, or automate more handoff. It begins after that layer. Its job is to govern what enters the session, what remains authoritative, what drift means, what violations matter, and how the model proves it actually loaded the governed archive at all.

That is why the answer is narrower than most people expect, and more specific than most framings allow.

MICA is not “the future of all context engineering.”
It is a governance answer to the part of context engineering that collection alone does not solve.

Part 6 will move back down from landscape to concrete operation.

It will show what MICA looks like in an actual project context: what a session opening report looks like, what a deviation log entry looks like in practice, and what happens when a self-test flags drift.

After that comes the harder question: what remains unresolved.

The series continues only where there is something concrete to specify, test, or correct.

Named decision from this post: Governance is not a layer you add after context engineering works. It is the layer that makes context engineering trustworthy — by declaring what is authoritative, recording what changes, and confirming what the AI actually loaded.

MICA is part of the Flamehaven governance-first AI systems practice. Schema, technical report, and production instance: flamehaven.space. Open-source tooling: AI-SLOP-Detector. All schema references follow the v0.1.8.1 Universal standard unless a specific earlier version is named.

The Model Already Read the README. MICA v0.1.8 Made It a Protocol

Kwansub Yun — Mon, 23 Mar 2026 08:20:43 +0000

Disclosure: This article was written by the author with AI assistance for editing. All technical content, architecture decisions, and design rationale are the author's own. #ABotWroteThis

Glossary: terms used in this article

🔸 MICA (Memory Invocation & Context Archive): A governance schema for AI context management. Defines how context should be structured, trusted, scored, and handed off across sessions.

🔸 Fail-Closed Gate: An admission rule that excludes a context item if it fails a required threshold — regardless of its score on other dimensions. No exceptions. Introduced in v0.1.7.

🔸 Invocation Protocol: The schema-level declaration of how a MICA archive reaches an AI session — and how the session confirms it was loaded. Formalized as a required field in v0.1.8.

🔸 Session Report Format: The structured opening report the model must produce at session start to confirm the archive was loaded. Required in v0.1.8.

🔸 Design Invariant Entry: A structured governance rule with identity, rule text, and severity. Replaced plain string invariants in v0.1.8.

🔸 Self-Test Policy: Machine-evaluable checks that validate the archive against the real project state — file existence, hash integrity, and README sync. Required in v0.1.8.

1. Where Part 3 Left Off

Part 3 covered v0.1.0 through v0.1.5.

By that point, the schema had become implementable, then auditable. Provenance gained a registry. Changes gained an audit trail. Invariants gained structure.

But two things still remained missing.

Scoring was still not structurally enforced at the schema level. The formula existed, but it was still closer to convention than contract.

And invocation_protocol still did not exist. The archive could be perfectly governed — verified, logged, ruled — and the model could still begin work with no instruction to locate it.

The governance was visible. It was not yet operative.

This part covers the versions that closed both gaps — and the observation that made the harder one easier to solve than expected.

2. v0.1.7: Scoring Becomes a Contract

Since v0.0.1, scoring had been one of the structurally weakest parts of MICA.

The earliest version used hardcoded test values with no defined combination rule. Later versions introduced an implementable formula, but the formula still lived as a declared behavior rather than a fully enforced contract. A conforming archive could still be structurally valid while leaving too much scoring behavior open to interpretation.

v0.1.7 changed that in two steps.

First, scoring became a structured policy.

Second, admission became fail-closed.

That second change matters more than it first appears.

A weighted score answers:

Which item ranks higher?

A fail-closed gate answers:

Should this item be considered at all?

That is the question Part 2 identified as missing from the usual output-reliability framing.

To illustrate the pattern, here is a simplified educational example:

{
  "scoring_policy": {
    "weights": {
      "sim": 0.4,
      "trust": 0.3,
      "continuity": 0.1,
      "...": "..."
    },
    "gate_policy": {
      "trust_floor": "distilled",
      "fail_behavior": "exclude"
    }
  }
}

This is not the production specification. It is a teaching example.

The point is simple: once admission becomes fail-closed, scoring stops being "rank everything and hope the top item is good enough." It becomes a governed decision boundary.

By v0.1.7, scoring had become a contract.

But scoring was never the deepest structural gap.

The deepest gap had always been invocation — how the archive reached the model at all.

3. The Observation That Changed the Question

Before v0.1.8, invocation was framed as an installation problem.

How do you make sure the model loads the archive before starting work?
The assumed answers were plugins, tools, configuration, or external services.

But that assumption missed something simpler.

In many repository-based AI sessions, the README is already the model's first orientation point. Not because it is explicitly configured to read it, but because it is the natural entry surface for understanding a workspace.

The invocation mechanism did not need to be invented.

It needed to be declared.

That insight changed the problem.

If the README is already where an AI session begins, then the README can carry a structured session protocol:

load the archive
run session-start checks
report readiness
stop if critical governance conditions fail

A simplified educational example looks like this:

## [AI Session Protocol]

Before starting any work:

1. Load `memory/example-service.mica.json`
2. Check critical invariants
3. Run session-start self-tests
4. Report session readiness
5. Stop if any blocking condition is present

Required opening report:
- gate
- active invariants
- drift status
- track

This is the key idea behind README-as-Protocol.

The README is no longer just documentation.
It becomes the session entry layer.

README-as-Protocol is not a workaround.
It is a recognition: the invocation mechanism already existed in practice. It simply had not yet been formalized.

v0.1.8 turned that recognition into a schema-level contract.

4. v0.1.8: The Schema Reaches the Model

At this stage, the important change was not one field. It was the structural shift.

The archive no longer just described governance.
It declared how governance entered the session.

In simplified terms, v0.1.8 added five kinds of structure:

the archive declares how it reaches the model
the model must prove it loaded the archive
invariants become structured governance objects
the archive declares how drift is handled
the archive can validate itself against the real project state

The following simplified example illustrates three of those five additions:

{
  "project": {
    "name": "example-service",
    "version": "1.0.0"
  },
  "design_invariants": [
    {
      "id": "DI-001",
      "statement": "Do not introduce undeclared dependencies.",
      "severity": "critical"
    }
  ],
  "invocation_protocol": {
    "primary_pattern": "readme_protocol"
  },
  "session_report_format": {
    "trigger": "session_start",
    "required_fields": ["gate", "active_invariants", "track"]
  }
}

This example leaves out most of the real system. That is intentional.

The purpose is to show the architectural shift:

design_invariants says what must remain true
invocation_protocol says how the archive reaches the session
session_report_format says how the model proves it loaded the archive

The id makes the invariant referenceable in audit output, and severity determines how violations are handled.

Before v0.1.8, a session could begin with a perfectly governed archive existing somewhere in the repository — but with no requirement that the model actually load it.

After v0.1.8, the archive no longer merely existed.

It had a declared path into the session.

And the session was not considered ready until it reported that the archive had been loaded.

That is the moment where governance stopped being documentation and became runtime behavior.

Files change. Hashes drift. Before this layer existed, that drift could remain silent. Now the schema requires a declared response when the archive no longer matches the project it describes.

5. v0.1.8.1: Precision Patches

Once the major shift was in place, several smaller ambiguities appeared in practice.

These did not change the architecture.
They clarified how the architecture behaved.

In simplified terms, the patches did three things:

they made self-tests more explicitly machine-evaluable
they declared which runtime was responsible for running those tests
they clarified authority between invariant definitions and derived track views so drift could be detected instead of silently accumulating

These are not the kinds of changes that make headlines.

They are the kinds of changes that make a governance system stable enough to survive repeated use.

That is what patch releases are for.

6. Playbook: What the Schema Cannot Enforce

At this point, a second document became necessary: the Playbook.

The schema can enforce structure:

required fields
valid shapes
invocation declaration
opening reports
self-tests
drift behavior

But it cannot enforce operator judgment:

when to use which invocation pattern
how to update invariants safely
what counts as a complete archive
which shortcuts are acceptable
how to avoid human-caused drift

So the division became explicit.

The schema enforces structure.
The Playbook enforces judgment.

That is not a limitation. It is a design boundary.

A governance system that tries to encode all human discipline into the schema becomes brittle and unmaintainable. Some rules belong in machine-validated structure. Others belong in operator practice.

A simplified educational example makes the difference concrete.

The schema requires that invocation be declared:

{
  "invocation_protocol": {
    "primary_pattern": "readme_protocol"
  }
}

The Playbook explains how to use it correctly:

Use README-as-Protocol as the default starting mode.
Do not submit a partially completed archive to the validator as if it were final.
Treat invariant track assignment as the single source of truth.
Validate in this order: schema, invocation completeness, self-test coverage, provenance completeness, then track consistency.

The schema can require that invocation exists.
It cannot require that the operator uses it well.

README-as-Protocol works for the widest range of projects without requiring anything beyond a text file. That makes it the recommended default. But it is not the only option. More installed or infrastructure-heavy approaches may be better in some environments. The Playbook documents when the default is no longer enough — and what to do next.

7. The Distance from v0.0.1 to an Operative Schema

Part 2 named three failures in v0.0.1.

Failure 1: Scoring with no defined semantics.
The earliest version used hardcoded values with no clear combination rule or enforced model.
→ Closed in v0.1.0 (implementable formula) and v0.1.7 (structured scoring plus fail-closed admission).

Failure 2: Invariants encoded as comments.
The earliest version used plain strings where governance objects were needed.
→ Partially closed in v0.1.5. Fully closed in v0.1.8 when invariants became structured, referenceable, severity-bearing entries.

Failure 3: No path to the model.
The archive existed, but the model had no reliable way to know it existed.
→ Closed in v0.1.8 when invocation became a declared protocol and session start required proof of archive load.

That progression can be summarized simply:

The schema existed.
Then governance moved into the schema.
Then the schema reached the model.
Only then did governance become operative.

That is the real distance from v0.0.1 to this point.

8. Where This Leaves MICA

At this stage, MICA is no longer just a memory format.

It is no longer just a structured archive.

It is a runtime governance contract:

what context is allowed in
what must remain true
how the archive is loaded
how drift is handled
how readiness is reported before work begins

That is a different category of system.

9. What Comes Next

The next part of this series steps back from version history and asks a broader question:

Where does MICA sit in the context engineering landscape that already existed around it?

The answer is narrower than most people expect, and more specific than most framings allow.

Named decision from this post: An AI session already uses the README as an orientation surface. Formalizing that behavior as a schema-level invocation protocol is not a trick. It is the recognition that the mechanism already existed — and needed to be declared, not invented.

MICA is part of the Flamehaven governance-first AI systems practice. Schema, technical report, and production instance: flamehaven.space. Open-source tooling: AI-SLOP-Detector. All schema references follow the v0.1.8.1 Universal standard unless a specific earlier version is named.

The Stake Was Governance Outside the Schema. MICA v0.1.5 Pulled It In

Kwansub Yun — Sat, 21 Mar 2026 17:09:51 +0000

Glossary: terms used in this article

🔸 MICA (Memory Invocation & Context Archive): A governance schema for AI context management. Defines how context should be structured, trusted, scored, and handed off across sessions.

🔸 Provenance Registry: A structured, hash-anchored record of where each context item came from. Requires uri, sha256, kind, created_at, and trust_class. Formalized as a required field in v0.1.5.

🔸 Deviation Log: An auditable record of every change to a governed archive. Each entry requires before_hash, after_hash, gate, approved_by, and rollback_ready. Formalized in v0.1.5.

🔸 Semantic Validation Policy: A machine-evaluable rule set applied to context items. Each rule requires id, expression, severity, and on_fail. Replaces string invariants in v0.1.5.

🔸 Semantic Collapse: The pattern in which a JSON Schema specification is applied to an LLM as a runtime contract rather than as a validator. The LLM executes the schema semantically and produces a contract-compliant artifact. First demonstrated in v0.1.4.

🔸 Invocation: The mechanism by which a MICA archive reaches an AI session. Still absent at v0.1.5. Formalized as a required field in v0.1.8.

1. Where Part 2 Left Off

Part 2 documented three failures in v0.0.1: scoring with no defined semantics, invariants encoded as comments rather than constraints, and no path for the archive to reach the model at all.

The first two were fixable by adding structure. The third was different in kind — a schema without an enforcement path is not a governance schema at all.

This part covers v0.1.0 through v0.1.5. What the intermediate versions fixed. What experiment revealed the need for v0.1.5. And what was still missing at the end of it.

2. The Stake Nobody Pulled

There is an old story about elephants. A young elephant chained to a small stake learns it cannot move. Years later, fully grown, it still does not pull — not because the stake holds, but because the boundary became an unquestioned assumption.

In practice, what I kept seeing across LLM tooling discussions was exactly that. Technical provenance tracked inside the system, governance detail left in documents. Who approved a change, under what conditions, with what rationale — always a README, a review checklist, a comment. Never the schema. Semantic rules written as strings, described in documentation, reviewed by humans. Everyone knew the constraint. Nobody encoded it.

The stake was real. It was not immovable.

v0.1.0 through v0.1.4 did not question it.

The stake was a single assumption: governance lived outside the schema.

3. v0.1.0 to v0.1.4: More Implementable, Not Yet Inspectable

These versions addressed the first two failures from v0.0.1. In brief:

v0.1.0: Scoring became implementable — hardcoded hints replaced with an explicit weighted formula: clamp01(0.55 * sim + 0.15 * recency + 0.10 * invoke + 0.10 * trust + 0.10 * continuity). Two implementations now produce the same number for the same inputs.
v0.1.0: invoke_role semantics defined — each role got explicit score bonus, pin behavior, compile behavior, and eviction priority.
v0.1.0: Eviction became a five-phase strategy — budget overflow had a defined sequence and a failure condition.
v0.1.0: Error handling defined — eight failure modes, each with an explicit action and severity.
v0.1.4: Description updated to include "semantic enforcement" — required field count unchanged. The label was ahead of the implementation.

These versions made the schema reproducible to implement. They did not make governance inspectable. design_invariants remained a plain string array. scoring_policy.function remained a free string in the JSON Schema. provenance_registry did not exist. The assumption from Section 2 was still in place.

4. The Experiment That Revealed the Gap

In March 2026, a semantic collapse experiment was run against v0.1.4. The procedure was direct: provide the schema and session context to an LLM, instruct it to execute the schema semantically, and verify the output against all required fields.

The result was a fully contract-compliant artifact. Zero field violations. Zero type constraint violations. The scoring pipeline ran. The eviction log was correct. The audit trail was structurally preserved. The schema worked — as a semantic contract against an LLM runtime.

But the artifact also contained this:

"provenance": {
  "uri": "mica://memory/flamehaven-labs/project/identity",
  "source_kind": "memory_export",
  "trust_class": "canonical",
  "collected_at": "UNKNOWN",
  "content_hash": "sha256:project_identity_anchor"
}

"collected_at": "UNKNOWN". Every memory-tier item in the artifact carried this. The timestamp was not available. The schema had no mechanism to require it. The provenance claim could not be verified — not because the data was missing, but because the schema had no structure to anchor it.

The experiment proved the schema was executable. It also proved that a contract-compliant archive could contain provenance that was structurally present but epistemically empty.

The same experiment revealed two more gaps. Design invariants were strings — the LLM read them, interpreted them, and applied them as best it could. But there was no id to reference in audit output, no severity to determine handling, no on_fail to define consequence. And every change to the archive between sessions had no record. No before_hash. No approved_by. No rollback_ready. The archive changed silently.

The semantic collapse experiment proved the schema was implementable. It also produced a precise map of what governance was still missing.

That map became v0.1.5.

5. The Question v0.1.5 Asked

v0.1.5 did not ask "what else should we add?"

It asked: what if governance itself belongs inside the schema?

Not as a description of what enforcement should look like. As a machine-evaluable structure that enforces it.

MICA v0.1.0 through v0.1.4 had followed the same pattern common in practice — governance detail kept implicit, schema focused on structure and scoring. The experiment made the cost of that pattern visible. v0.1.5 pulled three assumptions out of documentation and forced them into machine-readable structure.

6. What v0.1.5 Pulled In

Stake 1: Provenance was a field. v0.1.5 made it a registry.

The experiment artifact had "collected_at": "UNKNOWN" on every memory item. Before v0.1.5, there was no mechanism to prevent that — or detect it.

v0.1.5 introduced provenance_registry as a required top-level structure. Each record:

"required": ["uri", "sha256", "kind", "created_at", "trust_class"]

sha256 anchored the source to a specific state. trust_class declared reliability at registration — not inferred at use. created_at was now required — not optional, not "UNKNOWN". A context item's provenance claim could now be checked against the registry. If the hash did not match, the claim was false.

Provenance had been a field on a document. v0.1.5 made it a verifiable record.

Stake 2: Changes happened. v0.1.5 made them auditable.

Between sessions, the archive changed. There was no record of what changed, when, under whose authority, or whether it could be undone.

v0.1.5 introduced deviation_log as a required field. Each entry:

"required": [
  "change_id", "timestamp_utc", "gate", "reason",
  "before_hash", "after_hash", "approved_by", "rollback_ready"
]

before_hash and after_hash meant every change left a cryptographic record. gate associated the change with a defined governance checkpoint. approved_by declared authority explicitly. rollback_ready required the system to declare whether it could undo the change before committing it.

Change history had been implicit. v0.1.5 made it a required audit trail.

Stake 3: Invariants were strings. v0.1.5 made them rules.

The experiment applied string invariants as best as the model could interpret them. There was no id to reference in audit output, no severity to triage violations, no on_fail to define consequence.

v0.1.5 introduced semantic_validation_policy with structured rules:

"required": ["id", "expression", "severity", "on_fail"]

Each rule had an id — referenceable in audit output. Each had a severity — determining how a violation was handled. Each had an on_fail — defining the consequence. A violation could now be detected, logged, and traced.

String invariants described what the system must not do. Semantic rules could detect when it did.

7. What v0.1.5 Still Did Not Have

Two things remained missing.

Scoring was still a free string at the schema level.

scoring_policy.function was still { "type": "string" } in the JSON Schema. The weighted formula from v0.1.0 was a specified convention — implementors who followed it produced the same number. But the schema could not enforce that they did. A conforming archive could contain any string in that field. The formula was real. The contract was not.

The archive still had no path to the model.

invocation_protocol did not exist in v0.1.5. The provenance registry was hash-anchored. The deviation log was auditable. The semantic rules were machine-evaluable. A session could start with a perfectly governed archive — verified, logged, ruled — and the model would begin work with no instruction to locate it.

The governance was visible. It was not yet operative.

8. What This Series Covers Next

Part 1 defined the problem.

Part 2 documented v0.0.1 — what it was and why its most fundamental failure took the longest to fix.

This part covered v0.1.0 through v0.1.5: how the schema became implementable, then auditable, and what it still could not do.

Part 4 covers v0.1.7 and v0.1.8: the fail-closed gate, the provenance enforcement boundary, and the version that finally made invocation a required field — the stake that had been in the ground since v0.0.1.

The series continues only where there is something concrete to specify, test, or correct.

MICA is part of the Flamehaven governance-first AI systems practice. Schema, technical report, and production instance: flamehaven.space. Open-source tooling: AI-SLOP-Detector. All schema references follow the v0.1.8.1 Universal standard unless a specific earlier version is named.

Medical AI Repositories Need More Than Benchmarks. We Built STEM-AI to Audit Trust

Kwansub Yun — Fri, 20 Mar 2026 17:30:52 +0000

If you have been paying attention to GitHub recently — the past six months — you have seen the pattern.

A new bio-AI repository appears. It promises to automate genomic analysis, drug discovery, medical imaging, or clinical data interpretation. The README is polished. The architecture diagram looks serious. Within weeks it has hundreds of stars, a few forks, and a preprint on bioRxiv.

Then nothing.

No CI. No CHANGELOG. No response to issues. No clear statement of limitations. No clinical disclaimer anywhere in the repository. And a tool that may now be touching patient-adjacent workflows has exactly one quality gate: the author thought it was ready.

We are developers. We have built things in that atmosphere too. At some point, we had to ask a harder question than benchmark accuracy:

When a bio-AI repository gets close to real diagnostic, genomic, imaging, or therapeutic workflows — what does "trustworthy enough for serious review" actually mean?

The field has benchmarks for models. It has almost no shared standards for repository accountability.

So we built one.

What Makes This Moment Different

Bio-AI is now filling up with skill libraries, agent wrappers, orchestration pipelines, and plugin-style marketplaces that look far more deployable than they actually are. Surface maturity is easy to fake. A clean README, a marketplace entry, or a rising GitHub star count can make a repository look trustworthy long before it has earned that trust.

And this category cannot be judged like ordinary software.

If a note-taking app breaks, users get frustrated.
If an internal dashboard fails, a team loses time.
But when a biomedical or medical-AI repository fails quietly, the consequences do not stop at software quality.

A flawed genomics pipeline can distort interpretation.
A weak clinical model can normalize unsafe confidence.
A drug-discovery system can push the wrong candidates forward, bury better ones, and send time, capital, and downstream validation effort in the wrong direction.

In this category, failure is not just a debugging problem.
It is a patient-safety problem and a resource-allocation problem.

That is why these systems require more than code inspection. They require scrutiny of documentation, limits, provenance, maintenance behavior, and public claims — because the cost of being wrong is fundamentally different here.

What STEM-AI Is

STEM-AI (Sovereign Trust Evaluator for Medical AI) is a governance audit framework for public bio/medical AI repositories.

It does not ask whether a project sounds impressive.

It asks whether the repository shows observable signs of responsible engineering:

honest documentation
consistent public claims
maintenance discipline
biological data responsibility
explicit acknowledgement of limits

That distinction matters. A repository can look technically sophisticated and still fail the most basic governance test for patient-adjacent use. A polished README is not a safety surface by itself.

STEM-AI is not a regulatory verdict, clinical certification, or legal assessment. It is a structured review framework designed to give researchers, reviewers, procurement teams, and engineers a more reproducible starting point for discussion. The canonical spec requires a non-waivable disclaimer in every output stating exactly that.

STEM-AI is meant to support serious review, not replace it.

Why We Made It LLM-Native

STEM-AI runs as a structured specification executed by a major LLM.

The spec is the program. The LLM is the runtime.

That sounds strange until you look at the design constraint. We did not want a system that "vibes" its way to a trust score. We wanted a system that forces checklist-based scoring, explicit evidence chains, N/A handling for missing data, and hard floors for catastrophic claims. The goal is to reduce evaluator drift by replacing narrative judgment with traceable rubric logic.

The model is not supposed to "feel" trust.
It is supposed to count evidence.

How It Works

STEM-AI evaluates a repository across three stages. Each stage has a defined checklist. Each score must map back to observable evidence.

These three stages ask three different questions:

what the repository says
what public communication says
what the codebase actually proves

Stage 1 — README Intent

If a repository is patient-adjacent, the README is not marketing. It is the first governance surface.

WHAT WE LOOK FOR (positive signals):

  [R1] Does a Limitations or Known Issues section exist
       and cover a substantial portion of the README
       with specific, actionable content?

  [R2] Is any regulatory framework cited?
       (FDA SaMD guidelines, CE marking, IRB requirements,
        or equivalent)

  [R3] Is there a clinical disclaimer?
       ("This tool is NOT a substitute for clinical judgment"
        or equivalent)

  [R4] Are demographic bias limits or applicable population
       boundaries disclosed?

  [R5] Are reproducibility provisions present?
       (environment pinning, data version, seed values)

WHAT WE PENALIZE (negative signals):

  [H1] Performance superlatives without benchmark comparison
       "SOTA", "State-of-the-Art", "Best-in-class"

  [H2] Unsubstantiated innovation claims
       "Revolutionary", "Groundbreaking", "Game-changing"

  [H3] Fully autonomous framing in a clinical context
       "Fully Automated", "Zero Human Oversight"
       without any stated supervision requirement

  [H4] AGI or human-level capability claims
       "AGI", "Human-level", "Surpasses clinicians"

  [H5] Social proof substituted for technical credibility
       GitHub stars or download counts cited as a trust signal

  [H6] External optics substituted for technical evidence
       VC funding or press coverage without validation

![The Patient safety trapdoor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j4go8oh9dn80at4x4npu.png)

CRITICAL RULE:
  IF H3 AND H4 are both triggered simultaneously:
    → Hard floor activated.
    → Final Score = 0. Tier = T0. No further scoring.
    → A tool claiming fully automated clinical operation
      at AGI level is a patient safety issue,
      not a scoring edge case.
      No positive signals elsewhere can override this.

On clinical adjacency: STEM-AI does not assume all biomedical repositories carry the same deployment risk. The disclosure bar rises as a tool moves closer to patient-facing or decision-shaping use. If the repository contains medical imaging frameworks, drug docking engines, diagnostic genomics pipelines, or clinical language models, the absence of R2 and R3 is not a missed bonus — it becomes an active penalty.

Stage 2 — Cross-Platform Consistency

This stage is less about marketing tone and more about contradiction. If a README warns carefully but public posts erase those warnings, the repository's governance surface is inconsistent.

WHAT WE PENALIZE:

  [F1] Stars or downloads used as the primary message
       on external platforms, without technical substance

  [F2] Public dismissal or hostile response to
       critical feedback from the community

  [F3] Vanity metrics presented as clinical trust proxies

  [F4] External posts that directly contradict or omit
       warnings stated in the README
       → Most serious flag in Stage 2.
         A README that says "not for clinical use" while
         the author's LinkedIn announces clinical deployment
         is a consistency failure.

WHAT WE REWARD:

  [A1] Model errors or hallucination cases
       openly shared in public forums

  [A2] Reproducibility failures or known failure modes
       transparently acknowledged outside the repository

  [A3] Critical external feedback accepted and incorporated

  [A4] Regulatory or ethics body collaboration referenced

This stage is only as strong as the public evidence available. When live cross-platform data is unavailable, Stage 2 goes N/A and its weight moves to Stages 1 and 3. In medical-adjacent evaluation, pretending to know is worse than stating you do not.

Stage 3 — Code Infrastructure and Biological Integrity

This is where the framework becomes more than a documentation audit.

TECHNICAL RESPONSIBILITY:

  [T1] CI/CD pipeline
       Does automated testing run on push or PR?
       Is coverage stated or just implied?

  [T2] Domain-specific regression tests
       Not just unit tests — tests that verify biological
       or clinical correctness invariants.

  [T3] CHANGELOG transparency
       Does one exist?
       Does it document bugs and failures honestly,
       or only feature additions?
       Silent updates on a clinical-adjacent tool are
       a different risk profile than an honest bug log.

  [T4] Issue and PR response pattern
       When did the last maintainer response happen?
       What proportion of issues receive a response?

BIOLOGICAL INTEGRITY:

  [B1] Data provenance and consent
       Where did the training or reference data come from?
       Is there an IRB approval or data consent declaration?

  [B2] Algorithmic bias disclosure
       Are performance limits across demographic subgroups
       disclosed? Are actual measurements provided,
       or only a general acknowledgment?

  [B3] Conflict of interest transparency
       Is the funding source disclosed?
       If there is commercial interest, is it stated?

TRAJECTORY SIGNAL:
  v1.0.4 adds a trajectory modifier comparing issue close
  rates and release frequency across two consecutive
  90-day windows.

  This signal is deliberately bounded: it changes Stage 3
  by at most ±5 points, which translates to roughly ±2
  points on the final weighted score.
  Large enough to matter near tier boundaries.
  Small enough not to distort the audit.

What the Output Looks Like

Every STEM-AI audit produces a structured report. Here is an illustrative example — repository name withheld, structure unchanged:

# 🩺 STEM-AI Audit Report v1.0.4
──────────────────────────────────────────────
Target:        [Repository Name]
Audit Date:    2026-03-20
Report Expiry: 2026-09-16
Flags:         CLINICAL_ADJACENT: true
               NASCENT_REPO: false
               T0_HARD_FLOOR: false
──────────────────────────────────────────────
Stage 1 — README Intent        47 / 100  VC Pitch
Stage 2 — Cross-Platform       N/A       (MANUAL mode)
Stage 3 — Code Infrastructure  10 / 100  Hit-and-Run
──────────────────────────────────────────────
Final Score:   28 / 100
Tier:          T0 REJECTED
USE_SCOPE:     None — clinical use prohibited
──────────────────────────────────────────────
Priority Remediation:
  1. Add clinical disclaimer (active penalty applied —
     clinical-adjacent tooling without R3)
  2. Implement CI/CD pipeline (T1: 0 pts)
  3. Disclose data provenance and IRB status (B1: 0 pts)
──────────────────────────────────────────────
⚠ This report is LLM-generated. It is not a regulatory
  determination. Report expires: 2026-09-16.

The expiry date is not cosmetic. A report based on a repository that went inactive months ago should not circulate in procurement pipelines as if it still describes current reality. Version 1.0.4 computes that date from recent project activity automatically.

What the Tiers Mean — and What They Do Not

T0 Rejected:        Trust not established
                    Clinical use prohibited

T1 Quarantine:      High risk
                    Independent verification required

T2 Caution:         Research reference only
                    Clinical automation forbidden

T3 Review:          Supervised pilot eligible
                    Human oversight mandatory

T4 (highest tier):  Strongest observed governance signals
                    Still requires independent expert review
                    and formal regulatory clearance

Even at the highest tier, STEM-AI does not substitute for clinical validation, expert review, or regulatory clearance. That is not a disclaimer added to soften the framework. It is how the framework was designed.

One Design Decision That Matters

Author background does not affect the score.

Whether the author comes from biology, medicine, ML research, or pure software engineering is recorded as contextual information for human reviewers. It is explicitly non-scoring and carries a mandatory bias warning in every report.

Domain credentials are not the same thing as engineering integrity. And lack of domain pedigree does not imply carelessness. STEM-AI is built to evaluate observable repository governance — not to sort developers by prestige.

What STEM-AI Refuses To Be

A trust framework for medical AI can become dangerous if it turns into a personal attack engine.

STEM-AI is constrained on purpose. It is limited to public professional repositories and public professional material. It forbids PII inference, private-account speculation, and individual profiling as a use case. Without that boundary, a trust evaluator becomes a harassment tool.

Three Questions This Framework Is Built Around

If you remember nothing else from this framework, remember these:

Did the repository describe its limits honestly?

Did public communication remain consistent with the repository's stated limits?

Did the codebase show evidence of maintenance and biological responsibility?

Those are not performance questions. They are accountability questions. For tools that sit upstream of clinical decisions, accountability is not optional.

What Comes Tomorrow

Tomorrow we publish the first audit set from STEM-AI v1.0.4 across 10 open-source bio-AI repositories — including projects from research institutions, an actively used SaaS platform, and agent-style bioinformatics tooling running inside containerized environments.

What we can say now:

The strongest-looking repositories are not always the most accountable. In at least one case, a repository with solid engineering signals still falls short because a critical disclosure is missing from the surface a reviewer would actually read first.

In another, the safety control exists in the code but fails as governance because it does not activate under default deployment.

In another, the generation mechanism itself is not the problem. The problem is that users were never clearly told what it was. That is a disclosure failure, not a technical failure. The distinction matters.

The full breakdown publishes tomorrow.

Which bio-AI repositories would you want audited next? Drop them in the comments.

STEM-AI v1.0.4 — full audit results tomorrow.

"Code works. But does the author care about the patient?"

The Schema Existed. The Model Had No Way to Know.

Kwansub Yun — Thu, 19 Mar 2026 17:11:28 +0000

Glossary: terms used in this article

🔸 MICA (Memory Invocation & Context Archive): A governance schema for AI context management. Defines how context should be structured, trusted, scored, and handed off across sessions.

🔸 Session Loss: The architectural characteristic of LLMs where no information persists between independent conversations. Not a bug. A design property with real engineering consequences for long-running projects.

🔸 Trust Class: The reliability classification of a context item's source. In this series: canonical (repo truth), distilled (summarized from sessions), raw (unprocessed session output), symbolic (reference only).

🔸 Invoke Role: A governance label that defines a context item's eviction behavior. In this series: anchor (never evict), bridge (preserve across phases), hint (drop first under pressure), none (drop immediately).

🔸 Invocation: The mechanism by which a MICA archive reaches an AI session. Without explicit invocation, the archive exists but has no effect on the session. Formalized as a required field in v0.1.8.

🔸 Admission Gate: The point at which a context item is evaluated for inclusion. Decides what goes in — before output validation begins.

1. What v0.0.1 Was

Part 1 established the problem: session loss is not an inconvenience. For long-running projects, it is a structural failure. The standard responses — longer prompts, RAG, session summaries, flat memory exports — all treat context as a document to be read, not a governed structure with authority levels, eviction rules, and provenance.

v0.0.1 was the first attempt at a specification-level answer.

It proved that context could be structured. It did not prove that the structure could govern what context is allowed to shape the session. Those are different claims. v0.0.1 satisfied the first and failed the second — in three distinct ways.

Only one of those failures made the other two meaningless.

2. A Different Layer

While working through the problem, I was reading broadly. arXiv papers on memory management. Dev.to articles on LLM reliability. Community discussions about what people had tried and where it broke.

One article in particular was useful: Why Asking an LLM for JSON Isn't Enough. In that article and the discussion it generated, a clear framing had emerged:

Treat the LLM as an unreliable upstream service. Add schema, validation, retry, fallback.

The operative mental model was this:

LLM → output → validate → retry → fallback → use

This framing is correct, and useful, for what it solves: defensive parsing and output reliability. None of it addressed admission control — denying a context item before it reached the model at all.

The question I needed to answer was structurally different:

Context item → admitted? → trust level assigned → provenance checked → eviction priority set

The first asks "how do I handle what comes out?" The second asks "what goes in, and under what rules?"

MICA is not a replacement for output validation. It is the upstream governance layer that decides what is allowed to shape the session before output validation even begins.

3. Three Failures in v0.0.1

Failure 1: No defined semantics for scoring.

scoring_policy:
  relevance_inputs:
    - "query similarity"
    - "recency"
    - "invoke_role"
    - "trust_class"
    - "capsule continuity weight"
  scoring_hints:
    canonical_memory_bonus: 0.15
    continuity_bridge_bonus: 0.05
    raw_logs_penalty: 0.20
    symbolic_only_penalty: 0.10

These were hardcoded test values. They existed to confirm that the pipeline could apply differential weights at all — not to define what those weights should be.

The problem was not that the numbers were heuristic. The problem was that the heuristics had no defined semantics — no defined rule explained how they produced a score.

There was no combination rule.
No output range.
No normalization.

What canonical_memory_bonus: 0.15 meant relative to raw_logs_penalty: 0.20, how those four numbers combined into a final score, what the result represented — none of that was specified.

A conforming implementation was not possible. That means two tools claiming to implement v0.0.1 could rank the same archive differently and both still claim compliance.

Failure 2: Invariants encoded as comments, not constraints.

v0.0.1:

design_invariants:
  - "Context Broker must not generate SovDef."
  - "LawBinder must never receive raw Context objects."
  - "Only working_context may be passed to LLM-facing layers."

A plain list of strings. No id. No severity. No track. They were written as constraints, but encoded as comments. The difference matters: a constraint has enforcement. A note does not. These strings could not be machine-evaluated, could not be diff-checked against session behavior, and could not be reliably extracted by a model reading the archive.

What a machine-actionable invariant requires:

design_invariants:
  - id: INV-001
    rule: "Context Broker must not generate SovDef."
    severity: "critical"
    enforceable: true
    track: "compile-time"

Without id, an invariant cannot be referenced in audit output. Without severity, a violation cannot be triaged. Without track, enforcement has no defined point of application. v0.0.1 had none of these.

Failure 3: No path to the model.

There was no invocation_protocol. No session-start procedure. No confirmation that the archive had been loaded. The model would begin a session with no instruction to locate the archive, no way to confirm it had been read, and no defined behavior for what to do at session start.

The archive existed. The model had no reliable way to know it existed.

A new session could start, answer confidently, and never once acknowledge the archive it was supposed to be governed by.

4. Why Failure 3 Made the Others Meaningless

The three failures are not equal.

Failure 1 is a calibration problem. The formula can be fixed. The weights can be justified. That work happened in subsequent versions.
Failure 2 is a schema design problem. Fields can be added. Structure can be enforced. That work happened in the same range.
Failure 3 is different in kind. It is not a field that needs to be added. It is a requirement that the schema must define how it reaches the model at all. A context item classified as anchor with eviction_priority: 3 has no enforcement if the model never sees the archive. F

ailures 1 and 2 describe a schema that could not govern correctly. Failure 3 describes a schema that could not govern at all.

A context system does not fail only when it forgets. It also fails when it remembers without governance — and v0.0.1 could not even guarantee the second.

Every version from v0.1.0 through v0.1.7 addressed the first two failures. Scoring became implementable. Invariants gained structure. Eviction became a five-phase strategy. Error handling was defined.

The invocation problem was not formally addressed until v0.1.8, which introduced invocation_protocol as a required field. It declares how the archive reaches an AI session, what pattern is used, and what the session opening report must contain.

That is the distance between v0.0.1 and v0.1.8.

5. What This Series Covers Next

Part 1 defined the problem.

This part documented v0.0.1 — what it was, what it got wrong, and why the most fundamental failure took the longest to fix.

Part 3 covers v0.1.0 through v0.1.5: how scoring moved from hardcoded guesses to an implementable formula, what the eviction strategy revealed about context budget assumptions, and what was still missing at v0.1.5.

The series continues only where there is something concrete to specify, test, or correct.

MICA is part of the Flamehaven governance-first AI systems practice. Schema, technical report, and production instance: flamehaven.space. Open-source tooling: AI-SLOP-Detector. All schema references follow the v0.1.8.1 Universal standard unless a specific earlier version is named.

My LLM Kept Forgetting My Project. So I Built a Governance Schema.

Kwansub Yun — Mon, 16 Mar 2026 09:19:41 +0000

Disclosure: This article was written by the author with AI assistance for editing. All technical content, architecture decisions, and design rationale are the author's own.#ABotWroteThis

Glossary: terms used in this article

🔸 MICA (Memory Invocation & Context Archive): A governance schema for AI context management. Defines how context should be structured, trusted, scored, and handed off across sessions.

🔸 Anchor Item: A context item that cannot be dropped under any memory pressure. Eviction priority: 0. Later parts explain how this is enforced at the schema level.

🔸 Semantic Collapse: A pattern introduced in later parts of this series. A JSON Schema is applied to an LLM as a runtime contract rather than as a validator.

🔸 Fail-Closed Gate: An admission rule that excludes a context item if it fails any defined threshold. No exceptions. Formalized in v0.1.7.

1. The Problem

LLMs do not retain state between sessions. Every conversation starts from zero.

For a single task, that is manageable.

For a project maintained across dozens of sessions, it is not.

Such a project accumulates architectural decisions, non-negotiable constraints, protected files, and a decision history explaining why things are the way they are. In that setting, the lack of continuity becomes a structural failure with compounding consequences.

The failure mode is insidious. The model does not fail visibly. It produces well-formed, internally consistent output. What it cannot know is which of its inferences are wrong, because the context it received was incomplete. It cannot identify what is missing. Instead, it fills the gaps silently, drawing on training data rather than on the project's actual history.

Standard responses address parts of the problem. Longer system prompts, RAG pipelines, session summaries. None addresses the governance layer: which context items are authoritative, which are provisional, how they should be weighted against one another, and what happens when memory pressure forces eviction.

This post documents the specification I built to address that layer.

2. Where This Started

For a long time, my workaround was simple and inefficient. Copy a conversation from one AI, paste the whole thing into another, and ask the second one to analyze it. The idea was to get a fresh perspective. To avoid the first model's blind spots by bringing the work to a different context.

It rarely worked cleanly. The second model would inherit the framing, the vocabulary, even the confident wrong assumptions of the first session. It would pick up the conclusion without the reasoning errors that led to it. Then build on both. This pattern is well-documented in multi-model workflows. The receiving model anchors to the original session's assumptions rather than evaluating the content independently. Researchers studying AI review and self-critique systems have noted the same anchoring effect. Same-session review produces systematically worse error detection than fresh-session review.

So I kept looking for a better way to carry context across sessions.

At some point, I came across Claude's memory feature. Not while looking for it specifically. While exploring settings. Claude had introduced persistent memory for paid plans in October 2025, with the stated goal that users shouldn't have to re-explain their context at the start of every session. Anthropic's own description: "your first conversation feels like your hundredth." The export feature produces structured output. Categories, dates, entries, one per line.

Conceptually, this is similar to Claude's Skills system: a modular, reusable layer that carries working state between sessions without rebuilding it from scratch each time. The intent is sound.

I used it. It helped. But something was consistently off.

The output looked like this:

[2025-02-11] - user prefers concise responses
[2025-02-11] - medical AI governance pipeline must never skip the evidence gate
[2025-02-11] - currently exploring whether to add a search endpoint
[2025-02-11] - low-N prohibition is non-negotiable for clinical outputs

Four entries. Same format. Same weight. But they are not the same kind of thing. The second and fourth are hard constraints. In a medical AI context, violating them is not a quality issue. It is a safety issue. The first is a style preference. The third is an open question that may be abandoned next week. The export format has no mechanism to express that difference.

Others in the community have hit the same wall in different ways. From r/ClaudeAI:

A custom MCP memory server still caused the model to skip stored context ~40% of the time
"Notes are lossy. They capture what I thought was important, not what Claude actually found important."

The pattern is consistent. The flat format puts the entire prioritization burden on the user.

The Governance Gap

Anthropic labels the memory import feature experimental, with the explicit caveat that Claude may not always successfully incorporate imported memories. The export is a snapshot. What it cannot express is governance: which items are anchors, which are provisional, what survives memory pressure, and what the source of each item actually was.

At the same time, I had been building AI-SLOP-Detector. A static analysis tool for detecting low-quality AI-generated code. Its core premise: not all signals are equal. Some patterns are weighted more heavily. Some trigger hard blocks regardless of aggregate score. The scoring model is explicit, versioned, and auditable.

The same structure was missing from context management. A flat export has no weights, no eviction rules, no provenance, no trust hierarchy. It is a list. A governance schema is something different.

That gap is where MICA (Memory Invocation & Context Archive) started.

3. The Structural Problem with LLM Context

Language models do not retain information between sessions. This is a known architectural property, not an implementation gap. Every new conversation begins from zero.

For short, self-contained tasks, this is manageable. For long-running engineering projects, it is not. Codebases under active development. Governance systems with accumulated decision history. Architectures with non-negotiable invariants. Each creates a specific and compounding failure mode.

The failure is not that the model performs poorly. It is that the model performs well. Confidently. Using whatever context it was given. That context is almost always incomplete in ways the model cannot detect.

Consider what a project accumulates over months of development:

Type of knowledge	Survives session loss?	Consequence of loss
Current file contents	Partially (if re-provided)	Recoverable
Architecture decisions and rationale	No	Silent regression risk
Constraints that are non-negotiable	No	Violated without awareness
Solutions already tried and abandoned	No	Repeated work, repeated failures
Trust levels of different information sources	No	All context treated equally

The model cannot distinguish between a constraint that is load-bearing and one that is provisional. It does not know which decisions have downstream dependencies. It cannot identify that a suggested refactoring was already evaluated and rej

ected for a reason not present in the current context.

The result: the model fills gaps with plausible inference. Inference drawn from training data, not from project history.

4. Why the Standard Fixes Don't Fully Work

Every developer running a long project with an LLM eventually hits the wall and builds something to address it. The community has been running these experiments for a while now.

The diagram above shows the pattern. Each approach handles one or two dimensions. None handles governance.

Long system prompts are the first instinct. Write everything down, start every session with it. It works until maintaining the prompt becomes a second job. One developer described it precisely: "the scaffolding becomes the work." There is also a structural issue no token count fixes: the "lost in the middle" effect. Models attend well to the beginning and end of long contexts and degrade in between. An architecture constraint buried in paragraph 11 of a 3,000-token prompt may not receive adequate attention regardless of whether it fits in the window.
RAG pipelines handle knowledge injection well. They do not handle governance. Retrieving a paragraph about how a caching layer works is different from the model understanding that this specific caching layer cannot be modified without a deviation log entry. RAG provides facts. It does not provide the weight of facts.
Session summaries are a reasonable manual workaround and the one I used most. The problem: summarization is lossy by design.
Summaries strip the rationale behind decisions. The model gets the conclusion without the reasoning that produced it. When a new edge case appears that challenges an earlier decision, the model has no basis to evaluate whether the constraint still holds. It just follows the summary.
Flat memory exports — including Claude's own export format — do carry context across sessions. That part works. What they cannot express is priority. Which items are non-negotiable. Which are provisional. Which are stale. The burden of sorting that out falls entirely on the user, every time.

The common limitation across all four: they treat context as a document to be read. None treats it as a governed structure where different items have different trust levels, different eviction protection, and different behavior when they conflict.

5. Looking for an Answer

The question was whether anyone had already solved this at the specification level.

The research on LLM memory management is substantial. MemGPT (Packer et al., 2023) introduced the OS analogy: main context as RAM, external storage as disk. Later work extended this into hierarchical architectures and tiered storage models.

A 2025 survey (arXiv:2509.18868) maps the landscape and documents a problem the field had started to name: self-degradation. Naive "add everything" strategies cause memory inflation. Inflation leads to error propagation. The agent performs worse over time. Not better.

The direction is clear: memory needs to be tiered, decayed, and managed. Not accumulated.

What I could not find was a specification for the governance layer. Not for a multi-agent research system. For a single developer maintaining a real project across sessions.

How to express that some items are non-negotiable anchors and others are provisional notes.
How to define eviction behavior a conforming implementation must follow. How to require provenance.
How to version and audit context state the way you version and audit code.

The existing systems were engines. What was missing was a schema.

That is what v0.0.1 attempted. The first version was rough. It got several things wrong. Part 2 covers what those failures revealed.

6. What This Series Covers

That is the problem. MICA is the attempt at a specification-level answer to it.

This is Part 1. It documents the problem and the motivation.

Part 2: v0.0.1. The first schema version, what it defined, what it got wrong, and why those failures revealed exactly the problem this series is trying to solve.

The series runs as long as there is something concrete to document.

Named failure mode from this post: the governance gap. Context systems that store what you said, but not what it meant, where it came from, or how much it mattered.

MICA is part of the Flamehaven governance-first AI systems practice. Schema, technical report, and production instance: flamehaven.space. Open-source tooling: AI-SLOP-Detector. All schema references follow the v0.1.8.1 Universal standard unless a specific earlier version is named.

DEV Community: Kwansub Yun

AI-SLOP Detector v3.5.0 — Every Claim, Verified Against Source Code

What was claimed

Claim 1: "Every scan is recorded"

Claim 2: "Every re-scan becomes signal"

Claim 3: "Updates only when the signal is strong enough"

Claim 4: "Leaves behind a visible policy every time it changes"

Claim 5: "Explicit limits govern calibration"

Claim 6: "Detects empty implementations, phantom dependencies, disconnected pipelines"

Claim 7: "~1.4K downloads in the past week"

Why this format exists

It Gets Smarter Every Scan: AI-SLOP Detector v3.5.0 and the Self-Calibration Loop

The breach is the headline. The review gap is the story.

We ran into this ourselves

Static analyzers have a threshold problem

What AI-SLOP Detector actually does

The workflow is the product story

Self-calibration is the real headline

What the loop actually looks like

Why --init matters more now

JS, TS, and Go are not side quests

The honest boundary

Why this matters now

Quick start

Can AI Review Physics? Yes — That Is Why We Built SPAR

What SPAR Is

Why Ordinary Review Is Not Enough

A Minimal Divergence

The Core Mismatch Classes

The Three-Layer Structure

Layer A — Anchor Consistency

Layer B — Interpretation Validity

Layer C — Existence and Maturity Probes

Why the Registry Matters

Scoring Policy Is Explicit

A Concrete Example: The Omega Score Transition

Quick Start

Where This Framework Fits

A Lightweight Adoption Path

What SPAR Does Not Do

Reliability Is Not Enough

AI SLOP Detector v3.1: Three Formula Refinements and the Adversarial Tester That Found Them

Glossary — internal terminology used throughout this post

Quick context

Before v3.1.0: the self-scan

The adversarial tester: fhval SPAR

Refinement 1: The calibrator and scorer were using different formulas

Refinement 2: The complexity modifier had a dead zone at the common end

Refinement 3: Purity weight was documented but not connected

Two new detection patterns

Stub evasion: empty container returns

Fragmented god function: AST clone detection

Placeholder variable naming (v1.0)

SPAR result after v3.1.0

v3.1.1: the self-inspection patch

How this fits alongside existing tools

Score evolution

Honest limitations

Install / upgrade

My AI Maintainer Kept Making Wrong Calls. So I Made It Report Its State Before Touching Anything.

🔎 Glossary: terms used in this article

1. What Part 5 Left Open

2. The Session Opening Report

3. What the Self-Test Actually Checks

4. What Drift Detection Looks Like in Practice

5. What Happens When a Deployment Changes Something

6. What the Design Invariants Actually Govern

7. What MICA Cannot Do Here

8. What Part 7 Will Address

How Auditing 10 Bio-AI Repositories Shaped STEM-AI

What Text Could See — and What Code Revealed

What Code Access Makes Visible

Four Directions the Audits Opened

Direction 1: Clinical exposure is visible in imports, not just in README text.

Direction 2: Not all clinical proximity is the same.

Direction 3: Scoring stability is worth measuring directly.

Direction 4: Code-path behavior deserves its own scan layer.

What v1.0.6 Added — Carried Forward in v1.1.2

1. Two Evidence Paths, Not One

2. Clinical Dependency Detection

Why `--init` matters more now