DEV Community: Kwansub Yun

From Score to Workflow: Turning STEM BIO-AI Into a Local Audit System

Kwansub Yun — Fri, 08 May 2026 08:25:50 +0000

Earlier in this series, I wrote about why bio/medical AI repositories need more than benchmarks, what I learned after auditing 10 public repositories, and why an AI auditor itself needs a memory contract.

That work led to STEM-AI v1.1.2 and the MICA layer: a memory-contracted initialization step that forces the auditor to load bounded rules before scoring begins. If you have not read that part, the relevant post is here:

How Do You Trust the AI Auditor? STEM-AI v1.1.2 and Memory-Contracted Bio-AI Audits

For the broader arc, the full series is here:

STEM-AI / STEM BIO-AI series

But after that, a different engineering problem took over.

The audit logic was stricter.

The reports were richer.

The reasoning was more bounded.

But the developer workflow still felt too loose.

So the next question was no longer:

How do I score trust?

It became:

How does a bio-AI audit tool become something an engineer can actually run, gate, inspect, and integrate?

The answer turned out to be less about seeing more signals and more about refusing to confuse them.

That is the core argument of this post:

A detector becomes more trustworthy when it is strict about what it cannot conclude.

Once I took that seriously, STEM BIO-AI stopped looking like “one score plus some extra metadata” and started looking like a system with distinct lanes, distinct boundaries, and distinct operator workflows.

The problem was no longer scoring

By the time I reached the 1.6.x line, the rubric was no longer the main bottleneck.

The bottleneck was operational clarity.

A trust audit tool is not very useful if:

the normal path is one long command with too many flags
CI has to reverse-engineer the result from human-readable stdout
bio-specific diagnostics are mixed directly into the same surface as formal scoring
regulatory relevance shows up as vague implication instead of explicit traceability
advisory AI is present, but its relationship to the official score is unclear

At that point, the tool stops being hard to trust for conceptual reasons and starts being hard to trust for operational reasons.

That is a different class of problem.

The CLI had to reflect operator intent

The earlier CLI was functional, but too flat.

You could do things like this:

stem /path/to/repo --level 3 --format all --explain
stem /path/to/repo --tier-gate T3 --format json --quiet
stem /path/to/repo --advisory packet
stem /path/to/repo --advisory-response provider_advisory.json

All of that worked.

The issue was that it treated very different operator intents as one long option surface.

In practice, these are separate workflows:

scan a repository and generate artifacts
enforce a gate in CI/CD
export a bounded advisory packet
validate a downstream provider response
cross an explicit provider-call boundary

So I refactored the CLI around workflows instead of flag accumulation:

stem scan <folder>
stem gate <folder> --min-tier T2
stem advisory validate <folder>
stem advisory packet <folder>
stem advisory call <folder>
stem advisory check-response <folder> --response FILE

The older paths still exist for compatibility:

stem <folder>
stem audit <folder>
stem <folder> --tier-gate T2 --quiet
stem <folder> --advisory packet

But they are no longer the conceptual center.

That matters more than it sounds.

Once the command names match the operator’s intent, the system becomes easier to teach, easier to remember, and easier to wire into pipelines.

This is not just a DX cleanup. In a medical or bio-adjacent audit context, command ambiguity is part of the trust problem.

Repository trust needed four separate lanes

This was the biggest architectural shift.

I stopped treating repository trust as one object.

In practice, it needed four separate lanes:

deterministic structural scoring
deterministic diagnostics
regulatory traceability
optional AI advisory

If all of those collapse into one final confidence score, the tool becomes harder to reason about.

The more regulated the domain, the more dangerous it becomes to collapse every useful signal into one score.

Some evidence should change the score.
Some evidence should only raise review priority.
Some evidence should support traceability.
Some evidence should be handed to a human or advisory system.

The maturity of the tool is not that it sees all of them.

The maturity is that it does not confuse them.

This separation is not just conceptual. It exists in the code path.

One reasonable objection to any architecture write-up is: are these really separate lanes, or are they just different labels on the same output object?

In STEM BIO-AI, the answer is visible in the execution order.

The scanner computes the formal score first. In the result object, that means keys like:

Stage 1
Stage 2R
Stage 3
risk penalty
score cap
final_score
formal_tier

Only after that does it append the non-scoring layers, again as explicit result keys:

regulatory_basis
stage_traceability
regulatory_traceability
reasoning_model
optional ai_advisory

That ordering matters.

The score is not derived from the advisory lane.
The regulatory mapping does not mutate the formal tier.
The diagnostics lane can emit evidence without becoming a hidden score multiplier.

This is also why the JSON shape ended up more layered than earlier versions. The output had to preserve the distinction the code was already enforcing.

That execution order is the architectural reason the next four sections exist.

Once I had the lanes separated in code, each lane needed its own claim boundary, its own output semantics, and its own reason for not being collapsed into the others.

Put differently, the next four sections answer four different questions:

what is allowed to change the formal tier
what is useful enough to emit, but not yet mature enough to score
what can support regulatory review without pretending to be compliance
what can involve AI without letting AI become the scoring authority

1. Deterministic structural scoring

This remains the official score and tier.

It measures the main repository-visible signals:

README evidence
repo-local consistency
code and bio responsibility
dependency hygiene
changelog and provenance surfaces
code-integrity patterns

This lane is local, deterministic, and machine-checkable.

That is the part that can legitimately drive a formal triage tier.

I am not claiming this is the only possible architecture. A different system could have folded diagnostics or replication more aggressively into one unified score.

I chose not to, because the narrower score proved easier to defend. A smaller claim with cleaner boundaries was more valuable here than a broader score with ambiguous semantics.

2. Deterministic diagnostics

This is where the deterministic diagnostics spec became important.

I needed a place for findings that are real, useful, and inspectable, but should not silently perturb the main score until they are calibrated.

That is what docs/DETERMINISTIC_DIAGNOSTICS.md defines.

It separates the diagnostic problem into two lanes:

Lane A: deterministic local diagnostics
Lane B: optional AI-assisted semantic review

That separation is central.

The deterministic lane is authoritative for hard findings.
The AI lane is advisory only.

The local diagnostic lane currently focuses on evidence-bearing bio-specific signals such as:

malformed or suspicious SMILES-like outputs
missing parser guards
silent mock or simulated-data fallbacks
risky subprocess construction around bio tools
traceability manifest surfaces

The point was not to create a “bio slop detector” with a catchy label.

The point was to create a local evidence lane that could say:

here is the file
here is the line
here is the snippet
here is the bounded interpretation

That is much more useful than a vague semantic warning.

Why diagnostics stayed evidence-only

This was one of the harder engineering decisions.

It would have been easy to push every new bio-specific detector directly into the final score.

I did not do that.

The deterministic diagnostics spec is explicit that many of these findings begin as evidence-only. In practice, they are emitted as line-level records in the result object's evidence_ledger:

findings are emitted into the result object’s evidence_ledger
findings appear in Markdown and --explain
findings do not change final_score or formal_tier

That is the right default.

For example, the SMILES lane can be very useful for detecting:

malformed surface strings
low-entropy placeholders
repeated trivial outputs
missing parser guards

But it does not prove:

medicinal usefulness
synthetic feasibility
binding plausibility
biological efficacy
full chemical validity in every edge case

That boundary is important.

A detector becomes more trustworthy when it is strict about what it cannot conclude.

Just as importantly, this is not meant to be a permanent holding area for every detector. The diagnostics spec is explicit that score impact should only happen after commit-pinned benchmark evidence, explicit false-positive review, and reproducible calibration. In other words, evidence-only is the temporary safe default until a detector has earned score authority.

3. Regulatory traceability

The second document that became central was docs/REGULATORY_MAPPING.md.

This solved a different problem.

Once you audit clinical-adjacent repositories, people naturally ask:

does this align with EU AI Act themes?
does this help with FDA-oriented review?
is there anything relevant to IMDRF or SaMD evidence families?

The wrong answer would be to turn those questions into a fake compliance score.

So I did the opposite.

The regulatory layer is explicitly framed as:

a traceability aid, not a compliance verdict

That document maps observed evidence classes to requirement families with bounded confidence labels like:

strong
moderate
weak-moderate
weak
not assessed

And it makes an important distinction:

the confidence applies to the mapping relationship, not to legal acceptability.

Those confidence labels are not model outputs and they are not inferred at runtime. They are fixed, rule-level mapping judgments attached to evidence classes in the mapping document itself. For example, changelog / checksum / config-manifest style evidence is treated as a moderate traceability signal for Article 12-style review, while human-oversight interface signals stay weak because interface presence is not the same thing as oversight procedure.

That means the tool can say things like:

versioned manifests and changelogs may support record-keeping / traceability review
intended-use and disclaimer sections may support transparency scaffolding review
override interfaces may support human-oversight interface review
subgroup measurement language may support weak evidence of data-governance intent

without claiming:

legal compliance
regulatory clearance
clinical certification
deployer conformance

In a regulated domain, traceability is useful only when it does not pretend to be permission.

A concrete example: why Article 12 is traceability, not compliance

The best example here is EU AI Act Article 12 style traceability.

The regulatory mapping layer treats signals like:

changelogs
checksum manifests
versioned config surfaces
audit-log schema fragments
decision-event or override-event schema tokens

as evidence that a repository may have traceability scaffolding.

That is useful.

It is also bounded.

The mapping document is explicit that changelog presence is not the same thing as deploy-time event logging, and that current scope does not establish runtime log completeness.

So the output can legitimately say:

there is structural evidence relevant to traceability review

while refusing to say:

this system satisfies traceability obligations

That is exactly the kind of distinction I wanted this lane to enforce.

What this buys in practice is not a compliance shortcut, but a faster review question. If a repository exposes none of the scaffolding signals in this lane — no change history, no artifact hashes, no versioned manifests, no event-schema surfaces — then there is very little reason to treat it as traceability-ready for deeper institutional review. If those signals do exist, the next step is still expert inspection, but the scanner has at least opened the right folder and pointed at the right files.

Why regulatory mapping stayed subordinate to evidence

This was non-negotiable.

Regulatory relevance had to remain downstream from evidence, not a score multiplier pretending to be law.

That is why the output shape separates things like:

regulatory_basis
stage_traceability
regulatory_traceability

from the actual score computation.

And it is not just decorative structure.

The regulatory basis object is registry-driven. It can mark review_required when the basis registry is stale or required source families are missing. That is a traceability control on the mapping layer itself, not an input into the scoring formula.

This is also why the regulatory note belongs in a muted traceability panel, not next to the main score.

If a repo has traceability-relevant scaffolding, that is useful.

If a repo has traceability-relevant scaffolding, that is still not compliance.

The distinction has to remain visible in both the code and the artifacts.

4. Optional AI advisory

The fourth lane is the advisory layer.

This one exists for bounded model-assisted review, but it does not get to rewrite the official outcome.

That means workflows like:

stem advisory packet /path/to/repo
stem advisory check-response /path/to/repo --response provider_advisory.json

can exist without creating ambiguity about who owns the formal result.

The advisory layer can:

export a provider-neutral packet
validate downstream response structure
enforce finding-ID citation rules
reject prohibited claims
surface runtime and secret boundaries

What it cannot do is silently override:

score.final_score
score.formal_tier

How that rule is actually enforced

This is not just policy language in the README.

The advisory validator explicitly checks for score-override attempts. If a response includes fields like:

final_score
formal_tier
replication_score
replication_tier

or sets final_score_override, the response is marked invalid with final_score_override_requested.

The packet contract also exports the rule in plain language:

Do not modify or override final_score, formal_tier, replication_score, or replication_tier.

And provider responses must cite exact values from allowed_finding_ids; citation strings are not repaired or loosely matched later.

So the advisory lane is bounded in two ways:

it has no authority to change the deterministic result
it cannot cite evidence outside the bounded packet

That is the kind of mechanism I mean when I say “better boundaries.” If the rule cannot be checked, it is not really part of the architecture yet.

What operational use looks like now

Once these lanes were separated, the CLI became much easier to reason about.

Local engineering review:

stem scan /path/to/repo --level 3 --format all --explain

CI/CD gate:

stem gate /path/to/repo --min-tier T2 --summary off --output results

Offline advisory packet generation:

stem advisory packet /path/to/repo --output advisory_out

Downstream provider response validation:

stem advisory check-response /path/to/repo --response provider_advisory.json

The important point is not just that these commands exist.

It is that each one represents a distinct trust boundary.

That made the project feel more like engineering infrastructure and less like a scoring demo.

A real v1.6.2 packet

To make that less abstract, I re-ran STEM BIO-AI v1.6.2 against a local clone of ClawBio, which describes itself as a local-first, privacy-focused, reproducible bioinformatics-native AI skill library.

The command was:

python -m stem_ai.cli scan /path/to/ClawBio --level 3 --format all --explain

On my machine, that run took about 9.4 seconds and emitted the usual CLI output set: a machine-readable JSON result, a Markdown report, a 5-page PDF packet, and a line-level explain trace.

Before the numbers, the important context is that STEM BIO-AI uses a published triage scale:

T0 = 0-39
T1 = 40-54
T2 = 55-69
T3 = 70-84
T4 = 85-100

Stage 4 replication is reported separately as its own lane, where R2 means some reproducibility scaffolding is present, but not yet enough to call the repository replication-strong.

Governance note:
This is not a “bad repository” scoreboard, a clinical safety verdict, or a moral ranking. It is a deterministic evidence-surface pre-screen intended to support review, not replace it.

With that in mind, the result was:

67 / 100
T2 Caution
Replication lane: 55 / 100 (R2)
Clinical adjacency: CA-DIRECT (the repository surface makes direct healthcare-facing claims, even though it also carries an explicit non-clinical boundary)
Code integrity warnings: C2 dependency pinning, C4 exception handling

This is exactly the workflow shift I wanted the tool to support.

The same deterministic scan is rendered into multiple operator surfaces:

JSON for automation
Markdown for review
PDF for human-facing packet inspection
--explain for file / line / snippet proof tracing

That output shape is only possible because the result object already separates:

formal score and tier
replication lane
diagnostics lane
regulatory traceability
advisory boundary state

In other words, the PDF is not a separate product. It is a view over the same bounded audit object.

Two details from this run are worth calling out.

First, the scanner did not manufacture chemistry findings just because ClawBio is bio-adjacent. The deterministic diagnostics lane reported:

SMILES Surface Integrity: not_detected
SMILES RDKit Validation: not_applicable
SMILES Parser Guard: not_detected

That is the behavior I want. If a detector has no evidence, it should stay silent instead of inflating the report with domain-flavored noise. This is what the earlier thesis looks like when it hits real output: a detector becomes more trustworthy when it is strict about what it cannot conclude.

Second, the score is strict about observable repository conventions. ClawBio uses ClawBio_README_Repo.md rather than a root README.md, so the scan records S1_missing_readme: -20. A human reviewer might decide that this is acceptable contextually. The scanner does not make that leap for them. It only records what the repository exposes through the surfaces it knows how to measure.

That distinction matters. A T2 Caution result here does not mean “ClawBio is unsafe.” It means the current repository surface still raises review-relevant signals under the published deterministic rules, including dependency-pinning warnings, exception-handling warnings in a clinical-adjacent surface, and a stricter-than-human README convention check.

And that is exactly why the next section matters: once the workflow is concrete, the remaining question is not whether the tool can produce an answer, but where its current boundaries still need to stay visible.

What still has to stay bounded

The system is better than it was, but there are still obvious next steps.

1. The public surface is broad

There is now:

scoring
diagnostics
replication
advisory packeting
regulatory traceability
JSON / Markdown / PDF / explain outputs

That is useful, but it increases onboarding cost.

The CLI is clearer now, but the broader public surface has to stay disciplined.

2. The deterministic diagnostics lane is still missing a published calibration threshold

The diagnostics lane is evidence-first by design, but one practical gap remains: the public release does not yet ship a benchmark-backed threshold document saying exactly when a detector is mature enough to graduate from evidence-only into score-bearing territory.

Right now the rule is conceptually clear:

commit-pinned fixtures
reproducible detector output
explicit false-positive review

But the public decision boundary is still partly narrative. Until that calibration surface is published in a more operational form, keeping diagnostics evidence-only is the safer choice.

3. The regulatory confidence labels are rule-authored, not empirically validated

The mapping labels like strong, moderate, and weak-moderate are currently fixed rule-level judgments in the mapping document. They are not runtime model outputs, but they are also not yet backed by inter-rater reliability studies or a published reviewer-agreement benchmark.

That means they are useful as bounded structural mapping language, but they should not be treated as empirical proof that multiple auditors would converge on exactly the same label distribution.

Earlier context

Try it yourself

STEM BIO-AI is Apache 2.0 and fully open source.

If you want to know whether a bio/medical AI repository is actually exposing reviewable evidence, or whether your own repository is weaker than you think, run it yourself.

That is the real test.

GitHub: https://github.com/flamehaven01/STEM-BIO-AI
License: Apache 2.0

Final thought

The earlier STEM-AI posts were about why repository trust deserves its own audit layer.

This phase was about something more practical:

what does that audit layer have to look like if an engineer is actually going to run it, inspect it, and put it in a pipeline?

For me, the answer was simple:

Separate the workflows.
Separate the lanes.
Keep diagnostics evidence-first.
Keep regulatory mapping subordinate to evidence.
Keep advisory AI bounded.

Optimize for inspectability, not just score production.

That is what changed the project.

Not bigger claims.

Better boundaries.

How Do You Trust the AI Auditor? STEM-AI v1.1.2 and Memory-Contracted Bio-AI Audits

Kwansub Yun — Tue, 28 Apr 2026 13:51:45 +0000

Previous article:
How Auditing 10 Bio-AI Repositories Shaped STEM-AI

In the first STEM-AI write-up, I described what happened after auditing 10 open-source bio/medical AI repositories.

The important lesson was not just that some repositories lacked clinical disclaimers, tests, or governance artifacts.

The more useful lesson was this:

Text-only review is too weak for bio/medical AI. You have to inspect the code path.

That worked.

But it exposed the next problem.

If an AI system is auditing another AI or bioinformatics repository, how do you trust the auditor?

LLMs drift.
One session can enforce a clinical boundary strictly.
Another can invent a generous middle score for the same boundary case. In normal software review, that is annoying. In medical AI governance, it is a liability.

STEM-AI v1.1.2 is my answer to that problem.

It does not try to make the LLM deterministic by writing a longer prompt.

It binds the audit to a memory contract.

What v1.1.2 adds

STEM-AI v1.1.2 introduces MICA: Memory-Injected Contract Architecture.

The idea is simple:

before the auditor reads the target repository, it must load a fixed audit contract and self-check the rules it is not allowed to bend.

The v1.1.2 layer includes:

memory/mica.yaml -- composition contract
memory/stem-ai.mica.v1.1.2.json -- machine-checkable memory archive
memory/stem-ai-playbook.v1.1.2.md -- session playbook and drift guard
memory/stem-ai-lessons.v1.1.2.md -- historical failure-mode archive
spec/STEM-AI_v1.1.2_CORE.md -- canonical audit spec

The contract pins 18 invariants.

Examples:

Stage order is fixed: README intent, cross-platform evidence, code/bio evidence.
Stage weights are fixed.
Tier boundaries are fixed.
T0_HARD_FLOOR cannot be bypassed.
Stage 2 may use external evidence or Stage 2R repo-local consistency in LOCAL_ANALYSIS mode.
Governance overlay cannot raise the formal base tier.
C1-C4 code-integrity checks only run in LOCAL_ANALYSIS mode.
Mandatory clinical-use disclaimers cannot be omitted.

This is not a claim that the LLM becomes perfectly deterministic.

It is a narrower claim:

The auditor is forced to operate inside a contract whose scoring rules, hard floors, and evidence requirements are inspectable.

That is the useful layer.

What "loading the contract" means

MICA is not hidden model memory.

It is also not a claim that the model provider changed the LLM.

In v1.1.2, "loading the contract" means the audit session starts by reading a fixed set of repository files before it is allowed to score the target:

memory/mica.yaml
memory/stem-ai.mica.v1.1.2.json
memory/stem-ai-playbook.v1.1.2.md
memory/stem-ai-lessons.v1.1.2.md
spec/STEM-AI_v1.1.2_CORE.md

The auditor then performs a pre-execution contract test:

confirm the canonical spec exists
confirm the memory archive exists
confirm the invariant count is 18
confirm the fixed tier boundaries are present
confirm the Stage 2 / Stage 2R lane rule is present
confirm Stage 3G cannot raise the formal tier
confirm C1-C4 mode gating is active

Only after that does the audit proceed.

This does not make the LLM mathematically deterministic.

It makes the audit procedure file-backed, inspectable, and interruptible. If the session cannot load or reconcile the contract files, the correct behavior is to stop before scoring.

That is the difference between "please be consistent" and "execute this versioned contract."

The audit workflow

STEM-AI v1.1.2 runs as a structured audit workflow:

In LOCAL_ANALYSIS mode, the auditor is not limited to what the README says.

It can inspect:

package metadata
workflow files
test definitions
dependency manifests
source-code paths
deprecated or dead-code paths
exception handling
credential patterns
provenance and hash-checking logic

The output is intentionally split into two files:

report.md                  # human-readable audit judgment
experiment_results.json    # machine-readable evidence and score object

That split matters.

The report explains the reasoning.

The JSON lets another reviewer inspect the score, evidence fields, flags, and integrity checks without trusting the prose.

A real target audit, not a synthetic example

For this v1.1.2 demonstration, I used a real public repository:

artic-network/fieldbioinformatics

The target is not the protagonist of this post.

It is only the specimen used to show the audit workflow against a real bioinformatics codebase.

The local audit produced:

audits/fieldbioinformatics_v1_1_2/report.md
audits/fieldbioinformatics_v1_1_2/experiment_results.json

The target snapshot:

{
  "name": "artic-network/fieldbioinformatics",
  "remote": "https://github.com/artic-network/fieldbioinformatics",
  "branch": "master",
  "commit": "8008b4c97c2193a82308ff6f0be507b1d9306e36",
  "file_count": 114
}

This is the important part: the audit did not ask, "Does this README sound trustworthy?"

It asked:

Do README claims match actual package metadata and entry points?
Are there real CI and domain-specific tests?
Are dependencies reproducible enough?
Are there credential leaks?
Are there deprecated patient-adjacent paths?
Do clinical-adjacent output paths fail closed?
Does the repository include governance evidence, or only governance absence?

That is where STEM-AI is useful.

The score object

The machine-readable result records the score like this:

{
  "stage_1_readme_intent": 65,
  "stage_2_cross_platform": null,
  "stage_2_repo_local_consistency": 75,
  "stage_2_lane": "STAGE_2R_REPO_LOCAL_CONSISTENCY",
  "stage_2_policy": "External Stage 2 was not collected; LOCAL_ANALYSIS used Stage 2R in the fixed 0.20 Stage 2 slot.",
  "stage_3_code_bio": 55,
  "weights": {
    "stage_1": 0.4,
    "stage_2": 0.2,
    "stage_3": 0.4
  },
  "risk_penalty": 0,
  "final_score": 63,
  "formal_tier": "T2 Caution"
}

External Stage 2 is explicitly represented as null for this local-only audit.

That does not mean cross-platform consistency is unimportant.

It means this evidence slice was deliberately scoped to LOCAL_ANALYSIS. Instead of pretending to have social/web evidence, v1.1.2 uses Stage 2R: Repo-Local Consistency.

Stage 2R asks whether the repository's own surfaces agree with each other:

README vs package metadata and CLI entry points
README vs docs, tutorials, and troubleshooting
README test claims vs CI workflow and test definitions
clinical-adjacent outputs vs local intended-use boundaries

The contract defines the fixed-weight calculation:

Final = (Stage 1 x 0.40) + (Stage 2R x 0.20) + (Stage 3 x 0.40) - Risk Penalty
      = (65 x 0.40) + (75 x 0.20) + (55 x 0.40) - 0
      = 63

The final tier is therefore:

T2 Caution

Not because the prose sounded balanced.

Because the contract math forces that result.

Why the T0 hard floor did not trigger

T0_HARD_FLOOR is the rule that prevents a clinically dangerous repository from escaping rejection through good wording.

In simplified form:

If a repository is CA-DIRECT
and it has no substantive code implementation,
then final tier = T0 regardless of score math.

Examples of CA-DIRECT include patient-specific diagnosis, treatment recommendation, triage, risk scoring, or clinical decision support.

The audited repository did not trigger that floor because STEM-AI classified it as:

{
  "clinical_adjacent": true,
  "ca_severity": "CA-INDIRECT",
  "t0_hard_floor": false
}

It produces biological sequence artifacts that may sit near public-health or clinical workflows, but the inspected surface did not make direct autonomous diagnosis or treatment claims. It also has substantive implementation, CI, and domain-specific test definitions.

So the result is not T0.

But it is also not high-trust.

The bounded result is T2 Caution.

Code-integrity findings

The same JSON records C1-C4 LOCAL_ANALYSIS checks:

{
  "C1_hardcoded_credentials": {
    "status": "PASS"
  },
  "C2_dependency_pinning": {
    "status": "WARN"
  },
  "C3_dead_or_deprecated_patient_adjacent_paths": {
    "status": "WARN"
  },
  "C4_exception_handling_clinical_adjacent_paths": {
    "status": "WARN"
  }
}

That is the difference between a general review and a code-path audit.

A text review can say:

The project appears technically mature.

A code-path audit can say:

Credential patterns were checked. Dependency pinning is weak. Deprecated patient-adjacent metadata exists. One clinical-adjacent filtering path does not fail closed on missing depth.

That is a more useful governance object.

It is not a certificate.

It is a map of what a reviewer should trust, distrust, or inspect next.

A small Python verifier

Here is a small dependency-free Python script that reads the actual audit JSON and verifies the score calculation. It does not need target private code or patient data; it only checks the machine-readable audit result.

#!/usr/bin/env python3
import json
from pathlib import Path


RESULT = Path("audits/fieldbioinformatics_v1_1_2/experiment_results.json")


def tier(score: int) -> str:
    if score <= 39:
        return "T0"
    if score <= 54:
        return "T1"
    if score <= 69:
        return "T2"
    if score <= 84:
        return "T3"
    return "T4"


def bar(score: int, width: int = 20) -> str:
    filled = round((score / 100) * width)
    return "█" * filled + "░" * (width - filled)


data = json.loads(RESULT.read_text(encoding="utf-8"))
score = data["score"]

stage_1 = score["stage_1_readme_intent"]
stage_2 = score.get("stage_2_repo_local_consistency")
stage_3 = score["stage_3_code_bio"]
risk_penalty = score["risk_penalty"]

weights = score["weights"]
computed = round(
    (stage_1 * weights["stage_1"])
    + (stage_2 * weights["stage_2"])
    + (stage_3 * weights["stage_3"])
    - risk_penalty
)

assert computed == score["final_score"]
assert tier(computed) in score["formal_tier"]

print(f"Stage 1  {stage_1:3}/100  {bar(stage_1)}")
print(f"Stage 2R {stage_2:3}/100  {bar(stage_2)}")
print(f"Stage 3  {stage_3:3}/100  {bar(stage_3)}")
print(f"Final    {computed:3}/100  {bar(computed)}")
print(f"Tier     {score['formal_tier']}")

for key, item in data["code_integrity"].items():
    print(f"{key}: {item['status']}")

Expected digest:

Stage 1   65/100  █████████████░░░░░░░
Stage 2R  75/100  ███████████████░░░░░
Stage 3   55/100  ███████████░░░░░░░░░
Final     63/100  █████████████░░░░░░░
Tier      T2 Caution
C1_hardcoded_credentials: PASS
C2_dependency_pinning: WARN
C3_dead_or_deprecated_patient_adjacent_paths: WARN
C4_exception_handling_clinical_adjacent_paths: WARN

Why this matters

Bio/medical AI governance is full of language that sounds safe but is hard to verify:

"research use only"
"not medical advice"
"validated pipeline"
"clinical-grade"
"responsible AI"
"human-in-the-loop"

Those phrases are not enough.

STEM-AI asks for observable structure:

source-code reality
test reality
CI reality
dependency reality
clinical boundary reality
governance artifact reality
code-integrity reality

v1.1.2 adds another layer:

auditor reality.

The AI auditor itself has to load a memory contract before it scores.

That is what MICA is for.

The final answer is T2 Caution: research reference and supervised non-clinical technical review only. No autonomous clinical decision support.

Not hype.

Not rejection by default.

A bounded trust judgment with evidence paths.

What comes next

The follow-on lane should:

provision the target dependency environment
run selected target tests in a controlled shell
capture command, exit code, environment hash, and output digest
attach a replay manifest to experiment_results.json
keep runtime evidence separate from source/document/CI evidence

For the current demonstration, runtime execution status is recorded as an evidence boundary in the audit JSON. The score itself remains based on the official v1.1.2 LOCAL_ANALYSIS evidence basis: Stage 1 source/README evidence, Stage 2R repo-local consistency, Stage 3 code/bio evidence, and C1-C4 integrity checks.

Final thought

STEM-AI is not a clinical certifier.

It is also not trying to replace scientific review, regulatory review, or domain experts.

Its role is narrower: make the governance conversation start from observable evidence instead of presentation quality.

In practice, that means asking:

What did the repository claim?
What does the code actually implement?
Do the local surfaces agree with each other?
Are the tests domain-specific or merely infrastructural?
Are clinical-adjacent boundaries explicit?
Can the auditor's own scoring logic be inspected?

That is where I think STEM-AI belongs in AI governance.

Not as the final authority.

As the evidence gate before authority is invoked.

It turns a vague question, "Do we trust this bio/medical AI repository?", into a more reviewable one:

Does this repository establish enough observable trust to be considered, contained, or rejected?

Each /slop Is a Calibration Signal — AI-SLOP Detector v3.6.0 and the Claude Code Skill

Kwansub Yun — Tue, 28 Apr 2026 12:04:44 +0000

AI-assisted development has a quiet failure mode: the assistant that creates the pattern often becomes the assistant that reviews it.

When you and Claude work inside the same session, you drift together. The review criteria shift with the assistant's habits. After enough sessions, the same assistant that wrote the hollow function body is also the one approving the pull request. There is no external reference point — unless you build one.

That is the problem AI-SLOP Detector v3.6.0 addresses with the Claude Code skill.

Every time you run /slop inside a session, the scan result is recorded to a project-scoped history. When enough re-scan evidence accumulates, bounded self-calibration adjusts the detection weights for your codebase — automatically, without a manual command. The scanner does not drift with the session. It stays anchored to observed scan outcomes.

It does not get smarter every time. It builds calibration signal every time. That is a more accurate claim, and the distinction matters.

What the Skill Does

Install:

cp -r claude-skills/slop-detector ~/.claude/skills/slop-detector
# restart Claude Code

Four slash commands become available:

Command	What it does
`/slop`	Full project scan — interprets findings, prioritizes fixes, proposes patch plan
`/slop-file [path]`	Per-file deep-dive — explains each metric, gives concrete fix per pattern
`/slop-gate`	Hard gate decision — PASS or FAIL, lists blocking files with deficit_score >= 70
`/slop-spar`	Adversarial validation — probes metric boundaries, catches calibration drift

The intended workflow inside a Claude session:

1. /slop               → baseline scan, identify top offenders
2. review findings     → Claude prioritizes by deficit_score
3. patch files         → fix patterns with Claude's help
4. /slop-file <path>   → verify improvement per file
5. /slop               → confirm project aggregate improved
6. /slop-gate          → gate decision before merge

Quality policy lives in the skill layer. You do not re-explain what CRITICAL_DEFICIT means or which patterns are critical on every session.

The LEDA Flywheel

This is the part that matters.

LEDA is not model retraining. It is bounded weight calibration based on repeated scan outcomes.

/slop runs slop-detector --project . --json — without --no-history. Every invocation auto-records results to ~/.slop-detector/history.db, tagged with a project_id (sha256 of cwd) so signals never mix across different repositories.

After every 10 re-scanned files, the tool runs the LEDA self-calibration loop automatically:

/slop called
    │
    ├─► scan result → recorded to history.db (project-scoped)
    │
    ├─► 10 re-scanned files milestone?
    │       └─► SelfCalibrator: 4D grid-search over run history
    │               (ldr × inflation × ddc × purity weights)
    │               └─► confidence gap > 0.10?
    │                       └─► .slopconfig.yaml updated silently
    │
    └─► next /slop → calibrated weights, sharper detection

The calibrator uses re-scanned files as signal — not raw record count. A file counts toward the milestone only when the tool has seen it improve or degrade across at least two runs. This prevents first-time project scans from triggering calibration on noise.

Three constraints keep calibration bounded:

Domain-anchored — grid search is constrained to ±0.15 around domain baseline weights. Detection cannot drift outside the meaningful range for your project type.
Confidence gate — only applies when the top candidate weight set beats the second by > 0.10. Ambiguous signals produce no change.
Drift warnings — CalibrationResult.warnings flags any dimension that shifted > 0.25 from the anchor.

/slop-spar adds a separate adversarial layer: it probes known-pattern anchors, metric boundary cases, and existence conditions. When it detects that measured behavior has diverged from metric claims, it recommends --self-calibrate --apply-calibration explicitly.

What the Data Shows — and What We Won't Claim

We will not tell you that AI-SLOP Detector improves code quality by X%.

We have not run a controlled study. We have not compared matched projects with and without the tool. Any number we put here would be a claim we cannot prove, and this tool is built specifically to catch that kind of thing.

What we do have: the tool scanning itself. Every time a core module was changed, it got re-scanned. N = 14,367 records across all projects in ~/.slop-detector/history.db.

This is not outcome evidence. It is workflow telemetry. Here is what the scan history shows for the eight most-improved files in this codebase:

File                   Scans  Worst → Best   Improvement
─────────────────────────────────────────────────────────
ddc.py                   86   87.8 →  11.0    -76.8 pts
placeholder.py           92   70.3 →   0.0    -70.3 pts
cross_file.py            89   70.3 →   5.0    -65.3 pts
ci_gate.py               88   69.3 →   6.2    -63.1 pts
cli.py                   88   68.4 →   8.4    -60.0 pts
ldr.py                   90   58.0 →   0.1    -57.9 pts
python_advanced.py       95   74.0 →  18.0    -55.9 pts
context_jargon.py        86   55.7 →   5.0    -50.7 pts
─────────────────────────────────────────────────────────
Source: self-scan, history.db — not an independent study

And the weekly project aggregate (avg deficit score):

Week      Avg Deficit   Critical Files   Note
────────────────────────────────────────────────────────
2026-W09     11.9            3           baseline
2026-W10     22.1           20           structural refactor spike
2026-W14     20.0           58           large feature addition
2026-W15     11.9           14           post-refactor recovery
2026-W17     12.2           13           current — stable CLEAN state
────────────────────────────────────────────────────────

The mechanism is not mysterious. Scan reveals structural problems → Claude sees exact pattern names and line references → Claude (or the developer) fixes them → rescan confirms improvement → LEDA registers the delta and adjusts detection weights accordingly.

The loop does not guarantee quality. It makes quality visible, then measurable, then improvable.

Whether that loop improves your codebase is something your history.db will tell you — not us.

Also in v3.6.0

CI gate exit code fix. --ci-mode hard without --ci-report was returning exit 0 even on CRITICAL_DEFICIT files — a two-line fix in _evaluate_ci_gate() (commit 0d67997). This affected v3.1.1 through v3.5.0 on the specific path of using the gate without the reporting flag. A regression test at the subprocess level was added to prevent recurrence (commit 0208af4).

Pre-commit hooks rewritten. Three hook variants now use python -m slop_detector.cli as entry point (bypasses Windows .exe wrapper exit-code issue), and --severity high (nonexistent flag) replaced with --ci-mode:

repos:
  - repo: https://github.com/flamehaven01/AI-SLOP-Detector
    rev: v3.6.0
    hooks:
      - id: slop-detector           # hard gate
      # - id: slop-detector-warn    # report only
      # - id: slop-detector-patterns  # fast per-file

VS Code Extension v3.6.0. Version tracks core library. No behavior changes from v3.5.0.

The Shape of the Loop

The skill + LEDA loop is the external reference point. Detection weights stay grounded in observed scan outcomes — files that improved across re-scans, files that stayed problematic — rather than in what the assistant believes is correct at any given moment.

The loop does not guarantee quality. It makes quality visible, then measurable, then improvable.

We won't tell you what percentage your code will improve. That would make us the thing we are trying to detect.

The scanner is not Claude's opinion about code quality. It is a measurement that gets calibrated against reality, session by session. Your history.db will tell you the rest.

Links:

When an AI Pipeline Passes — But One Path Still Must Be Held: EXP-034

Kwansub Yun — Mon, 27 Apr 2026 10:09:19 +0000

No efficacy, causal, or clinical claims are made in this report.
RExSyn is an experimental Bio-AI governance pipeline.

You do not need to know the earlier experiments to read this report.

Most AI pipeline reports ask one question:

Did the system pass?

EXP-034 asked a stricter one:

Which path was allowed to count?

That distinction matters.

In a multi-stage AI pipeline, a final PASS can hide a lot of unresolved risk. A branch may be unstable. A regeneration path may drift. A new external API may enter the chain without being governed. A new modality may appear to improve the system while quietly changing the basis of judgment.

So EXP-034 was not designed to produce a clean success story.

It was designed to separate three things:

Path	Status	Meaning
Anchored expansion path	`GO`	Accepted path for EXP-034 reporting
Current regeneration path	`HOLD`	Diagnostic evidence, not acceptance baseline
Next remediation cycle	`EXP-035`	RCA and repair target

That is the real result.

EXP-034 passed, but not because every path passed.

It passed because the accepted anchor remained stable, the expansion tracks did not break the judgment system, and the unresolved regeneration path was explicitly held instead of being silently mixed into acceptance.

What EXP-034 tested

EXP-033 had already established a parity baseline.

EXP-034 asked whether that baseline could survive controlled expansion while adding:

a modal update track,
a live AlphaFold EBI observer endpoint,
and AlphaGenome / AG measurement.

The operating rule was simple:

Reproduce the parity baseline first.
Only then allow expansion.
Only then compare governance behavior across experiment cycles.

If the parity anchor breaks, the rest is not expansion.

It is regression.

The scope was also locked: methodology, governance, and reproducibility only. The experiment did not claim biological efficacy, causal inference, or clinical recommendation.

That boundary is important because this kind of system can easily sound more powerful than what was actually measured. EXP-034 was not asking whether the pipeline discovered a better biological answer.

It was asking whether the judgment system stayed governable after new signals entered the chain.

The key split: PASS did not mean everything passed

Track-A produced the defining decision of the experiment.

The accepted legacy replay anchor preserved the required PASS/BLOCK separation:

Metric	Legacy replay anchor
sample accuracy	`1.0`
sample balanced accuracy	`1.0`
arm accuracy	`1.0`
arm balanced accuracy	`1.0`
dangerous false-pass rate	`0.0`
false reject rate	`0.0`

That was the path allowed to anchor EXP-034.

But the current regeneration path did not recover:

Metric	Current regeneration
sample accuracy	`0.5`
sample balanced accuracy	`0.5`
status	`HOLD`

This is the most important part of the experiment:

EXP-034 did not pretend the regeneration path passed.

It kept that result inside the experiment as diagnostic evidence, but did not allow it to redefine the accepted baseline.

That separation is not a minor operational detail. It is the governance result.

A weak pipeline would have blended the two paths and still reported a final success. EXP-034 did the opposite. It allowed the stable anchor to proceed and held the unstable path for RCA.

That is how a stage-gated system avoids changing its own question after seeing the result.

Why path splitting matters

The concrete governance problem is this:

A pipeline can pass for the wrong reason.

valid_report = stable_anchor × traceable_extension × contained_instability

If the anchor is not stable, the report cannot be trusted.

If the extension is not traceable, the new signal becomes an ungoverned side channel.

If instability is not contained, a diagnostic failure can quietly contaminate acceptance.

A single final PASS is not enough when several branches contribute to a verdict. You need to know which branch produced the accepted decision, which branch failed, which branch was only diagnostic, and which branch is allowed to affect future work.

EXP-034 passed because all three conditions were enforced:

the legacy replay anchor held,
the new observer and AG paths were measured under governance,
and the regeneration HOLD remained outside acceptance.

That is the difference between a pipeline that merely outputs a verdict and a pipeline that controls which verdicts are allowed to count.

Adding AlphaFold EBI as an observer, not a predictor

Relative to EXP-033, EXP-034 added a live AlphaFold Protein Structure Database / EBI observer line.

This was not promoted into a primary predictor.

It was wired as an observer/reference oracle and traced into governance as ebi_g2.

The result:

Check	Result
AlphaFold EBI direct endpoint for `P23219`	`GO`
Stage 7 observer tests	`2 passed`
`ebi_g2` governance traceability	`PASS`
`BLOCKED_IDP` mapping path	validated in test

The point is not simply that an external endpoint responded.

The point is that the external signal entered the system through a governed path. It was not allowed to float beside the pipeline as informal context.

EXP-034 tested whether the new observer could be admitted without becoming an ungoverned side channel.

AG-live: non-degradation, not repair

Track-C tested a simple question:

If AG-live enters the pipeline, does it change the final decision?

The answer was no.

AG-live did enter the pipeline.

The AlphaGenome field was present with:

AG field	Value
source	`alphagenome_api_live`
pathogenicity_score	`0.5`
confidence	`0.7143`
clinical_significance	`uncertain`

These are sanitized branch artifact values, not implementation code or full raw artifacts.

AG-live did not change classification.

Both controls remained governed by the same conservative decision boundary:

Path	Expected	Observed	Interpretation
`EXP032-BLOCK-001` negative control	`BLOCK_EXPECTED`	`BLOCK / ESCALATE`	fail-closed behavior preserved
`EXP032-PASS-001` pass-eligible control	`PASS_ELIGIBLE`	`BLOCK / ESCALATE`	conservative over-blocking persisted

That is the key nuance.

AG-live did not create a dangerous false-pass. The negative control stayed blocked.

But AG-live also did not repair the current regeneration hold. The pass-eligible control still failed to recover and remained blocked under R2_component_floor.

The governance surface moved slightly, but the verdict did not:

Metric	Earlier AG branch	AG-live branch
`p_e2e`	`0.0912`	`0.0947`
clinical status	`BLOCK`	`BLOCK`
rule	`R2_component_floor`	`R2_component_floor`

So the correct conclusion is not:

AG improved the pipeline.

The correct conclusion is:

AG-live changed the measurement surface slightly, but did not change the decision boundary.

That is exactly what non-degradation means here.

It preserved fail-closed behavior on the negative control while leaving the pass-eligible control over-blocked.

This is why Track-C can only be called non-degradation, not repair.

Contract passed, but governance still blocked

One of the most useful details in EXP-034 is that the contract layer and governance layer did not collapse into one verdict.

The contract inspection reported:

Field	Value
pipeline contract score	`0.9077`
weakest connection	`C2`
dangerous pass risk	`0.0`
gate recommendation	`PASS`
overall OK	`true`

But the clinical governance layer still blocked the case.

That is not a contradiction.

It means the pipeline connection was valid enough to inspect, but the decision was not safe enough to accept.

This distinction matters.

A weaker system might treat a passing contract as permission to pass the whole output. EXP-034 did not do that. It allowed the contract layer to say:

The pipeline is connected.

while the governance layer could still say:

The claim should not pass.

That separation is exactly what a governance layer is supposed to preserve.

Cross-cycle comparison: EXP-032 → EXP-033 → EXP-034

Track-D compared the accepted anchor path across cycles.

You do not need the earlier experiments as background. They matter here for one reason only:

EXP-034 was not allowed to invent a new success criterion.

EXP-032 and EXP-033 provided the previous PASS/BLOCK baseline. EXP-034 tested whether that baseline survived expansion.

The classification baseline stayed fixed:

Compare	Accuracy / balanced accuracy
EXP-032 → EXP-034	`1.0 / 1.0`
EXP-033 → EXP-034	`1.0 / 1.0`

At the same time, governance signals moved:

Governance signal	Delta
`ccge_p_e2e_mean`	`+0.04447488775996111`
`nnsl_sr9_tech_mean`	`+0.04692394788063081`
`nnsl_di2_tech_mean`	`-0.03667940951579321`

The interpretation is narrow:

The judgment baseline stayed fixed while the governance surface became more measurable.

That is what EXP-034 was allowed to claim.

It did not prove biological efficacy.

It did not prove that every branch of the system was now stable.

It proved that controlled expansion could happen without breaking the accepted PASS/BLOCK baseline.

Stage-gate result

EXP-034 ended with all five stage gates passing:

Gate	Status
G1 parity	`PASS`
G2 reproducibility	`PASS`
G3 cross-experiment compare	`PASS`
G4 governance traceability	`PASS`
G5 extension safety	`PASS`

Final state:

Field	Value
overall status	`PASS`
anchor mode	`legacy_replay`
first failed gate	`null`
diagnostic hold	`Track-A current regeneration`

This is the important nuance:

The experiment passed with a retained diagnostic hold.

That is not a contradiction. It is the point of the control system.

The accepted anchor path was allowed to proceed. The current regeneration path was not. The remediation target was moved to EXP-035.

That separation is the actual proof EXP-034 provides: not that every branch became stable, but that instability was not allowed to contaminate acceptance.

What EXP-034 actually showed

EXP-034 did not show that the entire pipeline is now stable.

It showed something narrower and more useful:

A method-locked Bio-AI governance pipeline can admit modal expansion, AlphaFold EBI observer wiring, and AG-live measurement without losing its accepted PASS/BLOCK baseline — while keeping the unstable regeneration path out of acceptance.

Track-C sharpened that conclusion.

AG-live entered.

Metrics moved slightly.

The verdict did not change.

Dangerous false-pass did not appear.

Conservative over-blocking remained.

That is not a clean success story.

It is a governed result.

Closing

Stage-gated experimentation is not just about getting a result.

It is about deciding whether the result should be allowed to exist.

In EXP-034, the answer was:

GO   for the anchored expansion path
HOLD for current regeneration
NEXT for EXP-035 remediation

That may sound less dramatic than a clean success story.

But in governance work, that is exactly the point.

A mature AI pipeline is not the one that claims everything passed.

It is the one that can say:

This path passed.

This path did not.

And we did not mix them.

FLAMEHAVEN FileSearch: Why This RAG Engine Feels Different from the Usual Stack

Kwansub Yun — Mon, 20 Apr 2026 14:27:37 +0000

FLAMEHAVEN FileSearch: Why This RAG Engine Feels Different from the Usual Stack

RAG is no longer an exotic idea.

At this point, most developers have seen the familiar stack:

parser
chunker
embeddings
vector store
LLM
framework wrapper
demo query

That is not the interesting part anymore.

The interesting part is what happens after the diagram:
how much infrastructure the stack quietly demands, how much of the retrieval path is actually auditable, how much of the system is still mechanical rather than opaque, and how much operational tax the user is forced to absorb just to get a search engine running.

That is where FLAMEHAVEN FileSearch gets more interesting than the usual "another RAG repo" framing.

This is not a feature announcement. It is a technical look at what the project is actually doing differently.

The real problem with many RAG stacks

A lot of RAG systems are not products. They are assembly instructions.

They give you flexibility, but they also leave you responsible for stitching together:

file parsing
chunking strategy
embeddings
lexical retrieval
semantic retrieval
answer generation
attribution
storage
auth
monitoring
caching
deployment

That is fine if you want a blank canvas.

It is less fine if what you actually want is a document search engine that can be deployed without turning the setup itself into a second project.

That is the first reason this repo feels different: it is trying to compress more of that surface area into one codebase.

What is technically different here

1) Hybrid retrieval is treated as the baseline, not the upgrade path

A lot of RAG repos still behave as if semantic retrieval is the main event and lexical matching is an optional add-on.

That is backwards for real document systems.

FLAMEHAVEN FileSearch builds around three explicit modes:

keyword
semantic
hybrid

The interesting part is the hybrid path itself.

The retrieval stack combines:

BM25
Reciprocal Rank Fusion (RRF)
a Korean + English tokenizer
a lazy per-store BM25 rebuild path

That last point matters more than it sounds. The BM25 index is not eagerly rebuilt on every upload. It is marked dirty (_bm25_dirty) and rebuilt on first hybrid search after mutation. That is a very practical decision. It keeps ingestion cheaper without pretending indexing is free.

This is one of the deeper differences from many vector-first RAG demos: the system does not assume semantic retrieval should dominate exact-match behavior. It assumes production search needs both.

2) The indexing model is not just "document in, chunks out"

The second meaningful difference is the indexing granularity.

This repo introduces a KnowledgeAtom layer: a two-level indexing model with

file-level documents
chunk-level atoms

Those chunk atoms are not anonymous fragments. They carry stable fragment URIs of the form:

local://store/encoded_path#c0001

That design solves two very common problems at once:

precision retrieval
stable attribution

The file-level object remains available, but the system can also retrieve chunk-level units directly. That reduces the usual gap between "the document matched" and "the relevant passage was actually isolated."

The URI choice matters too. A lot of local-first search code still uses basename-style references that collide the moment two files share a name. This repo moves to a reversible, quoted absolute-path-based URI namespace (urllib.parse.quote(abs_path, safe='')), which is much less fragile.

That is not marketing polish. That is retrieval hygiene.

3) The chunking path is internal, structured, and mechanical

Another place where this codebase differs is that it does not outsource the core text pipeline by default.

Instead of treating chunking as a thin wrapper around an external library, it implements an internal text chunker with:

heading-boundary splitting
paragraph splitting
sentence fallback for oversized blocks
undersized chunk merging (default minimum: 64 tokens)
token-aware chunk sizing

The chunking system is actually two-pass under the hood. The structure-aware TextChunker handles the document splits above. On top of that, KnowledgeAtom applies a second windowing pass when generating chunk embeddings — 800-character windows, 120-character overlap, and an 80-character minimum before a fragment is dropped. These two paths are separate by design: TextChunker is responsible for semantic structure, KnowledgeAtom for granular embedding units.

The engine also ships a ContextExtractor — a sliding-window utility that can enrich each chunk with text from its neighboring chunks before retrieval. It is fully tested, but it is not yet wired into the default ingestion path. It is available for downstream pipeline extension.

So the pipeline architecture is:

text document → structure-aware split (TextChunker) → chunk atom embedding (KnowledgeAtom, 800-char windows) → multi-level indexing → retrieval

That is a better-shaped pipeline for document search than a naive chunk list.

4) The vector path is trying to remove operational weight, not add it

This is probably the most unusual architectural choice in the repo.

Instead of anchoring everything around a heavyweight embedding model stack, the project uses Gravitas Vectorizer v2.0, a deterministic vectorization path built on:

hybrid feature extraction (word tokens + character n-grams)
signed feature hashing for collision mitigation
SHA-256 based deterministic output
no torch, no transformers, no model download

The trade-off is obvious: this is not trying to win a leaderboard as a giant foundation-model embedding backend.

That is not the point.

The point is that it makes the semantic path much cheaper to deploy, easier to reason about, and viable in environments where "just load another model" is operationally the wrong answer.

Technically, that shows up in several ways:

deterministic vector generation
cold start under 1ms
no ML framework dependency in the core vector path
optional NumPy acceleration with pure-Python fallback

In other words, the semantic layer is being treated as infrastructure, not as a permanent excuse to expand infrastructure.

That is rare.

5) The repo is explicit about local-first and multi-provider execution

A lot of document search systems quietly assume one provider path.

This repo does not.

The provider layer supports:

Gemini
OpenAI
Anthropic
Ollama
OpenAI-compatible endpoints

That matters for two reasons.

First, it keeps the system from being hardwired to one hosted model assumption.

Second, it means the retrieval stack and the answer stack are not collapsed into the same dependency decision.

That is an important architectural separation.

For non-Gemini providers, the code takes a provider-RAG route: local semantic retrieval first, then prompt construction, then model answer generation. That is a much more honest design than pretending all providers support the same retrieval semantics natively.

The local Ollama path is especially relevant. Not because "local" is fashionable, but because self-hosted document search is often most attractive precisely when data boundary control matters more than marginal model quality gains.

6) The codebase has been refactored toward narrower responsibilities

One of the easiest ways to tell whether a repo is becoming more operationally serious is to look at whether the core orchestrator is shrinking or swelling.

Here, the architecture moved in the right direction.

The central core.py was split into focused mixins:

IngestMixin
LocalSearchMixin
CloudSearchMixin

That is not just aesthetic cleanup.

It clarifies the system boundary between:

ingestion
local retrieval/orchestration
provider-backed answer generation

The same pattern appears elsewhere:

BackendRegistry maps file extensions to parser classes via register() — new formats plug in without modifying existing dispatch logic
duplicate helper blocks were pulled out of cloud search paths
file parsing was reduced to dispatch instead of a single giant extractor module

These changes do not make a flashy screenshot.

They do make the code easier to maintain without quietly reintroducing the same complexity elsewhere.

That is a real engineering improvement.

Benchmark snapshot

System profile

Gravitas Vectorizer v2.0 (deterministic DSP, zero ML deps)

ChronosGrid vector backend with quantized storage (int8)

BM25 + RRF hybrid retrieval

Local / pgvector backends

Redis cache optional

Documented performance figures (Docker, Apple M1, 500 PDFs ~2GB)

Vector generation: <1ms

Search, cache hit: 9ms

Search, cache miss (includes Gemini API round-trip): 1,250ms

Batch search (10 queries, parallel): 2,500ms

Upload, 50MB file with indexing: 3,200ms

What matters more than the numbers

The cache-hit figure reflects the full path when semantic and lexical retrieval are served from warm indexes.

The cache-miss figure is dominated by the Gemini API round-trip, not local retrieval.

The performance story here is not just raw speed. It is that the repo achieves low-latency local retrieval by reducing dependency weight and simplifying the vector path, rather than by hiding heavy infrastructure behind abstraction.

A comparison that is actually worth making

The wrong comparison is:

"Is this the best RAG framework?"

That is too vague to be useful.

The better comparison is architectural.

Approach	Main idea	Common weakness	Why this repo differs
Framework-only RAG stack	Compose your own parser, retriever, vector store, and generator	High assembly burden; a lot of operational logic is still your job	This repo packages more of the retrieval, ingestion, attribution, and serving path together
Hosted RAG / SaaS search	Fastest time to first demo	External data boundary, vendor coupling, recurring service assumptions	This repo keeps self-hosted and local-first execution as first-class options
Vector-first DIY pipeline	Semantic retrieval drives everything	Lexical exactness and attribution often become second-class	This repo treats hybrid retrieval as the practical default
FLAMEHAVEN FileSearch	Retrieval + ingestion + serving compressed into one engine	Less of a blank canvas than a raw framework stack	Better fit for teams that want a mechanical, deployable search base instead of another assembly project

That is the actual niche.

Not "RAG but louder."

More like:

RAG with a lower operational tax.

Why this matters now

The RAG field has cooled compared to its peak hype cycle.

That is not a bad thing.

It means the novelty premium is lower, and the real questions are clearer:

Can it be deployed?
Can it run without a side quest in infrastructure?
Can it keep data local?
Can it support both lexical precision and semantic recall?
Can its retrieval behavior be inspected rather than mythologized?

That is why a repo like this becomes more interesting now than it would have been in the most hype-saturated phase of the RAG wave.

When everything is new, wrappers are enough.

When the field matures, the differentiator becomes whether the system removes real engineering burden.

This one is at least trying to solve that problem directly.

What is special about the code, specifically

If I had to reduce the repo's technical distinctiveness to a short list, it would be this:

BM25 + RRF is built in, not bolted on later
KnowledgeAtom indexing gives the system a more precise retrieval unit than document-only search
Stable chunk URIs (local://store/enc_path#c0001) make attribution less fragile
Two-pass chunking — structure-aware TextChunker + char-window KnowledgeAtom embedding pass — keeps the text pipeline mechanical and inspectable
Gravitas Vectorizer v2.0 reduces startup cost and dependency sprawl (zero torch/transformers)
Provider abstraction separates retrieval architecture from model vendor choice
Mixin segmentation and BackendRegistry pattern show a codebase moving away from monolithic orchestration

That is why this repo feels different from the usual RAG stack.

Not because it claims magic.

Because it makes several practical decisions that many RAG repos defer, externalize, or ignore.

The honest boundary

This is not a claim that the repo solves everything.

It does not.

And the codebase itself shows that.

Static inspection still flags complexity hotspots in:

api.py
admin_routes.py
eval_self.py
chronos_grid.py

There are also components that exist in the engine but are not yet connected to the default pipeline — ContextExtractor being the clearest example. The architecture is there; the wiring is not yet complete everywhere.

That is actually a good thing for a write-up like this, because it keeps the claim honest.

The interesting story here is not "perfect codebase."

It is:

a repo with a real architectural point of view, a recognizably lower dependency burden, and code decisions that are meaningfully different from the usual vector-wrapper pattern.

That is a much stronger claim than vague "enterprise-grade RAG" language.

Final take

FLAMEHAVEN FileSearch is interesting because it is not merely trying to make retrieval work.

It is trying to make retrieval:

more mechanical
more local
more attributable
less dependency-heavy
and less painful to deploy

That is a better differentiator than "supports RAG."

Most repositories do.

The more important question now is whether they reduce the actual engineering burden around RAG, or just rearrange it.

This repo is interesting because it appears to reduce some of it in code.

And in a field where many projects now converge into the same parser + vector store + model + wrapper pattern, that is a difference worth paying attention to.

Repository

GitHub: https://github.com/flamehaven01/Flamehaven-Filesearch

AI-SLOP Detector v3.5.0 — Every Claim, Verified Against Source Code

Kwansub Yun — Wed, 15 Apr 2026 06:19:37 +0000

I published a LinkedIn post about AI-SLOP Detector's self-calibration system and download numbers. Someone asked the reasonable question: "Can you actually back that up?"

Yes. Here's the source.

This isn't a feature announcement. It's a line-by-line audit of seven claims against the actual codebase. Every VERDICT links to a real file and real line numbers. The repo is public — go check it yourself.

What was claimed

Claim	Verdict
Every scan is recorded	✅ TRUE
Repeat scans become calibration signal	✅ TRUE
Updates only when signal is strong enough	✅ TRUE
Visible policy artifact (`.slopconfig.yaml`)	✅ TRUE
Explicit numeric limits govern calibration	✅ TRUE
Detects empty/stub/phantom/disconnected code	✅ TRUE
~1.4K downloads last week	✅ TRUE

All seven. No fabrications. No inflated numbers. Here's the proof.

Claim 1: "Every scan is recorded"

Source: src/slop_detector/history.py, lines 116–180

def record(self, file_analysis, git_commit=None, git_branch=None, project_id=None) -> None:

Auto-invoked on every CLI run. The only opt-out is --no-history. Each scan writes to SQLite at ~/.slop-detector/history.db and stores:

deficit_score, ldr_score, inflation_score, ddc_usage_ratio
n_critical_patterns, fired_rules
git_commit, git_branch, project_id

Schema is now at v5, auto-migrated on startup through every release from v2.9.0 to v3.5.0.

VERDICT: TRUE. The record() call is real. The schema is versioned. The behavior is not optional.

Claim 2: "Every re-scan becomes signal"

Source: src/slop_detector/history.py, lines 221–246

def count_files_with_multiple_runs(self, project_id=None) -> int:
    # Only files scanned >= 2 times count as calibration events
    SELECT file_path FROM history GROUP BY file_path HAVING COUNT(*) >= 2

Source: src/slop_detector/ml/self_calibrator.py, lines 301–309

def _extract_events(self, project_id=None):
    rows = self._load_history(project_id=project_id)
    by_file = self._group_runs_by_file(rows)

Single-scan files produce no calibration events. Only repeat scans generate improvement or fp_candidate labels. The threshold is hardcoded in SQL, not assumed.

VERDICT: TRUE. The repeat-scan requirement is enforced at the query level, not in documentation.

Claim 3: "Updates only when the signal is strong enough"

Source: src/slop_detector/ml/self_calibrator.py, lines 37–54 (constants) and 251–262 (enforcement)

CONFIDENCE_GAP: float = 0.10   # min gap between #1 and #2 candidate
MIN_IMPROVEMENTS: int = 5       # improvement events required
MIN_FP_CANDIDATES: int = 5      # fp_candidate events required

Gate 1 — confidence gap check (line 251):

if result.confidence_gap < CONFIDENCE_GAP:
    result.status = "insufficient_data"
    result.message = (
        f"Confidence gap {result.confidence_gap:.4f} < {CONFIDENCE_GAP}. "
        f"Candidates are too close — need more history data for reliable calibration."
    )
    return result  # NO UPDATE APPLIED

Gate 2 — score delta check (line 262):

if current_score - winner_score < 0.02:
    result.status = "no_change"  # also does not apply

Two independent guards. Both must pass before any weight update applies.

VERDICT: TRUE. Ambiguous signal is rejected twice before touching configuration.

Claim 4: "Leaves behind a visible policy every time it changes"

Source: src/slop_detector/ml/self_calibrator.py, docstring line 17–18

Return CalibrationResult; optionally write to .slopconfig.yaml via --apply-calibration

When --apply-calibration is passed and status == "ok", optimal weights are written to .slopconfig.yaml. Plain-text YAML. Human-readable. Git-versionable. Every calibration change is a diff.

VERDICT: TRUE. The policy artifact is explicit. You can git blame it.

Claim 5: "Explicit limits govern calibration"

Source: src/slop_detector/ml/self_calibrator.py, lines 37–54

MIN_W: float = 0.10             # minimum allowed weight per dimension
MAX_W: float = 0.65             # maximum allowed weight per dimension
MAX_PURITY_WEIGHT: float = 0.25 # purity ceiling
DOMAIN_TOLERANCE: float = 0.15  # max per-dimension deviation from domain anchor
DOMAIN_DRIFT_LIMIT: float = 0.25 # warn when optimal weight drifts this far
GRID_STEP: int = 20             # 0.05 increment resolution

No ML model. No learned bounds. Every constraint is a named constant with a comment explaining why it exists. The calibration space is a bounded grid, not an open optimization landscape.

VERDICT: TRUE. Every limit is auditable. Nothing is opaque.

Claim 6: "Detects empty implementations, phantom dependencies, disconnected pipelines"

These are the three canonical defect patterns AI code generation produces at scale. Each has a dedicated module.

Defect class	Implementation
Empty/stub functions	`src/slop_detector/metrics/ldr.py` — LDRCalculator detects `pass`, `...`, `raise NotImplementedError`, `TODO`
Phantom/unused imports	`src/slop_detector/metrics/hallucination_deps.py` — AST-based import vs usage analysis via `HallucinatedDependency` dataclass
Disconnected pipelines	`src/slop_detector/metrics/ddc.py` — DDC (Declared Dependency Completeness) usage ratio
Function clone clusters	`src/slop_detector/patterns/python_advanced.py` — Jensen-Shannon Divergence on 30-dim AST histograms, JSD < 0.05 = clone

The clone detection is worth noting. JSD on AST histograms catches structural duplication that string similarity misses entirely. LLMs produce a lot of this — same function logic, slightly renamed.

VERDICT: TRUE. Each defect class has a named module with a working implementation.

Claim 7: "~1.4K downloads in the past week"

Source: pypistats.org API (mirrors=false), queried 2026-04-15

last_week:  1,407  (mirrors excluded — actual pip install traffic)
last_month: 1,787
last_day:   83

"~1.4K" is within 0.5% of 1,407. Mirrors excluded means bot traffic is stripped — these are real install invocations.

VERDICT: TRUE. Verified against pypistats in real time. The number is not rounded up.

Why this format exists

Most open-source project posts make claims. Few back them up with file paths and line numbers.

That gap is the same problem AI-SLOP Detector is built to close. AI-generated code makes claims too — functions that look complete, imports that look used, pipelines that look connected. Static analysis finds the gap between what the code says and what it does.

This post applies the same standard to the project's own marketing copy. If a claim can be verified, it should be. If it can't, it shouldn't be made.

The codebase is public: github.com/flamehaven01/AI-SLOP-Detector

Pull requests welcome. Audits welcome more.

Verified by static code analysis + pypistats API, 2026-04-15

It Gets Smarter Every Scan: AI-SLOP Detector v3.5.0 and the Self-Calibration Loop

Kwansub Yun — Mon, 13 Apr 2026 16:15:49 +0000

Previously: 🔻v3.1.0 — Three Formula Refinements and the Adversarial Tester That Found Them ·
🔻v2.9.0/v2.9.1 — The Tool That Turned On Itself

By late 2025, everyone was building with AI. A weekend was enough to launch a SaaS app, and by Monday it was already on Product Hunt. The code looked finished, the UI worked, and the demo landed. That was also the problem.

In 2026, some of the consequences started arriving in public. Exposed databases, weak security boundaries, brittle automation, and production systems that looked polished enough to ship but had clearly not been understood at the level their surface confidence implied. Not every one of those failures belongs to static analysis, and it would be too easy to pretend otherwise. But many of them still point to the same upstream condition: code that looks complete long before it deserves trust.

That is the layer this release is about.

The breach is the headline. The review gap is the story.

A missing security rule is not the same thing as a stubbed auth function. A runtime-only bug is not the same thing as a phantom import.

A broken architecture is not the same thing as a buzzword-heavy helper. These are different failure classes, and any serious tool has to respect that difference.

What they often share, though, is the review environment that let them through. AI increased output volume, increased speed, and increased surface polish.

Review depth did not increase with it. That matters because AI-generated code has a very recognizable habit: it often looks complete before it is complete.

It compiles. It passes tests. It sounds like it knows what it is doing. Then you open the function.

def calculate_quality_score(data: Dict[str, Any]) -> float:
    """
    Advanced multi-dimensional quality assessment using
    proprietary algorithms with statistical normalization,
    entropy-based weighting, and dynamic threshold calibration.
    Returns a score between 0 and 100.
    """
    # TODO: implement the actual algorithm
    return 85.0

This is not noisy code. It is confident emptiness. In an analytics path, it becomes false certainty. In a payment path, it becomes a defect. In an auth path, it becomes risk.

The issue is not that AI writes ugly code. The issue is that AI reliably produces code that is structurally plausible while functionally thin.

That is a narrower claim than “AI is dangerous,” but it is also far more useful.

We ran into this ourselves

This did not begin as a theory about other people’s repos. It began when we found a flaw in our own scoring model. Back in v2.8.0, we discovered that our formula was accidentally rewarding spaghetti code.

A large god function could sometimes look healthier than a small clean function because complexity was dividing the penalty instead of amplifying it.

That was backwards, so the math changed.

AI-SLOP Detector now evaluates four dimensions:

LDR for logic density
Inflation for jargon density relative to real logic
DDC for dependency usage rather than dependency presence
Purity for critical structural defects that should drag the whole score down

These are combined with a weighted geometric mean, not an arithmetic average.

Why that matters:

one strong-looking axis should not be able to hide a collapsed one
a polished docstring should not rescue empty logic
if one important dimension fails, the whole score should feel it

That is the scoring philosophy underneath the tool. But even that was not enough.

Static analyzers have a threshold problem

Take a perfectly legitimate ML helper:

def prepare_training_batch(
    raw_samples: List[Dict[str, Any]],
    tokenizer: PreTrainedTokenizer,
    max_length: int = 512,
) -> Dict[str, torch.Tensor]:
    """
    Tokenize and pad samples for transformer training.
    Handles attention mask generation and HuggingFace
    tokenizer conventions for batch encoding.
    """
    return tokenizer(
        [s["text"] for s in raw_samples],
        padding=True,
        truncation=True,
        max_length=max_length,
        return_tensors="pt",
    )

There is nothing wrong with this code. But a generic detector may still overreact, because terms like tokenizer, attention mask, and HuggingFace can look suspicious if the analyzer does not understand the domain it is scanning. In a real ML codebase, those terms are normal. In a CRUD backend, some of them may be genuine anomaly signals.

That is the threshold problem. The same threshold can be wrong in one codebase and exactly right in another. A universal threshold sounds elegant, but real repositories are local. They have habits, idioms, and boilerplate that are legitimate inside one domain and suspicious inside another.

So the next problem became obvious: the tool had to learn the project it was scanning. That is the real center of v3.5.0.

What AI-SLOP Detector actually does

AI-SLOP Detector is a static analyzer built to catch a specific defect class that shows up repeatedly in AI-generated code: unimplemented stubs, disconnected pipelines, phantom imports, clone-shaped emptiness, placeholder-heavy production paths, and jargon inflation that outruns the actual logic. It is not a style linter, not a full security scanner, and not a runtime verifier. It is a detector for structural hollowness.

That distinction matters because it keeps the claim honest. The tool is not trying to solve every production risk. It is trying to catch one layer that becomes more expensive as AI output scales faster than human review.

pip install ai-slop-detector
slop-detector --init
slop-detector --project.

The workflow is the product story

What makes this release interesting is that it is not just “more patterns” or “more language support.” It is a workflow story.

The detector now has a real loop. It scans the file, classifies its role, computes the 4D score, applies structural pattern penalties, and writes the result to history. Then, once enough repeated scans exist, it revisits that history, extracts behavioral signals, tunes the weights inside bounded domain-aware limits, updates the configuration, and keeps scanning. That is the release.

That final stretch is what changed this from “detector upgrade” into “adaptive detector.” The tool no longer only evaluates code. It also learns from what happens after evaluation.

Self-calibration is the real headline

Every scan is recorded to a local SQLite history database. That history is not just there for reporting. It becomes the signal surface for the next tuning step. Once enough repeated scans accumulate, the detector begins asking a simple question: when this file was flagged, what happened next?

That produces two behavior-derived event types. An improvement event means the file was flagged, later changed, and its deficit dropped meaningfully. A false-positive candidate means the file was flagged, then scanned again with the same content and little meaningful score movement.

That difference is more important than it sounds. A lot of “self-improving” systems quietly learn from their own outputs. They mark something suspicious, then later use that same judgment as the truth signal for tuning. The system becomes better at agreeing with itself. That is not calibration. That is self-imitation with cleaner packaging.

v3.5.0 tries to avoid that trap. Its labels are not taken from the scoring formula. They are inferred from developer behavior around repeated scans. The formula says, “this looks suspicious.” The next run reveals whether a human treated that suspicion as real.

That signal is not perfect. An unchanged file is not always a false positive. It may be legacy code, low priority, or simply out of scope. But it is still a healthier signal than teaching the formula to imitate its own prior outputs.

What the loop actually looks like

The loop is not mystical. It is mechanical. Repeated scans accumulate, improvement and likely-FP events are extracted, candidate weight sets are evaluated, the search is bounded around the project’s current domain anchor, and if a strong enough winner appears, the config gets updated. If a calibrated weight drifts too far from the domain anchor, the system emits a warning.

This is what makes the title true. It gets smarter every scan, not because a hidden model is hallucinating taste, but because repeated use creates a bounded feedback loop. That is much less magical, and much more trustworthy.

Why `--init` matters more now

There is another reason the calibration story works better in v3.5.0. The detector no longer starts from a generic nowhere. --init now performs domain-aware bootstrap, detects the likely project type, and seeds the starting weights accordingly. That means calibration starts near the right neighborhood instead of wandering across the whole map.

That improves the first week of use, not just the tenth. And that matters, because bad first impressions kill adaptive tools. If the detector is only smart after a month of annoying you, it will never survive long enough to get smart.

Good initialization is not a convenience feature. It is part of whether the loop can gather clean signal at all.

JS, TS, and Go are not side quests

v3.5.0 also expands analysis coverage to Go, JS, JSX, TS, and TSX. That is useful on its own, but the deeper significance is architectural. Structurally hollow AI-generated code is not a Python-only phenomenon. If the detector’s long-term direction is project-local calibration rather than one-size-fits-all scoring, then wider language support is not a side feature. It is the natural expansion of the same idea.

Different languages. Same review gap. Same loop.

The honest boundary

This tool still does not close every gap. It will not fix missing infrastructure controls, catch every runtime bug, prove the architecture is correct, or replace security review. A clean structural profile is not proof of safety.

What it can do is narrow one expensive blind spot: the distance between code that looks finished and code that carries enough actual logic to deserve confidence. That is a smaller claim than “AI risk solved,” but it is also the kind of claim that survives production better.

Why this matters now

AI has made software generation dramatically cheaper. It has not made understanding cheaper. That difference is where governance debt begins to accumulate.

If teams can now generate far more code than they can truly review, then the review stack needs tools that operate below style and above syntax. Not tools that ask whether the code is pretty, but tools that ask whether the implementation carries enough substance for the confidence wrapped around it.

That is the space AI-SLOP Detector is trying to occupy. Not the whole problem. Just one layer that became impossible to ignore.

Quick start

pip install ai-slop-detector
cd my-project/
slop-detector --init
slop-detector --project .

Fix what clearly deserves fixing. Leave legitimate idioms alone. Then keep scanning. If the loop is doing its job, the next pass should know your codebase a little better than the first one did.

GitHub: flamehaven01/AI-SLOP-Detector

Can AI Review Physics? Yes — That Is Why We Built SPAR

Kwansub Yun — Sun, 12 Apr 2026 09:34:43 +0000

A standalone review framework for checking whether outputs deserve the claims attached to them.

Most review systems answer a familiar question:

Did the system still produce the expected output?

SPAR is built for a narrower and more dangerous one:

Does the output still deserve the claim attached to it?

That is the split. Not reliability alone, but admissibility.

In practical terms, admissibility means claim-worthiness: whether a result justifies the interpretation, governance status, or scientific statement built on top of it.

A system can be reliable and inadmissible at the same time.

A physics engine can compute beta_G_norm, return zero, pass regression, and stay green across the whole pipeline. The report can still say the background is admissible. But if the function producing beta_G_norm is a stub that always returns zero, the output is stable while the claim attached to it is false.

That is not hypothetical. It is one concrete class of review failure SPAR was designed to surface.

What SPAR Is

SPAR (Sovereign Physics Autonomous Review) is a deterministic framework for claim-aware review.

It does not replace unit tests. It does not replace regression benchmarks. It does not replace scoring systems. It reviews a different object:

the output
the claim attached to that output
the implementation state behind it
the maturity state that should travel with it

SPAR started inside Flamehaven-TOE, an open physics simulation and AI governance engine. It has since been extracted into a standalone open-source framework:

github.com/flamehaven01/SPAR-Framework

The framework includes a generic review kernel, explicit score and verdict policy, registry-backed review surfaces, and a first domain adapter for physics. Physics is the first adapter and the first domain where this review model was stress-tested. It is not the limit of the framework.

The core idea is simpler than the name: an output can pass while the claim attached to it drifts.

Why Ordinary Review Is Not Enough

Ordinary review is usually shallow by necessity. It asks questions like:

did the code run
did the output shape stay valid
did the score remain within bounds
did regression remain green

Those are necessary checks. They are not always enough.

The failure SPAR cares about is not, in the first instance, a crash. It is not even always a wrong number. It is this:

the code executes
the output looks plausible
the tests pass
and the interpretation is still overstated or structurally false

That failure can appear in several ways: a placeholder implementation returns stable-looking values; a maturity registry stays stale after the implementation improves; a score looks smooth before its epistemic basis is strong enough to justify the interpretation attached to it; an approximation gets reported as closure.

None of these failures is spectacular. That is exactly why they are easy to miss.

A Minimal Divergence

The clearest way to see the difference is in review form.

{
  "ordinary_regression": {
    "kernel_exec": "PASS",
    "output_contract": "PASS",
    "test_suite_status": "GREEN"
  },
  "spar_review": {
    "layer_a": "CONSISTENT",
    "layer_b": "CONSISTENT",
    "layer_c": "GAP_STATE_MISMATCH",
    "finding": "Implementation path is genuine; registry classification remains stale",
    "required_action": "RECLASSIFY"
  }
}

Ordinary regression says the system still works. SPAR says the system may no longer be describing its own computation truthfully.

SPAR is not "tests, but harsher." It can produce a different review outcome even when ordinary regression remains green. In this case, the required action is not rejection. It is reclassification. That is not the same as testing harder. It is reviewing a different object.

The Core Mismatch Classes

SPAR treats three mismatch classes as first-class review objects.

Anchor mismatch — the output conflicts with a declared analytical or contractual anchor.

Interpretation mismatch — the report language claims more than the implementation state justifies.

Maturity mismatch — the implementation, registry, and outward-facing claim have drifted out of sync.

Ordinary review mostly checks whether a system still passes. SPAR checks whether the result is still being described honestly.

The Three-Layer Structure

Layer A — Anchor Consistency

Layer A checks whether output agrees with a declared analytical or contractual anchor. The expected value is not "whatever the engine produced last time." It is "what the declared contract says must appear for this background, under this formulation."

# Flat Minkowski: beta residuals must vanish
status = CONSISTENT if abs(beta_G(output)) <= 1e-4 else ANOMALY

# de Sitter: admissibility gate must FAIL
# A PASS here indicates a gate defect, not a success.

Layer A tests agreement with a declared reference contract — not truth in some unconstrained universal sense. Analytical anchors depend on regime, normalization, and formulation. That distinction matters. Still, the engineering value is clear: reliability can remain intact while anchor-consistency fails. A Layer A anomaly means the output contradicts the contract the system claims to be using.

Layer B — Interpretation Validity

Layer B checks whether the interpretation attached to the output stays within declared scope. This layer is deterministic — it does not rely on a free-form LLM judge. It uses explicit rule tables over structured runtime artifacts, maturity states, and report text.

Typical checks: does the report claim full closure while the path is still heuristic or partial; is a bounded approximation being described as exact; is an environment-conditional bridge being written up as universal; are overclaim phrases appearing where runtime state does not support them.

Layer B does not eliminate semantic ambiguity. What it does is narrow the problem from "solve rhetoric in general" to "enforce explicit admissibility contracts against declared model states." That makes it auditable. Not complete. Auditable.

Layer C — Existence and Maturity Probes

Layer C asks what kind of implementation produced the result: genuine, approximate, gapped, environment-conditional, or research-only.

This is where SPAR becomes especially different from ordinary review. It does not merely score outputs. It checks the ontological status of the path that produced them. A result from a known-limited path is not the same thing as a result from a genuine path. A research probe is not production-grade closure. A dependency-bound bridge is not a universal capability. Those distinctions change what the output is allowed to claim.

Why the Registry Matters

A framework like this needs more than score outputs. It needs structured state that can travel with the result.

ARCHITECTURE_GAPS = {
    "C4_sidrce_omega": (
        "SIDRCE Omega (v4.5+): chi-squared Gaussian model. "
        "score = exp(-chi2/2), chi2 = (||beta||/tol)^2. "
        "Derived from zero-centered Gaussian likelihood. "
        "GAP: tolerance values are calibration parameters with "
        "no first-principles derivation of their exact magnitude."
    ),
    "C8_rg_linearized": (
        "RG flow: linearized dilaton ODE only. Metric does NOT flow. "
        "APPROXIMATION: valid only for small perturbations "
        "around a fixed background geometry."
    ),
}

A machine-readable registry turns caveat prose into runtime surface. That lets review results carry explicit maturity labels — open, partial, closed, environment_conditional, research_only — rather than vague prose. Without that surface, approximation and closure collapse into the same sentence.

Scoring Policy Is Explicit

SPAR keeps score policy visible. No hidden learned weights.

SCORE_TABLE = {
    "ANOMALY":       -15,   # contradicts an analytical anchor
    "FAIL":          -10,   # review-layer failure
    "GAP":            -5,   # honest gap disclosure
    "WARN":           -3,   # bounded concern
    "APPROXIMATION":  -2,   # known simplification
}

def journal_verdict(score, layer_a):
    if count_anomalies(layer_a) >= 2:  return "REJECT"
    if score >= 85:                    return "ACCEPT"
    if score >= 70:                    return "MINOR REVISION"
    if score >= 50:                    return "MAJOR REVISION"
    return "REJECT"

These are not laws of nature. They are review policy. A hidden learned scorer may feel sophisticated. An explicit policy is easier to inspect, debate, and change.

Two or more Layer A anomalies trigger unconditional REJECT regardless of total score. Mathematical contract failures are not averaged away by cleaner signals elsewhere.

A Concrete Example: The Omega Score Transition

Flamehaven-TOE's primary governance metric, SIDRCE Omega, once relied on a large arbitrary multiplicative constant applied to the raw residual. The outputs looked stable. Nothing felt obviously broken.

SPAR still flagged it. Stability was not the right question. The stronger question was whether the formula justified the interpretation being attached to it. A free scaling constant with no physical derivation is not the same thing as a physically motivated model.

The formula was replaced with a chi-squared Gaussian construction:

gs_score = math.exp(-0.5 * (beta_result.beta_G_norm / tol_G) ** 2)
b_score  = math.exp(-0.5 * (beta_result.beta_B_norm  / tol_B) ** 2)
si_score = math.exp(-0.5 * (beta_result.beta_Phi_norm / tol_Phi) ** 2)

omega_physics = (gs_score * b_score * si_score) ** (1/3)

That change matters because it introduces a reversible relation to the underlying residual:

||beta|| = tol * sqrt(-2 * ln(score))

Given a reported score, the residual is recoverable. The score is no longer just a presentation layer. It encodes a falsifiable relation to the quantity beneath it.

SPAR did not respond by declaring the problem solved. It updated the classification precisely: the formula ceased to be arbitrary, but the remaining gap shifted to the tolerance scales, which are still calibration parameters. That is a narrower and more honest claim than either "arbitrary" or "fully resolved."

That is exactly the kind of distinction SPAR is built to review.

Quick Start

pip install -e .[dev]

from spar_framework.engine import run_review
from spar_domain_physics.runtime import get_review_runtime

runtime = get_review_runtime()

result = run_review(
    runtime=runtime,
    subject={
        "beta_G_norm": 0.0,
        "beta_B_norm": 0.0,
        "beta_Phi_norm": 0.0,
        "sidrce_omega": 1.0,
        "eft_m_kk_gev": 1.0e16,
        "ricci_norm": 0.02,
    },
    source="flat minkowski",
    gate="PASS",
    report_text="Bounded report text.",
)

print(result.verdict)
print(result.score)
print(result.model_registry_snapshot["total_models"])

The review result carries more than pass/fail: a verdict, a score, and a maturity-aware review surface. That surface is what makes the output governable rather than just evaluable.

Where This Framework Fits

Physics remains the first adapter and strongest early testbed. The review pattern is broader.

It fits anywhere outputs can pass while the attached claim can drift: scientific computing pipelines, PDE and simulation workflows, scientific ML surrogates, inverse and calibration models, AI code review, model governance, regulated analytics and reporting.

That does not mean every team needs the full framework. Often the first useful step is smaller.

A Lightweight Adoption Path

Level 1 — Claim Check. Add three explicit questions to an existing workflow: What is the output actually claiming? Does that claim match the implementation state? Is this result exact, approximate, partial, or heuristic? Most teams can do this immediately with no new tooling.

Level 2 — Maturity Labels. Attach state labels to results: heuristic, partial, closed, environment_conditional. A small registry. Already a meaningful step beyond ordinary review.

Level 3 — Full SPAR. Layer A anchor consistency, Layer B interpretation validity, Layer C existence and maturity probes, registry-backed snapshots, explicit score and verdict policy.

SPAR can be used as a review habit before it is adopted as a full framework.

What SPAR Does Not Do

SPAR does not provide a universal truth engine, free-form LLM judging in the core, domain contracts inside the generic kernel, or certainty about whether a scientific claim is true in all possible senses.

SPAR is not a machine for declaring truth. Its narrower goal is to make claim drift reviewable.

Reliability Is Not Enough

Reliability asks whether a system produces stable, repeatable outputs. Admissibility asks whether those outputs deserve the meanings attached to them.

A stub that always returns zero can be reliable. A heuristic threshold can be reliable. A smoothly calibrated score can be reliable. None of those facts alone makes the resulting claim justified.

Current AI and scientific tooling are already better at measuring reliability than admissibility. That asymmetry is understandable — reliability is easier to benchmark, easier to automate, easier to ship in CI. But admissibility is where silent approximations, overstated claims, and maturity mismatches accumulate.

SPAR is one working answer to that problem. Not a universal answer. A technical one.

It turns implementation state, maturity state, analytical anchoring, and scope honesty into review objects that can travel with the result.

That is why the architecture may matter outside the domain that produced it.

Repository: github.com/flamehaven01/SPAR-Framework

Docs: What Is SPAR · Admissibility · Physics as the Proof Case · Use Cases

AI SLOP Detector v3.1: Three Formula Refinements and the Adversarial Tester That Found Them

Kwansub Yun — Thu, 09 Apr 2026 05:29:57 +0000

We shipped v2.9.0 with a scoring engine we trusted. We ran tests. Everything passed.

Then we built a tool specifically designed to find cases where the score was less precise than it could be — and it found three.

This is the story of v3.1.0. And the patch that followed six hours later.

Glossary — internal terminology used throughout this post

Term	What it means
Deficit score	The final output of the scorer. 0 = structurally clean, 100 = critical. Derived as `100 × (1 - GQG)`.
GQG	Geometric Quality Gate. A weighted geometric mean of LDR, Inflation quality, DDC, and Purity. The single formula the scorer evaluates.
LDR	Logic Density Ratio. Ratio of executable logic lines to total lines. Low LDR = file is mostly stubs, blanks, or comments.
Inflation	Metric that flags jargon-heavy docstrings unsupported by actual code complexity. A 2-line function with a 30-line docstring using 12 buzzwords scores badly.
DDC	Dead/Duplicate Code ratio. Tracks unreachable paths, copy-pasted blocks, phantom imports.
Purity	Pattern hit rate. How many structural anti-patterns (god functions, stub returns, nested complexity) fire on the file.
Cyclomatic Complexity (CC)	Count of independent code paths. A straight-line function = CC 1. Each `if`, `for`, `while`, `except` adds 1.
fhval	flamehaven-validator. An external tool that interrogates the scorer from outside the codebase. Its purpose is to catch cases where internal test consistency masquerades as correctness.
SPAR	Subcommand of fhval. Adversarial regression loop with three layers. Tests whether the scorer measures what it claims to measure.
JSD	Jensen-Shannon Divergence. A symmetric, bounded (0–1) measure of divergence between two probability distributions. Used here to compare AST node-type histograms between functions.
AST	Abstract Syntax Tree. The parsed structure of source code. An `if` statement, a `return`, a function call each become typed nodes.
function_clone_cluster	New pattern in v3.1.0. Detects files where many functions share near-identical AST structure — the fragmented god function evasion pattern.
placeholder_variable_naming	New pattern in v3.1.0. Detects vocabulary-clean code with zero semantic content: single-letter parameter floods, sequential numbered variables.
AM/GM gap	Core refinement in v3.1.0. The calibrator used an arithmetic mean (simpler approximation); the scorer uses a geometric mean (the precise target formula). Aligning them closes a ~5-7pt estimation gap on uneven files.

Quick context

AI SLOP Detector is a static analyzer that measures structural code quality — not style, not formatting. It scores each file across four dimensions and assigns a deficit between 0 (clean) and 100 (critical):

Dimension	What it measures
LDR	Ratio of executable logic to total lines
Inflation	Jargon, docstring bloat, unsupported claims
DDC	Unreachable paths, copy-pasted blocks
Purity	Pattern hit rate (stubs, god functions, etc.)

These four numbers feed a single formula — a weighted geometric mean — called the GQG. The output is the deficit score: 100 × (1 - GQG).

The calibrator's job is to find the best weights for that formula by searching over thousands of known cases.

Before v3.1.0: the self-scan

We don't ship a version without running the detector against itself. Before cutting v3.1.0, we ran v3.0.3 — a structural debt reduction pass on the three highest-deficit files in the codebase.

Self-scan before: avg_deficit=23.57, 15 deficit files, status=suspicious
Self-scan after:  avg_deficit=20.33, 12 deficit files, status=clean

analysis/cross_file.py dropped from 70.3 to 28.7 (critical → clean). ci_gate.py from 69.3 to 22.3. cli.py from 68.4 to 20.9. The fixes were mechanical: extracted nested closures to private methods, replaced if/elif/else dispatch chains with dict dispatch, removed re-declared constants.

The point is not that these numbers are good. It's that the tool had to earn its own PASS before we shipped the version that refines the formula. Shipping a scoring engine while your own codebase sits at suspicious would have been its own kind of slop.

The adversarial tester: fhval SPAR

In a previous post we described fhval — flamehaven-validator. The core concern: when every tool in an ecosystem is built by the same person against the same baseline, internal consistency can masquerade as correctness. Passing your own tests proves nothing about whether your tests are asking the right questions.

For v3.1.0 we added a spar subcommand — an adversarial regression loop that interrogates the scorer from the outside. Running SPAR against the v3.0.x scorer:

SPAR score: 55 / 100  [FAIL]

Layer A anomalies:
  A3 stub_class_8_methods     expected >= 30  got 20.0  [ANOMALY]
  A4 fragmented_god_function  expected >= 10  got  0.0  [ANOMALY]
  A5 vocab_clean_meaningless  expected >=  8  got  0.0  [ANOMALY]

Layer C blind spots:
  C2 inflation_blindspot      [BLIND_SPOT]
  C3 ddc_annotation_gap       [BLIND_SPOT]

Three gaps. Two documented scope limits. Score: 55 FAIL.

Each gap pointed at a specific detection weakness. The SPAR methodology itself — how Layer A/B/C work, why adversarial ground truth is hard to author from inside the codebase — is a separate topic covered in tomorrow's post. Here we focus on what the gaps told us and what we changed.

Refinement 1: The calibrator and scorer were using different formulas

The scorer computes a weighted geometric mean. The calibrator — which finds optimal weights — was computing a weighted arithmetic mean as its optimization target.

Those are not the same thing, and for a quality gate, the difference is structural.

Consider a file with three dimension scores: LDR=0.9 (good), inflation_quality=0.1 (very bad), DDC=0.8 (good).

Formula	Calculation	Result	Deficit
Arithmetic mean	(0.9 + 0.1 + 0.8) / 3	0.60	40
Geometric mean	(0.9 × 0.1 × 0.8) ^ (1/3)	0.42	58

The arithmetic mean gives deficit=40. The geometric mean gives deficit=58. The gap is 18 points — not rounding, but structural. The geometric mean amplifies weak dimensions because one bad score pulls the entire product down. The arithmetic mean averages over them.

The scorer uses the geometric mean for good reason: a file with excellent LDR but zero actual logic (all docstrings) should not score deficit=30. It should score much higher. The formula enforces that.

The first-generation calibrator used an arithmetic mean as a simpler starting approximation. So it was finding weights that minimize error against a different objective than the scorer actually computes. The result: roughly 5–7 point underestimation on files with uneven dimension profiles — which are precisely the target of this tool.

The AM ≥ GM inequality means the calibrator's scores were always optimistic. For balanced files (all dimensions similar) the gap is small and harmless. For uneven files, it was systematic — and those are the cases that matter most.

Refinement:

# Before (calibrator _recompute_deficit)
quality = (w_ldr * ldr + w_inflation * (1 - inflation_n) + w_ddc * ddc) / total_w

# After — mirrors the scorer's GQG formula exactly
from math import exp, log
gqg = exp(
    (
        w_ldr * log(max(1e-4, ldr))
        + w_inflation * log(max(1e-4, 1.0 - inflation_n))
        + w_ddc * log(max(1e-4, ddc))
    )
    / total_w
)
deficit = min(100.0, 100.0 * (1.0 - gqg))

This is why SPAR anomaly A3 (stub_class_8_methods) jumped from deficit 20.0 to 40.0: the stub class had heavily uneven dimensions, and the geometric mean scored it correctly once the calibrator was trained against the right target.

Refinement 2: The complexity modifier had a dead zone at the common end

The inflation metric applies a complexity modifier to penalize functions that are simultaneously simple and jargon-heavy — a common pattern in AI-generated code: a two-line function surrounded by an elaborate docstring.

The first-generation modifier formula:

# Before
complexity_modifier = max(1.0, 1.0 + (avg_complexity - 3.0) / 10.0)

For CC=1: 1.0 + (1-3)/10 = 0.8 → max(1.0, 0.8) = 1.0
For CC=2: 1.0 + (2-3)/10 = 0.9 → max(1.0, 0.9) = 1.0
For CC=3: 1.0 + (3-3)/10 = 1.0 → max(1.0, 1.0) = 1.0

CC=1, 2, and 3 all received the same modifier: 1.0. This meant simple functions — the three most common complexity levels — paid no complexity premium on inflation, regardless of how jargon-heavy they were. The modifier only activated from CC=4 upward.

Simple jargon-heavy functions are the most common AI code signature. The formula was least sensitive precisely where it needed to be most sensitive.

# After — CC=1 is the baseline, not CC=3
complexity_modifier = max(1.0, 1.0 + (avg_complexity - 1.0) / 10.0)

Now CC=2 gets a 1.10× modifier, CC=3 gets 1.20×. The penalty scales from the simplest meaningful function upward.

Refinement 3: Purity weight was documented but not connected

The GQG formula includes a purity dimension:

# Before
w_pur = 0.10  # hardcoded constant
final_score = gqg_score * (1.0 - w_pur * purity_penalty)

.slopconfig.yaml had a weights.purity field. The calibrator's weight search had a purity parameter. Neither was connected to this constant — users could configure weights.purity: 0.20 and nothing would change.

# After
w_pur = weights.get("purity", 0.10)  # default unchanged; now configurable

One line. The config surface now matches the implementation.

Two new detection patterns

Stub evasion: empty container returns

The existing return_constant_stub pattern caught return True, return 0, return "string" — but not return {}, return [], return (), return set(). These are equally common stub patterns in class skeletons:

class DataProcessor:
    def get_results(self) -> dict:
        return {}  # was not flagged before

    def list_items(self) -> list:
        return []  # was not flagged before

Both are now caught by return_constant_stub and interface_only_class.

Fragmented god function: AST clone detection

SPAR anomaly A4 was a file with 12 one-liner helper functions:

def _compute_r1(x): return x * 1.1
def _compute_r2(x): return x * 1.2
def _compute_r3(x): return x * 1.3
# ... through r12

Each function individually looks clean: low complexity, no nesting, short. No single function exceeds any per-function threshold. But collectively, this is a decomposed god function — a large computation split into structurally identical fragments that evade per-function gates.

The new pattern: function_clone_cluster.

How it works. For each file, build a 30-dimensional histogram of AST node types for every function: how many If nodes, Return nodes, Call nodes, BinOp nodes, and so on. The histogram is normalized to a probability distribution. Then compute pairwise Jensen-Shannon Divergence between all function pairs. JSD is bounded between 0 and 1. Two functions with near-identical AST structure produce JSD close to 0.

Functions with JSD < 0.05 get an edge in a graph. BFS finds connected components. The largest component is the clone cluster.

Thresholds:
  >= 6 functions in cluster: CRITICAL
  >= 4 functions in cluster: HIGH

Why JSD and not simpler metrics. Cosine similarity or Euclidean distance on raw histograms don't handle sparse distributions well — short functions have mostly empty histograms, and small absolute differences dominate. JSD compares distributions rather than raw vectors, stable when most histogram dimensions are near zero. It also has an upper bound of 1, which makes the 0.05 threshold interpretable rather than dataset-dependent.

The JSD threshold (0.05) was calibrated against the internal test corpus. It will produce false positives on files with many similar utility functions — for example, a large set of _validate_field_X() validators that are structurally identical by design. Adjust via --config if needed.

Placeholder variable naming (v1.0)

SPAR anomaly A5 was vocabulary-clean code with zero semantic content:

def aggregate(a, b, c, d, e, f, g):
    r1 = a + b
    r2 = r1 * c
    r3 = r2 - d
    # ... through r12
    return r12

No buzzwords. No docstring bloat. Every traditional linter passes this. The new placeholder_variable_naming pattern applies two checks:

Single-letter parameter density: 5 or more single-letter parameters (excluding self, cls, _) → HIGH.
Sequential numbered variables: a run of 8 or more → HIGH; 4 or more → MEDIUM.

This is v1.0: it detects naming style, not semantic quality. Known false positive zone: scientific and math libraries legitimately use single-letter conventions (x, y, z, mu, sigma). Suppress with domain_overrides in .slopconfig.yaml.

SPAR result after v3.1.0

SPAR score: 85 / 100  [PASS]

Layer A: 5/5 anchors consistent
Layer B: 4 documented limitations (no regressions)
Layer C: C2 inflation_blindspot [BLIND_SPOT — known scope limit]
         C3 ddc_annotation_gap  [BLIND_SPOT — known scope limit]

55 → 85 PASS.

The two remaining blind spots are not gaps to close — they're the documented scope limits of static analysis: a tool that reads AST cannot determine whether arithmetic is semantically meaningful, or whether annotation-heavy imports serve a real runtime purpose. Those require a different class of model. Documenting the ceiling is part of the job.

The full SPAR methodology — how Layer A/B/C work, why Layer A ground truth is hard to author from inside the codebase, and what "validating the validator" means in practice — is covered in tomorrow's post.

v3.1.1: the self-inspection patch

v3.1.0 and v3.1.1 shipped on the same day. The clone detection pattern introduced in v3.1.0 had a visibility gap: function_clone_cluster fired in the Issues section but produced no signal in the Core Metrics table. A community issue caught it within hours.

But before cutting v3.1.1, we ran the tool against itself — and the new patterns found something:

placeholder.py    deficit: 70.3  [CRITICAL_DEFICIT]
python_advanced.py  deficit: 74.0  [CRITICAL_DEFICIT]

Both files are part of the detection engine itself. Root cause: check_node methods with cyclomatic complexity 20–31, caused by compound boolean logic that had accumulated across releases. The tool was flagging its own pattern implementations as having the exact complexity problems it was designed to detect.

We extracted four module-level helpers in placeholder.py (_strip_docstring, _has_abstractmethod, _empty_container_repr, _is_placeholder_stmt) and added _make_god_issue() and _collect_numbered_vars() to python_advanced.py. Each check_node method went from 20–70 lines to 8–15. The detector earned its own PASS before shipping the patch.

placeholder.py      70.3 → 43.7  [CRITICAL → SUSPICIOUS]
python_advanced.py  74.0 → 66.7  [CRITICAL → INFLATED_SIGNAL]

Additional v3.1.1 refinements:

Clone Detection row added to Core Metrics table (CRITICAL/PASS at a glance).
Table style unified to box.ROUNDED across all project output (was mixing three styles).
VS Code extension: extractJson() strips [INFO] log lines before JSON.parse — previously caused silent parse failures when CLI log output appeared alongside JSON. Workspace analysis replaced with a QuickPick list of deficit files sorted by score; clicking opens the file in the editor.

If you installed 3.1.0, upgrade to 3.1.1 before using clone detection in CI.

How this fits alongside existing tools

Tool	Approach	What it sees that others don't
Semgrep	Pattern-matching on AST	Rule violations you've pre-authored
SonarQube	Cognitive complexity, duplication, coverage	Complexity, coverage gaps — not structural emptiness
Radon	Cyclomatic complexity	Raw CC values; used internally by AI SLOP Detector
Bandit	Security rules	Security vulnerabilities
mutmut / cosmic-ray	Mutation testing	Whether your test suite catches real bugs
AI SLOP Detector	Metric-based structural analysis	Docstring theater, stub pipelines, fragmented logic, phantom imports

The key gap: a file can be fully SonarQube-clean while containing zero actual logic — all stubs, all docstrings, all type annotations. Cognitive complexity doesn't measure whether the complexity is real. LDR does. Inflation does.

The complementary tool here is mutation testing. SPAR tests whether the scorer measures what it claims. Mutation testing tests whether your tests catch what they claim to catch. Both are adversarial approaches to the meta-problem: how do you validate the validator?

Score evolution

If you're running AI SLOP Detector on an existing project, upgrading to 3.1.x will change your scores. The formula alignment in Refinement 1 increases deficit on files with uneven dimension profiles, typically by 3–8 points. This is not drift — it's the scorer becoming more precise in the region where it matters most. Files that were borderline suspicious may move into inflated_signal. Check your CI threshold after upgrading.

Previous scores were valid estimates produced by the first-generation model. v3.1.x scores are tighter estimates with better sensitivity where dimensions are uneven — which is precisely the profile of AI-generated code.

Honest limitations

function_clone_cluster threshold (JSD < 0.05) was calibrated against the internal test corpus. It will fire false positives on legitimate utility function clusters. Adjust via --config.
placeholder_variable_naming v1.0 has no semantic context. def distance(x, y, z) is legitimate; the pattern doesn't know that.
SPAR score 85 means five ground truth anchors pass and eight of ten Layer C probes hold. The space of evasion patterns is open-ended. More in tomorrow's SPAR post.
The Layer A corpus is internally authored. External adversarial contributions would make it stronger.

Install / upgrade

pip install ai-slop-detector==3.1.1
# or
pip install --upgrade ai-slop-detector

VS Code extension: search "AI SLOP Detector" in Extensions, or install from VSIX:

code --install-extension vscode-slop-detector-3.1.1.vsix

# Scan a project
slop-detector --project ./your-project

# Machine-readable output
slop-detector --project ./your-project --json | jq '.file_results[] | {file: .file_path, deficit: .deficit_score}'

GitHub: flamehaven01/AI-SLOP-Detector

Previous posts in this series:

My AI Maintainer Kept Making Wrong Calls. So I Made It Report Its State Before Touching Anything.

Kwansub Yun — Tue, 07 Apr 2026 17:24:43 +0000

🔎 Glossary: terms used in this article

🔸 MICA (Memory Invocation & Context Archive): A governance schema for AI context management. Defines how context should be structured, trusted, scored, and handed off across sessions.

🔸 memory_injection: A MICA operational mode. The archive is updated after each maintenance session and read by the next AI session to compensate for session amnesia.

🔸 Session Report Format: The structured opening output the model must produce at session start — declaring archive version, self-test result, drift status, and active invariants — before any repository-level work begins.

🔸 Self-Test Policy: Machine-evaluable checks that validate the archive against the real project state: file existence, hash integrity, and invocation protocol presence.

🔸 Drift Response Policy: The schema-level declaration of how hash mismatches and missing files are handled. Different failure classes carry different response actions.

🔸 Design Invariant: A structured governance rule with identity, severity, and statement. Not a style guideline. A constraint the model cannot rationalize past.

🔸 Provenance Registry: The record of tracked files with SHA256 hashes. The basis for drift detection.

🔸 Deviation Log: The audit trail of formal exceptions to design invariants. Empty means no exceptions have been logged — not that no judgment calls were made.

1. What Part 5 Left Open

Part 5 placed MICA inside the context engineering landscape and drew one boundary: MICA is not a collection system. It begins after collection ends. Its job is to govern what enters the session, what remains authoritative, and how the model proves it actually loaded the governed archive at all.

That answer was correct. It was also still abstract.

This post comes down from that framing. It shows what MICA looks like when it is actually running — not as a concept, but as a protocol inside a real project.

The project is flamehaven.space, a Next.js B2B site maintained by a solo operator using a MICA package in memory_injection mode. Everything shown here is drawn from the live archive. Values that would expose internal configuration are anonymized; structure and behavior are real.

2. The Session Opening Report

Every MICA session in memory_injection mode begins with a declared output before any work starts. The archive specifies the required format:

[SESSION READY]
Archive: flamehaven-space-maintainer v0.2.0
Self-test: PASS (3 checks -- ST-001, ST-002, ST-003)
Drift: no hash mismatch detected
Active invariants: DI-001 (critical), DI-002 (critical) + 24 others loaded
Gate: PASS -- proceeding with maintenance

This is not a courtesy summary. It is a gate. The archive field session_report_format.gate_block_on is set to critical_self_test_failure — meaning the model must declare its load state before it is permitted to make any repository-level changes.

The format is specified in the archive itself:

json
"session_report_format": {
  "trigger": "session_start",
  "required_fields": ["archive_version", "self_test", "drift_status", "active_invariants", "gate"],
  "format_template": "[SESSION READY]\nArchive: {archive_version}\nSelf-test: {self_test}\nDrift: {drift_status}\nActive invariants: {active_invariants}\nGate: {gate}"
}

The model does not decide what to declare. The archive tells it what a valid session opening looks like.

3. What the Self-Test Actually Checks

The self_test_policy runs on session_start and pre_handoff. Three checks matter here:

ST-001 (provenance_sha256_format, severity: error) — verifies that provenance hashes in the registry match the expected format. A malformed hash means the file fingerprint is untrustworthy.
ST-002 (provenance_file_exists, severity: warning) — verifies that files listed in the provenance registry actually exist on disk. A missing file is not a formatting error; it is a ghost reference.
ST-003 (invocation_pattern_present, severity: error) — verifies that the invocation protocol is declared and readable. If the model cannot confirm how it was loaded, it cannot confirm the session is in a governed state.

Failure behavior is set per-check. The overall on_failure policy for this archive is warn_continue — the session proceeds, but the failure is surfaced explicitly in the opening report.

This is a deliberate calibration. A site maintenance session that blocks hard on every provenance warning would be too brittle for solo operation. The severity ladder reflects the actual cost of each failure mode, not a theoretical maximum.

4. What Drift Detection Looks Like in Practice

The archive's drift_response_policy is minimal by design:

"drift_response_policy": {
  "on_hash_mismatch": "warn_continue",
  "on_file_missing": "warn_block",
  "reminder_after_change": true,
  "inline_sync_required": false
}

The distinction between warn_continue and warn_block is operationally significant.

A hash mismatch means a tracked file changed — which happens legitimately during ordinary maintenance. The model surfaces the mismatch and continues.

A file that has gone missing entirely is a different failure class. The model blocks and requires operator acknowledgment before proceeding.

reminder_after_change: true means the archive instructs the model to remind the operator to refresh the provenance registry and artifact manifest before minting the next archive version. This is not automated enforcement. It is memory injection: the archive tells the next session what the previous session should have reminded the operator to do.

The deviation_log in v0.2.0 is empty. That is not a sign the system has never been used. It means no deviations have been formally logged yet — which is itself a state the archive captures.

5. What Happens When a Deployment Changes Something

A concrete scenario: the operator ships a writing refresh mode change — switching from automatic ISR to manual operator-triggered revalidation. Three files change: next.config.ts, a helper .bat script, and the playbook.

On the next session open:

Self-test runs. ST-002 may flag if the helper script path is not in the provenance registry yet.
Drift check runs. Hash mismatches fire for the changed files. on_hash_mismatch: warn_continue — the session proceeds.
The opening report surfaces the mismatch:

[SESSION READY]
Archive: flamehaven-space-maintainer v0.2.0
Self-test: PASS with warnings (ST-002: update-writing-now.bat not in provenance registry)
Drift: hash mismatch on next.config.ts, flamehaven-space-maintainer-playbook.v0.2.0.md
Active invariants: DI-001 (critical), DI-002 (critical) + 24 others
Gate: PASS WITH WARNINGS -- operator review required

The operator now has a concrete decision surface before touching anything: what changed, what the model knows about, and what it does not.

The model then follows the change process defined in the playbook: identify the canonical subsystem touched, patch the smallest coherent surface, run build and audit, verify route-level behavior, then update README, MICA, or playbook if the change alters maintainer truth.

At the end of the session, if a new archive version is minted, the synchronization rule is explicit: file name, project.version, and the archive handoff marker must be updated in the same change. Not sequentially. The same change.

This is the operational point of drift reporting: not merely to announce that something changed, but to force the model and the operator to see the same changed surface before any new work proceeds.

6. What the Design Invariants Actually Govern

The archive carries 26 design invariants. The first six establish the perimeter everything else operates within:

DI-001 (critical): Flamehaven is positioned as a governance-first, founder-led B2B AI systems practice, not a generic agency or AI wrapper shop.
DI-002 (critical): Primary conversion surface is the main domain flamehaven.space, not Medium, Substack, DEV.to, or LinkedIn.
DI-003 (high): Writing detail pages are authoritative artifacts linked to projects, selected work, contact, and SEO canonical ownership.
DI-004 (high): Selected Work must distinguish public, private, and internal systems without broken public links or placeholder copy.
DI-005 (high): Legacy WordPress-era routes must redirect away from obsolete agency messaging.
DI-006 (high): Operational choices favor deterministic behavior, inspectability, and maintenance continuity over decorative complexity.

These are not style guidelines. They are session-blocking constraints. An AI that proposes converting Selected Work to a live-fetch real-time surface is violating DI-006. An AI that treats cross-posting as the canonical publishing path is violating DI-002. An AI that leaves a legacy route live because a redirect “seems unnecessary” is violating DI-005.

The invariants exist so the model cannot rationalize its way past the operator's architectural intent — even across sessions, even with a new model instance that has never seen the project before.

7. What MICA Cannot Do Here

The deviation_log is empty because no formal deviation has been logged. But there have been judgment calls.

One example is the writing hero image fallback logic. The site had to support both modern Notion block structures and legacy imported posts with a different nesting format. That decision did not begin as an invariant. It began as a session-level judgment call, became a playbook rule, and only then became stable maintainer truth.

That path matters.

MICA does not automate the step from “we discussed this and made a call” to “this is now a governed constraint.” It provides the place to put the result. The operator decides what rises to the level of an invariant, what remains a lesson in the playbook, and what disappears when the session ends.

That boundary — what gets governed, what gets remembered, what gets lost — is not a gap in MICA. It is a design decision made with every archive update.

8. What Part 7 Will Address

Part 6 showed what MICA looks like in operation inside a single maintenance agent. The structure holds. The protocol runs. The session report is predictable.

But that is still the easier case.

The project being governed was a site: a relatively stable artifact, maintained by one operator, where the main problem was making sure the model did not forget what already mattered.

Part 7 moves into a harder setting.

The governed project is now a tool that runs inside AI workflows itself. That changes the governance problem. The issue is no longer only session amnesia. It is iterative accumulation: what the system learns across cycles, what becomes authoritative, what remains provisional, and what must be carried forward without allowing drift to harden into false memory.

Part 7 is that case.

Named decision from this post: A session report is not a polite summary. It is a hard gate. The model must declare — in the exact format dictated by the archive — what it loaded, what tests passed, and what drift it detected, before it is allowed to touch the repository.

MICA is part of the Flamehaven governance-first AI systems practice. Schema, technical report, and production instance: flamehaven.space. Open-source tooling: AI-SLOP-Detector. All schema references follow the v0.2.0 standard unless a specific earlier version is named.

How Auditing 10 Bio-AI Repositories Shaped STEM-AI

Kwansub Yun — Mon, 30 Mar 2026 12:07:20 +0000

Reading path:
This post is part of a series.
(1) STEM-AI introduction — what the framework is and why we built it
(2) Technical audit report — full findings across 10 repositories
(3) Narrative summary — what those findings actually mean

What Text Could See — and What Code Revealed

In March 2026, we ran STEM-AI against 10 high-visibility open-source bio/medical AI repositories.

The framework did what it was designed to do. It surfaced missing disclaimers, absent CI, weak reproducibility signals, and public-facing governance gaps. Those findings mattered, and the scores were directionally right.

But when we reviewed the audits more carefully, one pattern kept appearing: some of the most consequential failures were not visible in the artifact surface at all. They only became obvious when we looked directly at the code.

This function lives inside a repository presented as an AI-driven drug discovery workflow.

def generate_analogues(self, seed_smiles: str, count: int = 3):
    """
    Mocks a generative model (like REINVENT).
    In a real app, this would call a PyTorch model.
    """
    # Simple string manipulation for demo purposes
    analogues = []
    for i in range(count):
        if "C" in seed_smiles:
            new_smi = seed_smiles.replace("C", "C(C)", 1) if i == 0 else seed_smiles + "F"
            analogues.append(new_smi)
        else:
            analogues.append(seed_smiles + "C")
    return analogues

It does not generate molecules. It appends characters.

SMILES (Simplified Molecular Input Line Entry System) is a strict notation for molecular structure. A valid SMILES string encodes real geometry and bonding. Appending C produces a syntactically valid string that represents no real compound. The function runs without error, returns a list, and the pipeline continues downstream.

Our framework scored this repository T0. Correctly. But not because it saw this function.

It scored T0 because the README was missing disclaimers. The CI was absent. Reproducibility was undocumented. Text-path evaluation is designed to measure exactly that. It did.

The audit result was correct. The evidence surface had room to go deeper.

Running the audits showed us what code-path evaluation could add on top.

What Code Access Makes Visible

The drug discovery example was not unusual.

CellAgent's pipeline ends with this call:

# main.py, line 153
final_result = planner.generate_final_result(results)

The method exists. Its body is pass. The pipeline completes without error and produces nothing. A text audit reading the README would have no way to know this.

BioAgents includes a rate limiter for external API calls:

// rateLimiter.ts, lines 62-68
if (process.env.USE_JOB_QUEUE === 'true') {
    await this.rateLimiter.consume(userId);
} else {
    // Job queue disabled - skip rate limiting for direct calls
    return next();
}

USE_JOB_QUEUE defaults to false in .env.example. Every default deployment has rate limiting disabled. The function name implies protection. In default operation, there is none.

The pattern across all three:

the code looks governed.
the behavior tells a different story.
That story is only visible when you read the code.

Text scores and code behavior can diverge. Knowing where and how they diverge is the next layer of evidence worth capturing.

Four Directions the Audits Opened

Reviewing all ten audits, we identified four areas where code-path evaluation could extend what text auditing already does well.

Direction 1: Clinical exposure is visible in imports, not just in README text.

A repository importing pharmacogenomics allele tables has clinical exposure regardless of what its README says. Detecting that dependency at the import level — rather than waiting for a disclaimer — lets the framework flag exposure earlier.

The key distinction is severity: a direct pharmacogenomics import (CYP2D6, CPIC) signals live patient-facing risk and is classified CA-DIRECT.

A general-purpose medical imaging library like pydicom or MONAI is classified CA-INDIRECT — research-use exposure, not necessarily a live clinical output path. The import alone does not determine clinical risk; the classification tier does.

Direction 2: Not all clinical proximity is the same.

A live pharmacogenomics dosage tool and a README roadmap note about a future ClinVar integration are not equivalent risks. Differentiating them — live output vs. research context vs. planned feature — makes the evaluation more precise and makes the accountability expectations more appropriate.

Direction 3: Scoring stability is worth measuring directly.

We ran Stage 1 on one repository in multiple passes. The results ranged across 28 points on the same input. Overlapping trigger conditions between hype-detection items are one contributing factor.

LLM runtime stochasticity is another — the exact split between the two is still under measurement. Adding explicit discrimination examples — what exact phrasing triggers each item, what does not — makes the scoring surface cleaner and reduces the most obvious sources of variance.

Direction 4: Code-path behavior deserves its own scan layer.

A fail-open pattern is a control path that appears to enforce a constraint but defaults to bypassing it. The BioAgents rate limiter above is the example. In a clinical output path, a silent pass-through is not graceful degradation. It is an untraced result that looks like a real one. Building a dedicated scan for these patterns adds a check that text auditing was never meant to provide.

These four directions came directly from running the audits. The scores across the 10 repositories remain as published. Code-path evaluation is what the framework can now add on top of them.

What v1.0.6 Added — Carried Forward in v1.1.2

These changes were introduced in v1.0.6 and are carried forward in the current internal v1.1.2 package. They extend the framework's evidence surface into code-level behavior. Calibration is ongoing.

1. Two Evidence Paths, Not One

We narrowed one of the biggest divergence points by splitting evaluation into a text path and a code path.

The text path works as before: read the README, CHANGELOG, and public posts, score against the rubric. Always available regardless of access to the repository.
The code path activates when the audit has a local clone. It runs through Claude Code, Codex CLI, Gemini CLI, or Copilot CLI. Claims are not interpreted. They are measured. A README that says "IRB-approved data" earns no points for the statement. Points require a provenance artifact in the code.

When code confirms the README, that is a positive signal. When it contradicts it, that contradiction is the finding.

2. Clinical Dependency Detection

At the start of every local audit, a scan script reads Python imports and README keywords. It classifies the result into one of three severity levels:

# ca_detection_scan.sh -- pharmacogenomics section

# CA-DIRECT: live patient-facing output risk
check_import "CPIC|cpic|PharmGx|pharmacogenomic" \
  "Pharmacogenomics (CPIC/PharmGx)" "CA-DIRECT"

check_import "DPYD|CYP2D6|CYP2C19|allele" \
  "Pharmacogene alleles" "CA-DIRECT"

# CA-INDIRECT: research-use clinical exposure
check_import "import pydicom|from pydicom" \
  "pydicom (DICOM imaging)" "CA-INDIRECT"

check_import "import monai|from monai" \
  "MONAI (medical AI)" "CA-INDIRECT"

Accountability requirements follow the actual clinical proximity of the code. Not the aspirational proximity of the roadmap. A roadmap mention without active implementation is treated as CA-PLANNED rather than collapsed into the same bucket as live clinical output.

The pattern matching is against import statements and function names, not comment text. False positive calibration is still in progress.

3. Code Integrity Scanning (C1-C4)

A second scan handles four code-level checks:

hardcoded credentials (C1)
unpinned dependencies (C2)
clinical-path stubs (C3)
fail-open exception handlers (C4).

The C4 check targets the BioAgents-style pattern. Searching for clinical keywords on the except: line misses most real cases. Clinical context lives in function names and surrounding code. The scan uses a two-pass approach: first identify files with clinical-domain context, then find silent exception handlers within those files.

A silent except: pass in a clinical-context file is a trust-surface failure. The scan makes it visible without requiring a reviewer to read every exception block manually.

4. Discrimination Examples

To reduce the 28-point variance, we added explicit examples for each hype-detection item: what exact phrasing triggers it, what does not, and what the documented edge cases are.

The goal is to reduce obvious scoring drift enough that the same repository is no longer interpreted as two different trust surfaces across different auditors or different LLMs. That goal is not yet verified. The discrimination examples are the primary mechanism toward it.

The Three Questions Now Have a Fourth

In the first post, we described three questions the framework was built around:

Did the repository describe its limits honestly?

Did public communication remain consistent with those limits?

Did the codebase show evidence of maintenance and biological responsibility?

Running 10 real audits pointed toward a fourth:

4. Does the code actually do what the documentation says — and where it diverges, is that divergence visible and traceable?

That fourth question is what the audit outputs kept surfacing. A function name that sounds real. A pipeline that looks complete. An output that is plausible. An implementation that is a stub, or a control path that silently bypasses its own constraint.

The first three questions can be answered by reading. The fourth requires looking at the code.

What the Framework Added — and What Stays the Same

The first post ended with this:

"STEM-AI is meant to support serious review, not replace it."

That has not changed. Every report carries a non-removable disclaimer: LLM-generated audit, not a regulatory determination, not clinical certification. Every report carries an expiry date. The minimum threshold for supervised pilot consideration is still T3. None of the March 2026 repositories reached it.

What the audits added is narrower — broader evidence coverage on top of an already-working foundation:

A verifiable artifact shifts the accountability surface — it does not eliminate the possibility of falsification. The framework treats its presence as a necessary condition, not a sufficient one.

What the framework gained is that the evidence it counts now extends beyond what authors say about their own code.

Three Directions Still Ahead

Automated re-audit on repository changes. A score from three months ago may not describe the same repository. The trajectory signal measures issue close rate and release frequency across consecutive 90-day windows. It is a partial answer. A CI-triggered re-audit path is the logical next step.

The denominator problem. Zero of 10 repositories reached T3. This may accurately describe the ecosystem's current state. It may also reflect calibration issues in the upper tiers. Distinguishing between the two requires before-and-after auditing of repositories that have received systematic governance remediation.

The Stage 2 redistribution question. Most audits have no cross-platform consistency data. When that data is unavailable, the framework redistributes Stage 2's weight equally between documentation quality and engineering accountability.

For repositories with clinical-direct exposure, a well-written README can then compensate for weak code accountability. A guardrail flags this condition. The current redistribution rule is explicit but not yet final — it remains one of the framework's open calibration questions.

If there are open-source bio-AI repositories you think should be >audited next, drop them in the comments. Bonus if they claim clinical >relevance, drug discovery, or medical reasoning.

STEM-AI v1.1.2 — Trust Evaluation Framework for Medical AI Repositories.

"Code works. But does the author care about the patient?"

Everyone Was Talking About Context Engineering. Nobody Had Solved Governance.

Kwansub Yun — Wed, 25 Mar 2026 09:29:42 +0000

Disclosure: This article was written by the author with AI assistance for editing. All technical content, architecture decisions, and design rationale are the author's own. #ABotWroteThis

Glossary: terms used in this article

🔸 MICA (Memory Invocation & Context Archive): A governance schema for AI context management. Defines how context should be structured, trusted, scored, and handed off across sessions.

🔸 Fail-Closed Gate: An admission rule that excludes a context item if it fails a required threshold — regardless of its score on other dimensions. No exceptions. Introduced in v0.1.7.

🔸 README-as-Protocol: The pattern in which an AI session's natural behavior of reading the README first is formalized as the primary invocation mechanism. No installation required. Introduced in v0.1.8.

🔸 Invocation Protocol: The schema-level declaration of how a MICA archive reaches an AI session — and how the session confirms it was loaded. Formalized as a required field in v0.1.8.

🔸 Session Report Format: The structured opening report the model must produce at session start to confirm the archive was loaded. Required in v0.1.8.

🔸 Design Invariant Entry: A structured governance rule with identity, rule text, and severity. Replaced plain string invariants in v0.1.8.

🔸 Self-Test Policy: Machine-evaluable checks that validate the archive against the real project state — file existence, hash integrity, and README sync. Required in v0.1.8.

🔸 Playbook: The operator-facing discipline layer that sits outside the schema. The schema enforces structure; the Playbook enforces judgment.

🔸 Context Engineering: The practice of shaping what the model sees, in what order, with what boundaries, and under what assumptions — not just what you ask it, but what it actually knows at runtime.

🔸 CTX: A collection-first context packaging approach that gathers relevant workspace material and delivers it to the model. In this article, CTX represents the collection layer of context engineering — answering, “What does the AI see?”

1. What Parts 1 Through 4 Actually Established

The first four parts of this series already narrowed the problem considerably.

Part 1 defined the failure mode.
The issue was not that long-running AI work needed “more prompt.” The issue was that a model can only act on what it actually knows right now, and most project context systems still treat that as a document problem instead of a governance problem.
Part 2 established the first hard boundary.
A schema can exist, and the model can still have no reliable way to know it exists. That is the difference between a document and a constraint.
Part 3 moved governance into the schema.
Provenance, deviations, and semantic rules could no longer remain outside the system in READMEs, comments, or team habits. They had to become machine-readable structure.
Part 4 made that structure operative.
The model already treated the README as its natural entry surface. Once that behavior was declared as an invocation protocol, the schema stopped being a passive archive and became a runtime contract.

That progression defines what MICA is actually trying to solve.

It is not trying to be “better prompting.”

It is not trying to be “more retrieval.”

It is trying to answer a narrower question:

How does governed context reach the model, under declared rules, with confirmable load and auditable change?

That is the bridge into the broader landscape.

2. Context Engineering Was Never Just Prompting

One clarification matters before going further.

Context engineering is not just prompt writing.

At the broadest level, it is the practice of shaping what the model sees, in what order, with what boundaries, and under what assumptions. Prompts are one part of that. Retrieval is another. File selection, memory handoff, system instructions, and workspace state all belong to the same larger question:

What does the model actually know right now?

By that definition, MICA is part of context engineering.

Not because it retrieves context.

Not because it packs more tokens into a window.

But because it governs which context is allowed to shape the session, under what trust conditions, and with what record when those conditions are tested.

That distinction matters, because most of the field has focused on a different layer.

3. The Conversation Was Already Happening

Context engineering is not a new idea, and MICA does not claim to have invented the conversation.

The term was amplified by Andrej Karpathy and others, but the underlying practice — designing what the model sees, not just what you ask it — had already been emerging in serious AI work.

Collection-first tools already existed. CTX is a useful example of that layer: it gathers relevant workspace material and delivers it to the model without manual copy-paste. It answers an important question well:

What does the AI see?

At the same time, some of the sharper practitioner writing was already moving beyond collection alone. One such example was an OpenAI Developer Community post by Serge Liatko, “Prompt Engineering Is Dead, and Context Engineering Is Already Obsolete: Why the Future Is Automated Workflow Architecture with LLMs.”

The value of that piece was not the author's status, but the precision of the problem it named: manually maintained context eventually reaches its ceiling.

Once system state changes faster than humans can keep context aligned, the real question is no longer just how to collect context, but how to automate its ownership, maintenance, and validation as the system evolves.

That was an important move forward.

But one layer still remained underdefined.

4. The Missing Layer Was Already Visible

The same missing layer had already shown up elsewhere.

In Your Agentic Stack Has Two Layers. It Needs Three, I argued that the usual stack had matured around two strong layers:

MCP / tool calls — how the agent talks to systems
agent skills — what the agent can do

But something was still missing above both:

the layer that decides whether the agent should do it, under what constraints, and toward what end

That is the layer of intent, authority, and governance.

The same problem appears in context engineering.

A context pipeline can be excellent at retrieval and still be weak at governance. It can gather the right files, summarize the right notes, and deliver the right-looking material — and still fail to answer:

what is authoritative?
what is provisional?
what must never be violated?
what changed since last time?
how does the session prove it loaded the governed archive at all?

Those are not collection questions.

They are governance questions.

5. Where CTX Stops and MICA Begins

CTX solves the collection problem.

It answers:

What does the AI see?

That is a necessary layer. Without it, context management collapses back into manual copy-paste, repeated explanation, and fragile session startup.pro

But it does not answer the next set of questions:

Where did this context item come from, and can that claim be verified?
What happens when a file changes between sessions?
Which constraints are non-negotiable, and what is the consequence when they are violated?
Who approved the last change to the archive, and can that decision be audited?
When the session begins, how does the system confirm the archive was actually loaded?

Those are not retrieval questions.

They are governance questions.

That is the narrow but consequential difference.

CTX collects context and delivers it.

MICA governs what trust that context carries, what invariants it must not violate, what happens when it changes, and how the session proves that the governed archive was actually loaded.

One answers:

What does the AI see?

The other answers:

Under what rules does the AI operate — and what is the record when those rules are tested?

They are different layers. Neither replaces the other.

6. The Part That Was Still Open

A recurring theme in serious context-engineering discussion is that context cannot remain a hand-curated artifact forever. It has to become a function of system state.

But that still leaves one question open:

Who owns the specification for each step's input — and how is this versioned, tested, and audited as requirements shift?

MICA is a concrete, working answer to that gap.

Not the only possible answer. Not necessarily the final one. But a real one.

Its claim is not that context engineering needed to be invented.

Its claim is that context engineering still needed a governance layer with at least four properties:

machine-addressable invariants
versioned and auditable change records
self-tests against the real project state
declared invocation with confirmable session load

That is the layer MICA was built to supply.

7. What Governance Actually Means Here

Governance is an overloaded word. In this context it means something specific.

Provenance

Every context item must declare where it came from in a way that can be checked.

Auditability

Changes to the archive are recorded when they happen, not reconstructed later from memory.

Invariant enforcement

Constraints are not vague README prose. They are structured entries with identity and severity.

Self-testing

The archive is checked against the real project state, not only against its own internal shape.

Invocation confirmation

The model does not silently ignore the archive. Session start requires a structured acknowledgment that the governed archive was loaded.

None of these are abstract principles in MICA.

They are structural requirements in a running schema.

8. What This Is Not

It is worth being precise about the boundary.

MICA does not generate context automatically from the codebase. That is not its job. Collection-first systems already exist, and they are valuable. MICA governs what happens once context has been identified.

MICA does not replace human judgment. A schema can require structure, audit trail, drift response, and self-tests. It cannot eliminate operator discipline. That is why the boundary between schema and playbook matters.

MICA is also not a finished system. Parts 1 through 4 of this series were explicit about what each version got wrong, what each version corrected, and what remained unresolved.

That design history is part of the claim, not an embarrassment to it.

9. The Actual Claim

The actual claim is not that MICA solves all of context engineering.

It is this:

A small operation can already run a governed AI context system — with verifiable provenance, deviation audit trail, structured invariants, self-testing, and declared invocation — without waiting for future tooling that does not yet exist.

That claim is demonstrated by the design history already covered in this series:

v0.1.0 made scoring implementable
v0.1.5 brought governance structure into the schema
v0.1.7 made scoring fail-closed
v0.1.8 made invocation declared and confirmable
v0.1.8.1 clarified the remaining runtime ambiguities

And in practice, governance at runtime can look as concrete as this:

[SESSION READY]
Gate: PASS (self-tests: 7/7) | Track: A,B
Critical Invariants: 3/3 | Deviations: 0

If a critical check fails, the session does not proceed. That is what governance looks like when it becomes operative.

What can be said now is narrower and more solid: the gap is real, collection-first systems solve one side of it, and MICA addresses the governance side. The conversation about what comes after context engineering was already happening; MICA is one concrete answer to that part of it.

10. What Comes Next

Part 4 ended with a specific question:

Where does MICA sit in the context engineering landscape that already existed around it?

This post is the answer.

It sits inside context engineering — but not at the collection layer.

It does not compete with retrieval-first systems by trying to collect more files, pack more tokens, or automate more handoff. It begins after that layer. Its job is to govern what enters the session, what remains authoritative, what drift means, what violations matter, and how the model proves it actually loaded the governed archive at all.

That is why the answer is narrower than most people expect, and more specific than most framings allow.

MICA is not “the future of all context engineering.”
It is a governance answer to the part of context engineering that collection alone does not solve.

Part 6 will move back down from landscape to concrete operation.

It will show what MICA looks like in an actual project context: what a session opening report looks like, what a deviation log entry looks like in practice, and what happens when a self-test flags drift.

After that comes the harder question: what remains unresolved.

The series continues only where there is something concrete to specify, test, or correct.

Named decision from this post: Governance is not a layer you add after context engineering works. It is the layer that makes context engineering trustworthy — by declaring what is authoritative, recording what changes, and confirming what the AI actually loaded.

MICA is part of the Flamehaven governance-first AI systems practice. Schema, technical report, and production instance: flamehaven.space. Open-source tooling: AI-SLOP-Detector. All schema references follow the v0.1.8.1 Universal standard unless a specific earlier version is named.