DEV Community: mars70s

We Had a Single Source of Truth—and Still Had Stale State

mars70s — Mon, 12 Oct 2026 10:02:27 +0000

We had already chosen a Single Source of Truth.

The canonical state file was correct.

Later work updated that canonical state.

And yet another document still contained an obsolete view of the current work.

At first glance, this looks like a simple documentation mistake:

Someone forgot to update a file.

The Git history showed something more useful.

The stale copy was already inconsistent at the commit that introduced the Single Source of Truth rule.

Later updates did not create the inconsistency. They widened it.

That distinction changed how I thought about the fix.

The migration looked correct

The repository used several documents to manage an ongoing technical-writing workflow.

One of them, active/WORK_STATUS.md, was selected as the sole mutable authority for current state.

At the same time, a worker-instruction document, AGENTS.md, was changed so that it explicitly said current state should not be duplicated there.

Conceptually, the new rule was:

WORK_STATUS owns current state.

AGENTS contains durable worker rules.

AGENTS should not duplicate mutable current state.

That is a reasonable authority model.

But the migration was incomplete.

One section of AGENTS.md had been cleaned up.

Another section still contained state such as:

current target = TOP-04
current work boundary = early evidence construction
formal draft work should not proceed yet

The problem was not only that those values existed in two places.

They were already different.

The canonical WORK_STATUS.md had moved further ahead than the remaining state text in AGENTS.md.

So the repository contained this contradiction:

AGENTS:
Do not duplicate current state here.

AGENTS, another section:
Here is the current state.

Later updates followed the new authority model

After that migration, the workflow continued.

The formal draft moved forward and was Human Approved.

WORK_STATUS.md and related project documents were updated.

AGENTS.md was not part of that particular state-update commit.

That alone is not evidence of a problem.

If WORK_STATUS.md is the mutable state authority, updating the authority instead of every historical copy is exactly what the new model says to do.

But because the old copy had not been removed during migration, the difference became more visible.

The interesting part came later.

AGENTS.md itself was edited several more times for unrelated policy changes.

The obsolete state section still survived.

So the persistence cannot be explained only as:

AGENTS was outside the write scope.

The file was opened and changed later.

The stale section simply was not identified as part of the authority migration problem.

That made the detection gap more important than the individual write scope.

The Single Source of Truth was not the problem

This case did not show that SSOT is ineffective.

The canonical state did its job.

The problem was migration completeness.

Declaring:

WORK_STATUS is now authoritative

does not automatically remove every older mutable copy.

A migration has at least two sides:

1. Select the new authority.

2. Remove, convert, or explicitly retire the old mutable replicas.

If step 2 is incomplete, a stale copy can survive outside the canonical update path.

That is especially easy to miss when the old copy appears inside a document that otherwise contains durable rules.

A policy document can slowly become a mixed object:

durable rules
+
temporary current-state snapshot

Once that happens, it becomes unclear which parts should change when project state changes.

The tempting fix was not the structural fix

The easiest correction would have been:

TOP-04 → whatever the latest target is

That would make the document look correct again.

But it would preserve the same ownership problem.

The next state transition could make the copy stale again.

Instead, the relevant section was changed so that values such as:

current target
next gate
evidence authority
per-item state

were no longer duplicated there.

The durable document now points back to the mutable authority.

Conceptually:

Before:

WORK_STATUS:
current target = X

AGENTS:
current target = X


After:

WORK_STATUS:
current target = X

AGENTS:
read current target from WORK_STATUS

The fix was therefore not:

Update the old value.

It was:

Remove ownership of that value from the wrong document.

Bounded write scopes were not the villain

There was another tempting explanation:

Strict write scopes caused the stale state because nobody was allowed to touch the other file.

The Git history did not support that conclusion.

A bounded change scope does something useful: it limits the files changed by a particular operation.

What it does not do is discover every pre-existing inconsistency outside that scope.

Those are different properties.

So a safer conclusion is:

A bounded change scope can preserve a pre-existing inconsistency
without being the cause of that inconsistency.

That matters in tightly controlled Git workflows.

“Only change the authorized files” is still a useful rule.

It just should not be confused with a repository-wide semantic consistency check.

Syntax can be correct while document roles are wrong

Nothing about this failure required broken Markdown or an invalid Git state.

A document could be syntactically valid.

The diff could be clean.

The intended files could be the only files changed.

The canonical state could be correct.

And the repository could still contain a contradiction about which document owns current state.

That is a different type of invariant.

For example:

AGENTS must not retain duplicated mutable current-state values
when WORK_STATUS is the canonical mutable authority.

A normal diff or syntax check does not automatically establish that invariant.

This does not mean every repository needs another giant validation framework.

In this case, the useful lesson was much smaller:

When converting a project to a Single Source of Truth, explicitly look for the old mutable copies once.

Remove them, convert them into references, or mark them as historical.

Then go back to building the actual thing.

What I took from the case

My practical takeaway is:

An SSOT migration is not finished when the new authority is declared. It is finished when the old mutable replicas no longer compete with it.

A few narrower lessons followed:

Durable policy documents should avoid owning ephemeral project state when another file is authoritative.
Updating a copied value is sometimes legitimate, especially for an explicitly subordinate summary.
A bounded write scope protects scope; it does not repair unrelated existing inconsistencies.
Cross-document semantic invariants are different from syntax and diff correctness.
Sometimes the better fix is to remove state ownership rather than keep synchronizing copies.

This is one repository case, not evidence that every SSOT migration fails this way.

It is also not an AI-specific failure.

The mechanism is fundamentally about document authority, mutable state, and incomplete migration.

The full case study includes the commit chronology, evidence boundaries, and what each commit does and does not establish:

https://osiix.com/en/library/single-source-of-truth-stale-state.html

AI disclosure: AI-assisted drafting and review tools were used during preparation. The underlying Git evidence, claim boundaries, corrections, and publication decision were reviewed by the author.

Six Git Checks After an AI-Assisted Change — And What They Cannot Guarantee

mars70s — Thu, 08 Oct 2026 14:15:41 +0000

A coding assistant changes the two files you asked it to edit. The diff looks reasonable. Is the next Git operation safe to proceed?

Not necessarily. You might be in the wrong repository. An extra file might already be staged. The branch or configured push destination might have changed. Or the secret scanner might have failed to run at all.

These are not necessarily mistakes in the generated code. They are mismatches between the workflow you authorized and the Git state that actually exists.

I built a small, public reference Git Safety Harness to check selected mismatches before a later operation. It is a PowerShell-based, fail-closed workflow control—not a sandbox, not an independent enforcement boundary, and not a guarantee of safe AI-generated code.

A two-file example

Suppose the authorized work is:

Edit docs/article.md and public/article.html in a particular repository.
Stay on main.
For the next operation, stage only docs/article.md.
Check the configured push URL for origin against an explicitly approved URL.
Run a staged-diff secret scan before any separately authorized push.

The harness does not invent these expectations. A human or an authorized workflow must define them first. The harness then compares selected observed Git state against them.

Six checks, six different questions

Check	Question before proceeding	Example of a reason to stop
Repository Root	Is this the intended Git repository?	The command is running in another valid checkout.
Write Set	Are observed tracked changes and non-ignored untracked paths within the allowed set?	`README.md` changed even though it was not authorized.
Stage Set	Does the actual Git index contain only explicitly allowed staged paths?	An unrelated file was staged earlier.
Branch	Is the current symbolic branch the expected branch?	HEAD is detached or the branch differs.
Push URL	Does the selected remote have exactly one configured push URL, matching the approved URL?	`origin` points somewhere else, or has multiple push URLs.
Secret Scan	Can Gitleaks run and accept the staged diff?	Gitleaks is missing, cannot execute, or exits nonzero.

Each question has a different failure mode. A PASS from one check does not substitute for the others.

Why Write Set and Stage Set are separate

The Write Set asks which paths are allowed to change. The Stage Set asks a different question: which paths are allowed in the Git index for the next operation?

The reference implementation reads the actual index with git diff --cached --no-renames --name-only and compares staged paths with a separate allow-list. That is a separate comparison from the Write Set, which covers staged and unstaged tracked changes as well as non-ignored untracked paths, using its own allow-list.

Example: an allowed change, but an unapproved staged file

The approved Write Set contains both docs/article.md and public/article.html. The approved Stage Set contains only docs/article.md. Now imagine the following hypothetical output, not a captured test transcript:

$ git diff --cached --no-renames --name-only
docs/article.md
public/article.html

Both paths are allowed to change, so their presence alone does not violate the Write Set. But public/article.html is not allowed in the index for this operation. The reference implementation treats this as a Stage Set mismatch and reports GSH_STOP_STAGE_SET instead of accepting that staged state.

What would happen next? Consider two hypothetical paths from this same Git state:

Without an effective Stage Set check: If the workflow now runs git commit without changing the index, both docs/article.md and public/article.html are included in that commit. The second file was permitted to be edited, but was not authorized for this commit. If that commit is later pushed under a separate authorization, the unintended change can reach the remote repository. Neither a commit nor a push is claimed to have occurred in this example.
With the harness invoked and its exit respected: The Stage Set mismatch triggers GSH_STOP_STAGE_SET and a nonzero result. The calling workflow stops before its later commit or push action, and the staged files remain for inspection—the harness does not silently unstage or repair them. A human can then decide whether to correct the staging, change the authorization, or stop the work.

This illustrates a workflow boundary, not a measured incident or a new test result. It also does not mean that the harness itself blocks direct Git commands: if an actor bypasses the checked path, the local Layer-2 harness cannot enforce that stop.

Likewise, checking a remote's fetch address is not enough to establish its push destination. The harness inspects push URLs and requires one exact match.

What fail-closed means here

If a required state cannot be confirmed, the normal harness path stops rather than treating missing evidence as success.

That includes a repository root that cannot be resolved, an unexpected branch, an unavailable Gitleaks executable, and a scanner execution failure. The validated PowerShell wrapper also treats invocation errors, a missing child exit status, and nonzero exits as failures.

STOP is not an instruction for the assistant to silently repair the situation and continue. It is a point for inspecting the observed state and obtaining any necessary human decision.

What was actually tested?

The commit-pinned Validation Record reports results for the identified implementation in documented Windows reference environments:

28/28 fail-closed cases passed: 14 under PowerShell 7 and 14 under Windows PowerShell 5.1.
6/6 success-path cases passed: three under each shell, including a non-empty permitted staged diff scanned with real Gitleaks.
The record identifies the source and test files using SHA-256 hashes and the versions used: Git 2.54.0.windows.1, PowerShell 7.6.5, Windows PowerShell 5.1.26100.9444, and Gitleaks 8.30.1. A later preserved revalidation used the same source/test identities with PowerShell 7.6.6.

The tests used synthetic local remotes and recorded bounded results for the identified implementation and environments. They did not establish universal compatibility, unbypassability, or the safety of a later real-world push or deployment.

The Evidence Pack from that later revalidation preserves execution output, case results, exit codes, and a SHA-256 manifest. These are records of the later run, not the original run’s raw logs.

What the harness cannot guarantee

Three limits matter especially in AI-assisted workflows:

Allowed does not mean correct. A change inside docs/article.md can still be technically or semantically wrong.
A local check can be bypassed. If a person or agent can invoke raw Git, shell commands, or APIs outside the harness, the harness does not itself prohibit those paths. Independent restrictions belong outside this local workflow control (Layer 2), in a separate enforcement layer.
The check is point-in-time. State can change after verification. The harness checks a configured push URL but does not perform or authorize the later git push. Its Gitleaks check covers the staged diff examined by the scanner, not every file or every secret.

Those limitations are reasons to keep human approval separate from independent enforcement, such as repository-side controls. Human approval is a workflow decision, not independent enforcement; the local harness replaces neither.

The useful boundary

The goal is not to prove that an AI assistant is safe. It is narrower: before a selected Git workflow continues, make certain mismatches observable and stop when required checks cannot be accepted.

The full English design article on OSIIX explains the checks in detail. For the current implementation—including its separate AllowedStagedFiles parameter—use the commit-pinned source, Getting Started guide, and limitations, rather than treating every illustrative snippet in the earlier article as the latest tested code.

If you use AI-assisted Git automation, which of these checks is enforced outside the agent-controlled execution path, and which is only a convention the agent is expected to follow?

A Test Passed. But What Did It Actually Test?

mars70s — Wed, 07 Oct 2026 14:21:08 +0000

A test passed. That sounds reassuring.

But before treating that PASS as evidence, there is a more basic question:

What exactly did the test execute?

It is surprisingly easy to lose the connection between a test result and the implementation that result is supposed to validate. This became especially visible while reviewing a validation record from my own development work.

A PASS is not enough

A test result normally tells us something about a specific target under specific conditions.

At minimum, I want to know things such as:

What file or implementation was executed?
Which version was it?
Which test script produced the result?
Under what environment was it executed?
Can the result still be tied to that same target later?

Without that information, a PASS may be real while still being weak evidence for the claim we want to make.

The problem is not necessarily that the test itself is wrong. The problem may simply be that the evidence does not establish the identity of the thing that passed.

I initially made a stronger claim than the evidence supported

While reviewing an earlier validation record, I initially interpreted the evidence as a specific failure mechanism.

The working explanation was roughly:

One implementation may have been treated as evidence for another.

That was a coherent explanation, but the surviving evidence did not support it strongly enough.

When I went back through the retained scripts, outputs, filesystem metadata, and validation records, I could confirm one thing: the record itself did not identify the runner, the implementation, or the invocation path behind its PASS results.

But I could not reliably reconstruct the stronger causal details.

For example, the surviving evidence was not enough to establish:

what the intended validation target was,
whether a retained run script, which executed local functions, was the run that produced the PASS record,
whether those local functions were intended as a test double,
how the files involved relate to each other (lineage, copy direction, or creation order),
or whether AI was involved at all.

Those remained unknown.

So the stronger explanation had to be withdrawn.

What the evidence actually supported

The defensible conclusion was narrower:

The verification record did not sufficiently identify its executed validation target to attribute the PASS to a specific implementation from the record alone.

The finding is not:

"The wrong implementation was tested."

The evidence does not support that statement.

The observable problem was:

"The recorded PASS could not be attributed to a specific implementation from the record alone."

That distinction matters.

Why this matters in AI-assisted workflows

One caveat first: the evidence in this case does not establish whether AI was involved, and a single case says nothing about how often this happens. What follows is a general observation, not a finding from the case.

AI-assisted development can create many intermediate artifacts:

generated scripts,
wrappers,
temporary implementations,
test helpers,
copied code,
diagnostic scripts,
regenerated files,
and revised versions of the same component.

None of those are inherently bad.

But they increase the number of objects that can exist between:

what we intended to validate

and

what was actually executed.

If the validation record only says:

RESULT=PASS

we know very little.

Even this is better:

TEST=verify-something.ps1
RESULT=PASS

But it still may not tell us which implementation that test exercised.

For stronger evidence, the record needs enough identity information to reconstruct the relationship.

For example:

TARGET:
  path
  version or commit
  content hash where useful

TEST:
  path
  version or hash

ENVIRONMENT:
  relevant tool versions

RESULT:
  PASS / FAIL

BOUNDARY:
  what this result does and does not establish

The exact format is not important. The ability to trace the result back to the tested object is.

Where hashes are used as evidence, capturing them at the relevant execution or validation boundary is stronger than recomputing them later.

PASS should be a bounded statement

I now try to read a PASS as something closer to:

Under these conditions, this identified test produced this result against this identified target.

Not:

The implementation is correct.

And definitely not:

The system is safe.

Those are much larger claims.

A useful validation record therefore contains not only what was observed, but also what remains outside the evidence.

For example:

OBSERVED:
- identified test completed successfully
- expected exit code was observed
- target identity was recorded

NOT ESTABLISHED:
- universal compatibility
- production safety
- inability to bypass the control
- correctness outside the tested conditions

This may make the record look less impressive, but it also makes it much harder to accidentally reuse the result as evidence for something it never proved.

The uncomfortable part: correcting your own story

The most useful lesson from this case was not technical. It was methodological.

When investigating a result, it is easy to build a coherent explanation from incomplete evidence. Once that explanation has a name and several paragraphs around it, it becomes surprisingly difficult to abandon.

But the explanation is not the evidence.

If later review shows that the evidence only supports a narrower statement, the claim should become narrower too.

In my case, that meant replacing a satisfying causal story with a much more limited conclusion about target identity.

That was the right trade.

What a later validation record preserves

A later, separate validation record shows what more explicit identity information can look like. The historical case neither confirms nor invalidates that later record.

For one Git safety harness, the validation record includes items such as:

implementation commit,
source file SHA-256,
test file SHA-256,
Git version,
PowerShell versions,
Gitleaks version (a secret scanner),
individual fail-closed and success-path results,
and an explicit list of what the tests do not prove.

Here, fail-closed means stopping when a required check cannot be completed instead of treating an unverifiable state as success.

This does not make the harness universally correct or secure.

It does something more modest:

it makes the validation evidence easier to bind to the object that was actually tested.

A small checklist

Before accepting a PASS as evidence, I now ask:

What exactly was executed?
Can I identify its version or contents?
What test produced the result?
Under what conditions?
What claim does this result actually support?
What does it not establish?
Could I reconstruct that relationship later from the preserved evidence?

If those questions cannot be answered, running more tests may not solve the real problem.

The missing piece may be the identity of the verification target itself.

The underlying records for this case are not public, and the review was my own rather than a third-party audit.

"10 hours" and "around two weeks" are not the same kind of number

mars70s — Sun, 20 Sep 2026 16:44:27 +0000

In September 2026, Palo Alto Networks Unit 42 published a case involving
an AI-assisted intrusion into an enterprise network. The article includes
figures like "less than 10 hours," "more than 50 MITRE ATT&CK techniques,"
and "around two weeks" for what a human operator would normally need.

It's tempting to divide those numbers and conclude "AI made this 30x faster."
I don't think the source material supports that.

Two different kinds of numbers

"Less than 10 hours" is a reported duration of activity.

"Around two weeks" is Unit 42's comparative assessment of what a human
operator would typically require for impact of that scale.

The public article doesn't include a same-conditions human-only comparison,
team size, skill composition, tools used, or a detailed calculation method.
So "AI turned two weeks into 10 hours" isn't a measured speedup ratio —
it's two different types of numbers sitting next to each other.

The article changed after publication

I checked the Wayback Machine capture from the publication date against
the current version. Two things changed:

"as part of a ransomware attack" → "as part of a ransom attack" (the page's own update history later clarifies this was an intrusion, not a ransomware attack)
"public API endpoint" → "publicly accessible web service"

The second change isn't explained in the page's own changelog. I can't
establish when it happened or why — so I don't speculate. I just note
that the wording changed.

What's actually confirmed about AI use

Two separate evidence paths exist:

The threat actor's own account (told to Unit 42 during negotiations) that they used frontier AI models and attack-specific agentic frameworks
Technical indicators Unit 42 observed independently — parallel LLM calls, structured Markdown passed between agents, custom scripts

Both matter. Neither, alone or together, establishes that AI autonomously
ran every stage of the intrusion. Unit 42's own Figure 1 shows a human
actor setting objectives and making key decisions while agents execute —
closer to human-directed / agent-executed than fully autonomous.

A hypothesis worth testing: parallelization, not just speed

Here's where I'll go beyond what Unit 42 states. The public material
can't tell us how many times faster any single task became. But it does
describe multiple agents, parallel LLM calls, and state handed off across
sessions — which suggests a different question:

Maybe the effect of AI-assisted intrusion isn't mainly per-task speedup.
Maybe it's lower coordination cost and handoff latency when running
multiple workstreams in parallel.

If that's right, the interesting metrics aren't total elapsed time —
they're things like: how many agents/sessions ran concurrently, how much
independent tasks overlapped in time, how long from observation to next
action, what share of time went to human-approval waits.

This hypothesis also has a clear failure mode: if a detailed timeline
showed activity was mostly sequential, with few concurrent agents and
long approval waits, the parallelization story falls apart.

Full writeup

I go through the source comparison, the evidence paths, and the
full list of what remains unknown in more detail here:

https://osiix.com/en/library/unit42-ai-assisted-intrusion.html

This article independently reviews public material and does not
indicate any approval, supervision, or affiliation by Palo Alto
Networks or Unit 42.

Restorable Backup Safe Cyber Recovery

mars70s — Sat, 19 Sep 2026 13:58:41 +0000

You have multiple generations of backups. The newest one restores fine.
Ransomware was detected today. But you don't actually know when the
intrusion started.

The natural instinct: restore the latest backup. But "restorable" and
"safe to adopt as your recovery point" are not the same claim.

This is one chapter from a self-study book I've been writing on Backup /
Recovery / Archive design. Here's the short version.

Latest ≠ Correct

A backup taken after compromise may already contain:

encrypted data
corrupted state
some other form of unsafe state

This doesn't mean "newer is dangerous." It means newness alone doesn't
prove correctness.

False signals of "clean"

The chapter lists reasons people wrongly declare a backup safe — none of
which, alone, proves it:

it restored successfully
it was protected as immutable
the hash matched
antivirus scan passed
it predates the detection time

Each is a useful signal. None is proof on its own.

Two checkpoints, not one

NIST's guidance on cyber recovery separates verification into two stages:
checking the recovery asset before you use it, and checking the restored
system before you put it back into production. One check is not enough.

The discipline that matters most

"Don't turn 'unknown' into 'safe.'"

When the intrusion start time is uncertain — which it often is — that
uncertainty doesn't disappear just because you'd like to move forward.
The chapter treats "we don't know" as an honest state, not a failure to
be papered over.

Full chapter: https://osiix.com/en/library/backup-recovery-archive-fundamentals/chapter-11.html
Book index: https://osiix.com/en/library/backup-recovery-archive-fundamentals/