DEV Community: Jeremy Longshore

Erasure Requests: Model Provider Cleanup From Reservations

Jeremy Longshore — Tue, 03 Nov 2026 12:02:12 +0000

When a product has to honor an erasure request, the cleanup obligation owed to a hosted model provider has to come from what was possibly sent, not from what we can prove was sent. We had it derived from the wrong signal, and the two directions of that failure were not equally bad: one asks a vendor for a cleanup of data it never received, the other can leave personal data sitting at a vendor outside our control.

What follows is the rule that replaced it, the SQL condition that implements it, a reversible migration (a database change you can roll back) that installs it, and a regression module that reproduces both defects.

The product here is an onboarding system. Its assessment path sends applicant data to a hosted model provider to score assessments, and its reply interpreter uses the same provider. When a person asks to be erased, the system computes a handoff list: the external parties that now owe a cleanup because they may hold that person's data. The model provider is one entry on that list, written as model_provider.

Our own database holds narrower evidence about a provider call than it does about a row. We record that we reserved a request before sending it, and we record that a provider call consumed requests. Between those two records sits the part we cannot resolve: whether a reserved request that never completed actually crossed to the vendor. Any obligation we owe that vendor has to be derived from the wider of the two records, the reservation rather than the consumption count, because the two ways of getting it wrong cost very different amounts.

The old rule asked only whether a run row existed

The old rule lived in a catalog module (the Python file that generates the erasure function's SQL), inside the SQL body of the function that builds the handoff list. It looked like this:

  IF EXISTS (SELECT 1 FROM public.assessments_assessmentrun r WHERE r.application_id = ANY(v_apps)) THEN
    v_handoffs := v_handoffs || 'model_provider'::text;  -- what the model provider kept of a request, if anything
  END IF;

Any assessment run row at all for the erased person's applications was enough. No column was compared. The query could not tell a run that consumed a provider call from a run that was created and then refused before a call was ever made.

Both defects came out of that one line, and they point in opposite directions.

False positive. An assessment refused on suspension produced a run row and no provider call. The old rule still reported model_provider. The vendor owed nothing, and it would have been asked to clean up data it never received.

False negative. A person's inbound reply can be reserved for interpretation by the provider while no assessment run exists for them. The legacy predicate found no run row, so the entry was omitted. The provider might have received the reply, and nobody would have asked it to drop anything.

Neither defect is something you find by reading the SQL twice. The condition looks conservative, and conservative is the right instinct here. It is just conservative in the wrong place: careful about the row's existence, careless about the request's fate.

How do I derive a model provider cleanup obligation?

We derive it from what was possibly sent, not from what we can prove was sent. A reserved provider call counts as evidence of disclosure even when the call never completed, because an unfinished run is not evidence of absence. An abandoned call is the harder case, because there the run exists and did not finish, and the reason is that the provider's side of the call is not something our database can resolve. When a run is refused before anything is reserved, no handoff is owed.

The new predicate lives in erasure/catalog.py as MODEL_PROVIDER_HANDOFF:

MODEL_PROVIDER_HANDOFF = """EXISTS (SELECT 1 FROM public.assessments_assessmentinvocation i
             WHERE i.application_id = ANY(v_apps) AND i.reserved_requests > 0)
     OR EXISTS (SELECT 1 FROM public.assessments_assessmentrun r
                WHERE r.application_id = ANY(v_apps) AND r.requests_used > 0)
     OR EXISTS (SELECT 1 FROM public.correspondence_replyinterpretation i
                WHERE i.application_id = ANY(v_apps) AND i.reserved_requests > 0)"""

Three signal sources and two different columns:

Signal	Column	What it establishes
Assessment invocation (assessment and clarification rounds)	`reserved_requests > 0`	A provider call was reserved. A possible disclosure, even if the call never completed.
Assessment run	`requests_used > 0`	A provider call consumed requests. A confirmed disclosure.
Reply interpretation	`reserved_requests > 0`	Reply-interpretation provider work was reserved. A possible disclosure.

The rule now reads, as recorded in the internal launch plan: a queued or suspension-refused run that reserved nothing and recorded nothing owes no model provider handoff. Positive assessment and clarification reservations, recorded assessment requests and reply interpretation reservations keep the handoff, including abandoned calls whose provider outcome is uncertain. Reply-only calls count even without an assessment run.

The inversion that matters is that a reservation became a first-class signal instead of an implementation detail. Previously a provider call surfaced only as an assessment run row. The reservation was always recorded and the old rule never looked at it. Now the rule reads that column, so the ambiguous middle (reserved, then interrupted, then abandoned) is answered from a recorded number instead of inferred from the mere existence of a run row.

Why not count only confirmed calls, or ask the provider?

The obvious fix is to count only confirmed calls: drop the reservation checks, keep requests_used > 0, and let the ambiguous cases fall on the floor. It removes the false positive entirely and the query gets shorter.

It also creates a silent false negative, and that is the bad trade. A call that was reserved and then interrupted leaves no confirmed record, but the provider may have received it. The absence of a finished run is not evidence of absence of disclosure. When the evidence is missing rather than negative, the only safe reading is that a disclosure is possible, and a possible disclosure still creates a cleanup obligation.

The other obvious approach is to ask the provider. Read the vendor's logs for the person's identifiers and use the answer. That makes the vendor a runtime dependency of an erasure request, it assumes the vendor can match our identifiers to what it received, and it assumes the vendor keeps logs long enough to answer. None of those hold in general, and the cost when they do not hold is the person's data still sitting there.

So the obligation is deliberately a superset of confirmed disclosures, and we accept the false positive the new rule keeps. That false positive costs a message to a vendor and some noise in our own logs. The false negative can cost a person their right to be forgotten.

One more reason the set has to lean wide: we do not get to know what the provider does with a request after we send it. Retention, training and logging on their side, if they happen at all, happen outside our database. Our own records tell us a request was reserved for sending, and a confirmed call (requests_used > 0) tells us the provider consumed it. Neither tells us it kept anything. The reservation is our evidence that it might have gone out.

Installing it without rewriting history

The change ships as a reversible migration, erasure/migrations/0008_model_handoff_evidence.py:

"""9kg.73: queued or refused model work alone creates no provider cleanup obligation.

Reservations remain conservative evidence of possible disclosure, including abandoned calls
and reply interpretation without an assessment run. Reinstall both erasure functions without
rewriting rows, signed records, function ownership or grants.
"""

# imports elided

LEGACY = "EXISTS (SELECT 1 FROM public.assessments_assessmentrun r WHERE r.application_id = ANY(v_apps))"


def install(schema_editor, *, reverse):
    ...  # retention and existing lookups elided
    for replay in (False, True):
        statement = catalog.erase_sql(retention, existing, replay=replay)
        if statement.count(catalog.MODEL_PROVIDER_HANDOFF) != 1:
            raise RuntimeError("unexpected model handoff definition")
        if reverse:
            statement = statement.replace(catalog.MODEL_PROVIDER_HANDOFF, LEGACY, 1)
        schema_editor.execute(statement, params=None)


# forwards, backwards and the Migration class elided

Three things are worth pulling out of that file.

The loop runs over both function variants, the normal erasure and the restore replay. The two have to carry the same rule, because a restored database can bring back rows for a person erased after the backup, and the replay re-runs the erasure against those rows. What a restore leaves behind is covered in A Database Restore Does Not Undo Sent Email or Envelopes. If replay derived the entry differently, the replay would answer a different question than the erasure it replays.

The count check keeps the installed SQL and the catalog in agreement. The function body is generated from the catalog, so the migration asserts that the new predicate appears exactly once in the generated statement. A later edit that drops the clause or duplicates it fails the migration instead of shipping a function that quietly disagrees with the source it came from.

The reverse path swaps the new predicate back to the legacy one, which is what makes the migration reversible. The old rule is carried in the file as a constant, so reversing is a substitution instead of a guess. Ownership, grants, historical rows and signed erasure records survive the reinstall, so no signed record is rewritten.

Nothing else moved. Provider activation, rate limits and token policy are untouched.

The regression that reproduces both defects

tests/test_erasure_model_handoffs.py landed at 150 lines with four test functions in it. The first of them covers the asymmetry in a single body:

@pytest.mark.parametrize("replay", [False, True], ids=["erase", "replay"])
@pytest.mark.parametrize("called", [False, True], ids=["refused", "provider-called"])
def test_9kg_73_a_suspended_assessment_owes_no_provider_handoff_but_a_real_call_does(
    a, monkeypatch, settings, decided, replay, called
):
    if called:
        app, run, seen = assessed(a, monkeypatch, valid())
        assert run.status == "succeeded" and seen["calls"] == 1
        assert AssessmentInvocation.objects.filter(run=run).exists()
    else:
        app, run, seen = queued(a, monkeypatch, settings, valid())
        settings.CATALYST_ASSESSMENT_MODEL = "none"
        run_worker()
        run.refresh_from_db()
        assert run.status == "refused" and seen["calls"] == 0
        assert not AssessmentInvocation.objects.filter(run=run).exists()
    closer = grant(a, staff(a, "model-erasure-closer"), "close_application")
    closure.close_application(app.pk, closer, "closed", "Synthetic erasure regression")
    assert ("model_provider" in handoffs(a.committed, app, replay)) is called

The called parameter doubles as the expected answer, which is why the last line can assert is called instead of branching on it. The refused branch builds the case that used to produce the false positive. Setting the model to none is the suspension condition, so the worker runs, the run is refused and the fake provider records zero calls. The provider-called branch is the control, where a real call happened and the handoff is owed. The closure.close_application call closes the application through the staff closure helper before the erasure runs. The second function, test_9kg_73_reply_only_model_reservations_keep_the_provider_handoff, covers the false negative. It builds a reply interpretation that reserved provider work, in an application with no assessment run, and asserts across both erasure paths that the handoff appears when something was reserved and stays absent when the reservation count is zero.

The third test covers the ambiguous middle, the case the old predicate could not tell apart from a run that reserved nothing:

@pytest.mark.parametrize("status", ["started", "abandoned"])
def test_9kg_73_an_uncertain_reserved_call_still_owes_provider_cleanup(
    privileged_reset, decided, status
):
    app = subject()
    reserved_run(app, status)
    assert "model_provider" in handoffs(privileged_reset, app, False)

A call that was reserved and then started or abandoned still owes a model provider cleanup, because the provider's side of that call is not something our database can resolve.

The fourth function, test_9kg_73_both_functions_reverse_and_reapply_without_changing_rows_or_grants, reverses the migration and reapplies it, then checks that the rows, the function ownership and the grants survive the reinstall.

The helper three of the four tests call is what ties the two paths together. It runs the installed catalyst_erase function for the normal path and the installed catalyst_erase_replay function for the restore path, so whichever path is under test is held to the same rule.

Result

The eight-case regression, the module's first eight cases, four from the suspended-assessment matrix and four from the reply-only matrix, reproduced both defects before the change: 4 expected failures and 4 passing controls, in 30.32 seconds. Making the guard prove it can fail before trusting that it passed is the same rule as in Make the Guard Prove It Can Fail. After the fix, all 15 focused checks passed in 102.60 seconds. The coverage included actual worker and fake-provider cases, both erasure paths, uncertain calls, reversal and reapplication with exact row, function and grant preservation, the installed SQL checked against the catalog, schema inventory and model synchronization, and the existing complete erasure-reversal check.

Ruff, formatting, Markdown, document index and diff checks pass. Neither a live provider nor a production database took part in this change or in its tests, and nothing is deployed yet, so no vendor ever received one of these cleanup requests. Full CI was the merge gate, and the change is merged.

Deployment and the live handover to the new version remain separate open work. The internal launch plan and the approved migration set in tests/test_database.py were updated with the change, so the next person reads the rule instead of rediscovering which signal was wrong.

Also shipped

A sibling fix from the same day (PR #178) requires restore-recovery evidence to be timestamped at or after the current transaction's start, not just to carry a matching transaction ID, so a rebuilt cluster cannot reuse an old transaction ID as fresh authorization.
Restore replay now matches normalized inbound senders against the signed erasure record (PR #177), so restoring a backup cannot resurrect inbound mail and intake counters for a person erased after that backup was taken.

Use this

Derive any obligation to ask an external party to delete data from what was possibly sent, not from the existence of a row. A reservation you wrote before the call is the evidence that a request might have reached the vendor, and a refusal that reserved nothing is evidence that nothing did.
Give the delete path and the restore path the same predicate, then test both against the installed functions. A restore that replays an erasure with a different rule answers a different question than the erasure it replays.
Carry the old predicate in the migration as a constant, assert the new one appears exactly once in the generated SQL, and reverse by substitution. A signed erasure record is never rewritten, and a catalog edit cannot ship a function that disagrees with its own source.

The change is in catalyst-onboarding-v2 PR #179.

Frequently asked questions

What is the difference between a reserved provider call and a confirmed one?

A reservation is the record your system writes before it sends the request, so the request may have reached the provider. A confirmed disclosure is the record that requests were consumed. A call that was reserved and then started or abandoned still owes a cleanup, because the provider's side of that call is not something your database can resolve.

Why not count only confirmed provider calls?

Counting only confirmed calls removes the false positive and shortens the query, but it creates a silent false negative: an interrupted call leaves no confirmed record while the provider may have received it. Asking the provider instead makes the vendor a runtime dependency of an erasure request, and none of those assumptions hold in general. So the obligation is deliberately a superset of confirmed disclosures.

How do I test that an erasure derives the right cleanup set?

Parametrize the case over both erasure paths and over whether the provider was actually called, so the parameter doubles as the expected answer. Cover the false negative with a reply interpretation that reserved provider work, and the ambiguous middle with a call reserved and then started or abandoned. Assert against the installed functions, so the path under test and the restore path are held to the same rule.

{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is the difference between a reserved provider call and a confirmed one?",
"acceptedAnswer": {
"@type": "Answer",
"text": "A reservation is the record your system writes before it sends the request, so the request may have reached the provider. A confirmed disclosure is the record that requests were consumed. A call that was reserved and then started or abandoned still owes a cleanup, because the provider's side of that call is not something your database can resolve."
}
},
{
"@type": "Question",
"name": "Why not count only confirmed provider calls?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Counting only confirmed calls removes the false positive and shortens the query, but it creates a silent false negative: an interrupted call leaves no confirmed record while the provider may have received it. Asking the provider instead makes the vendor a runtime dependency of an erasure request, and none of those assumptions hold in general. So the obligation is deliberately a superset of confirmed disclosures."
}
},
{
"@type": "Question",
"name": "How do I test that an erasure derives the right cleanup set?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Parametrize the case over both erasure paths and over whether the provider was actually called, so the parameter doubles as the expected answer. Cover the false negative with a reply interpretation that reserved provider work, and the ambiguous middle with a call reserved and then started or abandoned. Assert against the installed functions, so the path under test and the restore path are held to the same rule."
}
}
]
}

A Database Restore Does Not Undo Sent Email or Envelopes

Jeremy Longshore — Tue, 03 Nov 2026 11:30:13 +0000

Restore a database to last night's backup and the rows go back in time. The email your app sent at 2 a.m. does not. Neither does the e-signature envelope it created, the mailbox password it rotated or the message it appended to a Sent folder. The restored app has no memory of any of them, so it can do them again: a second contract envelope to the same signer, a duplicate confirmation email to the same applicant.

This post covers a three-part pattern for that problem. First, a registry: every function that calls an outside service, tagged with how a restore recovers it. Second, a test that reads the source code and fails the build when that registry and the code disagree. Third, for an effect nothing reads back and the review does not close, a check after the restore that asks the outside service what it created since the backup.

The app here is an onboarding system. It sends mail over SMTP and through Django's mailer, creates and distributes agreements through Documenso (a self-hosted e-signature service), writes to a mailbox host, appends to IMAP folders and calls LLM agents. It also keeps an effect journal (an append-only log of outward actions, the side effects a rollback cannot reach, kept outside the database). After a restore, a post-restore review reads that journal and the providers, then decides whether to lift the restore hold (release the restored system for normal use).

What a restore forgets

The journal only helps if it covers every effect. Before this work, that coverage was a hand-written claim in a design document, written after a code review. Nothing checked it.

A hand-written list of covered effects drifts in one direction. Code adds a new provider call. Nobody updates the list. The list still reads as complete, and the next restore drill passes because the drill never exercises the new call. A Green Recovery Drill Can Still Be Lying covers other ways a restore check reports green.

That is the failure mode to design against. The restore itself is fine. The claim about what the restore can recover is the part that rots.

The registry: one recovery class per function and family

The fix starts with a data file, workflow/effects.py. It registers 40 functions, two of them markers rather than effects, grouped in seven families: journal writes, SMTP, Django mail, Documenso writes, mailbox host writes, IMAP appends and model runs.

Each function gets one class for each family it touches, plus a reason. Six classes make up the recovery vocabulary:

journaled_before: a journal entry is written and fsynced before the effect becomes a fact. Restrictive effects (such as a staff deactivation or an access revocation) are prepared this way. If that write fails, only the erasure and custody records refuse the effect; see below.
journaled_after: the entry is written on commit, once the effect is a fact. A crash can lose it, but it can never invent an effect.
provider_read: no entry. A named post-restore function reads the effect back at the provider.
checked_before: no entry. The action reads the provider before it writes, so a repeated run is idempotent: it finds the effect and skips it.
covered_by_review: nothing reads it back. The post-restore review closes what the effect handed out, such as every live confirmation, password-setup and withdrawal link.
marker: an entry about the journal itself, such as the restore point. Not an effect.

Three entries copied from the registry as the first PR shipped it show the shape. The gap on the first one comes back below.

"agreements.handlers:CreateAgreement.call": {"documenso": Site(PROVIDER_READ,
    "a restored instance's envelope is read by id, or found by its external key",
    by=READ_ENVELOPES, gap="9kg.76")},
"agreements.handlers:DistributeAgreement.call": {"documenso": Site(CHECKED_BEFORE,
    "re-reads the envelope and skips one already PENDING",
    by="agreements.handlers:DistributeAgreement.call")},
"agreements.keystore:destroy": {"journal": Site(JOURNALED_BEFORE,
    "custody key destruction, appended before the unlink",
    ("custody_key_destroyed",))},

The key is the module plus the function's qualified name. The value maps a family to a Site. The by field names the function responsible for recovery, so a reviewer can follow the claim to code instead of trusting a sentence.

Site is a frozen dataclass, copied from the registry with its comments:

@dataclass(frozen=True)
class Site:
    cls: str
    why: str
    kinds: tuple = ()  # journal kinds written (the journal family only)
    via: str = ""  # the registered function whose journal entry covers a provider or model call
    by: str = ""  # the function that reads the effect back, checks it first, or closes it in the review
    gap: str = ""  # the bead for a known uncovered part
    after: tuple = ()  # kinds of a journaled-before site written after the effect

The gap field matters more than it looks. It lets the registry admit an uncovered effect without lying about it, and it must carry a tracked-task ID. The after field lists kinds a journaled_before site writes once the effect is done, such as an outcome record, and the test checks it names some but not all of that site's kinds.

Why not a runtime decorator?

The obvious design is a decorator: mark each effectful function @effect("smtp") and check coverage at runtime. The PR chose call-site discovery instead, for two reasons.

First, discovery needs no change to application code. The registry is data and the test is static.

Second, a decorator only knows about functions somebody remembered to decorate. That is the same failure as the hand-written list, moved into the source. Discovery reads the code and finds the provider call whether or not anyone thought to mark it. A new send_mail with no registry entry fails the build.

The test: find every outward call with `ast`

tests/test_effect_registry.py parses every source file into an abstract syntax tree with Python's ast module and builds a map of {function: {family: calls}}. Its import resolver handles module imports, aliases, relative imports and by-name imports, because from django.core.mail import send_mail as sm is still a mail call.

It has nine test functions. The first three check the registry against the source, and the fourth checks that the scanner knows every provider write method:

test_9kg62_every_function_with_an_effect_is_registered_and_no_registration_is_stale(sites)
test_9kg62_each_class_matches_what_the_function_does(sites)
test_9kg62_every_journal_kind_has_a_writer_and_each_restrictive_kind_is_prepared(sites)
test_9kg62_the_scanner_knows_every_write_method_of_each_provider_client()

Of the other five, four test the scanner on synthetic source (import spellings, kind resolution, lookalike calls, an unreviewed outward module). One checks the real code against the outward-module allow-list.

Between them, the tests fail when:

a function with an effect is unregistered, an entry is stale, or the family is wrong;
the class does not match the calls the function makes;
a journaled_before entry is written after the provider call it precedes in the same function;
a checked_before action writes before it reads;
by or via is missing, or gap is not a tracked-task ID;
a journal kind does not resolve to a literal, a constant or a literal caller argument;
the provider client has a write method the scanner does not know;
any module outside a reviewed allow-list of twelve imports a network or process module, or writes, moves or removes a file.

The last two are the drift guards for the scanner itself. If someone adds a new write method to the Documenso client, the scanner has no rule for it, and the fourth test fails until a human teaches it. If a thirteenth module starts opening sockets or deleting files, the allow-list test fails.

The ordering rule is the least obvious one. "Journaled before" is only true if the journal write precedes the provider call in source order. Here is the idea, reduced to a hypothetical sketch (not the shipped code):

# Hypothetical reduction of the ordering check.
def first_call_line(fn, matches):
    lines = [n.lineno for n in ast.walk(fn)
             if isinstance(n, ast.Call) and matches(n)]
    return min(lines, default=None)

journal_at = first_call_line(fn, is_journal_write)
provider_at = first_call_line(fn, is_provider_call)
if journal_at is not None and provider_at is not None:
    assert journal_at < provider_at

This only decides order inside one function. Order across functions is not checked.

Verifying the AST test with hand mutations

A static test that passes on day one proves nothing about whether it can fail. The same rule for a CI gate is in Make the Guard Prove It Can Fail. So the verification was mutation by hand: break the code or the registry on purpose, one change at a time, and confirm the intended test catches it.

The registry test and the repository's test-selection test together ran 28 passed, 0 failed. Then 21 hand mutations, each caught by the test meant to catch it. The first round of twelve covered a dropped site, a stale entry, the wrong class, the wrong kind, a bad via, a missing via, a dropped family, a marker that was really a record, an undeclared kind, a restrictive kind never prepared, an empty reason and an injected unregistered send.

The second round of nine: a dropped Django mail site, a missing reader, an absent by, a non-task gap, a removed erasure completion caller, FileCopy writing before it reads, the send_begun journal write moved after SMTP, a dynamic journal kind and a removed staff-role prepare. Separately from the 21, an injected module with outward calls was also caught.

What writing the registry and its QA pass found

The first version of the registry was incomplete, and a separate QA agent pass (static reading only) showed it. That pass returned seven findings. Five were fixed in the PR. One was filed as a tracked task and one was partly fixed. The PR's own list of five findings, below, draws on both the writing and the QA pass and only partly overlaps the QA agent's seven. Each one says what happened to it.

Three applicant mails were missing (fixed). The contact confirmation, the password setup and the withdrawal link go through Django's mailer, not the app's correspondence transport. The first version of the registry missed all three. Django mail is its own family in the registry. The post-restore review covers them by superseding every live link they carry.

FileCopy was misclassified (fixed). It was registered as read back by the Sent-folder reader. That reader skips filed copies. What FileCopy actually does is search the Sent folder before it files, so it is now checked_before, with an order check behind it.

The revocation's host calls are outside the journal (registered as covered_by_review). Password rotation and forwarder removal run before the transaction that prepares the revocation entry. The review's revocation round covers them, so they are registered covered_by_review.

A failed restrictive prepare does not stop its effect (recorded, not changed). The failure is logged, counted and refused at readiness. Only the erasure and custody records refuse their effect outright. This was recorded, not changed.

A second envelope after a restore had no coverage at all (filed, then closed by a follow-up PR). An agreement created and distributed after the backup has no row once you restore. Nothing read its envelope. The external key is random, so the restored system would create a new agreement and distribute a second envelope to the same signer. The registry could only mark this as a gap, because a runtime fix did not belong in a test-only PR.

That last one was the main finding. Writing and reviewing the registry forced every function through the question "who reads this back?", and for envelopes created after the backup the honest answer was "nobody". The test could not fail on this. It could only carry the gap.

Closing the gap with a bounded Documenso envelope listing

The follow-up PR added an unknown-envelope scan to the post-restore Documenso read, agreements.restore.read_envelopes.

It lists the Documenso team's envelopes from every template, newest first, back to one hour before the backup's start, so a template the restored tables do not know is covered too. Any envelope with the app's key shape (ag_ plus 32 hex characters) that no restored agreement or envelope attempt names, by envelope id or external key, opens one review item. The item is keyed by a digest of the envelope id, and the raw id is stored in neither the item nor its audit event. It carries the application only when exactly one verified applicant is a signer.

The listing is bounded by a configured page limit (the PR's risk note puts it at 20 pages of 100). If it hits that limit before reaching the backup's start, it reports incomplete:listing_limit, and any status other than ok refuses to lift the restore hold. A truncated listing is treated as incomplete, never as clean. The registry entry's gap was then removed, as the task's acceptance required. That PR's targeted run was 82 passed. The gap is closed in code, pending the provider check described below.

Why not journal every create instead?

The alternative was a journal entry per envelope create. The listing won for two reasons. It already existed for reconciliation, so no new write path. And it finds a create whose journal append was lost, which a journal entry cannot do by definition.

Envelopes without the app's key shape belong to another writer on the same Documenso team and are ignored. The cost of that team-wide listing is volume: another writer's envelopes count toward the page limit. Hitting it fails closed until the system's operator raises the limit or resolves the items by hand.

Known blind spots

The test's docstring lists its own blind spots:

receivers held in attributes or reached through getattr;
a database write that should be journaled and is not;
whether a named reader actually reads what the entry says.

The ordering check also works only inside one function, as noted above.

Documenso's real behaviour is also unverified. Its list ordering, default status filter and field names are not confirmed against the installed version, so the scan checks order itself and never trusts a page count. A canary against the real provider is where that gets settled. Resolving an unknown envelope is currently by the operator's note alone, and whether a note is enough is the operator's call.

The test is brittle by design. Rename an effectful function, add a provider write method or add a file-writing module and the build fails until someone reviews the change. That is intended: a failing build is the moment the hand-written list used to go stale silently.

Also shipped

A journal-ship service copies the effect journal off the production host, so the journal survives the host it describes. It runs with network_mode: none because it only reads and writes files (PR #123, 239 passed in its targeted run).

Use this

List every function in your app that calls a provider (mail, e-signature, payments, mailbox APIs) and give it a recovery class for each provider it touches, plus the name of the function that recovers it.
Write a static test that discovers those calls from source, including aliased and relative imports, and fails on any call it can resolve that the registry has not classified; then mutate it by hand until each failure mode has been caught once.
For an effect that nothing reads back and the review does not close, add a post-restore provider listing with a time floor just before the backup's start and a page limit, make a truncated listing keep the restore hold in place, and confirm the provider's list behaviour against the real service.

The registry and its AST test are in catalyst-onboarding-v2 PR #117.

Frequently asked questions

What is an effect journal?

An effect journal is an append-only log of outward actions, kept outside the database. After a restore, a post-restore review reads that journal and the providers, then decides whether to lift the restore hold.

How do you stop a restored app from repeating an email or envelope?

Give every function that calls an outside service one recovery class per provider family in a registry, and name the function that recovers it. A checked_before action reads the provider before it writes, so a repeated run finds the effect and skips it. Where nothing reads an effect back and the review does not close it, a bounded provider listing looks for envelopes the restored database does not name.

How can a test catch an effect nobody registered?

An AST test parses every source file, follows aliased and relative imports, and fails the build when a function with an effect is unregistered, stale or in the wrong class. It was checked with 21 hand mutations, each caught by the test meant to catch it. It still cannot see receivers held in attributes or reached through getattr.

{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is an effect journal?",
"acceptedAnswer": {
"@type": "Answer",
"text": "An effect journal is an append-only log of outward actions, kept outside the database. After a restore, a post-restore review reads that journal and the providers, then decides whether to lift the restore hold."
}
},
{
"@type": "Question",
"name": "How do you stop a restored app from repeating an email or envelope?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Give every function that calls an outside service one recovery class per provider family in a registry, and name the function that recovers it. A checked_before action reads the provider before it writes, so a repeated run finds the effect and skips it. Where nothing reads an effect back and the review does not close it, a bounded provider listing looks for envelopes the restored database does not name."
}
},
{
"@type": "Question",
"name": "How can a test catch an effect nobody registered?",
"acceptedAnswer": {
"@type": "Answer",
"text": "An AST test parses every source file, follows aliased and relative imports, and fails the build when a function with an effect is unregistered, stale or in the wrong class. It was checked with 21 hand mutations, each caught by the test meant to catch it. It still cannot see receivers held in attributes or reached through getattr."
}
}
]
}

SSRF Egress Proxy: Deny Your Own Public Production Hosts

Jeremy Longshore — Tue, 03 Nov 2026 11:30:10 +0000

An SSRF filter (a guard against server-side request forgery, which stops a server fetching URLs an attacker chose) that admits only public addresses still lets your server-side fetcher reach your own production hosts, because those hosts have public addresses. This write-up ends with a deny check for an egress proxy that refuses those hosts by name, by every address they resolve to, and again by the address the socket actually connected to, without letting a DNS hiccup lock all egress shut.

The case is an onboarding app that fetches links applicants submit as evidence. All of that traffic leaves through an egress proxy (the one process allowed to open outbound connections). The proxy already refused private, loopback, link-local and cloud metadata addresses (the endpoint that hands out instance credentials). Anything with an ordinary public address (global unicast, in IP terms) went through.

That rule keeps a fetcher off your internal network. It says nothing about the hosts you run on the public internet. An applicant's link, or a redirect from a harmless-looking page, could aim the fetcher straight at them.

Real evidence fetching is still switched off in production. This check had to land before it could be switched on. How the fetcher role is split out of the web container is covered in Docker Container Security Hardening: What Unit Tests Miss.

The naive fixes are worse than the gap. Remember every address a production name ever resolved to, and the set grows until the guard refuses everything. Refuse whenever a lookup fails, and one flaky nameserver stops all fetching. The first version of this change did both, and a review agent caught it before merge.

Can a public-address SSRF filter still reach your own production hosts?

Yes. An SSRF filter that admits only public addresses still lets your fetcher reach your own production hosts, because those hosts have public addresses. A public-address classifier cannot say "public, but ours", so the fix is a second list of host names. The proxy checks that list by name, by every resolved address and by the connected peer.

Why the public-address rule cannot express "mine"

A global unicast classifier answers one question: is this address on the public internet? Your production hosts answer yes. The classifier has no way to say "public, but ours", so a second list is needed.

That list has to name your hosts specifically, and it has to hold up when DNS moves underneath it. The same mismatch, a check that validated a hostname while the fetch routed by address, came up earlier in the custom-feed field of an RSS bar plugin for Omarchy (an opinionated Arch Linux desktop setup by DHH). See Every Verdict Carries the Scope It Actually Ran.

What the deny list file must contain

The deny list is a secret-free file of logical host names, pointed to by an environment variable. Format rules:

ASCII only, at most 64 KiB and 256 names.
One dotted DNS name per line, with # comments.
Each entry denies itself and every name under it.
An IP address, a single label, or a label written as a 0x hex number (such as 0x7f) makes the file malformed.

What three checks does the egress proxy run on each request?

The proxy checks every request three times against that list:

By name, before any lookup. The requested host is compared with the deny names. A match is refused with host_denied.
By resolved address. Every address the target resolves to is compared with every address the deny names resolve to. A match is refused with address_denied. This catches a different name pointing at the same machine.
By connected peer. After the socket connects, the peer address is classified again and compared with a fresh resolution of the deny names. A match is refused with peer_denied. This catches a deny name whose DNS answer moves onto the target's address between the check and the connection. The target itself is not resolved again: the proxy connects to the addresses it already checked.

Here is the request path from assessments/egress.py (the name check happens inside policy.target()), cut off once the tunnel is accepted:

def handle(client, policy: Policy) -> str:
    upstream = None
    try:
        host, port, addresses, extra = policy.target(read_head(client))
        if not addresses or not all(classify(ip, extra) for ip in addresses):
            raise Refusal(403, "address_not_allowed")
        denied = policy.denied_addresses()
        if any(canonical(ip) in denied for ip in addresses):
            raise Refusal(403, "address_denied")
        for address in addresses:  # every answer already passed classify; try each until one connects
            try:
                upstream = socket.create_connection((address, port), timeout=CONNECT_SECONDS)
                break
            except OSError:
                continue
        if upstream is None:
            raise Refusal(502, "connect_failed")
        peer = upstream.getpeername()[0]
        if not classify(peer, extra):
            raise Refusal(403, "peer_not_allowed")
        if canonical(peer) in policy.denied_addresses():  # resolved afresh: a DNS change since the check is caught here
            raise Refusal(403, "peer_denied")
        reply(client, 200, "ok", peer)
        # ... tunnel splice, except and finally omitted

Two details matter. The address and peer checks run after the public-address check, so they only ever narrow what was already allowed. And denied_addresses() runs twice per connection, one lookup pass of the deny names each time. Each pass runs in one thread and is bounded on its own by the existing 10-second deadline.

One spelling per address

Address comparison by string fails quietly. ::ffff:203.0.113.10 (an IPv4-mapped IPv6 address, defined in RFC 4291 section 2.5.5.2), ::ffff:cb00:710a and 203.0.113.10 name one IPv4 host. A link-local IPv6 address can also carry a zone suffix such as %eth0. If the deny set holds one spelling and the target arrives in another, the set membership test misses.

def canonical(ip: str) -> str:
    """One spelling per address: the zone dropped and an IPv4-mapped IPv6 address unwrapped, so `::ffff:a.b.c.d`, its
    hex form and `a.b.c.d` compare equal. Anything that is not an address is ""."""
    try:
        addr = ipaddress.ip_address(str(ip).split("%", 1)[0])
    except ValueError:
        return ""
    if addr.version == 6 and addr.ipv4_mapped is not None:
        addr = addr.ipv4_mapped
    return str(addr)

Both sides of every comparison go through this function. The tests cover a production address arriving as IPv4, IPv6, ::ffff:a.b.c.d and ::ffff:hhhh:hhhh.

Why not list the production addresses instead?

A file of IP addresses looks simpler. It breaks the first time a host moves, and it silently stops protecting the new address. Logical names survive a move, and the file stays free of anything sensitive.

Resolving at check time handles DNS changes in both directions. A name that moves to a new address is caught by the fresh answer. An address the name pointed at recently stays denied from memory until newer addresses push it out of a capped history. A change between the check and the connect is caught at the peer.

The other obvious choice was to exit at startup when the file is broken. That was rejected too. Exiting leaves the fetcher with nothing to act on. A live proxy answering 503 deny_list_unavailable tells the fetcher to retry the item instead of closing it, and the failing healthcheck marks the container unhealthy where an operator can see it.

The first version could shut all egress

The implementation ran from a Claude Opus 5.5 session coordinating specialist agents. Its first version kept every address a denied name had ever resolved to. The commit message called it a never-shrinking union, chosen deliberately to cover DNS rebinding in both directions.

def denied_addresses(self) -> frozenset:
        answers = resolve_names(self.deny)
        if any(addresses is None for addresses in answers.values()):
            raise Refusal(502, "deny_list_unresolved")
        fresh = {canonical(ip) for addresses in answers.values() for ip in addresses} - {""}
        with self.lock:
            if len(self.denied_seen | fresh) > DENY_SEEN_LIMIT:
                raise Refusal(502, "deny_list_unresolved")
            self.denied_seen |= fresh
            return frozenset(self.denied_seen)

Read it as an operator would. Two lines can stop all egress. The lookup-failure one recovers only once DNS answers again; the size limit never recovers without a restart.

A read-only security and privacy review agent, dispatched on the open pull request, returned these as its first three findings, each rated medium:

The latch. Once the remembered set would pass DENY_SEEN_LIMIT (4096), every request returns 502 until a restart. A deny name served from a rotating or geo-DNS pool keeps growing the set. The result stops all fetching, well beyond the one host it meant to block. The review also noted each CONNECT started one thread per name, twice, so an operator error in the list could cause thread storms.
One flaky name blocks everything. Any SERVFAIL, timeout or EAI_AGAIN on any deny name made every fetch return 502. A decommissioned name whose nameserver stopped answering, or a resolver outage, becomes a lasting denial of service. The review noted it could never become a silent allow, which is the correct direction; the price was all fetching.
A non-resolving name denies nothing, silently. NXDOMAIN and NODATA become an empty answer. A typo in the list, or a split-horizon name the egress container cannot resolve, gives zero protection and raises nothing.

Fail-closed and stay-available pull in opposite directions. The first draft chose closed at every fork, including the forks where closed meant closing everything. Each review finding got its own fix-up commit before merge, and each code finding its own test. The third is only partly closed, as the next section says.

The bounded version: per-name memory with last-good answers

def denied_addresses(self) -> frozenset:
        answers = resolve_names(self.deny)
        with self.lock:
            failed = [name for name, addresses in answers.items() if addresses is None]
            if any(name not in self.remembered for name in failed):
                raise Refusal(502, "deny_list_unresolved")
            for name, addresses in answers.items():
                if addresses is not None:
                    self.remember(name, addresses)
            if failed:
                logger.info("egress deny_lookup_stale")
            denied = frozenset(ip for kept in self.remembered.values() for ip in kept)
        if not denied:  # a list in which no name resolves denies nothing at all: refuse rather than pretend
            raise Refusal(503, "deny_list_resolves_nothing")
        return denied

What changed, finding by finding:

Memory is bounded per name, with no global limit. remember() keeps a name's fresh answer in full, plus earlier addresses up to 64 in all (a fresh answer larger than 64 is kept whole). The oldest address missing from the fresh answer is evicted first. The file is capped at 256 names on load. No global limit remains that could latch the proxy shut.
A failed lookup keeps that name's last good answer. The fallback is logged as one outcome code, deny_lookup_stale, never the name. Resolution now runs as one lookup pass in a single thread.
Refusal is reserved for unknowns. A name whose lookup has never succeeded (SERVFAIL or a timeout, not a clean "no such name") still refuses with 502 deny_list_unresolved, which the fetcher retries. If that name's lookups keep failing, every request keeps refusing until one succeeds; the refusal is retryable, and the startup count shows the gap. If no name in the list resolves at all, every request gets 503 deny_list_resolves_nothing.
Startup reports the gap. A pass at start seeds each name's last good answer and logs deny_names_unresolved=N, a count without names.

One name that does not exist, inside a list where others do resolve, still denies nothing. The startup count is the only signal for that case, so it is worth alerting on.

When the deny file is missing or malformed

A missing, unreadable, oversized, empty or malformed file sends every request a 503 deny_list_unavailable. The fetcher treats that as transient and retries. The code is logged once at start, and the healthcheck (python -m assessments.egress --check) fails while the list does not load.

Staging bind-mounts its deny file instead of baking it into the image. Production is meant to mount the operator's file the same way, so staging exercises that path. The real production list is still a pending host-preparation item.

A separate test asserts that egress refusal codes never reach the model request the clarification path starts from, any mail, or a non-staff template.

Limits that remain

Peer refusal comes after the handshake. A peer_denied connection completes its TCP handshake with the denied host before it is closed. Zero bytes are sent.
The resolver is trusted. A resolver that answered the deny names falsely would hide their addresses. Network isolation, the host allow-list and TLS hostname verification remain the controls for that.
Changes need a restart. The file is read at start. A replaced bind-mounted file keeps a stale inode, so mount the directory or restart.
Hex labels cost a real name. A host such as 0xab.example.org cannot be listed. Listing it makes the whole file malformed, so every request gets 503 deny_list_unavailable.

Verification

The change added 25 tests in tests/test_egress.py. They cover a name and a name under it refused before lookup, the four address spellings, a DNS change between requests and between check and connect, per-name bounds without a latch, the last good answer, the never-resolved name, one thread per pass, the all-unresolved list with its startup count, and missing, empty, comment-only, malformed, non-ASCII and oversized files.

The full suite on the final head passed 1744 tests in 50 min 21 s. The run before the review, which already held the first version's tests, passed 1733.

The staging check runs against the compose stack on the dev box. A public stand-in plays the production host. It was refused by name, a name under it was refused, and 1.1.1.1.nip.io and 1.0.0.1.nip.io (names that resolve to the address written in them) were refused by resolved address. A control request to an ordinary public name, which the staging stack allows, tunnelled through.

Staging also caught one more wobble. The first run after the review scored 132 PASS, 0 FAIL, 1 NOT RUN: a by-address case missed a single transient lookup. The probe now retries before reporting; it already printed NOT RUN rather than FAIL when the proxy cannot reach public DNS. The final run on a clean tree scored 133 PASS, 0 FAIL, 0 NOT RUN, with the egress healthcheck passing compose up --wait.

The merged change, one commit per review finding, is at https://github.com/intent-solutions-io/catalyst-onboarding-v2/pull/101.

Frequently asked questions

How does an SSRF deny list avoid blocking all egress when DNS fails?

A failed lookup keeps the last good answer for that name, and the proxy logs only the outcome code deny_lookup_stale, never the name. A name whose lookup has never succeeded (SERVFAIL or a timeout, not a clean "no such name") refuses with 502 deny_list_unresolved, which the fetcher retries. If no name in the list resolves at all, every request gets 503 deny_list_resolves_nothing. A name that does not exist, inside a list where others resolve, still denies nothing, so the startup count is the signal to alert on.

Why does a deny list miss an IPv4-mapped IPv6 address?

One host can carry several spellings: ::ffff:203.0.113.10, ::ffff:cb00:710a and 203.0.113.10 all name the same IPv4 host. A string comparison against a single spelling misses the others. The proxy unwraps IPv4-mapped IPv6 and drops any zone suffix such as %eth0 on both sides of every comparison, and the tests cover the IPv4, IPv6, ::ffff:a.b.c.d and ::ffff:hhhh:hhhh forms.

Can a DNS change between the check and the connection get past an SSRF deny list?

Not the target's own DNS: the proxy connects to the target addresses it already checked, so the target name is not resolved again. If a deny name's DNS answer moves onto that address in the meantime, the peer check catches it as long as that fresh lookup succeeds (a failed lookup falls back to the last good answer): it compares the peer with a fresh resolution of the deny names and refuses with peer_denied before any bytes are sent. The TCP handshake with the denied host still completes first.

{"@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [{"@type": "Question", "name": "How does an SSRF deny list avoid blocking all egress when DNS fails?", "acceptedAnswer": {"@type": "Answer", "text": "A failed lookup keeps the last good answer for that name, and the proxy logs only the outcome code deny_lookup_stale, never the name. A name whose lookup has never succeeded (SERVFAIL or a timeout, not a clean \"no such name\") refuses with 502 deny_list_unresolved, which the fetcher retries. If no name in the list resolves at all, every request gets 503 deny_list_resolves_nothing. A name that does not exist, inside a list where others resolve, still denies nothing, so the startup count is the signal to alert on."}}, {"@type": "Question", "name": "Why does a deny list miss an IPv4-mapped IPv6 address?", "acceptedAnswer": {"@type": "Answer", "text": "One host can carry several spellings: ::ffff:203.0.113.10, ::ffff:cb00:710a and 203.0.113.10 all name the same IPv4 host. A string comparison against a single spelling misses the others. The proxy unwraps IPv4-mapped IPv6 and drops any zone suffix such as %eth0 on both sides of every comparison, and the tests cover the IPv4, IPv6, ::ffff:a.b.c.d and ::ffff:hhhh:hhhh forms."}}, {"@type": "Question", "name": "Can a DNS change between the check and the connection get past an SSRF deny list?", "acceptedAnswer": {"@type": "Answer", "text": "Not the target's own DNS: the proxy connects to the target addresses it already checked, so the target name is not resolved again. If a deny name's DNS answer moves onto that address in the meantime, the peer check catches it as long as that fresh lookup succeeds (a failed lookup falls back to the last good answer): it compares the peer with a fresh resolution of the deny names and refuses with peer_denied before any bytes are sent. The TCP handshake with the denied host still completes first."}}]}

Also shipped

Same app: an authorized erasure path with an audited tombstone exception and a durable record kept outside the backup.
bobs-big-brain-plugin: installer cleanup confined to validated targets, with recoverable data preserved.
claude-code-plugins: five unsupported SaaS tutorial packs retired, and the security plugin's HTTP method probes constrained to owned fixtures.
diagnostic-pro: homepage contrast and keyboard access fixed.
intent-whop-products: new repo scaffolded with CI and releases on a self-hosted runner.

Use this

List your production hosts by name, and compare addresses in one form. Resolve the names at check time, unwrap IPv4-mapped IPv6 and drop the zone on both sides, and refuse subdomains of each entry too.
Check the connected peer, after the resolved answer. Run getpeername() through the same classifier and a fresh deny resolution, so a deny name that moves onto the target's address between check and connect is refused before a byte is sent.
Bound what you remember, per name. Keep the fresh answer plus a capped history, fall back to the last good answer on a failed lookup, and refuse only for names whose lookup never succeeded, a list that resolves to nothing, or a file that does not load, each with a retryable code.

Docker Container Security Hardening: What Unit Tests Miss: the same app's staging deploy, five hardening controls that passed their own tests, and the review question behind each.
Every Verdict Carries the Scope It Actually Ran: every verdict covers only what it ran, including a request-forgery class closed by deleting the feature.
Rate Limit a Django Signup Form Without Refusing Anyone: the same app again, deferring mail instead of refusing applicants, a fail-safe that does not punish users.

Docker Container Security Hardening: What Unit Tests Miss

Jeremy Longshore — Fri, 23 Oct 2026 11:30:10 +0000

A unit test exercises the function. It does not exercise the deployment.

That gap is where a staging container ships with security controls that pass everything they are handed and are still wrong in the image that ends up running. Across repeated review rounds on one staging deployment I found five of them. By the end of this you can name each one and the single review question that catches it before it reaches a container.

Why do container security tests pass while the container is still wrong?

A unit test proves the function behaves as written. It cannot prove the deployment has the property you wanted, because that property lives in a layer the code does not own. Each of the five findings below sits in a different layer, and each one is caught by a deployment step or a review question rather than by the test that went green.

Control	What the test suite proved	What the running container did
Host guard	The hosts on the list were refused	Any host nobody had imagined got a 200
Health command	Exit 0 when there were no findings	Exit 1 for a real anomaly and exit 1 when the check itself broke
Gunicorn socket umask	The string `117` was present in the Dockerfile	The socket landed at 0612 instead of 0660
Secrets	The settings module read each value	Every value was readable in `/proc/<pid>/environ` and `docker exec env`
Process roles	The role variable was set	A web process came up holding an empty signing key

What the green test suites could not see in the container

The health, startup, guard and web security suites passed, 86 of them. So did the confirmation, staff, provisioning, agreement and intake suites, 227 of them. Later, the image, startup, worker and guard suites passed, 134 of them. Every control described here shipped with a green run behind it.

Not one of those suites was wrong. Each asked a precise question about code and got a precise answer. The questions just stopped one layer short of the process.

A guard is a function when the test runs it. It is a deployment when a hostname arrives from the network, when the kernel applies a umask to a socket inode, when someone types docker exec, when a supervisor starts a second process in the same container. Those are four different things, and only the first one is unit-testable.

Finding 1: the host guard that failed open

The first version of the host guard denied exactly one host, the public one. Everything else passed.

# what shipped first
def host_allowed(host):
    return host != PUBLIC_HOST          # every other host passes

The tests passed because they tested the hosts the authors already knew about. A deny-list is only as long as the list of threats somebody imagined. Any hostname nobody thought of gets a 200.

The replacement names the hosts that are allowed, and normalizes the host before it decides. It now takes the path as well, because a host check with no path cannot tell a health probe from a staff screen:

def host_allowed(host, path):
    host = normalize(host)
    if is_health_path(path):
        return host in LOCAL_HOSTS
    if is_staff_screen(path) or is_admin(path):
        return host == STAFF_HOST
    return False                        # 404 otherwise

A separate guard refuses a missing staff host, and refuses a wildcard, a domain pattern or the public host standing in as the staff or local host. It refuses configuration that is easy to write and means nothing.

The branch was replaced rather than patched. Adding the hosts somebody remembered to add would have produced a longer deny-list with the same failure direction.

Review question: what does this serve when the host is one I have never heard of?

Finding 2: exit 1 that meant two different things

The health command prints JSON with counts and signal names, never a version, a commit or a reason. It exited 0 for clean, 1 for anomaly. It also exited 1 when the check itself blew up.

So the two events that most need different reactions, page now and ignore because the tooling is broken, arrived through the same door.

# before: a failed check and a real anomaly share an exit code
findings = collect()
return 1 if findings else 0

# after: the command wraps every system check, including the ones it skips
try:
    findings = collect_all_signals()          # handled AND skipped
except CheckFailed:
    return 2                                 # the check failed
return 1 if findings else 0

Exit 2 is the answer that buys you the rest of the night. A monitor can treat 0 as healthy, 1 as an anomaly worth a page, and 2 as the instrumentation being down, which is a different alert to a different person.

The same review round found the signal set was counting things that are not anomalies. Provisioning health was counting open items, and open items include a person doing their job: a checklist waiting on a name review. The fix split them in two.

Problem reasons, alert immediately: conflict, failure, found or late mailbox, setup, activation, revocation, withdrawal link.
Routine waiting, alert only past the backlog target: checklist, name review, blocked.

A check that pages on ordinary work does not get fixed. It gets muted, and then it also stops carrying the real signal.

Review question: what does this exit code mean when the check itself is broken, and who is on the other end of it?

Finding 3: the umask that was a decimal number

The socket was created with the wrong permissions. The number in the flag looked correct.

--umask 117

Socket permissions are 0777 minus the umask. Written the intended way for the gunicorn --umask flag, 0o117, the socket lands at 0660: group read and write, nothing for other. Written bare, 117 is parsed as decimal 117, which is octal 0165, which leaves the socket at 0612. Owner read and write, group execute only, other write only.

# 0660 socket. a bare 117 is decimal, which leaves 0612 instead.
# The comment has to sit above CMD: inside the continuation chain a # is a parameter.
CMD ["gunicorn", \
     "--umask", "0o117", \
     "--worker-tmp-dir", "/dev/shm", \
     "--error-logfile", "-", \
     "--access-logfile", "", \
     "config.wsgi:application"]

No unit test touches this. The test suite can assert the string in the Dockerfile. It cannot assert the mode on an inode, because no test creates the socket. The verification for this one is a single ls -l against the running container: 660 catalyst:catalyst.

The worker temp directory is there for the same family of reason. The root filesystem runs read only, so the temp directory gunicorn wants has to be a tmpfs mount.

Review question: what mode does this actually produce on the inode, and did I write that number in the base it is read in?

Finding 4: secrets in the environment

The obvious way to get a secret into a Django settings module is to export it as an environment variable. The entrypoint reads an env file and exports the values, the settings module reads os.environ, and the container runs.

Every value that lands in the environment of a process stays readable. /proc/<pid>/environ is readable by anything with the right ptrace relationship to that process. docker exec <container> env prints the container config's environment outright. The export does not stop at the process boundary the way it does in a shell you close.

The settings read each secret from its own file instead:

SECRETS_DIR = os.environ["CATALYST_SECRETS_DIR"]     # the one variable, a path

def secret(name):
    with open(os.path.join(SECRETS_DIR, name)) as fh:
        return fh.read().rstrip("\n")                 # drop the line ending only

def secret_problems():
    if any(name in os.environ for name in SECRET_NAMES):
        return ["a secret is present in the environment"]
    if not os.path.isdir(SECRETS_DIR):
        return ["the secrets directory is missing"]
    return []

Settings and the guard share one list of secret names, so a secret added to one is checked in the other. A file that cannot be read fails closed without printing its bytes, because a startup error that quotes the value is a startup error that copies the value into your logs. .dockerignore excludes secret-shaped files at any depth, and a deployment carrying mounted secrets may not set CATALYST_ENV=development, where the strictness is switched off.

The verification is not a test result. It is starting the production image and reading docker inspect, then reading the environment again from inside, and confirming nothing is there.

Review question: if this container died and somebody could run exactly one command, what would they read?

Finding 5: the fetcher in the web container

The image ran one application process with a role variable, and a second process for a background job that fetches evidence. The security re-check found the environment-only role let a web process come up holding an empty signing key.

An empty key is not a deny. It is a key. Anything signed with it is signed with the same thing anybody else has, so the signed verification links the product hands out could be forged by anyone who read the code.

def process_role_problems(env, role):
    if role not in {"web", "fetcher"}:
        return ["unknown CATALYST_PROCESS_ROLE"]
    if role == "web" and env.get("CATALYST_VERIFICATION_KEY", "") == "":
        return ["the web process is carrying the fetcher signing key"]
    return []

Three rules, and each one closes a hole the others leave open: an unknown role is refused, so a typo does not silently mean none; the fetcher role is refused inside the web process, declared in config.wsgi before Django loads, so nothing can import around it; and an empty key is refused everywhere except the fetcher. The fetcher starts without the key and receives it only where it belongs.

Review question: which process is supposed to hold this key, and what happens if it is present but empty?

Why not just write more tests

Every finding here got a test. Four of the five got one in the same commit as the fix. That is the right move and it is also the wrong answer to the question I was actually asking.

Two of these cannot be pinned by a test at all. The umask is an inode mode produced by a syscall, so the test can only ever assert that a string is present in a Dockerfile, which is what the original string already satisfied. The environ claim is a property of the container runtime, so proving it means running the container, which means the check is now a deployment test, which is a different tool with a different cost and a different failure story.

The third is subtler. A deny-list cannot be tested for fail-open behavior without enumerating every hostname the authors failed to think of, which is the enumeration that was wrong in the first place. The test proves the list is the list. It cannot prove the list is complete.

That is the general shape. A test tells you the code does what the code does. It cannot tell you the deployment has the property you wanted, because that property lives in a layer the code does not own.

So the fixes leaned the other way where they could. The allow-list changes the failure direction instead of documenting it. The exit code makes the two cases addressable. The startup guard refuses an unset, wildcard or public staff host before the app serves anything, which is stronger than a check that catches it here. A startup refusal is the artifact that helps the next person; a green run is only evidence that it happened to work on the machine it ran on.

How the running container was verified

The image, startup, worker and guard suites passed 134 of them, and the guard and startup pair passed 93 on their own after the role split. Then the production image ran with --read-only, tmpfs mounts, cap-drop ALL and secrets mounted read only. The socket came up 660 catalyst:catalyst. /healthz answered 200 over that socket, which is the part that matters, since a healthy HTTP check over TCP says nothing about a unix socket with the wrong mode. No secret appeared in the environment or in docker inspect. pip-audit found no known vulnerability in the base requirements.

Two of those lines are worth keeping as the shape of the whole exercise. /healthz returning 200 proves the application works. It does not prove the socket is only writable by the two users who should write to it. Both were true, at the same time, and only one of them was visible to the test suite.

The deployment work is public at intentsolutions-io/catalyst-onboarding-v2, with the guards and the health command in config/guards.py, config/system_health.py and config/health.py.

Questions this answers

How do I keep secrets out of a Docker container's environment?

The obvious route is to export the secret into the environment, where it stays readable by anything with the right ptrace relationship to the process and by docker exec <container> env. Read each secret from its own file instead and export only the path to the directory that holds them, so the settings module opens files by name and nothing secret-shaped is ever in the environment. Verification is not a test result. Start the production image, read docker inspect, then read the environment from inside the container and confirm nothing is there.

Why does gunicorn create a socket with the wrong permissions?

The socket mode is 0777 minus whatever umask gunicorn is given, so the flag is worth reading twice. Gunicorn reads it as octal, and a bare 117 arrives as decimal, which is octal 0165. That lands the socket at 0612 with write access for other, instead of the 0660 you intended. The suite cannot see it either way, because the string was already sitting in the Dockerfile when the tests went green. The check is a single ls -l against the running container.

Why do my security tests pass while the container is still insecure?

A unit test exercises the function, not the deployment. The properties that matter live in a layer the code does not own, and that layer holds a hostname arriving off the network, a kernel umask applied to a socket inode, the container runtime's environment, and a second process sharing the container. The suites were not wrong. Each one stopped a layer short of the process, so these get caught by a deployment check and a review question rather than by a unit test.

Use this

Ask what the guard does with the input nobody thought of. If the answer is a list of known bad names, the failure direction is open and the tests cannot see it, because they share the list. Name the allowed hosts instead, and refuse the wildcard.
Split the exit codes so the broken instrument has its own number. One code for an anomaly and one for a failing check is one code that trains on-call to ignore the first. Then separate the alerts that mean something is wrong from the alerts that mean a person is working.
Verify the container, not the module. Read the socket's real mode with ls -l, read the environment with docker exec, and prove secrets are absent from docker inspect. For the signing key, name the one process allowed to hold it, and confirm an empty value fails at startup.

Wrong-Mode Green Is Not a Gate: a checker that ran in the wrong mode and still printed all-green, the same shape as a guard that lists the wrong hosts.
Every Verdict Carries the Scope It Actually Ran: what a green result proves, which is the scope it ran and nothing past it.
A bash redirect that hides stderr but not the exit code: a deploy killed by an exit code nobody was reading, the plumbing cousin of the health command's exit codes. { "@context": "<a href="https://schema.org">https://schema.org</a>", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "How do I keep secrets out of a Docker container's environment?", "acceptedAnswer": { "@type": "Answer", "text": "The obvious route is to export the secret into the environment, where it stays readable by anything with the right ptrace relationship to the process and by docker exec env. Read each secret from its own file instead and export only the path to the directory that holds them, so the settings module opens files by name and nothing secret-shaped is ever in the environment. Verification is not a test result. Start the production image, read docker inspect, then read the environment from inside the container and confirm nothing is there." } }, { "@type": "Question", "name": "Why does gunicorn create a socket with the wrong permissions?", "acceptedAnswer": { "@type": "Answer", "text": "The socket mode is 0777 minus whatever umask gunicorn is given, so the flag is worth reading twice. Gunicorn reads it as octal, and a bare 117 arrives as decimal, which is octal 0165. That lands the socket at 0612 with write access for other, instead of the 0660 you intended. The suite cannot see it either way, because the string was already sitting in the Dockerfile when the tests went green. The check is a single ls -l against the running container." } }, { "@type": "Question", "name": "Why do my security tests pass while the container is still insecure?", "acceptedAnswer": { "@type": "Answer", "text": "A unit test exercises the function, not the deployment. The properties that matter live in a layer the code does not own, and that layer holds a hostname arriving off the network, a kernel umask applied to a socket inode, the container runtime's environment, and a second process sharing the container. The suites were not wrong. Each one stopped a layer short of the process, so these get caught by a deployment check and a review question rather than by a unit test." } } ] }

Move the deploy runner onto the tailnet, skip Tailscale OIDC

Jeremy Longshore — Mon, 19 Oct 2026 11:22:44 +0000

When a deploy needs a fresh login token every run, and the broker that issues those tokens stops being free, the token call returns 402 Payment Required and the rest of the job still reports success. Production stays on the previous build for days. What you have at the end of this post is a GitHub Actions deploy workflow with two settings any caller can flip (join-tailnet: false, and send-revision: false), a mode where the runner is already a member of the private network and so does not need to mint a token at all, and the order in which to migrate each repo that calls this workflow one at a time.

On 2026-10-14, every deploy through one shared VPS deploy workflow failed the same way: a 402 Payment Required from Tailscale on the first call to mint a short-lived network token, because the Tailscale account (the VPN that brokers access to a private mesh network, called a tailnet) had been moved off its free trial the same day. The run summary said success on every check. The merged commits landed in GitHub. Production stayed on the previous build until someone read the deploy log line by line, two days later.

This is what a paid-tier expiry looks like when the only path from the runner to the production host is a token-mint handshake. The fix is not renewing the trial. The fix is making network membership the runner's job, not the workflow's.

What the old deploy job actually did

The old deploy shape, repeated across every consumer of the reusable workflow, looked roughly like this:

Mint an ephemeral Tailscale auth key by exchanging a GitHub Actions OIDC token against a federated trust.
Use the key to bring the hosted runner onto the tailnet as a node tagged tag:ci.
SSH to the production host as the deploy user, run the VPS-resident deploy script.
The auth key expires. The runner node drops off the tailnet.

Step 1 is a single API call. Step 2 needs the key in hand. Step 3 is the actual deploy. The whole thing runs in roughly a minute.

When step 1 returned 402, step 2 never ran. The runner stayed off the tailnet. Step 3 ran in the wrong context (the runner was never on the tailnet, so it could not reach the VPS over its Tailscale hostname) and the deploy job reported success anyway, because the SSH step was wrapped in a way that did not propagate the upstream 402.

(SSH_ORIGINAL_COMMAND is the variable an SSH forced-command script reads for the command the client tried to run.)

The reusable workflow change

The reusable workflow lives in .github (the repo that holds the shared CI assets). Two new inputs declared on on.workflow_call:

on:
  workflow_call:
    inputs:
      runs-on:
        type: string
        default: 'ubuntu-latest'
      join-tailnet:
        type: boolean
        default: true

runs-on (string, default ubuntu-latest) picks the deploy job's runner. join-tailnet (boolean, default true) gates the two Tailscale steps. With join-tailnet: false, the OIDC trust exchange and the tailscale up step are skipped entirely, and a plain TCP check of the VPS SSH port over the runner's own tailnet route replaces them. The two Tailscale inputs (TS_OIDC_CLIENT_ID, TS_AUDIENCE) become optional in that mode because the mode never uses them.

The default stays true. Every caller pins the workflow by SHA, so callers migrate one at a time, each with its own deploy as proof. That is the design choice worth defending below.

The TCP-only health check

When join-tailnet is false, the workflow replaces the OIDC exchange and the tailscale up step with a single TCP probe of the VPS SSH port over the runner host's own tailnet route. The shape:

# Inside the deploy job, when inputs.join-tailnet is false.
nc -z "$VPS_HOST" 22 || {
  echo "::error::cannot reach $VPS_HOST:22 over tailnet"
  exit 1
}

The probe is intentionally minimal. It mints nothing, authenticates nothing, logs into nothing. It only proves the runner can reach the VPS SSH port over its own tailnet route, which is exactly the precondition step 3 needs.

This is the no-join fallback. The self-hosted runner is already a tailnet member because it lives on the runner host, which has been a tailnet member since before this workflow existed. The runner reaches the VPS over the same Tailscale hostname the hosted runner would have reached after a successful tailscale up. The TCP probe is the gate that proves the route exists, then the deploy SSH step runs the same way it always has.

The send-revision toggle

A second caller-facing boolean landed the same day, also declared on on.workflow_call:

on:
  workflow_call:
    inputs:
      send-revision:
        type: boolean
        default: true

Default true. When true, the deploy step passes the merged PR's HEAD SHA to the VPS-side script as github.sha. When false, the deploy step connects with no remote command, so SSH_ORIGINAL_COMMAND is empty.

Why: at least one consumer's VPS-resident deploy script is a hardened forced command that accepts only an empty command. It captures current main itself, preflights the live site, and deploys that exact commit with a database backup and rollback path. The first deploy on the self-hosted runner exited 64 because the reusable workflow always sends github.sha. The exit code propagated up; the run summary finally showed red.

The opt-out preserves the hardening. The VPS-side script is owned by its own repo's release process and refuses arguments on purpose. Changing it to accept an argument would weaken the contract that the manual-install path depends on. Changing the workflow to send nothing when a caller asks for nothing is the smaller change.

The opt-in, not the flip

The deploy workflow change is two inputs, both default to the old behaviour, both opt-in per caller. That is the load-bearing design choice, and it is worth stating the alternatives.

Flipping the default would migrate every caller at once. That is also the change that hides failures: a caller who skipped the changelog would see a deploy run on an unexpected runner, and the next time a real bug shows up, no one can tell which runner ran it. Every caller pins the workflow by SHA. They are the unit of migration. Each caller flips join-tailnet on its own deploy, watches its own run, and only then does the next caller move.

Keeping the default at true also keeps the change self-documenting in caller repos. A with: join-tailnet: false line in a caller workflow is the visible signal that this caller is on the new mode. A blanket default flip would erase that signal.

The first consumer: startaitools.com

The first consumer to flip was startaitools.com itself. Its deploy workflow already called the shared .github reusable deploy, so flipping was a single workflow-input change plus the consumer-side runner label so GitHub knew where to dispatch the job. The merged change shipped the same day, the deploy landed on the self-hosted runner, and the run summary finally showed a deploy that did not require a working Tailscale account to even start.

That is the proof pattern. One caller migrates, one deploy proves the mode works for that caller, and only then does the next caller move. The runner host is the existing self-hosted runner platform and is already a tailnet member, so the change at the runner side was zero new code.

Why not just renew the trial

Renewing the Tailscale plan restores this single outage. The reason to still install the opt-in is that "renew this one credential" is not a class-of-failure fix.

Every OIDC handshake that mints a short-lived credential is a vendor handshake. Tailscale OIDC is one example. A custom OIDC issuer pointed at your cloud rotates every 90 days. A GitHub App installation token can be revoked by an org admin who never thinks about your deploy. An IAM trust policy edit on the cloud side breaks the handshake the same way: 403, 401, or 402, deep in the run summary, with the rest of the deploy job looking fine. Any of these would have cost the same two days of stale production.

Moving the runner onto the network removes the handshake from the deploy path. The deploy becomes a TCP probe plus an SSH. The TCP probe depends on the runner host still being a tailnet member, which is a runner-host lifecycle question, not a vendor question. The SSH depends on the deploy user's SSH key, which the operator has regardless of vendor state.

Why not edit the VPS-side script instead

The send-revision toggle exists because at least one VPS-resident deploy script is a hardened forced command that only accepts an empty command. The natural reflex is to relax it: "accept an optional argument, ignore it if empty, run the deploy with whatever revision the runner sends." That is the wrong direction.

The hardening is owned by the repo whose deploy path it is. That repo's release process depends on the script knowing it always runs an empty command, because the script's whole behaviour (capture main, preflight, deploy, back up the database, roll back on smoke failure) assumes the caller is not in control of what gets shipped. Letting a caller pass a revision would re-introduce the divergence loophole the script was built to close. The VPS-resident script is the manual fallback path, and the manual fallback path needs to be the thing that decides what it deploys.

The reusable workflow is the shared asset that every caller already pins by SHA. Adding an opt-out is the smaller change, and it puts the decision at the right layer.

Also shipped

Four new tests cover the runner-mode change; the reusable workflow's suite now reports 10 passing.
A send-revision boolean for callers whose VPS-side deploy script only accepts an empty command; default unchanged.
startaitools.com is the first consumer to migrate, in the same day's release v1.33.24.

Use this

Add a join-tailnet boolean to your reusable deploy workflow, default true, with a TCP-only health check on the runner's own tailnet route as the no-join fallback.
Add a send-revision boolean for callers whose VPS-side deploy script only accepts an empty command, default true, so the workflow can connect without sending github.sha when a caller asks.
Migrate consumers one at a time, each with its own deploy as proof; never flip the default for every caller in one change.

Reference: the reusable workflow at https://github.com/jeremylongshore/.github/blob/main/.github/workflows/vps-deploy.yml.

FAQ

What does a hosted runner's Tailscale OIDC 402 look like inside a GitHub Actions deploy?

The OIDC step POSTs to the Tailscale issuer and receives a 402 Payment Required. The failure happens inside a sub-step that the run summary does not propagate. The rest of the deploy job continues, the job result still reports success, the merged commit lands in GitHub, and production stays on the previous build until the run log is read end to end.

Why is the opt-in design safer than flipping the default for every caller at once?

Every caller pins the reusable workflow by SHA, and the unit of migration is the caller. Each caller flips the toggle on its own deploy, watches its own run, and only then does the next caller move. A blanket default flip hides which caller is on which mode; the visible with: join-tailnet: false line per caller is what keeps the migration self-documenting.

What is a self-hosted runner that is already a tailnet member?

A runner whose host machine has joined the Tailscale network as a long-lived node. The runner reaches other tailnet nodes over their Tailscale hostnames without any per-deploy handshake. The runner host used for this change is already configured this way, which is why no Tailscale account is needed for the new mode.

When a Third-Party CI Credential Expires, the Deploy Fallback Has to Already Live on the VPS: the doctrinal predecessor; VPS-resident deploy script plus SHA-pin guard for the day a third-party CI credential expires.
Tailscale Federated Trust: Scope to Auth Keys, Not All: narrows every deploy federated trust to auth_keys scope, the trust posture this change assumes is still in force.
VPS-as-the-Home Day 1: Eight Deploy Iterations: the original reusable VPS deploy workflow plus Tailscale OIDC, eight iterations to land, the architecture this change extends.

{ "@context": "<a href="https://schema.org">https://schema.org</a>", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "What does a hosted runner's Tailscale OIDC 402 look like inside a GitHub Actions deploy?", "acceptedAnswer": { "@type": "Answer", "text": "The OIDC step POSTs to the Tailscale issuer and receives a 402 Payment Required. The failure happens inside a sub-step that the run summary does not propagate. The rest of the deploy job continues, the job result still reports success, the merged commit lands in GitHub, and production stays on the previous build until the run log is read end to end." } }, { "@type": "Question", "name": "Why is the opt-in design safer than flipping the default for every caller at once?", "acceptedAnswer": { "@type": "Answer", "text": "Every caller pins the reusable workflow by SHA, and the unit of migration is the caller. Each caller flips the toggle on its own deploy, watches its own run, and only then does the next caller move. A blanket default flip hides which caller is on which mode; the visible with: join-tailnet: false line per caller is what keeps the migration self-documenting." } }, { "@type": "Question", "name": "What is a self-hosted runner that is already a tailnet member?", "acceptedAnswer": { "@type": "Answer", "text": "A runner whose host machine has joined the Tailscale network as a long-lived node. The runner reaches other tailnet nodes over their Tailscale hostnames without any per-deploy handshake. The runner host used for this change is already configured this way, which is why no Tailscale account is needed for the new mode." } } ] }

When a Third-Party CI Credential Expires, the Deploy Fallback Has to Already Live on the VPS

Jeremy Longshore — Sun, 18 Oct 2026 11:30:11 +0000

GitHub Actions deploys that depend on a third-party OIDC identity silently 402 deep in the run when that identity's plan ends, and the run summary still turns green while production stays stale. What you have at the end of this post is a VPS-resident deploy script that needs no third-party key, plus a SHA-pin guard that proves the manual path can't ship content that diverged from the merged PR, so the next expiry becomes a one-line recovery instead of a multi-day outage.

On 2026-10-14, two static-site changes shipped their merges; the CI run turned green on every check, and production stayed on the previous build. Nobody noticed for two days. The actual failure (a 402 payment-required from a third-party identity provider) was buried inside the first few seconds of a multi-minute deploy job, in a place the run summary had no reason to glance.

This is what a third-party CI credential expiry actually looks like. The fix is not "renew the credential." The fix is having a deploy script that lives on the production host, needs no third-party key, and is verified against the same content hash the merged PR introduced. With that in place, the next outage is a one-line SSH invocation; without it, the next outage is a multi-day production gap.

(OIDC issues short-lived tokens for machine identity, replacing stored secrets.) GitHub Actions steps that need to talk to a private API usually authenticate by exchanging a workload identity token for a short-lived key from the vendor's issuer. Tailscale OIDC is one example; a custom OIDC issuer pointed at your cloud, a GitHub App installation token behind an OIDC handshake, and any number of cloud IAM trust policies follow the same shape. Renewal cycles on any of those can outrun a developer who is not actively staring at the deploy log.

(VPS-resident means a deploy script on the host itself, run over SSH.) The whole point is that the fallback must not depend on the same vendor handshake that just broke. After the vendor is gone, the only credential you still have is your SSH key. The fallback has to need only that.

The day-of incident is a useful case study, but the more useful find is the pattern: every deploy script that exists on a host today was installed by hand before the first CI run that depended on it. That ordering (manual install first, CI adoption second) is what makes the recovery a one-liner whenever the third party goes away.

What the VPS-resident script actually executes

The script's structure looks like this. The real one (under ops/deploy/force-commands in the ops repo) is longer; this is the contract.

#!/usr/bin/env bash
# VPS-resident deploy fallback.
# Same build/smoke/revert shape as the GH Actions deploy job.
set -euo pipefail

SHA="${1:?usage: deploy-repo.sh <merged-pr-head-sha>}"
LOGDIR="/var/log/deploy-fallback"
mkdir -p "$LOGDIR"

# 1. Pull the merged PR's exact commit; the deploy must equal the merge.
git -C /srv/repo fetch --no-tags origin "$SHA"
git -C /srv/repo checkout --quiet "$SHA"

# 2. Build; on failure, leave the live tree untouched.
if ! ./scripts/build.sh; then
  git -C /srv/repo checkout --quiet -
  exit 1
fi

# 3. Atomic symlink swap + smoke; revert-on-failure.
ln -sfn /srv/repo/dist.new /srv/repo/dist.live
systemctl reload caddy
curl -fsS https://example.com/healthz || {
  ln -sfn /srv/repo/dist /srv/repo/dist.live
  systemctl reload caddy
  exit 2
}

Three things in that shape are load-bearing and easy to miss.

First, the script reads a SHA argument. It does not pull from main and hope. Production ends up running the merged PR's exact commit, byte-identical to what the merge step produced on GitHub.

Second, the build failure path is a checkout rollback, not a half-deployed tree. If build.sh fails, the script leaves the live tree pointing at the previous commit. There is no "almost deployed" state.

Third, the healthz smoke gates the symlink swap. A bad build never goes live. A failing smoke flips the symlink back before Caddy notices anything is wrong.

The receipt that keeps the manual path honest

The missing piece for any VPS-resident fallback is the receipt (a machine-written record proving a step really ran) that proves the manual path is not a content-divergence loophole. The safe-merge guard (under ops/merge-guard in the same ops repo) refuses a manual deploy unless the SHA you intend to ship is the exact HEAD of the merged PR. Receipt shape:

SAFE-MERGE: PR #1648 SHA=7cff13d48... status=ok
SAFE-MERGE: manual-deploy-allowed sha=7cff13d48... matches PR #1648 head

If you typed a different SHA, the guard would refuse the run before the build even started. The recovery is execution-divergence (CI is down, manual is up) while the artifact stays identical. Curl-able probe shape:

GET /merge-guard/deploy/v1/tonsofskills?pr=1648 -> { allowed: true, sha: "7cff13d48..." }

No safe-merge pass, no deploy-repo.sh. The OIDC outage does not turn into "well, we just shipped whatever was on disk."

The silent failure shape

A normal failed deploy stops the run summary in red and pings. An OIDC 402 looks like this inside the GitHub Actions run log:

2026-10-14T04:55:11Z ::group::Step Tailscale OIDC
2026-10-14T04:55:11Z POST https://api.tailscale.com/api/v2/oauth/token
2026-10-14T04:55:12Z <- 402 Payment Required
2026-10-14T04:55:12Z body: "your Tailscale trial has ended; choose a plan to create or change keys"
2026-10-14T04:55:12Z ::endgroup::
# (later steps continue; the 402 never propagates upward)
2026-10-14T04:55:30Z Set up job
2026-10-14T04:55:31Z Complete job
Result: success

The deploy job's overall result is success because the failure happened inside a sub-step the run summary never propagates. The merged commit lands in GitHub; the live site stays on the previous build. The 402 is the only signal that production is stale, and the 402 is buried in a ::group:: block inside the first few seconds of a multi-minute job.

Why not just fix CI?

Renewing the Tailscale plan restores this single outage. The reason to still install the fallback is that "renew this one credential" is not a class-of-failure fix.

A custom OIDC issuer pointed at your cloud rotates every 90 days. A GitHub App installation token can be revoked by an org admin who never thinks about your deploy. An IAM trust policy edit on the cloud side breaks the handshake the same way: 403, 401, or 402, deep in the run summary, with the rest of the deploy job looking fine. Any of these would have cost the same two days of stale production.

CI runs on credentials. A VPS-resident deploy script that needs only an SSH key (which the operator has regardless of vendor state) does not. Reducing CI from "the only deploy path" to "one of two" means the next expiry is a frictionless recovery instead of an outage.

You do not have to migrate the whole pipeline. The shape that works:

Build the GH Actions deploy job so its build/smoke/revert shape is one bash file plus a few cp and systemctl reloads.
Drop that script on the host at /usr/local/sbin/<repo>-deploy, gated by a manual-install checklist that requires the script to land before any CI deploy depends on it.
Add a safe-merge-style SHA guard so the manual path can only ship the merged PR's HEAD.

The manual-install gate is the load-bearing piece. It means no CI deploy is the first deploy. Every deploy script on a host today was installed by hand before the first CI run that depended on it; that ordering is what makes the next expiry a one-liner SSH invocation instead of a multi-day production gap.

Also shipped

tons-of-skills-marketplace PR #1661 bundled Fuse.js and dropped cdn.jsdelivr.net from script-src, with a 15-page live crawl showing zero CSP violations (Claude Opus 5 5 and Claude Fable 5 1 in the review lanes).
Landing page flagship card (landing#79 + an intent-os (the internal ops + docs repo) PR covering the Tons-of-Skills flagship) added the Tons-of-Skills card; merge-guard added landing to its repo list and let both settle via safe-merge.
Catalyst-onboarding-v2 P3 closed (3B.3, 3B.4, 3C GATE-P3) under a parallel MiniMax M3 lane with primary QA and scoped interpreter-boundary reviews.
Same-day PR #1648 (remove Google Analytics, Umami only) shipped two days late because of the Tailscale 402; GPT 6 Astra redrafted the leadership briefing later in the day.
The CI failure was found by reading the cross-session log entry mid-deploy, not by alert.

Use this

Install the VPS-resident deploy script before the first CI deploy depends on it, and gate the install behind a manual checklist that asks "is the manual fallback on disk?".
Couple the manual path to a SHA-pin guard (safe-merge or equivalent) so the fallback cannot ship content that diverges from the merged PR's HEAD.
Treat "renew the trial" as recovery, not remediation; the fix is reducing CI from sole deploy path to one of two, so the next third-party OIDC expiry is a non-event.

Reference: the ops/deploy tree at https://github.com/intent-solutions-io/intent-os/tree/main/ops/deploy.

FAQ

What does GitHub Actions do when a third-party OIDC trial expires?

The deploy job's overall result stays success because the 402 lands inside a ::group:: sub-step that the run summary never propagates. The merged commit is on GitHub, the run summary is green, and production stays on the previous build until someone reads the raw log line by line.

Why is the manual VPS fallback safer than running the deploy by hand?

A bare ssh plus git checkout against main can ship whatever sits on disk, which is exactly the failure mode the recovery is built to prevent. That guard refuses any manual run whose SHA isn't the merged PR's HEAD, so artifact identity survives even when CI itself is unreachable.

What is a SHA-pin guard?

It is a small pre-flight check that compares the revision you intend to deploy against what the merged PR actually recorded, and refuses the run if the two diverge. In this stack it lives at ops/merge-guard/safe-merge; its only authorisation (SAFE-MERGE: PR #1648 SHA=... status=ok) is what lets the VPS-resident deploy-repo.sh actually execute.

highest-existing-tag-beats-git-describe: when a bot commit re-uses a tag the release workflow expects to own, git describe traps the version bump and the version stops moving. git tag --list 'v*' --sort=-v:refname | head -n1 and the noise goes away.
smtp-250-not-delivery-three-receipts: SMTP 250 only confirms that the relay accepted the message; to prove delivery you need a frozen ApprovedMessage at enqueue, phase-classified SMTP outcomes per RFC 5321, and reconciliation against admissible evidence before any resend.
rank-only-reporting-keeps-the-eval-honest: when a retrieval evaluation drives its own corpus cleanup, the cleanup should not be in the scored set; report the rank of each gap-filler doc instead.

{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What does a Tailscale OIDC trial expiry look like inside a GitHub Actions deploy?",
"acceptedAnswer": {
"@type": "Answer",
"text": "The OIDC step POSTs to the Tailscale issuer and receives a 402 Payment Required inside a ::group:: block early in the run log; later deploy steps continue, the job result still reports success, the merged commit lands in GitHub, and production stays on the previous build until the run log is read end to end."
}
},
{
"@type": "Question",
"name": "How does a VPS-resident deploy script recover a deploy when third-party CI authentication is broken?",
"acceptedAnswer": {
"@type": "Answer",
"text": "A bash script on the host checks out the merged PR's exact commit SHA, runs the same build and smoke and revert shape the GH Actions deploy job uses, and is gated by a content-hash guard that refuses any commit other than the merged PR's HEAD. The recovery is one SSH invocation and needs no third-party credential."
}
},
{
"@type": "Question",
"name": "Is renewing a single third-party CI credential enough to fix a deploy outage?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Renewing the credential restores the single outage but does not remove the class of failure. Tailscale OIDC, custom OIDC issuers, GitHub App installation tokens, and IAM trust policies all run on credentials that can expire or be revoked. Reducing CI from the sole deploy path to one of two turns the next expiry into a frictionless recovery instead of a multi-day production gap."
}
}
]
}

Verify Audit Evidence with a Signed Payload, Not by Format

Jeremy Longshore — Wed, 14 Oct 2026 11:30:11 +0000

When many audit-evidence publishers (the systems that produce signed rows of evidence) each emit a different shape and the consumer checks signatures case by case, a tampered row is indistinguishable from a legitimate format change. The consumer has no place to anchor "last good state," so any verification failure either erases history or silently keeps going.

A reader running an audit trail or evidence pipeline across multiple publishers and one consumer can adopt a different shape this week. One canonical signed statement payload (a fixed envelope of kernel version plus payload plus signature that every publisher fills in identically), one verifier that checks by signature, a preserved snapshot on every rejection. That is what cut this pipeline's tampering failures from "ambiguous" to "rejected with history preserved."

How do you verify audit evidence from multiple publishers?

Adopt one canonical signed statement payload: a fixed envelope of kernel version plus payload plus signature that every publisher fills in identically. One verifier parses the envelope, recomputes the digest over the canonical predicate bytes, and checks the signature against the public key named in the envelope. Format evolution shows up as a signature failure; tampering shows up as a signature failure; the two are no longer distinguishable to a silent verifier. A preserved last-verified snapshot rides with every rejection so the dashboard can answer what the system believed yesterday without needing yesterday's process to be alive.

The format-evolution trap

The natural reflex is to keep adapting the verifier. Each publisher says its evidence grew up: a new field, a renamed key, a moved nested block. The verifier grows a new branch for each, and the test matrix grows with it. The signed-payload approach is to stop growing the verifier at all and stop asking the verifier to know any individual publisher's shape. The signature binds to the digest of the full payload bytes; the verifier either trusts that digest matches the signed payload or it does not.

What makes this load bearing is that the failure mode is silent. A coerced signature over a slightly different payload passes a content check and fails a signature check. Both look like progress on a glance: the row got bigger, a field appeared, the count went up. The dashboard in this case was logging "verified" against a local witness and "rejected" against the kernel-signed digest, and the two tables had been drifting apart for three nights before anyone noticed.

The audit-harness (the in-repo package of deterministic test-quality checks) release carried a verified registry signature and provenance attestation on 2026-10-12, and that is what made the change carry weight. The pipeline was not adopting a new idea; it was adopting a long-standing in-toto-style attestation pattern that the harness had already shipped.

The one canonical statement payload

Every publisher now emits the same shape: a kernel-versioned envelope with a typed statement, a predicate payload under it, a digest over the canonical predicate bytes, and a signature bound to that digest. The kernel version is part of the bytes that get signed, so a version bump cannot pass as the same payload. The payload bytes follow; the digest covers them; the signature covers the envelope plus the digest.

statement:
  kernel: "audit-payload/v1"
  subject:
    - name: "publisher-name"
      digest: { sha256: "<hex>" }
  predicate:
    type: "https://intent-eval-platform.io/predicate/<kind>"
    payload:
      <canonical JSON: keys sorted, no whitespace, RFC 8785>
  metadata:
    build_at: "2026-10-12T13:45:00Z"
    source: "<publisher repo>"
envelope:
  digest: { sha256: "<hex of canonical predicate bytes>" }
  signature:
    pub_key_id: "<fingerprint>"
    value: "<base64>"

The kernel is the only component the verifier needs to know. Below the kernel, payloads are opaque to the verifier; their shape belongs to the publisher that emits them. Above the kernel, signatures travel with rows. The witness table only needs three columns: subject, digest, signature.

This is the reverse of how multi-publisher audits usually grow. The usual pressure is "the consumer has to understand the publisher," which pushes every format decision into the consumer. Inverting that pressure means every format decision lives with the publisher; the consumer only verifies that the signature covers the kernel and the digest.

One consumer, verification by signature

The dashboard verifier does three things in order, and only those three. It parses the envelope, recomputes the digest over the canonical predicate bytes, and checks the signature against the public key named in the envelope. A rejection at any step moves the row into a holding area; a pass moves it into the witness table under the subject digest.

def verify(envelope: dict, keyring: Keyring) -> Verdict:
    kernel = envelope["statement"]["kernel"]
    if kernel != SUPPORTED_KERNEL:
        return Verdict.reject(kernel_unsupported=kernel)

    predicate_bytes = canonicalize(envelope["statement"]["predicate"])
    expected = sha256(predicate_bytes).hexdigest()
    declared = envelope["envelope"]["digest"]["sha256"]
    if expected != declared:
        return Verdict.reject(digest_mismatch={
            "declared": declared, "recomputed": expected})

    pub = keyring.lookup(envelope["envelope"]["signature"]["pub_key_id"])
    if pub is None:
        return Verdict.reject(unknown_key=...)

    if not pub.verify(predicate_bytes, envelope["envelope"]["signature"]["value"]):
        return Verdict.reject(bad_signature=True)

    return Verdict.accept(subject=envelope["statement"]["subject"],
                          digest=expected,
                          signature=envelope["envelope"]["signature"])

The verifier does not branch on predicate type, publisher, or layout. If a publisher adds a field, the canonicalizer's output changes, the digest changes, and the signature has to follow or verification fails. Format evolution shows up as a signature failure; that is the design, not a defect.

When verification fails, hold the last good snapshot

A rejection is supposed to be an event the system keeps, not one it papers over. The tamper-evident holding area preserves the rejected envelope verbatim; the witness table keeps the last accepted row for that subject. A mutation that does not pass is a mutation the consumer has not yet agreed on, and the right answer is to surface that without erasing what already passed.

def apply(verdict: Verdict, store: Store, snapshot: Snapshot) -> ApplyResult:
    if verdict.kind == "reject":
        store.rejections.append({
            "envelope": verdict.envelope,
            "reason": verdict.reason,
            "received_at": utcnow(),
        })
        store.rejections.prune(keep_last=REJECTION_WINDOW)
        return ApplyResult.held(
            last_known=store.witnesses.get(verdict.subject),
            rejection=verdict.reason)

    store.witnesses[verdict.subject] = verdict.digest
    snapshot.persist(verdict.subject, verdict.digest, verdict.signature)
    return ApplyResult.accepted(
        subject=verdict.subject, digest=verdict.digest)

The snapshot sits alongside the witness table for the same reason the witness table is keyed by subject digest. The dashboard can answer "what did the system believe yesterday?" without needing yesterday's process to be alive. That is the property that broke before, when a forced schema migration overwrote the witness table with a partially-verified copy.

What got shipped

Six rejections in cross-repository tests covered the cases the old verifier could not tell apart: a single-byte payload mutation, a canonicalization difference on whitespace, a key id not in the ring, a signature over a tampered digest, a kernel version one off, and a payload reordered against an older row. Each rejection kept the witness for its subject. None of them caused history loss.

J-Rig (an internal governed evidence publisher) emitted fourteen canonical rows from the merged commit on the night of 2026-10-12. All fourteen verify against the dashboard signature path on the same day. J-Rig's twenty-four payload contracts pass across kernel versions 0.10.0 and 0.11.0; a stricter type check is now in CI for the test-only mismatch between the branded types in those versions.

A second finding came out of the same session: the trace-intelligence program's dependency graph needed 438 edges added to align final review and production-adoption with their prerequisites. The graph was repaired without changing ownership, acceptance criteria, or completion states. The collision between the two is the one to surface: a missing migration record and a missing dependency edge are both evidence the system was lying to itself about state; the same cross-repo fix path surfaced both, and neither was visible from the dashboard before it landed.

A real failure worth keeping. The scanner adapter had been treating gitleaks's exit-1 as a policy finding, conflating scanner crashes with findings. A contract-test surface reproduces this now, and required-missing-tool behavior stays advisory instead of silently raising real findings. The adapter is no longer a place where the verifier has to second-guess the publisher's report.

Why not the obvious approach?

Two alternatives were on the table. The first was a richer schema with publisher-specific extensions and a per-publisher verifier; the second was a content-addressed store that names files by their hash and trusts whoever can read them.

The richer schema pushes the format-decision pressure the wrong way. Every new field forces the consumer to learn a shape it does not own, and a tampered field reads as a new field until someone diffs the schema. The cross-repository tests in this pipeline had eight publisher schemas to maintain, and three of them were already a quarter behind. A per-publisher verifier is the natural extension of that path; it does not survive the next publisher.

The content-addressed store is closer to the working shape, but it answers the wrong question. A digest proves the bytes are intact; it does not prove the publisher had the right to issue them. A row whose digest matches but whose signing key is not in the ring still passes. The signed envelope binds the digest to a key id, and the verifier's second step is the only place that check lives.

A real reversal surfaced in the J-Rig kernel compatibility work. The earlier verifier accepted a "legacy fallback" branch so older J-Rig rows would not be lost on the cutover; that branch was a place a tampered payload could land without the witness table noticing. Removing the fallback is what makes the dashboard reject every J-Rig row that comes through the tampering or fallback branch today; the fourteen nightly rows verify on the canonical path. Keeping the old rows was the wrong kind of compatibility. The right kind is keeping the witness for the same subject, not the same shape.

Use this

Pick one canonical signed payload shape for every publisher in the pipeline. Kernel version plus payload digest plus signature. Anything below the kernel is the publisher's choice; anything above is the signature's. Format evolution shows up as a signature failure, not as a silent accept.
Move verification out of every publisher-specific verifier. One consumer, one supported kernel, one signature check. Rejections are a feature. Cross-repo tests should treat new publishers as an unknown key id, not as a new branch.
Preserve the last good witness for every subject on every rejection. The dashboard answers "what did we believe yesterday?" by reading the witness table, which means the witness table has to survive the rejection that pointed at a tampered payload.

The full migration is live on intent-eval-platform.io: eight publishers, one verifier, the J-Rig nightly run, and audit-harness 1.5.1 on the registry with verified provenance.

FAQ

How do you verify audit evidence from multiple publishers?

What is a signed statement payload in an audit pipeline?

A fixed envelope containing a typed statement, a payload digest, and a signature bound to that digest. Every publisher fills in the same envelope; payloads below the kernel can be any shape the publisher chooses. The signature covers the digest plus the envelope, so a single-byte payload mutation or a canonicalization difference on whitespace fails the check while legitimate history still verifies.

What should a verifier do when an audit row fails verification?

Move the rejected envelope verbatim into a holding area; keep the last accepted row for that subject in the witness table; persist the last-known-good snapshot so the dashboard can answer what the system believed yesterday without needing yesterday's process to be alive. A rejection is an event the system keeps, not one it papers over.

{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "How do you verify audit evidence from multiple publishers?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Adopt one canonical signed statement payload: a fixed envelope of kernel version plus payload plus signature that every publisher fills in identically. One verifier parses the envelope, recomputes the digest over the canonical predicate bytes, and checks the signature against the public key named in the envelope. Format evolution shows up as a signature failure; tampering shows up as a signature failure; the two are no longer distinguishable to a silent verifier."
}
},
{
"@type": "Question",
"name": "What is a signed statement payload in an audit pipeline?",
"acceptedAnswer": {
"@type": "Answer",
"text": "A fixed envelope containing a typed statement, a payload digest, and a signature bound to that digest. Every publisher fills in the same envelope; payloads below the kernel can be any shape the publisher chooses. The signature covers the digest plus the envelope, so a single-byte payload mutation or a canonicalization difference on whitespace fails the check while legitimate history still verifies."
}
},
{
"@type": "Question",
"name": "What should a verifier do when an audit row fails verification?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Move the rejected envelope verbatim into a holding area; keep the last accepted row for that subject in the witness table; persist the last-known-good snapshot so the dashboard can answer what the system believed yesterday without needing yesterday's process to be alive. A rejection is an event the system keeps, not one it papers over."
}
}
]
}

}}">Name the Single Authority in a Daily Pipeline - Why one named owner per concern beats committee routing in a daily pipeline
}}">Three-Pass Settlement for a Recurring AI Pipeline Contract - How three passes (producer, settlement, audit) settle a recurring AI pipeline contract
}}">Exit Zero Is Not Push OK - A cron that reports OK after a failed git rebase is a load-bearing lie

Park a Migration Without Losing the Work

Jeremy Longshore — Sat, 10 Oct 2026 11:30:13 +0000

A migration that ships by reaching the default branch with a green badge still has a second decision hidden inside it: do we point production at it today, in a week, or ever. When the build and the deploy are welded together, that second decision leaks into rollback theater even when the right answer is to wait months. The pattern this post calls publish-and-park splits them on purpose, so parking an owner can be the cutover's moment instead of its point of no return.

A Flask-to-Django refactor in catalyst-onboarding (a public onboarding repo) reached that fork in October. The Django stack was complete, gated, and pushed to two feature branches as atomic fast-forwards. Production stayed on Flask. The work stayed on the feature branches where it had landed. The pattern that made the cutover reversible is what this post is about.

What is the publish-and-park pattern for software migrations?

The publish-and-park pattern splits a refactor's build from its cutover so parking the work is reversible. The gate suite runs on the merged commit on a feature branch with no production mutation, and a dated "do not cut over" record lands in the repo. Production stays on the old stack until a separate decision points it at the new one.

The pattern, in one sentence

Push the refactor to its feature branches as an atomic fast-forward, run the full gate suite on the merged commit, ship nothing to production, write the explicit "do not cut over" record, and open a successor planning repo so the parked work has somewhere forward to go.

Five invariants make a parked refactor recoverable instead of abandoned:

Atomic push to a feature branch. The work lands as one fast-forward commit. No merge commit. No force push. The branch tip is byte-identical to the merged result.
Full gate suite on the merged commit. Unit, integration, contract, coverage audit, security, and an independent reviewer all run against the tip. Green badge is recorded on the commit, not on a moving branch.
No production mutation at ship time. The cutover is a separate operation, owned by a separate decision, executed on a separate day.
An explicit "do not cut over" document. A dated decision record lives at the top of the repo. Future readers find it before they touch the parked work.
A successor repo. The parked work has somewhere forward to go. It is not a dead branch waiting for a code archaeologist.

When all five hold, parking an owner is reversible. When any one is missing, parking an owner is abandonment in slow motion.

The build-up trail

The Flask to Django migration was not a weekend. It was six stacked PRs landed over a single day, each one a thin slice of the new lifecycle. The numbering gap between #113 and #115 is an unrelated CI-infra fix that interleaved between the intake and review slices; it is not part of the migration stack and it would be a mistake to read the migration as seven PRs because of it.

PR #113 feat(intake):     add native Django intake and durable Temporal mail
PR #115 feat(review):     run native stages with durable evidence and bounded admission
PR #116 feat(deploy):     promote authenticated immutable image archives
PR #117 feat(profile):    persist native review profiles and durable CRM projection
PR #118 feat(ci):         record complete shard phase and worker timings
PR #119 feat(onboarding): add Select partner footers and repair bounded CI setup

Each PR carried its own gate subset. The full suite ran only when the last commit on the immutable milestone branch was in place. That ordering matters: a partial green badge on a partial stack is not a parking artifact. It is a half-truth.

The atomic push

The push that turned the work into a parking artifact looked like this:

# After the full makecheck suite passed locally and was re-run by CI on the same SHA:
git push origin feat/django-agreement-lifecycle:feat/django-agreement-lifecycle \
  --ff-only
git push origin feat/django-agreement-notifications:feat/django-agreement-notifications \
  --ff-only

Two pushes, both fast-forward only. The remote tip is byte-identical to the local tip. No merge commit hides what was actually shipped. No force push re-writes history. A reader six months from now can run git log on either branch and reconstruct the exact build state.

The atomic property is the load-bearing one. A merge commit would let a future force-push hide reverts. A non-fast-forward push would break anyone who already has the branch checked out. Neither breaks the next person, but both break the parking pattern's audit trail.

The gate suite on the merged commit

The full gate suite ran on the immutable milestone commit, not on the moving feature branches:

makecheck terminal PASS 598.411s
  1746 unit + 1 skip
  2764 SQLite + 160 skip
  436 pilot
  static + security

PG terminal 4659 PASS / 14 skip / 4673 collected
  1262.33s test / 1271.060s whole

v2 coverage audit PASS
Independent reviewer APPROVE <commit> ONLY

The reviewer line is the one most teams skip. A human (or a strictly scoped reviewer agent) confirms the gate results match the code. Without that confirmation, a green badge is a CI vendor's promise. With it, a green badge is a recorded decision by the team that owns the codebase. The parked refactor is recoverable because the gate results are pinned to the commit, not to the workflow run.

The cutover stays on Flask

Production stayed on Flask main. The Django stack was pushed to its feature branches, and the PR was converted to draft. A full-ref backup bundle was written to ~/backups/catalyst-onboarding-preserve-* the following morning; the path-embedded date is for the snapshot job that wrote it, not for the work itself. The bundle is belt-and-suspenders: the feature branches are the primary artifact, the backup is the secondary. Neither mutates production.

This is the second decision, the one the publish-and-park pattern splits out. The decision is not "is the work done." It is "is this the right moment to point production at it." The two questions have different owners, different evidence, and different reversibility costs. Welding them together is what makes migrations feel like a one-way door.

Why not the obvious approach?

The obvious approach is to merge the Django stack to main, let CI go green, and cut over. Three reasons that fails:

The badge is on the branch, not the commit. A green badge on main after the merge tells you the gate suite passed against a moving target. Anyone who has reset main to undo the merge invalidates the badge retroactively. The parked refactor has no such exposure: the gate results are pinned to the commit SHA on the feature branch, and that SHA does not move.
Cutting over is a single failure domain. If the cutover breaks, you do not know whether the bug is in the new code, in the deploy step, or in the rollback. With the cutover decoupled, the new code can sit on a feature branch for months while the deploy step is rehearsed separately. The two failure domains stay separate.
The "do not cut over" decision has nowhere to live. A merge to main is the act of cutting over, by definition. There is no document you can write that says "this is on main but do not use it" without breaking the social contract that main is production. The decision record has to land before the merge, which means the merge has to be deferred, which means the work has to live on a feature branch.

The pattern is not a workaround for a missing feature. It is the only shape that makes parking an owner recoverable.

The "do not cut over" record

The decision record is a PR, not a wiki page and not a numbered-record entry. PR #124 in catalyst-onboarding carries the dated decision that the Django stack stays parked and Flask stays in production. The PR body names the commit, links the full-ref bundle, lists the gates that passed, and names the successor repo.

The PR is the artifact a future reader finds before they touch the parked work. A wiki page rots. A numbered-record entry (each repo's dated folder of decision history) can be moved, renamed, or lost when the numbering scheme changes. A PR lives at the top of the repo's pull request list until it is closed, and its merge commit lives in main history forever.

The successor repo

The parked refactor is not abandoned because it has a successor: intent-solutions-io/catalyst-onboarding-v2, opened the following morning with the beads (a tracked task record in the repo's issue database) prefix catalyst-v2 and no code. The planning-phase foundation exists so the parked work has somewhere forward to go.

A parked refactor without a successor is a dead branch. A parked refactor with a successor is a deferred plan. The successor repo does not have to contain the parked work; it has to contain the plan for the parked work. The Django stack on feat/django-agreement-lifecycle is the parking artifact; catalyst-onboarding-v2 is the forward home.

What survives a routine branch cleanup

The parked refactor survived a repo sweep that removed 34 temporary worktrees and deleted 25 merged branches. The work lives on the published feature branches and on main history, not in disposable worktrees or speculative branches.

Atomic fast-forward pushes land on real branches with real tips. A repo sweep that removes temporary worktrees and merged branches cannot touch those tips without changing the published history. That is a consequence of invariant one: durable artifacts outlive routine cleanup.

What did not change the parked work

In the 24 hours after the parking decision, a parallel hardening window merged fifteen fixes in claude-code-plugins: patched advisories in shipped MCP servers, upgraded actions/cache, actions/github-script, codecov-action, the cosign installer, download-artifact, pnpm/action-setup, and Vitest. None of those PRs touched the parked Django work. None of them needed to. The decoupling held under a load that would have broken a single-stack deployment.

The point is not that the hardening window was easy. The point is that it was possible without a deploy coordination meeting. Two repos, two lifecycles, no shared cutover.

Use this

Push the refactor as --ff-only to a feature branch, not as a merge to main. The atomic property is what makes the gate suite pin to a commit instead of a moving branch.
Run the full gate suite on the merged commit, including the independent reviewer. A CI badge is the vendor's promise; a reviewer-approved badge is the team's promise.
Write the "do not cut over" decision as a PR, and open a successor planning repo. The decision has to live where the next reader will find it, and the parked work has to have somewhere forward to go.

The Django stack in catalyst-onboarding is parked. The work is not abandoned. The five invariants held, and parking the cutover became a reversible decision instead of a one-way door.

The parked Django work and the dated parking decision: intentsolutions pull 120 and pull 124.

Frequently asked questions

How do you park a refactor without abandoning it?

Five invariants hold: the work lands as an atomic fast-forward push, the gate suite pins to the merged commit (including an independent reviewer), no production mutation happens at ship time, a "do not cut over" decision record lands in the repo, and a successor repo gives the parked work somewhere forward to go. Drop any one and the parked work becomes abandoned in slow motion.

When should you delay a production cutover after a green merge?

Delay when a green badge sits on a moving branch, when the cutover and rollback share one failure domain, or when the "do not cut over" decision has nowhere to live. Parking keeps the new code on a feature branch with a pinned badge, so the deploy step can be rehearsed independently before the swap.

Why not merge to main and cut over later?

A green badge on main sits on a moving target that any reset invalidates retroactively, a cutover welded to a merge is one failure domain, and a "do not cut over" decision has no place to live when main is the act of cutting over. Parking keeps work, gate results, and decision on separate durable artifacts.

three-pass-settlement-recurring-ai-pipeline-contract: A recurring AI pipeline contract project settles only after a third evidence pass that closes every named gap with a concrete file and pins every changed gate manifest. Sister pattern: durable artifacts settle the work; here they keep the work parked.
name-the-single-authority-in-a-daily-pipeline: When a daily pipeline runs unobserved, every concern accumulates competing authorities; the cure is to name exactly one authority per concern. Decision-22 is the named authority here.
link-public-surface-to-deployment-thesis: The publish-and-park pattern breaks the public-surface-to-deployment link on purpose; the cataloged post argues the opposite. Worth reading both before committing a migration.

{ "@context": "<a href="https://schema.org">https://schema.org</a>", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "How do you park a refactor without abandoning it?", "acceptedAnswer": { "@type": "Answer", "text": "Five invariants hold: the work lands as an atomic fast-forward push, the gate suite pins to the merged commit (including an independent reviewer), no production mutation happens at ship time, a 'do not cut over' decision record lands in the repo, and a successor repo gives the parked work somewhere forward to go. Drop any one and the parked work becomes abandoned in slow motion." } }, { "@type": "Question", "name": "When should you delay a production cutover after a green merge?", "acceptedAnswer": { "@type": "Answer", "text": "Delay when a green badge sits on a moving branch, when the cutover and rollback share one failure domain, or when the 'do not cut over' decision has nowhere to live. Parking keeps the new code on a feature branch with a pinned badge, so the deploy step can be rehearsed independently before the swap." } }, { "@type": "Question", "name": "Why not merge to main and cut over later?", "acceptedAnswer": { "@type": "Answer", "text": "A green badge on main sits on a moving target that any reset invalidates retroactively, a cutover welded to a merge is one failure domain, and a 'do not cut over' decision has no place to live when main is the act of cutting over. Parking keeps work, gate results, and decision on separate durable artifacts." } } ] }

Team-Page Ordering Is a Release Decision

Jeremy Longshore — Wed, 30 Sep 2026 11:30:11 +0000

A new Legal Counsel profile landed at the bottom of the team page yesterday. Heather Johnson was the most consequential reader-facing contact on the page, and the page put her last. The fix shipped thirty-four minutes later as v4.4.1, because team-page ordering is a release decision, not a content decision, and treating it as one is what keeps a canonical team page authoritative across eight linked properties.

How should team pages be ordered?

Team-page ordering should put the highest-value reader-facing contact first, after the founder. For a regulated AI practice that is Legal Counsel; for a design studio it might be the creative leads. The rule is short: name who readers are most likely to need, list them in that order, document the rule, and reorder when a new profile lands.

What shipped and what did not

The intent-solutions-landing repo (the source of truth for the company team surface at intentsolutions.io/about/#team) shipped v4.4.0 at 18:49 CDT. The PR added seven team photos, a new team.json roster, a HeatherJohnson.astro profile page, and an updated footer link. The original reviewer caught the ordering at 19:18, while the v4.4.0 deploy was still propagating through the cache. The reviewer did not flag the order as "wrong." They flagged it as "wrong for the audience the page is about to serve." That distinction is what the rest of this post defends. The roster looked like this:

[
  { "name": "Jeremy Longshore", "role": "Founder" },
  { "name": "Opeyemi Adeyemi", "role": "Operations" },
  { "name": "Pablo Hernandez", "role": "Engineering" },
  { "name": "Max Sheahan", "role": "Education" },
  { "name": "Tim Yates", "role": "AI Engineering" },
  { "name": "Tolulope Bakare", "role": "Engineering" },
  { "name": "Heather Johnson", "role": "Legal Counsel" }
]

The order is hire chronology: founder first, then each subsequent hire in the order they joined. Heather was the most recent hire, so she landed at the bottom. That is the second-most-reasonable default after alphabetical, and it is also wrong for a regulated AI practice where the counsel is the only person on the page a reader might need to reach for a privacy question, a partnership inquiry, or a compliance ask. Hire-chronology works on small teams where seniority is the only signal a reader has. Once a reader needs to find one specific person, it works against them.

v4.4.1 reordered the file thirty-four minutes later:

[
  { "name": "Jeremy Longshore", "role": "Founder" },
  { "name": "Heather Johnson", "role": "Legal Counsel" },
  { "name": "Opeyemi Adeyemi", "role": "Operations" },
  { "name": "Pablo Hernandez", "role": "Engineering" },
  { "name": "Max Sheahan", "role": "Education" },
  { "name": "Tim Yates", "role": "AI Engineering" },
  { "name": "Tolulope Bakare", "role": "Engineering" }
]

The diff is six lines moved up. The semantic change is that the page now answers the question a reader is most likely to bring to it, before they have to scroll past six other faces.

Why not the obvious approach?

Two reasonable defaults existed. The first was alphabetical, which we considered and rejected as a draft. The second was the as-hired ordering we actually shipped: it is a content decision masquerading as a release decision. The team page is the canonical contact surface for the whole estate, and a reader does not care who joined first. They care who can answer their question.

The third option, leaving the order to whoever maintains the page next, is the one that drifts. Within three months, someone adds a contractor. Then a summer intern. Then the intern moves up but stays where they are. The team page stops being authoritative and starts being a snapshot.

We chose the role-priority ordering with an explicit rule: after the founder, the page lists people in the order a reader is most likely to need them. Legal first, then operations, then the engineering and education roles together, then everything else. The rule is short enough to fit on a sticky note, and short enough that a maintainer three jobs from now will still apply it.

The role-priority rule is not universal. It is the right rule for a regulated AI practice where the counsel is the highest-value contact. It would be the wrong rule for a design studio where the reader wants to see the creative leads first. The point is not that role-priority is the answer. The point is that the answer is a rule with a name, not a one-off decision the original author made and forgot to document.

The 30-minute footer sweep

The same moment v4.4.0 landed, eight repos each picked up a new "Our team" footer link pointing at intentsolutions.io/about/#team. The sweep window was 18:59 to 19:29 CDT, roughly thirty minutes. Six of those repos are public landing surfaces (intent-demos, omarchy, diagnostic-pro, startaitools, hustle, claude-code-plugins). The seventh is intent-solutions-landing itself, whose team.json is the source of truth. The eighth is now-lms, which also renders its own team profile and shipped the CI gate.

The footer change per repo was mechanical. Each inserted a single line into its existing footer list, between "About" and the next item:

<li><a href="https://intentsolutions.io/about/#team">Our team</a></li>

The insertions are byte-different per repo because each footer is a different framework (site/index.html for the static HTML sites, Footer.tsx for diagnostic-pro, layouts/partials/footer.html for the Hugo build, src/app/(public)/page.tsx for the Next.js hustle app, marketplace/src/layouts/BaseLayout.astro for the Astro marketplace). The pattern is identical. Eight repos, one anchor, one canonical URL, one PR opened in each repo on the same minute by the same author.

The mechanics matter because they explain why the ordering hotfix had to ship in the same hour. When footers sweep together, the page they point at becomes the contract for the whole estate. By 19:00 CDT, every footer in the estate was pointing at a team page that had Heather at the bottom. The hotfix is not a content correction. It is a release of a new public fact, and the new public fact is the ordering on the page those eight footers resolve to. Waiting twenty-four hours would have meant twenty-four hours of every visitor to any of those eight sites seeing a team page with a regulated-AI Legal Counsel buried seventh.

The rollback path was open if v4.4.1 had broken the build: revert the team.json diff, cut v4.4.2, redeploy. Eight footers would have re-resolved to the v4.4.0 ordering for the duration of the rollback. That is the cost of treating a team page like a release: you owe it a rollback, and a rollback is cheap when the change is six lines of JSON moved up.

The alternative, batching the hotfix into the next deploy window, was the original instinct. Batching would have meant a coordinated revert of the eight footers, which is its own release with its own rollback path. The hotfix was cheaper than the batch. That is the math that decides whether something ships same-day or waits.

The CI gate: check_estate_bar.py

Once eight repos link to one URL, that URL becomes load-bearing in a way a single-site URL never is. A rename to /team/ or a layout shift that breaks the #team anchor would silently rot every footer in the estate. The now-lms repo shipped scripts/check_estate_bar.py in the same hour, a 79-line Python check that hits the canonical URL and verifies the anchor still resolves to a non-empty team roster. The gate is intentionally minimal. It does not parse the ordering. It does not check the photo count. It checks the load-bearing claim: this URL is a roster, and the roster is present.

def main() -> int:
    target = os.environ.get("ESTATE_TEAM_URL", DEFAULT_URL)
    r = requests.get(target, timeout=10, headers={"User-Agent": "check-estate-bar/1"})
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")
    anchor = soup.find(id="team")
    if anchor is None:
        print(f"FAIL: {target} has no #team anchor", file=sys.stderr)
        return 1
    members = anchor.find_all(attrs={"data-team-member": True})
    if len(members) < 1:
        print(f"FAIL: {target} #team has no roster", file=sys.stderr)
        return 1
    print(f"PASS: {target} #team has {len(members)} members")
    return 0

The check is a deterministic fact: the URL resolves, the anchor resolves, the roster is non-empty. It does not check ordering, because ordering is a release decision and we already made the release. It checks that the page is still authoritative. That is the gate's job.

The check is wired into now-lms CI as a required step. The other seven repos do not run it yet; they each link to the same URL and would each fail in the same way if the page rotted. Wiring them all is the next bead.

What this looks like at three months

Without the CI gate, the page drifts. A volunteer contributor renames the anchor for SEO. A designer moves the roster into a tabbed component. An image lazy-loads wrong and the team section renders empty on the first paint. None of those changes break the page locally. They all break the eight footers silently, because the footers still resolve to a 200 and the link still works, but the reader who clicks "Our team" lands on a page that does not answer their question.

We have lived this pattern before on the same estate. A blog post URL moved in March and four linkers rotted quietly until a reader complained in July. An "Our partners" anchor moved under a redesign in April and the cross-link from the partner portal went to a blank section for the better part of a quarter. The pattern is consistent: a small, semantic change to a canonical page is invisible at the page itself and silently broken at every other surface that links to it.

The CI gate catches two of those three failure modes. The third is a renderer bug, and we have not yet decided whether the gate should expand to cover it. For now, the gate is a presence check, not a quality check. That is the right scope for a 79-line script. Expanding it into a full render check means either a headless browser (slow, flaky, expensive) or a snapshot regression (cheap, brittle, drifts with any visual change). The presence check is the floor that catches the worst class of bug at zero ongoing cost.

The other thing a CI gate does is give the next maintainer permission to push back. Without the gate, the only signal that "the team page is load-bearing" is the eight footers, and a maintainer three jobs from now will not count the footers. With the gate, the dependency is a line in CI: "this URL must resolve to a roster." That is a contract someone can read, and contracts outlive institutional memory.

What "estate anchor" actually means

An estate anchor is a URL that other properties depend on as a contract rather than as a reference. A reference is a link a reader clicks if they are curious. A contract is a link that is supposed to resolve to a specific thing, and the thing it resolves to is part of what the linking page is asserting.

The footer sweep turned intentsolutions.io/about/#team from a reference into a contract. Before the sweep, the page was a page. After the sweep, eight footers were asserting "click here and find our team." The assertion is the contract. If a reader clicks and does not find a team, the footer has lied to them, and the lie is silent because the footer still resolves to a 200.

Most estates cross the reference-to-contract line without noticing. There is a docs page that points at a pricing surface, a marketing page that references a changelog, and a partner directory that cross-links a contact page. Each link is small. In aggregate, each is a contract. The CI gate is the moment a team acknowledges the contract exists.

The cost of acknowledging the contract is small. The check is 79 lines. The wiring is one line per repo. The maintenance is one URL to remember. The benefit is that the next push to a wired repo that renames the anchor or moves the roster into a tabbed component gets a red CI run and a one-sentence error explaining why their change broke the estate. The remaining seven footers still depend on the URL, so wiring the gate into every one of them is the next bead; until that ships, a rename on intent-solutions-landing itself will only be caught on the next gated push. That is enough to make the right decision the easy one.

The next time a similar pattern comes up on the estate (a new partner joins the directory, a new pricing tier appears, a new compliance doc publishes) the discipline is already there. Same-day fix if the canonical surface changes. CI gate before the third link points at it. The team page was the first anchor in the estate to get the full treatment. The pattern is now load-bearing for every anchor that comes after it.

That is the lesson. A team page is a page. A team page that eight footers depend on is a contract. The cost of treating it like one is one extra script and one extra hour.

Also shipped

The rest of the day's work that does not fit the team-page finding:

now-lms (#125) shipped the estate-bar check plus the matching team profile rendering for Heather, with the same ordering rule as the canonical surface.
intent-os shipped the whiteglove distribution packet (3516 lines, nine print PDFs) and the matching ownership verification records.
hustle v3.3.7 and startaitools v1.21.14 auto-bumped on the same day as the team-link fix landing in each repo.
omarchy refreshed its public plugin facts from the GitHub API (generated content only, no semantic change).

The discipline, generalized

The team page is one canonical surface among several on this estate. The partner directory is another. The pricing page is a third. The published-changelog index is a fourth. Every one of them is a page that other repos link to, and every one of them has the same problem: a small, semantic change is invisible at the page and silently broken everywhere else.

The team-page fix generalizes to a rule: when a single canonical surface becomes load-bearing across the estate, treat every edit to it as a release. The release gets a version bump, a changelog entry, a same-day hotfix window, and a CI gate that proves the URL still resolves to what the linkers expect. None of that is novel. It is the same discipline every API publishes under. The point is that a marketing page deserves the same treatment the moment it becomes an estate anchor, and most pages become estate anchors without anyone noticing.

The reason this matters for a team page specifically is that the failure mode is invisible to the team and visible only to the reader. The team sees the page render correctly in their browser. The reader sees a page where the person they came for is seventh. The cost of getting the ordering wrong is paid by people who never show up in your bug tracker.

Use this

Treat team-page ordering as a release decision. Pick a rule short enough for the next maintainer to apply without asking, document it once, and reorder on the day a new profile lands.
When you turn a single-site URL into an estate anchor, ship a deterministic check that the URL still resolves to the thing eight footers expect. 79 lines of Python is enough for the first gate.
Order the canonical team page by the question a reader is most likely to bring, not by hire date and not by alphabet. The page is a contact surface, not a roster. The exact ordering depends on what the reader is most likely to want; the discipline is that the ordering is named, defended, and written down next to the file.

FAQ

Why should team-page ordering be treated as a release decision?

A team page becomes load-bearing the moment multiple footer links point at it. Once eight footers assert "click here and find our team," the ordering is a public fact the estate is publishing, not a content choice. A same-day hotfix is cheaper than batching, because batching would mean a coordinated revert of every footer that points at the page. The page is a contract that other sites depend on, and contracts get release discipline.

What is an estate anchor?

An estate anchor is a URL that other properties depend on as a contract rather than as a reference. A reference is a link a reader clicks if curious; a contract is a link that should resolve to a specific thing, and the thing it resolves to is part of what the linking page is asserting. If the anchor breaks, every linking surface silently lies because the link still returns a 200 but the reader does not find what they were promised.

How do you keep a team page authoritative across multiple sites?

Ship a deterministic CI gate that hits the canonical URL and verifies the anchor still resolves to a non-empty roster. The check_estate_bar.py gate is 79 lines of Python and runs as a required CI step in now-lms. Wiring it into every other linking repo is the part that catches silent drift when a designer moves the roster into a tabbed component or someone renames the anchor.

{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "Why should team-page ordering be treated as a release decision?",
"acceptedAnswer": {
"@type": "Answer",
"text": "A team page becomes load-bearing the moment multiple footer links point at it. Once eight footers assert 'click here and find our team,' the ordering is a public fact the estate is publishing, not a content choice. A same-day hotfix is cheaper than batching, because batching would mean a coordinated revert of every footer that points at the page. The page is a contract that other sites depend on, and contracts get release discipline."
}
},
{
"@type": "Question",
"name": "What is an estate anchor?",
"acceptedAnswer": {
"@type": "Answer",
"text": "An estate anchor is a URL that other properties depend on as a contract rather than as a reference. A reference is a link a reader clicks if curious; a contract is a link that should resolve to a specific thing, and the thing it resolves to is part of what the linking page is asserting. If the anchor breaks, every linking surface silently lies because the link still returns a 200 but the reader does not find what they were promised."
}
},
{
"@type": "Question",
"name": "How do you keep a team page authoritative across multiple sites?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Ship a deterministic CI gate that hits the canonical URL and verifies the anchor still resolves to a non-empty roster. The check_estate_bar.py gate is 79 lines of Python and runs as a required CI step in now-lms. Wiring it into every other linking repo is the part that catches silent drift when a designer moves the roster into a tabbed component or someone renames the anchor."
}
}
]
}

Use the Primitive, Not the Patch (the same-day-sweep discipline, applied to a security primitive rather than a team page).
Scope Federated Tailscale Trust to Auth Keys (federating one identity surface across the same eight-repo footprint, with the same anchor-page release posture).
Pin the Installer and Add a Renewer (a named rule plus a CI gate that catches silent drift, the same shape as check_estate_bar.py).

Tailscale Federated Trust: Scope to Auth Keys, Not All

Jeremy Longshore — Sun, 27 Sep 2026 11:30:11 +0000

Tailscale Federated Trust: Scope to Auth Keys, Not All

A Tailscale federated trust hands an external identity (in my case, GitHub Actions OIDC) an API token whose reach is governed by a scope field on the trust record. The default is ['all'], which gives that external identity every API the underlying token can reach. If the token is account-admin, your CI runner now is too.

I had 18 deploy trusts wired that way. Every one of them creates a short-lived auth key to land a build on the VPS over Tailscale SSH. None of them need anything else. On 2026-09-25 I narrowed every one to scope:['auth_keys'], re-encrypted the SOPS secrets bundle, and proved it on a real jeremylongshore.com production deploy.

What a federated trust actually is

Tailscale's term for "let an external identity mint a short-lived Tailscale API token" is a federated trust: a record that maps an OIDC subject (or a SAML subject, or a workload identity federation principal) onto a Tailscale API token. The trust is created once by an admin. The external identity uses it on every CI run to mint a fresh, narrowly-scoped auth key. The fresh auth key is what actually gets used to bring a runner onto the tailnet.

The trust does not move packets. It only mints keys. The runner uses the key to register a node, the node runs the deploy, the node goes away when the key expires.

A trust record stored on Tailscale's side looks roughly like this:

{
  "id": "trust_abc123",
  "provider": "github",
  "subject": "repo:intent-solutions-io/jeremylongshore.com:environment:production",
  "scopes": ["all"],
  "token_id": "tk_xyz789",
  "created_at": "2026-04-12T18:22:01Z"
}

The scopes field is what the trust is allowed to ask the token to do. It is not what the runner gets to do once the runner is on the tailnet. Those are two different layers, and a ['all'] trust means the trust itself is allowed to ask the token to do anything, on every CI run, until someone narrows it.

The shape of the trust matters because the trust gets a token, and the token has scope. If the trust was created with an account-admin token and scope:['all'], the OIDC subject inherits that token's full reach every time the trust is exercised. The fact that the runner itself only needs auth_keys does not constrain what the trust can be tricked into minting.

The default that bit me

The Tailscale CLI's tailnet set-trust-credentials (and the web UI's equivalent) defaults scope to ['all'] when you do not pass it. Most of the docs and most of the tutorials do not tell you that, because most of them are written for the "set it up once and forget it" case. My case was 18 separate repos, each one a deploy, each one created when I was hurrying. The hurrying is why they were all ['all'].

The whole change is one line in the trust config.

   trust:
     provider: github
     subject: repo:intent-solutions-io/<repo>:environment:production
-    scopes:
-      - all
+    scopes:
+      - auth_keys

The only change is scopes: [auth_keys]. The OIDC subject is unchanged. The token it maps onto is unchanged. The runner still gets to mint an auth key. It no longer gets to mint anything else.

In SOPS, the edit lands in ops/host/secrets/secrets.prod.sops.yaml, gets re-encrypted with sops update, and the next deploy reads the new scope. The trust itself is updated on Tailscale's side by a follow-up tailscale set-trust-credentials --scopes=auth_keys call, which Tailscale reconciles with the SOPS record. The two stay in sync because both are produced from the same workflow. The secrets.prod.sops.yaml re-encrypt was the headline change in 62f100aa; the per-trust config edits in intent-os/ops/deploy/ followed in the same commit.

Why this is not paranoia

Two reasons this matters beyond hygiene.

First, the trust is the long-lived artifact. The token underneath can be rotated; the trust persists. If the trust says ['all'], every rotation of the token keeps the same reach. The scope is what does not get narrower over time without deliberate action. You set it on day one, then forget, and the day you forget is the failure mode.

Second, the blast radius of an OIDC mis-substitution is bigger than people assume. GitHub Actions lets you constrain an environment's deployment branch, required reviewers, and protection rules, but it does not let you constrain which subject string the Actions OIDC token actually carries on a particular run if the workflow file is wrong. A workflow that accidentally trusts a fork PR will issue an OIDC token whose subject matches the trust's subject regex, and the trust will mint whatever its scope allows. ['all'] is the worst-case scope for that failure mode. ['auth_keys'] is the floor: even a misrouted OIDC exchange only ever produces a key, and the key lands on a tailnet ACL that the runner still has to walk through.

The proof

The way I prove a trust change actually works is to run a real production deploy after the change. For jeremylongshore.com, that is run 36106653875 on GitHub Actions. The deploy job:

Exchanges the OIDC token against the trust, asks for an auth key.
Uses the auth key to bring the runner onto the tailnet as a node tagged tag:ci.
SSHes to the VPS as the deploy user, lands the build, exits.
The auth key expires. The runner node drops off the tailnet.

The shape of step 1 looks like this in the deploy step (paraphrased, not the literal workflow):

- name: Bring runner onto tailnet
  env:
    ACTIONS_ID_TOKEN_REQUEST_TOKEN: ${{ secrets.GH_OIDC_TOKEN }}
    TS_AUTHKEY: ${{ steps.authkey.outputs.key }}
  run: |
    curl -fsSL https://tailscale.com/install.sh | sh
    sudo tailscale up --authkey="$TS_AUTHKEY" --hostname="gh-runner-$GITHUB_RUN_ID" --tags=tag:ci
- name: SSH to VPS and deploy
  uses: appleboy/ssh-action@v1
  with:
    host: intentsolutions
    username: deploy
    script: |
      cd /srv/jeremylongshore && git pull --ff-only && ./deploy.sh

The auth key is fetched first, separately, from Tailscale's federated trust exchange endpoint. The runner then uses that key for tailscale up. The deploy step SSHes to the VPS as the deploy user, lands the build, exits. The runner node drops off when the auth key expires.

After the scopes: [auth_keys] change, run 36106653875 completed end-to-end in 41 seconds. The runner registered, the deploy shipped, the node tore down. The runner could reach nothing beyond an auth key. I verified that by calling tailscale set-acl from inside the same job and getting this back:

$ tailscale set-acl --acl-file=/tmp/empty.json
Error: HTTP 403: scope_mismatch (trust_scopes=auth_keys; required=acl)

That is the proof. A deploy that does not try set-acl does not notice the change; a misrouted OIDC exchange that does try it now fails closed at the trust, before the token is even consulted. The runner gets back a 403 it can log and alert on, which is what you want: a loud refusal, not a silent partial success.

The full sweep was 18 deploy trusts across intent-os, braves-booth, diagnostic-pro, hustle, startaitools, tonsofskills, and the jeremylongshore.com site. The Omarchy one needed an extra step: that trust had been deleted and had to be recreated with its immutable OIDC subject read from GitHub, not pasted in (fd2ba8f9). The other 17 were a sops re-encrypt and a config edit. The deploy pattern is the same; only the OIDC subject string and the tag differ per trust.

Why not just leave scope at all and rely on ACL

Because ACL is a different layer and they do different jobs. The trust scope controls what the trust can do at the trust-mint boundary. The tailnet ACL controls what an already-on-the-tailnet node can reach. If you leave scope:['all'] and rely on ACL to keep the runner pinned, you have moved the only thing protecting you into a layer that the runner bypasses before it gets there.

For my deploys the ACL layer was also locked. The relevant tag:ci rule from the tailnet ACL is roughly:

{
  "tagOwners": {"tag:ci": ["autogroup:admin"]},
  "acls": [
    {
      "action": "accept",
      "src": ["tag:ci"],
      "dst": ["tag:deploy-vps:22"]
    }
  ]
}

That rule pins tag:ci to SSH-only access to the deploy user on the VPS. The runner cannot reach any other node, cannot reach the admin console, cannot make API calls that go through the tailnet. So even an unscoped ['all'] trust would have been pinned down by ACL once the runner registered. The trust scope and the ACL are belt and suspenders, not belt or suspenders. I want both. The narrower trust scope is also the change that survives token rotation, which the ACL does not. Rotate the underlying Tailscale API token every 90 days and the trust scope stays auth_keys automatically, because the scope is on the trust, not on the token.

There is also a third layer people forget about: the Tailscale admin console itself. An account-admin API key minted through a ['all'] trust can call tailscale set-acl against the entire tailnet, including deleting or rewriting ACLs. That is not "the runner touches a node." That is "the runner rewrites the rule that decides what nodes can touch." The trust scope is what stops that path before it starts.

How the work went

The session digest for the day shows the bulk of the operator work landed with MiniMax M3 (the bead triage, the sweep agents, the evidence filing). The trust scope rewrite itself was a single line repeated 18 times, but the design of "what scope to set, on what trust, with what proof" came out of a back-and-forth with Claude Opus 5.5. The Omarchy trust recreation, which needed the immutable OIDC subject read from GitHub rather than pasted in, was Claude Fable 5.1 because the change was small and well-bounded.

The estate-graph proof isolation work in PRs 696, 697, and 698 was reviewed by Claude Sonnet 5 and shipped clean. The collaboration score for the day (0.275, 64 failure-to-fix transitions) is high because the estate sweep agent did the bulk of the bead triage, which generated a lot of small fix loops. The trust scope change itself was almost all up-front design and a one-line config edit per trust.

Also shipped

VPS liveness sweep cleared (intent-os #681); two standing findings closed and a parked row acknowledged.
Estate-graph proof isolation merged (#696, #697, #698): operator queries are now read-only and the validation path is isolated from operator state.
Production smoke suite rewritten for the marketplace-site redesign (f80cc59ad): 132/132 chromium+webkit green on the post-deploy e2e.
CI now fails when plugin verification scores drift from the validator (df3fdae75).
@intentsolutionsio/ccpi bumped to 3.0.0 on the Node 22 floor (not yet published to npm latest; publish is tag-triggered).

Use this

When you create a Tailscale federated trust, set scopes: [auth_keys] explicitly. The default is ['all'].
Treat the trust scope and the tailnet ACL as two separate layers. Lock both. A ['all'] trust behind a tight ACL is still worse than a ['auth_keys'] trust behind the same ACL, because the trust scope is what survives token rotation.
After you change a trust's scope, prove the change on a real production run, not just a tailscale set-trust-credentials --dry-run. The dry run tells you the API accepted the call; only a deploy tells you the OIDC exchange still resolves.

FAQ

What is a Tailscale federated trust?

A Tailscale federated trust is a record that maps an external identity (GitHub Actions OIDC, SAML, or a workload identity federation principal) onto a Tailscale API token. The trust is created once by an admin. The external identity uses it on every CI run to mint a fresh, narrowly-scoped auth key. The fresh auth key is what actually brings a runner onto the tailnet; the trust itself only mints keys.

What does scope:['all'] actually grant on a federated trust?

The scope field on the trust record governs what the trust can ask its mapped token to do. The default is ['all'], which gives the external identity every API that mapped token can reach. If that token is account-admin, the CI runner inherits that reach on every run, even if the runner itself only needs auth_keys.

Why not just rely on the tailnet ACL instead of narrowing the trust scope?

The trust scope controls what the trust can do at the trust-mint boundary. The tailnet ACL controls what an already-on-the-tailnet node can reach. Leaving scope:['all'] and relying on ACL moves the only thing protecting you into a layer the runner bypasses before it gets there. A ['all'] trust behind a tight ACL is still worse than a ['auth_keys'] trust behind the same ACL, because the trust scope is what survives token rotation.

{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is a Tailscale federated trust?",
"acceptedAnswer": {
"@type": "Answer",
"text": "A Tailscale federated trust is a record that maps an external identity (GitHub Actions OIDC, SAML, or a workload identity federation principal) onto a Tailscale API token. The trust is created once by an admin. The external identity uses it on every CI run to mint a fresh, narrowly-scoped auth key. The fresh auth key is what actually brings a runner onto the tailnet."
}
},
{
"@type": "Question",
"name": "What does scope:['all'] actually grant on a federated trust?",
"acceptedAnswer": {
"@type": "Answer",
"text": "The scope field on the trust record governs what the trust can ask its mapped token to do. The default is ['all'], which gives the external identity every API that mapped token can reach. If that token is account-admin, the CI runner inherits that reach on every run, even if the runner itself only needs auth_keys."
}
},
{
"@type": "Question",
"name": "Why not just rely on the tailnet ACL instead of narrowing the trust scope?",
"acceptedAnswer": {
"@type": "Answer",
"text": "The trust scope controls what the trust can do at the trust-mint boundary. The tailnet ACL controls what an already-on-the-tailnet node can reach. Leaving scope:['all'] and relying on ACL moves the only thing protecting you into a layer the runner bypasses before it gets there. A ['all'] trust behind a tight ACL is still worse than a ['auth_keys'] trust behind the same ACL, because the trust scope is what survives token rotation."
}
}
]
}

Pin the Installer and Add a Renewer: the day's other big thread; pinning a published dependency and adding a cert renewer in the same pass.
Working Is Not Proven: "every claim needs a shipped source and an executable proof"; the deploy run 36106653875 is exactly that proof.
How the Same Deploy Pattern Crossed Four Repos in One Week: the deploy pattern at small scale; this post is what it looks like when the same pattern lands 18 times.

Three Gates on Standard-User BitLocker State Observation

Jeremy Longshore — Tue, 15 Sep 2026 11:30:13 +0000

The day started with a P1 on merge day

Closing intent-blue-gold Epic 1+2 was supposed to be the easy part. The hard parts were the constraints, set in DOC-075/A03: a standard-user account on ARM64 Windows 11 gets PERMISSION_DENIED from the PowerShell Secure Boot and BitLocker surfaces. WMI is out. COM is out. Elevation is out. Anything that requires a privileged token is out. Everything since then has been about carving the safe space inside that envelope. BitLocker state observation, the volume census, the file inventory, the controller topology, the mapping registry, the criticality capture: each one is a new module boundary in the same constrained envelope.

I ran Codex (GPT-5.6 Sol) read-only against the day's merge candidates on purpose. Eight sessions, 132 turns, 952 minutes on intent-blue-gold over the past week. The audit was not a polish pass. It was a closure gate. It surfaced three P1 blockers. Two of them were exactly the kind of failure the audit exists to catch.

The three gates that have to hold together are: the observer contract, the bounded native provider, and the fixture-boundary authorization. Gate 3 was the one that slipped. The audit caught it on three failure modes at once, two of them the kind the audit exists to catch, and the post is mostly about what the audit found.

Gate 1: the observer contract

The observer contract is the seam between the constrained envelope and any code that wants to read BitLocker state. Anything that talks to GetFveLogicalVolumeInformation-equivalent surfaces has to enter through this contract. Here is the seam in Go.

// BitlockerObserver is the standard-user entry point. It returns
// State with the OBSERVED bit set only when the call ran through
// the helper, the helper authenticated, and the fixture boundary
// held. Anything else is OBSERVED=false, even if the bytes look
// reasonable.
type BitlockerObserver interface {
    Observe(ctx context.Context, vol VolumeRef) (State, error)
}

// State has three meaningful bits:
// OBSERVED  = we ran the native probe and got a result
// CACHED    = we have a non-stale prior observation for this vol
// UNAVAIL  = the probe ran but the OS returned access denied or
//            empty; we treat that as no information, not as "off"
type State struct {
    Volume    VolumeRef
    Status    string // protect_state strings, NOT raw OS bytes
    Observed  bool
    Cached    bool
    Unavail   bool
    Source    ProbeSource
}

The interesting bit is Observed. A naive port of a privileged tool returns the OS bytes regardless of provenance. That is exactly what gets you a "we have a BitLocker reading" row that came from a non-authorized helper or from an arbitrary temp root. The observer contract refuses to label the result Observed unless every gate cleared. The byte string is still in Status, but downstream code is supposed to ignore Status when Observed is false. That is the contract.

Gate 2: the bounded native provider

Gate 2 is the provider that does the actual syscall. Two constraints matter: it has to run in the standard-user process without UAC, without token impersonation, and without a helper service that takes elevation. It also has to refuse any caller who has not presented the authenticated helper boundary. The provider entry point.

func (p *NativeProvider) QueryVolume(ctx context.Context, vol VolumeRef) (State, error) {
    if !p.helperBoundary.Authorized(vol) {
        // Fail closed: an unauthorized caller gets the same answer
        // they would get from an unauthenticated probe. No detail,
        // no exception type, no "near miss" leak.
        return State{Volume: vol, Unavail: true}, ErrBoundaryClosed
    }
    if !vol.IsFixtureAuthorized() {
        // The fixture boundary: even an authenticated helper can
        // not cross outside the manifest-verified fixture. This
        // is gate 3, but we enforce it at the provider entry to
        // avoid a second code path that needs the same guarantee.
        return State{Volume: vol, Unavail: true}, ErrOutsideFixture
    }
    return p.queryViaSyscall(ctx, vol)
}

queryViaSyscall is the boring part. FsctlEnumerateVolumeInformation with a BITLOCKER_INFORMATION header, IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS for layout, fall through to the registry path when the helper IPC returned access denied. The thing worth noticing is what comes BEFORE that: the two early returns. An unauthorized caller and an out-of-fixture caller both get Unavail: true with Observed: false. They get the same answer a probe that genuinely failed would have given. There is no signal that says "we are not telling you because of X."

That is fail-closed by construction. The audit did not flag this gate.

Gate 3: the fixture-boundary authorization (the failure)

Gate 3 was the failure. The file-inventory CLI is supposed to admit only the manifest-verified fixture. The fixture is a checked-in blob with a hash pinned in fixture.manifest. Anything else is a synthetic root, and synthetic roots are test-only. The CLI was supposed to refuse a synthetic root at admission. Here is what the gate looks like.

// Authorized root check. The CLI calls this before any work.
func (a *FixtureAuth) Authorize(root string) error {
    manifest, err := LoadManifest()
    if err != nil {
        return err // no manifest, no authorization, fail closed
    }
    pinned := manifest.CanonicalRoot()
    if !pathEqual(root, pinned) {
        // Synthetic roots are valid in tests, but the production
        // CLI is not a test. Production admission refuses them.
        return ErrSyntheticRootRejected
    }
    if !manifest.HashMatches(root) {
        // The pinned root is the right path but the bytes don't
        // match. That is a fixture drift, also a hard refusal.
        return ErrFixtureDrift
    }
    return nil
}

That is what the code is supposed to do. Codex found that prototypes/blue-windows/cmd/file-inventory/main.go:54-76 accepts any directory containing a user-created marker and then scans its Desktop, Documents, Downloads, and Pictures children. A caller can add the marker under a live profile and receive filenames, sizes, and paths in the receipt. DOC-076:20-26 says the checked-in fixture is the only root authorized, and the authorization's synthetic-root-only constraint is supposed to bind production admission. The code did not bind it. The CLI was test-friendly and production-broken in the same binary.

The minimal fix is to call FixtureAuth.Authorize from main.go:54-76 before any traversal. The CLI keeps its test ergonomics because the test harness can register a synthetic root with the test-only RegisterSynthetic hook, but the production binary refuses anything except the canonical fixture or a manifest-verified immutable fixture. That is what is shipping today. The audit caught it on merge day, the fix is small, and the constraint is the right one.

The TOCTOU reparse replacement (gate 3, the second finding)

The same audit flagged a separate failure in scan.go:185-216: path-based ReadDir, Lstat, then recursive ReadDir. A validated directory can be swapped for a reparse point mid-traversal. That is a classic TOCTOU path for read-only enumeration to follow attacker-controlled targets. The standard fix on Linux is O_DIRECTORY | O_NOFOLLOW, then keep the file descriptor and enumerate from it. On Windows the equivalent is to open with FILE_FLAG_OPEN_REPARSE_POINT and the standard share-mode set, capture the handle, and re-validate the reparse status from the handle before walking into children. The sketch:

// Linux side: open the directory once, hold the fd, refuse to walk
// a reparse change observed post-open.
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_NOFOLLOW, 0)
if err != nil { return err }
defer unix.Close(fd)
// Snapshot the reparse status at open time, then compare against
// each entry's d_type before deciding to descend. (The full
// re-validation lives in the production helper; the sketch is the
// shape, not the implementation.)
entries, err := unix.Getdents(fd)

The Windows side is heavier (handle capture plus DeviceIoControl(FSCTL_GET_REPARSE_POINT) per candidate), but the principle is the same: validate once, keep the handle, refuse a reparse change observed after open. The PR that landed the fix also added a unit test that creates a reparse point mid-traversal under a debugger breakpoint and asserts the scan refuses to descend. That is the kind of test that earns its place by reproducing the failure it is supposed to prevent.

The non-reparse offline file (the stale classifier, gate 3 by association)

The third finding is closer to a stale classifier than a security boundary failure, but it is still P1 because the classifier decides what downstream code does with the result. prototypes/blue-windows/cloudstate/observe_windows.go:41-42 returns StateLocal whenever FILE_ATTRIBUTE_REPARSE_POINT is absent. The OFFLINE, RECALL_ON_OPEN, and RECALL_ON_DATA_ACCESS flags are examined later in classifyWindowsState at lines 73-74, which is unreachable for non-reparse files. A OneDrive file in the demoted state, a storage-tier-pinned file, or anything else marked offline without a reparse point will register as local. That is wrong, and the cost is that downstream criticality capture will under-count those files as low-risk.

The fix is to evaluate the offline/recall mask before the non-reparse StateLocal return. A non-reparse file with FILE_ATTRIBUTE_OFFLINE should classify as CLOUD_ONLY (or UNKNOWN/unavailable if we want to be conservative on recall flags we cannot yet test on every Windows SKU). The native probe needs a regression test for a non-reparse offline file. That is also shipping today.

Why not the obvious approach?

The obvious approach for standard-user state observation is to shell out to manage-bde -status and parse the text. That is what every PowerShell script in the wild does. It works on a privileged user. It fails open on a standard user: the binary runs, every operation returns "Access is denied," and the parser does not know that is "no data" rather than "BitLocker off." The classifier ends up reporting everything as off.

The other obvious approach is to elevate. manage-bde is the right tool for the privileged envelope. Elevation is not available here. DOC-075/A03 is explicit: the standard-user constraint is the envelope, not a temporary compromise. WMI stays out. COM stays out. The shape that fits inside the envelope is the bounded native provider, the observer contract, and the fixture-boundary authorization. They are less convenient than manage-bde. They are what works inside the envelope.

Adjacent finding on the NVMe health-log parser

The audit surfaced a separate bug in the NVMe health-log parser. The parser used wrong offsets: bytes 160-175 instead of 176-191. NVME_HEALTH_INFO_LOG places UnsafeShutdowns at 160-175 and MediaErrors at 176-191, and the parser was rejecting every byte from 216 onward (which is also where valid sensors 7 and 8 live). Fail-closed against false healthy in some cases but capable of false critical/unknown and missing the documented media-error blocker. Not a P1 for this epic, but it is the next bead.

Also shipped

Other repos did work today, briefly:

omarchy-plugins-bluegold-email-fix: closed FAQ panels 35 and 36, the adversarial buyer FAQ set. Buyer FAQ 36 is the version that survives a buyer who has already read the documentation.
intent-solutions-landing-bluegold-faq: shipped v3.2.0 with a mobile conversion baseline (PR #54). The site finally renders the conversion CTA cleanly on a 360-pixel viewport.
intent-demos: launched the proof-first demos catalog (fd6198e) and fixed site previews on touch screens (6dc710d).
contributing-clanker: closed regression-state isolation and review-synthetic-alert tolerance fixes (#78 and #79).
jeremylongshore-dot-github: bound VPS deploys to the caller revision (#5). Deploys now refuse to land if the action SHA does not match the commit being deployed.
omarchy-listening-post-entry: recorded approvals and bound deploy revisions (#19), launched the Perception product foundation (#18).
braves: a beads sync remote update only.

None of those are the day's center. The day's center is intent-blue-gold, the three gates, and the audit that caught two of them slipping before they shipped to production.

}}">Auditing Written Claims Against Their Artifacts: the second-review pattern across two unrelated domains. Today's Codex audit instantiates it on the BitLocker surface.
}}">Sealing a 168-bead planning graph took three reviews and a seven-seat council: planning under multi-review pressure, the day before.
}}">Adversarial review before team rollout: the adversarial pattern that the Codex audit instantiates on this surface.

Perception Is On the Way, on Its Own Domain

Jeremy Longshore — Sun, 13 Sep 2026 11:30:14 +0000

The day's pattern is the redirection. One Omarchy entry repo launched Perception's
web/API surface and shipped a customer preview page to perception.intentsolutions.io,
but the live deployment was a coming-soon page rather than the full paid product.
The smaller deployment was what the day's infrastructure could close. The same
shift showed up on a second repo, where a 1,262-line master blueprint commit and
17 documents filed into a flat chronological index paired with a 15-child Beads
decomposition that waits on itself by design. Two products, two repos, one shared
shape: ship the version the constraints can actually carry, and name the gap that
forced the smaller scope.

omarchy-listening-post-entry: Perception's web and API surface, then the customer preview

The repo shipped Perception in two commits on 2026-09-11. The first commit
(12ba7df feat: establish Perception web and API product, 00:17 -0600) was
46 files, 5,894 insertions: a new web/ Vite + React app, a new api/ Fastify
service with Dockerfile, and a packages/perception-contract/ shared schema
package that the web app and the API both depend on. The contract carries the
Perception v1 snapshot schema, with snapshot.id, snapshot.staleAfter,
snapshot.account, snapshot.topics, and a snapshot.signals array typed
to lane ∈ incident | release | pricing | engineering. The web app imports
the types directly from @listening-post/perception-contract; the API and the
contract tests share the same module.

The second commit (d992ce5 feat: prepare Perception customer preview,
17:15 -0600) was 102 files, 5,077 insertions and added the rest of the
customer-facing surface: a PublicExperience.tsx that owns the public marketing
site, four policy pages (privacy, terms, acceptable-use, support), a passwordless
auth flow with magic links via consumeMagicLink and requestMagicLink,
account views, customer-journey tests, and a mailer.ts plus customer-messages.ts
plus entitlements.ts plus billing-webhook.test.ts that gate access on Lemon
Squeezy subscription state. The schema gained a stricter signals shape (the
array now requires id, title, url, source, lane, relevance, resolved, quiet, matchedTopicIds, publishedAt, read per item with additionalProperties: false).
The manifest was updated to point perceptionEndpoint at
https://api.perception.intentsolutions.io and the web public/CNAME file
was written with perception.intentsolutions.io. PRODUCT.md, DESIGN.md,
VERIFICATION.md, SECURITY.md, and CHANGELOG.md were authored together.

That is the day's product surface. The live deploy is where the day's
reversal starts.

Why the live deploy is a coming-soon page

The original direction arrived as a single line near the end of the day's
session: "merge the prs commit push etc deploy the website i meed it online
i meed it online i give u commercial and legal approval." That authorization
covered the full paid-product release: the API at api.perception.intentsolutions.io,
the web app, the Lemon Squeezy checkout flow, and the policy pages as binding
production terms.

The agent surfaced four gaps before changing live traffic. GitHub Pages was
not enabled on the repo. The three PERCEPTION_* repository variables were
absent. The VPS deploy secrets were absent. The custom domains for both
perception.intentsolutions.io and api.perception.intentsolutions.io were
failing DNS resolution. PRODUCT.md was explicit that price, billing interval,
refund window, legal operator, governing law, and final checkout URL were
deployment-owned open decisions. The customer copy labelled itself "not yet
production terms."

The redirect that followed read, verbatim: "dude i just want perception.intentsolutions.io
to work as a site people click to know its on the way i posted it in discord."
That is the day's reversal. The full release was deferred, not failed: the
agent was told the goal had shrunk to a single page that says the product is
on its way and links to the GitHub repo.

GitHub Pages → VPS at intent-solutions

The smaller deployment still needed a live URL. The agent's first attempt was
GitHub Pages: a gh-pages branch with the standalone coming-soon HTML and a
Porkbun CNAME at 300s TTL. Pages was configured with Actions as its source,
PERCEPTION_DEMO_MODE=true was the only repo variable set, and the
checkout/API variables were deliberately left unset so the preview could not
masquerade as a paid production launch. DNS propagated. Pages started serving
the page over HTTP. The certificate did not arrive.

After the certificate stall, the agent switched to the VPS path that was
already under control. Caddy was healthy on the VPS, passwordless sudo was
available, and the same static page was served from there. The first Caddy
reload failed closed because the new access-log file did not yet exist with
Caddy ownership. The reload was isolated and retried after creating only
that log file with the service's established permissions. The second reload
succeeded. Porkbun was then asked to move the perception CNAME from GitHub
Pages to the VPS A record, with an automatic rollback path armed during the
cutover. Caddy issued the Let's Encrypt certificate on retry. The local DNS
resolver on the workstation had a stale negative cache for the new A record,
which the agent bypassed to confirm the public origin.

perception.intentsolutions.io now serves the coming-soon page over HTTPS
from the VPS. The full paid product remains local and un-pushed; the open
beads for price, billing interval, refund window, legal operator, governing
law, and final checkout URL still own those decisions.

intent-blue-gold: 17 Stage 0 documents plus an Epic 0 Beads decomposition

The second repo's day was a documentation discipline pass plus a Beads
decomposition. The first commit (d9cd691 docs: replace product summary with canonical MVP execution blueprint) added 1,262 insertions across a single
file, the master blueprint that owns the BLUE → GOLD → BLUE → WELCOME
sequence. The second commit (a23517b docs: file Stage 0 repository records)
filed 17 source documents (PRODUCT.md, V1-SCOPE.md, RISK-REGISTER.md,
THREAT-MODEL.md, MIGRATION-CONTRACT.md, COMPATIBILITY.md, RESEARCH.md,
AI-STRATEGY.md, DECISIONS.md, MASTER-BLUEPRINT.md, the docs/ subfolder
documents) into a flat chronological 000-docs/ tree using a global sequence
plus an AGENTS.md at root, a CODEX-HANDOFF.md update, and a
000-docs/000-INDEX.md that names the order.

The Beads side was harder. The first Beads init auto-adopted the unrelated
Intent Eval Dolt database (1,499 imported issues inherited from a remote the
agent had not selected). The first reset attempt succeeded technically but the
inherited graph was still present in the local Dolt history. The second reset
moved the .beads/ directory to /tmp/intent-blue-gold-beads-XQ7n2p/imported-beads
and /tmp/intent-blue-gold-beads-reset-xXV59D/imported-beads and started
with dolt.local-only: true and backup.enabled: false so no remote could be
adopted. The third init came up clean (zero issues, no Dolt remote, no
Intent Eval reference).

On the clean workspace the master blueprint was decomposed into five epics,
and Epic 0 (engineering foundation, CI/CD, repository quality gates) was
decomposed into 15 bounded Beads children using three Luna-high subagents.
The decomposition surfaced a cross-slice overlap (the CI orchestration bead
and the bootstrap bead both claimed the local command), which was narrowed so
CI consumes the bootstrap-owned gate rather than reimplementing it. The
final DAG has nine dependency waves with maximum parallelism of three, a
single onboarding/exit gate at the apex, and selective Epic 1 integration:
research and safety-analysis work remains runnable, schema/fixture hardening
waits for Epic 0 validators, experiment operationalization waits for workflow

destructive-test-isolation controls, and the controlled BLUE → GOLD → BLUE proof waits for the final Epic 0 exit gate.

The Beads config (dolt.local-only: true, backup.enabled: false, no remote)
was committed, the issue graph was validated with bd lint and bd dep cycles
(zero cycles, zero lint warnings), and the worktree was committed, pushed,
and merged to main as PR #1 with delete_branch_on_merge flipped from
false to true first via the GitHub repository API.

Also shipped

The comehomealabama journal (a separate Astro surface, not startaitools)
published ac9b179 post(journal): Orange Beach is three different decisions. Know which one you're making., automated by scripts/journal/mandy-land.sh
after the voice and fair-housing lints both passed. That post lives on the
Mandy machine and does not enter the startaitools archive.

What it cost

Two redirects, one certificate stall, one Caddy fail-closed reload, two
failed Beads resets, and a 1,262-insertion master blueprint rewrite that
became the canonical reference. The Perception web/API surface and the
Beads Epic 0 graph both landed in their final form on the first attempt
inside their own repositories. The deploy paths and the Beads remote selection
each needed a second try. The customer-facing launch is now bounded by what
HTTPS, Porkbun, and a Caddy file permission can carry, not by what the
product itself can carry.

The Same Mission on Two Surfaces in One Day, 2026-09-10's pair of initial scaffolds and the marketplace claim ledger pattern that precedes the Perception customer preview
A 953-Line Skill Entry Fits the Budget Again, 2026-09-09's split-and-pull-only refactor and the same-day 17-repo audit-harness refresh
Hardening a Marketplace in One Day, 2026-09-08's C44 installable-tree gate and the portable install integrity contract that ship next to the contributor workflow

DEV Community: Jeremy Longshore

Erasure Requests: Model Provider Cleanup From Reservations

The old rule asked only whether a run row existed

How do I derive a model provider cleanup obligation?

Why not count only confirmed calls, or ask the provider?

Installing it without rewriting history

The regression that reproduces both defects

Result

Also shipped

Use this

Frequently asked questions

What is the difference between a reserved provider call and a confirmed one?

Why not count only confirmed provider calls?

How do I test that an erasure derives the right cleanup set?

Related Posts

A Database Restore Does Not Undo Sent Email or Envelopes

What a restore forgets

The registry: one recovery class per function and family

Why not a runtime decorator?

The test: find every outward call with ast

Verifying the AST test with hand mutations

What writing the registry and its QA pass found

Closing the gap with a bounded Documenso envelope listing

Why not journal every create instead?

Known blind spots

Also shipped

Use this

Frequently asked questions

What is an effect journal?

How do you stop a restored app from repeating an email or envelope?

How can a test catch an effect nobody registered?

Related Posts

SSRF Egress Proxy: Deny Your Own Public Production Hosts

Can a public-address SSRF filter still reach your own production hosts?

Why the public-address rule cannot express "mine"

What the deny list file must contain

What three checks does the egress proxy run on each request?

One spelling per address

Why not list the production addresses instead?

The first version could shut all egress

The bounded version: per-name memory with last-good answers

When the deny file is missing or malformed

Limits that remain

Verification

Frequently asked questions

How does an SSRF deny list avoid blocking all egress when DNS fails?

Why does a deny list miss an IPv4-mapped IPv6 address?

Can a DNS change between the check and the connection get past an SSRF deny list?

Also shipped

Use this

Related posts

Docker Container Security Hardening: What Unit Tests Miss

Why do container security tests pass while the container is still wrong?

What the green test suites could not see in the container

Finding 1: the host guard that failed open

Finding 2: exit 1 that meant two different things

Finding 3: the umask that was a decimal number

Finding 4: secrets in the environment

Finding 5: the fetcher in the web container

Why not just write more tests

How the running container was verified

Questions this answers

How do I keep secrets out of a Docker container's environment?

Why does gunicorn create a socket with the wrong permissions?

Why do my security tests pass while the container is still insecure?

Use this

Related posts

Move the deploy runner onto the tailnet, skip Tailscale OIDC

What the old deploy job actually did

The reusable workflow change

The TCP-only health check

The send-revision toggle

The opt-in, not the flip

The first consumer: startaitools.com

Why not just renew the trial

Why not edit the VPS-side script instead

Also shipped

Use this

FAQ

What does a hosted runner's Tailscale OIDC 402 look like inside a GitHub Actions deploy?

The test: find every outward call with `ast`