Treat Free-Server Eval as a Shadow Protocol Before It Shares a Write Key

Last month I let a free execution server replay an evaluation job that still carried a production write key. The model stream came back late, and the shadow worker applied that payload to the shared store anyway. Production then saw a committed key and skipped a different body that the planner still expected. Have you ever let a cheap canary plane borrow the same commit identity as the live path?

I keep replaying that order because the failure is not a slow model and it is not a flaky server. The failure is an architecture that treats free capacity as a harmless copy of the production plane. If that copy can still write, then it is not a copy anymore, is it really?

The invariant the common path drops

I want one invariant stated in plain words before I draw any boxes on the board. A shadow evaluation plane may observe live inputs, but it must not share commit keys with the serving plane. It also must not share queues or compensation topics, even when that makes diffs harder to read. The common path stamps the same idempotency key onto the free server job because diffs feel easier.

That shortcut is exactly what lets a late shadow completion bind a live write you did not intend. Would you still call it a shadow if its acknowledgement can move production state without a human check? I would not call it a shadow, because a second writer with a friendly name is still a writer.

Assumptions I am willing to defend

I am reviewing a design rather than reporting a load test, so these assumptions need to stay visible. Free model access and a free server option are availability choices for a canary, not a capacity contract. I do not know your quota, hardware, or how long that option lasts, and I will not invent those numbers. The planner can retry after a stall, and delivery to the execution plane is at least once.

Two planes that share a key namespace are one failure domain, even when their dashboards look completely separate. A free tier can reject, stall, or disappear without becoming a reason to spill work onto the live writer. I will treat any missing capacity number as unknown, rather than as permission to fan out blindly. If your measured limit arrives later, you can drop it into the fixture without changing the fence.

Where I place the canary

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am using the MonkeyCode project's free model access and free server option as the shadow plane in this review. That choice lets me exercise the fence without ever pointing the canary at the serving write path. If you remove the product from the story, the same invariant still holds for any shared execution tier.

How I want the data to move

I draw three planes and one fence, and I refuse to let any arrow skip that fence. The control plane admits a shadow job and rewrites its key before any model call starts. The model plane only proposes a payload, and the free server executes that payload against a shadow store. Nothing on that path should be able to append a record to the live commit log.

planner        control plane       model plane        free server       shadow store
   | admit           |                  |                  |                 |
   |---------------->| rewrite key      |                  |                 |
   |                 |----------------->| propose payload  |                 |
   |                 |                  |----------------->| execute         |
   |                 |                  |                  |---------------->| shadow write
   |                 | fence: reject live: prefix          |                 |
live commit log stays untouched unless prefix == live AND admitted_plane == live

violating order when the fence is missing:
1. control admits job-7 with key live:tenant:debit
2. model plane stalls after the proposal is buffered
3. client retries; free server accepts a second worker
4. late completion from worker A commits amount=10
5. worker B commits amount=12 under the same live key

Does that picture still hold if the free server may call the same tool URL as production? It does not hold, because a shared tool URL without a plane header is another shared write key. Have you seen a network address pretend to be an isolation boundary when the handler is shared? I treat that address as a door, not as a domain, until the handler checks the plane.

Failure domains I would not merge

I split failure by who can change state, not by which vendor logo happens to sit on the box. The model plane can stall, truncate, or return a late completion, and that should only invalidate a proposal. The free server can accept twice, die mid-write, or reject for capacity, and that should only touch the shadow store. The control plane can admit a duplicate when it forgets the rewritten key, and that leak is the dangerous one.

If I merge those domains, a capacity reject on the free server starts to look like a model failure. A retry then replays the same body into the live tool, because both errors were typed as timeout. Have you watched a retry policy follow the wrong plane just because the error string looked familiar? I have watched that happen, and the compensation topic then fired for a job the user never confirmed.

The review I rerun before I trust a diagram

I do not want a slide that says the shadow is isolated unless a fixture can fail the build. This simulator is a proposal fixture I can run locally, and it is not a benchmark of any kind. I am not claiming a throughput number, a model name, or a hardware shape I have not measured. I am staying on the design and the fixture, and I am not walking through deploy steps or a monitoring install.

1. Declare the key namespaces and the capacity bound as inputs, rather than as comments buried in a client.
2. Admit the shadow job only after the key is rewritten, and record the plane on the admission lease.
3. Inject a late model completion plus a duplicate delivery aimed at the free server under test.
4. Assert that no live key changes, and that a second admit with the same shadow key is rejected.
5. Fail the review if any event log line contains a live prefix after the shadow run has started.

'''Shadow-key fence fixture. Proposal, not a production client.'''
from dataclasses import dataclass

LIVE = 'live'
SHADOW = 'shadow'

@dataclass
class Write:
    key: str
    plane: str
    body: str

class Store:
    def __init__(self):
        self.rows = {}
        self.rejected = []

    def commit(self, write, admitted_plane):
        prefix = write.key.split(':', 1)[0]
        if prefix != admitted_plane or write.plane != admitted_plane:
            self.rejected.append(write)
            return False
        if write.key in self.rows:
            self.rejected.append(write)
            return False
        self.rows[write.key] = write
        return True

def rewrite(step, run):
    return f'{SHADOW}:{run}:{step}'

def late_duplicate_order():
    store = Store()
    key = rewrite('debit', 'run-19')
    first = Write(key, SHADOW, 'amount=10')
    late = Write(key, SHADOW, 'amount=10')
    leaked = Write('live:tenant:debit', LIVE, 'amount=10')
    ok1 = store.commit(first, SHADOW)
    ok2 = store.commit(late, SHADOW)
    ok3 = store.commit(leaked, SHADOW)
    return ok1, ok2, ok3, store

def live_untouched(rows):
    return all(not key.startswith('live:') for key in rows)

if __name__ == '__main__':
    ok1, ok2, ok3, store = late_duplicate_order()
    assert ok1 and not ok2 and not ok3
    assert list(store.rows) == ['shadow:run-19:debit']
    assert live_untouched(store.rows)
    print('fence_held', len(store.rejected))

python shadow_fence.py

I also want a property that does not depend on one scripted order in a single file. Generate permutations of stall, duplicate delivery, and capacity reject, then assert the live prefix never appears. If one permutation breaks that property, the design does not converge, no matter how clean the diagram looked. Would you really promote a design that passes one happy path and then fails the permutation test?

When I read the rejected list, I want every crossed-plane write to show up there, with its original key intact. A silent drop would hide the bug, and a rewrite of the leaked key would launder a live prefix into a shadow prefix. Have you ever fixed a test by normalizing the key before the assertion, and then missed the leak entirely? I keep the raw key in the rejection record so the next reviewer can see which plane tried to cross.

My load model is deliberately tiny, because the question is convergence under reordering, not a peak rate claim. I set capacity to one in-flight shadow job so a second admit must fail closed instead of borrowing serving room. You should replace that bound with your own measured limit, and you should not copy a number I refused to invent. If the free server starts rejecting for capacity, the control plane should queue or shed, not spill into the live plane.

The tradeoff I could not wave away

I could not wave this table away, because each row buys correctness with a different operational cost. I am not claiming a speedup, because I have not run a paired workload on a named model or machine. The denominator I care about is the count of event orders that violate the fence, not tokens per second. Acceptance rule: zero live keys may be written across the injected failure set, including a duplicate after a stall.

Choice	Latency effect	Correctness	Cost posture	What breaks
Shared write key on the free server	Fewer lookups	Shadow can bind live state	Looks cheap until replay	Late completion commits the wrong body
Rewritten shadow key plus a fence	One prefix check	Live prefix is rejected	Canary stays on free access	Diffs need a translation step
No free plane at all	Probes pay the serving path	Strong only if serving is fenced too	Higher for every review run	Failure injection gets skipped

What I would change next

I would stop letting the free server select tools by URL alone, without a plane check beside the key. I would require a plane header that the tool gateway compares with the key prefix before it accepts work. I would drop the request when they disagree, and I would also split compensation topics by plane. A shadow timeout must not enqueue a live undo, and an older run id must not join a newer comparison.

Should the gateway reject, replay, or compensate when the header and the key prefix disagree on a write? I want the gateway to reject, because replay would only amplify a client that is already confused. Compensation assumes a live write that the fence should have blocked before any side effect started. Would replaying a crossed-plane request ever make the invariant stronger than a hard reject at the gateway?

Who should leave this pattern alone

Do not use a free shadow plane as your only correctness proof when serving auth and data differ. Do not point the free server at production tools while you tell yourself that you are only testing prompts. Do not treat unknown free capacity as a queue you can fan out without some explicit admission bound. If a regulator requires an attested environment, a free shared server is the wrong domain for that evidence.

This review also will not tell you which model is smarter, because I am not publishing scores I did not measure. It will only tell you whether a late proposal from the free plane can still move a live key. If your serving path has a different retry budget, a green shadow run is not evidence that production will converge.

The order I want you to break

Which event order breaks the invariant for your system, and should that system reject, replay, or compensate? Take one shadow job, stall the model, deliver the execution twice, and sneak one live prefix into the batch. If the store accepts that prefix, the free server was never a shadow, and it was a second writer you forgot to fence. If you want a bounded place to run that fixture on free model access and a free server, keep the write key fenced.