Gate the Payload Before You Pick a Runtime

Most local-versus-cloud arguments are settled by taste. This one gets settled by four gates and a test file.

The failure repeats across teams. A feature ships with prompts pointed at a remote endpoint. Weeks later someone asks where the payload went. The answer is spread across three files and nobody's memory.

Speed is the wrong place to start. Order beats numbers. Gate the payload first, then let the surviving tasks argue about milliseconds.

Gate one: residency

Classify the data before you classify the model. Four buckets cover most systems: public, internal, confidential, regulated.

Confidential and regulated never leave the host. Not for a demo. Not for a benchmark. Not because the endpoint feels fast this week.

This is a code-level guard, not a compliance program. It stops the accidental leak. It does not stop a determined one.

Gate two: survivability

Ask what the feature does when DNS fails. If the honest answer is "nothing", the task belongs on the machine.

Offline capability is a product decision, not an ops detail. A router with one reachable branch is not a router.

Gate three: budget measured from cold

Latency arguments usually quote warm numbers. Users hit cold numbers. Measure cold start before comparing anything, and put it inside the budget instead of beside it.

A 400 ms cold start against a 300 ms budget is not a networking problem. It is an architectural mismatch, and no retry policy repairs it.

Gate four: recovery

Ask how the call fails and who retries it. A task with no retry path and no offline fallback gets exactly one shot at the network. Give that shot to the local runtime.

Now the code. Four gates, applied in order, no dependencies.

# router.py
from dataclasses import dataclass
from enum import Enum


class Route(str, Enum):
    LOCAL = "local"
    CLOUD = "cloud"


DATA_CLASSES = {"public", "internal", "confidential", "regulated"}
LOCAL_ONLY = {"confidential", "regulated"}


@dataclass(frozen=True)
class Task:
    name: str
    data_class: str            # public | internal | confidential | regulated
    must_survive_outage: bool
    p95_budget_ms: int         # what the user is promised
    cold_start_ms: int         # measured on your wire, not copied from a blog
    retryable: bool


def classify(t: Task) -> Route:
    if t.data_class not in DATA_CLASSES:
        raise ValueError(f"undeclared data class: {t.data_class!r}")

    # Gate 1: residency. The wire never sees these.
    if t.data_class in LOCAL_ONLY:
        return Route.LOCAL

    # Gate 2: survivability.
    if t.must_survive_outage:
        return Route.LOCAL

    # Gate 3: cold start counts against the budget.
    if t.p95_budget_ms < t.cold_start_ms:
        return Route.LOCAL

    # Gate 4: no retry path, no second chance.
    if not t.retryable:
        return Route.LOCAL

    return Route.CLOUD

The gate order is the whole design. Swap gates one and three and you will happily route regulated text to a fast endpoint on a good day.

The decision table below is what the function should produce. The cold_start_ms values are illustrative fixtures, not measurements of anyone's network.

data class	outage-safe required	p95 budget vs cold start	retryable	route
regulated	no	5000 vs 400	yes	local (gate 1)
public	yes	5000 vs 400	yes	local (gate 2)
public	no	300 vs 400	yes	local (gate 3)
internal	no	3000 vs 400	no	local (gate 4)
public	no	3000 vs 400	yes	cloud

One row does the real work here. The last line is the only traffic that earns the network.

Tests that encode policy

The router is only useful if a pull request cannot quietly weaken a gate. So the tests assert decisions, not timings.

# test_router.py
import pytest
from router import Task, Route, classify


def task(**overrides) -> Task:
    base = dict(
        name="fixture",
        data_class="public",
        must_survive_outage=False,
        p95_budget_ms=2000,
        cold_start_ms=400,
        retryable=True,
    )
    base.update(overrides)
    return Task(**base)


def test_regulated_never_reaches_the_wire():
    assert classify(task(data_class="regulated")) is Route.LOCAL


def test_offline_requirement_beats_a_generous_budget():
    assert classify(task(must_survive_outage=True, p95_budget_ms=9000)) is Route.LOCAL


def test_cold_start_can_blow_the_budget():
    assert classify(task(p95_budget_ms=300, cold_start_ms=400)) is Route.LOCAL


def test_retryable_public_work_goes_remote():
    assert classify(task()) is Route.CLOUD


def test_missing_retry_path_pins_to_local():
    assert classify(task(retryable=False)) is Route.LOCAL


def test_undeclared_class_fails_loudly():
    with pytest.raises(ValueError):
        classify(task(data_class="whatever"))

Run it with pytest -q. Six assertions, no network, no model. It runs in a CI job that costs nothing.

The last test matters more than it looks. An undeclared class should crash the build, not default to public. Silent defaults are how payloads leak.

The whole suite also gives you a regression signal. When someone adds a new feature, they must name a data class. That naming step is the review you actually wanted.

Where a shared remote endpoint wins

Some tasks survive all four gates. Those are predictable, retryable, and boring to run yourself.

Batch classification over public text fits. So do backfills, first-draft generation, and anything with a p95 budget measured in seconds rather than milliseconds.

That is the traffic I hand to MonkeyCode's free model access and free server option. The router decides, the free tier absorbs. Sensitive payloads never arrive, because gate one rejected them three lines earlier.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not quoting a token allowance in this post. Free-tier limits move, and a stale number in a tutorial is worse than no number. Check the current limits on the project's page before you depend on them, and treat free capacity as a measurement environment first.

The practical benefit is narrow and real. You get to run gate three against a live wire without buying anything. That is a measurement, not a migration.

Who should not use this

If you cannot measure cold start, gate three is fiction. Fix the instrumentation before you trust the router.

If your team has no agreed data taxonomy, stop. The router will encode an argument, not a policy.

If the feature needs sub-100 ms interactivity, treat it as local-only and skip the debate. Same for anything with a hard real-time contract.

If you have no local capacity at all, the router will send everything to LOCAL and your feature will simply be down. Gates describe intent. Capacity decides whether intent ships.

And keep the scope honest. The classifier is a guard against careless routing, not a legal opinion, and not a fix for prompt injection, verbose logs, or vendor retention. Those are separate controls with separate owners.

Run the four gates on your own traffic this week. Write the answers down before you write the prompt. If you want live numbers for the cloud branch, the free tier is the cheapest place to collect them.