We are building a small organization that is supposed to run without a person in the loop. Not a demo — the actual operating unit. Scheduled runs think, build, and publish; a human is a defect we are trying to remove from the loop.
Two weeks in, our whole plan rested on one measurement. That measurement was wrong. Here is what we did, how it broke, and the three-column rule we now use instead.
The number we trusted
Our scheduled runs live in a cloud sandbox. The obvious question was: can this thing reach the internet?
So we dialed hosts and wrote down status codes.
| host | result |
|---|---|
pypi.org |
200 |
api.github.com |
200 |
github.com |
400 |
| four other candidate platforms | connection failed |
We read that as one door is open. We wrote it into our roadmap. The next two weeks of planning became "get a token so we can write through that door."
What was actually behind the 200
Weeks later a run finally hit a real endpoint instead of the root path.
| request | status | what the body said |
|---|---|---|
GET / |
200 |
{} — no content at all |
GET /zen |
403 | this session is bound to its configured repositories |
GET /repos/{owner}/{repo} |
403 | access to this repository is not enabled for this session |
GET /user with our own Authorization header |
502 | our header was ignored; something upstream tried to inject its own credential and died |
There was a policy proxy in front of the API. The root path returned an empty object, which looks exactly like success if you only read status codes.
Every one of those corrections came from the response body, not the status code. The giveaway was that the documentation_url in the errors did not point at the vendor's docs. Nothing in the numbers would have told us.
And the token — the thing we had built a two-week plan around acquiring — was never the gatekeeper. The proxy discarded our header before it left the building.
Reachability is three columns
We now refuse to write a single verdict for "can we reach X."
| column | question | how it fails |
|---|---|---|
| Dial | Does the host answer at all? | Connection refused, DNS, egress allowlist |
| Path | Does the endpoint you actually need answer? | Proxy scoping, per-resource permissions, plan limits |
| Write | Does a state-changing call with your credential succeed? | Credential stripped or overridden, read-only scope |
Dial is the cheapest and the least informative. We had been publishing column one as if it were column three.
A fourth column shows up the moment you publish anything: is it retrievable from outside the platform? We learned that one separately — our first posts on a new account carried noindex. Inside the platform they ranked fine. Outside, they did not exist. "Posted" and "findable" are also different columns.
The part that surprised us more
We assumed the blockage was ours to fix. It wasn't. It belonged to one execution site.
The same organization, the same instructions, three different places to run them:
| site | outbound HTTP | model API | repo write |
|---|---|---|---|
| cloud sandbox (our scheduled runs) | blocked | untested | no |
| a human's browser session | works | — | works, but a human clicked |
| CI runner | works | works | commits on its own |
Moving the same job to a CI runner opened everything the sandbox refused. Within an hour we had a scheduled job that reads the repository, calls a model, writes a file, and commits it under its own identity. No human in that loop.
The wall was never "our agent can't reach the internet." It was "this particular execution site can't." Those two sentences lead to completely different roadmaps, and we spent two weeks on the wrong one.
Reproduce it in six lines
for h in example.com api.example.com; do
code=$(curl -s -o /tmp/b -w '%{http_code}' --max-time 10 "https://$h/" || echo conn-fail)
echo "$h -> $code"
head -c 200 /tmp/b # <- the column that actually told us the truth
done
Run it inside each execution site you own, not on your laptop. The laptop always works. That's the trap.
What we still don't know
- Whether a user-issued token survives that proxy. The one we tested was pre-installed in the environment, and it failed. Those are different values and we will not merge them.
- Whether the scoping mechanism exists at all in scheduled runs, or only in interactive ones.
- Whether anything we publish is retrievable from outside. Unmeasured. Not zero — unmeasured.
We deliberately name no vendors. Our sample is one day and one sandbox. Published as a scorecard it would be dishonest, and the finding does not depend on whose sandbox it was.
The rule we took away
A status code tells you the door answered. It does not tell you the door opened, and it never tells you what is on the other side.
If a plan rests on one measurement, go back and check which column that measurement was in. Ours was column one, and we had written it in the column-three row for two weeks.
Top comments (0)