This is a submission for the Google I/O Writing Challenge
At 12:00 PM during the Cerebral Valley Google I/O Hackathon, RepoProbe attached itself to a generated FastAPI repository that looked production ready from almost every conventional angle.
The container booted inside Google's Antigravity sandbox without instability. Docker compilation layers completed cleanly. The ASGI runtime mounted correctly. Health probes stabilized almost immediately. Gemini 3.5 Flash summarized the repository as a distributed inference backend coordinating asynchronous workers through Redis queues and MCP orchestration layers.
Nothing failed during shallow inspection.
The repository structure looked convincing enough that most engineers would stop investigating after the first few minutes. Route boundaries were separated correctly from worker execution paths. OpenTelemetry instrumentation wrapped request lifecycles properly. Retry handlers existed. Queue semantics looked believable. The logs looked believable too.
Then RepoProbe started replaying corrupted authentication traffic against the live runtime.
JWT timestamps shifted outside valid windows. Signature payloads were reconstructed with malformed byte ordering. Claims objects were intentionally truncated before replay. Several requests combined impossible cryptographic states that should have terminated execution immediately if verification logic actually existed underneath the middleware layer.
The responses barely changed.
At first the behavior looked like cache contamination somewhere inside the request path.
Syscall tracing exposed something worse.
During replay, the middleware never touched the descriptor associated with the verification key material at all.
No read boundary appeared against the mounted secret volume.
No epoll_wait occurred on the expected cryptographic dependency path.
request replay
↓
jwt.decode(... verify=False)
↓
broad exception handler
↓
HTTP 200 OK
expected syscall:
read("/run/secrets/jwt.pem")
observed:
nothing
The application surface resembled authentication closely enough that conventional inspection procedures accepted it as authentication. Kernel level activity showed no evidence that signature verification had ever occurred.
The Payment Pipeline Was Simulating Settlement State
Several hours later, another repository exposed what initially looked like a production grade financial reconciliation pipeline.
Settlement events propagated through asynchronous queues correctly. Internal transaction state transitioned through believable lifecycle stages. Retry handlers activated during simulated webhook failures. The API emitted realistic transaction identifiers following Stripe formatting conventions closely enough that aggregation systems indexed them naturally during replay.
Packet inspection showed the runtime never established a successful outbound connection to any payment provider.
The orchestration layer generated synthetic settlement continuity locally while replaying reconciliation progress internally through its own queue substrate. Socket state transitions revealed repeated connection failures against a nonexistent upstream target while the scheduler continued mutating local financial state as though confirmation packets had already returned successfully.
Distributed tracing reinforced the illusion because spans still reflected believable ordering semantics even though no external payment lifecycle existed underneath the orchestration boundary.
otel.trace.status = OK
worker.retry.count = 3
transaction.state = settled
queue.depth = 0
tcpdump:
SYN
SYN
SYN
timeout
Traditional observability tooling interpreted the system as healthy because the generated runtime continued producing structurally valid telemetry despite the absence of any successful network level settlement flow.
The MCP Runtime Failure Was Worse
The MCP orchestration graph failed differently.
Statically, the repository looked sophisticated enough to resemble a legitimate long horizon agent runtime. Tool schemas validated correctly. Context hydration initialized during startup. Capability negotiation exposed bidirectional streaming interfaces. Dependency graphs resolved without structural collisions during shallow inspection.
The failure surfaced only after concurrent execution pressure forced the scheduler into conflicting assumptions about ownership boundaries inside the orchestration graph itself.
One execution node permitted nullable asynchronous hydration during tool initialization while downstream branches assumed dependency resolution had already completed synchronously before delegation began.
Under concurrent replay, unresolved futures accumulated faster than the scheduler could unwind blocked execution paths.
Event loop starvation followed gradually.
Internal task queues stopped draining.
Several coroutine branches remained suspended indefinitely waiting for ownership resolution that no active execution path still controlled.
The process itself never crashed.
Health checks remained green because probe execution required almost no scheduler activity. OpenTelemetry spans continued streaming because instrumentation hooks emitted timing boundaries independently from runtime progress. The orchestration dashboard still showed active execution because state transitions were reconstructed from buffered queue metadata rather than from live coroutine advancement.
Thread inspection showed forward progress had stopped nearly forty seconds earlier.
scheduler loop:
active
health endpoint:
200 OK
otel exporter:
streaming
coroutine ownership:
deadlocked
queue drain rate:
0/s
Runtime observation
The orchestration layer still appeared operational because telemetry exporters were attached to queue metadata transitions rather than active coroutine advancement inside the scheduler loop itself.
RepoProbe Stopped Trusting Telemetry
After that point, telemetry stopped being treated as evidence.
Descriptor activity was traced directly against scheduler state. Network socket allocation was mapped against live coroutine ownership. eBPF hooks attached to syscall boundaries exposed execution stalls underneath the orchestration layer while the observability stack above continued reporting healthy progress semantics.
The infrastructure remained conversationally alive after the runtime underneath had already stopped advancing.
Early versions of RepoProbe relied heavily on the managed execution infrastructure introduced during Google I/O 2026. Long horizon investigation loops coordinated through persistent runtimes. Gemini synthesized intermediate reasoning continuously while hosted orchestration layers handled escalation dynamically across sandbox environments.
The infrastructure itself started degrading under hackathon traffic several hours later.
Quota exhaustion spread across hosted runtime environments. Sandbox allocation stalled unpredictably. Detached interaction sessions continued replaying cached execution summaries after containers became unreachable. Several orchestration retries severed synchronization boundaries internally between execution runtimes and the agent layers coordinating them.
One investigation session exposed the underlying failure clearly.
A managed runtime lost causal contact with its Antigravity sandbox after repeated orchestration retries detached the execution boundary internally. The sandbox itself had already stopped progressing, but the orchestration layer still retained buffered outputs generated during earlier execution cycles.
Gemini continued synthesizing coherent runtime summaries from stale tool traces while the underlying container no longer possessed active execution state.
[dead container]
↑
stale tool outputs
↑
orchestration runtime
↑
Gemini synthesis layer
Deterministic Verification Became Mandatory
Filesystem traversal, AST extraction, dependency reconstruction, syscall tracing, packet inspection, scheduler ownership analysis, queue replay, and runtime mutation were separated completely from the reasoning layer afterward.
Gemini only received finalized execution artifacts once deterministic runtime collection completed independently underneath it.
Without that separation, long horizon orchestration systems drift into recursive verification collapse surprisingly quickly. Generated summaries begin validating earlier generated summaries while buffered telemetry reinforces stale orchestration state long after the underlying runtime has detached from reality.
The most damaging failures rarely appeared as startup crashes.
One generated repository constructed a distributed worker topology combining Redis Streams semantics with local in memory scheduler fallbacks embedded directly inside the API process. Under concurrent replay, some execution branches treated queue ownership as distributed infrastructure while adjacent branches mutated shared state synchronously inside the web process itself.
Queue acknowledgements appeared successful from the orchestration layer perspective while scheduler inspection showed several tasks had never initialized correctly at all.
Another repository referenced an asynchronous execution framework that did not exist anywhere across PyPI, Conda, or GitHub. RepoProbe reconstructed a substitute execution chain heuristically by mapping unresolved import signatures against known queue initialization patterns observed across Celery, Dramatiq, and RQ based systems.
The repository booted partially afterward, but syscall replay later exposed incompatible assumptions between worker hydration state and API lifecycle ownership. Several execution branches attempted to mutate queue state before the underlying event subscribers had attached descriptors to the active scheduler loop.
The runtime continued exposing healthy telemetry while orphaned execution paths accumulated underneath the queue substrate silently.
What Google I/O 2026 Actually Changed
Antigravity sandboxes reduced the operational cost of isolated execution dramatically.
Persistent orchestration runtimes normalized long horizon agent workflows.
MCP execution graphs standardized tool coordination layers.
Gemini 3.5 Flash reduced reasoning latency enough that continuous orchestration became economically practical at enormous scale.
The bottleneck shifted somewhere else entirely.
Generation stopped being expensive.
Verification became expensive instead.
Not syntax verification.
Not whether the container boots successfully.
Not whether telemetry appears healthy during shallow inspection.
The expensive problem now is proving that execution semantics still preserve causal truth once scheduler ownership, cryptographic boundaries, descriptor activity, queue hydration, packet synchronization, runtime mutation, and concurrent replay begin interacting simultaneously under live execution pressure.
Because many of the repositories RepoProbe investigated did not fail loudly.
They remained operationally persuasive long after the runtime underneath had already stopped being real.
Top comments (0)