I created this piece of content for the purposes of entering the All Things Agentic Hackathon.
Over eight days, my teammate Muntaser and I built Gatehouse for the All Things Agentic Hackathon: an autonomous vendor-lifecycle fleet on Google Cloud. One document enters through a guarded HTTP door; a three-agent review fleet grounds findings in temporally valid evidence; a risk gate approves; an enablement agent recalls the findings from Memory Bank and onboards the vendor with training conditioned on the exact gaps the review found. Every step lands as a content-addressed node in a cryptographic ledger, and the committed golden review replays offline (model and database provably unreached) to the same seal digest.
Here's the thing I want to write down while it's fresh, because it's the lesson I'll reuse forever:
Across the entire build, the agent logic never failed in production. Not once. Every single cloud failure was an identity or addressing bug: a who or a where, never a what.
If you're deploying agents on Google Cloud (ADK, Agent Engine, the GEAP stack), these are the five walls we hit, in the order we hit them, each one formatted the way I wish someone had written it for me: the error, the wrong theory, the real cause, the fix.
Bug 1: Where does the model live? (The 404 that wasn't missing)
The error: 404 NOT_FOUND calling gemini-3.5-flash from a perfectly configured regional Vertex AI setup.
The wrong theory: model name typo; API not enabled; billing problem.
The real cause: endpoint topology is bimodal. Gemini models and embeddings serve on the global endpoint. Platform services (Agent Engine, Memory Bank, Firestore, Model Armor) are regional. Same project, same SDK, opposite routing.
The fix: GOOGLE_CLOUD_LOCATION=global for anything that touches the model; explicit regional locations, in code, for platform services.
The kicker: Model Armor's gcloud surface has the same split, but fails differently: without a regional endpoint override it returns PERMISSION_DENIED. A permissions error that is not a permissions error. We lost an hour to IAM theories before finding it. Bake the override into your scripts and never think about it again.
Bug 2: Who is GOOGLE_CLOUD_PROJECT? (The database that "didn't exist")
The error: first deployed run of the fleet on Agent Engine: The database (default) does not exist for project 126304062315. We had been querying that database, from the same repo, all week.
The wrong theory: the deploy didn't stage our packages; Firestore client version mismatch.
The real cause: Agent Engine's ambient GOOGLE_CLOUD_PROJECT is the project number. Firestore's default-database routing wants the project ID. Locally, your env says the ID, so the bug is invisible until the first real deploy.
The fix: never trust a numeric project env. We resolve through our own variable with a guard:
def _project_id() -> str:
p = os.environ.get("GATEHOUSE_PROJECT") or os.environ.get("GOOGLE_CLOUD_PROJECT", "")
return p if p and not p.isdigit() else "gatehouse-hackathon"
Three lines. It ended a class of bug permanently.
Bug 3: Who is the code, actually? (Your credentials are lying to you)
The error: same deployed fleet, one layer deeper: PermissionDenied: 403 on Firestore's run_query.
The wrong theory: the project-ID fix didn't take.
The real cause: every Firestore call for the previous five days had run as a human. My Application Default Credentials, with Owner. The deployed engine runs as its own service agent (service-<project-number>@gcp-sa-aiplatform-re.iam.gserviceaccount.com), which had platform permissions and nothing else. Dev credentials mask identity bugs perfectly until the first deploy, and then they all arrive at once.
The fix: one grant: roles/datastore.user to the reasoning-engine service agent. The durable lesson: before you deploy, ask who will this code be when it runs? Then grant that principal, not yourself.
Bug 4: Is anyone home? (The container that answered with nothing)
The error: after a tool-thread crash, every subsequent request to the engine returned an instant, empty, successful stream. 200, zero events, one second. For hours.
The wrong theory: our streaming client; the dispatcher; cold start.
The real cause: the crashed container was wedged. A dead process still answering the phone. Nothing in the response says "I am broken"; it just says nothing, politely.
The fix: an in-place redeploy (adk deploy --agent_engine_id <same-id>) cycles the containers without touching any wiring. The monitoring lesson: for streaming agents, zero events in under N seconds is a health signal, not a result. Alert on it.
Bug 5: The bug that fixed itself (retries are the autonomy)
The first time the full lifecycle ran end to end, the enablement engine had just been deployed. Pub/Sub delivered the vendor-approved event; the cold engine answered 429 Too Many Requests. Three times.
And then the fourth retry completed the first fully autonomous lap of the system, intake to onboarding, zero humans, while we watched the logs.
That wasn't luck; it was two boring decisions made earlier: the dispatcher acks only after the fleet run completes (so failures redeliver), and subscriptions carry 600-second ack deadlines (so long agent runs aren't falsely redelivered). At-least-once delivery plus honest acks turned a cold-start outage into a self-heal with receipts. If I could keep one design rule from this project: the retry design is the autonomy. An agent system that only works when every component is warm isn't autonomous; it's a demo with good weather.
The part that made all of this bearable
Every bug above was diagnosed from evidence, not guesswork, because the system writes everything down. The two libraries doing the writing, pollard (the content-addressed evidence ledger) and chronofy (temporal validity), are Muntaser's own open-source projects, and both predate the hackathon. What we built this week is Gatehouse itself: the fleet, the lifecycle, and the first production agent system running on top of both. Every model call and tool call becomes a ledger node. Retrieval passes a validity gate, so an 18-month-old pen test decays below threshold and gets pruned with a re-acquisition finding instead of silently trusted. Runs are sealed with a rolling SHA-256. Telemetry streams to Cloud Trace content-free: ids and digests, never documents. The failed runs sit in the trace explorer right next to the successful ones. Observability caught what a premature ack would have swallowed.
The end state: a judge, or you, can clone the repo and re-run our real Gemini review offline, zero credentials, with the model and Firestore provably unreached, landing on the identical seal digest:
pytest --pollard-mode=replay
The closer I can't resist
My day job is AI enablement: teaching people to work with these systems. Gatehouse's whole differentiator is that the training it generates is conditioned on what the review actually found. The reviewer flags weak MFA on a legacy tier, and the onboarding module that gets written is literally "MFA Setup for the Legacy Tier," citing the finding.
Debugging this project taught the same lesson from the other side. Generic guidance ("check your permissions") burned hours. Grounded, receipted specifics ("this exact principal lacks this exact role, and here is the node id that proves what was attempted") fixed things in minutes.
That's true for agents. It's true for the people using them. Build systems, and training, that cite their evidence.
Gatehouse was built by Katherine Burge and Muntaser Syed for the All Things Agentic Hackathon. Live console: try submitting a poisoned document at gatehouse-intake-cbk2rg5qgq-uc.a.run.app. Stack: Gemini 3.5 Flash, Google ADK, Vertex AI Agent Engine, Cloud Run, Pub/Sub, Firestore vector search, Model Armor, Memory Bank, Cloud Trace.
This post was created for the purposes of entering the All Things Agentic Hackathon.
Top comments (0)