If you have ever wanted to ship an AI-powered application without managing GPUs, model servers, or scaling infrastructure yourself, this guide is f...
For further actions, you may consider blocking this person and/or reporting abuse
Does the "default to batch" rule still hold once tools are involved? Batch makes sense for classifying 100k tickets with plain model calls, but I can't picture how an agent that needs to call a Cloud Run tool mid reasoning would run as a batch prediction job.
Great catch Nazar! you're right, and I've reworked that section. The old "default to batch" line overreached; it now uses a "Now vs Later" test instead. Online if a live user needs the answer in seconds, batch for high volume data that can wait.
On your specific point: batch prediction jobs are single shot (input → one model call → output), so they can't host an agent loop that pauses mid-reasoning to call a Cloud Run tool, even when nobody's waiting. For tool calling agents at scale, the pattern is to queue items (Pub/Sub or Cloud Tasks) and run them through the online agent endpoint asynchronously with capped concurrency, or pre-fetch the tool data upfront so each call becomes single-shot and genuinely batch-eligible. Thanks for sharpening this.
Really solid breakdown of the separation between the Agent Platform and Cloud Run — the "let the model call the platform, not the other way around" framing in the security section is a clean mental model I haven't seen explained that clearly elsewhere.
I teach cloud and AI concepts to students moving into DevOps/cloud roles, and one thing I always emphasize is exactly this kind of separation of concerns — orchestration vs. execution — but most intro-level material glosses over why it matters until something breaks in production. Curious how you'd explain the "offline vs. online inference" tradeoff to someone who's only ever worked with synchronous API calls — is the batch/async mental shift the biggest jump for people newer to this pattern?
Thanks, Midhun and great question.
The analogy I'd reach for with sync-only students: online inference (The Synchronous World You Know) is a phone call, batch is sending a mail truck. With a phone call, you dial, wait on the line, and get your answer in the same connection; exactly like every REST call they've ever made (Request - Response pattern).
With batch/offline, you load 100k letters onto a truck, it drives off, and you get a delivery receipt later; you never "wait on the line" at all. The mental shift is that there's no response to your request, the "response" is a job ID, and the actual results land somewhere else (Cloud Storage) at some later time you don't control.
In my experience the async model itself isn't the biggest jump, most devs have touched a message queue or a webhook. The real jump is designing for it; realizing your code can't just await the answer, so you now need job status polling or completion triggers, idempotent retries, and a place to put results. In other words, the hard part isn't understanding batch/offline, it's letting go of the assumption that request and response live in the same breath. Once that clicks, the "does a human need this right now?" question becomes the natural router.
Good breakdown of splitting orchestration from app logic. The 'default to batch' line is a useful heuristic, though tool-calling agents obviously complicate it.
Thanks, Kevinbai! Agreed
Nazar raised the same point above, and it pushed me to rework that section.
hello
Hi Jimmy!
dev.to/magopredator