A general model can take an image plus a question and return segmentation, bounding boxes, and object detection, work that used to require its own trained vision system. Use those boxes only after your code checks the schema, the coordinates, and that every box has a label. Fine-tuning can raise how often the model is right. The check in front of the boxes stays.
Logan Kilpatrick, product lead for Google AI Studio and the Gemini API at Google DeepMind, spelled out that shift on Chain of Thought episode 12 in February 2025. Paige Bailey, AI developer relations lead at Google DeepMind, described a later case on episode 47 in January 2026: a general model running vision, speech, and physical actions on a small open robot.
How long did one custom vision problem used to take?
On the order of nine to twelve months, in a successful case. Kilpatrick had lived that timeline as a machine learning engineer: collect the data, train the model end to end, put it in production, and build the infrastructure that kept it fault tolerant.
By that conversation, a vision-language model could do image segmentation, bounding boxes, and object detection before fine-tuning. Fine-tuning can help bring a use case to the reliability you need. Early-stage founders and business owners who lacked a team of machine learning PhDs can call a model API and try the vision use case. Months of data collection and end-to-end training dropped out of that path. The production bar remains.
the amount of time it would take to solve one of these, like, domain specific computer vision problems was on the order of nine to twelve months in a successful case. Like, you had to you had to go and collect all the data. You had to train the model end to end. You had to put that thing in production.
Logan Kilpatrick, product lead for Google AI Studio and the Gemini API at Google DeepMind, on Chain of Thought ep 12
The Making of Gemini 2.0: Deep… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts
Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · February 12, 2025 · 41m
podcasts.apple.com
Why did so few multimodal deployments reach production?
Kilpatrick's read was that multimodal AI was basically not in production, with very few successful deployments, while text deployments were already common. Early multimodal use cases were starting.
Two piles of work were left. Scaffolding is one. Inability to see and interact the way a person does is why current scaffolding exists, and a vision-capable setup could simplify it. Separately, the next multimodal agentic layer needs new scaffolding, and building it takes time. The other pile is knowing whether the system works. Of people who had already put AI in front of users, he bet that at least five out of ten would not have a good eval story and would not know what success looks like for what they shipped. He expected that lag to show up again for the multimodal agentic case, with the product surface ahead of the evals and the ops underneath it.
Define the success check before the first user sees a box. A returned list of boxes is a candidate, not a passing result.
I bet you at least five out of 10 of those folks would not have a good eval story.
Logan Kilpatrick, product lead for Google AI Studio and the Gemini API at Google DeepMind, on Chain of Thought ep 12
How do you check bounding boxes before you use them?
Reject the response unless every box sits inside the image, has a positive width and height, and carries a label. Pass the boxes to drawing, cropping, or an action only after that check holds.
Schema-check any structured output you take from the model, whether masks, boxes, or detections. The sketch below covers boxes. A coordinate past the frame, a zero-area box, or a missing label fails the answer, even when the surrounding text sounds confident. Store the image width and height with the request so the check has a bound. Keep the same gate if you fine-tune later. A bad structure should still be refused.
You still own the task definition, the refusal path, and the measurement. Choose which labels count, what happens when the check returns false, and which production images get scored against a known answer. Write that eval before you widen the rollout. Start with a small set of images that already have known boxes. Count schema failures apart from wrong labels.
By January 2026, Bailey was showing this class of model past a JSON reply. Stanford's open Pupper robots use a Raspberry Pi and Gemini APIs for vision, with Gemini Live supporting text to speech and robotic actions such as shaking hands or following someone. Validate the perception output before the action call.
A minimal sketch to adapt, not a drop-in library.
import math
def _finite(value):
if isinstance(value, bool) or not isinstance(value, (int, float)):
return None
if not math.isfinite(value):
return None
return value
def boxes_ok(boxes, width, height):
width, height = _finite(width), _finite(height)
if width is None or height is None or width <= 0 or height <= 0:
return False
if not isinstance(boxes, list) or not boxes:
return False
for box in boxes:
if not isinstance(box, dict):
return False
label = box.get("label")
if not isinstance(label, str) or not label.strip():
return False
x, y = _finite(box.get("x")), _finite(box.get("y"))
w, h = _finite(box.get("w")), _finite(box.get("h"))
if None in (x, y, w, h) or w <= 0 or h <= 0:
return False
if x < 0 or y < 0 or x + w > width or y + h > height:
return False
return True
def ask_about_image(image_bytes, question, width, height):
raw = call_model(
inputs={"image": image_bytes, "text": question},
response_format="boxes",
)
if not isinstance(raw, dict):
return {"ok": False, "boxes": []}
boxes = raw.get("boxes", [])
if not boxes_ok(boxes, width, height):
return {"ok": False, "boxes": []}
return {"ok": True, "boxes": boxes}
The models can do image segmentation now, and they can do bounding boxes and object detection.
Logan Kilpatrick, product lead for Google AI Studio and the Gemini API at Google DeepMind, on Chain of Thought ep 12
FAQ
Do I still need a custom vision model for boxes and detection?
For segmentation, bounding boxes, and object detection, Kilpatrick said general models already do that work, before fine-tuning. Fine-tuning is what he pointed to when you need a higher reliability bar. Keep a custom model when your labels, cameras, or error budget fall outside what you have measured on the general one.
Why were multimodal launches behind text launches?
He said multimodal AI was basically not in production, with very few successful deployments, while text deployments were common. Vision-first agents still needed new scaffolding, and that would take time. His separate bet was that at least five out of ten teams already shipping AI would not have a good eval story.
What should fail a bounding-box response?
Fail closed when the list is empty or the wrong type, when a box has no label, when width or height is not positive, or when a box extends outside the image. Put that check in front of any UI or robot command that would consume the boxes.
Takeaway
- Treat segmentation, bounding boxes, and object detection from a general model as a candidate answer, then validate the structure.
- Budget fine-tuning when the use case needs more reliability than the base model gives you.
- Define what success looks like before users, or a robot, depend on the output.
- Score schema failures separately from wrong labels.
- Expect the scaffolding and the evals to lag the demo, the way Kilpatrick saw multimodal deployments lag text.
The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)