I shipped a Chrome side panel that helps with "explain this page" requests, and for the first month the entire UX was: select text → click summarize → read paragraph.
It works. People use it. But the moments where it fails are loud and specific:
- A user pastes a screenshot of a broken UI and types "what's wrong here?" — there is no text to select, so summarize has nothing to summarize. They get back "I don't see any text content."
- A user wants to ask about a chart. Same problem.
- A user screenshots an error toast that disappears after 5 seconds. By the time they open the side panel, the context is gone.
- A user wants to ask "what does this button do?" — the text around it is generic. The visual is the actual signal.
The fix that worked for us: when the user pastes a screenshot (or drags one into the panel), treat it as the primary input. The model gets the image and a short prompt, not a forced summary of nonexistent text. Output an annotated overlay or a one-paragraph "what's visible" plus the answer.
Three implementation notes that took longer than expected:
Drag-and-drop into a side panel document is not the same drag-and-drop as a regular page. You need both the document-level handler and a fallback paste listener for users who Cmd-Shift-4 instead.
The model that summarizes text well and the model that reads images well are often not the same model. Routing by input type is a one-line change but the latency difference between vision and text-only models is real — users notice.
Once the panel accepts images, the upload UX has to stop pretending it's a chat. Users paste a screenshot expecting "what is this", not "uploading to your account" copy. We changed the empty state to "Paste a screenshot or ask anything" and the drop rate on first paste dropped.
The lesson I'd write on a sticky note if I were starting over: "Help me understand this" is a visual question more often than it is a text question. Build for the screenshot first. The text summarizer is the fallback, not the default.
Top comments (0)