Last week I almost shipped an image description generated by a model. It was clear, concise, and wrong in a way that should have been impossible: it described a disabled Save button as enabled and invented a validation error I had already removed. The screenshot showed neither thing. The model did not know that. It answered anyway.
That moment changed what I ask for. I stopped asking is this alt text good? and started asking which parts of this screen did the model never see? The second question gives you a work queue. The first just gives you an opinion.
Alt text is a boundary problem, not a writing problem
A static screenshot can tell you some things directly. It can show a visible label, a disabled state, a filled input, or an error summary. It cannot show what happened before the screen, what the code state is behind an element, or a validation error that is not visible at that moment.
When a model writes alt text and answers questions outside that boundary as if they were visible, that is the failure I care about. It is not bad writing. It is a fabricated screen detail that a human may later treat as fact.
To make this cheap to replicate, I moved the probe onto MonkeyCode, an open-source project that advertises free model access, a free server option, and a 30M-token allowance. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat those numbers as vendor claims, not independently measured facts, and I did not use any paid quota for the exercise.
The six-question blind-spot interview
I run the same six questions after every generated screen description. Each question is tagged with whether a static screenshot can answer it.
| # | Question | Can a static screenshot answer it? |
|---|---|---|
| 1 | What interactive elements are visible and what are their labels? | Yes, when the text is legible. |
| 2 | Which buttons appear disabled? | Yes or unsure, depending on contrast and styling. |
| 3 | What validation error is visible under each field? | Only if the error is actually rendered in the shot. |
| 4 | What happened immediately before this screen? | No. |
| 5 | What state will the button submit next? | No. |
| 6 | What data-testid values are in the markup? | No. |
The useful part is not accuracy on questions 1 and 2. The useful part is catching a confident answer to questions 3 through 6 when the correct answer is not visible.
Example probe card:
Screen: account settings
Generated description: `Save button is enabled. The email field shows an error: email is invalid.`
Question 3: What validation error is visible under the email field?
Model answer: `email is invalid`
Reality: the screenshot has no error text.
Flag: fabrication-risk. A static screenshot cannot prove an absent validation state.
This is schematized, not a benchmark result. The point is the decision rule: if the question cannot be answered from the image, unsure is the correct answer. A definite answer is not a better description. It is a warning.
Use the free server as a probe bench, not a leaderboard
I did not ask the model to summarize papers or win a benchmark. I sent six short questions with the same screenshot and asked for Yes, No, or Unsure on each. Then I only scored the boundary violations.
That keeps the workload tiny. It also keeps the failure easy to explain. You do not need a fine-tuned model to catch the most dangerous mistake, because the mistake is not visual. It is the refusal to say I cannot tell.
A free server is enough for this because the compute is low and the output is a few lines. It is a rehearsal loop, not a production captioning pipeline.
The handoff flow
Screenshot
|
v
Model writes a description
|
v
Six blind-spot questions
|
v
Confident answer to an unanswerable question -> flag
|
v
Human reviews the flag list first
I would not put the human in charge of rewriting every caption. I would put the human in charge of deciding whether a flagged detail can ship. That is a much smaller job, and it is the part that actually changes the record.
Accessibility checks for the review card
If this turns into a real interface, do not hide the flags behind color.
- Use a visible text label such as
fabrication-risk, not only a red dot. - Announce the hidden question before the generated answer, so a screen reader user hears what the model could not know before hearing what it claimed.
- Make the
Yes / No / Unsureselector keyboard reachable. - If the approve button is disabled because a flag exists, say why. A disabled button without a reason is just a locked door.
The same rule applies to generated alt text itself: a screen reader user should not get a sentence that quietly turns an unseen state into a fact.
Limitations
This protocol catches boundary overreach. It does not catch a model misreading a visible label, because that is an accuracy problem, not an unseen-detail problem. It also does not turn accessible captions into compliant content; you still need a human review of language, context, and purpose.
A model that over-hedges will create fewer flags but may also hide useful visible detail. That is the tradeoff. The flag rate is not a trust score. It is only a list of places to check.
Who should skip this
Skip this if you already have deterministic UI metadata for labels, states, and validation text. In that case, do not generate alt text from pixels; generate it from the metadata and keep the visual description for decorative images only.
Skip this if you need to process thousands of images a day without any editing step. This is deliberately slow. It pays off in the records that matter, not in volume.
The part I edit first
I now keep a tiny list of unanswerable screen questions next to any generated alt text. When the model answers one anyway, that is the first thing I edit. Not the tone. Not the grammar. The part it could not actually see.
If a free token allowance can pay for that lesson on one screenshot, it is worth more than another page of benchmark numbers. Try the six questions before you trust the next generated caption.
Top comments (0)