I maintain ImgKick, which includes an image-prompt catalog. While expanding it, we needed to avoid a misleading shortcut: treating the existence of an image as proof that a prompt works. This article was prepared with AI assistance from our implementation and test records.
Keep the source recipe separate from the run
A community recipe has an author, a source URL, a source model, a license notice and the original text. A test run adds a different set of facts: the text actually submitted, endpoint, quality, requested aspect ratio, output dimensions, optional reference and observations.
Those records should not overwrite each other. A recipe written for one model does not become native to another model because you display it on a new-model landing page. A prompt asking for 5:8 does not prove the API produced 5:8 when the request parameter was square.
We keep the imported recipes and our test records in separate JSON files, joined by a stable recipe ID. Here is a small illustrative validator for that pattern; it is deliberately narrower than our production checks:
def check_catalog(recipes, samples):
by_id = {}
for recipe in recipes:
key = recipe['id']
if key in by_id:
raise ValueError(f'duplicate recipe: {key}')
for field in ('prompt', 'source', 'source_model', 'license'):
if not recipe.get(field):
raise ValueError(f'{key}: missing {field}')
by_id[key] = recipe
for key, sample in samples.items():
if key not in by_id:
raise ValueError(f'unknown recipe: {key}')
if sample.get('tested_prompt') != by_id[key]['prompt']:
raise ValueError(f'{key}: adapted prompt needs its own record')
for field in ('model', 'quality', 'aspect_ratio',
'dimensions', 'image', 'observation'):
if not sample.get(field):
raise ValueError(f'{key}: missing test field {field}')
return {'has_output': len(samples),
'not_tested': len(by_id) - len(samples)}
The exact-text comparison is a policy choice for a catalog that presents a run as a test of the original recipe. If you support edited recipes, store a separate revision instead of relaxing the check and silently showing a different prompt.
Check the published asset, too
A valid JSON record can still point to the wrong thumbnail. Our build checks that the local file exists and matches its recorded SHA-256 hash. It also checks that the rendered text matches the source recipe and that source links remain present. An image hash establishes which file was published; it does not establish that the image is good, licensed or accurate.
For reference edits, link the actual input used in that run. Showing the output next to an unrelated input would defeat the purpose of recording evidence.
Give observations a place in the UI
One of our saved product-image runs contains recognizable branding despite a no-logo instruction. Another used square output for a prompt that requested a different aspect ratio. Both are useful examples of a run. Neither is a clean pass for every constraint.
That leads to different interface states: an untested recipe, an example with an output, and a note describing what the output did not satisfy. We do not have enough repeated trials to turn a single example into a reliability score.
A reader should be able to see the exact prompt, settings and observation together. Otherwise the gallery becomes a collection of attractive images with no reproducible connection to the text.
The same separation is useful in model demos and small evaluation datasets: source, execution and judgment answer different questions.
The live prompt catalog is the implementation context for this article. I operate that service; browsing the catalog is free, while image generation has limited free trials and paid credits.
Top comments (1)
The claim that a generated example is not a passed test deserves to become a schema constraint. I would keep
example,evaluation, andverified_runas different evidence types, with the verifier, environment, artifact hash, and timestamp attached to the last one. Then a UI cannot accidentally promote a plausible sample into proof just because both happen to contain an input and output.