One Idea, Seven Image Models: Building a Model-Aware Prompt Layer (With a Fallback That Never Breaks)

IPG Prompt Lab — Sat, 31 Oct 2026 03:47:59 +0000

Same subject, different wording per model: a small strategy-map pattern, plus a fallback for when the AI call fails

If you generate images with more than one model, you have probably hit this: a prompt that works well in one model comes out flat in another. A product-shot prompt that renders clean text in one model turns to mush in a second. A still that should feel like a frame from a video looks posed and static.

The cause is usually not the subject. It is the emphasis. Each model rewards a different kind of instruction: some want precise composition and text rules, some want motion language, some want structured, hierarchical instructions.

I ran into this while building a small prompt tool, and the fix turned out to be a pattern worth sharing on its own. I call it a model-strategy layer: a tiny lookup table that sits between "what the user wants" and "what the model is told". This post walks through it with real code from the project, including the part I like most: a deterministic fallback, so the tool still returns a usable prompt when the AI call fails.

The problem: one subject, many dialects

Take one plain-language idea:

a ceramic coffee mug on a wooden desk next to a laptop, soft morning window light

Now imagine sending it to seven different image models. The subject stays the same. What should change is the guidance around it:

One model is strong at real-world knowledge and text rendering, so you push precise composition and readable text.
Another is better at controllable edits and commercial design, so you push local refinements and layout.
A video-first model needs the still treated as a keyframe: action beat, motion direction, lens choice.
A model built for social content wants a bold, one-second visual hook.

Writing all of that by hand every time is tedious and easy to get wrong. So I encoded it.

Step 1: Write a strategy map, not seven prompt templates

The first instinct is to write a full template per model. Don't. The templates drift apart, and you end up maintaining seven copies of the same composition and quality rules.

Instead, keep one shared structure and a short strategy sentence per model. This is the actual map from the project (English side):

const MODEL_STRATEGY = {
  "GPT Image 2": "Prioritize real-world knowledge, accurate text rendering, precise composition, detail consistency, and polished final imagery.",
  "GPT Image 1.5": "Prioritize stable photographic realism, natural lighting, clear subjects, and safe compositions with minimal complex text.",
  "Nano Banana Pro": "Prioritize creative exploration, complex scenes, multi-image fusion, stylized visuals, and memorable compositions.",
  "Seedream 5 Pro": "Prioritize controllable edits, readable text, iterative image changes, local refinements, and commercial design output.",
  "Seedance 2.0": "Structure the prompt for both image keyframes and video storyboards; emphasize subject consistency, action beats, camera language, motion direction, and dynamic composition.",
  "Grok Imagine": "Prioritize trend-aware concepts, social shareability, strong visual contrast, and a clear one-second visual hook.",
  "Gemini 3 Pro": "Prioritize complex instruction following, multimodal reasoning, information structure, and long-context consistency.",
};

Each line is one sentence. You can review the whole map at a glance, and adding a new model means adding one line.

Step 2: Build the prompt from shared blocks

The shared structure is just a list of lines: subject, target model, strategy, aspect ratio, then the rules that apply to every model.

function buildLocalPrompt({ subject, model, ratio }) {
  const lines = [
    `Subject: ${subject}`,
    `Target model: ${model}`,
    `Model strategy: ${MODEL_STRATEGY[model] || "Create a high-quality visual result based on the model's strengths."}`,
    `Aspect ratio: ${ratio}`,
    "Composition: clear subject, readable foreground/midground/background layers, enough negative space, complete lighting and material details.",
    model === "Seedance 2.0" ? "Seedance keyframe note: write a single image as a video keyframe with a clear action moment, motion direction, lens focal length, and dynamic tension." : "",
    model === "Seedance 2.0" ? "Optional storyboard extension: for video, add 3-5 shots and specify framing, subject action, camera movement, duration, and transition for each shot." : "",
    "Text rule: if text appears in the image, keep it short, readable, and naturally integrated with the layout.",
    "Quality rule: no watermark, no extra distortions, no broken fingers, wrong perspective, messy background, or low-quality texture.",
    "Replaceable variables: swap [subject], [brand/person], [scene], and [text] with your specific need.",
  ];
  return lines.filter(Boolean).join("\n");
}

Two details matter here:

Conditional lines, not branches. The video-keyframe lines are only added for the one model that needs them. filter(Boolean) removes the empty strings. Nothing else forks.
Replaceable variables. The last line tells the person (or the next model) which parts are meant to be swapped. That turns a one-off prompt into a reusable template.

Here is what that looks like in the live tool when I pick Seedream 5 Pro for the mug example:

This output is plain, but it is structured, and that is the point. It is easy to edit, and every rule is visible.

Step 3: Let an LLM polish it, behind a thin Worker

A structured list is a good start. A flowing, natural prompt is often better. So the tool also has an AI path: a small Cloudflare Worker at /api/generate-prompt that sends the same ingredients to a chat model.

The Worker reuses the strategy map. Only the delivery changes. The system prompt is short and strict:

const system = [
  "You are the expert prompt editor behind Image Prompt Generator.",
  "Given the user's subject, target image model, and aspect ratio, produce one polished prompt ready to paste into an AI image generator.",
  "Output only the final prompt. Do not include explanations, Markdown, or a heading.",
  "Make it specific and actionable: subject, scene, composition, lens/camera language, lighting, materials, style, text rules, and quality constraints.",
].join("\n");

And the user message carries the same four fields the local builder uses: subject, target model, aspect ratio, model strategy.

A few defensive choices worth copying if you build something similar:

Origin allowlist. The Worker only answers requests whose Origin header is in an ALLOWED_ORIGINS list, and it returns 403 for anything else. That stops random sites from using your endpoint as a free prompt API.
Input clamping. The subject is trimmed, whitespace is collapsed, and it is capped at 3,000 characters. The model name is capped at 80 and the ratio at 20, with defaults (GPT Image 2, 1:1) if they are missing.
"Output only the final prompt." Without this line, chat models like to add "Here's your prompt:" and Markdown, which breaks copy-paste.
Bounded generation. max_tokens: 900 and temperature: 0.75: enough room for a rich prompt, enough variety to be useful, but not so loose that it starts inventing a different scene.

Step 4: Treat the fallback as a feature, not an error path

This is the part I would argue for in any AI-powered tool. The client code tries the AI path first and drops back to the local builder on any failure:

async function generatePrompt() {
  const localPrompt = buildLocalPrompt({ subject, model, ratio });
  try {
    const prompt = await requestGeneratedPrompt({ subject, model, ratio });
    render(prompt, "Generated by AI");
  } catch {
    render(localPrompt, "Generated locally");
    showToast("Live generation failed. Showing a local prompt instead.");
  }
}

Because the local builder and the Worker share the same strategy map, the fallback is not a degraded "sorry" message. It is the same prompt logic in a more literal format. If an upstream key expires, a rate limit hits, or the network drops, the person still leaves with something they can paste. The label tells them which path ran, so nothing is hidden.

The local prompt is built before the network call, so a failure costs nothing extra.

Step 5: Pair generation with real, attributed examples

A generated prompt tells you what to say. An example image shows you what the model actually did with similar words. The tool puts both on one page.

Below the generator there is a searchable prompt library. You can filter by model and category, sort it, and search by author, model or keyword. The data is split into one JSON file per model and loaded lazily with an IntersectionObserver as you scroll, so the page never has to download the full library at once. The current index lists 3,493 prompts across the models.

Each card keeps the author and source next to the image and the full prompt:

I think attribution is underrated in prompt libraries. Seeing who wrote a prompt and where it came from tells you its context: a product poster from a designer's thread is a different thing from a portrait study. It also lets you go back to the original collection. And if you want to reuse a card commercially, the site's FAQ asks you to check that card's source first.

A Try it button on the homepage cards copies a library prompt into the generator box, so you can change the model or ratio and see how the strategy line shifts.

Takeaways you can reuse

Encode model differences as one sentence per model in a strategy map, not as separate templates.
Build prompts from shared blocks + conditional lines. Only the model-specific bits branch.
Put the LLM behind a thin, allowlisted, input-clamped endpoint, and tell it to output only the prompt.
Make the deterministic fallback share the same logic, and label which path produced the result.
Show real examples with attribution next to generated output, so people can check the wording against actual results.

If you want to poke at the pattern live, pick a model in the Image Prompt Generator and watch the strategy line change while the rest of the structure stays put. It supports English and Chinese output, and the four aspect ratios (1:1, 4:5, 9:16, 16:9) feed straight into the prompt.

The pattern itself is not tied to images. Any time you target several models with one intent, whether that is text, code or video, a small strategy map plus a shared skeleton will save you from maintaining a pile of almost-identical templates.

DEV Community: IPG Prompt Lab