If you have ever built an image-to-image (I2I) consumer feature, you already know the dirty secret of prompt engineering: what works in a Cherry-Picked Twitter thread rarely survives contact with real user uploads.
A few weeks ago, I started building an AI Cartoonify effect. The goal seemed straightforward—let users upload a selfie or pet photo and transform it into a stylized 3D/2D cartoon while keeping the person recognizable.
At first, I tried throwing the reference image and a carefully crafted style prompt into a single generation pass. It was a disaster.
The "Identity vs. Stylization" Trap
Take a typical torture-test upload: a guy with a thick beard, wearing thin wire-rimmed glasses, standing under harsh side-lighting, holding up a peace sign.
When I ran this through a standard single-model I2I workflow, I kept hitting a brick wall:
- Failure Mode A (High Style Strength): The output looked like a gorgeous Pixar frame, except the wire-rimmed glasses melted into his eyebrows, his peace sign grew a sixth finger, and he looked like a generic stock character instead of the person in the photo.
- Failure Mode B (High Reference Conditioning): His face and glasses stayed intact, but the model refused to actually cartoonify the geometry. It just slapped a plasticky, airbrushed skin shader over a photorealistic face—straight into the depths of the uncanny valley.
The root cause? A single pass forces the model to figure out what is in the image and how to deform it artistically at the exact same time.
To ship this to production, I had to decouple semantic understanding from spatial rendering. Here is how I rebuilt the pipeline by pairing ChatGPT Image 2.5 with Nano Banana 2.1.
The Solution: A Two-Stage "Lock & Render" Pipeline
Instead of asking a single model to guess which pixels matter, I split the workload between two engines based on what they actually do best:
- Stage 1: The Semantic Lock (ChatGPT Image 2.5) — Used strictly as a visual parser and dynamic prompt compiler. Its job is to spot fragile details (like those wire-rimmed glasses or specific hand gestures) and convert them into hard text tokens.
- Stage 2: The Spatial Renderer (Nano Banana 2.1) — Used as the fast conditioning engine. It takes the original image as a structural blueprint alongside the dynamic tokens generated in Stage 1 to execute the actual style transfer.
How the Data Flows in Code
Rather than relying on static prompt templates, the backend builds a custom conditioning payload for every single upload. Here is a simplified look at the TypeScript structure driving the orchestration:
interface CartoonifyJob {
sourceImageUrl: string;
targetAesthetic: '3d-animation' | 'cel-shaded' | 'flat-illustration';
}
interface ConditioningPayload {
spatialReference: {
imageUrl: string;
edgeFidelity: 'strict' | 'balanced' | 'loose';
compositionLock: number; // 0.0 to 1.0
};
semanticTokens: {
identityLocks: string[]; // Extracted via ChatGPT Image 2.5
lightingCues: string[]; // Preserves mood without keeping raw photo noise
styleModifiers: string[]; // Drives the Nano Banana 2.1 shader
guardrails: string[]; // Anti-uncanny-valley negative tokens
};
}
async function buildCartoonPipeline(job: CartoonifyJob): Promise<ConditioningPayload> {
// Step 1: Extract fragile micro-features using ChatGPT Image 2.5
const analysis = await extractVisualAnchors(job.sourceImageUrl, {
model: 'chatgpt-image-2.5',
focusAreas: ['eyewear', 'facial_hair', 'hand_pose', 'garment_necklines']
});
// Step 2: Assemble the hybrid payload for Nano Banana 2.1
return {
spatialReference: {
imageUrl: job.sourceImageUrl,
edgeFidelity: 'balanced',
compositionLock: 0.70 // Enough room to exaggerate eyes/proportions
},
semanticTokens: {
identityLocks: analysis.detectedFeatures,
// e.g., ["thin gold wire-rimmed glasses", "trimmed full beard", "two-finger peace sign"]
lightingCues: [analysis.primaryLightingDirection, "soft rim light"],
styleModifiers: [
"stylized 3D character render",
"expressive stylized proportions",
"clean subsurface scattering"
],
guardrails: [
"photorealistic pores",
"hyper-detailed skin wrinkles",
"melted eyeglass frames",
"asymmetrical pupils"
]
}
};
}
3 Lessons Learned While Tuning the Models
Moving to a dual-model setup solved the macro architecture, but getting consistent outputs across thousands of edge cases took some serious trial and error.
1. Let the Reference Image Handle Macro, Let Keywords Handle Micro
Nano Banana 2.1 is fantastic at keeping macro composition intact—head tilt, shoulder angle, and background silhouettes stay rock solid when using a reference image. However, when you relax the reference lock enough to allow cartoon exaggeration (like slightly larger eyes or smoother jawlines), micro-details vanish. By having ChatGPT Image 2.5 explicitly inject tokens like "thin gold wire-rimmed glasses" into the prompt stream, Nano Banana 2.1 is forced to redraw those micro-details cleanly in the new cartoon style rather than blurring them away.
2. Strip Out Real-World Lighting Noise
User photos usually have terrible lighting—fluorescent office bulbs, harsh bathroom shadows, or blown-out window backlighting. If you rely purely on image conditioning, the cartoon output inherits those muddy color casts. By extracting only the directional intent of the lighting (e.g., "warm side-lighting") via keywords and letting Nano Banana 2.1 re-render the actual color palette from scratch, the output looks like it came out of an animation studio rather than a dark bedroom.
3. Kill Skin Texture Early
The #1 reason AI cartoons look creepy is leftover high-frequency skin texture. When a model puts stylized cartoon eyes on a face that still has realistic pores and nasolabial folds, the human brain immediately rejects it. Hard-coding anti-realism guardrails (photorealistic pores, raw skin texture) into the Nano Banana 2.1 conditioning step was the single biggest quality unlock.
Test It Out in the Wild
I have shipped this exact two-stage pipeline (ChatGPT Image 2.5 + Nano Banana 2.1) into production.
If you want to see how it handles tricky portraits, pets, or weird lighting—or if you just want to try breaking the semantic lock with your own photos—you can play with the live implementation here:
👉 Try the Live Tool: AI Cartoonify on VioEvo
Wrapping Up
As image models get faster and more specialized, chaining them together is becoming much more practical than waiting for one "god model" to do everything perfectly in a single shot. Using a heavy semantic model to write dynamic conditioning instructions for a fast spatial renderer has completely changed how I approach I2I features.
How are you all handling identity retention vs. style strength in your own generative pipelines? Drop a comment below if you've found other workarounds!
Top comments (0)