I’d like to take time out and discuss one of the core fundamental systems behind Campfire. Its entire Asset generation pipeline.
Campfire is an Agentic system that builds entire games with system prompts. Imagine going back and forth with a few prompts and then having it build everything you need, art, code, game design the whole shablam.
For a system like Campfire to even exist, we need to be able to take every human's basic inputs (A beginner will never describe in as much detail as is required) and turn that into an actual full game loop. And a full game loop not only requires base assets but you also need animations which are like 5-10 different images of the same base asset. Multiply this between enemies, tilemap, UI, sky, low level and high level decorations and you really need a robust system to reliably get base sprites and animations.
Over the last few months we’ve taken a great deal to really go over every possible process to see what produces the best results. This is really one of our most researched divisions as it's so essential to what we need Campfire to be so reliably. A means for everyday people to express their creativity without the limitations of our old crude learning practices of becoming an entire game developer to build a game.
When it comes to generating sprites and animations using AI there are 2 known methods:
- Using a Video generation model like Wan1.7, Grok etc.
- Using a Image generation model like Images 2 GPT or NBPro
I will go on to break down both of the known methods and some of the things that we have done to produce reliably enough results in Campfire. Everything labelled below has taken months of research and development through various testing and consultations between experts in this division.
Image Generation:
An image generation model is a model that generates static single png files. It can create remarkably well images especially in pixel art which is what most 2D platformers (currently all Campfire supports) choose as their art style. There is a known problem of “Mixels” which is where if you really zoom in you’ll see that the Image generation model is generating FAKE pixels but to the normal eye it's genuinely not noticeable. We’ve chosen to ignore this problem in Campfire because it's just a mere “nice to have”. The reason it generates mixels is because it's taught to generate not in specific pixels but in “whole chunks of the image” and there really is no work around to getting an image generation model to generate specific pixel perfect images for you.
As for sprite sheets, image generation has increased exponentially to be almost perfectly style consistent when generating different frames for an animation inside a singular png of a specific character. The problem does not lie with style, rather pure reasoning logic. In a non loop animation, say a sword swing, an accurate animation sequence would be: frame 1 ( idle), frame 2 (rising up), frame 3 (about to attack), frame 4 (climax of attack), frame 5 (cooling down), frame 6 (back to idle). A sort of incremental sequence of movements. However most often then not AI hallucinates this process, you get unnecessary frames like a frame 5 that instead of cooling down just points the sword up randomly. Or an issue where it may get the motion correct but it decides to switch the hand in which the character is attacking or just makes the sword disappear.
Another issue related to image generation for sprite sheets is that it fails to keep each frame inside a specific box and you result in “frame spillage” where frame 3’s sword tip falls into frame 4’s box. And besides that issue there is also the obvious problem (if not the hardest problem) of the character jumping and jittering around inside its “Dedicated canvas zone”.
A proper animation sequence requires a known canvas size to draw each frame in relative to a specific anchor point.
The point being when you first see the result it looks promising however when you try to take that result and actually use it in an animation by just removing the background you’ll see minor inconsistencies like this that ruin everything. And because of this you need a lot of post processing methods, some labelled below:
- Chroma Key Removal
- Cut to image bounds
- Anchor Point Alignment + creating your own unified canvas throughout all the animations and inter animation frames.
You can negate the lack of reasoning by increasing your prompt description to give frame by frame explanations but of course as any generative AI you will never get 100% perfect results.
GOOD RESULT VS BAD RESULT IMAGES
Good Sprite Sheet Results:
Bad Sprite Sheet Results:
As for a singular sprite, like a character's “Base” asset which is what we use to provide as reference for all the animation generations. The image generation model does a remarkably good job with the caveat that it fails to reliably produce a “solid flat base” for the character. A 2D side scrolling platformer requires a character with a solid leveled base. When you are generating the asset with chromakey as the background (the only way to properly remove the background from the generated sprite) the character generated is essentially “floating” within this chroma background which results in a subtle tilt on the character. What that looks like is essentially a: right foot lower than left foot type thing (just subtle enough to cause a problem):
You can try your best to fix it with prompt enforcement but as mentioned before you won’t get 100% results. The way we’ve understood to solve this is by enforcing a “Chromakey base line” in the same color of the chromakey background differed slightly so it’s visible but still easily removable. This enforces the character to have a solid base and works reliably well:
A few examples:
The only major restriction with generating using an image model is, regardless of what you try to do it will never manage to get a loop cycle animation. Suppose a run cycle or a swim cycle or anything that is supposed to loop. You can try to increase your results by providing skeletal pose annotations of the animation sequence but its still extremely unreliable and theres the fact that you have to create them specific for the type of character (e.g chibby proportions etc). The right way to handle loop animations is to go off singular sheet generation entirely and look into video generation.
Example of using Image Generation for run animation:
Video Generation:
This is by far the only way that we’ve managed to generate loop animations. A 3 second video describing a run cycle actually generates roughly 90 frames (depending on your settings) which is definitely more than what you require. Inside these 90 frames the character probably loops multiple times. And you wont have any “inter frame alignment” issues because its a video that has been generated coherently nor will you have terrible reasoning for the same benefit.
It seems like a perfect solution you just cut the middle frames e.g 30 frames and write an algorithm to determine the loop cycle.
*However what seems perfect on the top can only truly be understood inside post processing. *
To ensure a clean background removal we must add a chromakey background to this generation request, however during the animation the color of the chromakey mixes with some of the outer edges of your character within some animation frames so the character's ‘boot’ looks greenish in frame 12 and normal in frame 14.
This inconsistency in color doesn’t break the illusion of a smooth looping animation but it's noticeable enough for someone to point out “why does my character turn green when it starts running”.
Besides this, the biggest problem is probably specifically related to pixel art. Video generation models are not trained on pixel art animation, they're mainly trained on real life video. Because of this they make very mushy pixels during the animation. If you use the right model however, from the perspective of the user the mushness is as less noticeable as the mixels are. As for the color leakage we currently have not implemented any solution for that but thoughts do come to mind by creating a unified color palette using the base asset provided as reference along with the normal frames without color leakage and using that to fix the color leakage frames. We will test this and will update you on how it went or any other solution we ended up using.
What's the biggest problem Campfire has with using AI generation for base sprites and animations?
As for image generation, base sprites are generated remarkably well as mentioned. The main problem is with the** post processing singular canvas + anchor point for animation generation.** To help you understand let me take an example of a character's true animation requirements:
Let us picture a “Lumberjack with a gun” player. The basic animations required are as follows: Run, Shoot, Jump, Idle, Death, Reload
Not only do we have to run post processing and remove the jitter between a singular animation sequence (shoot for example known as the “inter animation frames alignment” but we also have to normalize a singular canvas and anchor point so that all other animations (already inter-aligned) can be aligned with each other.
A transition from Idle - Run cannot put the character on the right or move it up within the same canvas otherwise the transition isn’t smooth right?
What is the proposed solution to fix this?
Well it seems obvious what we must do to fix this, we must determine a canonical anchor point for the character itself. Something distinct that we can reference throughout every single animation (swim, run, jump, death, custom animation 1) whatever. Let us suppose when we are generating the base asset we specify a singular distinct anchor point. Something like the users right boot or a ghost's tail. Now when generating the animation sheet we reference this ‘anchor point’ inside the “Frame by Frame Description” and on the top level of the prompt: “Keep the character's right boot locked on the same anchor position throughout all frames”.
This doesn’t promise the character doesn’t jitter. What this promises is that the character's right boot looks the same (shape and color) throughout all frames wherever they choose to spawn inside its dedicated canvas space on the singular sprite sheet. Then we can use basic pattern matching to align all the frames. *And since we’ve agreed that this is the *“canonical anchor throughout this specific character” a jump, idle, shoot all share the same “right boot locked rule” so we can pattern match animation sequence 1 (idle) with animation sequence 2 (shoot) and it works amazingly. Quadruplets or characters like a bat or whatever imaginary character can also follow this method. The point is enforcing that every character or asset that we will generate has a “distinct” part of it that is easily recognizable.
The limitations to this method:
This is when there is a clearly distinct part of a character. What happens when the character is a slime blob? What happens when in the last frame of a death animation the character collapses entirely? How do you enforce a “lock right boot at the same place” while your frame by frame description says: "Collapse the character entirely as if he’s lying down”. And what happens when the character leaves the ground? Inside the raw generated sprite sheet he physically moves inside the canvas and the animation requires the boot to change into a more “pointed” way.
We’ve tested and we’ve gotten these “inconsistencies” to be caught best we can through ample prompt principles + other special case fine tuning methods but it's never 100% perfect.
Rest all problems are merely an increase in prompting logic and fine tuning here and there in post processing to get more accurate results.
Other game assets
AI Image models reliably produce all the relevant aspects of a game, the UI, the tilemap, the decorations all while maintaining style consistency.
The way Campfire does it so well is we use a multi agent architecture which just means we have a singular planning agent that decides beforehand the entire experience of the game and the style and the feel and then we have other sub agents that can interpret this “Design spec”.
The hard part isn’t producing a single static image (UI, tilemap etc), it’s mainly in the animations due to the fact that we turn a singular image to up to 40 different images (4-5 different animations) of the same base image. Needing to be style + Incremental logic sequence consistent.
My conclusion
The fundamental principle of Campfire is giving the user enough proof of concept and providing them with reliably enough iteration tools for them to fall into the creative cycle. We prefer to give the user only WHAT they need to know. Campfire is not for game developers or technicals it’s for their counterpart: Everyday people. An everyday person doesn’t care about canvas alignment or animation consistency they just say: “I want a run animation for this zombie and also throw in a bite one”. 100% of the process Campfire has to handle because the users job as we believe is High level intent where they belong.
Generating using AI 100% is prone to hallucination. That's why we’ve preferred to make it easy to iterate on an undesirable output. It's definitely not perfect but the prospect is here that these models are only ever getting *cheaper and better. *
To generate an entire game's assets have gone from thousands of dollars or hiring someone to spend 100s of hours to mere seconds and less than 5 dollars per game. It's incredible what this can allow and enable.
The loose end really is in how you get animations to be aligned well enough on their own that you see no jitter at all or no out of the ordinary frame or frame leakage throughout all your animations for all your characters. There's a lot of moving parts and post processing steps that rely on each other and if one fails it doesn’t matter how good the other steps worked the animation feels off.
Top comments (0)