DEV Community

Cover image for An AI Gave Me a Sprite Sheet. Here's Everything Wrong With It (and How to Salvage It)
Crane Zhou
Crane Zhou

Posted on Fully Autonomous

An AI Gave Me a Sprite Sheet. Here's Everything Wrong With It (and How to Salvage It)

You ask an image model for "a pixel art slime, jump animation, sprite sheet, 6 frames." Thirty seconds later you have something that looks exactly like a sprite sheet.

Then you drop it into your engine and it turns out to be a picture of a sprite sheet.

I make a sprite sheet cutter, and rescuing generated sheets is one of the jobs I built it for. Their problems are so consistent that I assembled one sheet with all of them at once, so I could show each fix cleanly:

A row of six green slimes on a bright green background, unevenly spaced, at different heights, with a small yellow spark floating above one of them

Looks fine, right? Count the problems.

  1. The background isn't transparent. It's green. Solid-looking green.
  2. It isn't even one green. Zoom in and every pixel is slightly different. Models paint texture; they don't flood-fill.
  3. There's no grid. The gaps between frames are 28, 13, 33, 21 and 35 pixels.
  4. The baselines wander. Each slime sits at a different height, and not because it's jumping.
  5. There's a loose spark floating above frame four that isn't attached to anything.
  6. The shadows are baked in, painted straight onto the background.

Each of those breaks a different assumption your engine makes. Let's go through them in the order you have to fix them.

Fix 1: remove a background that's almost one color

The classic approach is color keying: pick the background color, make every matching pixel transparent. Take the corner pixel, delete everything equal to it.

On a generated sheet, that removes almost nothing.

Top: with tolerance 0 the green background is almost untouched. Bottom: with tolerance 12 the background is gone and only the slimes remain

The corner pixel here is #20d455. The pixels around it are each a shade or two off in some direction. To your eye, identical. To an ===, strangers.

So you match by distance instead. Treat colors as points in 3D space and remove anything within a radius:

const dr = r - keyR, dg = g - keyG, db = b - keyB;
const close = dr * dr + dg * dg + db * db <= tolerance * tolerance * 3;
Enter fullscreen mode Exit fullscreen mode

A tolerance of 12 clears this whole backdrop. Push it too high and it starts eating the sprite, and a green slime on a green screen is the worst case for that. Which is why the second half of the trick matters more than the first.

Only remove background that's connected to the edge of the image. Start from the border pixels and spread inward through anything that matches. The fill stops when it hits the slime's dark outline, so greens inside the outline are never even tested. You can use a much more aggressive tolerance when the algorithm physically can't reach the inside of your character.

(The one thing this won't clear is an enclosed pocket, like the gap between a character's legs. That needs an "everywhere" mode, and then you're back to being careful with tolerance.)

Fix 2: find the sprites without a grid

With a real sheet you say "6 columns" and you're done. Here, an even 6-column cut slices through the middle of slimes, because nothing lines up with anything.

Instead, ask the image where the sprites are. Once the background is transparent, a sprite is just an island of non-transparent pixels. Flood-fill from each unvisited solid pixel, track the bounding box as you go, and you get one box per island.

Top: detection without joining finds 10 pieces, treating the spark and several shadows as their own frames. Bottom: with a join distance of 26 pixels there are 6 boxes, each containing a slime with its spark and shadow

The top row is the raw result: ten pieces from six slimes. The spark became a frame of its own, a tiny animation frame containing only a spark. So did every shadow that isn't touching its slime, which is all the ones where the slime is in the air.

The fix is one extra pass: merge any two boxes that are within N pixels of each other, and repeat until nothing merges. With a join distance of 26, the spark and the shadow get pulled into their slime's box. Six sprites, as intended.

There's a real judgement call in that number. Too small and a sword swing detaches from its owner. Too large and two neighbouring characters fuse into one wide frame. The right value is "bigger than the largest gap inside a character, smaller than the smallest gap between characters." On a sheet where those overlap, no algorithm saves you and you draw the boxes by hand.

Fix 3: make the frames the same size, and pick the right anchor

Detected boxes are all different sizes: a squashed slime is wide and short, a stretched one narrow and tall. Engines want uniform cells. So every frame gets padded out to the size of the largest.

Padded how is the last trap. Centre each sprite in its cell, and a character that gets taller mid-jump will appear to grow in both directions, with its feet dipping below the floor. Anchor to the bottom centre instead and the feet stay planted while the head does the moving.

Two animations side by side. Left, cut on an even grid: the slime jitters sideways and vertically. Right, detected and bottom-anchored: the slime squashes and hops in place

Left: the even grid cut. It's not an animation, it's a slime being shaken in a box. Right: detected, merged, padded, bottom-anchored. Same source pixels.

What you can't fix with a tool

I want to be straight about the limits, because "AI sprite sheet fixer" oversells it.

It isn't pixel art. Zoom into a generated "pixel" sprite and the pixels are different sizes and not on any grid. It's a painting of pixel art. You can downscale and re-quantize it, but that's a redraw, not a cut.

The frames don't agree with each other. The eyes move, the highlight changes shape, the character gains a pixel of height for no reason. Models draw six pictures of a slime, not one slime six times. No amount of cutting makes inconsistent drawings consistent.

Baked shadows stay baked. That soft shadow is now opaque dark green pixels. You can keep it or erase it. You can't turn it back into a real transparent shadow.

So here's how I'd actually use a generated sheet: as a storyboard. It's a fast way to see whether a six-frame hop reads at all. Then cut it, look at it moving, and redraw the frames that survive that test properly.

What's the most cursed sheet a model has handed you?

Top comments (0)