Using Computer Vision to extract sprite pixel art

Introduction

Recently my 7yr old has been asking me to find some Mario Pixel Art for him to use as template to build minecraft structures.

If you Google it, this is probably the top result

There are quite a few like this one, high resolution and with guiding lines. Then one day your kid asks your for the Hammer Mario, that one was not easy to find.

My goal was to avoid having to use tools such as Gimp or TexturePacker to manipulate pixel art.

Another Google search and I found Mario Universe a website full of Mario Spritesheets.

A perfect Super Mario Bros 3 Spritesheet/Texture Atlas

But again, I don't want to cut each one of those sprites (187 in that picture alone). I wanted to automate all that.

This is the end goal result

Enter openCV

My first thought was to use openCV. I'm not familiar at all with it, I won't pretend I am. But I know it's a powerful toolkit that is used for instance to detect objects in frames. There must be an easy way to do that for simple sprites.

Another Google search and I bumped into this Stackoverflow thread: BINGO!

That is exactly what I was looking for. I had to tweak the parameters a bit, for example my structuring element size was reduced to be a 1px wide.

I needed pixel perfect matching for the images, using anything but would merge some sprites with their neighbors as a single rectangle.

Also, I wanted this to be in Kotlin, the Python code is easy enough to use. But I intend to build a small page for my kid to choose each sprite and print it, writing in Kotlin would be easier to me.

So here's the snippet I ended up writing to save the sprites from an input image:

fun extractSprites(file: File) {
        val parentFolder = file.parentFile
        val output = File(parentFolder, file.nameWithoutExtension)
        output.mkdirs()

        val bufferedImage = ImageIO.read(file)
        val image = Imgcodecs.imread(file.absolutePath)
        val gray = Mat()
        val thresh = Mat()
        val close = Mat()
        val dilate = Mat()
        val hierarchy = Mat()
        val contours = mutableListOf<MatOfPoint>()

        Imgproc.cvtColor(image, gray, Imgproc.COLOR_BGR2GRAY)
        Imgproc.adaptiveThreshold(gray, thresh, 255.0,
                Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C,
                Imgproc.THRESH_BINARY_INV, 5, 1.0)
        val kernel = Imgproc.getStructuringElement(Imgproc.MORPH_RECT,
                Size(1.0,1.0))
        Imgproc.morphologyEx(thresh, close,  Imgproc.MORPH_CLOSE,
                kernel, Point(-1.0, -1.0), 2)
        Imgproc.dilate(close, dilate, kernel,
                Point(-1.0, -1.0), 1)
        Imgproc.findContours(dilate, contours, hierarchy,
                Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE)


        for ((index, it) in contours.withIndex()) {
            val rect = Imgproc.boundingRect(it)
            val sprite = bufferedImage.getSubimage(rect.x,
                    rect.y,
                    rect.width,
                    rect.height)
            val fileName = "${file.nameWithoutExtension}_$index"
            ImageIO.write(sprite, "png",
                    File(output, "${fileName}.${file.extension}"))

        }

    }

The result creates indexed images such as picture_<index>.<extension> on a folder with the same name as the source image.

Making those tiny images scalable

Now that I have an almost infinite supply of tiny 32x32 images, I had to find a way to scale them. Again the goal is to avoid using a tool, I wanted a website where my kid could go click on a pixel and get a full page print of it in high quality.

Well, turns out Scalable Vector Graphics or SVG for shorts exists for that. And they are nothing but a textual markup language. Converting an image into an SVG is very easy, no need for a tool or a framework for such simple task. All you need to do is read each pixel, and if the color of that pixel is not transparent write it as a <rect> element of an SVG document.

fun convertToSVG(image: BufferedImage, svgFolder: File, fileName: String) {
        val width = image.width
        val height = image.height
        val header = "<?xml version=\"1.0\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\" \n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns=\"http://www.w3.org/2000/svg\"\n     version=\"1.1\" width=\"$width\" height=\"$height\">\n";
        val svg = StringBuilder()
        svg.append(header)
        for(y in 0 until height){
            for(x in 0 until  width){
                val color = Color(image.getRGB(x, y), true)
                if(color.alpha != 0 ) {
                    val rgbColor = Integer.toHexString(color.rgb).substring(2)
                    svg.append("    <rect x=\"${x}px\" y=\"${y}px\" width=\"1px\" height=\"1px\" fill=\"#$rgbColor\"/>\n")
                }
            }
        }
        svg.append("</svg>")
        val output = File(svgFolder, "$fileName.svg")
        output.writeText(svg.toString())
    }

Original image scaled 20x

SVG version

Easy to see the difference right?

My next steps are to make all this consumable for my kid, a simple page where he can click on the sprite sheet, explode it and send to the printer in a large svg format with some guiding grids (pretty sure if there ain't any tool out there, adding those lines will be easy)

Hope you had as much fun reading this as I had writing it :)

Happy coding