Reading 839 game boards out of screenshots with OpenCV and k-means

I run Block Out Guide, a fan site for the mobile puzzle game Block Out! - Color Sort Puzzle. Every published level gets a static page with its video walkthrough, a board screenshot, and links to the neighbouring levels.

The easy version of that site is a template: fill in the level number, embed the YouTube video, done. But then every page says essentially the same thing. A player stuck on level 412 already knows the level number. What they don't know is what the board looks like.

The site already had one screenshot per level: the YouTube thumbnail, 1280×720. So I wrote extract-board-data.py, a Python script that reads the board out of each screenshot and emits a JSON description: palette, colour shares, background, and a compact layout map. It currently covers 839 boards, and the whole output is about 1.1 MB.

This post covers the image-processing decisions, including the ones that did not work the first time.

The screenshots are not standardized

There are two templates in the playlist:

• a promo card, with the board on the left and the "BLOCK OUT" logo on the right;
• an in-game screenshot, with the board in the middle, the logo on the left and a player watermark on the right.

So a fixed crop is out. It would either clip landscape boards or include the logo. Both templates do put the board inside the same purple rounded frame, so the first idea was to find the frame.

The frame works for most promo cards. On some boards, though, blocks in blue or purple pass the same colour test as the frame, and landscape boards are wider than they are tall, which breaks the frame's assumed proportions.

Find the board with a content mask

The second approach looks for the most vivid blob in the middle band of the image. The character sits to the left of the board; the promo logo and the watermark sit to its right. So the scan is restricted to x = 205..860, keeping pixels with high saturation and value, then closing the gaps and taking the largest connected component.

hsv = cv2.cvtColor(a, cv2.COLOR_RGB2HSV)
mask = ((hsv[..., 1] > 110) & (hsv[..., 2] > 100)).astype(np.uint8) * 255
mask[:, :BOARD_SCAN_X0] = 0
mask[:, min(w, BOARD_SCAN_X1):] = 0
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (17, 17))
closed = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)
n, _, stats, _ = cv2.connectedComponentsWithStats(closed)

The morphological close is what makes this work. Individual blocks are separated by thin dark gaps, and a 17×17 closing kernel bridges them into one component, so the board becomes one blob instead of a pile of separate rectangles.

Components smaller than 140×140 are discarded, and the largest remaining one by area wins. The fixed side margins keep the character and the watermark out of the result.

Ignore the frame, keep the pale blocks

The content box is still slightly too large: it includes part of the rounded purple shell. My first instinct was to trim it by saturation, which deletes white and pale blocks sitting on the edge. Boards with white pieces at the border made that obvious.

The fix is to measure the frame's own colour instead of assuming it. The script samples the purple frame mask inside the detected box, takes the median, and keeps the page navy as a constant. A pixel counts as content only if it is far enough from both:

frame_distance = np.abs(region - frame_rgb).sum(axis=2)
page_distance = np.abs(region - PAGE_RGB).sum(axis=2)
content = (frame_distance > 55) & (page_distance > 28)

The thresholds are sums across the three RGB channels, so 55 means roughly 18 per channel. That is cheap, and it held up across both templates. The tight bounding box of the content is then inset by 3% to drop anti-aliased edge pixels, with a floor of 3 pixels so it does not eat a real edge column.

Sample a lattice. Do not pretend it is the game grid

Next question: what is the cell grid? The honest answer is that it varies by level, and I do not have the game's internal map data. So the script does not try to recover the exact grid. It samples a fixed 20×28 lattice and takes the median colour of each patch.

for j in range(GRID_NY):
    for i in range(GRID_NX):
        patch = sub[py0:py1, px0:px1]
        cells[j, i] = np.median(patch.reshape(-1, 3), axis=0)

Median rather than mean: the blocks are flat shaded, but there are highlights, shadows and anti-aliased boundaries. The median throws those away. A mean would smear a boundary pixel into the cell colour.

The published data is colour shares and layout samples. A 20×28 sampling grid gives a stable resolution for comparing boards and a readable map. It does not claim the game has exactly 560 cells.

Cluster the colours in Lab, merge by hue

Each board has 3 to 13 distinct colours, with a median of 8. But k-means on raw RGB with k=14 splits one colour into its highlight and shadow variants. So the pipeline converts the samples to CIE Lab, runs k-means, and then merges the fragments.

The merge rules are deliberately asymmetric:

• chromatic clusters merge when the hue distance is at most 14° and the saturation difference is at most 0.25;
• achromatic clusters (chroma ≤ 25) merge by value, with a tolerance of 28/255;
• when a tiny cluster is folded into a neighbour, Lab distance down-weights lightness by a factor of 0.35.

def swarm_distance(lab_a, lab_b):
    return float(np.sqrt((lab_a[1] - lab_b[1]) ** 2 + (lab_a[2] - lab_b[2]) ** 2)
                 + LIGHTNESS_WEIGHT * abs(lab_a[0] - lab_b[0]))

One special case: the navy empty board and the purple frame share a hue. Merge purely by hue and they collapse into one cluster, after which the background gets reported as the frame. The fix is a lightness split. If both clusters sit in hue 220 to 285 and one is dark (value below 0.48) while the other is bright (value above 0.55), the script keeps them separate.

Clusters below 2% of the sampled cells are folded into their nearest neighbour, so a stray highlight does not become a palette entry.

Name the colours and find the empty board

Palette entries get coarse names from hue (red, orange, yellow, green, cyan, blue, purple, pink) plus greys by value. Duplicate names are disambiguated with lightness bands, which is where names like "pale blue" and "light blue" come from. Two greens that land in different bands are different colours on screen.

Then there is the background. The empty board is the largest dark-blue group that is not the frame. The criteria: blue leads red and green by at least 15, value is below 0.68, saturation is below 0.80, and the group covers at least 8% of the board. A bright blue block fails that test.

Every sampled cell is then assigned to its nearest remaining group in Lab. Cells excluded as frame or page become . in the map. Percentages are re-based on the assigned cells, so the published map and the published numbers cannot disagree. Finally the palette is sorted so index 0 is always the largest colour.

The output

Each level ends up with a compact record like this (real data from level 1):

{
  "b": [251, 62, 703, 669],
  "g": [20, 28],
  "bghex": "#232460",
  "bgname": "navy",
  "bgpct": 27.0,
  "c": [
    {"hex": "#EDE7E7", "name": "white", "pct": 17.3, "cells": 80, "edge": 5, "center": 12},
    {"hex": "#343234", "name": "near-black", "pct": 14.0, "cells": 65, "edge": 8, "center": 46},
    {"hex": "#EBA512", "name": "orange", "pct": 13.4, "cells": 62, "edge": 21, "center": 79}
  ],
  "l": [
    "....................",
    "223333333222444444..",
    "223333333222444444..",
    "223333332222444444.."
  ]
}

The edge and center fields are the share of a colour's cells in the outer two-cell border versus the central half of the board. They exist so a page can say "this colour mostly sits at the edges" without shipping the full map to every visitor.

The layout uses base-36 characters, so up to 36 colours can be encoded without changing the format. In practice the maximum is 13.

Running it over 839 images

A few operational details that saved time:

• The script reads from a local R2 WebP dump first and only fetches a missing thumbnail over HTTP, with exponential backoff, into a gitignored cache.
• ProcessPoolExecutor runs across all CPU cores with chunksize=8. The work is pure numpy and OpenCV, so processes are the right tool; per-image work does not benefit from threads in Python.
• --limit is the smoke test. --min-id / --max-id re-run a range and merge the results into the existing JSON, so fixing ten bad boards does not mean redoing 839.
• The final JSON is written with compact separators and keys sorted numerically. That keeps the payload small and the diffs stable.

Current stats: 839 boards, 3 to 13 colours per board (median 8), and 578 boards with a detected empty-board background. The other 261 boards did not meet the background test.

What this data is not

The limits:

• The map is not the game's exact grid. It is a 20×28 sampling lattice, so the shares describe the artwork, not legal moves.
• It says nothing about doors, paths or solvability. That would need the game's own data.
• The input can lie. Levels 1 to 10 share a single compilation video, so they share one thumbnail and produce ten identical board records.
• Template changes break it. The side margins that exclude the character and watermark are constants. A new screenshot layout would need those adjusted.

Even with those limits, each level page gets a palette, the dominant colours, and a map of where they sit. A player deciding whether to attempt a board can use that.

The script is about 650 lines including comments, and the pipeline is deliberately boring: NumPy, OpenCV and Pillow, no ML model, no GPU. If you are doing something similar with game screenshots, the two decisions that mattered most were measuring the frame colour instead of thresholding saturation, and being honest that the sampling grid is a sampling grid.

The site it feeds: https://block-out.org/