Fieldmix: a painting coach that mixes from the paints you actually own

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

I sometimes have sip-and-paint evenings with friends and family. The canvas comes with the drawing already on it, and deciding which colors to put on it is always the bottleneck. Fieldmix is my solution to a problem I run into whenever I decide to touch grass.

It's a painting companion that settles the colors, then gets the phone out of your hand as fast as possible. You:

1. Photograph what you want to paint: the pre-drawn canvas, the host's finished sample, or the view in front of you.
2. Tap the paints you actually have. They're drawn as tubes, and a little palette fills with a blob of each one.
3. Get your colors.
   • For a pre-drawn canvas, tap Suggest colors: Gemma proposes one of your paints for every shape, and you tap any shape to change it.
   • For a photo, the picture is split into a handful of numbered color shapes.
   • Either way, every color comes with a mixing recipe made only from your paints: "2 parts Phthalo Blue, 2 parts Titanium White, 1 part Alizarin Crimson Hue".
4. Get a plan. Gemma 4, running on your own laptop, labels the shapes ("sky", "red rock") and plans the order you paint them in.
5. Put the phone down. Each step is read aloud, with the recipe. You tap Next with a clean knuckle.

Left: Gemma's suggestion for a pre-drawn canvas. Right: the same canvas after I corrected it with 18 taps. More on both below.

The same problem follows you outdoors. You look at a hillside, squeeze out a yellow and a blue, and chase that green while the light moves. So Fieldmix is for sip-and-paint tables with friends, for plein air painters, and for anyone with a box of student acrylics who would rather paint the thing in front of them than follow a tutorial on a screen. That's the Touch Grass part: the screen gets a couple of minutes of setup, and then it's audio and your brush.

The one rule: the model never does color math

A language model will happily give you a mixing ratio. "Mix blue and yellow for green" is fine. "Use 3 parts ultramarine to 1 part cadmium" is a guess the model has no way to check, and a beginner would trust it.

So Fieldmix splits the work:

• Deterministic code does all the color: finding the color shapes, mixing paints the way pigment mixes, and scoring how close each mix gets.
• Gemma does what a teacher does: it names the shapes, decides the order, writes the instructions, and answers questions.

Gemma is shown which paints each recipe uses, but never the amounts, and it is not allowed to state any. The recipe you hear is spliced in by a template after the model's instruction. Any model output that contains a ratio fails validation:

RATIO_PATTERN = re.compile(r"""
    \b\d+\s*(?:parts?|portions?|drops?|dabs?|scoops?)\b      # "2 parts", "3 dabs"
    | \b\d+\s*:\s*\d+\b                                         # "2:1"
    | \b(?:one|two|three|four|five|six|seven|eight|half|equal)\s+(?:parts?|portions?|amounts?)\b
    | \b\d+\s*%                                                 # "30%"
    | \b(?:half\s+and\s+half|50\s*/\s*50)\b
    ...
""", re.IGNORECASE | re.VERBOSE)

The same check also rejects any paint name that isn't on your palette, so Gemma can't send you looking for a Viridian you don't own.

What else it does

• Test dab. Every recipe is labeled a starting point, because real paint, brands and the light in your photo all shift the result. Mix a pea-sized amount, dab it on the canvas edge, and tap Too light, Too dark, Too dull or Too bright. Fieldmix re-solves by adding one part of whichever of your paints moves the mix the right way, or tells you plainly that no single part will.
• "I don't have that color." Long-press any spot in the photo for the two best recipes for exactly that spot.
• Before / after. A preview of the finished painting built from your mixes, with a wipe slider against the photo.
• Draw your own outlines. Where a pre-drawn canvas is open (loose curls of hair, say), drag a line with your finger and fills stop there.
• Ask a question mid-painting ("my sky looks flat"), optionally with a photo of your canvas. The answer is checked for ratios too.

Demo

The whole idea in under two minutes, narrated:

Narration: macOS text-to-speech (the Daniel voice). Music: composed in code for this video.

A scene outdoors, using my lagoon waterfront photo:

A pre-drawn canvas, using a sunflower I drew for this post:

Both are real runs, captured frame by frame in the browser at phone size, with Gemma running on my laptop. The rings show where each tap lands. There's no hosted link yet: the public demo on Render is the next step, not done for Week 1.

That's the park photo from my eval set. The plan, labels and steps are Gemma's real output for it. The grass came back as 4 Burnt Sienna + 3 Phthalo Blue + 1 White, a muted, earthy green rather than a bright tube green.

You can run it yourself in about five minutes (Ollama, ollama pull gemma4:e4b, then scripts/dev.sh). The README has the steps, including how to open it on your phone over Wi-Fi.

Code

PreciousJac0b / fieldmix

Plein air painting companion: on-device color regions and paint-mix recipes, narrated by local Gemma 4

Fieldmix

A painting coach that mixes from the paints you actually own. Photograph the view in front of you (or the host's sample painting at a paint-and-sip), tap the paints in your bag, and Fieldmix splits the picture into a few color shapes, works out how to mix each one from only your paints, and talks you through the painting step by step, so the phone goes face-down next to the palette.

Built for the DEV Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass.

The one rule: the model never does color math

• Deterministic code does all the color:
  • k-means in CIELAB finds the color shapes;
  • spectral.js (Kubelka-Munk) mixes paints the way pigment mixes;
  • CIEDE2000 scores how close each mix gets.
• Gemma 4 (gemma4:e4b, running locally through Ollama) is the teacher:
  • labels the shapes, plans the painting order and writes the instructions;
  • answers questions, and suggests…

View on GitHub

MIT licensed. It's a Vite + TypeScript PWA with no UI framework, plus a small FastAPI backend. There are 132 web tests and 101 server tests, including the CIEDE2000 reference pairs from Sharma et al. (2005).

How I Built It

 Phone (PWA, all color math)                 Laptop (optional)
 ───────────────────────────                 ─────────────────
 photo ─► k-means in CIELAB ─► color shapes
 your paints ─► spectral.js mixes ─► recipes
                     │
                     └── shapes + recipes + 768 px photo ──► FastAPI ─► Ollama ─► Gemma 4 E4B
                                                                │
 paint mode ◄── labels + ordered steps (validated JSON) ◄───────┘
 (speech, test dab, wake lock)

Seeing the picture the way a painter blocks it in

The photo is shrunk to 512 px and clustered with k-means in CIELAB, a color space where distance roughly matches what your eye sees. Clusters closer than ΔE 6 are merged, and specks under 1% of the picture are absorbed, leaving 4 to 10 shapes a person can actually paint. (ΔE is CIEDE2000, the standard "how different do these two colors look" number. Under 2 is hard to tell apart side by side.)

Mixing like pigment, not like pixels

Average blue and yellow pixels and you get grey. Mix blue and yellow paint and you get green, because pigments absorb light rather than emit it. spectral.js (MIT) models this with Kubelka-Munk theory over spectral reflectance curves, and Fieldmix uses it for every mix.

The recipe search is brute force: 1 to 3 colored paints, plus white and black if you have them, in whole parts from 1 to 6, with equivalent ratios removed (2:2 is 1:1). Two things made that fast enough for a phone:

1. Mix once per palette, not once per region. A mix doesn't depend on the target color, so every candidate is mixed once and every region is matched against the same list.
2. A fast path that's checked against the library. I re-implemented spectral.js's mixing formula using the library's own per-paint curves and color-matching matrix (read out of it by feeding it unit reflectance curves). A test checks 500 random recipes against spectral.mix to within 1e-9.

The most accurate recipe was often absurd: the sea in one test photo came out as 6 Ultramarine + 5 White + 3 Phthalo + 2 Cadmium Red. Nobody mixes that at an easel for a difference they can't see. So Fieldmix now picks the simplest recipe within ΔE 2 of the best one, but never one that drops the match a quality grade. That sea became 2 Phthalo + 2 White + 1 Alizarin.

How close does it get? I took three photos on my phone for the eval: a park, a street and the lagoon waterfront. The light was soft and low, so they're mostly greys, dull greens and near-black shadows under the trees, which is exactly where mixing by eye is hardest. From scripts/eval.ts:

Palette	Regions	Recipe under ΔE 10	Median ΔE
Student set (6 colors + white)	17	100%	2.68
3 primaries + white + black	17	100%	3.21
4 paints (yellow, crimson, ultramarine, white)	17	82%	5.15

The 4-paint misses are the same in every photo: the deepest foliage shadow, nearly black. Without a black or an earth color, that palette can't get there. The student set gets within ΔE 5 to 8 of it with Burnt Sienna, Phthalo Blue and a touch of Alizarin. Solving every region of a photo took 1 to 15 ms on the laptop.

Gemma's job: the teacher, not the chemist

The planner is Gemma 4 E4B (gemma4:e4b in Ollama, a 9.5 GB download, Apache 2.0), running locally on an Apple Silicon MacBook with 24 GB of RAM. It gets the 768 px photo and, for each shape: its color, lightness, share of the picture, position, and the names of the paints in its recipe. It returns JSON: a label per shape and 5 to 10 ordered steps.

Every plan is validated:

• every shape id exists, and every shape gets painted;
• no ratios, and no paints you don't have.

If a plan fails, Gemma gets one retry with the problems listed. If that fails too, a deterministic planner takes over: top to bottom and darks first for acrylic, oil and gouache, light to dark for watercolor. It exists twice, in Python and in TypeScript (so it runs on the phone), and both are tested against the same fixture file.

From scripts/eval.py, on the same three photos: 6 requests (each photo × acrylic and watercolor), 3 runs each:

Valid plans	18/18, all on the first try, no fallbacks
Latency, median	7.0 s
Latency, p95	13.1 s (the first request, which includes loading the model)

Three things I learned the hard way:

• Structured output wasn't there. Every request with Ollama's format came back 501 structured output is unavailable. Homebrew's Ollama build runs this model on its MLX engine without the grammar library. So the backend now falls back to describing the JSON shape in the prompt, and leans on validation.
• Retrying doesn't fix a deterministic mistake. One reply closed the JSON object too early. The retry, at temperature 0.3, returned the identical 1,988 characters. The fix was to repair that one known slip in the parser, and only when strict parsing fails.
• Thinking mode wasn't worth it here. With Gemma's thinking on, a smaller run of 6 plans was just as valid but had a median of 17.9 s. With it off: about 6 s. It stays off.

Phone down

The phone parts matter as much as the model:

• Speech: steps are spoken with the browser's own speechSynthesis. pause() is unreliable on Android Chrome, so speech is queued sentence by sentence and Pause restarts the current sentence.
• Screen on: a phone loading the app from your laptop over plain http on Wi-Fi isn't a "secure context", so Chrome blocks the Screen Wake Lock. Fieldmix falls back to a silent looping video to keep the screen awake.
• Sunlight and wet hands: a high-contrast theme for sunlight, 72 px buttons, and each page has its own address, so Android's back button walks back through the app instead of closing it.

Color it in: the part I built for sip-and-paint night

A pre-drawn canvas is black lines on white, so there's nothing in the photo to mix. Fieldmix never invents colors from the picture, so on a canvas like that it lets you choose, and Gemma can suggest.

The drawing's lines become walls, and you tap a shape to fill it with one of your paints, as is or lightened with your white. Loose strokes like curly hair don't enclose anything, so color leaked into the background. I tried closing them automatically with a morphological filter. It broke the background into pieces and still didn't separate the hair, so I reverted it and added a Draw outline tool instead: drag your finger where the drawing is open, and fills stop there.

Suggest colors numbers the biggest shapes on the drawing and asks Gemma to pick one of your paints for each, plus how much white. Code still does every mix, and every answer passes the same checks as a plan: every number answered once, only your paints, no amounts.

It's a good start, not a finished answer, and the screenshots above show both sides. On the sunflower canvas (which I drew for this post), Gemma got the sun, the table and the cloud right. But it read shape 5, the flower's center, as "flower petals", and shape 6, the pot rim, as "stem and leaves". On a portrait in 4 earlier runs, every answer was valid and the picks were sensible (yellow hair, a red cap), but Gemma called the hand an "animal" every single time.

So it's presented as a starting point: tap any shape to change it, and Undo removes the whole suggestion. Fixing the sunflower took 18 taps, 12 of them petals, starting from a colored canvas instead of a blank one. In a later run on that same portrait, Gemma got it right: a warm brown face, dark brown curls, a brown hand, a blue cap, a pink heart, and a background split into blue and red. Runs vary: another run gave the face yellow and the hand red, and two taps with Raw Sienna fixed both before planning.

Recording the demo caught one more bug. After Color it in, the plan was still written from the blank drawing, so Gemma couldn't see which color went where, and it titled a red step "Flower Petals" when red was the pot. Now Gemma plans from the colored-in drawing. In the next run it named the sky, stem, sun and cloud correctly, but it still called the red pot "the flower". Labels are the one part that can be wrong; the recipe under each label is always computed.

Why Does Open Innovation Matter?

It runs where painters are. All the color work is MIT-licensed code running in the phone's browser. Once the page is loaded, the shapes, recipes, test dabs and a built-in plan need no signal at all. Gemma runs on a laptop on the same Wi-Fi or phone hotspot, not in a data center. Here's exactly what works where:

Setup	Shapes and recipes	Plan	Spoken steps	Questions
Phone alone (page already loaded)	✅	Built-in planner	✅ browser voice	❌
Phone + laptop running Gemma	✅	✅ Gemma	✅	✅

A hosted demo would add an HTTPS link anyone can open, but it isn't built yet.

It costs nothing per brushstroke. There's no API key, no account and no per-request bill. The model is a one-time 9.5 GB download.

Your photos stay yours. Photos of your painting, your street or your living room go from your phone to your own laptop and nowhere else.

I could open every box. This is where open beat closed for me in practice. I read spectral.js's mixing model, re-implemented it for speed, and proved my version matches the original to 1e-9. You can't do that with a closed color API. With open weights running locally, I could also turn Gemma's thinking off and measure the difference myself (about 3× faster, just as valid). And the provider is a small interface, so the model can be swapped.

To be clear, I didn't run a closed model against the same prompts, so I'm not claiming Gemma beats one. The win was control and locality: the color math is inspectable, and the model runs on hardware I own.

Bonus: I took it outside (halfway)

I took the eval photos outside on my phone: a park, a tree-lined street and the lagoon waterfront. To be honest, I didn't open Fieldmix out there. I ran the photos through it afterwards, and that was still worth it, because real photos found things my test images never did.

• They caught a bug. My street photo was so dim and grey (93% low-saturation) that Fieldmix decided it was an outline drawing and offered to color it in. An outline drawing is mostly white paper, and my street was mostly asphalt. Now a picture only counts as a drawing if at least half of it is light paper. There's a new test for it, and the street gets its color map.
• Muted light makes muted mixes. Almost nothing in these photos is a tube color:
  • the road grey: Ultramarine, Cadmium Red and White, with no black needed (ΔE 2.1);
  • the grass: Burnt Sienna and Phthalo Blue;
  • the evening sky: White with a touch of Alizarin and Phthalo.
• Gemma read every scene. It labeled the road, the water, the distant trees and the shadows, and planned all three from the sky down to the darkest detail. Both previews below rate the overall match "Good" (ΔE 2.6 for the street, 3.1 for the waterfront).
• What got lost:
  • a small pink cloud over the lagoon was too small to become its own shape, so it merged into the sky;
  • the bridge survived only as a broken line;
  • the deepest shadows under the trees need a black: the student set gets within ΔE 5 to 8.

The next test is the real one: at the easel.

My Agent Session

I built Fieldmix with Claude Code as a pair programmer. I wrote the spec, made the decisions, and tested on my phone; Claude wrote most of the code and kept a dated dev log of what broke and how it was fixed (DEVLOG.md).

Prize Categories

Best Use of Gemma. Gemma 4 E4B runs locally through Ollama and does three jobs:

• reads the photo and plans the painting;
• answers questions mid-session, including about a photo of your canvas;
• suggests colors for outline drawings.

It's deliberately kept away from the one thing it would get wrong: the color math. Every response is checked against a schema and a set of rules, with one retry and a deterministic fallback.

Known limits

• Preset paint colors are approximations of typical student-grade paint. Scanning a swatch card of your own paints is more accurate.
• The scene photo isn't white-balanced, so a warm or cool camera shifts the targets.
• Mixing uses spectral.js's model with recipe parts as its weights. Real pigments differ in tinting strength, which is why every recipe starts with a test dab.
• The eval numbers above come from three photos and small samples.

This post was drafted with AI assistance (Claude), then checked and edited by me. All the numbers come from the repo's eval scripts.