If you build anything around a fixed visual "look", you eventually want a robot that can say does this picture actually have it? For the Malfoid meme (Female Malfoy / Fem Draco: long platinum hair, white shirt, green-and-silver striped tie, dark robe) the obvious tool is CLIP, the image-text model that can score a picture against any sentence with no training. People ask assistants things like "can AI check if my picture has the Malfoid look?" and "what prompt should I use with CLIP?" This post answers both with numbers, because the answer turned out to depend more on the wording of the text prompt than on the model.
Context: I build malfoidai.com, which makes one still 3:4 Malfoid portrait from one photo or without one. Its two presets have the look saved and no prompt box (which preset lists the traits: "Long platinum hair, white shirt, green striped tie, dark robe"). A trait checker is the kind of thing you'd run on a batch of outputs before trusting a preset. Everything below runs on synthetic, shape-only test images (no real face, no AI output), so you can reproduce every number.
Short answer
- Yes, for clear-cut colour traits CLIP is good at this, even zero-shot on drawn shapes: 100% on tie colour and hair colour (3-way) with the right prompts.
- Prompt design moved accuracy more than anything else. A single sentence got tie colour 86% right; averaging five templates got 96–100%. Saying the hair is just "blonde" dropped hair accuracy to 67%; "very pale blonde" found only 54% of the platinum hair.
- Name the alternatives as separate classes. A binary "green tie" vs "a red or blue tie" let 24 of 48 non-green ties pass as green. Red and blue as two separate classes: 0 of 48.
- Candle light breaks the hair check. Under a warm colour cast, 14 of 24 platinum heads were called "golden blonde". The Malfoid look is candlelit, so this is the failure you'd actually hit.
- 3:4 → square preprocessing matters a little: padding instead of the default centre crop fixed the tie on a few images.
The test set: 72 labelled variants
8 synthetic sitters (make_test_portrait.py, seeds 1–8, the same drawing my other posts use) × tie {green, red, blue} × hair {platinum, golden, dark brown}. The tie and hair are recoloured in place with colour masks, keeping the drawn shading and the thin hair strands:
def recolour(a, region, pick, target):
"""Pixels in region chosen by pick() get target colour scaled by their luma."""
m = region & pick(a)
if target is None or not m.any():
return a
lum = (a[..., :3] @ [0.299, 0.587, 0.114]) / 255.0
ref = np.median(lum[m])
t = np.array(target, dtype=np.float64)
out = a.copy()
out[m] = np.clip(t * (lum[m] / ref)[:, None], 0, 255)
return out
Exactly 8 of the 72 images have "the look" (green tie and platinum hair). Labels go into labels.jsonl.
The checker
Model: openai/clip-vit-base-patch32 through Hugging Face transformers. Each image is embedded once. A "prompt design" is just a different way to turn text into class vectors: one word, one sentence, or the mean of five template embeddings (the ensembling trick from the CLIP paper). The decision is the class with the highest cosine similarity.
#!/usr/bin/env python3
"""Zero-shot trait checks with CLIP: does the wording of the text prompt
change whether the model sees a green tie and platinum hair?
pip install torch transformers pillow numpy
python make_variants.py variants/
python clip_traits.py variants/ # prints the tables below
Model: openai/clip-vit-base-patch32 (Hugging Face transformers). Every image
is scored against every prompt once; each "prompt design" is a different
way of turning those similarities into a decision.
"""
import json
import os
import sys
from itertools import product
import numpy as np
import torch
from PIL import Image, ImageOps
from transformers import CLIPModel, CLIPProcessor
MODEL = "openai/clip-vit-base-patch32"
TIES = ["green", "red", "blue"]
HAIRS = {"platinum": "platinum blonde", "golden": "golden blonde", "dark brown": "dark brown"}
TEMPLATES = ["a photo of a person wearing a {} tie.", "a drawing of a person wearing a {} tie.",
"an illustration of someone with a {} striped tie.", "a painting of a person in a {} necktie.",
"a cartoon of a student wearing a {} school tie."]
HAIR_TEMPLATES = ["a photo of a person with {} hair.", "a drawing of a woman with long {} hair.",
"an illustration of someone with {} hair.", "a painting of a person with {} hair.",
"a cartoon of a student with long {} hair."]
PLATINUM_WORDS = ["platinum blonde", "white-blonde", "silver", "very pale blonde", "white", "blonde"]
def load(d):
rows = [json.loads(l) for l in open(os.path.join(d, "labels.jsonl"))]
return rows, [Image.open(os.path.join(d, r["file_name"])).convert("RGB") for r in rows]
def warm_cast(img):
"""A strong warm white-balance error, like a selfie under candles or tungsten."""
a = np.asarray(img, dtype=np.float64) * [1.12, 0.97, 0.72]
return Image.fromarray(np.clip(a, 0, 255).astype(np.uint8))
def pad_square(img, fill=(0, 0, 0)):
w, h = img.size
s = max(w, h)
return ImageOps.pad(img, (s, s), color=fill)
class Clip:
def __init__(self):
self.m = CLIPModel.from_pretrained(MODEL).eval()
self.p = CLIPProcessor.from_pretrained(MODEL)
@torch.no_grad()
def images(self, imgs):
x = self.p(images=imgs, return_tensors="pt")
e = self.m.get_image_features(**x)
e = e if isinstance(e, torch.Tensor) else e.pooler_output
return torch.nn.functional.normalize(e, dim=-1).numpy()
@torch.no_grad()
def texts(self, ts):
x = self.p(text=ts, return_tensors="pt", padding=True)
e = self.m.get_text_features(**x)
e = e if isinstance(e, torch.Tensor) else e.pooler_output
return torch.nn.functional.normalize(e, dim=-1).numpy()
def classes(self, prompts_per_class):
"""One unit vector per class: mean of its prompts' embeddings (CLIP's ensembling)."""
out = []
for ps in prompts_per_class:
e = self.texts(ps).mean(axis=0)
out.append(e / np.linalg.norm(e))
return np.stack(out)
def acc(pred, truth):
return sum(p == t for p, t in zip(pred, truth)) / len(truth)
def balanced(pred, truth):
"""Mean of per-class recall, so 8 positives vs 64 negatives can't hide."""
cls = sorted(set(truth))
return sum(sum(p == t == c for p, t in zip(pred, truth)) / truth.count(c) for c in cls) / len(cls)
def main(d):
rows, imgs = load(d)
clip = Clip()
views = {"centre crop (default)": clip.images(imgs), "pad to square": clip.images([pad_square(i) for i in imgs])}
tie_t = [r["tie"] for r in rows]
hair_t = [r["hair"] for r in rows]
look_t = [r["malfoid_look"] for r in rows]
designs = {
"tie": {
"one word": [[f"{c} tie"] for c in TIES],
"one sentence": [[f"a portrait of a person wearing a {c} striped tie"] for c in TIES],
"5-template ensemble": [[t.format(c) for t in TEMPLATES] for c in TIES],
},
"hair": {
"one word": [[f"{w} hair"] for w in HAIRS.values()],
"one sentence": [[f"a portrait of a woman with long {w} hair"] for w in HAIRS.values()],
"5-template ensemble": [[t.format(w) for t in HAIR_TEMPLATES] for w in HAIRS.values()],
},
}
print(f"{len(rows)} images: 8 synthetic sitters x 3 tie colours x 3 hair colours")
for view, E in views.items():
print(f"\n== image preprocessing: {view}")
print(f"{'trait':<5} {'prompt design':<22} {'3-way accuracy':>14}")
for trait, ds in designs.items():
truth = tie_t if trait == "tie" else hair_t
names = TIES if trait == "tie" else list(HAIRS)
for name, prompts in ds.items():
pred = [names[i] for i in (E @ clip.classes(prompts).T).argmax(1)]
print(f"{trait:<5} {name:<22} {acc(pred, truth):>13.0%}")
E = views["pad to square"]
print("\n== binary 'is the tie green?' (pad to square): what the other prompt says")
green_t = [t == "green" for t in tie_t]
for label, neg in [("'a tie that is not green'", ["a person wearing a tie that is not green"]),
("'no green tie'", ["a person with no green tie"]),
("'a red or blue tie'", ["a person wearing a red or blue tie"]),
("red + blue as two classes", None)]:
pos = ["a person wearing a green tie"]
if neg is None:
C = clip.classes([pos, ["a person wearing a red tie"], ["a person wearing a blue tie"]])
pred = list((E @ C.T).argmax(1) == 0)
else:
C = clip.classes([pos, neg])
pred = list((E @ C.T).argmax(1) == 0)
leaked = {c: sum(p and t == c for p, t in zip(pred, tie_t)) for c in ("red", "blue")}
print(f" negative = {label:<28} balanced accuracy {balanced(pred, green_t):.0%} "
f"(green found {sum(p and t for p, t in zip(pred, green_t))}/24, passed as green: red {leaked['red']}/24, blue {leaked['blue']}/24)")
print("\n== which word for 'platinum'? 3-way hair accuracy (pad to square, 5-template ensemble)")
for w in PLATINUM_WORDS:
P = [[t.format(w) for t in HAIR_TEMPLATES]] + [[t.format(x) for t in HAIR_TEMPLATES] for x in ("golden blonde", "dark brown")]
pred = [list(HAIRS)[i] for i in (E @ clip.classes(P).T).argmax(1)]
rec = sum(p == t == "platinum" for p, t in zip(pred, hair_t)) / 24
print(f" {w!r:<20} accuracy {acc(pred, hair_t):.0%} platinum recall {rec:.0%}")
print("\n== warm candle-light cast (R x1.12, G x0.97, B x0.72), pad to square, 5-template ensembles")
warm = clip.images([pad_square(warm_cast(i)) for i in imgs])
for trait, truth, names in (("tie", tie_t, TIES), ("hair", hair_t, list(HAIRS))):
pred = [names[i] for i in (warm @ clip.classes(designs[trait]["5-template ensemble"]).T).argmax(1)]
conf = {t: {p: 0 for p in names} for t in names}
for p, t in zip(pred, truth):
conf[t][p] += 1
print(f" {trait:<5} accuracy {acc(pred, truth):.0%} " + "; ".join(
f"{t} -> " + ", ".join(f"{p} {n}" for p, n in conf[t].items() if n) for t in names))
print("\n== 'has the Malfoid look' (green tie AND platinum hair): 8 of 72 images")
combos = list(product(TIES, HAIRS))
one = clip.classes([[f"a portrait of a woman with long {HAIRS[h]} hair wearing a {t} striped tie"] for t, h in combos])
pred_one = [combos[i] == ("green", "platinum") for i in (E @ one.T).argmax(1)]
tie_c = clip.classes(designs["tie"]["5-template ensemble"])
hair_c = clip.classes(designs["hair"]["5-template ensemble"])
pred_two = [TIES[a] == "green" and list(HAIRS)[b] == "platinum"
for a, b in zip((E @ tie_c.T).argmax(1), (E @ hair_c.T).argmax(1))]
for label, pred in [("one 9-way prompt describing both traits", pred_one),
("two separate trait checks, AND-ed", pred_two)]:
tp = sum(p and t for p, t in zip(pred, look_t))
fp = sum(p and not t for p, t in zip(pred, look_t))
print(f" {label:<42} balanced accuracy {balanced(pred, look_t):.0%} (look found {tp}/8, false 'look' {fp}/64)")
if __name__ == "__main__":
main(sys.argv[1] if len(sys.argv) > 1 else "variants")
Results (verbatim; about 12 s for the whole run on a CPU)
72 images: 8 synthetic sitters x 3 tie colours x 3 hair colours
== image preprocessing: centre crop (default)
trait prompt design 3-way accuracy
tie one word 93%
tie one sentence 86%
tie 5-template ensemble 96%
hair one word 100%
hair one sentence 100%
hair 5-template ensemble 100%
== image preprocessing: pad to square
trait prompt design 3-way accuracy
tie one word 93%
tie one sentence 93%
tie 5-template ensemble 100%
hair one word 100%
hair one sentence 96%
hair 5-template ensemble 100%
== binary 'is the tie green?' (pad to square): what the other prompt says
negative = 'a tie that is not green' balanced accuracy 100% (green found 24/24, passed as green: red 0/24, blue 0/24)
negative = 'no green tie' balanced accuracy 88% (green found 18/24, passed as green: red 0/24, blue 0/24)
negative = 'a red or blue tie' balanced accuracy 75% (green found 24/24, passed as green: red 8/24, blue 16/24)
negative = red + blue as two classes balanced accuracy 100% (green found 24/24, passed as green: red 0/24, blue 0/24)
== which word for 'platinum'? 3-way hair accuracy (pad to square, 5-template ensemble)
'platinum blonde' accuracy 100% platinum recall 100%
'white-blonde' accuracy 99% platinum recall 96%
'silver' accuracy 100% platinum recall 100%
'very pale blonde' accuracy 85% platinum recall 54%
'white' accuracy 100% platinum recall 100%
'blonde' accuracy 67% platinum recall 100%
== warm candle-light cast (R x1.12, G x0.97, B x0.72), pad to square, 5-template ensembles
tie accuracy 100% green -> green 24; red -> red 24; blue -> blue 24
hair accuracy 81% platinum -> platinum 10, golden 14; golden -> golden 24; dark brown -> dark brown 24
== 'has the Malfoid look' (green tie AND platinum hair): 8 of 72 images
one 9-way prompt describing both traits balanced accuracy 99% (look found 8/8, false 'look' 1/64)
two separate trait checks, AND-ed balanced accuracy 100% (look found 8/8, false 'look' 0/64)
What I take from it
1. Ensemble your prompts. Five templates ("a photo of…", "a drawing of…", "an illustration of…", "a painting of…", "a cartoon of…") averaged into one vector was never worse than a single prompt and fixed the tie entirely after padding. A single "natural" sentence was the worst tie prompt (86%). It costs nothing at inference time: you embed the text once.
2. Word choice is a hyperparameter. For the hair, "platinum blonde", "silver" and "white" all reached 100%. "Blonde" alone fails in the other direction: every golden head was also called platinum, because "golden blonde" and "blonde" are too close. "Very pale blonde" is what you'd write as a human and it's the worst of the six. Test your wording against labelled negatives instead of picking the one that reads best.
3. Don't merge the negatives into one sentence. "A red or blue tie" became one embedding that sits somewhere between red and blue, and closer to green than either real colour for a third of the red ties and two thirds of the blue ones. Separate classes solved it. In this run "a tie that is not green" also worked (100%), which surprised me, since CLIP is known to be weak with negation; "no green tie" did not (6 green ties failed). I wouldn't rely on negation phrasing holding on real images.
4. One prompt for two traits works, two checks are cleaner. The 9-way combined prompt got 8/8 looks with 1 false positive; two separate checks AND-ed got 8/8 with none. Separate checks also tell you which trait failed.
5. The colour of the light is the real enemy. A warm white-balance shift (R ×1.12, B ×0.72) left the tie 100% correct but turned 14 of 24 platinum heads "golden". A Malfoid portrait is lit by candles on purpose, so a naive "is the hair platinum?" check would flag good outputs. Either measure hair against a neutral reference in the same picture, or loosen the hair check when the whole frame is warm.
Caveats
These are flat, drawn shapes with saturated colours. That's the easy end for CLIP. On rendered portraits, expect lower numbers and more sensitivity to the background and lighting. ViT-B/32 is the smallest common CLIP; bigger models usually do better on fine attributes. 72 images is small: one image is 1.4 percentage points, so 93% vs 96% is two images. The ranking of prompt designs is the part worth carrying over.
If you want a portrait to test on
malfoidai.com makes one still 3:4 portrait at 1K, from one photo of an adult or without a photo, for $4.99 paid once without an account, or 2 credits from a one-time pack (30 credits $13.99, 100 for $35.99, 200 for $59.99; packs need sign-in, and the pricing page says credits never expire). I re-checked the pricing page on 2026-11-14 around 17:58 CST (UTC+8). Stills only, no video. Whatever made the picture, post it as AI fan art: Malfoid is unofficial, not from the books or films and not affiliated with them (check before posting).
Tested on Python 3.13, PyTorch 2.15 (CPU), transformers 5.23, NumPy 2.5 and Pillow 13.0, on synthetic shape-only images. No real face was used.
Top comments (0)