How to Build an AI-Based Personalized Fashion Platform (Skin Tone, Color Matching and Recommendations)

Originate Soft Pvt Ltd — Wed, 07 Oct 2026 11:50:03 +0000

Most "personalized" fashion sites are really "people who bought this also bought that." That's fine, but it doesn't answer the question a shopper actually has: what will look good on me?
A proper personalized fashion platform tries to answer that one. It reads a few signals about the person (skin tone, undertone, size, style leanings, past behavior), matches them against what's in the catalog, and ranks products accordingly.
Founders are circling this idea too. One r/Startup_Ideas thread describes an AI fashion assistant that suggests clothes by skin tone, body type and fit, offers virtual try-on, and aims to recommend what is suitable and correctly sized, not just what's trending. That's the shape of product this article is about, so we'll go through each piece.
This guide walks through how to build that: the architecture, the skin tone pipeline, the color-matching logic, recommendations with embeddings, and the privacy and fairness work people tend to skip. There's code for the core pieces.
One honest note before we start. Matching clothes to complexion is a styling heuristic, not a law of nature. Build it as "colors that tend to flatter you" with an easy override, never as "this color is wrong for you." That framing will shape a lot of the decisions below.

The architecture at a glance

Selfie (processed on device)
-> face landmarks -> skin patches -> Lab color -> tone + undertone + confidence
|
Catalog images |
-> garment color extraction + FashionCLIP embeddings |
-> vector index v
User profile (tone, size, style, history) -> candidate retrieval -> re-ranker -> personalized feed
(embeddings) (color harmony + behavior)
Two pipelines meet in the middle. One understands the person, one understands the products, and the ranker decides which products to show to whom.

Step 1: Understand your catalog first

Skin tone detection gets the attention, but the catalog is where most projects quietly fail. If your product data is messy, the smartest matching engine in the world has nothing to match against.
For each product you want:
• Clean attributes: category, fit, fabric, pattern, occasion, size availability.
• Real garment colors: not the marketing name ("Midnight Mist"), but measured color values.
• Image embeddings: so you can search and compare visually.
To get garment colors, segment the garment from the background and from any model's skin (a clothing parser or a segmentation model both work), then cluster the remaining pixels in Lab space.
import numpy as np
from skimage import color
from sklearn.cluster import KMeans

def dominant_colors(garment_rgb_pixels, k=3):
"""garment_rgb_pixels: (N, 3) uint8 array of garment-only pixels."""
rgb = garment_rgb_pixels.astype(np.float32) / 255.0
lab = color.rgb2lab(rgb.reshape(-1, 1, 3)).reshape(-1, 3)
km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(lab)
counts = np.bincount(km.labels_, minlength=k)
order = np.argsort(-counts)
# return cluster centers (L*, a*, b*) with their share of the garment
return [(km.cluster_centers_[i], counts[i] / counts.sum()) for i in order]
Why Lab and not RGB? Because distances in CIELAB roughly follow how humans perceive color differences. RGB distances don't, and that matters when you start scoring "how different is this shirt from this person's skin."
Product photos also vary in lighting and editing. Studio shots are usually consistent within a brand, but if you aggregate several brands, expect color drift. Store the measured color, but keep the seller's color name too, and consider letting merchants correct it.

Step 2: Detect skin tone from a photo

Here's the pipeline most teams end up with:

Detect the face and landmarks. MediaPipe Face Mesh runs in the browser and on phones, which matters for privacy (more on that later).
Pick skin regions. Cheeks and forehead are the usual choice. Skip eyes, lips, eyebrows, beard, and anything near the hairline.
Remove bad pixels. Specular highlights and deep shadows will pull your estimate around.
Correct white balance as best you can.
Convert to Lab and take a robust statistic (the median).
Compute tone and undertone, plus a confidence score. import numpy as np from skimage import color

def skin_lab(skin_pixels_rgb):
"""skin_pixels_rgb: (N, 3) uint8 pixels sampled from cheeks/forehead."""
rgb = skin_pixels_rgb.astype(np.float32) / 255.0
lab = color.rgb2lab(rgb.reshape(-1, 1, 3)).reshape(-1, 3)
L = lab[:, 0]
lo, hi = np.percentile(L, [15, 85]) # drop deep shadows and highlights
kept = lab[(L >= lo) & (L <= hi)]
return np.median(kept, axis=0) # (L*, a*, b*)

def ita_degrees(L, b):
"""Individual Typology Angle, a standard skin-lightness measure."""
return np.degrees(np.arctan2(L - 50.0, b))
What ITA gives you
The Individual Typology Angle is a number computed from the L* and b* channels. It's widely used in dermatology research to estimate skin tone from images, and it correlates with how light or dark skin appears. The literature groups it roughly into very light, light, intermediate, tan, brown and dark bands (see this paper on estimating skin tone in dermatology datasets for how it's used in practice).
What I'd suggest: use ITA as a continuous feature, not as a label. Don't hard-bucket users and then show them a bucket name.
Which scale to show users
If you do show a scale, use one built for appearance. Google's Monk Skin Tone Scale has ten shades and was released to help test AI systems for bias, which also makes it a good fit for the evaluation work later in this article. It's more granular than the older six-type Fitzpatrick scale, which describes how skin reacts to sun exposure rather than how it looks, so I'd skip Fitzpatrick for styling.
There's no official, validated mapping from ITA values to the ten Monk shades. So don't pretend there is. Let the model suggest a shade and have the user confirm it with a swatch picker.
Undertone
Undertone (warm, cool, neutral) is harder than depth. A rough photo-based signal comes from the hue angle in the a*/b* plane, where more yellow reads warmer and more red or pink reads cooler.
def undertone_from_lab(a, b, warm_cut=58.0, cool_cut=50.0):
"""Cutoffs are placeholders. Calibrate them on labeled data from your own users."""
hue = np.degrees(np.arctan2(b, a))
if hue >= warm_cut:
return "warm"
if hue <= cool_cut:
return "cool"
return "neutral"
Those cutoffs are made up for the example, and I mean that literally: tune them on real, consented, labeled data from your audience. Because a photo alone is shaky here, combine it with a quick question or two ("Does gold or silver jewelry look better on you?"). Two weak signals together beat one weak signal.
Where this breaks in the real world
Almost every failure I'd expect here comes from the input, not the model:
• Lighting: warm bulbs, window backlight, and phone night mode all shift the result.
• White balance and filters: beauty filters and auto-processing change skin color before you ever see it.
• Makeup and tan: you'd be measuring what's on the skin today.
• Shadows and facial hair: these bias the sampled pixels.
• Cheap cameras: noise and heavy compression.
The practical answer: show a short capture guide ("face a window, no filter"), compute a confidence score (pixel variance, number of usable pixels, lighting checks), and fall back to the manual swatch picker when confidence is low. Never silently guess.

Step 3: Build the color-matching engine

Now you have a person (Lab values, undertone) and a garment (dominant colors). You need a score.
Start with rules. They're explainable, easy to debug, and they give you something to ship before you have any behavioral data. Three signals are a reasonable start:
• Contrast: how far the garment is from the skin tone (CIEDE2000 is the standard perceptual distance).
• Chroma: how vivid the garment color is.
• Undertone fit: whether the garment's hue sits on the warm or cool side.
import numpy as np
from skimage import color

def harmony_score(skin_lab, garment_lab, undertone):
"""Return a 0..1 score. Weights are starting points. Learn them from feedback."""
d_e = color.deltaE_ciede2000(skin_lab[None, None, :], garment_lab[None, None, :])[0, 0]
contrast = np.clip(d_e / 60.0, 0, 1)
chroma = np.hypot(garment_lab[1], garment_lab[2])
vivid = np.clip(chroma / 80.0, 0, 1)

garment_hue = np.degrees(np.arctan2(garment_lab[2], garment_lab[1])) % 360
warm_side = np.cos(np.radians(garment_hue - 60))   # +1 yellow/orange, -1 blue/green
if undertone == "warm":
    fit = warm_side
elif undertone == "cool":
    fit = -warm_side
else:
    fit = 0.0

return 0.4 * contrast + 0.2 * vivid + 0.4 * (0.5 + 0.5 * fit)

Be clear-eyed about what this is. The popular "seasonal color analysis" (spring, summer, autumn, winter) is a stylist tradition, not a validated science, and plenty of people look great in colors a rule says they shouldn't wear. High contrast also isn't universally "better"; some people prefer soft, low-contrast outfits.
So treat the rule score as one feature, not the verdict. Once you have click, save, purchase and return data, feed harmony_score into a learned ranker and let behavior decide how much it matters for each user. Returns tagged "color not as expected" or "didn't suit me" are especially valuable training signal.

Step 4: Recommendations with embeddings

Color match is only one part of "what will this person like." For the rest you need visual and semantic similarity.
FashionCLIP is a CLIP model fine-tuned on fashion data (the original work came from Farfetch's research team), and it's available on Hugging Face. It maps product images and text into the same vector space, so you can search "linen shirt dress" and also find products visually similar to one a user saved.
import torch
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("patrickjohncyh/fashion-clip")
processor = CLIPProcessor.from_pretrained("patrickjohncyh/fashion-clip")

def embed_images(pil_images):
inputs = processor(images=pil_images, return_tensors="pt")
with torch.no_grad():
emb = model.get_image_features(**inputs)
return emb / emb.norm(dim=-1, keepdim=True) # normalize for cosine search

def embed_text(texts):
inputs = processor(text=texts, return_tensors="pt", padding=True)
with torch.no_grad():
emb = model.get_text_features(**inputs)
return emb / emb.norm(dim=-1, keepdim=True)
Store the vectors in pgvector (if you're already on Postgres) or FAISS, and use a two-stage setup:

Candidate retrieval: hard filters first (in stock in the user's size, price range, category), then nearest neighbors to the user's taste vector (an average of items they've saved or bought).
Re-ranking: a learning-to-rank model (LightGBM's LambdaRank is a solid, boring choice) over features like embedding similarity, harmony_score, price fit, brand affinity, popularity and recency. Cold start is the real problem with a new user. Use a short onboarding quiz (styles, occasions, budget), the confirmed skin tone, and popularity within similar profiles until behavior data builds up. Measure what matters: click-through, add-to-cart, conversion and, for fashion especially, return rate. A recommender that lifts clicks but also lifts returns isn't working.

Step 5: Body type, fit and size

Skin tone is the visible feature, but "will this actually fit?" is where returns come from. It's a separate problem from color, and it needs its own data.
• Garment measurements, not just size labels. An "M" means different things in different brands. For each size, store measured dimensions (chest, waist, shoulder, length) where you can get them. Many merchants don't have these, so make collecting them part of seller onboarding.
• Start with a short questionnaire. Height, a weight range, the size the person usually wears in a couple of brands, and a fit preference (snug, regular, relaxed). It's cheap, private and a workable baseline.
• Learn from outcomes. Purchases that were kept, and returns tagged "too small" or "too large", are your best training data. A size recommender is often framed as predicting the size a person will keep, given their profile and the brand's history.
• Photo-based body measurement comes later. Pose estimation (MediaPipe Pose, for example) and body-shape models can estimate proportions from a photo. Accuracy depends on clothing, camera angle and pose, and full-body photos are very sensitive, so keep it optional and on-device, same rule as the skin tone photo.
• Be careful with "body type" labels. Shapes like "pear" or "rectangle" are styling shorthand, not measurements, and plenty of people dislike being labeled. Offer them as optional, editable hints and let measurements and behavior do the real work.

Step 6 (later): Virtual try-on

Virtual try-on is the feature everyone wants in the demo. Diffusion-based approaches such as IDM-VTON (from an ECCV 2024 paper on try-on "in the wild") can render a garment onto a person's photo with convincing detail.
It's also the heaviest part of the stack: GPU inference, noticeable latency, per-image cost, and a stricter privacy burden because you're handling full-body photos. My suggestion is to launch without it. Ship outfit boards and color-matched picks first, then add try-on once the core loop is earning its keep.
If you want users to share try-on images with friends or family for opinions, treat that as its own feature with its own risks. Use expiring links, label images as AI-generated, let users switch sharing off, and require them to confirm the uploaded photo is of themselves. Without that, you've built a tool that can be misused on other people's photos.

Fairness: test across skin tones, not on average

A skin tone feature that works well for lighter skin and poorly for darker skin isn't a minor bug. It's a broken product for a large part of your audience. So:
• Build a consented, diverse evaluation set labeled with a scale such as Monk.
• Report accuracy and failure rate per tone bucket, not just overall.
• Test across lighting conditions and devices, including low-end phones.
• Track the share of users who override the suggested tone, per bucket. A high override rate in one group is a red flag.
• Use positive, optional language in the UI ("colors that tend to flatter") and keep the override one tap away.

Privacy and compliance

Face photos are sensitive, so design to collect as little as possible:
• Process on the device. Run face landmarks and color sampling in the browser or app so the selfie never leaves the phone.
• Store derived values only. A few Lab numbers and an undertone label are far safer than an image.
• Ask for clear, specific consent, and make deleting the data as easy as giving it.
• Know your law. If you operate in India, the Digital Personal Data Protection Act and its Rules (notified in November 2025) phase in their main obligations over the following 12 to 18 months, according to this summary from Hogan Lovells. The Act expects consent that is specific and informed, and it can't be bundled with unrelated purposes. If you serve EU users, GDPR applies too. Check current dates with a lawyer, because this is moving.

A sensible build order

Phase What ships Why this order
1 Storefront, clean catalog, onboarding quiz with size and fit questions, manual skin tone picker Validates demand before any camera pipeline
2 Photo-based tone and undertone detection with confidence and override Removes friction once you know people use it
3 Embeddings, retrieval and learned re-ranker Needs real behavior data to be worth it
4 Virtual try-on, outfit builder Expensive, so earn it
A practical stack: a React or Next.js storefront plus a Flutter or React Native app, a FastAPI service for the models, Postgres with pgvector, and a small queue for batch embedding jobs.

Don't rebuild the boring half

Look back at the architecture diagram. Everything outside the recommender (catalog, size and color variants, cart, checkout, orders, returns and a mobile app) is standard commerce. It's necessary and well understood, but it's not what makes your platform different. Teams that build it all from scratch often spend months before a single user sees a color-matched recommendation.
That's the argument for starting from a ready-made fashion storefront and putting your engineering time into the AI layer. The base does need to understand apparel (variants per size and color, collections, returns) and, just as important, give your own services a way in through APIs or custom modules.
One option worth evaluating is a ready-made fashion and apparel e-commerce software solution that comes with a mobile app. (Disclosure: it's from the company that publishes this account.) Whichever base you look at, ask two things before committing: can you plug in your own ranking service, and can you export product data to generate embeddings?

The verdict

If I were building this today, I'd ship in this order: a clean catalog, a manual skin tone picker with a Monk-style swatch, and rule-based color matching. That alone tests whether shoppers care. Add the photo pipeline only after they do, and keep every AI suggestion optional, explainable and easy to override.
The technology is the easy part. The hard parts are honest framing, fair performance across skin tones, and respecting the data people hand you. Get those right and the rest is engineering.
If you've built something similar, tell us in the comments what broke first. My guess is lighting.

References

• Estimating skin tone and effects on classification performance in dermatology datasets (ITA in practice)
• Google's 10-shade skin tone scale (Monk Skin Tone Scale)
• Contrastive language and vision learning of general fashion concepts (FashionCLIP) and the model on Hugging Face
• Improving Diffusion Models for Authentic Virtual Try-on in the Wild (IDM-VTON)
• India's DPDP Act brought into force (Hogan Lovells)
• Building an AI based personalized fashion platform (r/Startup_Ideas) (a founder's discussion of the idea)

DEV Community: Originate Soft Pvt Ltd

How to Build an AI-Based Personalized Fashion Platform (Skin Tone, Color Matching and Recommendations)

The architecture at a glance

Step 1: Understand your catalog first

Step 2: Detect skin tone from a photo

Step 3: Build the color-matching engine

Step 4: Recommendations with embeddings

Step 5: Body type, fit and size

Step 6 (later): Virtual try-on

Fairness: test across skin tones, not on average

Privacy and compliance

A sensible build order

Don't rebuild the boring half

The verdict

References

References