DEV Community

Cover image for How face recognition works: embeddings, thresholds and why you never store faces
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

How face recognition works: embeddings, thresholds and why you never store faces

#facerecognition #embeddings #privacy #computervision

Face recognition is four steps, and only one of them is recognition. Understanding the pipeline is what lets you build one responsibly — which for FaceVision meant a system where the server has never seen a face.

The four steps

  1. Detection — find faces in the frame and return bounding boxes and landmarks.
  2. Alignment — rotate and crop each face to a canonical pose using the landmarks, so the next model sees consistent input.
  3. Embedding — a recognition model maps the aligned face to a vector, typically 128 to 512 numbers, such that the same person's faces land close together.
  4. Matching — compare the new embedding to stored ones with cosine similarity and decide with a threshold.

Choosing the threshold

Too strict and real users are rejected; too loose and strangers get in. Collect a small labelled set — pairs of the same person, pairs of different people — and plot the similarity distributions. The threshold sits where the curves separate; where they overlap is your error rate, and you should state it. Lighting, glasses and age change the distribution, so test with your actual users, not a benchmark.

Why you store embeddings, never images

An embedding is enough to verify a person and useless for showing anyone what they look like. Storing only embeddings — the FaceVision design — means a database breach leaks vectors rather than faces, consent conversations get simpler, and deletion is a single row. Run detection and embedding in the browser with ONNX Runtime Web and the raw image never leaves the device at all (see running ML models in the browser).

Failure modes to design for

  • A printed photo held up to the camera — this is why liveness detection exists.
  • Twins and look-alikes — accept that similarity is not identity and add a second factor for high-stakes actions.
  • Bias — recognition models perform unevenly across skin tones and ages; measure per group before deployment.

Build it this way and you can explain your system to a privacy officer in two sentences. That is a competitive advantage, not just an ethical one.

About Pranjul Rathour

Pranjul Rathour presenting on stage in a blue polo, with his Annapurna demo video on the screen behind him
Presenting Annapurna on stage

Pranjul Rathour giving a talk titled 'How and what I do', with demo videos of his products Vaidya and Annapurna on screen
Talking through the products he has shipped

Pranjul Rathour in a black t-shirt holding a microphone in front of a chequered wall
On the mic

Pranjul Rathour pitching with a microphone in front of a projector slide at CSJMU, Kanpur
Pitching at CSJMU, Kanpur

Pranjul Rathour speaking from the podium at VSICS, Kanpur
At the VSICS podium, Kanpur

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-06.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)