DEV Community

Cover image for Multimodal Face Recognition Pipeline with CocoIndex: Real-Time Image and Vector Search
Linghua Jin for CocoIndex

Posted on Edited on Originally published at cocoindex.io

Multimodal Face Recognition Pipeline with CocoIndex: Real-Time Image and Vector Search

This post is also on cocoindex.io, with interactive figures. The code uses CocoIndex v1.

A folder of group photos is full of people, but a file index sees only filenames and bytes. Ask it for "every photo with this person in it" and it has nothing to compare.

This post builds the index that can answer that question. It finds every face in every photo, turns each face into 128 numbers, and stores one Qdrant point per face, with the filename and the face's box. A face you search with then comes back with the photos it appears in and where it is in each one. The interactive demo on cocoindex.io uses the 1927 Solvay Conference photo, one of the 4 images that ship with the example. The boxes are the real output of the example's code.

Detection and embedding come from the face_recognition library, which wraps dlib. CocoIndex v1 runs the pipeline around it: it walks the image folder, runs each new or changed photo once, fans each photo out into one point per face, and removes a photo's points when the photo leaves the folder. The full code is in the CocoIndex repo.

What a face index lets you build

Face embeddings identify people, and under laws such as the EU's GDPR (Article 9) and Illinois' BIPA they can count as biometric data. Index only photos you have the right to process this way.

Image in, one point per face out

A text pipeline splits a document into chunks. This pipeline splits a photo into faces, so 1 image becomes N points:

  • Walk images/ and match .jpg, .jpeg and .png files. Each file becomes its own processing component, keyed by its path.
  • Detect every face with dlib's CNN detector, and crop each one out of the full-size photo.
  • Embed each crop into a 128-float vector, and declare one Qdrant point per face. The point ID is a uuid5 of the filename and the box, and the payload carries both.

You write each step as a plain Python function, target_state = transformation(source_state). CocoIndex tracks which files changed, skips unchanged photos through memoization, and keeps the Qdrant collection in step with the folder.

One photo, end to end

The stepper follows the Solvay photo through the 3 functions that do the work. Step with the buttons or the arrow keys.

The rest of the post shows the code in the order it runs.

Prerequisites

  • Qdrant on the default ports: docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant.
  • Python 3.11 or later. face_recognition needs dlib, which builds from source on some platforms; its installation notes cover CMake and the other build tools.

CocoIndex v1 keeps its own processing state in a local directory set by COCOINDEX_DB (the example's .env.example sets ./cocoindex.db), so you do not run a separate database for it.

Define the pipeline

Everything is in main.py. It starts with the imports, the constants and the Qdrant client's context key:

```python title=main.py
from future import annotations

import io
import os
import pathlib
import sys
import uuid
from dataclasses import dataclass
from typing import Any, AsyncIterator

import face_recognition
import numpy as np
from PIL import Image
from qdrant_client import QdrantClient

import cocoindex as coco
from cocoindex.connectors import localfs, qdrant
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
from cocoindex.resources.schema import VectorSchema

QDRANT_COLLECTION = "face_embeddings"
FACE_EMBED_DIM = 128 # face_recognition (dlib) face encodings are 128-d
MAX_IMAGE_WIDTH = 1280 # the CNN detector is slow on large images; downscale first
TOP_K = 5

def qdrant_url() -> str:
return os.getenv("QDRANT_URL", "http://localhost:6334/")

QDRANT_DB = coco.ContextKeyQdrantClient




`MAX_IMAGE_WIDTH` bounds the cost of detection: the CNN detector is slow on large images, so wider photos are shrunk for it. A face is a box plus the cropped pixels:



```python title=main.py
@dataclass
class ImageRect:
    min_x: int
    min_y: int
    max_x: int
    max_y: int

@dataclass
class Face:
    """One detected face: its bounding box and the cropped PNG."""

    rect: ImageRect
    image: bytes
Enter fullscreen mode Exit fullscreen mode

Detect and crop the faces

extract_faces runs the detector on a copy that is at most 1280 pixels wide, then multiplies each box by ratio so it lands on the original photo. The crop comes from the original, so the embedding sees every pixel of the face:

```python title=main.py
@coco.fn.as_async(runner=coco.GPU)
def extract_faces(content: bytes) -> list[Face]:
"""Detect faces in an image and return each as a (bounding box, cropped PNG)."""
orig = Image.open(io.BytesIO(content)).convert("RGB")

# Downscale large images for the detector, then map boxes back to full size.
if orig.width > MAX_IMAGE_WIDTH:
    ratio = orig.width / MAX_IMAGE_WIDTH
    small = orig.resize(
        (MAX_IMAGE_WIDTH, int(orig.height / ratio)),
        resample=Image.Resampling.BICUBIC,
    )
else:
    ratio = 1.0
    small = orig

faces: list[Face] = []
for top, right, bottom, left in face_recognition.face_locations(
    np.array(small), model="cnn"
):
    rect = ImageRect(
        min_x=int(left * ratio),
        min_y=int(top * ratio),
        max_x=int(right * ratio),
        max_y=int(bottom * ratio),
    )
    buf = io.BytesIO()
    orig.crop((rect.min_x, rect.min_y, rect.max_x, rect.max_y)).save(
        buf, format="PNG"
    )
    faces.append(Face(rect=rect, image=buf.getvalue()))

return faces
Enter fullscreen mode Exit fullscreen mode



`face_locations` returns boxes as `(top, right, bottom, left)`; `ImageRect` stores them as `min_x`, `min_y`, `max_x`, `max_y`. On the Solvay photo (2560 × 1853), the detector sees 1280 × 926 and finds 29 faces.

### Embed one face

`embed_face` passes the whole crop to dlib's encoder as the face location and returns 128 floats:



```python title=main.py
@coco.fn.as_async(runner=coco.GPU)
def embed_face(face_png: bytes) -> list[float]:
    """Embed a single cropped face into a 128-d vector."""
    img = Image.open(io.BytesIO(face_png)).convert("RGB")
    arr = np.array(img)
    encoding = face_recognition.face_encodings(
        arr, known_face_locations=[(0, img.width - 1, img.height - 1, 0)]
    )[0]
    return encoding.tolist()
Enter fullscreen mode Exit fullscreen mode

Both functions are synchronous dlib calls, so they are declared with @coco.fn.as_async(runner=coco.GPU). Callers await them, and the GPU runner runs them in a worker thread, one call at a time per GPU it finds (a machine without a GPU counts as 1).

One point per face

process_face embeds a face and declares its point. The ID comes from _face_id, a uuid5 of the filename and the box, so the same box in the same file always maps to the same point:

```python title=main.py
def _face_id(filename: str, rect: ImageRect) -> str:
key = f"{filename}|{rect.min_x},{rect.min_y},{rect.max_x},{rect.max_y}"
return str(uuid.uuid5(uuid.NAMESPACE_URL, key))

@coco.fn
async def process_face(
face: Face,
filename: str,
target: qdrant.CollectionTarget,
) -> None:
"""Embed one face and declare it as a Qdrant point."""
embedding = await embed_face(face.image)
target.declare_point(
qdrant.PointStruct(
id=_face_id(filename, face.rect),
vector=embedding,
payload={
"filename": filename,
"min_x": face.rect.min_x,
"min_y": face.rect.min_y,
"max_x": face.rect.max_x,
"max_y": face.rect.max_y,
},
)
)




`declare_point` does not write anything itself. It states which point should exist, and CocoIndex creates, updates or deletes points to match after the component finishes (see [target states](https://cocoindex.io/docs/programming_guide/target_state/)).

### One component per photo

`process_file` handles one image: it detects the faces, then runs `process_face` for each with [`coco.map`](https://cocoindex.io/docs/programming_guide/processing_component/#using-cocoindex-map-to-apply-transformations):



```python title=main.py
@coco.fn(memo=True)
async def process_file(file: FileLike, target: qdrant.CollectionTarget) -> None:
    """Detect every face in one image and index each one."""
    faces = await extract_faces(await file.read())
    filename = str(file.file_path.path)
    await coco.map(process_face, faces, filename, target)
Enter fullscreen mode Exit fullscreen mode
  • coco.map runs the calls concurrently inside the current component. It creates no child components, so all the points of one photo belong to that photo's component, and they are synced together.
  • memo=True memoizes the whole component. While a photo and the code are unchanged, the photo skips detection and embedding, and its points are carried over as they are.

Wire up the app

The lifespan creates one Qdrant client for the app, and app_main declares the collection and mounts one process_file per image:

```python title=main.py
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.provide(QDRANT_DB, qdrant.create_client(qdrant_url(), prefer_grpc=True))
yield

@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
target_collection = await qdrant.mount_collection_target(
QDRANT_DB,
collection_name=QDRANT_COLLECTION,
schema=await qdrant.CollectionSchema.create(
vectors=qdrant.QdrantVectorDef(
schema=VectorSchema(dtype=np.dtype(np.float32), size=FACE_EMBED_DIM),
distance="euclid", # dlib face encodings compare by Euclidean distance
)
),
)

files = localfs.walk_dir(
    sourcedir,
    recursive=True,
    path_matcher=PatternFilePathMatcher(
        included_patterns=["**/*.jpg", "**/*.jpeg", "**/*.png"]
    ),
    live=True,  # api supports live watch; pass -L to `cocoindex update` to run live
)
await coco.mount_each(process_file, files.items(), target_collection)
Enter fullscreen mode Exit fullscreen mode

app = coco.App(
coco.AppConfig(name="FaceRecognitionV1"),
app_main,
sourcedir=pathlib.Path("./images"),
)




- **`distance="euclid"`**: dlib's encodings are compared by Euclidean distance. `face_recognition` treats 2 faces at a distance of 0.6 or less as the same person by default (the `tolerance` of `compare_faces`).
- **`files.items()`** keys each file by its path relative to `images/`. That key names the file's component, so CocoIndex can tell which files are new, changed or gone.
- **`live=True`** lets the source keep watching the folder when the app runs in live mode (`cocoindex update -L main`). A plain `cocoindex update main` processes what changed and exits.

## Run it

From a clone of the [CocoIndex repo](https://github.com/cocoindex-io/cocoindex):



```sh
cd cocoindex/examples/face_recognition
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant
cp .env.example .env     # QDRANT_URL (defaults to the local container above)
pip install -e .
cocoindex update main    # or: cocoindex update -L main  (keep watching the folder)
Enter fullscreen mode Exit fullscreen mode

The example ships 4 photos in images/. The first run finds 36 faces: 29 in the Solvay photo, 3 in Carter_welcomes_Reagan.jpg, and 2 each in the Steve Jobs and Bill Gates photo and in einplanck3.jpg. Each face is one point in the face_embeddings collection.

What an edit to the folder costs

Run the update again after a change, and CocoIndex runs only the photos that changed. Each button below is one edit followed by cocoindex update main:

What decides each outcome:

  • New photo. A new path mounts a new component with no memo entry, so extract_faces runs once and embed_face runs once per face.
  • Same bytes, new modification time. The memo state of a file is its modification time plus a fingerprint of its bytes. The time changed, so CocoIndex fingerprints the bytes, finds them unchanged, and skips the component. Its points stay.
  • New bytes. The component runs again: one detection, and one embedding for every face in the photo, including faces that did not change, because neither process_face nor embed_face is memoized. Cropping the left or top edge shifts every box, so every face gets a new ID; the new points are upserted and the old ones, no longer declared, are deleted.
  • Rename. The old path's component is gone, so its points are deleted. The new path has no memo entry and runs in full, and since the filename is part of the ID, its points get new IDs.
  • Delete. The component is removed, and CocoIndex deletes all of its points.

A change to the logic of process_file, or of the @coco.fn functions it calls (extract_faces, process_face, embed_face), invalidates the memo too, and the next update re-runs every photo. Comments and formatting do not count.

Search by face

The original version of this post sketched the goal: a face found in one photo, looked up in another.

main.py also has a small query command. It detects the faces in a query image, embeds the first one it finds with the same dlib encoder (inside the whole photo rather than as a crop), and asks Qdrant for the 5 nearest points:

```python title=main.py
def _qdrant_search(
client: QdrantClient, query_vector: list[float], limit: int
) -> list[Any]:
return client.query_points(
collection_name=QDRANT_COLLECTION,
query=query_vector,
limit=limit,
with_payload=True,
).points

def query(image_path: str, *, top_k: int = TOP_K) -> None:
content = pathlib.Path(image_path).read_bytes()
img = Image.open(io.BytesIO(content)).convert("RGB")
arr = np.array(img)
locs = face_recognition.face_locations(arr, model="cnn")
if not locs:
print("No face found in the query image.")
return
query_vec = face_recognition.face_encodings(arr, known_face_locations=locs[:1])[
0
].tolist()

client = qdrant.create_client(qdrant_url(), prefer_grpc=True)
for r in _qdrant_search(client, query_vec, top_k):
    payload = r.payload or {}
    # Euclidean distance: smaller is more similar (same person typically < 0.6).
    print(f"[{r.score:.3f}] {payload.get('filename')}  {payload}")
Enter fullscreen mode Exit fullscreen mode





```sh
python main.py query images/einplanck3.jpg
Enter fullscreen mode Exit fullscreen mode

einplanck3.jpg shows Max Planck and Albert Einstein, who both sit in the front row of the Solvay photo. The interactive demo on cocoindex.io runs the query for each of them against the 36 indexed faces.

The results show how to use this kind of index:

  • The photo's own crop comes first. einplanck3.jpg is in the index too, and its stored crop of the query face is 0.230 away for Planck and 0.187 for Einstein, not 0, because the query encodes the face inside the whole photo and the index encodes the tight crop.
  • Small, old faces crowd together. Most Solvay faces are about 80 pixels wide in a 1927 scan. Planck's face in the Solvay photo is 0.472 away, 0.009 behind a different man. Einstein's is 0.587 away, under 0.6, but 11 points are closer.
  • Ranked results are candidates. Show the top hits with their boxes and let a person confirm, or require a tighter distance before you treat a hit as a match.

To go further, search with every face in a photo instead of the first one, filter the hits by filename in the payload, or draw each hit's box on its photo.

The face recognition walkthrough in the docs covers the same example, and the image search post shows a query API and a live index inside a FastAPI server.

Top comments (0)