DEV Community

Cover image for How to build image search with semantic understanding
Linghua Jin for CocoIndex

Posted on Edited on Originally published at cocoindex.io

How to build image search with semantic understanding

This post is also on cocoindex.io, with interactive figures. The code uses CocoIndex v1.

Search a folder of photos for "long neck" and a filename search returns nothing, unless someone named the giraffe photo that way.

This post builds image search that matches by meaning instead: you type "long neck" and the giraffe comes back, without captions or manual labels. Try it on the 4 photos that ship with the example. The scores are real CLIP scores.

The model that makes this work is CLIP. It embeds an image and a piece of text into the same vector space, so a text query and a matching picture land near each other. CocoIndex v1 walks a local image folder, embeds each picture with CLIP, and stores the vectors in Qdrant. The index runs in live mode inside a FastAPI server, so a photo you drop into the folder becomes searchable as soon as it is embedded, with no rebuild step.

The full code is in the CocoIndex repo and runs on CocoIndex v1.

Why search photos by meaning

You write the transformation in plain Python, target_state = transformation(source_state). CocoIndex handles the rest: the file watcher, change tracking, memoization, and a managed Qdrant collection, with change tracking and memoization running in its Rust engine.

The flow

Each image is its own processing component: read the bytes, embed them with the CLIP image encoder, and declare a Qdrant point. There is no text to chunk, so the indexing path is one embedding per image.

The shared image-text space

CLIP is trained to place a picture and its description near each other in one embedding space. At index time we only ever embed pixels (get_image_features); at query time we embed text (get_text_features). Both produce 768 floats in the same space, so "long neck" ends up closest to the giraffe photo even though that image was never described in words.

The vector database never learns that one vector came from an image and the other from text. It returns nearest neighbors by cosine similarity. A new image therefore needs no description to become searchable; adding the file to the folder is enough.

CLIP's image-to-text scores sit in a narrow band. In the post's demo, the best match for each query scores 0.21 to 0.23 and the other photos 0.11 to 0.19. Search ranks by those scores, so the order matters more than the absolute value.

One image, end to end

The stepper below follows a single photo through process_file before the code sections. Use the buttons, or let it play.

The rest of the post builds this pipeline in 3 pieces: the shared resources, the per-image function, and the app that connects them.

Technologies

CLIP ViT-L/14

CLIP ViT-L/14 is a vision-language model that embeds images and text into a shared 768-dimensional space. We use it two ways: to embed each image at index time, and to convert a natural-language query into that same space at search time.

The collection's vector size comes from model.config.projection_dim, so a smaller variant like CLIP ViT-B/32 (faster, slightly lower accuracy) needs only a new CLIP_MODEL_NAME, and the collection is created to match. A swap that changes the size (ViT-B/32 has 512 dimensions, against 768) makes CocoIndex drop and recreate the collection and re-embed every image on the next start. A swap that keeps the size is not noticed: the memo of process_file tracks the function's code, not the CLIP_MODEL_NAME constant. For that case, declare the dependency with @coco.fn(memo=True, deps=CLIP_MODEL_NAME), or stop the server and run cocoindex update pipeline.py --full-reprocess once. See memoization for what invalidates a memo.

CocoIndex

CocoIndex is an open-source data transformation framework for AI, with incremental processing built in. You declare the transformation in plain Python and your own types; change tracking and memoization run in a Rust engine underneath.

Qdrant

Qdrant is a vector database. We store the image vectors as points and query them by cosine distance through the CocoIndex Qdrant connector.

FastAPI

FastAPI serves the search API and hosts the live index in its lifespan, so one process both keeps the collection current and answers queries.

Prerequisites

  • Install Qdrant: docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant.
  • Python 3.11+ with torch, transformers, and pillow for CLIP. pip install -e . in the example folder installs them.

CocoIndex v1 keeps its own processing state in a local LMDB directory, set by COCOINDEX_DB (the example's .env sets it to ./cocoindex.db), so there is no separate tracking database to run.

Define the pipeline

The pipeline lives in pipeline.py. The excerpts below leave out the standard-library, torch, transformers, Pillow, NumPy, and qdrant_client imports; the linked file has them all. CocoIndex's Qdrant client is created in the CocoIndex lifespan and reached through the QDRANT_DB context key (api.py opens its own client for search):

```python title=pipeline.py
import cocoindex as coco
from cocoindex.connectors import localfs, qdrant
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
from cocoindex.resources.schema import VectorSchema

QDRANT_COLLECTION = "ImageSearch"
CLIP_MODEL_NAME = "openai/clip-vit-large-patch14"
TOP_K = 5

def qdrant_url() -> str:
return os.getenv("QDRANT_URL", "http://localhost:6334/")

QDRANT_DB = coco.ContextKeyQdrantClient
QDRANT_CLIENT = coco.ContextKeyQdrantClient






```python title=pipeline.py
@coco.lifespan
async def coco_lifespan(
    builder: coco.EnvironmentBuilder,
) -> AsyncIterator[None]:
    client = qdrant.create_client(qdrant_url(), prefer_grpc=True)
    builder.provide(QDRANT_DB, client)
    builder.provide(QDRANT_CLIENT, client)
    yield
Enter fullscreen mode Exit fullscreen mode

CLIP is loaded once and reused. The image encoder is an ordinary helper, so nothing about it is CocoIndex-specific. _projected_features handles a return-type change between transformers 4 and 5:

```python title=pipeline.py
@functools.cache
def get_clip_model() -> tuple[CLIPModel, CLIPProcessor]:
model = CLIPModel.from_pretrained(CLIP_MODEL_NAME)
processor = CLIPProcessor.from_pretrained(CLIP_MODEL_NAME)
return model, processor

def _projected_features(out: Any) -> torch.Tensor:
# transformers >=5 returns BaseModelOutputWithPooling with the projected features in pooler_output;
# transformers <5 returns the projected features tensor directly.
return out.pooler_output if hasattr(out, "pooler_output") else out






```python title=pipeline.py
def embed_image_bytes(img_bytes: bytes) -> list[float]:
    model, processor = get_clip_model()
    image = Image.open(io.BytesIO(img_bytes)).convert("RGB")
    inputs = processor(images=image, return_tensors="pt")
    with torch.no_grad():
        out = model.get_image_features(**inputs)
    return _projected_features(out)[0].tolist()
Enter fullscreen mode Exit fullscreen mode

One component per image

process_file handles a single image: read its bytes, embed them, and declare a Qdrant point keyed by a stable uuid5 of the path. memo=True skips an image whose bytes did not change, and because each image is its own component, deleting a file removes its point:

```python title=pipeline.py
@coco.fn(memo=True)
async def process_file(
file: FileLike,
target: qdrant.CollectionTarget,
) -> None:
content = await file.read()
embedding = embed_image_bytes(content)
row_id = _image_id(file.file_path.path)
point = qdrant.PointStruct(
id=row_id,
vector=embedding,
payload={"filename": str(file.file_path.path)},
)
target.declare_point(point)




`_image_id` is one line, `str(uuid.uuid5(uuid.NAMESPACE_URL, str(path)))`. The same path always produces the same point ID, so re-embedding an edited photo overwrites its point instead of adding a second one.

### Wire up the app

`app_main` creates the Qdrant collection (its vector size comes straight from CLIP's `projection_dim`), walks the image folder in live mode, and mounts one `process_file` per image with `mount_each`:



```python title=pipeline.py
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    model, _ = get_clip_model()
    dim: int = model.config.projection_dim  # type: ignore[assignment]

    target_collection = await qdrant.mount_collection_target(
        QDRANT_DB,
        collection_name=QDRANT_COLLECTION,
        schema=await qdrant.CollectionSchema.create(
            vectors=qdrant.QdrantVectorDef(
                schema=VectorSchema(dtype=np.dtype(np.float32), size=dim),
                distance="cosine",
            )
        ),
    )

    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(
            included_patterns=["**/*.jpg", "**/*.jpeg", "**/*.png"]
        ),
        live=True,  # source supports live watch; api.py runs the app with live=True
    )
    await coco.mount_each(process_file, files.items(), target_collection)

app = coco.App(
    coco.AppConfig(name="ImageSearchQdrantV1"),
    app_main,
    sourcedir=pathlib.Path("./img"),
)
Enter fullscreen mode Exit fullscreen mode
  • walk_dir(..., live=True) lists the matching files and, when the app runs live, keeps watching the folder. See the local file system connector for how live watching works.
  • files.items() keys each file by its path relative to img/. That key names the file's component, so CocoIndex can tell which files are new, which changed, and which disappeared.
  • mount_collection_target creates and manages the Qdrant collection: the schema, upserts, and removal of points whose target state is no longer declared.

Query the index

Search embeds the text with the same CLIP model, this time through the text encoder, and asks Qdrant for the nearest image vectors:

```python title=pipeline.py
def embed_query(text: str) -> list[float]:
model, processor = get_clip_model()
inputs = processor(text=[text], return_tensors="pt", padding=True)
with torch.no_grad():
out = model.get_text_features(**inputs)
return _projected_features(out)[0].tolist()




The FastAPI `/search` endpoint embeds the query and returns the top matches with their scores. `_qdrant_search` in `pipeline.py` calls whichever search method the installed `qdrant-client` version has:



```python title=api.py
@app.get("/search")
async def search(
    q: str = Query(..., description="Search query"),
    limit: int = Query(5, description="Number of results"),
) -> dict[str, Any]:
    query_embedding = pipeline.embed_query(q)

    if _client is None:
        raise RuntimeError("Qdrant client is not initialized.")

    results = pipeline._qdrant_search(
        _client,
        pipeline.QDRANT_COLLECTION,
        query_embedding,
        limit,
    )

    return {
        "results": [
            {
                "filename": (r.payload or {}).get("filename"),
                "score": r.score,
            }
            for r in results
        ]
    }
Enter fullscreen mode Exit fullscreen mode

Run it as a live service

There is no separate build step: api.py starts the app in live mode inside the FastAPI lifespan. It blocks startup until the first sweep reports READY, so the collection is queryable, then keeps watching img/ in the background while it serves requests:

```python title=api.py
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]: # type: ignore[override]
global _client
async with coco.runtime():
_client = qdrant.create_client(pipeline.qdrant_url(), prefer_grpc=True)

    # Start a live update; block startup until the initial sweep finishes so
    # the collection is queryable, then keep it running in the background.
    update_handle = pipeline.app.update(live=True)
    async for snap in update_handle.watch():
        if snap.status is coco.UpdateStatus.READY:
            break
    update_task = asyncio.create_task(update_handle.result())

    try:
        yield
    finally:
        update_task.cancel()
        with contextlib.suppress(asyncio.CancelledError):
            await update_task
        _client = None
Enter fullscreen mode Exit fullscreen mode



## Changes in the folder reach the index

Once the server is up, every change to a `.jpg`, `.jpeg`, or `.png` file in `img/` reaches the collection without a command (the patterns are case-sensitive, so `IMG_001.JPG` is skipped). The interactive demo on cocoindex.io shows what each kind of change costs.

What decides each outcome:

- **New file.** The watcher reports `created`, `mount_each` mounts a component for the new path, and its memo has no entry, so CLIP runs. A file still being copied in may fire several events; each one goes through the same memo check.
- **Same bytes, new modification time.** A `touch`, or any write of identical bytes, reports a change, usually `modified`. The memo state of a local file is its modification time plus a fingerprint of its bytes. The time changed, so CocoIndex fingerprints the bytes, finds them unchanged, and skips the function. The point stays as it is.
- **New bytes.** An edited photo has a new fingerprint, so `process_file` runs again and the point is overwritten under the same ID.
- **Rename.** A move is handled as a delete of the old path and a create of the new one. The memo is keyed by the path, so the renamed photo is embedded again and gets a new point ID; the old point is deleted.
- **Delete.** The component for the path is removed, and CocoIndex deletes the point it declared.

Code changes invalidate the memo too: changing the logic of `process_file` (comments and formatting do not count) re-runs it for every image on the next start. Edits to plain helpers it calls, like `embed_image_bytes`, are not tracked. As a safety net against a watcher that silently stops (macOS FSEvents can), `walk_dir` rescans the whole folder once an hour by default; set `rescan_interval` to change that.

## Run the example

From a clone of the [CocoIndex repo](https://github.com/cocoindex-io/cocoindex), start Qdrant, install the example, and run the server. The example ships an `img/` folder with a cat, a dog, an elephant, and a giraffe:



```sh
cd cocoindex/examples/image_search
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant
cp .env.example .env     # QDRANT_URL (defaults to the local container above)
pip install -e .
python -m uvicorn api:app --reload --host 0.0.0.0 --port 8000
Enter fullscreen mode Exit fullscreen mode

Then run the frontend and open it in the browser:

cd frontend && npm install && npm run dev   # http://localhost:5173
Enter fullscreen mode Exit fullscreen mode

Query "elephant" and the elephant ranks first; query "long neck" and the giraffe wins, then the other animals fall in line by CLIP similarity. The images have no tags or captions:

Now drop another image into the img/ folder, for example this squirrel. Because the folder is watched in live mode, it becomes searchable once CLIP has embedded it, with no command to re-run. Search "a bushy tail" and it ranks first.

Acknowledgement

This image search project was largely contributed by @par4m, Param Arora, an open source developer at Google Summer of Code.

Top comments (0)