Solving the Mixed Bin: Offline Visual Search for Apparel Robotics

First Published on Level Up Coding

A robot arm hovers over a tote during Myntra’s End of Reason Sale. Gripper open, waiting. Somewhere between its camera and an AWS endpoint two hundred kilometres away, a photo of the bin is stuck in a Wi-Fi dead zone created by six rows of steel shelving. By the time the embedding comes back, the conveyor has already moved the tote on. The arm picks air.

That is the whole problem in one sentence. Indian fashion e-commerce runs at a scale where fulfillment centers handle hundreds of orders per minute during peak events, and the giants already automate heavily. Myntra and Flipkart use Automated Guided Vehicles and conveyor systems to move racks and slide sealed packages down chutes. What they have not cracked is the actual pick, pulling one specific crumpled t-shirt out of a mixed tote to fulfill one specific order. That step is still mostly human.

This article is about closing that gap, and about the one architectural decision that makes it possible: the retrieval layer has to live on the robot. Not near the robot. On it. Everything below is built around that constraint, and all of it runs from a repository you can clone today.

Code for this article: github.com/rayl15/qdrant-edge-warehouse-robot. It runs on Qdrant Edge itself, end to end on a laptop with no GPU and no model downloads, and the same code is what you put on the robot.

Why fashion is the holy grail of robotic picking

Moving a rigid, identical cardboard box is a solved problem. The box is the same shape every time, so you can hand a robot fixed coordinates and a suction cup and walk away. Fashion breaks every assumption in that sentence.

Apparel is a deformable object. The same navy crew-neck t-shirt is a flat rectangle on a hanger, a soft lump when folded, and a shiny reflective blob when it is crumpled inside a polybag at the bottom of a tote. There is no fixed geometry to program against. A picking arm looking into a disorganized bin has to do three things in the time it takes the conveyor to pause. It has to parse chaotic visual data, recognize that a crumpled reflective shape is in fact the correct SKU, and work out a physical point where it can grip the item without tearing the packaging.

Barcode scanners do not help here, because the barcode is usually face-down or wrapped. Pre-programmed spatial grids do not help, because the item is never in the same pose twice. What the problem actually demands is zero-latency, zero-shot visual search. The robot needs to compare what its camera sees right now against a memory of the entire catalog, and it needs the answer in single-digit milliseconds.

That word memory is the useful frame. We are not building a classifier that has to be retrained every time Myntra adds a winter line. We are building a retrieval system, a searchable memory of what every product looks like, that the robot queries the way you query a database.

The metal cage: why cloud AI breaks down in a warehouse

The textbook way to build visual search looks reasonable on a whiteboard. The robot takes a high-resolution photo of the bin, sends it over Wi-Fi to a cloud API to extract embeddings, queries a hosted vector database, and receives the object data back. For a recommendation widget on a website, that round trip is fine. On a physical assembly line it fails for two independent reasons, and both of them are physics, not engineering.

The first is latency. A high-speed robotic arm moving along a line cannot wait two hundred to five hundred milliseconds for a cloud round trip. The physical state of the bin changes inside that window. Another item shifts, the tote rotates, the moment to grip has passed. Control loops for manipulation want to run at tens to hundreds of hertz, and a network hop blows straight through that budget.

The second reason is worse because you cannot buy your way out of it with a faster link. Massive fulfillment centers are, from a radio perspective, giant Faraday cages. Endless rows of steel shelving and metal totes reflect and absorb 2.4 and 5 gigahertz signals and create notorious dead zones. You can carpet the ceiling in access points and there will still be a pocket three shelves deep where a robot loses connectivity for the exact two seconds it needed to make a decision. A robot that turns into an expensive paperweight whenever it drives into a dead zone is not a robot you can deploy during your biggest sale of the year.

So the requirement is easy to state and hard to build. For real-time industrial robotics, both the inference and the retrieval layer have to live on the robot. Every decision happens offline.

Most teams get the first half of this. On-device inference is now normal, and a Jetson-class compute module can run a vision model locally without much drama. The half that gets forgotten is retrieval. People keep the vector database in the cloud out of habit, which quietly reintroduces the exact network dependency they just spent a lot of effort removing from the model. If the search is remote, the robot is still online-only. You have to move the index too.

The solution stack: in-process retrieval with Qdrant Edge
The piece that makes this practical is an embedded vector search engine. Qdrant Edge is the clearest example. The easiest way to think about it is as the SQLite of vector search. It is not a client that connects to a server, it is a search engine compiled as a library, in Rust with Python bindings, that runs directly inside your application’s process. The whole index is a directory on the robot’s local disk. When the process opens it, the index loads into RAM and stays resident there for the life of the process, so every query is a function call against memory rather than a request over a wire. There are no background optimizer threads, no update daemons, no connection to manage, and the install footprint is measured in single-digit megabytes.

This is the part I want to name properly, because it is the whole idea in four words. Pair that embedded index with a lightweight on-device Vision-Language Model and you get memory at the edge. The robot stops renting its recall from a data centre and starts carrying it. Its camera sees a crop, the VLM turns that crop into a vector, and the search compares that vector against a local index of every SKU. Perception and retrieval happen in the same process, on the same board, with the network cable unplugged.

There is no waiting list and no gate. pip install qdrant-edge-py works today, with wheels for macOS, Windows, and both x86 and aarch64 Linux, which is the one that matters because aarch64 is what a Jetson runs. It needs Python 3.10 or newer. Edge is still labelled beta, so pin the version and re-read the docs before you push to a fleet, but it is a package you can have on a board this afternoon.

Everything below is code I ran against that package, and every latency number is measured rather than quoted.

Picking the vision model

For the encoder, the workhorse is Apple’s MobileCLIP2. It is a family of CLIP-style image-text models built specifically for on-device inference. The important variant for this job is MobileCLIP2-S0. It has an image encoder of around eleven million parameters, encodes an image in single-digit milliseconds, and produces a 512-dimensional embedding that is cheap to index. Text and image land in the same vector space, which is what makes the whole thing zero-shot. You can describe a new SKU with a photo or a sentence and it drops straight into the index.

This is also where Mistral’s Robostral Navigate fits, and it helps to be precise about the division of labour. Robostral Navigate, released in July 2026, is an 8-billion-parameter vision-language-action model that reads an RGB frame plus a plain-language instruction and predicts where the robot should move next. It handles getting the robot to the tote, down the aisle and up to the pick station. It does not produce retrieval embeddings, so it is not the search engine. What it does have is the one thing the search engine also needs, a live camera feed.

So you run them together off the same camera. While Robostral is navigating, you sample frames from that video feed at intervals, and each sampled frame goes through MobileCLIP2 and into the Qdrant search. One model decides where to go, the other decides what it is looking at. Both are meant to run on the robot rather than in the cloud, which keeps the whole loop, move and recognize, offline. Navigation is the reason there is a camera feed at all, and visual search is what turns that feed into an understanding of the bin.

Modelling a garment as a set of vectors

Here is the modelling decision that most people get wrong, and the one that the second note in the brief for this piece pointed straight at. A SKU is not one vector. It is a set of vectors.

Think about why. If you store a single pristine product-shot embedding for the navy t-shirt, you are asking the robot to match a crumpled, half-occluded, glare-covered crop against a studio photo. In embedding space those two images can be surprisingly far apart. The single-vector index will happily hand you a confident wrong answer, because some other SKU’s product shot happened to sit closer to the messy crop than the correct SKU’s product shot did.

The fix is to store several reference views per SKU, front, back, folded, and crumpled, and let the vector database score against all of them at once. Qdrant supports this directly through multivector points using the MAX_SIM comparator. Each point carries a matrix of vectors instead of a single vector. At query time, MAX_SIM takes each query vector, finds its maximum similarity across all of a SKU’s stored views, and sums those maxima. A crumpled crop simply lights up whichever stored view is closest to it. That is the difference between matching the catalog photo and matching the item, however it happens to be lying in the bin.

This is the same late-interaction idea that ColBERT introduced for text retrieval, borrowed for apparel. Instead of one vector per token, we have one vector per view of a garment, and the fine-grained per-view matching is what buys the robustness.

That is easy to nod along to and hard to know whether it actually holds, so the repository pins it down as a test rather than a claim. Build two shards over the same two SKUs. One stores front views only. The other also stores a crumpled view of SKU-A. Then query both with a crumpled crop of A.

# tests/test_index.py, trimmed
query = _unit(0.35 * base_a + 1.0 * deform)    # a crumpled crop of A

single = EdgeShardIndex(tmp_path / "single", dim=64)
single.build([
    SkuRecord("SKU-A", "item a", np.stack([front_a])),               # front only
    SkuRecord("SKU-B", "item b", np.stack([front_b])),               # distractor
])

multi = EdgeShardIndex(tmp_path / "multi", dim=64)
multi.build([
    SkuRecord("SKU-A", "item a", np.stack([front_a, crumpled_a])),   # two views
    SkuRecord("SKU-B", "item b", np.stack([front_b])),
])

assert single.search(query, top_k=2)[0].sku == "SKU-B"   # wrong SKU wins
assert multi.search(query, top_k=2)[0].sku == "SKU-A"    # MAX_SIM recovers it

The scores are the part worth staring at. In the front-only index the distractor B wins at 0.60 while the correct answer A sits down at 0.182, which is exactly the confident wrong answer I described a paragraph ago. Add one crumpled reference view and A comes back at 0.982 and takes the top slot. The query did not change and neither did the encoder. The only thing that changed is what the index was allowed to remember about the garment.

I assumed when I started that this was the part I would have to give up at the edge, and that turned out to be wrong. Edge supports late interaction natively. MultiVectorConfig is right there in the embedded engine, and it is what turns an ordinary vector index into a late-interaction one, so nothing about the modelling above has to be watered down to fit on a robot.

Building the pipeline

The pick loop is small on purpose. The robot opens its shard, parses the bin into candidate garments, embeds each crop with the local vision model, and searches. Because retrieval never leaves the process, search is the cheapest step in the loop rather than the bottleneck it was in the cloud design.

First, initializing the Edge shard on the robot’s own disk. This is the line that makes the whole architecture real, so it is worth reading slowly. There is no host, no port, and no API key anywhere in it. EdgeShard.create takes a filesystem path, and on an NVIDIA Jetson that path is a directory on the module's local storage.

from qdrant_edge import (
    EdgeShard, EdgeConfig, EdgeVectorParams,
    Distance, MultiVectorConfig, MultiVectorComparator,
)

config = EdgeConfig(
    vectors={
        "vision": EdgeVectorParams(
            size=512,                          # MobileCLIP2-S0 embedding dim
            distance=Distance.Cosine,
            multivector_config=MultiVectorConfig(
                comparator=MultiVectorComparator.MaxSim
            ),
        )
    }
)

# /var/lib/mixed-bin on the Jetson. A directory, not a connection string.
shard = EdgeShard.create("/var/lib/mixed-bin", config)

Two things I got wrong on the first run, both worth inheriting from me rather than rediscovering. The directory has to exist before create is called, and create refuses to write over a shard that is already there, so a rebuild means clearing the path first. The second is nastier. Edge flushes to disk when the shard is dropped, so if you have already deleted the directory by then, it panics up from Rust with a stack trace that does not obviously point at your code. Close the shard before the directory goes away.

Each SKU then goes in as one point whose vector is a list of vectors, one row per reference view.

import numpy as np
from qdrant_edge import Point, UpdateOperation

points = []
for i, sku in enumerate(catalog):
    view_matrix = np.stack([embed(v) for v in sku.reference_views])  # (num_views, 512)
    points.append(
        Point(
            id=i,
            vector={"vision": view_matrix.tolist()},   # a matrix, not a single vector
            payload={"sku": sku.code, "title": sku.title},
        )
    )

shard.update(UpdateOperation.upsert_points(points))
shard.flush()

Second, the encoder. MobileCLIP2 comes through OpenCLIP, so loading it is a few lines. The one thing that trips people up is model.eval(), which is mandatory because MobileCLIP has BatchNorm layers and the output is unstable without it.

import torch, open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms(
    "hf-hub:timm/MobileCLIP2-S0-OpenCLIP"
)
model.eval()                                        # required: BatchNorm layers

def embed_image(image: Image.Image) -> "np.ndarray":
    tensor = preprocess(image.convert("RGB")).unsqueeze(0)
    with torch.no_grad():
        vec = model.encode_image(tensor)
    vec = vec / vec.norm(dim=-1, keepdim=True)      # L2 normalize for cosine
    return vec[0].cpu().numpy().astype("float32")

Third, scene parsing. The robot crops candidate garments out of the camera frame, and each crop carries a grip point, the pixel-space centre of its bounding box, which is what the arm’s inverse kinematics targets. The type that carries it is small enough to read in one go.

@dataclass(frozen=True)
class Crop:
    image: Image.Image
    # Bounding box in pixels: (left, top, right, bottom).
    box: tuple[int, int, int, int]

    @property
    def grip_point(self) -> tuple[int, int]:
        """Pixel-space centre of the box: where the arm goes."""
        left, top, right, bottom = self.box
        return (left + right) // 2, (top + bottom) // 2

In production you would use a detector like YOLO fine-tuned on your garment classes, and swapping one in means returning the same Crop objects from model.predict boxes, so nothing downstream notices. The point to hold onto is that detection gives you both the thing to search for and the place to grip.

Fourth, the sub-millisecond search. Each detected crop becomes a one-row query matrix. MAX_SIM scores it against every stored view of every SKU and returns the best match per SKU.

from qdrant_edge import Query, QueryRequest

def search(shard, query_vector, top_k=5):
    hits = shard.query(
        QueryRequest(
            query=Query.Nearest([query_vector.tolist()], using="vision"),
            limit=top_k,
            with_payload=True,                     # off by default, payload comes back None
        )
    )
    return [(p.payload["sku"], p.score) for p in hits]

Put the four together and the loop reads exactly like the physical action. The robot perceives the t-shirt, searches its local memory, and initiates the grip in one motion. The repository’s demo runs this with a deterministic stand-in encoder so you can see the wiring work before you download a single model weight.

Swap the stand-in for the real encoder and the claim holds on real photographs. scripts/demo_real.py indexes every view of each product except one, then queries with the held-out shot, so the query image is genuinely not in the index. On a sample catalog of nine products and twenty-five reference views, MobileCLIP2 and Edge get all nine right.

One honest note on that sample. Those nine products are footwear and accessories, not garments, because those are the multi-view listings Amazon Berkeley Objects ships permissively. Since apparel is the actual premise, the repository also carries a catalog of real crumpled t-shirts, shot wrinkled front and back, with a scripts/demo_crumpled.py that runs the same held-out test. Twenty-five near-identical tees is where a single reference view starts to strain, exactly as the previous section described.

How fast, honestly, and where it stops being sub-millisecond

Sub-millisecond is easy to assert, so the repo ships scripts/benchmark.py, run it rather than believe me. Apple silicon laptop, 512-dimensional vectors, four views per SKU:

Multivector search here is exact, so latency scales with catalog size instead of flattening the way an approximate index would. “Sub-millisecond” holds to roughly five thousand SKUs and not beyond, and I would rather write that down than benchmark only at the size that flatters the claim.

It also matters less than it looks. Fifty thousand SKUs still answer in about 30–70 faster than the cloud round trip it replaces while still being inside a manipulation control loop. For a six-figure catalog, shard by warehouse zone or trade exactness for an HNSW index through EdgeVectorParams(hnsw_config=...). A picking arm in aisle fourteen does not need the winter coats in aisle two.

One production detail worth building in from the start is a confidence floor. Cosine similarity below a threshold should not trigger a confident pick. It should route the item to a human. A robot that knows when it does not know is worth far more on the floor than one that grips at everything, and it really is a two-line change in the search step.

# src/mixed_bin/search.py
best = hits[0]
targets.append(
    PickTarget(
        sku=best.sku,
        confidence=best.score,
        grip_point=crop.grip_point,
        confident=best.score >= self.settings.min_confidence,   # the floor
    )
)
targets.sort(key=lambda t: t.confidence, reverse=True)          # best pick first

The floor defaults to 0.25 and is the one number you should tune against your own catalog rather than inherit from mine, because the right value depends on how close together your SKUs sit in embedding space. Twenty-five plain t-shirts that differ only in shade need a far higher floor than a mixed catalog of jackets, shoes, and handbags. A confident=False target is still a useful output, it just goes to a person instead of to the arm.

Zero-shot adaptability, or keeping up with fast fashion

Fashion seasons turn over fast, and this is where the retrieval framing pays for itself. A traditional image classifier needs thousands of labelled images and a retraining cycle every time a new clothing line lands. That cadence does not survive contact with a business that ships new SKUs weekly.

Because the system leans on a Vision-Language Model and vector similarity, it is zero-shot by construction. When a seller on Myntra introduces a new winter jacket, you do not retrain anything. You embed a few reference views of the jacket and upsert one new point into the index. The robot recognizes an item it has never been explicitly trained on, because recognition here is nearest-neighbour search in a shared space, not classification against a fixed label set. In the repository this is literally dropping a new folder of photos into the catalog directory and re-running the build. Adding an SKU is a database write, not a machine learning project.

That property also composes well with the async nature of a real catalog, and the fact that an Edge shard is a directory rather than a service is what makes it easy. Updating a fleet is a file sync. New products get embedded on a back-office machine when a new season lands, the rebuilt shard is pushed to each robot, and the compute-constrained board never does the heavy lifting of building an index, only querying one. Edge exposes snapshot_manifest and update_from_snapshot if you would rather ship differences than the whole directory.

Deploying it on the board

The split that matters in production is between building and querying, and it maps onto two phases. A back-office machine embeds the catalog and writes the shard. The robot opens that shard at boot and from then on only reads. EdgeShardIndex below is the repository's thin wrapper over the raw EdgeShard calls from earlier, and it exists mostly to handle the two footguns I mentioned, clearing the path before a rebuild and closing the shard before anything deletes it.

# Back office: embed the catalog, write the shard, close it cleanly.
with EdgeShardIndex("storage/mixed_bin", dim=512) as index:
    index.build(records)

# On the Jetson: open the shard that was synced down, then only ever query it.
index = EdgeShardIndex("storage/mixed_bin", dim=512)
index.load()
hits = index.search(query_vector, top_k=5)

For the vision model, the same board runs MobileCLIP2 offline once the weights are cached locally, and there is a community ONNX export if you want to push the encoder through TensorRT for extra headroom. Nothing in the hot path touches the network. The robot holds its own copy of the catalog, its own copy of the model, and makes every decision locally.

Sovereign agents and offline autonomy

This goes past one warehouse. The robots worth deploying are sovereign in a narrow, practical sense. They carry their own perception and their own memory instead of renting both from a data centre, so they keep working when the link drops. A picker that turns into a paperweight the moment the Wi-Fi flickers is not much use during peak seasons, like Diwali, when every second of uptime pays for the robot.

Offline visual search is a small, concrete version of that. Put the vision model on the robot, put the vector index next to it, store each deformable item as a set of views so the messy real world still matches, and the mixed bin stops being the job humans have to do by hand.

If you build industrial IoT or robotics, the useful next step is to time it yourself. Run pip install qdrant-edge-py, create an EdgeShard on a compute module you already have, load a few hundred of your own product photos, and measure a query with the network unplugged. The whole thing is on GitHub if you want the wiring, benchmark script included. My numbers came off a laptop. The ones that should convince you come off your hardware.

The part that surprised me was how little code it takes once the retrieval layer stops being a network call. The gap between a robot that needs the cloud and one that does not is smaller than it looks. It is mostly a matter of moving the index the last few centimetres, onto the machine that has to act.

References

• Qdrant Edge announcement: qdrant.tech/blog/qdrant-edge
• Qdrant Edge documentation and EdgeShard API: qdrant.tech/documentation/edge
• qdrant-edge-py on PyPI: pypi.org/project/qdrant-edge-py
• Qdrant multivector and MAX_SIM late interaction: qdrant.tech/documentation/concepts/vectors
• Apple MobileCLIP2: github.com/apple/ml-mobileclip
• Mistral Robostral Navigate: mistral.ai/news/robostral-navigate
• Code for this article: github.com/rayl15/qdrant-edge-warehouse-robot