Single-Stage Vector Filtering for AgriTech Mandi Logistics

Kritika Singh — Fri, 02 Oct 2026 11:51:07 +0000

Single-Stage Vector Filtering for AgriTech Mandi Logistics

Benchmarking pre-filtering, post-filtering, and Qdrant’s single-stage in-index filtering on a 120,000-listing Indian AgriTech dataset

The Problem: Vector Search Alone Isn’t Enough

Vector search is very good at answering questions such as:

“Find produce similar to this description.”

But real logistics systems rarely stop there.

A buyer may want produce that is semantically similar and satisfies strict operational constraints:

transit time ≤ 8 hours
cold storage available
moisture below a threshold
specific state
tax exemption enabled

The interesting engineering problem is therefore not simply semantic search.

It is:

Semantic similarity + hard metadata constraints = usable search result

This article builds that problem from scratch and compares three ways of solving it:

Legacy pre-filtering
Legacy post-filtering
Qdrant single-stage in-index filtering

The experiment uses 120,000 synthetic-but-structurally-realistic mandi listings, identical vectors, identical HNSW parameters, four filter-selectivity levels, 60 queries per scenario, and exact brute-force ground truth.

Code and complete benchmark pipeline:
https://github.com/ks-1220/mandi-single-stage-filtering/

1. When Similarity Isn’t Enough

Vector similarity captures semantic relevance, while metadata filters enforce hard operational constraints in mandi logistics.

Imagine a buyer searching for:

“firm, fresh tomatoes suitable for hotel supply”

A vector database can rank listings according to semantic similarity.

But suppose the buyer also requires:

constraints = {
    "transit_hours": "<= 8",
    "cold_storage_available": True,
    "state": "Maharashtra",
    "tax_exempt": True,
}

Now the search has two separate dimensions:

Semantic side → vector search

Constraint side → payload filter

The difficulty comes from the fact that approximate nearest-neighbor search does not examine every vector.

This experiment therefore asks one specific question:

Can we enforce increasingly selective metadata filters without sacrificing vector-search recall or causing latency to explode?

2. Meet the Mandi Search Problem

Semantic Similarity vs. Hard Constraints

The dataset represents a mandi listing using both free text and structured fields. A simplified record looks like:

listing = {
    "id": 1042,
    "crop": "Tomato",
    "mandi": "Junnar",
    "state": "Maharashtra",
    "grade": "A",
    "description": (
        "fresh tomato grade A firm red produce "
        "suitable for hotel supply"
    ),
    "transit_hours": 4.5,
    "cold_storage_available": True,
    "moisture_pct": 12.4,
    "tax_exempt": True,
}

The description is converted into a vector. The remaining fields become metadata.

So the architecture looks like:

Mandi listing

→ description → embedding

→ metadata → payload indexes

This separation is important.

The vector answers:

“How similar is this listing?”

The payload answers:

“Is this listing actually allowed?”

3. What We’re Actually Going to Benchmark

There are three approaches.

a. Pre-Filtering

Conceptually:

valid_points = filter(dataset)

search(
    graph=valid_points
)

The problem is that removing non-matching nodes can also remove graph connections that the HNSW traversal depends on.

b. Post-Filtering

Conceptually:

candidates = search(vector, top_k=K * overfetch)

results = [
    x for x in candidates
    if satisfies_filter(x)
][:K]

If the filter is highly selective, K * 5 may not contain ten valid results. So the system has to retry with more candidates.

c. Single-Stage Filtering

The third approach combines the vector query and payload filter:

client.query_points(
    collection_name="mandi_listings",
    query=query_vector,
    query_filter=metadata_filter,
    limit=10,
)

The filter becomes part of the search operation rather than a completely separate application-level step.

That is the behavior this experiment is testing.

Why Qdrant?

We chose Qdrant because this experiment focuses on filter-aware vector search: combining semantic similarity with strict metadata constraints in the same retrieval process.

Qdrant’s HNSW-based search supports payload filtering, with payload indexes for fields such as state, crop, transit_hours, and moisture_pct.

This makes Qdrant a suitable system for comparing single-stage filtering against traditional pre-filtering and post-filtering approaches, especially as the filter becomes increasingly selective.

4. Building the 120,000-Listing Mandi Dataset

A mandi listing is transformed into a 128-dimensional semantic vector and structured payload fields for combined vector and metadata search.

The repository deliberately separates data generation from the benchmark.

That means the experiment can later replace the synthetic dataset with an Agmarknet/eNAM CSV without changing the downstream benchmark pipeline.

The generator starts with real crop and mandi reference lists.

For example:

CROPS = {
    "Onion": [
        "Lasalgaon",
        "Pimpalgaon",
        "Nashik",
        "Rahata"
    ],
    "Potato": [
        "Agra",
        "Farrukhabad",
        "Hooghly",
        "Jalandhar"
    ],
    "Tomato": [
        "Kolar",
        "Madanapalle",
        "Narayangaon",
        "Junnar"
    ],
    "Wheat": [
        "Karnal",
        "Kota",
        "Indore",
        "Sehore"
    ],
    "Paddy (Rice)": [
        "Karnal",
        "Kaithal",
        "Raipur",
        "Bardhaman"
    ],
}

Then the generator creates logistics attributes with crop-aware distributions.

For example:

if crop in {"Wheat", "Paddy (Rice)"}:
    moisture_pct = float(
        np.clip(
            np.random.normal(12.5, 1.5),
            6,
            22
        )
    )

elif crop in GRAIN_CROPS:
    moisture_pct = float(
        np.clip(
            np.random.normal(10.0, 2.2),
            4,
            20
        )
    )

else:
    moisture_pct = float(
        np.clip(
            np.random.normal(78.0, 8.0),
            40,
            95
        )
    )

The final row contains both semantic and filtering information:

rows.append({
    "id": i,
    "crop": crop,
    "mandi": mandi,
    "state": state,
    "grade": grade,
    "description": desc,
    "transit_hours": round(transit_hours, 2),
    "cold_storage_available": cold_storage,
    "moisture_pct": round(moisture_pct, 2),
    "tax_exempt": tax_exempt,
    "price_per_quintal_inr": round(price_per_qtl, 2),
    "quantity_quintal": qty_qtl,
    "perishable": perishable,
})

So the benchmark is not operating on random vectors with random metadata.

It creates a dataset with an actual search shape:

                    120,000 listings
                           |
                           |
              +------------+------------+
              |            |            |
             text       logistics    compliance
              |            |            |
             crop        transit        tax
             grade       storage        state
             quality     moisture

5. Turning Mandi Listings into Vectors

For this benchmark, the embedding pipeline intentionally uses a local deterministic method:

description
    ↓
TF-IDF
    ↓
Truncated SVD
    ↓
128-dimensional vector

The implementation is:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.preprocessing import normalize

DIM = 128

vectorizer = TfidfVectorizer(
    max_features=20000,
    ngram_range=(1, 2),
    sublinear_tf=True
)

tfidf = vectorizer.fit_transform(
    df["description"]
)

svd = TruncatedSVD(
    n_components=DIM,
    random_state=42
)

emb = svd.fit_transform(tfidf)

emb = normalize(
    emb
).astype(np.float32)

np.save(
    DATA_DIR / "embeddings.npy",
    emb
)

This is deliberately not positioned as a production embedding model.

The point of the experiment is filtering.

As long as the vectors provide enough semantic structure to perform meaningful nearest-neighbor search, the filtering strategies can be compared.

The repository also performs a sanity check.

The reported mean cosine similarity is approximately:

same crop → 0.277
different crop → 0.093

So the representation contains enough crop-related structure for the benchmark to be meaningful.

6. Setting Up the HNSW Search Layer

Both legacy arms use the same HNSW configuration.

def build_hnsw(emb):
    n, dim = emb.shape

    index = hnswlib.Index(
        space="cosine",
        dim=dim
    )

    index.init_index(
        max_elements=n,
        ef_construction=200,
        M=16
    )

    index.add_items(
        emb,
        np.arange(n)
    )

    index.set_ef(64)

    return index

The important variables are:

M = 16
EF_CONSTRUCTION = 200
EF_SEARCH = 64
K = 10

The comparison keeps these parameters identical.

That matters because otherwise a faster or more accurate result could simply come from giving one method a larger search budget.

The experiment is instead asking:

same vectors
same dataset
same HNSW configuration
same query set
different filtering strategy

7. Approach #1 — Pre-Filtering

How Pre-Filtering Works

Imagine an HNSW graph like this:

A ---- B ---- C ---- D
       |      |
       E ---- F

Now apply a filter that removes:

B, C, E

The effective graph becomes:

A          D

which is much more disconnected.

The vector search algorithm may no longer have the same paths it used in the original graph.

That is the central pre-filtering problem.

The benchmark deliberately models this with a fixed candidate budget:

labels_raw, _ = hn.knn_query(
    query,
    k=min(
        EF_SEARCH,
        len(df) - 1
    )
)

kept_pre = [
    label
    for label in labels_raw[0]
    if mask_values[label]
][:K]

There is:

NO expansion
NO retry
NO increasing ef

The search gets its fixed budget and whatever valid candidates happen to survive that budget.

This is important because using hnswlib’s native filter callback would not reproduce the disconnected-graph behavior being tested.

The repository explicitly documents this methodological distinction.

8. Approach #2 — Post-Filtering

Post-filtering is easier to implement.

Search first.

Filter later.

fetch = K * 5

while True:
    labels_raw, _ = hn.knn_query(
        query,
        k=fetch
    )

    kept_post = [
        label
        for label in labels_raw[0]
        if mask_values[label]
    ][:K]

    if len(kept_post) >= K:
        break

    fetch = min(
        fetch * 4,
        max_fetch
    )

This implementation starts at:

10 × 5 = 50 candidates

If there are not ten valid results, it grows:

50
↓
200
↓
800
↓
3200
↓
...

That is the fundamental trade-off.

Recall can be recovered by searching more.

But the amount of searching now depends on the selectivity of the filter.

9. Approach #3 — Single-Stage Filtering with Qdrant

Now we build the Qdrant collection.

client = QdrantClient(
    url="http://localhost:6333",
    timeout=60
)

client.create_collection(
    collection_name="mandi_listings",
    vectors_config=qm.VectorParams(
        size=emb.shape[1],
        distance=qm.Distance.COSINE,
        hnsw_config=qm.HnswConfigDiff(
            m=16,
            ef_construct=200
        )
    )
)

The next step is critical.

We create payload indexes for fields that participate in filtering.

client.create_payload_index(
    "mandi_listings",
    field_name="transit_hours",
    field_schema=qm.PayloadSchemaType.FLOAT
)

client.create_payload_index(
    "mandi_listings",
    field_name="moisture_pct",
    field_schema=qm.PayloadSchemaType.FLOAT
)

client.create_payload_index(
    "mandi_listings",
    field_name="cold_storage_available",
    field_schema=qm.PayloadSchemaType.BOOL
)

client.create_payload_index(
    "mandi_listings",
    field_name="tax_exempt",
    field_schema=qm.PayloadSchemaType.BOOL
)

client.create_payload_index(
    "mandi_listings",
    field_name="state",
    field_schema=qm.PayloadSchemaType.KEYWORD
)

client.create_payload_index(
    "mandi_listings",
    field_name="crop",
    field_schema=qm.PayloadSchemaType.KEYWORD
)

The types correspond to how the fields are filtered:

transit_hours → range
moisture_pct → range
cold_storage_available → bool
tax_exempt → bool
state → keyword
crop → keyword

Qdrant’s payload-indexing and filtering model is designed so that filtering can participate in vector search rather than being treated as only an application-level operation.

10. Loading the Vectors and Payload

The benchmark uploads the collection in batches:

payload_cols = [
    "crop",
    "mandi",
    "state",
    "grade",
    "transit_hours",
    "cold_storage_available",
    "moisture_pct",
    "tax_exempt",
    "price_per_quintal_inr",
    "quantity_quintal"
]

batch = 1000

for start in range(
    0,
    len(df),
    batch
):
    end = min(
        start + batch,
        len(df)
    )

    chunk = df.iloc[start:end]

    client.upsert(
        collection_name="mandi_listings",
        points=qm.Batch(
            ids=chunk["id"].tolist(),
            vectors=emb[start:end].tolist(),
            payloads=chunk[
                payload_cols
            ].to_dict(
                orient="records"
            ),
        ),
        wait=False,
    )

Before benchmarking, the code waits for the collection to become ready.

Now all three approaches have the same underlying vectors.

11. Running the Same Query Across All Three Approaches

A Qdrant query combines the vector and metadata constraints:

qfilter = qm.Filter(
    must=[
        qm.FieldCondition(
            key="moisture_pct",
            range=qm.Range(lte=20)
        ),
        qm.FieldCondition(
            key="cold_storage_available",
            match=qm.MatchValue(value=True)
        ),
        qm.FieldCondition(
            key="transit_hours",
            range=qm.Range(lte=8)
        ),
        qm.FieldCondition(
            key="state",
            match=qm.MatchValue(
                value="Maharashtra"
            )
        ),
        qm.FieldCondition(
            key="tax_exempt",
            match=qm.MatchValue(value=True)
        ),
    ]
)

Then:

response = client.query_points(
    collection_name="mandi_listings",
    query=query.tolist(),
    query_filter=qfilter,
    limit=10,
    search_params=qm.SearchParams(
        hnsw_ef=64
    )
)

That is the entire semantic-plus-constraint query.

No separate:

ANN search
↓
fetch hundreds/thousands
↓
Python filtering
↓
retry

is required at the application layer.

12. How We Measure Search Quality

A benchmark needs a reference answer.

For every query, we calculate the exact nearest neighbors among only the records satisfying the filter.

def brute_force_topk(
    query,
    emb,
    mask_array,
    k=10
):
    idx = np.nonzero(
        mask_array.values
    )[0]

    if len(idx) == 0:
        return np.array(
            [],
            dtype=int
        )

    sims = emb[idx] @ query

    order = np.argsort(-sims)[:k]

    return idx[order]

This gives us:

exact filtered nearest neighbors

Then recall@10 is:

def recall_at_k(
    retrieved_ids,
    ground_truth_ids
):
    if len(ground_truth_ids) == 0:
        return None

    gt = set(
        ground_truth_ids.tolist()
    )

    hit = len(
        set(retrieved_ids) & gt
    )

    return hit / min(
        K,
        len(gt)
    )

So the question becomes:

How many of the true top-10 filtered neighbors did the approximate method retrieve?

We also record:

p50 latency
p95 latency
filter selectivity
recall@10

13. Designing the Benchmark

The four scenarios are deliberately nested.

m1 = df.moisture_pct <= 20

m2 = (
    m1
    & df.cold_storage_available
)

m3 = (
    m2
    & (df.transit_hours <= 8)
)

m4 = (
    m3
    & (df.state == "Maharashtra")
    & df.tax_exempt
)

They produce approximately:

Scenario	Matching listings	Selectivity
Loose — `moisture_pct <= 20`	60,087	50.1%
Medium — + cold storage	4,840	4.0%
Strict — + transit <= 8h	773	0.64%
Very strict — + Maharashtra + tax exemption	86	0.07%

The last scenario is particularly useful because only 86 of 120,000 records satisfy the complete predicate.

14. One Benchmark Detail We Had to Fix

This was one of the most important lessons from the experiment.

The first implementation used:

hnswlib.knn_query(
    query,
    k=...,
    filter=...
)

That sounds like pre-filtering.

But it is not the same experiment.

The callback can still traverse through rejected nodes as routing points. Therefore, graph connectivity remains available.

That means it can hide the failure mode we actually want to test.

The corrected benchmark instead does:

labels_raw, _ = hn.knn_query(
    query,
    k=EF_SEARCH
)

kept_pre = [
    x
    for x in labels_raw[0]
    if mask_values[x]
][:K]

No retry.

No additional candidate budget.

This models the fixed-budget disconnected case.

The post-filter arm was also changed from a fixed overfetch to adaptive retries:

5x
↓
20x
↓
80x
↓
320x
↓
...

until either enough results are found or the collection is exhausted.

The repository explicitly marks the earlier results as pre-fix and asks for the benchmark to be rerun to regenerate the corrected chart.

15. What the Benchmark Shows

The experiment is designed to demonstrate three different failure modes.

Pre-Filtering

As selectivity decreases, a fixed candidate budget contains fewer valid points.

At 50% selectivity:

64 candidates
×
50% valid
≈
32 potentially valid candidates

At 0.07% selectivity:

64 candidates
×
0.0007
≈
0.045 expected matches

That makes it intuitive why recall can collapse when the traversal is restricted to a disconnected candidate space.

Post-Filtering

Post-filtering has the opposite behavior.

The graph remains intact.

But the application has to search increasingly far to find enough valid points.

The benchmark therefore increases:

candidate count
↓
more HNSW work
↓
more retries
↓
higher latency

The repository’s benchmark code measures the sum of latency across those retries, rather than pretending that every retry is free.

Qdrant Single-Stage Filtering

The Qdrant arm keeps the same:

K = 10
M = 16
ef = 64

across scenarios.

The query itself remains structurally identical:

response = qc.query_points(
    collection_name=COLLECTION,
    query=query.tolist(),
    query_filter=qfilter,
    limit=K,
    search_params=qm.SearchParams(
        hnsw_ef=EF_SEARCH
    )
)

That is the property being evaluated:

filter selectivity changes
        ↓
search does not require
application-level overfetch tuning

16. Why Single-Stage Filtering Matters for Mandi Logistics

The actual business query is not:

“find similar tomatoes”

It is closer to:

query = semantic_vector(
    "firm fresh tomatoes for hotel supply"
)

constraints = {
    "transit_hours": 8,
    "cold_storage_available": True,
    "state": "Maharashtra",
    "tax_exempt": True
}

The search system has to satisfy both.

The architectural model is:

Buyer Query
     |
     +-------------------+
     |                   |
 Text description   Constraints
     |                   |
 Embedding             Payload
     |                   |
     +---------+---------+
               |
             Qdrant
               |
        Filter-aware HNSW
               |
           Top-K results

The result is not merely:

“semantically similar”

It is:

“semantically similar AND operationally eligible.”

That distinction is the entire motivation for the experiment.

17. From Experiment to Production Architecture

A production system can preserve the same separation:

Agmarknet / eNAM / seller feeds
              ↓
       Data Normalization
              ↓
       Listing Description
              ↓
        Embedding Model
              ↓
       +----------------+
       |    Qdrant      |
       |                |
       | Vector + HNSW  |
       | Payload Index  |
       +----------------+
              ↓
       Buyer Search API
              ↓
         Filtered Top-K
              ↓
         Logistics UI

The benchmark intentionally isolates the filtering layer so that the embedding model can later be replaced.

18. What About Real Agmarknet / eNAM Data?

The repository already separates this step.

A real CSV can be mapped into the same schema:

{
    "crop": ...,
    "mandi": ...,
    "state": ...,
    "grade": ...,
    "description": ...,
    "transit_hours": ...,
    "cold_storage_available": ...,
    "moisture_pct": ...,
    "tax_exempt": ...
}

The loader maps fields such as:

Commodity   → crop
Market      → mandi
State       → state
Variety     → grade
Modal_Price → price_per_quintal_inr

Agmarknet does not directly provide every logistics field needed for this benchmark, so the current loader derives those missing fields using the same distributional logic as the synthetic generator.

For a production benchmark, those fields should instead come from actual logistics and compliance data.

The important architectural point is that changing the data source does not require changing the filtering benchmark.

19. Reproduce the Experiment

The repository is structured so the experiment can be reproduced as a sequence.

Install dependencies

pip install \
qdrant-client \
hnswlib \
pandas \
numpy \
scikit-learn \
matplotlib \
pyarrow

Start Qdrant locally

wget https://github.com/qdrant/qdrant/releases/download/v1.19.1/qdrant-x86_64-unknown-linux-gnu.tar.gz

tar -xzf qdrant-x86_64-unknown-linux-gnu.tar.gz

chmod +x qdrant

QDRANT__STORAGE__STORAGE_PATH=./qdrant_storage \
./qdrant \
--disable-telemetry &

Generate the dataset

python3 scripts/01_generate_data.py

Build embeddings

python3 scripts/02_build_embeddings.py

Run the corrected benchmark

python3 scripts/03_benchmark.py

Generate the benchmark visualization

python3 scripts/04_plot_benchmark.py

Generate the filtering diagram

python3 scripts/05_diagram_filtering.py

Generate the architecture diagram

python3 scripts/06_diagram_architecture.py

There is also a Colab-ready notebook containing the end-to-end pipeline.

The repository notes that the notebook can launch a Qdrant server locally inside the runtime rather than requiring an external Qdrant deployment.

20. Design Decisions Worth Examining

The experiment intentionally keeps several things simple.

Why TF-IDF + SVD?

Because the experiment is about filtering behavior, not embedding-model quality.

TF-IDF
  ↓
SVD
  ↓
128 dimensions
  ↓
cosine similarity

A production implementation can replace this with a sentence encoder without changing the benchmark structure.

Why HNSW?

Because HNSW is the approximate nearest-neighbor structure being evaluated.

M = 16
ef_construction = 200
ef_search = 64

Why Recall@10?

Because the system is modeled as a top-K retrieval system.

K = 10

Why four selectivity levels?

Because the interesting behavior appears when the filter becomes rare:

50.1%
↓
4.0%
↓
0.64%
↓
0.07%

The experiment is therefore less about one latency number and more about how each strategy behaves as the valid search space shrinks.

21. What This Experiment Actually Tells Us

The important result is not simply:

“Qdrant is faster.”

That would be an incomplete interpretation.

The experiment is really testing whether semantic search and strict metadata constraints can remain part of the same retrieval operation as selectivity becomes extreme.

The three approaches expose three different failure modes.

Pre-Filter

filter first
↓
graph connectivity becomes difficult
↓
recall can collapse

Post-Filter

search first
↓
too few valid candidates
↓
over-fetch + retry
↓
latency grows

Single-Stage

vector + payload filter
↓
filter-aware indexed search
↓
no application-level overfetch loop

There is also an important benchmarking lesson here.

If the implementation of the baseline does not actually model the behavior you claim to be comparing against, the benchmark can produce a very convincing but incorrect story.

That is exactly why the hnswlib filter= implementation had to be replaced in this experiment.

A good benchmark should make the failure mode visible, not hide it behind a convenient API.

The central trade-off between recall, latency, and filter-aware retrieval as the number of valid candidates becomes increasingly selective.

22. What’s Next?

There are several obvious extensions.

a. Replace TF-IDF + SVD

Use a multilingual sentence encoder trained or adapted for agricultural listings:

emb = model.encode(
    df["description"].tolist(),
    normalize_embeddings=True
)

b. Replace synthetic logistics attributes

Use real operational data.

c. Test larger collections

120K
↓
1M
↓
10M+

d. Introduce more complex filters

qm.Filter(
    must=[...],
    should=[...],
    must_not=[...]
)

e. Build an actual buyer-facing retrieval system

natural-language buyer query
              ↓
        query embedding
              ↓
         Qdrant search
              ↓
      +----------------+
      |                |
  semantic        constraints
      |                |
      +-------+--------+
              ↓
       ranked mandi
          listings

That is where vector search becomes useful infrastructure rather than just a similarity demo.

Conclusion

A vector can tell us that two listings look similar.

A metadata filter can tell us whether a listing satisfies a constraint.

A real retrieval system needs both.

The interesting engineering problem is what happens when those constraints become highly selective.

This experiment builds the same 120,000-listing dataset into three retrieval strategies and measures the resulting recall and latency under increasingly strict filters.

The code makes the trade-off explicit:

Pre-filtering → risks recall

Post-filtering → risks latency

Single-stage in-index filtering → combines vector retrieval and payload constraints

The repository is intentionally structured so the experiment can be rerun, corrected, and extended rather than relying on a single static benchmark screenshot.

And that is probably the most useful lesson from the whole exercise:

When filtering is part of the meaning of a search result, filtering should be treated as part of the retrieval problem — not as cleanup performed after retrieval.

References

Repository

The complete implementation is available here:

https://github.com/ks-1220/mandi-single-stage-filtering/

mandi-single-stage-filtering/
│
├── article/
│   └── article.md
│
├── data/
│   └── mandi_listings_sample.csv
│
├── figures/
│   ├── benchmark_latency_recall.png
│   ├── filtering_strategies_diagram.png
│   └── system_architecture.png
│
├── notebook/
│   └── mandi_qdrant_single_stage_filtering.ipynb
│
├── scripts/
│   ├── 01_generate_data.py
│   ├── 02_build_embeddings.py
│   ├── 03_benchmark.py
│   ├── 04_plot_benchmark.py
│   ├── 05_diagram_filtering.py
│   └── 06_diagram_architecture.py
│
└── README.md

The benchmark, notebook, data generator, embedding pipeline, and figure-generation scripts are all included in the repository.

DEV Community: Kritika Singh

Single-Stage Vector Filtering for AgriTech Mandi Logistics

Single-Stage Vector Filtering for AgriTech Mandi Logistics

The Problem: Vector Search Alone Isn’t Enough

1. When Similarity Isn’t Enough

2. Meet the Mandi Search Problem

Semantic Similarity vs. Hard Constraints

3. What We’re Actually Going to Benchmark

a. Pre-Filtering

b. Post-Filtering

c. Single-Stage Filtering

Why Qdrant?

4. Building the 120,000-Listing Mandi Dataset

5. Turning Mandi Listings into Vectors

6. Setting Up the HNSW Search Layer

7. Approach #1 — Pre-Filtering

How Pre-Filtering Works

8. Approach #2 — Post-Filtering

9. Approach #3 — Single-Stage Filtering with Qdrant

10. Loading the Vectors and Payload

11. Running the Same Query Across All Three Approaches

12. How We Measure Search Quality

13. Designing the Benchmark

14. One Benchmark Detail We Had to Fix

15. What the Benchmark Shows

Pre-Filtering

Post-Filtering

Qdrant Single-Stage Filtering

16. Why Single-Stage Filtering Matters for Mandi Logistics

17. From Experiment to Production Architecture

18. What About Real Agmarknet / eNAM Data?

19. Reproduce the Experiment

Install dependencies

Start Qdrant locally

Generate the dataset

Build embeddings

Run the corrected benchmark

Generate the benchmark visualization

Generate the filtering diagram

Generate the architecture diagram

20. Design Decisions Worth Examining

Why TF-IDF + SVD?

Why HNSW?

Why Recall@10?

Why four selectivity levels?

21. What This Experiment Actually Tells Us

Pre-Filter

Post-Filter

Single-Stage

22. What’s Next?

a. Replace TF-IDF + SVD

b. Replace synthetic logistics attributes

c. Test larger collections

d. Introduce more complex filters

e. Build an actual buyer-facing retrieval system

Conclusion

References

Repository