🎯 Who This Is For
- You want to store and search vector data in DynamoDB
- You're frustrated that a single query returns at most 100 results, which leaves no room for aggregation or reranking
🆕 DynamoDB Now Supports Vector Data
In August 2026, DynamoDB added support for vector data. This post doesn't cover the basics of the feature itself — see the for details.
aws.amazon.com
🚧 The TopK=100 Problem
A popular RAG pattern today is to retrieve broadly, then rerank and aggregate on the application side before showing sources. In June 2026, Amazon S3 Vectors raised its TopK limit from 100 to 10,000, which made this "fetch wide, filter in the app" approach much more flexible.
DynamoDB vector search, however, still caps TopK (the number of results per query) at 100 😓. If the chunks of a single document occupy most of those 100 slots, you can't show multiple documents as sources.
As this article also points out, when you need results that overlap (e.g., multiple chunks from the same document), the effective number of useful results shrinks even further.
📚 A Quick Recap of DynamoDB's Vector Index Structure
When you create a table, you define the vector configuration: which attribute holds the vectors, and the partition key for the vector index. Specifying a partition key separates where data is stored, so searches run within a single partition.
At query time, you specify a partition key value to search only that partition.
🧩 Can Sharding Get Us Past TopK=100?
If one query only returns 100 results, why not just make several queries? That's the idea: split the partition key into multiple values, query each one, and merge the results.
Throughput limits for DynamoDB vector indexes are also defined per partition key value, so I used a hash to distribute the data evenly across shards.
Choosing the Partition Key
The partition key is derived from a hash of the item ID, which assigns each item evenly to one of 5 shard groups. Because the hash is deterministic, re-ingesting the data always puts each item back into the same shard — handy for rebuilds.
common.py
def shard_index_of(chunk_id: str) -> int:
digest = hashlib.sha256(chunk_id.encode("utf-8")).hexdigest()
return int(digest, 16) % N_SHARDS
Querying the Shards
Queries run in parallel — one per shard key, all fired at once.
scatter_gather.py
def one(shard_key: str) -> dict:
return search_shard(
client,
vector,
table_name=table_name,
index_name=index_name,
shard_key=shard_key,
top_k=top_k,
search_vector=search_vector,
**kwargs,
)
t0 = time.perf_counter()
if parallel:
workers = max_workers or len(keys)
with ThreadPoolExecutor(max_workers=workers) as pool:
per_shard = list(pool.map(one, keys))
else:
per_shard = [one(key) for key in keys]
Merging the Results
Once all shards respond, the results are combined and sorted by score. At this point, you effectively have the equivalent of a TopK=500 query.
scatter_gather.py
def sort_key(record: dict) -> tuple[float, str]:
"""Sort order for results. Ties on distance are broken by chunk_id."""
return (float(record["distance"]), record["chunk_id"])
def merge(*result_lists: Iterable[dict]) -> list[dict]:
"""Merge per-shard results into one list (dedupe -> sort by (distance, chunk_id))."""
flat: list[dict] = []
for one in result_lists:
flat.extend(one)
return sorted(dedupe(flat), key=sort_key)
🏇 Let's Test It!
For the test, I ran a RAG search UI locally that calls the backend with an API key and receives an answer along with its sources. I sent the same question through both a single-shard search and the sharded search, and checked whether the number of sources increased. Both searches use the same DynamoDB data — the table simply has two vector indexes, one with a shard key and one without.
Single-shard search, TopK=100 (the usual approach)
Multi-shard search, TopK=500 (the new approach)
It worked 👐 — 500 candidate chunks were retrieved, and the number of source documents went up to 10.
⏱️ Parallel Query Speed Depends on the Caller's CPU
The documentation states that vector search rate limits are defined per partition key value, and that throughput scales linearly across partition key values. In other words, sending queries to 5 shards at once shouldn't make them compete for capacity.
So 5 parallel queries should finish in roughly the time of one — or so I thought. When I measured from a Lambda function in the same region, my prediction was completely wrong.
| Caller | Single (ms) | 5 parallel (ms) | Parallel ÷ Single |
|---|---|---|---|
| Lambda 1,024 MB | 23.5 | 119.9 | 5.1x |
| Lambda 1,769 MB | 17.6 | 79.1 | 4.5x |
| Lambda 3,008 MB | 15.7 | 40.6 | 2.6x |
At 1,024 MB, 5 parallel queries took 5 times as long as a single query — parallelism bought nothing.
Firing 5 queries at once means the client-side work for all 5 has to run at the same time too. Without enough CPU, the threads end up waiting their turn, and it's no better than running them sequentially. Parallel search demands a beefier caller than a single search does.
⚠️ Things to Watch Out For
Changing the shard count means re-ingesting everything
Items are assigned by hash(chunk_id) % N, so changing the number of shards requires re-registering all data. You can keep running after changing only the shard constant, but the partitions become unbalanced and retrieval becomes uneven — not recommended.
Search cost scales with the number of shards
Each search now makes one query per shard, so billed bytes in DynamoDB go up. In my measurements, a single search was 112 KB, while the 5-shard total was 367 KB — 3.27x. That said, at 1 million searches per month, cost only goes from $0.24 to $0.78 (Tokyo region, pricing as of September 2026). Write cost and storage are essentially unchanged, since you're only adding one attribute.
It's not an exact match for a true TopK=500
Depending on how items are distributed, results near each shard's top-100 cutoff can differ from what a true global top 500 would return.
In my view, this is acceptable. The gap only affects the lowest-ranked part of the results — the tail end of the 500 — so the impact on retrieval quality (how good the LLM's answers feel) is minimal. If it does matter for your use case, adding more shards fixes it.
Choosing the shard count
Decide based on how many results you want. Want 300? Use 3 shards. Want 500? Use 5. If your sources are still skewed toward a few documents, add shards to fetch more.
🍜 Wrap-up
The overall architecture:
This time, I took the partition key — a tool that's normally meant to narrow the search scope — and repurposed it for sharding to widen how many results I can retrieve. A bit of a hack, maybe, but it gets past the TopK=100 limit.
It brings the classic RAG pattern of "fetch wide, filter in the app" to DynamoDB vector search. That could be a good fit if you're reranking with OpenSearch and want to cut costs, or if you'd rather not manage your data in both DynamoDB and S3 Vectors.
Honestly, though, I'm hoping DynamoDB eventually gets an update like S3 Vectors did — bulk retrieval or pagination would make all of this unnecessary 😊
Top comments (0)