Asked in an AI engineer interview.
What they are really testing:
Whether you can reason about partitioning under a latency SLO instead of just naming a shard key. Strong candidates address fan-out, skew, and memory reduction before they add machines.
A strong answer:
- Start with the math: 750M vectors at 768 dims in float32 is about 2.3 TB of raw data. Compress first with product or scalar quantization (PQ at 64 bytes per vector is roughly 48 GB), and keep full-precision vectors on disk for reranking.
- Shard by a hash of candidate ID, not by tenant or recruiter. Hashing spreads data evenly, while tenant sharding creates hot shards when a few large customers dominate size and query load.
- Accept scatter-gather but keep it cheap: each shard returns its local top-k (for example 100 candidates for reranking) and a coordinator merges them. Latency is bounded by the slowest shard, so monitor p99 per shard, not averages.
- Prune shards where it is safe. Store location, seniority, and tenant as filterable attributes, or partition coarse IVF clusters by region. Be explicit about pre-filtering versus post-filtering, since aggressive post-filtering can return too few results.
- Size throughput with replicas, not extra shards. Each shard group serves reads from several replicas, so 10k QPS is divided across them. Use consistent hashing so resharding moves only a small fraction of vectors.
- Treat tenant isolation as a separate concern. Enforce access control in the query filter or with per-tenant namespaces, independent of the physical shard key.
Common mistakes:
- Sharding by tenant or recruiter without accounting for skew, which leaves one node holding the largest account.
- Treating scatter-gather as free and not budgeting merge time and tail latency across shards.
- Trying to keep full float32 vectors in RAM instead of quantizing first, then adding machines to compensate.
How would you handle filters on fields that change daily, like candidate availability, without rebuilding the index? Curious how others would approach it.
Originally shared on LinkedIn.
An AI interview question every day, leadership for tech people and hands-on AI learning:
Top comments (0)