On building natural language search over a schema that changes without a deploy
The advanced filter screen is one of the oldest admissions of defeat in enterprise software. Pick a field from a dropdown. Pick an operator. Type a value. Add another condition if you need one. It works. It has always worked. And every person who has ever used one to find a specific set of cases in a fraud investigation has, at some point, wished they could just type what they were looking for and have the system understand it.
This system does that. A user types cases assigned to R. Iyer with the notice deadline past June thirtieth and gets back a real, filtered, paginated list of cases, the same list the dropdown screen would have produced, without touching a single dropdown. This is the story of how that works, why building it correctly is harder than it looks, and what I got wrong before I got it right.
The platform is a fraud case management tool used by banks and NBFCs for investigation and RBI regulatory reporting. Every case moves through nine or ten forms. Every form is not fixed. It is configured per client, per product type, through a forms engine that lets an administrator add, remove, or rename fields without a deployment. One client's investigation form might have three hundred fields. Another's might have four hundred, arranged differently, named differently, grouped into different forms entirely. A client opens the form builder on a Tuesday afternoon, adds a field, and from that moment the system has a shape it did not have an hour ago, with no migration, no release, and nobody on the engineering side aware it happened.
This is what makes the problem different from the one every natural language search tutorial is actually solving. Those are built over fixed schemas. Twenty columns, known at build time, stable forever. Natural language search over a fixed schema is a solved problem with good prior art and reasonable tooling. Natural language search over a schema that is a runtime value, different per tenant, reconfigurable at any hour by someone with no engineering background, is not the same problem in a different costume. It is a different problem, and the techniques that solve the first one break the second one in ways that are genuinely hard to diagnose.
The Reflex That Breaks It
The obvious implementation is obvious for a reason. Take the user's question. Hand it to a language model. Ask the model to produce a database query. Execute the query. Return the results.
This fails in a specific and dangerous way. The published failure modes of text-to-SQL are consistent across every generation of model: faulty joins, invented column names that sound plausible, aggregation logic that is subtly wrong but does not raise an error. In most systems, a wrong result is a minor inconvenience. In a fraud investigation platform used by regulated financial institutions, a query that runs, returns a result, and quietly filters on the wrong field is a compliance incident waiting to be discovered by someone downstream who trusted the output. The failure does not announce itself. It looks like an answer.
There is a second failure specific to this system. For the model to write a correct query, it would need to know which of three hundred to four hundred fields exist for this tenant, what they are named, how their values are stored, and how they relate to the case in front of the user. That is not context. That is the entire schema, restated per tenant, on every query, which is both enormous and still insufficient, because field names alone do not tell you what a field means.
But the deeper problem is not scale or context length. It is that handing a model a question and asking it to produce executable code collapses two problems that need to stay separate.
The first problem is interpretation: what does the investigator mean. The second is execution: what is true and safe to run against a live, regulated dataset. A language model is a reasonable instrument for the first. It has no business anywhere near the second. Collapse the two and every quiet misinterpretation becomes, silently, an incorrect result set, with no seam anywhere in between where the mistake could have been caught.
So the model in this system never produces a query. It never sees a table. It fills in a small, closed, typed object I define completely, an intent rather than a query, and a separate deterministic layer decides afterward whether that intent refers to anything real and whether it is safe to act on. Interpretation stays probabilistic. Execution stays deterministic. They never touch directly, and that separation is the spine everything else hangs off.
Three Problems, Not One
Once you commit to structured intent rather than raw query generation, a second realisation follows quickly: what looks like one problem from outside is actually three separate problems that need three different techniques. Collapse any two of them and the system breaks, usually silently, and usually in production rather than in testing.
The three problems are finding the right field, finding the right value, and preventing the model from acting on something it invented. Each one has a different shape, and the shape determines the tool. Applying the same technique to all three because it works for one of them is how systems end up confidently wrong about the cases that matter most.
Finding the Right Field Is a Meaning Problem
Say the field count out loud. Three to four hundred, per tenant, reconfigurable at runtime. Handing that entire catalog to a model on every question is not a viable starting point. It is expensive for no reason, since the overwhelming majority of those fields are irrelevant to any single question. More importantly, it is inaccurate. The more irrelevant options sit in front of a model, the more its attention is diluted across noise, and the higher the odds it settles on something plausible sounding rather than correct. A catalog that large is not context. It is noise with a small signal buried inside it.
The fix is retrieval before generation. Every active field across every form gets embedded once, using its name, its admin authored tooltip, and the name of the form and page it lives on. That last part matters more than it might seem. Three hundred fields spread across nine forms produce real naming collisions. Two fields both called Date are indistinguishable by name alone. Date from the Notice Configuration Form and Date from the KYC Verification Form are not the same field, and embedding the surrounding context alongside the name is what makes them retrievable separately.
Every stored field embedding sits in a vector column in PostgreSQL, one table per tenant schema, indexed with an approximate nearest neighbour index so similarity search stays fast as the field count grows rather than scanning the entire table every time. When a question arrives, it gets embedded the same way, and a single similarity search returns a small, relevant shortlist, fifteen or so candidates instead of four hundred. That shortlist, and only that shortlist, is what the model ever sees.
The catalog is not treated uniformly, and that distinction matters more than the retrieval mechanism itself. Alongside the three to four hundred dynamic fields, the system also carries a small, fixed set of structural attributes, case stage, current assignment, active status, a dozen or so, identical across every tenant. Those get shown to the model in full, every time, no retrieval involved. Semantic search earns its cost against a genuinely large, variable catalog. It buys nothing against twelve fixed items. Reaching for it there anyway would be applying a technique because it was already built, not because the problem in front of it needed it. Matching the tool to the actual shape of each sub-problem, rather than to whichever tool happens to be nearby, was one of the more consequential small decisions in the whole system.
Finding the Right Value Is a Spelling Problem
Retrieval solves which field a question is about. It does not solve who R. Iyer is, or what the audit group refers to, and those are a genuinely different kind of lookup that deserves its own mechanism rather than an extension of the first one.
Cases assigned to R. Iyer and cases with the audit group both name values, not fields. Those values are real rows sitting in real tables: a user directory with hundreds of names, a groups table with a few dozen, and a handful of smaller reference tables for case stage, sub status, product type, incident category, and branch hierarchy. Finding the right field was a meaning problem. Finding the right person named R. Iyer, or R. Iyer with a middle initial, or misspelled by one letter, is a spelling problem. Spelling problems do not call for embeddings. They call for character overlap, and that is a far older and cheaper technique.
This half of the system runs on trigram similarity, comparing overlapping three character sequences between strings. It is not sophisticated. It is correct for the shape of the problem. A short extraction pass reads the question first and identifies anything that will need a value lookup, a person's name, a group, a stage, a category. The resolver then searches every relevant table in parallel, users, groups, stage, sub status, product type, incident category, hierarchy, because every one of those tables shares an identical shape and one parameterised method covers all seven instead of a bespoke implementation for each.
The interesting failure appeared after this resolver was widened from two tables to seven, and it broke two questions that had worked cleanly until that moment. The bug taught something I had not fully understood about retrieval systems before building this one.
With only users and groups in scope, the top scoring match was almost always the only serious candidate, and a bare similarity floor was sufficient. Adding five more tables changed the statistics of the problem without anyone touching the threshold. A short name began weakly matching branch names it shared almost no real relationship with, purely on incidental letter overlap, at a score that still cleared the same floor that had been fine a moment earlier. The system was checking whether a candidate was good enough in isolation. It was never checking whether that candidate was good enough relative to everything else that also happened to match.
This is the distinction that changed how I think about retrieval. A similarity score tells you a match exists. It does not tell you whether that match is the only one that matters. Those are different questions, and a system that only answers the first one will eventually be confidently wrong in ways that pass every individual test.
The fix reframes what confidence means. A candidate only counts as trustworthy if it clears an absolute floor and stands meaningfully apart, within a defined margin, from the next best option. Two candidates within that margin are not one confident answer and one noise. They are a genuine tie, and the right response to a genuine tie is to say so rather than silently pick one.
There is a real limit here that no amount of threshold tuning removes. Strings three or four characters long simply do not carry enough information for character overlap to be reliable. Loosening the floor enough to correctly resolve a genuinely short name also lets short meaningless coincidences back in. That is not a bug waiting for a better number. It is where the technique runs out. The honest fix is architectural: let a person disambiguate a short or ambiguous reference explicitly, through the interface, rather than asking string similarity to divine intent from three letters. I have not built that yet. Naming the boundary precisely is worth more than pretending a cleverer threshold exists somewhere unexplored.
Preventing the Model From Acting on Something It Invented Is a Trust Problem
Retrieval decides what the model gets to see. It does not decide what happens when the model is wrong anyway, and on that question the most useful public account I found of anyone building this same shape of system is Netflix's writeup on rebuilding their internal Graph Search around natural language. Their system converts questions into a custom filter language rather than raw queries, validates generated output against real schema metadata, and ran into exactly the problem this system also ran into. Even with carefully assembled context, they found the model "sometimes hallucinates both fields and available values in the generated filter statement."
That sentence is not describing a flaw specific to their implementation. It is describing something structural about how these models work. No prompt instruction makes hallucination impossible. Instructions reduce frequency. They do not provide a guarantee, and a system that only has a reduction in frequency is asking you to trust a probability.
The answer is two genuinely different guarantees, stacked. They are not the same guarantee reinforced twice. They fail in different registers, and both are necessary.
The first is what the model is shown. Only the fields retrieval actually surfaced, only the values resolution actually found, and an explicit instruction to choose only from what is in front of it. This reduces how often the model invents something. It does not reduce that number to zero.
The second happens entirely outside the model, after generation, before anything is trusted. Every field the model names gets checked against the exact candidate list it was handed. Every value gets checked against the exact set of real rows the resolver already found. Anything that fails either check is discarded, logged as a likely fabrication, and never reaches a query, regardless of how plausible it reads. Netflix notes from their side that this validation costs almost nothing extra, since the data being checked against was already loaded to build the context in the first place. The same is true here. Validation is cheap precisely because the ground truth was never fetched twice.
The first layer is probabilistic and makes bad output rare. The second is deterministic and makes bad output, on the rare occasion it still occurs, structurally incapable of being acted on. A system is only as honest as the weaker of the two guarantees it actually has, not the stronger one it advertises.
One category of value earned a third, narrower form of grounding. Whether a case is open or closed is a boolean underneath, but nobody types false. They type pending, resolved, still active. Rather than trust the model to infer that mapping correctly on every query, which is exactly the kind of quiet inference that goes right ninety times and wrong on the hundredth in a way nobody notices until an audit, the accepted vocabulary and its synonyms are stated directly in what the model is shown. Closed maps to false because the system says so, not because the model guessed correctly and happened to keep guessing correctly.
What This Actually Claims
A person can type a plain question into a system carrying four hundred fields they have never seen named, against a schema an administrator reshaped last week without telling anyone, and get back a real, filtered, paginated list. The model that produced it never wrote a query, never touched a table, and was never taken at its word for anything that mattered. Every field it referenced was one it was shown and none other. Every person and group it named was a row already confirmed to exist before generation happened at all. Everything it produced was checked again, by code that owes the model nothing, before being allowed to run. The validated conditions never spawn a new query path. They become additional clauses on the same query builder every other screen in the product already runs through, because correctness belongs at whichever layer has already earned it.
That is a narrower claim than most natural language search demonstrations are implicitly making. Most of them are showing you that a model can sound like it understood a question. That is not hard anymore. Building a system that gives a model the freedom to interpret language while never once extending it the authority to be wrong is a different exercise, and it is the one that was actually worth doing here.
The dropdown filter is still there. It does not need to disappear. What changed is that an investigator no longer needs it to answer most of what they actually ask.
Caveats
This describes one system built against one set of constraints. A per tenant schema in the low hundreds of fields, lookup tables with hundreds rather than thousands of entries, and a domain where an incorrect result is a compliance exposure rather than an inconvenience. At a meaningfully larger field count, the retrieval step would need a reranking pass on top of raw similarity, which this version does not have. Past the low hundreds of users or groups, trigram matching alone begins to strain, and the disambiguation interface currently absent from this system stops being a future improvement and becomes a requirement. Every threshold value here was set against this system's real data by testing, and would be revised the moment testing said otherwise. None of them are constants. They are the record of specific tradeoffs made deliberately, which is the only kind of number worth publishing.
Netflix's writeup on rebuilding Graph Search around natural language is the closest public account I found of anyone solving this same shape of problem. Their system runs at a scale this one may never reach, against a custom filter language with an AST parser rather than a schema-native structured format, and they arrived at the same validation discipline independently. Convergent answers to the same problem, found separately, are usually a sign the answer is closer to correct than either team could tell alone. Their writeup is worth reading directly.
Nothing in the individual pieces here is novel. Embeddings, trigram matching, structured generation over raw query generation, none of it is new. What is less common is holding all of it together against a schema that refuses to sit still, changed by someone with no engineering background, at any hour, with no warning, and building a system that remains correct anyway, not because it trusts the model to get things right, but because it was designed from the start to survive the model getting things wrong.
The model interprets. The system decides what that interpretation is allowed to mean. That boundary, held precisely, is the only thing that makes any of this trustworthy.
Top comments (0)