95% mutation score, and a review still found a cross-tenant leak: Fuzzphony 0.11

Last time I wrote about building search next to a database I couldn't touch. Fuzzphony was at v0.4 then. It is v0.11.1 now, and the post ended with a section called "What I haven't solved yet". So this one starts there.

The short version: a typo-tolerant, ranked search for a PostgreSQL database you may not be allowed to change (a sidecar table, pg_trgm, full-text search, no new infrastructure). The longer version is below, including the part I find more interesting than the features: a 95% mutation score did not stop three real bugs, and an independent review found them the week before the release.

What I said I hadn't solved

Short words matched the wrong thing. mouse matched monitor, because a flat trigram threshold is too generous for a five-letter word and too strict for a fifteen-letter one. Since 0.7 the tolerance grows with the length of the word, so mouse stays mouse, and hedphones still finds headphones.

Very common words lost their best match. The candidate window (candidate_limit, 2000 by default) is cut before ranking, so for a word that appears in half the catalogue the best hit could fall off the edge. 0.9.8 changed how candidates are chosen so the best match of a very common word is no longer lost above the limit. The same release added tolerance for swapped letters (recieve) and a zero-result rescue: when a search finds nothing, it retries the closest spelling and tells you it did (correctedTo).

Operations. Reindexing without a gap (a shadow table that is swapped in), Prometheus metrics with sample Grafana dashboards, synonyms from Solr/Elasticsearch files that can be swapped at runtime, and "did you mean" from a vocabulary table all landed between 0.5 and 0.7. The rest of the rough edges I listed then are in the changelog.

0.10: secure by default

The part of search libraries I trust least is the part that decides who may see what, so 0.10 was the breaking release that made the safe thing the default.

Without a token the caller is anonymous. A system caller has to say so:

// a console command, a cron job: explicitly the system
$hits = $fuzzphony->in('customers')->asSystem()->query('kovacs')->get();

// a web request: the ambient actor is whoever is logged in
$hits = $fuzzphony->in('customers')->forTenant($tenantId)->query('kovacs')->get();

There is one tenant rule: an actor of tenant A cannot ask for tenant B, unless the call is asSystem(). Fields can be granted to a role, and a search by a viewer who does not hold the grant must not change because of a field they cannot see. I call that hidden-field independence: ids, ranks, totals, facets and highlights are the same whatever the hidden fields contain. There are also two database roles (a worker and an admin) with --dump-grants so the search process never needs DDL rights, and code: true fields for invoice numbers and SKUs (INV-001-style values are matched exactly, not "fuzzily"). The public API went from 102 types to 77, so the surface you can rely on is smaller and written down.

0.11: several sources, several values, one row to render

Real systems have a customer table, a company table, and orders. 0.11 lets one index read all of them (a simplified version of the demo's site index):

fuzzphony:
  indexes:
    site:
      sources:
        person:
          query: "SELECT id, name AS title, email AS body, account_id AS account FROM erp_customer WHERE kind = 'person'"
          watch: { erp_customer: "SELECT :id" }
        order:
          query: "SELECT id, number AS title, note AS body, account_id AS account FROM erp_order"
          watch: { erp_order: "SELECT :id" }
      fields: { title: { weight: A, fuzzy: true }, body: C }
      filters: { account: int }
      tenant: account

A hit says which source it comes from ($hit->kind) and carries that source's own typed id. Each source has its own watch, so changing a row refreshes only the right document, and the doctor now warns when the source query reads a table no watch covers.

A customer has several addresses. Instead of a folded string_agg(...) column, a field can say where the values live:

city:
  weight: C
  fuzzy: true
  many: { from: address, key: customer_id, column: city, limit: 50 }

and the explanation tells you which address matched ("found in the address: Budapest"). The next release is going to rank by the best single value instead of the whole field; that is a design I would rather finish properly than rush.

And the thing every search page needs: a result row. with() returns columns of the hit's source row, so a page can be rendered and linked without loading entities:

$result = $fuzzphony->in('products')->query('mouse')->limit(10)->with('title', 'slug', 'price')->get();

foreach ($result->hits as $hit) {
    echo $hit->data['title'], ' ', $hit->data['price'];
}

It is opt-in, reads the current value (nothing extra is copied into the index), and a column behind a granted field is returned only to someone who holds the grant. Remember that last sentence; it comes back below.

The demo grew an ERP data set (about half a million rows: customers, companies, addresses, orders, order lines), an ERP page with separate and combined search boxes, a live demonstration of the sync (move an address to another city, watch which indexes refreshed), and a search box in the header that searches every index of the demo at once. With about seven indexes the engine answers in 40 to 170 ms.

How I know it works

The test suite has about 3,700 tests, PHPStan at max level, and coverage close to 100%. Coverage tells you which lines ran. Mutation testing tells you whether the tests would notice if a line were wrong: the tool changes the code a little (> becomes >=, && becomes ||, a line disappears) and runs the tests; a mutant that survives is a place where nothing checks the behaviour.

The numbers, from CI:

Run	Mutants	Covered-code score
First full run (September)	3,944	85%
After 0.8.1	5,989	96%
0.11.0 branch	about 11,900	95%

The last full run takes 1 hour 45 minutes on CI, so it always lags one step behind the code; I run it once per release and work with the result. Of the 486 survivors in 0.11 I killed the ones in the new code with focused tests (assert what the code must do, never a mirror of the code), and I sorted the rest into a few classes that cannot be killed because the mutant is equivalent:

• a limit that the validator already guarantees (a replace "limited to 1" versus "limited to 2" when exactly one placeholder can exist),
• a cast on an array key that PHP normalises anyway,
• array_values() on a list that is only iterated,
• a branch the code itself marks as unreachable.

It sounds boring, and it is, until you read the next section.

What a mutation score cannot see

Before tagging 0.11.0 I asked for an independent, adversarial review of everything new in src/ (about 2,700 added lines), with one instruction: try to break the two promises that matter, hidden-field independence and no tenant crossing. It found three real bugs, behind a 95% score and a green suite.

1. A hit of the wrong kind became someone else's row. In a multi-source index the hits are emp:1 and post:2. The helper that turned a result into ids dropped the kind, so loading the Emp entity from the result looked up [1, 2], and Emp 2 is a row of another tenant that did not match anything:

$result = $fuzzphony->in('site')->forTenant(1)->query('ada')->get();   // hits: emp:1, post:2
$result->ids();                                                         // [1, 2]  (the kind is gone)
$loader->load($result, Emp::class);                                     // loads Emp 1 and Emp 2 (tenant 2)

The fix uses only the hits whose source reads the requested class. I never would have written a test for it, because the single-source world, where ids are enough, was the only one I was thinking about.

2. with() leaked a granted column under another name. The grant check looked at the field name; returnable lists column names. A field pay reading the column salary and granted to ROLE_HR was refused when asked as with('pay'), and returned to everybody when asked as with('salary'). The check now follows the column.

3. The ORM sync refreshed only the first index of an entity. A second index of the same entity, or a multi-source index reading it, was never refreshed: deleted rows stayed findable, and a row moved to another tenant stayed in the old tenant's results.

None of these has a surviving mutant to point at. The code for each was covered, the lines ran, and the tests asserted what I had thought of. A mutation score finds the tests I forgot to write for code I wrote; it cannot find the requirement I forgot. Only someone (or something) with a different mental model can, and so I now treat an independent review as part of the release, next to the suite, not after it.

Other safety nets that earned their place:

• A differential harness. For anything that touches ranking, a script runs 1,854 searches against the previous release and the new code and demands byte-identical results and statements, with the sequential-scan synchronisation switched off so timings are comparable.
• CI jobs that are not the happy path. The tests run against several PostgreSQL versions, with the oldest allowed dependencies, and against the next Symfony. In this release the first CI run was red for three reasons my machine had not shown: a demo smoke job that did not seed the new tables, a NAN in an error message that PHP turned into a warning (and PHPUnit into a failure), and an old Doctrine whose stub returned null where the new one returned an array.

What's next

0.12 is ranking by the best value of a field with several values (with a per-search switch for "one value must hold all the words"), a first step to a smaller sidecar, and the remaining mutation gaps. After that, 1.0: a stable API and a backward-compatibility promise.

The library is on GitHub (er2es/fuzzphony), and the demo runs with one docker compose up. If you have a database you cannot change and a search box that disappoints your users, I would like to hear what breaks first.