TL;DR
A Streamlit dashboard for retail foot traffic analysis needs typeahead search over about 1.2 million store locations in Japan. Streamlit's execution model makes per-keystroke interaction structurally awkward, and the search itself needs Japanese text normalization plus partial matching that a simple substring filter cannot express.
The fix is a small Rust service running beside the Streamlit container, serving search from an in-memory index.
The decisive reason for Rust was not raw speed. It was that a single Rust process can put a 210 MB index on the heap and let every CPU core read it concurrently. Node and Python cannot, and that turns into a multiple of the memory bill. Everything else in the design follows from that one fact.
The app and the data model
The product is an internal analytics dashboard for retail location intelligence. A user picks store locations and gets a report about who visits them: catchment areas, travel time isochrones, visitor demographics, and comparisons over time. It is built in Streamlit, which lets a small team ship data-heavy tooling in Python without a frontend build pipeline.
The data model has two levels, and they matter for everything downstream.
A chain is a brand or operator. A POI (point of interest) is one physical location belonging to that chain. So a convenience store brand is a chain, and each of its several thousand branches is a POI. There are roughly 11,000 chains and roughly 1.2 million POIs.
The display label for a location joins the two:
{chain name} - {location name}
That separator is presentation only. It is not part of the data and, as it turns out, it must not be part of the search text either.
The selection UI mirrors the hierarchy. Pick one or more chains, then pick locations within them, then render the report. That two step flow is where the trouble starts.
What Streamlit's execution model does to interactivity
Streamlit's core idea is that your script is the UI. On every interaction it reruns the script from the top, and whatever widget calls execute produce the current view. It is a genuinely elegant model for dashboards, and it removes an enormous amount of frontend work.
It also means there is no cheap way to handle "the user typed one character."
A rerun re-executes the whole script. Caching helps, and the usual toolkit is @st.cache_data, st.session_state, and fragments to limit what re-renders. But the mental model is still full script execution per interaction, and a per keystroke rerun over a report heavy page is not something you want to attempt.
The escape hatch is a custom component, which runs as JavaScript in an iframe and talks to Python over a message bridge. That works well, with one catch: sending a value back to Python triggers a rerun. So a component that filters a list has two options. Either it round trips to Python on every keystroke, incurring a rerun each time, or it receives all its options up front and filters entirely client side.
The second option is the only viable one, and it sets a hard ceiling: every candidate the user might search has to be in the browser before they type.
That is why the UI is two step. Sending 1.2 million options to the browser is not an option, so users must narrow by chain first. To keep even that tolerable, the app builds several layers of caching in front of the database: a shared Redis layer holding the chain list with a one day TTL, plus per chain caches of the location lists, so that a coordinator round trip to the Postgres cluster happens at most once per chain.
All of that work is compensating for a constraint that has nothing to do with the data and everything to do with where the filtering happens.
Three ways the search itself is hard
Even setting the delivery problem aside, the matching is not trivial.
Users do not think chain first
The two step flow assumes you know the brand before you know the store. Real usage is often the reverse. Someone wants "the locations in this district," across brands. The hierarchy in the data model got baked into the interaction model, and it should not have been.
Japanese text has many spellings for one string
A single name can appear in katakana or hiragana, full width or half width, with historical or modern kanji forms, and with Unicode variation selectors attached. These are entirely different byte sequences that a human reads as identical.
Unicode NFKC normalization handles width and compatibility forms. It does not handle the kanji variants, which are common in Japanese business names. So the normalization pipeline runs NFKC first, then a Japanese transliteration library for script folding, historical character forms, iteration marks, and variation selector removal, then lowercasing and punctuation stripping.
The important property is that normalization is additive, not destructive. The original text is kept for display, and the normalized text is stored as a separate derived column. Normalization is lossy and one way, so there is no reverse function. You match against the derived column and you display the original.
struct Corpus {
display: Vec<String>, // shown to the user, never modified
search: Vec<String>, // normalized, matched against, never shown
}
This is the same relationship a database has with its indexes. Use the index to find the row, then read the row.
Substring matching answers the wrong question
Here is the case that rules out a simple filter. Because a location name almost never repeats its own chain name (measured at 99.6% of records), the searchable text has to be the chain and the location concatenated, so that a query spanning both still matches.
But then a user types the brand plus a district, and the stored record has extra words in the middle. The query is genuinely not a substring of the record. haystack.contains(query) returns false, and no amount of optimization fixes that, because "does this contain that" is the wrong question.
The right question is "how much of this query does this record cover," which requires breaking the query into pieces and scoring partial matches.
The constraint that picked the language
The index has to live in memory. Roughly 210 MB for the text plus the postings, based on the layout described below, loaded once at startup and then serving every keystroke without touching Redis or the database.
The question that decided everything: what happens when you want more than one CPU core?
Node runs JavaScript on one thread, so one core. Four cores means four processes, and processes do not share heaps. That is four copies of the index. worker_threads does not rescue this, because each worker gets its own V8 isolate with its own heap. The only shareable thing is a SharedArrayBuffer, which holds raw bytes and no objects.
Python threads do share a heap, but the GIL means only one runs bytecode at a time. Sharing without parallelism. The standard fix is multiprocessing, which lands right back at N isolated heaps. Free threaded Python changes this and is worth watching, but it was not an option here.
Rust runs N OS threads inside one process, and threads share an address space. One index, every core reading it concurrently, no locks required because nothing mutates after load.
| Runtime | Shares one heap | Uses N cores | Index copies on 4 cores |
|---|---|---|---|
| Node, cluster | no | yes | 4 |
| Node, worker_threads | no | yes | 4 |
| Python, threads | yes | no | 1, serialized |
| Python, multiprocessing | no | yes | 4 |
| Rust, tokio | yes | yes | 1 |
At 210 MB per copy that is the difference between provisioning around 1 GB and around 2 GB per container, plus four independent cold start loads on every deploy instead of one.
That is the argument. Not that Rust is fast. That one process can hold one index and let every core read it.
How the service is built
Three layers and one rule.
controller HTTP only. Parse query params, call the service, serialize.
service Pure logic. Imports no web framework and no storage client.
repository I/O. Fetches bytes, returns plain domain types.
The rule is that service never imports the web framework or the storage driver. That single constraint is what keeps storage swappable, and it is worth more than any amount of interface ceremony.
Data flows one direction:
Postgres (source of truth)
| batch job builds a versioned snapshot
v
Redis (distribution, not a request path dependency)
| loaded once at boot, and on refresh
v
Process memory (answers every request)
Redis is a delivery mechanism, not a cache in front of queries. Once a process has loaded, Redis can go down and search keeps working. Snapshots publish blue green: write every chunk of the new version, verify, then flip a single pointer key. A reader sees the old version entirely or the new one entirely, never a mix.
The in memory layout is columnar rather than an array of structs:
// not this
struct Record { group: String, name: String, key: String }
let records: Vec<Record>;
// this
struct Corpus {
groups: Vec<String>, // deduplicated, referenced by index
group_idx: Vec<u32>, // which group each record belongs to
names: Vec<String>,
keys: Vec<String>,
search: Vec<String>, // normalized
}
"Record 7" does not exist as an object. It is the number 7, used to index every array. Two consequences.
Deduplicating the group name costs 4 bytes per record instead of a full string. With 11,000 distinct groups across 1.2 million records, that is roughly 4.8 MB instead of roughly 65 MB for identical information.
Search runs entirely on integers. A candidate set of 300 is 2.4 KB of numbers, not 300 objects. Strings are touched only for the handful of results actually returned, and the response payload is built on demand rather than stored per record.
The index
Japanese has no spaces, so splitting on whitespace is not available. Morphological analysis works but needs a dictionary measured in tens of megabytes. The alternative is n-grams, and bigrams are the practical choice: one character is not selective enough, three means short queries match nothing.
"alphabeta" -> al lp ph ha ab be et ta
Invert it, so each bigram maps to the records containing it:
"ph" -> [7, 91, 2043, ...]
"ha" -> [7, 512, ...]
A query then looks up its own bigrams, tallies how many each record matched, weights rare bigrams above common ones with standard IDF (ln(N / df)), keeps the top few hundred, and reranks those with more expensive checks. Partial coverage gets partial credit, which is exactly what substring matching could not express.
The layout matters more than the algorithm. A hash map from bigram to a vector of IDs means one heap allocation per bigram, plus hash overhead, plus per vector capacity slack, which measures at roughly 4 to 5 times the memory of the alternative.
The alternative is CSR, borrowed from sparse matrix code. Concatenate every posting list into one flat array and keep an array of offsets:
offsets: [ 0, 3, 7, 8, 12 ]
postings: [ 7,91,2043 | 5,7,88,90 | 3 | 12,44,55,67 ]
A posting list becomes a slice, &postings[offsets[b]..offsets[b+1]]. Two allocations for the whole index instead of hundreds of thousands, and every list is contiguous so hardware prefetching works. Build it by counting occurrences, prefix summing the counts into offsets, allocating exactly once, then filling. That is counting sort, and it is the entire trick.
Concurrency, measured
Tokio is an M:N scheduler. Many cheap tasks are multiplexed onto a small number of OS worker threads, one per core by default. Each .await is a yield point, and between yield points a task owns its worker outright.
Which means async provides no parallelism for CPU bound work. Parallelism comes from having N worker threads. A handler that never awaits is a synchronous function wearing an async hat.
Pinning a runtime to a fixed worker count and giving it 8 tasks, each needing 200 ms of pure CPU:
worker_threads( 1) -> 1600 ms
worker_threads( 2) -> 800 ms
worker_threads( 4) -> 400 ms
worker_threads( 8) -> 200 ms
Perfectly linear. Note that one worker is essentially the Node model, and no amount of async beats 1600 ms with one pair of hands.
The matching trap is blocking a worker. The same 8 requests, three handler styles:
CPU bound directly on the worker 1645 ms
offloaded via spawn_blocking 412 ms
real awaited IO 413 ms
The rule that falls out: if it does not .await and takes more than about a millisecond, it does not belong on a worker thread. The index build takes about 14 seconds, so it goes to the blocking pool. Blocking a worker for 14 seconds on a 4 core box costs 25% of capacity, health checks included.
The shared state itself is an atomically reference counted pointer, which is what makes the one copy design work. Handing it to a request costs a pointer and a counter bump rather than the data:
clone the shared pointer 11.46 ns
deep copy the same payload 51.7 ms (1M strings)
Roughly four and a half million times apart. That gap is the entire argument for shared ownership, and it is why per request state is 8 bytes rather than 210 MB.
Numbers
Dev machine, 12 cores, synthetic workloads unless marked otherwise.
| Metric | Value | Source |
|---|---|---|
| POIs | ~1.2 million | actual |
| Chains | ~11,000 | actual |
| Location names not containing their chain name | 99.6% | actual |
| Snapshot on the wire | ~49 MB gzipped | actual |
| Steady state index memory | ~210 MB | estimated from layout |
| Same index under a 4 process runtime | ~840 MB | estimated |
| Text normalization | ~12 microseconds per record | measured |
| Full index build | ~14 seconds, single threaded | measured |
| Shared pointer clone | 11.46 ns | measured |
| Deep copy, 1M strings | 51.7 ms | measured |
| Worker scaling, 8 x 200 ms tasks | 1600 / 800 / 400 / 200 ms | measured |
| Blocking vs offloaded | 1645 ms vs 412 ms | measured |
Lessons worth keeping
Build the slow version first. The plan deliberately runs a naive linear scan over plain vectors before the inverted index and the compact layout. It provides a correctness baseline and a number to beat. Skipping to the clever version leaves you unable to tell whether it is right or whether it helped.
Normalization belongs in exactly one place. Precomputing normalized text in the Python batch job would save the 14 second build, at the cost of two implementations of the same function in two languages that must agree byte for byte forever. When they drift, queries normalize one way and the index another, and search silently returns nothing. No error, no crash, just empty results. Paying 14 seconds on a background thread is clearly the better trade.
Health checks must gate on readiness. A fresh instance needs most of a minute to fetch, decompress, parse, and normalize before it can answer anything. If the health endpoint returns 200 during that window, the load balancer routes real users to an empty index and they get zero results with no error. That presents as a search bug and is actually a deployment bug.
The UI shape was a data model leak. Chain first selection existed because the delivery mechanism could not handle anything else, not because users think that way. Moving search server side did not just make it faster, it removed a step that should never have been there.
The interesting conclusion is that the decisive factor was not throughput. It was that one process can hold one index and let every core read it. Once that became the binding constraint, most of the rest of the design followed on its own.
Top comments (0)