DEV Community: Samuel Chan

Building of a Financial Search Engine

Samuel Chan — Wed, 19 Aug 2026 03:45:11 +0000

The following article is a technical piece I wrote for Sectors, a dev-friendly Financial Data Platform built for the Singapore and Indonesia financial markets (original version with more interactive examples here). You can read more about what I do on my profile: Samuel Chan.

The Technical Pieces of a Financial Search Engine

Before any of the engineering, it helps to be honest about what people are actually trying to do when they search a financial platform, because it is not one thing. Someone might know the three- or four-letter ticker and want to jump straight to it (e.g. "O39" for OCBC Bank, "BBCA" for Bank Central Asia).

More often they remember a fragment of a name, or the sector a company sits in, or the family that controls it, or a half-formed phrase like that "coal company in Kalimantan" or a "aerospace company that makes satellites." They might be looking for a company, a broker, a shareholder, a key person or a conglomerate -- any of which could be the right answer to a Search Console query that is only half-remembered, misspelled, or ambiguous.

Our Search Console must be able to absorb all of that imprecision and still land the user on the right page, and it must do so across more than one kind of entity. A company is searchable, but so is a sector, an index, a broker, a major shareholder, and a corporate group that ties dozens of listed companies together. In the search space, these are all first-class citizens and a multi-modal query should return any combination of them that is relevant.

So where does that leave the architectural requirements? We need breadth, precision under ambiguity, and trust.

Breadth means the corpus must be wide enough to cover every entity a user might be looking for, and it must be able to tolerate half-remembered input.
Precision means the ranking must be able to cut through near-duplicates and understand which of the matches is most likely what the user intended.
Trust means the system must be deterministic and fast enough to keep up with typing, so that a user can feel confident in the results they see.

That breadth is the first prerequisite, and it is mostly a data problem rather than an algorithmic one. Each entity carries several identities at once. A single company on the Indonesia Stock Exchange has a ticker, a formal legal name wrapped in boilerplate like "PT" and "Tbk," a common name people actually say out loud, a sector classification, a paragraph of business description, and a web of ownership relationships. Any one of those is a legitimate way in. The job of the corpus is to gather all of them, normalise away the noise that gets in the way of matching (the honorifics in a person's name, the legal suffixes on a company), and resolve identities that appear in more than one dataset so that a shareholder named one way in an ownership filing lines up with the same person named slightly differently elsewhere.

The second prerequisite is precision under ambiguity. Financial corpora are full of near-duplicates. Dozens of names begin with "Bank," a great many descriptions mention banking, and a substring as innocent as "pan" hides inside plenty of unrelated words. A search that merely finds everything matching the query is close to useless here, because everything is a lot. What separates a good financial search from a noisy one is the ranking: the ability to understand that a query is probably aimed at a name rather than a description, that a rarer word in the query carries more intent than a common one, and that a result reached only through an indirect relationship deserves to sit below a direct hit. Precision is not a finishing touch in this domain. It is the product.

The third prerequisite is trust, and trust in this context mostly means determinism and speed. People making decisions with money do not want a search box that occasionally invents an answer or pauses to think. For the common navigational query, they want the same correct result every time, returned faster than they can perceive. In the case of our AI Search, we want each query to return exactly the same answer every time, along with an explanation of the query path it takes to arrive there. Users of AI Search must be able to trust that the answer they get is the answer they would have gotten if they asked again, and that it is grounded in the data rather than hallucinated.

That speed requirement pushes us toward a structural, in-memory approach for the bulk of traffic, with heavier natural-language and AI-assisted retrieval reserved for the genuinely open-ended questions.

Gather every identity an entity has and normalise it, retrieve, rank while
cutting out near-duplicates, and deliver all of it at the speed of typing.

With those requirements in mind, we started designing a search architecture that could meet them, and this article is a tour of the pieces we ended up with, in the order that a query travels: how the Sectors engineering team builds and holds the corpus, how we retrieve against it, how we rank what comes back, how we sharpen that ranking with term weighting, and how we keep the whole pipeline off the thread the user is typing into.

An In-Browser Search Architecture

Even before the Search Console update, our Search was a critical part of the Sectors experience and ranked as a top feature in anonymized usage analytics. People reach for it the way they reach for the address bar in a browser, and that sets a high bar: the first keystroke should already be doing useful work, and the tenth should feel no slower than the first.

The decision that shapes everything else is that our primary search runs entirely in the browser. There is no round trip to a server for the common case of finding a ticker, a sector, a conglomerate, or a shareholder. The corpus that powers the search console experience is built and refresed periodically (typically twice a week) offline, and when ready, it is shipped to the client (your browser) and gets indexed there in-memory. This decision means that the slowest part of a query is no longer the network. It is whatever work we choose to do between the user pressing a key and the results painting.

We still lean on server-side and AI-assisted search for natural-language
queries, and we have written about that approach in Building Search Engines
in the age of AI. This article is about the layer
underneath it: the deterministic, instant, structural search that handles the
overwhelming majority of queries without ever leaving the tab.

There are four moving parts worth describing on their own terms. We build an in-memory index over several datasets, we query that index with substring semantics across multiple scopes at once, we re-rank the raw matches into something that respects user intent, and we do all of the indexing and searching off the main thread so the interface never stutters.

The sections below tackle each of those pieces one at a time.

Indexing

The retrieval primitive we build on is a compact in-memory index. It exposes a deliberately small surface: you add a document with index(uid, text), and you retrieve matching identifiers with search(query). The interesting design choice is not the primitive itself but how we feed it.

Every company has several fields a financial analyst or market research might search by: its ticker, its legal or display name, and a simple line of business description. Rather than maintain three separate indices, we register all three fields under the same identifier, the ticker. The index merges them into one searchable document keyed by that ticker, so a query that hits any field resolves back to the same company.

This collapsing of fields under one identifier is convenient for retrieval and, as we will see later, mildly inconvenient for ranking. It is a trade we make on purpose. Retrieval is the hot path and should stay simple; ranking, on the other hand, is where we are afforded more complexity to solve the precision problem.

The second indexing decision is about when each dataset becomes available, and here we are explicitly progressive rather than eager. The Indonesian stock index is built synchronously when the dialog mounts, because it is the dataset most queries touch and we want it ready before the user has finished typing the first word. The conglomerate dataset is heavier, so we import it lazily and build its index in the background once the more urgent work is done.

// IDX is ready immediately. Everything else streams in behind it.
useEffect(() => {
  buildIdxIndex();
  ...
  (async () => {
    const groups = await loadConglomerates();
    buildGroupIndex(groups);
  })();
  // SGX (Singapore) stocks waits on a small fetch
  // to determine which tickers are valid, then builds in the background.
  loadValidSgxTickers().then((valid) => buildSgxIndex(valid));
}, []);

The Singapore dataset index waits on a small fetch, filters the static description file against the set of valid tickers, and only then becomes searchable. The user sees Indonesian results the instant they type, Singaporean results a few milliseconds later, and conglomerate results once the larger payload has settled. Nobody waits on the slowest dataset to see the fastest one.

The reason this staggering is safe is the same reason the whole thing is fast: each index is independent, and the query layer treats a not-yet-built index as simply contributing nothing yet. There is no global "ready" gate that the entire feature blocks on. Readiness is per-index, and the experience degrades gracefully from the moment the dialog opens.

Searching

A query in our search bar is not always a plain string. It is closer to a command palette (like the one in VS Code, or Claude Code), where a leading slash selects a scope before the rest of the input is treated as the search argument. Typing /sg banks narrows the search to Singapore; /id narrows it to Indonesia. The grammar is small and deterministic, parsed before anything touches the index, so scoping never costs a model call or a network hop.

const parsed = parseCommand(query);
const rawQuery = parsed.arg.trim();

const scope = parsed.kind === "scope" ? parsed.scope : null;

Once we know the scope, we query every relevant index concurrently. There is no reason to search equities, sectors, and conglomerates in sequence when each is an independent in-memory lookup, so we fire them all off at once and wait for them to come back together. The search layer does not care which indices are in play, so it can treat a missing one as simply contributing no results rather than an error state.

const [idxUids, sgxUids, sectorUids, groupUids] = await Promise.all([
  searchIdx ? idxIndex.search(rawQuery) : Promise.resolve([]),
  searchSgx ? sgxIndex.search(rawQuery) : Promise.resolve([]),
  sectorIndex.search(rawQuery),
  groupIndex.search(rawQuery),
  ... // any future indices go here without
  // changing the shape of the code that follows
]);

The matching itself is substring-based across the merged document. A multi-word query is conjunctive: searching for bank pan returns companies whose indexed text contains both bank and pan somewhere, in any field. That is a powerful default for discovery, because it lets a half-remembered name or a sector keyword surface the right company.

It is also, by its nature, generous. A two-word query against a corpus this dense can return thirty or forty candidates ("noisy result sets"), and substring matching will happily count pan inside expand or Japan. Retrieval gives us a set of things that match. It says nothing about which of them the user actually meant (i.e. the "intent" behind the query).

That distinction, between membership and intent, requires Sectors engineering to build second layer of logic on top of the raw retrieval, a relevance model that can understand which of the matches is most likely what the user intended and rank it accordingly. The next two sections are about how we do that.

Relevance Ranking

The raw output of retrieval is a list of identifiers in index order. There is no score attached, and because we deliberately collapsed name, ticker, and description under a single identifier, the index cannot even tell us which field produced the match. A company that literally is "Bank Pan Indonesia" comes back indistinguishable from one whose description happens to contain both words in unrelated sentences.

In fact, when the Search Console first shipped, that was exactly the state of affairs. Sectors had a powerful index and an extremely fast retrieval, but the result sets were not ranked at all, with "Bank Pan Indonesia" sitting somewhere in the middle of the list rather than at the top. This was surfaced by @jigsawinthecity immediately, and confirm a problem we had anticipated but remained unsolved.

So we got to work and reconstruct the signals, building upon the earlier architectural foundations for our financial search engine.

After the retrieval step, we examine each candidate against the raw fields we already hold in memory and derive a small set of ranking signals. The priority order we want is intuitive and worth stating in plain language before any code: a match in the name beats a match in the description, and both beat a match reached only through an indirect connection such as a shareholding link. Within that, an exact phrase in the name is the strongest signal of all.

We capture those signals as a key, computed per candidate:

export type SearchRankKey = {
  inFull: number;
  inName: number;
  inDesc: number;
  lq45: number;
};

The temptation here is to flatten these into a single weighted score, something like 1000 * inFull + 100 * inName + 10 * inDesc. This is the approach taken by many search engines, and it can work well when the signals are continuous and the weights are carefully tuned. While seemingly robust, it actually can become brittle and opaque as scoring constants proliferate and evolve, sometimes in non-obvious ways. Programmers call them "magic numbers" for a reason -- they exist to make the math work, not because they have inherent meaning, and that makes them hard to justify and easy to break.

What we actually want is a stricter, more predictable ordering; one that uses a tiered approach to relevance, with a comparator consulting each signal in turn and only moving to the next one when there is a tie. A match in the name should always beat a match in the description, no matter how many description matches there are, and an exact phrase match should always beat a partial token match, no matter how many of those there are.

The comparator is simple and transparent, with no constants to tune and no risk of one signal quietly drowning out another.

// Each `||` is a tier. A later signal only matters when the earlier ones tie.
export function compareSearchRank(a: SearchRankKey, b: SearchRankKey): number {
  return (
    b.inFull - a.inFull ||
    b.inName - a.inName ||
    b.inDesc - a.inDesc ||
    b.lq45 - a.lq45
  );
}

This reads top to bottom as exactly the priority we described, with no constants to justify and no risk that twenty weak description hits quietly outweigh one strong name hit. Sorting is stable, so candidates that genuinely tie keep their retrieval order, which is predictable and deterministic (recalling our trust requirement from earlier). The comparator is also easy to test in isolation, which is a nice bonus.

The ranking layer also gives us a natural place to solve the "noisy result sets" problem. Once a query has produced any name-tier matches, the description-only and connection-only tail is almost always noise, e.g. the pan-inside-expand or pan-inside-Japan accidents.

So we apply a cutoff: if strong matches exist, we drop the weak tail; if nothing matched a name at all, we keep everything, because a narrow query should never be starved down to an empty list.

export function applyRelevanceCutoff<T extends { key: SearchRankKey }>(
  ranked: T[],
): T[] {
  const strong = ranked.filter((r) => r.key.inFull > 0 || r.key.inName > 0);
  return strong.length ? strong : ranked;
}

The cutoff is deterministic and explainable, which matters. When a user asks why a particular result did or did not show up, "it had no match in the name and there were better matches available" is an answer the Sectors Engineering team can stand behind. An opaque score threshold is not.

Lite-IDF Weighting for Intent and Discrimination

The comparator gets the tiers right, but it exposed a subtler problem the moment we used it on real queries. Consider bank pan again. "Panin Financial" matches on the name (albeit having ranked lower in the result sets), through the token pan. But then does "Bank Jago", through the token bank -- even when the user intent provides no evidence to support "Bank Jago" over "Panin Financial."

If inName is a simple count of matched tokens, both score one, they tie, and the tiebreaker decides the order more or less arbitrarily. The blue-chip banks float up and "Panin Financial", which is far closer to the intent of what someone typing "pan" is looking for, ends up buried below them.

The issue is that not all tokens carry the same amount of information. In a corpus of Indonesian listed companies, bank appears in dozens of names and discriminates almost nothing. pan appears in a handful and is highly discriminating. Treating a bank match and a pan match as equally valuable is the bug.

This is the exact problem that inverse document frequency was invented for, so Sectors Engineering borrowed this idea from information retrieval ¹ and implement in pure typescript code our own version of it, a version we call "lite-idf".

Here's the basic premise (not our own original idea, but worth stating plainly) of TF-IDF: a token that appears in many documents is less informative than one that appears in few, so we should weight the latter more heavily when it matches. In our case, the "documents" are the candidate names we already have in hand, and the "tokens" are the words in the query.

So here's the game plan we cooked up:

We take the query and split it into tokens, e.g. bank pan becomes ["bank", "pan"].
We take the candidate names and compute how many of them contain each token, e.g. ["Bank Jago", "Panin Financial", "Bank Central Asia"] contains bank in two names and pan in one.
Then, we compute the weight for each token based on its rarity, so rarer tokens contribute more to the score.

Rather than counting matched tokens, we weight each token by how rare it is, so matching a rare token contributes more to the score than matching a common one. We compute the document frequency over the candidate names we already have in hand, so there is no separate statistics table to build or keep in sync.

Here is how the math works out in code. The tokenNameWeights function takes the query tokens and the candidate names, counts how many names contain each token, and computes a weight for each token based on its rarity.

export function tokenNameWeights(
  terms: string[],
  names: (string | null | undefined)[],
): Map<string, number> {
  const lowered = names.map((n) => (n ?? "").toLowerCase());
  const total = lowered.length;
  const weights = new Map<string, number>();

  for (const t of terms) {
    let df = 0;
    for (const n of lowered) if (n.includes(t)) df++;
    // Rarer token -> larger weight. The +1 keeps it finite when df is 0.
    weights.set(t, Math.log(1 + total / (1 + df)));
  }
  return weights;
}

Two small details in the formula above: The 1 + df in the denominator handles the case where a token matches no name at all, which would otherwise result in a divide by zero error. The 1 + in the logarithm keeps the weight finite when a token matches every name, which would otherwise result in a log(0) error.

Finally, the logarithm compresses the dynamic range, so a token that is fifty times rarer does not get fifty times the weight and steamroll everything else. With those weights, pan ends up worth roughly twice what bank is worth, which is enough to lift "Panin Financial" above the generic banks without disturbing the tier structure around it.

The part I am happiest with is how it plugs in. The ranking function does not know that weighting exists (it doesn't have to). It accepts a weightOf callback that defaults to returning one, which means the ranker on its own is still the plain count-based version, fully testable in isolation. Weighting is injected from the outside, and the two concerns, the structure of the ranking and the importance of individual terms, stay orthogonal.

export function rankSearchEntry(
  entry: { name: string | null; description?: string; isLQ45?: boolean },
  terms: string[],
  weightOf: (token: string) => number = () => 1,
): SearchRankKey {
  const n = (entry.name ?? "").toLowerCase();
  let inName = 0;
  // ... accumulate weightOf(token) per matched token instead of a raw count
}

That additivity also guarantees an invariant we care about, for free.

A name that matches both tokens scores the sum of two positive weights and therefore always beats a name that matches only one. Weighting can reorder candidates within a tier, but it can never violate the larger rule that more name coverage is better. We get the nuance of term importance without giving up the guarantees of the tiered model.

OK, but surely this has limitations?

Yes, but a small one. The document frequency is computed over the result set rather than the whole corpus, which is cheap and self-calibrating but means a single-token query has nothing to discriminate on, every candidate matched the one token, so the weights collapse to equal. We consider this harmless, because rarity only needs to matter when there is more than one term in play. And because the underlying match is still substring-based, the frequency counts inherit that bluntness. These are deliberate simplifications, not oversights², and the implementation that Sectors Engineering ended up strikes a good balance between the ideal and the practical, with a meaningful boost for rarer tokens without the cost of building and maintaining a full statistics table over the corpus.

If you haven't got to try Sectors Search Console yet -- do it before you continue reading so you can see the effect of this weighting in action by typing bank pan into the search bar. The Panin entities should float to the top above the generically-named banks, with pan being worth about twice as much as bank in the name matches.

The whole addition we make since the Search Console update is this weighting layer, and it is a great example of how we can solve a problem that emerges from the real world with a small piece of code that plugs into the existing architecture, keeping the whole thing to a handful of lines that run in microseconds.

Debouncing and Highlighting the Query

Two small pieces sit on either end of the pipeline we have described, and both exist for the same reason: a person types in bursts, not in deliberate single keystrokes, and the experience should feel like it is keeping pace with the typing rather than reacting to every individual character.

At the front of the pipeline is debouncing³. The textarea updates query on every keystroke, because the input box must always feel immediate, but the value that actually drives indexing, retrieval, ranking, and re-sorting is a debouncedQuery that trails it by a short window. A burst of five characters typed in a tenth of a second collapses into a single search instead of five, and the expensive part of the pipeline only runs once the user has paused.

useEffect(() => {
  const id = setTimeout(() => setDebouncedQuery(query), 150);
  return () => clearTimeout(id);
}, [query]);

Is this truly robust? Almost. Adding debouncing is the kind of thing that is sometimes not worth the risk as it can introduce edge cases like stale queries or flickering results if not handled carefully, for seemingly innocent reasons. Consider the moment a user types the last character that crosses the minimum-length threshold for search. The debouncedQuery is still trailing behind, so for a moment it holds a value that no longer reflects what the user has on screen. If we fed that stale-but-too-short value into the search, the result list would flash the default state (e.g. "type at least three characters") before updating to the real results.

In our case though, that is fully mitigated by the fact that we fall back to the live query when debouncedQuery is below the threshold, so the search never runs against a stale-but-too-short string and the result list never flashes the default state on the way to a real answer.

const searchQuery =
  query.length < MIN_SEARCH_LENGTH
    ? ""
    : debouncedQuery.length >= MIN_SEARCH_LENGTH
      ? debouncedQuery
      : query;

The in-dialog debounce above is hand-rolled because it is tangled up with that threshold logic, but the same idea is needed all over the Sectors application wherever a keystroke triggers asynchronous work, so we also keep it as a tiny reusable hook, useDebouncedValue.

It returns a value that only updates once its input has stopped changing for the given window of time, so it can be used to debounce any value, not just the search query. In essence, a debounced value coalesces a burst of input changes into a single fetch rather than firing one request per character. The real-time news column in Sectors Search Console does exactly this, debouncing the target by 200ms before the fetch.

Search Highlighting

At the other end of the pipeline, once results are painted, is highlighting. Having gone to the trouble of understanding which tokens in the query carried intent, it would be a shame not to show the user where those tokens actually landed in each result. The useTextHighlight hook takes the same query tokens we parsed for ranking and wraps every occurrence of them in the rendered names, descriptions, and shareholder labels in a <mark>, so the match is visible at a glance rather than something the user has to hunt for.

useHightlight({
  searchWords: tokens,
  enabled: tokens.length > 0 && !(aiResult && Array.isArray(aiResult.results)),
  highlightClassName: "highlight",
});

The hook finds the spans and returns a highlight(text) function that the result rows call directly, e.g. {highlight(item.name)}.

A few light engineering touches: we apply deduplication and trim the search words so an empty or repeated token does nothing, and it is fed from the same debouncedQuery as the search itself, so the highlights never lag behind or run ahead of the result set they are annotating.

The visual reinforcement therefore closes a loop that the ranking opened, so the user can see not just that "pan" is more important than "bank" but also where "pan" is actually landing in the results, which is a nice bit of feedback to have when they are trying to figure out how to phrase their query.

Driving the Results from the Keyboard

We opened this article by comparing the search box to the address bar in a browser, and there is a corollary to that comparison that is easy to underrate: the address bar is something you never have to touch the mouse to use. You type, you arrow down to the suggestion you want, you press enter, and you are gone.

We want to ship a financial search engine that produces an identical console-like experience, where the more savvy powerusers can fly through the results by driving the search entirely from the keyboard, without having to lift their hands to reach for the trackpad and click on a result.

Sectors Search Console is therefore fully keyboard-drivable, and the machinery that makes it so is another part the Sectors Engineering team is deliberate about.

The first thing to notice is that the results are not a single list. They are laid out as parallel columns: equities and sectors on the left, then a stacked middle of ownership records, key people, and news. Each column carries its own cursor, so the navigation problem is genuinely two-dimensional rather than the one-dimensional up/down that most search boxes ship. We track that as a tuple of row positions, one per column, alongside the column that is currently active.

// activeCol: 0=tickers, 1=ownership, 2=people, 3=news
const [activeCol, setActiveCol] = useState<0 | 1 | 2 | 3>(0);
const [rowIdx, setRowIdx] = useState<[number, number, number, number]>([
  -1, -1, -1, 0,
]);

A column whose cursor is -1 is idle, with no highlighted row, which is exactly the state we want when the dialog opens or when the user has just edited the query and the old result set no longer exists. It means the visible highlight is always a faithful reflection of the internal state, because "nowhere yet" is a state we can represent rather than having to fake with row zero.

The down and up arrows move within a column, but they also know how to step between the stacked sections. When the cursor reaches the bottom of the ownership list, the next press of ↓ does not stop dead; it falls through to the next non-empty section below it, and ↑ does the symmetric thing. The left and right arrows treat the layout the way the user sees it, as two columns rather than four: a single hop between the left equities column and whichever of the stacked sections actually has rows in it.

The part that takes the most care, though, is not the grid itself but everything that has to be allowed to override it. The same handleKeyDown is the entry point whether the user is typing a scope command, sitting on the empty splash screen, looking at a "did you mean" suggestion, or navigating real results, and each of those is effectively a different modal surface fighting for the same keys.

More concretely, while the cursor is still resting on the search bar with no row selected, the left and right arrows are deliberately not captured, because at that moment the textarea is a text field first and a navigator second. The user should be able to move the caret and edit a word in the middle of their query; column navigation only takes over once ↓ has actually moved the cursor down into the results.

// Cursor still on the search bar: the textarea is a text field first,
//  so let ←/→ move the caret and edit the query mid-string.
if (
  (e.key === "ArrowLeft" || e.key === "ArrowRight") &&
  activeCol === 0 &&
  rowIdx[0] === -1
) {
  return;
}

On top of the grid sit a few hotkeys that only springs to live once the user is in keyboard-nav mode, so they can never react to a keystroke meant for the query.

Pressing b on a focused company or sector toggles it in the watchlist; Cmd/Ctrl+Enter escalates the current query to the heavier AI and natural-language search rather than the instant structural one. And there are small touches of feedback woven through, like a flash on the search bar when pops the cursor back out of the results, so the boundary between "navigating" and "typing" is something the user can see and not just infer.

None of this changes a single result or its ranking. It is entirely about the last few inches between a correct answer being on screen and the user landing on it, and experience is the sum of those inches. The Sectors Engineering team has spent a lot of time on them, and we are proud to present a keyboard-driven financial search engine that is as fast to navigate as it is comprehensive in coverage.

Web Worker Support

Everything described so far has a latency budget measured against a single unforgiving constraint: it has to keep up with typing. A person types several characters a second, and every one of them can trigger a fresh index build check, spawn several concurrent searches, triggers one or more re-rank, and one or more re-sort.

If any of that runs on the main thread, the browser cannot also be laying out the page and responding to the next keystroke, and the result is a stuttery, laggy experience that feels broken -- even when the search results are correct and the search engine is marvelously well-designed.

So we push all of that work off the main thread.

The indexing work? It does not happen on the main thread. The searching work? It does not happen on the main thread. The ranking work? It, too, does not happen on the main thread.

The only thing that happens on the main thread is the user typing and the interface painting, and that is exactly what we want. Everything else is a background worker⁴ sitting on a separate thread, doing the heavy lifting of execution (e.g. to build the index, search it, and rank the results) without ever blocking the main thread. The main thread, the one painting the interface and capturing keystrokes, never does the work beyond posting the financial queries across the thread boundary and getting a list of identifiers back.

This is why every interaction with the index in our code is asynchronous. A search call returns a promise that resolves when the worker has done its work and messaged back, and index construction is fire-and-forget from the main thread's point of view. The payoff is robustness under the kind of conditions that would otherwise hurt. The conglomerate index can be building in the worker while the user is already typing and getting Indonesian equity results, and the two never contend for the same thread.

Pushing work onto another thread does introduce one classic hazard, and it is worth showing how we handle it. Because results arrive asynchronously, a slow response to an earlier query can land after a faster response to a later one and overwrite fresh results with stale ones⁵. We guard every query with a cancellation flag tied to the lifetime of that specific keystroke's effect.

let cancelled = false;

runSearch(rawQuery).then((ranked) => {
  if (cancelled) return; // a newer query has superseded this one
  setResults(ranked);
});

When the query changes, a cleanup runs and sets its flag, so any in-flight response it was waiting on is discarded the moment it returns. The user always sees results for what they have currently typed, never a flicker of an older query's answer. It is a small pattern, but it is the difference between an asynchronous search that feels solid and one that feels haunted.

The combination is what makes the feature feel instant rather than merely fast. Retrieval is cheap because the corpus is in memory. The interface stays responsive because the corpus is in memory on a thread that is not the interface's. And the results stay correct because we are disciplined about which response is allowed to win.

Closing Notes

None of the individual pieces here are exotic. An in-memory index, conjunctive substring matching, a tiered comparator, a touch of inverse document frequency, a debounce, a highlighter, keyboard navigation, a worker thread.

Where Sectors Engineering shows its craft is in how those pieces are composed, with each layer solving a problem the previous one created.

Indexing under a shared identifier makes retrieval simple and forces ranking to reconstruct field information
Generous substring matching makes discovery powerful and forces a relevance model to tame the volume
A strict tiered ranking gets the order mostly right and exposes the term-importance gap that the lite-idf weighting closes
Debouncing keeps that pipeline from running on every intermediate keystroke, and highlighting reflects the matched terms back so the user can see why a result surfaced
Keyboard navigation turns the ranked columns into something a power user can fly through without the mouse
And the worker thread is what lets all of that run on every keystroke without the user ever noticing it happening.

We revisit these decisions often, the way we revisit most architecture at Supertype, and some of them will look different a year from now. But the shape of the system, retrieve broadly, rank strictly, and keep the heavy work off the thread the user is touching, has held up well, and it is a shape I would reach for again.

If you are building information-retrieval systems of your own and want to compare notes, Sectors for Enterprise is where those conversations start.

Footnotes

1: Inverse document frequency is a classic information-retrieval weighting that scores a term by the inverse of how many documents contain it, so that common terms count for little and rare terms count for more. The canonical reference is the family of tf-idf weighting schemes; our use is a deliberately reduced version of the idf half, computed over the candidate set rather than a precomputed corpus statistic.

2: The full-strength successor to plain tf-idf in information retrieval is Okapi BM25, which adds term-frequency saturation (so the tenth occurrence of a word counts
for far less than the first) and document-length normalisation. We deliberately stop short of it: our matches are boolean substring hits rather than counted term frequencies, and our "documents" are short names, so most of the machinery BM25 gets us would have little to act on here.

3: The term debounce is borrowed from electronics, where the contacts of a mechanical switch physically bounce for a few milliseconds and the circuit must wait for them to settle before registering a single press. The software analogue waits for input to "settle" for a fixed window before acting on it, so a flurry of events collapses into one.

4: Web Workers are the browser's primitive for running scripts on a background thread, communicating with the main thread only by passing messages. They have no direct access to the DOM, which is exactly what makes them safe for CPU-bound work like building and scanning an index: they cannot touch the page the user is interacting with even if they wanted to.

5: This is a classic out-of-order-response race. The cleanup-flag pattern shown here is the idiomatic way to handle it with React effects: when the query changes, it runs the previous effect's cleanup (flipping cancelled to true) before starting the next, so a late-arriving response from a superseded query is discarded rather than rendered.

My journey in data visualization, and how it took me to Altair

Samuel Chan — Thu, 27 Oct 2022 10:44:16 +0000

It is remarkable how far we've come in terms of python tooling for charting and visualization. I came into the python scientific computing ecosystem from R (user since 2015) and Matlab (user since 2013), and my experience has been largely pleasant. I wish I could say I've never looked back since, but the truth is that I have. For all the envy of Python's enormous ecosystem, scientific graphing in Matplotlib never felt as natural as it is with Matlab. This is a cruel irony as Matplotlib borrows heavily from the syntax of Matlab, a fact that isn't exactly subtle when you learn that the "Mat" in "Matplotlib" refers to Matlab and the library retains many of Matlab's plotting terminologies (i.e "markers").

My introduction to `ggplot2`

Attached above are some of the images I've created in R, with the trusty ggplot2 library.

I remember how I came across Hadley Wickham's ggplot2. I was relatively new to R and similar to Matlab, R has a rich feature set of built-in graphing utilities, catering to scientific computing users who needed to model complex simulations to statistical user who want a one-liner to produce a faceted plot (R users fondly call them "trellis plot"). Similar to Matlab, one didn't have to bother with importing external packages as R's core plotting library offers a lot out of the box.

Then I came across a challenge by Harvard's IQSS and I was rather intrigued. The challenge was about reproducing this plot on the Economist and the materials presented an introduction to ggplot2. I completed the challenge that weekend, and went ahead to reproduce a few more plots I found from the Economist as practice exercise, including this one.

ggplot2 is fantastic. It is ergonomic, natural, and intuitive. It is all of the above because it is an implementation of the grammar of graphics, which aims to bring a set of grammar to the art of the visualization.

ggplot2 was great, and I duly put my newfound skills to use by combining it with the shiny framework to produce visualization-heavy web dashboards like this one. When I taught data visualization, I only wanted to teach the ggplot system because it felt methodical, logical and highly organized.

The teaching experience never quite translate equally to the python universe. It wasn't just because ggplot2 is highly declarative -- the user is concerned only with "what" she wants to achieve, not "how" she wants it achieved. It wasn't just because ggplot2 adheres to a coherent system, known as the grammar of graphics. It wasn't just because ggplot2 is highly instructive -- it picks default values in ways that are seemingly minor but when compounded over many decisions, can mean the difference between an abhorrent mess and a convincing medium of message.

Altair

I know many Python developers, myself included, begrudgingly switch between the different ecosystems to get the best of what R and Python offer. Then I stumbled upon Altair in 2018, when Altair was in its early days (last few releases before the v2.0; current version is 3.x). I created my first web app with Altair and it is a project that I still maintain as it is actively used among our teaching staffs at Algoritma Data Science Education Center, a company I co-founded 6 years ago.

Altair is Jake Vanderplas's attempt to provide a Python wrapper over Vega-Lite / Vega.js (which in turn, is built on D3.js).

It adheres to the same grammar of graphics system. It is declarative. It is extensible and customizable.

On top of that, it does something that ggplot2 library couldn't: it emits JavaScript code, which means your graphics can be embedded in just about any HTML file. It runs in the browser and can support browser events like onMouseOver, onMouseOut, onClick etc, so charts created with Vega have that added interactivity and event binding that R plots couldn't.

This year, when Anaconda announced PyScript, I started a playlist that I initially named build with PyScript in which I build one web app per video, showcasing the different possibilities that come with it. In many of the interactive web dashboards throughout the PyScript series I have relied on the trusty Altair library:
- Visualizing MiBand runs
- Building an interactive data visualization dashboard

These videos demonstrates the utility of PyScript along with Altair to create user-friendly front-ends that are truly interactive and fun-to-use. Viewers love them, but I started to get comments that request for a separate series that cast the light on Altair. One that more centrally focus on the building blocks of Altair as a data visualization library.

With this, I spent the last two months conceptualizing and developing a video series that goes into the art and science of data visualization in Python, adopted from the data visualization curriculum developed by the University of Washington. The series will consist of 5 to 8 long-form videos, each focusing on a key aspect of data visualization and the accompanying mental models that go with it.

Data Visualization in Python (2022)

Here's part 1 of the video:
Data Visualization in Python: Altair 4.2 (altair-viz) Tutorial

Teaching visualization with Altair has been highly satisfying, and the data scientists where I work at are in agreement of its versatility; documenting my journey here so there is a point of reference and book-keeping for my future self has been equally refreshing.

Now, readers, if you'd love sharing with me, how do you decide on your data visualization / charting toolkit and how does it look like?

Interactive Visualization of my Mi Band ⌚️ runs w/ PyScript

Samuel Chan — Fri, 15 Jul 2022 13:24:18 +0000

I’ve heard this fantastic quote from a podcast that goes something alone the lines of

Covid, it’s safe to say, will be here temporarily and then goes away completely. But some of the damages it causes, will stay with us probably permanently, certainly with far longer consequences, long after covid has moved on.

It could have been Balaji Srinivasan, it could have been somebody else.

But it was the wake up call I needed. Being in this line of work as the rest of you, I am quite used to being glued to my chair working away for up to 10 hours a day. Then the pandemic struck.

I transported my work machines from office to home second week into the pandemic. This is promptly followed by 14-hour work days, and a complete obliteration of my health. I put on weight, and my already pitiful fitness level tanked to virtually non-existent.

Hearing that line, the thought that covid would take a few years off my lifespan sounds completely logical, and even inevitable.

So I started running. And my best friend and co-founder at work bought me Mi Band, a fitness tracker. "A form of encouragement" is how he calls it.

I kept at it for more than a year, and this week I thought to export the data and visualize my runs. The goal is to develop a responsive, mobile friendly dashboard that summarizes my running progress, milestones and personal best(s).

I chose to do it in PyScript and deployed the web app using GitHub pages. Since I have a video series on building apps with PyScript, I naturally recorded the whole development process and wanted to share this with my audience.

The tutorial features data wrangling and cleaning process using the pandas library, before delegating the visualization to altair, which is a wrapper over the popular vega-lite (vega) charting library. As with all videos in the PyScript tutorial series, we deploy it through GitHub Pages.

Here's the live dashboard:
onlyphantom.github.io/miband/

Here's all the code and data:
https://github.com/onlyphantom/miband

Here's each video in the PyScript tutorial series:

Part 1 PyScript Demo / Tutorial
Part 2 Deploying a PyScript App w/GitHub
Part 3 Build a PyScript interactive dashboard w/Altair
Part 4 Build a PyScript guestbook webapp, CRUD
Part 5 Building an interactive map w/ Folium & PyScript
Part 6 Dashboard to visualize Mi Fit runs with PyScript + Altair (⌚ -🏃)

So if also use a fitness tracker / wristband, and don't mind picking up pandas or altair for some personal dashboard-ing, follow along the tutorial! It's extremely satisfying to have a longer-term perspective on your run performances and I hope it helps, even if just a little bit, in helping you stay in shape.

Part 6 Dashboard to visualize Mi Fit runs with PyScript + Altair (⌚ -🏃)

See you on YouTube!

Todo + gamification with Taskwarrior & Taskquant

Samuel Chan — Wed, 06 Apr 2022 13:59:21 +0000

If you live and breathe in the terminal, you want as little context switching as possible. For many things (like web browsing and spreadsheet editing) a graphical user interface seems inevitable no matter how great the alternative is (i.e lynx browser).

For everything else, there is almost always a better option, one that allows you to stay within the terminal and gasp optionally works with VIM bindings. As I review my current work and processes, it strikes me that a task manager (todo app) is one such use-case.

A good todo app is stunning, aesthetically pleasing, and joyous to use.

But a great todo app should be almost the entirely opposite of a good todo app. It shouldn't even be joyous to use, lest you spend too much time in it. I want a todo app that is so minimal, so unobtrusive that it's barely there. If I find the todo app enjoyable, I'm spending too much time in it instead of doing the task.

Taskwarrior

Taskwarrior is a free, open source TUI that puts your TODO in the terminal and is as unobtrusive as to-do apps get.

Taskwarrior video introduction

It has built-in reporting features (task burndown.weekly), a tagging system (+meeting +report), a context switcher (task context office), project hierarchies (task project:Startup), a robust date system that blows anything, free or proprietary, out of the water, and a scriptable urgency formula for each task.

There are many other real-life examples of algorithms that are used to approximate qualities: the Apgar Score that assesses the health of newborns, Google's Page Rank that measures the importance of a web link, or credit ratings. These examples all approximate, but cannot perfectly represent, a complex concept.

Taskwarrior uses a polynomial expression to calculate urgency. This is a simple sum of terms, where each term represents some measurable quality of a task.

Custom Attributes

The best part of Taskwarrior is its support for user-defined attributes (UDA) beyond what it offers out of the box. No matter what productivity system (e.g GTD system) I choose, I can make it work the way I want to with a bit of customization.

One theme of my productivity system over the past many years is a way to quantitatively measure my productivity with a score attribute. Different apps may come and go, but to make it work for me, I need the score attribute as it's integral to how I manage my periodic reviews and longer-term reflections. It also pushes me to work on large projects since they come with bigger rewards (at least psychologically).

How do I add that to Taskwarrior? Simple. Add the following lines to the end of your .taskrc file:

uda.score.type=numeric
uda.score.label=Score 🏆 
urgency.uda.score.coefficient=2

Now all your task has an extra, optional score field (🏆)!

Productivity Scoresheet

One thing I often do in my periodic review is to collect data-points on my activities over the last period. This includes data from my work machines, my Mi Band, and Apple Health.

For example, I know how many hours and minutes I've spent on one specific app, or task, over the last calendar year.

For Taskwarrior, all that is left to do is to roll out a Python app that I can install and run. This package weighs less than 8kb (which is really nothing) and is available on PyPI. It installs under one second:

pip install taskquant

After which you can issue tq to get a productivity breakdown. Add the -w flag to have it grouped by week, in a rolling sum fashion:

tq -w 
+-------+-------+------------+
| Week# | Score | Cumulative |
+-------+-------+------------+
|  10   |  56   |     56     |
|  11   |  33   |     89     |
|  12   |  26   |    115     |
|  13   |  12   |    127     |
+-------+-------+------------+

If you want to hack away at your productivity system, the project is available on my GitHub:

TaskQuant

Bonus: Watch the whole process in action

I thought it'd be interesting to show how I build out the CLI tool so I've uploaded a recording of that on YouTube as well. If you're a data analyst from the pandas world, you may learn a thing or two about:

group by
aggregation
filling missing dates (padding)
tabular summary
colorful CLI
All without any dependencies except tasklib (the Taskwarrior libary itself). This brings the package down to 6.8kb of code, and with its 1 second installation time, is similarly unobtrusive. Gets you your productivity scores, and get out of the way. If you'd like to watch it, here are the links:
Building Taskquant ep.1
Building Taskquant, full playlist

What's next

We'll add terminal-based graphs and charts, and maybe more variety in terms of reporting format. No, there can never be too many ways to see how you're not doing enough. If you want to follow along the journey, drop by and 👋 on YouTube.

If you're feeling more driven than that, go ahead and install Taskwarrior and Taskquant and start knocking out some to-do! Collaborators welcomed!