DEV Community: Sourav Bhowmik

From Full-Table Scans to Millisecond Search in Go: Optimizing Prefix Search for 13.5 Million Places

Sourav Bhowmik — Wed, 28 Oct 2026 23:48:26 +0000

I built a minimal backend that searches a huge dataset and returns the 10 most popular matches. I wanted to find the bottlenecks in a simple search implementation and measure how each optimization changed performance. I started with the simplest version and improved it step by step, using the same data and query prefixes throughout. At the end of the final version, the query time for a single keyword a drops from 2.6 seconds to 2.9 milliseconds.

It is written in Go, with the standard library only. There is one search endpoint and one HTML page, both served by the same Go program. The code, the results and the git tags for each level are in the repo: https://github.com/bhowmik94/autocomplete-10M.

A query for the one-letter prefix a over 13.5 million places took 2.6 seconds in my first version. In the final version it takes 2.9 milliseconds. A longer prefix like dhaka went from 2.4 seconds to under 2 microseconds.

Disclaimer: These benchmarks measure the search function, not a deployed API.

Dataset and method

Data: I used the allCountries file from GeoNames, which has 13,472,243 rows. Each row is a place: a city, a village, a district and so on. The file has 19 columns, and I use only four: ID, name, country code and population.

I developed everything on cities15000 first, a smaller file with about 34,000 rows, and moved to allCountries once the naive version worked. I chose GeoNames over other free datasets, like Wikipedia article titles, because it has a population column, which gives a clear way to rank the results. The data is licensed under CC BY 4.0 and comes from GeoNames.

Ranking: Matches are sorted by population, highest first. If two places have the same population, the higher ID wins. Matching is case-insensitive and ignores leading spaces. I kept the ranking basic on purpose, because it was not the primary goal of this study.

Machine: AMD Ryzen 5 5500U, Windows, 12 threads, Go 1.26.4.

Benchmark prefixes: I measured four prefixes that behave very differently. The block size is the number of places whose name starts with the prefix. I estimated it from the memory the Level 1 version allocates, so treat it as approximate.

Prefix	Block size	Why it is in the set
`a`	about 590,000	Huge block
`dha`	about 12,000	Medium block
`dhaka`	about 200	Small block
`xyzq`	0	No match

Keeping it fair: Every level runs on the same data, the same prefixes and k = 10. I ran the same benchmark cases against every implementation. A separate test compares each version with the naive one for every prefix in a small dataset. The naive version is slow but simple, so I used it as the reference for correct answers. Before building each level, I wrote down what I expected to measure and compared it afterwards.

Level 0: the naive version

For every query, the naive version goes through all rows, lowercases each name, checks whether it starts with the prefix, collects the matches, sorts them by population and returns the top 10.

On the 34,000-row file, a query for dha took about 3.5 ms. On the full file it took about 1.4 seconds. That is roughly 105 ns per row in both cases, so the cost grows in line with the number of rows. A prefix with no match (xyzq) took as long as dha, which shows that the cost is the scan itself, not the matching. The one-letter prefix a was the slowest, because about 590,000 places match it and have to be collected and sorted.

Prefix	Time per query	Memory per query	Allocations per query
`a`	1.76 s	414 MB	13.47 million
`dha`	1.41 s	244 MB	13.47 million
`dhaka`	1.47 s	242 MB	13.47 million
`xyzq`	1.45 s	242 MB	13.47 million

The allocation count is the useful clue. It is one allocation per row, which is the lowercase copy of every name that is created on every query. That is why even a prefix with no matches allocates 242 MB. In the naive version, memory depends on the size of the dataset, not on the size of the matching block.

Loading the full file took about 9.8 seconds and the heap was 993.5 MB afterwards. Getting there needed one memory fix. Go’s strings.Split returns pieces that point into the original text, so keeping a single city name keeps the whole file text alive. On the small file the heap after loading was 10.2 MB. Copying each name with strings.Clone brought it down to 2.5 MB with the same load time. On the full file the unused text would have been over a gigabyte.

Level 1: sort once, binary search

The naive version repeats the same work on every query. Level 1 does that work once, at startup:

Each place gets a LowerName field, computed once when the data is loaded.
The whole list is sorted once by that field.

After sorting, all places whose names start with the same prefix sit next to each other in one unbroken stretch of the array. I call this stretch the block.

Two binary searches find the block: one for its start, one for its end.
To find the end, I search for the prefix with its last character incremented—for example, dha becomes dhb. Every dha match sorts before that boundary. Then I copy the block, sort the copy by population and return the first 10. The copy is important since the block is a window into the shared array, and sorting it in place would reorder the index for every later query.

The speedup has a price. The extra lowercase names raised the heap from 993.5 MB to 1,403.6 MB. The one-time sort takes about 10 seconds on top of about 10 seconds of parsing, so startup is now about 20 seconds.

Prefix	Naive	Level 1	Level 1 memory per query	Speedup
`a`	2.64 s	144 ms	37.7 MB	about 18x
`dha`	2.34 s	2.1 ms	771 KB	about 1,100x
`dhaka`	2.39 s	19 µs	14 KB	about 124,000x
`xyzq`	2.33 s	572 ns	32 B	about 4 million x

A note on the naive column: after the data was sorted, the naive version ran slower than in my first measurement (2.3 to 2.6 seconds instead of 1.4 to 1.8), although its code had not changed. I suspect cache misses: the sorted records still point to names laid out in file order. I haven’t tested that explanation. Every speedup in this article compares numbers from the same run.

Two things stand out. A no-match query takes 572 ns: it only needs two binary searches. For matching queries, time depends on the block size. a matches about 590,000 places and takes 144 ms; dha matches about 12,000 and takes 2.1 ms. The number of allocations is constant (6), but the bytes still grow with the block, because the block is copied and sorted.

Level 2: do not sort what you will not use

Level 1 copies and sorts 590,000 places to return 10 of them. Level 2 walks the block once and keeps a list of the best 10 places seen so far, sorted best first. Most places are worse than the current tenth-best result, so I skip them. If a place is better, I move it into the right position. Otherwise it replaces the worst one and moves up to its position. There is no copy and no sort. For k = 10, a small sorted slice is enough. I did not try a heap, which should only help for a large k.

Prefix	Level 1	Level 2	Level 2 memory per query	Speedup over Level 1
`a`	144 ms	2.9 ms	708 B	about 49x
`dha`	2.13 ms	47 µs	712 B	about 45x
`dhaka`	19.3 µs	1.9 µs	712 B	about 10x
`xyzq`	572 ns	454 ns	8 B	about 1.3x

Memory per query is now about 700 bytes (the ten places), no matter how big the block is. The no-match prefix barely changed, since it was already just two binary searches.

The remaining 2.9 ms for a looks like a memory limit. The block contains about 590,000 records of 64 bytes each, or roughly 38 MB. Scanning it in 2.9 ms works out to about 13 GB/s—close to the bandwidth I’d expect from a laptop. The loop only needs the population and the ID of each place, but it pulls in all 64 bytes. So the limit is now the memory layout, not the algorithm.

Across both steps:

Prefix	Naive	Level 1	Level 2	Total speedup
`a`	2.64 s	144 ms	2.9 ms	about 900x
`dha`	2.34 s	2.1 ms	47 µs	about 50,000x
`dhaka`	2.39 s	19 µs	1.9 µs	about 1.3 million x
`xyzq`	2.33 s	572 ns	454 ns	about 5 million x

Serving it over HTTP

There is one search endpoint, for example, GET /suggest?q=dha&limit=5. The default limit is 10, and values outside 1–50 are clamped. Non-integer values return a 400 response. The search function is a field of the server struct, so any of the three levels can run behind the same endpoint.

Go handles each request in its own goroutine, so many requests read the index at the same time. That is safe because nothing modifies the index after startup: each query only reads it and builds its own 10-place result. If a query had sorted the shared array in place, requests would have corrupted each other. A test with 50 concurrent requests, using Go’s httptest package, runs against the handler.

The HTML page is a single static file served by the same Go program at /. Serving it from the same place as the API avoids CORS issues and keeps the project small.

Limitations

No end-to-end load test. The timings above call the Go functions directly. They leave out the network, HTTP parsing and JSON encoding, so a real request takes longer. Everything ran on one laptop.
Estimates. The block sizes are estimated from allocated memory, and the slowdown of the naive version on sorted data is an untested guess. The naive version was measured with five iterations per prefix, because each run takes seconds.
Basic search quality. Only the name is matched, with no accent folding, so munchen does not find München. Alternate names are not indexed, places with the same name are listed separately, and ranking uses population only.
Startup and memory. Startup takes about 20 seconds and the index needs about 1.4 GB of RAM per instance.

What could come next

Store populations and IDs in separate compact arrays, so the scan for a reads a few bytes per place instead of 64.
Use a range-maximum structure (a sparse table or a segment tree) with a heap, so the cost depends on k and not on the block size.
Precompute the top 10 for the shortest prefixes.
Compare against Postgres with an index and a cache, and deploy it to the cloud.

What I learned

Write predictions down before measuring. It shows you what you did not understand yet.
Allocation counts show where time goes. 13.47 million allocations per query pointed straight at the lowercasing of every name.
Cost can depend on the part of the data a query touches, not on the dataset size. After Level 1, a query over 13.5 million places that matches nothing takes 572 ns.
Memory layout can matter as much as the algorithm. Level 2 runs at roughly the speed of memory, and the naive version may have got slower just because the data was reordered.
Keep the slow, simple version. The naive version was my reference for correct answers in every test.

Code, results and git tags: https://github.com/bhowmik94/autocomplete-10M. Data: GeoNames, CC BY 4.0.

How I Cut a React App's Initial Bundle by 89%

Sourav Bhowmik — Wed, 30 Sep 2026 17:40:53 +0000

Building and shipping applications is an integral part of a software developer's life. In this article I show you how I ran an experiment on the initial bundle size of a frontend web app build, specifically, which factors are responsible for bundle bloat, and how much of it can actually be reduced. My baseline build shipped a 634 KB (gzip) initial bundle. After running four optimization experiments, it dropped to 71 KB, a staggering 89% reduction!

Setup

I built a minimal activity-dashboard app in React, with help from Claude AI. The baseline version deliberately includes a few heavy dependencies: react-icons, lodash, recharts, and moment; each picked for a reason that seemed sensible at the time, not out of carelessness.

moment, because it's one of the most popular date libraries in JS and handling relative timestamps correctly (pluralization, day boundaries) felt like something worth not hand-rolling.
lodash, because I needed debounce for the search input and groupBy to cluster activities by date; pulling in a well-known utility library felt like the obvious choice. (There is a surprise bug for the lodash imports! Continue reading to find out 😉)
recharts, because the Dashboard page just needed one simple line chart, and it didn't seem worth reaching for something heavier like Chart.js.

The app has three pages: Feed (the landing page, showing dummy activity items), Dashboard (a line chart of weekly activity via recharts), and Settings (empty, included only to demonstrate route-based code splitting).

For measuring bundle composition, I used rollup-plugin-visualizer:

// vite.config.js
import { visualizer } from 'rollup-plugin-visualizer'

export default defineConfig({
  plugins: [
    react(),
    visualizer({ filename: 'dist/stats.html', gzipSize: true }),
  ],
})

This generates a stats.html treemap after every build, showing exactly which libraries and functions are taking up space.

Experiments

I ran four experiments from the baseline, each isolated on its own branch, to measure the individual impact of a single fix on initial bundle size. A final branch combines all four.

1. Fixing the icon import

This was the single biggest win. The baseline imported the entire icon set:

import * as Icons from 'react-icons/fa'

This looked necessary because the icon to render is resolved at runtime from a category field in the data — so it seemed like there was no way to know in advance which icons I'd need. But the category set is actually small and known ahead of time, so named imports work fine:

import { FaComment, FaUpload, FaAt, FaHeart, FaUserPlus } from 'react-icons/fa'

const CATEGORY_TO_ICON = {
  comment: FaComment,
  upload: FaUpload,
  mention: FaAt,
  like: FaHeart,
  join: FaUserPlus,
}

This single change cut the initial bundle by 67% — from 634 KB to 209 KB (gzip).

2. Lazy-loading the Dashboard route

The Dashboard page's line chart (recharts) was being loaded on every page visit, even for users who never opened the Dashboard. Wrapping the route in React.lazy + Suspense defers that cost until someone actually navigates there:

// Before
import Dashboard from './pages/Dashboard'

// After
const Dashboard = lazy(() => import('./pages/Dashboard'))

<Suspense fallback={<p>Loading...</p>}>
  <Routes>
    <Route path="/dashboard" element={<Dashboard />} />
    {/* ...other routes */}
  </Routes>
</Suspense>

This reduced the initial bundle by 17%, from 634 KB to 529 KB (gzip).

Worth calling out: this fix doesn't make recharts disappear; it rather moves it into a separate chunk that loads on demand. So while the initial download shrinks by 17%, the total JS the app ships (initial + Dashboard chunk combined) barely changes. That distinction, initial load vs. total weight, turned out to matter a lot for how I read all four results, and especially the combined one at the end.

3. Fixing the lodash import

The baseline imported lodash like this:

import { debounce, groupBy } from 'lodash'

This looks like a named import, but it isn't one; at least not in a way that helps. Lodash's main entry point is a CommonJS module, so the bundler can't statically determine which functions are actually used and ends up including the whole library anyway. It's effectively the same as:

import _ from 'lodash'

The fix is to import each function from its own subpath:

import debounce from 'lodash/debounce'
import groupBy from 'lodash/groupby'

This properly tree-shakes and cuts the bundle by 5%; a small number, but the mistake itself is the more useful finding: it's easy to write an import that looks tree-shaken and isn't.

N.B. There is a subtle and easy-to-miss bug in the second lodash import lodash/groupby. Named imports are case-sensitive, so the correct import path should be:

import groupBy from 'lodash/groupBy'

Then how did this work on my machine? Because Windows and MacOS filesystems are not case-sensitive. So the import path worked on my Windows environment. But this would raise an error in a Linux environment.

Switching to lodash-es with named imports (import { debounce, groupBy } from 'lodash-es') will completely remove this bug and make things even easier. And since lodash-es is a proper ES Module, tree shaking is embedded by default.

4. Replacing moment with date-fns

moment is a large, feature-complete library, and reaching for it as a first instinct for date handling is common since it's intuitive and well-documented. But most of that surface area goes unused in any single app, and unlike modern alternatives, moment isn't built to tree-shake. Swapping it for date-fns, which ships as individual function modules:

// Before
moment(item.timestamp).fromNow()

// After
formatDistanceToNow(item.timestamp, { addSuffix: true })

This produced the smallest change of the four — a 2% reduction. But in an app with heavier date usage than this one, that gap would likely be much larger; this app only touches three moment methods, so there wasn't much bulk to remove in the first place.

Combined result

Applying all four fixes together dropped the initial bundle from 634 KB to 71 KB (gzip) — an 89% reduction.

	Before	After	Change
Initial download (what a first-time visitor gets)	634 KB	71 KB	-89%
Total JS shipped (initial + Dashboard chunk)	634 KB	168 KB	-74%

The gap between those two rows is the point of lazy-loading: total app weight didn't shrink as much as the initial load did, because the chart code still exists; it just isn't downloaded until someone actually needs it.

Takeaway

The goal here was to find out which kinds of bundle bloat actually matter, not just to list generic advice. Of the four fixes, the wildcard icon import alone accounted for a 67% reduction — more than the other three combined. If you're short on time, auditing wildcard/dynamic imports is probably the highest-leverage place to start.

It's also worth knowing that named imports aren't automatically tree-shaken. Instead, it depends on how the library itself is structured, and it's easy to get this wrong without realizing it, as the lodash example shows.

The full project, including each experiment on its own branch and the combined final version, is on GitHub: [https://github.com/bhowmik94/bundle-experiment]. The experiments/ folder has the bundle visualizer output for each build, showing the exact size breakdown per library and function, if you want to dig deeper. You are welcome to reproduce the same results and see for yourself. Just fork the project and have fun! Please feel free to leave any questions or comments below for further discussions and improvements. And if you are feeling generous, consider leaving a heart in this article.