Hey there! I'm a software engineer who absolutely loves crafting web applications. Ruby is my go-to programming language, it's just my cup of tea! But hey, there's more to me than just programming.
That's actually in a slight contradiction with another common coding tip - "do NOT optimize early". Because then you can end up wasting time creating code you would never need.
I mean, if you can honestly expect your code will have to handle 100K+ values, then for sure you need to worry, but it is not always the case.
Hey there! I'm a software engineer who absolutely loves crafting web applications. Ruby is my go-to programming language, it's just my cup of tea! But hey, there's more to me than just programming.
You're right, and it's a fair tension. "Benchmark early" doesn't mean "optimize everything" — it means don't wait until production to discover your algorithm choice was wrong. The goal is awareness, not premature optimization.
The AI-naive version of our batch find_importers bit.ly/48yQaOX would be:
for fp in file_paths: # N target files
for src_file in all_source_files: # M source files
for imp in src_file.imports: # K imports each
if resolved == fp: ...
That's the same shape as the article's 50k-record dedup: O(N·M·K), "works in tests, dies on a 5k-file repo."
What the current code actually does (find_importers.py:106-129):
Pass 1 builds files_that_are_imported: set[str] — one scan, O(M·K).
Pass 2 builds import_map: dict[file_path → list[importer]] — another single scan, O(M·K).
The per-target loop is then import_map.get(file_path, []) — explicitly commented # O(1) lookup.
Total: O(M·K + N) instead of O(N·M·K). For a 500-file batch query against a 5k-file repo, that's the difference between one pass and 500 passes — the exact trap the article warns about, solved by the exact technique it recommends (dict/set membership for O(1) lookup instead of linear scans inside a loop).
Hey there! I'm a software engineer who absolutely loves crafting web applications. Ruby is my go-to programming language, it's just my cup of tea! But hey, there's more to me than just programming.
This is a great real-world example — thanks for sharing the actual code. O(M·K + N) vs O(N·M·K) is exactly the kind of thing that's invisible until it isn't. Fistbump right back 🤜
Engineering leader building AI systems and tools for real-world complexity. I write about scaling systems, team dynamics, and the gaps modern software still hasn’t solved.
https://calendly.com/kawacod
"We're getting used to not understanding the code we write" - the understatement of the century. Things are not looking bright at all for the future generations.
Hey there! I'm a software engineer who absolutely loves crafting web applications. Ruby is my go-to programming language, it's just my cup of tea! But hey, there's more to me than just programming.
Hey there! I'm a software engineer who absolutely loves crafting web applications. Ruby is my go-to programming language, it's just my cup of tea! But hey, there's more to me than just programming.
Fair point, thanks for the honest feedback. Those examples assumed some Ruby familiarity. I'll add more context to make them clearer for readers coming from other languages.
We're a place where coders share, stay up-to-date and grow their careers.
Nice tip: "Benchmark early"
😅 why not
That's actually in a slight contradiction with another common coding tip - "do NOT optimize early". Because then you can end up wasting time creating code you would never need.
I mean, if you can honestly expect your code will have to handle 100K+ values, then for sure you need to worry, but it is not always the case.
You're right, and it's a fair tension. "Benchmark early" doesn't mean "optimize everything" — it means don't wait until production to discover your algorithm choice was wrong. The goal is awareness, not premature optimization.
I agree!
The AI-naive version of our batch find_importers bit.ly/48yQaOX would be:
for fp in file_paths: # N target files
for src_file in all_source_files: # M source files
for imp in src_file.imports: # K imports each
if resolved == fp: ...
That's the same shape as the article's 50k-record dedup: O(N·M·K), "works in tests, dies on a 5k-file repo."
What the current code actually does (find_importers.py:106-129):
Pass 1 builds files_that_are_imported: set[str] — one scan, O(M·K).
Pass 2 builds import_map: dict[file_path → list[importer]] — another single scan, O(M·K).
The per-target loop is then import_map.get(file_path, []) — explicitly commented # O(1) lookup.
Total: O(M·K + N) instead of O(N·M·K). For a 500-file batch query against a 5k-file repo, that's the difference between one pass and 500 passes — the exact trap the article warns about, solved by the exact technique it recommends (dict/set membership for O(1) lookup instead of linear scans inside a loop).
===
TL;DR - Great minds. *Fistbump*...
-jjg
This is a great real-world example — thanks for sharing the actual code. O(M·K + N) vs O(N·M·K) is exactly the kind of thing that's invisible until it isn't. Fistbump right back 🤜
"We're getting used to not understanding the code we write" - the understatement of the century. Things are not looking bright at all for the future generations.
Couldn't agree more. The scary part is how quietly it normalizes. Nobody decides to stop understanding code — it just... happens.
Great article! Your first example is very clear but examples two and three are less so, especially for someone who doesn't use the same language.
Fair point, thanks for the honest feedback. Those examples assumed some Ruby familiarity. I'll add more context to make them clearer for readers coming from other languages.