DEV Community: sachin k

Smashing a hidden I/O bug: 100 disk reads per run for data that never changed

sachin k — Mon, 17 Aug 2026 16:48:54 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

awesome-tech-newsletter is an open-source project by Infrasity Labs that auto-discovers tech newsletters across the web. It runs a fleet of Python fetchers — for Substack, Hacker News, Medium, Product Hunt, Beehiiv, Hashnode, and more — that crawl each platform, classify what they find against a shared keyword config, and aggregate everything into a curated, categorized directory in the README.

Bug Fix or Performance Improvement

I picked this off the project's open issue tracker: issue #41, a performance bug in the Product Hunt fetcher.

The fetcher pulls 100 recent posts from the Product Hunt GraphQL API, then loops over them to classify each post against a list of keyword queries. The problem was this line, sitting inside the per-post loop:

for post_edge in posts:
    # ...build text_corpus...
    queries = get_search_queries(append_newsletter=False)  # <-- every iteration
    for query, cat in queries:
        ...

That looks like a harmless list rebuild — until you read what get_search_queries() actually does:

def get_search_queries(append_newsletter=True):
    config_path = os.path.join(..., "config.json")
    with open(config_path, "r", encoding="utf-8") as f:   # disk read
        categories = json.load(f)                          # JSON parse
    # ...build query list...

It opens and JSON-parses config.json from disk on every single call. So every fetch run was doing 100 file opens and 100 JSON parses for data that cannot change mid-loop.

The smoking gun that this was a bug and not a design choice: every other fetcher in the repo (hackernews.py, substack.py, medium.py, and six more) calls get_search_queries() exactly once, outside its loop. Only producthunt.py did it per-post.

Measured impact (benchmarked against the repo's real config.json — 9 categories, 86 generated queries — with 100 simulated posts):

Metric (per fetch run)	Before	After
`config.json` file opens	100	1
JSON parses	100	1
Classification loop time	5.14 ms	0.14 ms (~36× faster)

Honest framing: this is a scheduled crawler, so no user was staring at a spinner. But the cost scales with post count × config size, and redundant loop-invariant I/O is exactly the kind of silent tax that compounds as a project grows.

Code

PR: https://github.com/Infrasity-Labs/awesome-tech-newsletter/pull/43

The full change:

             posts = data.get('data', {}).get('posts', {}).get('edges', [])

+            # Load search queries once: get_search_queries() reads and parses
+            # config.json from disk on every call, and its result never changes
+            # within a fetch run, so calling it per-post did 100 redundant
+            # file reads + JSON parses.
+            queries = get_search_queries(append_newsletter=False)
+            seen_urls = set()
+
             for post_edge in posts:
                 node = post_edge.get('node', {})
@@
-                queries = get_search_queries(append_newsletter=False)
-                for query, cat in queries:
-                    if query in text_corpus:
+                for keyword, cat in queries:
+                    if keyword in text_corpus:
                         is_tech = True
                         category = cat
                         break
@@
-                    if not any(d['url'] == target_url for d in discovered):
+                    if target_url not in seen_urls:
+                        seen_urls.add(target_url)
                         logger.info("Discovered Product Hunt: %s", target_url)

My Improvements

Three changes, in decreasing order of importance:

1. Hoisted the loop-invariant call. get_search_queries() now runs once per fetch, right after the API response is parsed. This turns 100 disk reads + JSON parses into 1. I deliberately did not add caching inside get_search_queries() itself — that would change behavior for every fetcher and risk serving stale config in long-lived processes. Fixing it at the single bad call site keeps the change surgical and matches the pattern the other nine fetchers already use.

2. Fixed a variable-shadowing landmine. The inner loop used for query, cat in queries: — but query was already the name of the GraphQL query string defined earlier in the same function. It happened to be harmless today because the GraphQL string isn't reused after the request, but it's the kind of shadowing that turns a future "add retry logic" PR into a mystery bug. Renamed the loop variable to keyword.

3. Replaced an O(n²) dedup with a set. The duplicate-URL check was any(d['url'] == target_url for d in discovered) — a full list scan for every discovered post. A seen_urls set makes it O(1) per lookup. Minor at 100 posts, but free to fix while I was in the function.

Verification: the module compiles clean (python3 -m py_compile), imports, and runs correctly through its no-token code path. The benchmark hooked builtins.open to count file accesses, confirming the 100 → 1 drop, and used timeit over 20 repeats for the timing numbers.

Transparency note: I found, benchmarked, and fixed this bug working alongside Claude (Anthropic's AI assistant) — it ran the code analysis and benchmarking in a sandboxed environment while I directed the hunt, reviewed the changes, and submitted the PR.

Cybersecurity from Zero to Hero #1: The CIA Triad, or Why Your Bank App Keeps Logging You Out

sachin k — Mon, 17 Aug 2026 14:14:50 +0000

Welcome to post one of my new series, Cybersecurity from Zero to Hero. I am learning security from absolute scratch and writing about it as I go, so you get the explanations while the confusion is still fresh in my head. The plan is simple: concepts first, then networking, then Linux and web security, then real tools and practice platforms. No prior knowledge needed. If you are experienced, stick around anyway, because the comment sections are where you get to correct me and teach everyone else.

One rule before we start, and it applies to this entire series. Every hands on exercise happens only on systems I own or on platforms built for legal practice like TryHackMe and OWASP Juice Shop. Learning security responsibly is part of learning security.

Today we lay the foundation stone of all security thinking.

The one framework everything else builds on

Ask any security professional where to start and you will hear three letters: CIA. Not the agency. It stands for Confidentiality, Integrity and Availability, and it is the lens through which every security decision is made. Firewalls, passwords, backups, encryption, every tool you will ever meet in this field exists to protect one or more of these three properties.

Here is what surprised me on day one of studying this. Security is not about stopping hackers. It is about protecting these three promises, and hackers are only one of many things that threaten them. A flood in a data center is a security event. So is an intern deleting the wrong folder.

🔒 Confidentiality: only the right people can see it

Confidentiality means information is visible only to those authorized to see it. Your medical records, your salary, your private messages.

Real world example. When your banking app logs you out after five minutes of inactivity, that is confidentiality at work. The bank assumes you might have walked away from your phone in a coffee shop. The aggressive timeout is not bad design. It is a deliberate tradeoff that sacrifices your convenience to protect your account from the stranger at the next table.

Tools that protect it: encryption, passwords, multi factor authentication, access controls, and even the privacy screen on a laptop.

The classic violation: a data breach where customer records leak to the public.

✅ Integrity: the data has not been tampered with

Integrity means information stays accurate and unaltered except by authorized changes. It is the promise that what you read is what was written.

Real world example. When you transfer 100 dollars, integrity is the guarantee that it does not become 1000 dollars in transit, and that the recipient account number is not silently swapped. Banks obsess over this. An attacker who can change data is often more dangerous than one who can only read it.

I also ran my first integrity check today, and you can too. When you download software, sites often publish a checksum next to the file. On Mac or Linux:

bash shasum -a 256 downloaded_file.zip
On Windows PowerShell:

powershell Get-FileHash downloaded_file.zip

Compare the output to the published value. If even one character differs, the file was corrupted or tampered with somewhere between their server and your machine. That comparison is integrity verification in its purest form, and later in this series we will build our own hashing tool in Python.

Tools that protect it: hashing, checksums, digital signatures, version control, audit logs.

The classic violation: an attacker altering a student database to change grades.

⚡ Availability: the system works when you need it

Availability means authorized users can access the data and systems when they need them. Security that makes a system unusable has failed at its job.

Real world example. When a Distributed Denial of Service attack floods a website with junk traffic until it collapses, nothing was stolen and nothing was altered. Confidentiality and integrity are intact. But the service is down, so it is absolutely a security incident. Backups, failover systems and disaster recovery plans all exist to protect this leg of the triangle.

Tools that protect it: redundancy, backups, load balancing, DDoS protection, disaster recovery plans.

The classic violation: ransomware locking a hospital out of its own patient records.

🔺 The part that made it click for me: the triangle is a tension

Here is the insight most beginner summaries skip. The three properties pull against each other, and real security work is about balancing them, not maximizing all three.

Make a system maximally confidential with twelve authentication steps and you have destroyed availability. Nobody can get in, including the people who should. Make it maximally available with no logins at all and you have destroyed confidentiality. That annoying bank timeout is not a bug or lazy engineering. It is a chosen point on the triangle. Once I saw it this way, a dozen daily annoyances suddenly made sense as deliberate design decisions.

A question for the experienced folks. When you review a system, do you consciously walk the triad, or has it become instinct? And what is your favorite example of a control that helps one leg while hurting another? Drop it in the comments and I will feature the best answers in a future post.

🤔 What confused me today

Where does authenticity fit? Some sources add authenticity and non repudiation as extra properties, and some textbooks talk about a hexad instead of a triad. For now I am parking that question. The triad is the standard mental model and the extensions can wait until the fundamentals are solid.

Is a power outage really a security issue? My instinct said no, that is an IT problem. But under the triad, anything that threatens availability is in scope for security. The field is wider than the hacker movies suggest.

⏭️ Next in this series

Post 2 covers threats, vulnerabilities and risk, three words people use interchangeably that mean very different things. Once you can tell them apart, every security article you read becomes twice as clear. Follow the series so you do not miss it.