DEV Community: Mayank Srivastava

Why GitHub Cannot Paginate Through Millions of Repositories - Deep Pagination Explained

Mayank Srivastava — Mon, 03 Aug 2026 20:32:03 +0000

You've paginated through a big result set before. Page 2 loads fine. Page 200 takes a beat. Nobody stops to ask why — until something like this shows up instead of a slow page:

{
  "message": "Only the first 1000 search results are available.",
  "documentation_url": "https://docs.github.com/v3/search/",
  "status": "422"
}

That's GitHub's own search API. Ask it for page 9 or page 10 of a search and you get real results. Ask for page 11 and you get that, on purpose, every time.

I went and looked at why. Full video below, written breakdown after it for anyone who'd rather read.

Reproducing it

Not a guess — this is live, right now:

curl "https://api.github.com/search/repositories?q=javascript&page=9&per_page=100"
# -> real results

curl "https://api.github.com/search/repositories?q=javascript&page=10&per_page=100"
# -> real results

curl "https://api.github.com/search/repositories?q=javascript&page=11&per_page=100"
# -> 422, the message above

GitHub documents this cap directly — it's not rate limiting, and it's not a bug that slipped through. Once you understand the mechanism it's protecting against, the same wall shows up in a lot of places that don't bother telling you about it — including, probably, a table you've built yourself.

The query behind the page number

Most pagination UIs turn a page number into something like this:

SELECT * FROM videos
ORDER BY id
LIMIT 20 OFFSET 999980;

Twenty rows per page, page 50,000 requested — that's an offset of nearly a million. It's easy to assume the database can jump straight to row 999,980. It can't, and the reason is in how the underlying index is built, not in anything about this specific query.

What an index actually is

A sorted column is backed by a B-tree — a structure built for one thing: walking rows in order, fast. The leaf level is a chain of rows in sorted sequence. That chain is what makes ORDER BY cheap.

It is also the whole problem. There's no operation in a B-tree for "give me whatever's at position 999,980." Only for "give me whatever's next after this value." Position and value are not the same axis, and only one of them has a shortcut.

So OFFSET 999980 does the only thing it can: start at the beginning of the scan, count a row, discard it, count the next, discard it — one at a time, nearly a million times — before it starts collecting the 20 rows you actually asked for.

Cost = O(offset), not O(page size). Page 10 is nearly free. Page 50,000 means walking and throwing away roughly a million rows just to reach the part you wanted.

What actually happens, measured

Theory's cheap. Here's a real table — 5 million rows, local Postgres:

docker run --name pagination-demo -e POSTGRES_PASSWORD=demo -p 5432:5432 -d postgres:16
docker exec -it pagination-demo psql -U postgres

CREATE TABLE videos (
  id BIGINT PRIMARY KEY,
  title TEXT,
  created_at TIMESTAMP
);

INSERT INTO videos (id, title, created_at)
SELECT g, 'Video ' || g, NOW() - (random() * interval '3 years')
FROM generate_series(1, 5000000) AS g;

EXPLAIN ANALYZE
SELECT * FROM videos
ORDER BY id
LIMIT 20 OFFSET 999980;

Limit  (cost=33323.00..33323.67 rows=20 width=29) (actual time=121.583..121.586 rows=20 loops=1)
  ->  Index Scan using videos_pkey on videos  (cost=0.43..166616.30 rows=4999991 width=29)
        (actual time=0.012..94.512 rows=1000000 loops=1)
Planning Time: 0.201 ms
Execution Time: 121.603 ms

rows=1000000 on the scan node isn't an estimate — that's the actual count the engine walked through before it could return anything, exactly matching the mechanism above.

The fix everyone reaches for, and what it actually does

The usual advice is "use cursor pagination instead of OFFSET." Here's the query that advice produces:

EXPLAIN ANALYZE
SELECT * FROM videos
WHERE id < 845923
ORDER BY id DESC
LIMIT 20;

Limit  (cost=0.43..1.15 rows=20 width=29) (actual time=0.477..0.484 rows=20 loops=1)
  ->  Index Scan Backward using videos_pkey on videos  (cost=0.43..30516.09 rows=851752 width=29)
        (actual time=0.475..0.480 rows=20 loops=1)
        Index Cond: (id < 845923)
Planning Time: 0.129 ms
Execution Time: 0.503 ms

Same table. Same 20 rows back. 121.603 ms → 0.503 ms — roughly 242x, and it's not a rounding error, it's a different cost curve entirely.

What changed underneath: instead of walking the leaf chain from the start, the engine descends the tree directly — root, branch, leaf — to id = 845923, then reads 20 rows backward from there. A seek, not a scan. That's the entire mechanism. Nothing about the table changed, nothing about the index changed — only the shape of the question changed, from "what's at position N" to "what comes after this value."

This is also why Instagram says "load more" and X says "load older posts" instead of showing page numbers. Under the hood, that's a cursor — after=<id> — not a position.

Where this breaks silently

Keyset pagination only holds up if the sort key is unique and strictly ordered — a primary key like id qualifies. The moment you sort by something that can repeat, like created_at, where two rows can land on the exact same millisecond, a single-column cursor can silently skip a row or hand back a duplicate. Nothing errors. It just quietly returns the wrong set.

The fix is comparing a pair of columns, not one:

SELECT * FROM videos
WHERE (created_at, id) < (:last_created_at, :last_id)
ORDER BY created_at DESC, id DESC
LIMIT 20;

That line is the gap between "works in the demo" and "works in production."

What to check before you reach for either

In order:

Does the page number itself need to exist? If users are scrolling a feed, not jumping to "page 4,721," you don't need OFFSET's random access at all — cursor pagination is strictly better here, no tradeoff to weigh.
Do you actually need random access? Admin dashboards, audit tools, support panels — anywhere someone legitimately needs "go to page 42" — OFFSET's random access is the feature, not the bug, and the fix is usually capping how deep it's allowed to go rather than replacing it outright.
If you're building a public API, assume someone will request the deepest possible page. Elasticsearch refuses to paginate past 10,000 results by default and points you at search_after instead — the same fix GitHub ships, in a different system. Capping the depth is cheaper than absorbing the cost of someone finding it by accident.

Not a universal fix

OFFSET isn't wrong. It's the right tool when random access matters more than depth — small tables, internal tools, anywhere nobody's going past page 20 anyway. Cursor pagination isn't strictly "better," it trades away the ability to jump anywhere for a flat cost curve at any depth. The mistake isn't picking one — it's not knowing there's a choice, and finding out which one you picked at the exact moment your table crosses a few million rows.

If this was useful, the video walks through the same benchmark live — GitHub's actual API response, the terminal running both queries against the same 5M-row table, EXPLAIN ANALYZE output as it prints. Drop a comment if you want the follow-up on why COUNT(*) gets slow on the exact same kind of table — different query, same root cause.

The npm `overrides` Snippet Everyone Copy-Pastes — And What It Actually Does

Mayank Srivastava — Mon, 03 Aug 2026 20:13:35 +0000

You've seen this error. npm install fails with ERESOLVE, you search it, find a GitHub issue, copy an overrides snippet into package.json, and the red text disappears on your next run.

Nobody asks what that block actually did.

I hit this exact error yesterday setting up a cloned repo, and instead of moving on once the install went green, I went and looked. Full video below, written breakdown after it for anyone who'd rather read.

The error

npm error code ERESOLVE
npm error ERESOLVE unable to resolve dependency tree
npm error
npm error While resolving: forced-error-demo@1.0.0
npm error Found: react@19.2.8
npm error node_modules/react
npm error   react@"^19.0.0" from the root project
npm error
npm error Could not resolve dependency:
npm error peer react@"^16.9.0 || ^17.0.0" from @testing-library/react-hooks@8.0.1

Two lines in here matter, and they're not making the same kind of claim.

Found: react@19.2.8 — this is just what's already resolved, because it's what my own package.json asked for (^19.0.0).

peer react@"^16.9.0 || ^17.0.0" from @testing-library/react-hooks@8.0.1 — this is a different thing entirely: a peer dependency.

What a peer dependency actually is

Straight from @testing-library/react-hooks's own package.json:

{
  "name": "@testing-library/react-hooks",
  "version": "8.0.1",
  "peerDependencies": {
    "react": "^16.9.0 || ^17.0.0"
  }
}

A peerDependencies entry is a dependency the package needs, but expects the consumer to provide — the package isn't bundling its own copy of React, it's reaching into whatever React is already running in your project, and declaring the only versions it was built to reach into safely.

Compare that to a normal dependencies entry, which npm installs as its own separate copy in node_modules. A peer dependency installs nothing. It just checks.

npm has enforced that check by default since v7. Before that, npm 3 through 6 mostly let mismatches like this through silently — which is exactly why the --legacy-peer-deps flag is named what it is: it tells npm to go back to that old, unchecked behavior.

So this error isn't npm being broken. It's npm accurately reporting that a package in your tree is making a claim about your React version that isn't true.

The fix everyone copy-pastes

Here's the snippet you'll actually find on GitHub issues for this exact situation:

"overrides": {
  "@testing-library/react-hooks": {
    "react": "$react"
  }
}

The $react syntax means "whatever version I already have installed as a direct dependency, use that." So this line says: wherever @testing-library/react-hooks declares a requirement on react — including its peer requirement — force it to match root's react@19.0.0 instead.

Run npm install again. It succeeds. No red text. Looks fixed.

What actually changed (and what didn't)

overrides didn't touch a single line of code inside @testing-library/react-hooks. It didn't check whether that package's internals can actually run against React 19. It went into the resolved dependency graph and rewrote the version number the peer check compares against — from ^16.9.0 || ^17.0.0 to 19.0.0 — so the check has nothing left to disagree with.

That's the entire mechanism. overrides doesn't resolve incompatibility. It rewrites the number two packages are being compared on, whether or not the code underneath agrees.

What did not happen:

No code inside the overridden package ran
No check that its React-16/17-era internals still work against React 19's scheduling model
No test suite executed to confirm any of it

Why the gap matters

@testing-library/react-hooks was built before hook-testing support existed in @testing-library/react itself, and it's effectively frozen at how React 16 and 17 scheduled renders. React 19's internals have moved since then. The override doesn't close that gap — it just moves when you find out about it.

Without the override: you read two lines of log, thirty seconds, and you're acting on real information.

With the override: install succeeds clean, ships to CI, ships to prod — and the actual discovery happens whenever a hook test misbehaves or a render warning shows up with no line in your own code pointing back to it. Same incompatibility, if it's real. Just a much more expensive place to find it.

What to check before you reach for `overrides`

In order:

Does the conflicting package still need to exist? In this case — no. @testing-library/react has supported hook testing natively for a long time now, which makes @testing-library/react-hooks a dependency that's outlived its reason for being. Deleting it is a better fix than patching around it.
Can it be upgraded instead? A newer major of the conflicting package may not have this peer conflict at all. You might have to touch your own code too, since not every upgrade is backward-compatible — but that's still cheaper than a silent runtime bug.
If it's genuinely unavoidable, scope the override. Don't apply it globally:

// unscoped — forces every package in the tree that touches react
"overrides": {
  "react": "$react"
}

// scoped — forces it only where the actual conflict is
"overrides": {
  "@testing-library/react-hooks": {
    "react": "$react"
  }
}

Everything else in your tree keeps the version npm originally resolved. Smaller blast radius, easier to reason about, easier to remove later.

A temporary patch. Not permanent config.

If you do add an override, treat it like debt, not like settings. Leave a note on why it's there — package.json doesn't support comments, so put it in a README or an OVERRIDES.md: what it's patching, when to revisit it, who added it. Put it on your dependency-audit schedule, whatever that already is for your team, and pull it out the moment the upstream package makes it unnecessary.

An override with no expiry date doesn't leave. It just becomes the next person's mystery to reverse-engineer.

If this was useful, the video covers the same ground with the terminal running live — installing with and without the override, reading the actual npm ls output, watching what changes. Drop a comment if you want the follow-up on npm's resolution algorithm itself — why these conflicts happen before any override enters the picture.

How I Beat a 16-Click Anti-Bot Ad Trap

Mayank Srivastava — Wed, 20 May 2026 16:50:04 +0000

🎯 I Got Tired of Closing Popups. So I Reverse-Engineered the Entire Ad Stack.

We’ve all been there.
You open a streaming site.

You click play.

💥 A new tab appears.

You close it. Click again.

💥 Another popup.

Some sites are so aggressive that watching a single episode feels like defusing a bomb through 16 consecutive popups while trying not to accidentally download malware from 2007.

Most people tolerate it.
I couldn’t.

As a developer, the moment software starts fighting the user this aggressively, it stops being an inconvenience and becomes a challenge.
So I decided to break it.

🧪 Phase 1 — The “This Should Work” Delusion

My first thought was simple:
Override window.open.

window.open = () => null;

Done. Right?
Nope.
The site completely ignored it.
That’s when I realized this wasn’t normal popup logic anymore.
This was engineered hostility.

👻 Phase 2 — Chasing Invisible Click Traps

I opened DevTools and started tracing the DOM during every interaction.
Something strange kept happening.
Inside the video player container, the final

kept rapidly changing after every click.
Then it clicked.
The site was dynamically generating transparent overlay layers directly above the player.
The actual video button wasn’t receiving my clicks.
The invisible overlay was.
Every physical mouse interaction got hijacked by a temporary fullscreen layer designed to trigger an ad redirect using a trusted human click.
The sequence looked something like this:

┌───────────────────────────────┐
│ 🖱️ User Click                │
└──────────────┬────────────────┘
               │
               ▼
┌───────────────────────────────┐
│ 👻 Invisible Overlay Captures│
│    the Trusted Click Event   │
└──────────────┬────────────────┘
               │
               ▼
┌───────────────────────────────┐
│ 💥 Popup / Redirect Triggered│
└──────────────┬────────────────┘
               │
               ▼
┌───────────────────────────────┐
│ 🧹 Overlay Self-Deletes      │
└──────────────┬────────────────┘
               │
               ▼
┌───────────────────────────────┐
│ ♻️ New Overlay Injected      │
└──────────────┬────────────────┘
               │
               └─────── loops ───────▶

Ridiculously clever.
Because technically I initiated the click, the browser trusted the action.
So I escalated.

⚔️ Phase 3 — Fighting Back With a Chrome Extension

At this point, I realized manual debugging alone wasn't enough anymore.

So I decided to escalate.

I created a custom Chrome extension specifically for this battle.

The first version was primitive — a DOM-level interceptor designed to watch the page in realtime and surgically remove the invisible overlay traps before they could hijack clicks.

I used a MutationObserver to monitor the page for dynamically injected elements:

const observer = new MutationObserver(() => {
    document.querySelectorAll('.suspicious-overlay')
        .forEach(el => el.remove());
});

observer.observe(document.body, {
    childList: true,
    subtree: true
});

For a brief moment…
…it worked.
Then the site hit back.

🤖 It was at this moment I realized I was fighting an Anti-Bot System

Buried deep inside the bundled scripts was an aggressive anti-fraud service called Adscore.

The behavior suddenly changed:

🧹 Console logs started disappearing
🔄 The player entered infinite click loops
🕵️ Hardware fingerprinting routines activated
🎮 WebGL context checks triggered
📏 Screen dimension math got evaluated
🧠 Browser behavior started getting profiled

The site wasn’t just showing ads anymore.
It was actively detecting interference.
And then I realized my mistake:

I was still fighting inside the webpage sandbox.

That was their territory.

🧠 Phase 4 — Stop Fighting the UI. Attack the Network.

If the frontend is weaponized…
…you stop fighting visuals.

You cut the supply lines.

Instead of targeting the popup elements themselves, I started tracing the network behavior behind them.
During tiny windows before the console got wiped, I managed to capture the redirect domains responsible for the popup chains.
That changed everything.

Because once you identify the routing infrastructure the illusion collapses.

🔥 The Final Architecture

I rebuilt the extension entirely around Chrome’s native browser APIs:

declarativeNetRequest
tabs
background service workers

No more DOM fighting.
No more chasing invisible overlays.
No more playing inside their sandbox.
The browser itself would now intercept requests before the scripts could fully execute.

📦 manifest.json

{
  "manifest_version": 3,
  "name": "Native Ad Network Guard",
  "version": "1.0",
  "description": "Drops ad-network connections and instantly auto-closes leaked popups.",
  "permissions": [
    "declarativeNetRequest",
    "tabs"
  ],
  "host_permissions": [
    "<all_urls>"
  ],
  "background": {
    "service_worker": "background.js"
  }
}

⚡ background.js

// Domains responsible for popup routing,
// ad redirects, tracking, and anti-user behavior.
const BLOCKED_DOMAINS = [
    "xadsmart.com",
    "adsco.re"
];


// ---------------------------------------------------
// 1. Create browser-level blocking rules
// ---------------------------------------------------

// Chrome's declarativeNetRequest API works by defining
// static rule objects that the browser engine enforces
// BEFORE requests fully execute.

const rules = BLOCKED_DOMAINS.map((domain, index) => ({
    id: index + 1,
    priority: 1,

    // Action to perform when matched.
    // In this case: completely block the request.
    action: { type: 'block' },

    // Conditions that trigger the rule.
    condition: {
        // Match any request containing this domain.
        urlFilter: domain,
        // Types of browser resources to intercept.
        resourceTypes: [
            // Entire page navigations
            'main_frame',
            // Embedded iframes
            'sub_frame',
            // External JavaScript files
            'script',
            // AJAX/fetch/XHR requests
            'xmlhttprequest'
        ]
    }
}));


// ---------------------------------------------------
// 2. Register rules when extension installs
// ---------------------------------------------------

chrome.runtime.onInstalled.addListener(() => {

    chrome.declarativeNetRequest.updateDynamicRules({
        // Remove previously existing rules
        // to avoid duplicates during reinstalls.
        removeRuleIds: rules.map(r => r.id),

        // Inject the new blocking rules.
        addRules: rules
    });
});


// ---------------------------------------------------
// 3. Instant popup execution kill-switch
// ---------------------------------------------------

// Even if a popup somehow bypasses the network block,
// this listener acts as a secondary defense layer.
// It watches every tab URL update in realtime.

chrome.tabs.onUpdated.addListener((tabId, changeInfo) => {

    // Ignore updates that don't contain a URL.
    if (changeInfo.url) {

        // Normalize URL for safer comparisons.
        const url = changeInfo.url.toLowerCase();

        // Check whether the tab matches
        // known popup patterns or redirect signatures.
        const shouldKill =

            // Known blocked domains
            BLOCKED_DOMAINS.some(domain =>
                url.includes(domain)
            )

            // Common ad-network query patterns
            || url.includes('zoneid=')

            // Aggressive redirect handler pattern
            || url.includes('afu.php');

        // If detected:
        // instantly terminate the tab.
        if (shouldKill) {
            chrome.tabs.remove(tabId);
        }
    }
});

🚀 The Result

The transformation was surreal.
The site still attempts the same aggressive click-hijacking flow.
Invisible overlays still spawn.
Scripts still try redirect chains.
But now?

💀 The tabs die instantly.

No chaos.
No cleanup.
No losing immersion every five seconds.
Just a silent war happening underneath the browser while the video plays normally.

🧩 What This Actually Taught Me

The interesting part wasn’t blocking ads.

It was understanding how modern websites are willing to weaponize the browser itself:

Event hijacking
Trusted-click exploitation
Fingerprinting
Anti-debugging
Console wiping
Behavioral analysis
Redirect chaining
Dynamic overlay injection

👨‍💻 Final Thought

This entire project started because I got annoyed clicking “Close Tab” fifteen times.
It ended with me reverse-engineering popup infrastructure, anti-bot systems, and browser-level request interception pipelines.
And honestly?
That escalation path perfectly summarizes what I enjoy most about software engineering.

Tinkering.

And refusing to accept that a system is “unbeatable.”