DEV Community: Nick

Inside Instagram's Anti-Bot Stack: What Actually Catches You

Nick — Thu, 03 Sep 2026 11:11:48 +0000

Anti-bot protection is usually described as a wall. It is closer to a stack of independent checks, each reading a different part of the connection, each costing the operator something different to run and the automated client something different to defeat.

Meta's stack makes a good case study because an unusual amount of it is documented in public, in research papers and vendor specifications. What follows is assembled from those sources rather than from probing the platform, and contains no bypass instructions. The useful part is knowing which layers are cheap theatre and which ones decide the outcome.

The layers

     signal                          cost to fake     cost to run
  ---------------------------------------------------------------
  1  TLS handshake (JA3/JA4)         low              trivial
  2  HTTP/2 connection preface       low              trivial
  3  IP and ASN reputation           medium           cheap
  4  Browser surface consistency     medium           cheap
  5  Client attestation (mobile)     very high        cheap
  6  Behavioral telemetry            high             moderate
  7  Account graph position          impractical      expensive

Most anti-detection tooling addresses layers 1 through 4. Accounts die at layers 5 through 7. The ordering follows one rule: a signal is worth what it costs the attacker to control, not what it costs the defender to collect.

Layers 1 and 2: the connection describes the client

Before the first byte of a request exists, the TLS ClientHello has described the stack that produced it. Cipher list and order, extensions, supported groups, signature algorithms, ALPN. None of it is set by application code, which is why libraries differ from each other and why the fingerprint survives IP rotation and header spoofing. JA4, from FoxIO, encodes that as a partly readable string rather than JA3's opaque hash.

t13d1516h2_8daaf6152771_b186095e22b6
││ ││ │ │  │            └── extension + signature-algorithm hash
││ ││ │ │  └── cipher suite hash
││ ││ │ └── ALPN (h2)
││ ││ └── extension count (16)
││ │└── cipher count (15)
││ └── SNI present (d = domain, i = IP or absent)
│└── TLS version (1.3)
└── transport (t = TCP, q = QUIC)

HTTP/2 leaks the same way. An Akamai white paper from Black Hat EU 2017 identified four sources: the SETTINGS frame contents and ordering, the initial WINDOW_UPDATE increment, unsolicited PRIORITY frames, and the order of the request pseudo-headers. Those separate Chrome from Firefox from Go's standard library from curl cleanly, because each sits hardcoded in the implementation rather than exposed as configuration.

Both layers are cheap to normalize once known, so neither works as a standalone block. They filter the large volume of traffic from stock HTTP clients and contribute a feature to a larger model for everything else.

Layer 3: the address is a prior

IP and ASN reputation carries more weight in proxy marketing than it earns. What an address supplies is a prior probability. A datacenter ASN carrying 400 logins in an hour is a strong prior, a residential ASN carrying one is weak, and neither decides anything alone, because addresses are shared and a real user on mobile data changes IP several times a day.

Address history matters once it is joined to an account. One account appearing from twelve ASNs on three continents in an afternoon is an account-consistency signal that happens to arrive through addresses.

Layer 4: the tell is contradiction

Browser fingerprinting is commonly misread as a uniqueness problem where the goal is to look unremarkable. The stronger signal is internal contradiction. FP-Scanner (Vastel et al., USENIX Security 2018) showed that countermeasures defeat themselves, because altering attributes produces combinations that cannot occur naturally. Their consistency battery reached 1.0 accuracy detecting spoofing tooling against 0.45 for FingerprintJS2, and recovered the real browser family every time.

// Capture this before page script can shadow it. Calling
// navigator.hasOwnProperty() instead lets one assignment blind the check.
const hop = Object.prototype.hasOwnProperty;

const probe = {
  uaClaim:     navigator.userAgent,
  platform:    navigator.platform,
  evalLength:  eval.toString().length,   // 33 Chromium, 37 Firefox/Safari
  errorShape:  (() => { try { null.f() } catch (e) { return e.constructor.name } })(),
  touch:       'ontouchstart' in window,
  cores:       navigator.hardwareConcurrency,
  navOwnProps: Object.getOwnPropertyNames(navigator)
                 .filter(k => hop.call(navigator, k)),   // should be empty
};

A user agent claiming Firefox beside an eval length of 33 is a contradiction. So are touch events on a machine reporting a desktop OS, a Mesa renderer next to a MacIntel platform string, and own properties on the navigator instance, which the WebIDL spec places on the prototype.

Two limits are worth stating. The paper's rule treating ANGLE as a Windows indicator has expired, since ANGLE is now Chrome's default graphics layer on Linux, macOS and Android, so that check fires on ordinary users today. And the own-property test catches naive spoofing only, because prototype-level patching never touches the instance, which is where current automation frameworks work. Cost is not the constraint on running any of it: stopping at the first contradiction resolved 83 percent of fingerprints in 0.21 milliseconds.

Layer 5: attestation closes the mobile surface

A browser is a hostile execution environment from the site's perspective, since everything the page measures runs on the client and can be lied about. Native apps escape that through hardware. Google's Play Integrity API returns a signed verdict, and on Android 13 and above MEETS_DEVICE_INTEGRITY carries hardware-backed proof that the bootloader is locked and the running OS is a certified manufacturer image. An empty verdict indicates rooting, API hooking, or a failing emulator, and a separate PLAY_RECOGNIZED field confirms the package and certificate match what Play distributes. Apple's App Attest does the equivalent through the Secure Enclave.

The consequence runs against intuition. A large app's mobile API is usually the harder target of the two, because a signed attestation from a locked device is a claim that infrastructure cannot manufacture. Web endpoints stay comparatively open because browsers cannot make that claim either, which forces the defense there to be statistical rather than cryptographic.

Layers 6 and 7: the account graph

Meta published Deep Entity Classification at USENIX Security 2021. It describes Facebook rather than Instagram and is five years old, so read it as a window into how the company approaches this class of problem rather than a current specification.

The premise is that direct account features are attacker-controlled and therefore weak. Account age, connection count, posting frequency and profile completeness are trivially set by whoever runs the operation. So the system classifies on something else: over 20,000 deep features per account, aggregated from the properties of direct and indirect graph neighbors across multiple hops. In production the authors reported precision above 0.97, recall of 50 percent at 0.95 precision against 22 percent for a single-stage version, and hundreds of millions of accounts actioned over two years on 0.7 percent of global CPU.

The finding worth sitting with concerns evasion. Across the evaluation window the authors observed no adversarial adaptation to the deep features, unlike their experience with earlier systems. The reason is economic. Moving an account's aggregated neighbor statistics means controlling its neighbors and their neighbors, and doing that at volume produces the coordination side effects cheaper systems already catch. This is why enforcement arrives in batches rather than one account at a time. Whatever grouped those accounts was structural, and no header or proxy pool addresses a signal that is not about where the requests came from.

What the law allows

In January 2024, Judge Edward Chen of the Northern District of California ruled in Meta Platforms v. Bright Data (23-cv-00077-EMC) that Meta's terms did not prohibit scraping publicly available data while logged off, reasoning that Bright Data was not an active user in that state. Chen also declined to enforce a survival clause banning scraping in perpetuity after account termination, writing that letting companies decide who may collect and use data risks creating information monopolies. Meta was later reported to have abandoned the case.

The result is real and narrower than the coverage suggested. It is a contract ruling on one set of terms in one district, says nothing about logged-in access, does not address the CFAA, and does not touch GDPR, which is indifferent to contract position and very much concerned that profile data is personal data. The defensible zone is public, logged out, no account, no circumvention of a technical access control, and a lawful basis for anything involving identifiable people in Europe.

Designing collection around this

Start with the official API even when it looks insufficient, because it supplies a published budget rather than a guessed one. Meta's platform limit for app tokens is 200 calls per hour times user count, and Instagram's non-messaging endpoints run on 4800 times impressions per 24 hours. Usage returns in headers as you go, though the two endpoint families use different ones: flat X-App-Usage for app-token Graph calls, and X-Business-Use-Case-Usage keyed by business ID for Instagram Platform and Marketing endpoints. Handle only the first and the backoff never fires on exactly the endpoints that needed it. The unofficial path supplies no such number, so the limit gets learned by exceeding it.

For public logged-out collection, spend the engineering on coherence and pacing rather than volume. One internally consistent client moving at a plausible rate outlasts fifty that are individually perfect at the TLS layer and collectively obvious. Session stickiness matters more than pool size, since a pool rotating addresses mid-session manufactures the inconsistency layer 3 watches for.

Anything requiring authenticated accounts at scale belongs in the business risk column rather than the engineering one. It works for a while, which is the trap. The failure mode is losing accounts in batches with no diagnostic signal explaining why. That caps what can be collected, and the compensation is that the cap is stable, which is something to build on in a way a success rate that resets every quarter is not.

Five strategies for making your traffic look like a human's

Nick — Mon, 31 Aug 2026 14:19:54 +0000

Rotating IP addresses has a reputation problem. Everyone agrees you need it, and everyone also quietly suspects it does not work that well. The complaints sound the same: "we got banned anyway," "every IP was bad," "it was slower than going direct." Some of that is bad provider quality. Most of it is not.

The part that decides whether rotation works is the pattern sitting on top of the IPs. A flawless IP pool hammered with twenty requests per second from a known datacenter ASN gets blocked in minutes. A mediocre residential IP that makes three requests with realistic pauses between them sails through. The IP list matters. The behavior matters more.

This article covers five strategies that address the behavior side. None of them are exotic. They are the things most setups skip because they feel obvious or slow, and skipping them is exactly what gets you blocked.

Strategy 1: Throttle Before You Rotate

The most common mistake is the proxy choice itself. The request rate is the real problem. Developers rotate proxies and then still send ten or fifteen requests per second, which is the single most legible bot signal there is. Rotation does not fix a request rate that no human would ever produce.

Add delays between requests and vary them. For moderate targets, two to five seconds between requests works. For sensitive targets like logins or financial data, five to ten seconds or more. The variation matters as much as the delay itself. A fixed two-second pause is a pattern. A random two-to-five-second pause is behavior.

import random
import time
import requests

def request_with_delay(url, proxies):
    delay = random.uniform(2, 5)  # Random 2-5 second delay
    time.sleep(delay)
    return requests.get(url, proxies=proxies)

# Usage: space your calls. Do not batch them.
proxies = {"http": "http://user:pass@gateway:7000", "https": "http://user:pass@gateway:7000"}
for url in target_urls:
    resp = request_with_delay(url, proxies)
    if resp.status_code == 200:
        process(resp)

This slows you down. It also lets you keep going. A scraper that respects rate limits can run for weeks. A scraper that does not gets banned in hours, and no amount of IP rotation saves it.

Strategy 2: Rotate by Session, Not by Timer

Rotating proxies every N seconds regardless of what you have done with them is the second most common mistake. It wastes your pool and creates a detectable rhythm. A better approach ties rotation to actual usage.

Session-based rotation means each IP handles a small batch of requests, then retires. Five to ten requests per IP is a reasonable default for moderate workloads. The IP gets used, proves itself trustworthy with a few normal-looking requests, and then steps aside. This is what a real user's traffic looks like: a burst of activity, then a pause while the user reads or clicks elsewhere.

import itertools
import requests

class SessionRotator:
    def __init__(self, proxy_pool, requests_per_session=10):
        self.pool = itertools.cycle(proxy_pool)
        self.current = next(self.pool)
        self.count = 0
        self.requests_per_session = requests_per_session

    def get_proxy(self):
        if self.count >= self.requests_per_session:
            self.current = next(self.pool)
            self.count = 0
        self.count += 1
        return self.current

# Each IP handles up to 10 requests, then rotates.
# Adjust requests_per_session based on target sensitivity.
rotator = SessionRotator(proxy_pool, requests_per_session=10)
for url in target_urls:
    proxy = rotator.get_proxy()
    resp = requests.get(url, proxies={"http": proxy, "https": proxy})

For light monitoring, time-based rotation with a small pool is fine. For heavy scraping at hundreds of requests per minute, per-request rotation with a large pool is the only option that avoids clustering. The middle ground, session-based rotation with twenty to fifty IPs, covers most real workloads.

Strategy 3: Match Proxy Type to Target Sensitivity

Datacenter proxies are fast, cheap, and instantly recognizable. They come from cloud provider IP blocks, and any anti-bot system that checks ASN ownership can spot them immediately. They work fine for open APIs, public pages, and targets with weak bot detection.

Residential proxies route through real ISP connections. They cost five to ten times more but look like actual home users. They are worth the premium when the target has real anti-bot infrastructure, which is most sites that hold valuable data.

The hybrid approach is what most setups should run. Start with datacenter proxies for the bulk of your traffic. Keep a smaller residential pool for the targets that actually need it. Most scraping jobs fall into the first category. The sites that fight back are the exception, not the rule.

Proxy Type	Cost per GB	Detection Risk	Best For
Datacenter	$0.50–$2	High	Open APIs, public pages
Residential	$5–$30	Low	Login flows, guarded sites
Mobile	$15–$50+	Very low	App-native, heavily fingerprinted

One of the providers that builds its plans around this distinction is 2Extract, which bills residential and mobile proxies per port rather than per GB on some plans, on the theory that a bandwidth-heavy scraping job and a bursty session-based automation job should not be priced the same way. The provider question is worth revisiting once you know which workload you are actually running.

Strategy 4: Plan for Bans, Because They Happen

Even with good rotation and sensible throttling, bans happen. The difference between losing an hour of work and losing a day is what you do when a request comes back with a 403 or a captcha.

Rotate your request headers alongside your IP. A fresh IP carrying a stale user-agent or no referer is more suspicious than a reused IP with consistent headers. Keep a small pool of realistic user-agents and rotate them independently of the proxy pool.

import random
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0",
]

def request_with_headers(url, proxy):
    headers = {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept-Language": "en-US,en;q=0.9",
        "Referer": "https://www.google.com/",
    }
    return requests.get(url, proxies={"http": proxy, "https": proxy}, headers=headers)

Track your success rates per IP. An IP that accumulates errors is burning money whether you know it or not. Retire it. Most bans come from predictable patterns, not from bad IPs, so logging and retiring is more effective than buying a bigger pool.

Strategy 5: Measure What Actually Matters

The proxy that works is rarely the one with the most IPs or the cheapest GB rate. It is the one whose sessions survive, whose IPs are distributed across enough subnets and ASNs to look like real users, and whose pricing matches your actual usage pattern.

Ask providers for session survival rate before you ask for IP count. A hundred IPs from one narrow subnet behaves like one crowded neighborhood to a detection system, not like a hundred independent identities. Ask what percentage of sessions complete without the IP dropping mid-task. That number predicts real-world reliability better than pool size.

What This Looks Like in Practice

Putting it together, a resilient setup has five layers:

Delays. Randomized pauses that keep request rates in human territory.
Session-based rotation. Each IP handles a small batch, then retires.
Mixed proxy types. Datacenter for volume, residential for sensitive targets.
Header rotation. Fresh user-agents and referers alongside fresh IPs.
Monitoring. Track success rates, retire failing IPs, adjust pacing.

None of these layers is expensive. The expensive part is buying the wrong proxy type for the workload, which is what most people do because they guess instead of measure.

The Short Version

Proxy rotation is not a procurement problem. You do not need the most IPs or the cheapest rate. You need traffic that looks like it came from a person who is doing something specific and then pausing. Get the pacing right, rotate by session rather than by timer, match the proxy type to the target's sensitivity, and measure what breaks. The IPs are the easy part.

Why Indonesia Is an Underrated GEO for Mobile Proxies

Nick — Tue, 25 Aug 2026 14:22:41 +0000

Most proxy users default to the same three GEOs: United States, United Kingdom, Germany. Maybe Japan if they need Asian coverage. Indonesia rarely makes the shortlist, and that is a blind spot.

Indonesia has the largest e-commerce market in Southeast Asia at roughly $57 billion GMV. It has 140% mobile penetration, meaning more active SIM cards than people. It has three major carriers: Telkomsel, Indosat Ooredoo Hutchison, and XL Axiata, each with tens of millions of subscribers and distinct IP ranges. The anti-bot vendors protecting Indonesian e-commerce and fintech treat traffic from those carrier ranges as baseline human traffic.

If you are scraping Indonesian targets or need mobile IPs that carry high trust scores in the region, Indonesia is not a niche GEO; it is a primary one. Most providers just have not caught up.

The numbers that should get your attention

Mobile penetration hit 140% in 2024, and that is not a typo. Many Indonesians carry two SIMs: one for data, one for calls, or one for work and one for personal. Telkomsel alone serves roughly 153 million subscribers, about 45% of the market. Indosat Ooredoo Hutchison and XL Axiata split most of the remainder. These are not small carrier pools; they are carrier-grade IP ranges with millions of active devices behind them.

4G is practically universal in populated areas. 5G is live in selected cities and expanding. The government's Universal Service Obligation program deployed nearly 7,000 new base stations in underserved regions by late 2024, pushing national population coverage to 96%. The pool of clean, rotating mobile IPs is growing, not shrinking.

The e-commerce target surface

Indonesian e-commerce is dominated by three platforms: Shopee (roughly 52% market share), TikTok Shop (which absorbed Tokopedia), and Lazada. Together they control the vast majority of platform GMV. Blibli holds a smaller but significant slice in premium categories.

These platforms run real anti-bot. Cloudflare, DataDome, Akamai, and PerimeterX are all present. But they are tuned for the traffic patterns they actually see. In Indonesia, that traffic is overwhelmingly mobile. A request from a Telkomsel IP with a matching mobile user agent and a plausible TLS fingerprint looks like the 153 million legitimate users doing the same thing. A request from a US residential IP looks like an anomaly.

The trust score gap is real. Mobile IPs from Indonesian carriers start with higher baseline trust on Indonesian targets than residential IPs from any country, because the legitimate traffic mix is mobile-first by a wide margin.

Where the anti-bot vendors actually look

Cloudflare protects roughly a fifth of the web and leans on JA4 TLS fingerprinting plus managed challenges. Akamai goes deeper with sensor validation past the TLS layer and is common on high-value targets like banking and airlines. DataDome uses ML-first behavioral modeling across its network. PerimeterX, now HUMAN, combines CDN protection with behavioral biometrics.

On Indonesian e-commerce and fintech, you will see a mix. The common thread is that all of them weight IP reputation heavily, and Indonesian mobile carrier IPs carry strong reputation on Indonesian targets. A clean Telkomsel or Indosat IP passes the IP reputation layer that would flag a data center IP or a residential IP from a different region.

This is not unique to Indonesia. The same principle applies in any market where mobile traffic dominates. Indonesia happens to be a large market where the mobile dominance is extreme and the carrier IP ranges are well-defined.

The carrier specifics matter

Telkomsel's IP ranges are the broadest. If you need coverage across the archipelago, including tier-2 and tier-3 cities, Telkomsel is the pool with the reach. Indosat Ooredoo Hutchison has strong density in Java and Sumatra urban centers. XL Axiata has been expanding aggressively and its IP ranges are less saturated in proxy pools, which means cleaner reputations on average.

A provider that offers city-level and ASN-level targeting inside Indonesia lets you match the fingerprint to the expected traffic. Scraping a Jakarta-specific marketplace listing from a Jakarta Telkomsel IP with a Jakarta timezone and Indonesian locale is a completely different trust profile than the same request from a Singapore data center IP.

Most proxy providers do not expose this granularity for Indonesia. They offer "Indonesia" as a country-level target and back it with whatever IPs they can source. That works for low-stakes targets. It fails on anything with real anti-bot.

The legitimate use cases that drive demand

Price monitoring on Shopee, Tokopedia, and TikTok Shop is the most common driver. Brands and aggregators need to track SKUs across hundreds of thousands of listings. The platforms update prices dynamically and enforce geo-based pricing. You need Indonesian mobile IPs to see the same prices Indonesian users see.

Ad verification is the second. Brands running campaigns on TikTok, Instagram, and local publishers need to verify creative delivery in the Indonesian feed. The ad-serving logic is geo-aware and often carrier-aware. A mobile IP from the target carrier shows the same ads the target audience sees.

Fintech and super-app monitoring is the third: GoTo (Gojek plus Tokopedia), Dana, OVO, ShopeePay, these platforms run aggressive fraud detection because the stakes are financial. Mobile carrier IPs are the only ones that consistently pass their device and network fingerprinting at scale.

Travel and ticketing is the fourth. Airline and hotel pricing in Indonesia varies by point of sale. Mobile IPs from Indonesian carriers are the reliable way to see domestic pricing.

The trade-off nobody advertises

Indonesian mobile proxies cost more than Indonesian residential proxies, and the pool is smaller than the major carrier numbers suggest. Not every IP in a carrier's range is available, clean, and rotating. The effective pool size for a quality provider is a fraction of the subscriber count.

Throughput per IP is lower because you are sharing carrier NAT with real phones. Latency from outside Southeast Asia is higher. If your scraper runs from a US or EU server, the round-trip to an Indonesian mobile gateway adds 150 to 300 ms. That is fine for most scraping jobs. It is not fine for latency-sensitive arbitrage.

The honest limit is availability. If you need 500 concurrent Indonesian mobile IPs with city-level targeting, you will hit provider capacity limits faster than you would for US or EU GEOs. Plan for it.

What to ask your provider

If a provider claims Indonesian mobile coverage, ask three questions:

Which carriers do you source from, and can I target by carrier?
Do you support city-level and ASN-level targeting within Indonesia?
What is your session control, can I hold an IP for the duration of a checkout flow or a multi-step form?

If the answer to any of these is "we only offer country-level," the coverage is a checkbox, not a capability. Providers like 2Extract and a few others in the space build for this granularity because their customers hit the limits of coarse targeting.

Why the market underserves it

Proxy providers optimize for where the volume is. US and EU residential demand dwarfs everything else. Mobile proxy demand clusters in the same GEOs. Indonesia sits in a zone where the total addressable market looks smaller on a spreadsheet, but the trust-score advantage on Indonesian targets is structural.

The providers who invest in Indonesian carrier relationships, city-level targeting, and session control for that GEO are building a moat. The ones who treat it as a checkbox line item are leaving their customers exposed on the targets that matter.

The bottom line

If your scraping targets are Indonesian: e-commerce, fintech, travel, super-apps, and you are not using Indonesian mobile carrier IPs, you are fighting the trust score layer with one hand tied. The carrier IPs exist. The targeting granularity exists. The gap is provider coverage.

Start with a provider that offers carrier-level and city-level targeting for Telkomsel, Indosat Ooredoo Hutchison, and XL Axiata. Test on your hardest target. If residential was failing, mobile will likely close the gap. If mobile works, you have a GEO that most of your competitors are not even testing.

Real-World MCP Scenarios for Data Collection

Nick — Fri, 21 Aug 2026 15:11:57 +0000

Five practical scenarios where the Model Context Protocol replaces bespoke data-source adapters with a single uniform interface. Architecture patterns and code included

Every data pipeline turns into adapter maintenance eventually. You start with two sources, add a third, and by the time you hit a dozen your codebase is half glue code. Each source drags in its own auth flow, its own SDK, its own pagination quirks, its own rate-limit header parsing. The pipeline works, but the cost of adding the next source keeps climbing instead of falling.

The Model Context Protocol flips that cost curve. Instead of writing a bespoke integration for every source, you expose each one as an MCP server with a standardized set of tools. Any MCP-compatible client, whether that is an LLM agent, a data pipeline, or a notebook, queries them through the same interface. Swapping arXiv for Semantic Scholar becomes a one-server change instead of a pipeline rewrite.

This post walks through five data collection scenarios where MCP earns its keep, with architecture sketches and code for each one. None of them are theoretical. All of them are patterns I have seen in production or built myself.

1. Multi-Source Research Aggregation

Gathering information across arXiv, GitHub, web pages, and internal documentation, then synthesizing it into a structured report, is a problem that sounds simple until you try it. Each source has a different query language, a different response shape, and a different rate limit. The synthesis step is easy once you have the data. Getting the data is the bottleneck.

With MCP, each source becomes a server. The client does not care what is behind each tool.

          MCP Client
         (agent or pipeline)
              |
    +---------+---------+-----------+
    |         |         |           |
  arXiv     Web      GitHub    InternalDocs
 Server    Server    Server     Server

Each server exposes tools like search_papers, fetch_repo_metadata, extract_url_content, and query_internal_docs. The client calls them in sequence or parallel. Every response comes back in the same structured format.

results = await mcp_client.call_tool("search_papers", {"query": "graph neural networks"})
repos = await mcp_client.call_tool("search_repos", {"query": "graph neural networks"})
web = await mcp_client.call_tool("extract_url", {"url": "https://..."})

synthesized = synthesize(results, repos, web)

The trade-off is latency. You are adding a protocol layer between your client and the source, and for a single-source lookup that overhead is not worth it. MCP pays off when you have three or more sources and you want the freedom to swap any one of them without rewriting the pipeline.

2. Continuous Monitoring and Change Detection

Monitoring competitor websites, product listings, or research feeds for changes is a different shape of problem. You are not collecting data once. You are collecting it repeatedly and caring only about the diff.

Set up an MCP server that wraps each monitored source. A scheduled job calls the relevant tools at intervals, hashes the output, and triggers only when the hash changes.

# Cron job fires every 30 minutes
schedule: "*/30 * * * *"
prompt: |
  Call the "fetch_competitor_pricing" tool.
  Compare the result with the last known state.
  If prices changed, summarize what moved and by how much.

+-------------+     +--------------+     +--------------+
|  Scheduler  |---->|  MCP Client  |---->|  Alerting    |
|  (cron)     |     |  (LLM agent) |     |  (Slack/DM)  |
+-------------+     +-------+------+     +--------------+
                            |
                    +-------+-------+
                    |               |
              Pricing MCP      Blog RSS MCP
              Server            Server

The design choice that makes or breaks this pattern: the MCP server must emit stable output. No timestamps in the response body. No random ordering. No request_id field that changes every call. The scheduler hashes the exact bytes. Same hash means nothing changed, so the agent does not run and no alert fires. Different hash means something moved, so the agent runs, diffs the old and new state, and sends a digest.

The failure mode is subtle. If the server includes any non-deterministic field in its output, the hash changes every tick and you get alert fatigue. Discipline at the server level is what makes the monitoring pattern work.

For monitoring that involves web scraping, the MCP layer often sits in front of a proxy network rather than a direct HTTP client. The agent describes the job, the server provisions the right geo-targeted proxy, and the connection string comes back ready to use. 2extract ships an MCP server that does exactly this: you tell it "collect gaming-laptop prices from Amazon in Germany, the UK, and Japan" and it handles country-level geo targeting, creates a proxy resource, sets a traffic cap, and returns three connection strings. The agent never touches proxy configuration directly. It just calls the tool and gets a working endpoint back.

3. Structured Extraction from Unstructured Sources

PDFs, scanned documents, HTML pages, and Slack messages all contain structured information trapped inside unstructured formats. You need dates, obligations, action items, and dollar amounts pulled out and normalized into a single schema.

Each document type gets an MCP server with extraction tools tailored to its format.

Source Type	MCP Server	Key Tools
PDFs	`pdf-server`	`extract_text`, `extract_tables`, `fill_form`
Scans/OCR	`ocr-server`	`ocr_page`, `extract_fields`
Web pages	`web-server`	`extract_content`, `extract_metadata`
Chat logs	`chat-server`	`search_messages`, `extract_decisions`

The client orchestrates calls across all of them and normalizes into your target schema.

contract_fields = await mcp.call_tool("extract_fields", {
    "file": "contract.pdf",
    "fields": ["party_a", "party_b", "deadline", "amount"]
})

email_decisions = await mcp.call_tool("extract_decisions", {
    "thread_id": "t-12345",
    "since": "2025-01-01"
})

merged = merge_and_dedupe(contract_fields, email_decisions)

The honest limit is extraction quality. MCP standardizes the interface, not the underlying OCR or parsing engine. A badly tuned extraction server returns garbage through the same clean protocol. You still need to invest in the extraction quality itself, and for low-volume document sets the protocol overhead may not justify the architecture.

4. Database-to-Insights Pipeline

You have a warehouse, Postgres or BigQuery or Snowflake, and you want non-technical stakeholders to ask natural-language questions and get grounded answers. The catch is that you cannot hand them raw SQL access.

An MCP server wraps the database with read-only, parameterized query tools. The server only exposes curated queries, not a SQL passthrough.

SELECT * FROM sales WHERE region = $region AND quarter = $quarter

The client, an LLM agent, receives a natural-language question, picks the right tool, fills the parameters, and returns a structured answer.

User: "What were Q3 sales in EMEA?"

Agent flow:
  1. call_tool("query_sales", {"region": "EMEA", "quarter": "Q3"})
  2. Receive structured rows
  3. Summarize with numbers cited from the result

The security model is clean. The MCP server is the enforcement layer. It only exposes parameterized queries, so the LLM never touches the connection string and cannot inject arbitrary SQL. The principle of least privilege applies the same way it does for any REST API you would build in front of a database.

The trade-off is coverage. You can only expose queries you have pre-written and parameterized. When a stakeholder asks a question your tools cannot answer, you either write a new tool or tell them the data is not available through the interface yet. For most security-conscious teams that controlled surface is the whole point, but it does mean the interface grows organically rather than covering everything on day one.

5. Federated Data Collection Across Teams

Different teams in an org each own their data. Engineering owns GitHub metrics. Product owns Mixpanel. Finance owns Stripe. You need a unified view for a quarterly review, but you do not want to centralize all data in one warehouse just for that purpose.

Each team runs their own MCP server with their own auth, rate limits, and tool surface. A federated client queries across all of them and joins results in memory.

         +--------------------------+
         |   Federated MCP Client   |
         +--+------+------+--------+
            |      |      |
   +--------+  +---+      +--------+
   |           |                   |
 Engineering   Product             Finance
 MCP Server    MCP Server          MCP Server
 (GitHub,      (Mixpanel,         (Stripe,
  Linear)       Amplitude)         QuickBooks)

eng_metrics = await mcp.call_tool("get_deploy_frequency", {"quarter": "Q3"})
prod_metrics = await mcp.call_tool("get_feature_adoption", {"quarter": "Q3"})
fin_metrics = await mcp.call_tool("get_revenue", {"quarter": "Q3"})

report = build_qbr(eng_metrics, prod_metrics, fin_metrics)

Each team keeps control over their own data access. The federated client never sees credentials or connection details for any source. It just calls tools and gets structured results back. For a quarterly review that touches three teams, this is faster and cheaper than building a central ETL pipeline, and the teams do not have to hand over their keys.

The limitation is join complexity. You are joining in memory, which is fine for aggregate metrics like a QBR. If you need to join millions of rows across teams, you need a warehouse after all, and the federated pattern does not replace it.

Patterns That Apply Across All Five

A few design decisions recur across every scenario above. They are worth stating plainly.

Tool-per-source, not tool-per-action. Do not create get_github_issues, get_github_prs, get_github_releases as separate tools. Create one query_github tool with a flexible resource parameter. Fewer tools means less confusion for the LLM client and a smaller tool surface to maintain.

Stable output for monitoring. If you are using MCP for change detection, the server's output must be deterministic for a given input state. No timestamps, no random ordering, no request_id fields. Hash it, compare it, act on the diff. This came up in Scenario 2 but the discipline applies anywhere you hash server output.

Read-only by default. Data collection servers should be read-only. If a tool can write, it should require explicit confirmation on the client side. Treat MCP servers like REST APIs. Least privilege.

Pagination via cursor parameters. Do not try to return 10,000 rows in one tool call. Expose limit and cursor parameters and let the client page through.

while True:
    page = await mcp.call_tool("query_records", {
        "cursor": cursor,
        "limit": 100
    })
    results.extend(page["rows"])
    if not page["next_cursor"]:
        break

A Minimum Viable MCP Server

The spec is open and the reference SDKs are straightforward. Here is the smallest server that does something useful for data collection.

from mcp.server import Server
from mcp.types import Tool, TextContent

server = Server("my-data-collector")

@server.list_tools()
async def list_tools() -> list[Tool]:
    return [
        Tool(
            name="collect_data",
            description="Collect data from source X with optional filters",
            inputSchema={
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "limit": {"type": "integer", "default": 50},
                    "cursor": {"type": "string"}
                },
                "required": ["query"]
            }
        )
    ]

@server.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
    if name == "collect_data":
        data = await my_data_source.fetch(
            query=arguments["query"],
            limit=arguments.get("limit", 50),
            cursor=arguments.get("cursor")
        )
        return [TextContent(type="text", text=json.dumps(data))]

if __name__ == "__main__":
    import asyncio
    asyncio.run(server.run_stdio())

Wire up your data source behind my_data_source.fetch() and any MCP client can start collecting from it. You do not have to build every server yourself. For the proxy and scraping layer specifically, managed MCP servers like the one 2extract publishes at mcp.2extract.com handle provisioning, geo-targeting, and spend limits so your agent can request a working proxy connection string in plain language. The protocol makes the integration surface uniform, so you can evaluate managed servers on their data quality and coverage rather than their SDK ergonomics.

Where This Goes

MCP earns its place when you have multiple data sources and the cost of maintaining bespoke adapters for each one is climbing. The five scenarios above cover most of the data collection workflows I have encountered, and they compose. You can combine monitoring with extraction, or federation with aggregation, by pointing your client at more servers.

The protocol does not fix bad data sources or poor extraction quality. What it fixes is the integration tax, the one where adding the next source costs more than the last. With MCP, adding a source means writing one server, not rewriting your pipeline.