DEV Community: Evelyn Lim

How to Manage Authenticated Browser State for AI Agents

Evelyn Lim — Thu, 05 Nov 2026 02:53:34 +0000

TL;DR:

A browser session, a managed profile, and a storage snapshot preserve different things. Choose the state mechanism that matches the workflow's lifetime.
Every restored identity needs a live application check. Saved cookies can remain present after the server has expired the login.
Persistent state needs a single writer policy. Concurrent sessions that save to the same profile can overwrite each other's state.

Authenticated browser state can grant access to an account. Treating it as a convenient file in a project directory is an easy way to create cross-customer errors and secret exposure.

An AI browser agent needs more than a way to log in once. It needs to know whose state it is using, which destinations may receive that state, whether the login is still valid, and who may update the saved copy.

This tutorial uses Scrapeless Browser for remote browser execution and its documented Profiles feature for persistence. It also explains where Playwright storage snapshots fit and why proxy continuity is a separate concern.

Separate the Three State Lifetimes

An active browser session, a managed profile, and a Playwright storage snapshot have different persistence boundaries.

An active session holds the current browser instance. Keeping it alive can preserve open tabs, in-memory page state, and the current browsing flow. Ending that browser ends the live execution environment.

A managed profile stores supported browser data for use in later sessions. Scrapeless Profiles support reuse across browser sessions. They require deliberate save behavior when the current session should update the stored profile.

A Playwright storage snapshot serializes supported authentication-related storage. It is not a complete browser backup. Playwright's authentication guidance explains reuse of saved state and its security implications. Local storage and cookies are common parts of the snapshot; IndexedDB can be included with the relevant option.

Session storage needs separate consideration. Browser sessionStorage is scoped to a page session, and Playwright's ordinary authentication snapshot flow does not automatically make it a portable replacement for every application's login mechanism.

Determine what the target application actually uses. A restored cookie file can be syntactically valid while the login is unusable because the application relies on other state or the server has revoked the session.

Bind the Profile to an Account Before Allocation

A profile identifier should resolve through an application-owned account record.

Use a mapping that includes the tenant, approved destination origin, account identifier, profile identifier, and current access status. The worker should receive the assigned profile from this mapping. It should not select an arbitrary profile identifier from page content or a model-generated action.

Check that mapping before opening the remote browser. A profile for one customer must not enter another customer's job, even if both tasks visit the same website.

Keep credentials separate from observation artifacts. The model can receive a stable account label such as “approved research account.” It does not need the raw cookie values or the secret used to establish the session.

The practical risk is session-token compromise: possession of a usable session token can enable account access. A state export deserves the same care as the credentials that created it.

Bootstrap a Profile Through an Authorized Login

A reusable profile begins with a permitted login that has been checked against the assigned account.

The current Node.js SDK supplies the client used below; account ownership and profile allocation remain application responsibilities.

Create the profile through the Scrapeless dashboard or its documented Profiles interface. Open a browser using that profile and enable profilePersist: true for the designated bootstrap session. Complete the application's approved login and MFA flow, inspect the account identity, then close the session through the supported lifecycle.

The Scrapeless Profiles configuration documents that persistence is needed to save changes and that profile data is saved when the session closes. A reader session can load the stored profile with profilePersist: false.

That setting controls writing the managed profile; it does not make the website read-only. A browser with persistence disabled can still click a destructive button or modify server-side account data. Enforce allowed actions separately.

Use an application-owned test account when developing the flow. A documented login procedure is not permission to automate another person's account or export their state.

Validate the Assigned Account in the Browser

Restored state should be accepted only after the destination confirms the expected identity.

The following script opens a reader session, visits a configured account page, and requires an account identifier to match. The default selector is an example contract for an application you control; replace it with a reliable identity selector for your authorized target.

Prerequisites are Node.js, Browser access, a bootstrapped Scrapeless profile, an approved account-page URL, and the exact account identifier expected on that page. Install the same reference packages used by the example:

npm init -y
npm install playwright-core@1.64.1 @scrapeless-ai/sdk@1.12.1

Set SCRAPELESS_API_KEY, SCRAPELESS_PROFILE_ID, ACCOUNT_URL, and EXPECTED_ACCOUNT_ID through your environment. Optionally set ACCOUNT_SELECTOR. Save the script as authenticated-state.mjs and run node authenticated-state.mjs.

Note: The account checks and storage export were exercised against a local test application. Scrapeless profile persistence, a real account login, and proxy egress require your configured service access and are not claimed as completed cloud tests. CDP_WS_ENDPOINT is an alternative for an authorized CDP browser you operate.

import { chromium } from 'playwright-core';
import { Scrapeless } from '@scrapeless-ai/sdk';
import { mkdir, open } from 'node:fs/promises';
import path from 'node:path';

const directEndpoint = process.env.CDP_WS_ENDPOINT;
const apiKey = process.env.SCRAPELESS_API_KEY;
const profileId = process.env.SCRAPELESS_PROFILE_ID;
const accountId = process.env.EXPECTED_ACCOUNT_ID;
const accountUrl = process.env.ACCOUNT_URL;
if (!accountUrl || !accountId) {
  throw new Error('Set ACCOUNT_URL and EXPECTED_ACCOUNT_ID');
}
if (!directEndpoint && (!apiKey || !profileId)) {
  throw new Error('Set SCRAPELESS_API_KEY and SCRAPELESS_PROFILE_ID');
}
const target = new URL(accountUrl);
const endpoint = directEndpoint ?? new Scrapeless({ apiKey }).browser.create({
  sessionName: 'authorized-account-reader',
  sessionTTL: 180,
  proxyCountry: 'US',
  profileId,
  profilePersist: false,
}).browserWSEndpoint;

const browser = await chromium.connectOverCDP(endpoint, { timeout: 30000 });
try {
  const context = browser.contexts()[0];
  if (!context) throw new Error('Provider default context is unavailable');
  const page = await context.newPage();
  const response = await page.goto(target.href, {
    waitUntil: 'domcontentloaded', timeout: 30000,
  });
  if (!response || !response.ok()) throw new Error('Account page did not load');
  if (new URL(page.url()).origin !== target.origin) {
    throw new Error('Unexpected authentication destination');
  }
  const identity = page.locator(
    process.env.ACCOUNT_SELECTOR ?? '[data-testid="account-id"]'
  );
  await identity.waitFor({ state: 'visible', timeout: 10000 });
  if ((await identity.innerText()).trim() !== accountId) {
    throw new Error('Active account does not match the assigned account');
  }

  if (process.env.EXPORT_STATE === 'true') {
    const directory = path.resolve('.auth');
    await mkdir(directory, { recursive: true, mode: 0o700 });
    const state = await context.storageState({ indexedDB: true });
    const file = await open(path.join(directory, 'account.json'), 'wx', 0o600);
    try { await file.writeFile(JSON.stringify(state)); }
    finally { await file.close(); }
  }
  console.log('Assigned account validated; approved read workflow may proceed');
} finally {
  await browser.close();
}

The identity check belongs before data extraction or account actions. A welcome message or an unchanged URL is a weak substitute for a stable account identifier.

The final-origin check detects unexpected navigation after it occurs. A stronger destination restriction requires browser or network controls that prevent disallowed outbound requests. Keep credentials and exported state scoped to the intended application.

Use a Single Writer for Profile Updates

Profile updates should be serialized when several tasks share the same saved state.

The Scrapeless Profiles documentation states that, with persistence enabled, the last session to close supplies the saved data. Two independently modified browser sessions can therefore compete to write the same profile.

An application-level single writer policy avoids that ambiguity:

Mark the profile unavailable for normal allocation while refreshing it.
Wait for current readers to finish if they could interfere with the account workflow.
Acquire a profile lock owned by the refresh task.
Run the approved login or renewal process with persistence enabled.
Close the writer session and validate the saved state in a fresh reader session.
Release the lock and return the profile to the allocation pool.

This lock is a design responsibility of your application. The example does not claim a provider-managed distributed lock.

Reader sessions can still affect server-side state. Avoid concurrent workflows that change the same account resource even when they cannot save the managed profile. Some applications also invalidate old sessions when a new login is created, so test account concurrency with the target's rules.

Keep Proxy Policy Separate From State Policy

A browser profile preserves supported browser data; a proxy selects the network path.

When the same authorized workflow spans several pages, keep the intended egress policy consistent. A changed location can alter the content shown or cause the destination to request additional authentication. It does not follow that a stable IP guarantees account access.

Use the managed browser's routing configuration when the browser runs in Scrapeless. Use Scrapeless standalone proxies when you operate the browser yourself or configure a documented custom route.

Choose the proxy type around the workload. Sticky residential allocations have a lifetime. A static ISP product is a separate option when a workflow needs a longer-lived network assignment. Neither turns an expired server-side login into an authenticated session.

Export State Only for a Defined Portability Need

A local storage snapshot is useful when an authorized workflow needs to move supported state between controlled environments.

The optional export path in the script is disabled by default. When EXPORT_STATE=true, it writes .auth/account.json with restrictive file permissions and refuses to overwrite an existing file. Include .auth/ in your repository's ignore rules. Never attach this file to a bug report or publish it with the tutorial.

The file can later be passed to a supported Playwright browser.newContext({ storageState: ... }) flow. That creates a context from the snapshot; it does not merge the snapshot into an existing provider context. Confirm that the destination runtime supports the context behavior you require.

The example's file permissions apply on systems that implement the corresponding Unix permission model. They do not replace encryption, host access control, or a retention policy. In a service, store the minimum necessary state in an appropriately controlled secret storage system.

Make Invalid Authentication an Explicit Result

An agent should stop with a specific authentication outcome when its assigned identity cannot be established.

Distinguish a missing identity element, an unexpected account, an MFA requirement, and an application authorization denial. Do not let the model proceed by improvising with another profile.

For account revocation, stop new allocations, terminate or drain active jobs as appropriate, invalidate exported state, and revoke the destination session where supported. Deleting a local snapshot or a managed profile alone does not guarantee that the website has revoked its own server-side session.

The persistent-profile product workflow is useful context for managing saved browser data. Your account policy must cover the destination's session lifecycle as well.

Conclusion

Authenticated agents need a verified account binding, an explicit persistence mechanism, controlled profile updates, and separate routing decisions. Reusing state is safe operationally only when the application can explain whose state it is and why it remains valid for the assigned task.

Use Scrapeless Browser to operate the remote session, then validate the identity before useful work. Budget for the chosen browser and proxy product configuration as distinct parts of the deployment, with standalone proxies evaluated against their own routing requirements.

FAQ

Q: Does profilePersist: false prevent account changes?

That setting prevents saving the session's browser data back to the managed profile. It does not prevent website actions or server-side account changes.

Q: Is storageState a complete browser backup?

A Playwright storage snapshot preserves supported storage state. It does not reproduce every tab, in-memory object, extension, or application-specific session mechanism.

Q: Can two workers update one profile?

Concurrent profile writers can overwrite one another's saved state. Serialize persistence through an application-controlled writer policy.

Q: Does deleting a profile sign out the account everywhere?

Deleting stored browser data does not necessarily revoke the destination's server-side sessions. Use the destination's supported session-revocation controls when required.

How to Connect Playwright to a Remote Browser over CDP

Evelyn Lim — Wed, 04 Nov 2026 10:34:15 +0000

TL;DR:

Use chromium.connectOverCDP() for a Chromium CDP endpoint. A Playwright protocol endpoint belongs to a different connection method.
Configure routing where the remote browser starts. A local HTTP client's proxy settings do not automatically configure the remote browser.
Validate content after navigation and save artifacts before closing. A successful connection establishes control, not a successful extraction.

Playwright can control a browser that runs on another machine. Your Node.js process sends commands through a control connection; the remote browser loads pages, executes JavaScript, and maintains browser state.

With Scrapeless Scraping Browser, you can use that pattern without operating the browser host yourself. The current product site calls the service Agent Browser. This tutorial uses the official SDK to build its connection URL and Playwright to drive the browser.

Understand Which Endpoint You Have

CDP and Playwright's native protocol require different connection methods.

Use the current Scrapeless Node.js SDK interface when adapting service configuration; older examples can expose different class names or option styles.

Playwright's connectOverCDP() API connects to Chromium through the Chrome DevTools Protocol. Playwright's connect() method expects a Playwright protocol endpoint. Changing the method name does not convert one protocol into the other.

CDP also has lower fidelity than a native Playwright protocol connection. If a workflow depends on advanced context behavior, tracing, downloads, or other provider-sensitive features, test those operations on the intended service. A basic navigation check cannot establish feature parity.

For Scrapeless, the SDK's browser.create() method returns a browserWSEndpoint. In the SDK used here, this method constructs the connection URL locally. Receiving that string does not prove that your API key is accepted or a cloud browser has started. The network connection is the next step.

Prerequisites

You need Node.js, a Scrapeless API key with Browser access, and an approved public target page.

The example defaults to https://example.com and expects its Example Domain heading. When adapting it, change the URL and expected heading together. Choose an expected value that distinguishes the intended content from a generic login or error screen.

Note: The SDK URL construction and browser logic were checked locally. Running this example against Scrapeless requires your own configured API key; the cloud connection and residential egress are not presented here as a completed service test.

Install the packages in a new project. These exact package versions are the reference environment for the example:

npm init -y
npm install playwright-core@1.64.1 @scrapeless-ai/sdk@1.12.1

playwright-core supplies the client library. You do not need to download a local Chromium binary to connect to a managed remote browser.

Connect, Check the Page, and Save Evidence

A useful connection example should validate a page and release the browser even if validation fails.

Save the following as connect-cdp.mjs. Configure SCRAPELESS_API_KEY through your environment or secret manager, then run node connect-cdp.mjs.

Note: Cloud execution requires the Browser prerequisite above. CDP_WS_ENDPOINT is an optional alternative for a CDP browser you control; leave it unset when using the SDK-created Scrapeless connection.

import { chromium } from 'playwright-core';
import { Scrapeless } from '@scrapeless-ai/sdk';
import { mkdir, writeFile } from 'node:fs/promises';
import { createHash } from 'node:crypto';
import path from 'node:path';

const directEndpoint = process.env.CDP_WS_ENDPOINT;
const apiKey = process.env.SCRAPELESS_API_KEY;
if (!directEndpoint && !apiKey) {
  throw new Error('Set SCRAPELESS_API_KEY or an authorized CDP_WS_ENDPOINT');
}
const target = new URL(process.env.TARGET_URL ?? 'https://example.com');
const expectedHeading = process.env.EXPECTED_HEADING ?? 'Example Domain';
const output = process.env.OUTPUT_DIR ?? './artifacts';
const endpoint = directEndpoint ?? new Scrapeless({ apiKey }).browser.create({
  sessionName: 'cdp-public-page',
  sessionTTL: 180,
  proxyCountry: 'US',
}).browserWSEndpoint;

await mkdir(output, { recursive: true });
const browser = await chromium.connectOverCDP(endpoint, { timeout: 30000 });
try {
  const context = browser.contexts()[0];
  if (!context) throw new Error('Provider default context is unavailable');
  const page = await context.newPage();
  const response = await page.goto(target.href, {
    waitUntil: 'domcontentloaded', timeout: 30000,
  });
  if (!response || !response.ok()) throw new Error('Document request failed');
  if (new URL(page.url()).origin !== target.origin) {
    throw new Error('Navigation left the approved origin');
  }
  const heading = page.locator('h1').first();
  await heading.waitFor({ state: 'visible', timeout: 15000 });
  const h1 = (await heading.innerText()).trim();
  if (h1 !== expectedHeading) throw new Error('Expected page content is absent');
  const capturedAt = new Date().toISOString();
  const html = await page.content();
  await writeFile(path.join(output, 'page.html'), html);
  await page.screenshot({ path: path.join(output, 'page.png'), fullPage: true });
  const result = {
    requestedUrl: target.href,
    finalUrl: page.url(),
    title: await page.title(),
    h1,
    documentStatus: response.status(),
    capturedAt,
    htmlSha256: createHash('sha256').update(html).digest('hex'),
    artifacts: { html: 'page.html', screenshot: 'page.png' },
  };
  await writeFile(path.join(output, 'result.json'), JSON.stringify(result, null, 2));
  console.log('Accepted page capture; artifacts saved');
} finally {
  await browser.close();
}

The script selects the provider's existing default context. It deliberately fails when that context is unavailable. Automatically creating a different context could change the state assumptions of a provider-managed browser.

The page check has several independent conditions. The document request must succeed. The final origin must remain the approved origin. The heading must be visible and must contain the expected value. This small example validates one public page; a production extractor needs a domain-specific check for its actual output fields.

An origin check after navigation detects an unexpected redirect, but it is not a network firewall: the browser has already followed that navigation. Enforce a strict outbound allowlist at the browser or network layer when your application requires it.

Read the Artifacts as Evidence of a Capture

The output files describe what the browser captured during this job.

result.json records the requested URL, final URL, document status, heading, observation time, and artifact names. page.html stores serialized DOM content. page.png captures a visual representation of the page.

The HTML hash checks the integrity of the saved HTML bytes. It does not prove who published the page or whether its contents are correct. The HTML and screenshot are captured sequentially, so a page that changes continuously can differ between them.

For an extraction workflow, add the selected values and their source locations to the result record. Keep the raw observation separate from any model-written interpretation. That separation makes a later review possible without rerunning the model.

Configure the Destination Proxy on the Remote Side

The remote browser's routing determines the destination-visible exit path.

The example requests a US browser proxy country through the SDK. This is a configuration request, not a measurement of the actual exit. If geography affects your result, check the browser's observed public IP through an approved diagnostic page and record the relevant location evidence.

The Scrapeless browser proxy controls cover built-in routing and a custom proxy URL. If you already operate a local browser, Scrapeless Proxies provide a separate configuration path for that fleet. These two deployments should be evaluated separately.

Avoid setting a local Playwright launch proxy and assuming it applies to a remote browser that has already been allocated. In this tutorial there is no local launch() call: browser allocation and routing are requested through the remote service.

The distinction follows the general separation between a client connection and proxy routing to the destination. The machine running the browser makes the page requests.

Use the Right Readiness Condition

Page readiness should describe the content you need, rather than an arbitrary delay.

domcontentloaded establishes an initial document milestone. The example then waits for a visible heading. For a product page, the useful condition might be a populated price field. For a research page, it might be the main article region and its expected title.

Playwright actionability checks help interactions wait for usable elements. They do not certify business correctness. A visible “Submit” button can still belong to the wrong account or form.

Avoid treating an empty result as proof that a field does not exist. Record whether the target element was absent, the document was a different page, or the extractor could not establish readiness. Those outcomes need different downstream handling.

Diagnose Failures at the Correct Layer

Connection, navigation, readiness, and validation failures point to different parts of the system.

Connection failure: check whether the endpoint is CDP, the API key is configured, Browser access is available, and the network permits a secure WebSocket connection. Keep token-bearing URLs out of logs.

Navigation failure: inspect the approved target URL, document response, and destination accessibility from the browser's egress.

Readiness failure: inspect the page state and locator. A login screen, consent screen, or changed DOM can explain why the expected content never appeared.

Validation failure: compare the captured state with the task contract. A valid response from the wrong application page should remain an unsuccessful task.

The Playwright proxy integration discussion provides additional context for connecting browser configuration to routing. Use the current connection parameters demonstrated above when adapting older examples.

Close the Browser Within the Task Boundary

Browser cleanup belongs in a finally block so extraction errors still reach the close operation.

Save artifacts before closing. Do not put close calls only after the successful result path. At the application level, also release queue slots and state locks when a worker exits.

The lifetime of the provider session is separate from the Playwright object in your process. Follow the provider's documented close and session-lifetime behavior. For persistent profiles, verify the storage behavior separately rather than assuming that a disconnected client saved all state.

Use Scrapeless pricing to budget for the browser workload and any separately configured Scrapeless proxy product. A functional CDP example does not establish the cost of your page set.

Conclusion

Connecting Playwright over CDP requires the correct protocol endpoint, an explicit browser-context choice, and a content check after navigation. The complete workflow also saves inspectable artifacts and ends its resources predictably.

Start with the bounded example on Scrapeless Browser, then replace its heading assertion with the actual acceptance conditions of your application.

FAQ

Q: Does CDP work with every Playwright browser engine?

Playwright's CDP connection supports Chromium-based browsers. It is not the connection method for Firefox or WebKit.

Q: Does browser.create() prove the cloud session exists?

The SDK method used here constructs the WebSocket URL locally. The subsequent connection is required to contact the browser service.

Q: Can an HTTP 200 response still be the wrong page?

An HTTP 200 response can contain a login page or other unexpected content. Validate the final page state and required values.

Q: Can this script run without an AI model?

This example is deterministic Playwright code and needs no model call. An agent can use the same connection inside a separate, controlled orchestration loop.

Best Language for Web Scraping: Python, JavaScript, Go, Java, or C#?

Evelyn Lim — Wed, 21 Oct 2026 08:42:36 +0000

The best language for a scraper is usually the one that makes its hardest stage maintainable. A page of static records needs an HTTP client and a parser. An interactive application needs browser control. A production collection job also needs a place to run, a schema, and an owner when the source changes.

We compare Python, JavaScript, Go, Java, and C# by those responsibilities. A universal speed ranking would hide the browser startup, network waits, parsing choices, and downstream work that often determine the outcome.

Compare Languages by the Work You Need to Do

Language	A useful starting point	Evaluate carefully
Python	Parsing, cleaning, analysis, and data pipelines	Dependency management and the application's concurrency model
JavaScript / TypeScript	Browser automation and Node.js applications	Browser resources, async lifecycle, and typed output boundaries
Go	Services that already use Go and explicit concurrent HTTP work	HTML parsing choices and browser integration requirements
Java	Existing JVM services and application integration	Packaging, process resources, and whether a browser is needed
C#	Existing .NET applications and typed service code	Chosen HTML parser, browser bindings, and deployment runtime

These are workflow fits, rather than language performance scores. If the team already runs a Java service that consumes collected data, moving collection to Python creates an additional runtime and handoff. That may still be worthwhile, but the parsing library alone does not settle the decision.

Start with a representative target and a written output schema. Use identical fields and source pages when comparing implementations. Measure usable records and end-to-end time on your own environment if performance matters.

Python for Parsing and Data Work

Python is a convenient starting point when the output will be cleaned, analyzed, or transformed in Python. Requests handles HTTP; BeautifulSoup handles markup; ordinary dictionaries and JSON make the boundary visible.

Install the packages:

python -m pip install requests==2.34.2 beautifulsoup4==4.15.0

This example reads the regular, server-rendered practice page and produces a small normalized record list:

import json
import requests
from bs4 import BeautifulSoup

response = requests.get("https://quotes.toscrape.com/", timeout=(10, 30))
response.raise_for_status()
soup = BeautifulSoup(response.content, "html.parser")
records = []
for card in soup.select("div.quote"):
    text = card.select_one("span.text")
    author = card.select_one("small.author")
    if text is None or author is None:
        raise ValueError("Required quote field missing")
    records.append({"text": text.get_text(strip=True),
                    "author": author.get_text(strip=True)})
if not records:
    raise ValueError("Expected quote records")
print(json.dumps(records, ensure_ascii=False, indent=2))

The Requests quickstart explains response handling, while the Beautiful Soup documentation covers selectors and parser behavior. Keeping those jobs separate makes a source change easier to diagnose.

Python is not restricted to static HTML. It has browser automation bindings too. The practical question is whether the project benefits from Python's data tooling enough to use it for the browser stage as well.

JavaScript for Browser-Centered Workflows

JavaScript is a natural fit when your application already runs on Node.js or the collection depends on repeated interaction with browser state. TypeScript can add a compile-time contract around extracted records, although runtime validation is still necessary for external data.

Install the browser tooling and its Chromium build:

npm install --save-dev @playwright/test@1.64.0
npx playwright install chromium

Save this as quotes.mjs. It collects the same fields as the Python example, using the JavaScript-rendered practice page:

import { chromium } from '@playwright/test';

const browser = await chromium.launch();
try {
  const page = await browser.newPage();
  await page.goto('https://quotes.toscrape.com/js/');
  await page.locator('div.quote').first().waitFor();
  const records = await page.locator('div.quote').evaluateAll(cards =>
    cards.map(card => {
      const text = card.querySelector('span.text');
      const author = card.querySelector('small.author');
      if (!text || !author) throw new Error('Required quote field missing');
      return { text: text.textContent.trim(), author: author.textContent.trim() };
    })
  );
  if (!records.length) throw new Error('Expected quote records');
  console.log(JSON.stringify(records, null, 2));
} finally {
  await browser.close();
}

The browser waits for a meaningful record locator before reading it. The explicit finally block gives the browser an owner even if extraction fails. Once this grows into an application, put browser and page lifetimes into a small reusable component rather than scattering launches through business logic.

The two examples illustrate acquisition choices. They do not establish a Python-versus-JavaScript benchmark: one downloads static HTML and the other runs a full browser.

Go, Java, and C# in Existing Systems

Go

Go is worth considering when collection is part of a Go service and the team wants to manage HTTP work in that runtime. The standard HTTP client handles transport, and Go's HTML package provides parsing primitives. Decide whether the project needs those lower-level primitives or a selector abstraction before estimating development effort.

Concurrency requires explicit limits on open requests and collected results. A language with inexpensive concurrency does not remove the target's collection boundaries or the memory cost of storing responses. Browser-driven targets add a separate browser resource budget.

Java

For JVM applications, jsoup is a practical HTML parsing option. Its cookbook covers loading documents and finding elements. If a source exposes static HTML or a documented API, collection can fit inside the existing application lifecycle.

A Java scraper that needs client-side rendering still requires a browser-capable layer. Keep the boundary explicit: a parser processes a document; a browser executes the page and maintains interaction state.

C

C# fits teams that already deploy .NET services and want the same runtime for HTTP clients, validation, scheduling, and storage. Choose an HTML parser or browser binding according to the source, then evaluate how it integrates with the application's cancellation and resource ownership.

The strongest reason to choose Java or C# may be operational continuity: the team already knows how to ship, observe, and maintain those services. Reusing that capability can matter more than saving a few lines in the first parsing example.

Use the Same Managed HTTP Contract Across Languages

A managed acquisition service can reduce the importance of the local scraping language. Scrapeless Web Unlocker exposes an HTTP interface; your client sends an authenticated JSON request and receives page content. Each language can keep its own parsing and validation layer.

Prerequisite: the following request requires your own Scrapeless API key. It demonstrates the documented request shape; it is separate from the public practice-site examples above.

import os
import requests

response = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker",
          "proxy": {"country": "ANY"},
          "input": {"url": "https://quotes.toscrape.com/",
                    "response": {"type": "html"}}},
    timeout=(10, 180),
)
response.raise_for_status()
result = response.json()
if result.get("code") != 200 or not isinstance(result.get("data"), str):
    raise ValueError("Expected acquired HTML")
print({"html_characters": len(result["data"])})

For an interactive task, assess Agent Browser's Playwright connection separately. Sharing a transport protocol does not guarantee that every local automation feature behaves identically on a remote browser.

Make the Decision with a Small Implementation

Build one vertical slice: fetch the intended page, extract the required fields, reject an invalid record, and export a usable file. Then include the work needed to schedule and operate it.

Choose Python when its data workflow is the main advantage. Choose JavaScript or TypeScript when browser control and the Node.js application are central. Keep Go, Java, or C# when they fit the surrounding service and meet the target's acquisition requirements. A managed web layer can let that choice follow the application rather than a single target site's rendering behavior.

FAQ

Is Python Always the Fastest Choice?

It can be fast to develop with, but runtime performance depends on the actual workload. Benchmark equivalent acquisition and extraction tasks before making a speed claim.

Does a JavaScript Site Require a JavaScript Scraper?

No. Several languages have browser automation bindings, and a managed service can return rendered content over HTTP.

Should a Scraper Have Its Own Service?

Only when the isolation is useful. Consider deployment ownership, resource use, data contracts, and how the rest of the application consumes the result.

Legal note: collect and reuse only data you are authorized to access. Follow the relevant site terms, data rights, and application policies.

Nodriver in Python: Extract, Paginate, and Export JSON

Evelyn Lim — Wed, 21 Oct 2026 08:28:40 +0000

A browser scraper needs to know when the records exist, how to identify the next page, and what to do with missing fields. Changing the browser library does not remove those decisions.

Nodriver is an asynchronous Python library that controls a Chromium-based browser through the Chrome DevTools Protocol. It does not require Selenium's ChromeDriver process. We use it here for a bounded collection job on a public practice site: render quote cards, follow pagination, and export records with their source pages.

What Nodriver Does

Nodriver can open pages, select elements, interact with them, and inspect the rendered document. Its asynchronous interface lets a coroutine wait for browser operations without turning every step into a fixed delay.

Direct CDP control describes the connection to the browser. It does not prove that a target will accept an automated session. A site's response can also depend on the network, account state, browser environment, and the action being performed. Use a public practice site to establish that your extraction logic works before investigating a target-specific difference.

The official Nodriver documentation covers its API, while the project repository contains implementation and license details. Check dependency and license requirements before including it in a distributed application.

Install and Check Your Runtime

This example uses Nodriver with BeautifulSoup for the parsing stage. Install the packages in an isolated environment:

python -m pip install nodriver==0.50.6 beautifulsoup4==4.15.0

You also need a compatible local Chrome or Chromium installation. Nodriver normally discovers the executable; an application can supply browser_executable_path when it needs an explicit path. Keep that path in deployment configuration rather than assuming every machine uses the same operating system layout.

Use a separate automation profile. Reusing your personal browser profile can mix private sessions, extensions, and state with the collection job. It also makes a supposedly reproducible example depend on whatever happened in your everyday browser.

Launch, Wait, and Extract Record Cards

Save the following as collect_quotes.py. The page bound is intentional: the exercise reads two pages, then closes the browser.

import asyncio
import json
from pathlib import Path
from urllib.parse import urlsplit

import nodriver as uc
from bs4 import BeautifulSoup

START = "https://quotes.toscrape.com/js/"

async def main():
    browser = await uc.start(headless=True)
    records, visited = [], set()
    try:
        tab = await browser.get(START)
        for _ in range(2):
            await tab.select("div.quote", timeout=15)
            current_url = await tab.evaluate("window.location.href")
            if current_url in visited:
                raise ValueError("Pagination returned to an earlier page")
            if urlsplit(current_url).netloc != urlsplit(START).netloc:
                raise ValueError("Unexpected collection host")
            visited.add(current_url)
            soup = BeautifulSoup(await tab.get_content(), "html.parser")
            cards = soup.select("div.quote")
            if not cards:
                raise ValueError("Expected rendered quote cards")
            for card in cards:
                text = card.select_one("span.text")
                author = card.select_one("small.author")
                if text is None or author is None:
                    raise ValueError(f"Incomplete card at {current_url}")
                records.append({
                    "text": text.get_text(" ", strip=True),
                    "author": author.get_text(" ", strip=True),
                    "source_url": current_url,
                })
            if soup.select_one("li.next a") is None:
                break
            if len(visited) < 2:
                next_link = await tab.select("li.next a", timeout=5)
                await next_link.click()
                await tab.wait_for("div.quote", timeout=15)
        Path("nodriver-quotes.json").write_text(
            json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8"
        )
        print(f"Saved {len(records)} records from {len(visited)} pages")
    finally:
        browser.stop()
        await asyncio.sleep(0.5)

if __name__ == "__main__":
    asyncio.run(main())

tab.select() waits for a selector rather than immediately assuming that the page's first document contains records. The scraper then takes one HTML snapshot and parses fields relative to each card. That arrangement makes it easy to save the acquired HTML for diagnosis without changing the field extraction rules.

The short pause after stop() lets scheduled browser cleanup progress before the event loop exits. It is not a content-loading strategy. Page extraction waits on a record selector, which is the meaningful readiness condition for this exercise.

Make Pagination a State Transition

The script locates the next link in the rendered document and clicks the actual element. After navigation, it waits for quote cards and reads the current URL again. That gives the loop a page identity rather than relying on an incremented counter alone.

For a site that updates records without changing the URL, the readiness check must compare another stable property: a page index, a result identifier, or a content change you observed in the application. Do not copy the URL-based duplicate check into that kind of interface and assume it will work.

Avoid a broad text search for “Next” when the page has several controls with that label. A selector scoped to the pagination region gives the action a clearer target. If the site has a documented structured source for its records, inspect that source before building a longer click sequence.

Treat the Output as a Contract

Every exported record has text, author, and source_url. Required fields are checked before a record is appended, and Unicode is preserved when the file is written. Downstream code can therefore distinguish a valid empty optional field from a broken extraction rule.

A production job may need additional fields such as observation time and a source record ID. Add them deliberately. Browser titles and page URLs are useful provenance, but they do not necessarily identify a unique record across category pages or repeat visits.

Keep partial results out of the final export if the collection fails midway and the consumer expects a complete snapshot. An alternative is a separate partial-results file with an explicit status. The consumer should never have to infer completeness from the number of rows.

Decide Who Should Operate the Browser

Local Nodriver gives your application control over the browser process. That also means your deployment owns the executable, profile storage, dependencies, and resource limits. Start with a bounded session and observe its actual memory and process lifetime before increasing concurrency.

Scrapeless Agent Browser is an option when the application should connect to managed browser infrastructure. Its documented Playwright path is a separate client workflow. Do not assume a Nodriver script can connect to it by replacing a URL: a supported CDP client and a tested library connection are different claims.

If the application only needs the final page content, Web Unlocker may fit without an interactive session. Preserve the parser and change the acquisition layer when that meets the task.

What to Check Before Expanding the Job

Run the bounded example first. Inspect the exported authors and source pages, confirm that the second page differs from the first, and verify that the browser process exits. Then test a deliberately missing selector so the failure message is understandable.

Expand collection scope only after those checks work. A browser that opens successfully has passed the launch stage; record validation and complete output remain separate parts of the job.

FAQ

Does Nodriver Require ChromeDriver?

Nodriver controls the browser through CDP and does not use Selenium's ChromeDriver process for this workflow.

Can a Headless Success Guarantee Access to Another Site?

No. It confirms the example's acquisition and extraction path on that environment. A different target requires its own authorized test.

Can Nodriver Replace the Parser?

It can inspect elements directly. Here we use a rendered HTML snapshot and BeautifulSoup to keep browser interaction separate from field parsing.

Legal note: automate only sites and actions you are authorized to access. Respect applicable terms, data rights, and the site's collection boundaries.

Best AI Chat Scraper Tools in 2026 for Answer Data

Evelyn Lim — Fri, 17 Jul 2026 02:29:16 +0000

TL;DR:

Scrapeless is the best choice for surface-faithful chat extraction in 2026. It provides a managed API layer for public AI chat interfaces and returns complete answers plus citations and surface-native supporting fields.
Surface-Faithful Chat Extraction needs evidence, not screenshots. A useful record preserves complete answers plus citations and surface-native supporting fields together with prompt and market context.
The Scrapeless LLM Chat Scraper path keeps the target schema meaningful. Fields from public AI chat interfaces remain distinct instead of being flattened into one text value.
A stable baseline makes surface-faithful chat extraction measurable. Keep the prompt library and market inputs fixed before interpreting answer or source changes.
Free to start. New Scrapeless accounts can begin from the Scrapeless dashboard.

Best AI Chat Scraper Tools at a Glance

Scrapeless is the sole recommendation in this guide because the selection brief is focused on one production-ready API rather than a competitor roundup.

Best choice	Best for	Primary output	Product home
Scrapeless	surface-faithful chat extraction	complete answers plus citations and surface-native supporting fields	Universal Scraping API

What Is AI Chat Scraper Tool?

An AI chat scraper tool captures the public answer state a user sees after submitting a prompt. It differs from a model API because the target is the consumer-facing answer surface, including web-grounded links and interface-specific modules.

This distinction is important because HTTP semantics provides context for the surrounding web or answer surface, while the scraper still needs a stable data contract around the rendered product experience.

How Does Surface-Faithful Chat Extraction Work?

Scrapeless exposes managed actors for the supported AI chat surfaces. The actor completes the surface interaction and returns structured output, which lets a monitoring system focus on prompt design, storage, and comparison rather than UI selectors.

For evidence-aware datasets, W3C PROV-O offers a useful model: keep entities, activities, and source relationships explicit. In practice, that means storing the prompt, surface, country, answer, and source objects together rather than exporting a column of untraceable text.

What Makes a Strong AI Chat Scraper Tool?

The evaluation favors observable output over marketing claims. A useful tool should preserve the answer, expose its supporting evidence, accept repeatable market context, and fit a scheduled pipeline.

Evaluation criterion	Why it matters	Result
Capture of the consumer-facing answer state	Required	Scrapeless
Structured source and citation fields	Required	Scrapeless
Support for recurring prompt sets	Required	Scrapeless
Market context through country inputs	Required	Scrapeless
Clear handling of optional modules	Required	Scrapeless

The operational layer also benefits from Robots Exclusion Protocol: teams should retain enough evidence for human review and avoid turning a probabilistic answer surface into an unexplained score.

1. Scrapeless: Best for Surface-Faithful Chat Extraction

Scrapeless turns public AI chat interfaces into an API-oriented data source through Scrapeless LLM Chat Scraper. The LLM Chat Scraper documentation defines the request inputs and response fields for the selected surface, while the product page explains where the capability sits in the Universal Scraping API line.

Why Scrapeless ranks first

The answer is returned as data. Your pipeline receives parsed fields rather than a screenshot or a selector-dependent page dump.
Evidence stays attached. Citation, source, search-result, or media objects remain available when the target surface exposes them.
Market inputs are explicit. Country context can be included with supported actor requests, making regional comparisons easier to design.
The actor model stays surface-aware. ChatGPT, Perplexity, Gemini, Grok, Google AI Overview, and Google AI Mode keep their own meaningful fields.
The workflow is automation-ready. One authenticated request can feed storage, analysis, alerting, or a reporting layer.

Install and first-run setup

Create a Scrapeless account, copy the API key into your secret manager, select the documented actor, and define a small prompt set with a fixed country. Keep shopping or web-search options off unless the use case needs those extra modules. Review pricing before expanding the schedule.

How you actually use it: prompt your monitoring agent

Give the agent a bounded instruction such as: “Capture this prompt on the selected surface for the US market, store the complete answer and every cited URL, and label missing optional fields as null.” The agent should validate the actor name, submit the request, and write one normalized record without rewriting the answer.

60-second smoke test

Use one public, non-sensitive category prompt. Confirm that the response includes an answer field, preserves the original prompt context, and returns any available source objects as arrays. A smoke test passes when the record can be stored without scraping HTML or guessing field meaning.

Get your API key on the free plan: app.scrapeless.com

What Changes When You Use a Managed Actor for Public AI Chat Interfaces?

The useful comparison is between capture approaches, not vendor names.

Approach	Answer text	Structured sources	Market context	Maintenance burden
Manual copy and paste	Yes	No	Manual	High
Generic browser script	Yes	Custom parsing	Custom	High
Scrapeless managed actor	Yes	Yes, when exposed by the surface	Request input	Low at the integration layer

Selection Checklist for Surface-Faithful Chat Extraction

The right AI chat scraper tool should treat the final answer and its evidence as one record. Scrapeless is the strongest fit when you need repeatable captures across several chat surfaces without maintaining a browser workflow for each one.

Before committing, test three prompt shapes: a factual question, a category recommendation, and a location-sensitive query. Inspect whether citations, related prompts, media, products, or empty states are represented honestly. Do not accept a single opaque visibility score as a substitute for raw evidence.

Common Use Cases for AI Chat Scraper Tools

Archive research answers. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.
Watch how a category is described. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.
Compare localized recommendations. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.
Detect source changes. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.
Prepare prompt–answer datasets. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.

The Scrapeless LLM scraper overview shows how the actor family fits a broader answer-capture program without requiring a separate browser integration for every surface.

Why Is Surface-Faithful Chat Extraction Hard?

Chat interfaces assemble responses over time and often render supporting material separately from the main answer. DOM-only extraction can merge unrelated panels, drop late-arriving citations, or confuse navigation links with cited sources.

The safest design treats optional fields as nullable, stores the unmodified answer, and separates collection from interpretation. That keeps a parser change from silently rewriting historical results.

Conclusion

Public AI chat pages are products, not simple text endpoints. Their answers can include sources, shopping modules, media, follow-up prompts, and location-sensitive details. Scrapeless is the best API foundation for this work because it captures the answer surface as structured, source-aware data and leaves the scoring logic under your control.

Start with a small prompt library, pin the market context, retain raw evidence, and expand only after the records remain comparable across scheduled runs.

Ready to Build Your AI-Answer Data Pipeline?

Join developers building answer-monitoring and GEO pipelines in the Scrapeless community: Discord · Telegram.

Sign up at app.scrapeless.com, review Scrapeless pricing, and turn a fixed prompt set into structured records your team can audit.

FAQ

Q: Why is Scrapeless the best option in this guide?

Scrapeless is the best option because it provides dedicated managed actors for supported AI-answer surfaces and returns structured answer and evidence fields suitable for automation.

Q: Is an AI chat scraper the same as a model API?

No. An AI chat scraper captures the public chat product's rendered answer and supporting modules, while a model API returns data from a developer endpoint under its own contract.

Q: Can this workflow support regional comparisons?

Yes. Use the documented country or location inputs for the selected actor, keep the prompt fixed, and store the market context beside every response.

Q: Should monitoring use a single prompt?

No. Use a controlled library that covers factual, category, comparison, and location-sensitive intent, then keep that library stable long enough to establish a baseline.

Q: Is it acceptable to collect public AI answers?

Collection rules vary by jurisdiction and platform. Limit the workflow to public data, review applicable terms and policies, minimize retained personal data, and obtain legal advice for regulated use cases.

Best AI Competitor Analysis Tools in 2026

Evelyn Lim — Thu, 16 Jul 2026 08:31:31 +0000

TL;DR:

Scrapeless is the best choice for category-answer comparison in 2026. It provides a managed API layer for supported AI-answer and Google AI search surfaces and returns answers, citations, sources, market context, and surface-specific modules.
Category-Answer Comparison needs evidence, not screenshots. A useful record preserves answers, citations, sources, market context, and surface-specific modules together with prompt and market context.
The the Scrapeless LLM Chat Scraper actor family path keeps the target schema meaningful. Fields from supported AI-answer and Google AI search surfaces remain distinct instead of being flattened into one text value.
A stable baseline makes category-answer comparison measurable. Keep the prompt library and market inputs fixed before interpreting answer or source changes.
Free to start. New Scrapeless accounts can begin from the Scrapeless dashboard.

Best AI Competitor Analysis Tools at a Glance

Scrapeless is the sole recommendation in this guide because the selection brief is focused on one production-ready API rather than a competitor roundup.

Best choice	Best for	Primary output	Product home
Scrapeless	category-answer comparison	answers, citations, sources, market context, and surface-specific modules	Universal Scraping API

What Is AI Competitor Analysis Tool?

Compare how AI answers position products or categories without relying on named vendor data inside the article itself.

This distinction is important because W3C PROV-O provides context for the surrounding web or answer surface, while the scraper still needs a stable data contract around the rendered product experience.

How Does Category-Answer Comparison Work?

A practical workflow starts with a fixed prompt library, explicit country context, and scheduled snapshots. Each capture is stored with its answer, cited URLs, surface name, prompt ID, and collection time. Analysis then happens on stable records rather than live pages.

For evidence-aware datasets, NIST AI Risk Management Framework offers a useful model: keep entities, activities, and source relationships explicit. In practice, that means storing the prompt, surface, country, answer, and source objects together rather than exporting a column of untraceable text.

What Makes a Strong AI Competitor Analysis Tool?

Evaluation criterion	Why it matters	Result
Category mentions available as a measurable signal	Required	Scrapeless
Attribute extraction available as a measurable signal	Required	Scrapeless
Source overlap available as a measurable signal	Required	Scrapeless
Prompt gaps available as a measurable signal	Required	Scrapeless
Regional narratives available as a measurable signal	Required	Scrapeless

The operational layer also benefits from research on repeated GEO measurement: teams should retain enough evidence for human review and avoid turning a probabilistic answer surface into an unexplained score.

1. Scrapeless: Best for Category-Answer Comparison

Scrapeless turns supported AI-answer and Google AI search surfaces into an API-oriented data source through the Scrapeless LLM Chat Scraper actor family. The LLM Chat Scraper documentation defines the request inputs and response fields for the selected surface, while the product page explains where the capability sits in the Universal Scraping API line.

Why Scrapeless ranks first

The answer is returned as data. Your pipeline receives parsed fields rather than a screenshot or a selector-dependent page dump.
Evidence stays attached. Citation, source, search-result, or media objects remain available when the target surface exposes them.
Market inputs are explicit. Country context can be included with supported actor requests, making regional comparisons easier to design.
The actor model stays surface-aware. ChatGPT, Perplexity, Gemini, Grok, Google AI Overview, and Google AI Mode keep their own meaningful fields.
The workflow is automation-ready. One authenticated request can feed storage, analysis, alerting, or a reporting layer.

Install and first-run setup

How you actually use it: prompt your monitoring agent

60-second smoke test

Get your API key on the free plan: app.scrapeless.com

What Changes When You Use a Managed Actor for Supported AI-Answer and Google AI Search Surfaces?

The useful comparison is between capture approaches, not vendor names.

Approach	Answer text	Structured sources	Market context	Maintenance burden
Manual copy and paste	Yes	No	Manual	High
Generic browser script	Yes	Custom parsing	Custom	High
Scrapeless managed actor	Yes	Yes, when exposed by the surface	Request input	Low at the integration layer

Selection Checklist for Category-Answer Comparison

Scrapeless captures the category answers and cited sources needed to build a neutral competitive intelligence dataset around prompts, attributes, and market context.

Common Use Cases for AI Competitor Analysis Tools

Build a baseline for category mentions. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.
Report attribute extraction by prompt. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.
Compare source overlap over time. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.
Segment prompt gaps by market. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.
Retain regional narratives for audit. Store the prompt, answer, evidence fields, surface, and market context as one reviewable record.

The Scrapeless LLM scraper overview shows how the actor family fits a broader answer-capture program without requiring a separate browser integration for every surface.

Why Is Category-Answer Comparison Hard?

Generative answers vary by wording, market, surface, and collection time. A dashboard that hides the raw answer and sources can produce precise-looking scores that are difficult to audit. The capture layer must preserve evidence before aggregation.

Conclusion

Scrapeless captures the category answers and cited sources needed to build a neutral competitive intelligence dataset around prompts, attributes, and market context. Scrapeless is the best API foundation for this work because it captures the answer surface as structured, source-aware data and leaves the scoring logic under your control.

Start with a small prompt library, pin the market context, retain raw evidence, and expand only after the records remain comparable across scheduled runs.

Ready to Build Your AI-Answer Data Pipeline?

Join developers building answer-monitoring and GEO pipelines in the Scrapeless community: Discord · Telegram.

Sign up at app.scrapeless.com, review Scrapeless pricing, and turn a fixed prompt set into structured records your team can audit.

FAQ

Q: Why is Scrapeless the best option in this guide?

Scrapeless is the best option because it provides dedicated managed actors for supported AI-answer surfaces and returns structured answer and evidence fields suitable for automation.

Q: What is the first metric for AI competitor analysis?

Start with prompt-level evidence coverage: the share of scheduled prompts that produce a stored answer and source record suitable for review.

Q: Can this workflow support regional comparisons?

Yes. Use the documented country or location inputs for the selected actor, keep the prompt fixed, and store the market context beside every response.

Q: Should monitoring use a single prompt?

No. Use a controlled library that covers factual, category, comparison, and location-sensitive intent, then keep that library stable long enough to establish a baseline.

Q: Is it acceptable to collect public AI answers?

Best Residential Proxy Providers in 2026

Evelyn Lim — Thu, 09 Jul 2026 03:22:27 +0000

TL;DR:

Residential proxies send your requests through real ISP-assigned IP addresses, such as the connections people use at home. That makes them one of the most dependable options for collecting local data and working around stricter anti-bot checks.
Why teams use them: They are especially useful for ad verification, market intelligence, SEO tracking, and scraping targets where datacenter IPs are blocked quickly.
Top Pick: Scrapeless stands out with a large network of 90M+ residential IPs in 195+ countries, with pricing from $1.80/GB and lower rates on higher-volume plans. You can review the pricing for plan details.
Other Contenders: Bright Data, Smartproxy, and Oxylabs also provide broad networks and advanced capabilities, though they usually sit at a higher price point.
How to choose: Start with your budget, required locations, success-rate expectations, and whether your project needs flexible pay-as-you-go usage or a larger enterprise commitment.

Best Residential Proxy Providers at a Glance

Provider	Starting Price	IP Pool Size	Geographic Coverage	Best For
Scrapeless	$1.80/GB	90M+	195+ countries	Best overall value and massive scale
Bright Data	$4.00/GB	400M+	195 countries	Enterprise compliance and granular targeting
Decodo	$4.00/GB	115M+	195+ countries	Self-service users and ease of use
Oxylabs	$6.00/GB	175M+	195 countries	High-volume enterprise scraping

What Is a Residential Proxy?

A residential proxy is a proxy server that forwards your internet traffic through an IP address assigned by a standard Internet Service Provider (ISP) to a real household connection. For a technical view of how IP addresses are assigned and managed globally, see the IANA IP Address Allocation documentation. This is different from datacenter proxies (see also what is a proxy browser), which usually come from cloud hosting platforms such as AWS or DigitalOcean and are much easier for websites to identify.

In practice, a residential proxy makes your connection look like it is coming from an ordinary user device: a laptop in London, a phone in Tokyo, or a desktop in New York. Because that traffic carries a higher trust signal than many server-hosted IPs, residential proxies are widely used for web scraping workflows that need to reduce CAPTCHAs, blocks, and IP bans.

How Do Residential Proxies Work?

A residential proxy network works by routing requests through a large pool of opted-in devices around the world. The basic request path usually looks like this:

The Request: Your scraping script or browser sends a request to the proxy provider's gateway server.
The Routing: The gateway authenticates your request and chooses an available residential IP from the pool according to your targeting rules, such as country, state, or city.
The Forwarding: The gateway sends the request to the destination website through the selected residential device.
The Response: The website treats the request as traffic from a legitimate local user, returns the page or data, and the response travels back through the residential device and gateway to your client.

This setup also enables IP rotation. Depending on your configuration, the gateway can assign a new IP for each request or maintain the same address for a sticky session. That distribution helps scraping traffic avoid repeatedly hitting a target from the same IP address.

How We Evaluated These Tools

To compare the best residential proxy providers for 2026, the evaluation focused on the factors that matter most in real-world extraction projects:

Network Size and Diversity: The total number of available IPs and the depth of location coverage, including countries, cities, and ASNs. Larger, more diverse pools reduce repeated IP usage and can lower ban rates.
Pricing and Flexibility: Cost per GB, availability of pay-as-you-go plans, monthly commitment requirements, and overall value for the level of infrastructure provided.
Speed and Uptime: Success rates, response times, and network reliability during concurrent scraping or data collection tasks.
Targeting Capabilities: The ability to select specific countries, states, or cities without running into unnecessary premium charges.
Developer Experience: Documentation quality, dashboard clarity, credential management, and how easily the proxies fit into an existing scraping stack.

The Best Residential Proxy Providers: Ranked

1. Scrapeless: Best overall value and massive scale

Scrapeless ranks first for residential proxies in 2026 because it combines a very large IP network with low usage-based pricing. It is built for developers and data teams that need dependable extraction infrastructure without the cost profile often attached to enterprise proxy products. Its residential network is ethically sourced and globally distributed.

With more than 90 million IPs across 195+ countries, Scrapeless gives teams the coverage needed to access localized content around the world. It supports country, state, and city-level targeting, which makes it practical for workflows such as local SEO monitoring, regional market research, ad verification, and global pricing intelligence. The pricing model is also a major advantage: plans start at $1.80/GB on Basic and can drop to $1.44/GB on the Business plan, giving teams enterprise-grade proxy infrastructure at a more accessible cost.

Key Features:

Massive Pool: 90M+ ethically sourced residential IPs.
Global Reach: Coverage in 195+ countries with granular targeting.
High Performance: 99.9% uptime and exceptional success rates.
Low Pricing: Starting from $1.80/GB, dropping to $1.44/GB on higher-tier plans.

Install and Configure

Adding Scrapeless residential proxies to your pipeline is simple. Start by creating or retrieving your proxy credentials from the Scrapeless dashboard.

# Set your Scrapeless proxy credentials
export PROXY_HOST="proxy.scrapeless.com"
export PROXY_PORT="8000"
export PROXY_USER="your_username"
export PROXY_PASS="your_password"

How you actually use it: prompt your agent

If your workflow uses an AI agent to collect localized data, you can tell the agent to send requests through the Scrapeless residential proxy network.

"Fetch the search results for 'best running shoes' on Google UK. Route the request through a UK residential proxy using the credentials provided in the environment variables to ensure we see the localized results."

60-second smoke test

You can confirm that the proxy connection and location targeting are working with a short curl test that checks the visible IP and geolocation.

# Test the proxy connection and verify the IP location
curl -x http://$PROXY_USER:$PROXY_PASS@$PROXY_HOST:$PROXY_PORT https://ipinfo.io/json

{
  "ip": "82.132.21.14",
  "city": "London",
  "region": "England",
  "country": "GB",
  "loc": "51.5085,-0.1257",
  "org": "AS13285 Hutchison 3G UK Limited",
  "postal": "WC2N",
  "timezone": "Europe/London"
}

2. Bright Data: Best for enterprise compliance and granular targeting

Bright Data is one of the long-running names in the proxy market, with a reputation for broad infrastructure and strict compliance controls. Its residential proxy network includes more than 72 million IPs and is designed for larger organizations that need mature operational processes.

Its targeting options are a major strength. Users can narrow traffic by location and, for some use cases, by ASN or carrier. Bright Data also offers supporting tools such as Proxy Manager and Web Unlocker, which can reduce the amount of custom proxy-handling logic teams need to build themselves. Pricing is higher, starting around $8.40/GB, but the platform can fit organizations with strict compliance expectations, procurement processes, and enterprise budgets.

3. Smartproxy: Best for self-service users and ease of use

Smartproxy is known for being approachable and easy to set up. With more than 55 million residential IPs, it delivers consistent coverage across 195+ locations and is often a practical fit for self-service teams.

The product experience is its main appeal: a clear dashboard, strong documentation, and responsive support make it easier for developers to start quickly. Smartproxy offers multiple plan types, including pay-as-you-go options starting at $7.00/GB, so usage can scale as projects grow. Browser extensions and a proxy generator also help with quick configuration and smaller scraping tasks.

4. Oxylabs: Best for high-volume enterprise scraping

Oxylabs is another major provider in the residential proxy category, with a pool of more than 100 million residential IPs. It is positioned for high-volume enterprise data extraction where performance, uptime, and support are central requirements.

The platform includes features such as AI-powered proxy rotation and dedicated account management for large scraping operations. Its network is built for stability, which matters for mission-critical pipelines that need predictable performance. Pricing starts at $8.00/GB, reflecting the enterprise focus, but the infrastructure and support model can make sense for teams running extraction at significant scale.

Side-by-Side Comparison

Feature	Scrapeless	Bright Data	Smartproxy	Oxylabs
Starting Price	$1.80/GB	$8.40/GB	$7.00/GB	$8.00/GB
IP Pool Size	90M+	72M+	55M+	100M+
Locations	195+ countries	195 countries	195+ countries	195 countries
Targeting	Country, State, City	Country, State, City, ASN	Country, State, City	Country, State, City, ASN
Pay-as-you-go	Yes	Yes	Yes	Yes
Protocol Support	HTTP(S ), SOCKS5	HTTP(S), SOCKS5	HTTP(S), SOCKS5	HTTP(S), SOCKS5

Get your API key on the free plan: app.scrapeless.com

How Do You Pick the Right Tool?

The right residential proxy provider depends on the requirements of your project, not just the size of the advertised network. Use these factors to narrow the choice:

Budget: If cost efficiency is the main concern, Scrapeless offers competitive pricing at $1.80/GB, with rates as low as $1.44/GB on higher-tier plans, while still providing broad network scale and strong performance.
Scale: For large global scraping workloads, providers with the biggest IP pools, including Scrapeless and Oxylabs, can help reduce repeated IP usage and support higher concurrency.
Compliance: If your organization needs documented compliance workflows, enterprise procurement, and dedicated account management, Bright Data's enterprise-oriented model may be appropriate.
Ease of Use: For teams that want fast setup, an accessible dashboard, and straightforward documentation, Smartproxy offers a smooth developer experience.

Common Use Cases for Residential Proxies

Residential proxies support many data workflows where local access and IP trust are important:

Market Research and Pricing Intelligence: E-commerce teams use residential proxies to check competitor prices, inventory, and regional availability without being blocked or shown inaccurate localized pricing.
Ad Verification: Advertisers route requests through residential IPs to confirm that ads appear correctly in target regions and to detect suspicious click behavior.
SEO Monitoring: SEO teams track keyword rankings from specific cities and countries so they can see the same results a local searcher would see.
Social Media Management: Managing multiple accounts often requires distinct, high-trust IPs to reduce the chance of scripted bans, forced verification, or shadowbans.
Travel Fare Aggregation: Travel platforms collect airline and hotel pricing across regions, where rates can shift based on the user's location.

Why Is Residential Proxy Scraping Hard?

Residential proxies provide strong trust signals, but using them well still requires careful engineering:

IP Churn: Residential IPs are tied to real devices, so they can disappear when a user turns off a phone, closes a laptop, or restarts a router. Your scraping system needs to recover cleanly from sudden connection drops.
Geographic Precision: City or state targeting depends on the proxy gateway and IP geolocation data, which is not always perfect. Good tooling is needed to match the requested location as closely as possible.
Bandwidth Costs: Residential proxies are commonly billed by bandwidth. Heavy pages with large images, videos, or unnecessary assets can raise costs quickly, so blocking nonessential resources is important.
Protocol Handling: Modern anti-bot systems evaluate more than the IP address. TLS handshakes, HTTP/2 behavior, headers, and browser fingerprints can all affect trust. A residential IP helps, but the request should also look like it came from a real browser. For background, see MDN proxy documentation.

Conclusion

Residential proxies remain one of the strongest options for accessing localized data and reducing blocks from stricter anti-bot systems. In 2026, the market is shaped by the balance between network scale, performance, targeting depth, and cost.

Scrapeless is the leading choice in this comparison because it brings together a 90M+ residential IP pool, coverage in 195+ countries, granular targeting, and pricing from $1.80/GB. For teams building pricing pipelines, SEO monitors, ad verification systems, or AI data workflows, it provides reliable infrastructure for collecting public web data at scale.

Ready to Build Your AI-Powered Data Pipeline?

You can create an account at app.scrapeless.com and adapt the proxy patterns above to the public pages your pipeline needs to access.

FAQ

Is scraping with residential proxies legal?

Scraping publicly available data is generally legal, but rules vary by jurisdiction and by target website. Always review the website's Terms of Service and consult legal counsel to confirm compliance with local requirements.

Is a proxy necessary for web scraping?

For serious scraping workloads, yes. Many websites use rate limits and IP-based blocking to control automated access. Proxies distribute traffic across multiple IPs so data collection can continue with fewer interruptions.

How should WAF or Access Denied errors be handled?

When a Web Application Firewall (WAF) blocks your requests, use a residential proxy from the expected region of the target site. For more context on how WAFs work, see the OWASP WAF guidelines. It also helps to warm the session by loading the homepage before moving to the target page, which creates a more natural browsing pattern.

What is the best approach if the website's DOM changes?

Websites update markup frequently, which can break selectors and extraction logic. The W3C DOM specification explains how document structures are defined. Monitor scraping pipelines regularly and update selectors when the DOM changes.

How many concurrent requests are recommended?

To maintain higher success rates and avoid triggering anti-bot rules, keep concurrency low. A typical recommendation is no more than ≤3 workers per host for parallel runs.

Can residential proxies be used without an AI agent?

Yes. Residential proxies work with ordinary HTTP clients and scraping libraries such as Python's requests or Node.js axios. AI agents simply add orchestration and data extraction logic on top of the proxy connection.