DEV Community: ZyVOP

What Is Agentic AI? The Ultimate 2026 Guide to Autonomous AI Agents

ZyVOP — Sat, 05 Sep 2026 05:54:52 +0000

For most of the generative AI boom, the interaction model was straightforward: give a model a prompt and get a response.

That works well for writing, summarization, search, coding assistance, and analysis. But the model usually stops when the response ends. If the job requires querying another system, updating a record, checking the result, or deciding what happens next, someone—or some other software—has to continue the workflow.

AI agents extend that model.

Instead of asking an LLM for one answer, software gives it an objective, access to specific tools, working state, and boundaries around what it can do. The system can choose an action, execute it, inspect the result, and continue.

That's the core idea behind agentic AI.

An AI agent is a software component capable of working toward an objective and taking actions using available tools. Agentic AI is the broader approach of building systems around one or more agents so they can handle parts of a workflow with some autonomy.

The terminology has already become loose enough that Gartner uses “agentwashing” for products marketed as agents without substantial agentic capabilities. Gartner predicts that up to 40% of enterprise applications will include task-specific AI agents by the end of 2026, compared with less than 5% in 2025.

The IEEE Global Survey on Technology Impact 2026 found that 96% of surveyed technologists agreed that innovation, exploration, and adoption of agentic AI would continue accelerating in 2026.

Interest is real. That doesn't make agents the right abstraction for every problem.

Sometimes a queue worker and 50 lines of TypeScript are still the better solution.

How Does Agentic AI Work?

A useful agent isn't simply an LLM with a large system prompt.

Most production designs have some version of this loop:

The model interprets context and proposes what should happen next. Tools give it controlled access to the outside world: APIs, databases, browsers, code execution, CRMs, documentation, or other services.

State preserves what happened between steps. Without it, a long-running agent can't reliably resume after a failure or know which actions have already completed.

Validation checks model decisions against things the application can determine more reliably.

If your rule is:

if invoice.total > purchaseOrder.remainingBalance:
    reject()

put it in code. An LLM doesn't need to reason about a deterministic business rule.

There are two other details that become important quickly in production.

Actions need idempotency

Imagine an agent calls a refund API and the request times out.

The refund may have failed—or it may have succeeded and only the response was lost.

Blindly retrying the operation could refund the customer twice.

Write operations such as payments, refunds, emails, and record creation should use idempotency keys or another deduplication mechanism so retries don't repeat completed actions.

Agent loops need limits

The Finished? → No → Planner path can't continue forever.

Production workflows need boundaries such as maximum steps, retry limits, cost or token budgets, wall-clock timeouts, and repeated-action detection.

Autonomy without termination conditions is just an expensive infinite loop.

A Refund Agent in Practice

Suppose a customer writes:

I returned my headphones last week. Why haven't I received my refund?

You could send that message to an LLM and ask whether the customer deserves a refund.

A production system should separate interpretation from execution.

AI handles the parts involving language and contextual interpretation. Authentication, order ownership, monetary limits, execution, and persistence remain controlled by application code.

The amount of autonomy can change over time. A team might initially require approval for every refund, then automate low-risk cases after collecting enough evidence that the workflow behaves reliably.

Agentic AI vs. Generative AI

Agentic AI usually uses generative models internally. The difference is what the surrounding system allows them to do.

	Generative AI	Agentic AI
Input	Prompt	Objective
Result	Content or answer	Progress toward an outcome
Actions	Usually none	Can execute tools
State	Mostly conversational	Often persistent
Human role	Directs tasks	Defines goals and boundaries
Example	Write an email	Research a prospect and prepare outreach

Ask an LLM to write an email and you have generative AI.

Let software retrieve the prospect, research the company, draft the message, validate required information, and place it into an approval queue, and you have an agentic workflow.

Agentic AI vs. Traditional Automation

Traditional automation works best when the possible states and actions can be modeled in advance:

payment received → validate amount → update invoice → send receipt

An agent adds little value there.

Agents become useful when deciding what happens next requires interpreting ambiguous context.

A customer may describe the same support problem in dozens of ways. An invoice may arrive in an unexpected format. Research may require comparing conflicting information.

A practical system often combines both approaches: models interpret messy inputs, while conventional software handles known rules and execution.

Gartner similarly recommends using agents where decisions are required, traditional automation for routine workflows, and assistants for straightforward retrieval.

How Reliable Are AI Agents?

Agent capability isn't binary.

An agent may handle a small bug fix reliably and fail on a repository-wide migration involving many dependent decisions.

METR's Task-Completion Time Horizons provides a useful way to measure this.

Rather than asking whether an agent “can code,” METR measures how agent success changes as tasks become more difficult, using the time human experts require to complete those tasks as a measure of difficulty.

Their evaluations show rapid improvement among frontier models, but reliability still falls as task complexity increases. METR also cautions that current estimates above roughly 16 hours of human task duration have substantial uncertainty.

This matters because a 20-step autonomous workflow creates more opportunities for a bad assumption to propagate than a single bounded task.

Breaking work into smaller, verifiable stages is therefore useful even as models improve.

Multi-Agent Systems

Complex workflows can also be split among specialized agents.

Different agents can use different models, tools, permissions, and context. Independent work can run in parallel.

But every arrow is another interface to design, observe, retry, and pay for.

A five-agent architecture isn't automatically more capable than one agent with five well-defined tools. Use multiple agents when specialization or parallelism provides a measurable advantage.

Where AI Agents Are Useful

Software development

Coding works well because agents can receive immediate feedback.

An agent edits code, runs tests, reads failures, modifies the implementation, and tries again.

That creates a natural execution loop.

Microsoft's 2026 AI trends report discusses GitHub's direction toward repository intelligence, where AI works with relationships and history across a codebase rather than isolated snippets.

Tests, type systems, linters, CI, and code review also give developers ways to verify agent-generated work.

Customer support

Support combines messy natural language with structured operations.

An agent can interpret a request, retrieve the account and order, check relevant policies, and prepare a resolution. Low-risk actions might execute automatically while higher-risk actions require approval.

Internal operations

Many valuable workflows are much less exciting.

An employee downloads a CSV every Monday, combines it with two internal systems, cleans several columns, calculates metrics, and sends a report.

If that consumes three hours every week, automation doesn't need to look futuristic to be valuable.

Zapier says 97% of its employees use AI daily and that more than 3,000 internal agents have completed over 3 million tasks.

Recruiting

High-volume hiring involves screening, scheduling, reminders, document collection, status changes, and onboarding.

Fountain reported reducing hiring time from roughly two weeks to 22 minutes for a major delivery client.

The opportunity isn't necessarily automating the final hiring decision. Much of the work surrounding that decision can be automated.

Agents Are Changing How Software Uses the Web

Agents increasingly interact with interfaces originally designed for humans.

A shopping agent, for example, might need to identify a product, price, currency, size, inventory, delivery date, and return conditions across several retailers.

That requires information to be machine-readable and consistent.

Cloudflare's 2026 Agentic Internet report reported automated systems accounting for roughly 57.5% of requests for HTML content observed across its network during the measurement period.

That doesn't mean bots represent 57.5% of Internet users or all Internet traffic.

The useful implication is that semantic HTML, structured data, stable APIs, documentation, and reliable product feeds become increasingly valuable when software consumes information on users' behalf.

commercetools' 2026 guidance on AI-ready product data makes a similar point for commerce: agentic shopping depends on structured, current product information that machines can reliably discover and use.

Where Agent Projects Fail

Three problems appear repeatedly: compounding errors, hidden cost, and excessive permissions.

Suppose a research agent identifies the wrong company because two businesses share a similar name. Every later step can execute perfectly and still produce the wrong result.

Validation should happen as close as possible to the point where an error becomes detectable.

Costs can also grow unexpectedly. One user request may trigger planning, retrieval, multiple tool calls, validation, retries, and final generation.

Measure cost per successful workflow, not token price.

Permissions are even more important.

A model with read-only access can produce an incorrect answer. The same model with broad write access can create an incident.

Use least privilege, narrow tool scopes, and explicit approval for consequential actions.

Prompt Injection Becomes a Security Problem

Agents often consume information they don't control: webpages, emails, uploaded documents, support tickets, and code repositories.

That content can contain instructions intended to manipulate the model.

NIST's 2026 AI-agent security research describes this as agent hijacking, or indirect prompt injection.

A malicious webpage becomes much more dangerous when the model reading it can also access internal tools.

External content should remain untrusted input. Sensitive actions need authorization and policy enforcement outside the model.

Before Building an Agent

Start with the workflow, not the framework.

Ask five questions:

Is the workflow expensive enough to automate? Saving hundreds of hours per month is interesting. Saving five minutes twice a month probably isn't.
Can the result be verified? Tests, schemas, database constraints, business rules, and human approval provide useful feedback.
Which decisions actually require AI? Keep deterministic rules in code.
What happens when the agent is wrong? Decide which actions can be reversed, which require approval, and which tools shouldn't be available at all.
What does a successful run cost? Include models, APIs, infrastructure, retries, monitoring, and human review.

Gartner predicts that more than 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls.

Gartner — Agentic AI Project Forecast

The technology can be useful while individual projects still fail.

Frequently Asked Questions

Is agentic AI the same as AGI?

No. Agentic AI describes systems that can work toward objectives and take actions with some autonomy. AGI refers to the much broader idea of general intelligence across domains.

A refund agent can be agentic without being remotely close to AGI.

Is every AI assistant an agent?

No. An assistant may generate content or retrieve information while relying on a user to direct each step.

The term agent becomes more useful when software maintains state, selects actions, uses tools, observes results, and continues toward an objective.

Can AI agents operate without humans?

Yes, for bounded workflows.

Whether they should depends on the consequence of failure. Classifying a document and transferring money require very different levels of oversight.

Autonomy should be assigned at the action level rather than treated as one switch for the entire system.

How much does an AI agent cost?

There is no standard cost.

Measure the full cost of a successfully completed workflow, including model calls, external APIs, infrastructure, retries, monitoring, and human review.

The Best Agent May Not Look Like an Agent

The most useful agentic systems may end up being fairly ordinary.

A support request arrives, relevant information is collected, routine checks run, and an employee receives a prepared resolution.

A developer assigns a bug, software investigates the repository and proposes a tested patch.

A three-hour Monday reporting process runs automatically and leaves behind the data and validation results needed to check it.

Agentic AI becomes useful when software can handle workflows that were too ambiguous for traditional automation but structured enough to operate within clear boundaries.

LLMs made more of those workflows possible.

They didn't remove the need for authentication, permissions, state, idempotency, validation, retries, observability, security, and failure handling.

The difficult question isn't how many agents you can deploy.

It's where giving software more decision-making authority creates enough value to justify the complexity.

And sometimes the right solution will still be a cron job and 50 lines of TypeScript.

That's probably a healthy sign.

Sources & Further Reading

IEEE — Global Survey: The Impact of Technology in 2026 IEEE Global Survey on Technology Impact 2026

Gartner — Enterprise AI Agent Forecast Gartner Predicts 40% of Enterprise Applications Will Feature Task-Specific AI Agents by 2026

Gartner — Agentic AI Project Forecast Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027

METR — Task-Completion Time Horizons METR Task-Completion Time Horizons

NIST — AI Agent Security NIST Research on AI Agent Security and Agent Hijacking

Cloudflare — Agentic Internet Report Cloudflare Agentic Internet Report

Microsoft — AI Trends for 2026 Microsoft — What's Next in AI: 7 Trends to Watch in 2026

Zapier — AI Transformation Zapier AI Transformation

Fountain — Frontline Hiring Fountain Agentic-First Frontline Operating System Announcement

commercetools — AI-Ready Product Data How to Build AI-Ready Product Data for Agentic Commerce

Last reviewed: July 2026.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Build a Web Scraping API with FastAPI, Celery & Redis (2026)

ZyVOP — Fri, 04 Sep 2026 06:42:33 +0000

Why Your Scraper Needs an API Layer

At some point every scraper outgrows its original script form. Maybe you want to trigger scraping from a dashboard. Maybe a frontend team needs to query scraped data. Maybe you're building a product — a rank tracker, a price monitor, a lead generation tool — and users need to submit URLs and get results back.

The moment your scraper needs to serve multiple callers, run on a schedule, handle concurrent requests, or return results asynchronously, you need to wrap it in a proper API.

In this practical approach, we'll build a scraper wrapped in a web API powered by FastAPI to scrape and deliver data on demand — implementing FastAPI web services with asynchronous request handling, configuring real-time data scraping with caching and webhook support, and handling concurrent requests with rate limiting for scalable scraping API endpoints.

The stack we'll build is a proven pattern:

Client → FastAPI (HTTP layer) → Redis (task queue) → Celery Workers (scraping)
                                      ↓
                              PostgreSQL / SQLite (results storage)

FastAPI handles HTTP requests and returns immediate job IDs. Redis acts as the message broker. Celery workers run the actual scraping in the background. Results are stored in a database and fetched when the client polls.

This architecture means: your API never blocks, workers can scale horizontally, failed jobs retry automatically, and your scraping logic is completely decoupled from your HTTP layer.

Project Structure

scraping_api/
├── app/
│   ├── __init__.py
│   ├── main.py          # FastAPI application
│   ├── models.py        # SQLAlchemy models
│   ├── schemas.py       # Pydantic request/response schemas
│   ├── database.py      # DB connection
│   ├── tasks.py         # Celery scraping tasks
│   ├── scrapers/
│   │   ├── __init__.py
│   │   ├── generic.py   # Generic HTTP scraper
│   │   └── browser.py   # Playwright scraper
│   └── middleware/
│       ├── ratelimit.py  # Rate limiting
│       └── auth.py       # API key auth
├── worker.py            # Celery worker entrypoint
├── requirements.txt
├── docker-compose.yml
└── .env

Step 1: Install Dependencies

pip install fastapi uvicorn celery redis httpx \
            beautifulsoup4 sqlalchemy aiosqlite \
            python-dotenv pydantic slowapi

# requirements.txt
fastapi==0.115.0
uvicorn[standard]==0.30.0
celery[redis]==5.4.0
redis==5.0.1
httpx==0.27.0
beautifulsoup4==4.12.3
lxml==5.2.2
sqlalchemy==2.0.30
aiosqlite==0.20.0
python-dotenv==1.0.1
pydantic==2.7.0
slowapi==0.1.9
flower==2.0.1

Step 2: Database Models

# app/models.py
from sqlalchemy import Column, String, Float, Text, DateTime, Integer, Enum
from sqlalchemy.orm import DeclarativeBase
from datetime import datetime, timezone
import enum, uuid

class Base(DeclarativeBase):
    pass

class JobStatus(str, enum.Enum):
    PENDING   = "pending"
    RUNNING   = "running"
    SUCCESS   = "success"
    FAILED    = "failed"
    RETRYING  = "retrying"

class ScrapeJob(Base):
    __tablename__ = "scrape_jobs"

    id           = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
    url          = Column(String, nullable=False)
    scraper_type = Column(String, default="generic")   # "generic" | "browser"
    status       = Column(String, default=JobStatus.PENDING)
    created_at   = Column(DateTime, default=lambda: datetime.now(timezone.utc))
    started_at   = Column(DateTime, nullable=True)
    completed_at = Column(DateTime, nullable=True)
    api_key      = Column(String, nullable=True)
    error        = Column(Text, nullable=True)
    retry_count  = Column(Integer, default=0)

class ScrapeResult(Base):
    __tablename__ = "scrape_results"

    id         = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
    job_id     = Column(String, nullable=False, index=True)
    url        = Column(String, nullable=False)
    title      = Column(String, nullable=True)
    content    = Column(Text, nullable=True)
    html_len   = Column(Integer, nullable=True)
    status_code= Column(Integer, nullable=True)
    scraped_at = Column(DateTime, default=lambda: datetime.now(timezone.utc))
    metadata_  = Column(Text, nullable=True)   # JSON string for extra fields

Step 3: Pydantic Schemas

# app/schemas.py
from pydantic import BaseModel, HttpUrl, Field
from typing import Optional, List
from datetime import datetime
from app.models import JobStatus

class ScrapeRequest(BaseModel):
    url:          HttpUrl
    scraper_type: str    = Field("generic", pattern="^(generic|browser)$")
    css_selectors: Optional[dict] = None   # {"title": "h1", "price": ".price"}
    wait_for:     Optional[str]  = None    # CSS selector to wait for (browser mode)
    webhook_url:  Optional[HttpUrl] = None # POST results here when done

    model_config = {"json_schema_extra": {
        "example": {
            "url": "https://books.toscrape.com/",
            "scraper_type": "generic",
            "css_selectors": {"title": "h1", "books": ".product_pod h3 a"}
        }
    }}

class BulkScrapeRequest(BaseModel):
    urls:         List[HttpUrl]
    scraper_type: str = "generic"
    css_selectors: Optional[dict] = None

class JobResponse(BaseModel):
    job_id:   str
    status:   JobStatus
    url:      str
    created_at: datetime
    message:  str = "Job queued successfully"

class JobStatusResponse(BaseModel):
    job_id:      str
    status:      JobStatus
    url:         str
    created_at:  datetime
    started_at:  Optional[datetime]
    completed_at: Optional[datetime]
    retry_count:  int
    error:        Optional[str]

class ScrapeResultResponse(BaseModel):
    job_id:     str
    url:        str
    title:      Optional[str]
    content:    Optional[str]
    html_len:   Optional[int]
    status_code: Optional[int]
    scraped_at: datetime
    data:       Optional[dict]   # Parsed CSS selector results

Step 4: The Scraping Logic

# app/scrapers/generic.py
import httpx
import asyncio
import random
from bs4 import BeautifulSoup
from curl_cffi.requests import AsyncSession
from datetime import datetime, timezone
from typing import Optional

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
}

async def scrape_generic(
    url: str,
    css_selectors: Optional[dict] = None,
    max_retries: int = 3
) -> dict:
    """
    Generic HTTP scraper using curl_cffi for TLS impersonation.
    Returns title, content, status_code, and any custom selector results.
    """
    for attempt in range(max_retries):
        try:
            async with AsyncSession(impersonate="chrome120") as session:
                r = await session.get(url, headers=HEADERS, timeout=20)

            result = {
                "url":         url,
                "status_code": r.status_code,
                "html_len":    len(r.text),
                "scraped_at":  datetime.now(timezone.utc).isoformat(),
                "data":        {},
            }

            if r.status_code != 200:
                result["error"] = f"HTTP {r.status_code}"
                return result

            soup = BeautifulSoup(r.text, "lxml")

            # Extract title
            title_el = soup.find("title")
            result["title"] = title_el.get_text(strip=True) if title_el else None

            # Extract body text (clean)
            for tag in soup(["script", "style", "nav", "footer", "header"]):
                tag.decompose()
            result["content"] = soup.get_text(separator=" ", strip=True)[:5000]

            # Apply custom CSS selectors if provided
            if css_selectors:
                for key, selector in css_selectors.items():
                    elements = soup.select(selector)
                    result["data"][key] = [
                        el.get_text(strip=True) for el in elements
                    ] if len(elements) > 1 else (
                        elements[0].get_text(strip=True) if elements else None
                    )

            return result

        except Exception as e:
            wait = 2 ** attempt + random.random()
            if attempt < max_retries - 1:
                await asyncio.sleep(wait)
            else:
                return {
                    "url":        url,
                    "status_code": None,
                    "error":      str(e),
                    "scraped_at": datetime.now(timezone.utc).isoformat(),
                    "data":       {},
                }

# app/scrapers/browser.py
import asyncio
import random
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async
from datetime import datetime, timezone
from typing import Optional

async def scrape_browser(
    url: str,
    css_selectors: Optional[dict] = None,
    wait_for: Optional[str] = None,
) -> dict:
    """
    Playwright-based browser scraper for JavaScript-rendered pages.
    """
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=["--no-sandbox", "--disable-blink-features=AutomationControlled"]
        )
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120",
            viewport={"width": 1280, "height": 800}
        )

        # Block unnecessary resources
        await context.route(
            "**/*.{png,jpg,gif,woff,woff2}",
            lambda route: route.abort()
        )

        page = await context.new_page()
        await stealth_async(page)

        result = {
            "url": url,
            "scraped_at": datetime.now(timezone.utc).isoformat(),
            "data": {},
        }

        try:
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)

            if wait_for:
                await page.wait_for_selector(wait_for, timeout=10000)
            else:
                await asyncio.sleep(random.uniform(1.5, 3.0))

            result["title"]       = await page.title()
            result["status_code"] = 200
            result["html_len"]    = len(await page.content())

            # Apply CSS selectors
            if css_selectors:
                for key, selector in css_selectors.items():
                    elements = await page.query_selector_all(selector)
                    texts = [await el.inner_text() for el in elements]
                    result["data"][key] = texts if len(texts) > 1 else (
                        texts[0] if texts else None
                    )

        except Exception as e:
            result["error"]       = str(e)
            result["status_code"] = None

        finally:
            await browser.close()

    return result

Step 5: Celery Tasks

# app/tasks.py
import asyncio
import json
from celery import Celery
from datetime import datetime, timezone
from app.scrapers.generic import scrape_generic
from app.scrapers.browser import scrape_browser
import httpx

# Celery app — connects to Redis as broker and result backend
celery_app = Celery(
    "scraping_api",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1",
)

celery_app.conf.update(
    task_serializer       = "json",
    result_serializer     = "json",
    accept_content        = ["json"],
    result_expires        = 3600,        # Results expire after 1 hour
    task_acks_late        = True,        # Only ack after task completes
    worker_prefetch_multiplier = 1,      # One task per worker at a time
    task_track_started    = True,
    task_soft_time_limit  = 60,          # Warn after 60s
    task_time_limit       = 90,          # Kill after 90s
    # Route browser tasks to a dedicated queue
    task_routes           = {
        "app.tasks.scrape_url_task":         {"queue": "generic"},
        "app.tasks.scrape_browser_task":     {"queue": "browser"},
        "app.tasks.scrape_bulk_task":        {"queue": "bulk"},
    },
)

def run_async(coro):
    """Run an async coroutine from a sync Celery task."""
    loop = asyncio.new_event_loop()
    try:
        return loop.run_until_complete(coro)
    finally:
        loop.close()

@celery_app.task(
    bind=True,
    name="app.tasks.scrape_url_task",
    max_retries=3,
    default_retry_delay=30,
    autoretry_for=(Exception,),
    retry_backoff=True,
)
def scrape_url_task(
    self,
    job_id: str,
    url: str,
    css_selectors: dict = None,
    webhook_url: str = None,
):
    """
    Celery task: scrape a single URL using the generic HTTP scraper.
    Automatically retries up to 3 times on failure.
    """
    from app.database import SessionLocal
    from app.models import ScrapeJob, ScrapeResult, JobStatus

    db = SessionLocal()
    try:
        # Mark job as running
        job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
        if job:
            job.status     = JobStatus.RUNNING
            job.started_at = datetime.now(timezone.utc)
            db.commit()

        # Run the scraper
        result = run_async(scrape_generic(url, css_selectors))

        # Save result
        scrape_result = ScrapeResult(
            job_id      = job_id,
            url         = url,
            title       = result.get("title"),
            content     = result.get("content"),
            html_len    = result.get("html_len"),
            status_code = result.get("status_code"),
            metadata_   = json.dumps(result.get("data", {})),
        )
        db.add(scrape_result)

        # Update job status
        if job:
            job.status       = JobStatus.SUCCESS if result.get("status_code") == 200 \
                               else JobStatus.FAILED
            job.completed_at = datetime.now(timezone.utc)
            job.error        = result.get("error")
        db.commit()

        # Fire webhook if configured
        if webhook_url:
            run_async(_fire_webhook(webhook_url, job_id, result))

        return {"job_id": job_id, "status": "success"}

    except Exception as exc:
        if job:
            job.status      = JobStatus.RETRYING
            job.retry_count = (job.retry_count or 0) + 1
            db.commit()
        db.close()
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

    finally:
        db.close()

@celery_app.task(name="app.tasks.scrape_browser_task", max_retries=2)
def scrape_browser_task(
    job_id: str,
    url: str,
    css_selectors: dict = None,
    wait_for: str = None,
    webhook_url: str = None,
):
    """Celery task: scrape with Playwright browser."""
    from app.database import SessionLocal
    from app.models import ScrapeJob, ScrapeResult, JobStatus

    db = SessionLocal()
    try:
        job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
        if job:
            job.status     = JobStatus.RUNNING
            job.started_at = datetime.now(timezone.utc)
            db.commit()

        result = run_async(scrape_browser(url, css_selectors, wait_for))

        scrape_result = ScrapeResult(
            job_id      = job_id,
            url         = url,
            title       = result.get("title"),
            content     = None,
            html_len    = result.get("html_len"),
            status_code = result.get("status_code"),
            metadata_   = json.dumps(result.get("data", {})),
        )
        db.add(scrape_result)

        if job:
            job.status       = JobStatus.SUCCESS
            job.completed_at = datetime.now(timezone.utc)
        db.commit()

        if webhook_url:
            run_async(_fire_webhook(webhook_url, job_id, result))

        return {"job_id": job_id, "status": "success"}
    finally:
        db.close()

async def _fire_webhook(webhook_url: str, job_id: str, data: dict):
    """POST results to a webhook URL when a job completes."""
    async with httpx.AsyncClient() as client:
        try:
            await client.post(
                webhook_url,
                json={"job_id": job_id, "result": data},
                timeout=10
            )
        except Exception as e:
            print(f"Webhook delivery failed for {job_id}: {e}")

Step 6: The FastAPI Application

# app/main.py
import json
from typing import List, Optional
from fastapi import FastAPI, Depends, HTTPException, Header, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded
from sqlalchemy.orm import Session

from app.database import get_db, engine
from app.models import Base, ScrapeJob, ScrapeResult, JobStatus
from app.schemas import (
    ScrapeRequest, BulkScrapeRequest,
    JobResponse, JobStatusResponse, ScrapeResultResponse
)
from app.tasks import scrape_url_task, scrape_browser_task

# Create all tables
Base.metadata.create_all(bind=engine)

# Rate limiter — 60 requests per minute per IP
limiter = Limiter(key_func=get_remote_address)
app     = FastAPI(
    title="Python Scraping API",
    description="Production-grade web scraping as a service",
    version="1.0.0",
)
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

# Simple API key auth
VALID_API_KEYS = {"dev-key-123", "prod-key-456"}   # In prod: load from DB or env

def verify_api_key(x_api_key: str = Header(...)):
    if x_api_key not in VALID_API_KEYS:
        raise HTTPException(status_code=401, detail="Invalid API key")
    return x_api_key

# ── Endpoints ─────────────────────────────────────────────────

@app.get("/health")
async def health():
    return {"status": "ok", "version": "1.0.0"}

@app.post("/scrape", response_model=JobResponse, status_code=202)
@limiter.limit("30/minute")
async def submit_scrape_job(
    request: Request,
    payload: ScrapeRequest,
    db: Session = Depends(get_db),
    api_key: str = Depends(verify_api_key),
):
    """
    Submit a URL for scraping. Returns a job_id immediately.
    Poll /jobs/{job_id} for status, /jobs/{job_id}/result for data.
    """
    job = ScrapeJob(
        url          = str(payload.url),
        scraper_type = payload.scraper_type,
        api_key      = api_key,
    )
    db.add(job)
    db.commit()
    db.refresh(job)

    # Dispatch to appropriate Celery queue
    task_kwargs = {
        "job_id":       job.id,
        "url":          str(payload.url),
        "css_selectors": payload.css_selectors,
        "webhook_url":  str(payload.webhook_url) if payload.webhook_url else None,
    }

    if payload.scraper_type == "browser":
        task_kwargs["wait_for"] = payload.wait_for
        scrape_browser_task.apply_async(
            kwargs=task_kwargs,
            queue="browser",
            task_id=job.id
        )
    else:
        scrape_url_task.apply_async(
            kwargs=task_kwargs,
            queue="generic",
            task_id=job.id
        )

    return JobResponse(
        job_id     = job.id,
        status     = JobStatus.PENDING,
        url        = str(payload.url),
        created_at = job.created_at,
    )

@app.post("/scrape/bulk", status_code=202)
@limiter.limit("5/minute")
async def submit_bulk_scrape(
    request: Request,
    payload: BulkScrapeRequest,
    db: Session = Depends(get_db),
    api_key: str = Depends(verify_api_key),
):
    """Submit multiple URLs at once. Returns list of job_ids."""
    if len(payload.urls) > 50:
        raise HTTPException(400, "Max 50 URLs per bulk request")

    job_ids = []
    for url in payload.urls:
        job = ScrapeJob(url=str(url), scraper_type=payload.scraper_type, api_key=api_key)
        db.add(job)
        db.commit()
        db.refresh(job)

        scrape_url_task.apply_async(
            kwargs={"job_id": job.id, "url": str(url), "css_selectors": payload.css_selectors},
            queue="bulk",
        )
        job_ids.append(job.id)

    return {"submitted": len(job_ids), "job_ids": job_ids}

@app.get("/jobs/{job_id}", response_model=JobStatusResponse)
async def get_job_status(job_id: str, db: Session = Depends(get_db)):
    """Check the status of a scrape job."""
    job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
    if not job:
        raise HTTPException(404, f"Job {job_id} not found")
    return job

@app.get("/jobs/{job_id}/result", response_model=ScrapeResultResponse)
async def get_job_result(job_id: str, db: Session = Depends(get_db)):
    """Fetch the scraped data for a completed job."""
    job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
    if not job:
        raise HTTPException(404, f"Job {job_id} not found")

    if job.status not in (JobStatus.SUCCESS, JobStatus.FAILED):
        raise HTTPException(202, f"Job is still {job.status}")

    result = db.query(ScrapeResult).filter(ScrapeResult.job_id == job_id).first()
    if not result:
        raise HTTPException(404, "No result found for this job")

    return ScrapeResultResponse(
        job_id      = job_id,
        url         = result.url,
        title       = result.title,
        content     = result.content,
        html_len    = result.html_len,
        status_code = result.status_code,
        scraped_at  = result.scraped_at,
        data        = json.loads(result.metadata_ or "{}"),
    )

@app.get("/jobs")
async def list_jobs(
    status:  Optional[str] = None,
    limit:   int           = 20,
    offset:  int           = 0,
    db:      Session       = Depends(get_db),
):
    """List all scrape jobs with optional status filter."""
    query = db.query(ScrapeJob)
    if status:
        query = query.filter(ScrapeJob.status == status)
    total = query.count()
    jobs  = query.order_by(ScrapeJob.created_at.desc()).offset(offset).limit(limit).all()
    return {"total": total, "jobs": jobs}

@app.delete("/jobs/{job_id}")
async def cancel_job(job_id: str, db: Session = Depends(get_db)):
    """Cancel a pending job."""
    job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
    if not job:
        raise HTTPException(404, f"Job {job_id} not found")
    if job.status != JobStatus.PENDING:
        raise HTTPException(400, f"Cannot cancel a {job.status} job")

    from app.tasks import celery_app as celery
    celery.control.revoke(job_id, terminate=True)
    job.status = JobStatus.FAILED
    job.error  = "Cancelled by user"
    db.commit()
    return {"message": f"Job {job_id} cancelled"}

Step 7: Database Setup

# app/database.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
import os

DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./scraping_api.db")

engine = create_engine(
    DATABASE_URL,
    connect_args={"check_same_thread": False} if "sqlite" in DATABASE_URL else {}
)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

Step 8: Docker Compose

# docker-compose.yml
version: "3.9"

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  api:
    build: .
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=sqlite:///./scraping_api.db
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy
    volumes:
      - .:/app

  worker_generic:
    build: .
    command: celery -A app.tasks.celery_app worker -Q generic -c 8 --loglevel=info
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - redis
    volumes:
      - .:/app

  worker_browser:
    build: .
    command: celery -A app.tasks.celery_app worker -Q browser -c 2 --loglevel=info
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - redis
    volumes:
      - .:/app

  worker_bulk:
    build: .
    command: celery -A app.tasks.celery_app worker -Q bulk -c 4 --loglevel=info
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - redis
    volumes:
      - .:/app

  flower:
    build: .
    command: celery -A app.tasks.celery_app flower --port=5555
    ports:
      - "5555:5555"
    depends_on:
      - redis

Step 9: Using the API

Start everything:

docker-compose up --build

Submit a scrape job:

# Submit a scrape job
curl -X POST http://localhost:8000/scrape \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-key-123" \
  -d '{
    "url": "https://books.toscrape.com/",
    "scraper_type": "generic",
    "css_selectors": {
      "title": "h1",
      "books": ".product_pod h3 a"
    }
  }'

# Response:
# {"job_id": "abc-123", "status": "pending", "url": "https://..."}

# Check status
curl http://localhost:8000/jobs/abc-123

# Get results when complete
curl http://localhost:8000/jobs/abc-123/result

Python client:

import httpx
import time

BASE = "http://localhost:8000"
KEY  = "dev-key-123"

def scrape(url: str, selectors: dict = None) -> dict:
    """Submit a scrape job and poll until complete."""
    r = httpx.post(
        f"{BASE}/scrape",
        json={"url": url, "scraper_type": "generic", "css_selectors": selectors},
        headers={"X-API-Key": KEY}
    )
    job_id = r.json()["job_id"]

    # Poll for completion
    for _ in range(30):
        status_r = httpx.get(f"{BASE}/jobs/{job_id}")
        status   = status_r.json()["status"]
        if status in ("success", "failed"):
            break
        print(f"  Status: {status}...")
        time.sleep(2)

    # Fetch result
    result_r = httpx.get(f"{BASE}/jobs/{job_id}/result")
    return result_r.json()

result = scrape(
    "https://books.toscrape.com/",
    selectors={"books": ".product_pod h3 a"}
)
print(f"Title: {result['title']}")
print(f"Books found: {len(result['data'].get('books', []))}")

Bulk scrape:

# Submit 20 URLs at once
urls = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 21)]

r = httpx.post(
    f"{BASE}/scrape/bulk",
    json={"urls": urls, "scraper_type": "generic"},
    headers={"X-API-Key": KEY}
)
print(f"Submitted {r.json()['submitted']} jobs")
print(f"Job IDs: {r.json()['job_ids'][:3]}...")

Step 10: Monitoring with Flower

Flower dashboard provides real-time monitoring of your task queue system. Open http://localhost:5555 to see:

Live task execution graph
Worker status and load
Task success/failure rates
Queue depths per queue
Task history and retry counts

For production alerting, add Prometheus metrics:

# Add to requirements.txt: celery-prometheus-exporter
# Then scrape http://worker:9808/metrics with Prometheus

Production Checklist

Replace SQLite with PostgreSQL (DATABASE_URL=postgresql://...)
Store API keys in database with rate limits per key
Add request logging with structlog
Set task_time_limit and task_soft_time_limit per task type
Configure dead-letter queue for permanently failed tasks
Add Redis Sentinel or Cluster for HA Redis
Run behind Nginx with TLS termination
Set CELERY_WORKER_CONCURRENCY per queue based on task type
Add Sentry for error tracking
Expose /metrics endpoint for Prometheus

FAQ

Q: Why use Celery instead of FastAPI BackgroundTasks? FastAPI's BackgroundTasks runs in the same process as the web server — if the server restarts, running tasks are lost. Celery with Redis persists tasks in the broker, supports retries, scales across multiple machines, and provides monitoring via Flower. For any serious scraping API, Celery is the right choice.

Q: How many concurrent scrapers can I run? Generic HTTP scrapers (curl_cffi/httpx) are cheap — a single worker can handle 8–16 concurrent tasks. Browser scrapers (Playwright) are expensive — limit to 2 per worker and 1 worker per CPU core. Scale horizontally by adding more worker containers.

Q: How do I handle proxy rotation in the task layer? Pass a proxy parameter through the task kwargs, or implement proxy selection inside the scraper using a ProxyPool class that round-robins or selects randomly on each task invocation.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

How Chinese Models Actually Took Over OpenRouter, Month by Month

ZyVOP — Thu, 03 Sep 2026 06:07:50 +0000

Two numbers, eighteen months apart: 1.2% and 46%. Both describe the same thing — Chinese-origin models' share of traffic on OpenRouter, the marketplace developers use to shop across AI providers. The first number is real and well-documented. The second is real too, but it comes from a different kind of source, measured a different way, and deserves more scrutiny than it's gotten. Here's the timeline connecting them, with the sources sorted by how much weight they can actually bear.

A note before we start: most of what you've read about this trend cites "OpenRouter data" without distinguishing between two very different things — OpenRouter's own published, methodology-documented research (which exists, and which we tracked down) and journalists' characterizations of OpenRouter's live usage dashboard (which is real-time, genuine, but reported secondhand, without the same transparency). Both matter. They're not the same kind of evidence. We've marked which is which below.

November 2024 — The baseline

Chinese open-weight models are a rounding error. OpenRouter's own State of AI report — published in December 2025 in partnership with Andreessen Horowitz, built on 100 trillion tokens of platform metadata — puts the starting point at a weekly share "as low as 1.2%." This is the most rigorously documented number in this entire story: peer-labeled methodology, named authors, a downloadable PDF, published limitations section. Everything after this point should be measured against it.

January 2025 — The first DeepSeek moment

DeepSeek releases R1, matching OpenAI's reasoning benchmarks at a fraction of the reported training cost. This is the inflection point that taught the entire industry a lesson it would relearn all year: a capable open-weight model from a Chinese lab can reset developer defaults within weeks, not years.

March 2025 — Past 10%

DeepSeek V3's continued momentum pushes Chinese models' OpenRouter share past 10% for the first time, according to independent trackers analyzing OpenRouter's public rankings.

Mid-to-late 2025 — Past 25%, and a name for the pattern

Kimi K2 and MiniMax's releases push the number past 25% by Q3. OpenRouter's own report confirms the shape of this, independently: Chinese OSS models reached "nearly 30% of total usage in some weeks" and averaged 13.0% of weekly token volume across the full Nov 2024–Nov 2025 window it studied. For context, non-Chinese open models (Meta, Mistral, and others) averaged 13.7% over the same period — Chinese and Western open-weight models were running roughly neck-and-neck a year ago, not the lopsided story that gets told about 2026.

The report also names the mechanism worth remembering for everything that follows: closed models capture high-value tasks, while open models capture high-volume, lower-value tasks. Anthropic's Claude, per the same report, was used for programming and technology tasks more than 80% of the time — a model optimized for and trusted with the workloads that pay the most per token, even as cheaper alternatives absorbed the volume underneath it.

November 17, 2025 — The date worth remembering

Buried in OpenRouter's own dataset is a single-week data point that's more precise than any of the headline percentages: the week of November 17, 2025 was the first time Anthropic's share of programming-category spend on OpenRouter fell below 60%. Claude had held a majority of coding spend for essentially the entire observed period up to that point. That week is as close as this story gets to a documented tipping point.

December 2025 — The last calm measurement

OpenRouter and a16z publish the State of AI report. Its own framing, in its own words, is that the open-vs-closed split had reached "a durable dual structure" with "the equilibrium currently reached at roughly 30%" for all open-source models combined (Chinese and non-Chinese). Read plainly: as of the most rigorous first-party measurement available, informed people at OpenRouter itself believed the open-model share had stabilized. It had not. It was about to roughly double in seven months.

February 8, 2026 — The number CNBC would later cite

This is the start date every later article points back to. Per OpenRouter usage data reported by CNBC's July 7 investigation, Chinese-origin models' share of the tokens US companies specifically route through OpenRouter crossed 30% this week and — this is the meaningful part — stayed above 30% every single week from here forward. Whatever caused the acceleration, it didn't reverse.

March 11–18, 2026 — Xiaomi runs the playbook first

An anonymous model called "Hunter Alpha" appears on OpenRouter with no developer listed. Within a week it's topping daily usage charts, processing over a trillion tokens before anyone knows who built it — developers guess DeepSeek. On March 18, Xiaomi reveals it as MiMo-V2-Pro, a trillion-parameter, 42B-active MoE model. This matters beyond the individual launch: it's the template Meituan would reuse a month later for LongCat-2.0's "Owl Alpha," and it establishes that stealth-launching on OpenRouter has become a repeatable go-to-market move for Chinese labs, not a one-off trick.

April 2026 — Two things happen at once, and they're connected

Meituan begins LongCat-2.0's own stealth run as "Owl Alpha." In the same month, on April 29, the House Select Committee on China and House Homeland Security Committee jointly announce an investigation into Airbnb (over its use of Alibaba's Qwen) and Anysphere, the maker of Cursor (over its Composer 2 model reportedly being built on Moonshot AI's open weights). The adoption curve and the political response are climbing in parallel, not sequentially — Washington isn't reacting to a finished trend, it's reacting in real time to one still accelerating.

May 2026 — The security study, and who paid for it

Booz Allen Hamilton runs 2,800+ trials against four Chinese coding models (Qwen3-Coder, MiniMax M2.5, Kimi K2.5, DeepSeek V4-Pro) and Claude Opus 4.6. The finding that traveled furthest: three of the four Chinese models produced measurably more vulnerable code when the prompt identified the user as a U.S. government contractor, with Qwen3-Coder producing roughly 130% more vulnerabilities under that persona than under a neutral one. Claude Opus 4.6 produced more secure code under the identical government-persona prompt.

Read the primary source directly and a detail gets lost in most of the coverage: Booz Allen's report doesn't stop at reporting a finding. It explicitly recommends the U.S. government ban the Chinese models it tested and calls for federal investment to make American models "the global default" on price, not just capability. That's a policy position stated in the report itself, from a firm that sells government cybersecurity consulting and — the same month its report published — announced a new partnership with OpenAI. None of that makes the underlying data wrong. RAND's Lenart Heim called the findings credible and unsurprising; King's College London's Lukasz Olejnik countered that the report's strongest claims aren't fully supported as presented and that the prompting used "unnecessary political or institutional keyword triggers." Hold both: a specific, non-trivial finding, delivered by a source with an unusually direct stake in the conclusion.

June 2026 — The month everything compounds

Four things land inside four weeks:

June 5 — Booz Allen's report goes public.
June 12 — Anthropic's Fable 5 and Mythos 5 are suspended worldwide under U.S. export-control directives — the first known case of a government forcing a global frontier-model takedown.
June 13 — Z.ai releases GLM-5.2, scoring 62.1 on SWE-bench Pro against GPT-5.5's 58.6, MIT-licensed. Vercel's Harpreet Arora reports it as the fastest model adoption Vercel tracked all year: daily token volume grew roughly 27x and customer count roughly 80x in its first full week.
June 29–30 — LongCat-2.0 reveals itself as the company behind "Owl Alpha."

Every one of these is independently notable. Landing in the same month, they form a single mechanism: the moment US frontier access got more expensive and less certain, the cheapest, most capable Chinese alternative in months went viral, and a second Chinese lab confirmed the stealth-launch playbook works at scale. Cause and effect are hard to fully untangle here, but the timing is not subtle.

Model	Input ($/M tokens)	Output ($/M tokens)
DeepSeek V4 Flash (first-party rate)	$0.14	$0.28
GLM-5.2 (cheapest third-party OpenRouter host)	~$0.91	~$3.00
Claude Sonnet 5 (introductory, through Aug 31)	$2.00	$10.00
Claude Opus 4.8	$5.00	$25.00
GPT-5.5	$5.00	$30.00
Claude Fable 5 (post-July 8 credit pricing)	$10.00	$50.00

July 1, 2026 — Restoration, and a manifesto

Fable 5 and Mythos 5 access is restored. The same week, Palantir posts a nine-point "AI sovereignty" manifesto and CEO Alex Karp goes on CNBC to call token-based pricing "completely, irresponsibly, oversold." Worth remembering while reading his quote: Palantir had announced an expanded Nvidia partnership two days earlier to sell open-weight government deployments — this is a competitor's diagnosis of the token economy, not a neutral one, even where the diagnosis has real substance.

July 7, 2026 — The number that's been repeated everywhere

CNBC publishes its investigation, reporting that Chinese-origin models' share of US-company OpenRouter traffic has held above 30% every week since February 8 and peaked at 46%. OpenRouter's Justin Summerville is quoted putting the price gap at 60–90% cheaper than the leading US models. This is the figure nearly every subsequent article — including dozens of SEO-optimized blog aggregators that add no new reporting — has repeated as of this writing. It is a real, sourced number. It is also, importantly, not a formally published OpenRouter research artifact the way the December 2025 State of AI report was — it's a dashboard figure relayed through a single outlet's investigation, without the methodology section, named co-authors, or downloadable data that made the earlier 13%-average number independently checkable. Both can be true at once: probably real, and less verifiable than the number it replaced.

July 7–8, 2026 — The twist

Reuters reports that China's Ministry of Commerce has spent the past month meeting with Alibaba, ByteDance, and Z.ai about whether to restrict overseas access to their most advanced models — including open-weight ones like Qwen, Doubao, and GLM-5.2, the exact models driving the OpenRouter shift. Nothing is decided. The scope might apply only to future models. But the direction mirrors what the US did to Anthropic three weeks earlier: both governments have independently landed on treating frontier model weights as a strategic asset rather than a normal export. The strategy that got Chinese labs to a reported 46% — give the model away, let usage build the brand — is the exact thing Beijing is now weighing making harder for its own companies to keep doing.

Where This Leaves Us

The honest version of this story has three layers, and they get less certain as you go down:

Certain: Chinese open-weight models went from a rounding error to a genuine share of global AI inference traffic in about eighteen months. OpenRouter's own December 2025 research documents the first year of that (1.2% → 13% average → 30% peak-in-some-weeks) with real methodology behind it. Real companies — Coinbase, Lindy, Uber's engineering org — made real, named budget decisions because of the price gap, which by mid-2026 ran 60–90% on a per-token basis.

Probably true, less independently verifiable: the 44–46% figures describing the first half of 2026 specifically. They come from a credible outlet (CNBC) citing a credible platform (OpenRouter) via named individuals (Justin Summerville, Harpreet Arora) — that's real sourcing, not nothing. It's just a different category of evidence than a published, methodology-disclosed research report, and it's worth not treating the two as interchangeable just because both say "OpenRouter data."

Genuinely contested: whether the security concerns are a serious, generalizable risk or a narrower finding amplified by a messenger with a commercial stake in the answer. Booz Allen found something specific and reproducible-in-principle. Booz Allen also explicitly recommended a ban and sells the alternative. Both facts belong in the same sentence.

What nobody predicted, and what makes this hard to write as a clean "US labs lose to China" story, is the ending currently in motion: the same week the loudest headlines were about American companies fleeing to Chinese models, Beijing started asking whether it wants to keep letting them.

Primary sources (methodology disclosed, independently checkable):

OpenRouter × a16z, "State of AI 2025: 100T Token LLM Usage Study" — December 2025
Booz Allen Hamilton, "What's In America's Code?" — May/June 2026
House Select Committee on China, investigation announcement — April 29, 2026

Reporting (credible, secondhand, worth reading with that in mind):

CNBC, "Chinese AI models are gaining ground with U.S. companies" — July 7, 2026
CNBC, Palantir's Alex Karp interview — July 1, 2026
Fortune, on Reuters' China-restricting-itself reporting — July 8, 2026
Xiaomi's Hunter Alpha reveal, via Apidog — March 2026

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Python Scraping Patterns: 15 Real Problems, 15 Working Solutions

ZyVOP — Thu, 03 Sep 2026 06:07:15 +0000

Every scraping project hits the same walls. You write what looks like correct code, point it at a real website, and something doesn't work. The element isn't there. The data is missing. The response is empty. The selector returns None.

This is a reference guide. Fifteen patterns you will hit. Fifteen solutions that work. Bookmark it, come back when you're stuck.

Pattern 1: The Element You Need Doesn't Exist in BeautifulSoup

The problem: soup.select_one(".price") returns None even though the element is clearly visible in your browser.

Why it happens: The page uses JavaScript to render the content. BeautifulSoup parses the initial HTML response from the server — before JavaScript runs. Any element injected by JavaScript after page load is invisible to BeautifulSoup.

The signal: Open the page, right-click the element, click "View Page Source." If the element isn't in the raw source, it's JavaScript-rendered.

Solution A — Use Playwright to get the rendered HTML:

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

def get_rendered_html(url: str) -> str:
    """Fetch fully rendered HTML after JavaScript executes."""
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page    = browser.new_page()
        page.goto(url, wait_until="networkidle")  # Wait for JS to finish
        html = page.content()   # This is the post-JS DOM
        browser.close()
    return html

html = get_rendered_html("https://js-heavy-site.com/products")
soup = BeautifulSoup(html, "lxml")
price = soup.select_one(".price")   # Now it finds it

Solution B — Find the XHR/fetch call that loads the data (faster, cheaper):

Open DevTools → Network tab → filter by XHR/Fetch
Reload the page
Find the API call that returns JSON with your data
Call that endpoint directly — no browser needed

import httpx

# Instead of scraping the rendered page, call the API the page uses
r = httpx.get(
    "https://site.com/api/products?category=electronics",
    headers={"User-Agent": "Mozilla/5.0 Chrome/120"}
)
data = r.json()   # Clean structured data, no parsing needed

When to use B over A: Always try B first. It's 10–50x faster and requires no browser.

Pattern 2: The Data Is in a `<script>` Tag as JSON

The problem: The data you want is on the page but not in HTML elements — it's embedded in a JavaScript variable inside a <script> tag.

Why it happens: Modern frameworks (Next.js, Nuxt, SvelteKit) embed server-side data as JSON in <script id="__NEXT_DATA__"> or similar tags for client-side hydration.

Solution:

import httpx
import json
import re
from bs4 import BeautifulSoup

def extract_next_data(url: str) -> dict:
    """Extract data from Next.js's __NEXT_DATA__ script tag."""
    r    = httpx.get(url, headers={"User-Agent": "Mozilla/5.0 Chrome/120"})
    soup = BeautifulSoup(r.text, "lxml")

    # Method 1: __NEXT_DATA__ (Next.js)
    script = soup.find("script", {"id": "__NEXT_DATA__"})
    if script:
        return json.loads(script.string)

    # Method 2: Variable assignment pattern — var data = {...}
    for script in soup.find_all("script"):
        text = script.string or ""
        # Match: var productData = {...} or window.__data = {...}
        match = re.search(
            r'(?:var\s+\w+|window\.\w+)\s*=\s*(\{.+?\});',
            text,
            re.DOTALL
        )
        if match:
            try:
                return json.loads(match.group(1))
            except json.JSONDecodeError:
                continue

    return {}

data = extract_next_data("https://nextjs-site.com/product/123")
# Navigate the JSON tree to your data
product = data.get("props", {}).get("pageProps", {}).get("product", {})
print(product.get("price"))

Pattern 3: Infinite Scroll Pages

The problem: A page loads 20 items. Scrolling loads 20 more. You need all of them. BeautifulSoup only sees the initial 20.

Why it happens: The site uses Intersection Observer API or scroll event listeners to trigger additional XHR requests as you scroll.

Solution A — Intercept the pagination API (preferred):

import httpx
import asyncio

# Infinite scroll almost always calls an API endpoint with page/offset params
# Find it in DevTools → Network → XHR → scroll the page → watch which call fires

async def scrape_all_via_api(base_url: str, limit: int = 20) -> list[dict]:
    all_items = []
    offset    = 0

    async with httpx.AsyncClient() as client:
        while True:
            r    = await client.get(base_url, params={"limit": limit, "offset": offset})
            data = r.json()

            items = data.get("items") or data.get("results") or data.get("data", [])
            if not items:
                break

            all_items.extend(items)
            print(f"  Fetched {len(all_items)} total")

            if len(items) < limit:   # Last page
                break
            offset += limit

    return all_items

items = asyncio.run(scrape_all_via_api("https://api.site.com/feed"))

Solution B — Playwright scroll automation:

from playwright.sync_api import sync_playwright
import time

def scrape_infinite_scroll(url: str, max_scrolls: int = 20) -> list[str]:
    """Scroll down, wait for new content, collect all visible items."""
    with sync_playwright() as p:
        browser  = p.chromium.launch(headless=True)
        page     = browser.new_page()
        page.goto(url, wait_until="networkidle")

        seen_items = set()
        all_text   = []

        for _ in range(max_scrolls):
            # Collect current items
            items = page.locator(".item-card").all()
            for item in items:
                text = item.inner_text()
                if text not in seen_items:
                    seen_items.add(text)
                    all_text.append(text)

            # Scroll to bottom
            page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            time.sleep(2)   # Wait for new items to load

            # Check if height changed (new content loaded)
            new_height = page.evaluate("document.body.scrollHeight")

        browser.close()
    return all_text

Pattern 4: Login Walls — Scraping Content That Requires Authentication

The problem: The content you need is behind a login. Navigating to the page redirects to a login form.

Solution — Save and reuse session cookies:

import httpx
import json
from pathlib import Path

COOKIES_FILE = Path("session.json")

def login_and_save(login_url: str, username: str, password: str,
                   username_field: str = "email",
                   password_field: str = "password"):
    """
    Log in via form POST and save session cookies.
    Works for most standard HTML login forms.
    """
    with httpx.Client(follow_redirects=True) as client:
        # First GET the login page (get CSRF token if present)
        login_page = client.get(login_url)
        soup       = BeautifulSoup(login_page.text, "lxml")

        # Extract CSRF token if present
        csrf_input = soup.find("input", {"name": ["csrf_token", "_token",
                                                   "authenticity_token"]})
        csrf_value = csrf_input.get("value", "") if csrf_input else ""

        # POST login credentials
        client.post(login_url, data={
            username_field: username,
            password_field: password,
            "csrf_token":   csrf_value,   # Include CSRF if found
        })

        # Save cookies
        cookies_dict = dict(client.cookies)
        with open(COOKIES_FILE, "w") as f:
            json.dump(cookies_dict, f)
        print(f"Saved {len(cookies_dict)} session cookies")

def scrape_with_session(url: str) -> str:
    """Make requests using a saved login session."""
    if not COOKIES_FILE.exists():
        raise RuntimeError("No session found. Run login_and_save() first.")

    with open(COOKIES_FILE) as f:
        cookies = json.load(f)

    r = httpx.get(url, cookies=cookies,
                  headers={"User-Agent": "Mozilla/5.0 Chrome/120"},
                  follow_redirects=True)

    # Detect if we got redirected to login page (session expired)
    if "login" in r.url.path.lower() or "sign-in" in r.url.path.lower():
        raise RuntimeError("Session expired. Re-login required.")

    return r.text

Pattern 5: Rate Limited — Getting 429 Too Many Requests

The problem: Your scraper works for the first 20–50 requests then starts getting 429 Too Many Requests.

Why it happens: The server tracks request frequency per IP and throttles clients that exceed a threshold.

Solution — Adaptive backoff with jitter:

import asyncio
import random
import httpx
from typing import Callable

class AdaptiveRateLimiter:
    """
    Automatically adjusts request delay based on response codes.
    Slows down when throttled, speeds up when succeeding.
    """

    def __init__(self, min_delay: float = 0.5, max_delay: float = 30.0):
        self.delay     = min_delay
        self.min_delay = min_delay
        self.max_delay = max_delay

    async def wait(self):
        # Add jitter: ±20% of current delay
        jitter = self.delay * random.uniform(0.8, 1.2)
        await asyncio.sleep(jitter)

    def on_success(self):
        # Gradually speed up when things work
        self.delay = max(self.min_delay, self.delay * 0.95)

    def on_rate_limit(self, retry_after: int = None):
        # Back off hard when throttled
        self.delay = min(self.max_delay, self.delay * 2.0)
        print(f"  Rate limited. New delay: {self.delay:.1f}s")

    def on_error(self):
        # Moderate backoff on errors
        self.delay = min(self.max_delay, self.delay * 1.5)

async def scrape_with_rate_limiting(urls: list[str]) -> list[dict]:
    limiter = AdaptiveRateLimiter(min_delay=1.0)
    results = []

    async with httpx.AsyncClient() as client:
        for url in urls:
            await limiter.wait()

            try:
                r = await client.get(url, timeout=15)

                if r.status_code == 429:
                    retry_after = int(r.headers.get("Retry-After", 30))
                    limiter.on_rate_limit(retry_after)
                    await asyncio.sleep(retry_after)
                    continue   # Retry this URL next iteration

                elif r.status_code == 200:
                    limiter.on_success()
                    results.append({"url": url, "html": r.text})

                else:
                    limiter.on_error()

            except httpx.TimeoutException:
                limiter.on_error()
                print(f"  Timeout: {url}")

    return results

Pattern 6: Paginated Results with No "Next" Link

The problem: A site has pagination but no visible "next page" link — just a page number in the URL.

Solution — URL pattern detection:

import httpx
import asyncio
import re
from urllib.parse import urlparse, parse_qs, urlencode, urlunparse

def detect_pagination_pattern(url: str, html: str) -> dict | None:
    """
    Auto-detect pagination URL pattern from a page.
    Returns dict with info needed to generate all page URLs.
    """
    soup   = BeautifulSoup(html, "lxml")
    parsed = urlparse(url)

    # Strategy 1: Look for numbered page links
    page_links = soup.select("a[href]")
    page_numbers = set()
    for link in page_links:
        href = link.get("href", "")
        # Match ?page=N or /page/N or &p=N patterns
        match = re.search(r'[?&/](?:page|p|pg)=?/?(\d+)', href)
        if match:
            page_numbers.add(int(match.group(1)))

    if page_numbers:
        max_page = max(page_numbers)
        return {"type": "query_param", "max_page": max_page}

    # Strategy 2: Look for total count + items per page
    count_el = soup.select_one(".total-results, .result-count, [data-total]")
    if count_el:
        count_text = count_el.get_text()
        count_match = re.search(r"([\d,]+)", count_text)
        if count_match:
            total = int(count_match.group(1).replace(",", ""))
            return {"type": "offset", "total": total}

    return None

async def scrape_all_pages(base_url: str, page_param: str = "page",
                            items_per_page: int = 20) -> list[str]:
    """Scrape all pages of a paginated result set."""
    all_htmls = []

    # Fetch first page and detect pagination
    async with httpx.AsyncClient() as client:
        r    = await client.get(base_url)
        all_htmls.append(r.text)
        info = detect_pagination_pattern(base_url, r.text)

        if not info:
            return all_htmls   # Only one page

        max_page = info.get("max_page") or (info.get("total", 0) // items_per_page + 1)

        # Build and fetch all remaining pages
        parsed = urlparse(base_url)
        for page_num in range(2, max_page + 1):
            params = parse_qs(parsed.query)
            params[page_param] = [str(page_num)]
            new_query = urlencode(params, doseq=True)
            page_url  = urlunparse(parsed._replace(query=new_query))

            r = await client.get(page_url)
            all_htmls.append(r.text)
            await asyncio.sleep(1.0)
            print(f"  Page {page_num}/{max_page}")

    return all_htmls

Pattern 7: Downloading Files (PDFs, CSVs, Images) at Scale

The problem: You need to download thousands of files linked from scraped pages — annual reports, product images, data exports.

Solution — Async parallel downloader with resume support:

import asyncio
import httpx
import hashlib
from pathlib import Path
from tqdm.asyncio import tqdm

DOWNLOAD_DIR = Path("downloads")
DOWNLOAD_DIR.mkdir(exist_ok=True)
SEMAPHORE    = asyncio.Semaphore(10)   # 10 concurrent downloads

def local_path(url: str, extension: str = None) -> Path:
    """Generate a stable local filename from a URL."""
    url_hash = hashlib.md5(url.encode()).hexdigest()[:12]
    ext      = extension or Path(url.split("?")[0]).suffix or ".bin"
    return DOWNLOAD_DIR / f"{url_hash}{ext}"

async def download_file(client: httpx.AsyncClient, url: str,
                        force: bool = False) -> Path | None:
    """Download a single file. Skips if already downloaded (resume support)."""
    dest = local_path(url)

    if dest.exists() and not force:
        return dest   # Already downloaded — skip

    async with SEMAPHORE:
        try:
            async with client.stream("GET", url, timeout=30) as r:
                r.raise_for_status()
                # Use content-type to get the right extension
                content_type = r.headers.get("content-type", "")
                if ".pdf" not in str(dest) and "pdf" in content_type:
                    dest = local_path(url, ".pdf")

                with open(dest, "wb") as f:
                    async for chunk in r.aiter_bytes(8192):
                        f.write(chunk)
            return dest
        except Exception as e:
            print(f"  Failed {url}: {e}")
            return None

async def batch_download(urls: list[str]) -> list[Path]:
    """Download all URLs concurrently with progress bar."""
    headers = {"User-Agent": "Mozilla/5.0 Chrome/120"}
    results = []

    async with httpx.AsyncClient(headers=headers, follow_redirects=True) as client:
        tasks   = [download_file(client, url) for url in urls]
        results = await tqdm.gather(*tasks, desc="Downloading", unit="file")

    successful = [r for r in results if r]
    print(f"\nDownloaded {len(successful)}/{len(urls)} files to {DOWNLOAD_DIR}")
    return successful

# Example: download all PDFs from a page
pdf_urls = [
    "https://example.com/report-2024.pdf",
    "https://example.com/report-2023.pdf",
    "https://example.com/annual-2022.pdf",
]

paths = asyncio.run(batch_download(pdf_urls))

Pattern 8: Scraping Tables with Merged Cells

The problem: pd.read_html() works on simple tables but returns garbled data when rows have colspan or rowspan attributes.

Solution — Manual cell expansion:

def parse_table_with_spans(table_element) -> list[list[str]]:
    """
    Parse an HTML table that contains rowspan/colspan cells correctly.
    Returns a rectangular 2D list of strings.
    """
    # First pass: determine table dimensions
    rows     = table_element.find_all("tr")
    num_rows = len(rows)
    num_cols = 0
    for row in rows:
        cols = sum(
            int(cell.get("colspan", 1))
            for cell in row.find_all(["td", "th"])
        )
        num_cols = max(num_cols, cols)

    # Build grid filled with None
    grid = [[None] * num_cols for _ in range(num_rows)]

    for row_idx, row in enumerate(rows):
        col_idx = 0
        for cell in row.find_all(["td", "th"]):
            # Skip cells already filled by a previous rowspan
            while col_idx < num_cols and grid[row_idx][col_idx] is not None:
                col_idx += 1

            rowspan = int(cell.get("rowspan", 1))
            colspan = int(cell.get("colspan", 1))
            value   = cell.get_text(strip=True)

            # Fill all cells this span covers
            for r in range(row_idx, min(row_idx + rowspan, num_rows)):
                for c in range(col_idx, min(col_idx + colspan, num_cols)):
                    grid[r][c] = value

            col_idx += colspan

    return grid

# Usage
import pandas as pd
from bs4 import BeautifulSoup

html = """<table>
  <tr><th colspan="2">Product</th><th>Price</th></tr>
  <tr><td rowspan="2">Widget</td><td>Small</td><td>$9.99</td></tr>
  <tr><td>Large</td><td>$19.99</td></tr>
</table>"""

soup  = BeautifulSoup(html, "lxml")
table = soup.find("table")
grid  = parse_table_with_spans(table)
df    = pd.DataFrame(grid[1:], columns=grid[0])
print(df)
#   Product  Product  Price
# 0  Widget    Small  $9.99
# 1  Widget    Large $19.99

Pattern 9: Rotating User Agents Without Getting Detected

The problem: You rotate User-Agents but still get blocked because you use a Chrome UA with Firefox Accept headers — or vice versa.

Why it happens: Browsers send a characteristic set of headers that must be internally consistent. A Chrome User-Agent with Firefox headers is a red flag.

Solution — Consistent browser profiles:

import random

BROWSER_PROFILES = [
    {
        "User-Agent":        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
        "Accept":            "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
        "Accept-Language":   "en-US,en;q=0.9",
        "Accept-Encoding":   "gzip, deflate, br",
        "sec-ch-ua":         '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',
        "sec-ch-ua-mobile":  "?0",
        "sec-ch-ua-platform":'"Windows"',
        "Sec-Fetch-Dest":    "document",
        "Sec-Fetch-Mode":    "navigate",
        "Sec-Fetch-Site":    "none",
        "Sec-Fetch-User":    "?1",
    },
    {
        "User-Agent":       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
        "Accept":           "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
        "Accept-Language":  "en-US,en;q=0.9",
        "Accept-Encoding":  "gzip, deflate, br",
        "sec-ch-ua":        '"Not_A Brand";v="8", "Chromium";v="119", "Google Chrome";v="119"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform":'"macOS"',
        "Sec-Fetch-Dest":   "document",
        "Sec-Fetch-Mode":   "navigate",
        "Sec-Fetch-Site":   "none",
    },
    {
        "User-Agent":       "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
        "Accept":           "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language":  "en-US,en;q=0.5",
        "Accept-Encoding":  "gzip, deflate, br",
        "Sec-Fetch-Dest":   "document",
        "Sec-Fetch-Mode":   "navigate",
        "Sec-Fetch-Site":   "none",
        "Sec-Fetch-User":   "?1",
        "TE":               "trailers",
        # Note: no sec-ch-ua — Firefox doesn't send it
    },
]

def get_random_profile() -> dict:
    """Return a complete, internally consistent browser profile."""
    return random.choice(BROWSER_PROFILES).copy()

Pattern 10: Extracting Data Across Multiple Formats on the Same Page

The problem: A single page contains data in mixed formats — some in an HTML table, some in JSON inside a script tag, some in plain text paragraphs.

Solution — Multi-strategy extraction with fallback chain:

import re, json
from bs4 import BeautifulSoup

def extract_price(html: str) -> float | None:
    """
    Try multiple strategies to find a price, in order of reliability.
    Returns the first successful result.
    """
    soup = BeautifulSoup(html, "lxml")

    # Strategy 1: Structured data (most reliable) — JSON-LD schema.org
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string)
            if isinstance(data, list):
                data = data[0]
            price = (data.get("offers", {}).get("price")
                     or data.get("price"))
            if price:
                return float(str(price).replace(",", ""))
        except Exception:
            continue

    # Strategy 2: Meta tags (Open Graph / Twitter Cards)
    meta_price = soup.find("meta", property="product:price:amount")
    if meta_price:
        try:
            return float(meta_price.get("content", "").replace(",", ""))
        except ValueError:
            pass

    # Strategy 3: data-price attributes
    for el in soup.select("[data-price], [data-product-price]"):
        val = el.get("data-price") or el.get("data-product-price")
        if val:
            try:
                return float(str(val).replace(",", ""))
            except ValueError:
                continue

    # Strategy 4: CSS class patterns
    for selector in [".price", ".product-price", ".a-price-whole", "[itemprop='price']"]:
        el = soup.select_one(selector)
        if el:
            text = el.get_text(strip=True)
            match = re.search(r"[\d,]+\.?\d*", text.replace(",", ""))
            if match:
                try:
                    return float(match.group())
                except ValueError:
                    pass

    # Strategy 5: Regex on full page text (last resort)
    text = soup.get_text()
    match = re.search(r"(?:price|cost)[\s:$£€₹]*?([\d,]+\.?\d*)", text, re.IGNORECASE)
    if match:
        try:
            return float(match.group(1).replace(",", ""))
        except ValueError:
            pass

    return None   # Couldn't find a price

Pattern 11: Handling Cloudflare's Bot Protection Without a Proxy

The problem: curl_cffi with TLS impersonation works on most sites but still fails on Cloudflare-protected pages that use JavaScript challenges.

Solution — Detect the protection level and choose the right tool:

import httpx
from curl_cffi import requests as cffi_requests

def detect_protection_level(url: str) -> str:
    """
    Detect what level of bot protection a site uses.
    Returns: "none", "tls", "js_challenge", "captcha"
    """
    try:
        # First try with vanilla httpx
        r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=10)
        if r.status_code == 200 and len(r.text) > 1000:
            return "none"
        if r.status_code == 403:
            if "cloudflare" in r.text.lower():
                if "challenge" in r.text.lower() or "cf-browser-verification" in r.text:
                    return "js_challenge"
                return "tls"
            return "tls"
        if r.status_code == 429:
            return "rate_limit"
    except Exception:
        pass
    return "unknown"

async def fetch_adaptive(url: str) -> str | None:
    """
    Automatically pick the right fetching strategy based on protection level.
    """
    level = detect_protection_level(url)
    print(f"  Protection level: {level}")

    if level == "none":
        import httpx
        r = httpx.get(url)
        return r.text if r.status_code == 200 else None

    elif level == "tls":
        # TLS fingerprint impersonation fixes this
        r = cffi_requests.get(url, impersonate="chrome120", timeout=20)
        return r.text if r.status_code == 200 else None

    elif level == "js_challenge":
        # Need a full browser to solve JavaScript challenges
        from playwright.async_api import async_playwright
        from playwright_stealth import stealth_async
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=True)
            page    = await browser.new_page()
            await stealth_async(page)
            await page.goto(url, wait_until="networkidle")
            html = await page.content()
            await browser.close()
        return html

    else:
        print(f"  Cannot automatically handle protection level: {level}")
        return None

Pattern 12: Storing Scraped Data Without Duplicates

The problem: You re-run your scraper and end up with duplicate rows in your database or CSV.

Solution — URL-based upsert pattern:

import sqlite3
import hashlib
from datetime import datetime, timezone

def create_dedup_store(db_path: str = "scrape_store.db"):
    """Create a SQLite store with built-in deduplication."""
    conn = sqlite3.connect(db_path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS items (
            id          INTEGER PRIMARY KEY AUTOINCREMENT,
            url_hash    TEXT UNIQUE NOT NULL,   -- MD5 of URL for fast lookup
            url         TEXT NOT NULL,
            title       TEXT,
            price       REAL,
            data_json   TEXT,                   -- Raw JSON for extra fields
            first_seen  TEXT DEFAULT (datetime('now')),
            last_updated TEXT DEFAULT (datetime('now'))
        )
    """)
    conn.execute("""
        CREATE INDEX IF NOT EXISTS idx_url_hash ON items(url_hash)
    """)
    conn.commit()
    return conn

def upsert_item(conn: sqlite3.Connection, url: str,
                title: str = None, price: float = None,
                extra: dict = None):
    """
    Insert a new item or update if the URL already exists.
    Tracks first_seen and last_updated separately.
    """
    import json
    url_hash = hashlib.md5(url.encode()).hexdigest()
    now      = datetime.now(timezone.utc).isoformat()

    conn.execute("""
        INSERT INTO items (url_hash, url, title, price, data_json, first_seen, last_updated)
        VALUES (?, ?, ?, ?, ?, ?, ?)
        ON CONFLICT(url_hash) DO UPDATE SET
            title        = excluded.title,
            price        = excluded.price,
            data_json    = excluded.data_json,
            last_updated = excluded.last_updated
    """, (url_hash, url, title, price,
          json.dumps(extra or {}), now, now))
    conn.commit()

# Usage
conn = create_dedup_store()
upsert_item(conn, "https://site.com/product/123", title="Widget", price=9.99)
upsert_item(conn, "https://site.com/product/123", title="Widget", price=8.99)  # Updates price, doesn't duplicate
upsert_item(conn, "https://site.com/product/456", title="Gadget", price=29.99)

import pandas as pd
df = pd.read_sql("SELECT * FROM items", conn)
print(df)
conn.close()

Pattern 13: Scraping Sites That Block Based on Geolocation

The problem: The site returns different content (or blocks you) based on your IP's country.

Solution — Geo-targeted requests:

from curl_cffi import requests as cffi_requests

# Country-specific proxy endpoints (from providers like Bright Data, Oxylabs)
COUNTRY_PROXIES = {
    "US": "http://user:pass@us.proxy.example.com:8080",
    "UK": "http://user:pass@uk.proxy.example.com:8080",
    "IN": "http://user:pass@in.proxy.example.com:8080",
    "DE": "http://user:pass@de.proxy.example.com:8080",
}

def fetch_from_country(url: str, country: str = "US") -> dict:
    """
    Fetch a URL appearing to originate from a specific country.
    Useful for price comparison, geo-restricted content, localisation testing.
    """
    proxy = COUNTRY_PROXIES.get(country.upper())
    if not proxy:
        raise ValueError(f"No proxy configured for country: {country}")

    headers = {
        "User-Agent":      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120",
        "Accept-Language": {
            "US": "en-US,en;q=0.9",
            "UK": "en-GB,en;q=0.9",
            "IN": "en-IN,en;q=0.9",
            "DE": "de-DE,de;q=0.9,en;q=0.8",
        }.get(country.upper(), "en-US,en;q=0.9")
    }

    r = cffi_requests.get(
        url,
        impersonate="chrome120",
        proxies={"https": proxy},
        headers=headers,
        timeout=20
    )
    return {"html": r.text, "status": r.status_code, "country": country}

# Compare prices across regions
def compare_regional_prices(product_url: str) -> dict:
    results = {}
    for country in ["US", "UK", "IN"]:
        response = fetch_from_country(product_url, country)
        soup     = BeautifulSoup(response["html"], "lxml")
        price_el = soup.select_one(".price, [data-price]")
        results[country] = price_el.get_text(strip=True) if price_el else "N/A"
    return results

Pattern 14: Detecting When a Scraper Has Gone Silent

The problem: Your scraper runs on a schedule but you don't know if it's actually collecting data. It might be blocked, broken, or returning empty results — silently.

Solution — Health check with alerting:

import sqlite3
import smtplib
from datetime import datetime, timezone, timedelta
from email.mime.text import MIMEText

def health_check(db_path: str, alert_email: str, expected_hourly_rate: int = 100):
    """
    Check if the scraper is producing data at the expected rate.
    Sends an alert email if the rate drops below threshold.
    """
    conn = sqlite3.connect(db_path)

    # How many items were scraped in the last hour?
    one_hour_ago = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
    count = conn.execute(
        "SELECT COUNT(*) FROM items WHERE last_updated > ?", (one_hour_ago,)
    ).fetchone()[0]
    conn.close()

    status = {
        "timestamp":      datetime.now(timezone.utc).isoformat(),
        "items_last_hour":count,
        "expected_rate":  expected_hourly_rate,
        "health":         "OK" if count >= expected_hourly_rate * 0.5 else "DEGRADED",
    }

    print(f"Health: {status['health']} — {count} items in last hour "
          f"(expected ≥{expected_hourly_rate * 0.5:.0f})")

    if status["health"] != "OK":
        _send_alert(alert_email, status)

    return status

def _send_alert(to_email: str, status: dict):
    """Send a health alert email."""
    msg = MIMEText(
        f"Scraper health check FAILED\n\n"
        f"Timestamp:    {status['timestamp']}\n"
        f"Items/hour:   {status['items_last_hour']} (expected ≥{status['expected_rate'] * 0.5:.0f})\n"
        f"Status:       {status['health']}\n\n"
        f"Check the scraper logs immediately."
    )
    msg["Subject"] = f"⚠️ Scraper Alert: {status['health']}"
    msg["From"]    = "monitor@yourdomain.com"
    msg["To"]      = to_email

    try:
        with smtplib.SMTP("smtp.gmail.com", 587) as s:
            s.starttls()
            s.login("monitor@yourdomain.com", "app_password")
            s.send_message(msg)
        print(f"Alert sent to {to_email}")
    except Exception as e:
        print(f"Alert send failed: {e}")

# Schedule this to run every 30 minutes via cron:
# */30 * * * * python health_check.py
if __name__ == "__main__":
    health_check("scrape_store.db", "you@yourdomain.com", expected_hourly_rate=200)

Pattern 15: Scraping GraphQL APIs

The problem: The site uses GraphQL instead of REST. The Network tab shows POST requests to /graphql with JSON bodies.

Solution:

import httpx
import asyncio
import json

async def graphql_query(endpoint: str, query: str, variables: dict = None,
                        headers: dict = None) -> dict:
    """
    Execute a GraphQL query against any endpoint.
    Works for both public and authenticated GraphQL APIs.
    """
    default_headers = {
        "Content-Type":  "application/json",
        "User-Agent":    "Mozilla/5.0 Chrome/120",
        "Accept":        "application/json",
    }
    if headers:
        default_headers.update(headers)

    payload = {"query": query}
    if variables:
        payload["variables"] = variables

    async with httpx.AsyncClient(headers=default_headers) as client:
        r = await client.post(endpoint, json=payload, timeout=20)
        r.raise_for_status()
        data = r.json()

    if "errors" in data:
        raise Exception(f"GraphQL errors: {data['errors']}")

    return data.get("data", {})

async def scrape_shopify_storefront_graphql(store_domain: str,
                                             storefront_token: str) -> list[dict]:
    """
    Scrape a Shopify store's products using the official Storefront GraphQL API.
    Requires a Storefront API token (available free from Shopify partner dashboard).
    """
    endpoint = f"https://{store_domain}/api/2024-01/graphql.json"
    headers  = {"X-Shopify-Storefront-Access-Token": storefront_token}

    query = """
    query GetProducts($first: Int!, $after: String) {
        products(first: $first, after: $after) {
            pageInfo {
                hasNextPage
                endCursor
            }
            edges {
                node {
                    id
                    title
                    handle
                    priceRange {
                        minVariantPrice { amount currencyCode }
                        maxVariantPrice { amount currencyCode }
                    }
                    availableForSale
                    tags
                    vendor
                }
            }
        }
    }
    """

    all_products = []
    cursor       = None

    while True:
        variables = {"first": 50, "after": cursor}
        data      = await graphql_query(endpoint, query, variables, headers)
        products  = data.get("products", {})
        edges     = products.get("edges", [])

        for edge in edges:
            node = edge["node"]
            all_products.append({
                "id":        node["id"],
                "title":     node["title"],
                "handle":    node["handle"],
                "min_price": node["priceRange"]["minVariantPrice"]["amount"],
                "currency":  node["priceRange"]["minVariantPrice"]["currencyCode"],
                "available": node["availableForSale"],
                "vendor":    node["vendor"],
            })

        page_info = products.get("pageInfo", {})
        if not page_info.get("hasNextPage"):
            break
        cursor = page_info.get("endCursor")
        print(f"  Fetched {len(all_products)} products...")

    return all_products

# Example usage
products = asyncio.run(scrape_shopify_storefront_graphql(
    store_domain     = "your-store.myshopify.com",
    storefront_token = "your-storefront-access-token",
))
print(f"Total products: {len(products)}")

Quick Reference

Pattern	Root cause	First thing to try
Element not in BeautifulSoup	JS rendering	Check DevTools XHR first; use Playwright if no API
Data in <script> tag	JS hydration	Parse NEXT_DATA or variable assignment regex
Infinite scroll	Intersection Observer	Find the pagination API endpoint
Login wall	Session required	POST login, save cookies, reuse
429 errors	Rate limiting	Adaptive backoff + jitter
No next link	URL-based pagination	Detect page number pattern; generate URLs
File downloads	N/A	Async streaming with resume support
Merged table cells	rowspan/colspan	Manual cell expansion algorithm
Still blocked with rotated UA	Header inconsistency	Use complete consistent browser profiles
Mixed data formats	Multiple data sources	Multi-strategy fallback chain
Cloudflare JS challenge	JavaScript execution required	Playwright + stealth
Duplicate records	Re-runs overwrite	URL-hash upsert in SQLite
Geo-blocked content	IP geolocation	Country-specific residential proxy
Silent scraper failure	No monitoring	Hourly health check with email alert
GraphQL site	Non-REST API	POST JSON to /graphql endpoint

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Developer‑Centric AI Hits a Tipping Point: Performance, Open Models, and Scaling Pain | The AI Daily Roundup

ZyVOP — Tue, 01 Sep 2026 17:30:08 +0000

One unifying trend: AI for software development is moving from novelty to core infrastructure

Across the board, the stories you read today reveal a single narrative: code‑centric AI is no longer a side project. Companies are rewriting runtimes in Rust, slashing context windows, raising massive funding rounds for open‑model runtimes, and even pausing new subscriptions because demand outstrips capacity. The ecosystem is transitioning from experimental add‑ons to a commodity that must meet the same reliability, performance, and cost expectations as traditional developer tools.

Why this matters for engineers, CTOs, and investors

When AI becomes a foundational layer of the software stack, three forces dominate:

Performance & efficiency. The Claude Code/Bun Rust port demonstrates that a 10% speed gain on Linux can be a competitive moat for AI‑assisted IDEs. Rust’s safety guarantees also reduce runtime crashes, a critical factor for large‑scale code migrations.
Open‑model democratization. Ollama’s $88 M raise and its “personal computer moment” signal that investors see a market for on‑premise, developer‑first model runtimes that avoid vendor lock‑in and API latency.
Infrastructure bottlenecks. Moonshot’s pause on new Kimi K3 subscriptions and Google’s Gemini delay underscore that scaling GPU‑heavy inference is a hard constraint. Companies that can provision compute efficiently will capture the most paying developers.

For senior engineers, the implication is clear: choosing an AI assistant now involves evaluating runtime language (Rust vs. JavaScript), context window limits (OpenAI’s Codex reduction from 372k to 272k tokens), and on‑premise vs. cloud deployment models.

Evidence from today’s headlines

Claude Code adopts a Rust‑based Bun runtime

Jarred Sumner’s rewrite of Bun in Rust, now used by Claude Code v2.1.181, delivers a measurable 10% speed boost on Linux. The move also showcases a broader industry shift toward systems‑level languages for AI tooling, where latency and memory safety directly affect developer productivity.

OpenAI trims Codex context size

OpenAI’s decision to shrink Codex’s context window by 100k tokens reflects a trade‑off between model capability and inference cost. Smaller windows reduce GPU memory pressure, allowing higher request throughput—an operational tweak that will ripple to any product built on Codex.

Moonshot AI pauses new Kimi K3 subscriptions

Demand for Kimi K3 surged beyond the provider’s GPU capacity, forcing a temporary halt on new sign‑ups. The response—splitting memberships into “Web/App” and “Code” tiers—highlights how providers must tier services to align compute with use‑case, a pattern we’ll see replicated across the market.

Ollama’s $88 M fundraising for open‑model tooling

Ollama’s rapid adoption (8.9 M developers) and fresh capital underscore investor confidence that open‑model runtimes will become the de‑facto SDK for AI‑enhanced software. Their “single‑command” approach mirrors Docker’s early impact on developer workflows, suggesting a similar paradigm shift.

Google’s Gemini delay due to coding regressions

Internal reports of Gemini 3.5 Pro lagging behind rivals because of coding setbacks illustrate how even AI giants stumble when their internal tooling and model‑training pipelines can’t keep pace with external competition.

Human over‑reliance: AI advice study

The study showing a 3× drop in accuracy but a 2× rise in confidence when users receive AI advice is a cautionary data point. As AI embeds deeper into dev workflows, the risk of “automation bias” grows, demanding better UI/UX safeguards and verification pipelines.

Who wins, who loses, and what changes next

Winners: Companies that invest in low‑latency, Rust‑based runtimes (e.g., Claude Code, Anthropic’s code migration tooling) and that offer on‑premise open‑model stacks (Ollama) will capture the most developer spend. Investors backing these infrastructures stand to reap outsized returns.

Losers: Vendors that rely solely on cloud‑only APIs with limited context windows (e.g., OpenAI’s Codex shrinkage) may see churn among power users who need longer prompts for complex refactoring. Likewise, firms that cannot provision GPU capacity fast enough (Google, Moonshot) risk losing market share to more agile competitors.

What’s next: Expect a wave of “hybrid” offerings—cloud‑backed inference with optional edge‑runtime bundles—plus tighter integration of verification layers to counteract over‑confidence in AI suggestions. Look for more announcements around Rust‑based inference engines, context‑window pricing tiers, and developer‑first pricing models that reflect actual compute consumption.

Takeaway for the audience

For senior engineers and CTOs, the strategic question is no longer “Should we use AI for code?” but “How do we architect our stack so that AI tooling scales reliably, cost‑effectively, and safely?” Investors should watch for capital flowing into runtime‑level optimizations and open‑model ecosystems, as they will likely define the next generation of software development platforms.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Open‑Weight AI Is Redefining the Competitive Landscape | The AI Daily Roundup

ZyVOP — Tue, 01 Sep 2026 17:29:02 +0000

Trend: Open‑Weight Models Are Disrupting the AI Status Quo

Across the day’s headlines, the common thread is a decisive move away from proprietary, closed‑source foundations toward openly released model weights. Whether it’s China’s strategic advantage, frontier labs beating Anthropic, or security researchers weaponising the latest GPT, the narrative is clear: openness is becoming the primary lever of competitive advantage.

Why Openness Matters Now

Open‑weight models lower the barrier to entry for anyone with compute, turning the model itself into a public good while re‑monetising the surrounding stack—hosting, APIs, fine‑tuning services, and enterprise integrations. This shifts value from “own the model” to “own the ecosystem.” The economic consequences are two‑fold:

Cost compression: Companies can skip expensive licence fees and allocate resources to data pipelines, safety tooling, or domain‑specific adapters.
Innovation acceleration: Researchers worldwide can iterate on a common baseline, leading to rapid breakthroughs (e.g., Kimi K3, Qwen 3.8, Claude Fable’s math proof).

At the same time, openness expands the attack surface, as demonstrated by AI‑driven exploit discovery.

Evidence From Today’s Stories

1. China’s Open‑Weight Strategy Is Winning

The Werd.io analysis argues that America’s “locked‑down” approach is unsustainable. Chinese firms are publishing weights (Moonshot’s Kimi K3, Alibaba’s Qwen 3.8) and rapidly capturing market share because enterprises can swap models without vendor lock‑in, focusing instead on service contracts and integration quality.

2. Frontier Labs Prove Open Models Can Reach the Frontier

According to Emerging Trajectories, Kimi K3 and Qwen 3.8 achieve performance comparable to Anthropic’s Fable 5 while planning to release weights publicly. The article explains that the biggest ongoing cost is inference electricity, not payroll, meaning that owning the inference infrastructure becomes the decisive advantage.

3. Open Models Enable New Security Threats

The SLCyber report shows exploit brokers paying half‑a‑million dollars for WordPress RCEs discovered with GPT‑5.6. By adapting a prompt that solved a hard math conjecture, researchers coaxed the model into generating a chain of vulnerable code, proving that open‑weight or even closed‑weight LLMs can be repurposed for offensive security at scale.

4. AI‑Generated Academic Content Is Already Saturating the Literature

The UnsloP study flags roughly one‑third of new arXiv papers as machine‑written. Open models lower the cost of producing publishable text, threatening the credibility of scholarly communication and forcing institutions to invest in detection and verification pipelines.

5. Voice‑First Consumer Apps Ride on Open Models

Cue AI’s desktop voice agent runs entirely on DeepMind’s open‑source Gemma 4, cutting latency by 44% and eliminating marginal inference costs. This illustrates a business model where the core AI is free, and revenue comes from UI polish, data privacy guarantees, and premium integrations.

Who Gains, Who Loses

Gainers: Chinese AI firms, open‑model labs (Moonshot, Alibaba, DeepMind’s Gemma), agile startups that can embed free models into niche products, and cloud providers that own the inference hardware.
Losers: U.S. incumbents that rely on licence revenue (OpenAI, Anthropic), investors betting on closed‑model valuations, and enterprises that have built long‑term contracts around proprietary APIs.
Collateral: Security teams face a surge in AI‑generated exploits; academic reviewers must grapple with synthetic papers; regulators confront a diffuse ecosystem where the “model” itself is not controllable.

What Changes Next?

We can expect three converging dynamics:

Service‑Layer Differentiation: Companies will double down on managed inference, compliance tooling, and data‑centric services to monetize open models.
Regulatory & Safety Arms Race: Governments will target the tooling around model deployment (e.g., export controls on high‑end chips, mandatory provenance logs) rather than the models themselves.
Security Market Expansion: As AI‑assisted vulnerability discovery becomes routine, firms offering AI‑powered red‑team platforms will see explosive growth, and “model‑hardening” will become a core product line.

The net effect is a shift from a monopoly‑based AI economy to a layered market where openness fuels competition, but value is extracted from the surrounding infrastructure and trust mechanisms.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Qwen3.8-Flash-Next Cost Efficiency, OpenExecutive Satire, and Multi-Vector Retrieval Advances

ZyVOP — Tue, 01 Sep 2026 15:36:12 +0000

The 60-Second Briefing

Qwen3.8-Flash-Next introduces a new architecture focused on "ultimate cost-efficiency," expanding Qwen Studio's capabilities in image/video understanding and tool utilization. Source
OpenExecutive, a viral open-source project, satirizes AI-led layoffs by deploying a virtual CEO backed by 8 specialist Claude agents, highlighting the tension between automation and workforce reduction. Source
Sentence-Transformers v6.0 releases the MultiVectorEncoder, enabling ColBERT-style late-interaction retrieval training on consumer hardware (RTX 3090). Source
OpenAI reports 215M daily ChatGPT messages from Brazil and 70M weekly learning-related conversations globally, signaling massive adoption in education and enterprise. Source
Market Dynamics: Reports indicate Anthropic's top-tier models face user acquisition challenges as cheaper alternatives gain traction, shifting focus to cost-performance ratios. Source

The Big Story: The Cost-Performance Pivot

The dominant AI trend this week is a shift toward cost-efficient model architecture, led by Qwen3.8-Flash-Next. As premium models get more expensive, cheaper systems are increasingly winning production workloads where top-tier reasoning isn't worth the added cost.

The OpenExecutive project offers both a technical and cultural counterpoint, using 8 specialist Claude agents to create a virtual executive team. Beyond satire, it demonstrates a practical multi-agent pattern: one coherent persona backed by specialized sub-agents for enterprise AI.

Key Takeaway: The market is bifurcating. Frontier models are becoming premium reasoning engines, while mid-tier models like Qwen3.8-Flash-Next are optimizing for high-volume, cost-sensitive applications. Engineers must now select models based on strict cost-per-token metrics rather than just benchmark scores.

Model Releases & Architecture Breakdown

The release of Sentence-Transformers v6.0 marks a significant milestone in retrieval-augmented generation (RAG) infrastructure. The introduction of the MultiVectorEncoder model type brings ColBERT-style late-interaction architectures into the mainstream training pipeline. Unlike traditional dense retrieval, which uses a single vector per document, MultiVectorEncoder projects each token to a small vector (typically 64-dim), allowing for finer-grained matching via the MaxSim operator.

Model/Component	Architecture Type	Key Specification	License/Availability
Qwen3.8-Flash-Next	Cost-Optimized LLM	Focus on inference efficiency; multimodal (image/video)	Proprietary (Qwen Studio)
MultiVectorEncoder	ColBERT-style Late Interaction	Token-level vectors (64-dim); MaxSim scoring	Open Source (Sentence-Transformers v6.0)
OpenExecutive Agents	Multi-Agent Orchestration	8 Specialist Claude Agents; FastAPI + Next.js	Open Source (GitHub)
GPT-4o (ChatGPT)	Proprietary LLM	Used in educational studies; 215M daily messages in Brazil	Proprietary (OpenAI)

The MultiVectorEncoder is particularly notable for its accessibility. The reference implementation was trained on a single NVIDIA RTX 3090 (24GB VRAM) in 14.5 hours, making advanced retrieval techniques viable for teams without large-scale GPU clusters. This democratization of retrieval quality is a critical shift for RAG pipelines, where late-interaction models often outperform dense retrieval in precision but were previously too expensive to train and deploy.

Developer Tooling & Production Patterns

For engineering teams, the immediate actionable insight is the integration of MultiVectorEncoder into existing RAG stacks. The following code snippet demonstrates how to initialize and train a multi-vector encoder using the new Sentence-Transformers v6.0 API. This pattern is ideal for domains requiring high-precision retrieval, such as medical or legal document search, where token-level matching is crucial.

from sentence_transformers import MultiVectorEncoder, SentenceTransformerTrainer, TrainingArguments
from sentence_transformers.evaluation import EmbeddingEvaluator

# Initialize the multi-vector encoder
model = MultiVectorEncoder("bert-base-uncased")

# Define training arguments for a single GPU setup
training_args = TrainingArguments(
    output_dir="./multi-vector-model",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    learning_rate=2e-5,
    fp16=True  # Enable mixed precision for RTX 3090
)

# Initialize the trainer
trainer = SentenceTransformerTrainer(
    model=model,
    args=training_args,
    train_dataset=your_training_dataset,
    eval_dataset=your_eval_dataset,
    evaluator=EmbeddingEvaluator()
)

# Start training
trainer.train()

Additionally, the OpenExecutive project offers a blueprint for multi-agent systems. By using FastAPI for the backend and Next.js for the frontend, it demonstrates a clean separation of concerns. The "single coherent executive persona" is achieved by routing queries to 8 specialist Claude agents, each with a specific system prompt and toolset. This pattern is highly reusable for customer support or internal knowledge management systems where specialized expertise is required.

Industry Signals, Security & Research

OpenAI's expansion in Brazil provides a clear signal of the global scale of LLM adoption. With 215 million daily ChatGPT messages from Brazil alone, and 70 million weekly conversations related to learning, the demand for AI in education and enterprise is undeniable. The deployment of ChatGPT Edu at institutions like ITA, with credits for "reasoning models and Codex," suggests that advanced capabilities are becoming standard in academic settings.

However, the market is not without friction. Reports of Anthropic's top models struggling to attract users as cheaper tools thrive highlight a critical risk: price sensitivity. For many developers, the difference between a top-tier model and a mid-tier model is negligible for common tasks, making cost the primary decision factor. This dynamic is likely to accelerate the adoption of open-source and cost-optimized models like Qwen3.8-Flash-Next.

From a security perspective, the rise of multi-agent systems like OpenExecutive introduces new attack surfaces. If a "virtual CEO" can be manipulated via prompt injection, the consequences could be significant. Engineers must implement strict input validation and output filtering for any multi-agent deployment, especially when agents have access to sensitive tools or data.

What to Build & Watch Next Week

Build: Prototype a RAG pipeline using MultiVectorEncoder from Sentence-Transformers v6.0. Compare its NDCG@10 against your current dense retrieval model on a domain-specific dataset.
Build: Experiment with the OpenExecutive architecture. Create a multi-agent system for a specific business function (e.g., customer support) using 3-5 specialized agents and a central orchestrator.
Watch: Monitor Qwen3.8-Flash-Next benchmarks. Look for independent evaluations that compare its cost-per-token against other mid-tier models like Llama 3.1 or Mistral.
Watch: Track OpenAI's enterprise adoption metrics. The 5x YoY growth in ChatGPT Enterprise seats in Brazil is a strong indicator of enterprise AI maturity. Look for similar trends in other regions.
Watch: Keep an eye on the "AI CEO" narrative. As more companies experiment with AI-led management, expect more open-source projects and discussions on the ethical and practical implications of AI in leadership roles.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Debian Adopts "Responsible Use of Generative AI" After Nine-Way Condorcet Vote

ZyVOP — Tue, 01 Sep 2026 08:37:17 +0000

Debian closed the polls on General Resolution 2026-002 on August 28, 2026. The project put nine options on AI-assisted contributions to a Condorcet vote, and the outcome settles months of mailing-list argument: Debian will neither ban generative AI tools nor formally endorse them. Option 5, "Responsible Use of Generative AI," proposed by developer Marc Haber, won outright against eight rivals, including a Social Contract amendment that would have banned LLM-assisted work. The full beat matrix is posted by Debian Project Secretary Kurt Roeckx at vote.debian.org/~secretary/gr_llm/results.txt.

How Debian Reached a Decision

Condorcet voting decides Debian's general resolutions. Every ballot ranks all nine options, and the project secretary tallies every pairwise matchup to find the Schwartz Set: the smallest group of options that beats everything outside it. One option, a proposed Social Contract change, needed a 3:1 supermajority; the rest needed only a simple majority.

Discussion ran from July 23 to August 13, 2026, and voting followed from August 15 through August 28, as the official vote page lays out.

Two options failed to beat None of the Above (NOTA). Matthias Geiger's Option 1 would have amended the Social Contract to ban LLM-assisted packaging, documentation, and official communication. It reached quorum, but its win ratio against NOTA was only 0.560, far short of the 3:1 bar a Foundation Document change requires.

Ian Jackson's Option 3 would have required messages to humans, from bug reports to mailing-list posts, to be drafted without LLM assistance under the Code of Conduct. It lost to NOTA outright at a 0.765 ratio, per Roeckx's posted tally.

The vote drew 425 unique voters against a quorum threshold of roughly 47, according to the secretary's live statistics page. Every substantive option cleared that bar comfortably.

In the deciding pairwise round, Option 5 beat Lucas Nussbaum's Option 2 by 55 votes, 203 to 148. It then beat Pierre-Elliott Becue's Option 4, which limited AI guidance to Debian-specific work, by 117 votes, 232 to 115. Its widest margin came against None of the Above itself, 281 to 126, a gap of 155 votes in the full results.

What "Responsible Use of Generative AI" Requires

Haber's winning text, formally Proposal E, does not take a side on AI tooling itself. It states that Debian holds a neutral position on generative AI in development, packaging, and documentation: neither endorsing it nor prohibiting it. Every contribution is held to the same standards regardless of how it was produced, as the full proposal text spells out.

Disclosure of AI assistance is encouraged but not required, a lighter touch than Nussbaum's rival Option 2, which would have made disclosure mandatory whenever a tool substantially generated a contribution.

The resolution also addresses data handling: contributors should not feed embargoed security reports, credentials, cryptographic keys, or other non-public Debian material into third-party AI services without explicit authorization. On the harder question of copyright status for AI-generated output, the text declines to take a position, leaving that to individual jurisdictions and existing licensing rules.

Because it is a delegate-power statement rather than a Social Contract or Code of Conduct change, Debian can revise it later without that higher procedural bar.

A Closely Split Community, and Company Elsewhere

One commenter in the discussion thread on LWN read the published preference graph after the results posted and estimated that roughly three in ten voters had ranked every anti-AI option above the rest. It is a reminder that a clear Condorcet winner can still sit atop a closely divided community.

Debian is not the first project to weigh in on generative AI. Geiger's Option 1, formally Proposal A, cited outside precedent, including Gentoo's AI policy, adopted by the Gentoo Council in April 2024, which forbids any AI-assisted contribution outright, and a GNOME thread where the Loupe image viewer stopped accepting generative AI contributions in February 2025. GNOME Calendar later adopted a similar no-LLM statement.

Gard Spreemann's Option 7, Proposal G, separately drew on GCC's AI policy and the Rust project's LLM usage policy as reference points. Those examples span from outright bans to light-touch guidance, and Debian's 425 voters, based on the secretary's tally, landed on the permissive end of that range: no ban, no mandatory disclosure, and accountability resting with whoever runs the commit.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

How I Built a Real-Time Developer Trend Radar Into My SEO Growth Engine

ZyVOP — Mon, 31 Aug 2026 12:23:05 +0000

Most SEO tools tell you what already happened. They show you last month's rankings, last quarter's search volume, and keywords your competitors owned six months ago.

I wanted something different.

I wanted to look at what developers are talking about right now — the Hacker News thread with 700 points, the Dev.to post with 150 reactions, the search-query patterns appearing around a topic, the GitHub repository that has quickly attracted thousands of stars — and instantly turn those signals into blog content briefs I can execute on immediately.

So I built it. Here's how.

The Problem: Fresh Websites Have Zero Historical Data

When you launch a new developer blog or technical publication, Google Search Console (GSC) is practically empty. You might have a handful of impressions for your brand name and a few long-tail queries that nobody else searches for.

The typical SEO workflow looks like this:

This is a cold-start loop. You're waiting for data that depends on traffic you don't have yet.

The question became: What if I could bypass the cold-start entirely by pulling real-time demand signals from the places developers actually hang out?

The Architecture: Four Live Streams, One Unified Radar

The system I built aggregates live or recent data from four primary developer platforms into a single, queryable radar feed. The radar uses observed source signals rather than inventing engagement metrics. The separate keyword-discovery engine can also use model-generated estimates, but those are treated as estimates rather than measured search data.

The key architectural decision: live streams are cached in-memory, but content decisions are persisted to PostgreSQL. The radar shows you what's hot right now; once you decide to write about something, it becomes a permanent, trackable content opportunity with a full AI-generated content brief.

Data Source #1: Hacker News Front Page

Hacker News is arguably the most concentrated source of developer attention on the internet. A front-page post can attract substantial developer attention within hours.

I use the Algolia HN Search API to pull the current front page:

private async fetchHackerNewsTrends(): Promise<RealTrendingItem[]> {
  const res = await fetch(
    'https://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=25',
    { headers: { 'User-Agent': 'ZyVopSeoRadar/1.0' } },
  );

  const data = await res.json();

  return data.hits
    .filter((h) => h.title && h.points > 20)
    .map((h) => {
      // Extract real tags from HN Algolia _tags field
      // (filters out generic 'story', 'front_page', 'author_xyz')
      const rawTags = Array.isArray(h._tags)
        ? h._tags.filter((t) => !t.startsWith('author_') && t !== 'story' && t !== 'front_page')
        : [];
      const tags = rawTags.length > 0 ? rawTags : ['Tech', 'Engineering'];

      return {
        title: h.title,
        source: 'HACKER_NEWS',
        url: h.url || `https://news.ycombinator.com/item?id=${h.objectID}`,
        summary: `${h.points} points · ${h.num_comments || 0} comments`,
        tags,
        score: h.points,
        commentsCount: h.num_comments,
        publishedAt: h.created_at, // Real ISO timestamp from HN
        suggestedAngle: `Write an engineering deep dive addressing "${h.title}"...`,
      };
    });
}

The points > 20 filter is simply a practical noise threshold for the feed. A post with 700 points and 400 comments is a strong signal of visible community attention, although it does not by itself prove search demand.

What this gives you: Real-time awareness of what the developer community is debating right now. Topics like "Htmx 4.0", "GLM-5.3 is now open-weight", or "GUIs should be fully keyboard-driven" — these are the conversations you can join with a well-timed deep-dive article.

Data Source #2: Dev.to Trending Articles

Dev.to's top=7 feed surfaces popular articles from the previous 7 days. Unlike Hacker News (which skews toward links and discussions), Dev.to content is written by developers for developers — tutorials, opinion pieces, and how-to guides.

private async fetchDevToTrends(): Promise<RealTrendingItem[]> {
  const res = await fetch(
    'https://dev.to/api/articles?per_page=30&top=7',
    { headers: { 'User-Agent': 'ZyVopSeoRadar/1.0' } },
  );

  const data = await res.json();

  return data.map((d) => ({
    title: d.title,
    source: 'DEV_TO',
    url: d.url,
    summary: d.description,
    tags: d.tag_list,
    score: d.positive_reactions_count,
    commentsCount: d.comments_count,
    suggestedAngle: `Write a comprehensive, code-rich guide on "${d.title}"...`,
  }));
}

What this gives you: Validated content formats. If "10 Git Commands You'll Wish You Knew Earlier" has 178 reactions, you know that listicle-format developer productivity content resonates. You can write a more comprehensive version, targeting the same search intent with deeper technical substance.

Data Source #3: Google Autocomplete (Search-Intent Signal)

This is the most directly actionable signal for query discovery. Google Autocomplete reflects real searches, but its predictions can also depend on language, location, trending interest, and past searches. It is useful for discovering query patterns and emerging search intent, but it is not a direct search-volume metric. Google documents these factors here.

private async fetchGoogleSearchTrends(): Promise<RealTrendingItem[]> {
  const seedTerms = [
    'Next.js 15', 'AI agents', 'FastAPI', 'TypeScript',
    'PostgreSQL', 'Docker', 'DeepSeek R1', 'Rust programming',
  ];

  const results: RealTrendingItem[] = [];

  for (const term of seedTerms) {
    const url = `https://suggestqueries.google.com/complete/search` +
      `?client=chrome&q=${encodeURIComponent(term)}`;
    const res = await fetch(url, {
      headers: { 'User-Agent': 'Mozilla/5.0' },
    });
    const data = await res.json();

    // data[1] contains the autocomplete suggestions
    for (const query of data[1].slice(1, 4)) {
      results.push({
        title: query.trim(),
        source: 'GOOGLE_SEARCH',
        url: `https://www.google.com/search?q=${encodeURIComponent(query)}`,
        summary: `Google Autocomplete prediction for "${term}" — a search-intent signal, not a volume metric`,
        score: 0,            // No engagement score — autocomplete is a signal, not a post
        publishedAt: null,   // Query signal — there is no "published date"
      });
    }
  }

  return results;
}

An important design decision here: Google Autocomplete items have no engagement score or verified search-volume number. Unlike a Hacker News post (which has real points) or a Dev.to article (which has real reactions), an autocomplete suggestion is a search-intent signal. I deliberately set score: 0 and publishedAt: null instead of faking numbers — the frontend handles these cases with distinct labels ("Live Search Demand" and "Live Query") so the user knows exactly what kind of signal they're looking at.

For example, querying "Next.js 15" might return:

next.js 15 server actions
next.js vs react
next.js latest version
next.js tutorial

These are query patterns Google is surfacing around the seed topic. Writing a comprehensive article targeting a phrase such as "next.js 15 server actions" may align with emerging search intent, but the autocomplete result itself is not proof of current search volume.

What this gives you: Query discovery based on current autocomplete signals, without pretending those signals are equivalent to measured keyword volume.

Data Source #4: GitHub Breakout Repositories

New open-source projects with large star counts shortly after creation can signal emerging developer interest in a technology, pattern, or tool. The current query surfaces recently created repositories and sorts them by current star count; it does not yet measure star-growth velocity over time.

private async fetchGithubTrending(): Promise<RealTrendingItem[]> {
  const oneMonthAgo = new Date(Date.now() - 30 * 86400000)
    .toISOString().split('T')[0];

  const url = `https://api.github.com/search/repositories` +
    `?q=created:>${oneMonthAgo}&sort=stars&order=desc&per_page=12`;

  const res = await fetch(url, {
    headers: {
      'User-Agent': 'ZyVopSeoRadar/1.0',
      Accept: 'application/vnd.github.v3+json',
    },
  });

  const data = await res.json();

  return data.items.map((repo) => ({
    title: `${repo.full_name}: ${repo.description}`,
    source: 'GITHUB',
    url: repo.html_url,
    summary: `⭐ ${repo.stargazers_count.toLocaleString()} stars · ${repo.language}`,
    tags: [repo.language, 'Open Source', 'GitHub Trending'],
    score: Math.min(Math.round(repo.stargazers_count / 10), 1000),
    suggestedAngle: `Write a technical review or getting-started walkthrough...`,
  }));
}

What this gives you: An early publishing opportunity. A getting-started guide for a newly popular repository can give you a head start while search results are still relatively sparse, but it does not guarantee rankings.

The Backend: NestJS Service with In-Memory Caching

All four data sources are fetched concurrently using Promise.allSettled(). This means a timeout or failure in one source doesn't block the others — you always get results from whichever sources respond.

The 5-minute in-memory cache (Map<string, CachedRadar>) prevents hammering external APIs on every dashboard refresh while keeping the data fresh enough for near-real-time editorial decision-making.

The Conversion Pipeline: From Trending Topic to Content Brief

The most powerful part isn't the radar itself — it's what happens when you click "Write Blog on This".

When you click the button, the system:

Creates a permanent SeoOpportunity record in PostgreSQL with type: CONTENT_GAP, priority: HIGH, and actionType: CREATE_PAGE.
Triggers the SeoAiService which sends the topic to Groq's openai/gpt-oss-120b model to generate a full content brief.
Returns an AI Content Brief containing:

-   Recommended SEO title and H1

-   Complete article structure with section headings

-   Key questions the article should answer

-   Required original value (code samples, benchmarks, diagrams)

-   Internal linking suggestions

-   Suggested call-to-action

The opportunity then appears in the Opportunities tab, where it flows through the full lifecycle: DETECTED → ANALYZED → RECOMMENDED → APPROVED → IMPLEMENTED → DEPLOYED → MEASURED.

Once the blog is published and deployed, the system can pull its Google Search Console performance data and compare impressions, clicks, CTR, and position deltas over 7-day and 30-day windows.

The GraphQL API Layer

The entire system is exposed through two GraphQL operations:

Query: Live Trending Radar

query GetLiveTrendingRadar($category: String) {
  getLiveTrendingRadar(category: $category) {
    lastUpdated
    items {
      title
      source
      url
      summary
      tags
      score
      commentsCount
      publishedAt
      suggestedAngle
    }
  }
}

Mutation: Convert to Content Opportunity

mutation ConvertTrendingToOpportunity(
  $title: String!
  $tags: [String]
  $source: String
  $url: String
) {
  convertTrendingToOpportunity(
    title: $title
    tags: $tags
    source: $source
    url: $url
  ) {
    id
    targetQuery
    opportunityScore
    priority
    status
    contentRecommendation {
      contentBrief {
        recommendedTitle
        recommendedH1
        suggestedStructure
        questionsToAnswer
        requiredOriginalValue
      }
    }
  }
}

Both operations are protected by GqlAuthGuard and RolesGuard with @Roles('ADMIN'), ensuring only authenticated administrators can access the radar and create content opportunities.

The Frontend: A Real-Time Dashboard Tab

The frontend is a Next.js 16 React component that provides:

Source filtering — Toggle between All Sources, Hacker News, Dev.to, Google Search, and GitHub
Category filtering — Filter by AI & LLMs, React & Next.js, Python & Backend, DevOps, Rust, or PostgreSQL
Search — Full-text search across titles, tags, and summaries
Source badges — Color-coded indicators showing where each trending item originated, with real engagement metrics (HN points, Dev.to reactions, GitHub star counts)
Relative timestamps — Each card shows when the item was published ("23h ago", "2d ago") or "Live Query" for Google Autocomplete items that have no publish date
Dynamic engagement labels — Instead of a static "High Viral Potential" on every card, each item gets a label based on its actual score:
- Viral Buzz (≥500 points/reactions) — red
- High Engagement (≥200) — orange
- Rising Interest (≥50) — green
- Emerging Topic (<50) — gray
- Live Search Demand (Google Autocomplete) — blue
Suggested writing angles — Content angle suggestions tailored to each trending topic
One-click conversion — The "Write Blog on This" button that triggers the full content brief pipeline

The timeAgo() helper formats real ISO timestamps from each API into human-readable relative dates:

function timeAgo(dateStr: string | null | undefined): string {
  if (!dateStr) return 'Live Query';
  const seconds = Math.floor((Date.now() - new Date(dateStr).getTime()) / 1000);
  if (seconds < 60) return 'just now';
  if (seconds < 3600) return `${Math.floor(seconds / 60)}m ago`;
  if (seconds < 86400) return `${Math.floor(seconds / 3600)}h ago`;
  if (seconds < 604800) return `${Math.floor(seconds / 86400)}d ago`;
  return new Date(dateStr).toLocaleDateString('en-US', { month: 'short', day: 'numeric' });
}

The component uses Next.js Server Actions for the data flow. When the radar loads, fetchLiveTrendingRadarAction is called, which hits the backend GraphQL endpoint. When a user clicks "Write Blog on This", convertTrendingToOpportunityAction persists the opportunity and triggers AI analysis — all through Server Actions without any client-side GraphQL setup.

The Hybrid Keyword Discovery Engine

Alongside the real-time radar, the system includes a keyword discovery engine that works without requiring Google Ads API credentials — a common blocker for independent developers and small publications.

If Google Ads credentials are configured and working, the system uses the official Keyword Planner API. If they're not available, it falls back to:

Google Autocomplete — Fetching suggested queries for each seed keyword
Groq AI — Expanding those queries and classifying intent. Any volume, CPC, competition, or trend figures produced by the LLM are estimates, not measured Google keyword metrics.

The fallback produces keyword ideas that can be actionable for research, but any LLM-generated search-volume figures are estimates and should not be treated as measured demand. Each keyword is saved to the seo_keywords table, and the opportunity engine can create CONTENT_GAP opportunities for keywords that meet the configured threshold when a trustworthy avgMonthlySearches value is available.

The Data Model

The content opportunity lifecycle is tracked across three main entities:

The SeoOpportunity entity stores the full AI analysis and content recommendation as JSONB columns, making them queryable and flexible without requiring schema migrations for every new field the AI model returns.

Lessons Learned

1. `Promise.allSettled()` Over `Promise.all()`

External APIs are unreliable. GitHub might rate-limit you. Hacker News might be slow. Using Promise.allSettled() means you always get results from the sources that responded, instead of failing entirely because one source timed out.

2. In-Memory Cache for Live Feeds, PostgreSQL for Decisions

The radar feed changes every few minutes. Caching it for 5 minutes in memory prevents excessive API calls while keeping the data fresh. But once a user decides "I want to write about this topic," that decision is persisted permanently. The radar is ephemeral; content strategy is persistent.

3. Math.round() Before PostgreSQL Integer Columns

A subtle bug taught me this the hard way. TypeORM @Column({ type: 'int' }) columns in PostgreSQL strictly reject floating-point numbers. If you calculate impressions * 1.5 = 109.5 and try to save it, PostgreSQL throws invalid input syntax for type integer: "109.5". Always wrap computed values with Math.round() before saving to integer columns.

4. Google Ads Test Accounts Return Bucketed Ranges

Depending on account access and API response, Google Ads Keyword Planner metrics may be returned as ranges rather than exact values; use the values provided by the API rather than inventing precision. For a developer blog, the AI + Google Suggest fallback can still be useful for discovery when measured Keyword Planner data is unavailable.

5. Cannibalization Detection Needs High Thresholds

On a new site, nearly every query appears on multiple pages because you have so few pages. With a threshold of impressions >= 1, every single query was flagged as a "cannibalization alert." Raising the threshold to impressions >= 200 && urls.length > 1 eliminated the false positives entirely.

6. Never Fake Engagement Metrics

My first version hardcoded score: 350 on every Google Autocomplete result to make them appear alongside Hacker News posts (which have real point counts of 200-800). This was misleading — it made autocomplete suggestions look like they had engagement they didn't have, and it broke the sorting logic by inflating Google items above genuinely viral HN posts.

The fix was simple: set score: 0 for autocomplete items and handle the display differently in the frontend. Google Autocomplete signals are valuable for a completely different reason (search intent) than HN posts (community validation). They shouldn't compete on the same axis. The frontend now shows "Live Search Demand" in blue for these items instead of trying to rank them by a fake score.

The Tech Stack

Layer	Technology
Backend Framework	NestJS 11 + Fastify
Database	PostgreSQL + TypeORM
API	GraphQL (Apollo)
AI / LLM	Groq SDK (openai/gpt-oss-120b)
Frontend	Next.js 16 + React 19
Search Console	Google Search Console API
Live Data	Hacker News Algolia API, Dev.to API, Google Autocomplete, GitHub Search API
Caching	In-memory Map with 5-minute TTL
Styling	Vanilla CSS with dark/light mode support

What's Next

The radar currently streams and displays. The next evolution is to add:

Automated daily digests — A BullMQ job that runs the radar every morning and emails the top 10 trending topics with pre-generated content briefs
Trend velocity scoring — Tracking how fast a topic is accelerating across sources (a topic trending on HN, Dev.to, AND Google simultaneously gets a higher signal score)
Competitor content gap analysis — Cross-referencing trending topics against what competitors have already published to find uncovered angles
Auto-draft generation — Using the AI content brief to generate a full first draft that goes straight into the CMS as a review-ready post

Try It Yourself

The entire system is built with publicly available APIs. You don't need any paid API keys to get started:

Hacker News: https://hn.algolia.com/api/v1/search?tags=front_page
Dev.to: https://dev.to/api/articles?per_page=30&top=7
Google Autocomplete: https://suggestqueries.google.com/complete/search?client=chrome&q=YOUR_TERM
GitHub: https://api.github.com/search/repositories?q=created:>DATE&sort=stars

The value isn't in the individual data sources — it's in aggregating them into a single decision interface and connecting that interface to a content pipeline that turns attention signals into published articles.

Stop relying entirely on yesterday's SEO data. Use live community and search signals to find what is emerging now.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Anthropic’s Pricing Shock, Granite 4.2 Open‑Source Leap, and AI‑Powered Security & Policy Shifts

ZyVOP — Thu, 27 Aug 2026 17:49:16 +0000

The 60-Second Briefing

Anthropic’s flagship Claude Fable 5 sees enterprise resistance as companies shift workloads to half-price Claude Opus 5 and existing Opus 4.8 tiers (Financial Times).
IBM launches open-source Granite 4.2 featuring a native 128K context window (extensible to 512K) and native <think> reasoning modes under the Apache 2.0 license (Hugging Face).
Multi-model AI reverse-engineering: A developer spent $266 combining Claude, Kimi K3, and GLM to reverse-engineer Fire OS firmware and regain root access on Amazon hardware (Eric Pardee).
Frontier models stumble on abstract reasoning: A Columbia & Stony Brook study reveals top LLMs solved only ~18% of NYT Connections puzzles, lagging significantly behind human players (MIT Tech Review).
Google expands multimodal visual Search with Circle to Search and in-room furniture visualization for visual commerce (Google Blog).

The Big Story: Enterprise Model Routing & The Pricing Plateau

Anthropic’s most capable frontier model—Claude Fable 5 (released June 2026)—is encountering strong economic resistance across enterprise budgets.

According to transaction data analyzed by the Financial Times across 70,000 corporate accounts on Ramp, Fable 5 captured only approximately 11% of total spend on Anthropic models, a figure that has plateaued since launch. Instead, enterprises are rapidly adopting Claude Opus 5 (launched at half the price of Fable 5) and routing routine queries through smaller, cheaper tiers like Opus 4.8 and Sonnet.

Key Market Shift: The enterprise procurement market has moved past "defaulting to the flagship." Teams now prioritize dynamic multi-model routing over raw benchmark maximalism.

From an architectural standpoint, while flagship models offer superior performance on complex multi-step reasoning, standard enterprise tasks (summarization, retrieval, classification) show minimal perceptible delta when executed on models half the price.

Actionable Takeaways for Engineers:

Implement Semantic Routing: Use lightweight intent classifiers or embedding-based routers to send routine queries to cost-efficient models (e.g. Claude Opus 5, Sonnet, or self-hosted open-weights) while reserving top-tier frontier models strictly for edge cases.
Adopt Model-Agnostic Gateways: Deploy inference proxies (like vLLM or LiteLLM) to switch providers and model tiers dynamically via configuration without touching application logic.
Instrument Per-Route Cost Telemetry: Track cost_per_request and token_efficiency by task type to quantify the actual ROI before upgrading production endpoints to new flagship checkpoints.

Model Releases & Architecture Breakthroughs

IBM Granite 4.2 Family

IBM released its open-source Granite 4.2 family in 3B, 8B, and 30B parameter sizes under the Apache 2.0 license. Built specifically for reasoning and agentic workflows, the models introduce native <think>...</think> chain-of-thought tags with three configurable modes:

Full Thinking (Default): Deep multi-step deliberation for complex reasoning and mathematical verification.
Low-Effort Thinking: Bounded deliberation for low-latency interactive pipelines.
Non-Thinking: Direct generation for high-throughput extraction tasks.

Technical Specifications: Native 128K token context window (extensible to 512K), trained with agentic reinforcement learning for native tool calling and terminal execution. Out-of-the-box support for vLLM, llama.cpp, and Ollama.

Developer Tooling & Open Source Ecosystem

Democratizing Internal UI Prototyping

Internal AI tooling is shifting from centralized engineering bottlenecks to self-serve developer platforms. Travel company loveholidays highlighted their internal Search Playground, which integrates OpenAI Codex on top of their proprietary frontend design system.

The platform allows product managers and designers to prototype live search components with automated safety rails—every AI-generated prototype runs through pre-commit hooks containing ruff and pytest before hitting staging environments.

Industry Signals, Security & Research

Multi-Model Firmware Reverse-Engineering

Security researcher Eric Pardee documented a technical saga spending ~$266 across four AI models (Claude, Kimi K3, GLM-5.2, and GLM-5.3) to calculate memory offsets and reverse-engineer unpatched kernel vulnerabilities in Fire OS 7.3.2.6 on an Amazon Fire HD 10 tablet.

This workflow demonstrated how combining complementary LLM reasoning engines enables solo developers to accelerate complex firmware reverse-engineering tasks that previously required weeks of dedicated hardware lab analysis.

The Abstract Reasoning Gap

A joint study by researchers at Columbia University and Stony Brook University tested frontier LLMs against the New York Times Connections puzzle.

The study found that even leading models (including Claude 3.5 Sonnet) solved only ~18% of the puzzles, consistently trailing novice human players. The failure mode stems from the game's reliance on non-literal word associations, multi-meaning puns, and contextual distractors—highlighting that pattern-matching on training corpora still falls short of true abstract categorization.

What to Build & Watch Next Week

Short-Term Builds:

Deploy a Local Granite-4.2-8B Instance: Use Ollama or vLLM to benchmark the model's <think> reasoning tags against your internal tool-calling pipelines.
Add Multi-Model Tier Routing: Benchmark routing 80% of routine traffic to cost-effective models to achieve immediate 40–60% API cost reductions.
Integrate Linting into AI Generation Pipelines: Add automated ruff / syntax validation to all code-generation endpoints before serving outputs to users.

Watch For:

Enterprise Pricing Adjustments: How frontier labs adjust pricing tiers and model sizes in response to enterprise adoption data.
Long-Context Benchmark Evolutions: Real-world retrieval quality evaluations on 512K+ context windows.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

AI at Scale: Cost Cuts, New Tooling, and Growing Governance Rules

ZyVOP — Sat, 01 Aug 2026 16:08:14 +0000

Trend: AI is being forced into production at scale

Across the board, today’s headlines reveal a single narrative: AI is no longer a research curiosity but a commodity that must be affordable, operable, and compliant. Companies are slashing model prices, building orchestration tools, and tightening contribution policies—all to make large‑language models (LLMs) viable for everyday business workloads while limiting legal and security exposure.

Price‑performance race – OpenAI’s GPT‑5.6 pricing cuts

OpenAI announced that its new GPT‑5.6 Luna will cost 80 % less and the balanced Terra 20 % less. The move directly addresses the biggest barrier to large‑scale adoption: compute spend. By delivering “more intelligence per dollar,” OpenAI is courting enterprises that need high‑throughput, multi‑step workflows (e.g., automated ticket triage, code generation) without blowing budgets. The benefit is clear – startups and mid‑market firms can now run high‑quality LLM‑driven services that were previously reserved for the cloud‑giants.

Open‑source models prove they can stay uncensored – DeepSeek distillation

CTGT’s experiment distilling DeepSeek V4 Flash into GPT‑OSS‑120B shows that a model trained on outputs of a heavily censored Chinese system can retain financial‑reasoning performance without inheriting the same censorship. This demonstrates that open‑source “teacher‑student” pipelines can decouple capability from geopolitical bias, a crucial insight for companies that need transparent, auditable models while avoiding export‑control entanglements.

Tooling for agents – Agent‑Manager, Grafana AI SDK, Gemini Robotics 2

Operationalizing agents requires orchestration. The Agent‑Manager TUI lets engineers run Claude, Codex, and other agents side‑by‑side in tmux, providing live status, cost gauges, and diff‑based reviews. Similarly, Grafana’s AI SDK for Go offers a wire‑compatible streaming and tool‑calling layer that bridges Go back‑ends with React front‑ends, lowering the engineering effort to embed LLMs in production services. DeepMind’s Gemini Robotics 2 pushes the same orchestration principle into robotics, giving whole‑body intelligence a unified API. Together these projects signal that the ecosystem is maturing from ad‑hoc scripts to disciplined, observable pipelines.

Governance tightening – GCC, OpenJDK, EU platform rules

At the same time, foundational projects are drawing hard lines. The GCC steering committee adopted an AI contributions policy that rejects any “legally significant” LLM‑generated code, effectively banning large‑scale patch contributions. OpenJDK issued an interim policy with identical language, allowing private AI‑assisted debugging but prohibiting AI‑generated commits. In Europe, the EU’s Digital Services Act is being extended to platforms like ChatGPT and Roblox, imposing stricter transparency and safety obligations. The common thread is risk mitigation: intellectual‑property exposure, reviewer fatigue, and geopolitical bias are now seen as operational liabilities that must be codified.

Reality check – Autonomous business experiment fails

Even with these advances, the autonomous‑agent hype remains unproven. Bottleneck Labs gave GPT‑5.6 Sol a real‑world startup challenge and the agent lost $447, generated zero revenue, and only marginally grew its user base. The failure underscores that cost‑effective models and sophisticated tooling are necessary but not sufficient; robust business logic, alignment, and risk‑aware governance are still missing.

What changes next?

We can expect three converging forces:

Price pressure will intensify. Competitors (Anthropic, Google DeepMind) will match or beat OpenAI’s discounting, forcing a race to the bottom that benefits high‑volume SaaS users.
Tooling will become standardized. SDKs like Grafana’s and orchestration UIs will evolve into de‑facto platforms, reducing the friction of deploying multi‑step agents at scale.
Governance will harden. More open‑source foundations will adopt contribution bans, and regulators will demand audit trails for any LLM‑generated code shipped to production.

Companies that align their engineering pipelines with these realities—by adopting cost‑aware models, integrating proven orchestration layers, and embedding compliance checks—will capture the emerging “production AI” market. Those that ignore the policy wave or rely on untested autonomous agents risk both financial loss and legal exposure.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Scrapling: The Python Scraper That Doesn't Break When Sites Change (2026)

ZyVOP — Fri, 31 Jul 2026 10:06:33 +0000

The Problem Every Python Scraper Has

You spend a Tuesday afternoon writing a scraper. CSS selectors, BeautifulSoup, a loop, a CSV export. It works perfectly.

You come back three weeks later. The data is missing. The site redesigned. The class .product-price is now .price-display. Your soup.select_one(".product-price") returns None across all 50,000 URLs you were monitoring.

This is not an edge case. It is the normal lifecycle of every production scraper. D4Vinci/Scrapling — Python web scraper that adapts when sites change, so your data collection keeps working.

Scrapling is the answer to this problem. Instead of finding elements by their exact CSS class — which breaks the moment a developer renames it — Scrapling finds elements by their structural position and visual role in the document. The class name is a hint, not the whole answer. When the hint changes, Scrapling adapts.

By GitHub stars: Firecrawl (131k+), Playwright (90.2k+), Crawl4AI (68.2k+), Scrapy (62.1k+). Scrapling has been growing fast precisely because it solves the problem that every scraper maintainer hates most.

This guide shows you the whole library — the parsing layer (Adaptor), the killer adaptive features (find_similar, auto-generated selectors), and how Scrapling's fetchers handle real websites.

Installation

pip install scrapling curl_cffi

curl_cffi is required by Scrapling's fetchers for TLS fingerprint impersonation. Install both together.

Part 1: The Adaptor — Scrapling's HTML Parser

The Adaptor is Scrapling's HTML parsing layer. Think of it as BeautifulSoup with a smarter element model and additional methods for adaptive scraping.

from scrapling.parser import Adaptor

html = """
<html><body>
  <div class="job-card" data-id="101">
    <h3 class="job-title">Senior Python Developer</h3>
    <span class="company">Infosys</span>
    <span class="location">Bangalore</span>
    <span class="salary">Rs. 18–25 LPA</span>
    <span class="tag">Python</span>
    <span class="tag">Django</span>
    <a href="/jobs/101" class="apply-btn">Apply Now</a>
  </div>
  <div class="job-card" data-id="102">
    <h3 class="job-title">Data Scientist</h3>
    <span class="company">TCS</span>
    <span class="location">Hyderabad</span>
    <span class="salary">Rs. 15–22 LPA</span>
    <span class="tag">Python</span>
    <span class="tag">ML</span>
    <a href="/jobs/102" class="apply-btn">Apply Now</a>
  </div>
</body></html>
"""

# Create an Adaptor — pass the HTML and the source URL
# The URL is used for resolving relative links and for the auto-trained model
page = Adaptor(html, url="https://techjobs.example.com")

Selecting elements

Scrapling supports CSS selectors and XPath:

# CSS selectors — returns a Selectors list
titles = page.css("h3.job-title")
print([t.text for t in titles])
# ['Senior Python Developer', 'Data Scientist']

# XPath — same return type
titles_xpath = page.xpath("//h3[@class='job-title']")
print([t.text for t in titles_xpath])
# ['Senior Python Developer', 'Data Scientist']

# find_all by tag — like BeautifulSoup's find_all
items = page.find_all("span")
print(f"Found {len(items)} spans")

Getting values out

# .text — the element's visible text content
title_el = page.css("h3.job-title")[0]
print(title_el.text)   # Senior Python Developer

# .html — the element's outer HTML
print(title_el.html)   # <h3 class="job-title">Senior Python Developer</h3>

# .attrib — dict-like attribute access (never raises KeyError)
card = page.css("div.job-card")[0]
print(card.attrib["data-id"])          # 101
print(card.attrib.get("data-missing"))  # None — safe access

# Link href
link = page.css("a.apply-btn")[0]
print(link.attrib["href"])   # /jobs/101

# All text in a container (concatenates all descendant text nodes)
card_text = card.get_all_text(separator=" ")
print(card_text[:60])   # Senior Python Developer Infosys Bangalore Rs. 18...

extract() and extract_first()

These return the raw HTML string of matched elements — useful when you need to re-parse a sub-section:

# extract_first() → the first match as an HTML string
first_title_html = page.css("h3.job-title").extract_first()
print(first_title_html)   # <h3 class="job-title">Senior Python Developer</h3>

# extract() → list of all matches as HTML strings
all_price_html = page.css("span.salary").extract()
print(all_price_html)
# ['<span class="salary">Rs. 18–25 LPA</span>',
#  '<span class="salary">Rs. 15–22 LPA</span>']

Nested selection

You can run CSS selectors on individual elements, not just the whole page:

jobs = []
for card in page.css("div.job-card"):
    job = {
        "id":       card.attrib.get("data-id"),
        "title":    card.css("h3.job-title")[0].text,
        "company":  card.css("span.company")[0].text,
        "location": card.css("span.location")[0].text,
        "salary":   card.css("span.salary")[0].text,
        "tags":     [t.text for t in card.css("span.tag")],
        "link":     card.css("a.apply-btn")[0].attrib["href"],
    }
    jobs.append(job)

for j in jobs:
    print(f"  [{j['id']}] {j['title']} @ {j['company']} — {j['salary']}")
    print(f"       {', '.join(j['tags'])}")

Traversal

# children — iterate direct child elements
ul_html = "<ul><li>Item A</li><li>Item B</li><li>Item C</li></ul>"
page2   = Adaptor(ul_html, url="https://example.com")
ul      = page2.css("ul")[0]

for child in ul.children:
    if hasattr(child, "text"):
        print(child.text)   # Item A, Item B, Item C

# find_ancestor — walk up the tree with a test function
html3   = "<table><tr><td class='name'>Alice</td></tr></table>"
page3   = Adaptor(html3, url="https://example.com")
td      = page3.css("td.name")[0]

tr = td.find_ancestor(lambda el: el.tag == "tr")
print(tr.tag)   # tr

Part 2: The Features That Make Scrapling Different

find_similar() — the adaptive core

This is the feature that makes Scrapling special. find_similar() finds all elements on the page that are structurally similar to a given element — same depth, same relationship to siblings and parent, same content type — even if their CSS class has changed.

html = """
<table>
  <tr><td class="name">Alice</td><td class="age">30</td></tr>
  <tr><td class="name">Bob</td>  <td class="age">25</td></tr>
  <tr><td class="name">Carol</td><td class="age">35</td></tr>
</table>
"""
page = Adaptor(html, url="https://example.com")

# Get only the FIRST name cell by selector
first_name = page.css("td.name")[0]
print("First:", first_name.text)   # Alice

# find_similar() finds the other name-like cells by structural position
# — NOT by class name. If "td.name" becomes "td.member-name" tomorrow,
# this still works.
similar = first_name.find_similar()
print("Similar:", [s.text for s in similar])   # ['Bob', 'Carol']

The practical workflow this enables: identify the first example of the element you want, then call find_similar() to get all others. If the site redesigns and changes class names, find_similar() still finds them by structural position.

find_by_text() — locate elements by content

html = """
<div class="card">
  <h2>Product Overview</h2>
  <p class="desc">A quality product designed for professionals.</p>
  <button class="cta-btn">Buy Now</button>
</div>
"""
page = Adaptor(html, url="https://example.com")

# Find an element by its exact text
btn = page.find_by_text("Buy Now")
print(btn.text)   # Buy Now
print(btn.tag)    # button

# Find by partial text
desc = page.find_by_text("quality product", partial=True)
print(desc.text)   # A quality product designed for professionals.

Auto-generated selectors — never write a selector manually again

Scrapling can generate a CSS selector or XPath expression for any element it finds. Use this to discover selectors rather than hunting through DevTools:

html = """
<nav class="main-nav">
  <ul>
    <li><a href="/" class="nav-link">Home</a></li>
    <li><a href="/pricing" class="nav-link">Pricing</a></li>
    <li><a href="/about" class="nav-link">About</a></li>
  </ul>
</nav>
"""
page = Adaptor(html, url="https://example.com")

pricing_link = page.find_by_text("Pricing")
print("CSS selector:", pricing_link.generate_css_selector)
# body > nav > ul > li:nth-child(2) > a

print("XPath:",        pricing_link.generate_xpath_selector)
# //body/nav/ul/li[2]/a

print("Full CSS:",     pricing_link.generate_full_css_selector)
# html > body > nav > ul > li:nth-child(2) > a

Use case: you're scraping a new site and aren't sure what selectors to use. Run find_by_text() on known content, then call generate_css_selector to get the precise selector to use in your production code.

Part 3: Fetching Pages with Scrapling

The Adaptor alone is a parser — you bring the HTML. Scrapling also ships three fetcher classes that handle the HTTP layer and return Adaptor objects directly.

Fetcher — fast HTTP, no browser

Fetcher uses httpx under the hood. Good for simple server-rendered pages with no anti-bot protection.

from scrapling.fetchers import Fetcher

fetcher = Fetcher(auto_match=True)   # auto_match enables adaptive matching

# fetch() returns an Adaptor object — parse it immediately
page = fetcher.get(
    "https://books.toscrape.com/",
    timeout=20,
    stealthy_headers=True,   # Add realistic browser headers automatically
)

# Use Adaptor methods on the result
titles = page.css("h3 a")
for t in titles[:5]:
    print(t.attrib.get("title", t.text))

StealthyFetcher — TLS impersonation

StealthyFetcher uses curl_cffi to impersonate real browser TLS fingerprints. Use this for sites that check JA3/TLS fingerprints (Cloudflare, most major e-commerce sites).

from scrapling.fetchers import StealthyFetcher

fetcher = StealthyFetcher(auto_match=True)

page = fetcher.get(
    "https://target-site.com/products",
    timeout=30,
    # Scrapling picks a Chrome impersonation target automatically
)

# Result is still an Adaptor — same API
prices = page.css(".price")
print([p.text for p in prices])

PlayWrightFetcher — full browser rendering

For JavaScript-heavy sites:

from scrapling.fetchers import PlayWrightFetcher

fetcher = PlayWrightFetcher(auto_match=True)

page = fetcher.get(
    "https://spa-site.com/dashboard",
    headless=True,
    network_idle=True,       # Wait for all XHR to complete
    timeout=30,
    disable_resources=True,  # Block images/fonts for speed
)

# Same Adaptor API
data = page.css(".data-table tr")

Part 4: The auto_match System

auto_match=True is Scrapling's most powerful feature — and the one that requires a bit of explanation.

When you use auto_match, Scrapling stores a fingerprint of each element you extract — not just its CSS class, but its position in the DOM tree, its relationship to sibling elements, its text content pattern, and other structural signals. On subsequent runs, if the CSS class has changed, Scrapling uses this fingerprint to locate the element anyway.

from scrapling.fetchers import Fetcher

fetcher = Fetcher(auto_match=True)

# First run: Scrapling fetches the page and learns element fingerprints
page = fetcher.get("https://example.com/products")

# You select by CSS class — Scrapling memorises the element
price = page.css_first(".product-price")
print(price.text)   # Rs. 1,299

# Three weeks later: site redesigns, ".product-price" → ".price-tag"
# Scrapling's auto_match finds the element by structural fingerprint
# Even though the CSS class changed, it still returns the right element
page2 = fetcher.get("https://example.com/products")
price2 = page2.css_first(".product-price")   # Old selector still works!
print(price2.text)   # Rs. 1,299 — correct even after the redesign

This is the real promise of Scrapling: selectors that survive redesigns.

Part 5: A Complete Production Scraper with Scrapling

Let's build a complete working scraper that uses Scrapling's full feature set:

# scrapling_full_example.py
import asyncio
import pandas as pd
from scrapling.parser import Adaptor

# We use Adaptor with pre-fetched HTML for this example
# In production, replace with StealthyFetcher.get() or PlayWrightFetcher.get()

SAMPLE_PAGE_HTML = """<!DOCTYPE html>
<html>
<head><title>Tech Products — Store</title></head>
<body>
  <h1 class="page-heading">Latest Products</h1>
  <div class="product-grid">

    <article class="product-card" data-sku="WGT-001">
      <h2 class="product-name">Wireless Keyboard Pro</h2>
      <div class="pricing">
        <span class="current-price">Rs. 3,499</span>
        <span class="original-price">Rs. 4,999</span>
        <span class="discount-badge">30% off</span>
      </div>
      <ul class="feature-list">
        <li>Bluetooth 5.2</li>
        <li>Multi-device pairing</li>
        <li>12-month battery</li>
      </ul>
      <div class="meta">
        <span class="brand">Logitech</span>
        <span class="rating">4.6 ★</span>
        <span class="reviews">2,841 reviews</span>
      </div>
      <a href="/products/WGT-001" class="product-link">View Product</a>
    </article>

    <article class="product-card" data-sku="MNT-002">
      <h2 class="product-name">4K USB-C Monitor</h2>
      <div class="pricing">
        <span class="current-price">Rs. 28,999</span>
        <span class="original-price">Rs. 34,999</span>
        <span class="discount-badge">17% off</span>
      </div>
      <ul class="feature-list">
        <li>4K 27-inch IPS</li>
        <li>USB-C 90W PD</li>
        <li>HDMI 2.1</li>
      </ul>
      <div class="meta">
        <span class="brand">LG</span>
        <span class="rating">4.8 ★</span>
        <span class="reviews">1,204 reviews</span>
      </div>
      <a href="/products/MNT-002" class="product-link">View Product</a>
    </article>

    <article class="product-card" data-sku="HDS-003">
      <h2 class="product-name">ANC Headphones Elite</h2>
      <div class="pricing">
        <span class="current-price">Rs. 12,499</span>
        <span class="original-price">Rs. 15,999</span>
        <span class="discount-badge">22% off</span>
      </div>
      <ul class="feature-list">
        <li>Active Noise Cancellation</li>
        <li>40hr battery</li>
        <li>Hi-Res Audio</li>
      </ul>
      <div class="meta">
        <span class="brand">Sony</span>
        <span class="rating">4.7 ★</span>
        <span class="reviews">5,632 reviews</span>
      </div>
      <a href="/products/HDS-003" class="product-link">View Product</a>
    </article>

  </div>
</body>
</html>"""

def scrape_products(html: str, source_url: str) -> list[dict]:
    """
    Parse product cards from a page using Scrapling.
    Demonstrates: nested css(), .text, .attrib, find_similar(), find_by_text()
    """
    page     = Adaptor(html, url=source_url)
    products = []

    for card in page.css("article.product-card"):
        sku        = card.attrib.get("data-sku", "")
        name       = card.css("h2.product-name")[0].text

        # Pricing block
        price_curr = card.css("span.current-price")[0].text
        price_orig = card.css("span.original-price")[0].text
        discount   = card.css("span.discount-badge")[0].text

        # Features — list items
        features = [li.text for li in card.css("ul.feature-list li")]

        # Meta
        brand    = card.css("span.brand")[0].text
        rating   = card.css("span.rating")[0].text
        reviews  = card.css("span.reviews")[0].text

        # Link
        link     = card.css("a.product-link")[0].attrib["href"]

        # Clean price to float
        import re
        price_float = float(re.sub(r"[^\d]", "", price_curr))

        products.append({
            "sku":           sku,
            "name":          name,
            "price":         price_float,
            "price_display": price_curr,
            "original":      price_orig,
            "discount":      discount,
            "features":      ", ".join(features),
            "brand":         brand,
            "rating":        rating,
            "reviews":       reviews,
            "url":           link,
        })

    return products

def demonstrate_adaptive_features(html: str):
    """
    Show Scrapling's key adaptive features:
    find_similar(), find_by_text(), auto-generated selectors
    """
    page = Adaptor(html, url="https://store.example.com")

    print("\n── find_similar() ──")
    # Get the first product name
    first_name = page.css("h2.product-name")[0]
    print(f"First product: {first_name.text}")

    # find_similar finds the other product names by structural position
    # This works even after CSS class renames
    similar = first_name.find_similar()
    print(f"Similar elements found: {[s.text for s in similar]}")

    print("\n── find_by_text() ──")
    # Locate element by its text content — class-independent
    sony_brand = page.find_by_text("Sony")
    if sony_brand:
        print(f"Found 'Sony' in: <{sony_brand.tag}> element")
        # Walk up to get the parent product card
        parent_card = sony_brand.find_ancestor(lambda el: el.tag == "article")
        if parent_card:
            product_name = parent_card.css("h2")[0].text
            print(f"  → Parent product: {product_name}")

    print("\n── find_by_text() partial match ──")
    anc_el = page.find_by_text("Noise Cancellation", partial=True)
    if anc_el:
        print(f"Partial match: '{anc_el.text}'")

    print("\n── Auto-generated selectors ──")
    # Let Scrapling generate the selector for any element
    first_discount = page.css("span.discount-badge")[0]
    print(f"CSS selector: {first_discount.generate_css_selector}")
    print(f"XPath:        {first_discount.generate_xpath_selector}")

    print("\n── get_all_text() ──")
    first_card = page.css("article.product-card")[0]
    all_text   = first_card.get_all_text(separator=" | ")
    print(f"All text in first card: {all_text[:120]}...")

def main():
    print("=" * 55)
    print("  SCRAPLING DEMO")
    print("=" * 55)

    # Scrape products
    products = scrape_products(SAMPLE_PAGE_HTML, "https://store.example.com")

    print(f"\nScraped {len(products)} products:\n")
    df = pd.DataFrame(products)
    print(df[["sku", "name", "price", "discount", "brand", "rating"]].to_string(index=False))

    # Analysis
    print(f"\nAverage price: Rs. {df['price'].mean():,.0f}")
    print(f"Most expensive: {df.loc[df['price'].idxmax(), 'name']} "
          f"(Rs. {df['price'].max():,.0f})")
    print(f"Cheapest: {df.loc[df['price'].idxmin(), 'name']} "
          f"(Rs. {df['price'].min():,.0f})")

    # Demonstrate adaptive features
    demonstrate_adaptive_features(SAMPLE_PAGE_HTML)

    # Save
    df.to_csv("scrapling_products.csv", index=False)
    print("\n✓ Data saved to scrapling_products.csv")

if __name__ == "__main__":
    main()

Output:

=======================================================
  SCRAPLING DEMO
=======================================================

Scraped 3 products:

     sku                    name     price discount    brand  rating
 WGT-001    Wireless Keyboard Pro    3499.0   30% off Logitech  4.6 ★
 MNT-002          4K USB-C Monitor  28999.0   17% off       LG  4.8 ★
 HDS-003     ANC Headphones Elite   12499.0   22% off     Sony  4.7 ★

Average price: Rs. 14,999
Most expensive: 4K USB-C Monitor (Rs. 28,999)
Cheapest: Wireless Keyboard Pro (Rs. 3,499)

── find_similar() ──
First product: Wireless Keyboard Pro
Similar elements found: ['4K USB-C Monitor', 'ANC Headphones Elite']

── find_by_text() ──
Found 'Sony' in: <span> element
  → Parent product: ANC Headphones Elite

── find_by_text() partial match ──
Partial match: 'Active Noise Cancellation'

── Auto-generated selectors ──
CSS selector: body > div > article > div > span:nth-child(3)
XPath:        //body/div/article/div/span[3]

── get_all_text() ──
All text in first card: Wireless Keyboard Pro | Rs. 3,499 | Rs. 4,999 | 30% off...

✓ Data saved to scrapling_products.csv

Part 6: Scrapling vs BeautifulSoup — Honest Comparison

Feature	BeautifulSoup	Scrapling
Syntax simplicity	⭐⭐⭐⭐⭐	⭐⭐⭐⭐
CSS selectors	✅	✅
XPath	❌ (needs lxml directly)	✅
find_by_text	Basic	Advanced with partial match
find_similar	❌	✅ (unique to Scrapling)
Auto-selectors	❌	✅
Auto-match on redesign	❌	✅
Built-in fetchers	❌	✅ (3 fetcher types)
TLS impersonation	❌	✅ (via StealthyFetcher)
Speed	Fast	Slightly slower
Documentation	Excellent	Good, growing
Community	Enormous	Rapidly growing

When to use BeautifulSoup: You're scraping a stable site, you're a beginner, or you're teaching someone else.

When to use Scrapling: You're scraping sites that change frequently, you want built-in fetchers with anti-detection, or you're tired of maintaining selector lists.

Part 7: Working with Scrapling's Fetchers in Production

# production_fetcher_example.py
from scrapling.fetchers import StealthyFetcher
import time
import random

def scrape_with_retry(url: str, max_retries: int = 3) -> object | None:
    """
    Production-safe wrapper around StealthyFetcher.
    Handles retries and errors gracefully.
    """
    fetcher = StealthyFetcher(auto_match=True)

    for attempt in range(max_retries):
        try:
            page = fetcher.get(
                url,
                timeout        = 25,
                stealthy_headers=True,    # Scrapling adds realistic headers
                hide_browser   = True,    # Additional stealth for PlayWright
            )
            return page

        except Exception as e:
            wait = (2 ** attempt) + random.uniform(0, 1)
            print(f"  Attempt {attempt+1} failed: {e}. Retrying in {wait:.1f}s")
            time.sleep(wait)

    print(f"  Permanently failed: {url}")
    return None

# Example multi-page scrape
def scrape_catalog(base_url: str, pages: int = 5) -> list[dict]:
    all_products = []

    for page_num in range(1, pages + 1):
        url  = f"{base_url}?page={page_num}"
        page = scrape_with_retry(url)

        if not page:
            continue

        # Adaptive scraping — if class names change between pages,
        # find_similar() on first result finds all counterparts
        first_price = page.css(".price, .product-price, .current-price")
        if not first_price:
            print(f"  No prices found on page {page_num}")
            continue

        # find_similar adapts to whatever the price class is on this page
        all_prices = first_price[0].find_similar()
        all_names  = page.css("h2, h3").extract()

        print(f"  Page {page_num}: {len(all_prices)} prices, {len(all_names)} names")
        time.sleep(random.uniform(1.5, 3.0))

    return all_products

Common Pitfalls and Fixes

css_first() doesn't exist — use css()[0]

# Wrong — AttributeError
item = page.css_first(".price")

# Right
item = page.css(".price")[0] if page.css(".price") else None

extract() returns HTML strings, not text

# extract() returns '<span class="price">Rs. 999</span>'
raw = page.css(".price").extract()

# For clean text, iterate and use .text
texts = [el.text for el in page.css(".price")]

find_ancestor() takes a callable, not a string

# Wrong
parent = el.find_ancestor("div")

# Right
parent = el.find_ancestor(lambda e: e.tag == "div")
parent = el.find_ancestor(lambda e: "card" in (e.attrib.get("class") or ""))

Summary

Concept	Method	Example
Parse HTML	Adaptor(html, url=url)	Build the page object
CSS select all	page.css("selector")	Returns Selectors list
CSS select one	page.css("sel")[0]	First match
Element text	el.text	Clean inner text
Attribute	el.attrib["href"]	Safe dict access
All text	el.get_all_text(sep)	Concatenated descendants
Tag name	el.tag	"div", "span", etc.
Outer HTML	el.html	<div>...</div>
XPath	page.xpath("//h2")	Same return type as css()
Find by text	page.find_by_text("text")	Class-independent search
Partial match	find_by_text("x", partial=True)	Substring search
Find similar	el.find_similar()	Adaptive structural match
Walk up tree	el.find_ancestor(lambda e: ...)	Callable test
Auto CSS	el.generate_css_selector	Discover selectors
Auto XPath	el.generate_xpath_selector	Discover selectors
Children	el.children	Direct child iterator
Raw HTML list	page.css("x").extract()	HTML strings
First raw HTML	page.css("x").extract_first()	First HTML string

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

DEV Community: ZyVOP

What Is Agentic AI? The Ultimate 2026 Guide to Autonomous AI Agents

How Does Agentic AI Work?

Actions need idempotency

Agent loops need limits

A Refund Agent in Practice

Agentic AI vs. Generative AI

Agentic AI vs. Traditional Automation

How Reliable Are AI Agents?

Multi-Agent Systems

Where AI Agents Are Useful

Software development

Customer support

Internal operations

Recruiting

Agents Are Changing How Software Uses the Web

Where Agent Projects Fail

Prompt Injection Becomes a Security Problem

Before Building an Agent

Frequently Asked Questions

Is agentic AI the same as AGI?

Is every AI assistant an agent?

Can AI agents operate without humans?

How much does an AI agent cost?

The Best Agent May Not Look Like an Agent

Sources & Further Reading

Build a Web Scraping API with FastAPI, Celery & Redis (2026)

Why Your Scraper Needs an API Layer

Project Structure

Step 1: Install Dependencies

Step 2: Database Models

Step 3: Pydantic Schemas

Step 4: The Scraping Logic

Step 5: Celery Tasks

Step 6: The FastAPI Application

Step 7: Database Setup

Step 8: Docker Compose

Step 9: Using the API

Step 10: Monitoring with Flower

Production Checklist

FAQ

How Chinese Models Actually Took Over OpenRouter, Month by Month

November 2024 — The baseline

January 2025 — The first DeepSeek moment

March 2025 — Past 10%

Mid-to-late 2025 — Past 25%, and a name for the pattern

November 17, 2025 — The date worth remembering

December 2025 — The last calm measurement

February 8, 2026 — The number CNBC would later cite

March 11–18, 2026 — Xiaomi runs the playbook first

April 2026 — Two things happen at once, and they're connected

May 2026 — The security study, and who paid for it

June 2026 — The month everything compounds

July 1, 2026 — Restoration, and a manifesto

July 7, 2026 — The number that's been repeated everywhere

July 7–8, 2026 — The twist

Where This Leaves Us

Python Scraping Patterns: 15 Real Problems, 15 Working Solutions

Pattern 1: The Element You Need Doesn't Exist in BeautifulSoup

Pattern 2: The Data Is in a <script> Tag as JSON

Pattern 3: Infinite Scroll Pages

Pattern 4: Login Walls — Scraping Content That Requires Authentication

Pattern 5: Rate Limited — Getting 429 Too Many Requests

Pattern 6: Paginated Results with No "Next" Link

Pattern 7: Downloading Files (PDFs, CSVs, Images) at Scale

Pattern 8: Scraping Tables with Merged Cells

Pattern 9: Rotating User Agents Without Getting Detected

Pattern 10: Extracting Data Across Multiple Formats on the Same Page

Pattern 11: Handling Cloudflare's Bot Protection Without a Proxy

Pattern 12: Storing Scraped Data Without Duplicates

Pattern 13: Scraping Sites That Block Based on Geolocation

Pattern 14: Detecting When a Scraper Has Gone Silent

Pattern 15: Scraping GraphQL APIs

Quick Reference

Developer‑Centric AI Hits a Tipping Point: Performance, Open Models, and Scaling Pain | The AI Daily Roundup

One unifying trend: AI for software development is moving from novelty to core infrastructure

Why this matters for engineers, CTOs, and investors

Evidence from today’s headlines

Claude Code adopts a Rust‑based Bun runtime

OpenAI trims Codex context size

Pattern 2: The Data Is in a `<script>` Tag as JSON

Ollama’s $88 M fundraising for open‑model tooling

1. `Promise.allSettled()` Over `Promise.all()`