DEV Community: Ayan Pahwa

My agentic coding setup: Claude Code, multi-agent orchestration, and more

Ayan Pahwa — Mon, 01 Jun 2026 10:34:45 +0000

Every new agentic coding tool arrives with a version of the same implicit promise: this one will change how you build. I spent a good part of last year installing tools on that promise, configuring them, hitting their limits, and then either reaching for the next release or quietly uninstalling and going back to basics. The result, for a while, was a collection of half-configured assistants that each needed babysitting before they could help with anything.
What I have now is the setup that survived that process, not because the tools are exceptional in isolation, but because I made deliberate choices about what each one is actually for, what it is not for, and how they hand off to each other. I work on main projects at Zyte, on side projects, and on web scraping work, and this setup handles all three without requiring reconfiguration between them. This is that setup: the tools I kept, the habits that hold it together, and the reasoning behind each decision.

My unfair advantage

Before diving into the setup itself, I want to say something about what actually makes an agentic setup work, because it is not the tools.
I have been writing code for more than a decade, starting with Embedded C and C++ before gradually moving to higher-level languages and more recently into Python and web scraping. That background means I can usually tell when an agent is on the right track, when it is confidently producing something plausible but wrong, and when it is about to do something I will spend the next hour undoing. I do not need to read every line it writes to know whether the approach is sound. That accumulated judgment is the unfair advantage: years of building a mental model of how code actually behaves, which now applies directly to supervising what an agent produces.
But this is my advantage, not a universal prescription. Yours is different, and your setup should reflect that. If you have spent years in SEO, your unfair advantage is knowing precisely what good output looks like, what a manipulable signal looks like, and what an agent is getting subtly wrong before the metrics catch it. There are already excellent SEO-specific Claude skills available, and building a team of sub-agents around them (one for technical audits, one for content, one for structured data) with your domain knowledge as the quality filter is a genuinely powerful setup. If your background is in data engineering, you know what a clean pipeline looks like and what a silently broken one looks like, which is exactly the kind of judgment an agent cannot supply for itself. If you come from finance, security, or product management, the same principle holds.
The point is not that deep coding experience is required. Agentic tools amplify whatever domain judgment you already have. Think about where that knowledge lives in your case, and build your setup around it rather than copying someone else's wholesale. Everything in this post is what works for me and my context. Take what fits and ignore the rest.

The workspace that opens itself

The first friction point I fixed was the startup ritual. Every morning I was opening VS Code, arranging panels, launching a terminal, opening Claude Code, and getting everything positioned before I could do anything useful. Five minutes of overhead that was really ten minutes once you account for the mental cost of doing it on autopilot.
I now have a cw function in my ~/.zshrc:

cw() {
  local dir="${1:-.}"
  code "$dir"
  sleep 3 && osascript -e '
    tell application "Visual Studio Code" to activate
    delay 0.5
    tell application "System Events"
      key code 53 using {command down, shift down}
      delay 0.3
      key code 50 using {control down}
    end tell
  ' &
}

Type cw in any directory and VS Code opens in that directory, then AppleScript fires after three seconds to focus the window and drop you straight into the terminal panel where Claude Code is waiting. One command, full workspace ready.
The layout is fixed: file explorer on the left, terminal strip at the bottom, the agent chat panel in the middle, and outputs on the right. The chat pane is centered, not tucked in the sidebar, and that placement is deliberate. When the agent is your primary collaborator, putting it in the sidebar demotes it spatially. The center position is a constant reminder that orchestration is the primary activity and everything else supports it. For inline code review without breaking context, the Codex plugin for Claude runs directly in the editor.

Plan first, always

The habit that improved my output more than any tool was deciding that every task, regardless of scale, starts in plan mode before any file is touched. No exceptions.
Plan mode forces the agent to surface its assumptions, propose a concrete approach, and wait for sign-off before it touches anything. The default behavior of most agentic tools is to start executing immediately, and high-confidence execution in the wrong direction is the failure mode I have run into most. The agent is rarely wrong because the code is bad; it is wrong because it interpreted the brief differently than I intended, and three minutes of planning would have caught the gap.
I did not arrive at this habit entirely on my own. Dario Amodei, CEO of Anthropic, mentioned in a podcast that he spends the majority of his time in plan mode when working with Claude, and that once the plan is solid the actual execution becomes relatively straightforward. That framing stuck with me. If the person building the model treats planning as the primary activity, it is probably worth taking seriously.
For tasks where the problem is still fuzzy in my own head, I dictate into plan mode using WisprFlow. This is not just a comfort choice. I have noticed consistently that my spoken prompts produce better results than my typed ones: speaking forces me to construct a full sentence rather than tapping out a telegraphic shorthand, and that extra formality in the brief translates directly into more precise agent output. Describing the problem, the likely approach, and any constraints out loud usually clarifies the brief before the agent has responded, which means the plan mode exchange is a confirmation rather than a negotiation.
Something I have been testing recently on the planning side: Claude Code's /goal command. The idea is straightforward: before anything else in a session, you set a high-level goal that the agent holds as a persistent north star throughout all its subsequent actions. Where plan mode answers "how do we approach this specific task", /goal answers "what is this entire session ultimately in service of." I came across the same concept in Codex and liked the forcing function it created: it keeps a long session from gradually drifting away from what you actually opened it to achieve. I am still finding the edges of how to use it well, but the principle is sound: the more clearly you can state what done looks like before the first message, the less corrective steering you need to do mid-session. If you try it, be specific: "refactor the auth module to remove the session token storage" will serve you better than "clean up the auth code."

Two tools, different jobs

One thing worth saying before getting into the specifics: the agentic coding space is moving faster than almost any other area of software right now. Every major provider is shipping changes to tooling, pricing, context limits, and model capabilities on a cadence that would have seemed unrealistic two years ago. That pace is exciting, but it also means that going completely all-in on a single vendor is a real risk. If one provider changes pricing, deprecates a model, or ships a breaking change to their CLI, a setup that depends entirely on them stops working. The practical response is to not let that happen: maintain flexibility, keep alternatives warm, and make sure switching costs stay low.

My primary tool is Claude Code CLI with official Anthropic models, and it is where the majority of my serious work happens. I run it on the Claude API pay-per-usage plan, which means I pay for what I use and nothing beyond that. No monthly seat fee accumulating on days when I am not writing code, and no "I am already paying for it" pressure to keep the agent running past the point of diminishing returns. I also keep Codex in the mix via the Mac desktop app, not as a replacement, but as a parallel tool I use enough to stay current with how it is developing.
For model experimentation and usage overflow, I use OpenCode with OpenRouter via bring-your-own-key (BYOK). This is the experimentation layer and, frankly, the hedge against vendor lock-in. When a new model appears and I want to try it against a real task before committing to it in my main workflow, I reach for OpenCode. My ~/.aliases file has eight model shortcuts that make switching a single word in the terminal:

alias oc='opencode --model openrouter/anthropic/claude-sonnet-4-6'
alias oc-ds='opencode --model openrouter/deepseek/deepseek-chat-v3-1'
alias oc-free='opencode --model openrouter/deepseek/deepseek-r1:free'
alias oc-qwen='opencode --model openrouter/qwen/qwen3-coder'
alias oc-gemini='opencode --model openrouter/google/gemini-2.5-pro'
alias oc-opus='opencode --model openrouter/anthropic/claude-opus-4-7'
alias oc-cost='opencode --usage'
alias oc-stats='opencode --stats'

oc-cost and oc-stats are not cosmetic shortcuts. The pay-per-usage model only works as a cost discipline if you can see what you are spending.
OpenRouter also has an auto-router option (openrouter/auto) that selects the best available model for each prompt automatically, which is genuinely useful when you are unsure which model fits a task and do not want to think about it. My opencode.json config defines three routing entries that cover the main scenarios:

"openrouter/auto": {
  "name": "Auto Router (picks best model for prompt)",
  "tool_call": true,
  "limit": { "context": 200000, "output": 8192 }
},
"openrouter/free": {
  "name": "Free Router (picks best free model)",
  "tool_call": true,
  "limit": { "context": 128000, "output": 4096 }
},
"openrouter/pareto-code": {
  "name": "Pareto Code Router (auto-routes coding tasks)",
  "tool_call": true,
  "limit": { "context": 200000, "output": 8192 }
}

auto is the general-purpose router. free picks the best available free model, which is what I reach for when testing something throwaway. pareto-code is a coding-specific router that OpenRouter maintains separately, optimized for code tasks rather than general prompts.
Beyond Claude overflow, I use OpenRouter's free tier for testing with tools like OpenClaw and the Hermes agent, where spending money on a model I am just kicking the tires on makes no sense. This side of the setup is almost entirely for side projects and learning how different models actually behave on real tasks rather than benchmarks. My personal ranking after a fair amount of experimentation, in order: Qwen3 Coder, DeepSeek, MiniMax, Kimi, and Gemma. Qwen and DeepSeek are consistently good on code; MiniMax and Kimi are worth watching for longer-context tasks; Gemma punches above its weight for its size.
A note for developers still on Claude Pro ($20/month): the plan uses rolling usage windows that reset approximately every five hours. If you plan to code from noon, send Claude a short message at 7 AM. Your window resets right as you sit down, and the next reset lands around 5 PM, which means you can run two back-to-back full sessions from noon through the evening without hitting a cap mid-task. The exact window length has shifted with recent Anthropic updates, so check your own account to calibrate, but the principle holds: prime your session before you need it, not after you have already hit the limit. Since switching to pay-per-usage I no longer need this trick, but it was genuinely useful for the two years I ran on the flat-rate plan.

The four-agent team

Running a general-purpose agent at every task is a bit like asking one person to handle architecture, implementation, code review, and codebase archaeology simultaneously, and expecting them to be equally good at all four. I took inspiration from Gary Tan, CEO of Y Combinator, who described his own layered agent stack (which he calls GStack) as a way of giving each model a specific role rather than asking one model to do everything. I am not using GStack directly, but the framing shaped how I think about agent design: specialize the agents, not the prompts.

In OpenCode, I have defined four named agents, each with a specific model, a specific role, and a specific permission scope.
@architect runs on Claude Sonnet 4.6 and is read-only: no edit or bash access. It asks clarifying questions first, then produces ASCII diagrams and a numbered implementation plan. When it needs to understand the codebase before it can plan, it invokes @scout. The output is written to be clear enough for a junior developer, or for a cheaper model like DeepSeek, to execute without interpretation.
@scout runs on Gemini 2.5 Flash with a one-million token context window and is also read-only. It traces call chains, maps data flow, and produces structured reports with full file paths and line numbers. The large context window makes it the right choice for reading substantial portions of an unfamiliar codebase without losing thread.
@coder runs on DeepSeek V3.1, configured at 40 steps and temperature 0.1. It follows the architect's plan exactly and does not extend scope. Before marking any task complete, it invokes @reviewer.
@reviewer runs on Qwen3 Coder, is read-only, and works through a fixed priority order: security vulnerabilities first, then logic errors, missing error handling, performance bottlenecks, and finally code clarity. It cites exact file paths and line numbers for everything it flags.
The permission constraints are what most people skip, and they are what matter most. A read-only agent cannot accidentally delete files or run shell commands, which limits the blast radius when an agent misreads an instruction. In Claude Code, I apply the same model-tiering logic via the native multi-agent orchestrator: Claude Opus 4.7 for planning, Claude Sonnet 4.6 for execution, and Claude Haiku 4.5 for admin tasks like git summaries and log triage. For the heaviest parallel projects, I use Conductor, which runs multiple Claude Code and Codex instances across separate areas of a codebase without context bleed between them.
A word on scale: I am not trying to run 20 agents across 10 projects simultaneously, and I am not optimizing for that. At any given time I work across two or three projects at most, because beyond that I notice the creative block creeping in and the context-switching cost becoming real. Four concurrent agents is my personal ceiling before things start feeling chaotic rather than productive. Running more agents than you can coherently supervise is not a productivity gain; it is just noise with extra steps. The right number is the one where you still know what each agent is doing and why.

Teaching agents to remember: the CLAUDE.md file

Here is the thing nobody tells you when you start using agentic tools seriously: the agent forgets everything between sessions. Every conversation starts from zero. Your project conventions, the architectural decisions you made three weeks ago, the one gotcha in the authentication middleware that will silently break if you touch it: gone. The agent does not know any of it unless you tell it again.
The fix is a CLAUDE.md file in the root of every project. Claude Code reads this file automatically at the start of each session, which means it walks into the codebase already briefed: how the project is structured, what conventions to follow, what not to touch, and why certain decisions were made. It is the difference between starting a session with a junior developer who has never seen your codebase and starting a session with one who was briefed before they arrived. I include things like folder structure, key design decisions, known gotchas, and any non-obvious constraints. What is not in that file will surface at the worst possible time, usually when the agent is three files deep into something it should not have started. I have learned this lesson more than once.
There are actually two levels of CLAUDE.md worth maintaining separately. The per-project file, described above, carries context specific to that codebase. But there is also a global CLAUDE.md at ~/CLAUDE.md that Claude Code reads across every session, regardless of project. This is where the universal stuff lives: how you like responses formatted, code style preferences that never change, recurring patterns you reach for project after project. The smartest way I have found to populate it: ask Claude directly. Open a session after you have done a few projects and ask it what it has noticed you doing repeatedly across different codebases — preferences, habits, corrections you keep making. The answer is usually more accurate than anything you would write from memory, and it goes straight into the global file. You only have to do that calibration once, and every future session inherits it.
The same logic of "define it once, reuse it forever" applies to custom slash commands. Claude Code lets you create your own /commands by dropping a markdown file into .claude/commands/ inside a project, or into ~/.claude/commands/ for commands that travel with you globally. Anything you find yourself prompting repeatedly is a candidate: I have a /pr command that opens a pull request with a consistent format, a /review command that runs a code review against a checklist I care about, and a /standup command that summarizes what changed since the last commit in plain language. The rule of thumb is the same as for skills: if you have typed the same instruction more than twice, it should be a command. The overhead is a single markdown file; the return is that you never type it again.
In practice, this shapes how I move between tasks. One session handles one specific thing, 95% of the time. When that task is done and I want to start something different, I update the CLAUDE.md first, capturing any decisions made, gotchas discovered, or context the next session will need, and then launch fresh. This is not a workaround; it is the actual workflow. Each session stays focused on exactly one thing, the context window never accumulates unrelated baggage, and the token cost stays predictable because the session ends when the task ends.
One thing I want to push back on slightly: the idea that more persistent memory is always better. The "second brain" framing — building an ever-growing knowledge base that carries everything forward — is appealing in theory, but I have found clean starts genuinely valuable. A fresh session with a tight, well-written CLAUDE.md is often sharper and more focused than a long session carrying the accumulated noise of everything that came before. Starting fresh is not a disadvantage; sometimes it is the whole point. This is especially true when starting a brand new project: no CLAUDE.md, no prior context, no assumptions inherited from a different codebase. The agent approaches it with the same clean slate you do, which means nothing from the last project bleeds into this one. That is not a limitation of the tool; it is the right default. The CLAUDE.md approach hits the balance I actually want: enough persistent context to orient the agent quickly, without the clutter.

When the context window fills up

Agentic coding sessions on complex tasks will, eventually, produce a context window that is full or close to it. The agent's effective memory degrades as the window saturates, and you start getting responses that feel slightly off, repetitive, or strangely overconfident about something it got wrong two thousand tokens ago.
Claude Code handles this with automatic context compaction, which summarizes earlier parts of the conversation to make room. For sessions where I want manual control, I use the /compact command to trigger a summary on demand. When even that is not enough, the bluntest tool available is the right one: start a fresh session, point at the CLAUDE.md file, and re-brief the agent on exactly where the previous session left off. It feels a bit caveman, but a focused, fresh session outperforms a saturated long one almost every time. My experience is that a 10,000-token focused session produces better output than a 100,000-token sprawling one that has lost the thread.

Web scraping: where the Zyte layer comes in

Web data is not optional for serious agentic workflows. Agents that can research, verify facts, monitor changes, track competitors, or enrich datasets with live information are dramatically more useful than agents working from static knowledge alone. The web is the data source.
The problem is that the web does not cooperate equally. Some pages render entirely in JavaScript and return nothing useful to a basic HTTP request. Others sit behind rate limits, bot detection, or login walls. Some block entire cloud IP ranges outright. An agent that tries to fetch a page and gets a 403, a CAPTCHA, or a JavaScript shell with no content is effectively blind, and it will usually not tell you that clearly; it will just work with whatever it got. For the straightforward cases, Claude's built-in browsing is fine. For anything beyond that, you need a layer that actually understands how modern websites are built and how to get through them reliably.
That is where Zyte's tooling earns its place. For web scraping work, the setup picks up a layer specific to Zyte's tooling. Zyte publishes an official set of Claude Code skills at github.com/zyte-ai/claude-skills, installable in two commands:

claude plugin marketplace add zyte-ai/claude-skills
claude plugin install zyte-web-data@zyte-ai

Once installed, the skills slot into Claude Code as slash commands and activate automatically on relevant prompts. The ones I reach for most are:

/scrape: end-to-end workflow from a URL to a working Scrapy spider with web-poet page objects; this is the one you use when you just want to hand Claude a URL and a description of what to extract
/scrape-define: downloads a single detail page, discovers extractable fields, and iterates on the schema in the terminal until you approve it; good for quickly scoping what a site can give you
/scrape-explore-site: crawls from a start URL and saves a diverse set of pages (start, list, and detail) with classified links; useful before committing to a schema
/scrape-codegen: takes an extraction spec and generates the web-poet page object code; the output of /scrape-define feeds directly into this
/scrape-scrapy-cloud: deploys projects, schedules spiders, manages jobs, and surfaces logs and items from Scrapy Cloud, all from the terminal The skills integrate with the Web Scraping Copilot and are designed to pick up scraping prompts automatically, so you do not need to invoke a specific command for routine requests. If you are curious how these fit into a broader workflow, the post on supercharging web scraping with Claude skills covers the combination in detail. Everything is git-tracked, including personal side projects. The ghs alias in my shell switches git identity instantly between my Zyte work email and my personal email, which eliminates the risk of pushing to the wrong remote after a context switch between work and a side project. On MCP servers versus CLI tools: my standing rule is to reach for a CLI tool first and add an MCP server only when there is genuinely no CLI equivalent. MCP servers add indirection between the agent and the tool, and that indirection is not free: it makes the toolchain harder to audit, harder to debug, and slightly more likely to produce ambiguous outputs. If you are weighing your options, the comparison of Claude skills, MCP, and Web Scraping Copilot is worth reading before committing. One area where I have been rethinking the default recently is web search. Most agents fall back to keyword-based search, which is fine for locating a documentation page but falls apart when an agent needs to do actual research. I came across Exa at a local developer meetup, and it is built specifically for AI agents using semantic search rather than keyword matching, which produces noticeably better results when the agent needs to find conceptually related content rather than an exact phrase. The catch, and the reason I have not fully switched over, is that Exa currently only offers an MCP server and not a CLI utility. That puts it in direct conflict with the CLI-first rule: every time the agent invokes an MCP server there is a context switch, a round-trip, and a small but real cost in time and tokens that adds up over a long session. So for now I enable Exa selectively on projects where deep research is a core part of the work, and fall back to Claude's built-in search everywhere else. I am still exploring it, and if a CLI lands I will probably use it much more broadly. The last piece of the scraping layer is what I think of as the objective metric loop. Before running the agent on a scraping task, I define a concrete, measurable target: field fill rate above 95%, zero extraction errors across 100 test URLs, or a specific field-level accuracy requirement. The agent runs, the output is evaluated automatically against that metric, and I re-prompt with the delta. The loop continues until the metric is hit, not until the code looks right on inspection. "Looks right" is not a metric. ## A few principles, for what they are worth These are not best practices from a blog post. They are things I arrived at through repetition, usually after doing the opposite first. Stop obsessing over prompts. Models are meaningfully smarter than they were twelve months ago, and they will be smarter again in twelve more. A clear, complete description of what you want is almost always sufficient today. Intricate prompt engineering made more sense when models were brittle; spending that energy on your workflow instead will compound better. Anything done twice should become a skill. If you have guided an agent through the same process more than once, it belongs in a skill file. A skill is a reusable, well-described prompt with clear inputs and outputs. The overhead of writing one is low; the compounding return is not. Each skill should do exactly one thing. A skill that researches a topic, writes a script, and suggests titles is three skills waiting to be separated. Single-purpose skills are easier to debug, easier to improve, and much easier to reason about when something breaks. Skills can be chained into workflows. Three separate skills (research the next video topic, write a script from the research, suggest titles from the script) can be combined in sequence to produce a full workflow while remaining individually useful and testable. The composition is more flexible than a monolith, and any one skill can be swapped out without rebuilding everything. Bundle custom scripts with the skills that need them. If a skill depends on a helper script (a parser, a formatter, a validator), keep it in the same directory. Skills that rely on tools scattered elsewhere become fragile. Skills that travel with their dependencies stay portable. ## The rest of the bench Not everything in my setup is fully integrated or daily-use. A few tools I keep within reach at different stages of exploration: ChatGPT (GPT 4.5 and above) is where I go for conversational research: thinking through a problem in plain language, getting a second opinion on an approach before committing to it in code, or just having a broad discussion that would clog an agentic workflow. Not everything needs an agent. Perplexity covers manual search and research where I want cited sources rather than a generated answer. I am also currently poking at Perplexity Computer, though it is genuinely early days and I do not have a settled opinion on it yet. Local LLMs via LM Studio and Ollama, used for offline experimentation. I will be honest: my current hardware is the constraint, not the tooling. Running anything genuinely capable locally is a stretch on my machine. If you are in the same position and want to know what you can actually run before committing to a download, LLMFit is a handy utility that evaluates your system specs and tells you which models are feasible, and worth running before you spend an afternoon downloading a 70B model that will not fit in your RAM. ## Pick one thing The setup works because of the discipline behind it, not the tools themselves: plan before executing, give agents only the permissions they actually need, write the CLAUDE.md file before you need it (not after), evaluate against metrics rather than impressions, and restart aggressively when the context window is saturated. Most of what I have described here is free or pay-as-you-go, and none of it requires a large upfront commitment to try. If you are coming to this fresh, pick one piece rather than the whole stack. Enforcing plan mode before every task will return more value more quickly than any new tool installation, and adding a CLAUDE.md to a project you already work in will pay off in the first session. If you work with Scrapy and want to add the web scraping layer that connects this setup to Zyte's toolchain, the Zyte free trial is where to start. There is more to cover — the Karpathy metric loop in more depth, how I use CLAUDE.md across different project types, and how the local LLM setup is evolving as hardware catches up. If any of that sounds worth a Part 2, let me know in the comments. And if you want to stay across what the team at Zyte is building, subscribing to the Zyte newsletter means you will not miss it.

Stop using Python `requests` for web scraping: there are better & modern libraries instead

Ayan Pahwa — Thu, 09 Apr 2026 11:09:31 +0000

While the 'Requests' library remains the default choice for many Python developers due to its reliability and extensive documentation, the Python HTTP landscape has evolved considerably.

Modern alternatives now offer significant advantages, including built-in asynchronous support, HTTP/2 compatibility, enhanced performance, and up-to-date TLS handling.

This article introduces and compares three such contemporary clients: HTTPX, curl_cffi, and rnet, detailing their unique features and practical applications.

The problem with Requests for web scraping

It's important to clarify Requests' limitations before proceeding; for simple API interactions with well-behaved endpoints, it still remains the de facto standard.

However, a major drawback of the Requests library when it comes to web scraping is its predictable HTTP client fingerprint. This fingerprint, a unique combination of TLS version, cipher suites, HTTP headers, and connection characteristics, is sent with every request, and is well-known and cataloged by anti-bot systems.

Consequently, if you're interacting with any endpoint, including APIs or services protected by anti-ban vendors, your request can be blocked purely based on how the requests library identifies itself. This happens even before your credentials or payload are scrutinized, highlighting a significant limitation when targeting systems that perform client-side validation.

In addition to issues like fingerprinting, a major limitation of the requests library is its lack of native asynchronous support. This absence of async capability is particularly problematic when handling workloads that involve numerous HTTP requests. Without it, the calls execute sequentially, and the program's thread remains blocked for the entire duration of each individual request.

For straightforward scenarios, the standard requests API call remains perfectly functional, as demonstrated in a quick example.

import requests

response = requests.get(
    "https://jsonplaceholder.typicode.com/posts/1",
    timeout=10,
)
response.raise_for_status()
data = response.json()
print(data["title"])

Clean and simple. For a one-off call to a standard REST API, this is fine. The gaps start showing when you need concurrency, HTTP/2, or when the target endpoint does any kind of client validation.

Install the Alternatives

pip install httpx       or  uv add https
pip install curl-cffi       or  uv add curl-cffi
pip install rnet        or  uv add rnet &&
                    uv add asyncio

1. HTTPX

HTTPX is the most direct upgrade from Requests as the API is nearly identical. If you know Requests, you already know most of HTTPX. What it adds is first-class async support, HTTP/2, and a more modern internal architecture.

Where it differs from Requests is the explicit use of a Client context manager (strongly recommended over module-level function calls) and the AsyncClient for async usage. This gives you connection pooling and proper resource cleanup by default.

HTTPX is the right starting point if you're looking for a migration that requires minimal code changes.

Example: Sync

import httpx

with httpx.Client(timeout=10.0) as client:
    response = client.get("https://jsonplaceholder.typicode.com/posts/1")
    response.raise_for_status()
    data = response.json()

print(data["title"])

Example: Async (calling the Zyte API)

Async is where HTTPX really earns its keep. Here it's used to fire multiple requests to the Zyte API concurrently, each request blocks on the server side until extraction is complete, but your event loop stays free to send others in parallel:

import os
import asyncio
import httpx

API_KEY = os.environ["ZYTE_API_KEY"]
ENDPOINT = "https://api.zyte.com/v1/extract"

urls = [
    "https://example.com",
    "https://httpbin.org",
]

async def fetch(client: httpx.AsyncClient, url: str) -> dict:
    response = await client.post(
        ENDPOINT,
        json={"url": url, "browserHtml": True},
        auth=(API_KEY, ""),
    )
    response.raise_for_status()
    return response.json()

async def main():
    async with httpx.AsyncClient(timeout=60.0) as client:
        results = await asyncio.gather(*[fetch(client, url) for url in urls])
    for result in results:
        print(result["url"], "—", len(result["browserHtml"]), "chars")

asyncio.run(main())

Notes:

raise_for_status() raises httpx.HTTPStatusError on 4xx/5xx responses.
HTTP/2 support requires pip install httpx[http2] and passing http2=True to the client.
The 60-second timeout accounts for the Zyte API's server-side blocking behavior — it holds the connection open until extraction completes.

2. curl_cffi

curl_cffi wraps libcurl with Python bindings and adds something HTTPX doesn't have: TLS fingerprint impersonation. It can show the TLS handshake of Chrome, Firefox, Safari, and other browsers. For API calls hitting endpoints protected by anti-ban or similar systems, this can be the difference between getting a response and getting a 403.

The interface closely mirrors Requests, with the addition of the impersonate parameter. It supports both sync and async usage. For most API calls where fingerprinting isn't a concern, curl_cffi behaves just like Requests, the impersonate parameter is opt-in.

Example: Sync

from curl_cffi import requests

response = requests.get(
    "https://jsonplaceholder.typicode.com/posts/1",
    impersonate="chrome",
    timeout=10,
)
response.raise_for_status()
data = response.json()
print(data["title"])

Example: Async (calling the Zyte API)

import os
import asyncio
from curl_cffi.requests import AsyncSession

API_KEY = os.environ["ZYTE_API_KEY"]
ENDPOINT = "https://api.zyte.com/v1/extract"

payload = {
    "url": "https://example.com",
    "browserHtml": True,
}

async def call_zyte_api():
    async with AsyncSession(impersonate="chrome") as session:
        response = await session.post(
            ENDPOINT,
            json=payload,
            auth=(API_KEY, ""),
            timeout=60,
        )
        response.raise_for_status()
        data = response.json()
        print(data["url"], "—", len(data["browserHtml"]), "chars")

asyncio.run(call_zyte_api())

Notes:

impersonate="chrome" sends Chrome's TLS fingerprint on every request made through this session.
Other supported values include "firefox", "safari", "chrome110", and more — check the curl-cffi docs for the full list.
The sync interface (from curl_cffi import requests) is nearly identical to the requests module, making it the easiest drop-in if you only need sync.

3. rnet

rnet is the newest of the three. Like a lot of modern Python, it's built on Rust, making it async-first and performance-oriented. Like curl_cffi, it supports TLS impersonation, but its primary differentiator is throughput. It is designed for high-concurrency workloads where you're firing many requests simultaneously.

The API surface is different from Requests, so it's not a drop-in replacement. But the patterns are clean and modern, and for async-heavy workloads it's worth the minor adjustment.

Example: Sample library code

import asyncio
from rnet import Impersonate, Client


async def main():
    # Build a client
    client = Client(impersonate=Impersonate.Firefox139)

    # Use the API you're already familiar with
    resp = await client.get("https://tls.peet.ws/api/all")

    # Print the response
    print(await resp.text())


if __name__ == "__main__":
    asyncio.run(main())

Notes:

rnet is async-first; sync support is limited.
Response body methods like .json() and .text() are awaitable.
The Rust core makes it particularly well-suited for high-throughput concurrent workloads.

Comparison Table

Feature	Requests	HTTPX	curl_cffi	rnet
Sync Support	✅ Yes	✅ Yes	✅ Yes	⚠️ Limited
Async support	❌ No	✅ Yes	✅ Yes	✅ Yes (primary)
HTTP/2	❌ No	✅ With extra dependencies	✅ Via libcurl	✅ Built-in
Performance	Baseline	Good	Good–High	High
TLS changes	❌ No	❌ No	✅ Yes	✅ Yes

When to use which

Use Requests for simple, one-off scripts, internal tooling, or any situation where you're hitting a cooperative API endpoint and don't need concurrency. Nothing wrong with it in that context.

Use HTTPX when you need async, want the closest migration path from Requests, or need HTTP/2. It's the safest default upgrade for most projects.

Use curl_cffi when TLS fingerprint control matters, whether that's because you're hitting an anti-ban wall or an API with strict client validation, or any service that checks how a client identifies itself at the TLS layer.

Use rnet when raw async performance is the priority. Its Rust foundation makes it the strongest choice for high-concurrency workloads where you're firing many requests simultaneously and need low overhead.

The optimal choice is determined by several factors: your concurrency requirements, the target endpoint's sensitivity to client identification, and the desired similarity between the new code and your existing requests implementation.

Small models, big ideas: what Google Gemma and MoE mean for developers

Ayan Pahwa — Tue, 07 Apr 2026 12:50:03 +0000

We at zyte-devrel try to stay plugged into what is happening in the AI and developer tooling space, not just because it is interesting, but because a lot of it starts having real implications for how we build and think about web data pipelines. Lately, one development that has had us genuinely curious is Google's new Gemma 4 model family, and specifically the direction it points toward with Mixture of Experts (MoE) architecture.

This is not a deep tutorial. It is more of a "hey, here is what we have been poking at" - the kind of update we would share in a Slack channel or over coffee. If you wanna participate in such discussions, our discord is always a welcoming platform.

What is Gemma 4?

Gemma has been dubbed as stripped down versions of Google Gemini. The new Gemma 4 is Google's latest family of open-weight language models, released last week. The lineup covers four sizes:

2B: ultra-efficient, built for mobile and edge devices
4B: enhanced multimodal capabilities, still edge-deployable
26B: sparse model using Mixture of Experts architecture (more on this below)
31B: dense model for more demanding tasks

All four variants support multimodal input (text and images), over 140 languages, a 128K-256K token context window, and agentic workflows with tool use and JSON output. The 2B and 4B models are specifically designed to run fully offline on modern edge devices like smartphones, with no internet dependency at all.

According to Google's Gemma 4 model page, the family ranks third among open-weighted models on the LM Arena leaderboard and uses 2.5 times fewer tokens than comparable models for equivalent tasks.

The Gemma 4 26B MoE, specially caught my attention because unlike other variants it's based on MoE architecture and it does make a difference :

What is MoE, and why does it matter?

Mixture of Experts (MoE) is one of those ideas that sounds complex but is actually pretty intuitive once you hear the analogy.

In a traditional dense neural network, every parameter in the model activates for every input. It is like calling your entire company into a meeting every time someone has a question. It works, but it is expensive.

MoE works differently. Instead of one large model doing everything, you have a set of smaller "expert" sub-networks, each specialized in different patterns, plus a router that looks at each incoming token and decides which one or two experts to activate. Most of the model sits idle at any given moment.

The result: you get the quality of a much larger model at a fraction of the inference cost.

The Gemma 4 26B model is a great illustration of this. It has 26 billion total parameters, but during inference it only activates around 3.8 billion of them. You get near-26B quality at roughly 3.8B compute cost. That is the MoE advantage in one number.

Other models that take the same approach:

Mixtral 8x7B: eight experts, two active per token; it outperforms Llama 2 70B on most benchmarks at far lower inference cost
Kimi: Moonshot AI's model, also MoE-based, has been making similar waves in the open-model space

For a deep dive on how MoE works under the hood, the Hugging Face guide to mixture of experts is well worth the read.

Since the models are free, if you have the right machine you can host them lcoally using Ollama or call them using API services like OpenRouter.

My prefered way of using a new mode is through Claude, but I believe Gemma4 has a different tool calling structure so it is not compatible yet, but you can use it with LMStudio, or skil all that because you can now

Run Gemma 4 offline on an iPhone

Here is the part worth sharing, because it genuinely surprised us.

Using the Google Edge AI Gallery app from the App Store, you can load a Gemma 4 model and run it with airplane mode on. No API calls, no cloud round-trips, no data leaving the device. Just the model running locally on your phone.

The experience is not going to replace a foundational frontier model for complex reasoning. But that is not the point. For quick classification, summarization, or just experimenting with local inference, the 2B and 4B variants are remarkably capable, and there are zero API costs with no data leaving your device. And since it is multi-modal you can practically point your phone camera to a paper recipt and ask it to save the details in a spreadhseet.

If you have not tried running a local large language model (LLM) yet, this is probably the lowest-friction entry point on hardware you already own.

Why should developers building data pipelines care?

Here is where it connects back to what a lot of us are building.

When LLMs run on-device or at the edge, the calculus around data pipelines shifts in a few useful directions:

Tokens are getting expensive and when a model as good as Gemma 4 or Qwen-3.5 is free and open-weighted it's a welcome development. Everyone's complaining about running out of their claude usage quota last couple of weeks or getting huge bills, thanks to giving Opus API Keys to OpenClaw. These things can be significantly addressed using Open Models.

No API round-trips: on-device inference eliminates latency from cloud API calls. For classification tasks running inside a scraping pipeline, this is a meaningful difference.
Data privacy: running extraction locally means scraped content never leaves your infrastructure. For regulated industries or sensitive datasets, that is a significant advantage.
Cost at scale: if you are doing high-volume classification — is this a product page? is this content in the target language? — running a small local model beats paying per-token at scale.
Edge preprocessing: a small LLM can filter and classify pages before they ever reach a more expensive cloud model for deeper analysis, and I am personally looking forward to run them on SBCs like a Raspberry Pi.
Open Weights: people often confuse open-weights models with open-source models, while the lines may be blurry and even I don't fully understand the difference, one thing I know for sure is that Gemma 4 is available under the Apache 2.0 license, which allows building and selling products on top of it and open-weights allows you to fine-tune it for your use-case or application.

Here's me playing it with it on my iPhone 16, completely offline:

Just checking in

We do not have grand proclamations here. This is a space that is moving fast, and we are learning alongside everyone else.

If you have been experimenting with local LLMs in your scraping or data extraction workflows, we would genuinely love to hear about it. Drop a comment below, or find us on the Zyte discord and read more interesting blogs on Zyte Blog.

If you want to try this yourself, here are three good starting points:

Google Edge Gallery: available on the App Store and Playstore, runs Gemma 4 locally on iOS
Gemma models on Hugging Face: for running on desktop or server
Google's Gemma 4 model page: full family overview, benchmarks, and architecture details

Headless web exploration is the way to go!

Ayan Pahwa — Mon, 16 Mar 2026 09:21:25 +0000

I built a Claude Code skill that screenshots any website (and it handles anti-bot sites too)

Ayan Pahwa for Extract by Zyte ・ Mar 6

#claude #webscraping #python #ai

I built a Claude Code skill that screenshots any website (and it handles anti-bot sites too)

Ayan Pahwa — Fri, 06 Mar 2026 14:40:32 +0000

TLDR;

Automate screenshot capture for any URL with JavaScript rendering and anti-ban protection — straight from your AI assistant.

Taking a screenshot of a webpage sounds trivial, until you need to do it at scale. Modern websites throw every obstacle imaginable in your way: JavaScript-rendered content that only appears after a React bundle loads, bot-detection systems that serve blank pages to automated headless browsers, geo-blocked content, and CAPTCHAs that appear the moment traffic patterns look non-human. For a handful of URLs you can get away with Puppeteer or Playwright. For hundreds or thousands? You need infrastructure built for the job.

The Zyte API was designed specifically for this problem. It handles JavaScript rendering, anti-bot fingerprinting, rotating proxies, and headless browser management so you don't have to and what better way to do it straight from your LLM supplying the URLs? Hence I created this zyte-screenshots Claude Skill, which you can use to trigger the entire workflow- API call, base64 decode, PNG save on your filesystem, all just by chatting with Claude.

In this tutorial, we'll walk through exactly how the skill works, how to set it up, and how to use it to capture production-quality screenshots of any URL.

Why Use the Zyte API for Screenshots?

Before diving into the skill itself, it's worth understanding what makes the Zyte API uniquely suited to screenshot capture at scale.

1. Full JavaScript Rendering

Single-page applications built with React, Vue, Angular, or Next.js don't serve their content in the raw HTML response, they render it client-side after the page loads. Tools that capture the raw HTTP response will get a blank shell. Zyte's screenshot endpoint fires a real headless browser, waits for the DOM to fully settle, then captures the final rendered state.

2. Anti-Bot and Anti-Ban Protection

Enterprise-grade sites use fingerprinting libraries to detect automation. They check TLS fingerprints, browser headers, canvas rendering patterns, mouse movement entropy, and dozens of other signals. Zyte's infrastructure is battle-tested to pass these checks so your screenshots won't return a "Access Denied" page.

3. Scale Without Infrastructure

Managing a fleet of headless browser instances, proxy rotation, retries, and residential IP pools is a serious engineering investment. Zyte abstracts all of this into a single API call.

4. One API, Any URL

Whether the target is a static HTML page, a JS-heavy SPA, a behind-login dashboard (with session cookies), or a geo-restricted site, the same API call structure works. The skill you're about to install uses this endpoint.

What Is the zyte-screenshots Claude Skill?

Claude Skills are reusable instruction packages that extend Claude's capabilities with domain-specific workflows. The zyte-screenshots skill teaches Claude how to:

Accept a URL from the user in natural language
Read the ZYTE_API_KEY environment variable
Construct and execute the correct curl command against https://api.zyte.com/v1/extract
Pipe the JSON response through jq and base64 --decode to produce a PNG file
Derive a clean filename from the URL (e.g. https://quotes.toscrape.com becomes quotes.toscrape.png)
Report the exact file path and describe what's visible in the screenshot in one sentence

In practice, this means you can open Claude, say "screenshot https://example.com", and have a pixel-perfect PNG on your filesystem in seconds, no browser, no script, no Puppeteer config.

Prerequisites

Before installing the skill, make sure you have the following:

Tools

curl: Pre-installed on macOS and most Linux distributions. On Windows, use WSL or Git Bash.
jq: A lightweight JSON processor. Install via brew install jq (macOS) or sudo apt install jq (Ubuntu/Debian).
base64: Standard on all Unix-like systems.
Claude desktop app with Skills support enabled.

A Zyte API Key

Sign up at zyte.com and navigate to your API credentials. The free tier includes enough credits to get started with testing. Copy your API key, you'll set it as an environment variable.

💡 Pro tip: Set your ZYTE_API_KEY in your shell profile (~/.zshrc or ~/.bashrc) so it's always available: export ZYTE_API_KEY="your_key_here" or pass it along your prompt

Installing the Skill

Step 1: Download the Skill from GitHub

The skill is open source and available at github.com/apscrapes/claude-zyte-screenshots. Download the latest release ZIP from the repository's Releases page, or clone it directly:

git clone https://github.com/apscrapes/claude-zyte-screenshots.git

Step 2: Import into Claude

Open the Claude desktop app or go to Claude.ai
Navigate to Settings → Skills
Click Import Skill
Select the SKILL.md file from the downloaded ZIP
Claude will confirm the skill has been installed

Step 3: Set Your API Key

In your terminal, export your Zyte API key or pass it along your prompt:

export ZYTE_API_KEY="your_zyte_api_key_here"

To make this permanent across terminal sessions, add the export line to your ~/.zshrc or ~/.bashrc file.

Using the Skill: Step-by-Step

Basic Screenshot

Once the skill is installed and your API key is set, using it is as simple as asking Claude:

Screenshot https://quotes.toscrape.com using zyte-screenshots skill

Claude will:

Verify ZYTE_API_KEY is available in the environment
Derive the output filename: quotes.toscrape.png
Execute the following command:

curl -s https://api.zyte.com/v1/extract \
  -u "$ZYTE_API_KEY": \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com",
    "screenshot": true
  }' \
| jq -r '.screenshot' \
| base64 --decode > quotes.toscrape.png

Verify the file is non-empty
Open and inspect the image
Return the file location and a one-line visual description

Example output from Claude:

✅ Screenshot captured!

📁 Location: /home/you/quotes.toscrape.png

🖼️  What's in it: A clean light-themed page listing inspirational quotes
   with author attributions and tag filters in the sidebar.

Under the Hood: How the API Call Works

Let's break down the exact curl command the skill executes:

curl -s https://api.zyte.com/v1/extract \
  -u "$ZYTE_API_KEY": \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://target-site.com",
    "screenshot": true
  }' \
| jq -r '.screenshot' \
| base64 --decode > output.png

curl -s — Silent mode; suppresses progress output.

-u "$ZYTE_API_KEY": — HTTP Basic Auth. Zyte uses the API key as the username with an empty password.

-H "Content-Type: application/json" — Tells the API to expect a JSON body.

-d '{...}' — The JSON request body. Setting screenshot: true instructs Zyte to return a base64-encoded PNG of the fully rendered page.

| jq -r '.screenshot' — Extracts the raw base64 string from the JSON response.

| base64 --decode — Decodes the base64 string into binary PNG data.

> output.png — Writes the binary data to a PNG file.

The Zyte API handles everything in between — spinning up a headless Chromium instance, loading the page with real browser fingerprints, waiting for JavaScript execution to complete, and rendering the final DOM to a pixel buffer.

This was a fun weekend project I put together, let me know your thoughts on our Discord and feel free to play around with it. I'd also love to know if you create any useful claude skills or mcp server, so say hi on our discord.

Tags: web scraping • Zyte API • screenshots at scale • JavaScript rendering • anti-bot • Claude AI • Claude Skills • automation • headless browser • site APIs

Raspberry Pi & E-ink scrapes & displays the price of Gold today

Ayan Pahwa — Tue, 24 Feb 2026 08:39:31 +0000

They say that “data is the new oil”, but there’s another hot commodity that’s setting markets alight - precious metals.

In the last 12 months, the value of gold has surged about 75%, while silver has boomed more than 200%. That’s why I, like a growing number of others, now trade in the metal markets.

These days, it is possible to buy digital versions of precious metals. But I think of myself as a collector - I like to buy real, solid coins or bullions whenever I get a chance.

In the last two years, I have acquired a small collection of gold bullions and silver coins, which have appreciated healthily. But I am not planning to sell and book a profit just yet. In fact, I want to buy more, especially when there’s a dip in the price.

There’s just one problem that hits this hobby - prices of actual physical gold and silver bullions are very different in the retail market from stock exchanges’ spot prices and keeping track of them manually is cumbersome specially with a full time job.

To take advantage of the dips and price arbitrage, I need to automate my decisions. To buy gold old-style, I need a key resource from the modern trading toolset - data.

Turn data into gold

The All-India Gem And Jewellery Domestic Council (GJC), a national trade federation for the promotion and growth of trade in gems and jewellery across, is the go-to site listing latest retail rates for gold and silver.

Alas, it doesn’t offer an API to access that data. But fear not - with web scraping skills and Zyte API, I can extract these prices quickly and regularly.

And I can do it using some of the tech I love to tinker with.

I call it ExtractToInk - a custom project that pulls the latest prices on a two-inch, 250x122 e-ink display powered by a retired Raspberry Pi (total cost under US$50).

This is the story of how I power my quest for rapid riches using cheap old hardware and the world’s best web scraping engine - and how you can, too.

Mining for data

Like many modern sites, GJC’s includes both JavaScript rendering for HTML and protection mechanisms- technologies that can break brittle traditional scraping solutions.

This project connects all the dots:

Web → Extract → Parse → Render → Physical display

Tech stack

Hardware

Raspberry Pi (tested on Pi Zero 2 W), it should run on any Raspberry Pi Board
Pimoroni Inky pHAT (Black, SSD1608)

Software

Python 3
Zyte API: to get rendered HTML
BeautifulSoup: to parse HTML
Pillow and Inky Python libraries: for e-ink display stuff

Now let’s get building.

Step 1: Prepare hardware

Setup your Raspberry Pi. In my case, I am using Raspberry Pi OS booted from the SD card.

Depending on which display you use, it most probably will be connected to the Pi over i2c bus or SPI bus protocol - so, enable your display type by entering:

sudo raspi-config

Now attach your e-ink display and do a quick reboot

You might need to install libraries to use your e-ink display.

Step 2: Fetching rendered HTML with Zyte API

The source site, GJC, renders prices dynamically, using JavaScript - something which can make plain HTTP requests unreliable.

No problem. By accessing the page through Zyte API, we can set browserHTML mode to return the page content as though rendered in an actual browser.

Instead of fighting JavaScript, we let Zyte handle it.

html = requests.post(`  
    `"https://api.zyte.com/v1/extract",`  
    `auth=(ZYTE_API_KEY, ""),`  
    `json={"url": URL, "browserHtml": True},`  
`).json()["browserHtml"]

Note: there is no Selenium here, and no headless browsers. This is much more reliable for production-style scraping

Step 3: Parsing with CSS selectors

Once we have clean HTML, parsing becomes straightforward.

Gold prices

Let’s locate the actual prices in the page mark-up.

for row in soup.select(".gold_rate table tr"):`  
    `label = row.select_one("td strong")`  
    `values = row.select("td strong")`

    `if not label or len(values) < 2:`  
        `continue`

    `text = label.get_text(strip=True)`  
    `priceText = values[1].get_text()`

    `if "Standard Rate Buying" in text:`  
        `goldBuying = re.search(r"\d[\d,]*", priceText).group(0)`

    `if "Standard Rate Selling" in text:`  
        `goldSelling = re.search(r"\d[\d,]*", priceText).group(0)

We’re deliberately using:

CSS selectors (easy to find from your browser’s DevTools).
Minimal regular expressions (only for numeric extraction).
Defensive checks to avoid brittle parsing.

Silver prices

Silver appears outside the main table, so we filter it carefully:

for strong in soup.select("p > strong"):`  
    `text = strong.get_text(" ", strip=True)`

    `if "Standard Rate Selling" in text and not strong.find_parent("table"):`  
        `silver = re.search(r"\d[\d,]*", text).group(0)`  
        `break

Step 4: Rendering for e-ink

For this project, I did not want to pipe data into a web dashboard on a computer monitor.

E-ink is always-on, low power, distraction-free and perfect for “ambient information” like this.

So, it’s a great fit for data like prices, weather, status indicators and system health.

But e-ink displays are not normal screens.

They are typically black-and-white, have high contrast and are slow to refresh.

What’s more, no two e-ink displays are made the same way. Every vendor has different support packages so, whichever you end up using, make sure to read the documentation and change the code accordingly.

In my case, I am using Pimoroni inky PHat. The supplied Python library has great built-in examples to get you quickly up and running. I used the helper function to render texts on the display, ex, the build in draw.text() function comes handy:

Draw silver selling price

    draw.text((x, y), f"Silver : {silverPrice}", fill=(0, 0, 0), font=fontBig)

Taking it furtherSection about the finished product

I built this project to use web data thoughtfully, connecting it to the physical world, and building pipelines that feel calm, reliable, and purposeful. When I am at my work-desk the project actively tells me the current prices so I can buy new coins if I see a price drop.

I can further extend this to place automatic orders on the website and secure me a coin at my desired strike price.

If you want to take this further, you could also:

Run it via cron every 10 minutes : The website I am targeting only refreshes prices twice a day, so my cron job runs every 12 hours, but, if you need faster data, you can scrape a site with more real-time updates.
Add more commodities or currencies.
Turn it into a systemd service to run at start time.
Swap e-ink for another output (PDF, LED, dashboard).

If you’re exploring Zyte API, or looking for real-world scraping examples beyond CSVs and JSON files, this project is a great place to start.

You can get my code in the ExtractToInk GitHub repository now.

Holiday Gift Guide 2025: For Developers, Web Scrapers & Everyone In Between

Ayan Pahwa — Fri, 19 Dec 2025 07:55:12 +0000

It’s that time of the year when the coffee gets stronger, commits get messier, and everyone agrees to finally refactor that script in January. And let’s be honest, most of us won’t. But while it lasts let’s celebrate the season of enjoying laid back family time and exchanging gifts.
And to make gifting a little easier, we asked the Zyte team and community to share what they would love to receive. So if you’ve got a developer, a web scraper, or someone who just really enjoys arguing with APIs in your life… Here's your cheat sheet.
Grab a hot drink, settle into your favourite debugging position, and let’s dive in.

Disclaimer : It's not sponsored. All the recommendations are from community and author's personal experience using these products, hence no URL is provided but you should be able find these easily.

1. Book: Soul of a New Machine

For the dev who loves origin stories. It’s a classic engineering tale that reminds us why we fell in love with building things in the first place.

2. Logitech MX Master mouse

Smooth scrolling. Perfect ergonomics. Side buttons that feel like cheat codes for productivity. This is the mouse equivalent of finally finding that one undocumented API endpoint you needed all year.

3. AeroPress + Coffee Subscription

If coffee is the real runtime powering your favourite developer, this combo is basically a performance upgrade. AeroPress means fast, clean brews; fresh beans mean they might actually fix that bug before lunch.

4. Spider Plush Toy

For every web scraper who proudly identifies as “part human, part spider.” it sits silently on your desk reminding you to obey robots.txt (…most of the time).

5. Git Merch Based on Their Commit History

A mug that says “I survived the merge conflict of 2025.”
A T-shirt celebrating their 3,000-day streak.
Or maybe a gentle reminder that “WIP” is not a personality.
Funny, personal, and guaranteed to make them smile during standup.
Link : https://gitmerch.com/ (not-sponsored), just enter their github username

6. Nothing Headphones

Sleek, transparent, and great for tuning out noisy offices or noisy families. Perfect for deep work, debugging, or pretending you can’t hear someone asking, “Can you take a quick look at this?”

7. Mechanical Keyboard (NuPhy / Keychron)

Developers don’t just type on these; they perform.
Clicky keys, gorgeous layouts, RGB that could guide aircraft, what’s not to love? Warning: once they switch, they’ll never stop talking about actuation force.

8. Bambu Lab A1 Mini 3D Printer

For the dev who already has too many hobbies.
They’ll print brackets, cable holders, figurines, replacement parts… and things no one can identify but everyone politely admires. A creator’s playground in miniature.

9. BenQ Monitor Light Bar

Immediate upgrade to any desk setup. Reduces eye strain, looks clean, and helps developers see their code clearly even during late-night “just one more function” sessions without taking extra space on their desk.

10. 100W GaN Charger

Tiny but absurdly powerful, just like that one script they wrote at 3 AM that still runs in production. A GaN charger keeps everything powered: laptop, phone, tablet, e-reader, existential dread, everything.

If you love someone who spends their days coaxing data out of websites, obsessing over keyboard switches, or whispering sweet nothings to their terminal, this list has something that will make them light up.
Here’s to a warm, restful holiday season… and to fewer bugs in 2026. Happy gifting, happy coding, and as always, happy scraping! 🕷️✨

Build Your Own Holiday Deal Tracker with Python, Zyte API & IFTTT 🎁

Ayan Pahwa — Fri, 21 Nov 2025 11:06:18 +0000

(A simple, developer-friendly project to help you catch Black Friday, Cyber Monday, and Christmas deals before anyone else)

Watch the video tutorial :

It’s that time of the year again, everyone’s hunting for discounts, limited-time deals, and that one item you’ve been keeping an eye on all year. You know the drill: tabs open everywhere, price tracker sites breaking under traffic, browser extensions that promise magic but fail right when The Deal drops.

So this year, I decided to build something different, something that actually works when I need it the most.

A personal price-alert system, powered by:

Python 🐍
Zyte’s Automatic Extraction API 🕷️
An IFTTT mobile notification trigger 📱

And honestly? It ended up being one of the simplest, useful and most reliable holiday project I’ve made in a long time.

❌ No HTML parsing.
❌ No complex CSS selectors.
❌ No brittle scrapers that break during peak traffic.
✅Just a clean API call, structured product data, and a push notification when the price hits your set target.

Let me walk you through the whole thing :

🎁 Why Build Your Own Deal Tracker?

Because holiday deals don’t wait for anyone. Every year I get messages from friends asking,

“Bro, what’s the best price tracker? Nothing seems to work today.”

And they’re right. Most free services throttle, break, or go down completely during Black Friday and Cyber Monday because everyone hits them at once.

Meanwhile, with a few lines of Python and Zyte’s AI-powered extraction, you can build something that:

Works only for you
Deals with anti-ban, anti-bot and captchas
Checks as often as you want (make it run on your pc, github-actions, server or a raspberry pi)
Doesn’t rely on brittle selectors
Doesn’t choke on JavaScript
Doesn’t get rate-limited
Sends you a mobile ping instantly when price drops

It’s the perfect mix of practical dev fun + a tool you’ll actually use.

🎄 Why Zyte?

Scraping e-commerce sites for price data is usually a pain: blocking, JavaScript rendering, cookie rules, bot detection, dynamic DOMs, infinite variations in product page layouts... you get it.

The magic here is Zyte’s Automatic Extraction.

You literally tell AI powered Zyte API:

{ 
"url": "<product-url>", 
"product": true 
}

and it returns structured fields like:

price
currency
product name
sku
images
stock availability
description

You don’t write selectors. You don’t parse HTML. You don’t chase CSS changes. You just get clean data. And that makes this holiday project absurdly simple.

🎅 What We’re Building

A script that:

Takes:
a product URL
a target price
your Zyte API key
your IFTTT Webhook key
Fetches structured product data using Zyte API
Checks if the product price is ≤ your target
If yes → triggers an IFTTT event

Your phone instantly notifies you with product URL:
“Your product dropped to your target price. Go get it!”

You can run this:

manually (when you want)
on a cron job (in the background)
in GitHub Actions
on a Raspberry Pi
on a cloud function Your call.

🛠 Setup

Clone your project

git clone https://github.com/apscrapes/zyte-sale-alert.git
cd zyte-sale-alert

Create your virtual environment

python3 -m venv venv
source venv/bin/activate

Install dependencies

pip install -r requirements.txt

Create and add secrets in .env file

ZYTE_API_KEY=your_zyte_key_here (obtained from zyte)
IFTTT_KEY=your_webhooks_key (obtained from IFTTT, see next step)
IFTTT_EVENT=price_drop (name of your ifttt applet, see next step)
TARGET_PRICE=149.99 (example)
PRODUCT_URL=https://example.com/product/123

Create an IFTTT Webhook Applet

a. Download IFTTT mobile app, it’s paid but you get a 7-day trial and it has tons of automation you can build using no-code, so i think it’s worth it.

b. Click create new automation / applet

c. In “IF” field, select “Webhooks” > Add Event Name

d. In “THEN” field select Notification > App Notification > JSON Payload > High Priority

Note : You can instead of notification can also set other automation like getting an email for example

e. Here's how it should look like when it's successfully created:

Once the automation is created add your IFTTT_KEY and IFTTT_EVENT to your .env file

Set product parameters The main script is at src/pricedrop.py, edit the following variables to add your:

PRODUCT_URL = “ ”
DESIRED_PRICE = 250

PRODUCT_URL “ ” is the URL of the product you want to track and DESIRED_PRICE is the price at which you want to be notified, that is it.

Run the project

python src/pricedrop.py

You can run it manually or set up a cronjob to run at regular intervals. Whenever the price drops equal or below your target price you’ll get a notification from the IFTTT app on your phone with product URL so you can order it right away.

Let’s understand how it works.

🧩 Core Logic (Short Version)

resp = client.get({"url": url, "product": True})

“product : True” tells zyte API that the webpage we’re scraping has a product so the Machine Learning powered scrapper gets you all the relevant parameters like price, quantity, description, currency etc.

And that’s literally all you need.

The reason this works so beautifully is Zyte is handling:
JS rendering
blocking
retries
browser simulation
extraction logic
AI-powered field detection

In-detail :

Importing all the necessary libraries

import os
import sys
import requests
from zyte_api import ZyteAPI
from dotenv import load_dotenv

Setting required variables

PRODUCT_URL = "https://outdoor.hylnd7.com/product/a1b2c3d4-e5f6-4a7b-8c9d-000000000293"
DESIRED_PRICE = 250

Here we've setup a sample product link whose price we want to track and a price at which if it goes below we want to be notified.

Loading the API keys from .env file

load_dotenv()

# from Zyte API
ZYTE_API_KEY = os.getenv("ZYTE_API_KEY")

# from IFTTT Service applets
EVENT_NAME= os.getenv("EVENT_NAME")
IFTTT_KEY= os.getenv("IFTTT_KEY")

if not ZYTE_API_KEY:
    print("ERROR: ZYTE_API_KEY not found in environment.")
    sys.exit(1)

In the project root directory, create a .env file and add ZYTE API Key you'll get after logging into zyte.com and IFTTT webhook API key you get after creating the automation applet

Function to trigger the mobile notification :

def trigger_ifttt(event_name, key, value1):
    url = f"https://maker.ifttt.com/trigger/{event_name}/json/with/key/{key}"

    payload = {
        "value1": value1,
    }

What this funciton is doing is basically making an API call to IFTTT and IFTTT applet is set so whenever the API calls comes with payload it sends mobile notification with that payload, which in this case is product URL so you can directly click and open the product page and buy it before it goes out of stock, SMART right? 😉

Scraping init

client = ZyteAPI(api_key=ZYTE_API_KEY)

    payload = {
        "url": PRODUCT_URL,
        "product": True,          
    }

    resp = client.get(payload)

Making a GET request on Zyte API with product : True, we're asking zyte to treat the URL as product page and thus it's uses it's ML capabilities to fetch product relevant details, price in this case.

Compare price to SETPOINT

if price_float <= DESIRED_PRICE:
            trigger_ifttt(EVENT_NAME, IFTTT_KEY, value1 = PRODUCT_URL)

If price of the product reaches to or below our target price it will call the IFTTT function, thus triggering the notification.

🌟 Make It Even Better

You can extend this to:

Track multiple URLs
Log daily prices to CSV
Plot graphs
Send WhatsApp alerts
Push to a Telegram bot
Use GitHub Actions to check every hour
Deploy as a Streamlit dashboard

Zyte handles the extraction. You build the magic on top.

🧘 Final Thoughts

I love projects like this because they hit the sweet spot between:

seasonal usefulness
real-world scraping challenges
a clean developer experience
a fun weekend build

If you're new to the world of web scraping like me, this shows how powerful the right tools can be. If you're experienced, it’s refreshing to skip the boilerplate and let Zyte handle the messy parts.

And honestly, there’s something fun about getting a custom alert on your phone saying:
“Hey, that gadget you wanted all year just dropped to your target price.”

Happy building, happy holidays, and happy deal-hunting! 🎄🎁
Let me know what you end up tracking.

Join Zyte Discord to share what you're building or get any support :
https://discord.com/invite/DwTnbrm83s

@iayanpahwa

Build a crypto miner using Raspberry Pi in 10 minutes

Ayan Pahwa — Wed, 15 Jun 2022 13:44:17 +0000

Lately I’ve been hearing a lot of hype around web3- cryptocurrencies, NFTs, DAOs, DeFi, GameFi and all of these cool jargons giving me a lot of FOMO (Fear of missing out), so I decided that I’m gonna try some of them for myself and see what’s it all about.

I tried a few things like buying some cryptos on an exchange, made a couple of web3 projects and joined a gazillion of discord channels doing the morning ritual of gm(good morning!) and responding to WAGMI(we all gonna make it!) but this one specific project called Monero caught my attention and I wanted to share it with the community here.

Before proceeding I want to put out that this is in no way an endorsement of the project or whole blockchain/web3 ecosystem in general, nor it’s a financial, get rich by mining cryptocurrency advice. This is in fact a fun project which you can build along to learn more about web3 and being a developer advocate, I’m also gonna use this project to share about the concept of :

Using a separate docker images for build-time and run-time within the same Dockerfile, we will see why it’s important and how to achieve that later in this post

Nevertheless, this project is kind of a fun to build, it allows you to mine cryptocurrency called XMR of Monero blockchain, profitable or not you can no doubt brag about mining cryptocurrency at your home to your friends, so let’s get started 😉

You can skip the inner details and jump straight to the build part down in the post if you like.

Background

Without going too much into the detail, for someone new into the web3 ecosystem - A blockchain is a distributed ledger which keeps records of every transaction happening over it. Just like how a bank keeps the record of who sent money to whom, the amount as well as the date and time, similarly in case of blockchain this immutable information is maintained within distributed blocks connected by a network.

This validation of all the transactions on a blockchain is done by certain users who lend their compute power in the form of miners or validator nodes. There are also some programs called smart contracts which can run automatically on blockchain when certain conditions are met but that is out of scope for this blog so will not go into more details of smart contracts.

When you think of Bitcoin or Ethereum miners you might imagine a big server room consisting of massive GPUs or ASICs machines dedicatedly solving complex cryptographic problems sent to them by blockchain and once they solve it they earn rewards in the form of cryptocurrencies The process of successful submission to earn the rewards follows certain consensus mechanism which can vary depending on the type of blockchain

For example: Bitcoin blockchain works on a consensus mechanism known as Proof of Work (PoW), while Solana works on Proof of Stake (PoS) and Proof of History(PoH).
You can read all about types of consensus mechanism here

If you’re feeling disheartened seeing the cost of these mining computers crushing your dream to have a mining rig of yourself, don’t be because the project we’re talking today called Monero allows or in fact encourages mining on CPUs, so even a small single board computer like Raspberry Pi 4 can become a miner to help validate the transactions and get the rewards in form of XMRs, the cryptocurrency of monero blockchain.

The philosophy behind this is since the cost and entry barrier to mine is pretty high, the miners are usually owned by few persons or organizations which may not be good for blockchain’s decentralization (no ownership and trustlesness by design), so monero optimizes to allow more and more people to contribute making their blockchain more decentralized. Monero also works on Proof of Work but instead of your small device solving a really complex cryptography puzzle it can join a mining pool with other devices to lend the compute power and together they can solve it fast and depending on how much your device contributed to the solution it’ll be rewarded suitably for that.

Is it profitable? Maybe not, given the device needs to be powered on and running but it surely is fun and maybe you can really earn if you add a lot of devices in your mining pool with good resources, say Rpi with 8 GB RAM; nevertheless it’s not financial advice so please do your due diligence :)

You can read all about Monero project, how much it pays for mining, costs associated, etc all on official website: here

Build

Hardware Needed

A single board computer like raspberry pi 4 (more ram the better).

The following SBCs were tested by my friend Lambros and this was the hashrate result:

- Raspberry Pi 3  - 20 H/s
- Raspberry Pi 4, 1 GB RAM - 45 H/s
- Raspberry Pi 4, 4 GB RAM - 99 H/s
- Nvidia Jetson Nano 2GB (without GPU enabled) - 62 H/s

Hash rate(H/s) is the number of hashes a device can solve per second. I’ve not enabled CUDA(GPU backend) for Nvidia Jetson since monero encourages mining on CPU though if you want to give it a shot I’d love to see how it performs.

Other things you'll need are

SD card
Power supply
Connectivity to Internet via LAN or WiFi

Software needed

monero wallet (this is where you’ll get the rewards) download from https://www.getmonero.org/downloads/
Information of mining pools : as mentioned earlier we will be joining a mining pool, the address of active mining pools can be found by a simple google query, it’s better to use one near you ex: google search monero mining pools in Los Angeles. Each mining pool has their own threshold after which they start paying out to the wallets.
balenaCloud account to manage miners
balenaEtcher to flash the SD card

Deployment

The easy way (deploy with balena)

Sign up for a free balenaCloud account. Your first ten devices are free and fully-featured! Then use the button below to create and deploy the application:

Note: I have used a Raspberry Pi 4 in the image below but be sure to select the correct device type for the device you are using.

Select the Device
Add OS type: Production vs Development
(optionally) add in your home WiFi credentials, download and flash the OS to SD card using etcher

The Advanced way

If you are already a balena user it might be better for you to use this way. You can clone the project from this github repo and use the balena CLI command

balena push <fleet_name>

to push the application to your devices in the fleet created on dashboard. This is the best option if you want to tinker with the project and have full control. The Getting Started Guide covers this option. After you've created the application and pushed the code using the CLI, follow the steps below.

First device boot and configurations

When the device boots for the first time, it connects to the balenaCloud dashboard, after which you’ll be able to see it listed online. In the meanwhile we need to Install Monero wallet on our computer where we will be getting the mining rewards

Software Setup

Download the Monero wallet from https://www.getmonero.org/downloads/ GUI or CLI as per your preference. I recommend using a GUI wallet and create a new wallet using Simple Mode.

Once the wallet is created you’ll have a unique wallet address copy that to clipboard and head over to:

balena cloud > your device > Device Variables

tab and add the following

Device Variables

VARIABLE NAME	VALUE	CHANGE TYPE	DESCRIPTION
WALLET_ADDRESS	from last step	Must add	This is the wallet address where you’ll earn your mining rewards in the form of XMRs which you can trade on any crypto exchange
MINER_POOL	Default value is: http://xmr.2miners.com:2222/	Optional	This is the miner pool you will join by default. You can change this to another miner pool by searching addresses of miner pools on Google. The one near to your location will be good. For ex:

This will restart the container and your miner will be registered to the mining pool and start getting the jobs. All the rewards will be sent to your monero wallet after your device meets the threshold of the miner pool.

Now as I mentioned above this may not be profitable at all but it's definitely fun and let’s look at some totally unrelated things that we're gonna learn here.

Container Learnings

Here is the project’s Dockerfile.template. If you take a closer look at it, you’ll see there are two different base images being used. One of them is a build image and other one is the run image

Why we do that is simply because we don't want our run image to be bloated with extra packages which were needed to build a source code. So we conduct the build in a separate image and copy the artifacts or binaries in the run image using

COPY --from=build /usr/src/app/xmrig/build/xmrig /usr/local/bin

This way our main image is minimal and less bloated. All of the balena base images are available as build and run. The build image has additional packages such as gcc, build-essential which are needed to build from source whereas the run image is minimal. You can read more about it here.

So this was the container lesson from this blog. Now I hope you do use this project to dip into the world where IoT edge computing meets web3 and as always if you have any suggestions, feedbacks or questions, please write on balenaForums or any social media channel. I'll be back with another interesting project and lesson soon 🍻

Attribution

XMRig project

===