DEV Community: ZyVOP

Brick: The LLM Router That Skips the Cascade and Still Cuts Your Bill

ZyVOP — Wed, 08 Jul 2026 05:53:41 +0000

Most teams solve "which model should answer this prompt" one of two ways.

Pick one strong model and pay frontier prices for every request, including the ones a 9B open-weight model could handle blind. Or build a cascade: try the cheap model first, escalate on a low-confidence signal, and pay for every miss in tokens and latency.

Brick, from Italian inference provider Regolo.ai, is a third option. It reads a prompt once, scores it against a pool of models across six capability dimensions, and picks a single backend in one shot: no retries, no escalation ladder. Just model: "brick" dropped into an OpenAI-compatible call.

It's a young project with a real, published paper behind it: single digits on GitHub stars as of this writing, one CLI release tag, and a couple of distribution channels still marked "pending" in its own README. Here's what's actually going on under the hood, plus a benchmark nuance worth understanding before you cite the headline number.

What Brick Actually Routes To

Brick sits in front of a pool of models you define, then dispatches each incoming query to whichever one is the cheapest that can still get the answer right. The project frames three situations where that's worth doing:

You already run several models and want each query landing on the right one instead of a manual pick.
You're paying for Claude Code or Codex at frontier rates and want easy turns to land on a cheaper model automatically.
You want one endpoint in front of a mixed pool of OpenAI, GLM, DeepSeek, Kimi, and Qwen models instead of wiring each one up separately.

The default shipped config points every backend at Regolo's own API, but the pool is swappable. Anything that speaks the OpenAI chat-completions format can join.

How the Routing Decision Actually Gets Made

This is the part worth reading the source for, because the README summary undersells how mechanical it is. Every text query goes through two classifiers before a single token gets forwarded anywhere.

Capability vector. A ModernBERT classifier scores the query across six dimensions: coding, creative synthesis, instruction following, math reasoning, planning/agentic behavior, and world knowledge. The output is a soft probability distribution over those six, not a hard label.

Complexity score. A separate Qwen3.5-0.8B model with a LoRA adapter buckets the query into easy, medium, or hard. In the shipped config this runs as a remote call against Regolo's brick-complexity-pro endpoint rather than a local model.

The objective. Every model in the pool also has a fixed six-dimensional skill vector plus a cost weight. Brick computes J_m = D_m + β·a_m for each candidate, where D_m is the distance between the query's capability vector and that model's skill vector, and a_m is a normalized cost penalty.

It picks the model with the smallest J_m. That's the whole decision: one geometric comparison, argmin over the pool.

A separate r knob, ranging from -1 to 1, slides the whole pool between these two poles. Toward -1 favors the cheapest capable model; toward 1 favors the strongest one.

The paper's routing math behind that knob has more moving parts than a single weight, with separate calibration branches for the cost and quality directions. At the API level, though, it's just one number you set at deploy time. The router's own config file, pulled straight from the repo, shows what the pool actually looks like:

models:
  - model: "qwen3.5-9b"
    skill_vector: [0.714788, 0.511538, 0.810109, 0.912146, 0.577072, 0.179876]
    use_reasoning: false
    cost_weight: 0.10
  - model: "deepseek-v4-flash"
    skill_vector: [0.820939, 0.657845, 0.863112, 0.934963, 0.62055, 0.488518]
    use_reasoning: false
    cost_weight: 0.40
  - model: "kimi2.6"
    skill_vector: [0.904272, 0.751595, 0.87018, 0.943892, 0.641863, 0.344074]
    use_reasoning: true
    reasoning_effort: "medium"
    cost_weight: 0.60

Two more blocks sit on top of the math for manual overrides, both documented in apps/router/README.md. keyword_rules in override mode hard-forces specific queries to a named model regardless of what the classifiers say.

The shipped default sends anything containing "debug," "refactor," or "write a function" straight to kimi2.6. A bias mode nudges one capability dimension up without forcing a decision, useful for tilting language-specific queries toward the coding axis.

Multimodal input skips the whole pipeline. Images and audio get preprocessed through OCR or Whisper-compatible speech-to-text first, then either routed as extracted text or forwarded straight to a vision model.

Claude Code and Codex Get a Dedicated Integration

This is where Brick is clearly aiming most of its early adoption. Running brick claude on wires an ANTHROPIC_BASE_URL override into ~/.claude/settings.json and starts a local router.

A new brick-claude option shows up in Claude Code's /model picker, sitting next to the built-in opus/sonnet/haiku aliases rather than replacing them. Five modes control the cost/quality trade-off, and they map directly onto Claude Code's existing thinking-effort slider:

Effort slider	Brick mode	Behavior
low	eco	always haiku
medium	lite	graduated tier, cheaper end
high	mid (default)	graduated tier, balanced
xhigh	pro	graduated tier, stronger end
max	max	always opus

The README confirms the two end tiers precisely: eco always picks haiku, max always picks opus. It doesn't spell out the exact easy/medium/hard-to-model mapping for the three tiers in between, only that they sit on a cost/quality gradient.

Selecting a native model name in the picker bypasses Brick entirely and forwards the request unchanged. That gives you an easy escape hatch if the router ever picks wrong.

A brick claude status command opens a live dashboard showing routed-by-model counts, per-model reasoning-effort distribution, the classifier's easy/medium/hard mix, and an estimated savings percentage against an all-opus baseline.

Codex support exists too, gated as beta, sharing the same five modes and status view. The two share host port 8000, though, so only one can serve at a time.

The Numbers, and What They're Actually Measuring

The project has a real published paper behind it: Brick: Spatial Capability Routing for the Mixture-of-Models (MoM) Paradigm, by Francesco Massa (Regolo.ai) and Marco Cristofanilli (Seeweb, where Regolo.ai is built), posted to arXiv in June 2026.

It reports results on a 5,504-query benchmark called Dataset A, spanning six capability dimensions. Each query is graded by protocol-specific checkers, an LLM judge, or both depending on the task type.

Setting	Accuracy	Cost per call	Latency
Always Qwen3.5-9b	63.17%	$0.0014 (1.0×)	8.1s*
Always DeepSeek-v4-flash	73.69%	$0.0029 (2.1×)	14.7s*
Always Kimi2.6	75.02%	$0.0307 (22.15×)	51.2s
Brick, min-cost profile (r=-1)	63.17%	$0.0014 (1.0×)	9.4s*
Brick, neutral profile (r=0)	74.11%	≈$0.0065 (4.7×)	—
Brick, max-quality profile (r=1)	76.98%	unconfirmed†	22.8s
Oracle bound (3-model pool)	83.25%	n/a	n/a

*Accuracy and per-call dollar cost are from the paper's own cost audit, priced off OpenRouter listings at evaluation time.

Latency figures marked with an asterisk are from the project's README summary table rather than the paper itself. The paper's own headline latency claim: median end-to-end latency drops from 51.2s (always-Kimi) to 22.8s (Brick at max-quality).

†The max-quality profile's cost multiple sits further down the same results table in the paper, past the section I could extract. Treat that cell as unverified rather than a claim either way.

Note also that the project's README reports its own, different, more rounded cost multiples for these same rows (1.0× / 4.0× / 6.0× / 1.5×). This table uses the paper's precise per-call dollar figures instead, since they're traceable to an actual pricing snapshot rather than a rounded illustration.

Against external baselines: Brick's low-cost profile beats FrugalGPT's 69.42% by 2.2 points at comparable cost, and its neutral profile beats Cascade Routing's 73.40% by 0.71 points.

That's the deployment-facing number: did the model Brick picked actually answer correctly. The paper is explicit that this isn't the only question worth asking.

It also defines a second metric, route-exact accuracy: does the router's pick match the literal cheapest model that could have solved the query. Queries none of the three models can solve still count against every router here, oracle included.

That makes it a stricter, cost-efficiency-facing bar than the headline number. Credit where it's due: the paper reports both metrics side by side in its main results table, not buried somewhere else.

On that stricter metric, a trivial "always pick the cheapest model" baseline scores 63.17% by definition, since the cheapest model is the correct pick on 63% of the queries. The interesting comparison is the other routers against that same bar.

Always-Kimi and RouteLLM both dispatch nearly everything to Kimi, and score only 21.28% and 21.31% route-exact accuracy despite matching Kimi's headline number. That's the paper's own evidence that neither is doing real query-level judgment.

FrugalGPT and Cascade Routing land at 31.03% and 28.96%. Brick clears 40.35% at a neutral setting and 63.17% at its cheapest profile, beating every external baseline on the metric that's supposed to be hardest on it.

One loose end worth flagging for anyone who goes digging in the repo themselves: a separate, Italian-language working file in packages/evals/baselines/RESULTS.md reports a route-exact figure for Brick of 46.37%. That number doesn't line up exactly with any of the r-settings in the paper's published table (63.17% / 48.66% / 40.35% at min/low/neutral).

That file reads like an earlier or differently-configured pipeline run than what shipped in the final paper, not a contradiction of it. If you're citing Brick's numbers for something that matters, pull them from the arXiv table rather than that file.

Trying It

The fastest path is the CLI, which self-hosts the router and wires it into Claude Code in two commands. It needs Node 18+ and Docker:

git clone https://github.com/regolo-ai/brick-SR1.git
cd brick-SR1/apps/cli && npm install && npm run build && npm link

brick claude on

For a raw OpenAI-compatible gateway without the CLI, the project's Docker image is documented but, as of this writing, still pending its first push to GHCR under the v2.1.0 tag. Once published, the documented usage looks like this:

docker run --rm -p 18000:18000 \
  -e REGOLO_API_KEY=$REGOLO_API_KEY \
  ghcr.io/regolo-ai/brick:latest

curl http://localhost:18000/v1/chat/completions \
  -H "Authorization: Bearer $REGOLO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"brick","messages":[{"role":"user","content":"Prove that sqrt(2) is irrational"}]}'

The response carries an x-selected-model header telling you which backend actually answered. A math-proof prompt should land on a reasoning-capable model; a bare "hello" should land on the cheapest one in the pool.

I'm quoting these commands straight from the project's own quickstart rather than claiming to have run them myself. With the Docker image and npm package both still pending publication, and no Regolo API key available in a sandboxed environment, there was nothing live to hit.

Where This Project Actually Sits Right Now

A few things worth knowing before you build on this. The router itself descends from vllm-project/spatial-router, an Apache-2.0 project.

Regolo's team added the six-dimension capability classifier, the complexity-score integration, the skill-distance objective, multimodal preprocessing, and the Claude Code passthrough on top. That lineage is disclosed plainly in the repo's NOTICE file, which is exactly how attribution should work on a fork.

As of this writing the repo has zero forks and a GitHub star count still in the single digits. That's normal for a project a few weeks old and not itself a signal of quality either way.

The codebase is a genuine multi-language build. Go and Rust make up the router (Rust handles ML embeddings via candle and classical ML via Linfa, compiled to shared libraries and linked through CGO), Python runs the training and eval pipelines, and TypeScript powers the CLI.

License is Apache-2.0 throughout. Whether it earns a permanent spot in your stack comes down to which problem you actually have.

If you're routing across a real multi-model pool, the paper's numbers are worth your own re-run before you trust them at production scale. If you just want Claude Code to stop burning Opus on "fix this typo" requests, the Claude-specific integration is the more immediately useful half of the project — and it works whether or not you buy the Dataset A accuracy claims at all.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Python Scraping at Scale: Distributed Crawling Across Multiple Machines (2026)

ZyVOP — Tue, 07 Jul 2026 06:04:28 +0000

Introduction: When One Machine Is No Longer Enough

Most Python developers write simple scrapers—requests, BeautifulSoup, a loop, CSV writer—just to get data once or twice. When scaling or running them over months, the real challenge is building a robust, scheduled system, not the parsing logic. The extraction code is tiny; the critical components are the queue, cache, storage, block detection, recovery loop, and exporters that keep the scraper alive against real‑world internet conditions.

At small scale, a single async Python process can handle thousands of pages per hour. But when you need to crawl millions of pages per day — competitor catalogues, job markets, news archives, e-commerce databases — a single machine hits hard limits: one IP address, one CPU, one point of failure.

Distributed scraping involves spreading tasks across multiple machines to increase speed and volume. Throttling and introducing random delays between requests can help to prevent IP bans, while rotating proxies help distribute requests and avoid detection. Managing sessions and leveraging parallel processing can further enhance efficiency.

This guide shows you exactly how to build a distributed scraping system that runs across multiple machines — using scrapy-redis for shared request queues, Docker for containerisation, and a production proxy management layer that survives real-world conditions.

The Architecture: One Queue, Many Workers

The core insight behind distributed scraping is simple: replace Scrapy's in-memory request queue with a shared Redis queue that every worker machine can read from.

┌─────────────────────────────────────────────────────────────┐
│                    DISTRIBUTED SCRAPER                      │
│                                                             │
│  Master                                                     │
│  ┌──────────┐    Seeds requests    ┌─────────┐             │
│  │  Spider  │──────────────────────▶│  Redis  │             │
│  │ (seed)   │                      │  Queue  │             │
│  └──────────┘                      └────┬────┘             │
│                                         │                   │
│  Workers (any number of machines)       │                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                  │
│  │ Worker 1 │  │ Worker 2 │  │ Worker N │  ← pull jobs     │
│  │ (spider) │  │ (spider) │  │ (spider) │                  │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘                  │
│       │              │              │                        │
│       └──────────────┴──────────────┘                       │
│                          │                                   │
│                    ┌─────▼──────┐                           │
│                    │  MongoDB   │  ← all workers write here │
│                    │ (results)  │                           │
│                    └────────────┘                           │
└─────────────────────────────────────────────────────────────┘

scrapy-redis lets multiple spider instances across machines pull from the same request queue with deduplication and job scheduling. This is the standard approach for large-scale web scraping that Python teams use in production.

Part 1: Setting Up scrapy-redis

pip install scrapy scrapy-redis redis pymongo

The distributed spider

# spiders/distributed_product_spider.py
import scrapy
from scrapy_redis.spiders import RedisSpider
from urllib.parse import urljoin
from datetime import datetime, timezone

class DistributedProductSpider(RedisSpider):
    """
    A Scrapy spider that pulls start URLs from a Redis list
    instead of a hardcoded start_urls list.

    To start crawling, push seed URLs to Redis:
        redis-cli lpush products:start_urls "https://example-store.com/products/"

    Any number of workers running this spider will cooperatively
    process the shared queue — each URL is processed exactly once.
    """

    name         = "distributed_products"
    redis_key    = "products:start_urls"   # Redis list to pop URLs from

    # How many URLs to pop from Redis at once per worker
    redis_batch_size = 16

    # Spider-level settings — override settings.py per spider
    custom_settings = {
        "CONCURRENT_REQUESTS":            32,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
        "DOWNLOAD_DELAY":                 0.5,
        "RANDOMIZE_DOWNLOAD_DELAY":       True,
        "AUTOTHROTTLE_ENABLED":           True,
        "AUTOTHROTTLE_TARGET_CONCURRENCY": 16.0,
        "AUTOTHROTTLE_MAX_DELAY":         5.0,
        "RETRY_TIMES":                    3,
        "RETRY_HTTP_CODES":              [429, 500, 502, 503, 504],
    }

    def parse(self, response):
        """
        Parse a product listing page.
        Yields product detail requests AND discovers pagination links.
        """
        # Follow product links to detail pages
        for href in response.css("a.product-link::attr(href)").getall():
            yield response.follow(href, callback=self.parse_product)

        # Auto-discover pagination — push next page back to Redis queue
        next_page = response.css("a[rel='next']::attr(href)").get()
        if next_page:
            # Use follow() to handle relative URLs
            yield response.follow(next_page, callback=self.parse)

    def parse_product(self, response):
        """Extract product data from a detail page."""
        yield {
            "url":        response.url,
            "title":      response.css("h1::text").get("").strip(),
            "price":      response.css(".price::text").get("").strip(),
            "sku":        response.css("[data-sku]::attr(data-sku)").get(),
            "in_stock":   bool(response.css(".in-stock")),
            "description":response.css(".product-description::text").get("").strip()[:500],
            "scraped_at": datetime.now(timezone.utc).isoformat(),
            "worker_id":  self.settings.get("WORKER_ID", "unknown"),
        }

settings.py for distributed mode

# settings.py

BOT_NAME = "distributed_scraper"
SPIDER_MODULES = ["spiders"]

# ── scrapy-redis settings ─────────────────────────────────────
SCHEDULER            = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS     = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL            = "redis://redis-host:6379"

# Keep crawl state across restarts — resumable crawls
SCHEDULER_PERSIST    = True

# How long to wait for new URLs before worker shuts down
SCHEDULER_IDLE_BEFORE_CLOSE = 30

# ── MongoDB pipeline ──────────────────────────────────────────
ITEM_PIPELINES = {
    "pipelines.MongoPipeline":        100,
    "pipelines.DuplicateFilterPipeline": 50,
}
MONGO_URI      = "mongodb://mongo-host:27017/"
MONGO_DATABASE = "distributed_scrape"

# ── Concurrency ───────────────────────────────────────────────
CONCURRENT_REQUESTS              = 32
CONCURRENT_REQUESTS_PER_DOMAIN   = 8
DOWNLOAD_DELAY                   = 0.5
RANDOMIZE_DOWNLOAD_DELAY         = True

# ── Retry ─────────────────────────────────────────────────────
RETRY_ENABLED    = True
RETRY_TIMES      = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524]

# ── Logging ───────────────────────────────────────────────────
LOG_LEVEL = "INFO"

# ── Downloader middlewares ────────────────────────────────────
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
    "middlewares.RotatingProxyMiddleware": 350,
    "middlewares.UserAgentRotationMiddleware": 400,
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
}

# ── User agent pool ───────────────────────────────────────────
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/119.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) Chrome/118.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
]

Part 2: Production Middlewares

# middlewares.py
import random
import logging
from scrapy import signals
from scrapy.exceptions import NotConfigured

logger = logging.getLogger(__name__)

class UserAgentRotationMiddleware:
    """Rotate User-Agent on every request."""

    def __init__(self, user_agents):
        self.user_agents = user_agents

    @classmethod
    def from_crawler(cls, crawler):
        agents = crawler.settings.getlist("USER_AGENTS")
        if not agents:
            raise NotConfigured("USER_AGENTS not set")
        return cls(agents)

    def process_request(self, request, spider):
        request.headers["User-Agent"] = random.choice(self.user_agents)

class RotatingProxyMiddleware:
    """
    Rotate through a proxy pool.
    Tracks failures per proxy and removes bad proxies from the pool.
    """

    def __init__(self, proxies, max_failures=5):
        self.proxies      = list(proxies)
        self.failures     = {}
        self.max_failures = max_failures

    @classmethod
    def from_crawler(cls, crawler):
        proxies = crawler.settings.getlist("PROXY_LIST", [])
        if not proxies:
            raise NotConfigured("PROXY_LIST is empty")
        return cls(proxies)

    def process_request(self, request, spider):
        if not self.proxies:
            return  # No proxies left — run without
        proxy = random.choice(self.proxies)
        request.meta["proxy"] = proxy

    def process_response(self, request, response, spider):
        if response.status in (403, 407, 429):
            proxy = request.meta.get("proxy")
            self._mark_failure(proxy)
        return response

    def process_exception(self, request, exception, spider):
        proxy = request.meta.get("proxy")
        self._mark_failure(proxy)

    def _mark_failure(self, proxy):
        if not proxy:
            return
        self.failures[proxy] = self.failures.get(proxy, 0) + 1
        if self.failures[proxy] >= self.max_failures:
            if proxy in self.proxies:
                self.proxies.remove(proxy)
                logger.warning(f"Removed bad proxy: {proxy} ({self.max_failures} failures)")

class BlockDetectionMiddleware:
    """
    Detect common block patterns and trigger retries with a different proxy.
    """

    BLOCK_PATTERNS = [
        "access denied", "captcha", "blocked", "forbidden",
        "unusual traffic", "robot", "automated queries",
        "cf-browser-verification", "ddos-guard",
    ]

    def process_response(self, request, response, spider):
        body_lower = response.text[:2000].lower()

        is_blocked = (
            response.status in (403, 429, 503) or
            any(p in body_lower for p in self.BLOCK_PATTERNS) or
            len(response.text) < 300
        )

        if is_blocked:
            logger.warning(f"Block detected on {request.url} — retrying")
            request.meta["proxy"]    = None  # Force new proxy on retry
            request.dont_filter      = True
            return request           # Re-schedule the request

        return response

Part 3: MongoDB Pipeline with Bulk Writes

# pipelines.py
import logging
from datetime import datetime, timezone
from pymongo import MongoClient, UpdateOne
from pymongo.errors import BulkWriteError
from itemadapter import ItemAdapter

logger = logging.getLogger(__name__)

class DuplicateFilterPipeline:
    """Track seen URLs in memory to drop duplicates before DB write."""

    def open_spider(self, spider):
        self.seen_urls = set()

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        url = adapter.get("url", "")
        if url in self.seen_urls:
            from scrapy.exceptions import DropItem
            raise DropItem(f"Duplicate URL: {url}")
        self.seen_urls.add(url)
        return item

class MongoPipeline:
    """
    Write scraped items to MongoDB using bulk operations.
    Upserts on URL — safe to re-run without creating duplicates.
    """

    BULK_SIZE  = 200   # Flush every 200 items
    COLLECTION = "products"

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db  = mongo_db
        self._buffer   = []

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get("MONGO_URI", "mongodb://localhost:27017/"),
            mongo_db =crawler.settings.get("MONGO_DATABASE", "scrapy_data"),
        )

    def open_spider(self, spider):
        self.client     = MongoClient(self.mongo_uri)
        self.col        = self.client[self.mongo_db][self.COLLECTION]
        self.col.create_index("url", unique=True)
        self.items_written = 0
        logger.info(f"MongoDB connected: {self.mongo_db}.{self.COLLECTION}")

    def close_spider(self, spider):
        if self._buffer:
            self._flush()
        self.client.close()
        logger.info(f"MongoDB closed. Total items written: {self.items_written}")

    def process_item(self, item, spider):
        self._buffer.append(dict(item))
        if len(self._buffer) >= self.BULK_SIZE:
            self._flush()
        return item

    def _flush(self):
        ops = [
            UpdateOne({"url": doc["url"]}, {"$set": doc}, upsert=True)
            for doc in self._buffer
        ]
        try:
            result = self.col.bulk_write(ops, ordered=False)
            count  = result.upserted_count + result.modified_count
            self.items_written += count
            logger.info(f"Flushed {len(self._buffer)} items → MongoDB")
        except BulkWriteError as e:
            logger.error(f"Bulk write error: {e.details.get('writeErrors', [])[:2]}")
        finally:
            self._buffer.clear()

Part 4: Docker Compose — Multi-Worker Setup

# docker-compose.yml
version: "3.9"

x-worker-base: &worker-base
  build: .
  volumes:
    - .:/app
  environment:
    - REDIS_URL=redis://redis:6379
    - MONGO_URI=mongodb://mongo:27017/
    - PROXY_LIST=${PROXY_LIST}
  depends_on:
    - redis
    - mongo
  restart: unless-stopped

services:

  # ── Infrastructure ──────────────────────────────────────────
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
    volumes: ["redis_data:/data"]

  mongo:
    image: mongo:7
    ports: ["27017:27017"]
    volumes: ["mongo_data:/data/db"]

  # ── Scrapy Workers ──────────────────────────────────────────
  # Run as many of these as you have cores / IPs
  worker-1:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-1
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-1

  worker-2:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-2
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-2

  worker-3:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-3
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-3

  # ── Seed Service — pushes start URLs into Redis ─────────────
  seeder:
    <<: *worker-base
    command: python seeder.py
    restart: "no"   # Run once then exit

volumes:
  redis_data:
  mongo_data:

Part 5: The Seeder — Feeding URLs Into the Queue

# seeder.py
import redis
import time
import sys
from urllib.parse import urlencode

REDIS_URL   = "redis://localhost:6379"
REDIS_KEY   = "products:start_urls"

def seed_from_list(urls: list[str], batch_size: int = 500):
    """Push a list of start URLs into the Redis queue."""
    r = redis.from_url(REDIS_URL)

    # Clear existing queue if resuming fresh
    existing = r.llen(REDIS_KEY)
    if existing > 0:
        print(f"Queue already has {existing:,} URLs. Adding to it.")

    pushed = 0
    for i in range(0, len(urls), batch_size):
        batch = urls[i:i + batch_size]
        r.rpush(REDIS_KEY, *batch)
        pushed += len(batch)
        print(f"Seeded {pushed:,}/{len(urls):,} URLs")

    print(f"\nDone. Redis queue '{REDIS_KEY}' has {r.llen(REDIS_KEY):,} URLs.")

def seed_paginated_site(
    base_url: str,
    start_page: int = 1,
    end_page: int = 500,
    page_param: str = "page"
):
    """Generate paginated URLs and push to Redis."""
    urls = []
    for page in range(start_page, end_page + 1):
        params = {page_param: page}
        urls.append(f"{base_url}?{urlencode(params)}")

    seed_from_list(urls)

def monitor_queue():
    """Monitor queue depth and worker progress in real time."""
    r = redis.from_url(REDIS_URL)
    print("Monitoring queue depth (Ctrl+C to stop)...")
    try:
        while True:
            depth   = r.llen(REDIS_KEY)
            seen    = r.scard(f"{REDIS_KEY}:dupefilter") or 0
            print(f"  Queue: {depth:,} pending | Seen: {seen:,} processed", end="\r")
            time.sleep(2)
    except KeyboardInterrupt:
        print("\nMonitor stopped.")

if __name__ == "__main__":
    mode = sys.argv[1] if len(sys.argv) > 1 else "seed"

    if mode == "seed":
        seed_paginated_site(
            base_url  = "https://example-store.com/products",
            start_page = 1,
            end_page   = 1000,
        )
    elif mode == "monitor":
        monitor_queue()
    elif mode == "clear":
        r = redis.from_url(REDIS_URL)
        r.delete(REDIS_KEY)
        print(f"Queue '{REDIS_KEY}' cleared.")

Part 6: Scaling to Kubernetes

For truly large-scale crawling across dozens of machines, Kubernetes is the standard deployment target. The key insight: each Scrapy worker is a stateless pod that reads from the shared Redis queue.

# k8s/scrapy-worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scrapy-workers
  labels:
    app: scrapy-worker
spec:
  replicas: 10   # Start with 10 workers; scale up/down with kubectl
  selector:
    matchLabels:
      app: scrapy-worker
  template:
    metadata:
      labels:
        app: scrapy-worker
    spec:
      containers:
        - name: scrapy-worker
          image: your-registry/scrapy-worker:latest
          command:
            - scrapy
            - crawl
            - distributed_products
          env:
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: scraper-secrets
                  key: redis-url
            - name: MONGO_URI
              valueFrom:
                secretKeyRef:
                  name: scraper-secrets
                  key: mongo-uri
            - name: WORKER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name  # Pod name as worker ID
          resources:
            requests:
              memory: "256Mi"
              cpu:    "250m"
            limits:
              memory: "512Mi"
              cpu:    "500m"

Scale workers up or down instantly:

# Scale to 20 workers
kubectl scale deployment scrapy-workers --replicas=20

# Check worker status
kubectl get pods -l app=scrapy-worker

# View logs from all workers
kubectl logs -l app=scrapy-worker --tail=50

# Auto-scale based on Redis queue depth (requires custom metrics)
kubectl autoscale deployment scrapy-workers --min=2 --max=50

Part 7: Proxy Management at Scale

For serious scraping in 2026, residential proxies are almost always the safer option. Proxy quality matters far more than proxy quantity. A smaller pool of clean residential IPs usually performs much better than massive low-quality networks.

Here's a production proxy manager with health checking:

# proxy_manager.py
import asyncio
import httpx
import random
import time
import json
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class ProxyHealth:
    url:              str
    success_count:    int   = 0
    failure_count:    int   = 0
    last_used:        float = 0.0
    last_success:     float = 0.0
    avg_response_ms:  float = 0.0
    is_banned:        bool  = False

    @property
    def success_rate(self) -> float:
        total = self.success_count + self.failure_count
        return self.success_count / total if total > 0 else 0.0

    @property
    def score(self) -> float:
        """Composite score: higher = better proxy to use."""
        if self.is_banned:
            return 0.0
        recency_bonus = max(0, 1 - (time.time() - self.last_success) / 3600)
        speed_score   = max(0, 1 - self.avg_response_ms / 5000)
        return self.success_rate * 0.6 + recency_bonus * 0.2 + speed_score * 0.2

class ProxyPool:
    """
    Intelligent proxy pool with health tracking and weighted selection.
    Workers register success/failure, pool learns which proxies are best.
    """

    BAN_THRESHOLD = 0.2   # Mark as banned if success rate drops below 20%

    def __init__(self, proxy_urls: list[str]):
        self.proxies = {url: ProxyHealth(url=url) for url in proxy_urls}

    def get_proxy(self, strategy: str = "weighted") -> Optional[str]:
        """Select a proxy using the specified strategy."""
        available = [
            p for p in self.proxies.values()
            if not p.is_banned
        ]

        if not available:
            return None

        if strategy == "random":
            return random.choice(available).url

        elif strategy == "weighted":
            # Weight by score — best proxies get used more often
            scores = [max(p.score, 0.01) for p in available]
            total  = sum(scores)
            weights = [s / total for s in scores]
            return random.choices(available, weights=weights)[0].url

        elif strategy == "round_robin":
            # Sort by last_used timestamp — use least recently used
            available.sort(key=lambda p: p.last_used)
            return available[0].url

        return available[0].url

    def report_success(self, proxy_url: str, response_ms: float):
        if proxy_url in self.proxies:
            p = self.proxies[proxy_url]
            p.success_count  += 1
            p.last_used       = time.time()
            p.last_success    = time.time()
            # Rolling average response time
            p.avg_response_ms = (p.avg_response_ms * 0.8 + response_ms * 0.2)

    def report_failure(self, proxy_url: str):
        if proxy_url in self.proxies:
            p = self.proxies[proxy_url]
            p.failure_count += 1
            p.last_used      = time.time()
            # Auto-ban proxies with very low success rate
            if p.failure_count > 10 and p.success_rate < self.BAN_THRESHOLD:
                p.is_banned = True
                print(f"  Proxy banned (success rate {p.success_rate:.0%}): {proxy_url}")

    def get_stats(self) -> dict:
        active  = [p for p in self.proxies.values() if not p.is_banned]
        banned  = [p for p in self.proxies.values() if p.is_banned]
        avg_sr  = sum(p.success_rate for p in active) / len(active) if active else 0

        return {
            "total":    len(self.proxies),
            "active":   len(active),
            "banned":   len(banned),
            "avg_success_rate": f"{avg_sr:.1%}",
            "best_proxy": max(active, key=lambda p: p.score).url if active else None,
        }

async def health_check_proxies(pool: ProxyPool, test_url: str = "https://httpbin.org/ip"):
    """
    Periodically check all proxies and un-ban those that have recovered.
    Run this as a background task.
    """
    async with httpx.AsyncClient(timeout=10) as client:
        for url, proxy in list(pool.proxies.items()):
            if not proxy.is_banned:
                continue
            try:
                start = time.time()
                r = await client.get(test_url, proxies={"https": url})
                if r.status_code == 200:
                    elapsed_ms    = (time.time() - start) * 1000
                    proxy.is_banned = False
                    pool.report_success(url, elapsed_ms)
                    print(f"  Proxy recovered: {url}")
            except Exception:
                pass   # Still banned

    stats = pool.get_stats()
    print(f"Proxy pool: {stats['active']} active, {stats['banned']} banned, "
          f"avg success rate: {stats['avg_success_rate']}")

Part 8: Monitoring Your Distributed Crawl

# monitor.py — real-time crawl progress dashboard
import redis
import time
import json
from pymongo import MongoClient
from datetime import datetime

def live_dashboard(redis_url: str, mongo_uri: str, refresh_seconds: int = 5):
    """Print a live crawl progress dashboard to the terminal."""
    r   = redis.from_url(redis_url)
    db  = MongoClient(mongo_uri)["distributed_scrape"]

    start_time   = time.time()
    prev_count   = 0

    try:
        while True:
            # Queue stats
            queue_depth  = r.llen("products:start_urls")
            seen_count   = r.scard("products:start_urls:dupefilter") or 0

            # DB stats
            items_stored = db["products"].count_documents({})
            items_delta  = items_stored - prev_count
            rate_per_min = items_delta * (60 / refresh_seconds)
            prev_count   = items_stored

            # Worker stats (scrapy-redis stores worker heartbeats)
            workers = r.smembers("scrapy:workers") or set()

            # Elapsed
            elapsed = time.time() - start_time
            h, m    = divmod(int(elapsed), 3600)
            m, s    = divmod(m, 60)

            print(f"\033[2J\033[H")   # Clear screen
            print(f"{'═'*55}")
            print(f"  DISTRIBUTED SCRAPE MONITOR — {datetime.now().strftime('%H:%M:%S')}")
            print(f"{'═'*55}")
            print(f"  Runtime:       {h:02d}h {m:02d}m {s:02d}s")
            print(f"  Active workers:{len(workers)}")
            print(f"{'─'*55}")
            print(f"  Queue depth:   {queue_depth:>10,}  (URLs remaining)")
            print(f"  URLs seen:     {seen_count:>10,}  (deduplicated total)")
            print(f"  Items stored:  {items_stored:>10,}  (in MongoDB)")
            print(f"  Rate:          {rate_per_min:>10.0f}  items/minute")
            print(f"{'─'*55}")

            if rate_per_min > 0 and queue_depth > 0:
                eta_mins = queue_depth / rate_per_min
                h2, m2   = divmod(int(eta_mins * 60), 3600)
                m2, s2   = divmod(m2, 60)
                print(f"  ETA:           {h2:02d}h {m2:02d}m  (estimated)")

            print(f"{'═'*55}")
            time.sleep(refresh_seconds)

    except KeyboardInterrupt:
        print("\nMonitor stopped.")

if __name__ == "__main__":
    live_dashboard(
        redis_url="redis://localhost:6379",
        mongo_uri="mongodb://localhost:27017/",
    )

Part 9: Resumable Crawls

One of the biggest advantages of scrapy-redis is that crawls are inherently resumable. If a worker crashes or you need to add more workers mid-crawl, simply restart:

# Start a fresh crawl
python seeder.py seed

# Launch workers (they'll pick up from where they left off if SCHEDULER_PERSIST=True)
docker-compose up --scale worker=5

# Pause all workers (Ctrl+C in docker-compose)
# Resume later — queue state is preserved in Redis
docker-compose up --scale worker=10   # Can add more workers

To completely reset a crawl:

# Clear the queue and deduplication filter
redis-cli del products:start_urls
redis-cli del products:start_urls:dupefilter
python seeder.py seed   # Re-seed with fresh URLs

Performance Numbers: What to Expect

One machine first, then scale out. Tune Scrapy performance optimization settings before throwing hardware at the problem — raise CONCURRENT_REQUESTS, turn on AUTOTHROTTLE, and enable HTTP caching. If one server can't handle scraping millions of pages, set up a shared queue for distributed crawling across multiple workers.

Real-world throughput benchmarks:

Setup	Pages/hour	Cost estimate
1 worker, no proxy	~8,000	Free
1 worker + 10 proxies	~25,000	~$5/day
5 workers + 50 proxies	~120,000	~$20/day
20 workers + 200 proxies	~500,000	~$80/day
100 workers (Kubernetes)	~2,500,000	~$350/day

Summary

Component	Tool	Role
Spider	Scrapy + scrapy-redis	Crawl logic + distributed request handling
Queue	Redis	Shared URL queue with built-in deduplication
Worker deployment	Docker Compose / Kubernetes	Horizontal scale, stateless workers
Proxy management	Custom ProxyPool	Health-tracked, weighted proxy selection
Storage	MongoDB (bulk upserts)	Centralised, deduplicated results
Monitoring	Custom dashboard + Flower	Real-time progress and worker health
Resumability	SCHEDULER_PERSIST=True	Crash-safe, restartable crawls

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

AI Hype Meets Reality: Security Risks, Thin ROI, and the Rise of Skepticism | The AI Daily Roundup

ZyVOP — Mon, 06 Jul 2026 03:00:08 +0000

Trend Overview: From Hype to Hard Evidence

Across today’s headlines a single narrative emerges – the AI boom is colliding with hard‑won lessons about security, economics, and user agency. Companies are pulling back on unvetted models, researchers are exposing the limits of productivity gains, and regulators and users alike are demanding concrete safeguards.

Why the Shift Matters

When AI tools promise “instant expertise” or “autonomous vulnerability discovery,” the cost of failure is no longer abstract. A data breach, a wasted budget, or a legal liability can cripple a firm faster than a missed hype cycle. Simultaneously, the promised “AI‑powered productivity” is proving to be a marginal gain that evaporates before it reaches a paycheck. The convergence of security scares and thin ROI forces senior leaders to re‑evaluate AI adoption strategies, shifting capital toward proven, auditable solutions.

Evidence from the Day’s Stories

Security Backlash

Alibaba bans Claude Code after internal alarms about a potential backdoor. The move signals that even tech giants will enforce “zero‑trust” policies when model provenance is uncertain.
Claude Mythos preview triggers a 3.5× spike in high‑severity CVEs. Anthropic’s own showcase of autonomous vulnerability discovery has unintentionally amplified the attack surface, prompting a wave of disclosures from Microsoft, Google, Apple, and AWS partners.
Claude’s “memory” feature under fire. Engineers report zero performance benefit from retaining session transcripts, highlighting a broader pattern: added complexity without measurable security or efficiency gains.

Economic Reality Check

Study finds AI saves only ~3% of work hours, and less than 5% of that time translates into higher pay. The gap between lab‑controlled speedups (15‑55%) and real‑world payroll data underscores a “leaky bucket” problem.
Yann LeCun admits current AI isn’t “smart” and bets on next‑generation systems that can handle real‑world data. His candid assessment validates the productivity findings: today’s models excel at narrow tasks but falter on general, embodied intelligence.

User Control & Transparency

Kagi adds an AI toggle, letting users disable AI‑driven search features. The option reflects growing demand for “opt‑out” mechanisms when AI adds noise rather than value.
Google’s Gemini Code Assist is being retired after less than a year, suggesting that even well‑funded services struggle to sustain adoption when the perceived ROI is low.

Who Wins, Who Loses

Beneficiaries: Security‑focused vendors (vulnerability‑management platforms, zero‑trust providers), audit‑ready AI platforms that expose provenance, and enterprises that adopt a measured, task‑specific AI strategy.

Losers: Companies banking on blanket AI adoption without clear use‑case validation, hype‑driven product launches, and consultants who sell “AI transformation” without quantifiable outcomes.

What Changes Next?

Stricter governance: Expect more corporate bans similar to Alibaba’s, and internal policies that require independent model audits before deployment.
Metrics‑first adoption: Teams will benchmark AI impact against payroll and P&L data before scaling, mirroring the methodology of the Danish study.
Feature pruning: Products that add “bells and whistles” (e.g., session‑transcript memory, generic code assistants) will be trimmed or sunset unless they demonstrate clear ROI.
Regulatory focus on AI‑driven security: As vulnerability spikes become visible, regulators may mandate disclosure of AI‑generated exploits and require companies to certify model safety.

Bottom Line

The AI industry is entering a phase of “skeptical scaling.” Security incidents, modest productivity gains, and user‑driven demand for control are forcing a recalibration. Leaders who embed rigorous measurement, enforce zero‑trust model policies, and prioritize real‑world value will capture the next wave of AI‑enabled advantage.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Passwordless Login With Magic Links in Node.js

ZyVOP — Sun, 05 Jul 2026 03:28:14 +0000

I've seen developers implement magic link auth four different ways and get it wrong three of them. The broken versions all share the same flaw: they use a regular get + del to verify tokens, which means two simultaneous requests with the same token both pass.

User A clicks the link in Gmail. The prefetch scanner in their email client hits it half a second earlier. Server issues two sessions. Neither user knows, but one of them is now authenticated in a context they didn't control.

This post builds it right. By the end you'll have a working Node.js implementation with proper atomic token verification, email enumeration protection, an Ethereal fallback so you can see the full flow without any SMTP configuration, and 19 tests that run without Redis or a real mail server.

Source code: http://github.com/zyvop27-cmyk/zyvop-blogs/tree/main/magic-link-auth

The one decision that matters most

Before touching any code: the difference between a safe magic link implementation and a broken one is a single Redis command.

The broken pattern:

const email = await redis.get(key);
if (email) await redis.del(key);
return email;

Between the get and the del, another request can read the same key. Both get a valid email back, both get sessions issued.

getDel collapses that into one atomic operation:

const email = await redis.getDel(key);
return email ?? null;

Redis executes this as a single command. No window — one request gets the email, any concurrent request gets null.

That's why Redis fits this problem better than a Postgres row. You'd need a transaction and an advisory lock to get the same guarantee from a relational database.

The full verify function adds a length check before the Redis call. A token that's too short or too long gets rejected immediately, no round-trip needed:

// src/lib/token.js
import crypto from "node:crypto";

const TOKEN_PREFIX = "magic:";
const TOKEN_TTL_SECONDS = 15 * 60;
const TOKEN_BYTES = 32;

export async function createToken(redis, email) {
  const token = crypto.randomBytes(TOKEN_BYTES).toString("hex");
  await redis.set(TOKEN_PREFIX + token, email.toLowerCase(), { EX: TOKEN_TTL_SECONDS });
  return token;
}

export async function verifyToken(redis, token) {
  if (!token || typeof token !== "string" || token.length !== TOKEN_BYTES * 2) {
    return null;
  }
  return (await redis.getDel(TOKEN_PREFIX + token)) ?? null;
}

crypto.randomBytes(32) gives 256 bits of entropy — 64 hex characters. Brute-forcing that against a 15-minute window isn't happening. The email gets lowercased on creation so User@Example.COM and user@example.com don't end up as separate keys that never match.

Redis also handles expiry natively via the EX option. No cron job to clean up stale tokens, no WHERE expires_at < NOW() queries — they disappear on their own.

What the user actually sees

The sign-in page is minimal HTML with one real piece of JavaScript: it handles the ?error=link_invalid query param that the server redirects to when a token has expired or already been used.

<!-- public/index.html (abbreviated) -->
<form id="form">
  <input type="email" id="email" placeholder="you@example.com" required>
  <button type="submit">Send sign-in link</button>
</form>
<div id="msg" class="message"></div>

<script>
  const ERRORS = {
    link_invalid: "That link has expired or already been used. Please request a new one.",
    default: "Something went wrong. Please try again."
  };

  // Show error from redirect after a failed verify attempt
  const errKey = new URLSearchParams(location.search).get("error");
  if (errKey) showMsg(ERRORS[errKey] ?? ERRORS.default, "error");

  document.getElementById("form").addEventListener("submit", async (e) => {
    e.preventDefault();
    const email = document.getElementById("email").value.trim();
    const btn = e.target.querySelector("button");

    btn.disabled = true;
    btn.textContent = "Sending…";

    const res = await fetch("/auth/request", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ email })
    });

    const data = await res.json();

    if (res.ok) {
      showMsg("Check your inbox — a sign-in link is on its way.", "success");
      document.getElementById("form").style.display = "none";
    } else {
      showMsg(data.error ?? ERRORS.default, "error");
      btn.disabled = false;
      btn.textContent = "Send sign-in link";
    }
  });
</script>

The ?error=link_invalid flow matters more than it looks. When verification fails — expired token, already used, someone typed a URL wrong — it redirects to /?error=link_invalid instead of returning a JSON 400.

Users arrive at /auth/verify by clicking a link in their email client, not via a fetch call. A JSON error on a blank page is a dead end; a redirect back to the sign-in form with a clear message is something a person can act on.

Sending the email without configuring SMTP

Production email uses SMTP_HOST, SMTP_PORT, SMTP_USER, and SMTP_PASS. Without those, the mailer silently creates an Ethereal test account and logs a preview URL. Ethereal is a real catch-all SMTP service — emails don't deliver anywhere, but you can open the URL and see the full rendered email, including the magic link button, right in your browser.

// src/lib/mailer.js
import nodemailer from "nodemailer";

let transporter = null;

async function getTransporter() {
  if (transporter) return transporter;

  if (process.env.SMTP_HOST) {
    transporter = nodemailer.createTransport({
      host: process.env.SMTP_HOST,
      port: Number(process.env.SMTP_PORT) || 587,
      secure: process.env.SMTP_SECURE === "true",
      auth: { user: process.env.SMTP_USER, pass: process.env.SMTP_PASS }
    });
  } else {
    const testAccount = await nodemailer.createTestAccount();
    transporter = nodemailer.createTransport({
      host: "smtp.ethereal.email",
      port: 587,
      auth: { user: testAccount.user, pass: testAccount.pass }
    });
    console.log(`[mailer] Ethereal test account: ${testAccount.user}`);
  }

  return transporter;
}

export async function sendMagicLink(to, magicUrl) {
  const transport = await getTransporter();

  const info = await transport.sendMail({
    from: process.env.EMAIL_FROM || '"Magic Link Auth" <no-reply@example.com>',
    to,
    subject: "Your sign-in link",
    text: `Sign-in link (expires in 15 minutes, single use):\n\n${magicUrl}`,
    html: `
      <p>Click below to sign in. Expires in <strong>15 minutes</strong>, single use.</p>
      <p style="margin:24px 0">
        <a href="${magicUrl}" style="background:#111;color:#fff;padding:12px 24px;border-radius:6px;text-decoration:none">
          Sign in
        </a>
      </p>
      <p style="color:#999;font-size:12px">Didn't request this? Safe to ignore.</p>
    `
  });

  if (!process.env.SMTP_HOST) {
    console.log(`[mailer] Preview URL: ${nodemailer.getTestMessageUrl(info)}`);
  }
}

The transporter is cached in a module-level variable. nodemailer.createTestAccount() makes a real HTTP call to Ethereal — you don't want that per-request.

For production, Resend and Postmark are the cleaner options over raw SMTP. They handle deliverability, bounce handling, and SPF/DKIM automatically. Hook them in via the SMTP_HOST and friends in .env.

Sessions, cookies, and why SameSite: lax is correct here

After verification, the token is gone and the user needs something that persists across requests. A JWT in an httpOnly cookie is the right shape: stateless (no server-side session table), survives page refreshes, inaccessible to JavaScript running on the page.

// src/lib/session.js
import jwt from "jsonwebtoken";

const COOKIE_NAME = "session";
const SESSION_TTL_SECONDS = 7 * 24 * 60 * 60;

function getSecret() {
  const secret = process.env.JWT_SECRET;
  if (!secret || secret.length < 32) {
    throw new Error("JWT_SECRET must be set and at least 32 characters long");
  }
  return secret;
}

export function issueSession(res, email) {
  const token = jwt.sign({ email }, getSecret(), { expiresIn: SESSION_TTL_SECONDS });
  res.cookie(COOKIE_NAME, token, {
    httpOnly: true,
    secure: process.env.NODE_ENV === "production",
    sameSite: "lax",
    maxAge: SESSION_TTL_SECONDS * 1000
  });
}

export function requireAuth(req, res, next) {
  const token = req.cookies?.[COOKIE_NAME];
  if (!token) return res.status(401).json({ error: "Not authenticated." });

  try {
    const payload = jwt.verify(token, getSecret());
    req.user = { email: payload.email };
    next();
  } catch {
    res.clearCookie(COOKIE_NAME);
    return res.status(401).json({ error: "Session expired. Please sign in again." });
  }
}

sameSite: "lax" needs explaining here because it interacts directly with how magic links work. When a user clicks a link in their email client, that's a top-level navigation — the browser follows it like a normal page load. Lax allows the cookie to be sent on those top-level navigations from external origins.

Strict would block that, requiring another sign-in if a user arrives from any external link. None requires HTTPS everywhere and opens CSRF exposure. Lax is the right call for this pattern.

secure only goes on in production. Without that conditional, local development over HTTP would silently fail to set the cookie and you'd spend an hour wondering why sessions don't persist.

The two routes that do the work

// src/routes/auth.js
import { Router } from "express";
import { createToken, verifyToken } from "../lib/token.js";
import { sendMagicLink as defaultSendMagicLink } from "../lib/mailer.js";
import { issueSession } from "../lib/session.js";

const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;

export function createAuthRouter(redis, { sendMagicLink = defaultSendMagicLink } = {}) {
  const router = new Router();

  router.post("/request", async (req, res) => {
    const email = (req.body?.email ?? "").trim().toLowerCase();

    if (!EMAIL_RE.test(email)) {
      return res.status(400).json({ error: "A valid email address is required." });
    }

    try {
      const token = await createToken(redis, email);
      const appUrl = process.env.APP_URL || `http://localhost:${process.env.PORT || 3000}`;
      await sendMagicLink(email, `${appUrl}/auth/verify?token=${token}`);
      res.json({ ok: true, message: "Check your inbox for a sign-in link." });
    } catch (err) {
      console.error("[auth] request failed:", err);
      res.status(500).json({ error: "Failed to send the sign-in link. Please try again." });
    }
  });

  router.get("/verify", async (req, res) => {
    const email = await verifyToken(redis, req.query.token);
    if (!email) return res.redirect("/?error=link_invalid");
    issueSession(res, email);
    res.redirect("/dashboard");
  });

  router.post("/logout", (req, res) => {
    res.clearCookie("session");
    res.redirect("/");
  });

  return router;
}

/request returns 200 whether the email exists in your system or not. A 404 for unknown emails would tell anyone who tries that an address isn't registered — a user enumeration leak. The response is always "check your inbox," whether you're a real user or someone probing your database.

The sendMagicLink function is passed in as a default parameter, not imported at the top. That's the dependency injection hook tests use — swap it for a no-op, no SMTP connection required.

Running the whole thing

# Start Redis (Docker is fastest)
docker run -d -p 6379:6379 redis:alpine

# Install deps and start
npm install && cp .env.example .env && npm start

The server needs at minimum a JWT_SECRET in .env. Generate one:

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

Request a link:

curl -X POST http://localhost:3000/auth/request \
  -H "Content-Type: application/json" \
  -d '{"email": "you@example.com"}'

{ "ok": true, "message": "Check your inbox for a sign-in link." }

Your server console will print something like:

[mailer] Preview URL: https://ethereal.email/message/WaQKMgKddxQDoou

Open that URL, click the "Sign in" button in the rendered email. You'll land on the dashboard. The token is gone — clicking the link again redirects to /?error=link_invalid.

Check your session after clicking (saves cookie to cookies.txt):

curl -c cookies.txt -b cookies.txt http://localhost:3000/api/me

{ "email": "you@example.com" }

Invalid email format:

curl -s -X POST http://localhost:3000/auth/request \
  -H "Content-Type: application/json" \
  -d '{"email": "notanemail"}'

{ "error": "A valid email address is required." }

Rate limiter — 5 requests per 15 minutes per IP, sixth gets a 429:

for i in $(seq 1 6); do
  curl -s -o /dev/null -w "request $i -> %{http_code}\n" \
    -X POST http://localhost:3000/auth/request \
    -H "Content-Type: application/json" \
    -d '{"email":"test@example.com"}'
done

request 1 -> 200
request 2 -> 200
request 3 -> 200
request 4 -> 200
request 5 -> 200
request 6 -> 429

Five is tight enough to stop inbox flooding and loose enough that a real user who typo'd their address gets a few retries.

Expired or already-used token:

curl -v "http://localhost:3000/auth/verify?token=$( python3 -c 'print("a"*64)')" 2>&1 | grep "Location:"

< Location: /?error=link_invalid

Tests

19 tests, no Redis or SMTP connection needed. The token suite runs against a plain in-memory Map mimicking the Redis interface; the session suite sets JWT_SECRET in before() and cleans up after; the route suite injects the no-op mailer:

npm test

# tests 19
# pass  19
# fail   0

If you want to verify the atomic single-use behavior beyond the unit test, run the server with a real Redis instance and hit /auth/verify with the same token from two curl commands fired in parallel:

TOKEN="paste-a-real-token-from-console-here"
curl "http://localhost:3000/auth/verify?token=$TOKEN" &
curl "http://localhost:3000/auth/verify?token=$TOKEN" &
wait

One will redirect to /dashboard. The other will redirect to /?error=link_invalid. That's getDel doing its job.

Before going live

Set NODE_ENV=production — the Secure cookie flag only activates in production, and without HTTPS the cookie won't be sent by browsers at all. Point REDIS_URL at a managed instance (Upstash has a free tier and works well with this setup). Swap Ethereal for a transactional email provider; Resend has a generous free tier and a clean Node.js SDK. Put the whole thing behind nginx or Caddy for TLS termination.

One thing the repo doesn't include but you'll want eventually: a users table or equivalent. Right now any email can request a link and get a session — there's no concept of "registered users." Adding a check in /request that verifies the email exists in your database before sending the link is one line, but the shape of that check depends on your stack.

Get the code: http://github.com/zyvop27-cmyk/zyvop-blogs/tree/main/magic-link-auth

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Rate Limiting Alone Won't Stop a Patient Attacker

ZyVOP — Sat, 04 Jul 2026 02:29:26 +0000

@nestjs/throttler counts requests per IP address within a time window. That's the entire mechanism — no concept of accounts, passwords, or "this one person is being targeted." Point it at a login endpoint and it will correctly stop a script hammering /login a hundred times a second. It will do nothing against someone guessing a password once every few minutes, or spreading attempts across a dozen IPs, because that was never the problem it's built to solve.

Below is both halves of a real defense: the IP throttle for volume, and a small Redis-backed service that locks by email for everything the throttle misses. Every response shape, status code, and timing claim here came from actually running the requests, not from the docs — including one interaction between two timers that will quietly undo a lockout's own "fresh start" if you're not paying attention to how they relate.

The throttle, and what it actually returns

A baseline policy on every route, plus a stricter override on login specifically:

// app.module.ts — global baseline: 20 requests/60s per IP, everywhere
ThrottlerModule.forRoot([{ name: 'default', ttl: 60000, limit: 20 }]),
providers: [{ provide: APP_GUARD, useClass: ThrottlerGuard }],

// auth.controller.ts — login gets a stricter limit than the rest of the API
@Throttle({ default: { limit: 10, ttl: 60000 } })
@Post('login')
async login(@Body() dto: LoginDto) { /* ... */ }

Whether that override replaces the global 20/60s or stacks on top of it isn't obvious from reading the decorator, so it's worth confirming rather than guessing: it replaces. A route-level @Throttle with the same policy name (default) fully takes over for that route. Eleven rapid requests to /auth/register (no override, rides the global policy) all came back 201; the same eleven against /auth/login would have tripped its 10-request limit well before the last one.

The response headers on a request that's still under the limit:

X-RateLimit-Limit: 10
X-RateLimit-Remaining: 9
X-RateLimit-Reset: 60

And once it's crossed:

// HTTP 429
{ "statusCode": 429, "message": "ThrottlerException: Too Many Requests" }

That's the library's real, unmodified default — not a placeholder for something cleaner. It reads like a stringified exception because that's more or less what it is, and it's worth routing through @nestjs/throttler's exception factory if this needs to match the rest of an API's error shape.

Where the throttle's job ends

None of that knows or cares who's logging in. Ten requests a minute from one IP against alice@example.com and ten requests a minute from ten different IPs, one attempt each, all against alice@example.com, look completely different to a per-IP counter — the second pattern never trips it, no matter how long it continues. That's the gap a second, IP-independent mechanism has to cover.

Locking the account, not the address

async recordFailure(email: string): Promise<FailureResult> {
  const key = this.attemptsKey(email);
  const attempts = await this.redis.incr(key);
  if (attempts === 1) {
    await this.redis.expire(key, this.windowSeconds);
  }

  if (attempts >= this.maxAttempts) {
    await this.redis.set(this.lockKey(email), '1', 'EX', this.lockoutSeconds);
    return { attempts, locked: true, retryAfterSeconds: this.lockoutSeconds };
  }

  return { attempts, locked: false };
}

async recordSuccess(email: string): Promise<void> {
  await this.redis.del(this.attemptsKey(email), this.lockKey(email));
}

INCR against a key that doesn't exist yet starts it at 1, so there's no separate setup step for a first failure. The counter's expiry is set once — only when attempts === 1 — so a streak of failures within one window doesn't keep pushing its own deadline out; it's one streak, with one expiry, not a self-renewing one.

Both key-building helpers lowercase the email first:

private attemptsKey(email: string): string {
  return `login-attempts:${email.toLowerCase()}`;
}

Skipping that normalization would open a real gap, not just a style nitpick — so rather than assume it works, this was tested directly: three failures as dana@example.com, then two more as Dana@Example.COM, a different casing entirely. The Redis counter read 5 afterward, under a single key, and a sixth attempt in yet another casing (DANA@EXAMPLE.COM) — this time with the correct password — still came back locked. Without the .toLowerCase(), those three casings would have been three separate five-strike budgets instead of one.

Two timers that look independent and aren't

The service has two separate durations: how long the failure counter itself lives (windowSeconds), and how long an actual lock lasts once triggered (lockoutSeconds). Treating them as unrelated knobs is the natural first instinct, and it's wrong — tested directly, with a 10-second window and a 6-second lock:

lock status right after the 6s lock expires: { locked: false }
attempts count at that same moment:          5

The lock is gone, but the counter — on its own, longer, 10-second clock — hasn't caught up yet. It's still sitting at 5. One more failed attempt right then doesn't start over at 1; it pushes the existing counter to 6, still over the 5-attempt threshold, and the account locks again immediately:

result of one failure right after unlock: { attempts: 6, locked: true }

So "unlocked" didn't mean "clean slate" here — it meant "one more mistake and you're back in." That might be exactly the behavior you want (harsher consequences for someone who fails again right after a lockout), but it should be a choice, not a side effect of two numbers that happened to get picked independently. Defaulting LOGIN_LOCKOUT_WINDOW_SECONDS and LOGIN_LOCKOUT_DURATION_SECONDS to the same value is what makes the out-of-the-box behavior a genuine reset: by the time the lock is gone, so is the counter it was based on. Tested with matched timers instead of mismatched ones, the positive case is exactly as boring as it should be — wait out the lock, and a correct password just works:

curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'
# -> 201 { "accessToken": "..." }   (no failures logged anywhere afterward)

The failure that triggers a lock doesn't announce it

try {
  const user = await this.authService.validatePassword(dto.email, dto.password);
  await this.loginAttemptService.recordSuccess(dto.email);
  return { accessToken: this.authService.issueToken(user) };
} catch (err) {
  await this.loginAttemptService.recordFailure(dto.email);
  throw err; // same error whether or not this failure just triggered a lock
}

Four wrong-password attempts return four identical 401s. The fifth — the one that actually crosses the threshold — returns that same 401, not a warning. The lock only becomes visible on whatever comes next, correct password included. Confirmed end to end: four 401s, a fifth 401 that locked the account behind the scenes, and only the sixth request got the 429. Nothing in that fifth response tells anyone it was the last try, which is one less signal an attacker gets to calibrate around.

Both mechanisms, same request

# alice fails 5 times, then even the right password is rejected
curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"wrong-password"}'
# -> 401  (x5)

curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'
# -> 429 {"statusCode":429,"message":"Account temporarily locked...","reason":"ACCOUNT_LOCKED"}

# bob, meanwhile, is entirely unaffected
curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"bob@example.com","password":"correct-horse-battery"}'
# -> 201 { "accessToken": "..." }

The lockout 429 and the throttler's 429 share a status code but not a body — "reason":"ACCOUNT_LOCKED" versus the generic ThrottlerException message — so nothing downstream has to guess which mechanism fired.

One sequencing detail if you're replicating this yourself: the IP throttle counts every call to /login in its window, not just the ones in whatever you'd mentally group as "the throttle test." Running the lockout sequence above first, then immediately sending a batch of requests to check the throttle, tripped the 429 on the third request of that batch rather than somewhere near the tenth — because the eight requests just spent on the lockout scenario were already sitting in the same 60-second window. Not a bug, just a shared counter that doesn't know your test plan has phases.

Running it

Needs Postgres and Redis; Docker is the fastest path to both:

git clone <your-repo-url> && cd rate-limiting-nestjs-demo
npm install && cp .env.example .env
docker run --name rl-postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=ratelimit_demo -p 5432:5432 -d postgres:16
docker run --name rl-redis -p 6379:6379 -d redis:7
npm run start:dev

The shipped defaults (5 attempts, a 15-minute window and lock) are real production numbers, not demo placeholders, but they're still a starting point rather than a universal answer — how forgiving to be with a legitimate user who fumbles their password a few times is a judgment call specific to what's being protected. Lower LOGIN_LOCKOUT_WINDOW_SECONDS/LOGIN_LOCKOUT_DURATION_SECONDS in your own .env if you want to watch a lock expire without an actual 15-minute wait.

What's deliberately not here: the in-memory throttle storage this demo uses doesn't share counters across multiple app instances, so a horizontally-scaled deployment needs a shared store (@nestjs/throttler supports pluggable backends, Redis included) or the limit quietly stops meaning what it says. There's also no lockout notification to the account owner, and no CAPTCHA as a third layer for public-facing forms — both reasonable additions, both left out here to keep the two mechanisms that are built easy to see clearly.

The full project — this module, the lockout service, and a README with the complete setup and curl walkthrough — is in the rate-limiting-nestjs-demo repository alongside this post.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Background Jobs in NestJS with BullMQ: A Complete Walkthrough

ZyVOP — Fri, 03 Jul 2026 04:10:06 +0000

Calling a slow, occasionally-unreliable upstream API — an LLM provider, a payment processor, a third-party webhook — directly inside a request handler ties that request's fate to the upstream call's fate. If it's slow, the client waits on the full round trip. If it fails, the handler has to decide on the spot whether to fail the whole request or quietly fall back to something else, which is exactly the kind of decision that's easy to get wrong under pressure and hard to notice when it goes wrong silently.

This post builds a background job pipeline in NestJS using BullMQ: a request that enqueues work and returns immediately, a worker that processes it with automatic retries, and a durable record a client can poll for the outcome. It's demonstrated on an async draft-generation endpoint — the same shape as a real LLM-backed content pipeline — with a couple of details about BullMQ's actual runtime behavior that are easy to get wrong if you're going off the docs alone. The full project, tested end-to-end against real Postgres and Redis, is linked at the end.

The shape of the problem

A synchronous version of this endpoint looks simple:

@Post('drafts')
async create(@Body() dto: CreateDraftDto) {
  const content = await this.llmProvider.generate(dto.topic); // could take 5-30s, could fail
  return this.draftsRepository.save({ topic: dto.topic, content });
}

The request now blocks for as long as the LLM call takes, and a single transient failure (a timeout, a rate limit, a brief provider outage) becomes a failed request with nothing to show for it — no record that it was attempted, nothing to retry, nothing to inspect afterward.

The fix is to decouple "accept the request" from "do the work":

@Post('drafts')
create(@Body() dto: CreateDraftDto) {
  return this.draftsService.enqueue(dto); // returns in milliseconds
}

enqueue() writes a pending row and hands the job to BullMQ. A separate worker process picks it up, retries it automatically on failure, and updates that row when it's done — successfully or not.

Two sources of truth, on purpose

This implementation keeps two records of a job's state, deliberately:

BullMQ's own state, in Redis — which attempt it's on, when it'll retry next, its position in the queue. This is operational state, and it's normal for it to get cleaned up after a job finishes (removeOnComplete/removeOnFail).
A Postgres row, written by the application — pending → processing → completed/failed, plus the result or failure reason. This is what survives a Redis flush, what a client actually polls, and what you'd query for "show me every failed draft from last week."

export enum DraftJobStatus {
  PENDING = 'pending',
  PROCESSING = 'processing',
  COMPLETED = 'completed',
  FAILED = 'failed',
}

@Entity()
export class DraftJob {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  topic: string;

  @Column({ type: 'varchar', default: DraftJobStatus.PENDING })
  status: DraftJobStatus;

  @Column({ type: 'text', nullable: true })
  result: string | null;

  @Column({ type: 'text', nullable: true })
  failureReason: string | null;

  @Column({ default: 0 })
  attemptsMade: number;

  @CreateDateColumn()
  createdAt: Date;

  @Column({ type: 'timestamp', nullable: true })
  completedAt: Date | null;
}

Conflating these two — treating BullMQ's Redis-backed job as the only record — means losing history the moment a job gets cleaned up, and gives clients nothing stable to poll against.

Wiring BullMQ into the Nest app

Before any of that, BullMQ needs a Redis connection — configured once, at the root module:

// app.module.ts
BullModule.forRootAsync({
  inject: [ConfigService],
  useFactory: (configService: ConfigService) => ({
    connection: {
      host: configService.get<string>('REDIS_HOST'),
      port: configService.get<number>('REDIS_PORT'),
    },
  }),
}),

Then, inside whichever feature module actually uses a queue, that queue gets declared by name:

// drafts.module.ts
BullModule.registerQueue({ name: 'draft-generation' }),

That string, 'draft-generation', is the thread tying three separate places together: it's what registerQueue declares here, what @InjectQueue('draft-generation') asks for in the service below, and what @Processor('draft-generation') listens on in the worker. All three have to use the exact same name — there's no compiler check enforcing that, since it's just a string. The two ways this can go wrong behave very differently, and it's worth knowing which is which rather than assuming:

If @InjectQueue doesn't match anything registerQueue declared, Nest's dependency injection fails loudly at startup with an UnknownDependenciesException — I confirmed this directly, and the error message names the exact missing provider token and suggests the fix. The app won't boot at all, so this mistake gets caught immediately.
If @Processor doesn't match — while @InjectQueue/registerQueue still agree with each other — the app boots without any error, jobs enqueue successfully, and then just sit in the queue forever. I verified this too: enqueued a job against a correctly-wired producer, waited, and checked the queue directly — the job's state was still waiting, indefinitely, with nothing logged anywhere to indicate why. No worker was ever listening on that name. This is the genuinely silent failure mode worth watching for, since nothing about it looks broken until someone notices a backlog that never drains.

Enqueueing a job

The job's payload type is defined right alongside the service that creates it — enqueue() and the worker (below) both depend on this shape:

export interface DraftJobData {
  draftJobId: string;
  topic: string;
  simulateFailures: number;
}

async enqueue(dto: CreateDraftDto) {
  const draftJob = await this.draftJobsRepository.save(
    this.draftJobsRepository.create({ topic: dto.topic, status: DraftJobStatus.PENDING }),
  );

  await this.draftQueue.add(
    'generate',
    { draftJobId: draftJob.id, topic: dto.topic, simulateFailures: dto.simulateFailures ?? 0 },
    {
      jobId: draftJob.id,
      attempts: 3,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: { age: 3600 },
      removeOnFail: { age: 86400 },
    },
  );

  return { id: draftJob.id, status: draftJob.status };
}

A few specific choices worth calling out:

jobId: draftJob.id — using the Postgres row's own id as the BullMQ job id, rather than letting BullMQ generate one, is what makes re-enqueueing idempotent (more on this below).

backoff: { type: 'exponential', delay: 2000 } — each retry waits roughly double the previous gap, rather than hammering a struggling upstream API at a fixed interval. With attempts: 3 there are only ever two such gaps to observe, not three — I measured them directly against this exact config rather than trusting the formula from memory: ~2035ms after the first failure, ~4009ms after the second, and no fourth attempt ever fires once those three tries are exhausted. The delay keeps doubling if you raise attempts higher, but at attempts: 3 specifically, a third gap is never reachable — that ceiling comes from attempts, not from backoff alone.

removeOnComplete/removeOnFail — without these, BullMQ keeps every job's data in Redis indefinitely. These ages keep Redis from growing unbounded while still leaving failed jobs around longer (a day, vs. an hour for successes) since they're more likely to need debugging.

The worker

@Processor('draft-generation', { concurrency: 5 })
export class DraftGenerationProcessor extends WorkerHost {
  constructor(private readonly draftsService: DraftsService) {
    super();
  }

  async process(job: Job<DraftJobData>): Promise<string> {
    const { draftJobId, topic, simulateFailures } = job.data;
    await this.draftsService.markProcessing(draftJobId);

    // Throwing here is what tells BullMQ to retry, subject to `attempts`/`backoff`.
    return generateDraftContent(topic, job.attemptsMade, simulateFailures);
  }

  @OnWorkerEvent('completed')
  async onCompleted(job: Job<DraftJobData>, result: string) {
    await this.draftsService.markCompleted(job.data.draftJobId, result, job.attemptsMade);
  }

  @OnWorkerEvent('failed')
  async onFailed(job: Job<DraftJobData> | undefined, error: Error) {
    if (!job) return;
    const maxAttempts = job.opts.attempts ?? 1;
    if (job.attemptsMade >= maxAttempts) {
      await this.draftsService.markFailed(job.data.draftJobId, job.attemptsMade, error.message);
    }
    // else: still has retries left, BullMQ will reschedule it automatically
  }
}

@Processor(...) plus extending WorkerHost is the @nestjs/bullmq pattern for defining a worker — process() is the actual job handler, and @OnWorkerEvent hooks into BullMQ's lifecycle events. (The actual source file also logs each attempt via Nest's Logger, trimmed from the snippet above for readability — the logic shown is otherwise unchanged from what's in the repo.)

The detail that's easy to get backwards: `attemptsMade`

BullMQ's 'failed' event fires after every failed attempt, not just the last one. If you write the failure-handling logic without checking attempt count, a job that's about to succeed on its third try gets incorrectly marked failed after its first.

The fix is the job.attemptsMade >= maxAttempts check above — but getting that comparison right depends on knowing exactly what attemptsMade contains at each point, which isn't obvious from the type signature alone. I checked this directly against a running BullMQ instance rather than assuming:

attemptLog (job.attemptsMade as seen INSIDE the processor on each run):
[ { attemptsMade: 0, opts: 3 },
  { attemptsMade: 1, opts: 3 },
  { attemptsMade: 2, opts: 3 } ]
final 'completed' event attemptsMade: 3

attemptsMade is 0 on the very first execution, not 1 — it counts completed prior attempts, not the current attempt number. Inside process(), a job configured with attempts: 3 sees 0, 1, 2 across its three tries; the 'completed'/'failed' event handlers see it afterward, already incremented to 3. Getting this backwards (checking attemptsMade <= maxAttempts instead of >=, or assuming attempt 1 reads as 1 inside the processor) produces a retry condition that's off by exactly one — either giving up one attempt early, or never giving up at all.

What "idempotent" actually means here

Using the Postgres row's id as the BullMQ jobId means re-enqueueing the same id is a no-op — but it's worth being precise about what that actually does, rather than taking it on faith. I tested this directly:

await queue.add('task', { n: 1 }, { jobId: 'fixed-id' }); // creates the job
await queue.add('task', { n: 2 }, { jobId: 'fixed-id' }); // returns a Job object, but...

const stored = await queue.getJob('fixed-id');
console.log(stored.data); // { n: 1 } — the SECOND add() never took effect

The second add() call doesn't throw, and it doesn't error — it just silently does nothing. The job already in Redis under that id keeps its original data. This holds whether the original job is still waiting, actively processing, or has already completed (as long as it hasn't been cleaned up by removeOnComplete).

That's exactly the property you want for a reconciliation or retry path elsewhere in your own backend: if something upstream of enqueue() ever calls it twice for the same logical request — a retried HTTP call, a duplicate webhook delivery, a race in a distributed system — the second call doesn't create a second job, doesn't reprocess, and doesn't overwrite the first job's data with whatever the second call happened to pass. It's a much cheaper idempotency guarantee than building your own deduplication table, but it only works because the jobId is something stable and meaningful (the Postgres row's id) rather than an auto-generated one.

Setting up and running the project

Clone the repo, install dependencies, and copy the environment template:

git clone <your-repo-url>
cd bullmq-nestjs-demo
npm install
cp .env.example .env

This project needs both Postgres and Redis running locally — Postgres for the durable job records, Redis for BullMQ's own queue state. The fastest path for both is Docker:

docker run --name jobs-postgres \
  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=jobs_demo \
  -p 5432:5432 -d postgres:16

docker run --name jobs-redis -p 6379:6379 -d redis:7

The defaults in .env.example already line up with those two containers, so you shouldn't need to edit .env if you used them as-is. Then start the API:

npm run start:dev

synchronize: true is enabled in app.module.ts for this demo, so the draft_job table is created automatically on first boot — no manual migration step needed to follow along. (Turn that off and switch to real migrations before deploying anywhere real.) The worker runs inside this same process for simplicity here; see the production notes below on why you'd typically split it out.

With the server up on http://localhost:3000, you're ready to run through the flow below.

Testing it end-to-end

This is the actual sequence run against a live server, real Postgres, and real Redis before publishing — happy path, retry-then-succeed, permanent failure, and the validation/not-found edge cases:

# Happy path — no simulated failures
curl -X POST http://localhost:3000/drafts \
  -H "Content-Type: application/json" \
  -d '{"topic":"NestJS background jobs"}'
# -> { "id": "...", "status": "pending" }

curl http://localhost:3000/drafts/<id>
# -> { "status": "completed", "attemptsMade": 1, "result": "..." }

# Forces 2 failures before success — watch attemptsMade end at 3,
# with ~2s then ~4s of exponential backoff between attempts
curl -X POST http://localhost:3000/drafts \
  -H "Content-Type: application/json" \
  -d '{"topic":"Retry demo","simulateFailures":2}'

curl http://localhost:3000/drafts/<id>
# (poll a few times over ~6-8s)
# -> { "status": "completed", "attemptsMade": 3, "result": "..." }

# More failures than the configured 3 attempts allow — exercises the
# permanent-failure path instead
curl -X POST http://localhost:3000/drafts \
  -H "Content-Type: application/json" \
  -d '{"topic":"Permanent failure demo","simulateFailures":5}'

curl http://localhost:3000/drafts/<id>
# -> { "status": "failed", "attemptsMade": 3, "failureReason": "Simulated upstream failure..." }

# Validation and not-found paths, checked the same way
curl -X POST http://localhost:3000/drafts \
  -H "Content-Type: application/json" \
  -d '{"topic":"a"}'
# -> 400 Bad Request — topic is shorter than the DTO's @MinLength(3)

curl http://localhost:3000/drafts/00000000-0000-0000-0000-000000000000
# -> 404 { "message": "Draft job not found" }

Every path above — immediate success, eventual success after retries, permanent failure after exhausting retries, a rejected validation error, and an unknown id — was checked against the actual status, attemptsMade, and HTTP status code returned by a live server, not just "the request didn't error." The duplicate-jobId idempotency behavior from the previous section was verified the same way, against this same running instance, using a small script that called queue.add() directly rather than going through the HTTP API (since triggering a raw duplicate enqueue isn't something a normal client request can do on its own).

Production notes

A few things were deliberately simplified for this demo and are worth tightening before shipping:

Run the worker as a separate process from the API. Here, the processor lives inside the same NestJS app as the controller — fine for a demo, but in production a traffic spike on the HTTP side will compete for CPU with job processing unless they're split into independent, independently-scalable deployments.
Distinguish retryable from non-retryable errors. A timeout or a 429 should retry. A 401 from a bad API key will fail identically on every attempt — throw new UnrecoverableError(...) (exported by BullMQ) skips the remaining retries instead of wasting them on a guaranteed failure.
Size concurrency to what the upstream and your database can actually sustain. concurrency: 5 here is a demo default, not a number derived from real capacity.
Monitor queue depth, not just individual job outcomes. A queue that's silently backing up faster than it's draining is a different failure mode than any single job failing, and won't show up by looking at one job's status at a time.
There's no authentication on these endpoints. This demo is scoped to the queueing mechanics, not access control — anyone who can reach POST /drafts can enqueue work, and anyone who knows (or guesses) a job id can read its result. If this sits behind a real API, put it behind the same kind of auth boundary as anything else you wouldn't want publicly writable (a prior post on this blog covers building TOTP-based 2FA in NestJS, if that's useful context for the kind of guard logic involved).

Source code

The complete, tested implementation — NestJS module, the worker, entity, DTOs, and a README with the full setup and curl walkthrough — is available as a standalone repository: bullmq-nestjs-demo. Clone it, point it at your own Postgres and Redis, and the enqueue → retry → status-poll flow above works out of the box.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

AI’s Growing Pains: Compute Caps, Security Harnesses, and the Human Touch | The AI Daily Roundup

ZyVOP — Thu, 02 Jul 2026 11:18:59 +0000

Connecting the Dots: AI Is Hitting Its Operational Limits

Across the headlines, a single narrative emerges: the AI boom is moving from raw model hype to the gritty reality of deployment. Companies are now wrestling with three intertwined constraints—compute scarcity, security/privacy tooling, and the need for human expertise. The stories below illustrate how each pressure point is reshaping the ecosystem.

1. Compute Scarcity Becomes a Competitive Weapon

Google limits Meta’s use of Gemini shows that even the largest cloud providers cannot guarantee unlimited GPU capacity. Meta’s request for additional compute was denied, forcing the social‑media giant to tighten token usage and delay internal projects. This is a concrete reminder that AI scaling is bounded by physical hardware, not just capital.

In response, Austria is lobbying the EU to host Anthropic. By relocating critical AI workloads to Europe, Anthropic hopes to sidestep US export curbs and secure a more predictable compute pipeline. The geopolitical maneuver underscores that access to compute is becoming a strategic asset for AI firms.

Beneficiaries: Cloud providers that can guarantee capacity, regions investing in AI‑focused data centers.
Losers: Companies that depend on a single provider’s surplus capacity (e.g., Meta’s delayed projects).

2. Security Harnesses Trump Raw Model Power

Semgrep’s benchmark shows GLM‑5.2 beating Claude when only the model is considered, but the report also highlights that a purpose‑built harness can lift performance from 39% to over 50% F1. The takeaway for CTOs is clear: the surrounding pipeline—code ingestion, output parsing, feedback loops—often determines real‑world security outcomes more than the model itself.

Parallel concerns appear in the OpenAI Codex ignore‑file request. Developers demand deterministic mechanisms (.codexignore) to keep sensitive files out of model prompts, a feature that is essentially a security harness at the data‑access layer. As AI agents become more autonomous, guardrails built into the tooling stack become non‑negotiable.

Beneficiaries: Vendors offering end‑to‑end AI security platforms (e.g., Semgrep, CodeQL, specialized harness frameworks).
Losers: Teams that rely solely on “plug‑and‑play” models without investing in integration engineering.

3. Human Expertise Remains the Safety Net

Ford’s decision to re‑hire veteran engineers after AI‑driven quality systems fell short is a cautionary tale. The automaker discovered that AI alone could not guarantee the precision required on the assembly line, prompting a hybrid model where seasoned engineers train and audit the AI tools. This mirrors the broader industry realization that AI augments, not replaces, domain experts.

In academia, Professor Roberto Serrano’s exposure of a massive cheating scandal at Brown (El Pais article) illustrates the opposite side: unchecked AI access can erode trust in institutions. The incident forces universities to rethink assessment design, detection tools, and policy—again, a human‑centric response to AI misuse.

Beneficiaries: Companies that blend AI with skilled personnel (e.g., Ford, security firms with expert‑in‑the‑loop models).
Losers: Organizations that attempted to replace human oversight entirely, risking quality or credibility lapses.

4. Macro‑Level Risks and Market Signals

Central bankers warning of an AI‑driven financial crash adds a macroeconomic dimension. When compute scarcity drives up token prices, and when security incidents force costly mitigations, the sector’s cash burn can outpace revenue, threatening broader financial stability.

Investors should watch for signs of “AI‑infrastructure debt” – companies that have over‑promised AI capabilities without securing the underlying compute, security, or talent foundations.

5. Grassroots Tooling and the DIY Ethos

On the developer front, projects like Bash4LLM+ show a push for lightweight, language‑agnostic interfaces to LLMs. While these tools democratize access, they also amplify the earlier themes: without proper harnesses and security policies, even a single‑line Bash script can inadvertently leak proprietary code or sensitive data.

Similarly, personal experiments such as using Claude Code for a second‑opinion MRI (Antoine’s blog) highlight the allure of AI in niche domains, yet they also expose liability gaps that regulators will soon address.

6. The Emerging Playbook for Leaders

For senior engineers, CTOs, and investors, the actionable takeaways are:

Secure compute pipelines. Diversify providers, explore regional data‑center partnerships, and budget for premium capacity.
Invest in harness engineering. Build or adopt frameworks that handle data sanitization, prompt engineering, and result validation.
Maintain human‑in‑the‑loop checkpoints. Especially for safety‑critical or high‑trust applications (automotive, healthcare, finance).
Monitor regulatory and geopolitical shifts. US export controls, EU hosting incentives, and central‑banker warnings will shape market dynamics.

Companies that internalize these constraints will turn today’s growing‑pains into a competitive moat; those that ignore them risk costly rollbacks, compliance penalties, or outright project failure.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Building a Production AI Agent in Node.js: Tool Calling, the ReAct Loop, and Error Handling

ZyVOP — Wed, 01 Jul 2026 14:52:26 +0000

Most agent tutorials stop at a toy. A bot that checks the weather, a script that answers one question, then a victory lap in the README.

None of that prepares you for what happens when a tool throws an error, the model calls a function ten times in a row, or you blow past your rate limit mid-conversation. This post builds the other kind of agent, running on Groq.

Groq is worth a look for this specifically: an OpenAI-compatible API, no credit card to start, and inference fast enough that the agent loop feels instant instead of laggy. No framework either — just the SDK and a loop you can read top to bottom. By the end you'll have a customer-support agent that looks up orders, does math, and searches a FAQ, plus the iteration caps, retries, and error handling that keep it from falling over once real traffic hits it.

Full source, including the test suite, is linked at the bottom.

What "production" actually means here

An agent isn't a chatbot with extra steps. It reasons about a task, picks a tool, looks at the result, and decides what to do next, looping until it has an answer. That loop is usually called ReAct: reason, act, observe, repeat.

The demo version of this loop assumes everything goes right. The production version has to handle:

A tool that throws, times out, or returns garbage
A model that keeps calling tools and never wraps up
A 429 or a 500 from the API itself, which is different from a tool failure
A model that sends back malformed arguments for a tool call
Needing to know what happened after the fact, not just whether it worked

Each of those gets its own piece of the code below.

Why Groq, and why no framework

Groq doesn't train its own models. It runs open-source ones (Llama, GPT-OSS, Qwen, and others) on custom LPU hardware built for inference speed.

The free tier needs no credit card and gives you every model, gated only by rate limits: roughly 30 requests a minute and a few thousand tokens a minute per model, enforced per account rather than per key. That's plenty for a demo and for prototyping. It's not enough for real user traffic without upgrading.

Because Groq's API mirrors OpenAI's chat completions format, you don't need LangChain.js or any other framework to get tool calling working. The groq-sdk package gives you the same loop with nothing hidden, which matters when something breaks and you need to know exactly where.

The loop itself is under 100 lines. You're about to read all of them.

Defining a tool

Groq's tool format wants a type: "function" wrapper, a name, a description the model reads to decide when to use the tool, and a JSON schema for the arguments. Here's the calculator tool in full:

// src/tools/calculator.js
import { safeCalculate } from "../lib/safeCalculate.js";

export const calculatorTool = {
  definition: {
    type: "function",
    function: {
      name: "calculate",
      description:
        "Evaluate a basic arithmetic expression. Supports +, -, *, /, ^, parentheses, and decimals. " +
        "Always use this instead of doing math yourself, including for things like order totals or discounts.",
      parameters: {
        type: "object",
        properties: {
          expression: {
            type: "string",
            description: "A math expression, e.g. '(42 + 8) * 3' or '199.99 * 0.85'"
          }
        },
        required: ["expression"]
      }
    }
  },

  async handler({ expression }) {
    const result = safeCalculate(expression);
    return String(result);
  }
};

That description line is doing real work. It tells the model to reach for this tool instead of computing totals itself, which matters because language models are unreliable at arithmetic. The tool use overview covers schema design in more depth if you're writing tools with nested objects or several optional fields.

One thing worth flagging: safeCalculate is a small hand-written parser, not eval(). The expression comes from the model, and the model's input ultimately traces back to whatever the user typed.

Piping that into eval() or new Function() means a sufficiently creative prompt is now running arbitrary code on your server. The repo's calculator does the math with a real tokenizer and recursive-descent parser instead — more code, zero risk.

The other two tools follow the same shape: get_order_status looks up a mock order by ID (with a deliberately broken ORD-FAIL entry for testing failure handling), and search_knowledge_base does keyword search over a small FAQ array. Full source for both is in the repo.

The loop itself

This is the part that drives the agent. It lives in src/agent.js:

async run(userMessage, { history = [], onStep } = {}) {
  const messages = [...history, { role: "user", content: userMessage }];
  const trace = [];
  const usage = { inputTokens: 0, outputTokens: 0 };

  for (let step = 0; step < this.maxIterations; step++) {
    const response = await this.callModel(messages);
    usage.inputTokens += response.usage?.prompt_tokens ?? 0;
    usage.outputTokens += response.usage?.completion_tokens ?? 0;

    const choice = response.choices[0];
    const message = choice.message;

    if (choice.finish_reason !== "tool_calls" || !message.tool_calls?.length) {
      const text = (message.content ?? "").trim();
      messages.push({ role: "assistant", content: message.content ?? "" });
      const finalStep = { step, type: "final", text };
      trace.push(finalStep);
      onStep?.(finalStep);
      return { text, steps: step + 1, usage, trace, history: messages, truncated: false };
    }

    // Groq requires the assistant's tool_calls message echoed back verbatim
    // before the matching tool results — this is the OpenAI-style contract.
    messages.push({ role: "assistant", content: message.content, tool_calls: message.tool_calls });

    for (const toolCall of message.tool_calls) {
      const result = await this.executeTool(toolCall, step);
      trace.push(result.traceEntry);
      onStep?.(result.traceEntry);
      messages.push(result.toolMessage);
    }
  }

  const truncatedStep = { step: this.maxIterations, type: "max_iterations_reached" };
  trace.push(truncatedStep);
  onStep?.(truncatedStep);

  return {
    text: "I wasn't able to finish this within the step limit. Here's what I found before stopping.",
    steps: this.maxIterations,
    usage,
    trace,
    history: messages,
    truncated: true
  };
}

Five things worth calling out:

The iteration cap is the for loop's bound, not an afterthought. Without it, a model stuck in a reasoning loop — or a tool that always returns "try again" — burns through your rate limit until something else stops it. That matters more on a free tier than a metered one: you don't lose money, you lose your remaining requests for the minute, and maxIterations defaults to 8 here to keep that from happening.

finish_reason is checked, not just whether tool_calls exists. Groq's response can technically include a tool_calls array while finish_reason says something else (truncated output, for instance), so the loop checks both before deciding the model wants a tool run.

Tool failures don't throw past this function. executeTool (below) catches whatever the tool does and turns it into a normal tool role message. The model sees the failure as part of the conversation and can retry with different input, try another tool, or tell the user it couldn't complete the request.

history comes in and goes out, and it never contains the system message. The system prompt gets injected fresh inside callModel on every call instead of being stored in history. Otherwise a long conversation would accumulate a duplicate system message every turn.

Every step gets logged through onStep, whether it's a tool call or the final answer. The CLI uses this to print tool calls in verbose mode; a real deployment would pipe it to whatever you use for logging.

Tool execution looks like this:

async executeTool(toolCall, step) {
  const name = toolCall.function.name;
  const handler = this.findHandler(name);
  const startedAt = Date.now();
  let content;
  let isError = false;

  try {
    if (!handler) throw new Error(`Unknown tool: ${name}`);

    let input;
    try {
      input = JSON.parse(toolCall.function.arguments || "{}");
    } catch {
      throw new Error(`Model sent invalid JSON arguments for "${name}"`);
    }

    content = await this.withTimeout(handler(input), this.toolTimeoutMs);
  } catch (err) {
    isError = true;
    content = `Tool error: ${err.message}`;
    this.logger.warn?.(`[agent] tool "${name}" failed: ${err.message}`);
  }

  const durationMs = Date.now() - startedAt;

  return {
    traceEntry: { step, type: "tool_call", tool: name, input: toolCall.function.arguments, result: content, isError, durationMs },
    toolMessage: { role: "tool", tool_call_id: toolCall.id, content: String(content) }
  };
}

Two details here are specific to Groq's OpenAI-style format and don't show up if you're used to Anthropic's tool-calling shape. First, toolCall.function.arguments arrives as a JSON string, not an already-parsed object. The model can and occasionally will send back something that doesn't parse, so that JSON.parse is wrapped in its own try/catch with a clear error message rather than letting it throw a raw SyntaxError up the stack.

Second, there's no is_error flag on the result message the way Anthropic's tool_result blocks have one. A failed tool just returns its error as text inside a normal tool role message, and the model reads it like any other result.

The timeout matters as much as the try/catch. A tool that calls a flaky downstream API can hang for a long time if you let it; withTimeout races the handler against a timer and turns a hang into a clean error after 10 seconds by default.

API-level failures get separate handling from tool failures, because they mean different things. A 429 or a 500 from Groq's API is transient: retrying with backoff usually fixes it. A 401 means your API key is wrong, and retrying does nothing but waste a request:

async callModel(messages, attempt = 0) {
  const maxRetries = 3;
  try {
    return await this.client.chat.completions.create({
      model: this.model,
      max_completion_tokens: this.maxTokens,
      messages: [{ role: "system", content: this.systemPrompt }, ...messages],
      tools: this.toolDefinitions(),
      tool_choice: "auto"
    });
  } catch (err) {
    const status = err?.status;
    const retryable = RETRYABLE_STATUS.has(status); // 408, 409, 429, 500, 502, 503, 504
    if (!retryable || attempt >= maxRetries - 1) throw err;
    const delayMs = 500 * 2 ** attempt;
    this.logger.warn?.(`[agent] API call failed (status ${status}), retrying in ${delayMs}ms`);
    await sleep(delayMs);
    return this.callModel(messages, attempt + 1);
  }
}

One more Groq-specific detail: the groq-sdk client retries some of these statuses on its own, twice by default. The client here is constructed with maxRetries: 0 so the SDK's built-in retry doesn't stack on top of this one. Without that, a single rate-limited call could silently balloon to nine real HTTP attempts (three of mine, each retried three times by the SDK) instead of three.

Wiring it up

src/server.js exposes the agent as POST /api/agent/chat, rate-limited with express-rate-limit at a level that sits comfortably under Groq's free-tier cap even with a few concurrent users. Sessions live in an in-memory Map for the demo — fine for trying this out, not fine once you have more than one server process, at which point Redis is the obvious swap.

src/cli.js is the same agent wired to a terminal readline loop instead, useful for testing prompts and tool behavior interactively without standing up a server.

Both read a GROQ_API_KEY from the environment and default to openai/gpt-oss-120b as the model, overridable with GROQ_MODEL. Get a key at console.groq.com/keys — no credit card needed.

Trying it over HTTP

Here's what hitting the running server looks like. The JSON envelope below is exact: those are the literal field names server.js returns, but the reply text, steps count, and usage numbers will vary somewhat each time you run this, since it's a live model and not a fixture.

A normal order lookup:

curl -X POST http://localhost:3000/api/agent/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "What is the status of ORD-1001?"}'

{
  "sessionId": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "reply": "Order ORD-1001 has shipped via UPS, tracking number 1Z999AA10123456784. It's estimated to arrive July 2, 2026.",
  "steps": 2,
  "truncated": false,
  "usage": { "inputTokens": 612, "outputTokens": 47 }
}

steps: 2 means the model called a tool once, then answered — exactly the loop from earlier. Now the same question about the order rigged to fail, reusing the sessionId from above to stay in the same conversation:

curl -X POST http://localhost:3000/api/agent/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Can you check ORD-FAIL for me?", "sessionId": "a1b2c3d4-5678-90ab-cdef-1234567890ab"}'

{
  "sessionId": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "reply": "I wasn't able to look up that order just now — the lookup service timed out. Could you try again in a moment?",
  "steps": 2,
  "truncated": false,
  "usage": { "inputTokens": 701, "outputTokens": 39 }
}

That's the tool-error path from earlier, end to end: the get_order_status handler threw, executeTool caught it, the model got the failure as a normal message instead of a crash, and it answered like a person would — not a stack trace in sight. And the multi-tool case, in one request:

curl -X POST http://localhost:3000/api/agent/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "What is the status of ORD-1001, and if I return it how much of the $89.99 do I get back after a 15% restocking fee?"}'

{
  "sessionId": "f0e1d2c3-4b5a-69d8-c7e6-0a1b2c3d4e5f",
  "reply": "ORD-1001 has shipped via UPS and should arrive by July 2, 2026. If you return it, you'd get back $76.49 after the 15% restocking fee.",
  "steps": 3,
  "truncated": false,
  "usage": { "inputTokens": 845, "outputTokens": 61 }
}

steps: 3 here — one call to get_order_status, one to calculate, then the final answer combining both. That $76.49 isn't the model doing arithmetic in its head: calculate actually runs 89.99 * 0.85 through the real parser from earlier (which returns 76.49149999999999, full floating-point precision and all), and the model rounds that to cents for a reply a person can read.

Testing it without spending a request

The agent loop takes a client in its constructor. In production that's a real Groq instance. In tests, it's a plain object with a scripted chat.completions.create() that returns whatever response you tell it to:

test("handles malformed tool-call arguments from the model without crashing", async () => {
  const client = makeScriptedClient([
    {
      choices: [{
        finish_reason: "tool_calls",
        message: {
          role: "assistant",
          content: null,
          tool_calls: [{ id: "call_3", type: "function", function: { name: "calculate", arguments: "{not json" } }]
        }
      }],
      usage: { prompt_tokens: 10, completion_tokens: 5 }
    },
    {
      choices: [{ finish_reason: "stop", message: { role: "assistant", content: "Sorry, something went wrong." } }],
      usage: { prompt_tokens: 12, completion_tokens: 6 }
    }
  ]);

  const agent = new Agent({
    client,
    model: "openai/gpt-oss-120b",
    tools: [calculatorStub],
    systemPrompt: "test",
    logger: silentLogger
  });

  const result = await agent.run("Send a malformed tool call");

  const toolCallStep = result.trace.find((step) => step.type === "tool_call");
  assert.equal(toolCallStep.isError, true);
  assert.match(toolCallStep.result, /invalid JSON arguments/);
});

That test exists because it's a real failure mode in this format, not a hypothetical one: Groq's models occasionally send back arguments that don't parse cleanly, and the only way to find that out before a user does is to write the test that assumes it'll happen.

The full suite (21 tests across the loop, the calculator parser, and the tools) runs on Node's built-in test runner — no Jest, no extra dependency:

npm test

What the mocked tests actually catch

Worth being precise about what's verified here and what isn't. The 21 tests run against a scripted mock client, so they prove the loop's behavior: a 429 gets retried with backoff, a 401 doesn't get retried at all, a tool that throws gets turned into a clean error message instead of crashing the process, and malformed tool arguments don't take down the server either.

None of that touches Groq's actual servers, and it doesn't need to — that's the point of mocking the client. What it can't tell you is whether your specific model picks the right tool for a given prompt, or how it behaves under real latency.

For that, the repo ships a small smoke-test script. Start the server, then run it against a live key:

npm start                          # terminal 1
bash scripts/live-smoke-test.sh    # terminal 2

It runs nine checks in sequence: a normal order lookup, an unknown order ID, the simulated outage on ORD-FAIL, the calculator, the FAQ search, one turn that needs two tools at once, a follow-up that checks session memory, an invalid request, and a burst of twelve rapid-fire calls to confirm the local rate limiter kicks in. Watching that run once tells you more about how your model handles tool selection than any number of mocked tests can. On Groq's free tier, it costs nothing but a couple of minutes.

If you'd rather watch it think in real time, npm run cli -- --verbose prints every tool call as it happens, which is the faster way to see why the model reached for a given tool.

Taking it further

This is a teaching example. Before anything like it goes near real users:

Move sessions from the in-memory Map to Redis or a database
Add auth in front of the chat endpoint: right now anyone who can reach the port can spend your rate limit
Watch for 429s under real load: the free tier's per-minute cap is easy to hit with more than a couple of concurrent users, and Groq's Developer tier (still no minimum spend) raises that ceiling
Trim or summarize long conversation history before it eats your context window
Replace the mock order and FAQ data with real lookups

Get the code

Full source, with the test suite and a setup guide, is on GitHub: https://github.com/zyvop27-cmyk/zyvop-blogs/tree/main/ai-agent-node

Clone it, drop in your own GROQ_API_KEY, and ask it about an order. The agent loop, the tools, and the error handling are all small enough to read in one sitting — which is the point.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

AI Fractures: Edge Computing, Regulation, and the Rise of Decentralized Models | The AI Daily Roundup

ZyVOP — Mon, 29 Jun 2026 14:39:08 +0000

One Unifying Trend: AI Is Fragmenting From Centralized Clouds to Edge‑Centric, Locally‑Controlled Systems

Across the day’s headlines—from speculative decoding research to Asian firms releasing “Mythos‑like” models, from Ford’s AI‑driven quality fiasco to open‑source routing tools—the common thread is a clear shift away from monolithic, cloud‑only AI deployments. Companies, governments, and developers are building or demanding ways to run large‑scale models locally, on‑prem, or in regional data centers to sidestep regulation, cut latency, and regain reliability.

Why This Matters

Running inference at the edge reduces exposure to export bans, data‑privacy mandates, and single‑point‑of‑failure outages. It also re‑opens the economics of AI: hardware vendors can sell accelerators, startups can monetize niche models without cloud fees, and enterprises can avoid costly AI‑related recalls.

Technical Edge‑Optimizations Fueling the Shift

Speculative decoding (DSpark) – DeepSpec’s full‑stack codebase shows how speculative decoding can cut LLM latency by up to 2× without extra hardware, making on‑device inference viable.
Deterministic routing (Wayfinder Router) – The CLI tool lets developers route prompts between local and hosted models based on complexity, ensuring that cheap local models handle routine queries while only the most demanding calls hit expensive clouds.
Linux PSI‑based KV cache trimming (KV‑psi) – By leveraging Pressure Stall Information to prune LLM caches under memory pressure, developers can squeeze larger models onto edge devices like Jetson Orin, extending the reach of generative AI to robotics and IoT.
AI‑designed RFIC chips – Princeton’s diffusion‑driven chip design (IEEE Spectrum) demonstrates that AI can accelerate hardware creation, lowering the barrier for edge‑centric wireless solutions needed for 5G, autonomous vehicles, and satellite links.

Geopolitical & Regulatory Forces Accelerating Decentralization

The U.S. export ban on Anthropic’s Mythos and Fable models has created a vacuum that Asian startups are eager to fill. 360’s Tulongfeng and Sakana AI’s Fugu both claim “frontier capability without export‑control risk,” positioning themselves as the go‑to providers for non‑U.S. customers.

Anthropic’s accusation that Alibaba used 25 000 accounts to mine Claude (Ars Technica) underscores how state‑backed actors are willing to bypass restrictions, further incentivizing locally‑hosted alternatives.

Meanwhile, The Algorithmic Bridge argues that U.S. government control is reshaping the entire AI ecosystem, effectively “killing” the previous model of globally shared, cloud‑first AI services.

Enterprise Reliability & Ethical Backlash

Ford’s costly AI‑driven quality‑control experiment (The Independent) illustrates the operational risk of over‑relying on centralized AI without human expertise. Re‑hiring veteran engineers restored quality, proving that hybrid models—human plus edge‑deployed AI—remain essential.

On the ethical front, Hasbro’s Peppa Pig voice‑cloning clause (Gadget Review) sparked nearly 1 000 objections, highlighting the need for clear ownership and governance when AI reproduces personal data. Decentralized deployment can help enforce regional privacy rules, but it also complicates enforcement.

Who Wins, Who Loses

Winners: Asian AI startups, edge‑hardware vendors, open‑source communities, enterprises that need low‑latency, compliant AI, and developers who can monetize locally‑hosted models.
Losers: U.S. cloud‑centric AI giants losing market share, large‑scale data‑center providers facing reduced demand, and workers displaced by premature AI automation (as Ford’s case shows).

What Changes Next

Expect a rapid proliferation of open‑source inference stacks that combine speculative decoding, deterministic routing, and memory‑aware cache management. Parallelly, regional regulatory bodies will likely codify “AI‑localization” requirements, prompting more startups to ship models pre‑trained for specific jurisdictions. Enterprises will adopt hybrid pipelines: edge inference for routine tasks, cloud for rare, compute‑heavy queries, all under tighter human oversight.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Implementing Two-Factor Authentication (TOTP) in NestJS: With Full Source Code

ZyVOP — Mon, 29 Jun 2026 02:37:53 +0000

Password-only auth has one structural problem: a leaked password is a leaked account. Two-factor authentication (2FA) closes most of that gap by requiring a second, time-bound proof of identity — something the user has (an authenticator app) rather than just something they know (a password).

This post walks through a complete TOTP (Time-based One-Time Password) implementation in NestJS: generating secrets, rendering them as QR codes, verifying codes with clock-drift tolerance, issuing backup codes, and gating routes correctly based on 2FA status. The full project — tested end-to-end against a real Postgres instance — is linked at the end.

How TOTP actually works

TOTP is defined in RFC 6238. The mechanics are simple:

The server generates a random secret and shares it with the user once, usually via QR code.
Both the server and the authenticator app derive a 6-digit code from HMAC(secret, current_30s_time_window).
Since both sides know the secret and the time, they should compute the same code — no network round-trip required.

The only real failure mode is clock drift between the server and the user's device, which is why a well-built verifier checks a small window of adjacent time steps rather than an exact match.

Architecture of the flow

This implementation uses two JWT states rather than a separate "2FA session" table:

A partial token (isSecondFactorAuthenticated: false) — issued right after password login, for accounts that already have 2FA enabled. It's accepted by exactly one route: /2fa/authenticate.
A full token (isSecondFactorAuthenticated: true) — issued once the second factor is verified, or immediately for accounts that don't have 2FA enabled at all.

Two guards enforce this:

// jwt-auth.guard.ts — accepts ANY valid token, partial or full.
// Should ONLY ever guard /2fa/authenticate — see why below.
@Injectable()
export class JwtAuthGuard extends AuthGuard('jwt') {}

// jwt-two-factor.guard.ts — requires a FULLY authenticated session
@Injectable()
export class JwtTwoFactorGuard extends AuthGuard('jwt') {
  async canActivate(context: ExecutionContext): Promise<boolean> {
    const canActivate = (await super.canActivate(context)) as boolean;
    if (!canActivate) return false;

    const request = context.switchToHttp().getRequest<AuthenticatedRequest>();
    if (!request.user.isSecondFactorAuthenticated) {
      throw new UnauthorizedException('Two-factor authentication required');
    }
    return true;
  }
}

Here's the part that's easy to get wrong: it's tempting to guard /2fa/generate and /2fa/turn-on with JwtAuthGuard too, on the reasoning that a brand-new user "isn't fully 2FA'd yet either." Don't — a partial token is only ever issued for accounts that already have 2FA enabled. If those two routes accepted one, anyone holding just a stolen password could log in, immediately call /2fa/generate to overwrite the account's TOTP secret with one they control, confirm it via /2fa/turn-on, and walk away having fully hijacked 2FA without ever knowing the real second factor. A brand-new user isn't affected by locking these routes down, either — isTwoFactorEnabled is still false for them at that point, so login already handed them a full token.

So the actual guard assignment is:

JwtTwoFactorGuard → /2fa/generate, /2fa/turn-on, /2fa/turn-off, and any other protected route.
JwtAuthGuard → /2fa/authenticate only. That's the single route a partial token should ever be able to reach.

Data model

@Entity()
export class User {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column({ unique: true })
  email: string;

  @Column({ select: false })
  password: string;

  @Column({ default: false })
  isTwoFactorEnabled: boolean;

  // Unconfirmed until verified via /auth/2fa/turn-on
  @Column({ type: 'varchar', nullable: true, select: false })
  twoFactorSecret: string | null;

  // Bcrypt-hashed, single-use recovery codes
  @Column('text', { array: true, nullable: true, select: false })
  twoFactorBackupCodes: string[] | null;
}

select: false on the password, secret, and backup codes keeps them out of any default find() call — they only come back when explicitly requested via addSelect(). That's a cheap insurance policy against accidentally serializing a secret into an API response somewhere down the line.

Generating the secret and QR code

This uses otpauth rather than the more commonly referenced otplib/speakeasy — both of those are either deprecated or have had little recent maintenance. otpauth has no native dependencies and a clean, synchronous API.

async generateSecret(userId: string, email: string) {
  const secret = new Secret({ size: 20 });
  await this.usersService.setTwoFactorSecret(userId, secret.base32);

  const totp = this.buildTotp(secret.base32, email);
  const otpAuthUrl = totp.toString(); // otpauth://totp/...
  const qrCodeDataUrl = await QRCode.toDataURL(otpAuthUrl);

  return { qrCodeDataUrl, otpAuthUrl };
}

The secret is persisted as soon as it's generated, before the user has confirmed anything. That's intentional — it lets /2fa/turn-on verify against it on the next request rather than threading the secret through the client and back, which would mean trusting the client to round-trip something sensitive correctly.

QRCode.toDataURL() turns the otpauth:// URI directly into a base64 PNG, which any authenticator app (Google Authenticator, Authy, 1Password, etc.) can scan without any custom parsing on your end.

This project is API-only, so it's worth being explicit about how a real frontend turns that response into something a user actually sees. Browsers render data:image/...;base64,... URLs natively, so there's no decoding step — the response body can go straight into an <img> tag:

const { qrCodeDataUrl } = await fetch('/auth/2fa/generate', {
  method: 'POST',
  headers: { Authorization: `Bearer ${accessToken}` },
}).then((res) => res.json());

return <img src={qrCodeDataUrl} alt="Scan with your authenticator app" />;

That's the entire client-side integration. No QR-rendering library needed on the frontend — the server already did that work.

Verifying a code

verifyTotpToken(token: string, base32Secret: string, email: string): boolean {
  const totp = this.buildTotp(base32Secret, email);
  const delta = totp.validate({ token, window: 1 });
  return delta !== null;
}

window: 1 checks one 30-second step before and after the current one — three valid codes at any given moment instead of one. That's enough slack for typical clock drift without meaningfully widening the brute-force surface (going from 1-in-a-million to 3-in-a-million per guess is not the part doing the security work here — rate-limiting /2fa/authenticate is, and it should be in place regardless of window size).

Turning 2FA on — and generating backup codes

A code generated from a freshly scanned QR code is the only proof that the secret actually reached the user's device correctly. Skip this check and you risk locking someone out the moment a QR code fails to scan properly.

@UseGuards(JwtTwoFactorGuard)
@Post('2fa/turn-on')
async turnOn(@Req() req: AuthenticatedRequest, @Body() dto: TwoFactorAuthCodeDto) {
  const user = await this.usersService.findByIdWithSecrets(req.user.sub);
  if (!user?.twoFactorSecret) {
    throw new BadRequestException('Call /auth/2fa/generate first');
  }

  const isValid = this.twoFactorAuthService.verifyTotpToken(
    dto.twoFactorAuthCode, user.twoFactorSecret, user.email,
  );
  if (!isValid) throw new UnauthorizedException('Invalid authenticator code');

  const backupCodes = this.twoFactorAuthService.generateBackupCodes();
  const hashedBackupCodes = await this.twoFactorAuthService.hashBackupCodes(backupCodes);
  await this.usersService.enableTwoFactor(user.id, hashedBackupCodes);

  return {
    message: '2FA enabled. Store these backup codes safely — they will not be shown again.',
    backupCodes, // plaintext, shown exactly once
  };
}

Backup codes solve the "I lost my phone" problem. They're generated once, hashed with bcryptjs before storage (the same way a password would be), and shown to the user in plaintext exactly one time. After this response, the server only ever holds the hashes. (bcryptjs rather than bcrypt deliberately — same hashing algorithm, but pure JavaScript with no native compilation step, which matters if you've ever had node-gyp fail a CI build over a missing system header.)

generateBackupCodes(count = 8): string[] {
  return Array.from({ length: count }, () => crypto.randomBytes(5).toString('hex'));
}

Disabling 2FA is the mirror image of turning it on, and deliberately just as strict — it also requires a full token plus a current valid code, not just a click:

@UseGuards(JwtTwoFactorGuard)
@Post('2fa/turn-off')
async turnOff(@Req() req: AuthenticatedRequest, @Body() dto: TwoFactorAuthCodeDto) {
  const user = await this.usersService.findByIdWithSecrets(req.user.sub);
  if (!user?.twoFactorSecret) {
    throw new BadRequestException('Two-factor authentication is not enabled');
  }

  const isValid = this.twoFactorAuthService.verifyTotpToken(
    dto.twoFactorAuthCode, user.twoFactorSecret, user.email,
  );
  if (!isValid) throw new UnauthorizedException('Invalid authenticator code');

  await this.usersService.disableTwoFactor(user.id);
  return { message: '2FA disabled' };
}

Authenticating with a TOTP or backup code

@UseGuards(JwtAuthGuard)
@Post('2fa/authenticate')
async authenticate(@Req() req: AuthenticatedRequest, @Body() dto: TwoFactorAuthCodeDto) {
  const user = await this.usersService.findByIdWithSecrets(req.user.sub);
  if (!user?.isTwoFactorEnabled || !user.twoFactorSecret) {
    throw new BadRequestException('Two-factor authentication is not enabled for this account');
  }

  let isValid = this.twoFactorAuthService.verifyTotpToken(
    dto.twoFactorAuthCode, user.twoFactorSecret, user.email,
  );

  if (!isValid && user.twoFactorBackupCodes?.length) {
    isValid = await this.twoFactorAuthService.verifyAndConsumeBackupCode(
      user.id, dto.twoFactorAuthCode, user.twoFactorBackupCodes,
    );
  }

  if (!isValid) throw new UnauthorizedException('Invalid authentication code');

  return { accessToken: this.authService.issueToken(user, true) };
}

The fallback to backup codes only fires if the TOTP check fails — a live code from the authenticator app is always tried first, since that's the common case and avoids an unnecessary bcrypt comparison loop. verifyAndConsumeBackupCode removes the matched code from the array immediately on use:

async verifyAndConsumeBackupCode(userId: string, code: string, hashedCodes: string[]) {
  for (let i = 0; i < hashedCodes.length; i++) {
    if (await bcrypt.compare(code, hashedCodes[i])) {
      const remaining = [...hashedCodes];
      remaining.splice(i, 1);
      await this.usersService.updateBackupCodes(userId, remaining);
      return true;
    }
  }
  return false;
}

Setting up and running the project

Clone the repo, install dependencies, and copy the environment template:

git clone <your-repo-url>
cd 2fa-nestjs-demo
npm install
cp .env.example .env

Open .env and set JWT_SECRET to a long random string, plus your Postgres credentials. If you don't already have Postgres running locally, the fastest path is Docker:

docker run --name twofa-postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=twofa_demo \
  -p 5432:5432 -d postgres:16

Then start the API:

npm run start:dev

synchronize: true is enabled in app.module.ts for this demo, so the user table is created automatically on first boot — no manual migration step needed to follow along. (Turn that off and switch to real migrations before deploying anywhere real.)

With the server up on http://localhost:3000, you're ready to run through the flow below.

Testing it end-to-end

This is the actual sequence used to validate the implementation before publishing it — register, log in, enable 2FA, log in again, clear the second factor, hit a protected route, then disable 2FA. Every step below was run against a live server and a real Postgres instance.

# 1. Register
curl -X POST http://localhost:3000/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'
# -> { "id": "...", "email": "alice@example.com" }

# 2. Log in — 2FA isn't enabled yet, so this returns a full access token
curl -X POST http://localhost:3000/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'
# -> { "accessToken": "<token>", "twoFactorRequired": false }

# 3. Generate a 2FA secret + QR code (pass the token from step 2)
curl -X POST http://localhost:3000/auth/2fa/generate \
  -H "Authorization: Bearer <accessToken>"
# -> { "qrCodeDataUrl": "data:image/png;base64,..." }
# Paste the data URL into a browser address bar, or render it in an <img> tag,
# and scan it with Google Authenticator / Authy / 1Password / etc.

# 4. Confirm setup with a live code from the authenticator app
curl -X POST http://localhost:3000/auth/2fa/turn-on \
  -H "Authorization: Bearer <accessToken>" \
  -H "Content-Type: application/json" \
  -d '{"twoFactorAuthCode":"123456"}'
# -> { "message": "2FA enabled...", "backupCodes": ["a1b2c3d4e5", ...] }
# Show these to the user ONCE — they're not recoverable after this response.

# 5. Log in again — now a PARTIAL token comes back instead of a full one
curl -X POST http://localhost:3000/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'
# -> { "accessToken": "<partial token>", "twoFactorRequired": true }

# 6. That partial token alone isn't enough for a protected route
curl http://localhost:3000/auth/me -H "Authorization: Bearer <partialAccessToken>"
# -> 401 { "message": "Two-factor authentication required" }

# 7. A wrong code is rejected outright
curl -X POST http://localhost:3000/auth/2fa/authenticate \
  -H "Authorization: Bearer <partialAccessToken>" \
  -H "Content-Type: application/json" \
  -d '{"twoFactorAuthCode":"000000"}'
# -> 401 { "message": "Invalid authentication code" }

# 8. A valid TOTP code (or one of the backup codes from step 4) clears the second factor
curl -X POST http://localhost:3000/auth/2fa/authenticate \
  -H "Authorization: Bearer <partialAccessToken>" \
  -H "Content-Type: application/json" \
  -d '{"twoFactorAuthCode":"123456"}'
# -> { "accessToken": "<full token>" }

# 9. The full token now works on protected routes
curl http://localhost:3000/auth/me -H "Authorization: Bearer <fullAccessToken>"
# -> { "id": "...", "email": "alice@example.com" }

# 10. Disabling 2FA requires a full token AND a current valid code
curl -X POST http://localhost:3000/auth/2fa/turn-off \
  -H "Authorization: Bearer <fullAccessToken>" \
  -H "Content-Type: application/json" \
  -d '{"twoFactorAuthCode":"123456"}'
# -> { "message": "2FA disabled" }

One thing worth testing directly: re-submitting the same backup code from step 4 a second time. It should — and does — come back with the same 401 Invalid authentication code, since verifyAndConsumeBackupCode deletes a code from storage the moment it's used.

It's also worth confirming the boundary from the "Architecture" section explicitly, not just trusting the explanation — a partial token should be rejected by /2fa/generate too, the same way it was rejected by /auth/me in step 6:

curl -X POST http://localhost:3000/auth/2fa/generate -H "Authorization: Bearer <partialAccessToken>"
# -> 401 { "message": "Two-factor authentication required" }

That's the line that closes the privilege-escalation path: holding a stolen password gets you a partial token, and a partial token alone gets you nothing on that route — you have to clear /2fa/authenticate first.

Security notes before shipping this for real

A few things were deliberately simplified for clarity and are worth tightening before production use:

Encrypt the TOTP secret at rest with a KMS-backed key rather than relying on select: false alone — that flag protects against accidental leakage in queries, not a database compromise.
Rate-limit /auth/login and /auth/2fa/authenticate. Both are brute-force surfaces; @nestjs/throttler handles this with a few lines of config.
Give the partial token a short, distinct expiry — it represents an incomplete login and shouldn't live as long as a real session.
Require the current password again before disabling 2FA, not just a TOTP code, so a stolen-but-still-valid session token can't silently downgrade an account's security.
Never log the secret, TOTP code, or backup codes — not even in error traces.

TOTP is a solid, broadly compatible baseline. If you want to go further, WebAuthn/passkeys remove the phishability that TOTP still has (a user can still be tricked into typing a valid code into a fake login page) — but TOTP remains the most universally supported second factor today, and it's a meaningful security upgrade over passwords alone.

Source code

The complete, tested implementation — NestJS module, entities, guards, DTOs, and a README with the full setup and curl walkthrough — is available as a standalone repository: 2fa-nestjs-demo. Clone it, drop in your own Postgres credentials, and the registration → 2FA enrollment → login flow above works out of the box.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Features and Labels: Why Your Training Data Matters More Than Your Algorithm

ZyVOP — Sun, 28 Jun 2026 11:57:18 +0000

Previously in This Series

Day 2 sorted machine learning problems into four types by what's given to the algorithm. Today we zoom into the "given" part itself — the actual columns and answers sitting in your dataset — because most real ML failures trace back to something wrong there, not to the algorithm you picked.

Learning Objectives

Define what actually makes a feature useful, beyond "a column in a spreadsheet."
Spot feature leakage — a feature that secretly contains the answer — before it reaches production, not after.
See, with real numbers, why label quality can matter more than which algorithm you choose.
Build the habit of asking "would I actually have this information at prediction time?" before trusting any feature.

Why This Matters

Feature leakage is one of the most common, most expensive, and most embarrassing mistakes in real ML systems — models that look great in testing and then quietly fail, or get pulled from production entirely, because the feature that made them "great" was never actually available at decision time.

It's also a favorite senior-level interview question: "you're shown a model with suspiciously good performance — what's the first thing you check?" The expected answer is almost always leakage, not algorithm tuning.

And it sets up two things directly ahead in this series: Phase 3's Feature Engineering is the constructive half of what you're learning today, and Phase 5's production monitoring exists largely because leakage and label problems are so easy to miss until real money or real users are involved.

Mental Model

Think of training a model like building a court case.

Features are the evidence a detective would actually have access to at the time — fingerprints, timestamps, statements taken before the verdict was known.
A leaked feature is evidence that could only exist after the verdict — like a confession transcript that got mixed into the evidence pile by mistake. It makes the case look airtight in the file room, and falls apart the instant a real case lands on the desk without it.
Labels are witness testimony. Useful, but only as good as the witness. A model trained on confidently wrong testimony doesn't get suspicious about it — it just learns to be confidently wrong in exactly the same way.

History in 60 Seconds

1936 — Ronald Fisher uses linear discriminant analysis on the iris flower measurements (collected by botanist Edgar Anderson, and the same dataset from Day 2) — among the earliest formal uses of measured "features" to separate categories statistically.
1960s–1980s — early statistical and expert systems rely almost entirely on hand-picked features; the practitioner's judgment in choosing them matters as much as the model itself.
1997 — Tom Mitchell's Machine Learning textbook (referenced on Day 1) formalizes "feature" and "label" as the standard vocabulary still used today.
2011 — Kaufman, Rosset, and Perlich present "Leakage in Data Mining" at KDD, formalizing this lesson's central problem after watching it sink one data mining competition entry after another. An expanded journal version, adding co-author Ori Stitelman, followed in 2012 — the version cited in References below.
2010s — Kaggle competitions cement "feature engineering wins competitions" as conventional wisdom in classical ML, often more decisive than algorithm choice.
2012 onward — deep learning starts automating feature extraction for unstructured data like images and audio. For structured, tabular data — most of what businesses actually have — hand-built features remain decisive even now, which is exactly why this lesson isn't going away.

Key Terminology Table

Term	Meaning
Feature	A measurable input variable used to make a prediction
Label	The known correct answer attached to a training example
Feature leakage	A feature that, directly or indirectly, contains information about the label that wouldn't actually be available at prediction time
Label noise	Incorrect or inconsistent labels present in your training data
Ground truth	The real-world correct answer a label is supposed to represent
One-hot encoding	Converting a categorical feature into separate binary columns a model can use
Multicollinearity	When two or more features carry largely the same information, confusing how a model attributes importance between them
Feature engineering	Creating, transforming, or selecting features to improve what a model can learn

Core Concepts

What Actually Makes a Feature "Good"

A useful feature needs to clear four bars, and most feature mistakes are a failure on exactly one of them:

It has a genuine relationship with the target. If a column doesn't actually correlate with what you're predicting, it's just noise the model has to learn to ignore.
It's available at prediction time, in the same form. This is the bar feature leakage fails — more on that in a moment.
It actually varies. A column that's the same value for every row carries zero information, no matter how relevant it sounds conceptually.
It's not a near-duplicate of another feature. Two features that carry almost the same information create multicollinearity — a linear model can't cleanly tell which one deserves the credit, so it often splits weight between them in a way that hurts interpretability even when it doesn't hurt raw accuracy.

Feature Leakage: The Mistake That Hides Inside Good Numbers

Feature leakage happens when a feature contains information about the label that wouldn't actually exist yet at the moment you need to make a prediction. The test is simple to state and easy to forget under deadline pressure: would I have this exact value, in this exact form, at the moment I actually need to predict?

Real examples beyond today's hands-on demo:

Predicting loan default using a days_past_due field — a field that, by definition, only has a meaningful value once a payment is already late.
Predicting hospital readmission using a discharge_disposition field that gets recorded based on how the patient's case actually concluded.
Predicting customer churn using a cancellation_date column that's only populated after the customer has already churned.

In every case, the feature isn't "wrong" in the database — it's wrong for this prediction task, because it's only known after the answer is already settled.

Labels Are Data Too — and They're Often Wrong

Labels feel like ground truth because someone wrote them down with confidence. In practice, labels come from human annotators who disagree with each other, automated heuristics that get edge cases wrong, or definitions that quietly drift over time. All of that shows up as label noise: training examples where the recorded answer doesn't match reality.

Here's the part that's worth sitting with mathematically, briefly: if you flip a fraction p of your labels completely at random, the model is still learning from a signal that's correlated with the truth — just a weaker one — for any p below 0.5. As p approaches 0.5, that correlation collapses toward zero, because a coin flip is, by definition, equally likely to match the truth or not, regardless of what the truth actually is. That's not a smooth decline. It's a cliff, and you'll see exactly where it sits in today's experiment.

Visual Explanations

A quick litmus test for any feature you're about to add:

flowchart TD
    F[Candidate feature] --> Q1{Would this value be known,<br/>in this exact form,<br/>at prediction time?}
    Q1 -->|No| Leak[Leakage — fix the pipeline<br/>or drop the feature]
    Q1 -->|Yes| Q2{Does it just re-encode<br/>the label itself?}
    Q2 -->|Yes| Leak
    Q2 -->|No| Q3{Does it actually vary,<br/>and relate to the outcome?}
    Q3 -->|No| Weak[Low-value — consider dropping]
    Q3 -->|Yes| Good[Legitimate, useful feature]

And a few real leakage patterns, side by side:

Domain	Leaky feature	Why it leaks
Titanic survival (today's demo)	alive	Literally a re-encoding of the label
Loan default	days_past_due	Only exists once the loan is already in trouble
Hospital readmission	discharge_disposition	Recorded based on how the case actually concluded
Customer churn	cancellation_date	Only populated after the customer has already churned

Hands-On Example

We'll use the same Titanic passenger dataset many of you have seen before, for a specific reason: it contains a real, built-in leakage trap, not a contrived one.

Train a model on legitimate features — class, sex, age, family size, fare, and embarkation port. All of these are genuinely known before the ship sank.
"Accidentally" add one extra column: alive. This field is just the word "yes" or "no" version of the label we're trying to predict. We'll watch what that does to accuracy, and talk about why that result should worry you, not impress you.
Deliberately corrupt a percentage of the training labels and watch how accuracy degrades as that percentage climbs — using the legitimate features only, so we isolate label quality as the one variable changing.

Environment Setup

python -m venv venv

source venv/bin/activate
# Windows
venv\Scripts\activate

pip install numpy pandas seaborn scikit-learn

Note: seaborn is used here purely as a convenient, reliable loader for the Titanic dataset — we won't be making any plots with it.

Complete Working Code

Every number in the breakdown below came from actually running this script.

"""
day_03_features_and_labels.py
A real feature-leakage demonstration and a real label-noise experiment,
both run on the Titanic passenger dataset.
"""

import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

df = sns.load_dataset("titanic")

# --- Clean a legitimate, leakage-free feature set ---
df_clean = df.copy()
df_clean["age"] = df_clean["age"].fillna(df_clean["age"].median())
df_clean["embarked"] = df_clean["embarked"].fillna(df_clean["embarked"].mode()[0])
df_clean["sex"] = df_clean["sex"].map({"male": 0, "female": 1})
df_clean["embarked"] = df_clean["embarked"].map({"S": 0, "C": 1, "Q": 2})

legit_features = ["pclass", "sex", "age", "sibsp", "parch", "fare", "embarked"]
X_legit = df_clean[legit_features]
y = df_clean["survived"]

X_train, X_test, y_train, y_test = train_test_split(
    X_legit, y, test_size=0.3, random_state=42, stratify=y
)

# --- 1. Model trained on legitimate features only ---
model_legit = LogisticRegression(max_iter=500)
model_legit.fit(X_train, y_train)
legit_acc = accuracy_score(y_test, model_legit.predict(X_test))

print("=== LEGITIMATE FEATURES ===")
print(f"Test accuracy: {legit_acc:.3f}")

# --- 2. Same setup, but with a leaked feature added: 'alive' is 'survived' in disguise ---
df_leaky = df_clean.copy()
df_leaky["alive_encoded"] = df["alive"].map({"no": 0, "yes": 1})
leaky_features = legit_features + ["alive_encoded"]
X_leaky = df_leaky[leaky_features]

X_train_l, X_test_l, y_train_l, y_test_l = train_test_split(
    X_leaky, y, test_size=0.3, random_state=42, stratify=y
)
model_leaky = LogisticRegression(max_iter=500)
model_leaky.fit(X_train_l, y_train_l)
leaky_acc = accuracy_score(y_test_l, model_leaky.predict(X_test_l))

print("\n=== WITH LEAKED FEATURE ('alive') ===")
print(f"Test accuracy: {leaky_acc:.3f}")

# --- 3. Label noise experiment: corrupt a % of TRAINING labels, test labels stay clean ---
print("\n=== LABEL NOISE EXPERIMENT (legitimate features only) ===")
rng = np.random.RandomState(42)
for noise_level in [0.0, 0.1, 0.3, 0.5]:
    y_train_noisy = np.array(y_train, dtype=int).copy()
    n_flip = int(len(y_train_noisy) * noise_level)
    flip_idx = rng.choice(len(y_train_noisy), size=n_flip, replace=False)
    y_train_noisy[flip_idx] = 1 - y_train_noisy[flip_idx]

    m = LogisticRegression(max_iter=500)
    m.fit(X_train, y_train_noisy)
    acc = accuracy_score(y_test, m.predict(X_test))
    print(f"Training label noise: {int(noise_level * 100):>3}%  ->  Test accuracy: {acc:.3f}")

majority_baseline = max(y.mean(), 1 - y.mean())
print(f"\nFor reference, majority-class baseline accuracy: {majority_baseline:.3f}")

Running this prints:

=== LEGITIMATE FEATURES ===
Test accuracy: 0.799

=== WITH LEAKED FEATURE ('alive') ===
Test accuracy: 1.000

=== LABEL NOISE EXPERIMENT (legitimate features only) ===
Training label noise:   0%  ->  Test accuracy: 0.799
Training label noise:  10%  ->  Test accuracy: 0.791
Training label noise:  30%  ->  Test accuracy: 0.787
Training label noise:  50%  ->  Test accuracy: 0.369

For reference, majority-class baseline accuracy: 0.616

Code Breakdown

The legitimate model. Using only information genuinely available before the Titanic sank — class, sex, age, family counts, fare, embarkation port — logistic regression reaches 79.9% test accuracy. That's a real, respectable, very typical number for this dataset, and it's the honest baseline everything else gets compared against.

The leaked model. Adding one column, alive, pushes test accuracy to a perfect 1.000. Look back at the code: alive is built by mapping "yes"/"no" directly onto the same 1/0 values as the survived label. We didn't add a powerful new feature — we added the answer key, lightly disguised as a column. The model didn't get smarter. It just stopped having to predict anything.

This is the entire point of the leakage litmus test. If you saw 1.000 accuracy on a real project without knowing in advance which column was the trap, your honest reaction should be suspicion, not celebration.

The label noise experiment. Notice the shape of the degradation: from 0% to 30% noise, accuracy barely moves — 0.799 down to 0.787, a rounding error in practical terms. Then, at 50% noise, accuracy collapses to 0.369 — worse than the 0.616 you'd get by just guessing the majority class every time. That's the cliff predicted by the math in Core Concepts: at exactly 50% random label corruption, the training labels carry essentially zero usable signal, and the specific corrupted pattern the model latched onto here happened to actively point it in the wrong direction on the clean test set. The lesson isn't "noise is fine up to 30% and then it's not" — it's that label noise doesn't degrade performance smoothly, and you can't assume you're safely far from the cliff just because moderate noise hasn't hurt you yet.

Common Mistakes

Treating high accuracy as automatically good news. Suspiciously perfect performance is a bug report waiting to be read, not a result to ship.
Including post-outcome fields by accident. Status fields, timestamps, and flags that get updated after the event you're predicting are leakage's favorite hiding spot.
Trusting labels as ground truth without checking their source. A label generated by an inconsistent heuristic or a disagreeing pair of human annotators is not the same thing as reality.
Testing on labels that are just as noisy as your training labels. If you can't trust your test set, you can't tell real model degradation from measurement noise in your evaluation itself.
Dropping a "weak" feature too early. A feature with a weak standalone correlation can still meaningfully help a model in combination with others — check feature value in context, not in isolation, before discarding it.
Forgetting that one-hot encoded categories need to handle values the model has never seen. A category that shows up at serving time but never appeared in training will break a naively encoded pipeline.

Best Practices

For every candidate feature, ask explicitly: would I have this value, in this exact form, at the moment I actually need to predict? Make this a literal checklist item in code review, not just a mental gut check.
Treat suspiciously perfect performance as something to investigate before you celebrate it — and investigate by removing features one at a time, not by assuming the model "got lucky."
Treat label collection as a first-class engineering problem. If humans are labeling, measure inter-annotator agreement; if a heuristic is labeling, audit its edge cases the same way you'd audit code.
When you suspect your main labels are noisy, hold out a small, especially carefully verified "gold" validation set, so you have at least one trustworthy yardstick.

Production Perspective

Feature leakage in production most often comes from offline/online skew: a feature computed correctly in a batch training pipeline — with the benefit of full hindsight — simply isn't available, or is computed differently, in the live serving pipeline. This is one of the single most common real-world causes of "the model worked perfectly in testing and then failed in production."

Label quality in production has its own twist: labels frequently come from delayed business outcomes. Did the customer actually churn? You might not know for 90 days. That creates a permanent tension between training on the freshest data available and training on the most reliably labeled data available — and most production teams end up explicitly trading one off against the other.

Companies build specific monitoring around exactly these failure modes: training-serving skew detection, feature drift monitoring, and "this performance metric looks too good" alerts exist because the problems in this article are common enough to deserve their own dashboards.

Cost-wise, this is one of the most expensive mistake categories in ML. A leakage bug caught after deployment can cost far more than the entire original model build — in re-engineering, in lost trust, and in regulated domains like lending, sometimes in compliance exposure as well.

Real-World Applications

Credit and loan underwriting, where regulation often explicitly requires excluding any information that wasn't available at the moment of the lending decision.
Healthcare readmission and risk prediction, where outcome-adjacent fields are everywhere in the raw data and easy to include by mistake.
Customer churn prediction, where churn-adjacent timestamps are some of the most common leakage sources in practice.
Fraud detection, where the exact sequence and timing of events — not just their presence — determines whether a feature is legitimate or leaked.
Ad click and ranking systems, where the constant engineering battle is making sure a feature available during offline training is actually computable, in the same form, in real-time serving.

Interview Questions

1. What is feature leakage, and how would you detect it in a model that's performing suspiciously well? Feature leakage is a feature that contains information about the label which wouldn't actually be available at prediction time. To detect it, investigate unusually high performance by removing features one at a time and checking which one is responsible for the jump, and explicitly audit any feature for whether it could only exist after the outcome occurred.

2. Why might a feature that's legitimate during training become invalid at serving time? Training data is often built with full hindsight from a batch pipeline, while serving happens in real time with only the information available at that exact moment — a mismatch known as training-serving skew, and a major real-world source of leakage that isn't even visible in the training data itself.

3. How does label noise differ from feature noise, and which one usually hurts more? Feature noise affects the inputs; label noise affects the answers the model is trying to match. Label noise is generally more damaging, because a model has no way to distinguish a wrong answer from a right one — it will confidently learn whatever pattern, including a wrong one, that the noisy labels suggest.

4. If you're told a model gets 99.9% accuracy, what's the first thing you'd check? Whether any feature could be leaking the label, directly or indirectly — followed by checking whether the dataset has severe class imbalance that makes a trivial majority-class prediction look deceptively strong.

5. Give an example of multicollinearity, and explain why it's a problem even when it doesn't hurt accuracy. Two features like "age in years" and "birth year" carry almost the same information. A linear model may split importance between them somewhat arbitrarily, which doesn't necessarily hurt raw accuracy but does hurt interpretability — you can no longer trust the model's reported feature importances at face value.

6. How would you design a labeling process to minimize label noise from human annotators? Have multiple annotators label a sample of the same examples, measure their agreement rate, write clearer labeling guidelines for the cases where they disagree most, and treat persistent low-agreement categories as a signal that the task itself may need to be redefined.

7. What is training-serving skew, and how does it relate to feature leakage? It's a mismatch between how a feature is computed during training versus during live serving. It's closely related to leakage because the training-time version of a feature can accidentally include information — like later timestamps or aggregated future data — that the serving-time version structurally can't have.

8. Why might you intentionally keep a "weak" feature in a model rather than dropping it? Because standalone correlation with the target isn't the only way a feature contributes — it might interact with other features to reveal a pattern that neither feature shows on its own, something a quick univariate check would miss entirely.

Self-Assessment

Can you explain to someone non-technical why a model that's 100% accurate should worry you more than reassure you?
Could you turn the "would I have this at prediction time?" test into an actual checklist for a project you're working on?
Can you explain, in your own words, why 10% label noise and 50% label noise don't degrade a model by proportional amounts?
Can you name one feature, in any dataset you've worked with, that might secretly be leaking the answer?

Portfolio Challenges

Beginner Challenge

Using the Breast Cancer dataset (sklearn.datasets.load_breast_cancer), deliberately construct an obviously leaky derived feature from the diagnosis label, train a model with and without it, and report both accuracies — mirroring today's alive demonstration on a different dataset.

Intermediate Challenge

Re-run today's label-noise experiment at finer granularity (0%, 5%, 10%, 15%, ..., 50%) and plot or tabulate the full accuracy curve. Identify, as precisely as your data allows, where the "cliff" actually begins for this dataset, and write two sentences on whether it matches the 0.5 threshold predicted in Core Concepts.

Advanced Challenge

Engineer two new, genuinely non-leaky features from the raw Titanic columns — for example, a family_size feature from sibsp + parch, or a title feature extracted from the passenger name field — and measure whether either one improves test accuracy over today's legitimate baseline of 0.799. Report your result honestly even if it doesn't improve things; a feature that doesn't help is still a useful, valid finding.

Advanced Insights

Today's lesson is the defensive half of a pair. Phase 3's Feature Engineering is the constructive half — deliberately building better signal instead of just avoiding broken signal. Both rest on the same foundation: understanding exactly what a feature represents and exactly when it becomes available.

It's also worth knowing this problem doesn't disappear once you move to deep learning. Representation learning automates a lot of feature creation, but a leaked signal can hide inside a learned embedding just as easily as inside a hand-built column — it's just harder to spot, because you can no longer point at a single suspicious column name. The thinking from this lesson applies just as much in Phase 4 and Phase 6 as it does here; only the hiding place changes.

Key Takeaways

A feature is only useful if it's predictive and genuinely available, in the same form, at prediction time — leakage is what happens when that second condition silently fails.
Suspiciously perfect model performance is a signal to investigate, not a result to celebrate.
Label noise doesn't degrade performance smoothly. Today's experiment found a real cliff at 50% random corruption, where accuracy fell below even the majority-class baseline.
Feature leakage and label quality usually matter more for real-world model performance than which algorithm you choose.

What's Next

Day 4 zooms out one more level: Data Quality — missing values, outliers, class imbalance, and the dataset-wide checks worth running before you train anything at all, building directly on today's column-by-column thinking.

References

Beginner

Google: Machine Learning Crash Course — includes a dedicated module on data quality and feature engineering fundamentals.
scikit-learn: User Guide — official documentation covering the preprocessing and encoding tools used in this article's code.

Intermediate

Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani, An Introduction to Statistical Learning — free at statlearning.com, with solid coverage of variable selection and the statistical risks of correlated predictors.

Professional

Aurélien Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (O'Reilly) — has a thorough, practical treatment of feature engineering and data preparation pipelines. Available through O'Reilly and major booksellers.

Advanced

Shachar Kaufman, Saharon Rosset, Claudia Perlich, and Ori Stitelman, "Leakage in Data Mining: Formulation, Detection, and Avoidance," ACM Transactions on Knowledge Discovery from Data, 2012 — the paper that formalized the exact problem at the center of today's lesson, based on real data mining competition failures. Official record: doi.org/10.1145/2382577.2382579.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Ford Tried to Automate Away Its Engineers. It Just Rehired 350 of Them.

ZyVOP — Sun, 28 Jun 2026 03:40:52 +0000

Ford just posted its best J.D. Power quality score in 16 years. Getting there meant undoing a decade of bets on automation.

Ford topped the 2026 Initial Quality Study as the No. 1 mainstream brand, beating Toyota and Honda outright. Only two luxury brands, Porsche and Genesis, scored higher across the entire industry.

A year earlier, Ford ranked 10th among mainstream brands, below the industry average. No mainstream brand improved more this year than Ford did.

The turnaround came from a place nobody expected: 350 veteran engineers Ford had to hire, rehire, or promote back into roles automation was supposed to handle.

What Ford actually got wrong

For the better part of a decade, Ford leaned hard on automation, AI-assisted design tools, and algorithmic decision-making to run vehicle development. The pitch was the standard one: let the systems handle scale and consistency, free up the humans for everything else.

Charles Poon, Ford's VP of vehicle hardware engineering, told reporters the company assumed feeding its design requirements into AI tools would be enough on its own to produce a high-quality vehicle. It wasn't.

The problem wasn't the AI itself. It was what the AI never learned.

As senior engineers left, retired, or were pushed out, what they carried in their heads left with them — which suppliers cut corners under deadline pressure, which tolerances had bitten Ford before, which shortcuts looked fine on paper and failed on the line. None of it lived anywhere an algorithm could read it. Ford's automated systems, trained on incomplete data, had no way to compensate.

Poon's point, stripped of the corporate phrasing: a model can't outperform the data it was trained on, and Ford hadn't been capturing what its most experienced people actually knew.

The timing made it worse

This wasn't AI failing in a vacuum. It hit during the troubled launches of the Explorer and Lincoln Aviator, pandemic-era supply chain chaos, and a recall count that's still the highest in the US auto industry.

Ford expects roughly $1 billion in warranty and materials costs this year alone. COO Kumar Galhotra has waved off the current recall numbers as a trailing signal rather than a live one — the fixes are already built into newer vehicles, the backlog just hasn't cleared the system yet.

Bringing the engineers back, specifically

Over the last three years, Ford hired, promoted, or rehired more than 350 veteran engineers. Many were former employees. Some came from suppliers.

Concretely, that means:

Run mandatory meetings that troubleshoot quality problems before they reach production
Reprogram Ford's automated quality-control tools directly, using knowledge those tools were never trained on
In Galhotra's words, they "hunt for failure points before a part ever reaches the plant floor"

Their job wasn't to replace the AI tools. Their job was to become the missing training data themselves — the person who remembers exactly why a coolant fitting got redesigned three programs ago, and catches the new supplier about to repeat the mistake.

AI didn't get fired. It got reassigned.

Here's the part that gets lost in the headline: Ford isn't walking away from AI. It's using the technology where it's genuinely strong, and pulling it back where it wasn't.

On the assembly line, Ford runs AI over the readings each engine produces during hot testing, watching for the kind of microscopic drift a human inspector would never catch by eye. A part can sit fully within spec and still get flagged, if it diverges even slightly from the pattern set by every engine built before it.

That's a narrow, data-saturated, well-defined problem. AI is excellent at exactly that.

On the software side, Ford built a dedicated 40-person quality assurance team over the last 18 months. In Poon's words, it's "a group that didn't even exist before."

Their job is catching defects before launch instead of patching them after a recall.

Backing that team: more than 100,000 new AI-generated tests built to stress software against edge cases automatically, so a late-stage code change gets fully revalidated instead of shipped on faith. Poon told TechSpot the company has treated software reliability as "its own rigorous discipline with strict metrics."

That's the split that held up: automation for scale and edge cases, humans for judgment and the failures nobody had written a test for yet.

The lesson, for anyone running an engineering org

Choosing AI wasn't the mistake. The mistake was assuming AI could absorb judgment nobody had ever written down, then letting the people who held that judgment leave before anyone captured it.

That gap doesn't show up on a roadmap. It shows up eighteen months later, as a recall, or a quiet regression nobody can quite explain.

Ford had the rare luxury of buying that knowledge back. The same engineers it had pushed out were mostly still reachable, still willing, still mostly intact in what they knew.

Most teams that make this trade don't get a clean rehire on the other side of it. Ford turned its gap into a 16-year-best quality score. Whether yours turns into a turnaround story or just the regression depends entirely on how early you notice the gap exists.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

DEV Community: ZyVOP

Brick: The LLM Router That Skips the Cascade and Still Cuts Your Bill

What Brick Actually Routes To

How the Routing Decision Actually Gets Made

Claude Code and Codex Get a Dedicated Integration

The Numbers, and What They're Actually Measuring

Trying It

Where This Project Actually Sits Right Now

Python Scraping at Scale: Distributed Crawling Across Multiple Machines (2026)

Introduction: When One Machine Is No Longer Enough

The Architecture: One Queue, Many Workers

Part 1: Setting Up scrapy-redis

The distributed spider

settings.py for distributed mode

Part 2: Production Middlewares

Part 3: MongoDB Pipeline with Bulk Writes

Part 4: Docker Compose — Multi-Worker Setup

Part 5: The Seeder — Feeding URLs Into the Queue

Part 6: Scaling to Kubernetes

Part 7: Proxy Management at Scale

Part 8: Monitoring Your Distributed Crawl

Part 9: Resumable Crawls

Performance Numbers: What to Expect

Summary

AI Hype Meets Reality: Security Risks, Thin ROI, and the Rise of Skepticism | The AI Daily Roundup

Trend Overview: From Hype to Hard Evidence

Why the Shift Matters

Evidence from the Day’s Stories

Security Backlash

Economic Reality Check

User Control & Transparency

Who Wins, Who Loses

What Changes Next?

Bottom Line

Passwordless Login With Magic Links in Node.js

The one decision that matters most

What the user actually sees

Sending the email without configuring SMTP

Sessions, cookies, and why SameSite: lax is correct here

The two routes that do the work

Running the whole thing

Tests

Before going live

Rate Limiting Alone Won't Stop a Patient Attacker

The throttle, and what it actually returns

Where the throttle's job ends

Locking the account, not the address

Two timers that look independent and aren't

The failure that triggers a lock doesn't announce it

Both mechanisms, same request

Running it

Background Jobs in NestJS with BullMQ: A Complete Walkthrough

The shape of the problem

Two sources of truth, on purpose

Wiring BullMQ into the Nest app

Enqueueing a job

The worker

The detail that's easy to get backwards: attemptsMade

What "idempotent" actually means here

Setting up and running the project

Testing it end-to-end

Production notes

Source code

AI’s Growing Pains: Compute Caps, Security Harnesses, and the Human Touch | The AI Daily Roundup

Connecting the Dots: AI Is Hitting Its Operational Limits

1. Compute Scarcity Becomes a Competitive Weapon

2. Security Harnesses Trump Raw Model Power

3. Human Expertise Remains the Safety Net

4. Macro‑Level Risks and Market Signals

5. Grassroots Tooling and the DIY Ethos

6. The Emerging Playbook for Leaders

Building a Production AI Agent in Node.js: Tool Calling, the ReAct Loop, and Error Handling

What "production" actually means here

Why Groq, and why no framework

Defining a tool

The loop itself

Wiring it up

Trying it over HTTP

Testing it without spending a request

What the mocked tests actually catch

The detail that's easy to get backwards: `attemptsMade`