DEV Community: Jaison Hernando Velasco Camacho

Building Real-Time Asynchronous Web Crawlers with Python Asyncio, Playwright, and Webhooks

Jaison Hernando Velasco Camacho — Tue, 04 Aug 2026 19:52:42 +0000

Building Real-Time Asynchronous Web Crawlers with Python Asyncio, Playwright, and Webhooks

Web scraping has evolved from fetching static HTML using requests and BeautifulSoup to handling dynamic JavaScript-rendered Single Page Applications (SPAs). Modern web platforms rely heavily on client-side rendering, WebSocket connections, and complex DOM manipulations.

In this comprehensive tutorial, we will build a production-ready, asynchronous web crawler using Python's asyncio, Microsoft's Playwright (async API), and an automated Webhook dispatch pipeline to stream data in real-time.

🏗️ Architecture Overview

Our crawler follows an asynchronous Producer-Consumer design pattern:

URL Queue (Producer): Manages target URLs with concurrency limits and deduplication.
Browser Pool (Worker): Spawns headless chromium instances via Playwright async context managers.
Data Extractor: Extracts structured metadata, dynamic DOM elements, and network metrics.
Webhook Dispatcher (Consumer): Posts JSON payloads asynchronously to a designated webhook endpoint as soon as a page is scraped.

       +--------------------+
       |   Target URLs      |
       +---------+----------+
                 |
                 v
       +--------------------+
       |   asyncio.Queue    |
       +---------+----------+
                 |
        +--------+--------+
        |                 |
        v                 v
 +--------------+  +--------------+
 | Worker 1     |  | Worker 2     |  (Playwright Async Pages)
 +------+-------+  +------+-------+
        |                 |
        +--------+--------+
                 |
                 v
       +--------------------+
       |  Webhook Dispatch  |  (HTTPX / Async POST)
       +--------------------+

🛠️ Prerequisites & Setup

Ensure you have Python 3.9+ installed. Install the required dependencies:

pip install playwright httpx pydantic
playwright install chromium

🐍 Complete Implementation Code (`crawler.py`)

Here is the complete, runnable asynchronous web crawler engine:

import asyncio
import logging
import time
from typing import Dict, Any, List, Optional
from pydantic import BaseModel, HttpUrl
import httpx
from playwright.async_api import async_playwright, BrowserContext, Page

# Configure Logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.StreamHandler()]
)

# Models
class ScrapeTarget(BaseModel):
    url: str
    depth: int = 0
    max_depth: int = 1

class ScrapedPayload(BaseModel):
    url: str
    title: str
    status_code: int
    content_length: int
    scraped_at: float
    links: List[str]

class AsyncCrawlerEngine:
    def __init__(self, webhook_url: Optional[str] = None, concurrency: int = 3):
        self.webhook_url = webhook_url
        self.concurrency = concurrency
        self.queue: asyncio.Queue[ScrapeTarget] = asyncio.Queue()
        self.visited: set = set()
        self.http_client = httpx.AsyncClient(timeout=10.0)

    async def send_webhook(self, payload: ScrapedPayload):
        """Dispatches real-time scraped data via Webhook POST request."""
        if not self.webhook_url:
            logging.info(f"[Payload Processed] {payload.url} -> Title: '{payload.title}'")
            return

        try:
            response = await self.http_client.post(
                self.webhook_url,
                json=payload.model_dump()
            )
            logging.info(f"[Webhook Delivered] {payload.url} -> HTTP {response.status_code}")
        except Exception as e:
            logging.error(f"[Webhook Failed] {payload.url}: {e}")

    async def scrape_page(self, context: BrowserContext, target: ScrapeTarget):
        """Scrapes a single page using Playwright async page API."""
        page: Page = await context.new_page()
        try:
            logging.info(f"[Scraping] {target.url}")
            response = await page.goto(target.url, wait_until="domcontentloaded", timeout=30000)

            title = await page.title()
            content = await page.content()
            status = response.status if response else 200

            # Extract internal/external links
            href_elements = await page.query_selector_all("a[href]")
            links = []
            for elem in href_elements:
                href = await elem.get_attribute("href")
                if href and href.startswith("http"):
                    links.append(href)

            payload = ScrapedPayload(
                url=target.url,
                title=title,
                status_code=status,
                content_length=len(content),
                scraped_at=time.time(),
                links=links[:10]  # Cap top 10 links
            )

            await self.send_webhook(payload)

            # Enqueue child links if depth allows
            if target.depth < target.max_depth:
                for link in links[:5]:
                    if link not in self.visited:
                        self.visited.add(link)
                        await self.queue.put(ScrapeTarget(url=link, depth=target.depth + 1, max_depth=target.max_depth))

        except Exception as err:
            logging.error(f"[Error Scraping] {target.url}: {err}")
        finally:
            await page.close()

    async def worker(self, context: BrowserContext):
        """Worker task consuming URLs from the asyncio queue."""
        while True:
            try:
                target = await asyncio.wait_for(self.queue.get(), timeout=3.0)
                await self.scrape_page(context, target)
                self.queue.task_done()
            except asyncio.TimeoutError:
                break
            except Exception as e:
                logging.error(f"[Worker Exception] {e}")
                self.queue.task_done()

    async def run(self, seed_urls: List[str]):
        """Main engine loop establishing Playwright browser pool."""
        for url in seed_urls:
            self.visited.add(url)
            await self.queue.put(ScrapeTarget(url=url, depth=0, max_depth=1))

        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=True)
            context = await browser.new_context(
                user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
            )

            workers = [asyncio.create_task(self.worker(context)) for _ in range(self.concurrency)]

            await self.queue.join()
            await asyncio.gather(*workers)

            await context.close()
            await browser.close()
            await self.http_client.aclose()

if __name__ == "__main__":
    seeds = [
        "https://news.ycombinator.com",
        "https://httpbin.org/html"
    ]
    # Replace with your webhook receiver URL (e.g. webhook.site)
    webhook_endpoint = "https://httpbin.org/post"

    engine = AsyncCrawlerEngine(webhook_url=webhook_endpoint, concurrency=2)
    asyncio.run(engine.run(seeds))

⚡ Key Technical Highlights

Non-blocking I/O: Leverages Python asyncio loop with Playwright's native asynchronous driver to handle high concurrency without blocking CPU cycles.
Headless Execution & Anti-Detection: Uses custom User-Agents and browser contexts to render dynamic JavaScript SPA content cleanly.
Real-time Webhook Streaming: Uses httpx.AsyncClient to fire non-blocking JSON payloads instantly as pages complete scraping, enabling downstream message brokers (e.g. Kafka, RabbitMQ) to consume crawl data without delay.
Queue-driven Scaling: Scalable worker pool managing concurrency and deduplication via asyncio.Queue and thread-safe sets.

🛡️ Production Best Practices

Rate Limiting & Delays: Introduce random asyncio.sleep(random.uniform(1.0, 3.0)) between navigation requests to respect target servers.
Proxy Rotation: Pass proxy={"server": "http://user:pass@proxy_host:port"} inside browser.new_context().
Resource Cleanup: Always use try/finally blocks or async context managers to close browser pages, avoiding memory leaks.

🎯 Conclusion

By combining Python's asyncio, Playwright's headless browser control, and real-time Webhook streaming, you can build modern web scrapers capable of extracting dynamic JavaScript content at scale.

Building Serverless API Gateways with Node.js Express and GitHub Actions CI/CD

Jaison Hernando Velasco Camacho — Tue, 04 Aug 2026 19:52:23 +0000

Building Serverless API Gateways with Node.js Express and GitHub Actions CI/CD

Microservices architectures often require a central API Gateway to manage authentication, rate limiting, logging, and routing to downstream services. Running a traditional, always-on API Gateway on EC2 or Docker containers can lead to idle compute costs and maintenance overhead.

By leveraging Serverless architecture (AWS Lambda & API Gateway) with Node.js Express and GitHub Actions, you can build a highly resilient, auto-scaling API Gateway that costs virtually zero when idle and automates deployments seamlessly.

In this tutorial, we will construct a production-ready Serverless API Gateway from scratch using Node.js Express, wrap it for serverless deployment with serverless-http, and establish a zero-downtime CI/CD pipeline via GitHub Actions.

🏗️ Architecture Overview

The API Gateway acts as the single point of entry for client requests:

[ Client Request ] 
       │
       ▼
[ AWS API Gateway / Lambda (Express.js Gateway) ]
       ├── 1. Security Headers (Helmet) & CORS
       ├── 2. Rate Limiting (express-rate-limit)
       ├── 3. JWT & API Key Authentication
       └── 4. Dynamic Proxy Dispatcher -> [ Downstream Services ]

Key Components:

Express.js: Core routing engine and middleware handling.
Serverless Framework (serverless-http): Adapts Express request/response objects into AWS Lambda event handlers.
GitHub Actions: Automated linting, testing, and deployment to AWS on code pushes.

📁 Project Structure

serverless-api-gateway/
├── .github/
│   └── workflows/
│       └── deploy.yml
├── src/
│   ├── middleware/
│   │   ├── auth.js
│   │   └── rateLimiter.js
│   ├── routes/
│   │   └── gateway.js
│   └── app.js
├── tests/
│   └── gateway.test.js
├── handler.js
├── serverless.yml
├── package.json
└── README.md

💻 1. Setting Up `package.json`

First, initialize the Node.js project and install the required dependencies:

{
  "name": "serverless-api-gateway",
  "version": "1.0.0",
  "description": "Production-ready Serverless API Gateway with Express.js and GitHub Actions",
  "main": "handler.js",
  "scripts": {
    "start": "serverless offline",
    "test": "jest --detectOpenHandles"
  },
  "dependencies": {
    "axios": "^1.6.8",
    "cors": "^2.8.5",
    "dotenv": "^16.4.5",
    "express": "^4.19.2",
    "express-rate-limit": "^7.2.0",
    "helmet": "^7.1.0",
    "jsonwebtoken": "^9.0.2",
    "serverless-http": "^3.2.0"
  },
  "devDependencies": {
    "jest": "^29.7.0",
    "serverless": "^3.38.0",
    "serverless-offline": "^13.3.3",
    "supertest": "^6.3.4"
  }
}

🔒 2. Authentication & Rate Limiting Middleware

Authentication Middleware (`src/middleware/auth.js`)

const jwt = require('jsonwebtoken');

const API_KEY_SECRET = process.env.API_KEY_SECRET || 'super-secret-api-key';
const JWT_SECRET = process.env.JWT_SECRET || 'super-secret-jwt-key';

function authenticateRequest(req, res, next) {
  const apiKey = req.headers['x-api-key'];
  const authHeader = req.headers['authorization'];

  // 1. API Key Validation
  if (apiKey && apiKey === API_KEY_SECRET) {
    req.user = { role: 'service-account' };
    return next();
  }

  // 2. JWT Bearer Token Validation
  if (authHeader && authHeader.startsWith('Bearer ')) {
    const token = authHeader.split(' ')[1];
    try {
      const decoded = jwt.verify(token, JWT_SECRET);
      req.user = decoded;
      return next();
    } catch (err) {
      return res.status(401).json({ error: 'Invalid or expired authentication token' });
    }
  }

  return res.status(401).json({ error: 'Unauthorized: Missing API Key or Bearer Token' });
}

module.exports = { authenticateRequest };

Rate Limiter Middleware (`src/middleware/rateLimiter.js`)

const rateLimit = require('express-rate-limit');

const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100, // Limit each IP to 100 requests per windowMs
  standardHeaders: true,
  legacyHeaders: false,
  message: {
    status: 429,
    error: 'Too many requests from this IP, please try again after 15 minutes.'
  }
});

module.exports = { apiLimiter };

🚀 3. Core Gateway Express App (`src/app.js`)

const express = require('express');
const cors = require('cors');
const helmet = require('helmet');
const { apiLimiter } = require('./middleware/rateLimiter');
const gatewayRoutes = require('./routes/gateway');

const app = express();

// Security and utility middleware
app.use(helmet());
app.use(cors());
app.use(express.json());

// Global Rate Limiter
app.use('/api', apiLimiter);

// Health Check Endpoint
app.get('/health', (req, res) => {
  res.status(200).json({
    status: 'healthy',
    timestamp: new Date().toISOString(),
    uptime: process.uptime()
  });
});

// Gateway Proxy Routes
app.use('/api/v1', gatewayRoutes);

// Fallback 404 handler
app.use((req, res) => {
  res.status(404).json({ error: 'Route not found on API Gateway' });
});

module.exports = app;

🔀 4. Routing & Proxying Engine (`src/routes/gateway.js`)

const express = require('express');
const axios = require('axios');
const { authenticateRequest } = require('../middleware/auth');

const router = express.Router();

// Downstream service endpoints registry
const SERVICES = {
  USERS: process.env.USERS_SERVICE_URL || 'https://jsonplaceholder.typicode.com/users',
  POSTS: process.env.POSTS_SERVICE_URL || 'https://jsonplaceholder.typicode.com/posts'
};

// Protected User Microservice Proxy
router.get('/users', authenticateRequest, async (req, res) => {
  try {
    const response = await axios.get(SERVICES.USERS, { timeout: 5000 });
    res.status(200).json({
      gateway: 'Serverless-Node-Gateway',
      service: 'users-microservice',
      data: response.data
    });
  } catch (error) {
    res.status(502).json({ error: 'Bad Gateway: Microservice response failed', details: error.message });
  }
});

// Protected Posts Microservice Proxy
router.get('/posts', authenticateRequest, async (req, res) => {
  try {
    const response = await axios.get(SERVICES.POSTS, { timeout: 5000 });
    res.status(200).json({
      gateway: 'Serverless-Node-Gateway',
      service: 'posts-microservice',
      data: response.data.slice(0, 5) // Return sample posts
    });
  } catch (error) {
    res.status(502).json({ error: 'Bad Gateway: Microservice response failed', details: error.message });
  }
});

module.exports = router;

⚡ 5. Serverless Handler (`handler.js`) & Config (`serverless.yml`)

Lambda Entrypoint (`handler.js`)

const serverless = require('serverless-http');
const app = require('./src/app');

module.exports.handler = serverless(app);

Serverless Framework Configuration (`serverless.yml`)

service: serverless-api-gateway

provider:
  name: aws
  runtime: nodejs18.x
  region: us-east-1
  memorySize: 256
  timeout: 10
  environment:
    API_KEY_SECRET: ${env:API_KEY_SECRET, 'super-secret-api-key'}
    JWT_SECRET: ${env:JWT_SECRET, 'super-secret-jwt-key'}

functions:
  api:
    handler: handler.handler
    events:
      - httpApi:
          path: '*'
          method: '*'

plugins:
  - serverless-offline

⚙️ 6. GitHub Actions CI/CD Pipeline (`.github/workflows/deploy.yml`)

Automate testing and deployment whenever changes are pushed to main:

name: Serverless API Gateway CI/CD Pipeline

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test-and-lint:
    name: Run Tests & Audit
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Node.js 18.x
        uses: actions/setup-node@v4
        with:
          node-version: 18.x
          cache: 'npm'

      - name: Install Dependencies
        run: npm ci

      - name: Run Unit Tests
        run: npm test

  deploy:
    name: Deploy to AWS Lambda
    needs: test-and-lint
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Node.js 18.x
        uses: actions/setup-node@v4
        with:
          node-version: 18.x

      - name: Install Dependencies
        run: npm ci

      - name: Deploy Serverless Service
        uses: serverless/github-action@v3.2
        with:
          args: deploy
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          API_KEY_SECRET: ${{ secrets.API_KEY_SECRET }}
          JWT_SECRET: ${{ secrets.JWT_SECRET }}

🧪 7. Automated Unit Testing with Jest & Supertest (`tests/gateway.test.js`)

const request = require('supertest');
const app = require('../src/app');

describe('Serverless API Gateway Endpoints', () => {
  it('GET /health - should return status 200 OK', async () => {
    const res = await request(app).get('/health');
    expect(res.statusCode).toEqual(200);
    expect(res.body).toHaveProperty('status', 'healthy');
  });

  it('GET /api/v1/users - should reject unauthorized requests with 401', async () => {
    const res = await request(app).get('/api/v1/users');
    expect(res.statusCode).toEqual(401);
  });

  it('GET /api/v1/users - should succeed with valid X-API-KEY header', async () => {
    const res = await request(app)
      .get('/api/v1/users')
      .set('x-api-key', 'super-secret-api-key');
    expect(res.statusCode).toEqual(200);
    expect(res.body).toHaveProperty('service', 'users-microservice');
  });
});

🎯 Conclusion

By wrapping Express.js with serverless-http and managing deployments via GitHub Actions, you achieve:

Zero Cold-Start Overhead for Light Proxies: Lightweight Node.js routing.
Built-in Protection: Helmet security headers, rate-limiting, and centralized JWT validation.
Automated CI/CD: Automatic test execution and deployment to AWS Lambda on every git push.

This setup offers enterprise-grade API management with serverless cost efficiency.

Building Resilient Serverless Web Scrapers and Automated Webhook Pipeline with Python and GitHub Actions

Jaison Hernando Velasco Camacho — Tue, 04 Aug 2026 16:03:34 +0000

In modern data-driven environments, collecting data consistently from web sources without paying for heavy server infrastructure is a major engineering advantage. Serverless architectures coupled with CI/CD platforms like GitHub Actions provide an elegant, cost-effective, and highly reliable way to run scheduled data scraping pipelines.

In this comprehensive tutorial, we will build an enterprise-grade, serverless web scraper and webhook notification pipeline using Python 3, BeautifulSoup4, Requests, and GitHub Actions.

Architecture Overview

Scraper Core (scraper.py): Fetches target web content using robust HTTP session retries, custom User-Agents, and rate-limiting.
Parser Module: Extracts structured data efficiently with BeautifulSoup, handling missing DOM nodes gracefully.
Webhook Notifier: Posts newly discovered or updated records to Discord/Slack/Custom Webhook endpoints via JSON payload.
CI/CD Serverless Runner (.github/workflows/scraper.yml): Executes on a scheduled cron trigger (e.g., hourly/daily) or manual dispatch, persisting log output and caching HTTP responses if necessary.

Core Python Scraper Engine (`scraper.py`)

Below is the complete, self-contained Python implementation:

import os
import sys
import time
import json
import logging
import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

# Setup Structured Logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.StreamHandler(sys.stdout)]
)
logger = logging.getLogger("ServerlessScraper")

def create_resilient_session(retries=3, backoff_factor=1.5, status_forcelist=(500, 502, 503, 504)):
    """Configures a requests Session with automatic exponential backoff retries."""
    session = requests.Session()
    retry_strategy = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        raise_on_status=False
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    session.headers.update({
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    })
    return session

def parse_target_data(html_content):
    """Parses target HTML with BeautifulSoup and extracts structured items."""
    soup = BeautifulSoup(html_content, "html.parser")
    extracted_items = []

    articles = soup.find_all(["article", "div"], class_=lambda c: c and any(term in str(c) for term in ["card", "item", "post"]))

    for idx, item in enumerate(articles[:10]):
        title_elem = item.find(["h1", "h2", "h3", "a"])
        link_elem = item.find("a", href=True)

        if title_elem:
            title = title_elem.get_text(strip=True)
            link = link_elem["href"] if link_elem else ""
            if link and not link.startswith("http"):
                link = f"https://news.ycombinator.com/{link}"

            extracted_items.append({
                "id": idx + 1,
                "title": title,
                "url": link
            })

    return extracted_items

def send_webhook_notification(webhook_url, data):
    """Sends JSON payload to configured Webhook (Slack/Discord/Custom)."""
    if not webhook_url:
        logger.warning("No WEBHOOK_URL configured. Skipping notification step.")
        return False

    payload = {
        "content": f"🚀 **Serverless Scraper Pipeline Report**\nSuccessfully extracted {len(data)} items.",
        "embeds": [
            {
                "title": item["title"][:256],
                "url": item["url"],
                "color": 3447003
            } for item in data[:5]
        ]
    }

    try:
        response = requests.post(webhook_url, json=payload, timeout=10)
        response.raise_for_status()
        logger.info("Webhook notification delivered successfully.")
        return True
    except Exception as e:
        logger.error(f"Failed to send webhook notification: {e}")
        return False

def main():
    target_url = os.getenv("TARGET_URL", "https://news.ycombinator.com/")
    webhook_url = os.getenv("WEBHOOK_URL", "")

    logger.info(f"Starting resilient extraction run for: {target_url}")
    session = create_resilient_session()

    try:
        response = session.get(target_url, timeout=15)
        response.raise_for_status()

        logger.info(f"Received HTTP {response.status_code}. Parsing DOM structure...")
        parsed_data = parse_target_data(response.text)
        logger.info(f"Successfully extracted {len(parsed_data)} items.")

        print(json.dumps(parsed_data, indent=2))

        if webhook_url:
            send_webhook_notification(webhook_url, parsed_data)

    except requests.RequestException as err:
        logger.critical(f"Network error during execution: {err}")
        sys.exit(1)

if __name__ == "__main__":
    main()

GitHub Actions Automated Workflow (`.github/workflows/scraper.yml`)

name: Resilient Serverless Web Scraper

on:
  schedule:
    - cron: '0 */6 * * *'
  workflow_dispatch:

jobs:
  scrape-and-notify:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout Code Repository
      uses: actions/checkout@v4

    - name: Set up Python 3.11 Environment
      uses: actions/setup-python@v5
      with:
        python-version: '3.11'
        cache: 'pip'

    - name: Install Python Dependencies
      run: |
        python -m pip install --upgrade pip
        pip install requests beautifulsoup4 urllib3

    - name: Execute Serverless Scraper Pipeline
      env:
        TARGET_URL: 'https://news.ycombinator.com/'
        WEBHOOK_URL: ${{ secrets.WEBHOOK_URL }}
      run: |
        python scraper.py

Best Practices & Resilience Strategies

Exponential Backoff Retries: Handles temporary network degradation cleanly.
User-Agent & Header Rotation: Simulates real browser handshake headers.
Secret Management: Store webhook endpoints in secrets.WEBHOOK_URL.
Zero Cost: Completely free with GitHub Actions runners.

DEV Community: Jaison Hernando Velasco Camacho

Building Real-Time Asynchronous Web Crawlers with Python Asyncio, Playwright, and Webhooks

Building Real-Time Asynchronous Web Crawlers with Python Asyncio, Playwright, and Webhooks

🏗️ Architecture Overview

🛠️ Prerequisites & Setup

🐍 Complete Implementation Code (crawler.py)

⚡ Key Technical Highlights

🛡️ Production Best Practices

🎯 Conclusion

Building Serverless API Gateways with Node.js Express and GitHub Actions CI/CD

Building Serverless API Gateways with Node.js Express and GitHub Actions CI/CD

🏗️ Architecture Overview

Key Components:

📁 Project Structure

💻 1. Setting Up package.json

🔒 2. Authentication & Rate Limiting Middleware

Authentication Middleware (src/middleware/auth.js)

Rate Limiter Middleware (src/middleware/rateLimiter.js)

🚀 3. Core Gateway Express App (src/app.js)

🔀 4. Routing & Proxying Engine (src/routes/gateway.js)

⚡ 5. Serverless Handler (handler.js) & Config (serverless.yml)

Lambda Entrypoint (handler.js)

Serverless Framework Configuration (serverless.yml)

⚙️ 6. GitHub Actions CI/CD Pipeline (.github/workflows/deploy.yml)

🧪 7. Automated Unit Testing with Jest & Supertest (tests/gateway.test.js)

🎯 Conclusion

Building Resilient Serverless Web Scrapers and Automated Webhook Pipeline with Python and GitHub Actions

Architecture Overview

Core Python Scraper Engine (scraper.py)

GitHub Actions Automated Workflow (.github/workflows/scraper.yml)

Best Practices & Resilience Strategies

🐍 Complete Implementation Code (`crawler.py`)

💻 1. Setting Up `package.json`

Authentication Middleware (`src/middleware/auth.js`)

Rate Limiter Middleware (`src/middleware/rateLimiter.js`)

🚀 3. Core Gateway Express App (`src/app.js`)

🔀 4. Routing & Proxying Engine (`src/routes/gateway.js`)

⚡ 5. Serverless Handler (`handler.js`) & Config (`serverless.yml`)

Lambda Entrypoint (`handler.js`)

Serverless Framework Configuration (`serverless.yml`)

⚙️ 6. GitHub Actions CI/CD Pipeline (`.github/workflows/deploy.yml`)

🧪 7. Automated Unit Testing with Jest & Supertest (`tests/gateway.test.js`)

Core Python Scraper Engine (`scraper.py`)

GitHub Actions Automated Workflow (`.github/workflows/scraper.yml`)