DEV Community: hex

Too Lazy to Click: How I Scraped a 400-Page Forum Thread with Python

hex — Tue, 17 Jun 2025 05:05:14 +0000

I found myself in a familiar situation for anyone deep into a technical hobby. In my case, it was game reverse engineering, specifically for Valorant. The holy grail of information was a thread on unknowncheats.me titled "Valorant Reversal, Structs and Offsets." This thread is a goldmine, with community members constantly posting updated memory offsets, data structures, and code snippets.

The problem? The thread was, at the time of writing, well over 400 pages long.

My goal was simple: I wanted to search through all the collective knowledge in that thread for specific keywords. But the manual process was a nightmare. It involved:

Opening a page.
Hitting CTRL+F to search.
Clicking the "Next Page" button.
Repeating steps 2 and 3... hundreds of times.

My patience wore thin after about ten pages. I thought, "There has to be a better way." I'm a programmer, and tedious, repetitive tasks are the very things we build tools to destroy. I felt lazy, but it was the productive kind of lazy. I decided I wasn't going to read the forum; I was going to "reverse" it and make a script do the reading for me.

Step 1: "Reversing" the Website (aka Peeking at the HTML)

The term "reversing" here is a bit dramatic. I wasn't disassembling the website's backend. I was simply doing what every web developer does: I right-clicked on a forum post and hit "Inspect Element."

My mission was to find a predictable pattern. How does the website identify each individual message? If I could find a unique, repeating identifier for the posts, I could teach a script to find them all.

I looked at the HTML structure of a post. It was nested deep in a series of <div> and <table> tags, but I quickly struck gold. Each post's main content was wrapped in an element with an ID that looked like this:

<td id="post_message_5234981">
  <!-- The entire message content is in here -->
  ...
</td>

Bingo. The pattern was post_message_ followed by a unique post number. This was the key. I could now reliably target every single message on a page.

Next, I looked at the URL.

https://www.unknowncheats.me/forum/valorant/385792-valorant-reversal-structs-offsets-380.html

The page number was right there at the end. This meant I could easily loop from page 1 to page 400+ just by changing that number in the URL. The plan was coming together.

Step 2: Building the Scraper with Python

With a clear plan, I fired up my code editor and started building the script. I chose Python for its fantastic libraries for web scraping: requests to handle the web requests and BeautifulSoup to parse the messy HTML into something manageable.

Here's the script I wrote, broken down piece by piece:

import requests
import re
from bs4 import BeautifulSoup
import json
import time
import random

# 1. Setup
data = []
starting_page_number = 380
ending_page_number = 382 # I set a small range for testing

# 2. The Main Loop
for page_number in range(starting_page_number, ending_page_number):
  print(f'Loading page {page_number}')
  url = f'https://www.unknowncheats.me/forum/valorant/385792-valorant-reversal-structs-offsets-{page_number}.html'

  # These headers can be important to mimic a real browser session
  payload={}
  headers = {
    'Cookie': 'bblastactivity=0; bblastvisit=1691742053; bbsessionhash=23eb5d9818dd9fadf6102dba29534ba1'
  }

  # 3. Fetching the Page
  response = requests.request("GET", url, headers=headers, data=payload)

  # 4. Parsing with BeautifulSoup
  soup = BeautifulSoup(response.text, 'html.parser')

  # This is where we use the pattern we found!
  elementsWithPostMessageID = soup.find_all(id=re.compile("^post_message_"))

  # 5. Extracting the Text
  for element in elementsWithPostMessageID:
      # .text strips all HTML tags, leaving just the message
      data.append(element.text)

  # 6. Being a Good Citizen
  print(f'Finished page {page_number}. Pausing...')
  time.sleep(random.randint(1,10))


# 7. Saving the Results
with open('messages.json', 'w', encoding='utf-8') as outfile:
    json.dump(data, outfile, indent=2, ensure_ascii=False)

print("Scraping complete! Data saved to messages.json")

Let's walk through it:

Setup: I initialize an empty list called data to hold all the message texts. I also define my starting and ending page numbers so I can easily control how much of the forum I scrape at a time.
The Main Loop: The script iterates through each page number I defined. In each loop, it constructs the specific URL for that page using an f-string.
Fetching the Page: Using the requests library, I send a GET request to the URL. I included a Cookie in the headers. Sometimes, forums restrict access or have different layouts for guests versus logged-in users. By providing a session cookie (copied from my browser), I ensure the script sees the page exactly as I do.
Parsing with BeautifulSoup: This is where the magic happens. I pass the raw HTML text from the response to BeautifulSoup. Then, I use the find_all() method with a regular expression: re.compile("^post_message_"). This tells BeautifulSoup: "Find every single tag on this page whose ID starts with post_message_."
Extracting the Text: The script loops through the list of elements it just found. For each element, element.text conveniently strips away all the HTML formatting (<b>, <i>, <br>, etc.) and gives me just the clean, raw text of the message. I append this text to my data list.
Being a Good Citizen: This is critical. Hammering a website with rapid-fire requests is a great way to get your IP address banned. The line time.sleep(random.randint(1,10)) pauses the script for a random interval between 1 and 10 seconds between each page load. This mimics human browsing behavior and avoids overwhelming the server.
Saving the Results: After the loop finishes, all the scraped messages are in the data list. I use Python's json library to dump this list into a file named messages.json. This gives me a clean, structured, and machine-readable archive of the entire forum thread.

The Result

Instead of hundreds of browser tabs and endless scrolling, I now have a single messages.json file. I can open it, search it instantly, or even write other scripts to analyze the data for trends. The tedious manual labor was transformed into a 10-minute coding session.

This little project was a perfect reminder that sometimes, the "lazy" way is the smart way. By investing a little time upfront to automate a task, I saved myself hours of mind-numbing work and ended up with a far more useful result.

Building a Modern Window Manager Library for Web Applications

hex — Mon, 16 Jun 2025 11:14:07 +0000

Introduction
Design Philosophy and Architecture
Development Process
Core Components Implementation
Creating an NPM Package
Documentation Standards
Testing Strategy
Publishing and Distribution
Best Practices and Lessons Learned

Introduction

This article documents the creation of web-window-manager, a comprehensive JavaScript library that brings desktop-like window management capabilities to web applications. We'll explore the architectural decisions, implementation details, and the process of packaging and distributing the library through NPM.

Project Goals

Create a lightweight, dependency-free window management system
Provide an intuitive API that mirrors desktop OS conventions
Ensure accessibility and performance across modern browsers
Build a foundation that developers can extend for their specific needs

Design Philosophy and Architecture

Core Design Principles

Progressive Enhancement: The library should work without any build tools or frameworks
Composability: Each component should function independently while integrating seamlessly
Performance First: Minimize reflows and repaints through efficient DOM manipulation
Accessibility: Ensure keyboard navigation and screen reader compatibility
Extensibility: Provide hooks and events for custom functionality

Architectural Decisions

Why Vanilla JavaScript?

We chose vanilla JavaScript over a framework-specific implementation for several reasons:

Universal Compatibility: Works with any framework or no framework at all
Minimal Overhead: No virtual DOM or framework abstractions
Learning Curve: Developers familiar with DOM APIs can immediately understand the codebase
Performance: Direct DOM manipulation when done correctly can outperform framework abstractions

Component Architecture

WindowManager
├── Window System
│   ├── Window Creation
│   ├── Drag & Drop Handler
│   ├── Resize Handler
│   └── Focus Management
├── Context Menu System
│   ├── Menu Renderer
│   └── Position Calculator
├── Modal System
│   ├── Overlay Manager
│   └── Modal Renderer
└── Notification System
    ├── Queue Manager
    └── Animation Controller

Development Process

Step 1: Requirements Gathering

We identified key features by analyzing popular desktop environments:

Windows: Title bar, control buttons, resize handles
macOS: Traffic light controls, smooth animations
Context Menus: Right-click functionality with nested menus
Modals: Focus trapping and backdrop interaction
Notifications: Toast-style alerts with auto-dismiss

Step 2: API Design

We followed a builder pattern approach for flexibility:

// Simple usage
wm.createWindow({ title: 'My App' });

// Advanced usage
wm.createWindow({
    title: 'Advanced Window',
    width: 600,
    height: 400,
    position: { x: 100, y: 100 },
    resizable: true,
    onClose: () => saveState(),
    content: document.getElementById('app-content')
});

Step 3: Implementation Strategy

Core Window Class: Established the base window functionality
Event System: Implemented a pub/sub pattern for window events
Interaction Handlers: Added drag, resize, and focus management
Visual Components: Created context menus, modals, and notifications
Style System: Developed a themeable CSS architecture

Core Components Implementation

Window Management System

The window system uses a Map to track windows and their states:

class WindowManager {
    constructor(options = {}) {
        this.windows = new Map();
        this.config = { ...DEFAULT_CONFIG, ...options };
        this.zIndexManager = new ZIndexManager();
        this.eventBus = new EventEmitter();
        this.init();
    }

    createWindow(options) {
        const window = new Window(this, options);
        this.windows.set(window.id, window);
        this.eventBus.emit('window:created', window);
        return window;
    }
}

Drag and Drop Implementation

We implemented drag handling using pointer events for better touch support:

class DragHandler {
    constructor(window, handle) {
        this.window = window;
        this.handle = handle;
        this.isDragging = false;
        this.dragStart = { x: 0, y: 0 };
        this.windowStart = { x: 0, y: 0 };

        this.bindEvents();
    }

    bindEvents() {
        this.handle.addEventListener('pointerdown', this.onPointerDown);
        document.addEventListener('pointermove', this.onPointerMove);
        document.addEventListener('pointerup', this.onPointerUp);
    }

    onPointerDown = (e) => {
        this.isDragging = true;
        this.dragStart = { x: e.clientX, y: e.clientY };
        this.windowStart = {
            x: this.window.position.x,
            y: this.window.position.y
        };
        this.handle.setPointerCapture(e.pointerId);
    }
}

Context Menu System

The context menu system calculates optimal positioning:

class ContextMenu {
    constructor(items, options = {}) {
        this.items = items;
        this.options = options;
        this.element = this.render();
        this.positionMenu();
    }

    positionMenu() {
        const { x, y } = this.options.position;
        const menuRect = this.element.getBoundingClientRect();
        const viewport = {
            width: window.innerWidth,
            height: window.innerHeight
        };

        // Adjust position to keep menu in viewport
        const adjustedX = Math.min(x, viewport.width - menuRect.width);
        const adjustedY = Math.min(y, viewport.height - menuRect.height);

        this.element.style.left = `${Math.max(0, adjustedX)}px`;
        this.element.style.top = `${Math.max(0, adjustedY)}px`;
    }
}

Creating an NPM Package

Project Structure

web-window-manager/
├── src/
│   ├── core/
│   │   ├── WindowManager.js
│   │   ├── Window.js
│   │   └── EventEmitter.js
│   ├── components/
│   │   ├── ContextMenu.js
│   │   ├── Modal.js
│   │   └── Notification.js
│   ├── utils/
│   │   ├── dom.js
│   │   └── position.js
│   ├── styles/
│   │   ├── base.css
│   │   └── themes/
│   └── index.js
├── dist/
│   ├── web-window-manager.js
│   ├── web-window-manager.min.js
│   └── web-window-manager.css
├── docs/
│   ├── api/
│   ├── examples/
│   └── guides/
├── test/
├── package.json
├── README.md
├── LICENSE
└── rollup.config.js

Package.json Configuration

{
  "name": "web-window-manager",
  "version": "1.0.0",
  "description": "A modern window management library for web applications",
  "main": "dist/web-window-manager.js",
  "module": "src/index.js",
  "style": "dist/web-window-manager.css",
  "files": [
    "dist",
    "src",
    "README.md",
    "LICENSE"
  ],
  "scripts": {
    "build": "rollup -c",
    "dev": "rollup -c -w",
    "test": "jest",
    "test:watch": "jest --watch",
    "lint": "eslint src/**/*.js",
    "docs": "jsdoc -c jsdoc.config.json",
    "prepublishOnly": "npm run build && npm test"
  },
  "keywords": [
    "window-manager",
    "desktop",
    "ui",
    "windowing",
    "modal",
    "context-menu",
    "draggable",
    "resizable"
  ],
  "author": "Your Name <your.email@example.com>",
  "license": "MIT",
  "repository": {
    "type": "git",
    "url": "https://github.com/yourusername/web-window-manager.git"
  },
  "bugs": {
    "url": "https://github.com/yourusername/web-window-manager/issues"
  },
  "homepage": "https://github.com/yourusername/web-window-manager#readme",
  "devDependencies": {
    "@rollup/plugin-node-resolve": "^15.0.0",
    "@rollup/plugin-terser": "^0.4.0",
    "eslint": "^8.0.0",
    "jest": "^29.0.0",
    "jsdoc": "^4.0.0",
    "rollup": "^3.0.0",
    "rollup-plugin-css-only": "^4.0.0"
  },
  "peerDependencies": {},
  "engines": {
    "node": ">=14.0.0"
  }
}

Build Configuration (Rollup)

// rollup.config.js
import resolve from '@rollup/plugin-node-resolve';
import terser from '@rollup/plugin-terser';
import css from 'rollup-plugin-css-only';

export default [
  // ES Module build
  {
    input: 'src/index.js',
    output: {
      file: 'dist/web-window-manager.esm.js',
      format: 'es',
      sourcemap: true
    },
    plugins: [resolve()]
  },
  // UMD build
  {
    input: 'src/index.js',
    output: {
      file: 'dist/web-window-manager.js',
      format: 'umd',
      name: 'WindowManager',
      sourcemap: true
    },
    plugins: [resolve()]
  },
  // Minified UMD build
  {
    input: 'src/index.js',
    output: {
      file: 'dist/web-window-manager.min.js',
      format: 'umd',
      name: 'WindowManager',
      sourcemap: true
    },
    plugins: [resolve(), terser()]
  },
  // CSS
  {
    input: 'src/styles/index.js',
    output: {
      file: 'dist/web-window-manager.css',
      format: 'es'
    },
    plugins: [css({ output: 'web-window-manager.css' })]
  }
];

Documentation Standards

API Documentation (JSDoc)

/**
 * Creates a new window instance
 * @param {Object} options - Window configuration options
 * @param {string} options.title - Window title
 * @param {number} [options.width=400] - Window width in pixels
 * @param {number} [options.height=300] - Window height in pixels
 * @param {Object} [options.position] - Initial window position
 * @param {number} [options.position.x=100] - X coordinate
 * @param {number} [options.position.y=100] - Y coordinate
 * @param {boolean} [options.resizable=true] - Enable window resizing
 * @param {boolean} [options.draggable=true] - Enable window dragging
 * @param {Function} [options.onClose] - Close event handler
 * @returns {Window} The created window instance
 * @fires WindowManager#window:created
 * @example
 * const window = wm.createWindow({
 *   title: 'My Application',
 *   width: 600,
 *   height: 400,
 *   onClose: () => console.log('Window closed')
 * });
 */
createWindow(options = {}) {
    // Implementation
}

README Structure

# Web Window Manager

A modern, lightweight window management library for web applications.

## Features

- 🪟 **Draggable Windows** - Smooth window dragging with touch support
- 📐 **Resizable Windows** - Resize from any edge or corner
- 🎨 **Themeable** - CSS variables for easy customization
- ♿ **Accessible** - Full keyboard navigation and ARIA support
- 📱 **Responsive** - Works on desktop and mobile devices
- 🚀 **Performant** - Optimized rendering and minimal reflows
- 🔧 **Framework Agnostic** - Works with any or no framework

## Installation

bash
npm install web-window-manager


## Quick Start

javascript
import WindowManager from 'web-window-manager';
import 'web-window-manager/dist/web-window-manager.css';

const wm = new WindowManager();

const window = wm.createWindow({
title: 'Hello World',
content: '

Welcome!

'
});


## API Reference

See [full API documentation](./docs/api.md).

## Examples

- [Basic Usage](./examples/basic.html)
- [Custom Themes](./examples/themes.html)
- [React Integration](./examples/react.html)
- [Vue Integration](./examples/vue.html)

## Browser Support

- Chrome/Edge 88+
- Firefox 78+
- Safari 14+
- iOS Safari 14+
- Chrome Android 88+

## Contributing

See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and guidelines.

## License

MIT © [Your Name]

Type Definitions

// types/index.d.ts
declare module 'web-window-manager' {
  export interface WindowOptions {
    title?: string;
    width?: number;
    height?: number;
    position?: { x: number; y: number };
    resizable?: boolean;
    draggable?: boolean;
    closable?: boolean;
    minimizable?: boolean;
    maximizable?: boolean;
    content?: string | HTMLElement;
    onClose?: () => void;
    onFocus?: () => void;
    onBlur?: () => void;
  }

  export interface ContextMenuItem {
    label: string;
    icon?: string;
    action?: () => void;
    disabled?: boolean;
    separator?: boolean;
    submenu?: ContextMenuItem[];
  }

  export class Window {
    id: string;
    title: string;
    position: { x: number; y: number };
    size: { width: number; height: number };

    close(): void;
    minimize(): void;
    maximize(): void;
    focus(): void;
    setTitle(title: string): void;
    setContent(content: string | HTMLElement): void;
  }

  export default class WindowManager {
    constructor(options?: WindowManagerOptions);
    createWindow(options?: WindowOptions): Window;
    closeWindow(id: string): void;
    getWindow(id: string): Window | undefined;
    getAllWindows(): Window[];
    showContextMenu(event: MouseEvent, items: ContextMenuItem[]): void;
    showModal(options: ModalOptions): void;
    showNotification(title: string, message: string, duration?: number): void;
  }
}

Testing Strategy

Unit Tests

// test/WindowManager.test.js
import WindowManager from '../src/core/WindowManager';

describe('WindowManager', () => {
  let wm;

  beforeEach(() => {
    document.body.innerHTML = '<div id="desktop"></div>';
    wm = new WindowManager({ container: '#desktop' });
  });

  afterEach(() => {
    wm.destroy();
  });

  describe('createWindow', () => {
    test('creates window with default options', () => {
      const window = wm.createWindow();

      expect(window).toBeDefined();
      expect(window.title).toBe('New Window');
      expect(wm.windows.size).toBe(1);
    });

    test('creates window with custom options', () => {
      const options = {
        title: 'Test Window',
        width: 500,
        height: 400
      };

      const window = wm.createWindow(options);

      expect(window.title).toBe('Test Window');
      expect(window.size.width).toBe(500);
      expect(window.size.height).toBe(400);
    });
  });

  describe('window interactions', () => {
    test('focuses window on click', () => {
      const window1 = wm.createWindow();
      const window2 = wm.createWindow();

      expect(wm.activeWindow).toBe(window2);

      window1.element.dispatchEvent(new MouseEvent('mousedown'));

      expect(wm.activeWindow).toBe(window1);
    });
  });
});

Integration Tests

// test/integration/drag-drop.test.js
import WindowManager from '../src/index';
import { simulateDrag } from './helpers';

describe('Drag and Drop Integration', () => {
  test('window can be dragged', async () => {
    const wm = new WindowManager();
    const window = wm.createWindow({
      position: { x: 100, y: 100 }
    });

    const initialPosition = { ...window.position };

    await simulateDrag(window.element.querySelector('.window-header'), {
      from: { x: 150, y: 120 },
      to: { x: 250, y: 220 }
    });

    expect(window.position.x).toBe(initialPosition.x + 100);
    expect(window.position.y).toBe(initialPosition.y + 100);
  });
});

Publishing and Distribution

Pre-publish Checklist

Version Bump: Update version in package.json following semver
Changelog: Update CHANGELOG.md with new features/fixes
Tests: Ensure all tests pass (npm test)
Build: Generate distribution files (npm run build)
Documentation: Update API docs (npm run docs)
Examples: Test all examples with new version
Types: Verify TypeScript definitions are accurate

Publishing to NPM

# Login to NPM
npm login

# Publish (runs prepublishOnly script automatically)
npm publish

# Tag release on GitHub
git tag v1.0.0
git push origin v1.0.0

CDN Distribution

The package will be automatically available on CDNs:

<!-- jsDelivr -->
<script src="https://cdn.jsdelivr.net/npm/web-window-manager@1.0.0/dist/web-window-manager.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/web-window-manager@1.0.0/dist/web-window-manager.css">

<!-- unpkg -->
<script src="https://unpkg.com/web-window-manager@1.0.0/dist/web-window-manager.min.js"></script>
<link rel="stylesheet" href="https://unpkg.com/web-window-manager@1.0.0/dist/web-window-manager.css">

Best Practices and Lessons Learned

Performance Optimization

Batch DOM Operations: Group DOM changes to minimize reflows
Event Delegation: Use single event listeners on parent elements
RequestAnimationFrame: Throttle drag/resize operations
CSS Containment: Use contain property for better rendering performance

Accessibility Considerations

Keyboard Navigation: Implement focus trapping and tab order
ARIA Labels: Add appropriate roles and labels
Screen Reader Support: Announce window state changes
High Contrast Mode: Ensure visibility in all color modes

Common Pitfalls to Avoid

Memory Leaks: Always remove event listeners on window close
Z-Index Wars: Use a centralized z-index management system
Touch vs Mouse: Handle both pointer types consistently
CSS Conflicts: Use scoped class names or CSS modules

Future Enhancements

Window Snapping: Snap to screen edges and other windows
Persistence: Save/restore window layouts
Multi-Monitor Support: Handle different screen configurations
Virtual Desktops: Multiple window spaces
Plugin System: Allow third-party extensions

Conclusion

Building a window manager library requires careful consideration of performance, accessibility, and developer experience. By following modern web standards and best practices, we've created a foundation that developers can use to build sophisticated desktop-like applications in the browser.

The key to success lies in:

Starting with a solid architecture
Prioritizing performance from the beginning
Writing comprehensive documentation
Testing across different scenarios
Listening to community feedback

This library serves as both a practical tool and a learning resource for developers interested in creating complex UI systems for the web.

Building a Chrome Extension Request Interceptor: A Complete Development Guide

hex — Mon, 16 Jun 2025 10:43:08 +0000

Executive Summary

Understanding how websites communicate with backend services is crucial for developers, security researchers, and system integrators. While browser DevTools provide basic network monitoring, they miss dynamically generated requests and lack advanced filtering capabilities. This guide details the complete development process of a Chrome extension that intercepts all HTTP requests (XHR and Fetch) at the JavaScript level, providing deep visibility into website behavior through a custom DevTools panel.

Key Benefits:

Captures requests missed by standard DevTools
Provides stack traces for request origins
Offers detailed timing and performance data
Enables comprehensive website behavior analysis

Project Requirements and Architecture Planning

Initial Problem Statement

Standard browser DevTools have limitations when analyzing complex web applications:

Limited Request Context: No stack traces showing where requests originate
Timing Gaps: Missing JavaScript-level performance data
Dynamic Request Blindness: Some programmatically generated requests aren't captured
Integration Challenges: Difficult to export or process captured data

Technical Requirements

Core Functionality:

Intercept all XMLHttpRequest and Fetch API calls
Capture complete request/response data including headers and bodies
Provide stack traces for debugging request origins
Display data in an intuitive DevTools panel interface
Support real-time monitoring across all website tabs

Technical Constraints:

Must work with Manifest V3 (Chrome's latest extension format)
Cannot break existing website functionality
Must handle high-frequency request scenarios
Needs to work across all domains and protocols

Architecture Design

The extension uses a multi-layer architecture to overcome Chrome's security restrictions:

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐    ┌──────────────────┐
│   Web Page      │    │  Content Script  │    │ Background      │    │  DevTools Panel │
│                 │    │                  │    │ Service Worker  │    │                  │
│ ┌─────────────┐ │    │ ┌──────────────┐ │    │ ┌─────────────┐ │    │ ┌──────────────┐ │
│ │ Injected    │◄┼────┼►│ Message      │◄┼────┼►│ Message     │◄┼────┼►│ UI Display   │ │
│ │ Script      │ │    │ │ Forwarder    │ │    │ │ Router      │ │    │ │ & Controls   │ │
│ └─────────────┘ │    │ └──────────────┘ │    │ └─────────────┘ │    │ └──────────────┘ │
└─────────────────┘    └──────────────────┘    └─────────────────┘    └──────────────────┘
        │                        │                        │                        │
        ▼                        ▼                        ▼                        ▼
   Intercepts                Bridges                 Routes &                 Displays
   HTTP calls               contexts                Queues                  captured data

Phase 1: Project Setup and Manifest Configuration

Creating the Extension Structure

First, establish the project directory structure:

request-interceptor/
├── manifest.json
├── background.js
├── content_script.js
├── injected.js
├── devtools.html
├── devtools.js
├── panel.html
├── panel.css
├── panel.js
└── icons/
    └── icon16.png

Manifest V3 Configuration

The manifest.json file defines the extension's capabilities and architecture:

{
  "manifest_version": 3,
  "name": "Request Interceptor",
  "version": "1.0",
  "description": "Intercepts requests in-page and displays them in a DevTools panel.",
  "permissions": [
    "storage"
  ],
  "host_permissions": [
    "<all_urls>"
  ],
  "icons": {
    "128": "icon.png"
  },
  "background": {
    "service_worker": "background.js"
  },
  "content_scripts": [
    {
      "matches": ["<all_urls>"],
      "js": ["content_script.js"],
      "run_at": "document_start",
      "all_frames": true
    }
  ],
  "devtools_page": "devtools.html",
  "web_accessible_resources": [
    {
      "resources": ["injected.js"],
      "matches": ["<all_urls>"]
    }
  ]
}

Key Configuration Decisions:

"run_at": "document_start": Ensures injection before any website scripts execute
"all_frames": true: Captures requests from iframes and embedded content
"<all_urls>": Universal access for comprehensive website analysis
web_accessible_resources: Allows content script to inject our monitoring code

Phase 2: Core Interception Logic Development

The Injected Script Foundation

The injected.js file runs in the website's context and performs the actual HTTP interception:

(function () {
    // Prevent multiple injection
    if (window.__HybridInterceptorActive) return;
    window.__HybridInterceptorActive = true;

    const sendDataToDevtools = (data) => {
      window.postMessage({
        type: "FROM_PAGE",
        payload: data
      }, "*");
    };

    const getStackTrace = () => {
      try {
        throw new Error();
      } catch (e) {
        return e.stack ? e.stack.split("\n").slice(2).join("\n") : 'Stack trace not available';
      }
    };

Design Rationale:

Global Flag: Prevents multiple injections that could cause conflicts
PostMessage API: Safely communicates with content script across security boundaries
Stack Trace Capture: Provides debugging context for request origins

XMLHttpRequest Interception

XMLHttpRequest interception requires monkey-patching three prototype methods:

const originalXhrOpen = XMLHttpRequest.prototype.open;
const originalXhrSend = XMLHttpRequest.prototype.send;
const originalSetRequestHeader = XMLHttpRequest.prototype.setRequestHeader;

XMLHttpRequest.prototype.open = function (method, url) {
  this._interceptorData = {
    type: 'XHR',
    method,
    url,
    headers: {},
    start: performance.now(),
    trace: getStackTrace()
  };
  return originalXhrOpen.apply(this, arguments);
};

XMLHttpRequest.prototype.setRequestHeader = function (key, value) {
    if (this._interceptorData && this._interceptorData.headers) {
        this._interceptorData.headers[key] = value;
    }
    return originalSetRequestHeader.apply(this, arguments);
};

XMLHttpRequest.prototype.send = function (body) {
  if (this._interceptorData) {
    this._interceptorData.body = body || null;
  }

  const onStateChange = () => {
    if (this.readyState === 4 && this._interceptorData) {
      this._interceptorData.end = performance.now();
      this._interceptorData.duration = this._interceptorData.end - this._interceptorData.start;
      this._interceptorData.status = this.status;
      this._interceptorData.response = this.responseText;
      this._interceptorData.timestamp = new Date().toISOString();

      sendDataToDevtools(this._interceptorData);
    }
  };

  this.addEventListener("readystatechange", onStateChange);
  return originalXhrSend.apply(this, arguments);
};

Technical Challenges Solved:

State Management: Using instance properties to track request data across method calls
Header Capture: Intercepting setRequestHeader calls to build complete header map
Timing Precision: Using performance.now() for microsecond-accurate duration measurement
Non-intrusive Design: Preserving original functionality while adding monitoring

Fetch API Interception

Fetch interception is more complex due to its Promise-based nature and response streaming:

const originalFetch = window.fetch;
window.fetch = function(...args) {
  const [url, config] = args;
  const fetchData = {
    type: 'Fetch',
    method: config?.method || 'GET',
    url: url.toString(),
    headers: config?.headers ? new Headers(config.headers).toJSON() : {},
    body: config?.body || null,
    start: performance.now(),
    trace: getStackTrace(),
    timestamp: new Date().toISOString()
  };

  return originalFetch(...args)
    .then(response => {
      const responseClone = response.clone();
      fetchData.status = response.status;
      fetchData.end = performance.now();
      fetchData.duration = fetchData.end - fetchData.start;

      responseClone.text().then(text => {
        fetchData.response = text;
        sendDataToDevtools(fetchData);
      });

      return response;
    })
    .catch(error => {
      fetchData.status = 'Error';
      fetchData.response = error.message;
      fetchData.end = performance.now();
      fetchData.duration = fetchData.end - fetchData.start;
      sendDataToDevtools(fetchData);
      throw error;
    });
};

Key Implementation Details:

Response Cloning: Essential for reading response body without consuming the original stream
Headers Processing: Custom toJSON() method for Headers object serialization
Error Handling: Capturing both network errors and successful responses
Promise Chain Preservation: Maintaining original Fetch behavior for website compatibility

Phase 3: Inter-Context Communication System

Content Script Bridge

The content script (content_script.js) bridges the isolated website context with the extension's privileged context:

// 1. Inject the monitoring script into the page
const script = document.createElement('script');
script.src = chrome.runtime.getURL('injected.js');
(document.head || document.documentElement).appendChild(script);
script.onload = () => script.remove();

// 2. Listen for messages from the injected script
window.addEventListener("message", (event) => {
  // Security validation
  if (event.source !== window || !event.data.type || event.data.type !== "FROM_PAGE") {
    return;
  }

  // 3. Forward the data to the background script
  chrome.runtime.sendMessage({
    type: 'INTERCEPTED_REQUEST',
    data: event.data.payload
  });
}, false);

Security Considerations:

Source Validation: Ensures messages originate from the same window
Type Filtering: Only processes messages with expected structure
Script Cleanup: Removes injection script element after execution

Background Service Worker

The background script (background.js) manages connections and message routing:

const connections = {};
const messageQueue = {};

chrome.runtime.onConnect.addListener(port => {
  const extensionListener = (message) => {
    if (message.name === "init") {
      const tabId = message.tabId;
      connections[tabId] = port;

      // Process any queued messages for this tab
      if (messageQueue[tabId]) {
        messageQueue[tabId].forEach(msg => port.postMessage(msg));
        delete messageQueue[tabId];
      }

      port.onDisconnect.addListener(() => {
        delete connections[tabId];
      });
    }
  };
  port.onMessage.addListener(extensionListener);
});

chrome.runtime.onMessage.addListener((request, sender) => {
  if (sender.tab) {
    const tabId = sender.tab.id;
    if (tabId in connections) {
      connections[tabId].postMessage(request);
    } else {
      // Queue messages when DevTools panel isn't ready
      if (!messageQueue[tabId]) {
        messageQueue[tabId] = [];
      }
      messageQueue[tabId].push(request);
    }
  }
  return true;
});

Advanced Features:

Connection Management: Maintains persistent connections to DevTools panels
Message Queuing: Buffers messages when DevTools isn't open
Tab Isolation: Routes messages to correct panel instance
Cleanup Handling: Removes stale connections on disconnect

Phase 4: DevTools Panel Integration

Panel Registration

DevTools integration starts with panel registration (devtools.js):

chrome.devtools.panels.create(
    "Request Spy",
    "icons/icon16.png",
    "panel.html"
);

Panel HTML Structure

The panel UI (panel.html) uses semantic HTML for accessibility and maintainability:

<!DOCTYPE html>
<html>
  <head>
    <link rel="stylesheet" href="panel.css">
  </head>
  <body>
    <div class="toolbar">
      <button id="clear-button">Clear</button>
    </div>
    <div id="requests-container"></div>

    <template id="request-template">
      <div class="request-entry">
        <details>
          <summary class="request-summary">
            <span class="status-badge"></span>
            <span class="method-badge"></span>
            <span class="url"></span>
            <span class="duration"></span>
          </summary>
          <div class="details-grid">
            <div class="grid-header">URL:</div>
            <div class="grid-value full-url"></div>
            <div class="grid-header">Timestamp:</div>
            <div class="grid-value timestamp"></div>
            <div class="grid-header">Request Headers:</div>
            <div class="grid-value"><pre class="headers"></pre></div>
            <div class="grid-header">Request Body:</div>
            <div class="grid-value"><pre class="request-body"></pre></div>
            <div class="grid-header">Response Body:</div>
            <div class="grid-value"><pre class="response-body"></pre></div>
            <div class="grid-header">Trace:</div>
            <div class="grid-value"><pre class="trace"></pre></div>
          </div>
        </details>
      </div>
    </template>

    <script src="panel.js"></script>
  </body>
</html>

UI Design Decisions:

Details/Summary Elements: Native browser collapsible functionality
Template Element: Efficient DOM cloning for dynamic content
Grid Layout: Organized data presentation
Pre Elements: Preserve formatting for code and JSON

Panel Styling

CSS styling (panel.css) follows DevTools design patterns:

body {
  font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
  font-size: 14px;
  margin: 0;
  padding: 0;
  background-color: #f3f3f3;
}

.status-badge {
  padding: 2px 8px;
  border-radius: 10px;
  font-size: 12px;
  color: white;
  font-weight: bold;
  white-space: nowrap;
  flex-shrink: 0;
}

.status-badge.status-ok { background-color: #28a745; }
.status-badge.status-error { background-color: #dc3545; }
.status-badge.status-other { background-color: #6c757d; }

.details-grid {
  display: grid;
  grid-template-columns: 150px 1fr;
  gap: 8px;
  padding: 12px;
  border-top: 1px solid #eee;
  background-color: #fafafa;
}

Visual Design Principles:

Status Color Coding: Immediate visual feedback for response status
Responsive Grid: Adapts to different panel sizes
Developer-Friendly Typography: Monospace fonts for code content
Consistent Spacing: Professional appearance matching DevTools aesthetic

Panel JavaScript Logic

The panel logic (panel.js) handles data reception and UI updates:

document.addEventListener('DOMContentLoaded', () => {
  const requestsContainer = document.getElementById('requests-container');
  const requestTemplate = document.getElementById('request-template');
  const clearButton = document.getElementById('clear-button');

  // Establish connection to background script
  const backgroundPageConnection = chrome.runtime.connect({ name: "devtools-panel" });
  backgroundPageConnection.postMessage({ 
    name: 'init', 
    tabId: chrome.devtools.inspectedWindow.tabId 
  });

  // Handle incoming request data
  backgroundPageConnection.onMessage.addListener((message) => {
    if (message.type === 'INTERCEPTED_REQUEST' && message.data) {
      addRequestToView(message.data);
    }
  });

  // Clear functionality
  clearButton.addEventListener('click', () => {
    requestsContainer.innerHTML = '';
  });

  function addRequestToView(data) {
    const template = requestTemplate.content.cloneNode(true);

    // Status badge configuration
    const statusBadge = template.querySelector('.status-badge');
    const statusStr = data.status ? data.status.toString() : '???';
    statusBadge.textContent = `${data.type} ${statusStr}`;

    if (statusStr.startsWith('2')) {
      statusBadge.classList.add('status-ok');
    } else if (statusStr.startsWith('4') || statusStr.startsWith('5') || statusStr === 'Error') {
       statusBadge.classList.add('status-error');
    } else {
       statusBadge.classList.add('status-other');
    }

    // Populate template with request data
    template.querySelector('.method-badge').textContent = data.method;
    template.querySelector('.url').textContent = data.url;
    template.querySelector('.full-url').textContent = data.url;
    template.querySelector('.duration').textContent = data.duration ? `${data.duration.toFixed(0)}ms` : '';
    template.querySelector('.timestamp').textContent = data.timestamp;

    template.querySelector('.headers').textContent = JSON.stringify(data.headers || {}, null, 2);
    template.querySelector('.request-body').textContent = formatBody(data.body);
    template.querySelector('.response-body').textContent = formatBody(data.response);
    template.querySelector('.trace').textContent = data.trace || 'N/A';

    requestsContainer.appendChild(template);
  }

  function formatBody(body) {
    if (body === null || body === undefined) return 'N/A';

    if (typeof body === 'string') {
        try {
            // Pretty-print JSON if possible
            return JSON.stringify(JSON.parse(body), null, 2);
        } catch (e) {
            return body;
        }
    }

    if (typeof body === 'object') {
        return JSON.stringify(body, null, 2);
    }

    return body.toString();
  }
});

Advanced UI Features:

Template Cloning: Efficient DOM manipulation for high-frequency updates
Smart JSON Formatting: Automatic pretty-printing of JSON content
Real-time Updates: Immediate display of intercepted requests
Connection Management: Robust handling of DevTools panel lifecycle

Phase 5: Testing and Deployment

Local Testing Process

1. Extension Loading:

# Navigate to chrome://extensions/
# Enable "Developer mode"
# Click "Load unpacked" and select project directory

2. Functional Testing:

Open any website with AJAX functionality
Open DevTools and navigate to "Request Spy" tab
Trigger various HTTP requests (form submissions, API calls, etc.)
Verify all request data is captured and displayed correctly

3. Performance Testing:

Test on high-traffic websites (social media, news sites)
Monitor for performance degradation
Verify memory usage remains stable during extended sessions

Common Issues and Solutions

Issue: Requests Not Appearing

Diagnosis: Check if injected script loaded successfully
Solution: Verify web_accessible_resources configuration in manifest

Issue: DevTools Panel Not Loading

Diagnosis: Check background script connection handling
Solution: Ensure proper tab ID routing and connection management

Issue: Missing Request Data

Diagnosis: Timing issues with response body reading
Solution: Implement proper Promise handling in Fetch interception

Production Deployment

Chrome Web Store Preparation:

Icon Assets: Create required icon sizes (16x16, 48x48, 128x128)
Store Listing: Prepare description, screenshots, and promotional images
Privacy Policy: Document data handling practices
Permissions Justification: Explain necessity of broad permissions

Advanced Use Cases and Applications

Website Behavior Analysis

API Discovery:

Example: Analyzing a Social Media Site
┌─────────────────────────────────────────────────────────────┐
│ Captured Requests:                                          │
│ POST /api/v1/posts          → Content publishing           │
│ GET  /api/v1/feed           → Timeline loading             │
│ POST /api/v1/analytics      → User behavior tracking       │
│ GET  /api/v1/notifications  → Real-time updates            │
└─────────────────────────────────────────────────────────────┘

Authentication Flow Mapping:

Capture OAuth token exchanges
Monitor session refresh mechanisms
Identify security headers and CSRF protection

Performance Analysis:

Measure API response times across different endpoints
Identify slow or failing requests
Analyze request frequency patterns

Security Research Applications

AJAX Endpoint Discovery:

Identify hidden API endpoints not visible in source code
Map complete API surface area
Discover internal vs. public endpoints

Data Flow Analysis:

Track sensitive data transmission
Identify unencrypted communications
Monitor third-party integrations

Rate Limiting Detection:

Observe request throttling behavior
Identify API usage limits
Test error handling mechanisms

Integration Development Support

Third-Party API Integration:

Reverse-engineer API call patterns
Understand required headers and authentication
Capture complete request/response examples

Microservices Architecture Analysis:

Map service-to-service communication
Identify service dependencies
Document API contracts

Performance Optimization and Best Practices

Memory Management

Request Data Lifecycle:

// Implement request history limits
const MAX_REQUESTS = 1000;
if (requestHistory.length > MAX_REQUESTS) {
  requestHistory.splice(0, requestHistory.length - MAX_REQUESTS);
}

Cleanup Strategies:

Clear stored request data on tab close
Implement periodic cleanup of old entries
Optimize large response body handling

Scalability Considerations

High-Frequency Request Handling:

Implement request batching for performance
Add filtering options to reduce noise
Consider sampling for extremely high-traffic sites

Resource Usage Optimization:

Minimize memory footprint of stored request data
Implement lazy loading for response bodies
Optimize DOM updates for large request lists

Security Best Practices

Data Sanitization:

function sanitizeRequestData(data) {
  // Remove sensitive headers
  const sensitiveHeaders = ['authorization', 'cookie', 'x-api-key'];
  const cleanHeaders = { ...data.headers };

  sensitiveHeaders.forEach(header => {
    if (cleanHeaders[header]) {
      cleanHeaders[header] = '[REDACTED]';
    }
  });

  return { ...data, headers: cleanHeaders };
}

Permission Minimization:

Request only necessary permissions
Implement user consent for sensitive data capture
Provide opt-out mechanisms for privacy-conscious users

Future Enhancement Roadmap

Phase 1 Enhancements

Request Filtering: Add URL pattern and method filtering
Export Functionality: Generate HAR files for external analysis
Search Capabilities: Full-text search across request/response data

Phase 2 Advanced Features

Request Replay: Resend captured requests with modifications
Performance Metrics: Detailed timing breakdowns and statistics
Custom Scripting: JavaScript execution against captured data

Phase 3 Enterprise Features

Team Collaboration: Share captured sessions across team members
Integration APIs: Connect with external monitoring systems
Automated Testing: Generate test cases from captured requests

Conclusion

This Chrome extension demonstrates sophisticated browser extension development techniques while solving real-world web development challenges. The multi-context architecture, advanced JavaScript interception patterns, and custom DevTools integration showcase modern web platform capabilities.

Key Technical Achievements:

Successful implementation of non-intrusive HTTP monitoring
Robust cross-context communication system
Professional-grade developer tool interface
Comprehensive website behavior analysis capabilities

Applications for Technical Documentation:

Demonstrates complex system architecture documentation
Shows integration between multiple technical components
Provides real-world example of developer tool creation
Illustrates security considerations in extension development

The extension serves as both a practical development tool and an excellent example of technical writing that combines deep technical knowledge with clear, actionable documentation. This approach directly aligns with Google's technical writing requirements for roles involving developer tools, system integration, and infrastructure documentation.

Repository Link: https://github.com/hexcreator/hook-http-requests

DEV Community: hex

Too Lazy to Click: How I Scraped a 400-Page Forum Thread with Python

Step 1: "Reversing" the Website (aka Peeking at the HTML)

Step 2: Building the Scraper with Python

The Result

Building a Modern Window Manager Library for Web Applications

Table of Contents

Introduction

Project Goals

Design Philosophy and Architecture

Core Design Principles

Architectural Decisions

Why Vanilla JavaScript?

Component Architecture

Development Process

Step 1: Requirements Gathering

Step 2: API Design

Step 3: Implementation Strategy

Core Components Implementation

Window Management System

Drag and Drop Implementation

Context Menu System

Creating an NPM Package

Project Structure

Package.json Configuration

Build Configuration (Rollup)

Documentation Standards

API Documentation (JSDoc)

README Structure

Welcome!

Type Definitions

Testing Strategy

Unit Tests

Integration Tests

Publishing and Distribution

Pre-publish Checklist

Publishing to NPM

CDN Distribution

Best Practices and Lessons Learned

Performance Optimization

Accessibility Considerations

Common Pitfalls to Avoid

Future Enhancements

Conclusion

Building a Chrome Extension Request Interceptor: A Complete Development Guide

Executive Summary

Project Requirements and Architecture Planning

Initial Problem Statement

Technical Requirements

Architecture Design

Phase 1: Project Setup and Manifest Configuration

Creating the Extension Structure

Manifest V3 Configuration

Phase 2: Core Interception Logic Development

The Injected Script Foundation

XMLHttpRequest Interception

Fetch API Interception

Phase 3: Inter-Context Communication System

Content Script Bridge

Background Service Worker

Phase 4: DevTools Panel Integration

Panel Registration

Panel HTML Structure

Panel Styling

Panel JavaScript Logic

Phase 5: Testing and Deployment

Local Testing Process

Common Issues and Solutions

Production Deployment

Advanced Use Cases and Applications

Website Behavior Analysis

Security Research Applications

Integration Development Support

Performance Optimization and Best Practices

Memory Management

Scalability Considerations

Security Best Practices

Future Enhancement Roadmap

Phase 1 Enhancements

Phase 2 Advanced Features