DEV Community: Vinicius Porto

When the Scraper Breaks Itself: Building a Self-Healing CSS Selector Repair System

Vinicius Porto — Wed, 01 Apr 2026 23:15:54 +0000

A Python sidecar that watches your scraper fail, calls a local LLM, and fixes the problem before your users notice.

The Problem with Fragile Selectors

Production web scrapers have a hidden fragility: they depend on CSS selectors and XPath expressions authored against a snapshot of a third-party website's DOM. The moment the site redesigns its layout, renames a class, or restructures a table, those selectors silently return nothing — or, worse, return the wrong data.

For a surf alert system that monitors dozens of forecast sources, this is a recurring operational problem. A selector like tr.forecast-table__row[data-row-name="wave-heigth"] (yes, a typo the upstream site just fixed) breaks at 3 AM. The scraper records a failure, the forecast pipeline stalls, and users stop receiving alerts for a beach they care about. An engineer wakes up to a Slack notification, digs through logs, finds the selector, pushes a fix, and deploys.

The fix itself usually takes under five minutes. The detection, investigation, and deploy ceremony takes two hours.

This is not a scaling problem — it is a friction problem. The actual repairs are trivially mechanical: look at the new HTML, find the element, write a new selector. Automating that loop, safely, is what the Self-Healer is for.

Core Idea in One Paragraph

When the Ruby scraper fails to extract a field (e.g., wave_height returns nil), it publishes a repair job to a Redis queue. A Python sidecar — the Self-Healer — picks up that job, fetches the current HTML from the source URL, trims it to fit a token budget, and sends a targeted prompt to a local LLM running via MLX. The LLM proposes new CSS/XPath selector candidates with confidence scores and reasoning. Each candidate is then tested against the live HTML using BeautifulSoup and lxml, and the extracted value is validated against a type schema (e.g., float:0.1-20.0 for wave height). If a candidate passes, the new selector is written directly to data_sources.selector_overrides in PostgreSQL — no redeploy required. The next scraper run reads the updated config and proceeds as if nothing happened.

Design Principles

The shape of this system comes from a few deliberate constraints.

LLM as proposer, not decider

The model suggests selectors. Code decides whether they work. Every candidate goes through deterministic BeautifulSoup/lxml evaluation against the actual HTML before anything touches the database. The LLM output is treated like a PR from an intern: read with interest, merged only after review.

This matters because LLMs are confident even when wrong. A model can generate a plausible-looking XPath that matches the wrong column in a table — extracting the header label instead of the numeric value. Type validation catches this: "Wave Height (m)" fails float:0.1-20.0, so the candidate is rejected.

Sandbox before promotion

Candidates run against fetched HTML in-process, using the same parsing libraries the scraper would use. There is no staging environment, no shadow traffic, no A/B rollout. The sandbox is the gate. A selector that does not match the current HTML and extract a value that passes type validation never reaches the database.

Escalation over hallucination

Not all broken selectors can be repaired automatically. Two cases get escalated instead of guessed:

JS-rendered pages: If the fetched HTML is an empty SPA shell — detected via React/Vue/Angular framework markers, sparse body text relative to total HTML size, or loading spinners — there is no static selector to find. The repair status becomes ESCALATED_JS, the data_sources row is flagged with is_js_rendered = true, and a Slack message goes to the team.
Exhausted retries: If the LLM generates candidates across multiple context window widths and none pass validation, the status becomes ESCALATED_HUMAN. Automated repair has reached its limit; a human needs to look.

Both paths are first-class outcomes, not error states. They produce repair log entries, update DB flags, and send structured Slack notifications — so the team has full visibility without polling logs.

Decoupling via Redis

The Ruby scraper and Python sidecar share nothing except two Redis queues: healer:jobs (input) and healer:results (output). The scraper does LPUSH healer:jobs on failure; the healer does BRPOP healer:jobs and blocks until a job arrives. There is no shared process, no RPC, no HTTP call between services. Either side can restart independently without affecting the other.

Local inference

The LLM runs on-device via MLX, Apple's machine learning framework for Apple Silicon. This has three practical implications: fetched HTML never leaves the machine, there are no API costs per repair, and latency is bounded by local compute rather than network round-trips. The LLM client uses an OpenAI-compatible HTTP API (POST /chat/completions), so switching to a cloud model is a one-line config change if needed.

Traceability by default

Every repair attempt — successful or not — writes to selector_repair_logs. Jobs carry an optional scrape_failure_id that links back to the original scrape_failures row. When a repair succeeds, scrape_failures.resolved_at is stamped and resolved_by is set to 'healer'. The full audit trail is in the database, queryable, and joinable.

System Context

The Self-Healer is a container in the existing Docker Compose stack, enabled via --profile healer. It adds no new database (it reads and writes to shared PostgreSQL), no new queue infrastructure (it uses the existing Redis), and no new external dependencies beyond the local MLX process and a Slack webhook.

End-to-End Pipeline

A single repair job moves through eight steps:

1. Job arrives. RepairWorker pops from healer:jobs. The job carries beach_name, field_name, failed_selector, source_url, and optionally an html_snapshot and a last_known_good_value.

2. HTML fetch. HTMLFetcher makes a plain HTTP GET to source_url. If the job included an html_snapshot, it is used directly — useful for testing and replaying past failures.

3. JS detection. JSDetector checks the HTML for framework markers (<div id="root"></div>, data-reactroot, ng-app, etc.), loading indicators, and sparse body content relative to total HTML size. If any check fires, the job escalates immediately. Feeding an LLM trimmed markup from an empty SPA shell produces useless selectors.

4. HTML trimming. HTMLTrimmer reduces a 300 KB page to a ~5 KB snippet. It removes <script>, <style>, <svg>, and other non-content tags; strips non-selector attributes (keeping id, class, data-*, aria-*); locates the element nearest the target selector or the expected value; then walks up context_levels parent nodes to include enough surrounding structure. The default is 3 levels. Retries use 5, 7, then 9 levels.

5. LLM call. LLMClient builds a prompt with the field name, the failed selector, the expected type, the last known good value, and the trimmed HTML snippet. It posts to the MLX OpenAI-compatible endpoint with temperature=0.3. The response is parsed into a SelectorCandidates object. If the model outputs preamble before the JSON (some models do), _extract_balanced_brace finds the {...} block regardless.

6. Sandbox. SelectorSandbox runs each candidate against the full HTML using BeautifulSoup (CSS) or lxml (XPath) and extracts the matched value. Results are sorted: successful matches first, then by LLM confidence score.

7. Validation. Validator checks each passing sandbox result against the expected type spec. For float:0.1-20.0, it extracts the numeric portion via regex (handling mixed strings like "1.5ESE"), converts to float, and range-checks it. A header cell returning "Wave Height (m)" fails here.

8. Promote or retry/escalate. The first candidate that passes validation is promoted: selector_overrides is updated, the repair log is written, the scrape failure is resolved, and Slack gets a success notification. If no candidate passes and retries remain, the trimmer is called again with wider context. If retries are exhausted, the job escalates to human review.

The sequence diagram below shows how messages and data move across services for a single repair:

And the internal step-by-step flow inside a single repair attempt:

Deep Dives

Token budget and the context_levels retry loop

Sending a full HTML page to an LLM is wasteful and often impossible — 300 KB of HTML is around 75,000 tokens, well beyond typical context windows and full of noise. The trimmer's job is to find the smallest HTML fragment that still gives the LLM enough signal.

The context_levels parameter controls how far up the DOM tree to walk from the target element. At level 3 you might get table > tbody > tr > td — enough to understand the structure. At level 9 (used on the final retry) you might get the full forecast section container. Each retry widens the window, trading token cost for contextual richness.

This mirrors how a human engineer approaches the same problem: start by looking at the specific row, and if that isn't enough, zoom out to the table.

Prompting for structured output

The prompt template is explicit about output format: "You must respond with ONLY a single JSON object. Start your response with { and end with }." It also encodes rules that push the model toward stable selectors: prefer data-* and aria-* attributes over class names, prefer label-anchored selectors over positional nth-child, include both CSS and XPath variants when possible.

The structured output — a candidates array with selector, type, confidence, reasoning, and attribute — lets the pipeline operate deterministically on the model's output without parsing prose. The reasoning field is stored in the repair log, useful for post-incident review and prompt iteration.

Why type validation is not enough

A selector can match the right element type but the wrong element. Consider:

<tr data-row-name="wave-height">
  <th>Wave Height (m)</th>   <!-- header -->
  <td>1.5</td>               <!-- data    -->
</tr>

If the LLM generates tr[data-row-name="wave-height"] th instead of td, the sandbox extracts "Wave Height (m)". Type validation rejects this because it cannot parse a float from that string. Correct outcome, for the right reason.

The deeper issue is that the healer currently validates in isolation. It does not verify that the extracted HTML structure matches what the Ruby scraper's process_wave_data method expects downstream — for example, a <td> carrying a data-swell-state JSON attribute that encodes the full swell envelope. A selector that extracts 1.5 from a plain <span> would pass today's validation but silently break the downstream extraction logic.

This is an acknowledged gap. The roadmap includes a StructuralValidator that codifies field-level DOM requirements (element type, required attributes, required children) and a ValueCrossValidator that compares extracted values against recent historical data for the same beach — catching plausible-looking outliers that type ranges alone would pass.

Limitations and Honest Boundaries

Static HTML only. The automated repair path works only when target data is present in the raw HTTP response. Single-page applications that populate content via JavaScript require a headless browser for rendering, which is a different class of tooling. For now, JS-rendered sources are escalated rather than guessed at.

Scraper coupling. Even a correctly extracted value might not flow through the pipeline if the scraper's process_* methods expect a specific DOM structure the new selector doesn't deliver. The healer validates the selector in isolation; it cannot yet confirm end-to-end compatibility with Ruby-side extraction logic.

LLM correctness. A local model running at temperature=0.3 is not a theorem prover. The validation pipeline exists precisely because the model can propose syntactically valid but semantically wrong selectors. No incorrect LLM suggestion can reach production without passing independent deterministic checks — but monitoring scrape quality over time (alerts, golden samples, periodic audits of selector_overrides) remains necessary.

Roadmap: From Reactive Repair to Proactive Extraction Engine

The current system operates in repair mode: it reacts to production failures and fixes them. The building blocks — fetch, detect, trim, prompt, sandbox, validate, promote — compose naturally into a second mode.

Scout mode would run proactively for new sources: given a URL and a target schema (wave height, period, wind speed, tide, etc.), the system discovers selectors without waiting for a failure. It adds stricter series validation — not just "does this selector extract a float?" but "does it extract a time-aligned series of consistent floats across multiple forecast periods?" — and returns a structured config the Ruby scraper can consume immediately.

The architectural upgrade this enables is significant. Instead of onboarding a new surf forecast source by manually inspecting HTML and writing selectors, an engineer submits a URL and reviews the proposed config. The same Python service handles both ongoing repair and initial discovery with shared components and a single mental model. The system stops being "a scraper with a repair button" and becomes a forecast extraction engine with two modes.

The Staging Ground Symbiote: A Deep Dive into Monkey Patching for Bug and Exception Simulation (Part 1: Foundations & Risks)

Vinicius Porto — Sun, 19 Oct 2025 00:25:15 +0000

Introduction

The chasm between a developer's local environment and the complex, interconnected reality of a deployed staging or QA environment is the birthplace of some of software engineering's most frustrating bugs. The infamous phrase, "but it works on my machine," is more than a meme; it is a testament to the emergent, unpredictable behaviors that arise from the interplay of shared databases, third-party APIs, network latency, and concurrent user actions. Reproducing these elusive, environment-specific issues is often the first and most challenging step toward resolving them. How can an engineer reliably test the resilience of an application against a database connection pool exhaustion that only occurs under specific load, or a third-party payment gateway that intermittently times out? Waiting for these conditions to manifest organically is inefficient and unreliable.

This is where one of software development's most powerful and controversial techniques enters the stage: monkey patching. At its core, monkey patching is the practice of dynamically modifying or extending code at runtime. In dynamic languages like Ruby, this means having the ability to reopen any class—even core language classes or those from third-party libraries—and change its behavior on the fly. While its use in production code is widely debated and often condemned as an anti-pattern that creates brittle, unmaintainable systems, its application within the controlled confines of staging and QA environments presents a compelling case. Here, monkey patching transforms from a potential architectural liability into a precision tool for chaos engineering, enabling engineers to simulate a vast array of failure modes, exceptions, and bugs with surgical accuracy.

This three-part series provides a comprehensive, publication-ready exploration of using monkey patching for fault injection in staging and QA environments, targeted at intermediate to senior software engineers working primarily within the Ruby and Rails ecosystem. In this first installment, we navigate the theoretical underpinnings and the long-standing debate over its classification as a design pattern or an anti-pattern, followed by a thorough examination of the significant security considerations. Part 2 will dive deep into practical implementation with numerous complete code examples, while Part 3 will equip you with a decision-making framework, best practices, and thought-provoking discussion points to foster a deeper understanding of this potent practice.

Design Pattern and Theoretical Foundation

The practice of monkey patching occupies a contentious and ambiguous space within the established lexicon of software design. It is a technique born of the extreme flexibility offered by dynamic languages, yet it often stands in direct opposition to the principles of structure, predictability, and encapsulation that underpin classical software architecture. To truly understand its role, particularly in the context of testing and fault injection, one must first explore the vigorous debate surrounding its identity, its deep connection to the broader concept of metaprogramming, and how it contrasts with the canonical design patterns that offer more structured solutions to similar problems.

The central debate is whether monkey patching should be classified as a pragmatic design pattern or a dangerous anti-pattern. Proponents of the anti-pattern classification build their case on a foundation of core software engineering principles that the practice inherently violates. Monkey patching undermines modularity by creating invisible, runtime dependencies between otherwise disconnected parts of a system. A patch applied to a core library class in one part of the application can have unforeseen and disastrous consequences in another, completely unrelated part. It shatters the principle of information hiding by reaching into the internal implementation details of a class or module to alter its behavior, creating a tight and brittle coupling that is prone to breaking when the underlying library is updated. This discrepancy between the static source code and the dynamic runtime behavior dramatically increases cognitive load for developers, making the system harder to reason about, debug, and maintain. In a collaborative environment, this can lead to a maintenance nightmare, where multiple developers unknowingly patch the same methods, creating subtle conflicts that manifest as maddeningly intermittent bugs.

Despite this formidable list of criticisms, a pragmatic defense of monkey patching persists, arguing that in certain contexts, it is not an anti-pattern but a necessary tool—a "lesser evil" when no better options are available. In staging and QA environments, its value proposition shifts. The goal is no longer long-term maintainability but controlled, temporary chaos. It serves as an indispensable technique for simulating failure modes in third-party dependencies, allowing teams to test retry logic, circuit breakers, and user-facing error handling without relying on the unpredictable availability of external systems. For emergency hotfixes in production, a carefully crafted monkey patch can be a lifeline, restoring critical functionality in minutes while a proper, permanent fix is developed and deployed through standard processes. Similarly, when integrating with poorly designed legacy systems or APIs that lack proper extension points, monkey patching can provide a crucial bridge to achieve necessary functionality. In these scenarios, its defenders argue, the immediate, practical benefits outweigh the long-term architectural risks, provided the patch is treated as a temporary, well-documented, and highly targeted intervention.

Fundamentally, monkey patching is a specific application of metaprogramming—the concept of code that writes or manipulates other code. This connection is vital because it frames monkey patching not as an isolated hack but as part of a spectrum of powerful, language-level capabilities that include introspection (examining code at runtime) and dynamic code generation. Ruby, in particular, has a culture that deeply embraces metaprogramming, making runtime modification a natural and accessible feature. This cultural acceptance is a double-edged sword. The same dynamic features that enable elegant Domain-Specific Languages (DSLs) and the magic of frameworks like Ruby on Rails can become significant security vulnerabilities when misused. A related metaprogramming concept, often seen in the Ruby world, is "duck punching," which typically involves modifying a single object instance at runtime to alter its behavior, offering a more localized and slightly less dangerous alternative to class-level monkey patching.

When contrasted with classical design patterns from the seminal "Gang of Four" book, the unstructured nature of monkey patching becomes even more apparent. The Decorator pattern, for instance, also adds behavior to objects dynamically, but it does so through composition, wrapping objects in decorator classes that share the same interface. This approach is explicit, maintainable, and respects object boundaries, unlike monkey patching, which directly modifies the original class. The Adapter pattern solves the problem of incompatible interfaces by creating a new adapter class that acts as a translator, preserving the integrity of the original classes. Monkey patching might be used to crudely force compatibility by altering a class directly, but the Adapter pattern provides a clean, architecturally sound solution. Similarly, the Strategy pattern encapsulates interchangeable algorithms, allowing a client to select one at runtime. While one could mimic this by swapping out method implementations with a monkey patch, the Strategy pattern does so in a structured, type-safe, and explicit manner. These classical patterns promote key architectural principles like high cohesion, separation of concerns, and explicit dependencies, all of which are subverted by the use of monkey patching, which tends to create hidden coupling and unpredictable side effects that make systems harder to scale, refactor, and maintain. The historical evolution of the practice across dynamic languages tells a consistent story: an initial period of enthusiastic adoption for its flexibility, followed by a gradual shift toward cautious, restrained application as teams encountered the painful realities of its impact on large-scale systems.

Security Considerations

While the architectural debates surrounding monkey patching often focus on maintainability and complexity, the security implications are equally, if not more, critical, particularly in environments that handle sensitive data or act as a gateway to production systems. The very nature of monkey patching—the ability to alter code behavior at runtime—creates a potent attack surface that can bypass traditional security controls and static analysis tools. For engineers leveraging this technique in staging and QA, a deep understanding of the associated risks, real-world attack vectors, and effective mitigation strategies is not just best practice; it is an absolute necessity to prevent these pre-production environments from becoming a weak link in the organization's security posture.

The vulnerabilities introduced by monkey patching can be categorized into several distinct classes. The most direct threat is code poisoning and injection. By providing a mechanism for runtime code modification, monkey patching opens a door for malicious actors to alter critical system logic. An attacker who gains access to a dependency or the application's deployment pipeline could inject a patch that modifies authentication methods to bypass login checks, alters data serialization routines to exfiltrate sensitive information, or overrides security validation functions to allow malicious input. Because these modifications happen dynamically, they can be designed to evade static code analysis, making them exceptionally difficult to detect through standard code reviews. In JavaScript ecosystems, a related vulnerability known as "prototype pollution," where an attacker modifies Object.prototype, can have a devastatingly broad impact, injecting malicious behavior into nearly every object in an application.

Another significant risk involves the subversion of language-level security features, such as Ruby's tainted data mechanism. Tainted mode is designed to track data originating from untrusted external sources (like user input) and prevent it from being used in security-sensitive operations. However, several documented Common Vulnerabilities and Exposures (CVEs) have shown how metaprogramming and dynamic modification can undermine these protections. For example, CVE-2018-16396 revealed that tainted flags were not properly propagated in Ruby's Array#pack and String#unpack methods, allowing untrusted input to bypass security checks. Similarly, CVE-2015-7551 highlighted unsafe tainted string usage in the Fiddle and DL libraries, which could lead to arbitrary code execution. These incidents demonstrate that monkey patching can create subtle holes in a language's built-in security model.

Perhaps the most pervasive and modern threat is the role of monkey patching in supply chain attacks. When an application relies on dozens or hundreds of third-party dependencies, each of those dependencies becomes a potential vector for attack. An attacker who compromises a popular open-source library can inject a malicious monkey patch that will then be propagated to every application that uses it. This was a key concern in incidents like the SolarWinds attack, where the update mechanism itself was abused to distribute compromised code. The MITRE supply chain attack framework explicitly identifies malicious code insertion during the build or deployment process as a common pattern, a threat that is significantly amplified by the dynamic nature of monkey patching. In staging and QA environments, these risks are often heightened. These environments may have more relaxed security controls, may contain production-like (or even real, poorly sanitized) data, and can serve as a valuable testing ground for an attacker to validate a malicious patch before attempting to deploy it to production. A compromised staging environment can become a pivot point for lateral movement into the production network.

The most infamous real-world example of monkey patching being weaponized was the 2009 conflict between the Firefox extensions NoScript and Adblock Plus. The developer of NoScript allegedly used monkey patching to inject code that would disable Adblock Plus on his own websites, effectively forcing users to see his ads. This escalated into a digital arms race, with each extension pushing updates to override the other's patches, trapping users in the middle. The incident served as a stark demonstration of how monkey patching could be used to sabotage competing software and manipulate user experience without consent, ultimately eroding trust in the entire ecosystem. More recent Ruby CVEs, such as CVE-2019-16255 (a code injection vulnerability in Shell#[]) and CVE-2016-2098 (an arbitrary code execution flaw in Rails' render method), further underscore how the metaprogramming features that enable monkey patching can be exploited to create critical security flaws.

Mitigating these substantial risks requires a multi-layered approach. Rigorous code review and static analysis, while not foolproof, can help by scanning for common patching patterns and auditing dependencies for suspicious modifications. Maintaining a Software Bill of Materials (SBOM) provides a clear inventory of all components and their versions, aiding in vulnerability management. Since static analysis is insufficient, runtime monitoring and dynamic analysis become paramount. Dynamic Application Security Testing (DAST) tools, behavior monitoring, and anomaly detection can identify unexpected changes in application behavior at runtime that may indicate a malicious patch. In terms of process, enforcing principles of least privilege, requiring cryptographic code signing, and mandating security reviews for any monkey patch are essential controls. Ultimately, the most effective mitigation is to prioritize safer alternatives whenever possible. Language features like Ruby's scoped Refinements, architectural patterns like Decorators and Dependency Injection, and configuration-based tools like feature flags can often achieve the same testing goals without introducing the profound security risks associated with global, runtime code modification.

Conclusion to Part 1

In this first installment of our series, we have established the theoretical and security foundations necessary to understand monkey patching as a tool for fault injection. We've explored its contentious position in software design—straddling the line between pragmatic necessity and dangerous anti-pattern—and examined how it relates to metaprogramming and contrasts with classical design patterns. More importantly, we've confronted the significant security implications that arise when code can be altered at runtime, from supply chain attacks to the subversion of language-level security features.

Understanding these risks and the theoretical context is crucial because it shapes how we approach the practical application of this technique. Armed with this knowledge, we can now move forward with appropriate caution and respect for the power we're wielding.

In Part 2, we will transition from theory to practice, diving deep into concrete Ruby and Rails implementation patterns. You'll learn how to use Module#prepend to create clean, effective monkey patches that simulate real-world failures like API timeouts, database deadlocks, race conditions, and memory leaks—all within the controlled environment of the Rails console. We'll also explore real-world case studies from companies like DoorDash and Shopify, learning from both their successes and their cautionary tales.

Continue to Part 2: Implementation & Practice → LATER

References

Design Patterns and Theory

Security

General Resources

From Script to Library: Building a Firefox Tab Extractor for the Open Source Community

Vinicius Porto — Mon, 25 Aug 2025 00:50:32 +0000

Introduction

What started as a simple script to organize my browser tabs evolved into a full-fledged Python library with CI/CD, comprehensive testing, and PyPI publishing. This article chronicles the journey of transforming a personal productivity tool into an open-source library that others can benefit from.

The Problem: Tab Management Chaos

As a developer and researcher, I often find myself with dozens of Firefox tabs open - documentation, tutorials, research papers, and GitHub repositories. The challenge? Keeping track of what's important, what I've already read, and what needs attention.

My initial solution was a Python script that:

Extracted Firefox session data from recovery.jsonlz4
Parsed tab information (title, URL, access time, pinned status)
Exported to CSV for Notion integration
Helped organize study materials and research

But this was just a local script. What if others could benefit from this tool?

The Transformation: From Script to Library

Phase 1: Restructuring the Codebase

The original script was a monolithic file with everything mixed together. The first step was applying software engineering principles:

# Before: Everything in one file
def extract_firefox_tabs():
    # 200+ lines of mixed concerns

# After: Modular architecture
firefox_tab_extractor/
├── __init__.py
├── models.py          # Data structures
├── extractor.py       # Core logic
├── exceptions.py      # Error handling
├── cli.py            # Command-line interface
└── tests/
    └── test_extractor.py

Key Technical Decisions:

Data Models: Used @dataclass for Tab and Window objects with type hints
Error Handling: Custom exception hierarchy for specific failure scenarios
Separation of Concerns: CLI, core logic, and data models in separate modules
Logging: Standard Python logging for debugging and user feedback

Phase 2: Modern Python Packaging

Gone were the days of setup.py. Modern Python packaging with pyproject.toml:

[project]
name = "firefox-tab-extractor"
version = "1.0.0"
description = "Extract and organize Firefox browser tabs"
authors = [
    {name = "Vinicius Porto", email = "vinicius.alves.porto@gmail.com"}
]
dependencies = ["lz4>=3.1.0"]

[project.optional-dependencies]
dev = ["pytest", "black", "flake8", "mypy", "pre-commit"]

[project.scripts]
firefox-tab-extractor = "firefox_tab_extractor.cli:main"

Benefits:

Single source of truth for project metadata
Modern dependency specification
Entry points for CLI tools
Tool configurations (Black, MyPy, Pytest)

Phase 3: Quality Assurance

A library needs to be reliable. This meant implementing comprehensive testing and code quality tools:

# Example test structure
class TestFirefoxTabExtractor:
    @patch('firefox_tab_extractor.extractor.os.path.exists')
    def test_extractor_initialization(self, mock_exists):
        mock_exists.return_value = False
        extractor = FirefoxTabExtractor()
        assert extractor is not None

Testing Strategy:

Unit Tests: Mock external dependencies (file system, Firefox profiles)
Integration Tests: Test the complete workflow
Error Scenarios: Test exception handling
Edge Cases: Empty profiles, corrupted data, missing files

Code Quality Tools:

Black: Consistent code formatting
Flake8: Linting and style enforcement
MyPy: Static type checking
Pre-commit: Automated quality checks

Phase 4: Continuous Integration/Deployment

Automation is key for open source projects. GitHub Actions workflows handle:

# .github/workflows/publish.yml
name: Publish to PyPI
on:
  release:
    types: [published]
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]

  build-and-publish:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Build package
        run: python -m build

      - name: Publish to PyPI
        env:
          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
        run: twine upload dist/*

CI/CD Benefits:

Automated testing across Python versions
Quality checks on every commit
Automated PyPI publishing on releases
Consistent deployment process

Technical Challenges and Solutions

Challenge 1: Firefox Session Data Format

Firefox stores session data in LZ4-compressed JSON files. The technical approach:

import lz4.frame
import json

def decompress_session_data(file_path: str) -> dict:
    """Decompress Firefox session data from LZ4 format."""
    with open(file_path, 'rb') as f:
        compressed_data = f.read()

    # Remove Firefox-specific header
    json_data = compressed_data[8:]

    # Decompress LZ4 data
    decompressed = lz4.frame.decompress(json_data)

    # Parse JSON
    return json.loads(decompressed.decode('utf-8'))

Challenge 2: Cross-Platform Profile Detection

Firefox profiles are stored differently across operating systems:

def find_firefox_profile() -> str:
    """Find Firefox profile directory across different OS."""
    if sys.platform == "darwin":  # macOS
        base_path = os.path.expanduser("~/Library/Application Support/Firefox/Profiles")
    elif sys.platform == "win32":  # Windows
        base_path = os.path.expanduser("~/AppData/Roaming/Mozilla/Firefox/Profiles")
    else:  # Linux
        base_path = os.path.expanduser("~/.mozilla/firefox")

    # Find the default profile
    profiles = glob.glob(os.path.join(base_path, "*.default*"))
    return profiles[0] if profiles else None

Challenge 3: Data Model Design

The challenge was creating intuitive data structures:

@dataclass
class Tab:
    window_index: int
    tab_index: int
    title: str
    url: str
    last_accessed: int
    pinned: bool
    hidden: bool

    @property
    def domain(self) -> str:
        """Extract domain from URL for categorization."""
        try:
            return urlparse(self.url).netloc
        except Exception:
            return "unknown"

    @property
    def last_accessed_datetime(self) -> datetime:
        """Convert timestamp to datetime object."""
        return datetime.fromtimestamp(self.last_accessed / 1000)

Challenge 4: Error Handling Strategy

Robust error handling was crucial for a library:

class FirefoxTabExtractorError(Exception):
    """Base exception for Firefox tab extractor."""
    pass

class FirefoxProfileNotFoundError(FirefoxTabExtractorError):
    """Raised when Firefox profile cannot be found."""
    pass

class SessionDataError(FirefoxTabExtractorError):
    """Raised when session data cannot be parsed."""
    pass

class LZ4DecompressionError(FirefoxTabExtractorError):
    """Raised when LZ4 decompression fails."""
    pass

The Open Source Journey

Why Open Source Matters

Open source libraries are the backbone of modern software development. They:

Accelerate Development: Developers don't reinvent the wheel
Improve Quality: Community review and contributions
Foster Learning: Code becomes documentation and examples
Build Ecosystems: Tools that work together

Documentation and Community

A good open source project needs:

Clear README: Installation, usage, examples
API Documentation: Function signatures, parameters, return values
Contributing Guidelines: How others can help
Issue Templates: Structured bug reports and feature requests
Code of Conduct: Welcoming environment

Example: Our Documentation Structure

# Firefox Tab Extractor

## Quick Start
pip install firefox-tab-extractor
firefox-tab-extractor --help

## Features
- 🔍 Smart profile detection
- 📁 Multiple output formats (JSON/CSV)
- 🏷️ Rich metadata extraction
- 📊 Statistics and analytics
- 🛠️ Developer-friendly API

## Usage Examples
from firefox_tab_extractor import FirefoxTabExtractor

extractor = FirefoxTabExtractor()
tabs = extractor.extract_tabs()
stats = extractor.get_statistics(tabs)

Lessons Learned

1. Start Small, Scale Gradually

The initial script was functional. The library evolved through iterations:

First: Modular structure
Second: Testing and quality tools
Third: CI/CD and automation
Fourth: Documentation and community

2. Testing is Investment, Not Overhead

Good tests pay dividends:

Confidence in changes
Documentation of behavior
Easier refactoring
Community contributions

3. Automation Reduces Friction

CI/CD workflows mean:

No manual deployment steps
Consistent quality standards
Faster feedback loops
Reduced human error

4. Documentation is Code

Good documentation:

Reduces support burden
Attracts contributors
Serves as specification
Improves user experience

The Result

What started as a personal script became:

A Python library with 1,000+ lines of code
Comprehensive testing with 90%+ coverage
Automated publishing to PyPI
Cross-platform support (macOS, Windows, Linux)
Multiple output formats (JSON, CSV)
Rich metadata extraction (domains, timestamps, pinned status)
Command-line interface for easy use
Developer-friendly API for integration

Impact and Usage

The library enables workflows like:

# Study organization
tabs = extractor.extract_tabs()
study_tabs = [tab for tab in tabs if "tutorial" in tab.title.lower()]
extractor.save_to_csv(study_tabs, "study_materials.csv")

# Productivity analysis
stats = extractor.get_statistics(tabs)
print(f"Most visited domain: {stats['top_domains'][0]}")
print(f"Total reading time: {stats['estimated_reading_time']} hours")

# Notion integration
windows = extractor.get_windows(tabs)
for window in windows:
    print(f"Window {window.window_index}: {window.tab_count} tabs")

Conclusion

Building an open source library is more than just writing code. It's about:

Engineering Excellence: Clean architecture, testing, documentation
Community Building: Welcoming contributors, clear guidelines
Automation: CI/CD, quality tools, deployment pipelines
User Experience: Intuitive APIs, helpful error messages

The journey from script to library taught me that open source is about making tools that others can build upon. It's about contributing to the ecosystem that has given us so much.

The code is available at: github.com/ViniciusPuerto/firefox-tab-extractor

Install with: pip install firefox-tab-extractor

What started as a personal productivity tool became a contribution to the open source community. The next time you find yourself writing a script that others might find useful, consider taking that extra step to make it a proper library. The community will thank you for it.

Hanoi Tower with procs

Vinicius Porto — Thu, 21 Nov 2024 13:43:52 +0000

The Tower of Hanoi is a classic algorithmic problem that can be solved using recursive techniques. We can apply the concepts of procs and lambdas to implement a solution for the Tower of Hanoi problem in Ruby. Here's an example:


def tower_of_hanoi(n, source, destination, auxiliary, move_callback)
  if n == 1
    move_callback.call(source, destination)
  else
    tower_of_hanoi(n-1, source, auxiliary, destination, move_callback)
    move_callback.call(source, destination)
    tower_of_hanoi(n-1, auxiliary, destination, source, move_callback)
  end
end

move_proc = lambda { |source, destination| puts "Move disk from #{source} to #{destination}" }

tower_of_hanoi(3, 'A', 'C', 'B', move_proc)

In this implementation, the tower_of_hanoi method takes the number of disks (n), names of the source (source), destination (destination), and auxiliary (auxiliary) towers, as well as a move_callback proc as arguments.

The tower_of_hanoi method follows the recursive algorithm for solving the Tower of Hanoi problem. If n is 1, it simply calls the move callback to move the disk from the source to the destination tower. Otherwise, it recursively solves the problem for n-1 disks by moving them from the source tower to the auxiliary tower, then moves the largest disk from the source to the destination, and finally solves the problem for n-1 disks by moving them from the auxiliary tower to the destination tower.

We define a move_proc lambda that prints the move operations. You can pass any other proc or lambda with a different behavior to customize how the moves are handled.

Finally, we call the tower_of_hanoi method with n = 3, source tower 'A', destination tower 'C', auxiliary tower 'B', and the move_proc lambda as the move callback.

When you run this code, it will print the sequence of moves required to solve the Tower of Hanoi problem with 3 disks. You can adjust the value of n and the tower names to solve the problem for different numbers of disks or with different tower names.

Using Value Objects in Ruby on Rails

Vinicius Porto — Sat, 01 Apr 2023 04:10:00 +0000

In this article, we explore the concept of Value Objects in Ruby on Rails and how they can be used to encapsulate business logic and improve the maintainability of your code. We start by explaining what a Value Object is and how it differs from a Rails Model, before diving into how to create custom Value Objects in Ruby. We also provide an example of how to use a custom Value Object as an attribute in a Rails Model, along with the necessary configuration and implementation details. Finally, we discuss how to organize Value Objects within a Rails application for maximum clarity and ease of use.

Can a Rails Model be Considered a Value Object?

No, a Rails Model is not the same thing as a Value Object. A Value Object is a small, immutable object that represents a simple value or concept, such as a name, address, or monetary amount. In contrast, a Rails Model is a more complex object that represents a database table and includes functionality for querying and updating the associated records.

Can We Use Value Object Concepts in a Rails Model?

Yes, you can use Value Object concepts in a Rails Model by creating custom Value Objects and using them as attributes in your Model. This allows you to encapsulate business logic in a separate object and keep your Model lean and focused on database-related concerns.

Should We Create Separate Models or Use Value Objects?

It depends on the complexity of the business logic you need to represent. If the business logic involves complex relationships or requires extensive querying and updating of associated records, it may be better to create separate Models. If the business logic is relatively simple and can be represented as a standalone value or concept, it may be better to use a Value Object.

Creating a Value Object

Here is an example of how you might create a Value Object for representing monetary amounts:

class Money
  attr_reader :amount, :currency

  def initialize(amount, currency = 'USD')
    @amount = amount
    @currency = currency
  end

  def add(other)
    raise "Mismatched currency" unless currency == other.currency

    Money.new(amount + other.amount, currency)
  end

  # Other methods for performing arithmetic operations, formatting output, etc.
end

In this example, we define a Money class that represents a monetary amount with an associated currency. The class includes methods for performing arithmetic operations and other functionality related to monetary amounts.

Another example is a Value Object to represents an address:

class Address
    attr_reader :street, :number, :complement, :city, :state, :zip_code

    def initialize(street:, number:, complement:, city:, state:, zip_code:)
      @street = street
      @number = number
      @complement = complement
      @city = city
      @state = state
      @zip_code = zip_code
    end

    def eql?(other)
      return false unless other.is_a?(Address)
      street == other.street &&
        city == other.city &&
        state == other.state &&
        zip_code == other.zip_code
    end
    alias_method :==, :eql?

    def hash
      [street, number, complement, city, state, zip_code].hash
    end

    def to_s
      "#{street}, #{number}, #{complement} - #{city}/#{state}, #{zip_code}"
    end
end

Using a Value Object in a Rails Model

Here is an example of how you might use the Address Value Object as an attribute in a Rails Model:

class Order < ApplicationRecord
    attribute :shipping_address, AddressType.new  
end

using this method you need to make some aditional configurations on config/initializers/types.rb to register the new custom type like:

ActiveSupport.on_load(:active_record) do
    ActiveRecord::Type.register :address_type, AddressType
end

you still have to create a class, that inherits from ActiveRecord::Type::Value , that describes the convertion of the Address value object into a really usable type.

class AddressType < ActiveRecord::Type::Value
  def type
    :jsonb
  end

  def cast(value)
    return if value.blank?
    if value.is_a?(Address)
      value
    elsif value.is_a?(Hash)
      Address.new(value.symbolize_keys)
    else
      raise ArgumentError, "Invalid address value: #{value.inspect}"
    end
  end

  def serialize(value)
    return nil if value.blank?

    value.to_s
  end

  def deserialize(value)
    value
  end

  def ==(other)
    other.is_a?(AddressType)
  end
end

In the type method we are defining the real type that ‘ll be saved on the database
In the cast method has the responsability to casts a value from user input (e.g. from a setter). This value may be a string from the form builder, or a ruby object passed to a setter. There is currently no way to differentiate between which source it came from. The return value of this method will be returned from [ActiveRecord::AttributeMethods::Read#read_attribute](https://api.rubyonrails.org/classes/ActiveRecord/AttributeMethods/Read.html#method-i-read_attribute).
The serialize method casts a value from the ruby type to a type that the database knows how to understand. The returned value from this method should be a String, Numeric, Date, Time, Symbol, true, false, or nil .
and the deserialize method converts a value from database input to the appropriate ruby type. The return value of this method will be returned from ActiveRecord::AttributeMethods::Read#read_attribute. The default implementation just calls [Value#cast](https://api.rubyonrails.org/classes/ActiveModel/Type/Value.html#method-i-cast) .

Organizing Value Objects in a Rails Application

To organize your Value Objects in a Rails application, you can create a separate directory for them within the app directory. Here is an example directory structure:

app/
  ├── controllers/
  ├── models/
  │   ├── order.rb
  ├── value_objects/
  │   └── money.rb
      └── address.rb

In this example, we create a value_objects directory within the app directory to store our custom Value Objects. We then create a money.rb file within the value_objects directory to define our Money Value Object.

We can also use `composed_of` to use the value object

You can use the composed_of method in Rails to simplify the use of the Money value object in the Order model.

Here’s an example of how you could use composed_of:

class Order < ApplicationRecord
  composed_of :total_value,
              class_name: 'Money',
              mapping: %w[total_value amount currency],
              converter: ->(value) { Money.new(value.amount, value.currency) },
              allow_nil: true
end

In this example, the composed_of method is used to define a total_value attribute in the Order model. The :class_name option specifies the name of the value object class (in this case, Money). The :mapping option maps the total_value attribute to the cents attribute of the Money object. The :converter option specifies a lambda that converts the total_value attribute to a Money object. The :allow_nil option specifies that the total_value attribute can be set to nil.

With this setup, you can treat the total_value attribute as a Money object in your code, like this:

order = Order.new(total_value: Money.new(100, 'usd'))
order.total_value # returns a Money object with usd100

This approach allows you to abstract away the details of how the Money object is stored in the database and provides a simpler interface for working with the total_value attribute.

Conclusion

In conclusion, using value objects in a Ruby on Rails application can help you encapsulate business logic in a separate object and keep your models focused on database-related concerns. By using value objects to represent simple values or concepts, such as monetary amounts or addresses, you can make your code more readable, testable, and maintainable.

Creating a value object is relatively easy in Ruby, and you can use it as an attribute in your Rails models. You can also create custom types to handle the conversion of your value object to a database column. By organizing your value objects in a separate directory, you can make it easier to maintain and reuse them across your application.

references:

ActiveRecord::Aggregations::ClassMethods. Ruby on Rails API Documentation. Retrieved March 31, 2023, from https://api.rubyonrails.org/classes/ActiveRecord/Aggregations/ClassMethods.html
ActiveModel::Type::Value. Ruby on Rails API Documentation. Retrieved March 31, 2023, from https://api.rubyonrails.org/classes/ActiveModel/Type/Value.html#method-i-serialize
Fowler, M. (2003, August 20). ValueObject. Martin Fowler. Retrieved March 31, 2023, from https://martinfowler.com/bliki/ValueObject.html

DEV Community: Vinicius Porto

When the Scraper Breaks Itself: Building a Self-Healing CSS Selector Repair System

The Problem with Fragile Selectors

Core Idea in One Paragraph

Design Principles

LLM as proposer, not decider

Sandbox before promotion

Escalation over hallucination

Decoupling via Redis

Local inference

Traceability by default

System Context

End-to-End Pipeline

Deep Dives

Token budget and the context_levels retry loop

Prompting for structured output

Why type validation is not enough

Limitations and Honest Boundaries

Roadmap: From Reactive Repair to Proactive Extraction Engine

The Staging Ground Symbiote: A Deep Dive into Monkey Patching for Bug and Exception Simulation (Part 1: Foundations & Risks)

Introduction

Design Pattern and Theoretical Foundation

Security Considerations

Conclusion to Part 1

References

Design Patterns and Theory

Security

General Resources

From Script to Library: Building a Firefox Tab Extractor for the Open Source Community

Introduction

The Problem: Tab Management Chaos

The Transformation: From Script to Library

Phase 1: Restructuring the Codebase

Phase 2: Modern Python Packaging

Phase 3: Quality Assurance

Phase 4: Continuous Integration/Deployment

Technical Challenges and Solutions

Challenge 1: Firefox Session Data Format

Challenge 2: Cross-Platform Profile Detection

Challenge 3: Data Model Design

Challenge 4: Error Handling Strategy

The Open Source Journey

Why Open Source Matters

Documentation and Community

Example: Our Documentation Structure

Lessons Learned

1. Start Small, Scale Gradually

2. Testing is Investment, Not Overhead

3. Automation Reduces Friction

4. Documentation is Code

The Result

Impact and Usage

Conclusion

Hanoi Tower with procs

Using Value Objects in Ruby on Rails

Can a Rails Model be Considered a Value Object?

Can We Use Value Object Concepts in a Rails Model?

Should We Create Separate Models or Use Value Objects?

Creating a Value Object

Using a Value Object in a Rails Model

Organizing Value Objects in a Rails Application

We can also use composed_of to use the value object

Conclusion

We can also use `composed_of` to use the value object