I've wasted days of my life maintaining CSS selectors.
You know the drill — you write the perfect scraper, it works great for a week, then the site does a frontend redesign, your selectors break, and you spend another afternoon hunting through the DOM again.
So I built Opticparse — a completely different approach.
How It Works
Instead of selectors, Opticparse:
- Opens a real Chromium browser (via Playwright)
- Navigates to your URL and waits for JavaScript to load
- Screenshots the page
- Sends the image + your plain English query to a Vision-Language model (rotating through Groq, Gemini, and GitHub Models to prevent rate limits)
- Extracts and returns clean, structured JSON matching your target schema
Because it uses AI vision to look at the page exactly like a human does, it never breaks when the HTML structure changes.
🛠️ The Tech Stack
I built this using:
- FastAPI (Python): High-performance backend routing
- Playwright: Handles headless rendering, waits for dynamic content, and takes screenshots
- OpenAI / Gemini SDKs: Communicates with the vision models
- Free Key Rotation: Cascading fallback rotation across 6 providers so it runs completely for free
💻 Code Example: Scrape E-Commerce Prices
Here is how simple it is to query. You just pass the URL, a query prompt, and a JSON schema you want it to output:
bash
curl -X POST https://opticparse-python-sg.onrender.com/api/vision-scrape \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{
"target_url": "https://example.com/store",
"extraction_query": "Extract the product name, price, and currency",
"response_schema": {
"type": "object",
"properties": {
"product_name": {"type": "string"},
"price": {"type": "number"},
"currency": {"type": "string"}
}
}
}'
## 🚀 Get Started
I've open-sourced the code and deployed the API to Render.
1. **GitHub Repository**: If you want to run it locally or host it yourself: [parastejpal987-cmyk/opticparse](https://github.com/parastejpal987-cmyk/opticparse)
2. **RapidAPI Hub**: Access the free API tiers immediately: [Opticparse on RapidAPI](https://rapidapi.com/parastejpal987cmyk/api/opticparse-ai-vision-web-scraper)
Let me know what you think in the comments! Happy scraping.
Top comments (1)
That selector maintenance pain is all too real. I still remember the multi-day sprint we had a few years back when a major e-commerce client pushed a new site design, completely blowing up our product data ingestion pipeline. We were running a few hundred individual scrapers, and the cascade of failures was brutal. It really makes you question the robustness of any system reliant on specific DOM structure.
Moving away from explicit selectors towards a "plain English" model for extraction is a fascinating direction. The fundamental challenge with natural language queries on unstructured web data often boils down to ambiguity and context. For instance, if a page has multiple dates or prices, how does the system determine which one the "plain English" query refers to without some form of deeper semantic understanding or user feedback? I'm curious about the underlying heuristics or ML models you're employing to bridge that gap between human intent and the specific element on the page, especially across widely varying site layouts.
While CSS selectors are undeniably brittle to structural changes, their explicit nature means when they break, they do so predictably. A "plain English" system, if not robustly trained and maintained, could potentially introduce a different kind of "breakage" – subtle misinterpretations or incorrect data extractions that are harder to debug without explicit logs of the model's decision-making. The resilience to minor UI tweaks is a huge win if you can nail the consistency, but I imagine it's a continuous battle against the ever-evolving web and diverse semantic meanings.