DEV Community: 大橙子

Building a Self-Hosted Market Research Engine: Architecture, Scraping, and AI Analysis

大橙子 — Fri, 19 Jun 2026 10:55:44 +0000

Building a Self-Hosted Market Research Engine: Architecture, Scraping, and AI Analysis

When I started my last indie project, I needed to know two things: who's competing in my space, and what are they saying? Simple question. But the tools I found were either $200/month (hello, Semrush) or locked me into their data silo.

So I built my own. Self-hosted. Python-based. With actual control over my data.

Here's how it works under the hood — and why I think most indie devs should consider this approach instead of signing up for yet another SaaS.

The Core Architecture

The system has three layers, running in Docker:

┌─────────────────────┐
│   Data Collection    │  ← Scrapy + Playwright
├─────────────────────┤
│   Analysis Pipeline  │  ← LLM summarization + NLP
├─────────────────────┤
│   API + Dashboard    │  ← FastAPI + SQLite/Postgres
└─────────────────────┘

All three run on a $12/month VPS. No external API calls except the LLM endpoint.

Layer 1: Not Your Grandma's Web Scraper

The tricky part isn't scraping. It's scraping at scale without getting blocked. Here's what I learned:

Proxy rotation isn't optional. After about 50 requests to the same domain, you're cooked. I built a simple proxy rotator that pulls from a pool of residential proxies:

class RotatingProxyMiddleware:
    def __init__(self, proxies: list[str]):
        self.proxies = proxies
        self._index = 0
        self._lock = Lock()

    def fetch(self, url: str) -> Response:
        with self._lock:
            proxy = self.proxies[self._index % len(self.proxies)]
            self._index += 1
        return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)

Dead simple. But it solves 90% of the block problem. The last 10% is solved by random delays (2–5 seconds between requests) and realistic user-agent headers.

Playwright for JS-heavy sites. Some competitor sites are full SPA — no static HTML, all JavaScript rendering. For those, I drop into Playwright:

async def scrape_dynamic(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        content = await page.content()
        await browser.close()
        return content

This adds about 3 seconds per page, but it's the only way to get the actual rendered content.

Layer 2: The Analysis Pipeline — Where AI Actually Helps

Once you have the raw data (pricing pages, product descriptions, changelogs), you need to extract signal from noise. This is where LLMs shine — not for generating content, but for structuring it.

I use a cheap model (Claude Haiku or GPT-4o-mini) with a structured prompt:

ANALYSIS_PROMPT = """Extract the following from this competitor page:
1. Pricing tiers and amounts (list all)
2. Target customer (individual/team/enterprise)
3. Key features mentioned
4. Landing page positioning
Return as JSON."""

def analyze_page(content: str) -> dict:
    response = llm_client.chat(
        model="claude-3-haiku",
        messages=[{"role": "user", "content": ANALYSIS_PROMPT + "\n\n" + content[:8000]}],
        response_format={"type": "json_object"}
    )
    return json.loads(response.content)

The key insight: you don't need a huge context window. Most pages have the important info in the first 5000 characters. Truncate aggressively, batch cheap calls, and store results in SQLite.

Layer 3: The Dashboard

I built a minimal FastAPI dashboard that shows:

Competitor pricing changes over time (screenshot diff)
Feature comparison matrix (auto-populated)
Raw changelog entries with AI summaries

The dashboard runs on the same VPS. No frontend framework — just Jinja2 templates and htmx for interactivity. It's ugly, but it works, and I can access it from my phone.

Why Self-Hosted Matters

Three reasons:

Your data stays yours. Every market analysis tool I tried uploads your competitors' data to their databases. That's fine until you're analyzing something sensitive.
Cost scales to zero. The VPS costs $12/month regardless of how many competitors I track. SaaS tools charge per project/per seat.
You own the pipeline. Found a new data source? Add it in 20 lines of Python. Want a custom metric? Write it. No submitting feature requests.

The Stack I Wish I Started With

If I were building this again today, I'd start with:

Scrapy for crawling (not raw requests — Scrapy handles retries, backpressure, and pipelines out of the box)
SQLite for storage (faster than Postgres for read-heavy workloads, and zero ops)
Claude Haiku for analysis (cheapest capable model I've found)
htmx for dashboard (no JS build step)

What's Next

I've packaged this into a tool called MarketEye — it's the exact system I described above, with a ready-to-run Docker setup and a few pre-built scrapers for common competitor sources (Product Hunt, G2, Crunchbase, etc.).

If you want to check it out:

GitHub (open-source core): https://github.com/dachengzi065-gif/marketeYE
Gumroad (pre-built Docker image + scrapers): https://gumroad.com/l/kvnkhb

But honestly? Even if you never use MarketEye, I hope this post shows you that building your own self-hosted intelligence pipeline is totally doable. You don't need enterprise tools. You just need Python, a VPS, and a weekend.

Have you built any self-hosted tooling for your indie projects? I'd love to hear what you're running — drop a comment below.

I Open-Sourced MarketEye — Here's Why (and the GitHub Link)

大橙子 — Tue, 16 Jun 2026 03:35:29 +0000

I open-sourced MarketEye today.

For anyone who missed the first post: MarketEye is a self-hosted competitor price monitor I built because I didn't want to pay $99/month for Prisync.

The code is now up on GitHub under MIT license.

GitHub: github.com/dachengzi065-gif/marketeye

Why open source?

Three reasons:

1. People actually asked for it.

After my first post here, a few people DM'd me asking to see the code. They're developers too — they want to modify it, extend it, make it their own. That's fair. Selling source code to devs is like selling ice to eskimos.

2. Trust.

A closed-source price tracker that "runs on your machine" — you either trust the author or you don't. Open source removes that doubt. You can read every line, check what data leaves your machine (nothing), and build it yourself if you want.

3. Longevity.

Self-hosted tools have a dirty secret: if the developer disappears, you're stuck with a broken tool. Open source changes that. Even if I get hit by a bus tomorrow, you can fork the repo and keep going.

What this means for the $49 version

The Gumroad package still exists. It includes:

The same code, pre-packaged
Email support (I'll help you set it up)
A clear conscience subscription (you're paying for convenience, not software)

But honestly? If you can run pip install, just clone the repo. It's free.

What's next

I'm actively working on:

Docker image (one-command deploy)
More scrapers (plugins for different sites)
Discord/Telegram bot alerts (requested by several people)

PRs welcome. Issues welcome. Feedback welcome.

👉 github.com/dachengzi065-gif/marketeye

I Built a Self-Hosted Competitor Price Monitor Because SaaS Pricing Is Out of Control

大橙子 — Tue, 16 Jun 2026 01:29:40 +0000

I Built a Self-Hosted Competitor Price Monitor Because SaaS Pricing Is Out of Control

The Problem

I run a small online store. To stay competitive, I needed to track my competitors' pricing. The obvious solution? Sign up for a SaaS price monitoring tool.

Then I saw the prices:

Prisync: $99/month
Price2Spy: $20-58/month
Keepa: $29/month (Amazon only)

For a small operation, $99/month is absurd. So I built my own.

What I Built: MarketEye

MarketEye is a self-hosted competitor price monitor. You run it on your own machine, point it at any product URL, and it tracks price changes automatically.

Tech stack: Python + FastAPI + SQLite + APScheduler

Features:

Drop any product URL → auto price tracking
Email + webhook alerts on price changes
Price history charts + CSV/JSON export
Works with Amazon, Shopify, JD, Taobao, any HTML page
Web dashboard included (no CLI needed)

How It Works

Add a product URL through the web dashboard
MarketEye scrapes the page at configurable intervals
When a price changes, you get an alert
All data stays on your machine - no cloud, no third party

The scraping engine handles various e-commerce platforms out of the box and provides a plugin interface for custom scrapers.

Why Self-Hosted?

One-time payment: $49 vs $99/month ($3,564 over 3 years)
Unlimited products: no per-product fees
Data privacy: your pricing data never leaves your machine
Customizable: full Python source, extend it however you want

What's Next

I'll be sharing more technical deep-dives on the architecture, scraping challenges, and lessons learned from building and shipping this as a solo developer.

Check it out: MarketEye on Gumroad

Questions and feedback welcome! 🚀

Hello Dev.to 👋 — A Developer Joining the Community

大橙子 — Tue, 16 Jun 2026 01:22:25 +0000

Hey everyone!

Just created my Dev.to account and wanted to say hi properly.

I'm a software developer, working with Python and
building side projects that solve real problems.

My latest project is MarketEye — a self-hosted competitor price
monitor I built because I refused to pay $99/month for basic
price tracking. Python + FastAPI + SQLite, scrapes product
pages, sends alerts on price changes, generates history charts.

What I want to share here:
• Building and shipping a self-hosted product from idea to launch
• Web scraping techniques that actually work (and the pitfalls)
• Pricing and marketing indie software as a dev selling globally
• Technical deep-dives on Python, FastAPI, and deployment
• Getting your first customer without spending on ads

Looking for honest feedback and connecting with other indie
builders. Happy to help with anything Python/scraping related.

Thanks for having me! 🚀