DEV Community: Haimanot Getu

How Beaconmon Monitors Shopify Competitor Storefronts at Scale (Without Playwright)

Haimanot Getu — Wed, 10 Jun 2026 06:16:42 +0000

A practical look at content monitoring at scale without Playwright, without scraping farms, and without setting a single VPS on fire.

The 2am Alert That Started This Post

A skincare brand using Beaconmon woke up to an alert. A top competitor had quietly dropped prices across their bestselling serums by 18% and was running a sitewide free-shipping promo. No press release. No announcement. Just a Tuesday.

Their team repriced and matched the shipping threshold before the competitor's paid ads started driving volume. They saw it because a background worker had fetched that competitor's storefront, extracted the right DOM nodes with cheerio, diffed the result against a 24-hour-old snapshot, and scored the change as high-significance.

That is the whole product. Everything below is how we make it reliable at scale.

TL;DR

BullMQ workers fetch competitor HTML on a schedule
cheerio extracts price, promo, and product-grid content using a ranked selector cascade
Diffs are normalized, stored, and scored by rules first, AI second
No Playwright, no headless Chrome, no scraping farm
The lesson that mattered most: normalize before storage, or you will alert on whitespace forever

The Architecture in One Paragraph

Every tracked competitor is a Monitor record flagged is_competitor: true. A BullMQ scheduler enqueues a content-check job for each one on its configured interval (15 minutes on free, down to 5 on Growth). Workers pull jobs, fetch the page with undici, parse with cheerio, normalize the text, and compare against the last snapshot in Postgres. If anything changed, a second job scores the significance and fans out alerts.

No Playwright. No headless Chrome. HTML-only, on purpose. A single VPS will not survive 3,000 concurrent Chromium processes, and the signal we care about (price text, promo copy, product grid content) almost always exists in the initial HTML response.

Step 1: Map the Catalog Once, Then Leave It Alone

Shopify exposes a public JSON endpoint for any store's product catalog. We fetch it once at setup time to build an internal product map:

type ProductMap = Record<string, {
  title: string;
  variants: Variant[];
  priceRange: [number, number];
}>;

That map is what lets us say "3,000 products" instead of "3,000 URLs." We do not poll that endpoint on every check cycle. It is the seed, not the heartbeat.

The heartbeat is cheerio against the live HTML.

Step 2: The Selector Cascade

Shopify themes are not standardized, but they rhyme. Debut, Dawn, and most third-party themes share a small vocabulary of price-related class names. We try a ranked list of selectors and take the first match:

const PRICE_SELECTORS = [
  '.price__sale',
  '.price-item--sale',
  '.price__regular',
  '.price-item--regular',
  '.product-form__price',
  '.product__price',
  '[data-product-price]',
  '[data-price]',
  '[class*="price__"]',
  '[class*="ProductPrice"]',
  '.price',
];

If none match, we fall back to main. A fallback match is recorded but never marked as "detected," so a miss on the specific selector does not produce a false-positive event.

That distinction matters. We separate "we found the thing we wanted" from "we tracked something and it changed." If you collapse those two states, your alert quality dies.

The same pattern applies to announcement bars, collection grids, and sale pages. Each preset is a ranked selector list plus a fallback:

{
  id: 'announcement-bar',
  selectors: [
    '.announcement-bar',
    '.shopify-section--announcement-bar',
    '[class*="AnnouncementBar"]',
    '[data-announcement-bar]',
  ],
  fallbackSelector: 'header',
}

Step 3: The Actual Content Check

The fetch and parse is about 30 lines. The important parts are normalization, and using undici over node-fetch for connection pooling at volume.

import { request } from 'undici';
import * as cheerio from 'cheerio';

const { statusCode, body } = await request(url, {
  method: 'GET',
  headersTimeout: timeoutMs,
  bodyTimeout: timeoutMs,
  headers: {
    'user-agent': 'beaconmon/1.0 (+https://beaconmon.com)',
    accept: 'text/html,application/xhtml+xml',
  },
});

const html = await body.text();
const $ = cheerio.load(html);
const rawText = selector ? $(selector).text() : $('body').text();

const content = rawText
  .split('\n')
  .map((l) => l.replace(/[ \t]+/g, ' ').trim())
  .filter((l) => l.length > 0)
  .join('\n');

The normalization step matters more than it looks. Shopify themes render whitespace inconsistently across CDN edges and A/B test variants. Without collapsing horizontal whitespace per line and stripping blanks, you get false-positive diffs every few hours on high-traffic stores. With it, the diff is quiet until something actually changes.

Step 4: Score What Changed

Not all diffs are equal. A competitor updating their "About" copy is noise. A competitor dropping 18% off their hero SKU is a buying signal.

We score in two layers.

Rules First (fast, free, zero latency)

case 'price_changed': {
  const delta = Math.abs(next - old) / old;
  if (delta >= 0.05) return 'high';
  return 'medium';
}
case 'out_of_stock':
case 'back_in_stock':
case 'sale_started':
  return 'high';

AI Second (Growth and Scale plans only, fails open)

The diff text goes to Claude with a short system prompt describing the scoring rubric. It returns low, medium, or high with a 4-second hard timeout. If the model is unreachable or the plan does not include AI, the rule-based score is used. The system never blocks on the AI layer.

What the Data Actually Enables

This is the part that matters to the stores using it.

Repricing with context, not fear. When you see a competitor's price drop tagged high at 2am, you have a diff, a timestamp, and the old price. You are not guessing. You are deciding.

Promo calendar reconstruction. Six weeks of announcement-bar snapshots tells you when a competitor runs their sales, how long they run, and what copy they use. That is a content calendar you did not have to build yourself.

New-arrival velocity. A collection-grid monitor tells you how fast a competitor is dropping product. Twice a week means real buying power or a production operation worth watching. Once a month means they are coast-clearing.

Stock signal for sourcing. Out-of-stock events on a competitor's bestsellers can indicate a supply chain gap. That is a window to step up your own inventory or run a targeted ad against their unmet demand.

None of that requires crawling at scale or violating terms of service. It is the same HTML a browser would render, fetched politely with a declared user-agent, on a reasonable interval.

Three Lessons I'd Tell My Past Self

1. The selector cascade is the hard part, not the queue

Shopify's theme ecosystem is large enough that there is no single correct selector. A ranked list with a named fallback, and a clear distinction between "matched" and "fell back," is the right model. Get this wrong and alert quality collapses.

2. Normalize before storage. Always.

Storing raw HTML and diffing later sounds appealing until you realize CDN-injected attributes, nonce values in inline scripts, and whitespace drift will fire alerts on every check for a percentage of your monitors. Store the normalized text.

3. Fail open everywhere

The AI layer falls back. The selector cascade falls back. The check itself records the error type and moves on rather than retrying forever. A monitoring product that generates noise when its own dependencies hiccup is worse than useless.

Wrap

If you are building something similar, or you run a Shopify or WooCommerce store and want to see this in action, Beaconmon is in early access. I am happy to talk architecture, selector strategies, or where the AI layer earns its keep versus where rules are enough.

Find me at @haimanot_getu.

I built a bootstrapped uptime and competitor intelligence SaaS for eCommerce

Haimanot Getu — Mon, 08 Jun 2026 10:11:44 +0000

I run a small side project called Beaconmon. It monitors websites for uptime, tracks competitor pricing and promotions, and sends intelligence digests to eCommerce store owners. Shopify and WooCommerce are the primary target.

Here is how it is built and the decisions that shaped it.

The problem

Most uptime tools are overkill for a small eCommerce store. A Shopify merchant does not need Datadog. They need to know when their store goes down, when a competitor drops prices before a sale weekend, and when a checkout page breaks.

The interesting constraint: competitor tracking is not just a nice-to-have. It is the primary reason someone would pay for this over a free tier of Pingdom.

Tech stack

Next.js 15 (App Router) for the dashboard and the marketing site, combined in one app
TypeScript everywhere in a pnpm monorepo
PostgreSQL + Prisma for data, with a separate workers process for background jobs
BullMQ + Redis for the job queue
better-auth for authentication (sessions, OAuth)
Freemius as the merchant of record (not Stripe, more on that below)
Resend + React Email for transactional and lifecycle emails
Sentry + PostHog for error tracking and analytics
Docker Compose for both local dev and production on a single VPS

The monorepo has two apps and several packages:

apps/
  web/       # Next.js: dashboard + marketing
  workers/   # Background job workers
packages/
  checker/   # Core check evaluation logic
  queue/     # BullMQ queue definitions
  db/        # Prisma schema + repositories
  types/     # Shared TypeScript types
  billing/   # Freemius integration
  ai/        # AI provider abstraction (noop by default)
  logger/    # Structured logging (Pino)

Three architectural decisions worth talking about

1. Two consecutive failures before marking a monitor down

False positives are the biggest trust killer in uptime monitoring. A single timeout from a flaky CDN node will fire a 2am alert and destroy your credibility with the user.

The evaluation logic in packages/checker/ requires two consecutive failures before transitioning a monitor to down. On the first failure it records the check result but holds the incident open. If the next check passes, it closes clean. If it fails again, it fires.

This cut false positive alerts to near zero in testing against real Shopify stores.

2. Status pages read from Redis, not Postgres

Status pages are ISR with a 30-second revalidate. When a store goes down, the merchant's customers hit the status page simultaneously. That is easily 100 to 300 requests per minute on a small VPS.

If that read hits Postgres directly, the database becomes the bottleneck at exactly the worst moment. Instead, the workers write incident state to Redis on every check, and the status page reads from there. Postgres is never in the critical path during an incident.

3. Content monitoring with Cheerio, no Playwright

The competitor tracking feature diffs HTML content over time to detect price changes and promotional banners. The obvious tool is a headless browser, which handles JavaScript-rendered content perfectly.

I do not run Playwright in the worker fleet. The entire production setup runs on a single VPS. A Playwright worker pool would consume 2 to 4 GB of RAM and make the whole deployment fragile.

Cheerio handles the vast majority of eCommerce content just fine, JavaScript-rendered content is explicitly out of scope, and RAM stays predictable.

The billing choice: Freemius over Stripe

Freemius is a merchant of record for software. They handle VAT, sales tax, and payment processing. For a bootstrapped solo project, not dealing with tax compliance in 50 US states and the EU is worth the higher fee.

The tradeoff: Freemius has a clunkier API and their SDK is less polished than Stripe's. I wrapped it in a packages/billing package so the rest of the codebase does not care which billing provider is underneath.

One plan ID per tier, not two (monthly/annual). Freemius handles the billing cycle internally as an integer (1 = monthly, 12 = annual).

The workers

The workers app runs a set of specialized BullMQ workers:

check-worker runs HTTP checks on a schedule
content-worker diffs HTML snapshots for content monitoring
competitor-page-worker fetches competitor pages
alert-worker sends notifications through configured channels (email, Slack, Discord, SMS, webhook)
digest-worker generates intelligence digests
scheduler enqueues jobs at the correct intervals

The scheduler is pinned to one replica. BullMQ uses Redis for job storage with noeviction policy and AOF durability. Job data must not be evicted.

Alert sends are idempotent: before sending any notification, the worker checks for an existing send for the same (incident_id, channel_id) within the last 5 minutes.

This handles the case where a worker crashes mid-send and retries.

Tenant isolation

Every web-facing database function takes teamId as its first argument.

No exceptions.

This is enforced by convention and code review, not by row-level security at the database layer. Worker repositories are exempt because job payloads are already tenant-scoped at enqueue time.

Where it is now

Beaconmon is live.

The onboarding wizard detects Shopify and WooCommerce stores at step one and bulk-creates the competitor monitors at the end of the flow rather than incrementally.

If you are curious about the codebase or have questions about any of these decisions, drop them in the comments.

Built with Next.js, BullMQ, Prisma, and too much Redis.