I Built an Agent That Found Real Duplicate Content in My Production SEO Database

Asghar Shakir — Sat, 19 Sep 2026 09:57:39 +0000

I run RankinSEO, an AI-powered SEO article platform for trades businesses (roofing, HVAC, plumbing, solar, electrical). Over the past few months I've manually hunted down duplicate content in my Supabase database by eyeballing a sorted table — tedious, error-prone, and something I kept meaning to automate.

For the Sanity Challenge, I finally built that automation as a real agent.

What I built

Migrated a curated sample of my live content (5 articles per trade) from Supabase into Sanity's Content Lake as structured article documents
Wrote a GROQ-powered agent that queries the content and analyzes title similarity to flag potential duplicates
Ran it against my full 130-article production table to see if it would re-discover issues I'd already found manually

The GROQ query at the core of it

*[_type == "article"] {
  title,
  slug,
  trade
} | order(trade asc)

The agent then normalizes each title (strips stopwords and punctuation) and computes word-overlap similarity between every pair, flagging anything above a 40% threshold.

What it found

Running against my full dataset, the agent correctly re-discovered several duplicate clusters I'd already manually identified and fixed back in September — like "Gutter Cleaning Cost" vs "Gutter Cleaning Cost 2026" (75% overlap) and "Commercial Roofing" vs "Commercial Roofing Company" (67% overlap).

It also honestly surfaced a limitation: a Calgary-localized article with a typo in its slug ("roofing-contrators-near-me") got flagged as similar to several generic "near me" articles — purely because of shared words like "roofing" and "near me" — even though the actual content and intent were genuinely different. Pure word-overlap similarity can't distinguish "same topic, different angle" from "actual duplicate." That's a real constraint worth knowing before trusting this kind of tool blindly.

Why this matters

This isn't a toy demo — it's solving a problem I actually have, using my actual production content. The agent won't replace human judgment on which "duplicates" to merge (context still matters), but it turns a 20-minute manual scan into a 3-second automated first pass.

Stack

Next.js / Supabase (source data)
Sanity (Content Lake + GROQ)
Node.js

Built for the Sanity Challenge, Path One.

Doing everything right, but still on 2 clicks on my website

Asghar Shakir — Sat, 19 Sep 2026 02:30:16 +0000

Created an AI powered SEO website, that I always wanted to do, but having trouble with Seo, it's been 3 months still fighting with it everyday.

Registered with a lot of directories and tried a few high domain authority websites for backlinks, but no use.

Generating at least 5 articles daily, but failed.

Did a lot of keyword researches and also 'what people looking for' tried.

Hope Dev Community would be able to help!

Website for trade businesses and free audit for everyone.

Seo #RankinSEO, #TradeBusinesses, #WebAudit

Don't see any SEO discussion in here! Lets start a thread?

Asghar Shakir — Sat, 19 Sep 2026 02:21:24 +0000

DEV Community: Asghar Shakir