DEV Community: Joe Slade

How I Built a SERP Topic Gap Analyzer on Apify (And What the Prefill Bug Taught Me)

Joe Slade — Wed, 06 May 2026 02:18:11 +0000

Most content gap tools work backwards. They start with keyword data and try to reverse-engineer what your competitors rank for. I wanted to start from the other direction: given a set of real SERP results, which topics are your competitors covering that you're not?

That question led to the SERP Topic Gap Monitor — a deterministic, composable Apify actor that takes pre-fetched SERP data as input, runs a topic extraction and gap-scoring pipeline, and returns a ranked list of coverage gaps your site is missing.

Here's how it's built and what I learned along the way.

The Core Design Decision: Accept Data, Don't Fetch It

The first architectural choice was also the most important: the actor doesn't scrape Google. It accepts pre-fetched SERP results as structured input.

This sounds like a limitation. It's not.

Fetching Google reliably is a whole problem on its own — proxies, rate limits, bot detection, constantly shifting HTML structure. Bundling that complexity into a content analysis tool creates a fragile system where the analysis breaks every time Google changes its layout.

By accepting SERP data as input, the actor is decoupled from any specific data source. You can feed it results from Google Search API, SerpAPI, Apify's own SERP scrapers, or a fixture file you built by hand. The analysis stays stable regardless of where the data comes from.

The input schema is straightforward:

{
  "targetDomain": "yourblog.com",
  "serpResults": [
    {
      "keyword": "best nootropics",
      "results": [
        {
          "url": "https://healthline.com/...",
          "title": "...",
          "snippet": "...",
          "pageContent": "..."
        }
      ]
    }
  ]
}

pageContent is optional — when it's present, the topic extraction is richer. When it's not, the actor falls back to title and snippet.

The Pipeline: Four Pure Functions

The analysis runs as a linear pipeline of pure functions:

tokenize → topics → gaps → score

tokenize — takes raw text (title, snippet, page content), strips HTML, lowercases, removes punctuation, filters a bundled English stopword list (~170 words, NLTK/Snowball-based), and returns a token array.

topics — aggregates tokens across a result set into a frequency map. Higher frequency = more likely to be a genuine topic signal vs. noise.

gaps — compares your domain's topic set against the competitor topic set. Any topic present in competitor results but absent from your pages is a gap.

score — assigns a gapScore to each gap:

gapScore = uniqueCompetitorPages / totalUniqueCompetitorPages

A score of 1.0 means every competitor page in the result set covers this topic. Your site covers none of them. That's where you start writing.

The formula is simple on purpose. No black box, no magic weighting. If you understand the ratio, you understand the output.

Test-First, All the Way Down

Because all the logic is pure functions, the entire pipeline is fully testable without network access. I wrote 54 Vitest specs across 5 files before writing a line of implementation.

The test suite covers:

Tokenization edge cases (empty strings, HTML entities, mixed case, punctuation boundaries)
Stopword filtering (ensuring common words don't inflate topic scores)
Gap detection (correct identification of covered vs. uncovered topics)
Scoring math (boundary conditions at 0 and 1.0)
End-to-end pipeline integration with fixture SERP data

Running npm test gives you full confidence the analysis is correct before you push anything to the platform. This matters more than you'd think — debugging a scoring error in a live Apify run is significantly slower than catching it in a local test.

The Bug That Took Me Three QA Failures to Find

After publishing, I got a notice from Apify: the actor had been flagged "Under Maintenance" after three consecutive automated QA failures.

The error wasn't in the analysis code. It was in input_schema.json.

Apify's automated QA system runs each actor using the prefill values from the input schema as test input. If required fields don't have prefill values, the QA run receives undefined for those fields — which causes the actor to fail immediately.

My targetDomain and serpResults fields had no prefill. So every QA run hit this:

Actor.fail("targetDomain is required")

The fix took five minutes. Adding a prefill to both fields:

"targetDomain": {
  "type": "string",
  "prefill": "myblog.com"
},
"serpResults": {
  "type": "array",
  "prefill": [ /* 2-keyword fixture array */ ]
}

Build 0.1.4 deployed, QA passed, maintenance flag removed.

The lesson: Always set prefill for every required field in your Apify input_schema.json. The QA system can't test what it can't input. I've carried this forward to every actor since.

A Real Run

I ran it against one of my own sites — peakhealthprovisions.com, a wellness/nootropics site — using two keywords and five competitors (Healthline, Examine, WebMD, MedicalNewsToday, NootropicsExpert).

Results: 20 gaps, 0 topics covered.

nootropic   → gapScore: 1.0  (5/5 competitors)
nootropics  → gapScore: 1.0  (5/5 competitors)
cognitive   → gapScore: 0.8  (8 unique pages)
memory      → gapScore: 0.7  (7 unique pages)
focus       → gapScore: 0.6  (6 unique pages)

Every competitor is covering these topics. The site isn't covering any of them. That's not a content calendar suggestion — that's the content calendar.

What's Next

The actor's design creates an explicit upstream dependency: you need SERP data to feed it. One logical next step is a dedicated SERP actor with a clean JSON output schema designed to pipe directly into the Gap Monitor — one actor fetches, one actor analyzes, composable by design.

There's also a "Works well with" section in the README that cross-references the Changelog Triage Agent — another actor in the portfolio for teams who want to monitor API changelogs for breaking changes. Different use case, same philosophy: focused, testable, composable.

Try It

The actor is free to use on the Apify Store:
👉 apify.com/joeslade/serp-topic-gap-monitor

The README has a full walkthrough of the input format, output schema, and how to source SERP data to feed it. If you're building it into a content pipeline or have questions about the architecture, drop a comment — happy to dig into it.

Tags: apify seo typescript webdev devtools

What's Next for PolicyPilotPro?

Joe Slade — Sat, 05 Jul 2025 23:01:07 +0000

This is a submission for the World's Largest Hackathon Writing Challenge: After the Hack.

PolicyPilotPro addresses a critical need in the startup ecosystem by making compliance accessible, automated, and aligned with the fast-paced nature of technology companies. But this hackathon project is just the beginning.

Immediate Roadmap:

Market Validation: Target 10-20 high-urgency customers actively preparing for SOC 2 certification or Series A funding
Feature Enhancement: Expand compliance framework database and implement advanced analytics
Partnership Development: Build relationships with compliance consultants and startup accelerators

Long-Term Vision:

With a clear development roadmap and pricing strategy aligned with customer value ($99-$499/month based on company size), PolicyPilotPro is positioned to capture significant market share in the growing GRC and legal tech markets. Our target: $100K+ MRR within 18 months, with the potential to scale into a $10M ARR business.

The Bigger Picture:

This isn't just about building another SaaS product. It's about democratizing compliance for the next generation of innovative companies, ensuring that regulatory requirements become a competitive advantage rather than a growth barrier.

PolicyPilotPro represents the future of startup operations: AI-powered, proactive, and perfectly integrated into the workflows that drive innovation forward.

Built with Bolt.new - Powered by AI - Designed for the Future

PolicyPilotPro was built entirely using Bolt.new in less than 48 hours

Joe Slade — Sat, 05 Jul 2025 22:48:03 +0000

This is a submission for the World's Largest Hackathon Writing Challenge: Building with Bolt.

What PolicyPilotPro Does

PolicyPilotPro is an AI-powered compliance automation platform that transforms regulatory burden into strategic advantage for startups and growing companies. Think of it as having a compliance co-pilot that never sleeps, never misses a regulatory update, and never lets you fly blind into legal turbulence.

Core Capabilities:

Intelligent Document Scanning: Upload your existing policies, tech stack documentation, and workflows—our AI instantly identifies compliance gaps across multiple frameworks
Real-Time Policy Generation: Need a GDPR privacy policy? SOC 2 documentation? Our AI generates customized, legally-informed policies in minutes, not months
Regulatory Change Monitoring: Stay ahead of evolving regulations with proactive alerts that assess impact on your specific business context
Workflow Integration: Seamlessly connects with Notion, Linear, and Google Drive to make compliance part of your existing processes

The Problem We Solve:

In a market projected to grow by $44.22 billion from 2025-2029, startups are caught in a compliance paradox. They need robust regulatory frameworks to secure funding, enterprise deals, and SOC 2 certification, but they lack the resources for dedicated legal teams. PolicyPilotPro bridges this gap, serving as a "compliance insurance policy" for fast-moving teams who can't afford to slow down but also can't risk regulatory missteps.

How We Built It

The Tech Stack Symphony:

Bolt.new as our primary development platform, enabling rapid full-stack application creation
Supabase for scalable database architecture and real-time functionality
AI Integration leveraging GPT-4o for document analysis and policy generation
Modern UI/UX with responsive design optimized for startup workflows

The beauty of this approach? What traditionally would require months of development, multiple team members, and significant infrastructure investment was accomplished in a single weekend by leveraging the democratized power of AI-assisted development.

Challenges We Ran Into

The Full-Stack Reality Check: While I have a solid background in software development, my expertise has primarily been in front-end engineering. It's been years since I've navigated the complexities of full-stack architecture, database design, and server-side logic.

But here's where the paradigm shift becomes tangible. Bolt.new and Supabase didn't just make these challenges manageable—they made them disappear. Creating database schemas, implementing third-party integrations, and building server-side edge functions became as intuitive as describing what I wanted to accomplish.

The Compliance Complexity: Building a platform that handles multiple regulatory frameworks (SOC 2, GDPR, CPRA, AI-specific regulations) while maintaining accuracy and legal validity required a careful balance between automation and human oversight.

Accomplishments We're Proud Of

Speed of Innovation: Developing a complex, fully functional MVP application over a single weekend isn't just impressive—it's a testament to how AI-powered development tools are redefining what's possible for individual creators and small teams.

Market Validation: Our solution addresses a real pain point in the $46.76 billion legal technology market, specifically targeting the underserved startup segment.

Technical Achievement: Successfully integrating AI-powered document analysis, policy generation, and regulatory monitoring into a cohesive platform that actually works—not just a demo, but a production-ready solution.

User Experience: Creating an intuitive interface that makes complex compliance concepts accessible to non-legal professionals, complete with dashboard analytics, compliance scoring, and actionable recommendations.

What We Learned

This hackathon became a masterclass in modern development capabilities. I discovered that Bolt.new, combined with providers like Supabase, ElevenLabs, and other GTM tech solutions, can deliver real-world results with dramatically less time and fewer resources than traditional development approaches.

Key Insights:

AI-assisted development isn't just faster—it's fundamentally different, enabling individual creators to build enterprise-grade solutions
The barrier between idea and implementation has essentially vanished for those willing to embrace these new tools
Complex business logic, database relationships, and user interfaces can be created through conversational development