DEV Community

Apex Stack
Apex Stack

Posted on

Agentic SEO Is Here: How I Use AI Agents to Manage a 100,000-Page Website

Everyone's talking about AI agents replacing DevOps engineers and writing code. But nobody's talking about what happens when you point AI agents at SEO.

I run StockVS.com, a programmatic SEO site with over 100,000 pages covering 8,000+ stock tickers across 12 languages. Managing a site this size manually is impossible. So I built an agentic workflow that handles everything from content generation to cross-platform publishing to technical auditing.

Here's exactly how the system works, what surprised me, and where AI agents still fall short.

What "Agentic SEO" Actually Means

There's a difference between using AI to write a blog post and using AI agents to operate an SEO pipeline.

Traditional AI-assisted SEO looks like this: you open ChatGPT, ask it to write an article, copy-paste it into WordPress, and manually optimize it. That's AI as a tool.

Agentic SEO is different. You define the workflow once — data collection, content generation, quality checks, publishing, monitoring — and agents execute the entire pipeline autonomously. You shift from being the operator to being the architect.

For StockVS, that pipeline looks like this:

Data Layer:     yfinance API → Supabase PostgreSQL (8,000+ tickers)
Content Layer:  Local Llama 3 → 600-800 word analysis per ticker
Build Layer:    Astro static site → 12 language variants
Distribution:   Cloudflare CDN → Digital Ocean Spaces
Monitoring:     Google Search Console → Custom dashboards
Publishing:     AI skill → Dev.to + Medium + Hashnode simultaneously
Enter fullscreen mode Exit fullscreen mode

Each layer runs with minimal manual intervention. That's the shift.

Agent 1: Content Generation at Scale With a Local LLM

The backbone of StockVS is a local Llama 3 instance generating unique stock analysis for every ticker. Not generic filler — each page includes valuation metrics, financial health indicators, sector positioning, and a forward-looking assessment.

I covered this in detail in a previous article, but the key insight is cost. Running Llama 3 locally means generating 100,000+ pages of content costs exactly $0 in API fees. The tradeoff is compute time and the work of building the pipeline, but once it's running, the marginal cost of adding a new ticker is near zero.

What I'd do differently: I initially generated 300-400 word pages. Google flagged most of them as thin content (50,000+ pages "crawled but not indexed" in Search Console). After expanding to 600-800 words with more unique data points per page, the quality signal improved significantly.

Lesson: AI agents are only as good as the quality bar you set for them. An agent that generates fast but shallow content will scale your problems, not your results.

Agent 2: Cross-Platform Publishing in Under a Minute

Writing articles is half the battle. The other half is getting them in front of people across multiple platforms without spending 30 minutes copy-pasting and reformatting.

I built a publishing agent that takes a markdown article and pushes it to Dev.to, Hashnode, and Medium simultaneously. It handles:

  • Platform-specific formatting differences
  • Canonical URL management (so Google doesn't see duplicate content)
  • Tag optimization per platform
  • Frontmatter translation

A single command takes an article from draft to live on three platforms. The same article, properly formatted for each platform's quirks, with canonical URLs pointing back to the primary source.

Before this, publishing a single article across three platforms took 25-30 minutes of manual work. Now it takes under 60 seconds.

The Blog Cross-Publisher is available as a Claude skill if you want to try this workflow yourself.

Agent 3: Programmatic SEO Auditing

When you have 100,000+ pages, you can't manually check each one for SEO issues. You need an agent that crawls your own site, identifies problems, and prioritizes fixes.

My auditing agent checks for:

  • Missing or duplicate meta descriptions
  • Thin content pages (under word count threshold)
  • Broken internal links
  • Missing schema markup
  • Hreflang consistency across 12 languages
  • Canonical tag mismatches (Google flagged 297 pages for this)
  • Orphaned pages with no internal links

The output is a prioritized list of issues sorted by impact. Fix the top 10 items and you move the needle more than fixing 1,000 random pages.

This is where agents genuinely outperform humans. No person can audit 100,000 pages for hreflang consistency. An agent does it in minutes.

The Programmatic SEO Auditor is the skill I packaged from this workflow.

Agent 4: Financial Data Analysis

StockVS pages need actual financial data, not just AI-generated text. Each stock page pulls live data through yfinance: price history, P/E ratios, dividend yields, revenue growth, debt levels, and more.

An analysis agent processes this raw data and generates the narrative sections — "AAPL is trading at 28x earnings, above its 5-year average of 24x, suggesting the market is pricing in continued growth from its services segment."

This is where combining structured data with LLM reasoning gets interesting. The agent isn't hallucinating numbers — it's reading real financial data from a database and generating contextual analysis around verified facts.

I open-sourced a lite version of this as the Financial Data Analyzer skill.

The Hard Truth: What Agents Can't Do (Yet)

After 6 months of building agentic SEO workflows, here's where agents still struggle:

1. Editorial judgment. Agents can generate content that's technically accurate and well-structured, but they can't tell you if a page is interesting. The difference between a page that ranks #3 and one that ranks #30 is often the quality of insight, not the quality of grammar.

2. Link building. This is still fundamentally a human relationship activity. Agents can identify link opportunities and draft outreach emails, but the actual relationship — the "hey, I noticed your resource page" conversation — requires human nuance. My site has zero backlinks despite 100K+ pages, and no agent has solved that yet.

3. Strategic pivots. When Google's indexing data showed 50,000+ pages as "crawled but not indexed," the decision to remove comparison pages entirely and double down on stock analysis pages was a strategic call. Agents can surface the data, but the judgment to cut 200,000 pages requires understanding Google's signals in context.

4. Understanding Google's mood. Google's indexing behavior is partially opaque. Why did my Dutch pages get more impressions than English ones? Why did a copper industry page get a click before any individual stock page? Agents can track these patterns, but interpreting them still requires human pattern recognition.

The Real Numbers

Transparency matters more than hype, so here's where things actually stand:

Metric Current
Total pages 100,000+ across 12 languages
Pages indexed by Google 2,246 (2.1%)
Search clicks (3 months) 3
Impressions (3 months) 2,180
Average position 52.5
Backlinks 0
Revenue $0

Those numbers are humbling. And they're real.

The indexing rate is the critical blocker. Google has crawled 50,000+ pages and rejected them, and another 51,000+ pages it discovered but won't even crawl yet. Until the site builds authority through backlinks and proves content quality, the agentic pipeline is generating pages that Google isn't serving.

But here's why I'm still optimistic: the pipeline is built. When indexing improves — through backlink building, content quality signals, and time — the system can scale instantly. I don't need to manually write 100,000 pages. The agent infrastructure is ready.

How to Start With Agentic SEO

If you want to experiment with agentic SEO workflows, start small:

Step 1: Automate your publishing pipeline first. Cross-posting is the lowest risk, highest time-savings entry point.

Step 2: Build a site auditing agent. Even for a 50-page site, automated auditing catches things you'd miss manually.

Step 3: Add data-driven content generation for one content type. Don't try to automate everything at once.

Step 4: Monitor Google Search Console data programmatically. Build alerts for indexing drops, crawl errors, and ranking changes.

The goal isn't to remove humans from SEO. It's to let humans focus on strategy and relationships while agents handle the repetitive execution at scale.

What's Next

I'm currently focused on three things:

  1. Content thickening — adding related news, analyst ratings, and earnings data to each stock page to increase the quality signal
  2. Internal linking — building cross-references between stock, sector, and ETF pages so Google can better understand site structure
  3. Backlink building — the one thing agents can't automate, and the biggest blocker to indexing

If you're building something similar, or if you've cracked the indexing challenge at scale, I'd love to hear about it in the comments. What's your experience with using AI agents for SEO tasks?


I build tools for programmatic SEO and AI-powered content workflows. Check out the Apex Stack collection of Claude skills for SEO, publishing, and financial analysis.

Top comments (0)