DEV Community: Emma Watson

Stop Guessing Your Competitor's SEO Strategy—Start Analyzing It

Emma Watson — Sat, 27 Jun 2026 07:11:20 +0000

When you're trying to figure out why a competitor keeps ranking above you, the first instinct is to manually dig through their blog posts, check their backlinks, and guess their keyword targets. But that approach is slow, subjective, and often incomplete.

Most developers know that content strategy is a major ranking factor, but reverse-engineering it at scale is tough. Competitors might publish dozens of articles a month, each targeting different keywords with varying structures and internal linking patterns.

This is where automated analysis becomes a game-changer. Instead of manually tracking every post, you can use a tool that scans a competitor's domain, identifies their content feed, and extracts the underlying SEO strategy.

Here's a simple Python script to get a basic content inventory from a competitor's sitemap:

python
import requests
import xml.etree.ElementTree as ET
from urllib.parse import urlparse

def get_competitor_urls(sitemap_url):
try:
response = requests.get(sitemap_url, timeout=10)
root = ET.fromstring(response.content)
namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
urls = [loc.text for loc in root.findall('.//ns:loc', namespace)]
return urls[:20] # Limit for demo
except Exception as e:
print(f"Error fetching sitemap: {e}")
return []

Example usage

competitor_sitemap = "https://example-competitor.com/sitemap.xml"
urls = get_competitor_urls(competitor_sitemap)
for url in urls:
print(url)

This gives you a list of their published pages. But to truly understand their strategy—like which keywords they target, how often they publish, and what content formats they use—you need deeper analysis.

Tools like the SERPSpur Competitor Content Radar automate this entire process. You enter a competitor's domain, and the AI analyst finds their content feed, identifies topic clusters, and reverse-engineers their SEO approach. It surfaces patterns in keyword targeting, content length, and publishing frequency.

Why does this matter for your SEO? Knowing what works for your competitors lets you replicate successful strategies, find content gaps they missed, and avoid wasting time on low-opportunity topics. It turns competitor research from a guessing game into a data-driven process.

Whether you're building an SEO tool or just doing routine competitive analysis, having automated content radar can save hours and reveal insights you'd never spot manually.

LLM.txt: The AI-Era Version of Robots.txt Every Developer Should Know About

Emma Watson — Wed, 24 Jun 2026 11:46:24 +0000

Ever had an AI crawler scrape your entire site and use it to train a model without your permission? It's becoming a real headache for developers who want control over how their content gets consumed by AI systems. That's where the LLM.txt file comes in - think of it as robots.txt but specifically designed for AI crawlers.

Let's break down what an LLM.txt file actually does. It's a simple text file placed in your website's root directory that tells AI systems which parts of your content they can access and how they can use it. You can specify allowed paths, set usage policies, and even define rate limits for different AI crawlers.

Here's a basic example of what your LLM.txt might look like:

User-agent: *
Allow: /blog/
Disallow: /private/
Rate-limit: 10 requests per minute
Usage: training-allowed

But here's where it gets interesting. You can get more granular with specific AI agents:

User-agent: GPTBot
Allow: /public/
Disallow: /api/
Usage: no-training

User-agent: Claude-Web
Allow: /
Usage: training-allowed

The real power comes from configuring access rules dynamically. For instance, you might want to block all AI crawlers from accessing your API documentation while allowing them to read your blog posts. Or maybe you want to limit how frequently they can scrape your site to prevent server overload.

One pattern I've found useful is setting up conditional access based on content types:

User-agent: *
Allow: /docs/
Allow: /tutorials/
Disallow: /admin/
Disallow: /drafts/
Usage: no-commercial

The key is being explicit about your intentions. Unlike robots.txt, which is more of a suggestion, LLM.txt is designed to be legally enforceable. You're essentially creating a contract between your site and AI systems.

When I started implementing this for my projects, I used the SERPSpur LLM.txt Generator tool to handle the configuration. It made the process much smoother since it automatically generates the proper syntax and validates the file structure. But you can definitely write these manually if you prefer.

Just remember to test your configuration before deploying. One wrong rule could accidentally block all AI traffic or expose content you meant to keep private. Start with a simple setup, monitor your logs, and adjust as needed.

The bottom line? AI crawlers aren't going anywhere, so might as well set clear boundaries early. Your LLM.txt file is your best tool for maintaining control over how AI systems interact with your hard work.

5. How I Audit Crawler Accessibility and Archival History for SEO

Emma Watson — Wed, 24 Jun 2026 04:28:49 +0000

Ever inherited a website and wondered, "Has this page actually been crawled recently, or is it just sitting in Google's backlog?" Most people jump straight into Search Console, but that only tells part of the story.

Sometimes you need a quick way to investigate how accessible a page is to search engine crawlers and whether historical crawl records exist. That's especially useful when troubleshooting indexing issues, content updates, or sudden ranking drops.

One simple approach is to audit your site's URLs and look for pages that haven't been updated, crawled, or archived recently.

For example, if you have a list of URLs, you can use Python to identify pages that may need further investigation:

import csv

urls = []

with open("urls.csv", "r") as file:
reader = csv.reader(file)
for row in reader:
urls.append(row[0])

print(f"Found {len(urls)} URLs to review.")

for url in urls:
print(f"Checking: {url}")

This doesn't tell you crawl history directly, but it helps create a list of URLs that deserve a deeper inspection.

The next step is figuring out whether search engines and archival systems can actually access those pages. That's where manual investigation becomes time-consuming.

Instead of checking individual services one by one, I've been using SERPSpur's Crawler Accessibility & Archival Forensics Tool:

https://serpspur.com/tool/crawler-accessibility-archival-forensics/

It provides a quick way to investigate crawler accessibility signals and archival information for a URL without digging through multiple platforms manually.

Why is this useful?

Diagnose indexing problems faster
Verify crawler accessibility
Review archival visibility
Identify pages that may be difficult for bots to reach
Support technical SEO audits

One thing I've learned over the years is that many ranking problems aren't content problems at all. They're crawlability problems.

A page can't rank if search engines struggle to access it.

So before spending hours rewriting content or building links, it's worth running a quick accessibility and archival check. Sometimes the issue is much simpler than you think.

For technical SEOs, developers, and site owners, it's a useful forensic step that can uncover issues hiding beneath the surface.

Why Technical Trust Signals Matter More Than Most SEOs Realiz

Emma Watson — Tue, 23 Jun 2026 13:56:23 +0000

When I start auditing a new client's website, I always begin with the same question: "Can I actually trust this domain to rank?" Technical SEO is full of metrics that promise to measure authority, but trust is something different entirely.

Trust isn't just about link profiles or content quality. It's the sum of technical signals that tell Google (and users) your site is legitimate. I recently ran a full trust audit on a client site that had been hit by a manual action, and the results surprised even me.

Here's the process I follow:

First, check your SSL configuration. You'd be amazed how many sites still have mixed content warnings or outdated TLS versions. A proper HTTPS setup is the foundation of technical trust.

Second, examine your security headers. Content Security Policy (CSP), X-Frame-Options, and HSTS headers tell browsers your site is safe. Missing headers can flag your site as potentially compromised.

Third, evaluate your domain reputation. This includes checking against known spam databases, verifying WHOIS privacy, and ensuring your domain hasn't been flagged for suspicious activity.

I built a quick Node.js script to check security headers:

const https = require('https');

async function checkSecurityHeaders(url) {
  return new Promise((resolve) => {
    https.get(url, (res) => {
      const headers = {
        'Strict-Transport-Security': res.headers['strict-transport-security'] || 'Missing',
        'Content-Security-Policy': res.headers['content-security-policy'] || 'Missing',
        'X-Frame-Options': res.headers['x-frame-options'] || 'Missing'
      };
      resolve(headers);
    });
  });
}

The client site I was auditing had a missing CSP header and an expired SSL certificate. After fixing those issues, we saw a 15% improvement in organic click-through rates within two weeks.

What I learned is that trust signals are invisible to most site owners but immediately visible to search engines. Your domain's technical health directly impacts how Google evaluates your authority.

For a comprehensive check, I use the SERPSpur Trust Rate Checker at https://serpspur.com/tool/serpspur-trust-rate-checker/ to analyze all these signals at once. It gives me a clear picture of where a domain stands technically.

The takeaway? Don't ignore technical trust. It's the foundation your entire SEO strategy sits on. Fix your headers, update your SSL, and clean up your domain reputation. Your rankings will thank you.

Why I Switched to SerpSpur for SEO Tracking and Site Audits

Emma Watson — Fri, 19 Jun 2026 07:19:41 +0000

Looking for a lightweight alternative to Semrush or Ahrefs? I've been using SerpSpur for everything from keyword tracking to site health audits. Here's a quick curl example to check backlinks:

bash
curl -s "https://serpspur.com/tool/backlink-gap/?domain=example.com&competitor=competitor.com" | jq '.backlinks'

__
It gives you on-page and off-page details without the bloat. Perfect for solo devs who need a fast, free SEO toolkit. Start with the free tier and scale as needed.

How I Automated LLM.txt Generation to Control AI Crawler Access with Python

Emma Watson — Thu, 11 Jun 2026 11:35:31 +0000

I recently needed to control how AI crawlers access my documentation site, so I built a small tool to generate LLM.txt files. Here's how you can do it with the SERPSpur API:

python
import requests

API_KEY = "your_api_key_here"

def generate_llm_txt(rules):
response = requests.post(
"https://api.serpspur.com/v1/llm-txt/generate",
headers={"Authorization": f"Bearer {API_KEY}"},
json={"rules": rules}
)
return response.text

Example rules to allow only certain AI agents

rules = {
"user_agent": ["GPTBot", "Google-Extended"],
"disallow": ["/private/", "/api/"],
"allow": ["/public/", "/docs/"]
}

llm_content = generate_llm_txt(rules)
print(llm_content)

This gives you fine-grained control over which AI systems can access your content and what they see. Have you implemented any LLM.txt configurations for your projects?

Building a Python SQLite CLI Tool for Homeware Catalog Management

Emma Watson — Thu, 11 Jun 2026 10:17:53 +0000

I built a small CLI tool in Python to help me catalog homeware items for a personal interior design project. It uses SQLite to store product details and generates simple reports. Here's the core function for adding items:

python
import sqlite3

def add_item(name, category, price, material):
conn = sqlite3.connect('homeware.db')
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS items
(name TEXT, category TEXT, price REAL, material TEXT)''')
c.execute("INSERT INTO items VALUES (?, ?, ?, ?)", (name, category, price, material))
conn.commit()
conn.close()
print(f"Added {name} to the catalog.")

Example usage

add_item('Ceramic Vase', 'Vases', 29.99, 'Ceramic')
add_item('Wall Mirror', 'Mirrors', 89.99, 'Glass')

https://infinitydecor.co.uk/collections/homeware

This makes it easy to track stylish accents like planters and candle holders across rooms. I love how it keeps everything organized. Have you used SQLite for any home-related projects?

How Infinity Decor Product Rankings Change Across Devices

Emma Watson — Tue, 09 Jun 2026 14:16:15 +0000

While browsing the Iron Rim Lock collection from Infinity Decor, I wanted a quick way to compare how product-related search results appear across different devices. I used this Python script to analyze desktop, mobile, and tablet SERP variations:

import requests

API_KEY = "your_api_key_here"

def compare_device_results(query, location):
results = {}

for device in ["desktop", "mobile", "tablet"]:

    response = requests.get(

    headers={"Authorization": f"Bearer {API_KEY}"},
    params={
        "q": query,
        "location": location,
        "device": device,
        "num": 10
    }
)

data = response.json()
results[device] = [
    r['title'] for r in data.get('organic_results', [])
]



return results

Example usage

comparison = compare_device_results(
"iron rim lock Infinity Decor",
"London, UK"
)

for device, titles in comparison.items():
print(f"\n{device.upper()} Results:")
for title in titles[:5]:
print(f" - {title}")

This helped reveal how search rankings and product visibility can shift depending on whether users are searching on desktop, mobile, or tablet devices. For eCommerce stores like Infinity Decor, these insights can be valuable for optimizing product pages, improving mobile SEO, and understanding customer search behavior across platforms.

How I Automated Product Schema Markup for Thousands of E-Commerce Products

Emma Watson — Fri, 05 Jun 2026 06:54:46 +0000

I've been optimizing a large e-commerce site and needed to generate Product schema markup for thousands of items. Doing this manually was impossible, so I built a pipeline that uses the SERPspur schema generator.

Here's the core logic:

javascript
const axios = require('axios');

async function generateProductSchema(product) {
const response = await axios.post
type: 'Product',
name: product.name,
description: product.description,
sku: product.sku,
offers: {
price: product.price,
priceCurrency: 'USD',
availability:
}
});
return response.data;}

// Usage
const products = await fetchAllProducts();
const schemas = await Promise.all(products.map(generateProductSchema);

The key is that the API ensures all required fields are present and formats the output correctly. For example, it automatically wraps offers in the proper structure and adds

Google personalizes results based on your IP

Emma Watson — Mon, 01 Jun 2026 17:14:13 +0000

Ever noticed how your search results look completely different when you use a VPN? That's because Google personalizes results based on your IP, not just your query. This is a nightmare for SEO pros trying to audit local visibility.

I built a Python script that bypasses this by sending requests with custom headers and geolocation data. Here's a simplified version:

python
import requests

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
'Accept-Language': 'en-US,en;q=0.9',
'X-Forwarded-For': '8.8.8.8' # spoof IP
}

params = {
'q': 'coffee shop',
'gl': 'us', # country
'hl': 'en', # language
'uule': 'w+CAIQICINVXNhIE5ldyBZb3Jr' # encoded location
}

response = requests.get('https://www.google.com/search', headers=headers, params=params)
print(response.text[:500]) # check if results are localized

This is great for one-off tests, but managing multiple locations and languages quickly becomes tedious. That's why I now use a dedicated tool that automates all this with a simple API call. It's been a game-changer for my international SEO workflow.

https://serpspur.com

Tracking Keyword Rankings with Python + CSV: A Lightweight SEO Experiment

Emma Watson — Fri, 29 May 2026 15:35:39 +0000

I've been experimenting with a lightweight approach to track keyword ranking changes using Python and a simple CSV file. Here's the gist:

python
import csv
import requests
from datetime import datetime

def check_ranking(keyword, url):
# Simplified ranking check - actual implementation would use search API
# This is just a placeholder
return {'keyword': keyword, 'url': url, 'rank': 5, 'date': datetime.now().isoformat()}

def update_tracker(csv_path, keywords):
with open(csv_path, 'a', newline='') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=['keyword', 'url', 'rank', 'date'])
if csvfile.tell() == 0:
writer.writeheader()
for keyword in keywords:
data = check_ranking(keyword['url'], keyword['url'])
writer.writerow(data)

keywords = [
{'keyword': 'python tutorial', 'url': 'https://example.com/python'},
{'keyword': 'web scraping', 'url': 'https://example.com/scraping'}
]
update_tracker('rankings.csv', keywords)

For serious rank tracking, I rely on SERPSpur's API which handles search result parsing and historical data. But this simple approach works for hobby projects. How do you track your keyword rankings?