DEV Community: Jan S.

Automated Competitor Keyword Research Without Losing Your Weekend

Jan S. — Fri, 25 Sep 2026 22:30:00 +0000

You exported 20,000 keywords from a competitor gap analysis, and about half of them turn out to be the same keyword with the words shuffled. Someone on Reddit described exactly this a few months back, right down to building a spreadsheet macro that alphabetized every remaining term just so the duplicates would stand up and be counted. The comments then spent the whole thread arguing about whether one version "covers" the others. Nobody won.

That's the real bottleneck, by the way. It's not knowing that Semrush and Ahrefs can show you what your rivals rank for. Everyone knows that. It's the hour after the export, when a good question has turned into a wall of near-duplicates and three different definitions of search volume.

Say you run a small invoicing app, and three competitors seem to outrank you everywhere that matters. You want their keywords. Not all of them, just the ones worth a month of evenings. Here's a working version of automated competitor keyword research: the tedious middle gets handled or at least tamed, and you keep every decision that needs a brain.

Step 1: Pick the rivals you actually fight in search

Your business rivals and your search rivals are rarely the same list. In search, your competitor is whoever occupies the results you want: other software companies, sure, but also comparison sites, an industry blog that published one great guide in 2022 and still lives off it, and a Reddit thread Google can't stop ranking.

The low-tech way to find them: take the five searches you most want to win, type them into Google, and write down who keeps showing up. Do that before opening any tool. Then cross-check with the competitors report your SEO tool already has. Ahrefs calls it Organic Competitors; Semrush has its own version of the same idea. When the two lists disagree, believe the one you built by hand.

You'll almost certainly surface at least one site you never thought of as competition. Those are usually the interesting ones, because they're winning your searches without ever being on your radar.

Step 2: Pull keywords from two tools, not one

One tool gives you a list.

Two give you a reality check.

Semrush's Keyword Gap compares up to five domains side by side and shows where your rivals are present and you're missing. Ahrefs' Content Gap does the same job from the other direction: keywords other sites rank for that yours doesn't. Run both if you can, or run one and spot-check a handful of its answers in the other.

They will disagree, and not politely. Ahrefs' own guide to keyword difficulty admits that scores for the same keyword "will vary quite substantially" between tools, and that the score doesn't account for your site's authority. So a difficulty of 30 means nothing on its own. Thirty for whom?

Volumes drift too, for duller reasons. Ahrefs estimates from a recent 12-month window, Semrush averages the last 12 months, and Google hands advertisers bucketed ranges. Three definitions, one keyword, three numbers. Keep them side by side in your spreadsheet and resist averaging them, because the average of two different definitions is a third number nobody actually measures.

Step 3: Clean the list before it cleans you

Least glamorous step, biggest win. Four cuts, roughly in this order.

Word-order twins. "Invoicing software for freelancers" and "freelancer invoicing software" are one page, not two. Sort the list alphabetically and the twins end up next to each other, which does most of the work for you.

Branded terms. If the keyword contains a competitor's brand name, that click was never yours. Delete without guilt.

Mismatches. Gap reports show you the exact page that ranks for each term. When that page clearly isn't what the searcher wants, your competitor is ranking by accident. That's not a gap, that's noise in a gap costume.

Stuff you can't serve. You'll know it when you see it: the legal-question keyword, the enterprise term a two-person product can't honestly touch. Cut now, grieve later.

The point of cutting hard isn't tidiness. A shortlist you'll actually work through beats a masterpiece you abandon by Thursday.

Step 4: Cross everything against what Google already tells you about your site

Search Console is free, it's Google's own record of how your site performs, and it has two quirks worth knowing before you lean on it. The data runs two to three days behind, per Google's own documentation. And Google withholds some of the searches people actually typed, usually the rare ones, which Ahrefs pegs at close to half of all clicks. Slightly stale, slightly censored. Still the most honest data you own.

The cross-check itself is one question per keyword: do I already show up for this? If you're getting impressions for a term, it was never a gap. If you rank on page two and get no clicks, that's not a content gap either. That's a title-and-snippet problem, which is the cheaper kind and somehow the more annoying kind.

Step 5: Group keywords that share the same results

Here's the step most people skip, right before they publish a new post that competes with their own older one. Keywords that return roughly the same set of results are one topic, not ten.

Keyword Insights explains its approach without hiding the machinery: look at the top 7 results for each keyword, and if two keywords share 40% or more of the same results, they go in one cluster. The percentage is adjustable, and other tools do a version of the same thing. Semrush's Keyword Strategy Builder groups by similar results too.

The payoff is structural. You stop writing one page per keyword and start writing one page per group, aimed at the main term, with the variants riding along for free. What you're not doing is spinning up a page for every keyword left on the list. Google's guidance on people-first content is blunt about mass-producing pages to chase rankings, and gap reports are exactly how that habit starts.

Step 6: Check what the searcher wants, and whether it's your topic

Every tool slaps an intent label on keywords, informational or commercial or whatever, and the labels are mostly right. When a keyword genuinely matters to you and the label looks vague, open the results page yourself and look at it. The live results are the only intent check that can't be out of date.

Then comes the question no tool can answer: is this your topic? Ahrefs has a scoring idea worth stealing called business potential, running 0 to 3. Three means your product is essentially the answer to the search. Zero means there's no natural way to mention what you sell. An invoicing app should not chase "how to start a freelance career" no matter how many searches it has.

And the volume trap, because it catches everyone exactly once: Ahrefs' own strategy guide walks through a keyword with roughly a million US searches a month that sends only about 200 clicks, because an AI overview and a Reddit thread sit on top of the results. Sort a list by volume and work downward, and you can grind for months on keywords that were never going to send you a visitor.

Step 7: Check you're not competing with yourself

Before writing anything new, look at whether two of your pages are already fighting over the same term. Search Console's performance report, filtered to a single search term, shows every page of yours picking up impressions for it. Two pages, one keyword, split signals. Cannibalization is the most self-inflicted problem in SEO, and the fix is usually choosing a winner and pointing the losing page somewhere useful, not writing anything new at all.

Step 8: Automate the boring parts, keep the judgment

Look at what the tedious middle actually consisted of: sorting, deduplicating, grouping, and pulling the same report from three places every month. That's the automatable half. The other half is deciding that your invoicing app should own "late invoice letter template" this quarter and ignore "accounts receivable" entirely. No tool knows your margins, your roadmap, or what you're pushing this month. That part stays yours.

One warning before you hand work to AI: it's genuinely good at sorting and grouping a list you give it, and it should never be allowed near a number it can't look up. SpyFu documented ChatGPT estimating 5,000 to 10,000 monthly searches for a keyword whose real volume was around 700. So let the machine sort, group, and summarize. Never let it estimate volume or difficulty. Those come from a tool with actual data, every time.

This is the gap tools like Inteldo's SEO Analyst are built for: read-only connections to Search Console, PageSpeed Insights and Ahrefs, answers with citations instead of manually assembled reports, and the ability to pass a question to another specialist agent when it drifts into traffic or content territory. Useful, not mandatory. Everything above works with a spreadsheet, a couple of tool subscriptions, and a free Sunday.

What I'd do this week

Skip the grand setup and run the loop once, end to end.

Write down the five searches you most want to win and Google them. Whoever keeps showing up is your competitor list. Run one gap report across those sites, then cut until it hurts: twins, branded terms, mismatches. Cross what's left against Search Console, because anything you already get impressions for is not a gap. Then group what survives, pick one group, and write the single best page on that topic you're capable of. Stop there.

One pass of this beats a month of exporting, because the painful part was never the research. It was the shuffling.

I Thought Research Was About Finding Answers. I Was Wrong.

Jan S. — Mon, 31 Aug 2026 15:30:00 +0000

I've been working on a product where a surprisingly large part of the work comes down to answering questions.

At the beginning, I thought the difficult part would be getting the data. Connect the different sources, make sure the numbers are available, figure out how to pull everything together, and then the rest should be relatively straightforward.

That assumption didn't last very long.

The more I worked on it, the more I noticed something that felt slightly backwards. Getting an answer wasn't necessarily the end of the research. A lot of the time, getting the answer was what created the next problem.

And for a while, I thought that meant I wasn't doing the research properly.

The way I originally thought research worked

My mental model was pretty simple:

Question → Data → Answer → Done

It makes sense for a lot of basic questions.

How much revenue did we make?

How many users signed up?

Which page gets the most traffic?

You look something up, get the number, and move on.

But most of the questions that actually matter aren't really like that.

Take something simple like this:

Why did signups drop this week?

You look at the data and discover that traffic dropped.

Great. You have an answer.

Except now you have another questions.

Why did traffic drop?

You investigate that and find that most of the decline came from organic traffic.

Okay

Why did organic traffic drop?

Maybe a few pages lost rankings.

Then you start asking why those pages lost rankings.

Was there a technical change? Did the content change? Did search demand changes? Did competitors improve? Was there an indexing problem?

At this point, you've gone several questions away from where you started.

The strange part is that the original answer wasn't useless. It was actually what helped you get to the better question.

I used to think this meant the research was failing

This was probably the biggest mistake in how I thought about research.

If I asked a question and the answer immediately created another question, I felt like I hadn't really solved anything.

I was expecting a clean ending.

What I've started realizing is that real investigation usually doesn't work that way.

It's much closer to debugging.

When something is wrong in a codebase, you don't always know what the bug is before you start looking. You make an assumption, check something, find evidence that supports or disproves it, and then change your approach.

Sometimes the best result from debugging is discovering that the thing you thought was broken isn't actually the problem.

The same thing happens with business questions.

You might think you're investigating a conversion problem and discover that the conversion tracking changed.

You might think traffic is down because of SEO and discover that the bigger issue is a change in how the traffic is being measured.

You might think customers aren't using a feature because they don't want it, and then discover they couldn't figure out how to use it.

Those aren't failures.

They're corrections to the question.

More data didn't solve the problem either

This was another assumption I had.

If the problem is that you don't know what's happening, having access to more information should make things easier.

Sometimes it does.

Sometimes it makes things worse.

You can have analytics, payment data, product events, search data, customer feedback, spreadsheets, dashboards, and a dozen browser tabs open and still not know what you should actually do next.

I think I was confusing having information with making progress.

They're not the same thing.

A dashboard can tell you that something changed.

It doesn't necessarily tell you which explanation is worth investigating.

And having five different sources of data doesn't automatically help if you're not sure whether they're telling you the same story.

That was probably one of the more frustrating things to realize while working on this.

I didn't necessarily need more information.

I needed the information I already had to help narrow down what I should look at next.

The answer can be useful even when it isn't the answer you wanted

This has changed the way I look at research.

I used to think a successful investigation was one where I got the answer I was looking for.

Now I think there are several ways an answer can be useful.

It can confirm what you already suspected.

It can rule something out.

It can reveal something you weren't expecting.

Or it can completely change the question.

The last one might actually be the most valuable.

If I start with:

Why did revenue fall?

and discover that revenue didn't actually fall, but the reporting changed, that's a useful result.

I didn't solve the original problem.

I discovered that the original problem wasn't real.

That saves a lot of time.

I'm starting to think about questions differently

I've also noticed this changing the way I approach problems before I even start looking at the data.

Instead of only asking:

What's the answer?

I'm trying to think about:

What would I do differently depending on the answer?

That small change makes the investigation feel different.

If the answer could change the next thing I investigate, then it's probably worth digging into.

And sometimes the most useful question isn't the one that gives you the most information.

It's the one that eliminates the most possibilities.

I don't have this figured out completely. I'm still learning where that line is, and I'm sure I'll keep getting it wrong.

But I no longer see a new question appearing after an answer as a sign that the research failed.

Sometimes it means the research finally got somewhere.

What I'm taking away from this

The biggest change for me is that I don't think of research as:

Find the answer and finish.

I think of it more as:

Ask → investigate → learn something → adjust the question → investigate again.

That sounds obvious when I write it down, but it wasn't how I was thinking about it when we started building.

And maybe that's the part I find most interesting.

A good answer doesn't always close the investigation.

Sometimes it tells you what you should have been asking in the first place.

Building Against Real APIs Is Nothing Like Reading the Docs

Jan S. — Mon, 03 Aug 2026 08:30:12 +0000

A few months ago, I started spending a lot more time working with different APIs.

At first, I thought authentication would be the hardest part. Once OAuth was working and I could successfully make requests, I assumed everything else would mostly be connecting the dots.

It didn't take long to realize I was wrong.

The real challenge wasn't getting data. It was figuring out what that data actually meant.

The docs only show the happy path

API documentation is usually great at helping you get started.

You learn how to authenticate, which endpoint to call, and what a successful response looks like. Five minutes later, you're making your first request and everything feels pretty straightforward.

Production is a different story.

One API returns timestamps in UTC. Another uses your account's local timezone. Some return an empty array when there's no data, others return null, and a few simply leave the field out entirely.

None of those things are difficult by themselves.

The challenge is that every API is a little different, and those small differences slowly pile up.

The same event can mean different things

One thing that surprised me was how differently platforms describe the same event.

Let's say someone asks:

"How many customers signed up yesterday?"

It sounds like there should be one answer.

But depending on which platform you're looking at, "signed up" might mean:

created an account
started a free trial
verified an email
completed a payment
became an active customer

None of those definitions are wrong.

They're just measuring different moments.

That was probably the biggest mindset shift for me. Before combining data, you have to understand what each system is actually measuring.

Expect inconsistencies

After working with more integrations, I've stopped expecting APIs to behave the same way.

Different pagination styles.

Different rate limits.

Different error responses.

Different field names.

Sometimes even different behavior between endpoints from the same provider.

At first those inconsistencies felt frustrating.

Now I almost expect them.

Instead of assuming every response will match the documentation, I try to build integrations that can handle missing fields, unexpected values, and the occasional edge case.

It's usually those small details that save you hours of debugging later.

The biggest lesson I've learned

If there's one thing I'll carry into future projects, it's this:

Treat every external API as a system you don't control.

The documentation gets you connected.

The real work starts after the first successful request.

That's where you begin learning how the API behaves in the real world, and that's where most of the interesting engineering problems show up.

I'm still learning this myself, but it's already changed how I approach integrations.

I'd be interested to hear what unexpected API quirks you've run into. I'm sure there are plenty I haven't discovered yet.`