DEV Community: GasPriceCheck

I built a Node script that turns Google Search Console into a daily dashboard

GasPriceCheck — Thu, 28 May 2026 05:00:00 +0000

GSC's web UI is fine for occasional spelunking. It's slow for daily ops. Three pages of clicks to get to the same five queries I run every morning. The export is CSV. The date range picker is finicky.

So I built a Node script. About 150 lines. Three flags. Pulls everything I look at daily into the terminal in four seconds.

Here's what it does, what it doesn't do, and what I learned about the GSC API along the way.

What I check every morning

Five things, in order:

Total clicks last 7 days vs prior 7 days
Top 25 queries by clicks
Top 25 pages by clicks
Pages with the biggest position gain (rising stars)
Pages with the biggest position loss (warning signs)

In GSC's web UI, that's five-plus clicks and a date range adjustment per check. In a terminal, it's node gsc-query.js --all and four seconds.

One-time setup

You need OAuth. The GSC API requires a refresh token, which you get by going through a one-time auth flow.

I wrote a separate script for this (gsc-auth.js) that handles the dance:

const { google } = require('googleapis');
const open = require('open');
const http = require('http');
const url = require('url');

const oauth2Client = new google.auth.OAuth2(
  process.env.GSC_CLIENT_ID,
  process.env.GSC_CLIENT_SECRET,
  'http://localhost:3737/oauth-callback'
);

const authUrl = oauth2Client.generateAuthUrl({
  access_type: 'offline',
  scope: ['https://www.googleapis.com/auth/webmasters.readonly'],
});

console.log('Opening browser for authentication...');
open(authUrl);

http.createServer(async (req, res) => {
  const code = url.parse(req.url, true).query.code;
  if (code) {
    const { tokens } = await oauth2Client.getToken(code);
    console.log('Add this to your .env:');
    console.log('GSC_REFRESH_TOKEN=' + tokens.refresh_token);
    res.end('Auth complete. You can close this tab.');
  }
}).listen(3737);

You run it once, click through the consent screen, and it prints your refresh token to the terminal. Save that to .env and you never run this script again.

A note on the OAuth flow: you'll need to set up an OAuth consent screen and a Web Application credential in the Google Cloud Console first. Add http://localhost:3737/oauth-callback as an authorized redirect URI. The whole setup takes about 10 minutes if you've never done it before, and it's the most annoying part of the project.

The "domain property vs URL prefix" gotcha

This one cost me 30 minutes when I was setting up. Sharing it so it costs you zero.

GSC has two ways to verify a property:

URL prefix (e.g., https://www.gas-price-check.com/): covers exactly that protocol + subdomain combo
Domain property (e.g., gas-price-check.com): covers all subdomains and protocols

The API treats these as different objects. To call data for a domain property, you use the format sc-domain:gas-price-check.com. Without the prefix, the API returns "site not found" with no hint about the format issue.

const siteUrl = 'sc-domain:gas-price-check.com'; // domain property
// not 'https://www.gas-price-check.com'
// not 'gas-price-check.com'

const res = await searchconsole.searchanalytics.query({
  siteUrl,
  requestBody: {
    startDate: '2026-04-22',
    endDate: '2026-04-29',
    dimensions: ['query'],
    rowLimit: 25,
  },
});

If you set up your GSC property as a URL prefix, use the URL with the protocol. If you set up as a domain property, use the sc-domain: prefix. The error message gives you no hint about which format the API expects, so just remember the rule.

The query script

The actual gsc-query.js is straightforward. The interesting part is the flag handling:

const args = process.argv.slice(2);
const days = parseInt(args[args.indexOf('--days') + 1] || '7');
const showQueries = args.includes('--queries') || args.includes('--all');
const showPages = args.includes('--pages') || args.includes('--all');
const showRising = args.includes('--rising') || args.includes('--all');

The actual API call is shaped like this:

async function fetchData(dimensions, days = 7) {
  const endDate = new Date();
  const startDate = new Date(endDate);
  startDate.setDate(startDate.getDate() - days);

  const res = await searchconsole.searchanalytics.query({
    siteUrl: process.env.GSC_SITE_URL,
    requestBody: {
      startDate: startDate.toISOString().split('T')[0],
      endDate: endDate.toISOString().split('T')[0],
      dimensions,
      rowLimit: 25,
    },
  });
  return res.data.rows || [];
}

To get top queries: dimensions: ['query']. Top pages: dimensions: ['page']. To compute position changes over time: query twice with different date ranges and diff the results.

The five queries I run every morning

node gsc-query.js --days 7              # summary
node gsc-query.js --queries --days 28   # top queries last month
node gsc-query.js --pages --days 7      # top pages last week
node gsc-query.js --rising              # biggest position gains
node gsc-query.js --falling             # biggest position losses

Output looks like:

Top 25 queries (last 28 days):
cheap gas near me                   142 clicks  3,201 impr  4.4% CTR  pos 8.2
gas prices houston                   87 clicks  1,892 impr  4.6% CTR  pos 6.1
77386 gas prices                     61 clicks    520 impr 11.7% CTR  pos 2.8
cheapest gas in austin               52 clicks  1,104 impr  4.7% CTR  pos 9.4
...

Whole thing returns in four seconds. I run it while my coffee is brewing.

A few things I learned about the API

The free quota is generous. GSC API gives you 1,200 queries per minute and 30,000 per day. For a personal dashboard you'll never hit it.

The API returns the same data as the UI, but with no rounding. The web UI rounds clicks to the nearest digit and aggregates queries below 10 impressions into "..." rows. The API gives you actual numbers, including the long tail of queries with 1-3 impressions. For finding "rising star" queries, that long tail is gold. A query with 4 impressions today that had 0 impressions last week is a signal the web UI hides from you.

Rate limiting is per project, not per user. If you have multiple scripts hitting GSC, run them all from the same OAuth project to share the budget cleanly.

Caching helps a lot. I cache yesterday's data in a local JSON file and only re-fetch today's data on the second run of the day. Cuts run time roughly in half.

The searchAppearance dimension is underrated. Filter by searchAppearance: 'AMP_TOP_STORIES' or 'WEBLITE' to see how each rich-result type is performing separately. Most people never look at this.

What I haven't bothered to add

The script is about 150 lines. I've thought about adding:

A diff mode that flags queries that moved more than 5 positions week-over-week
An export-to-CSV flag for when I want to load data into a spreadsheet
A "branded queries only" filter
A daily Slack notifier so I don't have to run the script at all

Each is a 30-line addition. The fact that I haven't bothered is a sign the script does enough already. Premature feature creep is a real risk on personal tools.

What's the point

For a side project, GSC's web UI is fine. For a project where you check the same data daily, building a 150-line wrapper script will save you hours per month and surface signals the web UI hides. The whole thing took me about three hours to build, including the OAuth dance. It's paid that back in saved clicks within a week.

If you've built a similar GSC daily dashboard, what flags do you reach for? I'm curious what other people's morning checks look like. Mine is pretty narrow, but it's stable, and I've stopped opening the GSC web UI for daily ops entirely. Once a week I check the URL Inspection tool for specific pages in trouble. The rest is the script.

Templating got me to 33,620 pages. Indexing them was the hard part.

GasPriceCheck — Thu, 21 May 2026 05:00:00 +0000

I had a fantasy.

The fantasy was that programmatic SEO would let me skip the part where you write 200 individual articles and trick Google into ranking them. I had a structured dataset (US ZIP codes, cities, states) and a templated page generator. So I shipped 33,620 pages overnight. Then I sat back and waited for the long-tail traffic to roll in.

That's not what happened.

A few months later, I have a more nuanced view. Here are the five lessons I would have paid actual money for at the start.

The numbers, so we're working from data

Out of 33,620 templated pages on my site:

~6,200 are indexed
~18,000 are in "Crawled, currently not indexed" (CCNI)
~8,500 are "Discovered, currently not indexed"
~700 are flagged as soft 404
The rest are recently submitted and still in queue

That's an indexation rate of about 18%. Sounds low. It kind of is. But here's the catch: the 6,200 indexed pages drive the vast majority of organic traffic. The CCNI pile is mostly low-traffic ZIPs in low-population areas. Google has decided those aren't worth indexing, and honestly they're probably right.

If you came into programmatic SEO expecting "ship N pages, get N pages of traffic," recalibrate. The real math is closer to "ship N pages, get 15-25% indexed, and the indexed pile drives 95% of your traffic."

Lesson 1: Templating gives you the structure, not the content

This is the biggest one and the one most programmatic SEO tutorials skip.

My page generator spit out pages with this rough structure:

Hero (city, state, ZIP)
Search box
List of nearby gas stations
State average price comparison
"How to save on gas in {city}" section
Footer

Sounds reasonable. The problem: when I put two pages side by side (one for ZIPs in California, one for ZIPs in Maine), they were 95% identical text. The hero changed. The station list was different. Everything else was the same paragraph with one word swapped.

Google's templated-content detection is real and it's specifically looking for this. There's a metric (referenced in their public ranking patents as "boilerplate ratio") that measures what fraction of a page is shared with other pages on the same site. High boilerplate ratio plus thin unique content equals a soft 404 verdict.

The fix was to make the templated content actually vary. I built a state-context module with hand-written paragraphs for the 15 highest-traffic states. Each is 80 to 120 words covering the state's gas tax, refinery capacity, regulatory regime, and seasonal pricing patterns. The other 35 states got a parameterized template with state-specific variables (avg price, neighboring states, gas tax rate). After this shipped, indexation rates climbed about 15% over six weeks.

The 80/20 here: hand-writing the top 15 of 50 states covered about 75% of my traffic. The other 35 got the templated treatment, which is fine because they're not driving meaningful traffic anyway.

If I'd known this at the start, I would have spent the first weekend writing 15 paragraphs by hand and the second weekend generating the rest. Instead I spent four weekends generating everything and then went back to hand-write the top tier. Same end state, slower path.

Lesson 2: Internal URL types compete with each other

I had two URL types covering the same entity. A programmatic city page (/houston-tx) and a blog post about Houston gas prices (/blog/cheapest-gas-houston). Different formats, different content, both about Houston gas.

Google saw these as competitors. It picked one to index (the blog post) and put the other in CCNI. Then it picked the other for some sister cities (city page won, blog post in CCNI). The pattern was inconsistent and frustrating.

My first instinct was wrong. I tried to "fix" this by submitting the CCNI page for re-validation, hoping Google would just index both. That's not how this works. Google has a per-site indexation budget and a canonical-decision system that picks one URL per topic per intent. When you ask it to re-evaluate, it just makes the same decision again.

The actual fix was to differentiate the intent of each URL type:

Blog post: opinionated guide, current events, "why are gas prices high in Houston this month"
City page: factual reference, station list, ZIP grid, state context, "where to find cheap gas in Houston"

Once the content categories were clearly different, Google's canonical decisions became consistent. Blog posts started ranking for query-with-intent searches ("why are gas prices high in Houston"). City pages ranked for quantity-intent searches ("cheap gas Houston"). They stopped competing for the same impressions.

If you have two URL types covering similar entities, they're probably competing whether you intended it or not. Audit your URLs and make sure each type has a clear, different job.

Lesson 3: Word count is a proxy, not a target

I spent some time obsessing over word counts. I'd seen "1,500 word minimum for ranking" quoted in SEO guides. So I padded pages with filler to hit the number.

Google did not care. The pages that broke through to indexed weren't the ones with 1,500 words. They were the ones with 800 words of actually useful, page-specific content.

Word count is a proxy for depth. If you can get to 800 words by writing things that are genuinely useful and page-specific, you're done. If you're padding to hit a target, Google can tell, and the padding doesn't help. The detection isn't a single signal, it's a combination: sentence-level perplexity, semantic distinctness from other pages on your site, contextual relevance to the query intent. Modern ranking models are sophisticated enough to recognize stuffing.

I went from a 640-word baseline to an 810-word "after" state by adding things that were genuinely useful: the per-state context, an SSR-rendered nearby ZIP grid, a top-stations summary. None of it was filler. The +170 words mattered because of what was in them, not because of the count.

Lesson 4: GSC verdicts have memory

If a page gets flagged as soft 404, fixing the page doesn't immediately clear the verdict. The recovery flow is:

Fix the page
Submit a re-validation request in GSC
Wait 1 to 4 weeks for the re-crawl
Accept that some URLs won't come back even after you fix them

GSC's verdict is sticky. It's not because Google is malicious. It's because the indexation queue is finite and pages with prior negative verdicts get deprioritized. If a page has been "soft 404" for three months and you fix it, Google doesn't drop everything to re-evaluate. It works through the queue, and your fixed page joins the back.

The implication: when planning a programmatic SEO site, get the page quality right before you submit URLs to GSC. Submitting a thin page early creates a verdict you'll fight for months. Better to wait, polish, and submit clean.

I did the opposite. I submitted everything immediately because I was excited. About 700 pages got soft-404'd, and clearing those verdicts took about 8 weeks of patient post-mortem work even after the underlying content was fixed.

Lesson 5: Branded search beats backlinks for indexation

This one surprised me.

Pages that started getting branded search impressions (users typing "gas price check Houston" into Google) got indexed faster than pages that had inbound backlinks from other domains. The branded queries didn't even have to convert. Just the impressions seemed to register.

Google's signal here seems to be: "people are looking for this specific page on this site, by name." That signal is hard to fake (it requires real users typing branded queries) so Google trusts it heavily. A backlink can be paid for or manufactured. Branded search volume cannot.

The implication: if you want to accelerate indexation on a specific page, drive direct traffic to it from anywhere. Social posts, email, bio links, even paid ads briefly. The traffic sends a "this page is real and people want it" signal that Google's ranking model picks up.

I tested this informally. I posted a link to one of my CCNI city pages on Reddit. Got about 80 visits in a day. Within two weeks the page moved from CCNI to indexed, with a top-50 ranking on its target query. I didn't change the page at all. Just sent traffic.

I'm not claiming this works every time. But the pattern was strong enough that I now bias toward "drive traffic to anything stuck in CCNI" over "build more backlinks to anything stuck in CCNI."

What I'd do differently

If I were starting over:

Start with 50 hand-written pages, not 50,000 templated ones. Get those indexed and ranking before generating the long tail. The long tail benefits from established authority.
Differentiate URL types from day one. Decide what each URL type is for and don't blur the line.
Pre-compute everything. If you're going to template at scale, all unique-per-page content (geocoding, internal links, neighbor lists) should be resolved at build time, not runtime. Runtime fetches show up as "loading..." placeholders in the SSR HTML, and Google reads those as thin content.
Check GSC weekly, not monthly. Soft 404 verdicts compound silently. By the time you have 700 of them, recovery is a multi-month project.
Save the URL submissions for last. Polish first. Submit when the page is actually good. Once a verdict lands, it's sticky.

The bottom line

Programmatic SEO works. But the math isn't "ship 33,620 pages, get 33,620 pages of traffic." It's closer to "ship 33,620 pages, get 6,200 indexed, get traffic on the top 2,000 of those, and the rest are infrastructure." The work isn't generating the pages. The work is earning the right to keep them indexed.

If you've shipped a programmatic site at scale, what's your indexation rate? I'd love to hear what others are seeing. My hypothesis is that 15-25% indexation is typical for first-year programmatic sites, and the rate climbs over time as the site builds authority. But I have a sample size of one, so I'd love more data points.

When zippopotam.us 404s on a real US ZIP: building a 4-tier geocoding fallback

GasPriceCheck — Thu, 14 May 2026 05:00:00 +0000

ZIP-to-lat/lng feels like a solved problem. There are something like 41,000 active US ZIP codes. They're a public dataset. They're geocoded. Multiple free APIs serve them. You wire one up, it works, you ship.

That's what I thought. Then a user reported that the gas station on his block was showing as "8,247 miles away."

Quick context: I run a gas price finder side project. When you enter a ZIP, the site looks up that ZIP's lat/lng, then sorts nearby gas stations by driving distance. If the ZIP's lat/lng resolves to (0, 0), the Haversine math returns absurd distances because (0, 0) is in the middle of the Atlantic Ocean off the coast of Africa.

The user's ZIP was 75072. McKinney, Texas. Population 200,000. Not some obscure rural P.O. box. A real ZIP code with a real population center.

My geocoder was returning (0, 0).

What was actually happening

My geocoder, at the time, was a single fetch:

async function zipToLatLng(zip: string) {
  const res = await fetch('https://api.zippopotam.us/us/' + zip);
  if (!res.ok) return { lat: 0, lng: 0 };
  const data = await res.json();
  return {
    lat: parseFloat(data.places[0].latitude),
    lng: parseFloat(data.places[0].longitude),
  };
}

zippopotam.us is free, fast, and was working for everything I'd tested. So when 75072 returned a 404, my fallback was the (0, 0) sentinel. Garbage in, garbage out.

I dug a little. Curled the API directly:

$ curl -i https://api.zippopotam.us/us/75072
HTTP/1.1 404 Not Found

Then I tried 75070, the neighboring ZIP:

$ curl -i https://api.zippopotam.us/us/75070
HTTP/1.1 200 OK

So 75070 is in zippopotam's data, 75072 isn't. Two ZIPs in the same city, one block apart, one of them missing. There's no announced data freshness commitment from zippopotam (it's a free service maintained by one person), so this isn't a bug to report. It's just a gap.

I spot-checked. Out of a sample of 100 ZIPs from my user logs, zippopotam returned 404 on 4 of them. A 4% gap rate. On a programmatic site with 33,620 pages, that's about 1,300 pages where the geocoder silently returned (0, 0) and broke distance math.

This is the moment when "I'll just use a free API" stops scaling.

The chain I shipped

What I needed wasn't a better single source. It was a chain of sources, each with different tradeoffs, ordered from cheapest to most expensive:

Static file lookup (instant, zero network)
Redis cache (instant, one network hop)
zippopotam.us (~150ms, free, has gaps)
Nominatim (~400ms, free, rate-limited, covers gaps)
Sentinel (0, 0) with degraded behavior

Here's the actual implementation, slightly cleaned up:

import { Redis } from '@upstash/redis';
import zipContent from './zipContent.json';

const redis = Redis.fromEnv();

type LatLng = { lat: number; lng: number };

export async function zipToLatLng(zip: string): Promise<LatLng> {
  // Tier 1: static file (33,620 pre-resolved ZIPs)
  const staticEntry = (zipContent as Record<string, any>)[zip];
  if (staticEntry?.lat && staticEntry?.lng) {
    return { lat: staticEntry.lat, lng: staticEntry.lng };
  }

  // Tier 2: Redis cache (for ZIPs we've resolved at runtime)
  const cached = await redis.get<LatLng>('zip:' + zip);
  if (cached) return cached;

  // Tier 3: zippopotam.us
  try {
    const res = await fetch('https://api.zippopotam.us/us/' + zip, {
      signal: AbortSignal.timeout(2000),
    });
    if (res.ok) {
      const data = await res.json();
      const result = {
        lat: parseFloat(data.places[0].latitude),
        lng: parseFloat(data.places[0].longitude),
      };
      await redis.set('zip:' + zip, result, { ex: 60 * 60 * 24 * 30 });
      return result;
    }
  } catch (e) {
    // network error or timeout, fall through
  }

  // Tier 4: Nominatim
  try {
    const res = await fetch(
      'https://nominatim.openstreetmap.org/search?postalcode=' + zip + '&country=us&format=json&limit=1',
      {
        headers: { 'User-Agent': 'GasPriceCheck/1.0 (contact@example.com)' },
        signal: AbortSignal.timeout(3000),
      }
    );
    if (res.ok) {
      const data = await res.json();
      if (data.length > 0) {
        const result = {
          lat: parseFloat(data[0].lat),
          lng: parseFloat(data[0].lon),
        };
        await redis.set('zip:' + zip, result, { ex: 60 * 60 * 24 * 30 });
        return result;
      }
    }
  } catch (e) {
    // network error or timeout, fall through
  }

  // Tier 5: sentinel
  return { lat: 0, lng: 0 };
}

Five tiers, ordered by cost. Each one only runs if the previous one didn't find an answer. The static file handles 99% of requests in zero time. Redis handles the runtime-resolved ones. The two API tiers are the actual fallbacks.

A few notes on the implementation choices.

Tier 1: pre-resolving everything you can ahead of time

The biggest performance win wasn't the chain. It was Tier 1: a static JSON file with every ZIP I knew about, pre-resolved to lat/lng once at build time.

I wrote a Python script (enrich-zip-latlng.py) that walks my full ZIP list, calls zippopotam for each one, falls back to Nominatim for the gaps, and writes the results to zipContent.json. The script is resumable: it checkpoints to a JSON file every 100 ZIPs, so if it crashes (and it did, several times, on intermittent rate limits), you restart and it picks up where it left off.

import json, time
from pathlib import Path
import requests

CHECKPOINT = Path("zip_latlng_progress.json")
OUTPUT = Path("zipContent.json")

def load_progress():
    if CHECKPOINT.exists():
        return json.loads(CHECKPOINT.read_text())
    return {"resolved": {}, "remaining": load_zip_list()}

def resolve(zip_code):
    # try zippopotam, fall back to Nominatim
    try:
        r = requests.get(f"https://api.zippopotam.us/us/{zip_code}", timeout=5)
        if r.ok:
            d = r.json()
            return float(d["places"][0]["latitude"]), float(d["places"][0]["longitude"])
    except Exception:
        pass
    try:
        r = requests.get(
            "https://nominatim.openstreetmap.org/search",
            params={"postalcode": zip_code, "country": "us", "format": "json", "limit": 1},
            headers={"User-Agent": "ZipEnricher/1.0"},
            timeout=10,
        )
        if r.ok:
            data = r.json()
            if data:
                return float(data[0]["lat"]), float(data[0]["lon"])
    except Exception:
        pass
    return None, None

After running this against all 33,620 ZIPs (took about 9 hours of wall clock time, mostly idle waiting for rate limits), I had 100% coverage. Every known ZIP resolves in zero network calls at runtime.

The runtime fallback chain is now a safety net for the ZIPs that aren't in zipContent.json (mostly newly-issued ZIPs and edge-case P.O. box codes). It's load-bearing maybe 0.1% of the time. The other 99.9% of requests hit Tier 1 and exit.

Tier 4: Nominatim's etiquette is real

Nominatim is OpenStreetMap's geocoder. It's free. It's also explicit about its rate limits: 1 request per second, max. They publish a usage policy. They will rate-limit you (or ban your IP) if you violate it.

Two things matter for keeping Nominatim happy:

Set a real User-Agent header. They block requests with the default node-fetch/1.0 UA. Use something like MyApp/1.0 (contact@example.com).
Throttle. If you're calling Nominatim from a script, sleep at least 1 second between calls. From an API route serving real users, this is fine because user traffic is naturally sparse.

If you're going to depend on Nominatim at scale, you should run your own instance. It's open source and Dockerized. For a side project hitting it 1-2 times a day, the public instance is fine.

Tier 5: the sentinel and graceful degradation

When all four tiers fail, my function returns { lat: 0, lng: 0 }. The downstream code needs to handle this without producing garbage.

In my case, the downstream consumer is a "nearest gas stations" sorter that uses Haversine distance. If the user's coords are (0, 0), Haversine returns absurd distances for every station. So I added a guard:

const userCoords = await zipToLatLng(zip);
if (userCoords.lat === 0 && userCoords.lng === 0) {
  return stations; // unsorted, no distance column
}
return sortByDistance(stations, userCoords);

If we can't geocode, we still show stations. We just don't sort by distance, and we don't display the distance column. The page degrades gracefully instead of showing "8,247 miles away" everywhere.

This is the "fail open" pattern: when the dependency fails, the user still gets a useful page, just with less information. Better than failing the whole request.

What I'd tell my past self

Three things.

One. "Free APIs have gaps" is not a hypothesis, it's a default assumption. Any API where the provider is one person, a small org, or a "we'll keep this up as a community resource" project will have data quality issues somewhere. Plan for it from day one.

Two. Pre-resolve the long tail at build time, not request time. Runtime API calls are a tax on every user. If you have a finite, enumerable input space (like US ZIP codes), resolve the whole space once and cache the results to disk. This collapses the runtime cost from "1 API call per request" to "1 file lookup per request."

Three. Build the chain before you need it. The temptation when the first source works is to ship the single-source version and add fallbacks "later." Later means after a user reports a bug. Building the chain up front takes maybe 2 hours and saves you the embarrassment of "8,247 miles away" emails.

The architecture lesson here generalizes beyond ZIPs. Anywhere you have a dependency that maps "A → B" deterministically and the input space is finite, the move is the same: pre-resolve at build, cache at runtime, fall back to alternative providers, degrade gracefully when all else fails. ZIP geocoding, currency conversion, language code lookups, country data, time zone data, postal codes for any country with public data: same pattern.

The whole rebuild took about a weekend. I'd do it again in an afternoon now that I know the shape. The hard part wasn't the code, it was admitting that "free API + sentinel" wasn't actually a fallback strategy. It was a single-source-of-truth dressed up as one.

If you've shipped a layered geocoding chain on top of zippopotam, Nominatim, or a paid alternative, drop your tier ordering in the comments. Curious what other chains people are running.

Google deindexed half of my Next.js site. Here's the four-phase recovery.

GasPriceCheck — Thu, 07 May 2026 05:00:00 +0000

I run a side project: a gas price finder that's mostly programmatic content. Roughly 33,620 ZIP code pages plus a few hundred state and city pages, all built on Next.js 15 with ISR. It runs on Vercel, fronted by Cloudflare for DNS.

For about eight months it was indexing fine. Traffic was small but growing. Then on April 11 I checked Google Search Console and saw something I'd never seen at this scale: 87 URLs flagged as "not found (404)," 61 flagged as "soft 404," and a chunk of others sitting in "crawled, currently not indexed."

I'm a data analyst. This is my side project, not my day job. So I had a weekend, a coffee subscription, and the GSC export. Here's the four-phase recovery, what each phase actually fixed, and the things I'd do differently.

What the damage actually was

GSC's URL Inspection tool is the only way to figure out what Google thinks of any specific URL. The 87 hard 404s broke down into two groups when I sampled them:

Stale city slugs. URLs like /portland and /columbus that Google had picked up from somewhere (early sitemap drafts, third-party references, link typos in old social posts). My live routes use state-suffix disambiguation for city names that exist in multiple states (/portland-or and /portland-me for Portland, similar pattern for Columbus). Google still had the un-disambiguated slugs cached and was hitting 404s when it tried to refresh them.
Typo URLs in third-party referrers. A handful of blog posts I'd written had link typos to my own site. Google followed those, hit 404s, and now thought those URLs were canonical.

The 61 soft 404s were a different beast. These were real ZIP code pages that returned 200 OK but Google decided didn't have enough content to be "real" pages. Looking at the SSR output, I could see why: when a ZIP wasn't cached in Redis (which was most of the long tail), the page rendered a hero, a search box, and a thin "search results loading" placeholder. About 640 visible words in the SSR HTML. From Google's perspective: this is a glorified 404 wearing a 200 costume.

The "crawled, currently not indexed" pile was the soft-404 pile's pre-stage. Google had crawled them, decided they weren't worth indexing, but hadn't formally classified them as soft 404s yet.

That's the diagnosis. Now the fixes.

Phase 1: The boring redirect fix

This phase is unglamorous and important. For each of the 87 hard 404s, I had to decide: does this URL have a clear successor, and if so, where do I redirect it?

I did this in next.config.mjs:

const STALE_CITY_REDIRECTS = [
  { source: '/portland', destination: '/portland-or', permanent: true },
  { source: '/midland', destination: '/blog/gas-prices-midland', permanent: true },
  // ...64 more
];

module.exports = {
  async redirects() {
    return STALE_CITY_REDIRECTS;
  },
};

permanent: true emits a 308. That tells Google "this URL has moved permanently, transfer signals to the destination." A 301 would also work; I went with 308 because Next.js's defaults align with that and I didn't want to fight the framework.

The unglamorous part: most of the 87 URLs didn't have an obvious destination. Some pointed to cities I no longer covered. Some pointed to ZIPs that didn't exist. For those, I redirected to the parent state page (e.g. an unrecognized Texas city slug pointing to /texas) so the user lands somewhere useful, and the link equity isn't dropped on the floor.

Total output of Phase 1: 66 redirects, deployed in one commit. Pushed it. Watched GSC for a week.

Result: about half the hard 404s validated. The other half stuck. Why?

Phase 1.5: The Cloudflare redirect chain I didn't know I had

When I inspected one of the still-failing URLs in GSC, the URL Inspection tool said "page with redirect" and showed a chain. Google was hitting http://example.com/portland, getting redirected to https://www.example.com/portland, and then redirected again to https://www.example.com/portland-or.

Two hops. Google does not like two hops.

I had Cloudflare in front of Vercel as a DNS proxy. Cloudflare was handling the http to https and apex to www redirect. Vercel was handling the slug-to-slug redirect. Each layer worked. Together they made a chain.

The fix was a single Cloudflare Redirect Rule that does both transforms in one hop:

If: hostname matches "example.com" AND scheme is http
Then: 308 to https://www.example.com (preserving the original path)

After that rule landed, the chain collapsed from 2 hops to 1. Google reprocessed and started clearing the rest of the hard 404s. Lesson I should have known: when you have multiple layers (CDN, edge, framework), each one defaulting to "I'll handle the redirect" stacks. Audit the hop count with curl -IL <url> and look for chains.

Phase 2: The real work (thin SSR content)

The 61 soft 404s couldn't be redirected away. These were real pages I wanted indexed. Google just thought they were thin.

The diagnosis was straightforward once I started reading my own SSR output. When a ZIP wasn't cached, the page rendered:

A hero with the ZIP, city, state
A search box
A "loading prices..." placeholder (waiting for client-side fetch)
A footer

That's it. Maybe 640 visible words, of which 400 were the footer and global nav. The actual page-specific content was a hero header and a placeholder.

The fix had three sub-components.

2a. Server-render the nearby ZIP grid

I had a helper called getNearbyZips(zip, radius) that returned ZIP codes within a given mile radius. I'd been using it on the client. I moved it to the server component so the SSR HTML included an actual grid of "nearby ZIP codes" with links.

This added about 80 words of unique-per-ZIP content (different neighbors for each ZIP). More importantly, it added 8-12 internal links per page, which gave Google more signal about the URL's place in the site graph.

// Before: client-side, invisible to Google
const nearby = useNearbyZips(zip);

// After: server-rendered, visible to Google
const nearby = await getNearbyZips(zip, 25);
return (
  <section>
    <h2>Nearby ZIP codes</h2>
    <ul>
      {nearby.map(zipCode => (
        <li key={zipCode}>
          <Link href={'/' + zipCode}>{zipCode}</Link>
        </li>
      ))}
    </ul>
  </section>
);

2b. Unconditional save tips

I added a hand-written "How to save on gas in [city]" section that rendered regardless of cache state. About 120 words of static-but-locally-relevant content per city. This is templated, but with enough variable interpolation that no two pages have identical text.

2c. State backlink with name fallback

Every ZIP page already had a "Back to [state] state guide" link, but it relied on a state abbreviation lookup that returned null for some edge cases. So those pages were rendering "Back to undefined state guide" or worse, no link at all. Fixed it with a fallback:

const stateName = getStateByAbbr(state) ?? getStateByName(state) ?? state;

Small fix, but it meant every ZIP page now had a working internal link to its parent state, which closes a major site-graph gap.

After Phase 2, my SSR word count went from 640 to about 890 on previously-thin pages. That's the threshold I cared about. Google's "soft 404" verdict is based on relative content depth, not an absolute word count, but more depth is always better than less.

Phase 3: The geocoding gap I didn't know I had

While I was at it, I noticed something weird. Some ZIP pages were rendering with lat/lng of (0, 0). This made the distance calculations on the page nonsensical ("nearest gas station: 8,247 miles away"). It also meant the "nearby ZIPs" grid was showing up empty for those pages.

The cause: my ZIP-to-lat/lng resolver had a single source: zippopotam.us. It's free, fast, and most of the time correct. But for some valid US ZIPs (75072 in McKinney TX, for one), it returns a 404.

I rebuilt the resolver as a 4-tier fallback chain:

zipContent.json (a static file with 33,620 ZIPs and pre-resolved coords)
Redis cache (per-request resolved coords)
zippopotam.us API
Nominatim API (slower but covers the gaps)
Placeholder (0, 0) with degraded behavior

I'll write up the chain in detail in a separate post. The point for this post: when GSC flagged these pages as soft 404s, the broken geocoding was part of the picture even though it wasn't the headline issue.

Phase 7B: The second sweep

Three days after deploying Phases 1 through 3, I ran the GSC validation again. About 80% of the URLs had cleared. Some hadn't. So I wrote a script (find-redirect-candidates.js) that programmatically tested every plausible city slug variant against the live site:

const variants = [
  '/' + city,
  '/' + city + '-' + state,
  '/' + city.replace(' ', '-'),
  '/cheap-gas-' + city,
  // ...
];

for (const v of variants) {
  const res = await fetch('https://example.com' + v);
  if (res.status === 404) console.log(v);
}

This caught 35 more 404s I hadn't found in the GSC export. Stale links in old blog posts I'd written, third-party links from a directory submission I'd forgotten about, typo URLs in my own social media posts. Each one got a redirect.

Phase 7B added 35 new redirects, bringing the total to 101. I deployed those, and the second GSC validation came back clean.

Phase 8: Per-state context (the depth fix that actually moved the needle)

After all the redirects and SSR enrichment, I still had some pages stuck in "crawled, currently not indexed." Word count was up. Internal links were up. But Google was still skeptical.

The thing I hadn't done: make the templated content actually different across pages. My "save tips" section was different by city, but my page-level content above the fold was nearly identical. A page about ZIPs in California and a page about ZIPs in Maine had no state-specific context.

I built a stateContext.ts module:

const STATE_CONTEXT = {
  CA: "California's gas prices are shaped by the state's unique CARB...",
  TX: "Texas typically has some of the lowest gas prices in the country...",
  // ...15 hand-written for top-traffic states
  // ...35 generated from a parameterized template for the rest
};

export function getStateContext(state: string): string {
  return STATE_CONTEXT[state] ?? defaultContext(state);
}

The 15 hand-written paragraphs are 80 to 120 words each. They explain the state's gas tax, refinery capacity, regulatory regime, and seasonal pricing patterns. These are the things you'd say to a friend if they asked "why are gas prices weird in California?"

The other 35 states get a templated paragraph with state-specific variables (avg price, neighboring states, gas tax rate). Templated, but with enough variation that each is unique.

After deploying Phase 8, the previously-stuck pages started getting indexed within 10 days. Not all of them. But enough that I stopped worrying about them.

Final SSR word count on a representative ZIP page when measured post-deploy: 810 visible words, up from 635 before Phase 8 alone. The whole journey took it from 640 to 810: a +170 word lift made of mostly per-state context plus the SSR enrichment.

The diagnostic that lied to me

One quick aside that became its own post. While diagnosing the soft 404s, I wrote a script to grep the SSR output for content markers ("does this page have an EIA average price rendered?"). The script reported zero matches across 14 URLs. I spent an hour debugging the data layer before realizing the script was broken.

The cause: React inserts HTML comments between adjacent text expressions during SSR. My regex was failing on the comment boundary. The data was rendering correctly the whole time. My detector was the bug.

I wrote that one up separately. The summary: strip HTML comments before any content matching on Next.js SSR output.

The numbers

Before:

87 hard 404s in GSC
61 soft 404s in GSC
~640 SSR words on cached-miss ZIP pages
1 source of truth for ZIP geocoding (and gaps)
2-hop redirect chain from http apex to https www

After:

0 hard 404s (101 redirects in next.config.mjs, 1 Cloudflare Redirect Rule)
0 soft 404s after Phase 8 (per latest GSC validation)
~810 SSR words on the same pages
4-tier geocoding chain with 100% coverage of 33,620 ZIPs
1-hop redirect

Total commits: 7. Total deploy waves: 3 (April 17, April 23, April 27). Total weekend hours: I stopped counting somewhere around 14.

Things I'd do differently

If I were starting this side project again:

Set up GSC URL Inspection alerts before launch. I had GSC connected but wasn't watching it daily. The 404 problem accumulated for weeks before I noticed.
Add a curl -IL check to my deploy script. A redirect chain check would have caught the http+https+slug double-hop before it became a Google problem.
Server-render the unique parts of every templated page from day one. Anything that varies per page should be in the SSR HTML, not the client bundle. Loading states are the enemy of programmatic SEO.
Hand-write the top 10-20% of templated content. The remaining 80% can be generated, but the long-tail-of-the-long-tail is where Google will smell templating and dock you. Hand-writing the highest-traffic variants is high leverage.
Keep a diff log of redirect rules. Mine grew to 101 entries in next.config.mjs and I'd already lost track of which ones I added when. A separate JSON file with timestamps would have been smarter.
Don't use real URLs from your own site in code examples or narrative text in technical articles. Markdown auto-links them into real backlinks, which means typos and placeholders in your article become 404 reports on your site three weeks later. Use example.com for everything except actual call-to-action links.

What I learned about Google's verdict mechanism

Three things I genuinely didn't know before this:

Soft 404 is sticky. Once Google decides a page is a soft 404, fixing the page doesn't immediately clear the verdict. You have to ask GSC to re-validate, wait 1-4 weeks for re-crawl, and accept that some of those URLs will not come back even after you fix them. The verdict has memory.

"Crawled, currently not indexed" is the bench. Google has a finite indexation budget per site. Pages they don't think are worth indexing go on the bench. You can move pages off the bench by improving them, but it's not automatic and it's not fast.

Internal redirect hops add up. Each hop is a small signal-loss for Google. If your CDN does one redirect and your framework does another, you're below capacity even if both are individually correct. Audit your hop count.

If you've been through a similar programmatic-SEO recovery, I'd love to hear what your phase breakdown looked like. Mine was four phases over two weekends. The longest phase by far wasn't the bulk redirects (that was a few hours). It was Phase 8: the part where I had to admit my templated content was actually pretty thin and rewrite the per-state context by hand.

Templating gets you to 33,620 pages fast. Earning the right to keep those pages indexed takes longer.

Why your `[^<]+` regex is silently breaking on React SSR output

GasPriceCheck — Thu, 30 Apr 2026 06:02:34 +0000

Picture this. You've shipped a programmatic SEO site, a few thousand pages of templated content. Google flags 14 URLs as soft 404s. You write a quick diagnostic: hit each URL, fetch the SSR HTML, check for a few content markers (a price string, a state average, a section header). Confirm what's really rendering, fix what's missing, move on.

That was the plan. Forty minutes in, my script told me 0 of 14 pages had any EIA price data rendered. None. I was about to dig into the data fetching layer when something nagged at me. I curled one of the URLs by hand. The price was right there in the HTML. Plain text, easy to spot.

The script was lying. The regex was lying.

It took me longer than I want to admit to figure out what was happening, so here's the writeup so you don't burn the same hour.

The regex that "worked"

The diagnostic was straightforward. For each ZIP code page, I wanted to detect whether the EIA state average price was rendered in the SSR HTML. The component looked roughly like this:

<p>
  Texas average: $3.42 per gallon as of {eiaDate}.
</p>

So I grepped the SSR output for the marker pattern:

const html = await fetch(url).then(r => r.text());
const match = html.match(/Texas average: \$([^<]+) per gallon/);

The capture group uses [^<]+ to pick up "everything until the next tag." Standard pattern. I've used variations of this in dozens of throwaway scrapers.

Across all 14 URLs, this regex returned no match. Not "match with empty capture." No match at all.

Meanwhile, when I curled the same URL and read the response, the rendered text was right there:

Texas average: $2.97 per gallon as of 4/22/2026.

How does a regex miss text that's literally in the string?

What React is actually putting in your HTML

Here's the thing nobody told me about server-rendered React. When you have multiple adjacent text expressions inside a single element, React inserts an HTML comment as a hydration boundary marker. The actual SSR output for that paragraph looked like:

<p>Texas average: $<!-- -->2.97<!-- --> per gallon as of <!-- -->4/22/2026<!-- -->.</p>

Each {expression} interpolation gets bracketed by . This is intentional. React uses these markers during hydration to know where one text node ends and the next one begins. Without them, React can't reconcile the SSR text with what its virtual DOM says should be there, because adjacent text nodes get merged in the browser.

So when my regex hits Texas average: $, the next character is < (start of the comment). The [^<]+ capture group requires at least one non-< character. It fails immediately. The regex moves on, finds no other "Texas average:" anchor in the page, and reports no match.

The price ($2.97) is in the HTML. It's just not adjacent to the anchor text. There's a comment node between them.

The fix

Strip the comments before any content matching. Two lines:

const html = await fetch(url).then(r => r.text());
const stripped = html.replace(/<!--[\s\S]*?-->/g, '');
const match = stripped.match(/Texas average: \$([^<]+) per gallon/);

Use [\s\S] rather than . because comments can span multiple lines and . in JavaScript doesn't match newlines without the s flag (and s flag support is patchy enough that I just default to [\s\S]).

Once I added that one line, the regex worked on all 14 URLs. The EIA data was rendering correctly the whole time. My "soft 404 root cause" was a bug in my diagnostic, not a real content gap.

Why this gotcha matters more than you think

The reason I'm writing this up: the failure mode is silent and high-impact.

If your regex returned an obvious garbage value, you'd debug it in five minutes. But mine returned no match at all. That's the same return value as "this content is genuinely missing." Which is the exact thing I was trying to detect. The diagnostic was indistinguishable from the bug it was looking for.

I trusted the output. I started forming hypotheses based on the output. "The EIA fetch must be failing on the server. Let me check my fallback chain. Let me check Redis." I burned an hour on those wrong paths because the symptom was confirmed by my (broken) detector.

Two takeaways that generalize beyond this exact case.

First, when you grep server-rendered React HTML, always strip comments first. Any regex with [^<]+, \w+, or word-boundary anchors will trip on the hydration markers. Some browser View Source viewers hide comments by default, which is part of why this gotcha is invisible. View the raw response with curl -s URL | head -200 and look for the  pattern. You'll see them everywhere, especially on text-heavy pages with lots of variable interpolation.

Second, validate your diagnostic before you trust its output. Run it against a known-good page. If your "missing content" detector reports content as missing on a page where you can visually confirm the content exists, your detector is broken, not the content. I should have done this sanity check before chasing fix hypotheses. I didn't, because the detector was "obviously simple."

This bit me on a Next.js 15 / React 18 project. I checked: the same hydration-comment behavior is documented in the Next.js docs as part of how React handles hydration. It's not going away. If you're parsing SSR HTML programmatically, assume comments are everywhere.

One last thing worth knowing

There's a sister gotcha. If you're using cheerio or another DOM parser instead of regex, you don't have this problem. The DOM parser walks the tree, and adjacent text nodes are joined when you call .text(). So $('p').text() returns "Texas average: $2.97 per gallon as of 4/22/2026." with no comment artifacts.

But if you're using regex (faster, simpler for one-off scripts), strip first, match second. Or write a one-line helper:

const stripComments = (html) => html.replace(/<!--[\s\S]*?-->/g, '');

Drop it at the top of every SSR-parsing script you write. Future-you will thank you.

That's the whole post. One bug, one fix, one hour I'm not getting back. Hope I saved you the same hour.

If you've hit a similar silent-failure-mode debugging story, drop it in the comments. I collect these.