DEV Community: mrnoyy

Your Pages Are Not Indexed — a Debugging Checklist That Starts With the Boring Causes

mrnoyy — Thu, 17 Sep 2026 09:12:48 +0000

A site ships, weeks pass, and the pages are still not in the index. The usual advice jumps straight to content quality. Before that, there is a list of mechanical causes that are far more common and much easier to verify.

1. Can it actually be fetched

Use the URL inspection tool and fetch the live page. Not your browser — your browser has your cookies, your session, and possibly a cached copy.

Things that show up here regularly:

The page returns 200 to you and 403 to the crawler, usually via a bot-protection rule.
The HTML is nearly empty because content renders client-side after a fetch the crawler did not wait for.
A noindex tag left over from staging. This is embarrassingly common and takes ten seconds to check.

Check robots.txt too. A Disallow written for a staging path can match more than intended.

2. Canonical tags pointing elsewhere

If page A declares page B as canonical, you have asked search engines not to index A. That is working as designed.

The ways this happens by accident:

A template with a hardcoded canonical, so every page claims the homepage.
Cross-posting to a platform without setting the canonical there, so the copy on the bigger domain wins.
Protocol or host mismatch — canonical says https://www.example.com/page, the site serves https://example.com/page. Those are different URLs.

Every URL should declare itself canonical unless you genuinely mean otherwise.

3. Redirect chains from trailing slashes

/page redirects to /page/, which redirects to https://example.com/page/. Each hop costs a request and dilutes the signal, and if your sitemap lists the pre-redirect form, every crawl starts by bouncing.

Pick a form. Make internal links, the sitemap, and the canonical all use it. Then check what your host does automatically, because it may be adding a hop you did not configure.

4. Sitemap says one thing, the site says another

A sitemap generated from your content config lists what should exist. If the build dropped a page, the sitemap still lists it and you are pointing crawlers at 404s. Enough of those and the whole file is trusted less.

Generate the sitemap from the built output on disk, not from the source config. Then it cannot describe pages that were not shipped.

While you are there: lastmod should reflect real changes. Setting it to today's date on every build teaches crawlers that your dates mean nothing.

5. Slugs that changed without redirects

If URLs changed during a redesign and the old ones return 404, you did not migrate — you deleted a site and published a new one. Everything the old URLs had earned is gone.

Old URL to new URL, 301, one hop each. Keep them indefinitely. Links on other people's sites do not update themselves.

6. Near-duplicate pages competing

Twelve pages built from one template with a city name swapped in are, functionally, one page. Search engines pick one and ignore the rest, and "crawled, currently not indexed" is exactly what that looks like.

The fix is not more pages. It is fewer pages that each say something the others do not.

7. It has genuinely not been long enough

New domains get crawled slowly. Weeks is normal, not a symptom. Submitting the same URL repeatedly does not speed it up.

Do the useful things instead: internal links from pages that are already indexed, a clean sitemap referenced in robots.txt, and links from somewhere real.

A note on tooling

Check more than one search engine's webmaster tools. They surface different diagnostics, and a crawl problem that shows up in one is often the same problem in both — sometimes reported more clearly on the side you were not looking at.

The pattern I keep seeing: the cause is almost always in items 1 through 5, and almost always something mechanical that a person could have checked on day one. Content is worth improving. It is just rarely the reason a page is missing from the index entirely.

Building a Store Without a Payment Gateway — Manual Checkout on Next.js and Supabase

mrnoyy — Tue, 15 Sep 2026 09:14:06 +0000

Not every market has easy access to Stripe. In a lot of places the real payment rails are bank transfer and mobile wallets, confirmed by a human. That changes the architecture more than you would think: without a gateway webhook, your database is the source of truth for payment state, and you have to design it that way deliberately.

The order state machine

Five states, one direction, no shortcuts:

pending → awaiting_payment → payment_submitted → confirmed → fulfilled
                                    ↓
                                 rejected

pending — cart submitted, nothing else
awaiting_payment — customer has the transfer details
payment_submitted — customer says they paid and gave a reference
confirmed — a human matched it against the account
rejected — no match found

The important rule: only confirmed unlocks anything. payment_submitted is a customer claim, not a fact. Treating it as one is how you get taken.

Schema

create table orders (
  id uuid primary key default gen_random_uuid(),
  reference text unique not null,
  status text not null default 'pending',
  customer_name text not null,
  customer_phone text not null,
  total_amount numeric(10,2) not null,
  payment_method text,
  payment_reference text,
  confirmed_at timestamptz,
  confirmed_by uuid references auth.users(id),
  created_at timestamptz default now()
);

create table order_items (
  id uuid primary key default gen_random_uuid(),
  order_id uuid references orders(id) on delete cascade,
  product_id uuid references products(id),
  quantity int not null check (quantity > 0),
  unit_price numeric(10,2) not null
);

unit_price is copied onto the line item, not read through the product. Prices change; an order is a record of what was agreed at the time.

The short human-readable reference matters more than it looks. Customers type it into a transfer note and read it over the phone. A UUID is unusable for that.

Row Level Security is not optional

Supabase exposes your tables directly to the browser. The anon key is public — it is in your JavaScript. Without RLS, anyone can read every order in your database.

alter table orders enable row level security;

create policy "insert own order" on orders
  for insert to anon with check (status = 'pending');

create policy "staff read all" on orders
  for select to authenticated using (
    exists (select 1 from staff where user_id = auth.uid())
  );

Note what the insert policy does: an anonymous visitor can create an order, but only in pending. They cannot post one that is already confirmed. Every status transition after that happens server-side under a service role, never from the client.

Total on the server

The client sends product ids and quantities. It does not send a total.

const { data: products } = await supabase
  .from('products')
  .select('id, price')
  .in('id', items.map((i) => i.product_id));

const total = items.reduce((sum, item) => {
  const product = products.find((p) => p.id === item.product_id);
  if (!product) throw new Error('Unknown product');
  return sum + product.price * item.quantity;
}, 0);

Anything the browser sends is a suggestion. If a price can be set client-side, it will be.

Idempotency on submit

A customer on a slow connection taps the button three times. Without protection that is three orders and three confused phone calls.

Generate an idempotency key when the checkout form mounts, send it with the order, and put a unique constraint on it. The second and third submit hit the constraint and return the existing order instead of creating a new one.

The admin panel is the product

With a gateway, a webhook flips the status. Without one, a person does. So that screen deserves real attention:

Orders sorted oldest first — the customer waiting longest gets served first.
The payment reference copyable in one tap, because they are cross-checking it against a bank statement.
Confirm and reject as separate, deliberate actions with a note field.
Every transition written to an audit table with a user id. When something is disputed weeks later, "who confirmed this and when" is the only question that matters.

What you gain

No processing fees, no chargebacks, no gateway approval process. That is a genuinely good trade for a small catalogue in a market where cards are not the norm.

What you pay for it is latency. An order sits until a human looks at it, so set the expectation on the confirmation screen: tell the customer the window, and send something when the status changes. Most complaints in this model are not about the payment method, they are about silence.

Handling Rate Limits Properly — Backoff, Retry-After, and the Bugs In Between

mrnoyy — Thu, 10 Sep 2026 09:12:40 +0000

Every integration eventually meets a 429. The first version of the fix is usually a sleep(1000) between calls. Here is what actually holds up.

Read the response before you guess

Most APIs tell you what they want:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1643723400

Check those headers on every response, not just on 429s. X-RateLimit-Remaining dropping toward zero is your signal to slow down before you get blocked. Reacting only to failures means you are always one failure behind.

Retry-After has two legal formats

RFC 7231 allows both a delay in seconds and an HTTP-date:

Retry-After: 30
Retry-After: Fri, 31 Jul 2026 07:00:00 GMT

parseInt on the second returns 31, which is not thirty-one seconds — it is the day of the month. Your retry fires early, hits the limit again, and you burn attempts.

function retryAfterMs(header, fallbackMs = 30000) {
  if (!header) return fallbackMs;
  const raw = String(header).trim();

  if (/^\d+$/.test(raw)) return Number(raw) * 1000;

  const when = Date.parse(raw);
  if (!Number.isNaN(when)) {
    const diff = when - Date.now();
    return diff > 0 ? diff : fallbackMs;
  }
  return fallbackMs;
}

Note the diff > 0 check. A clock skew between your machine and the server can produce a date already in the past, and a negative timeout means you retry instantly.

Backoff for everything else

Retry-After is authoritative when present. When it is not — or for 500s and 502s — exponential backoff with a cap:

const waitMs = Math.min(2 ** attempt * 1000, 60000);

Add jitter if more than one worker is retrying. Without it, every client that failed at the same moment retries at the same moment, and you have rebuilt the stampede you were avoiding.

Cap the attempts. An unbounded retry loop against an API that is down is a job that never ends and a bill that keeps growing.

Separate the retryable from the fatal

This is the part people skip, and it is the one that causes real damage:

429, 502, 503, 504 — retry.
401, 403 — do not retry. Your key is wrong or revoked. Retrying will not fix it and some providers count repeated auth failures against you.
400, 422 — do not retry. The payload is malformed. It will be malformed the second time too.

A loop that retries a 422 will send the same broken request five times and then report a rate limit problem you do not have.

Concurrency is the actual cause

Most 429s come from Promise.all over an array. Ten items, ten simultaneous requests, and no limit anywhere:

// this is a rate limit waiting to happen
const results = await Promise.all(ids.map((id) => fetchItem(id)));

Promise.all has no concurrency control. For small batches, a sequential loop with a delay is honest and easy to reason about:

for (const id of ids) {
  await fetchItem(id);
  await sleep(delayMs);
}

For larger ones, a worker pool of three to five is usually plenty. The throughput difference against a rate-limited API is smaller than you expect, because the limit is the ceiling either way.

Do not fetch the same thing twice

The cheapest request is the one you skip. Cache responses that do not change within a run, and keep a local record of what you already did between runs. In my publishing pipeline a content hash decides whether a post needs an update at all — most runs make zero write calls because nothing changed.

Make the waiting visible

console.log(`429 — waiting ${Math.round(waitMs / 1000)}s`);

A CI job that sits silent for ninety seconds looks hung. One line of output turns a support question into a shrug.

None of this is complicated. It is just that the naive version works fine in testing with three items and fails on the first real batch, which is exactly when you are least in the mood to debug it.

Writing a Frontmatter Parser in 60 Lines Instead of Adding a Dependency

mrnoyy — Tue, 08 Sep 2026 09:13:11 +0000

Most markdown pipelines pull in a YAML library to read the block at the top of the file. If your frontmatter is a handful of strings, a boolean, and a list of tags, you can skip it. Here is the version I use, and the parts I got wrong first.

What the block actually contains

---
title: "My Post"
tags: [webdev, node]
canonical_url: https://example.com/blog/my-post
published: true
---

Strings, one array, one boolean. That is a tiny subset of YAML. Real YAML has anchors, multi-line scalars, nested maps, and a type-coercion table that has surprised people for years. None of it is in my files.

Splitting the file

const match = raw.match(/^---\r?\n([\s\S]*?)\r?\n---\r?\n?([\s\S]*)$/);

Three details in that regex, each learned the hard way:

\r?\n everywhere, because a file edited on Windows and committed without normalization has CRLF line endings and a \n-only pattern silently fails to match.
[\s\S]*? non-greedy, so a --- horizontal rule later in the body does not become the closing delimiter.
The ^ anchor, so a --- in the middle of a document is not mistaken for an opening block.

And before any of that:

const raw = fs.readFileSync(filePath, 'utf8').replace(/^\uFEFF/, '');

A byte order mark at the start of the file means the first character is not -, the regex does not match, and the error message says the frontmatter is missing when you can see it right there. That one cost me a genuinely embarrassing amount of time.

Parsing the keys

Line by line, with one piece of state for list continuation:

function parseFrontmatterBlock(block) {
  const out = {};
  let currentKey = null;

  for (const line of block.split(/\r?\n/)) {
    if (!line.trim()) continue;

    const dashItem = line.match(/^\s*-\s+(.*)$/);
    if (dashItem && currentKey) {
      if (!Array.isArray(out[currentKey])) out[currentKey] = [];
      out[currentKey].push(unquote(dashItem[1]));
      continue;
    }

    const kv = line.match(/^([A-Za-z0-9_]+)\s*:\s*(.*)$/);
    if (!kv) continue;

    const [, key, rawValue] = kv;
    currentKey = key;
    const value = rawValue.trim();

    if (value === '') {
      out[key] = [];
    } else if (value.startsWith('[') && value.endsWith(']')) {
      out[key] = value.slice(1, -1).split(',').map((v) => unquote(v.trim())).filter(Boolean);
    } else if (value === 'true' || value === 'false') {
      out[key] = value === 'true';
    } else {
      out[key] = unquote(value);
    }
  }

  return out;
}

The dashItem branch has to come first. A dashed list item like - something: else also matches the key-value pattern, and if you check that one first you get a key named - something.

The URL trap

The obvious key-value regex is /^(\w+):\s*(.*)$/. Now parse this:

canonical_url: https://example.com/blog/post

Split on the first colon and you are fine. Split on every colon, or use a lazy value group, and you get https as the value. Match the key against [A-Za-z0-9_]+ explicitly and take the rest of the line as the value, whatever is in it.

Validate at the boundary

Parsing is the cheap part. The value comes from failing loudly:

if (!meta.title) throw new Error(`${basename}: "title" is required`);
if (!body.trim()) throw new Error(`${basename}: body is empty`);

A file with a typo in a key name should stop the run, not publish an untitled post. My CI runs a check command over every file before anything touches the network, so a bad file fails in two seconds instead of halfway through a batch.

When to just use the library

Reach for a real parser the moment you need nested objects, multi-line values, dates as actual Date objects, or files written by people who did not write the parser. Sixty lines is worth it when you control both ends and the schema is five fields. It is not worth it as a matter of principle.

The point is not that dependencies are bad. It is that "read five known keys from a fenced block" is a smaller problem than "parse YAML", and the small problem has a small solution.

Next.js Static Export for Content Sites — the Settings That Actually Matter

mrnoyy — Thu, 03 Sep 2026 09:12:45 +0000

I run a set of content-heavy sites on Next.js with output: 'export'. No server, no serverless functions, just HTML on a CDN. It is fast and cheap, but the defaults are tuned for the server runtime and a few of them will quietly cost you traffic.

The config I end up with every time

/** @type {import('next').NextConfig} */
module.exports = {
  output: 'export',
  trailingSlash: true,
  images: { unoptimized: true },
};

Three lines, and each one solves a real problem.

trailingSlash

With output: 'export', a route like /reviews/example is written to disk. Without trailingSlash, you get reviews/example.html. With it, you get reviews/example/index.html.

Why that matters: most static hosts serve /reviews/example/ from the directory form without a redirect, but serve /reviews/example by redirecting first. If half your internal links point one way and your sitemap points the other, every crawl spends requests on 301s before reaching content. Pick one form, set it in config, and make your sitemap match exactly.

Also check what your host does by default. Some normalize trailing slashes, some do not, and finding out after launch means a wave of redirect chains.

images: unoptimized

The default Image Optimization API needs a running server. With a static export it either fails the build or, on some hosts, silently falls back to an optimizer you did not intend to use.

unoptimized: true makes next/image emit a plain <img> with the width, height, and lazy loading attributes intact. You still get the layout-shift protection, which is the part that affects Core Web Vitals. You lose automatic resizing, so generate the sizes you need at build time — or accept the original file size, but then actually check what you are shipping. A 2MB hero image will undo everything else on this list.

Dynamic routes need generateStaticParams

Every dynamic segment must enumerate its paths at build time:

export async function generateStaticParams() {
  const posts = await getAllPosts();
  return posts.map((post) => ({ slug: post.slug }));
}

Miss one and that page simply is not in your output. It is not a 404 you will see in dev — dev renders on demand and looks perfect. You find out from the live site or from a crawl.

I now diff the built out/ directory against my content source as a build step. A page count mismatch fails the build. That check has caught two silent drops for me.

The sitemap problem

Static export does not generate a sitemap. Plenty of tutorials hand you a sitemap.xml route handler, which works on the server runtime and does nothing here.

Two options that do work:

A Node script after the build that walks out/ and writes the XML from the actual files on disk.
next-sitemap as a postbuild script.

I prefer the first, because it reflects what was really exported rather than what the config says should exist. If generateStaticParams dropped a page, a config-driven sitemap still lists it and you send crawlers to a 404. A disk-driven sitemap cannot lie to you.

Same for robots.txt — put it in public/ and it gets copied verbatim.

What you give up

Be honest about this before choosing static export:

No middleware, no rewrites, no server-side redirects. Redirects move to your host's config, and every host spells it differently.
No ISR. Any content change means a rebuild and redeploy.
No route handlers. Forms need a third-party endpoint or an external function.

For a marketing site, docs, or a review site that changes a few times a week, none of that hurts. For anything with per-user content, you want the server runtime.

Build times

Static export builds every page every time. At a few hundred pages that is fine. Past a couple of thousand it becomes the slow part of your day.

What helped most was caching the data-fetching layer. If your pages come from an API or a database, fetch once into a local JSON cache and let the build read from that. The rendering is rarely the bottleneck — the network round trips are.

None of this is exotic. It is just that the export path and the server path share a framework and not a set of assumptions, and the docs cover both without always telling you which one you are reading about.

Publishing to Dev.to from GitHub Actions with Zero Dependencies

mrnoyy — Wed, 26 Aug 2026 07:30:34 +0000

I write posts in markdown, commit them, and they show up on Dev.to. No dashboard, no copy-paste, no npm install. Here is the whole thing.

Why zero dependencies

The pipeline does three things: read a file, hash it, send JSON over HTTP. Node 18+ has fs, crypto, and fetch built in. Adding axios, gray-matter, and dotenv would mean a node_modules restore on every CI run to save maybe forty lines of code.

My whole runner step is now:

- uses: actions/setup-node@v4
  with:
    node-version: '22'
- run: node src/index.js publish

No install step at all. The job finishes in about fifteen seconds.

The three things that broke first

1. The auth header is not Bearer

Almost every API I had touched before uses Authorization: Bearer <token>. Dev.to does not. The key goes in a header literally named api-key:

headers: {
  'api-key': apiKey,
  'Content-Type': 'application/json',
  accept: 'application/vnd.forem.api-v1+json',
}

I lost twenty minutes to 401s before reading the docs properly rather than assuming.

2. Retry-After comes back in two different formats

When you hit the rate limit you get a 429 with a Retry-After header. Sometimes it is a number of seconds. Sometimes it is a full HTTP-date string like Fri, 31 Jul 2026 07:00:00 GMT.

My first implementation was parseInt(header, 10). On the date format that returns 31, so the retry fires half a minute later, hits the limit again, and the loop burns through its attempts for nothing.

function retryAfterMs(header, fallbackMs = 30000) {
  if (!header) return fallbackMs;
  const raw = String(header).trim();

  if (/^\d+$/.test(raw)) return Number(raw) * 1000;

  const when = Date.parse(raw);
  if (!Number.isNaN(when)) {
    const diff = when - Date.now();
    return diff > 0 ? diff : fallbackMs;
  }
  return fallbackMs;
}

Handle both, and always keep a fallback for the case where the header is missing or garbage.

3. Tags are sanitized silently

Dev.to strips everything that is not a lowercase letter or a digit. next-js becomes nextjs. You get a 200 back and no warning at all.

That matters if you hash your content to decide whether to update a post, like I do. The local file says next-js, the remote says nextjs, so the hash never matches and every run pushes a pointless update. Normalize before hashing, not after:

const normalizeTags = (tags = []) =>
  tags
    .map((t) => String(t).toLowerCase().replace(/[^a-z0-9]/g, ''))
    .filter(Boolean)
    .slice(0, 4);

Four tags is the ceiling. The fifth is dropped, again silently.

Not creating duplicates

The API has no idempotency key, so POST /articles twice gives you two articles. I keep a small state file mapping slug to article id and a content hash:

{
  "my-post": {
    "id": 1234567,
    "url": "https://dev.to/user/my-post-abcd",
    "hash": "3f9a1c2b7e04"
  }
}

New slug means POST. Known slug with a changed hash means PUT /articles/:id. Known slug, same hash, do nothing. The workflow commits that file back to the repo after each run, with contents: write permission and a path filter on the trigger so the commit does not retrigger the workflow.

Canonical URLs

This is the part that actually matters if you have your own blog. Every cross-post should carry canonical_url pointing back to your original. Without it you have published the same article on a domain with far more authority than yours, and search engines will pick that one.

I derive it from the filename so I cannot forget:

if (!post.canonical_url && CANONICAL_BASE) {
  post.canonical_url = `${CANONICAL_BASE}/${post.slug}`;
}

And the script warns loudly on any post that ends up without one.

Dry run first

--dry-run prints exactly what would be sent, resolved tags and canonical included, and touches nothing. Every schema change I make gets a dry run before it gets an API call. It has caught more mistakes than any test I wrote.

The whole thing is about 250 lines across three files. If you cross-post to more than one platform, the same shape works — the state file just grows a key per platform.