DEV Community: Bcrypto

My monitor said "all clear" while it was completely blind

Bcrypto — Thu, 13 Aug 2026 03:09:15 +0000

I have a small watcher script. Its whole job is to notice when one specific web
page changes — a rewards program that, when it announces, opens a claim window
that closes in about 90 days. Miss the announcement, miss the window, miss the
money. So the watcher runs daily and tells me: changed, or no action needed.

One morning I read the logs and found this, the day before:

INCOME WATCHERS — no change
No action needed.

Reassuring. Except the line above it, from a different run, was:

⚠ fetch failed — fetch failed
⚠ fetch failed — fetch failed

Both of the watcher's fetches had failed that day. And it had still printed
"No action needed" and exited 0.

Why that's the worst possible bug for this program

A monitor has exactly one promise: if something happened, I'll tell you. The
one output it must never produce is a confident all-clear while blind.

Mine did. "I checked and nothing changed" and "I couldn't check at all" came out
identical — same message, same exit code, same green. If the announcement I'm
watching for had landed on a day my flaky connection dropped two requests, the
log would have said "no action needed" and the 90-day clock would have started
without me.

Here's the code that did it:

let html;
try {
  html = await fetchText(w.url);
} catch (e) {
  lines.push(`  ⚠ ${w.label}: fetch failed — ${e.message}`);
  continue;   // <-- skip this surface and carry on
}

continue skips a failed surface, and the run ends by reporting on the surfaces
it did read. Zero surfaces read, zero changes found, "no action needed." The
logic is technically correct and completely useless.

The fix

Two ideas. First, a transient failure isn't a verdict — retry before believing
it. Second, and the real point: "could not check" is a distinct state from
"nothing changed," and must never render as the second.

// Retry: transient DNS/TLS blips must not be mistaken for "nothing changed".
let html, lastErr;
for (let attempt = 1; attempt <= 3; attempt++) {
  try { html = await fetchText(w.url); lastErr = null; break; }
  catch (e) { lastErr = e; if (attempt < 3) await sleep(attempt * 3000); }
}

if (lastErr) {
  const fails = (state[w.id]?.consecutiveFailures || 0) + 1;
  state[w.id] = { ...state[w.id], consecutiveFailures: fails };
  failures.push({ label: w.label, error: lastErr.message, consecutive: fails });
  lines.push(`  ⚠ ${w.label}: COULD NOT CHECK (${fails}x) — ${lastErr.message}`);
  continue;
}

And the output now refuses to say all-clear when any surface went unchecked —
even in --quiet mode, because silence there is the whole bug:

if (failures.length) {
  console.log('\n  ⚠ DEGRADED — could not check ' + failures.length + ' surface(s).');
  console.log('    This is NOT "nothing changed". An announcement could have landed unseen.');
  const worst = Math.max(...failures.map(f => f.consecutive));
  if (worst >= 3) {
    console.log('    🚨 ' + worst + ' runs in a row failed. The watcher is effectively OFF.');
  }
  process.exitCode = 20;   // distinct from 10 (real change) so a cron can tell them apart
}

Three states now, three exit codes: 0 checked-and-clear, 10 something
changed, 20 couldn't check. A consecutive-failure counter means a watcher
that's been quietly dead for days announces it instead of blending into the
green.

The lesson

Absence of an alert can mean two very different things: nothing happened, or
I'm not watching anymore. Any monitor that renders both as the same calm
"all clear" isn't reassuring — it's the most dangerous kind of broken, because
it fails in the one direction you'll never think to check. Make "I couldn't
look" loud, and never let it wear the same face as "I looked, we're fine."

From the little automation stack behind
a project of mine. The watcher is boring.
Boring is the point — right up until it lies to you.

My security scanner cried wolf on Vercel and Linear. Here's the bug.

Bcrypto — Thu, 13 Aug 2026 03:09:10 +0000

I built a scanner that checks a live site for exposed secrets and files —
leaked API keys in JS bundles, a reachable .env, a published .git/config.
Before showing it to anyone, I ran it across a batch of well-known sites as a
sanity check.

It reported two CRITICAL findings on vercel.com. And two on linear.app.

Both companies employ people who do security for a living. My first thought was
the honest one: it's not them, it's my scanner. This is the story of why it was
wrong, because the failure mode is one every "does this file exist" check walks
into.

What it claimed

🔴 CRITICAL — Sensitive file exposed: /.git/config
🔴 CRITICAL — Sensitive file exposed: /.DS_Store

If true, that's a real leak: a served .git/config can expose repository
internals. So I checked by hand:

curl -s https://vercel.com/.git/config | head -c 200

<!DOCTYPE html><html><head><title>Vercel</title>...

That's not a git config. That's the homepage. Served with 200 OK, for a
path that does not exist.

The bug

Single-page apps route on the client. Ask the server for /.git/config,
/.env, /anything-at-all, and it can't know that's not a real app route — so
it returns 200 and the app shell, letting the client-side router sort it out.

My probe was, in effect:

const res = await fetch(origin + '/.git/config');
if (res.status === 200) {
  // treat as exposed
}

On a SPA, status === 200 is true for every path. So the scanner reported
.git/config exposed on every well-built SPA on the internet — which is to say,
precisely the sites whose engineers would notice and never trust it again.

Then it got worse. Looking at the older code, the "is this real" guard had this:

const looksReal =
  (path === '/.git/config' && /\[core\]/.test(body)) ||
  (path.endsWith('.json') && /[{[]/.test(body.trim())) ||
  (path === '/.git/config' ) ||   // <-- unconditionally true
  ...

There's a second /.git/config clause with no body check. Any 200 response for
that path matched. The one file whose content I most needed to verify was the
one I'd accidentally waved through.

The fix

A 200 proves the server answered. It proves nothing about what it answered.
So verify the content is actually the file, and treat any HTML response as the
catch-all it almost always is.

const SENSITIVE_FILES = [
  { path: '/.env',        sig: (b) => /^\s*(?:#|[A-Z][A-Z0-9_]*\s*=)/m.test(b) },
  { path: '/.git/config', sig: (b) => /\[core\]/i.test(b) },
  { path: '/.DS_Store',   sig: (b) => b.includes('Bud1') }, // binary magic bytes
  // ...
];

for (const { path: p, sig } of SENSITIVE_FILES) {
  const r = await safeFetch(origin + p);
  if (r.status !== 200) continue;
  const ctype = (r.headers.get('content-type') || '').toLowerCase();
  if (ctype.includes('text/html')) continue;          // SPA/404 catch-all
  const body = (await r.text()).slice(0, 800);
  if (/^\s*<!doctype html|^\s*<html/i.test(body)) continue; // mislabeled catch-all
  if (!sig(body)) continue;                            // 200, but not the real file
  report('critical', `Sensitive file exposed: ${p}`);
}

Three gates now: not HTML by content-type, not HTML by sniffing the body (some
servers mislabel), and a signature that matches the real file — [core] for a
git config, the Bud1 magic string for a .DS_Store, KEY=value for a
.env. A real exposed file passes all three; a SPA shell fails the first.

Proof it works both ways

The part I actually care about — a fix that stops false positives by also
suppressing real ones is worthless. So I tested both directions:

vercel.com, linear.app, github.com, stripe.com: 0 criticals (was 2 each on the first two)
A local server serving a genuine .env with SECRET_KEY=...: still caught

Then I locked it in with tests, because this is the highest-consequence logic in
the whole tool:

t('/.git/config rejects an HTML app shell', () =>
  assert.strictEqual(sig(SPA_SHELL), false));
t('/.git/config accepts a real git config', () =>
  assert.ok(sig('[core]\n\trepositoryformatversion = 0\n')));

The lesson

A scanner's credibility is entirely downside. Miss a real issue and you're no
worse than not scanning. Cry wolf on a well-run site once and everyone who saw
it stops believing every finding you'll ever produce. For a security tool the
false positive isn't a smaller bug than the false negative — in reputation terms
it's the larger one.

"HTTP 200 means the file exists" is an assumption almost everything on the web
quietly breaks. I'm glad I pointed the thing at Vercel before I pointed it at a
front page.

This was in Preflight, a free scanner for
exactly these exposures. It grades itself, and after this fix it earns the A.

I have 591 pages. Google indexed one. Here's how I'm debugging it.

Bcrypto — Tue, 11 Aug 2026 19:00:50 +0000

I run a directory site with 2,499 real vendor listings across Beirut, Dubai, Riyadh, Doha, Cairo and Casablanca. It has 591 programmatic landing pages and 1,182 URLs in its sitemap.

I checked what Google had actually indexed.

The homepage. That was it.

Not 591. Not 200. One.

This post is what I checked, what I ruled out, and the one diagnostic that actually decides what to do about it — because I got that last part wrong at first, and I suspect a lot of people do.

First instinct: something's broken on the site

That's where I went, and it's usually where everyone goes. So I worked the checklist.

robots.txt — correct. Allows everything, references the sitemap.

Sitemap — valid XML, 1,182 URLs, auto-generated, submitted.

Hub page — present. Something links into the page set; they aren't orphans.

Internal linking — 62 cross-links per page. Not a dead end.

Structured data — JSON-LD on every page.

Server-rendered content — real listings in the initial HTML. Not a JS shell.

Here's the fastest way to check that last one, and it's the one people skip:

curl -s https://yoursite.com/some-page | wc -w

If that number is small, that's roughly what a crawler sees. Your framework hydrating beautifully in a browser is irrelevant to a bot that doesn't run JavaScript. I've measured client sites with 10 words in the initial HTML that looked perfect to a human.

All six checks passed. Which meant the problem wasn't on the page.

The wrong conclusion I nearly published

My first explanation was: the site has no inbound links and no brand mentions anywhere, so Google has no reason to crawl it deeply. Authority problem. Go get links.

That's a coherent story. It might even be right. But I nearly wrote it up as fact, and it turns out there's a specific piece of data that decides it — and until you look at that data, you're guessing.

The diagnostic that actually matters

Google Search Console → Indexing → Pages → "Why pages aren't indexed".

Two statuses look similar and mean opposite things:

Discovered – currently not indexed
Google knows the URL exists but hasn't spent crawl budget on it. This is the authority/crawl-scheduling case. Links and internal-linking improvements genuinely help here.

Crawled – currently not indexed
Google fetched the page, looked at it, and decided not to index it. This is a quality judgement. For programmatic pages — city × category templates where only a couple of fields change — this usually means it read them as thin or near-duplicate.

If it's the second one, backlinks will not fix it. You can build links for six months and stay at one indexed page, because the pages themselves are what got rejected. The fix is consolidation: fewer, substantially different pages with real unique content, rather than more thin ones.

Same symptom. Opposite remedies. The export is free and takes two minutes, and I was about to skip it in favour of a narrative that felt right.

The trap in programmatic SEO

Programmatic SEO makes the satisfying work and the useful work feel identical.

Generating 600 pages is a fun afternoon. It produces a number that goes up. It looks like progress. And if your pages are being rejected rather than not reached, generating another 600 moves you from one indexed page to one indexed page.

I could have shipped a second batch and felt productive the whole time.

What I'd check on any site, in order

site:yourdomain.com in Google. Compare the count to how many pages you think you have. The gap is the actual story, and most people never look.
View source with JS disabled. If there's a loading spinner where your content should be, that's what you've published.
GSC → Indexing → Pages. Get the real status before choosing a fix.
Check whether you're blocking AI crawlers. Plenty of robots.txt files disallow GPTBot, ClaudeBot or PerplexityBot inherited from a template, and then people wonder why assistants never cite them:

   curl -s https://yoursite.com/robots.txt | grep -iE "gptbot|claudebot|perplexity"

Grep your bundles for keys. Unrelated to indexing, but while you're in there — anything starting sk_live, AKIA, or a service-role JWT that reached the browser is public regardless of what your backend does.

Where I've got to

The GSC export is the next thing I do, and it decides whether the next month is spent earning citations or consolidating pages. I'm writing this before I have that answer deliberately, because the interesting part isn't the resolution — it's that the symptom looked obvious and pointed at two incompatible fixes.

If you've got a programmatic site sitting at a fraction of its expected index coverage, check which status you have before you spend a month on the wrong one.

I automated the on-page half of these checks into a free scanner while debugging this — preflightscan.vercel.app, no signup. It won't tell you your GSC status; nothing can except GSC.