A 403 error page is not a noindex audit of the original page

A response inspector can make a subtle evidence mistake: it finds a robots tag on a block page, then labels the intended source page as noindex. The tag was observed, but on the wrong kind of response.

Keep two fields separate: what the request returned and what that response establishes about the intended page. A 403 observation deserves its own record. It does not establish that the publisher added a noindex rule to the original page.

Here is a small, dependency-free Node.js classification function. It makes no requests and does not retry a blocked response.

The contract

• Inspect the HTTP status before interpreting HTML robots rules.
• Leave source directives unresolved for failed requests, known challenges and responses outside this example's HTML scope.
• Pass only directives already resolved as applicable to the crawler being assessed. This function does not parse HTML, choose a user agent or resolve scoped X-Robots-Tag headers.
• Call the successful result no_noindex_observed, never indexed or SEO passed.

Save this as response-evidence.cjs:

function classifyHtmlObservation({ status, contentType = '', challenge = false, applicableDirectives = [] }) {
  if (!Number.isInteger(status) || status < 100 || status > 599)
    return { kind: 'invalid_status', sourceDirectives: null };
  if (status !== 200)
    return { kind: 'http_observation', status, sourceDirectives: null };
  if (challenge)
    return { kind: 'challenge_unverified', status, sourceDirectives: null };
  if (!/^(text\/html|application\/xhtml\+xml)(?:;|$)/i.test(contentType.trim()))
    return { kind: 'outside_html_scope', status, sourceDirectives: null };
  if (!Array.isArray(applicableDirectives) || applicableDirectives.some(x => typeof x !== 'string'))
    throw new TypeError('Resolve applicable directives to an array of strings first');
  const tokens = [...new Set(applicableDirectives.flatMap(x => x.toLowerCase().split(/[,\s]+/)).filter(Boolean))];
  return {
    kind: tokens.some(x => x === 'noindex' || x === 'none') ? 'noindex_observed' : 'no_noindex_observed',
    status,
    sourceDirectives: tokens
  };
}
module.exports = { classifyHtmlObservation };

Run the boundary cases

Save this beside it as response-evidence.test.cjs, then run node response-evidence.test.cjs:

const assert = require('node:assert/strict');
const { classifyHtmlObservation: check } = require('./response-evidence.cjs');
const base = { status: 200, contentType: 'text/html; charset=utf-8' };
const cases = [
  [{ ...base, status: 403, applicableDirectives: ['noindex'] }, 'http_observation'],
  [{ ...base, status: 429, applicableDirectives: ['noindex'] }, 'http_observation'],
  [{ ...base, challenge: true, applicableDirectives: ['noindex'] }, 'challenge_unverified'],
  [{ ...base, status: '200' }, 'invalid_status'],
  [{ ...base, contentType: 'application/pdf' }, 'outside_html_scope'],
  [{ ...base, applicableDirectives: ['NOINDEX, follow'] }, 'noindex_observed'],
  [{ ...base, applicableDirectives: ['none'] }, 'noindex_observed'],
  [{ ...base, applicableDirectives: ['noimageindex'] }, 'no_noindex_observed'],
  [{ ...base, applicableDirectives: [] }, 'no_noindex_observed'],
  [{ ...base, contentType: 'application/xhtml+xml', applicableDirectives: ['index'] }, 'no_noindex_observed']
];
for (const [input, kind] of cases) assert.equal(check(input).kind, kind);
for (const [input] of cases.slice(0, 5)) assert.equal(check(input).sourceDirectives, null);
const input = { ...base, applicableDirectives: ['index, follow', 'follow'] };
const before = JSON.stringify(input), out = check(input);
assert.deepEqual(out.sourceDirectives, ['index', 'follow']);
out.sourceDirectives.push('changed');
assert.equal(JSON.stringify(input), before);
assert.throws(() => check({ ...base, applicableDirectives: [null] }), TypeError);
console.log('PASS: 10 response classes, unresolved directives on five blocked/out-of-scope cases, deduplication, input preservation and type validation');

The fixtures cover ten response classes, including a 403 body containing noindex, a successful challenge page, the none directive, and noimageindex. They also check that the caller's input is preserved and the returned token array is independent.

What the result does not prove

Absence of an observed noindex rule is only one finding. This helper does not check robots.txt access, canonical selection, soft errors, rendered content, crawler-specific responses or actual search-index status. Preserve the response timestamp and final URL in the surrounding request record; redirects need separate treatment. A known challenge remains unresolved even when its HTTP status is 200.

Google documents both page-level robots directives and HTTP status handling. A successful response does not guarantee indexing.

I build AGProLabs. This is an engineering note about evidence labels, not a claim about search visibility or trading results. AI-assisted article and code; the included fixtures were executed locally.