How I Built a Company Career & ATS Hiring Signal Scraper + AI Markdown RAG Extractor in Node.js

#javascript #node #ai #webdev

When building LLM-powered RAG pipelines or B2B outbound intelligence systems, two data bottlenecks come up repeatedly:

Raw HTML is terrible for LLM context windows — navigation menus, cookie banners, inline scripts, and SVG icons burn 70–85% of your tokens before the model even sees the actual content.
Static company directories don't tell you who has active budget — knowing whether a company is actively hiring on Greenhouse, Lever, Ashby, or Workable (and what engineering/product roles are open right now) is one of the strongest B2B buying signals available.

As part of my suite of 8 production Node.js scrapers on the Apify Store, I built two focused actors to solve both problems using fast HTTP/Cheerio pipelines in Node.js:

Here is a practical walkthrough of the architecture and implementation behind both tools.

Part 1: Stripping DOM Noise & Chunking Semantic Markdown for RAG

Most naive web-to-Markdown scrapers run turndown or innerText across document.body. That leaves mega-menus, footer sitemaps, cookie consent modals, and hydration blobs inside your vector database embeddings.

In AI Website to Clean Markdown & RAG Chunk Extractor, the pipeline runs in three deterministic stages:

1. Pruning Non-Content DOM Subtrees First

Before converting anything to Markdown, we strip noisy tags and common boilerplate selectors while preserving semantic headings (h1–h4), paragraphs, lists, code blocks, and tables:

function cleanDomForMarkdown($) {
    const noiseSelectors = [
        'script', 'style', 'noscript', 'iframe', 'svg', 'canvas',
        'nav', 'footer', 'header', 'aside',
        '[role="navigation"]', '[role="banner"]', '[role="contentinfo"]',
        '.cookie-banner', '#cookie-consent', '.newsletter-popup', '.ads'
    ];
    $(noiseSelectors.join(', ')).remove();

    const root = $('main, article, [role="main"]').first();
    return root.length ? root : $('body');
}

2. Heading-Aware Token Chunking (~300 Words / ~400 Tokens)

Splitting Markdown by fixed character counts often slices tables or code blocks in half and loses the parent section heading. Instead, we chunk on paragraph boundaries while prepending the active # / ## heading hierarchy to each chunk so every vector embedding retains its semantic context:

function chunkMarkdownWithHeadings(markdown, maxWords = 300) {
    const blocks = markdown.split(/\n{2,}/);
    const chunks = [];
    let currentHeading = '';
    let buffer = [];
    let wordCount = 0;

    for (const block of blocks) {
        const trimmed = block.trim();
        if (!trimmed) continue;

        if (/^#{1,3}\s+/.test(trimmed)) {
            currentHeading = trimmed;
        }

        const words = trimmed.split(/\s+/).length;
        if (wordCount + words > maxWords && buffer.length > 0) {
            chunks.push({
                headingContext: currentHeading,
                content: buffer.join('\n\n'),
                wordCount,
                estimatedTokens: Math.ceil(wordCount * 1.33)
            });
            buffer = currentHeading ? [currentHeading] : [];
            wordCount = currentHeading ? currentHeading.split(/\s+/).length : 0;
        }

        buffer.push(trimmed);
        wordCount += words;
    }

    if (buffer.length > 0) {
        chunks.push({
            headingContext: currentHeading,
            content: buffer.join('\n\n'),
            wordCount,
            estimatedTokens: Math.ceil(wordCount * 1.33)
        });
    }

    return chunks;
}

Each output item includes the page metadata (title, canonical URL, OpenGraph/JSON-LD summary), full cleaned Markdown, word/token estimates, and the ready-to-embed ragChunks array for OpenAI, Claude, or LangChain.

Part 2: Detecting 10+ ATS Platforms & Hiring Signals from Company Websites

When enriching B2B leads, a company with 8 open engineering roles on Ashby or Greenhouse is a very different prospect from a dormant brochure site.

In Company Career Page, ATS & Hiring Signal B2B Lead Scraper, we start from a company's root domain, discover its /careers or /jobs links, and fingerprint the underlying Applicant Tracking System (ATS):

const ATS_SIGNATURES = [
    { name: 'Greenhouse', pattern: /boards\.greenhouse\.io|job-boards\.greenhouse\.io|greenhouse\.io\/embed/i },
    { name: 'Lever', pattern: /jobs\.lever\.co|api\.lever\.co/i },
    { name: 'Ashby', pattern: /jobs\.ashbyhq\.com|ashbyhq\.com/i },
    { name: 'Workable', pattern: /apply\.workable\.com|workable\.com/i },
    { name: 'BambooHR', pattern: /\.bamboohr\.com\/(careers|jobs)/i },
    { name: 'BreezyHR', pattern: /\.breezy\.hr/i },
    { name: 'Personio', pattern: /\.jobs\.personio\.(de|com)/i },
    { name: 'Teamtailor', pattern: /teamtailor\.com/i },
    { name: 'SmartRecruiters', pattern: /jobs\.smartrecruiters\.com/i },
    { name: 'Workday', pattern: /myworkdayjobs\.com/i }
];

function detectAtsPlatforms(html, outboundLinks) {
    const combined = [html, ...outboundLinks].join(' ');
    return ATS_SIGNATURES
        .filter(sig => sig.pattern.test(combined))
        .map(sig => sig.name);
}

Prioritizing Talent & Recruiting Contacts

Alongside open role titles and external ATS board URLs, the actor extracts emails and prioritizes recruiting-specific inboxes (careers@, hr@, talent@, jobs@, people@) ahead of generic support addresses:

function prioritizeHiringEmails(emails) {
    const priorityRegex = /^(careers|jobs|talent|hr|recruiting|people)@/i;
    return [...new Set(emails)].sort((a, b) => {
        const aPri = priorityRegex.test(a) ? 0 : 1;
        const bPri = priorityRegex.test(b) ? 0 : 1;
        return aPri - bPri;
    });
}

Try the Full Suite of 8 Actors & Browser Extensions

Both actors are live on the Apify Store alongside 6 additional B2B, e-commerce, and technical SEO scrapers:

All 8 Apify Actors: https://apify.com/kabeer099
Portfolio & MV3 Browser Extensions: https://kabeer.site/
GitHub: https://github.com/Kabeer099

If you have feedback on chunking heuristics or ATS edge cases, drop a comment below!

DEV Community