DEV Community: TORIX

One API for Greenhouse, Lever, Ashby & Workday Jobs: Hiring Signals in Python

TORIX — Thu, 15 Oct 2026 10:31:38 +0000

Job postings are among the most honest public data a company produces. A company hiring three data engineers is
building a data platform. A company hiring a "Head of Demand Gen" is about to buy marketing tools. A competitor posting
Kubernetes roles in Berlin is expanding to Europe.

Most of these jobs live on applicant tracking systems (ATS) such as Greenhouse, Lever, Ashby, Workable,
SmartRecruiters, Recruitee, Personio and Workday. Each has its own URL scheme, JSON format, pagination and quirks.
Workday is notoriously painful. LinkedIn and Indeed copies are delayed, deduplicated badly and hard to scrape.

This guide pulls jobs straight from the source with one call and one schema.

The tool

Career Site Jobs Scraper on Apify:

input: company domains (stripe.com), ATS board URLs (boards.greenhouse.io/stripe, jobs.lever.co/palantir, jobs.ashbyhq.com/ramp), or Workday URLs. It auto-detects the ATS from the company website;
output: one normalized row per job, with title, department, team, location(s), isRemote, workplaceType, employmentType, seniority, salary range when published, postedAt, apply URL and an optional clean-text description;
filters: title keywords (include/exclude), locations, remote only, posted within N days;
price: $6 per 1,000 jobs, with filtered-out jobs free.

Python: who is hiring data people this month?

import collections, os
from apify_client import ApifyClient

companies = ["stripe.com", "huggingface.co", "notion.so", "ramp.com", "https://jobs.lever.co/palantir"]

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("murch1k/career-site-jobs-scraper").call(run_input={
    "companies": companies,
    "keywords": ["data", "machine learning", "analytics"],
    "excludeKeywords": ["intern"],
    "postedWithinDays": 30,
    "includeDescription": False,
    "maxResults": 500,          # hard cost cap
})
jobs = list(client.dataset(run["defaultDatasetId"]).iterate_items())

by_company = collections.Counter(j["companyName"] for j in jobs)
for company, n in by_company.most_common():
    print(f"{company:25} {n} open data roles")

A row (shortened, values illustrative):

{
  "companyName": "Hugging Face",
  "ats": "workable",
  "title": "Machine Learning Engineer - Open Source",
  "location": "Remote, Europe",
  "isRemote": true,
  "workplaceType": "remote",
  "employmentType": "full-time",
  "salaryMin": null,
  "postedAt": "2026-10-02T00:00:00.000Z",
  "url": "https://apply.workable.com/huggingface/j/…"
}

Three things you can build in an afternoon

1. Sales trigger list. Run weekly over your target-account list with keywords that match your product
("data engineer" for a data tool, "SDR" for a sales tool). New rows mean a new buying signal. Push them to Slack or your
CRM with an Apify webhook.

2. Niche job board / newsletter. Pick 200 companies in a niche (climate tech, Rust, AI safety), schedule a daily run
with postedWithinDays: 1, and publish the feed. Every posting links back to the original apply URL.

3. Market research for LLMs. Turn on includeDescription and let a model extract the tech stack, seniority mix and
salary bands across a whole industry. Descriptions arrive as clean text, so they are cheap to embed.

Give it to an AI agent (MCP)

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=murch1k/career-site-jobs-scraper",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}

Ask Claude or Cursor: "Which of these 20 companies are hiring ML engineers in Europe right now? Group by company."

Gotchas

Unknown ATS: if a company uses an in-house careers page, auto-detection returns nothing and you are not charged.
Salary: only present when the employer publishes it (common in US states with pay-transparency laws).
Dates: some ATSs don't expose a posting date. Those jobs are kept even with a postedWithinDays filter.

Full Python and Node.js examples are in the Apify Scrapers Pack repo, alongside scrapers for LinkedIn
Jobs, StepStone, Dice and a remote-jobs aggregator.

The toolkit used in this series

All five are pay-per-result Actors on Apify (no subscription; the Apify free plan's monthly credits cover a test run):

YouTube Transcript Scraper: transcripts with timestamps from videos, channels and playlists.
Career Site Jobs Scraper: open roles from Greenhouse, Lever, Ashby, Workday and 4 more ATSs.
Google Trends Scraper: interest over time, related queries and topics.
Substack Scraper: posts and newsletter metadata as clean text for LLMs.
Website Contact Details Extractor: emails, phones and socials, one row per domain.

Python, Node.js, LangChain and MCP examples for each one are in apify-scrapers-pack on GitHub.
A star there helps other people find it.

Extract Emails, Phones & Socials From 1,000 Company Websites (Python + No-Code)

TORIX — Thu, 15 Oct 2026 10:31:36 +0000

You have a list of domains: a CRM export, a conference exhibitor list, results from Google Maps, or companies hiring
for a role you sell into. You need a contact row for each company: a general email, a phone number, LinkedIn and
other social profiles.

Doing it by hand takes about two minutes per site. Writing your own crawler is easy for 10 sites and painful for 1,000.
You end up dealing with contact pages hidden in footers, [email protected] Cloudflare obfuscation,
info [at] acme [dot] com, junk like logo@2x.png, five spellings of the same phone number, and sites that only answer
from residential IPs.

This guide shows a hosted approach that returns one row per domain and only charges for sites that worked.

The tool

Website Contact Details Extractor on Apify:

crawls the homepage plus likely contact pages (/contact, /about, /impressum, footer links), 6 pages per site by default;
decodes Cloudflare email protection and [at]/[dot] obfuscation, and reads schema.org / JSON-LD business data;
filters junk (image filenames, noreply@, Sentry DSNs, placeholder numbers) and deduplicates phones;
ranks a primaryEmail (on-domain role inboxes like info@, sales@, hello@ first) and records which page each email came from.

Benchmark on the platform: 300 company websites (~1,330 pages) in 77 seconds. Price: $3 per 1,000 websites, with
failed sites free.

Python

pip install apify-client
export APIFY_TOKEN=<your token>

import csv, os
from apify_client import ApifyClient

domains = [line.strip() for line in open("domains.txt") if line.strip()]

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("murch1k/website-contact-details-extractor").call(run_input={
    "websites": domains,
    "maxPagesPerSite": 6,
    "sameDomainEmailsOnly": True,   # drop gmail.com / agency emails that aren't the company's own
    "maxConcurrency": 50,
})

with open("contacts.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["domain", "primaryEmail", "allEmails", "phone", "linkedin"])
    for r in client.dataset(run["defaultDatasetId"]).iterate_items():
        socials = r.get("socials") or {}
        w.writerow([r["domain"], r.get("primaryEmail"), " ".join(r.get("emails") or []),
                    r.get("primaryPhone"), " ".join(socials.get("linkedin") or [])])

Each dataset row looks like this:

{
  "domain": "joesstonecrab.com",
  "status": "ok",
  "primaryEmail": "info@joesstonecrab.com",
  "emails": ["info@joesstonecrab.com"],
  "primaryPhone": "3056734611",
  "socials": {
    "facebook": ["https://facebook.com/JoesStoneCrab"],
    "instagram": ["https://instagram.com/joesstonecrab"]
  },
  "pagesScraped": 5
}

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('murch1k/website-contact-details-extractor').call({
    websites: ['https://apify.com', 'https://crawlee.dev'],
    maxPagesPerSite: 6,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.table(items.map(({ domain, primaryEmail, primaryPhone }) => ({ domain, primaryEmail, primaryPhone })));

No-code: Google Sheets / Make / n8n / Zapier

On Apify, open the Actor, paste domains into Websites, and click Start. Export as CSV or Excel, or connect the
Google Sheets integration so results land in a sheet. In Make, n8n or Zapier, use the Apify module "Run an Actor" and
map the dataset items to your CRM.

Pipeline ideas

Google Maps → contacts: search "dentist in Austin, TX" with a Maps scraper, then feed the website column here.
Hiring signal → contacts: companies posting "Head of Growth" roles are buying marketing tools. Pull open jobs from their career sites (see the next article in this series), then enrich the domains.
AI agent enrichment: expose the Actor through the Apify MCP server, and your agent can enrich a lead on demand: "find the contact email and LinkedIn page for acme.com".

Tips

403s: a few sites block all cloud traffic. Enable Apify Proxy and re-run only the failed rows (they weren't charged).
JS-only menus: if no contact links are found in the HTML, the actor still tries the standard contact paths.
Phones are returned the way the site prints them, normalised to digits with an optional +. Numbers without a country code are local to the site's country.

Use it responsibly

The actor only collects what websites publish about themselves, mostly role inboxes. Named personal emails are still
personal data under GDPR, CAN-SPAM and similar laws. Have a legitimate interest, keep outreach relevant, honour
opt-outs, and don't send bulk email from a domain you care about without warming it up.

Code for this and other scrapers is in the Apify Scrapers Pack repo.

The toolkit used in this series

All five are pay-per-result Actors on Apify (no subscription; the Apify free plan's monthly credits cover a test run):

YouTube Transcript Scraper: transcripts with timestamps from videos, channels and playlists.
Career Site Jobs Scraper: open roles from Greenhouse, Lever, Ashby, Workday and 4 more ATSs.
Google Trends Scraper: interest over time, related queries and topics.
Substack Scraper: posts and newsletter metadata as clean text for LLMs.
Website Contact Details Extractor: emails, phones and socials, one row per domain.

Python, Node.js, LangChain and MCP examples for each one are in apify-scrapers-pack on GitHub.
A star there helps other people find it.

YouTube Transcripts for RAG Without the YouTube Data API (Python, 30 Lines)

TORIX — Wed, 14 Oct 2026 16:26:09 +0000

YouTube is one of the best knowledge sources you can put behind an LLM: conference talks, tutorials, product demos,
earnings calls, podcasts. The problem is getting the text out.

The official route doesn't work for this. The YouTube Data API's captions.download endpoint needs OAuth and only
works for videos you own (or where the owner allows third-party contributions). For "index this competitor's channel"
or "build a RAG bot over 300 conference talks", it's a dead end. The usual Python workaround libraries break whenever
YouTube changes its internals and get your server IP blocked after a few hundred requests.

In this guide we'll take a hosted transcript scraper, turn transcripts into timestamped chunks, and load them into a vector
store. Each chunk links back to the exact second in the video.

What we're building

YouTube URLs / @channel / playlist / search
        │
        ▼
YouTube Transcript Scraper (Apify)  →  transcript + timestamped segments + metadata
        │
        ▼
chunker (300 words, 50 overlap, keeps start time)
        │
        ▼
chunks.jsonl  →  Chroma / pgvector / Pinecone / Qdrant

Step 1 — get transcripts

Create a free Apify account, copy your API token, and install the client:

pip install apify-client
export APIFY_TOKEN=<your token>

The YouTube Transcript Scraper accepts video links, Shorts,
@channels, playlists or plain search keywords. It returns one row per video with the full text and timestamped segments:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("murch1k/youtube-transcript-scraper").call(run_input={
    "urls": ["https://www.youtube.com/@AndrejKarpathy"],
    "maxVideosPerSource": 20,
    "languages": ["en"],          # ordered preference; manual captions beat auto-generated
    "includeSegments": True,
})
videos = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(len(videos), videos[0]["title"], videos[0]["wordCount"])

A row looks like this (shortened, values illustrative):

{
  "videoId": "zjkBMFhNj_g",
  "title": "[1hr Talk] Intro to Large Language Models",
  "channelName": "Andrej Karpathy",
  "url": "https://www.youtube.com/watch?v=zjkBMFhNj_g",
  "language": "en",
  "isAutoGenerated": false,
  "wordCount": 11873,
  "transcript": "hi everyone so recently I gave a 30-minute talk on large language models …",
  "segments": [{ "start": 0.16, "duration": 4.32, "text": "hi everyone so recently I gave a" }]
}

Pricing is per transcript ($5 per 1,000). Videos without captions, private or removed videos cost nothing. Apify's Free plan
monthly credits cover a few hundred videos, which is enough to prototype.

Step 2 — chunk with timestamps

Don't embed a one-hour transcript as a single vector. Split it into ~300-word windows with overlap, and keep the start
time of each window. A &t=123s deep link makes answers verifiable:

CHUNK_WORDS, OVERLAP = 300, 50

def chunks(video):
    buf, start = [], None
    for seg in video.get("segments") or []:
        start = seg["start"] if start is None else start
        buf.extend(seg["text"].split())
        if len(buf) >= CHUNK_WORDS:
            yield {"text": " ".join(buf), "url": f"{video['url']}&t={int(start)}s", "title": video["title"]}
            buf, start = buf[-OVERLAP:], seg["start"]
    if buf:
        yield {"text": " ".join(buf), "url": f"{video['url']}&t={int(start or 0)}s", "title": video["title"]}

docs = [c for v in videos for c in chunks(v)]

Step 3 — load into a vector store

Any store works. Here's Chroma, because it needs zero setup:

import chromadb

col = chromadb.PersistentClient("yt_index").get_or_create_collection("talks")
col.add(
    ids=[f"{i}" for i in range(len(docs))],
    documents=[d["text"] for d in docs],
    metadatas=[{"url": d["url"], "title": d["title"]} for d in docs],
)
hits = col.query(query_texts=["why do LLMs hallucinate?"], n_results=3)
for meta in hits["metadatas"][0]:
    print(meta["title"], "→", meta["url"])

Pass the hits to your model with an instruction like "answer only from the excerpts and cite the URLs" and you have
a cited, timestamp-accurate RAG bot over any channel.

Bonus — let the agent fetch transcripts itself (MCP / LangChain)

Instead of pre-indexing, you can give an agent the scraper as a tool. With the hosted Apify MCP server, add this to
Claude Desktop, Cursor or VS Code:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=murch1k/youtube-transcript-scraper",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}

Then ask: "Summarise the last 5 videos on @mkbhd and list every product he recommends."

In LangChain it is one line: ApifyActorsTool("murch1k/youtube-transcript-scraper") from the langchain-apify package.

Practical tips

Languages: pass "languages": ["en", "es"] in order of preference; set translateTo to get everything in one language.
Whole channels: maxVideosPerSource caps how far back you go; channelTab: "shorts" switches to Shorts.
Cost cap: maxResults puts a hard limit on any run, and your Apify spending limit is respected.
Freshness: schedule the run daily on Apify and only embed new videoIds.

The full script is in the Apify Scrapers Pack repo
(python/youtube_transcripts_rag.py), next to Node.js versions and examples for other scrapers.

Respect creators' copyright: use transcripts for analysis, search and summaries, not for republishing their content.

The toolkit used in this series

All five are pay-per-result Actors on Apify (no subscription; the Apify free plan's monthly credits cover a test run):

YouTube Transcript Scraper: transcripts with timestamps from videos, channels and playlists.
Career Site Jobs Scraper: open roles from Greenhouse, Lever, Ashby, Workday and 4 more ATSs.
Google Trends Scraper: interest over time, related queries and topics.
Substack Scraper: posts and newsletter metadata as clean text for LLMs.
Website Contact Details Extractor: emails, phones and socials, one row per domain.

Python, Node.js, LangChain and MCP examples for each one are in apify-scrapers-pack on GitHub.
A star there helps other people find it.