DEV Community: SteadyScrape

How to Check Whether ChatGPT and Perplexity Recommend Your Brand

SteadyScrape — Tue, 27 Oct 2026 02:54:56 +0000

More and more buying research starts with a question to an AI assistant instead of a list of blue links: "best CRM for a small business", "easiest invoicing tool for freelancers", "Notion alternatives for a 10-person team". The assistant answers with a short list of names. If your product isn't on that list, that buyer may never see you.

This post covers three things: why this is worth tracking, how to check it by hand in about an hour, and how to automate the check once a spreadsheet stops being enough. The example data is from a real test run on October 26, 2026.

Why AI answers are worth tracking

A few numbers, each from its original source:

ChatGPT reach. OpenAI says more than 1.2 billion people use ChatGPT each week (OpenAI, October 7, 2026).
Fewer clicks when an AI summary appears. Pew Research Center looked at the browsing data of 900 U.S. adults. When a Google search showed an AI summary, users clicked a traditional result in 8% of visits, compared with 15% when there was no summary. They clicked a link inside the summary in just 1% of visits (Pew Research Center, July 2025). That study covers Google's AI summaries, not ChatGPT, and it shows a correlation, not cause and effect. Still, the pattern is clear: when the answer is on the page, fewer people click through to compare.
Analysts expected the shift. In 2024 Gartner predicted that traditional search engine volume would drop 25% by 2026 as AI chatbots and virtual agents take over queries (Gartner press release, February 19, 2024). That was a forecast, not a measurement, but it explains why marketing teams started paying attention.

The practical takeaway is simple. For the questions your buyers ask, you want to know three things:

Are you mentioned at all?
Where do you appear (first pick, or fourth in the "also consider" list)?
Which sources does the assistant cite? Your own site, a review site, or a competitor's comparison page?

The third one matters more than it seems. Citations show you which pages the assistant actually trusts for your category, and those pages are where your content and PR effort will pay off.

The manual way (about an hour)

You don't need any tooling for a first look.

1. Write 10–20 buyer prompts. Use the words a buyer would use, not your brand name. Mix in:

category questions: "best CRM for small business"
use-case questions: "easiest CRM for a 5-person sales team"
alternative questions: "cheaper alternatives to Salesforce"

Keep your own brand out of most prompts. If you ask "is HubSpot good?" the answer will mention HubSpot, and that tells you nothing about organic visibility.

2. Ask each engine. Use ChatGPT and Perplexity in a private window or a fresh chat, so your history and memory don't steer the answer. Turn web search on if it isn't automatic.

3. Log the answer in a spreadsheet, one row per prompt and engine:

Prompt	Engine	We're mentioned?	Our position	Competitors mentioned	Our site cited?	Cited domains

4. Repeat it a few days later. This is the step people skip, and it's the important one. AI answers are not stable. In one of our own tests, we sent the same 6 prompts to Perplexity twice on the same day and compared the answers. Across the brands we tracked, 4 mentions disappeared and 2 new ones showed up between the two runs. One answer is an anecdote. You need a sample, repeated over time.

A small helper for the "position" column

Reading 40 answers and counting who came first gets old fast. If you paste answer text into a file, this short function finds which tracked brands are mentioned and in what order. It removes links first, so a cited URL like hubspot.com/pricing doesn't count as a mention:

import re

def brand_ranks(answer, brands):
    """brands: {"HubSpot": ["HubSpot", "HubSpot CRM"], ...} -> {brand: rank or None}"""
    text = re.sub(r"\[[^\]]*\]\([^)]*\)|https?://\S+", " ", answer)  # drop markdown links and bare URLs
    first = {}
    for brand, names in brands.items():
        hits = [m.start() for n in names
                for m in re.finditer(rf"(?<!\w){re.escape(n)}(?!\w)", text, re.I)]
        if hits:
            first[brand] = min(hits)
    order = sorted(first, key=first.get)
    return {b: (order.index(b) + 1 if b in first else None) for b in brands}

print(brand_ranks(answer, {
    "HubSpot": ["HubSpot", "HubSpot CRM"],
    "Pipedrive": ["Pipedrive"],
    "Zoho CRM": ["Zoho CRM", "Zoho"],
    "Salesforce": ["Salesforce"],
}))
# {'HubSpot': 2, 'Pipedrive': 1, 'Zoho CRM': None, 'Salesforce': None}

That output is from a real ChatGPT answer to "easiest CRM for a 5-person sales team". The answer did recommend Bigin, which is a Zoho product, but it never said "Zoho". The lesson: list product names as aliases, not just the company name. Otherwise you'll undercount.

Where the manual approach breaks down

It works for a first look. It gets painful when you want:

more than a couple dozen prompts, or several markets
both engines, every week, with the same settings each time
the cited URLs, not just the answer text
share of voice, meaning how often you're mentioned compared with each competitor across all prompts
a diff against last week: which prompts you gained, lost or moved on

That is the point where it makes sense to script it or use a hosted tool.

Automating it with the AI Visibility Tracker

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

We packaged this workflow into an Apify Actor: AI Visibility & Share of Voice Tracker (steady-scrape/ai-brand-visibility-tracker). For every prompt × engine it:

asks the engine through its official API with live web search (no browser automation, no scraping of chatgpt.com)
records, for your brand and each competitor: mentioned or not, mention count, rank, list position, a simple sentiment hint and the surrounding text
saves every cited URL and domain, and flags whether your own domain was cited
computes share of voice and visibility rate, overall and per engine
compares the run with your previous run if you give it a trackingId, so a weekly schedule becomes a tracker

An honest caveat first. The answers come from the developer APIs, not the consumer apps. The chatgpt engine uses an OpenAI model (by default gpt-6-luna, the model family OpenAI says powers ChatGPT's Free and Go tiers) with OpenAI's own web search, called through OpenRouter or OpenAI's API. The perplexity engine uses Perplexity's Agent API with web search. The apps add their own system prompts, personalization, memory and location, so a logged-in user can see a different answer. Treat the results as a repeatable, comparable sample, not a screenshot of what any one person sees.

Example input

This is the input from our real test run (2 prompts × 2 engines = 4 checks). maxChecks and maxCostUsd are safety caps:

{
  "prompts": [
    "best CRM for small business",
    "easiest CRM for a 5-person sales team"
  ],
  "brands": [
    { "name": "HubSpot", "aliases": ["HubSpot CRM"], "domains": ["hubspot.com"] }
  ],
  "competitors": [
    { "name": "Salesforce", "domains": ["salesforce.com"] },
    { "name": "Zoho CRM", "aliases": ["Zoho"], "domains": ["zoho.com"] },
    { "name": "Pipedrive", "domains": ["pipedrive.com"] },
    { "name": "Freshsales", "domains": ["freshworks.com"] }
  ],
  "engines": ["chatgpt", "perplexity"],
  "trackingId": "real-pricing-check",
  "maxChecks": 4,
  "maxCostUsd": 0.3
}

(HubSpot and friends are just an example category here. We have no connection to any of them.)

Example output (real run, trimmed)

All 4 checks succeeded in about 12 seconds. The interesting part is the second prompt, where the two engines disagreed. Here's the ChatGPT row, trimmed:

{
  "prompt": "easiest CRM for a 5-person sales team",
  "engine": "chatgpt",
  "model": "openai/gpt-6-luna",
  "answerSnippet": "**My pick: Pipedrive Lite** if you want a straightforward, sales-focused CRM your reps can learn quickly. ...",
  "brandMentioned": true,
  "bestBrandRank": 2,
  "brandDomainCited": true,
  "competitorsMentioned": ["Pipedrive"],
  "citedDomains": ["pipedrive.com", "zoho.com", "hubspot.com"],
  "brands": [
    { "brand": "HubSpot",   "mentioned": true,  "rank": 2, "listPosition": 2, "sentiment": "neutral", "domainCited": true },
    { "brand": "Pipedrive", "mentioned": true,  "rank": 1, "listPosition": null, "sentiment": "neutral", "domainCited": true },
    { "brand": "Zoho CRM",  "mentioned": false, "rank": null, "domainCited": true }
  ],
  "searchCalls": 2,
  "charged": true,
  "chargeEvent": "chatgpt-check"
}

And the Perplexity row for the same prompt:

{
  "prompt": "easiest CRM for a 5-person sales team",
  "engine": "perplexity",
  "answerSnippet": "**Pipedrive** is my pick for a five-person sales team: its visual deal pipeline is easy to understand ...",
  "brandMentioned": false,
  "bestBrandRank": null,
  "brandDomainCited": false,
  "competitorsMentioned": ["Pipedrive"],
  "citedDomains": ["capterra.com", "lessannoyingcrm.com"],
  "charged": true,
  "chargeEvent": "perplexity-check"
}

Same question, two engines: ChatGPT put HubSpot second and cited hubspot.com, while Perplexity didn't mention it at all. Perplexity built its answer from a Capterra review page and a smaller competitor's pricing page. That's exactly the kind of gap you'd want to know about, and it's invisible if you only check one engine.

The run also writes a summary row with share of voice across all 4 checks (trimmed):

{
  "recordType": "summary",
  "checksSucceeded": 4,
  "chargedUsd": 0.18,
  "shareOfVoice": [
    { "brand": "Pipedrive", "shareOfVoice": 0.3889, "visibilityRate": 1.0,  "avgRank": 2,    "citationRate": 0.5 },
    { "brand": "HubSpot",   "shareOfVoice": 0.3333, "visibilityRate": 0.75, "avgRank": 1.33, "citationRate": 0.75 },
    { "brand": "Zoho CRM",  "shareOfVoice": 0.2222, "visibilityRate": 0.5,  "avgRank": 2,    "citationRate": 0.75 },
    { "brand": "Freshsales","shareOfVoice": 0.0556, "visibilityRate": 0.25, "avgRank": 4,    "citationRate": 0 },
    { "brand": "Salesforce","shareOfVoice": 0,      "visibilityRate": 0,    "avgRank": null, "citationRate": 0.25 }
  ],
  "topCitedDomains": [
    { "domain": "hubspot.com", "checksCiting": 3 },
    { "domain": "zoho.com", "checksCiting": 3 },
    { "domain": "pipedrive.com", "checksCiting": 2 }
  ]
}

Two prompts is far too small a sample to conclude anything about these brands. It's here to show the format. Salesforce is a good example of why the citation column matters: it was never mentioned, but one answer still cited a salesforce.com page.

Running it from code

Any Apify Actor can be called over the Apify API. This runs the tracker and returns the dataset rows in one request:

curl -X POST \
  "https://api.apify.com/v2/acts/steady-scrape~ai-brand-visibility-tracker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @input.json

For a tracker, save the input as an Apify Task and put it on a weekly schedule. Keep the same trackingId and each run's summary includes a changes block against the previous run.

Pricing

It's pay per event, and you only pay for checks that return an answer:

Event	Price
ChatGPT check (one prompt answered by ChatGPT and analyzed)	$0.06
Perplexity check (one prompt answered by Perplexity and analyzed)	$0.03
Actor start	$0.00005 per run (Apify's standard fee)

Some examples:

The test run above (2 prompts × 2 engines) costs $0.18.
10 prompts on both engines cost $0.90 per run, so a weekly schedule comes to about $3.90 a month.
Failed, skipped and mock-mode checks are never charged. Set "mockMode": true to see the full output format for free before you spend anything.
Your Max charge per run setting is respected: the Actor only plans the checks that fit under it.

Wrapping up

Start by hand. Ten unbranded buyer prompts, two engines, one spreadsheet, and a second pass a few days later will tell you a lot about where you stand. Pay particular attention to the cited domains: they show you which review sites and comparison pages the assistants lean on for your category.

Once you want dozens of prompts, both engines, citations and week-over-week changes, automate it. You can write your own script, or use the AI Visibility Tracker if your time is worth more than $0.03–$0.06 a check.

Have you seen your brand show up (or vanish) in AI answers? We'd like to hear which prompts surprised you. Please share them in the comments.

How to Pull Job Postings Straight From Company Career Sites (Greenhouse, Lever, Ashby, Workday) as JSON

SteadyScrape — Mon, 26 Oct 2026 18:10:54 +0000

If you want job postings from specific companies (for a niche job board, a hiring-signal dashboard, salary research, or just an alert for your dream employer), you usually don't need to scrape HTML at all.

Most company careers pages are powered by an applicant tracking system (ATS) like Greenhouse, Lever, Ashby or Workday. These systems publish the job board through public endpoints, because the whole point is for the jobs to be seen. The careers page you look at in a browser is often just a thin front end over one JSON call.

This post shows how to call those endpoints directly with a few lines of Python, and what tends to go wrong once you do it for more than one company. Every endpoint below was tested with curl and the Python snippets were run on October 26, 2026.

Step 0: find the ATS and the "slug"

Open the company's careers page and click any job. The URL usually gives it away:

ATS	Job URL looks like	Slug
Greenhouse	`job-boards.greenhouse.io/stripe/jobs/...`	`stripe`
Lever	`jobs.lever.co/spotify/...`	`spotify`
Ashby	`jobs.ashbyhq.com/ramp/...`	`ramp`
Workday	`nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/...`	tenant `nvidia`, data center `wd5`, site `NVIDIAExternalCareerSite`

If the careers page lives on the company's own domain, the "Apply" button often links out to the ATS.

Greenhouse

import requests

HEADERS = {"User-Agent": "my-job-feed/0.1 (contact: you@example.com)"}

def greenhouse(slug):
    url = f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs"
    data = requests.get(url, params={"content": "true"}, headers=HEADERS, timeout=30).json()
    for j in data["jobs"]:
        yield {"id": f"greenhouse:{slug}:{j['id']}", "title": j["title"],
               "location": j["location"]["name"], "url": j["absolute_url"],
               "posted": j.get("first_published")}

greenhouse("stripe") returned 714 jobs in one request when I ran it (data["meta"]["total"] tells you the count). Leave out content=true if you don't need the HTML description; the response is much smaller.

Lever

def lever(slug):
    url = f"https://api.lever.co/v0/postings/{slug}"
    data = requests.get(url, params={"mode": "json"}, headers=HEADERS, timeout=30).json()
    for j in data:
        yield {"id": f"lever:{slug}:{j['id']}", "title": j["text"],
               "location": j["categories"].get("location"), "url": j["hostedUrl"],
               "posted": j["createdAt"]}  # epoch milliseconds, not ISO

lever("spotify") returned 89 postings as a plain JSON array. Lever also accepts skip and limit if you want to page through it. Note that createdAt is a Unix timestamp in milliseconds, and EU-hosted boards live on api.eu.lever.co instead.

Ashby

def ashby(slug):
    url = f"https://api.ashbyhq.com/posting-api/job-board/{slug}"
    data = requests.get(url, params={"includeCompensation": "true"}, headers=HEADERS, timeout=30).json()
    for j in data["jobs"]:
        yield {"id": f"ashby:{slug}:{j['id']}", "title": j["title"],
               "location": j["location"], "url": j["jobUrl"],
               "posted": j["publishedAt"],
               "pay": (j.get("compensation") or {}).get("compensationTierSummary")}

ashby("ramp") returned 173 jobs, descriptions included, and with includeCompensation=true many of them carry a pay range like $211.4K â€“ $290.6K â€¢ Offers Equity. Ashby is the friendliest of the bunch for salary data.

Workday (the hard one)

Workday has no documented public API for this, but every Workday careers site calls an internal JSON endpoint you can see in your browser's network tab. It's a POST, it's paginated, and it has quirks:

import time

def workday(tenant, dc, site, max_jobs=100):
    url = f"https://{tenant}.{dc}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs"
    offset, total = 0, None
    while offset < max_jobs:
        body = {"appliedFacets": {}, "limit": 20, "offset": offset, "searchText": ""}
        page = requests.post(url, json=body, headers=HEADERS, timeout=30).json()
        for j in page["jobPostings"]:
            yield {"title": j["title"], "location": j.get("locationsText"),
                   "path": j["externalPath"], "posted": j.get("postedOn")}
        if total is None:
            total = page["total"]  # only the first page reports it; later pages say 0
        offset += 20
        if offset >= total:
            break
        time.sleep(1)  # be polite

jobs = list(workday("nvidia", "wd5", "NVIDIAExternalCareerSite", max_jobs=60))

Things I hit while testing this against NVIDIA:

limit maxes out at 20. Asking for 50 returned HTTP 400.
total is only set on the first page. Pages after that report total: 0, so a naive while offset < page["total"] loop stops after page two. (I made exactly that mistake in my first version.)
The listing is thin. You get a title, a location string, a relative date like "Posted Today", and a path. The description, exact date and job ID need one extra request per job to .../wday/cxs/{tenant}/{site}{externalPath}.
Large boards seem to cap at 2,000. NVIDIA reports exactly total: 2000. Use searchText or facets to narrow it.
The data center (wd1, wd3, wd5, ...) and site name differ per company, so you can't guess them from the company name.

Pitfalls once you go beyond one company

Bad slugs return 404. Greenhouse, Lever and Ashby all return 404 for an unknown board. Handle it per company so one typo doesn't kill the whole run. Companies also switch ATS now and then, so a board that worked last month can 404 today.

Be polite. These are public endpoints, not an invitation to hammer them. Send a descriptive User-Agent, keep concurrency low (a handful of parallel requests), retry 429 and 5xx with exponential backoff, and don't poll more often than you need to. For most use cases, once a day is plenty.

Normalize early. Every ATS names things differently (text vs title, epoch ms vs ISO 8601, a location string vs an object). Map each one into your own schema right away and keep the raw payload around in case you need a field later.

Dedupe with a stable key. Use platform:slug:job_id, as in the snippets above. Titles aren't unique, and the same job can appear in several locations.

Track new and removed jobs. For alerts, what you usually want is the diff, not the full list. Store the IDs you've seen and compare:

import json, pathlib

def diff(board_key, jobs, state_file=pathlib.Path("seen.json")):
    state = json.loads(state_file.read_text()) if state_file.exists() else {}
    seen = set(state.get(board_key, []))
    current = {j["id"]: j for j in jobs}
    new = [j for jid, j in current.items() if jid not in seen]
    removed = sorted(seen - current.keys())
    state[board_key] = sorted(current)
    state_file.write_text(json.dumps(state))
    return new, removed

new, removed = diff("greenhouse:stripe", list(greenhouse("stripe")))

One catch: only mark a job as "removed" if you fetched the complete board. If you stopped paginating early (a Workday cap, a timeout, a keyword filter), missing jobs aren't necessarily gone.

Then there are the other ATSs. BambooHR, Workable, Recruitee, SmartRecruiters and Personio each have their own public endpoint, their own field names and their own pagination. None of them are hard on their own. Keeping nine of them working, normalized and tested is where the time goes.

If you'd rather not maintain this yourself

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

We packaged all of the above into an Apify Actor: ATS Job Feed (steady-scrape/ats-job-feed). What it does:

Covers 9 ATS platforms: Greenhouse, Lever, Ashby, Workday, BambooHR, Workable, Recruitee, SmartRecruiters and Personio, all mapped into one output schema.
Accepts board URLs, job URLs, platform:slug, or a company name or domain that it looks up in a bundled list of 381 boards (e.g. stripe.com). Names it can't find are reported, not guessed.
Has an "Only new jobs" mode that remembers what it already delivered and can also output removed-job records (only when it saw the full board listing, per the catch above).
Supports a per-company cap (maxJobsPerCompany), title/location/department/date filters, and optional full descriptions.
Pricing is pay-per-event: $1.50 per 1,000 jobs delivered. Removed-job records and the run summary are free.

Example input:

{
  "companies": [
    "https://job-boards.greenhouse.io/stripe",
    "lever:spotify",
    "ashby:ramp",
    "https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite"
  ],
  "keywords": ["engineer"],
  "excludeKeywords": ["intern"],
  "locations": ["US", "Canada"],
  "maxJobsPerCompany": 100,
  "includeDescription": true,
  "onlyNewSinceLastRun": true,
  "emitRemovedJobs": true
}

One output record. This is real data from one of our test runs (October 25, 2026), with the descriptions cut short here:

{
  "jobId": "greenhouse:discord:8806482002",
  "sourceJobId": "8806482002",
  "platform": "greenhouse",
  "boardSlug": "discord",
  "boardUrl": "https://job-boards.greenhouse.io/discord",
  "company": "Discord",
  "title": "Commercial Policy Lead",
  "department": "Policy",
  "team": null,
  "locations": [
    {"raw": "San Francisco Bay Area", "city": "San Francisco Bay Area", "region": "California",
     "country": "United States", "countryCode": "US", "isRemote": false}
  ],
  "locationText": "San Francisco Bay Area",
  "isRemote": false,
  "workplaceType": null,
  "employmentType": null,
  "salary": {"min": 192000.0, "max": 216000.0, "currency": "USD", "interval": "YEAR",
             "source": "structured", "raw": "Base Salary"},
  "postedAt": "2026-09-15T16:52:11Z",
  "updatedAt": "2026-10-21T00:27:20Z",
  "jobUrl": "https://job-boards.greenhouse.io/discord/jobs/8806482002",
  "applyUrl": "https://job-boards.greenhouse.io/discord/jobs/8806482002",
  "requisitionId": "R-107406",
  "descriptionHtml": "<div class=\"content-intro\"><p>Discord has a highly engaged c...",
  "descriptionText": "Discord has a highly engaged community of millions of daily ...",
  "changeType": null,
  "scrapedAt": "2026-10-25T21:07:46Z"
}

Fields the ATS doesn't publish come back as null rather than guessed (Greenhouse, for example, doesn't expose employment type here). For scale: a run with the default input (Stripe, Spotify, Ramp and NVIDIA, capped at 100 jobs per company) returned 389 jobs in about 29 seconds on Apify. Most of that time is the per-job detail requests on Workday.

It's not a search engine for every job on the internet. Like the scripts above, it fetches the boards you list.

Wrapping up

For one or two companies, the snippets above are all you need: one GET for Greenhouse, Lever or Ashby, and a paginated POST for Workday. Once you're tracking dozens of boards across several ATSs, the work moves to normalization, error handling, politeness and change tracking. You can build that yourself (the diff() function above is a decent start) or use a hosted tool if your time is worth more.

If you've found other ATS endpoints or quirks worth knowing about, please share them in the comments.