DEV Community: gafari radouane

Scrape YouTube videos and channels without the API quota (Python)

gafari radouane — Tue, 03 Nov 2026 10:06:52 +0000

The YouTube Data API is free — until you hit the 10,000-unit daily quota. A single search costs 100 units, so you get about 100 searches a day. For research, influencer lists or trend tracking that's not enough.

In this tutorial we'll collect YouTube search results, whole channels and full video stats without an API key or quota, then turn them into a ranked list of channels.

Disclosure: I built the Apify Actor used below (YouTube Scraper). It costs $0.50 per 1,000 results, video details included, and runs on Apify's free plan.

What you get

Search results: videos, Shorts, channels or playlists, with YouTube's filters (upload date, length, sort by views)
Videos: views, likes, comment count, exact publish date, duration, description, tags, category
Channels: subscribers, total views, video count, country, joined date, links, plus their videos, Shorts, live streams and playlists

1. Try it without code

Open YouTube Scraper, click Try for free and run:

{ "searchQueries": ["home workout"], "maxResults": 50, "uploadDate": "week", "sortBy": "views" }

You get the 50 most viewed "home workout" videos of the week. Export to CSV/Excel or JSON.

2. Whole channels

Paste channel URLs (@handle, /channel/ID, /c/name) in startUrls:

{
  "startUrls": ["https://www.youtube.com/@mkbhd", "https://www.youtube.com/@veritasium"],
  "channelTabs": ["videos", "shorts"],
  "maxVideosPerChannel": 100,
  "includeChannelInfo": true
}

You get one row per channel (subscribers, total views, country, links…) and its latest videos and Shorts.

3. From Python: find the fastest-growing channels in a niche

pip install "apify-client>=3"

from apify_client import ApifyClient
from collections import defaultdict

client = ApifyClient("YOUR_APIFY_TOKEN")  # Apify Console → Settings → API & Integrations

run = client.actor("rt-solutions/youtube-scraper").call(run_input={
    "searchQueries": ["ai agents tutorial", "n8n automation", "claude code"],
    "maxResults": 100,
    "resultType": "video",
    "uploadDate": "month",
    "includeVideoDetails": True,   # exact date, likes, comment count, tags
})

views = defaultdict(int); videos = defaultdict(int)
for v in client.dataset(run.default_dataset_id).iterate_items():
    if v.get("type") != "video":
        continue
    views[v["channelName"]] += v.get("viewCount") or 0
    videos[v["channelName"]] += 1

top = sorted(views.items(), key=lambda kv: kv[1], reverse=True)[:15]
for name, total in top:
    print(f"{name:35} {videos[name]:3} videos  {total:>12,} views this month")

A video item (shortened):

{
  "type": "video", "id": "2reQut8WNGc",
  "title": "Macbook PROLED XDR Impressions: They Said They Wouldn't!",
  "viewCount": 10027295, "likeCount": 129851, "commentCount": 11000,
  "publishedAt": "2026-10-28T19:50:34-07:00", "duration": "10:54",
  "channelName": "Marques Brownlee", "channelUrl": "https://www.youtube.com/@mkbhd"
}

4. Next steps

Need what is said in the videos? Feed the video URLs to YouTube Transcript.
Need audience reactions? Use YouTube Comments Scraper.
Schedule the run daily in Apify Console to track a niche over time.

Cost

$0.50 per 1,000 results (cheaper on paid plans), full video details included. Missing or private videos are free. The example above (~300 videos) costs about $0.15.

Notes

Comment counts are rounded the way YouTube shows them (e.g. 11K).
Channel business emails are hidden behind a YouTube login and are not collected.

Questions? Open an issue on the Actor page.

Scrape LinkedIn job postings without logging in (Python + Google Sheets)

gafari radouane — Mon, 02 Nov 2026 19:02:36 +0000

LinkedIn shows job postings to anyone, even without an account. In this tutorial we'll collect job postings — with full descriptions, salary when shown, applicant count, seniority and Easy Apply flag — for several keywords and locations, filter out what we don't want, and monitor new jobs every day.

No LinkedIn account, no cookies, no risk to your profile.

Disclosure: I built the Apify Actor used below (LinkedIn Jobs Scraper). It costs $0.30 per 1,000 jobs with full details and runs on Apify's free plan.

What you get for every job

title, company, location, exact posted date, applicants, salary (when LinkedIn shows it), seniority level, employment type, industries, apply type (Easy Apply or company site), job poster (when shown), full description (text and HTML), job URL.

1. Try it without code

Open LinkedIn Jobs Scraper, click Try for free and run:

{ "keywords": ["data analyst"], "locations": ["United States"], "datePosted": "week", "maxJobsPerSearch": 100 }

Or set your filters on linkedin.com/jobs, copy the URL and paste it into searchUrls.

2. Many keywords × many locations, cleaned

{
  "keywords": ["python developer", "data engineer"],
  "locations": ["London", "Berlin", "Remote"],
  "workplaceTypes": ["remote", "hybrid"],
  "experienceLevels": ["mid-senior"],
  "titleMustExclude": ["intern", "principal"],
  "companyExclude": ["Staffing", "Recruitment"],
  "maxJobsPerSearch": 200
}

Each keyword × location is a separate search; duplicates across searches are removed automatically.

3. From Python: daily "new jobs" report

pip install "apify-client>=3"

from apify_client import ApifyClient
import csv

client = ApifyClient("YOUR_APIFY_TOKEN")  # Apify Console → Settings → API & Integrations

run = client.actor("rt-solutions/linkedin-jobs-scraper").call(run_input={
    "keywords": ["machine learning engineer"],
    "locations": ["United States"],
    "postedWithinHours": 24,     # only jobs posted in the last 24 hours
    "sortBy": "date",
    "maxJobsPerSearch": 300,
})

jobs = list(client.dataset(run.default_dataset_id).iterate_items())
jobs = [j for j in jobs if j.get("jobId")]          # skip warning rows
# "Be among the first 25 applicants" is returned as 25
few_applicants = [j for j in jobs if (j.get("applicants") or 999) <= 25]

with open("new_ml_jobs.csv", "w", newline="") as fh:
    w = csv.writer(fh)
    w.writerow(["title", "company", "location", "salary", "applicants", "applyType", "url"])
    for j in few_applicants:
        w.writerow([j["title"], j["companyName"], j["location"], j.get("salary"), j.get("applicants"), j.get("applyType"), j["jobUrl"]])

print(f"{len(jobs)} new jobs, {len(few_applicants)} with 25 applicants or fewer")

A job item (shortened):

{
  "jobId": "4460087902", "title": "Senior Software Engineer, Payments",
  "companyName": "Whatnot", "location": "San Francisco, CA",
  "postedDate": "2026-10-28", "applicants": 200,
  "salary": "$155,000.00/yr - $160,000.00/yr",
  "seniorityLevel": "Mid-Senior level", "employmentType": "Full-time",
  "applyType": "external", "jobUrl": "https://www.linkedin.com/jobs/view/…"
}

4. Put it on autopilot

In Apify Console: Schedules → run daily with postedWithinHours: 24, and connect the Google Sheets integration so new jobs land in a sheet every morning.

Limits worth knowing

LinkedIn shows up to about 1,000 jobs per search. For more, use more keywords or more precise locations (cities instead of countries).
minSalary is a LinkedIn filter that only works for US searches.

Cost

$0.30 per 1,000 jobs with full details (cheaper on paid plans). A daily run of ~300 jobs costs about $0.09.

Need company data for the employers you find (website, size, headcount, HQ)? See LinkedIn Company Scraper.

How to track Google Flights prices with Python (no API key)

gafari radouane — Mon, 02 Nov 2026 18:58:33 +0000

Google Flights has no public API. If you want to watch fares for a route every day, compare dozens of dates, or feed flight prices into a spreadsheet, you need a scraper. In this tutorial we'll collect Google Flights prices for many routes and dates in one run, export them to CSV, and schedule a daily price check.

Disclosure: I built the Apify Actor used below (Google Flights Scraper). It costs $0.20 per 1,000 flights and runs on Apify's free plan.

What you get

For every flight shown by Google Flights:

total price and currency, rank on the results page
airlines, departure / arrival time, duration, number of stops and layovers
every leg: flight number, aircraft, legroom, codeshares
CO2 emissions vs. the typical value for the route
a direct Google Flights link

1. Try it without code

Open Google Flights Scraper and click Try for free.
Fill in origins, destinations and dates:

{ "origins": ["JFK"], "destinations": ["LHR"], "departureDates": ["2026-12-03"] }

Click Start. After a few seconds you can export the results as CSV, Excel or JSON.

2. Many routes × many dates in one run

You can give several airports and a date range (start..end). Every origin × destination × date becomes a search:

{
  "origins": ["JFK", "EWR"],
  "destinations": ["LHR", "CDG"],
  "departureDates": ["2026-12-01..2026-12-14"],
  "tripLengthDays": 7,
  "cabinClass": "economy",
  "stops": "nonstop",
  "currency": "USD"
}

tripLengthDays: 7 turns every search into a round trip returning 7 days later. Use returnDate instead for a fixed return day.

3. Run it from Python

pip install "apify-client>=3"

from apify_client import ApifyClient
import csv

client = ApifyClient("YOUR_APIFY_TOKEN")  # Apify Console → Settings → API & Integrations

run = client.actor("rt-solutions/google-flights-scraper").call(run_input={
    "origins": ["JFK"],
    "destinations": ["LHR"],
    "departureDates": ["2026-12-01..2026-12-07"],
    "stops": "nonstop",
})

rows = list(client.dataset(run.default_dataset_id).iterate_items())

# Cheapest flight per departure date
best = {}
for f in rows:
    d = f["departureDate"]
    if f.get("price") and (d not in best or f["price"] < best[d]["price"]):
        best[d] = f

with open("cheapest_jfk_lhr.csv", "w", newline="") as fh:
    w = csv.writer(fh)
    w.writerow(["date", "price", "currency", "airlines", "departure", "durationMinutes"])
    for d, f in sorted(best.items()):
        w.writerow([d, f["price"], f["currency"], "/".join(f["airlines"]), f["departure"], f["durationMinutes"]])

print(f"{len(rows)} flights, cheapest per day saved to cheapest_jfk_lhr.csv")

A flight item looks like this (shortened):

{
  "origin": "JFK", "destination": "LHR", "departureDate": "2026-12-03",
  "price": 711, "currency": "EUR", "airlines": ["American"],
  "departure": "2026-12-03T18:25", "arrival": "2026-12-04T06:20",
  "durationMinutes": 415, "stops": 0, "co2Kg": 421,
  "legs": [{ "flightNumber": "AA 100", "aircraft": "Boeing 777", "legroom": "31 inches" }]
}

4. Track fares every day

In Apify Console open the Actor → Schedules → run it daily with the same input. Each run is a new dataset, so you can append the cheapest price per date to a Google Sheet (Apify has a Google Sheets integration) or to your database and watch prices move.

Cost

$0.20 per 1,000 flights on the free plan ($0.08–0.10 on paid plans)
searches that return no flights are not charged
a 7-day × 1-route check (~150 flights) costs about $0.03

Notes

price is the total for all passengers; for round trips it is the round-trip price shown with the outbound flight, as on Google Flights.
No Google account or API key is needed.

That's it — a few lines of Python and you have a daily fare tracker. Questions or a missing field? Open an issue on the Actor page.

How to check if ChatGPT, Gemini and Perplexity recommend your brand (and track it weekly)

gafari radouane — Sun, 25 Oct 2026 17:12:21 +0000

More and more buyers skip Google and ask an AI assistant: "What is the best CRM for a small business?". If the answer doesn't name you, you are invisible to that buyer. This is what people now call GEO (generative engine optimization) or AEO.

Here is a simple, repeatable way to measure it, with real numbers from a test I ran in October 2026.

1. Write the questions your buyers really ask

Not "What is Pipedrive?" (you'll always be mentioned), but category questions:

What is the best CRM for a small business?
Which CRM is best for a sales team of 10 people?
Pipedrive vs HubSpot for startups?

10–30 questions per market is a good start. Do it per country and language: answers differ.

2. Ask every engine, and record four things

For each answer:

Is your brand mentioned?
At which position? Being 4th in a list of 5 is not the same as being 1st.
Which competitors are named, and in what order.
Which websites are cited as sources. ChatGPT, Gemini and Perplexity link the pages they used. Those review sites and comparison articles are where you need to appear.

3. Turn it into three numbers

Visibility %: share of answers that mention you.
Average position when you are mentioned.
Share of voice: your mentions ÷ all brand mentions (you + competitors).

A real test

Brand: Pipedrive. Competitors: HubSpot, Salesforce, Zoho, monday.com, Freshsales.
2 questions × 4 engines (ChatGPT with web search, Gemini, Perplexity, Claude) = 8 answers:

Metric	Result
Visibility	100% (8/8 answers)
Average position	2.1
First place	3 of 8
Pipedrive's own site cited as a source	3 of 8
Share of voice	HubSpot 20.5%, Pipedrive 20.5%, Zoho 20.5%, Salesforce 15.4%
Most cited domains	salesforce.com, techradar.com, pipedrive.com, zite.com…

Takeaways: Pipedrive is always there but comes first in only 3 answers out of 8, HubSpot and Zoho are named just as often, and third-party review sites like techradar.com shape the answers. That's the to-do list.

One answer proves little: AI answers change between runs. Track the averages weekly.

Automating it

By hand, this is fine for 5–10 questions. I got tired of copy-pasting, so I built a tool on Apify (disclosure: I'm the developer): AI Brand Visibility Tracker. You give it your brand, your competitors and your questions. It asks ChatGPT, Gemini, Perplexity and Claude and returns one row per answer, plus the summary above. It costs about $0.03 per answer, with no subscription, and you can schedule it weekly.

Example input:

{
  "brand": "Pipedrive",
  "brandDomain": "pipedrive.com",
  "competitors": ["HubSpot", "Salesforce", "Zoho"],
  "prompts": ["What is the best CRM for a small business?"],
  "engines": ["chatgpt", "gemini", "perplexity"]
}

Whatever tool you use, the method is what matters: fixed questions, several engines, averages over time, and work on the sources the AIs cite.

How to Scrape Gulf Job Listings with Salaries (UAE, Saudi Arabia, Qatar) in 2026

gafari radouane — Tue, 20 Oct 2026 09:28:52 +0000

The Gulf job market moves fast. Recruitment agencies in India, Pakistan, the Philippines and Egypt, HR analysts and job-board founders all need the same thing: fresh job listings from Dubai, Abu Dhabi, Riyadh, Doha or Kuwait, with the salary and the candidate profile the employer wants. NaukriGulf is one of the biggest job boards in the region — but there is no public API.

In this guide I'll show you how to export NaukriGulf jobs with salary (local currency and USD), experience, skills, full description and desired nationality into Excel, Google Sheets or your own code, without writing a scraper.

Disclosure: I built the tool used in this guide and publish it on the Apify Store. It has a free trial, so you can test everything below without paying.

What data you get per job

Field	Example
Title, company, location	Accounts & Sales Coordinator — Dubai, UAE
Salary (local currency)	AED 2,000 – 3,500 per month
Salary (USD)	$501 – $1,000 per month
Experience	1 – 5 years
Industry, function, contract	Retail, Accounts / Audit, Full-time, On-site
Desired candidate	Education, nationalities, gender
Skills	Tally ERP, invoicing, accounts receivable…
Full description	the complete job ad (text + HTML)
Posted date, vacancies, URL	✔

Only data the site shows publicly is returned: if an employer hides the salary or the recruiter e-mail, the tool leaves it empty.

Option 1 — No code

Open the NaukriGulf Scraper and click Try for free.
Enter keywords, one per line: accountant, civil engineer, nurse, sales manager…
Optionally a location: Dubai, Abu Dhabi, Riyadh, Jeddah, Doha, Kuwait, Saudi Arabia… (a typo like "Dubaii" is corrected automatically).
Set Posted within (days) to 1 if you only want today's new jobs.
Click Start, then Export → Excel or CSV.

Example of one result:

{
  "title": "Accounts & Sales Coordinator",
  "company": "HARDWARE POINT BUILDING MATERIALS TRADING LLC",
  "location": "Dubai - United Arab Emirates (UAE)",
  "salaryMin": 2000,
  "salaryMax": 3500,
  "salaryCurrency": "AED",
  "salaryUsdMonthlyMin": 501,
  "salaryUsdMonthlyMax": 1000,
  "experienceMin": 1,
  "experienceMax": 5,
  "industry": "Retail",
  "employmentType": "Full-time",
  "education": "Bachelor of Commerce, Bachelor of Business Administration",
  "nationalities": ["Indian", "Pakistani", "Nepali", "Bangladeshi"]
}

Tip for agencies: schedule a daily run with Posted within = 1 for your key roles and get every new Gulf job in a spreadsheet each morning.

Option 2 — From your own code

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('rt-solutions/naukrigulf-scraper').call({
  keywords: ['civil engineer', 'quantity surveyor'],
  location: 'Riyadh',
  postedWithinDays: 7,
  maxJobsPerSearch: 200,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const withSalary = items.filter((j) => j.salaryUsdMonthlyMin);
console.log(`${items.length} jobs, ${withSalary.length} with a public salary`);

Python is the same with pip install apify-client.

What it costs

$1 per 1,000 jobs, full details included. A daily run of 200 jobs costs about $6 per month. Apify's free plan includes monthly credits to test.

Use cases

Recruitment agencies: daily feed of new Gulf openings matching your candidates.
Salary benchmarks: average salary by role and city (AED, SAR, QAR, KWD…).
Market research: which industries and nationalities employers are hiring.
Job boards & newsletters: aggregate Gulf jobs for your audience.

Honest notes

About half of employers hide the salary; it is returned only when shown publicly.
Listings are mostly in English — use English keywords for the best results.
The tool collects only public job listings, visible to any visitor without login.

Working on Southeast Asian or Australian jobs too? I also built scrapers for JobStreet & JobsDB (6 countries) and SEEK (Australia & NZ).

How to Get Coles and Woolworths Grocery Prices Without Coding (2026)

gafari radouane — Tue, 20 Oct 2026 08:59:24 +0000

Australia's grocery prices have become a national topic: specials, "Down Down" campaigns, half-price deals and the endless Coles vs Woolworths debate. If you run a price-comparison app, track FMCG brands, write about cost of living or just want a spreadsheet of what milk costs this week, you need the data — and neither supermarket offers a public API.

In this guide I'll show you how to get Coles and Woolworths product prices, specials, unit prices and barcodes into Excel, Google Sheets or your own code in a few minutes, without writing a scraper yourself.

Disclosure: I built the two tools used in this guide and publish them on the Apify Store. Both have a free trial, so you can test everything below without paying.

What data you can get

For every product, from both supermarkets, in the same columns so you can compare them directly:

Field	Example
Name, brand, size	Nescafe Gold Original Premium Instant Coffee 100g
Price / was price	$7.25 / $14.50
On special, savings ($ and %)	yes, $7.25 (50%)
Offer text	"1/2 Price", "Pick any 2 for $44"
Unit price	$7.25 / 100g
Stock, category	in stock, Instant Coffee
Barcode (EAN/GTIN)	Woolworths, and Coles with product details
Image and product URL	✔

Option 1 — Compare Coles and Woolworths in one run (no code)

Open the Woolworths & Coles Scraper and click Try for free.
In Products to search, type what you want, one per line: milk, coffee pods, nappies, tim tam…
Keep both Woolworths and Coles selected. Tick Specials only if you only want this week's deals.
Click Start. A few seconds later you get a table like this:

{
  "store": "Coles",
  "name": "Nescafe Gold Original Premium Instant Coffee 100g",
  "price": 7.25,
  "wasPrice": 14.5,
  "isOnSpecial": true,
  "savingsPercent": 50,
  "offer": "1/2 Price",
  "unitPriceText": "$7.25/ 100g",
  "category": "Instant Coffee",
  "url": "https://www.coles.com.au/product/nescafe-gold-original-premium-instant-coffee-100g-8049750"
}

Click Export → Excel or CSV. Sort by unitPriceText or savingsPercent and you have your comparison.

Tip: schedule the run every Wednesday (when the new specials start) to build a weekly price history automatically.

Option 2 — The whole Coles catalogue, with ingredients and nutrition

If you need more than a keyword search — a full category, the entire weekly specials catalogue (~8,000 deals) or all ~30,000 products — use the Coles Scraper:

Mode "All specials": every product on special this week, de-duplicated.
Mode "Categories": e.g. Dairy, Eggs & Fridge or a sub-category URL like https://www.coles.com.au/browse/dairy-eggs-fridge/milk.
Mode "Full catalogue": every category in one run.
Include product details: barcode, country of origin ("Made in Australia from at least 90% Australian ingredients"), ingredients, allergens and the nutrition table per 100 g and per serving.

That last option is what nutrition apps, product databases and FMCG analysts usually can't find anywhere else.

Option 3 — Use it from your own code

Both tools have an API. With the official client (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('rt-solutions/au-supermarkets-scraper').call({
  keywords: ['milk', 'coffee'],
  stores: ['woolworths', 'coles'],
  specialsOnly: false,
  maxResultsPerSearch: 100,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const cheapest = items
  .filter((p) => p.unitPrice)
  .sort((a, b) => a.unitPrice - b.unitPrice)
  .slice(0, 10);
console.table(cheapest.map((p) => ({ store: p.store, name: p.name, price: p.price, unit: p.unitPriceText })));

Python works the same way with pip install apify-client.

What does it cost?

Woolworths & Coles Scraper: $1 per 1,000 products.
Coles Scraper: $1 per 1,000 products, + $1 per 1,000 if you add product details (barcode, ingredients, nutrition).

A weekly snapshot of 2,000 products is about $2. Apify's free plan includes monthly credits, enough to test properly.

A few honest notes

Prices are the standard online prices shown without a selected store.
Woolworths publishes barcodes for many products; Coles only on the product page (use Include product details).
Coles "Everyday" low prices and collectables are not counted as specials — only real deals are.
The tools only collect public information shown to any visitor, without logging in.

Ideas of what to build with it

A weekly "best deals" newsletter or Telegram bot
A price-history dashboard for your favourite products
Brand monitoring: how often is your product on special at each supermarket?
A nutrition comparison of house brands vs national brands

If you build something with this data, I'd love to see it — and if a field you need is missing, tell me in the comments.