DEV Community

Can Yılmaz
Can Yılmaz

Posted on • Originally published at apify.com

Why Internshala Internship & Jobs data is more interesting than you would think

#webscraping #apify #jobs #dataengineering

On the surface, Internshala Internship & Jobs sounds like the kind of dataset you would file under "boring infrastructure data" -- the sort of thing that lives in a corner of a warehouse and gets queried twice a quarter. After spending a bit of time actually looking at it, I have changed my mind. Here is why.

What is in it

Internshala Internship & Jobs Scraper Scrape Internshala.com Listings to JSON/CSV Scrape internship and fresher job listings from Internshala.com, India's #1 career platform trusted by 400K+ companies with 200K+ active listings. Each record carries a fairly rich set of fields:

  • listingId -- listing id
  • listingType -- listing type
  • url -- url
  • title -- title
  • company -- company
  • companyUrl -- company url
  • location -- location
  • isRemote -- is remote
  • stipend -- stipend
  • stipendMin -- stipend min
  • stipendMax -- stipend max
  • duration -- duration
  • startDate -- start date
  • applyBy -- apply by
  • openings -- openings
  • applicants -- applicants
  • skills -- skills
  • perks -- perks
  • description -- description
  • isPartTime -- is part time
  • hasJobOffer -- has job offer
  • postedAt -- posted at
  • category -- category
  • scrapedAt -- scraped at

The interesting bit is the combination. Individually, none of these fields is exotic. Together, they describe an entity precisely enough that you can do real analytics on it -- segmentation, trend analysis, even simple anomaly detection -- without needing a second data source.

Two records from a sample run 

{
  "listingId": "3150094",
  "listingType": "internships",
  "url": "https://internshala.com/internship/detail/work-from-home-web-development-internship-at-zdminds1778824887",
  "title": "Web Development",
  "company": "Zdminds",
  "companyUrl": "https://www.linkedin.com/company/zdmindsindia/?viewAsMember=true",
  "location": "Work from home",
  "isRemote": true,
  "stipend": "₹ 10,000 - 20,000 /month",
  "stipendMin": 10000
}
Enter fullscreen mode Exit fullscreen mode 
{
  "listingId": "3150096",
  "listingType": "internships",
  "url": "https://internshala.com/internship/detail/work-from-home-python-development-internship-at-zdminds1778824954",
  "title": "Python Development",
  "company": "Zdminds",
  "companyUrl": "https://www.linkedin.com/company/zdmindsindia/?viewAsMember=true",
  "location": "Work from home",
  "isRemote": true,
  "stipend": "₹ 10,000 - 20,000 /month",
  "stipendMin": 10000
}
Enter fullscreen mode Exit fullscreen mode

When you look at a couple of records side by side the analytical surface area opens up. The categorical fields invite grouping. The numeric fields invite ranking and distribution analysis. The timestamps invite time-series breakdowns. The text fields invite NLP.

Three things you can actually do with this

  1. Build a leaderboard. Pick a numeric field, group by a categorical field, sort. Trivial in SQL or Pandas, surprisingly useful for tracking hiring trends, building talent pipelines, salary benchmarking and competitive recruiting intelligence.
  2. Detect shifts over time. Snapshot the dataset daily, compute simple deltas between snapshots, alert on anything that moves more than a sensible threshold.
  3. Cluster the long tail. The categorical fields probably have a power-law distribution. The long tail is often where the interesting outliers live -- the new entrants, the niche players, the anomalies.

Why it is not just "another scrape"

The reason this dataset is more interesting than typical scrape output: the source has organic structure. The fields are not invented by the scraper, they reflect how the underlying domain organises itself. That gives the dataset a kind of semantic coherence that synthetic or heavily-derived datasets lack.

Caveats

  • Sample sizes from a one-off run will not let you do anything statistically serious -- you want a longitudinal feed.
  • Some optional fields are sparsely populated; check density before relying on them.
  • The source can change. Treat any production pipeline as something that will need maintenance.

How I would prove the analytical thesis

If I were trying to justify investing engineering time in this dataset for a real project, the path would be: pull a one-week recurring sample to get past the snapshot bias, run the three analytical patterns above on the larger pull, and judge whether the conclusions hold up. If you can get a single non-obvious insight out of that exercise, the dataset is worth keeping. If everything you find is something you already knew, it probably is not -- find a different feed. That bar sounds harsh, but it saves you from a portfolio of datasets that nobody actually queries.

For live, customizable extractions of this data, the actor that produced the dataset shown above is published on the Apify Store: logiover/internshala-scraper. It supports JSON, CSV and Excel exports and runs on a schedule.

Top comments (0)