Structuring US News Institutional Data for Pipeline Analysis

Building analytical pipelines around institutional benchmark data often hits a wall when aggregating metrics across higher education, healthcare, and K-12 systems. US News & World Report aggregates key performance indicators—such as undergraduate acceptance rates, hospital specialty scores, and private school tuition figures—behind distinct UI structures.

Manually collecting these figures or writing brittle custom parsers creates pipeline maintenance overhead. Using the US News Rankings Scraper allows data engineers to extract structured institutional metrics across multiple categories directly into automated workflows.

Standardizing Institutional Data Across Sectors

When ingesting higher education or healthcare data, schemas frequently diverge. US News standardizes these evaluations into distinct scoring systems. The scraper handles five primary extraction modes:

1. universities: Captures composite scores, enrollment figures, tuition costs, test score ranges, and peer-assessment reputation scores across specific program categories.
2. hospitals: Extracts national specialty rankings, composite scores (0–100), and contact details per medical discipline.
3. highSchools: Pulls state rankings, graduation rates, and College Readiness Index metrics for public secondary institutions.
4. privateHighSchools: Scrapes directory-level details like yearly tuition, student-teacher ratios, and sports programs.
5. gradSchools: Extracts program rankings across specific subjects such as law, medicine, and business.

Each extraction mode yields a consistent JSON structure containing a recordType tag, allowing ingestion jobs to route incoming payloads to the correct database tables without additional preprocessing.

Configuring Pipeline Input Parameters

The Actor's execution relies on programmatic input configuration. By defining explicit boundaries through schema parameters, you avoid fetching irrelevant records and keep run runtimes lean.

Key parameters include:

• mode (String, Required): Defines the targeted directory type (universities, hospitals, highSchools, privateHighSchools, or gradSchools).
• rankingType (String): Specifies the subset list when mode is set to universities (e.g., national-universities, best-value-schools, engineering).
• specialty (String): Filters medical rankings when mode is set to hospitals (e.g., cancer, cardiology, neurology).
• subject (String): Selects the field of study when mode is set to gradSchools (e.g., business, law).
• state (String): Limits results using a two-letter US state postal abbreviation (e.g., CA, TX, NY).
• maxItems (Integer): Controls the upper limit of records returned (1 to 2000).

For example, to extract public high school performance indicators limited to California, the JSON configuration uses:

{
  "mode": "highSchools",
  "state": "CA",
  "maxItems": 100
}

To extract top national business graduate programs, the configuration shifts to subject-based parameters:

{
  "mode": "gradSchools",
  "subject": "business",
  "maxItems": 50
}

Integrating Scraped Outputs into Python Workflows

The dataset items returned by the Actor present typed fields. A university record contains explicit numerical data for statistical analysis, including acceptanceRate as a float, tuition as an integer, and programReputationScores as an embedded object containing peer-assessment ratings on a 1.0–5.0 scale.

Here is an automated Python implementation using the Apify API client to fetch national university metrics and parse them directly into a data pipeline:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "mode": "universities",
    "rankingType": "national-universities",
    "maxItems": 50
}

# Run the Actor and wait for completion
run = client.actor("crawlerbros/usnews-rankings-scraper").call(run_input=run_input)

# Fetch dataset items from the run's default dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    name = item.get("name")
    rank = item.get("rank")
    tuition = item.get("tuition")
    acceptance_rate = item.get("acceptanceRate")

    # Access embedded program scores where available
    reputation = item.get("programReputationScores", {})
    cs_score = reputation.get("computerScience")

    print(f"#{rank} {name} - Tuition: ${tuition} | Acceptance: {acceptance_rate}% | CS Score: {cs_score}")

For hospital datasets (mode="hospitals"), output objects return medical discipline context:

{
  "rank": 1,
  "name": "Example Hospital",
  "city": "Boston",
  "state": "MA",
  "specialty": "cancer",
  "score": 100.0,
  "recordType": "hospital",
  "scrapedAt": "2026-03-30T12:00:00.000Z"
}

Understanding the Flat Pay-Per-Event Cost Model

This Actor utilizes a PAY_PER_EVENT pricing structure rather than charging variable usage fees based on duration. Every billable execution consists of predictable event charges:

1. Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run, charged once when the container initializes.
2. Result (apify-default-dataset-item): $0.005 per extracted record returned to the default dataset.

Volume discounts automatically reduce the per-result pricing depending on account tier structure:

• FREE: $0.005 per result
• BRONZE: $0.00433 per result
• SILVER: $0.00367 per result
• GOLD / PLATINUM / DIAMOND: $0.003 per result

For example, running an extraction job on a standard 1 GB container that yields 100 university records on the FREE tier incurs:

• $0.005 (Actor Start fee)
• 100 × $0.005 = $0.50 (Result charges)
• Total job cost: $0.505

Because charges scale directly with dataset output count rather than execution time, unexpected delays caused by network latency do not inflate run costs.

Schema Differences and Extraction Limits

When designing ingestion target schemas, note that field availability varies based on US News reporting patterns and the target institutional category.

Public high schools return structured metrics such as graduationRate and collegeReadinessIndex. In contrast, private high schools (mode="privateHighSchools") are unranked by US News due to the absence of mandatory state test reporting. Consequently, private school records omit ranking scores entirely and instead return directory data such as yearlyTuition, studentTeacherRatio, and sportsOffered. Furthermore, private school fields like yearlyTuition and apCoursesOffered are only populated if the institution has claimed and published its official profile; unclaimed profiles omit these keys entirely rather than returning empty strings or zeros.

Pipeline ingestion steps must account for these conditional keys using soft dictionary gets or optional schema validation rules to prevent null pointer exceptions during batch processing.

---

US News Rankings Scraper is the Actor behind these examples. If a selector in your own version breaks, compare your output against the fields listed in its README first.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-09-19. Check the Actor page for the current rates.