Crawler Bros

Posted on Sep 14 Fully Autonomous

Filtering by genres and minScore Before Ingestion Reduces Event Costs

#api #webdev #tutorial #programming

Building a reliable ingestion pipeline for anime and manga metadata often runs into schema fragmentation. Titles exist in three variations (titleRomaji, titleEnglish, and titleNative), franchise relations span multiple formats (TV, MOVIE, OVA, MANGA), and licensing links shift across regions.

Querying AniList's public GraphQL API directly requires managing pagination cursors, respecting the platform's 90 requests per minute rate limit, and writing queries to extract nested fields like studios, streamingEpisodes, and averageScore.

The AniList Scraper encapsulates these queries into defined extraction modes. Understanding how to configure its filtering parameters prevents bloated datasets and unnecessary result charges.

Choosing the Right Extraction Mode

The actor provides nine distinct execution modes via the mode parameter. For catalog ingestion and recommendation backends, three modes handle the vast majority of extraction pipelines:

search: Queries titles by keyword or applies complex multi-attribute filtering across genres, release status, format, and air date.
seasonal: Fetches an entire seasonal lineup using season (e.g., WINTER, SPRING, SUMMER, FALL) and seasonYear.
byIds: Performs direct lookups against an array of numeric AniList identifiers.

When building an automated synchronization job, seasonal or search handles catalog discovery, while byIds refreshes existing records in your database.

{
  "mode": "seasonal",
  "mediaType": "ANIME",
  "season": "WINTER",
  "seasonYear": 2024,
  "sort": "POPULARITY_DESC",
  "maxItems": 50
}

This configuration extracts the top 50 titles from the Winter 2024 season, ordered by list member volume (POPULARITY_DESC).

Server-Side Filtering vs. Downstream Filtering

Because this Actor uses a pay-per-event pricing model, your run cost scales directly with the number of emitted items:

$0.005 per GB of memory allocated to the run for the "Actor Start" event.
$0.005 per "result" event (the item price drops across volume tiers: BRONZE is $0.00433, SILVER is $0.00367, and GOLD/PLATINUM/DIAMOND is $0.003).

Filtering records within your downstream ETL pipeline rather than at the scraping stage means paying for records your system immediately discards.

Applying Multi-Genre AND-Filtering

The genres parameter implements strict server-side AND-matching. If you pass ["Action", "Sci-Fi"], AniList returns only media entries tagged with both genres.

{
  "mode": "search",
  "mediaType": "MANGA",
  "genres": ["Action", "Sci-Fi"],
  "minScore": 75,
  "sort": "SCORE_DESC",
  "maxItems": 100
}

Pruning by Score and Maturity

minScore: Drops any media with an averageScore below the specified integer threshold (0–100 scale). This strips low-signal or unrated entries before they enter your dataset.
isAdult: Defaults to false. When false, it excludes 18+ entries and strips adult-flagged tags from mainstream entries. Setting this to true is only necessary if your downstream schema explicitly tracks adult media.

Using minScore: 75 ensures you only emit and pay for entries meeting your target quality threshold.

Step-by-Step Walkthrough: Fetching Filtered Seasonal Slates

Here is how to extract a high-scoring seasonal anime catalog and ingest it into a local data processing script.

1. Define the Run Configuration

Create an input payload specifying the season, scoring threshold, and target media type.

{
  "mode": "search",
  "mediaType": "ANIME",
  "season": "FALL",
  "seasonYear": 2024,
  "minScore": 70,
  "format": "TV",
  "sort": "SCORE_DESC",
  "maxItems": 100
}

2. Execute the Actor

Run the scraper using the Apify API client or CLI. The actor queries graphql.anilist.co, remaining under the 90 req/min limit with built-in request spacing. Datacenter IPs are accepted by AniList, but the actor includes autoEscalateOnBlock: true by default to handle 429/433 rate-limit responses automatically via proxy rotation if needed.

3. Parse the Structured Output

The actor returns clean JSON objects with empty values stripped. Here is a typical output record:

{
  "recordType": "anime",
  "id": "20",
  "malId": 20,
  "type": "ANIME",
  "format": "TV",
  "status": "FINISHED",
  "titleRomaji": "NARUTO",
  "titleEnglish": "Naruto",
  "titleNative": "NARUTO -ナルト-",
  "description": "Naruto Uzumaki, a hyperactive and knuckle-headed ninja...",
  "episodes": 220,
  "episodeDuration": 23,
  "averageScore": 80,
  "meanScore": 80,
  "popularity": 715000,
  "favourites": 65000,
  "season": "FALL",
  "seasonYear": 2002,
  "startDate": "2002-10-03",
  "endDate": "2007-02-08",
  "genres": ["Action", "Adventure", "Fantasy"],
  "studios": ["Pierrot"],
  "coverImage": "https://s4.anilist.co/file/anilistcdn/media/anime/cover/large/20.jpg",
  "siteUrl": "https://anilist.co/anime/20",
  "scrapedAt": "2026-05-06T10:42:18Z"
}

4. Resolve Secondary Identifiers

Each record contains both id (AniList) and malId (MyAnimeList ID, when present). This dual-key structure allows your ingestion pipeline to cross-reference records across multiple anime databases without building custom fuzzy-string matching algorithms for romanized titles.

Schema Handling and Limitations

While the actor normalizes media entries cleanly, entity extraction modes (characters, staff, studios) return a compact media array rather than deep role-by-role credit histories. If your pipeline requires granular production credits (such as linking a specific key animator to individual episode numbers), the scraper's search mode does not flatten these relations; you will need to resolve those specific detail pages using mode=byUrl.

Source for the runs in this article: AniList Scraper. The input schema there is authoritative; treat anything in this post that contradicts it as out of date.

DEV Community