Extracting Track BPM and Key Data Past Beatport Search Limits

#programming #beginners #devops #webscraping

Building an automated ingestion pipeline for electronic music metadata often leads to Beatport. Music analytics platforms, DJ curation tools, and record labels rely on its catalog for critical track attributes: BPM, Camelot keys, catalog numbers, ISRC codes, and release dates.

However, querying Beatport directly presents engineering obstacles. The search endpoint behaves inconsistently, capping results at 10 items per page and looping result sets after one or two pages. Furthermore, the storefront utilizes anonymous token rotation that can reject requests with HTTP 401 status codes across certain IP ranges.

Using the Beatport Scraper actor bypasses these storefront quirks by wrapping the internal api.beatport.com/v4/catalog API, managing visitor session tokens, and offering targeted ingestion modes.

The Search Pagination Bottleneck

When sending free-text queries to Beatport's search endpoint via mode: "search", Beatport's API caps track results to 10 items per page regardless of request parameters. After the first one or two pages, the underlying API recycles results, making deep pagination via keyword search impossible. A free-text search typically tops out at 10 to 20 unique tracks before deduplication terminates the run.

If your pipeline requires systematic catalog coverage across genres or labels, keyword searching is the wrong ingestion path.

Instead, the catalog endpoints for genres, labels, releases, and charts do not suffer from this 10-item cap. Running a genre or release extraction allows consistent retrieval of full pages of tracks. For structured pipelines, shifting from mode: "search" to mode: "browseByGenre", mode: "browseReleases", or mode: "topCharts" provides stable, high-volume ingestion.

{
  "mode": "browseByGenre",
  "genre": "tech-house",
  "minBpm": 124,
  "maxBpm": 128,
  "key": "A Minor",
  "maxItems": 250
}

Structure of the Extracted Track Dataset

The actor emits clean records directly to the dataset, omitting empty fields rather than passing null values. A single track record includes identity, musical analysis, and media URLs:

{
  "id": 3731110,
  "title": "The Kid",
  "mixName": "Original Mix",
  "artists": ["Example Artist"],
  "primaryArtist": "Example Artist",
  "remixers": [],
  "label": "Example Records",
  "releaseId": 54321,
  "releaseName": "The Kid EP",
  "genre": {
    "id": 11,
    "name": "Tech House",
    "slug": "tech-house"
  },
  "bpm": 126,
  "key": "8A",
  "duration": "6:42",
  "durationMs": 402000,
  "price": {
    "value": 2.49,
    "currency": "USD",
    "display": "$2.49"
  },
  "isrc": "GB-AAA-24-00001",
  "catalogNumber": "EX001",
  "releaseDate": "2024-02-15",
  "previewUrl": "https://geo-samples.beatport.com/track/preview/3731110.mp3",
  "sampleStartMs": 0,
  "sampleEndMs": 60000,
  "artworkUrl": "https://geo-media.beatport.com/image_size/500x500/12345.jpg",
  "recordType": "track",
  "scrapedAt": "2025-02-23T12:00:00.000Z"
}

The payload delivers both standard musical keys and Camelot notations (such as 8A), making the output immediately usable in harmonic mixing engines or catalog tagging systems without custom mapping tables.

Filtering Parameters for Targeted Ingestion

To prevent downstream systems from discarding irrelevant records, the actor processes filter parameters before writing items to the dataset:

minBpm and maxBpm: Constrains the tempo range (valid between 40 and 200 BPM).
key: Restricts results to any of the 24 standard musical keys.
releasedAfter: Accepts an ISO date string (YYYY-MM-DD) to exclude historical back-catalogs during recurring monitoring runs.
containsKeyword: Performs case-insensitive matching on track titles.
includeReleaseTracks / includeChartTracks: When fetching parent objects in topCharts or browseReleases modes, setting these booleans to true extracts the nested individual track objects into the dataset alongside the parent record.

Running an Ingestion Workflow via Python

The Apify API client allows straightforward integration of the scraper into an ETL pipeline. The following script configures a release-monitoring extraction for a specific electronic music label.

import os
from apify_client import ApifyClient

client = ApifyClient(os.getenv("APIFY_TOKEN"))

run_input = {
    "mode": "browseReleases",
    "labelId": 20670,
    "releaseSort": "newest",
    "includeReleaseTracks": True,
    "maxItems": 100
}

# Run the actor and wait for completion
run = client.actor("crawlerbros/beatport-scraper").call(run_input=run_input)

# Fetch emitted dataset items
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

tracks = [item for item in dataset_items if item.get("recordType") == "track"]
print(f"Extracted {len(tracks)} tracks from label catalog.")

Step-by-Step Setup

Locate Target IDs: If targeting a specific artist or label, open their page on beatport.com. The ID is the integer at the end of the URL (e.g., beatport.com/artist/amy-dabbs/406088 yields artistId: 406088).
Select Ingestion Mode: Choose browseByGenre, browseByLabel, browseReleases, or topCharts for bulk ingestion. Reserve search strictly for targeted track name lookups.
Set Filters: Define releasedAfter to pull only new tracks since your last ingestion timestamp, and configure minBpm/maxBpm if your target taxonomy requires strict tempo boundaries.
Configure Proxy: If your runs encounter environment-level 401 blocks from Beatport's API, enable proxyConfiguration in the input payload to route catalog and token-minting requests through datacenter proxies.
Consume Dataset: Ingest the dataset items by streaming records from the default dataset into your analytical warehouse or music library database.

System Boundaries

This tool does not provide full-length audio downloads; it extracts only metadata and the public 30- to 60-second MP3 preview URLs hosted on Beatport's CDN. Additionally, because Beatport's API occasionally omits specific metadata fields (such as missing ISRC values on older releases), downstream schemas must handle missing fields rather than assuming every track record contains identical keys.

If you want to reproduce this, the Actor is Beatport Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

DEV Community