Building a Fast Multi-Source Music Discovery Pipeline with FastAPI

Nidhal Dalhoumi — Thu, 10 Sep 2026 10:31:46 +0000

When building TheExperience, I wanted the application to do more than search for albums.

The idea was to take an album search and turn it into a complete interactive experience starting from finding the album, retrieving reliable metadata, resolving its artwork, extracting its visual identity and finally using the information to drive the frontend.

The challenge was that no single API provides everything I needed.

The backend integrates Last.fm, MusicBrainz, and the Cover Art Archive, each with different APIs, identifiers, response times and failure modes.

The interesting engineering problem therefore became:

How do you combine several external APIs without making every search slow, wasteful, or fragile?

The solution was to separate discovery from enrichment, reuse identifiers whenever possible, execute independent requests concurrently, and treat external services as unreliable dependencies rather than extensions of the application itself.

The architecture

The backend is built with FastAPI and uses asynchronous HTTP clients to communicate with the external services.

The overall flow looks like this:

User searches for an album
        ↓
GET /albums/search
        ↓
Last.fm search
        ↓
Local ranking + result limiting
        ↓
Lightweight album results
        ↓
User selects an album
        ↓
GET /albums/details
        ↓
 ┌───────────────────────┐
 │                       │
Last.fm              MusicBrainz
popularity            enrichment
 │                       │
 └───────────┬───────────┘
             ↓
        Normalized Album
             ↓
        Cover Art Archive
             ↓
       Complete album data
             ↓
       React visual layer

One of the first decisions I made was to avoid doing all of this work during search.

1. Search is not enrichment

Album autocomplete needs to feel lightweight.

If every keystroke triggered Last.fm, MusicBrainz, and Cover Art requests, the application would perform unnecessary work before the user had even selected an album.

Instead, /albums/search only performs the lightweight Last.fm search.

lastfm_candidates = await search_albums_lastfm(
    query,
)

albums = [
    Album(
        id=album.get("id", ""),
        title=album.get("title", ""),
        artist=album.get("artist", ""),
        listeners=album.get("listeners", 0),
        playcount=album.get("playcount", 0),
    )
    for album in lastfm_candidates
    if album.get("title")
    and album.get("artist")
]

The backend receives a larger set of candidates from Last.fm, ranks them locally according to title and artist relevance, and returns a limited result set.

The deeper metadata pipeline only starts when the user selects an album.

This creates two distinct stages:

Discovery
---------
Last.fm search
      ↓
Local ranking
      ↓
Limited Album[]


Enrichment
----------
Last.fm popularity ─────┐
                        ├──→ Normalized Album
MusicBrainz ────────────┘
                        ↓
                    Cover Art

This separation reduced unnecessary external requests and made the search endpoint much simpler.

2. Reusing identifiers instead of searching again

One of the more interesting problems was matching data between Last.fm and MusicBrainz.

Last.fm can sometimes provide a MusicBrainz ID (MBID) for an album.

When that identifier exists, there is no reason to search MusicBrainz again using the album title and artist.

if musicbrainz_id:
    musicbrainz_task = asyncio.create_task(
        find_album_by_id(
            musicbrainz_id,
            title,
            artist,
        )
    )
else:
    musicbrainz_task = asyncio.create_task(
        find_album(
            title,
            artist,
        )
    )

Instead, I can resolve the album directly:

Last.fm result
      │
      ├── MusicBrainz ID exists
      │        ↓
      │   Direct release lookup
      │        ↓
      │   Release + release-group
      │
      └── No MBID
               ↓
        Release-group search
               ↓
          Match candidate
               ↓
          Release lookup

The direct path requires one MusicBrainz request.

The fallback path may require two: one release-group search followed by a release lookup.

This is a small optimization, but it illustrates an important principle when working with external APIs:

If an upstream service has already given you a stable identifier, reuse it instead of performing another search.

When an MBID is available, the lookup can be constructed directly:

url = f"{MUSICBRAINZ_RELEASE_URL}/{musicbrainz_id}"

The MusicBrainz response is then transformed into the application's Album model.

3. Concurrency — but only where dependencies allow it

Once an album has been selected, some pieces of information can be retrieved independently.

Last.fm popularity and MusicBrainz metadata do not depend on each other, so running them sequentially would unnecessarily increase latency.

FastAPI's asynchronous model makes it possible to execute these operations concurrently with asyncio.gather().

popularity_task = asyncio.create_task(
    get_album_popularity(
        title,
        artist,
    )
)

if musicbrainz_id:
    musicbrainz_task = asyncio.create_task(
        find_album_by_id(
            musicbrainz_id,
            title,
            artist,
        )
    )
else:
    musicbrainz_task = asyncio.create_task(
        find_album(
            title,
            artist,
        )
    )

(
    popularity_result,
    musicbrainz_album,
) = await asyncio.gather(
    popularity_task,
    musicbrainz_task,
)

The Cover Art request is different.

The Cover Art Archive lookup needs a specific MusicBrainz release ID, so it cannot start until MusicBrainz resolution has produced that identifier.

The pipeline is therefore not simply "make everything concurrent."

It is:

Last.fm popularity ──────┐
                         ├── concurrent
MusicBrainz resolution ──┘
                         ↓
                    release_id
                         ↓
                     Cover Art

cover_url = await get_cover_url(
    musicbrainz_album.release_id
    or musicbrainz_album.id
)

This distinction matters because asynchronous code is not automatically faster. The goal is to identify which operations are independent and parallelize only those.

4. External APIs need their own failure architecture

External services can fail for reasons that have nothing to do with the application.

A timeout, temporary service outage, invalid response, or network error should not necessarily turn the entire album experience into a server error.

For MusicBrainz, I added a small asynchronous request layer responsible for rate limiting, timeouts, retry handling, and converting recoverable failures into empty results.

MusicBrainz requests are coordinated with a module-level asynchronous lock and a minimum one-second interval between scheduled requests within the process.

The service also retries a 503 Service Unavailable response once after a two-second delay.

Other failures, such as timeouts or request errors, are converted into empty results so the application can continue where possible.

This is deliberately provider-specific rather than pretending that every API behaves the same way.

For example, Last.fm operations have different failure semantics, while Cover Art is treated as non-critical because missing artwork should not prevent an album's metadata from being displayed.

That leads to a useful design principle:

The importance of an external dependency should determine how its failure is handled.

Album metadata is important.

Album artwork is useful, but optional.

Those two failures should not have identical consequences.

5. Normalizing data from different systems

Another challenge was that the three services don't share the same data model.

Last.fm provides search and popularity information.

MusicBrainz provides structured music metadata and identifiers.

The Cover Art Archive provides artwork associated with MusicBrainz releases.

Rather than passing provider-specific response objects throughout the application, the backend converts the results into a common Album model.

MusicBrainz also distinguishes between a release group and an individual release.

The application keeps both concepts:

Album
├── id          → release-group identity
├── release_id  → specific release
├── title
├── artist
└── year

The release-group represents the album-level identity, while the specific release ID is useful when retrieving artwork.

This normalization keeps the frontend independent from the details of each external API.

6. Measuring instead of guessing

Another useful addition during development was instrumentation.

I used Python's time.perf_counter() around provider requests and larger sections of the enrichment pipeline.

This made it possible to see where time was actually being spent instead of treating the entire backend request as one opaque operation.

For example, during local testing, cached Last.fm searches were effectively instantaneous, while uncached searches were typically below roughly 1.5 seconds in my development environment.

Album enrichment was more variable because it depends heavily on external provider latency, particularly artwork retrieval.

These numbers are development observations, not production benchmarks.

The more important result was having visibility into individual stages of the pipeline.

Once external calls are measured independently, optimization becomes much more concrete.

7. What I would improve next

The current implementation works well for the project's scale, but there are several areas I would improve before treating the architecture as production-grade.

First, the rate limiter is process-local. A distributed deployment would need a shared mechanism if multiple backend instances were making MusicBrainz requests.

Second, retry handling could be expanded to account for rate-limit responses such as 429, with more sophisticated backoff behavior.

Third, the in-memory caches could eventually be replaced or supplemented with a shared cache if the application needed to run across multiple instances.

Finally, album matching could become more sophisticated for ambiguous metadata instead of relying on normalized exact title and artist matching in the MusicBrainz fallback path.

These are not problems that need to be solved simply because they exist. They are the next engineering trade-offs that would become relevant as scale and reliability requirements increase.

What I learned

The hardest part of integrating several APIs was not learning how to send HTTP requests.

It was deciding when to make those requests, which identifiers to reuse, which operations could run concurrently, and what should happen when a dependency fails.

The resulting architecture is relatively simple:

Keep search lightweight.
Enrich only after the user selects an album.
Reuse MusicBrainz IDs whenever available.
Run independent I/O concurrently.
Respect provider-specific constraints.
Normalize external data into an application-owned model.
Treat optional dependencies as optional.
Measure external latency instead of guessing where the bottleneck is.

For me, this project became a useful exercise in moving from simply integrating APIs to thinking about the behavior of a system around those APIs.

And that is ultimately what I found most interesting about building TheExperience: the visual experience is what you see, but the engineering challenge underneath it is making several independent systems behave like one coherent application.

Project: TheExperience — an interactive music discovery and visual album experience built with React, Angular, TypeScript, FastAPI, and external music metadata services.

Status: Active development.

DEV Community: Nidhal Dalhoumi