Four ways I tried to ingest data for my League of Legends hobby project

Intro

Hello fellow devs! This is my first post on dev.to. My name is Rinat, I'm a senior full stack developer and I'm developing professionally and for fun for more than 11 years now.

This spring I started my hobby project Riftmind - a web application that analyzes raw data from videogame League of Legends and extracts useful information. Most of you probably heard of that game - it's one of the most popular multiplayer games with very big esports community (by the way Worlds 2026 starts today!). There's a public Riot API that serves a lot of data that we can use - which champions were picked, which champions were banned, who dealt the most damage, who were dying the most, etc. Currently Riftmind calculates winrates by role, rank and region and shows synergies, counterpicks, analyzes draft.

Every 2 weeks Riot releases new patch with changes to the game - from small to very significant. So for our calculations we constantly need large amount of fresh data, desirable balanced by each rank and this brings me to the point I wanted to talk about in this post. The thing is - in order to find the useful effect - we need a lot of data, so for example to say that champion A is statistically confirmed to be stronger than champion B we need to have at least data from 1000 matches between them (and even then the margin of error is about ±3%), while the effect itself is small - for most matchups it's 1-2%, and only the strongest counterpicks reach 5-8%. Riot API has rate limits for small projects like mine - about 100 calls per 2 minutes for each server. And all the match data goes through only 4 regional servers, so it's roughly 200 match API calls per minute for the whole world (calls about players and ranks go to 14 other game servers, like EUW or NA, each with its own limit). And the worst part - there's no endpoint to fetch matches from certain rank directly, we can only get matches of a certain player and we can also get players from certain rank.

So my goal was to ingest as many matches as possible by making as less API calls as possible.

Architecture

As it is my hobby project that was never intended to bring any money I wanted to minimize my monthly spending on the infrastructure. That's why I split the project into 3 applications - Ingestion, Processing and Published Web App. Ingestion is running 24/7 constantly gathering data from RiotAPI and writing it raw to its own database. Processing is executed by schedule or on demand and it calculates aggregates and copies them to the production database at the end of each run. The published app is running on Render and only has small database that only contains aggregated data. Ingestion and processing I was running at my local PC first (the gathered data could reach about 150GB) and then I moved it to the cheap VPS.

Attempt 1 - crawling the players

So the first idea was simple as the idea but turned out to be very complex at execution - I manually went to the website that had some info about players in each rank, manually selected ones that had many games and used them as seeds - my ingestion worker then fetched all their games, saved info and then moved to the other players from these games (each game always has 10 players). It worked and first month or two the ingestion used this approach. But with time some problems became obvious:

• I couldn't reliably balance games between ranks. When we fetch the game - we don't know the ranks of all players, we only know the rank of the player whose game we're fetching. And the ranks in the same game can vary quite significantly. And if on the lower ranks it worked fine - on the higher ranks our queue would constantly drift to lower ranks. And making additional calls for each player would be very inefficient for our goal
• It wasn't clear how to pick who we crawl next - each game adds 10 candidates, how do we maximize the "useful" calls? I had to introduce some scoring for players - considering how many games do we play.

Eventually the logic of crawling grew more and more complex while introducing little to none improvements.

By the time I gave up, the crawler was 1,790 lines of code with 17 settings to tune, and the queue of players had to be capped at 250,000 so it wouldn't grow forever.

Attempt 2 - random sampling from the ladder

This time I tried different approach - Riot API has endpoint to fetch players from any rank - the ladder, a list of all players in a rank, about 200 per page. So we don't need to crawl, we already have list of players we can fetch directly. So algorithm was simpler:

1. Fetch players from the least represented rank in our data
2. For every player - fetch their 20 recent match ids, download games from the current patch and stop at the first older-patch game
3. Put these players on 48-hour cooldown, to wait until they play more games

This approach was way simpler - twice less code, games are ingesting, crawler logic was deleted at the same day.

The problem - most of the players we fetch didn't have any new games. So our ingestion was basically looking at the ladder, taking a random player and asked "did you play any games?" and answer was "no" most of the times.

In numbers: over the last full patch with this approach (16.19, two weeks) a random player gave us only 0.66 new matches on average, and only about 38 of every 100 match API calls brought a new match. That's about 2,900 new matches per hour.

Attempt 3 - catching live games

Before the next ladder idea, I tried one more thing. Riot API has a Spectator endpoint - you can ask if a player is in a game right now, and if they are, you get the game id. When the game ends, you download it with a single match API call. And Spectator calls go to those 14 game servers, whose limits I barely used. So the plan was to use these "free" calls to find games, and spend the expensive match calls only on downloading them.

I didn't want to break the working ingestion, so I ran it for an hour and a half as a side experiment - it only recorded the games it found, without downloading anything.

The games it found were perfect - every game I checked was a ranked game from the current patch. But there were too few of them: 5,340 calls found only 156 players in a game, which gave 62 unique games. Even if I used the whole spare limit of all 14 servers, it would find under 500 games per hour, while random sampling was already ingesting about 1,700 per hour at that time. Random players are rarely in a game right at the moment you ask. So I dropped the idea, but it showed me what I actually needed - a cheap way to know which players played recently.

Attempt 4 - Ladder, but better

This time instead of asking random player from the ladder - I requested 200 players from ladder in a single call (ladder calls go to the same 14 game servers with spare limits, so they're basically free), waited for 30 minutes, requested again and checked which player's game number (wins + losses) changed. And only fetched games from players who have played some actual games.

Before switching, I ran it for 7 hours as a side experiment, the same way as Spectator. The ladder turned out to be very precise: if a player's wins + losses grew by 1, they had on average 0.95 new games for us, by 2 - 1.95, by 3 - 2.91. So I switched to it the same day. Now the ingestion re-reads 20 ladder pages for every division every 2 hours (every 30 minutes was too heavy for my small VPS database), fetches games of the players with the most new games first, and only falls back to random players when the ladder runs out of active ones.

The results for the first week:

• 1.7 new matches per checked player, against 0.66 for random players
• about 62 of every 100 match API calls bring a new match, instead of about 38
• about 7,100 new matches per hour instead of 2,900 (together with a couple of other fixes I made the same day)
• the current patch got more matches in 9 days (1.23 million) than the previous one in two weeks (0.98 million)

Takeaways

• Write down what you actually need before picking an algorithm. I needed "many recent games in every rank", and the ladder already had players sorted by rank. Crawling was a solution looking for a problem.
• Find your bottleneck and measure everything against it. For me it was match API calls, so I compared every approach by how many new matches 100 calls bring.
• Use what's cheap to save what's expensive. Ladder calls were almost free, and they turned out to be the signal that tells me where to spend the expensive match calls.
• Try new ideas on the side first. Both Spectator and ladder ran next to the working ingestion and wrote to their own tables, so a bad idea couldn't break anything.

If you play League - check out Riftmind. And if you solved a similar problem differently - I'd love to hear about it in the comments!