Dealing with multi-level nested API data and creating a seeding file with it

Dealing with multi-level nested data from an API call can sometimes be overwhelming. We are going to go through an example from a project that I am currently working on and how it was broken down into smaller, more manageable pieces. The end goal is to create a seeding file to populate a data table.

Evaluating the API Response

The first step we need to do is look at the response that our GET request will receive from the API. We will look at what data we get, and what data we want to extract to use in our app. For this example we are using the ESPN API, and our goal is to get a list of all the current NFL players. We also want to get some information about each player. In the NFL there are 32 teams, each team having 53 active players. That alone adds up to 1,696 players. Then add in the players on each teams practice squad, and players on injured reserve and the number climbs well over 2,000 players. Now consider that we want to collect info on each player such as: their team name, team location, position they play, their playing status, and a couple other things and its easy to see how this can become overwhelming very quickly.

First things first, lets look at the API return and see what we are dealing with. The end point we are going to be using is:

/apis/site/v2/sports/football/nfl/teams/team_id/roster

When using this endpoint, the team_id will be replaced with the specific team that you want the roster info for. Here is the return when you use the team_id of 23.

We see that we have received an object with 6 keys: athletes, coach, season, status, team, and timestamp. So we see the team key with the info of the team and location of the team that the players on this roster play for. But the players are in another key that is split up into 3 array objects, one for offense, one for defense, and one for special teams. Lets look at the array items in the athlete key.

Now we see that each item in the under the athletes key contains an array of players. Lets go further and look in the first items array.

And now inside one of the players.

Lets not forget about the team info for this group of players.

So now we can see all the info we are looking for, but this is for just one team, there are still 31 other teams!

At this point its easy to feel overwhelmed but if we start to break it down into smaller steps it begins to feel much more manageable.

Breaking it Down

There are many ways to approach a task like this. I find the easiest is to start at top level and work down through. As we move down through each level, take note if there is any data on that level that we want to capture. If there is, then capture it and then continue moving down through the levels.

The API endpoint returns all the info for a single team so we will have to repeat the call for each team. This tells us we will need some kind of loop to make a call for each one. Because we know that there are a total of 32 teams, we can simply use the #times method to make a GET request for each team. Looking into the team_id we see that they start at 1 and increment up by one. We can declare a variable and set it to 1, then interpolate that variable into the API endpoint for the team_id, then just increment the variable by one at the end of each loop.

i = 0
32.times do
    res = RestClient.get "https://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/#{i}/roster"
    player_hash = JSON.parse(res)

    end
    i = i + 1
end

We now have a loop that will return an object for each team and store that object in player_hash. We can pull the team name, team location, and even the link to the team logo image from the player_hash. We can now check those data points off the list of what we want to collect and look at the rest of the keys on this level to see if there is anything else to collect.

We are not going to be using any of the data from the timestamp key, the status key, the season key, or the coach key. The only other key of interest is the athletes key. Looking at the athletes key, we can see that it is an array of 3 objects. We can use the#each method to iterate through them and continue down through the levels looking for data to collect.

player_hash["athletes"].each do |group|
    group["items"]

end

This takes us through the athletes array, which contains an items array, and position key: value pair. We are not going to be using the general position information, so we know that there is no data to collect yet and we can continue on. We now iterate through the items array which is an array containing the player objects. These objects contain all the player information that we want to collect and we can check all the data points we are looking for off our list. We now have access to everything we need to use the #create method and populate the players table in our database.

group["items"].each do |p|
    Player.create(
        name: p["fullName"],
        position: p["position"]["name"],
        number: p["jersey"],
        status: p["status"]["name"],
        is_drafted: false,
        team_name: player_hash["team"]["name"],
        team_location: player_hash["team"]["location"],
        team_logo: player_hash["team"]["logo"],
        image: p["headshot"]
    )
end      

As a last little touch we can give some feed back while the seeding process is taking place with some puts. Below is the completed seeds.rb file.

If you look at the code you will see the added line:

puts "Seeding #{player_hash["team"]["name"]} players ..."

If you are still unclear with what exactly is happening, this line might help to visualize what is going on. Lets see the seeding in action.

We have now sucessfully seeded our players table with all 2,393 NFL players!

A note for those that were looking closely, if you noticed that the .times was set to 34 instead of 32, good eye. This was done to compensate for the team_id that the API uses. In the id list, it skips numbers 31 and 32 so the team_id go from 1 to 30, then skips to 33 and 34.