So this is it, this is the moment that all of the overwhelming pieces of information I have supposedly obtained over the past month will come together, culminating in an “a-ha” moment, and I will have all the knowledge required to build whatever application I so chose. And then I started….
My first attempt was to scrape the Oklahoma City Thunder (my favorite basketball team) website, utilize the information obtained to present the user with a overview of the team roster, and give the user several options for getting more details regarding any player of their choosing.
I created the scraper for the first level, wrote methods for obtaining the information and presenting it to the user in a visually pleasing manner, and was well on my way to creating my application. However, upon creating the scraper for the second layer of the website, for some reason, I could not obtain the information I wanted. I tried every class name I could find, every div, every header, nothing. So, after more hours than I care to admit, I went to my cohort lead and asked him for advice. He told me that he was sorry, but the website I had chosen was inadequate for the project as the javascript was not going to permit me to get the information needed when the attempted scrape took place.
Great, 48 hours down the drain. However, the practice of creating the initial application did give me hope that I would be able to duplicate the process on another website in a fraction of the time. I then found my next idea on metacritic.com, presenting the user with the top 71 Nintendo 64 games sorted by critic reviews. So, I began toiling away, drafting my new scraper with methods to get all of the information I needed to present the user with. Again.
After getting this idea and building a simple scraper, I went to test it out so that I didn’t waste another day on a website that doesn’t work. And then, these words came across my terminal:
“403 Forbidden”.
This was in reference to this line of code:
html = (open(@base_url + "/browse/games/score/metascore/all/n64/filtered"))
Now, I can google all I want, but how in the world am I getting an error on my first line of code? This is the simplest part of the project, and I can’t even get the first line of code to work. I googled for a significant amount of time trying to figure out what this error message was referring to. I had my open-uri gem installed, I had Nokogiri gem as well, so why am I getting this error.
After many failed attempts to rectify this issue, I came across a thread on a Japanese website that suggested I simply change the user-agent. I figured I had tried a million other solutions, why not try one more. So I changed my scraper to this:
html = (open(@base_url + "/browse/games/score/metascore/all/n64/filtered", 'User-Agent' => ‘chrome’))
All of a sudden, everything was working. I did a bit more searching to try to figure out why this would work when all I did was change the User-Agent, but to no avail, and I certainly did not have the luxury of time at this point. So in the future, this is certainly something that I will want to look into further, but for now, it shall be one of the great mysteries of the universe to me (unless a lovely reader of this blog would like to share some insight with me in the comments below).
The next several portions of my code involved scraping the first and second layers of the metacritic website, establishing the has one / has many relationship required in the project (all games have one user review score, but there can be many games within one user review score), and writing some simple language to make the experience more interactive and fun for the user.
However, I then ran across an issue sorting my reviews into separate hashes for the menu so that the user can select whether they want any of the following four options:
puts "1. View all 71 games"
puts "2. View games with a user rating of 8 or greater"
puts "3. View games with a user rating under 8"
puts "4. Exit”
I had all of the information I wanted, I even pre-wrote the methods for options two and three to pull that information from the arrays I wanted to put the individual reviews in. I just could not figure out how to get the information into those arrays.
Luckily for me, I have a great cohort and spent several hours with several people in the cohort who walked me through the process and eventually led me to this simple code solution:
def self.sort_uscore
var = @@all.sort_by{|score| [score.number]}.reverse
var.each do |score|
if score.number >= 8
@@all_eight_plus << score
else
@@all_under_eight << score
end
end
end
All of a sudden everything was working, I had my three arrays (one for all games, one for games with a review score 8 or higher, and one for games with a review score under 8), my methods pulling from them were working perfectly. The only thing left I had for completing my project was to write a few lines for the user interaction, make sure that my app wouldn’t break with the wrong input, and done!
There are a few aspects of this application which I would have liked to have made more intricate, and may even do so moving forward. I would like additional looping features to be introduced for the user, and I would also like to make the styling more intricate, introducing colors and moving features.
However, for now, despite the simplistic nature of my application, it works, its mine, and I am proud to have built it.
Top comments (0)