First Project : Build a CLI app with External Data

My first application ever before.Half success, half failure. But It was a precious project where I could learn a lot.

Brief introduction of the application:

The CLI application I built for the first project at Flatiron is a very simple app that can list the menu of Dunkin' donut's espresso and coffee. By scraping the list of menu from the website, user can see more details about the menu chosen.

How I scrape the data?

require 'nokogiri'
require 'open-uri'
URL = open("https://www.webpage")
doc = Nokogiri::HTML(open(URL))
doc.css("class").
puts doc

By this way, I could get messy data that is difficult to read. The problem has begun here. I misjudged the data and I believed I could scrape the text I want. Without analyzing the data in details, I started to code my project, enjoyed CLI time for several days. The problem occurred at last. There were data not "scrapable". All 'Ingredients', 'calories' and 'Allergens' text data were unable to scrape. I struggle with scraping the data for several days because it seemed like just normal text data. What i learned a few days later was that it happened because the web page used 'JavaScript' for the certain data: which I needed.

I have not learnt JavaScript yet so I had to minimize and simplify my project finish up on time. As I mentioned before, there was actually a bigger plan than what this application I coded shows now. I should have scraped the data and see the details of it very carefully before I start to code.

For this project, scraping external data might give more sources to build various applications. However, since the way 'scraping' has certain risk, for instance, each website may have different designs and use different languages with special styles, it will affect the application and someday the app will be no longer useful.

So, next time? Definitely APIs rather than scraping web data.

What I learned from the first project

1. creating/ editing ruby project files using gems.
2. Coding CLI class method and testing by bin/filename and binding.pry
3. The use of web scraping and what risk it has. Also why programmers are more likely to use APIs
4. How to use github

git status => if I want to check the status
git add . = > add file
git commit -am "any commit message" => to save my changes to the local repository.
-a
Includes all currently changed files in this commit
-m
Sets the commit's message
git push => to upload local repository content to a my github repo