So a week ago I read a post here about Ruby would kill Python in the future. There are lots of debates, as always, on the comment section of that post. Someone mentions that it does not matter who would kill whom, every language or tool is suitable for something; You have to pick right tool for right work.
I think that is the right answer, however, it is important to consider that what market (or industry) thinks about tools and languages. Do they want a Ruby programmer as much as they want a Python developer?
To find out, I created a small Ruby project which does these tasks:
- Crawl Glass-door pages for predefined cities for software job ads
- Store pages
- Create anagram to count number of occurrences of each keyword (such as
pngfiles to visualize how industry is in need for each skill.
Here is the output for total ten cities around the world. Remember, this is the number of ads that contains each keyword. For example if ad contains
Java then it increase the number of total ads that contains this specific keyword.
And here is the number of technologies mentions:
The answer is easy:
Java. But if you are looking for script languages, then the answer is
Ruby. However there are interesting findings when you compare results for individual cities.
See the rest of these reports here. It includes chart for
Tallinn and etc.
The structure is easy. First there is a configuration file which you can define the cities, keywords, categories and etc.
Then by running
client.rb it will get first URL from the configuration file, crawls the web page, saves all URL specific parameter for each listing page, gets the second page and repeats it until the last page.
After that another class will crawl the web site again. This time it download the whole ad page and save it on disk.
The third class then create an anagram of all predefined keywords and scans every document that we saved in previous step. We save the results as a
yml file then.
Here is the sample of output:
At the end with the help of Gruff Gem we generate images from YAML files.
- This could be helpful if you are investigating your next career path or your next language to learn. Nothing serious more than that.
- The project is pretty much configurable. Just update the
configfile: add what city you want, the first URL, what keywords you are looking for and what categories. Then run it (wait minutes to get all the data) and check the output on result folder. link to project
- Have fun!