DEV Community: denvermullets

Tips on how to get project ideas when trying out new tech

denvermullets — Sun, 08 Nov 2020 21:24:09 +0000

Often times when I'm looking to learn new tech or learn a new technique, I like to start a new project and build something practical that'll keep me motivated to finish it. Here are a few tips I've collected that help me get started.

Keep a list

Sounds easy, right? Just keep a list! So simple!

Really, what I do is anytime I use an app and feel like it could be better, I write down the app idea of what I'm using. This could be a lot of different things depending on what the app is. Sometimes I just want to understand how things work (I love trying to figure stuff out without libraries if I can) and write that idea down. It could be as simple as just something that I feel like should exist and I can't find it easily.

A big part of why I got into programming was because I wanted to make tools to make life easier for others, as well as myself. So when I feel a tool can be improved upon, it goes on the list. I just rotate thru the list and the better I get at programming, the more feasible the 'harder' ideas become.

Here's a handful of ideas that I think would be fun to learn how to build:

Reddit clone
- This is kinda self explanatory but I always felt like the Reddit experience could be improved. It'd be a fun way to flex some CSS knowledge
Coffee brewing timer w/recipes
- I use an Aeropress to make coffee and feel like it's kinda hard to easily find other coffee setups and test them out, while keeping time (sometimes it's precise)
Google Drive clone
- I've bookmarked a couple of clones to see how they build it out but I'm always interested in decoupling my own dependency on external tools by building my own
Destiny 2 apps
- It's easy to look at things you spend a lot of time doing, for me it's video games, and try to make tools to make your life easier.

There's honestly a whole lot of room for apps to be improved upon or simplified that your list could always be expanding. Since I've been keeping track of stuff my list has grown to over 50 random ideas. So far I've built a handful of them and it's incredibly satisfying to build tools that you'll end up using.

How to figure out what skills are being hired for right now, programmatically

denvermullets — Mon, 26 Oct 2020 02:55:15 +0000

Ever curious what skills are really in demand on job boards? We're programmers, why not build a scraper to help figure that out?

Here's the type of output we want to get:

{
  "javascript": 648,
  "react": 442,
  "java": 382,
  "agile": 345,
  "cloud": 309,
  "css": 305,
  "python": 301,
  "apis": 243,
  "sql": 241,
}

Prerequisites

To get started you'll want to follow this guide which will walk you thru getting your environment setup and some basic scraping working.

Getting the job URL's

Following the same idea that the guide shows us, here's how we'll lay out our code. This code will get us our individual job listings to pull the words out. I opted to split this into 2 different files just in case there were errors. This will put the URL's of each job listing into a JSON file that we'll use in the next file.

I put in a few sample search result URL's, feel free to change the search terms or add as many as you'd like. The one I ended up running was about 80 URL's with a few different search terms.

# indeed_url.rb
require 'kimurai'
require "selenium-webdriver"

class Indeed < Kimurai::Base
  @name = 'indeed_scrape'
  @start_urls = [
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY', 
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=10', 
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=20',
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=30',
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=40',
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=50',
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=60',
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=70',
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=80',
    'https://www.indeed.com/jobs?q=full%20stack%20developer&l=New%20York%2C%20NY&start=90',
  ]
  @engine = :selenium_chrome
  @@jobs = []

  def scrape_page
    # Update response to current response after interaction with a browser
    doc = browser.current_response
    # browser.save_screenshot
    sleep 2

    while (doc.css('div.jobsearch-SerpJobCard')[0]) do
      # this loop goes thru the however many job listings are on the page
      doc = browser.current_response
      # get first job listing
      single_job = doc.css('div.jobsearch-SerpJobCard')[0]
      # get job information
      job_url = single_job.css('a.jobtitle').attribute('href')
      job_url = 'https://indeed.com' + job_url

      puts ' ===== '
      puts job_url
      puts " ===== "

      @@jobs << job_url if !@@jobs.include?(job_url)
      doc.css('div.jobsearch-SerpJobCard')[0].remove
 browser.execute_script("document.querySelector('div.jobsearch-SerpJobCard').remove()") 
      sleep 0.1

    end
  end

  def parse(response, url:, data: {})
    scrape_page
    File.open("tmp/indeed_jobs_urls.json","w") do |f|
      f.write(JSON.pretty_generate(@@jobs))
    end

    @@jobs
  end
end

Indeed.crawl!

Getting the words

Now that we have the URL's to parse let's go through each individual posting and then see which words pop up the most.

# indeed_posting.rb
require 'kimurai'
require 'json'

# this loads each url from the JSON file and pulls the description,
# removes all punctuation and converts it all to lowercase
# then, throw each word into a hash for JSON

class JobScraper < Kimurai::Base
  @name = 'indeed_scrape'
  @start_urls = JSON.parse(File.read("tmp/indeed_jobs_urls.json"))
  @engine = :selenium_chrome

  @@word_count = {}

  def scrape_page
    sleep 2
    doc = browser.current_response
    job_desc = doc.css('div.jobsearch-jobDescriptionText').text.gsub(/[[:punct:]]/, '').downcase
    job_array = job_desc.split(' ')
    job_array.each do |word|
      @@word_count[word] ? @@word_count[word] += 1 : @@word_count[word] = 1
    end

    puts @@word_count

  end

  def parse(response, url:, data: {})
    scrape_page
    sorted_hash = @@word_count.sort_by {|a,b| -b}
    sorted_hashery = sorted_hash.to_h  

    File.open("tmp/new_sorted_skills.json","w") do |f|
      f.write(JSON.pretty_generate(sorted_hashery))
    end

  end
end

JobScraper.crawl!
puts 'done scraping'

The only problem I have with this is that you'll get a JSON file that you'll have to eventually trim down and remove all the non tech words. It's pretty easy to see where the block of tech words pop up and only takes a few minutes.

I'm very open to improvement on this though, so let me know if there's something that I should tweak!

How to make gifs for your Github using Giphy - Is there a better way?

denvermullets — Mon, 19 Oct 2020 00:01:54 +0000

If you're trying to spruce up your Github readme's you're probably looking to add a gif or two showing how your app works or the UI navigation. I've been trying to find an easy way to make gifs without having to resort to my usual way of screen recording in Quicktime and editing it down in After Effects and then compressing it in Photoshop.

I think I found a good way to do if you're using a Mac. Giphy has made a Capture App that's really super useful and really easy to use! Just select the screensize and hit record.

The video clip shows at the bottom of the app and once you click on it it'll bring you to a customization menu where you can resize it to trim down on file size or skip frames.

If you're not uploading the final gif in your actual Github repo, then it'll need to be under 5mb in total size. Depending on how complex your readme is, you might need to split it into other gifs.

I think this is a pretty great and easy way to make gifs and if you've got Photoshop you can extend longer gifs by compressing them further with the options they provide. You can see my gifs in action here in my job tracker app

Do you know of any easier or better ways to make screencap gifs? I'd love to revamp the workflow.

Some notes on web scraping with Ruby featuring Kimurai and Selenium

denvermullets — Mon, 12 Oct 2020 01:44:43 +0000

I had previously written about scraping some stats from basketball reference and when I went back to do it again my links didn't work, meaning I would have to redo some of the work. I didn't particularly love my last exploration with Nokogiri and scraping so I took this as an opportunity to learn and expand my knowledge.

I'm not going to write this as a walkthru since this blog will really walk you thru it and really just posting this to show different ways you can get the information you need.

I feel like finding elements using CSS leads to a more stable approach vs hardcoding the XPath. As long as the class name or id is the same, it won't matter if the site ends up nesting the element and breaking your code.

One thing that I spent a lot of time trying to troubleshoot was my variables were coming up empty despite me doing it all seemingly correct. In the end I realized that if the site hadn't finished loading. I'm assuming there's a safer way to do this vs just using sleep 2 or something.

My Code

require 'kimurai'
require 'json'

class TequilaScraper < Kimurai::Base
  @name = 'tqdb_scrap'
  @start_urls = ['https://tequilamatchmaker.com/tequilas/2325-fortaleza-blanco']
  @engine = :selenium_chrome

  @@tequilas = []

  def scrape_page
    sleep 2
    doc = browser.current_response
    tequila = doc.css('div.product-actions')

    teq_name = tequila.css('h1[itemprop="name"]').text.gsub(/\n/, "")
    teq_type = tequila.css('div.product-type a').text.gsub(/\n/, "")
    teq_rating_p = tequila.css('ul.product-list__item__ratings li')[0].text.gsub(/\D/, '').gsub(/\n/, "")
    teq_rating_c = tequila.css('ul.product-list__item__ratings li')[1].text.gsub(/\D/, '').gsub(/\n/, "")
    teq_price_check = tequila.css('div.commerce-price-container div span')[1]

    if teq_price_check
      teq_price = tequila.css('div.commerce-price-container div span')[1].text.gsub(/\n/, "")
    else
      teq_price = 'n/a'
    end

    doc_mid = doc.css('div.container')
    teq_image = doc_mid.css('img.product-image').attr('src')

    teq_nom = doc_mid.css('div.production-details_product table tbody tr')[0].css('td a').text.gsub(/\n/, "")
    doc_mid.search('span.sr-only').each do |spans|
      # remove search result spans since it's just a comma
      spans.remove
    end 
    teq_agave = doc_mid.css('div.production-details_product table tbody tr')[1].css('td').text.gsub(/\n/, "")
    teq_agave_region = doc_mid.css('div.production-details_product table tbody tr')[2].css('td').text.gsub(/\n/, "")
    teq_region = doc_mid.css('div.production-details_product table tbody tr')[3].css('td').text.gsub(/\n/, "")
    teq_cooking = doc_mid.css('div.production-details_product table tbody tr')[4].css('td').text.gsub(/\n/, "")
    teq_extraction = doc_mid.css('div.production-details_product table tbody tr')[5].css('td').text.gsub(/\n/, "")
    teq_water = doc_mid.css('div.production-details_product table tbody tr')[6].css('td').text.gsub(/\n/, "")
    teq_fermentation = doc_mid.css('div.production-details_product table tbody tr')[7].css('td').text.gsub(/\n/, "")
    teq_distillation = doc_mid.css('div.production-details_product table tbody tr')[8].css('td').text.gsub(/\n/, "")
    teq_still = doc_mid.css('div.production-details_product table tbody tr')[9].css('td').text.gsub(/\n/, "")
    teq_aging = doc_mid.css('div.production-details_product table tbody tr')[10].css('td').text.gsub(/\n/, "")
    teq_abv = doc_mid.css('div.production-details_product table tbody tr')[11].css('td').text.gsub(/\n/, "")
    teq_other = doc_mid.css('div.production-details_product table tbody tr')[12].css('td').text.gsub(/\n/, "")

    tequila = {name: teq_name, type: teq_type, rating_p: teq_rating_p, rating_c: teq_rating_c,
      price: teq_price, image_url: teq_image, nom: teq_nom, agave: teq_agave, agave_region: teq_agave_region,
      region: teq_region, cooking: teq_cooking, extraction: teq_extraction, water: teq_water, fermentation: teq_fermentation,
      distillation: teq_distillation, still: teq_still, aging: teq_aging, abv: teq_abv, other: teq_other}

    @@tequilas << tequila #if !@@tequilas.include?(tequila)
  end

  def parse(response, url:, data: {})

    scrape_page

    File.open("tequila.json","w") do |f|
      f.write(JSON.pretty_generate(@@tequilas))
    end

    @@tequilas
  end
end

TequilaScraper.crawl!
puts 'done scraping'

Initially I was going to get mock data from a React site but opted for this tequila database just to test easily. Using Selenium is a bit overkill for this, but all in all it worked really well. I think next time I'm going to have to try to dig into some Python or JS scraping.

What I worked on in September and the idea of Learning in Public

denvermullets — Mon, 05 Oct 2020 01:21:56 +0000

A little bit ago I picked up a book written by swyx (The Coding Career Handbook - it's awesome, seriously scope it out) and in one of the chapters he talks about the idea of learning in public. Basically the idea is to document everything you do and don't be afraid to be wrong as long as you're trying to be correct. People notice when others are genuinely learning and are more willing to lend a hand if you're stuck or don't understand something.

There's a lot to this idea and definitely read his post about it. It's a scary thing starting out in a new field and trying to act like you belong and know what you're doing.

But you know what? It's ok if you don't know everything, you're going to be constantly learning. I remember how difficult some things in Ruby were back in my 2nd week of bootcamp, but I got over those things because I practiced and learned. I'd probably laugh at the things that were roadblocks then and I'll laugh at the things that are roadblocks now.

I need to do better about discussing the things I'm learning and this is a start to that.

September

September started off pretty busy, I officially started my job search and doing that made me realize how much work I had to do on my whole setup. Nothing like fear of failure to motivate you to improve.

I ended up deploying 4 projects and wrote a few blogs about that process since it was a pain. As a beginner to deploying production websites I had to learn a ton of new terms and concepts. I still feel I ended up accepting defeat and using a 3rd party service for some of it, but it was worth it to me to get back to coding.

It also was a busy month for getting back into talks and lectures as me and some of my cohort mates ended up attending a React conference by Geekle and then later I attended a Vue.js workshop by Frontend Masters.

I spent a little bit of time learning Express and Node.js since my previous knowledge was comically low. I wanted to make an app that let me control my Nanoleaf lights and I made good progress on that, hopefully I can finish that up in October.

September was honestly a lot of groundwork with my resume and portfolio projects, although I had a pretty active article on how to use github as your portfolio.

I just keep reminding myself that it's a marathon, not a race. Here's to an active October!

How to use your Github's profile readme as a portfolio page

denvermullets — Sun, 27 Sep 2020 23:38:55 +0000

A couple of months ago Github released the ability for each account to have its own profile readme that displays when someone visits your Github. There's lots of fun and neat ways that people have been sprucing theirs up. There are quite a few repo's that collect some of the fun ones and even clone down their readme file and modify your repo readme so that yours looks just like it. If you want to browse some other profiles, take a look at these:

To actually create your own all you need to do is to create a repo that matches your Github username. Pretty simple! Make sure you create that Read me file since this is where you'll be putting in any information to display.

Now while it's fun to make your Github profile look like it came straight out of Myspace, I actually think as a newer developer you might want to try to make it be your portfolio for a while. Back when I was a motion designer, people that were hiring would pay more attention to your reel than your portfolio page, and you typically wanted to keep that video under 60 seconds. People don't have a long attention span, especially if what they're looking at is in rough shape. The thing is, I think, most people that are looking to hire you are going to look at your Github page. I'm not sure I see the value in having them click on your portfolio only to click on your projects link or your Github profile link. Let's make it a one stop shop!

This is how I have mine setup and I'll explain some of the extra stuff you might see. My Github for reference

I'm a firm believer in keeping things simple and I try to keep things as minimal as I can, but sometimes that slips away. Outside of the basic information I put in my profile I found this great repo that has lots of different widgets that you can insert into your readme to spruce it up. There's quite a few different options that you might want to check out.

https://github.com/anuraghazra/github-readme-stats

The other thing on my profile is a rolling weekly stat of what languages I've been programming in. I don't think this is particularly important to anyone aside from me but as a long time Last.fm fan, I kinda grew to dig these kind of constantly updating charts. For this one you'll need some 3rd party help from a Wakatime plugin. You set that up and then create a Github action that updates your readme every day to update your stats. I'll provide another repo that connects in to your weekly stats if you want to have more customization, but the first link is the one I use.

Cool, now that you have that all setup here are the reasons why I think this is a good way to present yourself to employers when you're new.

It saves you from yourself. Sometimes having a simple design is better than having a bad design
It forces you to spruce up your project Readme's which is something every developer should be trying to keep updated
You don't have to worry about hosting a site OR listing your portfolio site as a project you've worked on, plus it's responsive already
With pinning of repos, or creating custom links using one of the above methods, you can still highlight certain projects that you want employers to see

I'm not sure this is the way to do a portfolio, but it's certainly a fun way and lets you experiment with some new stuff and be creative. Good luck out there and let me know if you come across any other snazzy looking profiles!

How to setup your domain w/Netlify when using a externally hosted backend

denvermullets — Mon, 21 Sep 2020 00:32:38 +0000

So let's say you've deployed your front-end to Netlify and you're going to host your backend on a Heroku / DigitalOcean type of setup. It's a pretty easy process, and only gets easier the more you do it.

The basic idea is that you're going to create a sub-domain and tell it where to look to find your backend. A sub-domain is anything extra before your domain, like api.yourdomain.com or store.yourdomain.com.

What we're going to need is the IP of your backend server to tell Netlify where to route your sub-domain. Inside of your control panel at Heroku or DigitalOcean you will need to add the domain to your server. I am using Hatchbox.io to deploy my Rails API's so I'll add my domain there.

Now that we have the IP address of the server, let's go in to your Netflify DNS Panel and we'll create a new record connecting them.

Let's create the A record and then you'll be all setup.

You'll need to determine how you're going to set your SSL certs up, but that another topic for another day.

Some Notes on Deploying your React Site w/a Rails API using Netlify and DigitalOcean

denvermullets — Mon, 14 Sep 2020 01:48:32 +0000

After finishing out my time at Flatiron School, I wanted to deploy some of the apps that I had made and ended up learning a hard lesson in how much there is to #devops and deploying your apps.

Using DigitalOcean to host your backend is great since you can host a bunch of personal projects on one server and avoid the Heroku spin up delay for only $5 a month and easily expand it if you start getting traffic. If you don't have an account, feel free to use my referral link and you'll get $100 in credit.

There are downsides to decoupling your frontend and backend servers. The biggest one to me is that your local development will have minor inconveniences since you won't be using localhost:3000 and using an actual URL. All in all, it's not the worst way to do it.

I'm going to assume you already own a domain ready to go. I like to use Namecheap but you can use any site you want.

Starting with React

Getting started with Netlify is honestly super slick. You create an account, link your github and you can push your React app pretty quickly. They even take care of your main domain SSL Certificates. When it comes to deploying your front-end, I found that I needed to do 2 things. I needed to create a netlify.toml file, in my parent directory and put the following inside of it:

[build]
  command = "CI= npm run build"
  publish="build"

[[redirects]]
  from = "/*"
  to = "/index.html"
  status = 200

The [build] sets it so that your code can get pushed while having React warnings (really be sure this is a good idea for your app). The [redirects] is what's used if you're using React Router. Add your domain to your account and make sure you point your domain to Netlify nameservers. You can additionally setup a subdomain to point at your backend from inside of the Domain panel.

Moving to Rails

Honestly, for the most part this guide is almost perfect - AND - it has a video so you can follow along if you want. They walk you thru creating your droplet and doing everything you need. If you're using a preloaded SSH key on your droplet you'll need to do make some changes before you can execute the ssh-copy-id commands:

Edit the config file with this command
sudo nano /etc/ssh/sshd_config

Change this line from 'no' to 'yes', then save and exit
PasswordAuthentication no

Then restart daemon and continue with the ssh-copy-id portion.
sudo systemctl restart sshd
Once you do that you'll want to come back and change the PasswordAuthentication back to no.

Rails API changes

If you're using Rails as an API only app then you'll need to tweak your Capfile. Comment out this line and add the 2 other lines.

# require 'capistrano/rails'

require 'capistrano/bundler'
require 'capistrano/rails/migrations'