DEV Community: Gavi Schneider

IsraBrew: A Craft Beer Scraper

Gavi Schneider — Wed, 24 Feb 2021 09:01:33 +0000

We may be small in size geographically, but the craft beer scene in Israel is booming. New beers are appearing on shelves on a weekly (if not daily) basis, and I’m not just talking about beers brewed locally. New beers from countries all over the world, some of them landing in Israel for the very first time are also popping up left and right. Last month (January ’21) was a particularly historic time for Israeli craft beer enthusiasts, with over 15 new beers from BrewDog (and their experimental brewery, OverWorks) making their debut appearance in Israeli stores all on the same day.

As someone who likes to stay up to date with all the latest beers making their way to our country, I was looking for a way to see everything that’s available from all the different stores and suppliers in one place. I could have just visited every store’s website and scrolled through their inventory, but I’d essentially be doing the same thing over and over again, only on different sites (assuming I even know where to look — something that might pose as a challenge to new Israeli beer enthusiasts).

When you want repeat some process over and over again, automation is your best friend. Specifically, in my case I turned to automating a web scraper. By writing a program that would visit all the websites for me, pull in their inventories and aggregate them into one location I wouldn’t need to bounce from site to site to know what beers are currently available.

This is how IsraBrew was born: The need for a website to view what craft beers are currently available from all the major Israeli suppliers. If you’re as obsessive as me and want to try every beer that you can get your hands on, you’re going to love this.

The IsraBrew project is made up of four main components:

The Web Scraper: The app that visits the websites and populates the database.
The API: Responds to requests with beers it retrieves from the database.
The Client: Web interface to view the beers.
Deployment: Infrastructure and services that aided in deploying the app.

Web Scraper

Although I really like using Puppeteer when building projects that utilize web scraping in JavaScript / Node.js, I decided that this time I’d go with Python for my web scraping needs. Selenium would have been an obvious choice, and I’ve even used it previously in both Python and Java, however I decided to go with Beautiful Soup — its simplicity and speed (relative to Selenium) were what won me over. I was looking to build a lightweight scraper that I could deploy on a server and run a few times a day, and Beautiful Soup did that perfectly. Well, almost perfectly, there was one thing it couldn’t do: scrape content that was generated via JavaScript after the page loaded. This initially became a problem, as a few of the suppliers used JS to load their content. After countless Stack Overflow and Reddit threads, I finally discovered ChompJS, which solved this exact problem.

With ChompJS, I was able to target script tags in the DOM and get the content that they would eventually produce. Problem solved. Using both Beautiful Soup and ChompJS, I was able to scrape all the desired websites without actually having to open a browser (headless or otherwise).

Once scraped, the beers were then stored in an in-memory SQLite database. Last time I had worked with a Flask backend I wrote out all the SQL commands inside of my Python code, but this time I decided to use the SQLAlchemy ORM package, which allowed me to do CRUD operations in Python without any need for raw SQL.

The web scraper is deployed with the Flask server and is scheduled to run multiple times throughout the day, so whenever you’re browsing the site the products will be up to date.

API

This part turned out to be the shortest and easiest to implement. I created a barebones Flask server that had a ‘/beers’ route and would retrieve beers from the database and send them to the requesting client. In my opinion, Flask’s simplicity is both it’s strength and it’s weakness — for smaller projects that require a web server, Flask is super easy to get started with and very easy to understand. For larger projects however, its lack of structure may cause uncertainty and confusion when choosing how to properly build out your application (similar to Express in the Node.js ecosystem, it’s not very opinionated, leaving the developer with many decisions to be made). I personally prefer using Django when building larger web applications with Python, but for this particular project Django seemed like overkill.

Client

A simple Frontend built with my usual stack: React, TypeScript and TailwindCSS. I discovered this NPM package which made implementing different tabs for different suppliers super easy. You all know about React, I’m not going to go into much detail here. What I will say is — if you’re using React but not using Tailwind to style your components, you should be. I still use old school CSS (or SASS) for specific things, but for me Tailwind has become almost as invaluable as React — If I’m building a React project, I’ll be styling it with TailwindCSS.

Deployment

If you’re even remotely interested in the Flask world, you’ve probably heard of Miguel Grinberg, who’s books I consider to be our Flask Bible. His blog is just as informative, and I found this article that really helped me understand how to properly deploy a Flask project on a server.

Essentially, Nginx is what holds everything together: It proxies all traffic starting with ‘/api’ to the backend (which is running on a Gunicorn server), and everything else to the React frontend. Not only does Nginx make this really easy to set up, but because my frontend and backend are deployed on the same machine, I never had to deal with any CORS issues in this project, which was… new. The server itself is a DigitalOcean Droplet, which are super easy to deploy and configure.

Serving my app over HTTPS turned out be a breeze thanks to Cloudflare. I’m not entirely sure how they’re staying business with their ridiculously good free tier, but I’m very thankful for it.

One feature that I really wanted to add was search — the ability to search for a specific beer and have results from all the different suppliers appear. Unfortunately, there was a slight language barrier — some of the suppliers named their beers in English while some named them in Hebrew. It’s possible that three suppliers are all carrying the same beer, but if two of them are named in Hebrew and you searched in English, you’d only get one result. I can implement the search with both English and Hebrew options and make it clear that searching in a certain language will only yield partial results, but I’d really like to find a way to overcome the language obstacle so that I can implement a proper search bar that can really show you all your different options when searching for a particular beer.

If you’re into programming, you can check out the repo here. If you’re in Israel and looking for some good beer, you can check out IsraBrew here.

Cheers 🍻

bandDotRand: A Random Artist Generator

Gavi Schneider — Mon, 07 Dec 2020 10:38:28 +0000

After completing my DevOps project last month, I wanted to take the app I had built and turn it into something more permanent, something with a graphical user interface that could be used outside of the terminal. I had built the app for my capstone project, but there the main focus was on the infrastructure, not on the app itself. I quickly created a program that exposed an API endpoint and returned a random song when it received a request. The song was generated through the Musixmatch API, and would include basic song properties, such as the song ID, name, album, artist, etc.

Instead of getting a random song, my new idea was to create an app that would get a random artist and display it alongside the artists discography and related artists. Every artist or album on the page would include a link to that artist / album on a music streaming service, so you could actually listen to anything you found interesting. Refreshing the page (or clicking on the next button) would load another artist, so the website could be used as a random artist generator.

After initially building out a prototype for the website that would eventually become bandDotRand, I realized that the Musixmatch API wasn’t going to cut it, and I had to find a music API that would better suit my needs. Spotify was my next choice, and after diving into their API documentation, it seemed like the right choice. The one thing Spotify didn’t have was an API endpoint to access a random artist, but after reading this article on how to get random tracks using the Spotify API, I realized I could do something similar in order to get a random artist. Once I had my random artist, I’d need to make one more API call to get the artist’s albums, and then I’d be set.

I had the artist data, now all that was left to do was to create a frontend that would display all that data nicely. I created all the React components that I needed, added some styles and tied it all together.

Lastly, I deployed the app on Heroku. Getting the frontend to communicate with the backend turned out to be way more time consuming than I thought it would be (I swear it ran on my machine), but in the end it ran just fine. “In the end”, meaning seven hours of debugging later, but nonetheless, the app is deployed.

The website is built with React on the frontend, and Node.js with Express on the backend. Both frontend and back are written in TypeScript — after watching some Pluralsight courses on the topic, I was looking for an excuse to try it out. Turns out if you’re coming from a C/C++ (or any statically typed language) background, it really isn’t that difficult to transition to from JavaScript.

All of the styling was done with TailwindCSS, which is my new favorite utility-class library. The combination of React components and utility-class styling makes building reusable components that look good really simple.

Looking ahead, I’d love to be able to integrate other streaming service API’s so that the user can be redirected to the service of their choice, instead of just Spotify. I’d have to figure out a way to match the artist on multiple platforms, like taking an artist I received from the Spotify API and finding it’s counterpart on Apple Music or Tidal. Getting the artists website and social accounts would also make a great addition, giving you more access to that new favorite (or not so favorite) band you just discovered.

You can try out bandDotRand here. Let me know what you think, and tell me which new bands you’ve discovered.

My Capstone Project for Udacity’s Cloud DevOps Engineer Nanodegree

Gavi Schneider — Thu, 12 Nov 2020 14:10:06 +0000

After three months of various DevOps related courses and smaller projects, I had reached the end of my Nanodegree, and it was time to build out my capstone project.

My project can be broken down into two parts: the application itself, and the infrastructure that deploys and hosts it.

The Application: Random Song

Random Song is a simple web app built using TypeScript, Node.js and Express. It serves as a web service that can send you a random song, using the Musixmatch API. To test out the app, simply go to the /random route, and you'll receive a random song object in JSON.

Going to the / route will return:

Welcome to my capstone project! To get a random song, go to the '/random' route.

And going to the /random route will return a random song:

{
  track_id: 160557034,
  track_name: 'Get Up and Fight',
  track_name_translation_list: [],
  track_rating: 26,
  commontrack_id: 86880624,
  instrumental: 0,
  explicit: 0,
  has_lyrics: 1,
  has_subtitles: 1,
  has_richsync: 1,
  num_favourite: 62,
  album_id: 30545841,
  album_name: 'Simulation Theory (Super Deluxe)',
  artist_id: 1248,
  artist_name: 'Muse',
  track_share_url: 'https://www.musixmatch.com/lyrics/Muse/Get-Up-and-Fight?utm_source=application&utm_campaign=api&utm_medium=Student+Developer%3A1409620630471',
  track_edit_url: 'https://www.musixmatch.com/lyrics/Muse/Get-Up-and-Fight/edit?utm_source=application&utm_campaign=api&utm_medium=Student+Developer%3A1409620630471',
  restricted: 0,
  updated_time: '2020-05-19T15:42:03Z',
  primary_genres: { music_genre_list: [[Object], [Object] ] }
}

The Infrastructure

After the application was built, the next task was deploying it. In this project I decided to go with a Rolling Deployment. My goal was to write out the necessary configuration files and required build commands, and then create a pipeline to automate the process of actually building the application and deploying the infrastructure. This way, it could be executed in the exact same manner every time I added new code or infrastructure to the project. I needed a server to host Jenkins, my CI/CD technology of choice for this project. After provisioning an AWS EC2 instance and installing Jenkins, it was time to start defining the tasks that I’d want Jenkins to run. After accessing my application’s code, here are the tasks that I created for Jenkins to run:

Install Node dependencies Simply running npm install would do the trick.
Build the application My application is written in TypeScript, so I needed to run npm run build to build out the JavaScript distribution folder.
Lint the code Running npm run lint to make sure everything is up to tslint's standards.
Build the Docker image Here Jenkins would build a Docker container based on the Dockerfile that I created. It was based off a simple Node image, and would copy my application code into the container and start it.
Upload the container to Docker Registry After being containerized, my application would then be uploaded to the Docker Registry for further availability.
Create the Kubernetes configuration file Here I needed to create a Kubernetes deployment file that would be used in the next step to actually deploy my application into a cluster. I used Kubernetes via AWS EKS.
Deploy application With the help of my Kubernetes deployment file and my Docker container that I uploaded to the registry, I was now able to deploy my application to my AWS EKS cluster. I also ran a kubectl get pods and kubectl get services to make sure everything was running as expected.

In the end, the app is deployed to the cluster and accessible to users. Random songs for days.

Unfortunately, the app is not currently deployed due to EKS not being a cheap service for a student to continuously pay for. However, I’m planning on taking the Random Song application and turning it into something that will be more permanently hosted in a future project. As far as the infrastructure goes, these are also things that can be repurposed in future projects — Docker containers, Kubernetes clusters and Jenkins pipelines are tools that can help build any software related project.

If you’d like to see the code, you can take a look at the project’s repo on GitHub.

Watchlist: Your New Movie Tracker

Gavi Schneider — Thu, 02 Jul 2020 14:20:16 +0000

Watchlist is my final project for CS50’s Introduction to Computer Science, by Harvard University.

Watchlist is your new to-do list for movies. Instead of trying to remember what movies you’d like to watch, just add them to Watchlist, where you can view them anywhere with an internet connection.

Watchlist is super simple: register, and you’ll be taken to your watchlist, which will initially be empty. Click on the ‘Add’ button in the navbar to start adding movies. Type in the movie you’d like to search for, and it will be added to your list.

Can’t remember the name of that movie you wanted to watch? Just log into Watchlist!

Finally got around to watching that movie? Click the button at the end of the movie’s row to delete it.

Watchlist was built with the following technologies:

HTML
CSS
Bootstrap
Python
Flask
SQL
SQLite

Watchlist’s movie data is from iTunes via their API, which technically means that a movie has to be on iTunes in order to add it to your Watchlist. As of April 2020, there are 65,000 movies in the iTunes store, so hopefully you won’t feel too limited when searching.

Lastly, I’d like to thank the CS50 staff who have put together a tremendous course. I took an Intro to Computer Science course in college over two years ago, yet by the end of this course I realized I had learnt so many new things that I hadn’t previously learned in my CS degree. I’m looking forward to taking my next CS50 course.

You can check out Watchlist here: https://gavischneider.pythonanywhere.com/login

Full Stack Certification

Gavi Schneider — Sat, 06 Jun 2020 22:38:37 +0000

Super proud of this one. Today I finished FreeCodeCamp's curriculum and earned the Full Stack Developer certificate.

https://www.freecodecamp.org/certification/gavischneider/full-stack

Music Genres: My Second NPM Package

Gavi Schneider — Sat, 23 May 2020 18:53:49 +0000

In case you haven’t already noticed, I love music. Not the way everyone says they do, I actually love it. So after yesterdays SLAAAYER rush, I quickly dove into my next music-related NPM project: Music Genres.

The package is simple: install and require it, and then you’ll have access to three functions:

getAllGenres()
getRandomGenre()
getRandomSubgenre()

The first will return all the stored genres and sub-genres (of which there are currently 13 and 222, respectively) in a JavaScript Object. The second will Return a random genre, an array holding sub-genres. Lastly, the third will return a random sub-genre, which is a string.

Example

const musicGenres = require("music-genres");

musicGenres.getAllGenres();
// Returns all genres - an object holding genres

musicGenres.getRandomGenre();
// Returns random genre - an array holding sub-genres

musicGenres.getRandomSubgenre();
// Returns random sub-genre - a string

You can install the package here: https://www.npmjs.com/package/music-genres

I’d love any and all feedback. Say hi on twitter @gschnei.

My Very First NPM Package: SLAAAYER!

Gavi Schneider — Fri, 22 May 2020 19:10:04 +0000

Originally published on my blog.

No, it doesn’t do much, as you probably guessed from the name. But it’s my first NPM package, and it works. A tiny piece of software that has my name on it. That’s good enough for me.

The package is simple: add it to your dependencies, require and name it like so:

const slayer = require(‘slaaayer’);

The name of the variable doesn’t actually have to be slayer, name it whatever you’d like.

Once you’ve done that, call slayer and pass it a number. The result is “SLAAAYER! 🤘🏻”, where the number you pass the function is the number of “A”s in the word slayer.

console.log(slayer(10));

will log: SLAAAAAAAAAAYER! 🤘🏻

You can install the package here: https://www.npmjs.com/package/slaaayer

Stay tuned for future NPM packages. I think Cannibal Corpse has a good ring to it…