How I built a simple "stock analytics" site with web scraping and serverless functions

Istvan Kreisz — Sun, 07 Feb 2021 04:07:32 +0000

A little backstory: the GameStop short squeeze

Everyone who even just loosely follows american news must have heard of the craziness going on on the US stock market in the past few weeks. In short the story is that a large group of Reddit users from the WallStreetBets subreddit, which describes itself as "a community for making money and being amused while doing it. Or, realistically, a place to come and upvote memes when your portfolio is down." almost brought Wall Street to its knees. They discovered that the GameStop stock was massively overshorted by a number of large hedge funds and in an effort to stick it to those greedy Wall Street investors they decided to start mass buying the company's shares. This drove up the stock's price so much that some hedge funds lost billions and needed to be bailed out to avoid bankruptcy. There's more to the story than this but let's get to the topic of the article.

We Like The Stock!

Okay, I admit calling WeLikeTheStock.live a stock analytics site is a bit generous for this ridiculous looking meme site. My goal was not to build a Yahoo Finance competitor, but be a small part of this historic movement and build something that can bring a smile to some disgruntled Reddit investors' faces.

Web scraping, serverless functions and serverless database

I've been a heavy user of React and Next.js so they were obvious choices for this project as well. Next.js makes the development of React sites a breeze with many useful features such as static & server side rendering, easy routing etc. If you use their own hosting solution, Vercel and hook it up to your site's repository then the whole deployment process is as easy as just pushing your changes to GitHub (or BitBucket or GitLab). Vercel also has built-in support for serverless functions which you can write in the same repo as your frontend code and push it to production along with your site. It doensn't get much easier than that!

For the web scraping part I went with Scarper API which is another wonderfully simple to use framework, allowing developers to easily grab the HTML content from any page.

I used Lambda Store to store data, which is a serverless database solution which takes only a few clicks to set up. It was my first time using it but I was suprised at the simplicity of it. It may not be suitable for a large scale application but it was perfect for my needs here.

Hooking it all up

So we've got everything we need but how does it all come together to form this gorgeous looking website? Well the frontend code is not much to look at, the juicy part is the serverless function. It's one simple endpoint which provides the data to the site through Next.js's getServerSideProps function, which lets you pre-render the page on each request before serving it to the user. This function is an async function which can go and fetch data or do any kind of work before returning the result which will be passed in as render props to the React site.

The serverless function itself does the following:

It checks the Redis database for recently updated data.
If it doesn't find anything or if the data is out of date (older than 20 mins), it goes ahead and scrapes https://www.highshortinterest.com/ which is a site listing the most shorted stocks. Scraper API takes care of all the scraping work, returning just the raw HTML code.
With the help of JSSoup it pulls out the data from the HTML and transforms it into a JSON object.
Once that's done it saves the new data to the database, and returns it for the frontend to use.

And that's how you build a site using web scraping and serverless functions in no time! While this toolset would not scale well for larger applications, it's best not to overcomplicate things when all we want is a simple website that will break America's financial system.

Go check it out now!

Do you like GeoGuessr? I built a multiplayer version of it (in React) that you need to try.

Istvan Kreisz — Thu, 15 Oct 2020 15:11:29 +0000

Geo... what?

In case you don't know GeoGuessr is an online game where you're dropped somewhere on Earth and with the help of Google Street View you need to figure out where you are. It's a very popular online game and a personal favorite of mine.

So why make another one?

A little background story: besides being a GeoGuessr fan, I've also spent countless hours playing .io-type online multiplayer games, like: skribbl.io and slither.io. The multiplayer element (among other things) makes these games a lot of fun and (unfortunately) highly addictive.

And that's what's missing from GeoGuessr.. Don't get me wrong it's a great game, but I figured it could be even greater if it had some sort of multiplayer functionality. So I built my own version of it.

Introducing... Hide & Seek World!

In Hide & Seek World you're up against real players instead of just getting computer-generated random locations.

The game consists of a few turns, and in each turn there's one hider who selects a location somewhere in the world, and 1-5 seekers who try to guess the location based on Google Street View images.
As a hider your goal is hide well because the better seekers are able to guess your location the less points you get. As a seeker you get more points if you guess better.

The concept is similar to the popular children's game, hence the name. The exception is that you have a bigger playground here, the whole world. At the end of each round you can see the guesses of all the players and how far everyone was from the actual hiding place.

Game modes

There's 2 game modes: Street View and Satellite. In Street View mode you hide by selecting a street view location somewhere and in Satellite mode your hiding spot is a zoomed in rectangular area on the surface of Earth. In Street View mode you need to be able to read visual cues about architectural styles, vegetation, people etc. to guess well, while Satellite mode relies more on your geographical knowledge.

You can also create a private room and play against your friends, where you can customize more things like: how much time you have to make a guess or restrict the map to only certain countries.

How I built it

I built the frontend using Next.js & React, and the backend with Firebase services: Authentication, Realtime Database, FireStore, Functions, Analytics. For the maps & Street View related things I used Bing Maps and Google Maps APIs. Payments are handled with Stripe Checkout. The site is hosted on Netlify.

DEV Community: Istvan Kreisz

How I built a simple "stock analytics" site with web scraping and serverless functions

A little backstory: the GameStop short squeeze

We Like The Stock!

Web scraping, serverless functions and serverless database

Hooking it all up

Go check it out now!

Do you like GeoGuessr? I built a multiplayer version of it (in React) that you need to try.

Geo... what?

So why make another one?

Introducing... Hide & Seek World!

Game modes

How I built it

Go check it out now!