DEV Community: Dirk Hoekstra

How to write a Web Scraper with Ruby

Dirk Hoekstra — Sat, 16 Mar 2024 10:51:27 +0000

https://scraperbox.com/blog/web-scraping-with-ruby

How I spend $500 per day because of a misconfiguration

Dirk Hoekstra — Fri, 05 Feb 2021 12:19:36 +0000

In January I unknowingly spend $500 per day because of a misconfiguration.

Read the full story here:
https://scraperbox.com/blog/spending-500-per-day-because-of-a-misconfiguration

Web Scraping with Ruby

Dirk Hoekstra — Tue, 22 Dec 2020 13:15:03 +0000

Hi guys 👋,

I spend the last week writing a super in-depth Ruby web scraping guide.

I show step-by-step how to build an indeed.com scraper using Ruby.

You can find the article here

Any feedback would be appreciated! :)

I couldn't find a house in Amsterdam, so I build a tool that helped me find a place to live in!

Dirk Hoekstra — Thu, 19 Nov 2020 17:46:40 +0000

Finding a rental in Amsterdam is really hard. There are way more people looking for a house than there are houses available.

I would often react to a rental listing only to find out that I was too late (even though I reacted the same day it was posted!)

So I started building some scrapers that would check all the rental listing websites every 5 minutes.

Each time a new listing came online the tool would send me an email! This helped me find a really nice place in Amsterdam and I thought "why not build a small SaaS tool out of it)

Fast forward a few weeks and WoningAlert.com is live!

Let me know what you guys think!

I build a Web Scraping API, curious what you guys think!

Dirk Hoekstra — Sun, 23 Aug 2020 15:44:00 +0000

I was playing around with Headless chrome browsers and proxy pools and decided it would be fun to turn it into a side project.

A few months later and the website is live :)

Basically it is an API that handles headless Chrome browsers and proxy pools for you so you can easily scrape data from the web.

I'm curious what you guys think of it!

Cloud Run vs App Engine

Dirk Hoekstra — Sun, 29 Dec 2019 09:56:49 +0000

Should you use Google’s Cloud Run or Google’s App Engine for your next project?

What is Cloud Run?

In a nutshell, you give Google’s Cloud Run a Docker container containing a webserver. Google will run this container and create an HTTP endpoint.

To deploy to the Google Cloud you should build a Docker container image on your local machine. Then, you should upload this container to Google Cloud.

gcloud auth configure-docker
docker push grc.io/PROJECT-ID/IMAGE

And finally, you can deploy the pushed container with the following command.

gcloud run deploy --image gcr.io/PROJECT-ID/IMAGE

All the scaling is automatically done for you by Google. If there is a lot of traffic it will add more containers containing the webserver.

Sounds great, but there’s a catch.

Cloud Run depends on the fact that your application should be stateless. If you want to host a traditional web application this means that you should divide it up into a stateless API and a frontend app.

What is App Engine?

With Google’s App Engine you tell Google how your app should be run. The App Engine will create and run a container from these instructions.

For example, to host a Node project on Google’s App Engine, I can simply create the following app.yml file.

runtime: nodejs12
entrypoint: node build/server.js

To deploy it to the App Engine I can run the following command.

gcloud app deploy

The App Engine will build a Docker Image with the nodejs12 environment and run the entrypoint command.

All the scaling is automatically done for you. And as a bonus, you can even store state information on the server.

This means that if you want to host an existing web application to the Google Cloud, the App Engine is probably your best bet.

Which service should you use?

If you have an application that is not stateless then you should go for the App Engine.

Also, deploying with App Engine is super easy. You simply fill out an app.yml file and Google handles everything for you.

However, with Cloud Run, you have more control. You can go crazy and build a ridiculous custom Docker image, no problem!

In my opinion Cloud Run is made for devops engineers, App Engine is made for developers.

How I structure my Express + Typescript + React applications

Dirk Hoekstra — Sat, 28 Dec 2019 14:13:13 +0000

In this article, I will show you how I set up and structure my Express — React projects.

Folder structure

When I set up a React Express app I always use the following folder structure.



├─app
├─build
├─frontend

The app directory will hold the Express backend application.
The build directory will hold the production build of the frontend and backend application
The frontend directory will hold the React frontend application.

Note that you are free to use any other folder structure that you like, this is simply my preferred way of doing things.

Creating the React app

Let’s begin with creating the React app. I’m going to use the create-react-app npm package for this.

You can run npm packages without installing them using the npx tool.



npx create-react-app frontend

The react app will be created in the frontend folder.

Let’s test if the installation went correctly.



cd frontend
yarn run start

The yarn run start command will run the React development server. Whenever you make changes to a file it will automatically recompile the react app and reload the browser! 🚀

The create-react-app package will also initialize a git repository in the frontend directory. However, I want to have a single git repository in the project root directory.

To remove the git repository in the frontend directory I simply remove the .git directory.



rm -rf .git

Creating the Express app

We now have a working frontend application, now it’s time to set up the backend Typescript Express app.

I start by creating a new package in the project root directory.

Then I add the Express and Typescript dependencies and finally, I create the app directory.



yarn init
yarn add express @types/express typescript
mkdir app

Next, I create a pretty standard tsconfig.json file. This file contains the settings for compiling Typescript to Javascript.



{
  "compilerOptions": {
    "module": "commonjs",
    "baseUrl": "./",
    "outDir": "build",
    "target": "es6",
    "moduleResolution": "node",
    "esModuleInterop": true,
    "lib": ["es6"],
    "allowJs": true,
    "forceConsistentCasingInFileNames": true,
  },
  "include": [
    "**.ts"
  ],
  "exclude": [
    "./frontend"
  ]
}

I only want to use Typescript in the backend — at least for now. That is why I exclude the frontend directory.

In the app directory, I will create a Server.ts that will contain a Server class.




import {Express, Request, Response} from "express";

export class Server {

    private app: Express;

    constructor(app: Express) {
        this.app = app;

        this.app.get("/api", (req: Request, res: Response): void => {
            res.send("You have reached the API!");
        })
    }

    public start(port: number): void {
        this.app.listen(port, () => console.log(`Server listening on port ${port}!`));
    }

}

This class will receive the Express app in the constructor and initialize the application routes.

In the real world, I would probably create another class Router.ts that will hold all the application routes, but that is out of scope for this article.

To initialize this server I create a index.ts file in the application root directory. All this does is create a new Server class and start the server.



import {Server} from "./app/Server";
import express from 'express';
const app = express();

const port = 8080;

const server = new Server(app);
server.start(port);

To start the backend server I add the following snippet to the package.json file. It will use the ts-node package to directly run Typescript code.

This way you won't have to worry about compiling the Typescript to Javascript as it is all done for you.



"scripts": {
    "start": "npx ts-node index.ts"
}

That why I can start the server running the following command.

As a bonus, you can use Nodemon to automatically restart ts-node when a file changes.

Building the React app

Let’s make a production build of the React app.

I will make a change to the frontend/package.json file. Because after building the React application I want to move the build files to the /build/frontend folder.

Find the "scripts" and update the "build" line.



"scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build && mv ./build ../build/frontend"
  },

Let’s run the yarn run build command and see if it works! 🙌



mkdir build 
cd frontend
yarn run build

If you navigate to the /build/frontend directory you will see the production-ready React app!

Building the Express app

Let’s add a new "build" script to the package.json file.



"scripts": {
    "start": "npx ts-node index.ts",
    "build": "npx tsc"
  }

This will simply call the Typescript compiler package tsc to compile — or transpile if you prefer 💁‍♂— the Typescript to Javascript.

Run the build command to test if it works!



yarn run build

If all went correctly your build directory should look like this.



build/
├─app/
├─frontend/
├─index.js

Connecting Express and React

We can develop the backend and frontend applications and build them. However, we should also connect Express to React.

For example, if I browse to localhost:8080/ I should get to see the React application.

If I browse to localhost:8080/api I should get to see the API message.

To do this I update the constructor of the Server class.




constructor(app: Express) {
    this.app = app;

    this.app.use(express.static(path.resolve("./") + "/build/frontend"));

    this.app.get("/api", (req: Request, res: Response): void => {
        res.send("You have reached the API!");
    });

    this.app.get("*", (req: Request, res: Response): void => {
        res.sendFile(path.resolve("./") + "/build/frontend/index.html");
    });
}

First I mark the build/frontend directory as a static asset directory. This means that Express will automatically serve the files in that directory.

Next, I add a wildcard * route and send those all to the index.html file. This is the file that holds the React frontend application.

Let’s rerun the backend application.



yarn run start

When navigation to localhost:8080 I get to see the React application 🎉

When navigating to localhost:8080/api I get to see the API message 🔥

That’s it! You can find the source code here on Github 🚀

Pitfalls of Vue's reactivity

Dirk Hoekstra — Fri, 01 Nov 2019 06:51:12 +0000

My Vue views would sometimes refuse to automatically re-render. After slamming Vue.$forceUpdate() everywhere in my codebase, I decided enough was enough and dove into Vue’s reactivity system.

View the post here

I build a Website Screenshot API, curious what you guys think!

Dirk Hoekstra — Tue, 29 Oct 2019 10:32:28 +0000

It started with me trying to figure out how to control a headless browser in Javascript. A few weeks later I had crafted a full SaaS application

I created a writeup on how I did it here

I'm curious what you guys think about it, would love some feedback!