DEV Community: zchtodd

Dockerizing Django: How to Create a Consistent Development Environment for Your Team

zchtodd — Sat, 25 Mar 2023 10:28:48 +0000

In this tutorial I'll show how you can create a shared and repeatable development environment that your entire team can use
while working on a Django application. We'll use Docker to containerize not only your Django application, but a PostgreSQL database,
Celery instances, and a Redis cache server. Along the way, we'll use best practices to improve developer productivity and keep configuration
maintainable.

Setting Up Docker

This article will assume we're using Ubuntu 22, but don't worry if you're on Windows or Mac. Installation instructions for all platforms
can be found on the Docker website.

To get started installing Docker for Ubuntu, we'll first need to make sure apt has the packages it needs to communicate with repositories over HTTP.

sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

Next we'll need the official GPG key from Docker to verify the installation.

sudo mkdir -m 0755 -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Now we'll add Docker as a repository source.

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

We should be able to issue an apt update to begin working with the new repository and then install Docker.

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Once Docker is installed, you should be able to run the following to download and run a test image.

sudo docker run hello-world

It would be nicer to run docker without root, so let's add a docker user group for that purpose. We'll also go ahead
and add

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Containerizing Django

As a first step, we'll focus on writing a Dockerfile for the Django container. Dockerfiles are essentially a recipe for building
an environment that has exactly what's required for an application, no more, and no less.

Let's take a look at the Dockerfile for Django that we'll use.

FROM python:3.11-alpine

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

To break down what's happening here, let's start with the first line.

The FROM statement defines the image that we're
inheriting from, or building off of. In this case, we are starting with an image built to provide python3.11. The alpine
means that the image is very bare-bones (i.e. it won't have many common programs such as bash) and so takes up much less
disk space than some other images. This can become important, for instance, if you're running tests on a platform like
GitHub, where the image will be frequently downloaded.

The WORKDIR command sets the current directory and also creates that directory if it doesn't exist yet. Consequently,
the COPY command on the next line is relative to the /app directory we set just above. The COPY command copies
the requirements.txt file from the host machine into the Docker image. As soon as that's available, we run pip install
to make sure all the dependencies are available.

Finally, we run one more COPY command to copy over all code in the current directory to the image.

At this point, you might wonder why we copied requirements.txt specifically when we're just going to copy everything a few
steps later.

Why not write the Dockerfile like this?

FROM python:3.11-alpine

WORKDIR /app

COPY . .
RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8000
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

To be sure, this would work just fine!

The issue lies in how Docker rebuilds images. To be efficient, Docker caches images into layers. Each command
that we run creates a new layer. When rebuilding, Docker will reuse cache layers when possible. The Dockerfile layout above, however, would require
rerunning pip install when any code is touched. Copying requirements.txt and running pip install above the last COPY allows Docker to cache the
dependencies as a separate layer, meaning we can change application code without triggering a time consuming pip install process.

There are a few pitfalls to watch out for with the COPY command. While we want to copy all the code, we don't want to pull in
unnecessary files and blow up the size of the final image. In addition to wasting disk space and taking longer to build, this can also
cause frequent rebuilds if constantly changing files are pulled in. Luckily, this is exactly what the .dockerignore file helps with.

The syntax of the .dockerignore file matches .gitignore if you've written one of those. Any file that matches a pattern in
.dockerignore will be ignored, so we can use COPY without fear of dragging in unnecessary files.

Here's an example of how we can prevent compiled Python bytecode from ending up inside the image. We'll also ignore the .git directory
as well, since we definitely don't need that living inside the image either.

*.pyc
__pycache__
.git

Assuming you have a Django project and a Dockerfile similar to the above, we should be ready to try starting a Django container.
You can build the image like below, using the -t or tag argument to give the image a name.

docker build -t myawesomeapp .

You can run the Docker container and expose port 8000 to test it once the image has finished building.

docker run -p 8000:8000 myawesomeapp

With the container up and running, the Django welcome page should display when you visit http://localhost:8000 in the browser.

Building a Simple Compose File

Having the app containerized is a great first step, but there's a lot more to a development environment, like the database, caching server, and static file server.
Although Django can serve static files, I find it simpler to go ahead and mirror the production setup, with a server such as Caddy in front for static files.

We'll start with a basic docker-compose.yml that contains just the app service.

 services:
   app:
     build: .
     command: python manage.py runserver 0.0.0.0:8000
     restart: unless-stopped
     volumes:
       - $PWD:/app
     ports:                                                 
       - 8000:8000

The service is named app in this example, but it could be named anything as long as it's valid YAML syntax.

The build key specifies that we want docker-compose to trigger a rebuild of the Docker image when dependencies have changed. This is useful, as we won't have to manually rebuild an image when we make a change, like adding a new module to requirements.txt. The command key will override whatever CMD is specified in the Dockerfile. We don't have to specify command if a CMD was given in the Dockerfile but I like to include it here for clarity.

Restart defines what we'd like to happen if the container stops. By default, a container that stops is not restarted. Here we unless-stopped to declare that we'd like the container to restart unless it was manually stopped. Most importantly, this means that our services will restart after a machine reboot, which is probably what we want in a development environment.

Volumes give a container access to host directories. The left side of the volume assignment maps to the host, while the right defines where the volume should appear within the container. The $PWD is just a short-hand for the directory where docker-compose is running so that we don't have to hard-code that path, which is likely different for every developer on the team.

But what use are volumes? Mapping the code into the container is very common in development setups. Otherwise, the code is frozen in time as of the moment we built the image. Without volumes, we'd have to rebuild the image every time we changed the code.

Lastly, we have the port key. Just as we did when launching the container manually, we're specifying that port 8000 on the host should forward traffic to port 8000 within the container.

Building and Running the App Service

With our first docker-compose.yml file written, we should be ready to start up the app.

Run the command docker-compose up within the Django project root to start the app container. Since it's the first time starting the app, Compose will build an image before running the app. Once the image is built you should be left with the app running and reachable on port 8000. I prefer running Compose in the background, which you can do with the docker-compose up -d command.

You'll notice that if you add a new app to the Django project and include a "hello world" view, that the view is automatically available, without needing to rebuild the container. This is due to the volume that we defined in the docker-compose.yml file.

On the other hand, if we add djangorestframework to the requirements.txt, that won't be available until we rebuild. This is because while our code is mapped into the container, the directories where Python stores third-party code are not. We can run the docker-compose build command to rebuild the image, taking into account new additions to the requirements.txt file. This will, however, leave the old container running if we started Compose in the background. To combine building and running the container, you can use the docker-compose up --build command.

Adding the Database Service

Now we'll add a PostgreSQL instance to the docker-compose.yml file. We'd like the data to be persisted so that it will survive the container being removed, and we also need the app server to start after the database has initialized.

Let's take a look at the updated file.

services:
   app:
     build: .
     command: python manage.py runserver 0.0.0.0:8000
     restart: unless-stopped
     depends_on:
       - database
     volumes:
       - $PWD:/app
     env_file:
       - app.env
     ports:                                                 
       - 8000:8000 

   database:
     image: postgres:13
     restart: unless-stopped
     environment:
       - POSTGRES_USER=postgres
       - POSTGRES_PASSWORD=example_password
       - POSTGRES_DB=example_db
     volumes:
       - ./postgres-data:/var/lib/postgresql/data
     ports:
       - 5432:5432

There are a few new concepts here. First, we're controlling what environment variables will exist inside the containers with the env_file and environment keys. The two function very similarly, but env_file pulls the values from a separate file. This can be nice when you have several services each using an overlapping set of variables.

Here's what the app.env file referenced above should contain at this point:

POSTGRES_USER=postgres
POSTGRES_PASSWORD=example_password
POSTGRES_HOST=database
POSTGRES_PORT=5432
POSTGRES_DB=example_db

One interesting thing to note here is that the POSTGRES_HOST is set to database to match the service name. Services are automatically reachable from one container to another, with each service name added as a DNS entry.

We should now be able to modify the Django settings.py file to connect to the database.

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": os.environ["POSTGRES_DB"],
        "USER": os.environ["POSTGRES_USER"],
        "PASSWORD": os.environ["POSTGRES_PASSWORD"],
        "HOST": os.environ["POSTGRES_HOST"],
        "PORT": os.environ["POSTGRES_PORT"],
    }
}

Assuming you've run docker-compose up -d again to create the database, it should be possible to run the Django database migrations to populate the initial schema.

docker exec -it tutorial_app_1 python manage.py migrate

The name of your app container might be different, so you should run docker ps first to see what name you'll need to use.

Adding the Caddy Service

It's typically not recommended to run Django with the development server facing the Internet, so we'll use Caddy to serve static files and pass requests to Django. As a bonus, the Caddy webserver will handle SSL setup, interacting with LetsEncrypt on our behalf.

Let's take a look at the updated docker-compose.yml and then dive into what's changed.

services:
   app:
     build: .
     command: gunicorn tutorial.wsgi:application -w 4 -b 0.0.0.0:8000
     restart: unless-stopped
     depends_on:
       - database
     volumes:
       - $PWD:/app
     env_file:
       - app.env
     ports:                                                 
       - 8000:8000    

  caddy:
    image: caddy:2.4.5-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy_data:/data
      - ./caddy_config:/config
      - ./static:/static

   database:
     image: postgres:13
     restart: unless-stopped
     environment:
       - POSTGRES_USER=postgres
       - POSTGRES_PASSWORD=example_password
       - POSTGRES_DB=example_db
     volumes:
       - ./postgres-data:/var/lib/postgresql/data
     ports:
       - 5432:5432

The app service is now using Gunicorn instead of the built-in Django development server. The Caddy server has volumes for its config file and for access to a directory where we'll put static files. Caddy will use the caddy_data and caddy_config volumes to store some of its own internal state, which means we can afford to lose the container, but recreate it if need be without any problems.

In order to switch to using Gunicorn, you'll need to add gunicorn to requirements.txt and issue a docker-compose build command.

Now let's examine the Caddyfile configuration.

localhost {
  handle_path /static/* {
    root * /static
    file_server
  }

  @app {
    not path /static/*
  }

  reverse_proxy @app {
    to app:8000
  }
}

This config proxies requests to all URLs except those beginning with /static over to the app service. If you recall from earlier, we use the name app here because it matches the service name in the docker-compose.yml file. Caddy handles all requests to /static with the file_server directive, so that Django is never involved in serving static files.

Once these updates are in place, you should be able to run docker-compose up -d to get Caddy running.

If you got to localhost in your browser, however, you'll notice that the self-signed certificate is not trusted. Normally, Caddy will add its own certificates to the system trust store. Since we're running Caddy inside a container, however, it can't make that update. If you like, however, you can copy the root certificate out of the container in order to avoid the browser warnings.

sudo cp ./caddy_data/caddy/pki/authorities/local/root.crt ./

For Firefox, for example, you can go to Settings, Privacy & Security, and click View Certificates. Under the Authorities tab, you should see an Import button. Import the root.crt file we copied out of the caddy_data directory. With that added, visiting localhost should no longer raise the SSL warning.

Adding Celery Instances

Now we'll dig into adding Celery instances so we can do some background job processing with our new project.

Celery requires a broker and a result backend and supports multiple destinations for both. Redis is a popular choice here, and we'll use it as the broker and result backend for simplicity sake. Adding Redis to the existing docker-compose.yml is straightforward. I'll show that below while omitting what we've already written, since it won't change.

  redis:
    image: 'redis:7.0'
    restart: unless-stopped
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
    ports:
      - 6379:6379

The ALLOW_EMPTY_PASSWORD setting should only be used in development as a convenience to avoid specifying passwords.

With redis added to the environment, we'll need to update the app.env file to give the app and worker containers enough information to connect. Add the following to the app.env to make the environment variables available to the containers.

CELERY_BROKER_URL=redis://redis:6379/0
CELERY_RESULT_BACKEND=redis://redis:6379/0

Typically when building an app that uses Celery, I like to separate jobs into different types that are assigned to their own queues. This offers a number of advantages, such as isolating trouble with one job type to a single queue, instead of causing problems across the entire app.

Having several queues requires that we spin up a container per queue. Normally, this would result in duplication, as we'd have to redefine much of the same configuration over and over. Fortunately, we can use the templating functionality of YAML to avoid this issue.

Here's an updated docker-compose.yml that features two Celery instances.

x-worker-opts: &worker-opts
  build: .
  restart: unless-stopped
  volumes:
    - $PWD:/app
  env_file:
    - app.env
  depends_on:
    - redis

services:
  tutorial-worker1:
    command: tools/start_celery.sh -Q queue1 --concurrency=1
    <<: *worker-opts

  tutorial-worker2:
    command: tools/start_celery.sh -Q queue2 --concurrency=1
    <<: *worker-opts

The start_celery.sh shell script contains the rather lengthy full command to start Celery. The command is wrapped with watchmedo which will reload the Celery instance any time a Python file changes inside the project directory.

#!/bin/sh
watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery -A tutorial worker "$@" --loglevel=info

The bash syntax $@ substitutes the arguments to the shell script into that position. In this case, that means inserting the queue argument into the command.

Next Steps

There is of course still much more to cover when it comes to Docker, such as adapting this setup to work with a CI/CD pipeline. Luckily, Docker and Compose have features that make running the same app in different environments fairly straightforward.

For now though, I hope this tutorial has been useful in setting up a development environment for your Django app.

Happy coding!

Beyond the Basics (Part V): Formik, D3, and More!

zchtodd — Sun, 18 Jul 2021 14:58:13 +0000

Once you've finished this post, you'll have a template for easily creating forms using Formik, as well as experience with D3 visualizations!

If you haven't read the first post in the series, this is a step by step guide on building a SaaS app that goes beyond the basics, showing you how to do everything from accept payments to manage users. The example project is a Google rank tracker that we'll build together piece by piece, but you can apply these lessons to any kind of SaaS app.

In the last post, we implemented user authentication in both Flask and React. Now that the basic structure is in place, we're going to implement a full "slice" of the application – we'll build the proxies page, where users can add and delete crawler proxies. I call it a slice because we'll build every part of the functionality in this post, from the data model to the user interface.

You can find the complete code on GitHub.

Building the Proxy Connection Data Model

The proxy model will contain all the details needed for Puppeteer to crawl Google using that connection, such as URL, username, and password. We'll also keep track of some stats, such as a counter of how many times the proxy is blocked, which will come in handy later when we want to visualize proxy performance with D3.

class ProxyConnection(db.Model):
    __tablename__ = "proxyconn"

    id = db.Column(db.Integer, primary_key=True)

    user_id = db.Column(
        db.Integer,
        db.ForeignKey("user.id", ondelete="CASCADE"),
        index=True,
        nullable=False,
    )

    proxy_url = db.Column(db.String, nullable=False)
    username = db.Column(db.String, nullable=False)
    password = db.Column(db.String, nullable=False)

    # Can this proxy support multiple parallel requests?
    allow_parallel = db.Column(
        db.Boolean, default=False, server_default="f", nullable=False
    )

    success_count = db.Column(db.Integer, default=0, server_default="0")
    block_count = db.Column(db.Integer, default=0, server_default="0")
    no_result_count = db.Column(db.Integer, default=0, server_default="0")
    consecutive_fails = db.Column(db.Integer, default=0, server_default="0")

    # Proxy is currently in use (only applicable when allow_parallel = 'f').
    engaged = db.Column(db.Boolean, default=False, server_default="f")

    # Must wait at least this long before allowing another request.
    min_wait_time = db.Column(db.Integer, default=0, server_default="0", nullable=False)

    # Use random delay when proxying with a static IP to avoid blocks.
    random_delay = db.Column(db.Integer, default=0, server_default="0", nullable=False)

    last_used = db.Column(db.DateTime, index=True, nullable=True)

    user = db.relationship("User")

I'll also define a Marshmallow schema as part of the data model. This will make it easier to accept form submissions in JSON format, as well as return data from the API.

from marshmallow_sqlalchemy import SQLAlchemyAutoSchema, auto_field
from app.models.proxyconn import ProxyConnection


class ProxySchema(SQLAlchemyAutoSchema):

    class Meta:
        model = ProxyConnection
        load_instance = True

    # Set password to load_only so that it is accepted during form
    # submissions, but never dumped back into JSON format.
    password = auto_field(load_only=True)

The SQLAlchemyAutoSchema class is a great convenience, because it automatically maps the model class to Marshmallow fields. When we need to treat a certain field differently, such as password here, it's easy enough to override the functionality.

Whenever new models are created in the project, we need those models to exist as actual tables in Postgres. We'll go through performing database migrations later, but for development purposes, it's easy to create new tables in Postgres using the Flask manager script.

docker exec -it openranktracker_app_1 python manage.py shell
>> db.create_all()

Creating and Deleting Proxy Connections

We're going to need GET, POST, and DELETE methods for the proxy model. Fortunately, this is pretty straightforward, especially because we'll be using Marshmallow to handle validation and serialization.

The ProxiesView handles creating new proxies, as well as returning all proxies belonging to a specific user.

from flask import request, g, abort
from marshmallow import ValidationError

from app.api.auth import AuthenticatedView
from app.models.proxyconn import ProxyConnection
from app.serde.proxy import ProxySchema
from app import db


class ProxiesView(AuthenticatedView):
    def get(self):
        return (
            ProxySchema().dump(
                ProxyConnection.query.filter_by(user_id=g.user.id)
                .order_by(ProxyConnection.id)
                .all(),
                many=True,
            ),
            200,
        )

    def post(self):
        try:
            proxy = ProxySchema().load(request.get_json(), session=db.session)
            proxy.user = g.user
        except ValidationError:
            abort(400)

        db.session.add(proxy)
        db.session.commit()

        return ProxySchema().dump(proxy), 201

We use the global Flask context to filter proxies by user, and to assign an owner to new proxies. The POST method simply returns a 400 Bad Request if the Marshmallow validation fails. This should not happen, however, because the front-end form will have its own validations to prevent bad submissions. More complex validations that can only be done on the back-end are sometimes necessary, but in this case we're concerned only with whether required fields are submitted.

The ProxyView will handle deletion of proxy connections.

from flask import g, abort

from app.api.auth import AuthenticatedView
from app.models.proxyconn import ProxyConnection
from app import db


class ProxyView(AuthenticatedView):
    def delete(self, proxy_id):
        proxy = ProxyConnection.query.get(proxy_id)

        if proxy.user_id != g.user.id:
            abort(403)

        db.session.delete(proxy)
        db.session.commit()

        return "", 200

Pretty simple, really! Unless you're trying to delete proxies that don't belong to you. In that case, we abort with a 403.

Finally, we make a quick stop in app/api/__init__.py to associate the new handlers with API routes.

api.add_resource(ProxyView, "/proxies/<int:proxy_id>/")
api.add_resource(ProxiesView, "/proxies/")

Creating the New Proxy Form

Now that the database models and API routes are in place, we need a form for submitting new proxies. This won't be the first form in the app – after all, we already have sign up and login forms. This time around, however, we're going to get a little fancier, and use the Formik library.

The login and signup forms were very simple. The proxy form, however, has five fields and additional validations beyond whether something is required or not. Handling all of that with Formik should cut down the amount of code we'll need to write.

The first step in building the form will be to define default values, as well as the validations we need to perform. Let's look at the first part of the ProxyPopup.js module to see how that's done.

import { Formik, Form, Field } from "formik";
import * as Yup from "yup";

const defaultProxy = {
    proxy_url: "",
    username: "",
    password: "",
    min_wait_time: 60,
    random_delay: 10
};  

const proxySchema = Yup.object().shape({
    proxy_url: Yup.string().required(),
    username: Yup.string().required(),
    password: Yup.string().required(),
    min_wait_time: Yup.number()
        .positive()
        .integer()
        .required(),
    random_delay: Yup.number()
        .positive()
        .integer()
        .required()
});

The Yup library integrates seamlessly with Formik, and allows you to build up different combinations of validators with ease.

Formik itself provides a base Formik component that expects a function as its child. We'll define our form inside that function, and Formik will pass arguments that include a values object, as well as touched and errors objects.

We can use these objects to drive the styling of the form, as you can see below.

The form relies on the touched and errors objects to flag the username field as an error. The password input isn't flagged, even though it's required, because the touched object indicates that it hasn't experienced a blur event yet. The errors object is updated automatically according to the Yup schema we provided. Formik simplifies tracking all of this state information.

I'll include a sample of the above form here, slightly abbreviated for length.

<Formik
    initialValues={defaultProxy}
    onSubmit={onSubmit}
    validationSchema={proxySchema}
    validateOnMount
>
    {({
        values,
        errors,
        touched,
        handleChange,
        handleBlur,
        handleSubmit,
        isValid
    }) => (
        <Form onSubmit={handleSubmit}>
            <div className="formGroup">
                <label className="formLabel">Proxy URL</label>
                <Input
                    name="proxy_url"
                    onChange={handleChange}
                    onBlur={handleBlur}
                    value={values.proxy_url}
                    border={
                        touched.proxy_url &&
                        errors.proxy_url &&
                        `1px solid ${COLORS.warning}`
                    }
                    style={{ width: "100%" }}
                />
            </div>
            <div className="formGroup">
                <label className="formLabel">
                    Proxy Username
                </label>
                <Input
                    name="username"
                    onBlur={handleBlur}
                    onChange={handleChange}
                    value={values.username}
                    border={
                        touched.username &&
                        errors.username &&
                        `1px solid ${COLORS.warning}`
                    }
                    style={{ width: "100%" }}
                />
            </div>
        </Form>
    )}
</Formik>

You may notice that I'm using custom classes such as Input instead of normal HTML inputs. These are simply convenience classes created using styled components. I have created a handful of these commonly required elements in order to avoid redefining their CSS over and over.

The custom form elements and buttons can be found in the util/controls.js module.

import styled from "styled-components";
import { BORDER_RADIUS, COLORS, PAD_XS, PAD_SM } from "./constants";

export const Input = styled.input`
    color: ${COLORS.fg1};
    background-color: ${COLORS.bg4};
    box-sizing: border-box;
    padding: ${PAD_XS} ${PAD_SM};
    outline: none;
    border-radius: ${BORDER_RADIUS};
    border: ${props => props.border || "none"};
`;

export const Button = styled.button`
    background: none;
    border: none;
    border-radius: ${BORDER_RADIUS};
    outline: none;
    cursor: pointer;

    &:disabled {
        filter: brightness(50%);
        cursor: default;
    }
`;

Building the Proxy Dashboard with Flexbox

We can create new proxies now, but we also need a place to view existing proxies, and monitor their performance.

How many proxies are needed depends on how many keywords we'd like to track, but we can assume it's easily possible to have a dozen or more. We'll use flexbox to create a layout that works as a grid, collapsing eventually to a single column when there isn't much space to work with.

First we'll take a look at the JSX that produces the dashboard.

<div className={styles.container}>
    <div className={styles.buttonRow}>
        <PrimaryButton
            style={{ padding: PAD_SM, marginLeft: "auto" }}
            onClick={addProxyServer}
        >
            Add Proxy Server
        </PrimaryButton>
    </div>
    <div className={styles.proxyList}>
        {proxies.map(proxy => (
        <div key={proxy.id} className={styles.proxyContainer}>
            <ProxyConnection proxy={proxy} onDelete={deleteProxy} />
        </div>
        ))}
    </div>
</div>

The buttonRow div is a flex container that houses the Add Proxy button, which is displayed on the right side of the page. Instead of using float: right here, it's possible to use margin-left: auto to achieve the same result. The proxyList class is also a flex container, of course, but with the flex-wrap property added.

The nowrap default of flex-wrap means items spill outside of their container when there isn't enough space. By changing to wrap, the children are instead allowed to break to the next line.

This is the relevant CSS that makes it all happen.

.container {
    padding: var(--pad-md);
    padding-top: var(--pad-sm);
    box-sizing: border-box;
}

.buttonRow {
    display: flex;
    margin-bottom: var(--margin-md);
}

.proxyList {
    display: flex;
    flex-wrap: wrap;
}

.proxyContainer {
    margin-right: var(--margin-sm);
    margin-bottom: var(--margin-sm);
}

The outer container class applies some padding so that the dashboard isn't pressed to the edges of the page. Using box-sizing: border-box prevents that added padding from creating scrollbars.

Adding a Donut Chart Using D3

If you recall the schema of the proxy table, we're keeping track of how many successful and failed requests each proxy has made. We'll display a donut chart for each proxy as an easy way to see performance at a glance.

The three donut slices represent successful and blocked requests, as well as requests that returned no results (in amber).

We'll create a DonutChart component that works with any kind of data having up to 3 categories. The component expects a category prop that has positive, neutral, and negative keys that map to integer values.

Unlike the vast majority of the app, the DonutChart is a class-based component. This is necessary because D3 works directly with the DOM. As a result, we can't rely on the normal rendering cycle. Instead, we'll have to manually watch for prop changes to determine when a re-render is necessary.

Fortunately, for class-based components we can use componentDidUpdate to determine if a re-render is required.

componentDidUpdate(prevProps) {
    if (prevProps.category != this.props.category) {
        this.drawChart();
    }
}

This is a simple example, but in more complex cases, allows us to have fine-grained control over what happens when props are changed.

The drawChart method contains the actual D3 rendering logic.

drawChart() {
    const svg = d3.select(this.svgRef.current).select("g");

    const radius = Math.min(this.width, this.height) / 2;
    const donutWidth = 10;

    const arc = d3
        .arc()
        .padAngle(0.05)
        .innerRadius(radius - donutWidth)
        .outerRadius(radius)
        .cornerRadius(15);

    const data = [
        this.props.category.POSITIVE,
        this.props.category.NEGATIVE,
        this.props.category.NEUTRAL
    ];

    const pie = d3
        .pie()
        .value(d => d)
        .sort(null);

    // Select all existing SVG path elements and associate them with
    // the positive, neutral, and negative sections of the donut
    // chart.
    const path = svg.selectAll("path").data(pie(data));

    // The enter() and append() methods take into account any existing
    // SVG paths (i.e. drawChart was already called) and appends
    // additional path elements if necessary.
    path.enter()
        .append("path")
        .merge(path)
        .attr("d", arc)
        .attr("fill", (d, i) => {
            return [COLORS.success, COLORS.warning, COLORS.caution][i];
        })
        .attr("transform", "translate(0, 0)");

    // The exit() method defines what should happen if there are more
    // SVG path elements than data elements.  In this case, we simply
    // remove the extra path elements, but we can do more here, such
    // as adding transition effects.
    path.exit().remove();
}

Remember that all of the code is on GitHub if you'd like to use this project as a template for setting up your own visualizations!

What's next?

In part six we'll work on building more visualizations to show ranking progress for the keywords that the user is tracking.

Building a SaaS App (Part IV): User Authentication in Flask and React

zchtodd — Mon, 05 Jul 2021 14:51:34 +0000

Once you've finished this post, you'll have a secure Flask application that handles the user login and sign-up process. As a bonus, we'll tackle not only traditional sign-up, but Google OAuth as well. We'll also introduce React to the series, and incorporate the concept of protected routes into the app.

In the last post, we introduced SQLAlchemy and covered some of the performance pitfalls that you should be aware of. We're going to cover a lot of ground in this post, including authentication on the back-end using Flask, but also how to protect pages that require login using React.

You can find the complete code on GitHub.

Securing a Flask REST API with JSON Web Tokens

We'll use JWTs to authenticate requests to the Open Rank Tracker API. JSON Web Tokens are, as the name implies, a JSON payload that resides either in a cookie or in local storage on the browser. The token is sent to the server with every API request, and contains at least a user ID or other identifying piece of information.

Given that we shouldn't blindly trust data coming from the front-end, how can we trust what's inside a JWT? How do we know someone hasn't changed the user ID inside the token to impersonate another user?

JWTs work because they are given a cryptographic signature using a secret that only resides on the back-end. This signature is verified with every request, and if the contents of the token are altered, the signature will no longer match. As long as the secret is truly secret, then we can verify that what we're receiving is unaltered.

Setting up authentication in Flask

Because we're using class based routes via Flask-RESTful, we can take advantage of inheritance to make protecting API routes simple. Routes that require authentication will inherit from AuthenticatedView, while public facing routes continue to use the Resource base class.

The decode_cookie function will use PyJWT to verify the token and store it in the Flask global context. We'll register the decoding function as a before_request handler so that verifying and storing the token is the first step in the request lifecycle.

from app.services.auth import decode_cookie

def create_app():
    app = Flask(__name__)

    app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
    app.config["SQLALCHEMY_DATABASE_URI"] = create_db_uri()
    app.config["SQLALCHEMY_POOL_RECYCLE"] = int(
        os.environ.get("SQLALCHEMY_POOL_RECYCLE", 300)
    )

    app.config["SECRET_KEY"] = os.environ.get("SECRET_KEY", "placeholder_key")
    app.config["SQLALCHEMY_ECHO"] = False

    app.before_request_funcs.setdefault(None, [decode_cookie])

    create_celery(app)
    return app

The decode_cookie function will run for every request, and before any route handler logic. This step only verifies the token and stores the object on g.cookie – it does not authenticate the user. We'll see that happen later in the require_login function. Below is the implementation for the decode_cookie function.

import os
import logging
import jwt

from flask import g, request, abort

def decode_cookie():
    cookie = request.cookies.get("user")

    if not cookie:
        g.cookie = {}
        return

    try:
        g.cookie = jwt.decode(cookie, os.environ["SECRET_KEY"], algorithms=["HS256"])
    except jwt.InvalidTokenError as err:
        logging.warning(str(err))
        abort(401)

Because this will run for every request, we simply return early if there is no cookie. We call the abort function with a 401 if the token fails to verify, so that the React front-end can redirect the user to the login page.

The require_login function does the actual check against the database. At this point, we've verified the token, and have a user ID extracted from that token. Now we just need to make sure that the user ID matches a real user in the database.

import logging

from flask import make_response, g, abort
from flask_restful import Resource, wraps

from app.models.user import User

def require_login(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        if "id" not in g.cookie:
            logging.warning("No authorization provided!")
            abort(401)

        g.user = User.query.get(g.cookie["id"])

        if not g.user:
            response = make_response("", 401)
            response.set_cookie("user", "")
            return response

        return func(*args, **kwargs)

    return wrapper


class AuthenticatedView(Resource):
    method_decorators = [require_login]

The decorator function also creates g.user so that the User instance is available wherever we might need it. If, for some reason the given ID is not found in the database, then we clear the cookie and send the user back to the login page with a 401.

Handling User Sign-Up in Flask

For this project, I want to walk through both traditional email/password sign-up, as well as using Google OAuth. Having run a SaaS app, I can say from my own experience that doing both worked out well – roughly half of users opted to use the Google OAuth option. Adding that option isn't too difficult, and I believe the convenience offered to the user is worth it.

To get started, let's take a look at the User database model.

from werkzeug.security import generate_password_hash, check_password_hash
from app import db

class User(db.Model):
    __tablename__ = "user"
    __table_args__ = (db.UniqueConstraint("google_id"), db.UniqueConstraint("email"))

    id = db.Column(db.Integer, primary_key=True)

    # An ID to use as a reference when sending email.
    external_id = db.Column(
        db.String, default=lambda: str(uuid.uuid4()), nullable=False
    )

    google_id = db.Column(db.String, nullable=True)
    activated = db.Column(db.Boolean, default=False, server_default="f", nullable=False)

    # When the user chooses to set up an account directly with the app.
    _password = db.Column(db.String)

    given_name = db.Column(db.String, nullable=True)
    email = db.Column(db.String, nullable=True)
    picture = db.Column(db.String, nullable=True)

    last_login = db.Column(db.DateTime, nullable=True)

    @property
    def password(self):
        raise AttributeError("Can't read password")

    @password.setter
    def password(self, password):
        self._password = generate_password_hash(password)

    def verify_password(self, password):
        return check_password_hash(self._password, password)

There are a few important things to note here. Firstly, this class uses property decorators for the password attribute, meaning that while it may look like an attribute on the outside, we're actually calling methods when that attribute is accessed.

Take the following example.

user = User()
user.username = "Bob"
user.password = "PasswordForBob"

Here we set the password, but behind the scenes, the User class is using the one-way hashing function generate_password_hash to create a scrambled version of the password that even we can't unscramble. The real value is stored in the _password attribute. This process ensures that even if an attacker gained access to the database, they would not find any user passwords.

The UniqueConstraint values added to the User class are also worth pointing out. Constraints at the database level are a great way to prevent certain kinds of bugs. Here we're saying it should be impossible to have two users with identical email addresses, or with the same Google ID. We'll also check for this situation in the Flask app, but it's good to have constraints as a fail-safe, in case there's a bug in the Python code.

Email verification and account activation

Creating new users with an email and password (as opposed to Oauth) is fairly straightforward. Most of the work comes from verifying the email address!

I was lazy in the beginning when building my own SaaS and neglected email verification. If you offer any kind of free trial, you're inevitably going to have abuse. I had one individual creating dozens of accounts with fake email addresses. Beyond just abusing your free trial, these users damage your email sending reputation, making it more likely that your emails end up in the spam folder.

Requiring an activation step won't 100% solve this problem, but it will go a long way.

We'll need a way for the app to send email. I'm using the Mailgun API for this project, and set up only takes a few minutes of fiddling with DNS records. Once you have an account with Mailgun and the correct DNS records are in place, sending email only requires a few more steps.

First, we'll update the variables.env and app/init.py files with the necessary config values.

MAIL_DOMAIN
MAIL_SENDER
MAILGUN_API_KEY
POSTGRES_USER
POSTGRES_PASSWORD
POSTGRES_HOST
POSTGRES_DB

If you'll recall from earlier, the variables.env file determines which environment variables are passed from the host machine into the Docker containers. The new values here are MAIL_DOMAIN and MAIL_SENDER, which in my case are mail.openranktracker.com and support@openranktracker.com respectively. The MAILGUN_API_KEY value is used to authenticate your requests to the Mailgun API.

Next we'll update the create_app function to add these new values to the global config dictionary, so that we can access them from anywhere.

app.config["MAILGUN_API_KEY"] = os.environ["MAILGUN_API_KEY"]
app.config["MAIL_SUBJECT_PREFIX"] = "[OpenRankTracker]"
app.config["MAIL_SENDER"] = os.environ.get("MAIL_SENDER")
app.config["MAIL_DOMAIN"] = os.environ["MAIL_DOMAIN"]

Sending an email requires a single API call to Mailgun. We can use the Requests module to make that call, and we'll wrap it all up as a re-usable utility function.

def send_email(to, subject, template, **kwargs):
    rendered = render_template(template, **kwargs)

    response = requests.post(
        "https://api.mailgun.net/v3/{}/messages".format(app.config["MAIL_DOMAIN"]),
        auth=("api", app.config["MAILGUN_API_KEY"]),
        data={
            "from": app.config["MAIL_SENDER"],
            "to": to,
            "subject": app.config["MAIL_SUBJECT_PREFIX"] + " " + subject,
            "html": rendered,
        },
    )

    return response.status_code == 201

Unlike the user interface, which is rendered using React, we'll create the emails with server side rendering via Jinja templates. The app/templates directory will contain all of the email templates, starting with our email verification template. The send_email function accepts extra keyword arguments, which are then passed into render_template, allowing us to have whatever variables we need while rendering the template.

The app/templates/verify_email.html template itself is very basic, but functional.

<p>Please follow the link below in order to verify your email address!</p>

<a href="{{ root_domain }}welcome/activate?user_uuid={{ user_uuid }}">Verify email and activate account</a>

The root_domain variable makes this code independent of the server it's deployed to, so that if we had a staging or test server, it would continue to work there. The user_uuid value is a long string of random letters and digits that identifies users outside of the system – we do this instead of using the primary key because it's best not to rely on an easily enumerated value that an attacker could iterate through.

When building a new template, keep in mind that most email clients support a limited subset of HTML and CSS – designing email templates, even today, will remind you of working with Internet Explorer 6.

Creating the user and sending the activation email

The verification process is kicked off once a user registers with an email and password. They'll have access to the app immediately, but some features will be restricted until the activation step is complete. This will be easy to keep track of thanks to the activated column on the user table.

Let's take a look at the signup.py route handler.

from app.services.user import send_email
from app.serde.user import UserSchema
from app.models.user import User
from app import db


class SignUpView(Resource):
    def post(self):
        data = request.get_json()

        user = User.query.filter(
            func.lower(User.email) == data["email"].strip().lower()
        ).first()

        if user:
            abort(400, "This email address is already in use.")

        user = User()
        user.email = data["email"].strip()
        user.password = data["password"].strip()
        user.last_login = datetime.now()

        db.session.add(user)
        db.session.commit()

        send_email(
            user.email,
            "Account activation",
            "verify_email.html",
            root_domain=request.url_root,
        )

        response = make_response("")
        response.set_cookie(
            "user",
            jwt.encode(
                UserSchema().dump(user), app.config["SECRET_KEY"], algorithm="HS256"
            ),
        )

        return response

This is pretty straightforward, but there are a few important "gotchas" to keep in mind. When checking whether an email is already registered, we're careful to make the comparison case insensitive and strip all white-space. The other point to remember here is that although we store the password in user.password, the plain-text password is never permanently stored anywhere – the one-way hashed value is stored in the _password table column.

The response returned to the client contains their new user details inside a JWT. From there, the front-end will send them to their app dashboard.

Securing pages within the React app

On the front-end side, we'd like to restrict certain pages to logged in users, while redirecting anyone else back to the login or signup area.

The first problem is how to determine whether a user is logged in or not. Because we're storing the JSON web token in a cookie, we'll use the js-cookie library to handle retrieving the cookie, and jwt-decode to parse the token itself. We'll perform a check in src/App.js when the page first loads to determine if the user has a token.

const App = () => {
    const [loadingApp, setLoadingApp] = useState(true);
    const [loggedIn, setLoggedIn] = useState(false);

    /* 
    ** Check for a user token when the app initializes.
    **
    ** Use the loadingApp variable to delay the routes from
    ** taking effect until loggedIn has been set (even logged in
    ** users would be immediately redirected to login page
    ** otherwise).
    */
    useEffect(() => {
        setLoggedIn(!!getUser());
        setLoadingApp(false);
    }, []);

    return (
        <UserContext.Provider value={{ loggedIn, setLoggedIn }}>
            {!loadingApp && (
                <Router style={{ minHeight: "100vh" }}>
                    <Splash path="/welcome/*" />
                    <ProtectedRoute path="/*" component={Home} />
                </Router>
            )}
        </UserContext.Provider>
    );
};

The UserContext is provided at the top level of the app, so code anywhere can determine if the user is currently logged in, and potentially change that state. The ProtectedRoute component simply wraps another component, and prevents that component from loading if the user isn't logged in, instead sending them back to the login page.

If we take a look at ProtectedRoute, we can see that it uses the UserContext to determine if it should load the wrapped component, or redirect to the login page.

const ProtectedRoute = ({ component: Component }) => {
    const { loggedIn } = useContext(UserContext);

    return loggedIn ? (
        <Component />
    ) : (
        <Redirect from="" to="welcome/login" noThrow />
    );
};

Adding Google Oauth as a Sign Up Option

As a bonus, now we'll turn to adding Google Oauth as a sign up and login option. You'll first need to create an account to access Google Developer Console if you haven't already.

After that, you'll need to configure what Google labels as the Oauth consent screen – this is the pop-up that users will see asking them to authorize your app. This step is filled with warnings about manual reviews, but as long as you avoid any sensitive or restricted scopes (i.e. account permissions), your consent screen should be immediately approved. Our app requires the non-sensitive OpenID and email scopes.

After configuring your consent screen, create a new Oauth 2.0 client under the Credentials tab. This is where you'll define your authorized origins and redirect URIs, or in other words, where the Oauth process is allowed to start from, and where the user should return to after interacting with the Google account page.

This is an example of my own settings. You'll also find your Client ID and secret on this page.

The GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET environment vars will need to find their way into variables.env so that the app container can pick them up.

The Flask app has 4 separate endpoints that handle the Oauth flow. The route handlers contained in oauthsignup.py and oauthlogin.py are very simple, and just redirect the browser over to Google while generating a callback URL. The React front-end will do a form submission to one of these, causing the browser to leave our application.

from flask import request, redirect
from flask_restful import Resource

from app.services.auth import oauth2_request_uri


class Oauth2SignUpView(Resource):
    def post(self):
        return redirect(
            oauth2_request_uri(request.url_root + "api/users/oauth2callback/signup/")
        )

Once the user has chosen an account for sign-up or login, they're directed back to our application using the Oauth2 request URI that we generated previously.

The sign-up and login callback handlers are actually very similar, except that during a login attempt the user must already exist. We could easily allow an oAuth login attempt to create a user if none exists, but this leads to confusion, as users forget which email account they used to sign in to the app.

This is the sign-up route handler that will execute when Google redirects the browser back to our domain.

from app.services.auth import get_user_info
from app.serde.user import UserSchema
from app.models.user import User
from app import db


class Oauth2SignUpCallbackView(Resource):
    def get(self):
        oauth_code = request.args.get("code")

        userinfo = get_user_info(oauth_code)
        google_id = userinfo["sub"]

        # Find existing authenticated Google ID or an existing email that the
        # user previously signed up with (they're logging in via Google for
        # the first time).
        user = User.query.filter(
            or_(
                User.google_id == google_id,
                func.lower(User.email) == userinfo["email"].lower(),
            )
        ).first()

        if not user:
            user = User()

        user.google_id = google_id
        user.given_name = userinfo["given_name"]
        user.email = userinfo["email"]
        user.last_login = datetime.now()
        user.activated = True

        db.session.add(user)
        db.session.commit()

        response = redirect(request.url_root)
        response.set_cookie(
            "user",
            jwt.encode(
                UserSchema().dump(user), app.config["SECRET_KEY"], algorithm="HS256"
            ),
        )

        return response

The get_user_info utility function combines the oAuth code returned from Google with our client ID and secret in order to fetch non-sensitive data about the user, including email address and given name.

The route handler also checks the database for an existing user, just to make sure we aren't creating new users when an existing user hits sign-up again for any reason. I've also chosen to sync up non-oAuth users with their Google ID if they should hit "Sign Up with Google" after going through the traditional sign-up process.

Remember that all of the code is on GitHub if you'd like to use this project as an example for setting up oAuth in your own application.

What's next?

In part five, we'll start working on the user dashboard, where we'll display ranking progress for the domains and keywords they are tracking.

Building a SaaS App: Beyond the Basics (Part III)

zchtodd — Sun, 27 Jun 2021 13:31:25 +0000

Once you've finished this post, you'll have a foundation on which to build the data model of your applications, using SQLAlchemy and Postgres.

In the last post, we set up NGINX and Flask using Docker, with both a local development version, as well as a version suitable for production deployment. In this post, we'll set up SQLAlchemy and explore a few of the performance pitfalls that lurk behind the scenes. Then we'll move to setting up our first real route handler, so that the scraper we built in part one can report its results.

You can find the complete code on GitHub.

Setting up SQLAlchemy and Postgres

Back in the first post, we built a working Google search scraper, but we didn't have anywhere to put the results. We're going to fix that problem now with the help of SQLAlchemy – by far the most popular ORM library for Python.

If you haven't used one before, using an ORM will allow us to work in terms of objects, instead of working with messy raw SQL strings in the Python code. Luckily, setting up SQLAlchemy to work with a Flask application is very straightforward, thanks to the Flask-SQLAlchemy package.

The app/__init__.py file contains all of the configuration necessary to get started.

from flask import Flask
from flask_sqlalchemy import SQLAlchemy

from app.util import create_db_uri

db = SQLAlchemy()

def init_app(app):
    db.init_app(app)
    return app

def create_app():
    app = Flask(__name__)
    app.config["SQLALCHEMY_DATABASE_URI"] = create_db_uri()
    app.config["SQLALCHEMY_ECHO"] = False

    return app

from app.models import *  # noqa

This is a reduced version of the init file containing just the minimum needed to set up Flask-SQLAlchemy. The config value SQLALCHEMY_DATABASE_URI tells Flask-SQLAlchemy how to connect to the database. This ultimately depends on the environment variables we saw in Part 2, such as POSTGRES_USER and POSTGRES_HOST.

The SQLALCHEMY_ECHO value is useful when debugging – when set to true, every SQL statement is logged, so you can see what's happening at every step. We'll see a lot of the global db variable throughout the application, because we'll import it wherever we need to interact with the database.

You might also notice the seemingly odd import at the bottom of the file, but it serves an important purpose. As you'll see soon, each of our models resides in its own file. Until a model is imported, SQLAlchemy won't know that it exists, even though we created the definition. Thus, the wildcard import at the bottom ensures that all of our models are imported at runtime.

Defining a model is easy. Model classes inherit from db.Model and define the columns, indexes, and constraints that belong to that model.

from app import db

class ProxyConnection(db.Model):
    __tablename__ = "proxyconn"

    id = db.Column(db.Integer, primary_key=True)

    proxy_url = db.Column(db.String, nullable=False)
    username = db.Column(db.String, nullable=False)
    password = db.Column(db.String, nullable=False)

    allow_parallel = db.Column(
        db.Boolean, default=False, server_default="f", nullable=False
    )

    usage_count = db.Column(db.Integer, default=0, server_default="0")
    block_count = db.Column(db.Integer, default=0, server_default="0")
    consecutive_fails = db.Column(db.Integer, default=0, server_default="0")

    engaged = db.Column(db.Boolean, default=False, server_default="f")

    min_wait_time = db.Column(db.Integer, default=0, server_default="0", nullable=False)
    random_delay = db.Column(db.Integer, default=0, server_default="0", nullable=False)
    last_used = db.Column(db.DateTime, index=True, nullable=True)

As we discussed in the first part of the series, we'll need to use proxy connections for the scraper – we'll keep track of those proxies in the database, as well as how they are performing. We can set a threshold, for instance, so that if a proxy has a certain number of consecutive_fails we take it out of rotation. The project will eventually have many tables, with a model for everything from users to ranking results.

At the moment, however, the database is empty. We need to create the tables defined in our models. To do that, we can use manage.py to create an interactive shell. This shell session is almost the same as an interactive Python session, but within the context of the Flask application.

docker exec -it openranktracker_app_1 python manage.py shell
>>> db.create_all()

The shell makes the db global available, and the create_all function will initialize the entire schema within Postgres. Once that step is complete, you can verify that the tables were created using a psql session.

docker exec -it openranktracker_database_1 psql -U pguser -d openranktracker

psql (11.4 (Debian 11.4-1.pgdg90+1))
Type "help" for help.

openranktracker=# \d
               List of relations
 Schema |       Name       |   Type   | Owner  
-------------+------------------+----------+--------
 public | domain           | table    | pguser
 public | domain_id_seq    | sequence | pguser
 public | keyword          | table    | pguser
 public | keyword_id_seq   | sequence | pguser
 public | proxyconn        | table    | pguser
 public | proxyconn_id_seq | sequence | pguser
(6 rows)

openranktracker=#

The tables have been created! Now we just need to put some data in there.

SQLAlchemy performance pitfalls to avoid

We want our users to have a fast and responsive experience with the application. In my experience, the most common source of slowness is mistakes made with the ORM. SQLAlchemy allows for great convenience and speed of development, but it's easy to lose track of what's happening behind the scenes.

So before moving on, let's cover some of the biggest performance killers, and how to avoid them in our application.

The N+1 Problem

Relationship loading is one of the killer features of any ORM. Instead of manually writing SQL, we can treat data more like objects and object properties. To take an example, think of artists, their albums, and the songs that make up those albums.

This would be a fairly typical way to represent such a problem.

class Artist(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String)

class Album(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String)

    artist_id = db.Column(
        db.Integer,
        db.ForeignKey("artist.id", ondelete="CASCADE"),
        index=True,
        nullable=False,
    )

    artist = db.relationship(
        Artist,
        backref=db.backref(
            "albums", order_by="Album.name", cascade="delete-orphan,all"
        ),
    )

class Song(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String)

    album_id = db.Column(
        db.Integer,
        db.ForeignKey("album.id", ondelete="CASCADE"),
        index=True,
        nullable=False,
    )    

    album = db.relationship(
        Album,
        backref=db.backref(
            "songs", order_by="Song.name", cascade="delete-orphan,all"
        ),
    )

Now say we wanted to show a big list of all artists, their albums, and songs in one place. Because of the relationships that we have defined, this would be one possible way of doing that.

result = []
for artist in Artist.query.all():
    for album in artist.albums:
        for song in album.songs:
            result.append((artist.name, album.name, song.name))

This may seem innocent enough, and in fact might work perfectly well with a small amount of data, but there is a lot happening behind the scenes. After setting SQLALCHEMY_ECHO to true, we see a surprising number of queries sent to the database for such a simple program.

[2021-06-26 17:03:27,602] INFO in log: SELECT artist.id AS artist_id, artist.name AS artist_name 
FROM artist
[2021-06-26 17:03:27,607] INFO in log: SELECT album.id AS album_id, album.name AS album_name, album.artist_id AS album_artist_id 
FROM album 
WHERE %(param_1)s = album.artist_id ORDER BY album.name
[2021-06-26 17:03:27,610] INFO in log: SELECT song.id AS song_id, song.name AS song_name, song.album_id AS song_album_id 
FROM song 
WHERE %(param_1)s = song.album_id ORDER BY song.name

And this is just with one artist and a single album! For each new artist or album, you can expect to see another query. The N+1 problem refers to the idea that an ORM is constantly issuing yet another query for each related object that you want to load.

This is a real problem, because each database round-trip quickly adds up, and that means our user is growing impatient as they stare at a loading spinner.

There are many variations on this problem, but the general idea is that we should aim to accomplish the task with fewer queries to the database. This isn't always worth obsessing over, especially if we're certain that the amount of data is always going to remain limited. When we know that a serious amount of data is expected, however, it's worth thinking in terms of how many queries are involved.

How can we speed up the code we saw above?

There is no one absolute best answer, but instead solutions that fit different scenarios. If we know, for instance, that we're almost always going to fetch artists, albums, and songs together, then we can change how the relationships are loaded.

    artist = db.relationship(
        Artist,
        backref=db.backref(
            "albums", order_by="Album.name", lazy="joined", cascade="delete-orphan,all"
        ),
    )

Adding lazy="joined" instructs SQLAlchemy to always load the related data upfront by issuing a query with a JOIN. This means fewer queries to the database, because the data is already available when accessing the relationship attributes.

[2021-06-26 17:21:44,224] INFO in log: SELECT artist.id AS artist_id, artist.name AS artist_name, album_1.id AS album_1_id, album_1.name AS album_1_name, album_1.artist_id AS album_1_artist_id, song_1.id AS song_1_id, song_1.name AS song_1_name, song_1.album_id AS song_1_album_id 
FROM artist LEFT OUTER JOIN album AS album_1 ON artist.id = album_1.artist_id LEFT OUTER JOIN song AS song_1 ON album_1.id = song_1.album_id ORDER BY album_1.name, song_1.name

Now all of the albums and songs are loaded alongside the artist data, and in a single query, instead of separate queries that waste round-trip time. Of course, if the relationships are rarely loaded, this becomes wasteful, because we're asking the database to do this extra work regardless.

You can also do the join yourself, which makes sense when always joining up-front is overkill.

result = Artist.query.join(Album, Song).all()

Calling commit too many times

Understanding when to call commit is also important. Performance and data integrity are two key reasons to call commit at the proper point in the code. Commit marks all of your changes as permanent (i.e. visible outside of your current transaction), and does so by forcing all of your updates to disk.

You want your changes persisted to disk, but calling commit multiple times repeats this process unnecessarily. Call commit only once, typically once you're done making all changes. This is usually simple in a web application, where you should expect to see commit() called near the end of a route handler.

Great convenience, but with caution required

SQLAlchemy brings great convenience, but also the ability to shoot yourself in the foot. Be mindful of what's happening in the background, and just because some code works fine now, doesn't ensure it won't grind to a halt later when flooded with real data.

Setting up our first API route handler

Now that SQLAlchemy is set up, we're almost ready to start storing some real data. The scraper agent from part one is already collecting ranking data, so let's build an API endpoint that can store those results.

First we'll make a slight modification of app/__init__.py to register the Flask blueprint that represents the API. A blueprint is a Flask concept that allows for endpoints sharing a common prefix (i.e. /api in our case) to be grouped together.

def init_app(app):
    db.init_app(app)

    from app.api import api_blueprint

    app.register_blueprint(api_blueprint)

    return app

Importing the api_blueprint object within the init_app function prevents circular import issues, since the code within those endpoints will need to import the global db object.

We'll need a place to wire up routes to their respective handlers, and app/api/__init__.py is where that happens. We'll start off with just one route, for handling data coming back from the Puppeteer search scraper.

from flask import Blueprint
from flask_restful import Api

from app.api.keywords.scan_callback import ScanCallbackView

api_blueprint = Blueprint("main", __name__, url_prefix="/api")
api = Api(api_blueprint)

api.add_resource(ScanCallbackView, "/keywords/<int:keyword_id>/callback/")

The <int:keyword_id> in the URL path is a placeholder for a variable that we expect to receive. The value will be passed along to the handler method, as we'll see in the next snippet of code.

from flask import request, abort
from flask import current_app as app

from flask_restful import Resource
from app.services.keyword import handle_scraper_response


class ScanCallbackView(Resource):
    def post(self, keyword_id):
        data = request.get_json()

        app.logger.debug("Keyword scan callback initiated")

        if data.get("secret_key") != app.config["SECRET_KEY"]:
            app.logger.warning(
                "Scan callback did not provide correct secret key: {}".format(
                    data.get("secret_key")
                )
            )
            abort(403)

        handle_scraper_response(keyword_id, data)
        return "", 201

This project uses Flask-RESTful, so the handlers are class based instead of functions – this allows us to handle GET, PUT, POST, and DELETE a little more elegantly. The scraper sends a POST request, and the handler extracts the data via request.get_json() before processing the results within the handle_scraper_response function.

We haven't covered users or authentication yet, so how do we prevent abuse of this endpoint? The app has a SECRET_KEY config value, which it will pass to scraper sessions. When a scraper sends its POST request, it will include that same secret key for verification.

That's all it takes to add a route handler! At this point we have some real functionality: the scraper runs, collects results, and now has a place to report those results.

What's next?

In part four, we'll break ground on the user interface for OpenRankTracker using React. We'll start off with the user sign-up and login flow and build from there! I hope you'll join me for part four.

Building a SaaS App: Beyond the Basics (Part II)

zchtodd — Thu, 24 Jun 2021 20:23:02 +0000

By the end of this post, you will have a deployable app that is ready to serve real users efficiently and securely!

In the last post, we built out the Puppeteer script that will do the actual scraping. In this post, we're going to focus on infrastructure – namely, how to set up and deploy the application.

For this project, I'm using NGINX, Flask, and Postgres on the back-end. We'll be using React for the front-end. Docker and Docker Compose will make it easier to deploy anywhere.

You can find the complete code on GitHub.

Setting up Docker and Docker Compose

A real SaaS app will be deployed to many environments: developer laptops, a staging environment, and a production server, to name just a few. Docker makes this both an easier and more consistent process.

Docker Compose orchestrates multiple containers, so that we can manage the entire application reliably. That orchestration is limited, however, to one host. Many apps will never need to scale beyond one host, but options like Kubernetes exist should your app become that successful!

To get started, we'll need to have Docker and Docker Compose installed on the host.

curl -fsSL https://get.docker.com -o get-docker.sh # Download install script.
sudo chmod u+x ./get-docker.sh # Make script executable.
sudo ./get-docker.sh 
sudo usermod -aG docker $USER # Add current user to the docker group.
newgrp docker # Reload groups so that changes take effect.

Docker should now be installed. Use docker ps to verify that Docker is installed correctly. You should see something like this.

ubuntu@ip-172-31-38-160:~$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Installing Compose is fairly straightforward as well.

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Deploying the development version

Now that Docker is installed, we can jump straight to starting the application. Use Git to clone the repository if you haven't already.

Once the repository is cloned, you can start up the application simply by running docker-compose up -d and waiting for the images to download and build. Docker will pull the NGINX and Postgres images, as well as build the image for the app container.

You can run docker ps after the image building and downloading is complete. The output should be similar to the below.

CONTAINER ID   IMAGE                 COMMAND                  CREATED          STATUS          PORTS                                       NAMES
0cc1d1798b49   nginx                 "/docker-entrypoint.…"   4 seconds ago    Up 3 seconds    0.0.0.0:80->80/tcp, :::80->80/tcp           openranktracker_nginx_1
eb3679729398   open-rank-tracker     "python tasks.py wor…"   51 seconds ago   Up 49 seconds                                               openranktracker_app-background_1
ab811719630a   open-rank-tracker     "gunicorn --preload …"   51 seconds ago   Up 49 seconds                                               openranktracker_app_1
df8e554d7b12   postgres              "docker-entrypoint.s…"   52 seconds ago   Up 50 seconds   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   openranktracker_database_1
68abe4d03f62   redis:5.0.4-stretch   "docker-entrypoint.s…"   52 seconds ago   Up 50 seconds   6379/tcp                                    openranktracker_redis_1

If you've never used Docker before, then this might seem like magic, but the Dockerfile and docker-compose.yml files contain all of the relevant details. The first contains instructions for building the Flask API container, and the second specifies all of the images that make up the application.

You may notice that we have docker-compose.yml as well as docker-compose.prod.yml. This is how we'll manage the differences in deployment between development and production versions. There are typically several important differences between environments, such as how SSL certificates are handled.

Understanding how NGINX and Flask work together

Although Flask has its own built-in web server, we'll use NGINX to process requests from the user. The Flask web server is meant only for development purposes and serves requests using a single thread, making it unsuitable for our API, and especially unsuitable for serving static files.

NGINX acts as a proxy, forwarding API requests over to Flask. We'll use Gunicorn to overcome our single-threaded Flask issue. Gunicorn manages a pool of processes, each running its own instance of Flask and load balancing between them. This may sound complicated, but the setup is managed within just a few small files.

Let's take a look at how nginx.conf is configured first.

worker_processes 4;

events { worker_connections 1024; }

http {
    include /etc/nginx/mime.types;

    server {
        listen 80;
        listen [::]:80;

        location / {
            root /static;
            try_files $uri $uri/ /index.html;

            add_header Cache-Control "no-cache, public, must-revalidate, proxy-revalidate";
        }

        location /api {
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Host $host;
            proxy_pass http://unix:/sock/app.sock:/api;
        }
    }
}

The server block tells NGINX to listen on port 80, while the location blocks define what should happen when a request URL matches a certain pattern. The order of location blocks is important – the first block can match any request, but the second block is more specific and applies to requests starting with /api as their path.

The second location block forwards the request to Flask by using the proxy_pass directive. The http://unix:/sock/ means that the network traffic will be over a Unix domain socket. The app.sock is a file that is shared between NGINX and Flask – both read and write from this domain socket file to communicate. Lastly, :/api means that the receiving side, Flask, should get requests prefixed with /api.

The X-Forwarded-Proto component will become important later when we introduce SSL in our production configuration. This directive will cause NGINX to proxy requests with the same protocol, so if a request was made over HTTPS, then Flask will receive that same request over HTTPS. This is important when implementing features like signing in with Google, because OAuth libraries require that every request be made over SSL.

Now let's take a look at the section of the docker-compose.yml file that defines how NGINX and Flask are deployed.

version: '3'

volumes:
    sock:

services:
    nginx:
        image: nginx
        restart: always
        volumes:
            - ./nginx.conf:/etc/nginx/nginx.conf
            - sock:/sock
        ports:
            - "80:80"

    app:
        command: gunicorn --preload --bind=unix:/sock/app.sock --workers=6 wsgi
        restart: always
        image: open-rank-tracker
        build: .
        volumes:
            - sock:/sock

The most relevant part here is the sock volume definition. By declaring sock as a top-level volume, we can share it between NGINX and Flask, allowing them to use it as a Unix domain socket.

Testing the NGINX and Flask configuration

We don't have to wait until we're building the UI to test whether this configuration is working or not. You can test this deployment using a browser, or even a simple command-line program like curl.

Because we haven't touched on the UI yet, we'll need to create a basic index.html file before we can really do any testing. Create an index.html file under the static directory within the project root.

sudo touch static/index.html
sudo bash -c 'echo "Hi, world" > static/index.html'
curl http://localhost

Using curl or going to http://localhost (or to the IP of your server if deployed elsewhere) in your browser should show Hi, world in response. This means the request matched the first location block in nginx.conf – in fact, any request you send that doesn't start with /api should return Hi, world at this point.

If you try going to http://localhost/api in your browser, you'll see the Flask 404 page instead. We haven't defined any routes in Flask yet, so the 404 is expected, but we know that NGINX and Flask are configured properly at this point.

Postgres Configuration

Setting up Postgres with Docker is fairly simple. I'll show you the docker-compose.yml configuration below, and walk through a few of the most important sections.

database:
    image: postgres
    restart: always
    volumes:
       - /var/lib/postgres:/var/lib/postgres
    expose:
       - 5432
    env_file:
       - variables.env

We name the service database, which is important, because that's the host name other containers can use to connect with Postgres. The volumes directive maps a directory on the host to a matching directory within the container, so that if the container is stopped or killed, we haven't lost the data.

The expose directive allows other containers access on port 5432, but does not allow access outside of the Docker network. This is an important distinction for security purposes. We could also use the ports directive, which would allow access to 5432 from the Internet. This can be helpful if you want to connect remotely, but at that point your Postgres password is the only thing preventing the entire world from gaining access.

Finally, the env_file tells Compose where to look for environment variables. These variables are then passed into the container. The Postgres image has just one required environment variable – POSTGRES_PASSWORD that must be defined, but we'll define a few others as well.

POSTGRES_USER
POSTGRES_PASSWORD
POSTGRES_HOST
POSTGRES_DB

Because they're listed without values in variables.env, each variable takes its value from the host environment. You can also hard code values inside the config file, but it's better to keep them out of source control, especially with values such as passwords or API keys.

Let's test out connecting to the Postgres instance using the psql command-line program. First, find the ID of the Postgres container using docker ps, and then we'll connect locally using docker exec.

docker exec -it ba52 psql -U pguser -d openranktracker

If all goes well, you'll be greeted with the Postgres interactive shell prompt.

Setting up SSL with Let's Encrypt

We'll need to set up SSL certificates via Let's Encrypt before we can deploy the production version of the app. This is a quick process that involves proving to Let's Encrypt that you are the owner of the server, after which they will issue certificate files.

You'll need a domain name before obtaining a certificate. I'm using Google Domains, but any domain registrar should work.

Installing the certbot agent is the first step in the process.

sudo apt-get install -y certbot

Now we can request a certificate, but first make sure that port 80 is available – if the app is running, be sure to stop it first so that certbot can use port 80.

sudo certbot certonly --standalone --preferred-challenges http -d openranktracker.com

Of course, you should replace openranktracker.com with your own domain name. Certificates are valid for 90 days, after which a simple renewal process is required. We'll go through setting up an automated renewal process a bit later.

Deploying the production version

What we've set up so far is great for local development on a laptop. In the real world, however, our app should at least have SSL enabled. Luckily, it's not hard to go that extra step for our production configuration.

We'll take advantage of a Compose technique known as stacking to make the configuration change as simple as possible. Instead of having to redefine everything in the separate docker-compose.prod.yml file, we only need to specify what is different, and those sections will take precedence.

version: '3'

services:
    nginx:
        image: nginx
        restart: always
        volumes:
            - /etc/letsencrypt:/etc/letsencrypt
            - ./nginx.prod.conf:/etc/nginx/nginx.conf
            - ./static:/static
            - sock:/sock
        ports:
            - "443:443"
            - "80:80"

This file contains only the NGINX service, because the configuration for the app and database remain the same. The volumes section exposes the Let's Encrypt certificate to the NGINX container, and the modified nginx.prod.conf makes use of the certificate to serve the application over HTTPS.

Let's take a look at the nginx.prod.conf file to see how SSL is handled.

worker_processes 4;

events { worker_connections 1024; }

http {
    include /etc/nginx/mime.types;

    server {
        listen 80;
        listen [::]:80;
        server_name _;
        return 301 https://$host$request_uri;
    }

    server {
        listen 443 ssl default_server;

        ssl_certificate /etc/letsencrypt/live/openranktracker.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/openranktracker.com/privkey.pem;

        location / {
            root /static;
            try_files $uri $uri/ /index.html;

            add_header Cache-Control "no-cache, public, must-revalidate, proxy-revalidate";
        }

        location /api {
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Host $host;
            proxy_pass http://unix:/sock/app.sock:/api;
        }
    }
}

This should look mostly familiar, except we now have two server blocks: one listens on port 80 and redirects traffic to port 443, while the other listens on 443 and serves the app as well as static files. If you try going to the HTTP version, your browser should be immediately redirected to the HTTPS version.

We'll use a stacked command with Compose to bring up the app with this configuration.

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

And presto! You should now have NGINX serving requests with SSL enabled.

What's next?

I hope you liked the second part of the SaaS app series! Up next, we'll start building the data model for the application, and set up the first route handler, so that the scraper we built in part one has a place to report its results.

Building a SaaS App: Beyond the Basics

zchtodd — Mon, 21 Jun 2021 20:28:04 +0000

This is the first post in a series on building your own SaaS application. We'll go step by step through what it takes to build a real product: taking payments, system monitoring, user management, and more.

So what kind of product are we going to build?

We're going to build a fully functioning (if minimal) Google rank tracker.

Enter a domain, some keywords, and the app will track performance on Google search over time. Does this idea make business sense? Probably not! But it's a fun idea that does something useful, it's a task we can accomplish, and you can take it as far as you like. We'll cover all the fundamentals of building a SaaS app along the way.

You can find the complete code on GitHub.

Building the Google Search scraper

Scraping Google search results is the core of this application. Although we could start building just about anywhere, I think beginning with the scraper itself makes sense.

The scraper should take a search query and load several pages of results. The scraper will then return those results to our app. That sounds so simple! But a lot can go wrong in-between. Because we don't want irate emails from unhappy customers, a great deal of the code will be dedicated to handling failures.

Setting up Puppeteer on an AWS instance

We'll use Puppeteer to do the scraping. Puppeteer provides a JavaScript API for remotely controlling a Chromium browser session. Best of all, the browser can run without a desktop environment (headless mode), so our code can execute independently on a server in the cloud. For this tutorial, we'll start with an Ubuntu 18.04 instance on AWS, and step through installing all of the dependencies needed for Puppeteer.

I'm using an EC2 tc2.medium instance for this project. This comes with 2 vCPUs and 4GB of RAM, so it's powerful enough to run Puppeteer, as well as what we're going to add later. An Ubuntu 18.04 instance is a good starting point.

Chromium comes bundled with Puppeteer, but there are a wide array of prerequisite system libraries that are needed before we can get started. Luckily, we can get all of that installed with this one liner.

sudo apt-get install -y ca-certificates fonts-liberation libappindicator3-1 libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 lsb-release wget xdg-utils

Once the Chromium dependencies are installed, we can move on to setting up Node v14. The simplest way to do this is via a downloadable setup script, which will tell our package manager how to find v14 of Node, instead of the much older version that it's already pointing to.

curl -sL https://deb.nodesource.com/setup_14.x -o nodesource_setup.sh
bash nodesource_setup.sh
apt-get install -y nodejs

At this point, we have Node and Chromium installed. Next we'll create a package.json file so that we can use NPM to install project dependencies (i.e. Puppeteer).

{
    "name": "agent-function",
    "version": "0.0.1",
    "dependencies": {
        "axios": "^0.19.2", // For communicating with the app server.
        "puppeteer": "10.0.0",
        "puppeteer-extra": "3.1.8",
        "puppeteer-extra-plugin-stealth": "2.7.8"
    }
}

After running npm install, you should have all the necessary pieces in place. Let's use a very simple Node script to verify that Puppeteer is installed and working.

const puppeteer = require("puppeteer-extra");

async function crawl() {
    console.log("It worked!!!");
}

puppeteer
    .launch({
        headless: true,
        executablePath:
            "./node_modules/puppeteer/.local-chromium/linux-884014/chrome-linux/chrome",
        ignoreHTTPSErrors: true,
        args: [
            "--start-fullscreen",
            "--no-sandbox",
            "--disable-setuid-sandbox"
        ]
    })
    .then(crawl)
    .catch(error => {
        console.error(error);
        process.exit();
    });

Notice the headless key in the config object. This means Chromium will launch without a GUI, which is what we want when running on a server in EC2. Hopefully, if all goes well, you'll see It worked!!! print to the console when you execute this script.

Making a simple Google search request

Now that we know everything is correctly installed, we should start with doing a simple Google search. We won't bother with any actual scraping at this point. The goal is simply to type a search query into the search bar, load the Google results, and take a screenshot to prove that it worked.

This is the crawl function after updating it to do what I just described.

async function crawl(browser) {
    const page = await browser.newPage();
    await page.goto("https://www.google.com/?hl=en");

    // Find an input with the name 'q' and type the search query into it, while 
    // pausing 100ms between keystrokes.
    const inputHandle = await page.waitForXPath("//input[@name = 'q']");
    await inputHandle.type("puppeteer", { delay: 100 });

    await page.keyboard.press("Enter");
    await page.waitForNavigation();

    await page.screenshot({ path: "./screenshot.png" });
    await browser.close();
}

Puppeteer loads the Google search page (adding hl=en to request the English version), enters the search query, and presses enter.

The waitForNavigation method pauses the script until the browser emits the load event (i.e. the page and all of its resources, such as CSS and images, have loaded). This is important, because we'd like to wait until the results are visible before we take the screenshot.

Hopefully you'll see something similar in screenshot.png after running the script.

Using a proxy network for scraper requests

Odds are good, however, that even if your first request was successful, you'll eventually be faced with a CAPTCHA. This is pretty much inevitable if you send too many requests from the same IP address.

The solution is to route requests through a proxy network to avoid triggering CAPTCHA blocks. The scraper will always be blocked from time to time, but with any luck, the majority of our requests will make it through.

There are many different types of proxies, and a huge number of vendor options. There are primarily three options for a scraping project like this.

Purchasing a single IP address, or a bundle of IP addresses, through a service like Proxyall. This is the lowest cost option. I purchased 5 IP addresses for about $5/month.
Data-center proxies that provide a wide range of IP addresses, but charge for bandwidth. Smartproxy, as an example, provides 100GB for $100. Many of these IP addresses, however, are already blocked.
Residential proxies also provide a wide range of IP addresses, but the addresses come from a residential or mobile ISP, and so will encounter CAPTCHA less frequently. The trade-off comes in price. Smartproxy charges $75 for 5GB of data transfer.

You may be able to get away with no proxy if your scraper works very slowly and makes infrequent requests. I actually want to track rankings for my own site, so going with a handful of dedicated IP addresses made sense.

Sending requests over the proxy, instead of the default network, is straightforward with Puppeteer. The start-up args list accepts a proxy-server value.

puppeteer
    .launch({
        headless: false,
        executablePath:
            "./node_modules/puppeteer/.local-chromium/linux-884014/chrome-linux/chrome",
        ignoreHTTPSErrors: true,
        args: [
            `--proxy-server=${proxyUrl}`, // Specifying a proxy URL.
            "--start-fullscreen",
            "--no-sandbox",
            "--disable-setuid-sandbox"
        ]
    })

The proxyUrl might be something like http://gate.dc.smartproxy.com:20000. Most proxy configurations will require a username and password, unless you're using IP white-listing as an authentication method. You'll need to authenticate with that username/password combination before making any requests.

async function crawl(browser) {
    const page = await browser.newPage();
    await page.authenticate({ username, password });
    await page.goto("https://www.google.com/?hl=en");
}

Any heavily used scraper is still going to experience getting blocked, but a decent proxy will make the process sustainable, as long as we build in good error handling.

Gathering the Search results

We turn now to the actual scraping part of the process. The overall goal of the app is to track rankings, but for simplicity's sake, the scraper doesn't care about any particular website or domain. Instead, the scraper simply returns a list of links (in the order seen on the page!) to the app server.

To do this, we're going to rely on XPath to select the correct elements on the page. CSS selectors are often not good enough when it comes to complex scraping scenarios. In this case, Google doesn't offer any easy ID or class name that we can use to identify the correct links. We'll have to rely on a combination of class names, as well as tag structure, to extract the correct set of links.

This code will extract the links and press the Next button a predetermined number of times, or until there is no more Next button.

let rankData = [];
while (pages) {
    // Find the search result links -- they are children of div elements
    // that have a class of 'g', while the links themselves must also
    // have an H3 tag as a child.
    const results = await page.$x("//div[@class = 'g']//a[h3]");

    // Extract the links from the tags using a call to 'evaluate', which
    // will execute the function in the context of the browser (i.e. not
    // within the current Node process).
    const links = await page.evaluate(
        (...results) => results.map(link => link.href),
        ...results
    );

    const [next] = await page.$x(
        "//div[@role = 'navigation']//a[descendant::span[contains(text(), 'Next')]]"
    );

    rankData = rankData.concat(links);

    if (!next) {
        break;
    }

    await next.click();
    await page.waitForNavigation();

    pages--;
}

Now that we have the search results, how do we get them out of the Node process and back to somewhere to be recorded?

There are a lot of ways to do this, but I chose to have the app make an API available for the scraper, so that it can send the results as a POST request. The Axios library makes this pretty easy, so I'll share what that looks like here.

    axios
        .post(`http://172.17.0.1/api/keywords/${keywordID}/callback/`, {
            secret_key: secretKey,
            proxy_id: proxyID,
            results: rankData,
            blocked: blocked,
            error: ""
        })
        .then(() => {
            console.log("Successfully returned ranking data.");
        });

Don't worry about the blocked or error variables here. We'll get into error handling in a moment. The most important thing here is the rankData variable, which refers to the list containing all of the search result links.

Scraper error handling

Handling the unexpected is important in any kind of programming, but especially so with a scraper. There is a lot that can go wrong: running into a CAPTCHA, proxy connection failures, our XPath becoming obsolete, general network flakiness, and more.

Some of our error handling will come later, because we can only do so much within the scraper code itself. The app will need to be smart enough to know when it should retry, or if it should retire a certain proxy IP address because it's getting blocked too frequently.

If you'll recall from earlier, the scraper returns a blocked value. Let's take a look at how we determine whether the scraper has been blocked.

    let blocked = false;

    try {
        const [captcha] = await page.$x("//form[@id = 'captcha-form']");
        if (captcha) {
            console.log("Agent encountered a CAPTCHA");
            blocked = true;
        }
    } catch (e) {}

This code simply looks for the presence of a form with the ID captcha-form and sets the blocked value to true if so. As we'll see later, if a proxy IP is reported as blocked too many times, the app will no longer use that IP address.

What's next?

I hope you've enjoyed this first part of the SaaS app series! Up next, I'll go through setting up NGINX, Flask, and Postgres using Docker, so that our scraper has an API to call. You can always find the complete code for the project on GitHub.

7 Tips for Building an Amazon Scraper

zchtodd — Wed, 16 Jun 2021 11:59:55 +0000

In this post I want to share some lessons learned while scraping Amazon product pages. For this project, I used the Scrapy framework to get a huge head start, but there were still plenty of pitfalls to overcome.

If you want to go straight to the code, you can find it on GitHub, along with instructions for setting it up.

My goal was to scrape thousands of product detail pages across several categories. The scraper stores product info in a PostgreSQL database. This data includes the name and price of the product, but also covers unstructured metadata that varies by category, such as CPU/RAM specs in the Computer category.

So, what do I wish I'd known at the beginning?

Build a spider to handle each product category

Starting out with a single spider class seemed like the natural choice. If you spend any time jumping between product categories, however, you'll soon realize that many of them have their own distinct layout.

Even when two product categories seem similar, their markup can have subtle differences. The path to madness lies in maintaining a single spider class that is aware of all these variations. Pretty soon, fixing a bug in one product category just introduces new bugs for other categories.

It's better to think of Amazon as many separate, if similar, websites. This may lead to some code duplication across spiders, but it makes each spider class much cleaner and more focused. Each product category is a start_url in Scrapy terms, making it easy to split up the crawling between different spider classes.

    start_urls = {
        "https://www.amazon.com/Exercise-Equipment-Gym-Equipment/b?ie=UTF8&node=3407731": ExerciseEquipmentSpider,
        "https://www.amazon.com/computer-pc-hardware-accessories-add-ons/b?ie=UTF8&node=541966": ComputerPCHardwareSpider,
    }

    for start_url, spider_class in start_urls.items():
        crawler.crawl(
            spider_class, start_urls=[start_url], allowed_domains=["amazon.com"],
        )
        crawler.start()

Use a residential proxy network

Like many big websites, Amazon has countermeasures in place to prevent scraping.

An IP address that sends thousands of requests will quickly end up on a blacklist. You'll need to route your requests through a proxy network to scrape at scale.

Luckily, setting this up with Scrapy is easy. Registering a middleware class that adds a proxy value to the request metadata is all it takes to route every request through a proxy.

class CustomProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = os.environ["PROXY_URL"]

Proxy networks mostly come in two flavors: Data-center and Residential. Roughly speaking, data-center proxies are fast and cheap, while residential networks are slower and more expensive. Although a data-center proxy was my first choice, I soon found that many IP addresses are already blacklisted.

I had far better luck with a residential proxy network. Your mileage may vary with different proxy vendors, but in my case using a data-center proved impractical.

Rely on Scrapy middleware to filter requests

Preventing Scrapy from going off the rails and crawling unnecessary pages was a bigger part of the challenge than I expected. On a big website like Amazon, it's very easy for Scrapy to follow one wrong link into oblivion.

Category pages have very complex markup, making it difficult to come up with a selector that will target only the correct links. XPath is flexible enough to do the job, but like regular expressions, quickly become unreadable. Instead of forcing the selectors to do all the work, I used a middleware class to filter out unwanted requests.

The builtin IgnoreRequest exception can be raised to avoid making a request. I quickly built up a list of "rabbit hole" links that were getting queued up and causing my crawl to ride off into the sunset instead of visiting product pages.

import os

from scrapy.linkextractors import IGNORED_EXTENSIONS
from scrapy.exceptions import IgnoreRequest


class IgnoredExtensionsMiddleware:
    IGNORE_PATTERNS = (
        "/stores/",
        "gp/profile",
        "gp/product",
        "gp/customer-reviews",
        "product-reviews",
        "ask/answer",
    )

    def process_request(self, request, spider):
        for pattern in self.IGNORE_PATTERNS:
            if pattern in request.url:
                raise IgnoreRequest()

        if request.url.lower().endswith(tuple(IGNORED_EXTENSIONS)):
            raise IgnoreRequest()

Scrapy Rules and LinkExtractors are your friend

Cramming all of the logic into a single parse method was a recipe for a jumbled mess. Refactoring the crawling logic to make use of rules and link extractors cleaned up the spider class considerably. Link extractors define what part(s) of the page Scrapy should examine for links to follow. This logic, in combination with the middleware, controls how the crawler moves through the website.

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class ComputerPCHardwareSpider(CrawlSpider):
    name = "computer_pc_hardware_spider"

    rules = (
        Rule(
            LinkExtractor(
                restrict_xpaths="//div[contains(@class, 'bxc-grid__container')]//*[img]"
            )
        ),
        Rule(LinkExtractor(restrict_text="See all results")),
        Rule(
            LinkExtractor(restrict_xpaths="//div[contains(@class, 's-main-slot')]//h2"),
            callback="parse_item",
        ),
    )

The XPath expression passed into restrict_xpath determines what part of the page to extract links from. Any link tag that is a descendant of a node that satisfies the XPath condition will be pulled into the crawl.

As long as you subclass CrawlSpider, it's unnecessary to have a parse callback for every request. Only the final rule, which covers links to product pages, has a defined parse method in this case.

Save data using an item pipeline

The crawler saves product details to a PostgreSQL database. To do that, I use an item pipeline, which allows for a nice separation of concerns. An item pipeline keeps logic involved in saving data apart from the crawling logic.

The pipeline.py file contains all of the database related code for the project.

class DatabasePipeline:
    def __init__(self):
        engine = sa.create_engine(
            "postgresql://{}:{}@{}:5432/{}".format(
                os.environ["POSTGRES_USER"],
                os.environ["POSTGRES_PASSWORD"],
                os.environ["POSTGRES_HOST"],
                os.environ["POSTGRES_DB"],
            )
        )

        Base.metadata.create_all(engine)
        Session = sessionmaker(bind=engine)

        self.session = Session()

    def process_item(self, item, spider):
        try:
            self.session.execute(
                insert(InventoryItem)
                .values([item])
                .on_conflict_do_nothing()
            )
        except:
            self.session.rollback()

        self.session.commit()

The process_item method will be invoked whenever the spider parse yields a non-request object. By convention, Scrapy expects parse functions to yield either an item or another request to be issued. When a product is scraped, the spider yields a dictionary, which in turn is inserted into the database.

Initially, I thought calling commit for each item would be too slow, but it was insignificant compared to the network requests.

Use AUTOTHROTTLE early on while debugging

Scrapy is fast, and early on when I wasn't sure if my crawler was following the right links, this meant burning proxy bandwidth on the wrong pages. By the time you realize something is wrong, the crawler is already pretty far off course.

Luckily, the AUTOTHROTTLE_ENABLED setting is an easy way to slow things down until your crawler is rock solid. The autothrottle setting adapts request rates based on the response time of the domain you're crawling. This results in a significant slow down, especially if you're behind a proxy.

There is also the DOWNLOAD_DELAY setting that will add a specified number of seconds to each request. Either one will work if you just need to slow things down while you're in debugging mode.

Use XPath for selecting elements

CSS selectors simply don't cut it when it comes to scraping sites like Amazon. Class names are often repeated on elements you don't want to include, and useful IDs are a rarity. Quite often, the only way to target the correct links is to pattern match against a sub-tree of elements, instead of one element in isolation.

        Rule(
            LinkExtractor(restrict_xpaths="//div[contains(@class, 's-main-slot')]//h2"),
            callback="parse_item",
        )

This rule, for instance, matches against not just a class name, but also requires that an h2 reside somewhere in the sub-tree of the parent.

XPath can be a little daunting, but the flexibility is worth spending time to become familiar with it.

Checkout the Amazon scraper code

The code is freely available for anyone to use or alter. You can check it out on GitHub for additional usage instructions.

DEV Community: zchtodd

Dockerizing Django: How to Create a Consistent Development Environment for Your Team

Setting Up Docker

Containerizing Django

Building a Simple Compose File

Building and Running the App Service

Adding the Database Service

Adding the Caddy Service

Adding Celery Instances

Next Steps

Beyond the Basics (Part V): Formik, D3, and More!

Table of Contents

Building the Proxy Connection Data Model

Creating and Deleting Proxy Connections

Creating the New Proxy Form

Building the Proxy Dashboard with Flexbox

Adding a Donut Chart Using D3

What's next?

Building a SaaS App (Part IV): User Authentication in Flask and React

Table of Contents

Securing a Flask REST API with JSON Web Tokens

Setting up authentication in Flask

Handling User Sign-Up in Flask

Email verification and account activation

Creating the user and sending the activation email

Securing pages within the React app

Adding Google Oauth as a Sign Up Option

What's next?

Building a SaaS App: Beyond the Basics (Part III)

Table of Contents

Setting up SQLAlchemy and Postgres

SQLAlchemy performance pitfalls to avoid

The N+1 Problem

Calling commit too many times

Great convenience, but with caution required

Setting up our first API route handler

What's next?

Building a SaaS App: Beyond the Basics (Part II)

Table of Contents

Setting up Docker and Docker Compose

Deploying the development version

Understanding how NGINX and Flask work together

Testing the NGINX and Flask configuration

Postgres Configuration

Setting up SSL with Let's Encrypt

Deploying the production version

What's next?

Building a SaaS App: Beyond the Basics

Table of Contents

Building the Google Search scraper

Setting up Puppeteer on an AWS instance

Making a simple Google search request

Using a proxy network for scraper requests

Gathering the Search results

Scraper error handling

What's next?

7 Tips for Building an Amazon Scraper

Build a spider to handle each product category

Use a residential proxy network

Rely on Scrapy middleware to filter requests

Scrapy Rules and LinkExtractors are your friend

Save data using an item pipeline

Use AUTOTHROTTLE early on while debugging

Use XPath for selecting elements

Checkout the Amazon scraper code