Using Selenium With Python in a Docker Container

While web scraping, I came across many useful applications such as listing old prices of some financial assets or finding current news topics. Although those examples are quite interesting to apply, frequently there was one main goal to reach at the end that is creating a database with the scraped information.

Whenever I went a bit further on scraping, I ended up in the websites using Javascript to display the data that I needed. Hence, I bumped into Selenium, which is a web testing and automation tool. In this small write up, I aim to list some steps that I find quite useful while setting up Selenium within a Docker container.

Introduction to Selenium WebDriver

Selenium WebDriver is a web automation or testing tool. It was created by Simon Stewart in 2006, as the first cross-platform testing framework that could control the browser from the OS level.

So with Selenium, I can run some automated actions on browsers (clicks, hovers, and fill forms) by directly communicating with them. Java, C#, PHP, Python, Perl, Go and Ruby are the supported languages for the bindings. Since I am more familiar with Python, I will be talking about it.

To work on a browser, I need to choose among a set of browser options like Firefox, Chrome (Chromium), Edge, and Safari. As a personal opinion, Chrome with a headless option (not generating a user interface) is the most performant one, hence I will be sticking to that.

Pulling the Image and Setting Up Google Chrome

To start with my custom Selenium-Python image, I need a Python image, here in this write-up I picked up the version 3.8.

Then I can install Google Chrome on top of it. Remember, without the Google Chrome itself, I cannot run Selenium on top of it to run our tasks. There are a few steps to apply for setting up Google Chrome in Linux:

1. Adding Google Chrome trusting keys to apt
2. Adding Google Chrome stable version to the repositories
3. Updating the repositories to see the stable version in apt
4. Installing google-chrome-stable

FROM python:3.8

# Adding trusting keys to apt for repositories
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -

# Adding Google Chrome to the repositories
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'

# Updating apt to see and install Google Chrome
RUN apt-get -y update

# Magic happens
RUN apt-get install -y google-chrome-stable

Installing Chrome Driver

Selenium requires a driver interface to work with the defined browser. Hence, I need to find a way to install Chrome Driver in our Linux image. Here are the steps to follow for doing this:

1. Installing unzip as we will need for the zipped Chrome Driver
2. Download the Chrome Driver into a folder called /tmp/chromedriver.zip, this name can be changed
3. Unzipping the /tmp/chromedriver.zip into the Linux executable path

After those steps, I need to set the display port (99) as Selenium is using this. It will avoid some crushes.

# Installing Unzip
RUN apt-get install -yqq unzip

# Download the Chrome Driver
RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip

# Unzip the Chrome Driver into /usr/local/bin directory
RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/

# Set display port as an environment variable
ENV DISPLAY=:99

Preparing the Docker for a Run

All the steps above were only for setting up Chrome in our Dockerfile. To run my Python application (app.py) using Docker, I might need the following lines into our Dockerfile.

COPY . /app
WORKDIR /app

RUN pip install --upgrade pip

RUN pip install -r requirements.txt

CMD ["python", "./app.py"]

Apart from those Docker settings, I would like to briefly mention some Docker specific chrome options while setting up the Chrome Driver via Python. I want to explicitly show those a few options in one function as set_chrome_options. Here I set up the example pseudocode with a function below. I need 4 specific arguments to run our Chrome Driver inside Docker:

1. Explicitly saying that this is a headless application with --headless
2. Explicitly bypassing the security level in Docker with --no-sandbox. There is a nice Stackoverflow thread over this, apparently as Docker deamon always runs as a root user, Chrome crushes.
3. Explicitly disabling the usage of /dev/shm/. The /dev/shm partition is too small in certain VM environments, causing Chrome to fail or crash.
4. Disabling the images with chrome_prefs["profile.default_content_settings"] = {"images": 2}.

from selenium.webdriver.chrome.options import Options
from selenium import webdriver

def set_chrome_options() -> None:
    """Sets chrome options for Selenium.
    Chrome options for headless browser is enabled.
    """
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_prefs = {}
    chrome_options.experimental_options["prefs"] = chrome_prefs
    chrome_prefs["profile.default_content_settings"] = {"images": 2}
    return chrome_options

if __name__ == "__main__":
    driver = webdriver.Chrome(options=chrome_options)
    # Do stuff with your driver
    driver.close()

Last Words

Here is the Dockerfile, that I took as an example. While creating this, I used the links that I shared to solve the problems that I faced. There might be other kinds of solutions to the problems that I faced. I am curious to listen to those.

Until now, I used it to scrape web archives for asset prices, books, yellow pages, and judgment texts. Although Selenium is not designed for web scraping, I leveraged this nice tool for taming Javascript using websites. But I should admit that, if the information that I was looking for was not hiding in Javascript, I would have been definitely a lot happier with using only Requests, BeautifulSoup4 and/or Scrapy for Python. Because all those are simpler to set up, and more performant.

Happy Scraping!

Comments

[Nick Crews]

Thanks for this! It might be easier to start off with an official selenium/standalone-chrome docker image, and then you don't have to install chrome and chromedriver yourself.

[Nazli Ander]

Thanks a lot for mentioning, that’s a very valid point. The most wordy part is installing the driver.

[martinstajnko]

Hi Nazli,

after building new image in Docker I got following error:
wget: invalid option -- 's'
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
The command '/bin/sh -c wget -O /tmp/chromedriver.zip chromedriver.storage.googleapis.co... -sS chromedriver.storage.googleapis.com/LATEST_RELEASE/chromedriver_linux64.zip' returned a non-zero code: 2

Can you help with this?

Thanks and best regards,

Martin

[Nazli Ander]

Hi Martin,

I was not able to reproduce the error. But I assume it is about the missing backtick in your Dockerfile RUN command. Because -sS (silent / show-error) option is not a part of the wget and it belongs to the command substitution starting with curl.

I will try explaining the line that you mentioned, then perhaps you can debug on your own in a faster way:

Command substitution allows the output of a command to replace the command itself, as it is stated in the GNU documentation. So the red part is evaluated first, then the main blue part is executed. The red part returns to an IP address states the location of the Chromedriver’s latest release. Then it is concatenated with the main URL. The main URL is used for downloading and storing the latest Chromedriver zip file.

With that part - we are able to pick up the latest release key without any hardcoding.

Hope the explanation is sufficient for you, and I was helpful. You can always see the repo that I stored the Dockerfile for the complete Dockerfile: github.com/nazliander/scrape-nr-of...

Best regards, N.

[khadyCi]

hi Nazil
I have used your example but when using:
driver = webdriver.Chrome(options=chrome_options)
This driver gives me the following error

[khadyCi]

please can you help me o someone else please