Spring Library: Selenium-Docker pool

I recently started working on my Master's thesis. Until now, I haven't wrote a single line of text, but only code instead. And since I wanted to share my experiences I already made along the way, I decided to create a small library of a part of my work.

There is no concrete title for the paper yet, but the main task is to model and automatize Twitter user behavior. To easily interact with Twitter, I decided to go for browser automation using Selenium. And since I wanted to embed this into an existing Spring application and needed to concurrently run different Selenium tasks, I wrote a module to provide Selenium-Docker containers.

The main concept 💌

In general, I'm making use of the awesome library Testcontainers to create the containers. Since we are talking about pools, it's all about taking containers and returning them to the pool after finishing work. The rest (environment cleanup, browser restart, ...) is done by the pool itself.

When acquiring a container from the pool, it is also possible to specify a profile, which the underlying Chrome browser should run with. By this, you have the chance to access cookies and stuff stored from previous runs.

Dive into the pool 🤿

Depending on your build tool, add the dependency to your project. For Maven it would be:

<dependency>
  <groupId>de.alxgrk</groupId>
  <artifactId>spring-selenium-pool-core</artifactId>
  <version>1.0.0</version>
</dependency>

Thanks to Spring auto-configuration, you are now able to get an @Autowired instance of WebDriverPool.

@Component
class SomeSeleniumTask(@Autowired webDriverPool: WebDriverPool) {

    init {
        val container: WebDriverForContainer? = webDriverPool.getWebDriverForContainer()
    }    

}

Since it might happen, that all containers are in use, this method may return null. To circumvent this issue, you can also wait until there is a free container. (If you've expected a suspending function at this point, have a look at CompletionStage.await() extension function for converting CompletableFuture)

@Component
class SomeSeleniumTask(@Autowired webDriverPool: WebDriverPool) {

    init {
        val container: CompletableFuture<WebDriverForContainer> = webDriverPool.getWebDriverForContainerAsync()

        // blocking get
        container.get()

        // ... or with timeout
        container.get(1, TimeUnit.SECONDS)

        // ... or coroutine style
        GlobalScope.launch {
            val webDriverForContainer = container.await()
        }
    }    

}

Doing the actual work 💪

Now you are able to do whatever you wanted to do. And to not forget to return the container to the pool (unless you are sure, you'll need the exact same instance later), WebDriverForContainer implements AutoClosable. Means you can simply use try-with-resources in Java or .use() extension function in Kotlin.

@Component
class SomeSeleniumTask(@Autowired webDriverPool: WebDriverPool) {

    init {
        webDriverPool.getWebDriverForContainer()!!.use {
            it.webDriver.get("https://foo.bar/")
        }
    }    

}

Configuration 🛠️

Last but not least, it is possible to configure the pool via Spring-Boot properties. Auto-Completion when working with IntelliJ included. This is an example configuration showing the default values:

# default values
selenium.pool.size=3
selenium.pool.recording-directory= # if empty, no recording will happen
selenium.pool.profiles-directory= # if empty, a temporary directory will be created and deleted on exit
selenium.pool.extension-files-in-classpath= # if empty, no extension files will be loaded

✨ Bonus: Custom Chrome extensions

If you read about the configuration properties carefully, you may have notice the parameter about extension files.
It is possible to install a custom Chrome extension by specifying a manifest.json and a corresponding index.js file.
This might be helpful for a couple of reasons including e.g. for grabbing the network traffic.
For an example on how to use this, see application.properties, manifest.json and index.js in spring-selenium-pool-example.

Feedback

I hope you like this library and find it useful. So please use it and tell me about it. I would be happy to hear what you like and what could be improved.

Check out the library on Github (a working example can be found in spring-selenium-pool-example) and leave a star if you like it:

alxgrk / spring-selenium-pool

A library that provides a pool of dockerized Selenium instances.

🏊 Selenium Pool

This small library helps to create a pool of Selenium Docker Container with a configurable size.

🩲 Prerequisites

Make sure to have the following installed:

• Docker

💿 Installation

For Maven, add this dependency:

<dependency>
  <groupId>de.alxgrk</groupId>
  <artifactId>spring-selenium-pool-core</artifactId>
  <version>1.0.0</version>
</dependency>

For Gradle use:

implementation("de.alxgrk:spring-selenium-pool-core:1.0.0")

🛠️ Configuration

This library is configurable via Spring-Boot properties. Auto-Completion when working with IntelliJ included. This is an example configuration showing the default values:

# default values
selenium.pool.size=3
selenium.pool.recording-directory= # if empty, no recording will happen
selenium.pool.profiles-directory= # if empty, a temporary directory will be created and deleted on exit
selenium.pool.extension-files-in-classpath= # if empty, no extension files will be loaded

🥽 Utilities

To connect to the running Selenium container, you have to have vncviewer & tigervnc-common installed. If correctly configured, simply…

View on GitHub