DEV Community: Alexander Girke

How useful are listings? 📜

Alexander Girke — Fri, 06 Jan 2023 22:33:24 +0000

I recently received some DEV credits that can be used to create a listing.

Since I've never really used / scrolled through the listing section on DEV, I was wondering how useful these credits are. Let's assume I would like to promote a side project, who would I be able to reach with that?

Does anyone have further information about listings in general and their effectiveness?

OSS Sponsoring by Usage? 💰📈

Alexander Girke — Tue, 20 Dec 2022 14:12:33 +0000

Hi there,

I recently thought about all the different OSS libraries I use every day at work and in personal projects.

Most of them are created and maintained by individuals that spend a lot of their time on that work. So, to give something back seems like a good thing to do - but to whom?

What I was asking myself is if there is already a tool/method to track the actual usage of a certain library or all dependencies of a project in general - and then use that data to aggregate & distribute donations to those people using the well-known platforms like GitHub Sponsors, Buy Me A Coffee, Open Collective, and all the others...

If there is no such thing, do you think people might be interested in something like that?
If so, what could the actual measure for a donation look like? Fixed rate per build / install task? Times the occurrence of that library in the dependency tree? Or a more sophisticated mechanism?

Please comment below, if you have any other ideas or anything to add 😊

#RedisHackathon - Comics Auctioning System

Alexander Girke — Mon, 29 Aug 2022 23:59:39 +0000

Overview of My Submission

https://redis-auctioning.alxgrk.de/

Submission Category:

Microservice Mavens

Language Used

Kotlin/Ktor
React/TS

Link to Code

alxgrk / redis-dev-to-hackathon-auction-system

A Contribution to the Redis-Dev.to-Hackathon.

Meme Auctioning System

Finally, you are able to sell your comics, memes and other artistic drawings on an auctioning platform.

How it works

How the data is stored:

Most of the data is stored as JSON in various keys and various data types.

  Refresh Tokens
     userId: like "cd63d6fd-3333-4877-9b8f-aab9e24271ca"
     refreshToken: like "c3adfc0d-6eb6-40f7-8663-49896d0c1855"
     expiryDate: like 1662422171471

  Auctions:
     id: like "8b0be9f2-3bb0-4972-87c6-423a7073832f"
     start: like "2022-08-07T22:23:07Z"
     end: like "2022-08-10T22:23:07Z"
     items: like ["95069680-670a-40c6-8068-c6cdd2561932"]
     title: like "First Auction"
     description: like "The very first Auction"
     seller: like "cd63d6fd-3333-4877-9b8f-aab9e24271ca"
     lowestBid:
       amount: like 123.45
       currency: like "EUR'
     isClosed: like false

  Items:
     id: like "d6118270-8404-4560-bee9-50c32870dd69"
     title: like "Move Fast and Break Things"
     description: like "Move Fast and Break Things"
     image: like "https://imgs.xkcd.com/comics/move_fast_and_break_things_2x.png"
     owner: like "735db4d6-658d-4ea6-aeed-d0718e076d6a"
  
  Biddings: (where key has prefix 'biddings:' followed by auctionId)
    0:
     id:

…

View on GitHub

BTW, your Github code’s README file should contain the details as per this template.

Additional Resources / Info

[Note:] # (Be sure to link to any open source projects that are using your workflow!)

[Note:] # Screenshots/demo videos are encouraged!

Check out Redis OM, client libraries for working with Redis as a multi-model database.
Use RedisInsight to visualize your data in Redis.
Sign up for a free Redis database.

Tracking Twitter User Behavior - The Browser Extension

Alexander Girke — Mon, 01 Feb 2021 08:20:20 +0000

Hello👋,

today we're gonna have a look at the first component of my data collection pipeline: the browser extension. If you want to peek into the code, find it here.

Intro

First, some background: what I needed was a browser extension, which runs on as many browsers as possible without having too much of a hassle supporting all of them. Luckily, Chrome (and all other Chromium-based browsers like Opera, Edge or Brave) and Firefox basically share the same API, called WebExtensions API. I won't go into much detail, so if you're not familiar with how browser extensions look like and what they can do, check out this awesome post:

Build Your First Chrome Extension

Ganesh Jaiwal ・ Sep 3 '20 ・ 4 min read

#javascript #codenewbie #computerscience #webdev

The resulting manifest.json looks like this:

{
  "manifest_version": 2,
  "name": "Twitter Tracking Extension",
  "version": "$VERSION",
  "description": "A browser extension that captures Twitter clicks - for scientific reasons.",
  "homepage_url": "https://github.com/alxgrk/twitter-tracking",
  "browser_specific_settings": {
    "gecko": {
      "id": "twittertracking@university.de",
      "strict_min_version": "48.0",
      "update_url": "https://github.com/alxgrk/twitter-tracking/blob/master/twitter-tracking-browser-extension/firefox-addon-update.json"
    }
  },

  "permissions": [
    "storage",
    "unlimitedStorage",
    "$ACCESS_SITE_PERMISSION"
  ],

  "icons": {
    "48": "icons/icon-48.png",
    "96": "icons/icon-96.png"
  },

  "browser_action": {
    "default_icon": {
      "32" : "icons/icon-32.png"
    },
    "default_title": "Twitter Tracking",
    "default_popup": "popup/tracked-events.html"
  },

  "content_scripts": [
    {
      "matches": ["*://*.twitter.com/*"],
      "js": ["js/main.js"]
    }
  ]

}

Note the variables starting with $...: they are going to be replaced by webpack. We'll see how in the next section.

The Building Process

First, let's have a look at how the final artifacts are build. I'm using Webpack's copy-webpack-plugin and webpack.DefinePlugin to differentiate when building for dev and prod. The CopyWebpackPlugin enables you to move third-party dependencies where needed or replace placeholders in files like the manifest.json. The following snippet shows how to set version and access site permission & bundle the webextension-polyfills necessary to ensure interoperability between Chrome and Firefox for production:

new CopyWebpackPlugin({
            patterns: [
                {
                    from: 'src',
                    globOptions: {
                        ignore: ['**/*.js'],
                    },
                    transform(content, absoluteFrom) {
                        if (absoluteFrom.includes("manifest.json")) {
                            return content.toString()
                                .replace("$VERSION", process.env.npm_package_version)
                                .replace("$ACCESS_SITE_PERMISSION", dotenv.parsed.PROD_API_URL);
                        } else {
                            return content
                        }
                    }
                }, {
                    from: 'node_modules/webextension-polyfill/dist/browser-polyfill.js',
                    to: path.resolve(distRoot, 'js')
                }
            ]
        }

To set environment variables like the endpoint of the REST-API, DefinePlugin has to be used as follows:

new webpack.DefinePlugin({
            'process.env.NODE_ENV': '"production"',
            API: JSON.stringify(dotenv.parsed.PROD_API_URL)
        })

As you may have noticed, the value for the API comes from a variable named dotenv, which is also the name of the library, that reads a .env file from your current working directory and provides its values as JS objects. To optimize the production artifact, I also added the TerserPlugin like this:

optimization: {
        minimize: true,
        minimizer: [new TerserPlugin()],
    }

Since we now have everything ready to build our artifacts, we need to make sure that Chrome, Firefox, etc. permit their installation. To achieve this, we need two different tools: web-ext for Firefox and chromium for Chrome. Check out this script for an example on how to use them. Also, make sure to follow the instructions on publishing extensions for Firefox and for Chrome.

The Collection Process

Once the user has the extension installed, the content-script located at src/main.js runs as soon as the user visits Twitter. And now the funny stuff begins.

What I did was to register some listeners to the standard browser events like load, beforeunload,click or scroll. If you remember the model from the previous post, we now need to deduct from these events what temporal dimension and what kind of user interactions we have.

load and beforeunload give us the start and end events, which are going to be published to the backend and additionally stored in local storage for the user to be displayed later. While constructing the events, we also have to create some kind of identifier for the user, ideally a reproducible one if we want to relate his/her Android app interactions. To achieve this, the username has to be found by inspecting the alt attribute of the user's profile image. And since we care (at least a little) about privacy, a consistent and "secure" hashing method like SHA256 is applied to it.

scroll maps trivially to the scroll events needed. Just use document.documentElement.scrollTop to get the current scroll position.

Concerning clicks, however, a little more fiddling is required. Unfortunately, Twitter HTML tags are almost all divs with randomized classnames. The only anchor are semantic tags like article, rare element IDs and the attribute data-testid. Luckily, with these and a great tool named @medv/finder, I was able to create RegExes that match the found selector to my predefined click types. So for any click events, the finder is run...

let selector = finder(event.target, {
        className: name => false // do not rely on classnames
    });

...and the result is mapped. All mapping functions are stored in an object, whose keys are the names of the click type. For e.g. a like click, this is the RegEx:

const TARGETS = {
    like: selector => /#tweet-action-buttons.+:nth-child\(3\).*/.test(selector),
    ...
}

As you can imagine, this approach is quite error-prone. But that is one of the difficulties when reverse-engineering a constantly changing external system and I haven't found an alternative yet. If you know it better, please comment.

Thanks for reading and have fun until next time when the post about the Android app is finished. ✌️

Tracking Twitter User Behavior - The Model

Alexander Girke — Mon, 25 Jan 2021 07:46:26 +0000

Hello again👋,
welcome to the second part of my series on understanding Twitter user behavior.

Before we dive deeper into the code, I want you to understand how I envision the characteristics of an user's behavior.

First of all, there are three main aspects:

temporal 🕰️ - how often does the user open Twitter and how long does a session last?
publicly visible 📻 - how often does a user post a tweet or follow someone?
publicly invisible 🕵️‍♀️ - how many tweets does a user scroll, how often do they click on images, video, etc.

For each of those three aspects I assigned some quantifiable characteristics. The result looks like this:

Temporal	Publicly Visible	Publicly Invisible
Sessions Per Day	Posting Prob.	Scroll Timeline Prob.
Session Length	Like Prob.	Click On Media Prob.
Session Distribution Over Day	Retweet Prob.	Open Details Prob.
	Follow After Searching Prob.	Click On Author Profile Prob.
	Follow From Tweet Prob.	Click On Hashtag Prob.

So the next step is to collect data from real users that will allow me to assign values to each of these cells. To do this, the Android app and the browser extension will have to send events that correspond to what can be described by this Kotlin data class:

data class Event(
    val eventType: EventType,
    val userId: String,
    val action: String,
    val timestamp: String,
    val target: String? = null,
    val selector: String? = null,
    val scrollPosition: Int? = null,
    val estimatedTweetsScrolled: Int? = null
) {
    enum class EventType {
        BROWSER,
        ANDROID
    }
}

As you can see, some of the properties are optional, depending on the value for action, which is either scroll, click, session_start or session_end. Note that this is the data class used in the backend, which is responsible for writing into the ElasticSearch instance only - hence, I didn't need to refine it anymore. The Android app, however, uses a more complex class hierarchy to facilitate event publishing. More on this in a later post.

A like event in the browser would look like this:

{
    "event_type": "BROWSER",
    "user_id": "b92427ae41e4649b934ca495991b7852b855...",
    "action": "click",
    "timestamp": "2021-01-24T16:11:36.822Z",
    "target": "like",
    "selector": "#tweet-action-buttons > div:nth-child(3) > div > div > div > div"
  }

... whereas a scroll event on Android would look similar to this:

{
    "event_type": "ANDROID",
    "user_id": "3a4cab1a6f566207d520622e0bbce78d7...",
    "action": "scroll",
    "timestamp": "2021-01-24T13:29:33.853",
    "scroll_position": 5938,
    "estimated_tweets_scrolled": 5
  }

With this information it is later possible to identify sessions, probabilities per session and probabilities per tweet. Thus, in the end, automated software that tries to mimic real users can be fed with these values.

Those are the basics to get an idea of what the whole infrastructure is about. In the next posts, we will look at the individual components, starting with the browser extension.

For those who haven't read the first post: here is the code involved.

Tracking Twitter User Behavior - The Overview

Alexander Girke — Sat, 23 Jan 2021 21:49:30 +0000

Hey there👋,

it's been a while since my last post - which doesn't mean, I've been lazy. I've constantly worked on my Master's Thesis and I'm almost done, but now I really wanted to share what I recently did: understanding Twitter users' behavior by spying on them.

In fact, one shouldn't call it spying if the victim wants it that way - or should one (#socialmedia)? Nonetheless, some people were crazy enough to help me by installing a browser extension or an Android app, respectively. And in this series I want to show you the infrastructure I used to collect and analyze the data they produced.

So let's start with an overview:

As you might see, I used AWS to deploy some of the components: I've created a Lambda function, which serves a REST API through an API Gateway and is based on Ktor Kotlin server framework. On invocation, incoming events are stored in an ElasticSearch instance also provided by AWS. Best of all: all of this is covered by the AWS Free Tier, so there is no cost.

To analyse the data, I again used Kotlin and combined 3 incredibly helpful tools: clikt, es-kotlin-client and plotly.kt. But more about that later.

You might know that tracking clicks in a browser is no big deal - but how the heck is that supposed to work in an Android app? Sorry, but again you'll have to be patient 🤫 - that's part of the next article in this series.

For those who don't want to wait for the solution: check out the source code of the whole infrastructure here.

Hope you'll stay tuned. ✌️

TIL: Connecting to the host from Testcontainers

Alexander Girke — Mon, 02 Nov 2020 22:51:23 +0000

TL;DR: Testcontainers allows you to expose a host's port to containers using Testcontainers.exposeHostPorts(localServerPort);.

A short reminder: for my Master's thesis I have to work with Selenium docker containers to automate Twitter accounts. My latest task was to make it possible to specify a proxy, which should be used by the Chrome browser running inside a container. That way, it want to fool Twitter about the true origin of an account, which is especially interesting if you want to automate multiple accounts at the same time on the same machine.

While setting the proxy has to be done using Selenium's JavascriptExecutor.executeScript(..) method in conjunction with JavaScript code accessing the Proxy API for Extensions (for more information, have a look at the example in spring-selenium-pool), I finally wanted to test my setup.

As a mock proxy server, I decided to use MockServer - integration with Java/Kotlin is easy, as is defining stub responses. The final question however was: how could I connect to the MockServer running on the host from inside the Selenium container?

Luckily, Testcontainers guys got me covered. The documentation provided the answer: Testcontainers.exposeHostPorts(localServerPort);.

That's all we need. Just make sure to run that line before your container starts, but after the mock server is running.

The host will then be available from inside the container at host.testcontainers.internal + your random local port.

What I understood it does under the hood, is to start another Docker container with an ssh-server that uses port forwarding to route all TCP traffic from the container of interest through the ssh-container to the host.

So in the end, here is the skeleton of the resulting test:

@SpringBootTest(classes = [TaskWithProxyTest.App::class],
        webEnvironment = SpringBootTest.WebEnvironment.NONE,
        properties = ["selenium.pool.size=1"])
class TaskWithProxyTest {

    // causes the container from spring-selenium-pool to come up
    @SpringBootApplication
    class App

    companion object {

        lateinit var mockProxy: ClientAndServer

        @BeforeAll
        @JvmStatic
        fun first() {
            val mockServerPort = SocketUtils.findAvailableTcpPort()
            mockProxy = startClientAndServer(mockServerPort)

            // necessary to access mockserver from container
            Testcontainers.exposeHostPorts(mockServerPort)
        }

        @AfterAll
        @JvmStatic
        fun tearDown() {
            mockProxy.stop(true)
        }
    }

    @Test
    fun `your test`() {
        // ...
    }
}

Playing around with Kotlin Sealed Classes

Alexander Girke — Thu, 24 Sep 2020 06:50:15 +0000

While working on my Master's thesis, I recently had to configure probabilities. These values were organized in a tree-like structure. Because of this, I thought of using inheritance - and with Kotlin already in place, using sealed classes seemed at least like an interesting opportunity to me.

Disclaimer: This post is just for fun and for showing some cool Kotlin-specific things. In the end, using Jackson to deserialize input into data classes is probably more effective.

Sample

Let's say, we want to configure the likeliness in percent of some actions using YAML for instance. This could look like the following:

Configuration:
    Greeting:
        SayHello: 70
        WaveAt: 80
        Hug: 5
    Talking:
        DoSmallTalk: 90
        Insult: 1

As you can see, there are two subsections. However, this can easily get an arbitrary depth.

Before we'll reproduce that structure, we should clarify what sealed classes are.

Sealed Classes

The official documentation calls them extensions to enum classes, used for representing restricted class hierarchies.

Sealed classes are declared using the keyword sealed, are abstract and must have only private constructors.

Simply put, a sealed class ensures, that all possible subtypes are known at compile time. This comes especially handy, when used in conjunction with Kotlin's when clause:

sealed class TruthOrDare
class Truth(val question: String) : TruthOrDare()
class Dare(val task: String) : TruthOrDare()

fun nextTurn(input: TruthOrDare) = when(input) {
    is Truth -> println("Answer the following question: ${input.question}")
    is Dare -> println("Your task is: ${input.task}")
    // the `else` clause is not required because we've covered all the cases
}

Sealed Classes in Action

Let's come back to the original task: using sealed classes for parsing configuration.

The structure of the above YAML snippet can be represented as follows:

sealed class Configuration {

    sealed class Greeting : Configuration() {

        object SayHello : Greeting()
        object WaveAt : Greeting()
        object Hug : Greeting()

    }

    sealed class Talking : Configuration() {

        object DoSmallTalk : Talking()
        object Insult : Talking()

    }

}

We have nested sealed subclasses and objects (a shortcut for Singletons in Kotlin). We'll see later, why we are using objects here.

However, yet there is nothing configured. So how do we use this structure?

The answer is: Reflection.

Digression

Before we come to the actual structure though, I would like to introduce a small wrapper class, to encapsulate our findings:

class Probabilities private constructor(
        private val backingMap: MutableMap<Configuration, Int> = mutableMapOf()
) : Map<Configuration, Int> by backingMap {

    constructor(configuration: Probabilities.() -> Unit) : this() {
        configuration()
    }

    infix fun Configuration.withProbabilityOf(percent: Int) = backingMap.put(this, percent)

    override fun toString(): String = backingMap.entries.joinToString { "${it.key::class.simpleName} = ${it.value}" }
}

In this class you can see some other cool Kotlin features: implementation by delegation, receiver functions and infix functions.

Implementing the Map interface can be a pain in Java. Kotlin provides a smart solution with implementation by delegation. All you need to do, is to use the by keyword in conjunction with a value, that already implements the interface. So in this case we simply delegate all Map-specific operations to a MutableMap, which is created by the default constructor.

To make configuration of the Probabilities class easy, we provide a second constructor taking a receiver function. This means, that the this keyword of the provided lambda points to an instance of Probabilities class.

Another receiver function is also provided by withProbabilityOf. However, this is also an infix function, which is marked by the infix keyword and enables you to write something like:

SayHello withProbabilityOf 99

Parsing

Finally, here is the code to parse e.g. a Map of String and Any:

fun fromMap(configuration: Map<String, Any>) = Probabilities { // (1)
    @Suppress("UNCHECKED_CAST")
    fun parseFor( // (2)
            configuration: Map<String, Any>, // (3)
            parent: KClass<out Configuration>?,
            clazz: KClass<out Configuration>
    ) {
        // (4)
        if (configuration.containsKey(clazz.simpleName) && parent?.isSuperclassOf(clazz) != false) {
            when (val value = configuration[clazz.simpleName]) {
                is Map<*, *> -> { // (5)
                    clazz.sealedSubclasses.forEach { subclass ->
                        parseFor(value as Map<String, Any>, clazz, subclass)
                    }
                }
                is Int -> { // (6)
                    if (clazz.objectInstance == null)
                        throw RuntimeException("${clazz.simpleName} should be an object")

                    clazz.objectInstance!! withProbabilityOf value
                }
                else -> throw RuntimeException("unknown property ${clazz.simpleName}")
            }
        }
    }

    parseFor(configuration, null, Configuration::class)
}

Some explanation (mind the comments in code):

(1) we construct a new instance of Probabilities by making use of one of Kotlin's greatest features: if the last parameter of any function or constructor is a lambda function, we can omit parentheses and only use curly braces
(2) another beautiful thing are local functions: since we don't need the recursive function parseFor anywhere else, we simply declare it inside our fromMap function
(3) the parameters of our parseFor function are:
- the subtree of the configuration
- the assumed parent as specified by the configuration
- the current class as specified by the configuration's property
(4) if somewhere on top level of the current subtree the expected class is found and the assumed parent matches the real superclass, then we use when on the property value's type
(5) if it's a Map, apply parseFor on each sealed subclass
(6) if it's an Int and clazz is an object, we configure the probability to be that value

In the above code, you might see, why having leaves being objects is a good idea: there will always be exactly one instance. So they are perfect for being used as a key in a map.

Sweet. - But why?

To be honest, all of the above might seem like an over-engineered construct. Compared to the example using sealed classes, an equivalent structure with data classes would be similar and could be easily used with e.g. Jackson out of the box:

data class Configuration(
    val greeting: Greeting?,
    val talking: Talking?
) {

    data class Greeting(
        val sayHello: Int?,
        val waveAt: Int?,
        val hug: Int?
    )

    data class Talking(
        val doSmallTalk: Int?,
        val insult: Int?
    )
}

This is the great thing about Kotlin: you have so many language constructs that help you build your software the best way, whatever this means. Therefore, it's necessary to play around with and get to know these possibilities.

Using Jackson

To make use of the concept of using sealed classes in conjunction with Jackson, we need to provide a custom deserializer. Since we already have the mapping logic however, this is an easy task:

class ProbabilitiesDeserializer : StdDeserializer<Probabilities>(Probabilities::class.java) {

    override fun deserialize(p: JsonParser?, ctxt: DeserializationContext?) =
            p?.readValueAs<Map<String, Any>>(object : TypeReference<Map<String, Any>>() {})
                    ?.let { fromMap(it) }

}

For this and the ability to parse YAML we need the following dependencies:

com.fasterxml.jackson.module:jackson-module-kotlin:2.11.2
com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.11.2

Finally, we are ready to create an ObjectMapper and read our YAML configuration:

val mapper = ObjectMapper(YAMLFactory()).apply {
    registerModule(SimpleModule().addDeserializer(Probabilities::class.java, ProbabilitiesDeserializer()))
    registerModule(KotlinModule())
}
val probabilities = mapper.readValue<Probabilities>(yamlInput)
println(probabilities.toString())
println("Say hello with probability of ${probabilities[SayHello]} percent.")

The result will be:

SayHello = 70, WaveAt = 80, Hug = 5, DoSmallTalk = 90, Insult = 1
Say hello with probability of 70 percent.

That's it.

Bonus: Ensuring all properties are set

One last thing I wanted to add, is how you enforce all objects to be configured.

All we have to do is to find all the leaves of our configuration tree and compare them to the input's content:

val leaves = leavesOf(Configuration::class)
fun leavesOf(baseClass: KClass<out Configuration>): List<KClass<out Configuration>> =
    if (!baseClass.isSealed) {
        listOf(baseClass)
    } else {
        baseClass.sealedSubclasses.flatMap(::leavesOf)
    }

fun Probabilities.ensureAllActionsCovered() {
    val keys = keys.map { it::class }
    val unconfigured = leaves.filter { leaf -> !keys.contains(leaf) }
    if (unconfigured.isNotEmpty())
        throw RuntimeException("Unconfigured leaves: ${unconfigured.joinToString()}")
}

Finally, we've seen a last cool Kotlin feature: extension functions. They allow us to add some functionality to an otherwise closed class. Like a bonus in a way.

You call them as if they were a class method:

probabilities.ensureAllActionsCovered()

Closing notes

Thanks for reading, I hope you liked it. You can find all the code in a kscript on Github as a Gist.

Kotlin: 'null' does not equal 'true', but also not 'false'

Alexander Girke — Mon, 21 Sep 2020 14:17:54 +0000

TL;DR Kotlin has the === operator for referential integrity and none of true, false or null references the same thing. Don't forget that, when comparing nullable booleans.

Today I stumbled upon a hint from IntelliJ. I wanted to write something like the following:

val potentiallyNull: Any? = null // in real world, we wouldn't know whether it's null or not
if (potentiallyNull?.equals("foo") ?: false) {
    println("not printed when null")
}

In words: if potentiallyNull is not null, check for equality with "foo"; otherwise return false.

So, IntelliJ kindly told me: Equality check should be used instead of elvis for nullable boolean check

I thought to myself, alright then, let's apply that nice hint.

What I got was:

if (potentiallyNull?.equals("foo") == true) {
    println("not printed when null")
}

However, later on, I wanted to invert the condition. So naively, I simply replaced true by false:

if (potentiallyNull?.equals("foo") == false) {
    println("print also when null")
}

Unfortunately, the result was not as expected. There was not a single character printed to the console. What went wrong?

Actually, it's quite obvious, but it took me a moment to realize: of course, null is neither the same as true nor as false.

So, I took a step back and rewrote my original condition:

if (potentiallyNull?.equals("foo") ?: true) {
    println("print also when null")
}

IntelliJ again told me to change that and use the equality check, so what I got now was:

if (potentiallyNull?.equals("foo") != false) {
    println("print also when null")
}

The first thing I thought was: How is ... != false different to ... == true?

But again, null is neither the same as true nor as false. So the comparison has actually three possible results: true, false or none of them.

... != false in this case means true or none of them, whereas ... != true means only true.

For more on equality in Kotlin, I recommend to read the official docs on equality

All in all, that wasn't my most brilliant moment for sure, but I thought others might stumble upon that as well. Hope you liked it.

Spring Library: Selenium-Docker pool

Alexander Girke — Fri, 14 Aug 2020 11:03:44 +0000

I recently started working on my Master's thesis. Until now, I haven't wrote a single line of text, but only code instead. And since I wanted to share my experiences I already made along the way, I decided to create a small library of a part of my work.

There is no concrete title for the paper yet, but the main task is to model and automatize Twitter user behavior. To easily interact with Twitter, I decided to go for browser automation using Selenium. And since I wanted to embed this into an existing Spring application and needed to concurrently run different Selenium tasks, I wrote a module to provide Selenium-Docker containers.

The main concept 💌

In general, I'm making use of the awesome library Testcontainers to create the containers. Since we are talking about pools, it's all about taking containers and returning them to the pool after finishing work. The rest (environment cleanup, browser restart, ...) is done by the pool itself.

When acquiring a container from the pool, it is also possible to specify a profile, which the underlying Chrome browser should run with. By this, you have the chance to access cookies and stuff stored from previous runs.

Dive into the pool 🤿

Depending on your build tool, add the dependency to your project. For Maven it would be:

<dependency>
  <groupId>de.alxgrk</groupId>
  <artifactId>spring-selenium-pool-core</artifactId>
  <version>1.0.0</version>
</dependency>

Thanks to Spring auto-configuration, you are now able to get an @Autowired instance of WebDriverPool.

@Component
class SomeSeleniumTask(@Autowired webDriverPool: WebDriverPool) {

    init {
        val container: WebDriverForContainer? = webDriverPool.getWebDriverForContainer()
    }    

}

Since it might happen, that all containers are in use, this method may return null. To circumvent this issue, you can also wait until there is a free container. (If you've expected a suspending function at this point, have a look at CompletionStage.await() extension function for converting CompletableFuture)

@Component
class SomeSeleniumTask(@Autowired webDriverPool: WebDriverPool) {

    init {
        val container: CompletableFuture<WebDriverForContainer> = webDriverPool.getWebDriverForContainerAsync()

        // blocking get
        container.get()

        // ... or with timeout
        container.get(1, TimeUnit.SECONDS)

        // ... or coroutine style
        GlobalScope.launch {
            val webDriverForContainer = container.await()
        }
    }    

}

Doing the actual work 💪

Now you are able to do whatever you wanted to do. And to not forget to return the container to the pool (unless you are sure, you'll need the exact same instance later), WebDriverForContainer implements AutoClosable. Means you can simply use try-with-resources in Java or .use() extension function in Kotlin.

@Component
class SomeSeleniumTask(@Autowired webDriverPool: WebDriverPool) {

    init {
        webDriverPool.getWebDriverForContainer()!!.use {
            it.webDriver.get("https://foo.bar/")
        }
    }    

}

Configuration 🛠️

Last but not least, it is possible to configure the pool via Spring-Boot properties. Auto-Completion when working with IntelliJ included. This is an example configuration showing the default values:

# default values
selenium.pool.size=3
selenium.pool.recording-directory= # if empty, no recording will happen
selenium.pool.profiles-directory= # if empty, a temporary directory will be created and deleted on exit
selenium.pool.extension-files-in-classpath= # if empty, no extension files will be loaded

✨ Bonus: Custom Chrome extensions

If you read about the configuration properties carefully, you may have notice the parameter about extension files.
It is possible to install a custom Chrome extension by specifying a manifest.json and a corresponding index.js file.
This might be helpful for a couple of reasons including e.g. for grabbing the network traffic.
For an example on how to use this, see application.properties, manifest.json and index.js in spring-selenium-pool-example.

Feedback

I hope you like this library and find it useful. So please use it and tell me about it. I would be happy to hear what you like and what could be improved.

Check out the library on Github (a working example can be found in spring-selenium-pool-example) and leave a star if you like it:

alxgrk / spring-selenium-pool

A library that provides a pool of dockerized Selenium instances.

🏊 Selenium Pool

This small library helps to create a pool of Selenium Docker Container with a configurable size.

🩲 Prerequisites

Make sure to have the following installed:

Docker

💿 Installation

For Maven, add this dependency:

<dependency>
  <groupId>de.alxgrk</groupId>
  <artifactId>spring-selenium-pool-core</artifactId>
  <version>1.0.0</version>
</dependency>

For Gradle use:

implementation("de.alxgrk:spring-selenium-pool-core:1.0.0")

🛠️ Configuration

This library is configurable via Spring-Boot properties. Auto-Completion when working with IntelliJ included. This is an example configuration showing the default values:

# default values
selenium.pool.size=3
selenium.pool.recording-directory= # if empty, no recording will happen
selenium.pool.profiles-directory= # if empty, a temporary directory will be created and deleted on exit
selenium.pool.extension-files-in-classpath= # if empty, no extension files will be loaded

🥽 Utilities

To connect to the running Selenium container, you have to have vncviewer & tigervnc-common installed. If correctly configured, simply…

View on GitHub