DEV Community: Daniel Kraus

Pre-push Commit Message Validation on Github (Enterprise)

Daniel Kraus — Sat, 31 Aug 2024 15:33:05 +0000

If you are looking for commit message validation on GitHub's Marketplace (or other places on the Internet), e.g., to enforce conventional commits, most solutions are based on GitHub Actions such as GsActions/commit-message-checker. While this does the job, I believe using GitHub Actions for commit message validation has two downsides:

Since validation happens as part of a workflow, it consumes runner minutes.
And because it is a workflow, the check happens post-push (from a Git server perspective).

If you are using GitHub Enterprise (Cloud, GHEC or Server, GHES), there is actually a GitHub-native solution: metadata restrictions.

Metadata restricions can be part of a branch or tag ruleset. That is, such a ruleset can be defined on the repository or organization level. In the latter case, you can also use custom properties to only target a subset of repositories (e.g., those of a specific team).

In any case, when you create the ruleset, you have to scroll to the bottom of the page to "Restrictions" and check "Restrict commit metadata", where it says:

Restrict commit author email addresses, committer email addresses, commit message content, and other metadata

Then click on "Add restriction", which opens the following dialog:

For "Requirement", you can choose among the following options:

Must start with a matching pattern
Must end with a matching pattern
Must contain a matching pattern
Must match a given regex pattern
Must not start with a matching pattern (negation of 1.)
Must not end with a matching pattern (negation of 2.)
Must not contain a matching pattern (negation of 3.)
Must not match a given regex pattern (negation of 4.)

Afterwards, click "Add" and activate (or evaluate) the ruleset – that's it!

I think this a great way to validate commit messages pre-push, GitHub-natively, without consuming runner minutes. And hopefully, this will someday also be part of GitHub's free tier. 🤞

Handling System Properties in JUnit 5

Daniel Kraus — Wed, 08 Jan 2020 11:30:02 +0000

In Java, properties are configuration values that are represented as key-value pairs, usually managed inside a Properties object. System properties are basically the Properties object of the System class, which "describes the configuration of the current working environment". This includes information about the current user, the OS or the Java runtime. The System class offers various methods to interact with its properties, however, the ones most used are:

clearProperty(String key): Removes the system property indicated by the specified key.
getProperty(String key): Gets the system property indicated by the specified key.
setProperty(String key, String value): Sets the system property indicated by the specified key.

Sometimes you need to control system properties when you test code that somehow uses System. When doing so, you shouldn't just wildly mutate their values, otherwise tests might inadvertently depend on possible side effects. Then, changes in your test code, or even changing the execution order, may lead to test failures that are hard to debug. Therefore, you want to have a proper restore mechanism in place that cleans up the environment after each test.

When it comes to JUnit 4, there are already off-the-shelf libraries for this such as the fantastic System Rules project. But for JUnit 5, you had to come up with a custom solution—at least until now.

There is JUnit Pioneer, a semi-official extension pack for JUnit 5. It offers various neat extensions, and as of version 0.5.0 also a mechanism to handle system properties in a safe manner. The @ClearSystemProperty and @SetSystemProperty annotation respectively, can be used to clear and set the values of system properties for a test execution. Both annotations work on the test method and class level, are repeatable as well as combinable. After the annotated method has been executed, the properties mentioned in the annotation will be restored to their original value or will be cleared if they didn't have one before. Other system properties that are changed during the test, are not restored.

For example, clearing a system property for a test execution can be done as follows:

@Test
@ClearSystemProperty(key = "some property")
void test() {
    assertNull(System.getProperty("some property"));
}

And setting a system property for a test execution:

@Test
@SetSystemProperty(key = "some property", value = "new value")
void test() {
    assertEquals("new value", System.getProperty("some property"));
}

As mentioned before, both annotations are repeatable and they can also be combined:

@Test
@ClearSystemProperty(key = "1st property")
@ClearSystemProperty(key = "2nd property")
@SetSystemProperty(key = "3rd property", value = "new value")
void test() {
    assertNull(System.getProperty("1st property"));
    assertNull(System.getProperty("2nd property"));
    assertEquals("new value", System.getProperty("3rd property"));
}

Note that class level configurations are overwritten by method level configurations:

@ClearSystemProperty(key = "some property")
class MySystemPropertyTest {
    @Test
    @SetSystemProperty(key = "some property", value = "new value")
    void test() {
        assertEquals("new value", System.getProperty("some property"));
    }
}

Sometimes, you might also need to set a system property to a value that is not a constant expression, which is required for annotation values. In this case, you can still leverage the restore mechanism:

@ParameterizedTest
@ValueSource(strings = { "foo", "bar" })
@ClearSystemProperty(key = "some property")
void test(String value) {
    System.setProperty("some property", value);
}

As you can see, both annotations are quite powerful and applicable in many situations. So, become a JUnit Pioneer stargazer on GitHub and take back control of your system properties!

A Recipe for Successful Exploratory Testing

Daniel Kraus — Tue, 13 Nov 2018 15:11:07 +0000

Originally posted on Medium.

In today’s agile and automated realm of software development, I often hear people ask: What actually is the role of manual testing? Assume you live in a perfect world where:

You always do test-driven development (TDD)
All your tests are automated
These tests produce 100% code coverage

Do you still need manual testing? Well, let’s first reflect the three points from above. TDD typically means that both production and test code get written by the same developer. In addition, these tests always test the same path. As a consequence, you need a lot of them; not just to achieve your 100% code coverage, but also to cover the risks of the system under test (SUT). And: automated tests are expensive. Not just in terms of maintenance, but also their creation — especially the higher you are in the test pyramid. Finally, many people within the testing community say that automated tests don’t do testing; they do checking:

Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.

(A test is an instance of testing.)

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

(A check is an instance of checking.)

Especially now, with AI-based testing tools emerging both in academia and industry, this is a pretty interesting topic. We can easily let machines perform automated checking, but automated testing is still an open research question.

I think a solid QA needs both. That is, smart automated checking but smarter manual testing. Automate the simple things to reveal trivial bugs and implement “[…] cleverly designed checks that monitor systems for important disturbances.” (Deep checking.) With regards to manual testing, not just do those tests that are too expensive to automate, explore and exploit the SUT constantly. This is where exploratory testing comes in. In “Exploratory Testing Explained”, James Bach defines it as “simultaneous learning, test design and test execution” and describes it as follows:

[…] any testing to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.

Doesn’t this sound good? Indeed, but without some guidance, exploratory testing can be a mess. Let me show you three basic ingredients, which I think make a successful recipe for exploratory testing.

1st Ingredient: Timebox ⏰

The timebox defines the amount of time available for exploratory testing, which is usually between 30 – 60 minutes. Within this time frame, people are extremely focused, which increases the chance of finding bugs. Moreover, it makes planning possible; a team can decide per sprint, week, or day, on how much time they want to spend on exploratory testing. This way, also others (e.g. developers or domain experts) can easily incorporate some testing in their daily routines.

2nd Ingredient: Testing Tour 🗺

The concept of testing tours has been around for a while. According to Michael Bolton’s blog post “Of Testing Tours and Dashboards”, the history of “touring” goes back to the mid 90’s and became more and more popular around 2005. The idea: if, for example, a tourist finds himself in a new city, he usually explores it on the basis of various topics that personally interest him, such as museums, shopping facilities, bars, etc. The very same concept can be applied to exploratory testing: instead of having no focus and no control during testing, one can specify topics around which the SUT can explored. Here are some examples:

Antisocial 🚷: always do the opposite
Garbage collector 🗑: go street by street, house by house
Landmark 🗽: most important features in different orders
Supermodel 💃: GUI only
…

Testing tours help to structure and organize exploratory testing so that, e.g., people don’t test the same feature over and over again, while other parts of the SUT aren’t tested at all. Using distinct and catchy names like the ones above, tours quickly become part of the team’s vocabulary and can also help new team members get used to the SUT.

3rd Ingredient: Protocol 📓

Protocols are usually quite unpopular; they require unpleasant writing effort and often they are read by no one. But: when it comes to exploratory testing, protocols can be a powerful tool. First of all, they simply record what has been tested. This allows to go back in time and see why a particular bug hasn’t been found. It also gives the opportunity to see what worked in the past and what didn’t. Furthermore, if a test case has proven itself to be very effective, the corresponding protocol(s) can be used as a blueprint for automating it. Let’s have a look at an example:

# 2018-11-02T09:30

Scope: my software
Tour: landmark
Timebox: 30 minutes
Legend: (C)orrect, (B)ug, (?) unknown

# Test Cases

## 1

1) open menu "x" via "y" - C
2) select "z" and type in "foo" - C
3) fill out form bottom to top - B

...

This protocol uses Markdown for formatting, which can be utilized to extract other formats such as HTML or PDF (e.g. with Pandoc). The protocol itself should be lightweight and informal. This encourages people to write and read it. Plus: a protocol like this can also help new team members to get on board easier.

Note that these are just loose guidelines. Some teams like to use Git-versioned Markdown files for their protocols, whereas others may prefer Excel sheets in a private cloud. Feel free to customize the entire process to your team’s/department’s/organization’s individual needs, but without disregarding the basic concepts.

In that sense, bon appétit!

What Are Your Top Three Conference Talks?

Daniel Kraus — Sun, 20 Aug 2017 12:26:15 +0000

I start off with these masterpieces:

What are yours?

A Formal Look at Selenium’s NoSuchElementException

Daniel Kraus — Sat, 12 Aug 2017 21:20:48 +0000

Originally posted on Medium.

As a Selenium user, you probably know the story: Out of a sudden, your tests fail due to the infamous NoSuchElementException. But this isn't something that only happens with Selenium. Most other test automation frameworks have their own (fancy) names for it, but the underlying problem is always the same: a particular UI element cannot be found.

Back in 2016, I did a small review on several Swing-compatible capture & replay tools as part of a project work at my alma mater. During this work, I stumbled across “An Extensible Heuristic-Based Framework for GUI Test Case Maintenance” by Scott McMaster and Atif M. Memon. In this paper, the authors introduced a formal model for the described situation—the GUI element identification problem. The problem typically occurs in regression testing, hence, between version n and n + i of the system under test (SUT). (Note that these versions are not necessarily stable releases, every change that is built can cause the phenomenon.) Since McMaster and Memon seemed to focus on desktop applications, it affected the terminology they have chosen. For instance, a separate window within the SUT is denoted as set W, containing various actionable GUI elements e. If you come from a web application background, you would probably speak of pages and UI elements. Sometimes, you also want to assert or verify non-actionable elements such as labels to ensure they contain a certain string. Nonetheless, the concept is essentially the same. W’ is a subsequent version of W, in which the number of elements e’ might vary (i.e. |W| ≠ |W’|, if you are unfamiliar with this notation, have a look at these quick references on logic and set theory). Now, the GUI element identification problem is defined as:

For each actionable GUI element e in W, find a corresponding (possibly modified) element e’ in W’ whose actions implement the same functionality.

In order to do so, each element e, e’ ∈ W ∪ W’ must be assigned to exactly one of the following three sets:

Deleted: elements from W with no corresponding elements in W’, therefore, they have been deleted in version n + i.

Created: elements from W’ without an assignment to elements in W, thus, they have been created in version n + i.

Maintained: elements from W which are still in W’, but that may have been modified.

If a test case uses elements from set D, then the underlying script must be adapted since mandatory elements are missing in the new version. Elements from C can affect test cases as well; for example, such an element might change the given layout, causing a different XPath somewhere else. It is usually also up to the developer or tester to decide whether these new elements need to be tested. Consequently, the accurate computation of M is the main interest of the GUI element identification problem. According to the authors, this “generally requires heuristic approaches in the absence of a model where each element has a programmer-defined unique identifier.”

Selenium offers various locator types (e.g. ID, name, or XPath), each with its own pros and cons. Additionally, more advanced test automation frameworks sometimes apply elaborate algorithms to not rely on a single locator. For instance, ReTest—the company I work for—embraces a novel paradigm called difference testing, where the complete UI state is being captured. As a result, all available attributes can be used at once. Another interesting approach is done by Testim. They incorporate historical data with the aid of machine learning to rank the locators for each element individually, which stabilizes the tests over time.

Although these (and other) techniques lead to a quite robust element recognition, they are no silver bullet … and I doubt the current state of the art offers one. But they shift the burden of the GUI element identification problem from humans to machines—reducing the amount of NoSuchElementExceptions (or whatever they are called by your framework of choice) developers and testers have to deal with.