DEV Community: Artur Neumann

[Boost]

Artur Neumann — Wed, 01 Apr 2026 07:47:30 +0000

Ashim Shrestha for JankariTech

Apr 1

Kubernetes - A Beginner's Guide to Container Orchestration

#kubernetes #devops #containers

Comments

10 min read

Configure Joplin to work with ownCloud Infinite Scale (oCIS)

Artur Neumann — Thu, 08 Jun 2023 06:02:37 +0000

I love to use Joplin to organize my notes. To synchronize the notes between different devices I have so far used the WebDAV sync option together with ownCloud 10. Now oCIS (ownCloud Infinite Scale) is the new cool kid in the cloud storage space and I would like to use it for syncing of Joplin.

Similar to ownCloud 10, oCIS offers a WebDAV API, but it has disabled basic-auth by default and the docs discourage using it in production. Instead, oCIS implements the OIDC workflow for authentication. I tried to get the OIDC authentication into Joplin, but sadly my PR for that was refused.

The workaround is the "Share via link" function of oCIS. The generated links support basic-auth by default. (Don't fear your notes will still be very private).

create a folder in oCIS (I've called it joplin)
click on the "Add people" button of that folder, that will bring you to the sharing functions
you should see the "Share via link" function
create a new link
change the permissions to "Anyone with the link can edit"
additionally set a password
to configure Joplin, open its synchronisation settings and:
1. select "WebDAV" as "Synchronisation target"
2. copy the last bit of the link URL (everything after the last /), this is your "public-link-token"
3. compose the "WebDAV URL" like <ocis-base-URL>/remote.php/dav/public-files/<public-link-token>
4. use the "public-link-token" as "WebDAV username"
5. add your password into the "WebDAV password" field

DONE!

To access your notes, one would need to know the random link AND your self selected password - so it should be pretty safe.
You could even have a different link, with a different password for every device you are using, so if you lose one device you would only have to delete that link to stop further syncing.

Automated Black-Box Tests for System Settings of a Micro-service Application

Artur Neumann — Mon, 22 May 2023 05:19:50 +0000

oCIS is the new generation open source file-sync and sharing platform, built by ownCloud GmbH, who also created the ownCloud 10 server based on the LAMP stack. The team of JankariTech is part of that development and mainly writes and maintains the automated tests.

The Challenge 🚀

Because a lot of APIs of oCIS still have to be compatible with ownCloud 10, we have used the existing API tests and (after some adjustments) ran them on oCIS to ensure backward compatibility. I have written about that process in BDD on Software Rewrite.

This worked well for a lot of test cases, and we could quickly reach good coverage. But there were some cases where the old tests could not be reused. One of those cases is specific system-wide settings. In ownCloud 10 those settings are often set by the UI, an API call, or the designated CLI tool. Also, the changes can be done during runtime and are immediately effective. This is great for an API test: change the setting => check the expected behaviour! Easy!

Because of the different architecture of oCIS, system-wide settings are set through YML files or environment variables. This is great if you run the service in a container environment, but it makes automated testing harder. Logically, the settings are only read once during the start of the service and for any change to take effect, the service needs to be restarted. How to do that as an external black-box API test?
Not to test those settings or to rely only on manual tests was never an option.

First Iteration 😵‍💫

We first started by creating different pipelines in CI that start oCIS with defined settings and ran specific tests against that instance. That gave quick results but soon became confusing because every combination of settings needs its own CI pipeline and a designated set of tests. Additionally, that system makes it hard for developers to run the tests in their environments.

Second Iteration 🥳

The next iteration was to write a small wrapper for the server (of course, it's written in Go, the same as oCIS).

This wrapper starts the oCIS server, monitors its state, and provides its own API that accepts lists of settings.
Whenever a test wants to change a system-wide setting of oCIS it sends the wrapper the desired environment variables. The wrapper then exposes those to oCIS, restarts oCIS and reports back the success of the reconfiguration. The test can then go on and contact oCIS through its APIs and check if the new behavior is as expected.

To decouple test-cases from each other, the wrapper also provides a special endpoint to completely reset oCIS to its default state.

All the kudos for this work goes to Sajan Gurung!

Future Challenges 🧑‍💻

As we move forward with extending the test-coverage to new APIs and various different use-cases of oCIS we face further challenges. For example, oCIS can be connected with different storage providers (POSIX filesystem, EOS, S3, ownCloud 10). We are currently exploring what would be the best way and layer to test the functionality of the storage drivers. Unit tests, Contract tests, E2E API tests, something else?

I will keep posting about interesting developments in this test automation area.

BDD (Behavior-driven development) mit Go

Artur Neumann — Tue, 18 May 2021 09:34:14 +0000

In Einstieg in BDD (Behavior-driven development) habe ich die Grundzüge von BDD erklärt und ihren Einsatz, um die Funktionen einer Anwendung zu beschreiben. Im Grunde genommen ist BDD dazu gedacht, alle Beteiligten zusammenzubringen und klar zu beschreiben, wie sich die "Features" einer Anwendung in verschiedenen Situationen zu verhalten haben.

Automatische Tests

Die Kommunikation und damit die Erfolgschancen der Entwicklung einer Anwendung zu verbessern, ist das Wichtigste bei BDD. Aber wir können noch einen Schritt weiter gehen und die entstandenen Feature-Beschreibungen nutzen, um die Anwendung automatisch zu testen.

(Wie schon im ersten Artikel werde ich den Quellcode und die Feature-Dateien nicht übersetzen, sondern so zeigen, wie sie in GitHub abgespeichert sind.)

Nach "Einstieg in BDD (Behavior-driven development)" haben wir ein Feature-File, das so aussieht:

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

  Scenario: converting a valid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"
    Then the HTTP-response code should be "200"
    And the response content should be "2003-07-17"

  Scenario: converting an invalid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01"
    Then the HTTP-response code should be "400"
    And the response content should be "not a valid date"

Um aus den Feature-Files automatische Tests zu machen, brauchen wir zunächst einen Interpreter, der die Gherkin Sprache versteht und die entsprechenden Tests ausführt.

Solche Interpreter gibt es für die verschiedensten Programmiersprachen. In diesem Artikel demonstriere ich, wie es mit godog package für Go funktioniert.

Um godog zu installieren, müssen wir zunächst eine einfache go.mod Datei anlegen

module github.com/JankariTech/bsDateServer

go 1.19

und dann go get github.com/cucumber/godog@v0.12.6 ausführen.

(Die Versionsnummer @v0.12.6 ist optional; ohne sie wird die neueste vorhandene Version installiert. Damit dieser Artikel aber länger verwendbar bleibt und ich ihn nicht ständig anpassen muss, gebe ich hier eine Versionsnummer an.)

Wir brauchen auch das godog Kommandozeilenwerkzeug, um das zu installieren muss
go install github.com/cucumber/godog/cmd/godog@v0.12.6

ausgeführt werden

Jetzt können wir godog mit $GOPATH/bin/godog *.feature ausführen. Die Ausgabe sollte in etwa so aussehen:

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"
    Then the HTTP-response code should be "200"
    And the response content should be "2003-07-17"

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01"
    Then the HTTP-response code should be "400"
    And the response content should be "not a valid date"

2 scenarios (2 undefined)
6 steps (6 undefined)
441.226µs

You can implement step definitions for undefined steps with these snippets:

func aRequestIsSentToTheEndpoint(arg1, arg2 string) error {
    return godog.ErrPending
}

func theHTTPresponseCodeShouldBe(arg1 string) error {
    return godog.ErrPending
}

func theResponseContentShouldBe(arg1 string) error {
    return godog.ErrPending
}

func InitializeScenario(ctx *godog.ScenarioContext) {
    ctx.Step(`^a "([^"]*)" request is sent to the endpoint "([^"]*)"$`, aRequestIsSentToTheEndpoint)
    ctx.Step(`^the HTTP-response code should be "([^"]*)"$`, theHTTPresponseCodeShouldBe)
    ctx.Step(`^the response content should be "([^"]*)"$`, theResponseContentShouldBe)
}

Godog listet alle Szenarien, die wir ausführen wollen, und sagt uns, dass es keine Ahnung hat, was es machen soll. Das ist keine Überraschung - schließlich haben wir noch keine Test-Schritte implementiert. Um das zu tun, erstellen wir eine Datei mit dem Namen bsdateServer_test.go und dem Inhalt:

package main

import (
    "github.com/cucumber/godog"
)

func aRequestIsSentToTheEndpoint(arg1, arg2 string) error {
    return godog.ErrPending
}

func theHTTPresponseCodeShouldBe(arg1 string) error {
    return godog.ErrPending
}

func theResponseContentShouldBe(arg1 string) error {
    return godog.ErrPending
}

func InitializeScenario(ctx *godog.ScenarioContext) {
    ctx.Step(`^a "([^"]*)" request is sent to the endpoint "([^"]*)"$`, aRequestIsSentToTheEndpoint)
    ctx.Step(`^the HTTP-response code should be "([^"]*)"$`, theHTTPresponseCodeShouldBe)
    ctx.Step(`^the response content should be "([^"]*)"$`, theResponseContentShouldBe)
}

Die InitializeScenario Funktion ist die Verbindung zwischen der menschenlesbaren Gherkin Sprache und dem Code, den der Computer ausführen soll. Mithilfe von RegularExpressions werden Teile der Sätze aus der Gherkin Zeile extrahiert und als Argumente an die jeweilige Funktion gesendet.
Aus When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" wird der Funktionsaufruf: aRequestIsSentToTheEndpoint("GET", "/ad-from-bs/2060-04-01")

Wenn wir wieder $GOPATH/bin/godog *.feature ausführen, sieht die Ausgabe schon anders aus:

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:8 -> aRequestIsSentToTheEndpoint
      TODO: write pending definition
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:12 -> theHTTPresponseCodeShouldBe
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:16 -> theResponseContentShouldBe

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bsdateServer_test.go:8 -> aRequestIsSentToTheEndpoint
      TODO: write pending definition
    Then the HTTP-response code should be "400"                         # bsdateServer_test.go:12 -> theHTTPresponseCodeShouldBe
    And the response content should be "not a valid date"               # bsdateServer_test.go:16 -> theResponseContentShouldBe

2 scenarios (2 pending)
6 steps (2 pending, 4 skipped)
576.1µs

Godog hat jetzt die Funktionen gefunden, die mit den jeweiligen Schritten korrespondieren, aber diese Funktionen tun, außer Fehler anzuzeigen, noch nichts.

Also implementieren wir die erste Funktion, die die HTTP Anfrage and unsere (noch nicht vorhandene) API sendet:

index c8b0144..f7ee56d 100644
--- a/bsdateServer_test.go
+++ b/bsdateServer_test.go
@@ -1,11 +1,26 @@
 package main

 import (
+    "fmt"
     "github.com/cucumber/godog"
+    "net/http"
+    "strings"
 )

-func aRequestIsSentToTheEndpoint(arg1, arg2 string) error {
-    return godog.ErrPending
+var host = "http://localhost:10000"
+var res *http.Response
+
+func aRequestIsSentToTheEndpoint(method, endpoint string) error {
+    var reader = strings.NewReader("")
+    var request, err = http.NewRequest(method, host+endpoint, reader)
+    if err != nil {
+        return fmt.Errorf("could not create request %s", err.Error())
+    }
+    res, err = http.DefaultClient.Do(request)
+    if err != nil {
+        return fmt.Errorf("could not send request %s", err.Error())
+    }
+    return nil
 }

 func theHTTPresponseCodeShouldBe(arg1 string) error {

Wir benutzen das net/http Go packet, um eine einfache HTTP Anfrage zu versenden. Der Trick bei godog ist, nil zurückzugeben, wenn kein Fehler aufgetreten ist. Das führt dazu, dass godog den Schritt als erfolgreich bewertet. Auf der anderen Seite wird ein Schritt als gescheitert markiert, wenn die Funktion irgendein Objekt zurückgibt, das die error Schnittstelle (Interface) implementiert.

Randbemerkung: die res Variable ist außerhalb der Funktion definiert, weil wir auf sie noch von anderen Funktionen zugreifen müssen.

Die Ausgabe von $GOPATH/bin/godog *.feature ist jetzt:

...
  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:13 -> aRequestIsSentToTheEndpoint
    could not send request Get "http://localhost:10000/ad-from-bs/2060-04-01": dial tcp 127.0.0.1:10000: connect: connection refused
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:31 -> theResponseContentShouldBe

...

Die HTTP Anfrage, die der Test sendet, schlägt fehl, weil nichts auf dem entsprechenden Port lauscht. Ganz Ähnlich wie bei TDD (Test Driven Development) haben wir erst den Test gebaut (oder einen Teil davon), bevor die Software implementiert wurde.

Deswegen implementieren wir jetzt einen minimal-Server, der praktisch nur den Port 10000 öffnet. Dafür kommt der folgende code in die Datei main.go und dann wird der Server mit go run main.go gestartet

package main

import (
    "fmt"
    "github.com/gorilla/mux"
    "log"
    "net/http"
)

func homePage(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Bikram Sambat Server")
}

func handleRequests() {
    myRouter := mux.NewRouter().StrictSlash(true)
    myRouter.HandleFunc("/", homePage)
    log.Fatal(http.ListenAndServe(":10000", myRouter))
}

func main() {
    handleRequests()
}

Wenn wir jetzt die Tests laufen lassen, während der Server läuft, sieht man, dass wir einen Schritt weiter gekommen sind:

  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:13 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
      TODO: write pending definition
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:31 -> theResponseContentShouldBe

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bsdateServer_test.go:13 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "400"                         # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
      TODO: write pending definition
    And the response content should be "not a valid date"               # bsdateServer_test.go:31 -> theResponseContentShouldBe

2 scenarios (2 pending)
6 steps (2 passed, 2 pending, 2 skipped)
1.849695ms

Die When Schritte funktionieren jetzt wie gewünscht. Als Nächstes müssen die Then Schritte implementiert werden:

--- a/bsdateServer_test.go
+++ b/bsdateServer_test.go
@@ -3,6 +3,7 @@ package main
 import (
     "fmt"
     "github.com/cucumber/godog"
+    "io/ioutil"
     "net/http"
     "strings"
 )
@@ -23,16 +24,23 @@ func aRequestIsSentToTheEndpoint(method, endpoint string) error {
     return nil
 }

-func theHTTPresponseCodeShouldBe(arg1 string) error {
-    return godog.ErrPending
+func theHTTPresponseCodeShouldBe(expectedCode int) error {
+    if expectedCode != res.StatusCode {
+        return fmt.Errorf("status code not as expected! Expected '%d', got '%d'", expectedCode, res.StatusCode)
+    }
+    return nil
 }

-func theResponseContentShouldBe(arg1 string) error {
-    return godog.ErrPending
+func theResponseContentShouldBe(expectedContent string) error {
+    body, _ := ioutil.ReadAll(res.Body)
+    if expectedContent != string(body) {
+        return fmt.Errorf("status code not as expected! Expected '%s', got '%s'", expectedContent, string(body))
+    }
+    return nil
 }

 func InitializeScenario(ctx *godog.ScenarioContext) {
     ctx.Step(`^a "([^"]*)" request is sent to the endpoint "([^"]*)"$`, aRequestIsSentToTheEndpoint)
-    ctx.Step(`^the HTTP-response code should be "([^"]*)"$`, theHTTPresponseCodeShouldBe)
+    ctx.Step(`^the HTTP-response code should be "(\d+)"$`, theHTTPresponseCodeShouldBe)
     ctx.Step(`^the response content should be "([^"]*)"$`, theResponseContentShouldBe)
 }

Hier lesen wir den HTTP Status Code und den Inhalt aus der HTTP Antwort und vergleichen die Werte mit den Erwartungen. Sollten die Resultate nicht mit den Erwartungen übereinstimmen, wird ein Fehler zurückgegeben.

Randnotiz: Es ist wichtig, gute Fehlermeldungen auszugeben. Das Ziel einer Fehlermeldung ist es, dem Entwickler die Fehlersuche zu erleichtern. Die Ausgabe des Tests muss den Entwickler zum Fehler führen. Diese Tests sollen schließlich nicht nur in der Entstehungsphase des Projekts benutzt werden, sondern auch später, um Regressionen zu vermeiden.

Die kleine Änderung in der Regular-Expression in InitializeScenario stellt sicher, dass nur Zahlen als HTTP Status Code akzeptiert werden.

Die Ausgabe der Tests ist jetzt:

...
  Scenario: converting a valid BS date # bs-to-ad-conversion.feature:6
    Then the HTTP-response code should be "200" # bs-to-ad-conversion.feature:8
      Error: status code not as expected! Expected '200', got '404'

  Scenario: converting an invalid BS date # bs-to-ad-conversion.feature:11
    Then the HTTP-response code should be "400" # bs-to-ad-conversion.feature:13
      Error: status code not as expected! Expected '400', got '404'


2 scenarios (2 failed)
6 steps (2 passed, 2 failed, 2 skipped)
1.766438ms

Das war zu erwarten; /ad-from-bs/ existiert noch nicht. Es ist an der Zeit, die API an sich zu implementieren.

Hier die Änderungen in main.go für eine einfache Konvertierung eines Bikram Sambat Datums in ein gregorianisches Datum:

index ae01ed0..06299b0 100644
--- a/main.go
+++ b/main.go
@@ -2,18 +2,34 @@ package main

 import (
        "fmt"
+       "github.com/JankariTech/GoBikramSambat"
        "github.com/gorilla/mux"
        "log"
        "net/http"
+       "strconv"
+       "strings"
 )

+func getAdFromBs(w http.ResponseWriter, r *http.Request) {
+       vars := mux.Vars(r)
+       dateString := vars["date"]
+       var splitedDate = strings.Split(dateString, "-")
+       day, _ := strconv.Atoi(splitedDate[2])
+       month, _ := strconv.Atoi(splitedDate[1])
+       year, _ := strconv.Atoi(splitedDate[0])
+       date, _ := bsdate.New(day, month, year)
+       gregorianDate, _ := date.GetGregorianDate()
+       fmt.Fprintf(w, gregorianDate.Format("2006-01-02"))
+}
+
 func handleRequests() {
        myRouter := mux.NewRouter().StrictSlash(true)
        myRouter.HandleFunc("/", homePage)
+       myRouter.HandleFunc("/ad-from-bs/{date}", getAdFromBs)
        log.Fatal(http.ListenAndServe(":10000", myRouter))
 }

Die Änderung ist eigentlich recht simpel: das BS Datum in Tag, Monat und Jahr aufspalten und es an die fertige GoBikramSambat Bibliothek übergeben. (Die Bibliothek wird mit go get github.com/JankariTech/GoBikramSambat installiert)

Und damit funktioniert schon das erste Szenario:

...
  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:14 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:34 -> theResponseContentShouldBe

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bsdateServer_test.go:14 -> aRequestIsSentToTheEndpoint
    could not send request Get "http://localhost:10000/ad-from-bs/60-13-01": EOF
    Then the HTTP-response code should be "400"                         # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "not a valid date"               # bsdateServer_test.go:34 -> theResponseContentShouldBe

--- Failed steps:

  Scenario: converting an invalid BS date # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bs-to-ad-conversion.feature:12
      Error: could not send request Get "http://localhost:10000/ad-from-bs/60-13-01": EOF


2 scenarios (1 passed, 1 failed)
6 steps (3 passed, 1 failed, 2 skipped)
2.035601ms

Mit ein paar kleinen Änderungen zur Behandlung von Fehlern sollte das zweite Scenario auch funktionieren:

index 8243aef..2850678 100644
--- a/main.go
+++ b/main.go
@@ -21,7 +21,11 @@ func getAdFromBs(w http.ResponseWriter, r *http.Request) {
     day, _ := strconv.Atoi(splitedDate[2])
     month, _ := strconv.Atoi(splitedDate[1])
     year, _ := strconv.Atoi(splitedDate[0])
-    date, _ := bsdate.New(day, month, year)
+    date, err := bsdate.New(day, month, year)
+    if err != nil {
+        http.Error(w, err.Error(), http.StatusBadRequest)
+        return
+    }
     gregorianDate, _ := date.GetGregorianDate()
     fmt.Fprintf(w, gregorianDate.Format("2006-01-02"))
 }


index b731d6d..9871219 100644
--- a/bsdateServer_test.go
+++ b/bsdateServer_test.go
@@ -33,7 +33,7 @@ func theHTTPresponseCodeShouldBe(expectedCode int) error {

 func theResponseContentShouldBe(expectedContent string) error {
     body, _ := ioutil.ReadAll(res.Body)
-    if expectedContent != string(body) {
+    if expectedContent != strings.TrimSpace(string(body)) {
         return fmt.Errorf("status code not as expected! Expected '%s', got '%s'", expectedContent, string(body))
     }
     return nil

Sollte die Konvertierung nicht funktionieren, wird jetzt in main.go ein Fehler ausgegeben. In den Tests benutzen wir TrimSpace, weil http.Error ein \n an den Fehlertext hängt.

Nun sollten beide Szenarien grün sein:

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:14 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:34 -> theResponseContentShouldBe

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bsdateServer_test.go:14 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "400"                         # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "not a valid date"               # bsdateServer_test.go:34 -> theResponseContentShouldBe

2 scenarios (2 passed)
6 steps (6 passed)
1.343415ms

Beispiel-Tabellen (Examples)

Um sicherzustellen, dass die Konvertierung richtig funktioniert, sollten wir noch mehr verschiedene Daten testen. Grundsätzlich sind beim Testen oft diese Dinge interessant:

höchste und niedrigste möglichen Werte - da die Umrechnung zwischen BS und AD auf Tabellen beruht, haben wir ein erstes Datum, das wir konvertieren können, und ein letztes; darüber hinaus ist keine Umrechnung möglich
Übergänge - zwischen Monaten und Jahren
andere besondere Fälle - Schaltjahre
falsche Eingaben - dreizehnter Monat, 32ter Tag, usw.
falsches Format - in unserem Fall z.b. 2012.12.03

Wir könnten für jeden Fall ein eigenes Szenario schreiben, aber das würde zu vielen Wiederholungen führen und die Datei schnell unübersichtlich machen. Besser ist es, mit dem Examples Schlüsselwort Beispiel-Tabellen anzulegen:

index 33f5d6c..9003cff 100644
--- a/bs-to-ad-conversion.feature
+++ b/bs-to-ad-conversion.feature
@@ -3,10 +3,15 @@ Feature: convert dates from BS to AD using an API
   I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
   So that I have a simple way to convert BS to AD dates, that can be used in different apps

-  Scenario: converting a valid BS date
-    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"
+  Scenario Outline: converting a valid BS date
+    When a "GET" request is sent to the endpoint "/ad-from-bs/<bs-date>"
     Then the HTTP-response code should be "200"
-    And the response content should be "2003-07-17"
+    And the response content should be "<ad-date>"
+    Examples:
+      | bs-date    | ad-date    |
+      | 2060-04-01 | 2003-07-17 |
+      | 2040-01-01 | 1983-04-14 |
+      | 2040-12-30 | 1984-04-12 |

Anstatt Scenario benutzen wir hier Scenario Outline als Schlüsselwort und am Ende des Szenarios ist eine Tabelle angefügt. Die Überschriften der Spalten werden als Variablennamen benutzt und in den Test-Schritten, in denen die Namen vorkommen, werden diese durch die Werte aus der Tabelle ersetzt.
Godog erstellt damit aus jeder Tabellenzeile ein separates Szenario.

Zusammenfassung

Die gewünschten Erwartungen an die Software in Gherkin Syntax niederzuschreiben, kann die Kommunikation zwischen allen Beteiligten verbessern und damit die Chancen auf den Erfolg des Projekts drastisch verbessern.
Die Beschreibungen der Funktionen werden zur Dokumentation.
Zusätzlich können die gleichen Beschreibungen benutzt werden, um die Software automatisch zu testen.

Wir helfen gerne bei der Umstellung auf BDD und der Erstellung von automatischen Tests:

Einstieg in BDD (Behavior-driven development)

Artur Neumann — Wed, 31 Mar 2021 11:05:39 +0000

In Demonstrating TDD (Test-driven development) in Go habe ich über TDD geschrieben und jetzt möchte ich zeigen wie BDD (Behavior-driven development) mit Go funktionieren kann.
In diesem Artikel gehe ich auf die Grundlagen von BDD ein und darauf, wie die Funktionen einer Applikation beschrieben werden. Der nächste Artikel wird dann technischer sein und sich darum drehen, wie man BDD Tests automatisch in einer Golang Umgebung laufen lässt.

Beide Artikel wurden bereits auf Englisch veröffentlicht Demonstrating BDD (Behavior-driven development) in Go

Anstatt BDD in Theorie zu beschreiben, werde ich praktische Beispiele benutzen, um die Prinzipien von BDD zu erklären.
Hier ein paar weiterführende Links zu BDD:

Wenn du weitere gute Ressourcen kennst, poste sie bitte in den Kommentaren.

Die Idee

Ich bin ein großer Fan davon, Dinge an anschaulichen Beispielen zu erklären. Daher habe ich in Demonstrating TDD (Test-driven development) in Go eine kleine Bibliothek gebaut, um Daten aus dem in Nepal benutzten Kalender (Bikram Sambat (BS), auch als Vikram Samvat bekannt) in gregorianische Daten umzuwandeln.
Diese Bibliothek verwende ich jetzt und baue eine API drumherum. (Das Projekt ist bei github zu finden)

Man könnte diese "Anforderung" auch an einen Entwickler geben und sehen, was passiert. Bei einem solch kleinen Projekt kann das gut gehen, aber es kann auch viel schief laufen:

die API wird sehr komplex
die API kann Daten umwandeln, aber behandelt Fehler nicht richtig
usw.

Wenn es schief läuft, werden viele Ressourcen verschwendet, Konflikte und Missverständnisse entstehen, usw. Es wäre also viel besser, die Anforderungen detaillierter aufzuschreiben, um das zu vermeiden. Dabei profitieren alle Beteiligten:

Als Kunde will man, dass die Applikation korrekt arbeitet (manchmal ohne zu wissen, was das genau bedeutet).
Als Entwickler will man genau das entwickeln, was gefordert ist (um Zeit zu sparen), und am Ende will man bezahlt werden.
Als Tester möchte man genau wissen, was zu testen ist und ob ein Verhalten ein Bug oder ein Feature ist.

Im Grunde genommen will man alle beteiligten Parteien (und es können mehr als die drei genannten sein) dazu bringen, ihre Erwartungen zu kommunizieren und sich auf das akzeptable Verhalten der Anwendung zu einigen. Und das ist die grundlegende Idee hinter BDD: die Kommunikation zwischen allen Beteiligten zu verbessern, um sicherzustellen, dass jeder weiß, worüber geredet wird.

Aber wie erreicht man das? Der Kunde könnte denken, dass eine Zeile zur Erklärung reicht: "API um Daten von BS zu AD und AD zu BS zu konvertieren". Der Manager möchte einen wasserdichten Vertrag und der Entwickler sagt: "Der Quellcode genügt als Dokumentation."
Eine gute Möglichkeit, das gemeinsame Verständnis zu verbessern, ist das Aufschreiben der Anforderungen in der Gherkin-Sprache. Gherkin ist eine teilstrukturierte Sprache, die so einfach ist, dass sogar eine Gurke sie verstehen kann.

Wer will was und wozu erreichen

Im Hauptverzeichnis unseres Projekts erstellen wir als Erstes eine Datei mit dem Namen bs-to-ad-conversion.feature. In dieser Datei werden wir beschreiben, wie die Konvertierung in die eine Richtung funktionieren soll.
Jede Funktion sollte in einer eigenen Datei beschrieben werden.

Randnotiz: es wird immer wieder darüber diskutiert, was eine Funktion / ein Feature ist. Konkret in unsrem Beispiel: Ist die Konvertierung in beide Richtungen ein oder zwei Features? Ist die Behandlung von Fehlern ein separates Feature oder ein Teil der Konvertierung? Wenn ich mir nicht sicher bin, ist die Lösung für mich oft einfach pragmatisch: Die Datei sollte nicht zu lang sein.

Wir fangen mit einer allgemeinen Beschreibung der Funktion an:
(Ich werde den Quellcode und die Feature-Dateien nicht übersetzen, sondern so zeigen, wie sie in GitHub abgespeichert sind. Aber Gherkin kann in jeder beliebigen Sprache verwendet werden. Je nachdem, was die gemeinsame Sprache der Beteiligten ist.)

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

Diese Zeilen sind sehr wichtig, weil sie die Fragen beantworten WER WAS und WOFÜR erreichen möchte. Wenn es niemanden gibt, der die Funktion benutzen möchte, warum sollte man die Funktion dann implementieren? Wenn man nicht weiß, was gemacht werden soll, gibt es eigentlich nichts zu programmieren. Und wenn es keinen übergeordneten Zweck für die Funktion gibt, hat sie keinen Wert.
Falls die Beteiligten (Entwickler, Kunden, Manager, QA, usw.) diese drei Fragen nicht beantworten können, sollte niemand Zeit und Geld investieren, um die Funktion zu implementieren.

Szenarien

Jede Funktion hat verschiedene Szenarien. Die Funktion "einen Artikel in den Warenkorb legen" in einem online shop kann diese Szenarien haben:

einen Artikel in den Warenkorb legen, wenn der Benutzer angemeldet ist
einen Artikel in den Warenkorb legen, wenn der Benutzer nicht angemeldet ist
einen Artikel in den Warenkorb legen, wenn der Warenkorb leer ist
einen Artikel in den Warenkorb legen, wenn sich schon Waren im Warenkorb befinden
mehrere Artikel auf einmal in den Warenkorb legen
usw.

Gut möglich, dass sich die Anwendung in jedem Szenario anders zu verhalten hat. Ist das spezifische Verhalten für einen der Beteiligten wichtig, dann sollte es auch beschrieben werden.

In Gherkin fängt jedes Szenario mit Schlüsselwort Scenario an und es folgt ein kurzer, frei wählbarer Text, der das Szenario beschreibt:

  Scenario: converting a valid BS date

  Scenario: converting an invalid BS date

Given, When, Then

Als Nächstes müssen wir das gewünschte Verhalten der Anwendung in diesem Szenario beschreiben. Dafür kennt Gherkin drei Schlüsselwörter:

Given - die Voraussetzungen für das Szenario
When - die Aktion, die getestet werden soll
Then - das gewünschte Ergebnis

Zusätzlich kennt Gherkin auch das And Schlüsselwort. Um die anderen Schlüsselwörter nicht zu wiederholen, kann man statt:

When doing A
When doing B

das And Schlüsselwort benutzen

 When doing A
 And doing B

In den meisten Fällen müssen irgendwelche Schritte ausgeführt werden, um die Anwendung in einen Zustand zu bringen, in dem die entsprechende Funktion getestet werden kann. Das kann z.B. das Anlegen von Benutzern oder das Navigieren zu einer bestimmten Stelle sein. Um die Anwendung in den testbaren Zustand zu bringen, wird das Given Schlüsselwort benutzt.
Für die Beispielanwendung in diesem Blog kommt mir da aber nichts in den Sinn, und so gehen wir jetzt weiter zum When Schlüsselwort.

Das When Schlüsselwort ist die Aktion die getestet werden soll.

  Scenario: converting a valid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"

  Scenario: converting an invalid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-13-01"

Was soll nun in den spezifischen Szenarien passieren? Was ist das Resultat, das der Benutzer beobachten kann? Wir benutzen das Then Schlüsselwort, um die Resultate zu beschreiben.

  Scenario: converting a valid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"
    Then the HTTP-response code should be "200"
    And the response content should be "2003-07-17"

  Scenario: converting an invalid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01"
    Then the HTTP-response code should be "400"
    And the response content should be "not a valid date"

Damit haben wir alle Zutaten zusammen, um die Applikation zu beschreiben:

Funktionen - eine Funktion pro Datei.
Szenarien - verschiedene Weisen, wie die Funktion sich verhalten soll.
Schritte - detaillierte Beschreibung jedes Szenarios. Jeder Schritt startet mit Given, When oder Then als Schlüsselwort.

Die komplette Beschreibung muss in natürlicher Sprache verfasst sein. Alle Beteiligten müssen die Beschreibung verstehen können. Was das genau bedeutet, ist ein Thema für einen separaten Blogpost. In unserem Fall hat der "Kunde" eine API bestellt, also sind technische Ausdrücke wie "HTTP-response code" meiner Meinung nach in Ordnung. Die Beschreibung einer grafischen Benutzeroberfläche würde wahrscheinlich weniger technische Begriffe enthalten. Grundsätzlich gilt: Benutze Worte die alle Beteiligten verstehen. BDD ist schließlich hauptsächlich dazu da, die Kommunikation zu verbessern.

Für weitere Informationen, wie die Schritte formuliert werden sollen, siehe: https://cucumber.io/docs/gherkin/reference/

Nachdem eine Funktion oder auch nur ein Szenario beschrieben wurde, kann die Entwicklung starten. In SCRUM-Sprache: jedes Feature ist eine user-story. Der ganze agile Zyklus funktioniert damit. Diese Beschreibung ist nicht nur die To-Do-Liste für die Entwicklung, sondern auch das Testprozedere für die Qualitätssicherung und sie dient auch als Dokumentation.

Im zweiten Teil des Blogs tauchen wir dann in die technischen Details ein und benutzen die Feature-Dateien, um die Anwendung automatisch zu testen.

Stay healthy and drink chiya 🍵

Artur Neumann — Fri, 19 Feb 2021 08:23:14 +0000

One thing, that you see in offices in Nepal 🇳🇵 at least once a day, is that someone makes chiya (milk tea) and brings it around to every desk.
That is a really good customs, because chiya keeps your brain working [Citation needed].

Still at JankariTech we introduced the policy that it's not permitted to deliver the drink to the desk. The reason for that policy is: we are sitting too much, and that makes us sick. To get up and to get the chiya from the kitchen gives you a good reason to move, to stretch your muscles and to activate the cardio vascular system

Even though all staff of JankariTech are still in home-offices at the moment, that policy still applies! So don't let anyone deliver you the chiya to the desk, and if anyone still does:

⬆️ get up
🚶‍♀️🖥 follow that person back to the kitchen
🤗 give the person a hug or at least say something nice
🖥🚶‍♀️ walk back to your desk
🍵 enjoy the chiya
👩‍💻 keep on coding

STAY HEALTHY AND DRINK CHIYA

P.S. For those who are so unfortunate not to to live in Nepal and don't know how to make chiya here a recipe

BDD on Software Rewrite

Artur Neumann — Mon, 15 Feb 2021 11:16:22 +0000

Imagine you have an application, it works great, but you need to rewrite it, maybe because the architecture is hitting some ceiling. There will be new features, but first, you need to make sure all the existing clients work with the new system. How do you make sure they do? It's simple: You need tests, tests that check the external behavior of the application. So if you have invested in a good UI and API test infrastructure in the first place it's a relatively easy task, and you can even do Behavior Driven Development while writing the new system without having to write new tests.

Let me show how it works at ownCloud. They are exactly in that situation, they have a stable and great product (ownCloud X written in PHP) but want to rewrite it in Go - OCIS (you can read about the reasons and background here).
We as JankariTech have been working since 2017 for ownCloud to improve the test infrastructure, increase the coverage and reduce manual testing effort. As a result of this partnership, ownCloud has a huge API and UI test-suite that covers nearly all functionality of ownCloud X. How do you transfer that over to OCIS?

Setup of the System under Test

We haven't started to use the API & UI tests from the first day of development, but only after there has been a bare minimum of functionalities implemented in OCIS. To run the tests the first challenge was how to provision the system. The goal is to have feature parity, but of course OCIS did lack a lot of functionality at that stage of development, including APIs to create and delete users, set system relevant settings, etc. Additionally, to run tests in ownCloud X, we heavily rely on the command-line tool to bring the system into a testable state. There is no equivalent (yet) in OCIS. The solution to those challenges was to extend the test-code to do slightly different things on both systems. E.g. OCIS would get users from an LDAP server, in ownCloud X we would provision users and groups through the provisioning api. Luckily we had the LDAP code already in the test-suite from testing the ownCloud LDAP app.

Those different code paths should be reduced to the setup of the SUT (System Under Test) - in Gherkin speak: Given - steps.
All external behaviour that needs to be tested and examined should be the same on both systems, the goal is to reach feature parity at some point.

By now a lot of the provisioning API is also implemented in OCIS, so we could switch off the provisioning by LDAP.

Failing tests

Of course at the beginning most of the tests would fail on OCIS, the application is not ready and does not claim to be ready. We started with skipping the failing tests and running only the tests that we knew would pass on OCIS. That way we got green CI and still prevented regressions. It was never an option to have CI failing because "we know those tests are allowed to fail". In that case the developers would have to check manually which tests are allowed to fail and which not. People would forget to do that or make wrong decisions about what is an expected failure and what not. Most important: red CI looks ugly on a pull request or worse a merge and is an embarrassment 🙈

Simply skipping failing tests also had big disadvantages. To make sure the test coverage is increased with every new feature added, the developer needs to know all the tests related to that feature and run them during or after the development. Or someone needs to run the skipped tests on a regular basis to see which of them started to pass and enable them. Both approaches are not practical because of 1. the laziness of human beings and 2. the amount of tests.

A lot of test frameworks have a feature that allows some specific tests to fail without failing the entire test-run. We took that idea further and implemented an "expected to fail" feature in CI. Tests listed in the expected-to-fail list have to fail, if they start to pass the CI run will fail.

The advantage of that above just a simple "these tests are allowed to fail" is that, after adding a feature or fixing a bug, the developer is forced to look into the tests. If tests start to pass, the only job the developer has to do, is to remove them from the expected-to-fail list, and what a joy is that 🎉.
From that point on the test has to pass in all future runs, and we are sure not to introduce any regressions. If we would only have an allowed-to-fail list, there would be no pressure to remove tests from that list, humans are humans, so they would forget or miss some. Potentially a bug could get fixed, then see some regression again and none of that would be noticed by the test-suite. So let the computers do what they are good at - automate!

human-readable code

To improve the readability of the expected-to-fail list, it got converted from a TXT file to an MD file. If you browse now the list you can jump directly to the issue report, to see WHY this particular tests fails, and you can jump to the test itself, to see WHAT exactly it does.

BDD for rewrite

With all that in place the developers

can use the existing test-suites for Behaviour Driven Development while rewriting the whole system.
know what features are missing and how far they are on the way to feature parity
don't need to rewrite all the tests for the new system
are safe from regressions for already implemented features

The only job left is to reduce the amount of expected-to-fail tests to 0, how hard can that be? 😜

BDD (Behavior Driven Development) with Flutter

Artur Neumann — Wed, 03 Jun 2020 06:46:57 +0000

This tutorial will first show how to test a flutter app using the Gherkin language and in the second part walk through an example of BDD (Behavior Driven Development) in the same App.

Flutter uses different types of tests (unit, widget, integration). You should have all types of tests in your app, most of your tests should be unit tests, less widget and a few integration tests. The test pyramid explains the principle well (using different words for the test-types).

In this tutorial I want to help you to start with integration tests but go a step further than the description in the flutter documentation and use the Gherkin language to describe the expected behavior.
The basic idea behind Gherkin/Cucumber is to have a semi-structured language to be able to define the expected behaviour and requirements in a way that all stakeholders of the project (customer, management, developer, QA, etc.) understand them. Using Gherkin helps to reduce misunderstandings, wasted resources and conflicts by improving the communication. Additionally, you get a documentation of your project and finally you can use the Gherkin files to run automated tests.

If you write the Gherkin files, before you write the code, you have reached the final level, as this is called BDD (Behaviour Driven Development)!

Here are some readings about BDD and Gherkin:

But enough theory, lets get our hands dirty. (You can find all the code of this tutorial here: https://github.com/JankariTech/flutterBDDexample)

The feature files

For the start you should have installed the flutter-tools stack and create a flutter test-drive app as explained in the get-started document

Inside the app folder create a folder called test_driver and inside another one called features. In features we will place all the Gherkin descriptions of the expected app behavior. So create here a file called: increment_counter.feature

We start the feature file with a very general description of the feature:

Feature: Increment Counter

  As the good shepherd
  I want to be able to count my sheep
  So that I notice if one is missing

The first line is just a title of the feature, the other three lines should answer the questions Who, wants to achieve what and why with this particular feature. If you cannot answer those questions for a particular feature of your app then you actually should not implement that feature, there is no use-case for it.

Next we have to describe the specific behavior of the app. For that Gherkin provides 3 different keywords:

Given - prerequisites for the scenario
When - the action to be tested
Then - the desired observable outcome

Add a scenario to the feature file.

  Scenario: Counter increases when the button is pressed
    Given the counter is set to "0"
    When I tap the "increment" button 10 times
    Then I expect the "counter" to be "10"

Later we will add more scenarios to the app, the feature might be the same, but in different scenarios it might have to react differently.

Now we can start the app and use our behaviour description to check if it works as it should.

Test-automation

Running manual tests from a description is nice, but not enough for us, we want to save time and reduce possible mistakes by running the tests automatically.

To interpret the Gherkin file and interact with the app we are using the flutter_gherkin package. Install it by placing flutter_gherkin: in the pubspec.yaml inside the dev_depencencies section.

dev_dependencies:
  flutter_test:
    sdk: flutter
  flutter_gherkin:

and run flutter pub get.

Now we also need some glue-code and configuration.

Inside test_driver create a file called app.dart with the content

import '../lib/main.dart';
import 'package:flutter/widgets.dart';
import 'package:flutter_driver/driver_extension.dart';

void main() {
  enableFlutterDriverExtension();
  runApp(MyApp());
}

and a file called app_test.dart with the content:

import 'dart:async';
import 'package:flutter_gherkin/flutter_gherkin.dart';
import 'package:gherkin/gherkin.dart';
import 'package:glob/glob.dart';

Future<void> main() {
  final config = FlutterTestConfiguration()
    ..features = [Glob(r"test_driver/features/**.feature")]
    ..reporters = [
      ProgressReporter(),
      TestRunSummaryReporter(),
      JsonReporter(path: './report.json')
    ]
    ..stepDefinitions = []
    ..customStepParameterDefinitions = []
    ..restartAppBetweenScenarios = true
    ..targetAppPath = "test_driver/app.dart"
    ..exitAfterTestRun = true; // set to false if debugging to exit cleanly
  return GherkinRunner().execute(config);
}

That was all we need to do for the installation, now we have to tell the test-software what actually to do with our Given, When and Then steps.
The library gives us some built-in steps, that should work "out-of-the-box" but others we need to implement ourself.
In our example the Then step is a built-in step but the Given and the When step have to be implemented. So let's do that. Inside test_driver create a folder called steps and in there create a file called tap_button_n_times_step.dart with the content:

import 'package:flutter_driver/flutter_driver.dart';
import 'package:flutter_gherkin/flutter_gherkin.dart';
import 'package:gherkin/gherkin.dart';

class GivenCounterIsSetTo extends Given1WithWorld<String, FlutterWorld> {
  @override
  RegExp get pattern => RegExp(r"the counter is set to {string}");

  @override
  Future<void> executeStep(String expectedCounter) async {
    final locator = find.byValueKey("counter");
    final actualCount = await FlutterDriverUtils.getText(world.driver, locator);
    expectMatch(actualCount, expectedCounter);
  }
}

class TapButtonNTimesStep extends When2WithWorld<String, int, FlutterWorld> {
  @override
  RegExp get pattern => RegExp(r"I tap the {string} button {int} times");

  @override
  Future<void> executeStep(String buttonKey, int amount) async {
    final locator = find.byValueKey(buttonKey);
    for (var i = 0; i < amount; i += 1) {
      await FlutterDriverUtils.tap(world.driver, locator, timeout: timeout);
    }
  }
}

In this file we have two classes, one for every step we want to implement. Every class extends an abstract class. The Given step extends a class which name starts with Given and analogously the When step extends a class which name starts with When. Then there is a number in the class name. That number tells how many parameters we can pass from the step to the implementation. In Given the counter is set to "0" there is one parameter (the 0) and in When I tap the "increment" button 10 times two (the button name, and the amount of taps).

The last part of the class to extend is WithWorld that gives us access to the Flutter context.

Next there is a variable called pattern with a regular expression, that is used to associate the step in the feature file with the class.

Last there is a function executeStep. This function receives the parameters from the feature file and finally does all the hard work.
In both cases it finds the element on the screen we want to interact with by using the find.byValueKey() method and then in the case of the Given step, gets the text of the element and checks if its as expected or, in the case of the When step, taps the button.

Similarly our Then step (remember it's a built-in step) will use the same find.byValueKey() method to get the value and assert the content. If you are interested in the implementation, the step is defined in flutter_gherkin-<version>/lib/src/flutter/steps/then_expect_element_to_have_value_step.dart.

The issue now is that the example code does not have any keys defined in the widgets. The test-code would not be able to locate the elements.
So edit the main.dart file and add key: Key('counter'), to the counter widget and key: Key('increment'), to the button widget.

You could also use find.byTooltip, find.Type or find.bySemanticsLabel.

Next the new .dart file with the step definitions need to be imported in app_test.dart:
import 'steps/tap_button_n_times_step.dart';

Additionally every class we add in the steps definitions we also have to register in the stepDefinitions array in app_test.dart, the line has to be:
..stepDefinitions = [TapButtonNTimesStep(), GivenCounterIsSetTo()]

Remember: The step Then I expect the "counter" to be "10" is a built-in-step. So we don't need to write any code for it, it will look for a text-widget with the key counter and assert its value.

run the tests

connect your phone or start the emulator
run dart test_driver/app_test.dart

after a while you should see an output like:

Running scenario: Counter increases when the button is pressed # ./test_driver/features/increment_counter.feature:5
   √ Given the counter is set to "0" # ./test_driver/features/increment_counter.feature:6 took 146ms
   √ When I tap the "increment" button 10 times # ./test_driver/features/increment_counter.feature:7 took 6420ms
   √ Then I expect the "counter" to be "10" # ./test_driver/features/increment_counter.feature:8 took 72ms
PASSED: Scenario Counter increases when the button is pressed # ./test_driver/features/increment_counter.feature:5
Restarting Flutter app under test
1 scenario (1 passed)
3 steps (3 passed)
0:00:16.767000

and the app working on the phone screen.

BDD (this time for real)

We know now how to write feature files and how to run automated tests from them, but that hasn't been BDD yet. We have only written a test for an existing feature in the app. To do BDD we have first to write the expected behaviour and then start coding.

1. write down the expected behaviour

Let's say we not only want to have a button to increment the counter, but also be able to decrement it. So in features create a file called decrement_counter.feature with this content:

Feature: Decrement Counter
  As the good shepherd
  I want to be able to decrement the count of my sheep when one is lost
  So that I can have extra joy incrementing the counter when I find the lost sheep

  Scenario: Counter decreases when the (-) button is pressed
    Given the counter is set to "10"
    When I tap the "decrement" button 1 time
    Then I expect the "counter" to be "9"

Trying to run this test we will have multiple issues:

the Given step only asserts the counter, but does not set it to a specific value
the regex will not match the When step because it says time and not times
there is no functionality and no button to decrement the counter

2. make the tests pass

For the first issue we would need to pre-set the counter with a value, but as we are doing end-to-end tests and acting as a user, the only way for the user to get the counter up to a specific value is to press the (+) button. Our test-code will do the same. (Side note: that will take time during test-execution, the faster option would be to have a back-channel to pre-set the value e.g. Data Handlers, but I could not make it work).

So lets refactor our step definition, so that the Given step pre-sets the counter to the expected value:

index e4eea51..e2e1a38 100644
--- a/myapp/test_driver/steps/tap_button_n_times_step.dart
+++ b/myapp/test_driver/steps/tap_button_n_times_step.dart
@@ -8,6 +8,7 @@ class GivenCounterIsSetTo extends Given1WithWorld<String, FlutterWorld> {

   @override
   Future<void> executeStep(String expectedCounter) async {
+    await tapButton(world, timeout, "increment", int.parse(expectedCounter));
     final locator = find.byValueKey("counter");
     final actualCount = await FlutterDriverUtils.getText(world.driver, locator);
     expectMatch(actualCount, expectedCounter);
@@ -20,9 +21,13 @@ class TapButtonNTimesStep extends When2WithWorld<String, int, FlutterWorld> {

   @override
   Future<void> executeStep(String buttonKey, int amount) async {
-    final locator = find.byValueKey(buttonKey);
-    for (var i = 0; i < amount; i += 1) {
-      await FlutterDriverUtils.tap(world.driver, locator, timeout: timeout);
-    }
+    await tapButton(world, timeout, buttonKey, amount);
+  }
+}
+
+Future<void> tapButton(FlutterWorld world, Duration timeout, String buttonKey, int amount) async {
 +  final locator = find.byValueKey(buttonKey);
 +  for (var i = 0; i < amount; i += 1) {
 +    await FlutterDriverUtils.tap(world.driver, locator, timeout: timeout);
    }

The second issue should be fixed easily with some regex-magic. Just place the s of times in a non-capturing regex group:
RegExp get pattern => RegExp(r"I tap the {string} button {int} time(?:s|)");
Non-capturing because a normal group would be passed as argument to TapButtonNTimesStep.

To fix the last issue, we actually need to implement a new functionality in the app. We need a decrement button in main.dart.

index 8795daa..068f558 100644
--- a/myapp/lib/main.dart
+++ b/myapp/lib/main.dart
@@ -63,6 +63,12 @@ class _MyHomePageState extends State<MyHomePage> {
     });
   }

+  void _decrementCounter() {
+    setState(() {
+      _counter--;
+    });
+  }
+
   @override
   Widget build(BuildContext context) {
     // This method is rerun every time setState is called, for instance as done
@@ -95,7 +101,7 @@ class _MyHomePageState extends State<MyHomePage> {
           // center the children vertically; the main axis here is the vertical
           // axis because Columns are vertical (the cross axis would be
           // horizontal).
-          mainAxisAlignment: MainAxisAlignment.center,
+          mainAxisAlignment: MainAxisAlignment.spaceEvenly,
           children: <Widget>[
             Text(
               'You have pushed the button this many times:',
@@ -105,15 +111,28 @@ class _MyHomePageState extends State<MyHomePage> {
               key: Key('counter'),
               style: Theme.of(context).textTheme.headline4,
             ),
+            Row(
+              mainAxisAlignment: MainAxisAlignment.spaceEvenly,
+                children: <Widget>[
+                  FloatingActionButton(
+                    onPressed: _decrementCounter,
+                    key: Key('decrement'),
+                    tooltip: 'decrement',
+                    child: Icon(Icons.remove),
+                  ),
+                  FloatingActionButton(
+                    // Provide a Key to this button. This allows finding this
+                    // specific button inside the test suite, and tapping it.
+                    key: Key('increment'),
+                    onPressed: _incrementCounter,
+                    tooltip: 'Increment',
+                    child: Icon(Icons.add),
+                  ),
+                ]
+            )
           ],
         ),
       ),
-      floatingActionButton: FloatingActionButton(
-        onPressed: _incrementCounter,
-        key: Key('increment'),
-        tooltip: 'Increment',

Now the tests should pass:

Running scenario: Counter decreases when the (-) button is pressed # ./test_driver/features/decrement_counter.feature:5
   √ Given the counter is set to "10" # ./test_driver/features/decrement_counter.feature:6 took 2877ms
   √ When I tap the "decrement" button 1 time # ./test_driver/features/decrement_counter.feature:7 took 255ms
   √ Then I expect the "counter" to be "9" # ./test_driver/features/decrement_counter.feature:8 took 43ms
PASSED: Scenario Counter decreases when the (-) button is pressed # ./test_driver/features/decrement_counter.feature:5
Restarting Flutter app under test
...
Running scenario: Counter increases when the button is pressed # ./test_driver/features/increment_counter.feature:5
   √ Given the counter is set to "0" # ./test_driver/features/increment_counter.feature:6 took 46ms
   √ When I tap the "increment" button 10 times # ./test_driver/features/increment_counter.feature:7 took 2835ms
   √ Then I expect the "counter" to be "10" # ./test_driver/features/increment_counter.feature:8 took 84ms
PASSED: Scenario Counter increases when the button is pressed # ./test_driver/features/increment_counter.feature:5
Restarting Flutter app under test
2 scenarios (2 passed)
6 steps (6 passed)
0:00:22.451000

3. multiply the scenarios by using an example table

Now we might want to test more cases than only tapping the (-) button once. For that we can just copy and paste the existing scenario, or more elegantly we add an example table:

  Scenario Outline: Counter decreases when the (-) button is pressed
    Given the counter is set to "<initial-counter>"
    When I tap the "decrement" button <decrement> time
    Then I expect the "counter" to be "<final-counter>"
    Examples:
      | initial-counter | decrement | final-counter |
      | 10              | 1         | 9             |
      | 10              | 9         | 1             |
      | 3               | 3         | 0             |

This will run the same scenario three different times with the values in the table substituted into the steps.

Running scenario: Counter decreases when the (-) button is pressed (Example 1) # ./test_driver/features/decrement_counter.feature:5
   √ Given the counter is set to "10" # ./test_driver/features/decrement_counter.feature:6 took 2658ms
   √ When I tap the "decrement" button 1 time # ./test_driver/features/decrement_counter.feature:7 took 243ms
   √ Then I expect the "counter" to be "9" # ./test_driver/features/decrement_counter.feature:8 took 60ms
PASSED: Scenario Counter decreases when the (-) button is pressed (Example 1) # ./test_driver/features/decrement_counter.feature:5

...

Running scenario: Counter decreases when the (-) button is pressed (Example 2) # ./test_driver/features/decrement_counter.feature:5
   √ Given the counter is set to "10" # ./test_driver/features/decrement_counter.feature:6 took 3325ms
   √ When I tap the "decrement" button 9 time # ./test_driver/features/decrement_counter.feature:7 took 2457ms
   √ Then I expect the "counter" to be "1" # ./test_driver/features/decrement_counter.feature:8 took 25ms
PASSED: Scenario Counter decreases when the (-) button is pressed (Example 2) # ./test_driver/features/decrement_counter.feature:5

...

Running scenario: Counter decreases when the (-) button is pressed (Example 3) # ./test_driver/features/decrement_counter.feature:5
   √ Given the counter is set to "3" # ./test_driver/features/decrement_counter.feature:6 took 878ms
   √ When I tap the "decrement" button 3 time # ./test_driver/features/decrement_counter.feature:7 took 877ms
   √ Then I expect the "counter" to be "0" # ./test_driver/features/decrement_counter.feature:8 took 63ms
PASSED: Scenario Counter decreases when the (-) button is pressed (Example 3) # ./test_driver/features/decrement_counter.feature:5

4. repeat

What about negative values? If a shepherd is using this app to count the sheep, there is no point to have a negative counter. To say it in Gherkin:

  Scenario: Counter should not be negative
    Given the counter is set to "0"
    When I tap the "decrement" button 1 time
    Then I expect the "counter" to be "0"

You also could add that to the previous table, but I would argue that it is another requirement and its easier to understand the feature file if its written out in a separate Scenario.

Running this test fails with:

   × Then I expect the "counter" to be "0" # ./test_driver/features/decrement_counter.feature:18 took 97ms 
      Expected: '0'
  Actual: '-1'
   Which: is different.
          Expected: 0
            Actual: -1
                    ^
           Differ at offset 0

The counter becomes negative. Let's fix it:

index 068f558..5e0d8d0 100644
--- a/myapp/lib/main.dart
+++ b/myapp/lib/main.dart
@@ -65,7 +65,9 @@ class _MyHomePageState extends State<MyHomePage> {

   void _decrementCounter() {
     setState(() {
-      _counter--;
+      if (_counter > 0) {
+        _counter--;
+      }
     });
   }

conclusion

You have seen how to write Gherkin files and how to run them as automated tests for a flutter application.
I personally find flutter_gherkin a bit more complicated than other BDD frameworks, but it's possible, and I believe using BDD will improve the quality of your project greatly.

If you need any help with the test-coverage of your app, BDD or other test-related topics, please contact us @JankariTech

Demonstrating BDD (Behavior-driven development) in Go

Artur Neumann — Wed, 05 Feb 2020 08:55:11 +0000

In Demonstrating TDD (Test-driven development) in Go I've written about TDD and this time I want to demonstrate BDD (Behavior-driven development) with Go.

I will not explain all principles of BDD upfront, but explain some of them as I use them in the example. You can read more about them here:

If you have more good resources, please post them in the comment section.

The basic idea

I'm a fan of explaining things with real examples, that's why in Demonstrating TDD (Test-driven development) in Go I've created that small library to convert from Bikram Sambat (BS) (also called Vikram Samvat) dates to Gregorian dates and vice-versa. Now I want to use that library to create an API-driven service to do the conversion. (The project can be found on github)

One could now give that "requirement" to a developer and see what happens. With that kind of small project, chances are, something good will come out, but bad things might also happen:

the API will be super-complex and over-engineered
the API does the conversion, but does not handle errors correctly
etc.

So there is a lot of potential for wasted resources, conflicts, misunderstandings etc. So it would be better to write down the requirements in more detail, because:

As customer you want your application to behave correctly (sometimes without knowing exactly what that means).
As developer your want to develop exactly what is requested and needed (to save time) and get paid afterwards.
As as QA-person, you want to know what you have to test, and you want to know what is a bug and what is a feature.

So basically the goal is to get all the stakeholders (there might be more than the listed 3) to communicate and agree on what should be the acceptable behavior of the application. And that is in a nutshell the idea of BDD: improve the communication between stakeholders so that everybody knows what is talked about.

But how to do that? The customer might think that the one-line explanation: "API to convert dates from BS to AD and vice-versa" is enough, the manager wants to write a contract and the developer says: "code is documentation enough".
A good way to bring everybody on the same page is to describe the features of an application using the Gherkin language. Its a semi-structured language, that is so simple a cucumber could understand.

Who wants to achieve what and why?

In the project folder we create a new file called bs-to-ad-conversion.feature. Here we want to describe the feature to convert the dates in one direction. The description of every feature of the app is supposed to go into a separate file.

Side note: there is always the discussion what is a "feature"? In our example: is the conversion in both directions one or two features? Is the error-handling a separate feature or a part of the conversion feature? If you are not sure, be practical and simply make sure the file does not get too long.

We start the feature file with a very general description of the feature:

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

These lines are very important. They answer the question WHO wants to achieve WHAT with that feature and WHY. If you don't know who will use that feature, why do you implement it? If there is nothing to achieve with that feature, you actually don't have a feature. And if there is no reason to use that feature, it doesn't have a business value. So if the stakeholders (developer, customer, manager, QA, etc.) cannot answer these 3 questions, nobody really should spend time and money to implement it.

Scenarios

Every feature has different scenarios. A "add item to shopping basket"-feature in an online-shop could have scenarios like:

adding item to the basket while user is logged in
adding item to the basket while user is not logged in
adding item to the basket when the card is empty
adding item to the basket when there is already the same item in the basket
adding multiple items to the basket at once
etc.

In every scenario your app might behave differently. If that specific behavior in that scenario matters for one or more stakeholders, better describe it.

In Gherkin we have to start the scenario description with the Scenario: keyword and a short free-text sentence:

  Scenario: converting a valid BS date

  Scenario: converting an invalid BS date

Given, When, Then

Now we want to describe the specific behavior of the app in that scenario. For that Gherkin provides 3 different keywords:

Given - prerequisites for the scenario
When - the action to be tested
Then - the desired observable outcome

Additionally there is And, if you have multiple of one of the above, you don't need to write

When doing A
When doing B

but you can use And (it just sounds and reads nicer)

 When doing A
 And doing B

For a complex application there will be most-likely some steps to bring the application into the state that you want to test (e.g. create users, navigate to a specific page, etc), for those prerequisites you should use the Given keyword.
For our app, I cannot really think of anything. So I skip over to the When keyword.

The When keyword is for the action (or multiple) you really want to test.

  Scenario: converting a valid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"

  Scenario: converting an invalid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-13-01"

Now, what should happen in those specific scenarios? What is the observable outcome? Use the Then keyword to describe that (if there are different outcomes connect multiple Thens with Ands)

  Scenario: converting a valid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"
    Then the HTTP-response code should be "200"
    And the response content should be "2003-07-17"

  Scenario: converting an invalid BS date
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01"
    Then the HTTP-response code should be "400"
    And the response content should be "not a valid date"

So as pieces of our description we have:

features - one feature per file
scenarios - different ways that the feature should behave
steps - detailed description of every scenario. Every step starts with Given, When or Then

All these pieces have to be written in a natural language, that all stakeholders can understand. What that means in detail would be a whole own post. In our case the "customer", requested an API, so IMO using technical terms like "HTTP-response code" should be OK. If you describe a GUI, the descriptions should be probably even less technical. The bottom line is: use words that all understand. Remember: BDD is all about improving communication!

For more information about how to phrase the steps definitions see: https://cucumber.io/docs/gherkin/reference/

After specifying one feature (or even one scenario) the developer could start developing. In SCRUM-terms: one feature is one user-story, so you do all your agile development cycle with it. Create one or multiple, put them in sprints, work on them, test them, etc. The description is not only the ToDo list for the developer, but also the test-procedure for QA and the documentation.

Test it automatically

We could stop there, but there is a great bonus-point: let's use these descriptions to run automatic tests.

For that we need software that interprets the Gherkin language and runs code that executes the tests. For Go there is the godog package.

To install godog we fist have to create a simple go.mod file with the content

module github.com/JankariTech/bsDateServer

go 1.19

and then run go get github.com/cucumber/godog@v0.12.6

If you are running within $GOPATH, you will need to set GO111MODULE=on, as:
(The version number @v0.12.6 is optional, if it's not given the latest version will be installed. I set the version here to make sure this blog-post stays valid also when s.th. changes in godog)

We also, need the godog cli command to run our tests. Run the following command to add the godog cli to $GOPATH/bin

go install github.com/cucumber/godog/cmd/godog@v0.12.6

Now you should be able to run godog with $GOPATH/bin/godog *.feature and the output would be something like:

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"
    Then the HTTP-response code should be "200"
    And the response content should be "2003-07-17"

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01"
    Then the HTTP-response code should be "400"
    And the response content should be "not a valid date"

2 scenarios (2 undefined)
6 steps (6 undefined)
281.282µs

You can implement step definitions for undefined steps with these snippets:

func aRequestIsSentToTheEndpoint(arg1, arg2 string) error {
    return godog.ErrPending
}

func theHTTPresponseCodeShouldBe(arg1 string) error {
    return godog.ErrPending
}

func theResponseContentShouldBe(arg1 string) error {
    return godog.ErrPending
}

func InitializeScenario(ctx *godog.ScenarioContext) {
    ctx.Step(`^a "([^"]*)" request is sent to the endpoint "([^"]*)"$`, aRequestIsSentToTheEndpoint)
    ctx.Step(`^the HTTP-response code should be "([^"]*)"$`, theHTTPresponseCodeShouldBe)
    ctx.Step(`^the response content should be "([^"]*)"$`, theResponseContentShouldBe)
}

Godog lists all the scenarios we want to run and tells us that it has no idea what to do with them, because we haven't implemented any of the steps. Now we actually need to write code to tell godog how to execute our scenarios.

For that create a file with the name bsdateServer_test.go and the content:

package main

import (
    "github.com/cucumber/godog"
)

func aRequestIsSentToTheEndpoint(arg1, arg2 string) error {
    return godog.ErrPending
}

func theHTTPresponseCodeShouldBe(arg1 string) error {
    return godog.ErrPending
}

func theResponseContentShouldBe(arg1 string) error {
    return godog.ErrPending
}

func InitializeScenario(ctx *godog.ScenarioContext) {
    ctx.Step(`^a "([^"]*)" request is sent to the endpoint "([^"]*)"$`, aRequestIsSentToTheEndpoint)
    ctx.Step(`^the HTTP-response code should be "([^"]*)"$`, theHTTPresponseCodeShouldBe)
    ctx.Step(`^the response content should be "([^"]*)"$`, theResponseContentShouldBe)
}

In the InitializeScenario function we have the link between the human-readable Gherkin, and the function that the computer has to execute for that step. The output of $GOPATH/bin/godog now looks a bit different:

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:8 -> aRequestIsSentToTheEndpoint
      TODO: write pending definition
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:12 -> theHTTPresponseCodeShouldBe
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:16 -> theResponseContentShouldBe

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bsdateServer_test.go:8 -> aRequestIsSentToTheEndpoint
      TODO: write pending definition
    Then the HTTP-response code should be "400"                         # bsdateServer_test.go:12 -> theHTTPresponseCodeShouldBe
    And the response content should be "not a valid date"               # bsdateServer_test.go:16 -> theResponseContentShouldBe

2 scenarios (2 pending)
6 steps (2 pending, 4 skipped)
576.1µs

Godog found the functions that correspond to every step, but those don't do anything yet, just returning an error.

Let's implement the first function to send the request:

index c8b0144..f7ee56d 100644
--- a/bsdateServer_test.go
+++ b/bsdateServer_test.go
@@ -1,11 +1,26 @@
 package main

 import (
+    "fmt"
     "github.com/cucumber/godog"
+    "net/http"
+    "strings"
 )

-func aRequestIsSentToTheEndpoint(arg1, arg2 string) error {
-    return godog.ErrPending
+var host = "http://localhost:10000"
+var res *http.Response
+
+func aRequestIsSentToTheEndpoint(method, endpoint string) error {
+    var reader = strings.NewReader("")
+    var request, err = http.NewRequest(method, host+endpoint, reader)
+    if err != nil {
+        return fmt.Errorf("could not create request %s", err.Error())
+    }
+    res, err = http.DefaultClient.Do(request)
+    if err != nil {
+        return fmt.Errorf("could not send request %s", err.Error())
+    }
+    return nil
 }

 func theHTTPresponseCodeShouldBe(arg1 string) error {

Here we create a request and send it using the net/http package. The trick in godog is to return nil if everything goes well, that will make the step pass. If a step function returns something that implements the error interface the step will fail.

BTW: the res variable is defined outside the function because we need to access it from other steps also.

Running godog now gives us this result

...
  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:13 -> aRequestIsSentToTheEndpoint
    could not send request Get "http://localhost:10000/ad-from-bs/2060-04-01": dial tcp 127.0.0.1:10000: connect: connection refused
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:31 -> theResponseContentShouldBe

...

It cannot connect to the server, because nothing is listening on that port. Let's change that. For a minimal implementation of a server waiting on the port put this code into main.go and run it with go run main.go

package main

import (
    "fmt"
    "github.com/gorilla/mux"
    "log"
    "net/http"
)

func homePage(w http.ResponseWriter, r *http.Request){
    fmt.Fprintf(w, "Bikram Sambat Server")
}

func handleRequests() {
    myRouter := mux.NewRouter().StrictSlash(true)
    myRouter.HandleFunc("/", homePage)
    log.Fatal(http.ListenAndServe(":10000", myRouter))
}

func main() {
    handleRequests()
}

Now we are a step further:

  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:13 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
      TODO: write pending definition
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:31 -> theResponseContentShouldBe

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bsdateServer_test.go:13 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "400"                         # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
      TODO: write pending definition
    And the response content should be "not a valid date"               # bsdateServer_test.go:31 -> theResponseContentShouldBe

2 scenarios (2 pending)
6 steps (2 passed, 2 pending, 2 skipped)
1.849695ms

The When step passed, it sent the request, but the first Then step failed as expected, because it's not implemented yet.

Let's do that:

--- a/bsdateServer_test.go
+++ b/bsdateServer_test.go
@@ -3,6 +3,7 @@ package main
 import (
     "fmt"
     "github.com/cucumber/godog"
+    "io/ioutil"
     "net/http"
     "strings"
 )
@@ -23,16 +24,23 @@ func aRequestIsSentToTheEndpoint(method, endpoint string) error {
     return nil
 }

-func theHTTPresponseCodeShouldBe(arg1 string) error {
-    return godog.ErrPending
+func theHTTPresponseCodeShouldBe(expectedCode int) error {
+    if expectedCode != res.StatusCode {
+        return fmt.Errorf("status code not as expected! Expected '%d', got '%d'", expectedCode, res.StatusCode)
+    }
+    return nil
 }

-func theResponseContentShouldBe(arg1 string) error {
-    return godog.ErrPending
+func theResponseContentShouldBe(expectedContent string) error {
+    body, _ := ioutil.ReadAll(res.Body)
+    if expectedContent != string(body) {
+        return fmt.Errorf("status code not as expected! Expected '%s', got '%s'", expectedContent, string(body))
+    }
+    return nil
 }

 func InitializeScenario(ctx *godog.ScenarioContext) {
     ctx.Step(`^a "([^"]*)" request is sent to the endpoint "([^"]*)"$`, aRequestIsSentToTheEndpoint)
-    ctx.Step(`^the HTTP-response code should be "([^"]*)"$`, theHTTPresponseCodeShouldBe)
+    ctx.Step(`^the HTTP-response code should be "(\d+)"$`, theHTTPresponseCodeShouldBe)
     ctx.Step(`^the response content should be "([^"]*)"$`, theResponseContentShouldBe)
 }

Here we simply get the status code and the result body and compare it with the expectation. If it does not match, return an error. Make sure you show good error messages, the goal is to direct the developer as much as possible to the problem. The clearer the message is the quicker the developer will be able to fix the issue. Remember: these tests will not only be used during the initial development but also in the future to prevent regressions.

The regular-expression change in the FeatureContext just makes sure that we only accept decimal numbers in that step.

Now the tests fail with:

...
  Scenario: converting a valid BS date # bs-to-ad-conversion.feature:6
    Then the HTTP-response code should be "200" # bs-to-ad-conversion.feature:8
      Error: status code not as expected! Expected '200', got '404'

  Scenario: converting an invalid BS date # bs-to-ad-conversion.feature:11
    Then the HTTP-response code should be "400" # bs-to-ad-conversion.feature:13
      Error: status code not as expected! Expected '400', got '404'

2 scenarios (2 failed)
6 steps (2 passed, 2 failed, 2 skipped)
1.766438ms

Why? Because the endpoint does not exist! The server returns 404. It's time to write the software itself!

Here are the changes in main.go to do a simple conversion:

index ae01ed0..06299b0 100644
--- a/main.go
+++ b/main.go
@@ -2,18 +2,34 @@ package main

 import (
        "fmt"
+       "github.com/JankariTech/GoBikramSambat"
        "github.com/gorilla/mux"
        "log"
        "net/http"
+       "strconv"
+       "strings"
 )

+func getAdFromBs(w http.ResponseWriter, r *http.Request) {
+       vars := mux.Vars(r)
+       dateString := vars["date"]
+       var splitedDate = strings.Split(dateString, "-")
+       day, _ := strconv.Atoi(splitedDate[2])
+       month, _ := strconv.Atoi(splitedDate[1])
+       year, _ := strconv.Atoi(splitedDate[0])
+       date, _ := bsdate.New(day, month, year)
+       gregorianDate, _ := date.GetGregorianDate()
+       fmt.Fprintf(w, gregorianDate.Format("2006-01-02"))
+}
+
 func handleRequests() {
        myRouter := mux.NewRouter().StrictSlash(true)
        myRouter.HandleFunc("/", homePage)
+       myRouter.HandleFunc("/ad-from-bs/{date}", getAdFromBs)
        log.Fatal(http.ListenAndServe(":10000", myRouter))
 }

Basically: split the incoming string, send it to the GoBikramSambat lib and return the formatted result.

And with that the first scenario passes:

...
  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:14 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:34 -> theResponseContentShouldBe

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bsdateServer_test.go:14 -> aRequestIsSentToTheEndpoint
    could not send request Get "http://localhost:10000/ad-from-bs/60-13-01": EOF
    Then the HTTP-response code should be "400"                         # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "not a valid date"               # bsdateServer_test.go:34 -> theResponseContentShouldBe

--- Failed steps:

  Scenario: converting an invalid BS date # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bs-to-ad-conversion.feature:12
      Error: could not send request Get "http://localhost:10000/ad-from-bs/60-13-01": EOF


2 scenarios (1 passed, 1 failed)
6 steps (3 passed, 1 failed, 2 skipped)
2.035601ms

With a bit of error-handling we should be able to make the other one pass also.

index 06299b0..a62eaf6 100644
--- a/main.go
+++ b/main.go
@@ -21,7 +21,11 @@ func getAdFromBs(w http.ResponseWriter, r *http.Request) {
        day, _ := strconv.Atoi(splitedDate[2])
        month, _ := strconv.Atoi(splitedDate[1])
        year, _ := strconv.Atoi(splitedDate[0])
-       date, _ := bsdate.New(day, month, year)
+       date, err := bsdate.New(day, month, year)
+       if err != nil {
+               http.Error(w, err.Error(), http.StatusBadRequest)
+               return
+       }
        gregorianDate, _ := date.GetGregorianDate()
        fmt.Fprintf(w, gregorianDate.Format("2006-01-02"))
 }


index 3156498..16c48ab 100644
--- a/bsdateServer_test.go
+++ b/bsdateServer_test.go
@@ -35,7 +35,7 @@ func theHTTPresponseCodeShouldBe(expectedCode int) error {

 func theResponseContentShouldBe(expectedContent string) error {
     body, _ := ioutil.ReadAll(res.Body)
-    if expectedContent != string(body) {
+    if expectedContent != strings.TrimSpace(string(body)) {
         return fmt.Errorf("status code not as expected! Expected '%s', got '%s'", expectedContent, string(body))
     }
     return nil

In main.go we now spit out an Error if the conversion does not work and in the tests we trim the body, because http.Error likes to send an \n at the end of the body.

Finally, the scenarios pass:

Feature: convert dates from BS to AD using an API
  As an app-developer in Nepal
  I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
  So that I have a simple way to convert BS to AD dates, that can be used in different apps

  Scenario: converting a valid BS date                                    # bs-to-ad-conversion.feature:6
    When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01" # bsdateServer_test.go:14 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "200"                           # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "2003-07-17"                       # bsdateServer_test.go:34 -> theResponseContentShouldBe

  Scenario: converting an invalid BS date                               # bs-to-ad-conversion.feature:11
    When a "GET" request is sent to the endpoint "/ad-from-bs/60-13-01" # bsdateServer_test.go:14 -> aRequestIsSentToTheEndpoint
    Then the HTTP-response code should be "400"                         # bsdateServer_test.go:27 -> theHTTPresponseCodeShouldBe
    And the response content should be "not a valid date"               # bsdateServer_test.go:34 -> theResponseContentShouldBe

2 scenarios (2 passed)
6 steps (6 passed)
1.343415ms

Examples

The scenarios we have written down are pretty limited, probably there are more requirements of the software. Specially there will be those that have not been spoken about. To reduce the size of the feature-file Gherkin has the Examples: keyword.

index 5a00814..18db1ed 100644
--- a/features/bs-to-ad-convertion.feature
+++ b/features/bs-to-ad-convertion.feature
@@ -3,12 +3,25 @@ Feature: convert dates from BS to AD using an API
   I want to be able to send BS dates to an API endpoint and receive the corresponding AD dates
   So that I have a simple way to convert BS to AD dates, that can be used in other apps

-  Scenario: converting a valid BS date
-  When a "GET" request is sent to the endpoint "/ad-from-bs/2060-04-01"
+  Scenario Outline: converting a valid BS date
+    When a "GET" request is sent to the endpoint "/ad-from-bs/<bs-date>"
     Then the HTTP-response code should be "200"
-    And the response content should be "2003-07-17"
+    And the response content should be "<ad-date>"
+    Examples:
+      | bs-date    | ad-date    |
+      | 2060-04-01 | 2003-07-17 |
+      | 2040-01-01 | 1983-04-14 |
+      | 2040-12-30 | 1984-04-12 |

Instead of Scenario we have to use Scenario Outline and at the bottom of the Outline we add a table. The headings of the table are used as "variables" and the table rows are substituted into the steps e.g. <bs-date> becomes 2060-04-01.
Godog will run a single scenario for every line in the examples table. That way you can very easily multiply out the test cases.

To learn more about Scenario Outlines and Example-tables read this blog post: Scenario Outline In Gherkin Feature File by

jasson99Follow

Computer Engineer

Conclusion

Writing down the expected behaviors using the Gherkin language can improve the communication between the different stakeholders and with that increase customer satisfaction, productivity and the chances to make the project a success.
The feature descriptions become the requirement documentation.
Additionally, the same feature descriptions can be used to run automatic tests.

If you need help with setting up BDD or you want to outsource your test-development, please contact us:

performance testing with locust - 04 - interpret the results

Artur Neumann — Tue, 14 Jan 2020 11:28:18 +0000

In the last posts of this series we setup locust and made some basic performance tests to test the ownCloud WebDAV-API. This time we will try to make some sense of the locust output.

Here is the locust file we are using:



 from locust import HttpLocust, TaskSet, task, constant
import uuid

userNo = 0

class UserBehaviour(TaskSet):
    adminUserName = 'admin'
    davEndpoint = "/remote.php/dav/files/"
    fileName = ''
    userName = ''

    def on_start(self):
        #create user
        global userNo
        self.userName = "locust" + str(userNo)
        userNo = userNo + 1
        self.client.post(
            "/ocs/v2.php/cloud/users",
            {"userid": self.userName, "password": self.userName},
            auth=(self.adminUserName, self.adminUserName)
        )

    def on_stop(self):
        from locust.clients import HttpSession
        self.admin_client = HttpSession(base_url=self.client.base_url)
        self.admin_client.delete(
            "/ocs/v2.php/cloud/users/" + self.userName,
            auth=(self.adminUserName, self.adminUserName)
        )

    @task(3)
    def downloadFile(self):
        self.client.get(
            self.davEndpoint  + self.userName + "/ownCloud%20Manual.pdf",
            auth=(self.userName, self.userName)
        )

    @task(1)
    def uploadFile(self):
        if self.fileName == '':
            self.fileName = "/locust-perfomance-test-file" + str(uuid.uuid4()) + ".txt"
        self.client.put(
            self.davEndpoint + self.userName + self.fileName,
            "my data",
            auth=(self.userName, self.userName)
        )

class User(HttpLocust):
    task_set = UserBehaviour
    wait_time = constant(1)

To start ownCloud we have used docker: docker run -p 8080:8080 --name owncloud owncloud/server

and then started locust with: locust --host=http://localhost:8080

test the right thing

When I now run both ownCloud and locust on my workstation (i5-7500 CPU @ 3.40GHz; 8GB RAM) and hatch 100 locust-users I get this graph:

But now have a look at the CPU usage (on Linux the easiest way to see it is to use the top command)

WOW, 61.7% CPU is used by locust itself. I'm not really testing the performance of ownCloud (or not alone). Beside locust gnome, X and Firefox are eating up a significant amount of resources, so the results will never be accurate. Better get some dedicated hardware to run ownCloud on.

I have here an old Lenovo X201 Laptop (i5 M 540 CPU @ 2.53GHz; 4GB RAM). Not really fast, but should be OK for this example. I will run ownCloud on that Laptop and locust on my workstation. That way hatching 100 users still eats up the same amount of resources on the workstation, but because its fast enough that should not be the limiting factor. We really don't want the test-runner computer to limit our performance tests. If you don't have a computer that is fast enough to fully load your SUT (System Under Test), you can run locus distributed and that way utilize multiple computers to fully load your SUT.

interpret the results

Lets start the tests and increase the amount of users.
I started the test with 20 users and 1 user/s hatch rate, then increased the users to 50 with 2 users/s hatch rate and finally to 100 users with 4 users/s hatch rate.

In the response-time graph the green line shows the median response time and the yellow one the 95th percentile (95% of the requests finish before that time).

To calculate the current response time a sliding window of (approximately) the last 10 seconds is used see: get_current_response_time_percentile function.

As you can see, the median response time goes up as we add more users. And there is a "bump" in the 95th percentile line every time new users are created. So it looks like user creation is "expensive". (The "bump" is also visible in the median-line, but not that obvious).

Rerunning the tests shows a similar result.

BTW: Because there is always other stuff happening on the server its always good to run performance tests multiple times and see if you get similar results.

So from that information, how many users can our system handle? Maybe the better question is how long do you want your user to wait? In our test-scenario the user sends one request every second, either a download or an upload request. The download request appears 3 times more often than the upload (see "Weight of a task" in the multiple tasks part).
If our server tries to serve 20 such users, 95% of the time it will be able to respond within 400-450ms or less (not taking the user-creation into account, in normal life we would not create new users all the time). When trying to serve 50 concurrent users, 95% of the time it will be able to respond within 1600-1800ms or less. And half of the time (median response time) users will have to wait for around 1000ms or more for a response. For 100 users that obviously looks even worse, 95th percentile is around 6000ms and median response time around 3200ms.

Would it be acceptable for your application to let the user wait for 3sec or more for half of the requests? If not, you need to optimize the software or buy more hardware.

To see more details and maybe make more analysis download the CSV data and open in a spreadsheets app.
These files have one line per request type & URL, because we have the username in the URL, there will be a lot of lines.

In the request statistics CSV file we have the median/average/min/max response time for all uploads user0 has done, and all uploads user1 has done and so on. We can e.g. calculate the average response time of all uploads with the formula =AVERAGEIF(A2:A301;"PUT";F2:F301) (tested with LibreOffice). Column A holds the method name, column F is the average response time and in my table there are 301 lines.

Be aware that this list will now hold ALL the results, from the time when we had 20 users, 50 users and 100 users, so if we want to know the average response time of uploads with a particular amount of users, we would have to rerun the test with a fixed amount of users and not change it in between.

optimization

When we have started the ownCloud docker container, it created an database and for that it used SQlite database, that is good for quick testing and evaluation, but its soooo slow. Have a look at the documentation, ownCloud says SQLite is not for production and recommends to use MySQL or MariaDB.
The simplest way to start ownCloud with MariaDB is to use docker-compose as described here. In addition you also receive a Redis server, to do some caching.

Running that proposed setup on my system shows that it improves the response time a lot when running with 20 users, the 95th percentile goes down to 220-250ms (vs 400-450 before), there is also some improvement when running with 50 users, but when running with 100 users, it actually gets worse (median 5200-6000ms and 95th percentile is often over 7000ms).

More tests showed that with 15 concurrent users there is still 20-30% CPU time left most of the time, but with 20+ users the CPU is basically flat out.
Another interesting finding is, that in the area around 15 users the CPU is still not fully utilized, but the hard-drive works already pretty hard (see iotop). My guess is that when running with <= 15 users a faster hard-drive, e.g. a SSD would improve the performance, but with more than 20 users an SSD would be a waste of money, because even if the data would arrive faster at the CPU, it struggles to do its calculation.

cross-check

Let's see if we can prove our assumption that ~15 users should be the max for our system. I'm simulating 30 users, but with a hatch-rate of 0.025 users/sec (I want to give the system enough time to create the user and to refresh the sliding window for the chart after user-creation).

Looking at the graph I see that up to ~10 users the median time does not change much (160-180ms), looking at the output of top at the same time I see that there is still a lot of CPU time unused and even with 14-15 users, the median time goes down to 190ms. After that pretty flat area in the graph, it goes up pretty steep, the CPU is totally flat out.

Also have a look at the "Total Requests per Second" graph. Up to 15 users it steadily climbs up, but then there are valleys and hills, but the system struggles to serve more requests/s.

conclusion

The system scales well up to 15 users, meaning the single user would not experience any performance issues up to 15 concurrent users. Also the user would not experience any faster system if she is the only user on the system.
Up to 15 users the system can be optimized by using a better DB, caching, faster HDD and memory
Above 15 users, the CPU is the bottleneck and working on the suggestions in point 2, would not help.

Demonstrating TDD (Test-driven development) in Go

Artur Neumann — Wed, 01 Jan 2020 09:40:39 +0000

TDD is the practice to write tests before code and it should reduce failure rates and defects in your software.
In this blog-post I want to demonstrate how it can work.

starting point

I'm writing an application in Go that should convert Bikram Sambat (BS) (also called Vikram Samvat) dates to Gregorian dates and vice-versa. Vikram Samvat is a calendar used mostly in Nepal and India. But even if you don't use it, this demonstration might be useful for you to understand TDD.

So far I have done a bit of work that makes it possible to create a BS (Bikram Sambat) date instance, to get its details and to convert it to a Gregorian date. See: https://github.com/JankariTech/GoBikramSambat/blob/b99c510b22faf8395becda9a6dec1d0239504bb1/bsdate.go

These functions are also tested: https://github.com/JankariTech/GoBikramSambat/blob/b99c510b22faf8395becda9a6dec1d0239504bb1/bsdate_test.go

Now I want to add the possibility to convert a Gregorian date to a Bikram Sambat date. To do so, I want to be able to create a BS-date-instance by using a Gregorian date, then I could just get the BS-date details and the conversion is done.

Something like nepaliDate, err := NewFromGregorian(gregorianDay, gregorianMonth, gregorianYear) would be great and then simply use the existing nepaliDate.GetDay() nepaliDate.GetMonth() and nepaliDate.GetYear()

1. create the test

According to TDD I first have to create a test.
So in the file bsdate_test.go I create a new function called TestCreateFromGregorian().
As I already have a table of test-dates that are used for the conversion from Nepali to Gregorian I will use that data also to test the reverse conversion.

Here is the test data and the test function:

type TestDateConversionStruc struct {
    bsDate        string
    gregorianDate string
}

var convertedDates = []TestDateConversionStruc{
    {"2068-04-01", "2011-07-17"}, //a random date
    {"2068-01-01", "2011-04-14"}, //1st Baisakh
    {"2037-11-28", "1981-03-11"},
    {"2038-09-17", "1982-01-01"}, //1st Jan
    {"2040-09-17", "1984-01-01"}, //1st Jan in a leap year
...
}

func TestCreateFromGregorian(t *testing.T) {
    for _, testCase := range convertedDates {
        t.Run(testCase.bsDate, func(t *testing.T) {
            var splitedBSDate = strings.Split(testCase.bsDate, "-")
            var expectedBsDay, _ = strconv.Atoi(splitedBSDate[2])
            var expectedBsMonth, _ = strconv.Atoi(splitedBSDate[1])
            var expectedBsYear, _ = strconv.Atoi(splitedBSDate[0])

            var splitedGregorianDate = strings.Split(testCase.gregorianDate, "-")
            var gregorianDay, _ = strconv.Atoi(splitedGregorianDate[2])
            var gregorianMonth, _ = strconv.Atoi(splitedGregorianDate[1])
            var gregorianYear, _ = strconv.Atoi(splitedGregorianDate[0])

            nepaliDate, err := NewFromGregorian(gregorianDay, gregorianMonth, gregorianYear)
            assert.Equal(t, err, nil)
            assert.Equal(t, nepaliDate.GetDay(), expectedBsDay)
            assert.Equal(t, nepaliDate.GetMonth(), expectedBsMonth)
            assert.Equal(t, nepaliDate.GetYear(), expectedBsYear)
        })
    }
}

The function takes entries from the convertedDates list, splits them, tries to create a BS date out of the particular gregorian test-case and then asserts that the BS date (day, month, year) is as expected.

2. run the tests

The test is done, according to TDD I have to run it.

go test -v

results in:

# NepaliCalendar/bsdate [NepaliCalendar/bsdate.test]
./bsdate_test.go:171:23: undefined: NewFromGregorian
FAIL    NepaliCalendar/bsdate [build failed]

That was expected, the function does not exist, no wonder my tests fail. What to do next? Guess what: implement the function.
That makes TDD so easy, you just do what the tests tell you to fix.

3. fix it

That's easy, add to bsdate.go a new function:

func NewFromGregorian(gregorianDay, gregorianMonth, gregorianYear int) (Date, error) {

}

4. repeat

running the tests again I get:

./bsdate.go:195:1: missing return at end of function

That is true, let's return something, but what? Hey let's just create a BS date with the Gregorian numbers

 func NewFromGregorian(gregorianDay, gregorianMonth, gregorianYear int) (Date, error) {
-
+       return New(gregorianDay, gregorianMonth, gregorianYear)
 }

You are saying that will not work? I don't care, I do TDD, the test tells me to return something, and I do return, I even return the correct type of value.

lets run the tests again:

=== RUN   TestCreateFromGregorian/2068-04-01
    assert.go:24: got '17' want '1'

    assert.go:24: got '7' want '4'

    assert.go:24: got '2011' want '2068'

=== RUN   TestCreateFromGregorian/2068-01-01
    assert.go:24: got '14' want '1'

    assert.go:24: got '4' want '1'

    assert.go:24: got '2011' want '2068'

....

a lot of failures, you have guessed it, the conversion does not work. So lets implement some bits.

We know that BS is 56 point something years ahead of Gregorian. So adding 56 to the gregorian year should help:

 func NewFromGregorian(gregorianDay, gregorianMonth, gregorianYear int) (Date, error) {
-       return New(gregorianDay, gregorianMonth, gregorianYear)
+       var bsYear = gregorianYear + 56
+       return New(gregorianDay, gregorianMonth, bsYear)
 }

test results look better, instead of

....
=== RUN   TestCreateFromGregorian/2037-11-28
    assert.go:24: got '11' want '28'

    assert.go:24: got '3' want '11'

    assert.go:24: got '1981' want '2037'

=== RUN   TestCreateFromGregorian/2038-09-17
    assert.go:24: got '1' want '17'

    assert.go:24: got '1' want '9'

    assert.go:24: got '1982' want '2038'
....

I get:

....
=== RUN   TestCreateFromGregorian/2037-11-28
    assert.go:24: got '11' want '28'

    assert.go:24: got '3' want '11'

=== RUN   TestCreateFromGregorian/2038-09-17
    assert.go:24: got '1' want '17'

    assert.go:24: got '1' want '9'
....

So some years are calculated correctly, at least. Lets fix more tests by calculating the years more accurate and also calculate the BS month.

Because of the way the BS-calendar works, there is no algorithm to convert the date directly from the Gregorian calendar, we need a table. We know that Jan 1st falls always in the 9th BS month (Paush). So we have a table of BS years where the first value is the day in Paush that is the 1st Jan in that year, then a list of days of every BS month.
We can easily get the day-of-year from the gregorian date. Starting from Paush, we count the days of each BS month, whenever we get over the gregorian day-of-year, we found the correct BS month.

2074: [13]int{17, 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30},
2075: [13]int{17, 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30},
2076: [13]int{16, 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30},
2077: [13]int{17, 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31},
2078: [13]int{17, 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30},

These details have nothing to do with TDD, but help you to understand the coming algorithm.

lets put it into code:

 func NewFromGregorian(gregorianDay, gregorianMonth, gregorianYear int) (Date, error) {
        var bsYear = gregorianYear + 56
-       return New(gregorianDay, gregorianMonth, bsYear)
+       var bsMonth = 9                         //Jan 1 always fall in BS month Paush which is the 9th month
+       var daysSinceJanFirstToEndOfBsMonth int //days calculated from 1st Jan till the end of the actual BS month,
+                                               // we use this value to check if the gregorian Date is in the actual BS month
+
+       year := time.Date(gregorianYear, time.Month(gregorianMonth), gregorianDay, 0, 0, 0, 0, time.UTC)
+       var gregorianDayOfYear = year.YearDay()
+
+       //get the BS day in Paush (month 9) of 1st January
+       var dayOfFirstJanInPaush = calendardata[bsYear][0]
+
+       //check how many days are left of Paush
+       daysSinceJanFirstToEndOfBsMonth = calendardata[bsYear][bsMonth] - dayOfFirstJanInPaush + 1
+
+       //If the gregorian day-of-year is smaller or equal to the sum of days between the 1st January and
+       //the end of the actual BS month we found the correct nepali month.
+       //Example:
+       //The 4th February 2011 is the gregorianDayOfYear 35 (31 days of January + 4)
+       //1st January 2011 is in the BS year 2067 and its the 17th day of Paush (9th month)
+       //In 2067 Paush had 30days, This means (30-17+1=14) there are 14days between 1st January and end of Paush
+       //(including 17th January)
+       //The gregorianDayOfYear (35) is bigger than 14, so we check the next month
+       //The next BS month (Mangh) has 29 days
+       //29+14=43, this is bigger than gregorianDayOfYear(35) so, we found the correct nepali month
+       for ; gregorianDayOfYear > daysSinceJanFirstToEndOfBsMonth; {
+               bsMonth++
+               if bsMonth > 12 {
+                       bsMonth = 1
+                       bsYear++
+               }
+               daysSinceJanFirstToEndOfBsMonth += calendardata[bsYear][bsMonth]
+       }
+
+       return New(gregorianDay, bsMonth, bsYear)
 }

and now? You guessed it! Run the tests:

=== RUN   TestCreateFromGregorian
=== RUN   TestCreateFromGregorian/2068-04-01
    assert.go:24: got '17' want '1'

=== RUN   TestCreateFromGregorian/2068-01-01
    assert.go:24: got '14' want '1'

=== RUN   TestCreateFromGregorian/2037-11-28
    assert.go:24: got '11' want '28'

....

Actually, while implementing the algorithm I've run the tests multiple times and found mistakes in mixed-up variable names and other rubbish. That's cool, the tests helped me to find the issues right away.

But the tests still fail, I better get the day calculation correct.
We know the correct BS month, and we know the days since 1st Jan till the end of this month. Subtracting the day-of-the-year of the gregorian calendar from the days since 1st Jan till the end of the correct BS month will give us the amount of days between the searched day and the end of the BS month. Subtracting that number from the amount of days in the BS month should bring us to the correct date.

So many words to describe it, so little effort to write it in code:

-       return New(gregorianDay, bsMonth, bsYear)
+       var bsDay = calendardata[bsYear][bsMonth] - (daysSinceJanFirstToEndOfBsMonth - gregorianDayOfYear)
+
+       return New(bsDay, bsMonth, bsYear)

I hear you shouting: "Run the tests, run the tests!" Don't worry, I will:

=== RUN   TestCreateFromGregorian
=== RUN   TestCreateFromGregorian/2068-04-01
=== RUN   TestCreateFromGregorian/2068-01-01
=== RUN   TestCreateFromGregorian/2037-11-28
=== RUN   TestCreateFromGregorian/2038-09-17
=== RUN   TestCreateFromGregorian/2040-09-17
=== RUN   TestCreateFromGregorian/2040-09-18
=== RUN   TestCreateFromGregorian/2041-09-17
=== RUN   TestCreateFromGregorian/2041-09-18
=== RUN   TestCreateFromGregorian/2068-09-01
=== RUN   TestCreateFromGregorian/2068-08-29
=== RUN   TestCreateFromGregorian/2068-09-20
=== RUN   TestCreateFromGregorian/2077-08-30
=== RUN   TestCreateFromGregorian/2077-09-16
=== RUN   TestCreateFromGregorian/2074-09-16
=== RUN   TestCreateFromGregorian/2077-09-17
=== RUN   TestCreateFromGregorian/2077-09-01
=== RUN   TestCreateFromGregorian/2076-11-17
=== RUN   TestCreateFromGregorian/2076-11-18
=== RUN   TestCreateFromGregorian/2075-11-16
=== RUN   TestCreateFromGregorian/2076-02-01
=== RUN   TestCreateFromGregorian/2076-02-32
=== RUN   TestCreateFromGregorian/2076-03-01
--- PASS: TestCreateFromGregorian (0.00s)
    --- PASS: TestCreateFromGregorian/2068-04-01 (0.00s)
    --- PASS: TestCreateFromGregorian/2068-01-01 (0.00s)
    --- PASS: TestCreateFromGregorian/2037-11-28 (0.00s)
    --- PASS: TestCreateFromGregorian/2038-09-17 (0.00s)
    --- PASS: TestCreateFromGregorian/2040-09-17 (0.00s)
    --- PASS: TestCreateFromGregorian/2040-09-18 (0.00s)
    --- PASS: TestCreateFromGregorian/2041-09-17 (0.00s)
    --- PASS: TestCreateFromGregorian/2041-09-18 (0.00s)
    --- PASS: TestCreateFromGregorian/2068-09-01 (0.00s)
    --- PASS: TestCreateFromGregorian/2068-08-29 (0.00s)
    --- PASS: TestCreateFromGregorian/2068-09-20 (0.00s)
    --- PASS: TestCreateFromGregorian/2077-08-30 (0.00s)
    --- PASS: TestCreateFromGregorian/2077-09-16 (0.00s)
    --- PASS: TestCreateFromGregorian/2074-09-16 (0.00s)
    --- PASS: TestCreateFromGregorian/2077-09-17 (0.00s)
    --- PASS: TestCreateFromGregorian/2077-09-01 (0.00s)
    --- PASS: TestCreateFromGregorian/2076-11-17 (0.00s)
    --- PASS: TestCreateFromGregorian/2076-11-18 (0.00s)
    --- PASS: TestCreateFromGregorian/2075-11-16 (0.00s)
    --- PASS: TestCreateFromGregorian/2076-02-01 (0.00s)
    --- PASS: TestCreateFromGregorian/2076-02-32 (0.00s)
    --- PASS: TestCreateFromGregorian/2076-03-01 (0.00s)
PASS
ok      NepaliCalendar/bsdate   0.002s

The tests pass, job done! Commit it, push it and get yourself some Chiya!

You can find all the changes of this post here: https://github.com/JankariTech/GoBikramSambat/pull/4/

conclusion

TDD is easy: think about what you want to achieve, write tests for it and wildly hack code till your tests pass.

An other big advantage is: I can refactor my code all I like and still be confident it works fine. Maybe I want to optimize the speed of the algorithm, maybe I don't like it altogether and come up with a better one, or I simply want to change variable names. I can do that all without fear of messing up the functionality, as long as my tests are passing I'm pretty sure the code reacts the same.

maybe the next step

An other useful principle in software development is BDD (Behavior-driven development), it emerged out of TDD and uses its general principles but focuses not on defining and testing a single unit (function) but on describing the behaviour of the system and by that improving the communication between different stakeholders of the project. I've written a post about BDD using the same project and taking it further: https://dev.to/jankaritech/demonstrating-bdd-behavior-driven-development-in-go-1eci

performance testing with locust - 03 - setup your system

Artur Neumann — Wed, 27 Nov 2019 08:43:24 +0000

In the last part of this series we created multiple tasks to be able to simulate more realistically uploading and downloading files.

Now we want to go the next step and use multiple ownCloud users.

I will use the term "ownCloud user" for users that are set-up in the ownCloud system, have a username and password and can be used to login into the system. When I'm using the term "locust user" I'm talking about simulated users that hammer the server with requests. So far we used only one ownCloud user "admin" and multiple locust users. All locust users used that one ownCloud user to access the ownCloud server.

In this part of the series we want to have one ownCloud user for every locust user, so every TaskSet will be connecting with an own ownCloud user to the ownCloud server.

setup and teardown

This situation is pretty common in any kind of automated testing, before starting the tests often you have to set your system up and bring it into a "testable" state, or simply into the state you want to test.
Basically all test frameworks will have some kind of setup and teardown methods or hooks. Same applies to locust, you even can have a separate setup and teardown method in your Locust class and your TaskSet class.

consider this simple locust script

from locust import HttpLocust, TaskSet, task, constant

class UserBehaviour(TaskSet):
    def setup(self):
        print ("setup of TaskSet")

    def teardown(self):
        print ("teardown of TaskSet")

    @task(2)
    def one_task(self):
        print ("running one task")

    @task(1)
    def an_other_task(self):
        print ("running another task")

class User(HttpLocust):
    def setup(self):
        print ("setup of Locust class")

    def teardown(self):
        print ("teardown of Locust class")

    task_set = UserBehaviour
    wait_time = constant(1)

We have two tasks and a setup and a teardown, one in the User class and one in the UserBehavior class

Now lets see what happens.

Starting locust from the CLI:

[2019-11-29 08:55:19,213] artur-OptiPlex-3050/INFO/locust.main: Starting web monitor at *:8089
[2019-11-29 08:55:19,213] artur-OptiPlex-3050/INFO/locust.main: Starting Locust 0.13.2

Starting the test from the webUI with 2 users

[2019-11-29 08:55:22,932] artur-OptiPlex-3050/INFO/locust.runners: Hatching and swarming 2 clients at the rate 1 clients/s...
[2019-11-29 08:55:22,932] artur-OptiPlex-3050/INFO/stdout: setup of Locust class
[2019-11-29 08:55:22,932] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:22,933] artur-OptiPlex-3050/INFO/stdout: setup of TaskSet
[2019-11-29 08:55:22,933] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:22,933] artur-OptiPlex-3050/INFO/stdout: running one task
[2019-11-29 08:55:22,933] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:23,933] artur-OptiPlex-3050/INFO/stdout: running another task
[2019-11-29 08:55:23,933] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:23,934] artur-OptiPlex-3050/INFO/stdout: running one task
[2019-11-29 08:55:23,934] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:24,934] artur-OptiPlex-3050/INFO/locust.runners: All locusts hatched: User: 2
[2019-11-29 08:55:24,934] artur-OptiPlex-3050/INFO/stdout: running one task
[2019-11-29 08:55:24,934] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:24,934] artur-OptiPlex-3050/INFO/stdout: running another task
[2019-11-29 08:55:24,935] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:25,935] artur-OptiPlex-3050/INFO/stdout: running one task
[2019-11-29 08:55:25,935] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:25,935] artur-OptiPlex-3050/INFO/stdout: running one task
[2019-11-29 08:55:25,935] artur-OptiPlex-3050/INFO/stdout: 

...

[2019-11-29 08:55:31,939] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:32,939] artur-OptiPlex-3050/INFO/stdout: running another task
[2019-11-29 08:55:32,939] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:32,939] artur-OptiPlex-3050/INFO/stdout: running one task
[2019-11-29 08:55:32,939] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:33,940] artur-OptiPlex-3050/INFO/stdout: running one task
[2019-11-29 08:55:33,940] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:33,940] artur-OptiPlex-3050/INFO/stdout: running another task
[2019-11-29 08:55:33,940] artur-OptiPlex-3050/INFO/stdout:

Stopping the tests on the webUI

pressing Ctrl+C on the CLI

^C[2019-11-29 08:55:57,191] artur-OptiPlex-3050/ERROR/stderr: KeyboardInterrupt
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/ERROR/stderr: 2019-11-29T03:10:57Z
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/ERROR/stderr: 
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/INFO/locust.main: Shutting down (exit code 0), bye.
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/INFO/locust.main: Cleaning up runner...
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/INFO/locust.main: Running teardowns...
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/INFO/stdout: teardown of TaskSet
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/INFO/stdout: 
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/INFO/stdout: teardown of Locust class
[2019-11-29 08:55:57,191] artur-OptiPlex-3050/INFO/stdout:

Did you expect that? teardown only runs after locust is completely stopped, not when the test is stopped.
Makes sense, but does not help us with our issue, we want to create users before running the actual test and delete them afterwards. We might start and stop the test, without stopping locust and we can increase the locust users during the test and in that case we want to create more ownCloud users on the fly.

Luckily we have also on_start and on_stop methods

create users with on_start

on_start is called when the locust user starts to run the TaskSet

Starting from the last script let's add an on_start method to create users

from locust import HttpLocust, TaskSet, task, constant
import uuid

userNo = 0

class UserBehaviour(TaskSet):
    adminUserName = 'admin'
    davEndpoint = "/remote.php/dav/files/"
    fileName = ''
    userName = ''

    def on_start(self):
        #create user
        global userNo
        self.userName = "locust" + str(userNo)
        userNo = userNo + 1
        self.client.post(
            "/ocs/v2.php/cloud/users",
            {"userid": self.userName, "password": self.userName},
            auth=(self.adminUserName, self.adminUserName)
        )

    @task(3)
    def downloadFile(self):
        self.client.get(
            self.davEndpoint  + self.userName + "/ownCloud%20Manual.pdf",
            auth=(self.userName, self.userName)
        )

    @task(1)
    def uploadFile(self):
        if self.fileName == '':
            self.fileName = "/locust-perfomance-test-file" + str(uuid.uuid4()) + ".txt"
        self.client.put(
            self.davEndpoint + self.userName + self.fileName,
            "my data",
            auth=(self.userName, self.userName)
        )

class User(HttpLocust):
    task_set = UserBehaviour
    wait_time = constant(1)

So what is new here?

The on_start method first constructs a ownCloud username out of "locust" & a number. The userNo variable has to be defined globally, so that it survives when locust initialize the next instance of the User class. Remember: the Locust class (HttpLocust inherits from Locust) represents one simulated user that accesses your application.

Next a POST request is send with the username as userid and password. That requests needs to be authenticated as the admin-user. (Check the ownCloud docu if you are interested to learn more about those requests.)

At last there is the davEndpoint, now it needs the specific username, so that information has been moved into the specific GET and PUT method.

If you run that script now with locust and start a test with, lets say 3 users, you should see something like that:

The first line tells us that 3 POST requests have been sent to /ocs/v2.php/cloud/users, that looks promising.
And in the PUT ans GET requests, the usernames "locust0" till "locust2" are mentioned, very good!

Now lets look into the users list of ownCloud. For that login with "admin" / "admin" to http://localhost:8080/ and in the top right corner click on "admin" and then on "Users".

Those three users were created and used. If you want to double check use them to login to ownCloud, you should see the uploaded file.

delete users with on_stop

The only thing left is to clean up after us. Obviously we can simply kill the docker container, delete it and start it fresh with no users, but wouldn't it be nice to delete the users after stopping the test?

Let's use on_stop to clean up! It is run when the TaskSet is stopped.

Just add a simple small method to the UserBehaviour class:

def on_stop(self):
    self.client.delete(
        "/ocs/v2.php/cloud/users/" + self.userName,
        auth=(self.adminUserName, self.adminUserName)
    )

Remember to delete the users from ownCloud before rerunning the script (or just do docker kill owncloud; docker rm owncloud and start it again)

Now when you start the test and stop it again, you will see DELETE requests in the list, one per hatched locust user.
But what's that? The DELETE requests fail with HTTPError('401 Client Error: Unauthorized for url: http://localhost:8080/ocs/v2.php/cloud/users/locust0',)

Digging deeper (e.g. with WireShark) shows that the requests not only had the correct Authorization header sent, but also some cookies.

DELETE /ocs/v2.php/cloud/users/locust0 HTTP/1.1
Host: localhost:8080
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Cookie: cookie_test=test; oc1j5vb7hdm0=q2mv4lb5f2b37ti3etn8s1e0f1; oc_sessionPassphrase=y2u2sfTFfk8xx4cyIQZbycNvit4q0ZcKr4nHiiA7vSrGN%2BOZI30Ruvf3B5NyZrAxwtDNGz1wI7F6Yb2gjGsn%2FCnZ8Xpw3U8qRur1NrNcpJv%2Bm9egvmUiflwp3j7Rd3IG
Content-Length: 0
Authorization: Basic YWRtaW46YWRtaW4=

locust got those cookies from the first GET request we have sent as the specific ownCloud user, and it has kept them for all future requests. Generally that is a good thing, but ownCloud now ignores the Authorization header and uses the cookies to authenticate. So we effectively authenticate as the specific ownCloud user e.g. locust0 and that user has no privilege to delete itself.

I could not find a way to clear the session, so we need a new one.
For that change the on_stop function to:

def on_stop(self):
    from locust.clients import HttpSession
    self.admin_client = HttpSession(base_url=self.client.base_url)
    self.admin_client.delete(
        "/ocs/v2.php/cloud/users/" + self.userName,
        auth=(self.adminUserName, self.adminUserName)
    )

Here we import the locust HttpSession class and use it to create a new session, with no cookies in our way.

And here we go, when starting and stopping the tests we have successful DELETE requests. One per hatched locust user.

what's next?

We have now some basic tests, now it's time to look closer into the metrics and try to understand the meaning of all the numbers locust throws at us.