Writing End-to-End test with pywinauto

In the previous blogpost, we discussed pros and cons of the automated End-to-End (E2E) testing and covered some tips on how to structure the tests and make them less flaky.

Let’s now try to use some of these tips and go over a simple example using pywinauto (python modules for Windows GUI automation). We will write UI tests for the Windows Notepad.

For the Windows Notepad, some of the user journeys that could be covered in the tests are:

• Entering text in the notepad window and saving the result to a file.
• Replacing a substring in the text with another value.
• Find all occurrences of the substring and going over them one by one.

Let’s use the first basic test case to get some glimpse into how pywinauto works and where we can apply tips we discussed in the previous blogpost.

Note: to use pywinauto, you can install it with pip install pywinauto.

def test_save_as():
    # Start notepad.exe and connect to it (when a new process is spawned).
    app = Application().start('notepad.exe')\
                .connect(title="Untitled - Notepad")

    # Type "Hello world!" in the notepad text editor.
    app.UntitledNotepad.Edit.type_keys("Hello world!", with_spaces=True)

    # Send ctrl+s signal to trigger "Save As".
    send_keys("^s")
    app.SaveAs.Edit.set_edit_text('tmp.txt')
    app.SaveAs.Save.click()

        # Assert that the content of the file is the same as in notepad.
    test_file = os.path.join(os.path.expanduser("~"), "Documents", "tmp.txt")
    assert open(test_file).read() == "Hello world!"

    # Kill the app.
    app.kill()

Let’s now take the first tip and introduce clean up in the beginning of the test. What would it solve here?

First of all, as of now, if there are multiple instance of Notepad running, the code will fail to connect to any of them. So our assumption is that there are no instances running before we launch the test - which we should either change or enforce this assumption. This assumption could be easily violated, in case the test didn’t reach app.kill() call in the end and failed before that.

We also need to ensure that the file tmp.txt doesn’t exist before starting the test. In fact, if the file already exists, the test will pass without rewriting it - Notepad will show a window to suggest replacing the file which we don’t process in the code.

To be on the safe side, let’s reset the state in the beginning of the test - the app may not have exited correctly or e.g. there might be another instance of Notepad open for the reasons outside of our control.

def remove_if_exists(path):
    if os.path.exists(path):
        os.remove(path)

def reset_state(file):
    os.system("taskkill /f /im notepad.exe")
    remove_if_exists(file)

def test_save_as():
    test_file = os.path.join(os.path.expanduser("~"), "Documents", "tmp.txt")
    reset_state(test_file)
        # ... the rest of the code

Let’s also consider where we might need to introduce some delays for actions to take effect. There are a few places where this is relevant, such as:

• We should verify before connecting to the application that it is ready and running.
• After sending Ctrl+S signal so save the file, we should make sure the expected dialog appeared. Note, this could be avoided if we used menu_select instead, but in this example we are sending a keyboard signal.
• Before asserting the content of the file, we should make sure that it has been created.

We will use the “wait until with retries” approach instead of just “sleep” as it is more reliable and doesn’t unnecessarily slow down the execution of the test.

Luckily, pywinauto already supports timeout and retries for many of its methods. For the rest of the methods, we will introduce execute_with_retries method.

def execute_with_retries(func, retries=5, time_between_retries=1):
    for _ in range(retries):
        try:
            return func()
        except Exception as e:
            print(e)
            time.sleep(time_between_retries)

    raise Exception(f"Method failed after {retries} retries.")

def get_dialog(app, dlg_name, retries=5, time_between_retries=1):
    return execute_with_retries(lambda: app[dlg_name], retries, time_between_retries)

def remove_if_exists(path):
    if os.path.exists(path):
        os.remove(path)

def reset_state(file):
    # Kill all currently open instances of notepad.
    os.system("taskkill /f /im notepad.exe")
    remove_if_exists(file)

def test_save_as():
    # RESET stage.
    test_file = os.path.join(os.path.expanduser("~"), "Documents", "tmp.txt")
    reset_state(test_file)

        # ARRANGE stage.
    app = Application().start('notepad.exe')\
                .connect(title="Untitled - Notepad"
    # get a handle on the window inside Notepad.exe process
    dlg = app.UntitledNotepad
    # wait till the window is really open
    dlg.wait('visible', timeout=5)

    # ACT stage.
        # Type "Hello world!" in the notepad text editor.
    dlg.Edit.type_keys("Hello world!", with_spaces=True)

        # Send ctrl+s signal to trigger "Save As".
    send_keys("^s")

        # Save into tmp.txt file.
    save_as = get_dialog(app, 'SaveAs')
    save_as.Edit.set_edit_text('tmp.txt')
    save_as.Save.click()

    # ASSERT stage.
        # Assert the file is created and has the correct content.
    assert execute_with_retries(lambda: open(test_file).read() == "Hello world!")

    # Kill the app.
    app.kill()

And now we have a UI test for ‘Save As’ functionality of the Notepad.

Although this is a trivial example, it still shows that even in simple cases many things can go wrong with UI tests, that are executed in complex environments which we don’t always fully control. Although this example would likely work fine with less precautions, this gives us a glimpse on what techniques can be used to reduce a chance of flakes when writing robust UI tests.

---

Darya Shirokova is a Software Engineer @ Google