DEV Community: Darya Shirokova

Document Q&A with RAG: Google Gen AI Capstone Project

Darya Shirokova — Tue, 15 Apr 2025 21:51:39 +0000

As part of the Google Generative AI Intensive Course, the final task was to experiment with Gen AI using the tools and techniques presented throughout the program.

For my project, I chose a Document Q&A system using Retrieval-Augmented Generation (RAG) and themed it around one of my hobbies: the Call of Cthulhu RPG. The notebook answers the question about the game rules as described in the Call of Cthulhu 7th Edition Quick-Start Rules guide available on Chaosium's website.

Some example queries the system handles:

Is there magic in the game?
What do I need to bring for the game
What is STR?

While large language models (LLMs) are very powerful, they have two key limitations:

They can’t “know” anything beyond their training data — so they don’t have access to private, recent, or domain-specific documents unless you explicitly give them that information.
They have limited context windows — meaning they can only handle a certain amount of text at once, which limits how much you can “feed” them per query.

Retrieval-Augmented Generation (RAG)

RAG is a technique that addresses both of these limitations. Instead of trying to cram all the necessary information into a single prompt or hoping the model already knows the answer, RAG combines the power of retrieval systems with language generation.

Here’s how it works, broken into three stages:

1. Indexing (Preprocessing)

Before anything can be retrieved, your documents need to be processed and stored in a way that enables fast, intelligent searching. This involves:

Splitting documents into smaller chunks (like paragraphs or sections),
Creating vector embeddings for each chunk—numerical representations that capture the meaning of the text,
Storing these embeddings in a vector database for quick lookup.

2. Retrieval (Query-Time Search)

When a user asks a question, the system:

Converts the question into an embedding,
Searches the vector database to find the most semantically similar chunks,
Returns these top matches as the most relevant context.

3. Generation (LLM Response)

The retrieved document chunks, the user’s original question, and any specific instructions are passed to an LLM, which then generates a natural language response grounded in the retrieved context.

This allows the system to provide answers even about topics the LLM has never seen during training.

Vector Databases

One of the most important parts of a RAG system is the vector database, such as ChromaDB, an open-source embedding store designed for fast, scalable retrieval.

Vector databases store embeddings, which allow you to search for semantic meaning, not just keyword matches. This makes them a great match for document Q&A applications.

Links

You can explore the full project notebook on Kaggle here.

Source of the material used for the article/notebook: Google Generative AI Intensive Course

This article was created with the help of AI - for better readability.

Learning C# After Java: 5 Differences Explored

Darya Shirokova — Tue, 13 Jun 2023 09:18:37 +0000

There are many similarities between Java and C# - they are both managed languages with somewhat similar syntax influenced by C, but there are many differences as well. In this blogpost we will explore some of the key distinctions that Java developers should be aware of when transitioning to C#.

1 - Syntax sugar

Let's explore three language features that C# offers to enhance code readability, which are not present in Java. This is by no means an exhaustive list.

Named and Optional Arguments
In C#, you can use the name of the parameter to specify arguments of the method in arbitrary order. For example, consider the following method:

static void PrintHello(string name, string surname) =>
            Console.WriteLine($"Hello, {name} {surname}!");

The following invocations are identical in terms of result:

PrintHello("John", "Doe");
PrintHello(name: "John", surname: "Doe");
PrintHello(surname: "Doe", name: "John");

Additionally, you can specify a default value for the parameter, in which case it becomes optional:

static void PrintHello(string value="World") =>
            Console.WriteLine($"Hello, {value}!");

static void Main(string[] args)
{
    PrintHello();
    PrintHello("Peggy");
}

Get/Set Properties

A property in C# is a member that offers a versatile approach to access, modify, or compute the value of a private field. It is an analogue of regular getters/setters in Java with a special syntax and an option to auto-implement them.

Consider the following example with auto-implemented properties:

class Student
{
    public string Name {get; init;}

    public string? Department {get; set;}
}

Here we declare auto-implemented properties. The Name property exposes get property, as well as setter that can only be used during initialization of the method (init). The Department property exposes both a getter and a setter. We can then access and modify the properties accordingly:

var student = new Student { Name = "Alice", Department = "Computer Science"};
student.Department = "Chemistry";
Console.WriteLine(student.Department);
Console.WriteLine(student.Name);

Null-Conditional (Elvis) and the Null-coalescing operator

Unlike Java, C# supports Elvis (?.) and the Null-coalescing operators (??). These operators aim to simplify work with nullable values. For example, this block of code:

string department = null;
if (student != null) {
    department = student.Department;
}

can be replaced with:

string department = student?.Department;

The null-coalescing operator allows to replace val != null ? val : defaultValue expression with val ?? defaultValue. It can also be used in combinations with the Elvis operator, for example:

Console.WriteLine(student?.Department ?? "Value not known");

2 - Exceptions

In Java, there are so called checked and unchecked exceptions. Checked exceptions are checked at compile time and if you use a method that throws a checked exception, you must process it in your code, otherwise it won’t compile. Unchecked exceptions on the other hand don’t have to be processed in the code. If they’re not caught, they will crash the thread they’re thrown on.

C# doesn’t have the concept of checked exceptions and this is always up to developers to decide how and when to process an exception.

3 - IEnumerable & LINQ

IEnumerable interface in C# is an analogue of Iterable interface in Java. To support lazy computations over Iterable objects, Java 8 introduces a separate Stream API. In C#, similar functionality is provided directly via the Enumerable class.

Here is an example of using Where method of Enumberable to filter even numbers in C#:

IEnumerable<int> evenNums = Enumerable.Range(1, 1000)
                .Take(5)
                .Where(num => num % 2 == 0);

foreach (var num in evenNums)
{
    Console.WriteLine(num);
}

In Java, the same code would look like this:

List<Integer> evenNums = Stream.iterate(1, i -> i + 1)
        .limit(5)
        .filter(num -> num % 2 == 0)
        .toList();

evenNums.forEach(System.out::println);

In addition to the invocation of the query methods as shown in the example above, C# provides Language-Integrated Query (LINQ) capabilities. This is a technology that allows to use query language on IEnumberable objects directly in C#.

The example to filter even number will look like this in a LINQ-style syntax:

var nums = new List<int> { 1, 2, 3, 4, 5 };

IEnumerable<int> evenNums = from num in nums
                                where num % 2 == 0
                                select num;

foreach (var num in evenNums)
{
    Console.WriteLine(num);
}

4 - Async / await

C# provides async/await keywords to write asynchronous code in a more readable and sequential manner. Consider the following example, where we declare an asynchronous method that generates a random integer after 2 seconds of delay. We first call the method twice to generate two random integers and then await for the result when we need to print the sum:

static async Task<int> RandomIntAsync()
{
    await Task.Delay(TimeSpan.FromSeconds(2));
    return new Random().Next(0, 100);
}

static async Task Main(string[] args)
{
    Task<int> val1 = RandomIntAsync();
    Task<int> val2 = RandomIntAsync();
    Console.WriteLine(await val1 + await val2);
}

You can check this blogpost where I explore the differences in how the asynchronous code is executed in C# and Java in more details.

5 - Nullable

C# treats all value type variables as non-nullable by default. To allow a null value, the type must be explicitly marked as nullable, one should simply add a question mark to the type definition (e.g. "int?"). This allows developers to process situations where data may be missing or undefined, for example:

class Person
{
    public string Name { get; init; }

    public int? Age { get; init; }
}

static async Task Main(string[] args)
{
    var person = new Person { Name = "Bob"};
    if (person.Age == null)
    {
        Console.WriteLine("Age not specified.");
    }
}

In addition to nullable value types, you can specify nullable reference type, just like we did above in a Student class example where we declared Department string as nullable. Historically in C#, all references are nullable by default. However starting with C# 8, references are assumed to be non-nullable by default unless explicitly stated, similar to how value types behave. Due to complications with backward compatibility, this feature is not enabled by default.

Conclusion

Here we had a brief overview of some (but not all) of the major differences between C# and Java. Hope this provides a starting point when switching from Java to C#!

Understanding Asynchronous Code: A Comparison of Asynchronous Programming in C# and Java

Darya Shirokova — Thu, 01 Jun 2023 14:53:00 +0000

In this blogpost we’ll discuss what the asynchronous code is and explore how its execution can vary in different languages. We will use C# and Java to illustrate different approaches.

Introduction

Let’s imaging we are doing a flat refurbishment. We want to paint the walls and install the new floor, but before we can do that, we need to place an order for the paint and the floor, wait for its delivery and remove all the furniture so that we don’t damage it in the process.

If we are to write a program that executes that for us, we have a choice between asynchronous and synchronous execution.

If we are doing this synchronously, the order will be the following:

Day 0: Place and order for the paint and wait a few days until it is delivered (2 days).
Day 2: Place an order for the wooden floor and wait until it is delivered (20 days - slow delivery).
Day 22: Remove all furniture from your flat (5 days).
Day 27: Paint the walls (5 days).
Day 32: Install the wooden floor (5 days).
Day 37: And you refurbishment is finished :) It took 37 days in total.

This doesn’t sound effective. Surely, we could place an order for the wooden floor without waiting for the delivery of the paint. Or we could start removing the furniture before deliveries have arrived. Or in other words, we could do many operations asynchronously.

The sequence might look like this:

Day 0: Place an order for the paint (the delivery will take 2 days).
Day 0: Place an order for the wooden floor (the delivery will take 20 days).
Day 0: Remove the furniture (5 days).
Day 5: The paint already arrived 3 days ago - let’s paint the walls (5 days).
Day 10: Still waiting for another 10 days until the floors are delivered - we don’t have anything to do right now (10 days).
Day 20: The floor has finally arrived - let’s install it (5 days).
Day 25: Aaand done, it took us 25 days in total.

More formally, in asynchronous execution model (also known as non-blocking), you can initiate the task but don’t necessarily wait for its completion until you actually need the result. In the synchronous model (also known as blocking) you perform all the actions sequentially one-by-one.

That’s a common introduction into synchronous and asynchronous execution models. Programming languages implement this model differently.

We will explore how this example can be implemented in C# and Java and compare their models.

Asynchronous execution models

Asynchronous execution doesn’t necessarily mean each asynchronous operations spawns or uses a separate thread - though multithreading and asynchronicity are closely related, these are separate concepts. For example, messaging queue is one of design patterns to asynchronously process tasks - it can use one or multiple threads (messages will be submitted from a separate thread).

In C#, according to Microsoft documentation, the async / await keywords do not result in the creation of extra threads as asynchronous methods do not execute on their own dedicated threads. Instead, the method runs within the current synchronization context and utilizes the thread's time only when it is actively processing. Synchronization Context sort of represents the execution context in which the code is running and there are some examples where it is single threaded.

If synchronization context is null (as it is in the console app example below), the asynchronous operations will be using a thread pool for execution. If the methods reaches await keyword and yields the execution, its continuation doesn’t necessarily use the same thread.

In Java, there are multiple ways to write asynchronous code. We will only discuss CompletableFuture in this blogpost (note: we won’t cover virtual threads which as of now are available as a preview feature). Java uses separate threads to perform asynchronous tasks and doesn’t switch the context on JVM level. Even if you create a single threaded executor (Executors.newSingleThreadExecutor()), it will still be a second thread (in addition to the main thread) and the main thread will be blocked if asked to await the results of the execution.

Another model is used in Node.js and it utilizes Event Loop for the asynchronous operations. You can read about it in this article.

Let’s now see some examples in C# and Java.

C# example

The code below performs renovation from our intro in the asynchronous paradigm, as explained in the introduction:

internal class Renovation
{
    static int Id => Thread.CurrentThread.ManagedThreadId;
    static async Task BuyPaint()
    {
        Console.WriteLine($"{Id} Placing an order for paint at a local shop - it'll be delivered quickly.");
        await Task.Delay(TimeSpan.FromSeconds(2));
        Console.WriteLine($"{Id} Paint delivered!");
    }

    static async Task BuyWoodenFloor()
    {
        Console.WriteLine($"{Id} Placing an order for wooden floor. The delivery will take over a month.");
        await Task.Delay(TimeSpan.FromSeconds(20));
        Console.WriteLine($"{Id} Wooden floor is finally delivered!");
    }

    static async Task RemoveFurniture()
    {
        Console.WriteLine($"{Id} Starting to remove the furniture... This will take a few days.");
        await Task.Delay(TimeSpan.FromSeconds(5));
        Console.WriteLine($"{Id} Removed all furniture from the room!");
    }

    static async Task PaintingTheWalls()
    {
        Console.WriteLine($"{Id} Now let's paint the walls - should be fairly quick.");
        await Task.Delay(TimeSpan.FromSeconds(5));
        Console.WriteLine($"{Id} Walls are painted!");
    }

    static async Task InstallNewFloor()
    {
        Console.WriteLine($"{Id} Let's install the floor.");
        await Task.Delay(TimeSpan.FromSeconds(5));
        Console.WriteLine($"{Id} Floors are installed!");
    }

    static async Task Main(string[] args)
    {
        Console.WriteLine($"Is Background thread? {Thread.CurrentThread.IsBackground}");
        Console.WriteLine($"Is SynchronizationContext null? {System.Threading.SynchronizationContext.Current == null}");

        Task paint = BuyPaint();
        Task woodenFloor = BuyWoodenFloor();
        Console.WriteLine($"{Thread.CurrentThread.ManagedThreadId} Placed all orders! Let's remove all the furniture from the room...");
        await RemoveFurniture();

        await paint;
        await PaintingTheWalls();

        await woodenFloor;
        await InstallNewFloor();

        Console.WriteLine($"{Thread.CurrentThread.ManagedThreadId} The renovation is done :)");
        Console.WriteLine($"Is Background thread? {Thread.CurrentThread.IsBackground}");
    }
}

And here is the output:

PS C:\Users\darya\VSCode> dotnet run
**Is Background thread?** False
Is SynchronizationContext null? True
1 Placing an order for paint at a local shop - it'll be delivered quickly.
1 Placing an order for wooden floor. The delivery will take over a month.
1 Placed all orders! Let's remove all the furniture from the room...
1 Starting to remove the furniture... This will take a few days.
5 Paint delivered!
5 Removed all furniture from the room!
5 Now let's paint the walls - should be fairly quick.
5 Walls are painted!
5 Wooden floor is finally delivered!
5 Let's install the floor.
5 Floors are installed!
5 The renovation is done :)
**Is Background thread?** True

There are a few interesting things to note:

C# never yields the control of the execution until it encounters await keyword. E.g. it first logs that it placed an order for the painting and yields, then it logs the same for the wooden floor and yields and only then it starts removing the furniture.
As mentioned above, they synchronization context is null so the code uses the thread pool. At peak, three methods run in parallel - Main, BuyPaint and BuyWoodenFloor. BuyPaint started on the thread #1, but when it resumes, it uses thread #5 and the execution of the main method continues on this thread afterwards as well.
The Main method started on the main thread, but after a few async invocations it logs that is is executed on the background thread in the end.

Java example

Let’s now implement the same example in Java using CompletableFuture:

public class Main {
    static int getId() {
        return (int) Thread.currentThread().threadId();
    }

    static void buyPaint() {
        try {
            System.out.println(getId() + " Placing an order for paint at a local shop - it'll be delivered quickly.");
            TimeUnit.SECONDS.sleep(2);
            System.out.println(getId() + " Paint delivered!");
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    static void buyWoodenFloor() {
        try {
            System.out.println(getId() + " Placing an order for wooden floor. The delivery will take over a month.");
            TimeUnit.SECONDS.sleep(20);
            System.out.println(getId() + " Wooden floor is finally delivered!");
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    static void removeFurniture() {
        try {
            System.out.println(getId() + " Starting to remove the furniture... This will take a few days.");
            TimeUnit.SECONDS.sleep(5);
            System.out.println(getId() + " Removed all furniture from the room!");
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    static void paintingTheWalls() {
        try {
            System.out.println(getId() + " Now let's paint the walls - should be fairly quick.");
            TimeUnit.SECONDS.sleep(5);
            System.out.println(getId() + " Walls are painted!");
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    static void installNewFloor() {
        try {
            System.out.println(getId() + " Let's install the floor.");
            TimeUnit.SECONDS.sleep(5);
            System.out.println(getId() + " Floors are installed!");
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    public static void main(String[] args) {
        Executor executor = Executors.newFixedThreadPool(/* nThreads= */ 3);
        CompletableFuture<Void> paint = CompletableFuture.runAsync(Main::buyPaint, executor);
        CompletableFuture<Void> woodenFloor = CompletableFuture.runAsync(Main::buyWoodenFloor, executor);

        System.out.println(getId() + " Placed all orders! Let's remove all the furniture from the room...");

        removeFurniture();

        paint.join();
        CompletableFuture.runAsync(Main::paintingTheWalls, executor).join();

        woodenFloor.join();
        CompletableFuture.runAsync(Main::installNewFloor, executor).join();

        System.out.println(getId() + " The renovation is done :)");
    }
}

And the output:

23 Placing an order for paint at a local shop - it'll be delivered quickly.
24 Placing an order for wooden floor. The delivery will take over a month.
1 Placed all orders! Let's remove all the furniture from the room...
1 Starting to remove the furniture... This will take a few days.
23 Paint delivered!
1 Removed all furniture from the room!
25 Now let's paint the walls - should be fairly quick.
25 Walls are painted!
24 Wooden floor is finally delivered!
23 Let's install the floor.
23 Floors are installed!
1 The renovation is done :)

We see that Java’s CompletableFuture used separate threads from the thread pool we defined for each method.
Unlike in C#, the asynchronous methods always fully executes on a separate thread from the beginning to end and don’t yield the execution.
The main thread is always occupied even when blocked - the main method always starts and ends on the main thread.

Conclusion

Asynchronous code is a powerful model that can boost responsiveness of the application when used properly. We just had a look into how it might work differently in various languages. This is a huge topic and I don’t attempt to cover all the details in one blogpost, but I hope it provides a demonstration of how seemingly same code can be executed very differently depending on the programming language.

Darya Shirokova is a Software Engineer @ Google

Writing End-to-End test with pywinauto

Darya Shirokova — Fri, 26 May 2023 14:42:00 +0000

In the previous blogpost, we discussed pros and cons of the automated End-to-End (E2E) testing and covered some tips on how to structure the tests and make them less flaky.

Let’s now try to use some of these tips and go over a simple example using pywinauto (python modules for Windows GUI automation). We will write UI tests for the Windows Notepad.

For the Windows Notepad, some of the user journeys that could be covered in the tests are:

Entering text in the notepad window and saving the result to a file.
Replacing a substring in the text with another value.
Find all occurrences of the substring and going over them one by one.

Let’s use the first basic test case to get some glimpse into how pywinauto works and where we can apply tips we discussed in the previous blogpost.

Note: to use pywinauto, you can install it with pip install pywinauto.

def test_save_as():
    # Start notepad.exe and connect to it (when a new process is spawned).
    app = Application().start('notepad.exe')\
                .connect(title="Untitled - Notepad")

    # Type "Hello world!" in the notepad text editor.
    app.UntitledNotepad.Edit.type_keys("Hello world!", with_spaces=True)

    # Send ctrl+s signal to trigger "Save As".
    send_keys("^s")
    app.SaveAs.Edit.set_edit_text('tmp.txt')
    app.SaveAs.Save.click()

        # Assert that the content of the file is the same as in notepad.
    test_file = os.path.join(os.path.expanduser("~"), "Documents", "tmp.txt")
    assert open(test_file).read() == "Hello world!"

    # Kill the app.
    app.kill()

Let’s now take the first tip and introduce clean up in the beginning of the test. What would it solve here?

First of all, as of now, if there are multiple instance of Notepad running, the code will fail to connect to any of them. So our assumption is that there are no instances running before we launch the test - which we should either change or enforce this assumption. This assumption could be easily violated, in case the test didn’t reach app.kill() call in the end and failed before that.

We also need to ensure that the file tmp.txt doesn’t exist before starting the test. In fact, if the file already exists, the test will pass without rewriting it - Notepad will show a window to suggest replacing the file which we don’t process in the code.

To be on the safe side, let’s reset the state in the beginning of the test - the app may not have exited correctly or e.g. there might be another instance of Notepad open for the reasons outside of our control.

def remove_if_exists(path):
    if os.path.exists(path):
        os.remove(path)

def reset_state(file):
    os.system("taskkill /f /im notepad.exe")
    remove_if_exists(file)

def test_save_as():
    test_file = os.path.join(os.path.expanduser("~"), "Documents", "tmp.txt")
    reset_state(test_file)
        # ... the rest of the code

Let’s also consider where we might need to introduce some delays for actions to take effect. There are a few places where this is relevant, such as:

We should verify before connecting to the application that it is ready and running.
After sending Ctrl+S signal so save the file, we should make sure the expected dialog appeared. Note, this could be avoided if we used menu_select instead, but in this example we are sending a keyboard signal.
Before asserting the content of the file, we should make sure that it has been created.

We will use the “wait until with retries” approach instead of just “sleep” as it is more reliable and doesn’t unnecessarily slow down the execution of the test.

Luckily, pywinauto already supports timeout and retries for many of its methods. For the rest of the methods, we will introduce execute_with_retries method.

def execute_with_retries(func, retries=5, time_between_retries=1):
    for _ in range(retries):
        try:
            return func()
        except Exception as e:
            print(e)
            time.sleep(time_between_retries)

    raise Exception(f"Method failed after {retries} retries.")

def get_dialog(app, dlg_name, retries=5, time_between_retries=1):
    return execute_with_retries(lambda: app[dlg_name], retries, time_between_retries)

def remove_if_exists(path):
    if os.path.exists(path):
        os.remove(path)

def reset_state(file):
    # Kill all currently open instances of notepad.
    os.system("taskkill /f /im notepad.exe")
    remove_if_exists(file)

def test_save_as():
    # RESET stage.
    test_file = os.path.join(os.path.expanduser("~"), "Documents", "tmp.txt")
    reset_state(test_file)

        # ARRANGE stage.
    app = Application().start('notepad.exe')\
                .connect(title="Untitled - Notepad"
    # get a handle on the window inside Notepad.exe process
    dlg = app.UntitledNotepad
    # wait till the window is really open
    dlg.wait('visible', timeout=5)

    # ACT stage.
        # Type "Hello world!" in the notepad text editor.
    dlg.Edit.type_keys("Hello world!", with_spaces=True)

        # Send ctrl+s signal to trigger "Save As".
    send_keys("^s")

        # Save into tmp.txt file.
    save_as = get_dialog(app, 'SaveAs')
    save_as.Edit.set_edit_text('tmp.txt')
    save_as.Save.click()

    # ASSERT stage.
        # Assert the file is created and has the correct content.
    assert execute_with_retries(lambda: open(test_file).read() == "Hello world!")

    # Kill the app.
    app.kill()

And now we have a UI test for ‘Save As’ functionality of the Notepad.

Although this is a trivial example, it still shows that even in simple cases many things can go wrong with UI tests, that are executed in complex environments which we don’t always fully control. Although this example would likely work fine with less precautions, this gives us a glimpse on what techniques can be used to reduce a chance of flakes when writing robust UI tests.

Darya Shirokova is a Software Engineer @ Google

Introduction into End-to-End Testing of GUI Applications

Darya Shirokova — Tue, 23 May 2023 21:19:01 +0000

This is a first blogpost from a number of blogposts on end-to-end testing. It covers the concept of end-to-end testing, what benefits it may provide and what trade-offs you might need to accept if you use end-to-end tests in you project.

What is End-To-End testing?

End-to-end (E2E) tests aim to test the application from users perspective. In case of GUI Applications, whether it is Web, Desktop or Mobile, user interacts with UI elements of your product such as buttons, text fields, checkboxes, etc, and this is the behaviour that the E2E tests should try to reproduce.

An E2E test covers core user journeys, such as creating an account, adding items to the shopping basket or saving an item to the wish-list.

Unlike units tests which only verify sizeable components of the application such as a module or a class, E2E tests verify the application as a whole. Another key difference is that unit tests could replace dependencies of the component under test (especially external dependencies like a database) with test doubles (such as mocks or fakes), while E2E tests use real dependencies, though they could (and often should) be isolated from the production environment.

Considerations on when to add E2E tests

So when should we consider End-to-end tests for GUI applications? E2E tests can automate manual testing and provide confidence that new changes don’t break the existing user journeys. They are a handy tool for applications with complex UI component to verify the interaction between frontend and backend, or to verify that all components work well together even when evolving independently.

There are a few things to consider before adding them:

Writing and maintaining robust and meaningful E2E tests can be a difficult task that consumes a lot of time during development.
You will need to set up and maintain the infrastructure to run the tests.
E2E tests are very often flaky, i.e. they might fail without detecting any actual issue in your product, even when you follow best practises. You will need to invest time into both maintenance and health of the tests or you will end up with a test suite that nobody trusts even when they actually detect some bugs.

Let’s discuss some tips on how to approach these problems when writing E2E tests.

Test cases structure

First of all, define what tests you want to add and how granular they need to be. Most likely, there are already lower level tests (such as unit tests) which extensively cover business logic of various subcomponents of your system, so you may want to focus on high-level core user journeys in E2E tests. According to the Test Pyramid concept, as you move to higher levels, the number of tests should decrease and you should focus more on testing integration between different component.

This approach allows to test how all the different components work together and whether users are able to achieve their goals when using you application.

Another rule of thumb is to consider adding an additional test case once there is a bug in your application which was not caught by the existing tests, but you should ask yourself whether this should be a unit or an E2E test.

One of the good practices is to include the following stages in your test case:

Reset the state of the application (or clean up after execution). Since E2E tests are often large and difficult to setup, individual test cases often share the state (database, file system, etc). To make sure that one test case doesn’t influence another, the shared state should be reset between test cases (more on this later on).
Arrange the test. This includes setting up the stage for the test - you prepare the state of the application so that preconditions for the test are satisfied. For example, if you want to test order cancellation, you’ll first create an order in the prepare stage. For GUI E2E tests, you usually have options on how to execute this stage. You can either prepare the state of the test from user perspective by interacting with UI elements (in which case this stage will effectively merge with the Act stage), or you can make a backend call / upload a configuration file to set the expected state of the test. While the first option better represents the user journey, it makes the test slower (thus increasing the probability of flakes) and consumes more resources. So this is a trade-off you’d need to consider. Often, the prepare part is already covered by other test cases, in which case executing this stage via a config file or a backend call might be a better approach.
Act. This is the stage where you simulate user’s behaviour by interacting with UI elements to complete the journey.
Assert. This stage can be different for E2E tests depending on your use case. You need to verify the state of the system changed as expected, e. g. by making a backend call. Another technique that might be useful is diffing (screenshot or text), that compares the current screen or the produced output with expected.

Flakes

Flakes in UI tests are almost inevitable. Flake is when a test fails from time to time for reasons not related to actual issues in the application under test. UI tests are especially susceptible to these kind of issues as they test the system as a whole and depend on response from different components. Additionally, due to their nature, they are run in a multithreading environment managed by an operating system which may cause various issues. Some examples:

Test tries to execute the next action after interacting with UI element, however the previous action is still in progress (e.g. you expect a new window to appear but it didn’t).
Some components introduce non-breaking changes which still break UI tests. For example, when I worked on tests for one of the plugins my team developed, we encountered an issue where the test would fail because the application suggests to install a new version when you open it.
Inconsistences in rendering of text or elements could cause flakiness, especially in screenshot diff tests (e.g. a slightly different font or font aliasing, an element moved by a pixel, etc).
Many more possibilities :)

Let’s discuss some techniques to reduce flakiness:

As mentioned above, always make sure test cases don’t depend on one another. Each state should either start execution from the scratch or at least make sure to reset the state of each test in the beginning of each test. In some cases it might be preferable to do it in the beginning of the test instead of making a clean up in the end of execution - some end-to-end tests are executed in the non-isolated environment (e.g. OS), and the state might have be changed from outside of tests or the previous execution could have aborted before completed cleaned up. This clean up can be done, for example, in special methods provided by test frameworks, such as methods using @Before and @After annotation in JUnit.
A common approach when interacting with UI elements is to give the application some time to act and propagate the change - a wait method. If feasible, it is better to use an upgraded version of the method - wait_until. Instead of just waiting for a specific time, define what state of the application you expect after the interaction. With that approach you can also specify a much larger timeout without unnecessarily slowing down the test as it will proceed as soon as the desired state is achieved - or fail after a time limit is reached with a high level of confidence that an actual bug is detected. An example of using wait_until is to verify that a certain form appeared on UI after you pressed a button.
For UI interactions, you can use retry with wait_until to minimise the chance of the flake.
Consider including intermediate validations in your tests. If the test requires a lot of steps to reproduce a user’s journey, consider periodically asserting the intermediate results of the operation. This doesn’t necessarily reduces the flakes, but might be handy to quickly define the root cause of the failure when it happens.

Infrastructure

Let’s discuss the question like when and how often to run the E2E tests.

In the perfect world, your tests would run before a pull request or a code change is merged, similar to other types of tests such as unit tests.

However, this is not always feasible. First of all, running E2E might be costly. It can also be time-consuming, and the last thing you want is to slow down the iteration cycle for developers (combine slowness with flakiness and E2E tests can be a real annoyance for engineers).

In that case, you would need to set up a runner which runs the tests with frequency feasible for your application, e.g. once a day. For this approach you’ll also need to set up monitoring & notification system to ensure failures are reported to the team’s email alias and / or bug tracker.

Maintenance

One of perhaps the most costly parts of the E2E tests is their maintenance. If this part is skipped, the most likely outcome is that the test will be flaky so that even when they fail for valid reason, nobody cares to check. In other words, tests will be useless without proper maintenance (so why bother writing them in the first place).

One of the approaches is to set up the on duty rota shared between all team members. It is important to make it’s not a one-person-job otherwise your colleague will be unhappy and the rest of the team won’t develop necessary skills to write useful tests.

The responsibilities should include monitoring failures, verifying whether a failure is an actual bug or a flake and making tests “green”.

To make this effort useful in the long run, once a flake is detected a good practice for the on-duty person in to try to address this flake so that it doesn’t happen in the future (e.g. introduce wait_until).

Conclusion

There are lot’s of tradeoffs to consider when adding E2E tests for UI applications. While it might not be realistic to have them for each project, they might be a great tool for complex systems with multiple components where it is hard to manually verify all the core user journeys. They are a great addition not only to Web Application, but also to the Desktop apps which often have scheduled releases and patching bugs after the code reaches end users might be challenging.

In the next blogpost, let’s explore an example using the pywinauto library to write a simple test case for a Windows GUI Application.

Darya Shirokova is a Software Engineer @ Google