DEV Community: AskUI

Use Playwright and AskUI Together

Johannes Dienst — Wed, 25 Sep 2024 08:42:53 +0000

Use Playwright and AskUI Together

Playwright is the de facto standard when it comes to reliably testing browser applications across browsers and operating systems.

But what if your workflow under test leaves your browser and requires you to automate on a desktop application?

AskUI is not confined to a specific application because it uses visual selectors and real human user-like input with keypresses and natural mouse movement. This makes it an ideal tool to accompany Playwright.

In this post you will learn to setup Playwright together with AskUI to define workflows that can leave the browser.

Prerequisites

AskUI installed and configured on your system (Windows, Linux, macOS)

Install Playwright

Playwright for JavaScript/TypeScript consists of two components.

The playwright node package
A browser installed through Playwright library

First you have to install playwright as a dev-dependency. Execute the following command in your terminal:

npm install --save-dev playwright

For this blog you will use Chromium as a browser. The installation through playwright is done like this:

npx playwright install --with-deps chromium

Configure Playwright

You will configure Playwright and the Chromium browser in the file helpers/askui-helper.ts, so it will work correctly together with Jest.

import { UiControlClient, UiController } from 'askui';
import { AskUIAllureStepReporter } from '@askui/askui-reporters';

// Server for controlling the operating system
let uiController: UiController;

// Client is necessary to use the askui API
let aui: UiControlClient;

// 1. Playwright setup
import { chromium, devices, Browser, BrowserContext } from 'playwright';
let pwBrowser: Browser;
let pwContext: BrowserContext;

jest.setTimeout(60 * 1000 * 60);

beforeAll(async () => {

  // 2. Playwright setup
  pwBrowser = await chromium.launch({ headless: false, slowMo: 100 });
  pwContext = await pwBrowser.newContext(devices['Desktop Chrome']);

  uiController = new UiController({
    /**
      * Select the display you want to run your tests on, display 0 is your main display;
      * ignore if you have only one display
      */
    display: 0,
  });

  await uiController.start();

  aui = await UiControlClient.build({
    credentials: {
      workspaceId: '<your-workspace-id>',
      token: '<your-access-token>',
    },
    reporter: new AskUIAllureStepReporter(),
  });

  await aui.connect();
});

...

afterAll(async () => {

  // 3. Playwright setup
  await pwContext.close();
  await pwBrowser.close();

  aui.disconnect();

  await uiController.stop();
});

export { aui };

// 4. Playwright setup
export { pwContext };

Import the necessary classes from playwright and define the variables
Configure chromium to be not headless and also slow down its execution. Also we want a Desktop Chrome as we are on desktop.
Close the BrowserContext and the Browser after the execution finished.
Export the Context so you can use it in your workflow

Write a Workflow

With the BrowserContext set up, hop over to the file my-first-askui-test-seuite.test.ts and import it there. Add this to the start of the file:

import { pwContext } from './helpers/askui-helper';

Now you can implement a simple workflow:

Open a new tab
Navigate to https://google.com/
Click away the cookie banner
Fill the search field with Playwright framework
Press Enter on keyboard
Switch to image search by clicking on the Button architecture
Move the mouse cursor to the element that has an image with a red and a green mask in it (Playwright logo description)

describe('jest with askui', () => {
  it('should open askui.com in Chrome with playwright', async () => {
    let page = await pwContext.newPage();
    await page.goto('https://google.com/');
    await page.getByRole('button', { name: 'Reject all' }).click();
    await page.locator('//textarea[@name="q"]').fill('Playwright framework');
    await page.keyboard.press('Enter');
    await page.waitForTimeout(1000);

    // AskUI workflow
    // Make sure you have the focus on the browser
    await aui.mouseLeftClick().exec();
    await aui.click().button().withText('Images').exec();
    await aui.click().text('architecture').exec();
    await aui.moveMouseTo().element().matching('image with a red and a green mask in it').exec()

    await page.waitForTimeout(5000);
  });
});

Testrun

To run this you need to run this command in your terminal:

// Linux + macOS
npm run askui

// Windows
AskUI-RunProject

The following gif shows the execution of the whole workflow:

Conclusion

Combining AskUI with Playwright enhances automation capabilities to extend beyond web browsers. While Playwright is a solid choice for browser automation, AskUI takes the reins when dealing with desktop apps and operating systems.

This integration offers a practical toolkit for workflows across various applications for efficient automation.

Run AskUI Workflows with Pipedream for Smoke Testing

Johannes Dienst — Wed, 07 Aug 2024 14:18:25 +0000

Dylan Pierce shared an interesting approach to smoke testing at our meetup: Fill out a contact form with the help of Computer Vision & Large Language Models. Especially the vision part was impressive but it lacked one specific feature: Interaction with the form. How can you be sure your form works if you do not try to fill it out and send it?

This is where I thought an AskUI integration into the Pipedream workflow could be useful. AskUI uses visual selectors instead of code selectors and can interact like a human with a form like that. Giving us a real smoke test!

In this blog I will describe how I integrated AskUI into a Pipedream workflow, so we get the benefit of visual selection and user interaction.

Prerequisites

AskUI Controller installed and configured on a remote accessible system like Gitpod or a cloud machine. You can use our Try-Out-Repository or install on your own system (Windows, Linux, macOS).
Pipedream account and a workflow. Follow their Introduction

What Will We Build? Aka the Use Case

Dylan Pierce showed an awesome use case with Pipedream and Puppeteer, where he implemented a smoke test with the help of AI without writing any selectors by himself. I highly recommend to watch the recording: https://youtu.be/o6hej9Ip2vs

Our Use Case: Smoke Test with Visual Selectors

Dylans use case involved querying a Large Language/Multimodal Model to implement the smoke test. We will modify this a little bit to use the visual selectors from AskUI, which do not rely on the specific UI-technology but identify elements through their appearance with an AI vision model.

Here are the steps we will implement:

Trigger the smoke test once per day
Perform the smoke test with AskUI on a remote system
Send an email if the smoke test was successful or not

1. Add a Trigger

The first thing Pipedream wants us to add is a trigger. We add a Schedule trigger that will run our Workflow everyday at 9:00 AM.

2. Prepare a Custom Code Action

AskUI does not have an action in Pipedream. So we will make use of the AskUI node package and run a custom code action. For this we a a _Code action. In this action we will fill out the simple authentication from https://authenticationtest.com/simpleFormAuth/. If we see the success page we will send

We have to do the following steps:

Add a property uiControllerUrl, so we do not have to hardcode it into the code
Store our credentials in environment variables
Import UiControlClient from the askui node package
Configure the AskUI UiControlClient to use the credentials and the uiControllerUrl

The first thing we want to add to our custom code is the uiControllerUrl. Click Refresh fields so the Configure tab shows up at the start of the action. Here is the relevant code snippet.

...
export default defineComponent({

  props: {
    uiControllerUrl: { type: "string" }
  },

  async run({ steps, $ }) {
...

Then head over to environment variables and add your workspaceId and accessToken there. You obtained those by following the prerequisites instructions above. You can now setup the connection to the AskUI Controller over the UiControlClient like this. Notice how easy it is to use arbitrary node-packages in Pipedream? I only needed to import UiControlClient and it just worked 🥳.

Note: The following code also contains the teardown.

import { UiControlClient } from 'askui';

...
  async run({ steps, $ }) {
    const result = {};

    const aui = await UiControlClient.build({
      credentials: {
        workspaceId: process.env.workspaceId,
        token: process.env.token,
      },
      uiControllerUrl: this.uiControllerUrl
    });
    await aui.connect();

    // AskUI Workflow will be added here

    aui.disconnect();

    return result;
  },
})

3. Write the AskUI Workflow

When you look at our example form, we want to fill out, you notice that we have to do the following things:

Write simpleForm@authenticationtest.com into the textfield E-Mail Address
Write pa$$w0rd into the next textfield
Click the button Log In
Validate the success

The less inference we invoke the faster the AskUI workflow will execute. Everything that prompts AskUI to search for an element on the screen invokes inference. So let us try to invoke the inference only once by finding the first textfield for the E-Mail Address. Then we will use keypresses to navigate the form. This is the code to achieve this:

// This line only works with the Gitpod setup shown in the next section
// Select the browser url textfield for your use case with the appropriate instruction!
await aui.typeIn('https://authenticationtest.com/simpleFormAuth/')
         .textfield()
         .contains()
         .text()
         .containsText('docs.askui')
         .exec();
await aui.pressKey('enter').exec();

// Fill out the form
await aui.typeIn('simpleForm@authenticationtest.com')
         .textfield()
         .contains()
         .text('E-Mail Address')
         .exec();
await aui.pressKey('tab').exec();
await aui.type('pa$$w0rd').exec();
await aui.pressKey('tab').exec();
await aui.pressKey('enter').exec();

// Check if the the login succeeded: Login Success is shown on the page
// Pass result to next step
try {
  await aui.expect().text('Login Success').exists().exec();
  result.success = "Smoke Test successful!";
} catch(error) {
  result.success = "Smoke Test failed!";
}

aui.disconnect();

return result;

4. Send an Email

Doing a smoke test without reporting about its success state would not help us. So we will just send us an email with the Send Yourself an Email action.

As subject we choose Smoke Test State and for the text we reference our success variable we returned in our action above with {{steps.code.$return_value.success}}.

Gitpod As Remote Machine

If you do not have a remote machine ready-to-go you can use a service like Gitpod. We have a prepared Try-Out-Repository which comes with AskUI already setup and a VNC to observe the AskUI workflow. Start the repository in Gitpod over the Open in Gitpod-button and let it finish the predefined AskUI workflow. When it reached the AskUI Docs (docs.askui.com) maximize the browser window in the Simple Browser tab.

Switch to the TERMINAL tab and start the AskUI-Controller with the following command:

./node_modules/askui/dist/release/latest/linux/askui-ui-controller.AppImage

Also make sure that you expose the port to the AskUI Controller (open lock icon). Head to the PORTS tab and make the port 6769 public. Then copy the URL and add it as the property uiControllerUrl in the Pipedream action.

Conclusion

Building a smoke test with Pipedream and AskUI was a practical use case to see how both tools integrate. The simplicity of Pipedream and its ability to integrate JavaScript code and Node packages was helpful. With that AskUI could be setup seamlessly inside an action and connected to an external AskUI Controller.

Use AskUI and Cucumber Together

Johannes Dienst — Wed, 24 Jul 2024 07:27:45 +0000

Use AskUI and Cucumber Together

By defining the behavior of a system in a structured format like Gherkin, Behavior-Driven Development (BDD) enables teams to bridge the gap between stakeholders, testers, and developers, avoiding misunderstandings and reducing rework. As a collaborative approach, BDD encourages all parties to work together from the outset, ensuring that everyone is on
the same page and that requirements are accurately captured.

In this process, Cucumber is a popular tool used to implement BDD, enabling teams to write clear, executable tests that ensure the system behaves as expected.

In this blog post, we'll show you how to set up Cucumber in conjunction with AskUI to define AskUI workflows using BDD principles.

Prerequisites

AskUI installed and configured on your system (Windows, Linux, macOS)
Delete askui_example/my-first-askui-test-suite.test.ts after initialization

Prepare Setup

Cucumber does not play nice with AskUI's default setup yet (Version 0.20.3). For AskUI to play nice with Cucumber you need to do two small preparations as AskUI uses Jest as its runner.

1. Change Jest's `testEnvironmentOptions`

In the file askui_example/helpers/jest.config.ts you have to disable that code is included in the run report. You achieve this by adding a testEnvironmentOptions property with the addCodeInReport property set to false.

const config: Config.InitialOptions = {
  ...
  testEnvironment: '@askui/jest-allure-circus',
  testEnvironmentOptions: {
    addCodeInReport: false
  },
};
...

2. Tell Jest Where to Find the Implementation for The Step Definitions

Also in askui_example/helpers/jest.config.ts you need to expand the default testMatch property. It must include files ending in step.ts because we will store the implementation there.

...
const config: Config.InitialOptions = {
  ...
  testEnvironment: '@askui/jest-allure-circus',
  testEnvironmentOptions: {
    addCodeInReport: false
  },
  testMatch: [ "**/__tests__/**/*.[jt]s?(x)", "**/?(*.)+(spec|test|step).[jt]s?(x)" ]
};
...

Install `jest-cucumber`

The easiest way to use Jest together with Cucumber is the npm-package jest-cucumber. Let's install it with the following command:

npm install --save-dev jest-cucumber

Create a Basic Features File

Create a folder features and in there a Feature file NavigateToWebsite.feature

project_root/
├─ askui_example/
├─ features/
  ├─ NavigateToWebsite.feature
├─ node_modules/
├─ ...

Write the following basic Feature into this file:

Feature: Navigate to a website

Scenario: Entering the correct URL into the browser address bar
  Given I am on the Google search page
  When I type in the URL for AskUI practice page
  Then I will land on the webpage

Create the Step Definitions Implementations

Create the step definition file askui_example/navigate-to-url.step.ts where each test maps to a specific scenario.

import { defineFeature, loadFeature } from 'jest-cucumber';
import { aui } from './helpers/askui-helper';

// Load the feature file
const feature = loadFeature('features/NavigateToWebsite.feature');

defineFeature(feature, test => {

  // Maps to 'Scenario' in your feature file
  test('Entering the correct URL into the browser address bar', ({ given, when, then }) => {

    given('I am on the Google search page', async () => {
      await aui.moveMouse(500, 500).exec();
      await aui.mouseLeftClick().exec();
      await aui.pressTwoKeys('command', 't').exec();
    });

    when('I type in the URL for AskUI practice page', async () => {
      await aui.typeIn('https://askui.github.io/askui-practice-page/').textfield().exec();
      await aui.pressKey('enter').exec();
    });

    then('I will land on the webpage', async () => {
      await aui.expect().text('Welcome to the AskUI Practice Page').exists().exec();
    });

  });
});

Run The Workflow

Open your browser in full screen and start the workflow with:

npm run askui

You should see that the workflow run will open a new tab and navigate to AskUI's practice page.

Conclusion

Combining AskUI with Cucumber enables you to write AskUI workflows in BDD style. Executing your tests like a real human-user will make them more realistic for every stakeholder.

Common Terms in Computer Vision

Johannes Dienst — Tue, 09 Jul 2024 10:14:42 +0000

When you read about Artificial Intelligence, Machine Learning and Computer Vision you often come across terms that seem to be common.

But do you know all of them and what they actually mean?

In this blog we will list the most common terms that we use in our day-to-day work.

Bounding Box

A Bounding Box is a rectangle around a detected element or region in a picture. In code they are represented with their coordinates. There are different formats used. Some examples:

pascal_voc: [xmin, ymin, ymax, ymax]
coco: [xmin, ymin, width, height]
yolo: normalized [xcenter, ycenter, width, height]

Example in JSON:

{
  "xmin": "1128.2720982142857",
  "ymin": "160.21332310267857",
  "xmax": "1178.8204241071428",
  "ymax": "180.83512834821428"
}

Annotation

An annotation can be the visual representation of a bounding box or just a keypoint. But also a textual description is possible.

Usually bounding boxes are used in grounding together with labels and to establish a ground-truth or reference for trainingdata and testdata for training computer vision models.

Annotations differ from labels in that they can include any additional metadata.

Label

A label is given to a bounding box or a keypoint and identifies which class or category an object or area the model assigned to it. In the above example the person-text above the bounding box is a label.

Labels are also used to train models where the training set consists of correctly labeled images.

Image Classification

Image classification is used to put a predefined label on an image.

For an image of a cat would get the label cat, while the image of a sunset would get the label sunset.

This seems like a trivial thing to do but it actually has a lot of practical use cases:

Recognition of objects or entities depicted in images
Classifying medical image for detecting abnormalities
Image retrieval based on categories

Image Segmentation

The goal of image segmentation is to divide an image into areas that belong together. The areas are visually distinctive from each other.

The output of the process is a set of segmentation masks that can be used to process the image further.

Take this segmentation of an avatar image where the masks define where a person is and where the background is. A practical application would be to remove the background based on these masks.

Object Detection

Object detection is used to identify objects in images and video. It is a crucial task in a lot of domains such as Surveillance and Security, Environmental Monitoring and Augmented Reality.

It takes visual input and determines the objects that are present. The objects are described with a bounding box and also get a class-label and sometimes a confidence score.

Feature

A distinctive feature in an area or keypoint of an image. Usually you attribute features to a specific class you want to detect.

For a mouse those could be the shape, surroundings or fur pattern. Those are represented as numerical vectors, called descriptors, that capture the visual properties of an area or keypoint.

Feature Vector

Feature vectors are not limited to Computer Vision, but used there for all kind of tasks.

A feature vector is generated from input data such as an image and contains one numerical entry for every feature. Feature vectors are typically high-dimensional.

In practice feature vectors get generated by a Backbone (see next heading) which are by themselves models trained to extract useful features.

Grounding

Grounding is the process of connecting high-level concepts, for example Mouse, with visual features, for example tail.

Now that you have high-level concepts and their representation in the model you can also put them into context and relation with each-other.

Grounding can also be done by humans for training data. There you annotate an input with bounding boxes and labels, so the model has a Ground Truth to learn from.

Referring Expression

Referring Expression in Computer Vision are used to describe objects or areas in images. They can contain attributes like color or relations to other objects.

Take this example: "The bike next to the child." You do not point to the bike directly but give a reference point child. Or you could give a description: "The red building next to the blue car.".

Referring Expressions are typically used in tasks like object localization, object detection or image retrieval.

Zero-Shot

In general zero-shot is when models can do tasks they were not specifically trained on.

To some extend, and most of them rather poorly, every model can perform tasks without specific training as the goal of training is to get a higher-level understanding of concepts.

Zero-shot learning in Computer Vision also includes more information than just labeled examples. The focus is on data that describes relationships between categories, so the model can generalize better to new categories.

This also requires large scale training on a lot of data, so the model is exposed to billions of examples and can generalize without specific training to new tasks.

Conclusion

That are all, but sure there are more. Do you know of any we should include?

Let us know on Social Media.

Tutorial: Automate Highlighting in WYSIWIG Editors

Johannes Dienst — Tue, 26 Sep 2023 22:00:00 +0000

Automate Highlighting in Wysiwyg Editors

Working in What-You-See-Is-What-You-Get editors (WYSIWYG) can be very tiring and also taxing on your wrists. Things like highlighting the same thing over and over again can often not be done with search-and-replace, because there is just no button for it 😤

I recently had to put a transcript of a podcast into such an editor and had to bold the speaker hints, so that you clearly know who was speaking in that part of the podcast. It looks like this:

JD: Welcome to J^2 Talk AI

JH: I think first of all we should understand...

It is always alternating between JD and JH as that is the nature of a conversation 😋.

Bolden the beginning speaker hints would need me to double-left-click the abbreviations and then hit the shortcut cmd+b (I am on macOS) in this particular editor.

I do not want to do this by myself as that would mean to do roughly 20*2 = 40 actions on average per episode. And I had 5 episodes... Extremely straining on your wrists and arms and it is dull work also!

In this tutorial I will show you how I automated this task with AskUI and also how to work around some kinks of WYSIWIG-Editors like microanimations and strange behaviour.

Prerequisites

AskUI installed: Windows, Linux, macOS
Open your favorite WYSIWIG-Editor

The Problem

So what do we need to do?

Find all the JD and JH
Double-left-click each one
Press cmd+b (macOS) or (ctrl+b Linux and Windows)

This is how it looks:

Let Us Develop It Step-By-Step

So the first thing that needs to be done is finding all the JD and JH. This can be done with the get() command. As you can see in the demo above there are possibly multiple pages, so we have to scroll somehow. The easiest way to do this is to press the pagedown-key. So let us be conservative and press that key three times.

Every time we press it there is a scroll animation that might mess up the inference. So let's wait two seconds with waitFor(2000). That should be enough to finish the animation.

for (let i = 0; i < 3; i++) {
  const jds = await aui.get().text().containsText("JD:").exec();

  const jhs = await aui.get().text().containsText("JH:").exec();

  await aui.pressKey("pagedown").exec();
  await aui.waitFor(2000).exec();
}

This already looks like it would be a good idea to refactor it and put the actual highlighting code into a function highlight().

for (let i = 0; i < 3; i++) {
  highlight();

  await aui.pressKey("pagedown").exec();
  await aui.waitFor(2000).exec();
}

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();

    const jhs = await aui.get().text().containsText("JH:").exec();
}

Great, now we need to mouse-double-left-click the text JD and JH one by one with a for-loop. But when we try it with await aui.moveMouseTo(jds[0]).exec() it will move the cursor to the middle of the line where JD/JH is at the beginning. We have to use the bounding box x-min and y-min values to get to the beginning of the line. Also we have to add like 10 pixels to the values to actually move the cursor over the JD/JH.

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();
    for (let i = 0; i < jds.length; i++) {
        const bbox = jds[i].bndbox;
        await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();
    }

    ...
}

Now we can do the highlighting by mouse-double-left-click and press the bold shortcut.

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();
    for (let i = 0; i < jds.length; i++) {
        const bbox = jds[i].bndbox;
        await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();
        await aui.mouseDoubleLeftClick().exec();
        await aui.pressTwoKeys("command", "b").exec(); // Use CTRL+b on Linux/Windows
    }

    ...
}

That should highlight all the JD. Let's do the same with JH. Also we need to add a keypress of escape here because the mini-menu that pops up when double-left-clicking the last JD might cover a JH

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();
    for (let i = 0; i < jds.length; i++) {
        const bbox = jds[i].bndbox;
        await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();
        await aui.mouseDoubleLeftClick().exec();
        await aui.pressTwoKeys("command", "b").exec();
    }

    await aui.pressKey("escape").exec();

    const jhs = await aui.get().text().containsText("JH:").exec();
    for (let i = 0; i < jhs.length; i++) {
        const bbox = jhs[i].bndbox;
        await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();
        await aui.mouseDoubleLeftClick().exec();
        await aui.pressTwoKeys("command", "b").exec();
    }
}

Two More Tweaks - Complete Code

Another problem you might run into is missing focus on the editor. This is why adding the first two lines to your code is beneficial. It moves the mouse cursor into the editor and with the left-click gets the focus.

Also extracting a function boldenText(bbox) that does the highlighting given a bounding box bbox avoids duplicate code.

Here it is, the complete code:

it('highlights all the JD and JH', async () => {
    await aui.moveMouse(500, 500).exec();
    await aui.mouseLeftClick().exec();

    for (let i = 0; i < 3; i++) {
        await highlight();
        await aui.pressKey("pagedown").exec();
        await aui.waitFor(2000).exec();
    }

})

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();
    for (let i = 0; i < jds.length; i++) {
        await boldenText(jds[i].bndbox);
    }

    await aui.pressKey("escape").exec();

    const jhs = await aui.get().text().containsText("JH:").exec();
    for (let i = 0; i < jhs.length; i++) {
        await boldenText(jhs[i].bndbox);
    }
}

async function boldenText(bbox) {
  await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();

  await aui.mouseLeftClick().exec();
  await aui.mouseDoubleLeftClick().exec();
  await aui.pressTwoKeys("command", "b").exec();
}

Conclusion

In this tutorial you learned how to automate tedious tasks like highlighting some recurring text in a WYSIWYG-Editor.

You also learned how to overcome some of the challenges user interfaces throw at you, like mini-popups and micro-animations.

Photo by Pankaj Patel on Unsplash

Visual Regression Testing with AskUI and Jest

Johannes Dienst — Mon, 07 Aug 2023 07:07:00 +0000

Visual Regression Testing (VRT) is a useful method to guard yourself against unwanted layout and visual changes that can occur with new version of a Graphical User Interface (GUI).

AskUI is not a testing tool in the original sense but an automation tool for all GUIs that is platform-independent. Wouldn't it be nice to combine VRT with AskUI, so you do not have to maintain two different projects.

Luckily you can do this because Jest - AskUIs favourite runner - can be extended to provide VRT, while AskUI delivers the screenshots to test on.

In this blog you will learn how to use AskUI in combination with Jest and the package jest-image-snapshot to include a visual regression test in your AskUI runs.

Prerequisites

AskUI installed: Windows, Linux, macOS

Setup `jest-image-snapshot`

First you need to install the package that provides the VRT capabilities: jest-image-snapshot.

Installation

Install it as a dev dependency:

npm install --save-dev jest-image-snapshot

Integration

The package provides a function toMatchImageSnapshot which implements Jest's Matchers<R> interface making it a Jest matcher that can be used with Jest's expect().

We have to add this matcher to Jest with expect.extend like this in your workflow file (See the docs):

const { toMatchImageSnapshot } = require('jest-image-snapshot');
expect.extend({ toMatchImageSnapshot });

Now the only thing you have to provide for a regression test in your workflows is an image to the function expect() as Buffer:

expect(imageBuffer).toMatchImageSnapshot();

You can use AskUI to get the screenshot as a base64 encoded string. The string is in the format of a data URL. So you have to strip the scheme data:[<mediatype>][;base64], away because it is not a valid image with the scheme. Here is the full code:

const annotate = await aui.annotate();
const imageBase64 = annotate.image;

// Strip away this: data:image/png;base64,
// Not valid png -> Only if used as a data URL
let imageBuffer = Buffer.from(
    imageBase64.split('base64,')[1], 'base64');

expect(imageBuffer).toMatchImageSnapshot();

How to Get it to work with TypeScript

When you run this you will get an error:

FAIL  test/my-first-askui-test-suite.test.ts
  ● Test suite failed to run

  test/my-first-askui-test-suite.test.ts:16:25 - error TS2551: Property 'toMatchImageSnapshot' does not exist on type 'JestMatchers<Buffer>'. Did you mean 'toMatchSnapshot'?

  16     expect(imageBuffer).toMatchImageSnapshot();
                            ~~~~~~~~~~~~~~~~~~~~

    node_modules/@types/jest/index.d.ts:1111:9
    1111         toMatchSnapshot<U extends { [P in keyof T]: any }>(propertyMatchers: Partial<U>, snapshotName?: string): R;
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      'toMatchSnapshot' is declared here.

This is because Jest's typings in jest.d.ts do not include toMatchImageSnapshot.

Luckily TypeScript has a mechanism called Declaration Merging. When we create a file jest.d.ts in our project and declare our matcher there, TypeScript will pick it up.

declare namespace jest {
    interface Matchers<R> {
      toMatchImageSnapshot(): CustomMatcherResult;
    }
}

When you rerun it you will get no error 🥳.

How `jest-image-snapshot` Works

From the repository:

Given an image (Buffer instance with PNG image data) the toMatchImageSnapshot() matcher will create a image_snapshots directory in the directory the test is in and will store the baseline snapshot image there on the first run. Note that if the customSnapshotsDir option is given then it will store baseline snapshot there instead.

On subsequent test runs the matcher will compare the image being passed against the stored snapshot.

To update the stored image snapshot run Jest with --updateSnapshot or -u argument. All this works the same way as Jest snapshots.

You also want to check out the options you can provide to toMatchImageSnapshot() to fine-tune the behaviour. For example you might want to set a threshold for a mismatch so that minimal differences do not fail a run:

expect(image).toMatchImageSnapshot({
    failureThreshold: 0.01,
    failureThresholdType: 'percent'
  });

Conclusion

You can use AskUI in combination with a Visual Regression Testing library that integrates with Jest like jest-image-snapshot.

With this you can validate your GUI's final state and ensure it is the one you expected 🥳.

Use the Power Of TypeScript in AskUI

Johannes Dienst — Fri, 30 Jun 2023 13:28:28 +0000

Optimizing UI Automation with Functions in TypeScript

In the past, we've discussed using if and else statements, try and catch blocks, and for loops in TypeScript for building UI automations.

However, to avoid repeating code and add real UI interactivity to your AskUI workflows, it's essential to harness the power of functions. Functions can help create reusable and efficient code for checking if elements are present and reacting accordingly.

The Problem

When you want to check if an element is present on your UI you might do it like this and do something else when the element is not found:

try {
    await aui.expect().text('<Your text here!>').exists().exec();
  } catch(error) {
    // Do something else
  }

This is totally fine, but once you create more sophisticated workflows you will notice that you need a variation of this code 1,2,3 ... x times.

You can do better if you extract this into a function!

Extract into a function

To get started, create a util.ts file in the same folder as your automation file. First, import the UiControlClient into this file:

import { aui } from './helper/jest.setup';

Then you can create a function in this file that takes a parameter text and returns true or false depending on if the element is present or not. This function is asynchronous and checks for the presence of a specific element on the screen. It returns a boolean value to indicate whether the element is present or not.

export async function checkSuccess(text: string) {
  try {
    await aui.expect().text().withText(text).exists().exec();
    return true;
  } catch(error) {
    console.log(`${text} not found!`);
  }
  return false;
}

The export keyword makes sure we can use it in our actual AskUI-workflow file.

// Do this in your actual AskUI workflow file at the start
import {checkSuccess} from './utils';

// Usage
if ((await checkSuccess('Success')) === false) {
  // React to text not being there
};

Implementing it like this makes it possible to reuse the this functionality for all text you need. It also improves the readability of your workflow file 🥳

Endless Possibilities

You can wrap (nearly) any functionality you need in a handy function.

Here's an example of a waitUntil() function that you can use to wait for an element or an app to appear on the screen, even if it takes longer than a few seconds:

// Retry the command 5 times with a
// wait time of 2 seconds between each try
async function waitUntil(askuiCommand: Promise<void>, maxTry = 5) {
    try {
        await askuiCommand;
    }
    catch (error) {
        if (maxTry == 0) {
            throw error;
        }
        console.log(`Retry predicting command, ${maxTry} tries left`);
        await aui.waitFor(2000).exec();
        await waitUntil(askuiCommand, maxTry - 1);
    }
}

// Wait for the text 'Github' to be displayed
await waitUntil(
    aui.expect().text().withText('Github').exists().exec();
)

Conclusion

Leveraging functions in TypeScript significantly streamlines your UI automation efforts and provides numerous opportunities for optimization.

Take a Screenshot with AskUI - Also on Fail!

Johannes Dienst — Tue, 02 May 2023 12:47:16 +0000

When you develop UI-Automations you often run into the problem that they fail inexplicably 😥.

This leads to frustration and oftentimes stops the effort entirely!

We will show you how can level up your debugging skills by teaching you a way to take screenshots during an AskUI execution and when on fail of an execution.

With this knowledge you can lower your frustration and become faster when developing UI-Workflows because you see what AskUI sees.

Prerequisites

AskUI SDK installed: [Windows, Linux, macOS

That is it 😉

Take a Screenshot with annotate()

Do you want to know what AskUI sees at a specific point in time when executing your code?

Luckily the image can be obtained by using AskUI's built-in method annotate() and a little bit of TypeScript.

First you need to import the fs module at the start of your automation file. You will use it to save the file to a specific place on your disc.

After you called annotate you save the returned object into a variable. This object contains the image encoded in base64.

You can extract it with a Buffer and write it to your disc with fs.writeFileSync(<path to file>, buffer)

// Add this to the start of your AskUI-file containing your workflows/instructions
import * as fs from 'fs';

// First, get all the information from the annotation
// This will also save an interactive HTML
// file to the 'report/' folder
const annotation = await aui.annotate();

// The screenshot is contained as a string in 'base64' format
// Create a buffer with the base64 image
let buffer = Buffer.from(annotation.image.split('base64,')[1], 'base64');

// Write the file
// Name as you want 
// (./ is your project folder)
fs.writeFileSync("./test.png", buffer);

Take a Screenshot on Fail

Normally you do not take a screenshot if nothing fails. But in case your UI-Automations fails at some point it is helpful to know Why it might have failed. Was an element missing? Or was the UI just too slow to render?

You can enable this behavior by enabling a setting in your AskUI UI Control Client. For the default Jest setup, you have to go to the file test/helper/jest.setup.ts.

There you have to set annotationLevel property of the UiControlClient to ON_FAILURE as shown in the following code:

aui = await UiControlClient.build({
  annotationLevel: AnnotationLevel.ON_FAILURE
});

A more detailed explanation is documented in the API Docs of the AskUI UI Control Client.

Conclusion

With these two methods you can see what AskUI sees at any given point of your UI-Automation. This will speed up your debugging.

Get Support

If you have a recurring or persisting issue, don’t hesitate to ask the Discord community for help!

'How the Hack?' AskUI Hackathon Winners Highlights

Johannes Dienst — Wed, 26 Apr 2023 06:35:24 +0000

It was an intensive and fun time to host our How the Hack? hackathon at AskUI with the theme Beyond the Browser.

In this blog post we will share our thoughts about the first three winners and highlight their awesome projects.

Let us dive right in with the first place 🥳

1st Winners: Gaurav Khurana, Anindita Rath, Mahathee Dandibhotla

The first place not only implemented one but three use cases. Hence they had to speed up their video presentation to stay in the five minute limit. Make sure to slow it down when you watch it.

They focused on mundane tasks that steal a lot of time in your daily live.

WhatsApp group chat/media cleaner
Get content from people you follow on social media by scraping it and send it via email
Recycle Bin cleaner

While all three sound easy to do they fit perfectly well into out theme Beyond the Browser and improve your quality of life effectively.

You can out the code from this Google Drive.

Judges Quotes

The feedback from our judges was positive all the way:

Very well explained problem statements. Problems that affect our everyday life was used the idea. AskUI was efficiently used.

Great at Presentation, Storytelling, Creating a compelling use case. Good one.

Video Presentation

2nd Winner: Sheilendra Omkar

The goal of our runner-up was to ease the use of Web3 wallets as that requires connecting it to a user account. This is usually tedious and has to be done for each product separately.

What impressed us the most besides the presentation and project idea was the amount of detail and care displayed in the accompanying Github-Repository.

Judges Quotes

The judges were very happy because the submission focused on the hackathon theme and also delivered a cross-platform use case:

Good use of the tool and sticking to this year's theme.

This project is automating a desktop app, which is beyond the browser and using AskUI library efficiently. Hence a good idea. Well structured code.

Video Presentation

3rd Winner: Phil Wong

Our second runner-up made an impression with a well-delivered presentation of his submission. And also the use case was very innovative as you can see in the judges comments below.

Phil automated the login to the AskUI User Portal. What does that have to do with our theme? He did it all with keypress-input only and captured the accessibility of the whole flow with the NVDA-Screenreader.

Check out the GitHub-Repository.

Judges Quotes

Our judges had some warm comments for this one, too:

Accessibility Testing & AskUI pairing is a great idea for me. Would love to see if you evolve this more in the future.

Very novel idea and fitting for this year's theme of 'Beyond the browser'. NVDA or any screen reader testing is a little hard and boring sometimes but this initiative lends a helping hand towards it.

Video Presentation

Conclusion

It was awesome to see how the participants used our tool to create innovative automations for their use cases.

Also the attention to detail when it comes to documentation and presentation astonished us.

We also learned a lot about the pain points in our product and are deeply grateful for all the feedback we got!

Thanks for making the hackathon worth the effort 🥰

Automating WebGL-/Canvas-based Website

HaramChoi — Tue, 18 Apr 2023 14:41:44 +0000

WebGL-based websites are becoming increasingly popular among web developers looking to create immersive and interactive user experiences. By drawing graphics and animations directly on the canvas, they offer developers a high degree of creative control over the user interface, providing a more flexible and dynamic platform for graphical compositions. They also offer a more interactive user experience, allowing users to interact with graphics and animations in real-time.

However, WebGL-based websites also present a unique challenge for automation, as they often lack DOM elements that conventional selector-based test frameworks rely on.

In this tutorial, we will demonstrate an automation case using AskUI to automate a WebGL-based website. Since AskUI relies solely on what is visible on the screen, you will discover how easy it is to automate even such a canvas-based website without relying on any DOM element.

Visit the Target Website

For the demonstration, we will use the website below. First, visit this website. If you inspect the source of this website, you will discover that it consists of a single canvas element without giving any clue on how to programmatically automate user interaction.

https://chartogne-taillet.com

Disclaimer: We use this website merely for our demonstration purpose and don't intend to promote any product or brand from this website.

Plan the Automation

Our final goal of this automation is to visit a page of this website that gives detailed information about a certain grapevine. Below are the steps we will automate:

Open the web browser and go to the website.
Click Accept for the cookie pop-up.
Click the text Enter on the first page.
Click the text that is a certain sort of grapevine.
Move the mouse cursor to the interaction point.
Scroll the mouse wheel.

1. Visiting the Website

The first step of the automation is to open the web browser and go to the website. Use the code below, and adjust the necessary part for your environment:

import { aui } from './helper/jest.setup';

describe('interact with WebGL', () => {
  it('opens browser and go to website', async () => {
    // Windows
    await aui.pressKey('command').exec(); 
    // macOS
    // await aui.pressTwoKeys('command', 'space').exec();
    await aui.type('chrome').exec();
    await aui.pressKey('enter').exec();
    // wait for the browser to open
    await aui.waitFor(1000).exec(); 

    // run this if the address bar is not focused after opening chrome
    await aui.click().text().withText('Search Google or type a URL').exec();
    await aui.type('https://chartogne-taillet.com').exec();
    await aui.pressKey('enter').exec();
  })

  it('click accept', async ()=>{
    await aui.click().text().withText('accept').exec();
  });
});

After successfully running the code, we will see this page:

2. Click Enter and go to Les Couarres

Nextly, we will click the text Enter and the page will transit to the next page. After that, we press the text Les Couarres on the bottom right part of the page. Note that these texts are not DOM elements, but merely texts drawn on the canvas as a bitmap. So there is no clue how to interact with these text-buttons if you inspect the source HTML of the page. Add this code block to the end of the code:

it('clicks enter', async ()=>{
  await aui.click().text().withText('Enter').exec();
   // wait for the page transition
  await aui.waitFor(5000).exec();
  await aui.click().text().withText('Les Couarres').exec();
});

After running the code, we will be on this page:

3. Scroll on the Text

Now we want to move on to the page where we can get detailed information about Les Couarres. To do so, we have to interact with the text DISCOVER that is placed at the mid-bottom of the screen:

it('scrolls on the text', async ()=>{
  await aui.moveMouseTo().text().withText('DISCOVER').exec();
   // negative direction on y axis will scroll down
  await aui.scroll(0, -2000).exec();
  await aui.scroll(0, -2000).exec();
  await aui.scroll(0, -2000).exec();
});

Try to adjust the amount of scrolling that suits your condition, for example, decrease it to -3000 if you have a high-resolution display. When scrolling on the text, you will see a smooth transition to another page. Clicking the text also triggers the same result, so give it a try if you want to!

There will be an animation until the page transition gets done. And after the transition, we will be on a page about a certain vineyard and grapes. Congratulations! You just successfully automated mouse interactions with WebGL contents!

Conclusion

In this article, we demonstrated how to automate a WebGL-based website using AskUI. By relying solely on what is visible on the screen, AskUI can interact with canvas-based websites without requiring any selectors or tags. We walked through a step-by-step guide to visit a certain page of a website, click on non-DOM text elements, move the mouse cursor toward a bitmap text, and scroll on a specific bitmap text, all of which are common challenges when automating WebGL-based websites. With AskUI, you can easily automate even the most complex canvas-based user interfaces, providing a more efficient way to test and validate the user experience of your WebGL applications.

Did you get any issues while using AskUI? Drop by our Discord if you are seeking help!

How To Do Assertions in AskUI

Johannes Dienst — Mon, 17 Apr 2023 07:38:19 +0000

Real interactivity is what characterizes what an end-user is doing on a User Interface (UI). This is why it is important to extract data from your UI into your tool and use it later, for assertions for example 🔧.

We will show two ways to use AskUI for real interactivity by extracting text out of your UI and use it for assertions:

AskUI assertions
Jest assertions

Prerequisites

AskUI SDK installed: Windows, Linux, macOS
A browser to open this page

Explaining the Use Case

When you open the page linked above it takes you to an interactive login form.

We will do the following steps to create a real end-user workflow:

Enter the E-Mail and Password
Extract the captcha value
Enter it into the designated captcha textfield
Click the Login-button
Verify the login succeeded by searching for the text Login Success

Enter E-mail and Password

The first thing we need to do is enter the given values for E-Mail and Password.

// Get focus on browser window
// Do not forget to move your mouse pointer
// over to the browser window!
await aui.mouseLeftClick().exec();

await aui.typeIn('bootstrap@authenticationtest.com')
         .textfield()
         .contains()
         .text()
         .withText('E-Mail Address')
         .exec();
await aui.typeIn('pa$$w0rd')
         .textfield()
         .contains()
         .text()
         .withText('Password')
         .exec();

Extract the CAPTCHA Value

Now we start with the interactive part 🥳. Extract the value of the text CAPTCHA with get(). We find the text above the textfield that contains the value 123456 and match the CAPTCHA with some TypeScript-RegEx-magic.

const texts = await aui.get()
                       .text()
                       .above()
                       .textfield()
                       .contains()
                       .text()
                       .withText('123456').exec();

// texts contains an array!
// The phrase with the captcha is the first element
const captchaInText = texts[0];

// We have to extract the value with TypeScript
// It is 6 digits long: Matching it with RegEx
// Match is also an array and we want the first element
const captcha = captchaInText.text.match(/\d{6}/)[0];

Enter the CAPTCHA Value Into the Designated Textfield

Feed the extracted value back to the textfield designated for it. Also lose the focus so we do not interfere with Optical Character Recognition (OCR) when we want to assert the value is really there.

await aui.typeIn(captcha)
         .textfield()
         .contains()
         .text().withText('123456')
         .exec();

// Lose focus on the textfield
// Cursor may interfere with OCR
await aui.pressKey('tab').exec();

Assert with Jest-Assertions That the Value was Entered Correctly

To assert if the CAPTCHA was entered successfully we will show you two approaches to find the correct textfield and extract its value:

// IMPORTANT:
// Add this line to the top of your file
import {expect} from '@jest/globals';

// APPROACH 1
// Assert that the captcha was actually entered
// Extract the actual value first
const actuallyEntered = await aui.get()
  .text()
  .withTextRegex('^\\d{6}')
  .exec();
expect(actuallyEntered[0].text).toBe(captcha);

// APPROACH 2    
const actuallyEntered2 = await aui.get()
  .text()
  .below()
  .text()
  .withTextRegex(".*in the box")
  .exec();
expect(actuallyEntered2[0].text).toBe(captcha);

Click the Login-Button

If the assertions do not fail we get to click the Log in-button.

await aui.click().button().withText('Log In').exec();

Verify Success with AskUI Assertions

When developing the code we noticed that sometimes the success window opened in a new browser tab. But sometimes not 😋

So we had to wrap our assertion in a try/catch-block to account for this scenario. If the Success-Page shows in the same browser tab the first instruction will succeed. If not it will throw an error.

In the catch-clause we will then try to switch to the next browser tab and find the Success-message there. If we fail again the login must have failed.

// Sometimes the site opens the success message
// in another tab
try {
  await aui.expect()
           .text()
           .withText('Login Success')
           .exists()
           .exec();
} catch(error) {
  // Switch to next browser tab
  // Windows Linux: control + tab
  // await aui.pressTwoKeys('control', 'tab').exec();

  // macOS: CMD + option + right arrow
  await aui.pressThreeKeys('command', 'alt', 'right')
           .exec();

  await aui.expect()
           .text()
           .withText('Login Success')
           .exists()
           .exec();
}

Complete Code

import { aui } from './helper/jest.setup';

import {expect} from '@jest/globals';

describe('jest with askui', () => {

  it('assert', async () => {

    // Get focus on browser window
    // Do not forget to move your mouse pointer
    // over to the browser window!
    await aui.mouseLeftClick().exec();

    await aui.typeIn('bootstrap@authenticationtest.com')
             .textfield()
             .contains()
             .text()
             .withText('E-Mail Address')
             .exec();
    await aui.typeIn('pa$$w0rd')
             .textfield()
             .contains()
             .text()
             .withText('Password')
             .exec();

    const texts = await aui.get()
                           .text()
                           .above()
                           .textfield()
                           .contains()
                           .text()
                           .withText('123456')
                           .exec();

    // texts contains an array!
    // The phrase with the captcha is the first element
    const captchaInText = texts[0];

    // We have to extract the value with TypeScript
    // It is 6 digits long: Matching it with RegEx
    // Match is also an array and we want the first element
    const captcha = captchaInText.text.match(/\d{6}/)[0];

    await aui.typeIn(captcha)
             .textfield()
             .contains()
             .text()
             .withText('123456')
             .exec();

    // Lose focus on the textfield
    // Cursor may interfere with OCR
    await aui.pressKey('tab').exec();

    // Assert that the captcha was actually entered
    // Extract the actual value first
    const actuallyEntered = await aui.get()
                                     .text()
                                     .withTextRegex('^\\d{6}')
                                     .exec();
    expect(actuallyEntered[0].text).toBe(captcha);

    const actuallyEntered2 = await aui.get()
                               .text()
                               .below()
                               .text()
                               .withTextRegex(".*in the box")
                               .exec();
    expect(actuallyEntered2[0].text).toBe(captcha);

    await aui.click()
             .button()
             .withText('Log In')
             .exec();

    // Sometimes the site opens the success message
    // in another tab
    try {
      await aui.expect()
               .text()
               .withText('Login Success')
               .exists()
               .exec();
    } catch(error) {
      // Switch to next browser tab
      // Windows Linux: control + tab
      // await aui.pressTwoKeys('control', 'tab').exec();

      // macOS: CMD + option + right arrow
      await aui.pressThreeKeys('command', 'alt', 'right')
               .exec();

      await aui.expect()
               .text()
               .withText('Login Success')
               .exists()
               .exec();
    }

  });
});

Conclusion

In this tutorial, you learned how to assert values in your UI

With this knowledge, you can now craft your own interactive UI-workflows 😎.

Get Support

If you have a recurring or persisting issue, don’t hesitate to ask the Discord community for help!

How AskUI Works

HaramChoi — Thu, 13 Apr 2023 07:54:46 +0000

Overview

In this article, we will give a detailed overview of AskUI's architecture and how it works under the hood.

AskUI is built on top of a number of components. We will cover what these components are and how they work together to provide a flexible and reliable way to automate interactions with UI elements of any operating system or platform.

By the end of this article, whether you're a software developer, QA engineer, or automation specialist, you'll have a solid understanding of how AskUI works, and be able to use this knowledge to build more efficient automation for your project.

AskUI consists of three building blocks:

AskUI Control Client
AskUI UI Controller
AskUI Inference Server

We will step through each of them and see how they work together to perform UI automation.

Glossary

Throughout this article, we will use some terms that describe certain parts of AskUI. Some of them are used only internally and not exposed by the AskUI Control Client API, but are important for understanding how AskUI works and what it can do. Please refer to this table while reading.

Term	Description
Element-description	A method in the AskUI Control Client API that searches for a specific type of UI element. For example `button()`, `textfield()`.
Action	A method in the AskUI Control Client API that describes an action to be taken against the operating system. For example `click()`, `type()`.
InputEvent (internal)	A specific type of action to be taken against the operating system. For example MouseInputEvent or KeyboardInputEvent.
ControlCommand (internal)	A command sent to the UI controller telling what to perform on the operating system. It consists of one or more InputEvents.

AskUI Control Client

The AskUI Control Client provides the API that tells AskUI what/how to automate. Once you start using AskUI, you will mostly interact with AskUI via the AskUI Control Client. In most of our tutorials and demonstrations, you will see let aui: UIControlClient is declared and combined with an Action and Element-descriptions which ends up forming an instruction, e.g:

await aui.click().button().withText('login').exec();

As shown above, you form an instruction by chaining an Action with Element-descriptions using the Fluent API of the AskUI Control Client. It is designed as a fluent interface to increase readability and make it more understandable.
The AskUI Control Client sends a request to the AskUI UI Controller:
- to take a screenshot.
- with a ControlCommand that tells what InputEvent to perform on the operating system.
The AskUI Control Client communicates with the AskUI Inference Server:
- to send a screenshot to be annotated with the instruction.
- to receive the annotation, e.g. detected elements.

To use the AskUI Control Client, user credentials are required. User credentials can be obtained via our User Portal.

See our API documentation for more information on this component.

AskUI UI Controller

The AskUI UI Controller is a binary that controls the operating system. This binary gets automatically downloaded when the UiController is initialized by calling UiController.start(). Once executed, it stays in the background and communicates with the AskUI Control Client on a specific port to receive the ControlCommand. Based on the given ControlCommand, it triggers InputEvents respectively.

The AskUI UI Controller is responsible for:
- Taking a screenshot.
- Triggering the InputEvent, i.e MouseInputEvent, KeyboardInputEvent, or shell execution.
- Running the interactive annotation.

See our API documentation for more information on this component.

AskUI Inference Server

The AskUI Inference Server is responsible for the prediction of UI elements within the given screenshot. As soon as it receives a request from the AskUI Control Client, it performs the prediction on the given image and returns the annotation to the AskUI Control Client.

For the inference, we use a machine-learning model that consists of several submodels:

Object Detector: Detects UI elements (e.g Button, Textfield).
Icon Classifier: Predicts the class of an icon based on the detected objects (e.g. a user icon 👤).
Optical Character Recognition (OCR): Converts the image of a text into text.
Custom Element Detector: Searches for an area in the given screenshot that matches the image given by the Element-description .customElement().

Them All in Action

Assuming that we run AskUI on the same device we want to automate, the simplest synopsis can be described as such:

When running AskUI,

The AskUI Control Client checks whether it is needed to be processed by the Inference Server.
If the code contains any of the Element-description or Getters, then the AskUI Control Client tells the AskUI UI Controller to take a screenshot of the given screen and sends it to the Inference Server.

After the AskUI Control Client has retrieved the annotation back from the server, it sends a ControlCommand to the AskUI UI Controller. Afterwards, the AskUI UI Controller triggers the InputEvent on the operating system.

// an example of AskUI code containing an element-identifier
await aui.click().button().withText('Confirm').exec();
// Here, the filter 'button()' is used together,
// so the client will fire a request to the server.

If the code contains an Action but no Element-description, then the AskUI Control Client sends the ControlCommand to the AskUI UI Controller to trigger the InputEvent directly.

// an example of AskUI code containing only a command
await aui.pressThreeKeys('control','alt','del').exec();
// It uses only a command 'pressThreeKeys()',
// so it will be executed by the UI Controller directly.

An Element-description represents a specific type of UI element that can be recognized by inference. Most of the commonly used UI elements such as Button, Textfield are supported and can be used.
An Action represents a specific type of action to be performed, i.e Mouse/Keyboard Input Event or Shell Command. This action can be performed on a specific element when combined with Element-descriptions or can be performed on its own as shown in the example right above.

Please visit our API Docs, if you want to learn more about different types of Element-description and Action.

Conclusion

Here we have seen the three core components of AskUI. If you aim to use AskUI in a more advanced way, e.g. integrating it into your CI/CD pipeline, it may be worthwhile to get an overview of how it is composed. For more practical examples, please refer to our Tutorials and API docs. And don't forget to come over to our Discord community, if you have any questions about AskUI!

DEV Community: AskUI

Use Playwright and AskUI Together

Use Playwright and AskUI Together

Prerequisites

Install Playwright

Configure Playwright

Write a Workflow

Testrun

Conclusion

Run AskUI Workflows with Pipedream for Smoke Testing

Prerequisites

What Will We Build? Aka the Use Case

Our Use Case: Smoke Test with Visual Selectors

1. Add a Trigger

2. Prepare a Custom Code Action

3. Write the AskUI Workflow

4. Send an Email

Gitpod As Remote Machine

Conclusion

Use AskUI and Cucumber Together

Use AskUI and Cucumber Together

Prerequisites

Prepare Setup

1. Change Jest's testEnvironmentOptions

2. Tell Jest Where to Find the Implementation for The Step Definitions

Install jest-cucumber

Create a Basic Features File

Create the Step Definitions Implementations

Run The Workflow

Conclusion

Common Terms in Computer Vision

Bounding Box

Annotation

Label

Image Classification

Image Segmentation

Object Detection

Feature

Feature Vector

Grounding

Referring Expression

Zero-Shot

Conclusion

Tutorial: Automate Highlighting in WYSIWIG Editors

Automate Highlighting in Wysiwyg Editors

Prerequisites

The Problem

Let Us Develop It Step-By-Step

Two More Tweaks - Complete Code

Conclusion

Visual Regression Testing with AskUI and Jest

Prerequisites

Setup jest-image-snapshot

Installation

Integration

How to Get it to work with TypeScript

How jest-image-snapshot Works

Conclusion

Use the Power Of TypeScript in AskUI

Optimizing UI Automation with Functions in TypeScript

The Problem

Extract into a function

Endless Possibilities

Conclusion

Take a Screenshot with AskUI - Also on Fail!

Prerequisites

Take a Screenshot with annotate()

Take a Screenshot on Fail

Conclusion

Get Support

'How the Hack?' AskUI Hackathon Winners Highlights

1st Winners: Gaurav Khurana, Anindita Rath, Mahathee Dandibhotla

Judges Quotes

Video Presentation

2nd Winner: Sheilendra Omkar

Judges Quotes

Video Presentation

3rd Winner: Phil Wong

Judges Quotes

Video Presentation

1. Change Jest's `testEnvironmentOptions`

Install `jest-cucumber`

Setup `jest-image-snapshot`

How `jest-image-snapshot` Works