Tutorial: How to Use the get() Command of AskUI

This tutorial will show you how you can utilize AskUI's get()-command to write more powerful automation.

Until now the AskUI SDK is only allowed to test for the existence of an element with the exists()-command. While that was useful, extracting information about elements such as text was impossible. If you wanted to act based on elements values or assert a state of an element with an assertion library, you could not do that!

With the new get()-command you can extract the information AskUI inferred. Many more automation possibilities become available 🔥.

We will show you how to assert if a textfield contains a text we tried to insert beforehand. Additionally, we will demonstrate how the get() command works regarding relational selectors as below().

Prerequisites

• You have AskUI installed: Windows, Linux, macOS
• You can access this practice page

Write to A Textfield

For our test, we first write a mobile telephone number into a textfield.

// Regain the focus on our browser
await aui.mouseLeftClick().exec();

// Type into the textfield below 'Mobile Number'
await aui
  .typeIn('[phone]')
  .textfield()
  .below()
  .text()
  .withText('Mobile Number')
  .exec();

Is There a Textfield Containing our Number?

Let us now see if there is a textfield that contains a text with the mobile number we just entered.

// Save the mobile number into a variable
// Log it to console
const mobileNumberTextfield =
  await aui
    .get()
    .textfield()
    .contains()
    .text()
    .withText('[phone]')
    .exec();

console.log(mobileNumberTextfield);

It should print out the following:

[
  DetectedElement {
   name: 'TEXTFIELD',
   text: '',
   colors: [],
   bndbox: BoundingBox {
      xmin: 388.80795200892857,
      ymin: 688.3126674107143,
      xmax: 943.7193917410715,
      ymax: 734.0479352678572
    }
  }
]

Notice that there is no text in there as the mobile number is contained in the textfield as a text element.

Is It the Correct Textfield?

Ok, so there is a textfield containing the correct number. But is it the correct textfield?

const mobileNumberLabel = 
  await aui
    .get()
    .text()
    .above()
    .textfield()
    .contains()
    .text()
    .withText('[phone]')
    .exec();

console.log(mobileNumberLabel[0]);

Should log this:

DetectedElement {
  name: 'TEXT',
  text: 'Mobile Number',
  colors: [ 'white', 'gray', 'gray' ],
  bndbox: BoundingBox {
    xmin: 390.4325474330357,
    ymin: 669.3268275669643,
    xmax: 496.69222935267857,
    ymax: 683.9144112723214
  }
}

With this you can now do assertions with your favorite library or manually:

console.log(
`Is the number entered into the textfield with the label 'Mobile Number'? -> ${'Mobile Number' === mobileNumberLabel[0].text}`
);

Additional Considerations for get()

When you use get(), you must pay attention to two details to avoid stumbling upon them.

UI Changes

When the UI changes the detected elements usually change. They may be moved to a new position, disappear entirely, or be populated with new data.

Thus the elements you saved into a variable may get invalid. Be aware of this fact and update your elements when you need an updated version.

Why Is the Sorting All Jumbled Up sometimes?

The get() command returns an ordered list based on the "distance" of the elements. The distance is not entirely based upon physical distance, but also takes into consideration the similarity when you use withText(). It also considers special cases, for example, modal dialogs. Therefore the order can be nonsensical when you look at it from a human perspective!

Let us demonstrate this with an example. With a relational selector, you can do something like this.

const belowElements = 
  await aui
    .get()
    .text()
    .below()
    .button()
    .contains()
    .text()
    .withText('Submit')
    .exec();

Take this example from our practice page:

If you try to get everything above the Submit button, as shown above, you will get this output. The first two elements make sense but the third element seems to be out of order:

[
      DetectedElement {
        name: 'TEXT',
        text: 'Enter your mobile number',
        colors: [ 'white', 'gray', 'gray' ],
        bndbox: BoundingBox {
          xmin: 353.01884765625,
          ymin: 756.9289620535715,
          xmax: 549.1419642857143,
          ymax: 776.7444475446429
        }
      },
      DetectedElement {
        name: 'TEXT',
        text: 'Mobile Number',
        colors: [ 'white', 'gray', 'gray' ],
        bndbox: BoundingBox {
          xmin: 332.73348214285716,
          ymin: 721.6995535714286,
          xmax: 449.13929966517856,
          ymax: 736.7938058035714
        }
      },
      DetectedElement {
        name: 'TEXT',
        text: 'User Email',
        colors: [ 'white', 'gray', 'gray' ],
        bndbox: BoundingBox {
          xmin: 331.9736258370536,
          ymin: 446.0734654017857,
          xmax: 419.82706473214284,
          ymax: 471.12338169642857
        }
      },
      DetectedElement {
        name: 'TEXT',
        text: 'Enter your company',
        colors: [ 'white', 'gray', 'gray' ],
        bndbox: BoundingBox {
          xmin: 353.16876395089287,
          ymin: 682.0184430803572,
          xmax: 503.8840262276786,
          ymax: 701.2948660714286
        }
      },
...
]

If you want to sort it by ymin for example, you have to implement a comparator function and pass it to the sort() function of the array.

aboveSubMitElements.sort(
    (element1, element2) => 
        (element1.bndbox.ymin <= element2.bndbox.ymin 
            ? -1 
            : 1))

For sorting based on the distance of the center of the bounding boxes of each element take a look at this article.

Conclusion

If you pay attention to the pitfalls of get() it is a powerful tool to get more out of your automation efforts.

Get Support

If you have a recurring or persisting issue, don’t hesitate to ask the Discord community for help!